Data Lakes

The term “Data Lake” itself is quite self-explanatory. It refers to a vast repository where lots of raw data is stored and managed. We can also say that a data lake holds the dumped data in its native format to be accessed as and when required.

The basic idea of a Data Lake is that it uses a flat architecture to store the data which is a different scenario as compared to a hierarchical data warehouse that stores data in file or folders. Data are set as elements in a Data Lake. Each of these Data Elements is assigned a unique identifier and all these elements are attached with a set of extendable ‘metadatatags’. These metadatatags allow for the easy searching of the data. Whenever a business need comes up that needs specific data from the lake for insights, the tag helps in querying for relevant data and creates a subset from the extended metadatatags. This smaller set of data can be then analysed to help answer the business question. Such storing and tagging of data, by allocating a specific and unique identifier to it, speeds the processing of the data.

Data Lake with Hadoop-Oriented Object Storage

Data Lakes are more commonly associated with Big Data for obvious reasons. Data Lakes can hold large amounts of data and both Big Data and Data Lake have object-oriented storage. That is why the term Data Lake is often associated with Hadoop-oriented object storage. When Data Lake merges with big data concept, the organisation’s data from the lake is loaded into the Hadoop platform. As such this object storage makes the access easier because of the unique identifier tagging of the data in the Lake. The business analytics and data mining tools are applied to the data where it resides on Hadoop’s cluster nodes of commodity computers. With the tools, the data can be then extracted for business requirement. Also, the Data Lake is accessible for cloud computing in the form of object-based storage and same for Hadoop nodes. 

However there is some negative rumblings about the concept of Data Lakes in the industry as it is often perceived to be a product that supports Hadoop but is inflexible to use with other products. Nevertheless, Data Lakes are growing in popularity, and research is continuing, in an effort to make the integration compatible with more products.

In the meantime however, the term Data Lake is being accepted as a way to describe any large data pool in which the schema and data requirements are not defined until the data is queried. This helps for accessing the data fast due to the unique identifier only when it is queried and hence helps with wastage of time and unnecessary storage of data information.

Related Articles:

How Important is The Human Element in Big Data?

Going Back to Basics. Understanding Big Data.