MongoDB for Real Time Analytics

Let’s first understand what real-time analytics is and how important it is in today’s world.

The Analytics industry has seen a tremendous growth in the recent past. There is plenty of data available for analysis through channels such as social media, websites etc. Many companies have developed the capability of analysing the data and coming up with business insights.

But going forward, the definitive differentiating factor is going to be how fast or in real-time can a business come up with insights and how quickly can they use it to their advantage.

Let’s take a quick look at some applications of real time analytics:

1. Recommendation Engines
   Consider e-commerce companies like Amazon, Flipkart etc. When a user visits the website and checks products, their recommendation engines analyse the user behaviour and recommend similar products within a few seconds. These systems heavily run real-time analytics on their backend.
2. Predicting machinery failures in manufacturing sectors
   The production units of big manufacturing companies have large number of expensive machines. Analysing the sensor data from these machines in real-time and predicting failures before they occur brings down the maintenance cost.
3. Predicting health risks by real time newborn monitoring
   A real time reliable monitoring of an infant’s health can prove to be a boon for saving lives. The prospects of real-time analytics in any industry are massive!

The Challenges of Real-Time Analytics

• The first challenge is storing the data reliably without any data loss. In real-time analytics, the data is usually coming in at different rates. At one moment the incoming rate could be 1MB/sec and another moment you may have to handle 1GB/sec.
• Every data point could be quite different from one another. The parameters present in every data point may differ vastly from each other. The system has to be flexible and reliable in handling these aspects.
• Next challenge is processing the data in real-time. When the data is coming in at different speeds, the system should have the flexibility to adapt to it.
• In case of traditional database systems and RDBMS systems, we know that data is stored in the form of tables and databases. Any data you want to ingest into the system has to comply with the schema definition. Clearly these systems are not suitable for the variety property of real-time data. Since data is managed as tables, traditional database system take some time in performing ETL, building data structures and ingesting data into databases. As most of the RDBMS systems are single server based, these systems are not scalable.
• When it comes to processing the data real time, RDBMS are not a convenient choice because the formats of the data read from these systems have to be processed before they can be used in real time applications.
• Systems like Hadoop are flexible and scalable, but cannot process data in real time. They are more suitable for batch jobs.

Clearly we need a solution that is reliable, not rigid in terms of the structure of the data, is scalable and is also fast in terms of performing reads and writes. 

This is where MongoDB fits in!

MongoDB is a free, open-source document based NoSQL database system. Some of the capabilities which MongoDB provides and how they are highly suitable for real time analytics:

• It is based on distributed database, making it horizontally scalable.
• It stores data in the form of documents – JSON or BSON. The documents that are inserted are not checked against any schema, which makes the system flexible for handling variety of data.
• Most of the real-time analytics applications are written in Java or a similar language. Converting a JSON/BSON format into a Java object and vice-versa is very quick, since JSON itself is the serialization notation of Java objects. This makes reading and writing into MongoDB very fast and suitable for real-time applications.
• MongoDB has official drivers for a variety of popular programming languages and development environments.

MongoDB is increasingly becoming popular because of these features. It is used widely for real time analytics in sectors like financial services, government, retail, high-tech and much more. Companies like Google, Flipkart, Facebook, Twitter, Bosch, Nokia, and MTV are all among the major users of MongoDB.