What is Hadoop?Modules of hadoop?

 What is Hadoop

Hadoop is an open source framework from Apache and is used to store process and analyze data which are very huge in volume. Hadoop is written in Java and is not OLAP (online analytical processing). It is used for batch/offline processing.It is being used by Facebook, Yahoo, Google, Twitter, LinkedIn and many more. Moreover it can be scaled up just by adding nodes in the cluster.

To learn hadoop course vsit:big data hadoop course

Modules of Hadoop

1. DFS: Hadoop Distributed File System. Google published its paper GFS and on the basis of that HDFS was developed. It states that the files will be broken into blocks and stored in nodes over the distributed architecture.
2. Yarn: Yet another Resource Negotiator is used for job scheduling and manage the cluster.
3. Map Reduce: This is a framework which helps Java programs to do the parallel computation on data using key value pair. The Map task takes input data and converts it into a data set which can be computed in Key value pair. The output of Map task is consumed by reduce task and then the out of reducer gives the desired result.
4. Hadoop Common: These Java libraries are used to start Hadoop and are used by other Hadoop modules.

Hadoop Architecture

The Hadoop architecture is a package of the file system, MapReduce engine and the HDFS (Hadoop Distributed File System). The MapReduce engine can be MapReduce/MR1 or YARN/MR2.

A Hadoop cluster consists of a single master and multiple slave nodes. The master node includes Job Tracker, Task Tracker, NameNode, and DataNode whereas the slave node includes DataNode and TaskTracker.

Hadoop Distributed File System

The Hadoop Distributed File System (HDFS) is a distributed file system for Hadoop. It contains a master/slave architecture. This architecture consist of a single NameNode performs the role of master, and multiple DataNodes performs the role of a slave.

Both NameNode and DataNode are capable enough to run on commodity machines. The Java language is used to develop HDFS. So any machine that supports Java language can easily run the NameNode and DataNode software.

To learn Big data visit:big data online course

NameNode

• It is a single master server exist in the HDFS cluster.
• As it is a single node, it may become the reason of single point failure.
• It manages the file system namespace by executing an operation like the opening, renaming and closing the files.
• It simplifies the architecture of the system.

DataNode

• The HDFS cluster contains multiple DataNodes.
• Each DataNode contains multiple data blocks.
• These data blocks are used to store data.
• It is the responsibility of DataNode to read and write requests from the file system’s clients.
• It performs block creation, deletion, and replication upon instruction from the NameNode.

Job Tracker

• The role of Job Tracker is to accept the MapReduce jobs from client and process the data by using NameNode.
• In response, NameNode provides metadata to Job Tracker.

Task Tracker

• It works as a slave node for Job Tracker.
• It receives task and code from Job Tracker and applies that code on the file. This process can also be called as a Mapper.

MapReduce Layer

The MapReduce comes into existence when the client application submits the MapReduce job to Job Tracker. In response, the Job Tracker sends the request to the appropriate Task Trackers. Sometimes, the TaskTracker fails or time out. In such a case, that part of the job is rescheduled.

Advantages of Hadoop

• Fast: In HDFS the data distributed over the cluster and are mapped which helps in faster retrieval. Even the tools to process the data are often on the same servers, thus reducing the processing time. It is able to process terabytes of data in minutes and Peta bytes in hours.
• Scalable: Hadoop cluster can be extended by just adding nodes in the cluster.
• Cost Effective: Hadoop is open source and uses commodity hardware to store data so it really cost effective as compared to traditional relational database management system.
• Resilient to failure: HDFS has the property with which it can replicate data over the network, so if one node is down or some other network failure happens, then Hadoop takes the other copy of data and use it. Normally, data are replicated thrice but the replication factor is configurable.

History of Hadoop

The Hadoop was started by Doug Cutting and Mike Cafarella in 2002. Its origin was the Google File System paper, published by Google.

Let’s focus on the history of Hadoop in the following steps: –

• In 2002, Doug Cutting and Mike Cafarella started to work on a project, Apache Nutch. It is an open source web crawler software project.
• While working on Apache Nutch, they were dealing with big data. To store that data they have to spend a lot of costs which becomes the consequence of that project. This problem becomes one of the important reason for the emergence of Hadoop.
• In 2003, Google introduced a file system known as GFS (Google file system). It is a proprietary distributed file system developed to provide efficient access to data.
• In 2004, Google released a white paper on Map Reduce. This technique simplifies the data processing on large clusters.
• In 2005, Doug Cutting and Mike Cafarella introduced a new file system known as NDFS (Nutch Distributed File System). This file system also includes Map reduce.
• In 2006, Doug Cutting quit Google and joined Yahoo. On the basis of the Nutch project, Dough Cutting introduces a new project Hadoop with a file system known as HDFS (Hadoop Distributed File System). Hadoop first version 0.1.0 released in this year.
• Doug Cutting gave named his project Hadoop after his son’s toy elephant.
• In 2007, Yahoo runs two clusters of 1000 machines.
• In 2008, Hadoop became the fastest system to sort 1 terabyte of data on a 900 node cluster within 209 seconds.
• In 2013, Hadoop 2.2 was released.
• In 2017, Hadoop 3.0 was released.

Year

Event

2003

Google released the paper, Google File System (GFS).

2004

Google released a white paper on Map Reduce.

2006

• Hadoop introduced.
• Hadoop 0.1.0 released.
• Yahoo deploys 300 machines and within this year reaches 600 machines.

2007

• Yahoo runs 2 clusters of 1000 machines.
• Hadoop includes HBase.

2008

• YARN JIRA opened
• Hadoop becomes the fastest system to sort 1 terabyte of data on a 900 node cluster within 209 seconds.
• Yahoo clusters loaded with 10 terabytes per day.
• Cloudera was founded as a Hadoop distributor.

2009

• Yahoo runs 17 clusters of 24,000 machines.
• Hadoop becomes capable enough to sort a petabyte.
• MapReduce and HDFS become separate subproject.

2010

• Hadoop added the support for Kerberos.
• Hadoop operates 4,000 nodes with 40 petabytes.
• Apache Hive and Pig released.

2011

• Apache Zookeeper released.
• Yahoo has 42,000 Hadoop nodes and hundreds of petabytes of storage.

2012

Apache Hadoop 1.0 version released.

2013

Apache Hadoop 2.2 version released.

2014

Apache Hadoop 2.6 version released.

2015

Apache Hadoop 2.7 version released.

2017

Apache Hadoop 3.0 version released.

2018

Apache Hadoop 3.1 version released.

Apache Commons

 Apache commons has an excellent Utility class, that is : StrSubstitutor.

Example 1:

String testString = "This is a test string @1@ @2@ @3@ that needs to be replaced with value";

Map<String,String> properties = new HashMap<String,String>();

      properties.put("1", "one");

      properties.put("2", "two");

      properties.put("3", "three");

StrSubstitutor substitutor = new StrSubstitutor(properties,"@","@");

System.out.println(substitutor.replace(testString));

Output:

This is a test string one two three that needs to be replaced with value

Example 2:

The default behavior of this class works similar to "Velocity Templates". i.e., ${var_name}.

  Map<String, String> map = new HashMap<String,String>();

      map.put("name", "Veera");

      map.put("city", "Hyderabad");

      

 String text = "My name is ${name}. I am from ${city}.";

      

 StrSubstitutor substitutor = new StrSubstitutor(map);

      

 System.out.println(substitutor.replace(text));

Output:

My name is Veera. I am from Hyderabad.

Become a Master in Big data at OnlineITGuru through big data hadoop training

key concepts about Big Data

 

What is Big data?

Big data is a term that describes the large volume of data – both structured and unstructured – that inundates a business on a day-to-day basis. ... It's what organizations do with the data that matters. Big data can be analyzed for insights that lead to better decisions and strategic business moves.More info visit:big data hadoop course

• 
  

1. Big Data Technologies

Software is needed when processing and analysing huge amounts of information. There are a lot of tools, but the majority of them are based on Hadoop Distributed File System (HDFS), a distributed, scalable, and portable file system.

HDFS is written in Java for Hadoop, a framework which allows applications to work with thousands of nodes and petabytes of information.

2. Real Time or Fast Data

This concept is related with the capacity to obtain data in real time, that is, at the same time it is generated. Streams occur at thousands of times per second.Besides the high frequency data intake, this concept also includes the capacity of processing and decision-making as quickly as possible.

3. NoSQL

NoSQL (“not only SQL”) comprehends a huge rank of database management systems that are distinguished because they do not need static structures such as tables. On the contrary, they are based in other storage systems like key-value, column mapping or graphs.

Unlike the traditional storage methods, NoSQL allows to manage a huge amount of information and avoids bottlenecks. Moreover, it doesn’t require computing, so it enables cost savings.

4. Data Analytics

A fundamental part of working with massive data is Analytics. This process consists of inspecting data series in order to draw conclusions about the information they contain.

Analytics allow companies to custom their services or products. Consequently, Data Analytics has increased decision-making and has made easier the commercial strategy of the companies.

5. Cloud Computing

Cloud is a key sector when working with Big Data, because it allows to process huge amounts of information. Moreover, it is a high-performance system which doesn’t require the installation of specific hardware.

Cloud Computing is, in short, a cheap, fast, comfortable, accessible and secure system. Companies increasingly use it: in 2019 it is expected that, near 100% of the companies will acquire information from the cloud related to their business.

Do you know any other concept related with Big Data? Write your comment below, we will glad to read your remarks.

urther more Topics Go through big data and hadoop online training.

Hadoop vs spark choosing the right framework

 With modern companies relying on a wealth of knowledge to better understand their customers and the industry, innovations such as Big Data are gaining tremendous traction.

Like AI, Big Data has not only landed on the list of top tech trends for 2020, but both start-ups and Fortune 500 businesses are expected to adopt it to enjoy rapid market growth and ensure greater consumer loyalty. Now, while everyone is highly motivated on the one hand to replace their conventional data analytics tools with Big Data-the one that prepares the ground for Blockchain and AI development, they are still puzzled about selecting the right Big Data tool. Apache Hadoop and Spark, the two titans of the Big Data universe, are facing the dilemma of picking.

So, given this idea, today we're going to cover an article about Apache Spark vs Hadoop and help you find out which one is the right choice for your needs,More info big data and hadoop course

But, first, let's give a brief introduction to what Hadoop and Spark are all about.

Apache Hadoop

Apache Hadoop is an open-source, distributed, and Java-based platform allowing users to use simple programming constructs to store and process big data through several device clusters. It consists of different modules that work together to provide an improved experience, which is as follows.

• Common Hadoop

• Distributed Hadoop File System (HDFS)

• YARN of Hadoop

• MapReduce Hadoop

Apache Spark

Apache Spark, however, is an open-source distributed big-data cluster computing platform that is 'easy-to-use' and provides faster services.

Because of the set of possibilities they bring, the two big data architectures are funded by several large corporations.

Benefits of Hadoop Consideration

1. Quick 

One of Hadoop's characteristics that makes it popular in the world of big data is that it is easy.

The method of storage is based on a distributed file system that primarily maps information wherever a cluster is located. Also, on the same server, data, and software used for data processing are typically available, making data processing a hassle-free and quicker task.

Hadoop was found to be able to process terabytes of unstructured data in just a few minutes, while petabytes can be processed in hours.

2. FLEXIBLE

Hadoop provides high-end versatility, unlike conventional data processing tools.

It helps organizations to collect data from various sources (such as social media, emails, etc.), work with different types of data (both structured and unstructured), and gain useful insights for various purposes (such as log processing, consumer campaign research, detection of fraud, etc.).

3. Scaleable

Another benefit of Hadoop is that it is incredibly scalable. Unlike conventional relational database systems (RDBMS), the platform allows organizations to store and distribute massive data sets from hundreds of parallel-operating servers.

4. Cost-Successful

When compared to other big data analytics software, Apache Hadoop is much cheaper. This is because no specialized machine is required; it runs on a commodity hardware group. Also, in the long run, it is easier to add more nodes.

In other words, one case easily increases nodes without suffering from any downtime of requirements for pre-planning.

5. High performance

Data is stored in a distributed way in the case of the Hadoop system, such that a small job is separated into several pieces of data in parallel. This makes it possible for companies to get more jobs completed in less time, resulting in higher throughput eventually.

6. Failure-resilient

Last but not least, Hadoop provides options for high fault tolerance that help to minimize the effects of failure. It stores a replica of each block, which allows data to be retrieved if any node goes down.

Apache Spark System benefits

1. In Nature Complex

As Apache Spark provides about 80 high-level operators, it can be dynamically used for data processing. The best Big Data tool to create and manage parallel apps can be considered.

2. Strong ones

It can handle numerous analytics challenges due to its low-latency in-memory data processing capability and availability of different built-in libraries for machine learning and graph analytics algorithms. This makes it a good business preference for big data to go with.

3. Advanced Analytics Section

Another distinctive aspect of Spark is that not only 'MAP' and 'reduce' are promoted, but also Machine Learning (ML), SQL queries, graph algorithms, and data streaming are enabled. This makes it appropriate to enjoy advanced analytics.

4. The reusability

Unlike Hadoop, it is possible to reuse Spark code for batch processing, run ad-hoc stream state queries, join streams against historical data, and more.

5. Production of Real-time Stream

Another benefit of going with Apache Spark is that it allows real-time information handling and processing.

6. Multilingual Assistance

Last but not least, several coding languages, including Java, Python, and Scala, support this Big Data Analytics tool.

Apache Spark vs Apache Hadoop

So, let's wait no more and head for their comparison to see which one is leading the battle of 'Spark vs Hadoop.'

1. Architecture in the Spark and Hadoop

The latter leads when it comes to Spark and Hadoop architecture, even when both function in a distributed computing environment.

This is because Hadoop's architecture has two primary components, HDFS (Hadoop Distributed File System) and YARN (Yet Another Resource Negotiator), unlike Spark. Here, HDFS manages massive data storage through different nodes, while YARN takes care of processing tasks through resource allocation and frameworks for job scheduling. In order to provide better solutions for services such as fault tolerance, these components are then further divided into more components.

2. Simplicity of Use

In their development environment, Apache Spark helps developers to implement different user-friendly APIs, such as Scala, Python, R, Java, and Spark SQL. It also comes loaded with an interactive mode which supports users as well as developers. It makes it easy to use and has a low learning curve.

Whereas, it provides add-ons to assist users while talking about Hadoop, but not an interactive mode. In this 'big data' battle, this causes Apache Spark to win over Hadoop.

3. Tolerance to Fault and Defense

Although both Apache Spark and Hadoop MapReduce have equipment for fault tolerance, the latter wins the fight.

This is because if a process crashes in the middle of the Spark environment procedure, one must start from scratch. But, when it comes to Hadoop, from the moment of the crash itself, they will proceed.

4. Performance

The former wins over the latter when it comes to considering the performance of Spark vs. MapReduce.

The Apache Spark device will run 10 times faster on the disc and 100 times faster on the memory. This allows 100 TB of data to be handled 3 times faster than Hadoop MapReduce.

5. Processing Data

Data processing is another aspect to remember during the Apache Spark vs Hadoop comparison.

While Apache Hadoop only provides an opportunity for batch processing, the other big data platform allows interactive, iterative, stream, graph, and batch processing to operate. Anything that shows that for better data processing facilities, Spark is a better choice to go for.

6. Compatibility

Spark and Hadoop MapReduce are somewhat similar in their compatibility.

Although both big data systems often serve as standalone applications, they can also run together. Spark can run on top of Hadoop YARN effectively, while Hadoop can combine with Sqoop and Flume easily. Both accept the data sources and file formats of each other because of this.

7. Security 

Various protection features such as event logging and the use of java servlet philtres for protecting web UIs are loaded into the Spark environment. It also promotes authentication through shared secrets and, when integrated with YARN and HDFS, can leverage the potential of HDFS file permissions, inter-mode encryption, and Kerberos.

Hadoop, on the other hand, supports Kerberos authentication, third-party authentication, traditional file permissions, and access control lists, and more, delivering better security results finally. So, the latter leads when considering the comparison of Spark vs. Hadoop in terms of defense.

8. Cost-Efficiency

When Hadoop and Apache Spark are compared, the former needs more disc memory, while the latter requires more RAM. Also, in contrast to Apache Hadoop, since Spark is very recent, developers working with Spark are rarer.

It makes partnering with Spark a costly affair. In other words, when one focuses on Hadoop vs. Spark cost, Hadoop provides cost-effective solutions.

9. Scope of Business

Although both Apache Spark and Hadoop are supported by large corporations and have been used for various purposes, in terms of business reach, the latter leads.

Conclusion

I hope you reach to a conclusion about Hadoop and spark. You can learn more through Big Data Online Training

What is Hadoop Ecosystem?

 Hadoop Ecosystem?

Core Hadoop ecosystem is nothing but the different components that are built on the Hadoop platform directly. However, there are a lot of complex interdependencies between these systems.

To know more about EcoSystem visit:Big data haoop course tutorials Blog

Before starting this Hadoop ecosystem tutorial, let’s see what we will be learning in this tutorial:

• What is Hadoop Ecosystem?
• HDFS
• YARN
• MapReduce
• Apache Pig
• Apache Hive
• Apache Ambari
• Mesos
• Apache Spark
• Tez
• Apache HBase
• Apache Storm
• Oozie
• ZooKeeper
• Data Ingestion

There are so many different ways in which you can organize these systems, and that is why you’ll see multiple images of the ecosystem all over the Internet. However, the graphical representation given below seems to be the best representation so far.

The light blue-colored boxes you see are part of Hadoop, and the rest of them are just add-on projects that have come out over time and integrated with Hadoop in order to solve some specific problems. So, let’s now talk about each one of these.

To learn Hadoop and Bigdata training visit:big data online training

HDFS

Starting from the base of the Hadoop ecosystem, there is HDFS or Hadoop Distributed File System. It is a system that allows you to distribute the storage of big data across a cluster of computers. That means, all of your hard drives look like a single giant cluster on your system. That’s not all; it also maintains the redundant copies of data. So, if one of your computers happen to randomly burst into flames or if some technical issues occur, HDFS can actually recover from that by creating a backup from a copy of the data that it had saved automatically, and you won’t even know if anything happened. So, that’s the power of HDFS, i.e., the data storage is in a distributed manner having redundant copies.

YARN

Next in the Hadoop ecosystem is YARN (Yet Another Resource Negotiator). It is the place where the data processing of Hadoop comes into play. YARN is a system that manages the resources on your computing cluster. It is the one that decides who gets to run the tasks, when and what nodes are available for extra work, and which nodes are not available to do so. So, it’s like the heartbeat of Hadoop that keeps your cluster going.

MapReduce

One interesting application that can be built on top of YARN is MapReduce. MapReduce, the next component of the Hadoop ecosystem, is just a programming model that allows you to process your data across an entire cluster. It basically consists of Mappers and Reducers that are different scripts, which you might write, or different functions you might use when writing a MapReduce program. Mappers have the ability to transform your data in parallel across your computing cluster in a very efficient manner; whereas, Reducers are responsible for aggregating your data together. This may sound like a simple model, but MapReduce is very versatile. Mappers and Reducers put together can be used to solve complex problems. We will talk about MapReduce in one of the upcoming sections of this Hadoop tutorial.

Apache Pig

Next up in the Hadoop ecosystem, we have a technology called Apache Pig. It is just a high-level scripting language that sits on top of MapReduce. If you don’t want to write Java or Python MapReduce codes and are more familiar with a scripting language that has somewhat SQL-style syntax, Pig is for you. It is a very high-level programming API that allows you to write simple scripts. You can get complex answers without actually writing Java code in the process. Pig Latin will transform that script into something that will run on MapReduce. So, in simpler terms, instead of writing your code in Java for MapReduce, you can go ahead and write your code in Pig Latin which is similar to SQL. By doing so, you won’t have to perform MapReduce jobs. Rather, just writing a Pig Latin code will perform MapReduce functions.

Hive

Now, in the Hadoop ecosystem, there comes Hive. It also sits on top of MapReduce and solves a similar type of problem like Pig, but it looks more like a SQL. So, Hive is a way of taking SQL queries and making the distributed data sitting on your file system somewhere look like a SQL database. It has a language known as Hive SQL. It is just a database in which you can connect to a shell client and ODBC (Open Database Connectivity) and execute SQL queries on the data that is stored on your Hadoop cluster even though it’s not really a relational database under the hood. If you’re familiar with SQL, Hive might be a very useful API or interface for you to use.

Apache Ambari

Apache Ambari is the next in the Hadoop ecosystem which sits on top of everything and gives you a view of your cluster. It is basically an open-source administration tool responsible for tracking applications and keeping their status.  It lets you visualize what runs on your cluster, what systems you’re using, and how much resources are being used. So, Ambari lets you have a view into the actual state of your cluster in terms of the applications that are running on it. It can be considered as a management tool that will manage the monitors along with the health of several Hadoop clusters.

Mesos

Mesos isn’t really a part of Hadoop, but it’s included in the Hadoop ecosystem as it is an alternative to YARN. It is also a resource negotiator just like YARN. Mesos and YARN solve the same problem in different ways. The main difference between Mesos and YARN is in their scheduler. In Mesos, when a job comes in, a job request is sent to the Mesos master, and what Mesos does is it determines the resources that are available and it makes offers back. These offers can be accepted or rejected. So, Mesos is another way of managing your resources in the cluster.

What is Big Data Hadoop? Enroll in our Big Data Hadoop Training now and kick-start your career!

Apache Spark

Spark is the most interesting technology of this Hadoop ecosystem. It sits on the same level as MapReduce and right above Mesos to run queries on your data. It is mainly a real-time data processing engine developed in order to provide faster and easy-to-use analytics than MapReduce. Spark is extremely fast and is under a lot of active development. It is a very powerful technology as it uses the in-memory processing of data. If you want to efficiently and reliably process your data on the Hadoop cluster, you can use Spark for that. It can handle SQL queries, do Machine Learning across an entire cluster of information, handle streaming data, etc.

Tez

Tez is similar to Spark and is next in the Hadoop ecosystem it uses some of the same techniques as Spark. It tells you what MapReduce does as it produces a more optimal plan for executing your queries. Tez, when used in conjunction with Hive, tends to accelerate Hive’s performance. Hive is placed on top of MapReduce, but you can place it on top of Tez, as Hive through Tez can be a lot faster than Hive through MapReduce. They are both different means of optimizing queries together.

What is Big Data? If you have more queries related to Big Data Hadoop, do post them on Big Data Hadoop and Spark Community!

Apache HBase

Next up in the Hadoop ecosystem is HBase. It is set on the side, and it is a way of exposing data on your cluster to the transactional platform.  So, it is called the NoSQL database, i.e., it is a columnar data store that is a very fast database and is meant for large transaction rates. It can expose data stored in your cluster which might be transformed in some way by Spark or MapReduce. It provides a very fast way of exposing those results to other systems.

What is Big Data? Learn more from Intellipaat’s Top Big Data Hadoop Interview Questions and crack all Big Data interviews!

Apache Storm

Apache Storm is basically a way of processing streaming data. So, if you have streaming data from sensors or weblogs, you can actually process it in real time using Storm. Processing data doesn’t have to be a batch thing anymore; you can update your Machine Learning models or transform data into the database, all in real time, as the data comes in.

Oozie

Next up in the Hadoop ecosystem, there Oozie. Oozie is just a way of scheduling jobs on your cluster. So, if you have a task that needs to be performed on your Hadoop cluster involving different steps and maybe different systems, Oozie is the way for scheduling all these things together into jobs that can be run on some order. So, when you have more complicated operations that require loading data into Hive, integrating that with Pig, and maybe querying it with Spark, and then transforming the results into HBase, Oozie can manage all that for you and make sure that it runs reliably on a consistent basis.

ZooKeeper

ZooKeeper is basically a technology for coordinating everything on your cluster. So, it is a technology that can be used for keeping track of the nodes that are up and the ones that are down.  It is a very reliable way of keeping track of shared states across your cluster that different applications can use. Many of these applications rely on ZooKeeper to maintain reliable and consistent performance across a cluster even when a node randomly goes down. Therefore, ZooKeeper can be used for keeping track of which the master node is, which node is up, or which node is down. Actually, it’s even more extensible than that.

Data Ingestion

The below-listed systems in the Hadoop ecosystem are focused mainly on the problem of data ingestion, i.e., how to get data into your cluster and into HDFS from external sources. Let’s have a look at them.

• Sqoop: Sqoop is a tool used for transferring data between relational database servers and Hadoop. Sqoop is used to import data from various relational databases like Oracle to Hadoop HDFS, MySQL, etc. and to export from HDFS to relational databases.
• Flume: Flume is a service for aggregating, collecting, and moving large amounts of log data. Flume has a flexible and simple architecture that is based on streaming data flows. Its architecture is robust and fault-tolerant with reliable and recovery mechanisms. It uses the extensible data model that allows for online analytic applications. Flume is used to move the log data generated by application servers into HDFS at a higher speed.
• Kafka: Kafka is also an open-source streaming data processing software that solves a similar problem as Flume. It is used for building real-time data pipelines and streaming apps reducing complexity. It is horizontally scalable and fault-tolerant. Kafka aims to provide a unified, low-latency platform to handle real-time data feeds. Asynchronous communication and messages can be established with the help of Kafka. This ensures reliable communication.