Setting up Hadoop Multi-Node Cluster

In this lesson, we will discuss Setting up Hadoop Multi-Node Cluster.It is a part of big data and hadoop course training,which is offered by OnlineITguru.

Installing Java
Syntax of java version command

$ java -version

Following output is presented.

java version "1.7.0_71" 
Java(TM) SE Runtime Environment (build 1.7.0_71-b13)
Java HotSpot(TM) Client VM (build 25.0-b02, mixed mode)

Creating User Account
System user account on both master and slave systems should be created to use the Hadoop installation.

# useradd hadoop 
# passwd hadoop

Mapping the nodes
hosts file should be edited in /etc/ folder on all nodes and IP address of each system followed by their host names must be specified.

# vi /etc/hosts

Enter the following lines in the /etc/hosts file.

192.168.1.109 hadoop-master
192.168.1.145 hadoop-slave-1
192.168.56.1 hadoop-slave-2

To learn Big data and hadoop course visit:big data and hadoop online training

Configuring Key Based Login
Ssh should be setup in each node such that they can converse with one another without any prompt for password.

# su hadoop 
$ ssh-keygen -t rsa 
$ ssh-copy-id -i ~/.ssh/id_rsa.pub tutorialspoint@hadoop-master
$ ssh-copy-id -i ~/.ssh/id_rsa.pub hadoop_tp1@hadoop-slave-1 
$ ssh-copy-id -i ~/.ssh/id_rsa.pub hadoop_tp2@hadoop-slave-2
$ chmod 0600 ~/.ssh/authorized_keys 
$ exit
 
Installing Hadoop
Hadoop should be downloaded in the master server.

# mkdir /opt/hadoop 
# cd /opt/hadoop/ 
# wget http://apache.mesi.com.ar/hadoop/common/hadoop-1.2.1/hadoop-1.2.0.tar.gz 
# tar -xzf hadoop-1.2.0.tar.gz 
# mv hadoop-1.2.0 hadoop 
# chown -R hadoop /opt/hadoop 
# cd /opt/hadoop/hadoop/

Configuring Hadoop
Hadoop server must be configured
core-site.xml should be edited.

fs.default.namehdfs://hadoop-master:9000/ dfs.permissions false

hdfs-site.xml file should be editted.

dfs.data.dir /opt/hadoop/hadoop/dfs/name/data true dfs.name.dir /opt/hadoop/hadoop/dfs/name true dfs.replication 1
mapred-site.xml file should be editted.

mapred.job.trackerhadoop-master:9001

JAVA_HOME, HADOOP_CONF_DIR, and HADOOP_OPTS should be edited.

export JAVA_HOME=/opt/jdk1.7.0_17 
export HADOOP_OPTS=-Djava.net.preferIPv4Stack=true
export HADOOP_CONF_DIR=/opt/hadoop/hadoop/conf 

Installing Hadoop on Slave Servers
Hadoop should be installed on all the slave servers

# su hadoop 
$ cd /opt/hadoop 
$ scp -r hadoop hadoop-slave-1:/opt/hadoop
$ scp -r hadoop hadoop-slave-2:/opt/hadoop

Configuring Hadoop on Master Server
Master server should be configured

# su hadoop 
$ cd /opt/hadoop/hadoop

Master Node Configuration

$ vi etc/hadoop/masters  
hadoop-master

Slave Node Configuration

$ vi etc/hadoop/slaves 
hadoop-slave-1
hadoop-slave-2

Name Node format on Hadoop Master

# su hadoop 
$ cd /opt/hadoop/hadoop 
$ bin/hadoop namenode –format
 

11/10/14 10:58:07 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = hadoop-master/192.168.1.109
STARTUP_MSG: args = [-format]
STARTUP_MSG: version = 1.2.0
STARTUP_MSG: build =
https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.2 -r 1479473; compiled by 'hortonfo' on Mon May 6 06:59:37 UTC 2013
STARTUP_MSG: java = 1.7.0_71
************************************************************/
11/10/14 10:58:08 INFO util.GSet: Computing capacity for map BlocksMap editlog=/opt/hadoop/hadoop/dfs/name/current/edits
………………………………………………….
………………………………………………….
………………………………………………….
11/10/14 10:58:08 INFO common.Storage: Storage directory /opt/hadoop/hadoop/dfs/name has been successfully formatted.
11/10/14 10:58:08 INFO namenode.NameNode: SHUTDOWN_MSG: /************************************************************
SHUTDOWN_MSG: Shutting down NameNode at hadoop-master/192.168.1.15
************************************************************/

Hadoop Services
Starting Hadoop services on the Hadoop-Master.

$ cd $HADOOP_HOME/sbin $ start-all.sh

+

To learn complete tutorial visit our blog,big data hadoop course

Addition of a New DataNode in the Hadoop Cluster
Networking
Add new nodes to an existing Hadoop cluster with some suitable network configuration. suppose the following network configuration.
For New node Configuration:

IP address : 192.168.1.103
netmask : 255.255.255.0 
hostname : slave3.in

Adding a User and SSH Access
Add a User
“hadoop” user must be added and password of Hadoop user can be set to anything one wants.

useradd hadoop
passwd hadoop

To be executed on master

mkdir -p $HOME/.ssh
chmod 700 $HOME/.ssh 
ssh-keygen -t rsa -P '' -f $HOME/.ssh/id_rsa 
cat $HOME/.ssh/id_rsa.pub >> $HOME/.ssh/authorized_keys
chmod 644 $HOME/.ssh/authorized_keys 
Copy the public key to new slave node in hadoop user $HOME directory 
scp $HOME/.ssh/id_rsa.pub hadoop@192.168.1.103:/home/hadoop/ 

To be executed on slaves

su hadoop ssh -X hadoop@192.168.1.103

Content of public key must be copied into file “$HOME/.ssh/authorized_keys” and then the permission for the same must be changed.

cd $HOME 
mkdir -p $HOME/.ssh
chmod 700 $HOME/.ssh  
cat id_rsa.pub >>$HOME/.ssh/authorized_keys
chmod 644 $HOME/.ssh/authorized_keys

ssh login must be changed from the master machine. Possibility of ssh to the new node without a password from the master must be verified.

ssh hadoop@192.168.1.103 or hadoop@slave3 

Set Hostname of New Node
Hostname is set in file /etc/sysconfig/network

On new slave3 machine 
NETWORKING=yes 
HOSTNAME=slave3.in

Machine must be restarted or hostname command should be run to a new machine with the respective hostname to make changes effective.
On slave3 node machine:
hostname slave3.in
/etc/hosts must be updated on all machines of the cluster

192.168.1.102 slave3.in slave3

ping the machine with hostnames to check whether it is resolving to IP.

ping master.in 

Start the DataNode on New Node

Datanode daemon should be started manually using $HADOOP_HOME/bin/hadoop-daemon.sh script. Master(NameNode) should join the cluster after being automatically contacted. New node should be added to the conf/slaves file in the master server. New node will be recognized by script-based commands.

Login to new node

su hadoop or ssh -X hadoop@192.168.1.103

HDFS is started on a newly added slave node

./bin/hadoop-daemon.sh start datanode

jps command output must be checked on a new node.

$ jps 
7141 DataNode 
10312 Jps

Removing a DataNodeNode can be removed from a cluster as it is running, without any data loss. A decommissioning feature is made available by HDFS which ensures that removing a node is performed securely.
Step 1Login to master machine user where Hadoop is installed.

$ su hadoop

Step 2Before starting the cluster an exclude file must be configured. A key named dfs.hosts.exclude should be added to our$HADOOP_HOME/etc/hadoop/hdfs-site.xmlfile.
NameNode’s local file system which contains a list of machines which are not permitted to connect to HDFS receives full path by this key and the value associated with it.

dfs.hosts.exclude/home/hadoop/hadoop-1.2.1/hdfs_exclude.txt>DFS exclude
 

Step 3Hosts to decommission are determined.
Additions should be made to file recognized by the hdfs_exclude.txt for every machine to be decommissioned which will prevent them from connecting to the NameNode.

slave2.in

Step 4Force configuration reload.
“$HADOOP_HOME/bin/hadoop dfsadmin -refreshNodes” should be run

$ $HADOOP_HOME/bin/hadoop dfsadmin -refreshNodes

NameNode will be forced to re-read its configuration, this is inclusive of the newly updated ‘excludes’ file. Nodes will be decommissioned over a period of time, allowing time for each node’s blocks to be replicated onto machines which are scheduled to remain active.
jps command output should be checked on slave2.in. DataNode process will shutdown automatically.

Step 5Shutdown nodes.
The decommissioned hardware can be carefully shut down for maintenance after the decommission process has been finished.

$ $HADOOP_HOME/bin/hadoop dfsadmin -report

Step 6Excludes are edited again and once the machines have been decommissioned, they can be removed from the ‘excludes’ file. “$HADOOP_HOME/bin/hadoop dfsadmin -refreshNodes” will read the excludes file back into the NameNode;DataNodes will rejoin the cluster after the maintenance has been completed, or if additional capacity is needed in the cluster again.
To run/shutdown tasktracker

$ $HADOOP_HOME/bin/hadoop-daemon.sh stop tasktracker
$ $HADOOP_HOME/bin/hadoop-daemon.sh start tasktracker

Hadoop 3: Comparison with Hadoop 2 and Spark

The release of Hadoop 3 in December 2017 marked the beginning of a new era for data science. The Hadoop framework is at the core of the entire Hadoop ecosystem, and various other libraries strongly depend on it.

In this article, we will discuss the major changes in Hadoop 3 when compared to Hadoop 2. We will also explain the differences between Hadoop and Apache Spark, and advise how to choose the best tool for your particular task.

To learn complete tutorials visit:online courses for big data and hadoop.

General information

Hadoop 2 and Hadoop 3 are data processing engines developed in Java and released in 2013 and 2017 respectively. Hadoop was created with the primary goal to maintain the data analysis from a disk, known as batch processing. Therefore, native Hadoop does not support the real-time analytics and interactivity.

Spark 2.X is a processing and analytics engine developed in Scala and released in 2016. The real-time analysis of the information was becoming crucial, as many giant internet services strongly relied on the ability to process data immediately. Consequently, Apache Spark was built for live data processing and is now popular because it can efficiently deal with live streams of information and process data in an interactive mode.

Both Hadoop and Spark are open source, Apache 2 licensed.

Level of abstraction and difficulty to learn and use

One of the major differences between these frameworks is the level of abstraction which is low for Hadoop and high for Spark. Therefore, Hadoop is more challenging to learn and use, as the developers must know how to code a lot of basic operations. Hadoop is only the core engine, so using advanced functionality requires plug-in of other components, which makes the system more complicated.

Unlike Hadoop, Apache Spark is a complete tool for data analytics. It has many useful built-in high-level functions that operate with the Resilient Distributed Dataset (RDD) — the core concept in Spark. This framework has many helpful libraries included in the cluster. For example, MLliballows using machine learning, Spark SQL can be used to perform SQL queries, etc.

Hardware and cost

Hadoop works with a disk, so it does not need a lot of RAM to operate. This can be cheaper than having large RAM. Hadoop 3 requires less disk space than Hadoop 2 due to changes in fault-tolerance providing system.

Spark needs a lot of RAM to operate in the in-memory mode so that the total cost can be more expensive than Hadoop.

Support of programming languages

Both versions of Hadoop support several programming languages using Hadoop Streaming, but the primary one is Java. Spark 2.X supports Scala, Java, Python, and R.

Speed

Generally, Hadoop is slower than Spark, as it works with a disk. Hadoop cannot cache the data in memory. Hadoop 3 can work up to 30% faster than Hadoop 2 due to the addition of native Java implementation of the map output collector to the MapReduce.

Spark can process the information in memory 100 times faster than Hadoop. If working with a disk, Spark is 10 times faster than Hadoop.

Security

Hadoop is considered to be more secure than Spark due to the usage of the Kerberos (computer network authentication protocol) and the support of the Access Control Lists (ACL). Spark in its turn provides authentication only with a shared secret password.

Fault tolerance

The fault tolerance in Hadoop 2 is provided by the replication technique where each block of information is copied to create 2 replicas. This means that instead of storing 1 piece of information, Hadoop 2 stores three times more. This raises the problem of wasting the disk space.

In Hadoop 3 the fault tolerance is provided by the erasure coding. This method allows recovering a block of information using the other block and the parity block. Hadoop 3 creates one parity block on every two blocks of data. This requires only 1,5 times more disk space compared with 3 times more with the replications in Hadoop 2. The level of fault tolerance in Hadoop 3 remains the same, but less disk space is required for its operations.

Spark can recover information by the recomputation of the DAG (Directed Acyclic Graph). DAG is formed by vertices and edges. Vertices represent RDDs, and edges represent the operations on the RDDs. In the situation, where some part of the data was lost, Spark can recover it by applying the sequence of operations to the RDDs. Note, that each time you will need to recompute RDD, you will need to wait until Spark performs all the necessary calculations. Spark also creates checkpoints to protect against failures.

YARN version

Hadoop 2 uses YARN version 1. YARN (Yet Another Resource Negotiator) is the resource manager. It manages the available resources (CPU, memory, disk). Besides, YARN performs Jobs Scheduling.

YARN was updated to version 2 in Hadoop 3. There are several significant changes improving usability and scalability. YARN 2 supports the flows — logical groups of YARN application and provides aggregating metrics at the level of flows. The separation between the collection processes (writing data) and the serving processes (reading data) improves the scalability. Also, YARN 2 uses Apache HBase as the primary backing storage.

Spark can operate independently, on a cluster with YARN, or with Mesos.

Number of NameNodes

Hadoop 2 supports single active NameNode and single standby NameNode for the entire Namespace while Hadoop 3 works with multiple standby NameNodes.

Spark runs Driver Program on the master node which manages SparkContext.

File systems

The main Hadoop 2 file system is HDFS — Hadoop Distributed File System. The framework is also compatible with several other file systems, Blob stores like Amazon S3 and Azure storage, as well as alternatively distributed file systems.

Hadoop 3 supports all the file systems, as Hadoop 2. In addition, Hadoop 3 is compatible with Microsoft Azure Data Lake and Aliyun Object Storage System.

Spark supports local file systems, Amazon S3 and HDFS.

For your convenience, we created a table that summarises all of the above information and presents a brief comparison of the key parameters of the two versions of Hadoop and Spark 2.X.

Conclusion

The major difference between Hadoop 3 and 2 is that the new version provides better optimization and usability, as well as certain architectural improvements.

Spark and Hadoop differ mainly in the level of abstraction. Hadoop was created as the engine for processing large amounts of existing data. It has a low level of abstraction that allows performing complex manipulations but can cause learning and managing difficulties. Spark is easier and faster, with a lot of convenient high-level tools and functions that can simplify your work. Spark operates on top of Hadoop and has many good libraries like Spark SQL or machine learning library MLlib. To summarize, if your work does not require special features, Spark can be the most reasonable choice.

To learn complete big data course visit:big data and hadoop course.

Big Data Development And Implementation

With happy, satisfied customers around the globe, BrickRed Systems’ information technology (IT) specialists know how important and how immediate the field of Big Data has become. Incredibly large data sets in mobile computing, social innovation, social media, artificial intelligence, machine learning and the overall internet of things (IoT) are being created in different forms in variety, velocity, and in volume from terabytes to zettabytes. To learn Big Data development visit:big data online training Information is being generated from all over the globe 24/7 and is gathered, computed, analyzed – all to bring more meaning to our lives, our world, our businesses – so that organizations can make smarter, more informed decisions, which will make our world an easier place to live in. BrickRed Systems’ IT technicians have the expertise and finesse that competent professionals own. They are always ready to help and assist you with any of your IT or Big Data needs. Our customers are important to us, and we strive for that 100 percent customer satisfaction, offering complete transparency in our work. Flexibility is essential in any good working relationship. That’s why we offer flexible implementation by using the eight steps in the waterfall model: conception, initiation, analysis, design, construction, testing, implementation, and maintenance – or the Agile process that offers these key steps and more: work is divided into smaller sections, keeping up-to-date on any changes that develop and keeping abreast of any problems, incremental development, iterative development, milestone retrospective, team and more. Our clients have consistently received successful implementation projects, and, therefore, we have the confidence that we can best assist you in development, service, quality, training and maintenance in your big data implementation needs. We are dedicated to offer our commitments on time and at affordable prices.

Big Data Development Service

• Customized application development based on Hadoop® MapReduce Program
• Development and modeling through algorithmic data
• Custom code and reduced code through data map developments

Big Data Implementation

• Using professional platforms, such as Hive-Project and Hadoop®, to provide social media or web log analysis
• Metadata management
• Analytic and visualization services – text mining, statistical programming, etc.

BrickRed Systems’ experts understand how valuable your time is, and we are committed to providing you with excellent service – on time, every time. We strive to gain your trust with our exceptional service because gaining your long-term relationship is our ultimate goal. To help your business reach its objectives and exceed its goals, we have flexible packages at affordable prices so that you can comfortably choose the service that will best meet your needs. To learn Big data please visit the blog:big data and hadoop course.