Implementation of Spark Applications Tutorial

In this lesson, we will discuss about Implementation of Spark Applications Tutorial

Welcome to the  lesson ‘Implementation of Spark Applications’ of Big Data Hadoop Tutorial which is a part of 'big data training’ offered by OnlineITGURU.

In this lesson, we will discuss how to implement Spark application. You will also be introduced to SparkContext and the Spark application cluster options.

Let us first look at the objectives of this lesson.

Objectives

After completing this lesson, you will be able to:

• Explain the difference between Spark Shell and Spark applications.
• Describe SparkContext.
• Explain the three supported cluster resource managers of Spark applications - Hadoop YARN, Spark Standalone, and Apache Mesos.
• List the steps to run Spark application.
• List the steps for dynamic resource allocation.
• Explain the different configurations related to Spark application.

You can check the Course Preview of big data hadoop course

Spark Shell vs. Spark Applications

The table below explains how spark shell differs from Spark Applications.

Spark Shell	Spark Applications
Allows interactive exploration and manipulation of data Example: REPL using Python or Scala	Run as independent programs Examples: Python, Scala, or Java, and jobs such as ETL processing, and streaming

The SparkContext

Every Spark program needs a SparkContext, which is the main entry point to Spark for a Spark application. The interactive Spark Shell creates it for you. The spark context is explained in the below diagram.

The first thing that any Spark program needs to do is to create a SparkContext object, which informs Spark how to access a cluster through a resource manager.

In order to create a SparkContext, begin by building a SparkConf object, which contains information about your application.

You can create SparkContext in Spark application. In a Spark shell, a special SparkContext is already created for you, in the variable called sc. When you want to terminate the program, you can call sc.stop.

For example, when you want to close files or network connections after you are done with them, you can call sc.stop, which lets the spark master know that your application is finished consuming resources.

Here is an example where you can see a SparkContext being initiated in Scala.

Here, the number of lines containing “a” and lines containing “b” are being calculated.

In the following sections, let’s learn about the different options of Spark application clusters and supported cluster resource managers.

Spark Application Cluster Options

Spark applications can run locally without any distributed processing, locally with multiple worker threads, and on a cluster.

Local mode

Spark application that runs locally is as shown below.

Local mode is useful for development and testing, while the cluster is preferred for production.

Cluster mode

Spark application that runs on a cluster is shown as below.

Supported Cluster Resource Managers

The three supported cluster resource managers of Spark applications are Hadoop YARN, Spark Standalone, and Apache Mesos.

Hadoop YARN

Hadoop YARN is included in Cloud Distribution Hadoop or CDH. Hadoop YARN is most commonly used for production sites and allows sharing cluster resources with other applications.

Spark Standalone

Spark Standalone is included with Spark. It has limited configurability and scalability but is easy to install and run. It is useful for testing, development, or small systems. However, there is no security support.

Apache Mesos

Apache Mesos was the first platform to be supported by Spark. However, currently, it is not as popular as the other resource managers.

In the next section, we will learn how to run Spark on Hadoop YARN, both in client mode and cluster mode.

Running Spark on YARN: Client Mode (1)

Now that you know what the three supported cluster resource managers are, let’s understand how Spark runs in Hadoop YARN.

As you can see from the diagram, when Spark is run in client mode, the SparkContext runs on the client machine.

The resource manager opens the application master, which, in turn, opens the executors and executes the program.

Running Spark on YARN: Client Mode (2)

In the diagram below, you can see that once the executors finish processing, they return the result to the SparkContext.

Now, let’s look at an example where another client has opened a SparkContext.

Running Spark on YARN: Client Mode (3)

This application has its own application master, which opens its own executors.

Once the executors finish processing, they return the result to the new SparkContext as shown in the below diagram.

Running Spark on YARN: Client Mode (4)

You’ve seen how Spark runs in client mode. Now let’s understand how Spark runs in Cluster mode.

Running Spark on YARN: Cluster-Mode (1)

In case of cluster mode, SparkContext, present in the cluster, opens new executors.

Once the executors finish the task, it returns the value to SparkContext. As mentioned earlier, a Spark application can run in different modes.

In the following sections, let’s see how to run a Spark application locally and starting a Spark shell on a cluster.

Running Spark Application

Let us first understand how to how to run Spark application locally.

Running a Spark Application Locally

To run a Spark application locally, use spark-submit--master to specify the cluster option.

Here are the different local options:

• Use local[*] to run the application locally with as many threads as cores. This is a default option.
• Use local[n] to run the application locally with n threads.
• Use local to run the application locally with a single thread.

Running a Spark Application on a Cluster

To run a Spark application on a cluster, use spark-submit--master to specify the cluster option. The different cluster options are:

• Yarn-client
• Yarn-cluster
• spark://masternode:port which is used in Spark Standalone
• mesos://masternode:port which is used in Apache Mesos.

Starting a Spark Application on a Cluster

Not only can you run a Spark application on a cluster, you can also run a Spark shell on a cluster. Both pyspark and spark-shell have a --master option.

Spark shell needs to be run in Hadoop YARN client mode only so that the machine you are running on acts as the driver.

To start the Spark shell, use the Spark or Apache Mesos cluster manager URL.

You can run local[*] with as many threads as cores by default. You can also run local[n] locally with n worker threads, and run locally without distributed processing.

Summary

Let’s now summarize what we learned in this lesson.

• The Spark Shell allows interactive exploration and manipulation of data, while Spark applications run as independent programs.
• Every Spark program needs a SparkContext.
• The interactive Spark Shell creates it for the user.
• Spark applications can run locally without any distributed processing, locally with multiple worker threads, and on a cluster.
• The three supported cluster resource managers of Spark applications are Hadoop YARN, Spark Standalone, and Apache Mesos.
• Spark provides numerous properties for configuring an application such as spark.master, spark.app.name, spark.local.dir, spark.ui.port, spark.executor.memory, and spark.driver.memory.
• Spark applications can be configured declaratively or programmatically.

Want to check our Big Data Hadoop and Spark Developer Certification course? big data and hadoop training

Conclusion

This concludes the lesson of “Implementation of Spark Applications.”