Welcome to the lesson ‘Spark Algorithm’ of Big Data Hadoop Tutorial which is a part of 'big data and hadoop online training' offered by OnlineItGuru.
In this lesson, you will learn about the kinds of processing and analysis that Spark supports. You will also learn about Spark machine learning and its applications, and how GraphX and MLlib work with Spark.
Let us look at the adjectives of this lesson in the next section.
Objectives
After completing this lesson, you will be able to:
Describe the kinds of processing and analysis that Spark supports
Describe the applications of Spark machine learning
Explain how GraphX and MLlib work with Spark.
When does Spark work best?
Spark is an open-source cluster-computing framework and provides up to 100 times faster performance for a few applications with in-memory primitives as compared to the disk-based, two-stage MapReduce of Hadoop.
Fast performance makes it suitable for Apache Spark machine learning algorithms, as it allows programs to load data into the memory of a cluster and query the data constantly.
Spark works best in the following use cases:
There is a large amount of data that requires distributed storage
There is intensive computation that requires distributed computing
There are instances where iterative algorithms are present that requires in-memory processing and pipelining.
Check out the big data hadoop course.
Uses of Apache Spark
Here are some examples where Spark is beneficial:
Spark helps you answer the question for risk analysis, “How likely is it that this borrower will pay back a loan?”
Spark can answer questions on recommendation such as, “Which products will this customer enjoy?”
It can help predict events by answering questions such as, “How can we prevent service outages instead of simply reacting to them?”
Spark helps to classify by answering the question, “How can we tell which mail is spam and which is legitimate?”
With these examples in mind, let’s delve more into the world of Spark. In the following sections, you will learn about how an iterative algorithm runs in Spark.
Spark: Iterative algorithm
Spark is designed for systems that are required to implement an iterative algorithm.
Let’s look at PageRank Algorithm, which is an iterative algorithm.
PageRank Algorithm: FEATURES
PageRank is an example of an iterative algorithm. It is one of the methods used to determine the relevance or importance of a webpage. It gives web pages a ranking score based on links from other pages.
A higher rank is given when the links are from many pages, and when the links are from high ranked pages. The algorithm outputs a probability distribution used to represent the likelihood that a person clicking on the links will arrive at a particular page.
PageRank is important because it is a classic example of big data analysis, like WordCount. As there is a lot of data, an algorithm is required that is distributable and scalable.
PageRank is iterative, which means the more the iteration, the better is the answer.
Let’s look at how the PageRank algorithm works.
PageRank Algorithm: WORKING
Start each page with a rank of 1.0. On each iteration, a page contributes to its neighbors its own rank divided by the number of its neighbors.
Here is the function:
contribp = rankp / neighborsp
You can then set each page’s new rank based on the sum of its neighbors contribution. The function is given here:
new-rank = Σcontribs * .85 + .15
Each iteration incrementally improves the page ranking as shown in the below diagram.
In the next section, you will learn about graph-parallel system.
Graph Parallel System
Today, big graphs exist in various important applications, be it web, advertising, or social networks. Examples of a few of such graphs are presented on the sections.
These graphs allow the users to perform tasks such as target advertising, identify communities, and decipher the meaning of documents. This is possible by modeling the relations between products, users, and ideas.
As the size and significance of graph data is growing, various new large-scale distributed graph-parallel frameworks, such as GraphLab, Giraph, and PowerGraph, have been developed. With each framework, a new programming abstraction is available. These abstractions allow the users to explain graph algorithms in a compact manner.
They also explain the related runtime engine that can execute these algorithms efficiently on distributed and multicore systems. Additionally, these frameworks abstract the issues of the large-scale distributed system design.
Therefore, they are capable of simplifying the design, application, and implementation of the new sophisticated graph algorithms to large-scale real-world graph problems.
Let’s look at a few limitations of the Graph-parallel system.
Limitations of Graph-Parallel System
Firstly, although the current graph-parallel system frameworks have various common properties, each of them presents different graph computations. These computations are custom-made for a specific graph applications and algorithms family or the original domain.
Secondly, all these frameworks depend on different runtime. Therefore, it is tricky to create these programming abstractions.
And finally, these frameworks cannot resolve the data Extract, Transform, Load or ETL issues and issues related to the process of deciphering and applying the computation results.
The new graph-parallel system frameworks, however, have built-in support available for interactive graph computation.
Next, you will learn about Spark GraphX.
What is Spark GraphX?
Spark GraphX is a graph computation system running in the framework of the data parallel system which focuses on distributing the data across different nodes that operate on the data in parallel.
It addresses the limitations posed by the graph parallel system.
Spark GraphX is more of a real-time processing framework for the data that can be represented in a graph form.
Spark GraphX extends the Resilient Distributed Dataset or RDD abstraction and hence, introduces a new feature called Resilient Distributed Graph or RDG.
Spark GraphX simplifies the graph ETL and analysis process substantially by providing new operations for viewing, filtering, and transforming graphs.
Features of Spark GraphX
Spark GraphX has many features like:
Spark GraphX combines the benefits of graph parallel and data parallel systems as it efficiently expresses graph computations within the framework of the data parallel system.
Spark GraphX distributes graphs efficiently as tabular data structures by leveraging new ideas in their representations.
It uses in-memory computation and fault tolerance by leveraging the improvements of the data flow systems.
Spark GraphX also simplifies the graph construction and transformation process by providing powerful new operations.
With the use of these features, you can see that Spark is well suited for graph-parallel algorithm.
In the next topic, let's look at machine learning, which explores the study and construction of algorithms that can learn from and make predictions on data, it's applications, and standard machine learning clustering algorithms like k means algorithm.
What is Machine Learning?
It is a subfield of artificial intelligence that has empowered various smart applications. It deals with the construction and study of systems that can learn from data.
For instance, machine learning can be used in medical diagnosis to answer a question such as "Is this cancer?"
It can learn from data and help diagnose a patient as a sufferer or not. Another example is fraud detection where machine learning can learn from data and provide an answer to a question such as "Is this credit card transaction fraudulent?"
Therefore, the objective of machine learning is to let a computer predict something. An obvious scenario is to predict an event in future. Apart from this, it can also predict unknown things or events.
This means that something that has not been programmed or inputted into it. In other words, computers act without being explicitly programmed. Machine learning can be seen as building blocks to make computers behave more intelligently.
History of Machine Learning
In 1959, Arthur Samuel defined machine learning as, "A field of study that gives computers the ability to learn without being explicitly programmed." Later, in 1997 Tom Mitchell gave another definition that proved more useful for engineering purposes.
"A machine learning computer program is said to learn from experience E concerning some class of tasks T and performance measure P. If its performance at tasks in T as measured by P improves with experience E."
As data plays a big part in machine learning, it's important to understand and use the right terminology when talking about data. This will help you to understand machine learning algorithms in general.
Common Terminologies in Machine Learning
The common terminologies in Machine Learning are :
Vector Feature
Samples
Feature Space
Labeled Data
Let's begin with vector feature.
Vector Feature
It is an n-dimensional vector of numerical features that represent some object. It is a typical setting which is provided by objects or data points collection.
Each item in this collection is described by some features such as categorical and continuous features.
Samples
Samples are the items to process. Examples include a row in a database, a picture, or a document.
Feature Space
Feature space refers to the collection of features that are used to characterize your data.
In other words, feature space refers to the n dimensions where your variables live.
If a feature vector is a vector length L, each data point can be considered being mapped to a D dimensional vector space called the feature space.
Labeled Data
Labeled data is the data with known classification results. Once a labeled data set is obtained, you can apply machine learning models to the data so that the new unlabeled data can be presented to the model.
A likely label can be guessed or predicted for that piece of unlabeled data.
Features Space: Example
Here is an example of features of two apples: one is red, and the other is green.
In machine learning, an object is used. In this example, the object is the Apple. The features of the object, which is Apple, include color, type, and shape.
In the first instance, the color is red, the type is fruit, and the shape is round.
In the second instance, there is a change in the feature described in the color of the Apple, which is now green.
Applications of Machine Learning
As machine learning is related to data mining, it is a way to fine-tune a system with tunable parameters. It can also identify patterns that humans tend to overlook or are unable to find quickly among large amounts of data.
As machine learning is transforming a wide variety of industries, it is helping companies make discoveries and identify and remediate issues faster. Here are some interesting real-world applications of machine learning.
Speech recognition
Machine learning has improved speech recognition systems. Machine learning uses automatic speech recognition or ASR as a large-scale realistic application to rigorously test the effectiveness of a given technique.
Effective web search
Machine learning techniques such as naive bayes extract the categories or a broad range of problems from the user entered a query to enhance the results quality. This is based on query logs to train the model.
Recommendation systems
According to Wikipedia, recommendation systems are a subclass of information filtering system that seeks to predict the rating or preference that a user would give to an item.
Recommendation systems have been using machine learning algorithms to provide users with product or service recommendations.
Here are some more applications of machine learning.
Computer vision: Computer vision, which is an extension of AI and cognitive neuroscience, is the field of building computer algorithms to automatically understand the contents of images.
By collecting a training data set of images and hand labeling each image appropriately, you can use ML algorithm to work out which patterns of pixels are relevant to your recognition tasks and which are nuisance factors.
Information retrieval: Information retrieval systems provide access to millions of documents from which users can recover any one document by providing an appropriate description.
Algorithms that mine documents are based on machine learning. These learning algorithms use examples, attributes, and values which information retrieval systems can supply in abundance.
Fraud detection: Machine learning aids financial leaders to understand their customer transactions better and to rapidly detect fraud.
It helps in extracting and analyzing a variety of different data sources to identify anomalies in real time to stop fraudulent activities as they happen.
No comments:
Post a Comment