You need a general purpose frame for your cluster. It is because all other types of systems can address a particular use case (e.g., graph processing, machine learning, etc.) and are not adequate by themselves to manage the variety of computing needs that are likely to occur in the organization. In comparison, many of the other frameworks depend on common-purpose frameworks. Also the special-purpose frameworks that don't build on general-purpose frameworks also depend on their bits and pieces.
To learn complete hadoop course visit:hadoop admin online course
Hadoop Frameworks
MapReduce, Spark, and Tez are the traditional frameworks in this category — and newer frameworks, such as Apache Flink, are emerging. Usually MapReduce is still built on clusters as of today. Certain general purpose systems, including input / output formats, rely on bits and pieces from the MapReduce stack. Nonetheless, other frameworks such as Tez or Spark can still be used without having MapReduce built on your cluster.
MapReduce is the most advanced, but it is the slowest, arguably. Both Spark and Tez are DAG systems and don't have the overhead of running a Map often accompanied by a Reduce job; both are more versatile than MapReduce. Spark is one of the Hadoop ecosystem's most successful ventures, and has a lot of traction. It's considered by many to be MapReduce 's successor — I advise you to use Spark over MapReduce whenever possible.
Notably, MapReduce and Spark have different APIs. That means you'll have to rewrite your jobs in Spark because you're using an abstraction system, if you're switching from MapReduce to Spark. It's also worth noting that while Spark is a general-purpose engine built on it with other abstraction systems, it also offers high-level processing APIs. Spark API can therefore also be seen as an abstraction system itself in this way. The amount of time and code needed to write a Spark job is therefore typically much less than writing an equivalent MapReduce job.
At this level, Tez is better suited to building abstraction frameworks as a framework, rather than developing applications using its API.
The important thing to remember is that just because you have a general purpose processing system built on your cluster doesn't mean you need to write all of your processing jobs using the API of that system. In general, it is recommended that abstraction frameworks (e.g., Pig, Crunch, Cascading) or SQL frameworks (e.g., Hive and Impala) be used whenever possible for writing processing jobs (there are two exceptions to this rule, as discussed in the next section).
Hadoop Abstraction and SQL frameworks:
Abstraction frameworks (e.g., Pig, Crunch, and Cascading) and SQL frameworks (e.g., Hive and Impala) minimize the amount of time spent explicitly writing jobs for general-purpose frames in Hadoop.
Abstraction frameworks:
Pig is an abstraction system which can run on MapReduce, Spark, or Tez as seen in the diagram above. Apache Crunch offers a higher level API for performing MapReduce or Spark jobs. Cascading is another abstraction system based on the API, which can run on either MapReduce or Tez.
SQL frameworks:
Hive can run on top of MapReduce or Tez as far as SQL engines go, and work is under way to make Hive run on Spark. There are several SQL engines specially designed for faster SQL, including Impala, Presto and Apache Drill
.
Main points about the benefits of using an Hadoop abstraction or SQL framework:
You can save a lot of time by not needing to use the low-level APIs of general purpose systems to implement common processing tasks.
You may change the underlying frameworks (as required and applicable) for general purpose processing. Coding directly on the frame means that if you decided to change systems, you would have to rewrite your jobs. Using an abstraction or SQL framework which builds on abstracts that away from a generic framework.
Running a job on an abstraction or SQL system needs just a small percentage of the overhead needed for an equivalent job written directly within the framework of general purpose. Also, running a query on a special purpose processing system ( e.g., Impala, or Presto for SQL) is much faster than running an equivalent MapReduce task, as they use a completely different execution model, designed to run fast SQL queries.
Hadoop Two examples, where a general purpose system can be used:
If you have other data (i.e. metadata) information that can not be expressed and exploited in an abstraction or SQL system. For example, if you construct a logical data set in an abstraction or SQL system, let 's assume that your data set is partitioned or ordered in a specific way that you can not express. Using such partitioning / sorting metadata in your job can also speed up processing. In such a scenario it makes sense to program directly inside a general-purpose processing framework's low-level API. In such situations, the time savings of running a job over and over again more than the extra time for growth pays off.
If a general purpose design is better suited to your use case. Generally there is a small percentage of use cases where the analysis is very complex and can not easily be represented in a DSL such as SQL or Pig Latin. Crunch and Cascading should be considered in these situations, but sometimes you can only need to program directly using a general purpose processing system.
If you have chosen to use an abstraction or SQL framework, which specific framework you typically use depends on the in-house knowledge and experience
.
Chart, machine learning, and real-time frameworks/streaming
Generally there is no need to ask users to follow graphs, machine learning and real-time / streaming systems. If a particular case of use is important to you, you will probably need to use a system that will solve the case.
Hadoop Frames in maps
The popular graph processing frameworks include Giraph, GraphX, and GraphLab.
Apache Giraph is a library running on MapReduce.
GraphX is a graph processing library running on Spark.
Graph Lab was a stand-alone, special purpose graph processing system now capable of handling tabular data as well.
Hadoop Frameworks for machine learning
Mahout, MLlib, Oryx, and H2O are widely used as frameworks for machine learning.
Mahout is a library on top of MapReduce, though plans are being made to get Mahout running on Spark.
MLlib is Spark's machine-learning library.
Oryx and H2O are machine learning engines which are stand-alone, special purpose.
Framework for real-time / streaming
Spark Streaming and Storm + Trident are widely used mechanisms for quasi-real-time data processing.
Spark Streaming is a micro-batch streaming research library which is built on top of Spark.
Apache Storm is a special purpose, distributed, real-time computing engine with Trident being used on top of that as an abstraction engine.
Hadoop Partitions:
Partition means dividing a table into coarse grained parts based on a column value like 'information.' It makes it easier to do queries on data slices
Hive Data Samples
What is Partition 's function, then?
Determines how data is stored by the partition keys. Each single value of the Partition key here determines a table partition. For simplicity the Partitions are numbered after the dates. This is close to HDFS's 'Block Splitting.'
Buckets:
Buckets offer the data extra structure which can be used for efficient queries. A combination of two tables bucketed on the same columns can be enforced as a Map-Side Combination, including the join column. Bucketing by the used ID means we can test a user-based query easily by running it on a randomized sample of the total user collection.
Conclusion
The Hadoop ecosystem has evolved to the point where using MapReduce isn't the only way to test Hadoop data anymore. With the variety of options available now, selecting which system to use to process the Hadoop data can be difficult. You can learn more through hadoop admin online training
No comments:
Post a Comment