MapReduce - API
In this chapter, we will take a close look at the classes and their methods that are involved in the operations of MapReduce programming. We will primarily keep our focus on the following −
- JobContext Interface
- Job Class
- Mapper Class
- Reducer Class
The JobContext interface is the super interface for all the classes, which defines different jobs in MapReduce. It gives you a read-only view of the job that is provided to the tasks while they are running.
The following are the sub-interfaces of JobContext interface.
|1.||MapContext<KEYIN, VALUEIN, KEYOUT, VALUEOUT>
Defines the context that is given to the Mapper.
|2.||ReduceContext<KEYIN, VALUEIN, KEYOUT, VALUEOUT>
Defines the context that is passed to the Reducer.
Job class is the main class that implements the JobContext interface.
The Job class is the most important class in the MapReduce API. It allows the user to configure the job, submit it, control its execution, and query the state. The set methods only work until the job is submitted, afterwards they will throw an IllegalStateException.
Normally, the user creates the application, describes the various facets of the job, and then submits the job and monitors its progress.
Here is an example of how to submit a job −
// Create a new Job Job job = new Job(new Configuration()); job.setJarByClass(MyJob.class); // Specify various job-specific parameters job.setJobName("myjob"); job.setInputPath(new Path("in")); job.setOutputPath(new Path("out")); job.setMapperClass(MyJob.MyMapper.class); job.setReducerClass(MyJob.MyReducer.class); // Submit the job, then poll for progress until the job is complete job.waitForCompletion(true);
Following are the constructor summary of Job class.
|3||Job(Configuration conf, String jobName)|
Some of the important methods of Job class are as follows −
User-specified job name.
Returns the current state of the Job.
Checks if the job is finished or not.
Sets the InputFormat for the job.
Sets the user-specified job name.
Sets the Output Format for the job.
Sets the Mapper for the job.
Sets the Reducer for the job.
Sets the Partitioner for the job.
Sets the Combiner for the job.
The Mapper class defines the Map job. Maps input key-value pairs to a set of intermediate key-value pairs. Maps are the individual tasks that transform the input records into intermediate records. The transformed intermediate records need not be of the same type as the input records. A given input pair may map to zero or many output pairs.
map is the most prominent method of the Mapper class. The syntax is defined below −
map(KEYIN key, VALUEIN value, org.apache.hadoop.mapreduce.Mapper.Context context)
This method is called once for each key-value pair in the input split.
The Reducer class defines the Reduce job in MapReduce. It reduces a set of intermediate values that share a key to a smaller set of values. Reducer implementations can access the Configuration for a job via the JobContext.getConfiguration() method. A Reducer has three primary phases − Shuffle, Sort, and Reduce.
Shuffle − The Reducer copies the sorted output from each Mapper using HTTP across the network.
Sort − The framework merge-sorts the Reducer inputs by keys (since different Mappers may have output the same key). The shuffle and sort phases occur simultaneously, i.e., while outputs are being fetched, they are merged.
Reduce − In this phase the reduce (Object, Iterable, Context) method is called for each <key, (collection of values)> in the sorted inputs.
reduce is the most prominent method of the Reducer class. The syntax is defined below −
reduce(KEYIN key, Iterable<VALUEIN> values, org.apache.hadoop.mapreduce.Reducer.Context context)
This method is called once for each key on the collection of key-value pairs.