Apache Pig - Group Operator


The GROUP operator is used to group the data in one or more relations. It collects the data having the same key.


Given below is the syntax of the group operator.

grunt> Group_data = GROUP Relation_name BY age;


Assume that we have a file named student_details.txt in the HDFS directory /pig_data/ as shown below.



And we have loaded this file into Apache Pig with the relation name student_details as shown below.

grunt> student_details = LOAD 'hdfs://localhost:9000/pig_data/student_details.txt' USING PigStorage(',')
   as (id:int, firstname:chararray, lastname:chararray, age:int, phone:chararray, city:chararray);

Now, let us group the records/tuples in the relation by age as shown below.

grunt> group_data = GROUP student_details by age;


Verify the relation group_data using the DUMP operator as shown below.

grunt> Dump group_data;


Then you will get output displaying the contents of the relation named group_data as shown below. Here you can observe that the resulting schema has two columns −

  • One is age, by which we have grouped the relation.

  • The other is a bag, which contains the group of tuples, student records with the respective age.

(21,{(4,Preethi,Agarwal,21,9848022330,Pune),(1,Rajiv,Reddy,21,9848022337,Hydera bad)})
(22,{(3,Rajesh,Khanna,22,9848022339,Delhi),(2,siddarth,Battacharya,22,984802233 8,Kolkata)})
(23,{(6,Archana,Mishra,23,9848022335,Chennai),(5,Trupthi,Mohanthy,23,9848022336 ,Bhuwaneshwar)})
(24,{(8,Bharathi,Nambiayar,24,9848022333,Chennai),(7,Komal,Nayak,24,9848022334, trivendram)})

You can see the schema of the table after grouping the data using the describe command as shown below.

grunt> Describe group_data;
group_data: {group: int,student_details: {(id: int,firstname: chararray,
               lastname: chararray,age: int,phone: chararray,city: chararray)}}

In the same way, you can get the sample illustration of the schema using the illustrate command as shown below.

$ Illustrate group_data;

It will produce the following output −

|group_data|  group:int | student_details:bag{:tuple(id:int,firstname:chararray,lastname:chararray,age:int,phone:chararray,city:chararray)}|
|          |     21     | { 4, Preethi, Agarwal, 21, 9848022330, Pune), (1, Rajiv, Reddy, 21, 9848022337, Hyderabad)}| 
|          |     2      | {(2,siddarth,Battacharya,22,9848022338,Kolkata),(003,Rajesh,Khanna,22,9848022339,Delhi)}| 

Grouping by Multiple Columns

Let us group the relation by age and city as shown below.

grunt> group_multiple = GROUP student_details by (age, city);

You can verify the content of the relation named group_multiple using the Dump operator as shown below.

grunt> Dump group_multiple; 

Group All

You can group a relation by all the columns as shown below.

grunt> group_all = GROUP student_details All;

Now, verify the content of the relation group_all as shown below.

grunt> Dump group_all;  
(all,{(8,Bharathi,Nambiayar,24,9848022333,Chennai),(7,Komal,Nayak,24,9848022334 ,trivendram), 
(6,Archana,Mishra,23,9848022335,Chennai),(5,Trupthi,Mohanthy,23,9848022336,Bhuw aneshwar), 
(2,siddarth,Battacharya,22,9848022338,Kolkata),(1,Rajiv,Reddy,21,9848022337,Hyd erabad)})

Useful Video Courses


Apache Spark Online Training

46 Lectures 3.5 hours

Arnab Chakraborty


Apache Spark with Scala - Hands On with Big Data

23 Lectures 1.5 hours

Mukund Kumar Mishra


Learn Apache Cordova using Visual Studio 2015 & Command line

16 Lectures 1 hours

Nilay Mehta


Delta Lake with Apache Spark using Scala

52 Lectures 1.5 hours

Bigdata Engineer


Apache Zeppelin - Big Data Visualization Tool

14 Lectures 1 hours

Bigdata Engineer


Olympic Games Analytics Project in Apache Spark for Beginner

23 Lectures 1 hours

Bigdata Engineer