Protocol Buffers - Quick Guide

---

---

Protocol Buffers - Introduction

Before we jump into Protocol Buffer, let us go over a brief background of Serialization which is what Protocol Buffer does.

What is Serialization and Deserialization?

Serialization is the process of converting an object (of any language) into bytes and storing them in persistent memory system. This memory system could be a file on the disk, messaging queue or a database. The major intention with serialization of object is that we can reuse the data and recreate the object on same or different machine. In deserialization, we convert the stored bytes back to an object.

Why do we need Serialization and Deserialization?

While there are a few other use-cases, the most basic and important one is that it provides a way to transfer object data over a network to a different service/machine etc. and then to recreate object for its further use. Transferring object data via API, database or messaging queue requires the object to be converted into bytes so that it can be sent over a network. And this is where serialization becomes important.

In microservice architecture, the application is broken down into small services and these services communicate with each other via messaging queue and APIs. And all of this communication happens over a network which requires frequent conversion of object to bytes and back to objects. So, serialization and deserialization becomes very critical aspects when it comes to distributed environment.

Why Google Protocol Buffers?

Google Protocol Buffers perform the serialization and deserialization of the objects to bytes which can be transferred over the network. But there are some other libraries and mechanisms to transfer data as well.

So, what makes Google Protocol Buffers special? Here are some of its important features −

• Language independent − Multiple languages have Protocol Buffers library, few famous ones being Java, Python, Go, etc. So, a Java object can be serialized into bytes from a Java program and can be deserialized to a a Python object.

• Efficient Data Compaction − In microservice environment, given that multiple communications take place over a network, it is critical that the data that we are sending is as succinct as possible. We need to avoid any superfluous information to ensure that the data is quickly transferred. Google Protocol Buffers have that as one of the focus areas.

• Efficient serialization and deserialization − In microservice environment, given that multiple communications take place over a network, it is critical how fast can we serialize and deserialize. Google Protocol Buffers ensure that it is as quick as possible in serializing and deserializing the data.

• Simple to use − Protocol Buffers library auto-generate serialization code (as we will see in the upcoming chapters), has a versioning scheme to ensure that the creator of data and the user of data can have separate versions of the serialization definition, etc.

Protocol Buffers vs Others (XML/JSON/Java serialization)

Let's take a look how other ways to transfer data over a network stack up against Protocol Buffers.

Protocol Buffers - Environment Setup

Install Java

We're using Java Based examples to demonstrate Protocol Buffers, this section guides you on how to download and set up Java on your machine. Please follow the following steps to set up the environment.

Java SE is freely available from the link Download Java. So you download a version based on your operating system.

Follow the instructions to download java and run the .exe to install Java on your machine. Once you installed Java on your machine, you would need to set environment variables to point to correct installation directories:

Setting up the path for windows 2000/XP:

Assuming you have installed Java in c:\Program Files\java\jdk directory:

• Right-click on 'My Computer' and select 'Properties'.

• Click on the 'Environment variables' button under the 'Advanced' tab.

• Now, alter the 'Path' variable so that it also contains the path to the Java executable. Example, if the path is currently set to 'C:\WINDOWS\SYSTEM32', then change your path to read 'C:\WINDOWS\SYSTEM32;c:\Program Files\java\jdk\bin'.

Setting up the path for windows 95/98/ME:

Assuming you have installed Java in c:\Program Files\java\jdk directory:

• Edit the 'C:\autoexec.bat' file and add the following line at the end:
  'SET PATH=%PATH%;C:\Program Files\java\jdk\bin'

Setting up the path for Linux, UNIX, Solaris, FreeBSD:

Environment variable PATH should be set to point to where the Java binaries have been installed. Refer to your shell documentation if you have trouble doing this.

Example, if you use bash as your shell, then you would add the following line to the end of your '.bashrc: export PATH=/path/to/java:$PATH'

Popular Java Editors:

To write your Java programs, you will need a text editor. There are even more sophisticated IDEs available in the market. But for now, you can consider one of the following:

• Notepad: On Windows machine you can use any simple text editor like Notepad (Recommended for this tutorial), TextPad.

• Netbeans:is a Java IDE that is open-source and free which can be downloaded from https://www.netbeans.org/index.html.

• Eclipse: is also a Java IDE developed by the eclipse open-source community and can be downloaded from https://www.eclipse.org/.

Install Maven

Maven is project management and build tool. We're using Maven to build our examples. This section guides on how to install and set up maven on your machine. Please follow the following steps to set up the environment.

Maven is freely available from the link Download Maven Installer. So you download a version based on your operating system.

Follow the instructions to download maven zip the extract Maven on your machine. Once you extract Maven files on your machine, you would need to set environment variables to point to correct installation directories. You can follow the instructions at Maven - Environment Setup Chapter.

Install Protocol Buffers Library

Once Java and Maven environment is ready. We can use Protocol Buffers dependency in pom.xml to use it in our java based projects as shown in snippet below −

<dependency>
   <groupId>com.google.protobuf</groupId>
   <artifactId>protobuf-java</artifactId>
   <version>4.31.1</version>
</dependency>

Build Maven Project

Now when you compile the maven project, maven will automatically download the Protocol Buffers library, build the code and run accordingly.

mvn clean install

Install Protocol Buffers Compiler

Let us install the "libprotoc" binary which we will use to autogenerate the code from Protocol Buffers schema. The binaries can be found at "https://github.com/google/protobuf/releases".

Choose the correct binary based on the OS. We will install protocol buffers compiler binary on Windows but the steps are not very different for Linux.

We've downloaded https://github.com/protocolbuffers/protobuf/releases/download/v31.1/protobuf-31.1.zip

Verify Protocol Buffers Compiler Setup

Once installed, ensure that you are able to access it via command line by adding its path in PATH environment variable −

protoc --version

libprotoc 31.1

It confirms that libprotoc is correctly installed.

Protocol Buffers - Basic App

Overview

Let us now use Google Protocol Buffer and see how it works with a simple Greeting app. In this example, we will create a simple application which would do the following −

• Greeting the Writer −

  • Take greeting and username from the user

  • Store the above information in a file in the disk

• Greeting Reader −

  • Reads the same file which we stored in the above file

  • Convert that data into an object and print the data

Protocol Buffer Definition file

The protocol buffer "definition file" contains the schema definition of the data we want to serialize. The data is stored in a human readable file with the extension ".proto".

Let us store the following data in greeting.proto and we will use this in our first application.

syntax = "proto3";
package tutorialspoint;
option java_package = "com.tutorialspoint.greeting";

message Greet {
   string greeting = 1;
   string username = 2;
}

Understanding each construct

Now, let us take a closer look at the data and see what each line of code does in the above code block.

syntax = "proto3";

The syntax here represents what version of Protobuf we are using. So, we are using the latest version 3 and the schema thus can use all the syntax which is valid for version 3.

package tutorialspoint;

The package here is used for conflict resolution if, say, we have multiple classes/members with same name.

option java_package = "com.tutorialspoint.greeting";

This argument is specific to Java, i.e., the package where the code from the .proto file will be auto-generated.

message Greet

Name of the base class for the object which would be created/recreated.

string greeting = 1;
string username = 2;

These are the attributes of the Greet class along with the data type and the position of the tag in the schema. If a new tag is to be added, it should have "3" as the position. Note that this position integer is important to ensure that the actual data is compact and there is scope of schema evolution.

Protocol Buffer Code Generation

Now that we have defined, let us install the "proto" binary which we will use to autogenerate the code for the above Greet class. The binaries can be found at "https://github.com/protocolbuffers/protobuf/releases/".

Choose the correct binary based on the OS. We will install proto binary on Windows but the steps are not very different for Linux.

We've downloaded https://github.com/protocolbuffers/protobuf/releases/download/v31.1/protobuf-31.1.zip

Verify Proto Compiler Setup

Once installed, ensure that you are able to access it via command line −

protoc --version

libprotoc 31.1

It confirms that Protobuf is correctly installed. Now let us move to creating the Greeting app described above for Java.

Project Structure

Here is the overall project structure that we would have −

Greeting App in Java

Now that we have installed protoc, we can auto-generate the code from the proto files using protoc. Let us first create a Java project though.

Following is the Maven configuration that we will use for our Java project. Note that it contains the required library for Protobuf-java as well.

pom.xml

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
   xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
   xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
	
   <modelVersion>4.0.0</modelVersion>
   <groupId>com.tutorials.point</groupId>
   <artifactId>protobuf-tutorial</artifactId>
   <version>1.0</version>
   <packaging>jar</packaging>

   <properties>
      <maven.compiler.source>21</maven.compiler.source>
      <maven.compiler.target>21</maven.compiler.target>
   </properties>

   <dependencies>
      <!-- https://mvnrepository.com/artifact/com.google.protobuf/protobuf-java -->
      <dependency>
         <groupId>com.google.protobuf</groupId>
         <artifactId>protobuf-java</artifactId>
         <version>4.27.3</version>
      </dependency>
   </dependencies>

   <build>
      <plugins>
         <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-shade-plugin</artifactId>
            <version>3.2.4</version>
            <configuration>
               <!--Put your configurations here-->
            </configuration>
            <executions>
               <execution>
                  <phase>package</phase>
                     <goals>
                     <goal>shade</goal>
                  </goals>
               </execution>
            </executions>
         </plugin>
      </plugins>
   </build>
</project>

All of our code would be present under src/main/java.

With the project structure out of the way, let us generate the code for the Greet class −

Generate Java Classes

protoc --java_out=. greeting.proto

Post execution of the command, you will notice a auto-generated class under com > tutorialspoint > greeting folder within current directory.

• Greeting.java

This file contains a class Greeting and an interface GreetOrBuilder which would help us with serialization and deserialization of the Greet object.

Using Generated Java Classes

Now, let us write the writer of the data, which will take the username and the greeting as its inputs −

GreetWriter.java

package com.tutorialspoint.greeting;

import java.io.FileOutputStream;
import java.io.IOException;
import com.tutorialspoint.greeting.Greeting.Greet;

public class GreetWriter{
   public static void main(String[] args) throws IOException {
      Greet greeting = Greet.newBuilder()
         .setGreeting(args[0])
         .setUsername(args[1])
         .build();
		
      String filename = "greeting_protobuf_output";
      System.out.println("Saving greeting to file: " + filename);
		
      try(FileOutputStream output = new FileOutputStream(filename)){
         greeting.writeTo(output);
      }
      System.out.println("Saved greeting with following data to disk: \n" + greeting);
   }
}

The writer simply takes CLI arguments, creates the Greet object, serializes it and then dumps it to a file.

Now let us write a reader which will read the file −

GreetReader.java

package com.tutorialspoint.greeting;

import java.io.FileInputStream;
import java.io.IOException;
import com.tutorialspoint.greeting.Greeting.Greet;

public class GreetReader{
   public static void main(String[] args) throws IOException {
      Greet.Builder greetBuilder = Greet.newBuilder();

      String filename = "greeting_protobuf_output";
      System.out.println("Reading from file " + filename);
	    
      try(FileInputStream input = new FileInputStream(filename)) {
         Greet greet = greetBuilder.mergeFrom(input).build();
         System.out.println("Greeting: " + greet.getGreeting() + "\n" + "Username: " + greet.getUsername());
      }
   }
}

The reader simply reads from the same file, deserializes it, and prints the data about the greeting.

Compile the project

Now that we have set up the reader and the writer, let us compile the project.

mvn clean install

Serialize the Java Object

And now, let us first execute the writer to serialize an object to file system.

java -cp .\target\protobuf-tutorial-1.0.jar com.tutorialspoint.greeting.GreetWriter Hello John

Saving greeting to file: 
greeting_protobuf_output

Saved greeting with following data to disk:
greeting: Hello
username: John

Deserialize the Serialized Object

And then, let us execute the reader to deserialize an object from file system.

java -cp .\target\protobuf-tutorial-1.0.jar com.tutorialspoint.greeting.GreetReader

Reading from file greeting_protobuf_output
Greeting: Hello
Username: John

So, as we see the data that was serialized by the writer and saved to the file, that exact data is correctly deserialized by the reader and printed accordingly.

Greeting App in Python

Let us now write the same example as a Python project −

Install protocol buffers library

We will need to install protobuf pip package before we proceed.

pip install protobuf

All of our code would be present under python directory.

Generate Python classes from proto file

With the project structure out of the way, let us generate the code for Greet class −

protoc  --python_out=. greeting.proto

Post execution of this command, you will notice an auto-generated class greeting_pb2.py in current directory. This class would help us with serialization and deserialization of the Greet object.

Using Generated Python Classes

Now, let us write the writer of the data, which will take the username and the greeting as its input −

greetWriter.py

import greeting_pb2

import sys

greet = greeting_pb2.Greet()
greet.username = sys.argv[1]
greet.greeting = sys.argv[2]

filename = "greeting_protobuf_output";
print("Saving to file: " + filename)

f = open(filename, "wb")
f.write(greet.SerializeToString())
f.close()
print("Saved following greeting to disk: \n" + str(greet))

The writer simply takes CLI arguments, creates the Greet object, serializes it, and then dumps it to a file.

Now let us create a reader which will read the file −

greetReader.py

import greeting_pb2

greet = greeting_pb2.Greet()

filename = "greeting_protobuf_output";
print("Reading from file: " + filename)

f = open(filename, "rb")
greet.ParseFromString(f.read())
f.close()

print("Read greeting from disk: \n" + str(greet))

The reader simply reads from the same file, deserializes it, and prints the data about the greeting.

Serialize the Python Object

Now, let us first execute the writer.

py greetWriter.py Hola Jane

Saving to file: greeting_protobuf_output
Saved following greeting to disk:
greeting: "Hola"
username: "Jane"

Deserialize the Python Object

And then, let us execute the reader.

python greetReader.py

Reading from file: greeting_protobuf_output
Read greeting from disk:
greeting: "Hola"
username: "Jane"

So, as we see, the data that was serialized by the writer and saved to a file. Next, the same data is correctly deserialized by the reader and printed accordingly.

Protocol Buffers - Constructs

Overview

Let us now look at a few basic data structures and data types which Google Protocol Buffers provides. We will look at these data structures using an example of a Movie theater.

Note that for this structure while we will be using Java code, using them in Python code should also be equally simple and possible.

In the next few chapters, we will discuss the following Protocol Buffers data types one by one −

Data Types

• message − "message" is a very basic building block of Protocol Buffers. This translates to a class in the languages that we use, for example, Java, Python, etc

• string − "string" data type translate to a string in the languages that we use, for example, Java, Python, etc

• Numbers − Numbers include Protocol Buffers types like int32, int64, float, double, which are basic building blocks of Protobuf. It translates to int, long float, double, respectively, in the languages that we use, for example, Java, Python, etc.

• bool − "bool" data type is one of the basic building blocks of Protocol Buffers. It translates to Boolean in the languages that we use, for example, Java, Python, etc.

• enum − "enum" is one of the composite datatypes of Protocol Buffers. It translates to an enum in the languages that we use, for example, Java.

• repeated − "repeated" is used to create array or list and is one of the composite datatypes of Protocol Buffers. Protocol Buffers translates this to a java.util.list interface in Java.

• map − "Map" is one of the composite datatypes of Protocol Buffers. Protocol Buffers translates this to a java.util.Map interface in Java.

• Nested Class − We can use an class created using "message" within another "message" and thus creating a nested class. Protocol Buffers translates translates this to a nested Java class.

Protocol Buffers - message

Overview

The very basic building block of Protocol Buffers is the message attribute. This translates to a class in the languages that we use, for example, Java, Python, etc.

Example Code

Following is the syntax that we need to have to instruct Protocol Buffers that we will be creating instances of a given class −

syntax = "proto3";
package theater;
option java_package = "com.tutorialspoint.theater";

message Theater {
}

We will save the above in "theater.proto" and we will use this when we explore other data structures.

Explanation

The "syntax" here represents what version of Protocol Buffers are we using. So, we are using the latest version 3 and the schema thus can use all the syntax which is valid for version 3.

syntax = "proto3";

The package here is used for conflict resolution, if, say, we have multiple class/message with the same name.

package tutorial;

This argument is specific to Java, i.e., the package where the code from the ".proto" file will be auto-generated.

option java_package = "com.tutorialspoint.greeting";

Now that we are done with the prerequisites, the last item here is −

message Theater

This is nothing but the class name of the base class for the object which would be created/recreated. Note that it is useless in its current shape, as it does not have any other attributes. But we will be more adding attributes as we move along.

Using multiple message attributes

A single proto file can also have multiple classes/messages. For example, if we want, we can add a Visitors message/class as well in the same file. Protocol Buffers would ensure to create two separate and independent classes for the same. For example −

syntax = "proto3";

package theater;

option java_package = "com.tutorialspoint.theater";

message Theater {
}
message Visitor {
}

Creating Java Classes from proto file

To use Protocol Buffers, we will now have to use protoc binary to create the required classes from this ".proto" file. Let us see how to do that −

protoc  --java_out=. proto_files\theater.proto

Using Java Classes created from a proto file

Well, that is it! The above command should create the required files in the current directory and now we can use them in our Java code −

Theater theater = Theater.newBuilder().build()
Visitor visitor = Visitor.newBuilder().build()

At this stage, it is not very useful, as we have not added any attributes to the members/classed. Let us do that when we look at strings in Protocol Buffers - string chapter.

Protocol Buffers - string

Overview

Protobuf strings translate to a string in the languages that we use, for example, Java, Python, etc. Continuing on the theater example, following is the syntax that we need to have to instruct Protobuf that we will be creating a string −

theater.proto

syntax = "proto3";
package theater;
option java_package = "com.tutorialspoint.theater";

message Theater {
   string name = 1;
   string address = 2;
}

Now our class/message contains two string attributes. Each of them also has a position which is what Protobuf uses while serialization and deserialization. Each attribute of a member needs to have a unique position attribute.

Creating Java Classes from Proto File

To use Protobuf, we will now have to use protoc binary to create the required classes from this ".proto" file. Let us see how to do that −

protoc  --java_out=. theater.proto

This will create a TheaterOuterClass.java class in com > tutorialspoint > theater folder in current directory. We're using this class in our application similar to as done in Protocol Buffers - Basic App chapter.

Using Java Classes created from Proto File

First let's create a writer to write the theater information −

TheaterWriter.java

package com.tutorialspoint.theater;

import java.io.FileOutputStream;
import java.io.IOException;
import com.tutorialspoint.theater.TheaterOuterClass.Theater;

public class TheaterWriter{
   public static void main(String[] args) throws IOException {
      Theater theater = Theater.newBuilder()
         .setName("Silver Screener")
         .setAddress("212, Maple Street, LA, California")
         .build();
		
      String filename = "theater_protobuf_output";
      System.out.println("Saving theater information to file: " + filename);
		
      try(FileOutputStream output = new FileOutputStream(filename)){
         theater.writeTo(output);
      }
      System.out.println("Saved theater information with following data to disk: \n" + theater);
   }
}	

Next, we will have a reader to read the theater information −

TheaterReader.java

package com.tutorialspoint.theater;

import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.IOException;
import com.tutorialspoint.greeting.Greeting.Greet;
import com.tutorialspoint.theater.TheaterOuterClass.Theater;
import com.tutorialspoint.theater.TheaterOuterClass.Theater.Builder;

public class TheaterReader{
   public static void main(String[] args) throws IOException {
      Builder theaterBuilder = Theater.newBuilder();

      String filename = "theater_protobuf_output";
      System.out.println("Reading from file " + filename);
        
      try(FileInputStream input = new FileInputStream(filename)) {
         Theater theater = theaterBuilder.mergeFrom(input).build();
         System.out.println(theater);
      }
   }
}

Compile the project

Now that we have set up the reader and the writer, let us compile the project.

mvn clean install

Serialize the Java Object

Now, post compilation, let us execute the writer first −

java -cp .\target\protobuf-tutorial-1.0.jar com.tutorialspoint.theater.TheaterWriter

Saving theater information to file: theater_protobuf_output
Saved theater information with following data to disk:
name: "Silver Screener"
address: "212, Maple Street, LA, California"

Deserialize the Serialized Object

Now, let us execute the reader to read from the same file −

java -cp .\target\protobuf-tutorial-1.0.jar com.tutorialspoint.theater.TheaterReader

Reading from file theater_protobuf_output
name: "Silver Screener"
address: "212, Maple Street, LA, California"

So, as we see, we are able to read the serialized strings by deserializing the binary data to the Theater object. Let us now look at numbers in the next chapter Protocol Buffers - Numbers .

Protocol Buffers - Numbers

Overview

Numbers include protobuf types like int32, int64, float, double, which are basic building blocks of Protobuf. It translates to int, long float, double, respectively, in the languages that we use, for example, Java, Python, etc.

Continuing with our theater example from Protocol Buffers - String chapter, following is the syntax that we need to have to instruct Protobuf that we will be creating numbers −

theater.proto

syntax = "proto3";
package theater;
option java_package = "com.tutorialspoint.theater";

message Theater {
   int32 total_capcity = 3;
   int64 mobile = 4;
   float base_ticket_price = 5;
}

Now our class/message contains numerical attributes. Each of them also has a position which is what Protobuf uses while serialization and deserialization. Each attribute of a member needs to have a unique number assigned.

Creating Java Classes from Proto File

To use Protobuf, we will now have to use protoc binary to create the required classes from this ".proto" file. Let us see how to do that −

protoc  --java_out=. theater.proto

This will create a TheaterOuterClass.java class in com > tutorialspoint > theater folder in current directory. We're using this class in our application similar to as done in Protocol Buffers - Basic App chapter.

Using Java Classes created from Proto File

TheaterWriter.java

package com.tutorialspoint.theater;

import java.io.FileOutputStream;
import java.io.IOException;
import com.tutorialspoint.theater.TheaterOuterClass.Theater;

public class TheaterWriter{
   public static void main(String[] args) throws IOException {
      Theater theater = Theater.newBuilder()
         .setTotalCapcity(320)
         .setMobile(98234567189L)
         .setBaseTicketPrice(22.45f)
         .build();
		
      String filename = "theater_protobuf_output";
      System.out.println("Saving theater information to file: " + filename);
		
      try(FileOutputStream output = new FileOutputStream(filename)){
         theater.writeTo(output);
      }
      System.out.println("Saved theater information with following data to disk: \n" + theater);
   }
}

Next, we will have a reader to read the theater information −

TheaterReader.java

package com.tutorialspoint.theater;

import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.IOException;
import com.tutorialspoint.greeting.Greeting.Greet;
import com.tutorialspoint.theater.TheaterOuterClass.Theater;
import com.tutorialspoint.theater.TheaterOuterClass.Theater.Builder;

public class TheaterReader{
   public static void main(String[] args) throws IOException {
      Builder theaterBuilder = Theater.newBuilder();

      String filename = "theater_protobuf_output";
      System.out.println("Reading from file " + filename);
        
      try(FileInputStream input = new FileInputStream(filename)) {
         Theater theater = theaterBuilder.mergeFrom(input).build();
         System.out.println(theater.getBaseTicketPrice());
         System.out.println(theater);
      }
   }
}

Compile the project

Now that we have set up the reader and the writer, let us compile the project.

mvn clean install

Serialize the Java Object

Now, post compilation, let us execute the writer first −

> java -cp .\target\protobuf-tutorial-1.0.jar 
com.tutorialspoint.theater.TheaterWriter

Saving theater information to file: theater_protobuf_output
Saved theater information with following data to disk:
total_capcity: 320
mobile: 98234567189
base_ticket_price: 22.45

Deserialize the Serialized Object

Now, let us execute the reader to read from the same file −

java -cp .\target\protobuf-tutorial-1.0.jar 
com.tutorialspoint.theater.TheaterReader

Reading from file theater_protobuf_output
22.45
total_capcity: 320
mobile: 98234567189
base_ticket_price: 22.45

So, as we see, we are able to read the serialized int, float, and long by deserializing the binary data to Theater object. In the next chapter Protocol Buffers - bool , we will look at the Boolean type.

Protocol Buffers - bool

Overview

The bool data type is one of the basic building blocks of Protobuf. It translates to Boolean in the languages that we use, for example, Java, Python, etc.

Continuing with our theater example from Protocol Buffers - String chapter, following is the syntax that we need to have to instruct Protobuf that we will be creating bool −

theater.proto

syntax = "proto3";
package theater;
option java_package = "com.tutorialspoint.theater";

message Theater {
   bool drive_in = 6;
}

Now our message class contains a bool attribute. It also has a position which is what Protobuf uses while serialization and deserialization. Each attribute of a member needs to have a unique number assigned.

Creating Java Classes from Proto File

To use Protobuf, we will now have to use protoc binary to create the required classes from this ".proto" file. Let us see how to do that −

protoc  --java_out=. theater.proto

This will create a TheaterOuterClass.java class in com > tutorialspoint > theater folder in current directory. We're using this class in our application similar to as done in Protocol Buffers - Basic App chapter.

Using Java Classes created from Proto File

TheaterWriter.java

package com.tutorialspoint.theater;

import java.io.FileOutputStream;
import java.io.IOException;

import com.tutorialspoint.theater.TheaterOuterClass.Theater;

public class TheaterWriter{
   public static void main(String[] args) throws IOException {
      Theater theater = Theater.newBuilder()
         .setDriveIn(true)
         .build();
		
      String filename = "theater_protobuf_output";
      System.out.println("Saving theater information to file: " + filename);
		
      try(FileOutputStream output = new FileOutputStream(filename)){
         theater.writeTo(output);
      }
      System.out.println("Saved theater information with following data to disk: \n" + theater);
   }
}

Next, we will have a reader to read the theater information −

TheaterReader.java

package com.tutorialspoint.theater;

import java.io.FileInputStream;
import java.io.IOException;

import com.tutorialspoint.theater.TheaterOuterClass.Theater;
import com.tutorialspoint.theater.TheaterOuterClass.Theater.Builder;

public class TheaterReader{
   public static void main(String[] args) throws IOException {
      Builder theaterBuilder = Theater.newBuilder();

      String filename = "theater_protobuf_output";
      System.out.println("Reading from file " + filename);
        
      try(FileInputStream input = new FileInputStream(filename)) {
         Theater theater = theaterBuilder.mergeFrom(input).build();
         System.out.println(theater.getDriveIn());
         System.out.println(theater);
      }
   }
}

Compile the project

Now that we have set up the reader and the writer, let us compile the project.

mvn clean install

Serialize the Java Object

Now, post compilation, let us execute the writer first −

> java -cp .\target\protobuf-tutorial-1.0.jar com.tutorialspoint.theater.TheaterWriter

Saving theater information to file: theater_protobuf_output
Saved theater information with following data to disk:
drive_in: true

Deserialize the Serialized Object

Now, let us execute the reader to read from the same file −

java -cp .\target\protobuf-tutorial-1.0.jar com.tutorialspoint.theater.TheaterReader

Reading from file theater_protobuf_output
drive_in: true

So, as we see, we are able to read the serialized bool by deserializing the binary data to Theater object. In the next chapter Protocol Buffers - enum, we will look at the enum, a composite type.

Protocol Buffers - enum

Overview

The enum data type is one of the composite datatypes of Protobuf. It translates to an enum in the languages that we use, for example, Java etc.

Continuing with our theater example from Protocol Buffers - String chapter, following is the syntax that we need to have to instruct Protobuf that we will be creating an enum −

theater.proto

syntax = "proto3";
package theater;
option java_package = "com.tutorialspoint.theater";

message Theater {
   enum PAYMENT_SYSTEM{
      CASH = 0;
      CREDIT_CARD = 1;
      DEBIT_CARD = 2;
      APP = 3;  
   }
   PAYMENT_SYSTEM payment = 7;
}

Now our message class contains an enum attribute. It also has a position which is what Protobuf uses while serialization and deserialization. Each attribute of a member needs to have a unique number assigned.

We define the enum and use it below as the data type along with "payment" attribute. Note that although we have defined enum inside the message class, it can also reside outside of it.

Creating Java Classes from Proto File

To use Protobuf, we will now have to use protoc binary to create the required classes from this ".proto" file. Let us see how to do that −

protoc  --java_out=. theater.proto

This will create a TheaterOuterClass.java class in com > tutorialspoint > theater folder in current directory. We're using this class in our application similar to as done in Protocol Buffers - Basic App chapter.

Using Java Classes created from Proto File

TheaterWriter.java

package com.tutorialspoint.theater;

import java.io.FileOutputStream;
import java.io.IOException;
import com.tutorialspoint.theater.TheaterOuterClass.Theater;
import com.tutorialspoint.theater.TheaterOuterClass.Theater.PAYMENT_SYSTEM;

public class TheaterWriter{
   public static void main(String[] args) throws IOException {
      Theater theater = Theater.newBuilder()
         .setPayment(PAYMENT_SYSTEM.CREDIT_CARD)
         .build();
		
      String filename = "theater_protobuf_output";
      System.out.println("Saving theater information to file: " + filename);
		
      try(FileOutputStream output = new FileOutputStream(filename)){
         theater.writeTo(output);
      }
	    
      System.out.println("Saved theater information with following data to disk: \n" + theater);
   }
}

Next, we will have a reader to read the theater information −

TheaterReader.java

package com.tutorialspoint.theater;

import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.IOException;
import com.tutorialspoint.greeting.Greeting.Greet;
import com.tutorialspoint.theater.TheaterOuterClass.Theater;
import com.tutorialspoint.theater.TheaterOuterClass.Theater.Builder;

public class TheaterReader{
   public static void main(String[] args) throws IOException {
      Builder theaterBuilder = Theater.newBuilder();

      String filename = "theater_protobuf_output";
      System.out.println("Reading from file " + filename);
        
      try(FileInputStream input = new FileInputStream(filename)) {
         Theater theater = theaterBuilder.mergeFrom(input).build();
         System.out.println(theater.getPayment());
         System.out.println(theater);
      }
   }
}

Compile the project

Now that we have set up the reader and the writer, let us compile the project.

mvn clean install

Serialize the Java Object

Now, post compilation, let us execute the writer first −

> java -cp .\target\protobuf-tutorial-1.0.jar com.tutorialspoint.theater.TheaterWriter

Saving theater information to file: theater_protobuf_output
Saved theater information with following data to disk:
payment: CREDIT_CARD

Deserialize the Serialized Object

Now, let us execute the reader to read from the same file −

java -cp .\target\protobuf-tutorial-1.0.jar com.tutorialspoint.theater.TheaterReader

Reading from file theater_protobuf_output
CREDIT_CARD
payment: CREDIT_CARD

So, as we see, we are able to read the serialized enum by deserializing the binary data to Theater object. In the next chapter Protocol Buffers - repeated, we will look at the repeated, a composite type.

Protocol Buffers - repeated

Overview

The repeated data type is one of the composite datatypes of Protobuf. It translates to a java.util.List interface in Java.

Continuing with our theater example from Protocol Buffers - String chapter, following is the syntax that we need to have to instruct Protobuf that we will be creating a repeated −

theater.proto

syntax = "proto3";
package theater;
option java_package = "com.tutorialspoint.theater";

message Theater {
   repeated string snacks = 8;
}

Now our message class contains an repeated attribute. Note that although we have a string repeated, we can as well have number, bool, custom data type list. It also has a position which is what Protobuf uses while serialization and deserialization. Each attribute of a member needs to have a unique number assigned.

Creating Java Classes from Proto File

To use Protobuf, we will now have to use protoc binary to create the required classes from this ".proto" file. Let us see how to do that −

protoc  --java_out=. theater.proto

This will create a TheaterOuterClass.java class in com > tutorialspoint > theater folder in current directory. We're using this class in our application similar to as done in Protocol Buffers - Basic App chapter.

Using Java Classes created from Proto File

TheaterWriter.java

package com.tutorialspoint.theater;

import java.util.List;
import java.io.FileOutputStream;
import java.io.IOException;
import java.util.ArrayList;
import com.tutorialspoint.theater.TheaterOuterClass.Theater;

public class TheaterWriter{
   public static void main(String[] args) throws IOException {
      List<String> snacks = new ArrayList<>();
      snacks.add("Popcorn");
      snacks.add("Coke");
      snacks.add("Chips");
      snacks.add("Soda");
	    
      Theater theater = Theater.newBuilder()
         .addAllSnacks(snacks)
         .build();
		
      String filename = "theater_protobuf_output";
      System.out.println("Saving theater information to file: " + filename);
		
      try(FileOutputStream output = new FileOutputStream(filename)){
         theater.writeTo(output);
      }
      System.out.println("Saved theater information with following data to disk: \n" + theater);
   }
}

Next, we will have a reader to read the theater information −

TheaterReader.java

package com.tutorialspoint.theater;

import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.IOException;
import com.tutorialspoint.greeting.Greeting.Greet;
import com.tutorialspoint.theater.TheaterOuterClass.Theater;
import com.tutorialspoint.theater.TheaterOuterClass.Theater.Builder;

public class TheaterReader{
   public static void main(String[] args) throws IOException {
      Builder theaterBuilder = Theater.newBuilder();

      String filename = "theater_protobuf_output";
      System.out.println("Reading from file " + filename);
        
      try(FileInputStream input = new FileInputStream(filename)) {
         Theater theater = theaterBuilder.mergeFrom(input).build();
         System.out.println(theater);
      }
   }
}

Compile the project

Now that we have set up the reader and the writer, let us compile the project.

mvn clean install

Serialize the Java Object

Now, post compilation, let us execute the writer first −

> java -cp .\target\protobuf-tutorial-1.0.jar com.tutorialspoint.theater.TheaterWriter

Saving theater information to file: theater_protobuf_output
Saved theater information with following data to disk:
snacks: "Popcorn"
snacks: "Coke"
snacks: "Chips"
snacks: "Soda"

Deserialize the Serialized Object

Now, let us execute the reader to read from the same file −

java -cp .\target\protobuf-tutorial-1.0.jar com.tutorialspoint.theater.TheaterReader

Reading from file theater_protobuf_output
snacks: "Popcorn"
snacks: "Coke"
snacks: "Chips"
snacks: "Soda"

So, as we see, we are able to read the serialized repeated by deserializing the binary data to Theater object. In the next chapter Protocol Buffers - map, we will look at the map, a composite type.

Protocol Buffers - map

Overview

The map data type is one of the composite datatypes of Protobuf. It translates to a java.util.Map interface in Java.

Continuing with our theater example from Protocol Buffers - String chapter, following is the syntax that we need to have to instruct Protobuf that we will be creating a repeated −

theater.proto

syntax = "proto3";
package theater;
option java_package = "com.tutorialspoint.theater";

message Theater {
   map<string, int32> movieTicketPrice = 9;
}

Now our message class contains a map of movie and their ticket price. Note that although we have "string -> int" map, we can as well have number, bool, and custom data types. However, note that we cannot have a nested map. It also has a position which is what Protobuf uses while serialization and deserialization. Each attribute of a member needs to have a unique number assigned.

Creating Java Classes from Proto File

To use Protobuf, we will now have to use protoc binary to create the required classes from this ".proto" file. Let us see how to do that −

protoc  --java_out=. theater.proto

This will create a TheaterOuterClass.java class in com > tutorialspoint > theater folder in current directory. We're using this class in our application similar to as done in Protocol Buffers - Basic App chapter.

Using Java Classes created from Proto File

TheaterWriter.java

package com.tutorialspoint.theater;

import java.io.FileOutputStream;
import java.io.IOException;
import java.util.HashMap;
import java.util.Map;

import com.tutorialspoint.theater.TheaterOuterClass.Theater;

public class TheaterWriter{
   public static void main(String[] args) throws IOException {
      Map<String, Integer> ticketPrice = new HashMap<>();
	    
      ticketPrice.put("Avengers Endgame", 700);
      ticketPrice.put("Captain America", 200);
      ticketPrice.put("Wonder Woman 1984", 400);
	    
      Theater theater = Theater.newBuilder()
         .putAllMovieTicketPrice(ticketPrice)
         .build();
		
      String filename = "theater_protobuf_output";
      System.out.println("Saving theater information to file: " + filename);
		
      try(FileOutputStream output = new FileOutputStream(filename)){
         theater.writeTo(output);
      }
      System.out.println("Saved theater information with following data to disk: \n" + theater);
   }
}

Next, we will have a reader to read the theater information −

TheaterReader.java

package com.tutorialspoint.theater;

import java.io.FileInputStream;
import java.io.IOException;

import com.tutorialspoint.theater.TheaterOuterClass.Theater;
import com.tutorialspoint.theater.TheaterOuterClass.Theater.Builder;

public class TheaterReader{
   public static void main(String[] args) throws IOException {
      Builder theaterBuilder = Theater.newBuilder();

      String filename = "theater_protobuf_output";
      System.out.println("Reading from file " + filename);
        
      try(FileInputStream input = new FileInputStream(filename)) {
         Theater theater = theaterBuilder.mergeFrom(input).build();
         System.out.println(theater);
      }
   }
}

Compile the project

Now that we have set up the reader and the writer, let us compile the project.

mvn clean install

Serialize the Java Object

Now, post compilation, let us execute the writer first −

> java -cp .\target\protobuf-tutorial-1.0.jar com.tutorialspoint.theater.TheaterWriter

Saving theater information to file: theater_protobuf_output
Saved theater information with following data to disk:
movieTicketPrice {
   key: "Avengers Endgame"
   value: 700
}
movieTicketPrice {
   key: "Captain America"
   value: 200
}
movieTicketPrice {
   key: "Wonder Woman 1984"
   value: 400
}

Deserialize the Serialized Object

Now, let us execute the reader to read from the same file −

java -cp .\target\protobuf-tutorial-1.0.jar com.tutorialspoint.theater.TheaterReader

Reading from file theater_protobuf_output
movieTicketPrice {
   key: "Avengers Endgame"
   value: 700
}
movieTicketPrice {
   key: "Captain America"
   value: 200
}
movieTicketPrice {
   key: "Wonder Woman 1984"
   value: 400
}

So, as we see, we are able to read the serialized map by deserializing the binary data to Theater object. In the next chapter Protocol Buffers - Nested Class, we will look at the nested classes.

Protocol Buffers - Nested Class

Overview

Here, we will see how to create a nested class. Protobuf translates this to a nested Java class.

Continuing with our theater example from Protocol Buffers - String chapter, following is the syntax that we need to have to instruct Protobuf that we will be creating a repeated −

theater.proto

syntax = "proto3";
package theater;
option java_package = "com.tutorialspoint.theater";

message Theater {
   TheaterOwner owner = 10;
}
message TheaterOwner{
   string name = 1;
   string address = 2;
}

Now our message class contains a nested class, i.e., information about the owner of the theater.

Creating Java Classes from Proto File

To use Protobuf, we will now have to use protoc binary to create the required classes from this ".proto" file. Let us see how to do that −

protoc  --java_out=. theater.proto

This will create a TheaterOuterClass.java class in com > tutorialspoint > theater folder in current directory. We're using this class in our application similar to as done in Protocol Buffers - Basic App chapter.

Using Java Classes created from Proto File

TheaterWriter.java

package com.tutorialspoint.theater;

import java.io.FileOutputStream;
import java.io.IOException;

import com.tutorialspoint.theater.TheaterOuterClass.Theater;
import com.tutorialspoint.theater.TheaterOuterClass.TheaterOwner;

public class TheaterWriter{
   public static void main(String[] args) throws IOException {
      TheaterOwner owner = TheaterOwner.newBuilder()
         .setName("Anthony Gonsalves")
         .setAddress("513, St Paul Street, West Coast, California")
         .build();
	    
      Theater theater = Theater.newBuilder()
         .setOwner(owner)
         .build();
		
      String filename = "theater_protobuf_output";
      System.out.println("Saving theater information to file: " + filename);
		
      try(FileOutputStream output = new FileOutputStream(filename)){
         theater.writeTo(output);
      }
      System.out.println("Saved theater information with following data to disk: \n" + theater);
   }
}

Next, we will have a reader to read the theater information −

TheaterReader.java

package com.tutorialspoint.theater;

import java.io.FileInputStream;
import java.io.IOException;

import com.tutorialspoint.theater.TheaterOuterClass.Theater;
import com.tutorialspoint.theater.TheaterOuterClass.Theater.Builder;

public class TheaterReader{
   public static void main(String[] args) throws IOException {
      Builder theaterBuilder = Theater.newBuilder();

      String filename = "theater_protobuf_output";
      System.out.println("Reading from file " + filename);
        
      try(FileInputStream input = new FileInputStream(filename)) {
         Theater theater = theaterBuilder.mergeFrom(input).build();
         System.out.println(theater);
      }
   }
}

Compile the project

Now that we have set up the reader and the writer, let us compile the project.

mvn clean install

Serialize the Java Object

Now, post compilation, let us execute the writer first −

> java -cp .\target\protobuf-tutorial-1.0.jar com.tutorialspoint.theater.TheaterWriter

Saving theater information to file: theater_protobuf_output
Saved theater information with following data to disk:
owner {
   name: "Anthony Gonsalves"
   address: "513, St Paul Street, West Coast, California"
}

Deserialize the Serialized Object

Now, let us execute the reader to read from the same file −

java -cp .\target\protobuf-tutorial-1.0.jar com.tutorialspoint.theater.TheaterReader

Reading from file theater_protobuf_output
owner {
   name: "Anthony Gonsalves"
   address: "513, St Paul Street, West Coast, California"
}

So, as we see, we are able to read the serialized nested class by deserializing the binary data to Theater object.

Protocol Buffers - Optionality & Defaults

Overview

While we looked at various data types and how to use them. What happens if we do not specify the values while serialization? "proto2" version supported "required" and "optional" tag which helped in figuring out if the serialization/deserialization should fail if the required parsing logic is unavailable. But "required" tag is removed in the "proto3" version. The failing part needs to be handled by respective code. Now every attibute is optional and have default value. Thus use of "optional" is reduntant in "proto3" version onwards.

Protocol Buffers supports default values of its data types as per given table below −

So, if one does not specify the data for these data types, then they would take the above default values. Now, let's continue with our theater example to demonstrate how it works.

In this example, we will let all the fields default. The only field which would be specified would be the name of the theater.

Continuing with our theater example from Protocol Buffers - String chapter, following is the syntax that we need to have to instruct Protobuf that we will be creating different datatypes −

theater.proto

syntax = "proto3";
package theater;
option java_package = "com.tutorialspoint.theater";

message Theater {
   string name = 1;
   string address = 2;
  
   int32 total_capcity = 3;
   int64 mobile = 4;
   float base_ticket_price = 5;
  
   bool drive_in = 6;
  
   enum PAYMENT_SYSTEM {
      CASH = 0;
      CREDIT_CARD = 1;
      DEBIT_CARD = 2;
      APP = 3;
   }
 
   PAYMENT_SYSTEM payment = 7;
   repeated string snacks = 8;
   map<string, int32> movieTicketPrice = 9;
   TheaterOwner owner = 10;
}
message TheaterOwner{
   string name = 1;
   string address = 2;
}

Now our message class contains multiple attributes.

Creating Java Classes from Proto File

To use Protobuf, we will now have to use protoc binary to create the required classes from this ".proto" file. Let us see how to do that −

protoc  --java_out=. theater.proto

This will create a TheaterOuterClass.java class in com > tutorialspoint > theater folder in current directory. We're using this class in our application similar to as done in Protocol Buffers - Basic App chapter.

Using Java Classes created from Proto File

TheaterWriter.java

package com.tutorialspoint.theater;

import java.io.FileOutputStream;
import java.io.IOException;

import com.tutorialspoint.theater.TheaterOuterClass.Theater;

public class TheaterWriter {
   public static void main(String[] args) throws IOException {
      Theater theater = Theater.newBuilder()
         .setName("SilverScreen")
         .build();
		
      String filename = "theater_protobuf_output";
      System.out.println("Saving theater information to file: " + filename);
		
      try(FileOutputStream output = new FileOutputStream(filename)){
         theater.writeTo(output);
      }
      System.out.println("Saved theater information with following data to disk: \n" + theater);
   }
}

Next, we will have a reader to read the theater information −

TheaterReader.java

package com.tutorialspoint.theater;

import java.io.FileInputStream;
import java.io.IOException;

import com.tutorialspoint.theater.TheaterOuterClass.Theater;
import com.tutorialspoint.theater.TheaterOuterClass.Theater.Builder;

public class TheaterReader{
   public static void main(String[] args) throws IOException {
      Builder theaterBuilder = Theater.newBuilder();

      String filename = "theater_protobuf_output";
      System.out.println("Reading from file " + filename);
        
      try(FileInputStream input = new FileInputStream(filename)) {
         Theater theater = theaterBuilder.mergeFrom(input).build();
         System.out.println(
            "Name:" + theater.getName() + "\n" +
            "Address:" + theater.getAddress() + "\n" +
            "Drive_In:" + theater.getDriveIn() + "\n" +
            "Total Capacity:" + theater.getTotalCapcity() + "\n" +
            "Base Ticket Prices: " + theater.getBaseTicketPrice() + "\n" +
            "Owner: " + theater.getOwner() + "\n" +
            "Snacks: " + theater.getSnacksList() + "\n" +
            "Payment: " + theater.getPayment()
         );
            
         //Map<FieldDescriptor, Object> f = theater.getAllFields();
         System.out.println("List of fields explicitly specified: " + theater.getAllFields());
      }
   }
}

Compile the project

Now that we have set up the reader and the writer, let us compile the project.

mvn clean install

Serialize the Java Object

Now, post compilation, let us execute the writer first −

> java -cp .\target\protobuf-tutorial-1.0.jar com.tutorialspoint.theater.TheaterWriter

Saving theater information to file: theater_protobuf_output
Saved theater information with following data to disk:
name: "SilverScreen"

Deserialize the Serialized Object

Now, let us execute the reader to read from the same file −

java -cp .\target\protobuf-tutorial-1.0.jar com.tutorialspoint.theater.TheaterReader

Reading from file theater_protobuf_output
Name:SilverScreen
Address:
Drive_In:false
Total Capacity:0
Base Ticket Prices: 0.0
Owner:
Snacks: []
Payment: CASH
List of fields explicitly specified: {theater.Theater.name=SilverScreen}

So, as we see, all the values defaulted accordingly apart from name which we have explicitly specified as seen in the bottom most line.

Protocol Buffers - Language Independence

Overview

Till now, we have been using Java to serialize and deserialize the Movie Theater data. However, one of the key features that Google Protocol buffers provides is "language independence". In this chapter, we will see how to serialize using Java and deserialize using Python.

Continuing with our theater example from Protocol Buffers - String chapter, following is the syntax that we need to have to instruct Protobuf that we will be creating different datatypes −

theater.proto

syntax = "proto3";
package theater;
option java_package = "com.tutorialspoint.theater";

message Theater {
   string name = 1;
   string address = 2;
  
   int32 total_capcity = 3;
   int64 mobile = 4;
   float base_ticket_price = 5;
  
   bool drive_in = 6;
   
   enum PAYMENT_SYSTEM { 
      CASH = 0;
      CREDIT_CARD = 1;
      DEBIT_CARD = 2;
      APP = 3;
   }
   PAYMENT_SYSTEM payment = 7;
   repeated string snacks = 8;
   map<string, int32> movieTicketPrice = 9;
   TheaterOwner owner = 10;
}
message TheaterOwner{
   string name = 1;
   string address = 2;
}

Serialization using Java

To use Protobuf, we will now have to use protoc binary to create the required classes from this ".proto" file. Let us see how to do that −

protoc  --java_out=. theater.proto

This will create a TheaterOuterClass.java class in com > tutorialspoint > theater folder in current directory. We're using this class in our application similar to as done in Protocol Buffers - Basic App chapter.

Using Java Classes created from Proto File

First, we will create a writer to write the theater information −

TheaterWriter.java

package com.tutorialspoint.theater;

import java.io.FileOutputStream;
import java.io.IOException;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import com.tutorialspoint.theater.TheaterOuterClass.Theater;
import com.tutorialspoint.theater.TheaterOuterClass.TheaterOwner;
import com.tutorialspoint.theater.TheaterOuterClass.Theater.PAYMENT_SYSTEM;

public class TheaterWriter {
   public static void main(String[] args) throws IOException {
      TheaterOwner owner = TheaterOwner.newBuilder()
         .setName("Anthony Gonsalves")
         .setAddress("513, St Paul Street, West Coast, California")
         .build();
	    
      List<String> snacks = new ArrayList<>();
      snacks.add("Popcorn");
      snacks.add("Coke");
      snacks.add("Chips");
      snacks.add("Soda");
	        
      Map<String, Integer> ticketPrice = new HashMap<>();
      ticketPrice.put("Avengers Endgame", 700);
      ticketPrice.put("Captain America", 200);
      ticketPrice.put("Wonder Woman 1984", 400);
	   
      Theater theater = Theater.newBuilder()
         .setName("Silver Screener")
         .setAddress("212, Maple Street, LA, California")
         .setDriveIn(true)
         .setTotalCapcity(320)
         .setMobile(98234567189L)
         .setBaseTicketPrice(22.45f)
         .setPayment(PAYMENT_SYSTEM.CREDIT_CARD)
         .putAllMovieTicketPrice(ticketPrice)
         .addAllSnacks(snacks)
         .setOwner(owner)
         .build();
		
      String filename = "E:/theater_protobuf_output";
      System.out.println("Saving theater information to file: " + filename);
		
      try(FileOutputStream output = new FileOutputStream(filename)){
         theater.writeTo(output);
      }
      System.out.println("Saved theater information with following data to disk: \n" + theater);
   }
}

Let us compile the project.

mvn clean install

Serialize the Java Object

Now, post compilation, let us execute the writer −

> java -cp .\target\protobuf-tutorial-1.0.jar com.tutorialspoint.theater.TheaterWriter

Saving theater information to file: E:/theater_protobuf_output
Saved theater information with following data to disk:
name: "Silver Screener"
address: "212, Maple Street, LA, California"
total_capcity: 320
mobile: 98234567189
base_ticket_price: 22.45
drive_in: true
payment: CREDIT_CARD
snacks: "Popcorn"
snacks: "Coke"
snacks: "Chips"
snacks: "Soda"
movieTicketPrice {
   key: "Avengers Endgame"
   value: 700
}
movieTicketPrice {
   key: "Captain America"
   value: 200
}
movieTicketPrice {
   key: "Wonder Woman 1984"
   value: 400
}
owner {
   name: "Anthony Gonsalves"
   address: "513, St Paul Street, West Coast, California"
}

Deserialize the Serialized Object Using Python

Generate Python classes from proto file

Let us generate the python code for Theater class −

protoc  --python_out=. theater.proto

Post execution of this command, you will notice an auto-generated class theater_pb2.py in current directory. This class would help us with deserialization of the Theater object.

Using Generated Python Classes

Now, let us write the reader of the data, which will read the file containing serialized object using java. −

theaterReader.py

import theater_pb2

filename = "E:/theater_protobuf_output";
print("Reading from file: " + filename)

theater = theater_pb2.Theater()

f = open(filename, "rb")
theater.ParseFromString(f.read())
f.close()

print("Read theater from disk: \n" + str(theater))

And then, let us execute the reader.

python theaterReader.py

Reading from file: E:/greeting_protobuf_output
Read theater from disk:
name: "Silver Screener"
address: "212, Maple Street, LA, California"
total_capcity: 320
mobile: 98234567189
base_ticket_price: 22.45
drive_in: true
payment: CREDIT_CARD
snacks: "Popcorn"
snacks: "Coke"
snacks: "Chips"
snacks: "Soda"
movieTicketPrice {
  key: "Wonder Woman 1984"
  value: 400
}
movieTicketPrice {
  key: "Captain America"
  value: 200
}
movieTicketPrice {
  key: "Avengers Endgame"
  value: 700
}
owner {
  name: "Anthony Gonsalves"
  address: "513, St Paul Street, West Coast, California"
}

So, as we see, all the values which were written by the Java client were correctly deserialized and read by our Python client which effectively means Protobuf is language independent.

Protocol Buffers - Compound Data Types

Overview

There are two more compound data types which may be useful for complicated use cases. They are "OneOf" and "Any". In this chapter, we will see how to use these two data types of Protobuf.

OneOf

We pass a few parameters to this OneOf data type and Protobuf ensures that only one of them is set. If we set one of them and try to set the other one, the first attribute gets reset. Let's us understand this via an example.

Continuing with our theater example from Protocol Buffers - String chapter, say, we have an API which is used to fetch the count of available employees. The value returned from this API is then set to 'count' tag in the following file. But if that API errors out, we can't really 'count', instead we attach the error log.

Ideally, we will always have one of them set, i.e., either the call is successful and we get the count OR the count calculation fails and we get the error message.

Following is the syntax that we need to have to instruct Protobuf that we will be creating an OneOf attribute −

theater.proto

syntax = "proto3";
package theater;
option java_package = "com.tutorialspoint.theater";

message Theater { 
   oneof availableEmployees {
      int32 count = 4;
      string errorLog = 5;
   }
}

Serialization using Java

To use Protobuf, we will now have to use protoc binary to create the required classes from this ".proto" file. Let us see how to do that −

protoc  --java_out=. theater.proto

This will create a TheaterOuterClass.java class in com > tutorialspoint > theater folder in current directory. We're using this class in our application similar to as done in Protocol Buffers - Basic App chapter.

Using Java Classes created from Proto File

First, we will create a writer to write the theater information −

TheaterWriter.java

package com.tutorialspoint.theater;

import java.io.FileOutputStream;
import java.io.IOException;

import com.tutorialspoint.theater.TheaterOuterClass.Theater;

public class TheaterWriter{
   public static void main(String[] args) throws IOException {
    	   
      Theater theater = Theater.newBuilder()
         .setCount(3)
         .setErrorLog("No employee found")
         .build();
		
      String filename = "theater_protobuf_output";
      System.out.println("Saving theater information to file: " + filename);
		
      try(FileOutputStream output = new FileOutputStream(filename)){
         theater.writeTo(output);
      }
      System.out.println("Saved theater information with following data to disk: \n" + theater);
   }
}

Next, we will have a reader to read the theater information −

TheaterReader.java

package com.tutorialspoint.theater;

import java.io.FileInputStream;
import java.io.IOException;

import com.tutorialspoint.theater.TheaterOuterClass.Theater;
import com.tutorialspoint.theater.TheaterOuterClass.Theater.Builder;

public class TheaterReader{
   public static void main(String[] args) throws IOException {
      Builder theaterBuilder = Theater.newBuilder();

      String filename = "theater_protobuf_output";
      System.out.println("Reading from file " + filename);
        
      try(FileInputStream input = new FileInputStream(filename)) {
         Theater theater = theaterBuilder.mergeFrom(input).build();
         System.out.println(theater);
      }
   }
}

Compile the project

Now that we have set up the reader and the writer, let us compile the project.

mvn clean install

Serialize the Java Object

Now, post compilation, let us execute the writer first −

> java -cp .\target\protobuf-tutorial-1.0.jar com.tutorialspoint.theater.TheaterWriter

Saving theater information to file: theater_protobuf_output
Saved theater information with following data to disk:
errorLog: "No employee found"

Deserialize the Serialized Object

Now, let us execute the reader to read from the same file −

java -cp .\target\protobuf-tutorial-1.0.jar com.tutorialspoint.theater.TheaterReader

Reading from file theater_protobuf_output
errorLog: "No employee found"

So, as we see, we are able to see only the error log being set as later.

Any

The next data type that can be of use for complicated uses cases is Any. We can pass any type/message/class to this data type and Protobuf would not complain. Let us understand this via an example.

Continuing with the theater example, say, we want to track people inside the theater. Some of them could be employees and others could be viewers. But ultimately they are people, so we will pass them in a single list which would contain both the types.

Now our class/message contains an Any attribute 'peopleInside' list along with Viewer and Employee class, i.e., information about the people inside theater. Let us see this in action.

Following is the syntax that we need to have to instruct Protobuf that we will be creating an Any attribute −

theater.proto

syntax = "proto3";
package theater;
option java_package = "com.tutorialspoint.theater";

import "google/protobuf/any.proto";

message Theater {
   string name = 1;
   string address = 2;
  
   repeated google.protobuf.Any peopleInside = 3;
}
message Employee{
   string name = 1;
   string address = 2;
}
message Viewer{
   string name = 1;
   int32 age = 2;
   string sex = 3;
}

Serialization using Java

To use Protobuf, we will now have to use protoc binary to create the required classes from this ".proto" file. Let us see how to do that −

protoc  --java_out=. theater.proto

This will create a TheaterOuterClass.java class in com > tutorialspoint > theater folder in current directory. We're using this class in our application similar to as done in Protocol Buffers - Basic App chapter.

Using Java Classes created from Proto File

First, we will create a writer to write the theater information −

TheaterWriter.java

package com.tutorialspoint.theater;

import java.util.List;
import java.io.FileOutputStream;
import java.io.IOException;
import java.util.ArrayList;

import com.google.protobuf.Any;
import com.tutorialspoint.theater.TheaterOuterClass.Employee;
import com.tutorialspoint.theater.TheaterOuterClass.Theater;
import com.tutorialspoint.theater.TheaterOuterClass.Viewer;

public class TheaterWriter{
   public static void main(String[] args) throws IOException {
	    
   List<Any> people = new ArrayList<>();
   people.add(Any.pack(Employee.newBuilder().setName("John").build()));
   people.add(Any.pack(Viewer.newBuilder().setName("Jane").setAge(30).build()));
   people.add(Any.pack(Employee.newBuilder().setName("Simon").build()));
   people.add(Any.pack(Viewer.newBuilder().setName("Janice").setAge(25).build()));
	    
   Theater theater = Theater.newBuilder()
      .setName("SilverScreen")
      .addAllPeopleInside(people)
      .build();
		
      String filename = "theater_protobuf_output_silver";
      System.out.println("Saving theater information to file: " + filename);
		
      try(FileOutputStream output = new FileOutputStream(filename)){
         theater.writeTo(output);
      }
	    
      System.out.println("Saved theater information with following data to disk: \n" + theater);
   }
}

Next, we will have a reader to read the theater information −

TheaterReader.java

package com.tutorialspoint.theater;

import java.io.FileInputStream;
import java.io.IOException;

import com.google.protobuf.Any;
import com.tutorialspoint.theater.TheaterOuterClass.Employee;
import com.tutorialspoint.theater.TheaterOuterClass.Theater;
import com.tutorialspoint.theater.TheaterOuterClass.Theater.Builder;
import com.tutorialspoint.theater.TheaterOuterClass.Viewer;

public class TheaterReader{
   public static void main(String[] args) throws IOException {
	   Builder theaterBuilder = Theater.newBuilder();

      String filename = "theater_protobuf_output_silver";
      System.out.println("Reading from file " + filename);
        
      try(FileInputStream input = new FileInputStream(filename)) {
         Theater theater = theaterBuilder.mergeFrom(input).build();
            
         System.out.println("Name:" + theater.getName() + "\n");
            
         for (Any anyPeople : theater.getPeopleInsideList()) {
            if(anyPeople.is(Employee.class)) {
               Employee employee = anyPeople.unpack(Employee.class);
               System.out.println("Employee:" + employee + "\n");
            }
                
            if(anyPeople.is(Viewer.class)) {
               Viewer viewer = anyPeople.unpack(Viewer.class);
               System.out.println("Viewer:" + viewer + "\n");
            }
         }
      }
   }
}	

Compile the project

Now that we have set up the reader and the writer, let us compile the project.

mvn clean install

Serialize the Java Object

Now, post compilation, let us execute the writer first −

> java -cp .\target\protobuf-tutorial-1.0.jar com.tutorialspoint.theater.TheaterWriter

Saving theater information to file: theater_protobuf_output
Saved theater information with following data to disk:
name: "SilverScreen"
peopleInside {
   type_url: "type.googleapis.com/theater.Employee"
   value: "\n\004John"
}
peopleInside {
   type_url: "type.googleapis.com/theater.Viewer"
   value: "\n\004Jane\020\036"
}
peopleInside {
   type_url: "type.googleapis.com/theater.Employee"
   value: "\n\005Simon"
}
peopleInside {
   type_url: "type.googleapis.com/theater.Viewer"
   value: "\n\006Janice\020\031"
}

Note − There are two points to note −

• In case of Any, Protobuf packs/serializes the contents inside any tag to bytes and then stores it as 'value'. Basically, that allows us to send any message type with this 'Any' tag.

• We also see "type.googleapis.com/theater.Viewer" and "type.googleapis.com/theater.Employee". This is used by Protobuf to save the type of object along with the data as the type of data in the Any data type can vary.

Deserialize the Serialized Object

Now let us execute the reader to read from the same file −

java -cp .\target\protobuf-tutorial-1.0.jar com.tutorialspoint.theater.TheaterReader

Reading from file theater_protobuf_output_silver
Name:SilverScreen

Employee:name: "John"

Viewer:name: "Jane"
age: 30

Employee:name: "Simon"

Viewer:name: "Janice"
age: 25

So, as we see, our reader code is successfully able to differentiate between Employee and the Viewer, even though they come in the same array.

Protocol Buffers - Command Line Usage

Overview

Protobuf serializes the data and stores it in a binary format. While this may not be a problem if we are dealing simply with strings, because ultimately Protobuf uses UTF-8. So, any text that it stores would be human readable if you are using a UTF8 enabled reader. However, things like int32, Boolean, list, maps are encoded using specific techniques to reduce space consumption.

That is why, at times, encoding/decoding a message via simple command line utility is useful for testing purposes. Let us see this in action −

Suppose we use the following simple "greeting_cli.proto" −

syntax = "proto3";
package tutorial;
option java_package = "com.tutorialspoint.greeting";

message Greet {
   string greeting = 1;
   string username = 2;
   int32 age = 3;
}  

And we create a message in cli_greeting_message −

greeting: "Yo"
username : "John"
age : 50

Now, let us encode this message using Protobuf CLI tool −

cat  .\cli_greeting_msg.proto |  protoc  --encode=tutorial.Greet .\greeting_cli.proto > encoded_greeting

If we look at what is inside this file or cat this file −

cat .\encoded_greeting

☻Yo↕John↑2

You will notice some weird characters apart from "Yo" and "John". That is because these encoding may not be a valid unicode/UTF-8 encoding. UTF-8 is what is used, generally speaking, at most of the places. And this is used for string in case of Protobuf, but ints, maps, Boolean, list have separate formats. Plus, this file also contains a metadata of the data.

That is why, we need a decoder/deserializer to read this data. Let us use that.

cat .\encoded_greeting | protoc --decode=tutorial.Greet .\greeting_cli.proto

greeting: "Yo"
username : "John"
age : 50

So, as we see, we are able to get the data back which was serialized and looked weird in the file.

Protocol Buffers - Rules to Update Definition

Overview

Assume you came out with the definition of the proto file that you will use in the production environment. There will obviously be times in future when this definition would have to change. In that case, it is essential that the changes we make adhere to certain rules so that the changes are backwards compatible. Let us see this in action with a few do's and dont's.

Add a new field in the writer, while the reader retains the older version of code.

Suppose, you decide to add a new field. Ideally, to have the new field to be added, we will have to update the writer and the reader simultaneously. However, in a large-scale deployment, this is not possible. There will be cases where the writer has been updated, but the reader is yet to be updated with the new field. This is where the above situation occurs. Let us see that in action.

Continuing with our theater example, say, we just have a single tag which is 'name' in our proto file. Following is the syntax that we need to have to instruct Protobuf −

theater.proto

syntax = "proto3";
package theater;
option java_package = "com.tutorialspoint.theater";

message Theater {
   string name = 1;
}

To use Protobuf, we will now have to use the protoc binary to create the required classes from this ".proto" file. Let us see how to do that −

protoc  --java_out=. theater.proto

The above command should create the required files and now we can use it in our Java code. First, we will create a writer to write the theater information −

TheaterWriter.java

package com.tutorialspoint.theater;
package com.tutorialspoint.theater;

import java.io.FileOutputStream;
import java.io.IOException;
import com.tutorialspoint.theater.TheaterOuterClass.Theater;

public class TheaterWriter{
   public static void main(String[] args) throws IOException {
      Theater theater = Theater.newBuilder()
         .setName("Silver Screener")
         .build();
		
      String filename = "theater_protobuf_output";
      System.out.println("Saving theater information to file: " + filename);
		
      try(FileOutputStream output = new FileOutputStream(filename)){
         theater.writeTo(output);
      }
	    
      System.out.println("Saved theater information with following data to disk: \n" + theater);
   }
}

Next, we will have a reader to read the theater information −

TheaterReader.java

package com.tutorialspoint.theater;

import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.IOException;
import com.google.protobuf.ProtocolStringList;
import com.tutorialspoint.greeting.Greeting.Greet;
import com.tutorialspoint.theater.TheaterOuterClass.Theater;
import com.tutorialspoint.theater.TheaterOuterClass.Theater.Builder;

public class TheaterReader{
   public static void main(String[] args) throws IOException {
	    
      Builder theaterBuilder = Theater.newBuilder();

      String filename = "theater_protobuf_output";
      System.out.println("Reading from file " + filename);
        
      try(FileInputStream input = new FileInputStream(filename)) {
         Theater theater = theaterBuilder.mergeFrom(input).build();
         System.out.println(theater);
         System.out.println("Unknwon fields: " + theater.getUnknownFields());
      }
   }
}		

Now, post compilation, let us execute the writer first −

> java -cp .\target\protobuf-tutorial-1.0.jar com.tutorialspoint.theater.TheaterWriter

Saving theater information to file: theater_protobuf_output
Saved theater information with following data to disk:
name: "Silver Screener"

Now let us execute the reader to read from the same file −

java -cp .\target\protobuf-tutorial-1.0.jar com.tutorialspoint.theater.TheaterReader

Reading from file theater_protobuf_output
name: "Silver Screener"

Unknown Fields

We just wrote a simple string as per our Protobuf definition and the reader was able to read the string. And we also saw that there were no unknown fields that the reader was not aware of.

But now, let us suppose we want add a new string 'address' to our Protobuf definition. Now, it will look like this −

theater.proto

syntax = "proto3";
package theater;
option java_package = "com.tutorialspoint.theater";

message Theater {
   string name = 1;
   string address = 2;
}

We will also update our writer and add an address field −

Theater theater = Theater.newBuilder()
   .setName("Silver Screener")
   .setAddress("212, Maple Street, LA, California")
   .build();

Before compiling, rename the JAR from the previous compilation to protobuf-tutorial-old-1.0.jar. And then compile.

Now, post compilation, let us execute the writer first −

> java -cp .\target\protobuf-tutorial-1.0.jar com.tutorialspoint.theater.TheaterWriter

Saving theater information to file: theater_protobuf_output
Saved theater information with following data to disk:
name: "Silver Screener"
address: "212, Maple Street, LA, California"

Now let us execute the reader to read from the same file but from the older JAR −

java -cp .\target\protobuf-tutorial-old-1.0.jar com.tutorialspoint.theater.TheaterReader

Reading from file theater_protobuf_output
Reading from file theater_protobuf_output
name: "Silver Screener"
2: "212, Maple Street, LA, California"

Unknown fields: 2: "212, Maple Street, LA, California"

As you can see from the last line of the output, the old reader is unaware of the address field which was added by the new writer. It just shows how a combination of "new writer - old reader" functions.

Deleting a Field

Suppose, you decide to delete an existing field. Ideally, for the deleted field to have an effect immediately, we will have to update the writer and the reader simultaneously. However, in a large-scale deployment, this is not possible. There will be cases where the writer has been updated, but the reader is yet to be updated. In such a case, the reader will still attempt to read the deleted field. Let us see that in action.

Continuing with the theater example, say, we just have two tags in our proto file. Following is the syntax that we need to have to instruct Protobuf −

theater.proto

syntax = "proto3";
package theater;
option java_package = "com.tutorialspoint.theater";

message Theater {
   string name = 1;
   string address = 2;
}

To use Protobuf we will now have to use the protoc binary to create the required classes from this ".proto" file. Let us see how to do that −

protoc  --java_out=java/src/main/java proto_files\theater.proto

The above command should create the required files and now we can use it in our Java code. First, we will create a writer to write the theater information −

TheaterWriter

package com.tutorialspoint.theater;

import java.io.FileOutputStream;
import java.io.IOException;
import com.tutorialspoint.theater.TheaterOuterClass.Theater;

public class TheaterWriter{
   public static void main(String[] args) throws IOException {
      Theater theater = Theater.newBuilder()
         .setName("Silver Screener")
         .setAddress("212, Maple Street, LA, California")
         .build();
		
      String filename = "theater_protobuf_output";
      System.out.println("Saving theater information to file: " + filename);
		
      try(FileOutputStream output = new FileOutputStream(filename)){
         theater.writeTo(output);
      }
	    
      System.out.println("Saved theater information with following data to disk: \n" + theater);
   }
}

Next, we will have a reader to read the theater information −

TheaterReader.java

package com.tutorialspoint.theater;

import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.IOException;
import com.google.protobuf.ProtocolStringList;
import com.tutorialspoint.greeting.Greeting.Greet;
import com.tutorialspoint.theater.TheaterOuterClass.Theater;
import com.tutorialspoint.theater.TheaterOuterClass.Theater.Builder;

public class TheaterReader{
   public static void main(String[] args) throws IOException {
      Builder theaterBuilder = Theater.newBuilder();

      String filename = "theater_protobuf_output";
      System.out.println("Reading from file " + filename);
        
      try(FileInputStream input = new FileInputStream(filename)) {
         Theater theater = theaterBuilder.mergeFrom(input).build();
         System.out.println(theater);
         System.out.println("Unknwon fields: " + theater.getUnknownFields());
      }
   }
}		

Now, post compilation, let us execute the writer first −

> java -cp .\target\protobuf-tutorial-1.0.jar com.tutorialspoint.theater.TheaterWriter

Saving theater information to file: theater_protobuf_output
Saved theater information with following data to disk:
name: "Silver Screener"
address: "212, Maple Street, LA, California"

Now let us execute the reader to read from the same file −

java -cp .\target\protobuf-tutorial-1.0.jar com.tutorialspoint.theater.TheaterReader

Reading from file theater_protobuf_output
name: "Silver Screener"
address: "212, Maple Street, LA, California"

So, nothing new here, we just wrote a simple string as per our Protobuf definition and the reader was able to read the string.

But now, let us suppose we want to delete the string 'address' from our Protobuf definition. So, the definition would look like this −

theater.proto

syntax = "proto3";
package theater;
option java_package = "com.tutorialspoint.theater";

message Theater {
   string name = 1;
}

We will also update our writer as follows −

Theater theater = Theater.newBuilder()
   .setName("Silver Screener")
   .build();

Before compiling, rename the JAR from the previous compilation to protobuf-tutorial-old-1.0.jar. And then compile.

Now, post compilation, let us execute the writer first −

> java -cp .\target\protobuf-tutorial-1.0.jar com.tutorialspoint.theater.TheaterWriter

Saving theater information to file: theater_protobuf_output
Saved theater information with following data to disk:
name: "Silver Screener"

Now let us execute the reader to read from the same file but from the older JAR −

java -cp .\target\protobuf-tutorial-old-1.0.jar com.tutorialspoint.theater.TheaterReader

Reading from file theater_protobuf_output
Reading from file theater_protobuf_output
name: "Silver Screener"
address:

As you can see from the last line of the output, the old reader defaults to the value of "address". It shows how a combination of "new writer - old reader" functions.

Avoid Reusing Serial Number of the Field

There may be cases where, by mistake, we update the "serial number" of a field. This can be problematic, as the serial number is very critical for Protobuf to understand and deserialize the data. And some old reader may be relying on this serial number to deserialize the data. So, it is recommended that you −

• Do not change serial number of field

• Do not reuse serial number of deleted field.

Let us see that in action by interchanging the field tags.

Continuing with the theater example, let's assume we just have two tags in our proto file. Following is the syntax that we need to have to instruct Protobuf −

theater.proto

syntax = "proto3";
package theater;
option java_package = "com.tutorialspoint.theater";

message Theater {
   string name = 1;
   string address = 2;
}

To use Protobuf, we will now have to use the protoc binary to create the required classes from this ".proto" file. Let us see how to do that −

protoc  --java_out=java/src/main/java proto_files\theater.proto

The above command should create the required files and now we can use it in our Java code. First, we will create a writer to write the theater information −

TheaterWriter.java

package com.tutorialspoint.theater;

import java.io.FileOutputStream;
import java.io.IOException;
import com.tutorialspoint.theater.TheaterOuterClass.Theater;

public class TheaterWriter{
   public static void main(String[] args) throws IOException {
      Theater theater = Theater.newBuilder()
         .setName("Silver Screener")
         .setAddress("212, Maple Street, LA, California")
         .build();
		
      String filename = "theater_protobuf_output";
      System.out.println("Saving theater information to file: " + filename);
		
      try(FileOutputStream output = new FileOutputStream(filename)){
         theater.writeTo(output);
      }
      System.out.println("Saved theater information with following data to disk: \n" + theater);
   }
}

Next, we will have a reader to read the theater information −

TheaterReader.java

package com.tutorialspoint.theater;

import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.IOException;
import com.google.protobuf.ProtocolStringList;
import com.tutorialspoint.greeting.Greeting.Greet;
import com.tutorialspoint.theater.TheaterOuterClass.Theater;
import com.tutorialspoint.theater.TheaterOuterClass.Theater.Builder;

public class TheaterReader{
   public static void main(String[] args) throws IOException {
      Builder theaterBuilder = Theater.newBuilder();
      String filename = "theater_protobuf_output";
      System.out.println("Reading from file " + filename);
        
      try(FileInputStream input = new FileInputStream(filename)) {
         Theater theater = theaterBuilder.mergeFrom(input).build();
         System.out.println(theater);
         System.out.println("Unknwon fields: " + theater.getUnknownFields());
      }
   }
}

Now, post compilation, let us execute the writer first −

> java -cp .\target\protobuf-tutorial-1.0.jar com.tutorialspoint.theater.TheaterWriter

Saving theater information to file: theater_protobuf_output
Saved theater information with following data to disk:
name: "Silver Screener"
address: "212, Maple Street, LA, California"

Next, let us execute the reader to read from the same file −

java -cp .\target\protobuf-tutorial-1.0.jar com.tutorialspoint.theater.TheaterReader

Reading from file theater_protobuf_output
name: "Silver Screener"
address: "212, Maple Street, LA, California"

Here, we just wrote simple strings as per our Protobuf definition and the reader was able to read the string. But now, let us interchange the serial number in our Protobuf definition and to make it like this −

theater.proto

syntax = "proto3";
package theater;
option java_package = "com.tutorialspoint.theater";

message Theater {
   string name = 2;
   string address = 1;
}		

Before compiling, rename the JAR from previous compilation to protobuf-tutorial-old-1.0.jar. And then compile.

Now, post compilation, let us execute the writer first −

> java -cp .\target\protobuf-tutorial-1.0.jar com.tutorialspoint.theater.TheaterWriter

Saving theater information to file: theater_protobuf_output
Saved theater information with following data to disk:
address: "212, Maple Street, LA, California"
name: "Silver Screener"

Now let us execute the reader to read from the same file but from the older JAR −

java -cp .\target\protobuf-tutorial-old-1.0.jar com.tutorialspoint.theater.TheaterReader

Reading from file theater_protobuf_output
name: "212, Maple Street, LA, California"
address: "Silver Screener"

As you can see from the output, the old reader interchanged the address and the name. It shows that updating the serial number along with a combination of "new writer-old reader" does not function as expected.

More importantly, here we had two strings, which is why we get to see the data. If we had used different data types, for example, int32, Boolean, map, etc., Protobuf would have given up and treated that as an unknown field.

So, it is imperative to not change the serial number of a field or reuse the serial number of a deleted field.

Changing the Field Type

There may be cases where we need to update the type of an attribute/field. Protobuf has certain compatibility rules for this. Not all the types can be converted to other types. Few basic ones to be aware of −

• string and bytes are compatible if the bytes are UTF-8. This is because, strings are anyways encoded/decoded as UTF-8 by Protobuf.

• enum is compatible with int32 and int64 in terms of the value, however, the client may not deserialize this as expected.

• int32, int64 (unsigned also) along with bool are compatible and thus can be interchanged. Excessive characters may get truncated similar to how casting works in languages.

But we need to be very careful when changing types. Let us see that in action with an incorrect example of converting int64 to int32.

Continuing with the theater example, suppose we just have two tags in our proto file. Following is the syntax that we need to have to instruct Protobuf −

theater.proto

syntax = "proto3";
package theater;
option java_package = "com.tutorialspoint.theater";
message Theater {
   string name = 1;
   int64 total_capacity = 2;
}

To use Protobuf, we will now have to use the protoc binary to create the required classes from this ".proto" file. Let us see how to do that −

protoc  --java_out=java/src/main/java proto_files\theater.proto

The above command should create the required files and now we can use it in our Java code. First, we will create a writer to write the theater information −

TheaterWriter.java

package com.tutorialspoint.theater;

import java.io.FileOutputStream;
import java.io.IOException;
import com.tutorialspoint.theater.TheaterOuterClass.Theater;

public class TheaterWriter{
   public static void main(String[] args) throws IOException {
      Theater theater = Theater.newBuilder()
         .setName("Silver Screener")
         .setTotalCapacity(2300000000L)
         .build();
		
      String filename = "theater_protobuf_output";
      System.out.println("Saving theater information to file: " + filename);
		
      try(FileOutputStream output = new FileOutputStream(filename)){
         theater.writeTo(output);
      }
	    
      System.out.println("Saved theater information with following data to disk: \n" + theater);
   }
}

Now, post compilation, let us execute the writer first −

> java -cp .\target\protobuf-tutorial-1.0.jar com.tutorialspoint.theater.TheaterWriter

Saving theater information to file: theater_protobuf_output
Saved theater information with following data to disk:
name: "Silver Screener"
total_capacity: 2300000000

Let us suppose, we use a different version of proto file for the reader −

theater.proto

syntax = "proto3";
package theater;
option java_package = "com.tutorialspoint.theater";

message Theater {
   string name = 1;
   int64 total_capacity = 2;
}

Next, we will have a reader to read the theater information −

TheaterReader.java

package com.tutorialspoint.theater;

import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.IOException;
import com.google.protobuf.ProtocolStringList;
import com.tutorialspoint.greeting.Greeting.Greet;
import com.tutorialspoint.theater.TheaterOuterClass.Theater;
import com.tutorialspoint.theater.TheaterOuterClass.Theater.Builder;

public class TheaterReader{
   public static void main(String[] args) throws IOException {
      Builder theaterBuilder = Theater.newBuilder();

      String filename = "theater_protobuf_output";
      System.out.println("Reading from file " + filename);
        
      try(FileInputStream input = new FileInputStream(filename)) {
         Theater theater = theaterBuilder.mergeFrom(input).build();
         System.out.println(theater);
         System.out.println("Unknwon fields: " + theater.getUnknownFields());
      }
   }
}		

Now let us execute the reader to read from the same file −

java -cp .\target\protobuf-tutorial-old-1.0.jar com.tutorialspoint.theater.TheaterReader

Reading from file theater_protobuf_output
name: "Silver Screener"
address: "212, Maple Street, LA, California"

So, nothing new here, we just wrote simple strings as per our Protobuf definition and the reader was able to read the string. But now, let us interchange the serial number in our Protobuf definition and make it like this −

theater.proto

syntax = "proto3";
package theater;
option java_package = "com.tutorialspoint.theater";

message Theater {
   string name = 2;
   int32 total_capacity = 2;
}		

Before compiling, rename the JAR from previous compilation to protobuf-tutorial-old-1.0.jar. And then compile.

Now, post compilation, let us execute the writer first −

> java -cp .\target\protobuf-tutorial-1.0.jar com.tutorialspoint.theater.TheaterWriter

Reading from file theater_protobuf_output
address: "Silver Screener"
total_capcity: -1994967296

As you can see from the output, the old reader converted the number from int64, however, the given int32 does not have enough space to contain the data, it wrapped around to negative number. This wrapping is Java specific and is not related to Protobuf.

So, we need to upgrade to int64 from int32 instead of other way around. If we still want to convert from int64 to int32, we need to ensure that the values can be actually held in 31 bits (1 bit for sign bit).

Protocol Buffers - Kafka Integration

We have covered quite a lot of examples of Protocol Buffers and its data types. In this chapter, let us take another example and see how Protocol Buffers integrates with a Schema Registry used by Kafka. Let us first understand what a "schema registry" is.

Schema Registry

Kafka is one of the widely used messaging queues. It is used to apply the publisher-subscriber model at scale. More information about Kafka can be found here − https://www.tutorialspoint.com/apache_kafka/index.htm

However, at the basic level, a Kafka producer is supposed to send a message, i.e., a piece of information which the Kafka consumer can read. And this sending and consuming of message is where we need a schema. It is especially required in large-scale organization where there are multiple teams reading/writing to Kafka topic. Kafka provides a way to store this schema in a schema registry which are then created/consumed when the producer/consumer creates/consumes the message.

There are two major benefits of maintaining a schema −

• Compatibility − In larger organizations, it is necessary that the team producing the message does not break the downstream tools which consume these messages. Schema registry ensures that changes are backwards compatible.

• Efficient encoding − Sending in a field name, its type with every message is space and compute inefficient. With schemas in place, we do not need to send this information with each message.

The schema registry supports Avro, Google Protocol Buffers and JSON Schema as the schema language. The schema in these languages can be stored in the schema registry. For this tutorial, we would require Kafka setup and Schema registry setup.

For installation of Kafka, you can check the following links −

• https://www.tutorialspoint.com/apache_kafka/apache_kafka_installation_steps.htm

• https://docs.confluent.io/platform/current/installation/installing_cp/deb-ubuntu.html#systemd-ubuntu-debian-install

Once you have Kafka installed, you can then setup the Schema Registry by updating the /etc/schema-registry/schema-registry.properties file.

# where should schema registry listen on
listeners=http://0.0.0.0:8081

# Schema registry uses Kafka beneath it, so we need to tell where are the Kafka brokers available
kafkastore.bootstrap.servers=PLAINTEXT://hostname:9092,SSL://hostname2:9092
Once done, you can then run:
sudo systemctl start confluent-schema-registry

With the setup out of the way, let us start using Google Protocol Buffers along with the Schema Registry.

Kafka Producer with Protocol Buffers Schema

Let us continue with our theater example. We will use the following Protocol Buffers schema −

theater.proto

syntax = "proto3";
package theater;
option java_package = "com.tutorialspoint.theater";

message Theater {
   string name = 1;
   string address = 2;
  
   int32 total_capcity = 3;
   int64 mobile = 4;
   float base_ticket_price = 5;
  
   bool drive_in = 6;
  
   enum PAYMENT_SYSTEM{
      CASH = 0;
      CREDIT_CARD = 1;
      DEBIT_CARD = 2;
      APP = 3;
   }
   PAYMENT_SYSTEM payment = 7;
   repeated string snacks = 8;
   
   map<string, int32> movieTicketPrice = 9;
}

Now, let us create a simple Kafka writer which would write the message encoded in this format to the Kafka topic. But for doing that, first, we need to add a few dependencies to our Maven POM −

• Kafka Client to use Kafka producer and consumer

• Kafka Protocol Buffers serializer to serialize and deserialize the message

• Slf4j simple to ensure we get logs from Kafka

<dependency>
   <groupId>org.apache.kafka</groupId>
   <artifactId>kafka-clients</artifactId>
   <version>2.5.0</version>
</dependency>

<!-- https://mvnrepository.com/artifact/io.confluent/kafka-protobuf-serializer -->
<dependency>
   <groupId>io.confluent</groupId>
   <artifactId>kafka-protobuf-serializer</artifactId>
   <version>5.5.1</version>
</dependency>

<!-- https://mvnrepository.com/artifact/org.slf4j/slf4j-simple -->
<dependency>
   <groupId>org.slf4j</groupId>
   <artifactId>slf4j-simple</artifactId>
   <version>1.7.30</version>
</dependency>

Once this is done, let us now create a Kafka producer. This producer will create and send a message which will contain the theater object.

KafkaProtbufProducer.java

package com.tutorialspoint.kafka;

import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.Properties;
import org.apache.kafka.clients.producer.Producer;
import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerRecord;
import com.tutorialspoint.theater.TheaterOuterClass.Theater;
import com.tutorialspoint.theater.TheaterOuterClass.Theater.PAYMENT_SYSTEM;

public class KafkaProtbufProducer {
   public static void main(String[] args) throws Exception{
      String topicName = "testy1";
      Properties props = new Properties();
      props.put("bootstrap.servers", "localhost:9092");
      props.put("clientid", "foo");
      props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
      props.put("value.serializer", "io.confluent.kafka.serializers.protobuf.KafkaProtocol BuffersSerializer");
      props.put("schema.registry.url", "http://localhost:8081");
      props.put("auto.register.schemas", "true");
      
      Producer<String, Theater> producer = new KafkaProducer<>(props);
      producer.send(new ProducerRecord<String, Theater>(topicName, "SilverScreen", getTheater())).get();
      System.out.println("Sent to Kafka: \n" + getTheater());
      producer.flush();
      producer.close();
   }
   public static Theater getTheater() {
      List<String> snacks = new ArrayList<>();
      snacks.add("Popcorn");
      snacks.add("Coke");
      snacks.add("Chips");
      snacks.add("Soda");
           
      Map<String, Integer> ticketPrice = new HashMap<>();
      ticketPrice.put("Avengers Endgame", 700);
      ticketPrice.put("Captain America", 200);
      ticketPrice.put("Wonder Woman 1984", 400);
                  
      Theater theater = Theater.newBuilder()
         .setName("Silver Screener")
         .setAddress("212, Maple Street, LA, California")
         .setDriveIn(true)
         .setTotalCapacity(320)
         .setMobile(98234567189L)
         .setBaseTicketPrice(22.45f)
         .setPayment(PAYMENT_SYSTEM.CREDIT_CARD)
         .putAllMovieTicketPrice(ticketPrice)
         .addAllSnacks(snacks)
         .build();
      return theater;
   }
}

Here is a list of a few points that we need to be aware of −

• We need to pass the Schema Registry URL to the Producer.

• We also need to pass the correct Protocol Buffers Serializer which is specific to the Schema Registry.

• Schema registry would automatically store the schema of the theater object when we are done sending.

• Lastly, we created a theater object from our auto-generated Java code and that is what we will be sending.

Output

Let us now compile and execute the code −

mvn clean install ; java -cp .\target\protobuf-tutorial-1.0.jar com.tutorialspoint.kafka.KafkaProtbufProducer

We will get to see the following output −

[main] INFO org.apache.kafka.common.utils.AppInfoParser - Kafka version: 2.5.0
[main] INFO org.apache.kafka.common.utils.AppInfoParser - Kafka commitId: 66563e712b0b9f84
[main] INFO org.apache.kafka.common.utils.AppInfoParser - Kafka startTimeMs: 1621692205607
[kafka-producer-network-thread | producer-1] INFO org.apache.kafka.clients.Metadata - 
[Producer clientId=producer-1] Cluster ID: 7kwQVXjYSz--bE47MiXmjw

Sent to Kafka

name: "Silver Screener"
address: "212, Maple Street, LA, California"
total_capacity: 320
mobile: 98234567189
base_ticket_price: 22.45
drive_in: true
payment: CREDIT_CARD
snacks: "Popcorn"
snacks: "Coke"
snacks: "Chips"
snacks: "Soda"
movieTicketPrice {
   key: "Avengers Endgame"
   value: 700
}
movieTicketPrice {
   key: "Captain America"
   value: 200
}
movieTicketPrice {
   key: "Wonder Woman 1984"
   value: 400
}
[main] INFO org.apache.kafka.clients.producer.KafkaProducer - 
[Producer clientId=producer-1] Closing the Kafka producer with timeoutMillis = 9223372036854775807 ms.

It means that our message has been sent.

Now, let us confirm that the schema has been stored in the Schema Registry.

curl  -X GET http://localhost:8081/subjects | jq

And the output which is displayed is "topicName" + "key/value"

[
   "testy1-value"
]

We can also see the schema which is stored by the registry −

curl  -X GET http://localhost:8081/schemas/ids/1 | jq {
   "schemaType": "PROTOBUF",
   "schema": "syntax = \"proto3\";\npackage theater;\n\noption java_package = \"com.tutorialspoint.theater\";\n\nmessage Theater {
      \n  string name = 1;\n  string address = 2;\n  int64 total_capacity = 3;\n  
      int64 mobile = 4;\n  float base_ticket_price = 5;\n  bool drive_in = 6;\n  
      .theater.Theater.PAYMENT_SYSTEM payment = 7;\n  repeated string snacks = 8;\n  
      repeated .theater.Theater.MovieTicketPriceEntry movieTicketPrice = 9;\n\n  
      message MovieTicketPriceEntry {\n    option map_entry = true;\n  \n    
      string key = 1;\n    int32 value = 2;\n  }\n  enum PAYMENT_SYSTEM {
         \n CASH = 0;\n    CREDIT_CARD = 1;\n    DEBIT_CARD = 2;\n    APP = 3;\n  
      }\n
   }\n"
}

Kafka Consumer with Protocol Buffers Schema

Let us now create a Kafka consumer. This consumer will consume the message which contains the theater object.

KafkaProtbufConsumer.java

package com.tutorialspoint.kafka;

import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.Properties;
import org.apache.kafka.clients.producer.Producer;
import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerRecord;
import com.tutorialspoint.theater.TheaterOuterClass.Theater;
import com.tutorialspoint.theater.TheaterOuterClass.Theater.PAYMENT_SYSTEM;

public class KafkaProtbufConsumer {
   public static void main(String[] args) throws Exception{
      String topicName = "testy1";
      Properties props = new Properties();
      props.put("bootstrap.servers", "localhost:9092");
      props.put("clientid", "foo");
      props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
         
      props.put("value.serializer", "io.confluent.kafka.serializers.protobuf.KafkaProtocol BuffersSerializer");
      props.put("schema.registry.url", "http://localhost:8081");
      props.put("auto.register.schemas", "true");
      Producer<String, Theater> producer = new KafkaProducer<>(props);
      producer.send(new ProducerRecord<String, Theater>(topicName, "SilverScreen", getTheater())).get();
      
      System.out.println("Sent to Kafka: \n" + getTheater());
      producer.flush();
      producer.close();
   }
   public static Theater getTheater() {
      List<String> snacks = new ArrayList<>();
      snacks.add("Popcorn");
      snacks.add("Coke");
      snacks.add("Chips");
      snacks.add("Soda");
           
      Map<String, Integer> ticketPrice = new HashMap<>();
      ticketPrice.put("Avengers Endgame", 700);
      ticketPrice.put("Captain America", 200);
      ticketPrice.put("Wonder Woman 1984", 400);
      
      Theater theater = Theater.newBuilder()
         .setName("Silver Screener")
         .setAddress("212, Maple Street, LA, California")
         .setDriveIn(true)
         .setTotalCapacity(320)
         .setMobile(98234567189L)
         .setBaseTicketPrice(22.45f)
         .setPayment(PAYMENT_SYSTEM.CREDIT_CARD)
         .putAllMovieTicketPrice(ticketPrice)
         .addAllSnacks(snacks)
         .build();
      return theater;
   }
}

Here is a list of points that we need to be aware of −

• We need to pass the Schema Registry URL to the Consumer.

• We also need to pass the correct Protocol Buffers Deserializer which is specific to the Schema Registry.

• The Schema Registry would automatically read the stored schema of the theater object when we are done consuming.

• Lastly, we created a theater object from our auto-generated Java code and that is what we will be sending.

Output

Let us now compile and execute the code −

mvn clean install ; java -cp .\target\protobuf-tutorial-1.0.jar com.tutorialspoint.kafka.KafkaProtbufConsumer

offset = 0, key = SilverScreen, value = May 22, 2021 7:50:15 PM com.google.protobuf.TextFormat$Printer$MapEntryAdapter compareTo
May 22, 2021 7:50:15 PM com.google.protobuf.TextFormat$Printer$MapEntryAdapter compareTo

name: "Silver Screener"
address: "212, Maple Street, LA, California"
total_capacity: 320
mobile: 98234567189
base_ticket_price: 22.45
drive_in: true
payment: CREDIT_CARD
snacks: "Popcorn"
snacks: "Coke"
snacks: "Chips"
snacks: "Soda"
movieTicketPrice {
   key: "Captain America"
   value: 200
}
movieTicketPrice {
   key: "Wonder Woman 1984"
   value: 400
}
movieTicketPrice {
   key: "Avengers Endgame"
   value: 700
}

So, as we can see, the message which was written into Kafka was correctly consumed by the Consumer. Plus, the Registry stored the schema which can also be accessed by a REST API.

Protocol Buffers - In Other Languages

We have been using Protocol Buffers in Java and Python. But there are multiple languages it supports including C++, C#, Kotlin, Dart, Go, etc. The basic stuff mostly remains the same, i.e., writing a proto schema, generating the source code via protoc binary which our code can use. Let us write a basic example for Go and Dart as part of this section.

We will use the following proto file −

greeting.proto

syntax = "proto3";
package tutorial;
message Greet {
   string greeting = 1;
   string username = 2;
}

Using Google Protocol Buffers in Go Lang

To use the above Protocol Buffers file, we will first have to generate the code for the Greet class in Go language. For that, we need to do the following −

Install the Go Protocol Buffers plugin (protoc-gen-go) which is a prerequisite for the protoc file which we have been using −

go install google.golang.org/protobuf/cmd/protoc-gen-go

Then, run the protoc with the provided ".proto" file and we will instruct it to generate the code under the "go" directory.

protoc  --go_out=go proto_files/greeting.proto

Post execution of the above command, you will notice an auto-generated class − "greeting.pb.go". This class would help us with the serialization and deserialization of the Greet object.

Now, let us create the writer of the data, which will take the username and greeting as its input −

greeting_writer.go

import "fmt"
import "io/ioutil"

func main() {
   greet := Greeting{}
   greet.username = "John"
   greet.greeting = "Hello"
   out, err := proto.Marshal(greet)
    
   ioutil.WriteFile("greeting_go_out", out , 0644)
    
   fmt.Println("Saved greeting with following data to disk:")
   fmt.Println(p)
}

Now let us create the reader which will read the file −

greeting_reader.go

import "fmt"
import "io/ioutil"

func main() {
   in, err := ioutil.ReadFile("greeting_go_out")

   greet := &pb.Greet{}
   proto.Unmarshal(in, greet)

   fmt.Println("Reading from file greeting_protobuf_output:")
   fmt.Println(greet)
}

Output

The reader simply reads from the same file, deserializes it, and prints the data about the greeting.

Now that we have setup the reader and the writer, let us compile the project.

Next, let us first execute the writer −

go run greeting_writer.go

Saved greeting with following data to disk:
{greeting: Hello, username: John}

Then, let us execute the reader −

go run greeting_reader.go

Reading from file greeting_protobuf_output
{greeting: Hello, username: John}

So, as we can see, the data that was serialized by the writer and saved to the file, that exact data is correctly deserialized by the reader and printed accordingly.

Using Google Protocol Buffers in Dart

To use the above Protocol Buffers file, we will first have to install and generate the code for the Greet class in Dart language. For that, we need to do the following −

Install the Dart Protocol Buffers plugin (protoc-gen-go) which is the prerequisite for the protoc file which we have been using. https://github.com/dart-lang/protobuf/tree/master/protoc_plugin#how-to-build-and-use

Then, run the protoc with the provided ".proto" file and we will instruct it to generate the code under the "dart" directory.

protoc  --go_out=dart proto_files/greeting.proto

Post execution of the above command, you will notice an auto-generated class − "greeting.pb.dart". This class would help us with the serialization and deserialization of the Greet object.

Now, let us create the writer of the data, which will take the username and greeting as its input −

greeting_writer.dart

import 'dart:io';
import 'dart/greeting.pb.dart';

main(List arguments) {
   Greeting greet = Greeting();
   greet.greeting = "Hello";
   greet.username = "John";
  
   File file = File("greeting_go_out");
  
   print("Saved greeting with following data to disk:")
   file.writeAsBytes(greet.writeToBuffer());
   print(greet)
}

Next, let us create a reader which will read the file −

greeting_reader.dart

import 'dart:io';
import 'dart/greeting.pb.dart';

main(List arguments) {
   File file = File("greeting_go_out");

   print("Reading from file greeting_protobuf_output:")
   Greeting greet = Greeting.fromBuffer(file.readAsBytesSync());
   print(greet)
}

Output

The reader simply reads from the same file, deserializes it, and prints the data about the greeting.

Now that we have setup the reader and the writer, let us compile the project.

Next, let us first execute the writer −

dart run greeting_writer.dart

Saved greeting with following data to disk:
greeting: Hello
username: John

And then, let us execute the reader.

dart run greeting_reader.dart

Reading from file greeting_protobuf_output
greeting: Hello
username: John

So, as we can see, the data that was serialized by the writer and saved to the file, that exact data is correctly deserialized by the reader and printed accordingly.