Protocol Buffers - Basic App

Quiz

Overview

Let us now use Google Protocol Buffer and see how it works with a simple Greeting app. In this example, we will create a simple application which would do the following −

Greeting the Writer −
- Take greeting and username from the user
- Store the above information in a file in the disk
Greeting Reader −
- Reads the same file which we stored in the above file
- Convert that data into an object and print the data

Protocol Buffer Definition file

The protocol buffer "definition file" contains the schema definition of the data we want to serialize. The data is stored in a human readable file with the extension ".proto".

Let us store the following data in greeting.proto and we will use this in our first application.

syntax = "proto3";
package tutorialspoint;
option java_package = "com.tutorialspoint.greeting";

message Greet {
   string greeting = 1;
   string username = 2;
}

Understanding each construct

Now, let us take a closer look at the data and see what each line of code does in the above code block.

syntax = "proto3";

The syntax here represents what version of Protobuf we are using. So, we are using the latest version 3 and the schema thus can use all the syntax which is valid for version 3.

package tutorialspoint;

The package here is used for conflict resolution if, say, we have multiple classes/members with same name.

option java_package = "com.tutorialspoint.greeting";

This argument is specific to Java, i.e., the package where the code from the .proto file will be auto-generated.

message Greet

Name of the base class for the object which would be created/recreated.

string greeting = 1;
string username = 2;

These are the attributes of the Greet class along with the data type and the position of the tag in the schema. If a new tag is to be added, it should have "3" as the position. Note that this position integer is important to ensure that the actual data is compact and there is scope of schema evolution.

Protocol Buffer Code Generation

Now that we have defined, let us install the "proto" binary which we will use to autogenerate the code for the above Greet class. The binaries can be found at "https://github.com/protocolbuffers/protobuf/releases/".

Choose the correct binary based on the OS. We will install proto binary on Windows but the steps are not very different for Linux.

We've downloaded https://github.com/protocolbuffers/protobuf/releases/download/v31.1/protobuf-31.1.zip

Verify Proto Compiler Setup

Once installed, ensure that you are able to access it via command line −

protoc --version

libprotoc 31.1

It confirms that Protobuf is correctly installed. Now let us move to creating the Greeting app described above for Java.

Project Structure

Here is the overall project structure that we would have −

Greeting App in Java

Now that we have installed protoc, we can auto-generate the code from the proto files using protoc. Let us first create a Java project though.

Following is the Maven configuration that we will use for our Java project. Note that it contains the required library for Protobuf-java as well.

pom.xml

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
   xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
   xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
	
   <modelVersion>4.0.0</modelVersion>
   <groupId>com.tutorials.point</groupId>
   <artifactId>protobuf-tutorial</artifactId>
   <version>1.0</version>
   <packaging>jar</packaging>

   <properties>
      <maven.compiler.source>21</maven.compiler.source>
      <maven.compiler.target>21</maven.compiler.target>
   </properties>

   <dependencies>
      <!-- https://mvnrepository.com/artifact/com.google.protobuf/protobuf-java -->
      <dependency>
         <groupId>com.google.protobuf</groupId>
         <artifactId>protobuf-java</artifactId>
         <version>4.31.1</version>
      </dependency>
   </dependencies>

   <build>
      <plugins>
         <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-shade-plugin</artifactId>
            <version>3.2.4</version>
            <configuration>
               <!--Put your configurations here-->
            </configuration>
            <executions>
               <execution>
                  <phase>package</phase>
                     <goals>
                     <goal>shade</goal>
                  </goals>
               </execution>
            </executions>
         </plugin>
      </plugins>
   </build>
</project>

All of our code would be present under src/main/java.

With the project structure out of the way, let us generate the code for the Greet class −

Generate Java Classes

protoc --java_out=. greeting.proto

Post execution of the command, you will notice a auto-generated class under com > tutorialspoint > greeting folder within current directory.

Greeting.java

This file contains a class Greeting and an interface GreetOrBuilder which would help us with serialization and deserialization of the Greet object.

Using Generated Java Classes

Now, let us write the writer of the data, which will take the username and the greeting as its inputs −

GreetWriter.java

package com.tutorialspoint.greeting;

import java.io.FileNotFoundException;
import java.io.FileOutputStream;
import java.io.IOException;

import com.google.flatbuffers.FlatBufferBuilder;

public class GreetWriter {
	public static void main(String[] args) throws FileNotFoundException, IOException {
		// create a flat buffer builder
		// it will be used to create Greet FlatBuffer
		FlatBufferBuilder builder = new FlatBufferBuilder(1024);
		
		// read greeting and username from console
		int greeting = builder.createString(args[0]);
		int username = builder.createString(args[1]);
		
		// create Greet FlatBuffers using startGreet() method
		Greet.startGreet(builder);
		// add the greeting and username to the Greet FlatBuffer
		Greet.addGreeting(builder, greeting);
		Greet.addUsername(builder, username);
		
		// mark, data being entered in Greet FlatBuffer
		int greet = Greet.endGreet(builder);
		
		// finish the builder
		builder.finish(greet);
		
		// get the bytes to be stored
		byte[] data = builder.sizedByteArray();
		
	      String filename = "greeting_flatbuffers_output";
	      System.out.println("Saving greeting to file: " + filename);
		// write the builder content to the file named	greeting_flatbuffers_output
	      try(FileOutputStream output = new FileOutputStream(filename)){
	    	  output.write(data);
	      }
	      System.out.println("Saved greeting with following data to disk: \n" + greeting);
	}
}

The writer simply takes CLI arguments, creates the Greet object, serializes it and then dumps it to a file.

Now let us write a reader which will read the file −

GreetReader.java

package com.tutorialspoint.greeting;

import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.IOException;

import com.google.flatbuffers.FlatBufferBuilder;

public class GreetReader {
	public static void main(String[] args) throws FileNotFoundException, IOException {
		// create a flat buffer builder
		// it will be used to read Greet FlatBuffer
		FlatBufferBuilder builder = new FlatBufferBuilder(1024);
		 String filename = "greeting_flatbuffers_output";
	      System.out.println("Reading from file " + filename);
		    
	      try(FileInputStream input = new FileInputStream(filename)) {
	    	  // get the serialized data
	         byte[] data = input.readAllBytes();
	         java.nio.ByteBuffer buf = java.nio.ByteBuffer.wrap(data);
	         // read the root object in serialized data
	         Greet greet = Greet.getRootAsGreet(buf);
	    	 
	         // print greet values 
	         System.out.println("Greeting: " + greet.greeting() + "\n" + "Username: " + greet.username());
	      }		
	}
}

The reader simply reads from the same file, deserializes it, and prints the data about the greeting.

Compile the project

Now that we have set up the reader and the writer, let us compile the project.

mvn clean install

Serialize the Java Object

And now, let us first execute the writer to serialize an object to file system.

java -cp .\target\protobuf-tutorial-1.0.jar com.tutorialspoint.greeting.GreetWriter Hello John

Saving greeting to file: 
greeting_protobuf_output

Saved greeting with following data to disk:
greeting: Hello
username: John

Deserialize the Serialized Object

And then, let us execute the reader to deserialize an object from file system.

java -cp .\target\protobuf-tutorial-1.0.jar com.tutorialspoint.greeting.GreetReader

Reading from file greeting_protobuf_output
Greeting: Hello
Username: John

So, as we see the data that was serialized by the writer and saved to the file, that exact data is correctly deserialized by the reader and printed accordingly.

Greeting App in Python

Let us now write the same example as a Python project −

Install protocol buffers library

We will need to install protobuf pip package before we proceed.

pip install protobuf

All of our code would be present under python directory.

Generate Python classes from proto file

With the project structure out of the way, let us generate the code for Greet class −

protoc  --python_out=. greeting.proto

Post execution of this command, you will notice an auto-generated class greeting_pb2.py in current directory. This class would help us with serialization and deserialization of the Greet object.

Using Generated Python Classes

Now, let us write the writer of the data, which will take the username and the greeting as its input −

greetWriter.py

import greeting_pb2

import sys

greet = greeting_pb2.Greet()
greet.username = sys.argv[1]
greet.greeting = sys.argv[2]

filename = "greeting_protobuf_output";
print("Saving to file: " + filename)

f = open(filename, "wb")
f.write(greet.SerializeToString())
f.close()
print("Saved following greeting to disk: \n" + str(greet))

The writer simply takes CLI arguments, creates the Greet object, serializes it, and then dumps it to a file.

Now let us create a reader which will read the file −

greetReader.py

import greeting_pb2

greet = greeting_pb2.Greet()

filename = "greeting_protobuf_output";
print("Reading from file: " + filename)

f = open(filename, "rb")
greet.ParseFromString(f.read())
f.close()

print("Read greeting from disk: \n" + str(greet))

The reader simply reads from the same file, deserializes it, and prints the data about the greeting.

Serialize the Python Object

Now, let us first execute the writer.

py greetWriter.py Hola Jane

Saving to file: greeting_protobuf_output
Saved following greeting to disk:
greeting: "Hola"
username: "Jane"

Deserialize the Python Object

And then, let us execute the reader.

python greetReader.py

Reading from file: greeting_protobuf_output
Read greeting from disk:
greeting: "Hola"
username: "Jane"

So, as we see, the data that was serialized by the writer and saved to a file. Next, the same data is correctly deserialized by the reader and printed accordingly.

Print Page