Flat Buffers - Introduction

Before we jump into Flat Buffers, let us go over a brief background of Serialization which is what Flat Buffers does.

What is Serialization and Deserialization?

Whenever we need to persist an object state to memory system, we need serialization. In serialization, we convert the object into bytes and stored the bytes in memory system. These stored bytes can then later be deserialized to get the object state back. As we are converting object to bytes, it can be stored anywhere including file system, messaging queues, databases and so on and then we can transfer these bytes to different machine and retrieve the object state as such.

Why do we need Serialization and Deserialization?

Serialization helps to persist an object state and then we can transfer it to anywhere over the network. Once received, we can deserialized the object or in other words, we can restore our object from bytes on different machine at any time. This is one of the many important use case of Serialization and Deserialization. Another important use case is where object is to be transferred over network. Message Queue, Database objects, REST APIs all work on this principle. In such cases, the object is serialized by the sender first and then transferred to the receiver. Receiver then deserialize the serialized object.

In REST APIs, Micro-services architecture, the application is generally broken down into small services and these services communicate with each other via messaging queue and APIs. As the communication is over network which requires frequent conversion of object to bytes and back to objects. So, serialization and deserialization becomes very critical aspects when it comes to distributed environment.

Why Flat Buffers?

Google Flat Buffers perform the serialization and deserialization of the objects to bytes which can be transferred over the network. But there are some other libraries and mechanisms to transfer data as well.

So, what makes Flat Buffers special? Here are some of its important features −

• Language independent − Flat Buffers compiler can create code for many languages like Java, Python, Go, C, C++ etc. So, a Java object can be serialized into bytes from a Java program and can be deserialized to a a Python object and vice versa.

• Efficient Data Compaction − Initially developed for Gaming environment and performance-critical systems, the flat buffers API is designed keeping data compaction and performance in mind. It is very memory efficient and even faster than Google Protocol Buffers, another Google Library for serialization and deserialization.

• Backward and Forward Compatability − Flat Buffers architecture is both backward and forward compatible. The schema of flat buffers supports adding changes in newer code and allows to deprecated older changes without breaking backward Compatability.

• Simple to use − Flat Buffers library auto-generate serialization code (as we will see in the upcoming chapters), has a versioning scheme to ensure that the creator of data and the user of data can have separate versions of the serialization definition, etc.

• JSON convertible Flat buffers schema file can be converted to JSON file and similarly we can convert a JSON file using flat buffers schema.

Flat Buffers vs Others (XML/JSON/Java serialization)

Let's take a look how other ways to transfer data over a network stack up against Flat Buffers.