Lossless Real-Time Data Compression with Zstandard (zstd)

Well-organized data compression is an essential component of effective data management. It allows us to conserve disk space, reduce data transfer times, and enhance overall data efficiency. Among the numerous data compression utilities that have gained popularity year by year, -zstd stands out as a high-performing option. Developed by Yann Collet, zstd is open-source software that offers impressive compression ratios while maintaining fast decompression speeds.

Zstd is a flexible and adaptable data compression utility suitable for compressing various types of data, such as text, images, audio, and video, among others. Moreover, it is compatible with several platforms, including Linux, Windows, MacOS, and FreeBSD, making it a reliable and accessible tool for different users. Additionally, zstd's flexibility extends to its command-line interface, which is user-friendly, and packed with multiple options and features.

In this article, we will provide a detailed explanation of zstd, exploring how it operates and how to install it. Additionally, we will offer practical examples of how to compress and decompress files using zstd and demonstrate some of its advanced capabilities, such as multi-threading, dictionary learning, and compression levels among others. By the end of this article, you will gain a comprehensive understanding of zstd and how it can be harnessed to meet your data compression needs.

Installation of Zstandard on Linux

Zstandard is available in most Linux distribution's official repositories, making it easy to install using the package manager. A few commands are used to install it on Ubuntu is mentioned below −

Input command to install Zstandard

sudo apt-get update
sudo apt-get install zstd

Output will give all the details about update and install of Zstandard

user@ubuntu:~$ sudo apt-get update
[sudo] password for user: 
Get:1 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB]
Hit:2 http://us.archive.ubuntu.com/ubuntu focal InRelease                
Get:3 http://us.archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB]             
Get:4 http://us.archive.ubuntu.com/ubuntu focal-backports InRelease [101 kB]             
Get:5 http://us.archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [1,341 kB]
Get:6 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [784 kB]
Get:7 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [236 kB]
Get:8 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [44.7 kB]
Get:9 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [11.6 kB]
Get:10 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [959 kB]
Get:11 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [172 kB]
Get:12 http://us.archive.ubuntu.com/ubuntu focal-updates/main Translation-en [341 kB]
Get:13 http://us.archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [70.1 kB]
Get:14 http://us.archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [12.9 kB]
Get:15 http://us.archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [900 kB]
Get:16 http://us.archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [197 kB]
Get:17 http://us.archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [8,680 B]
Get:18 http://us.archive.ubuntu.com/ubuntu focal-backports/main Translation-en [4,156 B]
Fetched 5,297 kB in 5s (1,163 kB/s)                          
Reading package lists... Done
Building dependency tree       
Reading state information... Done
All packages are up to date.
user@ubuntu:~$ sudo apt-get install zstd
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following NEW packages will be installed:
  zstd
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 249 kB of archives.
After this operation, 1,120 kB of additional disk space will be used.
Get:1 http://us.archive.ubuntu.com/ubuntu focal/universe amd64 zstd amd64 1.4.5+dfsg-4 [249 kB]
Fetched 249 kB in 1s (177 kB/s)    
Selecting previously unselected package zstd.
(Reading database ... 233077 files and directories currently installed.)
Preparing to unpack .../archives/zstd_1.4.5+dfsg-4_amd64.deb ...
Unpacking zstd (1.4.5+dfsg-4) ...
Setting up zstd (1.4.5+dfsg-4) ...
Processing triggers for man-db (2.9.1-1) ...
user@ubuntu:~$ 

After successfully installing Zstandard, we can now move to learn some basic usage of Zstd command examples in the following section.

List of Different Examples of how to use the “Zstd” Command in Linux

1. Compress a File

To compress a file with Zstd, use the command "zstd filename". The default compression level is 3, but we can adjust using "-# (number)". Zstd is great for large files, reducing their size for easier storage and transfer. It also has a fast decompression speed for easy restoration without efficiency loss. Zstd is a simple yet powerful tool for improving data management on Linux systems.

Input command to compress a file

zstd example.txt

Output for zstd command

user@ubuntu:~$ zstd example.txt
user@ubuntu:~$

2. Decompress a File

When you want to decompress a file compressed with Zstd, use the command "zstd -d" followed by the name of the compressed file. For instance, "zstd -d example.txt.zst" will restore the compressed file to its original format. Zstd's high compression ratios and fast decompression speeds make it a reliable and efficient way to manage large files on Linux.

The command for decompressing a file

zstd -d example.txt.zst

3. Compress a Directory

Compressing entire directories with the Zstd command is very simple. All we need to do is use the command "tar cf - directory_name/ | zstd > compressed_file_name.tar.zst". For instance, let's say we have a directory called "example" that we want to compress and create a compressed file called "example.tar.zst". We need to run this command in the terminal "tar cf - example/ | zstd > example.tar.zst". This will compress the entire directory into a single file in this way you can use this command.

Input command for compressing a directory

tar cf - example/ | zstd > example.tar.zst

4. Decompress a Directory

To decompress a Zstd-compressed directory in Linux, you can use the command "zstd -d example.tar.zst | tar xvf -". Specify the name of the compressed file with the "-d" option. The "tar" command extracts the contents of the archive file using the "-xvf" options. Remember to make a backup of the compressed directory before decompressing it to avoid overwriting existing directories with the same name.

Input for decompressing a directory

zstd -d example.tar.zst | tar xvf -

Output for “zstd -d” command

$ zstd -d example.tar.zst | tar xvf -
example/
example/file1.txt
example/file2.txt
example/file3.txt

5. Multi-threaded Compression

If you have a large file that needs to be compressed quickly, we can use Zstd, a fast compression tool that utilizes multiple CPU cores to enhance compression speed. By writing this command "zstd -T0 example.txt", you can activate multi-threaded compression with a thread count of 0, which will allow Zstd to use all available CPU cores to compress the file "example.txt".In this way, you can active Multi-threaded Compression in Linux.

Input command for multi-thread compression

zstd -T0 example.txt

Output for this command may look like this

zstd -T0 example.txt
example.txt         : 100%   [==================================================================================================]  10.00 MiB/s  2.20:1  

6. Set Compression Level

Zstd has 22 compression levels with varying tradeoffs between compression ratio and speed. Use the 'zstd -<level> <file>' command to set the compression level for a file. Higher levels require more CPU resources and time. Choose the right compression level for optimal results.

Input command for set compression level

zstd -3 example.txt

Output of zstd compression level

example.txt             :  10.00 MB
--
example.txt         :  3.55 MB        (35.45%)   (x3.18)
--

Conclusion

In conclusion, Zstandard is a valuable compression tool that can save disk space and speed up data transfer. Installing zstd on Linux is a simple process, and using it to compress and decompress files is a breeze. We trust that this article has assisted you in getting started with zstd and instructed you on how to use it to enhance your data compression requirements.