Processing Linux Commands in Parallel

Processing Linux commands in parallel allows administrators to execute multiple commands simultaneously, significantly improving efficiency when performing tasks like system updates, service management, and file operations. Command chaining is the technique of combining multiple commands using various operators and methods.

Using the & (Background) Operator

The & operator runs commands in the background, allowing the shell to execute the next command without waiting for the previous one to complete.

command1 & command2

Example running ls and cat commands simultaneously:

ls & cat Linux.txt

[1] 6178
Don't erase the existing data.
Desktop    file.sh      Myfile     prateek.pcap       Templates
Documents  Linux.txt    Pictures   Public             Videos

The cat command displays the file contents while ls shows directory listing concurrently.

Using Wait with & Operator

The wait command ensures all background processes complete before returning control to the shell:

command1 & command2 & wait

Example executing a script and listing files, then checking system uptime:

./file.sh & ls & wait; uptime

[1] 3650
Desktop    file.sh     Myfile    prateek.pcap      Templates
Documents  Linux.txt   Pictures  Public            Videos
19:06:59 up 1 min, 1 user, load average: 1.31, 0.55, 0.20
[1]+ Done            ./file.sh

Using Xargs for Parallel Processing

The xargs command controls the number of parallel processes using the -P flag for maximum processes and -n for arguments per process:

ls | xargs -P 3 -n 2 echo

Desktop Documents
Downloads example.txt
file.sh Linux.txt
Pictures Public
Templates Videos

This processes files in batches of 2 using up to 3 parallel processes.

Sequential Execution Operators

Semicolon (;) Operator

Executes commands sequentially regardless of success or failure:

pwd; ls; uptime

/home/prateek
Desktop    file.sh     Myfile    prateek.pcap      Templates
Documents  Linux.txt   Pictures  Public            Videos
12:36:51 up 1 min, 1 user, load average: 3.14, 1.09, 0.39

AND (&&) Operator

Executes the second command only if the first succeeds:

mkdir MyDirectory && cd MyDirectory

OR (||) Operator

Executes the second command only if the first fails:

[ -d ~/MyDirectory ] || mkdir ~/MyDirectory

Pipe (|) Operator

Pipes output from one command as input to another:

cat Linux.txt | grep "data"

Don't erase the existing data.

GNU Parallel

A powerful tool for running commands in parallel across multiple cores or machines. Install it using:

sudo apt install parallel -y     # Debian/Ubuntu
sudo yum install parallel        # RHEL/CentOS

Example usage:

echo -e "file1.txt\nfile2.txt\nfile3.txt" | parallel -j 3 wc -l

Bash Scripts for Parallel Execution

Create a script file to execute multiple commands:

#!/bin/bash
touch myfile.txt
echo "Tutorials Point" > myfile.txt
ls && cat myfile.txt

Make executable and run:

chmod +x script.sh
./script.sh

Combining Multiple Operators

Complex command chains using multiple operators:

[ -f ~/myfile.txt ] && echo "File exists" || touch ~/myfile.txt

Conclusion

Linux provides multiple methods for parallel command execution, from simple background operations using & to advanced tools like GNU Parallel. Understanding these operators and techniques enables efficient system administration and automation. Choose the appropriate method based on whether you need true parallelism, conditional execution, or sequential processing.