How to use the sub process module with pipes on Linux?

In Python, the subprocess module allows us to work with additional processes and provides a high-level interface for executing system commands. While other modules like os.spawn(), os.system(), and os.popen() offer similar functionality, subprocess is recommended because it provides better control, security, and flexibility.

When working with subprocess on Linux, pipes allow us to chain commands together, passing the output of one command as input to another. This is essential for building secure command pipelines without shell injection vulnerabilities.

Basic Subprocess Example

Let's start with a simple example that demonstrates basic subprocess usage:

import subprocess

# Execute a simple command
subprocess.call(['ls', '-ltr'], shell=True)

Running this code will display the directory contents:

$ python sample.py

__init__.py
index.rst
interaction.py
repeater.py
signal_child.py
signal_parent.py
subprocess_check_call.py
subprocess_check_output.py
subprocess_popen_read.py
subprocess_popen_write.py
...

Why Avoid Shell=True

Using shell=True can create security vulnerabilities through shell injection attacks. Consider this problematic example:

def count_number_of_lines(website):
    return subprocess.check_output('curl %s | wc -l' % website, shell=True)

While this returns the number of lines from a website (e.g., '7
' for Google), it's vulnerable to malicious input that could execute arbitrary commands.

Using Pipes for Secure Command Chaining

The secure approach uses subprocess.Popen() with pipes to chain commands without shell interpretation:

import subprocess

def count_number_of_lines(website):
    # First process: curl command
    args1 = ['curl', website]
    process_curl = subprocess.Popen(args1, stdout=subprocess.PIPE, shell=False)
    
    # Second process: wc command
    args2 = ['wc', '-l']
    process_wc = subprocess.Popen(args2, stdin=process_curl.stdout, 
                                  stdout=subprocess.PIPE, shell=False)
    
    # Close the stdout of the first process to allow it to receive SIGPIPE
    process_curl.stdout.close()
    
    # Get the output
    return process_wc.communicate()[0]

# Usage
result = count_number_of_lines('www.google.com')
print(result)  # Output: '7<br>'

How Pipes Work in This Example

The pipe mechanism works as follows:

1. Process 1 (curl): Fetches the webpage content and outputs to stdout=subprocess.PIPE
2. Process 2 (wc -l): Takes input from stdin=process_curl.stdout and counts lines
3. Pipe connection: The stdout of curl becomes the stdin of wc
4. Output collection: communicate()[0] returns the final result

Key Advantages of Using Pipes

• Security: No shell interpretation prevents injection attacks

• Control: Better error handling and process management

• Efficiency: Direct process communication without shell overhead

• Portability: Works consistently across different systems

Best Practices

• Always use shell=False when possible

• Pass commands as lists rather than strings

• Close intermediate stdout pipes to prevent deadlocks

• Use communicate() to properly wait for process completion

Conclusion

Using subprocess with pipes provides a secure and efficient way to chain Linux commands in Python. By avoiding shell=True and properly connecting processes through pipes, you can build robust command pipelines while maintaining security and control over process execution.