Send stdout to Multiple Commands

Sending stdout to multiple commands is a fundamental technique in Unix-like operating systems that allows you to process the output of one command through multiple operations simultaneously. This approach is essential for creating efficient data processing pipelines and automating complex workflows.

Standard output (stdout) is the default stream where programs write their output data. When you run a command in the terminal, the results typically appear on your screen through stdout. Understanding how to redirect and duplicate this output enables powerful command-line operations.

Methods for Sending stdout to Multiple Commands

Using Pipes

Pipes (|) redirect the output of one command as input to another command, creating a sequential processing chain.

cat file.txt | grep "pattern" | sort | uniq -c

This command reads a file, searches for a pattern, sorts the results, and counts unique occurrences in sequence.

Using the tee Command

The tee command reads from standard input and writes to both standard output and one or more files simultaneously, effectively duplicating the data stream.

cat data.txt | tee output.txt | wc -l

This saves the file contents to output.txt while also counting the lines.

For multiple destinations with tee and process substitution:

cat data.txt | tee >(grep "error" > errors.txt) >(wc -l > linecount.txt)

Process Substitution

Process substitution allows you to use the output of a command as input for multiple commands using <(command) or >(command) syntax.

diff <(sort file1.txt) <(sort file2.txt)

This compares two files after sorting them, without creating temporary files.

Practical Examples

Log Analysis Pipeline

tail -f access.log | tee >(grep "ERROR" >> errors.log) >(awk '{print $1}' | sort | uniq -c > ip_counts.txt)

This monitors a log file in real-time, saves errors to one file, and counts unique IP addresses in another.

Data Processing with Multiple Outputs

curl -s "https://api.example.com/data" | tee raw_data.json | jq '.results[]' | tee >(grep "active" > active_items.json) >(wc -l > total_count.txt)

This fetches API data, saves the raw response, processes it with jq, filters active items, and counts total results simultaneously.

Performance Comparison

Best Practices

• Use tee when you need to save intermediate results while continuing processing

• Combine methods for complex data workflows requiring multiple output streams

• Consider memory usage with large datasets pipes process data incrementally

• Test process substitution compatibility across different shell environments

Conclusion

Sending stdout to multiple commands enables efficient data processing pipelines that can simultaneously perform different operations on the same input stream. The combination of pipes, tee command, and process substitution provides flexible solutions for complex automation tasks and real-time data analysis workflows.