Redirect output of process to a file and streams?

Overview

We'll look at some ways to redirect the output of processes to files and standard streams such as STDOut and STDERR simultaneously.

The tee Command

Tee is one of the most common Linux command line tools that we can use to redirect a process' output. It's also known as "teeing" or "piping". The tee command takes two arguments− the file name where you want the redirected output saved, and another file name which will be used for writing the original input.

Redirect stdout

Here we go! We're going to look at a simple example of redirect the output of the ls (list) commands to stdout and a tempfile called out.log.

$ ls -C | tee /tmp/out.log
bin   dev  home mnt  proc  run  srv  tmp  var
boot  etc  lib64	 opt  root  sbin  sys  usr

We can confirm that the content of the file matches the output generated by executing the command.

$ cat /tmp/out.log
bin   dev  home mnt  proc  run srv  tmp  var
boot  etc  lib64	 opt  root  sbin  sys  usr

Another important point to remember is that the default behavior for the tee command is to replace the content of the file. However, if necessary, we can use the -a (append) command to add the new contents after the existing contents of a file.

Redirect stdout and stderr to the Same File

Internally, the tee command acts as a T-splitting device for the incoming stream so that data can be directed to the output stream and one or more files at once. We can use this knowledge to redirect stderrs of processes to stdouts and files.

$ (ls -C; cmd_with_err) 2>&1 | tee /tmp/out.log
bin   dev  home mnt  proc  run srv  tmp  var
boot  etc  lib64	 opt  root  sbin  sys  usr
bash: cmd_with_err: command not found

We can see that cmd_with_error is an unknown command, which means it produces an error message. We redirect the stderror fd (fd=2) into the stdin fd (fd=1), so that tee can read from both files at once.

Alternatively, we can also use |& as a shorthand notation for 2>&1| to get the same result −

$ (ls -C; cmd_with_err) |& tee /tmp/out.log
bin   dev  home mnt  proc  run srv  tmp  var
boot  etc  lib64	 opt  root  sbin  sys  usr
bash: cmd_with_err: command not found

Now, let’s verify the content of the /tmp/out.log file −

$ cat /tmp/out.log
bin   dev  home mnt  proc  run srv  tmp  var
boot  etc  lib64	 opt  root  sbin  sys  usr
bash: cmd_with_err: command not found

Redirect stdout and stderr to Separate Files

We may sometimes want to redirect the output of a process into two different files. We can use process substitutions to invoke the tee command. Before we get into the details of the tee() function, here's an example of a template code snippet that enables the tee() function to listen to a specific fd (file description) and write back to the original fd stream and a file.

fd> >(tee file_name fd>&fd)

We must point out that fd is just an example of a file descriptor; the actual value will be one for stdout, two for stderror, and zero for stdin.

Let’s now use our understanding of streams to redirect the stdout stream of the command to /tmp/out and the stderr stream to /tmp/err.

$ ((ls -C; cmd_with_err) 1> >(tee /tmp/out.log)) 2> >(tee /tmp/err.log 2>&2)
bin   dev  home mnt  proc  run srv  tmp  var
boot  etc  lib64	 opt  root  sbin  sys  usr
bash: cmd_with_err: command not found

We can confirm that /tmp/outfile.log has the correct output, but /tmp/errorfile.log has an incorrect error message.

$ cat /tmp/out.log
bin   dev  home mnt  proc  run srv  tmp  var
boot  etc  lib64	 opt  root  sbin  sys  usr
$ cat /tmp/err.log
bash: cmd_with_err: command not found

Redirection Delays

Invocation of the tee (tee) commands to direct the process' output to a file and STDOUT can introduce delays. We’ll look at a specific situation where we might want to avoid it and learn to mitigate it if it happens.

Scenario

Here we go! We're going to write a Python program that prints the current time once per second.

$ cat time.py
#!/usr/bin/python
from datetime import datetime
import time
import sys
from sys import stdout
while True:
   sys.stdout.write(datetime.today().strftime("%H:%M:%S %p
"))
    time.sleep(1)

If we run this script, we‘ll see a 1-sec delay between each timestamp printed out by the script.

$ ./time.py
   9:29:48 PM
   9:29:49 PM
   9:29:50 PM
   9:29:51 PM
   9:29:52 PM
   9:29:53 PM

Delayed Redirection

Let's now use the "te" command to redirect the output from this script to stdout and the "time.out" log.

$ ./time.py | tee time.out

We’ll see that there’s not any stdout printed out for a long period of times, then a huge chunk of stdout will be printed out at once.

Python’s internal buffer size is set to 1MB, which means that when writing to stdout, the output may be delayed for up to 1 second before actually appearing on screen. Buffering policies cause writes to stdout to be passed through a 4096–byte (or larger) buffer, which reduces the number of I/Os needed to output text to the terminal.

For interactive applications, any delay in redirecting from one page to another must be avoided. To address the redirecting delay problem, we need to look for ways to mitigate it.

Mitigation

One way to fix this problem is by ensuring that the output from the program is flushed to the console at regular intervals.

$ cat time.py
#!/usr/bin/python
from datetime import datetime
import time
import sys
from sys import stdout
while True:
   sys.stdout.write(datetime.today().strftime("%H:%M:%S %p
"))
   sys.stdout.flush()
   time.sleep(1)
Let's check if the delay has been removed:
$ ./time.py | tee time.out
21:46:12 PM
21:46:13 PM

We had direct control over the application code, so that allowed us to change it.

However, in many instances, the program could be a compiled binary, and we may not have access to modify its code. If we want to avoid the delay associated with writing to stdout, we can use the "unbuffer" program in Linux to solve the problem.

Let's remove the sys.stdout.flush() method call from our code and run the redirections again using the unbuffer command.

$ unbuffer ./time.py | tee time.out
21:52:22 PM
21:52:23 PM

There were no unexpected delays in stdout write operations after we changed our code.

Conclusion

We looked at several examples of redirecting a program's outputs to both stdin and stdouts simultaneously. We also learned some techniques for solving the problem of delayed redirections caused by buffered requests.