How to find out which process was killed by Linux OOM killer?

The Out of Memory (OOM) killer is a Linux kernel mechanism that terminates processes when the system runs out of available memory. When a process is killed by the OOM killer, the system logs this information to help administrators identify which processes were terminated and why.

To find out which process was killed by the Linux OOM killer, we use the grep command to search through system log files. The grep command filters searches in files for specific patterns and is essential for analyzing log data.

Basic grep Command Syntax

grep [options] pattern [files]

Common grep options for log analysis −

-c : Count lines that match the pattern
-h : Display matched lines only
-i : Ignore case for matching
-l : Print filenames only
-n : Show line numbers with matched lines
-v : Show lines that do NOT match the pattern
-r : Search recursively through directories

Finding OOM Killer Activity

OOM killer logs are stored in the /var/log directory. The primary log file to check is /var/log/syslog (or /var/log/messages on some distributions).

Command to Check OOM Kills

grep -i 'killed process' /var/log/syslog

Sample Output

[11686.043647] Killed process 2603 (ssrv) total-vm:149536kB, anon-rss:72174kB, file-rss:4228kB
[12890.234521] Killed process 3421 (mysql) total-vm:2048576kB, anon-rss:1843200kB, file-rss:51200kB

Alternative Search Commands

You can also search for other OOM-related patterns −

# Search for OOM killer invocation
grep -i 'out of memory' /var/log/syslog

# Search for memory allocation failures
grep -i 'oom-killer' /var/log/syslog

# Check kernel messages
dmesg | grep -i 'killed process'

Understanding the Output

The OOM killer log entry contains valuable information −

Searching in Multiple Log Files

# Search recursively in all log files
grep -rni "killed process" /var/log/

# Search in archived logs (compressed)
zgrep -i 'killed process' /var/log/syslog.*.gz

Conclusion

Finding OOM killer activity is crucial for system monitoring and debugging memory issues. Use grep -i 'killed process' /var/log/syslog to identify terminated processes, and analyze the output to understand memory usage patterns and system performance bottlenecks.