Understanding Stale file handles in Linux

In this article, we will discuss the concept of stale file handles in Linux and how to understand and resolve them. We'll explore the relationship between file handles, inodes, and the filesystem to understand why stale file handles occur and how to fix them.

The examples in this tutorial have been tested on Debian 10.10 (Buster) with GNU Bash 5.0.3. The concepts are POSIX-compliant and apply to most Unix-like systems.

What are File Handles?

A file handle (or file descriptor) is an integer value that represents a reference to an open file. The operating system maintains a table of all open files and their corresponding file handles. When a process opens a file, the OS creates a file handle and associates it with that file's inode (index node).

File handles are process-specific and stored in the file descriptor table (FDT). They don't live on the filesystem itself but are maintained in memory by the kernel to track open files for each process.

Understanding Inodes

An inode is a filesystem data structure that contains metadata about a file or directory, including permissions, timestamps, size, and the location of data blocks. Every filesystem object has an associated inode number that uniquely identifies it.

We can view inode information using the stat command:

user@system:~$ stat filename
File: filename
Size: 583 Blocks: 8 IO Block: 4096 regular file
Device: 810h/2064d Inode: 666 Links: 1
Access: (0644/-rw-r--r--) Uid: (1000/user) Gid: (1000/user)

This shows that the file references inode number 666.

File Handle to Inode Relationship

We can examine a process's open file descriptors by looking at /proc/PID/fd:

user@system:~$ ls -l /proc/1296/fd
total 0
lrwx------ 1 user user 64 Jul 1 10:49 0 -> /dev/pts/1
lrwx------ 1 user user 64 Jul 1 10:49 1 -> /dev/pts/1
lrwx------ 1 user user 64 Jul 1 10:49 2 -> /dev/pts/1
lrwx------ 1 user user 64 Jul 1 10:50 3 -> /home/user/filename

Here, file descriptor 3 points to /home/user/filename. We can verify the inode using:

user@system:~$ stat -L /proc/1296/fd/3
File: /proc/1296/fd/3
Size: 583 Blocks: 8 IO Block: 4096 regular file
Device: 810h/2064d Inode: 666 Links: 1

When File Handles Become Stale

A stale file handle occurs when a file handle references an inode that no longer exists or has been reused for a different file. This commonly happens when:

• A file is deleted while a process still has it open

• The filesystem is unmounted and remounted

• Network filesystems (NFS, CIFS) lose connection and the server reassigns inode numbers

• The inode is freed and reused by the filesystem

When attempting to access a stale file handle, you'll typically see an error like:

user@system:~$ cd /mounted_directory
-bash: cd: /mounted_directory: Stale file handle

The system returns ESTALE error code (116) with the message "Stale file handle".

Common Scenarios

Network Filesystems

Stale file handles are most common with NFS (Network File System) and CIFS/SMB mounts. When the network connection is lost or the server is restarted, inode numbers may change, causing existing file handles to become stale.

Local Filesystem Issues

On local filesystems, stale handles can occur after filesystem repairs, forced unmounts, or when files are deleted while still open by running processes.

Resolving Stale File Handles

The resolution depends on the type of filesystem and scenario:

For Network Filesystems

• Remount the filesystem ? Use umount and mount commands

• Restart the process ? Force the application to reopen files with fresh handles

• Use soft mounts ? Configure NFS with soft option for better error handling

For Local Filesystems

• Restart the affected process ? This forces reallocation of file handles

• Check filesystem integrity ? Use fsck if corruption is suspected

• Reboot the system ? As a last resort for persistent issues

Prevention Strategies

Conclusion

Stale file handles occur when the link between a process's file handle and the filesystem's inode is broken. Understanding this relationship helps in diagnosing and resolving these issues. Most stale handle problems can be resolved by remounting filesystems or restarting processes, though prevention through proper error handling is the best approach.