Today I discovered powerful Unix tools for process investigation and management
that make system debugging much more efficient and safer.
Using lsof to Scan Processes by Path
The lsof (list open files) command can identify processes that are using files
in a specific directory path, which is invaluable for debugging and system
maintenance.
Basic lsof Path Scanning:
Find Processes Using a Directory:
1
2
3
4
5
6
7
8
9
10
11
| # See all processes accessing files in /var/log
lsof +D /var/log
# More efficient for large directories (doesn't recurse)
lsof +d /var/log
# Find processes with files open in current directory
lsof +d .
# Find processes using any files under /home/user
lsof +D /home/user
|
Practical Use Cases:
1
2
3
4
5
6
7
8
9
10
11
| # Before unmounting a filesystem
lsof +D /mnt/external-drive
# Debug why a directory can't be deleted
lsof +D /tmp/app-cache
# Find processes preventing package updates
lsof +D /usr/lib/myapp
# Identify processes holding log files open
lsof +D /var/log/myapp
|
Advanced lsof Usage:
Combine with Other Filters:
1
2
3
4
5
6
7
8
9
10
11
| # Processes in path owned by specific user
lsof +D /home/user -u user
# Network connections from processes in specific path
lsof +D /opt/myapp -i
# Find processes with deleted files still open
lsof +D /var/lib/myapp | grep '(deleted)'
# Monitor real-time file access
lsof +D /var/log -r 2 # refresh every 2 seconds
|
Output Interpretation:
1
2
3
4
| # lsof output columns explained
$ lsof +d /tmp
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
chrome 1234 user 15u REG 8,1 12345 67890 /tmp/temp_file
|
- COMMAND: Process name
- PID: Process ID
- USER: Process owner
- FD: File descriptor (15u = file descriptor 15, read/write)
- TYPE: File type (REG = regular file, DIR = directory)
- DEVICE: Device identifier
- SIZE/OFF: File size or offset
- NODE: Inode number
- NAME: Full path to file
Troubleshooting Scenarios:
“Device or resource busy” Errors:
1
2
3
4
5
6
7
8
9
| # Can't unmount filesystem
umount: /mnt/data: device is busy.
# Find the culprit
lsof +D /mnt/data
# Shows processes still accessing files on the mounted filesystem
# Alternative approach
fuser -v /mnt/data # Shows processes using the mount point
|
Disk Space Issues:
1
2
3
4
5
| # Find processes with large deleted files still open
lsof +D /var/log | grep deleted | sort -k7 -nr
# Find processes writing to a specific directory
lsof +D /var/log -a -w # -a = AND, -w = write access only
|
Advanced kill Command with Verbose Signals
The kill command accepts human-readable signal names, making process
management safer and more self-documenting.
Verbose Signal Usage:
Common Readable Signals:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
| # Graceful termination (allows cleanup)
kill -TERM 1234
kill -SIGTERM 1234
# Force termination (immediate, no cleanup)
kill -KILL 1234
kill -SIGKILL 1234
# Stop/pause process (can be resumed)
kill -STOP 1234
kill -TSTP 1234 # Terminal stop (Ctrl+Z equivalent)
# Resume stopped process
kill -CONT 1234
kill -SIGCONT 1234
# Reload configuration (common in daemons)
kill -HUP 1234
kill -SIGHUP 1234
|
Advanced Signal Management:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
| # Check if process is running (signal 0 doesn't affect process)
if kill -0 1234 2>/dev/null; then
echo "Process 1234 is running"
else
echo "Process 1234 is not running"
fi
# Graceful restart script
graceful_restart() {
local pid=$1
echo "Sending TERM signal to process $pid"
kill -TERM $pid
# Wait up to 10 seconds for graceful shutdown
for i in {1..10}; do
if ! kill -0 $pid 2>/dev/null; then
echo "Process terminated gracefully"
return 0
fi
sleep 1
done
echo "Process didn't terminate gracefully, forcing..."
kill -KILL $pid
}
|
Combining lsof and kill:
Advanced Process Management:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
| # Kill all processes using files in a directory
lsof +D /path/to/app | awk 'NR>1 {print $2}' | sort -u | xargs kill -TERM
# More precise version with error handling
kill_processes_in_path() {
local path=$1
local signal=${2:-TERM}
echo "Finding processes using files in $path"
local pids=$(lsof +D "$path" 2>/dev/null | awk 'NR>1 {print $2}' | sort -u)
if [ -z "$pids" ]; then
echo "No processes found using files in $path"
return 0
fi
echo "Found processes: $pids"
echo "Sending $signal signal..."
for pid in $pids; do
if kill -0 "$pid" 2>/dev/null; then
echo "Killing process $pid"
kill -"$signal" "$pid"
fi
done
}
# Usage
kill_processes_in_path /opt/myapp TERM
|
Service Management Patterns:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
| # Safe service restart
restart_service() {
local service_path=$1
# Find main process
local main_pid=$(pgrep -f "$service_path/bin/main")
if [ -n "$main_pid" ]; then
echo "Stopping service (PID: $main_pid)"
kill -TERM "$main_pid"
# Wait and verify
sleep 5
if kill -0 "$main_pid" 2>/dev/null; then
echo "Service didn't stop gracefully, forcing..."
kill -KILL "$main_pid"
fi
fi
# Clean up any remaining processes
lsof +D "$service_path" | awk 'NR>1 {print $2}' | sort -u | while read pid; do
if [ -n "$pid" ] && kill -0 "$pid" 2>/dev/null; then
echo "Cleaning up remaining process: $pid"
kill -TERM "$pid"
fi
done
}
|
These tools provide powerful capabilities for system administration, debugging,
and process management, making it easier to understand what processes are doing
and manage them safely.
