strace and ltrace: Debugging Processes at the System Call Level

When a program behaves unexpectedly and you can’t attach a debugger, or you’re working with a binary you don’t have source for, strace and ltrace cut through the mystery. They show you exactly what a process is asking the kernel to do — every file it opens, every network connection it makes, every signal it sends. This guide covers both tools from practical first use through advanced production debugging techniques.

The Core Idea

Every program running on Linux interacts with the kernel through system calls (syscalls). Reading a file, writing to a socket, spawning a child process, allocating memory — none of these happen directly. They all go through a well-defined syscall interface.

strace intercepts these syscall transitions and prints them. You see every request the program makes to the kernel, its arguments, and the return value.

ltrace does the same thing but at the shared library level — it intercepts calls to functions in shared libraries like libc. Where strace shows read(3, ...), ltrace shows fread(...).

The two tools are complementary. strace is more fundamental and reliable; ltrace is useful when you want higher-level function names rather than raw syscalls.

Installation

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

# Debian / Ubuntu
apt-get install -y strace ltrace

# RHEL / Rocky / Fedora
dnf install -y strace ltrace

# Alpine
apk add strace

# Verify
strace -V
ltrace --version

Permissions

strace uses the ptrace syscall to attach to processes. You need either:

• Root (sudo)
• The same UID as the target process
• CAP_SYS_PTRACE capability

In containers, ptrace is often restricted by the seccomp profile. You may need --cap-add=SYS_PTRACE in Docker, or securityContext.capabilities.add: ["SYS_PTRACE"] in Kubernetes.

Basic Usage

Trace a New Process

1

strace ls /tmp

This runs ls /tmp under strace and prints every syscall to stderr. The output is dense — ls makes around 70 syscalls to list a directory.

Attach to a Running Process

1
2
3
4
5
6
7
8

# By PID
strace -p 1234

# Attach to all threads of a multi-threaded process
strace -p 1234 -f

# Attach to multiple PIDs at once
strace -p 1234 -p 5678

Press Ctrl+C to detach. The target process continues running.

Trace and Save Output

1
2
3
4
5

# Write strace output to a file (stderr by default)
strace -o /tmp/trace.txt ls /tmp

# Also show output on terminal
strace -o /tmp/trace.txt -v ls /tmp

Reading the Output

A typical strace line looks like this:

openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3

The format is: syscall_name(arg1, arg2, ...) = return_value

• openat — the syscall name
• AT_FDCWD — first argument: current working directory (a constant)
• "/etc/ld.so.cache" — second argument: the file path
• O_RDONLY|O_CLOEXEC — third argument: flags (decoded from integers to human-readable names)
• = 3 — return value: file descriptor number 3 was opened

Error returns look like this:

openat(AT_FDCWD, "/etc/missing.conf", O_RDONLY) = -1 ENOENT (No such file or directory)

The -1 is the error return value, ENOENT is the errno constant, and the parenthetical is the human-readable description.

Signal Notation

--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=7823, si_uid=1000, si_status=0} ---

Signals appear between --- markers, showing the signal name and metadata.

Process Exit

+++ exited with 0 +++

Essential Flags

-e trace= — Filter by Syscall or Category

This is the most important flag. Without filtering, the output is overwhelming.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

# Trace only file-related syscalls
strace -e trace=file ls /tmp

# Trace only network syscalls
strace -e trace=network curl -s https://example.com

# Trace only process creation
strace -e trace=process bash -c 'ls && echo done'

# Trace specific syscalls by name
strace -e trace=open,read,write,close cat /etc/hosts

# Trace everything EXCEPT a category
strace -e trace='!memory' myapp

Syscall categories:

-f — Follow Forks

Trace child processes too. Essential for daemons, shells, and anything that spawns subprocesses.

1

strace -f bash -c 'echo hello | grep hello'

With -f, each line is prefixed with the PID:

1234  execve("/bin/echo", ["echo", "hello"], ...) = 0
1235  execve("/bin/grep", ["grep", "hello"], ...) = 0

-ff -o — One File Per Process

When tracing many processes, write each PID’s trace to a separate file:

1
2

strace -ff -o /tmp/trace nginx
# Creates: /tmp/trace.1234, /tmp/trace.1235, ...

-t, -tt, -ttt — Timestamps

1
2
3
4

strace -t  myapp    # Wall clock time (HH:MM:SS)
strace -tt myapp    # Microsecond precision (HH:MM:SS.usec)
strace -ttt myapp   # Epoch time with microseconds
strace -T myapp     # Time spent in each syscall

-T — Time Per Syscall

Shows how long each syscall took in seconds:

1
2

strace -T -e trace=read myapp
# read(3, "...", 4096) = 512  <0.000123>

-c — Summary Statistics

Instead of printing each syscall, print a summary table at the end:

1

strace -c ls /tmp

% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 34.21    0.000285          14        20           mmap
 18.44    0.000154          11        14           read
 12.33    0.000103          51         2           getdents64
  9.87    0.000082           8        10           openat
  ...
------ ----------- ----------- --------- --------- ----------------
100.00    0.000834                    96         4 total

Combine with -e to profile specific categories:

1

strace -c -e trace=network myapp

-s — String Length

By default, strace truncates strings at 32 characters. Increase this to see full paths, data, etc.:

1
2

strace -s 256 myapp
strace -s 0 myapp    # No truncation (show full strings)

-y — Decode File Descriptors

Annotates file descriptor numbers with their actual paths:

1
2

strace -y cat /etc/hosts
# read(3</etc/hosts>, "127.0.0.1  localhost\n...", 131072) = 421

Without -y you’d just see read(3, ...) and have to infer what fd 3 is.

-P — Filter by Path

Only show syscalls that touch a specific path:

1
2

strace -P /etc/hosts cat /etc/hosts
strace -P /var/log/ -P /etc/nginx/ nginx -t

Real Debugging Scenarios

“Why is this program slow to start?”

1

strace -tt -T -e trace=file myapp 2>&1 | head -100

Look for:

• Repeated openat calls to paths that don’t exist (ENOENT)
• stat calls on missing paths
• Large gaps in timestamps (-tt helps spot these)

A common culprit is DNS lookup misconfiguration:

1
2
3

strace -tt -e trace=network myapp 2>&1 | grep -E "connect|send|recv"
# connect(4, {sa_family=AF_INET, sin_port=htons(53), ...  <1.823456>
# Spent 1.8 seconds on a DNS query

“Why can’t this program read its config file?”

1

strace -e trace=file,desc -s 256 myapp 2>&1 | grep -E "open|ENOENT|EACCES|EPERM"

This immediately shows every file the program tried to open and whether it succeeded. You’ll see exactly which path it’s looking in:

openat(AT_FDCWD, "/etc/myapp/config.yaml", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/local/etc/myapp.yaml", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/home/user/.config/myapp.yaml", O_RDONLY) = 5

Now you know where the program is looking and where it actually found something.

“Why is this program failing with a permission error?”

1

strace -e trace=file -s 256 myapp 2>&1 | grep -E "EACCES|EPERM"

openat(AT_FDCWD, "/var/run/myapp.pid", O_WRONLY|O_CREAT, 0644) = -1 EACCES (Permission denied)

Instant answer: it’s trying to write /var/run/myapp.pid and doesn’t have permission.

“What network connections is this program making?”

1

strace -e trace=network -s 256 -f myapp 2>&1 | grep connect

connect(4, {sa_family=AF_INET, sin_port=htons(443), sin_addr=inet_addr("93.184.216.34")}, 16) = 0
connect(5, {sa_family=AF_INET, sin_port=htons(6379), sin_addr=inet_addr("10.0.1.5")}, 16) = 0

Shows exactly which IPs and ports the process is connecting to.

“Why does this program crash intermittently?”

Run it under strace, saving full output, and look at the syscalls just before the crash:

1
2
3

strace -f -o /tmp/crash-trace.txt -s 512 myapp
# Wait for crash, then:
tail -50 /tmp/crash-trace.txt

Look for the signal that killed it:

--- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, si_addr=0x10} ---
+++ killed by SIGSEGV (core dumped) +++

si_addr=0x10 (a near-null pointer) tells you it’s a null pointer dereference.

“Is this program really exiting cleanly?”

1
2
3
4

strace -e trace=process myapp
# ...
# exit_group(0)           = ?   (clean exit with code 0)
# exit_group(1)           = ?   (exit with error code 1)

“What is this binary doing? I don’t have source.”

1
2
3

# Trace a mystery binary — follow forks, full strings, save to file
strace -f -s 512 -o /tmp/mystery.txt ./mystery-binary
cat /tmp/mystery.txt | grep -E "open|exec|connect|write" | head -40

“Why is my container failing to start?”

1
2

docker run --cap-add=SYS_PTRACE --security-opt seccomp=unconfined \
  myimage strace -f -e trace=file,process /app/start.sh

Profiling with -c — Finding Slow Syscalls

The -c summary is excellent for finding performance bottlenecks:

1
2

# Profile a slow command
strace -c -f myapp

Common findings:

• High futex time — thread contention, lock congestion
• High read/write count with tiny sizes — unbuffered I/O, should be larger reads
• High stat/lstat calls — excessive filesystem metadata queries (common in build tools)
• High poll/epoll_wait time — waiting on I/O (normal for servers, investigate if unexpected)
• Many mmap/munmap calls — memory allocation churn, possible allocator issue

1
2

# Find the 10 most-called syscalls
strace -c myapp 2>&1 | sort -k5 -rn | head -10

ltrace — Library Call Tracing

ltrace intercepts calls to shared library functions. It operates at a higher level than strace — you see malloc, fopen, strcmp instead of brk, openat, read.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

# Basic ltrace
ltrace ls /tmp

# Follow forks
ltrace -f myapp

# Filter by library function
ltrace -e malloc,free,realloc myapp

# Show only calls to functions matching a pattern
ltrace -e 'str*' myapp   # all str* functions (strcmp, strcpy, strlen...)

# Time each call
ltrace -T myapp

# Save to file
ltrace -o /tmp/ltrace.txt myapp

Sample ltrace Output

fopen("/etc/hosts", "r")                          = 0x55a3b2c1d020
fgets(0x7ffd2341a0e0, 1024, 0x55a3b2c1d020)      = 0x7ffd2341a0e0
strcmp("127.0.0.1", "10.0.0.1")                   = 1
fgets(0x7ffd2341a0e0, 1024, 0x55a3b2c1d020)      = 0x7ffd2341a0e0
strcmp("::1", "10.0.0.1")                         = -1
fclose(0x55a3b2c1d020)                             = 0

You can see the program opening /etc/hosts, reading lines, and comparing them — much more readable than the equivalent strace output.

When to Use ltrace vs strace

Use strace when:

• You need to see exactly what the kernel is doing
• You’re debugging permission errors, file not found, network issues
• The program is statically linked
• You need reliability (ltrace has more edge cases)

Use ltrace when:

• You want higher-level function names
• You’re debugging library interactions
• You want to see string operations, memory allocations
• strace output is too low-level to be useful

ltrace Limitations

• Doesn’t work well with statically linked binaries (no shared library calls to intercept)
• Can miss calls that go directly to syscalls via inline assembly
• More fragile than strace with some complex programs
• Not available in all container environments

Advanced Techniques

Tracing Without Stopping a Process (-p with signals)

When you attach with -p, the process is temporarily stopped (SIGSTOP) and then resumed. For latency-sensitive production processes, this pause matters. Consider:

1
2

# Trace for exactly 10 seconds then detach
timeout 10 strace -p 1234 -o /tmp/trace.txt; kill -0 1234 2>/dev/null && echo "Process still running"

Filtering Complex Expressions

1
2
3
4
5
6
7
8

# Trace only successful opens (return value > 0)
strace -e trace=openat myapp 2>&1 | grep -v "= -1"

# Trace only failed syscalls
strace -Z myapp   # -Z = only show failing syscalls (strace 5.2+)

# Equivalent for older strace
strace myapp 2>&1 | grep "= -[0-9]"

Counting Unique File Paths Accessed

1
2
3

strace -e trace=file -s 512 myapp 2>&1 \
  | grep -oP '(?<=openat\(AT_FDCWD, ")[^"]+' \
  | sort -u

Watch for a Specific Event and Alert

1
2
3
4
5
6
7

# Alert when a process accesses a sensitive file
strace -p 1234 -e trace=file -s 256 2>&1 | while IFS= read -r line; do
    if echo "$line" | grep -q "/etc/shadow"; then
        echo "ALERT: Process accessed /etc/shadow!" | mail -s "Security Alert" ops@example.com
        echo "$line"
    fi
done

Reconstructing What a Program Wrote

1
2
3
4
5

# Capture all write syscalls and decode the data
strace -e trace=write -s 4096 myapp 2>&1 \
  | grep 'write(' \
  | sed 's/write([0-9]*, "//; s/", [0-9]*).*//' \
  | sed 's/\\n/\n/g'

Tracing exec Chains (What Does This Script Actually Run?)

1
2
3
4

# See every program executed by a shell script
strace -f -e trace=execve -s 256 ./deploy.sh 2>&1 \
  | grep execve \
  | grep -v "= -1"

This is invaluable for auditing what CI scripts actually run.

Comparing Two Runs

1
2
3

strace -o /tmp/run1.txt myapp arg1
strace -o /tmp/run2.txt myapp arg2
diff /tmp/run1.txt /tmp/run2.txt

Key Syscalls to Know

You’ll see these constantly. Knowing what they do makes output readable at a glance.

File I/O

Process

Network

Memory

Synchronization

errno Values Quick Reference

These appear constantly in error returns:

Production Safety Notes

• Performance impact: strace adds measurable overhead — typically 2–20x slowdown due to the stop/inspect/resume cycle for every syscall. Don’t leave it running in production without a timeout.
• Use -c for profiling in production: The summary mode (-c) has lower overhead than per-syscall output because there’s no formatting cost per call.
• Detach carefully: When you Ctrl+C out of strace -p, the target process is resumed and continues normally. The trace stops cleanly.
• Sensitive data in output: strace will capture passwords, tokens, and keys if the program writes them to files or network sockets. Handle trace files as sensitive material.
• Containers: Most container runtimes restrict ptrace by default. You need --cap-add=SYS_PTRACE or a permissive seccomp profile. On Kubernetes, add SYS_PTRACE to the pod’s security context.

Practical Cheat Sheet

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41

# Trace a new command, filtered to file operations
strace -e trace=file -s 256 COMMAND

# Attach to running process, network only
strace -p PID -e trace=network -s 256

# Follow forks, save to file
strace -f -o /tmp/trace.txt COMMAND

# Per-syscall timing
strace -T -e trace=file COMMAND

# Summary table
strace -c COMMAND

# Only failed syscalls (strace 5.2+)
strace -Z COMMAND

# Decode fd numbers to paths
strace -y -e trace=desc COMMAND

# Filter to a specific file path
strace -P /etc/myapp.conf COMMAND

# One trace file per process
strace -ff -o /tmp/trace COMMAND

# ltrace — library calls
ltrace -f -e 'str*+mem*' COMMAND

# Find every file a program tries to open
strace -e trace=openat -s 512 COMMAND 2>&1 | grep openat

# Find permission errors
strace -e trace=file COMMAND 2>&1 | grep -E "EACCES|EPERM"

# Show network connections
strace -e trace=network -s 256 COMMAND 2>&1 | grep connect

# Trace what a script executes
strace -f -e trace=execve COMMAND 2>&1 | grep -v "= -1"

strace is one of those tools where the first real debugging win — “oh, it’s looking for the config file in /etc/app/ but I put it in /etc/myapp/” — makes everything click. From there it becomes instinctive: something isn’t working, reach for strace, know in thirty seconds what’s happening.