systemd Deep Dive
systemd is everywhere. On Debian 12 (Bookworm) it ships as version 252. On Ubuntu 24.04 LTS (Noble Numbat) it is version 255.4. On Fedora 40 and later it is version 256 or higher. If you run Linux in production — on bare metal, in VMs, inside containers that use it — you are using systemd, and the gap between “I can start and stop services” and “I actually understand how this works” is larger than most people admit.
This post is the reference I wish I had had when I first moved off SysV init. It covers the full surface area: unit file anatomy for all major unit types, the dependency and ordering model, socket activation, the journald log store, boot performance analysis, drop-in overrides, and user-mode instances. Code examples are real and production-oriented. The sharp edges are named, not glossed over.
Why systemd Replaced SysV Init
To understand systemd you need to understand what it replaced and why that replacement was contentious but ultimately necessary.
SysV init’s model was simple to the point of being primitive. PID 1 ran /etc/inittab, which launched shell scripts in sequence. Each runlevel had a directory of symlinks (/etc/rc3.d/S20foo, S30bar, etc.) and the scripts in that directory ran one at a time, in lexicographic order. To express “start nginx after mysql” you encoded it in the numeric prefix: S20mysql, S30nginx. If MySQL took forty seconds to start, nginx waited forty seconds. Every service waited for every service before it.
On a modern server with a dozen application services, a handful of databases, network setup, mount points, and device initialization, this serial model produced boot times measured in minutes. The S## numeric namespace was a global coordination problem — third-party packages fought over numbers. Error handling was whatever the shell script author decided to put in, which was often nothing. Log output went wherever the script sent it, which was often /dev/null. Monitoring a running service meant grepping for a PID file that may or may not have been written correctly.
Upstart, Ubuntu’s pre-systemd init, introduced an event-driven model that improved parallelism. Services declared what events they waited for (start on filesystem and net-device-up IFACE=lo) and upstart fired them when conditions were met. This was better, but the event model was hard to reason about for complex dependency chains and upstart never achieved the full parallelism that systemd would.
systemd, first released by Lennart Poettering and Kay Sievers in 2010, took a different approach. It modeled the entire system as a directed acyclic graph of units with explicit dependency and ordering edges. The dependency graph is resolved at startup, and everything that can run in parallel does run in parallel. A unit that is blocked waiting for a dependency burns no CPU and holds no locks — it simply does not start until its dependencies are satisfied.
More importantly, systemd’s scope extended far beyond process management. PID 1 in a systemd system manages:
- Services (daemon processes)
- Sockets (with activation — more on this later)
- Timers (cron replacement)
- Mount points and swap (replacing
/etc/fstabfor dynamic mounts) - Device management (via udev integration)
- Target units (synchronization points replacing runlevels)
- Scope and slice units (cgroup management for resource allocation)
- Login sessions (via logind)
- Network interfaces (via networkd, on systems that use it)
- Temporary file cleanup (systemd-tmpfiles)
- System time synchronization (systemd-timesyncd)
- Name resolution (systemd-resolved)
- Boot-time disk checks and encryption (systemd-fsck, systemd-cryptsetup)
This scope is also the source of much of the criticism. systemd does a lot, its binary is large, and its design choices sometimes prioritize operational uniformity over Unix philosophy composability. These are legitimate concerns. But for managing a production Linux system at scale, the uniformity is a feature: every service has the same interface for start/stop/status/logs, every service is cgroup-tracked so you can attribute resource usage, and every service failure is captured in the journal with structured metadata.
Unit File Anatomy
Everything in systemd is a unit. Units are described by plain text INI-format configuration files with a .service, .timer, .socket, .target, .mount, .slice, or other extension. Understanding the common structure across all unit types is the foundation.
Where Unit Files Live
systemd searches for unit files in several directories in a defined precedence order. Higher in the list wins:
/etc/systemd/system/ <- admin-managed, highest precedence
/run/systemd/system/ <- runtime-generated (e.g., by generators)
/usr/local/lib/systemd/system/ <- local admin installs
/lib/systemd/system/ <- vendor/distro units (package-provided)
/usr/lib/systemd/system/ <- same as above on some distros
The critical rule: never edit files in /lib/systemd/system/ or /usr/lib/systemd/system/. Package upgrades overwrite them. If you need to modify a vendor unit, place a drop-in override in /etc/systemd/system/<unit>.d/ or create a full replacement in /etc/systemd/system/. This is covered in depth in the drop-ins section.
The [Unit] Section
The [Unit] section is common to every unit type. It carries descriptive metadata and the dependency and ordering declarations.
|
|
Key directives:
Description — Human-readable name shown in systemctl status and the journal. Be specific: “PostgreSQL Database Server” is more useful than “Database”.
Documentation — Space-separated list of URIs and man page references. systemctl help <unit> opens them.
Wants — Soft dependency. systemd will attempt to start the listed units alongside this one, but if they fail or are not found, this unit starts anyway.
Requires — Hard dependency. If the required unit fails to start or stops, this unit is stopped too.
BindsTo — Stricter than Requires. If the bound-to unit stops for any reason (including a requested stop), this unit stops immediately. Use this for units that are meaningless without their dependency — for example, a service bound to a network interface that disappears when a VPN drops.
PartOf — One-directional propagation. Stop and restart operations on the listed unit propagate to this unit, but not vice versa. Used to group related services without creating a hard start dependency.
After / Before — Pure ordering, no dependency. After=foo.service means this unit starts after foo.service if both are being started. It does not cause foo.service to start. This is a common source of confusion and is covered in detail in the dependency section.
Conflicts — Mutual exclusion. Starting this unit stops the conflicting unit, and vice versa. The canonical example is Conflicts=rescue.service in many production units.
Condition directives* — These are checked before starting the unit. If the condition is false, the unit is skipped (not failed — it enters the “condition” state and systemd considers this a success). Common ones:
|
|
Assert directives* — Like Condition*, but failure causes the unit to enter a hard failed state rather than silently skip. Use Assert* when a missing condition indicates a genuine misconfiguration.
The [Install] Section
The [Install] section is not processed at runtime — it is read only by systemctl enable and systemctl disable. Understanding this distinction is important.
|
|
When you run systemctl enable myapp.service, systemd reads the [Install] section and creates symlinks in the target’s .wants or .requires directory:
/etc/systemd/system/multi-user.target.wants/myapp.service
-> /lib/systemd/system/myapp.service
That symlink is how a unit survives reboots. systemctl disable removes it. A unit without an [Install] section can be started manually but cannot be enabled.
WantedBy — Creates a .wants/ symlink in the target directory. This is a Wants= relationship from the target to this unit. The vast majority of services use WantedBy=multi-user.target.
RequiredBy — Creates a .requires/ symlink. Use this only when the target genuinely cannot function without this unit.
Alias — Additional names under which this unit can be addressed. A symlink is created pointing to the real unit file.
Also — When this unit is enabled, also enable the listed units. Useful for socket-activated services where you want systemctl enable myapp.service to also enable myapp.socket.
An Annotated Minimal Unit File
|
|
Service Units
Service units are the most common unit type. The [Service] section has dozens of directives. Here are the ones you will actually use.
Service Type
The Type= directive tells systemd how to determine when a service has finished starting. Getting this wrong leads to race conditions and dependency problems.
| Type | Semantics | Use When |
|---|---|---|
simple |
Process starts, systemd considers it ready immediately | Default; for processes that don’t fork and don’t notify |
exec |
Like simple but waits for execve() to complete before marking ready |
Preferred over simple for most modern services (available since systemd 240) |
forking |
Service forks; parent exits; child is the daemon | Legacy daemons that double-fork (Apache httpd with Daemonize on) |
oneshot |
Process exits after completing work; RemainAfterExit=yes keeps unit “active” |
Init scripts, one-time setup tasks |
notify |
Service calls sd_notify(READY=1) when ready |
Services that properly integrate with systemd notifications |
notify-reload |
Like notify, but also uses sd_notify for reload confirmation | Available since systemd 253; for reload completion signaling |
dbus |
Service acquires a D-Bus name; ready when name appears on bus | D-Bus services |
idle |
Startup delayed until job queue settles | Terminal services that should not interleave output with boot messages |
For new services you control the source code of, use notify if you call sd_notify(), otherwise use exec. For anything that double-forks, use forking and set PIDFile= so systemd can track the actual daemon PID.
Exec Directives
|
|
All paths must be absolute. This is not a limitation — it is intentional. systemd does not set $PATH in the service environment by default, and relative paths would be ambiguous given that WorkingDirectory can be set to anything. If you need shell expansion, prefix with a shell explicitly:
|
|
Prefixing an exec directive with - suppresses errors from that command:
|
|
The - means “if this command fails, that is fine, continue anyway.”
Prefixing with + runs the command with full root privileges even if User= is set. Use sparingly.
Restart Policy
|
|
| Restart= value | Restarts on clean exit | Restarts on failure | Restarts on watchdog timeout |
|---|---|---|---|
no |
No | No | No |
on-success |
Yes | No | No |
on-failure |
No | Yes | Yes |
on-abnormal |
No | No | Yes |
on-abort |
No | Abort only | No |
always |
Yes | Yes | Yes |
on-failure is the right default for production services. always is sometimes used for intentionally short-lived workers that should keep restarting indefinitely.
StartLimitBurst and StartLimitIntervalSec define a rate limiter. If a service restarts more than StartLimitBurst times in StartLimitIntervalSec, systemd stops trying and marks the unit as failed. A service in this state requires systemctl reset-failed myapp before it can be started again. This prevents a crash-looping service from hammering the system. Tune these to be generous for services that legitimately take time to recover after an infrastructure failure.
Environment and Configuration
|
|
The - prefix on EnvironmentFile= makes the file optional. This is useful for environment-specific overrides that may not exist on every host.
The /etc/default/ convention (Debian) or /etc/sysconfig/ (Red Hat) stores shell-format KEY=VALUE pairs that the service picks up. Note that this is not a sourced shell script — shell syntax like export or command substitution will not work. Just KEY=VALUE pairs.
Security Sandboxing
This is where systemd genuinely shines for service hardening. The [Service] section has a rich set of directives that build a privilege-reduced execution environment without requiring custom namespacing code in the service itself.
|
|
The systemd-analyze security myapp.service command produces an exposure score from 0 (fully locked down) to 10 (completely exposed). Running it against a default vendor unit for something like nginx and then iteratively adding directives to reduce the score is a straightforward hardening workflow.
One sharp edge: ProtectSystem=strict makes most of the filesystem read-only from the service’s perspective, including /etc. If your service needs to write config state to /etc, you will need ReadWritePaths=/etc/myapp. More commonly, this is a signal that the service should write to /var/lib/ instead.
Standard Output and Logging
|
|
By default both stdout and stderr go to the journal. This means printf and console.log and fmt.Println all end up in journalctl -u myapp without any configuration in the application. This is a significant operational convenience.
Other options:
|
|
The file: and append: options are useful for services that need traditional log files alongside journal storage, or for services whose log output is processed by external tools that cannot read from the journal API.
Timeouts
|
|
TimeoutStartSec is the time systemd waits for a service with Type=notify to send READY=1. If the service does not check in within this window, systemd kills it and marks it failed. For services that do slow initialization (database migrations, large cache warmups), this needs to be set appropriately. The default is 90 seconds.
TimeoutStopSec is how long systemd waits between sending SIGTERM and SIGKILL. The default is 90 seconds. For services with in-flight requests, set this to your p99 request duration plus a buffer.
A Complete Production Service Unit
|
|
After creating this file:
|
|
Timer Units
systemd timers are the modern replacement for cron. Every argument in favor of cron — familiarity, simplicity, universality — is true. But timers have advantages that matter in production.
Logging: Every timer invocation shows up in the journal with the full output of the triggered service. With cron, you configure a MAILTO or redirect output yourself, and missed runs are often silent.
Dependency system: A timer unit can express After= and Requires= just like any other unit. You can make a backup job wait for the network to be up, or require a mount point to be present.
Catch-up with Persistent=yes: If the system was offline when a timer was scheduled to fire, with Persistent=yes it will run immediately on next boot. cron silently skips missed runs.
Monotonic vs. realtime: cron is wall-clock based. systemd timers support both wall-clock (OnCalendar=) and relative-time (OnBootSec=, OnUnitActiveSec=) triggers. “Run 15 minutes after each boot” is trivial in systemd and awkward in cron.
Jitter: RandomizedDelaySec= adds a random delay to spread load when many systems run the same timer simultaneously.
The [Timer] Section
A timer unit must be named to match the service it triggers, or must explicitly declare Unit= to name a different service.
backup.timer automatically triggers backup.service. If you want to call the timer something different from the service, add Unit=backup.service to the [Timer] section.
|
|
The systemd.time format for OnCalendar= takes some getting used to but is more expressive than cron’s five-field syntax. The structure is DayOfWeek Year-Month-Day Hour:Minute:Second. Ranges use .., lists use ,, and * matches any value.
*-*-* 04:00:00 # every day at 04:00 (equivalent to cron: 0 4 * * *)
*-*-01 00:00:00 # first of every month at midnight
Mon *-*-* 00:00:00 # every Monday at midnight
*-*-* 00/6:00:00 # every 6 hours (00:00, 06:00, 12:00, 18:00)
Use systemd-analyze calendar '*-*-* 02:30:00' to verify a calendar expression and see the next scheduled times.
Timer + Service for a Daily Database Backup
|
|
|
|
Enable the timer (not the service — the timer manages the service):
|
|
systemctl list-timers shows all active timers with their last trigger time and next scheduled run:
NEXT LEFT LAST PASSED UNIT ACTIVATES
Sun 2026-06-01 02:37:44 UTC 9h left Sat 2026-05-31 02:32:01 UTC 14h ago db-backup.timer db-backup.service
The “PASSED” column showing 14 hours ago confirms the last run, and “LEFT” shows time until the next. If a timer shows “n/a” in LAST, it has never fired — worth investigating.
Socket Units and Socket Activation
Socket activation is one of systemd’s most powerful and least understood features. The concept is straightforward: instead of a service binding a port at startup and sitting idle waiting for connections, systemd binds and holds the port. When a connection arrives, systemd starts the service and hands it the pre-bound socket. Until that first connection, the service does not exist.
This has several benefits that compound in practice. First, services can start in parallel during boot even if they depend on each other’s sockets — systemd holds all the sockets open, so the first connection to service A might arrive while service B is still initializing, but both will be ready by the time a real workload arrives. Second, a service that crashes and is restarting does not cause connection failures to the client — systemd holds the socket and queues incoming connections during the restart window. Third, infrequently-used services consume no memory when idle.
How Socket Activation Works
When a service is started via socket activation, systemd passes the pre-opened socket file descriptors to the service process using a well-known convention. The number of passed file descriptors is in the LISTEN_FDS environment variable. The file descriptors themselves start at fd 3 (after stdin/stdout/stderr). The LISTEN_PID environment variable contains the PID to validate that the vars were not inherited from a parent.
LISTEN_FDS=1
LISTEN_PID=12345
The service reads these variables and uses the passed file descriptor instead of calling socket() + bind() + listen() itself. Most languages have libraries that handle this: sd_listen_fds() in C (from libsystemd), github.com/coreos/go-systemd/activation in Go, the sd-notify crate in Rust, and similar for Python and Node.js.
Socket Activation Flow:
systemd PID 1
+-----------------------------------------------------+
| |
| 1. On boot, systemd reads myapp.socket |
| and calls socket() + bind(:8080) + listen() |
| systemd holds the fd open |
| |
| 2. Connection arrives on :8080 |
| | |
| 3. systemd forks myapp.service process |
| passes fd via LISTEN_FDS=1, LISTEN_PID=<pid> |
| | |
| 4. myapp calls sd_listen_fds() |
| gets back fd 3 (already bound :8080) |
| calls accept() on fd 3 normally |
| | |
| 5. myapp handles the connection |
| |
| If myapp crashes: |
| systemd still holds the socket |
| incoming connections are queued (up to backlog) |
| systemd restarts myapp and hands fd back |
| |
+-----------------------------------------------------+
The [Socket] Section
|
|
The Accept=yes vs Accept=no distinction matters. With Accept=no (the default), systemd activates one instance of the service and passes it the listening socket. The service calls accept() itself and manages multiple connections. With Accept=yes, systemd calls accept() for each incoming connection and spawns a separate service instance for each one — instantiated services are named myapp@0.service, myapp@1.service, etc. The Accept=yes model is similar to inetd and is appropriate for stateless per-connection services.
Socket-Activated HTTP Service Example
|
|
|
|
Enable with:
|
|
One sharp edge: if your service does not implement sd_listen_fds() and tries to bind the same port itself, it will get EADDRINUSE because systemd already bound it. The service must be modified to support socket activation, or you must use Accept=yes (where systemd handles the accept loop) or simply not use socket activation for that service.
Target Units and the Boot Sequence
Targets are synchronization points. They group units and represent system states. They replaced SysV runlevels but the mapping is not one-to-one because targets are composable and multiple targets can be active simultaneously.
The Boot Sequence
systemd starts (PID 1)
|
v
default.target (alias -> graphical.target or multi-user.target)
|
+-- local-fs-pre.target
| +-- systemd-tmpfiles-setup-dev.service
|
+-- sysinit.target
| +-- systemd-udevd.service
| +-- systemd-journald.service
| +-- cryptsetup.target
| +-- local-fs.target
| | +-- mount units from /etc/fstab
| +-- swap.target
|
+-- basic.target
| +-- sockets.target <- all .socket units activate here
| +-- paths.target
| +-- timers.target
|
+-- network.target <- network interfaces exist (not necessarily up)
|
+-- network-online.target <- network is configured and routable
|
+-- multi-user.target <- normal multi-user text console
+-- graphical.target <- if display manager is present
The Critical network.target vs. network-online.target Distinction
This trips up almost everyone at some point. The names imply a graduated progression but the semantics are very different.
network.target is a passive target. It is “reached” when the network subsystem has started — meaning NetworkManager or networkd is running and has begun configuring interfaces. It does not mean any interface is up, any address is assigned, or any route is reachable. Starting your application After=network.target is almost certainly wrong if your application makes outbound TCP connections at startup.
network-online.target is an active target that is held until at least one interface reports it has achieved online status. It blocks the boot sequence while NetworkManager or networkd confirms connectivity. This is what most application services should use.
The trade-off: network-online.target delays boot. On a system with a slow DHCP server or a VLAN that takes time to come up, this can add 30 or more seconds to boot. For systems where fast boot is critical, some operators use After=network.target with application-level retry logic instead.
|
|
Key Targets Reference
| Target | Purpose | Notes |
|---|---|---|
sysinit.target |
Kernel + early system initialization | udev, journal, crypto, mounts |
basic.target |
Basic system facilities | Sockets, paths, timers ready |
network.target |
Network stack started | Interfaces may not be configured |
network-online.target |
Network is routable | Delays boot; use for apps that need connectivity |
multi-user.target |
Full text-mode multi-user system | Traditional runlevel 3 equivalent |
graphical.target |
Multi-user with display manager | Traditional runlevel 5 equivalent |
rescue.target |
Single-user recovery mode | Minimal services, root shell |
emergency.target |
Emergency shell | Only systemd and shell, no mounts |
sleep.target |
System is suspending | |
reboot.target |
System is rebooting | |
shutdown.target |
System is shutting down |
Custom Targets
Targets can be created to group related services. A myapp-stack.target that pulls in a web app, its worker process, and its Redis cache is a clean way to manage multi-component applications:
|
|
|
|
isolate is dangerous in production — it stops all units not required by the target. It is the systemd equivalent of telinit 1.
Dependency Ordering in Depth
The dependency model is the most conceptually important part of systemd. It is also where most misconfigurations occur.
Ordering Is Not Dependency
This distinction cannot be overstated: After= and Before= control only the order in which units start. They do not cause units to be started.
If your unit file says:
|
|
and nothing in the system would cause postgresql.service to start, then After= does nothing. Your service will start (if triggered by something else) and postgresql.service will remain stopped. The ordering only applies when both units are being started as part of the same transaction.
To express “I need postgresql running before I start, and if postgresql stops I stop too”, you need both:
|
|
The Requires= causes systemd to add postgresql.service to the transaction when your service is started. The After= then controls the order within that transaction.
The Four Dependency Types
Dependency Relationships:
Wants=B
A ---- B Start B when A starts. If B fails, A is fine.
Requires=B
A ==== B Start B when A starts. If B stops, A stops.
BindsTo=B
A #### B Like Requires, but also: if B stops for ANY reason
(including planned stop), A stops immediately.
PartOf=B
A - - B Restart/stop B -> restart/stop A.
Starting B does NOT propagate to A.
After=B Controls ordering within a transaction only.
Without Wants/Requires, After= does nothing on its own.
A Real Dependency Graph
Consider a web application with a database, a cache, and a message queue:
network-online.target
|
| Wants/After (from each service)
|
+------+---------------------------+
| | |
v v v
postgresql. redis.service rabbitmq.service
service
| | |
| Requires | Wants | Wants
| After | After | After
| | |
+------+-------+ |
| |
v |
mywebapp.service <----------------+
|
| PartOf
v
myworker.service
In this graph, mywebapp.service has Requires=postgresql.service because it cannot function without a database. It has Wants=redis.service because it can handle cache misses. It has Wants=rabbitmq.service because job queuing is a soft dependency. myworker.service has PartOf=mywebapp.service so that restarting the web app also restarts the worker, but the worker can be stopped independently.
|
|
Conflicts
Conflicts= creates mutual exclusion. When a unit with Conflicts=B starts, B is stopped, and vice versa. This is used in the boot sequence for targets that cannot coexist:
|
|
It is also occasionally useful for application-level exclusion — for instance, ensuring that a maintenance mode service and a production service cannot run simultaneously.
Drop-In Overrides
The drop-in system is how you modify the behavior of vendor-provided unit files without touching the files that package managers own. It deserves a full section because getting this wrong causes subtle bugs after package upgrades.
The Immutable Vendor Unit
Units in /lib/systemd/system/ (or /usr/lib/systemd/system/) are owned by packages. The package manager writes them on install and overwrites them on upgrade. Any change you make directly to these files will be silently reverted the next time apt upgrade runs. This is not a bug — it is the intended design.
The correct approach is a drop-in: a small .conf file in a .d/ directory that contains only the directives you want to change. systemd merges the base unit with all drop-ins at load time.
Creating Drop-Ins
|
|
After editing, systemd automatically runs daemon-reload. If you create the files manually, you must run:
|
|
Drop-In Syntax
Drop-in files must begin with the section header for the directive they override. They cannot omit the section:
|
|
The pattern of setting a directive empty first (ExecStart=) then setting the new value (ExecStart=/new/path) is important. Many directives are additive — setting them a second time adds to a list rather than replacing. Setting them empty first clears the accumulated list.
Common Drop-In Use Cases
Add a dependency the vendor unit does not know about:
|
|
Add memory limits to a service that predates cgroup v2 resource management:
|
|
Add environment variables to a package-managed service:
|
|
Override a service that defaults to Type=forking now that upstream added sd_notify support:
|
|
Inspecting the Merged Result
|
|
systemctl cat is invaluable for debugging. It shows exactly what systemd sees, including comments indicating which file each block came from.
Drop-in files are processed in lexicographic order within the .d/ directory. Naming conventions like 10-resource-limits.conf and 99-local-overrides.conf make the application order explicit at a glance.
journald and Log Management
systemd-journald is the logging daemon. It collects structured log entries from all services running under systemd, from the kernel, and from the initrd. The entries are stored in a binary format in /run/log/journal/ (volatile) or /var/log/journal/ (persistent).
Structured vs. Plain Text
The journal is not a collection of flat text files. Each entry is a structured record with mandatory fields (_SYSTEMD_UNIT, _PID, _UID, _COMM, _HOSTNAME, PRIORITY, MESSAGE, __REALTIME_TIMESTAMP) plus any custom fields the service writes via the journal API (sd_journal_send() or language-specific bindings). This structure enables filtering that would require grep | awk | sed pipelines against plain log files.
Storage Configuration
The storage mode is configured in /etc/systemd/journald.conf:
|
|
| Storage= | Behavior |
|---|---|
volatile |
Store only in memory (/run/log/journal/). Lost on reboot. |
persistent |
Store on disk (/var/log/journal/). Survives reboots. |
auto |
Persistent if /var/log/journal/ exists, otherwise volatile. Default. |
none |
Discard all logs immediately. Only forward to syslog. |
The default auto means logs are volatile on a fresh install. To make logs persistent:
|
|
The journal directory structure:
/var/log/journal/
+-- <machine-id>/ <- one directory per machine ID
+-- system.journal <- system/kernel messages
+-- system@<id>.journal <- rotated system journal files
+-- user-1000.journal <- per-user journal for UID 1000
+-- user-1000@<id>.journal
Size and Retention Configuration
|
|
The rate limiting defaults (RateLimitBurst=10000 in 30 seconds per service) are generous but finite. A service that enters a tight error loop can hit the limit and have subsequent log messages dropped, which is exactly when you most need logs. For verbose services, increase the limit with a drop-in:
|
|
journalctl Quick Reference
| Command | Purpose |
|---|---|
journalctl |
All journal entries, oldest first |
journalctl -r |
Reverse chronological (newest first) |
journalctl -f |
Follow new entries in real time |
journalctl -e |
Jump to end of journal |
journalctl -n 100 |
Show last 100 lines |
journalctl -u nginx.service |
Filter by unit |
journalctl -u nginx -u php-fpm |
Filter by multiple units |
journalctl _PID=12345 |
Filter by PID |
journalctl _UID=1000 |
Filter by user ID |
journalctl -p err |
Priority: emerg/alert/crit/err/warning/notice/info/debug |
journalctl -p warning..err |
Priority range |
journalctl -b |
Current boot only |
journalctl -b -1 |
Previous boot |
journalctl -b -2 |
Two boots ago |
journalctl --list-boots |
List all recorded boots with timestamps |
journalctl -k |
Kernel messages only (like dmesg) |
journalctl -t myapp |
Filter by syslog identifier |
journalctl --since "2026-05-31 08:00:00" |
Time range start |
journalctl --until "2026-05-31 09:00:00" |
Time range end |
journalctl --since "1 hour ago" |
Relative time |
journalctl -o json-pretty |
Structured JSON output |
journalctl -o cat |
Message text only, no metadata |
journalctl -o short-precise |
Microsecond timestamps |
journalctl --disk-usage |
Total journal disk usage |
journalctl --vacuum-size=500M |
Delete old files until under 500MB |
journalctl --vacuum-time=7d |
Delete entries older than 7 days |
journalctl --vacuum-files=10 |
Keep only 10 most recent journal files |
journalctl --verify |
Verify journal file integrity |
Some practical compound queries:
|
|
The -o json output mode is worth knowing for production work. When you need to ship logs to an external system (Elasticsearch, Loki, Splunk), parse structured fields, or do statistical analysis, the full structured journal record is far richer than what a plain text formatter shows:
|
|
This reveals fields like _SYSTEMD_CGROUP, _TRANSPORT, _COMM, _CMDLINE, and any custom fields your application wrote — none of which appear in the default text output.
systemd-analyze and Boot Performance
Boot performance optimization starts with measurement. systemd-analyze provides all the tools needed to understand where time is being spent.
Total Boot Time
|
|
The output breaks down time into four phases: firmware (UEFI POST), bootloader (GRUB/systemd-boot), kernel initialization, and userspace. For servers you generally cannot control the firmware phase. Kernel initialization is rarely the bottleneck. Userspace is where systemd lives and where most optimization work happens.
Finding Slow Units
|
|
systemd-analyze blame lists units sorted by activation time. The top offenders are usually obvious. NetworkManager-wait-online.service is the most common culprit — it waits for the network to come online and blocks everything that depends on network-online.target. On cloud instances with fast DHCP, it can be disabled if your services handle connectivity with retries:
|
|
Do this carefully. If your services genuinely need connectivity at startup and have no retry logic, disabling this will cause startup failures.
The Critical Chain
|
|
The critical chain is the path from the first started unit to the last unit needed to reach the default target. It shows exactly which dependency chain is determining the total boot time. This is far more actionable than blame alone, which does not show you whether a slow unit is on the critical path.
Generating a Visual Timeline
|
|
The SVG is a Gantt chart of the entire boot sequence with each unit as a bar, colored by type, with lines showing dependencies. For complex systems this is invaluable — you can immediately see which units started in parallel vs. which were serialized by dependencies.
Static Analysis
|
|
The security subcommand produces output like:
NAME DESCRIPTION EXPOSURE
x PrivateNetwork= Service has access to the host network 0.5
x User=/DynamicUser= Service runs as root user 0.4
x SystemCallFilter= System call allow list not defined 0.3
...
-> Overall exposure level for mywebapp.service: 4.7 MEDIUM
Each row shows a sandboxing directive and the exposure contribution of not having it. This is an excellent guide for iterative hardening. Start with a raw score and reduce it by adding directives that do not break the service’s functionality.
Common Boot Bottlenecks
NetworkManager-wait-online.service — Described above. The most common culprit on cloud VMs. On some systems, systemd-networkd-wait-online.service is the equivalent when using networkd instead of NetworkManager.
systemd-udev-settle.service — Waits for the udev event queue to drain. Often appears in older configurations. Modern systems generally do not need it; it can often be masked if specific device availability is handled by individual mount units or ConditionPathExists= checks.
dev-sdaX.device — If a disk device or LUN takes time to appear (common in SAN environments), its device unit will show up in blame. The fix is usually an appropriate timeout in the device unit or removing unused entries from /etc/fstab.
cloud-init.service — On cloud instances, cloud-init fetches metadata and applies configuration. On subsequent boots, if the data source is slow (IMDSv1 timeouts on AWS), this can be significant. Modern cloud-init versions have shorter timeouts by default, and you can configure the datasource explicitly.
Long TimeoutStartSec values in Type=notify services — If a service is Type=notify and takes a long time to call sd_notify(READY=1), it blocks its dependents. Check with blame and increase TimeoutStartSec if the service legitimately needs more time, or fix the service to call READY=1 sooner (optionally before full initialization, then use EXTEND_TIMEOUT_USEC to request more time).
User-Mode systemd Instances
Every logged-in user on a systemd system gets their own private systemd instance. This systemd --user process is PID 1 of the user’s process hierarchy and manages per-user services with the same unit file model as the system instance.
This is genuinely useful: you can run personal services — a development database, a file syncing daemon, a personal web server — as your own user, with full systemd management (journalctl logs, restart policies, socket activation), without any root access.
User Unit File Locations
~/.config/systemd/user/ <- user's own unit files (highest precedence)
/etc/systemd/user/ <- system-wide user units (admin-managed)
/usr/lib/systemd/user/ <- vendor user units (package-managed)
Managing User Services
|
|
Lingering: Services That Survive Logout
By default, the user’s systemd instance starts on login and stops on logout, taking all user services with it. For always-on services (syncthing, a home server, a background worker), this is wrong.
loginctl enable-linger keeps the user’s systemd instance running even when they are not logged in:
|
|
After enabling linger, the user’s services start at system boot and persist across logout and login cycles.
Syncthing User Service Example
|
|
|
|
Note WantedBy=default.target in the user unit’s [Install] section. default.target is the user analog of multi-user.target — it is the target that the user’s systemd instance reaches at startup.
Environment in User Services
User service environments are more limited than shell environments. Variables set in .bashrc or .profile are not automatically available to systemd user services because those files are sourced by interactive shells, not by systemd.
To pass environment variables to user services:
|
|
Or set them in the unit file directly:
|
|
The %h specifier expands to the user’s home directory. Other useful specifiers in user units: %u for username, %H for hostname, %n for unit name.
Practical systemctl Reference
Service Lifecycle
|
|
The distinction between reload and restart matters operationally. reload sends the reload signal (typically SIGHUP) which causes the service to re-read its configuration files without stopping. In-flight requests are not interrupted. restart stops the process entirely and starts a new one. For nginx, reload causes a graceful configuration reload; restart drops all active connections.
mask is stronger than disable. A disabled unit can still be started manually (systemctl start). A masked unit cannot be started by any means until it is unmasked. Use masking for units you never want running — systemctl mask bluetooth.service on a headless server, or systemctl mask ctrl-alt-del.target to prevent accidental reboots from a keyboard combination.
Inspecting State
|
|
Reading systemctl status output:
* mywebapp.service - My Go Web Application
Loaded: loaded (/etc/systemd/system/mywebapp.service; enabled; preset: enabled)
Drop-In: /etc/systemd/system/mywebapp.service.d
+--override.conf
Active: active (running) since Sat 2026-05-31 08:23:15 UTC; 1h 4min ago
Main PID: 1234 (mywebapp)
Tasks: 12 (limit: 4915)
Memory: 124.3M
CPU: 45.231s
CGroup: /system.slice/mywebapp.service
+--1234 /usr/local/bin/mywebapp --config /etc/mywebapp/config.yaml
May 31 09:15:22 hostname mywebapp[1234]: INFO: handled 1000 requests
May 31 09:22:01 hostname mywebapp[1234]: WARN: slow query detected: 2.3s
The output shows: loaded state (is the file parseable?), whether it is enabled (will it auto-start?), active state (is it running?), any drop-ins applied, the main PID, resource usage from cgroup accounting, and recent journal lines. This is the first thing to check when a service behaves unexpectedly.
Sharp Edges and Things That Bite You
No deep dive into systemd is complete without acknowledging where it genuinely causes pain.
daemon-reload is mandatory and easy to forget. After creating or editing any unit file, you must run systemctl daemon-reload or your changes are not seen. The old unit definition remains in systemd’s memory. Forgetting this and then wondering why your changes have no effect is a rite of passage.
Type=forking with a missing or stale PID file. If you configure Type=forking and the service does not write a PID file at the path specified by PIDFile=, systemd will track the wrong process. The service will appear running in systemctl status while the actual daemon is dead. Use Type=exec or Type=notify for modern services.
After=network.target is almost always wrong. Covered earlier, but worth repeating because it is extremely common in documentation and examples on the internet. network.target does not mean the network is usable. Use After=network-online.target with Wants=network-online.target for services that need connectivity.
ProtectSystem=strict breaks services that write to /etc. Several configuration management approaches and services write state to /etc. Enabling ProtectSystem=strict makes the entire filesystem read-only except for explicitly whitelisted paths. Test this thoroughly before enabling in production.
Cgroup v1 deprecation in systemd 256. systemd 256 (shipped in Fedora 40+, Ubuntu 25.04+) deprecated and by default refuses to boot under cgroup v1. Older container runtimes and some monitoring agents that depend on cgroup v1 paths will break. Verify your container runtime and observability stack support cgroup v2 before upgrading to distributions shipping systemd 256 or later.
Journal field size limits. By default, journald caps individual field values at 48KB and refuses to store messages larger than 8MB. Services that log large binary blobs or very long stack traces will have entries truncated. The defaults surprise people coming from file-based logging.
The systemctl edit empty file trap. When you run systemctl edit and close the editor without saving any content, it creates an empty override.conf file. An empty drop-in can cause unexpected behavior. Always verify with systemctl cat <unit> after editing, and use systemctl revert <unit> to clean up empty drop-ins.
StartLimitBurst and the reset-failed requirement. When a service hits its restart rate limit, it enters failed state and will not restart even if the underlying problem is resolved. The unit stays failed until you run systemctl reset-failed <unit>. This is intentional — it prevents infinite crash loops — but it means automated recovery scripts need to account for this state.
Version Notes: Debian 12 vs Ubuntu 24.04
The two most widely deployed enterprise Linux distributions diverge meaningfully on systemd version.
Debian 12 (Bookworm) ships systemd 252 (currently at the 252.x security patch series). This predates several features discussed in this post: Type=notify-reload was added in 253, ProcSubset=pid became widely stable in 254. On Debian 12, run systemd --version before using newer directives. Unknown directives are silently ignored in most contexts, but systemd-analyze verify will flag them as warnings.
Ubuntu 24.04 LTS (Noble Numbat) ships systemd 255.4. This includes socket activation improvements from 255 — PollLimitBurst= and PollLimitInterval= on socket units for controlling how often polling events trigger, DeferTriggerMaxSec= for deferred activation on conflicting units, and improved credential support. The journalctl --lines=+N syntax for showing the oldest N entries (rather than newest N) is a 255 addition. systemd 256’s run0 tool (a sudo alternative using transient units) arrived in Ubuntu 24.10, not 24.04.
For portable unit files that need to run across both Debian 12 and newer systems, stick to features available since systemd 245 and you will have broad compatibility. The sandboxing directives in this post (NoNewPrivileges, PrivateTmp, ProtectSystem, SystemCallFilter, etc.) have been stable since systemd 230 or earlier and are safe to use everywhere.
Putting It Together
systemd is not a simple tool. It manages hundreds of unit types across multiple subsystems with a dependency model that requires deliberate thought to get right. The learning curve is real and the man pages are dense — though thorough. man systemd.service, man systemd.unit, man systemd.timer, and man systemd.exec are worth reading in full at least once.
What you get in exchange: a uniform operational interface for every process on the system, automatic cgroup-based resource tracking and attribution, structured log storage with rich filtering, a timer system that handles edge cases cron does not, socket activation for resilient restarts, and a security sandboxing model that lets you write least-privilege service configurations without modifying application code.
The practical sequence for managing a new service:
- Write the unit file in
/etc/systemd/system/. Start simple —Type=exec, basicRestart=on-failure, correctAfter=andWants=dependencies. - Run
systemd-analyze verifyon the file to catch syntax errors before loading. - Run
systemctl daemon-reload && systemctl start <unit> && systemctl status <unit>. - Check logs with
journalctl -fu <unit>. - Enable it:
systemctl enable <unit>. - Harden it: run
systemd-analyze security <unit>and add sandboxing directives until the exposure score is acceptable. - Commit the unit file to your configuration management system alongside the application deployment.
Drop-ins for vendor units follow the same sequence, starting with systemctl edit <unit> instead of creating a new file.
For boot performance: systemd-analyze critical-chain first, then blame. Fix the root cause (usually network-online.target or a slow storage device), verify with systemd-analyze plot, and track total userspace time over time.
For logs: make them persistent (mkdir /var/log/journal), set reasonable size limits in journald.conf, and learn the journalctl filtering flags. The combination of journalctl -u <service> -b -p err --since yesterday becomes a reflex for first-pass triage on any service issue.
systemd is the plumbing of every modern Linux system. Understanding it at this level pays dividends every day.
Comments