Linux Security Hardening Baseline
A default Linux installation is a study in competing priorities. The distribution maintainers have to make choices that work across a wide range of hardware, use cases, and skill levels. Security is on their list, but so is “works out of the box on a laptop bought at a retail store.” The result is a system configured for usability, not for running production workloads exposed to the internet.
This is not a criticism — it is a design decision that puts the responsibility for hardening squarely on the operator. The question is what hardening actually means, what threat model it addresses, and which controls are worth the operational cost. That last part is where most hardening guides fail you: they hand you a checklist without explaining what breaks when you apply it.
This post covers a complete Linux security hardening baseline, the kind you would deploy on a production server running a business-critical workload. It covers kernel hardening, filesystem restrictions, SSH configuration, brute-force protection, mandatory access control with AppArmor, syscall filtering with seccomp, syscall auditing with auditd, systemd unit sandboxing, and automated patch management. Each section explains the threat model before showing the configuration.
The baseline targets Debian 12 (OpenSSH 9.2p1) and Ubuntu 24.04 LTS (OpenSSH 9.6p1). Most controls are portable across any systemd-based Linux distribution.
Defense in Depth: The Threat Model
The phrase “defense in depth” gets used so often it has lost meaning. Let’s be concrete about what it means for a Linux server.
An attacker progressing through a typical compromise follows a predictable sequence:
Initial Access Exploit a service, steal credentials, phish an admin
|
v
Establish Persistence Drop a cron job, install a backdoor, add SSH keys
|
v
Privilege Escalation Exploit a kernel bug, abuse a misconfigured setuid binary,
leverage a service running as root
|
v
Lateral Movement Use the compromised host to reach internal systems,
extract credentials from memory or config files
|
v
Objective Exfiltrate data, deploy ransomware, establish C2
Each layer of hardening does not prevent the attacker from completing their objective outright — it makes each transition in that chain harder, noisier, or impossible. A kernel exploit requires knowing addresses in memory; ASLR makes those addresses unpredictable. Privilege escalation via a setuid binary is blocked by NoNewPrivileges. Lateral movement via a compromised service is contained by a seccomp filter that prohibits socket() syscalls. An attacker who does succeed leaves traces that auditd captures.
The defense-in-depth model looks like this across layers:
+--------------------------------------------------+
| Network Layer |
| Firewall, rp_filter, reject ICMP redirects, |
| SYN cookies, fail2ban |
+--------------------------------------------------+
| Host Authentication Layer |
| SSH key-only auth, AllowGroups, MaxAuthTries, |
| unattended-upgrades for CVE patches |
+--------------------------------------------------+
| OS / Kernel Layer |
| ASLR, kptr_restrict, dmesg_restrict, |
| protected symlinks/hardlinks, suid_dumpable |
+--------------------------------------------------+
| Mandatory Access Control Layer |
| AppArmor profiles, seccomp filters, |
| systemd sandboxing directives |
+--------------------------------------------------+
| Audit / Detection Layer |
| auditd syscall rules, auth logging, |
| lynis/OpenSCAP scoring |
+--------------------------------------------------+
No single layer is sufficient. The network layer assumes the host layer can be compromised. The host layer assumes the application can be exploited. The MAС layer assumes the application is already running arbitrary code. The audit layer assumes all of the above and focuses on detection rather than prevention.
CIS Benchmarks are a reasonable starting point for a structured hardening program. The Center for Internet Security publishes scored profiles for most major Linux distributions. Level 1 targets practical security improvements with minimal operational impact. Level 2 is intended for high-security environments and includes controls that will break things — network services, desktop functionality, certain performance-sensitive features. This post aligns roughly with Level 1, with selected Level 2 controls where the operational cost is manageable.
The principle of least privilege applied to the OS level means: every process should have exactly the capabilities it needs to perform its function, access exactly the files it needs to read and write, and be able to make exactly the syscalls its code paths use. Nothing more. The controls in this baseline are all expressions of that principle at different layers of the stack.
Kernel Hardening via sysctl
The Linux kernel exposes a large number of tunable parameters through the /proc/sys/ virtual filesystem, settable persistently via sysctl. Most of the defaults are chosen for compatibility and functionality, not security. The right place to persist security-relevant settings is /etc/sysctl.d/99-hardening.conf — the 99- prefix ensures these settings load last and override any distribution defaults.
Apply a change immediately without rebooting:
|
|
Here is the rationale for each setting, followed by the complete file.
kernel.dmesg_restrict = 1 — The kernel ring buffer (dmesg) contains detailed information about hardware, loaded modules, memory layout, and driver messages. An unprivileged user reading dmesg can learn things that assist exploit development: ASLR bypass hints, kernel version specifics, driver names. Setting this to 1 restricts dmesg to CAP_SYSLOG. The operational cost is minimal — admins need to use sudo dmesg.
kernel.kptr_restrict = 2 — /proc/kallsyms exposes kernel symbol addresses. These are directly useful for constructing ROP chains in kernel exploits. With kptr_restrict = 1, kernel pointers are zeroed for unprivileged users but readable by root. Setting it to 2 hides them from everyone, including root. The cost is that debugging kernel issues becomes harder — perf tooling and some crash analysis tools need this. On a production server that should not be your concern.
kernel.randomize_va_space = 2 — Address Space Layout Randomization (ASLR). Setting 2 enables full randomization of stack, VDSO, and mmap regions. Setting 1 only randomizes stack and VDSO. The default on most distributions is already 2, but verifying it explicitly is worth doing. ASLR raises the cost of exploiting memory corruption vulnerabilities by making memory addresses unpredictable across runs — every exploit that relies on jumping to a fixed address in libc is broken unless the attacker can also leak a pointer.
kernel.yama.ptrace_scope = 1 — The ptrace syscall is used by debuggers to attach to running processes and read/write their memory. With ptrace_scope = 0 (default on many distros), any process can ptrace any process owned by the same user. This is a meaningful attack surface: a compromised process can use ptrace to inject code into another process running as the same user, including processes holding credentials (e.g., a GPG agent). Setting scope to 1 restricts ptrace to parent-child relationships. The cost: you cannot attach gdb to a running process without being its parent. Developers who need attach-to-process debugging will need sudo or a temporarily lowered scope.
kernel.perf_event_paranoid = 3 — The Linux performance counters subsystem has historically been a source of side-channel attacks. Setting paranoia to 3 restricts perf to root. Note that on some distributions, 3 is not a kernel-native value and must be enabled by a kernel patch (Ubuntu carries this patch; vanilla kernels stop at 2). Setting 2 restricts most perf operations to root; 3 adds additional restrictions. On a production server that does not need perf monitoring for unprivileged users, this is a sensible restriction.
net.ipv4.conf.all.rp_filter = 1 — Reverse path filtering (Strict mode). When the kernel receives a packet, it checks whether the source address would be reachable via the interface the packet arrived on. If the routing table says traffic to that source goes out a different interface, the packet is dropped. This is anti-spoofing protection: it prevents IP spoofing attacks that rely on asymmetric routing. Setting 1 enables strict mode; setting 2 enables loose mode (allows asymmetric paths). Use 1 unless you have legitimate asymmetric routing, in which case use 2.
net.ipv4.conf.all.accept_redirects = 0 and net.ipv6.conf.all.accept_redirects = 0 — ICMP redirect messages tell a host to update its routing table to use a different gateway. Accepting them allows an attacker on the local network to redirect traffic through a machine they control. There is essentially no legitimate reason for a server to accept ICMP redirects. Disable them.
net.ipv4.conf.all.send_redirects = 0 — Similarly, a non-router should never send ICMP redirects. If this host is acting as a gateway (running IP forwarding), consider this carefully. For most servers, disable it.
net.ipv4.conf.all.accept_source_route = 0 — Source routing allows a packet sender to specify the path the packet should take through the network. This was designed for diagnostic purposes and is now almost exclusively used for attacks. Disable it.
net.ipv4.tcp_syncookies = 1 — SYN flood protection. When the connection backlog fills, the kernel starts issuing cryptographic SYN cookies instead of allocating state. Legitimate connections can still complete; the flood cannot exhaust the connection table. This has been the default on most distributions for years, but it is worth verifying.
net.core.bpf_jit_harden = 2 — The eBPF JIT compiler has been a source of privilege escalation vulnerabilities. Hardening mode 2 adds constant blinding (randomizing immediate values in JIT-compiled programs), making JIT spraying attacks significantly harder. The cost is a small performance regression in eBPF-heavy workloads. On a server not doing large-scale packet processing, it is negligible.
fs.protected_hardlinks = 1 and fs.protected_symlinks = 1 — These prevent two classic TOCTOU (time-of-check time-of-use) race condition exploit classes. Without protection, an attacker can create hardlinks to setuid files in world-writable directories (like /tmp) to trigger privilege escalation. Symlink protection prevents following symlinks in world-writable sticky directories if the symlink owner does not match the follower or the directory owner. Both are enabled by default on modern Ubuntu but should be verified.
fs.suid_dumpable = 0 — Disables core dump generation for setuid and setgid programs. Core dumps from setuid programs can expose sensitive data from the privileged execution context. The default on most distributions is 0, but it is worth confirming.
Here is the complete file:
|
|
One important caveat: net.ipv6.conf.all.accept_ra = 0 will break IPv6 auto-configuration on hosts that rely on Router Advertisements for address assignment (most SLAAC setups). If your server gets its IPv6 address via SLAAC, either use stateful DHCPv6 or accept the RA restriction on the specific interface rather than globally.
Filesystem and Mount Hardening
The mount options on filesystems are a frequently overlooked hardening opportunity. Several dangerous operations — executing binaries, creating setuid files, creating device nodes — can be restricted at the filesystem level regardless of the permissions on individual files.
The key mount options are:
| Option | Effect |
|---|---|
noexec |
Prevents executing binaries on this filesystem |
nosuid |
Ignores setuid/setgid bits on files in this filesystem |
nodev |
Prevents interpreting device special files |
For /tmp, mounting it as a separate tmpfs with all three restrictions eliminates a common attacker staging area. An attacker who has achieved code execution and wants to drop a payload into /tmp and execute it is blocked:
tmpfs /tmp tmpfs defaults,noexec,nosuid,nodev,size=512M 0 0
tmpfs /var/tmp tmpfs defaults,noexec,nosuid,nodev,size=256M 0 0
Note the size limit — without it, an unbounded tmpfs can consume all available RAM. Size it based on what your applications actually write to /tmp.
For other filesystems:
# /var should not contain executables owned by users
/dev/sdaX /var ext4 defaults,nosuid,nodev 0 2
# /home should not need device files
/dev/sdaX /home ext4 defaults,nodev 0 2
The /proc filesystem with hidepid deserves a detailed treatment because the compatibility situation is complicated. The intent of hidepid=2 is to hide other users’ process information: a non-root user cannot see entries in /proc for processes they do not own. This is a meaningful privacy and information-disclosure control in multi-user environments.
The problem is that this breaks systemd. Specifically, systemd-logind, the user session manager (user@.service), and several other system services need to traverse /proc to enumerate process state, and they do so as accounts that do not own those processes. The failure mode is not subtle: users cannot log in, services fail to start at boot.
The workaround that actually works on systemd-based systems uses the gid option to grant a specific group exemption from hidepid:
|
|
Then add the services that need proc visibility to the proc group via drop-in overrides:
|
|
Even with this workaround, some services (sssd, polkit, certain monitoring agents) may still fail. The honest assessment is that on a systemd-based single-user server (where only one human account should ever log in), the security benefit of hidepid=2 is limited. Red Hat explicitly does not recommend it for RHEL 7+. For multi-user systems where tenant isolation matters, it is worth the configuration complexity. For single-workload production servers, the per-service ProtectProc= directive in systemd units (covered in the sandboxing section) achieves similar isolation with less operational risk.
Disabling unused kernel modules is a highly effective control that receives insufficient attention. A kernel module that is never loaded cannot be exploited. The mechanism is simple — override the module installation command in /etc/modprobe.d/:
|
|
The install <module> /bin/false trick works by replacing the module installation command with a call to /bin/false, which immediately exits with a nonzero status. Attempting to modprobe cramfs will silently fail.
setuid binary inventory — Every setuid binary on the system is a potential privilege escalation vector. Know what you have:
|
|
The -xdev flag prevents descending into other filesystems (important on systems with NFS mounts). Review the output and remove the setuid bit from any binary that does not require it: chmod u-s /path/to/binary. On minimal server installations, the list should be short — sudo, su, passwd, ping, mount, umount, and a handful of others. Any unexpected entries warrant investigation.
SSH Hardening
SSH is the administrative entry point for virtually every Linux server. It is also the service most consistently targeted by automated scanners and credential stuffing attacks. The default sshd_config has improved over the years but still has settings that need to be locked down for a production environment.
The hardening philosophy here is: keys only, explicit allowlists, modern cryptography, minimal feature exposure. Every forwarding feature, every authentication method, and every legacy compatibility option is an attack surface.
Here is a complete hardened sshd_config. The comments explain the security rationale for non-obvious settings:
# /etc/ssh/sshd_config
# Verify effective config after changes: sshd -T
Port 22
AddressFamily inet
##
## Authentication
##
PermitRootLogin no
PasswordAuthentication no
PubkeyAuthentication yes
AuthorizedKeysFile .ssh/authorized_keys
PermitEmptyPasswords no
ChallengeResponseAuthentication no
KerberosAuthentication no
GSSAPIAuthentication no
# Limit failed attempts - combined with fail2ban this blocks brute force
MaxAuthTries 3
MaxSessions 4
# Disconnect clients that have not authenticated within 30 seconds
LoginGraceTime 30
##
## Access control
## Replace with the actual user/group that should have SSH access
AllowGroups sshusers
##
## Idle session management
## Disconnect sessions idle for 10 minutes (300s * 2 checks)
ClientAliveInterval 300
ClientAliveCountMax 2
##
## Features - disable everything not in active use
X11Forwarding no
AllowAgentForwarding no
AllowTcpForwarding no
PermitTunnel no
GatewayPorts no
# Do not print /etc/motd through SSH - reduces information disclosure
PrintMotd no
PrintLastLog yes
# Avoid reverse DNS lookups - faster auth, removes dependency on DNS availability
UseDNS no
# Strict permissions check on ~/.ssh and authorized_keys
StrictModes yes
##
## Cryptography - modern algorithms only (OpenSSH 9.x)
##
# Host key algorithms - ed25519 preferred, ECDSA fallback
HostKeyAlgorithms ssh-ed25519,ecdsa-sha2-nistp256,rsa-sha2-512,rsa-sha2-256
# Key exchange - post-quantum hybrid first, then Curve25519
KexAlgorithms sntrup761x25519-sha512@openssh.com,curve25519-sha256,curve25519-sha256@libssh.org,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha256
# Symmetric ciphers - AEAD modes preferred (provide integrity as well as confidentiality)
Ciphers chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr
# MACs - ETM (encrypt-then-mac) modes preferred; -etm suffix
MACs hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,umac-128-etm@openssh.com
##
## Logging
##
SyslogFacility AUTH
LogLevel VERBOSE
A few items that warrant additional discussion:
AllowGroups sshusers — Create this group and add only the accounts that legitimately require SSH access. This is an explicit allowlist that means a newly created system account cannot be used for SSH login even if someone manages to set a password or drop in an authorized key. The operational cost is that you must remember to add the group when creating admin accounts.
PermitRootLogin no — The root account has no failed-login lockout by default. Permitting root login via SSH means an attacker who compromises a key or (if password auth is somehow re-enabled) guesses the password gets immediate full system access with no privilege escalation step needed. Always require login as a named user with sudo.
AllowTcpForwarding no — SSH tunneling can be used to bypass network-level controls, proxy traffic through your server, or create persistent connections that survive firewall rule changes. Disable it unless your use case explicitly requires it.
sntrup761x25519-sha512@openssh.com — This is a post-quantum hybrid key exchange that combines the NTRU Prime lattice-based algorithm with Curve25519. It is available in OpenSSH 8.5+ and is the default first preference in OpenSSH 9.x. Including it in your KexAlgorithms list means post-quantum-capable clients will use it automatically.
Validating your configuration:
|
|
sshd -T is invaluable for auditing — it shows every option’s effective value including compiled-in defaults, so you can verify that settings you have not explicitly configured are what you expect.
SSH key management — Use ed25519 keys exclusively for new deployments:
|
|
ed25519 keys are smaller, faster, and based on more carefully analyzed mathematics than RSA. A 256-bit ed25519 key provides stronger security guarantees than a 4096-bit RSA key. For RSA compatibility with legacy systems, use at minimum 4096 bits.
Deploy keys with ssh-copy-id, then verify permissions:
|
|
SSH Certificate Authority — For environments with more than a handful of servers, per-server authorized_keys management becomes a liability. SSH certificates solve this: a CA key signs user keys, servers trust the CA, and you revoke access by not renewing certificates rather than hunting through authorized_keys files on dozens of hosts. The operational investment pays off quickly in teams with 10+ servers. Tools like HashiCorp Vault’s SSH secrets engine automate certificate signing and can integrate with your identity provider.
fail2ban
fail2ban watches log files for patterns indicating failed authentication, then creates temporary firewall rules to block the source IP. It is not a substitute for disabling password authentication — that comes first — but it is a meaningful control against credential stuffing and key enumeration attacks.
Install and configure via /etc/fail2ban/jail.local. Never edit jail.conf directly — it gets overwritten on package updates. jail.local is the override file that survives upgrades.
|
|
Operational commands:
|
|
One nuance: fail2ban creates iptables/nftables rules in memory. The default configuration does not make ban rules persistent across reboots, which means a reboot clears the ban list. For high-value targets this is acceptable — the attacker needs to re-enumerate. For environments where you are tracking persistent adversaries, consider using bantime.persistent = true in the [DEFAULT] section, which stores ban state to disk.
Alternatives worth knowing: sshguard is a lighter-weight alternative focused specifically on SSH with a simpler configuration model. crowdsec takes a different approach: it combines local log analysis with a community-sourced IP reputation database, so you get the benefit of blocking IPs that have attacked other servers in the community before they attack yours. The trade-off is that crowdsec phones home with anonymized event data, which may not be acceptable in certain compliance contexts.
AppArmor: Mandatory Access Control
AppArmor is a Linux Security Module (LSM) that implements Mandatory Access Control (MAC). The distinction between MAC and traditional Unix DAC (Discretionary Access Control) matters: DAC permissions can be changed by the file owner; MAC policies are enforced by the kernel regardless of what the process thinks its permissions are. A confined process cannot grant itself access to resources that the MAC policy prohibits, even if it achieves code execution.
The AppArmor model is path-based: policies specify which filesystem paths a process may access and how (read, write, execute, memory-map), which capabilities it may exercise, which network operations it may perform, and what signals it may send or receive.
Process makes request
|
v
Linux VFS / Syscall layer
|
v
+-------------------+
| AppArmor LSM |
| hook intercepts |
+-------------------+
|
Profile loaded?
/ \
Yes No (unconfined — full access)
|
v
Profile check
| \
Allow Deny
| \
Continue [enforce mode] -> EPERM returned, audit log entry
[complain mode] -> audit log entry, operation allowed
The two modes matter operationally:
- Enforce mode: violations are blocked and logged. This is what you want in production.
- Complain mode: violations are logged but allowed. Use this when developing or testing a new profile — it lets the application run normally while revealing what the profile would block.
The standard workflow is: start in complain mode, exercise all code paths, review violations with aa-logprof, iterate until the profile is complete, switch to enforce.
|
|
Generating profiles with aa-genprof:
|
|
In another terminal, run your application through its typical use cases — start it, make requests to it, exercise its configuration loading, trigger any code paths that touch the filesystem. Then return to the aa-genprof terminal and press S to scan the log. It will present each access that was denied and ask whether to allow it, with options for globbing the path at different levels of specificity. Be conservative: prefer narrow paths over broad globs.
After the initial profile generation, aa-logprof handles ongoing tuning based on audit log entries from the running application:
|
|
Profile anatomy — Here is an AppArmor profile for a simple web server to illustrate the syntax:
# /etc/apparmor.d/usr.local.bin.webserver
#include <tunables/global>
/usr/local/bin/webserver {
#include <abstractions/base>
#include <abstractions/nameservice>
# Binary itself
/usr/local/bin/webserver mr,
# Configuration - read only
/etc/webserver/ r,
/etc/webserver/** r,
# Web content root - read only
/var/www/html/ r,
/var/www/html/** r,
# Log directory - write
/var/log/webserver/ rw,
/var/log/webserver/*.log rw,
# PID file
/run/webserver.pid rw,
# TLS certificates - read only
/etc/ssl/certs/ r,
/etc/ssl/certs/** r,
/etc/webserver/tls/ r,
/etc/webserver/tls/** r,
# Network - listen on TCP port 443
network inet stream,
network inet6 stream,
# Capabilities needed
capability net_bind_service, # bind to privileged ports
capability setuid, # drop privileges after binding
capability setgid,
# Explicitly deny sensitive paths
deny /etc/shadow r,
deny /etc/gshadow r,
deny /root/ r,
deny /home/ r,
deny /proc/*/mem rw,
deny @{PROC}/{[^1-9],[^1-9][^0-9]}/mem rw,
}
The deny rules are mostly belt-and-suspenders since the profile does not allow those paths — but explicit denies make the intent clear in code review and prevent future abstract include additions from accidentally granting access.
Operational trade-offs: AppArmor profile development has a real time cost. For widely deployed services (nginx, apache, bind, mysql, postgresql), distributions ship profiles or they are available in the apparmor-profiles and apparmor-profiles-extra packages. Start there and modify as needed. For custom applications, budget time for profiling during development, not as an afterthought — it is much harder to profile an application you do not understand.
Docker automatically applies the docker-default AppArmor profile to containers unless overridden. This profile provides meaningful restrictions (blocks mount, restricts /proc writes, denies kernel module loading) while being permissive enough for most containerized applications. Custom container profiles should be written for high-value or high-risk containers.
seccomp: Syscall Filtering
While AppArmor restricts what resources a process can access, seccomp restricts what system calls a process can make. A process that has achieved arbitrary code execution — say, via a buffer overflow or use-after-free — still has to make syscalls to do anything useful. It cannot read a file without open()/read(), cannot connect to a C2 server without socket()/connect(), cannot execute a new binary without execve(). seccomp blocks those syscalls at the kernel level, regardless of what the process is trying to do.
There are three seccomp modes:
- SECCOMP_SET_MODE_STRICT: only
read,write,_exit, andsigreturnare allowed. Used by highly specialized code (gpg agent internals, etc.) — most applications cannot operate under this. - SECCOMP_SET_MODE_FILTER: custom BPF filter program that evaluates each syscall and returns a verdict (allow, kill, return error, log). This is what most practical seccomp usage looks like.
- Disabled: the default for most processes.
Writing seccomp-bpf filters directly in C is tedious and error-prone. In practice, you use one of three interfaces:
Docker seccomp profiles:
|
|
Docker’s default seccomp profile blocks approximately 44 syscalls including kexec_load, create_module, init_module, ptrace, process_vm_readv/writev, and others that have no legitimate use in a containerized application but are useful for privilege escalation or container escape. Review the Docker documentation if you need to add permissions for specialized containers (e.g., containers running eBPF programs need bpf()).
systemd SystemCallFilter:
This is the most accessible interface for non-container workloads. systemd defines named groups of related syscalls that you can allowlist or denylist:
SystemCallFilter=@system-service
SystemCallFilter=~@privileged ~@mount ~@reboot
The @system-service group is a broad allowlist appropriate for most daemons — it includes the syscalls needed for normal service operation (file I/O, networking, IPC, memory management) while excluding the dangerous ones. The tilde prefix (~) denotes a deny rule.
Named syscall groups in systemd:
| Group | Contents |
|---|---|
@system-service |
read, write, open, stat, socket, connect, and normal service ops |
@privileged |
mount, umount, kexec, create_module, ptrace, rawsockets |
@mount |
mount, umount2, pivot_root, chroot |
@reboot |
reboot, kexec_load, kexec_file_load |
@network-io |
socket, bind, listen, accept, connect, send/recv variants |
@process |
clone, fork, execve, waitpid, kill, prctl |
@debug |
ptrace, process_vm_readv/writev, kcmp, perf_event_open |
@swap |
swapon, swapoff |
A practical hardened unit with SystemCallFilter:
|
|
SystemCallArchitectures=native is important: it prevents the process from switching to a 32-bit personality to call 32-bit syscall numbers that your filter does not cover. Without it, a clever exploit can sometimes bypass seccomp filters by using the 32-bit syscall ABI on a 64-bit kernel.
Kubernetes seccomp: Pods can have a seccompProfile specified in the pod spec, referencing either the runtime default (Docker/containerd default profile) or a custom profile stored on the node at /var/lib/kubelet/seccomp/. The RuntimeDefault profile is the safest starting point for most workloads.
|
|
The key trade-off with seccomp is development friction: if you block a syscall that the application uses on an uncommon code path (error handling, crash reporting, debugging features), the application will behave unexpectedly or crash when that path is hit. Always use SystemCallErrorNumber=EPERM rather than the default SIGSYS in development, and audit the audit.log for blocked calls before switching to strict enforcement.
auditd: Syscall Monitoring
auditd is the kernel’s audit subsystem — it generates tamper-evident logs of security-relevant events at the syscall level. Authentication events, privilege escalation, file access to sensitive paths, execution of commands, and network configuration changes all pass through the audit framework. The logs include process ancestry (who spawned the process that made the call), the UID/GID at the time, and the file paths involved.
The audit subsystem is covered in depth in a dedicated post on this blog. This section focuses on the rules appropriate for a security hardening baseline.
Rules live in /etc/audit/rules.d/. Files in that directory are merged and loaded by augenrules --load at boot. The rule syntax:
-w <path> -p <permissions> -k <key>— watch a file or directory-a <action>,<filter> -S <syscall> [-F <field>=<value>] -k <key>— audit a syscall
Permissions for -p: r (read), w (write), x (execute), a (attribute change).
Here is a baseline rules file:
|
|
A note on volume: the execve monitoring rule for all execution is deliberately not included in this baseline because on a busy server it generates enormous log volume that buries the signal you actually care about. The setuid execution rules (which capture privilege escalation via execve with differing uid/euid) provide the important subset. If you need full execution logging, consider filtering to specific directories or specific UIDs:
|
|
Querying the audit log:
|
|
The -e 2 (immutable mode) directive deserves special mention. Once loaded, it prevents any runtime changes to audit rules until the next reboot. An attacker who achieves root and wants to disable auditd to cover their tracks cannot do so — the kernel will reject all auditctl calls. The trade-off is that you also cannot modify rules at runtime; all changes require a reboot. Enable it once you are confident your rule set is stable.
systemd Unit Sandboxing
systemd’s security directives implement per-service namespacing and capability restrictions. They are easy to apply incrementally and can dramatically reduce the blast radius of a compromised service. The key metric is systemd-analyze security <unit>, which scores a service from 0.0 (fully sandboxed) to 10.0 (no sandboxing).
Check your current exposure before hardening:
|
|
The directives, in rough order of importance:
NoNewPrivileges=yes — The single most important sandboxing directive. Prevents the service and all its children from ever gaining additional privileges via execve() of a setuid/setgid binary or via prctl(PR_SET_SECUREBITS). A compromised nginx process cannot escalate to root by running sudo or a setuid binary, because the kernel blocks privilege elevation for processes with the no-new-privileges flag set. This should be set on every service that does not need to spawn privileged subprocesses.
PrivateTmp=yes — Creates a private /tmp and /var/tmp namespace for the service. The service sees an empty /tmp that is invisible to other services and users. Common attack path blocked: an attacker who has written a payload to /tmp via one vector cannot have a compromised service execute it, because the service’s /tmp is a different namespace. Also prevents symlink attacks between services via shared temp directories.
PrivateDevices=yes — Replaces /dev with a minimal pseudo-device filesystem containing only null, zero, full, random, urandom, tty, and a few others. The service cannot access raw disk devices, memory devices (/dev/mem), or port I/O devices (/dev/port), which eliminates several classes of privilege escalation.
ProtectSystem=strict — Mounts /usr, /boot, /efi, and /etc read-only. The service cannot modify system binaries, configuration, or kernel. Use ReadWritePaths= to explicitly grant write access to the specific directories the service needs (e.g., /var/lib/myservice).
ProtectHome=yes — Makes /home, /root, and /run/user inaccessible. Services should not be reading home directories.
CapabilityBoundingSet= — Sets the maximum set of capabilities the service can ever hold. Even if the service escalates privileges, it cannot gain capabilities not in the bounding set. For a service that runs as root to bind a privileged port and then drops to a lower-privilege user, you might set CapabilityBoundingSet=CAP_NET_BIND_SERVICE. For a service that only needs to read and write files as an unprivileged user, set it empty (CapabilityBoundingSet=).
AmbientCapabilities= — Combined with running as a non-root user and CapabilityBoundingSet, this grants specific capabilities to a non-root service without requiring it to be setuid. Use CAP_NET_BIND_SERVICE to let a service bind to port 80 or 443 without running as root.
RestrictNamespaces=yes — Prevents the service from creating new Linux namespaces. Namespace creation is a key primitive for container escapes and certain privilege escalation techniques (particularly user namespace abuse). Most services have no legitimate reason to create namespaces.
RestrictAddressFamilies= — Restricts the socket address families the service can use. A service that only needs IPv4 and IPv6 TCP sockets does not need AF_UNIX, AF_NETLINK, or AF_PACKET:
|
|
LockPersonality=yes — Prevents changing the execution personality (32-bit compatibility mode, etc.). This closes a seccomp bypass vector mentioned earlier.
RestrictSUIDSGID=yes — Prevents the service from setting setuid/setgid bits on files, even if it has write access to those files.
Here is a fully hardened service unit for a hypothetical web application:
|
|
After applying this configuration:
|
|
A fully hardened unit like this should score in the 1.0-2.5 range. Scores under 2.0 indicate strong sandboxing. The remaining exposure typically reflects the AF_INET/AF_INET6 network access that most services legitimately need.
Important caveat on MemoryDenyWriteExecute: This setting prevents creating memory that is simultaneously writable and executable. It breaks JIT compilers (Java, Node.js V8, Python with certain JIT extensions). Do not apply it to services running JIT-compiled runtimes. For compiled Go/Rust/C services, it is generally safe.
ProtectProc=invisible / ProcSubset=pid — These are the per-service alternatives to the global hidepid=2 mount option. ProtectProc=invisible hides other processes’ /proc entries from this service. ProcSubset=pid limits the service to only pid-related /proc entries. These avoid the systemd-wide compatibility issues associated with the global hidepid option.
unattended-upgrades and Patch Management
The most common initial access vector is a known, patched vulnerability in an outdated package. Hardening kernel parameters and MAC policies does nothing against an attacker who exploits CVE-2024-6387 on an unpatched SSH daemon. Automated security updates are not glamorous, but they are foundational.
|
|
The main configuration file:
# /etc/apt/apt.conf.d/50unattended-upgrades
Unattended-Upgrade::Allowed-Origins {
"${distro_id}:${distro_codename}";
"${distro_id}:${distro_codename}-security";
"${distro_id}ESMApps:${distro_codename}-apps-security";
"${distro_id}ESM:${distro_codename}-infra-security";
};
# Remove automatically installed dependencies no longer needed
Unattended-Upgrade::Remove-Unused-Dependencies "true";
# Automatically reboot if required - set to true only for non-critical systems
# or if you have a maintenance window process
Unattended-Upgrade::Automatic-Reboot "false";
Unattended-Upgrade::Automatic-Reboot-Time "02:00";
# Email notifications on errors
Unattended-Upgrade::Mail "ops@yourdomain.com";
Unattended-Upgrade::MailReport "on-change";
# Remove old debs from cache
Unattended-Upgrade::Remove-New-Unused-Dependencies "true";
The Allowed-Origins section limits auto-upgrades to security updates. General updates (new features, behavior changes) are not applied automatically — those go through your change management process. Security patches flow immediately.
Testing what would be upgraded:
|
|
The kernel reboot problem: Installing a kernel update does not apply it until the next reboot. A system running the new kernel package but booted on the old kernel is not actually patched. Three approaches:
- Schedule reboots: Accept a maintenance window after each kernel update. Use
needrestartto detect when a reboot is required. - kexec: Faster reboots via kernel execution — the new kernel takes over without a full hardware POST cycle. Boot time drops from ~minutes to ~seconds on most hardware. Enable with
kexec-tools. - Livepatch: Ubuntu Pro includes Livepatch for live kernel patching without reboots. Red Hat has kpatch, SUSE has kGraft. These work for most CVE-class kernel patches but not all.
After installing upgrades, identify services using old shared library versions:
|
|
This is important: a service that has loaded an old version of a library (e.g., an old openssl) continues to use that old code until it is restarted, even after the package is updated on disk.
Validation: Measuring Your Hardening Posture
Applying controls without measuring their effect is guesswork. These tools give you scored assessments of what is and is not hardened.
lynis audit system (version 3.1.6 as of late 2025) is the most accessible auditing tool. It runs a broad scan of system configuration, checks against known-good settings, and produces a scored report with specific remediation suggestions:
|
|
The hardening index score (0-100) is not a compliance score — it is a relative measure of how many of lynis’s checks your system passes. A score in the 80s is achievable with the baseline in this post. A perfect 100 would require controls that are impractical for most operational environments.
OpenSCAP provides formal compliance evaluation against CIS Benchmarks and DISA STIGs:
|
|
The HTML report maps each CIS control to a pass/fail result with the specific file path and required value. It is verbose but thorough.
rkhunter --check scans for known rootkits, suspicious files, and system configuration issues. It is not a substitute for the controls above — if you have a rootkit, your hardening has already failed — but it is a useful detection layer:
|
|
Run --propupd after any legitimate system changes to avoid false positives on the next scan.
A word on security theater: Several common “hardening” recommendations provide no measurable security benefit and should be avoided because they consume operational attention without improving security posture:
- Changing the SSH port to a non-standard number. This reduces automated scanner noise in logs but does not prevent a targeted attacker from scanning for it.
- Displaying a “you are being monitored” banner in
/etc/issueor/etc/motd. This does not deter attackers. - Obscuring system version information in service banners. Useful for reducing fingerprinting, but not a security control — an attacker who is present will enumerate the real version.
Focus on controls that change the attacker’s capability, not their knowledge. An attacker who knows your kernel version but cannot read kernel pointers, cannot execute payloads in /tmp, and has every service call monitored by auditd is effectively constrained. An attacker who does not know your kernel version but faces none of those controls is not.
Hardening Controls Quick Reference
| Control | Threat Mitigated | Operational Risk |
|---|---|---|
kernel.dmesg_restrict=1 |
Information disclosure for exploit dev | Low — sudo dmesg still works |
kernel.kptr_restrict=2 |
Kernel ASLR bypass via address leaks | Medium — breaks some perf/debug tooling |
kernel.randomize_va_space=2 |
Memory corruption exploit reliability | Very low — should be default |
kernel.yama.ptrace_scope=1 |
Process injection / credential theft | Medium — breaks attach-to-process debugging |
net.ipv4.conf.all.rp_filter=1 |
IP spoofing | Low — breaks only asymmetric routing |
net.ipv4.conf.all.accept_redirects=0 |
ICMP redirect MITM | Very low |
net.core.bpf_jit_harden=2 |
eBPF JIT spraying attacks | Low — small perf regression |
fs.protected_symlinks=1 |
/tmp symlink TOCTOU exploits |
Very low |
noexec on /tmp |
Payload staging and execution in /tmp | Medium — some apps write+exec from /tmp |
hidepid=2 on /proc |
Process information disclosure | High — breaks systemd services |
Disable usb-storage module |
Data exfiltration, BadUSB | High if USB storage is operationally needed |
SSH PasswordAuthentication no |
Credential brute force | Medium — requires key deployment first |
SSH AllowGroups |
Unauthorized account access | Low — must add group to new admin accounts |
| SSH modern KexAlgorithms | Downgrade attacks, weak crypto | Low — old clients may not connect |
| fail2ban SSH jail | Brute force / credential stuffing | Low — may ban shared-NAT office IPs |
| AppArmor (enforce mode) | Post-exploitation filesystem access | High — requires profile development |
seccomp / SystemCallFilter |
Post-exploitation syscall abuse | Medium — uncommon code paths may break |
NoNewPrivileges=yes |
Privilege escalation via setuid | Low — breaks services that need to spawn setuid |
PrivateTmp=yes |
Cross-service /tmp attacks |
Low |
ProtectSystem=strict |
Service writes to system paths | Low — must use ReadWritePaths for data dirs |
MemoryDenyWriteExecute=yes |
JIT-based shellcode injection | High — breaks JVM, V8, PyPy |
RestrictNamespaces=yes |
Container escape via user namespaces | Low for most services |
auditd immutable mode (-e 2) |
Attacker disabling audit | Low — requires reboot to change rules |
| unattended-upgrades | Known CVE exploitation | Medium — updates can break services |
Applying This Baseline
The controls in this post are not switches to flip all at once. The path to a hardened system that stays operational:
- Apply sysctl hardening first — lowest risk, highest coverage.
- Harden SSH and verify key-based auth works before disabling password auth.
- Enable fail2ban.
- Configure unattended-upgrades.
- For each service, run
systemd-analyze securityand apply sandboxing directives incrementally, testing after each change. - Run
lynis audit systemto identify remaining gaps. - Develop AppArmor profiles in complain mode, exercise all code paths, switch to enforce once clean.
- Add auditd rules, enable immutable mode last.
The measure of a good hardening baseline is not the score on a compliance report — it is whether a compromised application can cause harm beyond its immediate scope. If your web server gets exploited and the attacker cannot read SSH keys, cannot write to system paths, cannot make outbound connections, and every syscall is logged, you have succeeded. The compliance report is a byproduct of that outcome, not the goal.
Comments