PAM Configuration Deep Dive

PAM — the Pluggable Authentication Modules framework — is the layer that decides whether you get a shell, whether sudo trusts you, whether sshd lets you past the password prompt, and whether your screen locker believes you are really you. It has been sitting under your login prompt since 1995, and most engineers know enough PAM to be dangerous: they can copy a stanza from a blog post into /etc/pam.d/sshd, but when a failed login produces a silent rejection at 2 a.m., they cannot read the config they are looking at.

PAM is strange because it is simultaneously one of the most critical pieces of infrastructure on a Linux system and one of the least-understood. The file format looks like a shell script, but it is not. The stack runs top-to-bottom, but control flow is not what it appears. An auth line can fail silently for three different reasons. Modules have behaviors that are documented only in source code. And every distribution disagrees about what belongs in the default stacks, which means copy-pasting configs between a Red Hat box and a Debian box is how most PAM outages begin.

This post is what I wish I had when I first had to debug PAM for real. We will walk the stack semantics end-to-end, cover the modules you actually meet (pam_unix, pam_sss, pam_google_authenticator, pam_faillock, pam_mkhomedir, pam_systemd, pam_env), go through concrete debugging with strace, pamtester, and the PAM debug log, and finish with the configuration patterns you will want when you design an MFA rollout or integrate with an external identity provider.

What PAM actually is

PAM is a library — libpam.so — linked into every program that wants to authenticate a user. When sshd wants to know whether the password you typed is correct, it does not call getpwnam() and crypt() directly. It calls pam_authenticate(), and the PAM library walks a stack of shared objects (pam_unix.so, pam_faillock.so, etc.), each of which can participate in the decision. The service’s config file in /etc/pam.d/<service> tells PAM which modules to load and in what order.

This design means that the decision logic for “can this user log in?” lives outside of sshd, outside of login, outside of sudo. You can add two-factor authentication to every one of them by editing a config file — no recompilation, no service-specific code. It also means that a single typo in /etc/pam.d/system-auth can lock every user out of every service at once. PAM is both extremely powerful and extremely fragile, and the two properties are the same property.

There are four management groups — the categories of question PAM modules answer:

• auth — Is this the person they claim to be? Password checks, MFA tokens, biometric, Kerberos tickets.
• account — Is this account allowed to log in right now? Expiry checks, time-of-day restrictions, host restrictions, group membership.
• password — The user is changing their password. Enforce complexity, update the store.
• session — Set up and tear down the session: mount the home dir, set limits, log to wtmp, hand off to systemd.

When sshd calls pam_authenticate(), only the auth stack runs. When it then calls pam_acct_mgmt(), only the account stack runs. And so on. A common mistake is putting an MFA module in the session stack and wondering why it is never challenging users — session runs after you are already in.

The control field: the piece everyone misunderstands

Every non-comment line in a PAM config has this shape:

<type>   <control>   <module>   [arguments...]

The control field decides what happens based on the module’s return value. The traditional values — which you will see in almost every config — are required, requisite, sufficient, and optional.

• required — If this module fails, the overall stack will eventually fail. But PAM keeps evaluating subsequent modules. This is deliberate: it prevents an attacker from learning which check failed (password? account lockout? group membership?). The user just gets a generic failure at the end.
• requisite — Same as required, but fails the stack immediately and returns to the application. Used when continuing would be dangerous — e.g., a pre-auth check that rejects the connection before a password is even transmitted.
• sufficient — If this succeeds and no earlier required has failed, the stack short-circuits to success. If it fails, evaluation continues — the next module gets another chance.
• optional — The return code normally does not affect the overall result, unless this is the only module in the stack for that type.

The subtle part is that required and sufficient interact. A sufficient further down the stack can short-circuit past a required that already failed silently — which is exactly what happens in the classic stack: pam_unix is often sufficient (UNIX password works → you’re in), and pam_deny is required at the bottom (nothing matched → denied).

You will also see the bracketed control syntax:

auth [success=1 default=ignore] pam_google_authenticator.so
auth requisite                   pam_deny.so
auth sufficient                  pam_unix.so

This form is explicit: each possible return value (success, auth_err, user_unknown, etc.) maps to an action (ignore, bad, die, ok, done, reset, or an integer — skip N modules forward). The example above means: if Google Authenticator succeeds, jump past the pam_deny and hit pam_unix. Otherwise, fall through to the pam_deny and fail. This is how you express “MFA is required before password” compactly.

Return value semantics — this is the part most engineers never learn:

• success — the module passed
• ignore — the module did not participate (e.g., pam_sss when the user is local)
• user_unknown — the module does not know this user
• auth_err — authentication failed (wrong password, bad token)
• new_authtok_reqd — password is expired, must be changed
• perm_denied — the module actively rejects the user (account locked, wrong group)

When debugging, the difference between user_unknown and auth_err is often the root cause of a mystery. pam_unix returns user_unknown if the user is not in /etc/passwd — not a failure. pam_sss might then be expected to handle it. If the stack is written so user_unknown is treated as bad, an LDAP-only user will be rejected before pam_sss ever gets called.

Walking a real stack

Here is an annotated /etc/pam.d/sshd on a modern RHEL-family system, reading through includes:

# /etc/pam.d/sshd
auth       required     pam_sepermit.so
auth       substack     password-auth
auth       include      postlogin
account    required     pam_nologin.so
account    include      password-auth
password   include      password-auth
session    required     pam_selinux.so close
session    required     pam_loginuid.so
session    required     pam_selinux.so open env_params
session    required     pam_namespace.so
session    optional     pam_keyinit.so force revoke
session    include      password-auth
session    include      postlogin

The interesting part is substack and include. Both pull in another file, but they differ in one crucial way: a substack contains failures within itself — a requisite inside the substack only aborts the substack, not the caller. An include is textually inlined; a requisite inside an include aborts the whole thing.

password-auth is the real meat:

# /etc/pam.d/password-auth (common shared stack)
auth        required      pam_env.so
auth        required      pam_faildelay.so delay=2000000
auth        [default=1 ignore=ignore success=ok] pam_localuser.so
auth        [success=done default=die] pam_unix.so nullok
auth        [default=ignore] pam_succeed_if.so uid >= 1000 quiet_success
auth        [default=1 ignore=ignore success=ok] pam_usertype.so isregular
auth        sufficient    pam_sss.so forward_pass
auth        required      pam_deny.so

account     required      pam_unix.so
account     sufficient    pam_localuser.so
account     sufficient    pam_usertype.so issystem
account     [default=bad success=ok user_unknown=ignore] pam_sss.so
account     required      pam_permit.so

password    requisite     pam_pwquality.so local_users_only
password    sufficient    pam_unix.so yescrypt shadow nullok use_authtok
password    sufficient    pam_sss.so use_authtok
password    required      pam_deny.so

session     optional      pam_keyinit.so revoke
session     required      pam_limits.so
-session    optional      pam_systemd.so
session     [success=1 default=ignore] pam_succeed_if.so service in crond quiet use_uid
session     required      pam_unix.so
session     optional      pam_sss.so

Read the auth stack carefully. The first three lines set up environment, introduce a 2-second delay on failure (so you can’t use PAM to measure response times), and use pam_localuser to check whether the user is in /etc/passwd. If they are local, pam_unix.so runs and success=done short-circuits the entire stack on success — a remote LDAP lookup is never attempted. If they are not local, pam_localuser returns a failure which the bracket maps to skipping one line ahead, landing on pam_sss. This is a performance and availability pattern: local users must always work, even when LDAP is down.

The -session optional pam_systemd.so line has a leading -. That tells PAM: if this module is missing, silently ignore it. Useful for modules that are optional on some installs.

The final pam_deny.so in each stack is the safety net. If no module has said “yes,” deny by default. Remove it and you introduce a fail-open vulnerability.

The modules you’ll actually meet

pam_unix

The classic. Reads /etc/shadow (which means it needs to run as root or setuid, which is why chage works), compares a hashed password, returns success or auth_err. Knobs worth knowing:

• nullok — allow empty passwords (usually you don’t want this)
• try_first_pass / use_first_pass — accept the password already supplied by a previous module, no reprompt
• sha512, yescrypt, bcrypt — which hash algorithm to use for new passwords in the password stack
• rounds=N — configurable iteration count for stretching

pam_sss

Talks to sssd, which in turn talks to LDAP/AD/FreeIPA/Kerberos. Most enterprise Linux deployments have pam_sss in the stack behind pam_unix. Important flags:

• forward_pass — pass the password forward so subsequent modules don’t need to reprompt
• use_authtok — use a password already stored in the PAM conversation
• quiet — suppress certain log messages

The most common pam_sss surprise: sssd caches credentials. A user whose account was disabled in Active Directory may still log in for hours until the cache entry expires. Check /var/lib/sss/db/cache_*.ldb and clear with sss_cache -E if needed.

pam_google_authenticator

TOTP (RFC 6238). Reads ~/.google_authenticator for the per-user secret. Can be placed early (auth required) to force MFA before password, or late (after password success) to require both. Common options:

• nullok — if the user has not set up ~/.google_authenticator, let them through without MFA (for gradual rollout)
• secret=... — alternate path to the secret
• user=... — read as a specific user (useful when the secret lives in a privileged path)
• forward_pass and echo_verification_code for UX

pam_faillock

Replaces the old pam_tally2. Tracks failed authentication attempts and locks accounts. Two phases:

auth        required      pam_faillock.so preauth silent deny=5 unlock_time=900
auth        [default=die] pam_faillock.so authfail deny=5 unlock_time=900
account     required      pam_faillock.so

preauth runs before the password check and will deny immediately if the account is already locked. authfail runs after a failed attempt and increments the counter. account is where you clear the counter on successful login. If you forget the account line, counters never reset — users accumulate failures forever.

Inspect with faillock --user alice and reset with faillock --user alice --reset.

pam_mkhomedir

Creates a home directory on first login if it does not exist. Essential for LDAP users whose homes are not prepopulated:

session     required      pam_mkhomedir.so skel=/etc/skel umask=0077

oddjobd provides a privileged version (pam_oddjob_mkhomedir) for setups where the PAM service does not run as root.

pam_systemd

Registers the session with logind, sets up XDG_RUNTIME_DIR, applies cgroup placement, and — critically — starts the user’s systemd --user instance. Without it, systemctl --user does not work and your graphical session will feel broken in mysterious ways. It is almost always optional because you do not want a logind hiccup to prevent login.

pam_limits

Applies ulimit values from /etc/security/limits.conf (and /etc/security/limits.d/*). This is where you raise nofile for a user who runs a database, or cap memory for a developer account. Not applied retroactively — users logged in before the config change keep the old limits.

pam_env

Reads /etc/security/pam_env.conf (and optionally ~/.pam_environment) to inject environment variables. ~/.pam_environment is disabled by default in modern distributions because it was a local privilege-escalation risk through variables like LD_PRELOAD.

pam_succeed_if

A Swiss Army knife for conditional logic. Lets you match based on user properties (uid >= 1000, user ingroup wheel, service = sshd, rhost notin 10.0.0.0/8) and return success or ignore accordingly. Combined with bracketed control syntax, you can write “require MFA only for users in the engineers group” or “skip password complexity for service accounts.”

Writing a real MFA stack

Here is a production-quality /etc/pam.d/sshd for requiring TOTP in addition to the password, with a clean fallback for users who haven’t enrolled:

# /etc/pam.d/sshd — MFA enforced

# Fail closed if the account is locked
auth        required      pam_faillock.so preauth silent deny=5 unlock_time=900 even_deny_root

# Local users first, then SSSD
auth        [success=1 default=ignore] pam_succeed_if.so user ingroup mfa-exempt
auth        required      pam_google_authenticator.so nullok forward_pass

# Primary password (local or SSSD) — takes password from OATH prompt
auth        [success=done default=die] pam_unix.so try_first_pass
auth        sufficient    pam_sss.so use_authtok

# Increment failure counter
auth        [default=die] pam_faillock.so authfail deny=5 unlock_time=900

auth        required      pam_deny.so

account     required      pam_faillock.so
account     include       password-auth

password    include       password-auth

session     required      pam_loginuid.so
session     include       password-auth

Several patterns here worth noting:

1. mfa-exempt group: users in that group skip the TOTP prompt (for break-glass accounts).
2. TOTP prompts first: Google Authenticator runs before pam_unix, with forward_pass so the password entered after the TOTP is handed off to pam_unix. The user experience is: “Verification code: _____ Password: _____”.
3. even_deny_root: under faillock, also lock root. If you do not want root lockout, remove this. But if sshd_config has PermitRootLogin no, this is moot.
4. nullok: during rollout, users who have not run google-authenticator yet can still log in. Remove this flag after rollout to enforce MFA for everyone.

And you must also tell OpenSSH to allow keyboard-interactive, since TOTP is a keyboard-interactive challenge:

# /etc/ssh/sshd_config
ChallengeResponseAuthentication yes
UsePAM yes
AuthenticationMethods publickey,keyboard-interactive

That AuthenticationMethods line is the subtle one — it enforces that both SSH key and PAM (which will challenge for TOTP + password) must succeed. Without it, a user with an SSH key bypasses PAM entirely for auth (though PAM’s account and session stacks still run).

Debugging PAM: the tools you need

pamtester

Invaluable. Exercises a PAM stack as if you were a specific service, without needing to run the actual service:

pamtester -v sshd alice authenticate acct_mgmt

You get per-module success/failure printed to stderr, so you can see exactly where in the stack a denial happens. Install it (dnf install pamtester or apt install libpam-dev pamtester) on any host where you’re touching PAM. It is the fastest iteration loop.

Enabling debug logging

Add debug to the module arguments:

auth        sufficient    pam_sss.so forward_pass debug

Modules write to syslog (usually /var/log/auth.log on Debian, /var/log/secure on Red Hat). But the definitive trace — showing which module returned what — comes from enabling the PAM library itself:

echo "*" >> /etc/pam_debug

This is not a widely-advertised feature. The presence of /etc/pam_debug tells libpam to write verbose step-by-step logs to syslog. The contents (* or a list of service names) filter which services log. Remove the file when done — the log volume is heavy.

When authentication fails, check in this order

1. The application’s own logs — sshd will tell you Failed password, Invalid user, User not allowed. If you never see the login attempt in sshd’s log, the problem is not PAM — it’s networking, MaxStartups, or LoginGraceTime.
2. /var/log/secure or /var/log/auth.log — PAM modules log here. Look for the module name in the “module” field.
3. faillock --user <user> — is the account locked?
4. passwd -S <user> or chage -l <user> — is the password expired? Account expired?
5. getent passwd <user> — is the user even visible? If not, the problem is nsswitch.conf or sssd, not PAM.
6. sssctl user-checks <user> — does SSSD itself think the user is valid?
7. pamtester -v <service> <user> authenticate — reproduce from the shell with verbose output.
8. strace -f -o /tmp/pam.trace -e openat,read <the-command> — if all else fails, watch what files PAM is opening. Often reveals a missing config file, a wrong path, or SELinux blocking access.

The SELinux gotcha

If your service runs under a confined SELinux domain and a PAM module wants to open a file the domain can’t read (~/.google_authenticator being owned by the user but readable only inside the user’s home), you get a silent authentication failure. Check ausearch -m avc -ts recent for denials. The fix is either an SELinux boolean (like ssh_keysign) or a module-specific policy update.

Locked yourself out?

It happens. The recovery path on a machine with physical access:

1. Boot to a rescue shell (bootloader → single user, or boot a live USB and chroot).
2. Inspect /etc/pam.d/ — did you commit a file with a syntax error, or introduce a requisite pam_deny.so early in a stack?
3. For a remote server, keep a second SSH session open in a separate window before reloading sshd — the existing session is not affected by config changes, giving you a chance to revert.

Patterns and traps

Do not edit system-auth / common-auth directly on modern distros. Debian/Ubuntu provide pam-auth-update(8) and a fragment-based system under /usr/share/pam-configs/. Red Hat has authselect(8). Hand-editing the merged file means your changes will be reverted next time the system rebuilds the stack.

Service files are searched in /etc/pam.d/ first, then /usr/lib/pam.d/. If you want to override a vendor-provided service, put your copy in /etc/pam.d/ — the system-wide one takes precedence.

pam_tally2 is gone. If you find old blog posts telling you to add it, use pam_faillock instead. It has been the standard since RHEL 8 and Debian 10.

pam_securetty only applies to local consoles and has limited use in modern deployments (containers, cloud VMs). Do not rely on it as your only control.

Kerberos with pam_krb5 is mostly replaced by pam_sss in SSSD setups. Unless you have a specific reason to run pam_krb5 directly, let SSSD handle the Kerberos ticket acquisition.

Order matters for pam_env and pam_limits. They have to run before anything that depends on environment or ulimits — which in practice means near the top of the session stack. If pam_limits runs after pam_exec spawns your shell, the shell already inherits the defaults.

Do not put pam_exec in the auth stack without care. It runs an external program; if that program hangs, the login hangs. Prefer session for any hook where you want to trigger a script on login (set up scratch dirs, message the user, push a metric).

pam_access for host-based restrictions. When you want to say “user alice can only log in from inside 10.0.0.0/8”, /etc/security/access.conf is the place. It is quieter than iptables rules and more auditable than /etc/hosts.allow.

Module development and the conversation interface

If you are writing your own module — rare but not unheard-of for custom hardware tokens or integration with a proprietary MFA service — the API you target is pam_sm_authenticate(), pam_sm_acct_mgmt(), pam_sm_setcred(), etc. Your module communicates with the user through the PAM conversation function, which the application provides. That is why sshd can prompt for a token over a text channel and a GDM greeter can show a nice dialog — the module doesn’t know or care about the UI.

One gotcha: PAM conversations are synchronous. If your module calls out to a remote service, you must set reasonable timeouts. A module that hangs for a minute before returning makes every login that hits it hang for a minute.

Configuration hygiene for teams

If PAM is managed by multiple people, a few practices prevent outages:

• Version control /etc/pam.d/. Every change is a commit. When something breaks, git diff tells you what changed.
• Test on a non-production VM first. Use pamtester to exercise the full stack before deploying.
• Never deploy a PAM change without an open second session. Always keep a verified working shell until the change is confirmed.
• Distinguish “identity provider” from “authentication method.” PAM handles the latter. If you want to change who can log in, that’s sssd.conf, AD, LDAP — not PAM. If you want to change how a login is challenged, PAM is the right place.
• Keep MFA enrollment flows separate from enforcement. Run for a month with nullok logging everyone who would be locked out, audit the list, then remove nullok.
• Document the intent of each service’s /etc/pam.d/ file in a comment header. A PAM stack without intent documentation is a trap for the next operator.

Closing

PAM is unusual in that it gives you line-by-line control over an extremely security-sensitive process, while also offering nearly infinite room to shoot yourself in the foot. Once you internalize the management groups, the control field semantics, and the patterns for MFA and fallback, you will stop fearing it and start treating it like any other config language — one where the syntax is strict, the runtime behavior is debuggable, and the failure modes are well-understood.

The next time an engineer on your team wants to “just turn on 2FA,” you will know that the real question is: which stacks, which users, what fallback, and which modules run in what order. That is what PAM exists to express, and once you can read the stack, you can answer those questions in a few lines of config.