LUNAROPS · OPERATIONAL UPLINK 100% UPTIME 1,247d POSTS 893 JEFF.MOON@LUNAROPS.DEV UTC --:--:--

Vault Beyond Secrets: PKI, SSH CA, and Dynamic DB Credentials

17 min read Security
vaulthashicorppkissh-casecretssecuritytransit
Contents

Most people who use Vault use it as a key-value store with authentication. A Python app reads secret/data/app/config, gets back a JSON blob, parses out a DB password, and connects. That’s fine, and that’s what most documentation and tutorials focus on. But Vault’s KV secrets engine is the least interesting thing about it. The stuff that separates Vault from, say, writing secrets to S3 with KMS encryption and calling it a day, is the dynamic secrets engines — the ones that generate credentials on demand rather than store them.

A full-blown Vault deployment can issue short-lived X.509 certificates for mTLS, sign SSH certificates that expire in two hours, generate database credentials that exist only for the lifetime of a batch job, encrypt arbitrary data as a service, and mint cloud IAM credentials that vanish automatically. Using Vault this way changes the security posture fundamentally: instead of long-lived secrets distributed to many machines, you have ephemeral credentials issued to authenticated identities that automatically expire. Steal a cert? It expires in an hour. Steal an SSH key? It worked for the ten-minute window of the user’s session. Steal a database password? It was already revoked.

This post walks the secrets engines I’ve found most transformative in production: PKI (internal CA), SSH CA (signed SSH certificates), database secrets (dynamic DB users), and transit (encryption-as-a-service). These aren’t the “getting started with Vault” features — they’re the payoff for having Vault in your life.

Very brief architecture refresher

A Vault deployment is:

  • A cluster of Vault servers (typically 3 or 5, Raft-based or Consul-backed for HA).
  • A storage backend (integrated Raft is standard now; Consul is legacy but common).
  • Auth methods — how clients authenticate (Kubernetes SA tokens, AppRole, LDAP, OIDC, JWT, AWS IAM, GitHub, token, etc.).
  • Secrets engines — where secrets come from (KV, PKI, SSH, database, AWS, GCP, transit, etc.).
  • Policies — HCL documents granting path-scoped permissions.

The flow: authenticate (get a token) → the token has attached policies → policies grant access to specific paths on specific engines. Every API call to Vault is <engine mount path>/<operation>.

The secret engines live at mount points; you can mount the same engine multiple times. vault secrets enable -path=pki-internal pki mounts a PKI engine at pki-internal/. Your production and dev CAs can be separate mounts.

PKI: your own internal certificate authority

Let’s start with the one that pays back the fastest. Running an internal CA used to mean openssl, a binder full of sticky notes, and a handful of manually-issued certs with notAfter dates in 2029. Vault replaces all of that with an API that issues and revokes certs, enforces constraints, and rotates automatically.

Basic setup — a two-tier CA

Root CA — long-lived (10 years), issues nothing except intermediate certs, kept offline or at least restricted.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

vault secrets enable -path=pki-root pki
vault secrets tune -max-lease-ttl=87600h pki-root

vault write -field=certificate pki-root/root/generate/internal \
  common_name="Internal Root CA" \
  issuer_name="root-2026" \
  ttl=87600h > root_ca.crt

vault write pki-root/config/urls \
  issuing_certificates="https://vault.internal/v1/pki-root/ca" \
  crl_distribution_points="https://vault.internal/v1/pki-root/crl"

Intermediate CA — signed by the root, does the actual work of issuing certs. Shorter TTL (say 3 years), can be revoked without replacing the root.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

vault secrets enable -path=pki-intermediate pki
vault secrets tune -max-lease-ttl=26280h pki-intermediate

# Generate a CSR
vault write -format=json pki-intermediate/intermediate/generate/internal \
  common_name="Internal Intermediate CA" \
  issuer_name="intermediate-2026" \
  | jq -r '.data.csr' > intermediate.csr

# Root signs the CSR
vault write -format=json pki-root/root/sign-intermediate \
  issuer_ref="root-2026" \
  csr=@intermediate.csr \
  format=pem_bundle \
  ttl="26280h" \
  | jq -r '.data.certificate' > intermediate.crt

# Give the signed cert back to the intermediate engine
vault write pki-intermediate/intermediate/set-signed certificate=@intermediate.crt

Now configure a role — the template of what certs the intermediate will issue:

1
2
3
4
5
6

vault write pki-intermediate/roles/internal-services \
  allowed_domains="internal.corp" \
  allow_subdomains=true \
  max_ttl="720h" \
  key_type="rsa" \
  key_bits="2048"

Finally, issue a cert:

1
2
3

vault write pki-intermediate/issue/internal-services \
  common_name="api.internal.corp" \
  ttl="168h"

Response includes certificate, private_key, ca_chain, serial_number. The certificate is valid for 7 days, trusted by anything that trusts the root CA.

The workflow that pays back

Short-lived certs are the point. A 7-day cert means every workload re-issues weekly. The renewal happens automatically via Vault Agent, consul-template, or the relevant platform integration (cert-manager on Kubernetes, via the vault-issuer). Expired certs that aren’t renewed stop working — which is exactly what you want, because an attacker holding a leaked cert has at most a week to use it.

Compare to the classic “static cert valid for 3 years, auto-renewed by a script nobody owns”: at some point, the script breaks, the cert expires, production goes down on a Sunday, and someone has to manually issue a new one in a panic. With Vault PKI + short TTLs, expirations are continuous — if renewal breaks, it breaks immediately and loudly, not 2.5 years later.

Root rotation

Root CAs need to be rotatable. Vault supports multiple issuers per mount:

1
2
3
4

vault write pki-root/root/generate/internal \
  common_name="Internal Root CA 2027" \
  issuer_name="root-2027" \
  ttl=87600h

The root mount now has two issuers: root-2026 (old) and root-2027 (new). Configure clients to trust both during the transition, shift issuance to root-2027, let root-2026 age out, then remove it. Zero-downtime root rotation.

Revocation

vault write pki-intermediate/revoke serial_number=xx:xx:... revokes a specific cert. Vault publishes a CRL at /pki-intermediate/crl that clients can fetch. For busier shops, configure OCSP by enabling the OCSP Responder (Vault 1.12+).

cert-manager integration

On Kubernetes, cert-manager with the vault issuer means every Certificate CRD is fulfilled by Vault. A service annotates its Ingress with cert-manager.io/cluster-issuer: vault-issuer and gets a cert auto-renewed from Vault PKI. Combined with trust-manager to distribute your root CA into every namespace, you have internal mTLS that’s entirely automated.

Constraint patterns

The allowed_domains, allow_any_name, allow_localhost, server_flag, client_flag, ext_key_usage, no_store, require_cn fields on a role constrain what the role can issue. Two roles worth having:

  • server certs — allowed_domains=internal.corp, allow_subdomains=true, server_flag=true, client_flag=false, max_ttl=168h
  • client certs — allow_any_name=true with key_usage="client-auth", very short TTL (1-24h)

Then a Kubernetes pod authenticating to a database can request a fresh client cert per hour, the DB trusts the internal CA, and there are no long-lived DB credentials floating around.

SSH CA: signed SSH certificates

SSH keys are a disaster in most organizations. Engineers generate keys, put them on servers, never remove them when they leave, and the authorized_keys file becomes an archaeological record of everyone who ever had access. SSH certificates fix this.

An SSH certificate is a signed statement: “the bearer of this key may log in as user X, with these principals, until this time.” The server trusts any cert signed by the CA’s public key — TrustedUserCAKeys in sshd_config. No more authorized_keys per user; one CA trust, N users.

Setup

Enable the SSH engine:

1
2
3

vault secrets enable -path=ssh-ca ssh
vault write ssh-ca/config/ca generate_signing_key=true
vault read -field=public_key ssh-ca/config/ca > trusted-user-ca.pub

Distribute trusted-user-ca.pub to every server; add to /etc/ssh/sshd_config:

TrustedUserCAKeys /etc/ssh/trusted-user-ca.pub

Reload sshd. Now the server will accept any SSH cert signed by the Vault CA.

Define a role — what kinds of certs Vault will issue:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

vault write ssh-ca/roles/engineers -<<EOF
{
  "key_type": "ca",
  "allow_user_certificates": true,
  "default_user": "ubuntu",
  "allowed_users": "ubuntu,ec2-user,admin",
  "allowed_extensions": "permit-pty,permit-port-forwarding",
  "default_extensions": {"permit-pty": ""},
  "ttl": "8h",
  "max_ttl": "24h",
  "key_id_format": "{{identity.entity.name}}"
}
EOF

Now an engineer who’s authenticated to Vault can sign their own SSH public key:

1
2

vault write -field=signed_key ssh-ca/sign/engineers \
  public_key=@$HOME/.ssh/id_ed25519.pub > $HOME/.ssh/id_ed25519-cert.pub

SSH automatically uses the cert alongside the key: ssh -i ~/.ssh/id_ed25519 ubuntu@server. Server checks the cert, verifies the Vault CA signature, allows login.

The workflow

Wrap ssh in a tiny script:

1
2
3
4
5
6

#!/bin/bash
vault write -field=signed_key ssh-ca/sign/engineers \
  public_key=@$HOME/.ssh/id_ed25519.pub \
  valid_principals="ubuntu,ec2-user" \
  > $HOME/.ssh/id_ed25519-cert.pub
exec ssh "$@"

Engineer runs ssh-v server.internal, Vault signs a cert valid for 8 hours, ssh proceeds. Next day, they need to re-sign. Laptop stolen? The cert expires; the attacker can’t use the key beyond that.

The key_id_format: {{identity.entity.name}} puts the engineer’s Vault identity in the cert. SSH’s auth log will show “cert ID alice@corp used” — you get audit trail of who logged in, not just “someone with Ubuntu’s key.”

Host certs too

You can also sign host certificates — the server’s ed25519 host key, so clients can trust server identity without TOFU (trust-on-first-use). generate_signing_key on a separate host engine, sign host keys at boot, distribute the host CA to clients. Then known_hosts gets a @cert-authority *.internal ssh-ed25519 AAAA... line, and any server signed by that CA is trusted without individual fingerprints.

Key insights

  • SSH CA is a complete replacement for authorized_keys. You never manage per-user keys on servers again.
  • Revocation is by expiration: don’t issue long-lived certs and you don’t need a revocation mechanism. For belt-and-suspenders, Vault also supports OpenSSH revocation lists.
  • Principals restrict as who the cert can log in. A cert with valid_principals=deploy can only log in as the deploy user, regardless of the key.
  • Combine with Teleport if you want richer session recording, or use plain Vault + SSH CA + journald if you don’t.

Dynamic database credentials

The idea: your app doesn’t have a DB password — it has a Vault token. It asks Vault, “give me credentials for the reporting-db role,” Vault creates a temporary DB user, returns username+password, and the credentials are revoked automatically after the lease expires.

Setup

Enable the database engine:

1

vault secrets enable database

Configure a connection to your database. Vault will use a privileged admin account to create and drop temporary users:

1
2
3
4
5
6
7

vault write database/config/reporting-db \
  plugin_name="postgresql-database-plugin" \
  allowed_roles="reporting-ro,reporting-rw" \
  connection_url="postgresql://{{username}}:{{password}}@pg.internal:5432/reporting" \
  username="vault_admin" \
  password="redacted" \
  password_authentication="scram-sha-256"

Define a role that issues credentials:

1
2
3
4
5
6

vault write database/roles/reporting-ro \
  db_name="reporting-db" \
  creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; GRANT pg_read_all_data TO \"{{name}}\";" \
  revocation_statements="DROP ROLE \"{{name}}\";" \
  default_ttl="1h" \
  max_ttl="24h"

Read the role to get credentials:

1
2
3
4
5
6
7
8

$ vault read database/creds/reporting-ro
Key                Value
---                -----
lease_id           database/creds/reporting-ro/abc123
lease_duration     1h
lease_renewable    true
password           aJ...randomstring
username           v-token-reporting-ro-abc

Behind the scenes, Vault ran:

1
2

CREATE ROLE "v-token-reporting-ro-abc" WITH LOGIN PASSWORD '...' VALID UNTIL '2026-04-19T16:00:00Z';
GRANT pg_read_all_data TO "v-token-reporting-ro-abc";

The user is real, logs into Postgres, has pg_read_all_data, and will be dropped in 1 hour when Vault’s lease ends (or when Vault calls the revocation statement on lease revoke).

The value proposition

  • No long-lived DB password in Kubernetes secrets or env files. The app has a Vault token (scoped to this role); it gets fresh creds per-use or per-hour.
  • Audit trail. Every DB action is tied to a specific v-token-xxx user, which Vault maps to the app identity that requested it.
  • Instant revocation. Deployed a bad binary? Revoke all leases for that role; every active user is dropped simultaneously.
  • No secret to rotate. There’s no “the DB password changed” scramble — the password literally expires.

Integration patterns

  • Vault Agent — a sidecar running next to the app that acquires credentials, writes them to a file, and renews them. App reads the file; Vault Agent handles lifecycle.
  • Vault CSI provider — on Kubernetes, mount Vault secrets as files/env vars into pods.
  • Language SDKs — hvac (Python), vault/api (Go). App code calls vault.read('database/creds/x') when it needs to connect.
  • Database proxy — Teleport, Strongdm, or PgBouncer with Vault integration, where the connection itself gets its credentials dynamically.

The root rotation step

The privileged admin account Vault uses to create/drop users is itself a secret. Rotate it immediately after configuration so that only Vault knows it:

1

vault write -force database/rotate-root/reporting-db

Now no human knows the admin password; only Vault can use it. That’s the final piece — the only long-lived DB credential is Vault’s own internal one, inside the storage backend.

Works with more than Postgres

Database plugins Vault supports out of the box: PostgreSQL, MySQL, MariaDB, MongoDB, MSSQL, Oracle, Redis, Cassandra, InfluxDB, Elasticsearch, Snowflake, Redshift. For each, you define creation/revocation statements in the database’s dialect; the pattern is identical.

Transit: encryption-as-a-service

Transit is an under-appreciated gem. Vault exposes encrypt/decrypt/sign/verify/hmac operations over an HTTP API. Your app never sees the key material.

1
2

vault secrets enable -path=transit transit
vault write -f transit/keys/email-pii type="aes256-gcm96"

Encrypt some data:

1
2
3
4
5

$ vault write transit/encrypt/email-pii plaintext=$(echo -n "user@example.com" | base64)
Key            Value
---            -----
ciphertext     vault:v1:abcXYZencrypted...
key_version    1

Decrypt:

1
2
3
4

$ vault write transit/decrypt/email-pii ciphertext="vault:v1:abcXYZencrypted..."
Key          Value
---          -----
plaintext    dXNlckBleGFtcGxlLmNvbQ==

The use cases:

  • Field-level encryption in a database. Store vault:v1:... in your users table; decrypt on read. Key material never reaches the DB or the app’s memory for longer than the decrypt response.
  • Application-layer encryption for multi-tenant data. One key per tenant, rotatable independently.
  • Signing/verification. RSA or ECDSA keys for signing webhooks, JWTs, or artifacts.
  • HMAC generation for deterministic tokens (tokens that don’t store secrets but can be verified).

Transit supports key rotation without re-encrypting data:

1

vault write -f transit/keys/email-pii/rotate

New writes use the new key version. Old ciphertexts still decrypt (they carry v1 in the prefix). When you want to consolidate:

1
2

vault write transit/rewrap/email-pii ciphertext="vault:v1:..."
# returns vault:v2:...

Rewrap re-encrypts without decrypting; the new ciphertext is the same plaintext encrypted under the new key version. Over a period of weeks, rewrap all your data, then min_decryption_version=2 to disable v1.

Convergent encryption

Sometimes you want the same plaintext to always produce the same ciphertext — e.g., to enable equality searches on encrypted data:

1

vault write transit/keys/tenant-id type="aes256-gcm96" derived=true convergent_encryption=true

Encryption with the same context yields the same output; you can index and query encrypted fields for equality without decrypting. Trade-off: weakens semantic security (attacker who knows a plaintext can see if it’s in your DB). Use judiciously.

Other engines worth knowing

Briefly, because they deserve their own posts:

  • AWS secrets engine — dynamically issued IAM credentials. App asks Vault for a role, Vault calls AssumeRole, returns temporary AWS credentials. Same pattern with GCP, Azure.
  • Kubernetes secrets engine — dynamically issued service account tokens with scoped access to other clusters.
  • LDAP / AD secrets — dynamic AD user password rotation, static-role credential checkout.
  • KMIP — Vault as a KMIP server for legacy encryption clients.
  • Key Management — Vault as a key manager that synchronizes keys to AWS KMS, Azure Key Vault, GCP KMS for envelope encryption at cloud providers.

Auth: how clients get tokens

A secrets engine is only useful if clients can authenticate to it. The auth methods you’ll reach for:

  • Kubernetes — service account tokens. The canonical way to authenticate pods.
  • AppRole — role_id + secret_id pair. Older but still useful where the application can’t use a richer method.
  • JWT / OIDC — federate with your identity provider. Humans log in via OIDC; CI pipelines present signed JWTs from GitHub Actions / GitLab / Buildkite.
  • AWS IAM — AWS workloads (EC2, ECS, EKS, Lambda) authenticate via IAM signatures.
  • Cert auth — mTLS-based, useful for bootstrapping from PKI-issued client certs.
  • TLS cert + Token role chain — bootstrap a new service by pre-issuing a short-lived bootstrap token during deployment.

The pattern I find cleanest: OIDC for humans, Kubernetes/AWS-IAM for workloads, JWT for CI. Each class of caller has its own auth method; policies attached to those auth methods scope what each can see.

Policies

A policy is an HCL document mapping paths to capabilities:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

path "pki-intermediate/issue/internal-services" {
  capabilities = ["create", "update"]
}

path "database/creds/reporting-ro" {
  capabilities = ["read"]
}

path "ssh-ca/sign/engineers" {
  capabilities = ["create", "update"]
}

Attach the policy to an auth role (Kubernetes service account, OIDC group, etc.). Tokens issued through that auth inherit the policy.

Template variables in policies let you scope per-identity:

1
2
3

path "secret/data/apps/{{identity.entity.aliases.auth_kubernetes_abc.metadata.service_account_namespace}}/*" {
  capabilities = ["read"]
}

Each Kubernetes namespace gets its own secret tree, enforced by policy templating. No per-namespace policy to write; one template covers all.

Operational considerations

Unseal

Vault starts sealed. Unsealing requires threshold key shares (keyShares, keyThreshold). For production:

  • Auto-unseal using a cloud KMS or HSM. Vault uses the KMS to seal/unseal itself at startup. No human-held keys, no typing Shamir shares at 3am.
  • Transit unseal — use one Vault cluster to unseal another (useful for bootstrapping).
  • HSM unseal for compliance-heavy environments.

Raft cluster

Integrated Raft (now the recommended backend) means Vault stores its state in-process. Simple to operate; run 3 or 5 servers. Backup:

1

vault operator raft snapshot save vault-snapshot.snap

Schedule regularly. Store snapshots encrypted (because they contain everything — auth material, PKI keys, leases).

Monitoring

  • vault status — quick health check.
  • /v1/sys/metrics?format=prometheus — Prometheus metrics endpoint. Alert on vault_core_unsealed == 0, vault_raft_peers mismatches, vault_token_lookup latency spikes.
  • Audit log — enable at least one audit device (vault audit enable file file_path=/var/log/vault_audit.log). Every request and response is logged. Ship to SIEM. Do not leave audit disabled — it’s required for forensic analysis and for many compliance frameworks.

HA and disaster recovery

  • HA mode — Raft gives this inherently; any server can serve reads, the leader serves writes.
  • Performance standbys — Enterprise feature; non-leader servers serve reads to reduce leader load.
  • Disaster Recovery Replication — Enterprise feature; warm-standby cluster in another region.
  • Performance Replication — Enterprise; multi-region active-active.

For OSS Vault, the replication features aren’t available, but Raft snapshots + scripted restore to a standby cluster cover most DR needs.

Anti-patterns I’ve seen

  • Vault as a KV-only store. Fine for starters, but you’re missing 90% of the value. The PKI/SSH/DB engines are the payoff.
  • Long-lived tokens. If an app has a token that lives for a year, you haven’t really solved the secret problem. Use auth methods with short-lived tokens and frequent renewal.
  • vault write -force everywhere. The force flag bypasses some safety checks; learn when it’s required vs when you should be more deliberate.
  • Unrotated root credentials. After configuring a DB engine, always run database/rotate-root. Otherwise, the admin password is still floating around in your bootstrap process.
  • Missing audit log. Enabling audit is not optional. Make it one of the first post-install steps.
  • Vault as single point of failure without HA. A single-node Vault is a single-node critical dependency. Three-node Raft cluster is the minimum for anything load-bearing.
  • Policies that grant too much. capabilities = ["sudo"] on anything is rarely correct; scope tightly and prefer least privilege.

An opinionated rollout order

If you’re standing up Vault for real, an order that works:

  1. Raft HA cluster (3 servers), auto-unseal via KMS, audit log to a file shipped to your SIEM.
  2. OIDC auth for humans via your IdP (Okta/Google/etc.). Map group claims to Vault policies.
  3. Kubernetes auth for workloads. One role per service account.
  4. KV v2 for static secrets (API keys from third parties you don’t control).
  5. PKI with root + intermediate; start issuing internal service certs via cert-manager or Vault Agent.
  6. Database engine for your most sensitive databases; rotate root creds immediately.
  7. SSH CA; trust-distribute the CA pub key; migrate ops team to signed certs.
  8. Transit for any field-level encryption needs.
  9. AWS/GCP secrets engines once you’re comfortable with dynamic pattern.

Each step delivers value on its own. You don’t need to do all of them to get benefit — the PKI and Database engines alone justify Vault’s operational cost for most organizations.

Closing

Vault’s most powerful feature is that it makes dynamic the default. Certificates expire in days, not years. SSH sessions get a fresh cert every morning. Databases hand out and revoke users instead of holding static passwords. Cloud credentials exist for exactly as long as a workload runs.

That changes the security posture fundamentally, because the thing every attacker relies on — long-lived secrets that, once stolen, keep working — isn’t there to steal. What’s there to steal is either a short-lived credential that expires soon, or Vault’s internal state, which you’ve locked down with auto-unseal and audit logging and threshold-sharded root keys.

Getting there takes work. The auth method story, the policy templating, the rotation strategies — these take time to internalize. But each step compounds. And once you’re there, “did that leaked credential matter?” starts having an answer that’s usually “probably not, it expired.”

Comments