HAProxy Deep Dive: Load Balancing, ACLs, and SSL Termination at Scale
HAProxy is not the flashiest piece of infrastructure in a modern stack. It does not have a slick web UI, a cloud-managed tier, or a Kubernetes operator maintained by a VC-backed startup. What it has is twenty years of production hardening, a configuration language precise enough to express nearly any routing requirement, and performance characteristics that routinely handle millions of requests per second on commodity hardware. Nginx, Traefik, and Envoy each have their place, but when you need a load balancer where behavior is completely predictable and the configuration is auditable line by line, HAProxy is the reference implementation.
This post covers the full configuration model — global settings, defaults, frontends, backends — and builds toward production-ready patterns: ACL-based L7 routing, all load balancing algorithms with when to use each, stick tables for connection and rate limiting without external dependencies, SSL termination with automatic Let’s Encrypt renewal, the runtime API for zero-downtime updates, and Prometheus metrics. Every section includes a working configuration you can drop into a real deployment.
The Configuration Structure
An HAProxy config file has four sections, always in this order:
global — process-level settings: user, limits, logging, SSL
defaults — values inherited by all frontends and backends below
frontend — listening sockets and initial routing decisions
backend — server pools and health checks
A minimal working configuration:
global
log /dev/log local0
log /dev/log local1 notice
chroot /var/lib/haproxy
stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
stats timeout 30s
user haproxy
group haproxy
daemon
# TLS hardening
ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:RSA+AESGCM:RSA+AES:!aNULL:!MD5:!DSS
ssl-default-bind-options ssl-min-ver TLSv1.2 no-tls-tickets
ssl-default-server-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:RSA+AESGCM:RSA+AES:!aNULL:!MD5:!DSS
ssl-default-server-options ssl-min-ver TLSv1.2 no-tls-tickets
defaults
log global
mode http
option httplog
option dontlognull
option forwardfor # pass client IP via X-Forwarded-For
option http-server-close
timeout connect 5s
timeout client 50s
timeout server 50s
timeout http-request 10s
timeout http-keep-alive 2s
timeout queue 30s
errorfile 400 /etc/haproxy/errors/400.http
errorfile 503 /etc/haproxy/errors/503.http
frontend http-in
bind *:80
default_backend web_servers
backend web_servers
balance roundrobin
option httpchk GET /health HTTP/1.1\r\nHost:\ localhost
server web1 192.168.1.10:8080 check inter 5s rise 2 fall 3
server web2 192.168.1.11:8080 check inter 5s rise 2 fall 3
server web3 192.168.1.12:8080 check inter 5s rise 2 fall 3
Validate before reloading — always:
|
|
Frontends: Listening and First-Pass Routing
A frontend binds to one or more address/port combinations and decides which backend handles each request. It can match on host headers, paths, source IPs, methods, or any other aspect of the request.
frontend main
bind *:80
bind *:443 ssl crt /etc/haproxy/certs/
# Redirect HTTP to HTTPS
redirect scheme https code 301 if !{ ssl_fc }
# ACL definitions
acl is_api path_beg /api/
acl is_ws hdr(Upgrade) -i websocket
acl is_admin path_beg /admin/
acl local_net src 10.0.0.0/8 192.168.0.0/16
# Routing decisions — first match wins
use_backend admin_servers if is_admin local_net
http-request deny if is_admin !local_net
use_backend ws_servers if is_ws
use_backend api_servers if is_api
default_backend web_servers
The ACL language is where HAProxy’s power lives. Nearly any attribute of a request can be inspected:
# Match on host header
acl host_app1 hdr(host) -i app1.example.com
acl host_app2 hdr(host) -i app2.example.com
# Match on path prefix
acl path_api path_beg -i /api/
# Match on path extension
acl is_static path_end .css .js .ico .png .jpg .svg .woff2
# Match on HTTP method
acl is_post method POST
acl is_get method GET
# Match on specific header value
acl has_token req.hdr(Authorization) -m found
acl beta_user req.cook(beta) -m found
# Match on source IP or CIDR
acl office_ip src 203.0.113.0/24
# Match on SSL cipher or TLS version
acl tls13 ssl_fc_protocol TLSv1.3
# Combine with AND (consecutive acls) or OR (acl -o)
use_backend api_v2 if is_api tls13
use_backend api_v1 if is_api
# Block and return custom error
http-request deny deny_status 429 if is_post !has_token
Multiple bind lines support SNI-based certificate selection from a directory:
frontend https-in
# HAProxy reads all .pem files in the directory and selects based on SNI
bind *:443 ssl crt /etc/haproxy/certs/
# Or list them explicitly for priority control
bind *:443 ssl crt /etc/haproxy/certs/wildcard.example.com.pem \
crt /etc/haproxy/certs/api.example.com.pem
Backends: Server Pools and Health Checks
Backends define the servers HAProxy forwards to and how it monitors them:
backend api_servers
balance leastconn
option httpchk GET /health
http-check expect status 200
# HTTP/2 to backends
option http-use-htx
server-template api 1-5 192.168.1.1${0}-1${4}:8080 check inter 3s rise 2 fall 2 weight 1
# Explicit servers with individual tuning
server api1 192.168.1.10:8080 check inter 3s rise 2 fall 2
server api2 192.168.1.11:8080 check inter 3s rise 2 fall 2 weight 2 # twice the traffic
server api3 192.168.1.12:8080 check inter 3s rise 2 fall 2 backup # only if others down
# Retry on connection failure, not on successful response
retry-on conn-failure
retries 2
Health check parameters:
| Parameter | Meaning |
|---|---|
inter 3s |
Check every 3 seconds |
fastinter 500ms |
Check every 500ms while server is transitioning (up→down or down→up) |
downinter 10s |
Check every 10s once confirmed down (reduce noise) |
rise 2 |
2 consecutive successes to mark server UP |
fall 3 |
3 consecutive failures to mark server DOWN |
check |
Enable health checking |
check port 9000 |
Health check on a different port than traffic |
For TCP-mode backends (databases, gRPC without HTTP):
backend postgres_primary
mode tcp
balance first # send all to first available — primary/replica pattern
option tcp-check
tcp-check connect
server pg-primary 192.168.1.20:5432 check inter 5s
server pg-replica 192.168.1.21:5432 check inter 5s backup
Load Balancing Algorithms
HAProxy supports more algorithms than most load balancers expose:
# Round Robin — default, equal distribution, good general purpose
balance roundrobin
# Weighted Round Robin — proportional distribution
server web1 192.168.1.10:8080 weight 3 # gets 3x traffic of weight-1 servers
server web2 192.168.1.11:8080 weight 1
# Least Connections — best for long-lived or variable-duration requests
# Sends new requests to whichever server has fewest active connections
balance leastconn
# Source IP hash — client always reaches same server (simple session affinity)
# Consistent across reloads; breaks if server count changes
balance source
# URI hash — same URL always hits same backend (cache locality)
# Useful for caching layers — identical requests cluster on same server
balance uri whole # hash entire URI including query string
balance uri # hash path only
# URL parameter hash — route based on a query parameter value
# E.g., shard by user_id to maintain per-user session affinity
balance url_param user_id
# Header hash — route based on a header value
balance hdr(X-Tenant-ID)
# Random — uniform distribution with no state overhead
# Slightly better than roundrobin for very short connections
balance random
# First — always use the first server with available connection slots
# Good for active/passive failover
balance first
The decision tree:
Short-lived stateless requests?
└─ roundrobin (default)
or random (very high connection rates)
Variable request duration (API, proxy)?
└─ leastconn
Need session affinity?
├─ Can use cookies? → cookie-based persistence (see below)
├─ Client has stable IP? → balance source
└─ URL identifies session? → balance uri or url_param
Cache locality matters?
└─ balance uri
Multi-tenant, route by tenant?
└─ balance hdr(X-Tenant-ID)
Cookie-based persistence (stickiness) is more robust than IP-based because it survives NAT and CGNAT:
backend web_servers
balance roundrobin
cookie SERVERID insert indirect nocache
server web1 192.168.1.10:8080 check cookie web1
server web2 192.168.1.11:8080 check cookie web2
server web3 192.168.1.12:8080 check cookie web3
HAProxy inserts a SERVERID cookie on the first response. Subsequent requests with that cookie are routed to the same backend server. indirect means HAProxy strips the cookie before forwarding to the backend (backend never sees it). nocache prevents proxies from caching responses that contain the cookie.
Stick Tables: Rate Limiting Without External Dependencies
Stick tables are in-memory key-value stores built into HAProxy. They track per-source metrics — connection count, request rate, byte rate, error count — and can trigger ACL decisions. No Redis, no external rate limiter required.
backend st_src_tracking
stick-table type ip size 1m expire 30s store conn_cur,conn_rate(10s),http_req_rate(10s),http_err_rate(10s)
frontend main
bind *:80 *:443 ssl crt /etc/haproxy/certs/
# Track every request's source IP in the stick table
tcp-request connection track-sc0 src table st_src_tracking
# --- Connection limits ---
# Block if more than 50 concurrent connections from one IP
acl too_many_conn sc_conn_cur(0) gt 50
tcp-request connection reject if too_many_conn
# --- Request rate limits ---
# Block if more than 100 HTTP requests in the last 10 seconds from one IP
acl req_rate_abuse sc_http_req_rate(0) gt 100
http-request deny deny_status 429 if req_rate_abuse
# --- Error rate limits ---
# Block clients generating lots of 4xx errors (scanners, bad bots)
acl err_rate_high sc_http_err_rate(0) gt 20
http-request deny deny_status 429 if err_rate_high
default_backend web_servers
The stick table type ip size 1m expire 30s means:
- Keyed on source IP address
- Holds up to 1 million entries
- Entries expire after 30 seconds of inactivity
store conn_cur,conn_rate(10s),http_req_rate(10s) declares which counters to maintain per key. Multiple counters compound the memory use — 1M entries with four counters at ~150 bytes each is about 150MB. Size your table to your expected unique IP count.
For more granular rate limiting — per authenticated user rather than per IP, or per API endpoint:
# Rate limit per Authorization header value (API key abuse prevention)
backend st_api_key_tracking
stick-table type string len 64 size 100k expire 60s store http_req_rate(60s)
frontend api_gateway
bind *:443 ssl crt /etc/haproxy/certs/
# Track by the API key (first 64 chars of Authorization header)
http-request track-sc0 req.hdr(Authorization),lower,word(2,' '),substr(0,64) \
table st_api_key_tracking if { req.hdr(Authorization) -m found }
# 1000 requests/minute per API key
acl api_key_rate_exceeded sc_http_req_rate(0) gt 1000
http-request return status 429 \
hdr Retry-After "60" \
content-type "application/json" \
string '{"error":"rate limit exceeded"}' \
if api_key_rate_exceeded
default_backend api_servers
Stick tables also enable soft rate limiting with response headers rather than hard blocks:
# Add rate limit headers even when not blocking (RFC 6585)
http-response set-header X-RateLimit-Remaining %[sc_http_req_rate(0),sub(100)] \
if { sc_http_req_rate(0) lt 100 }
http-response set-header X-RateLimit-Remaining 0 \
if { sc_http_req_rate(0) ge 100 }
SSL Termination and Certificate Management
HAProxy terminates TLS on the frontend. Certificates are PEM files combining the certificate, any intermediate chain, and the private key:
|
|
frontend https
bind *:443 ssl crt /etc/haproxy/certs/ alpn h2,http/1.1
# HSTS — tell browsers to always use HTTPS for 1 year
http-response set-header Strict-Transport-Security "max-age=31536000; includeSubDomains; preload"
# Security headers
http-response set-header X-Frame-Options DENY
http-response set-header X-Content-Type-Options nosniff
http-response set-header Referrer-Policy strict-origin-when-cross-origin
default_backend web_servers
alpn h2,http/1.1 enables HTTP/2 negotiation via ALPN. HAProxy speaks H2 with clients and can either downgrade to HTTP/1.1 to backends or speak H2 to backends that support it.
Automatic Let’s Encrypt Renewal
HAProxy can serve ACME HTTP-01 challenges directly, enabling automated certificate renewal with no downtime:
|
|
|
|
# Serve ACME challenges from the webroot
frontend http-in
bind *:80
acl is_acme_challenge path_beg /.well-known/acme-challenge/
use_backend acme_challenge if is_acme_challenge
redirect scheme https code 301 if !is_acme_challenge
backend acme_challenge
mode http
server local 127.0.0.1:8888
# Or serve files directly with a local nginx/python server on 8888
Add the renewal script to cron:
|
|
The Runtime API
HAProxy’s Runtime API (the admin socket) allows you to inspect and modify live state without reloading the config file. This is the mechanism for dynamic server management, zero-downtime updates, and certificate hot-swapping.
|
|
The runtime API is the mechanism Kubernetes-style deployments use for drain-before-deregister patterns — drain the server in HAProxy, wait for zero connections, then stop the container.
Prometheus Metrics and Observability
HAProxy 2.0+ includes a native Prometheus exporter endpoint, compiled in by default in most distributions:
frontend stats
bind *:8404
stats enable
stats uri /stats
stats refresh 10s
stats auth admin:changeme # basic auth for the HTML dashboard
stats show-legends
stats show-node
# Prometheus endpoint — no auth by default, restrict by source IP
http-request use-service prometheus-exporter if { path /metrics }
acl prometheus_allowed src 10.0.0.0/8 127.0.0.0/8
http-request deny if { path /metrics } !prometheus_allowed
Key metrics to alert on:
# Backend availability
haproxy_backend_status{proxy="web_servers"} == 0 → all backends down
# Queue depth spike (backends can't keep up)
haproxy_backend_queue_length_average{proxy="api_servers"} > 50
# Session rate
haproxy_frontend_current_sessions{proxy="main"}
# 5xx rate per backend
haproxy_server_http_responses_total{proxy="api_servers",code="5xx"}
# Response time percentiles (requires stats-http-request-log)
haproxy_backend_response_time_average_seconds{proxy="web_servers"}
# Health check failures (server flapping)
haproxy_server_check_failures_total{proxy="web_servers",server="web1"}
Prometheus scrape config:
|
|
For structured log parsing, configure HAProxy to emit JSON logs:
global
log /dev/log local0
defaults
log global
option httplog
# JSON log format — easier to parse in Loki/Elasticsearch
log-format '{"time":"%t","frontend":"%f","backend":"%b","server":"%s","tsc":"%tsc","Tt":"%Tt","Tq":"%Tq","Tr":"%Tr","bytes_read":%B,"termination_state":"%ts","actconn":%ac,"feconn":%fc,"beconn":%bc,"srv_conn":%sc,"retries":%rc,"srv_queue":%sq,"backend_queue":%bq,"method":"%HM","uri":"%HU","status":%ST,"src":"%ci"}'
A Complete Production Config
Putting it together — HTTPS-only, ACL routing to multiple backends, rate limiting, health checks, and metrics:
global
log /dev/log local0
chroot /var/lib/haproxy
stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
stats timeout 30s
user haproxy
group haproxy
daemon
maxconn 50000
ssl-default-bind-options ssl-min-ver TLSv1.2 no-tls-tickets
ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384
defaults
log global
mode http
option httplog
option dontlognull
option forwardfor
option http-server-close
option redispatch # retry on different server if backend fails
timeout connect 5s
timeout client 30s
timeout server 30s
timeout http-request 10s
timeout http-keep-alive 2s
timeout queue 15s
retries 3
#---------------------------------------------------------------------
# Rate-limiting stick table
#---------------------------------------------------------------------
backend st_rate_limit
stick-table type ip size 500k expire 60s \
store conn_cur,conn_rate(10s),http_req_rate(60s),http_err_rate(10s)
#---------------------------------------------------------------------
# HTTP redirect to HTTPS + ACME challenge passthrough
#---------------------------------------------------------------------
frontend http-in
bind *:80
acl is_acme path_beg /.well-known/acme-challenge/
use_backend acme_challenge if is_acme
redirect scheme https code 301 if !is_acme
#---------------------------------------------------------------------
# Main HTTPS frontend
#---------------------------------------------------------------------
frontend https-in
bind *:443 ssl crt /etc/haproxy/certs/ alpn h2,http/1.1
# Rate limiting
tcp-request connection track-sc0 src table st_rate_limit
acl conn_abuse sc_conn_cur(0) gt 100
acl req_abuse sc_http_req_rate(0) gt 300
acl err_abuse sc_http_err_rate(0) gt 50
http-request deny deny_status 429 if req_abuse or err_abuse
tcp-request connection reject if conn_abuse
# Security headers
http-response set-header Strict-Transport-Security "max-age=31536000; includeSubDomains"
http-response set-header X-Frame-Options DENY
http-response set-header X-Content-Type-Options nosniff
# ACL definitions
acl host_api hdr(host) -i api.example.com
acl host_app hdr(host) -i app.example.com
acl path_api path_beg /api/
acl is_ws hdr(Upgrade) -i websocket
acl is_static path_end .css .js .ico .png .jpg .svg .woff2 .gz
# Routing
use_backend api_servers if host_api
use_backend api_servers if host_app path_api
use_backend ws_servers if is_ws
use_backend static_servers if is_static
default_backend app_servers
#---------------------------------------------------------------------
# Backends
#---------------------------------------------------------------------
backend api_servers
balance leastconn
option httpchk GET /api/health
http-check expect status 200
server api1 10.0.1.10:8080 check inter 3s rise 2 fall 3
server api2 10.0.1.11:8080 check inter 3s rise 2 fall 3
server api3 10.0.1.12:8080 check inter 3s rise 2 fall 3
backend app_servers
balance roundrobin
cookie SRVID insert indirect nocache
option httpchk GET /health
http-check expect status 200
server app1 10.0.1.20:3000 check inter 3s rise 2 fall 3 cookie app1
server app2 10.0.1.21:3000 check inter 3s rise 2 fall 3 cookie app2
backend ws_servers
balance source
timeout tunnel 3600s # long timeout for WebSocket connections
option http-server-close
server ws1 10.0.1.30:8090 check inter 5s
server ws2 10.0.1.31:8090 check inter 5s
backend static_servers
balance uri
option httpchk
server cdn1 10.0.1.40:80 check inter 10s
server cdn2 10.0.1.41:80 check inter 10s
backend acme_challenge
server local 127.0.0.1:8888 check
#---------------------------------------------------------------------
# Observability
#---------------------------------------------------------------------
frontend stats
bind *:8404
stats enable
stats uri /stats
stats auth admin:${HAPROXY_STATS_PASSWORD}
http-request use-service prometheus-exporter if { path /metrics }
acl internal src 10.0.0.0/8 127.0.0.0/8
http-request deny if !internal
Common Operational Patterns
Zero-downtime deployment: drain target server via runtime API, wait for sc_conn_cur to reach 0, deploy, re-enable. Script this into your CI pipeline:
|
|
Blue/green switch: maintain two backends (blue and green), switch the default_backend via a config reload, or use a map file to route by hostname dynamically without any reload.
Canary routing: use weighted servers to send 5% of traffic to a new version:
backend app_servers
balance roundrobin
server stable 10.0.1.20:3000 check weight 95
server canary 10.0.1.21:3000 check weight 5
Adjust weights via the runtime API during the rollout without a config reload.
Health-aware circuit breaking: combine fall/rise with http-check expect to distinguish between server down (TCP failure) and app broken (unexpected HTTP status) — HAProxy marks them down in both cases but the logging distinguishes them for your alerting.
HAProxy’s configuration language can seem verbose compared to Kubernetes Ingress YAML or Traefik’s automatic service discovery. The trade is explicitness and auditability: every routing decision is a named, inspectable ACL. Every server’s health state is visible in the stats page. Every operational change is a documented command against the runtime API. For infrastructure that is hard to reason about when it breaks, that explicitness pays for itself.
Comments