HAProxy Deep Dive: Load Balancing, Health Checks, ACLs, and Production Tuning

HAProxy has been the gold standard for software load balancing and proxying for over two decades. It powers high-traffic sites at GitHub, Reddit, Stack Overflow, and Airbnb. While cloud-native alternatives like Envoy and Nginx proliferate, HAProxy maintains a unique position: a single-purpose tool that does load balancing and proxying better than almost anything else, with a configuration language expressive enough to handle nearly any routing requirement, and performance characteristics that remain exceptional even at millions of requests per second.

This guide goes deep on the features that matter most in production: the full range of load balancing algorithms, health check strategies, ACL-based routing, SSL/TLS termination, connection and rate limiting, and the observability tools that let you see exactly what HAProxy is doing.

Core Concepts

HAProxy processes traffic through a pipeline: frontend → backend → server.

• Frontend: Where traffic enters. Binds to an IP/port, applies ACLs, and routes to a backend.
• Backend: A pool of servers. Defines load balancing algorithm, health checks, and server-specific settings.
• Server: An individual upstream — IP, port, weight, health check parameters.
• Listen: A combined frontend+backend shortcut for simple TCP proxies.

Client → [Frontend: bind :443] → [ACL routing] → [Backend: web-servers]
                                                          ↓
                                               ┌─ server: web-01:8080
                                               ├─ server: web-02:8080
                                               └─ server: web-03:8080

The global section sets process-level settings (logging, limits, SSL). The defaults section sets values inherited by all frontends and backends unless overridden.

The Configuration File

/etc/haproxy/haproxy.cfg

A well-structured production config:

#-----------------------------------------------------------------------
# GLOBAL — process-level settings
#-----------------------------------------------------------------------
global
    log         /dev/log local0          # syslog socket for logging
    log         /dev/log local1 notice   # notice-level to separate facility
    chroot      /var/lib/haproxy         # security: jail to this directory
    stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
    stats timeout 30s
    user        haproxy
    group       haproxy
    daemon

    # SSL/TLS hardening
    ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384
    ssl-default-bind-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
    ssl-default-bind-options prefer-client-ciphers no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets

    ssl-default-server-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384
    ssl-default-server-options no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets

    # Performance tuning
    maxconn     100000                   # global max simultaneous connections
    nbthread    4                        # use 4 threads (match CPU cores)
    cpu-map     auto:1/1-4 0-3          # pin threads to CPU cores 0-3

#-----------------------------------------------------------------------
# DEFAULTS — inherited by all frontends and backends
#-----------------------------------------------------------------------
defaults
    log         global
    mode        http                     # L7 HTTP proxy (vs tcp for L4)
    option      httplog                  # detailed HTTP logging
    option      dontlognull              # don't log empty connections (health checks)
    option      http-server-close        # close server-side after each request
    option      forwardfor               # add X-Forwarded-For header
    option      redispatch               # retry failed requests to another server

    timeout connect  5s                  # TCP handshake to backend
    timeout client   50s                 # client inactivity timeout
    timeout server   50s                 # server response timeout
    timeout http-request 15s            # max time to receive full HTTP headers
    timeout http-keep-alive 10s         # idle keep-alive connection timeout
    timeout queue    30s                 # max time in queue when all servers full
    timeout tunnel   3600s              # websocket / tunnel idle timeout

    errorfile 400 /etc/haproxy/errors/400.http
    errorfile 403 /etc/haproxy/errors/403.http
    errorfile 408 /etc/haproxy/errors/408.http
    errorfile 500 /etc/haproxy/errors/500.http
    errorfile 502 /etc/haproxy/errors/502.http
    errorfile 503 /etc/haproxy/errors/503.http
    errorfile 504 /etc/haproxy/errors/504.http

Load Balancing Algorithms

HAProxy supports nine scheduling algorithms. Choosing the right one has a significant impact on distribution quality and server resource utilization.

roundrobin — The Default

Requests distributed sequentially across servers. Each server receives an equal share. Supports real-time weight adjustment without removing servers from rotation.

backend web-servers
    balance roundrobin
    server web-01 192.168.1.10:8080 check
    server web-02 192.168.1.11:8080 check
    server web-03 192.168.1.12:8080 check

Best for: Stateless services with uniform request cost and similar server capacity. The correct default choice for most HTTP APIs.

leastconn — Minimize Active Connections

The server with the fewest active connections receives the next request. Ideal for long-lived connections (WebSockets, database connections, LDAP) where some requests tie up server resources longer than others.

backend db-proxies
    balance leastconn
    option tcp-check
    server pgbouncer-01 192.168.1.20:5432 check
    server pgbouncer-02 192.168.1.21:5432 check

Roundrobin is wrong for long-lived connections: if web-01 happens to get assigned several slow database queries, it accumulates connections while web-02 sits idle. Leastconn dynamically corrects this.

source — Client IP Affinity (Sticky Sessions Without Cookies)

A hash of the client’s source IP determines which server they hit. The same IP always reaches the same server (as long as server count doesn’t change). Persistent across connections.

backend api-servers
    balance source
    hash-type consistent  # consistent hashing — server additions/removals
                          # only disrupt a fraction of sessions
    server api-01 192.168.1.30:8080 check
    server api-02 192.168.1.31:8080 check
    server api-03 192.168.1.32:8080 check

hash-type consistent uses a consistent hashing ring. Without it, adding or removing a server reshuffles all sessions. With consistent hashing, only 1/N sessions move when a server is added or removed.

Best for: Services without session support that need some stickiness (e.g., internal gRPC where TLS session resumption benefits from hitting the same backend).

uri — URL-Based Hashing

The request URI (or a part of it) is hashed to select the server. The same URL consistently goes to the same backend.

backend cache-servers
    balance uri
    hash-type consistent
    # Optional: only hash on the path, ignore query string
    # balance uri whole   <- hash entire URI including query string
    # balance uri path    <- hash only the path component
    server cache-01 192.168.1.40:6379 check
    server cache-02 192.168.1.41:6379 check
    server cache-03 192.168.1.42:6379 check

Best for: CDN-like setups or cache servers where you want the same URL to hit the same cache node, maximizing cache hit rate.

hdr — HTTP Header Hashing

Hash on a specific request header value. Common use: route by Host header (virtual hosting), Authorization (sticky per API key), or a custom sharding header.

backend tenant-servers
    balance hdr(X-Tenant-ID)   # shard by tenant
    hash-type consistent
    server tenant-01 192.168.1.50:8080 check
    server tenant-02 192.168.1.51:8080 check

url_param — Query Parameter Hashing

Hash on a URL query parameter value. Useful for session affinity when the session ID is in the URL.

backend video-servers
    balance url_param user_id
    hash-type consistent
    server video-01 192.168.1.60:8080 check
    server video-02 192.168.1.61:8080 check

first — Fill One Server Before Spilling to the Next

Requests go to the first available server until it reaches maxconn, then overflow to the next. Good for minimizing the number of active servers (energy efficiency or licensing costs).

backend compute-servers
    balance first
    server compute-01 192.168.1.70:8080 maxconn 100 check
    server compute-02 192.168.1.71:8080 maxconn 100 check
    server compute-03 192.168.1.72:8080 maxconn 100 check

random — Randomized with Power of Two Choices

A random server is selected from the pool. With random(2) (the default), two servers are randomly selected and the one with fewer connections is used. This provides better distribution than pure random while being simpler than leastconn.

backend api-servers
    balance random(2)
    server api-01 192.168.1.80:8080 check
    server api-02 192.168.1.81:8080 check
    server api-03 192.168.1.82:8080 check
    server api-04 192.168.1.83:8080 check

Weights: Unequal Distribution

Any algorithm can be modified by server weights. Heavier servers receive proportionally more traffic:

backend mixed-capacity
    balance roundrobin
    server small-01  192.168.1.90:8080 check weight 1
    server medium-01 192.168.1.91:8080 check weight 2
    server large-01  192.168.1.92:8080 check weight 4
    # large-01 gets 4x the traffic of small-01

Adjust weights at runtime without a reload:

1
2

echo "set weight web-servers/large-01 8" | \
    socat stdio /run/haproxy/admin.sock

Health Checks

Health checks are HAProxy’s mechanism for detecting server failures and removing them from rotation before clients experience errors.

TCP Check (Layer 4)

The simplest check: can HAProxy establish a TCP connection to the server?

backend web-servers
    server web-01 192.168.1.10:8080 check
    # Defaults: check every 2s, 3 consecutive failures = down, 2 successes = up

    # Customized:
    server web-02 192.168.1.11:8080 check \
        inter 5s \          # check every 5 seconds
        fastinter 1s \      # check every 1s when in transition (up→down or down→up)
        downinter 10s \     # check every 10s when confirmed down
        rise 2 \            # require 2 consecutive successes to mark UP
        fall 3              # require 3 consecutive failures to mark DOWN

TCP checks are cheap and reliable for basic availability. They fail to detect application-level issues (deadlocked app, broken database connection).

HTTP Check (Layer 7)

Send an HTTP request and validate the response code and optionally the body:

backend api-servers
    option httpchk GET /healthz HTTP/1.1\r\nHost:\ api.internal

    # Expect a specific response code
    http-check expect status 200

    # Or a range
    http-check expect rstatus 2[0-9][0-9]

    # Or a string in the body
    http-check expect string "ok"

    # Full example with authentication header
    http-check send meth GET uri /health ver HTTP/1.1 \
        hdr Host api.internal \
        hdr Authorization "Bearer internal-health-check-token"

    server api-01 192.168.1.10:8080 check inter 10s rise 2 fall 3
    server api-02 192.168.1.11:8080 check inter 10s rise 2 fall 3

Agent Check: Application-Reported Health

The agent check connects to a sidecar port and asks the server how it wants to be treated. The server can respond:

• up — healthy, use normally
• down — unhealthy, remove from rotation
• maint — in maintenance, send 503 to clients
• ready — clear maintenance mode
• 50% — reduce my weight to 50%
• drain — stop sending new connections, allow existing to complete

backend web-servers
    server web-01 192.168.1.10:8080 check agent-check agent-port 9001 \
        agent-inter 5s

The agent is a small TCP server (often a simple shell script via socat or netcat) on each backend:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

#!/bin/bash
# health-agent.sh — serves as HAProxy agent on port 9001

# Check application health
if curl -sf http://localhost:8080/healthz > /dev/null; then
    # Check if we're under high load
    LOAD=$(cat /proc/loadavg | awk '{print $1}')
    CPU_COUNT=$(nproc)
    LOAD_RATIO=$(echo "$LOAD $CPU_COUNT" | awk '{printf "%.0f", ($1/$2)*100}')

    if [[ $LOAD_RATIO -gt 80 ]]; then
        echo "50%"  # reduce weight under high load
    else
        echo "up"
    fi
else
    echo "down"
fi

1
2

# Start the agent
socat TCP-LISTEN:9001,fork,reuseaddr EXEC:/usr/local/bin/health-agent.sh

External Check

For complex health verification logic, delegate to an external script:

global
    external-check   # must be enabled in global

backend db-servers
    option external-check
    external-check path "/usr/bin:/bin"
    external-check command /usr/local/bin/check-postgres.sh

    server pg-01 192.168.1.20:5432 check

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

#!/bin/bash
# /usr/local/bin/check-postgres.sh
# HAProxy passes: $1=haproxy_host, $2=haproxy_port, $3=server_host, $4=server_port

SERVER_HOST=$3
SERVER_PORT=$4

# Check PostgreSQL is accepting connections AND is not in recovery
RESULT=$(PGPASSWORD="$PG_CHECK_PASS" psql \
    -h "$SERVER_HOST" -p "$SERVER_PORT" \
    -U checker -d postgres \
    -t -c "SELECT pg_is_in_recovery();" 2>/dev/null)

if [[ "$RESULT" =~ "f" ]]; then  # "f" = not in recovery = primary
    exit 0  # healthy
else
    exit 1  # standby or unreachable
fi

ACLs and Routing

HAProxy’s Access Control Lists are the engine behind all routing decisions. An ACL is a named condition that evaluates to true or false. Use statements route traffic based on ACL combinations.

ACL Syntax

acl <name> <criterion> [flags] [operator] [value ...]

Common ACL Criteria

frontend https-in
    bind *:443 ssl crt /etc/ssl/certs/haproxy.pem

    # Match HTTP methods
    acl is_get      method GET
    acl is_post     method POST
    acl is_options  method OPTIONS

    # Match request path
    acl is_api      path_beg /api/
    acl is_static   path_beg /static/ /assets/ /images/
    acl is_health   path     /health /healthz /readyz
    acl is_ws       path_beg /ws /socket.io

    # Match Host header
    acl host_api    hdr(host) -i api.example.com
    acl host_web    hdr(host) -i example.com www.example.com

    # Match source IP (internal networks get different treatment)
    acl is_internal src 10.0.0.0/8 172.16.0.0/12 192.168.0.0/16
    acl is_trusted  src 10.10.0.0/24  # specific trusted subnet

    # Match HTTP header values
    acl is_json     hdr_val(Content-Type) -m sub application/json
    acl has_auth    hdr_cnt(Authorization) gt 0

    # Match URL query parameters
    acl is_v2       url_param(version) -m str v2

    # Match response (for use with http-after-response)
    # acl is_error    status 500 502 503 504

    # Routing decisions
    use_backend     health-check    if is_health
    use_backend     websocket-svrs  if is_ws
    use_backend     static-cache    if is_static
    use_backend     api-v2          if host_api is_v2
    use_backend     api-servers     if host_api
    use_backend     admin-servers   if is_internal { path_beg /admin/ }
    default_backend web-servers

Advanced ACL: Block Bad Actors

frontend https-in
    bind *:443 ssl crt /etc/ssl/certs/haproxy.pem

    # Block known bad user agents
    acl bad_bot     hdr_sub(User-Agent) -i masscan nmap sqlmap nikto
    acl empty_ua    hdr_cnt(User-Agent) eq 0

    # Block requests with suspicious patterns
    acl sql_inject  url_sub -i select%20 union%20 drop%20 insert%20
    acl path_trav   url_sub -i ../  ..\\ %2e%2e

    # Block specific IPs (maintain a file for dynamic updates)
    acl blocklist   src -f /etc/haproxy/blocked-ips.txt

    http-request deny deny_status 403 if bad_bot
    http-request deny deny_status 400 if sql_inject or path_trav
    http-request deny deny_status 403 if blocklist
    http-request deny deny_status 400 if empty_ua !is_internal

    # Return a tarpit (slow response) instead of immediate reject to
    # waste bot time without giving a clear signal
    http-request tarpit if bad_bot

Rewriting Requests and Responses

frontend https-in
    # Add security headers to all responses
    http-response set-header X-Frame-Options DENY
    http-response set-header X-Content-Type-Options nosniff
    http-response set-header Strict-Transport-Security "max-age=31536000; includeSubDomains; preload"
    http-response set-header Referrer-Policy "strict-origin-when-cross-origin"

    # Remove server fingerprinting headers from responses
    http-response del-header Server
    http-response del-header X-Powered-By
    http-response del-header X-AspNet-Version

    # Rewrite the Host header before forwarding upstream
    http-request set-header Host api-internal.svc.cluster.local

    # Add request ID for tracing (if not already set)
    http-request set-header X-Request-ID %[uuid()] if !{ hdr_cnt(X-Request-ID) gt 0 }

    # Strip internal headers that clients shouldn't be able to inject
    http-request del-header X-Internal-User
    http-request del-header X-Admin-Override

    # URL rewriting: /v1/users → /api/v1/users
    http-request replace-path ^/v1/(.*)$ /api/v1/\1

Sticky Sessions (Cookie-Based)

For applications that require session affinity, HAProxy can insert a cookie to track which backend server a client was assigned:

backend app-servers
    balance roundrobin
    cookie SERVERID insert indirect nocache httponly secure

    server app-01 192.168.1.10:8080 check cookie app-01
    server app-02 192.168.1.11:8080 check cookie app-02
    server app-03 192.168.1.12:8080 check cookie app-03

HAProxy inserts a SERVERID=app-01 cookie on the first response. Subsequent requests from that client include the cookie, and HAProxy routes them to app-01. If app-01 goes down, HAProxy routes to another server and updates the cookie.

Options:

• insert — HAProxy inserts the cookie on first response
• indirect — don’t forward the cookie to the backend (backend never sees it)
• nocache — add Cache-control: nocache to prevent caching responses with set cookies
• httponly — add HttpOnly flag to the cookie
• secure — add Secure flag (HTTPS only)

SSL/TLS Termination

HAProxy is excellent at TLS termination — it handles the encrypted connection from clients, then forwards plaintext (or re-encrypted) traffic to backends.

Basic TLS Termination

frontend https-in
    bind *:443 ssl crt /etc/ssl/private/example.com.pem \
                       alpn h2,http/1.1   # ALPN negotiation for HTTP/2

    # Redirect HTTP to HTTPS
    bind *:80
    http-request redirect scheme https unless { ssl_fc }

    default_backend web-servers

The certificate file is a PEM bundle containing the certificate, intermediates, and private key in order:

1
2

cat example.com.crt intermediate.crt ca.crt example.com.key > /etc/ssl/private/example.com.pem
chmod 600 /etc/ssl/private/example.com.pem

SNI-Based Multi-Certificate Termination

Serve different certificates for different hostnames on the same IP:

frontend https-in
    bind *:443 ssl \
        crt /etc/ssl/private/example.com.pem \
        crt /etc/ssl/private/api.example.com.pem \
        crt /etc/ssl/private/admin.example.com.pem

    # HAProxy selects the right cert based on SNI automatically
    # Route to different backends based on hostname
    use_backend api-servers   if { hdr(host) -i api.example.com }
    use_backend admin-servers if { hdr(host) -i admin.example.com }
    default_backend web-servers

For many certificates, use a directory — HAProxy loads all PEM files in the directory:

    bind *:443 ssl crt /etc/ssl/private/certs/

TLS Passthrough (L4 SNI Routing)

Route encrypted traffic to backends without terminating TLS — the backend handles decryption:

frontend tls-passthrough
    bind *:443
    mode tcp
    option tcplog

    # Route based on SNI (the hostname in the TLS ClientHello)
    use_backend internal-k8s     if { req.ssl_sni -i k8s.internal }
    use_backend secure-app       if { req.ssl_sni -m end .secure.example.com }
    default_backend web-servers-tls

backend internal-k8s
    mode tcp
    server k8s-api 10.0.0.1:6443 check

mTLS: Mutual TLS Authentication

Require clients to present a certificate signed by your CA:

frontend mtls-api
    bind *:8443 ssl \
        crt /etc/ssl/private/server.pem \
        ca-file /etc/ssl/private/client-ca.crt \
        verify required   # reject clients without a valid cert

    # Extract the client certificate subject for use in backends
    http-request set-header X-Client-CN %{+Q}[ssl_c_s_dn(cn)]
    http-request set-header X-Client-Serial %[ssl_c_serial,hex]

    # Only allow specific client certificates
    acl valid_client ssl_c_s_dn(o) -m str "Trusted Internal Services"
    http-request deny unless valid_client

    default_backend internal-api

backend internal-api
    # Re-encrypt to backend (backend also requires TLS)
    server api-01 192.168.1.10:8443 ssl \
        ca-file /etc/ssl/private/internal-ca.crt \
        verify required

Certificate Reloading Without Restart

HAProxy 2.2+ supports certificate updates via the Runtime API without a full reload:

1
2
3
4
5
6
7

# Update a certificate at runtime
echo "set ssl cert /etc/ssl/private/example.com.pem <<\n$(cat new-cert.pem)\n" | \
    socat stdio /run/haproxy/admin.sock

# Commit the change
echo "commit ssl cert /etc/ssl/private/example.com.pem" | \
    socat stdio /run/haproxy/admin.sock

For automated Let’s Encrypt renewal, a post-renewal hook:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

#!/bin/bash
# /etc/letsencrypt/renewal-hooks/deploy/haproxy.sh
DOMAIN="example.com"
CERT_DIR="/etc/letsencrypt/live/$DOMAIN"
HAPROXY_PEM="/etc/ssl/private/$DOMAIN.pem"

# Combine cert chain and key
cat "$CERT_DIR/fullchain.pem" "$CERT_DIR/privkey.pem" > "$HAPROXY_PEM"

# Reload HAProxy gracefully (no dropped connections)
systemctl reload haproxy
# Or for zero-downtime: send USR2 signal to trigger a seamless reload
# kill -USR2 $(cat /run/haproxy.pid)

Rate Limiting and DDoS Mitigation

HAProxy has built-in stick tables — in-memory key-value stores that track connection and request rates per source IP, cookie, or any other criterion.

Connection Rate Limiting per IP

# Define a stick table in the frontend or a dedicated "peers" section
frontend https-in
    bind *:443 ssl crt /etc/ssl/private/example.com.pem

    # Stick table: track HTTP request rate per source IP
    # type ip: key is source IP
    # size 100k: hold 100,000 entries
    # expire 1m: entries expire after 1 minute of inactivity
    # store: what to track
    stick-table type ip size 100k expire 1m \
        store conn_rate(10s),http_req_rate(10s),http_err_rate(1m),conn_cur

    # Track all connections
    tcp-request connection track-sc0 src

    # Deny if more than 100 connections per 10 seconds from one IP
    acl too_many_conns  sc_conn_rate(0) gt 100
    # Deny if more than 300 HTTP requests per 10 seconds
    acl too_many_reqs   sc_http_req_rate(0) gt 300
    # Deny if more than 20 errors per minute (likely scanning/fuzzing)
    acl too_many_errors sc_http_err_rate(0) gt 20
    # Deny if more than 50 simultaneous connections
    acl too_many_cur    sc_conn_cur(0) gt 50

    http-request deny deny_status 429 if too_many_reqs
    http-request deny deny_status 429 if too_many_errors
    tcp-request connection reject if too_many_conns
    tcp-request connection reject if too_many_cur

    default_backend web-servers

Tarpit Slow Connections (Catch Scanners)

frontend https-in
    # Tarpit abusive IPs — hold the connection open for timeout duration
    # Wastes attacker resources without giving information
    http-request tarpit if too_many_errors
    timeout tarpit 30s

Separate Rate Limits for Different Endpoints

frontend https-in
    # Tighter limits for authentication endpoints (brute force protection)
    stick-table type ip size 10k expire 5m \
        store http_req_rate(30s)

    acl is_auth path_beg /api/auth /api/login /api/token

    # Track auth requests separately
    http-request track-sc1 src if is_auth

    acl auth_rate_exceeded sc1_http_req_rate gt 10  # 10 auth attempts per 30s

    http-request deny deny_status 429 if is_auth auth_rate_exceeded

Sharing Stick Tables Across HAProxy Instances

In a multi-node setup, sync stick tables via the peers section:

peers haproxy-cluster
    bind *:1024
    peer haproxy-01 192.168.1.100:1024
    peer haproxy-02 192.168.1.101:1024

frontend https-in
    stick-table type ip size 100k expire 1m \
        store http_req_rate(10s) \
        peers haproxy-cluster   # sync this table across peers

The Stats Page and Runtime API

Stats Dashboard

listen stats
    bind *:8404
    stats enable
    stats uri /stats
    stats refresh 10s
    stats show-legends
    stats show-node
    stats auth admin:changeme       # basic auth
    stats admin if { src 192.168.0.0/24 }  # allow admin actions from LAN only

Navigate to http://your-haproxy:8404/stats for a live dashboard showing:

• Frontend and backend connection rates, data rates, error rates
• Server status (UP/DOWN/MAINT) and health check results
• Active sessions per backend
• Queue depths
• Cumulative stats since startup

Runtime API (socat)

The admin socket allows real-time management without reloads:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32

SOCKET="/run/haproxy/admin.sock"
CMD() { echo "$*" | socat stdio "$SOCKET"; }

# Show server status
CMD "show servers state web-servers"

# Drain a server (stop new connections, finish existing ones)
CMD "set server web-servers/web-01 state drain"

# Put a server in maintenance (return 503 to clients)
CMD "set server web-servers/web-01 state maint"

# Re-enable a server
CMD "set server web-servers/web-01 state ready"

# Dynamically adjust weight
CMD "set server web-servers/web-01 weight 50"

# View stick table contents
CMD "show table https-in"

# Clear an IP from a stick table (unban)
CMD "clear table https-in key 1.2.3.4"

# Show current connections
CMD "show info" | grep -E "^(Maxconn|CurrConns|MaxSessRate)"

# Show active sessions
CMD "show sess" | head -20

# Reload certificate at runtime
CMD "show ssl cert /etc/ssl/private/example.com.pem"

Prometheus Metrics

HAProxy 2.0+ includes a native Prometheus exporter:

frontend prometheus
    bind *:8405
    http-request use-service prometheus-exporter if { path /metrics }
    acl is_local src 127.0.0.1 ::1 192.168.0.0/24
    http-request deny unless is_local

Key metrics to alert on:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41

# prometheus-alerts.yaml
groups:
  - name: haproxy
    rules:
      - alert: HaproxyBackendDown
        expr: haproxy_backend_up == 0
        for: 0m
        labels:
          severity: critical
        annotations:
          summary: "HAProxy backend {{ $labels.proxy }} is down"

      - alert: HaproxyServerDown
        expr: haproxy_server_up == 0
        for: 0m
        labels:
          severity: warning
        annotations:
          summary: "HAProxy server {{ $labels.server }} in {{ $labels.proxy }} is down"

      - alert: HaproxyHighErrorRate
        expr: |
          rate(haproxy_backend_http_responses_total{code="5xx"}[5m])
          / rate(haproxy_backend_http_responses_total[5m]) > 0.05
        for: 2m
        annotations:
          summary: "HAProxy backend {{ $labels.proxy }} error rate > 5%"

      - alert: HaproxyHighQueueDepth
        expr: haproxy_backend_current_queue > 10
        for: 1m
        annotations:
          summary: "HAProxy backend {{ $labels.proxy }} queue depth {{ $value }}"

      - alert: HaproxyHighSessionUsage
        expr: |
          haproxy_process_current_connections
          / haproxy_process_max_connections > 0.8
        for: 5m
        annotations:
          summary: "HAProxy using >80% of max connections"

Graceful Reload and Zero-Downtime Deployment

HAProxy’s seamless reload is one of its killer features. On reload, the old process:

1. Stops accepting new connections
2. Transfers listening sockets to the new process
3. Finishes serving existing connections
4. Exits cleanly

No dropped connections. Clients don’t notice.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

# Validate config before reload
haproxy -c -f /etc/haproxy/haproxy.cfg

# Graceful reload (systemd)
systemctl reload haproxy

# Manual graceful reload (send SIGUSR2 to master process)
kill -USR2 $(cat /run/haproxy/haproxy.pid)

# Hard reload (drops connections — use only if graceful fails)
systemctl restart haproxy

Blue-Green Deployment via HAProxy

Use HAProxy’s runtime API to implement blue-green deploys without any config file changes:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41

#!/bin/bash
# blue-green-deploy.sh
# Deploy new version (green) while blue keeps serving traffic

SOCKET="/run/haproxy/admin.sock"
BACKEND="web-servers"
BLUE_SERVERS=(web-blue-01 web-blue-02 web-blue-03)
GREEN_SERVERS=(web-green-01 web-green-02 web-green-03)

cmd() { echo "$*" | socat stdio "$SOCKET"; }

# 1. Enable green servers
for server in "${GREEN_SERVERS[@]}"; do
    cmd "set server $BACKEND/$server state ready"
done
echo "Green servers enabled"

# 2. Wait for green to become healthy
sleep 10
for server in "${GREEN_SERVERS[@]}"; do
    STATUS=$(cmd "show servers state $BACKEND" | grep "$server" | awk '{print $6}')
    if [[ "$STATUS" != "2" ]]; then  # 2 = UP
        echo "ERROR: $server not healthy. Aborting."
        exit 1
    fi
done

# 3. Drain blue servers
for server in "${BLUE_SERVERS[@]}"; do
    cmd "set server $BACKEND/$server state drain"
done
echo "Blue servers draining..."

# 4. Wait for drain to complete
sleep 30

# 5. Disable blue servers
for server in "${BLUE_SERVERS[@]}"; do
    cmd "set server $BACKEND/$server state maint"
done
echo "Blue-green deploy complete"

Performance Tuning

Connection Limits

global
    maxconn 100000     # per-process limit (set to ~80% of OS limit)
    # Check OS limit: ulimit -n
    # For 100k: add to /etc/security/limits.conf:
    # haproxy soft nofile 200000
    # haproxy hard nofile 200000

defaults
    timeout connect  1s     # aggressive connect timeout for LAN backends
    timeout client   30s    # client-facing — tune to your application's needs
    timeout server   30s    # backend — must be > your slowest endpoint

    # For high-latency or large response backends, increase server timeout:
    # timeout server  300s

Multi-Threading

global
    nbthread 8             # match your CPU core count
    cpu-map auto:1/1-8 0-7 # pin each thread to a specific core (reduces cache thrashing)

Benchmark to verify multi-threading helps for your workload:

1
2
3

# Run haproxy with nbthread 1 vs 4 vs 8 and compare
# using wrk or hey for load testing
wrk -t12 -c400 -d30s http://haproxy:80/api/test

Buffer Sizes

global
    tune.bufsize 32768      # default 16384. Increase for large HTTP headers (JWT tokens)
    tune.maxrewrite 8192    # buffer for header rewrites, must be < bufsize/2
    tune.ssl.default-dh-param 2048
    tune.ssl.cachesize 100000   # SSL session cache entries

Connection Reuse to Backends

backend api-servers
    option http-reuse always   # aggressively reuse connections to backends
    # Options: never, safe, aggressive, always
    # "safe": reuse idle connections (most conservative — good default)
    # "always": reuse even non-idle connections when safe to do so

Complete Production Example

A realistic HAProxy configuration for a web application:

global
    log /dev/log local0
    chroot /var/lib/haproxy
    stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
    stats timeout 30s
    user haproxy
    group haproxy
    daemon
    maxconn 50000
    nbthread 4
    cpu-map auto:1/1-4 0-3
    ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256
    ssl-default-bind-options no-sslv3 no-tlsv10 no-tlsv11

defaults
    log global
    mode http
    option httplog
    option dontlognull
    option http-server-close
    option forwardfor
    option redispatch
    timeout connect  2s
    timeout client   30s
    timeout server   60s
    timeout queue    20s
    timeout http-request 10s

#--- Stats ---
listen stats
    bind *:8404
    stats enable
    stats uri /stats
    stats refresh 30s
    stats auth admin:${STATS_PASSWORD}
    stats admin if { src 10.0.0.0/8 }

#--- Prometheus ---
frontend prometheus
    bind 127.0.0.1:8405
    http-request use-service prometheus-exporter if { path /metrics }
    default_backend dev-null

backend dev-null
    server dummy 127.0.0.1:0 check

#--- HTTP → HTTPS redirect ---
frontend http-in
    bind *:80
    http-request redirect scheme https code 301

#--- Main HTTPS frontend ---
frontend https-in
    bind *:443 ssl crt /etc/ssl/private/example.com.pem alpn h2,http/1.1

    # Security headers on all responses
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    http-response set-header X-Frame-Options DENY
    http-response set-header X-Content-Type-Options nosniff
    http-response del-header Server
    http-response del-header X-Powered-By

    # Inject request ID
    http-request set-header X-Request-ID %[uuid()] if !{ hdr_cnt(X-Request-ID) gt 0 }

    # Rate limiting stick table
    stick-table type ip size 100k expire 1m \
        store conn_rate(10s),http_req_rate(10s),http_err_rate(1m)
    http-request track-sc0 src
    acl high_req_rate  sc_http_req_rate(0) gt 200
    acl high_err_rate  sc_http_err_rate(0) gt 30
    http-request deny deny_status 429 if high_req_rate
    http-request deny deny_status 429 if high_err_rate

    # Block bad patterns
    acl sql_inject url_sub -i select%20 union%20 drop%20
    acl path_trav  url_sub -i ../
    http-request deny deny_status 400 if sql_inject or path_trav

    # ACL definitions
    acl is_api      path_beg /api/
    acl is_static   path_beg /static/ /assets/
    acl is_health   path     /health /healthz
    acl is_ws       hdr(Upgrade) -i websocket

    # Routing
    use_backend websocket-svrs if is_ws
    use_backend static-cdn     if is_static
    use_backend health-check   if is_health
    use_backend api-servers    if is_api
    default_backend web-servers

#--- Backends ---
backend web-servers
    balance roundrobin
    option httpchk GET /healthz HTTP/1.1\r\nHost:\ example.com
    http-check expect status 200
    cookie SERVERID insert indirect nocache httponly secure
    server web-01 10.0.1.10:8080 check cookie web-01 inter 5s rise 2 fall 3
    server web-02 10.0.1.11:8080 check cookie web-02 inter 5s rise 2 fall 3
    server web-03 10.0.1.12:8080 check cookie web-03 inter 5s rise 2 fall 3 backup

backend api-servers
    balance leastconn
    option httpchk GET /api/healthz HTTP/1.1\r\nHost:\ api.example.com
    http-check expect status 200
    option http-reuse safe
    server api-01 10.0.2.10:8080 check inter 5s rise 2 fall 3
    server api-02 10.0.2.11:8080 check inter 5s rise 2 fall 3

backend websocket-svrs
    balance source
    option http-server-close
    timeout tunnel 3600s
    server ws-01 10.0.3.10:8080 check
    server ws-02 10.0.3.11:8080 check

backend static-cdn
    balance roundrobin
    server cdn-01 10.0.4.10:80 check

backend health-check
    server local 127.0.0.1:8080 check

Quick Reference: When to Use What

Related: Load Balancing, Traefik Complete Guide, Designing for Observability, SSL Certificates