MinIO: Self-Hosted S3 — Deployment, Erasure Coding, and Performance Tuning

April 18, 2026 14 min read Data & Databases

minios3object-storagestorageself-hostingerasure-coding

Contents

MinIO is the S3-compatible object storage server most people reach for when they need “S3, but not AWS”. It’s a single Go binary that can run on a laptop, scale to multi-node clusters serving petabytes, and claims line-rate throughput on commodity hardware. It’s also the subject of occasional community friction around licensing and the recent removal of the web console from the community edition. This post focuses on what it is, how it works, and how to run it well — not the licensing drama.

What MinIO actually is

MinIO implements the S3 API: PutObject, GetObject, ListObjectsV2, multipart uploads, presigned URLs, bucket policies, server-side encryption, lifecycle rules, event notifications, object locking, versioning. Almost every S3 SDK and tool works against it unmodified, usually by pointing at a custom endpoint URL. Your code doesn’t need to know it isn’t talking to AWS.

Internally, MinIO is a distributed system built around erasure coding over local disks. Data is split into data shards plus parity shards spread across disks and nodes, and reads reconstruct from whatever is available. There is no metadata server — objects and their metadata live together, addressed by a deterministic hash of the object name. The architecture is intentionally simple: no Paxos, no etcd, no external database. Coordination happens through the erasure-coded state itself.

The trade-off: MinIO is immutable at the cluster topology level. Once you deploy a set of nodes and drives, you do not add or remove disks within that set. You add entire new “server pools” of nodes, and MinIO federates across them. More on this in the “scaling” section.

Single-node deployment

The simplest deployment is one node, one drive:

1
2


mkdir -p /srv/minio/data
minio server /srv/minio/data --console-address ":9001"

This is fine for development. You have no redundancy; a drive failure loses everything. Useful for CI environments, local testing, and dev stacks.

Scale up to a single node with multiple drives, using erasure coding locally:

1

minio server /mnt/disk{1...4}/minio --console-address ":9001"

With 4 drives, MinIO defaults to EC:2 — 2 data shards + 2 parity shards — so any 2 drives can fail. Still a single point of failure (the node), but now you survive disk failures.

Docker Compose for single-node

A working docker-compose.yml for a single-node production-ish MinIO:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25


services:
  minio:
    image: quay.io/minio/minio:latest
    container_name: minio
    restart: unless-stopped
    command: server /data{1...4} --console-address ":9001"
    environment:
      MINIO_ROOT_USER: admin
      MINIO_ROOT_PASSWORD_FILE: /run/secrets/minio_root_password
      MINIO_BROWSER_REDIRECT_URL: https://minio-console.example.com
      MINIO_SERVER_URL: https://minio.example.com
    volumes:
      - ./data1:/data1
      - ./data2:/data2
      - ./data3:/data3
      - ./data4:/data4
    secrets:
      - minio_root_password
    ports:
      - "9000:9000"
      - "9001:9001"

secrets:
  minio_root_password:
    file: ./secrets/minio_root_password.txt

For real workloads, put each /dataN on a separate physical disk, not just separate directories on the same filesystem — erasure coding across directories on one disk is theater, not redundancy.

Distributed deployment

The real MinIO deployment model is 4+ nodes, each with multiple drives. This unlocks node-level failure tolerance.

A standard 4-node cluster, each with 4 drives:

1
2
3
4


# On each node, run the same command. Node hostnames must resolve.
minio server \
    http://minio{1...4}.example.com/mnt/disk{1...4}/minio \
    --console-address ":9001"

This produces a single “server pool” with 16 drives total. MinIO calculates an erasure set — typically 16 drives, meaning all 16 form one stripe with 12 data + 4 parity shards by default. You can tune the parity count:

1

MINIO_STORAGE_CLASS_STANDARD=EC:6  minio server ...   # 10 data + 6 parity

Higher parity = more failure tolerance but worse storage efficiency. The defaults (EC:4 for STANDARD, EC:2 for REDUCED_REDUNDANCY) are sensible for most deployments.

What “erasure set” means in practice

An erasure set is the group of drives across which one object is striped. MinIO picks set sizes from {4, 6, 8, 10, 12, 14, 16} based on your topology, aiming for even distribution across nodes. With 4 nodes × 4 drives = 16 drives, MinIO creates one erasure set of 16 — each object is spread across all 16 drives.

With 8 nodes × 4 drives = 32 drives, MinIO would create two erasure sets of 16, and each object lands in exactly one set (not both). Node affinity matters: MinIO tries to place drives from the same erasure set on different nodes so that losing a node doesn’t wipe out a whole set.

The practical consequence: you lose the cluster if you lose more than EC drives from any single erasure set. With 4 nodes and EC:4, you survive 4 drive failures total per erasure set, not 4 per node. Count carefully when sizing parity.

Minimum node count

MinIO does not run with fewer than 4 nodes in distributed mode. 4 is the minimum for quorum math to work. You can run 4 nodes with 1 drive each, but practical deployments use 4+ nodes with 4+ drives each.

Systemd deployment

For bare-metal production, deploy the binary under systemd. A minimal unit:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


# /etc/systemd/system/minio.service
[Unit]
Description=MinIO
After=network-online.target
Wants=network-online.target

[Service]
User=minio-user
Group=minio-user
EnvironmentFile=/etc/default/minio
ExecStart=/usr/local/bin/minio server $MINIO_OPTS $MINIO_VOLUMES
Restart=always
LimitNOFILE=1048576
TasksMax=infinity
TimeoutStopSec=infinity

[Install]
WantedBy=multi-user.target

With /etc/default/minio:

MINIO_ROOT_USER=admin
MINIO_ROOT_PASSWORD=redacted
MINIO_OPTS="--console-address :9001"
MINIO_VOLUMES="http://minio{1...4}.internal/mnt/disk{1...4}/minio"
MINIO_SERVER_URL=https://s3.example.com

The LimitNOFILE=1048576 matters. MinIO opens a lot of file descriptors under concurrent load and will emit too many open files errors at the default ulimit.

Filesystem choices for MinIO disks

MinIO is filesystem-agnostic but picky. Officially supported: XFS. Officially discouraged: everything else.

This recommendation is not arbitrary. MinIO relies on fsync behavior, sparse file support, extended attributes, and large directory listings — all of which XFS handles predictably under load. ext4 works but has reported performance issues on very large directories. Btrfs works but its COW semantics interact awkwardly with MinIO’s append-oriented write patterns. ZFS works but you’ll be paying for redundancy twice.

Format each MinIO drive directly with XFS, no LVM, no RAID beneath:

1
2


mkfs.xfs -L MINIO1 /dev/nvme0n1
echo "LABEL=MINIO1 /mnt/disk1 xfs defaults,noatime 0 2" >> /etc/fstab

Do not put RAID beneath MinIO. Erasure coding IS the redundancy layer. RAID5 beneath erasure coding wastes capacity, steals IOPS, and complicates failure handling. The MinIO docs are emphatic about this, and they’re right.

Do not use the same filesystem for multiple MinIO drives. Each drive entry in the MINIO_VOLUMES list needs a separate physical device. MinIO checks for and refuses to start on overlapping devices in strict mode.

TLS and load balancing

Put MinIO behind a real TLS terminator. Options:

Let MinIO handle TLS directly by placing certs in ~/.minio/certs/public.crt and ~/.minio/certs/private.key. Fine for small deployments.
Put a reverse proxy in front (Traefik, Nginx, Caddy). Required for multi-node clusters behind a single endpoint.

For distributed clusters, the S3 endpoint should resolve to all nodes (round-robin DNS, or a TCP load balancer). S3 clients connect to any node, and that node serves the request by pulling shards from peers as needed.

Important: don’t terminate TLS at each MinIO node with different certs and a load balancer in passthrough mode unless you know what you’re doing. Use a single cert at the LB with HTTP between LB and MinIO nodes on a trusted network, or terminate TLS identically at each node.

MinIO has two ports:

9000: the S3 API endpoint.
9001 (configurable): the web console.

The console is now a separate repository (minio/console) under AGPLv3 and has a paid enterprise variant. Community builds still include the basic console; self-compile if you want the newest version. Most production operators use mc (the command-line client) and monitoring dashboards rather than the web UI anyway.

The `mc` client

mc (MinIO Client) is the operational tool. It is to MinIO what kubectl is to Kubernetes — you will use it constantly.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


# Configure an alias
mc alias set myminio https://s3.example.com admin hunter2

# Common operations
mc ls myminio
mc mb myminio/backups                         # make bucket
mc cp /path/to/file myminio/backups/
mc mirror /local/dir myminio/backups/dir/
mc rm --recursive --force myminio/old-backups/

# Admin
mc admin info myminio                         # cluster status
mc admin heal -r myminio                      # heal damaged objects
mc admin service restart myminio              # rolling restart
mc admin user add myminio alice secretpass
mc admin policy attach myminio readwrite --user alice

The admin subcommand covers most cluster management: user creation, policy management, heal, rebalance, decommission, replication setup.

Erasure coding math and failure modes

A 4-node × 4-drive cluster with EC:4 has 16 drives per erasure set and 4 parity. In plain terms:

Any 4 drives can fail simultaneously → reads and writes continue normally.
5–7 drives fail → cluster is read-only (enough to reconstruct, not enough to safely write).
8+ drives fail → data loss.

Node-level math assuming balanced drive distribution:

Lose 1 node (4 drives) → cluster read/write, rebalancing queued.
Lose 2 nodes (8 drives) → 4 parity spent, cluster read-only at best, partial data loss likely.

Conclusion: EC:4 on a 4-node cluster is fragile against node-level failures. If you need node-level resilience, either:

Bump parity (EC:6 or EC:8, sacrificing usable space).
Spread across more nodes (8 or 12 nodes with fewer drives each).
Use site replication — two independent MinIO clusters replicating asynchronously.

For homelab-scale self-hosting, a 4-node EC:4 deployment is the sweet spot. For anything business-critical, aim for 8+ nodes or site replication.

Healing and rebalancing

When a drive fails and is replaced, MinIO detects it and begins healing — reconstructing the missing shards from surviving shards.

Check heal status:

1
2


mc admin heal -r myminio
mc admin heal myminio/bucketname --scan=deep   # slower, checksums every object

Healing runs in the background and throttles itself to avoid saturating the cluster. On a petabyte cluster, a full heal can take days. Don’t panic.

Rebalancing is different — it runs when you add a new server pool. MinIO redistributes existing objects so that the new pool shares the load. This also takes time and uses background I/O.

Decommissioning

You cannot remove drives from a server pool, but you can decommission an entire pool:

1
2


mc admin decommission start myminio http://old-node{1...4}/disk{1...4}
mc admin decommission status myminio

MinIO copies all objects out of the pool to remaining pools, then marks the pool removable. Once complete, stop the processes and remove the pool entries from your config.

Bucket features worth using

Versioning

1

mc version enable myminio/critical

Every PutObject creates a new version. GetObject returns the latest unless a version ID is specified. Deletes create delete markers rather than removing data. Enables point-in-time recovery — essential for backup targets.

Object locking (WORM)

1
2


mc mb --with-lock myminio/compliance
mc retention set --default compliance 1y myminio/compliance

Write-Once-Read-Many storage with compliance-grade retention. Objects cannot be deleted or overwritten until retention expires. Two modes: Governance (admins can override with special permission) and Compliance (nothing short of destroying the cluster can remove them before expiration). Required for many audit/backup scenarios.

Lifecycle rules

Automatically transition or expire objects by age and prefix:

1
2
3
4
5
6
7
8


{
  "Rules": [{
    "ID": "expire-old-logs",
    "Status": "Enabled",
    "Filter": { "Prefix": "logs/" },
    "Expiration": { "Days": 90 }
  }]
}

1

mc ilm import myminio/app-logs < lifecycle.json

Useful for log buckets, CI artifacts, and transient data.

Bucket replication

Synchronous or asynchronous replication to a secondary cluster:

1
2


mc admin replicate add primary secondary
mc replicate add --remote-bucket arn:minio:replication::xxx:target myminio/critical

For DR, combine with versioning on both sides so that a replication loop can’t destroy historical versions.

Performance tuning

Drive performance

MinIO is disk-bound. Use mc support perf drive myminio to benchmark each drive. A healthy NVMe should hit 2+ GB/s sequential, 200k+ 4k random IOPS. Slow drives will be identified, and one slow drive can cap the cluster.

Network

Distributed MinIO shuffles shard data between nodes on every request. 10 GbE is the practical minimum; 25 or 100 GbE is standard for larger clusters. MinIO’s own perf test:

1

mc support perf net myminio

If network bandwidth shows as the bottleneck, check for flow control issues, MTU mismatches (jumbo frames require consistent configuration), and bonding imbalance.

Concurrency and object size

MinIO throughput scales with concurrent object count. A single large upload from a single client will not saturate a cluster. Parallel multipart uploads with mc cp --parallel 16 or client-side concurrency is expected.

For small objects (< 1 MB), the overhead of erasure coding per object dominates. MinIO handles this reasonably but dedicated small-object stores (like Ceph’s RGW with Bluestore small-object pools) do better. For logs and metrics, use tools that batch into larger objects before writing to MinIO — don’t PUT one object per log line.

Multipart tuning

Multipart uploads help for large objects. Default part size with mc is 64 MB, minimum 5 MB, maximum 5 GB. For very large objects (100+ GB), increase part size so you don’t hit the 10,000-part ceiling:

1

mc cp --part-size 256MiB hugefile.tar myminio/backups/

Compression

MinIO can compress objects server-side with zstd or s2:

1
2
3
4


MINIO_COMPRESS_ENABLE=on \
MINIO_COMPRESS_MIME_TYPES="text/*,application/json,application/xml" \
MINIO_COMPRESS_EXTENSIONS=".txt,.log,.csv,.json,.xml" \
minio server ...

Compression runs on write, decompression on read. Disabled for media formats (already compressed) and encrypted uploads.

IAM, policies, and STS

MinIO implements the S3 IAM model:

Users with long-lived access keys.
Service accounts (scoped sub-keys under a user).
Policies in S3 JSON format, attached to users or groups.
STS for temporary credentials via OIDC or LDAP.

1
2
3


mc admin user add myminio ci-uploader $(openssl rand -hex 32)
mc admin policy create myminio uploads-only < uploads-policy.json
mc admin policy attach myminio uploads-only --user ci-uploader

Example bucket-scoped policy for CI uploads:

1
2
3
4
5
6
7
8


{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Action": ["s3:PutObject", "s3:GetObject"],
    "Resource": ["arn:aws:s3:::ci-artifacts/*"]
  }]
}

For OIDC integration (Keycloak, Authentik, Google), configure MINIO_IDENTITY_OPENID_* variables. Users authenticate to your IdP and receive time-limited STS credentials — the right pattern for human users.

Observability

MinIO exposes Prometheus metrics at /minio/v2/metrics/cluster and per-bucket metrics at /minio/v2/metrics/bucket. Point Prometheus at it and import the official Grafana dashboard. Key metrics:

minio_node_disk_free_bytes — per-drive free space.
minio_cluster_capacity_usable_total_bytes — usable post-erasure-coding.
minio_s3_requests_total — request rate by API.
minio_s3_errors_total — 4xx/5xx rates.
minio_cluster_nodes_offline_total — set alerts on this.
minio_heal_time_last_activity_nano_seconds — heal health.

The MinIO OpenTelemetry support also exports traces if you need per-request visibility.

Common pitfalls

Running MinIO on RAID. Waste of capacity and IOPS. Erasure coding is the redundancy layer.
Formatting drives with ext4 “because it’s easier”. It works until it doesn’t; XFS is the supported path.
Using fewer than 4 nodes in distributed mode. MinIO won’t start, and trying to work around this by running 4 processes on one node defeats the point.
Placing all drives of an erasure set on one node. A node failure loses the cluster. Verify with mc admin info.
Leaving LimitNOFILE at the default. High-concurrency workloads will exhaust file descriptors.
Not enabling versioning on backup buckets. A misconfigured mc rm --recursive will ruin your day.
Expecting to shrink the cluster. Plan for pool-level decommissioning, not drive-level.
Using MinIO as a database. It’s object storage. ListObjectsV2 at scale is O(N) on prefix depth — not a replacement for a real DB.

When MinIO is the right answer

S3-compatible backup target. Restic, Kopia, Velero, pgBackRest, rclone all work.
Artifact storage for CI. Docker registries (via S3 backend), Nexus, JFrog, build outputs.
Data lake on commodity hardware. Spark, Trino, DuckDB, and Iceberg all read MinIO as cheaply as they read AWS S3.
Development parity with production AWS. Dev teams point at local MinIO; prod points at AWS. Same SDK code.
Homelab photo and file backup. Combined with Immich’s S3 backend or Nextcloud’s external storage.

When to pick something else

You need hundreds of petabytes with complex tiering. Ceph RGW has better multi-tier stories.
You need POSIX and block semantics. MinIO is object-only. Add CephFS or GlusterFS for those.
You need a managed control plane. AWS S3 and GCS exist for a reason. MinIO is software you operate.
Your object size distribution is dominated by <100 KB objects. MinIO handles them but isn’t the best fit.

The licensing note

MinIO’s community edition is AGPLv3. The server binary is free to run. The paid “SUBNET” subscription adds support, the enterprise console, licensed compatibility certifications, and legal guarantees that AGPL doesn’t provide. For most self-hosters and small teams, the AGPL binary is fine — you’re not “distributing” anything by running it behind your firewall.

If you’re building a product that exposes MinIO-backed storage to customers, read the AGPL carefully. The network-use clause is broader than GPL’s distribution clause. The common alternative is Garage, SeaweedFS, or Ceph RGW, each with their own trade-offs.

MinIO’s power is its simplicity — one binary, a handful of environment variables, erasure coding that works. The operational complexity lands almost entirely at topology decisions (node count, drive layout, parity) made at deployment time. Get those right, run XFS underneath, monitor the Prometheus metrics, and it mostly runs itself.