Object Storage Internals: How S3-Compatible Storage Works and Self-Hosting with MinIO

March 26, 2026 17 min read Data & Databases

storages3minioobject-storageinfrastructureself-hosting

Contents

Object storage has become the default substrate for data at rest. Backups, ML training data, media files, data lake archives, container image layers, Terraform state — all of it flows through S3-compatible APIs. Most engineers interact with object storage as a black box: you PUT an object, you GET it back, done. But understanding how it actually works — consistency guarantees, durability mechanisms, performance characteristics — makes you dramatically better at choosing the right architecture and debugging the inevitable surprises.

This guide covers the internals of S3-compatible object storage and walks through running MinIO in production as a self-hosted alternative.

What Makes Object Storage Different

Object storage is not a filesystem and not a block device. Understanding what it is helps you understand its trade-offs.

The Object Model

Everything in object storage is an object: a blob of bytes with a globally unique key, metadata, and no internal structure visible to the storage system. Objects live in buckets (flat namespaces — there are no real directories, just key prefixes that clients interpret as paths). There’s no rename, no append, no partial write — you write entire objects atomically.

This design enables horizontal scalability that filesystems can’t match. A filesystem needs a consistent view of directory entries, which requires coordination. Object storage doesn’t — each object is independent, and the namespace can be sharded across thousands of nodes by key hash.

HTTP as the Protocol

S3’s wire protocol is HTTP. Operations map directly to HTTP verbs:

Operation	HTTP Method	Path
Create/replace object	`PUT`	`/bucket/key`
Read object	`GET`	`/bucket/key`
Delete object	`DELETE`	`/bucket/key`
List objects	`GET`	`/bucket?list-type=2&prefix=...`
Copy object	`PUT` + `x-amz-copy-source` header	`/bucket/dest-key`
Head (metadata only)	`HEAD`	`/bucket/key`
Initiate multipart upload	`POST`	`/bucket/key?uploads`

Authentication uses AWS Signature Version 4 — HMAC-SHA256 signed requests with AccessKeyID and SecretAccessKey. This is the de facto standard that every S3-compatible implementation supports.

Consistency Models

S3’s consistency model has evolved and is frequently misunderstood.

Strong Read-After-Write Consistency (2020+)

Amazon S3 now provides strong read-after-write consistency for all operations — PUTs, GETs, DELETEs, and LIST. After a successful PUT, any subsequent GET or LIST will see the new object. This replaced the eventual consistency model that existed before December 2020, when LIST could lag behind recent PUTs.

What this means in practice:

After PUT object returns 200, all subsequent GET calls return the new version
After DELETE returns 204, the object is immediately invisible to GET and LIST
After PUT object (overwrite), the old version is immediately replaced — no window where the old version is returned

What strong consistency does NOT mean:

It doesn’t mean transactions across multiple objects — two concurrent writers can both succeed, and the last write wins per-object
It doesn’t mean atomic multi-object operations — there’s no PUT /bucket/a AND /bucket/b atomically
It doesn’t mean linearizability across all clients globally in the CAP theorem sense — just that each object’s history is consistent from any client’s perspective

Self-Hosted Consistency

When you run MinIO or Ceph, consistency depends on your configuration. MinIO provides strong read-after-write consistency within a single cluster. Across geo-replicated sites, you get eventual consistency for replication — the primary site is always consistent, replicas lag.

Durability: Erasure Coding

S3 claims 99.999999999% (11 nines) durability. How? Not through replication — through erasure coding.

How Erasure Coding Works

Erasure coding is a mathematical technique that splits an object into k data shards and m parity shards, such that any k shards out of k+m total are sufficient to reconstruct the original object. You can lose any m shards and still recover.

Common configuration: EC:4+2 (4 data shards + 2 parity shards = 6 total). Any 4 of the 6 shards reconstruct the data. You can lose any 2 drives/nodes simultaneously.

Original object: 100MB

EC:4+2 encoding:
  Shard 1: 25MB (data)    → Drive 1
  Shard 2: 25MB (data)    → Drive 2
  Shard 3: 25MB (data)    → Drive 3
  Shard 4: 25MB (data)    → Drive 4
  Shard 5: 25MB (parity)  → Drive 5
  Shard 6: 25MB (parity)  → Drive 6

Storage overhead: 150MB total for 100MB data (1.5x)
Fault tolerance: any 2 drives can fail

Compare to 3-way replication: 300MB for 100MB data (3x overhead), but only tolerates 2 node failures if you read from any replica (and you can’t read if the primary fails in some configurations).

Erasure coding gives better fault tolerance per unit of storage overhead. This is why object storage is cheaper than replicated block storage at scale.

Minimum Drive/Node Requirements

MinIO requires a minimum of 4 drives for erasure coding to activate. Below that threshold, it runs in single-disk mode with no fault tolerance. For production:

Minimum: 4 drives (EC:2+2 — only tolerates 1 failure, not recommended)
Recommended: 8 drives minimum (EC:4+4 — tolerates 4 failures)
Standard production: 16 drives across 4 servers (4 drives per server)

Multipart Uploads

Large file uploads over HTTP have a reliability problem: if you lose the connection halfway through a 5GB upload, you start over. Multipart upload solves this.

The Protocol

1. InitiateMultipartUpload → UploadId

2. UploadPart (part 1, UploadId) → ETag
   UploadPart (part 2, UploadId) → ETag
   UploadPart (part 3, UploadId) → ETag
   (parts can be uploaded in parallel, in any order)

3. CompleteMultipartUpload(UploadId, [part1 ETag, part2 ETag, part3 ETag])
   → Final object assembled

OR

AbortMultipartUpload(UploadId)  → Cleanup

Parts can be uploaded concurrently — a 1GB file with 100MB parts can be uploaded with 10 parallel streams. Each part is individually checksummed. If a part upload fails, only that part needs to be retried. The final CompleteMultipartUpload assembles the parts server-side atomically — the object appears in its final form or not at all.

Part size constraints (AWS S3):

Minimum part size: 5MB (except the last part)
Maximum part size: 5GB
Maximum number of parts: 10,000
Maximum object size: 5TB

Multipart Upload with the AWS SDK

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28


import boto3
from boto3.s3.transfer import TransferConfig

s3 = boto3.client(
    's3',
    endpoint_url='http://minio:9000',    # MinIO endpoint
    aws_access_key_id='minioadmin',
    aws_secret_access_key='minioadmin',
)

# TransferConfig handles multipart automatically
config = TransferConfig(
    multipart_threshold=8 * 1024 * 1024,   # Use multipart above 8MB
    max_concurrency=10,                      # 10 parallel part uploads
    multipart_chunksize=8 * 1024 * 1024,    # 8MB per part
    use_threads=True,
)

# This automatically uses multipart for large files
s3.upload_file(
    '/path/to/large-file.tar.gz',
    'my-bucket',
    'backups/large-file.tar.gz',
    Config=config,
    ExtraArgs={
        'ChecksumAlgorithm': 'SHA256',  # Enable end-to-end checksum
    }
)

Manual Multipart Upload

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50


import boto3
import os

def multipart_upload(file_path: str, bucket: str, key: str, part_size: int = 100 * 1024 * 1024):
    """Upload a large file using multipart upload with manual control."""
    s3 = boto3.client('s3')

    # Step 1: Initiate
    response = s3.create_multipart_upload(Bucket=bucket, Key=key)
    upload_id = response['UploadId']
    print(f"Initiated multipart upload: {upload_id}")

    parts = []
    part_number = 1

    try:
        with open(file_path, 'rb') as f:
            while True:
                data = f.read(part_size)
                if not data:
                    break

                # Step 2: Upload each part
                response = s3.upload_part(
                    Bucket=bucket,
                    Key=key,
                    PartNumber=part_number,
                    UploadId=upload_id,
                    Body=data,
                )
                etag = response['ETag']
                parts.append({'PartNumber': part_number, 'ETag': etag})
                print(f"Uploaded part {part_number}: {etag}")
                part_number += 1

        # Step 3: Complete
        response = s3.complete_multipart_upload(
            Bucket=bucket,
            Key=key,
            UploadId=upload_id,
            MultipartUpload={'Parts': parts},
        )
        print(f"Upload complete: {response['Location']}")
        return response['ETag']

    except Exception as e:
        # Always abort on failure to avoid orphaned parts (you pay for them!)
        print(f"Upload failed: {e}. Aborting...")
        s3.abort_multipart_upload(Bucket=bucket, Key=key, UploadId=upload_id)
        raise

Cleaning Up Incomplete Multipart Uploads

Aborted or interrupted multipart uploads leave parts in storage that you pay for but can’t access. Always configure lifecycle rules to clean them up:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


s3.put_bucket_lifecycle_configuration(
    Bucket='my-bucket',
    LifecycleConfiguration={
        'Rules': [{
            'ID': 'abort-incomplete-multipart-uploads',
            'Status': 'Enabled',
            'Filter': {'Prefix': ''},
            'AbortIncompleteMultipartUpload': {'DaysAfterInitiation': 7},
        }]
    }
)

Lifecycle Policies

Lifecycle rules automate object management: transition objects to cheaper storage classes, expire old versions, clean up delete markers.

Common Lifecycle Patterns

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40


s3.put_bucket_lifecycle_configuration(
    Bucket='data-lake',
    LifecycleConfiguration={
        'Rules': [
            # Transition recent data to cheaper storage, expire old data
            {
                'ID': 'tiered-storage',
                'Status': 'Enabled',
                'Filter': {'Prefix': 'logs/'},
                'Transitions': [
                    # Standard → Infrequent Access after 30 days
                    {'Days': 30, 'StorageClass': 'STANDARD_IA'},
                    # → Glacier after 90 days
                    {'Days': 90, 'StorageClass': 'GLACIER'},
                ],
                # Delete after 2 years
                'Expiration': {'Days': 730},
            },
            # Clean up old versions (versioning enabled)
            {
                'ID': 'expire-old-versions',
                'Status': 'Enabled',
                'Filter': {'Prefix': ''},
                'NoncurrentVersionTransitions': [
                    {'NoncurrentDays': 30, 'StorageClass': 'GLACIER'},
                ],
                'NoncurrentVersionExpiration': {'NoncurrentDays': 90},
                # Clean up delete markers when no versions remain
                'ExpiredObjectDeleteMarker': True,
            },
            # Abort incomplete multipart uploads
            {
                'ID': 'cleanup-multipart',
                'Status': 'Enabled',
                'Filter': {'Prefix': ''},
                'AbortIncompleteMultipartUpload': {'DaysAfterInitiation': 3},
            },
        ]
    }
)

Versioning + Lifecycle for Backup Retention

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20


# Enable versioning first
s3.put_bucket_versioning(
    Bucket='backups',
    VersioningConfiguration={'Status': 'Enabled'}
)

# Lifecycle: keep last 30 days at full speed, then expire
s3.put_bucket_lifecycle_configuration(
    Bucket='backups',
    LifecycleConfiguration={
        'Rules': [{
            'ID': 'backup-retention',
            'Status': 'Enabled',
            'Filter': {'Prefix': ''},
            'NoncurrentVersionExpiration': {'NoncurrentDays': 30},
            'ExpiredObjectDeleteMarker': True,
            'AbortIncompleteMultipartUpload': {'DaysAfterInitiation': 1},
        }]
    }
)

Presigned URLs

Presigned URLs let you share temporary access to private objects without exposing credentials:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22


# Generate a URL valid for 1 hour
url = s3.generate_presigned_url(
    'get_object',
    Params={'Bucket': 'private-bucket', 'Key': 'sensitive/report.pdf'},
    ExpiresIn=3600,
)
# https://bucket.s3.amazonaws.com/key?X-Amz-Algorithm=...&X-Amz-Expires=3600&...

# Presigned POST for browser-based uploads (avoids proxying through your server)
response = s3.generate_presigned_post(
    Bucket='uploads',
    Key='user-uploads/${filename}',
    Fields={
        'Content-Type': 'image/jpeg',
    },
    Conditions=[
        ['content-length-range', 1, 10 * 1024 * 1024],  # Max 10MB
        {'Content-Type': 'image/jpeg'},
    ],
    ExpiresIn=300,
)
# Returns {'url': ..., 'fields': {...}} — POST directly from browser

Self-Hosting with MinIO

MinIO is the most widely deployed self-hosted S3-compatible object store. It’s written in Go, ships as a single binary, and is fully compatible with the AWS S3 API. Used at scale by GitLab, Cloudflare, and many others.

Single-Node Setup (Development)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


# Docker
docker run -d \
  --name minio \
  -p 9000:9000 \
  -p 9001:9001 \
  -v /data/minio:/data \
  -e MINIO_ROOT_USER=minioadmin \
  -e MINIO_ROOT_PASSWORD=changeme123! \
  quay.io/minio/minio server /data \
    --console-address ":9001"

# Access:
# API:     http://localhost:9000
# Console: http://localhost:9001

Production: Multi-Node Erasure Coding

For production MinIO with fault tolerance, you need at least 4 drives. Recommended: 4 servers × 4 drives each = 16 drives total.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78


# docker-compose.yml — 4-node MinIO cluster
# Run on 4 separate hosts or adjust for single-host testing

version: '3.8'

x-minio-common: &minio-common
  image: quay.io/minio/minio:latest
  command: server
    --console-address ":9001"
    http://minio{1...4}/data{1...4}
  environment:
    MINIO_ROOT_USER: minioadmin
    MINIO_ROOT_PASSWORD: ${MINIO_ROOT_PASSWORD}
    MINIO_VOLUMES: "http://minio{1...4}/data{1...4}"
    MINIO_SITE_NAME: "production-cluster"
  healthcheck:
    test: ["CMD", "mc", "ready", "local"]
    interval: 5s
    timeout: 5s
    retries: 5
  networks:
    - minio-net

services:
  minio1:
    <<: *minio-common
    hostname: minio1
    volumes:
      - /data1/minio:/data1
      - /data2/minio:/data2
      - /data3/minio:/data3
      - /data4/minio:/data4

  minio2:
    <<: *minio-common
    hostname: minio2
    volumes:
      - /data1/minio:/data1
      - /data2/minio:/data2
      - /data3/minio:/data3
      - /data4/minio:/data4

  minio3:
    <<: *minio-common
    hostname: minio3
    volumes:
      - /data1/minio:/data1
      - /data2/minio:/data2
      - /data3/minio:/data3
      - /data4/minio:/data4

  minio4:
    <<: *minio-common
    hostname: minio4
    volumes:
      - /data1/minio:/data1
      - /data2/minio:/data2
      - /data3/minio:/data3
      - /data4/minio:/data4

  nginx:
    image: nginx:alpine
    ports:
      - "9000:9000"
      - "9001:9001"
    volumes:
      - ./nginx.conf:/etc/nginx/nginx.conf:ro
    depends_on:
      - minio1
      - minio2
      - minio3
      - minio4
    networks:
      - minio-net

networks:
  minio-net:
    driver: bridge

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48


# nginx.conf — load balance across MinIO nodes
upstream minio_api {
    least_conn;
    server minio1:9000;
    server minio2:9000;
    server minio3:9000;
    server minio4:9000;
}

upstream minio_console {
    least_conn;
    server minio1:9001;
    server minio2:9001;
    server minio3:9001;
    server minio4:9001;
}

server {
    listen 9000;

    ignore_invalid_headers off;
    client_max_body_size 0;        # No upload size limit
    proxy_buffering off;
    proxy_request_buffering off;

    location / {
        proxy_set_header Host $http_host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_connect_timeout 300;
        proxy_http_version 1.1;
        proxy_set_header Connection "";
        chunked_transfer_encoding off;
        proxy_pass http://minio_api;
    }
}

server {
    listen 9001;
    location / {
        proxy_set_header Host $http_host;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_pass http://minio_console;
    }
}

MinIO on Kubernetes

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42


# Using the MinIO Operator (production recommended)
# Install operator first:
# kubectl apply -k github.com/minio/operator

apiVersion: minio.min.io/v2
kind: Tenant
metadata:
  name: minio-production
  namespace: minio-tenant
spec:
  image: quay.io/minio/minio:latest

  # Pool of 4 servers, 4 drives each = 16 drives total
  pools:
    - name: pool-0
      servers: 4
      volumesPerServer: 4
      volumeClaimTemplate:
        metadata:
          name: data
        spec:
          accessModes: [ReadWriteOnce]
          resources:
            requests:
              storage: 1Ti         # 1TB per drive
          storageClassName: fast-nvme

  # TLS via cert-manager
  requestAutoCert: true

  # Bucket configuration
  buckets:
    - name: data-lake
    - name: backups
    - name: ml-datasets

  # Users
  users:
    - name: minio-user-secret   # Reference to a Secret with MINIO_ACCESS_KEY + MINIO_SECRET_KEY

  # Prometheus metrics
  prometheusOperator: true

MinIO Client (mc) — Essential Commands

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45


# Install mc
curl -O https://dl.min.io/client/mc/release/linux-amd64/mc
chmod +x mc && sudo mv mc /usr/local/bin/

# Configure alias
mc alias set local http://localhost:9000 minioadmin changeme123!
mc alias set prod https://minio.example.com ACCESS_KEY SECRET_KEY

# Bucket operations
mc mb local/my-bucket
mc ls local/
mc ls local/my-bucket/
mc ls --recursive local/my-bucket/prefix/

# Object operations
mc cp ./file.txt local/my-bucket/path/file.txt
mc cp local/my-bucket/file.txt ./downloaded.txt
mc rm local/my-bucket/file.txt
mc rm --recursive --force local/my-bucket/old-prefix/

# Sync (like rsync for object storage)
mc mirror /local/data/ local/my-bucket/data/
mc mirror --remove local/my-bucket/ prod/my-bucket/   # Sync between clusters

# Policies
mc anonymous set public local/my-bucket/public/
mc anonymous set private local/my-bucket/

# Bucket policy (JSON)
mc anonymous set-json policy.json local/my-bucket

# Watch for events
mc watch local/my-bucket

# Performance benchmark
mc support perf object --duration 30s local/

# Disk usage
mc du local/my-bucket
mc du --depth 2 local/my-bucket/

# Admin operations
mc admin info local
mc admin heal local/my-bucket      # Trigger healing scan
mc admin prometheus generate local # Generate Prometheus bearer token

Bucket Policies and IAM

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39


import json
import boto3

s3 = boto3.client('s3', endpoint_url='http://localhost:9000',
                  aws_access_key_id='minioadmin',
                  aws_secret_access_key='changeme123!')

# Public read policy for a prefix
policy = {
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "PublicRead",
            "Effect": "Allow",
            "Principal": "*",
            "Action": ["s3:GetObject"],
            "Resource": ["arn:aws:s3:::my-bucket/public/*"]
        }
    ]
}

s3.put_bucket_policy(
    Bucket='my-bucket',
    Policy=json.dumps(policy)
)

# CORS configuration for browser uploads
s3.put_bucket_cors(
    Bucket='my-bucket',
    CORSConfiguration={
        'CORSRules': [{
            'AllowedHeaders': ['*'],
            'AllowedMethods': ['GET', 'PUT', 'POST', 'DELETE', 'HEAD'],
            'AllowedOrigins': ['https://app.example.com'],
            'ExposeHeaders': ['ETag'],
            'MaxAgeSeconds': 3000,
        }]
    }
)

Server-Side Encryption

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24


# SSE-S3: Server managed keys (MinIO manages keys)
s3.put_object(
    Bucket='secure-bucket',
    Key='sensitive.dat',
    Body=b'secret data',
    ServerSideEncryption='AES256',
)

# SSE-C: Customer provided keys (you manage keys, never stored by MinIO)
import os, base64
key = os.urandom(32)  # 256-bit AES key
key_b64 = base64.b64encode(key).decode()

import hashlib
key_md5 = base64.b64encode(hashlib.md5(key).digest()).decode()

s3.put_object(
    Bucket='secure-bucket',
    Key='sensitive.dat',
    Body=b'secret data',
    SSECustomerAlgorithm='AES256',
    SSECustomerKey=key_b64,
    SSECustomerKeyMD5=key_md5,
)

Performance Characteristics and Tuning

Throughput vs Latency

Object storage is optimized for throughput, not latency:

First-byte latency: typically 10–100ms (network round trip + metadata lookup)
Throughput: can saturate network bandwidth for large objects
Small objects (< 128KB): latency-dominated — avoid many small objects
Large objects (> 1MB): throughput-dominated — use multipart for > 8MB

Key Performance Patterns

Avoid sequential small writes: Writing 10,000 × 10KB objects is far slower than 100 × 1MB objects. If your workload produces small objects, aggregate them first (e.g., write to local disk, batch, then upload).

Parallelize GET operations: When downloading many objects, use a thread pool. Object storage handles high-concurrency requests well.

Use byte-range GETs for partial reads: No need to download a 10GB file to read a 1MB slice:

1
2
3
4
5


response = s3.get_object(
    Bucket='data-lake',
    Key='huge-file.parquet',
    Range='bytes=0-1048575',  # First 1MB only
)

Prefix distribution: S3 (and MinIO) partition the keyspace by prefix. Writing millions of objects with the same prefix (e.g., logs/2026-03-26/) can create hotspots on old S3. Modern S3 and MinIO handle this well, but adding a random prefix hash still improves throughput in extreme cases.

Connection pooling: Reuse HTTP connections. The AWS SDK does this automatically; if building custom integrations, use a persistent HTTP client.

Monitoring MinIO

1
2
3
4
5
6
7
8
9


# Prometheus scrape config for MinIO
scrape_configs:
  - job_name: minio
    metrics_path: /minio/v2/metrics/cluster
    scheme: http
    static_configs:
      - targets: ['minio:9000']
    authorization:
      credentials: <bearer-token-from-mc-admin-prometheus-generate>

Key metrics to alert on:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


# Disk usage (alert when > 80%)
minio_node_disk_used_bytes / minio_node_disk_total_bytes > 0.8

# Object count growth rate
rate(minio_bucket_objects_count[1h])

# Request error rate
rate(minio_s3_requests_errors_total[5m]) / rate(minio_s3_requests_total[5m]) > 0.01

# Healing activity (data being reconstructed — indicates recent drive failure)
minio_heal_objects_heal_total > 0

# Offline nodes
minio_cluster_nodes_offline_total > 0

# Replication lag (for geo-replicated setups)
minio_replication_pending_bytes > 1073741824  # > 1GB pending

Quick Reference

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38


# Single-node MinIO (dev/test)
docker run -p 9000:9000 -p 9001:9001 \
  -e MINIO_ROOT_USER=admin \
  -e MINIO_ROOT_PASSWORD=password123 \
  -v /data:/data \
  quay.io/minio/minio server /data --console-address :9001

# mc alias setup
mc alias set local http://localhost:9000 admin password123

# Create bucket with versioning
mc mb local/my-bucket
mc version enable local/my-bucket

# Sync local → bucket
mc mirror /local/path/ local/my-bucket/

# View storage usage
mc du local/my-bucket

# Admin cluster info
mc admin info local

# Python S3 client for MinIO
pip install boto3

import boto3
s3 = boto3.client('s3',
  endpoint_url='http://localhost:9000',
  aws_access_key_id='admin',
  aws_secret_access_key='password123',
  region_name='us-east-1',  # Required but ignored by MinIO
)

s3.create_bucket(Bucket='test')
s3.put_object(Bucket='test', Key='hello.txt', Body=b'Hello World')
obj = s3.get_object(Bucket='test', Key='hello.txt')
print(obj['Body'].read())

When to Self-Host vs Use S3

Self-hosting with MinIO makes sense when:

Data sovereignty: regulations require data to stay on-premises or in specific jurisdictions
Cost at scale: at hundreds of TBs, S3 egress costs and storage costs can exceed hardware costs
Air-gapped environments: no internet connectivity available
Latency: local MinIO eliminates WAN round trips for on-prem workloads
Integration: existing Ceph or storage infrastructure

Stick with S3 or equivalent managed services when:

Operational overhead: running distributed storage is non-trivial — disk failures, healing, upgrades
Low data volume: S3 pricing is very reasonable at < 50TB
No dedicated storage expertise on the team
Durability requirements: managed S3 has a better track record than most self-hosted setups

The S3 API compatibility of MinIO means you can develop and test locally, then deploy to S3 in production — or vice versa — with zero code changes. That flexibility alone makes MinIO worth knowing.