Self-Hosted GitHub Actions Runners: Ephemeral, Scalable, and Cost-Effective CI

March 27, 2026 21 min read DevOps

devopsci-cdgithub-actionskubernetesself-hostedplatform-engineering

Contents

GitHub-hosted runners are remarkably convenient — push a commit and a fresh Ubuntu VM spins up, runs your workflow, and disappears. But at scale, that convenience gets expensive fast. The 2,000 free minutes per month on the free plan, or even the generous allowances on Team/Enterprise plans, evaporate quickly when your team ships frequently or your builds take more than a few minutes.

Self-hosted runners put that compute on your own infrastructure. Done naively — a persistent VM with the runner agent installed — they work but come with problems: stale environments, credential leakage between jobs, a single point of failure. Done properly with ephemeral Kubernetes runners via Actions Runner Controller, you get isolation, autoscaling, and significant cost savings.

This guide covers the full stack: Actions Runner Controller (ARC) deployment, runner scaling strategies, build caching, security model, and the math on when self-hosted actually saves money.

The Case for Self-Hosted Runners

Before diving into setup, it’s worth being precise about the tradeoffs.

GitHub-Hosted Runner Limitations

Cost at scale: GitHub-hosted runners bill per minute. A 10-minute build running 100 times per day costs 1,000 minutes/day — 30,000 minutes/month. On Team plan that’s $96/month just for that one workflow. Enterprise plan: ~$240/month.

Fixed resources: GitHub-hosted runners come in fixed sizes (2-core/7GB for standard Ubuntu, 4-core/16GB for larger runners). You can’t get 8-core/32GB without paying premium rates or going Enterprise.

No private network access: GitHub-hosted runners run on GitHub’s infrastructure with no path to your private services. Testing against an internal database, deploying to a VPC, or pulling from a private artifact registry requires either making those services publicly reachable or using a VPN (added complexity, potential security risk).

Slow cold starts for large dependencies: Docker layer caching, npm/pip/Go module caches — GitHub-hosted runners start fresh every time. You can cache with actions/cache, but it’s slow to restore large caches over the internet.

Limited customization: You get the pre-installed toolchain. Adding custom dependencies requires apt-get install or script setup steps in every job.

Self-Hosted Runner Advantages

Private network access: Runners on your Kubernetes cluster can reach internal services — staging databases, private registries, internal APIs — directly without exposing them.

Persistent build caches: An NFS volume or distributed cache (Redis, Minio) shared across runner pods dramatically reduces build times. A Node project that takes 4 minutes downloading npm packages takes 30 seconds with a warm cache.

Custom hardware: GPU-enabled nodes for ML model training, high-memory instances for large test suites, ARM nodes for multi-arch builds.

Cost control: EC2 Spot instances, Hetzner dedicated servers, or existing on-premise hardware — all significantly cheaper per compute-minute than GitHub’s hosted prices.

Compliance: Some organizations can’t send source code through GitHub’s infrastructure. Self-hosted runners run your code on your servers.

When GitHub-Hosted Is Better

Low build volume (under ~5,000 minutes/month)
No private network requirements
Small team without platform engineering capacity
Security requirement for fully isolated build environments (ironically, ephemeral GitHub-hosted runners provide stronger isolation than a shared self-hosted runner)

Actions Runner Controller (ARC)

ARC is the official Kubernetes operator for running GitHub Actions runners. It manages RunnerDeployment and RunnerSet custom resources that describe runner pools, handles registration/deregistration with GitHub’s API, and scales pods up and down based on workflow demand.

Architecture

GitHub Actions                    Kubernetes Cluster
     │                                    │
     │  Webhook (job queued)              │
     │──────────────────────────────────> │
     │                          ARC Controller
     │                                    │
     │                          ┌─────────┴─────────┐
     │                          │   Runner Pods      │
     │  Job dispatched          │  (ephemeral)       │
     │<─────────────────────────│  - runner agent    │
     │                          │  - dind sidecar    │
     │  Job execution           │    (optional)      │
     │<─────────────────────────│                    │
     │                          └────────────────────┘
     │  Job complete → pod terminates

ARC operates in two modes:

Webhook-based scaling (recommended): GitHub sends a webhook when a job is queued. ARC scales up a runner pod immediately, the job runs, the pod terminates. Near-zero wait time.
Polling-based scaling: ARC polls the GitHub API. Simpler setup but adds latency before a runner is available.

Installing ARC

ARC is distributed as a Helm chart:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


# Add the ARC Helm repo
helm repo add actions-runner-controller \
  https://actions-runner-controller.github.io/actions-runner-controller
helm repo update

# Create namespace
kubectl create namespace arc-systems

# Install ARC (cert-manager is a prerequisite)
helm install arc \
  --namespace arc-systems \
  --create-namespace \
  actions-runner-controller/actions-runner-controller \
  --set authSecret.create=true \
  --set authSecret.github_token="${GITHUB_PAT}"

The GitHub PAT needs these scopes:

repo (for private repositories)
admin:org (for organization-level runners)
admin:enterprise (for enterprise-level runners, optional)

For GitHub App authentication (recommended for production — better rate limits, no personal token):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


# Create GitHub App with these permissions:
# - Actions: Read
# - Administration: Read & Write (for repo-level runners)
# - Metadata: Read
# Organization permissions:
# - Self-hosted runners: Read & Write

helm install arc \
  --namespace arc-systems \
  actions-runner-controller/actions-runner-controller \
  --set "authSecret.create=true" \
  --set "authSecret.github_app_id=${APP_ID}" \
  --set "authSecret.github_app_installation_id=${INSTALLATION_ID}" \
  --set "authSecret.github_app_private_key=${PRIVATE_KEY_BASE64}"

Defining Runner Pools

RunnerDeployment (Simple)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46


# runner-deployment.yaml
apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerDeployment
metadata:
  name: default-runners
  namespace: arc-runners
spec:
  replicas: 2           # Minimum always-on runners
  template:
    spec:
      repository: myorg/myrepo   # Or: organization: myorg
      labels:
        - self-hosted
        - linux
        - x64

      # Resource requests/limits per runner pod
      resources:
        requests:
          cpu: "1"
          memory: "2Gi"
        limits:
          cpu: "4"
          memory: "8Gi"

      # Docker-in-Docker for workflows that build container images
      dockerEnabled: true

      # Runner image (customize with your tools pre-installed)
      image: summerwind/actions-runner:latest
      # Or: image: myregistry/custom-runner:v1.2.0

      # Environment variables available to all jobs
      env:
        - name: DOCKER_BUILDKIT
          value: "1"

      # Mount a cache volume
      volumeMounts:
        - name: build-cache
          mountPath: /cache

      volumes:
        - name: build-cache
          persistentVolumeClaim:
            claimName: build-cache-pvc

HorizontalRunnerAutoscaler

Scale the runner pool based on pending jobs:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19


# autoscaler.yaml
apiVersion: actions.summerwind.dev/v1alpha1
kind: HorizontalRunnerAutoscaler
metadata:
  name: default-runners-autoscaler
  namespace: arc-runners
spec:
  scaleTargetRef:
    name: default-runners

  minReplicas: 0    # Scale to zero when idle (cost savings)
  maxReplicas: 20   # Cap for cost control

  metrics:
    - type: TotalNumberOfQueuedAndInProgressWorkflowRuns
      repositoryNames:
        - myorg/myrepo

  scaleDownDelaySecondsAfterScaleOut: 300  # 5 minutes before scaling down

With minReplicas: 0, you pay zero compute when CI is idle (nights, weekends). The first job after a quiet period waits for a pod to start (~30-60 seconds), but subsequent jobs in the same burst find warm runners.

ARC Scale Sets (Newer API)

ARC v0.8+ introduced a cleaner API called Runner Scale Sets:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71


# values.yaml for the runner scale set Helm chart
githubConfigUrl: "https://github.com/myorg"

githubConfigSecret: arc-github-secret

minRunners: 0
maxRunners: 20

runnerGroup: "default"

template:
  spec:
    initContainers:
      - name: init-dind-externals
        image: ghcr.io/actions/actions-runner:latest
        command: ["cp", "-r", "-v", "/home/runner/externals/.", "/home/runner/tmpDir/"]
        volumeMounts:
          - name: dind-externals
            mountPath: /home/runner/tmpDir

    containers:
      - name: runner
        image: ghcr.io/actions/actions-runner:latest
        command: ["/home/runner/run.sh"]
        env:
          - name: DOCKER_HOST
            value: unix:///var/run/docker.sock
        resources:
          requests:
            cpu: "1"
            memory: "2Gi"
          limits:
            cpu: "4"
            memory: "8Gi"
        volumeMounts:
          - name: work
            mountPath: /home/runner/_work
          - name: dind-sock
            mountPath: /var/run
          - name: build-cache
            mountPath: /cache

      - name: dind
        image: docker:dind
        args:
          - dockerd
          - --host=unix:///var/run/docker.sock
          - --group=$(DOCKER_GROUP_GID)
        env:
          - name: DOCKER_GROUP_GID
            value: "123"
        securityContext:
          privileged: true
        volumeMounts:
          - name: work
            mountPath: /home/runner/_work
          - name: dind-sock
            mountPath: /var/run
          - name: dind-externals
            mountPath: /home/runner/externals

    volumes:
      - name: work
        emptyDir: {}
      - name: dind-sock
        emptyDir: {}
      - name: dind-externals
        emptyDir: {}
      - name: build-cache
        persistentVolumeClaim:
          claimName: build-cache-pvc

1
2
3
4
5


helm install arc-runner-set \
  --namespace arc-runners \
  --create-namespace \
  oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set \
  -f values.yaml

Targeting Self-Hosted Runners in Workflows

1
2
3
4
5
6
7
8


# .github/workflows/build.yml
jobs:
  build:
    runs-on: [self-hosted, linux, x64]   # Match your runner labels
    steps:
      - uses: actions/checkout@v4
      - name: Build
        run: make build

For org-level runners shared across repos:

1
2
3


jobs:
  test:
    runs-on: [self-hosted, linux, x64, "size:large"]

Define label sets for different runner sizes so workflows can request appropriate resources.

Custom Runner Images

The default runner image (summerwind/actions-runner or ghcr.io/actions/actions-runner) works for basic workflows. For real workloads, build a custom image with your toolchain pre-installed — this eliminates setup steps that run on every job.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43


# Dockerfile.runner
FROM ghcr.io/actions/actions-runner:latest

USER root

# Install system dependencies
RUN apt-get update && apt-get install -y \
    build-essential \
    curl \
    git \
    jq \
    unzip \
    && rm -rf /var/lib/apt/lists/*

# Install Go
ARG GO_VERSION=1.22.0
RUN curl -fsSL "https://dl.google.com/go/go${GO_VERSION}.linux-amd64.tar.gz" \
    | tar -xz -C /usr/local
ENV PATH="/usr/local/go/bin:${PATH}"

# Install Node.js
ARG NODE_VERSION=20
RUN curl -fsSL https://deb.nodesource.com/setup_${NODE_VERSION}.x | bash - \
    && apt-get install -y nodejs

# Install kubectl
ARG KUBECTL_VERSION=v1.29.0
RUN curl -fsSL "https://dl.k8s.io/release/${KUBECTL_VERSION}/bin/linux/amd64/kubectl" \
    -o /usr/local/bin/kubectl && chmod +x /usr/local/bin/kubectl

# Install Helm
ARG HELM_VERSION=v3.14.0
RUN curl -fsSL "https://get.helm.sh/helm-${HELM_VERSION}-linux-amd64.tar.gz" \
    | tar -xz --strip-components=1 -C /usr/local/bin linux-amd64/helm

# Install AWS CLI
RUN curl -fsSL "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" \
    -o /tmp/awscli.zip \
    && unzip /tmp/awscli.zip -d /tmp \
    && /tmp/aws/install \
    && rm -rf /tmp/aws /tmp/awscli.zip

USER runner

Build and push to your registry:

1
2


docker build -f Dockerfile.runner -t myregistry/ci-runner:v1.0.0 .
docker push myregistry/ci-runner:v1.0.0

Update the runner template to use this image:

1
2
3
4
5


template:
  spec:
    containers:
      - name: runner
        image: myregistry/ci-runner:v1.0.0

Versioning Runner Images

Pin runner image versions in the ARC config — don’t use latest. Use a CI pipeline to build and test runner images before rolling them out:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21


# .github/workflows/build-runner-image.yml
on:
  push:
    paths: ["runner/Dockerfile.runner"]

jobs:
  build:
    runs-on: ubuntu-latest  # Use GitHub-hosted for meta-CI
    steps:
      - uses: actions/checkout@v4
      - name: Build and test runner image
        run: |
          docker build -f runner/Dockerfile.runner -t test-runner .
          # Smoke test: can it run a basic workflow step?
          docker run --rm test-runner go version
          docker run --rm test-runner node --version
      - name: Push to registry
        run: |
          IMAGE_TAG="myregistry/ci-runner:$(git rev-parse --short HEAD)"
          docker tag test-runner "${IMAGE_TAG}"
          docker push "${IMAGE_TAG}"

Build Caching

Caching is where self-hosted runners pay the biggest dividends over GitHub-hosted. With a shared cache volume accessible to all runner pods, npm install goes from 3 minutes to 10 seconds.

Shared PVC Cache

Create a ReadWriteMany PVC (requires NFS or a distributed storage class like Longhorn):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


# build-cache-pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: build-cache-pvc
  namespace: arc-runners
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: longhorn
  resources:
    requests:
      storage: 100Gi

Mount it in runner pods at /cache. Then configure tools to use it:

1
2
3
4
5
6


# In GitHub Actions workflow
env:
  npm_config_cache: /cache/npm
  GOPATH: /cache/go
  GOCACHE: /cache/go-build
  pip_cache_dir: /cache/pip

Or use actions/cache with a local path:

1
2
3
4
5
6


- uses: actions/cache@v4
  with:
    path: /cache/npm
    key: npm-${{ hashFiles('**/package-lock.json') }}
    restore-keys: |
      npm-

Unlike the GitHub-hosted cache (which uploads/downloads over the internet), this reads from a local volume — typically 1-2 GB/s vs. 10-50 MB/s.

Docker Layer Cache

For workflows that build container images, cache Docker layers on a local registry or use BuildKit’s inline cache:

Option 1: Local Registry Cache

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37


# local-registry.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: registry-cache
  namespace: arc-runners
spec:
  replicas: 1
  selector:
    matchLabels:
      app: registry-cache
  template:
    spec:
      containers:
        - name: registry
          image: registry:2
          env:
            - name: REGISTRY_PROXY_REMOTEURL
              value: https://registry-1.docker.io
          volumeMounts:
            - name: cache
              mountPath: /var/lib/registry
      volumes:
        - name: cache
          persistentVolumeClaim:
            claimName: registry-cache-pvc
---
apiVersion: v1
kind: Service
metadata:
  name: registry-cache
  namespace: arc-runners
spec:
  selector:
    app: registry-cache
  ports:
    - port: 5000

Configure Docker daemon in runner pods to use the pull-through cache:

1
2
3
4


// /etc/docker/daemon.json in runner image
{
  "registry-mirrors": ["http://registry-cache.arc-runners.svc.cluster.local:5000"]
}

Option 2: BuildKit with Registry Cache

1
2
3
4
5
6
7
8
9


# In workflow
- name: Build and push
  uses: docker/build-push-action@v5
  with:
    context: .
    push: true
    tags: myregistry/myapp:${{ github.sha }}
    cache-from: type=registry,ref=myregistry/myapp:cache
    cache-to: type=registry,ref=myregistry/myapp:cache,mode=max

This stores BuildKit cache layers in the registry itself — works with any registry (ECR, GCR, Harbor).

Option 3: BuildKit with Local Cache

For the fastest cache (no network transfer for cache hits):

1
2
3
4
5
6
7


- name: Build
  run: |
    docker buildx build \
      --cache-from type=local,src=/cache/buildkit \
      --cache-to type=local,dest=/cache/buildkit,mode=max \
      --tag myapp:${{ github.sha }} \
      .

Go Module Cache

Go modules download once and cache in GOPATH/pkg/mod. With a shared volume:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


env:
  GOPATH: /cache/go
  GOCACHE: /cache/go-build

steps:
  - uses: actions/checkout@v4
  - name: Build
    run: go build ./...
  - name: Test
    run: go test ./...

The module cache persists across all runner pods. First run downloads modules; subsequent runs on any runner are near-instant.

Maven / Gradle Cache

1
2
3
4
5
6
7


env:
  GRADLE_USER_HOME: /cache/gradle
  MAVEN_OPTS: -Dmaven.repo.local=/cache/maven

steps:
  - name: Build with Gradle
    run: ./gradlew build

Cache Invalidation Strategy

With persistent caches, stale entries accumulate. Implement periodic cleanup:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


#!/usr/bin/env bash
# cache-cleanup.sh — run weekly via Kubernetes CronJob
# Remove npm cache entries older than 14 days
find /cache/npm -type f -atime +14 -delete

# Remove Go build cache entries older than 14 days
go clean -cache  # Respects GOCACHE env var

# Remove Docker build cache
docker builder prune --filter until=336h --force  # 14 days

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24


# cache-cleanup-cronjob.yaml
apiVersion: batch/v1
kind: CronJob
metadata:
  name: cache-cleanup
  namespace: arc-runners
spec:
  schedule: "0 2 * * 0"  # Weekly Sunday 2am
  jobTemplate:
    spec:
      template:
        spec:
          containers:
            - name: cleanup
              image: myregistry/ci-runner:latest
              command: ["/scripts/cache-cleanup.sh"]
              volumeMounts:
                - name: build-cache
                  mountPath: /cache
          volumes:
            - name: build-cache
              persistentVolumeClaim:
                claimName: build-cache-pvc
          restartPolicy: OnFailure

Security Model

Self-hosted runners introduce security considerations that don’t exist with GitHub-hosted runners.

The Core Risk: Untrusted Code

A self-hosted runner executes whatever code is in the workflow. For public repositories, a malicious PR could attempt to exfiltrate secrets, pivot to other services on the network, or compromise the runner environment.

Rule: Never use self-hosted runners for public repositories unless you trust all contributors.

If you need to CI test PRs from forks of a public repo, use GitHub-hosted runners for that workflow and self-hosted only for trusted workflows (merges to main, releases).

1
2
3
4
5
6
7
8
9


# Pattern: use GitHub-hosted for PR checks, self-hosted for deploys
jobs:
  test-pr:
    runs-on: ubuntu-latest          # GitHub-hosted, isolated
    if: github.event_name == 'pull_request'

  deploy:
    runs-on: [self-hosted, linux]   # Self-hosted, has private network access
    if: github.ref == 'refs/heads/main'

Ephemeral Runners Are Critical

With ARC, each job runs in a fresh pod that terminates after the job completes. This is the most important security property:

No credential leakage between jobs
No persistent attacker access after a compromised job
Environment state can’t accumulate

Never use persistent (non-ephemeral) runner configurations in production. A runner that runs multiple jobs in sequence can have secrets from job 1 still in memory when job 2 starts.

Secrets Isolation

Don’t mount Kubernetes secrets broadly. Use a scoped approach:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


# Only inject secrets needed by CI — not cluster admin credentials
containers:
  - name: runner
    env:
      - name: DOCKER_REGISTRY_PASSWORD
        valueFrom:
          secretKeyRef:
            name: registry-credentials
            key: password
      # Do NOT mount kubeconfig here unless the job actually deploys

For deployment jobs that need cluster access, use IRSA (IAM Roles for Service Accounts) on EKS or Workload Identity on GKE rather than mounting long-lived credentials.

Network Policies

Isolate runner pods from other cluster workloads:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41


# network-policy.yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: runner-isolation
  namespace: arc-runners
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/name: gha-runner-scale-set
  policyTypes:
    - Ingress
    - Egress
  ingress: []  # No inbound connections to runners
  egress:
    # Allow DNS
    - ports:
        - port: 53
          protocol: UDP
    # Allow HTTPS to GitHub
    - to:
        - ipBlock:
            cidr: 0.0.0.0/0
            except:
              - 10.0.0.0/8
              - 172.16.0.0/12
              - 192.168.0.0/16
      ports:
        - port: 443
    # Allow access to internal registry
    - to:
        - namespaceSelector:
            matchLabels:
              name: registry
      ports:
        - port: 5000
    # Allow access to staging namespace (for integration tests)
    - to:
        - namespaceSelector:
            matchLabels:
              name: staging

This allows runners to reach GitHub and internal services while preventing access to other cluster namespaces.

Pod Security

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


template:
  spec:
    securityContext:
      runAsNonRoot: true
      runAsUser: 1000
      fsGroup: 1000
      seccompProfile:
        type: RuntimeDefault

    containers:
      - name: runner
        securityContext:
          allowPrivilegeEscalation: false
          readOnlyRootFilesystem: false  # Runners need to write
          capabilities:
            drop: ["ALL"]

For Docker-in-Docker (dind), the dind container requires privileged: true — this is unavoidable for Docker builds. Consider using Kaniko or Buildah as alternatives that don’t require privileged mode:

1
2
3
4
5
6
7
8


# Using Kaniko instead of docker build
- name: Build container image
  run: |
    /kaniko/executor \
      --context=dir://. \
      --destination=myregistry/myapp:${{ github.sha }} \
      --cache=true \
      --cache-repo=myregistry/myapp-cache

Autoscaling Deep Dive

Webhook-Based Scaling (Recommended)

Webhook scaling reacts to job queue events immediately rather than polling:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


# values.yaml — ARC controller with webhook support
githubWebhookServer:
  enabled: true
  ingress:
    enabled: true
    annotations:
      kubernetes.io/ingress.class: nginx
    hosts:
      - host: arc-webhook.yourdomain.com
        paths:
          - path: /
            pathType: Prefix

Configure the GitHub webhook (Organization Settings → Webhooks):

Payload URL: https://arc-webhook.yourdomain.com/webhook
Content type: application/json
Secret: generate with openssl rand -hex 32
Events: Workflow jobs only

With webhook scaling, the time from “job queued” to “runner available” is typically 15-30 seconds (pod startup time) rather than 1-2 minutes with polling.

Multi-Pool Configuration

Different workflows have different resource requirements. Create multiple pools:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43


# Small pool: quick jobs, lint, unit tests
apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerDeployment
metadata:
  name: runners-small
  namespace: arc-runners
spec:
  template:
    spec:
      labels: [self-hosted, linux, size-small]
      resources:
        requests: {cpu: "1", memory: "2Gi"}
        limits: {cpu: "2", memory: "4Gi"}
---
# Large pool: integration tests, builds with heavy compilation
apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerDeployment
metadata:
  name: runners-large
  namespace: arc-runners
spec:
  template:
    spec:
      labels: [self-hosted, linux, size-large]
      resources:
        requests: {cpu: "4", memory: "8Gi"}
        limits: {cpu: "8", memory: "16Gi"}
---
# GPU pool: ML model training (if you have GPU nodes)
apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerDeployment
metadata:
  name: runners-gpu
  namespace: arc-runners
spec:
  template:
    spec:
      labels: [self-hosted, linux, gpu]
      nodeSelector:
        accelerator: nvidia-tesla-t4
      resources:
        limits:
          nvidia.com/gpu: "1"

Workflows target specific pools:

1
2
3
4
5
6
7


jobs:
  lint:
    runs-on: [self-hosted, linux, size-small]
  integration-test:
    runs-on: [self-hosted, linux, size-large]
  train-model:
    runs-on: [self-hosted, linux, gpu]

Node Autoscaling with Cluster Autoscaler

Combine ARC’s runner scaling with Kubernetes Cluster Autoscaler for true elastic compute:

ARC scales runner pods up (HorizontalRunnerAutoscaler)
Cluster Autoscaler detects unschedulable pods and provisions new nodes
Runner pods schedule on new nodes
After jobs complete, ARC scales pods down
Cluster Autoscaler terminates idle nodes

For cost efficiency on AWS, use Spot instances for the CI node group:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


# eks-nodegroup.yaml (Spot instances for CI)
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
managedNodeGroups:
  - name: ci-runners-spot
    instanceTypes: ["m5.2xlarge", "m5a.2xlarge", "m4.2xlarge"]
    spot: true
    minSize: 0
    maxSize: 20
    labels:
      purpose: ci
    taints:
      - key: ci-runner
        value: "true"
        effect: NoSchedule

Runner pods tolerate the taint to land on CI nodes; regular workloads don’t, keeping CI compute separate.

Observability

Runner Metrics

ARC exposes Prometheus metrics:

1
2
3
4
5


# In ARC Helm values
metrics:
  serviceAnnotations:
    prometheus.io/scrape: "true"
    prometheus.io/port: "8443"

Key metrics:

Metric	Description
`controller_runtime_reconcile_total`	ARC reconciliation count
`runner_status`	Current status per runner
`horizontal_runner_autoscaler_desired_replicas`	Target replica count
`horizontal_runner_autoscaler_current_replicas`	Actual replica count

GitHub Actions Usage Dashboard

Query GitHub’s API for runner utilization:

1
2
3
4


# Workflow run timing
gh api repos/myorg/myrepo/actions/runs \
  --paginate \
  --jq '.workflow_runs[] | {name: .name, duration: ((.updated_at | fromdateiso8601) - (.run_started_at | fromdateiso8601))}'

Build a Grafana dashboard tracking:

Average job wait time (queue → runner available)
Average job duration per workflow
Runner pool utilization (% of time runners are busy)
Cache hit rate (compare build times with/without cache)

Alerting

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19


# Prometheus alert rules
groups:
  - name: arc-runners
    rules:
      - alert: RunnerPoolExhausted
        expr: |
          horizontal_runner_autoscaler_desired_replicas
          >= horizontal_runner_autoscaler_max_replicas
        for: 5m
        annotations:
          summary: "Runner pool at maximum capacity"
          description: "Jobs may be queuing. Consider increasing maxReplicas."

      - alert: RunnerJobStuck
        expr: |
          time() - runner_last_registration_time > 1800
        for: 5m
        annotations:
          summary: "Runner hasn't registered in 30 minutes"

Cost Analysis

Example: Mid-Size Engineering Team

Profile: 20 developers, ~500 workflow runs/day, average 8 minutes/run, mix of test and deploy workflows.

GitHub-hosted cost:

500 runs × 8 min = 4,000 minutes/day
4,000 × 30 = 120,000 minutes/month
GitHub Team plan: $0.008/minute for Linux
120,000 × $0.008 = $960/month

Self-hosted on AWS (EC2):

Resource	Spec	Cost
EKS cluster	2 m5.large control plane	$144/month
On-demand nodes (baseline)	2× m5.2xlarge (8 vCPU, 32GB)	$277/month
Spot nodes (peak scaling)	~4× m5.2xlarge avg utilized	~$185/month
EBS storage (cache)	200GB gp3	$16/month
Total		~$622/month

Savings: ~$338/month ($4,056/year) — a 35% reduction with better cache performance and private network access.

At higher volumes, the savings grow faster because GitHub charges per-minute while your own infrastructure has largely fixed costs.

Break-even point: Roughly 50,000 minutes/month (GitHub Team pricing). Below that, GitHub-hosted is likely cheaper when accounting for the engineering time to operate ARC.

Spot Instance Savings

Using Spot instances for the majority of CI compute (which tolerates interruption since failed jobs can retry):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


# Spot node group with on-demand fallback
managedNodeGroups:
  - name: ci-spot
    instanceTypes: ["m5.2xlarge", "m5a.2xlarge", "m4.2xlarge"]
    spot: true
    minSize: 0
    maxSize: 15
  - name: ci-ondemand
    instanceTypes: ["m5.2xlarge"]
    spot: false
    minSize: 0
    maxSize: 5  # Fallback when spot unavailable

Spot instances are typically 60-80% cheaper than on-demand. A $277/month on-demand bill becomes ~$90/month on Spot for the scaling nodes.

Putting It All Together: Production Setup Checklist

Infrastructure:
  ☐ Kubernetes cluster with autoscaling node group for CI
  ☐ Spot instance node group for cost efficiency
  ☐ NFS or distributed storage class for shared cache PVC
  ☐ Private container registry for runner images
  ☐ Network policies isolating runner namespace

ARC:
  ☐ ARC installed via Helm with GitHub App auth (not PAT)
  ☐ Webhook server configured for fast scaling
  ☐ Multiple runner pools (small, large, GPU if needed)
  ☐ HorizontalRunnerAutoscaler with minReplicas: 0
  ☐ ScaleDownDelay tuned to your burst pattern

Runner Images:
  ☐ Custom image with all toolchain pre-installed
  ☐ Image versioned and built via automated pipeline
  ☐ Smoke tests for each tool in the image

Caching:
  ☐ Shared PVC mounted at /cache in runner pods
  ☐ Tool caches (npm, Go, pip, Maven) pointing to /cache
  ☐ Docker layer cache (registry or local)
  ☐ Weekly cache cleanup CronJob

Security:
  ☐ Ephemeral runners (no persistent runners in production)
  ☐ Self-hosted runners NOT used for public repo PRs
  ☐ Network policies restricting runner egress
  ☐ No privileged containers except dind sidecar
  ☐ Secrets via Kubernetes secrets, not env files

Observability:
  ☐ Prometheus scraping ARC metrics
  ☐ Grafana dashboard for runner utilization
  ☐ Alerts for pool exhaustion and stuck jobs
  ☐ GitHub Actions usage tracked and reviewed monthly

The operational overhead of running ARC is real — you’re taking on infrastructure management that GitHub handles transparently. But for teams with regular CI load, private network requirements, or specialized compute needs, it pays off quickly. The combination of ephemeral pods, shared caching, and Spot instance scaling gets you CI infrastructure that’s faster, cheaper, and more flexible than GitHub-hosted runners while maintaining the security properties that make ephemeral execution trustworthy.