Most teams start with GitHub Actions by copying a workflow YAML, wiring up a few steps, and calling it done. That works fine for a single repo. But as your organization grows — more services, more repos, more engineers — those ad-hoc workflows become a maintenance nightmare: duplicated logic across dozens of repos, inconsistent security practices, cache misses that slow everyone down, and flaky tests nobody can diagnose.
This guide covers the patterns that make Actions scale: reusable workflows that centralize your CI standards, composite actions that compose building blocks, matrix strategies that parallelize intelligently, self-hosted runners that give you GPU access or private network reach, and the caching and security practices that turn a slow, fragile pipeline into a reliable engineering asset.
Reusable Workflows
Reusable workflows let one workflow call another, just like a function call. The called workflow runs in its own job context but shares the caller’s secrets (if explicitly passed) and outputs results back.
Defining a Reusable Workflow
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
|
# .github/workflows/reusable-build-test.yml
# This lives in your central "platform" or "devops" repo
name: Reusable — Build and Test
on:
workflow_call:
inputs:
go-version:
description: "Go version to use"
type: string
default: "1.22"
working-directory:
description: "Directory containing the Go module"
type: string
default: "."
run-integration-tests:
description: "Run integration tests (requires Docker)"
type: boolean
default: false
artifact-name:
description: "Name for the build artifact"
type: string
required: true
secrets:
SONAR_TOKEN:
description: "SonarCloud token for code quality analysis"
required: false
outputs:
image-tag:
description: "Docker image tag that was built"
value: ${{ jobs.build.outputs.image-tag }}
test-passed:
description: "Whether all tests passed"
value: ${{ jobs.test.outputs.passed }}
jobs:
test:
name: Test
runs-on: ubuntu-latest
outputs:
passed: ${{ steps.test.outcome == 'success' }}
defaults:
run:
working-directory: ${{ inputs.working-directory }}
steps:
- uses: actions/checkout@v4
- uses: actions/setup-go@v5
with:
go-version: ${{ inputs.go-version }}
cache-dependency-path: ${{ inputs.working-directory }}/go.sum
- name: Run unit tests
id: test
run: |
go test ./... \
-race \
-coverprofile=coverage.out \
-covermode=atomic \
-timeout 10m
- name: Upload coverage
uses: actions/upload-artifact@v4
with:
name: coverage-${{ inputs.artifact-name }}
path: ${{ inputs.working-directory }}/coverage.out
- name: SonarCloud analysis
if: secrets.SONAR_TOKEN != ''
uses: SonarSource/sonarcloud-github-action@master
env:
SONAR_TOKEN: ${{ secrets.SONAR_TOKEN }}
with:
projectBaseDir: ${{ inputs.working-directory }}
- name: Integration tests
if: inputs.run-integration-tests
run: |
docker compose -f docker-compose.test.yml up -d
go test ./integration/... -tags=integration -timeout 20m
docker compose -f docker-compose.test.yml down
build:
name: Build
needs: test
runs-on: ubuntu-latest
outputs:
image-tag: ${{ steps.meta.outputs.tags }}
steps:
- uses: actions/checkout@v4
- name: Docker meta
id: meta
uses: docker/metadata-action@v5
with:
images: ghcr.io/${{ github.repository }}
tags: |
type=sha,prefix=,format=short
type=ref,event=branch
type=semver,pattern={{version}}
- uses: docker/setup-buildx-action@v3
- uses: docker/login-action@v3
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- uses: docker/build-push-action@v5
with:
context: ${{ inputs.working-directory }}
push: true
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}
cache-from: type=gha
cache-to: type=gha,mode=max
provenance: true
sbom: true
|
Calling a Reusable Workflow
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
|
# .github/workflows/ci.yml — in your application repo
name: CI
on:
push:
branches: [main]
pull_request:
jobs:
# Call the centralized workflow — one line instead of 80
build-and-test:
uses: my-org/platform/.github/workflows/reusable-build-test.yml@main
with:
go-version: "1.22"
working-directory: "./services/api"
run-integration-tests: ${{ github.event_name == 'push' }}
artifact-name: api-service
secrets:
SONAR_TOKEN: ${{ secrets.SONAR_TOKEN }}
# Jobs that need the outputs from the reusable workflow
deploy-staging:
needs: build-and-test
if: github.ref == 'refs/heads/main'
runs-on: ubuntu-latest
steps:
- name: Deploy image
run: |
echo "Deploying: ${{ needs.build-and-test.outputs.image-tag }}"
kubectl set image deployment/api api=${{ needs.build-and-test.outputs.image-tag }}
|
Workflow Versioning Strategy
Pin called workflows to a specific ref — never @main in production:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
|
# Good: pinned to a release tag
uses: my-org/platform/.github/workflows/reusable-deploy.yml@v2.3.1
# Acceptable: pinned to a commit SHA
uses: my-org/platform/.github/workflows/reusable-deploy.yml@a3f8c2d
# Risky: floating tag (can break if the platform team makes changes)
uses: my-org/platform/.github/workflows/reusable-deploy.yml@main
# Automate updates with Dependabot:
# .github/dependabot.yml
version: 2
updates:
- package-ecosystem: "github-actions"
directory: "/"
schedule:
interval: "weekly"
groups:
actions:
patterns: ["*"]
|
Composite Actions
While reusable workflows operate at the job level (separate runner, separate context), composite actions are step-level building blocks. They run in the same job as the caller, which makes them ideal for encapsulating a sequence of steps that needs to share the caller’s environment, files, and context.
Creating a Composite Action
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
|
# actions/setup-node-pnpm/action.yml
# A composite action that sets up Node.js + pnpm with proper caching
name: Setup Node.js with pnpm
description: "Sets up Node.js and pnpm with dependency caching, restoring cache from multiple fallback keys"
inputs:
node-version:
description: "Node.js version"
default: "20"
pnpm-version:
description: "pnpm version"
default: "9"
working-directory:
description: "Directory containing package.json"
default: "."
outputs:
cache-hit:
description: "Whether the node_modules cache was fully restored"
value: ${{ steps.cache.outputs.cache-hit }}
runs:
using: composite
steps:
- name: Setup pnpm
uses: pnpm/action-setup@v4
with:
version: ${{ inputs.pnpm-version }}
- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: ${{ inputs.node-version }}
# Don't use setup-node's built-in cache — we manage it ourselves
# for more control over the cache key
- name: Get pnpm store path
id: pnpm-cache-dir
shell: bash
run: echo "dir=$(pnpm store path --silent)" >> $GITHUB_OUTPUT
- name: Cache pnpm store
id: cache
uses: actions/cache@v4
with:
path: ${{ steps.pnpm-cache-dir.outputs.dir }}
key: pnpm-${{ runner.os }}-${{ inputs.node-version }}-${{ hashFiles('**/pnpm-lock.yaml') }}
restore-keys: |
pnpm-${{ runner.os }}-${{ inputs.node-version }}-
pnpm-${{ runner.os }}-
- name: Install dependencies
shell: bash
working-directory: ${{ inputs.working-directory }}
run: pnpm install --frozen-lockfile
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
|
# actions/notify-deploy/action.yml
# Composite action for deployment notifications
name: Notify Deployment
description: "Posts deployment status to Slack and creates a GitHub deployment"
inputs:
status:
description: "Deployment status: success, failure, in_progress"
required: true
environment:
description: "Target environment"
required: true
service:
description: "Service being deployed"
required: true
image-tag:
description: "Docker image tag being deployed"
required: true
slack-webhook:
description: "Slack webhook URL"
required: true
runs:
using: composite
steps:
- name: Create GitHub deployment
uses: chrnorm/deployment-action@v2
id: deployment
with:
token: ${{ github.token }}
environment: ${{ inputs.environment }}
description: "Deploying ${{ inputs.service }} @ ${{ inputs.image-tag }}"
- name: Notify Slack
shell: bash
env:
STATUS: ${{ inputs.status }}
SERVICE: ${{ inputs.service }}
ENV: ${{ inputs.environment }}
TAG: ${{ inputs.image-tag }}
ACTOR: ${{ github.actor }}
RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
WEBHOOK: ${{ inputs.slack-webhook }}
run: |
COLOR="good"
EMOJI=":rocket:"
if [ "$STATUS" = "failure" ]; then
COLOR="danger"
EMOJI=":fire:"
elif [ "$STATUS" = "in_progress" ]; then
COLOR="warning"
EMOJI=":hourglass_flowing_sand:"
fi
curl -sf -X POST "$WEBHOOK" \
-H "Content-Type: application/json" \
-d "{
\"attachments\": [{
\"color\": \"$COLOR\",
\"title\": \"$EMOJI Deploy $STATUS — $SERVICE to $ENV\",
\"fields\": [
{\"title\": \"Image\", \"value\": \"\`$TAG\`\", \"short\": true},
{\"title\": \"Actor\", \"value\": \"$ACTOR\", \"short\": true}
],
\"actions\": [{\"type\": \"button\", \"text\": \"View Run\", \"url\": \"$RUN_URL\"}]
}]
}"
|
Using Composite Actions
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
|
# .github/workflows/deploy.yml
jobs:
deploy:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
# Use local composite action (relative path)
- uses: ./.github/actions/setup-node-pnpm
with:
node-version: "20"
# Use composite action from another repo
- uses: my-org/platform-actions/notify-deploy@v1
with:
status: in_progress
environment: staging
service: web-app
image-tag: ${{ steps.build.outputs.tag }}
slack-webhook: ${{ secrets.SLACK_DEPLOY_WEBHOOK }}
- name: Deploy
id: deploy
run: ./scripts/deploy.sh staging
- uses: my-org/platform-actions/notify-deploy@v1
if: always()
with:
status: ${{ steps.deploy.outcome }}
environment: staging
service: web-app
image-tag: ${{ steps.build.outputs.tag }}
slack-webhook: ${{ secrets.SLACK_DEPLOY_WEBHOOK }}
|
Matrix Strategies
Matrix builds multiply a job across combinations of inputs. Used well, they parallelize your test suite across language versions, OSes, or shards. Used carelessly, they create combinatorial explosions.
Basic Matrix
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
|
jobs:
test:
strategy:
matrix:
os: [ubuntu-latest, macos-latest, windows-latest]
go-version: ["1.21", "1.22"]
fail-fast: false # Don't cancel all matrix jobs if one fails
max-parallel: 4 # Limit concurrency to manage runner costs
runs-on: ${{ matrix.os }}
name: Test (Go ${{ matrix.go-version }}, ${{ matrix.os }})
steps:
- uses: actions/checkout@v4
- uses: actions/setup-go@v5
with:
go-version: ${{ matrix.go-version }}
- run: go test ./...
|
Dynamic Matrix from a Script
For large test suites, generate the matrix dynamically rather than hardcoding it:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
|
jobs:
# First job: discover test packages and generate matrix
discover:
runs-on: ubuntu-latest
outputs:
matrix: ${{ steps.set-matrix.outputs.matrix }}
steps:
- uses: actions/checkout@v4
- name: Generate test matrix
id: set-matrix
run: |
# Find all Go packages with tests, split into groups of ~20
packages=$(go list ./... | grep -v vendor)
count=$(echo "$packages" | wc -l)
shards=8
size=$(( (count + shards - 1) / shards ))
matrix_json=$(echo "$packages" | \
awk -v size=$size '
BEGIN { print "[" }
{ lines[NR] = $0 }
END {
shard=0
for (i=1; i<=NR; i++) {
if ((i-1) % size == 0) {
if (i > 1) printf ","
printf "{\"shard\":%d,\"packages\":\"", shard++
}
printf "%s ", lines[i]
if (i % size == 0 || i == NR) printf "\"}"
}
print "]"
}
'
)
echo "matrix={\"include\":$matrix_json}" >> $GITHUB_OUTPUT
# Second job: run each shard in parallel
test:
needs: discover
runs-on: ubuntu-latest
strategy:
matrix: ${{ fromJSON(needs.discover.outputs.matrix) }}
fail-fast: false
name: "Test shard ${{ matrix.shard }}"
steps:
- uses: actions/checkout@v4
- uses: actions/setup-go@v5
with:
go-version: "1.22"
cache: true
- name: Run tests
run: go test ${{ matrix.packages }} -timeout 10m -race
|
Matrix with Includes and Excludes
1
2
3
4
5
6
7
8
9
10
11
12
13
14
|
strategy:
matrix:
os: [ubuntu-latest, macos-latest, windows-latest]
node: [18, 20, 22]
exclude:
# Don't test Node 18 on Windows (known issues)
- os: windows-latest
node: 18
include:
# Add an extra job: Node 22 LTS on ubuntu with extra flags
- os: ubuntu-latest
node: 22
experimental: true
test-args: "--coverage"
|
Test Splitting for Speed
For large test suites, split by timing data to equalize shard duration:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
|
jobs:
test:
strategy:
matrix:
shard: [1, 2, 3, 4, 5, 6, 7, 8]
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Download timing data
uses: actions/cache@v4
with:
path: .test-timing.json
key: test-timing-${{ github.ref }}
restore-keys: test-timing-
- name: Run test shard
uses: hashicorp/vault-action@v3 # example: fetch DB creds
# Split tests by timing so each shard takes ~equal time
run: |
# Use gotestsum with --rerun-fails for flaky test handling
gotestsum \
--junitfile results.xml \
--rerun-fails=2 \
-- \
$(go-test-split \
--total ${{ strategy.job-total }} \
--index ${{ strategy.job-index }} \
--timing-file .test-timing.json \
./...)
- name: Update timing data
if: github.ref == 'refs/heads/main'
run: go-test-split --update-timing-file .test-timing.json results.xml
- name: Cache updated timing
if: github.ref == 'refs/heads/main'
uses: actions/cache/save@v4
with:
path: .test-timing.json
key: test-timing-${{ github.ref }}-${{ github.run_id }}
- name: Upload test results
if: always()
uses: actions/upload-artifact@v4
with:
name: test-results-shard-${{ matrix.shard }}
path: results.xml
|
Caching Strategies
Cache misses are the single biggest source of slow CI. Understand the cache key design before you write a single actions/cache step.
Cache Key Design
Cache key structure:
{scope}-{os}-{lockfile-hash}
scope: what the cache is for (npm, go, pip, gradle)
os: runner OS (cache is not cross-platform)
lockfile-hash: invalidate when dependencies change
restore-keys (fallback order, most specific first):
{scope}-{os}-{lockfile-hash} ← exact match (full hit)
{scope}-{os}- ← partial match on same OS (warm start)
{scope}- ← any OS (last resort)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
|
# Go modules — cache the module download cache and build cache
- name: Cache Go modules
uses: actions/cache@v4
with:
path: |
~/.cache/go/mod
~/.cache/go/build
key: go-${{ runner.os }}-${{ hashFiles('**/go.sum') }}
restore-keys: |
go-${{ runner.os }}-
# Node / pnpm — cache the content-addressed store
- name: Cache pnpm
uses: actions/cache@v4
with:
path: ~/.local/share/pnpm/store
key: pnpm-${{ runner.os }}-${{ hashFiles('**/pnpm-lock.yaml') }}
restore-keys: pnpm-${{ runner.os }}-
# Python / uv
- name: Cache uv
uses: actions/cache@v4
with:
path: |
~/.cache/uv
.venv
key: uv-${{ runner.os }}-${{ hashFiles('uv.lock') }}
restore-keys: uv-${{ runner.os }}-
# Rust / Cargo
- name: Cache Cargo
uses: actions/cache@v4
with:
path: |
~/.cargo/registry
~/.cargo/git
target/
key: cargo-${{ runner.os }}-${{ hashFiles('**/Cargo.lock') }}
restore-keys: cargo-${{ runner.os }}-
# Gradle (Android/JVM)
- name: Cache Gradle
uses: actions/cache@v4
with:
path: |
~/.gradle/caches
~/.gradle/wrapper
key: gradle-${{ runner.os }}-${{ hashFiles('**/*.gradle*', '**/gradle-wrapper.properties') }}
restore-keys: gradle-${{ runner.os }}-
|
Docker Layer Caching
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
|
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
with:
# Use docker-container driver for full cache support
driver-opts: |
network=host
- name: Build and push
uses: docker/build-push-action@v5
with:
context: .
push: true
tags: ${{ steps.meta.outputs.tags }}
# GitHub Actions cache (best for most cases)
cache-from: type=gha
cache-to: type=gha,mode=max
# Alternative: registry cache (persistent across runners, costs egress)
# cache-from: type=registry,ref=ghcr.io/${{ github.repository }}:buildcache
# cache-to: type=registry,ref=ghcr.io/${{ github.repository }}:buildcache,mode=max
# For multi-stage builds, cache individual stages
build-args: |
BUILDKIT_INLINE_CACHE=1
|
Saving Caches from PRs (Safely)
GitHub doesn’t allow PRs from forks to write to the cache. Handle this with a cache-warming workflow:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
|
# .github/workflows/warm-cache.yml
# Runs on main to pre-warm caches that PRs from forks can read
name: Warm Cache
on:
push:
branches: [main]
paths:
- "go.sum"
- "pnpm-lock.yaml"
- "Cargo.lock"
jobs:
warm:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Setup Go (warms cache)
uses: actions/setup-go@v5
with:
go-version: "1.22"
cache: true
- name: Download Go modules
run: go mod download
- name: Build (warms build cache)
run: go build ./...
|
Self-Hosted Runners
GitHub-hosted runners are convenient but limited: 2–4 CPU cores, no persistent storage, no access to private networks, and no GPUs. Self-hosted runners solve all of these — at the cost of managing the infrastructure yourself.
Runner Architecture
GitHub Actions
│
│ HTTPS long-poll (runner initiates, no inbound firewall rules needed)
▼
┌──────────────────────────────────────────┐
│ Self-hosted Runner Fleet │
│ │
│ ┌────────────┐ ┌────────────┐ │
│ │ Runner VM │ │ Runner VM │ ... │
│ │ (standard) │ │ (GPU) │ │
│ └────────────┘ └────────────┘ │
│ │
│ Private network access: │
│ - Internal Kubernetes clusters │
│ - Database hosts │
│ - Artifact registries │
└──────────────────────────────────────────┘
Ephemeral Runners with Actions Runner Controller (ARC)
The modern approach: Kubernetes-based ephemeral runners that scale to zero and spin up fresh for every job.
1
2
3
4
5
6
7
8
9
10
|
# Install ARC via Helm
helm install arc \
--namespace arc-systems \
--create-namespace \
oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set-controller
# Create a GitHub App or PAT for authentication
kubectl create secret generic github-token \
--namespace arc-runners \
--from-literal=github_token='ghp_your_token'
|
1
2
3
4
5
6
7
8
|
# arc-runner-scale-set.yml
# A scale set that runs one ephemeral runner per job
helm install arc-runner-set \
--namespace arc-runners \
--create-namespace \
--set githubConfigUrl="https://github.com/my-org/my-repo" \
--set githubConfigSecret=github-token \
oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
|
# values-runner.yml — customized runner pod
githubConfigUrl: "https://github.com/my-org"
githubConfigSecret: "github-token"
runnerScaleSetName: "k8s-runners"
minRunners: 0 # Scale to zero when idle
maxRunners: 20 # Hard cap
template:
spec:
initContainers:
- name: init-dind-externals
image: ghcr.io/actions/actions-runner:latest
command: ["cp", "-r", "-v", "/home/runner/externals/.", "/home/runner/tmpDir/"]
volumeMounts:
- name: dind-externals
mountPath: /home/runner/tmpDir
containers:
- name: runner
image: ghcr.io/actions/actions-runner:latest
env:
- name: DOCKER_HOST
value: unix:///var/run/docker.sock
resources:
requests:
cpu: "1"
memory: "2Gi"
limits:
cpu: "4"
memory: "8Gi"
volumeMounts:
- name: work
mountPath: /home/runner/_work
- name: dind-sock
mountPath: /var/run
- name: dind
image: docker:dind
securityContext:
privileged: true
volumeMounts:
- name: work
mountPath: /home/runner/_work
- name: dind-externals
mountPath: /home/runner/externals
- name: dind-sock
mountPath: /var/run
volumes:
- name: work
emptyDir: {}
- name: dind-sock
emptyDir: {}
- name: dind-externals
emptyDir: {}
|
GPU Runners for ML Workloads
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
|
# values-gpu-runner.yml
githubConfigUrl: "https://github.com/my-org"
githubConfigSecret: "github-token"
runnerScaleSetName: "gpu-runners"
minRunners: 0
maxRunners: 4 # Each runner uses one GPU
template:
spec:
containers:
- name: runner
image: my-registry/gpu-actions-runner:cuda12.3
resources:
requests:
nvidia.com/gpu: "1"
cpu: "8"
memory: "32Gi"
limits:
nvidia.com/gpu: "1"
cpu: "16"
memory: "64Gi"
volumeMounts:
- name: work
mountPath: /home/runner/_work
- name: model-cache
mountPath: /mnt/models # Persistent volume for model weights
volumes:
- name: work
emptyDir: {}
- name: model-cache
persistentVolumeClaim:
claimName: model-weights-pvc
nodeSelector:
nvidia.com/gpu.product: NVIDIA-A100-SXM4-80GB
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
|
# Using GPU runners in a workflow
jobs:
train-model:
runs-on: gpu-runners # Matches the runnerScaleSetName
steps:
- uses: actions/checkout@v4
- name: Train
run: |
nvidia-smi # Verify GPU is available
python train.py \
--model-dir /mnt/models/my-model \
--epochs 10 \
--device cuda
|
Autoscaling Runners on EC2 with Philips-labs Runner Manager
For AWS-native setups, ephemeral EC2 runners on spot instances are highly cost-effective:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
|
# terraform/runners.tf
module "github_runner" {
source = "philips-labs/github-runner/aws"
version = "~> 5.0"
aws_region = "us-east-1"
vpc_id = module.vpc.vpc_id
subnet_ids = module.vpc.private_subnets
prefix = "my-org"
github_app = {
key_base64 = var.github_app_key_base64
id = var.github_app_id
webhook_secret = var.github_webhook_secret
}
runner_extra_labels = ["self-hosted", "linux", "x64"]
# Use spot instances — 70-90% cheaper
market_options = "spot"
instance_types = ["c6i.2xlarge", "c6a.2xlarge", "m6i.2xlarge"]
runner_iam_role_managed_policy_arns = [
"arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore", # SSM access
aws_iam_policy.runner_ecr.arn, # Pull from ECR
aws_iam_policy.runner_s3.arn, # Read/write artifacts
]
# Scale from 0 to 20 runners
runners_maximum_count = 20
scale_down_schedule_expression = "cron(*/5 * * * ? *)"
# Use a custom AMI with your tools pre-installed
ami_filter = {
name = ["my-org-runner-*"]
state = ["available"]
}
}
|
Security Hardening
Pin Actions to Commit SHA
Floating tags like @v4 can be updated by the action’s owner at any time. A compromised action could exfiltrate your secrets.
1
2
3
4
5
6
7
8
9
|
# Risky: can change underneath you
- uses: actions/checkout@v4
# Safe: pinned to exact commit (immutable)
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
# Automate SHA pinning with OpenSSF's pin-github-action:
pip install pin-github-action
pin-github-action .github/workflows/ci.yml
|
Minimal Permissions
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
|
# Set minimal default permissions at the workflow level
permissions:
contents: read # Everything else is disabled
jobs:
test:
permissions:
contents: read # Read repo code
# ...
deploy:
permissions:
contents: read
id-token: write # For OIDC (AWS, GCP, Azure auth)
packages: write # Push to GHCR
deployments: write # Create GitHub deployments
|
OIDC Authentication (No Long-Lived Secrets)
Replace AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY with short-lived OIDC tokens:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
|
jobs:
deploy:
permissions:
id-token: write # Required for OIDC
contents: read
steps:
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683
- name: Configure AWS credentials (OIDC — no stored secrets!)
uses: aws-actions/configure-aws-credentials@e3dd6a429d7300a6a4c196c26e071d42e0343502
with:
role-to-assume: arn:aws:iam::123456789012:role/github-actions-deploy
aws-region: us-east-1
# Optionally restrict by branch:
# role-session-name: ${{ github.ref_name }}-deploy
- name: Deploy
run: aws s3 sync ./dist s3://my-bucket/
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
|
// AWS IAM trust policy for the OIDC role
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {
"Federated": "arn:aws:iam::123456789012:oidc-provider/token.actions.githubusercontent.com"
},
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": {
"StringEquals": {
"token.actions.githubusercontent.com:aud": "sts.amazonaws.com"
},
"StringLike": {
// Only allow from main branch of specific repo
"token.actions.githubusercontent.com:sub": "repo:my-org/my-repo:ref:refs/heads/main"
}
}
}]
}
|
Secret Scanning and Validation
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
|
# Prevent secrets from being committed
- name: Scan for secrets
uses: trufflesecurity/trufflehog@main
with:
path: ./
base: ${{ github.event.repository.default_branch }}
head: HEAD
extra_args: --debug --only-verified
# Validate that required secrets are set (fail fast, don't leak which are missing)
- name: Validate secrets
shell: bash
env:
REQUIRED_SECRETS_SET: ${{ secrets.DATABASE_URL != '' && secrets.API_KEY != '' }}
run: |
if [ "$REQUIRED_SECRETS_SET" != "true" ]; then
echo "::error::Required secrets are not configured"
exit 1
fi
|
Workflow Permissions for PRs from Forks
1
2
3
4
5
6
7
8
|
# For PRs from forks: use pull_request (restricted) not pull_request_target (dangerous)
on:
pull_request: # Safe: fork PRs run with read-only permissions and no secrets
# pull_request_target runs with write permissions and access to secrets
# NEVER run untrusted code in a pull_request_target workflow
on:
pull_request_target: # DANGEROUS if you run code from the PR
|
The safe pattern for fork PRs that need secrets (e.g., posting coverage):
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
|
# Workflow 1: Run tests on PR (no secrets needed)
on: [pull_request]
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- run: go test ./... -coverprofile=coverage.out
- uses: actions/upload-artifact@v4
with:
name: coverage
path: coverage.out
# Workflow 2: Post results (triggered by workflow completion, has secrets)
on:
workflow_run:
workflows: ["CI"]
types: [completed]
jobs:
report:
if: github.event.workflow_run.conclusion == 'success'
runs-on: ubuntu-latest
permissions:
pull-requests: write
steps:
- uses: actions/download-artifact@v4
with:
name: coverage
run-id: ${{ github.event.workflow_run.id }}
github-token: ${{ github.token }}
- name: Post coverage comment
uses: actions/github-script@v7
with:
script: |
const fs = require('fs');
// read coverage.out and post as PR comment
|
Monorepo Patterns
Path-Based Job Filtering
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
|
on:
push:
branches: [main]
pull_request:
jobs:
# Detect which services changed
changes:
runs-on: ubuntu-latest
outputs:
api: ${{ steps.filter.outputs.api }}
web: ${{ steps.filter.outputs.web }}
infra: ${{ steps.filter.outputs.infra }}
steps:
- uses: actions/checkout@v4
- uses: dorny/paths-filter@v3
id: filter
with:
filters: |
api:
- 'services/api/**'
- 'shared/proto/**'
- '.github/workflows/ci-api.yml'
web:
- 'apps/web/**'
- 'packages/ui/**'
infra:
- 'terraform/**'
- 'kubernetes/**'
# Only run if the API changed
build-api:
needs: changes
if: needs.changes.outputs.api == 'true'
uses: ./.github/workflows/reusable-go-build.yml
with:
service: api
working-directory: ./services/api
# Only run if web changed
build-web:
needs: changes
if: needs.changes.outputs.web == 'true'
uses: ./.github/workflows/reusable-node-build.yml
with:
service: web
working-directory: ./apps/web
# Gate: all changed services must pass
all-checks:
needs: [build-api, build-web]
if: always()
runs-on: ubuntu-latest
steps:
- name: Check all jobs passed
run: |
if [[ "${{ contains(needs.*.result, 'failure') }}" == "true" ]]; then
echo "One or more jobs failed"
exit 1
fi
echo "All checks passed"
|
Consistent Required Status Checks
When using path filtering, some jobs will be skipped. GitHub requires all “required” status checks to pass — but a skipped job counts as a pass. The trick: always report a final status:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
|
# This job always runs and checks that required jobs either passed or were skipped
required-checks:
if: always()
needs: [build-api, build-web, build-infra, lint, security-scan]
runs-on: ubuntu-latest
steps:
- name: Verify required checks
run: |
results='${{ toJSON(needs) }}'
failures=$(echo "$results" | jq '[.[] | select(.result == "failure")] | length')
if [ "$failures" -gt 0 ]; then
echo "::error::$failures required check(s) failed"
exit 1
fi
echo "All required checks passed or were skipped"
|
Workflow Observability
Job Summaries
1
2
3
4
5
6
7
8
9
10
11
12
13
|
- name: Write test summary
if: always()
run: |
echo "## Test Results" >> $GITHUB_STEP_SUMMARY
echo "" >> $GITHUB_STEP_SUMMARY
echo "| Test Suite | Result | Duration |" >> $GITHUB_STEP_SUMMARY
echo "|-----------|--------|----------|" >> $GITHUB_STEP_SUMMARY
echo "| Unit Tests | ✅ Passed | 2m 34s |" >> $GITHUB_STEP_SUMMARY
echo "| Integration Tests | ✅ Passed | 5m 12s |" >> $GITHUB_STEP_SUMMARY
echo "| Coverage | 87.3% | - |" >> $GITHUB_STEP_SUMMARY
echo "" >> $GITHUB_STEP_SUMMARY
echo "### Slowest Tests" >> $GITHUB_STEP_SUMMARY
# ... output from test timing data
|
Tracking CI Metrics
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
|
# Post workflow metrics to Datadog/Prometheus after every run
- name: Emit CI metrics
if: always()
env:
DD_API_KEY: ${{ secrets.DATADOG_API_KEY }}
WORKFLOW: ${{ github.workflow }}
JOB: ${{ github.job }}
STATUS: ${{ job.status }}
DURATION: ${{ ... }} # calculate from timestamps
BRANCH: ${{ github.ref_name }}
REPO: ${{ github.repository }}
run: |
curl -X POST "https://api.datadoghq.com/api/v2/series" \
-H "DD-API-KEY: $DD_API_KEY" \
-H "Content-Type: application/json" \
-d "{
\"series\": [{
\"metric\": \"ci.job.duration\",
\"points\": [[$(date +%s), $DURATION]],
\"tags\": [
\"workflow:$WORKFLOW\",
\"job:$JOB\",
\"status:$STATUS\",
\"branch:$BRANCH\",
\"repo:$REPO\"
]
}]
}"
|
Putting It Together: A Real-World Pipeline
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
|
# .github/workflows/ci.yml — production-grade pipeline
name: CI / CD
on:
push:
branches: [main]
pull_request:
types: [opened, synchronize, reopened]
# Cancel in-progress runs for the same branch/PR
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: ${{ github.ref != 'refs/heads/main' }}
permissions:
contents: read
pull-requests: write
jobs:
# 1. Detect changes
changes:
runs-on: ubuntu-latest
outputs:
api: ${{ steps.filter.outputs.api }}
web: ${{ steps.filter.outputs.web }}
steps:
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683
- uses: dorny/paths-filter@de90cc6fb38fc0963ad72b210f1f284cd68cea36
id: filter
with:
filters: |
api: ['services/api/**', 'shared/**']
web: ['apps/web/**']
# 2. Lint (fast, always runs)
lint:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683
- uses: golangci/golangci-lint-action@v6
with:
version: v1.62
# 3. Build + test API (only if changed)
api:
needs: [changes, lint]
if: needs.changes.outputs.api == 'true'
uses: my-org/platform/.github/workflows/reusable-go-build.yml@v2.1.0
with:
go-version: "1.22"
working-directory: ./services/api
artifact-name: api-${{ github.sha }}
run-integration-tests: ${{ github.ref == 'refs/heads/main' }}
secrets:
SONAR_TOKEN: ${{ secrets.SONAR_TOKEN }}
# 4. Build + test web (only if changed)
web:
needs: [changes, lint]
if: needs.changes.outputs.web == 'true'
uses: my-org/platform/.github/workflows/reusable-node-build.yml@v2.1.0
with:
node-version: "20"
working-directory: ./apps/web
artifact-name: web-${{ github.sha }}
# 5. Security scan (always)
security:
runs-on: ubuntu-latest
permissions:
security-events: write
steps:
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683
- uses: github/codeql-action/init@v3
with:
languages: go, javascript
- uses: github/codeql-action/autobuild@v3
- uses: github/codeql-action/analyze@v3
# 6. Required gate
all-green:
if: always()
needs: [lint, api, web, security]
runs-on: ubuntu-latest
steps:
- run: |
echo '${{ toJSON(needs) }}' | \
jq -e '[.[].result] | all(. == "success" or . == "skipped")'
# 7. Deploy to staging (main only)
deploy-staging:
needs: all-green
if: github.ref == 'refs/heads/main'
runs-on: ubuntu-latest
environment:
name: staging
url: https://staging.myapp.com
permissions:
id-token: write
contents: read
deployments: write
steps:
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683
- name: Configure AWS credentials
uses: aws-actions/configure-aws-credentials@e3dd6a429d7300a6a4c196c26e071d42e0343502
with:
role-to-assume: ${{ vars.AWS_DEPLOY_ROLE_STAGING }}
aws-region: us-east-1
- name: Deploy
run: |
aws eks update-kubeconfig --name staging-cluster --region us-east-1
kubectl set image deployment/api \
api=ghcr.io/${{ github.repository }}:${{ github.sha }} \
--namespace staging
kubectl rollout status deployment/api --namespace staging --timeout=5m
|
Quick Reference
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
|
# Useful GitHub CLI commands for Actions
# List workflow runs for a branch
gh run list --branch main --workflow ci.yml
# Watch a run in real-time
gh run watch <run-id>
# Download artifacts from a run
gh run download <run-id> --name my-artifact
# Re-run failed jobs only
gh run rerun <run-id> --failed
# View self-hosted runner status
gh api /orgs/my-org/actions/runners | jq '.runners[] | {name, status, busy}'
# List and delete old caches
gh api /repos/my-org/my-repo/actions/caches | jq '.actions_caches[] | {id, key, size_in_bytes}'
gh api -X DELETE /repos/my-org/my-repo/actions/caches/<id>
|
Filed under: Developer Tooling & Workflow
Comments