Temporal and Workflow Orchestration: Durable Execution for Engineers Who've Been Burned

Every engineer who has built a multi-step distributed process has the same scar tissue. An order processing flow that touches five services. A user onboarding pipeline that sends emails, provisions accounts, and calls three external APIs. A background job that needs to run for two minutes, retry on failure, and report completion back to the original request. You bolt it together with a task queue, add retry logic, write state to a database to track progress, and ship it. Six months later a partial failure leaves a user half-onboarded. A network blip causes a payment to be charged twice because the idempotency logic had a race condition. A queue worker crashes mid-job and the job silently disappears.

The pattern of problems is the same everywhere: distributed state is hard, failure handling is accidental rather than structural, and the coordination logic ends up smeared across application code, database schema, and queue configuration in ways that are impossible to test.

Temporal is a durable execution engine. The central claim is: write your business logic as regular code — functions calling functions — and Temporal guarantees it runs to completion even if every machine involved crashes and reboots, even if the process takes months, even if you redeploy in the middle of execution. It does this not through magic but through a specific architectural choice: every state transition is appended to a durable event log, and execution is always recoverable by replaying that log.

This post covers how the durable execution model works mechanically, workflows and activities in Go and Python with real examples, the retry and timeout system, the determinism constraints that bite new users, self-hosting Temporal on Kubernetes, and an honest comparison with the alternatives you might already be using.

The Core Problem with Task Queues

A task queue (Celery, Sidekiq, RQ, SQS workers) solves one problem well: defer a unit of work to a background process. It solves this problem in isolation. What it does not solve — and what engineers build around with escalating complexity — is coordinated multi-step execution with durable state.

Consider a user signup flow:

1. Create user record in database
2. Send welcome email
3. Provision workspace (calls external API, may take 30s)
4. Send Slack notification to #new-signups
5. Trigger trial period scheduler (needs to fire in 14 days)

With a task queue:

• Each step is a separate task
• You track progress in a database table (signup_state, with columns for each step’s completion)
• Each task checks the state table, does its work, updates state
• Failures retry the individual task — but if step 3 fails after step 2 completes, you need idempotency logic to avoid sending the welcome email again
• If you want to wait 14 days and then do something, you need a separate scheduled job system
• If you want to pause the workflow and wait for a human to approve something, you need another database table and a polling mechanism

The coordination logic — the state machine, the idempotency checks, the scheduler integration — is yours to build and maintain. It is never in one place, and it is rarely tested as a unit.

Temporal’s answer: the workflow is the coordination logic, written as ordinary code, and Temporal makes it durable. You do not write a state machine — you write the happy path, and Temporal handles everything else.

How Durable Execution Works

Temporal persists every state transition as an entry in an append-only event history. When an activity completes, that completion is recorded. When a timer fires, that firing is recorded. When a signal arrives, that signal is recorded. The workflow’s current state is always fully reconstructable from its event history.

When a Worker crashes mid-execution — or you redeploy, or the server restarts — Temporal does not try to resume from a checkpoint. It replays the event history from the beginning. The Workflow code runs again from the top, but instead of re-executing activities, it reads their recorded results from the history. The workflow function deterministically reaches the same state it was in before the crash. When replay reaches the point in history where execution was interrupted, it continues forward with new work.

Event History for workflow "order-12345":
┌────────────────────────────────────────────────────────┐
│ 1. WorkflowExecutionStarted  {input: {order_id: 12345}}│
│ 2. ActivityTaskScheduled     {activity: "charge_card"} │
│ 3. ActivityTaskStarted                                  │
│ 4. ActivityTaskCompleted     {result: {charge_id: "x"}}│
│ 5. ActivityTaskScheduled     {activity: "ship_order"}  │
│ 6. ActivityTaskStarted                                  │
│ ← WORKER CRASHED HERE                                  │
└────────────────────────────────────────────────────────┘

On recovery, Temporal replays events 1-6.
The workflow code runs again:
  - charge_card() → history says it already completed, return recorded result
  - ship_order()  → history says it started but never completed → re-execute

This is why Temporal workflows have a determinism requirement: the code must produce the same sequence of commands on every replay. If replay produces a different sequence of events than what is in the history, there is a non-determinism error and the workflow cannot recover. More on this constraint shortly.

The practical consequence: you do not need to write idempotency logic into your workflow code. Temporal handles it structurally. An activity that already completed in the history will never be re-executed on replay — its recorded result is returned directly.

Workflows and Activities

The fundamental distinction in Temporal:

Workflows are the coordinators. They orchestrate activities, wait on timers, handle signals, and express the business logic of what should happen and in what order. They must be deterministic — no external calls, no random numbers, no reading the current time directly.

Activities are the workers. They do the actual work: call external APIs, write to databases, send emails, process files. They are allowed to fail, and they are retried automatically. They are not deterministic and do not need to be.

Go example: order processing

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39

// activities.go — the actual work units
package order

import (
    "context"
    "time"
    "go.temporal.io/sdk/activity"
)

type OrderActivities struct {
    db          *Database
    emailClient *EmailClient
    shipClient  *ShippingClient
}

func (a *OrderActivities) ChargeCard(ctx context.Context, orderID string, amount int) (string, error) {
    activity.GetLogger(ctx).Info("Charging card", "orderID", orderID)

    // Heartbeat for long-running activities — lets Temporal know we're alive
    activity.RecordHeartbeat(ctx, "charging")

    chargeID, err := a.paymentGateway.Charge(ctx, orderID, amount)
    if err != nil {
        return "", fmt.Errorf("charge failed: %w", err)
    }
    return chargeID, nil
}

func (a *OrderActivities) ShipOrder(ctx context.Context, orderID, chargeID string) (string, error) {
    trackingNumber, err := a.shipClient.CreateShipment(ctx, orderID)
    if err != nil {
        return "", err
    }
    return trackingNumber, nil
}

func (a *OrderActivities) SendConfirmation(ctx context.Context, orderID, trackingNumber string) error {
    return a.emailClient.SendOrderConfirmation(ctx, orderID, trackingNumber)
}

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52

// workflow.go — the coordinator
package order

import (
    "time"
    "go.temporal.io/sdk/temporal"
    "go.temporal.io/sdk/workflow"
)

func OrderWorkflow(ctx workflow.Context, input OrderInput) error {
    logger := workflow.GetLogger(ctx)

    // Activity options: retry policy, timeouts
    activityOptions := workflow.ActivityOptions{
        StartToCloseTimeout: 30 * time.Second,
        RetryPolicy: &temporal.RetryPolicy{
            InitialInterval:    time.Second,
            BackoffCoefficient: 2.0,
            MaximumInterval:    30 * time.Second,
            MaximumAttempts:    5,
            // Don't retry on business logic failures
            NonRetryableErrorTypes: []string{"InvalidCardError", "InsufficientFundsError"},
        },
    }
    ctx = workflow.WithActivityOptions(ctx, activityOptions)

    // Step 1: Charge the card
    var chargeID string
    if err := workflow.ExecuteActivity(ctx, a.ChargeCard, input.OrderID, input.Amount).Get(ctx, &chargeID); err != nil {
        return fmt.Errorf("payment failed: %w", err)
    }
    logger.Info("Card charged", "chargeID", chargeID)

    // Step 2: Ship the order
    var trackingNumber string
    if err := workflow.ExecuteActivity(ctx, a.ShipOrder, input.OrderID, chargeID).Get(ctx, &trackingNumber); err != nil {
        // Payment succeeded but shipping failed — compensate
        workflow.ExecuteActivity(ctx, a.RefundCharge, chargeID)
        return fmt.Errorf("shipping failed, refund issued: %w", err)
    }

    // Step 3: Send confirmation
    workflow.ExecuteActivity(ctx, a.SendConfirmation, input.OrderID, trackingNumber)

    // Step 4: Schedule a follow-up review in 7 days
    // This sleeps the workflow for 7 days — zero resource consumption while waiting
    workflow.Sleep(ctx, 7*24*time.Hour)

    workflow.ExecuteActivity(ctx, a.SendReviewRequest, input.OrderID)

    return nil
}

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

// worker/main.go — the worker process
package main

import (
    "go.temporal.io/sdk/client"
    "go.temporal.io/sdk/worker"
    "order"
)

func main() {
    c, _ := client.Dial(client.Options{HostPort: "temporal:7233"})
    defer c.Close()

    w := worker.New(c, "order-processing", worker.Options{})

    // Register workflow and activities
    w.RegisterWorkflow(order.OrderWorkflow)
    activities := &order.OrderActivities{/* inject deps */}
    w.RegisterActivity(activities)

    w.Run(worker.InterruptCh())
}

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

// Starting a workflow from your API handler
func (h *Handler) HandleCheckout(w http.ResponseWriter, r *http.Request) {
    input := OrderInput{OrderID: "order-123", Amount: 4999}

    we, err := h.temporalClient.ExecuteWorkflow(r.Context(),
        client.StartWorkflowOptions{
            ID:        "order-" + input.OrderID,  // idempotent ID
            TaskQueue: "order-processing",
        },
        order.OrderWorkflow,
        input,
    )
    // we.GetID() and we.GetRunID() let you query this workflow later
}

Python example

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

# activities.py
import asyncio
from temporalio import activity

@activity.defn
async def charge_card(order_id: str, amount_cents: int) -> str:
    activity.heartbeat("Charging card")
    # call payment gateway...
    return charge_id

@activity.defn
async def ship_order(order_id: str, charge_id: str) -> str:
    # call shipping API...
    return tracking_number

@activity.defn
async def send_confirmation(order_id: str, tracking_number: str) -> None:
    # send email...
    pass

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46

# workflow.py
from datetime import timedelta
from temporalio import workflow
from temporalio.common import RetryPolicy

@workflow.defn
class OrderWorkflow:
    @workflow.run
    async def run(self, order_id: str, amount_cents: int) -> str:
        retry_policy = RetryPolicy(
            initial_interval=timedelta(seconds=1),
            backoff_coefficient=2.0,
            maximum_attempts=5,
            non_retryable_error_types=["InvalidCardError"],
        )

        charge_id = await workflow.execute_activity(
            charge_card,
            args=[order_id, amount_cents],
            start_to_close_timeout=timedelta(seconds=30),
            retry_policy=retry_policy,
        )

        tracking_number = await workflow.execute_activity(
            ship_order,
            args=[order_id, charge_id],
            start_to_close_timeout=timedelta(minutes=2),
            retry_policy=retry_policy,
        )

        await workflow.execute_activity(
            send_confirmation,
            args=[order_id, tracking_number],
            start_to_close_timeout=timedelta(seconds=10),
        )

        # Sleep for 7 days, zero resource consumption
        await asyncio.sleep(7 * 24 * 3600)

        await workflow.execute_activity(
            send_review_request,
            args=[order_id],
            start_to_close_timeout=timedelta(seconds=10),
        )

        return tracking_number

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

# worker.py
import asyncio
from temporalio.client import Client
from temporalio.worker import Worker
from workflow import OrderWorkflow
from activities import charge_card, ship_order, send_confirmation

async def main():
    client = await Client.connect("temporal:7233")
    async with Worker(
        client,
        task_queue="order-processing",
        workflows=[OrderWorkflow],
        activities=[charge_card, ship_order, send_confirmation],
    ):
        await asyncio.Event().wait()  # run forever

asyncio.run(main())

Retries and Timeouts

Temporal’s retry and timeout system is the most complete in any workflow engine, and understanding the distinct timeout types is critical for correct behavior.

Timeout types

Activity timeouts (apply to a single Activity Execution):
┌──────────────────────────────────────────────────────────────────────┐
│ Schedule-to-Start: how long to wait for a worker to pick up the task │
│   (catches: no workers running, queue backed up)                     │
│                                                                      │
│ Start-to-Close: how long the activity can run once started           │
│   (catches: activity hanging, infinite loop, slow external call)     │
│   THIS IS THE MOST IMPORTANT ONE — always set it                     │
│                                                                      │
│ Schedule-to-Close: total time budget for the whole activity attempt  │
│   including retries — a hard ceiling                                 │
│                                                                      │
│ Heartbeat timeout: if activity sends heartbeats, max gap between them│
│   (catches: worker crash mid-activity; requires RecordHeartbeat())   │
└──────────────────────────────────────────────────────────────────────┘

Workflow timeouts (apply to the entire Workflow Execution):
┌──────────────────────────────────────────────────────────────────────┐
│ Execution timeout: total clock time the workflow is allowed to run   │
│   including all retries and waits                                    │
│                                                                      │
│ Run timeout: maximum duration of a single run (between ContinueAsNew)│
│                                                                      │
│ Task timeout: maximum time a workflow task can run on a worker        │
│   (default 10s; increase for compute-intensive workflows)            │
└──────────────────────────────────────────────────────────────────────┘

The retry policy

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

RetryPolicy: &temporal.RetryPolicy{
    InitialInterval:    time.Second,      // first retry after 1s
    BackoffCoefficient: 2.0,             // doubles each retry: 1s, 2s, 4s, 8s...
    MaximumInterval:    5 * time.Minute, // cap at 5min
    MaximumAttempts:    0,               // 0 = unlimited retries
    NonRetryableErrorTypes: []string{
        "InvalidInputError",   // business logic failures — don't retry
        "NotFoundError",
    },
}

By default, activities have unlimited retries with exponential backoff. This is intentionally aggressive: Temporal’s philosophy is that transient failures (network issues, rate limits, downstream service blips) should be automatically handled and not require human intervention. You opt out by setting NonRetryableErrorTypes for errors that represent permanent failures.

Signals: external input to running workflows

A running workflow can receive signals — external events that trigger a change in workflow state. This replaces the pattern of polling a database table or using a message queue to communicate with a running process:

1
2
3
4
5
6
7
8
9

// In the workflow — wait for an approval signal
var approved bool
workflow.GetSignalChannel(ctx, "approval").Receive(ctx, &approved)
if !approved {
    return workflow.ExecuteActivity(ctx, a.CancelOrder, input.OrderID)
}

// From your API handler — send the signal
temporalClient.SignalWorkflow(ctx, workflowID, runID, "approval", true)

The workflow suspends at Receive() — no polling, no database writes, no resources consumed while waiting. When the signal arrives (minutes, hours, or days later), Temporal wakes a worker, replays the history to the suspension point, delivers the signal, and execution continues.

The Determinism Constraint

This is the part that surprises engineers new to Temporal. Because workflows are recovered by replaying event history, the workflow code must produce the exact same sequence of commands on every replay. Anything that introduces nondeterminism — random numbers, current time, direct external calls — breaks replay.

What you cannot do in workflow code:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

# WRONG: random number — different on replay
import random
delay = random.randint(1, 10)

# WRONG: current time — different on replay
import datetime
now = datetime.datetime.now()

# WRONG: external HTTP call — will re-execute on replay, breaks idempotency
import requests
result = requests.get("https://api.example.com/data")

# WRONG: goroutine/thread not managed by Temporal
go func() { doSomething() }()

What you do instead:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

// CORRECT: use workflow.Now() — returns time from event history on replay
now := workflow.Now(ctx)

// CORRECT: use workflow.NewRandom() — deterministic seeded from history
r := workflow.NewRandom(ctx)
delay := time.Duration(r.Intn(10)) * time.Second

// CORRECT: external calls go in Activities, not Workflows
var result MyResult
workflow.ExecuteActivity(ctx, myActivity).Get(ctx, &result)

// CORRECT: use workflow.Go() for managed goroutines
workflow.Go(ctx, func(ctx workflow.Context) { doSomething(ctx) })

The verifier at workflow startup does not enforce these rules statically — violations only surface as non-determinism errors during replay, which means they often appear in production under failure conditions. The rule of thumb: if it touches the outside world or produces a value that could differ between runs, put it in an Activity.

Code changes and versioning

Changing workflow code while instances are running requires care. Old instances have an event history produced by the old code. If you change the sequence of activities in a workflow and deploy, in-flight workflows will fail on replay because the new code produces a different command sequence than the history.

The solution is workflow.GetVersion() (Go) or workflow.patched() (Python):

1
2
3
4
5
6

v := workflow.GetVersion(ctx, "added-fraud-check", workflow.DefaultVersion, 1)
if v == 1 {
    // New path: includes fraud check
    workflow.ExecuteActivity(ctx, a.FraudCheck, input.OrderID)
}
workflow.ExecuteActivity(ctx, a.ChargeCard, input.OrderID, input.Amount)

Old instances replay with v == workflow.DefaultVersion and skip the new activity. New instances and new replays get v == 1 and include it. After all old instances complete, you clean up the version check.

Self-Hosting on Kubernetes

Temporal’s server is composed of four services: Frontend (API gateway, gRPC), History (event history storage and workflow state), Matching (task queue management, routes tasks to workers), and Worker (internal background tasks). They can run as separate Deployments or collapsed into a single binary for smaller deployments.

The official Helm chart is the recommended path for Kubernetes:

1
2
3
4
5
6
7
8

helm repo add temporalio https://go.temporal.io/helm-charts
helm repo update

# Install with PostgreSQL (you bring the database)
helm install temporal temporalio/temporal \
  --namespace temporal \
  --create-namespace \
  -f values.yaml

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53

# values.yaml — minimal production config
server:
  replicaCount: 3
  config:
    persistence:
      default:
        driver: sql
        sql:
          driver: postgres12
          host: postgres.temporal.svc.cluster.local
          port: 5432
          database: temporal
          user: temporal
          password: ${TEMPORAL_DB_PASSWORD}
          maxConns: 20
          maxIdleConns: 10
      visibility:
        driver: sql
        sql:
          driver: postgres12
          host: postgres.temporal.svc.cluster.local
          port: 5432
          database: temporal_visibility
          user: temporal
          password: ${TEMPORAL_DB_PASSWORD}

frontend:
  replicaCount: 2
  service:
    type: ClusterIP
    port: 7233

history:
  replicaCount: 3   # scale for throughput; each shard handles a portion of workflows

matching:
  replicaCount: 2

worker:
  replicaCount: 1

admintools:
  enabled: true

web:
  enabled: true
  replicaCount: 1
  service:
    type: ClusterIP
    port: 8080

elasticsearch:
  enabled: false   # optional; enables advanced search/visibility queries

The History service is the scaling bottleneck. It manages shards — each shard handles a subset of workflow executions. The default is 512 shards per cluster. For larger deployments, horizontal scaling of History replicas allows concurrent processing across shards. The database (PostgreSQL or Cassandra) is the actual persistence layer; Temporal strongly recommends Cassandra for large-scale deployments but PostgreSQL is well-supported and adequate for most self-hosted cases.

For local development, the docker-compose approach is simplest:

1
2
3

git clone https://github.com/temporalio/docker-compose.git
cd docker-compose
docker compose up   # starts temporal + postgres + temporal-ui on :8080

Connect workers to localhost:7233. The UI at localhost:8080 shows all running workflows, their histories, and their current state. This view alone is worth the setup — seeing the event history for a failed workflow in the UI immediately tells you exactly which activity failed, what error it returned, and how many retries it consumed.

Temporal vs. the Alternatives

The workflow orchestration space is crowded and the tools serve genuinely different problems:

Celery is the right choice when you have a Python application that needs to defer work to background processes and the coordination is simple (fan-out, simple chaining). The operational overhead is low, the ecosystem is mature, and the mental model maps to what most Python developers already know. Temporal adds significant complexity that Celery-appropriate workloads do not justify.

Airflow is the right choice for scheduled batch data pipelines. Its 60+ provider integrations (Snowflake, dbt, BigQuery, S3, Spark) and mature scheduler make it the default for data engineering. It is not designed for the “process a single user’s request reliably” use case — it is designed for “run this DAG every night at 2am.”

AWS Step Functions is the right choice when you are fully committed to AWS, need serverless execution (no workers to run), and your workflow logic fits into a state machine expressed in Amazon States Language JSON. The vendor lock-in and JSON-config-based workflow definition are significant trade-offs.

Temporal is the right choice when: your workflows have long durations or arbitrary waits, failure handling needs to be structural rather than bolted on, you want to write coordination logic in your primary language rather than a config DSL, and you are willing to operate the server (or pay for Temporal Cloud).

What Temporal Does Not Do

Temporal is not a scheduler. It does not replace cron for running jobs at regular intervals — though you can build cron-like behavior by having a workflow sleep and restart in a loop. workflow.Sleep(ctx, 24*time.Hour) is a valid pattern but is not the same as a cron job.

Temporal is not a stream processor. It is not a replacement for Kafka consumers or Flink. It handles individual workflow instances, not high-throughput event streams. If you are processing millions of events per second, Temporal is the wrong abstraction.

Temporal is not operationally free. The server has four services and a database dependency. Running it reliably in production requires attention to the History service’s shard distribution, database connection pooling, and careful schema migration on upgrades. Temporal Cloud (the managed offering) addresses this but costs money per action.

The honest characterization: Temporal solves a real and common problem — reliable multi-step distributed processes — better than anything else available. The complexity it introduces is real and front-loaded. Teams that operate it consistently report that it eliminates a category of bugs (partial failures, lost jobs, duplicate execution) that was previously a perpetual source of incidents. That trade is worth making for many systems. It is not worth making for a simple background job processor where Celery or a basic database-backed queue would suffice.