Apache Airflow Deep Dive: DAGs, Executors, and Production Realities

Apache Airflow is the default answer to “what should I use to orchestrate my data pipelines?” despite a loud chorus of newer tools (Dagster, Prefect, Temporal, Argo Workflows) trying to take its throne. It has won that position through sheer ubiquity — if you work in data, there is a non-trivial chance you have inherited an Airflow deployment at some point, whether you wanted to or not.

This post is the guide I wish existed when I first had to run Airflow in production. It covers what makes Airflow what it is, where its abstractions leak, how to write DAGs that stay maintainable past the first six months, which executor to pick, and how it compares to modern alternatives. I will assume Airflow 2.x with some references to 3.0 changes where they matter.

What Airflow actually is

Airflow is a scheduler that runs Python-defined workflows on a schedule. That sentence sounds trivial, but the pieces each matter:

• Scheduler — a long-running daemon that reads DAG definitions, decides which tasks should run now, and queues them.
• Python-defined workflows — DAGs are Python files that construct task objects. Airflow imports the files and interrogates the resulting Python objects to build its understanding of your pipelines.
• On a schedule — Airflow was designed around batch jobs that run on a cadence. It has grown to support event-triggered and near-real-time use cases, but the bones are cron.

An Airflow deployment has several components: the scheduler (decides what to run), the webserver (the UI and REST API), the metadata database (Postgres or MySQL — stores DAG parse results, task instances, connections, variables, XComs), workers (run tasks, in some executors), and for advanced setups, triggerer processes (for deferrable operators) and DAG processor daemons (for large installations that split parsing off the scheduler).

The scheduler does not run your tasks. It decides which tasks are ready and hands them to an executor, which is where “workers” come from. Pick the wrong executor and every other tuning decision is moot.

DAGs: the mental model

A DAG (Directed Acyclic Graph) is a Python object that contains tasks and dependencies. A minimal DAG:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

from airflow.decorators import dag, task
from datetime import datetime

@dag(
    dag_id="daily_revenue",
    schedule="0 2 * * *",
    start_date=datetime(2026, 4, 1),
    catchup=False,
    tags=["finance", "daily"],
)
def daily_revenue():
    @task
    def extract():
        return fetch_orders()

    @task
    def transform(orders):
        return aggregate(orders)

    @task
    def load(result):
        write_to_warehouse(result)

    load(transform(extract()))

daily_revenue()

The TaskFlow API (the @task decorator) was the biggest quality-of-life improvement in Airflow 2.0. Before it, tasks were operator subclasses and passing data between them required explicit xcom_pull calls. The decorator hides that machinery and makes Airflow look like ordinary Python.

Several concepts show up in every DAG and are worth getting right the first time:

start_date — the earliest date Airflow will consider running. Do not set this dynamically (datetime.now()). Airflow needs a stable reference point to compute which intervals to run. A fluctuating start_date breaks the scheduler’s interval math.

schedule — cron expression, timedelta, "@daily", "@hourly", a Dataset list (for data-aware scheduling), or None (only triggered manually). Prefer cron strings — they are precise, readable, and timezone-aware.

catchup — when True (the default for historical reasons), Airflow runs every scheduled interval between start_date and now. When you create a DAG with a start_date six months ago, catchup=True means 180 daily runs will immediately queue. Almost always set catchup=False.

max_active_runs — limits concurrent DAG runs. Default is 16. For DAGs with long-running tasks or shared resources, set this to 1 to guarantee serialized execution.

Data interval semantics — a DAG scheduled "@daily" with start_date=2026-04-01 does not run on 2026-04-01. It runs at the end of the interval 2026-04-01 → 2026-04-02, i.e., at the start of 2026-04-02. The data_interval_start and data_interval_end Jinja variables are the interval the run covers. This catches everyone once. Use logical_date / ds as the reference date for queries — it maps to the start of the interval.

Operators, sensors, hooks: the Airflow vocabulary

Operators are templates for tasks. PythonOperator runs Python code. BashOperator runs shell commands. KubernetesPodOperator runs a container. PostgresOperator runs SQL. There are hundreds of them in the provider packages (apache-airflow-providers-*).

Sensors are operators that wait for a condition: FileSensor, S3KeySensor, ExternalTaskSensor, SqlSensor. They poke at something on a schedule until it’s true. The critical knob is mode:

• poke (default) — holds a worker slot the entire time it’s waiting. A sensor waiting 6 hours blocks a worker for 6 hours.
• reschedule — releases the worker slot between pokes. For anything waiting more than a few minutes, use reschedule.
• Deferrable — sensors converted to deferrable operators run on the triggerer (an async event loop) and don’t hold worker slots at all. Prefer these when available.

Hooks are the connection clients: PostgresHook, S3Hook, SSHHook. They wrap the boilerplate of connecting to external systems using Airflow’s connection store. Don’t embed credentials in DAG code — define a Connection in the UI/env and use the hook.

Executors: the single biggest architectural choice

The executor decides how and where your tasks run. Picking the wrong one creates pain you can’t tune away.

SequentialExecutor — runs one task at a time, in process. For testing only. Uses SQLite. Never use in production.

LocalExecutor — runs tasks in subprocesses on the scheduler host. Good for small deployments where one beefy box handles both scheduling and execution. Limited to one host’s resources. Simple to operate. For teams with a few dozen DAGs and modest concurrency, LocalExecutor is genuinely underrated — it removes a whole class of distributed-system problems.

CeleryExecutor — tasks go into a Celery queue (Redis or RabbitMQ broker) and workers pull from it. Scales horizontally. Requires running and operating Redis/RabbitMQ, plus a pool of worker processes. The classic medium-to-large Airflow deployment.

KubernetesExecutor — every task runs in its own Kubernetes pod. Tasks are fully isolated, each can request its own resources, and idle resources cost nothing (pods terminate when the task finishes). Startup latency is the tradeoff — typically 15–30 seconds per task for pod scheduling and image pull. For short tasks this is death. For 10-minute+ tasks this is noise.

CeleryKubernetesExecutor — routes tasks with a queue="kubernetes" setting to K8s pods, everything else to Celery workers. The pragmatic “both” option: run frequent short tasks on persistent Celery workers, send heavy isolated tasks to K8s. This is what I run in production.

Decision matrix:

• Under 100 DAGs, single host OK → LocalExecutor
• Multi-host, stable workload, short tasks → CeleryExecutor
• Already on Kubernetes, mostly long tasks, varying resource needs → KubernetesExecutor
• Everything above, realistically → CeleryKubernetesExecutor

XCom: the nervous system and its limits

XCom (cross-communication) is how tasks pass data. A task returns a value, Airflow stores it in the metadata DB, downstream tasks pull it. The TaskFlow decorator handles this automatically:

1
2
3
4
5
6
7
8
9

@task
def extract() -> dict:
    return {"rows": 1_000_000, "path": "s3://..."}

@task
def load(info: dict):
    ...

load(extract())  # the dict flows through XCom

The trap: XCom is not for data. It is for pointers to data. The default backend stores XCom in the metadata database, serialized as JSON. Push a DataFrame there and your metadata DB will hate you — every task instance row carries that blob forever. Pass small metadata (paths, IDs, counts, timestamps); write the actual data to S3 / GCS / a database / a warehouse.

For larger XComs, Airflow 2.x supports custom XCom backends. The S3XComBackend pattern serializes large values to S3 and stores only the object key in the metadata DB. Worth configuring if teams are already abusing XCom.

Pools, priority, and concurrency controls

Airflow has a hierarchy of concurrency limits:

• core.parallelism — max tasks running globally across the whole install.
• core.max_active_tasks_per_dag — per-DAG cap.
• max_active_runs_per_dag — how many runs of a DAG can be active at once.
• Pools — named resource pools with a slot count. Tasks specify pool="bigquery", and Airflow ensures no more than the pool’s slots run concurrently.
• priority_weight — tie-breaker when multiple tasks compete for the same pool.

Pools are the underused power tool. Define a pool for every external system with a rate limit (bigquery_queries, salesforce_api, redshift_loaders) with a slot count matching your quota. Tasks that hit the limited system declare the pool. The scheduler will never overshoot the limit, and you stop worrying about pipelines DOS-ing a vendor API.

Idempotency: the property that makes Airflow usable

Every task in a production Airflow should be idempotent. Run it twice, get the same result. This is not optional; it is the difference between an orchestrator that saves your weekend and one that destroys it.

Idempotency shows up in a few concrete patterns:

• Writes go to a partitioned destination keyed by ds / data_interval_start. Re-running overwrites the partition cleanly.
• SQL operations use DELETE FROM ... WHERE ds = '{{ ds }}'; INSERT INTO ... SELECT ... or MERGE INTO. Never INSERT INTO without guards.
• File moves land in dated paths (s3://bucket/events/2026-04-19/...).
• External side effects (sending emails, kicking off API calls that do real things) are wrapped in a check-first pattern, or pushed to a step that explicitly does not re-run on retries.

The reward: airflow tasks clear ... becomes a safe, everyday operation instead of a last-resort panic button.

Templating: Jinja and the tricks that hurt

Airflow renders Jinja in fields that accept templates. {{ ds }} becomes 2026-04-19. {{ data_interval_start.strftime('%Y%m%d%H') }} becomes a timestamp. {{ ti.task_id }} is the current task ID. This is powerful and occasionally treacherous.

Templates render at task execution time, not DAG parse time. This matters because code that looks equivalent is not:

1
2
3
4
5
6
7
8

@task
def bad():
    return datetime.now().strftime("%Y-%m-%d")  # renders when the task runs

PostgresOperator(
    task_id="good",
    sql="SELECT * FROM events WHERE ds = '{{ ds }}'",  # renders from DAG context
)

The first captures a non-deterministic “now.” The second captures the logical date of the run, which is what you want.

Which fields are templated varies by operator. The TaskFlow API exposes the full context to a task via **context or specific kwargs (ti, params, ds, etc.). For regular operators, check the operator’s template_fields attribute.

DAG authoring patterns that hold up

A few practices separate DAGs that age well from ones that rot:

Keep DAG files lightweight. Airflow imports every DAG file on every parse loop (every 30 seconds by default). Top-level code runs each time. Don’t put database queries, API calls, or expensive imports at the module level. If you see scheduler_heartbeat alerts, the DAG files are the first suspect.

One DAG per file, named after the dag_id. Easier discovery, easier ownership, easier blast radius.

Use TaskGroups, not SubDAGs. SubDAGs are deprecated and were always a footgun (their own scheduler loop, nondeterministic triggering, confusing UI). TaskGroups are a purely visual construct — they group tasks in the UI without changing execution semantics.

Parameterize with params, not globals. The params dict flows into templated fields and can be overridden at trigger time, making DAGs reusable for backfills and ad-hoc runs.

Don’t generate DAGs dynamically at module scope unless you have to. It is tempting to read a YAML file and generate a hundred DAGs. This works until the YAML file changes during a parse, or the number grows past a few hundred and parse time explodes. If you must, use the dynamic DAG generation pattern with a dedicated DAG processor subprocess.

Make tasks fail loudly and specifically. Broad except Exception clauses that log and continue are how you end up with pipelines that “ran successfully” for a month while producing garbage. Airflow’s retry machinery handles transient errors — let it.

Dynamic task mapping: Airflow’s fan-out primitive

Added in 2.3, dynamic task mapping is Airflow’s answer to “run this task N times where N depends on upstream output”:

1
2
3
4
5
6
7
8
9

@task
def list_files() -> list[str]:
    return [f for f in s3.list("bucket/incoming/")]

@task
def process(file: str):
    do_work(file)

process.expand(file=list_files())

Each element of the upstream list becomes its own task instance. The UI shows them as a single mapped task with N parallel runs. This replaces the old pattern of generating N operators at DAG parse time based on some static input. Dynamic mapping evaluates at run time, so it adapts to however many files exist on a given day.

Good rule: the fan-out should be bounded. 10,000 tasks in a single mapped operator will slow the scheduler noticeably. For very wide fan-outs, offload to a single operator that does its own parallelism (a KubernetesPodOperator running a batched worker).

Deferrable operators: the quiet scaling fix

Airflow’s old sensors and long-polling operators burned worker slots while they waited. Deferrable operators move waiting to a lightweight async process (the triggerer). When the condition is met, the task resumes on a real worker for whatever finishing work it needs.

This changes what’s economically feasible. A team with 200 sensors waiting 8 hours each used to need 200 worker slots. With deferrable sensors, one triggerer process handles all of them on an asyncio event loop.

In provider packages, look for operators with deferrable=True options or *Async variants. Enable the triggerer in your deployment. The difference in resource usage is sometimes an order of magnitude.

Datasets and data-aware scheduling

Airflow 2.4 introduced Datasets: named entities that producer tasks update and consumer DAGs subscribe to. A DAG with schedule=[Dataset("s3://lake/events/")] runs when an upstream task declares it updated that dataset with outlets=[Dataset("s3://lake/events/")].

This is a step toward the event-driven model that Dagster’s assets and dbt’s refresh graph embrace natively. It is not as rich — Airflow Datasets are currently just URI triggers, no metadata or partitions — but it eliminates the old pattern of downstream DAGs using ExternalTaskSensor to wait for an upstream DAG. Airflow 3.0 is expanding this significantly with asset-like semantics; if you’re starting fresh, build around Datasets.

Connections, variables, and the secrets story

Airflow stores Connections (credentials for external systems) and Variables (key-value config) in the metadata database by default. Production Airflow should not use the metadata DB for secrets — it is the database your developers are dumping to troubleshoot.

Configure a secrets backend: AWS Secrets Manager, GCP Secret Manager, HashiCorp Vault, Azure Key Vault. Airflow looks there first, falls back to the DB. Set a prefix convention (airflow/connections/..., airflow/variables/...). Rotate through the secret manager, not the Airflow UI.

Observability: what to actually watch

Airflow is a scheduler; scheduler health is the metric to watch.

Scheduler latency (dag.dag_processing.last_duration and dag.dag_processing.total_parse_time) — if parsing takes longer than your scheduling interval, the scheduler falls behind and tasks don’t start on time.

Task duration histograms by DAG. Regressions here are the early warning that a pipeline is slowing down before it starts failing SLAs.

scheduler_heartbeat — if this stops, the scheduler died. Page immediately.

Pool utilization — pools consistently at 100% are bottlenecks; pools at 0% are probably misconfigured and not being used.

DAG run duration vs schedule interval — if a "@hourly" DAG routinely takes 90 minutes, the next run is late before it starts. Monitor the ratio.

Emit these to Prometheus via Airflow’s StatsD exporter or the OpenTelemetry integration. Grafana dashboards with per-DAG duration panels catch most production problems before they become incidents.

Testing DAGs: a realistic approach

Testing Airflow is genuinely hard because so much is runtime behavior. A pragmatic testing pyramid:

1. DAG loading tests — assert every DAG file imports without error and contains at least one task. Catches typos, missing imports, circular dependencies. Run in CI on every PR.

2. DAG structure tests — check the task graph against expectations (assert dag.task_ids == [...], assert "load" in dag.get_task("transform").downstream_task_ids). Catches accidental changes to dependencies.

3. Task logic tests — extract the non-Airflow logic into plain Python functions and unit-test those. The task itself becomes a thin wrapper. This is the biggest testability win available to Airflow authors: keep business logic out of operator subclasses.

4. Integration tests — use airflow tasks test <dag_id> <task_id> <date> or dag.test() in a pytest fixture. Runs a single task against real dependencies, without the scheduler. Good for verifying templating and XCom wiring.

5. End-to-end tests — optional, expensive. Spin up a full Airflow via docker-compose, fire a DAG, assert outputs. Worth it for a few critical DAGs.

Scaling Airflow: the hot spots

The parts that fail first as you grow:

Metadata DB load. The scheduler hammers Postgres. Every heartbeat, every task state transition, every serialized DAG, every XCom. At 10K+ task instances per day, use Postgres with connection pooling (PgBouncer), tune autovacuum aggressively on task_instance and dag_run tables, and consider the Standalone DAG Processor to offload DAG parsing writes.

DAG parse time. The scheduler walks every DAG file on a loop. Heavy imports at module scope kill parse time. Use the .airflowignore file to exclude Python files that aren’t DAGs. Set min_file_process_interval appropriately (default 30 seconds).

Celery broker. Redis with long lists of queued tasks. RabbitMQ with millions of messages. Both eventually hit limits. The fix is usually splitting into multiple queues with multiple worker pools, each backed by a smaller broker, or moving to KubernetesExecutor for the long-tail tasks.

Log shipping. Airflow writes task logs to worker disk by default. At scale, configure a remote logging backend (S3, GCS, or Elasticsearch). Worker disks fill up surprisingly fast.

How Airflow compares

Dagster — more opinionated, asset-oriented, better typing, stronger local development. Steeper initial learning curve; the abstractions are worth it if you’re building from scratch in the last few years. Real contender for new deployments.

Prefect — Python-first like Airflow, cleaner API, optional managed control plane. The “Airflow but friendlier” sell. Less ecosystem ubiquity.

Temporal — not really in the same category. Temporal is for durable application workflows (long-running business processes, saga patterns). If your “pipeline” is really “a ten-minute cross-service operation that needs retries and state,” Temporal fits better than Airflow.

Argo Workflows — Kubernetes-native, YAML-defined, container-first. Great for ML pipelines where every step is a container. Worse developer experience for iterating on Python business logic.

The honest summary: if you’re starting from scratch in 2026 and your use case is batch data pipelines, Dagster deserves serious consideration. If you inherit Airflow, or you’re in an ecosystem where everyone else uses it and interop matters, Airflow is fine — 2.x is a genuinely solid orchestrator and 3.0 is narrowing the gap with the newer tools.

An opinionated rollout

For a team adopting Airflow today:

1. Managed or self-hosted? Prefer managed (MWAA, Cloud Composer, Astronomer) unless you have strong reasons. Running Airflow is a real operational burden.
2. Executor: start with LocalExecutor on one host if you’re under 50 DAGs. CeleryKubernetesExecutor once you outgrow that.
3. Secrets backend from day one. Even if it’s just AWS SSM / Parameter Store.
4. Remote logging to S3/GCS from day one. Fixing this later is painful.
5. Use TaskFlow API, not old-style operators. Your future self will thank you.
6. One DAG per file, catchup=False unless you explicitly want backfills.
7. Pools for every rate-limited external system.
8. Tests for DAG loading, structure, and business logic (in that order of priority).
9. Monitor scheduler latency, task duration, and pool utilization — not just “did the DAG succeed.”
10. Start thinking in Datasets. Data-aware scheduling is where the platform is going.

Airflow rewards patience. The first few weeks are confusing — the scheduling semantics, the templating, the DAG parse model, the executor zoo. Once it clicks, it is a genuinely excellent tool for its job, and the installed base means there is almost always someone else who has already hit your problem.