You have dbt models. They run when you type dbt run in your terminal. At some point — usually the moment someone asks “why didn't the dashboard update this morning?” — you realise that manual runs are not a data platform. You need orchestration: something that runs your pipelines on a schedule, tells you when they fail, and retries them automatically. The three tools most data teams reach for are Airflow, Prefect, and dbt Cloud's built-in scheduler.
The analogy: train network, rideshare, and a dedicated shuttle
Airflow is a national train network. Massive, reliable, runs on fixed timetables, connects everything. But setting up a new route requires engineering work. The infrastructure is substantial. Miss one connection and everything downstream is affected. It handles complexity well — but it brings its own complexity.
Prefect is a rideshare platform. More flexible, faster to get started, handles route changes gracefully. The interface is modern. But it has fewer drivers in the ecosystem and has not been battle-tested at the same scale as Airflow.
dbt Cloud Scheduler is a dedicated shuttle service. It goes from A to B on a fixed schedule and does that one thing perfectly. Zero infrastructure to manage. But it only goes to one destination: dbt. If you need to stop somewhere else first — run a Python script, trigger an API, train a model — it cannot help you.
What orchestration actually means
An orchestrator does four things:
- Scheduling — run this pipeline at 6am every day
- Dependency management — run task B only after task A succeeds
- Retries — if task B fails, try again up to three times before alerting
- Observability — show me which tasks ran, which failed, and why
Without an orchestrator, you are doing all of this manually or with fragile cron jobs. A cron job with no retry logic, no alerting, and no logging is not orchestration — it is a time bomb.
Apache Airflow
Airflow is the industry standard. Released by Airbnb in 2014, it is the most widely deployed orchestrator in data engineering. Pipelines are defined as DAGs (Directed Acyclic Graphs) in Python — each node is a task, each edge is a dependency.
Where Airflow excels:
- Complex pipelines that involve multiple tools: ingest from Fivetran, run dbt, call an ML model, send a Slack notification, update a dashboard — all in one DAG.
- Large teams that need fine-grained control over task dependencies and scheduling.
- Organisations that already have Airflow infrastructure and experience.
The downsides: Airflow has a steep learning curve. Setting it up correctly — worker queues, the metadata database, scaling executors — is non-trivial. Managed options like AWS MWAA, Google Cloud Composer, and Astronomer remove the infrastructure burden but add cost.
Prefect
Prefect was built as a modern alternative to Airflow. Flows are written in plain Python — no DAG syntax, no XML, no boilerplate. The Prefect Cloud UI is significantly more intuitive than Airflow's. Failure handling and dynamic task mapping are first-class features.
Where Prefect excels:
- Teams starting fresh who want Airflow's power without Airflow's setup pain.
- Dynamic pipelines where the number of tasks is not known in advance.
- Python-heavy workflows where DAG syntax feels unnatural.
The downsides: smaller ecosystem than Airflow. Fewer pre-built provider integrations. Less battle-tested at extreme scale. The open-source server is available but Prefect Cloud is where most of the polished tooling lives.
dbt Cloud Scheduler
dbt Cloud is the managed hosting platform for dbt Core. Its built-in scheduler lets you run dbt build, dbt run, or dbt test on a cron schedule, triggered by a Git push, or via API. It shows model-level pass/fail results, integrates directly with dbt's lineage graph, and sends alerts on failure.
Where dbt Cloud excels:
- Your only orchestration need is running dbt. You do not need to orchestrate ingestion tools, Python scripts, or ML jobs.
- You want zero infrastructure to manage. It just works.
- The native dbt integration is unmatched — you can see which individual models failed, view the compiled SQL, and re-run only failed nodes.
The downsides: it cannot orchestrate non-dbt tasks. If your ingestion runs on Fivetran and your transformations run on dbt, you will need to coordinate these separately — dbt Cloud cannot wait for Fivetran to finish before triggering dbt. Pricing starts at around $100 USD/month for the developer plan.
The most common mistake: reaching for Airflow before you need it. Setting up Airflow for a one-engineer team with three dbt models is weeks of infrastructure work for no return. Start with dbt Cloud if dbt is all you need. Graduate to Airflow or Prefect when you have multi-tool pipelines that require it.
How to choose
Three questions narrow it down quickly:
- Do you need to orchestrate anything beyond dbt? If no — Python scripts, ingestion jobs, ML models — dbt Cloud Scheduler is the right answer. It is simpler, faster, and purpose-built for your use case.
- Are you starting fresh or inheriting an existing setup? If you are building from scratch and need a full orchestrator, Prefect has a lower setup cost than Airflow. If your company already runs Airflow, staying on it is usually right unless the pain is significant.
- How complex are your pipelines? If a DAG has more than a dozen tasks involving multiple systems, Airflow's maturity and ecosystem justify its complexity. For simpler pipelines, Prefect or dbt Cloud is proportionate.
A note on Fivetran and ingestion timing
A common question: how do you ensure dbt runs after Fivetran finishes loading? With dbt Cloud alone, you cannot. The cleanest solutions are:
- Use Fivetran's dbt Cloud integration — Fivetran can trigger a dbt Cloud job via API when a sync completes.
- Use Airflow or Prefect to sequence the Fivetran sync and the dbt run as steps in the same pipeline.
- Accept a time-based approach — schedule dbt to run 30 minutes after Fivetran is expected to finish. Simple and fragile, but fine for low-stakes pipelines.
For most early-stage data teams, the Fivetran → dbt Cloud API trigger is the cleanest option: no infrastructure, no Airflow, reliable sequencing.