Every data team faces the same choice sooner or later: buy a cloud data warehouse and get speed and structure, or dump everything into a data lake and save money on storage. Both options come with real trade-offs. Databricks was built on the premise that this is a false choice — and the platform they created in solving it has become the backbone of the modern data stack.
The analogy: a furnished apartment and an empty plot of land
A data warehouse is like renting a fully furnished apartment. Everything is in place. The rules are clear. Your analysts can move in and start querying the same day. But you pay premium rent, you cannot knock down walls, and when you eventually leave, your data goes with you in the vendor's proprietary format — not always easy to migrate.
A data lake is like buying an empty plot of land. It is cheap. You own it outright in your own cloud storage — Amazon S3, Google Cloud Storage, Azure ADLS. You can store anything: structured tables, raw JSON, logs, images. But it is still just land. There is no house, no plumbing, no electricity. Building those things requires serious engineering effort, and many teams never fully get there. The result is what the industry calls a data swamp: a field full of files that nobody can reliably query.
Databricks builds the house on the land. Your data stays in your own cloud storage, in open formats you control. But Delta Lake — the storage layer Databricks created — adds structure, transactions, and reliability on top. Databricks then adds the tools to work with that data: SQL for analysts, Python and Apache Spark for engineers, and a full suite of machine learning tools for data scientists. One copy of data. One platform. Three types of work.
Where Databricks came from
Databricks did not begin as a data platform. It started as a solution to a specific problem: Hadoop was slow.
In the late 2000s, Apache Hadoop was the dominant way to process large datasets. It worked by breaking data into chunks, distributing them across many machines, and processing them in parallel. The problem was that Hadoop wrote intermediate results to disk between every step. For complex multi-step jobs, that constant disk I/O added up fast — a job that should take minutes could take hours.
In 2012, researchers at UC Berkeley's AMPLab published a paper describing a new framework that kept intermediate data in memory between steps instead of writing to disk. They called it Apache Spark. For iterative workloads like machine learning and complex transformations, Spark was roughly 100 times faster than Hadoop.
In 2013, the same researchers founded Databricks to turn Apache Spark into a managed cloud service. Instead of provisioning and configuring a Spark cluster yourself, you could use Databricks and have a working environment in minutes. The company grew quickly, but the underlying problem remained: data in lakes was unreliable. No transactions, no schema enforcement, no way to roll back a failed job.
In 2019, Databricks open-sourced Delta Lake — a storage layer that adds ACID transactions, schema enforcement, and time travel to files sitting in cloud storage. In 2020, they named the resulting architecture the Lakehouse: the cost and openness of a data lake, combined with the reliability and performance of a data warehouse. The platform has expanded significantly since, but the Lakehouse concept remains its foundation.
Warehouse vs. lake vs. lakehouse
The five components of the Databricks platform
Databricks is five distinct tools that work together. Understanding what each one does makes the platform considerably less intimidating.
Apache Spark — the engine
Apache Spark is the distributed compute engine underneath everything on Databricks. When you run a Python transformation or a SQL query against hundreds of millions of rows, Spark splits the work across a cluster of machines that all process their slice simultaneously. You write the logic once — in Python, SQL, or Scala — and Spark handles the parallelisation. This is why Databricks can process data at a scale that would be impractical on a single machine.
Delta Lake — the storage layer
Your data files live in cloud storage in an open format called Parquet — a column-oriented file format designed for analytical workloads. Delta Lake wraps those files in a transaction log: a record of every change ever made to the dataset. This transaction log is what gives Delta Lake its key properties:
- ACID transactions — if a pipeline job crashes halfway through writing a million rows, the partial write is rolled back. No corrupted data left behind.
- Schema enforcement — a column renamed upstream breaks the pipeline with a clear error, rather than silently writing incorrect data downstream.
- Time travel — you can query the table as it existed at any point in the past. Useful for auditing, debugging, and recovering from accidental deletes.
Unity Catalog — governance
Unity Catalog is a centralised registry for all data across a Databricks environment. It tracks who can access which table, where each dataset originally came from (data lineage), and what tables and columns exist across all workspaces. Without centralised governance, access control tends to be managed through spreadsheets and tribal knowledge — both of which break down as teams grow.
MLflow — machine learning lifecycle
When data scientists train machine learning models, they run many experiments: different algorithms, different parameters, different feature sets. MLflow records every run — what inputs were used, what the performance metrics were, which version was promoted to production, and how to reproduce the result. Without experiment tracking, ML projects become difficult to audit and nearly impossible to reproduce reliably.
Databricks SQL — analytics for SQL users
Not everyone on a data team writes Python or works in notebooks. Databricks SQL provides a warehouse-style query interface sitting on top of the same Delta Lake tables that engineers and data scientists use. Analysts write SQL, connect tools like Tableau or Looker directly, and get fast results — without needing to understand clusters or Spark. Crucially, there is no separate copy of the data: everyone works from the same source.
How the platform fits together
Who uses Databricks and for what
Three distinct roles use Databricks in different ways, often within the same organisation.
Data engineers use it to build and run ETL pipelines. Instead of writing stored procedures or SQL-only transformations, they write Python logic using Spark, orchestrate jobs with Databricks Workflows, and store outputs in Delta Lake tables. This is where Databricks is strongest — complex, large-scale transformations that outgrow what SQL alone can handle.
Data scientists use it for the full machine learning workflow: exploring data in notebooks, engineering features on datasets that would not fit in memory on a laptop, training models with Spark's distributed compute, tracking experiments in MLflow, and deploying models to production.
Analysts use Databricks SQL to query Delta Lake tables without touching notebooks or Python. For many organisations, this is how analysts start accessing data that was previously only reachable by engineering teams.
Databricks and Snowflake are not mutually exclusive. A common pattern is to use Databricks for the heavy engineering work — ingestion, complex transformation, ML — and a warehouse like Snowflake or BigQuery as the serving layer for dashboards and reporting. The two tools overlap in some areas but are often used together in practice.
When Databricks makes sense — and when it does not
Databricks is powerful, but it is not always the right starting point.
It makes sense when your pipelines are outgrowing pure SQL and you need Python and Spark; when you are running ML workloads that require feature engineering at scale and proper experiment tracking; when you want to keep data in open formats you own rather than locked in a vendor's proprietary storage; or when your team is comfortable with Python, notebooks, and cloud infrastructure.
It adds complexity you may not need when your team works primarily in SQL; when you are just starting out and do not yet have the engineering capacity to manage clusters and runtimes; or when your data volumes are modest and a managed warehouse like BigQuery or Snowflake would handle them easily.
Most growing data teams reach Databricks eventually — particularly as machine learning and AI become part of the strategy. But it does not need to be the first tool. The question is not whether Databricks is capable (it is), but whether your team is at the point where its power justifies its complexity.