A data pipeline is an automated series of steps that moves data from where it is created to where it can be used, cleaning it on the way. Without one, every report starts with someone copying exports by hand, and no two reports quite agree.
Why raw data needs a pipeline
Most organisations collect data in many places at once. Sales sit in the shop's database, clicks arrive from the website's analytics, payments come from a provider's API and application logs pile up in files. Each source was built for its own job, not for analysis, so the data is scattered and inconsistent:
- Different formats. One system stores dates as
2026-03-01, another as01/03/2026, and a third as a Unix timestamp. - Different places. To answer 'which products sell best on mobile?' you need sales and clicks together, but they live in separate systems.
- Bad values. Missing prices, duplicate orders, test accounts and typos all creep in.
Querying the shop's live database directly for every report is also risky: a heavy analytical query can slow down the system customers are using. A pipeline solves both problems. It copies the data out on a schedule, fixes it once in one agreed way, and puts the result somewhere built for questions. The key word is automatically: the same steps run the same way every time, so the numbers are repeatable and nobody has to remember to do it.
ETL: extract, transform, load
The classic shape of a pipeline is ETL, named after its three steps in order.
Extract
Extracting means pulling data out of its sources: querying a database, calling an API, or reading files such as CSV or JSON exports. A good extract step changes nothing. It copies what is there, often only what is new since the last run (an extract), so the pipeline doesn't re-read years of history every night.
Wondering how this was made? How we make these