Database Technologies

What is a data pipeline? ETL explained

How a data pipeline works: ETL steps to extract, clean and load data into a warehouse, batch versus real-time runs, and where Airflow, Kafka and dbt fit in.

  • 7 min read
  • 5 questions
  • +35 XP
Transcript

Data Pipelines, explained by cats. Kitty collects data. Lots of it. Tuna sales. User clicks. App logs. But raw data is messy. Different formats. Different places. Chaos. So Kitty builds a data pipeline. A data pipeline moves and transforms data from one place to another.

Automatically. An ETL pipeline follows the following steps. First: Extract the data. Pull data from sources. Databases. APIs. or Files. Next: Transform the data. Clean it. Filter it. Fix bad values. Combine datasets. Then: Load the data. Store it somewhere useful.

A data warehouse. Analytics system. Now the data is ready for. Dashboards. Reports. or Machine learning. Pipelines can run on schedules. Or in real-time. Tools like Airflow, Kafka and DBT, all help move and shape the tuna. That’s a data pipeline. Messy data goes in. Organised data comes out. ready for Kitty to analyze and take over the world.