Introduction

To run data pipelines reliably in real-world production, you must understand how Airflow actually works behind the scenes. Airflow is essentially a system that coordinates and tracks jobs across different servers. It relies on four core components working together to schedule and run your workflows.

The 4 Core Architectural Components

     1. Webserver                  2. Scheduler
    (User Interface)              (Orchestration)
         │______________________________│
                        |
                        v
              3. Metadata Database
            (Central State Repository)
                        │
                        v
                   4. Executor
                (Execution Engine)

1. The Webserver

The Webserver is the user interface (UI) you see in your browser. It doesn't actually execute your code. Instead, it allows you to look inside the engine.
What it does: Shows you which tasks succeeded, which failed, how long they took, and lets you manually trigger or pause workflows.

2. The Scheduler

The Scheduler is the heart of Apache Airflow. It constantly monitors your workflow definitions and check times.
What it does: It looks at your schedule (e.g., “Run this DAG every day at midnight”) and checks task dependencies. When a task is ready to run, the Scheduler hands it off to be executed.

3. The Metadata Database

Airflow needs a single source of truth to remember everything that is happening. This is usually a relational database like PostgreSQL or MySQL.
What it does: It stores the state of your workflows. Every time a task starts, succeeds, or fails, the Scheduler writes that information here. The Webserver then reads this database to update your browser screen.

4. The Executor & Workers

While the Scheduler decides when a task should run, the Executor decides how it gets run. The Executor hands the actual work to Workers.
What it does: Depending on your setup, the Executor might run tasks sequentially on a single laptop, or scale them out across a massive cluster of computers (like Kubernetes) to process heavy data pipelines.

How a Workflow Flows: A Step-by-Step Example

Let's look at how these components interact when a workflow runs:

1. The Parsing: You write a Python file defining your DAG and save it. The Scheduler reads the file and saves the structure into the Metadata Database.
2. The Trigger: The clock strikes midnight. The Scheduler notices it is time to run your DAG. It updates the database status to "Running."
3. The Hand-off: The Scheduler hands the first task over to the Executor.
4. The Work: The Executor assigns the task to a Worker. The worker executes your script.
5. The Update: Once the worker finishes, it reports back. The status (Success or Failure) is written straight into the Metadata Database.
6. The Visual: You log into the Webserver UI, which reads the database and displays a nice green box showing you that your job completed successfully.

Writing Your First DAG: A Simple Python Example

Below is a simple workflow wrapped inside a clean with DAG block context manager. It has two steps: it extracts data, and then it processes that data.

from datetime import datetime
from airflow import DAG
from airflow.decorators import task

# Define the DAG configuration 
with DAG(
    dag_id="simple_data_pipeline",
    start_date=datetime(2026, 1, 1),
    schedule="@daily",
    catchup=False,
    tags=["beginner"],
) as dag:

    # Define the first task
    @task()
    def extract_data():
        print("Fetching data from the source API...")

    # Define the second task
    @task()
    def process_data():
        print("Data successfully cleaned and saved!")

    # Set the pipeline dependency flow using operators
    extract_data() >> process_data()

Code Breakdown

• The with DAG(...) as dag: Block: This acts as a container. Any task you write inside this indented block automatically belongs to this specific DAG, keeping things isolated and clean without needing messy function wrappers.
• The @task() Decorator: This turns a standard Python function into an official Airflow task. Airflow handles these independently, meaning if process_data fails, it won't break your underlying system.
• The >> Bitshift Operator: This acts like a physical arrow pointing downstream. It visually tells the Scheduler that extract_data() must run and finish successfully before process_data() is allowed to begin.
• The Parentheses (): When using @task decorators, you must add the parentheses (like extract_data()) in the layout. This explicitly tells Python to instantiate the task for Airflow to track.

When you save this file into your Airflow directory, the Scheduler will parse it, the Metadata Database will register it, and it will immediately show up as a visual diagram on your Webserver UI.

Conclusion

If you are just starting out in data engineering, building your first DAG is the ultimate way to turn chaotic scripts into a smart, automated system. By isolating work between the webserver, scheduler, database, and workers, Airflow is a great way to learn how modern, separate software systems work together.