AI Kafka Pipeline Demo

A minimal demo for Video 3 showing how a FastAPI gateway hands work to Kafka and how separate workers process the event chain.

Project Structure

ai-kafka-pipeline-demo/
├── api-gateway/
│   └── app/
│       ├── main.py
│       ├── routes/submit.py
│       ├── services/publisher.py
│       └── config.py
├── workers/
│   ├── extractor/
│   ├── summarizer/
│   └── notifier/
├── shared/
│   ├── kafka/
│   ├── schemas/
│   ├── config/
│   └── utils/
├── Dockerfile
├── docker-compose.yml
├── requirements.txt
└── .env

Demo Flow

1. POST /submit to the API gateway
2. Gateway publishes document.submitted
3. Extractor consumes and publishes text.extracted
4. Summarizer consumes and publishes summary.generated
5. Notifier consumes and logs final completion

Run

docker compose up --build

Test

Open another terminal:

curl -X POST http://localhost:8000/submit \
  -H "Content-Type: application/json" \
  -d '{
    "user_id": "user-1",
    "content": "Kafka helps decouple AI pipeline stages for scalable processing in production systems."
  }'

What you should see

• API returns Processing started
• Extractor logs the incoming event
• Summarizer logs the next event
• Notifier logs the final pipeline completion

Suggested narration

• FastAPI handles intake, not heavy processing.
• Kafka turns the request into an event.
• Each worker owns one stage.
• Shared schemas keep the contracts explicit.
• This is the simplest form of a production-style AI pipeline.
• I also recorded a full visual code walkthrough breaking down the project structure and explaining the design trade-offs here: https://youtu.be/c2ijN2KAWXw
• https://youtu.be/KjvbABpajUs