How to Integrate Apache Kafka with Spring Boot: A Production-Ready Guide

When a Spring Boot service needs to talk to another service without waiting on a synchronous HTTP call, message queues are the usual answer. Apache Kafka has become the default choice for this in most backend teams, but a lot of tutorials stop at a "hello world" producer and consumer that would never survive a real production load. Things like consumer retries, error handling, serialization of real objects, and graceful shutdown get skipped, and those are exactly the parts that page you at 2 a.m.

In this tutorial, you will build a Spring Boot application that produces and consumes JSON messages over Kafka. You will configure a producer and a consumer, send a typed object instead of a plain string, handle deserialization errors so one bad message does not block your whole consumer group, and verify the whole thing works end to end. By the end, you will have a small but realistic messaging setup you can build on.

Prerequisites

To follow along, you will need:

Java 17 or later installed. You can check your version by running java -version.
A Spring Boot 3.x project. You can generate one at start.spring.io with the Spring for Apache Kafka dependency added.
A running Kafka broker. The quickest way to get one locally is Docker, which the first step covers.
Basic familiarity with Spring Boot, including how @Component and application.yml work.

Step 1 — Running Kafka Locally with Docker

Before writing any code, you need a broker to talk to. Running Kafka by hand involves Zookeeper, broker configuration, and a fair amount of setup, so you will use Docker Compose to bring up a single-broker cluster instead.

Create a file named docker-compose.yml in your project root:

services:
  kafka:
    image: apache/kafka:3.7.0
    container_name: kafka
    ports:
      - "9092:9092"
    environment:
      KAFKA_NODE_ID: 1
      KAFKA_PROCESS_ROLES: broker,controller
      KAFKA_LISTENERS: PLAINTEXT://:9092,CONTROLLER://:9093
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@localhost:9093      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1

This uses Kafka's KRaft mode, which removes the Zookeeper dependency that older setups required. The KAFKA_ADVERTISED_LISTENERS value tells clients how to reach the broker; setting it to localhost:9092 is what lets your Spring Boot app connect from your machine.

Start the broker:

docker compose up -d

Confirm it is running:

docker compose ps

You should see the kafka container with a state of running.

With a broker available, you can now configure your application to connect to it.

Step 2 — Configuring the Producer and Consumer

Open src/main/resources/application.yml and add the Kafka configuration:

spring:
  kafka:
    bootstrap-servers: localhost:9092
    producer:
      key-serializer: org.apache.kafka.common.serialization.StringSerializer
      value-serializer: org.springframework.kafka.support.serializer.JsonSerializer
    consumer:
      group-id: orders-service
      auto-offset-reset: earliest
      key-deserializer: org.apache.kafka.common.serialization.StringDeserializer
      value-deserializer: org.springframework.kafka.support.serializer.ErrorHandlingDeserializer
      properties:
        spring.deserializer.value.delegate.class: org.springframework.kafka.support.serializer.JsonDeserializer
        spring.json.trusted.packages: "com.example.kafka.model"

A few of these settings are worth calling out, because they are the difference between a demo and something production-ready:

JsonSerializer and JsonDeserializer let you send and receive Java objects as JSON rather than hand-serializing strings.
auto-offset-reset: earliest means a new consumer group reads from the beginning of the topic rather than skipping existing messages, which is what you usually want during development.
ErrorHandlingDeserializer wraps the JSON deserializer. This is the important one. Without it, a single malformed message can throw a deserialization exception that the consumer cannot get past, and it will retry that same poisoned message forever, blocking every message behind it.
spring.json.trusted.packages restricts which packages the deserializer will instantiate, which prevents a class of deserialization security issues. Set it to the package where your message classes live.

Step 3 — Defining the Message Model

Create a simple record to represent the message payload. In a package called com.example.kafka.model, create OrderEvent.java:

package com.example.kafka.model;

public record OrderEvent(String orderId, String customer, double amount) {
}

Using a record keeps the model immutable and removes boilerplate. Because the package matches the spring.json.trusted.packages value from Step 2, the consumer will be allowed to deserialize incoming messages into this type.

Step 4 — Writing the Producer

Now create the producer that sends OrderEvent messages to a topic. In com.example.kafka, create OrderProducer.java:

package com.example.kafka;

import com.example.kafka.model.OrderEvent;
import org.springframework.kafka.core.KafkaTemplate;
import org.springframework.stereotype.Service;

@Service
public class OrderProducer {

    private static final String TOPIC = "orders";

    private final KafkaTemplate<String, OrderEvent> kafkaTemplate;

    public OrderProducer(KafkaTemplate<String, OrderEvent> kafkaTemplate) {
        this.kafkaTemplate = kafkaTemplate;
    }

    public void send(OrderEvent event) {
        kafkaTemplate.send(TOPIC, event.orderId(), event)
            .whenComplete((result, ex) -> {
                if (ex == null) {
                    System.out.println("Sent order " + event.orderId()
                        + " to partition "
                        + result.getRecordMetadata().partition());
                } else {
                    System.err.println("Failed to send order "
                        + event.orderId() + ": " + ex.getMessage());
                }
            });
    }
}

KafkaTemplate is the Spring abstraction for producing messages, and Spring Boot autoconfigures it from the properties you set in Step 2. Two details matter here:

Passing event.orderId() as the message key means all events for the same order land on the same partition, which preserves ordering for that order.
send() returns a CompletableFuture. Attaching a whenComplete callback lets you log delivery success or failure instead of firing the message and hoping it arrived. In production you would replace these println calls with a proper logger and metrics.

Step 5 — Writing the Consumer with Error Handling

Create the consumer that listens to the orders topic. In com.example.kafka, create OrderConsumer.java:

package com.example.kafka;

import com.example.kafka.model.OrderEvent;
import org.springframework.kafka.annotation.KafkaListener;
import org.springframework.stereotype.Component;

@Component
public class OrderConsumer {

    @KafkaListener(topics = "orders", groupId = "orders-service")
    public void consume(OrderEvent event) {
        System.out.println("Received order " + event.orderId()
            + " from " + event.customer()
            + " for $" + event.amount());
        // business logic goes here
    }
}

The @KafkaListener annotation does the heavy lifting: Spring polls the topic, deserializes each message into an OrderEvent, and calls this method for every record.

The error handling you set up in Step 2 is what makes this safe. Because the consumer uses ErrorHandlingDeserializer, a message that cannot be parsed into an OrderEvent is handed off as a failed record instead of crashing the listener. By default, Spring's error handler will retry a few times and then log and skip the bad record, so one corrupt message does not stall the entire partition.

Step 6 — Verifying It Works End to End

Wire the producer to a startup runner so you can see a message flow through. In your main application class, add a bean that sends a test event once the context is ready:

package com.example.kafka;

import com.example.kafka.model.OrderEvent;
import org.springframework.boot.CommandLineRunner;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.context.annotation.Bean;

@SpringBootApplication
public class KafkaApplication {

    public static void main(String[] args) {
        SpringApplication.run(KafkaApplication.class, args);
    }

    @Bean    CommandLineRunner runner(OrderProducer producer) {
        return args -> producer.send(
            new OrderEvent("ORD-1001", "Acme Corp", 249.99));
    }
}

Run the application:

./mvnw spring-boot:run

In the logs, you should see the producer confirm delivery and the consumer report receipt:

Sent order ORD-1001 to partition 0
Received order ORD-1001 from Acme Corp for $249.99

Seeing both lines means the message was serialized to JSON, published to the broker, pulled by the consumer group, deserialized back into an OrderEvent, and processed. The full round trip is working.

Conclusion

In this tutorial, you set up a local Kafka broker with Docker, configured a Spring Boot producer and consumer for typed JSON messages, and added deserialization error handling so a single bad message cannot block your consumer group. You also keyed messages by order ID to preserve per-order ordering and logged delivery results instead of sending blindly.

From here, you can extend this setup in several directions. You can route repeatedly failing messages to a dead letter topic so they can be inspected later, add a RetryTopic configuration for delayed retries, or introduce a schema registry with Avro if you need stronger contracts between producers and consumers.

推荐订阅源

DEV Community