Last updated June 30, 2026

Kafka topics are the structured representation of messages, and serve as the intermediary between producers and consumers. You can configure topic properties that define how data flows and persists through the topics in your Apache Kafka on Heroku add-on. Create and manage Kafka topics via the web dashboard and the CLI. This article covers the basics of CLI-based topic management.

Basic multi-tenant Kafka plans require a prefix on topics and consumer groups. See the differences between multi-tenant and dedicated. When integrating Kafka consumers, make sure to prefix topics and consumer groups with the value of the KAFKA_PREFIX environment variable. Otherwise, Kafka doesn’t receive the messages, and errors like Broker: Topic authorization failed or Broker: Group authorization failed can appear in Kafka debug events.

Automatic topic creation, or “create topic on first write,” isn’t available on Heroku.

Configuration Defaults and Limits

Parameter	Default	Lower Limit	Upper Limit
Replication	3	3	standard: 3, extended: 8 (number of brokers in a cluster)
Retention Period	24 hours	6 hours	standard: 2 weeks, extended: 6 weeks
Partitions per Topic	32	1	256
Partitions per Cluster	N/A	N/A	standard: 12,000, extended: 32,000 (4,000 x number of brokers in a cluster)

The Heroku-provided default settings are suitable for many apps, but for topics expected to handle billions of events per day, consider doing further research before entering production.

Plan your topic design carefully. You can’t change the partition count after topic creation. You can change retention and compaction settings at any time. You can also change the replication factor but this change triggers partition rebalancing across brokers.

Create a Topic

Use the heroku kafka:topics:create command to create a topic. Optionally, provide the topic name and options:

$ heroku kafka:topics:create my-example-topic --partitions 100 -a example-app
Creating topic my-example-topic with compaction disabled and retention time 1 day on kafka-animated-39618... done
Use `heroku kafka:topics:info my-example-topic` to monitor your topic.

Destroy a Topic

Use the heroku kafka:topics:destroy command to delete an existing topic. Provide the name of the topic to delete as an argument to the command:

$ heroku kafka:topics:destroy my-example-topic -a example-app
 ›   Warning: This command will affect the cluster: kafka-animated-39618, which is on example-app

✔ To proceed, type example-app or re-run this command with --confirm example-app example-app
Deleting topic my-example-topic... done
Your topic has been marked for deletion, and will be removed from the cluster shortly

List the Topics in your Cluster

You can list all topics on a cluster with the following CLI command:

$ heroku kafka:topics -a example-app
=== Kafka Topics on KAFKA_URL

  name               Messages   Traffic
 ─────────────────────────────────
  my-example-topic   45/sec      192 bytes/sec

See Topic Details

Use the heroku:topics:info command to check the details of an existing topic on your Apache Kafka on Heroku cluster. Provide the topic name to the command to specify the topic to review.

$ heroku kafka:topics:info my-example-topic -a example-app
=== kafka-animated-39618 :: my-example-topic

Producers:          45 messages/second (189 bytes/second) total
Consumers:          35 bytes/second total
Partitions:         32 partitions
Replication Factor: 3
Compaction:         Compaction is enabled for sample
Retention:          36 hours

Read or Write to a Topic

Use the heroku kafka:write and heroku kafka:tail commands to write and read messages from a topic.

kafka:write and kafka:tail only work in Private and Shield Spaces if you created IP rules for allowed sources.

Use the heroku kafka:topics:write to write a new message to an existing topic. Provide the topic name and the message.

$ heroku kafka:topics:write my-example-topic MESSAGE -a example-app

Use the heroku kafka:topic:tail command to subscribe to a topic and read any new messages written to it. Provide the name of the topic you want to consume messages from:

$ heroku kafka:topics:tail my-example-topic -a example-app

Understanding Partitions

Kafka topics consist of logical partitions. Partitions divide the log into shards. You can independently distribute each shard around the cluster and consume from them.

However, Kafka’s ordering guarantee only applies within an individual partition. This process means that Kafka consumes messages in the order they’re produced, but Kafka can interleave them if they span multiple partitions.

Most consumer libraries allocate a single consumer thread per partition. The number of partitions you select for your topics, and how you deliver messages to them, is crucial to the scalability of your app.

Use higher numbers of partitions if your consumers are relatively “slow” compared to your producers, such as if the consumers write into an external database. Apache Kafka on Heroku add-ons have a default of 32 partitions per topic, a maximum of 256 partitions per topic, and a maximum of 4,000 partitions times the number of brokers in a cluster. For example, 12,000 partitions for standard tier clusters, and 32,000 partitions for extended tier clusters.

Cleanup Policy

Kafka supports two primary modes of cleanup on topics: time-based retention and log compaction. Kafka supports the mixed use of these modes, for example, a topic can have time-based retention, compaction, or both modes enabled.

Time-Based Retention

Time-based retention is the default mode of cleanup for Kafka topics. Producers write messages to the partitions of a topic. Kafka timestamps each message and appends it to a log segment. Kafka periodically scans the log segments and removes messages that are older than the topic retention window.

$ heroku kafka:topics:retention-time my-example-topic '36 hours' -a example-app
Setting retention time to 36 hours for topic my-example-topic on kafka-animated-39618... done
Use `heroku kafka:topics:info my-example-topic` to monitor your topic.

Apache Kafka on Heroku has a minimum per topic retention time of 6 hours, and a maximum of 2 weeks for standard plans and 6 weeks for extended plans. Each topic can have different retention settings. The default retention time when creating a topic is 24 hours.

Log Compaction

Kafka supports an alternative configuration on topics known as log compaction. This configuration changes the semantics of a topic such that it keeps only the most recent message for a given key, tombstoning any predecessor. This process allows for creating a value-stream, or table-like view of data, and is a powerful construct in modeling your data and systems.

Compacted topics, without time-based retention enabled, don’t automatically reclaim storage space. The most recent version of a message for a given key persists until it’s actively tombstoned. The user tombstones it either by writing a new message of that key to the topic, or writing an explicit tombstone for that key. This process can cause compacted topics with unbounded keyspaces where the number of unique keys continuously grows, such as using timestamps or UUIDs as keys, to experience unbounded growth over time, driving unexpected resource utilization. Use and monitor compacted topics carefully to stay within plan limits.

Kafka doesn’t remove older messages of a given key that are tombstoned until the log-cleaner process clears them. This process is asynchronous, and multiple versions of a given key remain in the log until Kafka processes the current segment.

$ heroku kafka:topics:compaction my-example-topic enable -a example-app
Enabling compaction for topic my-example-topic on kafka-animated-39618... done
Use `heroku kafka:topics:info my-example-topic` to monitor your topic.

Replication

The replication factor of a topic determines the number of replicas or copies that Kafka maintains across the brokers in the cluster. Replication provides more durability and fault tolerance to the data in the Kafka cluster, while also increasing the volume of data managed over the cluster.

Kafka requires a minimum replication factor of 3 on the standard and extended plans. The maximum replication factor equals the number of brokers in the Kafka cluster (3 for standard plans, 8 for extended plans).

$ heroku kafka:topics:replication-factor my-example-topic 3 -a example-app
Setting replication factor for topic my-example-topic to 3... done
Use `heroku kafka:topics:info my-example-topic` to monitor your topic.

Increasing the replication factor of an existing topic can put more load on your Kafka cluster, as brokers create additional replicas. It also increases the size of the data stored in the cluster, which counts against plan capacity.

Producer Acknowledgment Configuration

The producer acknowledgment configuration, or producer acks, determines how many in sync replicas must acknowledge a write before it’s considered successful. This setting with the replication factor influences the latency and durability guarantees of your writes. This configuration resides in your producer code, but is important to consider alongside your cluster’s replication settings.

A configuration of acks=0 means that the producer doesn’t wait for any confirmation from the broker. The producer sends the message and immediately continues. This configuration minimizes latency but provides no guarantee Kafka received the message. The producer can’t retry a failed message delivery as it doesn’t receive any success or failure signals.

A configuration of acks=1 means that the producer waits for confirmation from only the leader broker. Producer retries can happen in this configuration, but data loss can occur if the leader broker fails before the data replicates to other brokers.

A configuration of acks=all or acks=-1 requires that all replicas are in sync and acknowledge the write before continuing. A high replication factor can increase the latency, but this factor yields the strongest guarantee of data resilience for writes.

Understanding ACLs

For customers using single-tenant standard and extended Apache Kafka on Heroku plans, we have a published, supported access control list (ACL) policy. We don’t change these ACLs without updating the changelog:

Your principal gets access to the following:

• Read,Write,Describe on name * for resource type Topic
• Read,Write,Describe,Delete on name * for resource type Group
• Describe,Write on name * for resource type TransactionalId

Maintenance

From time to time, Heroku performs maintenance tasks on Apache Kafka for Heroku clusters. Typical tasks include updating the underlying infrastructure of the cluster. Heroku handles these maintenance tasks automatically. Heroku doesn’t schedule maintenance events during an app’s maintenance window. Heroku sends a notice when the maintenance starts and ends. See the Apache Kafka on Heroku Maintenance FAQ for more information.