A topic is a set of ordered logs
Kafka partitions provide ordering within each partition. A consumer group assigns partitions among its members so the group can share work. A second group can independently read the same topic for another purpose. Kafka's introduction describes these core abstractions.
Imagine a topic with partitions P0, P1 and P2. In one illustrative assignment, consumer A handles P0 and P1 while B handles P2. If C joins, ownership may be redistributed. This diagram describes ownership, not the precise assignment produced by every strategy.
Choose keys from ordering requirements
If events for one order must remain ordered, use a consistent order identifier with a compatible partitioning strategy. Do not assume an event in P0 is processed before an event in P1 just because its timestamp is earlier. Cross-partition coordination belongs in the application.
Adding partitions can change key-to-partition mapping for common partitioners. Plan how the application handles older and newer events for the same business entity during that change. For a strict ordering requirement, test the actual producer strategy and migration procedure.
A customer’s events must stay in sequence
Imagine an order stream containing Created, Paid and Cancelled events. If the product requires those events to be consumed in order for one order, use a stable key that routes that order's events to the same partition. Kafka ordering is a partition boundary; a topic with multiple partitions does not provide one global timeline. A consumer may also launch asynchronous tasks whose completion order differs from the order of records it read, so application concurrency can break assumptions even when the log is ordered.
Define the business ordering rule explicitly. Some updates can be applied independently; others need a sequence number, version check or per-key serialization. Keep the original event identity with the resulting record so that a replay remains explainable. The partition count affects available consumer parallelism, but a single hot key can still dominate one partition. Adding more group members cannot split that key's work without a new business design. Observe lag and processing time per partition, because a healthy topic-level average can conceal one customer whose events are consistently delayed.
Choose keys and partition count around the workload
An order identifier distributes many independent orders while preserving each order's sequence. A tenant identifier preserves a broader tenant sequence but can concentrate a large customer's traffic. An unkeyed stream can spread records more freely, but the application must not assume per-entity order from that distribution. The right key follows the invariant and observed skew, not merely the field that is easiest to serialize.
A consumer group divides assigned partitions among its members; separate groups independently read the topic for separate applications. Extra members beyond the available partitions do not automatically increase useful processing parallelism. Increasing partition count is also a routing change for common partitioning schemes, so assess ordering assumptions and producer behavior before treating it as an invisible capacity adjustment. Keep partition growth, retention, broker resources and downstream concurrency in the same plan. A stream processor that reads faster can still overwhelm its database. Specify how many records may be in flight and when their offsets become eligible for commit.
Offsets express progress
| Sequence | Crash window |
|---|---|
| Commit, then perform effect | Work can be skipped |
| Perform effect, then commit | Work can repeat |
| Transactional Kafka output plus offsets | Can coordinate within Kafka's transactional scope |
A committed offset records where the group should resume. It does not prove an external payment or database effect happened exactly once. A restarted consumer can repeat effects performed before an uncommitted offset. Use idempotent writes or deduplication at that boundary.
Rebalancing changes the owner
A consumer joining, leaving or failing can trigger reassignment. Stop or fence old work appropriately when ownership is revoked; otherwise an old worker may finish after a new owner has started. Track partition lag, processing latency and rebalance activity rather than only aggregate throughput.
Kafka's design documentation describes delivery semantics and transactional scope. Its consumer protocol documentation covers the newer incremental protocol; check the documentation for your deployed version before changing settings.
Build a small ownership exercise
Produce numbered events for two order keys. Start two consumers, then add a third. Log partition, offset, key and operation identifier. Restart one consumer after its business effect but before its offset commit. Verify per-key order and inspect any duplicate effects. Do this with disposable topics and fixtures.
More consumers than partitions do not automatically create more parallel work in a standard consumer group. Partition count also affects management and ordering decisions, so choose it from the workload rather than maximizing it blindly.
Continue with delivery semantics and protocol migration.
Diagnose lag by ownership and progress
When lag rises, inspect partition assignments and the rate at which each member makes durable progress. A rebalance can change ownership while an old member still has asynchronous work in flight. Handlers should stop accepting work for revoked assignments and follow the client library's lifecycle rules. Offsets should reflect the processing guarantee you intend; committing beyond unfinished records can lose work after a crash, while committing only after durable effects can cause safe replays that require deduplication.
Rehearse a member restart with pending records and verify the resulting business state, not only that another member receives an assignment. Look for poison records that fail repeatedly and block progress; use a documented retry or quarantine policy that retains the original record identity and error context. Increasing batch size may improve efficiency while increasing recovery work and processing duration. Monitor rebalance frequency, oldest pending work, per-partition lag and downstream saturation. Those signals distinguish uneven keys from a slow dependency and make partition ownership an operational fact rather than an assumption hidden inside a diagram.
