Part 5 · 1 chapters · ~8 min

Exactly-Once and Transactions

Delivery semantics (at most once, at least once, exactly once), idempotent producers, transactional producers and the coordinator, read-process-write atomicity with offsets, read_committed consumers, zombie fencing with epochs, the boundary of exactly-once, and exactly-once in effect with idempotent sinks.

7

Transactions inside Kafka

code
// Java: a transactional read-process-write loop (Kafka Streams does this for you with processing.guarantee=exactly_once_v2)
producer.initTransactions();                       // fences older instances with the same transactional.id (zombies)
while (true) {
  var records = consumer.poll(Duration.ofMillis(200));
  producer.beginTransaction();
  for (var r : records) producer.send(new ProducerRecord<>("balances", r.key(), apply(r.value())));
  producer.sendOffsetsToTransaction(nextOffsets(records), consumer.groupMetadata());
  producer.commitTransaction();                    // or abortTransaction() on error
}
EXACTLY-ONCE: READ, PROCESS, WRITE
a transactional producer commits outputs and input offsets atomically
input topicstream processoroutput topic__consumer_offsetsread records (read_committed)beginTransaction()
swipe the figure sideways, or tap expand for full screen
1/4
transactional producer
A producer with a transactional.id can write to several partitions and commit consumer offsets in one atomic transaction, coordinated by a transaction coordinator on a broker.
a transactional.id groups writesa broker-side coordinator