Rollback Chains: When Payment Fails, What Actually Happens
In the previous post, I showed the orchestrator's state transition table. It knows which topic to publish on failure. But what happens on the receiving end? What does "rollback" actually look like in code?
This post walks through three real failure scenarios in my saga system. Each one triggers a different rollback chain, and each service handles compensation differently.
How Compensation Works
Every service implements two operations: the forward action and its compensation.
| Service | Forward | Compensation |
|---|---|---|
| product-validation | validateExistingProducts() |
rollbackEvent() |
| payment-service | realizePayment() |
realizeRefund() |
| inventory-service | updateInventory() |
rollbackInventory() |
The forward action does the work and publishes SUCCESS or ROLLBACK to the orchestrator. The compensation undoes the work and publishes FAIL.
There's an important distinction between ROLLBACK and FAIL. ROLLBACK means "I failed, and I need my own compensation first." FAIL means "I already rolled back, now the previous service needs to compensate." The orchestrator uses this to chain rollbacks in the correct order.
Scenario 1: Inventory Fails (Full Rollback Chain)
This is the worst case. Product validation passed. Payment was charged. Then inventory fails because the product is out of stock.
The rollback chain goes: Inventory → Payment → Product Validation → Finish Fail.
Step 1: Inventory Detects the Problem
public void updateInventory(Event event) {
try {
checkCurrentValidation(event);
createOrderInventory(event);
updateInventory(event.getOrder());
handleSuccess(event);
} catch (Exception ex) {
log.error("Error trying to update inventory: ", ex);
handleFailCurrentNotExecuted(event, ex.getMessage());
}
producer.sendEvent(jsonUtil.toJson(event).orElseThrow());
}
private void checkInventory(int available, int orderQuantity) {
if (orderQuantity > available) {
throw new ValidationException("Product is out of stock!");
}
}
When stock is insufficient, checkInventory throws. The catch block calls handleFailCurrentNotExecuted, which sets status to ROLLBACK. The event goes back to the orchestrator with source=INVENTORY_SERVICE, status=ROLLBACK.
private void handleFailCurrentNotExecuted(Event event, String message) {
event.setStatus(ROLLBACK);
event.setSource(CURRENT_SOURCE);
addHistory(event, "Fail to update inventory: ".concat(message));
}
Step 2: Orchestrator Routes the Rollback
The orchestrator looks up the table: INVENTORY_SERVICE + ROLLBACK → INVENTORY_FAIL. It publishes to inventory-fail.
Step 3: Inventory Compensates Itself
The inventory-service consumes from inventory-fail and restores the previous quantities:
public void rollbackInventory(Event event) {
event.setStatus(FAIL);
event.setSource(CURRENT_SOURCE);
try {
returnInventoryToPreviousValues(event);
addHistory(event, "Rollback executed for inventory!");
} catch (Exception ex) {
addHistory(event, "Rollback not executed for inventory: ".concat(ex.getMessage()));
}
producer.sendEvent(jsonUtil.toJson(event).orElseThrow());
}
private void returnInventoryToPreviousValues(Event event) {
orderInventoryRepository
.findByOrderIdAndTransactionId(event.getOrder().getOrderId(), event.getTransactionId())
.forEach(orderInventory -> {
var inventory = orderInventory.getInventory();
inventory.setAvailable(orderInventory.getOldQuantity());
inventoryRepository.save(inventory);
});
}
Notice that it uses oldQuantity from the OrderInventory record. When the forward action runs, it saves both old and new quantities. The rollback reads the old value and restores it. No guessing.
After rollback, inventory publishes FAIL back to the orchestrator. Now the orchestrator sees INVENTORY_SERVICE + FAIL → PAYMENT_FAIL and publishes to payment-fail.
Step 4: Payment Refunds
public void realizeRefund(Event event) {
event.setStatus(FAIL);
event.setSource(CURRENT_SOURCE);
try {
changePaymentStatusToRefund(event);
addHistory(event, "Rollback executed for payment!");
} catch (Exception ex) {
addHistory(event, "Rollback not executed for payment: ".concat(ex.getMessage()));
}
producer.sendEvent(jsonUtil.toJson(event).orElseThrow());
}
private void changePaymentStatusToRefund(Event event) {
var payment = findByOrderIdAndTransactionId(event);
payment.setStatus(PaymentStatus.REFUND);
setEventAmountItems(event, payment);
save(payment);
}
Payment changes the status to REFUND and publishes FAIL. The orchestrator sees PAYMENT_SERVICE + FAIL → PRODUCT_VALIDATION_FAIL and continues the chain.
Step 5: Product Validation Rolls Back
public void rollbackEvent(Event event) {
changeValidationToFail(event);
event.setStatus(FAIL);
event.setSource(CURRENT_SOURCE);
addHistory(event, "Rollback executed on product validation!");
producer.sendEvent(jsonUtil.toJson(event).orElseThrow());
}
private void changeValidationToFail(Event event) {
validationRepository
.findByOrderIdAndTransactionId(event.getOrderId(), event.getTransactionId())
.ifPresentOrElse(
validation -> {
validation.setSuccess(false);
validationRepository.save(validation);
},
() -> createValidation(event, false));
}
Finally, the orchestrator sees PRODUCT_VALIDATION_SERVICE + FAIL → FINISH_FAIL and the saga ends.
Scenario 2: Payment Fails (Partial Rollback)
Payment fails (card declined, fraud blocked). Inventory was never touched, so there's nothing to roll back there. Only product validation needs compensation.
Payment ROLLBACK → payment-fail → Payment refunds → FAIL
→ product-validation-fail → Validation marks as failed → FAIL
→ finish-fail
Shorter chain. Same mechanism.
Scenario 3: Product Validation Fails (No Rollback Needed)
Product validation is the first step. If the product doesn't exist in the catalog, nothing was charged and nothing was reserved. The orchestrator sees PRODUCT_VALIDATION_SERVICE + FAIL → FINISH_FAIL and skips straight to the end.
The Pattern: Save Before You Change
Every service follows the same pattern for safe rollbacks. Before changing any data, save the original values.
The inventory-service saves oldQuantity and newQuantity in OrderInventory:
private OrderInventory createOrderInventory(Event event,
OrderProducts product,
Inventory inventory) {
return OrderInventory.builder()
.inventory(inventory)
.oldQuantity(inventory.getAvailable()) // save current
.orderQuantity(product.getQuantity())
.newQuantity(inventory.getAvailable() - product.getQuantity()) // save new
.orderId(event.getOrder().getOrderId())
.transactionId(event.getTransactionId())
.build();
}
The payment-service saves the payment with status PENDING before processing:
private void createPendingPayment(Event event) {
var payment = Payment.builder()
.orderId(event.getOrder().getOrderId())
.transactionId(event.getTransactionId())
.totalAmount(totalAmount)
.totalItems(totalItems)
.build();
save(payment); // status defaults to PENDING via @PrePersist
}
This way, the compensation logic always has the data it needs to undo the change.
Idempotency Guards
Each service checks for duplicate transactions before processing:
private void checkCurrentValidation(Event event) {
if (orderInventoryRepository.existsByOrderIdAndTransactionId(
event.getOrder().getOrderId(), event.getTransactionId())) {
throw new ValidationException("There's another transactionId for this validation.");
}
}
If Kafka delivers the same message twice (at-least-once delivery), the service detects the duplicate and skips the forward action. Without this check, you'd charge the customer twice or double-deduct inventory.
What's Next
The rollback logic is clean when you read the code, but things get tricky in practice. What if the refund fails? What if Kafka drops a message? In the next post, I'll cover how I test these scenarios with unit tests and real failure injection.
The repo: github.com/pedrop3/saga-orchestration