Event Flow

Two views of one system

Customer view is the product: someone places an order, sees it appear immediately as pending, and watches it settle. When payment is refused the failure is handled in the interface rather than thrown at the user. When payment is merely slow — because the provider is not answering and the message is being retried — the interface says so, in those words, with the attempt number.

System view is the same events read as machinery: four services in a chain, each node lighting as a message lands, the real latency of each hop, the real depth of each queue.

Most distributed-systems demos show only the second view, which quietly assumes the audience already cares about queues. Showing both is the point — the customer never sees the diagram, and the diagram is why the customer’s experience holds together when a service goes down.

Failure, on purpose

There is a button that takes the payment service down.

What follows is not an animation. Orders keep being accepted. The message being worked on fails, and MassTransit retries it on an exponential backoff — waits of about 1.2, 1.8, 2.8, 4.8 and 8.8 seconds — while every order behind it waits in the RabbitMQ queue, where the system view counts it by asking the broker. Bring payments back inside that twenty seconds and the message succeeds on its next attempt. Leave it, and after the sixth attempt the message lands in payments-authorize_error, the order stops saying “processing” and starts saying “stuck, and nothing has been charged”, and a replay button appears.

Press replay twice and the second pass deduplicates, because every consumer writes an idempotency row in the same transaction as its work.

The four patterns worth naming — transactional outbox, consumer idempotency, retry with exponential backoff, dead-letter and replay — are each one small file with the reasoning written next to the code.

Measured, not asserted

Every number on screen comes from somewhere real. Hop latency is the gap between the message’s own send and receive timestamps. Queue depths come from RabbitMQ’s management API. Each service’s state is that service’s own /health response — the same five-field contract the rest of this site is built on, generated into C# and TypeScript from one JSON Schema.

So taking payments down does not just change a colour in a diagram. It changes what that service reports about itself, which changes its uptime figure, which changes the status dot on this site’s own home page. Nothing had to be wired up for that to happen; it falls out of everything reading the same contract.

Two things are simulated, and both are labelled in the source: the round trip to a payment provider, and the call to a warehouse. There is no provider and no warehouse. Everything between the browser and those two lines is the real thing.

On the broker

RabbitMQ, self-hosted in a container next to the services and orchestrated with .NET Aspire. It costs nothing to run, and a broker sitting beside its services is a better demonstration than one where the interesting part is somebody else’s dashboard.

Production work is on Azure Service Bus. The patterns here move across unchanged — the outbox, the idempotency ledger, the backoff, the dead-letter handling are all transport-agnostic by construction. What would change is the two lines that name the transport, and one property: Service Bus gives you sessions and scheduled delivery in the broker, so the delayed-redelivery machinery gets simpler rather than harder.