Event Flow

Two views of one system

Customer view is the product: an order appears immediately as pending and is tracked until it settles. If payment is refused, the failure is handled in the interface. If it is merely slow — because the provider is not answering and the message is being retried — the interface says so, with the attempt number.

System view is the same events as machinery: four services in a chain, each node lighting as a message lands, real hop latency, real queue depth.

Most distributed-systems demos show only the second view. Showing both is the point: the diagram is why the customer’s experience holds together when a service goes down.

Failure, on purpose

There is a button that takes the payment service down. Orders keep being accepted; the message being worked on fails, and MassTransit retries it on an exponential backoff (~1.2 to 8.8s) while the queue grows in RabbitMQ. Bring payments back within twenty seconds and the message succeeds on its next attempt. Leave it, and after the sixth attempt the message lands in payments-authorize_error, the order starts saying “stuck, and nothing has been charged”, and a replay button appears — which deduplicates, because every consumer writes an idempotency row in the same transaction as its work.

The four patterns worth naming (transactional outbox, consumer idempotency, retry with exponential backoff, dead-letter and replay) are each one small file, with the reasoning written next to the code.

Measured, not asserted

Every number on screen comes from somewhere real: hop latency is the gap between the message’s send and receive timestamps; queue depths come from RabbitMQ’s management API; each service’s state is that service’s own /health response — the same five-field contract the rest of this site is built on, generated into C# and TypeScript from one JSON Schema.

So taking payments down changes the reported uptime and the status dot on this site’s home page, with nothing wired up manually for that to happen.

Two things are simulated and labelled in the source: the round trip to a payment provider, and the call to a warehouse. Everything else, between the browser and those two lines, is real.

On the broker

RabbitMQ, self-hosted in a container next to the services and orchestrated with .NET Aspire — it costs nothing to run.

Production work is on Azure Service Bus. The patterns move across unchanged, because the outbox, idempotency, backoff and dead-letter handling are transport-agnostic by construction; only the line naming the transport would change. Service Bus still gives sessions and scheduled delivery in the broker, which simplifies the delayed-redelivery machinery.