Load Lab

Objectives before numbers

The plan was written before the first ramp. It sets service level objectives for each service: an error budget, a latency target for accepted requests, what load shedding is allowed to look like, what has to recover and how fast, and what a twenty-minute run should be able to show. It also sets a workload model, with every number marked as an assumption so it can be questioned.

There is no production traffic to derive that model from, since a portfolio gets a visitor at a time. The model is a design target instead: a post that gets attention, a hundred visitors in an hour, five minutes each. Little’s law puts that at about eight people on the site at once. The expected-load profile runs twenty, so the objectives are met with room to spare. Next to the editors, a fixed rate of spectators open a shared link and only read, which is the kind of traffic that keeps arriving no matter how busy the service already is.

Six shapes (smoke, load, stress, spike, soak, breakpoint) are defined in one file, so “stress” means the same staircase for either service. The scripts are journeys rather than single endpoints. An editor gets a session, creates a flow, saves it against its version, runs it and follows the run to the end, pausing between steps the way a person does. Every run that mattered also had dotnet-counters on the process while k6 was on the socket. A flat p95 can mean the service has headroom, or that it is close to a garbage collection that would change the picture. Only the process can tell those apart.

What counts as a failure

Both services refuse requests on purpose when they are busy. Flow Studio has six execution slots; the SQL lab runs four queries at a time and queues eight more. Past those limits the answer is an immediate 429 instead of a request that waits with a spinner.

So the first thing the lab has to get right is how it counts. A 429 is the service working as designed and is recorded as refused. An error is a 5xx, a timeout, a body that does not parse, or a run that never finishes, and that is the rate the thresholds apply to. A load test that treated load shedding as breakage would fail the service for doing what it was built to do under load. A test that ignored the 429s would miss the point where the service started refusing.

The bug found before any load

The smoke profile runs first, at two users. Its first run against Flow Studio reported every run as never finishing.

The runs were finishing. The polling endpoint that reports their progress was answering 500 on the .NET 10 SDK: System.Text.Json now rejects a property whose name matches the polymorphic discriminator, and the [JsonIgnore] on the base class is not read on the overrides. The streaming endpoint had the same fault, and since it had already sent its headers, all it could do was close the connection. The existing test would have crashed on an empty stream instead of reporting the cause. It reports it now.

The fix is five attributes and a comment. The useful part is that a load test with a smoke stage in front of it also checks the correctness of the path it is about to load, and this one paid for itself before the load started.

The staircase

Twenty, forty, sixty and eighty editors, one minute each, with ten spectators a second the whole time. The run is cut at the stage boundaries instead of averaged, because the question a stress test asks is what changed between the steps:

editors	req/s	p95	p99	runs tried	accepted	accepted/s	refused
20	50	1.5ms	2.2ms	240	240	4.0	0%
40	82	1.6ms	2.5ms	511	356	5.9	30%
60	108	1.8ms	2.9ms	808	370	6.2	54%
80	133	1.9ms	2.9ms	1,103	378	6.3	66%

The accepted column is the result. Six slots, each held for about the second a paced run takes, gives a ceiling of roughly six runs a second. The staircase reached it between forty and sixty editors and stayed there. The p95 on the row where two thirds of the runs were refused is the same as on the row where none were. The process, sampled every five seconds, used under one percent of the machine at the top step. Under overload a service can queue, slow down or refuse, and this one refuses.

What only load showed

The SQL lab’s catalogue, GET /api/scenarios, is a cached read that takes about two milliseconds. Under the eighty-visitor spike its p95 was 258ms, which failed the one objective written to hold regardless of load.

The reason was that the global concurrency limiter metered every path except /health, so the catalogue waited in the same queue as the 800ms queries it lists. A page that cannot load its list of scenarios because other people are running scenarios is not protecting anything useful. The cap now applies to runs only, and the re-run of the same profile reads 1.9ms.

The environment varied more than the code

Three runs of that spike gave different results. The slow side’s p95 went 1.04s, 1.41s, then 2.17s, and the second and third runs used the same code. The query time the API measures around its own call grew in step with the round trip, which points below the API: the Postgres container in Docker Desktop’s VM, its 128MB of shared buffers against a 273MB dataset, and the laptop that the whole stack shares with the load generator.

That stopwatch includes the driver and the hop to the container, so it could not clear the process by itself. The script now records the database’s own execution time for every run, and a fourth spike was run with counters on the process and stats on the container. The two clocks agreed to a tenth of a millisecond, the API process stayed at 1.2% CPU, and the Postgres container peaked at seven cores. The variance came from how much of the laptop the database got. The conclusion in the report is that the SQL lab’s absolute latencies from this machine are not numbers to quote; the shapes and the ordering between the sides are.

Twenty minutes

On the client side the soak showed nothing changing: p95 1.8ms from the first minute to the last, nothing refused, nothing failed. The process showed two changes. The working set grew from 39 to 96MB across every five-minute window. That is not a leak. The run registry keeps every finished run for fifteen minutes and sweeps every ten, so 4,973 runs were held at about 12KB each, and a twenty-minute soak of a retention that can reach twenty-five minutes only sees the climb. What came out of it is the bound: run rate times retention times 12KB, about 115MB over baseline at the six-slot ceiling.

The second change was lock contention, from 45 to 743 per window under a constant load. Every run start counted the running runs by enumerating a concurrent dictionary’s values, which takes all of its internal locks, and the dictionary had five thousand more entries at the end than at the start. It is a counter now, incremented on begin and decremented on complete, and a second soak confirmed the growth is gone.

Against production

The per-visitor bucket (twelve runs a minute, four back every twenty seconds) is the only control between a public endpoint that runs SQL and anyone with a for loop, so it is checked as arithmetic rather than as a timing: thirty-six admitted in two minutes, Retry-After on every refusal. Locally, 35 of 235 requests were admitted, with the header on all 200 refusals. Against the deployed service on Fly.io, 35 of 197, with the header on all 162. The configuration that shipped is the one that was tested.

The report, the plan, the JSON behind every table and the tools that produced them are in the repository. Every run had k6 and the service on the same laptop, so the generator competed with what it was measuring. The absolute latencies belong to that machine; the shapes belong to the service. The CI gate runs smoke and load on every change, asserts the bucket exactly, sets its latency thresholds an order of magnitude wide, and compares against the previous run on main, failing only when a p95 doubles, because a gate that fails on noise gets ignored.