A p95 from six samples is just your slowest call

A dashboard that says p95: 35s from six measurements is not telling you about the ninety-fifth percentile. It is telling you the slowest of six calls, under a name that implies something much stronger. This is not a rounding concern — below twenty samples the two are the same number by construction, and the label is the only thing that changes.

We shipped exactly this, caught it on 2026-08-31, and the correction is three lines. The reasoning is worth more than the fix.

The arithmetic

The nearest-rank method — the default in most quick implementations, and the one you get from a hand-written percentile function — sorts the readings and picks the value at index ceil(p × n).

For the 95th percentile that index is ceil(0.95 × n). Work it through:

The threshold falls out of the inequality rather than from taste. You need ceil(0.95n) < n, which holds only once n ≥ 20. Below that every "p95" you compute is the maximum, whatever you call it in the column header.

Medians are cheap; tails are expensive

The useful consequence is that the two figures on a latency column have very different costs, and treating them as one pair with one threshold is what causes the problem.

So a monitor that gates both behind the same count either publishes a fake tail early or withholds a perfectly good median for hours. We had the first version, which is the worse one: it printed a number that looked precise and was not.

Splitting the floors fixed both ends. The median appears within minutes of a model being probed, and the tail waits until it is a tail. In the meantime the column shows the median alone, which is less to look at and more true.

Interpolation does not fix it — it hides the tell

If you reach for a statistics library instead of writing the index yourself, you usually get linear interpolation rather than nearest-rank. NumPy and pandas both default to it. The index becomes (n − 1) × p, and when that lands between two readings the result is interpolated between them.

For six samples that is 5 × 0.95 = 4.75 — three quarters of the way from the fifth reading to the sixth. So the answer is no longer exactly the maximum, and this sounds like an improvement.

It is not. With one reading at 30 seconds and five around two, the interpolated p95 comes out near 23 — still overwhelmingly determined by the single worst call, but no longer equal to it. The problem is unchanged and the symptom is gone. Nearest-rank at least announces itself: a p95 that exactly equals your slowest call is a tell that anyone can spot. An interpolated number in between looks like a real estimate.

Neither method creates information that the sample size does not contain. A percentile is a claim about how often something happens, and no amount of arithmetic extracts a one-in-twenty claim from six draws.

What to show while you wait

Withholding a figure leaves a gap, and a column of dashes reads as broken rather than as pending. Three options, in order of how much we would recommend them:

• Show the median alone. It is available almost immediately and it is the number most readers want anyway. This is what we do.
• Rename the column to what you actually have. "Slowest of 12" is honest, informative, and nobody will misread it as a distribution.
• Widen the window instead of lowering the floor. A p95 over seven days on a low-traffic endpoint may clear twenty samples when a five-minute window never will — at the cost of reacting slowly to change.

What we would not do is keep the label and lower the threshold. The column header is a claim about what the number means, and it is the one part of a dashboard that readers take entirely on trust.

Why this survives review

What makes this bug interesting is not the arithmetic — it is why the wrong number stood.

Our first fix set the floor at twelve, with a comment explaining that twelve was "the point where the nearest-rank index stops landing on the last element". The reasoning was right and the number was wrong: at twelve, ceil(11.4) is 12, which is still the last element. A confident comment made a wrong constant look considered.

It also had a test. The test built a fixture of identical readings and asserted the p95 came out to that value — which passes at any floor, because when every reading is 2000ms, the maximum is also the median is also the p95. A fixture with no spread cannot detect a statistic that has collapsed to the maximum.

The test that catches it asserts the property rather than a value: given readings with one distinct outlier, the published p95 must not equal that outlier.

// One slow call among twenty. A p95 that returns it has collapsed
// to the maximum, whatever the column header says.
const spiky = [30000, ...Array(19).fill(2000)];

assert.notEqual(p95(spiky), 30000);   // fails at any floor below 20
assert.equal(p95(spiky), 2000);

// And one short, nearest-rank does land on the maximum --
// which is exactly the figure that must never be published.
assert.equal(p95(spiky.slice(0, 19)), 30000);

What to check in your own dashboards

1. 01Find the sample count behind each percentile. If a p95 is computed over a five-minute window on a low-traffic endpoint, count how many requests that actually is. Under twenty, it is a maximum.
2. 02Look for percentiles that never move. A p95 pinned to a suspiciously round worst-case, unchanged for hours, is usually a maximum in disguise.
3. 03Check your test fixtures have spread. A percentile test built on identical values validates nothing about percentiles.
4. 04Gate the median and the tail separately. They need different amounts of evidence, and one threshold cannot serve both.
5. 05Prefer withholding to guessing. A dash is a smaller lie than a confident number derived from four data points.

The same discipline applies to every ratio a monitoring page prints. An availability percentage from three checks, an error rate from two errors, a model-substitution rate from a handful of judged responses — all of them look like measurements and all of them are anecdotes until enough draws exist to support the decimal places on display.

Where to go next

• When your probe becomes the outage — three ways a monitor reports its own behaviour as somebody else’s failure.
• What uptime means when your client retries — the other number on a status page that needs its definition stated before it can be compared.
• AI API relays, explained — what is being measured, and why relay latency has a tail worth reporting at all.

Our own latency column now shows a median from five timed responses and holds the tail until twenty, which is why the status page shows a dash in that second position for newly measured models. The dash is the honest answer.