Synthetic versus real users
A Socratic walk-through of synthetic versus real users — reasoned out one step at a time, not lectured.
The question we started with
THE QUESTION #Why does a test that runs perfectly every minute miss the failure your users are living through?
A scripted check logs in, adds an item to a basket, and completes a purchase. It has done this every sixty seconds for the last nine months and it has never failed. Meanwhile support has forty tickets saying checkout is broken.
The tempting reading is that the check is faulty. Usually it is not: run it by hand and it passes, exactly as reported. So we are looking at something stranger and more useful — a measurement that is entirely truthful about what it measured, and yet says almost nothing about what we wanted to know. Working out why is really a question about what a sample can stand for.
Reasoning it through
REASONING #Start by asking what the probe actually is. It is one journey, from one place, on one browser, at one moment, with one account. Every property of it is held fixed — that is the point of it, because a fixed configuration is what makes a stable baseline possible. Change nothing and any change in the result must come from the system.
Now ask what a real user is. A different device, on a different network, in a different country, with a different amount of data in their account, on a different browser version, possibly in a different experiment cohort behind a feature flag, arriving at a moment when the caches happen to be cold. The population of real sessions varies along dozens of dimensions at once.
Put those two descriptions side by side and the answer is already visible. The probe samples a single point from a space with dozens of dimensions — and the sample is not random, it is deliberately constant. So what class of failure can it never detect? Precisely the failures correlated with anything it holds fixed. A broken edge node in one region: invisible, because the probe runs from a different region. A JavaScript error on one browser version: invisible. A page that collapses when an account has ten thousand rows: invisible, because the test account has three. A third-party payment script that fails to load on slow mobile networks: invisible, because the probe runs on a fast datacentre link.
Then there is a subtler effect, and it is worth pausing on because it is genuinely perverse. A probe that runs every minute keeps its own path warm — its queries stay in the database cache, its assets stay at the CDN edge, its connections stay pooled. The probe is not merely failing to detect the slow path; it is actively preventing itself from taking it. The busier the probe, the less like a real first-time visitor it becomes.
So what is the complementary measurement? Instrument the real sessions instead: have the actual browser or app report what actually happened — how long the page took for that person, on that device, in that country, and whether it errored. That is real user monitoring, and by construction it samples the true distribution, including the long tail nobody thought to script.
But do not swing all the way over, because RUM has a matching blind spot that follows from its own definition. It can only report on traffic that exists. If nobody is buying at four in the morning, there is nothing to see; if a page is so broken it never loads its own reporting script, the beacon never arrives; if a change has not shipped yet, there are no real users to measure. Notice these are exactly the cases where synthetic monitoring is strongest: it generates traffic in the quiet hours, it runs against pre-release environments, and it reports its own failure to complete because the probe lives outside the page under test.
That is the actual relationship. One is a controlled experiment with an unrepresentative sample; the other is a representative sample with no control. Neither is a weaker version of the other, and the argument about which to buy is malformed — the interesting question is which failures each can see, and where the union still leaves a gap.
The analogy
THE ANALOGY #Think of a hospital that checks its equipment by running the same demonstration patient through it every hour: a mannequin of standard height, standard weight, no allergies, no prior surgery. When the readings drift, you learn something real about the machine, immediately and unambiguously.
You will not learn that the cuff does not fit larger arms, or that the software crashes on a patient with two recorded surnames. Those failures are properties of the population, and the mannequin was chosen precisely for having none of its variation.
a hospital could in principle build a more representative set of mannequins, whereas the variation across real user sessions is unbounded and changes weekly as browsers, networks and account histories evolve — so the synthetic case can be broadened but never made representative, which is why it is a complement rather than a stepping stone.
Clarifying the model
THE MODEL #Three refinements.
The first corrects the usual misdiagnosis. When a probe is green during an outage the instinct is that the check is wrong and needs fixing. Frequently the check is doing exactly its job. What was wrong was the inference — treating "this one journey works" as "the journey works", which is a claim about a population that a sample of one cannot support.
The second is about what each measurement is really for. Synthetic monitoring is best understood as an availability and regression signal: a stable, comparable time series that will move if the system changes, and a way to know about a broken checkout before anyone tries to check out. Real user monitoring is best understood as an experience signal: a distribution you can slice by browser, region, device or cohort to find who is suffering. Asking synthetic monitoring for a performance verdict on your users, or RUM for a pre-release safety check, misuses both.
The third concerns distributions. RUM's value is destroyed by averaging. A mean page-load time can look excellent while a tenth of sessions are unusable, so the practice is to read percentiles and, better, to slice by dimension until the affected group is named — an average across users is a synthetic user of a different and equally misleading kind.
A picture of it
THE PICTURE #How to readThe vertical axis asks whether a failure can be caught without waiting for a real user to hit it; the horizontal asks whether catching it needs the diversity of real sessions. The top-left cluster — the 4am outage, the pre-release regression, the expiring certificate — is what a scripted probe is genuinely good for. The bottom-right cluster is what only real user data will show, because each of those failures is correlated with something the probe holds constant. The top-right quadrant is empty by construction: no instrument sees the real spread before real users arrive. "Cold cache path" sits low and central because the probe's own constant traffic is what hides it.
What became clearer
WHAT CLEARED #A perfectly reliable measurement can still be a poor sample. The scripted check is not lying; it is answering a narrower question than the one being asked of it — and the failures your users are living through are, almost by definition, the ones correlated with everything the probe was designed to hold constant.
Where to go next
ONWARD #- Percentiles versus averages, and why a good mean can hide a broken tenth of sessions.
- Instrumenting the client so that a page which fails to load can still report having failed.
- Multi-location and multi-browser synthetic checks: how much representativeness can be bought, and at what cost.
Key terms
TERMS #| Term | What it means |
|---|---|
| Synthetic monitoring | scripted transactions run on a schedule from controlled locations, giving a stable baseline independent of real traffic. |
| Real user monitoring (RUM) | telemetry reported from actual user sessions, capturing the true distribution of devices, networks and data. |
| Long tail | the many rare session configurations that collectively account for a substantial share of users but individually appear in no test plan. |
Every term the collection defines is gathered in the glossary.