I spent a summer building a speed camera to measure the street outside my house. That’s the previous post. This one is about the afternoon I sat down to report what it had found, and realised I had no idea whether any of it was true.
Not because I’d found a bug. Because I had roughly 108 days of continuous data, and I had been modifying the thing that produced it for most of those 108 days.
The number that looked great
Here’s what a naive query gives you: the 85th percentile speed on my street is 30.5mph, n = 24,375.
That is a wonderful-looking statistic. Twenty-four thousand observations. Three months of continuous coverage. It is more data than most municipal traffic studies collect, and it took one query.
It is also a blend of four different measuring instruments, and nothing about it says so.
Over that window I had: upgraded the board, run a resolution experiment, added a glitch gate that didn’t previously exist, tightened that gate later, started tracking two fields that hadn’t been recorded at the start, and changed the day/night exposure behaviour. Every one of those could move a speed measurement. None of them are visible in the output.
A pooled statistic across instrument versions is not one measurement with a large sample. It is several different measurements stacked on top of each other, presented with the authority of the biggest number in the pile. And the sample size — the very thing that makes it look rigorous — is what makes it dangerous, because n grows every time you leave the system running, whether or not the system stayed the same.
So before publishing anything, I had to reconstruct what the instrument had actually been doing, and when.
Reconstructing the epochs, and why git lied
The obvious way to date the changes is to read the commit history. I have a repo. The commits have dates. This took about ten minutes and produced a completely wrong timeline.
The tell was three commits whose messages say, in so many words, “sync from live Pi deployment.” Which means: the code on the device had been running ahead of the repository for extended stretches, and at some point I had copied the live file back into git and committed it. The commit date records when I tidied up, not when the behaviour changed.
How wrong does that get? One example. The database shows the glitch-flagging gate producing its first flag on June 5th. The commit that introduces that gate’s logic is dated August 3rd. The feature was live and altering the dataset for two months before the repository had any record of it existing.
If I’d built my epoch boundaries from commit dates, I’d have put a boundary in the wrong place by two months and then confidently sliced my data along it.
On a hand-deployed system, the repository is a lagging record of what ran, not a source of truth about it. The fix is to derive the timeline from the artifact the system actually produced — the data itself:
- Schema cutovers. A column that is uniformly null or zero before a date and populated after it marks the exact moment the code changed. Two of my four boundaries came from this and they’re the most reliable evidence in the whole exercise, because a database cannot be mistaken about when it started storing something.
- Flag onset. The first appearance of a new rejection reason dates the deployment of the rule that produces it, to the second.
- Step changes in a derived quantity. A distribution that shifts discontinuously overnight and stays shifted is a code change whether or not anything in git agrees.
Commit messages are a hypothesis. The data is the evidence. Where they disagreed, I trusted the data, and they disagreed more than I expected.
Two things I could not explain, and did not smooth over
Reconstructing this honestly meant ending up with loose ends. Both are worth stating, because the temptation to quietly resolve them was strong.
The step change with no code behind it. On June 28th, the mean track length jumps by a factor of about 5.6 — overnight, and it stays there. There’s no commit that day. There is a later commit whose message refers back to “a frame-rate regression since 6/28,” which sounds like an explanation until you check the direction: a genuine drop in frame rate should produce fewer frames per vehicle, and the data shows many more. So the commit message names the right date and, as far as I can tell, the wrong mechanism. My best guess is a change in tracking stability — tracks persisting and re-matching for longer — but I can’t prove it, so it stays labelled as an unexplained boundary.
That’s two separate failures of the written record in one project: commits dated long after the behaviour shipped, and a commit message confidently describing the wrong cause. Both were written by me, in good faith, about my own system.
The hardware swap I can’t find in the data. The board was upgraded during this window. The system journal shows a burst of reboots on one particular day that is exactly what a physical swap looks like. But when I went looking for a corresponding fingerprint in the detection record — some discontinuity in track lengths, in volume, in anything — there wasn’t one. The days around it look like the days beside them.
So I have circumstantial evidence for the date and no measurable effect. The honest report is that the change happened, I believe I know roughly when, and I cannot demonstrate that it moved the numbers. That’s less satisfying than either “it mattered” or “it didn’t,” and it’s what the evidence supports.
Did any of it matter?
Four epochs, recomputed independently:
| Epoch | n | mean | P85 | share ≤15mph |
|---|---|---|---|---|
| E0 — baseline, no gates | 4,661 | 23.13 | 30.80 | 14.5% |
| E1 — track length + first gate | 4,686 | 23.91 | 30.40 | 10.3% |
| E2 — direction, day/night, resolution test | 7,165 | 24.03 | 31.00 | 10.3% |
| E3 — gate tightened, jump-ratio added | 7,871 | 23.03 | 29.80 | 11.9% |
No. It didn’t matter. The spread across all four is 1.2mph — 29.80 to 31.00 — through a resolution change, two schema cutovers, an unexplained regime shift, and a gate-tightening pass. The naive pooled figure of 30.5 sits almost exactly between the last two epochs. The number I was worried about was fine.
I want to be careful about what that means, because there’s a wrong lesson available here: “you worried for nothing, the big-n number was right all along.”
The check was still correct to run, and I’d run it again. “The instrument changed and the measurement held” and “the instrument changed and I didn’t look” produce the same number and are not the same claim. Only one of them is something you can publish. A robustness check that comes back clean hasn’t wasted your time — it has converted an assumption into a finding, and the whole reason the headline is quotable now is that I know it survives being sliced four ways.
There’s also a real pattern in the table worth one line of honesty: E3, the epoch with the strictest filtering, has the lowest P85 of the four. The plausible reading is that tighter gates are removing more high-speed tracking artifacts — not that the street calmed down. I can’t separate those two with this data, so I won’t claim either.
What the street actually does
Restricting to the current instrument only — one configuration, no changes, 32.7 days, n = 7,871:
- 85th percentile: 29.8mph. On a 15mph street. That’s 1.99× the limit.
- 88.1% of vehicles exceed the limit. Compliance is 11.9%.
- Median: 23.1mph. The typical car is going more than half again the limit.
- Fastest: 58.9mph.
The 85th percentile is the number to sit with, because of what it’s conventionally used for. In traffic engineering, speed limits are frequently set at the 85th percentile of observed free-flowing traffic — the principle being that most drivers select a reasonable speed for the road, and the limit should reflect that rather than fight it.
Apply that convention here and this street would be posted at 30mph. It is posted at 15. The gap between those two numbers is the entire finding.
The shape matters more than the maximum
The distribution is the part that changed how I think about it. I went in expecting the interesting story to be the outliers — the 58mph pass, the one lunatic. It isn’t. Outliers are mostly bugs, which is what the previous post was about.
The mass is at 20–30mph: those two buckets alone hold 53.4% of all traffic. This is not a street where compliant driving is interrupted by occasional maniacs. It’s a street where nearly everyone drives 20–30 and essentially nobody drives 15.
That distinction determines what you’d even do about it. A few bad actors is an enforcement problem — catch them. A whole population sitting at double the limit is a road design problem: the street is built to feel like a 30mph street, and drivers are responding to the road rather than the sign. Those two situations produce similar-looking averages and call for completely different responses, and you can’t tell them apart without looking at the shape.
What I still can’t tell you
This is a summer sample. The 32.7-day window is August into early September. School schedules, holidays, and daylight all shift traffic, and I have exactly one season of a stable instrument. I’d expect the number to move; I have no idea in which direction.
Directional asymmetry is real but unmapped. Travel one way across the frame runs about 1.7mph faster at the 85th percentile than the other, and the difference in the tail is larger — roughly 20% versus 14% of vehicles above 30mph. But the code records direction as pixel motion, left or right across the frame, with no mapping to real-world orientation. I know one direction is faster. I can’t currently tell you which physical direction that is, which makes it an observation rather than a finding.
Night is not comparable to day. Overnight the camera drops to a lower frame rate to buy exposure, and tracks come out roughly half as long — which pushes more real vehicles toward the short-track gate than daylight does. The 85th percentile happens to land identically day and night, but I wouldn’t quote a night-specific volume figure. Less traffic at 3am is obviously real; how much less is entangled with the camera getting worse at seeing.
And I cannot audit the data I kept. The per-frame position history needed to compute the jump-ratio quality metric is only stored for tracks that were already flagged as suspect — a deliberate decision to limit disk usage. Of 24,375 valid detections, eleven have it.
Sit with that one, because I think it’s the sharpest thing this project taught me. I can characterise the health of everything I threw away in complete detail. I can say almost nothing about the health of everything I kept. The instrumentation was pointed at the rejects, which is exactly backwards from what you’d want, and it was backwards for a completely reasonable engineering reason that nobody would flag in review.
Store your quality metric for the population you’re going to publish, not just for the population you’re going to discard. Otherwise the only thing you can prove is that your filter caught what your filter caught.
The rule I’d carry to anything else
A system you are actively developing produces a dataset with versions in it, and the versions are invisible in the output.
Write down every change that could touch a measurement — hardware, calibration, filter logic, schema, exposure. Derive those boundaries from what the system produced, not from what your commit log claims, because a hand-deployed system will run ahead of its repository and the commit dates will quietly lie to you. Then check whether your headline survives being sliced along them.
Mine did, which I only get to say because I checked.