Every accuracy number has a truth definition underneath it, and the definition does more work than the number. Loosen it and everyone looks better. Tighten it and everyone looks worse. So the only figure worth reading is one that arrives with its standard attached.

Ours is called corroborated ground truth, and this page is the whole of it: what qualifies, what gets thrown out, what that choice costs us in every published score, and what it biases. The rule was written down before the measurement ran.

The ladder of evidence

Hail leaves three kinds of trace, and they are not equally trustworthy. Radar sees a signature and infers ice. An instrument estimates a size. A person stands in it, picks up a stone, and writes down what they measured. Only the third one is somebody's direct observation, which is why it sits at the top of every truth definition we use.

But one report is a single point of failure. Sizes get eyeballed, times get rounded, a stone gets described generously. The fix is not a better report — it is a second, independent one. Two people, reporting separately, clustered by the rule below into one storm event.

3,542 DEDUPLICATED REPORTS
1,441
distinct storm events reconstructed from the report record
THE COHORT WE GRADE ON
599
of them carry two or more independent observers — the corroborated set
THE LARGER GROUP
842
rest on a single report and are not treated as confirmation
Most storm events in the record rest on one observer. Requiring a second is a real filter, not a formality — it removes the larger group.

The rule, exactly

Here is the censoring rule as it is written in the campaign artifact, verbatim and unedited — the same string the measurement code reads:

FULL CENSORING. An evaluation row is removed from the population entirely — neither hit, nor miss, nor false alarm, for every system identically — if and only if it CARRIES GROUND TRUTH and that truth cannot be attributed to a corroborated storm event. A corroborated storm event is a connected component of deduped ground reports holding >= 2 member reports, computed by the same exact union-find over the same two edge relations as the storm-event POD (share a matched track_id; or within 25 km AND 60 min), over reports associated to storm objects by the same 15 km / +/- 10 min truth predicate. Rows carrying no truth are NEVER censored. A truth row whose storm object has no track_id is censored, because no event attribution is possible for it.

Two properties of that rule matter more than its wording. The first is that it is pre-registered: fixed before the numbers were computed, so it cannot have been shaped around a result. The second is that it is blind:

the rule reads only truth, report and track attributes and never any system's score, bar or fire flag, so the removed row set is identical for Anvil, ProbSevere v3 and MESH by construction

A truth rule that could read a model's score could be tuned to remove inconvenient storms. This one reads only the reports, the storm, and the track — so whatever it removes, it removes from all three systems identically, by construction rather than by good intentions.

One more property, and it is the one people assume backwards: a quiet storm is never evidence. Rows carrying no ground truth at all are never censored, and no missing report is ever scored as proof that no hail fell. Hail that lands where nobody is standing leaves no record — that is a limit of the instrument, and we treat it as one.

What it costs to use

Grading on this cohort is expensive, and the expense falls on us. Better than a third of the truth tracks in the clustering are removed from the population entirely:

2,586 TRACKS REMOVED
36.9%
of truth tracks are censored out of the population — for every system alike
WHAT WE GRADE AGAINST
4,430
corroborated truth tracks remain — the population every comparison is scored on
154,094 OF 1,744,182 TRACKS
8.8%
of all storm tracks in the campaign sit inside the corroborated set at all
The censoring rule is applied identically to Anvil, ProbSevere v3 and MESH before any of them is scored.

And here is the part that is easy to get wrong, stated in the artifact's own words:

censoring removes true positives from the fire population without removing any false alarm, so corroborated precision and CSI are mechanically lower, and FP-per-TP mechanically higher, than the all-truth scope for EVERY system before any skill difference is involved. Only the between-system comparison is interpretable; corroborated precision is never a product precision, because a user still receives the alarms on censored storms.

Read that carefully, because it cuts against us: censoring takes away confirmed catches and takes away no false alarms, so every system's absolute scores in this scope sit lower than they would elsewhere — ours included. A corroborated precision is not a product precision, and we never publish it as one. What the scope is for is the comparison between systems on truth nobody can wave away.

What survives it

Held to that standard, at track grain — one radar object's life — the balanced measure is the threat score, which charges misses and false alarms to the same account. Across 1,800,879 storm tracks and 4,430 corroborated truth tracks, the ordering holds:

Anvil leads both baselines on the strictest cohort Higher is better
Threat score (critical success index) on corroborated truth — misses and false alarms charged to one account, so no system can lift it by alarming more often. All three absolutes sit low because the scope removes confirmed catches from every one of them; the comparison is what the scope exists to make readable.

At the grain a person actually lives at — a storm, over a place, during an hour — the same standard produces the figure we lead with elsewhere: 93.6% of storms in the published cohort were confirmed while the hail was falling , 528 of 564, each one witnessed by at least 2 independent observers on the ground. Storms with no radar object at all are counted among the misses rather than excused.

Those two figures come from two different report pools and are two different populations. They are never averaged, never quoted in one breath, and each travels with the cohort it was measured over — which is the whole discipline this page describes, applied to itself.

What it biases

A cleaner instrument is not a neutral one, and the same artifact says so:

requiring two independent reports biases toward populated areas (more spotters per storm) and toward larger, longer-lived storms (more chances to be seen). Cleaner truth, less representative sample; a companion to the all-truth scope, never a replacement.

Two observers are easier to find where there are more people, and easier to accumulate under a storm that is bigger and lasts longer. So this cohort describes well-witnessed hail — not all hail. It is a companion to the unfiltered scope, never a replacement for it, and every figure we publish from it carries that clause wherever it goes.

That is the trade this standard makes, deliberately: a smaller, harder, less representative sample in exchange for truth that somebody stood under and wrote down. We would rather grade ourselves against that than against a number nobody can check.

Notes and scope

  • Two populations, never one. The corroborated track scope is built from one report pool and the storm-event cohort from a stricter one, so a track-grain figure and a storm-grain figure on this page are two measurements, not two views of one. They are never combined.
  • Corroborated precision is not product precision. Censoring removes confirmed catches and no false alarms, so every system's absolute scores sit mechanically lower in this scope. A user still receives the alarms raised on censored storms.
  • The rule reads no system's score. It is pre-registered and blind by construction, so the removed set is identical for Anvil, ProbSevere v3 and MESH.
  • Absence of a report is not absence of hail. Rows with no ground truth are never censored, and no missing report is ever scored as a system being right about silence.
  • Named baselines. MESH and ProbSevere v3 belong to their respective operators and are named here as the operational baselines Anvil is measured against. No endorsement by their operators is implied.
  • Offline measurement. These figures are an offline re-scoring of anvil-detect-5 and the baselines across historical rows — the right way to obtain a sample this size, and not a measurement of live behaviour. Scope: Contiguous United States only.