When "we've said this for sixty years" stopped counting as evidence

The bug, in one sentence

Arbiter's reasoning engine treated repetition over time as corroboration of the present. A claim reaffirmed for decades came out more credible than the arithmetic justified, and the older and more often repeated the claim, the worse the error. Issue #1343 fixed it. This post shows what the fix changes, run through the real engine on a real history rather than a hypothetical.

A claim that was true, reaffirmed, and then reversed

Dietary cholesterol is a clean example because the reversal is a matter of public record, not a contested opinion.

In 1961 the American Heart Association recommended limiting dietary cholesterol to reduce heart disease risk. Over the next four decades that guidance was independently reaffirmed: the 1977 Senate "Dietary Goals for the United States," the 1985 NIH Consensus Conference on lowering blood cholesterol, and the 2001 National Cholesterol Education Program's ATP III guidelines all restated the same limit. Four institutional bodies, forty years apart, all in agreement.

Then in 2015 the Dietary Guidelines Advisory Committee's scientific report concluded that dietary cholesterol was "not a nutrient of concern for overconsumption" for most people, and the numeric cholesterol ceiling that had stood in U.S. dietary guidance since 1980 was dropped. A 2019 cohort review corroborated that finding for the general population, while noting explicitly that it did not extend to every subgroup studied.

We modeled this history as an Arbiter investigation: one claim from 1961 ("dietary cholesterol meaningfully raises heart disease risk"), corroborated three more times through 2001; one claim from 2015 that refutes it, corroborated once more in 2019. Two actors, two claims, four corroboration events, one predicate, one real disagreement. This is the kind of case the engine exists to adjudicate.

What the old engine said

We ran the investigation against Arbiter's engine as it stood before #1343 landed.

	1961 claim (reaffirmed 1977, 1985, 2001)	2015 claim (reaffirmed 2019)
Credibility category	moderate	strong
Belief	0.403	0.515
Expected probability	0.701	0.757

The forty-year-old claim, superseded by the field's own current guidance, still read as moderately credible, not far behind the position that holds today. Each reaffirmation had added weight to the old position as if it were a fresh, independent confirmation of the present, rather than a confirmation of what was believed at the time. The math could not tell "this has stood the test of time" from "this was old and got repeated."

What the fixed engine says

Same investigation, same inputs, run against the engine after the fix.

	1961 claim (reaffirmed 1977, 1985, 2001)	2015 claim (reaffirmed 2019)
Credibility category	weak	moderate
Belief	0.165	0.446
Expected probability	0.582	0.723

The 1961 claim's belief drops by 59%, from "moderate" to "weak." The 2015 claim drops too, since the fix makes the engine more cautious about aged evidence in general, but only by 13%, and it stays at "moderate." The gap between the two widens in the direction the historical record points: the position that was superseded loses far more ground than the position that superseded it.

Why the correction is asymmetric

The fix is not a dial turned down on every old claim by the same amount. Think of the bug as a subsidy. It paid out on two things: age and repetition. The more years a claim had behind it, and the more times it had been restated, the larger the payment. The 1961 claim collected on both counts for forty years. The 2015 claim, far younger and restated once, collected a fraction of that. Remove the subsidy and each claim loses what it was being paid, no more and no less. That is why the corrections are 59% and 13% rather than a flat haircut. The size of the correction tracks how much each claim had benefited from the bug, which is exactly what the theory predicts.

The underlying reasoning is short. Testimony from 1961 is evidence about what was true in 1961. The question that matters is what that testimony is still worth as evidence about today, and the honest answer decays. It decays probabilistically, not by an accountant's straight-line depreciation. A reaffirmation in 1977 is real evidence too, but it is evidence about what was true in 1977. It does not reach back and make the 1961 testimony fresh again. The old engine conflated the two: it let each restatement count as a new confirmation of the present. The fix treats every observation as an observation made at its own time, and lets each one's relevance decay independently from that point. The discount is worked out from first principles, not picked because it looked plausible.

One thing this example is not

This is a constructed illustration built to show the arithmetic doing something different, not a verdict on the current state of nutritional science. We deliberately included the 2019 corroboration's own hedge, that its finding doesn't extend to every subgroup, because overstating how settled a real scientific question is would be exactly the kind of overclaiming this fix is meant to prevent the engine from producing. The number that matters here is the 59%-versus-13% asymmetry between the two claims, not either claim's absolute score.

---

This post was written by a coding agent working under Paul Nathan's supervision. Both engine runs described above are real, made against the actual Arbiter codebase before and after the fix; nothing here is simulated or estimated by hand.