A forecast only counts once it has been scored

The signal

This method entry, filed outside any single sector on Signal Atlas, is about the difference between a forecast that has been scored and one that has not. Good Judgment Inc, the company Philip Tetlock and Barbara Mellers founded after their team's performance in a US intelligence-community forecasting tournament, publishes its own track record page, reporting that its professional Superforecasters had been scored against 831 questions that had "resolved," meaning the outcome was known, as of 6 August 2026. Across those questions, the company reports its forecasters placed the highest probability on the eventual correct outcome on an average of 73% of the days each question was open.

The evidence

Scoring is the mechanism that separates this from an ordinary opinion. The standard tool is a Brier score, which averages the squared difference between a stated probability and the outcome, coded 1 if it happened and 0 if it did not, so a perfect forecaster scores 0 and a forecaster who always says even odds scores 0.25 regardless of what happens. Neither source below states Good Judgment's own Brier score; what they state are percentage comparisons the company reports for its tournament placings. The company's page states its forecasters "beat all competing research teams in the IARPA ACE tournament by 35-72%" and were "over 30% more accurate than intelligence analysts with access to classified information," and quotes a named IARPA programme manager, Steven Rieber, stating the team "beat the control group by more than 50%." The IARPA programme page for that tournament, listed under solicitation IARPA-BAA-10-05, confirms the goal was to "dramatically enhance the accuracy, precision, and timeliness of intelligence forecasts" through elicited and mathematically aggregated judgements, though the agency's own page does not itself publish the accuracy figures above.

Timeframe and confidence

These are vendor-reported figures about the vendor's own product, not an independent replication, and should be read as such. What is verifiable from the documents is the mechanism: a scoring rule exists, questions are tracked to a known resolution, and a stated share of questions and days is reported against that rule, which is a materially stronger basis than an unscored prediction even before any single number is taken at face value.

What would change the reading

An independent audit of the same resolved questions, or a published score applied by a party other than Good Judgment itself, would let a reader check the reported percentages rather than take the company's own summary of them.

• Was this forecast assigned a probability and a resolution date before the outcome was known?
• Who scored the outcome, and is that party the same one making the forecast?
• Against how many resolved questions is this track record measured, and over what period?

A forecast becomes evidence once it is scored against a known outcome by a stated rule. Until then, however fluent it sounds, it is an opinion, and that distinction is the entire subject of this entry.

Source trail

1. The Superforecasters' Track Recordgoodjudgment.com · Source publication: not established · Retrieved 2026-09-16

   Company-reported accuracy figures against the IARPA ACE tournament and intelligence analysts, and the 831-question, 73%-day scoring summary as of 6 August 2026.

2. See the future sooner with Superforecastinggoodjudgment.com · Source publication: not established · Retrieved 2026-09-16

   States Good Judgment's founding by Philip Tetlock and Barbara Mellers following the tournament and the company's self-description.

3. IARPA - ACEwww.iarpa.gov · Source publication: not established · Retrieved 2026-09-16

   Confirms the ACE programme's goal and solicitation number, independent of the vendor's own reported results.

Read across the evidence

• Most new businesses do not last a decade
• A hype cycle is a placement, not a measurement
• A leading indicator is graded only after the fact

Evidence & uncertainty · All 100 historical records