MODEL VALIDATION · 2018 AND 2022

Would MP-26 have got it right?

We replayed MP-26's probability engine on the last two midterms: the same per-race error, the same shared national error and the same 50,000-run simulation the live forecast uses, fed with each race's final expected margin from FiveThirtyEight's published forecasts. Then we compared it with what actually happened.

4/4chamber results inside the 80% interval940 races scored

Races called right96%85% in the closest races

Brier score (lower is better)0.032FiveThirtyEight's own odds: 0.030

Typical race miss±6.2points of margin, root mean square

National missR+1.91 / D+1.28average miss in 2018 / 2022: the error every race shares

CHAMBERS

Seat intervals against the result

MP-26 80% interval FiveThirtyEight 80% median actual

2018

HouseD majority 90% · actual 235 D seats

218235

Inside the 80% intervalFavourite won

SenateD majority 14% · actual 47 D seats

5147

Inside the 80% intervalFavourite won

2022

HouseD majority 13% · actual 213 D seats

218213

Inside the 80% intervalFavourite won

SenateD majority 51% · actual 51 D seats

5051

Inside the 80% intervalFavourite won

CALIBRATION

Do 30% races happen 30% of the time?

Each dot groups the competitive races (5–95%) by forecast. Dots below the diagonal on the left and above it on the right mean the engine was too cautious: favourites won more often than it said.

RACE BY RACE

Expected margin vs. result

Each dot is a contested race: the final expected margin (across) against the actual one (up). Gold rings mark races the favourite lost. 2018 2022

WHAT WE LEARNED

Findings and limits

Chambers: well calibrated. All four results fell inside MP-26's 80% interval, and the favourite won all four chambers, the 2022 Senate by a hair: Democrats were at 51%.
Races: on par, slightly cautious. Race-level scores are within a whisker of FiveThirtyEight's own odds on the same inputs. The engine's ±7.5 points of error is more than the day-before forecasts needed: the best-scoring error was ±4. We keep ±7.5 because the live forecast runs weeks before the election, when uncertainty is larger.
What this does not test. Vote-Scope, MP-26's benchmark, did not publish 2018 or 2022 forecasts, so the inputs here are FiveThirtyEight's. This validates the probability engine, not the benchmark the live model starts from. The model stays labelled experimental.

Log loss at ±7.5: 0.113; at ±4: 0.101. Inputs: FiveThirtyEight final 2018/2022 forecasts (classic), CC BY 4.0, via the Internet Archive. Outcomes: FiveThirtyEight election-results (CC BY 4.0).

How the model works →Sources and methodology →