Two features, two failures
Prediction models for sports project game outcomes before they're played. Building one involves testing candidate ingredients — factors that might improve how accurately the model projects a final score.
The standard first test is a regression — a statistical tool that measures whether a candidate correlates with what actually happened. If teams with a larger talent advantage tend to win by more, the talent measure gets a positive coefficient. That's a signal the ingredient might be worth including.
We ran that test on two college football features. Both showed a positive correlation. We built both.
The first — a measure of how highly a program's recruits were rated coming out of high school — made things worse at every weight setting we tried. When we gave it no influence at all, the model's correlation with actual game margins was 0.6713. At the lowest nonzero weight, it fell to 0.6632. At a moderate setting, 0.6459. At the highest, 0.5833. Every step in the same direction.
The second — a measure of recent team performance trends — added 0.0048 on the season we used for tuning. On the following season, one the tuning had never seen, it moved the number from 0.6903 to 0.6902.
The pattern behind both failures is the same.
What we were testing
The two features came from different hypotheses about what the model might be missing.
The first was built around the idea that roster talent has effects that persist beyond any single season's results. Services that track college football recruiting grade each program's incoming class of high-school players and produce a composite score per team — a single number meant to capture how talented the roster is relative to the rest of the sport. High-profile programs at well-funded schools with storied histories tend to score highest.
The intuition is real. Early in a season, before many games have been played, that composite is one of the better signals available for how good a team might be. The regression reflected that: across the early weeks of a season, the talent gap between two teams accounted for about 1.3% of the variance in actual margins. In statistical terms, that fraction is called R² — it measures how much of the spread in real outcomes a feature can explain on its own. 1.3% is modest, but it's positive.
The second feature came from the idea that recent performance trends matter independently of season-long ratings. A team on a hot streak might be better than its full-season average has caught up to yet. The regression agreed: 1.6% of the variance in margins during the middle weeks of a season. Another real, positive result.
We built both.
What a regression measures — and what it doesn't
A regression measures whether a candidate ingredient moves together with outcomes. Add the feature to the existing inputs, fit the whole combination to real results, and see whether the model fits better with it than without it. A positive coefficient is evidence of correlation.
This is genuinely useful. A candidate that shows no correlation isn't worth building.
The problem is in the reverse direction. A positive correlation doesn't mean including the feature in the full model will help. Inside the model, the feature has to prove it's adding something the existing inputs don't already capture. The regression doesn't test that. It measures whether the feature moves with outcomes, not whether it moves with outcomes independently of everything else already in the fit.
When you throw a new ingredient into a regression alongside existing ones, the test hands the new ingredient its own coefficient without asking whether the existing inputs were already carrying the same information. That's not the question the model asks when it has to actually use the feature — and the gap between those two questions is where both of these features fell.
Three traps
Each failure had a different concrete cause, but all three fit a recognizable pattern.
The first is redundancy. The talent-based feature correlated with outcomes because more highly rated programs tend to win. But the model already knew that — it carried last season's actual on-field results, which are a more direct measure of team quality than recruiting grades. The regression handed the talent measure a coefficient without asking whether the model already had the same information. It did. When the talent measure had to compete with the results the model was already using, it had nothing to add.
The second is that the sample changes with the feature. Recruiting scores exist only for programs that ranking services track, which means every game against a lower-division opponent dropped out of the regression automatically. Those guarantee games — where a top program hosts a weaker opponent — are often the most lopsided matchups on the schedule. The test ran on a narrower, harder sample than the model runs on. A feature that looked helpful on that sample had an easier case to make than what it would actually face.
The third is that a pooled result from one season can look like a law. The momentum feature added 0.0048 on the year we tested it on. That gain was real. When we checked the following year — a year the tuning had never touched — the same feature gained 0.0001. What looked like a consistent signal was specific to one season. The second year was the honest measurement.
The right process
A regression is the right starting point. A feature that shows no correlation isn't worth the plumbing. The mistake is treating the initial test as the final one.
The sequence that works: run the regression to decide whether a candidate is worth building. Then build it as the model would actually use it, with the existing inputs as competition rather than collaborators. Then test it on a season the tuning never saw. The first season is the exam you studied for. The second is the one you didn't.
For both features, the held-out season said the same thing. One made things monotonically worse at every setting we tried. The other contributed nothing.
Both results are preserved in an automated check that runs any time the relevant parts of the system change. It carries the numbers from each test — the coefficient that made each feature look promising and the measurement that showed it wasn't — so the experiment can be read rather than re-run. The regression result that motivated building each feature is easy to rederive and persuasive enough that it would be rederived. Without the negative result attached, it would be rederived as a positive one.
What this doesn't say
It doesn't say recruiting talent is uninformative. The 1.3% is real. The issue is that the model already carries the same underlying signal through a more direct route — last season's results, which reflect the talent already on the roster, encoded in actual football. A measure that enters the fit through recruiting grades is trying to tell the model something it already knows from a better source. When those two compete for weight, the more direct one wins.
It doesn't say momentum never matters. We tested one specific construction of the idea across two seasons. A different definition — a different window, a different measure of recent form — might behave differently. Two data points both pointing the same direction aren't a law.
And it doesn't say the model is finished. We track it against a public benchmark where roughly 50 college football models are scored against every game, with the betting market's own spread included. Our model's average error currently sits below the median of free public models. A feature that actually closes that gap would have to survive a held-out season and add something the existing inputs don't — which is exactly what neither of these did. We're still looking.
Sources
Every number here comes from the graded ledger the boards publish. Research from a statistical model, not betting advice; no outcome is guaranteed. Method: how MatchWiz works.