9.38 Training an LLM to Forecast: Outcome-Based RL and the Calibration Win

Here is a headline that should make you reach for the small print. A 14-billion-parameter open model, one you can run on a single node, was trained with reinforcement learning on old prediction-market questions and came out matching OpenAI's o1 on accuracy, beating it on calibration, and turning a profit in a Polymarket trading simulation. The paper is Turtel and coauthors' "Outcome-based Reinforcement Learning to Predict the Future," and the numbers are real: a seven-run ReMax ensemble hit a Brier score of 0.190 against o1's 0.202, with an expected calibration error of 0.062 against o1's 0.093, and its simulated bets earned $52 against o1's $39.

Read past the headline and the story sharpens into something more useful than "small model beats big model." The market itself still won. Polymarket's own prices scored a Brier of 0.151, cleaner than every model in the study by a wide margin. So the real result is not that a language model out-forecasts a liquid market. It is that outcome-only RL can drag a model's probabilities onto the diagonal, and that the leftover edge, wherever it exists, sits exactly where the market is least sure. That is a calibration story dressed up as an alpha story, and calibration is the part worth stealing.

The reward is a bet you grade yourself on

The whole training loop rests on one choice of reward. The model reads a yes/no question plus news headlines dated before the forecast, emits a single probability p-hat that the event happens, and later observes the binary outcome y. The reward is the negative Brier score:

$$ R = -(\hat{p} - y)^2 $$

Read it plainly. If the event happens, y is 1, and your reward is minus the squared distance from 1. If it does not, y is 0, and you are penalized by the squared distance from 0. Say the model forecasts 0.7. If the event happens, the reward is minus (0.7 minus 1) squared, which is minus 0.09. If it does not, the reward is minus (0.7 minus 0) squared, which is minus 0.49. Confidence in the wrong direction is punished far harder than the same confidence in the right one, and that asymmetry is the entire point.

Brier is a strictly proper scoring rule, which means the way to maximize your expected reward is to report your true probability, not to shade it. The old article "What a Market Price Actually Is: Capital-Weighted Consensus and the Brier Skill Score Gate" leaned on the same object to decide whether a price carries skill at all. Here it becomes the training signal itself: no human labels, no preference pairs, just the outcome and a squared error. Two guardrail variants matter later. During training a malformed or missing probability gets the maximum loss of 1.0 (reward minus 1); during evaluation the "soft Brier" assigns a gentler 0.25 to a failed parse, the same score you would get by guessing 50%, so a formatting slip does not masquerade as a catastrophic forecast.

The training signal that quietly breaks

The obvious way to optimize this reward is GRPO, the group-relative method behind DeepSeek's math models. For each question you sample a group of G answers, score each one, and turn the scores into advantages by subtracting the group mean and dividing by the group standard deviation:

$$ \mu = \frac{1}{G}\sum_{i=1}^{G} r^i, \qquad \sigma = \sqrt{\frac{1}{G}\sum_{i=1}^{G}\left(r^i - \mu\right)^2}, \qquad \hat{A}^i = \frac{r^i - \mu}{\sigma} $$

The mean subtraction is harmless. The division by sigma is the landmine. It rescales every question's spread of rewards to unit variance, which means the size of the gradient stops tracking how much Brier loss is actually at stake and starts tracking only the ranking of answers within that one group.

Work two questions through it. Question A draws two rollouts scoring minus 0.04 and minus 0.64, a genuine gap between a sharp answer and a badly wrong one. Its mean is minus 0.34, its sigma is 0.30, so the normalized advantages are plus and minus 1.0. Question B draws two rollouts scoring minus 0.25 and minus 0.26, a difference of one hundredth of nothing. Its sigma is 0.005, so the normalized advantages are also plus and minus 1.0. GRPO just told the model that Question B's meaningless 0.01 difference deserves exactly the same push as Question A's meaningful 0.60 one. Multiply that across 100,000 noisy questions and the model learns to chase noise, and the cheapest way to make the reward look decisive is to get loud: push probabilities toward 0 and 1.

The failure mode is not hypothetical. Raw GRPO drove 39.3% of predictions into the extreme 0-to-10% or 90-to-100% buckets, many landing exactly on 0 or 1. The model literally wrote "the probability is extremely low, almost zero" and printed 0.000 for an event that was merely unlikely, not impossible. A forecaster who says zero is a forecaster who has stopped forecasting.

The fix is to stop dividing

The correction is almost embarrassingly small. Modified-GRPO keeps the mean subtraction and deletes the sigma division; ReMax replaces the group standard deviation with a learned baseline:

$$ \hat{A}^i_{\text{Mod-GRPO}} = r^i - \mu \qquad\qquad \hat{A}^i_{\text{ReMax}} = r^i - b^i $$

Now the advantage keeps its units. A question where the model is confidently, expensively wrong produces a large negative advantage; a question where all the answers are mediocre produces a small one. The gradient stays proportional to the Brier loss, which is exactly what you want when the whole objective is calibration. Concretely, dropping the normalization cut the extreme-bucket share from 39.3% under raw GRPO to 7.9% under Modified-GRPO. The Brier bounded-ness in [0, 1] does the rest: because rewards cannot blow up, removing the variance scaling does not destabilize training the way it would with an unbounded loss.

Removing one normalization opened a different hole. Modified-GRPO without guardrails started emitting gibberish and switching mid-sentence into Chinese, because once the reward gradient cannot cleanly separate a valid forecast from an invalid one, the policy wanders off the manifold of coherent text. The fix was a scorer that docks reward for non-English passages, random-character strings, and missing rationales inside the think block, plus a hard truncation of any input past 16,000 characters and schema validation that zeroes the reward on any malformed output. With guardrails on the 100k corpus the extreme-bucket share settled at 13.1%, and the near-zero forecasts moved off the floor. This is DPO's neighborhood too: the paper keeps Direct Preference Optimization as an off-policy baseline, and it lands mid-pack, better than raw GRPO but behind the two on-policy fixes.

The part most "my AI predicts the future" claims get wrong

Before any of the accuracy numbers mean anything, ask one question: could the model have peeked? Temporal leakage is the disease of forecasting backtests. If the training data or the prompt contains any whiff of information dated on or after the forecast date, the model is not predicting, it is remembering, and the whole evaluation is fiction. This is the single most important paragraph in the paper and the one the headline never mentions.

The authors build the test set to make peeking hard. Every test question's forecast date falls after the latest resolution date of any training question, so the model cannot have trained on a resolved version of anything it is later asked. Questions with open-ended resolution dates, the "will X happen sometime before Y" kind that leak the future through survivorship, are dropped. And they run OpenAI o3 over the retrieved news to flag any story that quietly contains post-forecast information, then delete those questions. Even the news retrieval is date-gated at day-level granularity so the prompt cannot cite an article from after the cutoff. This is the discipline Paleka and coauthors' "Pitfalls in Evaluating Language Model Forecasters" spends a whole paper insisting on, and it is why most breathless forecasting demos evaporate under scrutiny. Trust the accuracy numbers only because the leakage controls came first.

What the training actually bought: calibration

Line the algorithms up and the pattern is clean. Accuracy barely moves; calibration moves a lot.

On the left, Brier scores cluster in a narrow band from about 0.19 to 0.22. The base DeepSeek-R1 14B sits at 0.215, o1 at 0.202, and the ReMax ensemble at 0.190. That is a real but modest accuracy gain, and the paired test against o1 is only marginally significant. On the right, expected calibration error tells the louder story: the base model at 0.089 and o1 at 0.093 both roughly halve to ReMax's 0.062 and Modified-GRPO's 0.054. The training did not mostly make the model smarter about which way events break. It made the model honest about how sure it is.

Expected calibration error is the number doing the work. Bin the forecasts into ten equal-mass buckets, and for each bucket compare the average confidence to the fraction of events that actually happened:

$$ \text{ECE} = \sum_{b=1}^{B} \frac{n_b}{N}\,\bigl|\,\text{acc}_b - \text{conf}_b\,\bigr| $$

Each term is a bin's share of the forecasts times the gap between what the model claimed and what reality delivered. Suppose the 0.8-to-0.9 bin holds 120 of 1,265 forecasts with an average confidence of 0.85, but only 68% of those events happened. That bin contributes (120 divided by 1,265) times the absolute value of 0.85 minus 0.68, which is 0.095 times 0.17, about 0.016. Sum the contributions across all ten bins and you get the ECE. An overconfident model piles up gaps in the extreme bins and its ECE balloons; a calibrated model keeps every bin's claim close to its outcome.

That is exactly the diagram the old article "A Reliability Diagram From Your Last 50 Trades" told you to build from your own trade history. The diagonal is perfect calibration: events you call 80% should happen 80% of the time. The overconfident curve, the raw-GRPO shape, sags below the line where it is bullish and rides above it where it is bearish, because it says 85% when reality is closer to 68%. The training win is geometric: it drags the curve back onto the diagonal. This is the same lesson as the old article "Bayesian Edge in Log-Odds: Better, Earlier, Calibrated, or It's Noise," which insisted that an edge you cannot calibrate is just noise wearing a confidence interval.

Did it actually make money?

In simulation, yes, and the shape of the profit is more honest than the total. Convert each forecast into a one-share Polymarket trade: if the model's probability beats the market price, buy the yes share at market plus a penny for fees; if it is below, short it; the extra cent stands in for slippage. Rank all 1,265 trades by expected edge and walk down the list. The ReMax ensemble earned $52 on $433 of cost and Modified-GRPO earned $54 on $428, both roughly a 10% return, against o1's $39. Every model turned a nominal profit even when forced to bet on all markets.

But look at where the edge lives.

The bars measure how much more often the model's bets won than the market price implied, split by how confident the market was. When the market was near certain, at the far ends, the model added nothing statistically distinguishable from zero, and the error bars swallow the bars whole. Essentially all the profit came from the 40-to-60% bucket, the questions where the market was closest to a coin flip. There the model's bets won 11.8 percentage points more often than the price predicted, worth about a 20% return on those questions alone. The paper even hands you the exit rule: truncate the strategy once the expected edge drops below your own ECE and you keep almost the entire upside while cutting the loss-making tail.

Stay skeptical about the money. This is a simulation on a small, low-liquidity venue with a one-cent fee stand-in, and the market's own Brier of 0.151 still beats the model's 0.190 outright. The model is not overturning the market; it is adding incremental information precisely where the market has the least conviction, and only a disciplined bettor who stops trading at the calibration threshold gets to keep it. The authors say as much: financial markets are thousands of times larger, and this is a tool to support human judgment, not replace the price.

Where this connects

This article is the machine-learning end of a thread the pillar has been pulling by hand. The old article "What a Market Price Actually Is: Capital-Weighted Consensus and the Brier Skill Score Gate" defined the price as a probability you grade with Brier; here Brier becomes the reward that trains a model to produce those probabilities. The old article "Bayesian Edge in Log-Odds: Better, Earlier, Calibrated, or It's Noise" argued that an uncalibrated edge is noise; this paper shows an RL pipeline whose entire measurable gain is calibration. And the old article "A Reliability Diagram From Your Last 50 Trades" handed you the audit tool; the reliability diagram here is that same tool pointed at a 14B model instead of your trade blotter.

The deeper lesson generalizes past language models. Any system optimized against a noisy proper score will drift toward overconfidence if the training signal rescales away the magnitude of its errors. The cure, keeping the gradient proportional to the actual loss and refusing to reward degenerate answers, is a discipline every model-builder in this book needs, whether the model is a transformer or a linear rule. Calibration is not a nicety you add at the end. It is the thing outcome-based RL is actually good at, and the only part of this result that survived contact with a real market.

KEY POINTS

• A 14B open model (DeepSeek-R1-Distill-Qwen-14B) trained with outcome-only RL matched o1 on accuracy (Brier 0.190 vs 0.202) and beat it on calibration (ECE 0.062 vs 0.093), but Polymarket's own prices still won outright at Brier 0.151.
• The reward is the negative Brier score, minus (forecast minus outcome) squared, a strictly proper rule: the only way to maximize it is to report your true probability.
• Standard GRPO divides advantages by each question's reward standard deviation, which decouples the gradient from the actual Brier loss and drove 39.3% of forecasts into the extreme 0-or-1 buckets. Dropping the division (Modified-GRPO) or using a learned baseline (ReMax) keeps the gradient proportional to the loss and cut extremes to 7.9%.
• Removing the normalization reintroduced gibberish and language-switching, fixed with guardrails that penalize non-English text, random strings, and missing rationales, plus schema validation that zeroes malformed outputs.
• The accuracy claim is only trustworthy because of aggressive anti-temporal-leakage controls: test dates after all training resolutions, no open-ended resolution questions, and an o3 pass to delete leaked news. Most forecasting demos skip this and are fiction.
• The training's measurable win is calibration, not smarts: Brier barely moved while ECE roughly halved. The reliability diagram straightens back onto the diagonal.
• In a Polymarket simulation the ReMax ensemble returned about 10% ($52 on $433), but essentially all the edge came from the 40-to-60% market-confidence bucket, worth about 20% there. Truncating at the edge-below-ECE threshold keeps the upside and drops the loss-making tail.

References

• Verification of Forecasts Expressed in Terms of Probability (Brier, 1950)
• DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models (Shao et al., 2024)
• ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Models (Li et al., 2023)
• Direct Preference Optimization: Your Language Model Is Secretly a Reward Model (Rafailov et al., 2023)
• DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning (Guo et al., 2025)
• Pitfalls in Evaluating Language Model Forecasters (Paleka et al., 2025)
• Outcome-based Reinforcement Learning to Predict the Future (Turtel, Franklin, Skotheim, Hewitt, and Schoenegger, 2025)

A note on AI. The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.