VIII · Validation
Station VIII of The Quant Map: is it real, and does it survive costs? Permutation tests, CSCV, walk-forward. Reading order plus the free map PDF.
is it real, and does it survive costs?, most ideas die here, and that is the point
Most ideas die here, and the archive is the output. A single Sharpe is a sample statistic with a wide interval: a Sharpe near 1 on a few years of daily data is often consistent with a true Sharpe near 0. The best of forty-one thresholds is a maximum. At a 5% bar the family-wise false-positive rate on that scan is about 88% before you correct it. Add the spread, the fees, and an honest slippage before you like the curve. A printed machine-learning paper can fail on its own numbers: a do-nothing baseline, a base rate, a cost line that was never subtracted.
Most ideas die here, and that is the point: the archive of rejected rules is the asset. The station's output is not a curve, it is a verdict with a trial count attached.
The Quant Map PDF works this spine through example research: five full descents from observation to live rule, and a library of 835 strategy entries, each with a named payer and a named failure mode. Get the free PDF on the Start Here page.
Reading order
Top to bottom. Stop at the first article you cannot apply to a live trade; that is the one to read slowly.
| Article | Why it is on this row |
|---|---|
| Why One Backtest Tells You Almost Nothing | Report the interval. A 12% annualized mean on a noisy daily sample can have a 95% band that includes a loss. |
| Backtesting Is an Experiment, Not a Screenshot | Pre-commit, detrend, permute, correct for the number of tries. One equity curve is one draw. |
| CSCV: A Direct Probability of Backtest Overfit | The probability that the in-sample winner is a loser out of sample, computed from the search you ran. |
| Permutation Tests for Indicator Significance | Shuffle the target, rerun the whole scan, and compare your best statistic to the best of each shuffle. |
| How to Test Indicator Thresholds Without Fooling Yourself | The RSI threshold that prints p = 0.016 in one scan can print about 0.3 once the scan is inside the test. |
| Why Walk-Forward Testing Is Better Than One Big OOS Split | Fit, grade the unseen window, roll. One frozen split is one anecdote about one date. |
| The Backtest Integrity Checklist | Point-in-time data, costs, trial count, regime split, a kill rule. Run it before capital. |
| The Hill, the Spike, and the Cliff: Reading Optimization Surfaces | A plateau you can move 20% and keep. A needle you found because you looked. |
| How to Spot a Fake ML Trading Paper | Three published accuracy numbers, and the baselines those papers did not print. |
| LLM Sentiment Trading: A Sharpe of Three That Needs an Autopsy | Mean 0.55% over volatility 2.49%, times the square root of 252, is about 3.5. The printed label says 3.05, and the split shares the label's window. |
On the map: ← VII · Hypothesis · The grand map · IX · Forecast / signal →