Pillar
The anti-overfitting lab. Stationarity, regime coverage, walk-forward, CSCV, Monte Carlo, permutation tests, parameter stability, degrees of freedom, and transaction-cost realism.
Every trading rule assumes stationarity. Markets violate it constantly. Strategies do not break because the rule fails; they break because the regime moves. Rolling stats first, then ADF and KPSS.
Slow drift is harder to detect than abrupt breaks because standard tests have weak power against it. CUSUM and long-vs-short divergence catch what ADF misses. Trust the alarm when it triggers.
Every strategy has a finite life. Four decay mechanisms: crowding, regime drift, microstructure, capacity. Size against half-life. Run a shadow book. Write the decommission policy at deployment.
A trading system never dies without warning. Eight diagnostic metrics flash months before the equity curve. Read the metrics. Hard kill switches written at deployment, automatic at threshold.
Most OOS failures get blamed on overfitting. Often it is regime change between IS and OOS. Run the diagnostic flow: permutation, pipeline audit, sample size, regime overlap test, in order.
Lifetime Sharpe is regime-weighted, not favorable-year. Each strategy has a vol regime where it lives and one where it dies. Gate at deployment, calibrate against the full distribution.
Returns are a near-random walk daily. Volatility has long memory: ACF +0.27, variance ratio 1.8 at 252 days. Vol is structurally easier to predict than direction. Build feature hierarchies around it.
Raw indicators have non-stationary distributions that break threshold rules across decades. Match the violation to the transformation. Eight classes, eight recipes. Verify causally. Do not stack.
Stationarization destroys signal when the strategy lives in the non-stationary part. 12-1 momentum raw Sharpe 0.55, time-series z-scored 0.18. Find the signal axis, transform only the orthogonal one.
The rolling window is a hyperparameter. Picking W by IS Sharpe inflates the estimate. Pick from a structural prior matched to the strategy timescale. Audit for look-ahead. Do not retune on live P&L.
A backtest is informative about regimes it covers, silent about the rest. Stratify by vol, trend, correlation, macro, microstructure. Report cell Sharpe with CI. Ship when every required cell passes.
Optimization climbs the sharpest peak. Robustness picks the widest plateau. The IS-optimal set is rarely robust. The robust set is rarely IS-optimal. OOS is the true distribution. Pick the plateau.
"Works on all markets" usually hides per-market tuning, cherry-picking, or long-bias. Real edges are family-specific. Demand the full matrix: fixed parameters, after costs, multi-test corrected.
Markets have personalities: efficiency ratio, autocorrelation, skew, vol regime, macro driver. Six families, six framework fits. SPX is not gold, gold is not BTC. Match framework to personality.
Right order: hypothesis, prototype at structural priors, parameter-stability map, then small validated optimization. Optimization first is search-width bias dressed as rigor. Test first, then tune.
Eight DoF categories, with numeric parameters as one of them. A "simple" strategy often hides thousands or millions of configurations. Count them all, apply the bias correction.
More parameters raise IS Sharpe and pitch quality. More parameters lower the bias-corrected Sharpe and OOS survival. The curves diverge after a small count. The mandate-winner is not the survivor.
The 10% rule: parameters / effective trades stays below 10%. Below 5% preferred. Above 10% is exploratory. Bias-to-noise scales with sqrt(p N). Count all categories.
The standard error of a Sharpe estimate is roughly 1/sqrt(N). At SR=1.0, 100 trades give CI +/-0.24, 500 give +/-0.11. Effective N corrects for hold-time, cross-section, gating. Report the bracket.
The 30-trade CLT threshold is about the sampling distribution of the mean, not deployment readiness. SE is 0.27 R for typical sigma. CI is wide. Each strategy property needs 200-1000 trades, not 30.
Monte Carlo has two flavors: bootstrap of trades (IS-distribution) and synthetic paths (model-conditional). Set kill-switches at bootstrap 99th percentile, not IS maximum. Not a substitute for OOS.
Permutation tests build the null by reordering the indicator while keeping returns fixed. No normality or i.i.d. assumed. Use block permutation for autocorrelated indicators. Pair with effect size.
PBO is the probability the IS-optimal parameter ranks below median OOS. CSCV enumerates all S-of-2S splits. Under 0.10 is robust; over 0.50 is pure overfit. Read the histogram, not the number.
Walk-forward chains many IS-OOS pairs, producing many OOS realizations and a temporal trend. A single split is one realization. Read the per-step distribution and trend, not the headline alone.
The IS-optimum is the peak. The plateau is a wide region of similar Sharpes. Pick the plateau center. Peak inherits search-width bias; plateau does not. Lower IS, higher OOS. Pick the plateau.
Optimization surfaces are hills, spikes, or cliffs. Hill: pick the center, deploy. Spike: reject. Cliff: pick away from the boundary. Same IS Sharpe; very different OOS. Read the shape.
A stop loss truncates losses (lowers drawdown) and cuts winners that briefly moved against entry (lowers expectancy). Mean reversion: second effect dominates. Trend following: first does. Test it.
MAE is the largest unrealized loss; MFE the largest unrealized gain. Plots reveal what aggregates hide: bounceback rates, profit capture, exit timing. Compute before adding stops or profit targets.
Profit factor is dominated by extreme trades. Two strategies at PF 2.0 can have very different OOS prospects. Report trimmed PF and Gini of trade P&L alongside the headline. Don't optimize on PF.
A strategy is more than net profit. The evaluation panel has 25 metrics across 7 categories. The panel reveals the failure axis. Headline numbers project away the operational info. Report all 25.
Apply realistic costs from Stage 2, not at the end. HFT strategies are cost-dominated; low-frequency edge-dominated. Teams that fall in love with pre-cost numbers resist the post-cost reality.
The deployment-readiness artifact: 32 items in 7 categories. Each box backed by analysis, reviewed by a second pair of eyes. Failures block or constrain.
Hold a position while an indicator stays above a threshold and there is no natural "trade." Group bars into holds and profit factor can hit infinity. Count bars with log returns and it stays honest.
A threshold optimizer always loves the tightest setting, because a few lucky bars beat thousands of honest ones in-sample. Floor the trade frequency, then permute the returns to prove the edge isn't the search.
Trend and reversion are one process split by the sign of beta. Regress a series on its own lag and OLS understates beta's magnitude in both cases, because the lagged regressor shares error terms with the target. You end up believing in less trend and less reversion than the market carries.
A NATGAS daytrader legend claims a guaranteed 20-25h sell-wave cycle, 44 trades a month. Run it through the gauntlet: cause test, multiple-comparisons, permutation null, and cost arithmetic. It is folklore until proven otherwise.
Sweeping 50/200, 60/210, 70/220 holds the ratio fixed: you test one concept along a line, not the space. That flat sweep is a thin ridge, not a plateau. Fix one parameter, vary the other, then reverse.