# Aligrithm > An independent research publication on market structure, dynamics, quantitative and systematic trading | By Ali H. Askar Public Ghost content for AI and LLM tooling. This file includes a bounded export of public pages first, then recent public posts. Append `.md` to any post or page URL to get the content in Markdown (for example, `/example-post.md`). ## Pages ### About aligrithm URL: https://aligrithm.com/about/ Last updated: 2026-05-20T10:15:19.000Z # Aligrithm is an independent research publication on systematic trading, quantitative research, market microstructure, and real-time adaptive systems. The work sits at the intersection of trading, AI, signal processing, statistics, physics, and systems engineering, with a strong empirical bias. Most trading content online sells predictions, indicators, and short-term excitement. Aligrithm covers what happens after the prediction layer. How real trading systems are researched, tested, engineered, and deployed under live market conditions. How edge survives the journey from notebook to live account, and how often it does not. How a rule that looks great in a backtest becomes a system that holds capital responsibly, then dies in the regime change that comes next, then gets replaced by the next one. The publication is built for traders, researchers, and engineers who would rather read about a permutation test than a "secret indicator." ## The eight pillars The content on this site is organized into eight content pillars. Each pillar has its own tag so articles can be filtered, indexed, and cross-referenced. ### [The Scientific Trader](https://aligrithm.com/tag/the-scientific-trader/) Evidence-based trading, hypothesis testing, benchmarks, falsifiability, and the scientific method applied to systematic trading. The philosophical and methodological foundation of everything else on the publication. The pillar that teaches the reader to stop asking "does this strategy make money" and start asking "what would prove this strategy wrong." ### [Indicator Engineering](https://aligrithm.com/tag/indicator-engineering/) Signal processing for traders. Entropy, stationarity, smoothing, transformations, frequency response, lag, tail behavior, Range/IQR, sigmoid transforms, band-pass filters, decyclers. Why a clean feature beats a complex model. ### [Robust Systems Lab](https://aligrithm.com/tag/robust-systems-lab/) Backtesting realism, walk-forward analysis, Monte Carlo, permutation tests, CSCV, parameter stability, regime coverage, transaction cost modeling, degrees of freedom, the backtest integrity checklist. The anti-overfitting laboratory. ### [Market Structure Notes](https://aligrithm.com/tag/market-structure-notes/) How real markets are organized. FX wholesale versus retail, intermarket relationships, execution mechanics, cross-asset signals, noise versus volatility, market personality, timeframe selection, currency strength decomposition. ### [Microstructure Alpha](https://aligrithm.com/tag/microstructure-alpha/) Short-horizon edge that depends on execution. Order book imbalance, microprice, markouts, maker/taker economics, fair value, adverse selection, spread design, skew, fill probability from trade-size CDFs. ### [Portfolio Construction & System Death](https://aligrithm.com/tag/portfolio-construction-and-system-death/) Position sizing, forecast combination, volatility targeting, instrument weighting, drawdown analysis, expectancy, profit factor, MAE/MFE, system decay, behavioral discipline, fat tails, random walk versus efficient markets. The survival layer. ### [Python Research Notebooks](https://aligrithm.com/tag/python-research-notebooks/) Reproducible code. Indicator quality reports, threshold testers, permutation tests, Monte Carlo drawdown simulators, optimization surface visualizers, OBI backtests, microprice empirical tests, portfolio simulators, system decay detectors. ### [Physics, Geometry & Event-Driven Markets](https://aligrithm.com/tag/physics-geometry-and-event-driven-markets/) The advanced research frontier. Intrinsic time, event-driven filters, market geometry, network causality, Ricci curvature, topological data analysis, optimal transport, Kuramoto synchronization, Tsallis entropy, Sornette log-periodic power laws. Markets as event-driven, nonlinear, geometric, information-processing systems. ## What you will not find on aligrithm Aligrithm avoids signal services, guru newsletters, secret-strategy vendors, paid trading rooms, chart-pattern alerts, and screenshots of called tops. The publication exists to make rigorous quantitative thinking accessible to readers who are willing to do the work. ## About the author [Ali H. Askar](https://www.linkedin.com/in/ali-h-askar/?ref=aligrithm.com) is a quantitative systems engineer and AI researcher. He builds production-grade trading systems across FX, crypto, equities, and derivatives markets. His work covers event-driven backtesting, market microstructure, execution systems, AI-driven signal research, and low-latency trading infrastructure, with the emphasis on robustness, realism, and live-market behavior over idealized backtests. Over the past decade he has developed systematic trading systems, execution engines, arbitrage infrastructure, and quantitative research frameworks designed to operate under real-world market constraints. The opinions on this site come from running those systems, watching them work, watching them decay, and rebuilding them. The writing tries to share that experience without selling anything. ## Subscribe Subscribing gets you full access to all articles, future research notes, and long-form technical content as it is published. No spam, unsubscribe anytime. The publication is for readers who want the work, not the noise. Subscribe Email sent! Check your inbox to complete your signup. No spam. Unsubscribe anytime. ### My Portfolio URL: https://aligrithm.com/my-portfolio/ Last updated: 2026-05-20T10:21:59.000Z *Research, trading systems, and low-latency market infrastructure.* I build quantitative trading systems that sit at the intersection of market microstructure, machine learning, statistical research, and high-performance execution infrastructure. The work covers the full lifecycle: signal research, backtesting, execution, risk, monitoring, and live deployment. The projects below cover graph-based alpha research, event-driven backtesting, low-latency execution, financial visualization, trade analytics, and multi-account trade management. DM me for more. [Linkedin ](https://www.linkedin.com/in/ali-h-askar/?ref=aligrithm.com) --- ## Featured projects ### Global Network Momentum Trading Strategy A production-grade implementation of network momentum, designed to capture momentum spillovers across global assets through graph-based learning. The system treats markets as a dynamic network where information, momentum, and structural relationships propagate across nodes, instead of treating assets as isolated return streams. The strategy builds adaptive asset networks using convex optimization, propagates signals across those networks with graph signal processing, and generates long/short portfolios via cross-sectional regression. The research pipeline includes walk-forward validation, hyperparameter optimization, parallelized computation, and strict no-lookahead controls. | Area | Implementation | | -------------------- | --------------------------------------------------- | | Network construction | Dynamic asset graphs from cross-asset relationships | | Signal propagation | Graph signal processing over financial networks | | Portfolio generation | Long/short cross-sectional regression | | Validation | Walk-forward testing with no-lookahead bias | | Scalability | Parallelized research and optimization pipeline | **Skills:** machine learning, statistical research, quantitative finance, Python. **Related pillars:** `event-driven`, `robust-systems`, `scientific-trader`, `python-notebooks`. --- ### PyCharting A high-performance financial charting library built for fast, interactive visualization of large-scale OHLC datasets. Quantitative researchers need to explore millions of price points, indicators, and trade signals without waiting on slow plotting workflows. PyCharting uses a FastAPI backend and a lightweight browser-based frontend to stream large financial datasets efficiently. It supports zooming, overlays, multi-panel indicators, trade signal visualization, and real-time interaction. Financial data exploration becomes instant and research-friendly. | Area | Implementation | | -------------- | ------------------------------------------------------- | | Backend | FastAPI data server | | Frontend | Browser-based lightweight charting | | Data support | Large-scale OHLC visualization | | Research tools | Overlays, indicators, signals, multi-panel charts | | Use case | Fast visual inspection of high-frequency financial data | **Skills:** Python, FastAPI, financial visualization, research tooling. **GitHub:** [alihaskar/pycharting](https://github.com/alihaskar/pycharting?ref=aligrithm.com) **Related pillars:** `python-notebooks`, `robust-systems`. --- ### Low-Latency Event-Driven Backtesting & Execution Engine A high-performance event-driven backtesting and execution engine built around point-in-time correctness, realistic simulation, and production-grade architecture. The engine eliminates lookahead bias by treating every market update, corporate action, order event, fill, and portfolio update as part of a deterministic event stream. It supports multi-asset strategies, corporate actions as real-time events, advanced order execution modeling, portfolio accounting, OMS/EMS components, and multi-timeframe strategy logic. The same architecture that tests a strategy supports its live deployment. | Area | Implementation | | ----------------- | ----------------------------------------------------- | | Backtesting model | Event-driven, point-in-time simulation | | Market data | Multi-asset support with corporate actions as events | | Execution | Advanced order and fill modeling | | Architecture | OMS, EMS, portfolio accounting, strategy layer | | Strategy support | Multi-timeframe logic and scalable research workflows | **Skills:** Python, high-frequency trading, backtesting, execution systems. **Related pillars:** `robust-systems`, `event-driven`, `market-structure`. --- ### HFT FX Arbitrage Engine A low-latency FX arbitrage system that detects and acts on cross-venue price dislocations across multiple liquidity sources. The architecture is built around hot-path optimization, shared memory, zero-copy data access, lock-free ring buffers, CPU core pinning, and C/Python bindings for deterministic performance. The system tracks latency across transport, buffer, and processing layers, measuring where every microsecond is spent. It targets high-throughput tick ingestion, real-time state management, and arbitrage detection across 16 venues. | Area | Implementation | | -------------- | -------------------------------------------------------- | | Venue coverage | 16 FX venues | | Data transport | Shared memory and memory-mapped access | | Processing | Lock-free ring buffers, hot-path optimization | | Latency | Sub-20µs tick-to-trade target with full latency tracking | | Throughput | Designed for 200M+ market events per day | **Skills:** Python, Rust, C/Python bindings, low-latency systems, FX microstructure. **Related pillars:** `microstructure-alpha`, `market-structure`. --- ### Trade Geometry Signal Analyzer A quantitative diagnostic engine for understanding what happens after a trading signal fires. Most trading research stops at the entry signal. This project starts where that research ends: how price moves after entry, how quickly it reaches favorable or adverse levels, and how different exit rules behave across regimes. The system analyzes maximum favorable excursion, maximum adverse excursion, time sequencing, TP/SL probabilities, volatility regimes, and trade archetypes. Exit design becomes data-driven rather than dependent on brute-force parameter optimization. | Area | Implementation | | -------------------- | ------------------------------------------------------- | | Trade path analytics | MFE, MAE, time-to-target, path sequencing | | Exit research | Probabilistic TP/SL outcome analysis | | Risk design | Risk/reward frontiers and adverse excursion diagnostics | | Regime analysis | Volatility-based behavior segmentation | | Clustering | Trade archetype discovery | **Skills:** quantitative research, Python, risk management, trade analytics. **GitHub:** [alihaskar/signal\_analyzer](https://github.com/alihaskar/signal%5Fanalyzer?ref=aligrithm.com) **Related pillars:** `portfolio-system-death`, `robust-systems`, `scientific-trader`. --- ### MT4/MT5 TradeManager A distributed trade replication and risk management system that synchronizes trades across multiple master and slave accounts in real time. The system combines WebSocket event ingestion, REST execution, Redis state management, PostgreSQL persistence, reconciliation logic, and real-time monitoring. Built for production-grade multi-account execution workflows where reliability matters more than cosmetic dashboards. It includes discrepancy detection, risk controls, fault-tolerant execution, account synchronization, and monitoring through Dash and Grafana. | Area | Implementation | | ---------------- | ------------------------------------------------------- | | Trade ingestion | WebSocket-based event flow | | Execution | REST-based order routing | | State management | Redis and PostgreSQL | | Reliability | Reconciliation, discrepancy correction, fault tolerance | | Monitoring | Dash GUI and Grafana dashboards | **Skills:** Python, Redis, PostgreSQL, WebSockets, MT4/MT5, trade replication. **Related pillars:** `market-structure`, `robust-systems`. --- ## What connects these projects All of these systems share the same focus. Build trading infrastructure that is fast, realistic, measurable, and production-ready. | Principle | Meaning | | -------------------------- | -------------------------------------------------------------------- | | Point-in-time correctness | No hidden lookahead bias, no unrealistic assumptions | | Execution realism | Model trading as it happens in live markets | | Latency awareness | Measure and optimize the full path from data to decision | | Research scalability | Large experiments stay repeatable and parallelizable | | Production discipline | Monitoring, reconciliation, and risk as core system components | | Market structure awareness | Build around how markets behave, not how they look in clean datasets | ## Areas of focus | Domain | Focus | | -------------------------- | ------------------------------------------------------------- | | Quantitative research | Alpha discovery, statistical validation, cross-asset modeling | | Market microstructure | Order flow, arbitrage, liquidity, execution behavior | | Machine learning | Feature engineering, graph learning, cross-sectional modeling | | Backtesting infrastructure | Event-driven simulation, PIT correctness, realistic execution | | Low-latency systems | Shared memory, lock-free design, hot-path optimization | | Trading operations | OMS/EMS, reconciliation, monitoring, risk controls | I build systems for the full lifecycle of quantitative trading: research, validation, execution, monitoring, and scaling. The goal is to build infrastructure where signals survive contact with real markets. --- ### Contact Us URL: https://aligrithm.com/contact/ Last updated: 2026-06-20T11:17:19.000Z Whatever that you have in your mind, please feel free to reach out: [Linkedin ](https://www.linkedin.com/in/ali-h-askar/?ref=aligrithm.com) ### I · Foundations URL: https://aligrithm.com/spine-i-foundations/ Last updated: 2026-10-16T09:34:18.000Z *the toolkit you think with, pedagogy, not alpha* A strategy that wins 55% of trades still hits eight losers in a row about 0.17% of the time, once or twice per thousand trades. That run is the strategy working. Risk 2% of a $5,000 account and the streak costs about $800\. Risk 15% because the setup looked clean and the same streak is the account. A second trap sits in the average: +60% then −40% advertises +10% a year, and $100 becomes $96\. Compounding pays the variance. The average does not. Nothing on this station earns money. It is the arithmetic that stops you from donating it: what a losing streak costs at your size, why the average of +60% and −40% is a loss, why an 87% win rate can still bleed. Pass through once before the first backtest and again before the first live dollar. The Quant Map PDF works this spine through example research: five full descents from observation to live rule, and a library of 835 strategy entries, each with a named payer and a named failure mode. [Get the free PDF on the Start Here page](https://aligrithm.com/start-here/). ## Reading order Top to bottom. Stop at the first article you cannot apply to a live trade; that is the one to read slowly. | Article | Why it is on this row | | -------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------- | | [Why Trading Is a Probability Business, Not a Certainty Business](https://aligrithm.com/why-trading-is-a-probability-business-not-a-certainty-business/) | One trade is one draw. The standard deviation of trade returns runs several times the edge, so the signal shows up through sample size. | | [Fat Tails: Why Gaussian Thinking Breaks Trading Systems](https://aligrithm.com/fat-tails-why-gaussian-thinking-breaks-trading-systems/) | The streak table gets worse once the left tail is fatter than a normal curve. Gaussian size is how the account dies on an "ordinary" day. | | [Get-Even-Itis: The Most Expensive Disease in Trading](https://aligrithm.com/get-even-itis-the-most-expensive-disease-in-trading/) | The streak is when you edit the size. The calm version of the rule has to be the one that is already written. | | [Average Drawdown vs Extreme Drawdown](https://aligrithm.com/average-drawdown-vs-extreme-drawdown/) | One worst loss is a single point. The average drawdown is the distribution you size to. | | [Why Percent Profitable Is Overrated](https://aligrithm.com/why-percent-profitable-is-overrated/) | An 87% win rate describes the pleasant trades. Expectancy includes the loss that gives the money back. | On the map: [The grand map](https://aligrithm.com/start-here/) · [II · Market architecture →](https://aligrithm.com/spine-ii-market-architecture/) ### II · Market architecture URL: https://aligrithm.com/spine-ii-market-architecture/ Last updated: 2026-10-16T09:34:19.000Z *where edges live: underlying markets × instruments, two different axes* The market is the thing whose price you care about. The wrapper is the contract you hold: spot, forward, future, perpetual, ETF, CFD. Four ways to be long bitcoin are one view and four P&Ls. Spot has custody. A quarterly future makes you pay or earn the basis every roll. A spot ETF sleeps while bitcoin does not. A perpetual never expires, and the crowded side pays funding, about 11% a year when the 8-hour print is 0.01%, and much more in a mania. In April 2020, spot oil recovered from the negative print. Holders of the rolled-futures wrapper did not get that recovery back. Being right about the market and wrong about the wrapper still loses the money. Architecture is two axes, markets times instruments, and most retail losses blamed on "the market" are the instrument: the roll, the funding, the swap on the retail wrapper. Learn the wrapper's cash flows before reading any signal on it. The Quant Map PDF works this spine through example research: five full descents from observation to live rule, and a library of 835 strategy entries, each with a named payer and a named failure mode. [Get the free PDF on the Start Here page](https://aligrithm.com/start-here/). ## Reading order Top to bottom. Stop at the first article you cannot apply to a live trade; that is the one to read slowly. | Article | Why it is on this row | | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------- | | [How FX PnL Actually Works](https://aligrithm.com/how-fx-pnl-actually-works/) | The pair, the account currency, and the roll are three different numbers. The backtest on a clean mid misses two of them. | | [Market Orders vs Limit Orders in FX](https://aligrithm.com/market-orders-vs-limit-orders-in-fx/) | The order type chooses who pays the spread. That choice belongs in the strategy. | | [Why Bid/Ask Bounce Matters for Intraday FX Systems](https://aligrithm.com/why-bid-ask-bounce-matters-for-intraday-fx-systems/) | On a short horizon the bounce is a large fraction of the move you think you forecast. | | [Currency Quoting Conventions: Numerator, Denominator, and the Priority Order](https://aligrithm.com/currency-quoting-conventions-numerator-denominator-and-the-priority-order/) | Which currency is the price, and which is the thing priced, changes the sign of every feature you build. | | [Cross-Exchange Fair Value for Crypto Perps](https://aligrithm.com/cross-exchange-fair-value-for-crypto-perps/) | The same coin on two wrappers. The gap is a basis, a funding stream, and a venue risk. | | [The 24-Hour Rolling-Return Artifact](https://aligrithm.com/the-24-hour-rolling-return-artifact/) | A clock you imposed on a market that does not close will manufacture a return that is not there. | | [Using Ratios as Trading Signals](https://aligrithm.com/using-ratios-as-trading-signals/) | A ratio is a wrapper around two prices. Know which leg the position is betting. | On the map: [← I · Foundations](https://aligrithm.com/spine-i-foundations/) · [The grand map](https://aligrithm.com/start-here/) · [III · Market theory →](https://aligrithm.com/spine-iii-market-theory/) ### III · Market theory URL: https://aligrithm.com/spine-iii-market-theory/ Last updated: 2026-10-16T09:34:20.000Z *how markets can work in principle, efficiency is the null hypothesis* Events arrive in bursts. Volatility clusters. A liquidity shock can rewire correlations inside an afternoon. Clock time treats those as evenly spaced samples, and a lot of the structure lives in the waiting time, the path between prints, and the shape of the book of correlations. The articles here are frontier reading: they assume you already know what a null is, what a feature is, and what a cost line is. Read them cold and you collect vocabulary. Vocabulary does not size a position. Theory is the station where you decide what would count as structure at all. Efficiency is the null: the burden of proof sits on the rule, and a random-looking path can still pay a premium for a risk. The frontier pieces here are optional until the rest of the spine is standing. The Quant Map PDF works this spine through example research: five full descents from observation to live rule, and a library of 835 strategy entries, each with a named payer and a named failure mode. [Get the free PDF on the Start Here page](https://aligrithm.com/start-here/). ## Reading order Top to bottom. Stop at the first article you cannot apply to a live trade; that is the one to read slowly. | Article | Why it is on this row | | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------- | | [Market Data Is Social Data: From Fractals to Topological Data Analysis](https://aligrithm.com/market-data-is-social-data-from-fractals-to-topological-data-analysis/) | The cross-section has a shape. Persistent homology is one way to measure when that shape tears. | | [The Omori Law: Why Aftershocks Follow Market Crashes](https://aligrithm.com/the-omori-law-why-aftershocks-follow-market-crashes/) | After a crash the intensity of large moves decays in a way you can fit. The clock for that decay is event time. | | [Long-Range Dependence: Real Memory or Just Short-Range Echo?](https://aligrithm.com/long-range-dependence-real-memory-or-just-short-range-echo/) | A slow autocorrelation can be real memory or a short-memory process with a break. The distinction changes the forecast. | | [Markets Are Getting More Random: Non-Stationarity of Randomness Itself](https://aligrithm.com/markets-are-getting-more-random-non-stationarity-of-randomness-itself/) | The amount of structure is a time series. A rule fit to a structured decade meets a noisier one. | | [Itô Calculus, Intuitively](https://aligrithm.com/ito-calculus-intuitively/) | The continuous-time account of how volatility enters the P&L. The drift you think you have and the drift you compound are different objects. | | [Random Walk and Efficient Markets Are Not the Same Thing](https://aligrithm.com/random-walk-and-efficient-markets-are-not-the-same-thing/) | A random-looking path can still pay a premium. Efficiency is the null you have to beat. | On the map: [← II · Market architecture](https://aligrithm.com/spine-ii-market-architecture/) · [The grand map](https://aligrithm.com/start-here/) · [IV · Edge mechanisms →](https://aligrithm.com/spine-iv-edge-mechanisms/) ### IV · Edge mechanisms URL: https://aligrithm.com/spine-iv-edge-mechanisms/ Last updated: 2026-10-16T09:34:21.000Z *why should an edge exist?, the root question of the map* Name the payer before you name the indicator. Reachable money is usually a risk someone better-resourced refuses to warehouse, a capacity limit that makes the trade a rounding error for them, or a flow that has to deal. A $5 million niche earning 40% a year is a living at your size and 0.01% of a $20 billion fund, which is why the fund leaves it. If the answer to "why does this pay" is "the backtest went up," you do not have a mechanism yet. This is the root question of the map. Ten mechanism families exist: risk premia, behavioral, macro transmission, relative pricing, positioning and leverage, microstructure, volatility, structural and events, information transmission, crypto-native. Every strategy in the library hangs off one of them, and the PDF names the family on each of its 835 rows. The Quant Map PDF works this spine through example research: five full descents from observation to live rule, and a library of 835 strategy entries, each with a named payer and a named failure mode. [Get the free PDF on the Start Here page](https://aligrithm.com/start-here/). ## Reading order Top to bottom. Stop at the first article you cannot apply to a live trade; that is the one to read slowly. | Article | Why it is on this row | | ------------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------- | | [The Theory of Edge: Why the Market Pays You](https://aligrithm.com/the-theory-of-edge-why-the-market-pays-you/) | Two checks: why you make money, and why you keep it. The barrier is measured against the participant across from you. | | [Who Is the Marginal Buyer?](https://aligrithm.com/who-is-the-marginal-buyer/) | Price is the last clip. If you cannot name who bids here, nothing is under the print. | | [Large Trades Are Insider Trades by Definition](https://aligrithm.com/large-trades-are-insider-trades-by-definition/) | Impact scales near the square root of size. The part that does not revert is the information. | | [Predictive Power vs Long Bias: The Hidden Trap in Backtests](https://aligrithm.com/predictive-power-vs-long-bias-the-hidden-trap-in-backtests/) | A QQQ trend rule's 22% split into about 16% equity drift, 5% tech tilt, and 1% timing. | | [The Carry Trade: Up the Escalator, Down the Elevator](https://aligrithm.com/the-carry-trade-up-the-escalator-down-the-elevator/) | The interest gap is the premium. The crash is the risk you have to date before you hold it. | | [Market Making Is Not Just Collecting the Spread](https://aligrithm.com/market-making-is-not-just-collecting-the-spread/) | The job is quoting a fair value and surviving the flow that knows more than you do. | | [Why Fat Tails Are the Reason Trend Following Works](https://aligrithm.com/why-fat-tails-are-the-reason-trend-following-works/) | The payer is the holder who cannot sit through the left tail you are being paid to take. | | [Alpha Decay Is Just Competition (and Papers Lie)](https://aligrithm.com/alpha-decay-is-just-competition-and-papers-lie/) | Realized edge shrinks as about 1 over (1 plus the number of people trading it). A printed paper is already a crowd. | On the map: [← III · Market theory](https://aligrithm.com/spine-iii-market-theory/) · [The grand map](https://aligrithm.com/start-here/) · [V · Observable world →](https://aligrithm.com/spine-v-observable-world/) ### V · Observable world URL: https://aligrithm.com/spine-v-observable-world/ Last updated: 2026-10-16T09:34:22.000Z *what can we measure?, point-in-time, or it does not exist* An observable is whatever was knowable then: price, a rate, funding, open interest, a commitment-of-traders print, the book. A revised series, a delisting filled in later, a hedge ratio fit on the whole sample, none of those existed that morning. Point-in-time is the filter. If the series fails it, the feature you build on top is a look at the answer. The station's one rule: if it was not knowable that morning, it does not exist. Revised series, backfilled delistings, and full-sample hedge ratios are all looks at the answer. The Quant Map PDF works this spine through example research: five full descents from observation to live rule, and a library of 835 strategy entries, each with a named payer and a named failure mode. [Get the free PDF on the Start Here page](https://aligrithm.com/start-here/). ## Reading order Top to bottom. Stop at the first article you cannot apply to a live trade; that is the one to read slowly. | Article | Why it is on this row | | ------------------------------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------- | | [Noise Is Not Volatility](https://aligrithm.com/noise-is-not-volatility/) | Two paths with the same range can be a trend or a chop. The efficiency ratio separates them: net move over path length. | | [Efficiency Ratio Explained for Traders](https://aligrithm.com/efficiency-ratio-explained-for-traders/) | The number, 0 to 1, that tells you whether the market you are about to trade is the kind your rule wants. | | [How to Rank Markets by Trend Quality](https://aligrithm.com/how-to-rank-markets-by-trend-quality/) | Rank the universe by the observable before you drop one system on all of them. | | [Reading the COT Report: Three Trader Groups, One Edge](https://aligrithm.com/reading-the-cot-report-three-trader-groups-one-edge/) | Positioning is public, weekly, and late. The lag is why a slow account can still use it. | | [Why Commercials Are Counter-Trend (and Lead by 2 Weeks)](https://aligrithm.com/why-commercials-are-counter-trend-and-lead-by-2-weeks/) | The group, the sign, and the lag. A COT rule that ignores which group is a story. | | [Order Book Imbalance: The First Microstructure Feature to Test](https://aligrithm.com/order-book-imbalance-the-first-microstructure-feature-to-test/) | The standing queue is the observable. Mid is a summary that throws most of it away. | | [Microprice: Better Than Mid Price?](https://aligrithm.com/microprice-better-than-mid-price/) | A size-weighted quote is a different observable from the average of bid and ask. | | [Fill Probability from Trade Size CDFs](https://aligrithm.com/fill-probability-from-trade-size-cdfs/) | Whether you get filled is measurable from the tape. A backtest that assumes the fill is assuming the answer. | On the map: [← IV · Edge mechanisms](https://aligrithm.com/spine-iv-edge-mechanisms/) · [The grand map](https://aligrithm.com/start-here/) · [VI · Feature engineering →](https://aligrithm.com/spine-vi-feature-engineering/) ### VI · Feature engineering URL: https://aligrithm.com/spine-vi-feature-engineering/ Last updated: 2026-10-16T09:34:23.000Z *transforms of observable state, one raw variable, many research questions* One raw series supports many questions, and they are different hypotheses. Carry level asks how big the paycheck is. Carry over volatility asks how big it is per unit of pain. Carry rank asks whether this pair is the best paid in the set. Carry change asks whether the gap is widening. Carry conditioned on trend asks whether price already agrees. Five features, two rates and a spot. RSI, MACD, and a stochastic on the same close are often one band agreeing with itself. Stacking them feels like confirmation. The frequency response says it is one feature in costume. The filter shelf past this table is the same question asked of one operator: which band it keeps, and how late. Butterworth, sinc, Hilbert, roofing, and the named oscillators are later visits. Start with the primer and the frequency response. A filter output is a feature. The forecast is a later station. A feature is a question asked of an observable, and one observable supports many questions. The filter shelf, moving averages, oscillators, band-pass constructions, is one operator family: which band it keeps, and how late it tells you. The Quant Map PDF works this spine through example research: five full descents from observation to live rule, and a library of 835 strategy entries, each with a named payer and a named failure mode. [Get the free PDF on the Start Here page](https://aligrithm.com/start-here/). ## Reading order Top to bottom. Stop at the first article you cannot apply to a live trade; that is the one to read slowly. | Article | Why it is on this row | | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- | | [The Indicator Is More Important Than the Model](https://aligrithm.com/the-indicator-is-more-important-than-the-model/) | No transform of a weak input carries more information about the target than the input did. The ceiling is the feature. | | [Why Most Indicators Should Be Transformed Before Modeling](https://aligrithm.com/why-most-indicators-should-be-transformed-before-modeling/) | Scale, tails, stationarity, and cross-market units. The transform is part of the feature's name, and its window is not a backtest knob. | | [Feature Engineering Before Machine Learning](https://aligrithm.com/feature-engineering-before-machine-learning/) | On daily bars the sample is small and the signal is thin. The pipeline that stationarizes and checks the histogram does more than the model swap. | | [No Filter Is Predictive: What Traders Misunderstand About Smoothing](https://aligrithm.com/no-filter-is-predictive-what-traders-misunderstand-about-smoothing/) | A causal filter is a function of the past. A crossover is the moment the less-lagged average caught up, after the turn. | | [DSP and Digital Filters for Traders: The Primer Nobody Wrote First](https://aligrithm.com/dsp-and-digital-filters-for-traders-the-primer-nobody-wrote-first/) | Trend, deviation, cycle, noise. Lag is the center of mass of the weights. Nyquist is two bars. | | [The Frequency Response of Trading Indicators](https://aligrithm.com/the-frequency-response-of-trading-indicators/) | RSI, MACD, stochastic, and ADX often sit in the same cycle band. Confluence of those four is one vote. | | [CMMA: A Better Momentum Primitive Than Price-minus-MA Alone](https://aligrithm.com/cmma-a-better-momentum-primitive-than-price-minus-ma-alone/) | Log price, a causal average, ATR, and a lookback correction, so 10-day and 200-day momentum share an axis. | | [Momentum Is a High-Pass Filter](https://aligrithm.com/momentum-is-a-high-pass-filter/) | Momentum is the deviations the low-pass threw away. Name the band or you are holding a smoother with a sign. | On the map: [← V · Observable world](https://aligrithm.com/spine-v-observable-world/) · [The grand map](https://aligrithm.com/start-here/) · [VII · Hypothesis →](https://aligrithm.com/spine-vii-hypothesis/) ### VII · Hypothesis URL: https://aligrithm.com/spine-vii-hypothesis/ Last updated: 2026-10-16T09:34:24.000Z *“if mechanism X is real, feature F should predict Y over horizon H”* Write the death condition before the run. "Ranked by carry-to-volatility, the top third of G10 beats the bottom third over the next one to three months, after costs" can die: the buckets fail to rise in order, or the whole result lives in 2002 to 2007\. "This chart looks like a head and shoulders" cannot die. Explanation is cheap after the print. A prediction is a dated, specific claim you cannot edit. The scientific bar is the pre-commit: hypothesis, null, statistic, threshold, held-out window, trial count, in a file whose timestamp is earlier than the result. A hypothesis is a dated, specific claim that can die. Write the death condition before the run, in a file whose timestamp is earlier than the result. The Quant Map PDF works this spine through example research: five full descents from observation to live rule, and a library of 835 strategy entries, each with a named payer and a named failure mode. [Get the free PDF on the Start Here page](https://aligrithm.com/start-here/). ## Reading order Top to bottom. Stop at the first article you cannot apply to a live trade; that is the one to read slowly. | Article | Why it is on this row | | --------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------ | | [Technical Analysis as a Scientific Hypothesis](https://aligrithm.com/technical-analysis-as-a-scientific-hypothesis/) | The rule, the statistic, the null, the threshold, written first. A popular moving-average cross on SPX often fails to clear 0.05. | | [How to Make Technical Analysis Falsifiable](https://aligrithm.com/how-to-make-technical-analysis-falsifiable/) | Turn the pattern into a function, a horizon, a success count, and a baseline. Then publish the rejection. | | [The Null Hypothesis for Trading Systems](https://aligrithm.com/the-null-hypothesis-for-trading-systems/) | Start from "no predictive power on detrended data." A rule that has not beaten that is a candidate. | | [The Difference Between Explanation and Prediction in Markets](https://aligrithm.com/the-difference-between-explanation-and-prediction-in-markets/) | A story fitted after the close had the answer. A forecast did not. Keep a dated log if you want the second. | | [The Scientific Method for Building Trading Systems](https://aligrithm.com/the-scientific-method-for-building-trading-systems/) | Eleven gates. A fail at any one stops the ship. Stacking them is how you buy a lower false-positive rate. | | [Induction in Trading: Why Past Patterns Are Always Uncertain](https://aligrithm.com/induction-in-trading-why-past-patterns-are-always-uncertain/) | The hit rate has a confidence interval, and the interval still assumes the future rhymes. Regime quality is the part the interval ignores. | On the map: [← VI · Feature engineering](https://aligrithm.com/spine-vi-feature-engineering/) · [The grand map](https://aligrithm.com/start-here/) · [VIII · Validation →](https://aligrithm.com/spine-viii-validation/) ### VIII · Validation URL: https://aligrithm.com/spine-viii-validation/ Last updated: 2026-10-16T09:34:25.000Z *is it real, and does it survive costs?, most ideas die here, and that is the point* Most ideas die here, and the archive is the output. A single Sharpe is a sample statistic with a wide interval: a Sharpe near 1 on a few years of daily data is often consistent with a true Sharpe near 0\. The best of forty-one thresholds is a maximum. At a 5% bar the family-wise false-positive rate on that scan is about 88% before you correct it. Add the spread, the fees, and an honest slippage before you like the curve. A printed machine-learning paper can fail on its own numbers: a do-nothing baseline, a base rate, a cost line that was never subtracted. Most ideas die here, and that is the point: the archive of rejected rules is the asset. The station's output is not a curve, it is a verdict with a trial count attached. The Quant Map PDF works this spine through example research: five full descents from observation to live rule, and a library of 835 strategy entries, each with a named payer and a named failure mode. [Get the free PDF on the Start Here page](https://aligrithm.com/start-here/). ## Reading order Top to bottom. Stop at the first article you cannot apply to a live trade; that is the one to read slowly. | Article | Why it is on this row | | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- | | [Why One Backtest Tells You Almost Nothing](https://aligrithm.com/why-one-backtest-tells-you-almost-nothing/) | Report the interval. A 12% annualized mean on a noisy daily sample can have a 95% band that includes a loss. | | [Backtesting Is an Experiment, Not a Screenshot](https://aligrithm.com/backtesting-is-an-experiment-not-a-screenshot/) | Pre-commit, detrend, permute, correct for the number of tries. One equity curve is one draw. | | [CSCV: A Direct Probability of Backtest Overfit](https://aligrithm.com/cscv-a-direct-probability-of-backtest-overfit/) | The probability that the in-sample winner is a loser out of sample, computed from the search you ran. | | [Permutation Tests for Indicator Significance](https://aligrithm.com/permutation-tests-for-indicator-significance/) | Shuffle the target, rerun the whole scan, and compare your best statistic to the best of each shuffle. | | [How to Test Indicator Thresholds Without Fooling Yourself](https://aligrithm.com/how-to-test-indicator-thresholds-without-fooling-yourself/) | The RSI threshold that prints p = 0.016 in one scan can print about 0.3 once the scan is inside the test. | | [Why Walk-Forward Testing Is Better Than One Big OOS Split](https://aligrithm.com/why-walk-forward-testing-is-better-than-one-big-oos-split/) | Fit, grade the unseen window, roll. One frozen split is one anecdote about one date. | | [The Backtest Integrity Checklist](https://aligrithm.com/the-backtest-integrity-checklist/) | Point-in-time data, costs, trial count, regime split, a kill rule. Run it before capital. | | [The Hill, the Spike, and the Cliff: Reading Optimization Surfaces](https://aligrithm.com/the-hill-the-spike-and-the-cliff-reading-optimization-surfaces/) | A plateau you can move 20% and keep. A needle you found because you looked. | | [How to Spot a Fake ML Trading Paper](https://aligrithm.com/how-to-spot-a-fake-ml-trading-paper/) | Three published accuracy numbers, and the baselines those papers did not print. | | [LLM Sentiment Trading: A Sharpe of Three That Needs an Autopsy](https://aligrithm.com/p/b84e9f26-90e4-4b96-8d6f-0b672c234e4c/) | Mean 0.55% over volatility 2.49%, times the square root of 252, is about 3.5\. The printed label says 3.05, and the split shares the label's window. | On the map: [← VII · Hypothesis](https://aligrithm.com/spine-vii-hypothesis/) · [The grand map](https://aligrithm.com/start-here/) · [IX · Forecast / signal →](https://aligrithm.com/spine-ix-forecast-signal/) ### IX · Forecast / signal URL: https://aligrithm.com/spine-ix-forecast-signal/ Last updated: 2026-10-16T09:34:27.000Z *a statement about the future, not a trade yet* AUDJPY at +1.3 and EURCHF at −0.8 are comparable units, and neither is a position. The station is the map from feature to expected value; in a cross-section, the stronger version is often an order, not a number. The forecast is still not a trade. It is a statement in comparable units across names and dates, and the cheapest honest version is often a rank. The Quant Map PDF works this spine through example research: five full descents from observation to live rule, and a library of 835 strategy entries, each with a named payer and a named failure mode. [Get the free PDF on the Start Here page](https://aligrithm.com/start-here/). ## Reading order Top to bottom. Stop at the first article you cannot apply to a live trade; that is the one to read slowly. | Article | Why it is on this row | | --------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------- | | [Ranking Beats Forecasting for Many Trading Problems](https://aligrithm.com/ranking-beats-forecasting-for-many-trading-problems/) | The cross-section often needs an order. Rank, then hold the wings. | | [Ranking Beats Forecasting — But Ranking What?](https://aligrithm.com/p/a9fbe29e-35a0-4e84-8723-b22f061d03ef/) | The target you rank on decides the trade. A rank of the wrong residual is a clean sort of noise. | | [From Indicator Value to Expected Value](https://aligrithm.com/from-indicator-value-to-expected-value/) | Map the feature to a conditional expectation before you turn it into a position. A z-score still has to become an edge in currency units. | On the map: [← VIII · Validation](https://aligrithm.com/spine-viii-validation/) · [The grand map](https://aligrithm.com/start-here/) · [X · Decision policy →](https://aligrithm.com/spine-x-decision-policy/) ### X · Decision policy URL: https://aligrithm.com/spine-x-decision-policy/ Last updated: 2026-10-16T09:34:28.000Z *forecast → position: trade only if E\[r\] clears costs + threshold* The decision is the arithmetic after the score: expected edge next month, about a 4.1% gap divided by 12, near 34 basis points, against a couple of basis points to get in and out at institutional size. Clear the toll with room, then size by risk so one wild pair cannot own the book. Refuse a rule whose expected gross edge per trade sits under about three times the all-in round trip. That filter deletes most of what gets marketed as a five-minute system. One number decides the station: expected edge per trade against the all-in round trip. Under about three to one, the rule is a donation schedule. The Quant Map PDF works this spine through example research: five full descents from observation to live rule, and a library of 835 strategy entries, each with a named payer and a named failure mode. [Get the free PDF on the Start Here page](https://aligrithm.com/start-here/). ## Reading order Top to bottom. Stop at the first article you cannot apply to a live trade; that is the one to read slowly. | Article | Why it is on this row | | -------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------- | | [Expectancy: The Most Important Formula in Trading](https://aligrithm.com/expectancy-the-most-important-formula-in-trading/) | Win rate times average win, minus loss rate times average loss. The tail term is the one the win rate hides. | | [Cost-Aware Ranking: The Missing Step in Cross-Sectional Strategies](https://aligrithm.com/cost-aware-ranking-the-missing-step-in-cross-sectional-strategies/) | A name that ranks first and costs 40 basis points to enter is a different trade from a name that ranks third and costs 2. | | [Why Transaction Costs Should Be Added Before You Fall in Love](https://aligrithm.com/why-transaction-costs-should-be-added-before-you-fall-in-love/) | Put the toll in the backtest on the first run. A curve you have already bonded with will argue for a kinder cost. | | [The Trader's Real Job: Control Losses, Not Predict Everything](https://aligrithm.com/the-traders-real-job-control-losses-not-predict-everything/) | Direction is the variable you do not set. Size, exposure, the drawdown stop, and the kill criterion are the ones you do. | On the map: [← IX · Forecast / signal](https://aligrithm.com/spine-ix-forecast-signal/) · [The grand map](https://aligrithm.com/start-here/) · [XI · Strategy →](https://aligrithm.com/spine-xi-strategy/) ### XI · Strategy URL: https://aligrithm.com/spine-xi-strategy/ Last updated: 2026-10-16T09:34:29.000Z *a decision rule repeated through time, and not everything that trades is alpha* A strategy is the decision frozen: monthly, rank by carry-to-vol, long the top three, short the bottom three, equal risk, rebalance. Morning mood is no longer an input. A clever way to buy is plumbing, a regime model is weather, and a risk limit is a seat belt. Useful, and none of them is a reason someone is paying you. High-noise markets want reversion. Low-noise markets want trend. One indicator on every chart is a refusal to read the observable. A strategy is the decision frozen so a stranger could run it. Everything that trades is not alpha: execution tricks, regime filters, and risk limits are plumbing, weather, and seat belts. The Quant Map PDF works this spine through example research: five full descents from observation to live rule, and a library of 835 strategy entries, each with a named payer and a named failure mode. [Get the free PDF on the Start Here page](https://aligrithm.com/start-here/). ## Reading order Top to bottom. Stop at the first article you cannot apply to a live trade; that is the one to read slowly. | Article | Why it is on this row | | ------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- | | [The Difference Between a Trading Rule, a Strategy, and a Portfolio](https://aligrithm.com/the-difference-between-a-trading-rule-a-strategy-and-a-portfolio/) | A rule emits a score. A strategy adds size, a vol target, and a kill. A portfolio is the set of strategies that can survive one of them dying. | | [Trading Systems Are Recipes, Not Predictions](https://aligrithm.com/trading-systems-are-recipes-not-predictions/) | Precise enough that a stranger gets your trades on your data. The recipe includes the kill, because the regime the recipe assumes will end. | | [Matching Strategy Families to Market Conditions](https://aligrithm.com/matching-strategy-families-to-market-conditions/) | Family first: trend, reversion, carry, breakout. The market's noise level picks the family. | | [High Noise Markets Are Mean-Reversion Markets](https://aligrithm.com/high-noise-markets-are-mean-reversion-markets/) | Path length long relative to net move. Fading is the family that matches the observable. | | [Low Noise Markets Are Trend-Following Markets](https://aligrithm.com/low-noise-markets-are-trend-following-markets/) | Net move large relative to path. Breakout and trend are the family. A fader here is fighting the structure. | | [Why "Works on All Markets" Is Usually a Red Flag](https://aligrithm.com/why-works-on-all-markets-is-usually-a-red-flag/) | A rule indifferent to noise, session, and wrapper has met a backtest universe. | | [Building a Trend Follower, Component by Component](https://aligrithm.com/building-a-trend-follower-component-by-component/) | Entry, filter, size, cost, kill, each as its own part. You can see which part failed. | | [Why Breakout Systems Need Low Noise Environments](https://aligrithm.com/why-breakout-systems-need-low-noise-environments/) | The breakout is a bet on path efficiency. In chop it pays the spread to reverse. | On the map: [← X · Decision policy](https://aligrithm.com/spine-x-decision-policy/) · [The grand map](https://aligrithm.com/start-here/) · [XII · Factors →](https://aligrithm.com/spine-xii-factors/) ### XII · Factors URL: https://aligrithm.com/spine-xii-factors/ Last updated: 2026-10-16T09:34:29.000Z *the same mechanism, systematically, across an entire universe* Factors come after one asset works end to end. A carry factor is the same question asked of every pair: high carry against low carry, and you no longer care whether AUDJPY rises. A factor is the same mechanism asked of a whole universe, long the high slice, short the low slice. Single-name conviction leaves; monotonic buckets arrive. The Quant Map PDF works this spine through example research: five full descents from observation to live rule, and a library of 835 strategy entries, each with a named payer and a named failure mode. [Get the free PDF on the Start Here page](https://aligrithm.com/start-here/). ## Reading order Top to bottom. Stop at the first article you cannot apply to a live trade; that is the one to read slowly. | Article | Why it is on this row | | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------- | | [What Is a Factor, Really](https://aligrithm.com/what-is-a-factor-really/) | The same mechanism, applied to a universe, long the high slice and short the low slice. | | [What a Factor Actually Is: α + βλ, and Why the Market Is Factor #1](https://aligrithm.com/what-a-factor-actually-is-a-bl-and-why-the-market-is-factor-1/) | Split the return into the market, the premia you are paid to hold, and a residual. Sell the residual last. | | [Predict Residual Returns, Not Gross](https://aligrithm.com/predict-residual-returns-not-gross/) | Gross return smuggles the beta. The thing you meant to forecast is what is left after the factor exposures. | | [Portfolio Sorts From Scratch: Deciles, Monotonicity, and the Long-Short Spread](https://aligrithm.com/portfolio-sorts-from-scratch-deciles-monotonicity-and-the-long-short-spread/) | Buckets have to rise in order. The long-short spread is the claim. A pretty top decile with a messy middle is a story. | | [Cross-Sectional Percentile Rank Within a Universe](https://aligrithm.com/cross-sectional-percentile-rank-within-a-universe/) | Today's rank against the names you could have traded today. A rank against a future member list is leakage. | On the map: [← XI · Strategy](https://aligrithm.com/spine-xi-strategy/) · [The grand map](https://aligrithm.com/start-here/) · [XIII · Multi-factor models →](https://aligrithm.com/spine-xiii-multi-factor-models/) ### XIII · Multi-factor models URL: https://aligrithm.com/spine-xiii-multi-factor-models/ Last updated: 2026-10-16T09:34:31.000Z *composites: carry + momentum + value, regime-conditioned* Composition reopens the search. A leftover alpha in a regression is often a factor you left out. A zoo of published factors is a counted search; the t-stat you remember was the maximum of many tries. Timing the factors, turning them on and off with a second model, mostly fails once you count that second search. Cross-sectional machine learning, in the study that measured it carefully, lifts about 20%. Composites inherit every search that built them, plus the new one that weights them. Count both before admiring the combined curve. The Quant Map PDF works this spine through example research: five full descents from observation to live rule, and a library of 835 strategy entries, each with a named payer and a named failure mode. [Get the free PDF on the Start Here page](https://aligrithm.com/start-here/). ## Reading order Top to bottom. Stop at the first article you cannot apply to a live trade; that is the one to read slowly. | Article | Why it is on this row | | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------- | | [The GRS Test: Your Alpha Is Probably a Missing Factor](https://aligrithm.com/the-grs-test-your-alpha-is-probably-a-missing-factor/) | A joint test on whether the leftovers are all zero. One shiny intercept is a single name in that test. | | [Fama-MacBeth Two-Pass and the Shanken Correction Nobody Applies](https://aligrithm.com/fama-macbeth-two-pass-and-the-shanken-correction-nobody-applies/) | Cross-sectional slopes, then the error in those slopes. Skipping the correction makes the t-stats look finished. | | [The Factor Zoo and the Multiple-Testing Reckoning](https://aligrithm.com/the-factor-zoo-and-the-multiple-testing-reckoning/) | Hundreds of published factors, one history. The bar moves once you count the tries. | | [Factor Timing Mostly Fails — the Honest Version](https://aligrithm.com/factor-timing-mostly-fails-the-honest-version-2/) | A second model that switches factors reopens the search. Most of the switching does not survive it. | | [Does ML Actually Help Asset Pricing? Kelly's 20%, Not 2–3×](https://aligrithm.com/does-ml-actually-help-asset-pricing-kellys-20-not-2-3x/) | The measured cross-sectional lift is about 20%. Read the multiple in the abstract against that. | On the map: [← XII · Factors](https://aligrithm.com/spine-xii-factors/) · [The grand map](https://aligrithm.com/start-here/) · [XIV · Alpha ensembles →](https://aligrithm.com/spine-xiv-alpha-ensembles/) ### XIV · Alpha ensembles URL: https://aligrithm.com/spine-xiv-alpha-ensembles/ Last updated: 2026-10-16T09:34:32.000Z *combining edges into one forecast engine* Two edges with a Sharpe near 0.5, roughly uncorrelated, combine to about 0.5 times the square root of 2, near 0.7\. Five copies of the same idea, correlation 0.85, give you almost nothing and one decay date. Combining works when the edges share a payer logic and differ in noise. Five paraphrases of one price series are one bet with one decay date. The Quant Map PDF works this spine through example research: five full descents from observation to live rule, and a library of 835 strategy entries, each with a named payer and a named failure mode. [Get the free PDF on the Start Here page](https://aligrithm.com/start-here/). ## Reading order Top to bottom. Stop at the first article you cannot apply to a live trade; that is the one to read slowly. | Article | Why it is on this row | | ------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------- | | [The Death of the Single-System Trader](https://aligrithm.com/the-death-of-the-single-system-trader/) | One rule is one decay away from a multi-year sideways grind. A handful of structurally different systems is the comparison set. | | [Signal Averaging: Killing Noise with Redundant Alphas](https://aligrithm.com/signal-averaging-killing-noise-with-redundant-alphas/) | Average signals that share a mechanism and differ in noise. Averaging five paraphrases of one price series does not. | | [Don't Trust Your Strategy-Weighting Scheme](https://aligrithm.com/dont-trust-your-strategy-weighting-scheme/) | The scheme that maximized in-sample Sharpe is another search. Flat risk weights are the baseline it has to beat. | On the map: [← XIII · Multi-factor models](https://aligrithm.com/spine-xiii-multi-factor-models/) · [The grand map](https://aligrithm.com/start-here/) · [XV · Portfolio construction →](https://aligrithm.com/spine-xv-portfolio-construction/) ### XV · Portfolio construction URL: https://aligrithm.com/spine-xv-portfolio-construction/ Last updated: 2026-10-16T09:34:33.000Z *from forecasts to risk: budget it, do not just add it* Budget a portfolio volatility, cut every position when realized vol doubles, cap one name's share of risk, and write the drawdown rule before the drawdown. Signal quality and portfolio quality come apart: a real signal with the wrong weights is a bad portfolio. Overbetting is the failure mode the Kelly fraction exists to refuse. The book, not the signal, decides survival: a real edge with wrong weights is a bad portfolio. Budget risk like money, because it is. The Quant Map PDF works this spine through example research: five full descents from observation to live rule, and a library of 835 strategy entries, each with a named payer and a named failure mode. [Get the free PDF on the Start Here page](https://aligrithm.com/start-here/). ## Reading order Top to bottom. Stop at the first article you cannot apply to a live trade; that is the one to read slowly. | Article | Why it is on this row | | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------ | | [Why Volatility-Adjusted Position Sizing Matters](https://aligrithm.com/why-volatility-adjusted-position-sizing-matters/) | Size to risk. Equal vol contribution stops one pair owning the book. | | [Why Portfolio Construction Is Part of the Signal](https://aligrithm.com/why-portfolio-construction-is-part-of-the-signal/) | The weights change the equity curve as much as the forecast does. Treat them as part of the recipe. | | [The Difference Between Signal Quality and Portfolio Quality](https://aligrithm.com/the-difference-between-signal-quality-and-portfolio-quality/) | A good score and a good book fail for different reasons. Diagnose the layer you broke. | | [Drawdown Is Not Just a Number: It Is a Diagnosis](https://aligrithm.com/drawdown-is-not-just-a-number-it-is-a-diagnosis/) | Depth, length, and whether the other systems are fine. Those three say different things. | | [When a Drawdown Means the System Is Broken](https://aligrithm.com/when-a-drawdown-means-the-system-is-broken/) | Compare the live drawdown to the historical distribution you wrote down. The 95th percentile is a line you can put on a chart. | | [Kelly Rejects Bad Trades Automatically: Why Overbetting Kills and Underbetting Doesn't](https://aligrithm.com/kelly-rejects-bad-trades-automatically-why-overbetting-kills-and-underbetting-doesnt/) | The fraction refuses a bad price. Half-Kelly is the version that survives an error in your own edge estimate. | On the map: [← XIV · Alpha ensembles](https://aligrithm.com/spine-xiv-alpha-ensembles/) · [The grand map](https://aligrithm.com/start-here/) · [XVI · Implementation →](https://aligrithm.com/spine-xvi-implementation/) ### XVI · Implementation URL: https://aligrithm.com/spine-xvi-implementation/ Last updated: 2026-10-16T09:34:34.000Z *where paper alpha meets the market* On Hyperliquid BTC in a studied window, a refused post-only buy returned about 0.31 basis points over ten seconds. The taker fee sat around 1.44 to 2.7 basis points. Stand aside as a taker; the edge does not clear the toll. Markouts are the truth serum: where the mid is a short time after you filled. Your own size walks the book at something near the square root of the fraction of volume you took. A simulation that fills you at mid, with no queue and no clock drift, is a different market from the one that sends the reject. Paper alpha meets the queue, the fee, the borrow, and the clock here. Markouts and impact curves are the honest accounting of what touching the market costs. The Quant Map PDF works this spine through example research: five full descents from observation to live rule, and a library of 835 strategy entries, each with a named payer and a named failure mode. [Get the free PDF on the Start Here page](https://aligrithm.com/start-here/). ## Reading order Top to bottom. Stop at the first article you cannot apply to a live trade; that is the one to read slowly. | Article | Why it is on this row | | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------ | | [Five Definitions of Fair Value, and the Trade Each One Implies](https://aligrithm.com/p/b545996b-0800-4e48-ade2-a484d5288611/) | Name the subtraction before the trade. If the forecast is inside the fee, there is no trade. | | [Markouts: The Truth Serum of Market Making](https://aligrithm.com/markouts-the-truth-serum-of-market-making/) | Mark the fill against the mid a fixed time later. The backtest mark is a story you told yourself. | | [Adverse Selection Explained for Traders](https://aligrithm.com/adverse-selection-explained-for-traders/) | The flow that hits your quote knew something. The spread is the rent you charge for that. | | [Adverse Selection Is Adverse Selection: Porting Fast-Fills-Are-Bad-Fills to FX and Futures](https://aligrithm.com/adverse-selection-is-adverse-selection-porting-fast-fills-are-bad-fills-to-fx-and-futures/) | The fast fill is the toxic one in FX and futures too. | | [Market Impact and the Square-Root Law: Walking the Book to Price Your Slippage](https://aligrithm.com/market-impact-and-the-square-root-law-walking-the-book-to-price-your-slippage/) | Impact grows near the square root of size over volume. Put that in the cost line before you scale. | | [Quotes From the Future: Clock Drift and the Latency Floor](https://aligrithm.com/quotes-from-the-future-clock-drift-and-the-latency-floor/) | A timestamp that lies by a few milliseconds invents a lead. Sync the clocks before you believe the lead-lag. | | [Why Market-Making Simulations Don't Work](https://aligrithm.com/why-market-making-simulations-dont-work/) | Queue position, cancel races, and adverse selection do not show up in a fill-at-mid simulator. | | [Execution Is Part of Expected Value: VWAP Slippage, the Five-Cent Threshold, and Why Copytrading an Arb Loses](https://aligrithm.com/execution-is-part-of-expected-value-vwap-slippage-the-five-cent-threshold-and-why-copytrading-an-arb-loses/) | The detector's profit and the copied order's profit differ by the spread and the queue. Count both. | On the map: [← XV · Portfolio construction](https://aligrithm.com/spine-xv-portfolio-construction/) · [The grand map](https://aligrithm.com/start-here/) · [XVII · Live research →](https://aligrithm.com/spine-xvii-live-research/) ### XVII · Live research URL: https://aligrithm.com/spine-xvii-live-research/ Last updated: 2026-10-16T09:34:35.000Z *strategies are mortal, monitor, adapt, retire* The live rule is an experiment that does not end. Watch the paycheck (the gap can close because policy changed), the crowding (positioning stretched means a narrow exit), and the weather. Two exits are both legitimate. Adapt: go back to the mechanism, the observable, and the feature with what the break taught you. Retire: archive the rule with a post-mortem so the next version of you does not pay the tuition twice. A crowded edge grinding lower as the trade gets popular does not heal by waiting. A barriered edge inside a historical-looking drawdown is often noise. The difference is the diagnosis, and it has to be written before you are in the drawdown. Live is a test that never ends. The two legitimate exits, adapt or retire, are both decided by a diagnosis you wrote before the drawdown. The Quant Map PDF works this spine through example research: five full descents from observation to live rule, and a library of 835 strategy entries, each with a named payer and a named failure mode. [Get the free PDF on the Start Here page](https://aligrithm.com/start-here/). ## Reading order Top to bottom. Stop at the first article you cannot apply to a live trade; that is the one to read slowly. | Article | Why it is on this row | | ------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------- | | [How to Detect When a Trading System Is Dying](https://aligrithm.com/how-to-detect-when-a-trading-system-is-dying/) | Decay is usually a grind. You need a measure that fires before the equity curve feels wrong. | | [When to Switch Off a Trading System](https://aligrithm.com/when-to-switch-off-a-trading-system/) | The kill criterion is a number you set on a calm day: drawdown, hit rate over a counted window, or a structural breach. | | [Why Systems Work Until They Don't](https://aligrithm.com/why-systems-work-until-they-dont/) | The rule was conditional on a regime. The post-mortem names which condition ended. | | [Why OOS Failure Is Often a Stationarity Failure](https://aligrithm.com/why-oos-failure-is-often-a-stationarity-failure/) | The held-out window failed because the distribution moved. | | [When a Drawdown Means the System Is Broken](https://aligrithm.com/when-a-drawdown-means-the-system-is-broken/) | Inside the historical band, sit. Outside it, the kill you pre-committed gets to fire. | | [Alpha Decay Is Just Competition (and Papers Lie)](https://aligrithm.com/alpha-decay-is-just-competition-and-papers-lie/) | Half-lives differ by how easy the edge is to copy. Crowding is a breach. Waiting it out is a hope. | On the map: [← XVI · Implementation](https://aligrithm.com/spine-xvi-implementation/) · [The grand map](https://aligrithm.com/start-here/) ## Posts ### 10.20 A Trend Factor for the Crypto Cross-Section (and Its Overfit Tell) URL: https://aligrithm.com/a-trend-factor-for-the-crypto-cross-section-and-its-overfit-tell/ Last updated: 2026-10-17T11:06:59.000Z Fieberg, Liedtke, Poddig, Walker, and Zaremba build a weekly score from 28 technical indicators, buy the value-weighted top fifth of coins, and short the bottom fifth. From April 2015 to May 2022 that hedge earns 3.87% a week, t-statistic 5.19, annualized Sharpe ratio 1.94\. They then rebuild the same trade under 55,296 research designs. The published Sharpe sits at the upper edge of that pile. The median Sharpe of the estimator they recommend is 1.34, and the cell that equal-weights coins with the million-dollar size screen turned off prints 10.92\. The ranking is a pattern in this sample. The 1.94 is the friendly cell. ## What this actually is A coin book has no earnings to sort on. Market cap is a supply figure the issuer can mint, and the tape you can recompute is price and volume. The desk question is whether a stack of chart readings, combined by a shrinkage regression, ranks next week's returns well enough to pay for replacing about two-thirds of the weights every week. Each week the method turns 28 chart readings into separate forecasts of next week's return, throws out the forecasts with the wrong sign, and averages the rest into one score. They call that average CTREND. The trade is a value-weighted long-short of the top and bottom fifths, held for one week. Picture a clerk who refuses to trust one dial. He writes down a relative-strength reading, a stochastic, seven moving averages, a volume oscillator, and a Bollinger band for every coin. He checks, over the last 52 weeks, which dials pointed the same way as the following week's return, throws out the dials that pointed the wrong way, and averages the survivors. The average is the score. Top fifth against bottom fifth is the trade. If that score survived a one-day delay, a validation slice, and a multiple-testing haircut, you would swap a 2-to-4-week momentum sort for it and you would drop momentum out of a crypto factor model. If 1.94 is the upper edge of a 55,296-cell grid on one sample of 423 weeks, you would underwrite the median cell, and you would put the t-statistic of 5.19 through the old article ["The Factor Zoo and the Multiple-Testing Reckoning"](https://aligrithm.com/the-factor-zoo-and-the-multiple-testing-reckoning/) before you sized a book on it. ## Twenty-eight dials, then an average This section settles whether the combiner beats the best single oscillator, or lands next to it. CoinMarketCap, weekly, April 2015 through May 2022, 423 weeks. A coin-week needs a close, a dollar volume, and a market cap. They delete the ten daily prints in which some coin's cap exceeds Bitcoin's, require a cap of at least 1 million dollars, and truncate returns at the 0.5 and 99.5 percentiles. Table 1 counts 3,245 unique coins. The prose says 3,244\. One coin, unexplained. The table stands. The 28 inputs sit in four groups, at fixed horizons taken from the technical-analysis literature, which is how the authors keep the window lengths out of the search. Oscillators: the 14-day relative strength index (average up-move divided by average down-move), a stochastic of that index, the fast and slow stochastics (where the close sits in the 14-day high-low range, and a 3-day average of that), and the commodity channel index. Moving averages: simple averages at 3, 5, 10, 20, 50, 100, and 200 days, each divided by the week's close so a high-priced coin does not dominate, plus the gap between a 12-day and a 26-day exponential average as a percent of the fast one, and that gap minus its 9-day signal. Volume: the same seven lengths of dollar volume divided by current volume, the volume versions of that gap and its signal, and Chaikin money flow. Volatility: Bollinger lower, middle, and upper bands divided by price, and the bandwidth. Every indicator is replaced by its cross-sectional rank, mapped into the interval from -0.5 to +0.5, so the lowest coin that week sits at -0.5 and the highest at +0.5\. The paper states the interval and does not print the interpolation. Regressions are fit by value-weighted least squares, weights proportional to market cap, so a microcap's residual barely moves a slope. The combiner is imported from the equity trend factor of Han, Zhou, and Zhu and from the cross-sectional elastic net of Han, He, Rapach, and Zhou. It was built for stocks. Crypto is the new tape. Each week, for each indicator, they run a cross-sectional regression of this week's return on last week's rank. The intercept and the slope that enter the forecast are the arithmetic means of those weekly estimates over a rolling 52-week window. The first score therefore does not exist in April 2015\. The table captions still date the study from April 2015. $$ \\hat{r}^{j}\_{i,t+1} = \\bar{\\alpha}\_{j,t} + \\bar{\\beta}\_{j,t}\\, z\_{i,j,t} $$ Read it as: the forecast of coin i's next-week return from indicator j equals the 52-week-averaged intercept, in percent per week, plus the averaged slope times the coin's current rank. The rank is a pure number between -0.5 and +0.5\. The forecast is in percent per week. Worked illustration, not a coefficient from the paper. An averaged intercept of 0.80 and an averaged slope of 5.0 send a coin at rank +0.30 to 0.80 + 5.0 times 0.30, which is 2.30 percent. The same dial at rank -0.30 sends it to 0.80 minus 1.50, which is -0.70 percent. They repeat that for all 28 indicators, then regress realized return on the 28 forecasts with an elastic net. The penalty is a half-and-half mix of the sum of absolute weights and the sum of squared weights, and the penalty strength is chosen by the corrected Akaike criterion. A forecast whose fitted weight is positive stays. A forecast whose weight hits zero or goes negative is dropped. The score is the equal-weighted average of the survivors. Their equation (11) is this average. The printed notation uses one symbol for the index, the set, and the count. The denominator is the number of forecasts with a positive weight. $$ \\mathrm{CTREND}\_{i,t+1} = \\frac{1}{|J^{+}|} \\sum\_{j \\in J^{+}} \\hat{r}^{j}\_{i,t+1}, \\qquad J^{+} = \\{ j : \\theta\_{j} > 0 \\} $$ Read it as: CTREND for coin i next week is the arithmetic mean of the univariate forecasts whose elastic-net weight theta is positive. Each forecast is in percent per week, and so is the mean. Worked continuation of the illustration. The coin's first forecast is 2.30 percent and a second survivor is 1.26 percent. The score is (2.30 + 1.26) / 2 = 1.78 percent. Dividing by 28 instead of by 2 would print 0.13 percent and would throw away the selection step. A third forecast with a negative weight does not enter, and it is not flipped into a reversal signal. In this sample that restriction fits the tape: the relative strength index, a textbook reversal tool, has a high-minus-low return of +3.52% a week (t-statistic 5.41). Coins the oscillator calls overbought keep rising. The ingredients already pay. The fast stochastic's high-minus-low earns 3.96% a week (t-statistic 5.73), and its alpha against the Liu, Tsyvinski, and Wu three-factor model is 2.56% (t-statistic 4.13). The combined score earns 3.87% (t-statistic 5.19) and a three-factor alpha of 2.62% (t-statistic 4.22). On raw return the stochastic leads by 0.09 points. On three-factor alpha the score leads by 0.06 points, six basis points a week. The authors' appendix ranks the middle Bollinger band, the commodity channel index, and the convergence/divergence as the heaviest inputs. A blend can load on several dials and still land its portfolio next to the strongest dial. Moving-average signs run the other way, and the reason is the scaling. They divide the average by price, so a high reading means the average sits above the price and the coin has been falling. The 5-day average's high-minus-low is -2.90% a week (t-statistic -3.35). The continuation trade is the low reading, price above its average. A negative high-minus-low column is the trend, with the sign flipped by the scaling. ## The 3.87% is the long book This section settles who pays the 3.87%, the coins you buy or the coins you short. ![Average weekly return by CTREND quintile: Low 0.12 percent, second 0.93, third 1.12, fourth 2.72, High 3.98](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/10/article_451-quintile_returns.png) Value-weighted quintiles, rebalanced every week. Average weekly returns, in percent: low 0.12 (t-statistic 0.16), then 0.93, 1.12, 2.72, and high 3.98 (t-statistic 4.80). The high-minus-low row prints 3.87\. The two extremes differ by 3.86, and the extra hundredth is rounding. The step from the third quintile to the fourth is 1.60 points. The step from the second to the third is 0.19\. The spread lives in the tails, and the bottom fifth earns about zero. $$ SR = \\frac{\\bar{r}}{s} \\sqrt{52} $$ Read it as: the annualized Sharpe ratio is the weekly mean return divided by the weekly standard deviation, times the square root of 52\. Both the mean and the standard deviation are in percent per week, so the ratio is a pure number. The square root of 52 is the right scale when the weeks are uncorrelated. Worked with Table 3's high-minus-low row. Mean 3.87, standard deviation 14.34\. The weekly ratio is 3.87 / 14.34 = 0.2699\. Times the square root of 52, which is 7.211, that is 1.946\. The table prints 1.94\. The third digit moves with rounding of the two inputs. The other five quintile Sharpes in the same table recover from their printed means and standard deviations without that gap. Multiply 3.87 by 52 and you get 201 percentage points, arithmetic, before a week with a 14.34% standard deviation has been allowed to hurt the book. The 1.94 is the number that includes the volatility. The hedge's market beta is 0.03 (t-statistic 0.50) and its size beta is -0.08 (t-statistic -1.79). Its momentum beta is 0.79 (t-statistic 14.88). Alpha against the one-factor crypto market model is 3.80% a week (t-statistic 5.03). Alpha against the three-factor model is 2.62% (t-statistic 4.22). Split that 2.62\. The high quintile's three-factor alpha is 1.10% with a t-statistic of 1.94, which misses the 5% line the authors use for boldface. The low quintile's alpha is -1.51% (t-statistic -3.23). Taking 1.10 minus a negative 1.51 gives 2.61, against the printed 2.62\. The raw return is the long book: 3.98% minus a bottom fifth that earns 0.12%. The alpha the factor model cares about is the short book failing to earn its premium. The bottom fifth is the smaller one. Average market cap is 135 million dollars there and 1,182 million in the top fifth. Average cumulative return over the prior three weeks is -21.63% in the bottom fifth and +62.74% in the top fifth. That gap is why the momentum beta is 0.79\. The top fifth itself carries a momentum beta of 0.52, and the bottom fifth carries -0.27\. The difference is the hedge's 0.79. A double sort leaves a long-short between 1.42% and 3.10% a week after a momentum control at one to four weeks. The Fama-MacBeth slope, a cross-sectional coefficient fit separately each week and then averaged, stays between 2.00 and 2.94 with those controls in, and prints 2.62 (Newey-West t-statistic 3.93; the standard errors allow for autocorrelation) when every control enters at once. The univariate slope is 2.36 (t-statistic 5.00). If the score is already in percent per week, 2.36 says one point of predicted return lines up with 2.36 points of realized return. Averaging noisy forecasts shrinks the score's cross-sectional spread, and a second-stage slope above one is what that shrinkage looks like. The paper never prints the spread, so the 2.36 stays a slope per unit of the score. The control that bites is the Amihud illiquidity ratio, absolute weekly return divided by dollar volume. After that double sort the three-factor alpha is 0.62% and clears the 10% line. ## One sample, 55,296 outfits This section settles what "it works in 55,296 specifications" is worth. ![Densities of annualized Sharpe ratios for CTREND, the size factor, and momentum across the research-design grid, plus a box plot comparing estimators](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/10/article_451-spec_sharpe_densities.png) The grid multiplies switches: how you treat outliers, whether stablecoins stay in, the cap screen, a one-dollar price screen, rolling versus expanding windows, 26 to 104 weeks of estimation, a lag of zero or one day, quintiles versus tertiles, value weights versus equal weights, and whether the volume indicators are included. Six estimators sit on that common grid. The figure caption's comparison uses 6,144 designs. The text says 55,296 combinations. Those two numbers divide evenly, 55,296 / 6,144 = 9\. Panel C of their design table adds a validation switch and a forecast-weight switch that are filled in only for the elastic-net column. Five plain estimators plus four elastic-net variants is nine. The paper does not write the multiplication. The product is the grid, and every cell is the same 423 weeks. Graph A is the density of Sharpe ratios across all 55,296 cells. The dashed line is the published specification, and the authors place it at the upper edge. Most of the mass lies between 0.5 and 2.5\. For the elastic-net estimator alone the Sharpe runs from -1.45 to 10.92, and the median is 1.34\. The 10.92 is the corner that equal-weights coins and turns the million-dollar cap screen off. Using Lo's test, which judges whether a Sharpe ratio is positive once the standard error allows for autocorrelation, 79% of all combinations clear 5%. The size factor clears that bar in 56% of its designs (median Sharpe 0.94, maximum 4.60). Momentum clears it in 49% (median 0.83, maximum 2.30, minimum -4.47). Read the entries in their Table 6 as conditional medians. Each row holds one switch fixed and takes the median over everything else. Holding the elastic net, a few switches move that median by enough to matter. With no validation sample the median Sharpe is 1.47\. When the most recent 12 weeks are used only to decide which forecasts survive, it falls to 1.19\. A zero implementation lag prints 1.46\. A one-day lag, the version you can run after the close, prints 1.24\. A price screen at zero prints 1.64\. A screen at one dollar prints 1.13\. Dropping every volume indicator moves the median from 1.30 to 1.39, so exchange wash volume is not where this Sharpe lives. The plain multivariate regression, the original Han, Zhou, and Zhu estimator, is the wide, low box in the comparison panel. Averaging the univariate forecasts and then keeping the positive weights is what tightens the distribution. That is a methods result, and it is the part worth taking to another market. The level of the crypto Sharpe is a separate claim, and it moves when you delay the trade by a day or when you stop selecting indicators on the weeks you score. A t-statistic of 5.19 belongs to one cell. The cells share one sample, and a 79% hit rate is what a small perturbation of a positive sample looks like. The old article ["The Factor Zoo and the Multiple-Testing Reckoning"](https://aligrithm.com/the-factor-zoo-and-the-multiple-testing-reckoning/) is the tax on publishing the cell near the top. The number their own grid supports is the elastic-net median, 1.34, and the validation-sample median, 1.19\. Both beat momentum's median of 0.83\. Quote 1.34, or 1.19 when the surviving indicators are chosen on a held-out 12 weeks. ## It prices the strategies it was built from This section settles whether swapping momentum for this score prices the cross-section, or prices the indicators that went into the score. They form tangency portfolios out of sample. Weights are estimated on the prior 52 weeks, the book is rescaled to a weekly volatility of 10%, and the weight is the usual mean-variance one: the inverse covariance matrix times the mean vector, normalized to sum to one. Adding the trend factor to the three-factor tangency portfolio produces an alpha of 2.77% a week (Newey-West t-statistic 3.80). Adding momentum to a model that already has the market, size, and the trend factor produces 0.37% (t-statistic 1.02). On this test, momentum is the factor that does not expand the frontier. ![Anomaly alphas against average returns under the crypto CAPM, the three-factor model, and the model that replaces momentum with CTREND](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/10/article_451-anomaly_alphas.png) The scatter compares each anomaly portfolio's average return with the alpha a model leaves behind. On the 45-degree line, alpha equals the average return. A cloud around the horizontal axis means the alphas are near zero. The Gibbons, Ross, and Shanken statistic tests the joint claim that every alpha is zero. The one-factor market model leaves an average absolute alpha of 2.69%, 20 portfolios still individually significant, joint-test p-value 0.00%. The three-factor model leaves 1.44%, 11 significant, p-value 0.38%. The model that swaps momentum for the trend factor leaves 0.68%, 2 significant, p-value 8.02%. Split the pooled row, because that is where the win sits. On the technical-indicator portfolios the p-value goes from 0.86% under the three-factor model to 25.6% under the trend model, and the count of significant alphas goes from 8 to 1\. On the Liu-Tsyvinski-Wu anomalies, which include size and the momentum horizons, the p-value is 0.19% under both models, and the weighted pricing error stays at 0.061\. The pooled pricing error falls from 0.159 to 0.112 because the technical-indicator error falls from 0.106 to 0.061\. A score blended from those indicators should price those indicators. The paper also says the trend model explains the momentum sorts, and names the two leftovers: the size quintile, and the fast stochastic, with a t-statistic of 1.98\. The size miss has a mechanical piece they state themselves. Their size factor is a tercile sort, and the quintile sort they are trying to price has a wider spread. No model they try passes the test on that anomaly set. ## Large coins and a 68% weekly turnover This section settles what is left once you refuse the microcaps and you pay to trade. Inside the largest half of coins by cap, the hedge earns 3.84% a week (t-statistic 5.00). Inside the largest tenth, 2.51% (t-statistic 3.01), with a three-factor alpha of 2.02% (t-statistic 3.18). Inside the most liquid tenth by the Amihud ratio, 2.20% (t-statistic 2.76), alpha 1.97% (t-statistic 3.18). The largest 100 coins earn 3.39% (t-statistic 4.49) in the size table and 3.40% (t-statistic 4.48) in the cost table. A hundredth. The spread survives after the small half is gone. It is largest in the liquid half, at 4.36% a week (t-statistic 5.62). The top tenth is where about a third of the full-sample 3.87% disappears. $$ r^{\\mathrm{net}} = r^{\\mathrm{gross}} - \\left( c\_{L} \\cdot 2 \\cdot TO\_{L} + c\_{S} \\cdot 2 \\cdot TO\_{S} \\right) $$ Read it as: net and gross returns are in percent per week. Turnover is the fraction of a leg they print, and the 2 turns that half-sum into the fraction of weight traded. The cost rates are in percent per unit of weight traded, so a 30 basis-point long leg enters as 0.30\. Worked with the printed turnovers, 62.84% on the long leg and 74.09% on the short leg, at 30 basis points long and 40 short. The long-leg cost is 0.30 times 2 times 0.6284, which is 0.377\. The short-leg cost is 0.40 times 2 times 0.7409, which is 0.593\. The sum is 0.970 percentage points. Subtract from 3.87 and you get 2.90, the net return they print (t-statistic 3.89). At 0.50 long and 0.60 short the cost is 0.50 times 1.2568 plus 0.60 times 1.4818, which is 1.517, and 3.87 minus 1.517 is 2.35 (t-statistic 3.16). Their net-return equation writes the cost on the raw change in weight. The turnover equation above it grows last week's weight by one plus the coin's return before taking the difference. The printed net returns reconcile with twice the printed turnover, so the card uses the turnover equation. A flat cost rate that zeros the 3.87% mean divides 3.87 by the total weight traded, 2 times (0.6284 + 0.7409) = 2.7386\. Then 3.87 / 2.7386 = 1.413%, and they print a breakeven of 1.41%. The rate that pushes the t-statistic through 5% is 0.88%. On the largest 100 coins the gross mean is 3.40%, turnover is 68.21%, the 30/40 basis-point net is 2.45%, and the breakeven flat rate is 1.25%. Holding for two weeks instead of one cuts the average weekly return by what they call 1.5 percentage points, to 2.34%. Subtracting 1.5 from 3.87 gives 2.37, not 2.34, so the "1.5" is rounded and 2.34 is the number in the supplement. They say the mean stays past the 5% line out to a four-week hold. That table is in the supplement. The hedge replaces 68.46% of its weights each week. The old article ["Percentile-Rank Momentum With Hysteresis: Low-Churn Signals"](https://aligrithm.com/percentile-rank-momentum-with-hysteresis-low-churn-signals/) is the relevant patch: a band that keeps a coin in the book until the rank crosses out of the quintile. The four-week result says the mean is still past the 5% line at a one-month hold. The paper does not test the band. A different path summary sits in the old article ["Price-Path Convexity: A New Cross-Sectional Anomaly (−45bp per σ)"](https://aligrithm.com/price-path-convexity-a-new-cross-sectional-anomaly-45bp-per-s-2/), which prices one functional of the equity path at -45 basis points per standard deviation. This score prices a bundle of 28 functionals of the crypto path. Leave the equity figure in that article. Own the long fifth of large coins if you want the raw return and you can live with that leg's momentum beta of 0.52\. Leave the 2.62% three-factor alpha on the hedge: 1.10 points of it sit on the long leg, t-statistic 1.94, and the rest is the short. Underwrite a Sharpe nearer 1.2 than 1.94 once the design includes a one-day lag and a validation slice, because that is where their own medians sit. The 68% weekly replacement against a 141 basis-point breakeven is a wide gate, measured on CoinMarketCap from 2015 to 2022, stablecoins out, returns already truncated at half a percent. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/10/article_451-visual-0.png) ## KEY POINTS - CTREND averages the univariate 52-week Fama-MacBeth forecasts, among 28 ranked technicals, that keep a positive elastic-net weight. The value-weighted high-minus-low quintile earns 3.87% a week (t-statistic 5.19), weekly standard deviation 14.34%, printed Sharpe 1.94\. The printed mean and standard deviation annualize to 1.946. - The raw return is the long fifth, at 3.98%. The short fifth earns 0.12%. The three-factor alpha of 2.62% is mostly the short leg's alpha of -1.51%. The long leg's alpha is 1.10%, t-statistic 1.94\. Momentum beta of the hedge is 0.79. - The fast stochastic alone earns 3.96% a week and a three-factor alpha of 2.56%. The combiner adds six basis points of weekly alpha. Relative strength continues in this sample (+3.52% high-minus-low), the opposite of the textbook reversal reading. - Across the elastic-net designs the Sharpe runs from -1.45 to 10.92, median 1.34\. The published 1.94 sits at the upper edge of the 55,296-cell density. A 12-week validation slice drops the conditional median from 1.47 to 1.19\. A one-day implementation lag drops it from 1.46 to 1.24\. The 10.92 cell equal-weights coins and drops the million-dollar cap screen. 55,296 / 6,144 = 9, a product of switches on one sample. - Swapping momentum for CTREND cuts average absolute anomaly alpha from 1.44% to 0.68% and lifts the Gibbons-Ross-Shanken p-value from 0.38% to 8.02%. The pricing-error drop lives in the technical-indicator portfolios. The Liu-Tsyvinski-Wu anomaly set still rejects, p-value 0.19%, with the weighted pricing error unchanged at 0.061. - The spread survives in the largest tenth (2.51%) and the most liquid tenth (2.20%). Weekly turnover is 68.46%. At 30 basis points long and 40 short the net is 2.90%. The flat breakeven rate is 1.41%. A two-week hold prints 2.34% a week in the supplement; their "drop of 1.5 points" from 3.87 lands on 2.37, so the 1.5 is rounded. ## References - [Han, He, Rapach, and Zhou (2024). Cross-Sectional Expected Returns: New Fama-MacBeth Regressions in the Era of Machine Learning](https://doi.org/10.1093/rof/rfae027?ref=aligrithm.com) - [Han, Zhou, and Zhu (2016). A Trend Factor: Any Economic Gains from Using Information over Investment Horizons?](https://doi.org/10.1016/j.jfineco.2016.01.029?ref=aligrithm.com) - [Liu, Tsyvinski, and Wu (2022). Common Risk Factors in Cryptocurrency](https://doi.org/10.1111/jofi.13119?ref=aligrithm.com) - [Lo (2002). The Statistics of Sharpe Ratios](https://doi.org/10.2469/faj.v58.n4.2453?ref=aligrithm.com) - [Menkveld et al. (2024). Nonstandard Errors](https://doi.org/10.1111/jofi.13337?ref=aligrithm.com) - [Fieberg, Liedtke, Poddig, Walker, and Zaremba (2025). A Trend Factor for the Cross Section of Cryptocurrency Returns](https://doi.org/10.1017/S0022109024000747?ref=aligrithm.com) --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ### 10.19 Crypto Has Fundamentals: A Seven-Factor Model (Read Skeptically) URL: https://aligrithm.com/crypto-has-fundamentals-a-seven-factor-model-read-skeptically/ Last updated: 2026-10-17T11:05:59.000Z Babayev and Aliyev's on-chain quality factor posts a Newey-West t-statistic of 9.80 and a top-minus-bottom quintile return of 244.5% a year. The same paper builds its universe by keeping the 90 tokens still trading on 13 May 2026 and walking them backward, so LUNA and FTT never enter the short leg. A t-statistic of that size, on a sample that drops the tokens that died, describes the survivors. The authors say the spread is not a return you can implement. Read the model as a risk decomposition with that limit stapled to the front. ## What this actually is A book of tokens needs a split of yesterday's P&L. How much was the capitalization-weighted market, how much was a cluster of small names, how much was a bet that networks people use would beat networks they ignore. Without the split, a good month and an accidental factor tilt look identical, and the tilt cannot be hedged. Q-7 is that split. Each day the authors score every token on seven characteristics, standardize the scores to a zero mean and a unit variance inside that day's cross-section, and regress the next day's return on the lagged scores. The average slope on a score is the premium attached to it. Picture a landlord who ranks buildings by foot traffic, by rent relative to that traffic, and by how violent the street is, then checks whether the busy buildings paid more rent the next month. The rank is the factor. The extra rent is the premium. Throw out every building that collapsed before the study ends and the busy buildings look like a brilliant allocation, because the quiet ones that fell down are missing from the average. If those slopes held up in a point-in-time universe, you would attribute a crypto book's P&L to the market, to sector, and to these style tilts, and you would hedge a quality exposure you did not mean to own. Treating 244.5% as a fill costs the bid-ask on names whose spreads the authors say can exceed 1% a trade, inside a quality score whose recipe they do not publish. ## Seven scores, two of them unpublished This section settles what each score computes, and which two you cannot rebuild. Market (MKT) is the capitalization-weighted return of the universe, weights from yesterday's market cap, computed only when at least 10 tokens are present. Size (SMB) is the negative logarithm of that same lagged cap, so a small token receives a positive score. Reversal (REV) negates the 21-day cumulative return with a one-day skip, so a recent loser receives a positive score. Volatility (VOL) is the exponentially weighted standard deviation of the residual after a time-varying beta on the market, and the authors negate it, so a calm token receives the positive score. Funding (FND) is the perpetual-futures funding rate, negated and then residualized, so a token whose longs are paying shorts receives a negative score. A perpetual trading above spot has longs paying shorts every eight hours. The factor takes the other side of a high rate. Value (VAL) and quality (QUA) are the unpublished pair. Value is a rank composite of five on-chain valuation ratios, each a market cap divided by some measure of network output, then residualized on size. Quality is a rank composite of four Coin Metrics activity series (unique participants, value transferred, transaction throughput, fees paid for block space), residualized on size and on value. Sakkas and Urquhart's earlier on-chain model was market plus a whale-concentration factor. This quality score is usage, and the formula is proprietary. The t-statistic of 9.80 sits on weights the paper does not print. The residualizing is a cross-sectional regression of that day's scores. It is not a regression of the factor-return series on each other. Two factors can be uncorrelated as scores on a Tuesday and still move together as portfolios, because the names inside each quintile change and a market shock hits both books. Size and volatility still correlate at -0.56 as returns after the authors have stripped size out of the volatility score. Value and quality correlate at +0.35, which follows from both of them drawing on on-chain data for a similar subset of names. Funding's largest absolute return correlation is 0.17, against volatility. The reversal window is the one place a desk can copy the construction without a vendor's black box. The paper says "21-day cumulative return" and does not write the price ratio. The reading below treats that phrase as the compounded change in price over the window. A sum of daily simple returns would differ on a token that moved this far. $$ \\mathrm{REV}\_{i,t} = -\\left(\\frac{P\_{i,t-2}}{P\_{i,t-23}} - 1\\right) $$ Read it as: for token i on day t, skip the most recent day, take the compounded return from the close 23 days ago to the close 2 days ago (21 trading days), and flip the sign. The price ratio is dimensionless. The score, before the cross-sectional z-score, is in return units. Worked example. A token that falls from 100 to 70 over that window has a compounded return of -0.30 and a raw reversal score of +0.30\. A token that rises from 100 to 130 scores -0.30\. After standardization, the loser is the name the long quintile wants. The long-short itself is an equal-weighted quintile spread, rebuilt each day from lagged scores. $$ R\_{k,t} = \\frac{1}{n\_H}\\sum\_{i \\in H} r\_{i,t} - \\frac{1}{n\_L}\\sum\_{j \\in L} r\_{j,t} $$ Read it as: the factor return on day t is the average simple return of the top quintile H minus the average simple return of the bottom quintile L. Both averages are equal-weighted. The unit is a daily return. Worked quality. Table 6 prints the high-quality quintile at +187% a year and the low quintile at -58%. The difference is 245 percentage points. Table 4 prints 244.5%. The half point is rounding; Table 6 is in whole percentages. Quality's fourth quintile, at +190%, sits a hair above the fifth. Value is messier: the fourth quintile prints +192% and the fifth +157%, so the spread is carried by the extremes, and the middle of the sort bounces. The authors mark every row except volatility as "approximate" monotonicity. That label matches the table. Coverage is uneven. Price-based scores cover the full 90 names. Quality covers 53 tokens and 62.1% of token-days, value 52 tokens and 60.6%, funding 89 tokens. A missing score enters the cross-sectional regression as a zero, a neutral loading. Quality's quintiles hold about 10 names. The sample runs from 2 January 2020 to 13 May 2026, 2,324 days, 170,006 token-days, 73 names a day on average, growing from 33 early in 2020 to 90 by late 2025\. Sector dummies are membership flags. The counts are 38 Layer-1 names, 21 DeFi, 10 infrastructure, 9 Layer-2, 6 gaming, and 6 meme. ## The annualized number is an arithmetic sum This section settles the units on the 244.5%, because the cumulative chart is easy to misread as a wealth multiple. $$ R^{\\mathrm{ann}} = 365 \\times \\bar{r}, \\qquad \\sigma^{\\mathrm{ann}} = \\sqrt{365} \\times s $$ Read it as: the annualized return is 365 times the mean daily simple return, and the annualized volatility is the square root of 365 times the standard deviation of the daily simple return. Both inputs are decimals per day. The outputs are decimals per year. This is an arithmetic convention. It is the one the paper states. Worked quality. 244.5% a year is 2.445 / 365 = 0.6699% a day. Volatility of 53.7% a year is 0.537 / 19.105 = 2.81% a day, where 19.105 is the square root of 365\. Return divided by volatility, with a cash rate of zero, is 2.445 / 0.537 = 4.55\. The paper does not publish a full-sample Sharpe ratio. The 4.55 is this ratio. The regime Sharpes in their sentiment table are the same ratio inside one bucket. The cumulative chart sums the daily returns. The sample is 2,324 / 365 = 6.367 years long. Quality's 244.5% times 6.367 is +1,557 percentage points. Volatility's -234.4% times 6.367 is -1,492 percentage points. Market lands at +360, size at +159, reversal at +958, funding at +862, value at +1,175\. The chart's quality line finishes above +1,500% and the volatility line finishes near -1,500%. The vertical scale of the figure is that multiplication. ![Cumulative summed factor returns, January 2020 to May 2026: quality climbs past +1,500%, volatility falls near -1,500%, and the other long-short books finish between them](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/10/article_450-cumulative_factor_returns.png) A constant 0.6699% a day, compounded for 2,324 days, finishes near 5.48 million times the starting stake. The natural log of 1.006699 is 0.006676, times 2,324 is 15.52, and e to that power is about 5.48 million. The chart does not show that path. The y-axis is the sum. Quality's worst peak-to-trough loss on the compounded wealth path is -42.2%, so that particular long-short did not go to zero inside the sample. The +1,557 label remains a sum of daily percentages. You cannot withdraw it. The authors warn, above Table 4 and again in the limitations, that survivorship, transaction costs, and capacity cut these numbers hard. Bid-ask spreads on smaller tokens can exceed 1% a trade. They report no turnover, so there is no net figure to quote. A 1% round trip once a week removes 52 percentage points a year. A 1% round trip once a day removes 365 points, larger than the quality spread. Which frequency a live book would face is a turnover number they do not give. Treat 52 and 365 as a scale check. The average token-day in the panel earned 0.15%. Under the same 365-day rule that is 54.8% a year, next to the capitalization-weighted market factor at 56.6%. Those are different series. One is the pooled token-day mean. The other is a cap-weighted portfolio. They happen to land close. On a given day the cross-section itself disperses by 4.74%, which the authors put at about 2.5 times the daily cross-sectional dispersion Fama and French reported for the S&P 500\. A quintile sort has a wide field to work in. Costs are a separate test, and this paper does not run one. ## Volatility got paid, and size dropped out This section settles the sign of the volatility premium, including one sentence in the source that fights the tables. Each day they regress token returns on the lagged z-scores. The premium is the average of those daily slopes. The t-statistic divides that average by a Newey-West standard error with five lags. $$ r\_{i,t} = a\_t + \\sum\_k \\lambda\_{k,t}\\, z\_{i,k,t-1} + e\_{i,t} $$ Read it as: token i's simple return on day t equals that day's intercept, plus a slope for each factor times the token's lagged z-score on that factor, plus a residual. The z-score is dimensionless. The slope lambda is a daily return per unit of z-score. The intercept is a daily return. Worked quality. The average slope is +0.003770\. A token one cross-sectional standard deviation higher on quality earned 0.377% more the next day, in sample, with the other scores held fixed. Times 365, that is 137.6% a year per unit of z-score. Divide 0.003770 by the t-statistic 9.80 and the Newey-West standard error is 0.000385 per day. Volatility's average slope is -0.004572, t-statistic -9.19, which is -166.9% a year per unit of z-score. The negative sign is the construction. Low residual volatility received the positive score, high residual volatility earned the subsequent return, and the slope on the positive score came out negative. Baker, Bradley, and Wurgler documented the opposite pattern in equities, where the calm stocks earned the premium. For volatility the autocorrelation adjustment raised the absolute t-statistic, from 7.46 with ordinary standard errors to 9.19 with five Newey-West lags. For quality it trimmed the t-statistic from 10.44 to 9.80\. Five lags is their choice. Divide the quintile spread by the annualized slope and the implied gap between the high and low quintiles is 1.78 z-scores for quality (2.445 / 1.376) and 1.40 z-scores for volatility (2.344 / 1.669). In a standard normal, the average of the top fifth sits 1.40 standard deviations above zero and the bottom fifth 1.40 below, a gap of 2.80\. The paper never reports the average z-score inside each quintile. The slope is a partial coefficient on the in-sample window. The quintile spread is a raw sort on the full sample. The two printed numbers stand. Neither one gets edited to force a match. Table 6's volatility row runs +223, +56, +42, +1, -12 from the column marked Q1 (Low) to the column marked Q5 (High). Q5 minus Q1 is -235 percentage points, against -234.4% in Table 4\. The high score earned -12%. The low score earned +223%. Section 3.5 defines the high score as low residual volatility, because the signal is negated before the sort. Under that definition the calm tokens earned -12% and the jumpy tokens earned +223%, the same sign as the t-statistic of -9.19 and the same sign as the factor return of -234.4%. The sentence in section 4.3 says the +223% belongs to the lowest-volatility quintile and the -12% to the highest. That sentence conflicts with Table 4, with Table 6's column labels, with the signal definition, and with the t-statistic. The sentence stays as printed. The consistent set is the tables and the t-statistic. The source may have swapped "lowest" and "highest" in that one line. Size's average slope is +0.000306, a t-statistic of 1.36, once volatility has been residualized on size. Annualized, that slope is 11.2% a year per unit of z-score. The size premium in earlier crypto work was this volatility premium. The authors keep size in the model as a control for value, quality, and funding, and because dropping it still costs 1.4 points of out-of-sample per-token R-squared. The old article ["What a Factor Actually Is: α + βλ, and Why the Market Is Factor #1"](https://aligrithm.com/what-a-factor-actually-is-a-bl-and-why-the-market-is-factor-1/) writes expected excess return as alpha plus beta-transpose lambda. The Fama-MacBeth slope is this paper's estimate of lambda. Quality's lambda is 0.377% a day per unit of z-score. The market, factor number one in that old article, still dominates the fit. ## The 63% is mostly the market This section settles what "63% out of sample" and "an in-sample/out-of-sample gap of -0.001" refer to. They are different regressions. The split is June 2024\. In sample: 1,626 days. Out of sample: 698 days. The two sum to 2,324, a 70/30 cut. The in-sample window holds the 2020-2021 rise, the 2022 break, and the recovery through mid-2024. The cross-sectional R-squared is the average, across days, of the fit from that day's returns on that day's lagged scores. Style scores alone: 0.231 in sample, 0.274 out of sample. Style plus six sector dummies: 0.334 in sample, 0.335 out of sample. In-sample minus out-of-sample is -0.001\. The raw cross-sectional fit did not fall after the split. The adjusted row is the degrees-of-freedom haircut. With a raw R-squared of 0.334, 66 tokens, and 12 regressors (six style scores and six sector dummies), the adjusted figure is 1 minus 0.666 times 65/53, which equals 0.183\. The paper prints 0.178\. The 0.005 difference is what you get when the token count is not 66 on every day and the reported number is an average of daily adjusted R-squared values. The printed 0.178 stands. The same plug-in on the style-only row, raw R-squared 0.231 and six regressors, gives 0.153, which matches their adjusted in-sample figure to the digit they print. Out of sample, 90 tokens and 12 regressors turn a raw 0.335 into a plug-in adjusted value of 0.231, against their 0.234\. The in-sample cross-sectional fit that survives the haircut is about 18%, on a cross-section of 66 names. The 63% is a time-series regression, one per token. Regress the token's daily returns on the factor-return series, average the R-squared across tokens. Style plus sectors: 0.533 in sample, 0.632 out of sample. On this metric the fit rose after June 2024\. The phrase "zero overfitting" in the paper is the cross-sectional gap of -0.001\. The per-token R-squared moved from 0.533 to 0.632\. Both numbers are in the paper. They answer different questions, and the second one went up. A random equal-weighted portfolio of 30 tokens has an out-of-sample R-squared of 0.848, against 0.851 in sample. At 5 tokens the out-of-sample figure is already 0.760, at 10 tokens 0.803, at 20 tokens 0.835\. Diversification removes the idiosyncratic noise, and the factor returns explain the common part. That is the attribution use the authors defend at the end. Drop one factor and record the fall in the 0.632\. The market costs 0.234 of R-squared, which is 0.234 / 0.632 = 37.0% of the fit, the share they print. Volatility costs 0.029\. Size costs 0.014\. Quality costs 0.004\. They call quality's share 0.7% of the total; 0.004 / 0.632 = 0.63%, so the 0.7% is the rounded label. The factor with the t-statistic of 9.80 moves the average token's explained variance by 0.4 points, because the score exists for 53 of 90 names. A slope can be pinned down on the names you observe and still be a small piece of a 90-token book. ![Leave-one-out drop in out-of-sample per-token R-squared. Bar lengths are Table 10. The t-statistics on the labels are the Newey-West figures from Table 8. The market factor has no cross-sectional t-statistic in that table](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/10/article_450-leave_one_out_r2.png) The old article ["The Factor Zoo and the Multiple-Testing Reckoning"](https://aligrithm.com/the-factor-zoo-and-the-multiple-testing-reckoning/) puts a practical hurdle near a t-statistic of 3 once a literature has tried hundreds of signals. A t-statistic of 9.80 clears that hurdle. The failure mode is the economic prior in that same old article. A premium this large has to show up as comovement, or it behaves like a near-arbitrage that costs and a point-in-time universe will take apart. Quality's full-sample return over volatility is 4.55\. Its leave-one-out contribution is 0.4 points of R-squared. The return is large. The covariance footprint is small. ## Losers won, except in greed This section settles the sign fight with the earlier crypto factor model, and the sentiment split. Liu, Tsyvinski, and Wu found that market, size, and momentum span the crypto cross-section, and that momentum subsumed the other price characteristics they tried. This sample, 90 tokens through May 2026, with data the authors say runs four years past that paper, finds the opposite sign at a 21-day horizon. The reversal t-statistic is 6.19\. The quintile spread is 150.5% in the main factor table and 151.3% in the lookback table, with volatility 70.7% against 71.0% and the same -82% max drawdown. The paper presents both rows as the 21-day specification. The 0.8 point gap in the annualized return is unexplained. Both figures stay as printed. Shorter windows print larger arithmetic returns and a compounded max drawdown of -100%. Seven days: +266.1% annualized, volatility 81.5%, max drawdown -100%. Fourteen days: +200.1%, drawdown -100%. Forty-two and 63 days also print a -100% max drawdown, at +109.5% and +115.7% annualized. The 21-day row is the specification whose drawdown is -82%. A -100% max drawdown means the compounded long-short touched zero, while the 365-day annualization of the mean still printed a triple-digit gain. The fast reversal book went bust inside the backtest on the compounded path. A sector-neutral variant, standardizing the 21-day return inside each sector before ranking, prints +202% against the raw +151% the authors quote. The reversal lives inside the sector. The old article ["Percentile-Rank Momentum With Hysteresis: Low-Churn Signals"](https://aligrithm.com/percentile-rank-momentum-with-hysteresis-low-churn-signals/) builds a continuation signal on Bitcoin and Ethereum and spends its engineering on a band that stops the position flipping every time the sign changes. A desk running that continuation book and this paper's 21-day reversal book stands on opposite sides of a recent loser. The horizon where this paper's arithmetic return is largest already shows a -100% drawdown. Sentiment uses the Alternative.me Fear and Greed index, smoothed over 30 days, cut at 35 and at 65\. Fear, below 35: 746 days. Neutral, 35 to 65: 947 days. Greed, above 65: 613 days. The three sum to 2,306, eighteen days short of the 2,324-day return sample. The authors call the shares 32%, 41%, and 27%. Dividing by 2,306 gives 32.4%, 41.1%, and 26.6%. Reversal in fear: +234.6%, Sharpe ratio 3.75, so volatility inside that bucket is 234.6 / 3.75 = 62.6%. Reversal in greed: -5.1%, Sharpe ratio -0.07\. Value runs the other way: +93.9% in fear, +365.2% in greed at a Sharpe ratio of 6.10, implied volatility 365.2 / 6.10 = 59.9%. Quality stays positive in all three buckets: +183.5% in fear (Sharpe ratio 4.16), +177.6% in neutral (Sharpe ratio 3.50), +424.4% in greed (Sharpe ratio 6.37). The greed-regime quality Sharpe ratio of 6.37 implies a volatility of 424.4 / 6.37 = 66.6%, against 53.7% on the full sample. Funding stays between +128.0% and +140.6%. Volatility is negative in every regime and worst in greed, at -452.1%, Sharpe ratio -4.82. ![Annualized arithmetic factor returns in fear, neutral, and greed, from the 30-day smoothed Fear and Greed index. Quality in greed is the tall positive bar; volatility in greed is the tall negative bar](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/10/article_450-regime_annualized_returns.png) A Sharpe ratio of 6.37 on a 613-day greed subsample, inside a survivor universe, with no costs, is the object the economic prior was written to catch. The cross-sectional R-squared stays at 0.331, 0.344, and 0.323 across the three buckets. The fit is stable. The premia move around. Conditioning a tilt on this index is an in-sample cut, and the authors say it wants a further test. In this sample the sentiment index ran with the market. Its correlation with the forward 63-day market return is +0.16, t-statistic 7.52\. Square 0.16 and you get 0.0256, the predictive R-squared they describe as under 3%. ## Keep the attribution, leave the spread Use the decomposition. On a 30-token equal-weighted book the factor returns explain 85% of return variance out of sample (0.848, which rounds to the 85% in the abstract), and the market accounts for 23.4 points of the 63.2-point per-token fit. Attribute the month to the cap-weighted market, to sector, and to the style tilts the holdings happen to carry. Hedge the tilt you did not want. Leave the quintile spreads on the page. Quality at 244.5%, value at 184.5%, reversal at 150.5%, and funding at 135.4% are arithmetic long-short means on a universe defined as tokens still alive on 13 May 2026\. The authors name the bias and the mechanism: the tokens that survived are the ones with high network use and enough capitalization, which is what the quality and value scores select. Both formulas are unpublished. Coverage is 53 and 52 names. Turnover is unpublished. Spreads on the small names can exceed 1% a trade. Funding, the score closest to orthogonal in the return correlations, is still a 135.4% gross spread with a -78.3% max drawdown and no cost model. Q-7 is a sketch of the attribution split. The 244.5% stays in the table. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/10/article_450-visual-0.png) ## KEY POINTS - Q-7 scores tokens on market, size, 21-day reversal, on-chain value, residual volatility, on-chain quality, and perpetual funding. Value and quality are proprietary rank composites. You cannot rebuild the t-statistic of 9.80 from the paper. - Quality's top-minus-bottom quintile earns 244.5% a year arithmetic (Table 6's quintiles differ by 245 points). That annualization is 365 times the mean daily return, 0.6699% a day. Summed over 2,324 days it is +1,557 points, which is the vertical scale of the cumulative chart. Compounding that same daily rate over 2,324 days produces a multiple near 5.48 million. The chart is the sum. - The Fama-MacBeth slope on quality is +0.377% a day per unit of z-score (137.6% a year), Newey-West t-statistic 9.80\. Volatility's slope is -0.457% a day, t-statistic -9.19, because the positive score is low residual volatility and the jumpy tokens earned the return. One sentence in section 4.3 swaps those quintile names; the tables and the t-statistic agree with each other. - Size's t-statistic falls to 1.36 once volatility is residualized on size. The apparent size premium was a volatility premium. Size stays in the model as a control. - Per-token out-of-sample R-squared is 0.632 with sectors, and a 30-token book reaches 0.848\. The market's leave-one-out share of the 0.632 is 0.234\. Quality's is 0.004\. The cross-sectional in-sample/out-of-sample gap of -0.001 is a different regression from the per-token R-squared, which rose from 0.533 to 0.632. - This sample finds 21-day reversal (t-statistic 6.19, about +151% a year), where Liu, Tsyvinski, and Wu found continuation. The 7-day version prints +266% and a -100% max drawdown. Reversal earns +235% in fear and -5% in greed. Quality's greed-regime Sharpe ratio is 6.37. - The universe is tokens alive in May 2026, projected backward. The authors say this inflates every factor return, and that quality and value select the characteristics survivors have. Use the model to attribute a book's P&L. Leave the quintile spreads in the table. Turnover is unreported, and smaller-token spreads can exceed 1% a trade. ## References - [Liu, Tsyvinski, and Wu (2022). Common Risk Factors in Cryptocurrency](https://doi.org/10.1111/jofi.13119?ref=aligrithm.com) - [Sakkas and Urquhart (2024). Blockchain Factors](https://doi.org/10.1016/j.intfin.2024.102012?ref=aligrithm.com) - [Babayev and Aliyev (2026). Crypto Has Fundamentals: A Seven-Factor Model for Digital Asset Returns](https://papers.ssrn.com/sol3/papers.cfm?abstract%5Fid=6818558&ref=aligrithm.com) --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ### 3.47 The Real IS→OOS Sharpe Haircut (Closed Form, Not "50%") URL: https://aligrithm.com/the-real-is-oos-sharpe-haircut-closed-form-not-50/ Last updated: 2026-10-16T08:41:59.000Z Twelve commodity futures, three momentum windows and an intercept, ten years of daily closes: 444 coefficients on 2,520 rows. Jacquier, Muhle-Karbe, and Mulligan compute the fraction of the in-sample Sharpe ratio this linear book is expected to keep on new data. Their commodity case study puts that fraction at 30 percent for a ten-year backtest. They attribute the rule of thumb that keeps 50 percent to Harvey and Liu. The fraction moves with the training length, the parameter count, and the true Sharpe. One future and one signal, true Sharpe 0.50: the formula keeps 20 percent after one year and 72 percent after ten. True Sharpe 2, ten years, still one signal: it keeps 98 percent. Twelve futures, 37 signals, a slope of zero, ten years: the in-sample Sharpe is 6.58 annualized and the fraction kept is zero. ## What this actually is You run twelve commodity futures, aluminium through sugar, and you fit a 5-day, a 1-year, and a 5-year momentum signal on each name, plus an intercept. That is 37 regressors and 444 coefficients. Ten years of closes is 2,520 rows. The backtest prints a Sharpe ratio, the mean profit-and-loss divided by its standard deviation, and you have to decide what fraction of it the same linear fit keeps on the next sample. The method is a closed form for that fraction. It takes the signal count, the asset count, the number of training days, and the Sharpe the book would earn if the slopes were known. It returns an expected in-sample Sharpe and an expected out-of-sample Sharpe. Divide the second by the first. Picture a student who sits the same exam twice. On the first sitting the answer key is on the desk, and every guess that matches the key is marked correct. On the second sitting the key is in the drawer. The drop equals those guesses. A short exam with a long question sheet drops harder. A long exam with one question the student knows keeps the first score. At the desk you compute the fraction before you size the book. Keeping half of the 6.58 Sharpe that this 444-coefficient design prints when the true slope is zero would put on a book the formula sets to zero out of sample. Keeping half of one signal with a true Sharpe of 2, on ten years, throws away a book the formula says keeps 98 percent. The formula is too kind once you have discarded losing trials, and it charges no spread. Those bills sit in the old article ["CSCV: A Direct Probability of Backtest Overfit"](https://aligrithm.com/cscv-a-direct-probability-of-backtest-overfit/) and the old article ["The Backtest Integrity Checklist"](https://aligrithm.com/the-backtest-integrity-checklist/). ## The in-sample mean carries an extra term The question is why the backtest Sharpe sits above the live one even when you fit the regression once and never touch it again. The return model is linear. Tomorrow's return on each asset is a slope times today's signals, plus a residual. $$ r\_{t+1} = \\beta s\_t + \\epsilon\_{t+1} $$ Read it as a forecast. The vector r is next period's return on m assets, in return units per day. The vector s is today's signals, p of them, standardized in the formulas below to mean zero and unit variance. Beta is the m-by-p matrix of slopes, in return per unit of signal. Epsilon is the residual, mean zero, covariance sigma-epsilon. In the commodity fit, m is 12 and p is 37. You estimate beta by ordinary least squares on the training window, then you trade the Markowitz book: weights equal the inverse residual covariance times beta times the current signal. The known-covariance assumption is deliberate. Estimating the covariance is a later section, and on twelve names the fraction stays put. The fitted profit-and-loss splits into three pieces. The true piece is the book you would hold if beta were known. The misestimation piece multiplies the error in beta by the true expected return; its expectation is zero in the training window and out of it, because the residual mean is zero. The overfitting piece multiplies the error in beta by the residual itself. Inside the training window that residual is a row the regression already saw, and the expectation of that piece is not zero. Outside the window the residual is a fresh draw, and the expectation is zero. That is the entire mean gap. $$ \\mathbb{E}\[\\text{mean}\_{\\text{IS}}\] = \\operatorname{tr}(\\Gamma) + \\frac{p m}{T\_1}, \\qquad \\mathbb{E}\[\\text{mean}\_{\\text{OOS}}\] = \\operatorname{tr}(\\Gamma), \\qquad \\Gamma = \\beta^{\\top} \\Sigma\_{\\epsilon}^{-1} \\beta $$ Read it as two expected daily profit-and-loss figures, in the units of that Markowitz book, plus the gap between them. Gamma is a p-by-p matrix: beta-transpose times the inverse residual covariance times beta. Its trace is the daily profit-and-loss you would expect if the slopes were known. T1 is the number of training observations. The in-sample expectation adds p times m over T1\. The out-of-sample expectation does not. This is their Proposition 3.1 for centred signals and Markowitz weights. The add-on is not a percent of net asset value. A constant rescaling of the weights scales the mean and the standard deviation together, so the Sharpe ratio, which is what you compare to a backtest, does not depend on that scaling. Worked count for the commodity design: 12 times 37 is 444, and 444 divided by 2,520 is 0.1762\. Every training day, the expected in-sample profit-and-loss sits 0.1762 above the expected out-of-sample profit-and-loss, in those units, at any true beta. Out of sample the add-on is zero. Add signals or add assets and the gap grows in a straight line. Lengthen the backtest and it shrinks as one over T1. ## One asset and one signal The question is what that gap does to the Sharpe ratio when there is only one slope to overfit. Set m = 1 and p = 1, scale the signal variance and the residual variance to 1, and leave the intercept out. The true per-day Sharpe, the one you would get with the slope known, collapses to a function of beta alone. $$ \\mathrm{SR} = \\frac{\\beta}{\\sqrt{1 + 2\\beta^{2}}} $$ Read it as a per-day Sharpe. Beta is the slope, in daily return per unit of signal, taken positive. Square root of 252 turns it into an annualized Sharpe under the model's own assumption that each day's profit-and-loss is independent of the last. The replication ratio below is a ratio of two Sharpes, so the square root of 252 cancels. It does not cancel on the horizontal axis of a chart drawn in annualized units, which is why the conversion is stated here. Solve for the slope that delivers a true annualized Sharpe of 0.50\. The square root of 252 is 15.8745, so the per-day Sharpe is 0.50 divided by 15.8745, which is 0.031497\. Beta-squared equals 0.031497 squared, divided by 1 minus twice 0.031497 squared, which is 0.00099404\. The true daily mean profit-and-loss in this scaling is that same 0.00099404\. Add 1 divided by 2,520\. The in-sample mean is 0.00139086\. The ratio of the two means, 0.00099404 divided by 0.00139086, is 0.715. The Sharpe ratio also has a variance, and the estimation error inflates it. Their simplified form of Proposition 3.1, for this one-asset case, is the pair below. Both are per-day Sharpes. The numerator is the expected profit-and-loss. The denominator is the square root of the expected variance. T1 is the training length in days. $$ \\mathrm{SR}\_{\\mathrm{EIS}} = \\frac{\\beta^{2} + 1/T\_1}{\\sqrt{2\\beta^{4} + \\left(1 + \\dfrac{15}{T\_1-2} - \\dfrac{2}{T\_1}\\right)\\beta^{2} + \\dfrac{4}{T\_1} - \\dfrac{3}{T\_1+2} - \\dfrac{1}{T\_1^{2}}}} $$$$ \\mathrm{SR}\_{\\mathrm{EOOS}} = \\frac{\\beta^{2}}{\\sqrt{2\\beta^{4} + \\left(1 + \\dfrac{2}{T\_1-2}\\right)\\beta^{2} + \\dfrac{1}{T\_1-2}}} $$ Read the first as the in-sample approximation and the second as the out-of-sample approximation. EIS is the ratio of the expected in-sample mean to the square root of the expected in-sample variance, and EOOS is the same construction on the new sample. They use this ratio-of-expectations because the expectation of the ratio has no closed form. In their checks the gap between the two stays within 2.5 percent of the Sharpe. Plug in beta-squared of 0.00099404 and T1 of 2,520, which is the unrounded slope behind the 0.50 Sharpe. The in-sample variance is three terms: 0.000001976 from two times beta to the fourth, 0.000999168 from the slope term, and 0.000397612 from the estimation-noise constant. The sum is 0.001398756\. Its square root is 0.037400\. The in-sample mean, 0.00139086, divided by that square root, is a per-day Sharpe of 0.03719, and 0.03719 times 15.8745 is 0.590 annualized. The out-of-sample variance sums to 0.00139394\. The per-day out-of-sample Sharpe is 0.02662, and the annualized figure is 0.423\. Divide 0.4226 by 0.5904 and the fraction kept is 0.716\. The ratio of the means alone was 0.715\. In this cell the variance adjustment is a rounding error. Hold that thought for the twelve-asset book, where it is not. $$ \\text{Replication ratio} = \\frac{\\mathrm{SR}\_{\\mathrm{EOOS}}}{\\mathrm{SR}\_{\\mathrm{EIS}}} $$ Read it as the fraction of the expected in-sample Sharpe that the expected out-of-sample Sharpe still covers. One means the backtest survives whole. Zero means the expected out-of-sample Sharpe is zero. The 50 percent rule is this ratio nailed to one half. The same plug-in at other cells, all annualized by the square root of 252: | Years | True Sharpe | In-sample | Out-of-sample | Fraction kept | | ----- | ----------- | --------- | ------------- | ------------- | | 1 | 0 | 0.9903 | 0 | 0 | | 1 | 0.50 | 1.1037 | 0.2231 | 0.202 | | 1 | 0.983 | 1.3779 | 0.6890 | 0.500 | | 10 | 0 | 0.3159 | 0 | 0 | | 10 | 0.50 | 0.5904 | 0.4226 | 0.716 | | 10 | 2.00 | 2.0198 | 1.9762 | 0.978 | Check the 50 percent row before trusting the rest. Half of 1.3779 is 0.6890, and 0.6890 divided by 1.3779 is 0.500\. That row is the folklore rule, and it is one cell: one asset, one signal, one year, a true Sharpe of 0.983\. The crossings nearby sit at a true Sharpe of about 0.45 on five years, 0.32 on ten years, and 0.16 on forty years. A flat stamp of 50 percent is that curve's intersection with a horizontal line, read off at whatever sample length you happen to have. On the one-year row with a true Sharpe of 0.50, the backtest is expected to print 1.10\. Half of 1.10 is 0.55\. The formula keeps 0.223\. The stamp is more than twice too generous. On the ten-year row with a true Sharpe of 2, the backtest prints 2.02 and the formula keeps 1.98\. Half of 2.02 is 1.01\. The stamp throws away a Sharpe near 2\. On a pure-noise signal the fraction kept is zero at every length, and a one-year sample still prints an in-sample Sharpe of 0.99\. Half of 0.99 is a position. Zero is the out-of-sample number. ![Replication ratio against true annualized Sharpe for one asset and one signal, at training lengths of 1, 5, 10, and 40 years, with a horizontal line at one half](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/10/article_490-univariate_replication.png) The curves are Proposition 3.1 in the one-asset case, not a fit to the commodity tape. A worthless signal sits at the origin. A strong signal on a long sample sits near one. The dashed line is the 50 percent rule, and each training length crosses it at a different true Sharpe. ## Twelve assets, thirty-seven signals, a zero slope The question is what this commodity parameter count prints when the signals carry nothing. Proposition 3.1 with a zero beta is exact on the variance side as well as the mean side: the leftover term they drop from the in-sample variance is a covariance between the inverse signal Gram matrix and a quadratic in beta, so a zero slope kills it. Set m = 12, p = 37, T1 = 2,520, beta = 0, no intercept, known residual covariance, Markowitz weights. The in-sample mean is 444 divided by 2,520, which is 0.17619 per day. The in-sample variance reduces to two terms: $$ \\mathbb{E}\[\\mathrm{var}\_{\\mathrm{IS}}\] = \\frac{p m (2m + p + T\_1 + 4)}{T\_1 (T\_1 + 2)} - \\frac{2 m^{2} p^{2}}{T\_1^{2} (T\_1 + 2)} $$ Read it as a daily variance of the Markowitz profit-and-loss when the true slope is zero. The first term grows with the parameter count. The second subtracts a smaller piece that comes from the square of the mean. Evaluated here, the variance is 0.18057\. That agrees with the long coefficient form in Proposition 3.1, the one with their c2 and c-tilde-2, digit for digit at this precision. The square root is 0.42493\. Divide 0.17619 by 0.42493 and the per-day Sharpe is 0.4146\. Times 15.8745, the annualized in-sample Sharpe is 6.58\. The out-of-sample mean is zero, so the out-of-sample Sharpe is zero and the fraction kept is zero. The out-of-sample variance is 0.179, next to an in-sample variance of 0.181\. The damage is the mean. A 50 percent stamp on 6.58 leaves a Sharpe of 3.29 to trade. The formula leaves zero. Two caveats sit on the 6.58\. Proposition 3.1 drops the intercept. The commodity regression puts an intercept inside the 37 columns, so the variance terms shift once a column of ones is in the design, and 6.58 is the no-intercept null at this dimension. The mean gap does not shift: in the general expression the add-on is p over T1 times the trace of the weight matrix times sigma-epsilon, and for Markowitz weights that trace equals the asset count, so the add-on is still 0.1762 per day. And 6.58 is not a decomposition of the 8.2 they later report for a simulated book with a nonzero slope. Different beta, different number. The use of 6.58 is as a floor. A design of this size prints an annualized Sharpe above 6 on empty signals. An in-sample 8 has not cleared much. ## The commodity case study prints two ten-year fractions The question is what happened when they ran the design on commodity futures rather than on a zero slope. The universe is twelve names, May 1998 through December 2023: aluminium, copper, nickel, zinc, and lead on the LME; gasoil on ICE; WTI on NYMEX; gold and silver on COMEX; coffee, cocoa, and sugar on NYBOT. Signals are the 5-day, 1-year, and 5-year Sharpe ratio of each future, centred and scaled to unit variance, plus an asset-level intercept. They fit the simulator's beta by ridge with penalty 0.1, then draw 10,000 paths. Inside each path the traded book is ordinary least squares on that path. A path is 13 years at 252 days, 10 years in sample and 3 years out. Average correlation across signals is 20 percent, across residuals 28 percent. Average excess kurtosis of the residuals is 9\. The diagonal of the fitted autoregressive matrix averages 0.88, a half-life of ln(2) over minus ln(0.88), which is 5.4 daily steps. ![Replication ratio against backtest length in years for the commodity case study, rising from about 30 percent near 10 years toward 80 percent near 120 years](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/10/article_490-commodity_backtest_length.png) The authors read this curve as 30 percent at ten years, and it is still climbing through 80 percent out near 120 years. The parameter count is what pulls the fraction down there. A panel regression, one coefficient per momentum window shared across the twelve names, cuts the free parameters from 444 to 3. The numbers they attach to that comparison do not all divide. In the simulation, where the data are drawn from the full model, they report an in-sample Sharpe of 8.2 and an out-of-sample Sharpe of 3.1 for the unrestricted book, and they call the replication ratio 38 percent. Those two Sharpes divide to 37.8 percent, so that pair checks. For the panel on the same simulated paths they report 0.9 in sample and 0.6 out of sample, and they call the ratio 70 percent. Divide 0.6 by 0.9 and you get 67 percent. On the real tape they report a panel ratio of 68 percent next to Sharpes of 0.7 and 0.5; 0.5 divided by 0.7 is 71 percent. They report an unrestricted ratio of 9 percent next to 3.6 and 0.3; 0.3 divided by 3.6 is 8.3 percent. An earlier sentence says a panel regression raises the replication ratio from 71 percent to 30 percent, which runs the wrong way, and which disagrees with both the 70-versus-38 simulation pair and the 68-versus-9 tape pair. The printed figures stay as printed. The ratios and the Sharpes do not reconcile, except the simulated unrestricted pair, and that one sentence has the direction backwards. The source may have swapped the 30 and the 71, or rounded the Sharpes past the point where the division survives. The paper also never shows the step that connects the 30 percent on the ten-year curve to the 38 percent in the simulation paragraph. Read the pair that comes with Sharpes, and keep the conflict visible. In the simulation the fat model has the higher out-of-sample Sharpe, 3.1 against 0.6, because the paths were drawn from the fat model. The panel keeps a larger fraction of a smaller backtest. On the real tape the panel has the higher out-of-sample Sharpe, 0.5 against 0.3, and the unrestricted in-sample Sharpe of 3.6 falls to 0.3\. The fraction kept and the level you earn are different questions. Keeping 70 percent of 0.9 leaves a smaller book than keeping 38 percent of 8.2, which is the simulation. On the tape the unrestricted 3.6 falls to 0.3\. Prefer the panel when the tape says the extra coefficients are empty. Prefer it for the out-of-sample level, not for the prettier fraction. The Gaussian, independent-signal simulation lands on the analytical values. They say the match is tight. Replacing the shocks with a t-distribution and the signals with an autoregression moves both Sharpes down and leaves the fraction close to the Gaussian number. Persistence has to get extreme before the independent-signal formula misleads: they put the gap against the independent case as small until the autoregressive coefficient exceeds 0.99\. A coefficient of 0.9, which they describe as a half-life of 6.5 daily steps, is 6.6 steps if you compute ln(2) over minus ln(0.9). The fitted commodity diagonals, at 0.88, sit in the region where the formula still tracked. ## Ten signals sit below the Kan, Wang, and Zheng curve The question is how much of this is new relative to the constant-mean result. Kan, Wang, and Zheng have a closed form for the expected in-sample and out-of-sample Sharpe of a Markowitz book when the means and the covariance do not move, and the only thing you estimate is the mean vector. In this paper's language that is one signal per asset, and the signal never decays. Jacquier, Muhle-Karbe, and Mulligan add signals that die. At 2,520 observations in sample and the same count out of sample, with the true Sharpe fixed at 1.5, their one-signal curve lies on top of Kan, Wang, and Zheng as the asset count rises. Their ten-signal curve lies below it. ![Replication ratio against the number of assets at 2,520 observations and a true Sharpe of 1.5, for Kan, Wang, and Zheng, for this paper with one signal, and for this paper with ten signals](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/10/article_490-replication_vs_assets.png) The true Sharpe is held fixed, so each extra signal is weaker. A new signal has to raise the true Sharpe by enough to pay for the drop in the fraction. Muller, in Quantitative Finance in 2001, said he would rather hold one strategy with a Sharpe ratio of 2 than stitch five uncorrelated Sharpe-1 strategies into a blend with a Sharpe ratio of 2.5\. The one-signal row of the table above is that preference with a denominator: a true Sharpe of 2 on ten years keeps 98 percent. Five weak signals are a larger p, and the fraction falls. The same moral shows up on the equity-premium side of the old article ["Does Complexity Actually Help? The Virtue-of-Complexity Autopsy"](https://aligrithm.com/does-complexity-actually-help-the-virtue-of-complexity-autopsy/). There, a 15-predictor ridge beat a 12,000-feature timing model because the sample could not carry the coefficients. Here the bill is the Sharpe ratio itself, explicit in p, m, and T1\. Kelly, Malamud, and Zhou are also the normalization this paper copies when it touches the Goyal, Welch, and Zafirov tape: returns divided by a trailing 12-month volatility, signals divided by an expanding volatility. ## Where the formula stops The question is which of your books this expression is allowed to tax. A few bounds are in the paper, and a few are outside it. They drop a covariance term from the in-sample variance. In the commodity simulation that term was negative, and the simulated variance ran about 3 percent below the formula with the term set to zero. Across a wider draw of parameters the gap stays under 3 percent when the signal count over the training length is below 0.1, which is ten rows per signal. Thirty-seven signals on 2,520 days is 68 rows per signal, inside the bound. Thirty-seven signals on 252 days is 6.8 rows per signal, outside it. Do not quote the formula on a one-year fit of this design. Estimating the residual covariance, instead of plugging in the true one, matches the known-covariance fraction at 12 assets and thousands of days. At 500 assets and a window shorter than the asset count, the sample covariance is singular and the out-of-sample Sharpe falls to about zero until the window exceeds the asset count. Their fix, once the asset count over the effective sample exceeds one, puts weight one over that ratio on the sample covariance and parks the rest on a scaled identity that preserves the trace. The slope error dominates the covariance error on the commodity book. On a 500-name book with a short window, the covariance error takes over. The equity-premium check uses the Goyal, Welch, and Zafirov monthly predictors: 39 series with coverage somewhere in 1926 to 2024, random subsets of size 1 to 39, random windows of 10 to 70 years with both legs at least five years, five million draws. For each draw they pick a single scale k on the slope vector so that the analytical out-of-sample Sharpe matches the realized out-of-sample Sharpe. That match is by construction. The content of the test is whether the in-sample side, and therefore the fraction, lines up after you average across draws. They report that it does. More signals, lower fraction. Longer window, higher fraction. Higher implied true Sharpe, higher fraction. Read that figure as a check on the in-sample formula after the out-of-sample level has been pinned. Next year's Sharpe was an input to k. Nothing here pays a spread, a borrow fee, or a missed fill. The old article ["The Backtest Integrity Checklist"](https://aligrithm.com/the-backtest-integrity-checklist/) is the place those charges get applied, and a 30 percent statistical fraction is the input to that checklist, not the output. Nothing here prices the trials you threw away. The derivation is one regression, fit once. If you searched a stack of signal sets and kept the winner, this fraction is the damage from fitting that one winner, and the old article ["CSCV: A Direct Probability of Backtest Overfit"](https://aligrithm.com/cscv-a-direct-probability-of-backtest-overfit/) is the damage from the search. Stack them. Do not substitute one for the other. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/10/article_490-visual-0.png) ## KEY POINTS - The fraction of in-sample Sharpe a linear book keeps out of sample has a closed form. It falls as you add signals or assets, and it rises with the training length and with the true Sharpe. The commodity case study, as the authors read their curve, keeps 30 percent at ten years. - The mean gap is p times m over T1\. For 12 assets, 37 signals, and 2,520 days that add-on is 0.1762 per day in Markowitz units, at any true slope. Out of sample the add-on is zero. - One asset, one signal, true annualized Sharpe 0.50: after one year the formula keeps 0.202 (in-sample 1.104, out-of-sample 0.223); after ten years it keeps 0.716 (in-sample 0.590, out-of-sample 0.423). A true Sharpe of 2 on ten years keeps 0.978\. The 50 percent rule is the cell at one year and a true Sharpe of 0.983. - With the intercept left out, 12 assets and 37 signals on 2,520 days and a zero slope print an in-sample Sharpe of 6.58 annualized and an out-of-sample Sharpe of 0\. A 50 percent stamp would trade 3.29 of that. The commodity regression includes an intercept, so 6.58 is the no-intercept null, not a split of their simulated 8.2. - In the simulation drawn from the full model they report unrestricted Sharpes of 8.2 in sample and 3.1 out of sample, ratio stated as 38 percent, and 3.1 divided by 8.2 is 37.8 percent. The panel pair 0.9 and 0.6 does not divide to the stated 70 percent (it divides to 67). On the real tape, 0.5/0.7 is 71 percent against a stated 68, and 0.3/3.6 is 8.3 percent against a stated 9\. One sentence runs the panel effect backwards, from 71 percent to 30 percent. The ten-year curve's 30 percent and the simulation's 38 percent are also unreconciled in the paper. - On the real tape the panel, 3 coefficients, has the higher out-of-sample Sharpe (0.5 against 0.3). In the simulation the fat model has the higher out-of-sample Sharpe (3.1 against 0.6), because the paths were drawn from it. Keep the fraction and the level as separate questions. - The dropped variance term is about 3 percent in their commodity check, and under 3 percent when you have at least ten rows per signal. Covariance error is secondary at 12 names and dominant at 500 names on a window shorter than the cross-section. The equity-premium figure pins the out-of-sample Sharpe by construction and then checks the in-sample side. No spread is charged. Selection across many trials is a separate bill. ## References - [Backtesting (Harvey, Liu, 2015)](https://doi.org/10.2139/ssrn.2345489?ref=aligrithm.com) - [In-Sample and Out-of-Sample Sharpe Ratios of Multi-Factor Asset Pricing Models (Kan, Wang, Zheng, 2024)](https://doi.org/10.1016/j.jfineco.2024.103837?ref=aligrithm.com) - [Dynamic Trading with Predictable Returns and Transaction Costs (Garleanu, Pedersen, 2013)](https://doi.org/10.1111/jofi.12080?ref=aligrithm.com) - [A Comprehensive 2022 Look at the Empirical Performance of Equity Premium Prediction (Goyal, Welch, Zafirov, 2024)](https://doi.org/10.1093/rfs/hhae044?ref=aligrithm.com) - [The Virtue of Complexity in Return Prediction (Kelly, Malamud, Zhou, 2024)](https://doi.org/10.1111/jofi.13298?ref=aligrithm.com) - [Proprietary Trading: Truth and Fiction (Muller, 2001)](https://doi.org/10.1080/713665546?ref=aligrithm.com) - [In-Sample and Out-of-Sample Sharpe Ratios for Linear Predictive Models (Jacquier, Muhle-Karbe, Mulligan, 2025)](https://arxiv.org/abs/2501.03938?ref=aligrithm.com) --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ### 5.51 Better Inputs Beat Deeper Nets: Crypto LOB Direction URL: https://aligrithm.com/better-inputs-beat-deeper-nets-crypto-lob-direction/ Last updated: 2026-10-16T08:09:02.000Z Wang reports logistic regression at 0.7284 binary accuracy on the next 500 milliseconds of the Bybit BTC/USDT limit order book, XGBoost, a gradient-boosted tree, at 0.7281, and DeepLOB, the convolutional network with a memory layer from Zhang, Zohren, and Roberts, at 0.7189\. All three use a 40-level book smoothed with a Savitzky-Golay filter, a local cubic average. Under a Kalman filter, a causal running estimate, that same DeepLOB score is 0.4825, below a coin flip and below the raw-book score of 0.6333\. His largest printed test, 148,340 rows at five levels, scores 0.5732 for XGBoost. The date is 2025-01-30, the clock is 100 milliseconds, and the smoother, as written, uses prices up to one second ahead. ## What this actually is You quote BTC/USDT on Bybit. A fresh book arrives every 100 milliseconds, and a one-lot quote can show up and vanish inside that gap, so the mid-price prints a move with no trade behind it. You want a single bit, up or down, over the next 500 milliseconds to the next second, so you can lean a bid or an offer before that flicker dies. Wang smooths the snapshots and has a linear classifier, a tree model, and several neural networks call that same bit. Picture a harbor radio bolted to the wall beside a pile driver. Extra circuitry in the receiver still picks up the hammer. Shutting the window removes more of the hammer than any new circuit does. His smoother is that window, and it faces forward: the value dated now mixes in prices from up to one second ahead, the same second he scores as a forecast. The Kalman filter faces backward and loses to the unsmoothed book on his table. If a backward-looking smooth still left the linear classifier level with the deep network on a second day, you would spend the research week on the smoother, the up/down cutoff, and the rule for dropping thin books, and you would ship the linear score as a quoting skew. Crossing the spread needs the size of the move, and he publishes the sign. Taking the published 0.7284 as a live hit rate means leaning on a feature that can see its own label, scored on the snapshots that still had 40 price levels, from a file he dates to one day. ## Four inputs you can compute from one book The question is what a logistic model can see before anyone stacks a hidden layer. Kercheval and Zhang ran hand-built book features of this kind through a linear classifier on equity limit-order books. Wang builds four inputs from the top of the book. The mid is the average of the best ask and the best bid. The top-of-book imbalance is the bid size minus the ask size, divided by the sum of the two sizes. The five-level version does the same with the summed sizes. The weighted mid change is a blend of recent mid changes at the first three depths, with weights proportional to one over the depth index and scaled so they sum to one. $$ m\_{t-1} = \\frac{a\_{t-1}^{1} + b\_{t-1}^{1}}{2} $$ $$ I\_t^{1} = \\frac{Q\_{t,1}^{b} - Q\_{t,1}^{a}}{Q\_{t,1}^{b} + Q\_{t,1}^{a}}, \\qquad I\_t^{5} = \\frac{\\sum\_{i=1}^{5} Q\_{t,i}^{b} - \\sum\_{i=1}^{5} Q\_{t,i}^{a}}{\\sum\_{i=1}^{5} Q\_{t,i}^{b} + \\sum\_{i=1}^{5} Q\_{t,i}^{a}} $$ $$ \\Delta m\_t^{w} = \\sum\_{i=1}^{3} w\_i \\left(m\_t^{i} - m\_{t-1}^{i}\\right), \\qquad w\_i \\propto \\frac{1}{i}, \\qquad \\sum\_{i=1}^{3} w\_i = 1 $$ Read it as: a and b are ask and bid prices in USDT, Q is size in BTC, m is a mid in USDT, I is a dimensionless ratio between minus one and plus one, and delta-m is a USDT change. The superscripts and the index i are depth, starting at the touch. Wang writes the level mid inside the weighted change without a separate definition; the reading here is the mid at that depth. Worked snapshot, invented so the arithmetic is visible, in the same 103,700 USDT neighborhood as his filter chart. Best ask 103,710, best bid 103,700, so the mid is 103,705 USDT. Touch sizes 12 BTC bid and 4 BTC ask: imbalance is (12 minus 4) over (12 plus 4), which is 8/16, so 0.50\. Five levels of bids 12, 8, 5, 3, 2 sum to 30 BTC. Five levels of asks 4, 6, 8, 10, 12 sum to 40 BTC. Five-level imbalance is (30 minus 40) over 70, which is minus 1/7, about minus 0.143\. The touch says the bid is heavy. The next four levels say the offer is stacked behind it. Both features belong in the model because they disagree on this book. Weights proportional to 1, 1/2, and 1/3, then normalized, are 6/11, 3/11, and 2/11\. Suppose the three depth mids changed by plus 1.0, plus 0.4, and minus 0.2 USDT. The weighted change is (6/11)(1) plus (3/11)(0.4) plus (2/11)(minus 0.2), which is 6.8/11, or 0.618 USDT. His introduction sends these inputs to logistic regression and to XGBoost. His methods section then says the XGBoost run flattens the whole snapshot, T past snapshots by F book columns, into one long vector. The tables never say which of those two recipes produced a cell. He does not reconcile them. The old article ["Reconstructing the Order Book from Incremental Deltas"](https://aligrithm.com/reconstructing-the-order-book-from-incremental-deltas/) is the step this file skips. Bybit hands him a snapshot, top 200 bids and asks, with a NaN where a level is missing. He drops any row that lacks the depth he asked for. A missing level 40 is often a thin book, and deleting the row is a sample choice, which shows up later as a test set of 5,442 instead of 20,000. ## The smooth value dated now uses the next second The question is whether the accuracy jump is a filter a live book can run. Before the model, he runs one of two smoothers over each series. The Savitzky-Golay smoother, Savitzky and Golay's 1964 local-polynomial fit, places a cubic on a centered window of 21 snapshots, j from minus 10 to plus 10, by least squares, then replaces the window with a fixed weighted average. At a 100-millisecond clock, 10 snapshots is 1.0 second. $$ \\min\_{a\_0,a\_1,a\_2,a\_3} \\sum\_{j=-10}^{10} \\left\[ v\_{t+j} - (a\_0 + a\_1 j + a\_2 j^2 + a\_3 j^3) \\right\]^2 $$ $$ \\hat{v}\_t = \\sum\_{j=-10}^{10} c\_j \\, v\_{t+j} $$ Read it as: v is the raw series in USDT (a mid, or a size, or a price), j is a snapshot offset with no unit other than "100 milliseconds each," a0 through a3 are the cubic's coefficients, c sub j is the dimensionless weight you get once that least-squares problem is solved for this fixed window, and v-hat is the smoothed value in the same unit as v. The weights sum to 1\. They are symmetric. Wang's equation (6) is the normal equation behind c; the convolution is the object you would code. Solving that window gives a center weight of 0.1076\. The weight one snapshot ahead, 100 milliseconds, is 0.1059, almost the same as the weight on now. The five future snapshots inside a 500-millisecond label carry a combined weight of 0.4479\. The current print carries 0.1076\. The smoothed "price" dated now is more the next half-second than it is the print in front of you. The weight at plus 10 snapshots is minus 0.0559, a negative lobe at the rim of the window, so the leak is signed, and a model can flip it. ![Savitzky-Golay weights for a cubic fit on 21 snapshots. Blue bars are the current snapshot and the past. Orange bars are the next 1.0 second. The center weight is about 0.11. The orange bars over the first half-second are about as tall as the center, and the bars at plus and minus 10 snapshots go negative.](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/10/article_460-sg_window_weights.png) Worked check. Twenty-one prints, all 103,700 USDT, smooth to 103,700 because the weights sum to 1\. Lift only the snapshot 100 milliseconds ahead by 10 USDT, to 103,710\. The smoothed value dated now becomes 103,700 plus 10 times 0.1059, which is 103,701.06 USDT. Lift only the snapshot 1.0 second ahead by that same 10 USDT and the smoothed value dated now becomes 103,700 plus 10 times minus 0.0559, which is 103,699.44 USDT. A 500-millisecond label is the sign of the mid five snapshots ahead. Those five snapshots are inside the feature. A 1,000-millisecond label's endpoint is the negative lobe. He publishes no lagged window, the causal version that uses j from minus 20 to 0 and dates the result at the right edge. That lag is 1.0 second. On a 1-second forecast the lagged smooth arrives as the horizon expires. The Kalman filter in his equations (8) through (10) is causal. A latent price does a random walk with process variance Q. You observe it plus noise of variance R. The gain, the updated estimate, and the updated variance are a recursion that uses the current print and the previous estimate. $$ K\_t = \\frac{P\_{t-1} + Q}{P\_{t-1} + Q + R} $$ $$ \\hat{x}\_t = \\hat{x}\_{t-1} + K\_t \\left(v\_t - \\hat{x}\_{t-1}\\right), \\qquad P\_t = (1 - K\_t)\\,(P\_{t-1} + Q) $$ Read it as: x-hat is the estimated price in USDT, v is the observed print in USDT, Q and R and P are variances in USDT squared, and K is a dimensionless gain between 0 and 1\. K is the fraction of the surprise you accept. Large R, a print you have declared noisy, keeps K small and the estimate glued to its past. He does not print Q or R. Illustration of the update, not his fit: Q = 0.25, R = 1, previous variance P = 0.25, all in USDT squared. The gain is (0.25 plus 0.25) over (0.25 plus 0.25 plus 1), which is 0.5/1.5, or 1/3\. Previous estimate 103,700, new mid 103,706\. The update is 103,700 plus (1/3) times 6, which is 103,702 USDT. The new variance is (2/3) times 0.5, which is 1/3\. Two-thirds of a 6 USDT jump is thrown away. Do that on every real move and the series lags the book the forecast is aimed at. He says the variances were grid-searched on a small sample and then frozen. The text points this filter at "Wu et al., 2011," and the bibliography's Wu paper is a 2021 note on perturbed books. The recursion above is what he wrote down. ![Bybit mid-price over 2,000 snapshots of 100 milliseconds, 200 seconds. The raw mid is the jagged gray line, between about 103,680 and 103,780 USDT. The Kalman filter in red and the Savitzky-Golay filter in green lie on top of each other.](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/10/article_460-midprice_filters.png) On that 200-second slice the two smoothers are hard to tell apart by eye. The accuracy table is where they split. Visual smoothness is a bad referee. ## Up, down, and a flat band he never sizes The question is what the hit rate is measured against. Binary labels are up or down. Ternary labels add a flat class inside a band of half-width epsilon around a zero return. He says he tunes epsilon, following Sirignano and Cont, so the three classes have about equal counts, and he reweights the loss by inverse frequency. He never prints epsilon, and the horizon tables never print the class counts. A balanced coin flip is 0.50 on the binary task and 1/3 on the ternary task. At the 500-millisecond, 40-level, Savitzky-Golay cell, logistic regression's 0.7284 sits 0.2284 above 0.50\. Its ternary score of 0.5434 sits 0.2101 above 1/3\. The ternary number looks worse and the lift over a dumb baseline is in the same range. At 100 milliseconds on a 5-level book the binary scores sit between 0.5271 and 0.5338, a lift of about three points over 0.50, and the best ternary score is DeepLOB at 0.4281, a lift of 0.0948 over 1/3\. Short horizon, thin book, small edge. The 0.73 does not live there. He quotes logistic regression at 48 to 64 percent on the FI-2010 equity benchmark, depending on horizon. That range is his literature summary, tied to the Ntakaris dataset paper. It is a different market, a different clock, and a vendor-cleaned book. Read 0.73 against that yardstick with the sample attached: one date, and the smoother described above. ## Eight points from the filter, one point from the network Hold the filter fixed and the remaining question is the gap across models. Same 80,000 training rows and a nominal 20,000-row test, one snapshot of history (his T equals 1), six models. DeepLOB here is three convolutional blocks and an LSTM of 64 units, a recurrent layer that carries a state across snapshots. The simpler net is one convolutional layer and a bidirectional LSTM of 32 units. CNN plus CatBoost, which the table header shortens to CatBoost, is the Bileki pipeline: two convolutional layers, a 64-dimensional embedding, then a boosted tree. XGBoost is the tree on its own. Logistic regression is the linear score. ![Binary accuracy on Bybit BTC/USDT from Wang's Table 2. Left panel, next 500 milliseconds. Right panel, next 1,000 milliseconds. Both are a 40-level book and T equals 1. Blue is the raw book, orange is the Kalman filter, green is Savitzky-Golay. The dashed line is 0.50.](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/10/article_460-binary_filter_accuracy.png) At 500 milliseconds, Savitzky-Golay, the order is logistic regression 0.7284, XGBoost 0.7281, DeepLOB 0.7189, the simpler net 0.7189, CNN plus XGBoost 0.7130, CNN plus CatBoost 0.6260\. Logistic regression minus DeepLOB is 0.0095\. Logistic regression minus CNN plus CatBoost is 0.1024\. The plain tree and the linear score tie. The convolutional embedding handed to CatBoost gives back ten points against the tree that saw the snapshot without that embedding. The filter moves the score by more than the model does. Logistic regression goes from 0.6517 on the raw book to 0.7284 after Savitzky-Golay, a gain of 0.0767, and to 0.4882 under Kalman, a drop of 0.1635\. DeepLOB goes from 0.6333 raw to 0.7189 smoothed, a gain of 0.0856, and to 0.4825 under Kalman, a drop of 0.1508\. XGBoost on the same Kalman book is 0.6301 against a raw 0.6542, a drop of 0.0241\. Every binary cell and every ternary cell in the three horizon blocks is lower under Kalman than on the raw book. The damage is a couple of points for the plain tree and fifteen points for the linear model and for DeepLOB. On the raw 500-millisecond book, before any smoother, XGBoost at 0.6542 and logistic regression at 0.6517 already lead DeepLOB at 0.6333\. Smoothing lifts the net by more than it lifts the logit, and the logit is still ahead. At 1,000 milliseconds the smoothed order is XGBoost 0.7150, logistic regression 0.7089, DeepLOB 0.6947, CNN plus CatBoost 0.6140\. XGBoost minus DeepLOB is 0.0203\. He calls this a 1 to 2 percent edge for the simpler models. Read it as 1 to 2 percentage points, and read it against the nets, because CNN plus CatBoost is ten points back (0.7150 minus 0.6140 is 0.1010). The worst Kalman cell in the binary table is the simpler net at 1,000 milliseconds, 0.4680, against 0.6372 on the raw book. The 0.0095 gap needs a noise scale. He prints two candidate test sizes and attaches neither to the 0.7284 cell. The 0.7150 cell, which is the smoothed XGBoost score at 1,000 milliseconds, is the one he later pairs with a support of 5,442. $$ SE = \\sqrt{\\frac{p\\,(1-p)}{n}} $$ Read it as: p is an accuracy between 0 and 1, n is the number of test snapshots, and SE is the sampling wobble of that accuracy in the same units. Worked at the headline accuracy, borrowing the 5,442-row support from the sister cell: p = 0.7284, so p times (1 minus p) is 0.19783, divided by 5,442 is 0.00003635, and the square root is 0.00603\. The 0.0095 gap is 1.6 of those wobbles. Borrow the 20,000-row test he names in the section header and the wobble falls to 0.00315, so the same gap is 3.0 wobbles. The ten-point gap against CNN plus CatBoost survives either sample size. The one-point gap against DeepLOB is small if the test is the 5,442-row book. Two confounders sit in the table headers. The 100-millisecond block is a 5-level book. The 500- and 1,000-millisecond blocks are 40-level books. A jump from 0.53 to 0.73 is a jump in horizon and a jump in depth at the same time. He calls the cells accuracy in the prose. In the paper, the section before the tables defines the F1 score, the harmonic mean of precision and recall, and the table header names no unit. The 0.7150 match with the later accuracy column is why these cells are accuracies. ## A full book is a smaller sample The question is whether 40 levels predict better, or whether a full book is an easier hour. He keeps the Savitzky-Golay smooth and XGBoost, drops any snapshot that lacks the k-th level, and lets k run from 40 down to 5\. The accuracy he dissects here is 0.7150, the 1,000-millisecond cell, not the 0.7284 maximum of the binary table. At 40 levels the accuracy is 0.7150, the two class F1 scores are 0.7115 and 0.7184, and the test support is 5,442\. The average of those two F1 scores is 0.71495, which matches 0.7150 to rounding, the pattern of near-equal class counts. At 10 levels the accuracy is 0.5837 on 17,471 rows. At 5 levels it is 0.5797 on 18,336 rows. From 40 levels to 10, accuracy drops by 0.1313 and the test set more than triples. From 10 levels to 5, accuracy drops by 0.0040, inside the 0.006 wobble above, and the test set grows by 865 rows. He says he trains on 80,000 and tests on 20,000\. Against that 20,000, a 40-level requirement keeps 5,442 rows, 27.2 percent, and a 5-level requirement keeps 18,336, 91.7 percent. Levels 11 through 40 are where his number moves. He does not separate "those levels carry signal" from "hours with a full 40-level book are easier." A quoter who abstains when level 40 is empty posts 0.715 and trades 27 percent of the snapshots. A quoter who always trades posts something near 0.58\. Those are different jobs, and the second one is the one a 24-hour book will hand you. The old article ["An Open L4 Order Book: What Hyperliquid Data Unlocks"](https://aligrithm.com/an-open-l4-order-book-what-hyperliquid-data-unlocks/) is the feed that would label the flicker instead of smoothing it. A 100-millisecond snapshot cannot show a cancel that lived 30 milliseconds. It shows a mid that twitched. He reaches for a smoother because the file stored pictures, not the cancels between them. ## Two points for ten snapshots The question is what a longer lookback buys once the model is already simple. He switches to 1,000,000 snapshots at five levels, Savitzky-Golay, and predicts the mid change over the next 1,000 milliseconds from either one snapshot or ten. Ten snapshots cover 0.9 seconds from the first to the last; he calls that a second. XGBoost at T equals 1 scores 0.5732, with class F1 scores 0.5757 and 0.5707 whose average is the accuracy, on supports 73,953 and 74,387\. That sum is 148,340 rows, 74.2 percent of a 200,000-row test, which he calls 75 percent. At T equals 10 the score is 0.5949\. Logistic regression goes from 0.5551 to 0.5716\. Gains of 0.0217 and 0.0165\. He calls that about a 2 percent accuracy gain. Training time on an M2 laptop, for the million rows, goes from 1 minute 36 seconds to 7 minutes 4 seconds for XGBoost (4.4 times) and from 1 minute 11 seconds to 9 minutes 13 seconds for logistic regression (7.8 times). His sentence places the T-equals-10 label one second after the sequence starts. A causal window of ten snapshots ends 0.9 seconds after it starts, so the label endpoint is 100 milliseconds past the last input. If the window is aligned the other way, most of the label path sits inside the input. He does not draw the alignment. A two-point gain is small for a feature that contained the answer, and it is the sort of gain extra history often produces. The million-row experiment is also the one that stays near 0.57, in line with the 0.5797 he got at five levels on the shorter file. The 0.73 does not reappear once the book is five levels deep and the test has 148,340 rows. ## One date, no markout, no causal rerun The question is what can go to a live quote. Section 3 says the study is the single day 2025-01-30\. The conclusion says the main tables use about 100,000 snapshots, which at 100 milliseconds is 10,000 seconds, 166.7 minutes, and he writes about 166 minutes. That is a slice of the day, 11.6 percent of the 864,000 slots in 24 hours, if the grid is full. The million-row run is 100,000 seconds, 27.8 hours, and he writes about 28 hours. One UTC day cannot hold it. Either the file crosses midnight, or the grid has gaps and the "28 hours" is a count of slots rather than a clock, or the single-day sentence is too strong. He writes both. Neither sentence repairs the other. He also never says the 80/20 split is chronological. "The first 100,000 snapshots" is a time prefix. The split inside it could be shuffled. On a 100-millisecond book a shuffled row shares its neighbors with the other side. Add the centered smoother, which already mixes ten future snapshots into every row, and a random split is a second leak on top of a first. Undescribed, so unproven. The centered window is proven from the summation limits. The abstract says the models are compared on latency. The only clock in the results is training time for two models on one laptop. The remark about a network that hits 80 percent and takes two seconds to forecast the next one second is a hypothetical. His best printed cell is 0.7284, and he does not time the forward pass. He prints no PnL, no spread, no fee, and no markout. A markout is the mid change over a fixed horizon after a fill, the number that says whether the fills were toxic. A sign hit rate has no PnL until the move clears the spread, and he never prints the move. Use the bit as a skew on a quote that was already going out. It becomes a trade after the markout clears the fee. He stops at the bit. The old article ["Does Complexity Actually Help? The Virtue-of-Complexity Autopsy"](https://aligrithm.com/does-complexity-actually-help-the-virtue-of-complexity-autopsy/) found the same shape on equity returns: once the comparison was specified, a plain ridge sat ahead of the high-order feature model. Here a linear score sits ahead of DeepLOB by a point, on a book that was smoothed with a window into its own label, on one date. The part worth keeping is the ranking of the filters. The 0.73 cell is a 40-level book on the short file. The large five-level test scores 0.57\. The part to rerun, before anyone skews a live BTC quote, is a backward-looking window, a printed epsilon, a chronological split, and a second day. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/10/article_460-visual-0.png) ## KEY POINTS - On Bybit BTC/USDT, 100-millisecond snapshots, Wang's best binary cell is logistic regression at 0.7284 for the next 500 milliseconds, on a Savitzky-Golay smoothed 40-level book. XGBoost is 0.7281\. DeepLOB is 0.7189\. The gap over DeepLOB is about one percentage point. - The same DeepLOB score is 0.6333 on the raw book and 0.4825 under his Kalman filter. Logistic regression falls from 0.6517 raw to 0.4882 under Kalman. Every binary cell and every ternary cell in the horizon tables is worse under Kalman than on the raw book. XGBoost loses only 0.024 on that 500-millisecond Kalman swap. - The Savitzky-Golay window he specifies is a centered cubic on 21 snapshots. The five future snapshots inside a 500-millisecond label carry combined weight 0.4479\. The current print carries 0.1076\. He publishes no lagged, causal rerun. The Kalman recursion he writes is causal, and it loses to the raw book. - Requiring 40 complete levels leaves 5,442 test rows at accuracy 0.7150\. Five levels leaves 18,336 rows at 0.5797\. Almost all of that drop happens between 40 and 10 levels (0.7150 to 0.5837). The 100-millisecond block is a 5-level book, so the horizon table mixes horizon with depth. - The only experiment with 148,340 printed test rows is the five-level run: XGBoost 0.5732 at one snapshot and 0.5949 at ten, logistic regression 0.5551 and 0.5716\. Training those on an M2 takes from 1 minute 11 seconds to 9 minutes 13 seconds. That is training time, not a forward-pass latency, and he never reports a PnL, a spread, or a markout. - The main tables are about 100,000 snapshots, 166.7 minutes of 2025-01-30\. The million-row run is 27.8 hours and does not fit in that single date as he also states it. The 80/20 split is not described as chronological. Treat 0.7284 as an upper bound from a smoother that can see its label, not as a live quoting edge. ## References - [Modelling high-frequency limit order book dynamics with support vector machines, Kercheval and Zhang (2015)](https://doi.org/10.1080/14697688.2015.1032546?ref=aligrithm.com) - [Benchmark dataset for mid-price forecasting of limit order book data with machine learning methods, Ntakaris, Magris, Kanniainen, Gabbouj, and Iosifidis (2018)](https://doi.org/10.1002/for.2543?ref=aligrithm.com) - [DeepLOB: Deep Convolutional Neural Networks for Limit Order Books, Zhang, Zohren, and Roberts (2019)](https://doi.org/10.1109/TSP.2019.2907260?ref=aligrithm.com) - [Universal features of price formation in financial markets: perspectives from deep learning, Sirignano and Cont (2019)](https://doi.org/10.1080/14697688.2019.1622295?ref=aligrithm.com) - [Smoothing and Differentiation of Data by Simplified Least Squares Procedures, Savitzky and Golay (1964)](https://doi.org/10.1021/ac60214a047?ref=aligrithm.com) - [Order book mid-price movement inference by CatBoost classifier from convolutional feature maps, Bileki, Barboza, Silva, and Bonato (2022)](https://doi.org/10.1016/j.asoc.2021.108274?ref=aligrithm.com) - [Exploring Microstructural Dynamics in Cryptocurrency Limit Order Books: Better Inputs Matter More Than Stacking Another Hidden Layer, Haochuan (Kevin) Wang](https://arxiv.org/abs/2506.05764?ref=aligrithm.com) --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ### 10.24 Learning to Rank, the Original: Pairwise and Listwise Cross-Sectional Momentum URL: https://aligrithm.com/learning-to-rank-the-original-pairwise-and-listwise-cross-sectional-momentum/ Last updated: 2026-10-16T08:09:03.000Z You rebalance a long-short of NYSE names once a month, and you keep the top 100 and the bottom 100\. The score's level never enters the book. A loss that matches next month's return, one name at a time, can land on the same average error and still hand you the opposite two names. Pairwise and listwise losses train the order the book actually trades. Poh, Lim, Zohren, and Roberts run that comparison from 1980 to 2019, volatility-scaled near a 15% target, with no costs. ## Squared error can swap the book The regress-then-sort network fits a score to next month's return divided by a trailing volatility, one name at a time. Their equation (11) is the mean squared gap. $$ \\mathcal{L} = \\frac{1}{M}\\sum\_{i}\\left(Y\_{i} - \\frac{r\_{i}}{\\sigma\_{i}}\\right)^{2} $$ Y is the score you are fitting. r is next month's return. Sigma is a volatility computed from past returns, so the target is in units of recent volatility. M is the number of name-months in the sum. Take two names whose targets are 0.30 and 0.10\. Forecast 0.10 and 0.30\. The errors are -0.20 and +0.20, and the mean squared error is 0.04\. Forecast 0.50 and 0.30\. The errors are +0.20 and +0.20, and the mean squared error is 0.04 again. The first forecast ranks the worse name above the better one. The second keeps the order and misses the level by the same amount on both. You cannot tell those two forecasts apart from the loss. You then throw the level away and keep the top of the list, the same percentile cut as the old article ["Cross-Sectional Percentile Rank Within a Universe."](https://aligrithm.com/cross-sectional-percentile-rank-within-a-universe/) The old article ["Ranking Beats Forecasting for Many Trading Problems"](https://aligrithm.com/ranking-beats-forecasting-for-many-trading-problems/) is this objection. On this sample the network, fit by that loss and then sorted, loses to a raw trailing-return sort. The rankers change the label as well as the loss. At each month the label is the next rebalance's return decile, from 1 to 10, and a higher decile is defined to outrank a lower one. They never publish the run that changes only the loss and keeps the return as the label. ## A pair has a winner RankNet trains on every pair. The probability that name i outranks name j is a logistic of the score gap, and the loss is the cross-entropy of that probability against a label that is 1 when i's decile is the higher one. The paper describes this and does not print it. $$ P\_{ij} = \\frac{1}{1+\\exp(-(s\_i-s\_j))}, \\qquad C\_{ij} = -\\bar{y}\_{ij}\\log P\_{ij}-(1-\\bar{y}\_{ij})\\log(1-P\_{ij}) $$ You compute that on one pair. Scores of 1 and 0, with the higher score on the name that should win, give a probability of 0.73 and a loss of 0.31\. Reverse the scores and the probability is 0.27 and the loss is 1.31, higher by exactly 1\. The squared-error sum has no pair in it, so it never charges that extra 1\. The cost of the method is the count. Training walks the pairs, and the work grows with the square of the number of names at the rebalance. ## The whole list is one draw ListNet drops the pairs. It turns the scores into the probability each name is ranked first, does the same for the labels, and charges the cross-entropy between those two distributions. A softmax does the turning. The paper describes this and does not print it. $$ P\_{j}(s)=\\frac{\\exp(s\_{j})}{\\sum\_{k}\\exp(s\_{k})}, \\qquad L=-\\sum\_{j}P\_{j}(y)\\log P\_{j}(s) $$ Three names, labels 3, 1, and 0\. The exponentials are 20.09, 2.72, and 1, and they sum to 23.81, so the label distribution is 0.844, 0.114, 0.042\. Scores of 1, 2, and 0 put the middle name on top. Their exponentials are 2.72, 7.39, and 1, sum 11.11, distribution 0.245, 0.665, 0.090\. The cross-entropy is 1.34\. Scores that copy the labels, 3, 1, and 0, bring it down to 0.52\. Both numbers use the whole list. A pairwise loss would have scored three separate coins and never seen that the top probability has to add up with the other two. ListMLE goes after the likelihood of the observed order, best name first. The paper picks it because the likelihood is consistent, and again does not print the product. You walk down the true order. At each step the name that should come next is scored against the names still remaining. $$ L=-\\sum\_{i=1}^{n}\\left(s\_{\\pi\_{i}}-\\log\\sum\_{k=i}^{n}\\exp(s\_{\\pi\_{k}})\\right) $$ Pi is the true order. The first term is the score of the name that should be first, minus the log of the sum of exponentials of everyone still in line, including that name. Then you drop that name and repeat. Scores 2, 1, 0 already in the true order cost 0.72\. The reversed scores 0, 1, 2 cost 3.72, higher by 3\. Consistency says a model that gets the order right should be preferred as the sample grows. It does not say the likelihood wins on a noisy five-year window. ## The top of the list is worth more LambdaMART never evaluates a loss value. It boosts trees on lambda-gradients. The paper's description is a pairwise gradient multiplied by the change in normalized discounted cumulative gain, NDCG, from swapping that pair. NDCG discounts a hit by its position, and they cut it off at 100 because that is the wing they trade. The gain itself is the usual one. The paper does not print it. $$ \\mathrm{DCG}=\\sum\_{i=1}^{n}\\frac{2^{\\mathrm{rel}\_{i}}-1}{\\log\_{2}(i+1)}, \\qquad \\mathrm{NDCG}=\\frac{\\mathrm{DCG}}{\\mathrm{DCG}^{\*}} $$ rel is the relevance of the name sitting in position i, here the return decile. DCG-star is the same sum on the perfect order, so NDCG equals 1 when the list is right. Three names with relevance 3, 1, 0, already in order, have a DCG of 7.63 and an NDCG of 1\. Swap the top two and the DCG falls to 5.42, so NDCG falls to 0.71, a drop of 0.29\. Swap the bottom two instead and NDCG only falls to 0.98\. The same pair-mistake costs more at the top, which is the end of the list the long-short actually holds. The lambda multiplies the RankNet pair-gradient by that drop, and the trees are fit to the lambdas. Their booster uses the pairwise ranking objective and NDCG as the evaluation metric. ## The test, and the haircut Look at the brown line against the orange and green lines sitting near the bottom of the log scale. ![Log cumulative returns, rescaled to a 15% volatility target, for a random book, Jegadeesh-Titman, Baz's MACD, a regress-then-sort network, RankNet, LambdaMART, ListNet, and ListMLE](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/10/article_602-cumulative_returns_vol_scaled.png) The brown line is LambdaMART. It finishes between the 10^3 and 10^4 gridlines. The trailing-return sort and Baz's volatility-normalized moving-average spread finish near 10\. The red network finishes down beside the random book. The paths run out of sample from the early 1990s through 2019\. They retune every five years, 50 hyperparameter trials in each window, and freeze the model for the next five. They rebalance on the last trading day of the month, and they pin every book a second time near 15%, so a Sharpe gap is a return gap. Sharpe here is mean return over volatility, with no risk-free rate removed. LambdaMART's Sharpe is 2.16\. Baz's is 0.70\. The trailing-return sort is 0.55\. The network, on the same momentum features as the rankers, is 0.27\. Dividing 2.16 by 0.70 gives 3.1, the comparison behind their approximate threefold. ListNet, ListMLE, and RankNet print 1.97, 1.61, and 1.50\. The consistent likelihood finishes behind the booster. They put that down to sample size. A listwise model sees one list per month. A pairwise model sees that list times the number of names again. They do not print the average universe. One hundred names is about a tenth of what they call tradeable, so the universe is on the order of a thousand, about 12,000 lists a year against about 12 million pairs. In the LambdaMART panel, compare the top line with the bottom line. The network's panel, top right, stays bunched. ![Log cumulative returns of decile portfolios, rescaled to the same volatility target. Decile 10 is the top line in each panel](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/10/article_602-decile_portfolio_paths.png) Kendall's tau, the rank correlation over the whole list, averages 0.03 for the rankers and 0.01 for the network. A tau of 0.03 barely orders the full list. The wings still separate. LambdaMART's worst decile prints a Sharpe of 0.08 and its best prints 1.23\. The network's decile Sharpes run from 0.44 to 0.83 and are not monotonic. You fit a pairwise or listwise loss, and you label each name with next month's return decile. The features are raw returns over 3, 6, and 12 months, those returns divided by volatility, and volatility-normalized moving-average signals lagged out to a year. You hold the outer 100 names on each side, scale each name by 15% over a 63-day exponentially weighted volatility, then scale the book to 15%. The neural rankers train on a 21-day forward return rather than the calendar month the book holds. I am against fitting the level and sorting the fit. On this test that drawdown is 0.64, against 0.33 for the trailing-return sort, and the annual return falls from 0.36 to 0.04 at the same 15% volatility. The second mistake is trading the gross Sharpe. The test turns 200 names a month and never charges a spread. Turnover is unreported, so the paper gives you no cost to subtract. The old article ["Momentum Is a Ranking Problem: Learning-to-Rank vs Regress-then-Rank"](https://aligrithm.com/momentum-is-a-ranking-problem-learning-to-rank-vs-regress-then-rank/) runs LambdaMART on a different cross-section, and the equity Sharpe shrinks once the rebalance slows. They also never say whether a delisting return enters the P&L. CRSP here is the Center for Research in Security Prices: NYSE share codes 10 and 11, price above one dollar, a full prior year of trading. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/10/article_602-visual-0.png) ## KEY POINTS - Fit the order. The book keeps the top and bottom 100 and throws the score's level away. - Treat a squared error of 0.04 as ambiguous. One forecast with that error keeps the order, and one swaps it. - Charge the pair. Scores of 1 and 0 cost 0.31 when the order is right and 1.31 when it is flipped. - Weight the top of the list. Swapping the top pair drops NDCG from 1 to 0.71\. Swapping the bottom pair drops it to 0.98. - Haircut the winner before Monday. LambdaMART's Sharpe is 2.16 on a test that turns 200 names a month and charges no spread. ## References - [Cross-Sectional Percentile Rank Within a Universe](https://aligrithm.com/cross-sectional-percentile-rank-within-a-universe/) - [Ranking Beats Forecasting for Many Trading Problems](https://aligrithm.com/ranking-beats-forecasting-for-many-trading-problems/) - [Momentum Is a Ranking Problem: Learning-to-Rank vs Regress-then-Rank](https://aligrithm.com/momentum-is-a-ranking-problem-learning-to-rank-vs-regress-then-rank/) - [Building Cross-Sectional Systematic Strategies By Learning to Rank (Poh, Lim, Zohren, and Roberts)](https://arxiv.org/abs/2012.07149?ref=aligrithm.com) --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ### 3.40 Does Complexity Actually Help? (flagship) URL: https://aligrithm.com/does-complexity-actually-help-flagship/ Last updated: 2026-10-16T08:09:04.000Z Buy-and-hold on the volatility-scaled US market earns an out-of-sample Sharpe ratio of 0.5494 from January 1940 to December 2020\. The 12,000-feature timing model that Kelly, Malamud, and Zhou offer as evidence that performance rises with complexity earns 0.4851 on that same window. A ridge regression on the 15 Goyal-Welch predictors, with an intercept and a 120-month expanding window, earns 0.6990\. Move the same question to a neural network and a cross-section of stocks in 24 developed markets, and the global network earns 0.74 percent a month against 0.62 percent for the same network trained region by region. Complexity helps when the sample can carry the parameters. The help is a fraction, not a multiple. The rising curve in the original paper is two choices in the plot. ## What this actually is You have a monthly decision on the S&P 500, or a long-short book across developed-market stocks, and a model with thousands of fitted weights. The S&P hands you about 120 months inside a training window. The cross-section hands you thousands of stocks in 24 countries. The method is a count. Put the number of parameters next to the number of observations, put the unconditional mean back into the regression, and only then ask whether a more flexible fit beats a short ridge or ordinary least squares. Picture a tailor with twelve measurements of one client and a pattern that places 12,000 darts. The coat fits last year's posture. Next month the client has moved, and the extra darts are memorized noise. Give that tailor measurements from every client in 24 cities and a more flexible pattern starts to earn a small improvement in the fit. At the desk the claim changes the build order. Add observations before adding parameters, leave the intercept unpenalized, and budget about a fifth on the Sharpe ratio before costs. Believing the headline and trading the 12,000-feature timing book means holding a 0.49 Sharpe strategy against a 0.55 buy-and-hold, before spreads. Refusing every flexible model means skipping the 0.12 percentage points a month that the global network earns over its regional twin. The two mistakes are different sizes. ## Two choices, and the curve flattens The question is whether portfolio performance rises as the model grows more complex. On this equity-premium sample, it rises only under two restrictions, and both of them are choices the researcher makes. Kelly, Malamud, and Zhou define complexity as c, the number of predictors P divided by the training-window length T. Their headline model uses P of 12,000 and T of 12, so c equals 1,000\. The predictors are not 12,000 new series. They are random Fourier features of the same 15 Goyal-Welch variables: draw a random weight vector, take its dot product with the predictor vector, and pass that scalar through a sine and a cosine. Buncic's equation (6) is that map. Nothing new enters the information set. The parameter count grows. $$ c = \\frac{P}{T} $$ Read it as a ratio of counts. P is the number of fitted coefficients. T is the number of months in the training window. Neither is a return. Worked values from the paper: 12,000 divided by 12 is 1,000, the complexity of the model they highlight. The best random-feature model once an intercept is allowed uses P of 200 and T of 120, and 200 divided by 120 is 5/3, about 1.67. The target being forecast is the market excess return divided by a trailing 12-month volatility, the standardization in Buncic's section 2.1\. Call that series R. $$ R\_t = \\frac{xret\_t}{\\sqrt{\\frac{1}{12}\\sum\_{i=1}^{12} xret\_{t-i}^{2}}} $$ Read it as this month's excess return, in percent, divided by the root-mean-square excess return over the previous 12 months. The ratio is in units of recent volatility. Worked sketch: a month with a 2 percent excess return, after a year whose root-mean-square excess return was 4 percent, scores 2 divided by 4, which is 0.5\. The sample mean of R is 0.1590\. A typical month sits 0.16 volatility units above zero. Buncic also prints a robust standard error of 0.0336 and a t-statistic of 4.4687\. Those three figures do not multiply through: 0.1590 divided by 0.0336 is 4.73, not 4.47, and 0.0336 times 4.4687 is 0.150, not 0.1590\. One of the three printed numbers is off. The direction is not. The mean is several standard errors above zero, and not one of the 15 predictors has a mean of zero either. Forcing the regression through the origin throws that mean away. The target's skewness is -0.4052, a mild negative skew, so there is no positive asymmetry sitting in the data for a model to "learn." The position in the timing book equals the forecast, Buncic's equation (11). Forecast 0.20 and then realize a standardized excess return of 0.50, and the strategy earns 0.20 times 0.50, which is 0.10 that month. Because the position is the forecast, a restriction that shoves every forecast toward zero shoves the book toward cash. Ridge is how they estimate a model with more coefficients than months. Buncic's equation (8) adds a penalty z to the diagonal of the feature second-moment matrix. In the one-predictor case that matrix is a single number. If the feature's average square equals 1 and its average product with next month's R equals 0.2, the coefficient is 0.2 divided by (z + 1). $$ \\hat{\\beta}(z) = \\frac{0.2}{z + 1} $$ Read the sketch as equation (8) collapsed to one feature. The 0.2 is an assumed covariance, in volatility units, chosen so the arithmetic is visible. It is not a fitted number from the study. z is the ridge penalty, in the same units as the feature's average square, here 1\. Worked values: z of 2 gives 0.2 divided by 3, which is 0.067\. z of 0 gives 0.2\. z of 1,000 gives 0.2 divided by 1,001, which is 0.0002\. A penalty of 1,000 on a 15-variable regression is a sledgehammer. The expanding-window ridge that earns 0.6990 uses z of 0.4. With the intercept forced to zero, the best random-feature Sharpe ratios on the common window are 0.4888, 0.4392, and 0.4305 at training windows of 12, 60, and 120 months. The whole curve shifts down as the window lengthens. That is the virtue-of-complexity pattern, and Buncic shows it is the zero intercept. Free the intercept, and keep it out of the penalty, and the best random-feature Sharpe ratios become 0.4745, 0.5157, and 0.5863 at those same three windows. Performance rises as T rises. Since c equals P over T, a longer window is a less complex model, and the less complex model wins. The best of them is 200 features, z of 2, T of 120, complexity 5/3, Sharpe ratio 0.5863\. That beats buy-and-hold by 0.5863 minus 0.5494, about 0.037\. It loses to the 15-predictor expanding ridge at 0.6990, and to the rolling 15-predictor ridge at z of 6, which earns 0.6127. The zero intercept is not what makes the short model win. With the intercept still forced to zero, the expanding 15-predictor ridge at z of 0.4 and T of 120 earns 0.6924, and a 120-month rolling historical average, used as the timing weight, earns 0.5446\. Both clear the complex model's 0.4851\. Kelly, Malamud, and Zhou only tabulate the 15-predictor ridge at z of 0 and z of 1,000\. One of those penalties is too small for a 12-month window. The other, from the sketch above, flattens the coefficients. The old article ["Does Complexity Actually Help? The Virtue-of-Complexity Autopsy"](https://aligrithm.com/does-complexity-actually-help-the-virtue-of-complexity-autopsy/) keeps the full grid. With the intercept forced off, the best expanding 15-predictor ridge on each window (penalties 60, 10, and 0.4) beats the 12,000-feature model at penalty 1,000: 0.6214 against 0.4851, 0.6524 against 0.4273, and 0.6924 against 0.4075\. With the intercept free, the same comparison at 120 months is 0.6990 against a best random-feature Sharpe of 0.5863. ![Out-of-sample Sharpe ratios on one common window. The 12,000-feature model is estimated with a zero intercept. The random-feature and ridge bars include an intercept. January 1940 to December 2020.](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/10/article_432-sharpe_ladder.png) The second choice is the way the 1,000 random draws get scored. For the table of portfolio results, the 1,000 forecasts are averaged first, and the Sharpe ratio is computed on that average book. For the curves that look strictly increasing, each draw is scored on its own, and the 1,000 Sharpe ratios are averaged. Sharpe is a ratio. The average of ratios is not the ratio of the average. At P of 2 and T of 120, that per-draw average produces Sharpe ratios of about -0.0017 for every penalty they plot, while expected returns and alphas stay positive. A portfolio with a positive mean return does not have a negative Sharpe ratio. An average of 1,000 separate ratios can. ![Out-of-sample Sharpe ratio against complexity, with an intercept. Training windows of 12, 60, and 120 months. Annotated peaks are 0.4785, 0.5191, and 0.5836, at complexity 2, 12, and 6.](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/10/article_432-voc_intercept_sharpe.png) The figure is Buncic's comparison once the intercept is free. Solid lines average the Sharpe ratios of the 1,000 draws. Dash-dotted lines take the Sharpe ratio of the averaged forecast. The annotated peaks, 0.4785 at complexity 2, 0.5191 at complexity 12, and 0.5836 at complexity 6, are the maxima on Kelly, Malamud, and Zhou's seven penalty values. The finer grid that produced 0.5863 uses z of 2, which is not on that seven-point list, so it does not appear as a label. The shape is the same on both grids. The peak sits at low or moderate complexity. The far right of each panel, where c is largest, is flat and no higher. The 120-month panel sits above the 12-month panel. Belkin, Hsu, Ma, and Mandal's double-descent picture, which the factor-investing deck uses as the theoretical alibi for huge models, has test error falling again after the model passes the interpolation boundary at c of 1\. There is a partial version of that dip and recovery in these panels. It tops out at complexity 2, 12, or 6, and then stops. It does not keep paying at complexity 1,000. The long-only claim is the same restriction. With a zero intercept, a 12,000-feature model and a 500-feature model produce timing weights that look alike, and a heavily shrunk linear regression produces the same positivity. Free the intercept and the 12,000-feature weights correlate 0.9999 with a 100-feature model and 0.9661 with a regression on the long-term yield alone. The machine did not discover a sign restriction. The origin did. Two cautions sit on these Sharpe ratios. They are the best of a wide penalty grid, picked on the same January 1940 to December 2020 window used to report them, with no deflation for the search. And month-t inflation, one of the 15 predictors, is not published until the middle of month t+1\. Buncic lags it and the random-feature Sharpe ratios fall across the board. He does not print a single replacement number, so none is supplied here. None of the ratios deduct a spread. A monthly timing overlay on the market is cheap relative to a cross-sectional book. It is not free. ## The bill is predictors over months The question is why a short monthly sample cannot carry 12,000 coefficients. The variance piece of the prediction error grows in proportion to the number of coefficients divided by the number of months. Shah's decomposition of expected squared error, his equation (1.5), splits the error of a fitted function into squared bias, variance of the fit, and the irreducible noise in the outcome. For ordinary least squares on d basis functions, the variance piece averages to his equation (3.1). $$ \\mathbb{E}\\,R(\\hat{h}\_{D}) - \\mathbb{E}\\,R(\\hat{h}\_{X}) \\approx \\frac{\\sigma^{2}\\, d}{n} $$ Read it as an extra out-of-sample squared error, in the square of whatever units the outcome is in. Sigma-squared is the variance of the outcome given the predictors. d is the number of coefficients. n is the number of training observations. The old article ["The Mathematics of Machine Learning, for Traders"](https://aligrithm.com/the-mathematics-of-machine-learning-for-traders/) derives the trace step. The usable fact is the ratio d over n. Worked with the variance of R itself, 1.1985, as a stand-in for sigma-squared. That stand-in is reasonable here because these predictors explain little of next month, which is the whole equity-premium problem. Fifteen coefficients and 120 months: 1.1985 times 15 divided by 120 equals 0.150\. Twelve thousand coefficients and 12 months: 1.1985 times 12,000 divided by 12 equals 1,198.5\. The second bill is the complexity ratio, 1,000, times the noise variance. Ridge and the average across random draws are attempts to refuse that bill. They do not repeal the ratio. Buncic's bias numbers say the refusal failed on its own terms. For the 12,000-feature model, squared bias of the forecast sits between 0.0130 and 0.0267 when the intercept is forced to zero, and under 0.0002 once the intercept is free. The flexible model was the biased one. The penalty you choose is a belief about how many of the predictors matter. Shah compares an ell-2 constraint, the ridge style, with an ell-1 constraint, the sparse style. Once each constraint is set to the smallest width that still contains the true coefficient vector, the excess-risk bounds are on the order of the square root of p over n for ell-2, and the square root of (s times the natural log of p) over n for ell-1, where s is the number of nonzero coefficients. Read the bounds as rates on excess prediction error. Smaller is a tighter guarantee. Take p of 15, n of 120, and suppose s equals 2, which is an illustration, not a test result. Buncic's univariate horse race keeps surfacing the term spread and book-to-market, so two is a serious guess and still a guess. The ell-2 order is the square root of 15 divided by 120, which is 0.35\. The ell-1 order is the square root of (2 times the natural log of 15) divided by 120, which is 0.21\. If instead all 15 predictors matter, the ell-1 constraint has to open wide enough to hold a dense vector, and that order worsens to 0.58, while ell-2 stays at 0.35\. A sparse penalty wins when you believe a few characteristics carry the premium. A ridge penalty wins when you believe the signal is spread out. That belief is the economic prior. It is an input. The fitting routine does not supply it. Expected returns in Buncic's zero-intercept block make the same point in portfolio units. The 12,000-feature model, penalty 1,000, has annualized expected timing-strategy returns of 0.273, 0.079, and 0.044 at windows of 12, 60, and 120 months. The best expanding 15-predictor ridges have 0.571, 0.639, and 0.859\. Divide 0.859 by 0.044 and you get 19.5\. Buncic calls that close to twenty-fold. I am keeping his ratio and his wording. The table transcription in the source file is too garbled to re-read those cells off the grid, so the prose figures are the ones used here. ## A network wants 24 countries; a regression does not The question is whether a more flexible model ever earns its parameters. It does when the training set is a cross-section, and the extra return is 0.12 percentage points a month. Chen, Hanauer, and Kalsbach run the comparison the equity-premium study cannot. One series gives you hundreds of months. A stock cross-section gives you a fresh panel every month. Their sample is 24 developed markets, 36 firm characteristics ranked each month within each country onto the interval from -1 to +1, microcaps removed (the smallest names making up 3 percent of country market cap), and value-weighted long-short quintiles. The test window is January 2002 to December 2021, 240 months. Training is an expanding window. The only lever is where the fit happens: a separate model for North America, Europe, Japan, and Asia Pacific, or one model on the pooled developed markets. The old article ["Global vs Regional: When Complexity Wants More Data"](https://aligrithm.com/global-vs-regional-when-complexity-wants-more-data/) keeps the screens and the hyperparameter grids. The numbers that decide the argument are in their Table 1. Local ordinary least squares earns 0.47 percent a month (standard deviation 2.64, t-statistic 2.76). Global ordinary least squares earns 0.43 percent (standard deviation 2.76, t-statistic 2.44). The elastic net is the same pattern, 0.43 local against 0.39 global. Random forests tie at 0.52\. Gradient-boosted trees go the other way, 0.51 local against 0.59 global (t-statistic 3.75). The neural-network ensemble goes further, 0.62 local (t-statistic 4.62) against 0.74 global (t-statistic 5.73). The crossover sits between the forest and the boosted trees. Linear models take a small local edge. The two flexible models take a global one. Check the printed t-statistics against the printed means and standard deviations over 240 months. Ordinary least squares local: 0.47 divided by (2.64 over the square root of 240) equals 2.76, matching the print. Boosted trees global: 0.59 divided by (2.44 over the square root of 240) equals 3.75, matching the print. The global network: 0.74 divided by (1.99 over the square root of 240) equals 5.76, against their printed 5.73\. The printed 5.73 stays. The gap fits a rounded standard deviation, or a robust standard error. The result does not move. The 0.12 percentage-point gap, 0.74 minus 0.62, is 0.12 divided by 0.62, or 19 percent of the local network's own return. Their cumulative-performance figure prints 456 percent for the global network over the twenty-year test. Compounding the 0.74 percent mean for 240 months produces 487 percent. The figure is the compounded path, which sits under the compounded mean because volatile months do not compound at the average. 456 stays as the path. Spanning regressions ask whether one book already contains the other. Regress the global network's long-short return on the local network's, and the intercept is 0.20 percent a month with a t-statistic of 3.45\. Multiply 0.20 by 12 and you get 2.4 percent a year, an arithmetic scaling, not a compound. The boosted-tree version of that intercept is 0.15 percent a month, t-statistic 2.04\. Run the regression the other way, local on global, and the network intercept is -0.07 with a t-statistic of -1.15\. The local flexible model adds nothing once the global one is in the book. The linear models show no reliable intercept in either direction. ![Sharpe ratios of long-short quintile books by region. Yellow bars are models trained inside the region. Blue bars are the same algorithm trained on all 24 developed markets.](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/10/article_432-global_local_sharpe.png) The regional Sharpe ratios, which the paper annualizes itself, repeat the split. The global network earns 1.03 in North America against 0.85 local, 0.96 against 0.81 in Europe, 0.68 against 0.42 in Japan, and 1.18 against 0.78 in Asia Pacific. Japan is the case the older linear literature treated as special, and the global network still leads there, by 0.68 minus 0.42, or 0.26\. For ordinary least squares and the elastic net the paper reports a mixed pattern: global fits better in Europe and Asia Pacific, regional fits better in North America and Japan, and the gaps are small. The rank of characteristics onto \[-1, +1\], done inside each country, is already an economic prior. It is the cross-sectional rank, not the raw accounting ratio, that enters the fit. The network then needs the pooled sample to estimate the nonlinear pieces of that rank. The regression does not. These Chen, Hanauer, and Kalsbach returns are gross. The paper does not report a cost cut. A monthly, value-weighted, country-neutral quintile book turns over less than a model that rebuilds every name's weight from scratch. How much less is not in the tables. ## A fifth, then the turnover The question is the size of the nonlinearity bonus a book should budget. The honest number in the factor-investing deck is about 20 percent on the Sharpe ratio, and the turnover attached to the published machine-learning books can erase it. The deck prints two out-of-sample Sharpe ratios for each model. The first cut: a random forest at 2.19, a neural network at 3.21, a principal-components model at 3.21, and Kelly, Pruitt, and Su's instrumented principal-components model (IPCA) at 3.39\. The second cut, with the first number repeated in parentheses: 0.86 (2.19), 2.39 (3.21), 2.77 (3.21), and 3.21 (3.39). These live on a different book from the 0.55 market-timing ratios above. They are cross-sectional factor portfolios, and the level is not comparable to a timing Sharpe on the market. The ranking inside the table is the usable fact. IPCA, a linear model that builds factors as combinations of characteristics, slips from 3.39 to 3.21\. Divide 3.21 by 3.39 and subtract 1: a 5 percent slip. The neural network slips from 3.21 to 2.39\. Divide 2.39 by 3.21 and subtract 1: a 25 percent slip. The most linear specification holds up. The old article ["Does ML Actually Help Asset Pricing? Kelly's 20%, Not 2–3×"](https://aligrithm.com/does-ml-actually-help-asset-pricing-kellys-20-not-2-3x/) goes after the unfair part of the comparison, a three-factor baseline against a model fed hundreds of characteristics. Bryan Kelly's own sentence, as the deck records it, is the budget: a 20 percent increase in the Sharpe ratio from adding nonlinearities feels realistic, and a doubling does not. Turnover is where that fifth goes to die. Avramov, Cheng, and Metzker measure monthly turnover of 0.976 on the Gu, Kelly, and Xiu machine-learning portfolio and 1.664 on a deep-learning stochastic-discount-factor portfolio. Size and value, in the same discussion, turn over below 10 percent a month. A book that replaces itself every month, or 1.66 times every month, pays the spread on the way in and on the way out. Avramov, Cheng, and Metzker also find the excess return concentrated in the short leg and in tiny, illiquid, distressed names, and they find the nonlinear models weaken once those names are dropped, while IPCA does not. A 20 percent Sharpe bump that lives in names you cannot short, at a turnover near 100 percent a month, is a research result. It is a poor order. ## The rule at the desk The question is what changes in the research process on Monday. On a single monthly series, fit the mean, keep the predictor count near the Goyal-Welch 15, shrink with a small z, and use the long window. I am against replacing that ridge with 12,000 random features. The complex model lost to buy-and-hold, lost to a rolling historical average, and lost to the 15-predictor ridge with the intercept forced off and with it left on. The best random-feature specification, 0.5863, beats buy-and-hold by 0.5863 minus 0.5494, which is 0.037, and then loses to the ridge by 0.6990 minus 0.5863, which is 0.113\. That 0.037 is the timing gain left behind by a flat refusal of every flexible model on this sample. It is smaller than the gain you give up by trading the headline model, and neither number has paid a spread or survived a penalty for searching the grid. On a cross-section, add countries before adding layers. The network's global-minus-local gap is 0.12 percent a month, and the spanning intercept on the local network is 0.20 percent a month. Ordinary least squares does not want the pool: 0.47 local against 0.43 global. The prior that earns its place is the one you can say before the fit, a rank inside the country, a factor structure like IPCA, a penalty matched to how many characteristics you believe are real. Shah's rates are the reason. The ratio inside the variance bill is d over n. Grow n, from one market to 24, and a larger d becomes affordable. Leave n at 12 months and no penalty in the grid makes 12,000 features the right trade. Budget a fifth on the Sharpe ratio for the nonlinear piece, then write down the turnover. If the book cannot pay a monthly turnover near 1, stay with the linear factor model that held 3.21 of its 3.39 in the deck's second cut. Complexity is a data-hungry regularizer with a modest payoff. The plot that called it a virtue was an intercept set to zero and an average of ratios. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/10/article_432-visual-0.png) ## KEY POINTS - On one common window, January 1940 to December 2020, buy-and-hold earns a Sharpe ratio of 0.5494\. The 12,000-feature model earns 0.4851\. A 15-predictor expanding ridge with an intercept earns 0.6990\. The best random-feature model, 200 features, penalty 2, 120 months, earns 0.5863. - The rising complexity curve needs a zero intercept and a per-draw average of Sharpe ratios. Free the intercept and the best random-feature Sharpe rises with the window, from 0.4745 to 0.5157 to 0.5863\. At 2 features and 120 months the per-draw average prints Sharpe ratios near -0.0017 next to positive expected returns. - The variance bill is sigma-squared times coefficients over months. With the variance of the standardized excess return at 1.1985, 15 coefficients on 120 months cost about 0.15, and 12,000 coefficients on 12 months cost about 1,198\. A sparse penalty tightens the bound only if few predictors matter. - On 24 developed markets, the global neural network earns 0.74 percent a month (printed t-statistic 5.73) against 0.62 percent local. Ordinary least squares goes the other way, 0.47 local against 0.43 global. The global network's spanning intercept over the local network is 0.20 percent a month, t-statistic 3.45\. Regional network Sharpes: 1.03, 0.96, 0.68, and 1.18 against local 0.85, 0.81, 0.42, and 0.78. - Budget about 20 percent on the Sharpe ratio for nonlinearities. In the deck's two cuts, linear IPCA slips from 3.39 to 3.21 and the neural network slips from 3.21 to 2.39\. Monthly turnover on the published machine-learning books is 0.976 and 1.664, against under 10 percent for size and value. - The 0.6990 and 0.5863 are maxima over a penalty grid, scored on the window used to pick them. Inflation in the predictor set prints two weeks late. No Sharpe ratio above has paid a spread. The global-network t-statistic implied by the printed mean and standard deviation is 5.76, against the printed 5.73\. Buncic's mean, standard error, and t-statistic for the standardized return do not multiply through. ## References - [The Virtue of Complexity in Return Prediction (Kelly, Malamud, Zhou, 2024)](https://doi.org/10.1111/jofi.13298?ref=aligrithm.com) - [Comment on 'The Virtue of Complexity in Return Prediction' (Berk, 2023)](https://papers.ssrn.com/sol3/papers.cfm?abstract%5Fid=4410125&ref=aligrithm.com) - [Simplified: A Closer Look at the Virtue of Complexity in Return Prediction (Buncic, 2025)](https://papers.ssrn.com/sol3/papers.cfm?abstract%5Fid=5239006&ref=aligrithm.com) - [Model Complexity and the Performance of Global versus Regional Models (Chen, Hanauer, Kalsbach, 2025)](https://doi.org/10.1016/j.econlet.2025.112641?ref=aligrithm.com) - [Mathematics of Machine Learning (Shah)](https://www.statslab.cam.ac.uk/~rds37/teaching/machine%5Flearning/notes%5Fcur.pdf?ref=aligrithm.com) - [Machine Learning in Factor Investing (DDA3600)](https://papers.ssrn.com/sol3/papers.cfm?abstract%5Fid=5301759&ref=aligrithm.com) --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ### 3.49 Building a Statistically Valid Backtest: The Checklist Nobody Runs URL: https://aligrithm.com/building-a-statistically-valid-backtest-the-checklist-nobody-runs/ Last updated: 2026-10-16T08:09:04.000Z McLean and Pontiff record stock-portfolio predictors whose performance is 26% lower out of sample than in sample, and 58% lower after publication. Suhonen, Lennkh, and Perez look at 215 smart-beta strategies banks offered and find a median deterioration of 73% in the Sharpe ratio from the backtest to the live period. Arakelian, Bolesta, Liu, Osterrieder, Poti, Schwendner, Sutiene, Vlah Jerić, and Weinberg take those two results as the case for a valid backtest. The machinery they assemble is the probability of backtest overfitting, White's reality check, Hansen's test of superior predictive ability, and Romano and Wolf's stepwise test. Underneath sit an out-of-sample split you do not retune, a cost, a size cap, and a permutation against chance. A Sharpe ratio that has not been through that machinery is a sales exhibit. ## What this actually is You have a monthly equity factor, or an ES rule, with an in-sample Sharpe ratio of 1.4 across ten years. You tried 40 variants. The memo holds the winner. The other 39 stay off the page. The 1.4 is the maximum of that search, and the maximum of a noisy search sits above the noise. The method scores how often a selection of that kind places the in-sample winner below the median on data the rule never saw. It also tests whether any rule beat a benchmark you named in advance, with the full set of rules as the hypothesis. A youth-league tryout runs the same way. Forty kids shoot free throws for an hour. You take the kid with the most makes. The next session that kid looks ordinary, because the hour mixed skill with luck and you kept the luck. A bigger tryout produces a higher score on the day and a worse next session. Ship under that claim and you freeze the rule before the holdout year, subtract slippage and commission before you quote a profit factor, cap leverage at the size the margin desk will fund, and replace the winner's t-statistic with a test of the whole set. Trust a backtest that skips those steps and the live Sharpe ratio is the one Suhonen, Lennkh, and Perez measured, a median 73% below the tear sheet, on 215 products a bank was willing to sell. Apply the same steps to a real edge that is too small to clear a multiple-testing bar and you stay with the benchmark. Staying with the benchmark is the cheaper mistake. ## The in-sample winner finishes below the median The question is whether the strategy you would have picked in sample still sits in the top half once the sample changes. Bailey, Borwein, Lopez de Prado, and Zhu supply the definition, and the discussion paper reprints it. There are N strategies. Rank each one from 1 to N on the in-sample performance measure, with N the best rank and 1 the worst, and rank them again on the out-of-sample measure. The selection overfits when the expected out-of-sample rank of the in-sample winner is at or below N/2. $$ \\sum\_{n=1}^{N} E\\left\[\\bar{r}\_{n} \\mid r \\in \\Omega\_{n}^{\*}\\right\] \\operatorname{Prob}\\left\[r \\in \\Omega\_{n}^{\*}\\right\] \\leq \\frac{N}{2} $$ Read it as: N is a count of strategies. r is the in-sample rank vector and r-bar is the out-of-sample rank vector, both in rank units from 1 to N. Omega-star sub n is the set of rankings in which strategy n is the in-sample winner, meaning its in-sample rank equals N. The probability in front is the chance that n wins the in-sample horse race. The term inside the expectation is n's out-of-sample rank on those occasions. The sum is one number, the expected out-of-sample rank of whoever won in sample. The process overfits when that number is at or below N/2. The probability of backtest overfitting, PBO, is the chance that this winner's out-of-sample rank falls below that same cutoff. $$ \\mathrm{PBO} = \\sum\_{n=1}^{N} \\operatorname{Prob}\\left\[\\bar{r}\_{n} < \\frac{N}{2} \\mid r \\in \\Omega\_{n}^{\*}\\right\] \\operatorname{Prob}\\left\[r \\in \\Omega\_{n}^{\*}\\right\] $$ Read it as: PBO is a probability between 0 and 1, with no market unit. For each strategy n, take the probability that its out-of-sample rank is below N/2 given that it won in sample, multiply by the probability that it won, and add the N products. The discussion paper prints a strict inequality. Rank equal to N/2 does not count as a miss. The ten splits below are constructed so both sums can be checked by hand. They are not a price series. N is 5, so N/2 is 2.5\. Ranks 1 and 2 are misses. Rank 3 is not. The paper calls N/2 the median of the ranks. For five strategies the middle rank is 3, and the formula's cutoff is 2.5\. The arithmetic below uses 2.5, the number in the formula. With an even count the printed cutoff is harsher than a bottom half: four strategies give N/2 equal to 2, and the strict inequality keeps only rank 1. | Split | In-sample winner | Out-of-sample rank | | ----- | ---------------- | ------------------ | | 1 | A | 1 | | 2 | A | 2 | | 3 | B | 4 | | 4 | C | 1 | | 5 | A | 2 | | 6 | D | 2 | | 7 | B | 5 | | 8 | C | 1 | | 9 | E | 4 | | 10 | A | 2 | The out-of-sample ranks of the in-sample winner are 1, 2, 4, 1, 2, 2, 5, 1, 4, 2\. Seven of the ten sit below 2.5, so the miss rate is 7/10, which is 0.70\. The weighted sum matches. A wins 4 of 10 splits and misses all 4, contribution 0.40 times 1\. B wins 2 and misses none, contribution 0\. C wins 2 and misses both, contribution 0.20\. D wins 1 and misses it, contribution 0.10\. E wins 1 and finishes at rank 4, contribution 0\. 0.40 plus 0.20 plus 0.10 equals 0.70. The expected out-of-sample rank uses the same weights. A's mean rank on its four wins is (1 + 2 + 2 + 2) / 4 = 1.75\. B's is (4 + 5) / 2 = 4.5\. C's is 1\. D's is 2\. E's is 4\. Then 1.75 times 0.40, plus 4.5 times 0.20, plus 1 times 0.20, plus 2 times 0.10, plus 4 times 0.10, equals 0.70 + 0.90 + 0.20 + 0.20 + 0.40 = 2.40\. The plain average of the ten ranks is 24/10 = 2.4\. The two routes agree. Because 2.4 is below 2.5, the paper's definition calls this selection overfit. The comparison for the 0.70 is a uniform out-of-sample rank, the case where the in-sample win carries no information about the out-of-sample rank. Each of the five ranks then has probability 1/5, and only ranks 1 and 2 sit below 2.5, so the miss probability is 2/5 = 0.40\. The expected rank is (1 + 2 + 3 + 4 + 5) / 5 = 3, which sits above the cutoff of 2.5, so a uniform rank does not meet the overfitting definition. The toy's 0.70 against that 0.40, and its 2.4 against that 3, are the gap the definition is built to catch. ![Out-of-sample rank of the in-sample winner on ten constructed splits of five strategies. Seven of the ten ranks sit below the line at 2.5](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/09/article_492-pbo_is_winner_oos_rank.png) The discussion paper stops at the definition. It does not compute a PBO on a price series, so 0.70 is not a market result. The number you would take to a committee comes from reshuffling which slice of your own sample is the training slice and recording where the training winner finishes on the held-out slice. Bailey, Borwein, Lopez de Prado, and Zhu estimate that probability with combinatorially symmetric cross-validation. The discussion paper names the probability and does not show the reshuffle. The old article ["The Backtest Integrity Checklist"](https://aligrithm.com/the-backtest-integrity-checklist/) is where that reshuffle becomes a line you either pass or you do not ship. ## White's reality check tests the whole search The question is whether any model beat a benchmark, once the hypothesis is the entire search and not the winner you noticed. White's reality check starts from a loss. For each model k and each date, form the loss of the benchmark minus the loss of model k. A positive gap means model k lost less than the benchmark on that date. $$ \\delta\_{k,t} = L\\left(y\_{t+h},\\hat{y}\_{t+h,\\mathrm{BM}}\\right) - L\\left(y\_{t+h},\\hat{y}\_{t+h,k}\\right) $$ Read it as: delta sub k,t is the loss gap for model k at date t, in whatever unit the loss uses. y at t plus h is the outcome h steps ahead. The two y-hats are the benchmark forecast and model k's forecast of that outcome, both formed with information dated t. L is the loss. The paragraph above this equation in the discussion paper writes the forecast with a lag, time t minus h. The equation writes t plus h, which is the timing in White: you score a forecast of the outcome h steps ahead. The equation is the one used here. The null says the best model has no positive expected gap. $$ H\_{0}: \\mu = \\max\_{k=1,\\ldots,m} E(\\delta\_{k}) \\leq 0 $$ Read it as: m is the count of models. E(delta sub k) is model k's expected loss gap, in loss units per date. Mu is the largest of those m expectations. The null says mu is zero or negative, so no model beats the benchmark on average. The alternative says mu is positive, so at least one does. The sentence beside this display in the discussion paper does not match it. That sentence says the null compares loss differentials to benchmark losses, and then says model losses are larger than the benchmark. Use the display. Take the loss to be the negative of the monthly return, in percentage points. Then delta equals the model's return minus the benchmark's return, and the units stay in percentage points per month. Four months, three models, built so the arithmetic is checkable. This is not a price history. The benchmark returns 1, 1, 2, 0\. Model 1 returns 3, 1, 3, 1, so its gaps are 2, 0, 1, 1\. Model 2 returns 4, 0, 2, 2, so its gaps are 3, minus 1, 0, 2\. Model 3 returns minus 1, 2, 2, minus 1, so its gaps are minus 2, 1, 0, minus 1\. The means are 1, 1, and minus 0.5 percentage points per month. The maximum is 1, shared by model 1 and model 2\. Check model 1 through the loss rather than the return shortcut: benchmark losses are minus 1, minus 1, minus 2, 0, and model 1's losses are minus 3, minus 1, minus 3, minus 1\. Subtracting gives 2, 0, 1, 1\. Same gaps. Model 1's gaps deviate from 1 by +1, minus 1, 0, 0\. The sum of squares is 2\. Divide by 3, one less than the four months, and the sample variance is 2/3\. The standard error is the square root of that variance, divided by the square root of 4\. The t-statistic is the mean gap divided by that standard error, a unit-free measure of how far the mean sits from zero given the scatter across months. For model 1 it equals the square root of 6, which is 2.45\. Model 2 has the same mean of 1 and a sum of squared deviations of 10, and the same steps give a t-statistic of 1.10\. The path is wider, from +3 to minus 1, so the same average edge is a weaker statistic. Model 3's mean is minus 0.5 and its sum of squared deviations is 5, and the t-statistic is minus 0.77. The one-sided p-value on model 1 alone, one-sided because the alternative is a positive gap, with 3 degrees of freedom left after estimating the mean, is 0.046\. The one-sided 5% critical value of that t distribution is 2.35, and 2.45 sits above it. That p-value belongs to a single model. The search had three. The discussion paper points at the Bonferroni bound as one way to cap the p-value of a multiple test. The bound splits one 5% budget for the whole search by the model count, so that a false pass anywhere in the set stays inside that budget. Here 0.05 divided by 3 is 0.0167, and the critical t-statistic on 3 degrees of freedom rises from 2.35 to 3.74\. Model 1's 2.45 misses 3.74. White's reality check replaces that bound with the joint distribution of the models. Bonferroni treats the models as separate. Models scored on the same four months share the months, so their gaps move together, and the null distribution of the maximum is tighter than the Bonferroni corner. The discussion paper says an analytic form for that distribution is not practical. You resample the dates, recompute the maximum on each resample, and read the p-value from the resampled maxima. Four months will not support a resample you should trust, because a draw can repeat one month until the standard deviation collapses. No bootstrap p-value is printed here for that reason. The object of the test is still the maximum, and the crude bound the paper cites sits above the best t-statistic in the toy. The paper says the procedure holds the family-wise error rate, the probability of one or more false discoveries, at a preset level alpha. Alpha is the false-rejection rate you pick before seeing the models. A common choice is 0.05\. One sentence then gets the error backwards. It says the procedure caps the chance of a wrong rejection when the null is false. A rejection of a false null is the correct decision. The cap applies when the null is true: no model beats the benchmark, and the test says one does. The formulas earlier in the section match that second statement. Hansen's test of superior predictive ability is a repair for a different defect. White's null is built under the least favorable configuration, the case in which every model, including model 3 with a mean gap of minus 0.5, is treated as if its expected gap were zero. A hopeless model can still win the maximum by luck, the critical value rises, and a real model fails the test. Stack more poor models into the race and that critical value rises further. Hansen recenters so that a model which is inferior by a clear margin does not enter the null as a tie with the benchmark. The discussion paper also calls this a test of superior predictive performance. The 2005 title is superior predictive ability. Hansen's answer is whether any model is superior. The names of those models are a separate step. Romano and Wolf walk the list. Rank the models by the statistic. Test the best one against a critical value computed from the whole set. If it fails, stop, and the set of superior models is empty. If it passes, drop it, recompute the critical value on what remains, and continue until a model fails. On this toy the best statistic is model 1's 2.45\. That sits above the single-test critical value of 2.35, and the Bonferroni bound on the same three models is 3.74\. The stepwise critical value is a resampled quantile. Four months will not support that resample, so the bound you can compute by hand is the Bonferroni one, and model 1 misses it. The illustrated output is an empty set. A later paragraph in the discussion paper calls the Romano and Wolf procedure a false-discovery-rate method. The 2005 paper controls the family-wise error rate, the chance of any false inclusion. So does the discussion paper's own account of the stepwise test, earlier in the same section. The false discovery rate would cap the share of rejected models that are false, and it would let more models through. The name in the later paragraph is a different test. A quadratic form in all the forecast errors at once is the other route the paper considers, and it needs a covariance matrix that is close to singular when the model count is large, so the chi-squared limit does not arrive. That is why these tests bootstrap a maximum t-statistic. The paper says the bootstrap is heavy enough that machine-learning backtests in production often skip it. Skip it and the number you are left holding is the single-model p-value of 0.046. ## Hold out, charge the cost, cap the size, permute The question is which operating rules keep the backtest in the same units as the account. The out-of-sample split comes first. The discussion paper's schematic puts portfolio weights, the holding period, and the rebalancing dates inside a train window, then a split point, then a test window. The drawing is the easy part. The rule is the sentence beside the overfitting definition: once you change the model so that it also looks good on the test window, that window is in sample. One retune after you have seen the holdout, and the split no longer separates anything. A longer test window estimates the performance with less noise. The window has to stay untouched, including by the person who picks the variant to report. Costs come next. Profit factor is the sum of winning trades divided by the sum of absolute losing trades, a ratio with no unit. Ten trades, wins of 5, 4, 3, 2, and 1 percentage points, absolute losses of 1, 1, 2, 2, and 3\. The profit factor is 15/9, which is 1.67\. Charge 50 basis points of slippage and commission on each trade. A basis point is 0.01 of a percentage point, so the charge is 0.5 percentage points, subtracted from each win and added to each absolute loss. The wins become 4.5, 3.5, 2.5, 1.5, and 0.5, summing to 12.5\. The absolute losses become 1.5, 1.5, 2.5, 2.5, and 3.5, summing to 11.5\. The profit factor is 12.5/11.5, which is 1.09\. The paper's instruction is to deduct that charge on every simulated trade and recompute the profit factor, the expectancy, and the drawdown. The gap between 1.67 and 1.09 is where a high-turnover rule loses the edge the gross number advertised. The paper does not choose the 50 basis points. You take the charge from the asset, the venue, and the size you intend to trade. A charge of zero is how the backtest overstates the live result. Size after that. An edge of 2% a year on notional, with a volatility of 15% a year, is a Sharpe ratio of 2/15, which is 0.13\. Both the return and the volatility are fractions of notional per year, so the ratio has no unit. At ten times leverage the account shows a 20% return and a 150% volatility, and 0.20 divided by 1.50 is still 0.13\. A year one standard deviation below the mean is 0.20 minus 1.50, a loss of 130% of equity. The tear sheet printed the 20%. The edge was 2%. Full leverage, a position in every name, and no size cap inflate the profit and ignore margin, liquidity, and a drawdown limit. Scale the simulated trades to a maximum the account can hold, then recompute the same ratios. The permutation is the comparison to chance. The paper wants the strategy's result set against a null of random hypothetical strategies, by Monte Carlo, with the costs left on. A profit factor of 1.09 that sits in the middle of that null is luck. The same comparison shows up inside the reality check as a resampling of the dates. Either way you compare the result to the distribution of random rules that have no edge. The old article "Replication Notebook Template: Claim → Replicate → Stress → Verdict" puts this comparison in the stress beat. Run it on the rule you froze before the holdout, on the same cost and the same size cap. Two further items from the paper's list, both easy to game. Sensitivity moves the entry, the size, and the indicator across a range you would have accepted before the search, and asks whether the profit factor stays above 1\. A profit that exists at one setting and vanishes next door is a fit to the path. Walk-forward rolls the train window, re-estimates, and stitches the test periods. A single calm decade is a weak test, which is the paper's point. Choosing the window length from the stitched result puts those test periods back into the search. The survey that precedes the list names the area under a ROC curve, a Kolmogorov-Smirnov comparison, information value, and a Hosmer-Lemeshow grouping. Those four score a predicted probability against an outcome, which is the job of a default model. A trading rule's result is a return after cost. Leave them off this checklist. The paper's closing recommendations are thinner than the list: hold out data and keep it clean of snooping, score stability and complexity rather than mean squared error alone, train the people who run the tests, and talk to outside researchers. The four that change a number on a tear sheet are the untouched holdout, the cost, the size cap, and the permutation. The old article ["The Backtest Integrity Checklist"](https://aligrithm.com/the-backtest-integrity-checklist/) turns those into pass-fail lines. This paper is the argument for having the lines. ## The haircut the citations report The question is how large a haircut the papers in the reference list put on a raw in-sample result. McLean and Pontiff, on predictors of the cross-section of stock returns: performance 26% lower out of sample than in sample, and 58% lower after publication. Index the in-sample figure at 100\. Subtract 26 and the out-of-sample index is 74\. Subtract 58 and the post-publication index is 42\. Those two complements are arithmetic from the declines the discussion paper quotes, not extra statistics. Falck, Rej, and Thesmar record a 50% decline in the performance of systematic strategies after publication, so the same style of index sits at 50\. Suhonen, Lennkh, and Perez, on the 215 smart-beta strategies: the median Sharpe ratio deteriorates 73% from the backtest to the live period, so a backtest Sharpe ratio indexed at 100 sits at 27 live. The four bars come from three measurements, a return, a performance decline, and a Sharpe ratio. All four point down. ![Remaining performance after the declines quoted in the discussion paper. In-sample or backtest is 100. McLean and Pontiff: 74 out of sample and 42 after publication. Falck, Rej, and Thesmar: 50 after publication. Suhonen, Lennkh, and Perez: 27 on the live Sharpe ratio](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/09/article_492-published_performance_haircut.png) Jensen, Kelly, and Pedersen reach the other conclusion. The discussion paper says they find a high degree of validity for factor strategies after publication, in the United States and abroad. It prints no magnitude for that finding, so they are absent from the chart. The discussion paper does not adjudicate the disagreement. A reader should not adjudicate it from this paper either. There is no new backtest in it. The MACD exhibits are terminal screens of an S&P 500 moving-average convergence rule, with buy marks, sell marks, and a block of descriptive trade counts. Change the sample or the bar size and the picture changes, which is the observation the screens are there to support. Descriptive counts are not a reality check. Wiecki, Campbell, Lent, and Stauth are cited for the statement that backtests overstate live performance, with no magnitude attached in this discussion. The magnitudes the discussion paper does print are McLean and Pontiff's 26% and 58%, Falck, Rej, and Thesmar's 50%, and the 73% median Sharpe-ratio deterioration on the 215 live products. Use those as a prior while your own reality check is still unrun. Take 26% off an in-sample return on the way to a fresh sample, and 58% off after publication. Take 73% off a marketed smart-beta Sharpe ratio on the way from the tear sheet to the account. That prior is a set of cited declines, and Jensen, Kelly, and Pedersen are the reminder that a factor with a reason can survive the publication date. You measure survival on a holdout you did not retune. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/09/article_492-visual-0.png) ## KEY POINTS - McLean and Pontiff's stock-portfolio predictors run 26% lower out of sample than in sample, and 58% lower after publication. On 215 smart-beta strategies that banks offered, Suhonen, Lennkh, and Perez find a median Sharpe-ratio deterioration of 73% from backtest to live. Falck, Rej, and Thesmar record a 50% decline after publication. Indexed to 100, the complements are 74, 42, 27, and 50. - The probability of backtest overfitting is the chance the in-sample winner finishes with out-of-sample rank below N/2\. On a constructed set of five strategies and ten splits, 7 of 10 winners finish below 2.5, the weighted sum equals 0.70, and the expected out-of-sample rank is 2.4, which meets the paper's overfitting cutoff of 2.5\. The 0.70 is a worked sum, not a market estimate. The discussion paper does not compute a PBO on prices. - White's reality check tests the best loss gap in the whole search. With loss equal to minus the return, three toy models over four months post mean gaps of 1, 1, and minus 0.5 percentage points. The t-statistics are 2.45, 1.10, and minus 0.77\. The single-model one-sided p-value of 0.046 dies under the Bonferroni bound the paper cites: critical t rises from 2.35 to 3.74. - Hansen's superior predictive ability test stops treating hopeless models as ties with the benchmark, which is what makes White's null conservative. Romano and Wolf's 2005 stepwise test then names which models survive, controlling the family-wise error rate. A later sentence in the discussion paper calls that procedure a false-discovery-rate method. That name is the wrong test. Another sentence caps rejections "when the null is false," which reverses a type I error. Follow the formulas. - Before a number leaves the building: an untouched holdout, a cost taken off every trade (the toy profit factor falls from 15/9 = 1.67 to 12.5/11.5 = 1.09 at 50 basis points), a size cap (2% on notional at 15% volatility is a Sharpe ratio of 0.13, and 10x leverage still has Sharpe 0.13 while a one-sigma year loses 130% of equity), and a permutation against random rules. ROC curves and Hosmer-Lemeshow tests score default probabilities. They do not score this. - Jensen, Kelly, and Pedersen are the dissent the discussion paper reports and does not settle: factor strategies can stay valid after publication. This paper runs no new backtest. The haircut above is a prior, and the measurement is your own holdout. ## References - [The Probability of Backtest Overfitting (Bailey, Borwein, Lopez de Prado, and Zhu)](https://doi.org/10.21314/JCF.2016.322?ref=aligrithm.com) - [A Reality Check for Data Snooping (White)](https://doi.org/10.1111/1468-0262.00152?ref=aligrithm.com) - [A Test for Superior Predictive Ability (Hansen)](https://doi.org/10.1198/073500105000000063?ref=aligrithm.com) - [Stepwise Multiple Testing as Formalized Data Snooping (Romano and Wolf)](https://doi.org/10.1111/j.1468-0262.2005.00615.x?ref=aligrithm.com) - [Does Academic Research Destroy Stock Return Predictability? (McLean and Pontiff)](https://doi.org/10.1111/jofi.12365?ref=aligrithm.com) - [Quantifying Backtest Overfitting in Alternative Beta Strategies (Suhonen, Lennkh, and Perez)](https://doi.org/10.3905/jpm.2017.43.2.090?ref=aligrithm.com) - [When do systematic strategies decay? (Falck, Rej, and Thesmar)](https://doi.org/10.1080/14697688.2022.2098810?ref=aligrithm.com) - [Is There a Replication Crisis in Finance? (Jensen, Kelly, and Pedersen)](https://doi.org/10.1111/jofi.13249?ref=aligrithm.com) - [All That Glitters Is Not Gold: Comparing Backtest and Out-of-Sample Performance on a Large Cohort of Trading Algorithms (Wiecki, Campbell, Lent, and Stauth)](https://doi.org/10.3905/joi.2016.25.3.069?ref=aligrithm.com) - [A discussion paper for possible approaches to building a statistically valid backtesting framework (Arakelian, Bolesta, Liu, Osterrieder, Poti, Schwendner, Sutiene, Vlah Jerić, and Weinberg)](https://papers.ssrn.com/sol3/papers.cfm?abstract%5Fid=4893677&ref=aligrithm.com) --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ### 6.62 Polynomial Regression Bands With a Profit Factor Below One URL: https://aligrithm.com/polynomial-regression-bands-with-a-profit-factor-below-one/ Last updated: 2026-10-16T08:09:05.000Z Gil Cohen fits second-, third-, and fourth-degree polynomial moving regression bands to Nasdaq-100 names from 2017 through March 2024, charges 0.3% on each fill, and names the four-degree model best. Table 3 puts the average net profit at 162.75 dollars per name. The abstract, the conclusion, and Table 4 print 162.73\. The risk case next to that crown is a profit factor of 0.55 and a minimum win rate of 41.51%. Profit factor is gross profit divided by gross loss. At 0.55 the gross loss is 1.82 dollars for each dollar of gross profit. That 0.55 is DoorDash. It is the smallest profit factor in the four-degree column, on a name that lost 52.38 dollars. The average of the hundred profit factors is 1.60, down from 1.76 at degree 2 and 1.67 at degree 3\. The model called the safe one has the worst average profit factor of the three. ## What this actually is You run Nasdaq-100 names, or you are about to let a script run them, and the offer is a band system with a seven-year daily backtest. The window is 2017 through March 2024, one hundred tickers, fills at the next open, and a commission of 0.3% on the buy and 0.3% on the sell. He says the printed results include those commissions. The curve he wants you to trade is a four-degree polynomial. The safety figure attached to it is a profit factor of 0.55, with a minimum win rate of 41.51% beside it. The system fits a polynomial of price on a time index, draws a band two residual standard deviations above the curve and two below it, buys the next open after the close has crossed the upper band, and sells the next open after the close has crossed the lower band. The book is long or flat. A flexible ruler on a paper chart is the same object. Allow one bend and the chalk line follows the broad swing. Allow two further bends and the chalk line ducks with the smaller turns, and the lane moves with it. You buy a step outside the lane on the way up. You sell a step outside the lane on the way down. Take the ranking as a portfolio result and you trade degree 4, and you size as if 0.55 were the strategy's loss ratio. The bill for that reading is specific. The 0.55 is one losing name. The average profit factor is lower at degree 4 than at degree 2\. The dollar average that won the horse race is fattened by a four-digit stock printed a second time under another ticker. The lookback, the number of days the ruler is pressed onto, is not in the paper. ## The close sends the order The order is a close through a band, filled at the next open. Degree changes the curve the bands hang on. The order stays that close. $$ Y = \\beta\_0 + \\beta\_1 X + \\beta\_2 X^{2} \\pm 2S $$ $$ Y = \\beta\_0 + \\beta\_1 X + \\beta\_2 X^{2} + \\beta\_3 X^{3} \\pm 2S $$ $$ Y = \\beta\_0 + \\beta\_1 X + \\beta\_2 X^{2} + \\beta\_3 X^{3} + \\beta\_4 X^{4} \\pm 2S $$ Read it as three fitted prices, in dollars per share. Y is the band, not a forecast he then scores on its own. X is a time index. Beta zero is the intercept in dollars. Each further beta multiplies a power of X, so the units of beta have to cancel that power back into dollars. S is the residual standard deviation in dollars, and the bands sit two of those above and below the curve. The first line is degree 2, the second degree 3, the third degree 4\. He publishes no betas, no S, and no length for X. The numbers below are an illustration of the arithmetic, with coefficients chosen here so the powers stay easy to check. They are not his fit. Set beta zero to 10 dollars, beta one to 0.5 dollars per day, beta two to 0.02 dollars per day squared, and S to 1.5 dollars. At day 10 the degree-2 curve is 10 plus 0.5 times 10 plus 0.02 times 100, which is 10 plus 5 plus 2, so 17 dollars. Two residual standard deviations are 3 dollars. The upper band is 20 and the lower band is 14\. A close at 20.50 sends a buy for the next open. A later close at 13.50 sends the sell. Add a degree-4 pair, beta three at 0.001 and beta four at minus 0.0001\. At day 10 those terms are 0.001 times 1,000 and minus 0.0001 times 10,000, which is plus 1 and minus 1\. The curve is still 17 dollars, and the bands do not move. At day 20 the degree-2 curve is 10 plus 0.5 times 20 plus 0.02 times 400, which is 28 dollars, and its upper band is 31\. The degree-4 curve is 28 plus 0.001 times 8,000 minus 0.0001 times 160,000, which is 28 plus 8 minus 16, so 20 dollars, and its upper band is 23\. A close at 24 is a buy on degree 4 and a nonevent on degree 2\. Same closes, different orders, eight dollars of curve from two extra powers. On a short window those extra powers fit noise. On a long window the extra powers add a small bend. The paper does not state the window. ## The 0.55 is DoorDash, and DoorDash lost money The profit factor in the abstract is one cell, and the identity that turns it back into dollars is fixed by his own definitions. $$ PF = \\frac{GP}{GL} $$ $$ GL = \\frac{NP}{PF - 1}, \\qquad GP = PF \\times GL $$ Read it as two steps, both in dollars except the ratio. PF is the profit factor, a pure number. GP is the gross profit, the sum of the winning trades, in dollars. GL is the gross loss, the sum of the losing trades written as a positive number of dollars. NP is net profit in dollars, and under those definitions it equals GP minus GL. Substitute GP equals PF times GL and you get NP equals GL times (PF minus 1), which rearranges to the second line. When the name made money, PF is above 1 and NP is positive. When the name lost money, PF is below 1 and NP is negative, so the ratio of two negatives is a positive gross loss. A profit factor of 1 means the gross profit equals the gross loss and the net profit is zero. DoorDash, degree 4: net profit minus 52.38 dollars, profit factor 0.55, win rate 42.86%, 21 closed trades, 19 days in the average trade. Gross loss is minus 52.38 divided by (0.55 minus 1), which is minus 52.38 divided by minus 0.45, so 116.40 dollars. Gross profit is 0.55 times 116.40, so 64.02 dollars. Check: 64.02 minus 116.40 is minus 52.38, and 64.02 divided by 116.40 is 0.55\. Divide 116.40 by 64.02 and the loss ratio is 1.818, which is 1 over 0.55\. DoorDash paid 1.82 dollars of gross loss for each dollar of gross profit. That is the number offered as evidence the four-degree strategy is less risky. The same inversion on a winner has to clear the other sign. Apple, degree 4: net profit 72.35 dollars, profit factor 1.70\. Gross loss is 72.35 divided by 0.70, so 103.357 dollars. Gross profit is 1.70 times 103.357, so 175.707 dollars. The difference is 72.35 and the ratio is 1.70\. The identity holds on a profit and on a loss. Three different names own the three "minimum" headlines in Table 4\. PDD Holdings owns the minimum net profit, minus 95.52 dollars, with a profit factor of 0.59 and a win rate of 51.28%. A name that wins half its trades and still prints a profit factor of 0.59 lost money because the losers were larger than the winners. Walgreens owns the minimum win rate, 41.51%, with a profit factor of 0.77 and a net profit of minus 20.07 dollars. DoorDash owns the minimum profit factor, 0.55\. The conclusion then calls minus 95.52 "the lowest average loss" and prints it as 95.52 dollars, without the minus. The average net profit on that model is positive 162.75 dollars. The minus 95.52 is one name's net profit, the worst one, and the sign is the result. Thirteen names in the four-degree table have a profit factor below 1\. Twelve do in the three-degree table. Twenty do in the two-degree table. The prose for the two-degree model says 19 names lost money, and 19%. The column contains 20 negative net profits, including CSGP at minus 0.86 dollars. Keep his sentence, and treat the count as a possible miscount. The source and the column disagree. ## Average dollars rise, and the average profit factor falls The horse race he publishes is an average of dollars. The profit factor he also publishes, the average of the hundred ratios, goes the other way. ![Average net profit rises from degree 2 to degree 4, while the average profit factor falls from 1.76 to 1.60 and the worst name's profit factor stays below 1](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/09/article_611-profit_factor_by_degree.png) Degree 2: average net profit 79.67 dollars, average win rate 52.89%, average profit factor 1.76, worst profit factor 0.37 (Walgreens). Degree 3: 110.33 dollars, 55.85%, 1.67, worst profit factor 0.48 (Walgreens again). Degree 4: 162.75 dollars, 55.45%, 1.60, worst profit factor 0.55 (DoorDash). Win rate peaks at degree 3, not at degree 4\. The floor he advertises does rise, from 0.37 to 0.48 to 0.55, and every one of those floors is a losing name. A higher floor under 1 is a smaller loss on the worst ticker. It is a loss. Sum the reconstructed gross profits and gross losses across names and you get a different profit factor, the one a single dollar pile would have. Degree 2: gross profit 26,122.01 dollars, gross loss 18,154.68, ratio 1.44\. Degree 3: 31,542.25 over 20,508.81, ratio 1.54\. Degree 4: 40,302.88 over 24,027.84, ratio 1.68\. Drop the cloned Baker Hughes row and the four-degree ratio is 36,542.58 over 22,434.49, which is 1.63\. The dollar-weighted ratio rises with degree. The average of the ratios falls. He ranked on the average of the dollars, then described the risk with the minimum ratio. Neither of those is 0.55 for the book, and the summed ratio still weighs a four-digit share price more than a single-digit one, because both sides of the fraction are raw dollars. Exelon is the one skipped row: degree 4 prints a profit factor of 1.00 next to a net profit of 0.13 dollars. A ratio of 1 forces a net profit of zero. The 0.13 is rounding, or the cell is wrong. Leaving it out moves the reconstructed net profit by that 0.13 dollars (16,275.04 against a column sum of 16,275.17). ## Booking Holdings, typed on the Baker Hughes line The four-degree average is an average of incompatible dollars, and one of the large dollars is a repeated row. Booking Holdings and Baker Hughes print the same five figures in Table 3: net profit 2,166.95 dollars, win rate 62.00%, profit factor 2.36, 50 closed trades, 19 days in the average trade. In the three-degree table Baker Hughes made 11.94 dollars, on a profit factor of 1.27, over 39 trades. The jump from 11.94 to 2,166.95 is 2,155.01 dollars. Summed net profit rises by 5,241.73 dollars from degree 3 to degree 4 (16,275.17 minus 11,033.44). That one jump is 41.1% of the entire improvement. MercadoLibre's jump is 886.14 dollars, 16.9% of the gap. Booking Holdings itself jumps by 665.70 dollars, 12.7%. Three rows account for 70.7% of the improvement. The other 97 names add 1,534.88 dollars between them, less than the Baker Hughes line alone. ![Horizontal bars of added dollar profit from degree 3 to degree 4: Baker Hughes 2,155 dollars, the other 97 names 1,535, MercadoLibre 886, Booking Holdings 666](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/09/article_611-where_the_extra_dollars_came_from.png) Blank the cloned row and the four-degree average falls from 162.75 to 141.08 dollars if you keep one hundred slots, or to 142.51 if you average the other 99\. Degree 3 sits at 110.33\. Degree 4 still leads on dollars, and the repeated row makes the lead look larger. Sirius XM, in the same four-degree table, made 2.38 dollars. Averaging 2.38 with 2,166.95 treats those two bets as the same size. The paper does not state the share count, the dollars of capital, or a volatility target. There is no buy-and-hold column next to a long-only book run through a Nasdaq advance. The best profit factor in the winning model is 3.82, on GE HealthCare, and that name has five closed trades. The worst is 0.55, on DoorDash, with 21\. A five-trade extreme and a 21-trade loss are both in the cross-section he summarizes with one minimum and one average. ## The summary cells are not all the column under them Table 3's average of 162.75 is the mean of the hundred net profits (162.7517). Table 4, the abstract, and the conclusion print 162.73\. Two cents. The standard-deviation row is the break. The sample standard deviation, the one with the n minus 1 divisor, matches the cells he got right, so that is the divisor in the row. Net profit prints 348 against a recomputed 348.02\. Profit factor prints 0.64 against 0.6406\. Trade count prints 9.01 against 9.012\. Win rate prints 54.40\. The same column, same divisor, is 6.76\. A standard deviation of 54 percentage points is not available to a series that runs from 41.51 to 80: put half the names at 41.51 and half at 80 and the spread is still about 19 points. The printed 54.40 is not this column. Average holding period prints a standard deviation of 1.12 against a recomputed 1.26\. In Table 2 the profit-factor standard deviation prints 0.85 against a recomputed 0.88 (0.876). Those three cells stay in his form here, and the columns do not produce them. Days in the market have a separate arithmetic error, and it is checkable without a new backtest. He multiplies the average trade count by the average holding period. Degree 4: 49.71 times 17.78 is 883.84 days, and he prints 883.8, which is 48.37% of the 1,827 days he counts in the sample. The average of the hundred products, trade count times holding period for each name, is 879.70 days, which is 48.15% of 1,827\. Degree 2: the product of the averages is 871.57 (he prints 871.56 in Table 4 and 871.5 in the text, and 871.57 divided by 1,827 is 47.7%). The average of the products is 867.14\. Degree 3: 872.40 against 867.04\. The gap is four to five days. Names that trade more often hold a bit less long, so the product of the averages overstates the average of the products. The minimum days in Table 4 are 115, 95, and 75\. Each of those is the minimum trade count times the minimum holding period, and the two minimums belong to different names. The name with the fewest days in a position is GE HealthCare in every model: 5 trades times 25 days is 125, 5 times 30 is 150, and 5 times 19 is 95\. His "only downside" of degree 4, an extra 11 to 12 days in the market, uses the products of the averages: 883.8 minus 871.6 is 12.2 days, and 883.8 minus 872.4 is 11.4\. On the average of the products the gap is 879.70 minus 867.14, which is 12.6 days, and 879.70 minus 867.04, which is 12.7\. About 13 days either way, on a 1,827-day sample. That comparison survives. It is the smallest fact in the paper, and it is the one he flags. ## The equations omit the window the word "moving" requires A moving regression has a length. Equations for the three models number the polynomial and the two-standard-deviation band. They do not number the lookback, and they do not define the time index as a bar count inside one. Degree 4 on five points interpolates. Degree 4 on two hundred points is a smooth bend. The ranking of degrees is a ranking of curves whose length is missing. The methods text also says the fit used early stopping on a validation set, a random search that tried degrees 5 and 6, and a search over how many standard deviations wide the bands should be, after which "various trading strategies" were applied. A polynomial fit by least squares has a closed form. Early stopping is a rule for an iterative fit. Either the optimizer is unnamed, or the paragraph does not describe the estimator that produced Tables 1 to 3\. The reported bands are all at two standard deviations, and the reported degrees are 2, 3, and 4\. No validation score appears. No trial count appears. The old article "How to Spot a Fake ML Trading Paper (house-style field guide)" treats a methods paragraph you cannot match to the estimator as a flag, and it treats citation sentences that do not match their papers the same way. Four of those sentences fail a title check. He writes that prior work used linear regression to predict financial assets and cites McMillan (2019) and Maroto (2018). McMillan's paper is "Cross-asset relations, correlations and economic implications." Maroto's paper is "Sharing or Limiting the Wealth? Coresidence, Parental Support, and Wealth Outcomes in Canada." He credits Chavarnakul and Enke with polynomial regression of stock prices. Their paper is "Intelligent technical analysis based equivolume charting for stock trading using neural networks." He credits Kim and Won with a hybrid of linear regression and polynomial terms for stock returns. Their paper is "Forecasting the volatility of stock price index: A hybrid model integrating LSTM with multiple GARCH-type models." The one precedent that is about polynomials, De Luna's 1998 projected polynomial autoregression for Treasury-bill yields, is a different object from the three band equations: an autoregression, not a band around a polynomial in a time index. Point-in-time index membership is undisclosed. GE HealthCare has five closed trades in a sample that starts in 2017, in every model. The old article ["The Backtest Integrity Checklist"](https://aligrithm.com/the-backtest-integrity-checklist/) is the list this writeup leaves blank: the window, the number of trials, the membership rule, a return with a capital denominator, and a holdout period. The 0.3% per side is stated and, on his account, already inside the net profit. Slippage on a next-open fill after a band break is not. The candlestick note under his Figure 1 says the green and red bars "symbolize the daily upper and lower price." A green candle is an up day and a red candle is a down day. Degree 4 is a lead on an average of unscaled dollars, inflated by a repeated row and still ahead if you delete that row. It is a loss on the average profit factor, 1.60 against 1.76 and 1.67\. The 0.55 in the risk sentence is DoorDash, at 1.82 dollars of gross loss per dollar of gross profit. A summed profit factor, rebuilt from his definitions, is about 1.68, or about 1.63 without the repeated row, and that rebuild still has no capital under it. Code the rule after someone states the window, the share count, and the trial list. Do not size a book off Table 4. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/09/article_611-visual-0.png) ## KEY POINTS - Cohen's four-degree polynomial band is named the winner on Nasdaq-100 names, 2017 through March 2024, at an average net profit of 162.75 dollars per name in Table 3\. The abstract, the conclusion, and Table 4 print 162.73\. The risk case is a profit factor of 0.55 and a minimum win rate of 41.51%. - Profit factor is gross profit over gross loss. DoorDash owns the 0.55: net profit minus 52.38 dollars, gross profit 64.02, gross loss 116.40, so 1.82 dollars lost per dollar gained. The average of the hundred profit factors is 1.60, the worst of the three models (1.76, then 1.67, then 1.60). - The three "minimum" headlines are three names. PDD Holdings owns minus 95.52 dollars (profit factor 0.59). Walgreens owns the 41.51% win rate (profit factor 0.77, net profit minus 20.07). DoorDash owns 0.55\. The conclusion calls minus 95.52 the lowest average loss and drops the minus. - Summed gross profit over summed gross loss rises with degree: 1.44, 1.54, 1.68\. Without the cloned row it is 1.63\. That dollar-weighted ratio favors degree 4 and still has no capital in the denominator. Thirteen four-degree names have a profit factor below 1\. The two-degree prose says 19 losers; the column has 20. - Booking Holdings and Baker Hughes are the same five cells in the four-degree table (2,166.95 dollars, 62%, profit factor 2.36, 50 trades, 19 days). Baker Hughes made 11.94 dollars at degree 3\. That jump is 2,155 dollars, 41% of the whole improvement in summed profit. Blank it and the four-degree average is about 141 dollars, still above degree 3's 110. - Days in the market are the product of the two averages: 871.6, 872.4, and 883.8 for degrees 2, 3, and 4\. The average of the per-name products is 867.1, 867.0, and 879.7\. Table 4's minimum days are 115, 95, and 75, each the product of two column minimums from different names. GE HealthCare's actual days in a position are 125, 150, and 95\. The win-rate standard deviation in Table 3 prints 54.40 against 6.76 on the column. The lookback is unstated, membership timing is unstated, and degrees 5 and 6 were tried and not shown. ## References - [Cross-asset relations, correlations and economic implications - McMillan (Global Finance Journal, 2019)](https://doi.org/10.1016/j.gfj.2019.02.003?ref=aligrithm.com) - [Polynomial Moving Regression Band Stocks Trading System - Cohen (Risks, 2024)](https://doi.org/10.3390/risks12100166?ref=aligrithm.com) --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ### 5.56 Spot-Futures Manipulation in Crypto: Pump Spot, Trap Perp Longs, Dump URL: https://aligrithm.com/spot-futures-manipulation-in-crypto-pump-spot-trap-perp-longs-dump/ Last updated: 2026-10-16T08:09:06.000Z Wang and Zhang study more than 300 altcoins on Binance from 2020 to 2024 and flag about 200 spikes as spot-futures manipulation. In a pooled regression those flagged spikes have forward returns 0.83% lower at two hours and 1.64% lower at 24 hours than other spikes, holding the pre-spike return, the basis change, the open-interest change, and volume fixed. The two-hour coefficient is the fragile one. Divide the printed -0.83% by its printed standard error of 0.41 percentage points and the t-statistic is 2.02\. A normal approximation to that t-statistic has a two-sided p-value of 0.043\. The 24-hour coefficient, -1.64% with standard error 0.54 percentage points, has a t-statistic of 3.04 and a p-value of 0.002\. No exchange, court, or wallet labels any of the 200 episodes as manipulation. Their schematic sets the coin at 10, then 40, then 60, then 30, with a fundamental value drawn at 10\. The estimated extra loss at a day is 1.64 percentage points. ## What this actually is You are about to buy a Binance alt perpetual because the coin's high stands 20% clear of the open, the close, and the low. The decision is whether that high is someone lifting a thin spot book so leveraged buyers will take the other side. Wang and Zhang keep days with that high, drop days whose low fell to 90% of the close or lower, drop days when Bitcoin spiked too, and then keep the subset where, on the way in, the perpetual cheapened against spot, open interest rose, and the long-short ratio of the largest-margin accounts fell. Picture a stall that marks up a thin item in the window, waits for a crowd to jam the side door with borrowed cash, sells the window stock into that crowd, and shorts the side door before marking the window down. The window is spot. The side door is the perpetual. In the 5-minute panel around these spikes, mean futures volume is 17.3 million dollars and mean spot volume is 4.2 million dollars. If the 1.64% gap holds, you skip the new perp long once those three signs are on the tape. Size a short off the schematic, a walk from 60 down to 10, and you hold a short in a coin that rallied the same day, on a path the authors assumed, on a day the filter required to hold its gain into the close. ## The flag is a daily high plus three pre-spike signs Membership in the 200 is a daily price rule plus three pre-spike signs. A day is a spike when the high clears 20% above the largest of the open, the close, and the low, and the low stays above 90% of the close. Wang and Zhang write that "the daily high price exceeds at least 20% of the maximum value among the open, close, or low." Read as a high greater than one fifth of that maximum, the rule would keep every day with a positive price, because the high is already at least that maximum. The next sentence says the threshold is there to keep large upward moves, so the reading used here is a 20% protrusion. They also drop an altcoin spike that lands on a Bitcoin spike day, and they give that rule no separate formula. The minute inside the day that they call the spike is unspecified. The event charts are clocked to a zero defined only by that word. $$ H \\geq 1.20 \\times \\max(O,\\, C,\\, L) $$ $$ L > 0.90 \\times C $$ Read it as two filters on the daily bar, in price units. H, O, C, and L are the high, the open, the close, and the low. The first line keeps a day when the high is at least 20% above whichever of the open, the close, and the low is largest. The second line drops a day whose low reached 90% of the close or lower. Worked day, prices in dollars. Open 1.00, high 1.40, low 1.00, close 1.10\. The max of open, close, and low is 1.10, and 1.20 times 1.10 is 1.32\. The high of 1.40 clears 1.32\. The low of 1.00 sits above 0.90 times 1.10, which is 0.99\. The day stays in. Move the low to 0.80 and leave the rest. The high of 1.40 still clears 1.32, and 0.80 is below 0.99, so the day goes out. They want the spike day to have held. On that sample they count about 2,000 spikes. The manipulation flag is a second cut, on the hours before the spike: the basis falls, open interest rises, and the top-trader long-short ratio falls. About 200 spikes survive. Binance funding updates every eight hours, so they leave it out of the flag and use the basis. $$ B = \\frac{F}{S} - 1 $$ Read it as the gap between the perpetual and spot. F is the perpetual price and S is the spot price, in the same units. Multiply by 100 and the gap is in percent. A negative gap means the perpetual is cheap to spot. Their Table 1 reports the basis in percent: mean -3.82, median -1.84, minimum -630.25\. A printed basis of -630.25% is a futures-over-spot ratio of 1 minus 6.3025, which equals -5.30\. That print cannot be a traded price. They publish the minimum, and the paper never says those ticks were dropped before the regressions. The means of 5-minute volume in the same table are 17.26 million dollars in the perpetual and 4.22 million in spot. Divide 17.26 by 4.22 and the ratio is 4.09, the "about four times" in their text. The medians are 0.188 million and 0.029 million. Divide 0.188 by 0.029 and the ratio is 6.48\. A typical bin is far smaller than the mean, and the perpetual is the busier book at both the mean and the median. The maximum futures volume in the table is 39,582 million dollars, so the mean is a tail. ## The schematic walks from 10 to 60\. One plotted episode gives back about a fifth The path they claim, and the path of the one coin they plot, are two different objects. Figure 1 of the paper is a drawing. The coin starts at 10, ends the pump at 40, peaks at 60 after retail arrives, ends the dump at 30, and drifts back to a fundamental value drawn at 10\. The dump is 30 points under the peak. The rest of the way, from 30 down to 10, is the drawn convergence. Table 2 does not estimate either leg. The prices are assumed. HOOKUSDT, 24 to 27 December 2023, is the episode they plot. Eyeball the futures line to the nearest cent: about 0.45 a day before the spike, about 0.54 at the spike, about 0.44 a day after. Take 0.54 minus 0.45, and divide by 0.45: the rise is 20%. Take 0.44 minus 0.54, and divide by 0.54: the decline is -18.5%. One episode, read off a chart. The average partial effect in the next section is 1.64% at a day. At the spike the basis on that chart is about -1%. With the perpetual at 0.54, spot is 0.54 divided by 0.99, which is 0.5455, or 0.55 cents above the perpetual. The spot-over-perp gap in the episode they chose is 0.55 cents on a coin that moved about 20%. Open interest runs from about 6 million dollars to about 15 million at the high and then fades. The top-trader position long-short ratio runs from about 1.3 down to about 1.0 at the high and back above 1.3 afterward. It stays above 1\. Large-margin accounts were less long at the top. The book, on this chart, stays net long. Funding, which they left out of the flag because it steps every eight hours, sits near -0.002% for the eight hours after the high. ![HOOKUSDT, 24 to 27 December 2023: futures price, basis, funding rate, open interest, and the top-trader position long-short ratio, clocked to the spike](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/09/article_465-hookusdt_spike.png) ## The extra loss is 0.83% at two hours and 1.64% at a day The load-bearing number is the gap in forward returns between flagged spikes and other spikes, with the pre-spike move held fixed. $$ R^{(h)} = \\alpha\_h + \\beta\_{B,h}\\,\\Delta B + \\beta\_{OI,h}\\,\\Delta OI + \\beta\_{V,h}\\,V + \\beta\_{R,h}\\,R\_{\\text{pre}} + \\beta\_{M,h}\\,M + \\varepsilon $$ Read it as a fitted forward return over horizon h, in decimal. R is that futures return. Delta B and delta OI are the pre-spike changes in basis and open interest, each measured over a window of the same length as h. V is average futures volume before the spike, in millions of dollars. R\_pre is the return over that same pre-spike window, in decimal. M is 1 when the three-way flag is on and 0 otherwise. Alpha is the fitted return when every regressor is 0\. Epsilon is the residual. They report five horizons: 30 minutes, 60 minutes, 2 hours, 4 hours, and 24 hours. The coefficient on M, converted to percent, is -1.18, -1.03, -0.83, -1.33, and -1.64\. Two stars at two hours, three stars on the other four. Worked 2-hour row. Set the basis change, the open-interest change, and volume to 0 so the arithmetic is visible. Zero is a reference point, and it is not the sample mean. Take a pre-window return of 20%, which is 0.20, and turn the flag on. The intercept is -0.0325, the pre-return slope is -0.2824, and the dummy is -0.0083. The fitted return is -0.0325 + (-0.2824)(0.20) + (-0.0083). The middle term is -0.05648\. The sum is -0.09728, or -9.73%. Flag off, drop the last term: -0.0325 - 0.05648 = -0.08898, or -8.90%. The gap is -0.0083, the dummy, recovered from the same arithmetic. Same exercise at 24 hours. The intercept is -0.0994, the pre-return slope is -0.1441, the dummy is -0.0164\. Flag on, with the same 20% pre-window return: -0.0994 + (-0.1441)(0.20) + (-0.0164) = -0.0994 - 0.02882 - 0.0164 = -0.14462, or -14.46%. Flag off: -0.12822, or -12.82%. The gap is 1.64 percentage points. The pre-return term does the heavy lifting. At two hours its slope is -0.2824 with standard error 0.0310, and 0.2824/0.0310 = 9.11\. A 20% pre-window rise lines up with 5.65 percentage points of giveback. The flag lines up with 0.83\. At 24 hours the same 20% lines up with 2.88 percentage points of giveback, against 1.64 from the flag. The chart puts those two contributions side by side. Whiskers are one standard error from Table 2\. The orange whisker is the standard error of the slope, multiplied by the same 20%. ![Partial effect of the manipulation flag, and the giveback from a 20% pre-window rise, at each horizon in Table 2. Whiskers are one standard error.](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/09/article_465-post_spike_partial_effects.png) Two other slopes belong next to the dummy. The open-interest change enters at +0.0156 at two hours, three stars. A larger pre-spike build in open interest lines up with a higher forward return in the pooled sample. The flag, which requires open interest to have risen, lines up with a lower forward return. The paper does not state the unit of the open-interest change, so 0.0156 stays in their units. Average volume prints as 0.0000 to four decimals and still carries stars. A coefficient smaller than 0.00005 in absolute value prints as 0.0000 to four decimals, and 0.00005 of return is half a basis point, so the per-million-dollar effect sits under half a basis point. The intercept runs from -1.46% at 30 minutes to -9.94% at 24 hours. It is the fitted return when the pre-window return is 0 and the other controls are 0\. A spike with a zero pre-window return is a strange spike. The intercept stays out of any sentence about the average dump. The regression keeps 1,267 events at 30 minutes and 1,209 at 24 hours. The text says about 2,000 spikes and about 200 flags. The three counts are never reconciled. R-squared is 0.20 at two hours and 0.16 at 24 hours. A normal approximation on the five printed t-statistics gives two-sided p-values, in horizon order, of 0.0001, 0.0054, 0.043, 0.003, and 0.002\. Five looks at a 5% Bonferroni cutoff means each test has to clear 0.01\. The two-hour result is the one that misses. The 24-hour result clears it. The five regressions share a sample, so this is a stress on the printed stars, and it leaves their coefficients as printed. ## Large accounts get less long into the high The second result is the slope of futures returns on large-account position changes, reported separately on flagged spikes and on the rest. $$ r = a + b\_{TT}\\,\\Delta LS^{TT} + b\_{TK}\\,\\Delta LS^{TK} + u $$ Read it as the futures return over one bar, in decimal, fit on the change in two long-short ratios over that same bar. Delta LS with the TT mark is the change in the long-short ratio of accounts in the top 20% by margin balance. Binance defines the group by margin. The HOOK panel is the position ratio, long position size over short position size. Delta LS with the TK mark is the change in the taker buy-versus-sell ratio. The b terms are slopes. u is the residual. They do not name the bar. Table 1 is built on five-minute data, and the row count matches that bar: four hours each side of the spike is 96 five-minute bars, and 19,175 divided by 96 is 199.7, in line with about 200 flags. A one-minute bar is 480 bars over the same eight hours, and 19,175 divided by 480 is 40 events, which misses the flag count. The other sample has 75,217 rows, 783.5 events at 96 bars. Table 2 kept 1,267 events at the 30-minute horizon. The long-short series is missing more often. Table 1 shows 1.71 million long-short rows against 2.24 million return rows. On the flagged sample the slope on the top-trader change is -0.0326, standard error 0.0084\. Divide 0.0326 by 0.0084 and the t-statistic is 3.88\. Episode fixed effects, and then average volume, leave it at -0.0327 or -0.0326\. On the other sample the slope is -0.0097, standard error 0.0060, t-statistic 1.62, p-value about 0.11 under the same normal approximation. A fixed effect puts a single star on -0.0095, and the star is gone at -0.0093 once volume is in the regression. The taker-ratio slope prints as -0.0000 in both samples. Table 1's taker ratio has a standard deviation of 79.6 and a maximum of 112,789, against a median of 0.98\. A printed zero is what that series produces. Worked bar, using the flagged slope of -0.0326\. The top-trader ratio falls by 0.10 in a five-minute bin, from 1.25 to 1.15\. The fitted contribution is -0.0326 times -0.10, which is +0.00326, or +32.6 basis points, in that bar. A rise of 0.10 lines up with -32.6 basis points. Into the high, the ratio falls while price rises. After the high, on the HOOK chart, the ratio rises while price falls. Both legs produce a negative slope. They do not report a lagged ratio. The slope is co-movement inside the episode. The flag already requires the ratio to fall before the spike, and the window includes those pre-spike bars. Part of the -0.0326 is the selection rule meeting the price path it was built on. A slope fit only on the bars after the spike would separate the two, and Table 3 pools both halves. The sample median of the ratio is 1.15 and the mean is 1.39\. The HOOK path bottoms near 1.0\. The shift is toward short, inside a book that stays net long on average and in the plotted episode. ## Sizing off 1.64 percentage points The coefficient supports a refusal. A Binance alt that has printed the daily spike and the three pre-spike signs has an estimated 24-hour futures return 1.64 percentage points below an otherwise similar spike, before any spread, fee, or funding. At two hours the gap is 0.83 percentage points, and that is the horizon a five-test cutoff drops. The old article ["How Manipulatable Are Prediction Markets? The Field Experiment"](https://aligrithm.com/how-manipulatable-are-prediction-markets-the-field-experiment/) measures a manipulation because the experimenter turned it on. Wang and Zhang keep the days that already match the story, then measure a further 1.64% at 24 hours. The old article "How to Spot a Fake ML Trading Paper (house-style field guide)" asks whether the label was fixed before the score was computed. Here the label is a threshold on basis, open interest, and the long-short ratio, chosen on this sample, then entered as a regressor on this sample. Funding on the HOOK chart steps to about -0.002% and stays there for eight hours after the high. Their description of the contract says that when the perpetual trades below spot, shorts pay longs. A rate of -0.002 percent is -0.00002 in decimal. On a 10,000 dollar short that is 0.20 dollars per eight hours. The futures line on the same chart moves about 20%. The paper reports no spread and no fee. Delisted coins, any trim of the -630% basis print, and a multiplicity adjustment are absent. Dhawan and Putnins study spot pump-and-dumps and the traders who enter them against a negative expected return. Li, Shin, and Wang map the spot version of the scheme. Wang and Zhang add the perpetual leg: short it while the spot inventory goes out. In their table the addition is 1.64 percentage points at a day. The schematic's trip from 60 to the drawn fundamental value of 10 is an assumed path. A short sized to that path is sized to the drawing. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/09/article_465-visual-0.png) ## KEY POINTS - Across 300-plus Binance altcoins, 2020 to 2024, a spike day has a high at least 20% above the max of open, close, and low, with the low still above 90% of the close, and with Bitcoin quiet that day. About 2,000 such days. About 200 also show a falling basis, rising open interest, and a falling top-trader long-short ratio before the high. - The flag's partial effect on the forward futures return is -1.18% (30 min), -1.03% (60 min), -0.83% (2 hr, t = 2.02, normal p = 0.043), -1.33% (4 hr), and -1.64% (24 hr, t = 3.04, normal p = 0.002). The pre-return slope at two hours is -0.2824 (t = 9.11). A 20% pre-window rise lines up with 5.65 percentage points of giveback, against 0.83 from the flag. - The schematic sets prices at 10, 40, 60, 30, and a fundamental value of 10\. HOOKUSDT, read off their figure to the nearest cent, runs about 0.45 to 0.54 to 0.44, a +20% rise and a -18.5% decline. Basis near -1% at the high puts spot 0.55 cents above a 0.54 perpetual. The long-short ratio bottoms near 1 and stays above 1. - Inside the flagged episodes the five-minute slope of futures returns on the change in the top-trader long-short ratio is -0.0326 (t = 3.88). A 0.10 drop in the ratio lines up with +32.6 basis points in that bar. The unflagged slope is -0.0097 (t = 1.62). The window includes the pre-spike bars the flag already required. 19,175 rows is about 200 events at 96 five-minute bars. - Table 1's basis minimum of -630.25% implies a futures price of -5.30 times spot under their own definition. The taker long-short ratio maxes at 112,789\. Regression event counts, 1,267 down to 1,209, are never lined up with "about 2,000" spikes. No costs, no holdout sample, no adjudication. Of the five horizons, the two-hour result is the one that misses a 0.01 Bonferroni line. - The usable decision is to skip the perp long when the three signs are already in. A short sized to the trip from 60 down to 10 is sized to the schematic. ## References - [A New Wolf in Town? Pump-and-Dump Manipulation in Cryptocurrency Markets - Dhawan, Putnins (Review of Finance, 2023)](https://doi.org/10.1093/rof/rfac051?ref=aligrithm.com) - [Cryptocurrency Pump-and-Dump Schemes - Li, Shin, Wang (Journal of Financial and Quantitative Analysis)](https://doi.org/10.1017/s0022109025000201?ref=aligrithm.com) - [Spot-Futures Manipulations in Cryptocurrency Markets - Wang, Zhang (ssrn-5125326)](https://papers.ssrn.com/sol3/papers.cfm?abstract%5Fid=5125326&ref=aligrithm.com) --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ### 10.21 Is Attention a Factor? WSB Herding, BERT Sentiment, and the Meme Confound URL: https://aligrithm.com/is-attention-a-factor-wsb-herding-bert-sentiment-and-the-meme-confound/ Last updated: 2026-10-16T08:09:07.000Z Huang and Shum Nolan buy three names. Each month they take the tickers WallStreetBets mentioned most, keep a name only when the posts were majority bullish, and hold the equal-weight book for the next month. Regress that book's daily percent return on the market, size, value, and momentum portfolios (the Fama-French-Carhart regression) and the intercept is 0.433 percent a day. Multiply by 260, which is the annualization that matches their printed figure, and the intercept is 112.58 percent a year from 2016 through 2022\. Remove GameStop and AMC from the regression and the same product is 64.22 percent. The event-time cousin of this book, an equal-weight of the three loudest bullish names, has a 20-day buy-and-hold abnormal return of -0.045 percent. A characteristic ranked beside value or momentum has to show up in the window the rank leaves the book holding. The holding-period intercept and the post-event drift describe different trades. ## What this actually is You run a US equity book. Someone offers a monthly sleeve: the three tickers a Reddit forum shouted about last month, kept only when more than half the posts were bullish, equal-weighted, turned over at month-end. The pitch is a four-factor intercept of 112.58 percent a year, still 64.22 percent once GameStop and AMC are out of the regression. You need the intercept that survives once the book is three names, and once you look at the 20 days after the shout, which is the window a month-end fill earns. The method counts posts that name a ticker, labels each post bullish or bearish with a language model retrained on retail comments, and buys last month's three loudest bullish names at the close. Picture a bar where the same three songs get requested all evening. The bartender starts the next round because the requests arrived in a pile. Attention herding is that pile. The open question is whether the round was already paid for by the time the pile peaked. Their event study is the tab for that question, and the tab for the three-name version comes back flat. If the intercept is a characteristic, you rank on last month's bullish mention count the way you rank on value, and you size the sleeve like any other three-name bet, with a cost line written down. If the intercept is a meme residual, you have funded a book whose leftover daily move is about 3 percent once GameStop and AMC are gone, and one name can do that in an afternoon. The old article ["Alternative Data and the Short-Horizon Decay Tax"](https://aligrithm.com/alternative-data-and-the-short-horizon-decay-tax/) is the lease on this feed. Forum attention is short-horizon data, and short-horizon data is what the next desk copies first. ## Two labels, then a fraction A post enters the book only after two labels, and the accuracy they publish was measured on a different website's comments. They start with 2.2 million WallStreetBets posts from 1 January 2016 through 31 December 2022\. A ticker counts if users wrote it with a dollar sign, or if the word matches a real symbol and that symbol shows up with a dollar sign somewhere in the sample. Posts that name two tickers are thrown out. A language model then answers two yes-or-no questions: is the post about a trade, and if so is it bullish. They split the job in two because a wrong sign does more damage, for their purpose, than a discarded post. The model is BERT, retrained on 12,000 hand-labelled comments from Investing.com. Out of sample, on that comment style, relevance accuracy is 80 percent and sentiment accuracy is 85 percent. FinBERT, the same architecture trained on filings, scores this slang worse. A dictionary such as Loughran and McDonald counts words one at a time, and Antweiler and Frank's message-board study treated word arrivals as independent. BERT is in the paper because the next word can flip the sign. A later check with a 2023 chat model adds little accuracy, and they print no figure for it. They publish no confusion matrix on WallStreetBets text. The 85 percent is an Investing.com number. Each surviving post carries a sign, 1 for bullish and 0 for bearish, and an upvote score equal to the maximum of zero and upvotes minus downvotes. They do not split bots from people. The feed in front of a user is the object, including whatever in it is fake. Daily ticker sentiment is the bullish share of that day's posts about the name. $$ S\_{i,t} = \\frac{N^{\\mathrm{pos}}\_{i,t}}{N^{\\mathrm{pos}}\_{i,t} + N^{\\mathrm{neg}}\_{i,t}} $$ Read it as: N-pos is the count of bullish posts about ticker i on day t, N-neg the count of bearish posts. S is a fraction between 0 and 1\. Counts divide counts, so the unit cancels. Worked example: 40 bullish posts and 10 bearish posts give 40/50 = 0.80\. The monthly screen uses the same ratio over the whole month and keeps a name only when bullish mentions exceed 50 percent. A stricter cut in the event study asks for a daily average above 0.70, or below 0.30 on the bearish side. Their variable glossary once writes the cutoff as "greater than 50," next to a score that lives between 0 and 1\. The figure captions use 0.50\. Treat "50" as a unit slip in the glossary, and use 0.50. Forum-wide sentiment is the same fraction over every labeled post that day. Smoothed over 20 days, it moves opposite the VIX, the S&P 500 implied-volatility index, from 2016 through 2021\. In a regression they do not print, a one-standard-deviation rise in sentiment lines up with a 0.1-standard-deviation drop in the VIX. A tenth of a standard deviation is a comovement with no size attached, and the fitted line is not in the paper. ## The intercept is a daily percent times 260 The portfolio question is what last month's three loudest bullish names earn over the next month, after the four standard factors. Rank names in month t by mention count. A ticker named twice in one post counts once. Keep names with a bullish share above 0.50\. Buy the top three at the close of the last session of month t, equal-weight them, and hold to the close of the last session of month t+1\. They rebalance monthly, citing noise and transaction costs, and then never subtract a cost. The regression uses daily percent returns so the time series is long. $$ R\_{p,t} = a + b\_{\\mathrm{MKT}} \\mathrm{MKT}\_t + b\_{\\mathrm{SMB}} \\mathrm{SMB}\_t + b\_{\\mathrm{HML}} \\mathrm{HML}\_t + b\_{\\mathrm{MOM}} \\mathrm{MOM}\_t + e\_t $$ Read it as: R sub p,t is the equal-weight return of the three names on day t, in percent. a is the daily intercept, in percent per day. The four slopes are loadings on the market, size, value, and momentum factor portfolios, which are themselves percent returns, so each slope is dimensionless. e is the residual in percent. Worked reading of their full-sample column, 1,743 days: a = 0.433, market loading 1.283 (t = 17.76), size loading 1.072 (t = 5.47), value loading -0.259 (t = -2.27), momentum loading 0.082 (t = 0.86). R-squared is 0.177, so 82 percent of the daily variance sits outside the four factors. That is what a three-name book looks like. They annualize by multiplying the daily intercept by 260. $$ a\_{\\mathrm{ann}} = 260 \\times a $$ Read it as: a is percent per day, 260 is a day count, and the product is percent per year. Worked: 260 times 0.433 = 112.58, their printed full-sample figure. Drop GameStop and the constant is 0.270, and 260 times 0.270 = 70.20\. Drop GameStop and AMC and the constant is 0.247, and 260 times 0.247 = 64.22\. Ending the sample in 2021, their constants 0.483 and 0.257 become 125.58 and 66.82, which they print, and the middle constant 0.284 becomes 73.84, which they do not print and which sits between those two. Times 252, the full-sample 0.433 and the ex-meme 0.247 are 109.12 and 62.24\. The printed 112.58 and 64.22 match 260\. Compounding the mean, (1 + a/100) to the power 260, minus 1, prints 207.52 percent and 89.92 percent. That compound stacks the mean and leaves a multi-percent daily residual out of the wealth math. Their published figures are the simple product. The full-sample t of 4.43 on the intercept is a two-sided p-value near 0.00001\. Three asterisks, under their legend, mean p under 0.001, and that star is earned. The ex-GameStop-and-AMC t of 2.79 on 1,197 days is a two-sided p-value of 0.0054 against a Student-t with 1,192 degrees of freedom (observations minus the intercept and four slopes). That clears 0.01 and misses 0.001, and the cell is marked with three asterisks. The value loading of -0.259, t = -2.27, is marked with two asterisks; the same reading gives p = 0.023, which is one asterisk under their legend. The old article ["How to Spot a Fake ML Trading Paper"](https://aligrithm.com/how-to-spot-a-fake-ml-trading-paper/) is the same habit: recompute the printed number before trusting the decoration around it. The intercept's standard error in the full sample is 0.433 / 4.43 = 0.0977 percent. Multiply by the square root of 1,743 and the residual scale is 4.08 percent a day. On the ex-GameStop-and-AMC sample the standard error is 0.247 / 2.79 = 0.0885 percent, and times the square root of 1,197 the scale is 3.06 percent a day. This treats intercept variance as residual variance over N. Factor portfolios with nonzero means push the true residual higher. A three-name book with a 3 percent daily residual can print a large intercept for a few years because one name's path is the residual. ## The size loading leaves with the meme days The question here is what the column "GME Excluded" did to the day count. A rule that swaps GameStop for the fourth name keeps the day count. Their day count falls from 1,743 to 1,240, a loss of 503 days, 28.9 percent of the sample. Drop AMC as well and the count falls another 43 days, to 1,197\. Ending in 2021 removes 251 days, 1,743 minus 1,492, which is a full year of sessions. Inside that split, the GameStop deletion removes 273 days from 2016-2021 and 230 days from 2022, leaving 21 days of 2022 in the ex-GameStop column. The extra 43 AMC days sit in 2016-2021, because the same 43-day gap appears in both panels. The column header says "GME Excluded" and never says the fourth name enters. The observation count is what deleting those days produces. Read 70.20 percent and 64.22 percent as the intercept on days the deleted name was absent from the top three. ![Four-factor intercepts, annualized as the daily intercept times 260, for 2016-2022 and 2016-2021, beside the SMB and HML loadings as GameStop and AMC leave the sample](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/09/article_452-ff4_alpha_and_loadings.png) The left panel is their intercepts. The right panel is the size loading. Size loading goes from 1.072 (t = 5.47) to 0.151 (t = 0.92) without GameStop, and to -0.039 (t = -0.26) without GameStop and AMC. The value loading goes from -0.259 to -0.463 to -0.564, and the last of those has t = -3.78\. Momentum stays insignificant in every column of the main panel (t = 0.86, then -0.58, then -1.53). Market loading stays near 1.3\. What remains is a high-beta growth book with an intercept of 0.247 percent a day, on a sample that skipped the days the meme names were the forum's topic. The bearish twin, top three names by attention with majority-negative posts, prints one column three times, because GameStop and AMC never sat in that book. The constant is -0.106 percent a day, t = -1.49, a two-sided p-value of 0.14\. Apply their 260-day rule and that constant is -27.56 percent a year; they do not print the annualized figure, so -27.56 is the product, not a number from the table. The bearish book does load on market (1.119), size (0.778), and momentum (-0.319, t = -3.88). R-squared is 0.289, higher than the bullish book's 0.177\. They point at Avila, Martineau, and Mondria on an optimism bias in social-network retail, and at Robinhood's ban on short sales, to explain the weak bearish intercept. A characteristic is a spread. They never report bullish attention minus bearish attention. With a short-leg t of -1.49, that spread has no estimated price. ## The 20-day window and the monthly intercept The question here is whether the 20 days after peak attention earn the 5.35 percent a month implied by the intercept. Their event is stricter than the monthly book. A name has a complete attention day if, between midnight and 3:30 pm Eastern, it ranks first in post count and first in upvotes. During the Robintrack window, May 2018 through August 2020, that rule fires 305 times across 125 tickers, 207 bullish days and 98 bearish days. Robintrack is the public series of how many Robinhood accounts held each stock. They prefer a post count to Google search volume, the attention series in Da, Engelberg, and Gao, because that index is rescaled when the window updates. Abnormal return, following the market-adjusted definition they take from Barber, Huang, Odean, and Schwarz, and the bad-model warning in Fama (1998), is the stock's percent return minus the market's percent return. $$ AR\_{i,t} = R\_{i,t} - R\_{m,t} \\qquad BHAR\_{0,T} = \\prod\_{t=1}^{T} (1 + AR\_{i,t}) - 1 $$ Read it as: R is the stock's simple return, R sub m the market's simple return. Use decimals inside the product and percent in their table. AR is a difference of returns, same units as R. BHAR is the compounded abnormal return over the window. Worked illustration with round numbers: day 1 the stock is up 2 percent and the market is up 1 percent, so AR is 1 percent; day 2 both are up 0.5 percent, so AR is 0\. The product form gives 1.01 times 1.00, minus 1, which is 1.00 percent. Their table caption writes BHAR as the product of one plus AR minus the product of one plus the market return. On the same two days that second product is 1.01 times 1.005 = 1.01505, and 1.01 minus 1.01505 is -0.505 percent. If AR is already market-adjusted, the caption removes the market a second time. The printed cells cannot be recomputed from here. Keep the cells, and flag the caption as a possible wording error. On the single loudest bullish name, the run-up from 10 days before the event to the event close is a BHAR of 9.65 percent, standard error 3.91, t = 2.47\. The 20 days after the event, in the winsorized table, are a BHAR of 1.4268 percent, standard error 2.7713, t = 1.4268 / 2.7713 = 0.51\. A 95 percent interval is 1.4268 plus or minus 1.96 times 2.7713, which runs from -4.00 percent to 6.86 percent. Zero sits inside it. Barber, Huang, Odean, and Schwarz report -4.7 percent over 20 days on their Robinhood herding rule. That figure sits 2.21 standard errors below 1.4268, outside the interval above. Account holdings do jump: user change on the event day averages 2,334 accounts, standard error 757, t = 3.08\. The spillover into Robinhood is in the holdings. The subsequent abnormal return is a point estimate inside a wide interval. The prose quotes a post-event BHAR of 1.95 percent for the bullish sample and -0.89 percent for the bearish sample, and calls the 1.95 insignificant "per Table 4." A footnote says the winsorized BHAR is 1.42 percent and pins that 1.42 on Table 4\. Table 3, the single-name winsorized panel, is the table whose day-20 cell is 1.4268\. Table 4, the equal-weight of the top three bullish names, has a day-20 BHAR of -0.045, standard error 2.478\. The footnote cites the wrong table. The 1.95, the 1.43, and the -0.045 are three different objects, and the -0.045 is the cell that matches the portfolio they trade. They also annualize the bearish -0.89 percent as -10.68 percent. That product is -0.89 times 12, which treats a 20-day window as a calendar month stacked twelve times. The events overlap, so the twelve months are not separate trades. ![Reported 20-day buy-and-hold abnormal returns: the winsorized single-name cell with its 95 percent interval, the prose figures of 1.95 and -0.89, Barber user-change herding at -4.7, and the post-2020 sample at 4.68 and -6.05](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/09/article_452-post_event_bhar.png) The whisker is the single standard error they print for a 20-day BHAR. From September 2020 through December 2022, with GameStop removed, they report a bullish 20-day BHAR of 4.68 percent, a peak of 8.62 percent on day 13, and a bearish BHAR of -6.05 percent. The giveback from the peak to day 20 is 8.62 minus 4.68 = 3.94 percentage points. Of those events, 155 fall in late 2020, 249 in 2021, and 325 in 2022, and 325 / 729 = 44.6 percent, the share they print. No standard error is attached to 4.68 or to 8.62. Their monthly intercept of 0.247 percent a day is 0.247 times 260 / 12 = 5.35 percent per month under the same 260-day rule. The winsorized single-name drift is 1.43 percent over a similar horizon. The gap is about 3.9 percentage points, and the three-name event drift is -0.045 percent. One bridge would close that gap with no arithmetic error: if the same names stay in the top three next month, the holding period contains the next herd's pre-event run-up, the 9.65 percent with t = 2.47\. The flat post-event window is a different object. They do not report the month-to-month repeat rate. The sample split with Barber, Huang, Odean, and Schwarz is a definition, and it is why a reversal in their paper and a flat line in this one can both be real. $$ \\mathrm{UCR}\_{i,t} = \\frac{\\mathrm{UserCount}\_{i,t}}{\\mathrm{UserCount}\_{i,t-1}} $$ Read it as: user count is the number of Robinhood accounts holding stock i. The ratio is dimensionless. Barber and coauthors call a herding day the top 0.5 percent of this ratio. Worked example: an account count that goes from 100 to 200 is a ratio of 2.00 and a change of 100 accounts. A count that goes from 100,000 to 102,500 is a ratio of 1.025 and a change of 2,500 accounts. The ratio picks the first stock. A rank on post count picks names people already hold. Their attention sample has an average market cap of 175 billion dollars, against 2.23 billion in the Robinhood-ratio sample. 175 / 2.23 = 78\. Large-cap names, 10 billion and up, are 60.57 percent of the stock-day observations and 35.44 percent of the unique names, so the same large names recur. Two names in furniture and home stores account for 9.24 percent of the observations. Average account change on their bullish event days is 2,497, against 900 in the ratio sample. Robinhood blocks short sales, so a bearish post has a weak path into account changes for a mechanical reason, and the bearish bars are where that shows. Influencer posts are a smaller version of the same event. A user in the top 5 percent of trailing-twelve-month upvotes is labeled an influencer for the next month. A post clears the bar if its upvotes exceed that cutoff user's upvote count. In the Robintrack window they have 170 bullish names and 115 bearish names. Both groups are up under 5 percent in the 10 days before the post. Pedersen's claim is that the influencer blesses a view already moving. The pre-event gain matches that timing. After the post the paths split, then fade. A bearish post cuts the prior gain in half over 20 days, and the gain is gone by day 32\. A bullish post keeps rising through day 15, and by day 40 the buy-and-hold abnormal return is back at the event-day level. A path that round-trips inside 40 days is the short-horizon lease in the old article ["Alternative Data and the Short-Horizon Decay Tax"](https://aligrithm.com/alternative-data-and-the-short-horizon-decay-tax/). ## Write the cost line they skipped The question here is how much of 64.22 percent a stated cost removes, and the paper leaves the turnover blank so the answer has to be a bound. Let f be the fraction of the book replaced at month-end. Selling fraction f and buying fraction f trades 2f of net asset value. Let kappa be the one-way cost in percent. $$ c\_{\\mathrm{month}} = 2 f \\kappa \\qquad c\_{\\mathrm{year}} = 12 \\times c\_{\\mathrm{month}} $$ Read it as: f is a fraction of the book, dimensionless. Kappa is a one-way cost in percent of notional. The 2 counts the sale and the purchase. The month cost is in percent of the book, and times 12 it is percent per year. Worked upper bound, full replacement, f = 1\. At 10 basis points one way, kappa = 0.10, the month cost is 0.20 percent and the year cost is 2.40 percent. Against 64.22 that leaves 61.82\. At 50 basis points one way the year cost is 12 percent and the leftover is 52.22\. At 200 basis points one way, a squeeze-name cost on a stock far smaller than this sample's 175 billion dollar average, the year cost is 48 percent and the leftover is 16.22\. Replace one of the three names, f = 1/3, and 10 basis points costs 0.80 percent a year. They never report f. Full replacement is the expensive case, and 64.22 percent is still there afterward. Search the portfolio section for a transaction cost, a slippage number, a Sharpe ratio, or a net figure. None is there. That is red flag three in the old article ["How to Spot a Fake ML Trading Paper"](https://aligrithm.com/how-to-spot-a-fake-ml-trading-paper/): a gross intercept with no cost model. On this book a punitive haircut still leaves a double-digit intercept, so the missing line is a disclosure hole. The book is three names. The ex-meme column deletes 503 days. The three-name event drift is -0.045 percent over 20 days, standard error 2.478\. The repeat rate that would reconnect that drift to a 5.35 percent monthly intercept is absent. Attention, measured this way, is a short-horizon description of which large names the forum is shouting about. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/09/article_452-visual-0.png) ## KEY POINTS - Ticker sentiment is the bullish share of posts about the name. Forty bullish and ten bearish is 0.80\. The published 85 percent sentiment accuracy and 80 percent relevance accuracy are out-of-sample on Investing.com comments, not on WallStreetBets text. The model was retrained on 12,000 of those comments. - The four-factor intercept is 0.433 percent a day. Times 260 that is 112.58 percent a year. Without GameStop and AMC it is 0.247 percent a day, 64.22 percent a year. The ex-meme t of 2.79 is a p-value of 0.0054, which misses the 0.001 bar their three asterisks claim. The full-sample t of 4.43 does clear that bar. - "GME excluded" drops 503 days out of 1,743\. The size loading falls from 1.072 to -0.039 once GameStop and AMC are out. The value loading goes to -0.564\. The bearish book's intercept is -0.106 percent a day, t = -1.49, and they never report the long-minus-short spread. - The single-name 20-day drift is 1.4268 percent in the winsorized table, t = 0.51, interval -4.00 to 6.86\. The prose says 1.95 percent. The three-name event drift, the cell that matches the portfolio, is -0.045 percent. A footnote pins 1.42 percent on the wrong table. The caption's BHAR wording subtracts the market a second time. - Under their 260-day rule the ex-meme intercept is 5.35 percent a month. The post-event window prints 1.43 percent on one name and -0.045 percent on three. A full monthly replacement at 50 basis points one way removes 12 percent a year and leaves 52\. The repeat rate that could reconnect the intercept to the pre-event run-up of 9.65 percent is not reported. - Influencer posts arrive after a gain of under 5 percent. Bullish posts round-trip the post-event piece by day 40\. Bearish posts give the gain back by day 32\. That is a short-horizon path, the lease in the old article ["Alternative Data and the Short-Horizon Decay Tax"](https://aligrithm.com/alternative-data-and-the-short-horizon-decay-tax/). ## References - [Pedersen, Game On: Social Networks and Markets, Journal of Financial Economics (2022)](https://doi.org/10.1016/j.jfineco.2022.05.002?ref=aligrithm.com) - [Huang and Shum Nolan, Social Network Attention Herding, Sentiment, and Markets](https://papers.ssrn.com/sol3/papers.cfm?abstract%5Fid=4798796&ref=aligrithm.com) --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ### 10.16 ML in the Cross Section: Avramov's Companion to the DDA3600 Spine URL: https://aligrithm.com/ml-in-the-cross-section-avramovs-companion-to-the-dda3600-spine/ Last updated: 2026-10-16T08:09:08.000Z On the non-microcap book, 0.50% times the Gu-Kelly-Xiu neural net's turnover of 0.869 costs 0.4345% a month. Its Fama-French six-factor alpha on that book is 0.312%. The ticket is larger than the alpha. Instrumented principal components, the linear model that lets betas move with firm characteristics, posts a six-factor alpha of 0.613% against a cost of 0.565% and clears by 0.048 percentage points, 4.8 basis points a month. Doron Avramov's lecture is the algebra under that split. The old article ["Does ML Actually Help Asset Pricing? Kelly's 20%, Not 2–3×"](https://aligrithm.com/does-ml-actually-help-asset-pricing-kellys-20-not-2-3x/) already showed the headline Sharpes are fragile. This is the companion ledger, plus the mean-squared-error formula that forces the shrink before anyone sorts a portfolio. ## What this actually is You run a US equity book. Someone offers a signal built from 94 firm characteristics and a neural net, and quotes a long-short Sharpe near 1\. You need the Sharpe that remains once the book is cut to stocks above the 20th NYSE size percentile, and once you pay half a percent for every unit of the book you turn over in a month. The lecture does two concrete things. It writes the mean squared error of ordinary least squares as residual variance times the trace of the inverse of X-transpose X, which blows up when characteristics move together, and that blow-up is the reason to shrink. It then files the machines traders argue about into two pricing stories: forecast each stock's beta on a few factors, or build one discount factor as a portfolio of the stocks. Picture a surveyor with two tape measures aimed almost the same way. Each tape is fine alone. A small wobble in one reading becomes a large wobble in how much length you assign to each direction. Shrinkage is the surveyor pulling that split back toward a dull answer. Beta pricing and the discount factor are two choices of what "dull" was allowed to mean before the tapes came out. If the ledger holds, you keep the linear instrumented-factor sleeve on names large enough to trade. If the ledger is wrong, you have skipped a nonlinear premium in liquid stocks, and 4.8 basis points a month will not pay for that miss. The old article ["The SDF View: One Equation Behind Every Factor Model"](https://aligrithm.com/the-sdf-view-one-equation-behind-every-factor-model/) is the same discount factor from the Kelly lecture. Avramov writes the cost column beside it. ## The trace that blows up Two predictors at a correlation of 0.9 make the OLS coefficient error 5.263 times the orthogonal case. That ratio is the whole case for a penalty. Avramov starts from ordinary least squares because every later machine is a repair of it. With spherical residuals of variance sigma-squared, and a fixed predictor matrix X with T rows and M columns, the mean squared error of the coefficient vector is residual variance times a trace. $$ \\mathrm{MSE}(\\hat\\beta) = \\sigma^2 \\, \\mathrm{tr}\\!\\left\[(X'X)^{-1}\\right\] $$ Read it as: sigma-squared is the residual variance, in return-squared units. X-transpose X is the M by M Gram matrix of the predictors. The trace of its inverse adds one term per coefficient, and each term is large when that direction in X is thin. Mean squared error here is the expected sum of squared coefficient errors, so the unit is coefficient-squared. The old article ["The Mathematics of Machine Learning, for Traders"](https://aligrithm.com/the-mathematics-of-machine-learning-for-traders/) is the general split of that error into variance plus bias squared. OLS is unbiased under the lecture's assumptions, so the trace is the whole error. Work it with two predictors and T = 240 months, columns scaled so X-transpose X equals 240 times the correlation matrix. At zero correlation the trace is 2/240 = 0.008333\. At correlation 0.9 the eigenvalues of the correlation matrix are 1.9 and 0.1, and the trace is (1/1.9 + 1/0.1) / 240 = 10.5263 / 240 = 0.04386\. The ratio of the two traces is 1 / (1 - 0.9 squared) = 1 / 0.19 = 5.263\. Same residual variance, 5.263 times the coefficient error. At correlation 0.99 the factor is 1 / (1 - 0.99 squared) = 50.25\. Firm characteristics live in that region. Size, value, profitability, and investment do not point in orthogonal directions. ![Trace of the inverse Gram matrix for two predictors over 240 months, OLS exploding as correlation approaches 1, ridge with penalty 24 staying flatter](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/09/article_447-ols_trace_vs_correlation.png) The orange line is a ridge penalty of 24, which is 10% of T, added to the diagonal before the inverse. At correlation 0.9 its trace is 0.0229, against the OLS trace of 0.04386, so the variance piece falls to about half. Read the orange line as the variance piece only. Ridge is biased, so its full mean squared error also carries a squared-bias term the trace leaves out. The chart's job is the blue line: the variance was about to explode, and a small diagonal load stops the explosion. ## The same shrinkage, written as a discount factor The question here is where the penalty sits inside a pricing equation, and which first-order condition the lecture's slope formula obeys. Hoerl and Kennard's ridge estimator adds lambda times the identity to X-transpose X before inverting. As lambda goes to 0 you recover OLS. As lambda goes to infinity every slope goes to 0\. The useful object is the weight on each principal component of the fitted values: eigenvalue j over (eigenvalue j plus lambda). In the two-predictor example the eigenvalues of X-transpose X are 24 and 456\. With lambda = 24 the small component is kept at 24/48 = 0.5, and the large one at 456/480 = 0.95\. Ridge barely touches the direction the data can see, and it cuts the thin direction in half. Avramov then writes the same penalty as a prior on the stochastic discount factor. The kernel, in the Hansen-Jagannathan projection the lecture uses, is one minus a portfolio of demeaned returns. $$ M\_t = 1 - b'(r\_t - \\mu) = 1 - \\mu' V^{-1} (r\_t - \\mu) $$ Read it as: M\_t is the discount factor, a pure number. r\_t is the vector of excess returns, mu is their mean, same units as return. b is the vector of portfolio weights. V is the covariance matrix of those returns. The second equality sets b equal to V-inverse times mu. One asset, V = 0.04 (monthly variance), mu = 0.01\. Then b = 0.01 / 0.04 = 0.25\. A month that realizes 0.06 has a demeaned return of 0.05, the portfolio inside M is 0.25 times 0.05 = 0.0125, and M = 1 - 0.0125 = 0.9875\. The discount factor is low when the high-mean asset pays off. Check the first-order condition against the lecture's own line. The population identity E\[M r\] = mu - V b. Plug in b = V-inverse mu and that product is zero: the kernel prices the excess returns. The lecture writes the condition as E\[M (r - mu)\] = 0\. That residual equals -mu, which is -0.01 in the example, and the only b that sets it to zero is b = 0\. The slope formula in the slide is the standard one. The residual printed next to it does not deliver that slope. Keep the formula, and treat the printed Euler residual as a slip in the notes. Kozak, Nagel, and Santosh put a normal prior on b and the posterior mean is ridge. $$ E(b) = (V + \\lambda I)^{-1} \\hat\\mu, \\qquad \\lambda = \\frac{s^2}{T \\sigma^2} $$ Read it as: mu-hat is the sample mean return. s-squared is the trace of V, a sum of variances. T is the number of months. Sigma-squared here is the prior's confidence knob, a different object from the residual variance in the OLS section. Lambda is the ridge penalty, in variance units. With V = 0.04, mu-hat = 0.01, T = 120, and sigma-squared = 1/120, lambda = 0.04 / (120 times 1/120) = 0.04\. The posterior weight is 0.01 / (0.04 + 0.04) = 0.125, half the OLS weight of 0.25\. You gave up unbiasedness and cut the position in half. The lecture's eta parameter decides whether that prior's expected squared Sharpe grows with the number of components. Eta = 2, the case that makes the prior on b independent of V, sets the expected squared Sharpe equal to sigma-squared. In the example that is 1/120 per month, and the root-mean-square Sharpe is the square root of 0.1, which is 0.316 annualized. Eta = 1 multiplies by the number of components. Feed the same sigma-squared into 94 managed portfolios, the characteristic count the lecture hands Kozak, and the expected squared Sharpe is 94 / 4.8 = 19.583 per month. The root-mean-square Sharpe is the square root of 19.583 times 12, equal to 15.33 annualized. That 15.33 is an illustration of their formula under a monthly covariance. The lecture does not print it. You would not sign the eta = 1 prior at that width. Eta = 2 is the prior that stays humble, and it is the prior that lines up with ridge. ## Beta pricing versus the kernel The question is which restriction each famous machine imposes. They run on different samples, so a ranking is the lecture's composite. Avramov sorts them on one page. Gu, Kelly, and Xiu (2020) is a reduced form: a three-layer net, 32, 16, and 8 neurons, predicting the next return from firm characteristics and macro variables, with no pricing restriction. The predictor count is (8 + 1) times 94 characteristics, plus 74 industry dummies, which is 920\. A toy net with 4 inputs, 5 hidden units, and 1 output has (4 + 1) times 5 plus 6 = 31 parameters, their own count, so you can see the scaling before the real net. Training is 1957 to 1974, validation 1975 to 1986, and the out-of-sample window is 1987 to 2017. Kelly, Pruitt, and Su price with betas. The conditional loading is linear in characteristics, and the factors are latent. $$ r\_{i,t+1} = \\beta\_{i,t}' f\_{t+1} + \\varepsilon\_{i,t+1}, \\qquad \\beta\_{i,t}' = x\_{i,t}' \\Gamma\_\\beta $$ Read it as: r is stock i's excess return next month. f is a K-vector of latent factor returns. beta is that stock's loading, dimensionless if f is a return. x is the stock's characteristics, including a one for the intercept. Gamma-beta is an M by K matrix shared across stocks. No alpha in this line: the characteristics move the loading, and the loading times the factor is the whole expected return. Their implementation uses 6 factors and 94 characteristics, re-estimated every month, out of sample 1987 to 2017\. Work one factor. A stock with x = (1, 0.5) and Gamma-beta = (0.8, -0.4) has beta = 0.8 + 0.5 times -0.4 = 0.6\. A factor realization of 0.02 (2% that month) contributes 0.6 times 0.02 = 0.012, or 1.2%, to the stock. A stock with the size coordinate at 0 has beta 0.8 and a contribution of 1.6%. The characteristic changed the loading. It did not add a separate cash intercept. Gu, Kelly, and Xiu (2021) replace the linear map x-transpose Gamma with a neural net, two hidden layers of 32 and 16 units, and 5 latent factors. Same beta-pricing story, nonlinear loading. They call it a conditional autoencoder. Out of sample runs 1987 to 2016. Chen, Pelger, and Zhu stay on the kernel side and let an adversary pick the moments. The discount factor is one minus a portfolio whose weights are a network of characteristics and a macro state. $$ \\min\_w \\max\_g \\ \\frac{1}{N} \\sum\_{j=1}^{N} \\left\\| E\\!\\left\[\\left(1 - \\sum\_{i=1}^{N} w(I\_t, I\_{t,i}) R^e\_{t+1,i}\\right) R^e\_{t+1,j}\\, g(I\_t, I\_{t,j})\\right\] \\right\\|^2 $$ Read it as: w is the discount-factor portfolio, a weight per stock, a function of macro information I\_t and stock information I\_{t,i}. g is the adversary's instrument, also a function of those inputs, scaled to sit between -1 and 1\. The object inside the norm is a pricing error: discount factor times excess return times instrument, averaged over time, then squared and averaged across stocks. The modeler picks w to shrink those errors. The adversary picks g to inflate them. They alternate. Hansen and Jagannathan (1997) motivate the minimax: when the kernel is only a proxy, minimizing the largest pricing error pulls it toward an admissible kernel in least squares. A one-month sketch of the portfolio inside M, with the expectation left to the sample average: weights 0.6 and -0.4, excess returns 0.03 and -0.01, portfolio return 0.6 times 0.03 plus -0.4 times -0.01 = 0.022, so M = 0.978\. Their benchmark uses on the order of 10,000 stocks and 8 instruments, about 80,000 instrumented assets, so the adversary can only "win" on mispricing that hits a large share of names. Training is 1967 to 1986, validation 1987 to 1991, out of sample 1992 to 2016, on 46 firm characteristics plus 178 macro predictors. Those windows do not match, and the stock sets do not match either. The Gu-Kelly-Xiu, IPCA, and autoencoder samples cover 21,882 NYSE, AMEX, and Nasdaq names, 5,117 to 7,877 in a given month. Chen-Pelger-Zhu cover 7,904 names, 1,933 to 2,755 a month. Read a ranking of their full-sample spreads as that composite. ## The book you can trade The question is which of those spreads survives the names an institution can hold, after a turnover charge. The figures are Avramov, Cheng, and Metzker's, printed as the lecture's appendix. The lecture's reason for throwing out small names is the anomaly record it cites. Harvey, Liu, and Zhu count 296 published anomalies and judge 27% to 53% of them false discoveries. Hou, Xue, and Zhang find that 82% of 452 anomalies lose significance once microcaps are out and the portfolios are value-weighted. On the full sample the lecture's own bars read 0.95% a month for IPCA (six-factor alpha 0.62%), 1.56% for the neural net (alpha 0.92%), and 2.18% for the adversarial kernel (alpha 1.87%). The alpha column matches the printed table at two decimals: 0.624, 0.916, and 1.867\. Sharpes in that table are 0.967, 0.944, and 1.225, against a market Sharpe of 0.527\. A market Sharpe of 0.53 is an annualized equity number, so read the Sharpe column as annualized. Read the return and alpha columns as percent per month: dividing the neural net's full-sample alpha 0.916 by its turnover 0.976 recovers the printed break-even of 0.94, and that division only lands if alpha and turnover share a month. Drop stocks below the 20th NYSE size percentile and the table changes shape. Neural-net Sharpe goes from 0.944 to 0.644\. Six-factor alpha goes from 0.916 to 0.312, a 66% cut. The adversarial kernel goes from Sharpe 1.225 to 0.839, and from alpha 1.867 to 0.548, a 71% cut. The conditional autoencoder's alpha goes from 0.746 to 0.387, and (0.746 - 0.387) / 0.746 = 48%, which is the lecture's own "48% lower" on that method. IPCA goes from Sharpe 0.967 to 0.978, and from alpha 0.624 to 0.613, a 1.8% cut. The linear beta model's alpha barely moves when the microcaps leave. The lecture also states that value-weighting the neural-net spread cuts performance 48% relative to equal-weighting. That 48% is the lecture's sentence. The figure behind it lacks labels tight enough to recompute the percent from the bar heights. Turnover is why the alpha that remains fails to cover the ticket. One-way monthly turnover is 0.976 for the neural net, 1.664 for the adversarial kernel, 1.186 for IPCA, and 1.565 for the autoencoder, against under 0.10 for size and value in the lecture's comparison set. The non-microcap break-evens, alpha divided by turnover, print as 0.36, 0.34, 0.54, and 0.26 percent per unit of turnover. Recomputed: 0.312 / 0.869 = 0.359, 0.548 / 1.625 = 0.337, 0.613 / 1.130 = 0.542, 0.387 / 1.478 = 0.262\. The lecture's rounding matches. $$ \\text{cost} = 0.50\\% \\times \\text{turnover}, \\qquad \\text{net} = \\text{FF6} - \\text{cost} $$ Read it as: 0.50% multiplies one-way turnover, so a turnover of 1.130 produces a charge of 0.565%. Turnover is a fraction of the book traded in the month. FF6 is the six-factor alpha in percent per month. Net is alpha left after that charge, same units. Non-microcap turnover is 0.869, 1.625, 1.130, and 1.478\. Costs are 0.4345, 0.8125, 0.565, and 0.739 percent per month. Nets are 0.312 - 0.4345 = -0.1225, 0.548 - 0.8125 = -0.2645, 0.613 - 0.565 = +0.048, and 0.387 - 0.739 = -0.352\. One method is positive. ![Grouped bars of six-factor alpha against a 0.50 percent times turnover cost, non-microcaps, for the neural net, the adversarial kernel, IPCA, and the conditional autoencoder](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/09/article_447-ff6_vs_turnover_cost.png) The lecture's chart of the same comparison labels the bars 0.31 against 0.43, 0.55 against 0.81, 0.61 against 0.57, and 0.39 against 0.74\. Same arithmetic, one-decimal labels. IPCA is the only bar pair where alpha stands above the cost. The full-sample neural-net break-even of 0.94% looked like it could pay this rate. That 0.94 includes the microcaps. On the tradable book the break-even is 0.36%, and 0.50% is above it. The edge also shows up in the months when arbitrage is expensive. On the neural-net long-short the lecture prints a high-sentiment coefficient of 1.534 (t = 2.43) and, in a sibling specification, a high-VIX coefficient of 1.851 (t = 2.85). IPCA and the autoencoder, the lecture says, show low time-series variation and mixed evidence in those states. The deep reduced form and the adversarial kernel earn more of their keep when the market is hard to trade. That is the same fact as the microcap result, moved from the cross-section to the calendar. ## A stock pick inside the industry The question is whether the forecast is picking stocks or picking industries. For the neural net, the lecture's split says stocks. Define the unconditional long-short from the gap between each stock's predicted return and the predicted market, scaled so the positive weights sum to one. $$ WML\_{t+1} = \\frac{1}{H\_t} \\sum\_{i=1}^{N\_t} (\\hat R\_{i,t} - \\hat R\_{m,t}) R\_{i,t+1}, \\qquad H\_t = \\frac{1}{2} \\sum\_{i=1}^{N\_t} |\\hat R\_{i,t} - \\hat R\_{m,t}| $$ Read it as: R-hat is the model's predicted return, R is the realized return, both in the same units. The market prediction is the cross-sectional average of the predictions. H is half the sum of absolute gaps, so dividing by H turns the gaps into long-short weights. The lecture then splits each gap into a within-industry piece and an across-industry piece, and the two pieces add back to the unconditional spread. Three stocks, predictions 4%, 2%, and 0%, so the market prediction is 2% and H = 2\. Realized returns 5%, 1%, and -3%. Unconditional payoff: (2/2) times 5 + (0/2) times 1 + (-2/2) times -3 = 8 percent that month. Put the first two stocks in one industry, predicted industry mean 3%, and the third alone in another. Within-industry payoff is 2 percent. Across-industry payoff is 6 percent. Sum is 8\. In this toy the industry bet is most of the money. In the lecture's full sample the mix flips: the within-industry piece is 84% of the raw neural-net payoff and 93% of the risk-adjusted payoff. The signal is a stock pick inside an industry. The longs the lecture describes line up with the usual anomaly book: small, cheap, illiquid, low recent issuance. Two exceptions, high investment and high idiosyncratic volatility, are where they say the nonlinearity is doing something a linear sort would miss. None of that makes the 4.8 basis points a strategy. It tells you which restriction was binding. A reduced-form net and an adversarial kernel find a large in-sample-looking spread, then give most of it back once microcaps, distress, and a half-percent turnover charge are in the ledger. A linear beta that is allowed to move with characteristics keeps a thin residual. The old article ["Does ML Actually Help Asset Pricing? Kelly's 20%, Not 2–3×"](https://aligrithm.com/does-ml-actually-help-asset-pricing-kellys-20-not-2-3x/) put the practitioner's 20% next to the published 2-to-3 times. Avramov prices the turnover, and on the tradable book the linear model is the one with a positive net. If you deploy one of these four, deploy IPCA. The neural net's full-sample Sharpe is the wrong number to budget. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/09/article_447-visual-0.png) ## KEY POINTS - OLS mean squared error equals sigma-squared times the trace of (X-transpose X) inverse. Two predictors, 240 months, correlation 0.9: the trace is 0.04386 against 0.008333 when the columns are orthogonal, a factor of 5.263\. That explosion is why the lecture shrinks. - Ridge is the same object as a normal prior on the discount-factor weights. M equals 1 minus b-transpose times (r minus mu), with b equal to V-inverse mu. The lecture's printed condition E\[M (r minus mu)\] equals 0 does not deliver that b. The condition that does is E\[M r\] equals 0\. Treat the printed residual as a slip. - With the prior that sets lambda to 0.04, an OLS weight of 0.25 shrinks to 0.125\. The eta-equals-2 prior keeps expected squared Sharpe at sigma-squared (0.316 annualized in the worked example). Eta equals 1, on 94 portfolios with the same knob, implies a root-mean-square Sharpe of 15.33\. That 15.33 is an illustration of the formula. The lecture does not print it. - The four machines are two stories. IPCA and the conditional autoencoder are beta pricing, linear versus a net. Kozak-style ridge and the Chen-Pelger-Zhu adversary are the kernel. Gu-Kelly-Xiu is a reduced form with 920 predictors and no pricing restriction. The out-of-sample windows and the stock universes (21,882 names versus 7,904) do not match. - On non-microcaps, a 0.50% times turnover charge leaves the neural net, the adversarial kernel, and the autoencoder with negative nets (-0.1225, -0.2645, -0.352 percent per month). IPCA's six-factor alpha of 0.613 against a cost of 0.565 is the only positive net, +0.048 percent per month. Its Sharpe does not fall when microcaps leave (0.967 to 0.978). - The neural-net payoff is a within-industry stock pick: 84% of the raw spread and 93% of the risk-adjusted spread. High-sentiment and high-VIX months add 1.534 (t = 2.43) and 1.851 (t = 2.85) to that long-short. The alpha lives where the book is hardest to trade. ## References - [Chen, Pelger, and Zhu, Deep Learning in Asset Pricing, Management Science (2024)](https://doi.org/10.1287/mnsc.2023.4695?ref=aligrithm.com) - [Avramov, Cheng, and Metzker, Machine Learning vs. Economic Restrictions: Evidence from Stock Return Predictability, Management Science (2023)](https://doi.org/10.1287/mnsc.2022.4449?ref=aligrithm.com) - [Avramov, Machine Learning in the Cross Section of Asset Returns](https://faculty.runi.ac.il/davramov/machine%5Flearning.pdf?ref=aligrithm.com) --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ### 8.12 Information Theory for Traders: Entropy, Mutual Information, Channel Limits URL: https://aligrithm.com/information-theory-for-traders-entropy-mutual-information-channel-limits/ Last updated: 2026-10-16T08:09:08.000Z A feature that matches the sign of the next E-mini return on 4 days out of 5 leaves 0.722 bits of uncertainty about that sign. The part the feature removes is 0.278 bits of mutual information, out of the 1 bit in a fair coin. The binary channel that flips 1 transmitted bit in 5 has that 0.278 as its capacity, and no choice of how often the feature calls up versus down raises it. ## What this actually is You run a daily signal on the E-mini. Lagged order-flow imbalance, or yesterday's return, agrees with the sign of the next return on 4 days out of 5\. The backtest reports an 80% hit rate. The information in that agreement is 0.278 bits per day. Entropy is the average surprise of one series, in bits. Mutual information is how many of those bits two series share. Channel capacity is the largest mutual information a noisy map can carry, once you may choose the distribution of what you send. A harbor telegraph delivers the right character 4 times in 5 and flips it the fifth. Repeat each instruction until a majority vote is sure, and the dock gets few new instructions per hour. Shannon's coding theorem says the error can be pushed as low as you want without driving the information rate to zero, as long as the rate stays under a bits-per-use ceiling. Above the ceiling, the decoder misses a fraction of the messages at every block length. At the desk, rank a feature by its mutual information with the next return, and treat capacity as a cap. A moving average, a rank, or a model fitted on the feature cannot beat the mutual information the raw series shares with the return. Taking the 0.278 from a 252-day book at face value ignores the finite-blocklength gap. On this channel the leading correction, at a 1% chance the decoded block is wrong, is 0.117 bits per day. ## A four-bin day contains 1.75 bits A four-bin E-mini return with probabilities 1/2, 1/4, 1/8, and 1/8 contains 1.75 bits, and that number is a floor on a lossless record of the bins. Pinkard and Waller put those same weights on four marble colors. Read them as a large up move, a small up move, a small down move, and a large down move. Entropy, the sum in their section 2.1, is the probability-weighted average surprise. $$ H(X) = \\sum\_{x} p(x) \\log\_2 \\frac{1}{p(x)} $$ Read it as: p(x) is the probability of bin x, a fraction with no units. The log is base 2, so each term is in bits, and the sum is bits per draw. A bin of probability 1/2 has surprise 1 bit. A bin of probability 1/8 has surprise 3 bits. Worked on the four bins: (1/2)(1) + (1/4)(2) + (1/8)(3) + (1/8)(3) = 0.5 + 0.5 + 0.375 + 0.375 = 1.75 bits. Spread the same probability evenly over four bins and the entropy hits its maximum, the log base 2 of 4, which is 2 bits. Redundancy, their W(X), is the gap: 2 minus 1.75, so 0.25 bits. A prefix code assigns the large-up bin the single bit 1, the small-up bin the two bits 01, and the two down bins the strings 001 and 000\. No codeword is a prefix of another, so a concatenated string still splits. On 16 draws that land on the expected counts, 8 large-up, 4 small-up, 2 small-down, and 2 large-down, the length is 8(1) + 4(2) + 2(3) + 2(3) = 28 bits, and 28/16 = 1.75\. The paper prints a binary string beside that 28\. The string in the transcription is 31 characters. The length implied by the probabilities is 28\. The drawn string and the printed length disagree, so the figure may be wrong. The sum is the part that checks. Over 252 trading days the same histogram needs 252 times 1.75 = 441 bits for a lossless record of the bins, against 504 bits if you budgeted the 2-bit maximum every day. The 63-bit gap is redundancy of the histogram, compressibility of the record. A forecast needs the next bin to depend on the ones before it. That dependence is the entropy-rate gap, computed below. The 0.25 here is histogram redundancy. Their asymptotic equipartition result, from the law of large numbers, puts the probability of a long sequence onto a typical set of about 2 raised to N times H(X) sequences, each with probability about 2 raised to minus that. For N = 16 and H(X) = 1.75 the set holds 2 to the 28, which is 268,435,456 sequences. All possible 16-draw strings from four bins number 4 to the 16, which is 4,294,967,296\. The typical set is 1/16 of that space, and addressing it takes 28 bits. Fewer bits and two typical sequences share a codeword. The old article ["Entropy as a Market Choppiness Gauge"](https://aligrithm.com/entropy-as-a-market-choppiness-gauge/) divides entropy by the log of the number of bins and reads a ratio near 1 as chop. On these four bins that ratio is 1.75/2 = 0.875\. The ratio describes the histogram. Dependence from one bin to the next can be large while the ratio sits near 1, which is the sticky chain below. ## The shared part is mutual information The 80% feature shares 0.278 bits with the next sign. The correlation of the same table is 0.6\. The two numbers do not convert into each other. Mutual information is their equation (1), the probability-weighted average of the pointwise score log of joint over the product of the marginals. $$ I(X; Y) = \\sum\_{x} \\sum\_{y} p(x, y) \\log\_2 \\frac{p(x, y)}{p(x)\\, p(y)} $$ Read it as: p(x, y) is the joint probability of signal x and next sign y. p(x) and p(y) are the separate probabilities. The fraction inside the log equals 1 when the two are independent, and the log is then zero. The sum is in bits per observation. Work the 80% table. Signal and next sign are each up half the time, and the signal is wrong on 1 day in 5\. The four joint probabilities are 0.4 for up-up, 0.1 for up-down, 0.1 for down-up, and 0.4 for down-down. Each diagonal cell contributes 0.4 times log base 2 of (0.4/0.25) = 0.4 times log base 2 of 1.6 = 0.4 times 0.6781 = 0.27124\. Each off-diagonal cell contributes 0.1 times log base 2 of (0.1/0.25) = 0.1 times log base 2 of 0.4 = 0.1 times -1.3219 = -0.13219\. Two of each: 2(0.27124) + 2(-0.13219) = 0.54248 - 0.26438 = 0.2781 bits. Code up as +1 and down as -1 on that same table. The product averages 0.4 + 0.4 - 0.1 - 0.1 = 0.6, and both variables have variance 1, so the correlation is 0.6\. One table, two summaries. Pinkard and Waller then rewrite equation (1) three ways. $$ I(X; Y) = H(Y) - H(Y \\mid X) = H(X) - H(X \\mid Y) = H(X) + H(Y) - H(X, Y) $$ Read it as: H(Y) is the uncertainty in the next sign before you see the feature, in bits. H(Y given X) is the uncertainty that remains after you see it. The difference is what the feature removed. The third form subtracts the joint entropy from the sum of the two separate entropies, which is the overlap. ![Bars relating joint entropy H(X, Y), the conditional entropies, and mutual information I(X; Y). The overlap of the two entropy bars is the mutual information.](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/09/article_493-entropy_mi_bars.png) Worked on the same table. The next sign is a fair coin, so H(Y) = 1 bit. Given the feature, the sign still flips with probability 1/5, so the remaining entropy is the binary entropy of 1/5\. That is -(1/5) log base 2 of (1/5) minus (4/5) log base 2 of (4/5) = (1/5)(2.3219) + (4/5)(0.3219) = 0.4644 + 0.2575 = 0.7219 bits, which rounds to 0.722\. Then 1 minus 0.722 = 0.278 bits. Same number as the double sum. The bar diagram is that subtraction drawn as lengths. Let X be one of -2, -1, +1, +2, each with probability 1/4, and let Y equal X squared. Then Y is 4 or 1, each with probability 1/2\. X is symmetric about zero, so the correlation is 0\. H(X) is 2 bits. Given Y, two values of X remain and they are equally likely, so the conditional entropy is 1 bit and the mutual information is 1 bit. You learned the magnitude. You learned nothing about the sign. The old article ["Mutual Information as a Regime / Noise Filter"](https://aligrithm.com/mutual-information-as-a-regime-noise-filter/) uses a gap of this kind as a go or no-go on a system. The mutual information is the ceiling on that test. A correlation of zero can sit next to a ceiling of 1 bit. The ceiling drops further once the series has memory, and the marginal entropy does not show it. Pinkard and Waller's sticky chain repeats the last outcome with probability 5/8 and jumps to each of the other three with probability 1/8\. The chain is symmetric, so the marginal stays uniform and each draw still has entropy 2 bits if you have not seen the previous draw. The conditional entropy is $$ H(X\_{n+1} \\mid X\_n) = -\\frac{5}{8}\\log\_2\\frac{5}{8} - 3\\cdot\\frac{1}{8}\\log\_2\\frac{1}{8} $$ Read it as: 5/8 is the probability of repeating the last bin, and each 1/8 is a jump to one of the other three. The log is base 2\. The result is bits of uncertainty about the next bin once you know the last one. The three jumps contribute 3 times (1/8) times 3 = 1.125 bits, since log base 2 of 8 is 3\. The repeat contributes (5/8) times log base 2 of (8/5) = 0.625 times 0.6781 = 0.4238 bits. Add them: 1.125 + 0.4238 = 1.5488, which rounds to 1.549 bits. Mutual information from one bin to the next is 2 minus 1.549 = 0.451 bits. Their equations (3) and (4) are two writings of the entropy rate: joint entropy divided by length, and the conditional entropy of the next draw given the whole past. On a stationary process the two meet as the length grows. Their equation (5) is the Markov cut. Only the last draw matters, so the conditioning set collapses to one variable. A one-lag model of this chain cannot pull out more than 0.451 bits per step. Point the choppiness ratio at the marginal and it reads 2/2 = 1, a fully chopped tape, while the lag still carries 0.451 bits. The directed version across two series, how many bits one series' past removes from the uncertainty of another's future, is what a conditional-mutual-information test estimates. The old article ["Stop Using Pairwise Granger: PCMCI for Financial Causality"](https://aligrithm.com/stop-using-pairwise-granger-pcmci-for-financial-causality/) plugs that test in as CMI, beside a linear partial correlation and a Gaussian-process residual test. Pairwise mutual information still counts bits that a third series explains. The conditional number is the subtraction H(X given Z) minus H(X given Y and Z). A pairwise screen has not run that subtraction. ## Capacity is the best input on a fixed channel The flip channel's capacity is 0.278 bits per use. A 252-day block does not get to spend all of it. Capacity, their equations (7) and (8), is the mutual information at the best input distribution. The channel stays fixed. You choose how the inputs are distributed. $$ C = \\max\_{p\_X} I(X; Y) $$ Read it as: p\_X is a probability distribution over the channel inputs, a list of fractions that sum to 1\. I(X; Y) is mutual information in bits per use. C is that mutual information at the maximizing list. For the binary symmetric channel the maximizing input is a fair coin, and capacity equals 1 minus the binary entropy of the flip probability. $$ h(p) = -p \\log\_2 p - (1-p)\\log\_2(1-p), \\qquad C = 1 - h(p) $$ Read it as: p is the probability that a transmitted bit arrives flipped, a fraction. h(p) and C are in bits per use. At p = 0 the channel is clean and C = 1\. At p = 1/2 the output is independent of the input and C = 0. Worked at p = 1/5: h(1/5) = 0.722 bits, so C = 0.278 bits per use. That is the mutual information of the 80% table, because that table is this channel fed with a fair input. You were already on the ceiling. Changing the long-run share of up calls versus down calls does not raise it. Pinkard and Waller set this flip probability at 1/5 in the body of the coding section. The caption of their coding figure says the channel flips each bit with probability 1/2\. At 1/2 the capacity is zero, so the caption and the body disagree. The arithmetic here follows the body. If the caption is what they solved, the capacity result collapses. Their three-input channel shows what happens when you optimize one term of I(X; Y) = H(Y) minus H(Y given X) and drop the other. Maximizing output entropy alone, spreading probability onto inputs whose outputs do not pile up, transmits 1 bit on that channel. Minimizing output noise alone piles every input onto the quietest symbol and transmits 0 bits, because a channel that always sends the same symbol sends nothing. Maximizing the difference transmits 1.13 bits. Those three numbers are the ones they print. They draw the channel matrix rather than tabulate it, so this article does not recompute 1.13\. Treat 1.13 as their reported optimum, not as a check you can redo from the text. ![Binary entropy and channel capacity of a binary symmetric channel versus flip probability. The mark at p = 1/5 sits on a capacity of 0.278 bits per use. At p = 1/2 capacity is zero.](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/09/article_493-bsc_capacity.png) Repetition coding, sending the same bit many times and taking a majority vote, can push the error down, and the rate (source bits divided by transmitted bits) goes to zero as you demand a smaller error. The noisy-channel coding theorem says block codes do better. Encoder and decoder pairs exist that keep the probability of a wrong block as small as you want, at any rate below capacity, once the block is long enough. No pair does that above capacity. Shannon's argument shows that a random encoder achieves the bound on average. It does not hand you the encoder. Pinkard and Waller note that codes with fast decoders arrived decades after the existence proof. The rate you can lock in at a finite block length sits under capacity. They report, citing Polyanskiy, Poor, and Verdú, that at a fixed block-error probability the gap shrinks in proportion to 1 over the square root of the block length. They do not print the constant. On this channel the pointwise information is 1 + log base 2 of (4/5) = 0.678 bits when the bit arrives intact, and 1 + log base 2 of (1/5) = -1.322 bits when it flips. The mean of that two-point score, weighted by 4/5 and 1/5, is the capacity, 0.278\. The variance is 0.8 times (0.678) squared plus 0.2 times (1.322) squared, minus (0.278) squared, which equals 0.640 bits squared. The normal approximation in the finite-blocklength paper sets the leading gap at the square root of (variance over N), times the standard-normal quantile of 1 minus the block-error target. At a 1% block-error target that quantile is 2.326\. At N = 252, $$ \\sqrt{\\frac{0.64}{252}} \\times 2.326 = 0.117 $$ Read it as: 0.64 is the variance of the pointwise information, in bits squared. 252 is a count of trading days, the block length. 2.326 is a pure number, the normal quantile. The product is in bits per day. So 0.278 minus 0.117 leaves 0.161 bits per day as the leading finite-blocklength rate at a 1% chance the whole block is decoded wrong. The next term in the approximation, of order log(N) over N, adds about 0.016 bits, half of log base 2 of 252, divided by 252\. Constants that stay flat as N grows sit in neither term. The 0.117 is checkable from the variance above. Pinkard and Waller do not print it. That 1% is the probability the decoded block is wrong. A daily hit rate answers a different question, and reading the 1% as "right on 99% of days" misreads the theorem. The gap shrinks slowly. The leading gap falls to a tenth of capacity, 0.0278 bits, when N = 0.64 times (2.326 / 0.0278) squared, which is 4,480 trading days, about 18 years. A mutual-information number from one year of daily bars sits outside the limit the coding theorem describes. The same paper flags the cliff effect, again from Polyanskiy, Poor, and Verdú. Design the code for one noise level, then run it on a noisier channel, and the error jumps. The theorem tells you the capacity at the new noise. It does not tell you how to build a code that degrades gently when the noise you assumed was wrong. A feature whose 80% hit rate you measured in one regime and traded in another has this failure, and capacity at the new noise will not patch the code you already froze. ## A transform spends bits, and units can fake entropy A function of a series cannot carry more information about a target than the series already shares with that target. Their data-processing inequality, in section 3.3, covers a chain A to B to C. What C tells you about A is at most what B tells you about A. $$ I(A; B) \\ge I(A; C) $$ Read it as: A, B, and C are random variables. The inequality is in bits. The hypothesis is that C depends on A only through B. Work it on the four bins. Entropy is 1.75 bits. The sign is a function of the bin. Given a down sign, the two down bins each had probability 1/8, so they split the conditional probability in half and the conditional entropy is 1 bit. Given an up sign, the small-up probability 1/4 and the large-up probability 1/2 become 1/3 and 2/3\. That conditional entropy is -(1/3) log base 2 of (1/3) minus (2/3) log base 2 of (2/3) = 0.9183 bits. The sign is down with probability 1/4, so the average entropy left is (1/4)(1) + (3/4)(0.9183) = 0.9387 bits. Mutual information between the four-bin return and its own sign is 1.75 minus 0.9387 = 0.8113 bits. Any further function of the sign keeps at most those 0.8113 bits about the four-bin return. The sign already dropped 0.9387 bits. The same inequality binds the feature. A moving average of order-flow imbalance is a function of the imbalance, so the average cannot carry more information about the next return than the imbalance did. Mix extra noise into the average and it carries less. The word length in the old choppiness gauge is a map of this kind. A coarser alphabet is a channel. It keeps some bits and spends others. It does not raise the mutual information with a target the raw tape did not already share. Differential entropy is the integral that replaces the sum when the variable is continuous, and Pinkard and Waller spend section 2.9 on why you should refuse to read it as a bit count. It can come out negative. It changes when you change units, because a density has units of 1 over x, and logging it folds that unit into the number. $$ h(X) = \\frac{1}{2} \\log\_2 \\left(2 \\pi e \\sigma^2\\right) $$ Read it as: this is the differential entropy of a Gaussian, in bits, with sigma the standard deviation of the return in whatever unit you stored. Pi and e are the usual constants. The log is base 2. A Gaussian daily return with standard deviation 0.01, decimal form, has differential entropy -4.597 bits. Write the same returns in percent and the standard deviation is 1, and the differential entropy is +2.047 bits. The shift equals log base 2 of 100, which is 6.644 bits, and -4.597 + 6.644 = 2.047\. Nothing in the market changed. Mutual information does not move under that rescaling: the log runs on a ratio of densities, and the units cancel. Ross showed mutual information between a discrete series and a continuous series is well defined, which is the case of a binned signal against a raw return. Use that, or bin both sides and accept that the bin width is part of the answer. Ranking features by the differential entropy of a residual, in a unit you picked for the file format, ranks the unit. Rate distortion is the lossy form of the same floor. To hold average distortion under a cap you chose, you need at least R(D) bits about the source. If you already sit on the best distortion your current mutual information can buy, a lower distortion requires more bits. For a discrete source and a distortion that is zero only when the reconstruction matches, that curve meets zero distortion at the entropy. The sign map above is a lossy code at 0.8113 bits, under the 1.75-bit lossless floor, and the 0.9387 bits it dropped are the distortion you accepted when you kept the sign and threw out the size of the move. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/09/article_493-visual-0.png) ## KEY POINTS - An 80% hit rate on a balanced up/down signal is 0.278 bits of mutual information. The binary entropy of a 1-in-5 flip is 0.722 bits, and that is the uncertainty left after you see the feature. - Entropy of the bins 1/2, 1/4, 1/8, 1/8 is 1.75 bits. The maximum on four bins is 2\. Redundancy is 0.25 bits. Over 252 days that is a 441-bit lossless record against a 504-bit budget. The 63-bit gap is compressibility of the histogram. Forecasts live in the entropy rate, the conditional entropy of the next bin. - The sticky chain that repeats with probability 5/8 has marginal entropy 2 bits and conditional entropy 1.549 bits, so one lag carries 0.451 bits. A choppiness ratio on the marginal reads 1 and misses it. Conditional mutual information is the subtraction the old article ["Stop Using Pairwise Granger: PCMCI for Financial Causality"](https://aligrithm.com/stop-using-pairwise-granger-pcmci-for-financial-causality/) plugs in as the CMI test. - Channel capacity is the maximum of mutual information over input distributions. On this flip channel the fair input already achieves it, at 0.278 bits per use. The paper's coding-figure caption says flip probability 1/2, which would make capacity zero. The body says 1/5\. The body is the calculation that checks. - At block length 252 and a 1% chance the whole block is wrong, the leading finite-blocklength gap on this channel is 0.117 bits per day, from a pointwise-information variance of 0.64\. The square root shrinks slowly: cutting that gap to a tenth of capacity takes about 4,480 trading days. The theorem does not supply a code, and it does not protect you when the noise level moves. - Data processing forbids a moving average, a rank, or a sign from carrying more information about a target than the raw series did. The sign of the four-bin return keeps 0.8113 bits and drops 0.9387\. Differential entropy of a 1% Gaussian return is -4.597 bits in decimal and +2.047 bits in percent. Rank features by mutual information, which does not move when you change units. ## References - [A Mathematical Theory of Communication (Shannon)](https://doi.org/10.1002/j.1538-7305.1948.tb01338.x?ref=aligrithm.com) - [Channel Coding Rate in the Finite Blocklength Regime (Polyanskiy, Poor, and Verdú)](https://doi.org/10.1109/TIT.2010.2043769?ref=aligrithm.com) - [Mutual Information between Discrete and Continuous Data Sets (Ross)](https://doi.org/10.1371/journal.pone.0087357?ref=aligrithm.com) - [A visual introduction to information theory (Pinkard and Waller)](https://arxiv.org/abs/2206.07867?ref=aligrithm.com) --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ### 5.54 Lead-Lag Is Multidimensional: Trade Price and OBI Predict Others' Midpoints URL: https://aligrithm.com/lead-lag-is-multidimensional-trade-price-and-obi-predict-others-midpoints/ Last updated: 2026-10-16T08:09:09.000Z The Euro Stoxx 50 future's trade price led the DAX future's midpoint on 99% of trading days from January to June 2021, and the lag that maximized the correlation was 100 microseconds. Bender, Cestonaro, and Schmidt measure that lead on 19 Xetra and Eurex instruments, across nine microstructure series, with a lag-shifted Hayashi-Yoshida correlation: the sum of price changes whose time windows overlap, with no resampling onto a clock. Their lead-lag strength, a score that grows only when one series leads and the two series move together, sits at or above 2.29 for the top 5% of pairs when the leader is a trade price and the lagger is a midpoint. Ask the leader's last change to call the direction of the lagger's next change and the average hit rate is 51.8%, against 50.5% for a benchmark that always calls the more common direction. Hit rate and lead-lag strength correlate 0.67\. The ordering of who moves first is a stable fact about a handful of related contracts. A 1.3 point edge on direction, inside a window of 100 microseconds, is a quote you cancel if you are making a market in the same data center, and a spread you donate if you try to lift it from a remote server. ## What this actually is You quote the DAX future. A trade prints in the Euro Stoxx 50 future. In this sample the DAX midpoint follows that trade, same direction, and the match peaks 100 microseconds later. Leave the resting quote and a faster participant lifts it. That pickup is the whole economic content of the paper. The method lines up two uneven streams of trades and book updates and asks which stream's changes match the other's later changes. Nothing is forced onto a one-second grid. Picture two clerks in one room, each stamping a ticket only when their own book changes. The stamps do not share a metronome. Count the pairs of stamps whose time windows overlap, multiply the two price changes, and add. Then slide one clerk's clock forward by a chosen lag and count again. The lag where the sum peaks is how far ahead that clerk is. If the claim holds, you read the leader's trade before you leave a resting order in the lagger, or you widen the quote by enough to pay for being picked off. Treating the 51.8% hit rate as a signal you can take from outside the matching engine means you cross a spread to bet a coin flip that wins an extra 1.3 times in a hundred, on a clock your connection cannot see. In vacuum, light covers 30 kilometers in 100 microseconds. A desk outside that radius is already late when the message arrives. ## Overlapping increments, not a resampled clock Two series that tick at different times still have a correlation, and forcing them onto a fast clock drives that correlation to zero. Downsample both to one-minute bars and you can run a regression, which is what the older lead-lag papers did, and what the old article ["Lead-Lag Relationships in Global Markets"](https://aligrithm.com/lead-lag-relationships-in-global-markets/) does with lagged cross-correlation on a regular clock. Push that same grid down toward the speed of the trades and the correlation shrinks toward zero even when the two prices are the same price. Epps documented that bias in 1979\. Interpolation manufactures the bias. Hayashi and Yoshida skip the grid. Split each series into the intervals between its own observations. Multiply the change on an interval of the first series by the change on an interval of the second only when the two intervals overlap, add every such product, and divide by the square root of the summed squared changes of each series. Hoffmann, Rosenbaum, and Yoshida then slide the second series by a lag ell before testing the overlap. A positive ell lays the second series' earlier changes on top of the first series' later changes, so a large correlation at a positive ell means the second series leads. $$ \\hat{\\rho} = \\frac{\\sum\_{i,j} \\Delta X(I\_i)\\, \\Delta Y(J\_j)\\, \\mathbf{1}\\{I\_i \\cap J\_j \\neq \\emptyset\\}} {\\sqrt{\\sum\_i \[\\Delta X(I\_i)\]^2 \\; \\sum\_j \[\\Delta Y(J\_j)\]^2}} $$ $$ \\hat{\\rho}(\\ell) = \\frac{\\sum\_{i,j} \\Delta X(I\_i)\\, \\Delta Y(J\_j)\\, \\mathbf{1}\\{I\_i \\cap (J\_j + \\ell) \\neq \\emptyset\\}} {\\sqrt{\\sum\_i \[\\Delta X(I\_i)\]^2 \\; \\sum\_j \[\\Delta Y(J\_j)\]^2}} $$ Read the first line as an overlap correlation. Delta X on interval i is the price change of series X between two of its own prints, in price points. The indicator is 1 when that interval shares any time with interval j of series Y, and 0 otherwise. The denominator rescales by each series' own size, so the ratio is unitless. The second line is the same sum after every Y interval has been moved by ell seconds. Positive ell: Y leads X. Negative ell: X leads Y. Worked toy, not a market. X rises 4 points from 0 to 10 seconds and 3 points from 10 to 20 seconds. Y rises 4 points from 1 to 2 seconds and 3 points from 11 to 12 seconds. Each X interval overlaps exactly one Y interval, the products are 16 and 9, the numerator is 25, and each series' summed squares are 25, so the correlation at lag zero is 25 over 25, which is 1\. Slide Y forward by 15 seconds and the only surviving overlap is the 3-point X change with the shifted 4-point Y change. The numerator falls to 12 and the correlation falls to 12/25, which is 0.48\. Slide Y backward by 15 seconds and nothing overlaps, so the correlation is 0\. Y's earlier move lines up with X's later move. Y leads. The same arithmetic, run on nanosecond stamps, is the whole estimator. Two design choices bound what that number can mean. The authors keep a measure only at times when a trade prints, the same filter Hoffmann and Huth used for prices. An order-book imbalance that changes between trades never enters the sum. And they estimate one curve per instrument pair, per measure pair, per day, on a fixed menu of 35 lags: 17 leads, 17 lags, and the contemporaneous point, from 10 microseconds out to 60 seconds. Nineteen names, nine measures, and cross-name pairs only (19 times 18 times 81) give 27,702 series pairs. Times 124 sessions, that is 3,435,048 daily curves. They drop a day when the two series share fewer than 100 overlapping intervals, which leaves 2,119,921 curves, 61.7% of the total. They call that about 61%. Four pairs, all involving the silver note, fail a 21-day minimum and leave the sample. ## A strength score that ignores fake asymmetry The lag of the peak is a thin summary of the curve, and the lead-lag ratio is a misleading one. The maximizing lag records a direction and ignores every other point. Huth and Abergel's lead-lag ratio keeps the whole curve: summed squared correlations on positive lags, divided by the same sum on negative lags. Above 1, the shifted series leads. A ratio of 4 can come from correlations of 0.4 or from correlations of 0.0004. $$ LLR = \\frac{\\sum\_{\\ell>0} \\hat{\\rho}(\\ell)^2}{\\sum\_{\\ell<0} \\hat{\\rho}(\\ell)^2} $$ $$ LLS = \\begin{cases} \\left(\\dfrac{\\sum\_{\\ell>0}\\hat{\\rho}(\\ell)^2}{\\sum\_{\\ell<0}\\hat{\\rho}(\\ell)^2}-1\\right) \\dfrac{100}{n\_L}\\sum\_{\\ell} \\hat{\\rho}(\\ell)^2 & \\text{if that ratio}\\ge 1,\\\\\[1em\] \\left(\\dfrac{\\sum\_{\\ell<0}\\hat{\\rho}(\\ell)^2}{\\sum\_{\\ell>0}\\hat{\\rho}(\\ell)^2}-1\\right) \\dfrac{-100}{n\_L}\\sum\_{\\ell} \\hat{\\rho}(\\ell)^2 & \\text{otherwise.} \\end{cases} $$ Read the lead-lag ratio as a comparison of two areas under the curve. Read the lead-lag strength as that comparison, minus one, multiplied by 100 times the average squared correlation across the grid. n\_L is the number of lags on the grid, here 35, counting lag zero. The sum in the last factor runs over every lag, including zero. Positive strength means the shifted series leads, and the absolute value is the same if you swap which series you shift. The authors' typeset equation is missing the opening parenthesis in front of the ratio, so the line as printed does not parse. The card is the grouping their prose describes, with that parenthesis restored. Their own footnote is the reason the ratio is not enough. Set every positive-lag correlation to 0.001 and every negative-lag correlation to 0.0005\. With 17 lags on each side the summed squares are 17 times 0.000001 and 17 times 0.00000025, and the ratio is 4\. A lead-lag ratio of 4 sounds like a strong lead. The correlations are noise. Put lag zero at 0, which the footnote leaves unspecified, and the lead-lag strength is (4 minus 1) times (100/35) times 0.00002125, which equals 0.000182\. Swap which series is shifted and the strength is minus 0.000182\. The same absolute value, opposite sign, which the ratio does not give you: the swapped ratio is 0.25, not minus 4\. In the real sample the 95th percentile of strength for trade price leading midpoint is 2.29, about 12,600 times the footnote's fake lead. The strength stays near zero when the curve is flat and tiny, and it gets large only when the curve is both lopsided and tall. ## Trade price and imbalance lead; quoted depth does not Trade prices and best-quote imbalance lead other names' midpoints. Quoted spread, volume, and resting depth do not. The authors rank measure pairs by the 95th percentile, across instrument pairs, of the daily peak correlation and of the lead-lag strength. That percentile is the value the best 5% of pairs clear, so a quiet cell means even the strongest pairs in that cell are quiet. Price against price is the loud cell. Five percent of pairs have a peak midpoint correlation of 0.88 or higher, and the midpoint-against-trade-price cell is 0.76 on the same percentile. Quoted spread, traded volume, size at the best quotes, and size at the ten best levels sit near 0.01 to 0.05 against the other series. Two off-diagonal leads survive. Trade price of one name leading the midpoint of another has a 95th-percentile strength of 2.29\. Order-book imbalance (OBI) at the best quotes, the euro amount on the bid minus the euro amount on the ask divided by their sum, leading the other name's midpoint has a 95th-percentile strength of 0.55\. The reverse, midpoint leading future imbalance, is about zero. Transaction-price volatility leading midpoint volatility is the third pair they keep, with a 95th-percentile peak correlation of 0.21 or higher. They then study one pair from each family: trade price into midpoint, imbalance into midpoint, and transaction-price volatility into midpoint volatility. $$ OBI\_t = \\frac{P^{bid}\_t Q^{bid}\_t - P^{ask}\_t Q^{ask}\_t} {P^{bid}\_t Q^{bid}\_t + P^{ask}\_t Q^{ask}\_t} $$ Read it as a signed fraction of euro depth at the inside. Prices are in currency per unit, sizes are in units, both products are in currency, and the ratio is unitless, between minus 1 and 1\. Worked book: bid 100.00 for 500, ask 100.10 for 200\. Euro bid is 50,000, euro ask is 20,020, and the imbalance is 29,980 / 70,020, which is 0.428\. The share-count version used in the old article ["The Worst-Kept Secret: Imbalance Is the MM's Optimal Response, Not Alpha"](https://aligrithm.com/the-worst-kept-secret-imbalance-is-the-mms-optimal-response-not-alpha/), (500 minus 200) over (500 plus 200), is 0.429\. At the inside quote the two agree. The lead this paper finds is the object that article warns you about. Imbalance is the market maker's reply to a price she already expects, so a lead from one name's imbalance into another's midpoint can be two books answering the same latent price. Cont, Cucuringu, and Zhang document the same cross-name pattern: order-flow imbalance in one equity moves the midpoint of another. The instrument map is a map of shared underlyings. Volkswagen and Daimler, both autos, correlate with each other and with the DAX and Euro Stoxx futures and ETFs. Pair either of them with SAP and the correlation drops. The Euro Stoxx future is the dominant leader: its trade price leads the midpoint of every stock and equity future in the sample, and the strength keeps the same sign on 99% of days, which is the three-star mark in their figure. The Swiss future does the same. The DAX future and the Mini-DAX future lead the three DAX stocks, and the stocks lead them back, with SAP, the heaviest DAX weight, the strongest and most persistent of those reverse leads. That two-way result is what the law of one price says you should see. A DAX constituent trade moves the index future because the index is the constituents. An index-future trade moves the constituent because both are the same bet. ![Hayashi-Yoshida correlation against lag in seconds for FDAX midpoint versus FESX trade price, and for FGBL midpoint versus FGBM trade price. Both curves peak immediately after zero and decay faster on the negative-lag side.](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/09/article_463-hy_curve_fdax_fesx.png) On the zoomed curve the Euro Stoxx trade price leads the DAX midpoint by 100 microseconds, and the Bobl trade price leads the Bund midpoint by the same 100 microseconds. The curve stays elevated for several seconds away from zero and decays faster when the lag is negative. The peak itself is a tenth of a millisecond. The authors, citing Deutsche Boerse, put the fastest members' reaction under 3 microseconds, so a colocated trader can act many times inside that 100 microsecond lead. Budish, Cramton, and Shim's stale-quote sniping is this picture with a name on it: the fast trader hits the DAX quote that has not yet moved to the Euro Stoxx trade. Imbalance is slower, and one-directional. The curves fall to about zero after five to ten seconds, imbalance leads midpoint, and midpoint does not lead imbalance. Daimler's imbalance leads the DAX future's midpoint by one second and Volkswagen's midpoint by two seconds. One second is a different operational problem from 100 microseconds. A fast quoter can see it. The futures-versus-futures trade-price lead is for members who already live inside the exchange's latency budget. Volatility splits by how often the name trades. The DAX future's transaction-price volatility leads the Mini-DAX future's midpoint volatility by one millisecond. Between the two bitcoin notes the lead is two seconds, and the correlation only falls from 0.461 to 0.446 at a ten-second lead. The thinner note, 21XB, trades about 2.0 million euros a day with 77 seconds between trades. The thicker one, BTCE, trades about 26.6 million euros a day with 21 seconds between trades. The correlation cannot die in ten seconds when the next print may not exist yet. The authors searched every pair, with no pre-chosen fundamental link. The pairs that light up share an underlying or a risk factor. ## Liquid names lead, until you leave equities The same pair's lead does not track that day's volume, and the more active name leads only inside equities. Bin each pair's days by the combined rank of the two names' volume, by the gap between their trades, and by midpoint volatility. The lead-lag strength inside those bins barely moves. The same null shows up for turnover per trade, relative spread, and the share of trades that hit past the first price level. The day-to-day wobble in the strength is not the day's liquidity. The cross-section is where the old equity result shows up, and then breaks. Huth and Abergel, on midpoints, found that the name with more trades and a tighter book tends to lead. Here the authors regress daily strength on the log of an indicator ratio, indicator of the midpoint name over indicator of the shifted name, one indicator at a time so the liquidity proxies do not fight each other. A negative coefficient means the shifted name's lead gets stronger as its own indicator rises relative to the other name. They print coefficients, standard errors, and adjusted R-squared in percentage points. N is 113,520 in the full sample, pooling the three measure pairs. $$ LLS\_{m,i,t} = \\beta\_0 + \\beta\_1 \\log\\frac{I\_{X,i,t}}{I\_{Y,i,t}} + \\varepsilon\_{m,i,t} $$ Read it as a line. The left side is the lead-lag strength for measure pair m, instrument pair i, day t. The ratio inside the log is a liquidity or volatility statistic of the midpoint name divided by the same statistic of the shifted name, unitless. Beta-one is the change in strength for a one-unit rise in that log, which multiplies the ratio by e, about 2.718. On average turnover per trade, the equity subsample prints beta-one of minus 18.0 with a standard error of 0.9 and an adjusted R-squared of 0.8%. Add dummies for whether each leg is a stock, an equity future, or an equity ETF and the slope goes to minus 25.3 (standard error 1.7) while the adjusted R-squared goes to 11.0%. The slope has the sign the equity literature wants: more turnover on the leader, stronger lead. The fit says the turnover ratio accounts for under one percent of the variation, and knowing the instrument types accounts for the jump to 11%. In the full sample the same slope is minus 2.4 (0.1) with an adjusted R-squared of 0.1%, and minus 6.3 (0.5) with 13.3% once asset-class dummies enter. The equity slope is an equity fact. Bonds and notes dilute it. The sign is not even stable inside equities. On intertrade duration, seconds between trades, the equity slope is minus 1.2 (0.3) with no dummies and plus 10.2 (1.1) with them. The positive sign matches "the name that trades more often leads," because a higher duration ratio means the shifted name's trades are closer together. That sign shows up once the instrument-type dummies are in the regression. On midpoint volatility the equity slope is plus 17.3 (0.9) without dummies, the less-volatile name leading, and plus 1.9 (2.0) with dummies, a standard error larger than the coefficient. The volatility story dies as soon as you know what you paired. The bond futures are the counterexample you can see without a regression. The Bund trades about 92.8 million euros a day, with 0.24 seconds between trades. The Schatz trades about 18.6 million euros a day, five times less, with 3.91 seconds between trades, about 16 times less often. The Bobl sits in between, at 39.1 million euros. Trade prices of the Schatz and the Bobl lead the Bund's midpoint. The Bund does not lead them back. The less active contracts lead the most active one. Whatever you learned about leadership from stocks and equity futures, do not carry it into the bond curve. The equity subsample is 39,040 rows and the non-equity subsample is 16,787\. Those do not add to 113,520\. Pairs that mix a stock or equity future with a bond, a metal note, or a bitcoin note sit in the full sample and in neither subsample, which is why the full-sample slope is so much flatter. ## Direction accuracy ranks the lead and does not pay the spread A higher lead-lag strength ranks which pairs call the next change, and the average call is 51.8%. The authors average the daily strength over January through May 2021 and score June. For every pair whose average strength is positive, the shifted series is the leader. Each time the leader changes, they call the lagger's next change in the same direction when the peak correlation is positive, and in the opposite direction when it is negative. They keep only the latest leader change as the live call. Accuracy is correct calls divided by the number of changes in the lagger. The benchmark always calls whichever direction was more common for that lagger. ![Scatter of June 2021 direction accuracy against average lead-lag strength, for midpoint versus trade price, midpoint versus imbalance, and midpoint volatility versus transaction-price volatility. The cloud slopes up with correlation 0.67. A corner table shows mean accuracy 0.518 against a benchmark of 0.505, and a 95th percentile of 0.636 against 0.545.](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/09/article_463-lls_vs_accuracy.png) Six hundred seventy-four pairs have a positive average strength. Mean accuracy is 51.8% for the lead-lag call and 50.5% for the benchmark. The median is 50.4% against 50.1%. The 90th percentile is 57.0% against 53.5%, the 95th is 63.6% against 54.5%, and the best pair reaches 81.6% against a benchmark whose own best is 60.1%. Those two maxima need not be the same pair. The worst lead-lag call is 43.8%, worse than the benchmark's worst of 47.0%. On a dead pair, following the "leader" loses to the base rate. The correlation of 0.67 (they print a p-value of 0.00) says the strength ranks pairs. It does not say the average pair has a trade. Restrict the scored changes to a window from half the estimated lead time to one and a half times that lead time, and the mean accuracy rises to 53.3%. The authors say the upper tenth of pairs gains another 4.1 percentage points relative to the unwindowed call. For the Euro Stoxx into DAX pair the lead time is 100 microseconds, so the window is 50 to 150 microseconds after the Euro Stoxx trade. That is the only horizon on which the improved number applies to the headline pair. Put a spread next to the hit rate. The appendix table reports a relative quoted spread of 1.00 basis points on the DAX future and 2.56 basis points on the Euro Stoxx future. The same appendix prints the spread formula as bid minus ask, over the midpoint, which is negative on any two-sided book, while every spread in the table is positive. The formula and the table disagree. The positive table values are the ones used here. The printed sign is a typesetting error. A 51.8% call on the direction of the next midpoint change, before any queue or fee, is a 1.3 point edge over a coin flip. Crossing a 1 basis point spread to express it hands back more than the edge on a one-tick move. The paper reports no profit, no cost, and no fill model. The out-of-sample window is June 2021, one month after a five-month average, inside a single half-year on one exchange group. The useful object is the ranking. Strength near zero, and the other series has nothing to say. Strength in the right tail, and the leader's last change calls direction well above the base rate, on a horizon equal to the lag that maximized the curve. For a market maker in the DAX future, cancel or skew when the Euro Stoxx future trades, inside a tenth of a millisecond, or widen by the spread you will donate to the member who saw it. For anyone else the 0.67 describes whose midpoint is stale. The old article ["From Intermarket Analysis to Network Momentum"](https://aligrithm.com/from-intermarket-analysis-to-network-momentum/) wants a learned graph of who spills into whom. This sample's graph is a star with the Euro Stoxx future at the center, plus a bond-curve edge that runs from the thin contracts toward the Bund, and it is stable within these six months. The old article ["Stop Using Pairwise Granger: PCMCI for Financial Causality"](https://aligrithm.com/stop-using-pairwise-granger-pcmci-for-financial-causality/) is the caveat on reading that star as a cause. Euro Stoxx and DAX share the European equity factor. The more active contract can print the factor first without being the parent of the other. Pairwise overlap correlation, like pairwise Granger, cannot see the common driver. It can still tell you which quote is the stale one. That is the market maker's question, and it is the only question these numbers answer. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/09/article_463-visual-0.png) ## KEY POINTS - The Euro Stoxx 50 future's trade price leads the DAX future's midpoint on 99% of days in January to June 2021, and the Hayashi-Yoshida curve peaks at 100 microseconds. The Bobl leads the Bund by the same 100 microseconds. - The estimator multiplies price changes whose intervals overlap, then slides one series by a lag. Resampling onto a fast clock biases the correlation toward zero, which is the Epps effect. Measures are recorded only when a trade prints, so imbalance changes between trades are invisible. - Lead-lag strength multiplies curve asymmetry by the average squared correlation. The authors' example of correlations at 0.001 against 0.0005 produces a lead-lag ratio of 4 and a strength of 0.000182\. The real 95th percentile for trade price leading midpoint is 2.29\. Their typeset formula is missing an opening parenthesis; the prose version is the one used here. - Of nine microstructure series, trade price leading another name's midpoint (strength 2.29 at the 95th percentile) and best-quote imbalance leading another name's midpoint (strength 0.55) are the price-side results. Quoted spread, volume, and depth barely correlate. Daimler's imbalance leads the DAX future by one second, not by 100 microseconds. - More turnover and more frequent trades line up with leadership in equities, with adjusted R-squared under 1% until instrument-type dummies lift it to about 11%. The Schatz trades five times less euro volume than the Bund and leads it anyway. Day-to-day liquidity does not explain day-to-day strength. - June 2021 direction accuracy averages 51.8% against a 50.5% base-rate benchmark, the median is 50.4%, and accuracy correlates 0.67 with lead-lag strength. The 95th percentile is 63.6%. A window around the estimated lead time lifts the mean to 53.3%. No costed P&L is reported. The DAX future's relative spread in their table is 1.00 basis points, wider than the average edge. ## References - [Cross-impact of order flow imbalance in equity markets - Cont, Cucuringu, Zhang (Quantitative Finance, 2023)](https://doi.org/10.1080/14697688.2023.2236159?ref=aligrithm.com) - [The High-Frequency Trading Arms Race: Frequent Batch Auctions as a Market Design Response - Budish, Cramton, Shim (Quarterly Journal of Economics, 2015)](https://doi.org/10.1093/qje/qjv027?ref=aligrithm.com) - [Lead-Lag Relationships in Market Microstructure - Bender, Cestonaro, Schmidt (ssrn-5043260)](https://papers.ssrn.com/sol3/papers.cfm?abstract%5Fid=5043260&ref=aligrithm.com) --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ### 3.43 Stat-Arb Is a Clustering Problem: Multi-View Spectral > Any Signal URL: https://aligrithm.com/stat-arb-is-a-clustering-problem-multi-view-spectral-any-signal/ Last updated: 2026-10-16T08:09:10.000Z On S&P 500 names from 2000 to 2022, the best book in Raymond Leung's grid is multi-view co-regularized spectral clustering with 25 clusters, a plain long-short spread, and an Ornstein-Uhlenbeck entry and exit. After a 5 basis point charge on each leg at entry and again at exit, that book has an annualized Sharpe ratio of 0.830, a mean excess return of 2.7% a year, and a maximum drawdown of 7.0%. Hold the feature set fixed at the full intraday menu and the cost fixed, and the best distance-method book falls to a Sharpe ratio of 0.187\. The best single-view book falls to 0.495\. The clusterer moves the Sharpe ratio. Changing the signal inside one clusterer moves it less. The introduction also prints a pre-cost Sharpe ratio of 1.166 and a 4.6% drawdown for that same best book. No results table prints either figure. The after-cost table prints 7.0%. ## What this actually is You run pairs on the S&P 500\. The formation window is 252 business days, the trading window is the next 126, and the book holds 50 pairs. The classical membership test asks whether two normalized price paths stayed close over the formation window. Leung's claim is that this test admits pairs whose one-minute variance, skew, and betas disagree, and those are the pairs whose gaps fail to close. The method builds a separate similarity graph for each feature, the closing price included, fuses the graphs, and cuts the result into clusters before any spread rule runs. Think of a stack of tracing sheets. One sheet is the price path. The next is realized variance. The next is realized skew. Single-view clustering glues the sheets into one long strip and measures distance along the strip. Multi-view clustering leaves each sheet intact, draws a who-resembles-whom map on each sheet, and keeps the pair when the maps agree. A price match that fails the variance sheet loses its ticket. If that claim holds at the desk, the research budget goes to the clusterer and to a one-minute realized-moment pipeline, and the entry rule can stay a plain spread threshold. If the claim is the winner of more than 2,000 specifications on the same 22 years, the cost of believing it is a data build whose published Sharpe ratio was the maximum of a search the paper does not haircut. ## Price distance, after five basis points Price distance on large-cap US names after 2000 is the benchmark, and after 5 basis points it does not clear a t-statistic of 2. The formation window is 12 times 21, so 252 business days. The trading window is 6 times 21, so 126 business days. Six such trading windows overlap, the same staggered construction Gatev, Goetzmann, and Rouwenhorst used. A normalized price is that day's close divided by the close on the first day of the formation window. The distance method keeps the 50 pairs with the smallest gap. Leung prints the gap as follows. $$ \\mathrm{MSE}\_{ij} = \\frac{1}{T} \\sum\_{t=1}^{T} \\| z\_i - z\_j \\|\_2^2 $$ Read it as: z\_i is the stacked feature path of name i, a vector of length p times T, where p is the number of features and T is 252 formation days. The double-bar is Euclidean length. As printed, the quantity inside the sum does not depend on the day index t, because z\_i already contains every day. The sum therefore collapses, and the whole expression equals the squared Euclidean distance between the two stacked paths. Dividing by T, or skipping that division, does not change which 50 pairs win. On price alone, that distance is the sum of squared gaps in the normalized price index, which is the Gatev ranking. Worked on three formation days. Name A prints 1.00, 1.02, 1.01\. Name B prints 1.00, 1.05, 0.98\. Squared gaps are 0, 0.0009, and 0.0009\. The sum is 0.0018\. The mean is 0.0006\. Either number ranks this pair the same way against every other pair. On price alone, the two-standard-deviation rule has a Sharpe ratio of 0.571 before costs and 0.338 after, with a drawdown of 18.8% after costs. The after-cost mean excess return is 1.5% a year. Divide 0.015 by 0.338 and the annualized volatility is 4.4%. The t-statistic on that mean is 1.545\. Rad, Low, and Faff document the same fade on a broader CRSP universe: distance-method mean excess returns peaked near 1.80% a month in the early 1970s, which is 21.6% a year, and fell to about 0.20% a month by the 2010s, which is 2.4% a year. Leung's pre-cost mean on this rule is 2.6% a year, in that neighborhood, on a more liquid universe. Do and Faff asked whether simple pairs trading still works. Avellaneda and Lee, and Krauss's survey, report the same decline. On S&P 500 names after 2000, with 5 basis points, the classical book does not pay. ## Moments from the one-minute tape The extra features are one-minute moments, computed once a day, and they are what the multi-view books cluster on. Leung's prose defines realized variance as the sum of squared one-minute log returns. That is the estimator in Andersen, Bollerslev, Diebold, and Labys. Table 2 of the paper prints a different expression, the sum of squared changes in those returns. $$ \\mathrm{RV}\_{t,i} = \\sum\_{m=1}^{M} R\_{t\_m,i}^{2}, \\qquad R\_{t\_m,i} = \\log P\_{t\_m,i} - \\log P\_{t\_{m-1},i} $$ Read it as: on day t, for name i, take each one-minute log return R, square it, and add across the M minutes in the day. P is the one-minute price. The sum is that day's realized variance, in the units of a squared log return. Leung uses a one-minute grid throughout. Worked on three minutes: log returns 0.001, minus 0.002, and 0.0015\. Sum of squares is 0.000001 plus 0.000004 plus 0.00000225, which is 0.00000725\. The Table 2 first-difference form on the same three numbers, squared changes of the returns rather than squared returns, equals 0.00002125\. The prose definition is the one that matches the paper he cites. Table 2 does not match the prose. The source may have transcribed the cell wrong. There is no code in the paper, so which expression built the clusters is not recoverable from the text. The rest of this article treats realized variance as the sum of squared one-minute log returns, and treats the table cell as unreliable. Realized skewness and kurtosis follow Amaya, Christoffersen, Jacobs, and Vasquez. $$ \\mathrm{Skew}\_{t,i} = \\frac{\\sqrt{M} \\sum\_{m=1}^{M} R\_{t\_m,i}^{3}}{\\left(\\sum\_{m=1}^{M} R\_{t\_m,i}^{2}\\right)^{3/2}}, \\qquad \\mathrm{Kurt}\_{t,i} = \\frac{M \\sum\_{m=1}^{M} R\_{t\_m,i}^{4}}{\\left(\\sum\_{m=1}^{M} R\_{t\_m,i}^{2}\\right)^{2}} $$ Read it as: skewness is the third-moment sum, scaled by the square root of the minute count M, divided by the realized-variance term to the power 3/2\. Kurtosis is M times the fourth-moment sum, divided by the square of the sum of squares. Both are pure numbers. A negative skewness means the down minutes were fatter than the up minutes that day. Same three returns. The sum of cubes is 0.000000001 minus 0.000000008 plus 0.000000003375, which is minus 0.000000003625\. Skewness comes out at minus 0.322\. Kurtosis comes out at 1.26\. Three minutes is a toy day, enough to see the arithmetic, not a market fact. He also builds upside and downside semivariance, and an intraday beta of each name on SPY and on the sector ETFs: realized covariance with the reference, divided by the reference's realized variance. Daily market, size, value, and momentum betas, plus idiosyncratic volatility and total volatility, sit on the same menu, from a trailing regression of 252 days with a minimum of 126\. He standardizes each feature across names, one day at a time, to zero mean and unit standard deviation, before any clustering. The kitchen-sink collection, his feature set 14, is the one the best book uses. He labels it All. ## One graph per feature Single-view clustering flattens every feature into one vector. Multi-view clustering keeps a graph per feature. That split is the whole method. With p features, single-view stacks the p matrices, each of size T by N, into one matrix of height p times T. Each name is one long vector. A price match and a variance match are the same Euclidean inches. Multi-view leaves the p matrices separate and builds one similarity per feature. Leung's local scale uses the distance to the 7th nearest neighbor. $$ w\_{ij} = \\exp\\left( -\\frac{\\| x\_i - x\_j \\|^2}{\\sigma\_i \\, \\sigma\_j} \\right) $$ Read it as: w\_ij is the similarity of names i and j on one feature, a number between 0 and 1\. x\_i is that name's T-day path on the feature. Sigma\_i is the distance from i to its 7th nearest neighbor on the same feature. Closer paths produce a larger weight. Worked: squared distance 0.25, sigma\_i equal to 1, sigma\_j equal to 0.5\. The fraction inside the exponent is 0.25 divided by 0.5, so 0.5\. The weight is e to the power minus 0.5, which is 0.607. From that similarity matrix W he builds the symmetric normalization D to the power minus one half, times W, times D to the power minus one half. D is the diagonal matrix of row sums. He takes the top K eigenvectors, rescales each row to length 1, and runs k-means. That is his single-view spectral step, the procedure of Ng, Jordan, and Weiss. K is either 10 or 25\. He says he has no rule for the right K once several views are fused, so the two values are a sensitivity check. A single pairwise price gap is a thin graph, which is the complaint in the old article ["Stop Using Pairwise Granger: PCMCI for Financial Causality"](https://aligrithm.com/stop-using-pairwise-granger-pcmci-for-financial-causality/). Multi-view clustering builds one similarity graph per feature and fuses them. It is a clustering step, not a search for causal parents. The co-regularized multi-view step, his Algorithm 4, maximizes the sum across views of a trace term on each view's eigenvectors, plus a penalty, scaled by lambda, that rewards eigenvectors which agree across views. Each view's eigenvector matrix is constrained to have orthonormal columns. He freezes lambda at 1/2 and does not cross-validate it. He then runs k-means on the eigenvectors of the price view, which he treats as the most informative view. His reference list points this objective at Kumar and Daumé (2011). That paper's abstract says the method has no free parameter. The objective he prints has lambda. Keep his equation, and keep lambda at 1/2 as an untested choice. The citation may point at the wrong writeup of the same idea. Clusters from this cut can hold more than two names. He screens back to 50 pairs on the price gap, taking the ceiling of 50 divided by K pairs from each cluster, then the 50 smallest gaps overall. For K equal to 25 that is 2 pairs from each cluster. For K equal to 10 it is 5\. The screen stops the list from collapsing into the single tightest cluster. He also notes that insisting on 50 pairs, where the pairs literature often keeps 20, pulls in worse price fits, and he treats that as a downward bias on the results. ![Schematic: multi-view clustering keeps a separate T-by-N matrix for each feature, while single-view clustering stacks those matrices into one tall matrix of height p times T](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/09/article_454-multiview_vs_singleview.png) The left block is the multi-view input, one sheet per feature. The right block is the single-view input, the same sheets glued into one column. Everything after this picture is a consequence of which block you hand to the clusterer. ## Fifty pairs, six months, five basis points A cluster becomes a position through a spread, a threshold, and a 5 basis point toll on every leg at entry and at exit. The plain signal is the undirected spread: normalized price of name 1 minus normalized price of name 2\. The direction of the trade is fixed when the spread crosses a threshold, not when the pair is chosen. The cointegration signal follows the Engle and Granger two-step in the form Rad, Low, and Faff use. Over the formation window he regresses each normalized price on the other, keeps the regression with the smaller Augmented Dickey-Fuller statistic, and sets the spread equal to P1 minus beta-hat times P2\. The Dickey-Fuller statistic picks a direction. He does not require it to pass as a test of cointegration. The copula signal fits marginal distributions and a copula on formation returns, builds a cumulative mispricing index for each leg, and opens when the two indices sit on opposite sides of plus or minus 0.5, one dollar against one dollar. Multivariate cointegration, a Johansen vector on every name in the cluster, produced the worst books in the grid. He drops it from the rest of the reported results. Every Sharpe ratio below is a two-name trade. The two-standard-deviation rule opens when the spread crosses the 2.5 or 97.5 percent point of the distribution fitted on the formation window, and closes when the spread crosses zero. He fits several distributions and keeps the one with the lowest AIC; the Gaussian case is the classical Gatev threshold, with the round percentiles 0.025 and 0.975 in place of a strict two-sigma cut. The Ornstein-Uhlenbeck rule models the spread as a reverting diffusion. $$ dX\_t = -\\alpha X\_t \\, dt + \\eta \\, dB\_t $$ Read it as: X is the spread. Alpha is the reversion speed per day. Eta is the diffusion scale, in spread units per square root of a day. B is a standard Brownian motion. He estimates alpha and eta by maximum likelihood on the formation window. Bertram's optimal barriers are symmetric: enter at a-star, exit at minus a-star. Leung's cost input to that calculation is $$ c = 2 \\times 0.0005 \\times \\| \\beta \\|\_1 $$ Read it as: 0.0005 is 5 basis points. The leading 2 makes it a round trip. The L1 norm adds the absolute values of the position weights. For a one-dollar long and a one-dollar short, beta is the pair (1, minus 1), the L1 norm is 2, and c equals 0.002\. That is 20 basis points on one dollar of long notional, paid across both legs and both sides. He also prints a maximand for the optimal entry. It has the shape alpha times (2a plus kappa), divided by 2 pi times an error function of the square root of alpha over eta. Kappa is never defined. The denominator does not depend on the entry level a, so the printed expression has no interior maximum. The operational rule is still usable: fit alpha and eta on the formation spread, charge that c, enter at a-star, exit at minus a-star. The maximand as transcribed is not. The source may have dropped terms. This article does not replace them with a repaired formula. Daily portfolio return is an employed-capital average, following Gatev and Do and Faff. $$ R\_s = \\frac{1}{L\_s} \\sum\_{l=1}^{L\_s} \\mathrm{PnL}\_{s,l} $$ Read it as: L\_s is the number of trades open on day s, counted across all six overlapping windows. PnL of trade l is the dollar mark-to-market on a one-dollar long leg and a one-dollar short leg. The day's portfolio figure is the average of those dollar PnLs. He calls the series an excess return because the pair is dollar-neutral at the open and the accounting charges no financing. $$ \\mathrm{SR} = \\frac{\\overline{R}}{\\sigma} $$ Read it as: the annualized mean of that excess-return series, divided by its annualized standard deviation. He uses 252 business days wherever he annualizes. No separate risk-free rate appears in the tables. Worked on his own prices, with the arithmetic corrected. Long one dollar of a name at 100 dollars, short one dollar of the other at 5 dollars. The share counts are 0.01 and minus 0.2\. At the open the position value is zero, and the open cost is 0.0005 times (1 plus 1), so 0.0010 dollars. At the close the long name is at 150 and the short name is at 3.50, against prior marks of 145.75 and 4.25\. That day's mark-to-market is 0.01 times 4.25, plus minus 0.2 times (3.50 minus 4.25), which is 0.0425 plus 0.15, so 0.1925 dollars. The exit cost is 0.0005 times (1.50 plus 0.70), so 0.0011 dollars. From the open prices to the close prices the gross profit is 0.01 times (150 minus 100), plus minus 0.2 times (3.50 minus 5), which is 0.50 plus 0.30, so 0.80 dollars. Round-trip cost is 0.0021 dollars. Net is 0.7979 dollars. His writeup of this same trade prints a gross of 0.20 dollars and a net of 0.1989\. The 0.20 subtracts the short leg's gain instead of adding it. The 0.1989 then subtracts the exit cost alone, not the 0.0021 round trip. Both steps disagree with the share counts he set above them, which do produce the 0.1925 closing-day mark he also prints. The corrected totals are 0.80 gross and 0.7979 net. The 5 basis point rate is the median market-impact cost Frazzini, Israel, and Moskowitz report for US equities, 5.08 basis points on NYSE-AMEX and 5.03 on NASDAQ, over 1998 to 2016\. It is a large-cap number. He applies the same 5 basis points to ETFs, futures, and foreign exchange, and he says the cost literature off equities is thin. That flat rate on the other asset classes is his assumption. ## The clusterer, then the signal With the feature set held at All and costs turned on, the clusterer ranks the books, and the signal does not reorder that ranking. The best distance-method row has a Sharpe ratio of 0.187 and a drawdown of 14.3%. The best single-view row has 0.495 and a drawdown of 14.0%. The best multi-view row has 0.830 and a drawdown of 7.0%. That row is co-regularized spectral clustering, K equal to 25, the All feature set, the bivariate long-short, and Ornstein-Uhlenbeck barriers. Its mean excess return is 2.7% a year and the t-statistic is 3.797\. Divide 0.027 by 0.830 and the annualized volatility is 3.25%. Divide the t-statistic by the Sharpe ratio and square the result: 3.797 divided by 0.830, squared, is 20.9 years. That matches a 2000 to 2022 sample after a one-year formation. The distance-method rows produce the same 20.9 years. The mean, the Sharpe ratio, and the t-statistic are one identity. The tenth-best multi-view row on that same All set still prints a Sharpe ratio of 0.611, above the best single-view row. Across feature subsets, the top of the after-cost table is a run of multi-view long-short Ornstein-Uhlenbeck books from 0.748 to 0.830\. A copula row and a cointegration row sit inside the multi-view top ten, so no one signal owns the list. He reports the same for trading rules, in the supplement. The ordering that survives is distance, then single-view, then multi-view. Picking the 50 pairs is itself a ranking, and the old article ["Ranking Beats Forecasting for Many Trading Problems"](https://aligrithm.com/ranking-beats-forecasting-for-many-trading-problems/) is the same moral in another setting: the ordering imposed on the names does more work than the forecast attached afterward. Here the ordering is the cluster assignment. ![After-cost scatter of annualized Sharpe ratio against maximum drawdown for every clusterer, feature set, signal, and trading rule in the grid. Distance-method points sit toward low or negative Sharpe ratios and larger drawdowns. Multi-view points occupy the right edge, near Sharpe 0.6 to 0.83 and drawdowns near 5 to 10 percent.](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/09/article_454-sharpe_vs_drawdown_after_costs.png) Red points are the distance method. Green and olive points are multi-view. The right edge of the cloud, Sharpe ratios out near 0.8 and drawdowns near 7%, is multi-view. A mass of red points sits left of zero, with drawdowns from about 15% to about 50%. He says the grid contains well over 2,000 combinations, and he cites Harvey, Liu, and Zhu on multiple testing. His answer is the top-down sort and the plateau: many multi-view rows land near one another. He does not report a deflated Sharpe ratio. The plateau is in the table. The 0.830 is still the maximum of a search over one 22-year path. ![Cumulative excess return of the highlighted books, 2002 to 2022, compounded from daily employed-capital returns. The lower panel includes the 5 basis point cost. The multi-view Ornstein-Uhlenbeck book ends near 1.75. The distance method on the full feature set with the two-standard-deviation rule ends near 0.85.](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/09/article_454-cumulative_excess_highlighted.png) The lower panel is after costs. The pink line is the best book and finishes near 1.75 dollars per starting dollar. The olive dash-dot line is the distance method on the same All feature set with the two-standard-deviation rule, and it finishes near 0.85\. Stuffing realized moments into a price-distance match, and leaving the clusterer alone, does not rescue the path. The two lines differ by the clusterer. On the Fama-French five-factor model plus Carhart momentum, with Newey-West standard errors and 6 lags, the after-cost alpha of the best book is 2.4% a year, t-statistic 3.655\. The two-standard-deviation cousin of that same clusterer has an alpha of 3.0%, t-statistic 2.276\. Plain multi-view spectral clustering on All, with the two-standard-deviation rule, has 2.0%, t-statistic 2.139\. Distance-method alphas are indistinguishable from zero on this regression. Read the scale against the table, because the table and the caption disagree. Table 9 prints the best book's alpha as 0.024, and the caption says that 1.00 means 1.00% a year, which would turn 0.024 into 0.024%. The prose in the paper calls the same entry 2.4%, and it calls the neighboring entries 2.0% and 3.0% for the printed values 0.020 and 0.030\. The mean column of the performance table prints 0.027 for a return that has to be 2.7% a year, or the Sharpe-ratio identity above breaks. The market loading in the next column is minus 0.067, a slope. The coherent reading is the prose: 2.4% a year. The caption's scale sentence does not reconcile with the printed cells. The source may have written the example for a percent-formatted table it did not ship. That market loading is minus 0.067 with a t-statistic of minus 12.9\. The momentum loading is minus 0.047 with a t-statistic of minus 7.5\. Profitability and investment loadings are also away from zero. The book is small in its market exposure and short momentum, which is what fading a gap does, and the 2.4% is what remains after those loadings. Per trade, after costs, the best book shows a positive cumulative profit on 69.2% of trades, and 62.2% of trades close by the rule rather than by the end of the 126-day window. Mean holding period is 46.3 days. He also prints a mean daily profit of 38.13 basis points and a mean profit per trade of 54.32 basis points. Those two figures do not multiply. Daily profit of 38.13 basis points over 46.3 days is about 1,765 basis points, not 54\. The 54.32 basis points spread over 46.3 days is 1.17 basis points a day, next to the portfolio's 1.07 basis points a day from the 2.7% annualized mean. The daily mean and the daily standard deviation inside that table do agree with the printed information ratio, 38.13 divided by 109.34 equals 0.349\. They do not agree with the cumulative-profit column or with the portfolio mean. One of the basis-point columns is on a different scale from the label. The win rate and the holding period have no such conflict. The 38.13 figure should not be used. The introduction's 4.6% drawdown is the other mismatch. The after-cost ranking table prints 0.070 on this book. The 18.8% benchmark drawdown does match the price-only distance-method row. The pre-cost scatter reaches Sharpe ratios a bit above 1.1 with drawdowns near 5%, so a pre-cost 1.166 and a drawdown near 4.6% could be unlabeled points on that figure. They are not in a table. The numbers to carry are 0.830 and 7.0%. The best book has 221 trades open on an average day, against a committed-capital denominator of 6 times 50, which is 300\. Occupancy is about 0.74\. Reserving all 300 slots would scale the 2.7% mean to about 2.0%, since 0.027 times 221 divided by 300 is 0.0199\. A constant rescaling leaves the Sharpe ratio unchanged. Occupancy varies, with a standard deviation of 52 open trades, so a committed-capital Sharpe ratio would move. He says the ranking survived that version and he does not print it. The book is also not 50 stable couples: Table 11 counts 5,191 distinct pairs over the sample. Futures and the three foreign-exchange quote currencies, on shorter samples and with K equal to 5 and 10 pairs, show the same pattern in his summary: the best Sharpe ratio is a multi-view book, and it beats the distance-method benchmark on Sharpe ratio and on drawdown. On the ETF list the Sharpe-ratio winner was a single-view book, and he says the t-statistic on its mean was weak. He does not claim a tradable ETF result. The S&P 500 universe is the index's own daily constituents from CRSP, with Kibot prices adjusted for splits and dividends. Where two share classes of one company both sit in the index, he keeps the longer history. The sample starts at the end of 2000 because earlier one-minute coverage of the constituents is poor. He says a five-minute grid in place of the one-minute grid did not change the study in a material way. That comparison is not in the tables. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/09/article_454-visual-0.png) ## KEY POINTS - On S&P 500 constituents from 2000 to 2022, multi-view co-regularized spectral clustering with 25 clusters, a long-short spread, and Ornstein-Uhlenbeck barriers posts an after-cost Sharpe ratio of 0.830, a mean excess return of 2.7% a year, and a maximum drawdown of 7.0%, at a 5 basis point cost per leg per side. - With the feature set and the cost held fixed, the best distance-method Sharpe ratio is 0.187 and the best single-view Sharpe ratio is 0.495\. The tenth-best multi-view row on that set is still 0.611\. The clusterer ranks the books. The signal and the trading rule do not. - The classical price-distance book with the two-standard-deviation rule falls from a Sharpe ratio of 0.571 to 0.338 after costs, with an 18.8% drawdown and a t-statistic of 1.545\. That matches the post-2000 fade reported by Rad, Low, and Faff, Do and Faff, and Avellaneda and Lee. - Realized variance in the prose is the sum of squared one-minute log returns. Table 2 prints the sum of squared changes in those returns. On a three-return toy those are 0.00000725 and 0.00002125\. The clusters may have been built from either. The text does not say which. - The introduction's pre-cost Sharpe ratio of 1.166 and its 4.6% drawdown are not in any results table. The after-cost table prints 7.0% for the same named book. A worked round trip in the paper prints 0.20 dollars of gross profit where the share counts give 0.80\. The per-trade daily profit of 38.13 basis points does not multiply out to the printed 54.32 basis points over a 46.3-day hold. - The grid exceeds 2,000 specifications and there is no deflated Sharpe ratio. Alphas of 2.0% to 3.0% a year on the multi-view books come with a market loading near minus 0.07 and a short momentum loading. The ETF universe does not deliver a tradable version of the result. The 5 basis point cost on futures and foreign exchange is an assumption carried over from US equities. ## References - [Statistical Arbitrage via Single-view and Multi-view Spectral Clustering on Mixed Frequency Data (Raymond C. W. Leung)](https://papers.ssrn.com/sol3/papers.cfm?abstract%5Fid=4975855&ref=aligrithm.com) --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ### 8.11 Stop Using Pairwise Granger (causality upgrade) URL: https://aligrithm.com/stop-using-pairwise-granger-causality-upgrade/ Last updated: 2026-10-16T08:09:10.000Z A pairwise screen of the 64-futures book, at a 5% level and five lags, expects 1,008 false links when every cross-link is null. That book is the one in the old article ["Network Momentum as a Cross-Asset Factor,"](https://aligrithm.com/network-momentum-as-a-cross-asset-factor/) and the screen is the bivariate test the old article ["Stop Using Pairwise Granger: PCMCI for Financial Causality"](https://aligrithm.com/stop-using-pairwise-granger-pcmci-for-financial-causality/) took apart. The sequel is the object you trade. Both live graphs in this pillar are still pair graphs: one connects futures whose momentum features co-move, and one connects every currency the market lists. The parent graph is the replacement. ## What this actually is You run 64 futures. Crude, the S&P, the bund, and the euro sit on one graph, and the old article ["Network Momentum as a Cross-Asset Factor"](https://aligrithm.com/network-momentum-as-a-cross-asset-factor/) turns that graph into a position: each contract's signal is a weighted average of its neighbors' momentum. The published book earns a Sharpe of 1.51 and 22% a year, gross, from 2000 to 2022\. The neighbors are whoever co-moves. A dollar squeeze lifts crude and the S&P together, and the graph writes an edge the spillover did not produce. The method keeps an edge when the link remains after the other series that explain both sides have been removed, then treats the surviving edges as one graph. A harbor switchboard works the same way. Every pair of docks that rings at the same hour gets a patch cable. Most of the ring is the building generator. Filter the generator out and pull every cable that goes silent. Calls go out on the cables that stay live. You operate the board. At the desk, crude's signal would come from Brent and heating oil, and the S&P cable would be gone. Delete that cable and a short crude signal becomes a long one, on the same neighbor returns. The cost of a mistaken deletion is the cross-class sleeve the old article measured: equity momentum built inside equities scores a Sharpe of 0.30, and the same equity trades fed by the full cross-asset graph score 0.95\. Miss a real parent, because the link is same-day or the true driver was left out of the file, and you have cut the sleeve that carried that gap. Keep a false parent and the 1.51 is a factor bet wearing a momentum label. ## A pair screen on these two books paints a thousand false arrows The count this section settles is the false-arrow pile on the two books you would hand the screen. Call the false-positive level alpha, the number of assets N, and the longest lag you test tau-max. Each ordered pair of distinct assets, at each lag, is one test. Under a null graph and a calibrated test, the expected number of false directed links is alpha times N times (N minus 1) times tau-max. $$ \\mathbb{E}\[\\text{false links}\] = \\alpha \\, N \\, (N - 1) \\, \\tau\_{\\max} $$ Read it as: alpha is the per-test false-positive level (a fraction, no units), N is a count of assets, tau-max is a count of lags, and the product is a count of arrows you expect to keep even though none of them is real. Five lags is an arbitrary trading-week window; the formula is linear in that choice, so halve the lags and you halve the pile. Work the two books. For the futures book, N is 64 and tau-max is 5, so the test count is 64 times 63 times 5, which is 20,160, and 5% of that is 1,008\. One lag instead of five still leaves 64 times 63 times 0.05, which is 201.6 false arrows. For the ten-currency book in the old article ["Graph Learning for FX: Interest-Rate-Parity Statarb Done Right,"](https://aligrithm.com/graph-learning-for-fx-interest-rate-parity-statarb-done-right/) the same five-lag screen runs 10 times 9 times 5 tests, which is 450, and expects 22.5 false arrows. One lag expects 4.5\. The chart is that quadratic: the ten-currency dot sits near the origin, and the 64-futures dot sits at 1,008. ![Expected false directed links from a pairwise screen at 5 percent and five lags, marked at 10 currencies (22.5) and 64 futures (1,008)](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/09/article_434-expected_false_links.png) That 1,008 is the optimistic count. It assumes the test is calibrated. Runge and coauthors show the original PC algorithm, the condition-selection skeleton behind PCMCI, shows an upper tail of false positives reaching 25% on links whose average autocorrelation exceeds 0.7\. Futures momentum features are smooth by construction. A pairwise Granger test conditions on the target's own lags and leaves the driver's parents out of the conditioning set, which is the same failure. Treat 1,008 as a floor. Full conditional independence, the multivariate Granger test, conditions on N times tau-max minus one lagged series. At five lags that is 319 controls in one regression. The shortest lookback Pu and coauthors use to learn their graph is 252 trading days. 319 coefficients on 252 days is an unidentified regression. Their longest lookback, 1,260 days, can host 319 coefficients and leaves the power problem the old article walked on the climate sample: full conditioning held about 80% detection at five series and 40% at twenty, on 150 observations. Conditioning on everything produces a blind test, or an equation the 252-day window cannot fit. ## The false-discovery cut on the whole graph The question this section settles is why a 5% test on each link still hands you a junk graph, and what cut replaces it. PCMCI, as the old article sets it out, holds each link's false-positive rate near 5%. On a null graph that rate times 20,160 tests is the same 1,008 arrows. The momentary conditional independence step scores one link at a time. Runge and coauthors say so: in the numerical experiments they scored links one at a time and left the false-discovery rate uncontrolled. They do print the adjustment you need, the Benjamini-Hochberg q-value on the whole lagged graph. $$ q = \\min\\left(p \\,\\frac{m}{r},\\, 1\\right), \\qquad m = N^{2} \\, \\tau\_{\\max} $$ Read it as: p is the raw p-value of one lagged link (a probability, no units), r is that p-value's rank after you sort every test from smallest to largest (a count), m is the number of tests, and q is the adjusted value you compare with your false-discovery target. Runge sets m to N squared times tau-max, which counts own-lags as well as cross-links. For 64 futures and five lags, m is 4,096 times 5, which is 20,480\. The rank-1 bar for a 5% false-discovery target is 0.05 divided by 20,480, which is 2.44 times 10 to the minus 6\. A link you would have called at 5% in a single test is nowhere near that bar. Work three links on that book. A raw p of 0.000001 at rank 1 gives q equal to 0.000001 times 20,480, which is 0.0205, and it survives a 5% target. A raw p of 0.001 at rank 1 gives 0.001 times 20,480, which is 20.48, so the cap sets q to 1 and the link fails. A raw p of 0.0001 at rank 50 gives 0.0001 times 20,480 divided by 50, which is 0.041, and it survives. A raw p of 0.0002 at rank 51 gives 0.0002 times 20,480 divided by 51, which is 0.080, and it fails. The 1995 procedure also replaces each q with the smallest q at that rank or any worse rank, so the adjusted values stay nondecreasing as you walk down the list. On these three numbers the running minimum changes nothing, because 0.0205, 0.041, and 0.080 rise with rank. The formula Runge prints is the per-rank factor with the cap at 1, and a worse rank with a much smaller p can undercut an earlier q. Apply the running minimum on top of the printed factor. Two consequences for the desk. First, PC1's loose threshold, the one the old article tells you to keep liberal so real parents survive, is a regularizer inside the condition search. The graph comes from the MCI test plus this q cut. At 100 series, Runge reports that more than 80% of the conditions PC1 kept were false positives, even while it recovered more than 80% of the true parents. The MCI test, then this q cut, turns that superset into edges you can propagate. Stop after PC1 and you have rebuilt the spaghetti with extra steps. Second, the q cut is severe on a 64-name book because m is twenty thousand. The Nino-to-British-Columbia link, partial correlation about 0.10 on 468 months, posted p = 0.036\. At rank 1 on this book that p-value gives q = min(0.036 times 20,480, 1) = 1\. A link with that p-value fails the cut at every rank. A thick graph at this m is a graph that skipped the cut. ## Parent weights flip the crude position The question this section settles is what the network-momentum position does when the edge weight is a surviving parent instead of a co-movement score. Pu and coauthors learn a symmetric, nonnegative adjacency. The objective rewards assets whose momentum-feature paths vary little across an edge, and two penalties keep the graph from falling into isolated nodes or a handful of huge weights. The result connects series that look alike. Alike includes a common driver. They then scale the adjacency by its degrees and propagate eight momentum features along the scaled edges. $$ \\tilde{A}\_{ij} = \\frac{A\_{ij}}{\\sqrt{d\_i \\, d\_j}}, \\qquad \\tilde{u}\_{i} = \\sum\_{j} \\tilde{A}\_{ij} \\, u\_{j} $$ Read it as: A-sub-ij is the learned similarity between assets i and j (a nonnegative weight, no dollar units), d-sub-i is the sum of row i (the degree), A-tilde is the degree-scaled weight, u-sub-j is asset j's vector of eight momentum features, and u-tilde-sub-i is the feature vector you hand the regression for asset i. The sum runs over neighbors, and there is no self-loop. The regression in the old article then maps that vector to a sign. The sign is the position. Put three names on it. Call them crude (CL), Brent (BZ), and the S&P (ES). Choose similarity weights so the arithmetic is exact: CL-BZ is 0.8, CL-ES is 0.6, BZ-ES is 0.2\. Degrees are then 1.4, 1.0, and 0.8\. The scaled CL-BZ weight is 0.8 divided by the square root of 1.4, which is 0.676\. The scaled CL-ES weight is 0.6 divided by the square root of 1.12, which is 0.567\. Give Brent a vol-scaled one-month return of +1.2 and the S&P a return of minus 2.0\. Crude's propagated signal is 0.676 times 1.2 plus 0.567 times minus 2.0, which is 0.811 minus 1.134, so minus 0.323\. Short crude, because the index dumped and the similarity graph trusts that dump. Now delete the CL-ES edge the way a failed parent test deletes it, leave BZ-ES in place, and recompute the degrees. Crude's degree falls to 0.8, Brent's stays at 1.0\. The scaled CL-BZ weight is 0.8 divided by the square root of 0.8, which is 0.894\. The signal is 0.894 times 1.2, which is +1.073\. Long crude. Same neighbor returns, opposite position. The entire difference is the 0.6 similarity the S&P was allowed to write. That sign flip is an illustration on chosen weights, and neither paper reran the 2000-2022 book on a parent graph, so the Sharpe of 1.51, the 22% gross annual return, and the equity gap from 0.30 to 0.95 belong to the similarity graph. Read the ablation before you spend it. Intra-class edges alone score 1.21\. Inter-class edges alone score 0.91, with a max drawdown of 60.7% against 19.9% for the full book, and the two sleeves correlate about 0.51\. The cross-class edges are where a missing dollar factor hides, and they are also where the extra equity Sharpe sits. The test has to decide edge by edge. Deleting every cross-class cable throws away a sleeve that diversifies the intra-class book. Keeping every cross-class cable because the Sharpe went up is how the pair screen gets back in. Two facts about their learned graph make the substitution concrete. Edge sparsity, the share of possible pairs that carry a nonzero weight, sits under 0.4 for most of the sample, and the Jaccard overlap of successive graphs sits above 0.99\. The similarity graph is sparse and sticky. A dollar regime can glue the same edges in place for years, and the Jaccard index will read 0.99\. The overlap of successive similarity graphs leaves the conditional-independence question open. Costs stay where the old article left them until someone measures turnover on the pruned graph. Network momentum keeps a positive cost-adjusted Sharpe through 3 basis points and prints minus 0.67 at 5 basis points. A thinner parent graph ought to turn the book over less. The paper reports no turnover on a pruned graph. I am against quoting 1.51 as the Sharpe of a causal graph. ## Ten currencies, ninety directed pairs, one constraint graph The question this section settles is which edges in the FX paper are discovered links, and which are the pairs the market lists. Hong and Klabjan build the prediction graph with currencies as nodes and tradable exchanges as edges, and they put government-bond yields on the nodes. For ten currencies that is 90 directed crosses, or 45 reciprocal pairs once you impose that the rate one way times the rate the other way equals one. Interest-rate parity is the reason the yields sit on the nodes. Which cross predicts which is a separate test. The triangle identity makes a direct cross a function of two others, so a pairwise regression of euro-dollar on sterling-dollar lights up because both are functions of the dollar leg. PCMCI on the ten rates, with the bond yields in the conditioning pool, asks whether a cross moves the target after the other crosses and the yields are accounted for. The five-lag screen above expects 22.5 false directed links out of 450\. Their prediction graph keeps all 90 by construction. Their second graph is a different object, and mixing the two is how this upgrade goes wrong. In the arbitrage stage, exchanges become nodes, and an edge exists when an entry of the projection matrix is large. That entry says this trade's quantity moves when that trade's quantity moves, because the flat-holdings constraint says so. It is an accounting edge. Leave it alone. Parents answer what predicts. The projection answers what the flat-book constraint couples. Feed parent weights into the projection and you have changed the constraint. Feed the projection into the momentum propagation and you have called an accounting identity a cause. The scoreboard the old article reports is a risk cut on that constraint graph, and one column does not match the others. Averaged across USD, EUR, and JPY as the home currency, the graph-net information ratio is 61.89% higher than the linear-program benchmark, Sortino is 45.51% higher, annual volatility is 52.23% lower, max drawdown is 44.77% lower, and annual return is 22.73% lower, from 6.19% to 4.80% on the USD row. Recomputing the three-currency average from the rounded cells lands within a hundredth of those printed percents, so the averages are the table. The information-ratio column is a different series. On the USD row, annual return divided by annual volatility is 6.19 divided by 1.41, which is 4.39, for the benchmark, and 4.80 divided by 0.68, which is 7.06, for the graph net. The information-ratio cells on that same row read 27.21 and 43.86, in a table headed as percents. Dividing 4.39 and 7.06 by the square root of 252 lands near 0.277 and 0.445, against 0.272 and 0.439 if you slide the decimal on the cells. Close, and not equal. The source may be computing the information ratio on a different gain series than the annual return and volatility rows. Keep the printed 61.89% relative lift. Dividing 4.80 by 0.68 recovers 7.06, the return-to-volatility ratio, and leaves the printed cell at 43.86. The forecast moved little, which is the other reason the parent screen belongs in front of this net. Mean squared error on next-day log-rate changes, FX rates plus interest rates, is 2.8635 against 2.9083 in units of 10 to the minus 5\. The relative gap is (2.9083 minus 2.8635) divided by 2.9083, which is 1.54%. Their paired tests detect the gap, and the change in the rate you hand the arbitrage stage is small. The 61.89% lift came from the risk objective and the constraints, on a graph that keeps every listed pair. A parent screen would ask whether that risk cut survives once the indirect legs are gone. Hong and Klabjan report no such comparison. The paper also charges no spread. Spot loops of a few percent die in the spread, so the lift is a gross, costless number on top of the column mismatch. ## Run the parent graph, and leave the published Sharpe where it is The question this section settles is the Monday order, and which published number stays on the graph that earned it. Use the universe the strategy trades, and add the factor you suspect is the common driver. For the futures book that means a dollar proxy or a rates series inside the condition set. Leave it out and every surviving cross-asset edge can be that factor in disguise. Run the two PCMCI stages from the old article with partial correlation on the vol-scaled returns. Price levels fail the stationarity the test assumes. Then apply the q cut across the whole lagged graph, target 5% false discoveries, with m equal to N squared times tau-max. The edge weight you propagate is the momentary partial correlation on the links that survive, entered as A-sub-ij in the degree scaling above, and zero otherwise. Skip the Laplacian objective. You replaced it. For the currency book, run the parent screen on the rates before the prediction net, and leave the projection graph as the constraint layer. Estimate parents on data through date t and trade date t plus one. The futures paper refits every five years. Refit inside each window. Common drivers rewire when their community ratio drops, which the futures paper records in the 2007 to 2008 window, and a Jaccard above 0.99 on the similarity graph says nothing about that refit. The assumptions you inherit from Runge bind harder here than in the climate sample. Causal sufficiency fails when the dollar, the funding factor, or a policy shock is absent from the file, and the method will hand you parents that are that missing series. Lagged links get a direction from the clock; same-bar links stay undirected, and a large piece of equity-bond spillover is same-bar. The equity Sharpe gap of 0.65 points may live in that bucket, and this screen will not see it. Observational noise at the scale of the signal breaks the false-positive control, because the common driver stops being detectable. Market closes are noisy. An absent link is the more believable output. A present link is a hypothesis. I am against wiring 1.51, or the 61.89% information-ratio lift, to this screen. Those numbers were earned by a similarity graph and a constraint graph. The upgrade is the graph you are allowed to propagate. Reproduce the cross-class gap on the parent graph, net of your own costs, before you spend either headline. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/09/article_434-visual-0.png) ## KEY POINTS - A pairwise screen of 64 futures at 5% and five lags runs 20,160 directed tests and expects 1,008 false links on a null graph. Ten currencies expect 22.5\. One lag cuts those piles to 201.6 and 4.5\. The 1,008 figure assumes a calibrated test; on strong autocorrelation the PC algorithm's false-positive rate reaches 25%, so treat 1,008 as a floor. - Full conditioning on the same futures book asks for 319 controls. The 252-day graph window cannot fit that regression. The 1,260-day window can, and detection power is the collapse the PCMCI article measured: about 80% at five series, 40% at twenty. - Each link's 5% rate on 20,160 tests is the same 1,008 arrows. Benjamini-Hochberg on m = N squared times tau-max, which is 20,480 for this book, demands a rank-1 p-value under 2.44 times 10 to the minus 6\. A p of 0.000001 at rank 1 survives (q = 0.0205); a p of 0.001 dies; the climate link's p of 0.036 fails at every rank. Apply the running minimum the 1995 procedure adds on top of the per-rank factor Runge prints. Runge's own experiments skipped this cut. - Swap co-movement weights for parent weights inside the same degree scaling. With CL-BZ at 0.8, CL-ES at 0.6, Brent at +1.2 and the S&P at minus 2.0, the similarity signal is minus 0.323 and the parent signal, after the S&P edge is deleted, is +1.073\. Same features, opposite position. - The published 1.51 Sharpe, 22% gross year, and the equity gap from 0.30 to 0.95 belong to the similarity graph. Intra-class alone is 1.21, inter-class alone is 0.91 with a 60.7% max drawdown against 19.9% for the full book, and the sleeves correlate about 0.51\. Decide cross-class edges one at a time. Costs still go negative at 5 basis points (Sharpe minus 0.67). No paper measures turnover after the prune. - The FX prediction graph keeps all 90 directed crosses because they are listed, not because they survived a test. The second graph's edges are entries of a constraint projection. Keep those two graphs apart. The 61.89% information-ratio lift is a gross risk cut, and annual return over annual volatility is 7.06 on the USD graph-net row while the printed information-ratio cell is 43.86\. The next-day forecast gap is 1.54% relative. - Monday's order: put the suspected common driver in the file, run PCMCI on vol-scaled returns, cut the graph with the q-value, propagate the surviving partial correlations, leave the FX projection alone, and refit inside each window. Do not attach 1.51 or 61.89% to a screen nobody has run. ## References - [Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing (Benjamini and Hochberg 1995)](https://doi.org/10.1111/j.2517-6161.1995.tb02031.x?ref=aligrithm.com) - [Detecting and quantifying causal associations in large nonlinear time series datasets (Runge, Nowack, Kretschmer, Flaxman, and Sejdinovic 2019)](https://doi.org/10.1126/sciadv.aau4996?ref=aligrithm.com) - [Network Momentum across Asset Classes (Pu, Roberts, Dong, and Zohren 2023)](https://doi.org/10.2139/ssrn.4540651?ref=aligrithm.com) - [Graph Learning for Foreign Exchange Rate Prediction and Statistical Arbitrage (Hong and Klabjan)](https://arxiv.org/abs/2508.14784?ref=aligrithm.com) --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ### 10.23 Moving Average Distance: The Technical Indicator That Passed the Cross-Section URL: https://aligrithm.com/moving-average-distance-the-technical-indicator-that-passed-the-cross-section/ Last updated: 2026-10-16T08:09:11.000Z Divide a stock's 21-day moving average by its 200-day moving average. That is the whole signal, and it is about as retail as a signal gets. Avramov, Kaplanski and Subrahmanyam ran it across 13,828 US firms and 1,353,679 monthly returns from July 1977 through December 2018, and the value-weighted hedge portfolio produced an annual alpha of 9.05% against the five Fama-French factors plus momentum plus a crisis dummy, with a t-statistic of 3.02\. Momentum does not survive the encounter: the standard winner-minus-loser spread earns a 17.58% three-factor alpha on its own, and 2.25% with a t of 0.96 once you add the moving average distance factor to the right-hand side. One dollar placed in the top decile in July 1977 finishes at 959.69 dollars against 92.47 for the CRSP value-weighted market. Now the part that decides whether you can trade it. The entire 9% sits on the long leg. The short leg's alpha is minus 0.05% a year with a t-statistic of minus 0.02, which is a rounding error wearing a minus sign. ## What this actually is You own a trend filter. Someone told you that when the 50-day crosses the 200-day you buy, and when it crosses back you sell, and every clean test of that crossover on an index comes back flat or worse. The authors check this directly: a dummy for whether the golden cross or death cross fired this month carries a t-statistic of 1.17 in the 21-day version, which is nothing. The binary event is dead. The question is whether anything in the moving average pair is alive. What they do about it: stop treating the crossover as an event and start treating the gap between the two averages as a number, then rank every stock in the market by that number each month and buy the widest gaps while selling the narrowest. The mechanism they propose is anchoring. Move to a new city and your sense of what a coffee costs stays pinned to your old town for months. A price that is twenty cents off you accept on the spot. A price that is triple you dismiss as a tourist trap and only update grudgingly, over many visits. The distance between your running one-month impression and your running year-long impression measures how much updating you still owe. Their claim is that stock prices owe the same updates, and that the gap between the short and long averages is a readable meter of the debt. At the desk this changes two things. You rank the cross-section on the ratio instead of trading crossovers, and you hold for three to twelve months instead of reacting to the cross. Believing it when it is wrong costs you specifically: the ratio correlates 0.58 with momentum and 0.64 with the 52-week high, so a book you think is a new anomaly may be a momentum book with extra turnover and extra fees. ## The ratio, and the screen that is supposed to sharpen it The construction runs in two steps, and the second one does less work than it appears to. $$ MA\_k(t) = \\frac{1}{k}\\sum\_{j=0}^{k-1} P\_{t-j}, \\qquad MRAT\_t = \\frac{MA\_{21}(t)}{MA\_{200}(t)} $$ Read it as: MA sub k at time t is the arithmetic average of the last k daily closing prices in dollars, split- and dividend-adjusted, and MRAT is the 21-day average divided by the 200-day average, a pure ratio with no units. A value above one says the recent month has traded above the year-long baseline. Worked example. A stock whose last 21 closes average 52.00 dollars and whose last 200 closes average 45.00 dollars has an MRAT of 52.00 divided by 45.00, which is 1.1556\. A stock averaging 33.00 dollars over the month against 40.00 over the year has an MRAT of 0.8250\. Across the full sample the mean MRAT is 1.047 with a standard deviation of 0.204, so 1.1556 is a bit more than half a standard deviation above average and 0.8250 is about one standard deviation below. The second step converts the ratio into a plus-one, minus-one, zero variable. $$ MAD\_{i,t} = \\begin{cases} +1 & \\text{if } MRAT\_{i,t} \\in D\_{10} \\text{ and } MRAT\_{i,t} > 1 + \\sigma\_t \\\\\[4pt\] -1 & \\text{if } MRAT\_{i,t} \\in D\_{1} \\text{ and } MRAT\_{i,t} < 1 - \\sigma\_t \\\\\[4pt\] \\ \\ \\ 0 & \\text{otherwise} \\end{cases} \\qquad \\sigma\_t = \\operatorname{sd}\_i\\!\\left(MRAT\_{i,t}\\right) $$ Read it as: a stock scores plus one if it lands in the top MRAT decile that month and its ratio also clears one plus sigma, minus one if it lands in the bottom decile and falls below one minus sigma, and zero otherwise. Sigma is not a time-series volatility. It is the standard deviation of MRAT computed across all stocks in that single month, so the bar moves with how dispersed the market is. Worked example with the sample-wide sigma of 0.204\. The long bar sits at 1.204 and the short bar at 0.796\. The stock at 1.1556 fails the long bar even if it makes the top decile, so it scores zero. A stock at 58.00 over 45.00, or 1.2889, clears it and scores plus one. The stock at 0.8250 misses the short bar of 0.796 and also scores zero. That sounds like a tight double filter. Fit a lognormal to the Table 1 moments and it mostly is not. ![Density of the MRAT ratio fitted to the paper's Table 1 moments, showing the bottom decile cutoff at 0.803 nearly coinciding with the one-minus-sigma line at 0.796, and the top decile cutoff at 1.316 sitting above the one-plus-sigma line at 1.204](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/09/article_600-mrat_double_screen.png) The decile cutoffs land at 0.803 and 1.316\. The sigma bars land at 0.796 and 1.204\. On the short side the two cutoffs are almost the same number, and on the long side the decile boundary is the binding one. Using the pooled dispersion, the sigma screen removes close to nothing, and "MAD equals plus one" is close to "top MRAT decile." The screen only bites in months when cross-sectional dispersion runs well above its sample average, which is exactly when the top decile is already extreme. The authors note that going to two sigma strengthens the results "at the cost of having several months with relatively small number of investable stocks," which is a polite way of saying the screen and the portfolio count trade off directly. Treat the sigma condition as a mild trim, not as the source of the edge. ## Surviving twenty-odd controls The first test is whether the ratio still predicts once you throw the standard anomaly zoo at it. They use Fama-MacBeth, which means one cross-sectional regression per month and then a t-test on the time series of slopes. $$ R\_{i,t+1} = a\_t + b\_t\\,MAD\_{i,t} + \\sum\_{k} c\_{k,t}\\,X\_{k,i,t} + e\_{i,t+1}, \\qquad \\hat{b} = \\frac{1}{T}\\sum\_{t=1}^{T} b\_t, \\qquad t(\\hat{b}) = \\frac{\\hat{b}}{\\operatorname{se}\_{NW}(b\_t)} $$ Read it as: for each of the 498 months, regress next month's return on the signal and on every control X, keep the slope b on the signal, then average those 498 slopes and divide by their Newey-West standard error. The controls here run past twenty variables, including size, book-to-market, idiosyncratic volatility, turnover, Amihud illiquidity, standardized unexpected earnings, net stock issues, asset growth, gross profitability, accruals, return on assets and equity, net operating assets, the Ohlson distress score, a dummy for price above the 200-day average, MACD, and four separate past-return windows. The results. For one-month-ahead returns the continuous ratio carries a t of 4.92 and the plus-one-minus-one version carries a t of 6.65\. Over months two through six the slopes hold at t equals 3.60 and 6.12\. Over months seven through twelve both collapse, to t equals 1.51 and 0.66\. In the same regressions momentum turns negative and insignificant, minus 0.00 with a t of minus 0.03, and the 52-week high does the same at minus 0.76 with a t of minus 1.67\. The trend factor of Han, Zhou and Zhu is the one control that stays strongly positive alongside. One arithmetic problem, and I am leaving it as the source printed it. Table 2's note says the slopes are multiplied by ten thousand. Take that literally and the MAD slope of 0.51 implies a top-minus-bottom spread of 2 times 0.51 times ten to the minus four, which is 0.0102% a month. Table 3 reports the actual raw spread at 1.58% a month. Reading the same slopes as percent per month gives 1.02% a month, which sits below the raw 1.58% by about the amount you would expect twenty controls to absorb. The scaling note is off by two orders of magnitude. The coefficients themselves are internally consistent, so nothing in the paper's conclusions moves, but do not quote the slope magnitudes without fixing the units yourself. Two more things in this table deserve flagging. Harvey, Liu and Zhu argue that with thousands of candidate factors in circulation, a new one should clear a t-statistic of 3.0 rather than 2.0\. Most of the MAD t-statistics do. The 2001 through 2018 subsample does not: it comes in at 2.61 for MAD and 3.06 for the continuous ratio. So the claim that the effect persists into the modern era rests on a coefficient that fails the bar the same paper cites two pages later. And in negative market states, defined by a negative trailing two-year market return, the effect vanishes entirely, with t-statistics of 1.82 and 0.40\. The authors blame the small sample, and they are right that only 10.6% of sample months qualify, but the honest reading is that we have close to no evidence about how this signal behaves through a sustained bear market. ## The long leg is the strategy Sort into value-weighted deciles and the raw spread is 1.58% for the next month and 7.27% cumulative over months two through six, both significant at 1%. Risk-adjust it and the picture splits. $$ R\_{p,t} - R\_{f,t} = \\alpha\_p + \\beta\_1 MKT\_t + \\beta\_2 SMB\_t + \\beta\_3 HML\_t + \\beta\_4 RMW\_t + \\beta\_5 CMA\_t + \\beta\_6 UMD\_t + \\delta\\,Crisis\_t + \\varepsilon\_{p,t} $$ Read it as: regress the portfolio's monthly excess return, in percent, on the five Fama-French factors plus the momentum factor plus a dummy for 2008 and 2009, and call the intercept alpha. Alpha is the part of the return that the factor exposures do not explain, in percent per month; the paper annualizes by multiplying by twelve without compounding. Worked example. The one-month top-minus-bottom portfolio has a monthly alpha of 0.7542%, and 12 times 0.7542 gives the 9.05% printed in Table 4\. That annualization convention matters when you compare against compounded numbers, because it overstates by whatever the compounding would add. ![Grouped bar chart of annual alphas by holding period for the hedge portfolio, the top decile alone, and the bottom decile alone, showing the long leg carrying essentially all of the alpha at every horizon](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/09/article_600-mad_alpha_by_leg.png) At a one-month holding period the hedge earns 9.05%, the top decile alone earns 9.00%, and the bottom decile alone earns minus 0.05%. Out to twelve months the long leg actually grows stronger, at 8.26% with a t of 7.10, while the hedge decays to 6.67% because the bottom decile turns positive. Stambaugh, Yu and Yuan established that most anomalies live on the short leg, where arbitrage is expensive. This one inverts that, which is good news for anyone who cannot borrow shares and bad news for the story that the effect is mispricing that short sellers cannot reach. The wealth curves make the same point in a way that also traps the unwary. ![Value of one dollar invested monthly from July 1977 to December 2018 in the MAD top decile, the CRSP value-weighted market, and the MAD bottom decile, on a log scale with NBER recessions shaded](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/09/article_600-mad_wealth_curves_1977_2018.png) The bottom decile ends at 0.35 dollars, a loss of 65% over 41.42 years, which compounds to minus 2.50% a year. The top decile's 959.69 dollars compounds to 18.03% a year and the market's 92.47 dollars to 11.55%. So the short leg looks like a machine for losing money and yet its risk-adjusted alpha is zero. Both facts are true because the bottom decile is loaded with small, high-beta, high-volatility names whose losses the factor model already prices. That distinction is the whole point of the old article ["What a Factor Actually Is: α + βλ, and Why the Market Is Factor #1"](https://aligrithm.com/what-a-factor-actually-is-a-bl-and-why-the-market-is-factor-1/): a portfolio that loses money is not delivering negative alpha, it is delivering negative beta-times-lambda, and shorting it earns you the factor exposure rather than a premium. ## What it costs to actually run Turnover is the reason most anomalies die on contact with a broker. The MAD top decile turns over 34.9% a month on average, the bottom decile 38.5%, which implies an average holding time of 1 divided by 0.349, or 2.87 months. The authors compute the transaction cost that would zero the alpha rather than assuming one. $$ \\left\[R\_{l,t} - TO\_{l,t}\\times TC\\right\] - \\left\[R\_{s,t} + TO\_{s,t}\\times TC\\right\] \\ \\longrightarrow\\ \\text{regress on factors, solve } \\alpha(TC)=0 $$ Read it as: charge the long leg TC on every dollar it turns over and charge the short leg the same, both in percent per month, then find the TC that drives the regression intercept to zero. Everything in the bracket is a monthly percentage return. Worked example. The three-factor-plus-momentum alpha at a one-month holding period is 8.35% a year, or 0.6958% a month. Combined turnover is 0.349 plus 0.385, or 0.734\. Dividing 0.6958 by 0.734 gives 0.948%, or 94.8 basis points, against the 94 basis points the paper reports. The five-factor specification gives 0.7542 divided by 0.734, or 102.7 basis points, against their 101\. The published long-short numbers reproduce. The long-only row does not. The paper reports a break-even of 142 basis points for the top decile alone, but 9.00% a year is 0.75% a month, and 0.75 divided by the reported 34.9% turnover is 214.9 basis points. Charging a round trip instead gives 107.4\. Neither is 142\. The long leg's own turnover under that specification is never printed, so the figure cannot be checked. I am leaving it as published and flagging it as unverifiable rather than substituting my own number. ![Bar chart of break-even transaction costs by holding period against an institutional cost ceiling of 40 basis points and a retail one-way cost of about 208 basis points](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/09/article_600-breakeven_vs_real_costs.png) Against Novy-Marx and Velikov's estimate of 10 to 40 basis points a month for trading momentum and post-earnings drift, every horizon clears comfortably. Against retail costs it is a different verdict. The value-weighted quoted spread averages 321 basis points for the top decile and 510 for the bottom, so a one-way cost near 208 basis points kills the one-month version outright and leaves the three-month version barely alive. The twelve-month version, with a break-even of 1,102 basis points, works for anyone. If you are trading your own account, the horizon is not a tuning parameter, it is the thing that decides whether the strategy exists. The old article ["Percentile-Rank Momentum With Hysteresis: Low-Churn Signals"](https://aligrithm.com/percentile-rank-momentum-with-hysteresis-low-churn-signals/) attacks the same problem from the signal side, by refusing to rerank on small changes, and that machinery transfers directly here. ## Anchoring, or a jump detector with a good story The authors test their mechanism by splitting on suddenness. For each top-decile stock they take the largest positive monthly change in MRAT over the previous three quarters, and set SuddenUp to one if that change exceeds the decile's monthly median. SuddenDown is the mirror. The results support anchoring in sign and undercut MAD in magnitude. SuddenUp carries a coefficient of 1.04 with a t of 6.66, SuddenDown minus 0.54 with a t of minus 3.01, and both point the way the theory predicts: the wider the jump away from the anchor, the more underreaction is left to harvest. Splitting the hedge portfolio, the sudden half earns 23.97% a year with an alpha of 12.57% at a t of 3.51, while the gradual half earns 14.30% with an alpha of 5.62% at a t of 2.60. Here is what I take from that table and the authors do not spell out. MAD's own coefficient falls from 0.51 to 0.18 once the two suddenness interactions enter. More than half the effect is not "the averages are far apart" but "the averages got far apart in a hurry." That is a different signal with a different name. It is closer to a gap-and-drift detector, and it lands in the same territory as post-earnings announcement drift, though standardized unexpected earnings is among the controls and keeps its own positive slope at a t of 4.18, so the two are not the same thing. The overlap still deserves more than a footnote, and it means anyone implementing this should compute the suddenness split first and decide whether they want the whole signal or just the fast half. For context on how large 9% is for a chart-derived quantity, the old article ["Price-Path Convexity: A New Cross-Sectional Anomaly (−45bp per σ)"](https://aligrithm.com/price-path-convexity-a-new-cross-sectional-anomaly-45bp-per-s-2/) covers a signal built from the shape of the price path rather than the level of two averages, priced at 45 basis points per standard deviation. MAD is a bigger number, computed from cruder inputs, on a longer sample. That asymmetry is itself a reason to keep checking it. ## The verdict The result is real inside the sample and the authors earned it with a harder control set than most anomaly papers bother with. Three things keep me from sizing it like a discovered risk premium. The sample ends in December 2018, so the strategy has no out-of-sample record at all in the period since, which includes a pandemic crash, a zero-rate melt-up and a rate shock. The effect is untested in bear markets because 89% of sample months follow a positive two-year market return. And more than half the coefficient belongs to the suddenness interaction rather than to the distance itself. Run it long-only at a six- to twelve-month horizon, size it as a momentum variant rather than as an independent factor, and reserve judgment until someone posts the 2019 through 2025 numbers. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/09/article_600-visual-0.png) ## KEY POINTS - The distance between the 21-day and 200-day moving averages, not their crossing, is what predicts. The crossover dummy carries a t of 1.17 and dies; the ratio carries a t of 4.92 in a Fama-MacBeth regression against more than twenty controls, and the discretized version carries 6.65. - The hedge portfolio's annual alpha is 9.05% against five Fama-French factors plus momentum plus a crisis dummy, and momentum's own alpha drops from 17.58% to an insignificant 2.25% once MAD enters. One dollar compounds to 959.69 in the top decile against 92.47 for the market over 41.42 years. - All of the alpha is on the long side. Top decile 9.00%, bottom decile minus 0.05% with a t of minus 0.02\. The bottom decile loses 2.50% a year compounded, but that loss is factor exposure, not negative alpha, so shorting it pays nothing. - Break-even costs reproduce exactly from the published turnover: 0.6958% monthly alpha divided by 73.4% combined turnover gives 94.8 basis points against the 94 printed. Institutions clear that at every horizon; retail, facing 321 and 510 basis point quoted spreads, only clears it past three months. - Two numbers do not reconcile. Table 2's "multiplied by ten thousand" scaling note is off by two orders of magnitude against the portfolio spreads, and the 142 basis point long-only break-even does not follow from the published 34.9% turnover. Both are left as the source printed them. - More than half the effect is a suddenness effect. MAD's coefficient falls from 0.51 to 0.18 once SuddenUp and SuddenDown enter, and the sudden half of the portfolio earns a 12.57% alpha against 5.62% for the gradual half. - The sample stops at December 2018, the 2001 to 2018 subsample fails the t equals 3.0 bar at 2.61, and 89% of months follow a positive two-year market return. Nobody has shown this works in a bear market or after 2018. ## References - [Moving Average Distance as a Predictor of Equity Returns - Doron Avramov, Guy Kaplanski and Avanidhar Subrahmanyam (SSRN 3111334)](https://papers.ssrn.com/sol3/papers.cfm?abstract%5Fid=3111334&ref=aligrithm.com) --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ### 6.55 State-Dependent (In)Efficiency: Meta-Learning a Directional-Change Threshold URL: https://aligrithm.com/state-dependent-in-efficiency-meta-learning-a-directional-change-threshold/ Last updated: 2026-10-16T08:09:12.000Z Barak, Razmi and Mousavi report an out-of-sample Sharpe ratio of 1.34 against 0.59 for the best static version of the identical trading logic, on 50 crypto futures from January 2022 to January 2024, with a Ledoit-Wolf bootstrap p-value of 0.008 on the difference. The machine doing the work is a LightGBM classifier that picks tomorrow's directional-change threshold out of {0.01, 0.02, 0.04}. One table earlier, the same paper reports that running the same strategy on Bitcoin alone scored 0.63, beating every static portfolio version they built, and the headline table drops that comparison. The conclusion states the entire study is gross of transaction costs on a minute-level futures strategy that opens and closes a position on every threshold breach. Both facts sit in the paper. Neither survives into the abstract. The framing is the Adaptive Market Hypothesis of Lo: efficiency is not a property a market has or lacks, it is a quantity that moves as participants learn and conditions change. Urquhart found Bitcoin weak-form inefficient in 2016, Makarov and Schoar found arbitrage closing fast by 2020, and both can be right if inefficiency comes and goes. The interesting part of this paper is that it converts that debate into a parameter-selection problem with a measurable payoff, rather than another multifractal diagnostic. The part worth arguing with is the arithmetic. ## What a directional change event is Directional change throws out the calendar. Instead of sampling price every minute or every day, you declare a threshold, called theta, and record an event only when price reverses by more than theta from the last extreme. In an uptrend you track the running high; the moment price falls below that high by theta, you confirm a downward trend. $$ P\_c \\leq P\_{high} \\times (1 - \\theta) \\qquad\\text{and}\\qquad P\_c \\geq P\_{low} \\times (1 + \\theta) $$ The left condition confirms a new downtrend, the right one confirms a new uptrend. P-c is the current price, P-high is the highest price seen since the last confirmation, P-low the lowest. Worked numbers with theta of 0.02: if the running high on the portfolio index is 30,000, the downward event fires the first tick at or below 29,400\. Widen theta to 0.04 and you wait for 28,800\. Tighten it to 0.01 and 29,700 does it. The threshold is the only knob, and it controls how much noise you agree to sit through before you call the trend dead. Two points get labelled at each event. The confirmation point, DCC, is where the threshold gets breached, and it is the only point you can trade. The extreme point, EXT, is the high or low that the move ran away from, and it gets stamped in hindsight once the confirmation happens. Everything after the confirmation, until the next extreme, is the overshoot. ![Directional change anatomy on intraday price: black price line, green circles at directional change confirmation points, green squares at extreme points, solid green segments for the DC interval and dash-dot segments for the overshoot interval](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/09/article_486-dc_event_anatomy.png) Look at the gap between EXT (a) and DCC (b). The extreme sits at roughly 29,470 and the confirmation at roughly 29,170\. That 300-point gap is the toll you pay for waiting for confirmation, and it is charged on every single trade. Any backtest that enters at the extreme point rather than the confirmation point has already leaked the future. Barak and colleagues enter at confirmation, which is correct. ## The dual threshold turns a trend into a label The base strategy runs two thresholds at once, a small one and a big one. Call them theta-small and theta-big, with theta-big the larger. The question the model answers is whether a small-threshold trend has the legs to become a big-threshold trend. $$ y\_i = \\begin{cases} 1 & \\text{if } P^{s}\_{EXT,i} \\in \\{P^{b}\_{EXT}\\} \\\\\[4pt\] 0 & \\text{otherwise} \\end{cases} $$ Read it as a membership test. For the i-th small-threshold event, take its extreme point and ask whether that same price also registers as an extreme in the big-threshold series. If yes, the small move was the start of something large and the label is 1, meaning momentum. If no, the small move fizzled and the label is 0, meaning reversal. ![Dual directional-change summarisation with two thresholds overlaid: green markers and lines for the small threshold, red markers and lines for the big threshold, with EXT (a) shared by both series and EXT (b) appearing only in the green series](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/09/article_486-dual_threshold_label.png) EXT (a) near 29,470 carries a green square and a red square, so its label is 1\. EXT (b) near 29,170 carries only a green square, so its label is 0\. The base model forecasts that label from four directional-change features, of which the workhorse is total move value, the size of the trend measured in thresholds rather than in percent. $$ TMV = \\frac{P^{s}\_{EXT,i} - P^{s}\_{EXT,i-1}}{P^{s}\_{EXT,i-1} \\times \\theta^{s}} $$ Dividing by theta is what makes this comparable across regimes. A move from 28,000 to 29,680 is 6.00%, and at theta-small of 0.02 that is a TMV of 3.00: the trend ran three thresholds before dying. Run the same 6% move with theta of 0.04 and TMV is 1.50\. Same price path, half the number, because the yardstick doubled. The trading rule is then perverse on purpose: when the model predicts 0 the strategy opens a counter-trend position, betting the small move reverses. ## No single threshold wins for long The static benchmark comes from a grid search on January 2021 to January 2022, which lands on theta of 0.026. ![Scatter plot of in-sample Sharpe against the small theta threshold on a log axis, points coloured by optimisation trial, rising from minus 0.45 below theta of 0.004, crossing zero near 0.009, peaking near 0.9 around theta of 0.026 and falling to minus 1.3 by theta of 0.09](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/09/article_486-theta_optimisation_scatter.png) The surface is not noise. It is a clean hump: negative below theta of 0.004, crossing zero near 0.009, peaking near 0.9 around 0.026, collapsing to minus 1.3 by 0.09\. Two things spoil the comfort that gives you. First, at the peak itself the repeated trials scatter between roughly 0.25 and 0.95, so the optimum is a smear rather than a point. Second, that in-sample peak of about 0.9 turned into an out-of-sample Sharpe of 0.46 at a maximum drawdown of 56.31%, a haircut of roughly half. Worse for the story the grid tells, theta of 0.01 scores near zero in-sample and then posts the best static out-of-sample Sharpe in the paper at 0.59\. The grid ranked the eventual winner as worthless. ![Monthly returns for three static directional-change thresholds, 0.01 in blue, 0.02 in orange and 0.04 in green, showing the winning threshold rotating month to month with no persistent leader](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/09/article_486-monthly_returns_static_thetas.png) The rotation is real. April 2021 belongs to 0.04, August 2021 to 0.01, June 2022 to 0.04 again, October 2023 to 0.02\. Note that this figure starts in April 2021 while the paper defines its out-of-sample window as January 2022 to January 2024, so nine of the bars come from the period used to fit the static benchmark. The caption calls the whole span out-of-sample. That is sloppy labelling rather than a broken result, but it is the second caption in the paper that disagrees with its own axis, the first being Figure 7, whose legend shows theta of 0.026 while its note claims it plots theta of 0.01. ## Predicting which parameter wins tomorrow The meta-learner treats threshold choice as a three-class classification problem. The label is whichever base strategy made the most money. $$ \\theta^{s}\_{\*} = \\arg\\max\_{\\theta\_i \\in \\Theta} \\text{Return}(\\theta\_i, d) \\qquad \\hat{\\theta}^{s}\_{d+1} = f\_{meta}(\\mathbf{m}\_d) $$ The first expression takes the day's realised returns across the three base strategies and names the winner. The second maps the market-state feature vector m observed at the close of day d into a forecast of the winner for day d+1\. Worked example: on some day theta of 0.01 returns +1.8%, theta of 0.02 returns minus 0.4%, theta of 0.04 returns +0.9%. The label for that day is 0.01\. The model sees the features from the prior close and tries to have named 0.01 in advance. The index on the first expression is wrong as printed: it labels day d using day d's own returns, while the text and the stated training pairs both use day d+1\. Take it as a typo, because the alternative reading is a model trained on today's answer sheet. Feature vector m carries four blocks: recent performance of each base strategy, 43 on-chain metrics for Bitcoin and Ethereum culled from around 300 by importance scoring inside the walk-forward loop, directional-change trend statistics, and plain market statistics like RSI, skewness and a price Z-score. Base models roll on a fixed window, the meta-model expands from a fixed origin so it keeps long memory of regimes it saw years earlier. Run each block alone and the ranking is not what the abstract advertises. Market regime statistics alone score 1.19 and strategy performance history alone scores 1.17, while the two blocks the paper markets as its novel contribution, on-chain data and directional-change indicators, score 0.93 and 0.92\. The novel information sets are the weakest two of four. They do add something, since the combined model at 1.34 beats every single block, and the appendix shows the four models' predicted probabilities are close to uncorrelated. Diversification across feature families, not blockchain fundamentals, is what lifts the number. ![Horizontal bar chart of every out-of-sample Sharpe ratio the paper reports for January 2022 to January 2024, ranging from 0.07 for equal weight up to 1.34 for the adaptive LightGBM model, with the 0.63 bar for the same strategy run on Bitcoin alone highlighted because the headline table omits it](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/09/article_486-sharpe_ladder.png) Every bar above comes from the same out-of-sample window. The orange one is the comparison the headline table leaves out. ## Weighting the loss by how much the day matters This is the one piece worth stealing. A classifier minimising log loss treats every misclassification the same, which is wrong when the cost of picking the wrong threshold varies by two orders of magnitude across days. The fix weights each training day by the spread between the best and worst base strategy that day. $$ w\_d = \\exp\\!\\big(\\alpha \\cdot (r\_{\\max,d} - r\_{\\min,d})\\big), \\qquad \\mathcal{L} = \\sum\_{i=1}^{n} w\_i \\cdot \\ell(y\_i, \\hat{y}\_i) $$ The spread r-max minus r-min is non-negative by construction, so the weight is at least 1 and grows without bound as the day gets more decisive. Alpha is a scaling constant, set to 5\. Worked example: on a loud day the best threshold makes +8% and the worst loses 4%, so the spread is 0.12 and the weight is exp(5 times 0.12) = 1.822\. On a quiet day the spread is 0.01 and the weight is exp(0.05) = 1.051\. The loud day counts 1.73 times as much. Their weight histogram tops out at 2.2, which back-solves to a spread of ln(2.2)/5 = 0.158, consistent with the return clipping they describe. Then check whether it worked. Appendix H runs every model with and without the weights. Strategy-history, on-chain and regime models all improve. Directional-change indicators drop from Sharpe 1.10 to 0.92\. The combined model, the one carrying the 1.34 headline, drops from 1.37 to 1.34 and from 496.57% to 445.98% total return. Three of five improved, not four as the text claims, and the flagship model got worse. The honest version of this contribution is that the weighted loss trades a little return for a shallower drawdown, 29.91% against 37.54%, on the model they shipped. ## What the model learned Temporal feature importance shifts with volatility. Through the high-volatility stretch of early 2022 the model leans on directional-change indicators, overshoot magnitudes and trend asymmetries. As volatility falls through late 2022 and 2023 it shifts toward on-chain metrics like exchange flows and holder behaviour, and toward recent strategy performance. In a panic the information sits in the price path, and in calm the price path stops telling you much. The SHAP attribution is narrower than the summary suggests. It says the model picks the small threshold when the price Z-score is high and Ethereum exchange netflows are positive, which reads as an overbought market with coins moving onto exchanges ahead of selling. It picks the large threshold when average position lengths are long and netflows are negative, which reads as accumulation. That is a statement about mean reversion versus trend, not about volatility. A shorthand that says high volatility implies a tighter threshold is not what the paper reports, and it would be strange if it were, because a tighter threshold in a noisier tape fires more false events, not fewer. ## Do the reported numbers close? Sharpe and compounded return are not free to move independently. If a strategy has daily mean m and daily standard deviation s, its compounded growth per day is roughly m minus s-squared over two, and the daily Sharpe k pins m to k times s. $$ g(s) = k\\,s - \\tfrac{1}{2}s^{2}, \\qquad g\_{\\max} = \\tfrac{1}{2}k^{2} \\quad\\text{at}\\quad s = k, \\qquad k = \\frac{SR\_{ann}}{\\sqrt{A}} $$ Growth g is a downward parabola in volatility: too little volatility and you cannot compound, too much and variance drag eats you. The peak sits at s equal to k, so the most a strategy with a given Sharpe can compound to over 730 days is exp(730 times g-max) minus 1, whatever volatility it runs. A is the annualisation factor, 365 for a market that never closes. ![Two panels plotting two-year compounded return against daily volatility for a fixed annualised Sharpe: left panel Sharpe 0.59 peaks at 42 percent while the paper reports 63 percent, right panel Sharpe 1.34 peaks at 502 percent with the reported 446 percent intersecting the curve at 5.4 percent daily volatility](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/09/article_486-sharpe_return_feasibility.png) Run the static theta of 0.01 row through it. Sharpe 0.59 over 365 days gives k = 0.59/19.105 = 0.030882, so g-max = 0.000477 per day, and over 730 days that compounds to 41.6%. The paper reports 63.19%. No volatility makes that pair work. The same failure hits theta of 0.04 (ceiling 32.4% against a reported 45.23%) and the theta of 0.026 benchmark (ceiling 23.6% against 31.25%). Reversing the calculation, a 63.19% two-year return needs an annualised Sharpe of at least 0.70 on a 365-day clock, and re-annualising 0.59 from 252 days to 365 gives 0.59 times the square root of 365/252, which is 0.71\. That clears the floor with nothing to spare, so the likely explanation is a 252-day convention applied to a market that never closes rather than invented returns. Vidal-Tomás made the same point about crypto data hygiene. Keep the source form and read every Sharpe in the paper as about 20% low against its own return column. The adaptive row is feasible even on 365 days, and the price of feasibility is the interesting part. A Sharpe of 1.34 compounding to 445.98% requires daily volatility of 5.37%, which annualises to 103%. Simulate 20,000 two-year paths at that mean and volatility and the median maximum drawdown is 67%, with only 1 in 10,000 coming in shallower than the reported 29.91%. Assume instead that the Sharpe came off a 252-day clock and the implied daily volatility falls to 3.47%, which lifts the odds to 3.5%. Still the lucky tail. The reported combination annualises to a 133.7% return against a 29.91% drawdown, a Calmar ratio of 4.47\. Event-based strategies do sit flat for stretches, which makes daily returns lumpy rather than normal, so treat this as a smell rather than a proof. It is a strong smell. ## What a fill costs The paper says outright that everything is gross of fees, funding and slippage, and leaves net returns to future work. Price that gap. Profit factor on the adaptive model is 1.27\. Suppose the average winner gains 1.50% and the average loser drops 1.20%; a win rate of 50.4% produces exactly 1.27\. Now charge every round trip. At 4 basis points the profit factor falls to 1.196\. At 8 basis points, roughly Binance futures taker fees on both sides for a large account, it falls to 1.127\. At 16 basis points it hits 1.001 and the edge is gone. That threshold is not far away once you remember what the strategy trades: an aggregated index of every selected coin, so each event fires a simultaneous rebalance across all legs, several of them illiquid alts where the spread alone clears 10 basis points. Add perpetual funding, which a two-sided strategy holding through funding stamps pays on every stamp. The base strategy at theta of 0.01 fires on every 1% move in a market whose daily range clears 4% often enough, which is a churn problem before it is a cost problem. The old article ["Percentile-Rank Momentum With Hysteresis: Low-Churn Signals"](https://aligrithm.com/percentile-rank-momentum-with-hysteresis-low-churn-signals/) handles the same sensitivity with a dead band instead of a classifier, and a dead band costs nothing to fit. One more crack in the sample: the universe is the top 50 coins by market capitalisation as reported by CoinMarketCap in 2024, applied back to January 2021\. LUNA was a top-ten coin in 2021 and worthless by May 2022\. FTT was top-thirty until November 2022\. Neither appears. That omission matters more to this strategy than to most, because the base rule is counter-trend: it opens against the move when the model predicts the small trend will fizzle. Run that rule into LUNA in May 2022 and it buys every dip of a coin going to zero. The same hindsight also lifts the equal-weight and hierarchical risk parity benchmarks, so their Sharpes of 0.07 and 0.08 are already the flattered versions. ## What survives Strip the headline and a defensible result remains. Threshold optimisation on directional-change strategies does not hold up: the in-sample optimum of 0.026 delivered 0.46 out of sample, worse than two of the three round-number thresholds nobody tuned. Switching thresholds on observable state beats committing to one, by a margin the Ledoit-Wolf bootstrap puts at p = 0.008, and that test is conservative because the benchmark is the best of three chosen after seeing the results. The XGBoost replication at 1.28 says the gain belongs to the framework rather than to LightGBM. Doubling a Sharpe from 0.59 to 1.34 is the claim the evidence supports, and it is a real claim. The 445.98% is not. That number is 7.06 times the static strategy's total return while the Sharpe is only 2.27 times higher, and the whole gap is compounding at 100% annualised volatility over two years. The old article ["Does ML Actually Help Asset Pricing? Kelly's 20%, Not 2–3×"](https://aligrithm.com/does-ml-actually-help-asset-pricing-kellys-20-not-2-3x/) makes the same correction in equities: machine learning buys you a real but bounded improvement, and the eye-catching multiples come from the reporting convention rather than the model. Report the Sharpe, net of costs, against the best benchmark you have, which here is the same strategy on Bitcoin alone at 0.63 rather than the portfolio at 0.59\. On that comparison the adaptive model is a 2.13 times improvement, gross, on a survivor-selected universe, over two years that contained one bear market and one bull market and nothing else. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/09/article_486-visual-0.png) ## KEY POINTS - A LightGBM meta-learner that reselects the directional-change threshold each day from {0.01, 0.02, 0.04} posts an out-of-sample Sharpe of 1.34 against 0.59 for the best static threshold, Ledoit-Wolf bootstrap p = 0.008, on 50 crypto futures from January 2022 to January 2024. - The same strategy run on Bitcoin alone scored 0.63, better than every static portfolio version, and the headline table omits it. Measure the adaptive gain against 0.63, not 0.59, and it is 2.13 times rather than 2.27 times. - Threshold optimisation fails on its own terms. The in-sample grid peaks near Sharpe 0.9 at theta of 0.026, that setting delivered 0.46 out of sample, and theta of 0.01, which the grid scored near zero, went on to post the best static out-of-sample Sharpe at 0.59. - The two feature blocks sold as novel are the two weakest. On-chain metrics score 0.93 and directional-change indicators 0.92, against 1.19 for plain market statistics and 1.17 for strategy performance history. The lift comes from combining near-uncorrelated blocks. - The weighted loss, w = exp(5 times the best-minus-worst daily return spread), is the transferable idea, but Appendix H shows it improved three of five models, not four, and made the flagship model worse on Sharpe (1.37 to 1.34) while cutting drawdown from 37.54% to 29.91%. - The static rows do not reconcile. A Sharpe of 0.59 annualised on 365 days caps two-year compounded return at 41.6%, and the paper reports 63.19%. A 252-day convention makes the rows feasible, so read the reported Sharpes as too low against their own returns rather than the returns as invented. - The adaptive row implies 103% annualised volatility on a 365-day convention, under which a 29.91% maximum drawdown shows up in 1 of 10,000 simulated paths. Calmar of 4.47 on a gross-of-cost backtest is a number to distrust. - Everything is gross of fees. At profit factor 1.27, a 16 basis point round trip takes the edge to zero, and the universe is the 2024 top-50 applied back to 2021, so LUNA and FTT never appear. ## References - [State-Dependent Market (In)Efficiency in Cryptocurrency Markets, Sasan Barak, Siavash Razmi and Alireza Mousavi (SSRN 5525163)](https://papers.ssrn.com/sol3/papers.cfm?abstract%5Fid=5525163&ref=aligrithm.com) --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ### 4.82 Good vs Bad COVOL in Crypto: A Common-Volatility Tilt URL: https://aligrithm.com/good-vs-bad-covol-in-crypto-a-common-volatility-tilt/ Last updated: 2026-10-16T08:09:12.000Z Pham, Han, Nguyen, Pham and Do build one index and then trade it backwards. In Section 5.4 they write that an RCI near zero marks "widespread panic selling and a potentially buying opportunity," and that exuberance at the top "can indicate a market peak and a potential selling opportunity." Nine pages later, in Section 6, they lever the portfolio up to 125% when the same index rises above 70 and cut it to 75% when it falls below 30\. Contrarian in the diagnosis, momentum in the backtest, same number on both sides. The momentum version is the one that produced the headline, so the headline is the version that was not the paper's own stated reading of its own index. That is worth sitting with before we get to the numbers, because the index itself is a genuinely good idea. Engle and Campos-Martins showed that if you fit a GARCH model to each asset separately, the leftover squared residuals still move together across assets. They named the latent factor driving that co-movement COVOL, and it measures something no news index captures: the days on which everything surprised at once. This paper applies it to 25 cryptocurrencies from January 2015 to April 2024, splits it by the sign of returns, and turns the split into a tradable tilt. The construction is clean. The evaluation is not. ## One latent factor that rescales everybody's variance Start with what COVOL actually is, because the word "volatility" is doing two jobs here. Each coin gets an AR(1) return equation with a first-principal-component factor and GARCH(1,1) errors, which strips out that coin's own predictable variance. What remains is the standardized residual, which by construction has unit variance. COVOL is a common multiplier on the variance of that residual. $$ e\_{i,t} = \\sqrt{g(s\_i\\, x\_t^{\\sigma})}\\; \\epsilon\_{i,t}, \\qquad g(s\_i\\, x\_t^{\\sigma}) = s\_i\\,(x\_t^{\\sigma} - 1) + 1 $$ Read the second piece first. The latent factor x has mean one by construction, so on an average day the bracket is zero and g equals one, leaving the residual at its nominal unit variance. On a common-shock day x jumps above one, and each coin's variance gets multiplied by g, where the loading s decides how much of the shock that coin absorbs. Coins with a high s are the ones that blow up when the whole sector blows up. Worked example with the paper's own numbers. Bitcoin's bad-COVOL loading is 0.2430, and the largest bad COVOL in the sample is 98.4933 on 3 April 2017\. So g equals 0.2430 times 97.4933 plus 1, which is 24.69\. Variance multiplied by 24.69 means volatility multiplied by the square root, 4.97\. Bitcoin's return distribution that day was five times as wide as on a normal day, and that is on top of whatever its own GARCH process already knew. Run the same arithmetic on 12 March 2020, where bad COVOL was 43.5893: g is 11.35, so the volatility multiplier is 3.37\. Toncoin, with a loading of 0.1123, gets 11.95 and 3.46 on the April 2017 date, so a coin with less than half Bitcoin's loading still absorbs a third more than Bitcoin did in March 2020\. The loading compresses the extremes but does not remove them. The test for whether a common factor exists at all is an equicorrelation test on the squared residuals, and it is worth checking because it is the gate everything else passes through. $$ T\_{e^2} \\;=\\; \\hat{\\rho}\_{e^2}\\,\\sqrt{\\frac{T\\,N\\,(N-1)}{2}} $$ The statistic is the average pairwise correlation of the squared standardized residuals, scaled by the square root of the number of pair-days in the panel, and it is standard normal under the null of no common volatility. The paper prints the statistic in an expanded form; collapsing it gives this, and the collapse is where the check lives. They report a correlation of 0.043 and a statistic of 20.67 with 25 coins. Invert it: the implied number of full-panel-equivalent days is 20.67 squared divided by 0.043 squared divided by 300, which is 770\. Their goodness-of-fit pass, after the common factor is removed, reports a correlation of 0.0013 and a statistic of 0.635, and the same inversion gives 795\. Those two agree, which confirms both statistics come from the same scale and that the printed equation is the right one. The p-value they quote for 0.635, namely 0.2627, is exactly one minus the standard normal CDF at 0.635, so that reconciles too. The gap worth naming: the panel runs 3,361 days for Bitcoin, and 770 full-panel-equivalent days is what you get when most of the 25 coins only exist for the back half of the sample. Solana contributes 1,367 days, Toncoin 490, Mantle 267\. The common factor in the early years is estimated off a handful of coins, and the RCI's historical extremes all sit in exactly those early years. ## Splitting the factor by truncating returns, which breaks the error assumption The good and bad decomposition is mechanically simple. Take each coin's return series, keep the positive days and set the rest to zero to get one series, then keep the negative days and set the rest to zero to get the other. $$ \\tilde{r}\_{i,t}^{+} = \\begin{cases} \\tilde{r}\_{i,t} & \\tilde{r}\_{i,t} > 0\\\\\[2pt\] 0 & \\text{otherwise} \\end{cases} \\qquad \\tilde{r}\_{i,t}^{-} = \\begin{cases} \\tilde{r}\_{i,t} & \\tilde{r}\_{i,t} \\leq 0\\\\\[2pt\] 0 & \\text{otherwise} \\end{cases} $$ Then run the whole AR(1)-GARCH-plus-COVOL machine on each half separately. Good COVOL is the common variance factor of the positive-only series, bad COVOL the same for the negative-only series. The problem is what that truncation does to the series being modelled. Roughly half the observations are now exactly zero. Take a coin with 3,361 days and a return that is positive on about 52% of them: the positive-only series has about 1,600 real numbers and 1,760 exact zeros. A GARCH(1,1) fit to that produces standardized residuals with a point mass at a single value, which cannot be the independent standard normal that the COVOL specification assumes for epsilon. The estimator does not know it is being fed a censored series. Patton and Sheppard, whom the paper cites as precedent, avoided this by building signed realized semivariances out of intraday returns, so every day carries both an up and a down measure and nothing gets zeroed. Splitting daily closes does not have that escape. The good and bad COVOL estimates are therefore point estimates under a violated distributional assumption, and the paper never reports the fit diagnostics for the split models the way it does for the pooled one. Separately, the transcription is loose in places that matter for a reader trying to reimplement. The GARCH variance equation carries time subscripts on its parameters, which would make them non-identified; beta serves as both the factor loading in the mean equation and the persistence term in the variance equation; Section 5.3.3 points at "Equation (10)" when it means Equation (13); Figure 3's discussion cites "Table 10" when it means Table 5; Table 1's note dates the sample from January 2006 when the data section says January 2015; and Bitcoin's minimum excess return prints as 21.59 without the minus sign. None of that changes a result. All of it means you should check the arithmetic yourself rather than lift numbers. ![Twenty-day moving averages of good COVOL (blue) and bad COVOL (red) for the crypto market, 2015 to 2024, both spiking to between 5 and 8 in 2015 to 2017 and settling under 3 after 2018](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/09/article_477-good_vs_bad_covol.png) Two features of that picture carry the paper. Bad COVOL spikes harder, topping 8 in early 2017 against good COVOL's 5\. Good COVOL happens more often: the authors count 1,754 days where good exceeded bad against 1,588 the other way, so 52.5% of days lean bullish. Frequent mild optimism against rare violent fear is the same asymmetry equity researchers have documented for decades, and finding it again in crypto is a reasonable contribution on its own. The amplitude also collapses after 2018 in both series, which tells you the whole sector's common-shock intensity fell by more than half once the panel filled out with real coins. Any threshold calibrated on the 2015 to 2017 amplitude describes a market that stopped existing. ## The RCI, and why you could not have computed it at the time The index is the difference between the two factors, smoothed and rescaled. $$ d\_t = \\hat{\\chi}\_t^{\\sigma,+} - \\hat{\\chi}\_t^{\\sigma,-}, \\qquad dma\_{20,t} = \\frac{1}{20}\\sum\_{k=0}^{19} d\_{t-k}, \\qquad RCI\_t = 100 \\times \\frac{dma\_{20,t} - \\min(dma\_{20})}{\\max(dma\_{20}) - \\min(dma\_{20})} $$ Subtract bad from good, average the last 20 days of that difference, then map the result onto 0 to 100 by its distance between the minimum and the maximum. Worked example: suppose the 20-day average difference today is +1.2, the sample minimum is -4.0 and the sample maximum is +3.0\. Then RCI equals 100 times (1.2 plus 4.0) divided by 7.0, which is 74.3, and the strategy levers up. Now suppose a later crash drops the minimum to -6.0\. The identical +1.2 reading becomes 100 times 7.2 divided by 9.0, which is 80, and if the crash instead pushed the maximum to +5.0 the same reading becomes 100 times 5.2 divided by 11.0, which is 47.3, and the strategy sits flat rather than levering. One reading, three answers, decided by data that had not happened yet. That is the defect. The min and max in Equation (12) are taken over the whole sample, so the 30 and 70 thresholds are not observable in real time. A trader standing in June 2018 does not know what the 2021 extremes will be. ![Two panels of a simulated RCI: the top uses full-sample min and max as published, the bottom uses only data available up to each day, with the bullish 70 line crossed far more often in the tradable version](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/09/article_477-rci_lookahead.png) I simulated two spiky mean-one factors, differenced them, and computed the index both ways to see how much the choice matters. With full-sample scaling the index sits above 70 on 10.2% of days after burn-in; with expanding-window scaling, the only version you could trade, it sits above 70 on 15.9%. The three-state signal disagrees on 6% of days. A strategy that takes leverage 56% more often than the published version is a different strategy. Now the fair part, because the size of this particular leak is smaller than the principle suggests. The extremes that set the denominator for this sample land early: good COVOL peaks in September 2016 at 58.9349 and bad COVOL peaks in April 2017 at 98.4933\. The portfolio simulation starts in March 2017\. So the full-sample range is essentially pinned within two months of the simulation's first trade, and only about one month of the 86 is contaminated. The leak is structural rather than fatal here. Had the extremes arrived in 2021 instead, every signal before them would have been fiction, and nothing in the method tells you in advance which sample you are in. Rebuild it with an expanding or rolling range and report both. The old article ["Percentile-Rank Momentum With Hysteresis: Low-Churn Signals"](https://aligrithm.com/percentile-rank-momentum-with-hysteresis-low-churn-signals/) makes the same point from the other direction: rank-based signals need their reference window fixed to information available at decision time, and they need hysteresis so a threshold crossing does not flip the book on noise. ![RCI in grey against the logarithm of the S&P Cryptocurrency Broad Digital Market index in black, 2017 to 2024, with dashed lines at 30, 50 and 70](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/09/article_477-rci_vs_bdm_index.png) Look at that chart with the direction question in mind. The RCI is a difference of two variance factors, and neither factor knows the sign of the market's return. Good COVOL is high when positive returns were commonly large in magnitude, which happens in melt-ups and also in violent bear-market rallies. The index reads direction only because in crypto, large synchronized up-moves cluster inside bull markets. That is an empirical regularity, not an identity, and it is the reason the paper could argue itself into both a contrarian and a momentum reading without noticing. The old article ["Regime-Switching That Works, Factors That Don't (MS-GARCH)"](https://aligrithm.com/regime-switching-that-works-factors-that-dont-ms-garch/) found the same asymmetry: a second-moment model can label the regime you are in with real skill while telling you close to nothing about the sign of the next return. ## The tilt is smaller than the result it is credited with Here is what the strategy actually does. Above 70, hold 125% of the crypto index. Below 30, hold 75%. In between, hold 100%. Borrow and lend at the one-year Treasury yield. The thresholds come from RSI convention, which the authors state openly, so at least they are not fitted. Everything the tilt earns over buy-and-hold comes from the covariance between the weight and the return. $$ R\_t^{\\text{RCI}} - R\_t^{\\text{B\\&H}} = (w\_t - 1)\\,(r\_t - r\_f), \\qquad \\mathbb{E}\\!\\left\[R^{\\text{RCI}} - R^{\\text{B\\&H}}\\right\] = 0.25\\left(p\_{+}\\,m\_{+} - p\_{-}\\,m\_{-}\\right) $$ The weight deviates from one by 0.25 in either direction, so the monthly edge is 0.25 times the frequency of bullish months times their mean excess return, minus 0.25 times the frequency of bearish months times theirs. Worked example with the paper's means: RCI Momentum 1 averages 2.48% a month against 1.91% for buy-and-hold, a gap of 0.57%. Divide by 0.25 and the signal must deliver 2.28 percentage points a month of frequency-weighted spread. If the bullish and bearish states each occupy 20% of months, the conditional means must differ by 11.4 points a month. At 30% each, 7.6 points. Crypto monthly returns are wide enough that this is possible, and that is the entire bet: a ±25% exposure wobble pays only if the index separates monthly returns by high single digits at minimum. Except the paper's own outperformance columns disagree about how large that spread was. The terminal wealth ratio, 141.98 against 45.58, is 3.115 over 86 months, which compounds to 1.33% a month of relative gain and by the same arithmetic demands 5.32 points a month of spread. The arithmetic means demand 2.28\. The two are a factor of 2.3 apart. ## Table 6 does not describe one return series That inconsistency is worth running down, because the headline lives in this table. $$ g \\;\\approx\\; m - \\tfrac{1}{2}\\sigma^2, \\qquad SR = \\frac{(1+g)^{12} - 1 - r\_f}{\\sigma\\sqrt{12}} $$ The geometric monthly return is the arithmetic mean less half the variance, and the annualized Sharpe compounds that geometric mean, subtracts the risk-free rate and divides by the annualized volatility. Run it on buy-and-hold: m is 1.91%, sigma is 10.89%, so g is 1.317%, compounding to 17.00% a year against an annualized volatility of 37.72%, and with a 1.8% Treasury yield the Sharpe is 0.403\. Reported, 0.3971\. Run it on RCI Momentum 1: g is 1.832%, annualizing to 24.35% against 39.42%, giving 0.572 against a reported 0.5648\. The mean, standard deviation and Sharpe columns hang together. The accumulated return column does not hang with them, on either reading. Read 45.58 as a percentage and buy-and-hold compounded at 0.438% a month for 86 months, which against an arithmetic mean of 1.91% implies a volatility drag of 1.47 points a month and therefore a monthly standard deviation of 17.2%, not the 10.89% printed beside it. Read 45.58 as a terminal wealth multiple and the geometric mean becomes 4.54% a month, which exceeds the arithmetic mean of 1.91%, and a geometric mean can never exceed an arithmetic one. Neither reading survives. ![Accumulated returns of the RCI Momentum portfolio in blue against the buy-and-hold benchmark in red, March 2017 to March 2024, the benchmark rising from about 1 to a peak near 60 and ending near 46](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/09/article_477-accumulated_returns_rci_vs_benchmark.png) The figures settle which column describes reality. Figure 5 puts the logarithm of the BDM index at about 1.95 in February 2017 and about 3.65 in early 2024\. Base-ten logs on an index based near 100 give a rise from roughly 89 to roughly 4,470, a factor near 50, which matches the market: Bitcoin went from about 1,180 dollars to about 63,000 dollars over that window, a factor of 53\. So 45.58 is a wealth multiple and buy-and-hold made about 4,458%, not 45.58%. And then the printed volatility of 10.89% a month, or 37.7% a year, belongs to no crypto index over 2017 to 2024; realized Bitcoin volatility over that period runs closer to 60% or 70% a year. Keep the source form, because guessing at a correction is worse than reporting the conflict. Stated plainly: the risk-adjusted numbers everyone will quote, 0.5648 against 0.3971, were computed from a mean and a standard deviation that cannot have come from the series plotted in Figure 6\. The relative result, 3.1 times the terminal wealth from a ±25% tilt, is the claim that matches the chart. Both cannot be right, and the paper does not reconcile them. This is also gross of costs. The tilt trades a broad index at monthly frequency, so turnover is low and the cost drag is small, but the leveraged leg borrows at the Treasury yield rather than a crypto broker's rate, and in 2021 those differed by more than a thousand basis points. ## The macro drivers explain one percent The last section regresses daily COVOL on a risk-aversion index, the S&P 500 return, the dollar index, the ten-year-minus-two-year term spread and the two-year yield. The signs are sensible: the dollar comes in negative, the term spread and the short yield positive. The adjusted R-squared is 0.0105. One percent of the variation in COVOL, across 2,219 days. The paper calls these "crucial macroeconomic and sector-specific drivers." Significance at that sample size costs almost nothing, and two of the stars do not survive their own standard errors: the dollar coefficient of -0.05 with a standard error of 0.03 is a t of -1.67, which is the 10% level and not the 5% level it is marked at, and the monthly crypto-price-uncertainty coefficient of 0.017 with a standard error of 0.09 is a t of 0.19, marked significant at 10%. That last one is almost certainly a typo for 0.009, which gives a t of 1.89 and does clear 10%, but the table as printed does not support its own asterisk. The monthly specification reaches an adjusted R-squared of 0.1657 on 102 observations, which is more respectable and also 102 observations. The validation tests are the strongest empirical section and get the least attention. Squared COVOL predicts the broad index's realized volatility with a coefficient of 0.42 and survives at 0.38 in the full model with global policy uncertainty, VIX changes and geopolitical risk alongside it, where policy uncertainty loses its significance. Squared COVOL also raises the average pairwise correlation among the 25 coins with a coefficient of 0.02: a move in monthly squared COVOL from 1 to 10 lifts average pairwise correlation by 0.18, which is the difference between a diversified crypto book and one position. That is the finding I would build on. It says nothing about direction and everything about when your diversification stops working. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/09/article_477-visual-0.png) ## KEY POINTS - COVOL is one latent factor that multiplies every coin's post-GARCH residual variance, scaled per coin by a loading. Bitcoin's bad-COVOL loading of 0.2430 against the sample's worst bad COVOL of 98.4933 gives a variance multiplier of 24.69, so a volatility multiplier of 4.97\. On 12 March 2020 the same arithmetic gives 3.37. - The equicorrelation test reconciles twice. A correlation of 0.043 with a statistic of 20.67 and a post-fit correlation of 0.0013 with a statistic of 0.635 both imply about 780 full-panel-equivalent days out of a 3,361-day panel, which is what an unbalanced panel of late-listing coins produces. The quoted p-value of 0.2627 matches the normal CDF exactly. - The good and bad split truncates daily returns to zero on half the days and then fits GARCH to the result. Standardized residuals from a series with a point mass at zero cannot be the independent standard normal the COVOL specification assumes. Patton and Sheppard dodged this with intraday semivariances; splitting daily closes has no such escape. - Equation (12) rescales the index by the whole sample's minimum and maximum, so the 30 and 70 thresholds were unknowable in real time. In simulation the tradable expanding-window version sits above 70 on 15.9% of days against the published 10.2%. For this particular sample the extremes land in September 2016 and April 2017, just before a March 2017 start, so the leak is structural rather than decisive. Rebuild with a rolling range and report both. - The RCI is a difference of two variance factors, so it carries no information about the sign of returns except through crypto's habit of clustering large synchronized up-moves inside bull markets. The paper reads it contrarian in Section 5.4 and momentum in Section 6, and only the momentum reading produced the headline. - A ±25% exposure tilt earns the covariance between weight and return, nothing else. The 0.57% monthly gap over buy-and-hold requires 2.28 points a month of frequency-weighted conditional spread, or 11.4 points if the bullish and bearish states each occupy a fifth of months. The terminal wealth ratio of 3.115 demands 5.32 points instead, so the paper's two outperformance measures disagree by a factor of 2.3. - Table 6's mean, standard deviation and Sharpe columns reconcile with each other: 1.91% and 10.89% monthly compound to 17.0% a year against 37.7% volatility and a Sharpe of 0.403 against the reported 0.3971\. The accumulated return column reconciles with neither. As a percentage it implies 17.2% monthly volatility; as a wealth multiple it implies a geometric mean above the arithmetic mean, which is impossible. Figure 5's log scale and Bitcoin's 53-fold rise say it is a wealth multiple, which leaves the printed 37.7% annualized volatility describing no crypto index that existed. Source numbers kept as printed; the source appears wrong. - Macro drivers explain 1.05% of daily COVOL variation across 2,219 days, with the dollar index starred at 5% on a t of -1.67 and crypto price uncertainty starred at 10% on a t of 0.19\. Build instead on the validation result: squared COVOL moving from 1 to 10 lifts the average pairwise correlation across the 25 coins by 0.18, which is diversification failing on schedule. ## References - [What are the events that shake our world? Measuring and hedging global COVOL - Engle & Campos-Martins (2023)](https://doi.org/10.1016/j.jfineco.2022.09.009?ref=aligrithm.com) - [Good Volatility, Bad Volatility: Signed Jumps and the Persistence of Volatility - Patton & Sheppard (2015)](https://doi.org/10.1162/REST%5Fa%5F00503?ref=aligrithm.com) - [Good Volatility, Bad Volatility, and the Cross Section of Stock Returns - Bollerslev, Li & Zhao (2020)](https://doi.org/10.1017/S0022109019000097?ref=aligrithm.com) - [Asymmetric connectedness on the U.S. stock market: Bad and good volatility spillovers - Baruník, Kočenda & Vácha (2016)](https://doi.org/10.1016/j.finmar.2015.09.003?ref=aligrithm.com) - [The cryptocurrency uncertainty index - Lucey, Vigne, Yarovaya & Wang (2022)](https://doi.org/10.1016/j.frl.2021.102147?ref=aligrithm.com) - [The Time Variation in Risk Appetite and Uncertainty - Bekaert, Engstrom & Xu (2022)](https://doi.org/10.1287/mnsc.2021.4068?ref=aligrithm.com) - [Measuring Geopolitical Risk - Caldara & Iacoviello (2022)](https://doi.org/10.1257/aer.20191823?ref=aligrithm.com) - [Good versus Bad COVOL in Cryptocurrency Markets: A Measure of Asymmetric Common Volatility - Pham, Han, Nguyen, Pham & Do (ssrn-6038623)](https://papers.ssrn.com/sol3/papers.cfm?abstract%5Fid=6038623&ref=aligrithm.com) --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ### 9.46 Trend Quality Near Settlement: A Kalshi State Variable, Not Alpha URL: https://aligrithm.com/trend-quality-near-settlement-a-kalshi-state-variable-not-alpha/ Last updated: 2026-10-16T08:09:13.000Z A 70.8% continuation rate looks like a trade. Greene sorts 4,061 Kalshi contracts by the quality of their price trend over the window from 30 to 12 minutes before close, and the top decile keeps moving in the trend's direction 70.8% of the time against 51.2% in the bottom decile. Ex-post forecast error falls from 14.75 cents to 6.07 cents across the same sort. Both gaps carry contract-level bootstrap p-values under 0.001\. Then the same paper reports that the out-of-sample R-squared of the trend-quality signal is negative on every continuous target it tries, and that recomputing the whole exercise from quote midprices instead of last trades cuts the continuation gap from 19.6 points to 1.9 points with a p-value of 0.62\. Both facts are in the paper. Only one of them made the abstract. The statistic is called the Trend Quality Ratio, and the honest reading is that it describes a Kalshi market's state near settlement rather than predicting its next move. High trend quality marks a contract whose price has already worked out the answer. That is worth knowing. It is the opposite of a signal to trade, because you profit when a price is wrong, not when it is right. This sits next to the old article ["Do Polymarket Prices Converge? Bias and Volatility Evidence,"](https://aligrithm.com/do-polymarket-prices-converge-bias-and-volatility-evidence/) which measured the same convergence from the other side, and next to the old article ["Percentile-Rank Momentum With Hysteresis: Low-Churn Signals,"](https://aligrithm.com/percentile-rank-momentum-with-hysteresis-low-churn-signals/) which treats a trend score as a state gate rather than an entry trigger. The data deserves a note before any of the numbers. The panel holds 4,858,323 near-expiry snapshots across 29,590 tickers from 25 January to 12 March 2026, which sounds enormous. Greene discloses that a single BTC price-range series, KXBTC, contributes about 89% of those raw rows while making up 12% of tickers and 19% of the baseline contract windows. The analysis runs at contract level, so the skew does not weight the results, but "4.86 million snapshots" describes the disk, not the evidence. The evidence is 4,061 contracts with a non-zero trend, of which 1,094 resolved inside the sample. ## The statistic, and the normalization that cancels itself Fit a straight line to a contract's price over the pre-close window, with elapsed time running from 0 at 30 minutes remaining to 18 at 12 minutes remaining, and require at least 5 observations. Take the slope, divide by mean price to get a directionality index. Take the root mean squared residual, divide by mean price to get a normalized RMSE. Divide the first by the second. $$ \\text{DI} = \\frac{\\beta\_1}{\\bar{y}}, \\qquad \\text{nRMSE} = \\frac{\\sqrt{\\frac{1}{n}\\sum\_{i=1}^{n} (y\_i - \\hat{y}\_i)^2}}{\\bar{y}}, \\qquad \\text{TQR} = \\frac{\\text{DI}}{\\text{nRMSE}} = \\frac{\\beta\_1}{\\text{RMSE}} $$ Read it as drift per unit of noise. Beta-one is how many cents the fitted line climbs per minute; RMSE is how far the actual prints scatter around that line in cents. A contract that walks 9 cents higher in a tidy line scores high. A contract that ends 9 cents higher after thrashing around scores low. Note the last equality, because the paper's own equations 2, 3 and 4 imply it and the paper never writes it down. Mean price appears in the numerator of the directionality index and in the numerator of the normalized RMSE, so it cancels exactly. Worked example: a contract averaging 40 cents, slope 0.50 cents per minute, residual RMSE 1.20 cents. The directionality index is 0.50 divided by 40, which is 0.0125 per minute. The normalized RMSE is 1.20 divided by 40, which is 0.030\. The ratio is 0.0125 over 0.030, which is 0.4167\. Now take the identical price path shifted up to an 80-cent mean: the directionality index halves to 0.00625, the normalized RMSE halves to 0.015, and the ratio is 0.4167 again. Straight to the point, beta-one over RMSE is 0.50 over 1.20, which is 0.4167. Section 4.1 spends a paragraph arguing that dividing both pieces by mean price makes TQR "comparable across contracts with heterogeneous price levels." Nothing in the construction does that work. The ratio of a slope in cents per minute to a dispersion in cents is scale-free before you touch it, because both quantities scale linearly in the price. The two divisions by mean price are algebraic decoration. The same paragraph calls TQR a "dimensionless" score. It is not. Cents per minute divided by cents leaves one over minutes. Our worked example scores 0.4167 per minute; measure elapsed time in seconds instead of minutes and the same price path scores 0.00694\. Every decile cut point in the paper depends on the choice of clock. Greene fixes the window at 18 minutes and the unit at minutes for every contract, so the cross-sectional comparison inside this paper survives. The dimensional claim in the text does not, and anyone porting TQR to a 5-minute or 2-hour window has to refit the thresholds from scratch. I am keeping the source's form and flagging it rather than quietly redefining the statistic. ## TQR is an effect size, not evidence A reader who has fit a trend line before will ask why not use the t-statistic on the slope. The paper addresses this in prose and drops the algebra, which hides the interesting part. $$ t\_{\\beta\_1} = \\frac{\\beta\_1}{s\\,/\\sqrt{S\_{xx}}}, \\qquad s = \\text{RMSE}\\sqrt{\\frac{n}{n-2}}, \\qquad S\_{xx} = \\sum\_{i=1}^{n} (x\_i - \\bar{x})^2 $$ $$ \\Longrightarrow \\quad t\_{\\beta\_1} = \\text{TQR}\\,\\sqrt{S\_{xx}\\,\\frac{n-2}{n}} $$ The OLS standard error uses the residual standard deviation s, which divides the squared residuals by n minus 2, while the paper's RMSE divides by n. Correct for that and the t-statistic is TQR multiplied by the square root of the regressor's total sum of squares, scaled by n minus 2 over n. Sum over the window observations; x is elapsed minutes. Worked example with the dense case. Nineteen one-minute prints across the 18-minute window put x at 0 through 18, so the mean is 9 and the sum of squared deviations is 570\. The factor 17 over 19 gives 510, whose square root is 22.58\. Our TQR of 0.4167 corresponds to t equal to 9.41\. Now the sparse case: the same TQR on the 5-point minimum, with prints at 0, 4.5, 9, 13.5 and 18 minutes elapsed, gives a sum of squared deviations of 202.5, a factor of 3 over 5, and t equal to 4.59\. Same trend quality, half the statistical evidence. That is the honest description of what a TQR sort does. It ranks contracts by effect size, the way a Cohen's d ranks group differences, and stays silent about how much data backs each estimate. Applying it to the bottom decile makes the point: mean TQR of 0.024 on 19 prints implies t equal to 0.54, which is no trend at all. The top decile's 0.574 implies t equal to 12.96\. So the sort separates flat contracts from steep ones, and inside any one decile it pools a five-print guess with a nineteen-print measurement. Greene does carry window observations as a regression control, and that control lands at 0.065 with a standard error of 0.025 on forecast error, so sampling density is doing measurable work. The decile sorts do not control for it at all. ## The continuation ladder, and the ties it is built on Continuation asks whether the price observed closest to expiration sits on the trend's side of the price at 12 minutes remaining. Roughly 49% of non-zero-trend windows show exactly zero change over that stretch, which leaves the binary outcome undefined for half the sample. Greene handles this with a tie-adjusted score that scores a tie as half a win. $$ \\text{Cont}^{0.5} = (1-z)\\,c \\;+\\; \\tfrac{1}{2}z \\qquad \\Longleftrightarrow \\qquad 2\\,\\text{Cont}^{0.5} - 1 = (1-z)\\,(2c-1) $$ Here z is the share of windows with zero post-window change and c is the continuation rate among the windows that did move. The right-hand form is the one a trader wants, because 2c minus 1 is the directional edge on a coin-flip payoff and one minus z is the fraction of the time you get paid at all. Check it against the paper's own table. The top decile reports c equal to 70.8% with z equal to 42.6%, so the edge is 0.574 times 0.416, which is 0.2388, and the tie-adjusted score is 61.94%. Greene reports 61.9%. The bottom decile reports 51.2% with z equal to 30.0%, giving 0.700 times 0.024, which is 0.0168, and a score of 50.8% against the paper's 50.9%. Decile 4 reports 57.2% with z equal to 64.3%, giving 52.6% on the nose. Table 1 and Table 2 reconcile to a tenth of a point. ![Continuation probability by absolute TQR decile on Kalshi last-trade prices, rising from 0.51 in decile 1 to 0.71 in decile 10 with a dip at decile 8](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/09/article_446-continuation_by_tqr_decile.png) The ladder is the bull case, and it is not a clean ladder. Deciles 3 through 5 sit flat at 57%, decile 8 drops to 59.4% below deciles 6 and 7, and the whole rise is 19.6 points across a nine-step sort whose fitted slope is 1.8 points per decile. Two other things matter more than the wiggle. Continuation conditions on the post-window move being non-zero, which is a property of the outcome, so you cannot filter on it at entry. And continuation records only the sign of the move. Section 4.2 defines a magnitude-weighted version, the signed post-window change in cents, and no table in the paper reports it. ## The error result reads backwards for a trader Absolute forecast error is the gap between the price at 12 minutes remaining and the settlement value of 0 or 100 cents. Across the |TQR| sort it falls from 14.75 cents in the bottom decile to 6.07 cents in the top, an 8.68-cent improvement that Greene correctly reports as a 58.9% reduction. Then look at the path it takes to get there. ![Mean absolute forecast error in cents by absolute TQR decile, non-monotone with a peak of 15.60 at decile 3, a trough of 3.64 at decile 5, and 6.07 at decile 10](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/09/article_446-abs_error_by_tqr_decile.png) Deciles 4 and 5 post the lowest errors in the table at 4.00 and 3.64 cents, well below the top decile's 6.07\. Decile 3 posts the highest at 15.60, above the bottom decile. The paper says this plainly, that the series "is not strictly monotone" and that individual decile means are noisy at a resolved sample of 1,094\. Two further things are worth extracting. First, the arithmetic. Average the ten decile error means without weights and you get 8.62 cents, while Table 4 reports an overall mean of 7.41\. The gap has to come from unequal numbers of resolved markets per decile, which means the N column in Table 1 (407, then 406 nine times) describes the trend-quality deciles and not the error column that sits beside it. The resolved counts behind each decile mean are never printed. Under the midrange restriction this bites hard: 1,317 contracts pass the 20-to-80-cent filter, only 227 of them resolved, and the paper still prints ten decile error means with bootstrap intervals. That is roughly 23 resolved markets per decile carrying the headline 20.98-cent midrange result. Second, and this is the part that decides whether TQR is tradable, low forecast error means the market price was already close to the truth. Deciles 4 and 5 show zero-change shares of 64.3% and 64.2%, the highest in the table, alongside the lowest errors. Those are contracts that have stopped trading because the answer is settled and the price is pinned near a boundary. Knowing that a price is accurate gives you nothing to bet on. You need the price to be wrong and to know which way. Low trend quality flags the contracts with the largest errors, so that is where the mispricing lives, and by construction those are the contracts with no discernible direction. The error result and the directional result point at disjoint sets of markets. ## Midquotes erase the direction and keep the error Greene rebuilds everything from the midpoint of the best bid and ask, which cures stale last trades and raises the usable sample from 18,987 to 21,464 contracts and the non-zero-trend set from 4,061 to 5,077\. The two versions of TQR correlate at about 0.77, so this is the same statistic measured on a cleaner price. ![Continuation rate by absolute TQR decile, last-trade prices versus quote midprices, showing a 19.6 point top-minus-bottom gap on trades collapsing to 1.9 points on midquotes](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/09/article_446-continuation_lasttrade_vs_midquote.png) The directional gradient disappears. Top minus bottom falls to 1.9 points with a bootstrap p-value of 0.62, the tie-adjusted version falls to 1.4 points at p equal to 0.63, and the top decile itself prints 49.1%, under a coin flip. The out-of-sample classification result is worse than null: trend quality alone reaches an AUC of 0.448 on midquotes, against 0.583 on last trades. A ranker below 0.5 is ranking backwards. Meanwhile the price-level and spread controls, with no trend quality at all, reach an AUC of 0.617. The error result survives the switch and gets larger, from 25.52 cents in the bottom decile to 5.51 in the top, a 78.4% cut. Read the rest of that column before crediting it to trend quality. In the midquote regression of absolute error on trend quality plus controls, mean spread carries a coefficient of 0.571 and the specification reaches an R-squared of 0.576, while adding trend quality contributes 0.005\. Out of sample the controls deliver an R-squared of 0.311 and adding trend quality drops it to 0.291\. Spread is doing the work. Wide-spread contracts are unresolved contracts trading in the middle of the range, and those carry large settlement errors for reasons that have nothing to do with how straight their last 18 minutes looked. The baseline sample tells a quieter version of the same story. The decile sort gives 8.68 cents, but the conditional regression of absolute error on trend quality with controls gives a coefficient of minus 6.177 with a standard error of 3.259, so t equals minus 1.90\. Multiply that coefficient by the 0.550 spread in mean |TQR| between the bottom and top deciles and you get minus 3.40 cents, which is 39% of the sorted gap and not significant at 5%. The continuation regression holds up better: 0.166 times 0.550 gives 9.1 points against the sorted 11.1. ## The break-even the paper never runs Take the top decile at face value and price the trade. Buy in the trend's direction at 12 minutes remaining, hold to close, exit at the prevailing price. $$ \\mathbb{E}\[\\pi\] \\;=\\; \\bigl(2\\,\\text{Cont}^{0.5} - 1\\bigr)\\,d \\;-\\; k \\qquad\\Longrightarrow\\qquad d^{\*} \\;=\\; \\frac{k}{2\\,\\text{Cont}^{0.5} - 1} $$ Expected profit per contract in cents is the directional edge multiplied by d, the average absolute size of the post-window move in cents, minus k, the round-trip cost in cents from spread and fees. This assumes the winning and losing moves average the same size, which is generous near a boundary. The break-even move size d-star is the cost divided by the edge. The top decile's edge is 2 times 0.619 minus 1, which is 0.238\. At a 1-cent round trip you need the average post-window move to exceed 4.2 cents to break even, and at 2 cents you need 8.4 cents. Put a plausible 2-cent move through it and the gross take is 0.48 cents per contract, which a 1-cent round trip erases twice over. In the bottom decile the edge is 0.018 and the gross take on the same 2-cent move is 0.036 cents. Whether d clears 4.2 cents cannot be answered from this paper, because the signed post-window change is defined and never tabulated, spread is carried as a control and never reported as a level, and no cost assumption appears anywhere in the text. Those three omissions are the gap between a state variable and a strategy. ![Reliability diagram for the trend-quality-only continuation classifier on the test split, with predicted probabilities compressed between 0.54 and 0.72 and wide Wilson intervals](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/09/article_446-continuation_reliability_diagram.png) Greene's own reliability check says the same thing in a picture. Predicted continuation probabilities compress into the band from 0.54 to 0.72, the two highest-predicted bins both realize about 64.5% and so fall below their predictions, and every bin's 95% Wilson interval spans roughly 12 points. Nothing in that plot separates the contracts you would size up from the ones you would skip. Greene labels it a noisy check rather than a validation, which is the right call. ## What TQR is actually for Trend quality near settlement earns a place in a Kalshi dashboard as a convergence gauge. High absolute TQR on last trades says the market has picked a side and is walking to it, so there is little left to win and the post-window continuation you would be betting on is mostly the pin. Low absolute TQR says the price is still wrong by 12 to 15 cents on average and nobody knows which way, which is where mispricing sits and where a directional trend signal has nothing to say. Used as a filter for standing down, it costs nothing and it is cheap to compute from a bid, an ask and a timestamp. Used as alpha it fails on the paper's own evidence. Out-of-sample R-squared is negative for both continuous targets in the baseline, from minus 0.003 to minus 0.019 on the tie-adjusted score and minus 0.017 to minus 0.064 on absolute error, which says a constant beats the model on held-out contracts. The AUC of 0.583 is the single number that survives, and it evaporates to 0.448 the moment you measure price from quotes rather than from the last trade someone happened to print. The direction result depends on stale transaction prices. Glosten and Milgrom explained why quotes and trades carry different information 40 years ago, and the sensible conclusion here is that a trend fitted to sporadic last trades in a thin near-expiry contract is partly measuring the arrival pattern of trades. Moskowitz, Ooi and Pedersen built time-series momentum across decades of futures data with cost models attached. An 18-minute regression on a contract that will not exist in 12 minutes is not in that category, and Greene says so in the discussion: TQR "should not be viewed as a universal alpha." That sentence is the finding. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/09/article_446-visual-0.png) ## KEY POINTS - The Trend Quality Ratio is the pre-close trend slope divided by its residual RMSE on the window from 30 to 12 minutes before a Kalshi contract closes. The paper's two divisions by mean price cancel exactly, so the price-level normalization described in section 4.1 does nothing. - TQR carries units of one over minutes, not the "dimensionless" score section 4.1 claims. The same path scores 0.4167 per minute and 0.00694 per second, so every decile threshold is tied to the 18-minute window and the minute clock. Treat this as a source error, not something to silently redefine. - TQR equals the slope t-statistic divided by the square root of the regressor sum of squares. Same TQR of 0.4167 gives t equal to 9.41 on 19 prints and 4.59 on the 5-print minimum, so the sort ranks effect size and ignores how much data backs it. - The headline direction result is a top-minus-bottom continuation gap of 19.6 points (70.8% against 51.2%). It conditions on the post-window move being non-zero, which you cannot filter on at entry, and Table 1 and Table 2 reconcile through the identity 2 times Cont-0.5 minus 1 equals (1 minus tie share) times (2c minus 1). - The 8.68-cent forecast-error improvement (a 58.9% cut) points the wrong way for a trader. Low error means the price at 12 minutes was already right, and the lowest-error deciles are the pinned ones with 64% zero-change shares. Mispricing sits in the low-TQR contracts, which by construction have no direction. - Rebuild from quote midprices and the direction dies: top minus bottom falls to 1.9 points at p equal to 0.62, the top decile prints 49.1%, and the trend-quality-only AUC falls to 0.448 while price and spread controls alone reach 0.617\. Spread carries the midquote error result, with an R-squared of 0.576 and trend quality adding 0.005. - Out-of-sample R-squared is negative on both continuous targets in the baseline, so a constant beats the model on held-out contracts. Break-even needs the average post-window move to clear 4.2 cents at a 1-cent round trip, and the paper never reports the move size, the spread level or any cost assumption. - Sample caveats the paper discloses or implies: one BTC series supplies 89% of raw snapshots, only 1,094 of 4,061 non-zero-trend contracts resolved, per-decile resolved counts are never printed, and the midrange error result rests on about 23 resolved markets per decile. ## References - [Trend Quality and Predictability in Prediction Markets: Evidence from Minute-Level Kalshi Data - Joseph James Clyde Greene](https://papers.ssrn.com/sol3/papers.cfm?abstract%5Fid=6442939&ref=aligrithm.com) --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ### 3.45 Do Intra-Bar Timestamps Carry Alpha? OHLC + Timing Features URL: https://aligrithm.com/do-intra-bar-timestamps-carry-alpha-ohlc-timing-features/ Last updated: 2026-10-16T08:09:14.000Z Bloomberg started selling the clock inside the bar. Not just the high price for the minute, but the second at which the high printed, the same for the low, the first and last trade times, condition-code tick counts, and a Bloomberg VWAP for the interval. Ruslan Tepelyan trained three neural networks on that feed across the Russell 3000 for all of 2021 and reports that the timing features improve every architecture on every metric he tracks. The claim holds. Check the size of it before you call your data vendor. Test-set negative log-likelihood for his best Transformer runs 0.4311 with OHLC and past VWAP returns alone, 0.4088 once he adds volume, tick counts, bar-shape ratios and prior-activity statistics, and 0.4075 once he adds the timestamps. The timestamps move the number by 0.0013 out of a total improvement of 0.0236\. Directional accuracy goes 71.889%, then 72.288%, then 72.294%. Six thousandths of a percentage point of direction. The 72.3% headline is real and owes nothing to the clock. The timestamps earn their keep in the second moment. Root mean squared error between predicted variance and realized squared error improves 4.2% on the test set when you add them, and the no-timing feature set is worse than plain OHLC on that metric (4.4752 against 4.4261). Every bit of the conditional-variance gain, and then some, comes from the two timestamps. That is a volatility feature, and calling it alpha requires a target you can trade. The target here was chosen because you cannot. ![Share of the basic-to-full test set improvement contributed by the timing features, by metric: conditional variance RMSE 135.6 percent, mean squared error 29.0 percent, coefficient of determination 28.8 percent, negative log-likelihood 5.5 percent, directional accuracy 1.5 percent, calibration error minus 2.4 percent](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/09/article_456-timing_share_by_metric.png) ## What the enhanced bars actually contain The new fields are the timestamp of the first trade at the open, the first occurrence of the high, the first occurrence of the low, and the last trade at the close, each rescaled so that 0 is the start of the bar and 1 is the end. From those he derives two features: the time difference between the high and the low, and a "timing surprise" indicator. $$ \\tau\_{H},\\ \\tau\_{L} \\in \[0,1\], \\qquad S\_{t} \\;=\\; \\mathbb{1}\\!\\left\[\\ \\operatorname{sign}\\!\\left(C\_{t}-O\_{t}\\right)\\cdot\\operatorname{sign}\\!\\left(\\tau\_{H,t}-\\tau\_{L,t}\\right) \\;<\\; 0 \\ \\right\] $$ Read the indicator as a disagreement flag. Tau-H and tau-L are the fractional times of the high and the low. A bar that closes above its open usually prints its low first and its high later, because that is what an up path looks like. Timing surprise fires when the order reverses. Worked example: a bar opens at 100.00 and closes at 100.22, prints its high at second 12 (tau-H of 0.20) and its low at second 47 (tau-L of 0.78). The close-minus-open sign is positive, the tau difference is negative, the product is negative, so the flag is 1\. The bar finished up, yet the high came 35 seconds before the low, which means the entire up move happened in the last 13 seconds. How often should that fire? Tepelyan says up bars are "much more likely" to print the high after the low and leaves it there. I simulated 400,000 driftless 60-step random walks: the high arrives after the low on 88.2% of up bars, so timing surprise fires on 11.8% of bars under a pure random walk. A flag that fires one minute in eight has enough variation to matter and is rare enough that a model needs a large sample to learn from it. He has one. He also does not use the open and close timestamps as features, only the high and low times. On a one-minute bar with at least 30 ticks the open and close times sit within a second or two of the bar boundaries, so they carry almost no information. Sensible omission, but it means the whole result rests on two numbers per bar. ## The target was chosen so that it cannot be arbitraged The prediction target is the log change in Bloomberg's VWAP from one bar to the next. Tepelyan gives four reasons, and the fourth is the one to sit with: "Weaker arbitrage incentives. Predicting the direction of VWAP changes does not lend itself to straightforward arbitrage." His argument is that knowing next minute's VWAP will be higher does not help unless you already executed at the current VWAP, so the pattern can survive in an efficient market. That is an honest statement and it settles the title question. A signal that persists because nobody can trade it is a measurement, not an edge. He reports no transaction costs, no execution model, and no strategy anywhere in the paper. Fair enough, since he is evaluating a data product rather than pitching a system. Anyone reading 72.3% directional accuracy as a trading result has stopped reading at the abstract. Worse for the trading interpretation: the target is standardized by the training-set mean and standard deviation, and the raw standard deviation is never printed. Every MSE, every R-squared, every variance RMSE in the paper lives in units of "training-set sigma" with no bridge back to basis points. You cannot check any result in the paper against a spread, a commission, or a tick. For a paper about a commercial data feed, that is the single most useful number omitted. ## Where 72% direction and R-squared of 0.37 come from An R-squared of 0.3744 on next-bar returns should stop you cold. Nothing forecasts one-minute equity returns to 37% of variance. The explanation sits in the overlap between the two VWAP windows. VWAP is an average across the bar, so the return from VWAP at t-1 to VWAP at t measures a move from roughly the middle of one minute to the middle of the next. By the time you predict, you have already seen every trade in bar t-1, including its close. Under a random walk the expected price during bar t is the last observed price, so the expected VWAP-to-VWAP return is approximately the log ratio of the previous close to the previous VWAP, which is an observable you already hold. Tepelyan lists it as a feature: "Close / VWAP Log Return". His Appendix E tests the coefficient-of-one version of that idea and it fails, with validation MSE of 0.88756 against 0.62892 for predicting zero. Take those two numbers and solve for what a shrunk version would do. $$ \\mathrm{MSE}(b) \\;=\\; \\operatorname{Var}(y) \\;-\\; 2\\,b\\,\\operatorname{Cov}(x,y) \\;+\\; b^{2}\\operatorname{Var}(x) $$ $$ b^{\*} \\;=\\; \\frac{\\operatorname{Cov}(x,y)}{\\operatorname{Var}(x)}, \\qquad R^{2} \\;=\\; \\frac{\\operatorname{Cov}(x,y)^{2}}{\\operatorname{Var}(x)\\,\\operatorname{Var}(y)} $$ Here y is the next bar's standardized log VWAP return and x is the previous bar's close-to-VWAP log return. The first line is the mean squared error of predicting b times x; the second gives the coefficient that minimizes it and the variance share it explains. Worked example on his own table. Predicting zero gives MSE equal to the variance of y, so Var(y) = 0.62892\. Setting b to 1 gives 0.88756, so Var(x) minus twice Cov(x,y) equals 0.25864\. If the Transformer's entire explained variance came from x alone, then Cov(x,y) squared over Var(x) equals 0.62892 minus 0.39551, which is 0.23341\. Solving the pair gives Cov(x,y) = 0.572 and Var(x) = 1.403, hence an optimal coefficient of 0.41 and a correlation of 0.61, for an R-squared of 0.371\. His model posts 0.371 on that same subset. So the paper's entire explanatory power is reproducible by one regression on one observable with a shrinkage factor of 0.41\. The martingale baseline fails only because it used a coefficient of 1\. The shrinkage is heavy for a good microstructure reason: the close is a single trade carrying half the spread, while VWAP averages hundreds of trades, so x is the noisier series (implied variance 1.403 against 0.629 for y) and the optimal loading falls well below one. This is my arithmetic on his Appendix E, not his claim, and it does not prove his network is a one-variable regression. Covariance is not separately identified from three MSE numbers, so treat it as a consistency check rather than a decomposition. What it does establish is that the reported R-squared is reachable with no forecasting at all, from the mechanical overlap of two averaging windows plus bid-ask bounce. The decile table supports that reading: the top decile of predicted magnitude hits 88.21% direction and the bottom decile 52.85%, which is exactly the pattern you get when the signal is "how far did the last bar close from its own VWAP". The old article ["Price-Path Convexity: A New Cross-Sectional Anomaly (−45bp per σ)"](https://aligrithm.com/price-path-convexity-a-new-cross-sectional-anomaly-45bp-per-s-2/) makes the same structural point at a monthly horizon, that the shape of the path between two endpoints carries information the endpoints do not. The difference is that convexity is measured on a target you can hold for a month. Here the extra path information is measured against a target defined by an average that overlaps the information set. ## Same OHLC, four times the volatility Two bars can share an identical open, high, low and close and still differ in volatility by a factor of four. The separator is how long the price took to travel between the extremes, and that is what the two timestamps report. ![Two synthetic one-minute paths with identical open 100.00, high 100.20, low 99.90 and close 100.15. In bar A the low prints at second 3 and the high at second 57, implying 19.0 basis points per minute. In bar B the low prints at second 28 and the high at second 31, implying 80.5 basis points per minute](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/09/article_456-intrabar_timing_variance.png) Both bars above have the same OHLC. Bar A crawls from its low to its high over 54 seconds. Bar B covers the same 30 cents in 3 seconds and then chops sideways. A range estimator that only sees high and low assigns them the same volatility. Add the timestamps and the diffusion scaling separates them. $$ \\hat{\\sigma}^{2}\_{t} \\;=\\; \\frac{\\left(\\log H\_{t} - \\log L\_{t}\\right)^{2}}{4\\log 2 \\;\\cdot\\; \\left|\\tau\_{H,t}-\\tau\_{L,t}\\right|} $$ The numerator is the squared log range. The constant 4 log 2 is the usual correction that turns an expected squared range into a variance for a Brownian path over the full interval, so with a tau difference of 1 this is the classic range estimator. Dividing by the tau difference uses the fact that variance accumulates in proportion to elapsed time: if both extremes landed inside a window covering 5% of the bar, the bar's range was generated by 5% of the bar's clock, and the per-minute variance implied is twenty times larger. Worked example with the figure's numbers. The log range is log(100.20/99.90) = 0.0029985, squared to 8.991e-6, and 4 log 2 = 2.7726\. With a tau difference of 1 you get 18.0 basis points per minute, the plain range estimate. Bar A has a tau difference of 54/60 = 0.90, giving 19.0 basis points. Bar B has 3/60 = 0.05, giving 80.5 basis points. Identical OHLC, a factor of 4.24 in volatility, and the only inputs that changed are two timestamps. That factor is the square root of 0.90 over 0.05, the square root of 18. Flag the bias before anyone uses this as a level estimate. Conditioning on the window that happens to contain both extremes picks the most active stretch of the minute, so the estimator overstates variance and the overstatement grows as the tau difference shrinks. It works as a ranking device between bars that share a range. It is not a drop-in volatility number. The paper's variance results follow the mechanics. Conditional-variance RMSE on the test set: 4.4261 for basic OHLC, 4.4752 for the no-timing set, and 4.2880 with the timestamps. The non-timing features made variance prediction worse, which Tepelyan flags as unexpected and does not resolve. The timestamps improve it by 4.2% over the no-timing set and 4.9% on validation. Relative to a 0.13% likelihood gain and a 0.008% directional gain, a 4% improvement in variance estimation is the only number in the paper large enough to survive a skeptical reading. ## The distribution choice beat the data product All three architectures output the parameters of a Student-t distribution and train by maximum likelihood. Read the loss and its interpretation before reading any of his tables. $$ \\mathrm{NLL} \\;=\\; -\\frac{1}{N}\\sum\_{n=1}^{N}\\log f\\!\\left(y\_{n}\\mid \\mu\_{n}, s\_{n}, \\nu\_{n}\\right), \\qquad \\frac{\\bar{f}\_{B}}{\\bar{f}\_{A}} \\;=\\; \\exp\\!\\left(\\mathrm{NLL}\_{A}-\\mathrm{NLL}\_{B}\\right) $$ The left expression is the average negative log density the model assigns to what happened. The right identity converts any NLL gap into a ratio of geometric-mean densities, which is the only way to read these numbers as sizes rather than decorations. Worked example. All of the feature engineering, basic to full, is exp(0.4311 − 0.4075) = 1.0239, so 2.4% more probability density per bar. The timestamps alone are exp(0.0013) = 1.0013, or 0.13% per bar. Against the unconditional standard normal at 1.2830, the full model is exp(0.8755) = 2.400, so it assigns 2.4 times the density of an unconditional baseline. That gap of 0.8755 is 37 times the entire feature-engineering gain, which is the right way to read the ranking: knowing the conditional scale of the next bar is the bulk of the model, and the features fight over a thin margin on top of it. Now the distribution ablation. Replacing the Student-t with a Gaussian of matched mean and variance costs 0.0297 of test NLL, which is exp(0.0297) = 1.0301, a 3.0% density loss. Choosing the right tail shape buys 23 times what the intra-bar timestamps buy. $$ f(y) \\;=\\; \\frac{\\Gamma\\!\\left(\\frac{\\nu+1}{2}\\right)}{\\Gamma\\!\\left(\\frac{\\nu}{2}\\right)\\sqrt{\\nu\\pi}\\,s}\\left(1+\\frac{1}{\\nu}\\left(\\frac{y-\\mu}{s}\\right)^{2}\\right)^{-\\frac{\\nu+1}{2}}, \\qquad \\operatorname{Var}(y) \\;=\\; s^{2}\\,\\frac{\\nu}{\\nu-2} $$ Mu locates the distribution, s scales it, and nu controls how slowly the tails decay, with the variance blowing up as nu approaches 2\. Worked example at nu = 4, s = 1, mu = 0, so the variance is 2 and the matched Gaussian has standard deviation 1.414\. At the centre the t density is 0.375 against the Gaussian's 0.282, so 33% more mass where most bars land. Eight units out, which is 5.66 Gaussian sigmas, the t gives 3.15e-4 and the Gaussian gives 3.18e-8, a ratio near 9,900\. His test target carries excess kurtosis of 82.7, so those tails are not hypothetical. Calibration is where the tail choice shows up in something you can audit. Following Kuleshov, Fenner and Ermon, he scores the model by how often reality falls below each predicted quantile. $$ \\hat{p}\_{j} \\;=\\; \\frac{1}{N}\\left|\\left\\{\\,y\_{n}\\;:\\;\\mathcal{F}\_{X\_{n}}(y\_{n}) < p\_{j}\\,\\right\\}\\right|, \\qquad \\mathrm{cal} \\;=\\; \\sum\_{j=1}^{M}\\left(p\_{j}-\\hat{p}\_{j}\\right)^{2} $$ For each nominal quantile p-j, count the fraction of observations that landed below the model's own p-j quantile, then sum the squared misses over M = 100 evenly spaced quantiles. Perfect calibration means the 30th percentile prediction is exceeded 70% of the time. Worked example. His full-feature validation calibration is 0.0330 after the paper's factor of 100, so cal = 3.30e-4\. Spread evenly over 100 quantiles that is 3.30e-6 per quantile, and the square root is 0.0018, so the typical quantile is off by 0.18 percentage points. The Gaussian ablation posts 2.0310, which works out to 1.43 points per quantile, eight times worse. The unconditional normal posts 173.77, which is 13.2 points per quantile, a model that does not know what a quantile is. ![Calibration histograms on the test set. The full and no-timing feature sets track the uniform ideal with chi-squared of 0.0262 and 0.0257, the basic set is skewed toward low CDF values at 0.3138, and the Gaussian ablation spikes at both extremes at 1.0631](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/09/article_456-calibration_test_set.png) The red line is the Gaussian ablation, piling up density at CDF values of 0 and 1, which is what a model looks like when reality keeps landing outside its tails. Note the two middle curves: the no-timing set scores 0.0257 and the full set 0.0262, so the timestamps made test calibration slightly worse. Tepelyan calls the difference negligible in practice, which it is. It also means the timestamps lose on one of his six metrics out of sample after winning on it in validation, where they cut calibration error from 0.2236 to 0.0330. ## What would change my mind Three things in the setup keep me from reading the 0.0013 as a finding. The comparison is not architecture-matched. He tuned hyperparameters separately per feature set, which is the right protocol, and the winners differ: the no-timing Transformer took 6 attention blocks with an embedding size of 32, while the full-feature Transformer took 10 blocks with an embedding of 128 and weight decay three orders of magnitude smaller. The full model is bigger. The 0.0013 gap bundles the timestamps with roughly double the attention depth and four times the embedding width. No dispersion is reported. Each configuration ran three random seeds and he reports the mean across them. Three seeds with no standard error, and differences of 0.0008 to 0.0013 in validation NLL. Across architectures the full-feature spread is 0.0038 (MLP 0.2480, GRU 0.2452, Transformer 0.2442), so the entire timing effect is a fifth of the architecture noise band. The monotone ordering across nine architecture-by-feature-set cells is the strongest evidence he has, and it is circumstantial. The filters remove most of the market. He keeps only bars with at least 30 ticks and a lookback-window low above 4 dollars, which leaves 1,021 unique names over 63 validation days, about a third of the Russell 3000\. His own arithmetic: a complete panel would have been 18 times larger. If all 1,021 names appeared on all 63 days, the average stock contributes roughly 20 valid minutes per day out of 370 usable ones. The surviving sample is the busiest minutes of the busiest names, which is exactly where intra-bar timestamps are measurable and where their information content should be largest. Generalizing to a quiet mid-cap at 2pm is not supported. Architecture barely matters here, which agrees with the old article ["State-Space Models for Price: CryptoMamba vs Transformers (Skeptical)"](https://aligrithm.com/state-space-models-for-price-cryptomamba-vs-transformers-skeptical/). Swapping an MLP for a GRU for a Transformer moves validation NLL by 0.0038 on the full feature set. Swapping the feature set inside a single architecture moves it by 0.0225, about six times more. Data beats topology, as usual. ## The verdict Intra-bar timestamps carry information. They do not carry alpha, and the paper never claims they do. What they carry is a 4% improvement in conditional-variance estimation, which has a clean mechanical explanation: the gap between the high time and the low time tells you how much clock the range consumed, and that separates a jump from a drift at identical OHLC. If you size positions by predicted volatility, run a range-based estimator, or price short-dated intraday options, that is worth having and you can test it in an afternoon with the formula above. If you came for direction, the numbers say no. The timestamps add 0.006 percentage points of directional accuracy out of sample, moving the win-to-loss odds from 2.6085 to 2.6093, an improvement of 0.03%. The non-timing features, volume and tick count and bar shape, add 2.00% to those odds. And the 72.3% itself is largely the arithmetic of two overlapping VWAP windows and a bid-ask bounce, not a forecast. Buy the feed for the volatility model. Do not buy it for the signal. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/09/article_456-visual-0.png) ## KEY POINTS - Bloomberg's enhanced one-minute bars add the timestamps of the high and low, the tick count, and a bar VWAP. On the Russell 3000 in 2021, Tepelyan's best Transformer improves test NLL from 0.4311 (OHLC only) to 0.4088 (all non-timing features) to 0.4075 (with timestamps). The timestamps supply 0.0013 of the 0.0236 total, or 5.5%. - Decompose by metric and the timestamps are a volatility feature. They supply 136% of the conditional-variance RMSE improvement (the non-timing features made variance prediction worse), 29% of the MSE improvement, 5.5% of the likelihood improvement, 1.5% of the directional improvement, and minus 2.4% of the calibration improvement. - The mechanism is diffusion scaling. Two bars with identical OHLC differ in volatility by the square root of the ratio of their high-to-low time gaps. A 30-cent range traversed in 54 seconds implies 19 basis points per minute; the same range in 3 seconds implies 80.5\. Divide the squared log range by 4 log 2 times the tau gap, and treat it as a ranking device, since conditioning on the window containing both extremes biases the level upward. - The 72.3% directional accuracy and R-squared of 0.3744 are mostly mechanical. Solving Tepelyan's Appendix E baselines gives an implied coefficient of 0.41 on the previous bar's close-to-VWAP return, a correlation of 0.61, and an R-squared of 0.371, matching his model. One observable plus shrinkage reproduces the entire result with no forecasting. - The target cannot be traded, by design. Tepelyan picked VWAP-to-VWAP return partly for "weaker arbitrage incentives", reports no transaction costs, and standardizes the target without ever publishing its raw standard deviation, so no result in the paper can be checked against a spread. - Choosing the Student-t over a matched Gaussian buys 0.0297 of test NLL, 23 times what the timestamps buy, and cuts per-quantile calibration error from 1.43 points to 0.18\. Distribution shape beat the data product. - Three caveats keep the timing gain provisional: the no-timing and full Transformers differ in size (6 blocks and embedding 32 against 10 blocks and embedding 128), only three seeds ran with no standard errors against effects of 0.0008 to 0.0013, and the tick and price filters keep about 1,021 names and roughly 20 valid minutes per stock-day out of 370, a sample 18 times smaller than the full panel and concentrated in the most active minutes. ## References - A new methodology to exploit predictive power in OHLC data - Andrew D. Mann, Denise Gorse, ICANN 2017 - Machine learning for forecasting mid-price movements using limit order book data - Paraskevi Nousi et al., IEEE Access 7, 2019 - [Enhancing OHLC Data with Timing Features: A Machine Learning Evaluation - Ruslan Tepelyan (arXiv 2509.16137)](https://arxiv.org/abs/2509.16137?ref=aligrithm.com) --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ### 6.56 Piecewise-Linear Regime Identification (Interpretable, No Black Box) URL: https://aligrithm.com/piecewise-linear-regime-identification-interpretable-no-black-box/ Last updated: 2026-10-16T08:09:15.000Z Fit a connected piecewise-linear trend to the Swiss Leaders Index with fifteen breakpoints and you get an in-sample R-squared of 0.954\. Fit the same model to a driftless random walk of the same length and you get 0.966\. I ran 200 of them. The noise wins at every single value of K, from zero breakpoints to fifteen. Whatever the 0.954 is measuring, it is not evidence that the SLI has regimes. This research note from Andreas Steiner Consulting proposes segmented regression as an interpretable substitute for black-box regime models: fit straight lines joined end to end, optimize the joint locations, pick the number of joints by the elbow of the R-squared curve, and read off a bullish or bearish label from each segment's slope. The machinery is clean and the presentation is honest about being descriptive. The problem is what happens between the equations and the chart. Their own reported R-squared ladder violates a property that nested least-squares fits cannot violate, and the breakpoint count they ship sits exactly on top of that violation. Start with what the model is, because the paper's own Section on splines gives away more than it intends. ## The model is a one-layer ReLU network wearing a suit Standard linear regression on a time index assumes one slope for the whole sample. Segmented regression relaxes that by letting the slope change at K locations. Enforcing continuity at those locations, which the paper does and which matters because a jumping fitted price is nonsense, gives the truncated power form. $$ y\_t = \\beta\_0 + \\beta\_1 t + \\sum\_{j=1}^{K} \\beta\_{j+1}\\,(t-\\tau\_j)\_+ + \\epsilon\_t, \\qquad (x)\_+ = \\max(0,x) $$ $$ \\text{slope on segment } m \\;=\\; \\beta\_1 + \\sum\_{j=1}^{m-1}\\beta\_{j+1} $$ The hinge term contributes nothing before its breakpoint and grows linearly after it, so beta-sub-(j+1) is the change in slope at breakpoint tau-sub-j, and the slope on any segment is the starting slope plus every change that has already fired. Worked example using the four-breakpoint SLI fit, with coefficients I reconstructed by reading levels and dates off the paper's own chart (the paper prints no coefficient table, so treat these as my reconstruction, not their output). Take beta-0 = 1.205 and beta-1 = 0.00184 index units per day, with breakpoints at trading days 58, 66, 91 and 216, which map to 27 March 2025, 8 April 2025, 14 May 2025 and 11 November 2025\. Set beta-2 = -0.02144, beta-3 = +0.02292, beta-4 = -0.00310 and beta-5 = +0.00092. Segment 1 runs at 0.00184 per day and carries the index from 1.205 to 1.205 + 58(0.00184) = 1.3117 by 27 March. Segment 2 runs at 0.00184 - 0.02144 = -0.01960 per day, so over the eight days into 8 April the index falls to 1.3117 - 8(0.01960) = 1.1549, a 12% drawdown. Segment 3 runs at -0.01960 + 0.02292 = +0.00332, recovering to 1.1549 + 25(0.00332) = 1.2379 by 14 May. Segment 4 runs at +0.00022, which over 125 trading days moves the index to 1.2654 and is a flat line with extra steps. Segment 5 runs at +0.00114 and closes at 1.3133\. Those five endpoint levels match the fitted dashed line in the paper's chart to within the width of the line. ![Piecewise linear model fitted to the Swiss Leaders Index from 2025-01-03 to 2026-01-12 with four breakpoints, green shading for positive-slope regimes, red shading for the March-April crash, and two out-of-sample points at the right edge](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/09/article_487-sli_piecewise_regimes.png) Now look at what that equation actually is. Input t, K hidden units of the form max(0, t minus a bias), a linear skip connection, and a linear output layer. That is a one-hidden-layer ReLU network with the input weights frozen at one, and fitting it means solving OLS for the output layer while running a quasi-Newton optimizer on the hidden biases. The paper's title claim of no black box rests on the network being tiny and the plot being legible, which is a fair reason to prefer it. It is not a different species of model, and it inherits the training pathologies of the species it belongs to. Hold that thought, because those pathologies show up in their results table. ## Locating the breakpoints, and why L-BFGS-B was the wrong hammer With K fixed, the coefficients come from OLS in closed form, so the only hard part is the breakpoint positions. The paper concentrates the coefficients out and minimizes the residual sum of squares over the knot vector. $$ \\min\_{\\tau\_1,\\dots,\\tau\_K}\\ \\mathrm{RSS}(\\tau)=\\sum\_{t=1}^{T}\\bigl(y\_t-\\hat y\_t(\\tau)\\bigr)^2 \\qquad \\text{s.t.}\\quad \\tau\_{\\min}<\\tau\_1<\\tau\_2<\\cdots<\\tau\_K<\\tau\_{\\max} $$ Read it as: slide the joints around until the fitted polyline hugs the data as tightly as possible, keeping the joints in order and keeping a minimum number of observations inside each segment. The paper uses L-BFGS-B, a quasi-Newton method with box constraints, initialized at equally spaced knots, and enforces the ordering by sorting inside the loop. Worked scale: with T of about 255 daily observations, the number of distinct knot configurations at K=4 is 254-choose-4, which is 169,362,501\. At K=15 it is 254-choose-15, about 5.9 times 10 to the 23\. The paper calls exhaustive search infeasible and it is right about K=15\. It is wrong about K=4, and K=4 is the model they ship. On a laptop with rank-one residual updates, 169 million configurations is an overnight job at worst. Two things make L-BFGS-B a poor fit here, and both are visible in the output. First, the profiled RSS is continuous in tau but kinked at every integer, because moving a knot past a data point changes which observations that hinge column touches. Quasi-Newton methods build a curvature estimate that assumes smoothness, and on a kinked surface the curvature estimate is garbage near the kinks. Second, sorting the knots inside the objective makes the function non-differentiable wherever two knots cross, which corrupts the same curvature estimate a second time. Add the dead-unit failure mode from the network view: if a knot wanders past the end of the sample, its hinge column becomes all zeros, its gradient becomes exactly zero, and it never comes back. ## Their own R-squared ladder proves the optimizer failed Here is the check that takes thirty seconds and that nobody ran. Piecewise-linear models are nested in the number of breakpoints. Any K-breakpoint solution is reachable from a (K+1)-breakpoint model by setting the extra slope change to zero or by stacking two knots on the same date. So at the global optimum, R-squared must be non-decreasing in K. It cannot go down. Ever. ![Grid of sixteen small charts, one per breakpoint count from 0 to 15, each showing the observed SLI in black and the fitted piecewise-linear model in red, with the in-sample R-squared printed above each panel](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/09/article_487-piecewise_fits_k0_to_k15.png) Read the R-squared values off the panel titles: 0.231, 0.377, 0.403, 0.392, 0.861, 0.876, 0.879, 0.884, 0.893, 0.940, 0.941, 0.930, 0.943, 0.944, 0.956, 0.954\. It goes down three times. K=3 scores 0.392 against K=2's 0.403\. K=11 scores 0.930 against K=10's 0.941\. And K=15, the headline number in the abstract, scores 0.954 against K=14's 0.956\. Three violations out of fifteen steps. Each one is a proof that the optimizer returned a local minimum rather than the global one, and the K=3 failure is the largest of the three. I am not correcting these numbers. They are what the source reports, and the source may simply be wrong about them. What I will say is that the reported ladder and the stated method are inconsistent with each other, and the paper does not flag it. ## The knee is a real algorithm sitting on a broken input The elbow rule is the paper's answer to model selection, and the algorithm itself is fine. Draw the chord from the first point of the R-squared curve to the last, then take the point with the largest perpendicular distance above it. $$ R^2(K)=1-\\frac{\\mathrm{RSS}(K)}{\\mathrm{SS}\_{\\mathrm{tot}}},\\qquad \\mathbf{v}=\\bigl\[K\_{\\max}-0,\\ R^2(K\_{\\max})-R^2(0)\\bigr\] $$ $$ d\_k=\\frac{\\bigl|\\,v\_x\\bigl(R^2(k)-R^2(0)\\bigr)-v\_y\\,k\\,\\bigr|}{\\|\\mathbf{v}\\|}, \\qquad k^{\*}=\\arg\\max\_k d\_k $$ The numerator is a two-dimensional cross product measuring how far the curve bulges above the straight line, and the denominator rescales it into a distance. Worked example on the paper's own ladder. The chord runs from (0, 0.231) to (15, 0.954), so v = \[15, 0.723\] and the norm of v is 15.017\. At k=4 the numerator is 15(0.861 - 0.231) - 0.723(4) = 9.450 - 2.892 = 6.558, giving d = 0.4367\. At k=5 it is 15(0.876 - 0.231) - 0.723(5) = 6.060, giving 0.4035\. At k=9 it is 4.128, giving 0.2749\. The maximum sits at k=4 with 0.4367, which reproduces the red dot in their figure exactly. Two observations. The norm of v is the same constant for every k, so dividing by it changes nothing about which k wins. Strip it out and the knee is just the point of maximum vertical gap above the chord, and at k=4 that gap is 6.558 divided by 15, or 0.437 of R-squared. The perpendicular framing is decoration. ![Chart of in-sample R-squared against number of breakpoints for the SLI, with a straight baseline chord from the first to the last point and a red marker at the knee, K equals 4](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/09/article_487-r2_vs_breakpoints_knee.png) The second observation is the damaging one. K=4 wins the knee contest by a wide margin because of the vertical cliff from 0.392 to 0.861 between K=3 and K=4, and that cliff is partly an artifact of the K=3 optimization failure. Part of the jump is real: the March-April crash is a V, and a V needs two knots to model, so no three-knot fit can be as good as a four-knot one. But the reported K=3 value of 0.392 is provably not the global optimum, since 0.403 is achievable by construction. If the true K=3 optimum lands anywhere near 0.8, and a rally-crash-rebound-drift configuration with three knots is available in the data, the cliff shrinks and the knee moves or disappears. The knee inherits every error in the curve it reads. ## Why they needed the knee in the first place The paper lists AIC and BIC before reaching for the elbow, then drops them without showing the numbers. Run them. With T = 253 and RSS proportional to one minus R-squared, moving from K=4 to K=15 changes the goodness-of-fit term by T times the log of (1 - 0.954) over (1 - 0.861), which is 253 times the log of 0.3309, or -279.8\. The AIC penalty as the paper writes it, 2(K+2), rises by only 22 over the same interval. The BIC penalty, (K+2) log T with log 253 = 5.5334, rises by 60.9\. Both criteria pick K=15 by a margin of more than 200 units. It gets worse under an honest parameter count. The paper counts K+2 coefficients and ignores the K breakpoint locations, which are estimated from the same data and cost degrees of freedom. Count 2K+2 and the AIC penalty gap becomes 44 and the BIC gap becomes 121.7\. Still nowhere near 279.8\. Zhang and Siegmund built a modified BIC for precisely this undercount, and the paper cites it as reference \[3\] without using it. So AIC and BIC both run away to the maximum complexity the analyst allows, the analyst does not want fifteen regimes on one year of data, and the knee delivers four. That is the actual sequence. It is a defensible piece of judgment and the paper should present it as judgment rather than as a third selection criterion sitting alongside the other two. ## Noise fits better than the SLI does Here is the test that settles whether any of this R-squared is informative. I generated 200 driftless random walks of 253 observations, fitted connected piecewise-linear models with K from 0 to 15 by greedy forward knot selection on an integer grid, and recorded the in-sample R-squared. R-squared is scale invariant for a driftless random walk, so the daily volatility does not enter and there is nothing to tune. ![In-sample R-squared against number of breakpoints, showing the median and 5th-to-95th percentile band for 200 random walks sitting above the SLI curve reported in the paper at every K](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/09/article_487-random_walk_null.png) Median random-walk R-squared: 0.438 at K=0, 0.892 at K=4, 0.966 at K=15\. The SLI: 0.231, 0.861, 0.954\. Pure noise beats the real index at every breakpoint count. The comparison is generous to the paper, because greedy selection on a coarse integer grid is a weaker optimizer than continuous L-BFGS-B, so the null numbers are a floor rather than a ceiling. None of this says the SLI lacked structure in 2025\. It says in-sample R-squared on a price level cannot tell you whether it did. A random walk is locally trending everywhere, so a polyline with free knots will always find trends in it, and the number it reports back is a measure of how many knots you granted, not of how regime-like the series is. The paper reports 0.954 as a headline. It is the least informative number in the note. ## The regime labels are arithmetic, not estimation The final step assigns each segment a label from its slope, and the paper estimates that slope by regressing the fitted values on time within the segment. $$ \\hat\\beta\_{\\text{seg}}=\\frac{\\sum\_{t=s}^{e}(t-\\bar t)\\,\\hat y\_t}{\\sum\_{t=s}^{e}(t-\\bar t)^2}, \\qquad \\bar t=\\frac{s+e}{2} $$ $$ \\text{label}=\\begin{cases}\\text{bullish} & \\hat\\beta\_{\\text{seg}}>\\theta\\\\\[2pt\] \\text{bearish} & \\hat\\beta\_{\\text{seg}}<-\\theta\\\\\[2pt\] \\text{consolidation} & |\\hat\\beta\_{\\text{seg}}|\\le\\theta\\end{cases} \\qquad \\theta=\\text{66th pct of }\\{\\hat\\beta\_{\\text{seg}}\\} $$ The regression is a no-op. Inside a segment the fitted values are exactly a straight line by construction, so OLS of y-hat on t returns that line's slope with an R-squared of one and recovers beta-1 plus the accumulated slope changes to machine precision. Running a regression here dresses up a subtraction. The threshold is worse. Worked example on the five SLI segments, using the slopes reconstructed above: -0.01960, +0.00022, +0.00114, +0.00184, +0.00332\. Sorted, the 66th percentile by linear interpolation sits at position 0.66 times 4 = 2.64, between 0.00114 and 0.00184, giving theta = 0.001588\. Apply it: segments 1 and 3 come out bullish, segment 2 bearish, segments 4 and 5 consolidation. Change nothing about the market and set theta at the 50th percentile instead and segment 5 flips to bullish. A percentile threshold on five numbers does not classify market states, it partitions a five-element list into buckets whose sizes you chose in advance. With K=4 you will get roughly 1.7 bullish segments no matter what the index did. ## What it is good for, and what it cannot do The fatal constraint for anyone thinking about trading this: the breakpoints are located using the whole sample. That last knot on 11 November 2025 was placed knowing what happened through 12 January 2026\. On 11 November itself, with no future data, the optimizer has no reason to put a knot there. The paper's chart legend holds out two observations as out-of-sample. Two. The knee, the breakpoint count and every knot position come from the full history. That gap is the whole difference between this and a state-space regime model. The old article ["Regime-Switching That Works, Factors That Don't (MS-GARCH)"](https://aligrithm.com/regime-switching-that-works-factors-that-dont-ms-garch/) covers the alternative: a Markov-switching specification gives you a transition matrix and a filtered probability of being in each state using information available at time t, which is a number you can size a position with. Segmented regression gives you a retrospective annotation. Both are called regime detection and only one of them runs forward. Used inside its actual range, the method earns its place. As a compression device it is excellent: five slopes and four dates summarize a year of daily prices in a form a risk committee can read without a tutorial, and the paper's stated applications of conditioning performance attribution and factor exposures on identified periods are legitimate ex-post analysis where hindsight is allowed. The extensions the paper sketches are the right ones, particularly robust loss functions in place of OLS, since a single crash segment of eight observations carries enormous leverage on the squared-error objective. Warwicker and Rebennack solve exactly the robust continuous piecewise-linear problem to global optimality with a Benders decomposition, which would also fix the local-minimum failures shown above. Muggeo's iterative linearization would at least give standard errors on the breakpoint dates, which the paper does not report and which would immediately show how loosely the 14 May knot is pinned down. The verdict: correct model, honest framing of its purpose, and a validation section that measures nothing. Report the segment slopes and dates. Delete the R-squared. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/09/article_487-visual-0.png) ## KEY POINTS - The connected piecewise-linear model is a truncated power spline with free knots, which is identical to a one-hidden-layer ReLU network with unit input weights and a linear skip. Calling it "no black box" describes the size of the plot, not a difference in kind, and the model inherits the same training pathologies. - The paper's own R-squared ladder decreases three times as breakpoints are added: 0.403 to 0.392 at K=3, 0.941 to 0.930 at K=11, and 0.956 to 0.954 at K=15\. Nested least-squares fits cannot do that at the global optimum, so L-BFGS-B returned local minima. The source reports these numbers without flagging the contradiction. - The geometric knee reproduces at K=4 (perpendicular distance 0.4367 against 0.4035 at K=5), but it wins on the 0.392-to-0.861 cliff, and the 0.392 endpoint of that cliff is the largest of the three optimizer failures. Fix K=3 and the knee may move. - AIC and BIC both select K=15 by more than 200 units, even after correcting the paper's parameter count from K+2 to 2K+2\. The knee is the analyst overruling both criteria, which is reasonable, and should be labelled as judgment rather than as a third criterion. - Across 200 driftless random walks of 253 observations, median in-sample R-squared is 0.892 at K=4 and 0.966 at K=15, above the SLI's 0.861 and 0.954 at every K. In-sample R-squared on a price level measures knot count, not regime structure. - Regressing fitted values on time inside a segment recovers the segment slope exactly and adds nothing. A 66th-percentile threshold applied to five slopes fixes the number of bullish segments before looking at the market. - Every breakpoint is placed with full-sample hindsight and the paper holds out two observations. Use this to summarize history and condition attribution after the fact. For a signal that runs forward, you need filtered state probabilities, not a retrospective polyline. ## References - [Automated regime detection in multidimensional time series data using sliced Wasserstein k-means clustering - Knottenbelt & Lowe (2023), arXiv:2310.01285](https://arxiv.org/abs/2310.01285?ref=aligrithm.com) - [A Piecewise-Linear Model For Market Regime Identification - Andreas Steiner Consulting GmbH (ssrn-6067668)](https://papers.ssrn.com/sol3/papers.cfm?abstract%5Fid=6067668&ref=aligrithm.com) --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ### 6.54 Volatility Clustering in Bitcoin: Regime Persistence as a Forecast URL: https://aligrithm.com/volatility-clustering-in-bitcoin-regime-persistence-as-a-forecast/ Last updated: 2026-10-16T08:09:15.000Z Bitcoin's volatility does not mix randomly across time. High-vol periods follow high-vol periods, low-vol stretches extend themselves, and the transition probabilities are stable enough to build a forecast from. Borrego Roldán's 2024 study quantifies this with hourly and daily data from August 2018 to November 2024, constructing 3-state Markov chains around rolling log-volatility and checking whether knowing today's regime tells you anything about tomorrow's moves. It does. But the infrastructure underneath carries a problem worth understanding before you act on any of it. ## How you measure the thing in the first place The study computes rolling historical volatility the standard way: standard deviation of returns over a trailing window. Returns are price ratios, not log returns. $$ R\_t = \\frac{O\_t}{O\_{t-1}}, \\qquad \\sigma\_t = \\text{std}(R\_{t-n+1}, \\ldots, R\_t) $$ O is the open price at time t, and n is the window length. The author then takes the natural log of that series. $$ LV\_t = \\ln(\\sigma\_t) $$ Worked example: suppose the last 24 hourly returns have a standard deviation of 0.003 (0.3% per hour). Log-volatility at that point is ln(0.003) = -5.81\. A more turbulent stretch with std 0.008 gives ln(0.008) = -4.82\. The logged series is stationary across almost all windows tested, with ADF p-values well below 0.01\. The exception is the 90-day daily window, which required differencing. Everything below uses log-volatility. The paper runs this across five timeframes (1h, 4h, 6h, 12h, 1d) and multiple window sizes in each. Autocorrelation functions confirm the clustering: for 1h data with a 24-period window, the ACF at lag 24 sits around 0.65\. The lag at which autocorrelation becomes statistically insignificant roughly corresponds to the window length itself. Short windows cluster fast and go quiet fast; longer windows carry structure out to larger lags. ![Log-volatility (14-day and 60-day rolling windows) for Bitcoin daily data 2019-2025, with fixed full-sample 20th and 80th percentile bands. The series spends long unbroken stretches above or below each band.](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/08/article_485-log_vol_percentile_bands_2019_2025.png) Look at how long vol stays above the 80th percentile band (the upper dashed line) versus how rarely it crosses between bands in a single day. That is the clustering, made visible. Also notice the problem: the bands are fixed horizontal lines, computed from the full 2019-2025 history. From 2019's vantage point those lines did not exist yet. ## Carving the series into three states Three states are enough: Low (below the 20th percentile of log-volatility over the full sample), Medium (between the 20th and 80th), and High (above the 80th). With the 1h, 24-period window those thresholds are fixed numbers. Each hour, the current log-vol drops into one of the three buckets. $$ \\text{State}\_t = \\begin{cases} 1 & \\text{if } LV\_t < P\_{20} \\\\ 2 & \\text{if } P\_{20} \\le LV\_t \\le P\_{80} \\\\ 3 & \\text{if } LV\_t > P\_{80} \\end{cases} $$ Worked: P20 = -5.0, P80 = -4.4 (illustrative values; the actual cutoffs depend on the window and timeframe). A current LV of -5.5 goes into State 1; LV of -4.7 goes into State 2; LV of -4.0 goes into State 3\. From there the analysis asks: given you are in State i right now, what is the probability you are in State j one period later? ## The Markov numbers The transitions are estimated by counting: out of all the hours where vol was in State 1, what fraction transitioned to State 1, State 2, or State 3 the next hour? That gives one row of the matrix. Repeat for all three states. $$ P\_{ij} = \\Pr(\\text{State}\_{t+1} = j \\mid \\text{State}\_t = i), \\qquad \\sum\_j P\_{ij} = 1 $$ P sub ij is the probability of moving from state i to state j. Each row sums to 1\. The diagonal entries are what matter here: they are the persistence probabilities, the chance of staying in the same state. For 1h data with a 24-period vol window: | State | Stay in state | Leak to Medium | Leak to other | | ----- | ------------- | -------------- | --------------- | | Low | **0.9505** | 0.0488 | 0.0007 | | Med | **0.9715** | (own state) | 0.0164 / 0.0121 | | High | **0.9629** | 0.0369 | 0.0003 | With a 48-period window the persistence rises further: Low stays at 0.975, Medium at 0.985, High at 0.980\. Widen the window and the state becomes stickier, because a 48-hour rolling std does not shake off a single noisy hour. Cross to the daily timeframe with a 7-period window and you get the lowest persistence in the whole study: the Low state stays at only 0.8167\. That is the floor. Still above 80%, but meaningfully lower than the hourly figures. By the time you use a 90-period daily window, persistence climbs back to 0.9705/0.9819/0.9774 across the three states. Worked interpretation of the 1h, 24-period matrix: you are in the Low vol state. Tomorrow's probability distribution across states is 95.05% Low, 4.88% Medium, 0.07% High. The direct transition from Low to High in a single step is essentially zero. To reach High, vol has to transit through Medium first, which takes multiple steps. That is the clustering in matrix form. ## What each state predicts about moves This is the part that has a trading application. For each vol state, the paper measures subsequent absolute price moves over standardized lookahead horizons. For 1h data, the primary lookahead is Move\_1 = 48 hours ahead. For daily data, Move\_1 = 7 days ahead. From the 1h data with a 24-period vol window: | State | 48h mean abs move | 48h std | | ----- | ----------------- | ------- | | Low | 2.39% | 2.94% | | High | 4.42% | 4.37% | In the High state, mean plus one standard deviation is 4.42 + 4.37 = 8.79%, so roughly 9% captures one standard deviation of 48-hour outcomes. The paper states this figure directly. t-test statistic between Low and High is -40.5 (p essentially zero), and the Levene test on variance equality rejects even more cleanly, confirming that the distributions differ in both mean and spread. From daily data with a 7-period vol window: | State | 7-day mean abs move | 7-day std | | ----- | ------------------- | --------- | | Low | 2.56% | 2.81% | | High | 4.42% | 4.17% | In the Low state over 7 days: average move is 2.56%, one standard deviation is 2.81%, so the 1-stdev upper boundary sits at 5.37%. The paper summarizes this as "often less than 5%"; the exact table gives 5.37%, which is a small rounding overstatement. t-statistic between Low and High is -7.79 (p near zero), weaker than the hourly case because the daily sample is smaller, but still decisive. ![7-day absolute moves from 2019 to 2025, with green dots marking instances preceded by Low vol state (below 20th percentile of 1d/7-period log-vol) and red crosses for High vol state. Large moves cluster around red crosses.](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/08/article_485-move_timeline_by_vol_state.png) The large spikes visible from 2019 to 2021 are almost all marked red. The quieter 2023-2024 period shows a preponderance of green. The clustering is visible even without the t-tests. ## The part that breaks it in live trading The percentile thresholds are computed from the entire 2018-2024 dataset. P20 and P80 are fixed constants derived from the full history. That means from a 2019 vantage point those thresholds incorporate vol observations from 2020, 2021, 2022, and later years -- years that did not happen yet. This is not a minor efficiency issue; it is a structural look-ahead problem. The paper acknowledges it directly in the Discussion section: "The primary limitation is the static nature of percentile calculation over time. Percentiles were derived from the entire dataset, which introduces periods where certain volatility states are under-represented or even absent. For instance, the year 2021 saw very few moves preceded by low volatility states." Look at the figure above: in 2021, log-vol spent almost its entire time above the 80th percentile band, because that band was calibrated on a dataset that includes the comparatively quiet 2023-2024 period. A trader running this in 2021 would see almost nothing classified as Low vol. The fix the paper proposes is a rolling window for percentile computation: compute P20 and P80 using only the trailing year of data, then update the thresholds daily. That converts the static classification into an adaptive one. The paper does not implement this -- it is flagged as future work -- so the transition matrices and move statistics reported above are in-sample over the full period by construction. Treat them as an upper bound on what you would see in deployment. The old article ["Regime-Switching That Works, Factors That Don't (MS-GARCH)"](https://aligrithm.com/regime-switching-that-works-factors-that-dont-ms-garch/) covers a structurally similar problem from the other direction: MS-GARCH estimates regime parameters from the full sample, which creates similar in-sample favorability. The pattern repeats whenever regimes are defined post-hoc from aggregate statistics rather than estimated online. The old article ["Loose Pants Fit Everyone: Why General Trading Ideas Survive Longer"](https://aligrithm.com/loose-pants-fit-everyone-why-general-trading-ideas-survive-longer/) is relevant here too. A rule that says "expect quiet when vol is low" is loose enough to survive multiple market structures, even if the exact percentile thresholds shift. The direction of the relationship is durable; the magnitude in the tables above is not. ## What the persistence does and does not tell you A persistence of 0.95 in the High vol state does not mean vol will stay elevated for exactly 20 periods (1/0.05). It means each step has a 5% exit probability, giving a geometric distribution with mean 1/(1-0.95) = 20 periods, but the variance around that expectation is wide. You can be in High vol for 2 hours or 200 hours; the Markov model treats both as equally consistent with a 0.95 diagonal. What the matrix does give you is a prior for the next single step. If vol is High now, the probability that it exits to Medium in the next hourly bar is 3.7%, and the direct jump to Low is 0.03%. Building a hedge or sizing a position on that prior is sensible; sizing on "vol will stay High for the next 20 hours" is not. The minimum persistence in the data (0.8167, 1d/7-period Low state) translates to a mean residence of 1/(1-0.8167) = 5.5 days. Short-window daily vol is visibly noisier, so the Low state exits more often. Once you switch to 90-day daily windows, the state is almost absorbing: mean residence at 0.9705 persistence is 1/(1-0.9705) = 34 days. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/08/article_485-visual-0.png) ## KEY POINTS - Bitcoin log-volatility is stationary across all timeframes from 1h to 1d (except the 90-day daily window before differencing) and its ACF decays to near zero at lags beyond the computation window -- confirming persistence, not indefinite memory. - A 3-state Markov chain (Low/Medium/High, split at the 20th and 80th percentile of log-vol) shows diagonal persistence above 0.9 in all but two cases across the entire study. The minimum is 0.8167 for the 1d/7-period Low state. - In the High state, the 48-hour expected absolute move is 4.42% with a standard deviation of 4.37%, giving a mean-plus-one-sigma bound of roughly 9%. In the Low state, the 7-day expected absolute move averages 2.56% with std 2.81%; one-sigma boundary is 5.37%, not the "less than 5%" the paper text claims. - The t-test between Low and High states rejects at t = -40.5 (1h data), confirming the difference is real. The Levene test confirms the variance gap too -- high state spreads are roughly 50% wider than low state spreads. - The percentile thresholds are fixed on the full 2018-2024 dataset. This is a structural look-ahead: the P20/P80 values you see in the tables include post-sample data from the perspective of any date before 2024\. A deployable version needs rolling-window thresholds. - The framework works as a prior for option sizing and hedging, not as a point forecast. Persistence of 0.95 gives a mean residence of 20 periods, but variance around that is wide. Use the next-step transition probabilities, not the mean duration. ## References - [Gyamerah (2019) — Modelling the Volatility of Bitcoin Returns Using GARCH Models — Quantitative Finance and Economics](https://doi.org/10.3934/QFE.2019.4.739?ref=aligrithm.com) - [Yildirim & Bekun (2023) — Predicting Volatility of Bitcoin Returns with ARCH, GARCH and EGARCH — Future Business Journal](https://doi.org/10.1186/s43093-023-00255-8?ref=aligrithm.com) - [Borrego Roldán (2024) — Volatility Clustering in Bitcoin — SSRN](https://papers.ssrn.com/sol3/papers.cfm?abstract%5Fid=5073986&ref=aligrithm.com) --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ### 4.77 Stock-Bond Correlation: The Sign Flips and So Do Its Drivers URL: https://aligrithm.com/stock-bond-correlation-the-sign-flips-and-so-do-its-drivers/ Last updated: 2026-10-16T08:09:16.000Z A 50/50 stock-bond portfolio that earned its target return with moderate volatility before 2000 requires an 80/20 allocation after 2000 to maintain the same profile. That 30-point shift is not a style choice or a bet on equities. It is the portfolio adjustment needed to compensate for the stock-bond correlation flipping from positive to negative, which McMillan documents across the G7 using DCC-GARCH and rolling windows from 1980 to 2023\. The flip happened around 2000 for most markets, a decade earlier for Japan, and barely at all for Italy. The correlation recently shows signs of reverting toward positive around 2022, following the inflation surge. Before you assume this is another dotcom-crash flight-to-safety story, look at the drivers. They are not constant. Real interest rates are the most stable predictor post-2000, but inflation and growth flip sign and significance across Bai-Perron structural breaks. The drivers themselves are time-varying. ## The correlation flips, measured two ways Stock-bond correlation is usually estimated with rolling windows or a dynamic model. McMillan uses both: a three-year rolling Pearson correlation and Engle's DCC-GARCH, which estimates time-varying correlations from a multivariate GARCH model that lets the correlation evolve with its own equation. The DCC model is: $$ \\rho\_{sb,t} = \\text{corr}(r\_{stocks,t}, r\_{bonds,t} \\mid \\text{info up to } t-1) $$ At each time t, the model computes the conditional correlation between stock and bond returns given past volatilities and correlations. The rolling window is simpler: take the last 36 months of returns and compute the Pearson correlation. Both methods agree on the broad pattern. Worked reading: if the DCC correlation for the US in January 2000 is +0.22 and by December 2000 it is −0.05, the correlation switched from positive to negative that year. A positive correlation means stocks and bonds tend to move together, so bonds diversify but do not hedge. A negative correlation means they move opposite, so bonds hedge stocks. The switch from +0.22 to −0.05 is the switch from diversifier to hedger. The sample is monthly total returns on stock and bond indices for Canada, France, Germany, Italy, Japan, the UK, and the US, spanning 1980 to 2023 where data permits. Table 2 in the paper splits the sample at the midpoint for each country. Pre-2000 correlations are positive for every market: Canada +0.16, France +0.22, Germany +0.18, Japan near zero at +0.02, UK +0.32, US +0.22\. Italy is the outlier at +0.30\. Post-2000, the correlation flips negative for six of the seven: Canada −0.12, France −0.03, Germany −0.20, Japan −0.24, UK −0.10, US −0.24\. Italy stays positive at +0.27\. The sign switch is not a US-only phenomenon. Looking at the DCC time series directly, the switch occurs around 2000 for Canada, Germany, the UK, and the US. France switches around the same time but shows earlier evidence of mean reversion back toward positive in the mid-2010s. Japan switched in mid-1992 and remained negative through 2023, with only weak signs of reversion at the end. Italy had a brief negative period from 2000 to 2004 but otherwise maintained a positive correlation throughout. The rolling correlations exhibit greater volatility around the same pattern, which is expected given the smaller effective sample in each window. By the end of 2022 and into 2023, the DCC correlations for most markets shift back toward positive or less negative, consistent with the view that the 2022 inflation surge and rate hikes moved both stocks and bonds in the same direction again. ## Drivers: real rates stable, the rest unstable The paper regresses the time-varying correlation on a set of macro variables: $$ \\rho\_{sb,t} = \\alpha\_0 + \\beta\_1 \\, \\text{Inflation}\_{t-1} + \\beta\_2 \\, \\text{Infl. Vol.}\_{t-1} + \\beta\_3 \\, \\text{Output}\_{t-1} + \\beta\_4 \\, \\text{Output Vol.}\_{t-1} + \\beta\_5 \\, \\text{Real IR}\_{t-1} + \\beta\_6 \\, \\text{Corr(Infl., Output)}\_{t-1} + \\varepsilon\_t $$ Inflation is the month-on-month log change in CPI, output is the log change in industrial production, inflation volatility and output volatility are estimated via GARCH or rolling standard deviation to match the correlation series, the real interest rate is the 10-year Treasury yield minus inflation, and the inflation-output correlation is itself time-varying, estimated by DCC-GARCH or a rolling window. All predictors are lagged one period. Worked example for interpreting a coefficient: if the real interest rate coefficient is +0.055 (the US DCC regression result), a 1 percentage point increase in the real rate is associated with a +0.055 increase in the stock-bond correlation. In practice, real rates varied from near zero to 5% over the sample, so a 2-point move from 1% to 3% would lift the correlation by 0.11, which is enough to move it from neutral to moderately positive. Table 5 reports the full-sample regression for both DCC and rolling correlations. Real interest rates are positive and significant for all markets except Canada and Germany in the DCC specification, and significant for all seven in the rolling specification. Inflation is positive and significant for Germany, Japan, the UK, and the US (DCC), and for Canada, Germany, Japan, the UK, and the US (rolling). Inflation volatility is positive and significant for France, Germany, Italy, Japan, the UK, and the US (DCC), fewer for rolling. Output growth is positive and significant in a handful of cases, including for Canada, France, Italy, and the US in the rolling specification. The typical theoretical story is that higher real rates and higher inflation both depress stock and bond prices via higher discount rates, producing a positive correlation, while higher growth should raise stock prices and lower bond prices, producing a negative correlation. The results here support the first part but contradict the second: output largely has a positive effect, suggesting that in practice higher growth raises demand for all assets rather than rotating from bonds to stocks. The inflation-output correlation (the "nominal-real" correlation) is mostly insignificant, which contradicts the recent literature that links the stock-bond correlation flip to a flip in the inflation-output correlation from negative to positive. ## Bai-Perron breaks: the drivers change over time The above regressions assume constant coefficients. The Bai-Perron procedure tests for multiple structural breaks and estimates separate coefficients for each regime. Table 6 reports the break dates. Most markets show breaks around 1998-1999 (dotcom period), the mid-2000s (pre-financial crisis), and the mid-2010s (post-debt crises). Japan and the US have an additional early-1990s break. Germany and Italy have breaks associated with the financial crisis itself. Table 7 presents the regime-specific coefficients, grouped roughly by date. The variables that affect the correlation change both in sign and in significance across regimes. For the US, inflation and output volatility are significant in the 1980s, only output volatility remains significant through the 1990s, only inflation volatility is significant in the early 2000s, and neither is significant post-financial crisis. Real interest rates, in contrast, are positive and significant from the late 1990s onward for most markets, with some country variation. Germany shows a negative significant real-rate coefficient in the first half of the 2010s while most others are positive. Inflation and inflation volatility become increasingly positive and significant from the late 1990s forward. Output growth remains mixed, sometimes positive and sometimes negative, with no stable pattern. The nominal-real correlation is also mixed. The key takeaway is that the factors driving the stock-bond correlation are not stable. Real interest rates are the most consistent predictor post-2000, typically positive, meaning higher real rates push the correlation toward positive. Inflation also tends to push positive. Growth is unstable. The notion that the correlation is driven by a single unchanging mechanism is not supported. ## Portfolio implications: you need more stocks to stand still The correlation flip is not an academic curiosity. It directly affects portfolio returns and risk. Figure 3 in the paper plots the cumulative return of a 50/50 stock-bond portfolio for each G7 market. For Canada, France, Germany, the UK, and the US, the cumulative path shows a visible flattening or kink around 2000, consistent with the switch to negative correlation detracting from performance. Japan shows a steeper path pre-1995 when the correlation was still positive. Italy, which maintained a mostly positive correlation, does not show the same kink. The US case is the cleanest illustration. Figure 4 (reproduced below) plots the cumulative 50/50 portfolio for the US alongside a trend fitted to the pre-2000 period and extrapolated forward. The realized portfolio diverges sharply below the trend after 2000\. The lower panel overlays portfolios with different stock weights. From 2000 to 2008, a 65/35 stock-bond allocation is needed to track the pre-2000 trend. From 2009 onward, an 80/20 allocation is required. ![US stock-bond portfolio performance with trend and alternative weights](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/08/article_471-us_portfolio_weights.png) Worked reading of the figure: the orange line is the pre-2000 trend. The blue 50/50 portfolio diverges below it after 2000\. The green 65/35 portfolio tracks the trend from 2000 to 2008\. The red 80/20 portfolio tracks the trend from 2009 to 2023\. This is not a bet on higher stock returns. It is the allocation needed to offset the drag from the negative correlation. Table 8 quantifies the same effect from a risk perspective. The mean and standard deviation of the 50/50 portfolio over the full sample are shown, then split into pre-2000 and post-2000 periods. The table also reports the stock weight W needed in each sub-period to match the full-sample standard deviation. For the US, the full-sample 50/50 portfolio has a mean return of 0.40% per month and a standard deviation of 2.57%. Pre-2000, the same 50/50 portfolio had a mean of 0.52% and a standard deviation of 2.72%, so to bring that down to the full-sample 2.57%, you would need only 41% in stocks (W1 = 0.41). Post-2000, the 50/50 portfolio had a mean of 0.29% and a standard deviation of 2.42%, so to lift that up to 2.57%, you need 54% in stocks (W2 = 0.54). The risk shift is symmetric: lower weight pre-2000, higher weight post-2000, to maintain constant volatility. The same pattern holds for returns. To maintain a constant return, you need to increase the stock allocation post-2000 because the negative correlation no longer provides the same diversification boost. ## Variance decomposition: the correlation contribution switches sign Portfolio variance for a two-asset portfolio is the sum of the variance contribution from each asset plus twice the covariance. For a 50/50 stock-bond portfolio: $$ \\sigma\_p^2 = 0.25 \\, \\sigma\_{stocks}^2 + 0.25 \\, \\sigma\_{bonds}^2 + 2 \\times 0.25 \\times \\sigma\_{stocks} \\times \\sigma\_{bonds} \\times \\rho\_{sb} $$ Dividing each term by the total portfolio variance gives the fractional contribution of each component. Table 9 reports these contributions for the full sample and the two sub-periods, using a five-year rolling window to compute variances. For the US over the full sample, stocks contribute 88% of portfolio variance, bonds 24%, and the covariance term contributes −12%. The negative contribution from covariance is the hedging benefit: the negative correlation reduces total variance. Split by period, the story is clearer. Pre-2000, stocks contribute 55%, bonds 22%, and covariance +23%. The positive covariance contribution means the correlation is positive and adds to variance. Post-2000, stocks contribute 108%, bonds 25%, and covariance −34%. The covariance term is now large and negative, providing a substantial hedge. The stock contribution exceeds 100% because the covariance term is negative, so the total sums to 100%. The same flip from positive to negative covariance contribution occurs for Canada, France, Germany, Japan, and the UK. Italy, which maintained a positive correlation, shows a positive covariance contribution in both periods. The bond contribution remains broadly stable across periods for all markets, while the stock contribution increases post-2000 as equity volatility rose. The variance decomposition confirms that the correlation flip is not cosmetic. It fundamentally changes the risk profile of a stock-bond portfolio, shifting the covariance term from a drag to a hedge. ## What caused the flip, and is it stable The paper does not settle causation. The literature points to several candidates: the dotcom crash and subsequent flight-to-safety, the prolonged low-rate environment post-financial crisis, changing inflation dynamics, and shifts in the correlation between inflation and growth. The regression results show that real rates and inflation are associated with higher correlation, while growth is mixed. The Bai-Perron breaks coincide with major crisis periods, which supports the view that regime shifts in macro risk drove the correlation changes. The reversion toward positive correlation in 2022-2023 coincides with the inflation surge and rate hikes, where both stocks and bonds sold off together. If inflation remains elevated or volatile, the correlation may stay positive or neutral. If inflation recedes and rates stabilize, the correlation could flip back to negative. The drivers themselves are unstable, so the correlation is not anchored. For portfolio managers, the implication is that the 60/40 or 50/50 portfolio is not a static allocation. The correlation between stocks and bonds is a regime variable, and the regime can persist for decades. Rebalancing rules that assume a constant correlation will mis-allocate. Monitoring the correlation directly, or the macro drivers that predict it, is necessary to maintain a target return or risk profile. The paper is limited to G7 developed markets and monthly data. Higher-frequency data or emerging markets may show different patterns. The regression framework is reduced-form and does not distinguish between structural shocks (supply vs demand, monetary vs real). The variance decomposition assumes static weights and does not account for dynamic rebalancing strategies that might exploit the correlation flip. The sample ends in December 2023, before the full effects of the 2022-2023 rate cycle are observable. The finding that the stock-bond correlation flips sign, that the drivers of the flip are themselves time-varying, and that the flip has direct portfolio consequences, is robust across methods and markets. The 80/20 allocation needed post-2000 to replicate pre-2000 50/50 performance is not a recommendation. It is a measurement of how much the correlation regime changed. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/08/article_471-visual-0.png) ## KEY POINTS - Stock-bond correlation flipped from positive to negative around 2000 for six of seven G7 markets (earlier for Japan, barely for Italy), measured via DCC-GARCH and 3-year rolling windows. Pre-2000 correlations ranged from +0.02 (Japan) to +0.32 (UK); post-2000 they turned negative for most, reaching −0.24 for the US and Japan. - Correlation shows partial reversion toward positive around 2022-2023, coinciding with the inflation surge and rate hikes that moved stocks and bonds together again. - Real interest rates are the most stable driver of the correlation post-2000, with a positive coefficient: higher real rates push the correlation toward positive. Inflation also typically positive. Output growth mixed and unstable. - Bai-Perron structural breaks show the drivers themselves are time-varying. Breaks cluster around 1998-1999 (dotcom), mid-2000s (pre-crisis), and mid-2010s (post-debt crises). The factors that predict the correlation change in sign and significance across regimes. - Portfolio impact is large. US investors needed to shift from 50% stocks to 80% stocks after 2000 to maintain the same return and risk profile as a pre-2000 50/50 allocation. The 30-point shift compensates for the negative correlation no longer providing diversification. - Variance decomposition: covariance contribution to portfolio variance flipped from +23% (pre-2000) to −34% (post-2000) for the US, turning bonds from a drag into a hedge. Stock contribution increased, bond contribution stable. ## References - [Why does the correlation between stock and bond returns vary over time? - Andersson et al. (2008)](https://scholar.google.com/scholar?q=Why+does+the+correlation+between+stock+and+bond+returns+vary+over+time+Andersson&ref=aligrithm.com) - [Smooth transition patterns in the realized stock-bond correlation - Aslanidis & Christiansen (2012)](https://scholar.google.com/scholar?q=Smooth+transition+patterns+in+the+realized+stock-bond+correlation&ref=aligrithm.com) - [The determinants of stock and bond return comovements - Baele et al. (2010)](https://doi.org/10.1093/rfs/hhq014?ref=aligrithm.com) - [Estimating and testing linear models with multiple structural changes - Bai & Perron (1998)](https://doi.org/10.2307/2998540?ref=aligrithm.com) - [Macroeconomic drivers of bond and equity risks - Campbell et al. (2020)](https://doi.org/10.1086/707766?ref=aligrithm.com) - [Dynamic conditional correlation - Engle (2002)](https://doi.org/10.1198/073500102288618487?ref=aligrithm.com) - [Macroeconomic factors and the correlation of stock and bond returns - Li (2002)](https://scholar.google.com/scholar?q=Macroeconomic+factors+and+the+correlation+of+stock+and+bond+returns+Li+2002&ref=aligrithm.com) - [Stock-Bond Return Correlation: Understanding the Changing Behaviour - McMillan (2025)](https://papers.ssrn.com/sol3/papers.cfm?abstract%5Fid=5193234&ref=aligrithm.com) --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ### 5.50 Market Making With Competitors You Can't See URL: https://aligrithm.com/market-making-with-competitors-you-cant-see/ Last updated: 2026-10-16T08:09:16.000Z The Avellaneda-Stoikov framework gives you a clean optimization: post at some depth from the midprice, collect fills according to an exponential probability, penalize inventory quadratically, and done. The framework treats you as the only market maker. Every market order that does not fill you vanishes. That assumption is false in one way that has a real consequence: every unfilled order goes to someone else, and that someone adjusts their quotes accordingly. Boyce, Herdegen, and Sánchez-Betancourt (2024) patch this. They keep the A-S skeleton and add one extra state variable: the cumulative inventory of the competition. When a buy order arrives and fills the competitor instead of you, the competitor becomes one unit longer. Because the competitor's quotes depend on their inventory, that missed fill shifts their next ask, changes the gap between your depth and theirs, and degrades your fill rate from that moment forward. The fill you passed on is not neutral. It is a future liability. The paper derives an approximate closed-form solution through a matrix ordinary differential equation, shows it beats PPO trained for 500 million epochs with 99% confidence on a paired t-test, and documents that the reference market maker only needed to post more aggressively than the competition on 13 out of 10,000 simulated paths (0.13%). The math extends the old article ["The Three Pillars of Market Making: Fair Price, Spread, Skew"](https://aligrithm.com/the-three-pillars-of-market-making-fair-price-spread-skew/) directly: same objective function, same exponential fill model, two inventories instead of one. ## The competitor's rule of thumb The setup has one reference market maker who optimizes and one competing market maker who follows an inventory-linear heuristic. Poisson buy and sell orders arrive at intensities lambda^a and lambda^b. Each order fills either the reference MM or the competitor depending on who posts the tighter depth. The competitor uses an A-S-derived linear rule. Their ask and bid depths follow $$ \\tilde{\\delta}^a\_t = \\bar{a} - \\beta \\tilde{Q}\_t - Z\_t, \\qquad \\tilde{\\delta}^b\_t = \\bar{b} + \\beta \\tilde{Q}\_t + Z\_t $$ where tilde\_Q is the competitor's cumulative inventory, a\_bar and b\_bar are their base spread levels, beta controls how aggressively they skew quotes for inventory management, and Z\_t is an independent noise Brownian motion. The reference MM observes tilde\_Q and Z at each instant, so the competitor's posted depth is known now. What the reference MM cannot control is how tilde\_Q evolves next, because that depends on how many orders she lets through. Worked numbers using the paper's parameters (a\_bar = b\_bar = 0.1, beta = 0.05, no noise at Z = 0): - Competitor at tilde\_Q = +1 (net long by 1): ask depth = 0.1 - 0.05*1 = 0.05, bid depth = 0.1 + 0.05*1 = 0.15\. The competitor narrows their ask (wants to sell) and widens their bid (slow to buy more). - Competitor at tilde\_Q = -1 (net short by 1): ask depth = 0.1 + 0.05*1 = 0.15, bid depth = 0.1 - 0.05*1 = 0.05\. They widen their ask (not competing for buy fills) and tighten their bid (wants to buy to cover). The reference MM watches these shifts in real time. When the competitor tightens their ask, that is the cue to match or stay passive and take fewer fills. ## How missed fills feed back The departure from classical A-S is in the fill probability. The reference MM's ask fill rate depends on the gap between her own depth and the competitor's, not on the absolute depth: $$ \\Lambda^a = \\lambda^a \\min\\!\\left(\\exp\\!\\bigl(-\\kappa\\,(\\delta^a - \\tilde{\\delta}^a)\\bigr),\\; 1\\right) $$ and the bid side mirrors this. If delta^a equals tilde\_delta^a, fill rate is lambda^a (full arrival intensity). If delta^a is wider (less aggressive), fill rate decays exponentially at rate kappa. If delta^a is tighter (more aggressive), fill rate caps at lambda^a because every arriving market order hits you first. Worked example (kappa = 2, lambda^a = 10, competitor ask at tilde\_delta^a = 0.10): - Reference posts delta^a = 0.20 (10 bps wider than competition): Lambda^a = 10 \* exp(-2 \* (0.20 - 0.10)) = 10 \* exp(-0.20) = 10 \* 0.819 = 8.19 fills per unit time. - Reference posts delta^a = 0.30 (20 bps wider): Lambda^a = 10 \* exp(-2 \* (0.30 - 0.10)) = 10 \* exp(-0.40) = 10 \* 0.670 = 6.70 fills per unit time. - Reference posts delta^a = 0.05 (5 bps tighter, inside competition): Lambda^a = min(10 \* exp(-2 \* (0.05 - 0.10)), 10) = min(10 \* exp(0.10), 10) = min(11.05, 10) = 10.0\. Capped at full rate. The feedback loop works as follows. Suppose the reference MM posts wide and a buy order arrives. The competitor fills it. Competitor inventory rises by 1\. Their next ask depth shrinks by beta = 0.05\. The competition just got tighter on the ask, so the gap (delta^a - tilde\_delta^a) grows, and the reference MM's next fill rate falls. She must manage her own inventory risk against the cost of ceding orders that worsen the competitive environment she operates in. The goal functional uses the competitor's midprice for terminal inventory liquidation: $$ J(\\delta) = \\mathbb{E}\\!\\left\[X\_T + Q\_T\\!\\left(S\_T + \\frac{\\bar{a}-\\bar{b}}{2} - \\beta\\tilde{Q}\_T - Z\_T\\right) - \\gamma Q\_T^2 - \\phi\\!\\int\_0^T Q\_r^2\\,dr\\right\] $$ The term inside the parenthesis is the competitor's midprice. If the competitor holds large long inventory (tilde\_Q large), their mid is below S, which marks the reference MM's terminal inventory at a lower price and increases the effective terminal penalty. The two inventories are coupled not just through fill rates but through the terminal condition. ## The closed-form matrix ODE The value function depends on t, Q, tilde\_Q, Z, and S. The S and X dependence separates cleanly via the linear-quadratic ansatz. What remains is a function g(t, q) that satisfies a nonlinear ODE system over the inventory grid from q\_underline to q\_bar. The substitution omega(t, q) = exp(kappa \* g(t, q)) converts that nonlinear system into a linear one -- the same exponential change of variables that Guéant, Lehalle, and Fernandez-Tapia (2013) used for the single-MM problem. The linear system is then a matrix ODE whose solution is a matrix exponential. Theorem 3.1 defines the tridiagonal matrix A with entries: $$ A\_{q,q} = -\\phi\\kappa q^2 + \\beta\\kappa(\\lambda^a - \\lambda^b)q $$$$ A\_{q,q-1} = \\lambda^a \\exp\\!\\bigl(-1 - \\kappa(\\tfrac{\\beta}{2} - \\bar{a})\\bigr), \\qquad A\_{q,q+1} = \\lambda^b \\exp\\!\\bigl(-1 - \\kappa(\\tfrac{\\beta}{2} - \\bar{b})\\bigr) $$ The terminal vector v has entries v\_q = exp(kappa \* ((a\_bar - b\_bar)/2 \* q - (gamma - beta/2) \* q^2)). The solution is then: $$ \\omega(t,\\, q) = \\bigl(\\exp(\\mathbf{A}(T - t))\\;\\mathbf{v}\\bigr)\_q $$ Worked reading with the paper's parameters (symmetric case, lambda^a = lambda^b = 10, kappa = 2, a\_bar = b\_bar = 0.1, beta = 0.05, phi = 0.1, gamma = 0.03): Diagonal entries of A at q = 0: -phi \* kappa \* 0 + beta \* kappa \* (10-10) \* 0 = 0\. At q = 1: -0.1 \* 2 \* 1 + 0.05 \* 2 \* 0 \* 1 = -0.2\. Off-diagonal entries: lambda^a \* exp(-1 - 2\*(0.025 - 0.1)) = 10 \* exp(-1 + 0.15) = 10 \* exp(-0.85) = 10 \* 0.427 = 4.27\. A is a (21 x 21) tridiagonal matrix (inventories -10 through +10). The matrix exponential runs in milliseconds and needs computing only once per market session unless parameters drift. The optimal depths follow from log-ratios of adjacent omega values: $$ \\hat{\\delta}^a = \\frac{\\beta}{2} + \\frac{1}{\\kappa}\\!\\left(1 + \\log\\frac{\\omega(t,q)}{\\omega(t,q-1)}\\right) - \\beta\\tilde{q} - z $$$$ \\hat{\\delta}^b = \\frac{\\beta}{2} + \\frac{1}{\\kappa}\\!\\left(1 + \\log\\frac{\\omega(t,q)}{\\omega(t,q+1)}\\right) + \\beta\\tilde{q} + z $$ The -beta \* tilde\_q - z term in the ask depth is the competitor inventory adjustment. When tilde\_q = 1 (competitor is net long), ask depth reduces by 0.05 to match the competitor's own tighter ask. When tilde\_q = -1 (competitor is net short), ask depth rises by 0.05 because the competitor is not competing hard on the ask side and there is no need to tighten further. The actual posted depth clips the unconstrained optimum at the competitor's level: delta^{\*,a} = max(hat\_delta^a, competitor's depth). Taking the maximum of the two depths means: if the unconstrained optimal is already wider than the competitor, post at the unconstrained optimal and take whatever fill rate that gives. If the unconstrained optimal happens to go below the competitor's depth (which forces you to top the book), post exactly at the competitor's level and fill every arriving order. In practice this happens rarely. ## What the numbers show The paper runs 10,000 simulations with 1,000 timesteps across T = 1 under symmetric parameters. Three strategies compete head to head: the closed-form from Theorem 3.1, a numerical Euler-scheme approximation of the exact value function, and PPO trained over 500 million epochs. ![Optimal ask depths (solid lines) and bid depths (dashed lines) at t = 0.5, plotted as a function of competitor inventory on the x-axis, with reference MM inventory shown in the color bar from -9 (purple) to +9 (yellow). Red lines mark the competitor's own depths; the reference MM almost always posts wider.](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/08/article_459-depth_vs_competitor_inventory.png) The depth surface shows the two-inventory structure directly. Ask depths slope downward as the competitor's inventory rises (tighter competition for buy-order fills) and bid depths slope upward. Each colored band is a different reference MM inventory. The purple bands (reference MM net short) push the ask tighter and the blue-green bands (reference MM net long) push the bid tighter, consistent with A-S inventory skew. The red lines mark the competitor's own depths: almost every colored line sits above (wider than) the red line, meaning the reference MM almost never tops the book. ![Bar chart comparing mean performance criterion with one-standard-deviation error bars for closed-form approximation (3.64), Euler numerical scheme (3.66), and PPO reinforcement learning (3.14) across 10,000 simulations](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/08/article_459-strategy_comparison.png) Numbers: closed-form 3.64 (standard deviation 2.57), Euler 3.66 (2.56), PPO 3.14 (2.76). The 0.55% shortfall from the closed-form to the Euler exact solution is significant at 99% confidence by a paired t-test. PPO trails the closed-form by 0.50 units, about 14% below the Euler benchmark, despite 500 million training epochs. The authors flag that PPO hyperparameters and architecture choices may explain part of the gap. As-is, the analytic approach costs essentially nothing to compute and dominates. The 0.13% figure is the most memorable output: in 10,000 simulated paths, only 13 ever required the reference MM to clip to the competitor's depth (i.e., the constraint in equation (3.7) bound). In the other 9,987 paths, the unconstrained optimal sat wider than the competition at every moment, and the reference MM never needed to top the book. The optimal strategy under symmetric arrival rates is to follow the competition's depth rather than race to beat it. The old article ["DRL Market Making With a Periodic Signal and Real Latency"](https://aligrithm.com/drl-market-making-with-a-periodic-signal-and-real-latency/) showed that DRL wins over A-S when signals and realistic latency enter the picture. This paper shows the reverse in a signal-free two-inventory setting: the structured analytic solution dominates RL, and the structure costs a single matrix exponential per session. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/08/article_459-visual-0.png) ## KEY POINTS - Classical A-S treats every missed fill as a neutral event. This paper tracks missed fills explicitly: orders not filled by the reference MM go to the competitor, raise the competitor's inventory, shift their quotes, and degrade the reference MM's future fill rate. - The competitor follows a linear inventory rule (A-S-optimal form): their ask shrinks and bid grows when they hold net long inventory (beta = 0.05 per unit in the paper's parameterization). The reference MM observes this state and adjusts quotes accordingly. - Fill probability depends on the gap between the reference MM's depth and the competitor's depth, not on the absolute depth. A 10 bps wider post with kappa = 2 and lambda^a = 10 reduces the fill rate from 10 to 8.19 per unit time. - The closed-form uses a matrix exponential: omega(q, t) = (exp(A\*(T-t)) \* v)\_q on a (q\_bar - q\_underline + 1)-dimensional grid. The approximation holds when the unconstrained optimal depth stays at or above the competitor's depth, which holds for the paper's symmetric parameters. - Performance: closed-form 3.64 beats PPO 3.14 by 14%, with 99% confidence. The Euler exact solution (3.66) sits only 0.55% above the closed-form, a statistically significant but economically small gap. - The reference MM topped the book in only 13 of 10,000 simulated paths (0.13%). The optimal behavior under symmetric order flow is to follow the competitor's depth rather than undercut it. Competition creates a natural inventory-sharing equilibrium. ## References - Avellaneda, M. and Stoikov, S. (2008). [High-frequency trading in a limit order book](https://doi.org/10.1080/14697680701381228?ref=aligrithm.com). *Quantitative Finance*, 8(3), 217–224. - Guéant, O., Lehalle, C.-A. and Fernandez-Tapia, J. (2013). [Dealing with the inventory risk: a solution to the market making problem](https://doi.org/10.1007/s11579-012-0087-0?ref=aligrithm.com). *Mathematics and Financial Economics*, 7(4), 477–507. - Chordia, T., Roll, R. and Subrahmanyam, A. (2002). [Order imbalance, liquidity, and market returns](https://doi.org/10.1016/S0304-405X%2802%2900136-8?ref=aligrithm.com). *Journal of Financial Economics*, 65(1), 111–130. - Boyce, R., Herdegen, M. and Sánchez-Betancourt, L. (2024). [Market Making with Exogenous Competition](https://arxiv.org/abs/2407.17393?ref=aligrithm.com). arXiv:2407.17393. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ### 5.49 The Worst-Kept Secret: Imbalance Is the MM's Optimal Response, Not Alpha URL: https://aligrithm.com/the-worst-kept-secret-imbalance-is-the-mms-optimal-response-not-alpha/ Last updated: 2026-10-16T08:09:17.000Z Sasha Stoikov called it the "worst-kept secret of high-frequency trading" in a 2014 seminar: the volume imbalance in a limit order book predicts the next mid-price move. Define it as I = (q^b − q^a)/(q^b + q^a), where q^b is the size resting at the best bid and q^a the size at the best ask. Push I toward 1 and the next tick tends to be up. Push it toward −1 and the next tick tends to be down. Huang, Lehalle and Rosenbaum documented it in 2015, Lehalle and Mounjid confirmed it in 2017, Sfendourakis and Muni Toke again in 2023\. Every writeup since treats the correlation as a free signal: watch the depth, trade the flip. Pulido, Rosenbaum and Sfendourakis build a model that removes the mystery entirely. Their market maker already knows where the price is going. The imbalance sitting in her book is not information leaking out of someone else's trade. It is the volume she chose to post, given what she knows, to maximize her own expected utility. Run her optimization and the predictive imbalance falls out as a byproduct, not an assumption. You are not detecting an informed trader's footprint. You are reading a solved control problem. ## A market maker who already knows the answer Strip the setup to its parts. There is an efficient price S, the "true" value nobody quotes directly, following a Bachelier random walk: dS\_t = σ dW\_t, plain Brownian motion with volatility σ. There is a mid-price P that sits on a tick grid and only moves in discrete half-tick jumps, governed by the model with uncertainty zones from the old article's neighboring literature (Robert and Rosenbaum, 2011): when S drifts far enough from P, P snaps to catch up. And there is a single market maker who observes S directly, which nobody trading against her does. She is the platform's only liquidity provider. Because the stock is large-tick, the spread is stuck at one tick almost always, so she cannot move the price. Her only lever is size: how much she posts at the best bid (q^b) and the best ask (q^a). Market orders arrive against those quotes as two independent Poisson processes, and their arrival rate depends on Y := S − P, the gap between the efficient price and the tradeable one. When Y is large and positive, buy orders arrive faster because market takers see the same drift she does. She does not control that arrival rate or its size. Market orders show up whether or not she is quoting, which rules out price manipulation by construction: there is no channel through which her quotes change the tape's behavior, only how much of that behavior she captures. $$ I = \\frac{q^b - q^a}{q^b + q^a} $$ Worked example: she posts 30 units at the bid and 10 at the ask. I = (30 − 10)/(30 + 10) = 0.5\. An outside observer sees a book tilted toward buying pressure and, per the empirical studies above, expects the next move to be up. In this model that expectation is correct, but not because 20 extra units of buy-side size are "information." They are her answer to a question she already knows the answer to. ## An efficient price nobody else can see The jump rule is concrete enough to check with numbers. Take the paper's own example parameters: tick size δ = 0.01, uncertainty-zone parameter η = 0.2\. The mid-price jumps up by half a tick the instant S climbs to P + δ(η + 1/2). Plug in: 0.01 × (0.2 + 0.5) = 0.01 × 0.7 = 0.007\. So the efficient price has to run 0.7 of a full tick past the current mid-price before the quoted price catches up. Below that, P sits still, S wanders, and the gap Y = S − P widens and narrows on its own. ![Sample path of the efficient price S (top, blue) against the sticky mid-price P (top, orange steps) with uncertainty-zone barriers in red, and the resulting gap Y = S minus P (bottom)](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/08/article_458-efficient_price_vs_sticky_midprice.png) Read the top panel first: S wanders continuously while P only steps when S has drifted far enough away, exactly the δ(η + 1/2) = 0.007 rule above. The bottom panel is Y itself, oscillating inside a band and occasionally testing the red barrier lines, at which point P catches up and Y snaps back toward zero because S is continuous but P just jumped by half a tick. η itself is not a free parameter you have to guess: Robert and Rosenbaum show it can be estimated from tick data as η̂ = N^c/(2N^a), the count of price-move continuations over twice the count of alternations, so this whole barrier distance is checkable against real quote data, not fitted to make the chart look nice. ## What one fill is worth The market maker's actual decision reduces to comparing two numbers every instant: what she nets if a sell order hits her bid, versus what she nets if a buy order lifts her ask. From the model's profit-and-loss decomposition, filling one unit on the ask side pays δ/2 − Y, and filling one unit on the bid side pays δ/2 + Y. $$ \\text{Ask fill edge} = \\frac{\\delta}{2} - Y, \\qquad \\text{Bid fill edge} = \\frac{\\delta}{2} + Y $$ Worked example, continuing with δ = 0.01 so δ/2 = 0.005: say Y = 0.002, meaning the efficient price sits 0.2 of a tick above the mid. Selling one unit at the ask nets 0.005 − 0.002 = 0.003\. Buying one unit at the bid nets 0.005 + 0.002 = 0.007\. Buying is more than twice as attractive as selling at this instant, because selling now hands away a unit at a price that is about to look cheap once S catches up, while buying locks in a discount. That asymmetry is the entire mechanism. She responds by posting less on the ask and more on the bid, which mechanically raises I. Nobody told her the imbalance formula; it falls out of comparing two fill payoffs. ## Risk aversion caps the response, cleverness doesn't If fill edges were the only consideration, she would post unlimited size on whichever side pays more and let inventory run wherever it wants. She doesn't, because she maximizes the exponential utility of her terminal profit with a risk-aversion coefficient γ, not the expected profit itself. Holding a large inventory Q exposes her to the swings of σ dW\_t on every unit she carries, and exponential utility punishes that variance directly. The Hamilton-Jacobi-Bellman equation that comes out of her optimization (equation 2 in the paper) carries a running term (σ²γ²Q²)/2 that acts as a continuous inventory penalty, independent of any single trade's edge. $$ \\text{Inventory risk penalty rate} = \\frac{\\sigma^2 \\gamma^2 Q^2}{2} $$ Worked example: take σ = 0.005 and Q = 15 units, the same values used in the paper's own sensitivity charts. At γ = 0.5, the penalty rate is (0.005² × 0.5² × 15²)/2 = (0.000025 × 0.25 × 225)/2 ≈ 0.0007, about a fourteenth of a tick. Quadruple the risk aversion to γ = 2 and the rate jumps to (0.000025 × 4 × 225)/2 ≈ 0.01125, more than a full tick. Because the penalty scales with γ squared while a trade's edge only scales linearly with γ through the exponential's argument, doubling risk aversion doesn't double the caution, it roughly quadruples it. That is why the paper's own numerical experiments show a more risk-averse market maker pulling her quoted volumes back toward zero inventory well before her physical limit, and why the imbalance she posts depends on Q as well as on Y: a market maker sitting on inventory of −15 needs a much smaller Y before she's willing to buy aggressively than one sitting on +15. ## The dial: quoted imbalance as a readout of S minus P Solve the full control problem numerically and plot the resulting imbalance against Y for a few fixed inventory levels, and you get a monotone staircase from −1 to 1. ![Optimal quoted imbalance as a function of Y = S minus P, for inventory levels Q from -15 to +15, across three parameter configurations](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/08/article_458-quoted_imbalance_monotone_in_y.png) Each panel fixes a different combination of risk aversion, volatility and order-flow intensity; within a panel, each line is one inventory level. Read the leftmost panel at Y = 0, meaning the efficient price sits exactly on the mid: the Q = −15 line (she is short 15 units and needs to buy back) is already pinned near an imbalance of +1, quoting almost nothing on the ask and heavily on the bid. The Q = +15 line (she is long 15 and needs to sell) is still pinned near −1 at that same Y = 0 and only starts climbing once Y has moved a noticeable distance toward positive territory. Being long and being short shift the entire staircase sideways along the Y axis; a long inventory needs a stronger bullish signal before she'll flip to aggressive buying, exactly as the risk-penalty math above predicts. One inconsistency worth flagging rather than quietly fixing: the paper's introduction defines volume imbalance as I = (q^b − q^a)/(q^b + q^a), the standard convention this article uses throughout, and states correctly that a higher I predicts a higher next-period price. But the paragraph introducing Figure 4 in Section 5.1 defines the plotted quantity as (q^a − q^b)/(q^a + q^b), the opposite sign. Since the same paragraph and Figure 3 both establish that q^a is decreasing and q^b is increasing in Y, that reversed formula would produce a curve falling from 1 to −1 as Y rises, not the rising staircase the paper plots and describes. The chart and the surrounding prose only make sense under the introduction's original sign convention; the formula printed in Section 5.1 looks like a transcription slip in the source, and this article keeps the source's plotted result (imbalance rising with Y) rather than silently rewriting the paper's formula for it. ## A free byproduct: sizing the tick Because the model already computes how often market orders hit each side, it also answers a question regulators and exchanges care about: what tick size maximizes traded volume? Too small a tick and each fill barely pays the market maker anything, so she trades less; too large a tick and her posted price sits too far from the efficient price for market orders to bother crossing it. Running the platform's expected-volume objective across a grid of tick sizes at σ = 0.005 (using the empirical relation η = η₀√(δ₀/δ) from Dayri and Rosenbaum, with η₀ = 0.2 and δ₀ = 0.1 as anchors), the paper finds an optimum at δ = 0.0032\. Push volatility higher and the optimal tick grows with it: at σ = 0.0075 it is 0.0044, at σ = 0.01 it is 0.0064, and at σ = 0.015 it is 0.015\. A more volatile stock needs a fatter tick so the market maker earns enough per fill to keep quoting through the extra risk. This is one derived application, not the paper's main claim, but it shows the same coupled HJB system pays for itself twice: once to explain imbalance, once to size the venue. Treat all of this as a caution against any strategy built on "buy when imbalance spikes." The old article ["Order Book Imbalance: The First Microstructure Feature to Test"](https://aligrithm.com/order-book-imbalance-the-first-microstructure-feature-to-test/) tested the correlation empirically and found it real; this model explains why the correlation exists without needing any informed trader on the other side of the book at all. A rational, uninformed-about-your-order market maker generates a predictive imbalance purely as a side effect of managing her own inventory risk against a price she can see and you cannot. A retail signal built on it captures a mechanical echo of someone else's risk aversion, not leaked information. The three-pillar quoting framework from the old article ["The Three Pillars of Market Making: Fair Price, Spread, Skew"](https://aligrithm.com/the-three-pillars-of-market-making-fair-price-spread-skew/) already treats skew as an inventory-management tool; this paper is the formal proof that the same skew, observed from outside, is indistinguishable from a forecasting signal. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/08/article_458-visual-0.png) ## KEY POINTS - Volume imbalance I = (q^b − q^a)/(q^b + q^a) is treated across the microstructure literature as a predictive signal; this model shows it is instead the optimal quoting response of a market maker who already observes the efficient price and cannot be gamed by market orders she does not control. - The efficient price S follows a Bachelier walk (dS = σ dW) while the mid-price P only jumps by half a tick once S drifts δ(η + 1/2) away from it; with δ = 0.01 and η = 0.2, that barrier is 0.007, checkable against real data since η itself is estimable as N^c/(2N^a). - Each fill has a clean payoff: δ/2 − Y on the ask side, δ/2 + Y on the bid side. When Y > 0, buying pays more than selling, so the market maker posts more bid size than ask size, mechanically raising I. - Risk aversion, not signal cleverness, caps her inventory: the HJB equation's penalty term (σ²γ²Q²)/2 grows with the square of γ, so quadrupling risk aversion multiplies the inventory cost sixteenfold and pulls her book back toward zero well before her hard limit. - Solved numerically, the optimal imbalance is a monotone staircase from −1 to 1 in Y = S − P, shifted sideways by current inventory; a short position needs less bullish evidence to trigger aggressive buying than a long position does. - The paper's own Section 5.1 prints the plotted imbalance formula with a flipped sign relative to its introduction and relative to what the chart actually shows; this article keeps the plotted result and flags the formula as likely mistranscribed rather than silently correcting it. - As a free byproduct, the same coupled HJB system picks an optimal tick size: 0.0032 at σ = 0.005, rising to 0.015 at σ = 0.015, since a more volatile stock needs a fatter tick to keep the market maker's per-fill edge worth the risk. ## References - [High-Frequency Trading in a Limit Order Book - Avellaneda and Stoikov (2008)](https://scholar.google.com/scholar?q=%22High-Frequency+Trading+in+a+Limit+Order+Book%22+Avellaneda+Stoikov+2008&ref=aligrithm.com) - [A New Approach for the Dynamics of Ultra-High-Frequency Data: The Model with Uncertainty Zones - Robert and Rosenbaum (2011)](https://scholar.google.com/scholar?q=%22A+New+Approach+for+the+Dynamics+of+Ultra-High-Frequency+Data%22+Robert+Rosenbaum+2011&ref=aligrithm.com) - [The Micro-Price: A High-Frequency Estimator of Future Prices - Stoikov (2018)](https://scholar.google.com/scholar?q=%22The+Micro-Price%3A+A+High-Frequency+Estimator+of+Future+Prices%22+Stoikov+2018&ref=aligrithm.com) - [Simulating and Analyzing Order Book Data: The Queue-Reactive Model - Huang, Lehalle and Rosenbaum (2015)](https://scholar.google.com/scholar?q=%22Simulating+and+Analyzing+Order+Book+Data%3A+The+Queue-Reactive+Model%22+Huang+Lehalle+Rosenbaum+2015&ref=aligrithm.com) - [On Bid and Ask Side-Specific Tick Sizes - Baldacci, Bergault, Derchu and Rosenbaum (2023)](https://scholar.google.com/scholar?q=%22On+Bid+and+Ask+Side-Specific+Tick+Sizes%22+Baldacci+Bergault+Derchu+Rosenbaum+2023&ref=aligrithm.com) - [Understanding the Worst-Kept Secret of High-Frequency Trading - Pulido, Rosenbaum and Sfendourakis (2024)](https://arxiv.org/abs/2307.15599?ref=aligrithm.com) --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ### 5.52 Crypto Isn't Structurally Alien: Roll/VPIN/Amihud Predict Distribution Shifts URL: https://aligrithm.com/crypto-isnt-structurally-alien-roll-vpin-amihud-predict-distribution-shifts/ Last updated: 2026-10-16T08:09:18.000Z Easley, O'Hara, Yang and Zhang take five textbook microstructure measures, compute them on Binance one-minute bars for BTC, ETH, XRP, SOL and ADA over January 2021 to July 2023, and ask a random forest to call the sign of tomorrow's change in the return distribution. Two numbers carry the paper. Crypto VPIN averages 0.47 against the 0.22 that the same authors found in E-mini and crude futures, which they read as heavier toxicity. And accuracy on the sign of the change in realized volatility runs 0.56 to 0.58, well above the 0.50 a coin flip gives you. I reproduced the machinery behind both numbers and neither one means what the paper says it means. The VPIN gap is an artifact of the estimator, and the volatility accuracy leans on a feature that is mostly a volatility measure wearing a spread estimator's name. One finding survives both objections, and the abstract buries it. The five features are the Roll measure, the Roll impact measure, Kyle's lambda, the Amihud illiquidity ratio and VPIN, each computed for each of the five coins over a trailing window W of 50 or 100 one-minute bars. Each coin's model therefore sees 25 features: its own microstructure and the other four coins'. The labels are the sign of the change, 1,500 bars ahead, in realized volatility, autocorrelation, skewness, kurtosis and the Jarque-Bera statistic. The choice of 1,500 bars, roughly a trading day, exists to stop the label at bar t and the label at bar t+1 from sharing data. Hold on to that number, because it decides how much evidence the paper has. ## The Roll measure is a volatility measure with extra steps Roll's 1984 estimator recovers an effective spread from the bid-ask bounce: with a constant spread and no drift, consecutive price changes have a negative autocovariance equal to minus the squared half-spread. The paper drops the sign and takes an absolute value. $$ \\text{Roll}\_t \\;=\\; 2\\sqrt{\\bigl|\\operatorname{cov}(\\Delta P\_t,\\; \\Delta P\_{t-1})\\bigr|}, \\qquad \\Delta p\_t = p\_t - p\_{t-1} $$ $$ \\text{if } \\Delta p \\text{ is AR(1) with volatility } \\sigma \\text{ and autocorrelation } \\rho: \\qquad \\text{Roll}\_t \\;=\\; 2\\,\\sigma\\,\\sqrt{|\\rho|} $$ The first line is the paper's equation, taking the covariance between the vector of price changes over the window and the same vector lagged one bar. The second line is what that expression collapses to when price changes follow a first-order autoregression: the covariance is rho times sigma squared, so the square root pulls out sigma times the square root of the absolute autocorrelation. The absolute value is what makes it usable at all, since crypto minute bars often show positive autocovariance and the original estimator would demand a square root of a negative number. It also destroys the interpretation. A momentum-driven positive autocovariance and a bounce-driven negative one of equal size print the identical "spread". Read the elasticities off the second line. Roll is linear in sigma and square-root in the absolute autocorrelation, so doubling volatility doubles Roll while doubling autocorrelation raises it by 41%. Worked example on a BTC window where minute price changes have a standard deviation of 2 dollars and an autocorrelation of 0.05: Roll equals 2 times 2 times the square root of 0.05, which is 0.894 dollars. Push the autocorrelation to 0.10 and Roll goes to 1.265, up 41%. Leave the autocorrelation at 0.05 and push volatility to 4 dollars instead, and Roll goes to 1.789, up 100%. The feature answers to volatility twice as hard as it answers to the autocorrelation it is supposed to be measuring. ![Two scatter plots from simulated AR(1) price changes over 50-bar windows: the Roll measure against per-bar volatility with correlation 0.71, and the Roll measure against absolute autocorrelation with correlation 0.36](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/08/article_461-roll_is_a_vol_proxy.png) I simulated 800 windows of 50 AR(1) price changes, drawing sigma uniformly between 0.5 and 5 and rho uniformly between minus 0.45 and 0.45, then computed the paper's Roll measure on each. Correlation with sigma comes out at 0.71, correlation with the absolute autocorrelation at 0.36\. Those exact levels depend on the ranges I drew from, so treat the ratio rather than the levels as the result: Roll loads on volatility about twice as hard as on autocorrelation, which is the elasticity the algebra predicted. Now put that next to the paper's own feature-importance chart for the label it predicts best. ![Bar chart of mean decreased accuracy for own-coin features predicting the sign of the change in realized volatility, aggregated over five cryptocurrencies at a 50-bar window: Roll Measure about 0.06, Roll Impact and VPIN about 0.009, Kyle's Lambda 0.003, Amihud near zero](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/08/article_461-mda_own_features_realized_vol.png) The Roll measure scores a mean decreased accuracy of 0.058 for predicting the sign of the change in realized volatility. Every other feature sits under 0.01, and Amihud registers 0.001\. So the forest predicts future volatility mostly from one feature, and that feature is a scaled estimate of current volatility. Volatility clusters. The paper's strongest result is a well-behaved model rediscovering volatility persistence through a proxy, and the conclusion that the Roll measure "provides a microstructure foundation for the momentum observed in crypto prices" does not follow from an importance score on a statistic that tracks sigma first and rho second. The old article ["Bid-Ask Spread From OHLC: The GMM Estimator That Beats Roll/CS"](https://aligrithm.com/bid-ask-spread-from-ohlc-the-gmm-estimator-that-beats-roll-cs/) is the relevant comparison. That work keeps the spread interpretation intact by combining several OHLC-based estimators under GMM, pushing minute-level correlation with a high-frequency benchmark from 56% to 89% and cutting non-positive estimates from 34% of observations to 0.02%. The absolute-value patch here takes the opposite route: it guarantees a positive number by throwing away the sign that made the number a spread. ## VPIN's null value is 0.50, and Binance prints 0.47 VPIN splits each bar's volume into buyer-initiated and seller-initiated parts using bulk volume classification, then averages the absolute imbalance. Write out the paper's version and the volume cancels. $$ \\text{VPIN}\_t=\\frac{1}{W}\\sum\_{i=t-W+1}^{t}\\frac{\\bigl|V\_i^{S}-V\_i^{B}\\bigr|}{V\_i}, \\qquad V\_i^{B}=V\_i\\,Z\\!\\left(\\frac{\\Delta p\_i}{\\sigma\_{\\Delta p}}\\right), \\qquad V\_i^{S}=V\_i-V\_i^{B} $$ $$ \\Longrightarrow\\qquad \\text{VPIN}\_t=\\frac{1}{W}\\sum\_{i=t-W+1}^{t}\\Bigl|\\,1-2\\,Z\\!\\left(\\tfrac{\\Delta p\_i}{\\sigma\_{\\Delta p}}\\right)\\Bigr| $$ Substitute the buy volume into the numerator and every V cancels: the buy side is V times Z, the sell side is V times one minus Z, their difference is V times the quantity two Z minus one, and dividing by V leaves the absolute value of one minus twice Z. The measure contains no volume information at all. It is an average of standardized one-minute price changes pushed through the normal CDF, which is worth knowing before anyone builds a toxicity dashboard on it. Worked example over two bars. The first bar moves plus 0.8 standard deviations, so Z equals 0.7881 and the term is the absolute value of one minus 1.5762, which is 0.5762\. The second bar moves minus 0.1 standard deviations, so Z equals 0.4602 and the term is 0.0797\. The two-bar VPIN is 0.328\. Now do it in general. Under a pure random walk with Gaussian price changes, Z of a standard normal draw is uniform on the unit interval, so the absolute value of one minus twice a uniform is itself uniform on the unit interval, and its expected value is exactly 0.500\. Zero information, no toxicity, no informed trader anywhere: the paper's VPIN still reads 0.50. ![Left panel: histogram of the per-bar order-flow imbalance term under Gaussian price changes, flat with mean 0.500, and under t(3) price changes, mean 0.392. Right panel: mean VPIN against Student-t degrees of freedom rising toward the Gaussian null of 0.500, crossing the shaded band of Binance VPINs reported between 0.457 and 0.471 around 5 to 7 degrees of freedom](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/08/article_461-vpin_null_is_half.png) Fat tails push it down rather than up. When price changes are leptokurtic, standardizing by the window's own standard deviation packs most bars near zero and Z clusters near one half, so the imbalance term shrinks. Simulating 400,000 draws, Gaussian price changes give a mean of 0.4996 and Student-t with 3 degrees of freedom gives 0.392\. The Binance VPINs in Table 1 run from 0.4566 for XRP at 100 bars to 0.4705 for ETH at 50 bars, averaging 0.4678 across the five coins at 50 bars. That band lands exactly where pure noise with a tail index around 5 to 7 degrees of freedom lands. Crypto minute returns have tails in that range. The 0.22 for E-mini and crude comes from Easley, Lopez de Prado and O'Hara's 2012 construction, which aggregates buy and sell volume across a whole equal-volume bucket before taking the absolute difference. Summing signed imbalances first lets them cancel; taking absolute values bar by bar first, as this paper does, forbids cancellation. Jensen's inequality guarantees the bar-by-bar version prints higher, on any data, informed or not. The paper concedes "a slightly different methodology" in a single clause and then treats 0.47 against 0.22 as evidence of greater information-based trading in crypto. Two estimators with different null values are not a comparison. ## The sample holds 904 observations, not 1.4 million The paper reports no standard errors on any accuracy figure. It predicts 1,500 bars ahead and it makes a prediction every minute, so consecutive labels share 1,499 of 1,500 bars of look-ahead. Independent observations arrive once every 1,500 minutes, not once a minute. $$ N\_{\\text{eff}}=\\frac{T}{h}, \\qquad \\mathrm{SE}=\\sqrt{\\frac{0.25}{N\_{\\text{eff}}}}, \\qquad z=\\frac{\\widehat{\\mathrm{acc}}-0.5}{\\mathrm{SE}} $$ T is the number of one-minute bars in the sample and h is the 1,500-bar look-ahead, so their ratio counts non-overlapping blocks. The standard error uses the binomial variance of a fair coin, one quarter, because the null being tested is accuracy of 0.50\. Worked example: January 2021 through July 2023 spans 942 days, or 1,356,480 minute bars, which divided by 1,500 gives 904 independent blocks. The standard error is the square root of 0.25 over 904, or 0.0166\. Realized volatility at 0.5816 sits (0.5816 minus 0.5) over 0.0166, a z of 4.91\. Autocorrelation at 0.5317 gives a z of 1.91. ![Bar chart of z-statistics on reported 50-bar accuracy against 904 non-overlapping observations: realized volatility 4.91, kurtosis 2.66, Jarque-Bera 1.98, autocorrelation 1.91, skewness 0.00, with reference lines at 1.96 and 3.29](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/08/article_461-accuracy_tstats.png) Realized volatility clears any bar you set. Kurtosis clears 5% and nothing else. Jarque-Bera and autocorrelation sit on the 1.96 line, and skewness is a coin flip by the paper's own admission. Apply the multiple-testing tax and the picture tightens: the paper runs five labels across five coins at two window lengths, which is 50 cells before you count the crypto-winter split, and a Bonferroni threshold at 5% over 50 tests sits at a z of 3.29\. One label survives. That calculation is also generous, because it credits the model with the full sample when only a held-out test slice produced those accuracies, and the paper never states the split. Shrink the test set to a third and every z falls by a factor of 1.7, leaving realized volatility at 2.8 and nothing else standing. Two more things follow from the same arithmetic. The crypto-winter robustness test compares aggregated accuracy of 0.5379 before November 2021 against 0.5395 after, and calls the 0.0016 gap "remarkable stability". The two subperiods hold 301 and 603 independent blocks, so the standard error on their difference is 0.035 and the test cannot see a true gap smaller than about 7 accuracy points. Stability that loose is compatible with the market changing a great deal. And the logistic-regression check in Figure 10 reports realized-volatility accuracy of 0.61 against the forest's 0.58\. A linear model beat 100 trees on 25 features, which is what you expect when a single feature carries the signal. ## Kyle's lambda and Amihud never worked here, and the definitions say why $$ \\lambda\_t=\\frac{p\_t-p\_{t-W}}{\\sum\_{i=t-W}^{t} b\_i V\_i}, \\qquad b\_i=\\operatorname{sign}(p\_i-p\_{i-1}), \\qquad \\text{Amihud}\_t=\\frac{1}{W}\\sum\_{i=t-W+1}^{t}\\frac{|r\_i|}{p\_i V\_i} $$ Kyle's lambda divides a price change over the window by the net signed volume over the same window. Over 50 bars of two-sided flow that denominator is a near-cancelling sum, so lambda measures the reciprocal of net imbalance rather than price impact. Worked example: 500 million dollars of gross volume over the window with a 1% net imbalance gives a 5 million dollar denominator, so a 30 dollar BTC move prints a lambda of 6 times ten to the minus six. Same 30 dollar move, same gross volume, but a 0.01% net imbalance, and the denominator is 50,000 dollars and lambda prints 6 times ten to the minus four, a hundredfold jump from a quantity that has nothing to do with liquidity. The paper's own Tables 1, 9 and 10 show the damage: BTC's 50-bar lambda reads 42.22 before November 2021, 0.391 after, and 15.99 over the full sample, while the 100-bar version of the same statistic reads 1.26\. A twelvefold gap between a 50-bar and a 100-bar estimate of the same quantity is not a market changing, it is an estimator failing. Kyle's lambda earns a near-zero mean decreased accuracy in every table, and that is the model correctly ignoring noise. The Amihud definition has a separate problem. Section II defines V\_t as the dollar volume in bar t, yet Amihud divides by p\_i times V\_i and Roll impact divides by p\_t times V\_t. Under the stated definition both expressions count price twice and neither is the Amihud ratio. Read V as share volume and all three formulas become the standard ones while the prose becomes wrong. I keep the printed form and flag the inconsistency rather than pick a reading for the authors, but note which way it cuts: Amihud scores near-zero importance everywhere, including a negative 0.0009 for realized volatility before crypto winter, and a mis-specified denominator is a plausible reason. ## What survives The cross-coin ranking survives all of it. Each coin's own Roll measure dominates its own predictions, and after that the Roll measures of BTC and ETH outrank every other coin's features. ![Two bar charts of mean decreased accuracy for XRP labels using all 25 features, with rollmeasure_XRP largest by a wide margin, VPIN_XRP second for kurtosis, and rollmeasure_BTC and rollmeasure_ETH the only cross-coin features with visible importance](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/08/article_461-mda_xrp_all_25_features.png) For XRP kurtosis, the own Roll measure scores 0.030, own VPIN 0.025, and then BTC's Roll measure at 0.005 and ETH's at 0.004 lift clear of everything else, while SOL and ADA features sit at zero. That pattern repeats across all five coins. Since the Roll measure is largely a volatility proxy, the honest reading is that BTC and ETH volatility leads volatility in the smaller coins at a one-day horizon, and no other pair of coins carries information. That is a real and usable statement about crypto structure, consistent with Makarov and Schoar on where crypto price discovery concentrates, and it does not depend on any of the estimators behaving as advertised. It also survives the winter split, and unlike the accuracy comparison, a rank ordering of features is a coarse enough claim that low statistical power does not sink it. Two of the paper's own summary numbers do not reconcile with its tables, so check them before you quote them. The introduction claims an AUC above 0.55 averaged across currencies and variables, while Table 2 reports aggregated AUC of 0.5381 at 50 bars and 0.5304 at 100 bars. The introduction also gives a per-label AUC range of 0.54 to 0.61, while the tables produce 0.53 to 0.58 for the four predictable labels; the 0.54 to 0.61 range is the futures result the authors quote later from their 2021 work. I keep the tables. The title claim holds. Crypto market dynamics are not structurally alien, and the same measures that predict distribution shifts in futures predict them on Binance at similar magnitudes. Nimalendran, Pathak, Petryk and Qiu reach a compatible verdict from variance-ratio tests rather than machine learning, so the inefficiency itself is not in dispute. Trading on it is another matter. Predicting the sign of the change in one-day-ahead realized volatility at 0.58 accuracy is a vol-forecasting result, and you can already get that from a GARCH fit on the same bars without a 25-feature forest. The paper is a careful negative for four of its five labels and a tautology-adjacent positive for the fifth. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/08/article_461-visual-0.png) ## KEY POINTS - The Roll measure the forest leans on is not a spread estimate. Taking the absolute value of the autocovariance makes it equal 2 times sigma times the square root of the absolute autocorrelation, so it responds to volatility with elasticity 1 and to autocorrelation with elasticity 0.5\. Simulated 50-bar windows give correlation 0.71 with sigma against 0.36 with the absolute autocorrelation. - Predicting the sign of the change in realized volatility from a volatility proxy at 0.58 accuracy is volatility clustering, not a microstructure foundation for momentum. The Roll measure scores mean decreased accuracy of 0.058 while every other feature sits under 0.01. - The paper's VPIN reduces to the average of the absolute value of one minus twice the normal CDF of standardized price changes, so volume cancels out entirely. Its expected value under a Gaussian random walk is exactly 0.500, and the reported Binance range of 0.457 to 0.471 is what noise with 5 to 7 degrees of freedom of tail weight produces. - Comparing 0.47 to the 0.22 that Easley, Lopez de Prado and O'Hara found in futures compares two estimators with different null values. Their version aggregates signed volume across a bucket before taking absolute values; this one takes absolute values bar by bar, which Jensen's inequality guarantees prints higher on any data. - With a 1,500-bar look-ahead over 942 days, the sample holds 904 independent observations and the standard error on accuracy is 0.0166\. Realized volatility gives z of 4.91, kurtosis 2.66, Jarque-Bera 1.98, autocorrelation 1.91, skewness zero. Against a Bonferroni threshold of 3.29 for 50 label-coin-window cells, one label survives. - The crypto-winter test compares 0.5379 to 0.5395 across subperiods holding 301 and 603 independent blocks. The standard error on that difference is 0.035, so the test cannot detect a true gap under about 7 accuracy points. Read the result as no detectable change, not as stability. - Kyle's lambda divides by net signed volume, which nearly cancels over 50 bars, so a hundredfold change in net imbalance moves lambda a hundredfold with the same price move. BTC prints 42.22 before winter, 0.391 after, and 1.26 at a 100-bar window. Its near-zero importance scores are the forest ignoring noise. - Take the cross-coin ranking as the durable finding: BTC and ETH Roll measures are the only cross-coin features with importance anywhere, so BTC and ETH volatility leads the smaller coins at a one-day horizon and no other pair carries information. ## References - [Flow Toxicity and Liquidity in a High-frequency World - Easley, Lopez de Prado, O'Hara (Review of Financial Studies, 2012)](https://doi.org/10.1093/rfs/hhs053?ref=aligrithm.com) - [Microstructure in the Machine Age - Easley, Lopez de Prado, O'Hara, Zhang (Review of Financial Studies, 2021)](https://doi.org/10.1093/rfs/hhaa078?ref=aligrithm.com) - [Trading and Arbitrage in Cryptocurrency Markets - Makarov, Schoar (Journal of Financial Economics, 2020)](https://doi.org/10.1016/j.jfineco.2019.07.001?ref=aligrithm.com) - [Informational Efficiency of Cryptocurrency Markets - Nimalendran, Pathak, Petryk, Qiu (Journal of Financial and Quantitative Analysis, 2024)](https://doi.org/10.1017/S0022109024000310?ref=aligrithm.com) - [Microstructure and Market Dynamics in Crypto Markets - Easley, O'Hara, Yang, Zhang (ssrn-4814346)](https://papers.ssrn.com/sol3/papers.cfm?abstract%5Fid=4814346&ref=aligrithm.com) --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ### 4.74 FX Edge Lives in Other Markets (cross-asset series) URL: https://aligrithm.com/fx-edge-lives-in-other-markets-cross-asset-series/ Last updated: 2026-10-16T08:09:18.000Z A currency futures book built from other currency futures scores a Sharpe of 0.16\. The same contracts, wired into a graph that also contains equities, bonds, and commodities, jump the FX sleeve to 0.66\. That is Pu, Roberts, Dong, and Zohren, 64 futures, 2000 to 2022, volatility-targeted to 15%. Phylaktis and Yamani ran the pairwise version of the same instinct and got a harsher split: a country's own stocks and bonds forecast next month's exchange rate in emerging markets, 0.59% a month net of costs, and fail the random-walk test in the G10\. Hong and Klabjan then showed that even those G10 pairs still pay once they are treated as a ten-currency graph with interest rates on the nodes. The old article ["Network Momentum as a Cross-Asset Factor"](https://aligrithm.com/network-momentum-as-a-cross-asset-factor/) already made the graph the unit of analysis. This piece is that claim run on FX, at three clocks, with the plumbing that can cut the wires. The pair is a slice. The object is the graph of stocks, bonds, rates, and other currencies. ## Pairwise first: stocks lead FX, and only where the market is slow Phylaktis and Yamani, covered in the old article ["Stocks and Bonds Predict FX — But Only in Emerging Markets,"](https://aligrithm.com/stocks-and-bonds-predict-fx-but-only-in-emerging-markets/) put one differential in front of one currency. Local stock return minus US stock return, one month, 28 countries, November 1989 to December 2019\. The forecast is a regression. $$ \\Delta s\_{i,t+1} = \\alpha + \\beta \\left( r^{E}\_{i,t} - r^{E,US}\_{t} \\right) + \\varepsilon\_{i,t+1} $$ Delta-s is next month's log change in foreign-currency units per one US dollar, so a negative number means the foreign currency strengthened. The term in parentheses is the equity differential. Beta is the whole signal. The bond model swaps in government-bond returns; the combined model uses both. Across the combined model the equity coefficient came in negative for every basket: minus 0.105 developed, minus 0.129 emerging, minus 0.083 global. Negative means a hot local equity market, relative to the US, is followed by a stronger local currency next month. That is return-chasing: foreign capital buys the equity, and has to buy the currency to do it. Work the emerging coefficient. Stocks beat the US by 2 percentage points this month. Predicted delta-s is minus 0.129 times 2, which is minus 0.258%, call it a quarter percent of appreciation, so the rule buys the currency. One coefficient, one differential, one month. The bar is the random walk, unbeaten in developed FX since Meese and Rogoff. They score it with out-of-sample R-squared. $$ R^{2}\_{OOS} = 1 - \\frac{MSFE^{\\chi}}{MSFE^{RW}} $$ MSFE is mean squared forecast error. The ratio of the model's squared misses to the random walk's, subtracted from one. Positive means the model beat the coin flip. The emerging combined model posts a relative MSFE of 0.95, so R-squared is 0.05: squared errors about 5% below the random walk. (The table caption says "percent units"; Rel-MSFE of 0.95 only matches if 0.05 is a fraction, so the caption and the arithmetic disagree, and the arithmetic is the one that is internally consistent.) The developed combined model posts 1.016, R-squared minus 0.016, worse than doing nothing. Clark-West t-stats, the formal nested-model test, draw the line. Emerging equity 3.113, emerging combined 2.724, both past 1%. Developed equity 0.626, developed combined 1.405, neither clearing 10%. Bonds fail the full sample in both groups. Trading the sign of the forecast, net of half the bid-ask, earned 0.30% a month on emerging equity signals and 0.59% a month combined, about 7% a year, with positive skew, beating carry and currency momentum. Developed currencies produced nothing tradable. The pairwise channel is real, and it is a leak. Emerging markets are thinner, more segmented, slower to absorb the equity flow. Developed markets arbitrage the same flow before a monthly signal can cash it. That is the old article's result, and it is also the trap: stop there and G10 FX looks like a dead letter. ## One hop wider: other currencies and the rate on the node Hong and Klabjan, in the old article ["Graph Learning for FX: Interest-Rate-Parity Statarb Done Right,"](https://aligrithm.com/graph-learning-for-fx-interest-rate-parity-statarb-done-right/) refuse the pair. Ten currencies are nodes. Tradable exchanges are edges. Government-bond interest rates sit on the nodes. The exchange rate rides the edge. A graph neural network passes messages along that web, which is the structure interest-rate parity already implies. Covered parity is the lock. $$ (1 + Y^{f}\_{i})\\, F\_{ij} = (1 + Y^{f}\_{j})\\, X\_{ji} $$ The left side grows a unit at country i's risk-free rate and converts forward. The right side converts spot into j and grows at j's rate. If the two sides disagree, borrow the cheap leg and lend the rich one. Work it: i pays 5%, j pays 1%, spot from j to i is 1.00\. The forward from i to j has to be 1.01 over 1.05, about 0.962, so the high-rate currency trades at a forward discount. Uncovered parity swaps the forward for the expected future spot. Either way the rate on the node carries information about the edge, which is why a flat table of pairs is the wrong input. The residual they hunt is the loop that should close and does not. $$ \\text{Loop payoff} = X\_{ik}\\, X\_{kj}\\, X\_{ji} - 1 $$ Convert one unit of i into k, k into j, j back into i, subtract the starting unit. Product above one is a paying loop. USD to EUR at 0.92, EUR to JPY at 162.0, JPY back to USD at 0.0068: 0.92 times 162.0 is 149.04, times 0.0068 is 1.0135\. The product lands at 1.0135 dollars, a 1.35% sliver. Aiba and coauthors documented those slivers as a real interaction among FX rates. Fenn and coauthors called the spot version a mirage once the time between the screen and the fill is in the model. Hong and Klabjan take Fenn's objection as the problem statement: decide at t minus 1, execute at t where the fill is random, hold overnight, convert home at t plus 1, and maximize expected gain over the standard deviation of that gain, not raw profit. Against a linear-program benchmark that uses the same rate forecasts, the graph method lifts information ratio 61.89% and Sortino 45.51%, cuts annual volatility 52.23% and max drawdown 44.77%, and pays for it with 22.73% less raw return: 4.80% a year versus 6.19% on a USD base, vol 0.68% versus 1.41%. Divide 4.80 by 0.68 and the ratio is about 7.1, against 6.19 over 1.41, about 4.4\. The table prints "Info. Ratio" as 43.86 versus 27.21, labeled in percent; those figures match a daily ratio times 100 (4.80% over 252 days, 0.68% annual vol, daily IR near 0.44), not the annual Sharpe. The 61.89% lift is the same ratio either way. Chaboud, Hjalmarsson, Vega, and Chiquoine already showed that algorithmic traders on EBS eat triangular opportunities at a five-second clock and cut the absolute autocorrelation of five-second returns, because machines reprice faster than people. That is not a contradiction. The five-second triangle is gone. The daily, rate-informed, lag-aware residual across ten currencies is a different object. G10 pairs fail Phylaktis's monthly stock regression and still contain a graph residual Hong can trade, on paper, with no transaction costs in the backtest and unlimited borrowing assumed. Treat the 4.80% as a method result, not a paycheck. ## Widen again: the FX node in a 64-asset graph Pu and coauthors, in the old article ["Network Momentum as a Cross-Asset Factor,"](https://aligrithm.com/network-momentum-as-a-cross-asset-factor/) take the next hop. Sixty-four futures across commodities, equities, fixed income, and FX. No firm-level tie connects cotton to the dollar. They learn the graph from prices: eight momentum features per contract (volatility-scaled returns at 1 day, 1 month, 3 months, 6 months, 1 year, plus three MACDs), then a convex program that puts large edges between assets whose feature vectors look alike and zeros the rest. Network momentum for a target is the edge-weighted average of its neighbors' features. The target's own past return does not enter. $$ \\tilde{u}\_{i,t} = \\sum\_{j \\in \\mathcal{N}(i)} \\tilde{A}\_{ij,t}\\, u\_{j,t}, \\qquad y\_{i,t} = \\tilde{u}\_{i,t}^{\\top} \\beta + b $$ The first sum is the propagated feature vector. The second is one cross-sectional OLS across all assets, predicting next day's volatility-scaled return; the sign is the trade. Work a scalar version. Neighbors weighted 0.5, 0.3, 0.2, with one-month vol-scaled momentum of plus 1.2, minus 0.5, and plus 0.8\. Propagated signal: 0.5 times 1.2 plus 0.3 times minus 0.5 plus 0.2 times 0.8, which is 0.60 minus 0.15 plus 0.16, so plus 0.61\. Long the target, even if its own recent return was flat. The ablation is the FX result that belongs in this article. Rebuild the network inside FX alone, then trade the same FX contracts off the full cross-class graph. ![Two-panel figure: left, Phylaktis combined-model out-of-sample R-squared, negative for developed pairs versus USD and plus 0.05 for emerging; right, Pu et al. Sharpe as the graph around FX widens, from 0.16 FX-only to 0.66 as an FX sleeve of the cross-asset graph to 1.51 for the full 64-asset book](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/08/article_437-fx_graph_width.png) FX-only network: Sharpe 0.16, 2.4% a year at the 15% vol target, 40% of days underwater. The same FX sleeve off the full graph: Sharpe 0.66, 9.7% a year, underwater time down to 29%. Equities make the same jump, 0.30 to 0.95\. The information that prices a currency future is sitting in the bonds, commodities, and equity indexes that share its momentum regime. Intra-class edges still carry more weight than inter-class edges alone (Sharpe 1.21 versus 0.91 for the full book), and the two are only 0.51 correlated, which is why the full graph beats either piece. Inter-class links are the ingredient that makes FX more than FX. They are not the whole meal. Gross, daily rebalance. The 64-asset book stays positive up to about 3 basis points of pseudo-cost and goes negative at 5\. Nobody should read 1.51 as a number that survives a futures roll and a real commission. ## The graph has a kill switch Golub, Dupuis, and Olsen, in the old article ["HFT Supplies Liquidity Until It Doesn't: FX Flash-Crash Cascades,"](https://aligrithm.com/hft-supplies-liquidity-until-it-doesnt-fx-flash-crash-cascades/) put the microstructure under the three papers above. The FX market is a mesh, not a ladder: EBS and Reuters at the center, Currenex and Hotspot as ECNs, Oanda as an FCM, CME futures off to the side, about 1.4 trillion dollars of spot a day. On EUR/USD, Ultra-HFT posts 61.6% of orders and fills 6.8% of them. Manual traders post 3.7% and fill about half. The depth on the screen is 93% cancellable. Chaboud's machines, the same cohort that closes five-second triangles, do not trade with each other as much as random matching would predict. Their strategies are more correlated than humans'. On average that is useful: prices update faster, arb dies, five-second autocorrelation falls. The same commonality means that when the shared signal says exit, they exit together and act as one enormous trader. Cespa and Foucault write the fragility: a small liquidity drop in one asset feeds back into a market-wide crash, with a bad equilibrium that looks like May 6, 2010. March 17, 2011 is the clean FX case. USD/JPY around 79.50 at 05:55 Tokyo, below 76.50 twenty-five minutes later, 300 pips, no fresh fundamental in that window. Retail stop-losses into a thin tape, banks yank quotes or widen until their bids sit under the last print, more stops, down to 76.25\. Recover to 78.23, then another automated stop-out wave, about two billion dollars, down to 77.10\. HFT and traditional makers had both left. Designated market makers widened so far the obligation was theater. So the graph the other three papers trade is not a fixed object. At a monthly clock, emerging equity flow still leaks into the currency. At a daily clock, G10 loops and cross-asset momentum still show up in closes. At a millisecond clock the edges are quotes, and quotes vanish in a cascade. A strategy that sizes off yesterday's graph and hits a market order into a 05:55 Tokyo tape is trading a different graph than the one it estimated. Hau and Rey, and Kremens and Martin, give the economics that makes the slower clocks possible: equity and currency returns are correlated through capital flow, and a stock-based currency risk premium has genuine predictive content. Phylaktis is that premium as a monthly trading rule. Hong is the intra-FX, intra-rate version of the same interrelatedness. Pu is the extra-FX version, learned from prices because cotton has no quanto on the dollar. Golub is the reminder that all of it clears through a book whose majority-posted liquidity is optional. ## Three clocks, one object Put the four papers on one desk and the disagreement dissolves into horizon. Monthly, pairwise, 28 countries: stocks (and, after 2013, bonds) predict emerging FX and lose to the random walk in the G10\. The edge is a flow lag, and it has an off switch. From September 2006 to April 2013, through the crisis dollar spike, the combined strategies made no reliable money and the emerging bond rule lost 0.68% a month. From May 2013 to December 2019 the same bond rule made 0.47% a month. One six-year window carries the full-sample 0.59%. Daily, ten currencies plus government-bond rates: G10 pairs that failed the monthly stock test still contain a graph residual. Information ratio up 61.89% versus an LP that ignores the lag, return down 22.73%, no costs in the test, drawdowns in 2020, 2022, and 2024 when regimes the training set never saw arrived. Daily, 64 futures: FX as a class is a weak momentum book on its own (Sharpe 0.16) and a decent one once equities, bonds, and commodities are allowed to speak (0.66). The full cross-asset network prints 1.51 gross and dies somewhere between 3 and 5 basis points. Millisecond: the transmission lines are HFT quotes. They tighten spreads and kill triangles in calm, and they drop the quote in a cascade. Retail is on the wrong side of both. A desk that stares at EUR/USD, or even at a basket of G10 pairs, is using the unit of analysis Phylaktis already falsified for developed markets and Pu already beat by a factor of four on FX futures. The trade is: estimate the graph that is live at your clock, trade the node off its neighbors, and assume the edges can go to zero without a press release. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/08/article_437-visual-0.png) ## KEY POINTS - The unit of analysis for FX is the graph of stocks, bonds, rates, and other currencies. A pair versus the dollar is a slice of that graph, and the slice is where the edge is weakest. - Pairwise, Phylaktis and Yamani: local minus US stock (and bond) returns forecast next month's FX in emerging markets, Clark-West t of 3.11 equity and 2.72 combined, 0.30% and 0.59% a month net of costs. Developed currencies fail the random walk. The leak is slow, segmented, and off from 2006 to 2013. - Intra-FX, Hong and Klabjan: currencies as nodes, rates as node features, loops as the residual. Against an LP that assumes a fill at the observed rate, information ratio up 61.89%, vol down 52%, return 4.80% versus 6.19%. Five-second triangles are already gone (Chaboud); this is a daily, lag-aware residual, costless in the paper, not a paycheck. - Cross-asset, Pu et al.: FX-only network Sharpe 0.16; the same FX sleeve off a 64-asset graph 0.66; full book 1.51 gross, 22% a year, 2000 to 2022\. The extra alpha is in the links between classes. Daily turnover kills it past about 3 basis points. - Microstructure, Golub, Dupuis, Olsen: Ultra-HFT posts 61.6% of EUR/USD orders and fills 6.8%. Correlated machines close arb in calm and withdraw together in stress. March 2011 USD/JPY dropped 300 pips in 25 minutes with no news. The estimated graph is not the graph a market order hits in a cascade. - Three clocks, one object. Monthly flow (emerging equities), daily graph residual (G10 plus rates, plus cross-asset momentum), millisecond quotes. Size the node off its live neighbors, and do not assume the edges stay posted. ## References - [Empirical exchange rate models of the seventies: Do they fit out of sample? (Meese and Rogoff, 1983)](https://doi.org/10.1016/0022-1996%2883%2990017-X?ref=aligrithm.com) - [Approximately normal tests for equal predictive accuracy in nested models (Clark and West, 2007)](https://doi.org/10.1016/j.jeconom.2006.05.023?ref=aligrithm.com) - [Exchange Rates, Equity Prices, and Capital Flows (Hau and Rey, 2006)](https://doi.org/10.1093/rfs/hhj008?ref=aligrithm.com) - [The Quanto Theory of Exchange Rates (Kremens and Martin, 2019)](https://doi.org/10.1257/aer.20180019?ref=aligrithm.com) - [Triangular arbitrage as an interaction among foreign exchange rates (Aiba et al., 2002)](https://doi.org/10.1016/s0378-4371%2802%2900799-9?ref=aligrithm.com) - [The mirage of triangular arbitrage in the spot foreign exchange market (Fenn et al., 2009)](https://doi.org/10.1142/s0219024909005609?ref=aligrithm.com) - [Rise of the Machines: Algorithmic Trading in the Foreign Exchange Market (Chaboud et al., 2014)](https://doi.org/10.1111/jofi.12186?ref=aligrithm.com) - [Illiquidity Contagion and Liquidity Crashes (Cespa and Foucault, 2014)](https://doi.org/10.1093/rfs/hhu016?ref=aligrithm.com) - [Foreign currency forecasting in emerging markets: What can stock and bond markets tell us? (Phylaktis and Yamani, 2025)](https://doi.org/10.1016/j.jempfin.2025.101641?ref=aligrithm.com) - [Graph Learning for Foreign Exchange Rate Prediction and Statistical Arbitrage (Hong and Klabjan, 2025)](https://arxiv.org/abs/2508.14784?ref=aligrithm.com) - [High-Frequency Trading in FX Markets (Golub, Dupuis, and Olsen)](https://scholar.google.com/scholar?q=High-Frequency+Trading+in+FX+Markets+Golub+Dupuis+Olsen&ref=aligrithm.com) - [Network Momentum across Asset Classes (Pu, Roberts, Dong, and Zohren, 2023)](https://arxiv.org/abs/2308.11294?ref=aligrithm.com) --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ### 9.47 There Is No Black-Scholes for Prediction Markets (Yet) URL: https://aligrithm.com/there-is-no-black-scholes-for-prediction-markets-yet/ Last updated: 2026-10-15T13:47:28.000Z Shaw Dalen and the Daedalus Research Team want to give prediction markets what Black and Scholes gave options: one shared kernel, one quotable risk factor, and a derivative layer on top. The construction is clean. Map the traded probability to log-odds so standard tools apply, force the price to be a martingale under the risk-neutral measure, and whatever is left over (belief volatility, jump intensity, cross-event dependence) becomes the thing you quote and hedge. The old article ["A Black-Scholes for Beliefs: Logit Jump-Diffusion and Tradable Belief-Vol"](https://aligrithm.com/a-black-scholes-for-beliefs-logit-jump-diffusion-and-tradable-belief-vol/) walked through the machinery in full. The machinery rests on two claims about the market underneath it. First, the traded price is the risk-neutral probability, so a dollar of payoff costs a coherent amount no matter which leg you buy. Second, that price carries no predictable drift beyond the convexity correction the model itself supplies. Three other papers, all reading real Polymarket and Manifold transaction data, say both claims are false right now and by wide margins. Prices do not sum to a dollar, and roughly $39.6 million walked out of the gap. Price and mean belief separate by construction under log utility, not by friction. And a two-dollar shove still moves a market price sixty days later. ## The kernel, and the assumption holding it up Write p for the traded price of a contract paying $1 if the event happens. The logit map sends p in the open interval from 0 to 1 onto the whole real line, so the modeller can use ordinary jump-diffusion tools without the price wandering outside its bounds. $$ x\_t = \\operatorname{logit}(p\_t) = \\log\\frac{p\_t}{1-p\_t}, \\qquad dx\_t = \\mu(t,x\_t)\\,dt + \\sigma\_b(t,x\_t)\\,dW\_t + \\int\_{\\mathbb{R}} z\\,\\tilde N(dt,dz) $$ _This post is for paying subscribers only._ ### 3.39 Does Complexity Actually Help? The Virtue-of-Complexity Autopsy URL: https://aligrithm.com/does-complexity-actually-help-the-virtue-of-complexity-autopsy/ Last updated: 2026-08-22T23:59:59.000Z Kelly, Malamud and Zhou opened their Journal of Finance paper by reporting out-of-sample market timing Sharpe ratio improvements "relative to market buy-and-hold" of roughly 0.47 per year, earned by a model with 12,000 predictors trained on 12 monthly observations. Daniel Buncic re-ran it on the same data with the same code and found the first problem in the first sentence. Buy-and-hold on that same standardized return series earns 0.5494 over the common evaluation window. The complex model earns 0.4851\. It does not improve on buy-and-hold by 0.47\. It loses to buy-and-hold by 0.06, before anyone pays a spread. That reads like a gotcha, and on its own it would be one. The substance is what Buncic does next. He shows that the headline theoretical prediction, that portfolio performance is strictly increasing in model complexity, only appears in the data because of two choices Kelly, Malamud and Zhou made in their implementation. Remove either one and the curve flattens or reverses. Neither choice has anything to do with complexity. ## The machinery you need to hold in your head The target is the excess return on the value-weighted CRSP market index, divided by a trailing 12-month volatility estimate. Call it R. The predictor set is the 15 Goyal-Welch variables, which is the standard equity premium prediction panel, plus last month's excess return. Sample runs December 1929 to December 2020 after standardization. The random Fourier feature trick converts those 15 predictors into as many regressors as you want. Draw a random 15-by-1 weight vector omega from a standard normal, form the dot product omega-transpose G, and feed it through a sine and a cosine. Each draw manufactures a pair of new signals out of the same underlying information. Draw 6,000 times and you have P equals 12,000 regressors built from 15 variables. Nothing new enters the information set. Only the parameter count grows. Fit by ridge regression, with shrinkage z: $$ \\hat{\\beta}(z) = \\left( zI + \\frac{1}{T}\\sum\_{t=1}^{T} S\_t S\_t' \\right)^{-1} \\frac{1}{T}\\sum\_{t=1}^{T} S\_t R\_{t+1} $$ Read the middle term as the sample covariance between the regressors and next month's return, and the inverted term as the regressor covariance matrix with z added down the diagonal. Larger z pushes the coefficients toward zero. Worked scalar example, one regressor, with the regressor's mean square equal to 1 and its covariance with next month's return equal to 0.20\. At z equals 0 the coefficient is 0.20 divided by 1, so 0.20\. At z equals 6 it is 0.20 divided by 7, so 0.0286\. At z equals 1000 it is 0.20 divided by 1001, so 0.0002\. That last number matters, because Kelly, Malamud and Zhou evaluate their 15-predictor linear benchmark at only two shrinkage settings, z equals 0 and z equals 1000\. On 15 predictors, z equals 1000 does not regularize the benchmark. It deletes it. Forecasts become positions directly. The predicted return is the portfolio weight. $$ \\pi\_t = \\hat{R}\_{t+1|t}, \\qquad R^{\\pi}\_{t+1} = \\pi\_t R\_{t+1} $$ You size your bet at the size of your forecast, then earn the forecast times the realization. Worked example: forecast plus 0.30 against a realized plus 1.20 pays 0.36\. Forecast minus 0.30 against a realized minus 1.20 also pays 0.36, because two negatives multiply to a positive. Being confidently right in either direction pays; being confidently wrong costs the same in reverse. Complexity is the ratio of parameters to training observations. $$ c = \\frac{P}{T} $$ With P equals 12,000 and T equals 12, c is 1000\. With P equals 200 and T equals 120, c is 5/3\. Hold that second identity, because it does the work later: lengthening the training window lowers complexity even when the model is unchanged. So "Sharpe ratio increases with T" and "Sharpe ratio increases as complexity falls" are the same sentence. ## Choice one: a regression forced through the origin Kelly, Malamud and Zhou fit every model without an intercept. Their footnote 35 defends it by saying that when they do add a constant, it gets shrunk so heavily that nothing changes. That defence contains the bug. Ridge penalizes the intercept along with everything else, and the intercept is the one coefficient you should never penalize, because it carries the unconditional mean of the target. The target here has a mean of 0.1590 with a robust standard error of 0.0336, so a t-statistic of 4.47\. It is not zero and the data say so loudly. None of the 15 predictors has a zero mean either. Force the fit through the origin and the model has to manufacture that non-zero mean out of slope coefficients on non-zero-mean regressors, which is a job the slopes were not hired for. The damage shows in the bias decomposition. With the zero-intercept restriction, squared bias in the forecasts from the headline model runs between 0.0130 and 0.0267 across the three training windows. Let the intercept float and exclude it from the penalty, and squared bias drops below 0.0002 in all three. That is a reduction of at least 65-fold, and it is not the bias that ridge shrinkage is supposed to buy you in exchange for lower variance. It is bias that a modelling restriction put there. Then the ordering flips. Estimated without an intercept, the whole Sharpe ratio surface for the random-feature models shifts down as the training window grows: 0.4888 at T equals 12, 0.4392 at T equals 60, 0.4305 at T equals 120\. Shorter training window, higher complexity, better performance, exactly as advertised. Estimated with an intercept, the same surface shifts up with the training window: 0.4745, 0.5157, 0.5863\. The best random-feature model in the entire exercise uses P equals 200 features on T equals 120 observations, which is a complexity of 1.67, and it is the only one that beats buy-and-hold. ## Choice two: averaging Sharpe ratios instead of averaging forecasts The random Fourier feature construction is random, so Kelly, Malamud and Zhou draw 1,000 independent sets of features and produce 1,000 forecasts for every month. Something has to collapse those into one number. They do it two different ways in the same paper. $$ \\overline{SR} = \\frac{1}{N}\\sum\_{n=1}^{N} SR\\!\\left(R^{\\pi,(n)}\\right) \\qquad\\text{versus}\\qquad SR^{\*} = SR\\!\\left(\\frac{1}{N}\\sum\_{n=1}^{N} R^{\\pi,(n)}\\right) $$ On the left, score each of the 1,000 draws separately and average the 1,000 scores. That is what produces the virtue-of-complexity plots in their Figures 7 through 9\. On the right, average the 1,000 strategies into one portfolio and score that. That is what produces their Table I, and it is the only one of the two you could trade, since holding 1,000 separate books and averaging their Sharpe ratios is not a portfolio. Expected return and alpha are linear, so both routes give identical answers for those. Sharpe ratio, information ratio, alpha t-statistic and R-squared are nonlinear, and the two routes diverge hard at low complexity. Here is why the divergence has to grow the way it does, and this mechanism is my reading rather than something the paper spells out. A single draw with P features already averages P/2 independent feature pairs inside itself. Its idiosyncratic error therefore shrinks roughly like one over the square root of P. Scoring one draw at a time means scoring a noisy strategy at small P and a self-averaged one at large P. Scoring the pooled portfolio averages across 1,000 draws no matter what P is, so it is quiet everywhere. Put numbers on it. Let each draw's monthly timing return be a common component with mean 0.1299 and standard deviation 1, plus idiosyncratic noise of standard deviation 21.2 divided by the square root of P. The true annualized Sharpe ratio of the signal is 0.1299 divided by 1, times the square root of 12, so 0.45, and it never changes with P. At P equals 2 a single draw has standard deviation equal to the square root of 1 plus 224.7, so 15.02, and an annualized Sharpe ratio of 0.1299 divided by 15.02 times the square root of 12, so 0.030\. At P equals 12,000 the same draw has standard deviation 1.019 and a Sharpe ratio of 0.44\. Average those per-draw scores and you trace a curve climbing from 0.03 to 0.44, monotone and concave, on a strategy whose predictive content is constant by construction. ![Simulated strategy with constant true Sharpe ratio of 0.45, scored two ways across complexity from 0.017 to 100: the average of 1000 per-draw Sharpe ratios climbs from 0.03 to 0.44 in a monotone concave curve, while the Sharpe ratio of the averaged strategy stays flat near 0.45](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/08/article_431-aggregation_artifact.png) Compare the shape against the real thing. Buncic replicates both aggregations on the actual data. The solid lines are the per-draw averaging that generates the published virtue-of-complexity curves. The dash-dotted lines are the pooled portfolio. ![Out-of-sample Sharpe ratios against complexity for training windows of 12, 60 and 120 months, models estimated without an intercept, showing solid lines rising steeply from near zero under per-draw averaging and dash-dotted lines starting high and flat under pooled averaging](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/08/article_431-voc_curves_no_intercept.png) The dash-dotted lines sit above the solid lines at every level of complexity and every level of shrinkage, and they are close to flat. The rise in the solid lines is the measurement catching up, not the model improving. Buncic's sharpest diagnostic sits in the bottom panel. At P equals 2 and T equals 120, per-draw averaging returns Sharpe ratios of about minus 0.0017 and information ratios of about minus 0.0084 across all seven shrinkage settings, while expected returns and alphas for the same models stay positive throughout. A strategy cannot have a positive expected excess return and a negative Sharpe ratio. The average of a thousand ratios can, which is the tell. Now add the intercept back and redraw the same plot. ![Out-of-sample Sharpe ratios against complexity for training windows of 12, 60 and 120 months, models estimated with an unpenalized intercept, showing curves starting high near 0.45 to 0.55 at low complexity, dipping toward the interpolation boundary at c equals 1, then recovering](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/08/article_431-voc_curves_with_intercept.png) The monotone concave shape is gone even under the per-draw averaging. The curves now start high, sag toward a trough at the interpolation boundary where c equals 1, then recover. Adding random features to a simple model makes it worse up to that boundary, which is the opposite of the advertised result. Maximum Sharpe ratios land at complexity 2, 12 and 6 for the three training windows, not at 1000. One more detail closes the loop. With an intercept, the simplest possible random-feature model, P equals 2, at heavy shrinkage reproduces the rolling historic average forecast to four decimal places: 0.4443, 0.4842 and 0.5446 for the three windows. The intercept-only model was always there. The zero restriction was hiding it. ## The machine did not learn to be long-only Kelly, Malamud and Zhou also claim their complex model discovers the Campbell-Thompson positivity constraint on its own, calling the high-complexity strategy "long-only at heart" and noting that the machine appears to learn the rule without being told. Their evidence is the asymmetry of the timing weights. ![Market timing weights from January 1931 to December 2020 without an intercept, six-month moving average and raw series, showing the complex model, a 500-feature model, a 15-predictor ridge regression and a single book-to-market regression all skewed positive](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/08/article_431-timing_weights_no_intercept.png) The weights do skew positive. They also skew positive for the 500-feature model, for a 15-predictor ridge regression at heavy shrinkage, and for a univariate regression on book-to-market alone. Any model fit through the origin on a target with a positive mean produces this. The correlation between the complex model's weights and the 500-feature model's weights is 0.9987. Let the intercept float and the asymmetry vanishes for every model at once. ![Market timing weights from January 1931 to December 2020 with an unpenalized intercept, showing the complex 12000-feature model, a 100-feature model, a 15-predictor ridge regression and a single long-term-yield regression as four nearly indistinguishable symmetric lines](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/08/article_431-timing_weights_with_intercept.png) Four lines, one visible series. The correlation between the 12,000-feature model and a 100-feature model rises to 0.9999\. Against a univariate ridge regression on long-term yield alone it is 0.9661\. The complex model and a single-variable regression are producing the same positions. And since these forecasts come from rolling 12-month windows, they are time-varying weighted moving averages of the last 12 monthly returns, which is enough on its own to reproduce the celebrated behaviour of divesting ahead of recessions. Buncic quotes Samuelson on the market having predicted nine of the last five recessions, which is the right response. ## What actually wins, and why you should not buy that either Over the common January 1940 to December 2020 window, the ranking runs: expanding-window ridge on the 15 Goyal-Welch predictors with mild shrinkage at 0.6990, rolling-window ridge on the same 15 at 0.6127, a univariate regression on the term spread at 0.6789, buy-and-hold at 0.5494, the rolling 120-month historic average at 0.5446, the best random-feature model at 0.5863, and the headline complex model at 0.4851\. Expected returns tell the same story: 0.044 for the complex model at T equals 120 against 0.859 for the best expanding-window linear model, a factor of roughly 20. This is where I part company with the paper's framing. Those 0.699 and 0.6127 figures are grid maxima. Buncic searches roughly 40 shrinkage values, crossed with rolling and expanding windows and three training window sizes, and reports the best, evaluated on the same 1940 to 2020 sample used to pick it. The 0.4851 he compares them against is a single pre-specified configuration. Comparing a maximum over a few hundred configurations to a point estimate is the selection bias that the old article ["Parameter Stability Beats Best Parameter"](https://aligrithm.com/parameter-stability-beats-best-parameter/) spends its length warning about, and the paper reports no deflated or selection-adjusted number. The honest version of the comparison is narrower: with an intercept, simple models are competitive with the complex one and the ordering in complexity reverses. Whether a 15-predictor ridge regression beats buy-and-hold out of sample by 0.15 of a Sharpe ratio is a separate question this paper does not settle. Two other cracks are worth naming. The predictor set uses contemporaneous inflation, which posts about two weeks into the following month, so every model here carries a look-ahead. Buncic reruns with lagged inflation and reports that random-feature Sharpe ratios drop across the board for all P and z, which means the complex models were leaning on the leak harder than the simple ones. And nothing in the exercise charges a cost. A strategy that re-sizes its position every month off a 12-observation refit turns over constantly, and the pooled random-feature portfolio is an average of 1,000 books. The generalizable lesson is not that complexity fails. It is that a theoretical result about complexity got confirmed by an empirical design where two settings unrelated to complexity, an origin restriction and a scoring convention, controlled the entire shape of the answer. The old article ["Why Simplicity Is a Statistical Weapon"](https://aligrithm.com/why-simplicity-is-a-statistical-weapon/) makes the case for small models on variance grounds. This is a different and more uncomfortable case: the measurement apparatus produced the result. Before you believe any curve that rises with a knob, turn the other knobs and check that the curve is still there. The old article ["Does ML Actually Help Asset Pricing? Kelly's 20%, Not 2–3×"](https://aligrithm.com/does-ml-actually-help-asset-pricing-kellys-20-not-2-3x/) reaches the same destination from the returns side. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/08/article_431-visual-0.png) ## KEY POINTS - KMZ's stated 0.47 annual Sharpe ratio improvement over buy-and-hold is not an improvement. Buy-and-hold earns 0.5494 on the same standardized series over the common January 1940 to December 2020 window, and their headline 12,000-feature model earns 0.4851. - The zero-intercept restriction is doing the work. The target has a mean of 0.1590 with a t-statistic of 4.47, so forcing the fit through the origin makes the slopes manufacture the mean. Squared bias falls from 0.0130 to 0.0267 down to under 0.0002 once the intercept floats and stays out of the ridge penalty. - Allow an intercept and the complexity ordering reverses. Best random-feature Sharpe ratios go 0.4745, 0.5157, 0.5863 for training windows of 12, 60 and 120 months. Longer window means lower complexity, since c equals P over T, so performance improves as complexity falls. - The VoC curve comes from scoring 1,000 random draws separately and averaging the scores, while KMZ's own Table I averages the forecasts first. A single draw with P features self-averages P/2 feature pairs, so per-draw noise shrinks like one over the square root of P. A simulated strategy with a fixed true Sharpe ratio of 0.45 traces the same monotone concave curve from 0.03 to 0.44 under that scoring rule. - Watch for the impossible number: at P equals 2 and T equals 120 the per-draw averaging returns Sharpe ratios near minus 0.0017 alongside positive expected returns and positive alphas. A real portfolio cannot do that. An average of ratios can. - The "machine learns to be long-only" claim is the same restriction seen twice. Free the intercept and the timing weights from a 12,000-feature model correlate 0.9999 with a 100-feature model and 0.9661 with a single regression on long-term yield. - Do not swap one grid maximum for another. The 0.6990 and 0.6127 linear results are the best of roughly 40 shrinkage values crossed with two window types and three window sizes, scored on the sample used to select them, with no deflation reported. Add contemporaneous inflation in the predictor set, which posts two weeks late, and zero transaction costs on a monthly-rebalanced book, and treat every number here as gross. ## References - [The Virtue of Complexity in Return Prediction (Kelly, Malamud, Zhou, 2024)](https://doi.org/10.1111/jofi.13298?ref=aligrithm.com) - [A Comprehensive Look at the Empirical Performance of Equity Premium Prediction (Goyal, Welch, 2008)](https://doi.org/10.1093/rfs/hhm014?ref=aligrithm.com) - [Predicting Excess Stock Returns Out of Sample: Can Anything Beat the Historical Average? (Campbell, Thompson, 2008)](https://doi.org/10.1093/rfs/hhm055?ref=aligrithm.com) - [A Comprehensive 2022 Look at the Empirical Performance of Equity Premium Prediction (Goyal, Welch, Zafiropoulos, 2024)](https://doi.org/10.1093/rfs/hhae044?ref=aligrithm.com) - [Empirical Asset Pricing via Machine Learning (Gu, Kelly, Xiu, 2020)](https://doi.org/10.1093/rfs/hhaa009?ref=aligrithm.com) - [Comment on 'The Virtue of Complexity in Return Prediction' (Berk, 2023)](https://ssrn.com/abstract=4410125?ref=aligrithm.com) - [Simplified: A Closer Look at the Virtue of Complexity in Return Prediction (Daniel Buncic, 2025)](https://papers.ssrn.com/sol3/papers.cfm?abstract%5Fid=5239006&ref=aligrithm.com) --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ### 3.56 Metaheuristics for Rule Optimization — With Diversity as the Guardrail URL: https://aligrithm.com/metaheuristics-for-rule-optimization-with-diversity-as-the-guardrail/ Last updated: 2026-10-15T19:01:15.000Z Hernández-Romo and co-authors ran four population-based optimizers 31 times each, 1000 generations per run, population of 50, tuning an eight-parameter moving-average strategy on five-minute BTC/USDC bars from January 2020 to March 2025\. Differential Evolution won on average annualized return with 107.36%, against 92.05% for the Whale Optimization Algorithm, 73.47% for Particle Swarm and 61.80% for Grey Wolf. Skip that ranking. The number worth keeping sits in Section 5.4: after 1000 generations, DE's population was still 80 to 85% as spread out as the random population it started from. The search never converged onto a point, and the authors published the trace instead of hiding it. That trace is why the paper earns a slot in a robust-systems reading list when most parameter-search papers do not. It reports how well the space got explored rather than only the best coordinate found, and that is the exact evidence a plateau argument needs. The authors then spend the conclusion ranking algorithms by peak return, which throws the evidence away. We can keep the useful half. ## What they optimized, and why a grid was never an option The strategy uses four simple moving averages, not two. SMA1 crossing above SMA2 opens a long; SMA3 crossing below SMA4 opens a short. Two independent crossover systems compete for one position slot, since the engine runs one position at a time and never holds both sides. $$ \\text{long at } t \\iff SMA1\_{t-2} \\le SMA2\_{t-2} \\;\\wedge\\; SMA1\_{t-1} > SMA2\_{t-1} $$ Read it as a crossover confirmed one bar late. Two bars back the fast average sat at or below the slow one; one bar back it sat above. The signal uses bars t-2 and t-1 and fills at the close of t-1, which is how the authors keep look-ahead bias out of the fill. Worked example on a 20-period and a 90-period average: two bars back SMA1 reads 61,400 and SMA2 reads 61,550, so the first condition holds. One bar back SMA1 reads 61,620 and SMA2 reads 61,570, so the second holds. You buy at 61,620. Exits run on multiplicative trailing levels. With the long stop factor k-sub-sll set to 0.95, that entry puts the stop at 61,620 times 0.95, or 58,539\. Price runs to 68,000 and the stop ratchets to 64,600\. Price falls back and the stop stays put. Eight parameters carry the strategy: four average lengths on \[5, 200\] and four exit factors, the long take-profit on \[1.0, 1.3\], the long stop on \[0.9, 0.99\], the short take-profit on \[0.9, 0.99\] and the short stop on \[1.0, 1.1\]. Grid the lengths at integer resolution alone and you get 196 to the fourth power, or 1,475,789,056 combinations, before touching the four continuous factors. Each evaluation means a full backtest over five-minute bars. That arithmetic is the honest case for a population-based search, and the authors make it. Costs are stated: 5 basis points per transaction, 1x leverage, 100% of capital per position, one order live at a time. ## The objective function rewards volatility Here is what the optimizers actually maximize. $$ \\mu(x) = \\frac{1}{M}\\sum\_{j=1}^{M} r^{(j)}(x) \\qquad f(x) = \\mu(x) \\;\\;\\text{maximised} \\qquad g(x) = \\max\_{t} D\_t - D\_{\\text{threshold}} \\le 0 $$ The fitness of a parameter vector x is the plain average of the returns it earns across M training windows. The constraint g rejects any configuration whose longest trade exceeds D-threshold, set to five days or 432,000 seconds. Training covers January 2020 through December 2022, 36 months, with an 8-month window sliding in 4-month steps, so windows start at months 0, 4, 8 and on through 28 and M equals 8. Worked example. Say a configuration returns 40%, 10%, -25%, 60%, 5%, -10%, 30% and 20% across those eight windows. The sum is 130, so mu equals 130 divided by 8, or 16.25%. Now watch what that objective prefers. A configuration that gains 100% in one window and loses 50% in the next scores an average of 25%, while the account ends exactly where it started: 2.0 times 0.5 equals 1.0\. A steadier configuration that gains 12% in both windows scores 12% and compounds to 1.12 squared, or 25.44%. The objective ranks the flat account above the one that made a quarter of your money. Arithmetic averaging of window returns pays for variance. The authors open the paper by criticizing the literature for judging strategies on absolute return while ignoring volatility and drawdown. Then f(x) is absolute return. Sharpe and Sortino show up in the reporting tables and never in the objective. You get what you optimize, and they optimized the thing they said was insufficient. The constraint has its own tell. DE's best configuration held trades for an average of 39 days in the validation period. The five-day cap binds on training trades only, so there is no formal contradiction, and the filter meant to exclude long holds still selected a configuration whose live behavior is long holds. The paper does not mention it. ## Rolling windows here are an averaging device, not a walk-forward ![Three horizontal bars labelled Iteration 1, 2 and 3, each a fixed-length training window sliding forward in time with roughly 50 percent overlap, and a legend containing only one entry, Training Window](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/08/article_612-rolling_window_scheme.png) Look at the legend. One category: Training Window. No test block follows any of the three bars. All eight windows sit inside the 2020 to 2022 block and every one of them feeds the objective, then a single holdout runs from January 2023 to March 2025. That design is worth something. Averaging across eight overlapping periods stops a configuration from winning on one lucky quarter, which is a robustness objective and a good one. It is not walk-forward validation, which fits on a window and then tests on the block immediately after it before rolling. The authors cite the walk-forward literature and use the term anyway. Call the scheme what it is: multi-period in-sample averaging plus one holdout. The 50% overlap also means each middle month enters the objective twice while the first and last four months enter once, so the sample's midsection carries double weight. ## Diversity, the part worth stealing The authors track a population-spread statistic every generation, following Hussain and co-authors. $$ Div = \\frac{1}{l \\cdot n} \\sum\_{d=1}^{l} \\sum\_{i=1}^{n} \\left| \\bar{x}^{d} - x\_{i}^{d} \\right| $$ For each dimension d, take the population mean along that dimension, measure how far every individual sits from it, and average the whole thing over n individuals and l dimensions. Worked example on one dimension with a population of 50\. If 25 individuals carry an SMA1 length of 20 and 25 carry 180, the mean is 100 and each individual sits 80 away, so that dimension contributes 80\. Collapse all 50 onto a length of 45 and it contributes 0\. Here n is 50 and l is 8. They then convert the raw spread into two percentages. $$ XPL\\% = \\frac{Div}{Div\_{max}} \\cdot 100 \\qquad XPT\\% = \\frac{\\left| Div - Div\_{max} \\right|}{Div\_{max}} \\cdot 100 \\qquad XPL\\% + XPT\\% = 100\\% $$ Div-max is the largest spread observed during the run, which lands at generation 1 when the population is still a random draw. So XPL% reads as a percentage of the initial random scatter, and XPT% is whatever remains. Worked example: a population whose spread has fallen to 12.2 from a starting 24.4 reports 50% exploration and 50% exploitation. A population at 82% exploration in generation 1000 is still 82% as scattered as the random draw it began with. ![Four line charts, one per optimizer, plotting exploration and exploitation percentages across 1000 generations: DE holds exploration near 82 percent throughout, GWO and PSO see exploration collapse early, WOA hands over from exploration to exploitation in a straight line](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/08/article_612-exploration_exploitation_by_optimizer.png) DE runs flat at 80 to 85% for all 1000 generations. GWO thrashes for roughly 350 generations and then exploration falls to near zero, and PSO settles around 15 to 20% after an early collapse. WOA does the textbook thing, a smooth linear handover that reaches full exploitation right at the end. ## A flat exploration line is a plateau statement Fitness stopped improving in DE's runs long before generation 1000, yet the population stayed spread out. Parameter vectors sitting hundreds of bars apart in SMA length were scoring the same. That is a fitness surface with many separate basins of near-equal quality, and it is the same claim the old article ["Parameter Stability Beats Best Parameter"](https://aligrithm.com/parameter-stability-beats-best-parameter/) makes from the other direction: the peak of an in-sample surface is a sampling artifact, and the thing to ship is a region. The old article ["Optimization Comes After Testing, Not Before"](https://aligrithm.com/optimization-comes-after-testing-not-before/) argues the same point procedurally, that a search which reports only its argmax has thrown away the diagnostic you needed. A diversity trace is that diagnostic, delivered for free by any population-based method, because the population is already a sample of the competitive region. Nobody has to build a separate stability map. Log the spread and read it. One caution the authors skip. A persistently high spread has a second explanation: a flat, noisy fitness surface exerts weak selection pressure, so the population drifts rather than occupying distinct optima. The paper picks the flattering reading, "multiple distant yet competitive optima." Both readings hand a trader the same instruction. No single parameter vector is special, so do not ship one. ## The metric adds bars to ratios The equation above sums absolute deviations across dimensions with no per-dimension scaling. Four of those dimensions are moving-average lengths measured in five-minute bars with a range of 195\. The other four are dimensionless exit multipliers with ranges from 0.09 to 0.30\. The sum adds bars to ratios, and the units do not cancel because there are no units to cancel against. Work out the damage. For a population spread uniformly across a range \[a, b\], the mean absolute deviation from the centre is (b minus a) divided by 4\. Each SMA length contributes (200 minus 5) over 4, or 48.75, and four of them contribute 195.0\. The long take-profit contributes 0.30 over 4, or 0.075\. The long stop and the short take-profit contribute 0.09 over 4, or 0.0225 each. The short stop contributes 0.10 over 4, or 0.025\. All four exit factors together contribute 0.145. ![Horizontal bar chart on a log axis showing each of the eight parameters' contribution to the diversity sum: the four SMA lengths each contribute 48.75, about 25 percent apiece, while the four exit factors contribute between 0.0225 and 0.075, about 0.01 to 0.04 percent apiece](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/08/article_612-diversity_dimension_contribution.png) Div comes to 195.145 divided by 8, or 24.39, and the exit factors supply 0.145 of 195.145, which is 0.07% of the total. DE's 82% exploration is a statement about SMA lengths and close to nothing else. Whether the optimizer explored the stop-loss and take-profit factors, the four parameters that set the risk profile of every trade, the metric cannot say. The fix costs one line: divide each dimension by its range before summing, so every parameter contributes on a 0 to 1 scale. Then the percentage means what the paper claims it means. Anyone borrowing this diagnostic should normalize first. ## The best-of-31 column measures dispersion, not skill Table 5 reports an average and a best across the 31 seeds. Averages run 61.80% to 107.36%, a spread of 45.56 points. The bests run 116.79% to 121.48%, a spread of 4.69 points. Four optimizers with wildly different averages produce peaks within five points of each other, and that is what the arithmetic of a maximum predicts. $$ E\\left\[\\max\_{1 \\le i \\le n} X\_i\\right\] \\;\\approx\\; \\mu + \\sigma \\cdot \\Phi^{-1}\\!\\left(\\frac{n - 0.375}{n + 0.25}\\right) $$ For draws from a normal distribution, the expected maximum of n of them sits a fixed number of standard deviations above the mean, and that multiplier grows with n. This is Blom's approximation. With n equal to 31, the fraction is 30.625 over 31.25, or 0.98, and the inverse normal of 0.98 is 2.05\. So the best of 31 seeds should land about 2.05 sigma above the average, and the gap between the two columns backs out the seed-to-seed sigma. Run it on all four. DE's gap of 119.62 minus 107.36 is 12.26, so sigma is 12.26 over 2.05, or 6.0\. WOA gives 26.10 over 2.05, or 12.7\. PSO gives 48.01 over 2.05, or 23.4\. GWO gives 54.99 over 2.05, or 26.8. ![Strip plot of 31 simulated seed outcomes per optimizer using the implied sigmas, with the reported average marked by a black bar, the reported best by a red star, and a dashed green line at the 104.2 percent annualized buy-and-hold return](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/08/article_612-best_of_31_selection.png) The back-out agrees with the paper's own claim that DE showed the lowest dispersion across runs, which is a small numerical check that passes and makes the rest of the reading safer. DE's seeds cluster inside a sigma of 6 while GWO's scatter across 27, a factor of four and a half. That is DE's actual result. It is not that DE finds better parameters. Its peak of 119.62% is second to PSO's 121.48%. DE is the optimizer you can rerun on Tuesday and get Monday's answer from, which for anyone re-optimizing on a schedule matters more than a peak. ![Violin plot of average return per optimizer showing DE with the narrowest distribution but the lowest median, while GWO, WOA and PSO have wider distributions with long negative tails](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/08/article_612-average_return_violin_by_optimizer.png) The violin plot confirms the dispersion story and contradicts the ranking. DE's shape is the tightest and its long tail into negative territory is missing, which is the point. Its median sits lowest of the four, around 130 on an unlabelled axis, while Table 5 hands DE the highest average. Take the picture for the widths and treat the levels as unresolved against the tables. Two more discrepancies belong on the record. Table 5's best column and Table 7's best execution both claim to report the highest validation return per optimizer, and they agree only for GWO at 116.79%. DE reads 119.62% against 108.86%, WOA 118.15% against 120.28%, PSO 121.48% against 110.84%. Keep both as printed rather than picking one. Separately, the authors select Table 7's configurations by highest validation-period return, so the holdout did the choosing, and Table 7's Sharpe of 1.086 and its drawdown of -28.66% are in-sample with respect to that selection. ## The benchmark comparison collapses under the paper's own formula The claim: "All optimized strategies significantly outperformed a passive Buy and Hold (397.93%) benchmark." That 397.93% is cumulative over the validation window. The strategy figures, 107.36% and 120.28% and the rest, are annualized. Put them on one clock using Equation 20 from the paper itself. $$ r\_{annual} = (1 + r)^{365/d} - 1 $$ Take the total return r over d calendar days and stretch it to a year. Worked example on the benchmark. The validation period runs 1 January 2023 through 31 March 2025, which is 365 plus 366 plus 90, or 821 days. With r equal to 3.9793, the base is 4.9793 and the exponent is 365 over 821, or 0.4446\. That gives 4.9793 to the power 0.4446, which is 2.042, so the annualized buy-and-hold return is 104.2%. Against 104.2%, DE's average of 107.36% clears the benchmark by 3.2 points while carrying 108.85% annualized volatility. WOA at 92.05%, PSO at 73.47% and GWO at 61.80% all lose to holding the coin. Only the cherry-picked best runs beat it with any margin, and those are max-of-31 statistics selected on the holdout. The paper never reports buy-and-hold's Sharpe, Sortino or maximum drawdown. For a study whose stated contribution is that prior work compares raw returns and ignores risk, omitting the benchmark's risk metrics repeats the exact failure it set out to correct. And the reproducibility section states a "356 day trading year" two lines above an equation using 365. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/08/article_612-visual-0.png) ## KEY POINTS - Log the population spread every generation and report it. Any population-based optimizer gives you a sample of the competitive region for free, so you never need a separate stability map. DE held 80 to 85% exploration for all 1000 generations, meaning parameter vectors far apart in the space scored the same. - A flat exploration line says the surface has many equal basins, which is the plateau argument from the old article ["Parameter Stability Beats Best Parameter"](https://aligrithm.com/parameter-stability-beats-best-parameter/) arriving from the search side. The second explanation, a flat noisy surface with weak selection pressure, leads to the same instruction: ship a region, not a point. - Normalize each dimension by its range before summing the diversity. Unnormalized, the four SMA lengths supply 195.0 of the 195.145 total and the four exit factors supply 0.07%, so the reported percentage says nothing about whether the stop and take-profit parameters got explored. - Best-of-31 is a dispersion statistic. Blom's approximation puts the expected maximum at 2.05 sigma above the mean for 31 draws, which backs out seed sigmas of 6.0 for DE, 12.7 for WOA, 23.4 for PSO and 26.8 for GWO. Averages spread over 45.56 points collapse to peaks spread over 4.69. - DE's real advantage is reproducibility, not return. Its peak of 119.62% trails PSO's 121.48%, and its seed sigma of 6.0 beats GWO's 26.8 by a factor of four and a half. Rerun DE and you get roughly the same answer. - Optimize the metric you claim to care about. The objective is the arithmetic mean of window returns, which scores a configuration that doubles then halves at 25% while the account ends flat, and a steady 12% twice at 12% while it compounds to 25.44%. Sharpe and Sortino appear only in the reporting tables. - Annualize the benchmark before comparing. The 397.93% buy-and-hold is cumulative over 821 days, which is 104.2% annualized by the paper's own Equation 20\. Three of the four optimizers lose to it on average, and the study never reports the benchmark's Sharpe, Sortino or drawdown. ## References - [Storn, R. and Price, K. (1997), Differential Evolution: A Simple and Efficient Heuristic for Global Optimization over Continuous Spaces, Journal of Global Optimization 11, 341-359](https://doi.org/10.1023/A:1008202821328?ref=aligrithm.com) - [Kennedy, J. and Eberhart, R. (1995), Particle Swarm Optimization, Proceedings of ICNN'95, 1942-1948](https://doi.org/10.1109/ICNN.1995.488968?ref=aligrithm.com) - [Mirjalili, S. and Lewis, A. (2016), The Whale Optimization Algorithm, Advances in Engineering Software 95, 51-67](https://doi.org/10.1016/j.advengsoft.2016.01.008?ref=aligrithm.com) - [Mirjalili, S., Mirjalili, S.M. and Lewis, A. (2014), Grey Wolf Optimizer, Advances in Engineering Software 69, 46-61](https://doi.org/10.1016/j.advengsoft.2013.12.007?ref=aligrithm.com) - [Hussain, K., Zhu, W. and Mohd Salleh, M.N. (2019), Long-Term Memory Harris' Hawk Optimization for High Dimensional and Optimal Power Flow Problems, IEEE Access 7, 147596-147616](https://doi.org/10.1109/ACCESS.2019.2946664?ref=aligrithm.com) - [Morales-Castañeda, B., Zaldívar, D., Cuevas, E., Fausto, F. and Rodríguez, A. (2020), A Better Balance in Metaheuristic Algorithms: Does It Exist?, Swarm and Evolutionary Computation 54, 100671](https://doi.org/10.1016/j.swevo.2020.100671?ref=aligrithm.com) - [Brock, W., Lakonishok, J. and LeBaron, B. (1992), Simple Technical Trading Rules and the Stochastic Properties of Stock Returns, Journal of Finance 47, 1731-1764](https://doi.org/10.1111/j.1540-6261.1992.tb04681.x?ref=aligrithm.com) - [Fama, E.F. and Blume, M.E. (1966), Filter Rules and Stock-Market Trading, Journal of Business 39, 226-241](https://doi.org/10.1086/294849?ref=aligrithm.com) - [Cerqueira, V., Torgo, L. and Mozetič, I. (2020), Evaluating Time Series Forecasting Models: An Empirical Study on Performance Estimation Methods, Machine Learning 109, 1997-2028](https://doi.org/10.1007/s10994-020-05910-7?ref=aligrithm.com) - [Suhonen, A., Lennkh, M. and Perez, F. (2017), Quantifying Backtest Overfitting in Alternative Beta Strategies, Journal of Portfolio Management 43, 90-104](https://doi.org/10.3905/jpm.2017.43.2.090?ref=aligrithm.com) - [Hernández-Romo, K., Lemus-Romani, J., Vega, E., Becerra-Rozas, M. and Romo, A., Robust Metaheuristic Optimization for Algorithmic Trading: A Comparative Study of Optimization Techniques, Mathematics 2026, 14, 69](https://doi.org/10.3390/math14010069?ref=aligrithm.com) --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ### 1.28 Model-Based or Data-Mined: Lotter's Framing of the Whole Problem URL: https://aligrithm.com/model-based-or-data-mined-lotters-framing-of-the-whole-problem/ Last updated: 2026-08-18T04:39:05.000Z Johann Christian Lotter opens his workshop deck with four lines that do more work than most methodology chapters. A model is not the reality. The reality is unknown. The same reality can be described with many different models. The best model must be selected by experiment. Every argument in Pillar 1 reduces to those four lines, and the deck states them in under thirty words before showing a single chart. The split he builds on top of them is the useful part. Model-based development starts with a market theory and looks for it in the data. Data mining starts with the price data and hunts for patterns that predict. Both end with a rule you can code, and the deck is honest that neither is safe. What follows takes the framing seriously, checks the arithmetic on the deck's own slides, and finds two places where the illustrations do not support the claim printed above them. ## The order book model, and where its arithmetic slips The first model is mechanical rather than statistical. Lotter puts up a book with asks of 10k at 1.03, 20k at 1.02 and 10k at 1.01, and bids of 20k at 0.99, 10k at 0.98 and 10k at 0.97\. The broker quotes price 1.01 and spread 0.02, which is the best ask and the distance to the best bid. Buy 10k at market and the 1.01 level disappears. The new best ask is 1.02, the best bid is still 0.99, and the quoted price and spread become 1.02 and 0.03\. A buy order pushes the price up. This is not a theory about trader psychology, it is bookkeeping. The next slide is where the arithmetic slips. You buy 10k at market and someone else sells 10k at market. Lotter marks both the 1.01 ask and the 0.99 bid as filled, then reports the new price as 1.01 and the spread as 0.02, unchanged, with the conclusion that buy and sell orders cancel each other. Against the static book he drew, they do not. Consuming the 1.01 ask leaves 1.02 as the best offer and consuming part of the 20k at 0.99 leaves 10k there, so the quote moves to 1.02 by 0.99 and the spread widens to 0.03\. His arithmetic holds only if the two market orders cross against each other rather than against the resting book, or if a dealer reposts the consumed levels straightaway. Both are reasonable assumptions about a retail forex broker, and neither appears on the slide. Keep the conclusion, which is that balanced two-way flow leaves the mid unmoved, and treat the ladder as a sketch rather than a matching engine. ## The random walk is the null, and it looks exactly like a price chart The second model is two dice. Roll one for buyers, roll one for sellers, and set the next price to the previous price plus buyers minus sellers. Lotter then plots a real EUR/USD curve against a synthetic random one and asks which is which. ![EUR/USD price in red against a synthetic random walk in blue, 2011 to 2016, both fluctuating between roughly 0.95 and 1.50 with visually indistinguishable structure](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/08/article_613-real_vs_random_eurusd.png) The red line is the real EUR/USD. The blue line is dice. Both trend, both consolidate, both put in what a chartist would call double tops and channel breakouts. If your rule discovery process cannot tell these two curves apart, it is not discovering anything, which is the argument the old article ["The Null Hypothesis for Trading Systems"](https://aligrithm.com/the-null-hypothesis-for-trading-systems/) makes from the statistical side and this chart makes from the visual side. Two rules fall out of the dice model. The first is that a pure random walk curve cannot be traded, which Lotter calls the rule of no roulette system. The second is the one people ignore. $$ \\sigma(T) \\;=\\; \\sigma\_1 \\sqrt{T} $$ Volatility over T bars equals the one-bar volatility times the square root of T. This comes from adding independent increments: variances add, standard deviations add in quadrature, so the standard deviation grows as the square root of the number of steps rather than in proportion to it. Work the dice case. One fair die has variance (36 minus 1) over 12, which is 2.9167\. The difference of two independent dice has variance 5.8333 and standard deviation 2.4152\. After 100 rolls the spread of outcomes has standard deviation 2.4152 times 10, which is 24.15\. After 400 rolls it is 2.4152 times 20, which is 48.30\. Four times the duration, twice the dispersion. The same rule on EUR/USD with a 0.6% daily standard deviation gives 0.6 times the square root of 20, which is 2.68% over a trading month, and 0.6 times the square root of 250, which is 9.49% over a year. The consequence Lotter draws is the one worth stealing. Maximum drawdown scales the same way, so a drawdown measured on a short backtest is not comparable to one measured on a long backtest. A system that showed a 12% drawdown over three years should be expected to show 12 times the square root of 2, which is 17%, over six years. Compute a Calmar ratio from a 10% annual return and the three-year drawdown and you get 0.83\. Use the six-year-equivalent drawdown and you get 0.59\. Skipping the normalization inflated the ratio by 41%. ## Every strategy in the deck is one equation The third model is the general one, and it is the reason the deck is worth reading twice. $$ y\_t \\;=\\; y\_{t-1} \\;+\\; f\\!\\left(y\_{t-1}, \\ldots, y\_{t-n}\\right) \\;+\\; \\varepsilon\_f \\;+\\; \\varepsilon\_r $$ Today's price equals yesterday's price, plus a deterministic function f of the recent price history, plus a fundamental shock epsilon-f, plus a random shock epsilon-r. Set f to zero and you have the random walk. Everything a trader calls an edge lives inside f, and Lotter defines inefficiency as a systematic deviation from the random walk, which is the same thing as saying f is not zero. Roughly 99% of tomorrow's price is yesterday's price; the fight is over the remainder. Momentum and mean reversion are two choices of f. $$ \\begin{aligned} \\text{Momentum:}\\quad y\_t &= y\_{t-1} + a\_1\\,(y\_{t-1}-y\_{t-2}) + a\_2\\,(y\_{t-2}-y\_{t-3}) + \\ldots + \\varepsilon\_f + \\varepsilon\_r \\\\\[6pt\] \\text{Mean reversion:}\\quad y\_t &= y\_{t-1} - \\frac{1}{\\lambda}\\,\\bigl(y\_{t-1}-\\hat{y}\\bigr) + \\varepsilon\_f + \\varepsilon\_r \\end{aligned} $$ The momentum form says a fraction a-one of the last bar's move carries into the next bar, plus smaller contributions from older bars. The mean reversion form says the price gets pulled back toward a reference level y-hat by one over lambda of its current distance from it, so lambda sets the speed. Work the reversion case with lambda equal to 20, a reference of 1.1000 and a current price of 1.1200\. The pull is one twentieth of 0.0200, which is 0.0010 against you per bar. The gap decays by a factor of 0.95 each bar, so the half-life is the log of 0.5 divided by the log of 0.95, which is 13.5 bars. The continuous approximation, lambda times the log of 2, gives 13.9\. Use the discrete number when your bars are the unit you trade. Now the momentum case, because the arithmetic there explains why so many systems die. Take a-one equal to 0.05 and hourly EUR/USD bars with a return standard deviation of 0.0010, which is 10 pips. The average absolute move is 0.8 times the standard deviation, so 8 pips, and the expected drift the momentum term buys you is 0.05 times 8 pips, which is 0.4 pips per bar. The noise on that same bar is 10 pips. Signal over noise per bar is 0.04\. Because the drift accumulates in proportion to the number of bars while the noise accumulates as the square root, the ratio grows as 0.04 times the square root of N. Reaching a ratio of 2 needs the square root of N to be 50, so 2,500 bars. That is the rule of square root volatility working against you rather than for you, and it says a 5% autocorrelation is real and undetectable on a few hundred bars at the same time. ![Volatility divided by the square root of horizon times one-bar volatility, plotted against horizon from 1 to 64 bars, for a momentum series, a pure random walk and a mean-reverting series, with a shaded band showing the 90 percent range of the same statistic measured on 2,000 bars of a pure random walk](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/08/article_613-sqrt_volatility_deviation.png) I simulated the three cases on 400,000 bars each and divided the T-bar volatility by the square root of T times the one-bar volatility, so a pure random walk sits at 1.0 by construction. The momentum series with a return autocorrelation of +0.15 climbs to 1.17 by 64 bars and the mean-reverting one falls to 0.86, while the random walk stays within 2% of the line. The shaded band is the honest part: it shows where that same statistic lands on 2,000 bars of pure dice, at the 5th and 95th percentiles across 500 draws. At a 2-bar horizon the band is narrow, roughly 0.98 to 1.02, and both inefficiencies sit far outside it. At 64 bars the band runs from 0.81 to 1.15 and swallows the mean-reverting curve whole. Two thousand bars holds only 31 non-overlapping 64-bar windows, and 31 observations cannot resolve a 14% effect. The deviation from the random walk is what you are hunting, it is small, and your ability to see it collapses as the horizon grows. ## Three prerequisites, and the test the slide does not run Lotter's bar for accepting a model has three parts. It has a rational basis in market structure or trader behavior. It can be written as a program flow or a formula. It has statistical significance in real price curves. Then the procedure: find an algorithm that detects the inefficiency, compute a statistic, plot a histogram, and compare against random walk curves or shuffled price curves. His last instruction is blunt, which is not to rely on other people's research because scam is ubiquitous. He names Elliott Waves as the example. The illustration he chooses is a frequency spectrum of EUR/USD. ![Bar chart of a EUR/USD frequency spectrum, horizontal axis running from 12 to 196 bars, with a pronounced cluster of tall bars peaking near 42 at a period of about 88 bars against a typical bar height near 25](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/08/article_613-eurusd_frequency_spectrum.png) The peak sits at a period near 88 bars and reaches about 42 against a typical bar height near 25\. Reading the other bars off the chart, they scatter roughly between 17 and 31, so a standard deviation near 4\. That puts the peak about 4.3 standard deviations above the middle of the pack, and 11 points clear of the next tallest bar anywhere in the spectrum. With 47 bins, the largest of 47 independent draws would be expected to land about 2.2 standard deviations above the mean, so 4.3 is beyond what bin-picking alone explains, and the fact that the peak is a cluster of five adjacent tall bars rather than one spike makes it harder to dismiss. That is my calculation, not his, and it is the calculation the slide skips. The deck tells you to compare against shuffled price curves and then shows a spectrum with no shuffled baseline on it, no confidence band, and no statement of the sample period or the number of assets scanned. A cycle detected on one currency over one window, presented without the comparison the previous slide demanded, is exactly the evidence standard the deck warns against three slides earlier. ## The filter matters more than the algorithm One line in the development process deserves more attention than the seven-step list around it. A market inefficiency does not exist all the time, so you need a filter that decides whether it is present, and in most cases the filter matters more than the algorithm. This inverts how most people allocate their effort. The cycle strategy is four lines: detect the dominant cycle and phase, generate a forerunning sine, short at a peak, long at a valley. The filter is one line: trade only when the amplitude of the dominant cycle exceeds a threshold. The four lines are the part everyone tunes and the one line is the part that decides whether the system makes money, because a cycle system running through a trending regime bleeds on every entry. The same asymmetry shows up in the trend and reversion strategies, both of which open with the identical instruction to detect the market regime first. His warnings on parameter fitting are the standard ones and they are correct. Do not optimize too many parameters. Do not optimize for peaks, which rules out brute force and genetic search as the final step. Find the most robust parameter range instead, the sweet spot. The old article ["Why Simplicity Is a Statistical Weapon"](https://aligrithm.com/why-simplicity-is-a-statistical-weapon/) gives the reason in statistical terms: every parameter you add hands noise another lever. ## The reality check in the deck does not pass Walk-forward analysis gets a fair treatment. Roll a window across the simulation, split it into a training and a test section, and the test is out of sample while still covering most of the data. Lotter names the cost immediately, which is that the system now depends on two more parameters, the cycle count and the split. Then he says that even after walk-forward and Monte Carlo the results still carry bias introduced by the development process itself, and he lists four ways to measure it, with White's Reality Check first. ![Histogram of a bootstrap reality check, horizontal axis showing profit factor from 1.185 to 1.420, red bars for randomized runs centered near 1.26 and a single black benchmark bar at about 1.32 with a visible red tail extending past it](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/08/article_613-whites_reality_check.png) Read this chart the way the method intends. The red histogram is the strategy run on many randomized versions of the price curve, and the black bar is the real result, a profit factor near 1.32\. The p-value is the fraction of red mass at or to the right of the black bar. Counting the red bars past 1.32 by eye gives roughly 85 out of about 1,000 runs, so a p-value near 0.085\. That does not clear a 5% bar. The deck's own illustration of how to detect bias shows a system whose edge fails the test, and the slide does not say so. Two more things sit in that picture. The randomized runs average a profit factor around 1.26, so the shuffled curves are profitable before any skill is involved, which is the long-bias trap in miniature. And a profit factor of 1.32 against a random mean of 1.26 is a thin margin to begin with, whatever the p-value turns out to be. I would not trade this, and I do not think the chart was chosen to be flattering. It was chosen to show the method, and the method worked. ## Reinvest the square root of your profits The money management slides carry the deck's only clean numbers, and they reward a Calmar comparison the deck itself does not run. The same EUR/USD system compounds at 15% with no reinvestment and 16% at 1% reinvestment. Push to half of OptimalF and the compound rate jumps to 48%. Reinvest only the square root of accumulated profits and it comes back to 43%. Taken alone, half of OptimalF wins. The equity charts say otherwise. Reading the drawdown troughs off both plots, the half-OptimalF run peaks near 175,000 and takes a worst drawdown near 60,000, which is 34% of peak. The square-root run peaks near 125,000 and takes a worst drawdown near 25,000, which is 20%. Calmar for the aggressive version is 48 over 34, so 1.41\. For the square-root version it is 43 over 20, so 2.15\. Giving up a tenth of the compound rate bought a 52% better return-to-drawdown ratio. That is the argument for the square root rule, and it survives the arithmetic. Those trough depths are eyeballed from chart axes rather than printed in the deck, so treat the ratio as approximate and the direction as solid. ## Step 1: visit trader forums The deck closes with seven steps labelled the real development process. Find a thread about a fabulous new indicator. Code it, find the backtest disappointing, and debug. Add a trailing stop, run a day-of-week analysis, discover Tuesday is bad, add a Tuesday filter, add a midday filter, and watch the backtest profit appear. Run a walk-forward on 23 optimized parameters, and when that disappoints, try different cycle counts and bar periods until a sensational out-of-sample number arrives. Trade it live. Watch it fail. Hold seminars. The joke has a precise structure. Every step is a legitimate technique applied in the wrong order. Walk-forward analysis is sound. Day-of-week analysis is sound. Trailing stops are sound. Running them as a search for a number that looks good converts each one into another trial in an uncounted multiple-testing experiment, which is the failure the old article ["The Scientific Method for Building Trading Systems"](https://aligrithm.com/the-scientific-method-for-building-trading-systems/) builds its pass and fail gates to prevent. Here is where I would push back on the framing itself. Lotter presents model-based development as the disciplined alternative to data mining, and the deck's own structure undercuts that. He lists ten families of inefficiency: momentum, mean reversion, cycles, statistical arbitrage, price constraints, clusters, patterns, gaps, seasonality and heteroskedasticity. Choosing among ten models, across a handful of assets, with the best one selected by experiment, is a search over 30 or 50 specifications. The multiple-testing tax on that search is identical to the tax on mining 50 rules out of the price data. Starting from a market theory buys you a rational basis, which is worth something when you are deciding whether a survivor is plausible, and it buys you fewer free parameters inside each candidate. It does not buy you a free pass on counting your trials. The deck never counts them. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/08/article_613-visual-0.png) ## KEY POINTS - Model-based development starts with a market theory and looks for it in the data; data mining starts with the data and hunts for patterns. Lotter's four-line epistemology sets the standard for both: a model is not reality, reality is unknown, many models fit the same reality, pick by experiment. - The general price curve model, price equals previous price plus a function f of recent history plus a fundamental and a random shock, contains every strategy in the deck. Momentum, mean reversion, cycles and GARCH are choices of f, and inefficiency means f is not zero. - Volatility scales as the square root of duration, so a drawdown from a three-year backtest must be multiplied by the square root of 2 before you compare it with a six-year one. Skipping that step inflated a Calmar ratio from 0.59 to 0.83 in the worked case. - The square root rule works against detection too. A 5% momentum autocorrelation on 10-pip bars produces 0.4 pips of drift against 10 pips of noise, and needs about 2,500 bars to reach a signal-to-noise ratio of 2. - The deck's frequency spectrum peak at 88 bars sits about 4.3 standard deviations above the other bins, which is beyond bin-picking, but it is shown with no shuffled baseline despite the previous slide demanding one. - The White's Reality Check illustration puts roughly 85 of about 1,000 randomized runs at or above the benchmark profit factor of 1.32, so a p-value near 0.085\. The deck's example of how to detect bias is a system that fails the test. - Reinvesting the square root of profits returned 43% against 48% for half of OptimalF, with the worst drawdown falling from about 34% of peak to about 20%. Calmar goes from 1.41 to 2.15. - Model-based development does not exempt you from the multiple-testing tax. Selecting the best of ten inefficiency families across several assets is a search with 30 or 50 trials in it, and the deck never counts them. ## References - [A Reality Check for Data Snooping (White, 2000)](https://doi.org/10.1111/1468-0262.00152?ref=aligrithm.com) - [Generalized Autoregressive Conditional Heteroskedasticity (Bollerslev, 1986)](https://doi.org/10.1016/0304-4076%2886%2990063-1?ref=aligrithm.com) - [Efficient Capital Markets: A Review of Theory and Empirical Work (Fama, 1970)](https://doi.org/10.2307/2325486?ref=aligrithm.com) - [A New Interpretation of Information Rate (Kelly, 1956)](https://doi.org/10.1002/j.1538-7305.1956.tb03809.x?ref=aligrithm.com) - [Portfolio Selection (Markowitz, 1952)](https://doi.org/10.1111/j.1540-6261.1952.tb01525.x?ref=aligrithm.com) - [Model-Based Trading Strategies - Johann Christian Lotter, Financial-Hacker.com](https://financial-hacker.com/build-better-strategies-part-2-model-based-systems/?ref=aligrithm.com) --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ### 10.14 Price-Path Convexity: A New Cross-Sectional Anomaly (−45bp per σ) URL: https://aligrithm.com/price-path-convexity-a-new-cross-sectional-anomaly-45bp-per-s-2/ Last updated: 2026-08-16T10:21:16.000Z Take two stocks that both end the month flat. The first bled lower for two weeks, then clawed all the way back. The second ran up for two weeks, then gave it all back. Same start, same finish, same zero return. Sort every stock in the market by past return and these two land in the identical bucket, because return only sees the endpoints. Gulen and Woeppel show that the shape between the endpoints predicts next month, and it predicts it hard: the stock that ended by rising underperforms the stock that ended by falling by about 0.84% a month, and no standard factor touches the spread. The old article ["Wave Velocity and Acceleration: Reading When the Market Runs Out of Gas"](https://aligrithm.com/wave-velocity-and-acceleration-reading-when-the-market-runs-out-of-gas/) argued that the second derivative of a price path carries an early read the level and slope miss. This is the cross-sectional version of that claim, tested on sixty years of the entire U.S. market instead of one instrument's sine fit. Return is the first derivative. Convexity, the curvature of the path, is a scaled cousin of the second derivative, and it prices the cross-section on its own. ## What convexity measures, and how to compute it Convexity compares two summaries of the same price path. Take the midpoint of the first and last closing price of the month. Subtract the average of every daily close in the month. Divide by the midpoint to standardize across price levels. That difference is the whole variable. $$ \\text{Convexity}\_{it} = \\frac{\\dfrac{P\_{1t} + P\_{N\_{it}}}{2} - \\dfrac{1}{N\_{it}}\\sum\_{k=1}^{N\_{it}} P\_{kt}}{\\dfrac{P\_{1t} + P\_{N\_{it}}}{2}} $$ Read it as midpoint of the endpoints, minus mean of the closes, over the midpoint. P-one is the first close of the month, P-N is the last close, and the middle sum is the average of all the closes in between. When the average of the closes sits below the midpoint of the endpoints, the path sagged in the middle and recovered, so convexity is positive and the path is convex, a valley. When the average sits above the midpoint, the path bulged up in the middle and faded, so convexity is negative and the path is concave, a hump. ![Convex versus concave price paths, showing the midpoint of the endpoints against the mean of the daily closes](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/08/article_408-convexity_geometry.png) Work the paper's own example, in dollars. A stock opens the month at 20, drops 1 a day for ten days down to 10, then climbs 1 a day for ten days back to 20\. First close and last close are both near 20, so the midpoint is 20\. The average of all the closes is dragged down toward 15 by the trough. Midpoint minus mean is about 4.8, and 4.8 over 20 is roughly 0.24\. That is a strongly convex path. Now steepen the front half: drop 1.50 a day for ten days to 5, then climb 1 a day, ending at 15\. The average sinks further, the midpoint falls to 17.5, and convexity rises to about 0.42\. A return measure would call both paths the same, since it only reads start and end. Convexity separates them because it weighs the whole trajectory. The authors build it from price changes in dollars, not percentage returns, on purpose. Investors think about a stock in dollar terms as much as in percent terms (Shue and Townsend document the non-proportional thinking), and dollar changes track the path's actual dynamics better than a return that collapses to its endpoints. ## The cross-sectional result: a monotone decline and a clean spread Sort every NYSE, AMEX, and NASDAQ common stock into convexity quintiles at each month-end, form value-weighted portfolios, and hold one month. The pattern is monotone. The lowest-convexity quintile, the concave humps that just faded, earns 1.24% the next month. The highest-convexity quintile, the convex valleys that just rallied, earns 0.41%. Every step down in convexity rank buys you return. ![Average next-month return by convexity quintile, declining monotonically from CON1 to CON5](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/08/article_408-quintile_returns.png) Buy the lowest quintile, short the highest, and the zero-investment spread is 0.84% a month with a t-stat of 6.29\. That is a Sharpe of 0.23 monthly, which annualizes near 0.80\. The old article ["Portfolio Sorts From Scratch: Deciles, Monotonicity, and the Long-Short Spread"](https://aligrithm.com/portfolio-sorts-from-scratch-deciles-monotonicity-and-the-long-short-spread/) set the bar for trusting a sort: you want a monotone gradient across the buckets, not a lone extreme quintile carrying the whole spread. Convexity clears it, and it clears it at shorter horizons too, earning 0.64% per 10-day hold (t of 7.31) and 0.37% per 5-day hold (t of 4.94). Then run the spread through the factor gauntlet. Regress the long-short return on CAPM, then the Fama-French three-factor model, then the five-factor model, then five-factor plus momentum, short-term reversal, long-term reversal, and liquidity. The alpha stays between 0.81% and 0.96% a month, every t-stat above 5.6\. The factor models explain almost none of it. The spread's return also beats the best single factor in the sample (momentum) by 20 basis points, and its Sharpe runs more than half again above the second-best factor. During recessions the spread pays 1.55% a month. A premium that gets larger when the world is worse is the opposite of what a risk story predicts. ## Is this just short-term reversal wearing a costume? Short-term reversal is the oldest short-horizon anomaly in the book: last month's losers beat last month's winners next month. The old article ["Reading Reversals from Autocorrelation"](https://aligrithm.com/reading-reversals-from-autocorrelation/) traced that effect to negative serial correlation at short lags. Convexity smells related, since a convex valley ends on an up-move and a concave hump ends on a down-move, and reversal would fade both. The obvious worry is that convexity is reversal with extra steps. The double sort kills that worry. Sort independently on convexity and on lagged one-month return, then read the convexity spread inside each reversal bucket. ![Convexity long-short spread within each short-term-reversal quintile, all near 0.8 to 1.0 percent and significant](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/08/article_408-distinct_from_reversal.png) Inside every reversal quintile, from the biggest losers to the biggest winners, the convexity spread survives: 0.97%, 0.83%, 0.78%, 0.92%, and 0.81% a month, with t-stats from 4.60 to 5.69\. Convexity predicts among stocks that all had the same past return. Run it the other way, hold convexity fixed and vary past return, and reversal mostly flattens out; in the lowest convexity quintile lagged return does not predict future return at all. So the curvature carries information the endpoints do not, which is the same lesson the old article ["Legendre-Polynomial Trend and Trend Relative to Local Variation"](https://aligrithm.com/legendre-polynomial-trend-and-trend-relative-to-local-variation/) pushed for a single series: a clean straight climb and a drunk stagger with the same net move are different trades, and you want a number that scores the shape, not just the displacement. ## The 45 basis points that survive every control Portfolio sorts can hide a correlated characteristic doing the real work. The fix is a Fama-MacBeth cross-sectional regression: each month, regress next month's return on this month's convexity plus a stack of controls, then average the monthly slopes and test whether the average slope is different from zero. $$ R\_{i,t+1} = \\gamma\_{0t} + \\gamma\_{1t}\\,\\text{Convexity}\_{it} + \\sum\_{j} \\gamma\_{jt}\\,\\text{Control}\_{j,it} + \\varepsilon\_{i,t+1} $$ R next month is stock i's return, the gammas are the slopes estimated fresh each month, and the reported number is the time-series average of the convexity slope with a t-stat corrected for autocorrelation. The controls are the usual suspects: size, book-to-market, profitability, asset growth, momentum, one-month return (that is short-term reversal), illiquidity, idiosyncratic volatility, skewness, and maximum daily return. With every control in the same regression, a one-standard-deviation increase in convexity is associated with a 45 basis point decline in the next month's return, at a robust t-stat of −11.15. Put a number on it. Say the convexity distribution has a standard deviation of 0.05 in this month's cross-section. A stock one standard deviation above average convexity, so 0.05 more convex than the typical name, carries a predicted return roughly 0.45% lower next month than the average stock, holding size, momentum, reversal, and the rest fixed. That is a large tilt for a variable built from nothing but daily closes, and the t of −11.15 says it is not a fluke of one decade. It also is not end-of-month drift in disguise; the relation holds after controlling for the lagged 1-day, 5-day, and 10-day returns, so it is not just the last few days of the month bleeding into the next. ## Why it works: overextrapolation, not risk or illiquidity The authors run the three standard explanations and only one stands. Risk fails because the effect lives at very short horizons where valuations barely move, survives every risk-factor adjustment, and strengthens in bad states. Illiquidity fails because the aggregate effect is stronger when illiquidity is low, and the firm-level spread holds among the most liquid stocks. What is left is mispricing, and the mispricing has a mechanism. Decompose convexity and it turns into a weighted average of daily price changes, with more recent changes getting more weight, minus a term that stands in for the cumulative return. $$ \\text{Convexity} \\;\\approx\\; \\frac{P\_N}{P\_1 + P\_N}\\left\[\\frac{N\\,\\Delta P\_N + (N-1)\\,\\Delta P\_{N-1} + \\cdots + 1\\cdot\\Delta P\_1}{N + (N-1) + \\cdots + 1} \\;-\\; \\frac{P\_N - P\_1}{N}\\right\] $$ The first bracketed term is a weighted mean of the daily price changes delta-P, where the newest change gets weight N, the next gets N minus 1, on down to weight 1 for the oldest. In a 21-day month, N is 21, so the last day's move counts twenty-one times as heavily as the first day's, and the second-to-last counts twenty times, and so on down. Recency-weighted averages like this are exactly how the literature models an extrapolative investor: someone who forms beliefs about the next move by leaning on the most recent moves. So convexity mechanically loads on the same recency-weighted signal that drives extrapolation. Then test it against real expectations. At the market level, the authors take Yale survey data on one-month DJIA return expectations and fit a decay model measuring how much investors lean on recent weekly returns. Investors extrapolate: recent returns push expectations up, and the extrapolative part of those expectations predicts lower future returns. Regress that extrapolative component on convexity and the R-squared clears 40%. At the firm level they repeat it with Forcerank, a platform where people rank stocks by expected next-week return; convexity explains the extrapolative slice of those rankings with R-squared near 28%, while it explains none of the residual slice. Investors see a stock that has been climbing into the close, extrapolate the climb, bid it up, and eat the reversal. Convexity is a clean read on how far that extrapolation has gone. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/08/article_408-visual-0.png) ## Where this sits, and what would break it Convexity is the cross-sectional payoff of a bet this platform has made in single-series form for a while: the second derivative of a price path is information, not decoration. The old article "Wave Velocity and Acceleration" built acceleration from a sine model to catch a single instrument running out of gas. Gulen and Woeppel build a scaled curvature from raw closes and show the same instinct prices six decades of the whole market, with a t-stat that dwarfs anything a single-instrument fit produces. Stay skeptical about the trade, not the finding. The spread is a monthly-rebalanced long-short over the full cross-section including small and illiquid names, and the paper reports gross returns; short-horizon anomalies are where transaction costs do their worst, so the honest question for a live book is how much of 0.84% a month survives spreads, borrow, and the turnover of a signal that reshuffles every month. The recession outperformance is a real point in its favor, and the extrapolation channel gives it a mechanism instead of a data-mined coincidence. But the number to reproduce before trusting it is not the alpha, it is the alpha net of costs at your rebalance frequency, on your investable universe. That test is the point of the whole factor-zoo discipline, and convexity earns the right to sit in it. ## KEY POINTS - Convexity is the curvature of a monthly price path: the midpoint of the first and last close, minus the average of all closes, over the midpoint. Positive means a convex valley that recovered into the close; negative means a concave hump that faded. - Return reads only the endpoints; convexity reads the whole trajectory. Two stocks with the same start, end, and return can have opposite convexity. - Sorted into value-weighted quintiles, next-month returns fall monotonically from 1.24% (low convexity) to 0.41% (high convexity). The long-short spread is 0.84% a month, t-stat 6.29, Sharpe near 0.80 annualized. - No standard factor model explains it. Alphas against CAPM, FF3, FF5, and FF5 plus momentum, reversal, and liquidity all stay between 0.81% and 0.96% a month, every t above 5.6\. The spread pays 1.55% in recessions, which argues against a risk story. - It is distinct from short-term reversal. In a double sort, the convexity spread survives inside every past-return quintile (0.78% to 0.97%, all significant), while past return often does not predict inside a convexity quintile. - In a Fama-MacBeth regression with size, momentum, reversal, illiquidity, idiosyncratic vol, skewness, and max daily return all controlled, a one-standard-deviation rise in convexity predicts a 45 basis point lower next-month return, t of −11.15. - The mechanism is overextrapolation. Convexity decomposes into a recency-weighted average of price changes, the same object used to model extrapolative beliefs, and it explains the extrapolative component of survey and Forcerank return expectations with R-squared above 40% and near 28%. - Before trading it, reproduce the alpha net of costs at your rebalance frequency on your own universe, not the gross number. Short-horizon spreads are where costs bite hardest. ## References - [Systematic Trading - Robert Carver (Amazon)](https://www.amazon.com/s?k=Systematic+Trading+Robert+Carver&ref=aligrithm.com) - [Evidence-Based Technical Analysis - David Aronson (Amazon)](https://www.amazon.com/s?k=Evidence-Based+Technical+Analysis+David+Aronson&ref=aligrithm.com) - [Price-Path Convexity and Short-Horizon Return Predictability (price-path-convexity-and-short-horizon-return-predictability)](https://scholar.google.com/scholar?q=Price-Path+Convexity+and+Short-Horizon+Return+Predictability&ref=aligrithm.com) --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ### 10.12 Factor Timing Mostly Fails — the Honest Version URL: https://aligrithm.com/factor-timing-mostly-fails-the-honest-version-2/ Last updated: 2026-08-15T06:37:00.000Z Factor timing is the seductive idea that you can do better than holding a basket of factors: lean into value when value looks cheap, load momentum when momentum is running, cut a factor when its volatility spikes. Every one of those moves has a tidy formula and a plausible story. And when you run them out of sample on a clean universe, almost all of them lose to the dumbest thing you could have done, which is hold every factor in equal weight and never touch it again. That is the honest version, and it is not a rounding error. Chuan Shi ran the four classic timing signals, valuation spread, factor momentum, factor volatility, and investor sentiment, on the Chinese market from 2000 to 2020, and none of them beat an equal-weighted seven-factor portfolio at any horizon he tried. The equal-weight benchmark itself was real, earning 0.27% a month over the market with a Newey-West t-statistic of 3.98\. The timing overlays added nothing on top. Meanwhile there is a genuine theoretical argument, from Haddad, Kozak and Santosh, that timing should matter enormously. Reconciling those two facts is the whole article, and the resolution is the same lesson the old article ["Collinearity in Parameter Sweeps: Plateaus, Not Peaks"](https://aligrithm.com/collinearity-in-parameter-sweeps-plateaus-not-peaks/) taught for backtests: robust edges live on wide plateaus, and factor timing is mostly an attempt to stand on a sharp peak that estimation error knocks you off of. ## The four timing recipes, and their formulas Start with valuation timing, the most respectable of the four. You measure each factor's value spread, the valuation of its long leg minus its short leg, and you overweight a factor when its own spread is wide relative to its history. The key discipline is that you compare a factor to itself, never across factors, because every factor lives in its own valuation range. $$ w\_{kt}^{\\text{raw}} = 1 + \\frac{VS\_{kt} - VS\_{kt}^{\\text{mean}}}{VS\_{kt}^{\\text{max}} - VS\_{kt}^{\\text{min}}}, \\qquad w\_{kt} = \\frac{w\_{kt}^{\\text{raw}}}{\\sum\_{j=1}^{K} w\_{jt}^{\\text{raw}}} $$ VS-kt is factor k's value spread now, and the mean, max and min are taken over its own trailing window of T months. The raw weight is a number near 1 that rises above 1 when the factor is cheap relative to its own past and falls below 1 when it is expensive, and then you normalize the raw weights across the K factors so they sum to one. Work a number. Suppose value's spread today sits at 0.8, its trailing mean is 0.5, its max is 1.0 and its min is 0.2\. The raw weight is 1 plus (0.8 minus 0.5) over (1.0 minus 0.2), which is 1 plus 0.3 over 0.8, or 1.375\. A factor trading at its historical median gets exactly 1.0, and one at its cheapest gets close to 2.0\. Tidy. The problem is not the recipe, it is that when Shi actually traded it in China the excess return over equal weight was negative at every lookback from 24 to 60 months, the least bad being minus 0.03% a month with a t-statistic of minus 0.53. Factor momentum is the second recipe and needs no calculus. Rank the K factors by their trailing twelve-month return, hand the best factor a score of K, the next K minus 1, on down to 1 for the worst, and normalize those scores into weights. It is a winner-loser bet on factors themselves. Sentiment timing is the fourth: build an index from five gauges, the ratio of upside to downside volatility, turnover, the futures basis, new-account growth, and margin-balance growth, each turned into a rolling 252-day percentile, then average them and split into high and low sentiment states. In the Chinese data only momentum and growth factors even showed a statistically detectable sentiment dependence, and timing on it still did not beat equal weight. ## Volatility timing, and the correlation that guts it The third recipe is inverse-volatility weighting. Give each factor a weight proportional to the reciprocal of its own volatility, so the calm factors get more capital and the wild ones get less, which lowers total portfolio volatility. $$ w\_k = \\frac{1/\\sigma\_k}{\\sum\_{j=1}^{K} 1/\\sigma\_j} $$ Sigma-k is factor k's return volatility. If three factors have volatilities of 2%, 3% and 6% a month, their reciprocals are 0.50, 0.33 and 0.167, summing to 1.0, so the weights are 0.50, 0.33 and 0.167\. The low-vol factor gets triple the weight of the high-vol one. Sensible in isolation, and it is genuinely a risk-reduction tool. But as a timing signal it is largely dead on arrival, because factor volatilities are not independent. Kapadia, Linn and Paye showed factor vols move together, and Shi confirms it: the four style factors in the Chinese five-factor model have an average pairwise volatility correlation of 0.65, and across 92 factors the average is still 0.55\. When every factor gets more volatile at the same time, tilting toward whichever is momentarily calmer is noise, not information. Traded in China, volatility timing produced negative excess returns over equal weight at every horizon, none significant. ![Bar chart of monthly excess return over equal weight for valuation, momentum, volatility, and sentiment timing in China, all with t-statistics below 2, against a note that equal weight itself earned 0.27 percent per month at t equals 3.98](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/08/article_406-china_timing_fails.png) Line the four up and the verdict is blunt. Even choosing the single most favorable lookback for each signal, the steelman case, not one clears a t-statistic of 2\. The honest read is not that these signals contain zero information ever, it is that the information is too weak and too correlated to survive real trading once you already hold the factors equally. ## The theory says timing should be a big deal Here is the twist that keeps factor timing alive as a research program. Asset pricing says a stochastic discount factor prices everything, and you can write it as one minus a loadings vector times the surprise in factor returns. $$ m\_{t+1} = 1 - \\delta\_t^\\top \\big(f\_{t+1} - \\mathbb{E}\_t\[f\_{t+1}\]\\big), \\qquad \\delta\_t = \\Sigma\_{f,t}^{-1}\\, \\mathbb{E}\_t\[f\_{t+1}\] $$ Here f-t-plus-1 is the vector of factor returns, E-t of f is what you expected given today's information, delta-t is the SDF's loadings, and Sigma-f-t is the conditional covariance of the factors. The optimal loadings are the inverse conditional covariance times the conditional expected returns, which is just the mean-variance efficient weights computed with time-varying inputs. The instant you let E-t of f move over time, you are doing factor timing, so timing is not some optional overlay, it sits inside the definition of the efficient portfolio. And Hansen and Jagannathan proved the maximum attainable Sharpe ratio is bounded by the variance of the SDF, which means anything that raises the SDF's variance raises the ceiling on how good a portfolio can be. Haddad and coauthors measured exactly that: moving from static factor investing to full factor timing lifted the average conditional SDF variance from 1.67 to 2.96, nearly doubling the theoretical Sharpe ceiling. On paper, timing is worth a fortune. ## Why the fortune does not show up in your account The reconciliation is that the predictable part of factor returns is concentrated in a few big dimensions, and everything else is noise you cannot trade. Haddad, Kozak and Santosh ran principal components on fifty anomaly portfolios. The first component alone explains 25.8% of the variance and the first five together explain 60%. Then they asked which components are actually forecastable using their own book-to-market spreads, and the answer is the dominant ones, not the obscure ones. ![Bar chart of in-sample and out-of-sample R-squared for predicting each of fifty principal components with its own book-to-market ratio, with the first few components boxed as the only ones showing meaningful out-of-sample predictability](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/08/article_406-pc_predictability.png) The big components, PC1 and PC4, show out-of-sample monthly R-squared of 4.82% and 3.52%, real and tradeable. The long tail of small components has R-squared indistinguishable from zero and often negative out of sample. This is the surprise the original authors emphasized, and it inverts the folk wisdom. The common intuition is that you time individual boutique factors on their own momentum or sentiment. The data says the only reliably timeable object is the handful of dominant common components, timed by valuation. That is why Shi's individual-factor timing in China fails while Haddad's "anomaly timing" of the big US components earns a positive out-of-sample information ratio of 0.60, against minus 0.64 for naive market timing. Timing works, narrowly, exactly where the naive practitioner is not looking. ## The plateau beats the peak Now connect it to allocation, because the failure of timing is really the failure of precision. The mean-variance efficient portfolio is weights proportional to inverse covariance times expected returns, and it delivers the maximum Sharpe ratio if your inputs are right. They never are. Shi says it plainly: the method is highly sensitive to its inputs and estimating expected returns and covariances ex ante is the hard part. So look at what happens when you actually trade the fancy optimizer against the dumb ones. ![Cumulative returns of equal weight, mean-variance efficient, maximum diversification, and equal risk contribution allocations against the market, with equal weight and equal risk contribution on top and the mean-variance optimizer lagging](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/08/article_406-allocation_methods.png) Equal weight and equal risk contribution finish at the top. The mean-variance efficient portfolio, the one that is theoretically optimal, actually lags them, because the estimation error in its inputs costs more than its cleverness earns. In Shi's stylized comparison the methods are identical when factors share Sharpe ratios and correlations, and the optimizer only pulls ahead in the fully general case, on paper, at Sharpe 0.670 versus equal weight's 0.504, which is precisely the case where the input sensitivity bites hardest in live trading. This is the old article ["Collinearity in Parameter Sweeps: Plateaus, Not Peaks"](https://aligrithm.com/collinearity-in-parameter-sweeps-plateaus-not-peaks/) applied to portfolios instead of parameter grids. There, the lesson was to trust a wide plateau of stable performance over a razor-thin peak that only exists because two parameters move together. Here, equal weight is the plateau: no inputs to estimate, nothing to overfit, robust by construction. Factor timing and mean-variance optimization are the peak: gorgeous at the estimated optimum, and standing on air the moment your estimates are off. None of this says timing is impossible. It says the honest baseline is brutal. Before you deploy any timing signal, you owe it a comparison against equal weight with a real out-of-sample window and Newey-West standard errors, and most signals will not clear the bar. The edge that survives is narrow, it lives in the big common components rather than individual factors, it is timed by valuation rather than momentum or sentiment, and it is small enough that transaction costs can erase it. Sell the plateau, not the peak. ## KEY POINTS - In Chuan Shi's Chinese study (2000 to 2020, top-1000 universe, seven long-leg factors), valuation, momentum, volatility, and sentiment timing all failed to beat an equal-weighted portfolio at every horizon, none reaching a t-statistic of 2. - The equal-weight benchmark itself was real, earning 0.27% a month over the market with a Newey-West t-statistic of 3.98\. The timing overlays added nothing to it. - Valuation timing weights a factor by where its own value spread sits in its historical range, comparing each factor only to itself, never across factors. - Inverse-volatility weighting is undercut by common volatility: factor vols carry an average pairwise correlation of 0.65 across four style factors and 0.55 across 92 factors, so tilting toward the calmer factor is mostly noise. - Theory says timing should be huge: the mean-variance-efficient SDF loadings are inverse conditional covariance times conditional expected returns, and moving from static investing to full timing lifted the average SDF variance from 1.67 to 2.96, nearly doubling the Hansen-Jagannathan Sharpe ceiling. - The predictability is concentrated: the first principal component of fifty anomalies explains 25.8% of variance, and only the dominant components (PC1, PC4) show tradeable out-of-sample R-squared (4.82% and 3.52%). The obscure components are noise. - Timing the big common components by valuation earns a positive out-of-sample information ratio (0.60) while naive market timing earns minus 0.64\. Timing works where the naive practitioner is not looking. - Equal weight and equal risk contribution beat the mean-variance optimizer in live trading because the optimizer's input-estimation error costs more than its cleverness earns. Prefer the robust plateau over the fragile peak. ## References - [Systematic Trading - Robert Carver (Amazon)](https://www.amazon.com/s?k=Systematic+Trading+Robert+Carver&ref=aligrithm.com) - [Evidence-Based Technical Analysis - David Aronson (Amazon)](https://www.amazon.com/s?k=Evidence-Based+Technical+Analysis+David+Aronson&ref=aligrithm.com) - [Factor Timing and Factor Allocation (ssrn-5264439)](https://papers.ssrn.com/sol3/papers.cfm?abstract%5Fid=5264439&ref=aligrithm.com) --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ### 6.58 Dual Momentum Between Gold and Bitcoin (Two Stores of Value) URL: https://aligrithm.com/dual-momentum-between-gold-and-bitcoin-two-stores-of-value/ Last updated: 2026-09-22T14:20:44.000Z An 8-week dual-momentum switch between GLD and IBIT returns 79.91% a year at a Sharpe of 1.64\. Vojtko and Dujava report that number on a Quantpedia note covering 31 December 2018 through April 2026, weekly rebalance at Wednesday's close, cash when both lookbacks print negative. Hold the other column: maximum drawdown is still -43.94%. They tested ten lookbacks and published the peak. The equity curve that produces the 80% sits with the rest of the grid until 2024, then rips. The rule is Antonacci relative-plus-absolute momentum on two tickers. The result is an in-sample trophy from the easiest Bitcoin stretch in the sample. I am against treating 79.91% as a live expectation. The number that a mandate could live with is the 20% volatility cap on their 4/8/12 composite: 12.01% a year, Sharpe 1.37, drawdown -12.27%. That one still comes off the same grid. ## Two stores of value, one blend that does not help Gold's above-ground stock grows about 1 to 2% a year, which is Erb and Harvey's supply argument for why the metal keeps a monetary premium. Bitcoin copies the scarcity story with a programmed halving. Baur, Hong and Lee treat Bitcoin as a speculative asset, not a medium of exchange. Corbet, Meegan, Larkin, Lucey and Yarovaya show that in stress it trades with risk assets, which is the opposite of a store of value. The Quantpedia note asks a narrower question: if you refuse the ideology and just rotate the two, does dual momentum beat holding either one, or a 50/50. They use GLD for gold. Bitcoin is a splice: Bitfinex hourly BTC, then BITO, then IBIT after the spot ETF, timestamped to GLD's NYSE close. IBIT listed in January 2024\. Everything before that is a proxy, not the live vehicle. ![Cumulative wealth of Bitcoin, GLD, and a 50/50 blend from late 2018 through early 2026](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/08/article_489-benchmark_btc_gld_5050.png) Bitcoin compounds at 46.66% a year with 64.01% volatility and a -77.49% drawdown, Sharpe 0.73 at a 0% risk-free rate. GLD does 18.80% at 17.09% vol and -20.71% drawdown, Sharpe 1.10\. The 50/50 prints 38.65% at 34.44% vol and still eats a -50.99% hole, Sharpe 1.12\. Check the ratios off their Table 1: 46.66 / 64.01 = 0.73, 18.80 / 17.09 = 1.10, 38.65 / 34.44 = 1.12\. Calmar is performance over the absolute drawdown, so Bitcoin 46.66 / 77.49 = 0.60, gold 18.80 / 20.71 = 0.91, 50/50 38.65 / 50.99 = 0.76\. The blend captures Bitcoin's direction and a large fraction of Bitcoin's crash. Equal weights do not solve a 64-versus-17 volatility gap. ## Relative plus absolute, never both Antonacci's 2016 dual-momentum rule has two gates. Relative momentum picks the winner between the two assets over a lookback of X weeks. Absolute momentum requires that winner's lookback return to be positive, otherwise the book goes to cash at 0%. They never hold both. It is a switch, not a blend. $$ w\_t = \\begin{cases} \\text{IBIT} & \\text{if } R^{\\text{IBIT}}\_{t-X:t} > R^{\\text{GLD}}\_{t-X:t} \\text{ and } R^{\\text{IBIT}}\_{t-X:t} > 0 \\\\ \\text{GLD} & \\text{if } R^{\\text{GLD}}\_{t-X:t} > R^{\\text{IBIT}}\_{t-X:t} \\text{ and } R^{\\text{GLD}}\_{t-X:t} > 0 \\\\ \\text{cash} & \\text{otherwise} \\end{cases} $$ R-IBIT over t-minus-X to t is IBIT's total return across the last X weeks, same for GLD. At each Wednesday close the rule compares those two numbers to each other and to zero, then holds one name or nothing until next Wednesday. Worked reading with X = 8\. IBIT +12% over eight weeks, GLD +3%: relative winner is IBIT, absolute gate is open, so 100% IBIT. IBIT -8%, GLD +2%: long GLD. IBIT -8%, GLD -3%: both fail the zero test, cash. IBIT +5%, GLD +7%: long GLD. The cash state is the crash brake. It is also an 8-week delayed crash brake, which is why a -43.94% drawdown survives the "otherwise cash" clause. Bitcoin's 2022 drawdown ran for months. An 8-week window stays positive deep into a decline, then flips after the damage is already on the sheet. X is the only free parameter. They try 1, 2, 3, 4, 6, 8, 12, 20, 24 and 28 weeks. Ten specs, one sample, no walk-forward. The old article ["Percentile-Rank Momentum With Hysteresis: Low-Churn Signals"](https://aligrithm.com/percentile-rank-momentum-with-hysteresis-low-churn-signals/) attacked the same problem from the other side: rank a move against its own signed history and put a hysteresis band on entries so the signal stops flip-flopping, then check the rule on a grid of walk-forward splits. This note does the opposite. It mines a single lookback on one path and calls the peak a sweet spot. ## Ten lookbacks, one trophy ![Cumulative wealth of the pure dual-momentum switch across lookbacks of 1 to 28 weeks, plus a 4/8/12 composite](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/08/article_489-pure_dual_momentum_equity.png) Table 2, pure dual momentum, 31 December 2018 to April 2026: | X (weeks) | 1 | 2 | 3 | 4 | 6 | 8 | 12 | 20 | 24 | 28 | Comp. | | --------- | ------- | ------- | ------- | ------- | ------- | ----------- | ------- | ------- | ------- | ------- | ------- | | Ann. % | 22.04 | 45.21 | 47.22 | 52.00 | 68.66 | **79.91** | 56.09 | 40.31 | 23.03 | 22.75 | 64.73 | | Vol % | 45.48 | 45.68 | 47.28 | 47.78 | 46.12 | 48.85 | 47.88 | 48.68 | 47.25 | 46.55 | 44.14 | | Sharpe | 0.48 | 0.99 | 1.00 | 1.09 | 1.49 | **1.64** | 1.17 | 0.83 | 0.49 | 0.49 | 1.47 | | Max DD % | \-65.19 | \-69.42 | \-63.78 | \-66.78 | \-48.23 | **\-43.94** | \-49.98 | \-59.32 | \-66.02 | \-72.84 | \-49.40 | | Calmar | 0.34 | 0.65 | 0.74 | 0.78 | 1.42 | **1.82** | 1.12 | 0.68 | 0.35 | 0.31 | 1.31 | Comp. is an equal-weight mix of the 4-, 8- and 12-week variants. Check the bold column: 79.91 / 48.85 = 1.64, 79.91 / 43.94 = 1.82\. Composite: 64.73 / 44.14 = 1.47, 64.73 / 49.40 = 1.31\. The arithmetic matches the table. The 50/50 Sharpe is 1.12\. Seven of the ten lookbacks print Sharpe at or below 1.09\. The three that beat 1.12 are 6, 8 and 12 weeks. Their "composite to mitigate parameter sensitivity" averages 4, 8 and 12, which is the interior of that cluster plus the 4-week edge that almost ties the blend. It is not an average of the grid. It is an average of the specs that work. ![Sharpe and maximum drawdown against lookback length for the pure switch and the 20% vol-capped switch, with the 50/50 Sharpe as a dashed line](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/08/article_489-lookback_grid.png) The top panel is an inverted U with a spike at 8 weeks. The bottom panel never gets the pure switch's drawdown above -43%. Volatility across lookbacks sits in a tight band around 46 to 49%, so the Sharpe ranking is almost a return ranking. The strategy does not change risk. It changes how often you are in Bitcoin during a Bitcoin bull. Read the equity chart against that grid. The 8-week line (teal) tracks the pack through 2023 and then leaves everyone behind in 2024-2026, finishing near 80x. A 79.91% geometric rate over about 7.3 years compounds to roughly (1.7991)^7.33 ≈ 74x, which is the same picture as the chart's "near 80" once you allow for lookback burn-in and a visual read of the last print. The outperformance is concentrated in the last two years of the sample, which is also when the authors wrote the note (6 May 2026, sample through April 2026). ## The 20 percent cap is the actual product Bitcoin's 64% vol against gold's 17% means a 100% switch into IBIT is an equity-like book one week and a crypto book the next. They add a hard 20% annualized volatility ceiling, not a target. After the dual-momentum signal picks IBIT or GLD, they take that name's 12-week rolling weekly standard deviation, annualize it, and scale the position. $$ \\sigma^{\\mathrm{ann}}\_t = \\sigma^{\\mathrm{weekly}}\_{t-12:t} \\times \\sqrt{52}, \\qquad w\_t = \\min\\left(\\frac{0.20}{\\sigma^{\\mathrm{ann}}\_t},\\, 1\\right) $$ Sigma-weekly is the standard deviation of the selected asset's Wednesday-to-Wednesday returns over the last 12 weeks. Square root of 52 is about 7.21, so a weekly sigma of 8.87% annualizes to 8.87 × 7.21 ≈ 64%, Bitcoin's full-sample vol. The weight is then 0.20 / 0.64 = 0.3125, so 31% in IBIT and 69% in cash. Their own examples: 23% annualized vol → 20/23 ≈ 87% invested, 13% cash; 15% vol → 100%, because the min with 1 caps the book at fully invested; 40% vol → 50%. Units cancel: percent over percent. The 20% is an arbitrary institutional round number. They say so. It is chosen to look like equity risk. Table 3, same dates, 20% cap: | X (weeks) | 1 | 2 | 3 | 4 | 6 | 8 | 12 | 20 | 24 | 28 | Comp. | | --------- | ------- | ------- | ------- | ------- | ------- | ----------- | ------- | ------- | ------- | ------- | ----------- | | Ann. % | 5.75 | 8.85 | 9.95 | 10.57 | 12.69 | **14.36** | 10.90 | 8.61 | 5.75 | 5.90 | 12.01 | | Vol % | 9.10 | 9.14 | 9.48 | 9.55 | 9.21 | 9.75 | 9.50 | 9.70 | 9.39 | 9.24 | 8.77 | | Sharpe | 0.63 | 0.97 | 1.05 | 1.11 | 1.38 | **1.47** | 1.15 | 0.89 | 0.61 | 0.64 | 1.37 | | Max DD % | \-16.58 | \-18.63 | \-16.28 | \-17.36 | \-11.69 | **\-10.78** | \-10.46 | \-13.70 | \-17.85 | \-19.85 | **\-12.27** | | Calmar | 0.35 | 0.48 | 0.61 | 0.61 | 1.09 | **1.33** | 1.04 | 0.63 | 0.32 | 0.30 | 0.98 | 8-week column: 14.36 / 9.75 = 1.47, 14.36 / 10.78 = 1.33\. Composite: 12.01 / 8.77 = 1.37, 12.01 / 12.27 = 0.98\. Again the table is internally consistent. The cap takes the composite from 64.73% at 44.14% vol and -49.40% drawdown down to 12.01% at 8.77% vol and -12.27% drawdown. Realized vol at 8.77% is less than half the 20% ceiling, and less than gold's standalone 17.09%. That gap is the cash. Absolute momentum parks the book when both lookbacks are negative, and the cap itself cuts IBIT to \~30% whenever Bitcoin is running at 60%+ vol. The Sharpe holds up (1.47 → 1.37 on the composite) because return and vol fall together. The Calmar does not: 1.31 down to 0.98\. Drawdown compression is smaller, relative to the return haircut, than vol compression. ## What the backtest leaves on the table The Sharpe uses a 0% risk-free rate. They print that in the table note. From 2023 onward T-bills paid around 5%. Subtract 5 points from the capped composite and keep vol at 8.77: (12.01 - 5) / 8.77 = 0.80\. The pure 8-week is more robust to that haircut because 79.91 is large: (79.91 - 5) / 48.85 = 1.53\. The product a risk desk would fund is the capped one, and that is the one the zero-rate convention flatters. No transaction costs. Weekly rotation between two liquid ETFs is cheap, so this is not the usual crypto-cost ambush. It is still a missing line. A Wednesday GLD-to-IBIT switch pays two spreads plus any premium/discount on the ETF. BITO, the pre-IBIT sleeve, is a futures fund with roll drag the Bitfinex BTC series does not have. The backtest is cleaner than the live book on the Bitcoin leg until January 2024. Ten lookbacks on one 7-year path, then a composite built from the three that cluster at the peak. No out-of-sample split, no permutation of the Wednesday clock, no embargo. The 2024-2026 burst in Figure 2 is inside the estimation sample. Dual momentum as a framework can be sound and this particular 8-week print can still be a fitted spike. Those two statements do not contradict. The authors disclose that Qwen3.6-Plus helped with LaTeX, structure, and typesetting, and they claim the research design and numbers are theirs. Take them at their word on the tables. The disclosure does not change the lookback grid. One more structural limit they do state. When Bitcoin and gold both sell off, relative momentum has nothing to rotate into, and absolute momentum is only as fast as X. The 2022 risk-off window is why the pure switch still prints drawdowns of 44 to 73% depending on lookback. The cap is what turns that into a -12% event. If the pitch is "digital gold plus physical gold, systematically combined," the honest version is "rotate them, sit in cash when both lookbacks are negative, and never let the book run above 20% vol." The 79.91% is what happens if you skip the last clause and pick the lookback after you see the path. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/08/article_489-visual-0.png) ## KEY POINTS - Dual momentum here is a weekly Wednesday switch between GLD and IBIT, long the lookback winner only if that lookback return is positive, else cash. Never both names at once. - Buy-and-hold: Bitcoin 46.66% a year, vol 64.01%, Sharpe 0.73, drawdown -77.49%. GLD 18.80%, 17.09%, 1.10, -20.71%. 50/50 38.65%, 34.44%, 1.12, -50.99%. The blend does not fix Bitcoin's crash. - Pure 8-week switch: 79.91% a year, Sharpe 1.64, drawdown -43.94%, Calmar 1.82\. That lookback is the peak of a 10-spec in-sample grid. Seven of ten lookbacks fail to beat the 50/50 Sharpe of 1.12. - The 4/8/12 composite (64.73%, Sharpe 1.47, DD -49.40%) averages the interior of the peak, not the full grid. The 8-week equity curve separates from the pack in 2024-2026. - A 20% vol cap, weight = min(0.20 / annualized 12-week vol, 1), turns the composite into 12.01% a year, Sharpe 1.37, drawdown -12.27%, realized vol 8.77%. Realized vol sits under the cap because of cash from the absolute gate plus IBIT being sized to \~30% at Bitcoin-like vol. - Sharpe is computed at rf = 0\. Haircut the capped composite by a 5% bill rate and Sharpe falls to about 0.80\. Pre-January 2024 Bitcoin is a Bitfinex/BITO proxy, not IBIT. No costs, no out-of-sample split. ## References - [Risk Premia Harvesting Through Dual Momentum](https://ssrn.com/abstract=2042750?ref=aligrithm.com) - [The Golden Dilemma](https://ssrn.com/abstract=2078535?ref=aligrithm.com) - [Bitcoin: Medium of Exchange or Speculative Asset?](https://econpapers.repec.org/RePEc:eee:intfin:v:54:y:2018:i:c:p:177-189?ref=aligrithm.com) - [Exploring the Dynamic Relationships Between Cryptocurrencies and Other Financial Assets](https://econpapers.repec.org/RePEc:eee:ecolet:v:165:y:2018:i:c:p:28-34?ref=aligrithm.com) - [Dual Momentum Allocation Between Physical Gold and Bitcoin (Digital Gold)](https://papers.ssrn.com/sol3/papers.cfm?abstract%5Fid=6729918&ref=aligrithm.com) --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ### 3.42 Stat-Arb Without Cointegration: Moving-Band Portfolios via Convex-Concave Optimization URL: https://aligrithm.com/stat-arb-without-cointegration-moving-band-portfolios-via-convex-concave-optimization/ Last updated: 2026-10-14T08:28:35.000Z Every pairs-trading tutorial starts the same way. Find two assets whose prices moved together, run an Engle-Granger cointegration test, confirm the spread is stationary, then trade the spread back to its mean. It works often enough to keep the ritual alive, but the objective is quietly wrong. Cointegration asks whether a combination of prices is stationary. It never asks whether that combination swings enough to pay you. A dead-flat stationary spread passes every test and earns nothing. Kasper Johansson, Thomas Schmelzer, and Stephen Boyd threw the whole testing apparatus out and optimized for the thing you actually get paid for: a portfolio price that stays inside a band and thrashes across it as violently as possible. No cointegration, no stationarity test, no exhaustive pair search. Just an optimization problem, and one clever trick to solve it. The payoff is a method that finds stat-arbs with ten assets instead of two, scales to a universe of fifteen thousand stocks, and comes with a second idea, the moving band, that keeps the trade alive far longer out of sample. On fourteen years of CRSP data it produced hundreds of tradeable portfolios. The catch, as always, is in the out-of-sample line, and this article spends real time there. ## The objective is the profit, so optimize it directly Start with the setup. You have a universe of n assets with a price vector at each day, and you want a share portfolio, call it s, with short positions allowed as negative entries. The portfolio price on any day is s-transpose times the price vector, a single number that can go negative because s can. You want two things at once from that price series: it should stay inside a band, and it should move across the band constantly. The first property is mean reversion. The second is what pays. Here is the whole problem. $$ \\begin{array}{ll} \\text{maximize} & \\sum\_{t=2}^{T} (p\_t - p\_{t-1})^2 \\\\\[4pt\] \\text{subject to} & -1 \\le p\_t - \\mu \\le 1, \\quad p\_t = s^{\\top} P\_t, \\quad t = 1, \\ldots, T \\\\\[2pt\] & |s|^{\\top} \\bar{P} \\le L, \\quad \\mu \\ge 0 \\end{array} $$ Read it in plain language. Maximize the sum of squared day-to-day price changes, which is the portfolio's total price variation. Hold the price within one unit of the band midpoint mu on every day, so it never escapes the band. Cap the leverage, the average dollar size of the position, at a limit L, which is a weighted sum of absolute holdings and therefore pushes most weights to zero and leaves you a sparse portfolio of a handful of names. The band width is fixed at 2 because you can always rescale the shares and midpoint to make it so. Work the number to feel why the objective is right. Take two portfolios that both obey the band. Portfolio A barely moves, drifting a hundredth of a unit per day, so each squared change is about 0.0001 and over 62 days the objective sums to roughly 0.0006\. Portfolio B slams from the bottom of the band to the top and back, a change of about 2 units each day, so each squared change is about 4, and over 62 days the objective is about 248\. Both are stationary. Both would pass a cointegration test. Only B is worth trading, and the objective ranks B four hundred thousand times higher than A. Cointegration cannot see that difference. This objective is built from nothing but that difference. ## Why maximizing variation inside a band is exactly the trade The objective is not a proxy. It is the trading profit, almost exactly, and the paper proves it with a simple policy. Hold a quantity of the portfolio equal to the midpoint minus the current price. When the price sits at the bottom of the band, one unit below mu, you hold plus one and you are long. When it sits at the top, one unit above, you hold minus one and you are short. Buy low, sell high, mechanically. _This post is for paying subscribers only._ ### 1.27 How to Spot a Fake ML Trading Paper URL: https://aligrithm.com/how-to-spot-a-fake-ml-trading-paper/ Last updated: 2026-08-15T06:37:02.000Z Three papers crossed my desk this month. One detects market manipulation with graph neural networks and reports 98.7% accuracy at 8.3 milliseconds per decision. A second, same author cluster, detects anomalies in high-frequency trading and reports a 15% accuracy improvement with an F1 of 0.915\. A third uses XGBoost on Bitcoin technical indicators and reports 92.40% accuracy with a ROC AUC of 0.9817\. All three are real, published, and citable. All three are worthless as trading evidence, and every one of them betrays itself with numbers printed in its own tables. Set them next to the one honest paper in the same stack. Mathias Mesfin took fourteen popular intraday setups, wired them to next-bar fills and two points of cost on Micro Nasdaq futures, and ran an honest gauntlet across 947 days. Zero survived. That study is the subject of the old article ["The Signal Ceiling: Why No Single-Bar OHLCV Edge Beats Costs in MNQ,"](https://aligrithm.ghost.io/the-signal-ceiling-why-no-single-bar-ohlcv-edge-beats-costs-in-mnq/?ref=aligrithm.com) and it is the control specimen for this field guide. The fakes report 92% and 98.7%. The honest paper reports zero. The gap between them is the whole lesson. This is a forensic checklist. Six red flags, each with a worked number you can compute from the paper's own reported figures before you trust a single claim. ## Red flag one: the accuracy measures the wrong thing Start with the Hafid XGBoost paper because its leak is the cleanest to see. The authors build a target label with a moving-average crossover rule: buy when the 10-bar moving average of Bitcoin's close sits above the 60-bar moving average, sell when it does not. Then they feed the model features that include exponential moving averages, the stochastic oscillator %K and %D, momentum, and RSI, all computed from that same closing price. The label is a smoothed function of the price. The features are smoothed functions of the same price. The model predicts one from the other. $$ \\mathcal{Y}(t\_i) = \\begin{cases} 1 & \\text{if } \\mathrm{MA}\_{10}(t\_i) \\ge \\mathrm{MA}\_{60}(t\_i) \\\\ -1 & \\text{if } \\mathrm{MA}\_{10}(t\_i) < \\mathrm{MA}\_{60}(t\_i) \\end{cases} \\qquad \\text{persistence accuracy} = \\Pr\\!\\big(\\mathcal{Y}(t\_i) = \\mathcal{Y}(t\_{i-1})\\big) $$ The label is 1 when the short moving average is at or above the long one, minus 1 otherwise. The persistence accuracy is the fraction of bars where the label equals the previous bar's label, which is what a model with zero information scores by predicting "same as last bar." A crossover of a 10-bar and a 60-bar average flips rarely, so the state persists for long stretches. I simulated a 15-minute Bitcoin-like series, built the exact MA(10,60) label, and measured the no-information persistence baseline: 97.5%. That beats the paper's 92.4%. A model that ignores every feature and echoes the last bar scores higher than the fine-tuned XGBoost the paper spent nine hyperparameters tuning. ![Bar chart comparing a coin flip, majority-class, and zero-information persistence baseline against the paper's reported 92.4% XGBoost accuracy on the moving-average-crossover label](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_435-leakage_baseline.png) The 92% is real. It measures the persistence of a deterministic rule, not a forecast of anything you can trade. The test to run: ask what the label is a function of. If the target is computed from the same price series as the features, the accuracy is circular, and the correct baseline is not a coin flip, it is the persistence of the label itself. The old article ["Technical Analysis as a Scientific Hypothesis"](https://aligrithm.com/technical-analysis-as-a-scientific-hypothesis/) makes the general point: a rule is a hypothesis only when it produces a falsifiable prediction about something not already baked into its inputs. ## Red flag two: an accuracy with no base rate attached The multi-modal GNN paper reports 98.7% accuracy detecting manipulation. Its own dataset table gives you the number that kills it: on NYSE, 8.2 billion transactions contained 12,847 labeled manipulations. That is a base rate of one manipulation per 638,000 clean transactions. $$ \\text{Accuracy} = \\frac{TP + TN}{TP + TN + FP + FN} \\qquad \\text{Accuracy}\_{\\text{do-nothing}} = 1 - p, \\quad p = \\frac{12{,}847}{8.2 \\times 10^{9}} \\approx 1.57 \\times 10^{-6} $$ Accuracy is the share of all decisions the model gets right. A do-nothing classifier that flags every transaction as clean gets a share equal to 1 minus the prevalence of the positive class. Plug in the paper's base rate: 1 minus 0.00000157 is 99.99984%. A model that detects nothing, that is a single line of code returning "clean," scores 99.99984%, which is 1.3 percentage points above the paper's headline 98.7%. Reporting raw accuracy on a needle-in-a-haystack problem is meaningless, and worse, the paper's own number sits below the trivial baseline. ![Accuracy of a do-nothing classifier as a function of class prevalence, showing that at a one-in-638,000 base rate the trivial model scores 99.99984 percent, above the paper's claimed 98.7 percent](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_435-accuracy_trap.png) The same paper reports precision of 97.9%, and here the arithmetic gets funnier. Precision at that base rate demands an almost supernatural false-positive rate. $$ \\text{Precision} = \\frac{\\text{TPR}\\cdot p}{\\text{TPR}\\cdot p + \\text{FPR}\\cdot (1-p)} \\;\\Rightarrow\\; \\text{FPR} = \\frac{\\text{TPR}\\cdot p\\,(1-\\text{Precision})}{\\text{Precision}\\,(1-p)} $$ Precision is the share of flagged transactions that are truly manipulation. Solve the identity for the false-positive rate the paper needs. With a true-positive rate of 0.982, a base rate of 0.00000157, and a claimed precision of 0.979, the required false-positive rate works out to about 3.3 times 10 to the minus 8\. That means fewer than one false alarm per 30 million clean transactions. No graph neural network on noisy tick data flags clean flow that cleanly. When a paper reports high precision and high recall on an extreme-imbalance problem without showing you the false-positive rate in raw counts, compute the implied FPR yourself. If it lands below one in a million, the labels or the split are broken. ## Red flag three: no cost model anywhere None of the three papers subtracts a transaction cost. Not one reports a Sharpe ratio, a profit factor, or a dollar of net P&L. They report classification metrics and stop. The old article ["The Signal Ceiling: Why No Single-Bar OHLCV Edge Beats Costs in MNQ"](https://aligrithm.ghost.io/the-signal-ceiling-why-no-single-bar-ohlcv-edge-beats-costs-in-mnq/?ref=aligrithm.com) showed why that omission is fatal on real instruments, where the entire finding was that gross edge tops out below the cost of trading it. $$ \\text{Net}\_{\\text{trade}} = \\text{Gross}\_{\\text{trade}} - \\text{Friction}\_{\\text{round-trip}} $$ Net return per trade is the gross directional move captured minus the cost of getting in and out once. Run it on the Hafid signal. The MA(10,60) label flips whenever the crossover changes state; on the paper's 15-minute cadence that is on the order of 150 to 200 round trips across the test window. Binance taker fees plus spread run roughly 15 basis points round-trip. A signal that flips 180 times pays about 27% of notional in friction over the period, and the paper never demonstrates a gross edge large enough to clear that, because it never measures a forward return at all. A directional-accuracy number that is never converted into net-of-cost P&L is not a trading result. The test: search the PDF for "transaction cost," "slippage," "Sharpe," and "net." If all four are absent, the paper has not shown an edge, it has shown a curve-fit. ## Red flag four: figures that illustrate nothing Open the anomaly-detection GNN paper's data figure and look at what it actually shows. ![Multi-panel figure from the anomaly-detection paper: five near-identical symmetric violin plots of daily trading volume, an anomaly heatmap that looks like uniform noise, and a feature-correlation matrix](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_435-synthetic_looking_figure.png) Five stocks, five violin plots of daily trading volume, all five near-identical symmetric blobs. Real trading volume is heavily right-skewed with fat upper tails; AAPL and INTC do not share a distribution shape. The anomaly heatmap beside it is a wash of uniform random color with no temporal structure, and the correlation matrix shows the tidy block pattern you get from synthetic Gaussians, not from real market features. Across all three papers the figure captions repeat the same boilerplate: "the visualization employs a hierarchical layout with color-coded components," "mathematical formulas are included," "confidence intervals around the curves." Prose describing a figure that carries no data. A real results figure lets you read a number off an axis. These let you read nothing. The test: pick one figure and try to extract a single quantity from it. If you cannot, it is decoration standing in for evidence. ## Red flag five: the reference list is a citation ring The reference lists are where the paper mill leaves fingerprints. The multi-modal manipulation paper, a study about detecting market manipulation, cites work on monoclonal antibody production, drug discovery, medical CT image captioning, P2P microloan default, and acute ischemic stroke thrombolysis in the McGill Journal of Medicine. The anomaly-detection paper's acknowledgments thank colleagues for research on IoT network traffic and cloud data-center energy optimization. The same names cycle through both author lists and both bibliographies, publishing across finance, oncology, and Kubernetes in the same year, in journals with names like the Journal of Artificial Intelligence General Science. This is self-citation as camouflage. The references exist to inflate a citation count inside a closed group, not to build on prior manipulation-detection results, because there is no coherent prior literature being extended. The test: sample five references and check whether they are about the paper's actual topic. If a market-surveillance paper cites antibodies and strokes, the reference list is a ring, and the old article ["Alpha Decay Is Just Competition (and Papers Lie)"](https://aligrithm.com/alpha-decay-is-just-competition-and-papers-lie/) tells you why the incentive exists: publication and citation reward the appearance of results, and nothing in the pipeline checks whether the results survive contact with a market. The fabrication can be blunter. The Hafid paper's comparison table lists Lin et al. (2020) with an accuracy of 92.4%, the exact figure the current paper reports for itself, alongside feature names lifted from the current paper. Lin's actual reported accuracy, stated three pages earlier in the same paper's own related-work section, is 59.4%. The comparison row is copied from the authors' own result. When a paper's "comparison with prior work" hands a competitor the paper's own headline number, someone filled the table by pattern, not by reading the cited work. ## Red flag six: round numbers and no null The deepest tell is structural, and it is the one the old article "Why "Quantitative" Does Not Automatically Mean Scientific" is built around. A scientific result states a null hypothesis, a benchmark, and a rejection threshold before running the test. None of these three papers states a null. None runs a permutation test. None corrects for the many model and feature combinations it searched. The accuracies arrive clean and round, 92.40% and 98.7%, with no distribution behind them and no account of how many configurations were tried before those landed. $$ \\mathbb{E}\\big\[\\max\_{1 \\le k \\le K} |t\_k|\\big\] \\approx \\sqrt{2 \\ln K} $$ Under the null of no edge, the largest t-statistic you expect to see across K independent tries grows like the square root of twice the natural log of K. Try 20 feature-and-model combinations on noise and the best one shows an expected t near 2.45, comfortably past the naive 1.96 threshold that says "significant." This is why Harvey, Liu, and Zhu argue the hurdle for a genuine factor claim should sit near 3.0, not 2.0, once you account for the hundreds of specifications the profession quietly searched. A paper that reports one round accuracy from an unstated search has not cleared any hurdle, because it never built the null distribution to clear. ## What the honest paper does instead Put the control specimen back on the table. The MNQ falsification study fixes every hole above before it reports a result. It fills every entry at the next bar's open, so no signal touches its own entry price. It subtracts two points of round-trip friction on every trade. It demands all five gates at once: an out-of-sample t-statistic at or above 2.0, at least 30 trades, positive net return after cost, stability across every year, and a permutation p-value below 0.05\. It plants two known-good signals as positive controls and confirms the harness accepts them, at t-statistics of 5.83 and 5.15\. Then it reports the finding without spin: fourteen popular setups, zero survivors. That is what a real ML trading result looks like, and the contrast is the point of this whole desk. The fakes are easy to produce and easy to publish; a language model can generate the prose, the tables, and the boilerplate figures in an afternoon, and a predatory journal will print it. The honest version is expensive, it usually returns a null, and the null is the value. When you cannot tell the two apart from the abstract, run the six tests: check what the label is a function of, compute the do-nothing accuracy from the base rate, search for a cost model, try to read a number off a figure, sample the reference list, and look for a stated null. Any single failure is enough to set the paper down. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_435-visual-0.png) ## KEY POINTS - Three published ML trading papers report 98.7%, 92.4%, and a 15% improvement. All three fail on their own printed numbers. High accuracy is the first thing to distrust, not the last thing to celebrate. - Check what the label is a function of. The Hafid XGBoost paper predicts a moving-average crossover from features built on the same price; a zero-information "predict last bar" baseline scores 97.5%, above the reported 92.4%. Circular labels manufacture accuracy. - Compute the do-nothing accuracy from the base rate. At one manipulation per 638,000 transactions, a model that flags nothing scores 99.99984%, higher than the multi-modal paper's 98.7% headline. Raw accuracy on an imbalanced problem is a non-result. - Solve for the implied false-positive rate. The multi-modal paper's 97.9% precision at that base rate requires fewer than one false alarm per 30 million clean transactions, which no noisy classifier delivers. - Search for a cost model. None of the three subtracts friction or reports net P&L, a Sharpe, or a profit factor. Net equals gross minus cost, and a directional-accuracy number never converted to net-of-cost return is not a trading edge. - Try to read a number off the figures. Synthetic-looking violin plots, noise heatmaps, and boilerplate captions ("hierarchical layout with color-coded components") are decoration, not evidence. - Sample the reference list. A market-surveillance paper citing monoclonal antibodies and stroke thrombolysis, with the same author cluster recycling across bibliographies, is a citation ring. One paper even copied its own 92.4% into a competitor's comparison row. - Look for a stated null. Under noise, the best of 20 tries shows an expected t near 2.45, which is why the real hurdle sits near 3.0\. A single round accuracy from an unstated search clears nothing. The honest MNQ study states its null, subtracts costs, plants positive controls, and reports zero survivors. ## References - [Evidence-Based Technical Analysis - David Aronson (Amazon)](https://www.amazon.com/s?k=Evidence-Based+Technical+Analysis+David+Aronson&ref=aligrithm.com) - [Systematic Trading - Robert Carver (Amazon)](https://www.amazon.com/s?k=Systematic+Trading+Robert+Carver&ref=aligrithm.com) - [\[teardown target\] Multi-modal Market Manipulation Detection in High-Frequency Trading Using Graph Neural Networks (Multi-modal Market Manipulation Detection in High)](https://scholar.google.com/scholar?q=Multi-modal+Market+Manipulation+Detection+in+High-Frequency+Trading+Using+Graph+Neural+Networks&ref=aligrithm.com) - [\[teardown target\] Anomaly Pattern Detection in High-Frequency Trading Using Graph Neural Networks (Anomaly Pattern Detection in High-Frequency Trading)](https://scholar.google.com/scholar?q=Anomaly+Pattern+Detection+in+High-Frequency+Trading+Using+Graph+Neural+Networks&ref=aligrithm.com) - [\[teardown target\] Predicting Market Trends with Enhanced Technical Indicator Integration and Classification Models (predicting market trends with indicators and classification)](https://scholar.google.com/scholar?q=Predicting+Market+Trends+with+Enhanced+Technical+Indicator+Integration+and+Classification+Models&ref=aligrithm.com) - [Structural Limits of OHLCV-Based Intraday Signals in MNQ Futures: A Systematic Falsification Study (ssrn-6709401)](https://papers.ssrn.com/sol3/papers.cfm?abstract%5Fid=6709401&ref=aligrithm.com) --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ### 8.10 Stop Using Pairwise Granger: PCMCI for Financial Causality URL: https://aligrithm.com/stop-using-pairwise-granger-pcmci-for-financial-causality/ Last updated: 2026-08-15T06:37:03.000Z Run pairwise Granger tests over a basket of thirty instruments and you get a causal network that looks like a plate of spaghetti. AUD/USD drives copper, copper drives the Aussie back, oil leads the loonie, the loonie leads oil, and somewhere in the mess a small-cap ETF appears to Granger-cause the ten-year yield. Most of those arrows are fake. They come from three things markets are drowning in: common drivers that move everything at once, autocorrelation that never dies, and indirect chains that get read as direct links. The old article ["Granger Causality: Finding What's Driving Your Currency Right Now"](https://aligrithm.com/granger-causality-finding-whats-driving-your-currency-right-now/) showed how to run the bivariate test cleanly. This article is the sequel that has to be said: the bivariate test, run across many series, is a false-positive machine, and the obvious fix makes it worse. Jakob Runge and coauthors hit exactly this wall in climate science, where you have hundreds of interdependent, strongly autocorrelated series and a few hundred observations. Their answer, PCMCI, is a two-stage causal discovery algorithm that keeps detection power high and false positives controlled where plain Granger falls apart. The setting they solved is the setting a quant lives in. Steal the method. ## Why your pairwise Granger network is mostly noise Start with the disease before the cure. Take a monthly climate index, Nino sea-surface temperature, and land air temperature over British Columbia, 468 months of data. The correlation between them runs about 0.3 at a two-month lag with a p-value under ten to the minus four. Strong, significant, and useless for causality, because both directions light up: temperature appears to correlate with Nino at negative lags too, which is physically impossible on that timescale. Correlation cannot tell you direction, and it cannot even tell you the right lag. The same trap sits in any financial cross-section. A risk-on regime lifts the Australian dollar and copper together. Run a pairwise test on just those two and you find a clean, significant, bidirectional link. There is no direct causal edge between them at all. The regime is a common driver, and the test has no way to see it because it only ever looks at two series at a time. ![Left panel, a common risk regime drives both AUD/USD and copper, producing a spurious dashed link between them; right panel, PCMCI conditions on the regime and the spurious link disappears](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_429-spurious_common_driver.png) The fix your instincts reach for is to add controls. Condition on the past of every other series so the common drivers get accounted for. That is the right idea and it is the trap this whole article is about. ## The dimensionality trap kills the honest test Conditioning on the full past of every variable has a name in the paper: full conditional independence testing, FullCI. It is the rigorous, multivariate version of Granger causality. To test whether series i at lag tau drives series j now, you condition on the entire past of all N series up to the maximum lag, minus the one link you are testing. $$ X^{i}\_{t-\\tau} \\perp\\!\\!\\!\\perp X^{j}\_{t} \\;\\big|\\; \\mathbf{X}^{-}\_{t} \\setminus \\{X^{i}\_{t-\\tau}\\} $$ Read it as: driver at a lag is independent of the target now, given the past of everything else. The conditioning set is the whole history of the system, so the test controls for every common driver and indirect path you have measured. That is exactly what you wanted. The problem is the size of that conditioning set. With N series and a maximum lag of tau\_max, you are conditioning on roughly N times tau\_max minus one variables in a single test. Work the number a quant would actually hit. Thirty instruments, five daily lags, so the conditioning set is 30 times 5 minus 1, which is 149 variables. If you have one trading year of clean daily data, call it 250 observations, you are fitting a relationship with 149 controls on 250 points. The degrees of freedom are almost gone, and the effect size you are trying to detect gets diluted by every irrelevant control you throw in. In the climate example the true link Nino to British Columbia had a FullCI partial correlation of about 0.1 with a p-value of 0.037 in the bare bivariate case. Add one variable that is itself driven by Nino and the effect size drops to 0.09, and the link is detected in only 53% of realizations. Add six more autocorrelated series and detection power falls to 40%. The link never changed. The test just went blind. ![Detection power against network size, FullCI collapsing from 80 percent at five variables to 40 percent at twenty and unusable beyond thirty, while PCMCI stays above 70 percent out to a hundred variables](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_429-power_vs_dimension.png) The pattern is brutal and monotone. FullCI holds about 80% power at five variables, drops to 40% at twenty, and simply cannot run once N times tau\_max exceeds your sample size. So you are stuck between two bad options: bivariate Granger that reports spurious links, or full-conditioning Granger that goes deaf the moment the network gets interesting. That is the dilemma, and adding more data is rarely the escape because markets are non-stationary and last year's regime is not this year's. ## PCMCI step one: find the few parents that matter The way out is to stop conditioning on everything and condition only on the variables that actually matter, the direct causes. Causal discovery theory gives you a clean result: the parents of a variable, its direct causal drivers, are a sufficient set to condition on. You do not need the whole history. You need the handful of series that genuinely drive the target. PCMCI's first stage, called PC1, is a condition-selection algorithm that hunts down that handful. It starts by assuming every lagged variable is a potential parent, then iteratively strips out the ones that wash out. First pass removes anything with no unconditional association at all. Second pass removes anything that becomes independent once you condition on the single strongest driver found so far. Third pass conditions on the two strongest, and so on, until nothing more drops. What survives is a short list of likely parents. The design choice that matters here is that PC1 runs with a deliberately loose significance level. You want it to over-include, to keep the real parents even at the cost of dragging in some false ones, because the next stage cleans up false positives. In the experiments, PC1 still recovered more than 80% of true parents even at 100 variables. At that size more than 80% of the conditions it selected were false positives, but the list was still a tiny fraction of the 500 lagged variables FullCI would have conditioned on. You traded a saturated 149-control regression for a lean five-control one, and that is where the power comes back. ## PCMCI step two: MCI conditions on both ends Getting the conditioning set small is half the fix. The other half is the false-positive problem that autocorrelation creates, and this is the part plain PC-style algorithms get wrong. The momentary conditional independence test, MCI, tests the link from driver to target while conditioning on two parent sets at once: the parents of the target, and the parents of the driver. $$ X^{i}\_{t-\\tau} \\perp\\!\\!\\!\\perp X^{j}\_{t} \\;\\big|\\; \\mathcal{P}(X^{j}\_{t}) \\setminus \\{X^{i}\_{t-\\tau}\\},\\; \\mathcal{P}(X^{i}\_{t-\\tau}) $$ Read it as: driver at a lag is independent of the target now, given the parents of the target and the parents of the driver. Conditioning on the target's parents is what kills spurious and indirect links, the same job FullCI was doing, but now with five controls instead of 149\. Conditioning on the driver's own parents is the extra piece, and it is there to soak up autocorrelation. A strongly autocorrelated series carries information about its own past, and if you do not control for that, a mechanically persistent series will look like it is causing things it is not. Adding the driver's parents to the conditioning set corrects the false-positive rate back to the nominal level, even when both series are heavily autocorrelated. Work it against the climate case. The bare FullCI link Nino to British Columbia had effect size 0.1 that decayed to 0.09 and then vanished into 40% power as controls piled up. MCI, conditioning only on the few real parents, holds the partial correlation at about 0.10 with a p-value of 0.036 in all three variants, and detection power stays above 80% even in the high-dimensional case. Same link, same effect size, no collapse. For finance this is the whole ballgame: it means a causal link you find in a two-asset study does not evaporate when you expand the analysis to the full basket, which is precisely what happens with FullCI. Autocorrelation is not a corner case here either. Volatility clustering, momentum, and slow-moving macro series make financial data some of the most autocorrelated you will ever test, so the FDR control that MCI buys is not a nicety, it is the difference between a network you can act on and one that is mostly artifacts. ## Effect size is causal strength, not correlation There is a bonus in MCI that changes how you read the output. The MCI statistic is not just a pass-fail flag, it is an estimator of causal strength, and it behaves far better than correlation as a ranking tool. The paper defines the standardized causal effect of a link with coefficient c as follows. $$ \\text{causal strength} = \\frac{|c|}{\\sqrt{1 + c^{2}}} $$ Read it as: the true strength of a link scales with its coupling coefficient, squashed into a bounded range. Work two: a weak link with c equal to 0.15 has a causal strength of 0.15 divided by the square root of 1.0225, which is 0.148\. A strong link with c equal to 0.5 has strength 0.5 over the square root of 1.25, which is 0.447\. Now the point. When you fix every link in a synthetic network to the same c and measure how three methods report it, correlation is all over the map, spanning nearly zero to high values for links that are all equally causal. It is essentially unrelated to causal strength, which is why a correlation-ranked list of drivers is nearly worthless. FullCI partial correlation is tighter but systematically smaller than the truth. MCI sits right on the true causal-strength line. ![Effect size against coupling coefficient, correlation spread across a wide grey band unrelated to true strength, FullCI running below the truth, and MCI tracking the true causal-strength line almost exactly](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_429-effect_size_vs_correlation.png) For a quant this means the MCI value on each surviving edge is a rankable number. You can say the rate-to-currency link is genuinely stronger than the equity-to-currency link and mean it, rather than mistaking a big correlation for a big cause. The old article ["Regularization from First Principles"](https://aligrithm.com/regularization-from-first-principles/) made the case that Lasso is the standard tool for picking a few active drivers out of many. Runge's experiments put Lasso in the same test, and the sting is in the worst case: even at ten variables Lasso constantly overlooks a slice of the true links, while PCMCI kept detection power above 70% on 99% of links. Lasso optimizes prediction, not causal significance, and the gap shows up exactly where you care. ## Pick the independence test to match your data PCMCI does not hard-code how it measures dependence. Both PC1 and MCI plug into any conditional independence test, and the paper works three. Linear partial correlation, ParCorr, is fast and assumes linear additive noise, the right default when you believe your relationships are roughly linear. GPDC uses Gaussian process regression with a distance-correlation test on the residuals, which catches nonlinear links with additive noise at the cost of speed. CMI is a fully non-parametric conditional-mutual-information test built on a k-nearest-neighbor estimator, and it catches almost any dependence structure, including the messy nonlinear couplings that live in stressed markets. The old article ["Mutual Information as a Regime / Noise Filter"](https://aligrithm.com/mutual-information-as-a-regime-noise-filter/) already made the case for mutual information over correlation when relationships bend; CMI is that idea wired into a causal discovery engine. The tradeoff is the usual one. The more general the test, the more data it needs. GPDC and CMI have lower power than ParCorr on genuinely linear relationships at small samples. So the honest workflow is to run ParCorr first, and only reach for GPDC or CMI when you have reason to believe the linear test is missing a nonlinear driver and you have the sample size to support the switch. ## What PCMCI does not promise Keep the skepticism pointed at the method too. PCMCI rests on assumptions that finance violates more often than climate does. Causal Sufficiency assumes every common driver is in your dataset. If the true regime variable, a central-bank stance or a liquidity shock, is not one of your series, PCMCI will still hand you a confident network and some of its edges will be that missing driver in disguise. The word causal here is always relative to the variables you included, nothing more. Two more limits. PCMCI as described handles time-lagged links only, where the arrow of time settles direction; it does not resolve contemporaneous, same-bar causality, which is where a lot of high-frequency action lives. And observational noise degrades it: the paper found noise up to 25% of the dynamical noise standard deviation had only minor effects, but at noise levels comparable to the signal the false-positive control breaks because common drivers stop being detectable. Market data is noisy, so treat a PCMCI network as a hypothesis generator that narrows hundreds of candidate drivers to a testable few, not as a finished truth. The one asymmetry worth trusting: a link PCMCI fails to find is more believable as a genuine non-link than a link it finds is as a genuine cause, because absence of a detectable relationship rests on weaker assumptions than presence. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_429-visual-0.png) ## KEY POINTS - Pairwise Granger across many instruments reports mostly spurious links, because markets are full of common drivers, heavy autocorrelation, and indirect chains that a two-series test cannot see. - The obvious fix, conditioning on the full past of every series (FullCI), controls spurious links but collapses in power. Thirty instruments at five lags means 149 controls in one test; power fell from 80% at five variables to 40% at twenty and became unusable once controls exceeded the sample. - PCMCI stage one, PC1, selects a short list of likely parents by iterative conditioning, deliberately over-including to protect real drivers. It recovered over 80% of true parents even at 100 variables. - PCMCI stage two, MCI, tests each link conditioning on the parents of both the target and the driver. The target's parents kill spurious and indirect links with a handful of controls instead of 149; the driver's parents soak up autocorrelation and restore correct false-positive control. - MCI is an estimator of causal strength, so its value ranks links honestly. Correlation is nearly unrelated to true causal strength; FullCI understates it; MCI tracks it. Even at ten variables Lasso overlooked links that PCMCI caught above 70% power on 99% of links. - The dependence test is pluggable: ParCorr for linear and fast, GPDC and CMI for nonlinear at the cost of sample size. Run ParCorr first, escalate only with evidence and data. - The assumptions bite in finance: Causal Sufficiency needs every common driver in the dataset, the method resolves lagged links only, and it degrades under signal-scale noise. Treat the output as a hypothesis generator, and trust an absent link more than a present one. ## References - [Mathematical Techniques in Financial Markets - Don Mak (Amazon)](https://www.amazon.com/s?k=Mathematical+Techniques+in+Financial+Markets+Don+Mak&ref=aligrithm.com) - [Cycle Analytics for Traders - John Ehlers (Amazon)](https://www.amazon.com/s?k=Cycle+Analytics+for+Traders+John+Ehlers&ref=aligrithm.com) - [Detecting causal associations in large nonlinear time series datasets (Detecting causal associations in large nonlinear time series datasets)](https://scholar.google.com/scholar?q=Detecting+causal+associations+in+large+nonlinear+time+series+datasets&ref=aligrithm.com) --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ### 4.72 Graph Learning for FX: Interest-Rate-Parity Statarb Done Right URL: https://aligrithm.com/graph-learning-for-fx-interest-rate-parity-statarb-done-right/ Last updated: 2026-08-15T06:37:03.000Z Take one dollar. Buy euros with it, buy yen with those euros, then buy dollars back with the yen. In a clean market you end where you started, holding exactly one dollar, because the three exchange rates are locked together by an identity. Sometimes you end with slightly more than a dollar. That sliver is triangular arbitrage, and it exists in real spot data even though the textbook says it cannot. Yoonsik Hong and Diego Klabjan built a two-stage graph neural network across ten currencies to hunt those slivers, and the interesting part is not the machine learning. It is that they beat the standard arbitrage solver by 61.89% on information ratio while cutting volatility in half and drawdown by 45%, and they did it by fixing a bias the whole prior literature ignored. The old article ["From Intermarket Analysis to Network Momentum"](https://aligrithm.com/from-intermarket-analysis-to-network-momentum/) made the case that markets are a graph and a signal flows along the edges. The old article ["Granger Causality: Finding What's Driving Your Currency Right Now"](https://aligrithm.com/granger-causality-finding-whats-driving-your-currency-right-now/) made the case that you should find the live driver of a currency instead of guessing. This paper fuses both instincts into one FX arbitrage engine and then does something most quant papers skip: it takes the look-ahead bias seriously. ## The triangle that should close and does not Two identities govern currency conversion. The first is direct: the rate from currency i to j times the rate from j back to i should equal one, so a round trip through a single pair leaves you flat. The second is triangular: the rate from i to j should equal the rate from i to k times the rate from k to j. When that second identity breaks, a three-currency loop pays. $$ \\text{Loop payoff} = X\_{ik}\\, X\_{kj}\\, X\_{ji} - 1 $$ The loop payoff is what you keep after converting one unit of i into k, then k into j, then j back into i, minus the unit you started with. If the product of the three rates lands above one, the loop is profitable. Work it with numbers. Say the rate from USD to EUR is 0.92, from EUR to JPY is 162.0, and from JPY back to USD is 0.0068\. Multiply: 0.92 times 162.0 is 149.04, times 0.0068 is 1.0135\. You end with 1.0135 dollars, a 1.35% gain on the loop, because the three quotes drifted out of alignment for a moment. In theory this is zero. In practice, Aiba and coauthors documented it as a real interaction among FX rates, and Hong and Klabjan build their whole method on exploiting the loops that appear across three or more currencies, not just three. The classic way to scale this past a single triangle is a linear program. Soon and Ye wrote the template: observe all the rates, set up cash-flow-balance constraints so you hold nothing but your home currency at the end, and let the LP maximize the profit across every possible loop. That LP is the benchmark this paper is trying to beat, and its fatal assumption is coming up. ## Why interest rates belong in the graph FX rates are not free-floating numbers. Interest rate parity ties every currency pair to the interest rate gap between the two countries. Covered interest rate parity is the exact version. _This post is for paying subscribers only._ ### 4.73 HFT Supplies Liquidity Until It Doesn't: FX Flash-Crash Cascades URL: https://aligrithm.com/hft-supplies-liquidity-until-it-doesnt-fx-flash-crash-cascades/ Last updated: 2026-08-15T06:37:04.000Z At 05:55 Tokyo time on March 17, 2011, six days after the Fukushima earthquake, USD/JPY sat around 79.50 on a thin overnight tape. Twenty-five minutes later it was below 76.50\. Three hundred pips, gone, with no fresh fundamental news in that window. Retail stop-losses tripped, margin desks fired liquidations into a market where banks had already yanked their quotes or widened them so far their bids sat well under the last print. Then it snapped back to 78.23 within half an hour, then dropped again to 77.10 around 07:00 as a second wave of forced stop-outs, roughly two billion dollars of USD/JPY selling, hit six market makers who had obligations to quote. Nobody decided the yen was worth that. The microstructure decided. The comfortable academic consensus says high-frequency traders are good for you: they narrow spreads, speed price discovery, and dampen short-term volatility. Golub, Dupuis, and Olsen at Olsen Ltd show that consensus is true on average and false exactly when it matters. Their claim is blunt: FX price trajectories are path-dependent, a black swan can be manufactured by market structure alone with no link to fundamentals, and the liquidity HFT provides is real right up to the moment the machines all decide to leave at once. The old article ["Why Retail FX Execution Is Not the Same as Interbank FX"](https://aligrithm.com/why-retail-fx-execution-is-not-the-same-as-interbank-fx/) explained why your fill is not the interbank fill. This article is about what happens to that whole plumbing system under stress. ## The FX market is a mesh, not a ladder First fix the map, because most retail traders picture FX as a clean hierarchy with banks on top and them at the bottom. It is not hierarchical. It is a mesh of venues with different rules, and orders route sideways as often as up. At the center sit two inter-dealer platforms: EBS and Reuters, both limit order books. EBS is the venue for USD, EUR, GBP, CHF, and JPY crosses; Reuters owns the Commonwealth and Scandinavian currencies. Both carry a minimum ticket size of one million units, which is what makes them institutional-only. That minimum created a business opportunity, and ECNs like Currenex and Hotspot filled it with anonymous market making, a 0.1-pip tick, and much smaller or zero minimum tickets (Hotspot sets 50,000 units). Oanda, an FCM, quotes the same firm price for a ticket as small as one dollar or as large as ten million and hedges its net exposure with institutional makers. The CME runs the largest currency-futures market at roughly 100 billion dollars notional a day, 49 contracts, a 125,000-dollar ticket, and, unlike spot, a single centralized price and a clearinghouse guarantee. Retail aggregators together are about 10% of spot volume. The whole thing turns over about 1.4 trillion dollars of spot a day. Two mechanical details matter for what follows. EBS publishes time-sliced order-book snapshots every 250 milliseconds, or every 100 on its premium feed, and imposes a minimum quote lifetime of 250 milliseconds so an order cannot be cancelled the instant it is posted. The ECNs are faster and looser, and some let liquidity providers use "last look," a few-hundred-millisecond option to reject your fill if the price moved against them. Last look is the same asymmetry the old article "Why Retail FX Execution Is Not the Same as Interbank FX" flagged: the maker keeps the good fills and passes the bad ones back to you. Even the paper notes Hotspot's last-look feed runs wider than Currenex's non-last-look feed, so the "protection" makers get is paid for by everyone else in spread. ## Who is actually in the book Now count the participants, because the order book you see is mostly a mirage. Schmidt's EBS study splits traders into four tiers by trades per day: manual traders (MT) on a GUI, "Slow AI" under 500 trades a day, HFT between 500 and 3,000, and Ultra-HFT above 3,000\. Manual traders are 75% of EBS customers, and more than 90% of them submit under 100 orders a day. But on EUR/USD, the busiest pair, the share of orders submitted flips hard: MT 3.7%, Slow AI 5.7%, HFT 29%, and Ultra-HFT 61.6%. The machines dominate the order flow you observe. Here is the trap. Posting an order is not filling one. Fill ratios run the opposite direction: about 50% for manual traders, 26.6% for Slow AI, 8.1% for HFT, and 6.8% for Ultra-HFT. The tier that floods the book with 61.6% of orders cancels roughly 93% of them. ![Grouped bar chart comparing each EBS trader tier's share of EUR/USD orders posted against the fraction of those orders that actually fill, showing manual traders post few orders but fill half, while Ultra-HFT posts most orders but fills under 7%](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_422-trader_taxonomy.png) Multiply the two and you get the share of actually-filled trades per tier, Schmidt's Table 4.1: MT 1.9%, Slow AI 1.5%, HFT 2.3%, Ultra-HFT 4.2%. So the depth on your screen is dominated by quotes that will vanish before you can hit them. In calm markets this phantom liquidity is harmless, even helpful, because someone usually refreshes. In stress it is the whole problem: the 61.6% of the book that is cancellable evaporates in milliseconds, and the depth you were counting on was never contractually there. Masry's Oanda study, 110 million transactions across 46,000 accounts and 48 pairs from 2007 to 2009, finds the same shape on the retail side: assuming manual traders trade at most 50 times a day, their 27.7% share lines up with the 26.6% institutional figure. ## Path dependence: a crash with no cause The paper's central and least comfortable claim is that price is path-dependent. Second-by-second transaction volume is a trickle, so a minor market order can spike the price, the spike trips stop-losses and margin calls, the liquidations are more market orders, and the loop feeds itself. No fundamental input required. Cespa and Foucault formalize the same fragility: a small drop in one asset's liquidity propagates through a feedback loop into a large, market-wide liquidity crash, with multiple equilibria where the bad one looks like the May 6, 2010 Flash Crash. The FX record is a list of these. The August 2007 yen carry unwind produced one of the highest realized volatilities and the highest absolute serial correlation in five-second returns on record; algorithms as a group aggressively sold dollars and bought yen while humans on the other side bought and sold in roughly equal amounts. The machines moved together; the humans did not. On May 6, 2010, the DJIA lost about 900 points in minutes, a 1,010-point intraday swing, triggered by a mutual fund's 4.1-billion-dollar E-mini sell that HFT did not start but amplified into a "hot potato." In FX that day, EBS algorithmic activity actually rose to 53.5% of the total against 46.5% manual, versus a 45/55 norm, so the algos did not flee, they leaned in. Menkveld and Yueshen later showed the mutual fund was only 4% of net E-mini sells during the crash itself, which means the cascade, not the trigger, did the damage. Then March 2011, the cleanest FX example. ![Line chart of the March 17 2011 USD/JPY cascade, dropping from 79.50 to a 76.25 low in 25 minutes as banks pull quotes, rebounding to 78.23, then dropping again to 77.10 on a second forced stop-out wave](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_422-jpy_cascade.png) Retail margin stop-losses set off the first wave of dollar selling into a thin tape. Banks withdrew from making prices or widened spreads until their bids were far below the last trade, which is exactly the volatility-conditioned spread behavior the old article ["Spread Widening During Volatility Expansion"](https://aligrithm.com/spread-widening-during-volatility-expansion/) described, except here it went all the way to withdrawal. That drove the price lower, which tripped more stops, a positive feedback loop down to 76.25\. The recovery to 78.23 came only when fresh longs stepped in and banks resumed quoting. The second leg down to 77.10 was another automated stop-out cascade, about two billion dollars, when a Tokyo platform restarted and dumped compulsory stop orders on six obligated market makers. Both HFT and traditional makers had stepped away. The lesson the authors draw is sharp: even venues with designated, obligated market makers give you no guarantee on quote quality, because obligated makers widened their spreads so far the obligation was meaningless. ## The nuance the headline skips If the story stopped there it would be dishonest, and the paper does not stop there. During two central bank interventions, HFT was the good guy. At the BOJ intervention at 01:00 GMT on August 4, 2011, the USD/JPY jump did not break the two-sided market: manual traders quoted 100% of the time, HFT and Slow AI failed to provide two-sided liquidity for only two and eight seconds respectively, and HFT set the spread throughout. Only the Ultra-HFT tier withdrew for several minutes around 02:40\. At the SNB intervention on September 6, 2011, from 08:00 to 08:30 GMT, HFT was the quickest of all groups to restore two-sided quoting and the most active in setting the spread, while Ultra-HFT was the slowest to come back. So the real finding is conditional, not a slogan. HFT reliably supplies liquidity, tightens spreads, and restores two-sided markets fast during large but orderly moves like a telegraphed central bank action. It withdraws and amplifies during disorderly, surprise cascades where its own positions are suddenly underwater. The distinction that keeps showing up is the Ultra-HFT tier: fastest to post in calm, first to vanish in stress. "HFT is good for markets" and "HFT causes flash crashes" are both true, on different days, and anyone selling you only one half is selling you something. ## Correlation is the hidden fragility The mechanism that turns many independent makers into one fragile maker is strategy correlation. Chaboud and coauthors, using EBS data from 2003 to 2007 on EUR/USD, USD/JPY, and EUR/JPY where human and computer trades are separately tagged, found strong evidence that computers do not trade with each other as much as random matching would predict. Their strategies are more correlated and less diverse than humans'. On average this is benign and even useful: algorithmic quoting reduced triangular arbitrage opportunities and cut the absolute autocorrelation of five-second returns, because machines reprice on new information faster than people. But the same commonality means that when the shared signal says "get out," they all get out together, collectively acting as one enormous trader. Jarrow and Protter warned about precisely this: common signals plus speed produce momentum and less informative prices. Correlated liquidity is liquidity that disappears all at once, which is no liquidity at all in the one moment you need depth. ## A fix at the queue, not the tax Regulators mostly reached for blunt instruments: circuit breakers, a 500-millisecond minimum resting time under MiFID, order-count caps, and France's 0.2% Tobin tax. Becchetti and coauthors found the French tax did cut intraday volatility, but it also cut transaction volume, which is the brute-force tradeoff: you buy calm by killing activity. The paper's own proposal is subtler. Instead of punishing makers, reward the ones who quote both sides. It changes the order-book queue itself, from the standard price-time priority to what the authors call spread/price-time priority. Under normal price-time priority, the best price wins and ties break by who was first. The proposal blends price rank with spread rank using one knob, alpha between 0 and 1. $$ \\text{rank}(\\alpha) = \\alpha \\times \\text{price} + (1 - \\alpha) \\times \\text{spread} $$ The rank of a limit order is a weighted average of its price rank (weight alpha) and its spread rank (weight one minus alpha), and the lowest rank gets executed first. Set alpha to 1 and you are back to pure price-time priority, spread ignored. Set alpha to 0 and it is pure spread-time priority, price ignored. Lower alpha means more weight on the spread, which hands better queue position to traders who quote a tight two-sided market. Worked example: suppose a one-sided sell order is 0.4 pips off the best ask, and the widest resting two-sided quote has a 1.0-pip spread, so the one-sided order inherits spread rank 1.0\. A competing two-sided maker sits 0.6 pips off the best price but quotes a tight 0.3-pip spread. At alpha of 0.5, the one-sided order ranks 0.5 times 0.4 plus 0.5 times 1.0, which is 0.70, while the two-sided maker ranks 0.5 times 0.6 plus 0.5 times 0.3, which is 0.45\. The two-sided maker jumps the queue despite the worse price, because it revealed information about both sides. That requires defining the two ranks. Price rank is your distance from the current best price on your side, and spread rank depends on whether you quote one side or both. $$ \\text{price} = \\text{ask} - \\text{ask}^{\\text{best}} \\quad (\\text{sell}), \\qquad \\text{spread} = \\text{ask} - \\text{bid} \\quad (\\text{two-sided}) $$ A sell order's price rank is its ask minus the best resting ask, so a new best ask ranks 0 and worse asks rank higher (a buy order is symmetric, best bid minus your bid). A two-sided order's spread rank is simply its own ask minus bid. A one-sided order does not get to skip the spread term: it is assigned the worst spread among resting two-sided orders, so it is always "at par" with the least competitive real maker. Worked example: best ask is 1.2000, you post a sell at 1.2002, so your price rank is 0.0002, two pips. If you also post a bid at 1.1999, your spread rank is 1.2002 minus 1.1999, which is 0.0003, three pips. Quote only the sell and you inherit whatever the widest two-sided spread in the book is, which is a strictly worse deal. The design makes one-sided quoting structurally expensive in queue position, which is the whole point: it pays makers to stay two-sided. ## Does the queue fix actually work The authors test it in Bartolozzi's agent-based limit-order-book model, which reproduces the empirical fingerprints of real microstructure: negative return autocorrelation, clustered volatility and volume, and non-linear price response to size. They add two knobs: alpha, and p, the probability a trader posts a two-sided rather than one-sided quote, capped at a 10-pip spread. They sweep both on a 0.1-to-0.9 grid, 81 combinations, 1,000 iterations each. Volatility is measured as an average across eight timescales so it captures both the scalper's world and the swing trader's. $$ \\sigma = \\frac{\\sigma\_{1} + \\sigma\_{2} + \\sigma\_{5} + \\sigma\_{10} + \\sigma\_{15} + \\sigma\_{20} + \\sigma\_{25} + \\sigma\_{50}}{8} $$ Overall volatility is the mean of the return standard deviations measured at 1, 2, 5, 10, 15, 20, 25, and 50 time-step horizons. Averaging across scales stops the metric from flattering a mechanism that only calms one horizon while roughening another. Worked example: if the per-horizon standard deviations came out at 0.9, 0.8, 0.7, 0.6, 0.55, 0.5, 0.48, and 0.47 model units, the reported sigma is their sum, 4.90, divided by 8, which is 0.61\. A fix that cut only the 1-step number to 0.3 would barely move this average, which is the design intent. The result is the useful part. ![Two-panel line chart: left panel shows average volatility rising with alpha for every value of p, so lower alpha means lower volatility; right panel shows total volume flat across alpha but scaling with p, meaning the two-sided quote rate sets volume, not the alpha knob](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_422-spread_priority.png) Lowering alpha, putting more weight on spread, reduces volatility regardless of p. Total volume, meanwhile, tracks p and is essentially flat in alpha. Read together, that is the claim: you can dial down volatility with the queue rule without sacrificing volume, which is exactly the tradeoff the Tobin tax loses. HFT keeps its market-making revenue, long-term traders get a calmer tape, and volume survives. ## The skeptic's ledger Hold the enthusiasm at the right level. Take the taxonomy and the cascade mechanics as real: they rest on transaction-level EBS and Oanda data and documented events. Treat the fix as an untested hypothesis, because that is what it is. It lives inside a single agent-based model, Bartolozzi's, 81 runs of 1,000 synthetic iterations, with no live venue ever running spread/price-time priority. Agent-based models are calibrated to reproduce stylized facts, which means they can also reproduce a convenient result; a real order book has adverse selection, latency arbitrage, and gaming incentives that a toy will not surface. The authors themselves note the rule can produce crossed or locked quotes for any alpha above zero, which is a real implementation headache. The event evidence is anecdotal by the paper's own admission, four episodes across different stages of HFT's evolution, not a controlled panel. And the liquidity-provision measurements are from EBS, the institutional venue with a one-million-unit minimum. Retail did not get that liquidity in March 2011; retail stop-losses were the fuel. So the honest read for someone trading a retail account is grim: you carry the wider spread and the last-look rejection in calm markets, per the old article ["Why Retail FX Execution Is Not the Same as Interbank FX,"](https://aligrithm.com/why-retail-fx-execution-is-not-the-same-as-interbank-fx/) and in a cascade you are the first to be liquidated into a book whose depth was 93% cancellable to begin with. The machines supply liquidity until they don't, and when they don't, you are not on the EBS side of that trade. ## Where this connects For Pillar 4, this is the market-structure floor under every FX strategy: your edge is computed against quoted prices, and quoted prices are path-dependent artifacts of a correlated, mostly-cancellable book that thins out precisely in the tails your risk model cares about. For Pillar 5 microstructure, the queue-priority idea is the constructive counterpart to the destructive cascade: the same limit-order-book mechanics that manufacture a flash crash can, in principle, be re-engineered to dampen one. Both pillars share the same uncomfortable premise, that in FX the plumbing is the strategy, and pretending price is a clean signal on top of infinite liquidity is how retail accounts get stopped out at 76.25. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_422-visual-0.png) ## KEY POINTS - The FX market is a non-hierarchical mesh of venues (EBS and Reuters interbank books, Currenex and Hotspot ECNs, Oanda FCM, CME futures) with wildly different tick sizes, minimum tickets, and rules like last look and a 250-millisecond minimum quote lifetime. - The order book is mostly phantom liquidity. On EUR/USD, Ultra-HFT posts 61.6% of orders but fills only 6.8% of them; manual traders post 3.7% but fill about 50%. Depth you see can be cancelled before you hit it. - FX prices are path-dependent. A thin tape plus stop-losses and margin calls creates self-feeding cascades with no fundamental cause: August 2007, May 2010, and the March 2011 USD/JPY drop of 300 pips in 25 minutes. - HFT's role is conditional, not one-sided. It supplied and restored liquidity fastest during the orderly BOJ and SNB interventions, but withdrew and amplified during disorderly surprise cascades. The Ultra-HFT tier is consistently first to leave. - Correlated strategies are the hidden fragility. Machines trade less diversely than humans, so when a shared signal says exit, they all exit at once and act as one giant trader, which is when depth vanishes. - The proposed fix is spread/price-time priority: rank equals alpha times price rank plus one minus alpha times spread rank. Lower alpha rewards two-sided quoting, and in an agent-based model it cuts volatility without cutting volume, unlike a Tobin tax. - The taxonomy and cascade mechanics are solidly evidenced; the queue fix is a single agent-based model with no live-venue test, so treat it as a hypothesis. Retail sits on the wrong side of all of it. ## References - [Trading Systems and Methods - Perry Kaufman (Amazon)](https://www.amazon.com/s?k=Trading+Systems+and+Methods+Perry+Kaufman&ref=aligrithm.com) - [Cybernetic Trading Strategies - Murray Ruggiero (Amazon)](https://www.amazon.com/s?k=Cybernetic+Trading+Strategies+Murray+Ruggiero&ref=aligrithm.com) - [The Art of Currency Trading - Brent Donnelly (Amazon)](https://www.amazon.com/s?k=The+Art+of+Currency+Trading+Brent+Donnelly&ref=aligrithm.com) - [High-Frequency Trading in FX Markets (HFT in FX markets - Glob)](https://scholar.google.com/scholar?q=High-Frequency+Trading+in+FX+Markets&ref=aligrithm.com) --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ### 5.46 Bid-Ask Spread From OHLC: The GMM Estimator That Beats Roll/CS URL: https://aligrithm.com/bid-ask-spread-from-ohlc-the-gmm-estimator-that-beats-roll-cs/ Last updated: 2026-08-15T06:37:05.000Z You backtest a strategy, subtract "a spread" for costs, and move on. Where did that number come from? If you typed in a guess, or pulled a single quoted spread from a vendor, you are almost certainly wrong, and wrong in the one direction that flatters your worst assets. The effective spread you actually pay is unobservable without trade-and-quote data, and quote data for anything outside recent U.S. large-caps is expensive, patchy, or nonexistent. So most people estimate the spread from the only thing they have: open, high, low, close candles. The problem is that the popular estimators for doing that were built on an assumption that is false in exactly the markets where costs bite hardest. The old article ["Why Transaction Costs Should Be Added Before You Fall in Love"](https://aligrithm.com/why-transaction-costs-should-be-added-before-you-fall-in-love/) argued you must apply realistic costs from the prototype stage, before you get emotionally attached to a pre-cost Sharpe. This article is the missing piece of that: how to get the spread number honestly from OHLC alone. David Ardia, Emanuele Guidotti, and Tim Kroencke derive an estimator they call EDGE, the Efficient Discrete Generalized Estimator, and show it dominates the seminal Roll estimator and the more recent Corwin-Schultz and Abdi-Ranaldo estimators across simulations and 1.6 million stock-months of real data. The single number that should make you nervous: when trading is thin, Corwin-Schultz reports a spread of 0.04% where the true spread is 1.00%. That is not a small error. That is your cost model understating reality by a factor of 25 on precisely the illiquid names where the spread is the whole game. ## What "effective spread" even means Start with the thing being estimated, because half the confusion in cost modeling is people measuring different quantities and calling them the same word. The relative effective spread for a single trade is the distance between the price you got and the true, unobserved fundamental value. $$ S = \\frac{2D(P - \\bar{P})}{\\bar{P}} $$ S is the relative effective spread, P is the transaction price you actually paid, P-bar is the unobserved fundamental price, and D is a direction indicator that is plus one for a buyer-initiated trade and minus one for a seller-initiated trade. The factor of two turns a one-sided distance from the midpoint into a full round-trip spread. Work a number: you buy at 100.10, D is plus one, and the fundamental price is 100.00\. Then S is 2 times 1 times (100.10 minus 100.00) divided by 100.00, which is 0.20 divided by 100, or 0.20%. If you had instead sold at 99.90 with D minus one, you would get the same 0.20%. The catch is right there in the formula: P-bar is unobservable. Every estimator in this fight is just a different way of guessing it. High-frequency methods proxy P-bar with the quoted midpoint from actual quote data. Low-frequency methods, the subject here, assume a model of how prices move and back the spread out of transaction prices alone. ## The old guard, and why they break The low-frequency lineage starts with Roll in 1984\. His insight: if the fundamental price is a random walk and trades bounce between bid and ask, then consecutive observed price changes are negatively correlated, and the size of that negative serial covariance tells you the spread. $$ S^2 = -4\\,\\mathrm{Cov}\[\\Delta c\_t, \\Delta c\_{t-1}\] $$ The squared spread equals minus four times the serial covariance of consecutive close-to-close log-return changes, where delta-c-t is the change in the closing log-price from one period to the next. Plain version: the bid-ask bounce makes an up-tick tend to follow a down-tick, that alternation shows up as negative covariance, and Roll inverts it to recover the spread. Worked example: if the covariance of consecutive daily close changes is minus 0.000025, then S-squared is minus four times minus 0.000025, which is 0.0001, and S is the square root, 0.01, or 1.00%. The fatal weakness is variance. On a yearly sample of daily closes that covariance is so noisy it comes out positive about half the time, which makes S-squared negative, which is nonsense, so you truncate to zero and throw away half your estimates. Corwin-Schultz improved on Roll using the high-low range, and Abdi-Ranaldo improved again using close plus the high-low midpoint, each with lower variance. But all three share one buried assumption: that prices are observed continuously, meaning there is always at least one trade between any two observations. That assumption is the crack the whole edifice falls through. ## The discreteness fix nobody else made Real markets are discrete. Some days a thin stock trades a handful of times; some intraday bars have zero trades and the OHLC just gets forward-filled from the previous close. When that happens, the observed price stops bouncing, the negative serial covariance shrinks, and every continuous-time estimator reads that as a smaller spread. It is not smaller. It is undersampled. The authors' contribution is an analytical correction: the probability that the open or close price coincides with the high or low. Under continuous observation that probability is zero. In reality it ran between 25% for large stocks and 75% for small stocks over 1926 to 2021, and it climbs as you move to intraday sampling where each bar holds fewer trades. The corrected spread-at-open estimator looks like this. $$ S\_o^2 = \\frac{-8\\,\\mathbb{E}\[(\\eta\_t - o\_t)(o\_t - c\_{t-1})\]}{\\mathbb{P}\[o\_t \\neq h\_t\] + \\mathbb{P}\[o\_t \\neq l\_t\]} $$ The squared spread at the open equals minus eight times the expected product of two returns, the mid-to-open return and the open-to-previous-close return, divided by a correction term: the probability that the open differs from the high plus the probability that the open differs from the low. Here o, h, l, c are open, high, low, close log-prices and eta is the high-low midpoint. The numerator is the same covariance idea as Roll and Abdi-Ranaldo. The denominator is the fix. When trading is frequent, open rarely equals high or low, both probabilities are near one, the denominator is near two, and you recover the classic estimator. When trading is thin, open often equals the high or the low, the probabilities shrink, the denominator shrinks, and dividing by a smaller number scales the estimate back up to where it belongs. Worked example: suppose the raw covariance term gives a numerator implying an uncorrected S-squared of 0.36 (a 0.60% spread), but the stock is thin so the probability the open differs from the high is 0.6 and from the low is 0.6, summing to 1.2 instead of 2.0\. The corrected estimate is scaled by 2.0 over 1.2, lifting S-squared to 0.60 and the spread to about 0.77%. The continuous-time estimator would have reported the 0.60% and quietly understated your cost. ## Combine everything, optimally One estimator uses open, high, low. You can build others from different price combinations: close-high-low, open-high-low-close, close-high-low-open. The authors derive four such building blocks and prove something useful about them. Two of them (the open-high-low-close and close-high-low-open versions) have the lowest variance when the spread is small relative to volatility. The other two (open-high-low and close-high-low) win when the spread is large relative to volatility. No single one is best everywhere. So rather than pick, they combine all four using the generalized method of moments from Hansen 1982: write each estimator as a moment condition and blend them to minimize the variance of the result. The efficient combination weights each piece inversely to its variance. $$ S\_{EDGE}^2 = w\_1\\,\\mu\_1 + w\_2\\,\\mu\_2, \\qquad w\_i \\propto \\frac{1}{\\mathrm{Var}\[x\_i\]} $$ The EDGE estimate is a weighted average of the moment estimates mu-1 and mu-2, where each weight is proportional to the inverse of that estimator's variance, so the more precise building block gets more say. Plain version: it is the same logic as combining two noisy measurements of the same thing by trusting the tighter one more. Worked example: say estimator one has variance 4 and estimator two has variance 1\. The raw inverse-variance weights are 0.25 and 1.0, which normalize to 0.20 and 0.80\. If estimator one reads 0.90% and estimator two reads 0.95%, EDGE reports 0.20 times 0.90 plus 0.80 times 0.95, which is 0.94%, leaning hard on the precise estimator. Because the low-variance building blocks are different for small and large spreads, EDGE automatically shifts its weight toward whichever pair is trustworthy in the current regime, which is how it achieves minimum variance across both. That regime-adaptive weighting is the "efficient" in Efficient Discrete Generalized Estimator, and it is what lets a single formula beat Roll and CS everywhere instead of just somewhere. ## Does it work: the simulation The authors run the standard Corwin-Schultz simulation setup: 10,000 months, 21 trading days each, 390 minutes a day, a fundamental price on a geometric random walk with 3% daily volatility, and a bid-ask bounce of a known size. Then they vary how often a trade is actually observed, from every minute down to roughly once a day, and check whether each estimator recovers the true 1.00% spread. ![Line chart of estimated bid-ask spread versus average trades per day on a log axis, showing EDGE flat at the true 1.00% while Corwin-Schultz collapses toward 0.04% and Abdi-Ranaldo sags as trading thins](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_423-bias_vs_frequency.png) At 390 trades a day, everyone is fine: all estimators land near 1.00%. As trading thins, the story splits hard. Corwin-Schultz reads 0.75% at 100 trades a day and craters toward zero below ten trades a day, bottoming near 0.04% in the sparse simulations. Abdi-Ranaldo is less sensitive but still sags below the truth. EDGE stays flat at 1.00% down to roughly a single trade per day. The variance results are just as decisive: across simulated spreads from 0.50% to 8.00%, EDGE has the lowest standard deviation of any estimator, uniformly, small spreads and large. It wins on bias when others are biased, and it wins on precision when everyone is unbiased. ## Does it work: the real data Simulations can be rigged to flatter the home team, so the empirical test matters more. The authors take every U.S. common stock in CRSP from 1993 to 2021, about 1.64 million stock-months, estimate the spread from daily OHLC with each method, and compare against a ground-truth effective spread built by matching high-frequency trades to quotes in the TAQ database, the Holden-Jacobsen benchmark. ![Bar chart of Pearson correlation with the high-frequency effective-spread benchmark, EDGE at 78.9% ahead of Abdi-Ranaldo 68.1%, Roll 55.2%, and Corwin-Schultz 45.5%](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_423-empirical_correlation.png) EDGE hits 78.9% Pearson correlation with the benchmark. Abdi-Ranaldo manages 68.1%, Roll 55.2%, and Corwin-Schultz only 45.5%. EDGE also posts the lowest mean absolute percentage error, 16.2% against CS's 35.9%, and the smallest fraction of non-positive estimates. It wins in every subsample the authors slice: every market venue, every sub-period, every size decile, every spread-size bucket, and every trading-frequency group. One more finding worth carrying into your cost model: the end-of-day quoted spread runs about double the effective spread, because dealers routinely trade inside their own quotes. If you have been costing trades at the quoted spread, you have been roughly doubling your assumed transaction cost, which is conservative but wasteful for sizing. ## The frequency upgrade that reconciles two literatures Here is the part that should interest anyone doing microstructure work. Because EDGE handles discreteness correctly, you can feed it intraday candles instead of daily ones and it just gets better, where the old estimators break. On the hard sample from October 2003 to December 2021, where spreads are tiny relative to volatility and daily estimates are noisy, switching EDGE from daily to minute OHLC does this. ![Two-panel bar chart: left shows EDGE correlation with the benchmark rising from 56.17% on daily candles to 88.79% on minute candles; right shows the fraction of non-positive estimates falling from 34.15% to 0.02%](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_423-minute_upgrade.png) Correlation with the benchmark jumps from 56.17% to 88.79%. The fraction of non-positive estimates, the garbage zeros you have to truncate, falls from 34.15% to 0.02%, so the upward bias that truncation induces essentially disappears. And it is more effective to raise the sampling frequency than to pile on more years of daily data. That is the reconciliation: the same estimator spans the low-frequency and high-frequency literatures depending on what candles you feed it. Outside equities it holds too. For cryptocurrencies, the old estimators produce a tenfold gap between daily and intraday estimates, a clear sign of the discreteness bias, while EDGE's daily estimates overlap its hourly and minute estimates. For a 24/7 asset with no official quote feed, an estimator that gives the same answer at any frequency is close to the only honest option. ## How to use it, and where it stops Practically, this is not a research toy you have to reimplement. The authors ship it as the open-source `bidask` package in R, Python, C++, and more. You hand it a frame of open, high, low, close columns and it returns the effective spread for the window. Then it drops straight into the net-edge accounting from the old article ["Why Transaction Costs Should Be Added Before You Fall in Love"](https://aligrithm.com/why-transaction-costs-should-be-added-before-you-fall-in-love/): net edge is gross edge minus cost, and EDGE gives you a defensible, asset-specific, frequency-appropriate estimate of the largest slice of that cost without paying for quote data. Now the honest limits, because a spread estimator is not a cost model. First, EDGE estimates the average effective spread over the estimation window, not the spread on your specific fill at your specific size. It is a population parameter, not a quote. Second, it rests on the standard assumptions that fundamental returns are serially uncorrelated and that bid-ask bounces are zero-mean and uncorrelated with those returns; strong intraday autocorrelation, momentum, or jumps violate those and will bias it, so it is a spread estimator, not a free lunch. Third, negative estimates still get truncated to zero, so in very small samples a residual upward bias survives even for EDGE, which is why more data or higher frequency helps. Fourth and most important, the effective spread is only one cost term. It does not capture market impact, the extra price you move by trading size, which is a separate and often larger cost for anyone trading meaningful volume. That is the domain of the old article ["Market Impact and the Square-Root Law: Walking the Book to Price Your Slippage."](https://aligrithm.com/market-impact-and-the-square-root-law-walking-the-book-to-price-your-slippage/) Your real per-trade cost is commission plus effective spread plus impact. EDGE nails the middle term cheaply and honestly; you still owe the other two. ## Where this connects For Pillar 5, this is the cheapest reliable microstructure measurement you can make: liquidity, quantified from the candles you already have, at whatever frequency you sample, across asset classes that never had quote data. For the data and backtesting work, it is a direct upgrade to your cost assumptions, and it kills a real source of what the literature calls non-standard errors, the silent variation in results caused by everyone estimating costs differently. The uncomfortable lesson underneath all of it: the "spread" in your backtest was probably a continuous-time estimate that understated the cost of your thinnest, most seductive-looking assets, and understating cost on illiquid names is exactly how a strategy looks profitable on paper and bleeds in production. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_423-visual-0.png) ## KEY POINTS - The effective spread is the distance of your fill from the unobserved fundamental price, S equals 2D times (P minus P-bar) over P-bar. Every low-frequency estimator is just a different guess at the fundamental price from OHLC data. - Roll, Corwin-Schultz, and Abdi-Ranaldo all assume prices are observed continuously. Real markets are discrete, and when trading is thin those estimators are downward biased, understating the spread exactly when it is largest. - EDGE's fix is an analytical correction term: the probability that the open or close equals the high or low. That probability is zero under continuous observation but ran 25% to 75% for real stocks, and it rescales the estimate back to unbiased. - EDGE combines four OHLC building blocks via GMM (Hansen 1982), weighting each inversely to its variance, so it achieves minimum variance across both small and large spreads instead of winning only in one regime. - In simulation, Corwin-Schultz reads 0.04% on a true 1.00% spread when trading is thin; EDGE stays unbiased down to about one trade per day and has the lowest variance uniformly. - On 1.64 million CRSP-TAQ stock-months, EDGE hits 78.9% correlation with the trade-and-quote benchmark versus 68.1% (AR), 55.2% (Roll), and 45.5% (CS), and wins in every venue, period, size, and frequency bucket. - Feeding EDGE minute instead of daily candles lifts correlation from 56.17% to 88.79% and cuts non-positive estimates from 34.15% to 0.02%, reconciling low- and high-frequency spread measurement in one estimator, and it holds up on crypto where the others show a tenfold daily-vs-intraday gap. - It measures the average effective spread, not your fill and not market impact. Use it for the spread term of your cost model, then add commission and impact separately. ## References - [The Art of Currency Trading - Brent Donnelly (Amazon)](https://www.amazon.com/s?k=The+Art+of+Currency+Trading+Brent+Donnelly&ref=aligrithm.com) - [Efficient estimation of bid–ask spreads from open, high, low, and close prices (Efficient estimation of bid–ask spreads from ohlc)](https://scholar.google.com/scholar?q=Efficient+estimation+of+bid%E2%80%93ask+spreads+from+open%2C+high%2C+low%2C+and+close+prices&ref=aligrithm.com) --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ### 7.5 The Mathematics of Machine Learning, for Traders URL: https://aligrithm.com/the-mathematics-of-machine-learning-for-traders/ Last updated: 2026-08-15T06:37:05.000Z You already ran the models. The old article ["From One Tree to Forests to Boosting"](https://aligrithm.com/from-one-tree-to-forests-to-boosting/) walked you from a single decision tree to XGBoost, ["How a Decision Tree Engineers a New Alpha"](https://aligrithm.com/how-a-decision-tree-engineers-a-new-alpha/) showed a tree carving conditional edges out of order-book features, and ["Ridge Above 1h, XGBoost Below 5min"](https://aligrithm.com/ridge-above-1h-xgboost-below-5min/) handed you a timeframe rule for which model to point at which horizon. None of those articles answered the one question that decides whether any of it survives contact with next month's data: why should a function fit on the past predict the future at all? Nothing in the arithmetic of a backtest forces that to be true. The math of machine learning is the part that tells you when it is, and by how much, and what you pay for reaching for a bigger model. This is the theory sitting under the whole ML arc. It is not decoration. Every skeptical instinct the robustness articles kept repeating, that complexity is expensive, that in-sample fit is a liar, that more parameters means more variance, turns out to be a theorem with a rate attached. Once you see the rate you stop arguing about it. ## The best you can possibly do, and why you never reach it Start with the target. You have an input X (your features) and an output Y (next return, or its sign). You pick a function h to predict Y from X, and you grade it with a loss. For a sign classifier the natural loss is misclassification: zero when you call the direction right, one when you call it wrong. The risk is the expected loss over the true, unknown distribution of the market. The function that minimizes that risk is not mysterious. In the sign-prediction case it is the Bayes classifier, and it depends on one quantity, eta of x, the probability that Y equals plus one given the features equal x. $$ \\eta(x) := \\mathbb{P}(Y = 1 \\mid X = x), \\qquad h\_0(x) = \\begin{cases} +1 & \\text{if } \\eta(x) > 1/2 \\\\ -1 & \\text{otherwise.} \\end{cases} $$ Read it plainly. If the features say the odds of an up-move are better than even, bet up. Otherwise bet down. That is the best classifier that exists, and its error rate is the Bayes risk, the floor no model beats. Work a number. Say for one feature bucket the true up-probability eta is 0.58\. The Bayes rule bets up and is wrong 42% of the time. That 42% is irreducible, market noise that no feature removes. A trader who expects a direction model to hit 90% on liquid futures is asking to beat the Bayes risk, which is impossible. The realistic question is never "can I hit the floor," it is "how far above the floor does my model sit, and is that gap worth trading." You never reach the floor for a blunt reason: you do not know eta of x. You have a finite sample, and you minimize the loss you can compute, the training error, over some class of candidate functions. That substitution, true risk replaced by average training loss, is empirical risk minimization, and it is what every fit you have ever run actually does. ## Every parameter costs you sigma-squared over n Here is the first theorem with real teeth for a trader. Fit ordinary least squares with d basis functions (d features, or d columns of a polynomial expansion) on n data points, where the noise variance is sigma-squared. Decompose the expected prediction error and the variance you added by estimating d coefficients comes out exactly: $$ \\mathbb{E}R(\\hat h\_D) \\;=\\; \\underbrace{\\text{squared bias}}\_{\\text{falls as } d \\uparrow} \\;+\\; \\underbrace{\\frac{\\sigma^2 d}{n}}\_{\\text{variance}} \\;+\\; \\underbrace{\\mathbb{E}\\,\\mathrm{Var}(Y\\mid X)}\_{\\text{irreducible}}. $$ The middle term is the price of complexity, and it is linear in d. Every parameter you add pours another sigma-squared over n of variance into your out-of-sample error, whether or not that parameter helps. Bias falls when you add useful features; variance rises no matter what. Put numbers on it. Take sigma-squared equal to 1 and n equal to 200 bars. A five-parameter model carries variance 5 over 200, which is 0.025\. Push to forty parameters and variance becomes 40 over 200, which is 0.20\. You bought 35 extra coefficients and paid 0.175 in guaranteed out-of-sample error, and they only earn their keep if they cut squared bias by more than that. On 200 bars they almost never do. ![Bias falls and variance rises as you add parameters; the sum is U-shaped with a minimum well short of the biggest model](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_430-bias_variance.png) The curve is U-shaped and the minimum sits well short of the richest model your software will let you fit. This is the exact mechanism behind the old article ["The Limits of Linear Models"](https://aligrithm.com/the-limits-of-linear-models/) and the reason ["Regularization from First Principles"](https://aligrithm.com/regularization-from-first-principles/) exists: shrinkage deliberately biases the coefficients toward zero to buy back some of that sigma-squared d over n. The trade is more bias for less variance, and on short, noisy financial samples it usually wins. The math here is why. ## Cross-validation estimates the wrong thing, on purpose You cannot see the true risk, so you estimate it. v-fold cross-validation splits the data into v blocks, trains on v minus one of them, scores on the held-out block, and rotates. $$ \\mathrm{CV}(j) := \\frac{1}{n} \\sum\_{k=1}^{v} \\sum\_{i \\in A\_k} \\ell\\!\\left( \\hat h\_{-k}^{(j)}(X\_i),\\, Y\_i \\right). $$ The honest caveat the notes make explicit: this is a biased estimate. Each fold trains on only a fraction of the data, so cross-validation reports the error of a model fit on n times (v minus one) over v points, not on all n. Run 5-fold on 500 bars and every model is scored while trained on 400 bars, then rotated across the five held-out blocks of 100\. The number you read off is the error of a 400-bar model, and it is pessimistic for the 500-bar model you will actually deploy. Leave-one-out (v equal to n) has the least bias but the folds overlap almost completely, so the scores are highly correlated and the estimate is noisy. Typical choice is v equal to 5 or 10, a deliberate compromise between bias and variance of the estimate itself. For a trader this maps onto a familiar pitfall. Cross-validation on shuffled financial data leaks the future into the training folds through autocorrelation, so the CV number flatters the model. The purged, embargoed splits the robustness articles push are the same v-fold idea with the leakage plugged. The estimator is only as good as the independence of its folds, and market data violates independence by construction. ## Trees are unstable, and averaging has a floor A single regression tree splits the feature space greedily to cut residual sum of squares, then predicts the mean response in each leaf. The old article ["How a Decision Tree Engineers a New Alpha"](https://aligrithm.com/how-a-decision-tree-engineers-a-new-alpha/) covered that machinery. The theory adds the warning: the greedy build is unstable, so a small change in the training data flips an early split and hands you a different tree. High variance, in the exact sense of the decomposition above. Random forests attack that variance by averaging B trees grown on bootstrap resamples, each split restricted to a random subset of features. The reason the feature sampling matters falls straight out of the variance of an average of correlated things: $$ \\mathrm{Var}\\big(f\_{\\text{rf}}(x)\\big) = \\frac{1-\\rho}{B}\\,\\mathrm{Var}\\big(\\hat T^{(1)}(x)\\big) \\;+\\; \\rho\\,\\mathrm{Var}\\big(\\hat T^{(1)}(x)\\big). $$ Rho is the correlation between two trees' predictions. The first term shrinks to nothing as you grow more trees. The second term does not depend on B at all. Grow ten thousand trees and the variance still cannot fall below rho times a single tree's variance. Numbers. Let a single tree's variance be 1 and let the trees correlate at rho equal to 0.3\. With B equal to 100 trees the forest variance is 0.7 over 100 plus 0.3, which is 0.307\. With B infinite it is still 0.30\. Adding trees past a point is free of benefit; the floor is set by rho. That is why the algorithm samples features at each split: it forces the trees apart, dropping rho toward 0.05, which drops the floor to 0.05\. The whole point of a random forest is not the bagging, it is the decorrelation. ![Growing more trees kills the 1/B term but the rho floor stays fixed; lower tree correlation means a lower floor](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_430-forest_variance.png) For a trader this is the quantitative version of "your ensemble members are too similar." Ten models built on the same features with the same target are one model wearing ten hats, and no amount of averaging escapes their shared variance. ## The price of complexity, as a bound you can compute Now the centerpiece. How far can training error drift from true error across a whole class of candidate models? For a finite class of size the-cardinality-of-H, Hoeffding's inequality plus a union bound gives, with probability at least 1 minus delta: $$ R(\\hat h) - R(h^\*) \\;\\leq\\; \\sqrt{\\frac{2\\big(\\log|\\mathcal{H}| + \\log(1/\\delta)\\big)}{n}}. $$ The left side is excess risk, how much worse your fitted model is than the best model in the class. The right side is the tax you pay for searching. It grows with the log of how many models you tried and shrinks like one over the square root of the sample. Worked example. You grid-search 1000 candidate configurations, n equal to 500 bars, delta equal to 0.05\. The bound is the square root of 2 times (log 1000 plus log 20) over 500, which is the square root of 2 times (6.91 plus 3.00) over 500, which is the square root of 0.0396, about 0.20\. Your out-of-sample error could sit a full 20 percentage points above the best-in-class model purely from the search. Push n to 50,000 and the same bound falls to about 0.02\. Sample size is the only thing that buys the tax down, and it buys slowly. Infinite model classes (linear models over a continuum of coefficients, all trees) need a finer tool than counting. The count gets replaced by the VC dimension, the largest number of points the class can label in all possible ways. The Sauer-Shelah lemma says a class of VC dimension d can only produce polynomially many labelings, (n plus 1) to the d, instead of the two-to-the-n a truly unrestricted class would, and that collapses the bound to: $$ \\mathcal{R}\_n(\\mathcal{F}) \\;\\leq\\; \\sqrt{\\frac{2\\,\\mathrm{VC}(\\mathcal{H})\\,\\log(n+1)}{n}}. $$ The useful fact for a trader: for linear classifiers on p features, the VC dimension is exactly p, bounded above by the dimension of the underlying function space. So a linear sign model on 10 features, with n equal to 1000, carries an excess-risk bound near the square root of 20 times log(1001) over 1000, about 0.37\. At n equal to 100,000 it drops to about 0.05\. The dimension of your feature set enters the guarantee directly, and it enters through a one-over-root-n rate that no clever fit accelerates. A tree over the same features has a much larger VC dimension, which is one more reason the old article ["Linear Models' Hidden Symmetry Advantage"](https://aligrithm.com/linear-models-hidden-symmetry-advantage/) preferred the linear form for short-horizon sign models: fewer effective degrees of freedom, a tighter bound, and the buy-sell symmetry built in for free. ![The excess-risk bound falls only as roughly one over the square root of n, and higher VC dimension shifts the whole curve up](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_430-generalization_bound.png) This is the theorem behind every warning about degrees of freedom. More candidate models, more features, richer class, all push the bound up, and the only counterweight is more genuinely independent data. On a strategy with 2000 non-overlapping bars, the bound is loose enough that a large model's in-sample edge tells you almost nothing. ## Why you never optimize the loss you actually care about Misclassification loss is a step function, flat then a cliff at zero. Minimizing it over linear models is not convex and not even continuous, so it is computationally hopeless. The fix that makes machine learning tractable is to replace the 0-1 loss with a convex surrogate that hugs it from above. ![Zero-one loss against three convex surrogates: hinge, logistic, and exponential, all decreasing in the margin u](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_430-surrogate_losses.png) The three that run the field are hinge loss (max of 1 minus u and 0, the support vector machine), exponential loss (e to the minus u, the engine of AdaBoost), and logistic loss (log of 1 plus e to the minus u, logistic regression), where u is the margin y times h of x, positive when you are right. $$ \\phi\_{\\text{hinge}}(u) = \\max(1-u,\\,0), \\qquad \\phi\_{\\text{exp}}(u) = e^{-u}, \\qquad \\phi\_{\\text{log}}(u) = \\log\_2\\!\\big(1 + e^{-u}\\big). $$ Put a margin through them. At a correct-but-marginal call, u equal to 0.5, hinge charges 0.5, logistic charges log-base-2 of 1 plus e-to-the-minus-0.5, about 0.66, and exponential charges e-to-the-minus-0.5, about 0.61\. At a confident miss, u equal to minus 1, exponential explodes to e, about 2.72, while hinge charges only 2 and logistic about 1.88\. Exponential loss punishes bad misses hardest, which is why AdaBoost chases outliers so aggressively and why a few mislabeled bars can dominate its fit. The reason this substitution is legitimate, not a hack, is classification calibration: a convex surrogate whose derivative at zero is negative drives its minimizer to the same sign as the Bayes classifier. All three above qualify. You optimize a smooth convex proxy, and you still recover the right decision boundary. That single result is why gradient-based training works and why your logistic regression and your SVM are chasing the same target through different smooth approximations of a cliff. ## Sparsity is a bet, and the math prices it Constraining the coefficient norm controls the class complexity, and the choice of norm decides how the guarantee scales with the number of features p. Bounding the L2 norm (ridge) gives an excess-risk rate of order the square root of p over n. Bounding the L1 norm (lasso) gives order the square root of s log p over n, where s is the number of features that actually matter. $$ \\ell\_2:\\quad \\sqrt{\\frac{p}{n}} \\qquad\\qquad \\ell\_1:\\quad \\sqrt{\\frac{s\\,\\log p}{n}}. $$ Worked case. You have p equal to 1000 candidate features, but suspect only s equal to 10 carry signal, on n equal to 2000 bars. The L2 rate is the square root of 1000 over 2000, about 0.71\. The L1 rate is the square root of 10 times log 1000 over 2000, about 0.19\. When the truth is sparse, L1 trades a factor of p for a factor of s log p and wins by a mile. When every feature matters equally, the two rates converge and L2 is the cleaner choice. This is the formal statement of what ["Regularization from First Principles"](https://aligrithm.com/regularization-from-first-principles/) argued by intuition: lasso is a bet that most of your features are noise, and the bet pays only when it is true. ## Gradient descent converges at one over root k, and stochastic descent matches it Fitting these convex objectives means minimizing a convex function, often non-differentiable (hinge loss has a kink). Subgradients handle the kink: any supporting line below the function counts as a gradient, and gradient descent steps against it, projecting back onto the constraint set each step. The convergence guarantee is clean: $$ f(\\bar\\beta) - f(\\hat\\beta) \\;\\leq\\; \\frac{2LR}{\\sqrt{k}}, $$ where k is the number of iterations, R bounds the size of the feasible set, and L bounds the subgradient norm. Error falls like one over the square root of the step count. To get within 0.02 of the optimum with L and R both equal to 1 you need about 10,000 iterations, since 2 over the square root of 10,000 is 0.02. The move that makes this scale to real data is stochastic gradient descent. Computing the full gradient sweeps all n points every step. SGD replaces the full gradient with the gradient at a single randomly drawn point, and the expected-error bound is identical, 2LR over root k. You pay nothing in the rate for using one sample at a time instead of all n, which is why SGD trains models on datasets too large to hold a full gradient for. The catch the notes are honest about: this rate is for convex objectives, and a neural network is wildly non-convex, so the guarantee does not transfer. SGD works on deep nets empirically, not because a theorem promises it. ## Boosting and backprop, in one breath each AdaBoost is greedy empirical risk minimization on the exponential loss. Each round it fits a weak classifier to reweighted data and adds it with weight: $$ \\hat\\beta\_m = \\frac{1}{2}\\,\\log\\!\\left(\\frac{1 - \\operatorname{err}\_m}{\\operatorname{err}\_m}\\right), $$ where err-sub-m is the weighted error of that round's weak learner. A learner with error 0.4 gets weight one-half log of 0.6 over 0.4, which is 0.20\. A sharper learner with error 0.1 gets one-half log of 9, which is 1.10\. Better weak classifiers earn exponentially more say, and misclassified points get upweighted so the next round focuses on them. Gradient boosting drops the exponential-loss specialization and treats the whole thing as gradient descent in function space: fit each new tree to the negative gradient of the loss at the current predictions. For squared error that gradient is proportional to the residual, so gradient boosting is repeatedly fitting a tree to what the current model got wrong, then adding a shrunk version of it. That is the literal mechanism inside the LightGBM and XGBoost the old article ["From One Tree to Forests to Boosting"](https://aligrithm.com/from-one-tree-to-forests-to-boosting/) told you to use below five minutes. Backpropagation is the chain rule applied to the layered composition of a feedforward network, computing the gradient of the loss with respect to every weight in one backward sweep. Forward pass to get the activations, backward pass to accumulate derivatives layer by layer. It is fast because the compositional structure lets each layer reuse the derivative computed by the layer above it, and it is the only reason training a deep network is feasible at all. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_430-visual-0.png) ## Where this lands for a trader None of this tells you which features carry alpha. It tells you the exchange rate between complexity and data, and that rate is unforgiving. Excess risk falls like one over the square root of n, so quadrupling your sample halves your guarantee. Complexity, measured by parameter count or VC dimension, enters that guarantee directly and multiplies the tax. Variance from estimating coefficients is sigma-squared d over n whether the coefficients help or not. Averaging correlated models hits a floor set by their correlation, not by how many you stack. Every one of those is a theorem, not an opinion, and every one of them says the same thing the empirical robustness work has been saying: on short, noisy, dependent financial data, the small model with the honest out-of-sample test beats the big model with the pretty backtest, and the reason is arithmetic. The old article ["QLike: The Right Loss Function for Vol Forecasting"](https://aligrithm.com/qlike-the-right-loss-function-for-vol-forecasting/) made the point that the loss you optimize is a modeling choice with money attached; the surrogate-loss and calibration results here are the same point one level down, about the loss you can actually compute versus the one you care about. Pick the model your sample size can afford, and let the rate decide, not the fit. ## KEY POINTS - The Bayes classifier bets on whichever sign has probability above one-half, and its error rate is the irreducible floor. No model beats it, so a direction model hitting 90% on liquid futures is a red flag, not a triumph. - Empirical risk minimization substitutes average training loss for the true risk you cannot see. Every fit you run does this, and the gap between the two is the whole subject. - Each estimated parameter adds sigma-squared over n of variance to out-of-sample error, useful or not. Variance is linear in the number of parameters d; bias only falls if the parameter helps. This is why the optimal model is U-shaped short of the biggest one you can fit. - Cross-validation is a deliberately biased estimate that scores a model trained on a fraction of the data. On autocorrelated financial data the folds leak the future unless you purge and embargo, which is why the CV number flatters bad models. - A random forest's variance cannot fall below the tree-to-tree correlation rho times a single tree's variance, no matter how many trees you grow. The feature sampling exists to push rho down, not to add trees. - Excess risk is bounded by roughly the square root of (VC dimension times log n over n). More candidate models, more features, and richer classes all raise the tax; only more independent data lowers it, and it lowers like one over root n. - You never minimize misclassification loss directly; you minimize a convex surrogate (hinge, logistic, exponential) that is classification-calibrated, meaning its minimizer has the right sign. This is why gradient training works at all. - L1 (lasso) beats L2 (ridge) at a rate of square-root s-log-p over n versus square-root p over n, but only when the truth is sparse. Sparsity is a bet the math prices explicitly. - Gradient descent and stochastic gradient descent both converge like 2LR over root k on convex objectives, so using one data point at a time costs nothing in the rate. The guarantee does not extend to non-convex nets. - Gradient boosting fits each new tree to the current residual; AdaBoost weights each weak learner by one-half log of (1 minus error) over error; backprop is the chain rule in one backward sweep. All three are the machinery under the models the ML arc told you to run. ## References - [Statistically Sound Indicators for Financial Market Prediction - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Statistically+Sound+Indicators+for+Financial+Market+Prediction+Timothy+Masters&ref=aligrithm.com) - [Systematic Trading - Robert Carver (Amazon)](https://www.amazon.com/s?k=Systematic+Trading+Robert+Carver&ref=aligrithm.com) - [Mathematics of Machine Learning (maths of ML)](https://scholar.google.com/scholar?q=Mathematics+of+Machine+Learning&ref=aligrithm.com) --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ### 5.47 DRL Market Making With a Periodic Signal and Real Latency URL: https://aligrithm.com/drl-market-making-with-a-periodic-signal-and-real-latency/ Last updated: 2026-08-15T06:37:06.000Z Most reinforcement-learning market-making papers cheat in the same two places, and once you know where to look you stop believing their equity curves. First, they take event-driven tick data, the actual stream of order placements, cancellations, and fills, and resample it onto a fixed clock, one snapshot every second or every hundred ticks, which throws away the exact microstructure that a market maker lives and dies on. Second, they assume your order hits the book the instant you decide to send it, which is a fantasy in a business where the whole game is measured in milliseconds. Jiafa He, Cong Zheng, and Can Yang at HKUST built a market maker that refuses both shortcuts: it runs tick-by-tick in physical time on a reconstructed level-3 order book from Coinbase BTC-USD, and it carries explicit latency for both submitting and cancelling orders. The headline results are what you would hope for. Proximal Policy Optimization beats Double, Dueling, and Noisy DQN, and beats a fixed-spread quoter and a simplified Avellaneda-Stoikov model. Bolting a periodical alpha signal onto the tick-level agent nearly doubles PPO's return on real data. But the interesting parts are the two things that should make you cautious: a latency result that looks like good news and is actually a risk artifact, and a simulator the authors themselves admit is missing the one behavior that matters most. The old article ["The Three Pillars of Market Making: Fair Price, Spread, Skew"](https://aligrithm.com/the-three-pillars-of-market-making-fair-price-spread-skew/) laid out what a quote actually is; the old article ["Why Market-Making Simulations Don't Work"](https://aligrithm.com/why-market-making-simulations-dont-work/) laid out why most of these backtests lie. This paper is a good chance to hold both in your head at once. ## The two risks you are actually fighting A market maker posts a bid and an ask and hopes to earn the spread on the round trip. Two risks stand between the maker and that spread. The first is inventory risk: you accumulate a position you did not want because only one of your two quotes kept filling. The paper's own example: if the true fair price is 90 dollars and you misjudge it as somewhere between 95 and 105, you quote a bid at 90 and an ask at 105, only your bid fills, and now you are long an asset that is about to fall. The second is adverse selection: your resting limit orders are public, so a faster, better-informed trader picks off the stale one before you can move it. The old article ["The Three Pillars of Market Making: Fair Price, Spread, Skew"](https://aligrithm.com/the-three-pillars-of-market-making-fair-price-spread-skew/) made the point that skew's real job is not managing ordinary variance but blocking informed traders from loading you with toxic inventory. Everything in this paper is an attempt to control those two risks tick by tick: the tick-level agent fights inventory risk with fast reactions, and the periodical alpha signal fights adverse selection by predicting where price is about to go. ## The simulator: tick intervals and a random-walk mid Before touching real data, the authors build a simulator calibrated to real Coinbase top-of-book data so they can test in controlled conditions. Two random pieces drive it. The time between consecutive ticks is drawn from a log-normal distribution. $$ \\Delta T = e^{\\mu + \\sigma Z} $$ _This post is for paying subscribers only._ ### 3.38 The Signal Ceiling: Why No Single-Bar OHLCV Edge Beats Costs in MNQ URL: https://aligrithm.com/the-signal-ceiling-why-no-single-bar-ohlcv-edge-beats-costs-in-mnq/ Last updated: 2026-08-15T06:37:06.000Z Open any retail futures forum and you will find the same fourteen setups traded as gospel: the opening range breakout, the gap fade, the gap continuation, the volume spike, the liquidity grab reversal, the Asia session expansion. Each comes with a chart, a win rate, and a story about why it works. Mathias Mesfin took the whole list, wired it to a bar-close signal and next-bar-open fill on Micro E-Mini Nasdaq futures, and ran it across 947 trading days of five-minute data. Fourteen signal families. Zero survived a serious test. Not one cleared a T-statistic of 2.0 out of sample with enough trades to trust and a positive return after two points of cost. That is not a bad backtest. That is the answer. This is the article the whole robustness pillar has been building toward, so read the null result as the finding, not the disappointment. The old article ["The Backtest Integrity Checklist"](https://aligrithm.com/the-backtest-integrity-checklist/) gave you the 32 items a strategy has to pass before deployment. This article shows what happens when you run an honest gauntlet against the most popular intraday patterns on one of the most liquid instruments a retail trader can touch. They die, and they die for a reason with a number attached. ## The setup that makes the test honest Start with the data because the design is where most retail backtests already lost. The primary dataset is 72,604 five-minute OHLCV bars for MNQ continuous front-month futures, regular trading hours only, 09:30 to 16:00 ET, December 2021 through August 2025\. After dropping partial days and session-boundary junk, 947 clean trading days remain. A second instrument, Micro Gold (MGC), gets 1,091 days for a cross-instrument check. Two design choices carry the whole study. First, every signal is computed at bar close, and every entry fills at the open of the next bar. No signal touches the bar it fires on for its entry price. That single rule kills same-bar fill bias, the quiet lookahead that inflates most retail results, where the "entry" secretly happens at a price you could never have gotten. Second, a fixed two-point round-trip friction cost hits every MNQ trade, about $4.00 per micro contract, covering the bid-ask spread, exchange fees, and conservative slippage. In high-volatility windows where the MNQ spread widens to two ticks or more, two points understates the real cost, so the assumption leans generous to the strategies, not to the conclusion. Parameter selection runs on an expanding walk-forward: train on 2022 and test on 2023, then train on 2022 to 2023 and test on 2024, then train on 2022 to 2024 and test on 2025\. The test window is never touched until the final scoring step. The old article ["CSCV: A Direct Probability of Backtest Overfit"](https://aligrithm.com/cscv-a-direct-probability-of-backtest-overfit/) showed why a single in-sample-best pick lies to you about capacity. Walk-forward is the cheaper defense: you never let future data choose your parameters, so the out-of-sample number is a genuine estimate instead of a flattering one. ## The five-gate gauntlet A signal passes only if it clears all five criteria at once. Clearing four is a fail. | Criterion | Threshold | Why it is there | | ---------------- | ---------------------------- | -------------------------------------------------------------------------- | | T-statistic | T >= 2.0, out-of-sample only | Minimum statistical significance; in-sample T-stats count as zero evidence | | Trade count | N >= 30, out-of-sample | Below 30 trades the variance is too wide for any inference | | Net return | Positive after 2-pt friction | Gross edge is not economic edge; costs come out first | | Year stability | Consistent across all years | A single good year is treated as a fluke until proven otherwise | | Permutation test | p < 0.05 where applicable | Confirms the result is not a random-sequence artifact | The permutation gate is the same idea from the old article ["Permutation Tests for Indicator Significance"](https://aligrithm.com/permutation-tests-for-indicator-significance/): shuffle the labels, rebuild the null distribution, and ask whether the observed edge sits outside what randomness produces. The trade-count floor of 30 echoes the old article ["The Trade-Frequency Floor: Choosing a Threshold Honestly,"](https://aligrithm.com/the-trade-frequency-floor-choosing-a-threshold-honestly/) where tightening a rule until it fires ten glorious times is how you manufacture a profit factor out of luck. These gates are deliberately harsh. In a market as competitive as MNQ, the prior probability that free OHLCV data hides a deployable edge is low, so a loose bar would just print false positives. ## The body count Fourteen families, run through the gauntlet. The verdicts, in net T-statistic terms, look like this. ![Net out-of-sample T-statistic for all fourteen MNQ signal families against the T equals 2.0 pass gate, with the two positive controls plotted in green above the line](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_418-tstat_ceiling.png) Walk the list. The opening range breakout, immediate entry, produces a net T of 1.17 long and negative short. Stretch the hold to fifteen bars and the long breakout climbs to T = 1.50, the best of the genuine OHLCV signals, still short of the gate. The pullback-entry variant, waiting for price to retrace to within five points of the breakout before entering, stops out 80.7% of the time at a twenty-point stop and returns net -4.44 at T = -1.27\. In MNQ a large share of breakouts fail and reverse, so the pullback entry is systematically wrong, not merely weak. The Asia session expansion signal is worse than useless, and instructive about why. Fading nothing, just trading continuation on bars whose range exceeds 1.5x the rolling mean, gives T = -10.96 at the next bar. That is one of the strongest directional results in the entire study, pointing the wrong way. The expansion burst is real, but it lives and dies inside the expansion bar. By the time the bar closes, the signal fires, and you fill at the next open, the move is spent, and you are buying the top of a spike that is already reversing. The liquidity grab reversal has the same disease: 6,442 events, fade direction net -2.20 at T = -14.12, continuation direction net -1.80 at T = -13.24\. Both directions lose. The signal carries 0.20 to 0.80 points of real directional content, and friction of two points eats it whole regardless of which way you lean. Gaps split into two hypotheses and both fail. The gap fill fade, betting overnight gaps close during RTH, returns net -1.31 to -2.24 across entry times at 09:30, 09:45, and 10:00, with T-stats of -0.32 to -0.59, indistinguishable from noise. MNQ gaps do not reliably fill. Volume tells the same null: the volume spike momentum signal on 2,119 up-spike bars returns T = +0.07, and the down-spike variant on 2,409 bars returns T = -0.64\. With samples over two thousand trades, those near-zero T-stats are precise estimates of nothing, not small-sample scatter. Volume magnitude at the bar level does not predict next-bar direction. The volatility-volume-gap classifier fires on 4.4% of days and flags genuinely distinct behavior, a 25.6 basis point next-day return spread and a 77.6% peak-reversal rate, but every directional rule built on it fails the gauntlet. Event-day trend on 993 high-impact releases dies because the drift is fully contained in bars one to five, the news spike itself; measured from bar six, T sits between 0.14 and 0.69, and the two biggest releases (NFP and CPI at 08:30 ET) are outside RTH prop-firm rules anyway. Even the cross-instrument check fails: Ornstein-Uhlenbeck mean reversion on MGC gold returns T = -4.49 at the tightest threshold, and the 60-minute version has an 7.85-bar half-life that overruns a single session. ## The gross edge ceiling Strip out the friction and look at the raw directional content, because that is where the structural finding lives. Across all fourteen families, the best gross return per trade at the most favorable horizon lands between roughly 1.05 and 1.50 points for the signals that trade often enough and consistently enough to count. The round-trip friction is 2.0 points. The math is not subtle. $$ \\text{Net}\_{\\text{trade}} = \\text{Gross}\_{\\text{trade}} - \\text{Friction}\_{\\text{round-trip}} $$ Net per trade is the gross directional move the signal captures minus the cost of getting in and out once. Work a real one: the fifteen-bar ORB long, the best-behaved OHLCV signal in the study, captures +4.82 points gross at its best horizon. Subtract two points of friction and you keep +2.82 net. That sounds fine until you remember it still fails, because 2.82 net across 447 trades only produces T = 1.50, and the year split (net -1.3 in 2022, +4.5 in 2023, +2.3 in 2024) shows the pooled positive number leans on one strong year. Now run the same subtraction on a signal near the true ceiling: the Asia expansion 2.5x captures +1.06 gross, and 1.06 minus 2.0 is -0.94 net. Negative before you account for anything else. Most of the fourteen families sit right there, with a gross edge smaller than the cost of trading it. ![Best gross return per trade for each signal family plotted against the two-point round-trip friction line, most families falling short of the cost](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_418-gross_edge_ceiling.png) Two bars in that chart clear the friction line, and both are traps the gauntlet catches. ORB long at +4.82 gross fails on T and year-stability, as above. The gap continuation short prints +16.52 gross and +14.52 net at T = 3.23, the single most tempting result in the paper, and it dies on the trade-count floor: 22 trades across three years, twelve in 2022, six in 2023, four in 2024, a signal decaying toward extinction. That is exactly the failure mode the old article ["The Trade-Frequency Floor: Choosing a Threshold Honestly"](https://aligrithm.com/the-trade-frequency-floor-choosing-a-threshold-honestly/) warned about, a gorgeous T-stat resting on a sample too thin to bet on and shrinking every year. Why does the ceiling sit near the friction cost instead of somewhere higher or lower? Competition sets it. MNQ is one of the most heavily traded futures on earth. Any OHLCV-visible pattern that predicts next-bar direction is an arbitrage that faster participants with lower costs will trade until the gross edge shrinks to roughly the friction cost of the marginal player. For a retail-accessible instrument that clearing level lands around one to two points gross on five-minute bars. The market does not leave a free point on the table for a signal every forum already knows. ## Why the trade-count floor is not optional The gauntlet's second gate, thirty trades minimum, is the one traders resent most, because it kills their prettiest signals. The T-statistic is why it cannot bend. $$ T = \\frac{\\bar{x}\_{\\text{net}}}{s\_{\\text{net}}} \\times \\sqrt{N} $$ The T-statistic is the mean net return per trade divided by the standard deviation of net returns, scaled by the square root of the trade count. The mean over the standard deviation is the per-trade signal-to-noise; the square-root-of-N term is how much you trust it after seeing N samples. Push the gap continuation short through it. Mean net is +14.52 and T is 3.23 on 22 trades, so the standard deviation implied is 14.52 times the square root of 22 divided by 3.23, about 21.1 points. A per-trade edge of 14.52 against a swing of 21.1 is genuine, but the square root of 22 is only 4.69, so the sample barely lifts it over the line. Add a handful of losing trades from a fourth year and that T collapses. The square-root scaling is unforgiving in both directions: it means a real edge needs volume to prove itself, and it means a thin sample can fake significance until the next drawdown arrives. Thirty is the floor because below it the standard error on the Sharpe and the T is too wide to separate skill from a good run. ## Instability is the real killer One pattern shows up again and again once you split the results by year. A signal looks alive because a single strong year drags the pooled number positive, while the other years sit flat or negative. ![Heatmap of year-by-year net return in points for each signal family across 2022, 2023, and 2024, showing single strong years next to losing years](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_418-year_instability.png) Read the ORB long row: net -1.3 in 2022, +4.5 in 2023, +2.3 in 2024\. The 2023 and 2024 numbers look tradeable in isolation, and a researcher who ran 2024 alone would report T = 2.07 with +9.14 net on the VVG continuation strategy and call it an edge. The full picture undoes it: that same VVG rule prints -9.51 in 2022 and a weak 2023, so the pooled result is noise wearing a good year as a costume. The heatmap is a field of red and blue with no column that stays one color, which is the visual signature of a market whose behavior shifts regime to regime rather than a durable structural effect. This is the single most common reason a signal fails the gauntlet, and it is invisible to anyone who reports one aggregate number. ## The controls that prove the harness works A falsification study has an obvious failure mode of its own: maybe the test is so strict it rejects everything, edge or not. So the paper plants two known-good signals as positive controls, the same move a lab runs when it spikes a sample with a substance it knows the assay should detect. The RTH Confluence signal returns mean net +15.77 points at T = 5.83 on 538 trades, with a walk-forward out-of-sample T of 3.11\. London Session Signal B returns net +5.77 at T = 5.15 on 289 trades, profit factor 2.42\. Both clear all five gates with T-stats near three times the threshold, which proves the gauntlet accepts real edge and rejects the rest. What separates the survivors from the fourteen corpses is the interesting part. The controls do not trade single-bar price patterns. They fire on regime classification, a GMM label plus a Markov transition probability plus a volume z-score, and they hold for twelve to fifteen bars, sixty to seventy-five minutes, not one to six. That longer hold lets a structural state transition accumulate enough points to clear friction, where a single-bar breakout never can. The ceiling applies to single-bar directional guesses from features every participant can see. It does not apply to slower, structural signals that read market state instead of the last candle. ## What this means if you trade the forum playbook The uncomfortable read for a retail trader: if your intraday MNQ strategy fires on a bar-close pattern and holds for a few bars, the study says your gross edge is too small to survive the cost of trading it, and the backtest that told you otherwise probably leaned on one of four confounds. It filled at bid-mid or some price you could not get. It skipped the full round-trip cost. It optimized on in-sample data that will not generalize. Or it reported the one winner out of a pile of tests and buried the losers, the survivorship bias the old article ["The Backtest Integrity Checklist"](https://aligrithm.com/the-backtest-integrity-checklist/) spends whole sections guarding against. The paper's whole value is that it logged every rejection instead of publishing only the survivor. None of this proves MNQ has no intraday edge. It proves a narrower, sharper thing: single-bar directional signals from public OHLCV features do not clear costs at five-minute resolution on this instrument under honest execution. Edge above the ceiling exists, but it lives in tick-level order flow, in structural regime detection, or in execution advantages retail does not have. That is where the next Pillar 3 work points, and it is a more honest map than the fourteen setups you started with. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_418-visual-0.png) ## KEY POINTS - Fourteen popular intraday OHLCV signal families were tested on 947 days of five-minute MNQ data with honest bar-close signals, next-bar-open fills, and two points of friction. None cleared the five-gate gauntlet. - The gauntlet requires all five at once: out-of-sample T >= 2.0, at least 30 trades, positive net after friction, stability across every year, and a permutation p below 0.05\. Passing four is a fail. - The gross edge ceiling is the finding. The best-behaved signals capture roughly 1.0 to 1.5 points gross per trade, and a two-point round-trip cost erases it. Net per trade equals gross minus friction, and for most families that number is negative before anything else. - The ceiling sits near the friction cost because competition arbitrages any public single-bar pattern down to the marginal participant's cost. The market leaves no free point for a setup every forum knows. - The trade-count floor is not bureaucracy. Because T scales with the square root of N, a thin sample can fake significance; the gap continuation short printed T = 3.23 on 22 trades and dies the moment the sample grows. - Year instability is the most common failure mode. One strong year drags the pooled number positive while the others sit red, and a single-year backtest would have called several dead signals tradeable. - The two positive controls (T = 5.83 and 5.15) prove the test can find real edge. They win by reading regime state and holding twelve to fifteen bars, not by trading single-bar price patterns, which is exactly the category the ceiling constrains. ## References - [Testing and Tuning Market Trading Systems - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Testing+and+Tuning+Market+Trading+Systems+Timothy+Masters&ref=aligrithm.com) - [Data Mining Algorithms in C++ - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Data+Mining+Algorithms+in+C%2B%2B+Timothy+Masters&ref=aligrithm.com) - [Structural Limits of OHLCV-Based Intraday Signals in MNQ Futures: A Systematic Falsification Study (ssrn-6709401)](https://papers.ssrn.com/sol3/papers.cfm?abstract%5Fid=6709401&ref=aligrithm.com) --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ### 6.57 Fair Value as an Adaptive Low-Pass Filter: LAFO for Mean Reversion URL: https://aligrithm.com/fair-value-as-an-adaptive-low-pass-filter-lafo-for-mean-reversion/ Last updated: 2026-09-22T14:20:43.000Z Every mean-reversion trade starts with a lie you tell yourself about where price "should" be. You call it fair value, you subtract it from the spot price, and you bet the gap closes. Most traders reach for a moving average and stop thinking. Xu, Firoozye, Koukorinis, Treleaven, and Zhu at UCL take the question seriously and reframe fair value as the output of a tunable low-pass filter, then train neural networks to produce it. The framework is clean and the math is honest. The trading results are a demo the authors themselves refuse to defend, and one of them is an in-sample 2-minute Sharpe of 11, which should make you laugh before it makes you curious. The old article ["Percentile-Rank Momentum With Hysteresis: Low-Churn Signals"](https://aligrithm.ghost.io/percentile-rank-momentum-with-hysteresis-low-churn-signals/?ref=aligrithm.com) fought the same enemy from the other side: a signal that flips too often bleeds you on costs, so you smooth it and add a deadband. This paper gives the smoothing a name, an objective function, and a dial. Fair value is a low-frequency component you extract from a noisy price; the only real decision is how much high-frequency noise to let through, and everything else follows from that. ## The dial: one window that spans MSE to the global mean The paper's core object is the local average filtering objective, LAFO. Instead of fitting the filtered line to every price point (which chases noise), it fits the average residual inside a sliding window of length K. $$ \\mathcal{L}\_{\\text{LAFO}}(\\hat{y}) = \\frac{1}{T} \\sum\_{t=1}^{T} \\bar{e}\_t^{\\,2} = \\frac{1}{T} \\sum\_{t=1}^{T} \\left( \\frac{1}{n\_t} \\sum\_{k=a\_t}^{b\_t} (y\_k - \\hat{y}\_k) \\right)^2 = \\frac{1}{T} \\, \\lVert A(\\mathbf{y} - \\hat{\\mathbf{y}}) \\rVert\_2^2 $$ Read it in pieces. The term y minus y-hat is the residual, the gap between the real price and your fitted fair value. The inner sum averages that gap over the last K bars, so e-bar-t is the local average residual ending at time t. LAFO squares those local averages and averages them over the whole series. The matrix A is just the sliding-average operator that turns raw residuals into windowed ones, so the whole thing collapses to the squared length of A times the residual vector. Because A-transpose-A is positive semidefinite, the objective is convex, so it has a well-behaved minimum. Work the window at its two extremes, because that is where the intuition lives. Set K equal to 1 and the window is a single bar: the local average residual is just the raw residual, A becomes the identity, and LAFO reduces to plain mean squared error, which fits every wiggle. Set K equal to T, the whole sample, and the objective only cares about the average of all residuals, so the best fit is a flat line at the global mean. A three-point series with residuals of plus 3, 0, and minus 3 has an MSE of 6 (the average of 9, 0, 9) at K equals 1, but a LAFO of 0 at K equals T because the residuals sum to zero. Same fit, opposite verdict. Every useful K sits between those poles, trading local responsiveness against smoothness. $$ K = 1 \\;\\Rightarrow\\; A = I \\;\\Rightarrow\\; \\text{standard MSE (fits all noise)} \\qquad\\qquad K = T \\;\\Rightarrow\\; \\text{rank}(W) = 1 \\;\\Rightarrow\\; \\text{global mean only} $$ That single knob is the whole pitch. Fair value is not a fixed recipe; it is a frequency cutoff, and K is where you set it. A Fourier analysis in the paper confirms the picture: larger K passes lower frequencies and kills more of the high-frequency chop. ![LAFO fitted to the same noisy series at K = 1, 100, 300, 500, 1500, 2500. K = 1 tracks every spike; larger K flattens toward a slow trend](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_488-lafo_window_size.png) Look at the red line, K equals 1: it lies on top of the noise, useless as a fair value because it says price is always already at fair value. The yellow line, K equals 2500, barely moves, so it calls almost everything a deviation and would have you fading real regime shifts. The middle windows are the only ones that separate signal from noise, and picking one is the actual job. ## Setting K without fooling yourself The paper is candid that K is a choice, not a constant handed down by nature, and it offers three data-driven ways to set it. Match K to the expected regime length, near the median regime half-life, so the filter accentuates deviations on the horizon you actually trade. Or use the return autocorrelation: take K as the smallest lag where the absolute autocorrelation drops below a threshold like 0.1, which is the point past which recent moves stop predicting the next one. Or run rolling walk-forward cross-validation over a grid of K values (8, 16, 32, 64, 128, and so on) and let out-of-sample error pick. Only the third method is honest for trading, and it is the one the paper skips in its actual backtest. The first two calibrate K to properties of the same sample you then trade, which is the in-sample bias the old article ["State-Space Models for Price: CryptoMamba vs Transformers (Skeptical)"](https://aligrithm.ghost.io/state-space-models-for-price-cryptomamba-vs-transformers-skeptical/?ref=aligrithm.com) spent its whole length warning about. A cutoff frequency tuned on the data it filters will always look sharp in the picture and blurry in production. ## The penalty picks the regime shape The window controls how much you smooth. A penalty term controls the shape of what survives, and this is where the framework earns its keep over a moving average. Add one of three penalties on the differences of the filtered line. $$ \\text{TV: } \\lambda \\sum\_{t=2}^{T} \\lvert \\hat{y}\_t - \\hat{y}\_{t-1} \\rvert \\qquad \\ell\_1\\text{ TF: } \\lambda \\sum\_{t=3}^{T} \\lvert \\hat{y}\_t - 2\\hat{y}\_{t-1} + \\hat{y}\_{t-2} \\rvert \\qquad \\ell\_2\\text{: } \\lambda \\sum\_{t=2}^{T} (\\hat{y}\_t - \\hat{y}\_{t-1})^2 $$ Each penalty punishes a different kind of movement. Total variation penalizes the sum of absolute first differences, the step sizes, so it drives most steps to exactly zero and leaves a piecewise-constant line that jumps only at genuine level breaks. The l1 trend-filtering penalty hits second differences, the changes in slope, so it produces a piecewise-linear line that bends only at a few regime transitions. The l2 penalty squares the first differences, which never forces anything to zero and instead spreads adjustment across every bar, giving a globally smooth curve. Lambda sets how hard each penalty pushes. Take three consecutive fair-value points at 100, 101, 100.5: the first difference is plus 1 then minus 0.5, and the second difference is minus 1.5, so TV pays 1.5 in penalty on the levels while l1 TF pays 1.5 on the single slope change. The choice of penalty is a bet on how the market's fair value actually moves: in flat steps, in straight trends, or in gentle drift. ![Penalised LAFO on piecewise-constant and piecewise-linear test data. TV holds flat steps, l1 TF tracks straight trends, l2 stays smooth, and both EMAs lag the regime breaks](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_488-lafo_penalties.png) The comparison against exponential moving averages is the point of the figure. The dashed EMA lines round off every corner and arrive late at each regime break, which is exactly the lag that costs mean-reversion traders money when the "deviation" they are fading is really the fair value having moved. The l1 penalties turn corners on time. That is a real improvement over a moving average, and it is the honest contribution of the paper. ## From filter to trade Once you have a fair value, the strategy is mechanical. Measure the relative gap between spot and filter, and take a position against it. $$ \\delta\_t = \\frac{y\_t - \\hat{y}\_t}{\\hat{y}\_t} \\qquad w\_t = \\begin{cases} -\\operatorname{sgn}(\\delta\_t)\\,\\min\\{1,\\ \\alpha \\max(0,\\ \\lvert \\delta\_t \\rvert - \\tau\_{\\text{entry}})\\}, & \\lvert \\delta\_t \\rvert \\geq \\tau\_{\\text{entry}} \\\\ 0, & \\lvert \\delta\_t \\rvert < \\tau\_{\\text{entry}} \\end{cases} $$ Delta-t is the mispricing: how far spot sits above or below fair value, in percent. The position rule does nothing while the gap is inside a deadband of width tau-entry, then leans against the gap with size scaling up to a cap of 1\. The minus-sign-of-delta means you short when price is above fair value and buy when it is below, the whole mean-reversion premise. Work it with the paper's settings, tau-entry of 0.0005 and alpha of 200\. If spot is 5000.5 and fair value is 5000, then delta is 0.5 over 5000, which is 0.0001, below the 0.0005 band, so you stay flat. If spot is 5005, delta is 0.001, above the band; the position is alpha times (0.001 minus 0.0005), which is 200 times 0.0005, so 0.1, a tenth of full size short. The deadband is the same anti-churn hysteresis idea as the old article ["Percentile-Rank Momentum With Hysteresis: Low-Churn Signals,"](https://aligrithm.ghost.io/percentile-rank-momentum-with-hysteresis-low-churn-signals/?ref=aligrithm.com) here applied to a reversion signal instead of a momentum one. ## The results, and why the Sharpe of 11 is a warning The paper trades this on 2-minute S&P 500 data over two months, feeds every model the same LAFO window (K equals 80) and the same trading parameters, and charges 3 basis points per unit of turnover. The neural filters beat the moving averages by a wide margin. ![In-sample annualized Sharpe by filter on 2-minute SPX: both EMAs deeply negative, the neural filters strongly positive, 2-layer CNN highest at 11.0](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_488-strategy_sharpe.png) The 2-layer CNN posts an annualized Sharpe of 11.049, a hit rate of 0.479, and a Sortino of 39.13\. WaveNet lands 8.005, the deep Kalman filter 4.920\. Both exponential moving averages are deeply negative: EMA(10) at minus 18.265, EMA(30) at minus 14.761\. Read literally, a two-layer convolutional net turned the same fair-value framework that sinks a moving average into a Sharpe north of 11. Do not read it literally. A Sharpe of 11 on two months of 2-minute bars is not an edge, it is a flashing in-sample-overfit light. The authors say so themselves, in plain text: the models are "applied without systematic parameter calibration," the study is "primarily methodological," and out-of-sample evaluation "should be addressed in a separate paper." There is no walk-forward, no retraining, no second dataset. The window K, the penalty lambda, the entry band, and the position scale were all set on the same two months that produced the Sharpe. The negative moving-average numbers are just as suspect in the other direction: a fixed EMA lag fed the same tiny deadband on the same noisy series is being set up to lose. What the table actually shows is that a flexible filter fits two months of 2-minute SPX better than a rigid one. That is unsurprising and not tradable. The one durable finding hides in the same table. Model complexity did not help. The small 2-layer CNN beat the heavier WaveNet and the deep Kalman filter, and the paper attributes this to the larger models being data-hungry and unstable on a 6,000-sample set. That rhymes with the old article ["State-Space Models for Price: CryptoMamba vs Transformers (Skeptical)"](https://aligrithm.ghost.io/state-space-models-for-price-cryptomamba-vs-transformers-skeptical/?ref=aligrithm.com): on short financial samples, the fancy architecture usually loses to the simple one, and the burden of proof sits with anyone claiming otherwise. ## What to keep and what to bin Keep the framing. Fair value as a low-pass filter with an explicit cutoff is a genuinely better mental model than "use the 20-day average," because it forces you to name the frequency you are separating and to choose a penalty that matches how you think fair value moves. Keep the penalty menu: TV for level regimes, l1 trend filtering for trending regimes, l2 for smooth drift, each with a lag advantage over an EMA at the regime breaks. Keep the deadband trade rule; it is the same low-churn discipline that survives contact with transaction costs. Bin the performance numbers. Every Sharpe in the table is in-sample, single-asset, two months, with all filter and trade parameters fitted on the test window and no out-of-sample check, which is why the authors label it a demo. Treat the 11.0 as a stress test of your own credulity, not as evidence. Before any of this touches capital it needs the full gauntlet: walk-forward K selection, purged cross-validation, multiple assets, and a permutation test on the reversion signal. The paper builds a clean pipe and honestly declines to certify the water. That honesty is worth more than the headline. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_488-visual-0.png) ## Where this connects This is the smoothing engine under the old article ["Percentile-Rank Momentum With Hysteresis: Low-Churn Signals."](https://aligrithm.ghost.io/percentile-rank-momentum-with-hysteresis-low-churn-signals/?ref=aligrithm.com) That piece built a low-churn signal by ranking, then adding a deadband so the position stops flickering; here the deadband reappears as tau-entry, and the ranking-versus-raw-price question becomes a filtering-frequency question. Both are fights against the same tax: a signal that reacts to every tick pays the spread to death, so you decide, on purpose, how much of the high-frequency world to ignore. It also belongs next to the old article ["Wave Velocity and Acceleration: Reading When the Market Runs Out of Gas,"](https://aligrithm.com/wave-velocity-and-acceleration-reading-when-the-market-runs-out-of-gas/) which chased the same grail from digital signal processing: reading a turn early, without the lag a moving average adds. That piece got its early read from the exact derivative of a local sine model; LAFO gets its from penalty structure. The moving average lags because it is a fixed low-pass filter with no penalty; the l1 penalties in LAFO cut the lag at regime breaks precisely because they allow sharp, sparse changes instead of spreading every adjustment across the window. Same target, a filter that is smooth in the quiet and fast at the breaks, reached with an explicit objective instead of a clever recursion, and both inherit the same fragility when the model of the underlying shape is wrong. And the skepticism is borrowed straight from the old article ["State-Space Models for Price: CryptoMamba vs Transformers (Skeptical)."](https://aligrithm.ghost.io/state-space-models-for-price-cryptomamba-vs-transformers-skeptical/?ref=aligrithm.com) A tiny out-of-sample window, a single asset, and a headline number too good to be true are the exact tells that piece flagged. The 2-layer CNN beating WaveNet and the Kalman filter is the same lesson twice: on short, noisy financial data, complexity is a liability until proven otherwise, and a two-month Sharpe of 11 is a hypothesis about overfitting, not a strategy. ## KEY POINTS - Fair value is better modeled as the output of a tunable low-pass filter than as a fixed moving average. LAFO fits the average residual inside a window of length K, so K is an explicit dial for how much high-frequency noise you let through. - The window spans the two things traders already do badly: K equal to 1 is plain MSE that fits all noise (fair value equals price, useless), and K equal to the whole sample is a flat global mean that ignores regimes. The tradable filter lives strictly between. - The penalty sets the regime shape. Total variation gives flat steps, l1 trend filtering gives straight trends, l2 gives smooth drift, and all three turn corners at regime breaks faster than an exponential moving average, which lags. - Trading is a deadband rule on the relative gap between spot and filter: stay flat inside tau-entry, then lean against the gap up to a size cap. This is the same anti-churn hysteresis that keeps costs from eating a reversion signal. - On 2-minute SPX over two months at 3 bps cost, the 2-layer CNN posted an in-sample Sharpe of 11.0, WaveNet 8.0, the deep Kalman filter 4.9, while EMA(10) and EMA(30) were deeply negative at minus 18.3 and minus 14.8. - Do not trust those Sharpes. The authors state the study is methodological, with no calibration and no out-of-sample test, and all parameters were fitted on the same two months. A Sharpe of 11 on 2-minute bars is an overfit tell, not an edge. - The one durable result: the small CNN beat the larger WaveNet and Kalman models, because complex architectures are data-hungry and unstable on a 6,000-sample set. On short financial data, simple filters win until proven otherwise. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Systematic Trading - Robert Carver (Amazon)](https://www.amazon.com/s?k=Systematic+Trading+Robert+Carver&ref=aligrithm.com) - [Trading Systems - Urban Jaekle Emilio Tomasini (Amazon)](https://www.amazon.com/s?k=Trading+Systems+Urban+Jaekle+Emilio+Tomasini&ref=aligrithm.com) - [The Price Impact of Nothing: Rejected Orders as Predictors of Future Returns\* (ssrn-6250378)](https://papers.ssrn.com/sol3/papers.cfm?abstract%5Fid=6250378&ref=aligrithm.com) ### 6.53 Bitcoin's Overnight Returns Forecast the VIX URL: https://aligrithm.com/bitcoins-overnight-returns-forecast-the-vix/ Last updated: 2026-07-30T09:41:30.000Z Split one Bitcoin day into two pieces and only one of them predicts anything. The piece that runs while US stock exchanges are closed, from yesterday's 4pm close to today's 9:30am open, carries a signal for tomorrow's VIX. The piece that runs while those exchanges are open carries nothing. Gu, Lin, and Liu ran five-minute Bitcoin data from 2018 to 2023 through that split and found the overnight leg forecasts next-session equity volatility with a coefficient of minus 0.262 (t-statistic minus 2.49), while the trading-hour leg dies the moment you put the overnight leg beside it. The old article ["Chicken and Egg: Use the SPX to Time the VIX, Not Vice Versa"](https://aligrithm.ghost.io/chicken-and-egg-use-the-spx-to-time-the-vix-not-vice-versa/?ref=aligrithm.com) established that the VIX is downstream, a reaction to what the market does rather than a leading gauge of it. This paper adds a second upstream driver from a market that never closes. Bitcoin trades 24/7, so while the S&P sleeps, retail sentiment keeps pricing itself into BTC, and that overnight move front-runs the fear gauge when Wall Street reopens. ## The one decomposition that matters The whole result rides on cutting the Bitcoin day at US-equity boundaries. Take BTC prices, convert from UTC to Eastern Time, and split each day into an overnight session (4pm close to 9:30am open) and a trading-hour session (9:30am to 4pm). Three returns fall out. $$ R^{CC}\_{BTC,t} = \\ln BTC^{Close}\_t - \\ln BTC^{Close}\_{t-1} $$ $$ R^{ON}\_{BTC,t} = \\ln BTC^{Open}\_t - \\ln BTC^{Close}\_{t-1} $$ $$ R^{TH}\_{BTC,t} = \\ln BTC^{Close}\_t - \\ln BTC^{Open}\_t $$ Close-to-close is the full day. Overnight is open minus yesterday's close, the window while equities are shut. Trading-hour is close minus open, the window while they trade. The three tie together: overnight plus trading-hour equals close-to-close, so the decomposition just partitions the day, it does not throw anything away. Work a number. Bitcoin closes at 40,000 on day t-1\. It opens at 40,400 on day t, so the overnight return is the log of 40,400 over 40,000, which is 0.00995, about plus 1.00%. It then closes day t at 40,100, so the trading-hour return is the log of 40,100 over 40,400, which is minus 0.00746, about minus 0.74%. The close-to-close return is the log of 40,100 over 40,000, about plus 0.25%, and it equals the overnight plus the trading-hour, 1.00 minus 0.74\. That specific shape, a positive overnight move that partly reverses during the day, is the sentiment fingerprint the paper hunts for. ## Why the overnight leg is sentiment Across the full sample, overnight Bitcoin returns average plus 0.093% and trading-hour returns average minus 0.029%, a gap of 0.122% that repeats day after day. Positive overnight, negative during the day. That is the same pattern Berkman and Aboody documented in stocks, where retail traders pile in at the open and pay for it, and the paper imports it to Bitcoin as a sentiment test. The clean test sorts days by retail attention and watches what happens to each leg. Attention is measured two ways, Google search volume for "Bitcoin" and the count of unique active Bitcoin addresses, each taken as a deviation from its own trailing monthly average. Sort into deciles, compare the top attention decile to the bottom, and the overnight leg lights up while the trading-hour leg stays dark. ![Overnight BTC returns rise with retail attention while trading-hour returns stay flat, so the daily reversal grows with attention](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_484-attention_reversal.png) The high-minus-low spread in overnight returns is 1.133% (t-statistic 2.072) using Google search and 0.987% (t-statistic 2.033) using active addresses. Both clear significance. The trading-hour spread is the opposite: minus 0.114% (t-statistic minus 0.361) and minus 0.288% (t-statistic minus 0.880), neither distinguishable from zero. The daily reversal spread, overnight minus trading-hour, comes in at 1.247% and 1.275%, both significant. Read plainly: when retail piles into Bitcoin overnight, the overnight return jumps and then gives part of it back during the day, and the jump-and-reverse grows with how much attention the crowd is paying. That is overreaction, and overreaction is sentiment. Hold the skepticism for a second. Those t-statistics sit around 2.0 to 2.3, which is barely across the conventional line, on six years of data. The pattern is real in this sample, but it is not the kind of overwhelming signal that survives every subperiod without a scratch. ## Overnight Bitcoin predicts the VIX, trading-hour Bitcoin does not Now point the overnight leg at the fear gauge. The paper regresses the next trading-hour change in log VIX on contemporaneous overnight Bitcoin returns, controlling for S&P 500 futures returns (both overnight and lagged trading-hour), contemporaneous overnight VIX changes, three lags of VIX changes, holiday dummies for the first day after a market closure, and FOMC announcement dummies. $$ \\Delta \\ln V^{TH}\_t = \\alpha + \\beta\_1 R^{ON}\_{BTC,t} + \\beta\_2 R^{TH}\_{BTC,t-1} + \\text{(SPX futures, VIX lags, holiday, FOMC)} + e\_t $$ The left side is the trading-hour change in log VIX, how much the fear gauge moves once equities reopen. Beta-one is the payload: the loading on Bitcoin's overnight return. Beta-two is the loading on lagged trading-hour Bitcoin, the leg that turns out to be noise. Everything in the parenthetical is a control, there to make sure the Bitcoin signal is not just repackaged S&P futures movement, a holiday effect, or a Fed-day effect. Beta-one comes out at minus 0.262 with a t-statistic of minus 2.49\. Negative and significant: a stronger overnight Bitcoin move (more sentiment) predicts a lower VIX in the next session. Add the S&P futures controls and holiday effects and it barely budges, minus 0.255 (t-statistic minus 2.44). Add FOMC effects and it holds. And once the overnight leg is in the regression, the lagged trading-hour Bitcoin coefficient goes insignificant. The predictive information sits entirely in the hours when US equities are closed. ![Only overnight Bitcoin returns carry a significant loading on the next trading-hour VIX change](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_484-overnight_predicts_vix.png) The economic size is modest, and the paper is honest about it. $$ \\text{effect of } 1\\sigma = \\frac{0.036 \\times 0.262}{0.072} = \\frac{0.0094}{0.072} \\approx 13\\% $$ Bitcoin overnight returns have a standard deviation of about 0.036\. Multiply by the coefficient 0.262 and a one-standard-deviation move shifts the predicted VIX change by 0.0094\. The standard deviation of the trading-hour VIX change is 0.072, so 0.0094 is 13% of it. One big overnight swing in Bitcoin shaves 13% off the typical size of the next VIX move. That is a real dent, not a market-mover. This forecasts the change in the VIX, not its level, and the fraction of variance explained is small. ## Out of sample, and the workhorse model In-sample coefficients are cheap. The paper runs a proper out-of-sample horse race with a 500-day rolling window (a third of the sample), producing one-day-ahead forecasts. Two model families compete: vector autoregressions (VAR) that model the overnight and trading-hour VIX legs jointly, and a heterogeneous autoregressive model (HAR) that predicts the VIX level from its own recent history at three horizons. $$ \\ln V^C\_t = \\alpha + \\beta\_1 \\ln V^C\_{t-1} + \\beta\_2 \\ln V^{C(5)}\_{t-1} + \\beta\_3 \\ln V^{C(22)}\_{t-1} + \\delta\_1 R^{ON}\_{SPF,t} + \\delta\_2 R^{TH}\_{SPF,t-1} + Z\_t + e\_t $$ The HAR predicts today's log VIX from yesterday's log VIX, its 5-day average, and its 22-day average, the daily/weekly/monthly memory horizons, plus S&P futures returns. The Z term is where the Bitcoin signal plugs in. Set Z to the overnight Bitcoin return and you get HAR-ON; set it to trading-hour or close-to-close Bitcoin and you get the alternatives. HAR-ON wins. It posts the lowest forecast errors, and in the Model Confidence Set test it reaches a p-value of 1.000 across all four loss metrics, meaning you cannot statistically exclude it from the set of best models under any of them. The Diebold-Mariano tests add the kicker: close-to-close Bitcoin returns are significantly worse than their overnight counterparts. Blending the useful overnight leg with the useless trading-hour leg into a single daily number destroys the signal. ## Does it make money? The VIX is not tradable, so the paper trades VIXY, the short-term VIX futures ETF, with a plain directional rule. At each close, go one dollar long if the model predicts the VIX rises, one dollar short if it predicts a fall, and unwind before the next close. Costs are charged at 9.7 basis points per dollar traded. Performance is graded against buy-and-hold on two metrics. $$ IR = \\frac{\\text{mean(strategy return} - \\text{buy-and-hold)}}{\\text{sd(strategy return} - \\text{buy-and-hold)}}, \\qquad CER = \\mu - \\tfrac{1}{2}\\gamma\\sigma^2 $$ The information ratio is the average return above buy-and-hold divided by the standard deviation of that excess, a Sharpe-style score for the timing skill. The certainty-equivalent return is the return a risk-averse investor would accept for certain instead of the risky strategy, with gamma the risk aversion; the CER gain is the strategy's CER minus the benchmark's. Worked out, HAR-ON with daily rebalancing lands an annualized information ratio of 0.986, with CER gains of 1.20 for a low-risk-aversion investor (gamma equals 1) and 0.519 for a high-risk-aversion one (gamma equals 3). Swap VIXY for the VXX ETN and HAR-ON still posts 0.894\. The semi-daily VAR versions trail, VAR3-ON at 0.406 on VIXY and VAR2-ON at 0.322 on VXX. ![Annualized information ratios of the overnight-augmented strategies across VIXY and VXX, daily versus semi-daily rebalancing](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_484-strategy_ir.png) An information ratio near one after costs is a strong number. Read what it is measured against before you get excited: excess return over buy-and-hold VIXY. Buy-and-hold VIXY is a structural bleeder, the same downhill-forever instrument the old article ["Chicken and Egg: Use the SPX to Time the VIX, Not Vice Versa"](https://aligrithm.ghost.io/chicken-and-egg-use-the-spx-to-time-the-vix-not-vice-versa/?ref=aligrithm.com) built its whole short-VX case around. A directional timer that goes short more often than long will beat a guaranteed loser, and part of this information ratio is that tailwind rather than pure VIX-direction skill. ## What to distrust The signal is real in this window and thin everywhere it counts, so price it before you believe it. Six years, one of them COVID. The sample runs January 2018 to December 2023, and the out-of-sample period opens in 2020, right into the largest volatility shock in the series. The paper splits pandemic (through May 2023) from post-pandemic and reports the edge holds in both, which helps, but the post-pandemic slice is barely seven months. The sentiment sorts carry t-statistics around 2.0 and the headline regression a t-statistic of 2.49\. Cross the multiple-testing bar the old article ["What a Market Price Actually Is: Capital-Weighted Consensus and the Brier Skill Score Gate"](https://aligrithm.com/what-a-market-price-actually-is-capital-weighted-consensus-and-the-brier-skill-score-gate/) insists on, and a t of 2.49 on a single hand-chosen predictor is suggestive, not settled. The overnight window is a construction, not a law. Bitcoin trades continuously, so "overnight" is defined by a market Bitcoin has nothing to do with, the US equity session. The choice is motivated (retail sentiment leaks in when stocks are closed) but it is still a researcher's cut of a 24/7 series, and the result depends on that cut lining up the way it does over these particular years. The trade is timing a decaying instrument. VIXY and VXX bleed by design, so the strategy's real job is deciding when to be short the bleed and when to duck a spike, exactly the short-volatility problem with the fat left tail. The paper charges 9.7 basis points and rebalances daily, but a directional short-vol book is one gap away from a very bad day, and an information ratio computed on daily returns understates that. Sizing and a hard stop are the difference between collecting the premium and getting run over, and this paper models neither. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_484-visual-0.png) ## Where this connects This sits directly on top of the old article ["Chicken and Egg: Use the SPX to Time the VIX, Not Vice Versa."](https://aligrithm.ghost.io/chicken-and-egg-use-the-spx-to-time-the-vix-not-vice-versa/?ref=aligrithm.com) That piece said the VIX is an effect, best predicted by reading its causes, and the tradable form is timing a short-vol book. This paper hands you a second cause, Bitcoin's overnight sentiment, that fires before the S&P even opens, and it trades the same kind of VIX ETP with the same directional logic. Two assets, one message: the fear gauge is downstream of everything. It also extends the belief-pricing thesis of the old article ["What a Market Price Actually Is: Capital-Weighted Consensus and the Brier Skill Score Gate."](https://aligrithm.com/what-a-market-price-actually-is-capital-weighted-consensus-and-the-brier-skill-score-gate/) A Bitcoin overnight print is a capital-weighted vote by whoever is awake and trading while the US sleeps, and the paper's whole claim is that this vote carries sentiment information not yet in equity prices. And it rhymes with the cross-market logic of the old article ["Why FX Traders Must Watch Gold, Rates, and Equities,"](https://aligrithm.com/why-fx-traders-must-watch-gold-rates-and-equities/) where the driver of an instrument lives outside it. The VIX has no life of its own; its next move is written partly in an asset that trades while its own market is dark. Watch the market that never closes, and read the fear gauge before it opens. ## KEY POINTS - Split the Bitcoin day at US-equity hours: overnight is open minus prior close, trading-hour is close minus open. Overnight plus trading-hour equals the full close-to-close day, so the cut partitions the day without discarding data. - The overnight leg is a retail-sentiment measure. Overnight returns average plus 0.093% against minus 0.029% for trading-hour, and the overnight return rises with retail attention (high-minus-low spread 1.133% by Google search, 0.987% by active addresses) while the trading-hour leg shows no attention pattern. - Overnight Bitcoin predicts the next-session VIX change with a coefficient of minus 0.262 (t-statistic minus 2.49); more overnight sentiment forecasts a lower VIX. Once overnight is included, lagged trading-hour Bitcoin goes insignificant. - The economic size is modest: a one-standard-deviation overnight move cuts the standard deviation of the next VIX change by about 13%, and it forecasts the change, not the level. - Out of sample, the HAR model augmented with overnight Bitcoin (HAR-ON) wins, reaching a Model Confidence Set p-value of 1.000 on all four loss metrics. Close-to-close Bitcoin is significantly worse, so mixing the useful overnight leg with the useless day leg destroys the signal. - Trading VIXY directionally on the forecast, HAR-ON posts an annualized information ratio of 0.986 after 9.7 bps costs (0.894 on VXX), with the semi-daily VAR versions trailing at 0.32 to 0.41. - Distrust the magnitude. Six years with COVID in the out-of-sample window, marginal t-statistics near 2, an overnight window defined by a market Bitcoin does not trade in, and an information ratio measured against a structurally bleeding ETF that flatters any short-biased timer. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Systematic Trading - Robert Carver (Amazon)](https://www.amazon.com/s?k=Systematic+Trading+Robert+Carver&ref=aligrithm.com) - [Trading Systems - Urban Jaekle Emilio Tomasini (Amazon)](https://www.amazon.com/s?k=Trading+Systems+Urban+Jaekle+Emilio+Tomasini&ref=aligrithm.com) - [Beyond Conventional Sentiment Indicators: Bitcoin’s Hidden Potential in VIX Forecasting (ssrn-5186527)](https://papers.ssrn.com/sol3/papers.cfm?abstract%5Fid=5186527&ref=aligrithm.com) ### 6.52 Chicken and Egg: Use the SPX to Time the VIX, Not Vice Versa URL: https://aligrithm.com/chicken-and-egg-use-the-spx-to-time-the-vix-not-vice-versa/ Last updated: 2026-07-29T05:01:44.000Z Twenty years of retail research points the arrow one way. You read the VIX, and the VIX tells you where the S&P 500 is going. Oversold VIX means complacency, sell stocks; spiked VIX means panic, buy the dip. Connors built a cottage industry on it, and every trading forum still runs some version of the "VIX says buy" screenshot. Rob Hanna ran the tests both directions and found the arrow points the other way. SPX action predicts the VIX better than the VIX predicts the SPX. And the tradable version of that is not a stock-timing signal at all, it is a filter for shorting VIX futures. The old article "Volatility-Regime Filter for Any Strategy" already made the softer version of this point: volatility is the thing you can measure with a small sample and trust, while direction needs hundreds of trades before the number means anything. This piece takes it further. It says the predictable object in the SPX/VIX pair is the volatility instrument itself, and the best signal for it is an overbought/oversold reading on the index, not on the fear gauge. ## The direction everyone runs backward The VIX exists because the SPX moves, not the other way around. It measures 30-day implied volatility priced into SPX options, pulling from contracts in the 23-to-37-day window. When the market falls, traders bid up portfolio protection and option premiums rise, so the VIX rises. When the market drifts up quietly, nobody pays up for puts and the VIX sags. From 1990 to 2023 the closing VIX ranged from 9.14 to 82.69, and every spike lined up with an SPX drawdown: 1998, 2000-2003, 2008, 2020. That causal order matters. The VIX is a reaction to SPX price action, and option traders react rather than anticipate. If the VIX is downstream of the SPX, then reading the VIX to forecast the SPX is reading the effect to predict the cause. Reading the SPX to forecast the VIX is reading the cause to predict the effect. Hanna tests exactly that, and the data agrees with the plumbing. ## What a VIX reading actually says Before testing anything, get literal about what a VIX number means, because most people misread it. The VIX is annualized implied volatility. To turn it into an expected daily move you divide by the square root of trading days in a year. $$ \\sigma\_{\\text{daily}} \\approx \\frac{\\text{VIX}}{\\sqrt{252}} = \\frac{\\text{VIX}}{15.875} \\approx \\frac{\\text{VIX}}{16} $$ This is the "Rule of 16." The square root of 252 trading days is 15.875, near enough to 16\. A VIX of 16 prices in daily SPX moves of about 1%, meaning roughly two thirds of days stay under 1% and a third exceed it. Work the rest: a VIX of 32 implies 2% daily moves, 48 implies 3%, and for the VIX to sit at 80 the options market has to be pricing 5% daily moves. Realized 2% days in a row are rare, and stringing them together is rarer still, which is why a VIX above 30 or 40 does not last. High readings require high realized volatility to sustain them, and realized volatility mean-reverts fast. That single fact, that the VIX cannot stay elevated unless the SPX keeps thrashing, is the engine behind everything below. ## RSI(2), and why two periods instead of fourteen Hanna measures overbought/oversold with Wilder's RSI, but at a 2-day lookback, not the textbook 14. $$ \\text{RSI} = 100 - \\frac{100}{1 + RS}, \\qquad RS = \\frac{\\text{average gain over } n \\text{ bars}}{\\text{average loss over } n \\text{ bars}} $$ RSI runs 0 to 100\. When recent bars are mostly up, average loss shrinks, RS blows up, and RSI pushes toward 100 (overbought). When recent bars are mostly down, RSI collapses toward 0 (oversold). Work a 2-day example on the SPX. Say the last two daily changes were +0.2% and minus 1.8%. Average gain is 0.2 divided by 2, so 0.1; average loss is 1.8 divided by 2, so 0.9\. Then RS is 0.1 over 0.9, about 0.111, and RSI is 100 minus 100 divided by 1.111, which is 100 minus 90, so RSI(2) equals 10\. Deeply oversold after one ugly day. The short lookback is deliberate, and Hanna is honest that it is a choice rather than a law. A 14-period RSI parks almost everything between 30 and 70, so the oversold and overbought tails barely fire. A 2-period RSI spends real time below 20 and above 80, which gives cleaner delineation between buckets. It also makes the signal a mean-reversion detector, not a trend gauge. Treat the 2 as tuned to short-horizon reversion, and hold the thought when the overfitting question comes up later. ## Test one: the VIX is no better than the SPX at timing the SPX Start with the long horizon. Hanna runs three simple SPX trend filters (price above where it closed a year ago, price above the 200-day average, and the 50-day above the 200-day golden cross), goes to cash in Fed funds otherwise, and compares to buy-and-hold. Then he builds the mirror image on the VIX (long SPX when VIX is below its year-ago level, below its 200-day, or its 50-day below its 200-day) since the VIX runs inverse. Buy-and-hold returned 7.91% a year with a brutal 56.78% max drawdown, a CAR/MDD of 0.14\. Two of the three SPX filters beat that total return, and all three cut drawdown hard; the "SPX up year" filter posted 7.97% at a 25.04% drawdown, CAR/MDD 0.32, more than double the return-per-unit-of-pain. The VIX filters did nothing useful. Every one lagged buy-and-hold on net profit and matched or beat it on CAR/MDD only by accident, all landing at 0.11 to 0.13\. Long-term VIX levels do not filter the SPX. The short horizon tells the same story with the RSI(2) readings, 2007 to 2023\. An SPX RSI(2) at or below 20 (no long-term filter) led to a next-day win rate of 57.66% and a profit factor of 1.41; an SPX RSI(2) above 80 dropped to 51.5% wins and a profit factor of 0.91\. The overbought/oversold split is sharp, and it gets more extreme in downtrends, where oversold below the 200-day averaged +0.32% next day (profit factor 1.53) and overbought averaged minus 0.19% (profit factor 0.71). Now swap in RSI(2) of the VIX to time the SPX. An oversold VIX (RSI at or below 20) produced a 55.51% win rate and a 1.14 profit factor, against a no-filter baseline of 54.79% and 1.11\. That is noise. An overbought VIX carried a little edge, but the marquee "VIX oversold, buy stocks" trade barely clears doing nothing. ## The easier question: where is VX headed Here is the move most traders miss. You do not have to forecast the SPX at all. Ask the easier question, where are VIX futures going, because they have a structural downhill tilt that the SPX does not. VIX itself is not tradable. VIX futures (ticker VX) are, and they bleed. Hanna builds a continuous VX contract rolled the Tuesday before each Wednesday expiry, and it falls from a roll-adjusted 229.9 on October 31, 2006 to 13.05 at the end of 2023\. Counter-trend spikes are violent, but the recovery to new lows is fast, and that asymmetry is the whole edge. ![Time underwater for long SPX versus short VX across the two bear markets the paper details](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_483-drawdown_length.png) Look at the two bear markets. In 2007, the SPX topped in October and did not make a new high until May 2013, roughly five and a half years underwater. VX bottomed in May 2008, seven months after the SPX top, then made a new low in February 2010, about a year and a half. In 2022, the SPX spent two years getting back to a new high; VX barely went six months without a new low and was printing fresh lows in July 2022 before the SPX had even bottomed. A short-vol position does not need a rally to recover. It just needs the market to calm down, and markets calm down faster than they climb back. That is the same "downside persistency" the abstract leans on, and it is why a short-VX book spends far less time in drawdown than a long-SPX book. ## Filtering the short-VX trade: SPX RSI beats VIX RSI Shorting VX and holding is profitable but ugly. Over 2007-2023, a naked short of one VX contract captured 213.52 points with a 68.86-point max drawdown, profit factor 1.13\. The question is whether an overbought/oversold filter improves it, and whether the filter should read the SPX or the VIX. Since VX trends down, a positive number below means VX fell (good for the short). ![Net VX points captured shorting, filtered on SPX RSI(2) versus VIX RSI(2), by bucket](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_483-spx_vs_vix_filter.png) Filtering on SPX RSI(2), an oversold SPX (RSI at or below 20) let the short capture 178.6 points with a max drawdown of only 21.04, a 62.9% win rate, and a recovery factor of 8.49\. Shorting VX whenever SPX RSI(2) was at or below 40 and sitting out otherwise captured more than the full 213.52 downside points while being in the market only 34% of the time. And an overbought SPX (RSI above 80) flipped the sign: VX rose 45.57 points, meaning short-term SPX strength is a dangerous time to be short vol and a candidate for going long VX. Now filter on VIX RSI(2) instead. The oversold-VIX short captured 80.23 points and the overbought-VIX short 90.81, both roughly half the SPX oversold result, and the muddle is obvious in the chart: the VIX filter's biggest bar sits in the overbought bucket, the opposite end from where the SPX filter pays. Reading the SPX gives you a clean, monotonic edge and a usable long-VX signal on the other tail. Reading the VIX gives you a weaker, scrambled one. $$ \\text{Profit Factor} = \\frac{\\text{gross gains}}{\\text{gross losses}}, \\qquad \\text{Recovery Factor} = \\frac{\\text{net profit}}{\\text{max drawdown}} $$ Both metrics grade reward against risk. Profit factor above 1 means the strategy made money; the SPX-oversold short scored 1.47 against the naked short's 1.13\. Recovery factor is net profit divided by the worst peak-to-trough loss, so the SPX-oversold short's 178.6 points over a 21.04-point drawdown gives 8.49, against 3.1 for the naked short. Higher on both means you earned more per unit of pain and dug a shallower hole to do it. _This post is for subscribers only._ ### 4.80 Cross-Venue Arb: Deribit Inverse Options × Polymarket Binaries URL: https://aligrithm.com/cross-venue-arb-deribit-inverse-options-x-polymarket-binaries/ Last updated: 2026-10-15T10:13:00.000Z Open two browser tabs. In one, a Deribit Bitcoin call expiring Friday at 08:00 UTC, priced and settled in BTC. In the other, a Polymarket contract asking "Will Bitcoin be above $90,000 on Friday?", priced in cents between zero and one dollar. One is a regulated-adjacent institutional derivative quoted by market makers running Black-76\. The other is a bet that pays a flat dollar, priced by whoever showed up to click yes or no. Nobody prices these two against each other, because they look nothing alike. That gap is the trade. Augustin Valery's paper works out the exact condition under which you can buy a vanilla option on one venue and a pile of binary contracts on the other, and walk to expiry knowing you cannot lose money. He tests it on Bitcoin and Ether over 580 days and reports 24 trades, zero losses, and about 20.7% average return per trade. Strong headline. The fine print is that the thing is barely ever available, and calling it "arbitrage" hides that it is a directional bet with a floor bolted on. Both halves of that sentence are true, and both matter. ## Two instruments that look nothing alike Start with the Deribit leg, because its payoff is weird before it is useful. A Deribit call is an inverse option: you pay for it in Bitcoin, and it settles in Bitcoin. The intrinsic value in coin terms is the notional times max of one minus strike over spot, or zero. $$ V\_C^{\\mathbb{B}}(K, T) = N^{\\mathbb{B}} \\cdot \\max\\!\\left( 1 - \\frac{K}{S\_T},\\, 0 \\right) $$ Read it in plain terms. N is how many coins of notional. K is the strike, S\_T is the spot at expiry. When the option finishes in the money, its coin-denominated payoff is one minus the ratio of strike to spot. The higher BTC goes, the smaller K over S\_T gets, so the payoff in coins keeps climbing but at a slowing rate. That decay is the "inverse" quirk: a call that pays in the same asset it bets on has a concave payoff when you measure it in coins. Work one number. Take one BTC of notional, strike $88,588, and suppose BTC prints $92,000 at expiry. The coin payoff is one minus 88,588 over 92,000, which is one minus 0.9629, so 0.0371 BTC. Convert that to dollars at the $92,000 expiry price and you get $3,412, which is exactly spot minus strike. The one-over-S\_T factor that bent the coin payoff cancels the moment you sell the coins for dollars. That is the whole reason the paper insists on converting the vanilla leg to dollars first: only in dollar terms does the inverse option flatten back into the ordinary piecewise-linear call payoff that lines up against a binary. ![Inverse vanilla call payoff in BTC (solid, concave) versus the same payoff in dollars (dashed, linear), plotted against the underlying at expiry](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_475-inverse_vanilla_payoff.png) The Polymarket leg is the opposite kind of animal. A binary pays a fixed one dollar if its outcome resolves true and nothing if it resolves false. Its value at expiry lives in the set of zero or one: $$ V\_C(K, T) = \\begin{cases} 1 & \\text{if } S\_T > K \\\\ 0 & \\text{otherwise} \\end{cases} $$ You pay a price P between zero and one for it, so your net outcome is either minus P if you are wrong or one minus P if you are right. A binary "yes" is a digital call, a binary "no" is a digital put, and the price is the crowd's implied probability. Buy the "no" at $0.15 and you are paying 15 cents to collect 85 cents if the price finishes below the strike. Flat payoff on each side of the strike, a cliff at the strike. Nothing concave about it. ## The trade: bolt a floor onto a call Here is the move. A vanilla call bleeds its premium whenever the underlying finishes below the strike. Its payoff is minus P for any S\_T below K, then negative-but-rising between K and K plus P, then positive above that. So the call loses money across a whole region of outcomes. The binary that points the other way pays off in exactly that region. Buy enough binary puts and their winnings cover the call's premium whenever the call dies. Do it right and the combined position never settles negative. The paper is careful about the word "enough," because a naive hedge breaks in the other direction. When BTC rallies past the binary's strike, the binary puts you bought all expire worthless, and now you are down their combined premium. You need the call's upside to cover that too. Two conditions, then: enough binaries to cover the call, and enough distance between the strikes that the call's gain covers the binaries when they lose. Miss either and the floor has a hole. ![Combined vanilla-binary payoff for the long (call) and short (put) portfolios, showing the individual legs and their sum against the underlying at expiry](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_475-combined_arb_payoff.png) The combined line is the point. Below the vanilla strike K\_V the two legs cancel to a flat zero, not a loss. Inside the interval between K\_V and K\_B both legs pay, giving a bump. Above K\_B the call runs on its own. The shape is an ordinary long-option payoff with its downside sawed off at zero. That is also the tell: this is a call in disguise, not a market-neutral arbitrage. You still need Bitcoin to move your way to make anything. _This post is for paying subscribers only._ ### 11.5 Prop Firm Math: Edge, Barriers, and Cash After the Fee URL: https://aligrithm.com/prop-firm-math-edge-barriers-and-cash-after-the-fee/ Last updated: 2026-08-12T09:15:37.000Z You buy a discretionary payout contract written on a simulated book. The fee buys an attempt to reach a profit target without piercing absorbing floors. Clear the gates and the firm may share a fraction of the simulated profit as cash, at its discretion. Traders who size and optimize for the challenge alone turn positive-looking strategies into negative-EV fee machines. The objective that runs through the book: **maximize discounted realized cash payouts after fees, trading costs, rule risk, and counterparty risk.** Pass probability, verification survival, funded lifetime, and growth sit inside that objective as intermediate metrics. Maximize any one of them alone and you can destroy total value. Strategy, sizing stack, literal rule schema, payout process, and counterparty form one system. A signal with post-cost edge still loses money if the path dies on a daily cliff or the firm never pays. ## Who this is for I assume you can read a barrier equation and a block bootstrap. You size from code or a spreadsheet. "Risk 1%" and "pass the challenge" fail as decision procedures under daily-loss cliffs, trailing floors, lot quantization, and a firm that moves second. Pip primers belong elsewhere. The book estimates cash from a validated edge, or shows before the next fee that the conversion fails. ## The chain Cash requires six links: 1. **Edge.** Post-cost, out-of-sample drift mu bounded away from zero. Lower bound straddles zero: stop. Risk geometry does not rescue a zero-EV strategy. 2. **Path.** Rules score path shape. At equal expectancy, lumpy paths breach; smoother paths survive. 3. **Sizing.** Size from the cushion above the active floor, well below full Kelly, as the minimum over every live cliff, rounded down to executable lots. 4. **Simulation.** No closed form covers the full rule set. Monte Carlo on the trade log under the literal rules returns survival-adjusted Net EV. 5. **Counterparty-adjusted EV.** Pass probability is simulated bookkeeping. Settlement fractions and payment hazard push the break-even above naive fee recovery. 6. **Cash.** Settled payout from a firm that still exists and still pays. Farm, churn, or hybrid follows from that hazard. Retail prop screens stop at headline returns. They leave links five and six unmodeled. ## What the book covers **Part I — The Contract.** Purchase object, single objective, farm/churn/hybrid fork from payment hazard, rules as executable constraints, running example: dated 10k FTMO 2-Step CFD account with barriers tight enough that mistakes show. **Part II — The Path to the Barrier.** First-passage math, path dependence of daily and trailing floors, five-layer sizing stack, four-state controller from challenge through post-payout farming. **Part III — The Return Process.** Edge gate (walk-forward, nested CV, block-bootstrap lower bound), path geometry the rules reward, portfolios on joint breach probability, execution as a term in realized edge. **Part IV — Simulation and Economics.** Rule-aware simulator, payout-EV chain, attempt portfolios, firm selection under adversarial counterparty risk. **Part V — Stress Test and Operation.** Synthetic end-to-end stress test (Strategy A): designed inputs, measured outputs, companion code. Live health engine for funded accounts after edge decay. Closing synthesis. **Appendices.** Frictionless academic core, notation, rule-schema detail, bibliography by source class. ## Reading order Parts I–II first. Chapter 9 is mandatory; everything downstream assumes mu is real. Part IV decides before you pay a fee. Part V is the rejection template. Strategy A is synthetic; swap in a real trade log. The decision the book forces: can this strategy, at this size, on this symbol, survive these rules long enough to reach a payout and get paid by this counterparty often enough across enough attempts to beat the full cost chain? A negative answer saves the fee. ![Reading order from Part I through the edge and Net EV gates to Part V.](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_489-introduction_reading_order.png) ## Download the full book (PDF) This post is the introduction only. The full pre-publication edition — contract framing, barrier math, sizing stack, edge gate, simulation, counterparty-adjusted Net EV, stress test, and appendices — is in the PDF. [**Download Prop Firm Math (PDF)**](https://aligrithm.com/content/files/2026/07/Prop-Firm-Math-1.pdf) Pre-publication edition, July 2026 (v0.9). Free to read; treat it as a working draft, not a finished textbook. ## KEY POINTS - A prop account is a discretionary payout contract on a simulated book, not brokerage capital. - Optimize discounted realized cash after fees, costs, rule risk, and counterparty risk — not pass rate alone. - Six links from edge to cash; retail screens usually stop before counterparty-adjusted EV. - If the post-cost lower bound on mu straddles zero, or Net EV after hazard is negative, skip the fee. - Full derivations, figures, worked FTMO 10k example, and appendices are in the PDF above. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ### 6.51 Momentum Is a Ranking Problem: Learning-to-Rank vs Regress-then-Rank URL: https://aligrithm.com/momentum-is-a-ranking-problem-learning-to-rank-vs-regress-then-rank/ Last updated: 2026-07-27T13:25:55.000Z Cross-sectional momentum has one job: at each rebalance, order a universe of assets from worst to best, buy the top, sell the bottom. Everyone agrees on that. Where strategies quietly disagree is on how they produce the order. Classic momentum sorts on the past twelve-month return. A neural net predicts each asset's next return and sorts on the prediction. Both treat the ranking as a byproduct of something else, a raw return or a point forecast. But if the only thing you ever use is the order, why not train the model to get the order right directly? That is the question this paper picks up, borrowing the answer from search engines. Information retrieval solved "put the most relevant thing on top" decades ago with learning-to-rank algorithms, and the paper drops one of them, LambdaMART, into a four-asset-class momentum backtest. The intellectual case is clean and it matches the old article ["Ranking Beats Forecasting for Many Trading Problems"](https://aligrithm.com/ranking-beats-forecasting-for-many-trading-problems/): you should optimize the objective you actually trade on. The empirical case is where you need to keep your hand on your wallet. The headline number, a 3.40 Sharpe in equities, is a method demonstration, not a strategy, and the paper's own weekly results are what give the game away. ## Four ways to build the same ordered list Fix the portfolio first so the only thing that changes is the ranker. Every strategy scores each asset, sorts the scores, longs the top three, shorts the bottom three, equal-weights each leg, and scales positions so each asset contributes the same risk. The portfolio return on the day is the average of those signed, vol-scaled asset returns. $$ r\_{t,t+1}^{\\text{CSM}} = \\frac{1}{n} \\sum\_{i=1}^{n} S\_t^{(i)} \\, \\frac{\\sigma^{\\text{tgt}}}{\\sigma\_t^{(i)}} \\, r\_{t,t+1}^{(i)}, \\qquad S\_t^{(i)} = \\begin{cases} +1 & \\text{rank} > N\_t - 3 \\\\ -1 & \\text{rank} \\le 3 \\\\ 0 & \\text{otherwise} \\end{cases} $$ In words: the signal S is plus one for the three top-ranked assets, minus one for the three bottom-ranked, and zero for everyone in the middle. The ratio of target volatility to the asset's own volatility scales each position so a calm asset gets a bigger notional than a wild one, which is the same inverse-vol sizing behind the old article ["Selection vs Diversification: Why |t|>3 Throws Away Alpha."](https://aligrithm.com/selection-vs-diversification-why-t-3-throws-away-alpha/) Target vol is 15% annualized; each asset's vol is a 63-day exponentially weighted estimate. Work one leg through it. Say the target is 15% annual (about 0.94% daily) and an equity index is running at 20% annual vol. Its scale factor is 15 divided by 20, so 0.75\. If it is a top-three long and gains 1% that day, it contributes 0.75 times 1%, so 0.75%, to the long leg before averaging across the six positions. A 30%-vol index in the same slot gets scaled to 0.50 and contributes less. The sizing is fixed. The ranking is the only decision. Now the four rankers. **JT** (Jegadeesh-Titman) scores each asset by its raw 252-day return and sorts. **Baz** scores by a blend of volatility-normalized MACD signals across three time scales, the CTA-style trend indicator. Both are hand-built formulas with no fitting. The two machine-learning rankers share the same 22 features per asset: the one-month raw return, volatility-normalized returns over 1, 3, 5, 10, and 21 days, and a stack of MACD indicators at different horizons. ## Regress-then-rank: predict the return, sort the predictions The multilayer perceptron is the honest strawman for "just forecast returns." It maps the feature vector to a number trained to match next day's volatility-scaled return under mean squared error, then you sort those numbers. $$ L(\\theta) = \\frac{1}{M} \\sum\_{\\Omega} \\left( Y\_t^{(i)} - \\frac{r\_{t,t+1}^{(i)}}{\\sigma\_t^{(i)}} \\right)^2 $$ Read it as: average, over every asset-day in the training set, the squared gap between the model's score Y and the realized vol-scaled return. Minimizing this makes Y a good point forecast of return. The ranking is whatever falls out of sorting those forecasts. Here is the leak. MSE spends its effort where the errors are largest in magnitude, which is the middle of the pack and the fat tails, not the top and bottom edges you trade. Suppose two assets both belong in your short basket. The truth says asset A returns minus 2% and asset B minus 3%. A model that predicts minus 2.5% for both has real squared error but ranks them correctly enough for a top-3/bottom-3 trade. A second model nails A at minus 2% but calls B plus 1%. It has lower error on A yet flips B out of the short leg entirely. MSE cannot tell you the second mistake is the expensive one, because it grades forecast accuracy, not order. That mismatch between the loss and the objective is the whole opening for learning-to-rank. ## Learn-to-rank: optimize the order you actually trade LambdaMART attacks the order directly. It is a gradient-boosted tree ensemble descended from RankNet and LambdaRank, and instead of minimizing forecast error it optimizes a ranking-quality metric, NDCG, by defining gradients ("lambdas") that push assets toward their correct list positions. $$ L(\\theta) = \\sum\_{t \\in \\mathcal{T}} \\ell\\big(y\_t^{(i)}, f(\\mathbf{x}\_t^{(i)}; \\theta)\\big)\_{i \\in \\mathcal{U}}, \\qquad \\text{aggregated}\_{\\text{ndcg}} = \\tfrac{1}{2}\\big(\\text{top}\_{3,\\text{ndcg}} + \\text{bottom}\_{3,\\text{ndcg}}\\big) $$ The left piece says the loss compares the true ordering y against the model's scores across the whole universe at each rebalance, not one asset at a time. The right piece is the scorecard: NDCG@3 measured on the top three, then again on the bottom three (by flipping the labels so "most negative" counts as "most relevant"), and averaged. NDCG rewards putting the right names in the right slots with more credit for getting the very top and very bottom right, using an exponential gain of two-to-the-relevance minus one. That exponential is also why the paper caps the universe near 31 assets, since larger relevance grades overflow a 32-bit integer in the ranking library. A worked NDCG intuition. Your short basket should hold the three worst assets. If your model ranks them 1st, 2nd, and 4th worst, NDCG@3 is high but not perfect because one true bottom-3 name slipped to slot four and an impostor took its place. If it ranks them 1st, 5th, and 9th, NDCG@3 drops hard, because the discount for lower positions is steep and you are missing two of the three names that carry the leg. The metric grades exactly the thing the trade cares about, which MSE does not. The training is walk-forward and disciplined: fit on a 5-year block, freeze the parameters, trade the next 5-year block, with a 90/10 train-validation split and early stopping. Data runs 2000 to 2023 across 16 commodities, 18 bond indices, 29 currencies, and 26 equity indices. ## The equity result, and why it is a demo In equity indices with daily rebalancing, the ranking framing looks decisive. LambdaMART posts a 3.40 Sharpe, the MLP 3.08, while sort-by-past-return (JT) sits at minus 1.88 and the CTA signal (Baz) at minus 0.25\. Random ranking gives minus 0.11\. The learners also cut max drawdown to minus 20% against JT's minus 100%, and lift the hit rate to 0.59. ![Bar chart of daily equity Sharpe ratios by strategy, showing LM at 3.40 and MLP at 3.08 far above the negative Sharpes of JT, Baz, and random](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_482-equity_daily_sharpe.png) Before you wire up a broker, read the paper's own footnotes, because the author does not hide them. The equity results carry no transaction costs. The index universe comes from a screen that keeps major, actively traded indices, which is survivorship bias baked into the sample. Shorting indices is expensive and sometimes impossible, and daily rebalancing multiplies that cost. The author says it plainly: weekly performance is the more realistic indicator, and daily rebalancing "could significantly underestimate the transaction costs involved in shorting indices." A 3.40 Sharpe that assumes free, frictionless daily shorting of a curated index list is a statement about the algorithm, not about a tradable P&L. The ranking-quality edge that drives it is real but tiny in level. NDCG@3 in equities is 0.556 for LambdaMART and 0.555 for the MLP, against 0.540 for random, and both machine learners clear significance at the 1% level while JT (0.542) and Baz (0.540) do not. So the learners genuinely order equities better than a past-return sort does. The leverage from that small ranking gain to a 3.40 Sharpe is the part that only survives under frictionless daily rebalancing. ![Bar chart of equity NDCG@3 scores by strategy, with MLP 0.555 and LM 0.556 just above random 0.540, JT, and Baz](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_482-ndcg_equity.png) ## Slow the rebalance down and the story deflates The cleanest test of whether the edge is tradable is to move to weekly rebalancing, where the author notes trading costs are negligible because only six positions turn over. If the daily Sharpe were a real signal rather than a rebalancing artifact, it should mostly survive. It does not. Weekly equity Sharpe falls to 0.54 for LambdaMART and 0.64 for the MLP. The learners still beat the traditional strategies, which stay negative, so the ordering advantage is genuine and it persists. But the headline number drops by roughly a factor of six. Whatever produced 3.40 lived in the daily rebalance, exactly where costs are worst and least accounted for. ![Grouped bar chart comparing daily versus weekly equity Sharpe for LM (3.40 to 0.54) and MLP (3.08 to 0.64), showing the sharp decline](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_482-daily_vs_weekly.png) The author is candid about the mechanism: some cross-sectional signal decays at longer horizons, and the weekly sample is also thinner, which starves the learners of training data (35,211 daily training samples versus 6,786 weekly for currencies). Both cut the same way. The daily number is inflated by frictionless assumptions and possibly by overfitting a data-rich daily panel. Take 0.54, not 3.40, as the honest reading of what an equity learning-to-rank momentum strategy delivers, and even that ignores the survivorship bias in the index list. ## Ranking structure is not universal The paper's most useful finding for a practitioner is not the equity headline. It is that "momentum is a ranking problem" is true in some markets and false in others, which is why you cannot bolt LambdaMART onto everything and call it done. | Asset class (daily) | Best ranker | LM Sharpe | MLP Sharpe | Best traditional | Read | | ------------------- | ----------- | --------- | ---------- | ---------------- | ------------------------------------------ | | Equity | LM | 3.40 | 3.08 | JT −1.88 | Ranking framing helps most, but demo-only | | Bonds | MLP | 1.12 | 1.98 | JT 1.34 | Regress-then-rank wins; order matters less | | Commodity | LM | 0.44 | 0.08 | JT −0.35 | LM the only consistent positive | | Currency | LM | 0.47 | 0.00 | Baz 0.37 | Barely edges a hand-built signal | In bonds the MLP's regress-then-rank approach beats LambdaMART, 1.98 to 1.12, and even simple JT momentum (1.34) beats the dedicated ranker. When you can forecast the level well, sorting the forecasts is fine and the ranking machinery adds nothing. In currencies the learner's 0.47 barely clears the hand-built Baz signal's 0.37, and weekly it gets worse: random ranking posts the best weekly currency Sharpe at 0.34 while LambdaMART manages 0.20\. The author also strips out a hidden driver here, showing that the strong currency results in the prior literature were largely the Turkish Lira's collapse; excluding it pulls the numbers back to earth. Two more cautions the paper surfaces. NDCG and Sharpe do not track each other cleanly, because in weekly currencies LambdaMART has the highest NDCG@3 yet no return edge, so better ranking on the metric does not guarantee better money. And random ranking scores NDCG@3 between 0.540 and 0.558 everywhere, which tells you the ranking task itself is hard and the achievable edge over a coin flip is thin. This is the same discipline the old article ["Cross-Sectional Percentile Rank Within a Universe"](https://aligrithm.com/cross-sectional-percentile-rank-within-a-universe/) insists on: a rank is only worth trading when the spread between top and bottom is real, not when the ordering is barely better than noise. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_482-visual-0.png) ## Where this lands The idea is right and worth internalizing: if you only trade the order, train on the order. Learning-to-rank beats sort-by-past-return in equities and commodities on the metric that matters, and it does so with the same features a regress-then-rank model uses, which isolates the objective as the source of the gain. That is a real methodological upgrade over the reflex to forecast returns and sort them, and it extends the case the old article ["Ranking Beats Forecasting for Many Trading Problems"](https://aligrithm.com/ranking-beats-forecasting-for-many-trading-problems/) makes at the single-instrument level to the cross-section. What is not right is the headline. A 3.40 equity Sharpe assembled from frictionless daily shorting of a survivorship-screened index list is a benchmark for the algorithm, not a live edge, and the author's own weekly test collapses it to 0.54\. Treat the daily equity number as a demo, treat the weekly number as the floor of what is real, and treat the cross-asset table as the actual lesson: ranking structure pays in equities and commodities, regress-then-rank is fine in bonds, and currencies barely beat noise. Match the tool to the market, and cost the trade before you believe the Sharpe. ## KEY POINTS - Cross-sectional momentum only uses the ordering of assets, so the sharpest framing is a ranking problem, not a return-forecasting problem. That is the paper's thesis and it matches ["Ranking Beats Forecasting for Many Trading Problems."](https://aligrithm.com/ranking-beats-forecasting-for-many-trading-problems/) - Regress-then-rank (an MLP under MSE) optimizes forecast accuracy, which spends effort in the middle and tails, not on the top-3/bottom-3 edges you actually trade. Learning-to-rank (LambdaMART) optimizes NDCG on those edges directly. - With the portfolio and 22 features held fixed, the only variable is the ranker, so performance gaps isolate the objective. - Daily equity Sharpe: LambdaMART 3.40, MLP 3.08, versus JT −1.88 and Baz −0.25\. The ranking-quality edge is real but tiny in level (NDCG@3 0.556 vs 0.540 random, significant at 1%). - The 3.40 is a method demo: no transaction costs, a survivorship-biased index universe, and expensive daily index shorting. Weekly rebalancing, where costs are negligible, drops it to 0.54. - Ranking structure is not universal. Bonds prefer regress-then-rank (MLP 1.98 > LM 1.12 > JT 1.34), currencies barely beat a hand-built signal, and weekly random ranking wins in currencies. - NDCG and Sharpe do not move together, and random ranking scores 0.540 to 0.558 everywhere, so the achievable ordering edge over noise is thin. Only trade a rank when the top-to-bottom spread is real. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Systematic Trading - Robert Carver (Amazon)](https://www.amazon.com/s?k=Systematic+Trading+Robert+Carver&ref=aligrithm.com) - [Trading Systems - Urban Jaekle Emilio Tomasini (Amazon)](https://www.amazon.com/s?k=Trading+Systems+Urban+Jaekle+Emilio+Tomasini&ref=aligrithm.com) - [Learning to Rank: Enhancing Momentum Strategies Across Asset Classes (ssrn-5255258)](https://papers.ssrn.com/sol3/papers.cfm?abstract%5Fid=5255258&ref=aligrithm.com) ### 6.50 Crypto TSMOM Lives in Volume-Weighted Returns URL: https://aligrithm.com/crypto-tsmom-lives-in-volume-weighted-returns/ Last updated: 2026-10-15T11:56:51.000Z A crypto momentum strategy that earns 0.94% a day at an annualized Sharpe of 2.17 should set off every alarm you own. Huang, Sangiorgi, and Urquhart report exactly that, buying coins that went up and shorting coins that went down, rebalanced daily across 3,192 cryptocurrencies from 2014 to 2023\. Before you wire money, look at one number they also report: run the identical momentum signal with equal weights instead of volume weights and it loses 1.19% a day. The signal does not carry the strategy. The weighting scheme does. That single contrast is the whole paper, and it is both the interesting finding and the reason to be careful. Time series momentum, in the old article ["Building a Trend Follower, Component by Component,"](https://aligrithm.com/building-a-trend-follower-component-by-component/) is the plainest edge in trading: assets that went up tend to keep going up, so you go long positive past returns and short negative ones. Moskowitz, Ooi, and Pedersen documented it across stocks, bonds, currencies, and commodities. In crypto the record was a mess, with some papers finding strong momentum and others finding none. This paper argues the disagreement comes from a preprocessing choice nobody scrutinized: how you weight the coins when you build the market return. ## Volume, not market cap, because market cap is a lie The standard crypto "market return" weights each coin by market capitalization, following Liu and Tsyvinski. The authors refuse to, and the reason is specific. A coin's market cap is supply times price, and supply is whatever the founders say it is. Mint more coins, burn coins to manufacture scarcity, and the cap moves without a single trade happening. They point to coins like Peercoin and Namecoin carrying market caps above 1 million dollars while trading under 10,000 dollars a day. Cap says "large," the tape says "nobody is here." Volume is harder to fake with a keystroke, so they weight by trading volume instead. $$ R\_{VW,t} = \\sum\_{i=1}^{N} \\frac{v\_{i,t}}{\\sum\_{j=1}^{N} v\_{j,t}} \\, r\_{i,t} $$ Read it as a weighted average: each coin i gets a weight equal to its dollar volume v that day divided by the total volume of every coin, then you sum those weighted returns. Worked example with three coins on a given day. Bitcoin trades 40 billion and returns +2%, Ethereum trades 15 billion and returns +1%, some alt trades 1 billion and returns +30%. Total volume is 56 billion. The volume-weighted return is (40/56)(2%) plus (15/56)(1%) plus (1/56)(30%), which is 1.43% plus 0.27% plus 0.54%, so about 2.24%. The alt's 30% pop barely registers because almost nobody traded it. Under equal weighting that same alt would contribute a full 10%, letting a coin nobody touched swing the index. The choice matters before any strategy runs. The volume-weighted market averages 0.62% a day over the sample, against 0.21% for cap-weighted, 0.33% for equal-weighted, and 0.11% for Bitcoin alone. The index you pick already decides most of your answer. ## The momentum is real, at least inside the index They test time series momentum first by regressing the next day's volume-weighted market return on its own past cumulative return over horizons from one day to two weeks. $$ R\_{VW,t} = \\alpha + b \\, R\_{VW,\\,t-1:t-h} + e\_t $$ The slope b answers whether the past h-day return predicts tomorrow. Positive and significant means momentum; the intercept alpha is the part unrelated to the past, and e is noise. The slope lands between 0.024 and 0.042 and is significant at the 1% level from the 3-day horizon out to two weeks, across the full sample and both subperiods. Worked reading: a b of 0.04 means that for every 1% of cumulative return over the past week, tomorrow's expected return rises by 0.04%. Small per unit, but past weekly moves are often tens of percent in crypto, so it adds up. The one-day and two-day horizons are insignificant, which fits the story that a single day is mostly noise. Sorting sharpens it. Group each day by the tercile of past return and look at the next day. The bottom tercile earns almost nothing the next day, the middle earns more, and the top tercile earns the most, a clean monotonic ladder that holds even when the tercile cutoffs are fixed using only 2014 to 2015 data and applied out of sample through 2023. ![Bar chart of next-day average return by past 7-day return tercile: Low 0.11 percent, Middle 0.46 percent, High 1.29 percent, with High minus Low of 1.17 percent](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_481-tercile_momentum.png) The High-minus-Low gap runs above 1% a day and the weekly version peaks near 2.9% in-sample and above 4% out-of-sample. Continuation is there. _This post is for paying subscribers only._ ### 10.18 Can Machines Learn Weak Signals? Ridge > Zero > Lasso URL: https://aligrithm.com/can-machines-learn-weak-signals-ridge-zero-lasso/ Last updated: 2026-09-28T08:05:41.000Z Feed 920 firm characteristics into a Lasso to predict next month's stock returns and it will do something that should stop you cold: it loses to a model that predicts zero for every stock. Not "underperforms a good benchmark." Loses to the number 0\. Shen and Xiu prove this is not bad luck or a coding bug. In the regime where economics and finance actually live, where signals are individually tiny and predictors are plentiful, Lasso cannot beat the zero estimator for any choice of its penalty, while Ridge can. The ranking is Ridge, then zero, then Lasso, and it holds in theory and across six real datasets. Start with why the regime matters. The authors hand-collected the R-squared from every regression in 411 economics papers and 380 finance papers published in 2022, more than 20,000 values. The 25% quantile sits at 9.7% for economics and 5.8% for finance, and that is after publication bias filters out the weakest results. Most published work in these fields runs on covariates with feeble explanatory power. ![Two histograms of reported R-squared values from five economics journals and three finance journals in 2022, both heavily concentrated near zero](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_449-weak_signals_everywhere.png) That concentration near zero is the whole setup. When no single predictor carries much signal, the useful question stops being "which model fits best" and becomes "can any model beat doing nothing at all." The old article "Does ML Actually Help Asset Pricing? Kelly's 20%, Not 2-3x" landed on a modest verdict for machine learning in the cross-section; this paper explains the mechanism underneath that modesty and tells you which regularizer to reach for when the signal is this thin. ## The zero estimator, and why it is the bar to clear A weak signal is a coefficient so small its estimation error swallows the bias it would remove. Formally the paper defines the weak regime by the total signal strength vanishing as the number of predictors grows. $$ \\| \\beta\_0 \\|\_2^2 \\asymp\_P \\tau \\to 0, \\qquad p \\to \\infty $$ Read it plainly: beta-zero is the vector of true coefficients, and the squared length of that vector, its total signal energy, shrinks toward zero (call that shrinking budget tau) while the number of predictors p grows without bound. The population R-squared is proportional to tau, so tau near zero means an R-squared near zero. Worked example: if tau delivers a population R-squared of 1%, then even the true coefficients explain one part in a hundred of return variance. There is almost nothing to learn, and the question is whether a method can extract even that. The natural benchmark in this world is the zero estimator, which ignores every predictor and forecasts zero. It sounds like a strawman. It is not. Shen and Xiu prove that when signals are this weak, the zero estimator is asymptotically Bayes-optimal: its prediction risk matches the best possible predictor that knows the true prior distribution of the coefficients. $$ \\lim\_{n \\to \\infty} \\frac{\\mathcal{R}(\\hat{\\beta}, F)}{\\mathcal{R}(F)} = 1 \\quad\\text{for } \\hat{\\beta} = 0 $$ The numerator is the risk of the estimator you built; the denominator is the risk of the infeasible Bayes predictor. When that ratio goes to 1, your estimator is doing as well as anything possibly could. The theorem says the do-nothing forecast hits that mark. So beating zero is not a low bar you clear by accident. Beating zero means you genuinely pulled signal out of the noise, because zero is already optimal to first order. Any estimator that fails to beat it has learned nothing. ## The test that magnifies the difference Because Ridge, Lasso, and zero all hit the same first-order optimal risk, comparing their risk ratios tells you nothing; they all look like 1\. To separate them you need a microscope. The paper defines a relative-error metric that blows up the higher-order gap and carries a sign. $$ \\Delta(\\hat{\\beta}) = p\\, n^{-1} \\tau^{-2} \\left( \\| \\Sigma\_2^{1/2}(\\hat{\\beta} - \\beta\_0) \\|^2 - \\| \\Sigma\_2^{1/2} \\beta\_0 \\|^2 \\right) $$ The bracket is the estimator's prediction error minus the zero estimator's prediction error (the zero estimator's error is just the second norm, since it sets beta-hat to zero). Multiply by p over n times tau-squared-inverse to rescale the tiny difference into something that converges to a finite number. The sign is what matters: if Delta is negative with probability approaching one, your estimator beats zero; if positive, it loses to zero. Worked reading: Delta equal to -1 means you edged out the do-nothing forecast; Delta equal to +2 means you would have been better off forecasting zero and going home. ## Ridge dips below the line; Lasso never does Run Ridge through this metric and the payoff is a clean closed form. With the tuning parameter written as lambda over tau, the relative error converges to a value the paper calls alpha-star. $$ \\Delta(\\hat{\\beta}\_r) \\xrightarrow{P} \\alpha^\* = 2\\theta\_2 \\sigma\_x^4 \\left( \\frac{\\sigma\_\\varepsilon^2 \\theta\_1}{2\\lambda^2} - \\frac{\\sigma\_\\beta^2}{\\lambda} \\right) $$ The term in parentheses is a parabola in one-over-lambda: a positive piece that dies like lambda-squared and a negative piece that dies like lambda. Set the noise, signal, and design constants all to one and it collapses to alpha-star equal to one-over-lambda-squared minus two-over-lambda. Plug in numbers. At lambda equal to 1, alpha-star is 1 minus 2, so -1: Ridge beats zero by the most. At lambda equal to 2, it is 0.25 minus 1, so -0.75, still negative. As lambda grows the curve creeps back up toward zero from below, meaning Ridge with a huge penalty just becomes the zero estimator and ties it. Push lambda toward zero and alpha-star runs to positive infinity: the Ridgeless (no-penalty) interpolator is catastrophically worse than doing nothing, because with weak signals its implicit regularization cannot control variance and the fitted coefficients balloon far past the true ones. ![Plot of the Ridge relative-error curve alpha-star against the tuning parameter lambda, dipping to a minimum of minus one near lambda equal to one and rising back toward zero, with the region above zero shaded to mark where Lasso is stuck](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_449-ridge_delta_curve.png) Now Lasso. The paper cannot get a closed form because Lasso has no closed-form solution, so it derives probability bounds instead, and the bounds are the punchline: both the lower and upper bounds on Lasso's relative error are non-negative, for every choice of the penalty. Lasso lives in the shaded region above the zero line in the chart and cannot cross it. Crank its penalty up and it converges down to zero (it becomes the zero estimator); crank it down and it diverges to positive infinity (worse than zero). There is no setting where it dips negative. The reason is structural. Lasso is a selection method that thrives when there is a crisp line between real and fake predictors. Weak signals erase that line, so Lasso's willingness to zero out coefficients throws away exactly the diffuse, collective signal that Ridge's shrinkage keeps. The elastic net inherits the problem to the extent it leans on the l1 penalty. One practical rescue: cross-validation still works for Ridge here. Even though the optimal penalty diverges as the sample grows (because tau is unknown and shrinking), the K-fold cross-validated choice converges to the noise-to-signal ratio and lands you on the useful part of the curve without knowing tau in advance. ## Six datasets say the same thing Theory is cheap. Shen and Xiu run seven methods across six datasets from finance, macro, and micro. The pattern from Table 1 is stubborn. ![Grouped bar chart of out-of-sample R-squared for Ridge versus Lasso across five datasets, with Ridge positive in all five and Lasso negative in four](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_449-empirical_ridge_beats_lasso.png) Predicting the equity premium from the Welch-Goyal indicators (16 predictors, annual data since 1948), Ridge posts an out-of-sample R-squared of 0.80% while Lasso posts -12.19%, and plain OLS, with barely more observations than predictors, craters to -81.08%. On the Gu-Kelly-Xiu cross-section of individual stock returns, extended with 920 characteristics, the l2-penalized neural net leads at 0.26% and Ridge follows at 0.19%, both beating the l1 neural net at 0.14% and Lasso at 0.10%. Turn those thin R-squared numbers into a long-short decile portfolio and the l2 neural net earns a Sharpe of 2.13 and Ridge 1.64, while the sparse GBRT trails at 0.80\. Cross-country GDP growth: Ridge 6.58%, Lasso -14.58%. Crime rates: Ridge 0.48%, Lasso -1.01%. Only one of the five low-signal cases has Lasso clearing zero at all. The pairing extends past linear models. Random Forest, which builds dense trees using nearly all variables, beats Gradient Boosting, which builds sparse sequential trees, in every weak-signal dataset. Neural nets with an l2 penalty beat the same nets with an l1 penalty. Across the board, the denser method wins. Look at the variable-importance plots and the winners spread small weights across almost every covariate; the losers concentrate on a handful. That is the collective-signal story made visible. ## Signal weakness, not the absence of sparsity Here is where the paper picks a fight with a famous result. Giannone, Lenza, and Primiceri argued that sparsity is an "illusion" in economics: the best predictive models keep lots of covariates, so the world is not really sparse. Shen and Xiu accept the empirics and reinterpret the cause. Ridge can beat Lasso even when the true model is genuinely sparse, as long as the non-zero coefficients are weak. So Lasso's failure does not prove the world is dense. It proves the signals are weak. Watching Lasso underperform tells you about signal strength, not about the number of true predictors. This reframes the old article ["The Factor Zoo and the Multiple-Testing Reckoning"](https://aligrithm.com/the-factor-zoo-and-the-multiple-testing-reckoning/) from the shrinkage side. That piece, through the Kozak-Nagel-Santosh lens, argued the cross-section is better described by a dense stochastic discount factor than by a few clean anomalies, and that shrinking many weak factors beats selecting a few strong ones. This paper supplies the estimation-theory reason: when factors are individually weak, an l2 shrinker that keeps all of them is asymptotically the right tool, and an l1 selector that discards most of them is asymptotically the wrong one. The debate about whether alpha is sparse or dense partly dissolves into a debate about whether the signals are strong or weak, and in asset pricing they are weak. ## What survives when signals are weak The proposition on out-of-sample R-squared drives home how thin the payoff is even when you win. $$ R^2\_{oos}\\big(\\hat{\\beta}\_r(\\lambda\_n^{opt})\\big) = \\frac{n}{p}\\, \\theta\_2 \\, (R^2)^2 \\,\\big(1 + o\_P(1)\\big) $$ The optimal Ridge's out-of-sample R-squared is proportional to the square of the population R-squared, scaled by the sample-to-predictor ratio n over p. Square a small number and it gets much smaller. Worked example: population R-squared of 5% (0.05), with n over p times theta-2 near one, gives an out-of-sample R-squared around 0.05 squared, so about 0.25%. The signal is real, Ridge captures it, and the visible out-of-sample R-squared is still a quarter of a percent. That is not a bug in your pipeline. It is what learning a weak signal looks like, and it is exactly why the Sharpe ratios above come from R-squared values you could mistake for zero. The trap the old article ["Does Complexity Actually Help? The Virtue-of-Complexity Autopsy"](https://aligrithm.ghost.io/does-complexity-actually-help-the-virtue-of-complexity-autopsy/?ref=aligrithm.com) warned about sits right next to this result. Complexity and flexibility are not free in a low-signal world: the Ridgeless interpolator, the most complex linear fit, is the one that blows up worst here, and the shallow neural nets can lose to plain Ridge because their extra parameters chase noise. Flexible models help only after you have shrunk hard, and shrinkage with an l2 penalty is doing most of the work. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_449-visual-0.png) ## Where this connects The spine of this pillar is the equation from the old article "What a Factor Actually Is: alpha + beta-lambda, and Why the Market Is Factor #1," where the cross-section of expected returns is a set of weak, noisy loadings on priced factors. This paper is the estimation manual for that world. When the true premia are small and the characteristics are many, ridge-style shrinkage of the whole coefficient vector beats lasso-style selection, boosting-style sparse trees, and unregularized fits, and the honest out-of-sample R-squared you should expect is the square of an already small number. The practical takeaways are blunt. Benchmark against the zero estimator, not against a flattering baseline, because zero is the optimal do-nothing forecast when signals are weak. Prefer l2 penalties (Ridge, l2 neural nets, Random Forest) over l1 penalties (Lasso, GBRT) in these regimes. Read a low out-of-sample R-squared as evidence of weak signal, not proof of a sparse world. And treat Lasso's popularity as a modern OLS with suspicion: in economics and finance, where the R-squared histograms pile up near zero, it is the wrong default. ## KEY POINTS - In the high-dimensional, low-R-squared regime that describes most of economics and finance (25% quantile R-squared of 9.7% and 5.8%), the zero estimator is asymptotically Bayes-optimal. Beating it means you genuinely learned a weak signal. - Ridge can beat the zero estimator with the right l2 penalty; its relative error dips negative, bottoming out at the optimal tuning value. Lasso cannot beat zero for any l1 penalty, because its relative-error bounds are provably non-negative. - The Ridgeless (no-penalty) interpolator is worse than doing nothing in this regime; its implicit regularization fails to control variance and the fitted coefficients balloon. - The ranking generalizes: dense Random Forest beats sparse Gradient Boosting, and l2 neural nets beat l1 neural nets, in every weak-signal dataset. Denser methods win because weak signal is spread across many predictors. - Six real datasets confirm it. Equity premium: Ridge 0.80% versus Lasso -12.19%. Cross-section of returns: l2 neural net Sharpe 2.13, Ridge 1.64, sparse GBRT 0.80\. Ridge clears zero in all five low-signal cases; Lasso clears it in one. - Lasso failing does not prove the model is dense. Ridge beats Lasso even under true sparsity if the non-zero coefficients are weak. The binding constraint is signal weakness, not the absence of sparsity. - Even when Ridge wins, out-of-sample R-squared is proportional to the square of the population R-squared, so a 5% population R-squared shows up as roughly 0.25% out of sample. Cross-validation still selects a valid Ridge penalty without knowing the signal strength. ## References - [Economic Predictions With Big Data: The Illusion of Sparsity (Giannone, Lenza, and Primiceri, 2021)](https://onlinelibrary.wiley.com/doi/10.3982/ECTA17842?ref=aligrithm.com) - [Empirical Asset Pricing via Machine Learning (Gu, Kelly, and Xiu, 2020)](https://academic.oup.com/rfs/article/33/5/2223/5758276?ref=aligrithm.com) - [Ridge Regression: Biased Estimation for Nonorthogonal Problems (Hoerl and Kennard, 1970)](https://www.tandfonline.com/doi/abs/10.1080/00401706.1970.10488634?ref=aligrithm.com) - [Regression Shrinkage and Selection via the Lasso (Tibshirani, 1996)](https://rss.onlinelibrary.wiley.com/doi/10.1111/j.2517-6161.1996.tb02080.x?ref=aligrithm.com) - [Regularization and Variable Selection via the Elastic Net (Zou and Hastie, 2005)](https://rss.onlinelibrary.wiley.com/doi/10.1111/j.1467-9868.2005.00503.x?ref=aligrithm.com) - [The Lasso Risk for Gaussian Matrices (Bayati and Montanari, 2012)](https://ieeexplore.ieee.org/document/6151144?ref=aligrithm.com) - [A Comprehensive Review of Prediction of Stock Returns via the Equity Premium (Welch and Goyal, 2008)](https://academic.oup.com/rfs/article/21/4/1455/1565737?ref=aligrithm.com) - [Shrinking the Cross-Section (Kozak, Nagel, and Santosh, 2020)](https://www.sciencedirect.com/science/article/abs/pii/S0304405X19301655?ref=aligrithm.com) - [Can Machines Learn Weak Signals? (Shen and Xiu, 2024)](https://bfi.uchicago.edu/working-paper/can-machines-learn-weak-signals/?ref=aligrithm.com) --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ### 10.17 Getting the Target Right: The Transform That Beats 147 Features URL: https://aligrithm.com/getting-the-target-right-the-transform-that-beats-147-features/ Last updated: 2026-09-28T08:05:40.000Z Most machine-learning asset-pricing papers spend their budget on the same three things: more characteristics, a fancier model, a longer training window. Gu, Kelly, and Xiu fed 94 signals into a neural net. Jensen, Kelly, and Pedersen assembled 153 characteristics into a global replication dataset. The arms race is on the input side. Cakici and Zaremba took that whole machine, froze the 147 characteristics and the eight-model ensemble, and changed one thing almost nobody touches: how the *return you are predicting* is written down. That single knob moved more alpha than any feature choice they tried. The setup is boring on purpose. A panel of 80,148 firms across 35 developed and emerging markets, monthly, 1994 to 2024, out-of-sample from 2003\. The 147 firm characteristics come straight from the Jensen-Kelly-Pedersen catalog. The forecast is an equal-weighted average of eight standard models: OLS, ridge, LASSO, elastic net, PLS, random forest, gradient boosting, and a shallow neural net. Nothing here is new. What they vary is a grid of six transformations of the target return crossed with five transformations of the input features, thirty combinations in total, run identically through the same pipeline so the only thing that can explain a performance difference is the transformation itself. The result is a hierarchy. Get the target right and the features barely matter. Get the features right while leaving the target raw and you are optimizing the wrong margin. This is the cross-sectional cousin of the point the old article ["Selection vs Diversification: Why |t|>3 Throws Away Alpha"](https://aligrithm.com/selection-vs-diversification-why-t-3-throws-away-alpha/) made about significance thresholds: the decision that looks like plumbing is the one that decides your P&L. ## The asymmetry nobody questioned Look at how the field actually preprocesses data and you find a lopsided habit. Features almost always get transformed. The standard move is to rank-map each characteristic into an interval like \[-1, 1\] so that book-to-market and momentum sit on the same scale and no single variable dominates the fit for mechanical reasons. The target, though, usually enters raw, as the plain one-month-ahead excess return. Gu, Kelly, and Xiu do it. Most of the papers that followed do it. Raw returns are a bad target for a cross-sectional model, and the reason is the market. In a down month almost every stock is red; in an up month almost every stock is green. A model trained on raw returns learns to associate predictors with those market-wide swings, so it ends up encoding "returns tend to be high or low in months that look like this" instead of "stock A beats stock B this month." That is a time-series lesson bolted onto a cross-sectional question. The objective and the target disagree, and the model splits the difference by learning something you did not ask for. _This post is for paying subscribers only._ ### 6.49 Percentile-Rank Momentum With Hysteresis: Low-Churn Signals URL: https://aligrithm.com/percentile-rank-momentum-with-hysteresis-low-churn-signals/ Last updated: 2026-07-27T13:25:54.000Z Momentum is the oldest anomaly in the book, and a new momentum paper has to justify why it exists. Landolfi's percentile-rank framework does not sell you the momentum. It sells the plumbing around it: rank each move against its own sign-consistent history instead of a raw threshold, gate entries and exits with a hysteresis band so the signal stops flip-flopping, and validate with a grid of walk-forward splits instead of one arbitrary calibration. The strategy underneath is long/short trend on BTC and ETH. The engineering is the contribution. Two of the three ideas are worth stealing whatever you think of the evidence, and the evidence deserves suspicion. This is a company whitepaper on crypto, tested across 2018 to 2025, the easiest trending market of the decade, with no benchmark and no reference list. The old article ["Ranking Beats Forecasting for Many Trading Problems"](https://aligrithm.com/ranking-beats-forecasting-for-many-trading-problems/) argued that you should order things rather than predict their magnitude, because magnitude is where forecasts die. This paper takes that instinct and points it at a single instrument's own past. That move is clever and it is also where the first question mark sits. ## Rank the move against its own past, by sign Start with the raw material. Momentum is measured over several lookback horizons, and each horizon spits out a lagged return. Returns from a calm week and a violent week are not comparable, so the framework divides each by a recent volatility estimate before doing anything else. $$ \\tilde{r}\_n\[t\] = \\frac{r\_n\[t\]}{\\sigma\_{t-1}} $$ The normalized return r-tilde over horizon n is the raw n-bar return divided by yesterday's volatility estimate, sigma at t minus 1, usually an exponentially weighted standard deviation. The point is comparability. A 6% move means something different in a 2% world than in a 5% world. Work it. A 20-bar return of 6% in a regime where weekly volatility sits at 4% gives a normalized reading of 0.06 divided by 0.04, or 1.5\. The same 6% move when volatility has fallen to 2% reads 3.0, twice as extreme. Without the division you would treat two very different events as identical. With it, the signal speaks in standard deviations. Then the actual novelty. Instead of scoring that normalized move against one fixed cutoff, the framework ranks it as a percentile against its own history, and it splits the history by sign. Up-moves are ranked only against past up-moves; down-moves are ranked only against past down-moves in absolute terms. $$ \\operatorname{rank}^{(+)}\_{n}\[t\] = \\frac{\\#\\{\\, r\_n\[\\tau\] > 0 \\ \\text{and}\\ r\_n\[\\tau\] \\le r\_n\[t\] \\,\\}}{\\#\\{\\, r\_n\[\\tau\] > 0 \\,\\}} \\in \[0, 1\] $$ The long-side rank counts how many past positive n-bar moves were smaller than or equal to today's move, divided by the total count of past positive moves. It answers one question: among all the up-moves this market has ever made, how big is this one. The short side does the same with the magnitude of negative moves against past negative moves. Work it. Today's 20-bar move is a positive 6%. Look back and find 200 prior positive 20-bar moves, of which 170 were 6% or smaller. The long rank is 170 divided by 200, or 0.85\. Today sits in the 85th percentile of this market's up-moves. Rank the same move against all history, up and down mixed, and you dilute it with a crowd of down-moves it has nothing to do with. Splitting by sign keeps a strong rally judged against rallies. This is where the paper diverges from the old article ["Cross-Sectional Percentile Rank Within a Universe."](https://aligrithm.com/cross-sectional-percentile-rank-within-a-universe/) That method ranks one market against its peers at a single moment, a snapshot across instruments. This one ranks one market against its own past, a filmstrip through time. Both output a clean 0-to-1 number, and both dodge the scaling problems of raw indicators, but they answer different questions. Peer ranking asks who is strongest now. Self ranking asks whether this is a big move for me. Self ranking carries a hidden cost the cross-sectional version avoids: your reference distribution is one asset's history, so a structural shift in that asset quietly poisons every rank that follows. ## The composite score Each horizon produces its own rank. The framework blends them into one number per side with a convex weighting, weights that are non-negative and sum to one. $$ S\_{\\text{long}}\[t\] = \\sum\_i w\_i \\operatorname{rank}^{(+)}\_{n\_i}\[t\], \\qquad w\_i \\ge 0, \\ \\sum\_i w\_i = 1 $$ The long score S-long is a weighted average of the long-side ranks across horizons. Because the weights sum to one and each rank lives in 0 to 1, the composite stays in 0 to 1 too. The paper's heuristic is to weight shorter horizons more for responsiveness and taper the weight as horizons lengthen, so a fast move can move the score while the slow horizons anchor it. Work it. Use two horizons, a short one weighted 0.6 and a long one weighted 0.4\. If the short-horizon long rank is 0.85 and the long-horizon long rank is 0.70, the composite is 0.6 times 0.85 plus 0.4 times 0.70, which is 0.51 plus 0.28, or 0.79\. One number, bounded, that you can hang a decision rule on. ## Hysteresis is the actual product The title of this piece is the part that earns its keep. A single threshold trades badly. Put an entry cutoff at 0.85 and the moment the score wobbles between 0.84 and 0.86 you sell, buy, sell, buy, paying the spread every time. The fix is two thresholds with a gap between them. $$ 0 < q\_x < q\_e < 1 $$ You enter when the composite score rises above the high entry quantile q-e, and you do not exit until it falls below the lower exit quantile q-x, or a maximum holding time forces you out. The gap between q-x and q-e is the hysteresis. The score has to fall a long way, not wobble a little, before the position closes. Engineers call this a Schmitt trigger; it is the same reason a thermostat does not switch the furnace on and off every thirty seconds. Work it. Set entry at 0.85 and exit at 0.55\. The score climbs to 0.87 and you go long. It then drifts down through 0.80, 0.72, 0.60, and you hold through all of it because none breached 0.55\. Only when it finally hits 0.50 do you exit. A single 0.85 line would have thrown you out at 0.80 and back in at any bounce, several round-trips over the same path. The band traded twice. ![A noisy composite score crossing a single 0.85 threshold versus a 0.55-to-0.85 hysteresis band. The single line flips 28 times over the same path; the band enters once on the way up and exits once on the way down.](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_428-hysteresis_churn.png) The chart is the whole argument for why this survives costs. Over one simulated hump the naive threshold flips 28 times, and each flip is a round-trip spread. The band flips twice. The paper's own defense of its results leans entirely on this: average trade size lands in the 30 to 40 basis point range, so even a fat cost model does not erase the profit. That claim only holds because the hysteresis keeps trades few and large. Strip the band out, keep everything else, and the same signal would churn its edge into the spread. Low churn is not a nice property here. It is the load-bearing wall. ## The walk-forward bundle The last idea attacks a quieter problem: how you validate. Classic walk-forward analysis picks one in-sample length and one out-of-sample step, re-fits on each in-sample window, and tests on the next out-of-sample block. The catch is that both numbers are arbitrary. A 12-month window with monthly re-calibration and a 60-month window with semiannual re-calibration can tell opposite stories on the same data. The bundle runs the whole grid instead of picking one. In-sample lengths from 12 to 60 months, out-of-sample steps of 1, 3, and 6 months, both anchored and rolling schemes, first out-of-sample block starting January 2019\. Every out-of-sample segment from every configuration gets pooled into one distribution of equity curves. A tight cluster of curves says the result does not depend on the split you happened to choose; a fan of diverging curves says it does. ![Walk-forward bundle out-of-sample equity for BTCUSDT (top) and ETHUSDT (bottom). Each line is a different in-sample length, step, and anchoring choice; the tight grouping is the paper's robustness claim.](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_428-walk_forward_bundle.png) The bundles are tight. BTC lands around 290 to 350 percent cumulative across the grid, ETH around 500 to 620 percent, and the spread between the most and least aggressive calibration schedules is narrow. The cross-sectional Sharpe distribution centers well above one, roughly 1.4 to 1.8 depending on the asset. Read at face value, that is a strategy whose performance does not hinge on a lucky calibration choice. Read skeptically, the bundle proves less than it looks. It shows insensitivity to the split schedule, not to the market. Every one of those curves rides the same single history: the same 2020-2021 crypto boom, the same 2022 drawdown, the same recovery. This is the exact trap the old article ["Collinearity in Parameter Sweeps: Plateaus, Not Peaks"](https://aligrithm.com/collinearity-in-parameter-sweeps-plateaus-not-peaks/) named. A sweep can hold up beautifully while testing one line through the space and proving nothing, because the flat result is the parameters moving together, not the strategy surviving genuinely new conditions. A compact walk-forward bundle on one asset's one timeline is a more honest robustness check than a single split, and it is still one timeline wearing a robustness costume. ## What the evidence doesn't show The framework ships without the numbers that would let you trust it. There is no benchmark. BTC ran from a few thousand dollars to six figures over the test window, so buy-and-hold returned thousands of percent, and a long/short program that made 300 percent might have trailed the asset it trades by a wide margin. The paper never puts the two side by side, and for a market-neutral book that comparison is the first thing an allocator asks. There is no drawdown series, no realized turnover count, no cost model actually applied, only the assertion that 30 to 40 basis point trades make costs a rounding error. It is a whitepaper with no reference list and no peer review, on the single most trend-friendly asset class of the era. Keep what transfers and drop the marketing. The sign-asymmetric percentile rank is a clean way to make a raw move comparable to its own kind. The hysteresis band is a genuine turnover weapon that belongs on any threshold system you run. The walk-forward bundle is a better habit than the single split everyone defaults to. None of that depends on believing the crypto equity curves. The ideas are portable; the backtest is one asset class in one bull market, and you should treat the Sharpe of 1.5 as a hypothesis, not a result. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_428-visual-0.png) ## KEY POINTS - The contribution is the machinery around momentum, not the momentum: sign-asymmetric percentile ranking, a hysteresis entry/exit band, and a walk-forward bundle for validation. - Normalize each lookback return by recent volatility before ranking, so a move is read in standard deviations and stays comparable across calm and violent regimes. - Rank up-moves only against past up-moves and down-moves only against past down-moves. This keeps a rally judged against rallies, but ties every rank to one asset's own history, which a structural shift quietly corrupts. - Blend horizon ranks with convex weights that sum to one, weighting short horizons more for responsiveness. The composite stays bounded in 0 to 1. - The hysteresis band is the load-bearing idea. Enter above a high quantile, exit only below a lower one or at a hold cap. In a simple simulation a single threshold flipped 28 times where the band flipped twice. The 30-to-40 basis point average trade only survives costs because churn stays low. - The walk-forward bundle pools out-of-sample results across a grid of in-sample lengths, steps, and anchoring schemes, replacing one arbitrary split with a distribution. Tight bundles mean insensitivity to the split, not to the market. - Treat the crypto results with suspicion: no benchmark against a market that rose thousands of percent, no drawdown or turnover disclosed, costs asserted not applied, a whitepaper with no references on the decade's most trend-friendly assets. Keep the ranking and the band; doubt the Sharpe of 1.5. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Systematic Trading - Robert Carver (Amazon)](https://www.amazon.com/s?k=Systematic+Trading+Robert+Carver&ref=aligrithm.com) - [Trading Systems - Urban Jaekle Emilio Tomasini (Amazon)](https://www.amazon.com/s?k=Trading+Systems+Urban+Jaekle+Emilio+Tomasini&ref=aligrithm.com) - [A Percentile-Rank Momentum Framework (percentile rank momentum framework)](https://scholar.google.com/scholar?q=A+Percentile-Rank+Momentum+Framework&ref=aligrithm.com) ### 9.40 Do Polymarket Prices Converge? Bias and Volatility Evidence URL: https://aligrithm.com/do-polymarket-prices-converge-bias-and-volatility-evidence/ Last updated: 2026-07-27T13:26:24.000Z Everyone repeats the same slogan: prediction market prices are probabilities, and the crowd is wise. A 2024 master's thesis out of Charles University actually checked, on 187 Polymarket markets that resolved between December 2023 and April 2024, and the answer is more interesting than the slogan. Prices do converge to the truth as resolution nears, which is the good news. But the ride is far more violent than stocks or crypto, and the prices carry two measurable, exploitable biases: they overprice longshots and they lean toward "yes." Three findings, one reassuring and two that should keep you honest. The old article ["What a Market Price Actually Is: Capital-Weighted Consensus and the Brier Skill Score Gate"](https://aligrithm.com/what-a-market-price-actually-is-capital-weighted-consensus-and-the-brier-skill-score-gate/) argued that a price is only a probability if it survives a calibration test. This thesis runs that test on real Polymarket data and finds the price passing on convergence while failing on calibration in a specific, patterned way. That gap is where an edge lives, and also where you can fool yourself, so read the numbers before you read the conclusion. ## The prices do converge, and you can clock the speed Start with the reassuring part. Take a snapshot of a market's probability 90, 30, and 5 days before it resolves, and ask whether the probability assigned to the eventual winner climbs as the clock runs down. It does, and the movement is large. The average jump in the probability of the correct outcome from 90 days out to 30 days out is 12.6 percentage points, with a standard deviation of 21.3 points. From 30 days to 5 days it climbs another 7.9 points, standard deviation 20 points. Those gaps are not noise. A one-sample t-test on the 90-to-30 move returns a t-statistic of 8.01; the 30-to-5 move returns 5.36\. Both p-values sit far below 0.0001, so the market genuinely sharpens as information arrives and the window for surprises shrinks. The author also fit a crude daily linear model: $$ \\widehat{p}\_{\\text{correct}} = 0.860 + 0.003 \\times \\text{day} $$ Read it plainly. The probability the market assigns to the eventual outcome starts around 0.860 and creeps up by roughly 0.003, three tenths of a percentage point, for each additional day closer to resolution. Work a number: move 40 days closer and the model adds 0.003 times 40, about 0.12, or 12 percentage points of extra confidence in the right answer. The slope's t-statistic is 40, which looks decisive, but the author is refreshingly blunt that daily probabilities are autocorrelated, so the regression's standard errors are understated and this line "serves only to point out the relationship." Trust the t-tests on the 90/30/5 snapshots, not the pretty slope. ![Mean predicted probability of the outcome rising from 90 to 0 days before resolution, split by market category](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_417-convergence_by_category.png) Split by category and the convergence holds everywhere, but the starting point differs. Ninety days out, political markets already sit at 68.1% confidence in the eventual outcome against 59.3% for geopolitical ones, and that gap is the only category difference that clears significance at 90 days (geopolitical versus political, p = 0.04). By five days out, every category has climbed into the mid-to-high 80s and no pairwise comparison is significant anymore. The lesson for a trader is that the informational edge across categories exists early and evaporates late. If you think you read geopolitics better than the crowd, the time to express that is three months out, not the week of resolution. ## The ride is brutal: volatility Now the part the slogan never mentions. A prediction market token is not a gentle probability drifting to its answer. Measured as the standard deviation of daily returns, Polymarket is wildly more volatile than either stocks or crypto. $$ \\sigma = \\sqrt{\\frac{1}{N-1}\\sum\_{t=1}^{N}\\left(r\_t - \\bar{r}\\right)^2} $$ That is just the standard deviation of daily returns r-t around their mean r-bar. Work a tiny example: a token posting daily returns of +20%, minus 10%, +15%, and minus 25% has a mean of zero and a standard deviation of about 0.19, meaning a typical day moves the price roughly 19%. A blue-chip stock's daily standard deviation sits near 1%. That is the gap the thesis measures. ![Daily-return volatility of Polymarket tokens versus stocks and crypto, with Welch t-test results](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_417-volatility_comparison.png) The comparison uses Welch's t-test, chosen because the variances are nowhere near equal. Polymarket versus stocks returns a t-value of 25.1, an absurd distance past the critical value of about 1.97\. Polymarket versus major cryptocurrencies returns 4.6, past the critical 2.365\. So the ordering is stocks, then crypto, then prediction markets, each step up a real jump in daily risk. Within Polymarket, economic-indicator markets are the most volatile at a mean daily volatility of 0.13, significantly above the rest (t = 2.63, p = 0.01), while celebrity markets are the calmest at 0.10 (t = minus 2.16, p = 0.03). The practical read: position sizing that feels sane on a stock will get you liquidated on a prediction market, because the same nominal edge rides on three-to-ten times the daily swing. Here is the contrarian kicker the author half-buries. The obvious hypothesis, that volatility spikes as resolution nears, because liquidity providers pull out to dodge divergence loss and news arrives faster, does not hold. Comparing volatility 90-to-61 days out (mean 0.073), 60-to-31 days (0.058), and 30-to-1 days (0.059), no pairwise t-test is significant; the largest is t = 1.04, p = 0.15\. If anything, volatility drifts down into resolution, the opposite of the story. Do not assume the endgame is the wild part. On this sample it was the opening. ## The prices are biased: longshots are overpriced Convergence and volatility are about dynamics. The sharper edge is static mispricing. Bundle every market by its predicted probability and check how often each bucket actually resolves "yes." A well-calibrated market has the 10-to-20% bucket resolving yes about 15% of the time. Instead, the low buckets resolve far less often than their price implies. Ninety days out, the 0-to-10% bucket resolved yes just 2.4% of the time, and the 10-to-20% bucket only 10.5%. The market was charging longshot prices for events that almost never happened. Formalize it with a weighted least squares regression of each bundle's actual outcome rate on its average predicted probability, weighting by how many markets fall in each bundle: $$ \\text{actual}\_b = \\beta\_0 + \\beta\_1 \\cdot \\bar{p}\_b, \\qquad H\_0: \\beta\_1 = 1 \\;\\; \\text{vs} \\;\\; H\_1: \\beta\_1 > 1 $$ If the market were perfectly calibrated, the slope beta-1 would be exactly 1: a bucket priced at 30% should resolve yes 30% of the time. A slope above 1 means the line is too steep, so low prices undershoot reality (longshots overpriced) and high prices overshoot (favorites underpriced). The fit gives beta-1 = 1.13 with a standard error of 0.058 and an R-squared of 0.955\. Test it: the t-statistic is (1.13 minus 1) divided by 0.058, which is 0.13 over 0.058, about 2.18\. Against the critical value of 1.734 for 18 degrees of freedom at the 5% level, 2.18 clears it, so you reject calibration and confirm the overestimation of low probabilities. ![Average outcome by predicted-probability bundle with the fitted overestimation curve](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_417-low_prob_overestimation.png) The straight-line model was flagged as misspecified by a Ramsey test, because the true relationship is an S-curve, not a line. The author swapped in a monotone, centrally-symmetric bias function pinned to pass through the point (0.5, 0.5), and it fit at R-squared 0.98 with its own slope of 1.04\. Either way the verdict is identical: bet the favorites. Systematically buying high-probability tokens and avoiding longshots was profitable on this sample, which is the same probability-weighting distortion Kahneman and Tversky documented in prospect theory, now showing up in a crypto order book. This is exactly the kind of durable, ranked mispricing the old article ["The Term Structure of Prediction-Market Strategy and Crowding: Why the Fastest Three Bots Take 80 Percent"](https://aligrithm.com/the-term-structure-of-prediction-market-strategy-and-crowding-why-the-fastest-three-bots-take-80-percent/) warned gets arbitraged by whoever moves first, so treat the edge as real but crowded. ## The prices lean yes: acquiescence bias There is a second, subtler tilt. Because every market is phrased as a yes/no question, people disproportionately buy "yes," which should make yes tokens systematically overpriced. Test it by summing each market's implied probability into an expected count of yes outcomes and comparing to how many actually happened: $$ \\mathbb{E}\[\\text{yes}\] = \\sum\_{i=1}^{N} p\_i, \\qquad H\_0: \\mathbb{E}\[\\text{yes}\] = \\text{actual yes} \\;\\; \\text{vs} \\;\\; H\_1: \\mathbb{E}\[\\text{yes}\] > \\text{actual yes} $$ The expected yes count is just the sum of the individual yes probabilities across all N markets; a binomial test then asks whether the market expected more yeses than reality delivered. Ninety days out, the market's prices implied 72.5 yes outcomes, but only 44 of the 187 markets actually resolved yes, a gap with p far below 0.001\. At 60 days the implied count was 67.8, at 30 days 58.3, both still significant. By 15 days the implied count had fallen to 49.5 against the same 44 actual, and the gap was no longer significant (p = 0.41). So yes, the market leans affirmative, and the lean fades as resolution approaches, another face of the self-correction the convergence section found. But stay skeptical about naming it. Only 44 of 187 markets resolved yes, a base rate of 24%, so a chunk of what looks like acquiescence bias is really the same longshot overpricing seen earlier: most of these markets asked "will this unusual thing happen," and the honest answer was usually no. The author says as much. Two biases pointing the same direction on a small sample are hard to cleanly separate, and 187 markets with 44 yeses is a thin dataset to hang a trading system on. The direction is credible; the precise magnitude is not bankable. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_417-visual-0.png) ## Where this connects This thesis is an empirical stress test of the claim the whole pillar rests on. The old article ["What a Market Price Actually Is: Capital-Weighted Consensus and the Brier Skill Score Gate"](https://aligrithm.com/what-a-market-price-actually-is-capital-weighted-consensus-and-the-brier-skill-score-gate/) said a price earns the name "probability" only by passing a calibration gate. Here the gate is a weighted regression whose slope should be 1, and Polymarket flunks it at 1.13: the price is a biased probability, not a clean one. The convergence result rescues the slogan for the endgame, since the bias shrinks toward resolution, but the opening is where the price and the truth disagree most. The trading implication ties straight into the old article ["The Term Structure of Prediction-Market Strategy and Crowding: Why the Fastest Three Bots Take 80 Percent."](https://aligrithm.com/the-term-structure-of-prediction-market-strategy-and-crowding-why-the-fastest-three-bots-take-80-percent/) A durable, direction-stable mispricing, longshots overpriced and yes overbought, is precisely the kind of signal that gets crowded and front-run by the fastest capital. The edge here is real on 2023-to-2024 data, but it is the most obvious edge in the room, which means its half-life is short and its capacity is small. Fade the longshots, lean on the favorites, act early while categories still disagree, and size for volatility that will humble a stock trader. And remember the sample is 187 markets on a young venue; the pattern is a hypothesis with good p-values, not a law. ## KEY POINTS - On 187 Polymarket markets resolving December 2023 to April 2024, prices converge to the truth as resolution nears: the probability on the eventual outcome rises 12.6 points from 90 to 30 days out (t = 8.01) and 7.9 more from 30 to 5 days (t = 5.36). - Category edges exist early and vanish late. Political markets are sharpest 90 days out (68.1% on the eventual outcome), but by 5 days every category clusters in the high 80s with no significant differences. - Prediction markets are far more volatile than traditional assets. Daily-return volatility beats stocks (Welch t = 25.1) and crypto (t = 4.6); economic markets are most volatile (0.13), celebrity least (0.10). - Volatility does not rise into resolution. Across 90-61, 60-31, and 30-1 day windows no difference is significant (largest t = 1.04), contradicting the liquidity-withdrawal hypothesis; if anything it declines. - Prices overestimate low probabilities. A weighted regression of actual outcomes on predicted probability gives a slope of 1.13 (SE 0.058), significantly above the calibrated value of 1 (t = 2.18), so longshots are overpriced and favorites underpriced. Bet the favorites. - Prices also lean "yes" (acquiescence bias): implied yes counts of 72.5 at 90 days versus 44 actual (p far below 0.001), fading to insignificance by 15 days (p = 0.41). - Stay skeptical: only 44 of 187 markets resolved yes, so acquiescence and longshot overpricing are entangled, the convergence regression is autocorrelated, and 187 markets on a young venue is a thin base for a trading system. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Convex Optimization - Stephen Boyd Lieven Vandenberghe (Amazon)](https://www.amazon.com/s?k=Convex+Optimization+Stephen+Boyd+Lieven+Vandenberghe&ref=aligrithm.com) - [An Introduction to Copulas - Roger Nelsen (Amazon)](https://www.amazon.com/s?k=An+Introduction+to+Copulas+Roger+Nelsen&ref=aligrithm.com) - [CHARLES UNIVERSITY (analysis of prediction markets in crypto)](https://scholar.google.com/scholar?q=CHARLES+UNIVERSITY&ref=aligrithm.com) ### 9.39 The DePM Design Space: An 8-Stage Microstructure Map URL: https://aligrithm.com/the-depm-design-space-an-8-stage-microstructure-map/ Last updated: 2026-07-27T13:26:24.000Z Polymarket looks like the obvious way to build a prediction market on a blockchain. It is not. Rahman, Al-Chami, and Clark went back through the history of decentralized prediction markets, which starts in 2011 and runs to hundreds of proposals, distilled the field down to 35-plus notable systems, and showed that the design everyone now copies bucked the trend rather than following it. The two founding blueprints, both from 2014, point in opposite directions: Truthcoin, where the market creator sets the odds and risks its own money, and the Princeton academic paper, where the creator is passive and risks nothing. Polymarket picked the passive one, then bolted on pieces neither original design had. To compare systems that disagree on almost everything, the authors do what the old article ["Separate Detection from Execution: A Three-Layer, Typed-Agent Architecture"](https://aligrithm.com/separate-detection-from-execution-a-three-layer-typed-agent-architecture/) did for a trading bot: they refuse to argue about whole systems and instead cut the machine into modules you can swap and reason about one at a time. Their cut is an eight-stage workflow: underlying infrastructure, market topic, share structure and pricing, initialization, trading, resolution, settlement, and archiving. This article walks that map, pulls out the math that makes a prediction market solvent by construction, and lands on the one stage that is genuinely dangerous. ## The eight stages, side by side The payoff of a modular workflow is a single table where systems that look nothing alike line up column by column. Read across a row and you get a system's full design; read down a column and you see the menu of choices for one stage. ![Classification of selected decentralized prediction markets across the eight workflow stages, from Truthcoin in 2014 to Zeitgeist in 2022, showing how Polymarket differs from earlier designs](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_416-depm_classification.png) The table is the argument. Truthcoin and Augur v1 sit on the left with sidechains, winner-take-all shares, automated bookmaking, and reputation-token voting. Polymarket sits apart: a smart contract on Polygon, a negative-risk yes-no share structure, USDC as the numeraire, splitting for initialization, an optimistic vote only when a market is disputed, and gasless push settlement. No single row is the "best" design. Each column is a trade-off between decentralization, expressiveness, and resistance to manipulation, and the authors spend the paper naming those trade-offs and marking ten open research gaps where nobody has a good answer yet. ## What a prediction market actually is Before the design choices, the paper pins down what all these systems have in common with a definition sparse enough to cover every variant. A single market is a tuple of three objects. $$ M = (\\Omega,\\; J,\\; R) $$ Omega is the outcome space, the set of ways the event can resolve. J is the finite set of contract labels, the shares you can hold. R is the payoff map: it says how many units of the numeraire one unit of share j pays when the world lands on outcome omega. Take the HBO Satoshi documentary market. Omega is the list of people the film might name, J is one share per name plus an "other or multiple" share, and R pays $1 to the share matching whoever the film actually named and $0 to the rest. When Omega is a set of mutually exclusive names and each share pays either 0 or 1 with exactly one winner, you have the winner-take-all special case, the Arrow-Debreu market that most people picture when they think "prediction market." The load-bearing property is not the payoff, it is the collateral behind it. A bookmaker quotes odds and promises to pay; it does not have to hold the money. A prediction market must hold enough to cover every outcome at all times, starting from the very first trade. The paper writes this as a solvency invariant. $$ L(S) = \\sup\_{\\omega \\in \\Omega} \\sum\_{j \\in J} S\_j \\, R(\\omega)\_j \\qquad\\text{and}\\qquad \\mathrm{Treas} \\geq L(S) $$ S is the vector of how many of each share are outstanding. L(S) is the worst-case liability: for each possible outcome you total up what you would owe, then take the largest such total across all outcomes. The treasury must always sit at or above that worst case. Work it: a three-outcome winner-take-all market with 100 shares outstanding of each label owes at most 100 times $1 in every outcome, because only one label wins, so L(S) is $100 and the treasury has to hold at least $100\. Any operation that mints more shares has to top the treasury up by at least the increase in worst-case liability. The old article ["The Market Maker's Impossibility: Speed Versus Accuracy, and Why Arbitrage Is Tolerated"](https://aligrithm.com/the-market-makers-impossibility-speed-versus-accuracy-and-why-arbitrage-is-tolerated/) described why a maker cannot both quote fast and quote right; this invariant is the flip side, the reason a prediction market never has to fight a winning ticket. It is collateralized to pay by construction, so it is financially indifferent to who wins. ## Share structures: four ways to slice a bet The stage with the most quiet consequence is how you structure the shares. The paper lays out three main types and a variant, and the choice determines what markets you can even express. Winner-take-all issues one share per outcome, and the shares are built to be mutually exclusive and complete, so exactly one pays $1\. That is why holding one of every share equals holding $1, the fact the whole splitting mechanism runs on. The prices work as probabilities: if the "A wins" share trades at $0.54, the market implies a 54% chance. The tidy claim that the prices sum to $1.00 is imprecise, though. Shares have a bid and an ask, and the honest statement is that the sum of bids and the sum of asks straddle $1.00, with a spread that can be wide. A single displayed "price," like the last trade or the bid-ask midpoint, need not sum to a dollar across outcomes. That is not a broken market, just a misread screen. Yes-no bundles issue two shares per outcome, a yes and a no, so each outcome is its own little binary market. This buys flexibility: multiple outcomes can resolve yes, or none can, which is exactly what you want for a market on which words Trump will say in a speech. New outcomes can be added mid-life because the bundle does not have to be complete up front. Polymarket runs a disciplined variant called negative risk, where the yes shares are forced to be complete and mutually exclusive after all, so both a per-outcome constraint and a market-wide one hold. $$ \\sum\_{k \\in \\Omega} R(\\omega)\_{j\_{k\_Y}} = 1 \\qquad\\text{and}\\qquad \\sum\_{k \\in \\Omega} R(\\omega)\_{j\_{k\_N}} = |\\Omega| - 1 $$ The yes shares sum to $1, the no shares sum to the number of outcomes minus one. In the six-way HBO market, one "yes" wins and pays $1, while five of the six "no" shares win, so the no shares pay $5 in total, which is the six outcomes minus one. The practical consequence is a conversion gadget: holding a "no" on Hal Finney pays exactly the same as holding a "yes" on every other candidate at once, so a trader can swap a single no share for a portfolio of yes shares and back. That gadget is what quietly aligns yes and no prices and hands arbitrageurs a low-friction way to erase inconsistencies, the same inconsistencies measured at roughly $40M in the combinatorial-arbitrage study. The fourth type is scalar, where the payout tracks a quantity like a vote share or a temperature, normalized to a 0-to-1 range. Polymarket mostly cheats here, chopping the quantity into buckets and running a yes-no market per bucket. That unifies the interface but creates its own trap: if your forecast of the popular vote straddles two buckets, you have to buy both and dilute your return, and the market jumps hard when consensus crosses a bucket boundary. ## Initialization: someone has to be the first counterparty The stage the authors call the biggest evolution from Truthcoin to Polymarket is initialization, the question of how the very first trade happens when there is nobody on the other side yet. There are three answers, and they trade the same thing the old article ["Fast Fills Are Bad Fills: Make vs Take, Adverse Selection, and Why the Spread Is Posterior Variance"](https://aligrithm.com/fast-fills-are-bad-fills-make-vs-take-adverse-selection-and-why-the-spread-is-posterior-variance/) worried about: who takes the risk of being first. Automated bookmaking posts a loss fund and an algorithmic price rule, originally Hanson's logarithmic market scoring rule, so the first trader trades against the operator's money instantly. The operator is the counterparty, which means it can lose, and it carries the burden of setting the opening odds well. Splitting takes the opposite stance: a trader pays $1, receives one share of every outcome, and lists some for sale, so the operator posts zero risk but the first trader has to wait for a second trader to show up. Matching, the third route, mirrors a futures exchange, escrowing each side's maximum loss and minting shares only when a buyer and seller coincide on price and size. The clean way to see the tension is the paper's insight: if the first trader is going to be able to make a dollar, someone has to be able to lose a dollar, and that someone is either the operator or a later trader. You cannot have instant liquidity and zero operator risk at the same time. Splitting picks zero risk and pays for it with a waiting first trader. Automated bookmaking picks instant liquidity and pays for it with a loss fund. That is the whole design decision, stated in one sentence. ## Trading: make and take, on a chain that watches you Once shares exist, they trade like any blockchain token, and here the old make-take question from centralized order books changes shape. A fully on-chain order book is too expensive to run at scale, so the market maker's world splits three ways: centralized exchanges that custody tokens and run a limit order book, partially decentralized designs like Polymarket that match off-chain but settle on-chain, and fully on-chain automated market makers. Automated market makers were born out of prediction-market research, but they fit prediction shares badly. A share price is trapped between $0 and $1, it can snap to either end the instant an event resolves, and once resolved it is frozen there forever. When news breaks, an automated market maker gets drained faster than its liquidity providers can pull out, so the maker eats the loss that the old article ["The Market Maker's Impossibility: Speed Versus Accuracy, and Why Arbitrage Is Tolerated"](https://aligrithm.com/the-market-makers-impossibility-speed-versus-accuracy-and-why-arbitrage-is-tolerated/) said was unavoidable. On-chain trading adds a second tax the centralized world does not have: miners and bots can see your order and front-run it, the maximal-extractable-value problem. The mitigation the Princeton paper proposed back in 2014, before the acronym existed, is the frequent batch auction, which collects orders over a short window and clears them at one uniform price so there is no ordering left to exploit. The whole make-take story you know from a normal exchange still runs here, but every quote is now public, front-runnable, and settled by a chain. ## Resolution: the stage that can be bought Seven of the eight stages are engineering trade-offs. Resolution is where a decentralized prediction market can be attacked outright, because deciding who won is a uniquely decentralized problem: a centralized platform just rules on its own markets, while a decentralized one has to source a truth nobody controls. Polymarket escalates disputed markets to UMA's data verification mechanism, where token holders vote and the majority is paid while dissenters can be slashed. The failure mode is a Keynesian beauty contest. A rational voter is paid for matching the majority, not for reporting the truth, so the system only works when the truth is an obvious focal point everyone expects everyone else to pick. Break that, and voters start guessing what the whales will do. And the whales are concentrated. ![Cumulative share of UMA governance votes by wallet rank, with the top five wallets controlling 45.6 percent and the top thirteen exceeding 65 percent](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_416-uma_vote_concentration.png) At the time of the paper, the top five UMA wallets controlled 45.6% of the votes and the top thirteen exceeded 65%, enough to clear the passing threshold on their own. A market on whether Ukraine agreed to a mineral deal was resolved in a way that raised exactly this alarm: large holders staking early on an outcome can scare honest voters away from the truth, and if a successful attack pushes the token price down, the next attack is cheaper, which is a spiral. This is the concrete reason the "wisdom of the crowd" that a prediction market advertises has a soft spot: the crowd priced the event, but a small committee of token holders decides whether that price pays out. Settlement itself is the easy part after that, a winning share redeems for $1 of stablecoin, and the operator, being collateralized by construction, has no reason to stall the way a bookmaker does. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_416-visual-0.png) ## Where this connects This SoK is the map that the earlier Pillar 9 pieces were drawing freehand. The old article ["Separate Detection from Execution: A Three-Layer, Typed-Agent Architecture"](https://aligrithm.com/separate-detection-from-execution-a-three-layer-typed-agent-architecture/) argued for splitting a system into modules you can reason about alone; the eight-stage workflow is that same instinct applied to the market itself, and it lets you say precisely where two platforms differ instead of waving at "Polymarket versus Augur." The old article ["Fast Fills Are Bad Fills: Make vs Take, Adverse Selection, and Why the Spread Is Posterior Variance"](https://aligrithm.com/fast-fills-are-bad-fills-make-vs-take-adverse-selection-and-why-the-spread-is-posterior-variance/) lives inside the trading and initialization stages, now with the blockchain twist that a full order book is too costly on-chain and every quote is front-runnable. The old article ["The Market Maker's Impossibility: Speed Versus Accuracy, and Why Arbitrage Is Tolerated"](https://aligrithm.com/the-market-makers-impossibility-speed-versus-accuracy-and-why-arbitrage-is-tolerated/) is the solvency invariant read from the other side: the market is built to pay, so arbitrage and divergence loss are the tax the design accepts rather than a bug it fixes. The one thing the map makes unavoidable is that the danger is not in the pricing math. It is in resolution, where a handful of token wallets, not the crowd, get the last word on what the price meant. ## KEY POINTS - Decentralized prediction markets go back to 2011 and number in the hundreds; the paper distills 35-plus notable systems into one eight-stage workflow: infrastructure, topic, share structure and pricing, initialization, trading, resolution, settlement, archiving. - Polymarket did not follow the original blueprints. It took the passive-creator Princeton design over the odds-setting Truthcoin design, then added negative-risk shares, USDC, splitting, and an optimistic oracle. - A market is the tuple (Omega, J, R): outcome space, share labels, and a payoff map. The winner-take-all case pays $1 to one share and $0 to the rest. - Solvency is built in: the treasury must always cover the worst-case liability, the largest total payout across all outcomes, so a prediction market never has to fight a winning ticket the way a bookmaker does. - Share structure sets expressiveness: winner-take-all, yes-no bundles, Polymarket's negative-risk variant where yes shares sum to $1 and no shares sum to the outcome count minus one, and rarely-used scalar markets that Polymarket fakes with buckets. - Initialization forces a choice: instant liquidity with operator risk (automated bookmaking) or zero operator risk with a waiting first trader (splitting). If the first trader can win a dollar, someone else must be able to lose one. - On-chain trading makes a full order book too expensive, exposes every quote to front-running (MEV), and drains automated market makers when news hits; frequent batch auctions are the standing mitigation. - Resolution is the attackable stage. UMA's vote is a Keynesian beauty contest, and with the top five wallets at 45.6% and the top thirteen above 65%, a small set of holders can settle a disputed market against the truth. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Convex Optimization - Stephen Boyd Lieven Vandenberghe (Amazon)](https://www.amazon.com/s?k=Convex+Optimization+Stephen+Boyd+Lieven+Vandenberghe&ref=aligrithm.com) - [An Introduction to Copulas - Roger Nelsen (Amazon)](https://www.amazon.com/s?k=An+Introduction+to+Copulas+Roger+Nelsen&ref=aligrithm.com) - [SoK: Market Microstructure for Decentralized Prediction Markets (DePMs) (market microstructure for prediction markets)](https://scholar.google.com/scholar?q=SoK%3A+Market+Microstructure+for+Decentralized+Prediction+Markets+%28DePMs%29&ref=aligrithm.com) ### 7.4 State-Space Models for Price: CryptoMamba vs Transformers (Skeptical) URL: https://aligrithm.com/state-space-models-for-price-cryptomamba-vs-transformers-skeptical/ Last updated: 2026-07-27T13:25:59.000Z Every few years a new architecture gets pointed at Bitcoin and a paper announces it won. This round it is Mamba, the selective state-space model that is genuinely reshaping language and vision. Sepehri, Mehradfar, Soltanolkotabi, and Avestimehr at USC built CryptoMamba, a compact Mamba network that reads 14 days of Bitcoin OHLCV and predicts the next day's close. The numbers are real and they reported them cleanly: a test-set RMSE of 1598, a 136k-parameter model beating an S-Mamba baseline with 330k, and an Extended Smart trading rule that turns $100 into $262 over a year. The paper concludes there is "a huge advantage for SSMs" in price forecasting. The architecture is legitimate and the engineering is careful. The evidence for the trading claim is not, and the gap between those two things is the whole point of this teardown. The problem is not the model. It is the benchmark the paper never ran and the sample it ran on: one year of daily bars, no naive baseline, no buy-and-hold line, and a ranking that flips between validation and test. Predicting the level of a near-random-walk is easy and says almost nothing about whether you can trade it. ## What a selective state-space model actually is Strip the branding and an SSM is a linear system with a hidden memory. A small state vector carries information forward, decays part of it each step, absorbs the new input, and emits an output. That is the same machinery Kalman wrote down in 1960, now stacked and trained by gradient descent. $$ \\dot{x}(t) = A\\,x(t) + B\\,u(t), \\qquad y(t) = C\\,x(t) + D\\,u(t) $$ Read it plainly. The input u is today's scalar feature, the state x is a low-dimensional memory, and A, B, C, D are learned. A controls how fast the memory fades, B injects the new input into the memory, C reads the memory out into a prediction, and D is a direct passthrough usually dropped. Work a scalar case: set A to -0.5 and B to 1\. If the state is 2 and the input is 1, then x-dot is -0.5 times 2 plus 1 times 1, which is 0, so the memory holds. Cut the input to 0 and x-dot becomes -1, so the memory bleeds off toward zero. The state is a leaky record of everything it has seen. To run on daily bars the continuous system is discretized with a zero-order hold over a step size Delta. $$ x\_k = \\bar{A}\\,x\_{k-1} + \\bar{B}\\,u\_k, \\qquad y\_k = \\bar{C}\\,x\_k, \\qquad \\bar{A} = \\exp(A\\,\\Delta) $$ The bar terms are the discrete versions of A and B. With A at -0.5 and Delta at 1, A-bar is exp(-0.5), which is 0.607, and B-bar works out to about 0.786\. So x-k equals 0.607 times the last state plus 0.786 times today's input. The one trick Mamba adds is selectivity: it makes B-bar, C, and Delta functions of the input itself, so the decay rate and the read-out change with the data instead of staying fixed. That is what lets it react to regime shifts, and because the scan is hardware-aware, cost scales linearly with sequence length rather than quadratically like attention. This is real progress over a plain LTI state-space model, and over the quadratic cost of a Transformer. None of it is in dispute. ## The accuracy number, and the baseline they never ran Here is where the story gets thin. The reported accuracy metric is MAPE, the mean absolute percentage error between predicted and actual next-day close. $$ \\text{MAPE} = \\frac{100}{n} \\sum\_{i=1}^{n} \\left| \\frac{y\_i - \\hat{y}\_i}{y\_i} \\right| $$ CryptoMamba's volume-inclusive variant posts a MAPE of 2.034%. On a $60,000 Bitcoin that is a typical miss near $1,200, consistent with its reported MAE of 1121\. Sounds sharp. Now run the one baseline the paper skips: predict tomorrow's close equals today's close. For that naive rule the error on each day is exactly that day's absolute return, so its MAPE equals the average absolute daily return over the test window. Bitcoin's daily moves across that year ran roughly 2 to 3% (an estimate from the price path, not a paper number). That puts naive persistence at a MAPE somewhere around 2 to 3%, sitting right on top of CryptoMamba's 2.034%. ![All six models' next-day-close predictions overlaid on the actual Bitcoin price across train, validation, and test. From this distance every model traces the price line, because predicting the level of a near-random-walk is easy for anything with memory.](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_427-forecast_all_models.png) Look at the figure and the tell is obvious: all six models, from a plain LSTM to CryptoMamba, produce prediction lines that hug the actual price. They look almost identical because they are all doing the same easy thing, echoing a level that is dominated by yesterday's level. The old article ["Synthetic Prices: EMA of Random Numbers as a Market Model"](https://aligrithm.com/synthetic-prices-ema-of-random-numbers-as-a-market-model/) made this exact point: prices behave like random numbers with memory, so a model that forecasts the price level will look accurate while telling you nothing about the increment. RMSE and MAPE on the level reward you for the autocorrelation of the level, not for skill on the change. The paper compares CryptoMamba to LSTM, GRU, iTransformer, and S-Mamba, and never to the random walk that would expose how little any of them add. ## One year is not a sample The test set runs from September 2023 to September 2024, and predictions start 14 days in, so the entire out-of-sample record is roughly 350 daily points on a single asset in a single regime. That is one bull market, not a distribution of outcomes. The instability shows up the moment you read both tables. On validation, CryptoMamba is mediocre: its Vanilla strategy ends at $124 while Bi-LSTM-v hits $157 and GRU-v hits $154\. On test, the ranking inverts and CryptoMamba jumps to the top. A model that ranks fifth on one year and first on the next has not demonstrated skill, it has demonstrated that one year of daily data cannot separate these models. The paper also fixes a single random seed, which means there is no run-to-run variance reported at all, so a 1598 versus 1651 RMSE gap between CryptoMamba and S-Mamba has no error bar attached. On 350 points those two numbers are almost certainly inside each other's noise. The timeframe is the other quiet problem. The old article ["Ridge Above 1h, XGBoost Below 5min"](https://aligrithm.com/ridge-above-1h-xgboost-below-5min/) argued that the right model depends entirely on the bar size, and that at the daily scale a heavier architecture rarely earns its complexity. This paper is the daily scale, predicting one-step-ahead level, which is close to the easiest and least informative forecasting target there is. The architecture race is being run on the track where architecture matters least. ## The trading result barely clears buy-and-hold The trading section is meant to prove the forecasts are useful, using three rules of rising aggression: Vanilla, Smart, and Extended Smart, the last of which can short up to 0.002 BTC. Risk is measured with maximum drawdown. $$ \\text{MDD} = \\max\_{t \\in \[1,T\]} \\left( \\frac{\\max\_{i \\in \[1,t\]} P\_i - P\_t}{\\max\_{i \\in \[1,t\]} P\_i} \\right) $$ MDD is the worst peak-to-trough drop as a fraction of the peak. Work it: if the portfolio peaks at $130 then falls to $110, the drawdown is 130 minus 110 over 130, which is 15.4%. CryptoMamba-v's Extended Smart run reports an MDD of 11.09%, the best risk control in the field, while LSTM's Extended Smart run blew a 71.46% drawdown. So far so good for the winner. The catch is the missing reference line. ![Extended Smart test-year final balances for the volume-inclusive models, against an approximate buy-and-hold BTC line. Only CryptoMamba-v clearly beats holding Bitcoin; the rest match it or lose.](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_427-final_balance_vs_buyhold.png) Bitcoin ran from roughly $26,500 to about $60,000 over the test year, near +125%, so a passive $100 in spot BTC would have become about $225 with zero modeling (this line is my estimate, not a paper figure, since the paper never plots it). Against that line the results deflate. CryptoMamba-v's Extended Smart ends at $262.78, a real beat of about 38 points, but its Smart variant ends at $213.20, which loses to just holding. S-Mamba-v lands at $222.50, a tie with spot. iTransformer-v's Extended Smart at $207.67 loses to spot, and its Vanilla at $241.50 sits within a hair of CryptoMamba-v's $246.58\. Every recurrent baseline finishes below buy-and-hold. In other words, the single best configuration of the best model, with shorting switched on, beat holding Bitcoin by a modest margin over one bull year, and most of the "winning" trading strategies underperformed doing nothing. That is not a result you lever up on. ## What actually holds up Two things survive the skepticism. The efficiency claim is real: 136k parameters beating a 330k S-Mamba and a 569k Bi-LSTM, with a 29-minute train and 1.17 ms inference, is a genuine argument for selective SSMs as compact sequence models. And using trading algorithms as an evaluation lens, rather than RMSE alone, is the right instinct, because regression error and profit are not the same objective. The authors half-built the honest test. They just left out the two benchmarks that would have graded it: naive persistence for the forecast and buy-and-hold for the trade. Put those back in and CryptoMamba looks like a tidy, efficient model that has not yet been shown to beat the two dumbest strategies in finance by a margin that survives a single year of daily bars. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_427-visual-0.png) ## KEY POINTS - CryptoMamba is a selective state-space (Mamba) model reading 14 days of OHLCV to predict the next day's Bitcoin close. The architecture is sound: input-dependent decay and read-out, linear-time scan, and only 136k parameters versus 330k for S-Mamba and 569k for Bi-LSTM. - The headline accuracy, a 2.034% MAPE, is measured on the price level, which is dominated by yesterday's level. A naive "tomorrow equals today" rule has a MAPE equal to the average daily absolute return, roughly 2 to 3% here, so the model's edge over the untested baseline is a few tenths of a percent. - All six models' prediction lines hug the actual price in the paper's own figure, because forecasting the level of a near-random-walk is easy for anything with memory. The paper never compares to a random walk. - The out-of-sample test is one bull year, about 350 daily points on one asset. CryptoMamba ranks mid-pack on validation and first on test, and only one seed is run, so key RMSE gaps (1598 vs 1651) carry no error bar. - The trading win barely clears buy-and-hold. BTC rose about +125% in the test year (a $100 to roughly $225 passive result). CryptoMamba-v's best rule ends at $262.78, but its Smart variant ($213.20) and every recurrent baseline lose to spot, and iTransformer sits within a hair of it. - What holds up: the efficiency case for compact SSMs, and the instinct to grade forecasts by trading outcomes rather than RMSE alone. What is missing: the naive-persistence and buy-and-hold benchmarks that would decide whether any of it is skill. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Statistically Sound Indicators for Financial Market Prediction - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Statistically+Sound+Indicators+for+Financial+Market+Prediction+Timothy+Masters&ref=aligrithm.com) - [Systematic Trading - Robert Carver (Amazon)](https://www.amazon.com/s?k=Systematic+Trading+Robert+Carver&ref=aligrithm.com) - [CRYPTOMAMBA: LEVERAGING STATE SPACE MODELS FOR ACCURATE BITCOIN PRICE PREDICTION (2501.01010v2)](https://arxiv.org/abs/2501.01010?ref=aligrithm.com) ### 9.38 Training an LLM to Forecast: Outcome-Based RL and the Calibration Win URL: https://aligrithm.com/training-an-llm-to-forecast-outcome-based-rl-and-the-calibration-win/ Last updated: 2026-07-27T13:26:23.000Z Here is a headline that should make you reach for the small print. A 14-billion-parameter open model, one you can run on a single node, was trained with reinforcement learning on old prediction-market questions and came out matching OpenAI's o1 on accuracy, beating it on calibration, and turning a profit in a Polymarket trading simulation. The paper is Turtel and coauthors' "Outcome-based Reinforcement Learning to Predict the Future," and the numbers are real: a seven-run ReMax ensemble hit a Brier score of 0.190 against o1's 0.202, with an expected calibration error of 0.062 against o1's 0.093, and its simulated bets earned $52 against o1's $39. Read past the headline and the story sharpens into something more useful than "small model beats big model." The market itself still won. Polymarket's own prices scored a Brier of 0.151, cleaner than every model in the study by a wide margin. So the real result is not that a language model out-forecasts a liquid market. It is that outcome-only RL can drag a model's probabilities onto the diagonal, and that the leftover edge, wherever it exists, sits exactly where the market is least sure. That is a calibration story dressed up as an alpha story, and calibration is the part worth stealing. ## The reward is a bet you grade yourself on The whole training loop rests on one choice of reward. The model reads a yes/no question plus news headlines dated before the forecast, emits a single probability p-hat that the event happens, and later observes the binary outcome y. The reward is the negative Brier score: $$ R = -(\\hat{p} - y)^2 $$ Read it plainly. If the event happens, y is 1, and your reward is minus the squared distance from 1\. If it does not, y is 0, and you are penalized by the squared distance from 0\. Say the model forecasts 0.7\. If the event happens, the reward is minus (0.7 minus 1) squared, which is minus 0.09\. If it does not, the reward is minus (0.7 minus 0) squared, which is minus 0.49\. Confidence in the wrong direction is punished far harder than the same confidence in the right one, and that asymmetry is the entire point. Brier is a strictly proper scoring rule, which means the way to maximize your expected reward is to report your true probability, not to shade it. The old article ["What a Market Price Actually Is: Capital-Weighted Consensus and the Brier Skill Score Gate"](https://aligrithm.com/what-a-market-price-actually-is-capital-weighted-consensus-and-the-brier-skill-score-gate/) leaned on the same object to decide whether a price carries skill at all. Here it becomes the training signal itself: no human labels, no preference pairs, just the outcome and a squared error. Two guardrail variants matter later. During training a malformed or missing probability gets the maximum loss of 1.0 (reward minus 1); during evaluation the "soft Brier" assigns a gentler 0.25 to a failed parse, the same score you would get by guessing 50%, so a formatting slip does not masquerade as a catastrophic forecast. ## The training signal that quietly breaks The obvious way to optimize this reward is GRPO, the group-relative method behind DeepSeek's math models. For each question you sample a group of G answers, score each one, and turn the scores into advantages by subtracting the group mean and dividing by the group standard deviation: $$ \\mu = \\frac{1}{G}\\sum\_{i=1}^{G} r^i, \\qquad \\sigma = \\sqrt{\\frac{1}{G}\\sum\_{i=1}^{G}\\left(r^i - \\mu\\right)^2}, \\qquad \\hat{A}^i = \\frac{r^i - \\mu}{\\sigma} $$ The mean subtraction is harmless. The division by sigma is the landmine. It rescales every question's spread of rewards to unit variance, which means the size of the gradient stops tracking how much Brier loss is actually at stake and starts tracking only the ranking of answers within that one group. Work two questions through it. Question A draws two rollouts scoring minus 0.04 and minus 0.64, a genuine gap between a sharp answer and a badly wrong one. Its mean is minus 0.34, its sigma is 0.30, so the normalized advantages are plus and minus 1.0\. Question B draws two rollouts scoring minus 0.25 and minus 0.26, a difference of one hundredth of nothing. Its sigma is 0.005, so the normalized advantages are also plus and minus 1.0\. GRPO just told the model that Question B's meaningless 0.01 difference deserves exactly the same push as Question A's meaningful 0.60 one. Multiply that across 100,000 noisy questions and the model learns to chase noise, and the cheapest way to make the reward look decisive is to get loud: push probabilities toward 0 and 1. The failure mode is not hypothetical. Raw GRPO drove 39.3% of predictions into the extreme 0-to-10% or 90-to-100% buckets, many landing exactly on 0 or 1\. The model literally wrote "the probability is extremely low, almost zero" and printed 0.000 for an event that was merely unlikely, not impossible. A forecaster who says zero is a forecaster who has stopped forecasting. ## The fix is to stop dividing The correction is almost embarrassingly small. Modified-GRPO keeps the mean subtraction and deletes the sigma division; ReMax replaces the group standard deviation with a learned baseline: $$ \\hat{A}^i\_{\\text{Mod-GRPO}} = r^i - \\mu \\qquad\\qquad \\hat{A}^i\_{\\text{ReMax}} = r^i - b^i $$ Now the advantage keeps its units. A question where the model is confidently, expensively wrong produces a large negative advantage; a question where all the answers are mediocre produces a small one. The gradient stays proportional to the Brier loss, which is exactly what you want when the whole objective is calibration. Concretely, dropping the normalization cut the extreme-bucket share from 39.3% under raw GRPO to 7.9% under Modified-GRPO. The Brier bounded-ness in \[0, 1\] does the rest: because rewards cannot blow up, removing the variance scaling does not destabilize training the way it would with an unbounded loss. Removing one normalization opened a different hole. Modified-GRPO without guardrails started emitting gibberish and switching mid-sentence into Chinese, because once the reward gradient cannot cleanly separate a valid forecast from an invalid one, the policy wanders off the manifold of coherent text. The fix was a scorer that docks reward for non-English passages, random-character strings, and missing rationales inside the think block, plus a hard truncation of any input past 16,000 characters and schema validation that zeroes the reward on any malformed output. With guardrails on the 100k corpus the extreme-bucket share settled at 13.1%, and the near-zero forecasts moved off the floor. This is DPO's neighborhood too: the paper keeps Direct Preference Optimization as an off-policy baseline, and it lands mid-pack, better than raw GRPO but behind the two on-policy fixes. ## The part most "my AI predicts the future" claims get wrong Before any of the accuracy numbers mean anything, ask one question: could the model have peeked? Temporal leakage is the disease of forecasting backtests. If the training data or the prompt contains any whiff of information dated on or after the forecast date, the model is not predicting, it is remembering, and the whole evaluation is fiction. This is the single most important paragraph in the paper and the one the headline never mentions. The authors build the test set to make peeking hard. Every test question's forecast date falls after the latest resolution date of any training question, so the model cannot have trained on a resolved version of anything it is later asked. Questions with open-ended resolution dates, the "will X happen sometime before Y" kind that leak the future through survivorship, are dropped. And they run OpenAI o3 over the retrieved news to flag any story that quietly contains post-forecast information, then delete those questions. Even the news retrieval is date-gated at day-level granularity so the prompt cannot cite an article from after the cutoff. This is the discipline Paleka and coauthors' "Pitfalls in Evaluating Language Model Forecasters" spends a whole paper insisting on, and it is why most breathless forecasting demos evaporate under scrutiny. Trust the accuracy numbers only because the leakage controls came first. ## What the training actually bought: calibration Line the algorithms up and the pattern is clean. Accuracy barely moves; calibration moves a lot. ![Brier score and expected calibration error for each training algorithm, split by training set, with error bars](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_415-algo_comparison.png) On the left, Brier scores cluster in a narrow band from about 0.19 to 0.22\. The base DeepSeek-R1 14B sits at 0.215, o1 at 0.202, and the ReMax ensemble at 0.190\. That is a real but modest accuracy gain, and the paired test against o1 is only marginally significant. On the right, expected calibration error tells the louder story: the base model at 0.089 and o1 at 0.093 both roughly halve to ReMax's 0.062 and Modified-GRPO's 0.054\. The training did not mostly make the model smarter about which way events break. It made the model honest about how sure it is. Expected calibration error is the number doing the work. Bin the forecasts into ten equal-mass buckets, and for each bucket compare the average confidence to the fraction of events that actually happened: $$ \\text{ECE} = \\sum\_{b=1}^{B} \\frac{n\_b}{N}\\,\\bigl|\\,\\text{acc}\_b - \\text{conf}\_b\\,\\bigr| $$ Each term is a bin's share of the forecasts times the gap between what the model claimed and what reality delivered. Suppose the 0.8-to-0.9 bin holds 120 of 1,265 forecasts with an average confidence of 0.85, but only 68% of those events happened. That bin contributes (120 divided by 1,265) times the absolute value of 0.85 minus 0.68, which is 0.095 times 0.17, about 0.016\. Sum the contributions across all ten bins and you get the ECE. An overconfident model piles up gaps in the extreme bins and its ECE balloons; a calibrated model keeps every bin's claim close to its outcome. ![Reliability diagram contrasting an overconfident forecaster with a calibrated one against the diagonal](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_415-reliability_diagram.png) That is exactly the diagram the old article "A Reliability Diagram From Your Last 50 Trades" told you to build from your own trade history. The diagonal is perfect calibration: events you call 80% should happen 80% of the time. The overconfident curve, the raw-GRPO shape, sags below the line where it is bullish and rides above it where it is bearish, because it says 85% when reality is closer to 68%. The training win is geometric: it drags the curve back onto the diagonal. This is the same lesson as the old article "Bayesian Edge in Log-Odds: Better, Earlier, Calibrated, or It's Noise," which insisted that an edge you cannot calibrate is just noise wearing a confidence interval. ## Did it actually make money? In simulation, yes, and the shape of the profit is more honest than the total. Convert each forecast into a one-share Polymarket trade: if the model's probability beats the market price, buy the yes share at market plus a penny for fees; if it is below, short it; the extra cent stands in for slippage. Rank all 1,265 trades by expected edge and walk down the list. The ReMax ensemble earned $52 on $433 of cost and Modified-GRPO earned $54 on $428, both roughly a 10% return, against o1's $39\. Every model turned a nominal profit even when forced to bet on all markets. But look at where the edge lives. ![Bet win probability minus Polymarket prediction, by market confidence bucket](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_415-edge_by_market_confidence.png) The bars measure how much more often the model's bets won than the market price implied, split by how confident the market was. When the market was near certain, at the far ends, the model added nothing statistically distinguishable from zero, and the error bars swallow the bars whole. Essentially all the profit came from the 40-to-60% bucket, the questions where the market was closest to a coin flip. There the model's bets won 11.8 percentage points more often than the price predicted, worth about a 20% return on those questions alone. The paper even hands you the exit rule: truncate the strategy once the expected edge drops below your own ECE and you keep almost the entire upside while cutting the loss-making tail. Stay skeptical about the money. This is a simulation on a small, low-liquidity venue with a one-cent fee stand-in, and the market's own Brier of 0.151 still beats the model's 0.190 outright. The model is not overturning the market; it is adding incremental information precisely where the market has the least conviction, and only a disciplined bettor who stops trading at the calibration threshold gets to keep it. The authors say as much: financial markets are thousands of times larger, and this is a tool to support human judgment, not replace the price. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_415-visual-0.png) ## Where this connects This article is the machine-learning end of a thread the pillar has been pulling by hand. The old article ["What a Market Price Actually Is: Capital-Weighted Consensus and the Brier Skill Score Gate"](https://aligrithm.com/what-a-market-price-actually-is-capital-weighted-consensus-and-the-brier-skill-score-gate/) defined the price as a probability you grade with Brier; here Brier becomes the reward that trains a model to produce those probabilities. The old article "Bayesian Edge in Log-Odds: Better, Earlier, Calibrated, or It's Noise" argued that an uncalibrated edge is noise; this paper shows an RL pipeline whose entire measurable gain is calibration. And the old article "A Reliability Diagram From Your Last 50 Trades" handed you the audit tool; the reliability diagram here is that same tool pointed at a 14B model instead of your trade blotter. The deeper lesson generalizes past language models. Any system optimized against a noisy proper score will drift toward overconfidence if the training signal rescales away the magnitude of its errors. The cure, keeping the gradient proportional to the actual loss and refusing to reward degenerate answers, is a discipline every model-builder in this book needs, whether the model is a transformer or a linear rule. Calibration is not a nicety you add at the end. It is the thing outcome-based RL is actually good at, and the only part of this result that survived contact with a real market. ## KEY POINTS - A 14B open model (DeepSeek-R1-Distill-Qwen-14B) trained with outcome-only RL matched o1 on accuracy (Brier 0.190 vs 0.202) and beat it on calibration (ECE 0.062 vs 0.093), but Polymarket's own prices still won outright at Brier 0.151. - The reward is the negative Brier score, minus (forecast minus outcome) squared, a strictly proper rule: the only way to maximize it is to report your true probability. - Standard GRPO divides advantages by each question's reward standard deviation, which decouples the gradient from the actual Brier loss and drove 39.3% of forecasts into the extreme 0-or-1 buckets. Dropping the division (Modified-GRPO) or using a learned baseline (ReMax) keeps the gradient proportional to the loss and cut extremes to 7.9%. - Removing the normalization reintroduced gibberish and language-switching, fixed with guardrails that penalize non-English text, random strings, and missing rationales, plus schema validation that zeroes malformed outputs. - The accuracy claim is only trustworthy because of aggressive anti-temporal-leakage controls: test dates after all training resolutions, no open-ended resolution questions, and an o3 pass to delete leaked news. Most forecasting demos skip this and are fiction. - The training's measurable win is calibration, not smarts: Brier barely moved while ECE roughly halved. The reliability diagram straightens back onto the diagonal. - In a Polymarket simulation the ReMax ensemble returned about 10% ($52 on $433), but essentially all the edge came from the 40-to-60% market-confidence bucket, worth about 20% there. Truncating at the edge-below-ECE threshold keeps the upside and drops the loss-making tail. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Convex Optimization - Stephen Boyd Lieven Vandenberghe (Amazon)](https://www.amazon.com/s?k=Convex+Optimization+Stephen+Boyd+Lieven+Vandenberghe&ref=aligrithm.com) - [An Introduction to Copulas - Roger Nelsen (Amazon)](https://www.amazon.com/s?k=An+Introduction+to+Copulas+Roger+Nelsen&ref=aligrithm.com) - [Outcome-based Reinforcement Learning to Predict the Future (outcome based RL to predict the future)](https://scholar.google.com/scholar?q=Outcome-based+Reinforcement+Learning+to+Predict+the+Future&ref=aligrithm.com) ### 9.37 Bold vs Timid: The Math of Betting When You Must URL: https://aligrithm.com/bold-vs-timid-the-math-of-betting-when-you-must/ Last updated: 2026-07-27T13:26:23.000Z Here is a result that offends every instinct a careful trader has. You are in a game that is rigged against you, a small negative edge on every bet, and you have to turn a starting stake into a target before you go broke. The disciplined move feels obvious: bet small, spread the risk, give the law of large numbers time to work. That instinct is exactly backwards. Kyle Siegrist's expository treatment of the red-and-black problem proves that when the game is unfair, the strategy that maximizes your chance of hitting the target is bold play: bet your whole fortune, or just enough to reach the goal, on every single game. Grinding small bets, timid play, is provably the worst thing you can do. It hands you a near-certain slow death. This is not a trick or an edge case. It is a theorem, and it flips again when the game turns in your favor. If you have a real edge, bet tiny. If you are behind the house, bet everything. The whole content of "how to gamble if you must" is that the optimal bet size depends on the sign of your edge in a way that has nothing to do with growth rate and everything to do with the geometry of absorbing barriers. That distinction matters because a trader in a funded-account challenge, or anyone trying to hit a fixed target before a drawdown floor, is playing exactly this game, and the Kelly reflex from the old article ["Kelly Rejects Bad Trades Automatically: Why Overbetting Kills and Underbetting Doesn't"](https://aligrithm.com/kelly-rejects-bad-trades-automatically-why-overbetting-kills-and-underbetting-doesnt/) is answering a different question than the one on the table. ## Every bet you make bleeds you Start with the one fact that governs everything. Let your fortune after game n be X-n, and let Y-n be the amount you wager on game n in a game you win with probability p. The expected change in your fortune per bet is fixed. $$ \\mathbb{E}(X\_n) = \\mathbb{E}(X\_{n-1}) + (2p - 1)\\,\\mathbb{E}(Y\_n) $$ Read it plainly. Whatever you bet, your expected fortune moves by two-p-minus-one times the stake. When p is below one half the multiplier is negative, so every dollar you put at risk loses money on average; when p is above one half it gains; when p equals one half it is a martingale that drifts nowhere. Work a number: at p equal to 0.45 you wager 10 dollars, and your expected change is 2 times 0.45 minus 1, which is minus 0.10, times 10, so minus one dollar. Every ten dollars through the game costs you one dollar in expectation. Now chain the logic. Your total expected loss is that edge times the total volume you wager over the whole session. If you want to minimize expected bleed, you minimize the number of dollars that pass through the machine, which means you want to reach the target or bust in as few bets as possible. Timid play does the opposite: it forces enormous volume through a losing edge. This is the seed of why bold wins. You are not trying to grow money, you are trying to touch a target before the edge grinds you to zero, and exposure is the enemy. ## Timid play is gambler's ruin, and gambler's ruin is a trap Timid play, betting one dollar a game until you hit zero or the target a, is a random walk with two absorbing barriers. The probability of reaching a from a starting fortune x has the classic gambler's-ruin form. $$ f(x, a) = \\frac{(q/p)^{x} - 1}{(q/p)^{a} - 1} \\quad (p \\ne \\tfrac{1}{2}), \\qquad f(x, a) = \\frac{x}{a} \\quad (p = \\tfrac{1}{2}) $$ The letter q is one minus p, so q over p is the odds against you, a number bigger than one whenever the game is unfair. Because that ratio is raised to the power of your fortune, the success function is convex and hugs the floor: you need to climb almost all the way to the target before your odds of getting there become respectable. Work it at p equal to 0.45 with a target of 20 dollars, starting at 10, exactly half way. The ratio q over p is 1.222, so f is 1.222 to the tenth minus one, over 1.222 to the twentieth minus one, which is 6.44 over 54.4, about 0.118\. You start at the halfway mark and still reach the goal only twelve percent of the time. ![Timid-play success function for three win probabilities, convex and hugging the floor when the game is unfair](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_414-timid_success.png) The figure is the paper's own picture of this. For p above one half the curve bulges upward and you are likely to win from almost anywhere; for p below one half it sags into the floor and you are doomed across most of the range; at exactly one half it is the straight diagonal, where your chance of success is just your fortune divided by the target. The old article ["Kelly in Prediction Markets vs a Prop Challenge: Same Formula, Opposite Geometry"](https://aligrithm.com/kelly-in-prediction-markets-vs-a-prop-challenge-same-formula-opposite-geometry/) built the bridge from this gambler's-ruin barrier to a trading account with a hard floor. The lesson there and here is the same: a fixed barrier plus a negative drift is a machine for slow, near-certain ruin, and betting small only lets the machine run longer. ## Bold play, the doubling map, and an exotic staircase Bold play scales, so set the target to one dollar and let your fortune be a fraction x between zero and one. You bet the smaller of x and one minus x, driving straight for the goal. Condition on the first game and the success function F satisfies a functional equation instead of a linear difference equation. $$ F(x) = \\begin{cases} p\\,F(2x), & 0 \\le x \\le \\tfrac{1}{2} \\\\\[4pt\] p + q\\,F(2x - 1), & \\tfrac{1}{2} \\le x \\le 1 \\end{cases} $$ The recursion says: from below the midpoint you must win the next game (probability p) and then you are playing bold from twice your fortune; from above the midpoint you either win outright with probability p or, failing that, restart from 2x minus 1\. The doubling map, the operation that sends x to the fractional part of 2x, is doing the work, which is why bold play is secretly a dynamical system on the binary digits of your fortune. Work two values with p equal to 0.45: at x equal to one half, F is just p, so 0.45\. At x equal to one quarter, F is p times F of one half, which is p squared, 0.2025\. Compare that quarter-fortune bold number, 0.20, against the timid result, which from a quarter of a twenty-dollar target is a dismal few percent. The strangest part is what the success function looks like across all starting fortunes. Write W as the number whose binary digits are the complements of the game outcomes. $$ W = \\sum\_{j=1}^{\\infty} \\frac{1 - I\_j}{2^{j}}, \\qquad F(x) = \\mathbb{P}(W \\le x) $$ Each I-j is one if you win game j and zero if you lose, so W packs the entire sequence of outcomes into the bits of a single number between zero and one, and the bold success function is exactly the distribution function of W. When the game is fair, W is uniform and F is the plain diagonal. When p is not one half, W has a singular continuous distribution: F is continuous and strictly increasing, yet its derivative is zero almost everywhere. It is a devil's-staircase, all of its increase concentrated on a set of measure zero, the same kind of pathological object as the Cantor function, arising here from an honest casino question. ![Bold-play success function for several win probabilities, a singular staircase that bows below the diagonal for unfair games](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_414-bold_success_singular.png) The paper's plot shows F for a few values of p, and you can see the staircase texture and the way the unfair curves bow below the diagonal. The point for a bettor is not the fractal beauty; it is that even this optimal bold curve sits below break-even when the game is against you. Bold is the best you can do, and the best you can do is still a losing proposition. ## The flip: bold when unfair, timid when favorable What makes one strategy optimal? Siegrist gives a single sufficient condition. A strategy with success function V beats every other strategy if, at every fortune x and every allowable bet y, mixing a bet does not help. $$ p\\,V(x + y) + q\\,V(x - y) \\le V(x) $$ In words: the chance-weighted average of where a bet of size y could land you, winning to x plus y or losing to x minus y, must be no greater than staying put in value terms. This is a concavity-flavored test, and which strategy passes it depends entirely on the sign of the edge. When the game is favorable, p at least one half, the timid success function satisfies the condition and timid play is optimal; with a real edge you want maximum exposure to your advantage, so you bet as small and as often as the house allows and let the edge compound. When the game is unfair, p at most one half, the bold success function satisfies it instead and bold play is optimal; with a disadvantage you want minimum exposure, so you resolve the bet in as few games as possible. ![Bold versus timid probability of reaching the target in an unfair game, with bold dominating timid at every starting fortune](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_414-bold_vs_timid.png) The generated figure puts the two head to head in the unfair case at p equal to 0.45\. Bold sits above timid at every single starting fortune, and the gap is brutal near the middle: from half your target, bold reaches the goal 45 percent of the time while timid manages 12 percent. There is a final twist Siegrist proves and I will only flag: in the unfair case bold is not the unique optimum. An entire infinite family of higher-order bold strategies, built by rescaling the basic one, achieves the identical success probability. So "bet boldly" is really "resolve the game fast," and there are many ways to do that equally well. ## What this actually means for a trader Strip away the elegance and the practical warning is sharp. Bold play maximizes the probability of hitting a fixed target before ruin. It does not maximize growth, it does not maximize expected wealth, and it cheerfully accepts near-certain ruin in exchange for a better shot at the goal. That is the opposite objective from Kelly. Kelly, the subject of the old article ["Kelly Rejects Bad Trades Automatically: Why Overbetting Kills and Underbetting Doesn't,"](https://aligrithm.com/kelly-rejects-bad-trades-automatically-why-overbetting-kills-and-underbetting-doesnt/) maximizes long-run growth on a favorable edge and structurally refuses to risk total ruin. Bold play is what you reach for precisely when Kelly has nothing to offer, because your edge is zero or negative and there is no long run to grow into, only a target and a barrier. A funded-account challenge is this problem wearing a suit. You must reach a profit target before hitting a drawdown floor, and after fees, spread, and slippage your per-trade edge is often at or below break-even. The red-and-black theorem says your best chance in that spot is to concentrate risk, take a small number of large, decisive swings, not to grind a hundred tiny scalps that feed volume into a losing expectation. But read the theorem honestly. It tells you how to make the least-bad of a losing situation, not how to win. If your true edge is negative, bold play raises your pass probability from dismal to merely bad, and no bet-sizing rule invented can turn a house edge into money. The only durable fix is an actual edge, at which point the math flips and you should be betting timid anyway. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_414-visual-0.png) ## Where this connects This is the geometry underneath the Kelly pillar. The old article ["Kelly Rejects Bad Trades Automatically: Why Overbetting Kills and Underbetting Doesn't"](https://aligrithm.com/kelly-rejects-bad-trades-automatically-why-overbetting-kills-and-underbetting-doesnt/) showed Kelly sizing on a favorable edge to maximize growth; red-and-black is the mirror world where the edge is against you and the objective is a target, not growth, so the answer inverts to bold. The old article ["Kelly in Prediction Markets vs a Prop Challenge: Same Formula, Opposite Geometry"](https://aligrithm.com/kelly-in-prediction-markets-vs-a-prop-challenge-same-formula-opposite-geometry/) already carried the gambler's-ruin barrier into a funded-account setting; this article supplies the theorem that says what to do once you are trapped behind that barrier. The honest through-line is a warning about objectives. Before you copy a bet-sizing rule, check which quantity it maximizes. Kelly maximizes growth and protects against ruin. Bold play maximizes the probability of touching a target and embraces ruin as the price. Pick the wrong one for your situation and you will grind yourself out with tiny disciplined bets in a game where the only rational move was to swing once and get it over with, or you will blow up swinging in a game where patience would have compounded a real edge. ## KEY POINTS - In an unfair game, bold play, betting your whole fortune toward the target on each game, maximizes the probability of reaching a target before ruin. Timid grinding is provably the worst choice. - The strategy flips with the sign of your edge: bold when the game is against you, timid when it favors you. It is proven by a single optimality condition, p times V of x-plus-y plus q times V of x-minus-y at most V of x. - Every bet moves expected fortune by two-p-minus-one times the stake, so with a negative edge your total expected loss scales with total volume wagered. Minimizing exposure, not maximizing it, is the goal. - Timid play is gambler's ruin: the success function is convex and floor-hugging when unfair. Starting at half a 20-dollar target with p equal to 0.45, you reach the goal only 12 percent of the time. - Bold play scales and obeys a doubling-map functional equation. Its success function is a singular continuous staircase, continuous and strictly increasing yet with zero derivative almost everywhere, and it still sits below break-even in an unfair game. - From half your target at p equal to 0.45, bold reaches the goal 45 percent of the time versus timid's 12 percent, and bold dominates timid at every starting fortune. - Bold play maximizes hitting a target, not growth, and accepts near-certain ruin. It is the opposite objective from Kelly and is only the right tool when your edge is gone. It makes a losing game less lost, never won. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Convex Optimization - Stephen Boyd Lieven Vandenberghe (Amazon)](https://www.amazon.com/s?k=Convex+Optimization+Stephen+Boyd+Lieven+Vandenberghe&ref=aligrithm.com) - [An Introduction to Copulas - Roger Nelsen (Amazon)](https://www.amazon.com/s?k=An+Introduction+to+Copulas+Roger+Nelsen&ref=aligrithm.com) - [How to Gamble If You Must (how to gamble if you must)](https://scholar.google.com/scholar?q=How+to+Gamble+If+You+Must&ref=aligrithm.com) ### 5.48 An Open L4 Order Book: What Hyperliquid Data Unlocks URL: https://aligrithm.com/an-open-l4-order-book-what-hyperliquid-data-unlocks/ Last updated: 2026-07-27T13:25:29.000Z Academic microstructure research runs on a thin diet. Most papers you have read about limit order books were built on LOBSTER, a reconstruction of NASDAQ message data that is genuinely useful and also blind in specific ways: it shows you the orders that reached the book, not the ones that were rejected, and it never tells you who submitted them. Everything else is proprietary exchange data behind a negotiated contract you cannot share, which means nobody can check your results. Albers, Cucuringu, Howison, and Shestopaloff, out of Oxford, UCLA, and Queen Mary, went around the whole problem by running a non-validating node on Hyperliquid and dumping the raw blockchain into a structured research dataset. They put roughly 196 GB of it on Zenodo under an open license. The reason this matters is not that crypto is exciting. It is that a fully on-chain exchange has no private ledger. Every order submission, cancellation, modification, fill, and rejection is a public blockchain event, timestamped to the nanosecond and permanently tagged to the wallet that sent it. That gives you the three things conventional feeds strip out: the identity of every actor, the counterparty on both sides of every trade with their inventory before it, and the roughly 89% of order events that get rejected and never appear in any normal dataset. This piece walks through what is actually in the data, does the arithmetic on why it is big, and then argues about how much of it is a real edge versus a well-marketed sample. ## What "Level 4" actually means Start with the taxonomy, because Hyperliquid's marketing muddies it. The standard market-data ladder goes: Level 1 is best bid and offer, Level 2 is aggregated depth per price level, Level 3 is individual order events (this is LOBSTER). Hyperliquid calls its node feed "Level 4," which is a coined term, not an industry standard. Strip the branding and it is Level 3 plus four additions: a pseudonymous wallet address on every action, the full order specification (type, time-in-force, limit price, size), every failed attempt, and the identity plus pre-trade inventory of both counterparties on each fill. The node writes three streams. Order status records log every event in every order's life: accepted onto the book (open), filled, canceled, or rejected. Trade records log each fill with both wallet addresses, their positions before the trade, and order IDs that link back to the status stream. Raw book diffs log every visible change to the book as new, remove, or update. That third stream is the input to a reconstruction engine: fold the diffs forward in sequence and you rebuild the full book at any instant. The old article ["Reconstructing the Order Book from Incremental Deltas"](https://aligrithm.com/reconstructing-the-order-book-from-incremental-deltas/) laid out that fold, and the old article "LOB Reconstruction Engine" built the state machine for it. Hyperliquid's book diff stream is exactly the delta feed those pieces assumed, handed to you with a wallet ID stapled to every delta, which is the part LOBSTER can never give you. ## The 89% you never saw Here is the headline unlock, and it is a real one. The order status stream captures every attempt, not just the survivors. In the December 2025 sample, roughly 89% of all order status records are rejections, the most common being badAloPxRejected, a post-only order that would have crossed the spread and so was bounced by the matching engine to protect its post-only guarantee. Conventional data records none of this. The accept rate is small. $$ p\_{\\text{accept}} = \\frac{N\_{\\text{book}}}{N\_{\\text{status}}} = 1 - p\_{\\text{reject}} $$ The accept probability is the count of events that reached the book divided by the count of all order status events, which is one minus the rejection rate. Work it with the paper's daily figure. The node logged about 880 million order status events per day across BTC, ETH, and SOL. With a 0.89 rejection rate, only 880 million times 0.11, or roughly 97 million events a day, actually landed on the book. The other 783 million, the ones a LOBSTER-style feed would silently drop, are the new material. ![Of roughly 880 million daily order-status events across BTC, ETH, and SOL, only about 11% are accepted onto the book. Conventional Level 3 data captures that slice and drops the other 89% of rejected and failed attempts.](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_425-visibility_iceberg.png) Why care about orders that never traded and never even rested? Because they are behavior. The companion papers by the same authors show that a small cohort uses rejected post-only orders to grab queue priority right after the spread widens, and that this rejected flow is significantly correlated with future returns, which is a direct shot at the textbook idea that only executed order flow has price impact. The old article "The MM Feature Catalog: Arrival, Cancellation, Update Rates" defined arrival, cancellation, and update rates as the core market-making features. This dataset lets you compute all three per wallet, and adds a fourth the catalog could not: rejection rate, the frequency and pattern of a trader's failed attempts, which is often a cleaner fingerprint of intent than what they managed to execute. ## The plumbing: a node, 100 GB a day, and a 54-byte record None of this is free to handle, and the size is the first reason most people will not replicate it. In native JSON each order status record runs about 300 to 450 bytes. Multiply that out. $$ V\_{\\text{day}} = N\_{\\text{events}} \\times b\_{\\text{record}} $$ Daily volume is the event count times the bytes per record. At 880 million events and 400 bytes, that is about 352 GB per day in raw JSON for three coins, which matches the paper's "exceeding 300 GB" and its note of roughly 100 GB per day of raw node logs before expansion. That is a firehose, and a month of it is unusable for most academic infrastructure. So they re-encoded. $$ r = 1 - \\frac{b\_{\\text{binary}}}{b\_{\\text{json}}} $$ The size reduction is one minus the binary record size over the JSON record size. Packing each record into a fixed 54-byte binary layout (wallet addresses mapped to 32-bit integer IDs, categorical fields to small integer codes, prices and sizes to fixed-point integers) gives one minus 54 over 400, or 0.865, an 86.5% cut, squarely in the paper's stated 85 to 88% range. That is what drops a 352 GB day to about 47 GB, and what let the whole sample fit under Zenodo's 200 GB ceiling. The re-encoding was not a nicety; it was the difference between a public dataset and a private one. ![The released archives, about 196 GB total. For every coin, the rejected-order archive is roughly two to three times the size of the accepted-order archive, so the invisible flow is literally the bigger half of the file.](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_425-zenodo_archives.png) The archive breakdown makes the point physical: for BTC the accepted-order file is 19 GB and the rejected-order file is 46 GB. The stuff no other dataset carries is the larger share of the download. ## What it costs to play, and why HFT lives here The fee schedule explains why a cancel-heavy, rejection-heavy market exists at all. Hyperliquid charges maker-taker fees with a baseline of 1.5 basis points for makers and 4.5 for takers, and the most active makers get down to a 0.3 basis point rebate against 1.44 for takers. Compare the two ways to turn a position around. $$ C\_{\\text{taker}} = 2 \\times f\_{\\text{taker}}, \\qquad C\_{\\text{maker}} = 2 \\times f\_{\\text{maker}} $$ A round trip costs twice the per-side fee. Crossing the spread both ways at the baseline taker rate costs 2 times 4.5, or 9 basis points, before you have earned a single tick. Providing both ways as a top-tier maker earns 2 times 0.3, or 0.6 basis points of rebate, a swing of nearly 10 basis points purely from resting instead of crossing. That is the entire economic case for quoting rather than taking, and it is the same logic the maker-side execution pieces in this pillar rest on. The other half of the case is structural: order submissions and cancellations on Hyperliquid are gasless. On any normal blockchain a per-message network fee would make the high cancel-to-trade ratios of market making ruinous. Remove that fee and the 89% rejection rate stops being a bug and becomes the expected texture of a viable HFT venue. ## What the data does not settle Now the skepticism, because the paper is careful and the resharers usually will not be. First, the decentralization is thin. Hyperliquid's active validator set is 21 nodes chosen by stake, which is closer to a consortium than to Bitcoin's thousands of participants. The openness that makes this data exist is a property of 21 operators agreeing to run the chain, not a law of nature. Second, the intra-block ordering is less exploitable than it looks. Within each 100 to 200 millisecond block the engine processes passive orders first, cancellations second, and takers last, which sounds like a clean edge for maker cancellation. But which block your message lands in is set by a pseudorandomly selected proposer with its own local view, so two messages sent together can land blocks apart, and a cancellation meant to pull a stale quote can arrive too late or, by luck, too early. The hierarchy is real; your ability to trade on it is bounded by routing randomness the paper is explicit about. Third, know what the sample is before you generalize from it. The public release is one month of order status and book diffs (December 2025) for three perpetual contracts, plus four months of trades, all from a single venue, all crypto perps, all pseudonymous rather than identified. One wallet is not one human; a single desk can spin up unlimited wallets, and fee tiers even push them to consolidate, so wallet-level population studies inherit an identity-resolution problem the data cannot close on its own. The full firehose, reconstructed snapshots, and the raw replica command logs are only available on request, so the open 196 GB is a generous sample, not the whole thing. The unlock is genuine: rejected orders and counterparty inventory at nanosecond resolution are new, open, and checkable. Just do not confuse an open window into one young exchange with a general law of how markets work. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_425-visual-0.png) ## KEY POINTS - Hyperliquid is a fully on-chain limit-order-book exchange, so its entire message history is a public blockchain feed. A non-validating node (16 vCPU, 64 GB RAM, a binary, three logging flags) captures every order event with no special access. - "Level 4" is Hyperliquid's coined label, not a standard. In practice it is Level 3 plus a pseudonymous wallet ID on every action, full order parameters, every failed attempt, and both counterparties' identities and pre-trade inventory on each fill, all at nanosecond precision. - The real unlock is rejected orders. About 89% of the roughly 880 million daily order-status events (BTC, ETH, SOL) are rejections, so only about 97 million a day reach the book. Conventional data drops the other 783 million. - Rejected flow is not noise. The authors' companion work shows traders use rejected post-only orders to grab queue priority after spread widening, and that this never-executed flow predicts future returns, challenging the idea that only filled orders have price impact. - The engineering is the barrier: raw JSON runs over 300 GB per day. A fixed 54-byte binary record cuts size by about 86.5% (one minus 54 over 400), which is what let 196 GB fit on Zenodo. - Fees explain the venue: a baseline taker round trip costs 9 basis points while a top-tier maker earns a 0.6 basis point rebate, and order submission and cancellation are gasless, which is why cancel-heavy, rejection-heavy market making is structurally viable here. - Be skeptical of scope. Only 21 validators, intra-block priority bounded by routing randomness, one month, three coins, one venue, crypto perps, and pseudonymous wallets that do not map cleanly to traders. It is an open sample of one young exchange, not a general market law. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [The Art of Currency Trading - Brent Donnelly (Amazon)](https://www.amazon.com/s?k=The+Art+of+Currency+Trading+Brent+Donnelly&ref=aligrithm.com) - [An Open Book: Level 4 Order Book Data from the Hyperliquid Exchange (ssrn-6465720)](https://papers.ssrn.com/sol3/papers.cfm?abstract%5Fid=6465720&ref=aligrithm.com) ### 9.36 How Manipulatable Are Prediction Markets? The Field Experiment URL: https://aligrithm.com/how-manipulatable-are-prediction-markets-the-field-experiment/ Last updated: 2026-07-27T13:26:22.000Z Two economists walked onto Manifold Markets and bought their way into 817 separate prediction markets, on purpose, to move the prices. In each one they either pushed the yes price up by 5 percentage points, pushed it down by 5, or left it alone as a control. Then they walked away and watched. They logged the price every hour for 30 days, took one more snapshot at 60 days, and ended up with about 600,000 price observations. The question was simple and old: does a manipulative trade get washed out fast by everyone else, or does the shove stick? The textbook answer, going back to Wolfers and Zitzewitz, is that manipulation has no discernible effect except during a short transition, because rational traders pounce on the mispricing and undo it. Itzhak Rasooly and Roberto Rozzi ran the first large field experiment built to test that claim, and the textbook lost. The shove stuck. Sixty days later, the markets they had pushed up were still trading higher than the markets they had pushed down, and the gap was still significant at better than the 1% level. Prices did revert, but only part way, and the reversion ran out of steam after the first week. If you trade the price a prediction market shows, this is the risk the old article ["The Six Ways to Lose Money on Polymarket"](https://aligrithm.com/the-six-ways-to-lose-money-on-polymarket/) was pointing at: the number on the screen is movable, and it stays moved longer than the efficient-markets story admits. ## The experiment, and why the design is the point The clever part is that the randomization happens at the market level, not the trader level. Each of the 817 markets was assigned by coin flip to yes, no, or control, so the difference in average price between the yes group and the no group has a clean causal reading. No storytelling about who traded and why. You shocked a random half up and a random half down, and any surviving gap between them is your treatment effect. The setup was disciplined in ways most trading "studies" are not, and the discipline is worth copying. They pre-registered the whole analysis plan in December 2023 before placing a single bet, so the hypotheses and the regressions were locked in advance. They set the sample size from a power calculation: 849 markets to get 90% power to detect a 3 point gap after a week, which after dropping markets that broke the inclusion rules left 817\. They threw out markets with fewer than 10 traders, markets too expensive to move 5 points on their budget, brand-new markets, and any market closely related to another already in the sample, so one bet could not leak into another through arbitrage. This is a backtest done as an actual experiment, the standard the old article ["The Six Ways to Lose Money on Polymarket"](https://aligrithm.com/the-six-ways-to-lose-money-on-polymarket/) keeps demanding and rarely gets. ## The pricing engine you are betting against To understand why a shove might stick, you have to understand how the price is even set. Manifold runs an automated market maker on the constant product rule. The maker holds a reserve of yes shares y and no shares n. When you spend x currency to buy yes, the maker first turns your x into x yes shares and x no shares, so reserves jump to (y+x, n+x), then hands you the number of yes shares q that keeps the product of its reserves where it was. $$ (y + x - q)(n + x) = y\\,n $$ Read it as a conservation law. Before your trade the reserves multiplied to y times n, and after your trade the maker keeps that product constant by choosing how many yes shares q to give you. Work it: start with balanced reserves of y = 1000 and n = 1000, so the product is 1,000,000\. Spend x = 40 to buy yes. Reserves become (1040, 1040) before the maker pays you, and it needs (1040 minus q) times 1040 to equal 1,000,000, so 1040 minus q equals 961.54, which means q equals 78.46 yes shares. Your $40 did not buy 80 shares at 50 cents. It bought 78.46, because the price climbed as you ate the reserve. The price the market reports is the marginal price, the cost of the next tiny share, and it comes straight out of the reserve ratio. $$ p = \\frac{n}{n + y} $$ This is the implied probability the market is quoting. With reserves of (1000, 1000) it is 1000 divided by 2000, which is 0.50, a coin flip. After the $40 yes buy above, the yes reserve dropped to 961.54 and the no reserve rose to 1040, so the new marginal price is 1040 divided by (1040 plus 961.54), which is 0.5196\. A $40 bet moved the quoted probability from 50.0% to about 52.0%. Push harder and it moves more, but never linearly, because the reserve fights back. This marginal price also decides who does what: a trader whose personal belief sits above the current p buys yes, a trader whose belief sits below it buys no, and a trader who agrees with p sits out. Manifold's actual rule, Maniswap, generalizes this by holding y-to-the-p times n-to-the-(1 minus p) constant so markets can start off balance, but at a 50% start it collapses back to plain constant product. ## Why the shove should snap back, in theory Rasooly and Rozzi build a model where traders are risk-averse, hold different beliefs, and understand that their own trades move the price. The load-bearing assumption is decreasing absolute risk aversion, meaning a person puts more dollars at risk as they get wealthier, which is both realistic and what makes the math tractable. From it comes the reversion logic. Suppose a manipulator buys yes and drags the marginal price up from p to p plus some jump. Traders who were comfortable holding yes now find yes too expensive relative to their belief and trim it; traders sitting in the middle flip from yes to no; traders below the old price pile further into no. Every one of those moves pushes the price back down toward where it started. So the model says reversion is real, but it does not promise the reversion is complete, and that is the whole fight. Simulating a baseline market with 11 traders, balanced reserves, and beliefs spread evenly, a shock from 0.50 to 0.55 reverts only about 40% of the way even after prices settle. Three knobs change how much sticks. More traders means more wealth aimed at correcting the error, so faster and fuller reversion. More "learning", where traders treat the market price itself as evidence and update their own beliefs toward it, makes the shove stickier; at full learning the manipulation never reverts at all, because the fake price teaches everyone the fake is true. More prior agreement among traders makes a market hard to move in the first place, since a crowd that already agrees the answer is 0.50 will fight anything else. ![Simulated price paths after a manipulation shock, under the baseline and under variations in the number of traders, the learning rate, and prior agreement among traders](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_413-reversion_simulations.png) ## What actually happened The raw picture is the cleanest evidence. Before the trade, all three groups sit together near 0.50\. At hour zero the yes group jumps up 5 points and the no group drops 5, opening a 10 point gap. Then the gap narrows, but slowly, and it never closes. ![Average price in the yes, no, and control groups from 24 hours before the trade through 168 hours after, showing the 10-point gap opening at the trade and only partly closing over a week](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_413-manipulation_persists.png) They put numbers on it with a regression estimated hour by hour. $$ p\_{t,i} = \\beta\_0 + \\beta\_1 \\, \\mathbb{1}\_{Y,i} + \\beta\_2 \\, \\mathbb{1}\_{C,i} + \\beta\_3 \\, p\_{-1,i} + u\_i $$ Here p at time t in market i is explained by a yes dummy, a control dummy, the pre-bet price p-minus-1 (thrown in to soak up baseline noise and buy power), and an error. The no group is the baseline, so the coefficient beta-1 is exactly the yes-minus-no price gap at time t. Trace it over time and the story is quantitative: the gap opens near 10 points, stands at about 7.3 points after one week, so only about a quarter of the shock washed out in the week everyone said would erase it. Push the horizon out and reversion keeps crawling: 25% gone after the first week, but just another 6% over the following three weeks. At 60 days the gap is still 5.9 points among markets that had not yet resolved, which is 41% reversion and 59% still standing. The standard error grew as expected, from 0.004 at a week to 0.014 at 60 days, and the effect stayed significant at p below 0.01 the whole way. A quick symmetry check, testing whether the yes push and no push were mirror images, could not be rejected (p = 0.124), so the effect is not an artifact of one direction. ## Reversion is front-loaded, and some markets barely move The averages hide a pattern the model predicted: how much of the shove survives depends on the market. Splitting the sample at the median of each characteristic and measuring one-week reversion, the thin, quiet, isolated markets are the soft targets. ![Grouped bars comparing one-week reversion for markets with versus without an external source, with more versus fewer traders, and with higher versus lower volume](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_413-reversion_by_market_type.png) Markets whose question also lived on Metaculus reverted 47%, against 23% for markets with no outside price to anchor on, which fits the learning story: when traders have an external probability to check against, they lean less on the manipulated market price and the shove teaches them nothing. Markets with above-median traders reverted 33% against 17% for the thin ones. Above-median volume reverted 31% against 19%. Push into the top quarter by trader count, markets averaging 56 traders and 349 final trades, and reversion climbs to 37%, but even there the effect stays significant. The soft caveat: these market traits were not randomly assigned, only the yes-no-control label was, so the heterogeneity is suggestive rather than airtight causal. The direction is exactly what the model called. ## Real money did not fix it The obvious objection is that Manifold runs on Mana, a play currency you cannot cash out, only donate to charity, so maybe traders just do not fight hard enough to correct prices. Manifold then launched Sweepcash markets, redeemable near 1-to-1 with dollars, and the authors reran the experiment on 110 of them, shifting the shock to 4 points to avoid tipping off traders who had seen the working paper. Same two findings. The yes group stayed above the no group on every recorded day, and reversion showed up again and slowed over time. Matching markets by activity, Mana reverted 51% and Sweepcash 43%, close enough that swapping play money for real money did not rescue the efficient-price story. They spent $15.72 per trade on average, about $1,729 total, to move a set of real-money markets and make it stick for the days the experiment survived before Manifold pulled Sweepcash. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_413-visual-0.png) ## Where this connects The old article ["The Dutch Book Theorem, Plainly, and Why Polymarket Will Always Have Arbitrage"](https://aligrithm.com/the-dutch-book-theorem-plainly-and-why-polymarket-will-always-have-arbitrage/) argued that incoherence is a permanent feature of these markets, not a bug competition erases. This experiment is the companion fact on the price itself. If a random 5 point shove can persist for two months, then the "wisdom of the crowd" number a prediction market advertises is not a clean probability you can lift and trade. It is a price that a modestly funded actor can bend and hold, most easily in the thin, unwatched markets that make up the long tail of every platform. That is a direct entry in the ledger the old article ["The Six Ways to Lose Money on Polymarket"](https://aligrithm.com/the-six-ways-to-lose-money-on-polymarket/) keeps: before you treat a market price as truth, ask how many traders stand behind it, whether an outside source pins it, and how much it would cost someone to move it. On Manifold, moving it 5 points and keeping it there for a week cost less than the price of dinner, and the crowd took two months to mostly not fix it. ## KEY POINTS - The study is a pre-registered field experiment: 817 Manifold markets randomly assigned to a yes shock (+5 points), a no shock (-5 points), or control, with hourly prices logged for 30 days and a snapshot at 60, about 600,000 observations. - The manipulations persisted. The yes-minus-no price gap opened at 10 points, still sat at 7.3 after a week, and was 5.9 points (41% reverted, 59% surviving) among unresolved markets at 60 days, significant at p below 0.01 throughout. - Reversion is real but partial and front-loaded: about 25% in week one, only another 6% over the next three weeks. The old efficient-price claim that manipulation washes out fast does not hold here. - Prices are set by a constant-product automated market maker, keeping (y+x-q)(n+x) equal to y times n, with the quoted probability equal to n divided by (n+y). Pricing is nonlinear, so a bet moves the price less the deeper it eats the reserve. - The model, built on decreasing absolute risk aversion, predicts reversion because a higher price makes existing yes-holders trim and pessimists pile into no. It also predicts stickier manipulation when traders learn from the price itself. - Harder-to-manipulate markets are the ones with an external anchor (on Metaculus, 47% reversion versus 23%), more traders (33% versus 17%), and higher volume (31% versus 19%). - Rerunning on real-money Sweepcash markets gave the same result, so play money was not the reason prices failed to correct. Moving a market 5 points and holding it for a week cost only a few dollars per trade. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Convex Optimization - Stephen Boyd Lieven Vandenberghe (Amazon)](https://www.amazon.com/s?k=Convex+Optimization+Stephen+Boyd+Lieven+Vandenberghe&ref=aligrithm.com) - [An Introduction to Copulas - Roger Nelsen (Amazon)](https://www.amazon.com/s?k=An+Introduction+to+Copulas+Roger+Nelsen&ref=aligrithm.com) - [How manipulable are prediction markets?\* (how manipulatable are prediction markets)](https://scholar.google.com/scholar?q=How+manipulable+are+prediction+markets%3F%2A&ref=aligrithm.com) ### 9.35 Kelly When Price ≠ Probability: Why the Gap Persists URL: https://aligrithm.com/kelly-when-price-probability-why-the-gap-persists/ Last updated: 2026-07-27T13:26:22.000Z Open Polymarket's own help page and it tells you the price is the probability. "Prices (odds) on Polymarket represent the current probability of an event occurring." That sentence is wrong, and it is wrong for a reason that has nothing to do with manipulation, wash trading, or dumb money. It is wrong because the people setting the price are sizing their bets with logarithmic utility, and log-utility does not clear a market at the average belief. It clears it somewhere else. A short 2024 paper by Bernhard Meister works out exactly where, and the answer is that the gap between price and probability is a structural feature of how rational bettors allocate capital, not a bug you can arbitrage away. This matters because the whole appeal of prediction markets rests on the claim that the price is a forecast. If a contract trades at 0.30, the pitch is that the event has a thirty percent chance. Meister's point is that a room full of Kelly bettors will push that price to 0.30 even when their capital-weighted average belief is 0.45\. The price is a clearing condition, not a poll. This article walks through the optimal fraction that produces the gap, the two-investor toy that shows how wide it gets, and the finite-horizon result that should worry anyone trading these: getting the probability wrong costs you more than getting the bet size wrong. ## The fraction you should bet is the gap, not the belief Start with one contract. It costs p to buy, pays one dollar if the event happens, zero otherwise. You believe the true probability is q. You cannot borrow and you cannot go naked short. A log-utility maximizer picks the fraction f of bankroll that maximizes expected log wealth. $$ U(q, p, f) = (1-q)\\log(1-f) + q\\log\\!\\left(1 + f\\,\\frac{1-p}{p}\\right) $$ The first term is the log of what you keep if the event fails, weighted by your belief it fails. The second is the log of what you hold if it happens, where the payout multiplier is one minus p over p, the odds the market is offering. Maximize over f and the answer collapses to something clean. $$ f = q - p\\,\\frac{1-q}{1-p} = \\frac{Q - P}{1 + Q}, \\qquad P = \\frac{p}{1-p}, \\quad Q = \\frac{q}{1-q} $$ Read the right-hand form. Capital-P is the market's odds, capital-Q is your odds. You bet a fraction proportional to the gap between your odds and the market's odds. Work a number: the contract trades at p equal to 0.40, you believe q equal to 0.60\. Then P is 0.40 over 0.60, about 0.667, and Q is 0.60 over 0.40, which is 1.5\. The fraction is 1.5 minus 0.667, over 1 plus 1.5, which is 0.833 over 2.5, about 0.333\. You put a third of your bankroll long. Now set your belief equal to the price, q equal to p. Then Q equals P, the numerator is zero, and f is zero. You bet nothing. This is the whole thesis in one line. You do not bet because you think the event is likely. You bet because your probability differs from the price, and you bet the gap. The old article ["Kelly Rejects Bad Trades Automatically: Why Overbetting Kills and Underbetting Doesn't"](https://aligrithm.com/kelly-rejects-bad-trades-automatically-why-overbetting-kills-and-underbetting-doesnt/) made the same point from the stock-market side: Kelly hands you a zero when the edge is gone. Here the edge is literally the distance between q and p. ![Optimal Kelly fraction as a function of belief q, for three market prices, crossing zero exactly where belief equals price](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_412-kelly_fraction.png) The figure draws f against your belief for three fixed prices. Each line crosses zero exactly at its price and tilts positive above, negative below. When your belief sits under the price you want to short the contract, and the no-short rule in real binary markets is why platforms list a mirror contract instead. The slope also changes with the price, and that asymmetry is what pries price and probability apart. ## Why the crowd's average belief is not the price Meister defines the mean belief honestly: the capital-weighted average of everyone's subjective probability. Investor i brings capital C-i and belief q-i, so the mean belief is the sum of C-i times q-i. Their position in the market is C-i times their fraction f-i. The market clears when those positions net to zero. $$ \\sum\_{i=1}^{N} C\_i\\, f\_i(p, q\_i) = 0 $$ In plain terms: the price p is whatever value makes all the Kelly bets cancel. Longs equal shorts in capital terms. Nothing in that condition forces p to equal the capital-weighted mean belief, because each f-i is a nonlinear, asymmetric function of p. To see how far apart they can drift, take the simplest possible market: two investors, one of them dead certain, the other with belief q. Solve the clearing condition for the case where the confident investor is certain the event will not happen, and the capital-weighted expectation lands at $$ E\_-(q\_1^-, p) = \\frac{1-p}{\\,q\_1^- - 2p + 1\\,}\\, q\_1^- $$ Work it. Say the price is p equal to 0.30, and the second investor genuinely believes q equal to 0.70\. Plug in: the numerator is 0.70 times 0.70, which is 0.49; the denominator is 0.70 minus 0.60 plus 1, which is 1.10\. The mean belief is 0.445\. So the capital-weighted crowd believes 0.445, but the market prints 0.30\. That is a fourteen-point gap with only two rational participants and no manipulation. Meister shows the mean belief in this setup can range across the whole interval from 0 to one-half whenever the price sits below one-half. Flip the confident investor to certain-yes and the algebra gives the mirror result, with the expectation pinned back at p. The lesson is not the exact number, it is that a market can clear at a price nowhere near what its own participants, on average and weighted by their money, actually believe. This is a different failure than the incoherence the old article ["The Dutch Book Theorem, Plainly, and Why Polymarket Will Always Have Arbitrage"](https://aligrithm.com/the-dutch-book-theorem-plainly-and-why-polymarket-will-always-have-arbitrage/) described. There the prices across related contracts violated the probability axioms and left a standing arbitrage. Here a single price is internally fine, it just is not the average belief, and no arbitrage closes the gap because every bettor is already at their optimum. You cannot trade against rationality. ## Bending the payout to pull liquidity If the gap is a product of the payout structure, you can change the payout structure. Meister floats a one-parameter deformation: raise the market's odds multiplier to a power alpha. $$ U\_\\alpha(q, p, f) = (1-q)\\log(1-f) + q\\log\\!\\left(1 + f\\left(\\frac{p}{1-p}\\right)^{\\!\\alpha}\\right) $$ At alpha equal to one you recover the standard contract. Push alpha below one and the payout near the extremes gets compressed, so a high-probability bet with a normally tiny relative payout becomes more attractive; a bettor who would otherwise ignore a contract priced at 0.95 now has a reason to size into it. Push alpha above one and you do the opposite. The practical idea is that an exchange sitting on a lopsided market, price hugging a boundary where liquidity dries up, could tune alpha to make the unloved side worth trading and pull depth back in. Be skeptical about how far this goes, and Meister is. He calls it explicitly not a panacea: the moment the market flips toward the other boundary, the same deformation that attracted liquidity now repels it, and depth drains from the side you just starved. He also notes the full effect on the price-belief gap requires working through several cases he defers to a later paper. So treat alpha as a suggestive knob, not a solved market-design fix. It shows the gap is a function of contract terms, which is the useful insight, without pretending anyone has found the right terms. ## Finite time is where the theory bites Kelly's guarantee is asymptotic. Over infinite rounds the log-optimal bettor beats every other strategy with probability one. Nobody trades infinite rounds. Meister models the finite case as a biased coin flipped N times, k of them favorable, and asks how likely you are to fall short of a growth target. The tail probability of too few favorable steps is bounded by the Chernoff inequality, whose exponent is the Kullback-Leibler divergence. $$ F(k, N, p) \\le e^{-N\\,D\\!\\left(\\frac{k}{N}\\,\\middle\\|\\,p\\right)}, \\qquad D\\!\\left(\\frac{k}{N}\\,\\middle\\|\\,p\\right) = \\frac{k}{N}\\log\\frac{k}{pN} + \\left(1-\\frac{k}{N}\\right)\\log\\frac{1-\\frac{k}{N}}{1-p} $$ The left side is the probability that you get k or fewer favorable steps out of N. The bound says that probability decays exponentially in N, at a rate set by the KL divergence D between the fraction you need, k over N, and the true bias p. Work it: a favorable coin with p equal to 0.55, over N equal to 100 flips, and you want to know the odds of ending underwater with 45 or fewer wins, so k over N is 0.45\. The divergence is 0.45 times log of 0.45 over 0.55, plus 0.55 times log of 0.55 over 0.45, which comes to about 0.020\. The bound is e to the minus 100 times 0.020, which is e to the minus 2.0, about 0.135\. Even with a real edge, you have roughly a one-in-seven chance of a hundred-flip stretch leaving you behind. That is the finite-horizon reality the asymptotic theorem hides. ## The asymmetry that should change how you trade Now the result worth the whole paper. Meister perturbs the growth calculation two ways and compares the leading terms. First, hold your bet size fixed and get the probability wrong by a small amount epsilon. The change in the divergence, which controls your achievable growth, is linear in epsilon. $$ D\\!\\left(\\frac{k}{N}\\,\\middle\\|\\,p+\\epsilon\\right) - D\\!\\left(\\frac{k}{N}\\,\\middle\\|\\,p\\right) = \\frac{p - \\frac{k}{N}}{p(1-p)}\\,\\epsilon + O(\\epsilon^2) $$ Second, hold the probability fixed and get the bet fraction wrong by epsilon, around the optimum f-star equal to 2p minus 1\. The change in the growth rate is quadratic. $$ U(p,\\, 2p-1+\\epsilon) - U(p,\\, 2p-1) = -\\frac{1}{4(1-p)p}\\,\\epsilon^2 + O(\\epsilon^3) $$ Put the two side by side. A probability error hits you at first order; a bet-size error only at second order. Work the numbers with p equal to 0.55 and k over N equal to 0.45\. The bias slope is 0.55 minus 0.45, over 0.55 times 0.45, which is 0.10 over 0.2475, about 0.40\. So an epsilon of 0.02 in your probability estimate costs about 0.008 per step, linearly. The bet-size penalty coefficient is one over four times 0.55 times 0.45, about 1.01, so the same 0.02 error in fraction costs only 1.01 times 0.0004, about 0.0004 per step. The probability mistake is roughly twenty times more expensive at that scale. ![Two penalty curves versus error size: bias error rises linearly and steeply, bet-size error rises only quadratically and stays flat near zero](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_412-bias_vs_fraction_penalty.png) The figure makes the ranking visual. Near zero error the quadratic bet-size curve is almost flat while the linear bias curve climbs immediately. This is the same flat-topped growth curve the old article "The Overbet Runs Through Everything" traced across shrinkage, asymmetry, and anti-patterns: the penalty for missing the Kelly fraction is quadratic, so small sizing errors barely register. ![Log-growth rate versus fraction wagered, showing a flat top at the Kelly optimum and a steep decline into overbetting](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_412-growth_penalty.png) That flat top is exactly why practitioners bet half-Kelly and sleep fine: giving up a little size costs almost nothing in growth but slashes variance. Meister's contribution is to put the other error on the same axes and show it is worse. Fine-tuning your stake is second-order comfort. The first-order danger is your probability estimate, and in a prediction market that estimate is what you are supposedly better at than the crowd. If you are not, the market's price, gap and all, will take your money at linear speed while you fuss over bet sizing at quadratic speed. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_412-visual-0.png) ## Where this connects This paper is the bridge between the Kelly pillar and the prediction-market pillar. The optimal fraction f equal to the gap over one plus your odds is the same Kelly logic the old article ["Kelly Rejects Bad Trades Automatically: Why Overbetting Kills and Underbetting Doesn't"](https://aligrithm.com/kelly-rejects-bad-trades-automatically-why-overbetting-kills-and-underbetting-doesnt/) laid out, specialized to a contract whose price is bounded above as well as below. The linear-versus-quadratic asymmetry is the growth-penalty geometry from the old article "The Overbet Runs Through Everything," now carrying a second lesson: the overbet penalty is quadratic, but the probability-estimate penalty is linear, so estimation error dominates sizing error. The practical takeaway is a warning about reading prices. A prediction-market price is a capital-weighted clearing point for log-utility bettors, not a probability, and the gap does not vanish because it is nobody's mistake. If you are going to trade these, spend your effort on the probability, because that is the error the math punishes first and hardest, and treat any exchange's claim that "price equals probability" as marketing rather than mathematics. ## KEY POINTS - A prediction-market price is not the probability. It is the point where capital-weighted Kelly bets cancel, and log-utility does not clear at the average belief. - You bet the gap, not the belief. The optimal fraction is f equals Q minus P over one plus Q, where P and Q are the market's and your odds. When your belief equals the price, you bet exactly zero. - The price-belief gap is structural, not an arbitrage. A two-investor toy market with one certain participant prints a price of 0.30 while the capital-weighted mean belief is 0.445, and no trade closes it because everyone is already optimal. - Bending the payout with a power alpha can pull liquidity toward an unloved boundary, but it is explicitly not a panacea: flip the market and the same knob drains depth from the other side. - Finite horizons kill the Kelly guarantee's comfort. Even a real edge, p equal to 0.55 over 100 flips, leaves roughly a one-in-seven chance of ending behind, bounded by a Chernoff exponent equal to the KL divergence. - The asymmetry that matters: misjudging the probability hurts growth at linear order, mis-sizing the bet only at quadratic order. Getting the probability wrong is roughly twenty times more expensive than the same-sized bet error in the worked case. - Trade accordingly. Half-Kelly sizing is cheap insurance because the growth top is flat, but your probability estimate is where the money is won or lost. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Convex Optimization - Stephen Boyd Lieven Vandenberghe (Amazon)](https://www.amazon.com/s?k=Convex+Optimization+Stephen+Boyd+Lieven+Vandenberghe&ref=aligrithm.com) - [An Introduction to Copulas - Roger Nelsen (Amazon)](https://www.amazon.com/s?k=An+Introduction+to+Copulas+Roger+Nelsen&ref=aligrithm.com) - [Application of the Kelly Criterion to Prediction Markets (application of kelly critereon in prediction markets)](https://scholar.google.com/scholar?q=Application+of+the+Kelly+Criterion+to+Prediction+Markets&ref=aligrithm.com) ### 9.34 A Black-Scholes for Beliefs: Logit Jump-Diffusion and Tradable Belief-Vol URL: https://aligrithm.com/a-black-scholes-for-beliefs-logit-jump-diffusion-and-tradable-belief-vol/ Last updated: 2026-07-27T13:26:21.000Z Options got a common language in 1973\. Before Black-Scholes, a dealer quoting a call had to argue about the whole distribution of the stock at expiry. After it, everyone agreed to argue about one number, implied volatility, and hedge the rest with Greeks. The model was never "true." It coordinated a market around a small set of state variables, and that coordination is what let the options business scale into a derivative stack worth trillions. Prediction markets have no such language. Polymarket, Kalshi and their cousins trade a contract that pays a dollar if an event happens and nothing if it doesn't, and the price sits in the open interval between zero and one so you can read it as a probability. But there is no shared model for how that probability should move through time, gap on news, or co-move with a related market. A recent paper from the Daedalus Research Team proposes to borrow the Black-Scholes trick wholesale: model the log-odds of the price as a jump-diffusion, force the price to be a martingale so the drift is not a free parameter, and whatever is left over becomes the tradable risk. This article walks through that kernel, the derivative layer it implies, and why the empirical case for it is thinner than the theory. ## Trade the log-odds, not the probability A probability is a bad state variable. It lives between zero and one, so a naive random walk will eventually push it through a boundary where it makes no sense, and its volatility has to shrink to nothing near the edges or the process escapes. The fix is the same one a softmax layer uses. Map the probability to its log-odds, run your dynamics on the unbounded line, and map back with a sigmoid. $$ x\_t = \\operatorname{logit}(p\_t) = \\log \\frac{p\_t}{1-p\_t}, \\qquad p\_t = S(x\_t) = \\frac{1}{1+e^{-x\_t}} $$ The variable x is the log-odds, free to wander anywhere on the real line, and S is the logistic squashing function that pulls it back into a valid probability. Work a number: a price of p equal to 0.6 is x equal to log of 0.6 over 0.4, which is log 1.5, about 0.405\. A price of 0.5 is x equal to zero. Push x out to 3 and p is about 0.95; push it to minus 3 and p is about 0.05\. The same move in x buys you less and less probability as you approach the boundary, which is exactly the behavior you want. If this map feels familiar it should. The old article ["If You've Used a Softmax Layer, You Already Know the Price Formula"](https://aligrithm.com/if-youve-used-a-softmax-layer-you-already-know-the-price-formula/) showed that the logistic and softmax are the same object the market maker uses to turn an unbounded score into normalized prices. This paper is running that map in reverse and then putting a stochastic process on the score. ## The kernel is a jump-diffusion in log-odds With x on the real line you can use the standard machinery. The paper writes belief dynamics as an Itô-Lévy process with three pieces: a drift, a diffusion, and a jump term. $$ dx\_t = \\mu(t, x\_t)\\, dt + \\sigma\_b(t, x\_t)\\, dW\_t + \\int\_{\\mathbb{R}} z\\, \\tilde{N}(dt, dz) $$ Read the three pieces. The drift mu is the slow, predictable pull. The middle term is a Brownian motion scaled by sigma-b, the belief volatility, which is how fast log-odds wobble on ordinary order flow. The last term is the jump: a compensated random measure that lets the log-odds lurch by an amount z when news lands, rather than crawl there continuously. That third piece is the honest part of the model, because event prices really do gap. A court ruling drops, a candidate concedes, an economic print surprises, and the price teleports. ![A simulated belief path: the traded probability diffuses smoothly and then jumps discretely at two news events](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_411-belief_path.png) The figure is one simulated path. Between the dashed lines the probability grinds around on diffusion, and at each dashed line a jump snaps it to a new level in a single step. A market maker warehousing this contract is not mainly exposed to where the price ends up. They are exposed to the path: the wobble around the middle, and the gaps on announcements. That is the risk the rest of the framework is trying to make tradable. ## The drift is not yours to choose Here is the load-bearing idea, and it is the direct analogue of Black-Scholes. Because the price is the discounted risk-neutral value of a one-dollar payoff, it has to be a martingale: no expected drift under the pricing measure. But the log-odds map is nonlinear, so a zero-drift price does not mean a zero-drift x. Apply Itô's formula to the sigmoid and the convexity forces a specific drift on x. $$ \\mu(t,x) = -\\,\\frac{\\tfrac{1}{2} S''(x)\\, \\sigma\_b^2(t,x) + \\int\_{\\mathbb{R}} \\big( S(x+z) - S(x) - S'(x)\\chi(z) \\big)\\, \\nu\_t(dz)}{S'(x)} $$ In words: the drift of the log-odds is whatever it has to be to cancel the curvature and jump-compensation terms, so that the probability itself has no expected change. Take the diffusion-only case with no jumps, where S-prime is p times one minus p and S-double-prime is that times one minus two-p. The drift collapses to minus one-half times one minus two-p times sigma-b squared. Plug in p equal to 0.6 and a belief vol of 0.8: the drift is minus one-half times minus 0.2 times 0.64, which is plus 0.064\. The log-odds is nudged gently upward not because anyone thinks the event got likelier, but purely to offset the convexity of the sigmoid and keep the price a fair game. This is the same discipline as the old article ["Arbitrage Is Just Projection: One Calculation Gives Trade, Profit, and Target Prices."](https://aligrithm.com/arbitrage-is-just-projection-one-calculation-gives-trade-profit-and-target-prices/) No-arbitrage is not a vibe, it is a constraint that pins down a specific quantity. There, projecting onto the no-arbitrage subspace gave you the trade and the target prices in one shot. Here, imposing the martingale property gives you the drift in one shot. Once the drift is spoken for, it stops being a knob. What remains free, and therefore what you can actually quote and hedge, is belief volatility, jump intensity and size, and cross-event correlation. ## The derivative layer: sell spread, buy variance If belief volatility is the free risk, you want to trade it directly instead of eating it through inventory. The options market solved this with variance and volatility swaps, and the paper transplants them. A belief-variance swap pays the realized quadratic variation of the log-odds against a fixed strike, and under slowly varying parameters that strike has a clean form. $$ K\_{t,T}^{x,\\text{var}} \\approx \\int\_t^T \\sigma\_b^2(u)\\, du + \\int\_t^T \\lambda(u)\\, \\mathbb{E}\[z^2(u)\]\\, du $$ The fair strike is the sum of two variance sources: the diffusion part, the integral of belief vol squared, plus the jump part, the jump intensity lambda times the expected squared jump size. Work it over ten days. Say belief vol squared runs at 0.64 per day, so the diffusion piece is 6.4\. Say jumps arrive at intensity 0.2 per day with expected squared size 0.25, so the jump piece is 0.2 times 0.25 times ten, which is 0.5\. The strike is about 6.9\. A maker who sells tight spreads and is therefore short realized variance can buy this swap to neutralize the exposure, and the split tells them how much of the premium is paying for ordinary wobble versus gap risk. ![A belief-volatility term structure with a local bump at a scheduled news date](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_411-belief_vol_term.png) The strike is an integral over a belief-vol term structure, and that structure is not flat. The figure sketches the shape the paper wants you to calibrate: a baseline that rises into resolution as the outcome sharpens, plus a local bump on a scheduled announcement where you know intensity will spike. Once you can see that bump, you can budget your hedge for it, buying variance concentrated in the window where it actually accrues. The paper extends the menu with corridor variance, which only accrues while the price sits in a swing zone away from the boundaries, and correlation swaps that offset co-movement across related markets on something like an election night. Whether any venue will actually list these instruments is a separate question the paper does not answer. ## Greeks that live in the middle of the board To hedge any of this you need sensitivities, and in the log-odds parameterization they are almost embarrassingly simple. The delta and gamma of a vanilla contract with respect to x are just the first two derivatives of the sigmoid. $$ \\Delta\_x = \\frac{\\partial p}{\\partial x} = p(1-p), \\qquad \\Gamma\_x = \\frac{\\partial^2 p}{\\partial x^2} = p(1-p)(1-2p) $$ Delta is p times one minus p, which is maximized at 0.5 and dies at both boundaries. Gamma is that times one minus two-p, which is positive below the midpoint, zero at 0.5, and negative above it. Work the corners: at p equal to 0.5 delta is 0.25, its peak, and gamma is zero. At p equal to 0.9 delta is only 0.09 and gamma is 0.09 times minus 0.8, about minus 0.072\. So a market pinned near 0.9 barely moves in probability terms even when the log-odds swings hard, while a coin-flip market at 0.5 has the most price sensitivity and the most dangerous curvature. ![Logit-domain Greeks: delta peaks at p equal to 0.5, gamma flips sign there, and both vanish at the boundaries](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_411-greeks_logit.png) This is why liquidity and toxicity concentrate in the middle of the board. Order flow is most dangerous where gamma is largest and inventory turns fastest, which is the swing zone the corridor products target. The paper wraps this into an inventory-aware quoting rule, a version of the standard dealer reservation-price-and-spread recipe written in log-odds units, with the spread widening on toxic flow and pulling in near the boundaries where the contract barely moves. It is a sensible mapping, and it inherits the same limitation as every such rule: it is only as good as your estimate of belief vol. ## Getting the numbers out of the tape None of this is tradable without calibration, and the price you see is not the latent log-odds. It is that log-odds plus microstructure noise: stale quotes, bid-ask bounce, thin-book flicker. The paper's pipeline is a two-stage estimator. First a heteroskedastic state-space filter, a Kalman variant whose measurement noise is modeled as a function of spread, depth, trade rate and imbalance, recovers a smoothed latent log-odds from the noisy tape. Then an expectation-maximization step treats each log-odds increment as a mixture of a diffusion draw and a jump draw, and assigns each increment a posterior probability of being a jump. Increments that look like jumps get pulled out before estimating diffusion variance, so the belief-vol number is not contaminated by gaps. The refined variance and jump estimates feed the martingale drift formula, and one or two outer loops tighten it. The output is the belief-vol surface, a jump layer, and a dependence layer, indexed by time-to-resolution and moneyness. The idea is coherent. Separate the signal from the noise, then separate the diffusion from the jumps, then enforce the no-arbitrage drift, and you get a stable object to quote off. The question is whether it actually forecasts better than something dumber. ## Does it work? The evidence is one synthetic path Here the paper gets thin, and honesty demands flagging it. The headline experiment is a single end-to-end forecast task: at each second, predict the realized log-odds variance over the next sixty seconds, then compare to what happened. The full model, RN-JD, is put up against a random walk in log-odds, a constant-volatility logit diffusion, a boundary-respecting Wright-Fisher diffusion, and an AR(1)-GARCH model, all evaluated causally on the same series. | Model | MSE | MAE | QLIKE | | ---------------------- | ------ | ----- | ----- | | RN-JD (this paper) | 70.28 | 1.588 | 1.462 | | Random walk in logit | 77.41 | 1.163 | 4.732 | | Constant-vol logit | 76.75 | 2.078 | 2.659 | | Wright-Fisher (mapped) | 1.7e17 | 3.7e7 | 1.948 | | AR(1)-GARCH (mapped) | 1.1e19 | 5.3e8 | 0.796 | Read the table like a skeptic. RN-JD wins on squared and absolute error, which is the paper's claim. But the two probability-space baselines post error terms of ten-to-the-seventeenth and ten-to-the-nineteenth, which is not "worse," it is numerically detonating. When you map a GARCH forecast from probability space back to log-odds you divide by p times one minus p squared, and near a boundary that blows up. So most of RN-JD's MSE advantage is really a statement that its competitors explode when mapped, not that its variance forecast is sharp. Worse for the headline, on QLIKE, the loss that volatility researchers actually trust because it penalizes under-prediction and tolerates noise, the exploding AR(1)-GARCH still wins at 0.796 against RN-JD's 1.462\. The paper's own preferred model loses on the metric its own field prefers. And this is one path. The table is a single synthetic, risk-neutral-consistent simulation of six thousand steps, plus a mention of twenty Polymarket contracts that never turns into a real out-of-sample horse race. There is no cross-event calibration, no live A/B test, no confidence intervals. The paper is upfront that multi-event dependence and real deployment are future work. So take it for what it is: a well-constructed conceptual kernel with a proof-of-concept that does not yet clear the bar it sets for itself. ## Why the idea might still matter The paper's best argument is the one in its own conclusion: standardization beats realism. Black-Scholes coordinated options around implied vol despite being wrong about fat tails, jumps and stochastic volatility, all of which got bolted on later. What mattered was the shared vocabulary. If prediction markets are about to scale, and the ICE stake in Polymarket and billion-dollar monthly volumes suggest they are, then a common way to name belief volatility, jump intensity and cross-event correlation is worth more than any single forecast being accurate. The kernel's real contribution is not the sixty-second variance number. It is forcing the drift to be non-negotiable and exposing the remaining three factors as the things a desk should quote, hedge and eventually list swaps against. Whether the market adopts that vocabulary is an empirical question the paper cannot settle from a synthetic path, and it would be a mistake to trade real inventory on this calibration as though it were validated. But as a map of what belief risk decomposes into, and where a maker's danger actually lives, the logit jump-diffusion is a clean and useful frame. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_411-visual-0.png) ## Where this connects This kernel sits on top of two ideas this pillar has already built. The martingale drift is the same move as the old article ["Arbitrage Is Just Projection: One Calculation Gives Trade, Profit, and Target Prices"](https://aligrithm.com/arbitrage-is-just-projection-one-calculation-gives-trade-profit-and-target-prices/): no-arbitrage is a hard constraint that hands you a specific number rather than a vibe, and here the number is the drift of the log-odds. The logit map at the center is the object from the old article ["If You've Used a Softmax Layer, You Already Know the Price Formula,"](https://aligrithm.com/if-youve-used-a-softmax-layer-you-already-know-the-price-formula/) run backward so you can put a stochastic process on the score and squash it into a valid price. The market-maker layer connects forward to everything this pillar says about quoting under adverse selection. Belief vega, corridor variance and the swing-zone concentration of gamma are just precise names for where inventory hurts most, which is the same place tight spreads get picked off. The honest read is that the theory is ahead of the evidence. Treat the decomposition as a lens, not a live trading signal, until someone runs it out of sample on real multi-event data. ## KEY POINTS - Prediction markets lack the shared pricing language options got from Black-Scholes. The paper proposes the analogue: model the log-odds of the price as a jump-diffusion and standardize around belief volatility, jump intensity, and correlation. - Trade the log-odds, not the probability. The logit map sends the bounded price to the whole real line where standard tools apply, and the sigmoid maps it back. It is the same map a softmax uses, run in reverse. - The drift is not a free parameter. Forcing the price to be a martingale, the no-arbitrage condition, pins the log-odds drift to a convexity-plus-jump correction. In the diffusion-only case it is minus one-half times one minus two-p times belief vol squared. - What is left free is what you can trade: belief volatility, jump size and intensity, and cross-event correlation. The paper defines variance swaps, corridor variance, and correlation swaps on these, with a fair variance strike that splits cleanly into a diffusion piece and a jump piece. - The Greeks are trivial in log-odds units: delta is p times one minus p, gamma is that times one minus two-p. Delta peaks and gamma flips sign at 0.5, and both vanish at the boundaries, which is why toxicity concentrates in the swing zone. - Calibration is a state-space filter to strip microstructure noise, then an EM step to separate diffusion from jumps, then enforcement of the martingale drift, yielding a belief-vol surface. - The evidence is weak. The headline result is one synthetic path where the model wins on MSE mostly because its baselines numerically explode when mapped, and it loses on QLIKE, the metric volatility researchers prefer. Treat the kernel as a useful frame, not a validated trading signal. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Convex Optimization - Stephen Boyd Lieven Vandenberghe (Amazon)](https://www.amazon.com/s?k=Convex+Optimization+Stephen+Boyd+Lieven+Vandenberghe&ref=aligrithm.com) - [An Introduction to Copulas - Roger Nelsen (Amazon)](https://www.amazon.com/s?k=An+Introduction+to+Copulas+Roger+Nelsen&ref=aligrithm.com) - [Toward Black–Scholes for Prediction Markets: A Unified Kernel and Market-Maker’s Handbook (black\_and\_sholes\_prediction\_markets)](https://scholar.google.com/scholar?q=Toward+Black%E2%80%93Scholes+for+Prediction+Markets%3A+A+Unified+Kernel+and+Market-Maker%E2%80%99s+Handbook&ref=aligrithm.com) ### 8.9 Regime-Switching That Works, Factors That Don't (MS-GARCH) URL: https://aligrithm.com/regime-switching-that-works-factors-that-dont-ms-garch/ Last updated: 2026-07-27T13:26:04.000Z Regime-switching gets sold as the model that tells you when to leave. Turn it on, it flags the storm, you step aside before the drawdown. De la Torre-Torres, Álvarez-García, and del Río-Rama ran the honest version of that pitch on one commodity and reported every number, the ugly ones included. They traded random-length lumber futures weekly from 2004 to 2023 with a single rule: if a two-regime Markov-switching model says next week is high-volatility, sit in 3-month Treasury bills; otherwise hold lumber. The headline is a 158% cumulative return against 7% for buy-and-hold. The two results that actually teach you something sit underneath it. The first: the entire edge depends on which variance model you bolt onto the regime switch, not on the regime switch itself. Swap the variance engine and the same rule goes from losing 53% to making 158%. The second: the authors then loaded the model with market and behavioral factors, expecting a lift, and the good versions got worse. The old article "Volatility-Regime Filter for Any Strategy" made the general case for gating any strategy by its volatility regime. This paper is that idea run through a real estimation engine on a real instrument, and it shows where the idea earns its keep and where the fancy version quietly breaks. ## The rule is one inequality The trader holds lumber or holds cash, nothing in between. Each week the model estimates the probability of being in the low-volatility regime (call it s=1) and the high-volatility regime (s=2), then projects that probability forward one week and checks a threshold. If the forecasted probability of the high-vol regime next week, written xi for s=2 at t+1, is 50% or lower, take a full long position in the 1-month lumber future. If it is above 50%, move 100% into the T-bill fund. The forecast itself is a matrix multiply on the current regime probabilities. $$ \\begin{bmatrix} \\xi\_{s=1,t+n} \\\\ \\xi\_{s=2,t+n} \\end{bmatrix} = P^{n} \\begin{bmatrix} \\xi\_{s=1,t} \\\\ \\xi\_{s=2,t} \\end{bmatrix}, \\qquad P = \\begin{bmatrix} \\pi\_{11} & \\pi\_{12} \\\\ \\pi\_{21} & \\pi\_{22} \\end{bmatrix} $$ Read it plainly. The vector on the right is where the model thinks you are right now: the smoothed probabilities of each regime at time t. P is the transition matrix, where each entry is the probability of moving from one regime to another. Raising P to the power n and applying it walks those probabilities n weeks ahead. For the trade you only need n equal to one, the one-week-ahead forecast. Work a number. Say the model puts you at 80% low-vol and 20% high-vol today, so the current vector is 0.80 and 0.20\. Suppose the transition matrix says low-vol stays low with probability 0.90 and flips to high with 0.10, while high-vol reverts to low with 0.30 and stays high with 0.70\. The one-week high-vol forecast is 0.10 times 0.80 plus 0.70 times 0.20, which is 0.08 plus 0.14, or 0.22\. That is 22%, below the 50% line, so you hold lumber into next week. The threshold is the whole discretion in the system. Everything hard happens upstream, in how the model decides those probabilities. That matters because the benchmark is close to flat. Buy-and-hold lumber returned 7.13% over the full 1022-week sample, about 0.34% a year. Lumber went almost nowhere across two decades while whipping violently around that flat line. A rule that sidesteps the worst weeks has room to add a lot without predicting direction at all. ## The variance model is the whole game Here is where the paper stops being a lumber study and becomes a lesson about regime models. The authors did not test one model. They crossed three probability distributions for the likelihood (Gaussian, Student's t, and the fat-tailed generalized error distribution) with four ways of modeling the regime's volatility: a time-fixed variance, a plain ARCH, a symmetric GARCH, and an asymmetric EGARCH. Same rule, same data, twelve engines. The spread in results is enormous. The time-fixed variance models, the ones that assume each regime has a constant volatility, were the worst. The Gaussian version lost 52.9% and the GED version lost 20.1%. The moment the variance is allowed to move within a regime, the picture flips. The symmetric GARCH models made 86% to 151%, and the asymmetric EGARCH with a GED likelihood topped the table at 158.3%. The mechanism is a time-varying variance recursion running inside each regime. $$ \\hat{\\sigma}^2\_t = \\omega + \\alpha\\,\\varepsilon^2\_{t-1} + \\beta\\,\\hat{\\sigma}^2\_{t-1} $$ The forecast variance today, sigma-squared-t, is a baseline omega, plus a reaction to yesterday's squared shock (alpha times epsilon-squared), plus a memory of yesterday's variance (beta times the prior sigma-squared). Alpha controls how sharply volatility jumps after a surprise; beta controls how long the elevated volatility lingers. Work it. Set omega to 0.00002, alpha to 0.10, and beta to 0.85\. Suppose last week delivered a 3% shock, so epsilon-squared is 0.0009, and the prior variance was 0.0004, a 2% weekly volatility. This week's variance is 0.00002 plus 0.10 times 0.0009 plus 0.85 times 0.0004, which is 0.00002 plus 0.00009 plus 0.00034, or 0.00045\. Take the square root and volatility has climbed from 2.00% to 2.12%, and the 0.85 memory term keeps it elevated for weeks. The EGARCH variant goes one step further: it models the logarithm of variance and lets a negative shock raise volatility more than a positive shock of the same size. That asymmetry, the well-documented fact that lumber and most risk assets get more turbulent on the way down, is why the EGARCH caught the onset of high-vol regimes earlier and produced cleaner sell signals. A constant-variance regime model cannot see a storm building inside the calm regime; a GARCH one can. ![The best no-factor portfolio (orange) against buy-and-hold lumber (blue), with the forecasted high-volatility regime probability in the lower panel. The sustained spike in 2021-2022 is where the model exits to T-bills and the buy-and-hold holder eats the crash.](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_426-regime_probability_selloff.png) The lower panel is the tell. For most of the sample the forecasted high-vol probability sits near the floor, so the algorithm stays long. It spikes hard through the 2021-2022 lumber blow-off and collapse, and that is exactly when the model parks in T-bills while a buy-and-hold trader rides the price from roughly 500 back down to 100\. The algorithm never forecasts price. It forecasts turbulence and refuses to hold through it. ## Adding factors made the winners worse The obvious next move is to feed the model more information. Instead of estimating regimes off lumber returns alone, estimate a mean equation first, packed with market and behavioral factors, and run the regime model on what is left over. The authors did exactly this, with the commodity-index return, Working's speculation ratio (volume over open interest), the VIX, and the dollar index as market factors, plus three sentiment trackers: economic policy uncertainty, a stock-market volatility news tracker, and an infectious-disease news tracker. $$ \\hat{r}\_t = \\alpha + \\beta\_1 \\text{comm}\_t + \\beta\_2 \\text{WHR}\_t + \\beta\_3 \\text{VIX}\_t + \\beta\_4 \\text{DXY}\_t + (\\text{sentiment}) + \\varepsilon\_t, \\qquad \\varepsilon\_t = r\_t - \\hat{r}\_t $$ The first part fits a predicted return, r-hat-t, from the factors. The second part is the piece that matters: the Markov-switching GARCH model is estimated on the residual, epsilon-t, which is the actual return minus the factor prediction, not on the return itself. This is a structural constraint, not a choice. A GARCH process cannot be estimated jointly with a moving mean because of path dependence, so the factors are forced into the mean equation and the regime model only ever sees the leftovers. Work it. Suppose lumber returns 1.2% one week and the factor model predicts 0.5%. The residual fed to the regime model is 0.7%. Change the factor set and every residual in the history changes, which means the regimes the model detects change too. That is the trap. Richer factors do not sharpen the volatility signal; they redraw the entire residual series the volatility model learns from. The result was blunt. Factors helped the models that did not work and hurt the ones that did. The broken time-fixed MS models jumped (the Student's t version posted 282%, the highest number in the paper), but the authors trace that spike almost entirely to the single 2020 COVID episode, not to any repeatable skill. The GARCH and EGARCH engines that carried the no-factor results went backward: GARCH with a GED likelihood fell from 151% to 92.6%, and the EGARCH-GED winner dropped from 158.3% to 123.5%. ![Cumulative return of all nine model specifications, no factors versus market plus behavioral factors, against the flat 7.1% buy-and-hold line. Factors lift the weak time-fixed models and drag down the GARCH and EGARCH ones that actually worked.](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_426-accumulated_returns.png) This is a clean instance of the old article ["Loose Pants Fit Everyone: Why General Trading Ideas Survive Longer."](https://aligrithm.com/loose-pants-fit-everyone-why-general-trading-ideas-survive-longer/) The lean model, lumber returns and nothing else, generalized. The tailored model, seven factors and three sentiment indexes stitched into the mean, fit its residuals to a narrower set of past conditions and lost ground where it counted. The authors set out to prove that factors help (their Hypothesis 2) and reported, against their own thesis, that they do not. ## The alpha that mostly wasn't The 158% headline needs a second look before anyone builds on it. The paper measures skill with Jensen's alpha against the broad commodity futures market, which strips out the return you got just for holding a commodity when commodities happened to rise. $$ \\alpha = r\_t - \\beta\\, r\_{m,t} $$ Alpha is the portfolio return, r-t, minus its market exposure beta times the commodity-market return, r-m-t. If alpha is positive by a meaningful amount, the algorithm added something the market did not hand it for free. Work it: a 0.12% weekly portfolio return, a beta of 0.9, and a 0.10% commodity-market week give alpha of 0.12% minus 0.09%, or 0.03% per week, roughly 1.6% a year. That 0.03% is close to what the winning model actually earned in alpha terms, and it is thin. Only three of the nine no-factor portfolios posted any positive alpha at all; the rest matched or lagged the broad commodity market. The winner's Sharpe ratio was 0.059, barely off zero. The rule beats buy-and-hold, but buy-and-hold was a flat 0.34% a year, so clearing it is a low bar. Measured against the thing that pays you nothing for skill, the broad commodity index, most of these portfolios added almost nothing. The regime switch is real and it dodges crashes; the risk-adjusted excess return on top of just owning commodities is small. ## What the backtest quietly leaves out Two omissions decide whether any of this survives contact with a broker. The authors state plainly that no trading fees were included. A rule that flips 100% in and 100% out on a weekly signal churns hard, and on an illiquid contract the round-trip cost is not a rounding error. Strip the fees out and the thin 1.6%-a-year alpha has to cover a spread it never paid in the simulation. The bigger problem is the instrument. CME wound down the random-length lumber contract in 2023, right where the sample ends, and replaced it with a smaller, differently specified lumber future. The exact series tested here stopped trading. On top of that, this is one commodity, one sample, with the "best fitting" variant chosen each week by an in-sample information criterion, which is a form of selection the out-of-sample story has to carry. The transferable lesson is not "trade lumber this way." It is the structural finding: gate an asset by a regime model, put the effort into an asymmetric time-varying variance, and resist the urge to feed the model factors that only redraw its residuals. The regime switch worked because of the variance engine. The factors failed because they never touched it. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_426-visual-0.png) ## KEY POINTS - The strategy is one inequality: if the forecasted probability of next week's high-volatility regime is 50% or lower, hold lumber; otherwise hold 3-month T-bills. The forecast is the current regime probabilities pushed one step through the transition matrix. - The edge lives in the variance model, not the regime switch. Constant-variance regime models lost 20% to 53%; symmetric GARCH made 86% to 151%; the asymmetric EGARCH with a fat-tailed GED likelihood topped out at 158.3% against 7.1% for buy-and-hold. - Asymmetry is why EGARCH won. Modeling log-variance and letting negative shocks raise volatility more than positive ones caught the onset of turbulent regimes earlier and produced cleaner exits, visible in the 2021-2022 lumber crash. - Adding market and behavioral factors backfired on the models that worked. Factors enter only through the mean equation, so the regime model is fit on residuals; richer factors redraw those residuals and dragged GARCH-GED from 151% to 92.6% and EGARCH-GED from 158% to 123.5%. - The 282% factor "winner" was a broken time-fixed model rescued by a single 2020 COVID episode, not repeatable skill. - The alpha is thin. Against the broad commodity index most portfolios added almost nothing; the winner's Sharpe was 0.059 and its alpha near 0.03% a week. Beating a flat 0.34%-a-year buy-and-hold is a low bar. - The backtest charged no trading fees on a weekly full-flip rule, and the tested random-length lumber contract was discontinued in 2023\. Treat the transferable result as a method, not a lumber trade. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Mathematical Techniques in Financial Markets - Don Mak (Amazon)](https://www.amazon.com/s?k=Mathematical+Techniques+in+Financial+Markets+Don+Mak&ref=aligrithm.com) - [Cycle Analytics for Traders - John Ehlers (Amazon)](https://www.amazon.com/s?k=Cycle+Analytics+for+Traders+John+Ehlers&ref=aligrithm.com) - [An EM/MCMC Markov-Switching GARCH Behavioral Algorithm for Random-Length Lumber Futures Trading (An EMMCMC Markov-Switching GARCH Behavioral)](https://scholar.google.com/scholar?q=An+EM%2FMCMC+Markov-Switching+GARCH+Behavioral+Algorithm+for+Random-Length+Lumber+Futures+Trading&ref=aligrithm.com) ### 4.71 Stocks and Bonds Predict FX — But Only in Emerging Markets URL: https://aligrithm.com/stocks-and-bonds-predict-fx-but-only-in-emerging-markets/ Last updated: 2026-07-27T13:25:00.000Z Meese and Rogoff broke exchange-rate forecasting in 1983\. They showed that no macro model, not interest rates, not money supply, not trade balances, beat a coin-flip random walk out of sample. Four decades later that result still stands for the dollar, the euro, the yen. Phylaktis and Yamani ran a different predictor through the same gauntlet: not the usual macro fundamentals, but the return on a country's own stock and bond markets relative to the US. The random walk survived again for developed currencies. For emerging currencies it did not. An equity-return signal predicted the next month's move well enough to trade, net of costs, and the profit did not exist in the developed world. The old article ["Why FX Traders Must Watch Gold, Rates, and Equities"](https://aligrithm.com/why-fx-traders-must-watch-gold-rates-and-equities/) made the structural case that a currency has no standalone fundamentals, so the force that moves it lives in other markets. This paper is that claim run as a strict out-of-sample horse race across 28 countries, and it hands back a sharp, uncomfortable answer: the cross-market signal is real, but only where the market is inefficient enough to leak it. ## One regression, one differential The whole paper hangs on a single predictive regression. Take a currency, take the return on that country's stock market minus the return on the US stock market, and use that differential to forecast next month's exchange-rate move. $$ \\Delta s\_{i,t+1} = \\alpha + \\beta \\left( r^{E}\_{i,t} - r^{E,US}\_{t} \\right) + \\varepsilon\_{i,t+1} $$ The left side, delta-s, is next month's log change in the exchange rate quoted as foreign-currency units per one US dollar, so a fall in delta-s means the foreign currency strengthened. The predictor in parentheses is the local stock return minus the US stock return, the equity differential. Beta is the one number that decides whether the signal points the right way, and alpha is the drift the random walk also gets. The bond version swaps in the government-bond return differential; the combined version uses both. The sign of beta is the mechanism. Across the combined model the equity coefficient came in negative for every basket: minus 0.105 developed, minus 0.129 emerging, minus 0.083 global. Negative means that when the foreign stock market outruns the US, the foreign currency tends to appreciate next month. That is the return-chasing story: foreign investors see a hot local equity market, buy in, and bid up the local currency to do it. Work a number. Emerging equity coefficient is minus 0.129\. Suppose an emerging market's stocks beat US stocks by 2 percentage points this month, so the differential is plus 2\. The model predicts delta-s of minus 0.129 times 2, which is minus 0.26%. Delta-s negative means the foreign currency is forecast to appreciate about a quarter percent next month, so you buy that currency. That is the entire forecast. One coefficient, one differential, one month ahead. ## The random walk is the bar, and it is a high one A positive-looking regression means nothing here, because the benchmark is not zero. It is the random walk, and beating it out of sample is the whole game the FX literature has lost for forty years. The paper scores every forecast against the random walk with the out-of-sample R-squared. $$ R^{2}\_{OOS} = 1 - \\frac{MSFE^{\\chi}}{MSFE^{RW}} $$ MSFE is the mean squared forecast error, the average of the squared misses. The formula divides your model's error by the random walk's error and subtracts from one. If your model's squared errors are smaller, the ratio drops below one and the out-of-sample R-squared turns positive. If you forecast worse than the random walk, it goes negative. Zero is the coin flip. Work it for the combined model on emerging currencies. Its MSFE is 4.479 and the random walk's is about 4.715, so the ratio is 0.95 and the out-of-sample R-squared is 1 minus 0.95, which is 0.05\. The model's squared errors run about 5% below the random walk. For monthly FX that is a large edge, and it only appears in emerging currencies. The developed combined model posts a ratio of 1.016, an out-of-sample R-squared of minus 0.016, so it forecasts worse than doing nothing. The paper backs this with a Clark-West test, a formal t-test of whether two nested models have equal predictive accuracy, because a raw R-squared gap can be luck. ## The asymmetry is the whole result The Clark-West statistics draw a clean line between the two worlds. Read them and the paper's thesis is obvious in one glance. ![Clark-West t-statistics comparing developed and emerging portfolios across equity, bond, and combined forecasting models, with significance thresholds marked](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_420-cw_tstats.png) Emerging equity forecasts hit a Clark-West t of 3.11, past the 1% bar. Emerging combined forecasts hit 2.72, also past 1%. Both crush the random walk. The developed portfolio's equity model manages 0.63 and its combined model 1.41, neither clearing even the 10% line. Bonds fail everywhere in this full-sample test, with negative t-stats for both groups. Stock returns carry information about emerging currencies and almost nothing about developed ones. This is the empirical face of what the old article ["Using Bonds to Filter Equity Signals"](https://aligrithm.com/using-bonds-to-filter-equity-signals/) argued in the other direction: stocks and bonds and currencies are three prices set by overlapping forces, so one market's return is a legitimate gate or predictor for another. Here the equity market is the leading price and the currency lags it by a month, but only where the lag is wide enough to trade. ## Statistical significance is not a paycheck A significant t-stat does not pay rent. The paper's second test turns each forecast into a position: go long the currency if the predicted return is positive, short it if negative. $$ Signal\_t = \\operatorname{sign}\\left( \\widehat{\\Delta s}^{\\chi}\_{t+1|t} \\right), \\qquad \\Delta s^{\\chi}\_{i,t} = Signal\_t \\times \\Delta s\_{i,t+1} $$ The signal is plus one or minus one depending on the sign of the forecast. The strategy return is that signal times the actual realized currency move. When the forecast calls direction right, sign and realized move share a sign and the product is positive. When it calls direction wrong, the product is negative. Direction is all that matters; the paper cites the old point that getting the sign right beats getting the magnitude right for a trading rule. Then it subtracts transaction costs, half the bid-ask spread on each turn, before reporting anything. Net of those costs the emerging equity strategy earned 0.30% a month and the combined strategy 0.59% a month, about 7% a year, with positive skew, meaning the rare big months were gains not losses. Both beat the random walk, and the combined rule beat currency momentum and the carry trade, the two workhorses of FX. In developed currencies the same machinery produced nothing worth trading. ![Net-of-cost mean monthly returns for the equity and combined strategies, emerging versus developed currencies](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_420-emerging_returns.png) ## Why emerging and not developed The split is not an accident of the sample; it is the point. Developed currency markets are deep, integrated, and picked over, so any stock-to-currency signal gets arbitraged before you can trade it. Emerging markets are thinner, more segmented, and carry more information leakage, so prices absorb news sluggishly and the equity lead-in survives long enough to harvest. The paper leans on the return-chasing channel to explain the mechanism: capital flows into hot emerging equity markets drag the currency up behind them, and that flow is slow and observable. This is where the old article ["PPP and the Big Mac Index: Why Valuation Only Matters Long-Term"](https://aligrithm.com/ppp-and-the-big-mac-index-why-valuation-only-matters-long-term/) fits as a foil. Purchasing power parity says a currency has a fair value it eventually reverts to, but it can stay mispriced for years, so valuation is useless for a one-month bet. This paper works the other clock. It ignores fair value entirely and rides the short-horizon flow signal from equities, the thing that actually moves the pair next month while PPP sleeps. Two different horizons, two different drivers, and only one of them pays inside a month. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_420-visual-0.png) ## The catch: it comes and goes Here the skeptic earns their seat. The paper splits the out-of-sample period in two, and the edge is not stable. From September 2006 to April 2013, through the global financial crisis and the sharp dollar spike that came with it, none of the combined-model forecasts beat the random walk and the strategies made no reliable money. The bond strategy actually lost 0.68% a month on emerging currencies in that window. Only from May 2013 to December 2019 did the signal switch on: the combined model turned significant, and the bond strategy flipped from that 0.68% loss to a 0.47% monthly gain. So the headline "stocks predict emerging FX" is really "stocks predicted emerging FX in one six-year window of a twenty-year sample." That is a live warning, not a footnote. A monthly signal whose sign of profitability inverts across regimes is exactly the kind of edge that looks great in the full-sample table and gets you run over when you deploy it into the wrong half of the cycle. The authors are honest about it, tying the dead patch to the crisis-era dollar and calling the profitability time-varying. Treat the 7%-a-year combined number as a regime-conditional result with a real off switch, size it small, and watch the same rate-equity-gold confirmation the old FX articles insist on before trusting the stock signal alone. ## KEY POINTS - The predictor is a country's own stock (or bond) return minus the US return, run through one linear regression to forecast next month's exchange-rate move. - The benchmark is the random walk, unbeaten in developed FX since Meese and Rogoff 1983\. Beating it out of sample, not posting a positive backtest, is the entire bar. - Equity return differentials beat the random walk for emerging currencies (Clark-West t of 3.11 equity, 2.72 combined, both 1% significant) and fail for developed currencies (t of 0.63 and 1.41, not significant). Bonds fail in the full sample for both. - The combined emerging model cut mean squared forecast error about 5% below the random walk, an out-of-sample R-squared near 0.05, large for monthly FX. - Trading the sign of the forecast net of costs earned about 0.30% a month on emerging equity signals and 0.59% a month combined, with positive skew, beating carry and momentum. Developed currencies produced nothing tradable. - The mechanism is return-chasing: capital flows into hot emerging equity markets pull the currency up behind them, and emerging markets are slow and segmented enough that the lag survives arbitrage. - The edge is time-varying. It was dead from 2006 to 2013 (the bond rule lost 0.68% a month through the crisis) and alive from 2013 to 2019 (bond rule +0.47%). One six-year window carries the full-sample result. - Valuation anchors like PPP work on a multi-year clock and are useless here; this is a one-month flow signal, a different driver at a different horizon. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Trading Systems and Methods - Perry Kaufman (Amazon)](https://www.amazon.com/s?k=Trading+Systems+and+Methods+Perry+Kaufman&ref=aligrithm.com) - [Cybernetic Trading Strategies - Murray Ruggiero (Amazon)](https://www.amazon.com/s?k=Cybernetic+Trading+Strategies+Murray+Ruggiero&ref=aligrithm.com) - [The Art of Currency Trading - Brent Donnelly (Amazon)](https://www.amazon.com/s?k=The+Art+of+Currency+Trading+Brent+Donnelly&ref=aligrithm.com) - [Foreign currency forecasting: What can stock and bond markets tell us?\* (Foreign currency forecasting What can stock and bond markets tell us)](https://scholar.google.com/scholar?q=Foreign+currency+forecasting%3A+What+can+stock+and+bond+markets+tell+us%3F%2A&ref=aligrithm.com) ### 6.48 Trend-Following P&L Is a Function of Autocorrelation (Closed Form) URL: https://aligrithm.com/trend-following-p-l-is-a-function-of-autocorrelation-closed-form/ Last updated: 2026-07-27T13:25:54.000Z Ask a CTA salesperson why their fund makes money and you get a story: markets trend, we ride the trend, we cut losers and let winners run. That story is untestable. Sepp and Lucic did something the industry rarely does. They wrote down the exact profit-and-loss of a standard European trend-follower and factored it into two things you can measure directly: the autocorrelation of the traded returns and the square of the drift. No story, no hand-waving about market psychology. The whole P&L reduces to one signed number, and if that number is negative the strategy loses no matter how disciplined you are. The old article ["Building a Trend Follower, Component by Component"](https://aligrithm.com/building-a-trend-follower-component-by-component/) walked the full construction pipeline: universe selection, a volatility-normalized fast-minus-slow filter, a sigmoid position map, inverse-vol sizing. This piece takes the European branch of that pipeline and asks the harder question. Given the machine is built correctly, what does it actually earn, and on what does that number depend? The answer is a closed form, and it is unforgiving. ## The daily return is signal yesterday times return today Start with what a European trend-follower does on any given day. It holds a position whose size is the smoothed signal times a volatility target, and it earns that position times today's return. Strip the bookkeeping and the daily P&L collapses to a product. $$ f\_t = w\_{t-1} r\_t = \\frac{\\sigma\_{target}}{\\sqrt{a}} \\, s\_{t-1} \\, z\_t $$ Read it plainly. The term f-sub-t is the strategy's return today. The s-sub-t-minus-1 is the signal computed through yesterday, an exponentially weighted moving average of past volatility-normalized returns, which you can read as a running estimate of the Sharpe ratio. The z-sub-t is today's return divided by its estimated volatility. Sigma-target over root-a is a constant that just sets the risk level. The strategy return today is yesterday's trend signal multiplied by today's normalized move. Work a number. Set the annual volatility target to 15%, and the annualization factor a to 260 trading days, so root-a is about 16.1 and the constant is 0.15 divided by 16.1, or 0.0093\. Suppose the signal through yesterday sits at 1.5 (a decently strong uptrend on a z-score scale) and today the market delivers a normalized return of 0.8\. The day's P&L is 0.0093 times 1.5 times 0.8, which is 0.011, or 1.1%. If today the market instead drops and z-sub-t comes in at minus 0.8, the same positive signal now loses 1.1%. The strategy is betting that a positive signal is followed by a positive return. That bet only pays if returns are positively autocorrelated, and that is exactly what the closed form makes precise. ## The whole cumulative return is autocorrelation minus one, plus Sharpe squared Sum those daily products over a period and the mess of cross terms telescopes into something clean. This is the load-bearing result of the paper, the part the authors flag as new. $$ \\hat{F}\_T \\propto \\hat{\\gamma}\_T(0)\\left(\\left\[\\sum\_{m=0}^{\\infty} \\nu^m \\hat{\\rho}\_T(m) - 1\\right\] + \\frac{\\nu}{1-\\nu}\\frac{(\\overline{z}\_T)^2}{\\hat{\\gamma}\_T(0)}\\right) $$ The cumulative return F-hat-sub-T splits into two drivers. The first is a weighted sum of the sample autocorrelations rho-hat at every lag m, with the weight nu-to-the-m shrinking as the lag grows, and then you subtract one. The subtracted one is the lag-zero term, rho(0), which always equals one; it is the price of admission, the cost of trading against your own noise. The second driver is the square of the sample mean of normalized returns over its variance, which is just the squared Sharpe ratio of the underlying. The subtraction is the whole skeptical point. The weighted autocorrelation sum starts at one because a series is perfectly correlated with itself at lag zero. Subtract that one and you are left with the sum of the genuine forward-looking autocorrelations at lags one, two, three and beyond, discounted by the filter. If those lagged autocorrelations are on balance positive, the bracket is positive and the trend-follower makes money. If they are on balance negative, meaning the market mean-reverts at the horizons your filter cares about, the bracket goes negative and the strategy bleeds. The average autocorrelation must be positive for trend-following to work. Everything else is sizing. Note what does not appear. The sign of the average return, z-bar, is squared, so a trend-follower is indifferent to whether the drift was up or down. It profits from a strong down-move as readily as a strong up-move, because it can go short. That squared term is why trend-following is a long-volatility, direction-agnostic bet, not a disguised long position. ## White noise: no memory, no edge, only drift squared Feed the formula a process with zero autocorrelation at every nonzero lag, plain white noise with a drift. Every rho(m) for m at least one is zero, so the entire autocorrelation bracket vanishes. Only the drift term survives. $$ \\hat{F}\_{1y} = \\left(\\frac{l\\,\\sigma\_{target}}{\\sqrt{a}}\\right)\\left(\\mu^{z}\_{an}\\right)^{2}, \\qquad l = \\sqrt{span} $$ The annual return F-hat-1y equals a constant times the square of the annualized drift mu-z-an, which is the Sharpe ratio of the underlying. The loading l for the variance-preserving filter equals the square root of the filter span. Two things fall out. First, without autocorrelation a trend-follower can still make money, but only from drift, and only from its square, so a Sharpe of zero on the underlying gives you nothing. Second, that drift capture scales with the square root of the lookback, so longer filters harvest more of it. Work it. Take an underlying with an annualized Sharpe of 0.5, so mu-z-an is 0.5 and its square is 0.25\. On a one-year span of 250 days, l is root-250, about 15.8, and the annual return is 15.8 times 0.15 divided by 16.1, times 0.25, which is 3.7%. Drop to a one-month span of 21 days and l falls to about 4.6, giving 15.8 becomes 4.6, and the return collapses to roughly 1.1%. Same market, same drift, and the long filter earned more than triple the short one. This is the paper's "drift squared benefits long lookbacks" result stated in dollars: if all you have is a persistent drift and no genuine trending memory, you want a slow system, because a slow system spends more of its time leaning into the drift. ## AR(1): the sign of one coefficient decides everything Now give the process real memory. An AR(1) process has autocorrelation that decays geometrically, rho(m) equals phi-to-the-m, and the infinite weighted sum has a tidy closed form. $$ \\hat{F}\_{1y} = c\_{1y}\\left\[\\frac{\\phi\\nu}{1-\\phi\\nu}\\right\] + \\left(\\frac{l\\,\\sigma\_{target}}{\\sqrt{a}}\\right)\\left(\\mu^{z}\_{an}\\right)^{2}, \\qquad c\_{1y} = l\\,\\sigma\_{target}\\sqrt{a}\\,\\frac{1-\\nu}{\\nu} $$ The bracket phi-nu over one-minus-phi-nu carries the sign of phi, the autoregressive coefficient. Positive phi means today's return tends to follow yesterday's, a diverging or trending market, and the bracket is positive. Negative phi means returns reverse, a mean-reverting market, and the bracket goes negative. The drift term on the right is unchanged from the white-noise case. The strategy's edge is now literally the sign of a single autocorrelation parameter. Take a one-month span, 21 days, so nu is 1 minus 2 over 22, about 0.909, and l is root-21, about 4.58\. With a modest phi of 0.05 and zero drift, the bracket is 0.05 times 0.909 over one minus that product, which is 0.0455 over 0.9545, about 0.0476\. The constant c-1y works out near 1.11, so the annual return is about 5.3%. Flip phi to minus 0.05 and every term keeps its magnitude but the bracket turns negative, so the same system loses about 5% a year on the same volatility. Nothing changed except whether the market trended or reversed at the daily scale. ![Expected annual trend-following return and turnover as a function of filter span, for a mildly trending AR(1) process (phi = 0.05) and a mildly mean-reverting one (phi = -0.05)](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_419-ar1_return_vs_span.png) The left panel is the mirror image the theory predicts: the phi equals 0.05 curve sits entirely above zero and its phi equals minus 0.05 twin sits entirely below, roughly symmetric around it. There is also a lesson about span. For a trending market the return is highest at the shortest spans and decays as the filter slows, because a short filter reacts fast enough to ride short-lived momentum. For a mean-reverting market the loss is also deepest at short spans. The takeaway from the old article ["Trend and Reversion Are the Same, and OLS Understates Both"](https://aligrithm.com/trend-and-reversion-are-the-same-and-ols-understates-both/) lands here with a vengeance: trend and reversion are one process read at two signs of phi, and the same closed form prices both. A trend-follower is a short-mean-reversion position and nothing more. ## ARFIMA: trend can win even when the short term reverses Real markets are not AR(1). They mean-revert over days and trend over months, and a one-parameter model cannot hold both. The paper reaches for the fractionally-integrated ARFIMA process, which carries a long-memory parameter d controlling how slowly the autocorrelation decays, layered on top of a short-term AR(1) term phi. This is where the result earns its keep. ![Autocorrelation functions for AR(1) versus long-memory ARFIMA processes: AR(1) dies after a few lags, positive-d ARFIMA keeps a small positive tail out to fifty lags, negative-d ARFIMA keeps a negative tail](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_419-autocorrelation_shapes.png) Look at the shapes. Panel A, the plain AR(1), spikes at lag one and is dead by lag three; all its memory is short. Panel B, ARFIMA with positive d, has the same short-term spike but then a long, slowly decaying tail of small positive autocorrelations stretching out to fifty lags. Panel C, negative d, keeps a persistent negative tail. Now recall the closed form: the trend-follower sums nu-to-the-m times rho(m) over every lag. Even when the lag-one autocorrelation is negative (short-term reversion, the blue phi equals minus 0.1 line), a long positive tail from a positive d can outweigh it once you use a slow enough filter that reaches those far lags. That is the paper's most useful and least intuitive claim, and it drops straight out of the arithmetic. A market can reverse day-to-day and still be a profitable trend-following market, provided it trends over the long horizon and you run a slow filter. The weighted sum is what matters, not any single lag. Conversely, with negative d the long tail is negative and even a positive drift only rescues the strategy if the drift is large and the span is long. The formula turns "does trend-following work here" from a vibe into an integral you can evaluate against a process's autocorrelation function. This is the measurable version of the rhyme from the old article ["Why the Market Does Not Repeat, But Still Rhymes."](https://aligrithm.com/why-the-market-does-not-repeat-but-still-rhymes/) The rhyme a trend-follower harvests is not a chart shape you eyeball; it is the sign of a discounted sum of autocorrelations. If that sum is positive at your horizon, the market rhymes in your favor. If it is negative, no amount of discipline saves you. ## Turnover is fixed by the span, not the market One more result, because it decides whether any of the above survives costs. The expected volatility-adjusted turnover of the European system depends only on the filter span, not on the return process at all. $$ \\mathbb{E}\[a\\,U\_t\] = \\frac{2a}{\\sqrt{\\pi}}\\,\\sigma\_{target}\\,\\sqrt{1-\\nu}, \\qquad 1-\\nu \\approx \\frac{2}{span} $$ The expected annual turnover is proportional to root of one-minus-nu, which is proportional to one over the square root of the span. Halve nothing about the market; just shorten the filter and turnover, and therefore cost, climbs. For a one-month span, one-minus-nu is about 0.091 and the formula gives roughly 1300% annualized volatility-adjusted turnover. Stretch to a three-month span and it falls to about 780%. The right panel of the AR(1) figure confirms it: turnover decays with span regardless of phi. Here is the squeeze. Short filters capture short-term trending better (higher gross return when phi is positive) but they trade far more (higher cost). The paper's grid backtest lands on a long span of 250 and a short span of 20 as the practical optimum, where realized costs run about 1.7% a year, in line with a real CTA. Push faster and costs eat the extra gross. That trade-off is not a market opinion; it is baked into these two formulas. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_419-visual-0.png) ## Does the closed form survive real data A clean formula is worthless if the market's autocorrelation is too small to beat costs, so the honest question is the size of the effect. The paper runs all three systems on a broad liquid-futures universe. Net of transaction costs and a 2-and-20 fee load, the European, American, and TSMOM systems and the SG Trend index all cluster around a Sharpe of 0.45 from 2000 to 2025, the same as a plain 60/40 stock-bond portfolio. On the longer sample back to 1965 the net Sharpe is about 0.63\. These are not spectacular numbers. The genuine long-horizon autocorrelation in futures is real but thin, exactly what the discounted-sum formula would tell you to expect from a market that is mostly efficient. The value shows up somewhere else. All three systems are near-uncorrelated with 60/40 over the full sample but strongly anticorrelated in the worst quarters, so almost all of their Sharpe is earned in the bear regime. Blend 40% into a 60/40 book and the combined Sharpe roughly doubles to about 0.9 while the bear-regime drawdown collapses. Adding portfolio volatility targeting lifts the European system's long-run Sharpe from 0.77 to 1.12, at the cost of some positive skew. The closed form explains why: the payoff is convex and direction-agnostic because the drift enters squared, which is exactly the profile you want as a hedge against a long-only portfolio, not as a standalone alpha engine. Trend-following is a diversifier that happens to be free-standing, priced by the sign of an autocorrelation sum. Read that way, the modest Sharpe is a feature, not a disappointment. ## KEY POINTS - A European trend-follower's daily return is yesterday's signal times today's normalized return, so the strategy is a pure bet that positive signals precede positive returns. - The exact cumulative P&L factors into two drivers: a filter-weighted sum of the return autocorrelations minus one, plus the squared Sharpe of the underlying. The subtracted one is the cost of trading your own noise. - Trend-following makes money only when the average autocorrelation at your filter's horizons is positive. The sign of that sum, not any story about trends, decides the outcome. - Drift enters as its square, so the strategy is direction-agnostic and long-volatility; it profits equally from strong up and down moves, and longer lookbacks capture more of the drift. - Under white noise the edge is purely drift-squared and grows with the square root of the span. Under AR(1) the sign of the single coefficient phi flips profit to loss and back. - Under long-memory ARFIMA a market can mean-revert day-to-day and still be a profitable trend market, if it trends over long horizons and you run a slow filter, because the weighted autocorrelation sum can stay positive even when the first lag is negative. - Turnover depends only on the span, scaling with one over the square root of the lookback, which forces the trade-off between capturing fast trends and paying costs. The paper's optimum is a long span near 250, short near 20, at about 1.7% annual cost. - Net of fees the real edge is a Sharpe around 0.45, no better than 60/40 on its own, but its bear-regime, convex, uncorrelated payoff nearly doubles a 60/40 Sharpe when blended. It is a diversifier, not an alpha machine. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Systematic Trading - Robert Carver (Amazon)](https://www.amazon.com/s?k=Systematic+Trading+Robert+Carver&ref=aligrithm.com) - [Trading Systems - Urban Jaekle Emilio Tomasini (Amazon)](https://www.amazon.com/s?k=Trading+Systems+Urban+Jaekle+Emilio+Tomasini&ref=aligrithm.com) - [The Science and Practice of Trend-following Systems (The Science and Practice of Trend-following Systems)](https://scholar.google.com/scholar?q=The+Science+and+Practice+of+Trend-following+Systems&ref=aligrithm.com) ### 9.33 The $40M, Verified: Reading the Probabilistic-Forest Arbitrage Paper URL: https://aligrithm.com/the-40m-verified-reading-the-probabilistic-forest-arbitrage-paper/ Last updated: 2026-07-27T13:26:21.000Z Take a market with three mutually exclusive outcomes: a Republican wins New York, a Democrat wins, or a third party wins. Exactly one of those pays $1 at resolution, the other two pay nothing. So the three YES prices have to add up to $1\. If a trader can buy one share of each for a combined $0.95, they lock in a nickel of guaranteed profit before knowing who wins. That gap is not a forecast and not a bet. It is a bookkeeping error someone left on the table. Oriol Saguillo and coauthors went looking for those errors across a year of Polymarket data, April 2024 to April 2025, and put a number on them: about $40 million extracted. The old article ["The Marginal Polytope: One Shape That Contains Every Fair Price"](https://aligrithm.com/the-marginal-polytope-one-shape-that-contains-every-fair-price/) argued the theory of this, that every coherent set of prices has to live inside one convex shape, and that stepping outside it is a free lunch. This paper is the empirical audit. It counts how often prices stepped outside, how much money was sitting there, and whether anyone actually picked it up. My job here is to read the pipeline they built, check what the $40M is made of, and separate the parts that generalize from the parts that were a one-time election windfall. ## The one rule the whole thing rests on A Polymarket market is a question with a set of conditions, and the conditions are built to be exhaustive and mutually exclusive: one and only one resolves True. The price of a YES token is read as the market-implied probability of that outcome. Stack the definition together and you get a hard constraint on the prices. $$ \\sum\_{i} \\mathrm{val}(Y\_i, t) = 1 $$ This says the YES prices of all the conditions in a market, summed at any single moment t, must equal $1\. It is the same statement as "probabilities of a complete, mutually exclusive set of outcomes add to one," dressed in trading-token clothes. Work a number through it. A New York election market lists three conditions priced at Republican $0.42, Democrat $0.50, third party $0.03\. Those sum to $0.95, not $1\. The market is quoting a 95% chance that somebody wins the election it was built to cover, which is nonsense, because somebody wins with certainty. The missing 5 cents is the error, and the constraint is what tells you it is an error rather than an opinion. ## Two ways the sum breaks The paper splits arbitrage into two clean types by where the incoherence lives. The first is Market Rebalancing Arbitrage, inside a single market. When the YES prices sum below $1, you buy one share of every condition and wait. $$ \\text{Long: } \\sum\_{i} \\mathrm{val}(Y\_i, t) < 1 \\qquad \\text{profit} = 1 - \\sum\_{i} \\mathrm{val}(Y\_i, t) $$ Buying all three New York conditions costs $0.95\. One of them resolves to $1, the other two to $0\. You paid $0.95 and collect $1, so the guaranteed profit is $0.05 per set, independent of who actually wins. The mirror case runs the other way. When the YES prices sum above $1, the NO tokens are the underpriced side. $$ \\text{Short: } \\sum\_{i} \\mathrm{val}(Y\_i, t) > 1 \\qquad \\text{profit} = \\sum\_{i} \\mathrm{val}(Y\_i, t) - 1 $$ Say the same three conditions instead sum to $1.08\. Buy one NO for each condition. Across three mutually exclusive outcomes, exactly one YES wins, so exactly two NO tokens pay $1 each, giving $2 total. The NO tokens cost the complement of the YES prices, which is 3 minus $1.08, or $1.92\. You collect $2 against $1.92 paid, banking $0.08, again the exact size of the overshoot. Both directions pay you the distance between the price sum and $1, and the paper found the overwhelming majority of single-condition opportunities were long, meaning prices summed below $1. The second type is Combinatorial Arbitrage, and it spans two markets whose outcomes are logically linked. A market on "which party wins the state" and a market on "the winning margin" are not independent: winning by 5 or more points is a subset of winning at all, so its price can never sit above the price of winning. $$ \\text{Arbitrage if } \\quad \\sum\_{c \\in S} \\mathrm{val}(T\_c, t) \\neq \\sum\_{c' \\in S'} \\mathrm{val}(T\_{c'}, t) $$ The dependent subset S in one market and S' in the other must carry the same total probability, so any gap between the two sums is money. Concrete version: "Republican wins the state" trades at $0.55, while "Republican wins by 5 or more" trades at $0.60\. The narrower event is priced richer than the broader event that contains it, which is impossible if the market is coherent. Buy the cheap leg, take the opposing position on the rich leg, and the $0.05 gap is locked regardless of the final margin. The profit is the difference in the two dependent sums, same shape as rebalancing, just read across two order books instead of one. ## The scaling wall, and the triage that gets around it Finding the first type is cheap, because you only look inside one market. Finding the second type is where the paper earns its keep, because you have to check pairs of markets for hidden logical dependence, and the honest way to do that blows up fast. $$ \\text{naive comparisons per market pair} = O\\!\\left(2^{\\,n+m}\\right) $$ For two markets with n and m conditions, the joint space of True/False assignments has 2 to the power of n plus m entries, and something has to reason over all of them to decide whether the two markets constrain each other. Take a pair where each market has 5 conditions: 2 to the power of 10 is 1,024 combinations for that one pair. Now recall the election group alone held 46,360 candidate pairs. Enumerating 1,024 combinations across 46,360 pairs is roughly 47 million reasoning steps, and that is before you widen n and m past 5\. No human checks 46,360 pairs, which is the whole point of the old article "46,360 Pairs No Human Can Check." The authors cut the space with three filters stacked in order. First, timeliness: only compare markets that share the same end date, since Polymarket markets about the same event are created to resolve on the same day. Second, topic: run each market's question through the Linq-Embed-Mistral text embedder, the best open-source embedder on the MTEB leaderboard at the time, take cosine similarity against seven topic labels (Politics, Economy, Technology, Crypto, Twitter, Culture, Sports), and only compare markets inside the same topic. A hand-check of 100 labels put topic accuracy at 92%. Third, and this is the load-bearing trick, hand the reduced condition sets to a large language model, DeepSeek-R1-Distill-Qwen-32B, and ask it to return the set of valid joint outcomes as strict JSON, using chain-of-thought reasoning to work out which combinations are logically impossible. ![The dependency-detection pipeline: reduce each market to its outcome vectors, feed condition sets to the LLM, and validate the returned outcome space to label pairs independent or dependent](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_410-dependency_pipeline.png) The LLM has a real limit the paper does not hide: it chokes past four or five conditions. On the single-market sanity check, where every condition is dependent by construction, 128 markets went in and only 101 (81.45%) came back fully correct, with 4 returning invalid JSON and the rest failing the outcome-vector checks. When a market has too many conditions, the model falls into a reasoning loop and tries to spit out the full 2-to-the-power exhaustive table, the exact explosion the pipeline was built to avoid. The fix is blunt: reduce any market to its top four conditions by traded volume plus a fifth catch-all "everything else" condition. Over 90% of a market's liquidity sits in the top four conditions, so the reduction loses little, and the fifth condition preserves the logical dependencies by standing in as a logical OR of the tail. ![Triage funnel from 46,360 candidate election pairs down to 13 confirmed arbitrage relationships](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_410-pair_triage_funnel.png) The funnel is brutal. Of 46,360 election pairs, 353 returned no JSON and 4,374 returned malformed output, leaving valid results on the rest. The LLM tagged 40,057 pairs as independent and 1,576 as dependent. A subset checker cut the 1,576 down to 374 that could structurally hold arbitrage, and a manual pass over those 374 confirmed 13 pairs that satisfy the strict combinatorial definition, 11 NegRisk-to-NegRisk and 2 NegRisk-to-Single. Outside the election, 2,267 pairs produced exactly one dependent pair, and that one did not even meet the strict definition. Thirteen real relationships out of tens of thousands of candidates. Keep that ratio in mind when you read the headline number. ## Did anyone actually take it Opportunities existing is not money made. The paper reconstructs 86 million on-chain bids, computes a volume-weighted average price per token per block, and flags an opportunity when the absolute gap between the price sum and $1 exceeds two cents, restricting to moments when no outcome is already priced above $0.95 (so the event is still live and liquid). Then it checks which addresses actually held the winning basket of positions inside a roughly one-hour window. ![Volume-weighted YES and NO prices for a single condition over time, with the price-sum deviation from $1 and the moments arbitrageurs took the opportunity](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_410-arbitrage_window.png) The example condition shows the mechanism plainly: the YES and NO prices sit calm and coherent for weeks, then the sum lurches away from $1 during a burst of volatility, and blue markers show traders stepping in during exactly those windows. Arbitrage is a volatility phenomenon here. When nothing is happening, the book stays coherent and there is no gap to harvest. ![Realized profit by strategy, showing buying NO across markets and buying YES across markets dominate the roughly $40M total](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_410-realized_profit_breakdown.png) Adding the strategies up gives $39,587,585, call it $40 million, under the paper's assumption of at least $1 profit per counted trade. The composition matters more than the total. Buying NO positions across multi-condition markets brought in $17.3M and buying YES another $11.1M; single-condition rebalancing added $5.9M on the buy side and $4.7M on the sell side. Combinatorial arbitrage, the exotic cross-market type that needed the whole LLM pipeline to find, extracted about $95,000 across five of the 13 pairs. The clever part of the paper produced a rounding error of the profit. Almost all $40M is plain single-market rebalancing that needs no embeddings and no language model, only a script that adds up prices and notices they miss $1. The extraction is also concentrated. The top account cleared $2.0M across 4,049 trades, the top ten accounts show bot-like trade counts in the thousands, and only about 1% of the estimated election-window opportunities were ever taken. One trader, @Tutaaa91, bought both sides of a condition for under $0.02 each and booked $58,983 on a single trade when prices detached from any real probability. This is not a crowded, efficient market grinding out basis points. It is a thin field of a few automated players cleaning up obvious mistakes, with most of the theoretical money never touched. ## What survives the skepticism Three caveats keep this from being a repeatable strategy pitch. First, the window is a single 2024-25 US election cycle, the most liquid event in Polymarket's history at over $3.7 billion in volume, and the profit concentrates there. The old article "$40M, One Election Cycle: Where Did the Arbitrage Come From?" makes the point that this is a snapshot, not a run rate. Strip the election and the numbers shrink hard. Second, every leg is non-atomic. On an order-book venue you place one order, then another, and the second can fail after the first fills, so the "guaranteed" profit carries real execution risk that the clean inequalities above ignore. This is not a DeFi flash-loan swap that reverts if any leg misses. You can get stuck holding half a position. Third, the $40M is a floor built from volume-weighted average prices, which smooth away the sharpest intrabar dislocations, so the paper underestimates the margin an alert bot actually captured, while also counting gross profit with no accounting for the capital tied up or the tail of trades that lost. The honest read: incoherence on Polymarket is real, common in single markets, and mostly harvested by a handful of bots during volatility spikes, while the cross-market combinatorial arbitrage that makes the paper novel is rare, hard to find, and barely worth money so far. The pipeline is the contribution. The $40M is mostly the easy kind. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_410-visual-0.png) ## Where this connects This paper is the empirical body attached to the theory head of the old article ["The Marginal Polytope: One Shape That Contains Every Fair Price."](https://aligrithm.com/the-marginal-polytope-one-shape-that-contains-every-fair-price/) The polytope says coherent prices live inside one convex hull; Saguillo and coauthors measured how far and how often Polymarket prices sat outside it, and who profited from dragging them back in. The combinatorial-explosion filter, temporal proximity plus topical embeddings plus an LLM reading logical dependencies, is the practical answer to the structural moat described in the old article "46,360 Pairs No Human Can Check," and the composition of the profit is the receipt for the framing in the old article "$40M, One Election Cycle: Where Did the Arbitrage Come From?" Read together, the lesson is narrow and useful: the coherence constraint is enforceable and enforced, but the enforcement is thin, automated, and concentrated in the loudest event of the year, not a standing river of alpha you can wade into. ## KEY POINTS - A prediction market's mutually exclusive YES prices must sum to $1\. When they sum to less, buy one share of each condition and keep the shortfall; when they sum to more, buy the NO side and keep the overshoot. The profit is the exact distance from $1. - Market Rebalancing Arbitrage lives inside one market and is trivial to spot. Combinatorial Arbitrage spans two logically linked markets, where a narrower outcome is priced above the broader outcome that contains it. - Checking every market pair for hidden dependence scales like 2 to the power of n plus m per pair, which is hopeless across 46,360 election pairs. The paper cuts it with same-end-date filtering, Linq-Embed-Mistral topic embeddings, and a DeepSeek LLM that returns valid joint outcomes as JSON. - The LLM breaks past four or five conditions and falls into reasoning loops, so markets are reduced to their top four conditions by volume plus a catch-all fifth, which holds over 90% of liquidity. - The funnel goes 46,360 candidate pairs to 1,576 dependent to 374 structural candidates to 13 confirmed combinatorial relationships. Real cross-market arbitrage is rare. - The realized total is about $40M over April 2024 to April 2025, but roughly $39.9M of it is plain single-market rebalancing. Combinatorial arbitrage, the novel part, extracted only about $95,000 across five pairs. - Extraction is concentrated in a few bot-like accounts, only about 1% of opportunities were ever taken, every leg is non-atomic and carries execution risk, and the whole figure is a single election-cycle snapshot, not a run rate. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Convex Optimization - Stephen Boyd Lieven Vandenberghe (Amazon)](https://www.amazon.com/s?k=Convex+Optimization+Stephen+Boyd+Lieven+Vandenberghe&ref=aligrithm.com) - [An Introduction to Copulas - Roger Nelsen (Amazon)](https://www.amazon.com/s?k=An+Introduction+to+Copulas+Roger+Nelsen&ref=aligrithm.com) - [Unravelling the Probabilistic Forest: Arbitrage in Prediction Markets (probabilistic forest prediction markets)](https://scholar.google.com/scholar?q=Unravelling+the+Probabilistic+Forest%3A+Arbitrage+in+Prediction+Markets&ref=aligrithm.com) ### 10.15 Network Momentum as a Cross-Asset Factor URL: https://aligrithm.com/network-momentum-as-a-cross-asset-factor/ Last updated: 2026-07-10T10:57:25.000Z Momentum is the one factor nobody argues about. Winners keep winning, losers keep losing, and the effect shows up in stocks, bonds, commodities, and currencies across a century of data. The old article ["From Intermarket Analysis to Network Momentum"](https://aligrithm.com/from-intermarket-analysis-to-network-momentum/) pushed a harder claim: an asset's momentum can leak into the assets it is linked to, so the return of one contract carries information about the next one's future. That was the thesis. Pu, Roberts, Dong, and Zohren built the machine that tests it on 64 futures across four asset classes, and the payoff is a strategy with a Sharpe of 1.51 and 22% a year, out of sample, from 2000 to 2022, using nothing but prices. The trick the paper solves is the one that stopped everyone before it. Momentum spillover was documented between a company's stock and its bond, or between firms in the same supply chain, because those pairs share an obvious economic tie. Cotton and the ten-year Treasury share nothing you can write down. So instead of hand-coding links, the authors learn the graph straight from the momentum data, then propagate each asset's signal along the learned edges. This is the cross-sectional, multi-asset version of the lead-lag idea the old article ["Lead-Lag Relationships in Global Markets"](https://aligrithm.com/lead-lag-relationships-in-global-markets/) introduced, with the graph itself as the unit of analysis instead of a single pair. ## Step one: eight momentum features per asset Before any network exists, each asset needs a momentum description. The paper uses eight features per contract per day: volatility-scaled returns over 1 day, 1 month, 3 months, 6 months, and 1 year, plus three normalized MACD indicators tuned to short, medium, and long time scales. Volatility-scaling matters because it puts a quiet bond and a wild commodity on the same axis. You divide each return by the asset's own recent volatility, so a signal of "+1" means the same size of move everywhere. $$ \\text{feature}\_{i,t}(\\Delta) = \\frac{r\_{i,\\,t-\\Delta:t}}{\\sigma\_{i,t}\\sqrt{\\Delta}}, \\qquad \\Delta \\in \\{1, 21, 63, 126, 252\\}\\ \\text{days} $$ Read it as the return of asset i over the last delta days, divided by that asset's daily volatility times the square root of the horizon. Sigma is an exponentially weighted moving standard deviation with a 60-day span, so recent days count more. Say crude oil returned 8% over the past month (21 days) and its daily volatility is 2%. The denominator is 0.02 times the square root of 21, about 0.0917, so the feature is 0.08 over 0.0917, roughly 0.87\. A gold contract that also returned 8% but at 1% daily vol scores about 1.74, twice as strong, because the same move is more impressive in a calmer asset. Stack the eight features for every asset and you have the raw material the network is built from. ## Step two: learn the graph from the features The links are learned, not assumed. Each day, the model stacks the feature matrix over a lookback window and solves a convex optimization for the adjacency matrix A, the table of edge weights that says how strongly every pair of assets is connected. $$ \\min\_{A\_t}\\ \\operatorname{tr}\\!\\left(V\_t^\\top (D\_t - A\_t)\\, V\_t\\right)\\; -\\; \\alpha\\,\\mathbf{1}^\\top \\log(A\_t \\mathbf{1})\\; +\\; \\beta\\,\\lVert A\_t \\rVert\_F^2 $$ $$ \\text{s.t.}\\quad A\_{ij,t} = A\_{ji,t},\\quad A\_{ij,t} \\ge 0,\\quad A\_{ii,t} = 0 $$ The first term, the trace, is the Laplacian quadratic form. It equals a weighted sum over pairs of the edge weight times the squared distance between the two assets' feature vectors. Minimizing it puts big edges between assets whose momentum features look alike and near-zero edges between assets that do not. The middle log term forces every node to keep at least some connection, so nothing gets stranded. The last term, the squared Frobenius norm scaled by beta, penalizes huge weights and keeps the graph smooth. The constraints make the matrix symmetric, non-negative, and free of self-loops. Work the intuition on two assets. If contract i and contract j have almost identical eight-feature vectors, the squared distance between them is tiny, so the optimizer pays almost nothing to set a large edge weight A between them, and it does. If their feature vectors point in different directions, the distance is large, and any weight there gets multiplied by a big number, so the optimizer drives that edge to zero. Two knobs, alpha and beta, tune how sparse the result is, picked by grid search on in-sample data. Smaller values give a sparser graph, which means each asset hears from only a few others and less noise propagates. The authors then average five graphs from lookback windows of one to five years to steady the weights, and the resulting network is stable: the Jaccard overlap of edges from one day to the next stays above 0.99 outside crises. ## Step three: propagate, then regress With the graph in hand, network momentum for a target asset is the edge-weighted average of its neighbors' momentum features. You do not use the asset's own past returns at all; you borrow the momentum of everything it connects to. $$ \\tilde{u}\_{i,t} = \\sum\_{j \\,\\in\\, \\mathcal{N}(i)} \\tilde{A}\_{ij,t}\\, u\_{j,t}, \\qquad\\qquad y\_{i,t} = \\tilde{u}\_{i,t}^\\top \\beta + b $$ The first equation says the network momentum feature vector for asset i is the sum over its neighbors j of the normalized edge weight times neighbor j's own feature vector. The second feeds that propagated vector into one cross-sectional OLS regression, trained across all assets at once, to predict the next day's volatility-scaled return. Its sign becomes the trade: positive means go long, negative means go short. Take a scalar version. Suppose a target contract connects to three assets with normalized weights 0.5, 0.3, and 0.2, and their one-month volatility-scaled momentum reads +1.2, −0.5, and +0.8\. The propagated signal is 0.5 times 1.2, plus 0.3 times −0.5, plus 0.2 times 0.8, which is 0.60 minus 0.15 plus 0.16, so +0.61\. Positive, so you go long the target, even if the target's own recent return was flat or negative. That is the whole point: the position comes from the neighborhood, not the asset. The regression across eight features per neighbor is what lets the model weigh a 1-day signal against a 1-year signal and net them into one call. The book is then volatility-targeted. Each position is sized inversely to the asset's own volatility so every name contributes the same risk, and the portfolio is scaled to a 15% annual volatility target. $$ r^{\\text{port}}\_{t:t+1} = \\frac{1}{N\_t}\\sum\_{i=1}^{N\_t} \\operatorname{sign}(y\_{i,t})\\,\\frac{\\sigma\_{\\text{tgt}}}{\\sigma\_{i,t}}\\, r\_{i,t:t+1}, \\qquad \\sigma\_{\\text{tgt}} = 0.15 $$ The sign of the prediction sets direction, the ratio of target volatility to the asset's volatility sets size, and you average over all assets. A contract running at 30% annualized volatility gets a scale of 0.15 over 0.30, so half a unit; one at 10% gets 1.5 units. Same risk budget per name, which is standard trend-following hygiene. ## The result: network momentum is its own signal Run the whole thing out of sample from 2000 to 2022, retraining every five years, and the network momentum portfolio (the paper calls it GMOM) clears every benchmark. Volatility-scaled to the same 15% target, a long-only book scores a Sharpe of 0.73, a plain MACD rule 0.70, and individual momentum run through the identical regression (LinReg) 1.13\. Network momentum hits 1.51, at 22% annual return, with the smallest drawdown and the shortest time underwater of the group. ![Sharpe ratios of long-only, MACD, individual momentum, and network momentum at a 15% volatility target](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_409-strategy_sharpe.png) The comparison that matters is network momentum against individual momentum, because they eat the exact same eight features. The only difference is that network momentum propagates them across the graph first. Their return correlation is about 0.65 and they agree on direction 72% of the time, so network momentum is not a repackaging of the individual signal; it captures something the plain version misses. Averaging the two positions does not beat network momentum alone, which tells you the network version already contains most of what the individual version knows, plus extra. Stay honest about what "1.51" includes. These are gross returns before costs, and the strategy rebalances daily, so turnover is high. The paper runs the cost sensitivity: network momentum stays positive up to a pseudo-cost of 3 basis points per trade and holds a Sharpe above 0.5 up to 2 basis points, but at 5 basis points it turns negative (−0.67), though it still beats individual momentum, which craters to −1.05 at the same cost. The edge is real in the data and more cost-robust than the individual version, but it lives or dies on execution, and nobody should read 1.51 as a number you keep. ## Where the alpha actually comes from: the links between classes The finding that earns this a place in the factor pillar is not the headline Sharpe. It is where the return comes from. The authors run the ablation you would demand: rebuild network momentum inside each asset class alone (a single-class graph), then compare it to the same class traded off the full cross-class graph. Cross-class links win in every class. ![Sharpe by asset class from a single-class network versus a cross-class network](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_409-cross_vs_single_class.png) Equities are the clean tell. Built from an equity-only network, the momentum signal scores a Sharpe of 0.30\. Built from the full cross-class network, the same equity trades score 0.95, more than triple. Currencies go from 0.16 to 0.66, fixed income from 0.78 to 1.23, commodities from 0.81 to 0.90\. The information that helps you trade equities is not sitting in other equities; it is sitting in the bonds, commodities, and currencies that share a momentum regime with them. That is the empirical case for treating the cross-asset graph, not the single instrument, as the thing you analyze, which is exactly the graph-as-unit argument the Pillar 8 chapters on networks make. One caution the authors surface themselves. Splitting the graph into intra-class edges only versus inter-class edges only, the intra-class version (Sharpe 1.21) beats the inter-class version (0.91), so connections within a class still carry the most weight, and the pure inter-class book took a hard hit in the 2007 to 2008 crisis. The two are only about 0.51 correlated, which is why combining them into the full graph beats either piece. Inter-class links are the ingredient that makes it better than single-class momentum; they are not the whole meal. ## Reversals show up, and the regression catches them Individual momentum has a well-known short-horizon reversal: last month's winners tend to give some back next month. Whether that reversal survives in network momentum was an open question, and the single-feature strategies could not answer it because they only carried one horizon. Feeding eight horizons into a regression lets the coefficients speak. The past 1-day return loads positive and significant in both individual and network momentum, so very recent moves continue. The past 1-month return loads negative, the classic reversal, in both. In the network version, the 3-month and 1-year signals also turn negative after 2010, which the plain individual momentum does not show, so the reversal structure of network momentum differs from the individual kind. The regression is not just a predictor here; it is a diagnostic that tells you which horizons trend and which snap back, and it lets the model net them instead of betting blindly on continuation. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_409-visual-0.png) ## What to keep and what to distrust Network momentum is a genuine second signal, not a relabeled trend follower, and the mechanism has an economic story: momentum regimes spill across asset classes even when no company-level tie connects the instruments. The graph-learning approach is interpretable and needs only prices, which is rare for something this good on paper. The cross-class ablation is the part worth stealing even if you never trade the full system, because it says your equity book should be listening to bonds and commodities, and it measures how much that listening is worth. The distrust is the usual short-horizon distrust. Daily rebalancing, gross Sharpe, and a cost cliff past 3 basis points mean the live version depends entirely on cheap execution and turnover control, which the authors flag and leave to future work. The training is a five-year rolling refit with a grid search over the sparsity knobs, so some of the elegance hides ordinary hyperparameter tuning. And 22 years out of sample spans only a handful of real regimes, so the 1.51 is one draw, not a law. Reproduce the number the way this pillar always tells you to: rebuild the cross-class-versus-single-class gap net of your own costs, because that gap, not the headline Sharpe, is the actual claim. ## KEY POINTS - Network momentum trades an asset off the momentum of the assets it is linked to, not its own past returns. Winners' neighbors tend to win next. - The links are learned from prices, not assumed. A convex graph-learning objective connects assets whose eight momentum features (vol-scaled returns at five horizons plus three MACDs) look alike, and zeros out the rest. - A target's signal is the edge-weighted average of its neighbors' features, fed through one cross-sectional OLS regression whose sign sets the long or short position, then volatility-targeted to 15%. - Out of sample 2000 to 2022, the network momentum book (GMOM) posts a Sharpe of 1.51 and 22% a year, beating individual momentum (1.13), a MACD rule (0.70), and long-only (0.73), with the smallest drawdown. - It is a distinct signal: about 0.65 correlated with individual momentum and agreeing on direction only 72% of the time, so it adds information rather than repackaging it. - The alpha lives in the cross-class links. Trading each class off the full cross-asset graph beats trading it off a single-class graph in every class: equities jump from a Sharpe of 0.30 to 0.95, currencies from 0.16 to 0.66. - Intra-class edges still carry the most weight (Sharpe 1.21 versus 0.91 for inter-class alone), but the two are only 0.51 correlated, so the full graph beats either piece. - The regression exposes reversals: 1-day returns continue, 1-month returns reverse, and in the network version longer horizons turn negative after 2010, differing from individual momentum. - These are gross returns with daily turnover. The strategy stays positive only up to about 3 basis points of cost and goes negative at 5\. Reproduce the cross-class gap net of your own costs before trusting it. ## References - [Value and Momentum Everywhere (Asness, Moskowitz, and Pedersen, 2013)](https://onlinelibrary.wiley.com/doi/10.1111/jofi.12021?ref=aligrithm.com) - [Stock and Bond Market Interaction: Does Momentum Spill Over? (Gebhardt, Hvidkjaer, and Swaminathan, 2005)](https://www.sciencedirect.com/science/article/abs/pii/S0304405X04001291?ref=aligrithm.com) - [Economic Links and Predictable Returns (Cohen and Frazzini, 2008)](https://onlinelibrary.wiley.com/doi/10.1111/j.1540-6261.2008.01379.x?ref=aligrithm.com) - [Shared Analyst Coverage: Unifying Momentum Spillover Effects (Ali and Hirshleifer, 2020)](https://www.sciencedirect.com/science/article/abs/pii/S0304405X20300118?ref=aligrithm.com) - [How to Learn a Graph from Smooth Signals (Kalofolias, 2016)](https://proceedings.mlr.press/v51/kalofolias16.html?ref=aligrithm.com) - [Learning Laplacian Matrix in Smooth Graph Signal Representations (Dong, Thanou, Frossard, and Vandergheynst, 2016)](https://ieeexplore.ieee.org/document/7472871?ref=aligrithm.com) - [Enhancing Time Series Momentum Strategies Using Deep Neural Networks (Lim, Zohren, and Roberts, 2019)](https://jfds.pm-research.com/content/1/4/19?ref=aligrithm.com) - [Understanding Momentum and Reversal (Kelly, Moskowitz, and Pruitt, 2021)](https://www.sciencedirect.com/science/article/abs/pii/S0304405X21000027?ref=aligrithm.com) - [Network Momentum across Asset Classes (Pu, Roberts, Dong, and Zohren, 2023)](https://arxiv.org/abs/2308.11294?ref=aligrithm.com) --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ### 10.14 Price-Path Convexity: A New Cross-Sectional Anomaly URL: https://aligrithm.com/price-path-convexity-a-new-cross-sectional-anomaly-45bp-per-s/ Last updated: 2026-07-10T10:57:26.000Z Take two stocks that both end the month flat. The first bled lower for two weeks, then clawed all the way back. The second ran up for two weeks, then gave it all back. Same start, same finish, same zero return. Sort every stock in the market by past return and these two land in the identical bucket, because return only sees the endpoints. Gulen and Woeppel show that the shape between the endpoints predicts next month, and it predicts it hard: the stock that ended by rising underperforms the stock that ended by falling by about 0.84% a month, and no standard factor touches the spread. The old article ["Wave Velocity and Acceleration: Reading When the Market Runs Out of Gas"](https://aligrithm.com/wave-velocity-and-acceleration-reading-when-the-market-runs-out-of-gas/) argued that the second derivative of a price path carries an early read the level and slope miss. This is the cross-sectional version of that claim, tested on sixty years of the entire U.S. market instead of one instrument's sine fit. Return is the first derivative. Convexity, the curvature of the path, is a scaled cousin of the second derivative, and it prices the cross-section on its own. ## What convexity measures, and how to compute it Convexity compares two summaries of the same price path. Take the midpoint of the first and last closing price of the month. Subtract the average of every daily close in the month. Divide by the midpoint to standardize across price levels. That difference is the whole variable. $$ \\text{Convexity}\_{it} = \\frac{\\dfrac{P\_{1t} + P\_{N\_{it}}}{2} - \\dfrac{1}{N\_{it}}\\sum\_{k=1}^{N\_{it}} P\_{kt}}{\\dfrac{P\_{1t} + P\_{N\_{it}}}{2}} $$ Read it as midpoint of the endpoints, minus mean of the closes, over the midpoint. P-one is the first close of the month, P-N is the last close, and the middle sum is the average of all the closes in between. When the average of the closes sits below the midpoint of the endpoints, the path sagged in the middle and recovered, so convexity is positive and the path is convex, a valley. When the average sits above the midpoint, the path bulged up in the middle and faded, so convexity is negative and the path is concave, a hump. ![Convex versus concave price paths, showing the midpoint of the endpoints against the mean of the daily closes](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_408-convexity_geometry.png) Work the paper's own example, in dollars. A stock opens the month at 20, drops 1 a day for ten days down to 10, then climbs 1 a day for ten days back to 20\. First close and last close are both near 20, so the midpoint is 20\. The average of all the closes is dragged down toward 15 by the trough. Midpoint minus mean is about 4.8, and 4.8 over 20 is roughly 0.24\. That is a strongly convex path. Now steepen the front half: drop 1.50 a day for ten days to 5, then climb 1 a day, ending at 15\. The average sinks further, the midpoint falls to 17.5, and convexity rises to about 0.42\. A return measure would call both paths the same, since it only reads start and end. Convexity separates them because it weighs the whole trajectory. The authors build it from price changes in dollars, not percentage returns, on purpose. Investors think about a stock in dollar terms as much as in percent terms (Shue and Townsend document the non-proportional thinking), and dollar changes track the path's actual dynamics better than a return that collapses to its endpoints. _This post is for paying subscribers only._ ### 10.10 Behavioral Factors: Prospect-Theory Value and Capital-Gains Overhang URL: https://aligrithm.com/behavioral-factors-prospect-theory-value-and-capital-gains-overhang/ Last updated: 2026-07-10T10:57:27.000Z Most factor stories start with risk. You earn a premium because you hold something painful, and the pain is compensated. Behavioral factors flip the logic. Here the premium exists because other people are making a predictable mistake, and the mistake is old, robust, and written into how humans weigh gains against losses. Kahneman and Tversky mapped that machinery half a century ago, and it turns out you can turn their psychology into two tradable numbers: a score for how attractive a stock looks to a loss-averse gambler, and a score for whether the average holder is sitting on a gain or a loss. The punchline is uncomfortable for anyone who thinks appeal and return go together. Rank stocks by how good they look through the lens of prospect theory, and the most attractive decile earns a four-factor alpha of minus 0.21% a month while the least attractive earns plus 1.03%. Long the ugly, short the pretty, and the spread is 1.24% a month with a t-statistic of 6.83\. The market pays you to hold what behavioral investors cannot stomach. The old article ["The Flawed Human Brain in Trading"](https://aligrithm.com/the-flawed-human-brain-in-trading/) catalogued these biases as things to purge from your own decisions. This article does the other half: once you accept the brain is broken in known ways, you can price the breakage. It also connects to the old article ["When an Alpha Metric Is U-Shaped,"](https://aligrithm.com/when-an-alpha-metric-is-u-shaped/) because the behavioral effects here are conditional, they flip sign depending on whether holders are winning or losing, and a naive linear read misses them entirely. ## The value function: losses hurt more, and both ends flatten Prospect theory replaces the textbook idea that people maximize final wealth. Instead they react to changes from a reference point, and the reaction is asymmetric. The cumulative-prospect-theory value function writes it down in closed form. $$ v(x) = \\begin{cases} x^{c} & \\text{if } x \\geq 0, \\\\ -\\kappa\\,(-x)^{c} & \\text{if } x < 0\. \\end{cases} $$ Here x is the outcome measured against the reference point, a gain if positive and a loss if negative. The exponent c, between 0 and 1, bends both arms of the curve so that the tenth extra dollar matters less than the first, which is diminishing sensitivity. The coefficient kappa, bigger than 1, is loss aversion, and it steepens the loss arm so a loss stings more than the equal gain pleases. Tversky and Kahneman's estimates put c around 0.88 and kappa around 2.25. Work a symmetric bet. A 10% gain scores 10 to the power 0.88, which is about 7.6 units of value. A 10% loss scores minus 2.25 times that same 7.6, which is minus 17.1\. The loss registers 2.25 times as hard as the identical gain. Put a coin flip between plus 10% and minus 10% through this and its perceived value is minus 4.8, deeply negative, even though its expected payoff is zero. That single asymmetry is why people refuse fair gambles and, later, why they cannot sell losers. ![Prospect-theory value function, concave over gains and convex and steeper over losses, with c equal to 0.5 and kappa equal to 2.5](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_404-value_function.png) The curve makes the shape concrete. Above the reference point it is concave, so people are risk-averse in gains and grab a sure profit. Below it the curve is convex and much steeper near zero, so people turn risk-seeking in losses, holding a bad position and hoping to get back to even. That is the disposition effect in one picture, and it is the same instinct the old article ["Why Traders Take Profits Too Early and Losses Too Late"](https://aligrithm.com/why-traders-take-profits-too-early-and-losses-too-late/) warned you about from the inside. ## The weighting function: rare events get overweighted The second engine is that people do not use raw probabilities. They run probabilities through a weighting function that overweights the unlikely and underweights the near-certain. $$ w^{-}(p) = \\frac{p^{\\delta}}{\\left(p^{\\delta} + (1-p)^{\\delta}\\right)^{1/\\delta}} $$ The gain-side twin w-plus uses an exponent gamma instead of delta, but the form is identical. p is the true probability of an outcome, and w of p is the decision weight it actually gets. When delta is below 1, the function is an inverse-S: it lifts small probabilities above the 45-degree line and pushes large ones below. Tversky and Kahneman estimated gamma at 0.61 and delta at 0.69. Run a 1% tail through it with delta at 0.69\. The numerator, 0.01 to the power 0.69, is about 0.037\. The denominator works out near 0.68, so the decision weight is roughly 0.054\. A one-in-a-hundred event gets treated like a one-in-twenty event, overweighted more than fivefold. That is the whole economics of lottery tickets and cheap out-of-the-money options: people pay for the fat right tail they imagine is fatter than it is. ![Probability weighting function for three values of delta, showing the inverse-S that overweights small probabilities and underweights large ones](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_404-weighting_function.png) The chart shows the effect getting stronger as delta falls. In markets this is why right-skewed "lottery stocks" and hot IPOs command high prices and then deliver low returns. Demand for the imagined jackpot bids the price up, and the future return pays for it. ## Assembling the TK score of a stock Barberis, Mukherjee and Wang turned these two functions into a single number per stock. You cannot know a stock's future return distribution, so they use its past 60 monthly returns as the outcome set, each with probability one over 60, and score that distribution the way a prospect-theory agent would. $$ \\text{TK} = \\sum\_{i=-m}^{-1} v(r\_i)\\left\[w^{-}\\!\\left(\\tfrac{i+m+1}{60}\\right) - w^{-}\\!\\left(\\tfrac{i+m}{60}\\right)\\right\] + \\sum\_{i=1}^{n} v(r\_i)\\left\[w^{+}\\!\\left(\\tfrac{n-i+1}{60}\\right) - w^{+}\\!\\left(\\tfrac{n-i}{60}\\right)\\right\] $$ Read it as a weighted average of felt value. Sort the 60 returns, run each through the value function v to get how it feels, and weight it by the decision weight from w, computed cumulatively so the extreme months in each tail get the overweighting the theory demands. The left sum handles losses with w-minus, the right sum handles gains with w-plus. A stock with a big right tail and mild losses scores high, it looks like a great prospect. Barberis and coauthors deliberately fixed the parameters at Tversky and Kahneman's originals, c 0.88, kappa 2.25, gamma 0.61, delta 0.69, so there is no fitting, no knobs to snoop. Because attractive stocks get overbought, TK should predict returns negatively, and it does, monotonically. ![Bar chart of four-factor alpha across TK-value deciles, declining from plus 1.03% for the lowest to minus 0.21% for the highest](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_404-tk_decile_alpha.png) The lowest-TK decile earns a four-factor equal-weighted alpha of 1.03% a month at a t of 6.05, and the alpha falls almost straight across the deciles to minus 0.21% at a t of minus 3.19 for the highest. The long-short spread is 1.24% a month with a t of 6.83, and it survives a five-factor model at 1.30%. Value-weighted the spread shrinks to 0.62% but stays significant at a t of 3.67, which tells you the effect is not purely a microcap artifact. When Barberis and coauthors switched the components off one at a time, the probability weighting piece did most of the work, the overweighted tails matter more than loss aversion for the cross-section. ## Capital-gains overhang: are the holders winning or losing The value function has a second market consequence through the disposition effect. Investors sitting on gains rush to sell and lock in the win, which pushes those stocks down and leaves them undervalued, so they earn higher future returns. Investors sitting on losses refuse to sell, propping the price up and leaving the stock overvalued, so it earns lower future returns. To trade it you need to know the crowd's average cost, and Grinblatt and Han built the proxy. $$ RP\_t = \\frac{1}{k}\\sum\_{n=1}^{T}\\left(V\_{t-n}\\prod\_{\\tau=1}^{n-1}(1 - V\_{t-n+\\tau})\\right)P\_{t-n}, \\qquad CGO\_t = \\frac{P\_t - RP\_t}{P\_t} $$ The reference price RP is a weighted average of past prices, where each past price P at lag n is weighted by the chance that shares bought then have not turned over since, using the turnover ratio V, with k a normalizer that makes the weights sum to one. High recent turnover means old prices are unlikely to still be held, so they get little weight. Capital-gains overhang is then just the percentage distance of today's price above that average cost. Put numbers on it. A stock trades at 50, the turnover-weighted reference price is 40, so CGO is 50 minus 40 over 50, which is 0.20\. The average holder is up 20%, deep in the concave, sell-and-lock-in region of the value function, so disposition selling should press the price and set up higher future returns. A stock at 30 against a reference of 40 has a CGO of minus 0.25, holders underwater and clinging, price propped, lower future returns ahead. The overhang is not just a standalone signal, it conditions other anomalies, and this is where the U-shape shows up. Sort by risk and by CGO together, and the low-risk anomaly reverses sign across the overhang. Among stocks whose holders are deep in losses, high-idiosyncratic-volatility names underperform low-vol names by 1.92% a month, the familiar low-risk anomaly in force. Among stocks whose holders are sitting on gains, the relationship flips positive. The difference-in-difference is 2.24% a month at a t of 7.97\. The risk-return line is not flat or downward everywhere, it bends one way or the other depending on whether the marginal holder is chasing a recovery or protecting a profit. Read risk on its own and you average two opposite regimes into mush. ## Where this connects Behavioral factors are the mirror image of the discipline articles. The old article ["Get-Even-Itis: The Most Expensive Disease in Trading"](https://aligrithm.com/get-even-itis-the-most-expensive-disease-in-trading/) described refusing to sell a loser as a personal failure. Capital-gains overhang is that same refusal, measured across every holder of every stock and priced into a long-short portfolio. The bias you are told to remove from your own trading is the exact edge you harvest from everyone who did not remove it. Prospect theory sits underneath both: the concave-convex value function generates the disposition effect that becomes overhang, and the overweighted tails generate the lottery demand that becomes the TK effect. Two cautions keep this honest. First, these are behavioral premiums, so they depend on limits to arbitrage, if the mispricing were trivially shortable it would already be gone, and short constraints on ugly lottery stocks are exactly why it is not. Second, the effects live in the cross-section and lean on small and hard-to-trade names, the value-weighted TK spread is half the equal-weighted one, so transaction costs and capacity matter before you size anything. The signal is real and old and grounded in measured psychology. That does not make it free. ## KEY POINTS - Behavioral factors pay a premium because other investors make predictable, well-documented mistakes, not because you hold extra risk. The edge is the bias everyone is told to avoid. - The prospect-theory value function is concave over gains and convex and steeper over losses, with loss aversion near 2.25, so a loss hurts about twice as much as an equal gain feels good. This drives risk-seeking in losses and the disposition effect. - The probability weighting function is an inverse-S that overweights rare events. A 1% tail is treated like roughly 5%, which is why lottery stocks and hot IPOs are overpriced and then underperform. - The TK score runs a stock's past 60 monthly returns through both functions with fixed Tversky-Kahneman parameters. Higher TK means more attractive, and attractive predicts low returns: the low-minus-high decile spread is a 1.24% monthly four-factor alpha at t 6.83. - Probability weighting, not loss aversion, does most of the cross-sectional work in the TK effect, and the value-weighted spread of 0.62% shows it is not only a microcap story. - Capital-gains overhang, CGO equals price minus a turnover-weighted reference price over price, proxies whether the average holder is winning or losing. Positive overhang predicts higher future returns, negative predicts lower, via the disposition effect. - Overhang conditions other anomalies. The low-risk effect reverses sign across CGO, a difference-in-difference of 2.24% a month, so the risk-return relationship is regime-dependent rather than linear. ## References - [Prospect Theory: An Analysis of Decision under Risk (Kahneman and Tversky, 1979)](https://www.jstor.org/stable/1914185?ref=aligrithm.com) - [Advances in Prospect Theory: Cumulative Representation of Uncertainty (Tversky and Kahneman, 1992)](https://link.springer.com/article/10.1007/BF00122574?ref=aligrithm.com) - [Prospect Theory and Stock Returns: An Empirical Test (Barberis, Mukherjee, and Wang, 2016)](https://doi.org/10.1093/rfs/hhw049?ref=aligrithm.com) - [Prospect Theory, Mental Accounting, and Momentum (Grinblatt and Han, 2005)](https://doi.org/10.1016/j.jfineco.2004.10.006?ref=aligrithm.com) - [The Disposition Effect and Underreaction to News (Frazzini, 2006)](https://doi.org/10.1111/j.1540-6261.2006.00896.x?ref=aligrithm.com) - [Lottery-Related Anomalies: The Role of Reference-Dependent Preferences (An, Wang, Wang, and Yu, 2020)](https://doi.org/10.1287/mnsc.2018.3205?ref=aligrithm.com) - [Stocks as Lotteries: The Implications of Probability Weighting for Security Prices (Barberis and Huang, 2008)](https://www.aeaweb.org/articles?id=10.1257/aer.98.5.2066&ref=aligrithm.com) - [Mispricing Factors (Stambaugh and Yuan, 2017)](https://doi.org/10.1093/rfs/hhw107?ref=aligrithm.com) - [Behavioral Finance and Factor Investing (Shi, 2024)](https://papers.ssrn.com/sol3/papers.cfm?abstract%5Fid=5268293&ref=aligrithm.com) --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ### 10.13 Alternative Data and the Short-Horizon Decay Tax URL: https://aligrithm.com/alternative-data-and-the-short-horizon-decay-tax/ Last updated: 2026-07-10T10:57:27.000Z In October 2018 a firm called Thasos drew a digital fence around Tesla's 370-acre Fremont factory, leased trillions of smartphone location pings, and counted the phones inside the fence overnight. The overnight shift had swelled 30% from June to October. Its hedge-fund clients bought Tesla before the Q3 earnings report, and when the report landed the stock jumped 9.14% on a surprise profit. That is the fantasy alternative data sells: see the trucks, count the phones, read the earnings before the market does. The reality has a tax attached, and this article is about the tax. Alternative data is almost entirely short-horizon information. It tells you about this quarter's phones, not the firm's ten-year cash flows. And Dessaint, Foucault and Frésard showed that when short-horizon data gets cheap, forecasters shift their effort toward the short term and their long-horizon accuracy actually falls. You buy a short-term edge and pay for it with a long-term blind spot. Worse, the short-term edge is exactly the kind that competition erodes fastest, which is the point of the old article ["Alpha Decay Is Just Competition (and Papers Lie)"](https://aligrithm.com/alpha-decay-is-just-competition-and-papers-lie/): a signal decays because other people find it, and the cheaper and more short-dated the data, the faster the crowd arrives. ## Cross-firm links: the alpha hiding in public data Before the satellite feeds there is a cheaper cousin, what the source calls alternative-like data: information that is public and dull enough that people ignore it. The workhorse example is cross-firm links. Two firms tied by a supply chain, a shared customer, a technology overlap, or the same covering analyst tend to move together, but the lagging firm reacts late. So you build a predictor by taking a link-weighted average of the linked firms' returns. $$ x\_{i,t} = \\frac{\\sum\_{j \\in \\mathcal{S}} \\mathrm{LINK}\_{i,j,t} \\times \\mathrm{RET}\_{j,t}}{\\sum\_{j \\in \\mathcal{S}} \\mathrm{LINK}\_{i,j,t}} $$ Firm i is the focal firm whose next-month return you want to predict, S is its set of linked firms, LINK-i-j is the strength of the tie to firm j, and RET-j is firm j's current return. The numerator is the link-weighted sum of the linked firms' returns and the denominator normalizes by total link strength, so x is a weighted average return of everyone connected to you. Work it. Suppose a chipmaker has three technology-linked peers with link weights 0.5, 0.3 and 0.2, and this month they returned plus 4%, minus 1% and plus 2%. Then x equals 0.5 times 4 plus 0.3 times minus 1 plus 0.2 times 2, all over 1.0, which is 2.0 minus 0.3 plus 0.4, or 2.1%. A high linked return predicts a high focal return next month, because the focal stock has not caught up yet. ![Bar chart of long-short monthly factor-adjusted alphas for shared-analyst, customer, technological, complicated-firm, and geographic links, each significant](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_407-cross_firm_link_alpha.png) The alphas are real and they are not small. Sorting focal firms on this signal and going long the top decile, short the bottom, the four-factor monthly alpha runs 1.81% for shared-analyst links (t of 6.15), 1.38% for customer links (t of 3.13), 1.08% for both technological and complicated-firm links (t of 4.98 and 4.48), and 0.42% for geographic links (t of 3.40). Every one clears the bar. The mechanism is not risk, it is limited attention plus limits to arbitrage: the link is public, but nobody bothers to trade it until the lagging firm's own news forces the issue. ## Why the edge exists, and why it is fragile If the link is public, why does anyone leave money on the table? Because of how the information arrives. Da, Gurun and Warachka called it the frog in the pan: a signal delivered in many tiny continuous increments slips past investors, while the same signal delivered in one discrete jump gets noticed and priced. You can measure the discreteness of a linked firm's recent move directly. $$ \\mathrm{ID}\_{j,t} = \\mathrm{sign}(CR\_{j,t}) \\times (\\%neg\_{j,t} - \\%pos\_{j,t}) $$ CR-j is firm j's cumulative return over the past three months, sign of CR is plus or minus one, and percent-neg and percent-pos are the fractions of days in that window with negative and positive returns. Read what it captures. Take a stock up over three months, so the sign is plus one. If the gain came from many small up days, percent-pos is large, say 0.60, and percent-neg small, say 0.30, so ID equals plus one times 0.30 minus 0.60, which is minus 0.30, a low and continuous reading. If instead the same gain came from a single earnings-day spike with otherwise mixed days, percent-pos and percent-neg are both near 0.30, so ID sits near zero, a discrete reading. Continuous information, the low-ID frog in the pan, produces the strongest lead-lag: in the data the high-linked-return spread earns 0.98% a month (t of 4.65) for the most continuous firms versus a statistically dead 0.21% (t of 1.10) for the most discrete. The edge lives precisely where the information was too quiet to notice, which is also why it is fragile. The moment someone builds the ID screen and trades it, the quiet gets loud. ## Fund-implied alpha: reading the smart money's homework A second alternative-like source is mutual-fund holdings. If a skilled fund holds a stock heavily, that vote carries information. Define a fund's skill as the holdings-weighted average of its stocks' future alphas. $$ S\_{i,t+1} = \\sum\_{n=1}^{N\_t^s} w\_{n,t}^i\\, \\alpha\_{n,t+1} $$ S-i is the skill of fund i, w-n is the weight of stock n in that fund, and alpha-n is stock n's future alpha. Forward, it is trivial: a fund with weights 0.5, 0.3, 0.2 on stocks with alphas of 2%, 1% and minus 1% has skill of 0.5 times 2 plus 0.3 times 1 plus 0.2 times minus 1, which is 1.1%. The clever move is to run it backwards. You observe many funds' realized skills and you want to recover the individual stock alphas that must have produced them, which is stacking the skill equation across all funds and solving for the alpha vector. The catch is that there are far more stocks than funds, so ordinary least squares has no unique solution and you fall back on a generalized inverse. Wermers and coauthors did exactly this: the top-minus-bottom decile sorted on fund-implied alpha earned 1.10% a month with a t of 3.99\. The smart money's portfolio weights are a signal, if you invert them carefully. ## Common flow: when everyone gets redeemed at once Fund holdings also drag stocks around mechanically. When investors pour money into funds or yank it out, managers buy or sell their existing holdings in proportion, and these flows are not independent across funds. One dominant component drives them: the first principal component of fund-flow shocks explains over 60% of the common variation. That means a stock's sensitivity to aggregate flow is a priced risk, which you estimate with a time-series regression. $$ R\_{i,t} = a\_{i,t} + \\beta\_{i,t}^{flow} \\times \\mathrm{common\\\_flow}\_t + e\_{i,t} $$ R-i is the stock's return, common-flow is the aggregate flow factor, and beta-flow is the stock's loading on it, estimated over a trailing 36 months. Here is the twist in the sign. Fund managers prefer to hold low-flow-beta stocks because those insulate them from forced fire-sale trading, so demand bids low-beta stocks up and pushes their future returns down, while high-flow-beta stocks are shunned, sit cheaper, and earn more. Dou, Kogan and Wu found the top-minus-bottom flow-beta quintile earned a CAPM alpha of 7.62% a year (t of 2.42). Common flow carries a positive risk premium, and it is invisible unless you decompose the flows. ## The short-horizon decay tax Now the tax. Comb through 26 academic papers that use alternative data, from satellite imagery to social media to employer reviews, and not one finds predictability beyond a one-year horizon. Alternative data is structurally short-dated. Dessaint, Foucault and Frésard turned that fact into a model: analysts split limited effort between forecasting short-term and long-term earnings, and when cheap short-term data arrives it lowers the cost of the short-term task, so a rational analyst pours more effort there and pulls effort away from the long term. The prediction is not that everything improves. It is that the term structure of forecast accuracy steepens. ![Line chart of forecast informativeness against horizon, with the alt-data-era curve higher at short horizons and lower at long horizons than the pre-alt-data curve, crossing near two years](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_407-horizon_effect.png) They confirmed it two ways. Over the long run, US analyst forecast informativeness rose at short horizons and fell at long horizons after 2000, exactly when alternative data began arriving. Cross-sectionally, analysts more exposed to StockTwits social-media data show the same steepening. The Chinese evidence repeats it: across 4,208,520 analyst forecasts from 2012 to 2023, short-horizon accuracy improved after 2016 while two-year accuracy significantly declined, with no change beyond two years. This is the decay tax stated precisely. The short-horizon accuracy you gain is paid for in long-horizon accuracy you lose, because attention is finite and the data only helps the near term. Stack that on top of the crowding argument from the old article ["Alpha Decay Is Just Competition (and Papers Lie)"](https://aligrithm.com/alpha-decay-is-just-competition-and-papers-lie/) and the picture is bleak for anyone treating alternative data as a permanent edge. A short-horizon signal is the easiest kind for a competitor to replicate, because it needs no long track record to validate, just this quarter's data feed. So the very signals alternative data is good at producing are the ones that decay fastest once the data vendor sells the same feed to your competitors. The lead-lag link alphas above are underreaction stories that survive only while attention stays scarce, and every new subscriber to the same dataset makes attention less scarce. The honest read: alternative data buys you a real but short-dated and rapidly-decaying edge, and it quietly degrades your ability to see far. Treat it as a lease, not a purchase, and never let the short-horizon signal convince you that you understand the long horizon better than you did before. You understand it worse. ## KEY POINTS - Alternative data (satellite, social media, employer reviews, search) is structurally short-horizon: across 26 studies, none finds predictability beyond one year. - Cross-firm links are cheap alternative-like data. A link-weighted average of linked firms' returns predicts a focal firm's next-month return, earning long-short four-factor alphas of 1.81% (shared analyst), 1.38% (customer), 1.08% (technological and complicated firms), and 0.42% (geographic), all significant. - The lead-lag edge is underreaction, not risk. Information discreteness (the frog-in-the-pan measure) shows the effect is strongest for continuous, low-discreteness information (0.98% a month, t of 4.65) and dead for discrete information (0.21%, t of 1.10). - Fund-implied alpha reads the smart money: a fund's skill is its holdings-weighted average stock alpha, and inverting the fund-to-stock map (via generalized inverse, since stocks outnumber funds) yields a decile spread of 1.10% a month (t of 3.99). - Common fund flow is a priced risk. Its first principal component explains over 60% of flow variation, and high-flow-beta stocks earn a CAPM alpha of 7.62% a year (t of 2.42) because managers overpay for low-beta hedges. - The short-horizon decay tax: cheap short-term data induces analysts to shift effort to the short term, raising short-horizon forecast accuracy but lowering long-horizon accuracy. Confirmed in US data after 2000, via StockTwits exposure, and in 4.2 million Chinese forecasts after 2016. - Short-horizon signals decay fastest because they are the easiest to replicate and need no long validation history. Alternative-data alpha is a lease, not a purchase, and it degrades your long-horizon vision while it lasts. ## References - [Does Alternative Data Improve Financial Forecasting? The Horizon Effect (Dessaint, Foucault, and Frésard, 2024)](https://doi.org/10.1111/jofi.13323?ref=aligrithm.com) - [Economic Links and Predictable Returns (Cohen and Frazzini, 2008)](https://doi.org/10.1111/j.1540-6261.2008.01379.x?ref=aligrithm.com) - [Complicated Firms (Cohen and Lou, 2012)](https://doi.org/10.1016/j.jfineco.2011.08.006?ref=aligrithm.com) - [Technological Links and Predictable Returns (Lee, Sun, Wang, and Zhang, 2019)](https://doi.org/10.1016/j.jfineco.2018.11.008?ref=aligrithm.com) - [Shared Analyst Coverage: Unifying Momentum Spillover Effects (Ali and Hirshleifer, 2020)](https://doi.org/10.1016/j.jfineco.2019.10.007?ref=aligrithm.com) - [Frog in the Pan: Continuous Information and Momentum (Da, Gurun, and Warachka, 2014)](https://doi.org/10.1093/rfs/hhu003?ref=aligrithm.com) - [A Flow-Based Explanation for Return Predictability (Lou, 2012)](https://doi.org/10.1093/rfs/hhs103?ref=aligrithm.com) - [Crowdsourced Employer Reviews and Stock Returns (Green, Huang, Wen, and Zhou, 2019)](https://doi.org/10.1016/j.jfineco.2019.03.012?ref=aligrithm.com) - [Alternative Data (Shi, 2024)](https://papers.ssrn.com/sol3/papers.cfm?abstract%5Fid=5326961&ref=aligrithm.com) --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ### 10.12 Factor Timing Mostly Fails URL: https://aligrithm.com/factor-timing-mostly-fails-the-honest-version/ Last updated: 2026-07-10T10:57:28.000Z Factor timing is the seductive idea that you can do better than holding a basket of factors: lean into value when value looks cheap, load momentum when momentum is running, cut a factor when its volatility spikes. Every one of those moves has a tidy formula and a plausible story. And when you run them out of sample on a clean universe, almost all of them lose to the dumbest thing you could have done, which is hold every factor in equal weight and never touch it again. That is the honest version, and it is not a rounding error. Chuan Shi ran the four classic timing signals, valuation spread, factor momentum, factor volatility, and investor sentiment, on the Chinese market from 2000 to 2020, and none of them beat an equal-weighted seven-factor portfolio at any horizon he tried. The equal-weight benchmark itself was real, earning 0.27% a month over the market with a Newey-West t-statistic of 3.98\. The timing overlays added nothing on top. Meanwhile there is a genuine theoretical argument, from Haddad, Kozak and Santosh, that timing should matter enormously. Reconciling those two facts is the whole article, and the resolution is the same lesson the old article ["Collinearity in Parameter Sweeps: Plateaus, Not Peaks"](https://aligrithm.com/collinearity-in-parameter-sweeps-plateaus-not-peaks/) taught for backtests: robust edges live on wide plateaus, and factor timing is mostly an attempt to stand on a sharp peak that estimation error knocks you off of. ## The four timing recipes, and their formulas Start with valuation timing, the most respectable of the four. You measure each factor's value spread, the valuation of its long leg minus its short leg, and you overweight a factor when its own spread is wide relative to its history. The key discipline is that you compare a factor to itself, never across factors, because every factor lives in its own valuation range. $$ w\_{kt}^{\\text{raw}} = 1 + \\frac{VS\_{kt} - VS\_{kt}^{\\text{mean}}}{VS\_{kt}^{\\text{max}} - VS\_{kt}^{\\text{min}}}, \\qquad w\_{kt} = \\frac{w\_{kt}^{\\text{raw}}}{\\sum\_{j=1}^{K} w\_{jt}^{\\text{raw}}} $$ VS-kt is factor k's value spread now, and the mean, max and min are taken over its own trailing window of T months. The raw weight is a number near 1 that rises above 1 when the factor is cheap relative to its own past and falls below 1 when it is expensive, and then you normalize the raw weights across the K factors so they sum to one. Work a number. Suppose value's spread today sits at 0.8, its trailing mean is 0.5, its max is 1.0 and its min is 0.2\. The raw weight is 1 plus (0.8 minus 0.5) over (1.0 minus 0.2), which is 1 plus 0.3 over 0.8, or 1.375\. A factor trading at its historical median gets exactly 1.0, and one at its cheapest gets close to 2.0\. Tidy. The problem is not the recipe, it is that when Shi actually traded it in China the excess return over equal weight was negative at every lookback from 24 to 60 months, the least bad being minus 0.03% a month with a t-statistic of minus 0.53. Factor momentum is the second recipe and needs no calculus. Rank the K factors by their trailing twelve-month return, hand the best factor a score of K, the next K minus 1, on down to 1 for the worst, and normalize those scores into weights. It is a winner-loser bet on factors themselves. Sentiment timing is the fourth: build an index from five gauges, the ratio of upside to downside volatility, turnover, the futures basis, new-account growth, and margin-balance growth, each turned into a rolling 252-day percentile, then average them and split into high and low sentiment states. In the Chinese data only momentum and growth factors even showed a statistically detectable sentiment dependence, and timing on it still did not beat equal weight. ## Volatility timing, and the correlation that guts it The third recipe is inverse-volatility weighting. Give each factor a weight proportional to the reciprocal of its own volatility, so the calm factors get more capital and the wild ones get less, which lowers total portfolio volatility. $$ w\_k = \\frac{1/\\sigma\_k}{\\sum\_{j=1}^{K} 1/\\sigma\_j} $$ Sigma-k is factor k's return volatility. If three factors have volatilities of 2%, 3% and 6% a month, their reciprocals are 0.50, 0.33 and 0.167, summing to 1.0, so the weights are 0.50, 0.33 and 0.167\. The low-vol factor gets triple the weight of the high-vol one. Sensible in isolation, and it is genuinely a risk-reduction tool. But as a timing signal it is largely dead on arrival, because factor volatilities are not independent. Kapadia, Linn and Paye showed factor vols move together, and Shi confirms it: the four style factors in the Chinese five-factor model have an average pairwise volatility correlation of 0.65, and across 92 factors the average is still 0.55\. When every factor gets more volatile at the same time, tilting toward whichever is momentarily calmer is noise, not information. Traded in China, volatility timing produced negative excess returns over equal weight at every horizon, none significant. ![Bar chart of monthly excess return over equal weight for valuation, momentum, volatility, and sentiment timing in China, all with t-statistics below 2, against a note that equal weight itself earned 0.27 percent per month at t equals 3.98](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_406-china_timing_fails.png) Line the four up and the verdict is blunt. Even choosing the single most favorable lookback for each signal, the steelman case, not one clears a t-statistic of 2\. The honest read is not that these signals contain zero information ever, it is that the information is too weak and too correlated to survive real trading once you already hold the factors equally. ## The theory says timing should be a big deal Here is the twist that keeps factor timing alive as a research program. Asset pricing says a stochastic discount factor prices everything, and you can write it as one minus a loadings vector times the surprise in factor returns. $$ m\_{t+1} = 1 - \\delta\_t^\\top \\big(f\_{t+1} - \\mathbb{E}\_t\[f\_{t+1}\]\\big), \\qquad \\delta\_t = \\Sigma\_{f,t}^{-1}\\, \\mathbb{E}\_t\[f\_{t+1}\] $$ Here f-t-plus-1 is the vector of factor returns, E-t of f is what you expected given today's information, delta-t is the SDF's loadings, and Sigma-f-t is the conditional covariance of the factors. The optimal loadings are the inverse conditional covariance times the conditional expected returns, which is just the mean-variance efficient weights computed with time-varying inputs. The instant you let E-t of f move over time, you are doing factor timing, so timing is not some optional overlay, it sits inside the definition of the efficient portfolio. And Hansen and Jagannathan proved the maximum attainable Sharpe ratio is bounded by the variance of the SDF, which means anything that raises the SDF's variance raises the ceiling on how good a portfolio can be. Haddad and coauthors measured exactly that: moving from static factor investing to full factor timing lifted the average conditional SDF variance from 1.67 to 2.96, nearly doubling the theoretical Sharpe ceiling. On paper, timing is worth a fortune. ## Why the fortune does not show up in your account The reconciliation is that the predictable part of factor returns is concentrated in a few big dimensions, and everything else is noise you cannot trade. Haddad, Kozak and Santosh ran principal components on fifty anomaly portfolios. The first component alone explains 25.8% of the variance and the first five together explain 60%. Then they asked which components are actually forecastable using their own book-to-market spreads, and the answer is the dominant ones, not the obscure ones. ![Bar chart of in-sample and out-of-sample R-squared for predicting each of fifty principal components with its own book-to-market ratio, with the first few components boxed as the only ones showing meaningful out-of-sample predictability](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_406-pc_predictability.png) The big components, PC1 and PC4, show out-of-sample monthly R-squared of 4.82% and 3.52%, real and tradeable. The long tail of small components has R-squared indistinguishable from zero and often negative out of sample. This is the surprise the original authors emphasized, and it inverts the folk wisdom. The common intuition is that you time individual boutique factors on their own momentum or sentiment. The data says the only reliably timeable object is the handful of dominant common components, timed by valuation. That is why Shi's individual-factor timing in China fails while Haddad's "anomaly timing" of the big US components earns a positive out-of-sample information ratio of 0.60, against minus 0.64 for naive market timing. Timing works, narrowly, exactly where the naive practitioner is not looking. ## The plateau beats the peak Now connect it to allocation, because the failure of timing is really the failure of precision. The mean-variance efficient portfolio is weights proportional to inverse covariance times expected returns, and it delivers the maximum Sharpe ratio if your inputs are right. They never are. Shi says it plainly: the method is highly sensitive to its inputs and estimating expected returns and covariances ex ante is the hard part. So look at what happens when you actually trade the fancy optimizer against the dumb ones. ![Cumulative returns of equal weight, mean-variance efficient, maximum diversification, and equal risk contribution allocations against the market, with equal weight and equal risk contribution on top and the mean-variance optimizer lagging](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_406-allocation_methods.png) Equal weight and equal risk contribution finish at the top. The mean-variance efficient portfolio, the one that is theoretically optimal, actually lags them, because the estimation error in its inputs costs more than its cleverness earns. In Shi's stylized comparison the methods are identical when factors share Sharpe ratios and correlations, and the optimizer only pulls ahead in the fully general case, on paper, at Sharpe 0.670 versus equal weight's 0.504, which is precisely the case where the input sensitivity bites hardest in live trading. This is the old article ["Collinearity in Parameter Sweeps: Plateaus, Not Peaks"](https://aligrithm.com/collinearity-in-parameter-sweeps-plateaus-not-peaks/) applied to portfolios instead of parameter grids. There, the lesson was to trust a wide plateau of stable performance over a razor-thin peak that only exists because two parameters move together. Here, equal weight is the plateau: no inputs to estimate, nothing to overfit, robust by construction. Factor timing and mean-variance optimization are the peak: gorgeous at the estimated optimum, and standing on air the moment your estimates are off. None of this says timing is impossible. It says the honest baseline is brutal. Before you deploy any timing signal, you owe it a comparison against equal weight with a real out-of-sample window and Newey-West standard errors, and most signals will not clear the bar. The edge that survives is narrow, it lives in the big common components rather than individual factors, it is timed by valuation rather than momentum or sentiment, and it is small enough that transaction costs can erase it. Sell the plateau, not the peak. ## KEY POINTS - In Chuan Shi's Chinese study (2000 to 2020, top-1000 universe, seven long-leg factors), valuation, momentum, volatility, and sentiment timing all failed to beat an equal-weighted portfolio at every horizon, none reaching a t-statistic of 2. - The equal-weight benchmark itself was real, earning 0.27% a month over the market with a Newey-West t-statistic of 3.98\. The timing overlays added nothing to it. - Valuation timing weights a factor by where its own value spread sits in its historical range, comparing each factor only to itself, never across factors. - Inverse-volatility weighting is undercut by common volatility: factor vols carry an average pairwise correlation of 0.65 across four style factors and 0.55 across 92 factors, so tilting toward the calmer factor is mostly noise. - Theory says timing should be huge: the mean-variance-efficient SDF loadings are inverse conditional covariance times conditional expected returns, and moving from static investing to full timing lifted the average SDF variance from 1.67 to 2.96, nearly doubling the Hansen-Jagannathan Sharpe ceiling. - The predictability is concentrated: the first principal component of fifty anomalies explains 25.8% of variance, and only the dominant components (PC1, PC4) show tradeable out-of-sample R-squared (4.82% and 3.52%). The obscure components are noise. - Timing the big common components by valuation earns a positive out-of-sample information ratio (0.60) while naive market timing earns minus 0.64\. Timing works where the naive practitioner is not looking. - Equal weight and equal risk contribution beat the mean-variance optimizer in live trading because the optimizer's input-estimation error costs more than its cleverness earns. Prefer the robust plateau over the fragile peak. ## References - [Factor Timing (Haddad, Kozak, and Santosh, 2020)](https://doi.org/10.1093/rfs/hhaa017?ref=aligrithm.com) - [Implications of Security Market Data for Models of Dynamic Economies (Hansen and Jagannathan, 1991)](https://doi.org/10.1086/261749?ref=aligrithm.com) - [Macroeconomic Perceptions, Financial Constraints, and Anomalies (He, Su, and Yu, 2024)](https://doi.org/10.1016/j.jfineco.2024.103952?ref=aligrithm.com) - [One Vol to Rule Them All: Common Volatility Dynamics in Factor Returns (Kapadia, Linn, and Paye, 2024)](https://doi.org/10.1017/S0022109023000443?ref=aligrithm.com) - [Factor Timing and Factor Allocation (Shi, 2024)](https://papers.ssrn.com/sol3/papers.cfm?abstract%5Fid=5264439&ref=aligrithm.com) --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ### 10.11 Is Value Dead? The Revaluation Decomposition URL: https://aligrithm.com/is-value-dead-the-revaluation-decomposition/ Last updated: 2026-07-10T10:57:29.000Z By the middle of 2020 the obituaries were written. The classic value factor, long cheap stocks and short expensive ones, had lost about 55% from its December 2006 peak, a drawdown that ran more than thirteen years. Every quant who had loaded up on book-to-market was underwater for a decade, and the natural conclusion was that the world had changed, the trade was arbitraged away, and value was structurally dead. That conclusion is the expensive one, and it is usually wrong, because it confuses a factor getting cheaper with a factor stopping working. Arnott, Harvey, Kalesnik and Linnainmaa took the drawdown apart into its moving pieces and found something the death narrative cannot survive. The engine that generates the value premium, the part that comes from cheap stocks staying cheap and paying you income while they do, never stopped running. It contributed a positive 1.1% a year even through the worst stretch. What sank the factor was a separate term: the price gap between value and growth stocks blew out to the widest it had ever been, a 100th-percentile spread. That is a valuation move, and valuation moves mean-revert. The factor was not dying, it was going on sale. This is the empirical version of two arguments made before. The old article ["Why Systems Work Until They Don't"](https://aligrithm.com/why-systems-work-until-they-dont/) laid out how every strategy decays and how to tell decay from noise. The old article ["When to Switch Off a Trading System"](https://aligrithm.com/when-to-switch-off-a-trading-system/) built the kill rule: hold inside the envelope, cut on a high-percentile breach, and separate a statistical breach from a structural one. Value in 2020 is the textbook case where getting that distinction wrong would have cost you the recovery. ## Split the return into what lasts and what mean-reverts Start with the accounting identity for a portfolio's log return over one period. Whatever you earn comes from the price moving plus the dividend you collected. $$ \\log(1 + r\_t) = \\log\\!\\left(\\frac{P\_{t^-}}{P\_{t-1}}\\right) + \\log\\!\\left(1 + \\frac{D\_{t^-}}{P\_{t^-}}\\right) $$ P is the portfolio's market value and D the dividend, with t-minus meaning the old portfolio carried to time t before rebalancing. The first term is the capital gain, the second the dividend yield. True so far but useless, because the price move mixes two completely different things: the company genuinely getting more valuable, and the market simply paying a higher multiple for the same book value. You need to separate those, so Arnott and coauthors add and subtract price-to-book terms to break the return into three named pieces. $$ \\log(1 + r\_t) = \\underbrace{\\left\[\\log\\tfrac{P\_t^+}{B\_t^+} - \\log\\tfrac{P\_{t-1}}{B\_{t-1}}\\right\]}\_{\\text{revaluation}} + \\underbrace{\\left\[\\log\\tfrac{P\_{t^-}}{B\_{t^-}} - \\log\\tfrac{P\_t^+}{B\_t^+}\\right\]}\_{\\text{migration}} + \\underbrace{\\left\[\\log\\tfrac{B\_{t^-}}{B\_{t-1}} + \\log\\!\\left(1 + \\tfrac{D\_{t^-}}{P\_{t^-}}\\right)\\right\]}\_{\\text{income yield}} $$ Read the three pieces. B is book value, and a superscript plus marks the freshly rebalanced portfolio formed at time t, while t-minus marks the old portfolio drifting into time t. Revaluation is the change in the portfolio's price-to-book multiple across the period, pure repricing. Migration is the jump when you rebalance, capturing the value premium harvested as cheap stocks get repriced upward and graduate out of the portfolio, which Fama and French named the migration effect. Income yield is book-value growth plus dividends, the intrinsic accumulation. Group them and the split that matters falls out: revaluation is the mean-reverting part that should wobble around zero over the long run, while migration plus income yield is the structural premium, the durable payment for holding cheap stocks. ## The decomposition verdict: a repricing, not a death Run the HML factor through this and the death narrative collapses. ![Bar chart of HML return decomposition across 1963-2007, 2007-2020, and the full sample, showing structural premium positive throughout while revaluation drives the post-2007 loss](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_405-value_decomposition.png) From 1963 to 2007 the factor earned 6.1% a year, and it was almost all structural, 5.9% of it, with revaluation contributing a trivial 0.2%. That is what a working factor looks like: the return comes from the durable engine, not from multiples drifting. Now the crisis window. From July 2007 to June 2020 the factor lost 6.1% a year, and the revaluation term was minus 7.2%. The structural premium over the same stretch stayed positive at plus 1.1%. Read that again. Cheap stocks kept paying the premium while their price relative to growth stocks cratered. The entire loss, and then some, was the value-growth spread widening. Over the full 1963 to 2020 span the structural premium is a healthy 4.8% a year and revaluation is a small negative 1.5%, exactly the near-zero long-run behavior the theory predicts. The valuation evidence backs it up hard. Asness measured the price-to-book spread between cheap and expensive stocks and found it at the 100th percentile of its entire history in early 2020, more than four standard deviations above the median. Strip out technology, media and telecom, drop the megacaps, drop the 10% most expensive stocks, hold industries neutral, and it stays pinned near the 100th percentile. This was not one bubble sector dragging the number. Value stocks were categorically, broadly, historically cheap relative to growth. A spread that wide is a coiled spring, not a tombstone. ## Book value forgot about intangibles There is a real problem underneath the repricing, and it is not that value stopped working. It is that the ruler got worse. Book value was designed for a world of factories and inventory, and it quietly expenses the things that now build the most corporate value: research and development, brand, software, patents. A firm that pours money into R&D looks like it has thin book value and gets misclassified as expensive, when economically it may be the cheap one. Arnott and coauthors rebuild the value signal by capitalizing intangibles into book value, producing an intangibles-adjusted factor, iHML. ![Cumulative performance of traditional HML versus intangibles-adjusted iHML from 1963 to 2020, with iHML consistently ahead](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_405-hml_vs_ihml.png) The intangibles version wins across every window. Full sample it earns 4.3% a year against traditional HML's 3.0%. Through the post-2007 drawdown it still lost, but only minus 3.2% a year versus minus 5.4%, and its cumulative drawdown was minus 43% against minus 55%. Better, not saved. iHML peaked in December 2016 and then fell too, because the same valuation-spread blowout hit it. So the two findings stack: the book-value ruler is genuinely outdated and fixing it helps, and on top of that the whole value complex got repriced to an extreme. One is a measurement fix, the other is a mean-reverting cheapness. Neither is death. ## Was a 55% drawdown even surprising Before killing a strategy you owe it one question: is this loss inside the range the strategy could always have produced. The way to answer is a block bootstrap, resampling historical returns in six-month blocks to preserve their autocorrelation, using only pre-2007 data so the recent drawdown does not contaminate the distribution, then reading where the realized loss falls. ![Bootstrap distribution of the largest HML drawdown, with the realized drawdown sitting at the 2.3rd percentile of the pre-2007-calibrated distribution](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_405-bootstrap_drawdown.png) The median simulated worst drawdown is around 32%. The realized drawdown lands at roughly the 2.3rd percentile, deep in the left tail but not off the chart. This is the operational lesson from the old article ["When to Switch Off a Trading System"](https://aligrithm.com/when-to-switch-off-a-trading-system/) made concrete. A 2.3rd-percentile event is rare, so a disciplined trader cuts risk on a breach that extreme, but it is not proof the edge is gone, because a healthy strategy visits its 2nd percentile roughly one time in forty-something. The mistake is treating a tail draw as a structural break when your own pre-crisis data said such draws were possible. And because the block bootstrap used data ending in 2006, it if anything understated the true odds. The drawdown was ugly and legal at the same time. ## The honest kill test: update, do not react None of this means you hold a dying factor forever out of stubbornness. It means you decide with a rule, not a mood, and the right rule is Bayesian. You start with a prior belief about the factor's premium, you observe a run of recent returns, and you blend the two in proportion to how precise each one is. $$ \\mu\_p = \\frac{\\text{se}^2}{\\sigma^2 + \\text{se}^2}\\,\\mu + \\frac{\\sigma^2}{\\sigma^2 + \\text{se}^2}\\,\\bar{r}, \\qquad \\sigma\_p^2 = \\left(\\frac{1}{\\sigma^2} + \\frac{1}{\\text{se}^2}\\right)^{-1} $$ Here mu and sigma are the prior mean and standard deviation of the factor's expected return, r-bar and se are the sample mean and its standard error from recent data, and mu-p and sigma-p are the posterior. The posterior mean is a precision-weighted average: the noisier the recent sample, the less it moves your belief. Then you judge failure by the posterior t-statistic, mu-p divided by sigma-p, not by the raw bad run. Work it. Say your prior on a monthly value premium is a mean of 0.40% with a standard deviation of 0.20%, so the prior variance is 0.04\. A grim recent stretch prints a sample mean of minus 0.50% with a standard error of 0.30%, so its variance is 0.09\. The weight on the prior is 0.09 divided by 0.13, about 0.69, and the weight on the data is 0.04 divided by 0.13, about 0.31\. The posterior mean is 0.69 times 0.40 plus 0.31 times minus 0.50, which is 0.12% a month, still positive. The posterior variance is one over the sum of 25 and 11.1, about 0.028, so the posterior standard deviation is 0.17% and the t-statistic is roughly 0.7\. The verdict is not "dead," it is "shrunk and no longer significant, keep it small and keep watching." That is a decision, and it is the opposite of panic-selling the bottom. ## Where this connects The value debate is the cleanest live example of the decay-versus-noise problem from the old article ["Why Systems Work Until They Don't."](https://aligrithm.com/why-systems-work-until-they-dont/) Four decay mechanisms exist, crowding, regime drift, microstructure change, and capacity saturation, and each has a fingerprint. The 2007 to 2020 value drawdown matched none of them. Its fingerprint was a widening valuation spread with an intact structural premium, which is the signature of a repricing, and repricings are the one kind of drawdown that predicts higher future returns rather than lower. Confuse it with genuine decay and you commit the exact error the old article ["When to Switch Off a Trading System"](https://aligrithm.com/when-to-switch-off-a-trading-system/) warns is most common and most expensive: quitting inside the envelope and missing the recovery. Two cautions keep this from becoming a blank check for stubbornness. The decomposition is an attribution done with hindsight, and knowing the loss was revaluation does not tell you when the spread reverts, only that its expected long-run contribution is near zero. And the same accounting that exonerates traditional book value also indicts it, so if your value signal ignores intangibles you are carrying a measurement bug regardless of the spread. The lesson is not "value can never die." It is that death and cheapness leave different fingerprints, and you owe every strategy a decomposition and a bootstrap before you write its obituary. ## KEY POINTS - The classic value factor lost about 55% from December 2006 to June 2020, prompting widespread claims that value was structurally dead. - Decomposing HML into revaluation, migration, and income yield shows the structural premium (migration plus income yield) stayed positive even in the crisis, at plus 1.1% a year, while revaluation of minus 7.2% a year drove the entire loss. - Revaluation is the mean-reverting price-multiple term that should hover near zero long-run, and over 1963 to 2020 it was just minus 1.5% while the structural premium was a healthy 4.8%. The engine never broke. - The value-growth valuation spread hit the 100th percentile of its history, over four standard deviations above median, and stayed extreme after removing tech, megacaps, the priciest names, and industry effects. That is a coiled spring, not a tombstone. - Book value ignores intangibles like R&D and brand, misclassifying research-heavy firms. Capitalizing intangibles into iHML beats traditional HML in every window, 4.3% versus 3.0% full sample, though it too fell in the spread blowout. - A block bootstrap on pre-2007 data puts the realized drawdown at roughly the 2.3rd percentile: a deep tail event, but one the strategy's own history said was possible, not proof of a broken edge. - Judge factor death with a Bayesian update, blending prior and recent sample by precision and testing the posterior t-statistic, rather than reacting to a raw bad run. A grim stretch can leave the posterior positive but shrunk, which argues for smaller size, not abandonment. ## References - [Reports of Value's Death May Be Greatly Exaggerated (Arnott, Harvey, Kalesnik, and Linnainmaa, 2021)](https://doi.org/10.1080/0015198X.2020.1842704?ref=aligrithm.com) - [Is (Systematic) Value Investing Dead? (Asness, 2020)](https://www.aqr.com/Insights/Perspectives/Is-Systematic-Value-Investing-Dead?ref=aligrithm.com) - [The Quant Crisis of 2018-2020: Cornered by Big Growth (Blitz, 2021)](https://doi.org/10.3905/jpm.2021.1.232?ref=aligrithm.com) - [A Five-Factor Asset Pricing Model (Fama and French, 2015)](https://www.sciencedirect.com/science/article/abs/pii/S0304405X14002323?ref=aligrithm.com) - [The History of the Cross-Section of Stock Returns (Linnainmaa and Roberts, 2018)](https://doi.org/10.1093/rfs/hhy030?ref=aligrithm.com) - [Market Efficiency in the Age of Big Data (Martin and Nagel, 2022)](https://doi.org/10.1016/j.jfineco.2021.10.006?ref=aligrithm.com) - [Factor Failure (Shi, 2024)](https://papers.ssrn.com/sol3/papers.cfm?abstract%5Fid=5258481&ref=aligrithm.com) --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ### 10.9 The SDF View: One Equation Behind Every Factor Model URL: https://aligrithm.com/the-sdf-view-one-equation-behind-every-factor-model/ Last updated: 2026-07-10T10:57:29.000Z Ask ten quants what a factor model is and you get ten answers: Fama-French three-factor, the q-factor model, a random forest, a neural net, principal components on a thousand characteristics. They sound like rival religions. They are not. Every one of them is the same single equation with different furniture bolted on. That equation is the stochastic discount factor, and once you see it, the factor zoo stops looking like a menagerie and starts looking like one animal wearing costumes. Chuan Shi's lecture notes lay the frame out cleanly. Write down one pricing object, the SDF, let its portfolio weights be a function of firm characteristics, and every model from a 1990s double sort to a 2024 adversarial network becomes a choice of two things: which characteristics to feed in, and what function turns them into weights. The payoff is not philosophical. On the same test, machine-learning SDF estimators post out-of-sample Sharpe ratios of 3.2 to 3.4 while the Fama-French three-factor model manages 0.45\. Seven times the risk-adjusted return from the same equation, just better choices inside it. The old article ["What Is a Factor, Really"](https://aligrithm.com/what-is-a-factor-really/) defined a factor operationally: an alpha so pervasive you keep re-finding it, so you regress it out before hunting anything new. The old article "What a Factor Actually Is: alpha plus beta-transpose lambda" gave the cross-sectional line every model argues over. This article goes one level deeper, to the object those factor premiums are shadows of. Everything in this pillar is a special case of what follows. ## One law: the SDF prices everything Start with the only assumption that matters, no near-arbitrage. If there is no portfolio out there earning an absurd Sharpe ratio for free, then there exists a single random variable, call it m, that prices every asset at once. The pricing law is that the expected product of m and any asset's excess return is zero. For every asset i, at every date t, the market sets the price so that m and the payoff cancel in expectation. One equation, the whole cross-section. The trick the modern literature uses is to write m itself as one minus a portfolio of the very returns it prices. $$ m\_{t+1} = 1 - \\mathbf{b}\_t^\\top \\mathbf{R}\_{t+1}^e, \\qquad b\_{i,t} = g(\\mathbf{c}\_{i,t}) $$ Read it slowly. m-sub-t-plus-1 is next period's discount factor. R-e is the vector of excess returns on all N stocks. b-sub-t is a vector of weights, one per stock, and the dot product b-transpose R-e is just the return on a portfolio built with those weights. The second piece is the whole idea: the weight on stock i is not a free parameter, it is a function g of that stock's characteristics c-sub-i-t, its size, its book-to-market, its momentum, its 300 other features. The SDF is a characteristic-weighted portfolio, and g is the recipe that turns a company's traits into how much it belongs in that portfolio. Work a toy version. Two stocks, weights driven only by book-to-market, and suppose g just copies the value score, so a stock with value score 0.8 gets weight 0.8\. If the two stocks return 3% and minus 1% next month, then b-transpose R-e is 0.8 times 3 plus 0.2 times minus 1, which is 2.2%, and m is 1 minus 0.022, or 0.978\. The SDF is low in states where the value portfolio pays off. That is the entire content of "value is a factor": the discount factor moves with it. ## Collapse the weights into managed portfolios The function g can be anything, and anything is hard to estimate. So you approximate it as a weighted sum of simpler building blocks, each one a portfolio you can actually form. $$ m\_{t+1} = 1 - \\sum\_{k=1}^{K} \\theta\_k \\, \\tilde{R}\_{k,t+1}^e, \\qquad \\tilde{R}\_{k,t+1}^e = \\sum\_{i=1}^{N} g\_k(\\mathbf{c}\_{i,t})\\, R\_{i,t+1}^e $$ The inner sum builds factor k: take every stock, weight it by g-sub-k of its characteristics, add up the returns, and you have R-tilde-k, a managed portfolio. The outer sum mixes those K factors with coefficients theta. Those thetas are the SDF loadings, the thing you estimate. This is the unconditional form, and it is where the costumes go on. A sign convention floats around here, some write it with a plus, because the sign gets absorbed into theta, so do not read anything into it. Now watch every model fall out of this one line. Fama-French three-factor: K is three, the characteristics are market cap and book-to-market, and each g-sub-k is a double portfolio sort into big-small and value-growth buckets. The q-factor model swaps in investment and profitability sorts. Principal-components approaches let the data pick the g-sub-k as eigenvectors of the return covariance. A neural net lets g-sub-k be an arbitrary nonlinear function learned by gradient descent. Same skeleton, different g. The research problem splits exactly two ways: find C, the characteristics worth including, and find g, the function mapping them to weights. The rest is estimation. The goal also splits, and the split is the whole fight in the field. The traditional academic goal is a parsimonious model that minimizes in-sample pricing errors on a handful of published anomalies. The industry goal is to maximize the out-of-sample Sharpe ratio of the mean-variance-efficient portfolio the SDF implies. Those are not the same target, and optimizing the first has a long history of failing the second. ## The scoreboard: same equation, seven times the Sharpe Here is why the framing earns its keep. Put several choices of C and g on the same out-of-sample test and the spread is enormous. ![Bar chart of out-of-sample Sharpe ratios, machine-learning SDF estimators near 3.2 to 3.4 versus ad-hoc factor models near 0.45 to 1.1](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_403-oos_sharpe_ml_vs_adhoc.png) The textbook models cluster low. Fama-French three-factor posts an out-of-sample Sharpe of 0.45, the six-factor version 1.03, the q-factor model 1.10, the Stambaugh-Yuan mispricing model 0.61\. The machine-learning SDF estimators sit in another league: a random forest at 2.19, a neural net at 3.21, a PCA-based model at 3.21, and instrumented PCA (IPCA) at 3.39\. IPCA, note, is barely nonlinear, it is essentially a smarter linear factor model that lets loadings depend on characteristics, and it wins the whole table. That detail matters: you do not need a monstrous black box, you need the right characteristics feeding a disciplined g. None of this is a different equation. It is the same SDF. The three-factor model chose three characteristics and a crude g and left most of the risk-return relationship on the table. The winners chose many characteristics and a g flexible enough to capture nonlinearities and interactions. The equation was never the constraint. The choices were. ## Sparsity is the illusion, and priors are the fix The obvious objection: if more characteristics win, why not dump in everything and let the machine sort it out? Because asset pricing is the worst possible environment for naive machine learning. The signal-to-noise ratio is very low, the number of predictors rivals the number of observations, prediction error feeds straight into portfolio risk rather than staying an abstract statistic, and the data-generating process does not even hold still. Turn a flexible model loose on that and it fits noise with confidence. Two disciplines rescue it, and they pull in opposite directions from the old "keep it simple" instinct. First, sparsity is a myth here. The illusion of sparsity result shows that cross-sectional return prediction does not collapse to three or five true factors. Many characteristics carry information, they are correlated, and model uncertainty is large, so throwing away all but a few costs you out-of-sample. Second, you cannot just estimate freely, you need an economic prior. The figure below, from the shrinking-the-cross-section work, plots the sweet spot: out-of-sample performance is highest when you use a large number of nonzero coefficients and pair it with a prior of the right strength. ![Contour plot of out-of-sample performance against prior strength and number of nonzero coefficients, peaking with many coefficients and a moderate prior](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_403-sparsity_needs_priors.png) The bright region sits high on the vertical axis, many characteristics, not down at the bottom where sparse models live. But it also needs the prior dialed in along the horizontal axis. Too weak a prior and you overfit, too strong and you shrink real signal to zero. This is where the SDF view repays you again: the natural prior comes from theory. Under no near-arbitrage, the factors that carry high risk premiums must also be the ones that drive how stocks move together. That is not a soft suggestion, it is a testable restriction that tells the estimator where to look, and it is exactly the economic prior the plan for this pillar keeps insisting on. ## The prior, made concrete: ridge is Bayesian shrinkage "Impose an economic prior" sounds vague until you write the estimator. The cleanest case is ridge regression, and its whole appeal is that it is Bayesian shrinkage wearing a linear-algebra disguise. $$ \\hat{\\boldsymbol{\\theta}} = \\left( \\mathbf{X}^\\top \\mathbf{X} + \\frac{\\sigma^2}{\\sigma\_g^2}\\,\\mathbf{I} \\right)^{-1} \\mathbf{X}^\\top \\mathbf{y} $$ X is the matrix of factor returns, y the thing you are pricing, and the term added inside the inverse is the whole story. Ordinary least squares is what you get when that term is zero. The ratio sigma-squared over sigma-g-squared is the noise variance divided by your prior variance on the coefficients. When you believe the true loadings are small (small sigma-g), that ratio is large, and it pulls the estimates hard toward zero. When you are confident the data is clean (small sigma-squared) it barely nudges them. A tight prior is literally a big number added to the diagonal. Put numbers on it. Say the noise variance is 4 and your prior variance on the coefficients is 1, so the penalty is 4 over 1, which is 4\. A raw signal that OLS would estimate near, say, 1.0 gets shrunk down toward roughly 1 over 1 plus 4, about 0.2, in the simplest one-regressor case. You gave up unbiasedness on purpose and bought a large cut in variance, and out-of-sample that trade wins. The deck's own experiments show ridge beating OLS on both out-of-sample R-squared and Sharpe ratio for exactly this reason. Overfitting, in one line, is placing too much weight on this dataset and too little on prior experience, and in investing, ignoring prior experience is how you blow up. ## The same skeleton, from double sorts to trees to GANs Because everything is a choice of g, the frontier is just more expressive g functions. One line of work builds g as a decision tree: split the universe by size, then by value, then by size again, and each leaf becomes a portfolio. The picture is a literal tree. ![Decision tree splitting stocks first by size, then value, then size again, with each leaf node forming a portfolio](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_403-gk_decision_tree.png) A Fama-French double sort is just this tree pruned to two levels. Grow it deeper and let the estimator shrink both the expected returns and the covariance of the resulting factors, and you get a far richer SDF from the same equation. Push further and g becomes an adversarial network: one network builds the SDF to minimize pricing errors while a second network hunts for the portfolios where the SDF fails worst, and they fight until the SDF prices even the hardest test assets. The no-near-arbitrage condition itself becomes the loss function. Different g, same m. The genealogy runs straight from a hand-drawn double sort to a neural net, and nothing in between changed the underlying law. ## Where this connects The SDF is the equation the rest of this pillar keeps circling. The cross-sectional line from the old article "What a Factor Actually Is: alpha plus beta-transpose lambda" is what you get when you take the SDF's implications for expected returns, so alpha and the premium lambda are downstream of m, not separate objects. The factor-premium fights, the tests of whether a set of alphas is jointly zero, the second-pass premium estimates, all of them are questions about the theta coefficients and the choice of g in this one framework. And the t-cutoff debate from the reckoning articles is the "find C" half of the problem: which characteristics survive honest testing, given that true factors attenuate roughly 36% out of sample. There is also a geometric reading worth flagging for the information-geometry thread elsewhere in the project. The set of SDFs consistent with no-arbitrage is a convex region, choosing a model is picking a point in it, and the mean-variance-efficient portfolio the SDF implies is the projection that maximizes squared Sharpe. Priors are curvature imposed on that space, and shrinkage is a walk back toward the prior's center. That is speculative framing on my part, not from the source, but the SDF is the natural object to hang it on. Name the one equation, and every model in the zoo becomes a coordinate rather than a creature. ## KEY POINTS - Under no near-arbitrage there is a single stochastic discount factor m that prices every asset at once, and its defining property is that the expected product of m and any excess return is zero. Every factor model is a way of writing m. - Write m as one minus a characteristic-weighted portfolio, m equals 1 minus b-transpose R-e with the weight b on each stock a function g of its characteristics. The SDF is just a portfolio whose weights are learned from firm traits. - Approximating g by a sum of managed portfolios turns the model into m equals 1 minus a sum of theta-k times factor-k. Fama-French, q-factor, PCA, and neural-net models are all this one equation with different characteristics C and different functions g. - On the same out-of-sample test, machine-learning SDF estimators hit Sharpe ratios of 3.2 to 3.4 (IPCA 3.39, neural net 3.21) versus 0.45 for Fama-French three-factor. The equation is identical, the choices of C and g are not. - Sparsity is an illusion in the cross-section: many characteristics carry information, so sparse three-to-five-factor models leave performance on the table. Best out-of-sample results use many coefficients plus a well-calibrated prior. - Economic priors are not optional. Ridge regression is Bayesian shrinkage, adding noise-over-prior-variance to the diagonal, and no-near-arbitrage supplies the key restriction: high-premium factors must also drive comovement. - The frontier is just more expressive g functions, from double sorts to decision trees to adversarial networks that turn the no-arbitrage condition into a loss. Same skeleton throughout. ## References - [Common Risk Factors in the Returns on Stocks and Bonds (Fama and French, 1993)](https://www.sciencedirect.com/science/article/abs/pii/0304405X93900235?ref=aligrithm.com) - [Interpreting Factor Models (Kozak, Nagel, and Santosh, 2018)](https://onlinelibrary.wiley.com/doi/10.1111/jofi.12612?ref=aligrithm.com) - [Shrinking the Cross-Section (Kozak, Nagel, and Santosh, 2020)](https://www.sciencedirect.com/science/article/abs/pii/S0304405X19301655?ref=aligrithm.com) - [Empirical Asset Pricing via Machine Learning (Gu, Kelly, and Xiu, 2020)](https://academic.oup.com/rfs/article/33/5/2223/5758276?ref=aligrithm.com) - [Characteristics Are Covariances: A Unified Model of Risk and Return (Kelly, Pruitt, and Su, 2019)](https://www.sciencedirect.com/science/article/abs/pii/S0304405X19301151?ref=aligrithm.com) - [Deep Learning in Asset Pricing (Chen, Pelger, and Zhu, 2024)](https://doi.org/10.1287/mnsc.2023.4695?ref=aligrithm.com) - [Forest Through the Trees: Building Cross-Sections of Asset Returns (Bryzgalova, Pelger, and Zhu)](https://papers.ssrn.com/sol3/papers.cfm?abstract%5Fid=3493458&ref=aligrithm.com) - [Economic Predictions with Big Data: The Illusion of Sparsity (Giannone, Lenza, and Primiceri, 2021)](https://onlinelibrary.wiley.com/doi/10.3982/ECTA17842?ref=aligrithm.com) - [The Factor Model Failure Puzzle (Baba-Yara, Boyer, and Davis)](https://papers.ssrn.com/sol3/papers.cfm?abstract%5Fid=3967588&ref=aligrithm.com) - [A Forward Looking View of Factor Investing (Shi, 2024)](https://papers.ssrn.com/sol3/papers.cfm?abstract%5Fid=5261822&ref=aligrithm.com) --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ### 10.8 Selection vs Diversification: Why |t|>3 Throws Away Alpha URL: https://aligrithm.com/selection-vs-diversification-why-t-3-throws-away-alpha/ Last updated: 2026-07-10T10:57:30.000Z The factor-zoo reckoning handed the profession a clean rule: raise the bar. If a new predictor cannot clear a t-statistic of 3, treat it as noise. Harvey, Liu and Zhu built the case, Chordia, Goyal and Saretto pushed the cross-sectional hurdle to 3.4, and the field mostly nodded along. That rule is correct for the question it answers. The question is whether one specific signal is real. But almost nobody who quotes the rule is answering that question. They are building portfolios, and for portfolio construction the same rule quietly destroys performance. Shingo Goto and Toru Yamada put a number on the damage. They rebuilt Yan and Zheng's data-mining machine, generated more than 18,000 fundamental-ratio signals, and found that roughly 80% of them are statistical nulls. Then they did the thing the inference crowd never bothers to do: they measured the out-of-sample information ratio of portfolios sorted by how strict the significance filter was. The strict bucket, the one that survives the |t|>3 hurdle, is the worst portfolio in the study. A portfolio that keeps every single signal, 80% garbage included, beats it. This article is about why, and about the exact point where the statistician's advice and the portfolio manager's payoff pull in opposite directions. The old article ["The Factor Zoo and the Multiple-Testing Reckoning"](https://aligrithm.ghost.io/the-factor-zoo-and-the-multiple-testing-reckoning/?ref=aligrithm.com) made the case for the high bar. This is the other side of that trade. And the old article ["The Backtest Integrity Checklist"](https://aligrithm.com/the-backtest-integrity-checklist/) already flagged the discipline that decides who is right here: know what you are optimizing for before you pick a threshold. ## The brackish environment: mostly noise, some real signal Start with the raw material. Following Yan and Zheng, Goto and Yamada compute financial ratios across 213 Compustat accounting variables and 15 base denominators, sort stocks into deciles on each ratio every year, and form value-weighted long-short portfolios (top decile minus bottom decile). After requiring at least 120 consecutive months of returns, roughly 13,932 signal portfolios are eligible in an average year, peaking above 15,000 after 2005\. That is an order of magnitude more predictors than there are stocks to predict, which is the defining feature of the modern factor problem. How much of it is real? Storey's estimator says most of it is not. The fraction of null signals, the ones with a true alpha of exactly zero, comes in at 79.1% for CAPM-adjusted returns and 80.6% for FF6-adjusted returns. The picture below shows why that estimate is credible: the empirical distribution of p-values hugs the uniform 45-degree line for thresholds above 50%, exactly what pure noise produces, but piles up an obvious excess mass near zero. ![Empirical distribution of in-sample p-values for CAPM and FF6 alphas, showing a spike of small p-values against a mostly uniform bulk](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_402-pvalue_distribution.png) The estimator reads that right tail. If almost every p-value above some cutoff belongs to a null signal, count them and scale up. $$ \\hat{\\pi}\_0(\\lambda) = \\frac{\\#\\{\\, p\_i > \\lambda \\,\\}}{(1-\\lambda)\\, m} $$ Here m is the total number of signals, p-sub-i is the p-value of signal i, and lambda is a cutoff pushed toward one where non-null signals rarely survive. The count of p-values above lambda, divided by the number a uniform distribution would put there, estimates the null fraction pi-zero. Work it at lambda equal to 0.5: if a genuinely null world would scatter half its p-values above 0.5, and you observe about 40% of your 13,932 signals sitting up there, then pi-zero is roughly 0.40 divided by 0.50, near 0.80\. Goto and Yamada minimize this across several lambdas and average across years to stay conservative, and still land at 79 to 81% null. That leaves 19 to 21% non-null, about 2,700 to 2,900 real signals a year. Real, but pathetic on their own. The paper's maximum-likelihood fit puts the average non-null signal at a t-ratio of 1.62, which corresponds to a p-value near 10.5%. The typical true signal would fail |t|>3 by a mile. This is the brackish water in the paper's title: weak fresh signal dissolved in an ocean of salt, impossible to separate cleanly one drop at a time. ## Two ways to count the false discoveries Before measuring portfolios, measure the contamination. The false discovery proportion is the share of your "significant" signals that are actually null. Two estimators bracket it. $$ \\widehat{\\text{FDP}}(\\tau) \\;=\\; \\hat{\\pi}\_0 \\cdot \\frac{\\tau}{\\hat{F}\_p(\\tau)} \\qquad\\qquad \\text{Easy FDP}(\\tau) \\;=\\; \\frac{\\tau}{\\hat{F}\_p(\\tau)} $$ Read the pieces. Tau is the p-value threshold you call significant. F-hat-p of tau is the fraction of all signals that actually clear tau. The ratio tau over F-hat-p of tau is Chen's Easy FDP, an upper bound that needs no estimate of the null fraction, since under pure noise you expect a fraction tau to clear by luck and you compare that against how many actually did. Multiply by the estimated null fraction pi-zero and you get Storey's sharper plug-in estimate. Work the conventional 5% threshold on CAPM alphas. Only 12.1% of signals clear it, so F-hat-p of 0.05 is 0.121\. Easy FDP is 0.05 divided by 0.121, which is 0.413\. So up to 41% of everything you just blessed at the 5% level is spurious. Fold in the 79.1% null rate and Storey's estimate drops to 0.791 times 0.413, about 0.327, still a third false. Even at the stringent 0.1% threshold the bound sits near 8 to 10%. The inference lesson is real: to get false discoveries below 5% you genuinely need to go tighter than |t|>3\. If your job is to certify a single factor, the high bar is not paranoia, it is arithmetic. ## The metric that matters for a portfolio is the information ratio Now switch jobs. A portfolio manager does not care whether signal number 8,412 is individually real. The manager cares about return per unit of risk on the combined book. The relevant object is the information ratio. $$ \\text{IR} \\;=\\; \\frac{\\hat{\\alpha}}{\\hat{\\omega}} \\sqrt{12} \\qquad\\qquad t \\;=\\; \\frac{\\hat{\\alpha}}{\\hat{\\omega}} \\sqrt{T} $$ Alpha-hat is the portfolio's monthly benchmark-adjusted return, omega-hat is its residual risk, the standard deviation of what the benchmark cannot explain. Their ratio is the monthly information ratio, and multiplying by the square root of 12 annualizes it. The t-statistic is the same ratio scaled by the square root of the number of months, so a higher information ratio and a stronger t-stat are the same statement. This is where strict selection destroys itself. Tightening the filter raises alpha per signal but shrinks the number of signals, and fewer signals means less diversification and higher residual risk. The residual risk climbs faster than the alpha. Take the CAPM numbers straight from the study. The strict p-less-than-0.1% bucket earns a monthly alpha of 0.268% on residual risk of 2.195%, so its information ratio is 0.268 divided by 2.195, times the square root of 12, which is 0.42\. The loose p-less-than-6% bucket earns a smaller alpha of 0.191% but on residual risk of only 1.272%, giving 0.191 divided by 1.272, times root 12, which is 0.52\. Lower alpha, higher information ratio. The strict bucket won the alpha contest and lost the only contest that pays. ## The information ratio peaks where the statistician tells you to stop Plot the information ratio against the significance threshold and the whole argument lands in one line. ![Out-of-sample annualized information ratio by in-sample p-value threshold for CAPM and FF6, peaking near 6 to 9 percent and lowest at the strict bucket](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_402-info_ratio_by_threshold.png) The peaks sit nowhere near the inference hurdle. For CAPM the best bucket is p-less-than-6%, holding around 1,900 signals at roughly 35% false discovery proportion, with an information ratio of 0.52 and a t-ratio of 3.29\. For FF6 the peak is p-less-than-9%, information ratio 1.01, t-ratio 6.40\. The strict p-less-than-0.1% bucket, the one that passes |t|>3, is the low point of both curves at 0.42 and 0.91\. Harvey, Liu and Zhu recommend p-less-than-0.27%, Chordia and coauthors p-less-than-0.07%. Both sit to the left of the dashed line, in the exact region where the portfolio does worst. Push it to the absurd end and the result holds. The bucket that keeps every signal, p-less-than-100%, roughly 80% of it null by construction, still beats the strict bucket: 0.48 versus 0.42 for CAPM, 0.94 versus 0.91 for FF6\. Feeding a portfolio 80% pure noise beat the disciplined filter that the profession spent a decade defending. That is not a rounding error, it is the thesis. ## Diversification, formalized: the Dybvig-Ross spanning test "Beats" needs a test, not a Sharpe-ratio eyeball. Goto and Yamada use the classical spanning regression from Dybvig and Ross: to ask whether portfolio A adds anything a mean-variance investor wants beyond an existing portfolio B, regress A's returns on B's and look at the intercept. $$ r\_{A,t} \\;=\\; a \\;+\\; b\\, r\_{B,t} \\;+\\; \\varepsilon\_t $$ r-A-t and r-B-t are the excess returns of the two portfolios, b is A's beta to B, and epsilon is the leftover. The intercept a is the whole point. A positive, significant a means A delivers return that B cannot span, so adding A improves the frontier. A negative a means A is a drag. Set B to the strict p-less-than-0.1% benchmark and let A run through the looser buckets. ![Incremental Dybvig-Ross alphas and t-statistics of each bucket over the strict benchmark, plotted against the proportion of signals included](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_402-incremental_diversification_alpha.png) The intercepts are positive almost everywhere. For FF6-adjusted returns every bucket above 3% delivers a positive, statistically significant incremental alpha, the largest at p-less-than-20% with 0.765% per year and a t-stat of 2.97, and the highest t-stat of 3.02 at p-less-than-40%. For CAPM the gains are more modest but still positive across a broad band, topping out at 0.635% per year at p-less-than-6%. Diversifying away from the strict bucket does not just look better, it spans the strict bucket and adds a frontier gain on top. The weak and null signals the inference rule told you to discard were carrying diversification you paid to throw away. ## Why noise does not hurt: equal weighting is shrinkage in disguise The mechanism is not mystical. A null signal has zero expected alpha by definition, so averaging it into an equal-weighted book does not drag the mean return down. What it does is dilute idiosyncratic estimation error. Every signal portfolio carries its own noise; average thousands of them and that noise partially cancels while any faint common alpha survives. Equal weighting across 14,000 signal portfolios is a brute-force version of the 1/N rule that Ledoit and Wolf and DeMiguel and coauthors show is hard to beat, and it behaves like implicit shrinkage and model averaging. It is the same instinct behind combining many weak indicators rather than betting the book on the single strongest one. Strict selection does the opposite. Keeping only the handful of most-significant signals maximizes in-sample alpha and maximizes prediction-error variance at the same time. Weights get concentrated and unstable, residual risk balloons, and the information ratio collapses, which is exactly the curve you saw. There is even a harder version of the claim in the statistics literature the paper leans on: Bishop showed that deliberately injecting noise into training is equivalent to Tikhonov regularization. Under the right conditions, noise is not just harmless, it disciplines the fit. ## Where this connects, and where it does not Read this next to the old article ["The Factor Zoo and the Multiple-Testing Reckoning"](https://aligrithm.ghost.io/the-factor-zoo-and-the-multiple-testing-reckoning/?ref=aligrithm.com) and the tension resolves. That reckoning is right that the tightest bar kills false factors. Goto and Yamada measure what else the tightest bar kills: the diversifying, individually-weak signals that carry portfolio performance. Same evidence, opposite prescription, because the objective changed from inference to construction. The old article ["The Backtest Integrity Checklist"](https://aligrithm.com/the-backtest-integrity-checklist/) already made this the first question to ask, and this paper is the cleanest case for why. Pick your threshold to match your goal, not your journal. Two honest caveats keep this from becoming a license to hoard noise. First, the portfolios are value-weighted on purpose, which strips out the microcap inflation that makes equal-weighted anomaly results look better than they trade. Second, the edge decayed. FF6 alphas in the strict bucket fell from 0.66% per month before 2003 to 0.19% after, a drop of roughly 70%, and Storey's diagnostics confirm it was not driven by more false discoveries but by genuine market efficiency and transaction costs eating the alpha. Diversification across signals still helps contain those costs through offsetting trades, but the free lunch shrank. The durable lesson is not "keep everything forever." It is that statistical significance and economic usefulness are different quantities, and a rule tuned for one will misprice the other. ## KEY POINTS - The |t|>3 hurdle from the factor-zoo literature answers an inference question, whether one signal is real. It is the wrong tool for portfolio construction, where the payoff is return per unit of risk on the combined book. - Data-mining more than 18,000 fundamental signals, Goto and Yamada estimate roughly 80% are statistical nulls (Storey pi-zero of 79 to 81%), and the average real signal has a t-ratio near 1.62, far below any strict hurdle. - The out-of-sample information ratio peaks at loose thresholds, p near 6% for CAPM (IR 0.52) and 9% for FF6 (IR 1.01), while the strict p-less-than-0.1% bucket is the worst performer at 0.42 and 0.91. - A portfolio holding every signal, about 80% of it null, still beats the strict bucket, because tightening selection raises alpha per signal but raises residual risk faster, sinking the information ratio. - The Dybvig-Ross spanning test confirms it: looser buckets deliver positive, significant incremental alpha over the strict benchmark, up to 0.77% per year for FF6, so the discarded weak signals were carrying diversification. - Equal weighting across thousands of signals acts as implicit shrinkage and model averaging, diluting estimation noise while faint common alpha survives, which is why adding null signals does not hurt and can help. - Caveats: results are value-weighted to avoid microcap inflation, and FF6 alphas decayed roughly 70% after 2003 from market efficiency and costs, not from more false discoveries. Match the threshold to the objective. ## References - [Fundamental Analysis and the Cross-Section of Stock Returns: A Data-Mining Approach (Yan and Zheng, 2017)](https://academic.oup.com/rfs/article-abstract/30/4/1382/2739544?ref=aligrithm.com) - [...And the Cross-Section of Expected Returns (Harvey, Liu, and Zhu, 2016)](https://academic.oup.com/rfs/article-abstract/29/1/5/1843824?ref=aligrithm.com) - [Anomalies and False Rejections (Chordia, Goyal, and Saretto, 2020)](https://academic.oup.com/rfs/article-abstract/33/5/2134/5555879?ref=aligrithm.com) - [A Direct Approach to False Discovery Rates (Storey, 2002)](https://doi.org/10.1111/1467-9868.00346?ref=aligrithm.com) - [Most Claimed Statistical Findings in Cross-Sectional Return Predictability Are Likely True (Chen, 2025)](https://arxiv.org/abs/2206.15365?ref=aligrithm.com) - [The Analytics of Performance Measurement Using a Security Market Line (Dybvig and Ross, 1985)](https://doi.org/10.1111/j.1540-6261.1985.tb04964.x?ref=aligrithm.com) - [Choosing Factors (Fama and French, 2018)](https://doi.org/10.1016/j.jfineco.2018.02.012?ref=aligrithm.com) - [Optimal Versus Naive Diversification: How Inefficient Is the 1/N Portfolio Strategy? (DeMiguel, Garlappi, and Uppal, 2009)](https://doi.org/10.1093/rfs/hhn075?ref=aligrithm.com) - [Selection Versus Diversification in Noisy Alpha Environments (Goto and Yamada, 2026)](https://www.sciencedirect.com/science/article/pii/S0378426626001007?ref=aligrithm.com) --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ### 10.7 Global vs Regional: When Complexity Wants More Data URL: https://aligrithm.com/global-vs-regional-when-complexity-wants-more-data/ Last updated: 2026-07-10T10:57:31.000Z For twenty years the international asset-pricing literature had a settled answer: regional beats global. Griffin showed Fama-French factors work better country by country than pooled. Fama and French themselves found local four-factor models explain international returns better than global ones. Hollstein agreed. The story was that markets are not fully integrated, momentum is weak in Japan, value behaves differently in Europe, so you should train a separate model for each region and let it learn the local quirks. That answer has a hole in it. Every one of those studies was ex-post and linear. Ex-post means they looked back and noticed that momentum was weak in Japan, which is not the same as a model predicting Japanese returns out of sample. Linear means they never asked what happens when the model is a gradient-boosted tree or a neural network instead of a regression. Chen, Hanauer, and Kalsbach redo the whole comparison the honest way, ex-ante across 24 developed markets with five algorithms, and the settled answer flips depending on one thing: model complexity. Simple linear models still slightly prefer local training. Complex models strongly prefer global. The old article ["Does ML Actually Help Asset Pricing? Kelly's 20%, Not 2–3×"](https://aligrithm.ghost.io/does-ml-actually-help-asset-pricing-kellys-20-not-2-3x/?ref=aligrithm.com) argued the nonlinear gain is real but modest. This is the sequel: that gain grows when you feed the complex model more data, and pooling the planet is how you feed it. ## The setup: same 36 signals, two training geographies The test is clean because only one thing changes. Twenty-four developed markets from July 1990 to December 2021, 36 firm characteristics, everything from book-to-market and momentum to Amihud illiquidity. Every characteristic is ranked cross-sectionally within each country and month onto the interval from minus one to plus one, so outliers cannot dominate, and missing values are set to zero. Microcaps are excluded, breakpoints come from big stocks, and portfolios are value-weighted long-short quintiles rebalanced monthly. Five algorithms run the gauntlet: OLS, elastic net, random forest, gradient-boosted regression trees, and a neural network ensemble. Then the only lever: where the training data comes from. Regional training fits a separate model for each of North America, Europe, Japan, and Asia Pacific excluding Japan, then pools the forecasts. Global training fits one model on all markets at once. Same characteristics, same portfolios, same expanding-window discipline that never lets future data leak backward. The only question is whether learning is better done locally, where the patterns are region-specific, or globally, where there is far more data. ## The headline: complexity flips the sign The long-short portfolios get judged by the t-statistic of their mean monthly return. $$ t = \\frac{\\bar{r}}{\\operatorname{std}(r)/\\sqrt{T}} $$ Read it as the average monthly spread return divided by its standard error, where the standard error is the return's standard deviation over the square root of the number of months. Work the winning model: the globally trained neural network earns 0.74% per month with a standard deviation of 1.99% over the 240 months from 2002 to 2021, so the t-stat is 0.74 divided by (1.99 over the square root of 240), which is 0.74 over 0.128, about 5.76, matching the paper's reported 5.73\. Its local twin earns only 0.62% with a t of 4.62\. For the linear models the ranking reverses: local OLS earns 0.47% against global OLS at 0.43%, and local elastic net beats global 0.43% to 0.39%. Simple models want to stay home; complex models want to travel. ![Cumulative performance of four long-short portfolios from 2002 to 2022, with the globally trained neural network climbing far above the local neural network and both OLS lines](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_401-cumulative_global_vs_local.png) The cumulative curve makes the gap physical. Over the 20-year sample the globally trained neural network compounds to a 456% cumulative return, pulling away from the local neural network and leaving both OLS lines flat on the floor. The two linear models, local and global, are almost indistinguishable down there. All the action, and all the local-versus-global divergence, lives in the complex models. ## Spanning tests: which one is redundant Higher raw returns are not proof of added information, because the two models might just be repackaging the same signal. The spanning regression settles it. Regress one model's long-short return on the other's and look at the intercept. $$ r^{\\text{global}}\_t = \\alpha + \\beta\\, r^{\\text{local}}\_t + \\varepsilon\_t $$ The slope beta soaks up whatever the two portfolios share. The intercept alpha is the part of the global model's return that the local model cannot explain. A positive, significant alpha means global training delivers information local training misses. Run it both directions and the asymmetry is stark. Regress global on local and the neural network alpha is 0.20% per month with a t-stat of 3.45, and gradient-boosted trees give 0.15% with a t of 2.04, both significant. Reverse it, regress local on global, and the alphas are insignificant or negative, minus 0.07% for the neural network. Global spans local; local does not span global. The local model is the redundant one. ![Bar chart of global-on-local spanning alphas rising with model complexity, negative for OLS and elastic net, significantly positive for gradient-boosted trees and neural networks](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_401-spanning_alpha_by_complexity.png) Line the alphas up by complexity and you see the sign flip directly. OLS and elastic net sit below zero and insignificant, meaning global adds nothing to linear models. Random forest is positive but weak. Only at the complex end, gradient-boosted trees and the neural network, does the global-on-local alpha turn significantly positive. Complexity is precisely the regime where global data earns its keep. ## Why complexity is data-hungry The mechanism is a counting argument, not magic. What matters for estimation is the ratio of observations to parameters. $$ \\text{observations-to-parameters} = \\frac{N\_{\\text{stocks}} \\times T\_{\\text{months}}}{P\_{\\text{parameters}}} $$ The more data points you have per parameter you must estimate, the more reliably you pin each parameter down. A linear model over 36 characteristics estimates 36 coefficients, and a single region has plenty of stock-months to nail 36 numbers, so pooling the world buys it almost nothing. The neural network is a different animal. Its deepest ensemble member runs 36 inputs into layers of 32, 16, 8, 4, and 2 neurons, which works out to roughly 1,900 weights to estimate (that count is approximate, from the architecture in the paper's appendix). Feed 1,900 parameters the stock-months of Japan alone and the model is starving, fitting noise. Feed it 24 pooled markets and the observations-to-parameters ratio jumps by whatever multiple the pooling provides, and the same architecture that overfit locally now estimates cleanly. Asset-pricing signal is faint and the process drifts, so complex models need every observation they can get, and geography is a cheap way to multiply observations without multiplying parameters. This is the same logic as the old article ["The Difference Between Robustness and Optimization"](https://aligrithm.com/the-difference-between-robustness-and-optimization/): more data pushes the estimate toward the robust plateau and away from region-specific noise the model would otherwise mistake for signal. ## Even Japan bends If any market should reward local training, it is Japan, the poster child for idiosyncratic behavior and dead momentum. It does not. ![Grouped bar chart of Sharpe ratios by region and model, with globally trained neural networks highest in all four regions including Japan](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_401-sharpe_by_region.png) Split the results into four subregions and the globally trained neural network wins everywhere. North America: 1.03 global against 0.85 local. Europe: 0.96 against 0.81\. Japan, supposedly the exception: 0.68 global against 0.42 local, the global edge actually wider here than elsewhere. Asia Pacific: 1.18 against 0.78\. The region-specific-patterns story, the whole justification for local models, does not survive out of sample once the model is complex enough to use the global data. The linear models keep their split personality: global wins in Europe and Asia Pacific, local wins in North America and Japan, no consistent edge either way, which is exactly why the old linear literature could conclude that local wins and not be wrong on its own narrow terms. ## Where this connects This article extends the old article ["Does ML Actually Help Asset Pricing? Kelly's 20%, Not 2–3×"](https://aligrithm.ghost.io/does-ml-actually-help-asset-pricing-kellys-20-not-2-3x/?ref=aligrithm.com) by adding the missing variable. There the verdict was that nonlinear models beat linear ones by a modest margin. Here the margin turns out to be conditional: the complex model only delivers if you feed it enough data, and starving it on a single region throws the advantage away. It also sharpens the old article ["The Difference Between Robustness and Optimization."](https://aligrithm.com/the-difference-between-robustness-and-optimization/) Global training is a robustness move dressed as a data-scale move. Pooling 24 markets does not optimize to any region's quirks; it forces the model onto patterns that repeat across geographies, which is the definition of robustness by cross-market variation. Two caveats keep this honest. The sample is developed markets from 2002 to 2021 and excludes microcaps by construction, so it sidesteps the tiny-illiquid-stock problem that haunted the machine-learning gains in the previous article. And the finding sits inside a live and unresolved fight over whether complexity is a virtue at all, with Kelly, Malamud, and Zhou arguing more parameters keep helping and Buncic arguing the effect is a mirage of how you scale the inputs. This paper is a data point for the pro-complexity side, not a verdict. The honest read is narrow and useful: if you are going to run a complex model, train it on as much of the world as you can. ## KEY POINTS - The old international-asset-pricing consensus, that regional models beat global, rested on ex-post, linear studies. Redone ex-ante with five algorithms across 24 developed markets, the answer flips with model complexity. - Linear models slightly prefer local training (local OLS 0.47% per month versus global 0.43%). Complex models strongly prefer global: the global neural network earns 0.74% per month with a t of 5.73 versus 0.62% local, and compounds to a 456% cumulative return. - Spanning tests break the tie. Global-on-local alpha is 0.20% (t of 3.45) for the neural network and 0.15% (t of 2.04) for boosted trees, both significant; local-on-global alphas are insignificant or negative. Global spans local, not the reverse. - The mechanism is the observations-to-parameters ratio. A linear model estimates 36 coefficients and a single region suffices; a neural network with roughly 1,900 weights starves on one region and needs pooled global data to estimate cleanly. - The result is robust across regions. The global neural network posts the highest Sharpe in all four subregions, including Japan (0.68 versus 0.42 local), killing the region-specific-patterns justification for local models. - Global training is a robustness move: pooling markets forces the model onto patterns that repeat across geographies rather than fitting one region's noise. - Caveats: developed markets only, microcaps excluded, 2002 to 2021, and the finding sits inside the unresolved virtue-of-complexity debate. The takeaway is conditional, not a law: if you run a complex model, feed it the whole world. ## References - [Are the Fama and French Factors Global or Country Specific? (Griffin, 2002)](https://doi.org/10.1093/rfs/15.3.783?ref=aligrithm.com) - [Size, Value, and Momentum in International Stock Returns (Fama and French, 2012)](https://doi.org/10.1016/j.jfineco.2012.05.011?ref=aligrithm.com) - [Local, Regional, or Global Asset Pricing? (Hollstein, 2022)](https://doi.org/10.1017/S0022109021000028?ref=aligrithm.com) - [Empirical Asset Pricing via Machine Learning (Gu, Kelly, and Xiu, 2020)](https://doi.org/10.1093/rfs/hhaa009?ref=aligrithm.com) - [Characteristics Are Covariances: A Unified Model of Risk and Return (Kelly, Pruitt, and Su, 2019)](https://doi.org/10.1016/j.jfineco.2019.05.001?ref=aligrithm.com) - [The Virtue of Complexity in Return Prediction (Kelly, Malamud, and Zhou, 2024)](https://doi.org/10.1111/jofi.13298?ref=aligrithm.com) - [Simplified: A Closer Look at the Virtue of Complexity in Return Prediction (Buncic, 2025)](https://papers.ssrn.com/sol3/papers.cfm?abstract%5Fid=5239006&ref=aligrithm.com) - [Replicating Anomalies (Hou, Xue, and Zhang, 2020)](https://doi.org/10.1093/rfs/hhy131?ref=aligrithm.com) - [Model Complexity and the Performance of Global versus Regional Models (Chen, Hanauer, and Kalsbach, 2025)](https://doi.org/10.1016/j.econlet.2025.112641?ref=aligrithm.com) --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ### 10.5 The Factor Zoo and the Multiple-Testing Reckoning URL: https://aligrithm.com/the-factor-zoo-and-the-multiple-testing-reckoning/ Last updated: 2026-07-10T10:57:31.000Z The old articles built the machine that grades a factor. Sort the cross-section, run the GRS test, run Fama-MacBeth, and out pops a t-stat. Every one of those tools treats your factor as if it were the only hypothesis you ever tried. It never is. The profession has run that machine on hundreds of candidate signals, kept the winners, and buried the losers in unpublished drawers, and the survivors get reported with a t-stat of 2 as if it meant something. It does not. A t of 2 is a 5% false-positive bar, and if you test three hundred dead signals, roughly fifteen of them clear that bar by luck alone. That pile of survivors has a name: the factor zoo. Harvey, Liu, and Zhu cataloged more than three hundred published factors, and the number keeps climbing as researchers mine trading accounts, order books, social media, news text, satellite images, weather, and Google search volume for anything that correlates with next month's returns. Most of the zoo is noise that got published because nobody adjusted the significance bar for the search. This is the cross-sectional twin of the backtest-overfitting problem from the old articles ["CSCV: A Direct Probability of Backtest Overfit"](https://aligrithm.com/cscv-a-direct-probability-of-backtest-overfit/) and ["Permutation Tests for Indicator Significance."](https://aligrithm.com/permutation-tests-for-indicator-significance/) There the danger was one researcher sweeping parameters. Here it is a whole field sweeping signals. ## The zoo is real and it is enormous Look at where the candidate signals come from now. Trading account records, limit order book events, message-board sentiment, crowdsourced earnings forecasts, the text of 10-K filings, photographs in news articles, sunshine in the city where the exchange sits, and search-engine attention. Each data source spawns dozens of characteristics, each characteristic gets sorted into a long-short portfolio, and each portfolio gets a t-stat. Multiply it out and the count of tested signals runs into the thousands even if only three hundred reach print. Quantity is the tell. When a field produces a new priced factor every few weeks for two decades, the base rate says most of them cannot be real, because real compensated risks are scarce and the market is not that inefficient in that many independent ways. The skeptic's default is that a freshly published anomaly is a false discovery until it survives a correction for how many signals were tried to find it. ## Why one t-stat lies when you ran three hundred Start with the arithmetic that breaks the naive t-stat. A single test at the 5% level has a 5% chance of a false positive when the factor is truly worthless. Run K independent such tests on worthless factors and the chance that at least one prints significant is not 5%, it climbs fast. $$ \\text{FWER} = \\operatorname{prob}(\\text{at least one false positive}) = 1 - (1-\\alpha)^K $$ That is the family-wise error rate, the probability of making even one false discovery across the whole family of K tests, where alpha is the per-test level. Read it as one minus the probability that every single test correctly stays quiet. Work it for the zoo. With alpha of 0.05 and K equal to 316, one minus 0.95 to the power 316 is 0.99999991, indistinguishable from certainty. You are not at risk of a false factor, you are guaranteed a crowd of them. The expected count is even more concrete: if all 316 candidates were truly worthless, alpha times K is 0.05 times 316, about 16 factors clearing a t of 2 by pure luck. The zoo has a false-discovery floor baked in. ![Curve showing the required t-statistic hurdle rising with the number of factors tested on a log scale, crossing t equals 3 near 20 factors and reaching 3.78 at 316 factors](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_399-bonferroni_threshold.png) The fix is to raise the bar as the search widens. The curve shows exactly how much. Test one factor and a t of 1.96 is fine. Test twenty and the honest hurdle is already past 3\. Test the 316 of the published zoo and it sits near 3.8\. The single-test t of 2 that fills asset-pricing papers is calibrated for a world where you tried one idea, and nobody tried one idea. ## The error table: FWER, FDR, FDP Multiple-testing control formalizes the bookkeeping. Sort every test into whether the factor is truly worthless or truly priced, and whether you rejected the null or not. Of K factors, K-zero are truly worthless and K-one are truly priced. You reject L of them, and those rejections split into T-one true discoveries and F-one false discoveries. Three different error rates target three different fears. $$ \\text{FWER}=\\operatorname{prob}(F\_1 \\geq 1)\\leq\\alpha, \\qquad \\text{FDR}=\\mathbb{E}\\!\\left\[\\frac{F\_1}{L}\\right\]\\leq\\alpha, \\qquad \\text{FDP}=\\operatorname{prob}\\!\\left(\\frac{F\_1}{L}\\geq\\gamma\\right)\\leq\\alpha $$ The family-wise error rate caps the probability of even one false factor in the whole batch, which is brutally strict when K is large. The false discovery rate is gentler: it caps the expected fraction of your discoveries that are false, so if you announce 20 factors and accept an FDR of 5%, you tolerate about one of them being junk on average. The false discovery proportion caps the probability that the junk fraction exceeds some threshold gamma, a middle ground. Work the FDR: announce L equal to 20 factors at an FDR of 0.05 and you expect F-one near 1, so nineteen are likely real and one is likely noise, which is a very different promise from "each of these has a t of 2." ## Two corrections: Bonferroni and Benjamini-Hochberg The oldest correction is Bonferroni, and it controls the family-wise rate by brute force. Divide the significance level by the number of tests. $$ \\text{reject factor } i \\text{ if } \\; p\_i \\leq \\frac{\\alpha}{K} $$ Every factor must clear a per-test bar of alpha over K. With alpha of 0.05 and K of 316, the threshold is 0.05 divided by 316, which is 0.000158, and a two-sided p-value that small corresponds to a t-stat of about 3.8\. Bonferroni is simple and safe and throws away real factors, because holding the probability of any false positive to 5% across hundreds of tests forces the bar so high that weak-but-genuine premia get rejected too. Benjamini and Hochberg traded that severity for power. Instead of guarding against any false positive, they control the false discovery rate. Sort the p-values from smallest to largest, then walk down the list. $$ \\text{reject the } i \\text{ smallest } p\\text{-values, where } i \\text{ is the largest index with } \\; p\_{(i)} \\leq \\frac{i}{K}\\,\\alpha $$ The threshold slides: the smallest p-value is judged against alpha over K, the second against two-alpha over K, and so on, so a factor does not have to clear the full Bonferroni bar if many other factors are also significant. It keeps more true discoveries while still capping the junk fraction. Harvey, Liu, and Zhu ran these corrections across the published zoo and landed on a practical hurdle: a factor needs a t-stat above about 3.0, not 2.0, before you should believe it. That single number retires a large slice of the literature, and it is the correction the old articles ["CSCV: A Direct Probability of Backtest Overfit"](https://aligrithm.com/cscv-a-direct-probability-of-backtest-overfit/) and ["Permutation Tests for Indicator Significance"](https://aligrithm.com/permutation-tests-for-indicator-significance/) were pointing at from the time-series side. ## The 36% haircut: attenuation out of sample Clearing the corrected bar in-sample is still not the finish line. Even factors that pass decay once you leave the sample they were discovered in. Harvey and Liu measured the drop at about 36% for U.S. stocks, and the deck's author reports a similar attenuation for the Chinese market. A factor with an in-sample premium of 6% a year is a 3.8% factor once real trading meets it, before costs. ![Scatter of in-sample against out-of-sample annualized Sharpe ratios for fifteen anomaly strategies, with every point sitting below the 45-degree line](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_399-in_vs_oos_sharpe.png) The scatter makes the haircut visual. Fifteen anomalies, in-sample Sharpe on one axis and out-of-sample Sharpe on the other, and every point sits below the 45-degree line where the two would match. Nothing lands above it. The ranking survives, better in-sample strategies tend to be better out of sample, but the level shrinks across the board. That gap is the combined bill for the multiple testing that found the factor and the crowding that arrives once it is public, the same alpha-decay-is-competition theme from earlier pillars, now measured on the cross-section. ## The illusion of sparsity The obvious reaction is to prune hard: keep three or four factors and throw the zoo out. The data says that overcorrects. Giannone and coauthors called it the illusion of sparsity, and Kozak, Nagel, and Santosh showed it directly for the cross-section of returns. When you build the pricing kernel that actually maximizes out-of-sample Sharpe, it does not load on a handful of characteristics. It spreads weight across almost all of them. ![Contour plot of out-of-sample performance against prior strength on one axis and number of nonzero coefficients on the other, with the bright high-performance region requiring many nonzero coefficients](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_399-illusion_of_sparsity.png) The bright region, where out-of-sample predictability peaks, sits at a large number of nonzero coefficients paired with a moderate prior. Shrink to a sparse handful of factors and you slide off the good region into the dark. So the reckoning cuts both ways. Most individual published factors are false, which argues for a higher bar. But the true pricing kernel is not sparse, which argues against compressing everything into five factors. The resolution is not a t-stat lottery and not a five-factor dogma. It is heavy shrinkage toward the whole cross-section under an economic prior, and the prior comes from theory: Kozak, Nagel, and Santosh argue that any factor with a high premium must also be a major source of comovement, because otherwise it would offer a near-arbitrage Sharpe ratio that cannot survive. Return has to be tied to risk structure, or it is a ghost. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_399-visual-0.png) ## Where this connects The factor zoo is backtest overfitting scaled from one desk to an entire profession. The old article ["CSCV: A Direct Probability of Backtest Overfit"](https://aligrithm.com/cscv-a-direct-probability-of-backtest-overfit/) measures how badly a single parameter search will disappoint out of sample; the multiple-testing correction here measures how badly a signal search across thousands of candidates will. The old article ["Permutation Tests for Indicator Significance"](https://aligrithm.com/permutation-tests-for-indicator-significance/) builds a null distribution from the data so one indicator's p-value is honest; the corrections in this article fix the threshold once you admit you computed that p-value hundreds of times. Same disease, two scales. The next step is that the tightest bar is not the right bar. Pushing the hurdle to a t of 3 kills false factors but also throws away real, diversifying ones, and the out-of-sample information ratio of a factor book can peak well below the strictest cutoff. Where that optimum sits, and why the selection-versus-diversification tradeoff means the academy's significance bar and the practitioner's diversification bar point in opposite directions, is the subject of the FDR-curve article to come. ## KEY POINTS - The factor zoo is the pile of more than three hundred published factors mined from trading accounts, order books, text, images, weather, and search data. Most are false discoveries because the significance bar was never adjusted for how many signals were tried. - A t-stat of 2 is a single-test 5% bar. Test K worthless factors and the chance of at least one false positive is one minus 0.95 to the power K, which for the 316-factor zoo is essentially certain, with about 16 factors expected to clear t of 2 by luck. - Multiple-testing control targets three different errors: the family-wise error rate caps the probability of any false factor, the false discovery rate caps the expected fraction of discoveries that are false, and the false discovery proportion caps the probability that fraction exceeds a threshold. - Bonferroni divides the significance level by K and needs a t near 3.8 for 316 tests, which is safe but throws away real factors. Benjamini-Hochberg controls the false discovery rate with a sliding threshold, keeping more true discoveries. Harvey, Liu, and Zhu land on a practical hurdle of t above 3.0, not 2.0. - Survivors still decay: Harvey and Liu found about 36% out-of-sample attenuation for U.S. stocks, similar in China. A 6% in-sample premium becomes about 3.8% before costs, and every one of fifteen anomalies had a lower out-of-sample than in-sample Sharpe. - The illusion of sparsity: you cannot fix the zoo by pruning to a handful of factors, because the pricing kernel that maximizes out-of-sample Sharpe loads on almost all characteristics. The fix is heavy shrinkage toward the whole cross-section under an economic prior, not a five-factor dogma. - The economic prior: a high-premium factor must also drive comovement, or it implies a near-arbitrage Sharpe ratio that will not survive. Return tied to risk structure is real; return untethered from it is a ghost. ## References - [... and the Cross-Section of Expected Returns (Harvey, Liu, and Zhu, 2016)](https://academic.oup.com/rfs/article-abstract/29/1/5/1843824?ref=aligrithm.com) - [Presidential Address: The Scientific Outlook in Financial Economics (Harvey, 2017)](https://onlinelibrary.wiley.com/doi/10.1111/jofi.12530?ref=aligrithm.com) - [Lucky Factors (Harvey and Liu, 2021)](https://doi.org/10.1016/j.jfineco.2021.04.014?ref=aligrithm.com) - [Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing (Benjamini and Hochberg, 1995)](https://www.jstor.org/stable/2346101?ref=aligrithm.com) - [Shrinking the Cross-Section (Kozak, Nagel, and Santosh, 2020)](https://doi.org/10.1016/j.jfineco.2019.06.008?ref=aligrithm.com) - [Interpreting Factor Models (Kozak, Nagel, and Santosh, 2018)](https://onlinelibrary.wiley.com/doi/10.1111/jofi.12612?ref=aligrithm.com) - [Economic Predictions with Big Data: The Illusion of Sparsity (Giannone, Lenza, and Primiceri, 2021)](https://onlinelibrary.wiley.com/doi/10.3982/ECTA17842?ref=aligrithm.com) - [A Forward Looking View of Factor Investing, DDA3600: Factor Investing (Chuan Shi, 2024)](https://papers.ssrn.com/sol3/papers.cfm?abstract%5Fid=5261822&ref=aligrithm.com) --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ### 10.6 Does ML Actually Help Asset Pricing? Kelly's 20%, Not 2–3× URL: https://aligrithm.com/does-ml-actually-help-asset-pricing-kellys-20-not-2-3x/ Last updated: 2026-07-10T10:57:32.000Z The pitch writes itself. Feed a neural network four hundred firm characteristics, let it learn the nonlinear interactions that linear factor models miss, and watch the out-of-sample Sharpe ratio jump. The published tables cooperate: a random forest at 2.19, a neural network at 3.21, an IPCA model at 3.39, against a Fama-French three-factor Sharpe closer to 0.5\. Read those numbers straight and machine learning looks like it quadruples your risk-adjusted return. Then listen to Bryan Kelly, who builds these models for a living. His own words: you should not expect machine learning to double or triple your Sharpe ratio, and a 20 percent bump from adding nonlinearities feels realistic to him. That gap, between the 2-to-3x the tables imply and the 20% the practitioner will admit to, is the whole story. This deck by Chuan Shi is a facts-and-fictions teardown of machine learning in factor investing, and the honest verdict is that ML helps at the margin while the headline numbers are a magic trick built on unfair comparisons, fragile tuning, and untradable stocks. It is the same discipline as the old article ["Optimization Comes After Testing, Not Before"](https://aligrithm.com/optimization-comes-after-testing-not-before/): the impressive first number is never the deployable number. ## The evaluation has to survive itself Before you believe any machine-learning Sharpe, ask how it was validated, because the validation protocol decides whether the number means anything. Plain k-fold cross-validation shuffles the data into folds and assumes the observations are independent. Financial data is not independent. Consecutive months are correlated, so a fold that trains on month t and tests on month t plus one lets the model learn the answer through the back door. Walk-forward respects time order but gives you exactly one path through history, one lucky or unlucky sequence, and no distribution to judge it against. Combinatorial purged cross-validation, from Lopez de Prado, fixes both. Split the sample into N blocks, use K of them for testing, and enumerate every way to choose those K blocks. $$ \\text{number of backtest paths} = \\binom{N}{K} $$ The count is the binomial coefficient, N choose K, the number of distinct test groups you can form. Work it with the deck's example: N equal to 6 blocks and K equal to 2 gives 6 choose 2, which is 15 backtest paths instead of one. Each path trains on the other blocks, with a purge buffer deleting the observations adjacent to the test blocks so no correlated neighbor leaks across the boundary. Now you have a distribution of 15 out-of-sample Sharpes, and you can ask whether the median is any good rather than betting on a single draw. This is the cross-sectional cousin of the overfitting diagnostics from Pillar 3, and it is the first filter any ML result has to pass. ## The penalized-regression zoo is priors with a dial Most of what gets sold as machine learning in asset pricing is regularized linear regression. You take the OLS objective and bolt on a penalty that punishes large coefficients. $$ \\min\_{\\beta\_1,\\dots,\\beta\_K} \\sum\_{i=1}^{N}\\left(y\_i - \\beta\_0 - \\beta\_1 x\_{1i} - \\cdots - \\beta\_K x\_{Ki}\\right)^2 + \\lambda \\sum\_{j=1}^{K} f(\\beta\_j) $$ The first term is ordinary least squares, the sum of squared prediction errors. The second term is the penalty, lambda times a function of the coefficients, and the intercept is deliberately left out of it. Lambda is the dial. Set the penalty function to the sum of squared coefficients and you get ridge regression; set it to the sum of absolute values and you get the lasso, which drives weak coefficients exactly to zero and performs variable selection for free. Worked at the extremes: as lambda goes to zero the penalty vanishes and you are back to plain OLS, and as lambda goes to infinity every coefficient is crushed to zero and the model predicts the mean. Everything useful happens in between, and validation picks the spot. The honest way to read this is Bayesian, and the deck says so. Ridge is what you get when you put a normal prior centered at zero on the coefficients, and the ridge lambda is precisely the ratio of noise variance to prior variance, sigma-squared over sigma-zero-squared. The lasso is the same idea with a Laplace prior, which has a spike at zero and so produces sparsity. There is no magic here. Penalized regression is you telling the model, before it sees the data, that most characteristics probably do not matter much, and then letting the data argue you out of it. On the deck's own test, 54 characteristics on the CSI 500 with a rolling 36-month window, the elastic net beat OLS out of sample. Real, useful, and about as revolutionary as admitting you have a prior. ## The nonlinear end: autoencoders and an adversarial SDF The genuinely nonlinear models are where the 2-to-3x dream lives. A conditional autoencoder, from Gu, Kelly, and Xiu, lets factor exposures move: instead of a static beta, the loading on each latent factor becomes a neural-network function of the firm's characteristics, so a stock's risk profile updates as its fundamentals change. The deepest version is the adversarial approach of Chen, Pelger, and Zhu. Start from the no-arbitrage condition, that the stochastic discount factor prices every excess return to zero, and turn model-building into a fight. $$ \\min\_{w}\\ \\max\_{g}\\ \\frac{1}{N}\\sum\_{j=1}^{N}\\left\\|\\ \\mathbb{E}\\!\\left\[\\left(1 - \\sum\_{i=1}^{N} w(I\_t, I\_{i,t})\\, R^e\_{i,t+1}\\right) R^e\_{i,t+1}\\, g(I\_t, I\_{i,t})\\right\]\\right\\|^2 $$ Read it as a zero-sum game. The inner term is a pricing error: one minus the SDF's weighted returns, multiplied by a test-asset return built by the function g. One network, w, plays the asset pricer and tries to minimize those pricing errors. The adversary network, g, plays the attacker and searches for the managed portfolios where the SDF fails worst, maximizing the error. They iterate: the attacker finds the most mispriced corner of the market, the pricer patches it, repeat until the attacker cannot find a big miss. On Chen and coauthors' numbers this genuinely wins, a test Sharpe of 0.75 for the GAN model against 0.44 for a plain feed-forward network and 0.50 for the elastic net, with the highest cross-sectional R-squared of the batch at 0.23\. So nonlinearity does add something real. The question is how much of it you keep. ![Classic double-descent curve: test risk falls, rises to a peak at the interpolation threshold, then falls again in the over-parameterized regime, while training risk decays monotonically](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_400-double_descent.png) The theoretical cover for going complex is double descent. The old bias-variance story says test error rises once a model has more parameters than it can justify. The modern picture, from Belkin and coauthors, adds a second descent: push past the interpolation threshold, where the model has enough capacity to fit the training data exactly, and test error can fall again. Over-parameterized models can generalize. Fine. But asset-pricing data has a signal-to-noise ratio near the floor and a data-generating process that will not sit still, so the second descent is a much dicier bet here than in image recognition. ## The headline Sharpes are a magic trick Now the teardown. Three things inflate the published numbers. First, the comparison is rigged: the machine-learning model gets hundreds of characteristics while the Fama-French benchmark gets three, so of course the kitchen sink wins. Give both the same information and the gap shrinks. Second, the results are fragile. Re-evaluate the same models carefully and the Sharpes fall hard. ![Grouped bar chart of headline versus robust out-of-sample Sharpe ratios for four models, with the random forest collapsing from 2.19 to 0.86 and the linear IPCA barely moving from 3.39 to 3.21](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_400-headline_vs_robust_sharpe.png) The random forest headline of 2.19 becomes 0.86 under a fair test, a 61% haircut. The neural network drops from 3.21 to 2.39\. The one model that barely flinches is IPCA, from 3.39 to 3.21, and IPCA is essentially a linear model, characteristics-as-covariances estimated by cross-sectional regression. The most linear method is the most robust and finishes on top. That is the opposite of the ML-doubles-your-Sharpe story. Third, and this is Kelly's point stated as math, linear models are the first-order approximation and first-order approximations capture most of the signal, so nonlinearity is a second-and-third-order correction. A 20% Sharpe improvement is a good result. Expecting 200% is expecting the correction terms to dwarf the main term, which is not how good approximations work. ## The untradable fine print Suppose the gain survives all that. You still have to trade it, and here the machine-learning portfolios fall apart. Their edge comes from constant rebalancing. ![Bar chart of monthly turnover: machine-learning models between 0.98 and 1.66, against traditional size and value factors at roughly 0.10](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_400-ml_turnover.png) The turnover numbers from Avramov and coauthors are brutal: the Gu-Kelly-Xiu model turns over 98% of the book every month, Kelly-Pruitt-Su 119%, the autoencoder 156%, the Chen-Pelger-Zhu GAN 166%. Traditional size and value factors sit below 10%. A model that rebuilds itself every month is paying spreads and impact ten to sixteen times as often as a value sleeve. At a rough 30 basis points round-trip cost, 100% monthly turnover is about 3.6% a year in frictions, enough to swallow the entire nonlinear premium (that cost figure is a back-of-envelope, flag it as illustrative). Worse, the deck's Fact 5 shows where the predictability lives: in the short legs and the tiny, illiquid, distressed, unrated stocks. Exclude the microcaps and the nonlinear models decline sharply while IPCA stays consistent across the full sample and the subsamples. The alpha is concentrated exactly in the names an institution cannot short cheaply or size into. On paper it is real. In a fund it is a rumor. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_400-visual-0.png) ## Where this connects This is the same lesson the old article ["Optimization Comes After Testing, Not Before"](https://aligrithm.com/optimization-comes-after-testing-not-before/) taught in the time-series world, transplanted to the cross-section. There the trap was a team polishing an in-sample Sharpe of 1.4 that evaporated on deployment. Here the trap is a published out-of-sample Sharpe of 3.2 that halves under a fair comparison and dies after costs. Combinatorial purged cross-validation plays the role the parameter-stability map played there: it turns one flattering number into a distribution you can be skeptical about. The machine-learning arc that runs through the Idea 3 articles on regression trees, random forests, and gradient boosting supplies the algorithms; this article supplies the reason to distrust their brochure. The next question is when the complexity is actually worth paying for. Global models trained on 24 markets and dozens of characteristics can beat local ones, but only when there is enough data to feed the extra parameters, and linear methods still slightly prefer staying local. That data-hunger-versus-complexity tradeoff is the subject of the global-versus-regional article to come. ## KEY POINTS - The headline pitch is that machine learning multiplies your Sharpe ratio 2-to-3x. Bryan Kelly, who builds these models, says expect about 20% from nonlinearity, not 200%. That gap is the entire argument. - Validate before you believe. K-fold assumes independent data and leaks through correlated neighbors; walk-forward gives one path. Combinatorial purged cross-validation enumerates N-choose-K test groups (15 paths for N=6, K=2) with a purge buffer, turning one Sharpe into a distribution. - Most ML in asset pricing is penalized regression: OLS plus a lambda-weighted coefficient penalty. Ridge is a Gaussian prior with lambda equal to noise-to-prior variance ratio; the lasso is a Laplace prior that zeroes weak coefficients. It is priors with a dial, not magic. - The nonlinear models are real: conditional autoencoders let betas move with characteristics, and the Chen-Pelger-Zhu adversarial GAN-SDF (a min-max game between a pricer and an attacker) hit a test Sharpe of 0.75 versus 0.44 for a plain network. - The headline Sharpes are inflated three ways: unfair comparisons (hundreds of characteristics versus three factors), fragility (random forest 2.19 to 0.86, a 61% cut, under a fair test), and Kelly's first-order point that linear models already capture most of the signal. - IPCA, essentially a linear model, is the most robust of the batch (3.39 to 3.21) and finishes on top. The most linear method wins, the opposite of the ML-doubles-your-Sharpe story. - Even real gains may be untradable. ML portfolios turn over 98% to 166% monthly versus under 10% for size and value, and their alpha concentrates in tiny, illiquid, distressed, short-leg stocks that institutions cannot cheaply trade. ## References - [Empirical Asset Pricing via Machine Learning (Gu, Kelly, and Xiu, 2020)](https://doi.org/10.1093/rfs/hhaa009?ref=aligrithm.com) - [Autoencoder Asset Pricing Models (Gu, Kelly, and Xiu, 2021)](https://doi.org/10.1016/j.jeconom.2020.07.009?ref=aligrithm.com) - [Deep Learning in Asset Pricing (Chen, Pelger, and Zhu, 2024)](https://doi.org/10.1287/mnsc.2023.4695?ref=aligrithm.com) - [Machine Learning versus Economic Restrictions: Evidence from Stock Return Predictability (Avramov, Cheng, and Metzker, 2023)](https://doi.org/10.1287/mnsc.2022.4449?ref=aligrithm.com) - [Characteristics Are Covariances: A Unified Model of Risk and Return (Kelly, Pruitt, and Su, 2019)](https://doi.org/10.1016/j.jfineco.2019.05.001?ref=aligrithm.com) - [Reconciling Modern Machine-Learning Practice and the Classical Bias-Variance Trade-off (Belkin, Hsu, Ma, and Mandal, 2019)](https://doi.org/10.1073/pnas.1903070116?ref=aligrithm.com) - [Regression Shrinkage and Selection via the Lasso (Tibshirani, 1996)](https://www.jstor.org/stable/2346178?ref=aligrithm.com) - [Ridge Regression: Biased Estimation for Nonorthogonal Problems (Hoerl and Kennard, 1970)](https://doi.org/10.1080/00401706.1970.10488634?ref=aligrithm.com) - [Machine Learning in Factor Investing, DDA3600: Factor Investing (Chuan Shi, 2025)](https://papers.ssrn.com/sol3/papers.cfm?abstract%5Fid=5301759&ref=aligrithm.com) --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ### 10.3 The GRS Test: Your Alpha Is Probably a Missing Factor URL: https://aligrithm.com/the-grs-test-your-alpha-is-probably-a-missing-factor/ Last updated: 2026-07-10T10:57:32.000Z The old article ["Portfolio Sorts From Scratch: Deciles, Monotonicity, and the Long-Short Spread"](https://aligrithm.com/portfolio-sorts-from-scratch-deciles-monotonicity-and-the-long-short-spread/) left you with one spread portfolio and one t-stat on its average return. That is fine when you have a single factor to judge. But the real question a factor model faces is not "does this one spread pay," it is "does my model price everything at once." You hand the model a whole panel of test portfolios, twenty-five size-and-value cells, ten momentum deciles, whatever, and you ask whether it leaves any of them with a return it cannot explain. Every leftover is an alpha, and a model that prices the cross-section should leave all of them at zero together. Alpha here is not a bragging number. It is the pricing error, the vertical gap between what a portfolio actually earned and what the model says it should have earned given its exposures. A significant alpha means one of three things: the asset is genuinely mispriced, you got unlucky with a small sample, or, most of the time, your model is missing a factor that the asset happens to load on. The Gibbons-Ross-Shanken test is the one number that grades all the pricing errors jointly, and it usually delivers the same verdict: your alpha is a factor you forgot to include. ## Alpha is the gap, not the glory Start with the workhorse. For each test asset you run a time-series regression of its excess return on the factor returns, month by month: $$ R\_{it}^e = \\alpha\_i + \\beta\_i^\\top \\lambda\_t + e\_{it}, \\qquad t = 1, 2, \\cdots, T $$ Read it left to right. The excess return of asset i in month t equals a constant alpha, plus the asset's exposures beta-transpose times the factor returns that month, plus a shock. Run OLS and you get two things per asset: beta-i, how much of each factor it carries, and alpha-i, the part of its average return the factors never touched. Average over time and the model's prediction for the asset is beta-transpose times the average factor premium, so alpha is whatever sits above or below that prediction. On a plot of average return against beta, the model is a straight line and alpha is the height of the asset above the line. ![Scatter of average excess return against beta with the factor model drawn as a straight line, and alpha-i marked as the vertical distance from an asset up to the line](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_397-alpha_pricing_error.png) The dots that sit off the line are the model's failures. Work a number. Say a value portfolio averaged 1.2% a month, its market beta is 1.5, and the market premium averaged 0.6% a month. The single-factor model predicts 1.5 times 0.6, which is 0.9%. The portfolio earned 1.2%, so its alpha is 0.3% a month of return the market factor never accounted for. That 0.3% is either a genuine anomaly, noise, or a signal that a value factor belongs in the model. The regression alone will not tell you which, and that ambiguity is the whole game. ## One test per asset lies, so test them jointly You could eyeball each alpha and its own t-stat, but that road leads straight into the multiple-testing trap the old article ["Portfolio Sorts From Scratch: Deciles, Monotonicity, and the Long-Short Spread"](https://aligrithm.com/portfolio-sorts-from-scratch-deciles-monotonicity-and-the-long-short-spread/) flagged. Run twenty-five separate t-tests at the 5% level and you expect one or two to print significant by chance even if every true alpha is zero. The fix is to test the entire vector of alphas in one shot. Stack them into alpha equals the column of alpha-1 through alpha-N, and ask whether that whole vector is jointly zero. The natural statistic is a quadratic form: the alpha vector, squeezed through the inverse of its own covariance matrix, back against itself. $$ \\hat{\\alpha}^\\top \\, \\operatorname{cov}(\\hat{\\alpha})^{-1} \\, \\hat{\\alpha} \\; \\sim \\; \\chi^2\_N $$ The middle term weights each pricing error by how precisely it was estimated and how the errors move together, so a big alpha on a noisy, correlated portfolio counts for less than a modest alpha on a clean, independent one. Under the null that all alphas are zero, this quadratic form follows a chi-squared distribution with N degrees of freedom, one for each test asset. Large value, small p-value, reject the model. That is the asymptotic version, and it leans on having a long sample T for the central limit theorem to bite. ## The GRS F-statistic Gibbons, Ross, and Shanken took one more step in 1989\. Instead of the large-sample chi-squared approximation, they derived the exact small-sample distribution of the scaled pricing errors, which turns out to be an F: $$ \\frac{T-N-K}{N}\\left(1 + \\hat{\\theta}^2\\right)^{-1}\\,\\hat{\\alpha}^\\top \\hat{\\Sigma}^{-1}\\hat{\\alpha} \\;\\sim\\; F\_{N,\\,T-N-K} $$ Take it apart. Alpha-transpose Sigma-inverse alpha is the size of the pricing errors, scaled by the residual covariance Sigma, so correlated residuals do not get double-counted. The term one over one-plus-theta-squared is a correction for how good the factors themselves are, where theta is the factors' own Sharpe ratio, and more on that in a second. The fraction (T minus N minus K) over N converts the whole thing into an F statistic with N and T-minus-N-minus-K degrees of freedom, where K is the number of factors. The F distribution is exact for finite T, which matters because asset-pricing samples are short. Twenty years of monthly data is only 240 observations, and the chi-squared approximation is optimistic at that length. The verdict is a single p-value on the null that the model prices everything. The China data in the deck shows both outcomes cleanly. Test the two Fama-French factors as assets against the Chinese three-factor model and the GRS F is 0.88 with a p-value of 0.41, so you cannot reject, the Chinese model prices the imported factors. Reverse it, test the Chinese size and value factors against Fama-French, and the GRS F is 33.90 with a p-value of 2.14 times ten to the minus thirteen. That is not a close call. Fama-French leaves the Chinese factors with alphas so large the model is rejected into the ground. ## What GRS is really measuring: a Sharpe-ratio gap The F formula looks like bookkeeping until you rewrite it. The same statistic equals a comparison of two Sharpe ratios: $$ \\frac{T-N-K}{N}\\left(\\frac{1+\\hat{\\theta}\_{N+K}^2}{1+\\hat{\\theta}\_K^2} - 1\\right) \\;\\sim\\; F\_{N,\\,T-N-K} $$ Here theta-K squared is the maximum squared Sharpe ratio you can build from the K factors alone, and theta-(N+K) squared is the maximum squared Sharpe ratio from the factors plus all N test assets. GRS is asking one question: do the test assets let you build a materially better tangency portfolio than the factors already give you? If the factors span everything worth spanning, adding the test assets barely nudges the Sharpe ratio, the ratio sits near one, the statistic is small, and the model survives. If the test assets open up a much steeper Sharpe frontier, the factors are missing something, the alphas are real, and GRS rejects. ![Two minimum-variance frontiers on a return-versus-risk plot, the outer one spanned by factors plus test assets reaching a higher tangency Sharpe ratio than the inner one spanned by the factors alone](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_397-grs_sharpe_frontier.png) The picture is the argument. The inner frontier is what the factors can reach, the outer frontier is what factors plus assets can reach, and the gap between the two tangency lines is exactly what GRS measures. This is why the test is really a factor-completeness check. The deck reports the maximum annual Sharpe ratio spanned by three competing models: a composite four-factor model reaches 2.02, Stambaugh-Yuan reaches 0.98, and Daniel-Hirshleifer-Sun reaches 0.80\. A model whose factors already span a Sharpe of 2.02 leaves little room for test assets to improve on it, so it will reject far fewer of them. The higher your factors' own Sharpe ratio, the harder it is for any anomaly to show a significant alpha against you. ## Your alpha is probably a missing factor Now the title earns itself. A big alpha against your model is rarely free money. Feed one model's factors in as the other model's test assets and the pattern is stark. Against the Chinese three-factor model, the Fama-French value factor carries an alpha of 0.34 with a t-stat of 0.97, statistically indistinguishable from zero, and the Fama-French size factor an alpha of minus 0.04 with a t-stat of minus 0.66\. The Chinese model spans them; they add nothing it does not already have. Run it the other way and the Chinese value-minus-growth factor shows an alpha of 1.39 with a t-stat of 7.93 against Fama-French, and the Chinese size factor an alpha of 0.47 with a t-stat of 7.03\. Those enormous alphas are not evidence that China prints free returns. They are evidence that Fama-French is missing the factor the Chinese model built. That is the joint-hypothesis problem from the old article ["What a Factor Actually Is: α + βλ, and Why the Market Is Factor #1,"](https://aligrithm.com/what-a-factor-actually-is-a-bl-and-why-the-market-is-factor-1/) made operational. A nonzero alpha says the market is inefficient or your model is wrong, and the test cannot separate the two. GRS does not break that tie. It quantifies how badly your current model fails, which almost always gets fixed by adding a factor rather than declaring the market broken. The lazy academic move is to keep bolting on factors until every alpha vanishes in-sample, which is curve-fitting with a respectable vocabulary, and it manufactured a large share of the factor zoo that a later article tears apart. ## The test-asset trap GRS has a soft underbelly, and the deck names it. The entire verdict depends on which test assets you feed it. Swap the test portfolios and the same model can pass one gauntlet and fail another, because you are only ever asking whether the model prices these particular assets. Nothing stops a researcher from quietly choosing the test assets that flatter their own model, which is data snooping wearing a lab coat. Worse, adding a factor to a model can sometimes raise the pricing errors on a given set of test assets rather than lower them, so "more factors" is not monotonically "better model." There is one honest use of GRS for model comparison: count how many test assets still show a significant alpha against each model. Fewer survivors means the model prices more of the cross-section. ![Grouped bar chart of the number of anomalies that still show a significant alpha against three factor models, at t-stat thresholds of 2.0 and 3.0, with the composite four-factor model leaving the fewest](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_397-anomaly_counts.png) The composite four-factor model leaves 18 anomalies alive at a t-stat above 2.0 and only 1 above 3.0, against 28 and 7 for Stambaugh-Yuan and 23 and 5 for Daniel-Hirshleifer-Sun. By this scoreboard the composite model wins. But read the gap between the two thresholds. Raising the bar from a t of 2.0 to a t of 3.0 collapses the survivor count for every model, which is the whole multiple-testing reckoning in one picture. Most of those t-between-2-and-3 anomalies are noise dressed as discovery, and the honest threshold in a field that tested hundreds of signals is closer to 3 than 2\. That fight gets its own article. And every number here is in-sample; none of it promises the model prices anything out of sample, which is where factor claims go to die. ## Where this connects GRS is the joint version of the single t-stat from the old article ["Portfolio Sorts From Scratch: Deciles, Monotonicity, and the Long-Short Spread."](https://aligrithm.com/portfolio-sorts-from-scratch-deciles-monotonicity-and-the-long-short-spread/) One sort gives you one premium and one t-test; a whole panel of sorts gives you a vector of alphas and needs GRS. The next step, the two-pass Fama-MacBeth regression, attacks the same panel from the other direction, estimating premiums cross-section by cross-section instead of asset by asset, and it comes with the Shanken correction almost nobody applies. It also lands on the same point as the old article ["Why Benchmarks Matter in Rule Evaluation."](https://aligrithm.com/why-benchmarks-matter-in-rule-evaluation/) A rule's return means nothing until you net it against the right benchmark, and alpha is exactly that: return minus what the model already explains. GRS just does the netting for a whole book of portfolios at once and asks whether anything survives. Pick a weak benchmark and everything looks like alpha; pick the right factor model and most of your edge turns out to be exposure you already owned. ## KEY POINTS - Alpha is the pricing error, the gap between what a test asset earned and what the model predicts from its exposures. A significant alpha usually means the model is missing a factor, not that you found free money. - Testing alphas one asset at a time invites false positives across many portfolios. GRS tests the whole vector of alphas jointly, so a model that prices the cross-section leaves every pricing error at zero together. - The joint quadratic form alpha-transpose times the inverse covariance times alpha follows a chi-squared with N degrees of freedom asymptotically. Gibbons-Ross-Shanken sharpened this into an exact small-sample F with N and T-minus-N-minus-K degrees of freedom, which matters because asset-pricing samples are short. - GRS is really a Sharpe-ratio comparison: it asks whether the test assets build a materially higher maximum Sharpe ratio than the factors alone. A model whose factors already span a high Sharpe (the composite model reaches 2.02 annual) rejects far fewer anomalies. - Worked verdict from the China data: the Chinese three-factor model prices the Fama-French factors (GRS F 0.88, p 0.41), but Fama-French fails badly on the Chinese factors (GRS F 33.90, p about ten to the minus thirteen; value-minus-growth alpha 1.39, t 7.93). The huge alpha is a missing factor. - The verdict depends entirely on the choice of test assets, which opens the door to data snooping, and adding a factor can even raise pricing errors on a given test set. More factors is not automatically a better model. - Counting surviving anomalies is a fair model-comparison scoreboard, but raising the bar from t above 2 to t above 3 collapses every model's survivor count, and it is all in-sample. The out-of-sample test is where factor claims usually break. ## References - [A Test of the Efficiency of a Given Portfolio (Gibbons, Ross, and Shanken, 1989)](https://www.jstor.org/stable/1913625?ref=aligrithm.com) - [Common Risk Factors in the Returns on Stocks and Bonds (Fama and French, 1993)](https://www.sciencedirect.com/science/article/abs/pii/0304405X93900235?ref=aligrithm.com) - [A Five-Factor Asset Pricing Model (Fama and French, 2015)](https://www.sciencedirect.com/science/article/abs/pii/S0304405X14002323?ref=aligrithm.com) - [Digesting Anomalies: An Investment Approach (Hou, Xue, and Zhang, 2015)](https://academic.oup.com/rfs/article-abstract/28/3/650/1574802?ref=aligrithm.com) - [Size and Value in China (Liu, Stambaugh, and Yuan, 2019)](https://www.sciencedirect.com/science/article/pii/S0304405X19300625?ref=aligrithm.com) - [... and the Cross-Section of Expected Returns (Harvey, Liu, and Zhu, 2016)](https://academic.oup.com/rfs/article-abstract/29/1/5/1843824?ref=aligrithm.com) - [Regression-Based Test, DDA3600: Factor Investing (Chuan Shi, 2024)](https://papers.ssrn.com/sol3/papers.cfm?abstract%5Fid=5255118&ref=aligrithm.com) --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ### 10.4 Fama-MacBeth Two-Pass and the Shanken Correction Nobody Applies URL: https://aligrithm.com/fama-macbeth-two-pass-and-the-shanken-correction-nobody-applies/ Last updated: 2026-07-10T10:57:33.000Z The old article ["The GRS Test: Your Alpha Is Probably a Missing Factor"](https://aligrithm.ghost.io/the-grs-test-your-alpha-is-probably-a-missing-factor/?ref=aligrithm.com) graded a model by running a time-series regression per asset and asking whether the alphas were jointly zero. That machine only works when your factors are tradable portfolios you can put on the right-hand side. Half the interesting factors are not. Industrial production, inflation surprises, a liquidity shock, an aggregate sentiment index: you cannot buy a share of GDP, so you cannot form its factor-mimicking portfolio and you cannot run the time-series test. You also often want a different thing from the test. GRS hands you a pass or fail on the whole model. Fama-MacBeth hands you the premium itself, a number with a t-stat, for each factor, tradable or not. Fama-MacBeth is the most-run regression in empirical asset pricing. It is also the one whose standard errors are almost always reported wrong. The betas that go into the second pass were estimated in the first pass, and pretending they are known truth shrinks your standard errors and inflates your t-stats. Shanken wrote down the fix in 1992\. Most practitioners still skip it, which is how a t of 1.99 gets published as a t of 2.3. ## Two passes: betas first, premia second Both cross-sectional methods share a first pass. Regress each asset's excess return on the factor returns over time to get its exposures, exactly as in the old article ["The GRS Test: Your Alpha Is Probably a Missing Factor."](https://aligrithm.com/the-grs-test-your-alpha-is-probably-a-missing-factor/) That gives you beta-hat for every asset. The second pass is where the cross-sectional approach diverges: instead of reading the premium off a tradable factor's average return, you regress average returns across assets on their estimated betas. $$ \\mathbb{E}\_T\[R\_i^e\] = \\hat{\\beta}\_i^\\top \\lambda + \\alpha\_i, \\qquad \\hat{\\lambda} = (\\hat{\\beta}^\\top \\hat{\\beta})^{-1}\\hat{\\beta}^\\top \\, \\mathbb{E}\_T\[R^e\] $$ Read it plainly. The left equation says each asset's long-run average excess return should equal its exposures times the factor premiums, and whatever is left over is the pricing error alpha-i. The right equation is just the OLS slope of that cross-sectional line: regress the vector of average returns on the matrix of betas, and the slope is lambda-hat, the estimated premium per unit of exposure. The line is fit to minimize the sum of squared pricing errors across assets. ![Scatter of average excess return against beta across assets with a fitted cross-sectional line, and alpha-i marked as the vertical distance of one asset from the line](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_398-cross_sectional_second_pass.png) Work a number. Suppose you have five test portfolios and, after the first pass, their market betas are 0.6, 0.8, 1.0, 1.2, 1.4, and their average monthly excess returns line up roughly on a line with slope 0.5% and a small scatter. The cross-sectional regression returns lambda-hat near 0.5% per unit of beta, so a portfolio with beta 1.2 is predicted to earn 0.6% a month. If it actually averaged 0.75%, its alpha is 0.15%, its distance above the line. The premium is the slope, the pricing errors are the residuals, and both fall out of one regression. ## Fama-MacBeth: estimate then average The plain cross-sectional regression runs once on the time-averaged returns. Fama-MacBeth flips the order. It runs the cross-sectional regression separately in every single period, producing a fresh premium estimate each month, then averages those estimates over time. $$ \\hat{\\lambda} = \\frac{1}{T}\\sum\_{t=1}^{T}\\hat{\\lambda}\_t, \\qquad \\sigma(\\hat{\\lambda}\_k) = \\left\[\\frac{1}{T^2}\\sum\_{t=1}^{T}\\left(\\hat{\\lambda}\_{kt}-\\hat{\\lambda}\_k\\right)^2\\right\]^{1/2}, \\qquad t = \\frac{\\hat{\\lambda}\_k}{\\sigma(\\hat{\\lambda}\_k)} $$ Each month t, regress that month's returns on the betas to get lambda-t, the factor's return that month. Average the lambda-t series to get the premium, and take its standard error the usual way for a mean: the standard deviation of the monthly estimates divided by the square root of T, which is why the sum carries one over T squared rather than one over T. The t-stat is the premium over its standard error. Plain cross-section averages first then estimates; Fama-MacBeth estimates first then averages. If the betas never changed over time the two give the identical premium, but in practice betas drift, and the original paper re-estimated them on a rolling window, using the beta known at t minus 1 to explain the return at t. ![Two fitted lines on a plot of average return against beta, one from cross-sectional regression and one from time-series regression, meeting near the factor portfolio point at beta equal to one](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_398-ts_vs_cs_regression.png) The two lines are not the same line. The time-series line is pinned to run through the origin and the tradable factor portfolio, so its slope is forced to be the factor's own average return. The cross-sectional line is free to tilt and shift to minimize pricing errors across the test assets, so its slope can disagree with the factor's realized return. That freedom is the point when the factor is not tradable, and it is also the source of the problem in the next section. Fama-MacBeth's real advantage is that computing the premium period by period sidesteps the cross-sectional correlation among residuals that would wreck a single pooled regression's standard errors. Its weakness is that it assumes the monthly estimates are independent over time, which is why almost every modern paper layers a Newey-West adjustment on top to soak up autocorrelation. ## The generated-regressor problem Here is the crack everyone steps over. The second pass treats beta-hat as if it were the true beta. It is not. Beta-hat came out of a finite time-series regression in the first pass, so it carries sampling error, and now it sits on the right-hand side of the second regression as an explanatory variable. Regressing on a noisy regressor is the classic errors-in-variables problem. Two things happen. The premium estimate gets biased toward zero, because noise in a regressor flattens the slope. And, worse for anyone chasing significance, the naive standard error is too small, because it accounts for the scatter of returns around the line but ignores the scatter in the betas themselves. The result is a t-stat that looks sturdier than it is. You did two estimation steps and only paid for the uncertainty in one. Every marginal factor whose t-stat sits between 2 and 3, exactly the band where the factor zoo lives, is a candidate to have its significance evaporate once you account for the first-pass noise. ## The Shanken correction Shanken derived the adjustment. The errors-in-variables piece of the premium's variance gets multiplied by a correction factor that depends on how good the factors are: $$ \\operatorname{cov}(\\hat{\\lambda}) \\;=\\; \\frac{1}{T}\\left\[\\left(1 + \\hat{\\lambda}^\\top \\hat{\\Sigma}\_f^{-1}\\hat{\\lambda}\\right)\\left(\\hat{\\beta}^\\top\\hat{\\beta}\\right)^{-1}\\hat{\\beta}^\\top \\Sigma \\,\\hat{\\beta}\\left(\\hat{\\beta}^\\top\\hat{\\beta}\\right)^{-1} + \\hat{\\Sigma}\_f\\right\] $$ The term that matters is the multiplier one plus lambda-transpose Sigma-f-inverse lambda. That quantity, lambda-transpose Sigma-f-inverse lambda, is the maximum squared Sharpe ratio you can build from the factors per period. It is the same theta-squared that appears inside the GRS F-statistic from the old article ["The GRS Test: Your Alpha Is Probably a Missing Factor."](https://aligrithm.com/the-grs-test-your-alpha-is-probably-a-missing-factor/) So the correction inflates the standard errors by the square root of one plus the factors' own squared Sharpe ratio. Good factors punish you: the more return-per-risk your factors already deliver, the more the correction widens your error bars on the premia estimated from them. Put numbers on it. A plain three-factor model with an annual maximum Sharpe around 0.77 has a monthly squared Sharpe near 0.05, so the correction multiplies the variance by 1.05 and inflates the standard error by about 2.4%. Negligible. But take the composite four-factor model from the old article's China evidence, with an annual maximum Sharpe of 2.02\. Its monthly squared Sharpe is about 0.34, the multiplier is 1.34, and the standard error inflates by about 16%. A premium that printed a t-stat of 2.3 with naive errors drops to roughly 1.99 once corrected, straight through the significance line. ![Curve showing standard-error inflation from the Shanken correction rising with the factors' annual Sharpe ratio, marked at 2.4 percent for a Sharpe of 0.77 and 15.8 percent for a Sharpe of 2.02](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_398-shanken_inflation.png) The curve is the whole argument for why this is not pedantry. For weak factors the correction rounds to nothing, which is why sloppy work survived for a while. For the high-Sharpe models that quant desks actually chase, it moves t-stats by enough to overturn a marginal discovery. Nobody applies it because the default output of every regression package is the uncorrected Fama-MacBeth standard error, and adding the multiplier takes an extra line of code plus an estimate of the factor covariance. The path of least resistance publishes the bigger t-stat. ## Noisy betas also bias the premium, not just the error bar The correction fixes the standard error, but the attenuation bias in the premium itself is a separate leak worth naming. Because beta-hat is measured with error, the estimated premium is dragged toward zero and the intercept is pushed up, so a real premium can look smaller than it is even as its significance looks larger than it is. Sorting stocks into portfolios before the second pass, the workhorse move from the old article ["Portfolio Sorts From Scratch: Deciles, Monotonicity, and the Long-Short Spread,"](https://aligrithm.com/portfolio-sorts-from-scratch-deciles-monotonicity-and-the-long-short-spread/) is partly a defense against this: a portfolio's beta is averaged over many stocks, so it is estimated far more precisely than any single stock's beta, and the errors-in-variables damage shrinks. The cost is that you throw away cross-sectional spread by collapsing thousands of stocks into ten or twenty-five buckets. Precise betas or a wide cross-section, and you rarely get both. ## Where this connects Fama-MacBeth is the cross-sectional twin of the time-series GRS test from the old article ["The GRS Test: Your Alpha Is Probably a Missing Factor."](https://aligrithm.com/the-grs-test-your-alpha-is-probably-a-missing-factor/) GRS forces the model through tradable factor portfolios and tests the alphas jointly; Fama-MacBeth estimates the premium directly, tradable factor or not, and tests it with a t-stat. The pricing error alpha-i is the same object in both, and it is the same object as the residual return in the old article ["Predict Residual Returns, Not Gross."](https://aligrithm.com/predict-residual-returns-not-gross/) That article argued you should forecast the idiosyncratic leftover after stripping the factors; the Fama-MacBeth second pass is how you measure what each unit of exposure earned so you know exactly what to strip. Estimate the premia, subtract beta times premium, and forecast only what survives. The Shanken correction is one more reason the honest significance bar sits higher than the reflexive t of 2\. Between generated-regressor noise, the correction most people skip, and the multiple-testing tax across hundreds of candidate factors, a premium needs a lot more than a naive t of 2 to be real. That reckoning gets its own article on the factor zoo. ## KEY POINTS - Fama-MacBeth is a two-pass estimator: first estimate each asset's betas from time-series regressions, then regress average returns on those betas across assets to get the factor premium. It works even when the factor is not a tradable portfolio, which the time-series GRS test cannot handle. - The second pass fits a cross-sectional line whose slope is the premium lambda and whose residuals are the pricing errors alpha. The line minimizes squared pricing errors across the test assets. - Fama-MacBeth runs the cross-sectional regression every period and averages the resulting premium estimates, so its standard error is the standard deviation of the monthly estimates over the square root of T. This sidesteps cross-sectional residual correlation but assumes no time-series autocorrelation, which Newey-West patches. - The betas fed into the second pass were estimated in the first pass, so they carry noise. This errors-in-variables problem biases the premium toward zero and makes the naive standard error too small, inflating t-stats. - The Shanken correction multiplies the errors-in-variables part of the premium variance by one plus the factors' own maximum squared Sharpe ratio, the same theta-squared that appears in the GRS F. Standard errors inflate by the square root of that factor. - The correction is tiny for weak factors (about 2% for an annual Sharpe of 0.77) but large for high-Sharpe models (about 16% for the composite four-factor model's Sharpe of 2.02, enough to drop a t of 2.3 to 1.99). Nobody applies it because uncorrected errors are the default output. - Sorting into portfolios before the second pass averages betas over many stocks, cutting the errors-in-variables damage, at the cost of collapsing cross-sectional spread. ## References - [On the Estimation of Beta-Pricing Models (Shanken, 1992)](https://doi.org/10.1093/rfs/5.1.1?ref=aligrithm.com) - [Risk, Return, and Equilibrium: Empirical Tests (Fama and MacBeth, 1973)](https://www.journals.uchicago.edu/doi/10.1086/260061?ref=aligrithm.com) - [A Test of the Efficiency of a Given Portfolio (Gibbons, Ross, and Shanken, 1989)](https://www.jstor.org/stable/1913625?ref=aligrithm.com) - [Common Risk Factors in the Returns on Stocks and Bonds (Fama and French, 1993)](https://www.sciencedirect.com/science/article/abs/pii/0304405X93900235?ref=aligrithm.com) - [A Simple, Positive Semi-Definite, Heteroskedasticity and Autocorrelation Consistent Covariance Matrix (Newey and West, 1987)](https://doi.org/10.2307/1913610?ref=aligrithm.com) - [Size and Value in China (Liu, Stambaugh, and Yuan, 2019)](https://www.sciencedirect.com/science/article/pii/S0304405X19300625?ref=aligrithm.com) - [Regression-Based Test, DDA3600: Factor Investing (Chuan Shi, 2024)](https://papers.ssrn.com/sol3/papers.cfm?abstract%5Fid=5255118&ref=aligrithm.com) --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ### 10.2 Portfolio Sorts From Scratch: Deciles, Monotonicity, and the Long-Short Spread URL: https://aligrithm.com/portfolio-sorts-from-scratch-deciles-monotonicity-and-the-long-short-spread/ Last updated: 2026-07-10T10:57:34.000Z The old article "What a Factor Actually Is" ended on the equation expected excess return equals alpha plus beta-transpose lambda, and left a hole. Lambda is the factor premium, the number you actually get paid. But you never observe a stock's factor exposure, and you never observe lambda directly either. Both are hidden. So how does anyone put a number on the value premium or the size premium? They sort. Portfolio sorts are the workhorse that turns a characteristic you can measure into a factor return you can trade, and they are the first thing any factor claim has to survive. Get the sort wrong and every t-stat downstream is noise. The trick is to stop pretending you can measure exposure. You can't read a stock's true loading on the value factor off a screen. What you can read is its book-to-market ratio, and the sort assumes only that book-to-market is correlated with the value loading. Not equal to it, not a linear function of it, just correlated in the right direction. That weak assumption is the whole method, and it is why sorting is more honest than regressing returns on a characteristic and pretending the coefficient is a premium. ## Rank on a proxy, not a measurement In a portfolio sort the variable you rank on is the sorting variable, and book-to-market is the classic one. The method uses that variable as a stand-in for exposure to the factor. High book-to-market means "probably high value loading," low book-to-market means "probably low value loading," and that is all it claims. No functional form, no assumption that a book-to-market of 2.0 means twice the loading of 1.0\. The only requirement is that the ranking lines up with the exposure often enough. This is the same instinct as the old article ["Ranking Beats Forecasting for Many Trading Problems."](https://aligrithm.com/ranking-beats-forecasting-for-many-trading-problems/) You do not need to predict the magnitude of a stock's return to trade the cross-section, you only need to get the order roughly right, because expected returns are tiny next to their own standard error and any point forecast of them is mostly noise. Sorting throws the magnitude away on purpose and keeps the order. A monotone error in your proxy leaves the ranking intact, so the sort survives a badly calibrated signal in a way a return regression does not. The old article ["Cross-Sectional Percentile Rank Within a Universe"](https://aligrithm.com/cross-sectional-percentile-rank-within-a-universe/) ran the same play from the indicator side, ranking one market's normalized trend against its peers. A Fama-French decile sort is that idea industrialized across a few thousand stocks. ## The three-step recipe The mechanics are boring, which is the point. Boring means fewer places to cheat. First, sorting. Fix a stock pool and rank every name in it by the sorting variable, high to low or low to high depending on whether the variable is positively or negatively related to return. Second, grouping. Cut the ranked list into L groups, and by convention L is 10, so you get deciles. Build a hedged portfolio by going long the top decile and shorting the bottom decile. That long-short book is the spread portfolio, and its return is your estimate of the factor return for that month. The two legs are dollar neutral, equal money long and short, and within each leg you weight stocks either equally or by market capitalization. Third, rebalance. A stock's book-to-market drifts over time, so you redo the sort on a schedule. Academics rebalance monthly or annually. Trading desks often rebalance a factor book daily, which sounds more diligent until you price in the turnover and the spread you pay on every name that crosses a decile boundary. Notice what the spread portfolio quietly does. Going long the top and short the bottom nets out the exposures both groups share. If the whole market rallies, both legs rise and the move cancels. What is left is the return that specifically separates high book-to-market stocks from low ones, which is exactly the factor return you wanted and nothing else. That is the same neutralizing logic behind any ranked long-short book: the market move you cannot forecast gets subtracted by construction. ## Is the spread real, or is it two lucky months You now have a time series of monthly spread returns, one lambda per month. Testing whether the factor earns a premium is a t-test on that series, and it is the least glamorous, most load-bearing step in the whole pillar. $$ \\hat{\\lambda} = \\frac{1}{T}\\sum\_{t=1}^{T}\\lambda\_t, \\qquad \\text{se}(\\hat{\\lambda}) = \\frac{\\operatorname{std}(\\lambda\_t)}{\\sqrt{T}}, \\qquad t = \\frac{\\hat{\\lambda}}{\\text{se}(\\hat{\\lambda})} $$ Read it left to right. Lambda-hat is the average of your monthly spread returns, the estimated premium. The standard error is the sample standard deviation of those monthly returns divided by the square root of the number of months, which is just the standard error of a mean. The t-statistic is the premium divided by its standard error, and it follows a t-distribution with T minus 1 degrees of freedom. The minus one is there because you burned a degree of freedom estimating the standard deviation from the same sample. Work the paper's own numbers. Sort Chinese stocks into ten book-to-market deciles, value-weight each decile, rebalance monthly. The high-minus-low spread averages 0.88% a month with a standard error of 0.47%. The t-stat is 0.88 divided by 0.47, which is 1.85, and the p-value is 0.07\. That clears the 10% significance bar and nothing tighter. A 0.88% monthly premium sounds like a fortune, over 10% a year before costs, and the t-stat of 1.85 says you should hold your applause. It would not survive the multiple-testing tax that a later article covers, where the honest threshold for a factor dredged out of hundreds of candidates sits closer to a t of 3 than a t of 2\. One sort, one barely-significant premium, is where the factor zoo's graveyard starts filling. ## Monotonicity: the spread can lie A positive high-minus-low spread is necessary and nowhere near sufficient. The number 0.88% is a difference between two endpoints, and two endpoints can look great while everything between them is garbage. Suppose deciles 1 through 9 are a flat, random mess and only the top decile spikes on a couple of lucky momentum names. The spread is still positive, the t-stat still prints, and you have a portfolio built on two data points pretending to be a monotone risk premium. A real factor should explain the cross-section smoothly, so the ten decile returns should rise more or less in order as the sorting variable rises. The check is the rank correlation coefficient between the decile group labels and the decile returns. $$ \\rho\_s = \\frac{\\operatorname{cov}(X\_r, X\_g)}{\\sigma\_{X\_r}\\,\\sigma\_{X\_g}} $$ X\_g is the group number, 1 through 10, and X\_r is the rank of that group's average return, also 1 through 10\. Rho-s is the ordinary correlation computed on those two sets of ranks, so it measures the monotonic relationship and ignores how big the gaps are. It runs from minus 1 to plus 1\. A rho-s of plus 1 means the returns increase in perfect lockstep with the deciles, every step up in book-to-market buys a step up in return. Near zero means the ordering is noise and your spread is an accident at the ends. Run it on the paper's deciles. The average returns from Low to High are 0.57, 0.53, 0.57, 0.72, 0.94, 0.90, 1.08, 1.32, 1.21, 1.44\. Not a perfect staircase, decile 2 dips below Low and decile 6 slips under decile 5 and decile 9 under decile 8, but the trend up the ranking is unmistakable. Rank the returns, correlate against 1 through 10, and rho-s lands around 0.95\. High, not perfect, which is what a genuine but noisy factor looks like. ![Bar chart of the ten book-to-market decile returns rising from Low to High, with the 0.88 percent high-minus-low spread shown as a separate red bar](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_396-decile_sort_returns.png) The picture is the tell. A clean upward climb is what separates a factor from a fluke. If the bars looked like a flat field with one tall spike on the right, the spread would be the same 0.88% and the rank correlation would collapse, and you would know the premium is riding on a handful of names rather than a systematic pattern across the whole cross-section. ## One sort controls nothing Univariate sorting has a hole you can drive a truck through. Ranking on book-to-market alone gives you a portfolio that is long high book-to-market and short low book-to-market, but it does not check what else those two groups differ on. If high book-to-market stocks happen to be small caps and low book-to-market stocks happen to be large caps, your "value" spread is quietly also a long-small, short-large size bet. You wanted one factor and you built a blend of two, and you cannot tell how much of the 0.88% is value and how much is size leaking in. The fix is to sort on two variables at once, double sorting. Cut the universe into L1 groups on the first variable and L2 groups on the second, giving L1 times L2 cells, and quintiles on each side are the common choice, so 25 portfolios. From that grid you build a factor for one variable by averaging across the other. For the first variable the return is the average of the top-X1 cells minus the average of the bottom-X1 cells: $$ \\lambda\_{X\_1,t} = \\frac{1}{L\_2}\\sum\_{i=1}^{L\_2} R\_{L\_1 i,\\,t} - \\frac{1}{L\_2}\\sum\_{i=1}^{L\_2} R\_{1 i,\\,t} $$ The first sum runs across the whole top row of the grid, the high-X1 cells at every level of X2, and averages them. The second sum does the same for the bottom row, the low-X1 cells. Subtract, and X2 is averaged out of the picture, so the leftover is the X1 premium purged of X2\. Concretely, with value as X1 and size as X2, you average value's spread across small, mid, and large stocks separately, so the size tilt cancels and you are left with a cleaner value premium. ## Independent versus dependent, and why the cells go empty Double sorting splits into two flavors, and the difference decides whether your grid is usable. Independent double sorting, also called unconditional, cuts each variable into quintiles using its own global breakpoints, then drops every stock into whatever cell its two ranks land in. Simple, symmetric, and it breaks the moment the two variables correlate. If book-to-market and size move together in the cross-section, a high book-to-market stock is usually also small. Under independent breakpoints the stocks pile onto the diagonal of the grid, the high-high and low-low cells overflow while the off-diagonal cells, high on one variable and low on the other, run nearly empty. A cell holding six stocks is not a portfolio, it is a rounding error, and one outlier in it swings the whole factor return. ![Two 5 by 5 grids of stock counts, the independent sort with counts piled on the diagonal and near-empty corners, the dependent sort with a flat 200 stocks in every cell](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_396-double_sort_counts.png) The left grid is the disease. Two correlated characteristics, and the corner cells hold hundreds of stocks while the opposite corners hold single digits. The right grid is the cure. Dependent double sorting, also called conditional, sorts on the first variable into L1 groups, then sorts on the second variable within each of those groups. Because the second cut happens inside each first-variable bucket, every cell gets the same count by construction, no empty corners, no outlier hostage situation. Dependent sorting also answers a sharper question. It treats the two variables unequally on purpose. The first variable is a pure control, and the real question is whether the second variable still predicts returns after the first is held fixed. Does momentum add anything once you have already conditioned on size, or is it just size in disguise? A factor should be built only on the second, conditioned variable, because that is the one whose incremental information you just isolated. Independent sorting treats both variables as co-equal factors, dependent sorting treats the first as a nuisance you are controlling away, and knowing which you want is the difference between measuring a factor and measuring a confound. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_396-visual-0.png) ## Where this connects The sort is the machine that produces lambda, and everything else in the pillar is a check on what comes out of it. The old article "What a Factor Actually Is" gave the equation; this article gave the estimator for its most slippery term. The premium test here, the t-stat on the mean spread, is the single-factor warm-up for the GRS test, which asks the same significance question about a whole vector of alphas at once. Fama-MacBeth, coming next, is the sort's regression-based cousin, estimating premiums in a second cross-sectional pass instead of a long-short book. And the barely-significant t of 1.85 on that value premium is the thread that leads straight into the factor zoo, where the reckoning is that most published sorts print a spread that dies the moment you correct for how many characteristics people tried. Reach back too. The neutralizing logic of the spread portfolio, long the top and short the bottom so the shared exposure cancels, is the same move as the ranked long-short book from the Carver-side ranking articles and the cross-sectional percentile rank from the old article ["Cross-Sectional Percentile Rank Within a Universe."](https://aligrithm.com/cross-sectional-percentile-rank-within-a-universe/) Rank on a proxy, trade the ends, subtract what both ends share. The sort just makes that discipline exact enough to slap a t-stat on. ## KEY POINTS - You never observe a stock's factor exposure or the factor premium. A portfolio sort ranks stocks on a measurable proxy like book-to-market and assumes only that the proxy is correlated with the true exposure, no functional form required. - The recipe is three steps: rank the universe on the sorting variable, cut into L groups (L equals 10 by convention), go long the top decile and short the bottom to form a dollar-neutral spread portfolio, then rebalance. The spread's monthly return is your estimate of the factor return. - Test the premium with a t-stat on the monthly spread returns: lambda-hat is the mean, the standard error is the standard deviation over the square root of the number of months, and the t-stat is the ratio on T minus 1 degrees of freedom. - The paper's book-to-market sort earns 0.88% a month with a t-stat of only 1.85 and a p-value of 0.07\. That clears the 10% bar and nothing tighter, and it would not survive the multiple-testing correction a later article applies. - A positive high-minus-low spread can be two lucky endpoints. Check monotonicity with the rank correlation between decile number and decile return; a rho-s near 1 (about 0.95 in the example) means the returns climb smoothly with the ranking rather than spiking at one end. - Univariate sorting cannot control for other factors, so a value sort can secretly be a size bet. Double sorting cuts on two variables at once and averages one out to purge it from the other. - Independent double sorting uses global breakpoints and empties the off-diagonal cells when the two variables correlate, leaving factor returns hostage to outliers. Dependent double sorting sorts the second variable within each group of the first, keeps cell counts even, and isolates the second variable's incremental predictive power after controlling for the first. ## References - [The Cross-Section of Expected Stock Returns (Fama and French, 1992)](https://onlinelibrary.wiley.com/doi/10.1111/j.1540-6261.1992.tb04398.x?ref=aligrithm.com) - [Common Risk Factors in the Returns on Stocks and Bonds (Fama and French, 1993)](https://www.sciencedirect.com/science/article/abs/pii/0304405X93900235?ref=aligrithm.com) - [A Five-Factor Asset Pricing Model (Fama and French, 2015)](https://www.sciencedirect.com/science/article/abs/pii/S0304405X14002323?ref=aligrithm.com) - [Monotonicity in Asset Returns: New Tests with Applications to the Term Structure, the CAPM, and Portfolio Sorts (Patton and Timmermann, 2010)](https://www.sciencedirect.com/science/article/abs/pii/S0304405X10000723?ref=aligrithm.com) - [Digesting Anomalies: An Investment Approach (Hou, Xue, and Zhang, 2015)](https://academic.oup.com/rfs/article-abstract/28/3/650/1574802?ref=aligrithm.com) - [... and the Cross-Section of Expected Returns (Harvey, Liu, and Zhu, 2016)](https://academic.oup.com/rfs/article-abstract/29/1/5/1843824?ref=aligrithm.com) - [Portfolio Sort Analysis, DDA3600: Factor Investing (Chuan Shi, 2024)](https://papers.ssrn.com/sol3/papers.cfm?abstract%5Fid=5254966&ref=aligrithm.com) --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ### 10.1 What a Factor Actually Is: α + βλ, and Why the Market Is Factor #1 URL: https://aligrithm.com/what-a-factor-actually-is-a-bl-and-why-the-market-is-factor-1/ Last updated: 2026-07-10T10:57:35.000Z Pull fourteen years of returns for Alphabet and Citigroup. One compounds to roughly ten times your money, the other crawls sideways near two. The reflex is to explain each path on its own: Google had better products, Citi carried a bad balance sheet through the crisis. That reflex is the wrong axis. The question underneath factor investing is not "where does this one stock go next," it is "why do average returns differ across the whole cross-section of stocks at the same time." Answer that and you stop trading stories about single names and start trading the thing that actually separates the winners from the losers. The old article ["What Is a Factor, Really"](https://aligrithm.com/what-is-a-factor-really/) gave the practitioner's version of the answer: a factor is an alpha that explains so much return variance you keep re-finding it, so you subtract it before hunting for new edge. This article gives the equation that both the academics and the fund managers are arguing over when they say the word "factor," and shows they are staring at the same three symbols from opposite ends. ## The one equation everything hangs on Every factor model, from a 1960s single-market story to a 2020s neural net, is a claim about one line: $$ \\mathbb{E}\[R\_i^e\] = \\alpha\_i + \\beta\_i^\\top \\lambda $$ Read it left to right. The term on the left, expected excess R, is the asset's average return above the risk-free rate. Lambda is a vector of factor risk premiums, the extra return the market pays for holding each factor. Beta is that asset's vector of exposures, how much of each factor it carries. The dot product beta-transpose lambda is the return the model says the asset earns for its exposures. Whatever the equation cannot account for lands in alpha, the pricing error. ![The core cross-sectional equation, expected excess return equals alpha plus beta-transpose lambda](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_395-core_equation.png) Work a number through it. Say a stock loads 1.2 on the market, 0.4 on value, and −0.3 on momentum, and the annual premiums are 6% for the market, 3% for value, and 4% for momentum. The model return is 1.2 times 6, plus 0.4 times 3, minus 0.3 times 4, which is 7.2 plus 1.2 minus 1.2, so 7.2% a year. If the stock actually averaged 9%, the leftover 1.8% is its alpha. If it averaged 7.2%, its alpha is zero and the factors explained everything. That leftover is the whole game. A significant nonzero alpha means either your model is missing a factor, or the asset is genuinely mispriced, or you got fooled by a small sample. The equation itself does not tell you which. It just quarantines the part you cannot explain and dares you to explain it. ## The market is factor number one, and it fails on its own The first factor anyone wrote down was the market. The Capital Asset Pricing Model says an asset's expected excess return is its market beta times the market's excess return, and nothing else: $$ \\mathbb{E}\[R\_i\] - R\_f = \\beta\_i \\big( \\mathbb{E}\[R\_M\] - R\_f \\big) $$ One factor, one premium. High-beta stocks should earn more, low-beta stocks less, in a straight line through the origin. The market is factor number one for the same reason the old article ["What Is a Factor, Really"](https://aligrithm.com/what-is-a-factor-really/) strips market beta before anything else: it explains more return variance than any characteristic you will ever measure, so it contaminates every other signal you test. Remove it first because it is the biggest liar in the room about whether you found something new. The trouble is that the line does not hold. Sort real stocks by beta and average their returns, and the relationship goes flat, then bends down at the high-beta end. ![Scatter of average monthly return against beta, showing returns rise then fall as beta increases instead of climbing in a straight line](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_395-capm_fails.png) Returns climb a little from a beta of 0.7 to about 1.0, then stall, then slide as beta keeps rising. The stocks with the most market risk did not earn the most return. That single scatter killed CAPM as a complete story and forced the field to admit that expected returns depend on more than one exposure. Ross packaged the fix in Arbitrage Pricing Theory: keep the linear form, add more factors, and the market becomes the first term in a longer sum rather than the only term. Size showed up (small beats large), then value (cheap beats expensive), and the single-factor world was over. ## Two conditions, or it is not a factor A characteristic that predicts returns is not automatically a factor. The paper's definition sets a bar with two conditions, and both have to clear. First, the factor has to drive the common co-movement of asset returns, which means it has to show up in the covariance matrix. Second, it has to be priced, meaning holding exposure to it earns you a premium over time. Miss either one and you have a curiosity, not a factor. The cleanest intuition is that factors are to assets as nutrients are to food. ![Two trees showing an asset built from market, size, momentum, value, and volatility exposures, next to food built from water, carbs, protein, fats, and fiber](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_395-nutrient_analogy.png) A meal is a bundle of nutrients; a stock is a bundle of factor exposures. You do not eat "protein" directly, you eat chicken that carries protein, and you do not buy "value" directly, you buy a cheap stock that carries value loading. The analogy also flags the skeptic's job. Plenty of characteristics correlate with returns in one sample and vanish in the next, which is the equivalent of a fad ingredient that turns out to do nothing. Whether a proposed factor survives the "is it priced out of sample" test is the entire content of the factor-zoo problem, and most of the roughly three-hundred published factors do not survive it. That reckoning gets its own article; here the point is narrower. Both conditions are load-bearing, and the second one is where the graveyard fills up. ## Alpha is a test of the market, and a trap for the researcher Once you accept the equation, alpha stops being a bragging number and becomes a diagnostic. When a long-short portfolio built on some sorting variable shows a significant nonzero alpha against your factor model, that portfolio is an anomaly, and the variable is an anomaly variable. Academics care about this because a real anomaly challenges market efficiency: if the market priced everything correctly, no characteristic should leave a systematic alpha behind. Fama named the catch. You can never test market efficiency by itself, because you always test it jointly with whatever pricing model you assumed. This is the joint hypothesis problem. A nonzero alpha means the market is inefficient, or your model is wrong, and the test cannot separate the two. The side effect poisoned decades of research. If a nonzero alpha just means "add a factor," the lazy move is to keep adding factors until the alpha disappears in-sample. That is curve-fitting with a respectable vocabulary, and it is the cross-sectional cousin of the overfitting the robustness articles keep warning about. The academy's habit of chasing the model that minimizes pricing errors in-sample manufactured a large share of the factor zoo it now cannot replicate. The industry reads the same alpha differently. A fund manager does not want alpha to test a philosophy of markets, the manager wants alpha to survive transaction costs and land in the P&L. Same symbol, opposite motive. One camp uses alpha to grade a theory, the other uses it to grade a trade. ## The same equation, read from both ends Split the equation into its two moving parts and the academia-versus-industry split becomes obvious. Nobody is disagreeing about the math. They are pointing at different terms. | Term | Academia asks | Industry asks | | --------------------- | ------------------------------------------------------------- | --------------------------------------------------------------------- | | beta-transpose lambda | Which is the "best" factor model? What are the true premiums? | Which factors carry a positive premium I can allocate to? | | alpha | Is the market efficient? Is model A better than model B? | Can I earn this after costs, and how do I attribute my fund's return? | The beta-transpose lambda term is the model of the mean, the part that explains why average returns differ across assets. Academics fight over the correct list of factors and the econometrics for estimating premiums, using alpha to compare models where fewer surviving anomalies means a better model. Fund managers take the factors as given, look for the ones with durable positive premiums, and load up exposure under a diversification budget, which is exactly what the smart-beta ETF industry sells. Ranking a whole universe on a characteristic and holding the top against the bottom is the cross-sectional engine behind the old articles ["Ranking Beats Forecasting for Many Trading Problems"](https://aligrithm.com/ranking-beats-forecasting-for-many-trading-problems/) and ["Order Book Imbalance: The First Microstructure Feature to Test,"](https://aligrithm.com/order-book-imbalance-the-first-microstructure-feature-to-test/) where imbalance is treated as the first thing to rank on at the tick level. ## Model of the mean, model of covariance The equation you have been reading is the cross-sectional view, one expected return per asset. Run the same relationship through time and it becomes a regression of each asset's realized excess return on the factor realizations: $$ R\_{it}^e = \\alpha\_i + \\beta\_i^\\top \\lambda\_t + \\varepsilon\_{it} \\qquad\\Longrightarrow\\qquad \\Sigma = \\beta \\, \\Sigma\_\\lambda \\, \\beta^\\top + \\Sigma\_\\varepsilon $$ The left piece is the time-series version: this month's return equals the constant alpha, plus the exposures times this month's factor returns, plus an idiosyncratic shock. Take the covariance of both sides, use the fact that factor returns and shocks are uncorrelated, and you get the right piece. The full covariance matrix of assets, Sigma, splits into a systematic part driven by the factors, beta times the factor covariance times beta-transpose, plus a diagonal matrix of stock-specific variances. This is the model of covariance, and it is the same factors doing double duty. The payoff is practical, not decorative. Estimating a raw covariance matrix for N assets needs at least N periods of data, or the sample matrix is singular and useless. A five-factor model replaces the N-by-N estimation problem with a handful of factor covariances plus N diagonal terms, which is why risk models are built as factor models. The factors that explain the mean return are the same ones that structure the risk. That is not a coincidence the theory tolerates, it is the theory's central claim: if a characteristic prices the cross-section of returns, it should also organize how those returns move together, and if it does one without the other, it fails the definition. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_395-visual-0.png) ## Where this connects The equation expected excess return equals alpha plus beta-transpose lambda is the spine of this whole pillar. Portfolio sorts estimate the lambda term by ranking and spreading. The GRS test asks whether the whole vector of alphas is jointly zero. Fama-MacBeth estimates the premiums in a second pass. The factor zoo is the fight over how many terms belong in the sum before you are just fitting noise. Every one of those is a move on this single line. It also reaches back into the earlier pillars. Predicting the residual instead of the gross return, the thesis of the old article ["Predict Residual Returns, Not Gross,"](https://aligrithm.com/predict-residual-returns-not-gross/) is the same instinct as forecasting alpha after the factors are removed. Benchmarks, the subject of ["Why Benchmarks Matter in Rule Evaluation,"](https://aligrithm.com/why-benchmarks-matter-in-rule-evaluation/) are just the beta-transpose lambda an honest test subtracts before crediting you with skill. Name the systematic part, remove it, and pay attention only to what survives. The equation is the bookkeeping that makes that discipline exact. ## KEY POINTS - Factor investing asks a cross-sectional question, why average returns differ across assets at the same time, not a time-series question about where one stock goes next. - Everything hangs on one equation: expected excess return equals alpha plus beta-transpose lambda. Beta is exposure, lambda is the factor premium, and alpha is whatever the model cannot explain. - The market is factor number one because it explains the most return variance, so you remove it first. CAPM says returns rise in a straight line with beta, but real data bends flat then downward, which is what forced multi-factor models. - A factor has to clear two conditions: it must drive the common co-movement of returns (show up in the covariance matrix) and it must be priced. Most published factors fail the second test out of sample. - Alpha is a diagnostic, not a trophy. A significant alpha means a missing factor, a real mispricing, or a small-sample fluke, and Fama's joint hypothesis problem says you cannot cleanly separate the first two. Minimizing alpha in-sample by piling on factors is curve-fitting with a respectable name. - Academia and industry read the same equation from opposite ends: academics fight over the best model and use alpha to test efficiency, managers take the factors as given and allocate to durable premiums after costs. - The cross-sectional equation has a time-series twin whose covariance splits as Sigma equals beta Sigma-lambda beta-transpose plus a diagonal. The factors that price the mean are the same ones that structure the risk, which is why risk models are factor models. ## References - [The Arbitrage Theory of Capital Asset Pricing (Ross, 1976)](https://www.sciencedirect.com/science/article/abs/pii/0022053176900466?ref=aligrithm.com) - [Common Risk Factors in the Returns on Stocks and Bonds (Fama and French, 1993)](https://www.sciencedirect.com/science/article/abs/pii/0304405X93900235?ref=aligrithm.com) - [A Five-Factor Asset Pricing Model (Fama and French, 2015)](https://www.sciencedirect.com/science/article/abs/pii/S0304405X14002323?ref=aligrithm.com) - [Digesting Anomalies: An Investment Approach (Hou, Xue, and Zhang, 2015)](https://academic.oup.com/rfs/article-abstract/28/3/650/1574802?ref=aligrithm.com) - [A Census of the Factor Zoo (Harvey and Liu, 2019)](https://papers.ssrn.com/sol3/papers.cfm?abstract%5Fid=3341728&ref=aligrithm.com) - [Shrinking the Cross-Section (Kozak, Nagel, and Santosh, 2020)](https://www.sciencedirect.com/science/article/abs/pii/S0304405X19301655?ref=aligrithm.com) - [A Test of the Efficiency of a Given Portfolio (Gibbons, Ross, and Shanken, 1989)](https://www.jstor.org/stable/1913625?ref=aligrithm.com) - [Presidential Address: Discount Rates (Cochrane, 2011)](https://onlinelibrary.wiley.com/doi/10.1111/j.1540-6261.2011.01671.x?ref=aligrithm.com) - [Arbitrage Asymmetry and the Idiosyncratic Volatility Puzzle (Stambaugh, Yu, and Yuan, 2015)](https://onlinelibrary.wiley.com/doi/10.1111/jofi.12286?ref=aligrithm.com) --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ### 3.37 Collinearity in Parameter Sweeps: Plateaus, Not Peaks URL: https://aligrithm.com/collinearity-in-parameter-sweeps-plateaus-not-peaks/ Last updated: 2026-07-27T13:23:59.000Z You vary your parameters, watch performance hold up across the range, and conclude the system is robust. The old article ["Parameter Stability Beats Best Parameter"](https://aligrithm.com/parameter-stability-beats-best-parameter/) told you to prefer the stable region over the lucky peak, and you did. The trap is that you can run a parameter sweep that holds up beautifully and proves nothing, because the sweep never tested the parameter space at all. It tested one concept along a single line through that space, and the flatness you saw was collinearity wearing a robustness costume. ## What collinearity does to a sweep Collinearity means two or more of your parameters move together, linearly correlated, so changing one drags the other. Take the canonical moving-average crossover with a fast length of 50 and a slow length of 200\. You decide to test sensitivity, so you sweep 50/200, then 60/210, then 70/220, and the performance barely budges across all three. Robust, you think. $$ (50,\\,200),\\;(60,\\,210),\\;(70,\\,220),\\;\\dots \\;\\Rightarrow\\; \\text{slow} - \\text{fast} = 150 \\text{ (held fixed)} $$ $$ \\text{the tested points lie on a straight line in } (\\text{fast},\\,\\text{slow}) \\text{ space, correlation} \\approx 1 $$ Read that as: every point you tested sits on a single line through the two-parameter plane, because you moved fast and slow in lockstep and kept their gap pinned at 150\. The fast-to-slow relationship stayed put, so the crossover fires at the same relative speed in all three tests, and a strategy whose behavior is governed by that relationship behaves the same in all three. You did not probe the space around your parameters. You slid your one parameter set up and down a diagonal and confirmed that the diagonal is flat, which says nothing about what happens when you step off it. ## The flat sweep is a thin ridge, not a plateau The old article ["The Hill, the Spike, and the Cliff: Reading Optimization Surfaces"](https://aligrithm.com/the-hill-the-spike-and-the-cliff-reading-optimization-surfaces/) named this shape directly: the thin ridge, collinearity in disguise. On a two-parameter heatmap, real robustness shows up as an area, a wide plateau with gentle gradients where performance survives moves in any direction. Collinear robustness shows up as a bright line, high along the diagonal you swept and collapsing the moment you move perpendicular to it. Both look like a stretch of strong performance if you only sampled along the line. Only one of them survives a parameter that drifts off the line in live trading, and parameters always drift off the line, because the market does not keep your fast-to-slow gap pinned at 150 for you. So the heatmap is the right tool, as that article argued, but only if you read the topography honestly. Wide plateau and gentle slope mean the strategy tolerates being wrong about the parameters. Sharp cliff means a regime boundary is one step away. Thin ridge means you have been admiring a one-dimensional slice and calling it an area. The flat sweep along the diagonal is consistent with all three shapes, which is exactly why the flat sweep alone cannot certify anything. ## Break the collinearity on purpose The fix is to stop moving the parameters together. Fix one, vary the other on its own, then reverse and do it again. $$ \\text{fix fast} = 50,\\;\\text{vary slow} \\in \\{100,\\,200,\\,300\\} $$ $$ \\text{then fix slow} = 200,\\;\\text{vary fast} \\in \\{20,\\,50,\\,80\\} $$ Read that as: first hold the fast length still and push the slow length across a wide range on its own, which changes the fast-to-slow relationship instead of preserving it, then swap and hold the slow length still while you move the fast one. Now your tested points spread out into the plane rather than tracing a line, so you are sampling the actual neighborhood around your parameters. If performance holds up across this cross-shaped probe, you have evidence of a genuine area of stability. If it falls apart the instant the relationship between fast and slow changes, you just caught a strategy that was robust only along a line it has no way of staying on. ## Two checks past the eyeball Reading a surface by eye gets you most of the way, and two numerical checks catch what the eye misses. Run a correlation matrix on your top-performing parameter sets. If the winners are tightly correlated with each other, sitting along one diagonal, the apparent robustness is collinearity: the system found one configuration and you re-listed it under several labels. If the winners are spread across the parameter space and still perform, that spread is the real thing, robustness that does not depend on any one relationship holding. This is the same logic as the old article ["The Backtest Integrity Checklist,"](https://aligrithm.com/the-backtest-integrity-checklist/) whose perturbation item asks for stability under independent moves in each parameter, not stability along a convenient diagonal. Then take the top 10 to 15 percent of performers and plot their equity curves on top of each other. Outcome metrics like final Sharpe can agree while the paths underneath them disagree violently, one curve grinding steadily and another lurching to the same endpoint through a 40 percent drawdown. Overlaying the curves shows you consistency in path and volatility, not just in the summary number, and a cluster of genuinely robust parameters produces curves that look like siblings rather than strangers who happened to finish in the same place. ## The discipline Topography testing is the right instinct and a flat sweep is the right thing to want. The failure is reading a line as an area. Vary parameters independently before you trust a plateau, correlation-matrix the winners before you believe they are diverse, and overlay the equity curves before you believe the stability is in the path and not only in the final tally. Look for flatlands, the wide region that tolerates being wrong, not peaks and not the thin bright ridge that looks like a flatland from one angle and disappears from every other. ## Visualizing the ridge versus the plateau ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_340-visual-0.png) ## KEY POINTS - A parameter sweep can hold up beautifully and prove nothing if the parameters move together. Collinearity means two or more parameters are linearly correlated, so a sweep that varies them in lockstep tests one concept along a line, not the parameter space. - The classic case: sweeping 50/200, 60/210, 70/220 holds the fast-to-slow gap fixed, so every tested point sits on one diagonal and the crossover fires at the same relative speed throughout. The flatness you see is the diagonal being flat, not the neighborhood. - A flat collinear sweep is a thin ridge, the shape the old article ["The Hill, the Spike, and the Cliff: Reading Optimization Surfaces"](https://aligrithm.com/the-hill-the-spike-and-the-cliff-reading-optimization-surfaces/) flagged as collinearity in disguise. Real robustness is a wide plateau that survives moves in any direction; a ridge collapses the moment a parameter drifts off the line, and parameters always drift. - Break the collinearity: fix one parameter and vary the other independently across a wide range, then reverse. This spreads the tested points into the plane and probes the actual neighborhood instead of a diagonal. - Run a correlation matrix on the top performers. Tightly correlated winners strung along one diagonal are collinearity; winners spread across the space that still perform are genuine robustness. - Overlay the equity curves of the top 10 to 15 percent of performers. Summary metrics can agree while the paths disagree; a cluster of robust parameters produces curves that look like siblings, not strangers who finished in the same place. - Read topography logically. Look for flatlands that tolerate being wrong, not peaks and not the thin bright ridge that passes for a plateau from one angle and vanishes from every other. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Testing and Tuning Market Trading Systems - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Testing+and+Tuning+Market+Trading+Systems+Timothy+Masters&ref=aligrithm.com) - [Data Mining Algorithms in C++ - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Data+Mining+Algorithms+in+C%2B%2B+Timothy+Masters&ref=aligrithm.com) ### 9.32 25:1: How Five "Diversified" Election Bets Blow Up on One Shock URL: https://aligrithm.com/25-1-how-five-diversified-election-bets-blow-up-on-one-shock/ Last updated: 2026-07-27T13:26:20.000Z You hold arbitrage positions in five separate state-election markets. Five different states, five independent-looking books, and a spreadsheet that says your correlation is a comfortable 0.75\. You call it diversified and size each position like it is one of five roughly independent bets. Then a national result lands, one number that moves every state at once, and the five positions go down together in a single afternoon. The loss is not five times a normal wobble. It is twenty-five times it. The diversification you priced in was never there, and this article works out exactly where it went. The full treatment of copulas and how to fit a Student-t dependence structure lives in the article "Correlation Lies; Tail Dependence Tells the Truth: Student-t Copulas for Election Portfolios." Here the scope is one worked blowup and the single quantity that predicts it. _This post is for paying subscribers only._ ### 9.31 Prediction Markets Live on Boundaries URL: https://aligrithm.com/prediction-markets-live-on-boundaries/ Last updated: 2026-07-27T13:26:20.000Z Open a prediction-market venue and scroll the board. Most contracts are not sitting at fifty cents having an honest fight. They cluster at the edges: 3 cents, 96 cents, 1 cent, 99 cents. Will the incumbent win a safe seat, will a scheduled event happen on schedule, will a range hold. The interesting mid-book races are the minority. Prices spend their lives pressed against zero and one, because most questions about the world are close to settled most of the time. That empirical fact is the reason the elegant arbitrage math from the article "Arbitrage Is Just Projection" can hand you a wrong trade and never tell you it did. This is the numerical-analysis half of the story, narrowed to one point. The full fix, the contraction schedule, and the complete solver pipeline live in the article ["Why Your Arbitrage Solver Crashes at 99 Cents: Barrier Frank-Wolfe."](https://aligrithm.com/why-your-arbitrage-solver-crashes-at-99-cents-barrier-frank-wolfe/) Here the question is smaller and nastier: why does the theory break exactly where the prices actually are, and why does it break without a warning. ## The price scale lies about distance A move from 50 cents to 60 cents and a move from 1 cent to 11 cents are both a dime. On the price ruler they look identical. In information terms they are nothing alike, and the market maker prices in information terms, not cents. The natural scale is log-odds, the log of the probability over one minus the probability. Watch what a fixed price step does to it near the middle versus near the wall. _This post is for paying subscribers only._ ### 9.30 The Term Structure of Strategy: Momentum Far Out, Reversion Near Resolution URL: https://aligrithm.com/the-term-structure-of-strategy-momentum-far-out-reversion-near-resolution/ Last updated: 2026-07-27T13:26:19.000Z A momentum rule that printed money on an election market sixty days out will hand it all back in the final week. Same market, same rule, opposite result, and nothing about your code changed. What changed is the distance to resolution, and that distance flips which strategy pays. A prediction market is not one environment. It is a sequence of them, ordered by time-to-resolution, and the strategy that fits at the far end is the strategy that bleeds at the near end. This is the piece of the regime story that is specific to prediction markets and has no clean analog in FX or equities. The full regime treatment, including crowding and the winner-take-most concentration of arbitrage profit, lives in the article "The Term Structure of Prediction-Market Strategy and Crowding: Why the Fastest 3 Bots Take 80%." Here the scope is one axis: calendar time before the contract settles. ## Why prediction markets have a term structure at all A stock has no forced destination. A prediction-market contract does. It resolves to exactly one dollar or exactly zero on a known date, and the price is a probability crawling toward one of those two walls. That terminal boundary is what gives the market a term structure of behavior, because the character of price movement depends on how much genuine information is left to arrive before the wall. _This post is for subscribers only._ ### 9.29 Fast Fills Are Bad Fills: The Numbers URL: https://aligrithm.com/fast-fills-are-bad-fills-the-numbers/ Last updated: 2026-07-27T13:26:19.000Z You buy YES on a market order and it fills in the same block. The confirmation feels like competence. Track that fill for ten minutes and you find the mid drifted below what you paid, and the trade you were proud of started life underwater. Do that a few hundred times, average the result, and one number falls out: your fills are worth about seventy cents of the spread against you. This piece takes the single ratio behind that number, fill quality, and works it out in full, because the slogan "fast fills are bad fills" only bites once you can compute it on your own tape. The broader case for when to cross the spread and when to rest lives in the article ["Fast Fills Are Bad Fills: Make vs Take, Adverse Selection, and Why the Spread Is Posterior Variance."](https://aligrithm.com/fast-fills-are-bad-fills-make-vs-take-adverse-selection-and-why-the-spread-is-posterior-variance/) Here the scope is narrow on purpose. One metric, one table, and what the table forces you to do. ## Fill quality, defined so the sign means something Fill quality measures where you traded relative to fair value, scaled by the spread you were quoted. For a buy: $$ \\mathrm{FQ} \\;=\\; \\frac{p\_{\\text{mid}} - p\_{\\text{exec}}}{p\_{\\text{ask}} - p\_{\\text{bid}}} $$ The numerator is how far below the midpoint you executed, and the denominator is the spread. Buy under the mid and you got price improvement, so fill quality is positive. Pay above the mid and you lost ground before the position even opened, so it is negative. Scaling by the spread makes the number comparable across a thin book quoting four-cent spreads and a deep one quoting half a cent. The trap in reading this is which midpoint you use. Take the midpoint at the instant your order rests and you only measure the spread you crossed, not the information you crossed into. The honest version measures the mid a short interval after the fill, the markout mid, because that is where fair value settled once your trade and whatever moved with it cleared. The old article ["Markouts: The Truth Serum of Market Making"](https://aligrithm.com/markouts-the-truth-serum-of-market-making/) makes the general case: the spread you nominally captured is a story, and the mid a few minutes later is the fact. Fill quality is a markout wearing a normalizing denominator. ## Working one buy through Book on a binary contract: best bid 0.48, best ask 0.52, so the mid sits at 0.50 and the spread is 0.04. Send a market buy. It crosses to the ask and fills at 0.52\. Measured against the mid at that instant, fill quality is (0.50 minus 0.52) over 0.04, which is minus 0.5\. That minus a half is not adverse selection yet. It is the mechanical cost of crossing a full spread, and every market order pays it by construction. Now watch the mid. Ten minutes later it has slid to 0.49, because the flow you traded against knew the same news you did and it kept coming. Recompute against the markout mid: (0.49 minus 0.52) over 0.04, which is minus 0.75\. The extra quarter of a spread past the mechanical minus a half is the information you bought into. Average enough of these and you land near the market-order figure below. $$ \\underbrace{-0.72}\_{\\text{market-order FQ}} \\;\\approx\\; \\underbrace{-0.5}\_{\\text{spread you crossed}} \\;+\\; \\underbrace{-0.22}\_{\\text{adverse drift after the fill}} $$ Read that decomposition out loud. Half the damage is the spread, a cost you chose the moment you sent a taker order. The rest is the market moving against you right after, which is the definition of trading against someone better informed. ## The table Bucket fills by how you got them and how long they took, then average the markout fill quality in each bucket. | How the order filled | Average fill quality | | -------------------------------------- | -------------------- | | Market orders | \-0.72 | | Limit orders filled in under 1 minute | \-0.31 | | Limit orders filled in over 10 minutes | +0.43 | Market orders sit at minus 0.72, and the decomposition above already explains them: half a spread crossed, a fifth of a spread of adverse drift on top. The middle row is the one that catches people who think a limit order is safety. You rest a buy at 0.49, inside the spread, and it fills in twenty seconds. At the instant of the fill you look clever, because 0.49 is a cent below the 0.50 mid, a plus 0.25 on the immediate measure. Then the mid drops to 0.47 because the reason your order filled so fast was the market coming down through your price on its way lower. Markout fill quality is now (0.47 minus 0.49) over 0.04, minus 0.5, and averaged against the occasional benign fast fill you get minus 0.31\. A limit order that fills in seconds is a market order in disguise; you set the price and the market decided it agreed for a reason you will regret. The bottom row is where limit orders earn their keep. You rest at 0.49 and it fills eleven minutes later, filled by someone who needed liquidity rather than someone who knew something. The mid is still near 0.50 or has drifted up. You bought a cent under fair value and fair value held, so fill quality comes out plus 0.43\. You collected the spread you posted for instead of paying it. Speed and fill quality move in opposite directions, and the ranking is monotone: the faster you got filled, the worse you did. That is not a coincidence you can tune away. A resting order fills fast precisely when price is rushing through it, and price rushes through a level when something is pushing it, and that something is usually against you. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_391-visual-0.png) ## What the number makes you do Fill quality is not a scoreboard to admire, it is an instruction. Compute it on your own last hundred fills, split by order type and time-to-fill, and the average tells you whether your execution is donating to the people on the other side. If your market-order bucket averages worse than minus 0.5, you are paying more than the spread, which means you are taking liquidity into informed flow and should either slow down or stop taking on that signal. The old article ["Adverse Selection Explained for Traders"](https://aligrithm.com/adverse-selection-explained-for-traders/) gives the general mechanism; the number here tells you whether it is happening to you specifically. If your fast-limit bucket is negative, resting inside the spread is not protecting you, and posting tighter will not help because the fills that hurt are the ones that come quickest. If your only positive bucket is the slow-fill one, your edge is patience, and any part of the system that rushes is burning it. There is one place the sign flips on purpose. Structural arbitrage, the trade from the article "Arbitrage Is Just Projection," dies inside a single block, so you take it with a market order and you accept the minus 0.72 as a cost of capture, because a fill worth minus 0.72 of the spread still clears when the guaranteed edge is several spreads wide. Fill quality tells you the price of speed. Whether to pay it is the make-versus-take decision, and everywhere outside a vanishing arbitrage the honest fill quality says the same thing: wait for the slow fill, or do not trade. ## KEY POINTS - Fill quality for a buy is the midpoint minus your execution price, divided by the spread. Positive means you bought below fair value; negative means you paid up. Measure the midpoint after the fill, not at order time, or you see the spread you crossed but miss the information you traded against. - A market buy at the ask starts at minus 0.5 from crossing a full spread alone. The empirical minus 0.72 is that minus 0.5 plus roughly minus 0.22 of adverse drift after the fill, so half the damage is spread and half is being picked off. - The buckets rank monotone with speed: market orders minus 0.72, limit orders filled under a minute minus 0.31, limit orders filled over ten minutes plus 0.43\. A limit order that fills in seconds is a market order in disguise, because price rushed through your level on its way against you. - The only positive bucket is the slow fill, where you collected the spread from a liquidity trader instead of paying it to an informed one. Patience is the edge; anything that rushes spends it. - Compute fill quality on your own last hundred fills, split by order type and time-to-fill. A market-order average worse than minus 0.5, or any negative fast-limit average, means your execution is leaking to better-informed counterparties. - Structural arbitrage is the deliberate exception: it dies in one block, so you take it and eat the minus 0.72 because the guaranteed edge is wider than the fill cost. Everywhere else, wait for the slow fill or skip the trade. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Convex Optimization - Stephen Boyd Lieven Vandenberghe (Amazon)](https://www.amazon.com/s?k=Convex+Optimization+Stephen+Boyd+Lieven+Vandenberghe&ref=aligrithm.com) - [An Introduction to Copulas - Roger Nelsen (Amazon)](https://www.amazon.com/s?k=An+Introduction+to+Copulas+Roger+Nelsen&ref=aligrithm.com) ### 9.28 Adverse Selection Is Adverse Selection: Porting Fast-Fills-Are-Bad-Fills to FX and Futures URL: https://aligrithm.com/adverse-selection-is-adverse-selection-porting-fast-fills-are-bad-fills-to-fx-and-futures/ Last updated: 2026-07-27T13:26:18.000Z Your limit order fills in two seconds and you feel quick. On Polymarket that fast fill means the market ran through your price toward something you will not like. On EURUSD it means the same thing. On a bund future it means the same thing. The counterparty who took your resting order was faster to the news or knew more, and the speed of the fill is the tell in every market with an order book, because adverse selection is a property of who chooses to trade against you, not of what the contract pays at the end. The article "Fast Fills Are Bad Fills" made this case for prediction markets, where the contract pays a dollar or zero and the information asymmetry is sharp. The old article ["Adverse Selection Explained for Traders"](https://aligrithm.com/adverse-selection-explained-for-traders/) made the general case for order books before that. This is the bridge: the same fill-quality metric, the same market-making profit equation, and the same markout diagnostic move between a Polymarket CLOB and an FX or futures book without changing a symbol. ## Fill quality: one metric, every book When your order fills, someone took the other side. Ask why, and measure the answer with a single ratio. $$ \\mathrm{FQ} \\;=\\; \\frac{p\_{\\text{exec}} - p\_{\\text{mid}}}{p\_{\\text{ask}} - p\_{\\text{bid}}} $$ For a buy, the numerator is how far above the midpoint you actually executed and the denominator is the spread. Negative fill quality means you bought above where the market thinks fair value sits, the fingerprint of adverse selection. The prices in the formula are a probability on Polymarket and a currency rate on EURUSD, but the ratio is dimensionless and reads identically. Track it by order type and the pattern holds across markets. | Order type | Average fill quality | | -------------------------------------- | -------------------- | | Market orders | \-0.72 | | Limit orders filled in under 1 minute | \-0.31 | | Limit orders filled in over 10 minutes | +0.43 | Market orders are badly adversely selected because you cross the spread against resting flow that had every chance to pull if your price was wrong. Fast limit fills are still negative because a quick fill means the market came to your price on its way through it. Slow limit fills come out positive because someone traded against you for liquidity reasons, not information, and you earned the spread you posted for. The slogan is market-agnostic: the faster the fill, the worse the adverse selection. The old article ["Why Bid/Ask Bounce Matters for Intraday FX Systems"](https://aligrithm.com/why-bid-ask-bounce-matters-for-intraday-fx-systems/) showed the FX residue of the same effect, where the spread-sized oscillation that a naive mean-reversion system reads as signal is the microstructure cost you pay on every fast fill. ## The market maker's equation is the same equation A maker in any of these markets is selling short-term insurance against price fluctuation, not forecasting the outcome. The profit model has two terms and the second is what separates a desk from a donor. $$ \\text{Profit} \\;=\\; \\text{spread} \\times \\text{volume} \\;-\\; \\text{adverse selection} \\times \\text{informed volume} $$ The first term is the spread earned across uninformed flow that trades for liquidity. The second is what informed traders extract when they pick you off. The business is profitable only when the spread collected from noise exceeds what the informed take, and this is true whether the contract is a binary on an election or a quarterly rate future. The old article ["Market Making Is Not Just Collecting the Spread"](https://aligrithm.com/market-making-is-not-just-collecting-the-spread/) built exactly this point for order books: the quoted spread is gross, one adversely-selected fill erases several spreads, and net-of-pickoff spread is what you keep. Nothing in that argument used a feature specific to any one market. The defenses port too. Inventory skew, tightening the ask and widening the bid when you are already long to attract offsetting flow, is the old article ["Why Skewing Is Simpler Than People Think"](https://aligrithm.com/why-skewing-is-simpler-than-people-think/) applied to whatever you are quoting. And the truth serum for whether you are making money or accumulating toxic inventory is markouts, the old article ["Markouts: The Truth Serum of Market Making"](https://aligrithm.com/markouts-the-truth-serum-of-market-making/): look at where price sits a few minutes after your fill, not at the spread you nominally captured, because a book can show a healthy gross spread and a deeply negative markout at the same time on Polymarket and on crude alike. ## Spread is posterior variance, on any instrument How wide should you quote? As wide as your uncertainty about fair value. On Polymarket the fair value is a probability you hold a posterior distribution over, and the optimal spread tracks that posterior variance: confident, quote tight and capture volume; uncertain because news is breaking or the book is thin, quote wide, because a tight quote under uncertainty invites informed traders to pick the side you got wrong. Rename fair value from a probability to a currency rate and the rule is unchanged. The old article ["Spread Widening During Volatility Expansion"](https://aligrithm.com/spread-widening-during-volatility-expansion/) is this reflex on continuous instruments, mapping short-horizon volatility to quoted width, and volatility is just the observable proxy for how uncertain you are about where price will be when you get filled. Uncertainty up, spread up, in both markets. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_390-visual-0.png) ## What actually differs, and why it does not matter The honest difference is the payoff structure. A prediction-market contract terminates at a dollar or zero, so as resolution approaches the information asymmetry gets discrete and sharp, and a fast fill near resolution can mean the counterparty already knows the answer. FX and futures have no terminal binary payoff; adverse selection there is about short-horizon drift, someone trading ahead of a move measured in pips or ticks. The mechanism is the same, informed flow choosing your resting order at the moment it is about to hurt, and the difference is only in the shape of what "hurt" means. It changes your markout horizon and your spread calibration, not the existence or the sign of the effect. This is why the make-versus-take decision is the same decision everywhere. The old article ["Maker vs Taker Edge: Same Signal, Different Economics"](https://aligrithm.com/maker-vs-taker-edge-same-signal-different-economics/) made the general point that one signal is two different businesses depending on which side of the spread you sit on, and the old article ["Market Orders vs Limit Orders in FX"](https://aligrithm.com/market-orders-vs-limit-orders-in-fx/) turned it into an order-type rule: momentum and breakout need market orders because a resting limit never fills on a real breakout and captures only the failed ones, while mean-reversion suits limit orders because you want to fill at the overshoot you intend to fade. Structural arbitrage from "Arbitrage Is Just Projection" is a taking decision in prediction markets because the edge dies in a Polygon block; a breakout in futures is a taking decision because a resting limit inverts the edge. Same logic, edge magnitude against time decay, deciding whether paying the spread is worth the certainty. The lesson survives the port intact. Fast fills are bad fills, the spread is not free money, markouts not nominal spread tell you if you are winning, and quote width is a readout of what you know. None of it is a Polymarket fact or an FX fact. It is an order-book fact. ## KEY POINTS - Fill quality, execution price minus midpoint over the spread, is dimensionless and reads identically on a Polymarket CLOB, EURUSD, and a bund future. Negative means adverse selection. - The fill-quality pattern is market-agnostic: market orders average -0.72, sub-minute limit fills -0.31, ten-minute-plus limit fills +0.43\. The faster the fill, the worse the adverse selection, in every order book. - Market-maker profit equals spread times volume minus adverse selection times informed volume, whether the contract is a binary or a currency pair. It is positive only when noise-trader spread beats informed-trader losses. - The defenses port without change: skew quotes to manage inventory, and check markouts a few minutes after the fill rather than the nominal spread, because a healthy gross spread can sit on a deeply negative markout in any market. - Spread width equals posterior variance over fair value, and fair value being a probability or an exchange rate does not change the rule: confident, quote tight; uncertain, quote wide. Volatility is the observable proxy on continuous instruments. - The only real difference is the payoff structure: prediction-market asymmetry gets sharp and discrete near resolution, FX and futures adverse selection is short-horizon drift. That changes the markout horizon and spread calibration, not the existence or sign of adverse selection. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Convex Optimization - Stephen Boyd Lieven Vandenberghe (Amazon)](https://www.amazon.com/s?k=Convex+Optimization+Stephen+Boyd+Lieven+Vandenberghe&ref=aligrithm.com) - [An Introduction to Copulas - Roger Nelsen (Amazon)](https://www.amazon.com/s?k=An+Introduction+to+Copulas+Roger+Nelsen&ref=aligrithm.com) ### 9.27 CVaR Sizing: One Tool, Two Markets URL: https://aligrithm.com/cvar-sizing-one-tool-two-markets/ Last updated: 2026-07-27T13:26:18.000Z Five Polymarket election arbs that each look independent all lose together when one national result surprises. Ten trend-following futures systems that each look independent all bleed together in the same choppy regime. Different markets, different instruments, identical failure: a book of individually-good positions that is secretly one bet, and a risk model that priced them as many. The tool that stops both is the same conditional-value-at-risk constraint inside the same quadratic program, and once you see it work on election arbs you already know how to run a systematic futures book. The article "Kelly Isn't Enough" built the CVaR-constrained portfolio QP for prediction-market arbitrage. This is that machine pointed at the other pillar. The prediction-market half of it is the correlated blowup of arbitrage legs; the systematic half is what the old article ["Smooth Equity Curves Are Built, Not Found"](https://aligrithm.com/smooth-equity-curves-are-built-not-found/) called the correlation that rises in a crisis exactly when you need it to stay low. Same math. ## The number VaR hides, in both markets Value at Risk answers where the bad region starts and says nothing about what is inside it. Conditional Value at Risk answers the question that empties accounts: once you are past the boundary, how bad is it on average. $$ \\mathrm{CVaR}\_\\alpha = \\mathbb{E}\\big\[\\, \\text{Loss} \\;\\big|\\; \\text{Loss} > \\mathrm{VaR}\_\\alpha \\,\\big\] $$ CVaR is the expected loss conditional on being past the VaR line. Sort 100 trades worst to best; say the worst five lose 80, 60, 45, 30, and 20 dollars. VaR at 95 percent is 20, the boundary. CVaR at 95 percent is the average of the five. $$ \\mathrm{VaR}\_{95\\%} = \\$20, \\qquad \\mathrm{CVaR}\_{95\\%} = \\frac{80 + 60 + 45 + 30 + 20}{5} = \\$47 $$ VaR reports 20 and stops. CVaR reports 47, and the gap is the whole point: a strategy with a VaR of 20 and a CVaR of 200 has a fat tail VaR completely conceals. The old article ["Fat Tails: Why Gaussian Thinking Breaks Trading Systems"](https://aligrithm.com/fat-tails-why-gaussian-thinking-breaks-trading-systems/) made the systematic version of this argument, that a normal-distribution risk model sizes you for an ordinary day and an ordinary day is not when you blow up. CVaR is the answer in both places because it is defined on the tail itself, not on a Gaussian approximation of it. ## The loss distributions differ; the tail metric does not The two markets have different-shaped losses, and this is where people wrongly assume they need different tools. A prediction-market arbitrage loss is discrete, three branches, and the middle branch is the killer. $$ \\text{Loss} = \\begin{cases} -(\\Pi^\* - \\Delta) & \\text{full execution (the intended profit)}\\\\ -(\\text{partial cost} - \\text{recovery}) & \\text{partial execution (one leg stranded)}\\\\ 0 & \\text{no execution} \\end{cases} $$ Full execution pays the guaranteed profit, no execution costs only opportunity, and partial execution is where a hedged arb becomes a naked directional bet because one leg filled and the other did not. A systematic futures or FX strategy has no branches; its loss is a continuous return with a left tail fattened by gaps, regime shifts, and volatility clustering. Two completely different distributions. CVaR does not care. It reads the worst alpha percent of whatever distribution you hand it and reports the average loss there, so the same constraint bounds the partial-execution disaster on Polymarket and the gap-risk tail on a bund future without changing form. ## One allocation, both books: the portfolio QP Sizing positions one at a time misses the co-movement that does the damage. Allocate across the whole book at once. $$ \\max\_{f\_1, \\ldots, f\_k}\\; \\sum\_i f\_i e\_i - \\tfrac{1}{2}\\, \\mathbf{f}^\\top \\Sigma\\, \\mathbf{f} \\quad \\text{s.t.} \\quad \\sum\_i f\_i \\le F\_{\\max},\\;\\; 0 \\le f\_i \\le f\_i^{\\max},\\;\\; \\mathrm{CVaR} \\le C\_{\\max} $$ The first sum is total expected profit, each fraction times its edge. The covariance term with sigma is a penalty for correlated risk: when two positions tend to lose together, sigma grows and the optimizer holds less of both, forcing diversification the trader would not do by eye. The constraints cap total capital, cap each position, and cap the portfolio tail. This is a quadratic program, a squared objective with linear constraints, and CVXPY or OSQP or Gurobi solve hundreds of positions in milliseconds. Read the covariance term through the systematic lens and it is the same object the old article ["Why Portfolio Construction Is Part of the Signal"](https://aligrithm.com/why-portfolio-construction-is-part-of-the-signal/) insisted on: how you weight and net correlated positions changes what the strategy predicts, so ten correlated energy futures equal-weighted are one bet stacked ten times, and the QP's sigma penalty is what stops you from stacking. The old article ["The Difference Between Signal Quality and Portfolio Quality"](https://aligrithm.com/the-difference-between-signal-quality-and-portfolio-quality/) split those into two dimensions; the QP is the construction dimension made explicit, the part that decides how much of a good signal you actually keep. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_389-visual-0.png) ## Where the correlation actually hides The QP is only as honest as the sigma you feed it, and plain correlation lies in exactly the moments that matter. Five election arbs can show a linear correlation around 0.75 in calm conditions and a tail dependence near 0.90 when a national shock hits, so the joint extreme loss runs many times what the correlation predicts. The article "Correlation Lies; Tail Dependence Tells the Truth" is where the prediction-market pillar swaps the covariance matrix for a Student-t copula that captures joint tails. The systematic pillar reached the same wall from the other side: the old article ["Smooth Equity Curves Are Built, Not Found"](https://aligrithm.com/smooth-equity-curves-are-built-not-found/) ends on the warning that correlations rise in crises, so the diversification that smooths your equity curve in normal times fails precisely when the crisis arrives. Two pillars, one defect in linear correlation, one fix, model the tail dependence and let CVaR bound the joint loss rather than the marginal one. The systematic reader has one more familiar friend here. The old article ["Average Drawdown vs Extreme Drawdown"](https://aligrithm.com/average-drawdown-vs-extreme-drawdown/) separated the comfortable typical dip from the survival-question extreme, and used Monte Carlo permutation to build a distribution instead of trusting one realized max. That is the same move as simulating the loss distribution to estimate CVaR: you refuse to size off a single sample and you plan around the tail of the distribution, not its middle. ## The through-line CVaR does not raise your edge in either market. It stops the fat tail VaR cannot see and, wrapped in the QP with a covariance penalty, it stops the correlated blowup you did not price. On Polymarket that blowup is partial-execution losses correlating across arbs in a bad block. On a futures book it is ten systems that share a regime dependence and draw down as one. The article ["Match Your Edge to Your Resources"](https://aligrithm.com/match-your-edge-to-your-resources/) argued that once the math is done the job is staying alive to collect the edge, and the CVaR-constrained QP is the single piece of machinery that does that job identically across both pillars. ## KEY POINTS - CVaR is the expected loss in the worst alpha percent of outcomes, not the boundary VaR reports. In the worked example VaR is 20 dollars and CVaR is 47; a VaR of 20 with a CVaR of 200 has a fat tail VaR hides. Constrain CVaR in both markets. - Prediction-market arbitrage losses are three-branch and discrete, with partial execution the dangerous branch. Systematic-strategy losses are continuous with a gap-fattened left tail. CVaR is defined on the tail of whatever distribution you feed it, so the same constraint bounds both. - The portfolio QP maximizes edge minus a covariance penalty subject to caps on total capital, per-position size, and portfolio CVaR. The covariance term forces diversification, and QP solvers handle hundreds of positions in milliseconds. It is the same machine for election arbs and a futures book. - The covariance penalty is the systematic pillar's "portfolio construction is part of the signal" made explicit: ten correlated positions equal-weighted are one bet stacked ten times, and sigma is what stops the stacking. - Plain correlation understates joint tails in both markets: election arbs show correlation near 0.75 but tail dependence near 0.90 under a shock, and systematic books see correlations rise in a crisis. Model the tail dependence and let CVaR bound the joint loss. - CVaR raises no edge. It stops the fat tail VaR conceals and the correlated blowup a per-trade sizer never prices, identically across prediction markets and systematic trading. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Convex Optimization - Stephen Boyd Lieven Vandenberghe (Amazon)](https://www.amazon.com/s?k=Convex+Optimization+Stephen+Boyd+Lieven+Vandenberghe&ref=aligrithm.com) - [An Introduction to Copulas - Roger Nelsen (Amazon)](https://www.amazon.com/s?k=An+Introduction+to+Copulas+Roger+Nelsen&ref=aligrithm.com) ### 9.26 Kelly in Prediction Markets vs a Prop Challenge: Same Formula, Opposite Geometry URL: https://aligrithm.com/kelly-in-prediction-markets-vs-a-prop-challenge-same-formula-opposite-geometry/ Last updated: 2026-07-27T13:26:17.000Z Kelly's formula is identical in a prediction market and a prop challenge, but one optimizes long-run growth and the other a first-passage race to a target before an absorbing trapdoor. Opposite geometry, opposite sizes. _This post is for paying subscribers only._ ### 9.25 The Market Maker's Impossibility: Speed Versus Accuracy, and Why Arbitrage Is Tolerated URL: https://aligrithm.com/the-market-makers-impossibility-speed-versus-accuracy-and-why-arbitrage-is-tolerated/ Last updated: 2026-07-27T13:26:17.000Z Polymarket could eliminate structural arbitrage tomorrow. It chooses not to, and the choice is forced by a theorem, not by laziness. A combinatorial market maker faces a hard fork: price fast, or price coherently, and no mechanism does both unless one of the deepest open problems in computer science quietly turns out to be trivial. The platform picked speed. The bill for that choice is the entire structural half of this pillar, the roughly $40M that arbitrageurs extracted in a single election cycle. That is not a bug the exchange is racing to patch. It is a line item it has decided to pay. The Dutch-book argument in "The Dutch Book Theorem, Plainly" showed that incoherent prices exist. This is the harder question behind it: why can a well-run, well-capitalized exchange not simply keep its prices coherent and shut the arbitrageurs out? The answer is computational, and once you see it, the permanence of structural edge stops looking like a temporary inefficiency and starts looking like a design constant. ## The fork: fast or coherent, pick one State the tradeoff as the proposition it is. For any cost-function market maker running combinatorial markets with logical dependencies between them, exactly one of two things holds. Fast price updates, meaning constant-time per trade, imply arbitrage exists. Coherent prices, meaning the price vector always sits inside the arbitrage-free set, imply each update requires solving an NP-hard integer program. No mechanism achieves both at once. The reason is membership testing. Keeping prices coherent after every trade means verifying, every time, that the new price vector lies inside the marginal polytope from ["The Marginal Polytope: One Shape That Contains Every Fair Price."](https://aligrithm.com/the-marginal-polytope-one-shape-that-contains-every-fair-price/) For general logical constraints of the integer-programming form, that verification is co-NP-hard, the complement of checking integer feasibility. A maker that enforces coherence in real time must therefore solve a co-NP-hard problem on every single trade, which is flatly incompatible with the sub-100-millisecond updates a live exchange needs, unless P equals NP. The same computational wall that ["Frank-Wolfe: Solving a Quintillion-Vertex Problem in 100 Steps"](https://aligrithm.com/frank-wolfe-solving-a-quintillion-vertex-problem-in-100-steps/) climbs from the arbitrageur's side, approximating the projection with a hundred oracle calls, is the wall the exchange would have to climb exactly, on every order, for every user. The arbitrageur gets to be approximate and occasional. The maker would have to be exact and constant. That asymmetry is the whole game. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_367-visual-0.png) ## Polymarket chose speed, on purpose Faced with the fork, Polymarket runs independent logarithmic-market-scoring-rule sub-markets, each pricing in under 100 milliseconds, with no real-time mechanism enforcing consistency across related markets. Fast, cheap, responsive, and permanently arbitrageable. The platform tolerates the arbitrage for three concrete reasons, and each one is a business decision, not an oversight. Arbitrageurs supply volume and liquidity. Someone racing to close an incoherence is placing orders, tightening books, and moving prices toward fair, which is exactly the activity a young exchange wants more of. The maker's loss is bounded, at b times the log of the outcome count from ["If You've Used a Softmax Layer, You Already Know the Price Formula,"](https://aligrithm.com/if-youve-used-a-softmax-layer-you-already-know-the-price-formula/) so the downside of running fast is capped by construction and known in advance. And removing the arbitrage would require the slow, coherent pricing on the other side of the fork, which would throttle trading volume and drive users away. The cure is worse than the disease. Put a number on the disease. Benchmark the roughly $40M in estimated arbitrage profit over the cycle against public figures for Polymarket's gross fee revenue in the same window, and it lands on the order of a few tens of percent, around 20 percent in a back-of-the-envelope read. Significant, and evidently acceptable. The exchange is handing arbitrageurs a fifth of its gross fee take as the price of pricing fast, and the trade is worth it to them. This reframes the arbitrageur from parasite to paid liquidity provider, and it is why the edge is not a race the platform is trying to win. As long as the platform prices fast, the edge regenerates continuously, trade after trade, which is the permanence the article "The Term Structure of Prediction-Market Strategy and Crowding" relies on when it treats structural incoherence as a renewable resource rather than a one-time find. ## Why this is not a market inefficiency that competition erases Retail intuition says any free money gets competed away, so structural arbitrage must be a fluke that closes. That intuition, the one behind the old article ["Random Walk and Efficient Markets Are Not the Same Thing,"](https://aligrithm.com/random-walk-and-efficient-markets-are-not-the-same-thing/) is right about informational edges and wrong here. Competition among arbitrageurs decides who captures a given incoherence and how fast, the crowding dynamics from the term-structure article, but it does not stop new incoherences from being born. Each new trade on a fast independent sub-market can push prices back outside the coherent set, and the maker, pricing in constant time, does nothing to pull them back. The generator of the edge is the exchange's own pricing mechanism, running on every order, forever. You are not front-running a mispricing that will correct. You are harvesting the exhaust of a machine that is designed to keep producing it. ## Reflexivity: where the exchange's choice does bite the trader One caveat separates the two halves of the pillar. Structural arbitrage needs no forecast, so it is immune to reflexivity, the feedback loop where the price itself changes the reality it is pricing. For informational trades on high-visibility events, that loop is a first-order effect. On a big election, the market price shifts campaign spending, media narratives, and turnout, which shifts the true probability, which shifts the price again, an oscillation. $$ \\text{no feedback:} \\quad x\_t = f(x\_{t-1}) + w\_t \\qquad\\qquad \\text{reflexive:} \\quad x\_t = f(x\_{t-1},\\, y\_{t-1}) + w\_t $$ Read that as: the standard model evolves the hidden true probability x from its own past plus noise, while the reflexive model lets the past price y feed back into the true probability's evolution. The first is the clean world the structural arbitrageur lives in. The second is the messy world the informational trader from "Bayesian Edge in Log-Odds" actually trades, where the price is both a measurement of reality and a cause of it. The oscillation reflexivity creates is exploitable with mean reversion if you can estimate the feedback strength, but it also means an informational edge on a high-profile event is fighting a moving target it is helping to move. That is one more reason the pillar ranks structural edge above informational: the structural trader is exempt from the feedback the informational trader cannot escape. ## KEY POINTS - A combinatorial market maker faces a forced fork: fast updates imply arbitrage exists, coherent prices imply an NP-hard integer solve per trade. No mechanism does both unless P equals NP. - The reason is membership testing. Keeping prices inside the marginal polytope after every trade is co-NP-hard, incompatible with sub-100-millisecond updates. The arbitrageur gets to be approximate and occasional; the maker would have to be exact and constant. - Polymarket chose speed: independent LMSR sub-markets, no real-time cross-market consistency. Fast, cheap, and permanently arbitrageable. - The platform tolerates arbitrage for three reasons: arbitrageurs supply volume and liquidity, the maker's loss is bounded at b times the log of the outcome count, and removing the arbitrage would require slow pricing that kills volume. - The bill is roughly 20 percent of gross fees, around $40M in one election cycle. The exchange pays a fifth of its fee take to price fast, and finds the trade worth it. The arbitrageur is a paid liquidity provider, not a parasite. - This is not an inefficiency competition erases. Competition decides who captures each incoherence, but the exchange's fast pricing keeps generating new ones on every trade. The edge is a renewable resource, not a closing window. - Structural arbitrage is immune to reflexivity because it needs no forecast. Informational trading on high-visibility events fights a price-reality feedback loop, one more reason structural edge outranks informational. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Convex Optimization - Stephen Boyd Lieven Vandenberghe (Amazon)](https://www.amazon.com/s?k=Convex+Optimization+Stephen+Boyd+Lieven+Vandenberghe&ref=aligrithm.com) - [An Introduction to Copulas - Roger Nelsen (Amazon)](https://www.amazon.com/s?k=An+Introduction+to+Copulas+Roger+Nelsen&ref=aligrithm.com) ### 9.24 If You've Used a Softmax Layer, You Already Know the Price Formula URL: https://aligrithm.com/if-youve-used-a-softmax-layer-you-already-know-the-price-formula/ Last updated: 2026-07-27T13:26:16.000Z Train a classifier and you have already written the Polymarket pricing engine. The layer that turns raw network outputs into class probabilities, softmax, is the exact function a logarithmic market-scoring-rule maker uses to turn accumulated trades into prices. Same formula, same math, different bank account. The gateway article "Convex Geometry for Traders" said this in one line to keep you moving; this is the full derivation, with the numbers worked out, so you can read a market maker as a softmax layer with money on the line and know exactly why prices move the way they do. The whole point of pinning down the pricing function is that every structural result in the pillar rides on it. The projection in "Arbitrage Is Just Projection" measures profit in a distance defined by this cost function. The solver crashes in "Why Your Arbitrage Solver Crashes at 99 Cents" because of this function's curvature at the boundary. Get the price formula concrete and those results stop being abstract. ## Proper scoring rules: why truth is the profit-maximizing report Start one level below prices, at the incentive. A scoring rule pays a forecaster based on the probability they report and the outcome that happens. It is proper when reporting your true belief maximizes your expected score, so honesty is not a virtue you hope for, it is the strategy that pays best. $$ \\mathbb{E}\_q\\bigl\[S(q, \\omega)\\bigr\] \\;\\ge\\; \\mathbb{E}\_q\\bigl\[S(p, \\omega)\\bigr\] \\quad \\text{for all } p \\neq q $$ Read that as: if your true belief is q, then scoring yourself with your honest q gives at least as much expected score as reporting any other p. Two rules matter. The log score pays the log of the probability you assigned to what actually happened, and it is the seed of the market maker below. The quadratic, or Brier, score pays based on squared error and decomposes into calibration, discrimination, and base-rate uncertainty, which is the same decomposition the article "What a Market Price Actually Is" uses to gate informational trading. Every proper scoring rule converts into a market-maker cost function through one transform, so the maker's pricing inherits the honesty incentive of the rule it came from. ## The cost function and the softmax price The logarithmic market-scoring-rule maker, the one Polymarket and its kin use, holds a state vector, the running count of how many shares of each outcome have been bought. It sets a cost for reaching any state, and the price of each outcome is how fast that cost rises as you buy more of it. $$ C(\\theta) = b \\, \\ln\\!\\left( \\sum\_i \\exp(\\theta\_i / b) \\right), \\qquad p\_i(\\theta) = \\frac{\\exp(\\theta\_i / b)}{\\sum\_j \\exp(\\theta\_j / b)} $$ Read the left formula as: the cost of a state is a smoothed maximum of the state coordinates, scaled by a liquidity parameter b. Read the right as: the price of outcome i is the softmax of the state, e raised to that coordinate over b, divided by the sum so all prices are positive and sum to one. The state plays the role of logits, the prices are the softmax output, and buying delta shares of outcome i adds delta to that coordinate, which pushes its price up and drags every other price down because they must still sum to one. A machine-learning reader arrives fluent. This is also the substrate the article "Bayesian Edge in Log-Odds" leans on, because the state is a log-odds vector and updating a belief is adding a number to a coordinate, the same act as trading. ## A trade, worked to the cent Take a binary YES-NO market with liquidity parameter b equal to 100 and a fresh state of zero shares on each side. Both coordinates are zero, so the softmax gives each side e-to-the-zero over two e-to-the-zeros, which is 0.50\. Fair coin, as it should be on an untouched market. Now a trader buys 10 YES shares. The state becomes 10 on YES, 0 on NO. The new YES price is e-to-the-0.1 over the quantity e-to-the-0.1 plus 1, roughly 1.105 over 2.105, about 0.525\. The price moved from 50 percent to 52.5 percent. What did that cost? The cost of the trade is the cost at the new state minus the cost at the old state. $$ C(10, 0) - C(0, 0) = 100 \\ln\\!\\left(e^{0.1} + 1\\right) - 100 \\ln 2 \\approx 100\\,(0.744 - 0.693) = \\$5.12 $$ Read that as: buying 10 YES shares that will pay $10 if YES resolves true cost $5.12 and moved the market 2.5 points. The maker did not consult a forecast or a news feed. It ran the cost function, charged the integral of the price over the shares you bought, and updated the state. That is the entire pricing act. ## Bounded loss and the meaning of b Two properties make this maker safe to run and tunable to your market. The maker's worst-case loss is bounded, and the bound is b times the log of the number of outcomes. For a binary market that is b times the log of 2, about 0.693 times b. At b equal to 100 the maker cannot lose more than $69.30 no matter how many trades cross, no matter who is right. That bound is why "The Market Maker's Impossibility" can argue the platform tolerates arbitrage: the downside is capped by construction. The liquidity parameter b is the one knob, and it sets the tradeoff between price sensitivity and depth. $$ \\Delta p\_i \\;\\approx\\; \\delta \\, \\frac{p\_i \\, (1 - p\_i)}{b} $$ Read that as: the price impact of buying delta units of outcome i is roughly delta times p times one-minus-p, divided by b. Small b, say 10, means prices lurch on small orders, good for fast price discovery and brutal for anyone trying to fill size. Large b, say 1,000, means the book barely moves, good for large orders and expensive for the maker who is subsidizing that depth. The impact is largest at a price of 0.50, where p times one-minus-p peaks, and it vanishes as the price approaches 0 or 1\. That last fact is the seed of a problem: near the boundary the price barely moves per share on the way in, but the curvature of the cost function explodes, which is precisely the gradient blow-up that forces Barrier Frank-Wolfe in "Why Your Arbitrage Solver Crashes at 99 Cents." ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_366-visual-0.png) ## Two coordinate systems, and why a trader needs both A market has two honest descriptions, and you need to move between them. The state space counts shares, unbounded, where a trade is just addition. The price space shows probabilities, bounded in zero to one, summing to one, which is what the screen displays. The convex conjugate, the Legendre-Fenchel transform, is the translator. $$ \\theta \\to p: \\; p = \\nabla C(\\theta) \\quad (\\text{softmax}), \\qquad p \\to \\theta: \\; \\theta = \\nabla R^\*(p) \\quad (\\text{log}) $$ Read that as: apply the gradient of the cost function to a state and you get prices, the softmax direction; apply the gradient of the conjugate to prices and you get back the state, the log direction. For this maker the conjugate is negative entropy scaled by b, the sum of p-i times log p-i. You need the log direction for one concrete reason. The arbitrage math in "Arbitrage Is Just Projection" hands you target prices, the fair vector the market should show, but an exchange only accepts orders in shares. The optimal trade is the gradient of the conjugate at the target prices minus your current state, which is the log-direction map turning "the market should be priced here" into "buy this many of these contracts." Skip the conjugate and you have a target you cannot place. ## The curved ruler: Bregman and KL Distance in this market is not Euclidean, and using the wrong ruler costs money. The Bregman divergence built from the cost function is the honest measure. $$ D\_R(p \\,\\|\\, q) = R(p) - R(q) - \\nabla R(q) \\cdot (p - q) $$ Read that as: take the value of the convex function R at p, subtract its value at q, then subtract the tangent-line approximation of R at q evaluated at p; what is left is the gap between the true curve and its tangent, a distance that respects the function's curvature. Plug in negative entropy, the conjugate of this maker's cost, and the divergence collapses to KL divergence, the sum of p-i times log of p-i over q-i. Plug in one-half the squared norm and it collapses to plain Euclidean distance. The maker measures in KL because the maker's cost function is built on the log score, so the guaranteed arbitrage profit in the projection theorem is a KL distance from the current prices to the fair set. The old article ["Entropy as a Market Concept"](https://aligrithm.com/entropy-as-a-market-concept/) builds the intuition for why entropy and its divergences are the natural language of an information market, and this is that idea made into a pricing engine. ## KEY POINTS - The logarithmic market-scoring-rule maker prices with softmax. The state vector is the logits, the prices are the softmax output, and buying shares adds to a coordinate and pushes that price up. If you have used softmax, you know the price formula. - Proper scoring rules pay most for honest reports; the log score seeds this maker, the Brier score gives the calibration decomposition. Every proper scoring rule becomes a cost function through the Legendre-Fenchel transform. - Worked trade: binary market, b equal to 100, buy 10 YES shares, price moves 50 percent to 52.5 percent and costs $5.12\. The maker runs the cost function, charges the integral of price over shares, and updates the state. No forecast involved. - The maker's worst-case loss is bounded at b times the log of the outcome count, about $69.30 for a binary market at b equal to 100\. This bound is why the platform can tolerate arbitrage. - The liquidity parameter b sets sensitivity against depth. Price impact is roughly delta times p times one-minus-p over b, largest at 0.50 and vanishing near 0 and 1\. That boundary behavior is the seed of the solver blow-up that forces Barrier Frank-Wolfe. - The convex conjugate translates between shares and prices: softmax one way, log the other. You need the log direction to convert the arbitrage math's target prices into actual share orders. - Distance in this market is KL, not Euclidean, because the cost is built on the log score. The Bregman divergence from negative entropy is KL, and it is the ruler the projection theorem measures guaranteed profit with. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Convex Optimization - Stephen Boyd Lieven Vandenberghe (Amazon)](https://www.amazon.com/s?k=Convex+Optimization+Stephen+Boyd+Lieven+Vandenberghe&ref=aligrithm.com) - [An Introduction to Copulas - Roger Nelsen (Amazon)](https://www.amazon.com/s?k=An+Introduction+to+Copulas+Roger+Nelsen&ref=aligrithm.com) ### 9.23 Was It Skill, Structure, or Luck? Execution Efficiency and Regret in Four Buckets URL: https://aligrithm.com/was-it-skill-structure-or-luck-execution-efficiency-and-regret-in-four-buckets/ Last updated: 2026-07-27T13:26:16.000Z A single P&L number hides four systems and a lot of luck. Grade execution with eta, attribute profit by edge source, split regret into detection, computation, execution, and sizing, and you finally know whether it was skill, structure, or luck. _This post is for paying subscribers only._ ### 9.22 The Central Allocator and Non-Negotiable Risk Governors: Proposals, Not Orders URL: https://aligrithm.com/the-central-allocator-and-non-negotiable-risk-governors-proposals-not-orders/ Last updated: 2026-07-27T13:26:15.000Z Give five agents direct access to the wallet and they will each size their trade as if the other four do not exist. The structural solver wants 4 percent on a clean arb, the Bayesian agent wants 3 percent on a directional bet, the market maker wants inventory on six books, and none of them can see that three of those positions are the same election wearing different jerseys. Each is individually reasonable. Together they blow the capital budget, stack correlated exposure, and breach the tail-risk limit nobody was watching. The fix is a rule of authority: agents do not spend money. They ask. A single allocator holds the wallet and decides who gets funded, at what size, subject to every constraint at once. This is the operational half of the architecture from ["Separate Detection from Execution: A Three-Layer, Typed-Agent Architecture."](https://aligrithm.com/separate-detection-from-execution-a-three-layer-typed-agent-architecture/) That article said agents emit proposals across a narrow interface. This one is what sits on the other side of the interface, reading every proposal, solving one optimization, and enforcing the limits that keep a good week from turning into a blown account. ## Proposals, not orders An agent does not send an order. It sends a proposal, a structured request for capital that carries everything the allocator needs to rank and size it. Eight fields. The strategy tag, so the allocator knows the edge source. Expected profit and confidence, the two numbers that drive the objective. Requested size and minimum viable size, because some trades are worthless below a threshold, a combinatorial arb that needs all legs or none. Execution risk, the probability this fills cleanly. Urgency, how fast the opportunity decays, so a structural arb dying in one block outranks a directional bet good for hours. And a correlation key, the field that groups proposals sharing a hidden risk factor so the allocator can cap them together. The allocator is the sole authority over capital. Agents propose; it disposes. That single rule is what lets you reason about total risk at all, because there is exactly one place where every position is visible at once. ## One optimization funds the book Each cycle, the allocator solves a constrained optimization: pick the sizes that maximize confidence-weighted expected profit without breaking any limit. $$ \\max\_{\\{s\_i\\}} \\; \\sum\_i s\_i \\cdot \\pi\_{\\text{exp},i} \\cdot \\gamma\_i \\quad \\text{subject to} \\quad \\begin{cases} \\sum\_i s\_i \\le W\_{\\text{available}} \\\\\[2pt\] s\_i \\le s\_{\\text{req},i} \\\\\[2pt\] s\_i \\ge s\_{\\text{min},i} \\;\\text{ or }\\; s\_i = 0 \\\\\[2pt\] \\sum\_{i \\in \\text{group}(\\kappa)} s\_i \\le E\_{\\max}(\\kappa) \\\\\[2pt\] \\mathrm{CVaR}(\\text{portfolio} + \\text{new}) \\le C\_{\\max} \\end{cases} $$ Read the objective as: choose each trade's size so the total of size times expected profit times confidence is as large as possible. Then read the constraints top to bottom, each one a way the naive per-agent sizing kills you. Total size cannot exceed available capital. No trade gets more than it asked for. Each trade is funded at or above its minimum viable size, or not at all, which is the min-size-or-zero rule. Everything sharing a correlation key is capped as a group, so five state-election arbs cannot each look small while jointly betting the whole account on one national shock. And the tail-risk measure of the whole book, including the new trades, stays under budget. That third constraint, size at least the minimum or exactly zero, is what makes this a mixed-integer program rather than a smooth one, because it forces a discrete yes-or-no on each trade. With a moderate number of proposals it still solves in milliseconds. The correlation-group cap and the tail-risk constraint are where this connects to ["Kelly Isn't Enough: Drawdown- and CVaR-Constrained Sizing and the Portfolio QP."](https://aligrithm.com/kelly-isnt-enough-drawdown-and-cvar-constrained-sizing-and-the-portfolio-qp/) The tail-risk term here is the CVaR from that article, and the correlation key is the operational form of the tail-dependence lesson from "Correlation Lies, Tail Dependence Tells the Truth," because grouping by a shared risk factor is how you stop the allocator from funding five trades it wrongly believes are independent. When capital runs short and not every good proposal can be funded, rank them by priority. $$ \\text{Priority}\_i = \\frac{\\pi\_{\\text{exp},i} \\times \\gamma\_i}{\\rho\_{\\text{exec},i} \\times \\left(1 + \\tau\_i^{-1}\\right)} $$ Read that as: priority rises with expected profit and confidence, and falls with execution risk and with how long you can afford to wait. The urgency term in the denominator, one over tau, means a fast-decaying opportunity gets pushed up the queue, because a slow one can wait for the next cycle. Fund highest priority first until the capital or a risk limit binds. ## Conflicts resolve by rule, not by argument Two agents will collide, and the allocator settles it mechanically. | Conflict | Resolution | | -------------------------------- | ---------------------------------------- | | Same market, opposite directions | Net the positions | | Capital competition | Fund higher priority first | | Risk limit reached | Cut size to a compliant level, or reject | Same market opposite directions net out, so you are not paying spread twice to hold offsetting bets. Capital competition goes to priority. A binding risk limit shrinks the trade to fit or drops it. No discretion, no override, no agent lobbying for special treatment. ## Risk governors are non-negotiable The optimization maximizes profit. The governors below do not care about profit at all. They exist to keep you alive, and they override everything. The kill switch halts all trading if portfolio drawdown passes 15 percent. The position cap forbids any single trade above 2 percent of capital. The exposure cap forbids any single market above 10 percent of capital. The failure-rate monitor pauses trading and forces a diagnosis if execution failures exceed 30 percent, because a system missing a third of its fills is broken in a way more capital will not fix. The solver-timeout monitor shrinks the opportunity window when integer-programming solves start timing out, so you stop chasing problems you cannot compute in time. These do not increase returns. They dramatically improve survival, and survival is the whole game. This is the same message as the old article ["Why Loss Control Is the Only Thing You Fully Control,"](https://aligrithm.com/why-loss-control-is-the-only-thing-you-fully-control/) that you cannot control the edge but you can control the size and the stop, and it is the operational core of the old article ["When to Switch Off a Trading System,"](https://aligrithm.com/when-to-switch-off-a-trading-system/) which is exactly what a kill switch automates. Operational discipline is the ninth source of edge in ["The Nine Sources of Edge, Ranked Cleanest to Dirtiest,"](https://aligrithm.com/the-nine-sources-of-edge-ranked-cleanest-to-dirtiest/) ranked last not because it makes money but because skipping it eventually loses all of it. The governors are that edge, written as code that cannot be argued with at 3 a.m. during a drawdown. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_364-visual-0.png) ## KEY POINTS - Agents propose, they do not execute. A single allocator holds the wallet, which is the only way to reason about total risk, because it is the one place every position is visible at once. - A proposal is eight fields: strategy tag, expected profit, confidence, requested size, minimum viable size, execution risk, urgency, and a correlation key for grouping shared risk. - The allocator maximizes confidence-weighted expected profit subject to five constraints: total capital, per-trade requested cap, min-size-or-zero, per-correlation-group exposure cap, and a portfolio CVaR tail-risk budget. The min-or-zero rule makes it a mixed-integer program that still solves in milliseconds. - The correlation-key cap is the operational form of tail dependence: it stops the allocator from funding five trades it wrongly believes are independent. The CVaR term is the same tail-risk budget from the portfolio-QP article. - Under capital scarcity, rank by priority: profit times confidence, divided by execution risk times a term that rises as urgency falls. Fast-decaying opportunities jump the queue. - Conflicts resolve by rule: net opposite positions in one market, fund higher priority under capital competition, shrink or reject when a risk limit binds. - Risk governors are non-negotiable and override the optimization: kill switch at 15 percent drawdown, 2 percent per-trade cap, 10 percent per-market cap, pause above 30 percent fill failures, shrink the window on solver timeouts. They do not raise returns; they keep you solvent, which is the operational edge. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Convex Optimization - Stephen Boyd Lieven Vandenberghe (Amazon)](https://www.amazon.com/s?k=Convex+Optimization+Stephen+Boyd+Lieven+Vandenberghe&ref=aligrithm.com) - [An Introduction to Copulas - Roger Nelsen (Amazon)](https://www.amazon.com/s?k=An+Introduction+to+Copulas+Roger+Nelsen&ref=aligrithm.com) ### 9.21 Separate Detection from Execution: A Three-Layer, Typed-Agent Architecture URL: https://aligrithm.com/separate-detection-from-execution-a-three-layer-typed-agent-architecture/ Last updated: 2026-07-27T13:26:15.000Z The moment your solver is also your order router, one crash takes down both. A structural-arbitrage solver runs for thirty minutes on a hard problem, hits an edge case, and dies. If that same process was responsible for filling the proposals it computed an hour ago, those fills die with it, and you are now holding a half-legged position on a market that has moved. Wire detection and execution together and every failure in one becomes a failure in both, every scaling decision for one constrains the other, and you cannot test either without the other live. The fix is a design rule that predates prediction markets and survives them: keep the thing that finds trades separate from the thing that fills them. The old article ["The Difference Between a Trading Rule, a Strategy, and a Portfolio"](https://aligrithm.com/the-difference-between-a-trading-rule-a-strategy-and-a-portfolio/) makes the layered version of this argument for signals, that a rule, a strategy, and a portfolio are different objects living in different files and failing for different reasons. The same discipline scales up to a live prediction-market system running several strategies at once, structural arbitrage, informational directional bets, and market making, each with its own detection logic, sizing needs, and ways to break. Coupling them is the mistake. Separation is the architecture. ## The separation principle Split the system into two halves that talk through one narrow interface. Opportunity engines answer what to trade. Execution engines answer how to fill it. Between them passes a single object: a trade proposal. | Opportunity engine (what to trade) | Execution engine (how to fill) | | ---------------------------------- | -------------------------------- | | A0: structural arbitrage solvers | Order construction | | A1: fast constraint scanners | Parallel submission, under 30 ms | | A2: market-making quoters | Fill monitoring | | B0: Bayesian directional agents | Partial-fill handling | | R: regime detection and gating | Post-trade accounting | Three reasons force the split, and each one is a failure you avoid. Independent scaling. Detection wants high-throughput market monitoring, watching thousands of order books and dependency graphs at once. Execution wants low-latency I/O, getting one set of orders to the mempool as fast as physically possible. Those are different machines with different bottlenecks, and jamming both onto one process means neither is tuned right. Independent failure. When the integer-programming solver crashes, the execution engine keeps filling the proposals it already holds. When execution latency spikes because the chain is congested, the opportunity engine keeps detecting and queuing. Neither outage cascades into the other. This is the same logic behind the old article ["The Death of the Single-System Trader,"](https://aligrithm.com/the-death-of-the-single-system-trader/) that a single coupled point of failure is how a system dies quietly, and the answer is structural independence rather than one monolith you pray does not fall over. Independent testing. Opportunity engines backtest on historical data with no live execution at all. Execution engines test against synthetic orders with no real edge required. Decoupled, you can validate each half in isolation, which is the only way to know which half is actually broken when live performance disappoints. ## Three layers, three questions, three latency budgets Inside that split, stack three layers, each answering one question on its own clock. Layer 1, inference, answers what is happening right now. It ingests WebSocket price feeds, order-book snapshots, on-chain events, and news, and it outputs price vectors, a regime classification, and the dependency graph linking related markets. Budget: under 100 milliseconds for prices, under 10 seconds for a regime update. Layer 2, decision, answers what should we do. It takes Layer 1's outputs plus current portfolio state and risk parameters, runs arbitrage detection, the Bregman projection from "Arbitrage Is Just Projection," and Kelly sizing, and it emits trade proposals of the form market, direction, size, limit price. Budget: under 1 second for a simple arbitrage check, up to 30 minutes for a full Frank-Wolfe projection on a hard combinatorial problem. That wide spread is not sloppiness. The article ["Frank-Wolfe: Solving a Quintillion-Vertex Problem in 100 Steps"](https://aligrithm.com/frank-wolfe-solving-a-quintillion-vertex-problem-in-100-steps/) explains why a genuinely hard structural problem can take real wall-clock time to converge, and the architecture has to tolerate a Layer 2 component that thinks for half an hour without stalling everything else. Layer 3, execution, answers did we get filled. It takes proposals, queries the live order book and a gas oracle, constructs orders, submits them in parallel, monitors fills, and handles partials. Budget: under 30 milliseconds from decision to mempool. That number is not arbitrary either. The article "Execution Is Part of Expected Value" shows why all legs of a multi-leg arb have to land inside one Polygon block or the price drifts between them, and the 30-millisecond target is what it takes to make that window. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_363-visual-0.png) ## Typed agents, and the fast-scanner-triggers-deep-solver trick Not every opportunity engine is the same shape. Type them by edge source, speed, and compute cost, and the system becomes legible: you know what each agent is for, how fast it runs, and how much it costs to run. | Agent | Edge source | Speed | Compute | | --------------------------- | -------------- | ------------ | -------------------------------- | | A0: structural arb solver | Structural | Seconds | High, integer-programming solver | | A1: fast constraint scanner | Structural | Milliseconds | Low, linear checks | | A2: market maker | Microstructure | Milliseconds | Medium | | B0: Bayesian directional | Informational | Seconds | Medium | | R: regime detector | Regime | Seconds | Low | Each agent maps to a strategy the pillar already built. A0 runs the projection machinery. A2 quotes with the make-or-take logic from "Fast Fills Are Bad Fills." B0 runs the posterior from "Bayesian Edge in Log-Odds." R is the regime gate from "The Term Structure of Prediction-Market Strategy and Crowding," and it does the most valuable thing in the whole system, telling the others when to stand down. The A0-A1 pairing is the pattern worth stealing. Running a full integer-programming solve on every price tick across thousands of markets is impossible, the compute cost is too high. So A1, the fast constraint scanner, runs cheap linear checks in milliseconds across everything, and when it spots a cheap violation that smells like an arbitrage, it triggers A0 to run the expensive full solve on that one market. A fast, dumb pre-filter guards a slow, smart solver. You get millisecond coverage of the whole market and second-scale depth only where it pays, instead of choosing one or the other. ## KEY POINTS - Coupling detection and execution means one crash kills both, one scaling decision constrains both, and you cannot test either in isolation. Separate them behind a single narrow interface, the trade proposal. - Three reasons to separate: independent scaling (detection wants throughput, execution wants low latency), independent failure (a dead solver still lets fillers work), independent testing (backtest opportunity engines on history, test execution engines on synthetic orders). - Three layers, each with its own clock. Inference, what is happening, under 100 ms for prices. Decision, what should we do, 1 second to 30 minutes for a full projection. Execution, did we get filled, under 30 ms to mempool so all legs land in one block. - The 30-minute decision budget and the 30-millisecond execution budget are both load-bearing: hard combinatorial projections genuinely take time, and multi-leg arbs genuinely need one-block execution. - Type your agents by edge source, speed, and compute: A0 structural solver, A1 fast scanner, A2 market maker, B0 Bayesian directional, R regime detector. Each maps to a strategy the pillar already built. - The best latency trick is A1 triggering A0: a cheap millisecond scanner across all markets fires the expensive integer-programming solve only where a violation appears. Full-market coverage plus deep computation only where it pays. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Convex Optimization - Stephen Boyd Lieven Vandenberghe (Amazon)](https://www.amazon.com/s?k=Convex+Optimization+Stephen+Boyd+Lieven+Vandenberghe&ref=aligrithm.com) - [An Introduction to Copulas - Roger Nelsen (Amazon)](https://www.amazon.com/s?k=An+Introduction+to+Copulas+Roger+Nelsen&ref=aligrithm.com) ### 9.20 Correlation Lies, Tail Dependence Tells the Truth: Student-t Copulas for Election Portfolios URL: https://aligrithm.com/correlation-lies-tail-dependence-tells-the-truth-student-t-copulas-for-election-portfolios/ Last updated: 2026-07-27T13:26:14.000Z You hold arbitrage positions across five US state-election markets and you feel diversified. Five separate markets, five separate resolutions, the losses should wash out. Then a national-level shock hits, a debate implosion, a scandal, a macro print, and all five markets move the same direction at once. The diversification you counted on evaporates precisely when you needed it, and the portfolio that was supposed to lose a dime loses two and a half dollars. The linear correlation number on your dashboard said 0.75 and looked manageable. It was measuring the wrong thing. Correlation describes how markets move together on an average day. It says almost nothing about how they move together on the worst day, and the worst day is the only one that can end you. This is anti-pattern number four from ["The Six Ways to Lose Money on Polymarket"](https://aligrithm.com/the-six-ways-to-lose-money-on-polymarket/) written out in full. That article named correlation blindness as a way to blow up; this one gives you the math that separates the average-day co-movement you can ignore from the bad-day co-movement that bounds your real downside, and the tool, a copula, that lets you size for the second instead of the first. ## Correlation understates the tail The linear correlation coefficient between two election markets might be 0.75\. That describes the ordinary scatter, how the two returns track each other across the full cloud of normal days. What you actually care about is a different quantity, the tail dependence coefficient, which asks a sharper question. $$ \\lambda\_{\\text{upper}} = \\lim\_{q \\to 1} \\Pr\\!\\bigl(Y > F\_Y^{-1}(q) \\,\\big|\\, X > F\_X^{-1}(q)\\bigr) $$ Read that as: given that market X just had an extreme move, past its q-th percentile with q pushed toward 1, what is the probability that market Y also had an extreme move? That conditional probability, in the limit at the very edge of the distribution, is the tail dependence. Linear correlation averages over the whole distribution and drowns this out. Two markets with correlation 0.75 can carry tail dependence of 0.85 to 0.95, meaning that when one goes to an extreme the other almost always follows. A risk model built on the 0.75 alone underestimates the joint extreme loss by a factor of three to five, which is the exact gap between the loss you budgeted for and the loss that shows up. The old article ["Fat Tails: Why Gaussian Thinking Breaks Trading Systems"](https://aligrithm.com/fat-tails-why-gaussian-thinking-breaks-trading-systems/) makes the univariate version of this mistake: assume a normal distribution and the extremes that actually drive your P&L are dismissed as impossible. Tail dependence is the multivariate cousin. It is not that each market has fat tails, it is that their tails are wired together, and a Gaussian view of the joint distribution treats a synchronized five-market crash as a once-in-the-history-of-the-universe event while the market delivers one every election cycle. ## The 25 to 1 blow-up, worked Put numbers on the five-market portfolio. Under normal conditions each position swings a nickel, and because the day-to-day moves are close to independent you get a genuine diversification benefit, the portfolio swings about a dime, roughly the square-root-of-five shrinkage you would expect. Then the national shock arrives and every position moves adversely at the same time. | Scenario | Per position | Portfolio | Diversified? | | --------------------------- | --------------------- | ---------------------- | ------------ | | Normal day | plus or minus 5 cents | plus or minus 10 cents | Yes | | Joint tail (national shock) | minus 50 cents | minus $2.50 | No | The normal-day row shows real diversification: five independent nickels net to about a dime, not a quarter. The joint-tail row shows it vanishing: five positions each down 50 cents net to $2.50, with no cancellation, because on the bad day they are not five independent bets, they are one bet on the national environment wearing five costumes. The tail-to-normal loss ratio is 25 to 1\. An independence assumption would have predicted 5 to 1\. The old article ["The Death of the Single-System Trader"](https://aligrithm.com/the-death-of-the-single-system-trader/) derives the portfolio-volatility benefit of combining uncorrelated systems, the square-root scaling that makes diversification work, and the lesson here is the sharp edge of that same blade: the benefit is real on normal days and conditional on the correlations staying low, and when a shock drives the correlations to one, the square-root math you were relying on simply does not apply. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_362-visual-0.png) ## A copula separates how each market behaves from how they move together The fix is to stop modeling the joint distribution as one lump and split it in two. A copula does exactly that: it separates the marginal behavior of each market from the dependence structure that links them. $$ F(p\_1, p\_2) = C\\bigl(F\_1(p\_1),\\, F\_2(p\_2)\\bigr) $$ Read that as: the joint distribution F of the two prices factors into the individual distributions F-one and F-two, each describing one market on its own, wrapped inside a copula function C that describes only how they move together. You model each market's own behavior however you like, then choose a copula to glue them, and the choice of glue is where the tail dependence lives. This is the same separation the old article ["Why Portfolio Construction Is Part of the Signal"](https://aligrithm.com/why-portfolio-construction-is-part-of-the-signal/) argues for at the portfolio layer: the quality of the individual trades and the quality of how you combine them are two different problems, and here the copula is the formal object that carries the combination half. ## Gaussian glue hides the crash, Student-t glue shows it The choice of copula is not cosmetic. Run 10,000 simulated scenarios for the five correlated election markets and count how many produce a joint extreme loss. Under a Gaussian copula you might see about 5 such scenarios. Under a Student-t copula you see 50 or more. Same marginals, same nominal correlation, ten times the joint-tail events, because the Gaussian copula has thin tails and forces extreme co-movement to be rare, while the Student-t copula lets the markets clutch together in the tails the way real correlated markets do under a shock. For election markets, use a Student-t copula with the degrees-of-freedom parameter around 4 to 6\. That setting gives you moderate correlation on normal days and strong co-movement in the tails, which matches how state markets actually behave: loosely linked most of the time, tightly linked when a national event moves everything. The parameter controls how fat the joint tails are. Push it to infinity and you recover the thin-tailed Gaussian copula that just underestimated your risk; drop it to 3 or 4 and you get the heavy, crisis-like dependence seen in stressed financial assets. Around 4 to 6 is the honest middle for election portfolios. This is the modeling engine behind the companion notebook for this pillar, which runs the 10,000-scenario simulation both ways and reproduces the 25-to-1 tail ratio, so you can see the Gaussian model undercount the crashes on your own positions. ## Independence is assumed, correlation is paid for The guideline is one line, and it is the whole chapter: compute portfolio risk with copula-based simulation, not a multivariate-normal assumption, because the chance of simultaneous adverse moves across correlated markets is far higher than a Gaussian model admits. Many individually good trades can combine into a portfolio that dies on a single shock, which means the portfolio-sizing framework from ["Kelly Isn't Enough: Drawdown- and CVaR-Constrained Sizing and the Portfolio QP"](https://aligrithm.com/kelly-isnt-enough-drawdown-and-cvar-constrained-sizing-and-the-portfolio-qp/) is incomplete if it feeds on linear correlation. The correlation-risk penalty term in that quadratic program has to be built from the tail-dependence structure, or the optimizer will happily load up five state markets it believes are diversified and hand you the 25-to-1 loss the first time the nation moves as one. Portfolio construction is the seventh source of edge in the hierarchy, a meta-edge that prevents catastrophe rather than generating profit, and the way you earn it is by refusing to assume independence you have not measured, and by pricing the correlation you are actually carrying. ## KEY POINTS - Linear correlation describes average-day co-movement and says almost nothing about the worst day. Two election markets at correlation 0.75 can have tail dependence 0.85 to 0.95, and the worst day is the only one that ends you. - Tail dependence is the conditional probability that market Y hits an extreme given that market X did, taken at the edge of the distribution. A model using only linear correlation underestimates joint extreme losses by three to five times. - Worked five-market portfolio: normal days, plus or minus 5 cents per position nets to plus or minus 10 cents (real square-root diversification). National shock, minus 50 cents per position nets to minus $2.50 with no cancellation. The tail-to-normal ratio is 25 to 1, not the 5 to 1 independence predicts. - A copula separates each market's own behavior (the marginals) from how they move together (the dependence). The choice of copula is where tail dependence lives. - Gaussian copula versus Student-t on 10,000 scenarios: about 5 joint extreme losses versus 50 or more. The Gaussian has thin tails and forces synchronized crashes to be rare; the Student-t lets markets clutch together in the tails. Use a Student-t with degrees of freedom around 4 to 6 for election portfolios. - Independence is assumed; correlation is paid for. The portfolio QP must build its correlation penalty from tail dependence, not linear correlation, or it will load up markets it wrongly believes are diversified. This is the portfolio meta-edge: preventing the one-shock blow-up. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Convex Optimization - Stephen Boyd Lieven Vandenberghe (Amazon)](https://www.amazon.com/s?k=Convex+Optimization+Stephen+Boyd+Lieven+Vandenberghe&ref=aligrithm.com) - [An Introduction to Copulas - Roger Nelsen (Amazon)](https://www.amazon.com/s?k=An+Introduction+to+Copulas+Roger+Nelsen&ref=aligrithm.com) ### 9.19 The Term Structure of Prediction-Market Strategy and Crowding: Why the Fastest Three Bots Take 80 Percent URL: https://aligrithm.com/the-term-structure-of-prediction-market-strategy-and-crowding-why-the-fastest-three-bots-take-80-percent/ Last updated: 2026-07-27T13:26:14.000Z A structural arbitrage that printed money in October 2024 was worthless by February 2025, and nothing about the math changed. The polytope was still the polytope, the projection still returned a positive guaranteed profit, the solver still converged. What changed was the market around the trade: the volume dried up, the same opportunity now had eight bots racing for it instead of one, and the election that generated all the incoherence had resolved. A trade is not just its own edge. It lives inside a regime, and the regime decides whether that edge is real, shrinking, or already gone to someone faster than you. This is the meta-layer the earlier pillar articles skipped. "Arbitrage Is Just Projection" tells you the trade exists. "Execution Is Part of Expected Value" tells you whether you can fill it. Neither tells you whether you should be trading this class of opportunity at all right now, or whether you are the third bot in line about to provide exit liquidity to the first two. That is a regime question, and regime sits at rank six in ["The Nine Sources of Edge, Ranked Cleanest to Dirtiest"](https://aligrithm.com/the-nine-sources-of-edge-ranked-cleanest-to-dirtiest/) for a reason: it is a meta-edge that improves everything above it, and its single most valuable output is the instruction to not trade. ## Four regimes, and the strategy each one allows Markets sit in qualitative states. Each state sets how much edge is on the table, how fast it decays, and which strategy survives. Detection is not exotic: keep rolling-window statistics on three numbers, spread, order-book depth, and your own fill rate, and let regime boundaries trigger a change in behavior. | Regime | Spread | Depth | Edge density | Typical trigger | | ----------------- | ------------- | ----------- | ------------------ | ----------------- | | Deep liquidity | under 2 cents | over $10K | Moderate | Normal conditions | | Thin liquidity | 2 to 8 cents | $500 to $5K | Low | Off-hours | | Liquidity crisis | over 8 cents | under $500 | High if executable | News shocks | | Crowded arbitrage | Normal | Normal | Very low | Post-discovery | Read the table by what it does to your fill. Deep liquidity is where you want to be: tight spreads, enough depth to size into, moderate but reliable edge. Thin liquidity looks fine on a screenshot and starves you on execution, because the depth is not there to absorb your order without slippage eating the edge. Liquidity crisis is the seductive one: spreads blow out to 8 cents and the mispricings look enormous, but the depth collapses under $500, so the edge is real and mostly unfillable, and the same news shock that opened the gap is moving the fair value under you. Crowded arbitrage is the quiet killer, spread and depth look normal, but the edge density has been competed to near zero because everyone already found this opportunity. The old article ["Volatility Regimes and Strategy Survival"](https://aligrithm.com/volatility-regimes-and-strategy-survival/) makes the general version of this point on continuous instruments: a rule is only ever profitable conditional on the regime that produced its backtest, and the old article ["Regime Coverage: Why Your Backtest Needs Different Market States"](https://aligrithm.com/regime-coverage-why-your-backtest-needs-different-market-states/) is the warning that a backtest spanning only one of these four states is lying to you about the other three. ## Volatility clusters, so track it cheaply You do not need a full GARCH fit to know whether the market is calm or violent. Volatility clusters, big moves follow big moves, and an exponentially weighted moving average of squared returns captures that with one parameter. $$ \\hat{\\sigma}^2\_t = \\lambda\\, \\hat{\\sigma}^2\_{t-1} + (1 - \\lambda)\\, r\_t^2, \\qquad \\lambda \\approx 0.94 $$ Read that as: today's variance estimate is mostly yesterday's estimate, nudged by today's squared return. The parameter lambda near 0.94 sets the memory, recent returns matter more than old ones, and nothing ever fully leaves the average. High reading means the market is volatile and prices are swinging; low reading means it is calm. This is the same non-stationarity the old article ["Why Volatility Is More Non-Stationary Than Trend"](https://aligrithm.com/why-volatility-is-more-non-stationary-than-trend/) hammers on: your volatility estimate drifts faster than any directional signal, so you re-measure it constantly rather than assuming a fixed level. ## The term structure of strategy: momentum far out, reversion near the close Here is the genuinely prediction-market-specific insight, the one continuous-instrument traders do not get for free. A prediction contract has a resolution date, and how far you sit from that date flips which strategy pays. Far from resolution, more than 30 days out, information arrives directionally. New polls, new endorsements, new fundamentals keep pushing the true probability one way as the picture clarifies, and the price trends toward the eventual answer. Momentum pays. Near resolution, inside about 7 days, most of the real information is already priced. What is left is noise, and the price overreacts to every rumor and every thin-market print, snapping back once the noise clears. Mean reversion pays. The old article ["High Noise Markets Are Mean-Reversion Markets"](https://aligrithm.com/high-noise-markets-are-mean-reversion-markets/) gives the statistical backbone: when the efficiency ratio drops and price churns without net progress, you fade the moves rather than follow them, which is exactly the near-resolution state. A strategy that works at 60 days to resolution can lose money at 2 days to resolution, on the same market, with no bug in the code. The date is a state variable, and a system that does not condition on time-to-resolution is running one strategy across two regimes that demand opposite behavior. ## Crowding: the fastest few take almost everything Structural arbitrage does not pay everyone who finds it. It pays whoever gets there first, and the math of that is brutal. When N arbitrageurs detect the same opportunity, each fill moves the price against the next arrival, so the k-th trader inherits a shrunken profit. $$ \\Pi\_k \\approx \\Pi\_0 - \\sum\_{j < k} |\\delta\_j| \\cdot \\lambda $$ Read that as: the k-th bot's profit is the original guaranteed profit minus the cumulative price impact of every faster bot ahead of it, where lambda is the Kyle impact parameter measuring how much each unit of order size moves the price. The first bot captures near the full edge. By the time the trade reaches the third or fourth arrival, the earlier fills have already dragged the price into the fair set, and the latecomers trade against a book that has moved against them. The fastest two or three traders capture over 80 percent of the available profit; the rest lose money on adverse fills. This is the same winner-take-most concentration that "Execution Is Part of Expected Value" describes when it explains why copytrading an arb is exit liquidity: by the time you see the block that captured it, you are trader number five, and the equation above already spent your edge. The Polymarket data from the 2024 to 2025 election cycle shows exactly this shape. Of the roughly $40M in estimated arbitrage profit extracted, a small handful of accounts took a large share, on the order of 20 percent for the top ten by one reading of the breakdown, with a long tail of smaller extractors, many of whom likely finished net negative after failed executions ate their gas and slippage. Structural arbitrage is a math problem where the answer is guaranteed and the profit still goes to almost no one, because the guarantee is on the trade, not on your latency. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_361-visual-0.png) ## The four-phase life of an arbitrage Every structural edge ages through the same four phases, and knowing which phase you are in tells you whether to scale up, hold, or walk away. | Phase | Participants | What it feels like | | -------------- | ------------ | -------------------------------------------------------------- | | 1\. Discovery | Few | High profit per trade, almost no competition | | 2\. Adoption | Growing | Profit per trade falling; infrastructure is the differentiator | | 3\. Saturation | Many | Profit roughly equals execution cost; only the fastest survive | | 4\. Commodity | Stable | Returns converge to the risk-adjusted market rate | In discovery you make outsized returns because you are alone. In adoption the returns fall and the game quietly becomes an infrastructure race, faster detection, faster solver, faster submission, which is the moment most retail participants should recognize they have already lost. In saturation the profit per trade drops to roughly the cost of executing it, so only the lowest-latency operators clear anything, and everyone else is trading for the exchange's benefit. In commodity the edge is gone as an edge. This is ["The Death of the Single-System Trader"](https://aligrithm.com/the-death-of-the-single-system-trader/) told in fast-forward: the old article warns that every edge decays and the death is a quiet grind rather than a crash, and structural arbitrage runs that entire arc in months instead of years because the competition is automated and the opportunity is public the instant a price prints. The $40M number is a snapshot of a specific window, April 2024 to April 2025, which included a US Presidential election, the moment of maximum volume and maximum structural inconsistency across thousands of hastily-listed related markets. That was a discovery-and-adoption window for the whole category. Do not backtest against it and assume the same edge density persists in a quiet off-cycle month, because you would be extrapolating discovery-phase profits into a saturated market. ## Not trading is a position The discipline to abstain during crowded or thin-liquidity regimes beats the ability to detect one more marginal opportunity. That is the whole guideline, and it is worth more than it sounds. A detector that fires on every incoherence will hand you a stream of trades that are real arbitrages and losing propositions at once, because they are crowded, or thin, or two days from resolution running a momentum rule that should be a reversion rule. The old article ["When to Switch Off a Trading System"](https://aligrithm.com/when-to-switch-off-a-trading-system/) frames the general skill: knowing the conditions under which your edge does not exist, and standing down when they hold, is a rule in its own right. In prediction markets the abstain signal is concrete, wide spreads with no depth, normal spreads with competed-away edge density, or a resolution date close enough to flip your strategy's sign. Reading that signal and doing nothing is a regime edge. It is source number six, and it protects every edge ranked above it. ## KEY POINTS - A trade lives inside a regime, and the regime decides whether its edge is real, shrinking, or already captured. Regime is a meta-edge (source six in the nine) whose most valuable output is the instruction to not trade. - Four regimes, detected from rolling spread, depth, and fill-rate stats: deep liquidity (trade here), thin liquidity (looks fine, starves execution), liquidity crisis (huge edge, mostly unfillable, fair value moving), crowded arbitrage (normal book, edge competed to near zero). - Track volatility cheaply with an EWMA of squared returns, lambda near 0.94\. Volatility clusters and drifts faster than any directional signal, so re-measure it constantly. - The term structure of strategy is prediction-market specific: more than 30 days from resolution, information arrives directionally and momentum pays; inside 7 days, information is priced and price overreacts to noise, so mean reversion pays. The same market flips which strategy works as the date approaches. - Crowding math: the k-th arbitrageur's profit is the original minus the cumulative price impact of everyone faster. The fastest two or three take over 80 percent; latecomers trade adverse fills. In the 2024 to 2025 cycle a handful of accounts took roughly a fifth of about $40M, with a long net-negative tail. - Every structural edge ages through discovery, adoption, saturation, commodity. The $40M was a discovery-and-adoption snapshot around a Presidential election. Do not extrapolate that edge density into a quiet off-cycle month. - Abstaining is a position. Wide-and-shallow, competed-away, or wrong-side-of-the-term-structure conditions are concrete signals to stand down, and standing down protects every cleaner edge above regime in the hierarchy. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Convex Optimization - Stephen Boyd Lieven Vandenberghe (Amazon)](https://www.amazon.com/s?k=Convex+Optimization+Stephen+Boyd+Lieven+Vandenberghe&ref=aligrithm.com) - [An Introduction to Copulas - Roger Nelsen (Amazon)](https://www.amazon.com/s?k=An+Introduction+to+Copulas+Roger+Nelsen&ref=aligrithm.com) ### 5.39 Big Moves in Lead-Lag: Liquidations, Impact, and News URL: https://aligrithm.com/big-moves-in-lead-lag-liquidations-impact-and-news/ Last updated: 2026-07-27T13:25:24.000Z A lead-lag regression fit on all the data is mostly fit on noise. Most bars are small, directionless wiggles where A and B drift around their spreads, and the ordinary least-squares slope spends its variance explaining those. The old article ["Lead-Lag Relationships in Global Markets"](https://aligrithm.com/lead-lag-relationships-in-global-markets/) measured the relationship with a lagged regression and treated every observation equally. On the HFT side, that is the wrong weighting. The money in a follow-the-leader trade is in the big moves, and a model that averages them in with a thousand tiny ones is diluting the only part that pays. Two fixes, one statistical and one structural. Reweight the regression toward the tail, and classify what caused the big move so you know whether to trade it or hide from it. ## Fit the tail, not the middle The plain lead-lag slope answers "on average, how much does B follow A." The question that matters when A prints a large move is "when A moves big, how much does B follow," and those are not the same number. Average and tail behavior diverge because impact is nonlinear: a 5 bps move in A barely perturbs B's book, a 50 bps move sweeps levels and drags B with it. You want the slope conditioned on the large moves. Two ways to get it. The blunt one is to filter: keep only the observations where A's move exceeds a size threshold, throw away the small bars, and regress on what remains. You lose sample but you stop fitting the noise. The sharper one is quantile regression, which fits the relationship at a chosen quantile of B's move distribution instead of at the mean. $$ \\min\_{\\beta}\\ \\sum\_{t}\\ \\rho\_{\\tau}\\!\\left(\\Delta B\_t - \\beta\\,\\Delta A\_{t-k}\\right), \\qquad \\rho\_{\\tau}(u)=u\\,\\bigl(\\tau - \\mathbb{1}\[u<0\]\\bigr) $$ Read it as a tilted version of least squares. Ordinary regression squares the error and so chases the mean; quantile regression weights positive and negative errors unequally by the factor tau, the check function rho, so the fitted line lands at the tau-th quantile of B's move rather than its average. Set tau high, near 0.9, and beta describes how B behaves in its large upper-tail moves, k is the lead lag you already measured, and the slope you get is the one that governs the trades worth taking. The middle of the distribution stops dominating the fit. The practical payoff is a beta that is honest about size. If the all-data slope is 0.4 but the upper-tail slope is 0.7, then when A prints a big move you should forecast B following harder than the average regression would tell you, and sizing off the average leaves edge on the table on exactly the moves where edge is largest. ## Volume flags which moves matter Before you condition on move size, you need to know a move is real and not a one-print artifact. Volume is the cleanest proxy. A large price move on heavy volume is information being absorbed; the same move on a single thin print is a glitch or a spoof clearing, and it will not propagate to B in any reliable way. Weighting or filtering large moves by the volume that accompanied them keeps the lead-lag signal anchored to moves that actually shifted the book, and it is one of the strongest features to fold into the leading statistic. ## Classify the cause: liquidation, impact, or news A big move in A is not one thing. It has a cause, and the cause changes whether the lead-lag follow-through is tradable. Three buckets cover most of crypto's large moves, and you can separate them from data you already have. - **Market impact.** A large directional trade walks the book. You see it as a cluster of aggressive prints in one direction with large cumulative size and no external trigger. This is the cleanest case for lead-lag: real flow in A, the leader, that genuinely pulls B, the laggard, after it. Trade-size data tells you it happened. - **Liquidations.** A forced unwind, visible in the exchange's liquidation feed. These are violent, one-directional, and often mean-revert hard once the cascade exhausts, so the follow-through in B is shorter-lived and the snap-back risk is higher. The liquidation feed is the label; use it to mark these moves and treat their lead-lag signal with a faster exit. - **News.** A scheduled or unscheduled headline repriced A on information, not flow. Trade size around a news move looks different, broad participation rather than one whale, and the timing lines up with a known release. The separator is mechanical. Examine trade size and check the liquidation feed: concentrated aggressive size with no liquidation flag points to impact, a liquidation flag points to forced flow, and a move with broad size at a scheduled time points to news. The old article ["Cross-Exchange Fair Value for Crypto Perps"](https://aligrithm.com/cross-exchange-fair-value-for-crypto-perps/) already layered liquidation and flow signals onto fair value across venues; this is the same data feeding a different decision, whether the leader's move is the kind that B will follow cleanly or the kind that will reverse on you. ## News is a reason to widen, not to trade The lead-lag trade is a maker's trade as often as a taker's, and for a maker, scheduled news flips the logic. You do not want to be quoting tight into a release. The old article ["Using Trade Flow to Predict Short-Term Price Movement"](https://aligrithm.com/using-trade-flow-to-predict-short-term-price-movement/) framed the maker's job as predicting flow; a scheduled news event is the one time you get advance warning that toxic, informed flow is about to arrive on a known timetable. So the news bucket is not primarily a signal to trade, it is a signal to defend. Ahead of a scheduled release, widen quotes or pull them entirely, because the move that is coming will be fast and informed and your resting orders are the liquidity it picks off. The advantage of news over impact and liquidations is precisely that it is on a clock, so unlike the surprise of a whale or a cascade, you can prepare for it before the first print lands. ## The traps, unchanged and sharpened Conditioning on big moves makes the fragile parts of lead-lag worse, not better. Tail samples are small, so the upper-quantile beta is noisier than the average beta and overfits faster; the old lead-lag article's data-mining warning bites harder when you have already thrown away most of your observations to isolate the tail. Liquidation cascades are the regime where the lead can invert, the laggard leading the snap-back, so a beta fit in calm tails can have the wrong sign in a real cascade. And all of it is still a latency race against players reading the same liquidation feed and the same economic calendar. Re-estimate on a rolling window, keep the size threshold honest, and remember that the cleaner the classification looks in backtest, the more you should suspect you labeled the moves with hindsight. ## Visualizing the three causes of a big move ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_308-visual-0.png) ## KEY POINTS - A lead-lag regression on all data fits mostly noise; on the HFT side the edge is in the big moves, so condition the slope on large moves instead of averaging them in. - Get the tail slope two ways: filter to observations where A's move exceeds a size threshold, or use quantile regression at a high tau to fit B's large-move behavior directly. Average and tail betas differ because impact is nonlinear. - Volume flags which moves are real. A large move on heavy volume is information absorbed; the same move on a thin print is a glitch, so weight large moves by accompanying volume. - Classify the cause from data you have: concentrated aggressive trade size means market impact (clean follow-through), a liquidation-feed flag means forced flow (short follow-through, snap-back risk, faster exit), broad size at a scheduled time means news. - For a maker, scheduled news is a reason to widen or pull quotes, not to trade, because it warns of informed flow on a known clock (the old article ["Using Trade Flow to Predict Short-Term Price Movement"](https://aligrithm.com/using-trade-flow-to-predict-short-term-price-movement/)). - Tail conditioning sharpens the fragility: small tail samples overfit faster, leads can invert during liquidation cascades, and it remains a latency race, so re-estimate on a rolling window and distrust clean hindsight labels. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [The Art of Currency Trading - Brent Donnelly (Amazon)](https://www.amazon.com/s?k=The+Art+of+Currency+Trading+Brent+Donnelly&ref=aligrithm.com) ### 5.38 Lead-Lag Done Right: Predict and Manage, Don't Naively Wait URL: https://aligrithm.com/lead-lag-done-right-predict-and-manage-dont-naively-wait/ Last updated: 2026-07-27T13:25:23.000Z The textbook lead-lag trade goes like this. Asset A jumps. You believe B follows A with a delay, so you buy B and close after a fixed five seconds. Simple, mechanical, and wrong in a way that quietly bleeds you. The old article ["Lead-Lag Relationships in Global Markets"](https://aligrithm.com/lead-lag-relationships-in-global-markets/) established the relationship itself, one market moving first and a related market following with a measurable delay, and warned that the lead is the most fragile object in intermarket analysis. This piece is about what you do once you have a lead worth trading, and the answer is not "wait five seconds." The fixed-wait version throws away the two things that actually make the trade work: a number for how far B should move, and a rule for getting out when it does. ## Why the fixed-wait trade is broken A leads B means the regression of B's future return on A's recent return has a real slope at some lag. The old article "Crypto Cross-Exchange Lead-Lag Study" built exactly this on crypto venues, regressing one exchange's returns against another's at a shifted timeframe to find where the slope peaks. That slope is not just a yes/no signal that B will move. It is a magnitude. It tells you roughly how much B should move given the size of A's jump. The naive trade ignores the magnitude entirely. It enters B and exits on a clock, which means three bad outcomes get lumped together. B reaches the predicted move in two seconds and then keeps drifting or reverses, and you sit through the give-back until your timer fires. B reaches the move instantly and you hold for three more seconds of pure noise and adverse selection. B never moves at all because this was one of the times the lead failed, and you still hold to the timer hoping. A clock cannot tell these apart. A target can. ## Predict the move, then manage the exit Use the lead the way it was measured: as a forecast of B's move, not a trigger to wait. $$ \\widehat{\\Delta B} = \\beta \\cdot \\Delta A\_{\\text{recent}} $$ Read it plainly: the predicted move in the lagging asset B equals the lead-lag slope beta times A's recent move. Beta is the peak-lag regression coefficient from the old lead-lag study, the number you already estimated to confirm the lead exists. If A jumps 20 basis points and your fitted beta is 0.6, your forecast is a 12 basis point move in B. That 12 bps is the whole trade. It is your target, and it is what turns a vague "B should follow" into a managed position. Now the management rule, which is the part the naive version skips: - Enter B in the direction of the forecast as soon as A moves. - If B reaches the predicted move before your horizon expires, close immediately and take the edge. You got what the lead promised; there is no reason to keep holding into noise. - If B has not reached the target by the horizon, close anyway, at whatever small profit or loss is there. The lead either misfired or got arbitraged before you, and holding longer is hope, not edge. The horizon still exists. It is a backstop, not the exit logic. Most of your good exits come from hitting the target early, not from the clock. ## Why early exit is the whole point The edge in a lead-lag trade lives in a narrow time window, the lag you measured, and it decays fast on either side of it. Hold past the moment B catches up to A and you are no longer trading the lead. You are holding naked exposure to B's own noise, paying spread and eating adverse selection with no forecast backing you. The early-exit rule cashes the edge exactly when it exists and refuses to pay for time after it is gone. A worked sequence makes the asymmetry obvious. Say your measured edge per fired signal is around 12 bps of predicted B-move, your round-trip cost is 4 bps, and B's noise over the holding window is roughly 8 bps one-sigma. Exit the instant B hits 12 bps and you net about 8 bps before the noise has a chance to drag you back. Sit on a fixed timer instead and that 8 bps of noise is now a coin flip layered on top of your realized move, so half your hit-the-target trades give back several basis points before the clock lets you out. Same signal, same entry, worse exit, and the difference is pure management. The forecast also gates entry size. A big predicted move relative to cost is worth a full clip; a predicted move barely above your round-trip cost is not worth trading at all, no matter how clean the lead looked historically. The fixed-wait trade has no way to express this because it never computes the number. ## The traps that still apply None of this rescues a lead that is not real. The old lead-lag article's three fake-lead mechanisms are all still live here: a shared slow driver hitting both assets with different delays, a real lead that got arbitraged away as venues sped up, and a data-mined lead from scanning too many pairs and lags. A managed exit on a fake lead just loses more efficiently. Re-estimate beta on a rolling window and watch the peak lag migrate toward zero, the same early-decay warning from before, because the day the lag collapses is the day this trade stops paying and starts feeding the faster player on the other side. The other honest limit is that this is a latency race. The cross-exchange version from the old fair-value work, ["Cross-Exchange Fair Value for Crypto Perps,"](https://aligrithm.com/cross-exchange-fair-value-for-crypto-perps/) is the same physics: the leader's move is public, and everyone with a colocated box sees it too. Your forecast of B is only tradable in the slice of time between A moving and B catching up, and if that slice is shorter than your reaction time, the predicted move is real and still unreachable. Measure your own latency against the lag before you assume the edge is yours. ## Visualizing predict-and-manage vs naive-wait ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_307-visual-0.png) ## KEY POINTS - The naive lead-lag trade (enter B after A moves, exit on a fixed timer) wastes the two things that make the trade work: a magnitude for B's expected move and a rule to exit when it is reached. - The lead-lag slope you already fit (the old article "Crypto Cross-Exchange Lead-Lag Study") is a forecast, not just a trigger. Predicted B-move equals beta times A's recent move, so a 20 bps A-jump with beta 0.6 forecasts a 12 bps B-move. - Manage the exit: close as soon as B reaches the predicted move, and close at the horizon otherwise at whatever small profit or loss exists. The horizon is a backstop, not the exit logic. - Early exit cashes the edge inside the narrow lag window where it exists; holding past B catching up to A is naked exposure to B's noise with no forecast and full costs. - The forecast also gates size and entry: a predicted move barely above round-trip cost is not worth trading, a fact the fixed-wait version cannot express because it never computes the number. - The fragility from the old article ["Lead-Lag Relationships in Global Markets"](https://aligrithm.com/lead-lag-relationships-in-global-markets/) is unchanged: shared-driver fakes, arbitraged-away leads, and data-mined leads, plus it is a latency race, so re-estimate beta and check the lag against your own reaction time. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [The Art of Currency Trading - Brent Donnelly (Amazon)](https://www.amazon.com/s?k=The+Art+of+Currency+Trading+Brent+Donnelly&ref=aligrithm.com) ### 5.36 Reconstructing the Order Book from Incremental Deltas URL: https://aligrithm.com/reconstructing-the-order-book-from-incremental-deltas/ Last updated: 2026-07-27T13:25:22.000Z Every microstructure feature in this pillar assumes you have the order book. Order book imbalance from the old article ["Order Book Imbalance: The First Microstructure Feature to Test,"](https://aligrithm.com/order-book-imbalance-the-first-microstructure-feature-to-test/) the density placement, the spoofing detection, the market-impact walk, all of them read the full state of resting liquidity at each moment. None of it works until you can produce that state, and the exchange does not hand it to you. It hands you a snapshot once and then a firehose of changes, and you have to fold those changes back into a live book yourself. This is the unglamorous engine under the entire pillar, and the notebook from the old article "Order Book Imbalance Backtest" silently depends on it. ## What the exchange actually sends You get two things on an L2 feed. First, a snapshot: the full book at one instant, every price level with its resting size, sent once when you connect. Second, an endless stream of incremental deltas: tiny messages each saying "at this price, the size is now this," or "this price level is gone." The exchange does not resend the whole book on every change because that would be enormous and slow. It sends the diff, and reconstructing the book means starting from the snapshot and applying every delta in order. A delta is one of three operations at a price level. An insert adds a level that did not exist. An update changes the size at a level that does. A delete removes a level whose size went to zero. Critically, a delta gives you the new absolute size at that price, or tells you to remove it, it is not a relative change you add, it is a replacement. Apply it by overwriting whatever you had at that price. $$ \\text{book}\_t = \\text{snapshot}\_0 \\;\\oplus\\; \\delta\_1 \\oplus \\delta\_2 \\oplus \\dots \\oplus \\delta\_t $$ Read it as: the book at time t is the initial snapshot with every delta folded in, in sequence, where each fold either sets a price level to a new size or removes it. The book is the snapshot plus the running sum of changes, and any feature you compute reads off book at t. ## The data structure: a hashmap and a sorted tree The naive implementation is a list of levels you scan and edit. It is too slow, because deltas arrive thousands per second and each one would force a linear search. You need two structures working together, one for finding a level fast and one for keeping levels sorted. The hashmap maps price to size. When a delta lands for a price, you look it up in O(1), overwrite the size, or delete the key. That handles the update itself at constant cost no matter how deep the book. But a hashmap has no order, and every feature you want, the best bid and ask, the top-N levels for imbalance, the level-by-level walk for impact, needs the prices sorted. So you pair the hashmap with a sorted structure, a balanced binary search tree, a B-tree, or a red-black tree, keyed by price. The tree keeps the levels in price order so the best bid is the max of the bid tree and the best ask is the min of the ask tree, both reachable in logarithmic time, and an in-order traversal gives you the top levels for any depth-based feature. The split is deliberate: the hashmap gives constant-time edits, the tree gives ordered access, and you maintain both on every delta. Insert a new price: add to the hashmap and the tree. Update a size: overwrite in the hashmap, and if the price already existed the tree needs no reorder. Delete a level: remove from both. Bid and ask sides are two separate pairs of structures, mirror images. ## Sequence numbers are the whole game Reconstruction is correct only if you apply deltas in exactly the order the exchange produced them, with none missing. Feeds carry sequence numbers for this reason: each message is stamped, and consecutive messages must have consecutive numbers. If you receive sequence 1004 right after 1002, you dropped 1003, and your book is now wrong, silently, because you applied a change to a state that had not received its predecessor. From that point your imbalance, your spreads, your everything are computed on a corrupted book. The discipline is mechanical. Track the last sequence number you applied. If the next message is not exactly one higher, you have a gap, and the only safe response is to discard your book, request a fresh snapshot, and resynchronize. Do not paper over a gap by applying the out-of-order message; a book that is quietly wrong is worse than no book, because it produces confident garbage features that poison every model downstream. This is the single most common way a reconstruction engine fails in production, and it fails without an error message unless you check the sequence. ## Where it sits and how to build it The reconstruction engine is the prerequisite for the whole pillar, which is why this is a Pillar 7 engineering piece rather than a strategy one. It produces the book state that the imbalance feature, the density placement, and the impact walk all consume. Build it as its own component, not tangled into your strategy code, because it runs hot, thousands of deltas a second, and it benefits from being written in a fast language while your strategy logic stays in something slower and more flexible. A common architecture splits exactly here: a performant data server folds the deltas and maintains the book, then hands the clean state or the precomputed features to a Python strategy, keeping the high-frequency churn out of the slow language. A worked sanity check before you trust it: reconstruct the book from a feed, and independently subscribe to the exchange's periodic full snapshots if it offers them. Every time a fresh snapshot arrives, compare it to your reconstructed state. They must match exactly. If they drift, you are dropping or misordering deltas, and the snapshot comparison is the only way you will catch it, because the reconstruction itself never complains. ## Visualizing the reconstruction ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_305-visual-0.png) ## KEY POINTS - Exchanges send a one-time full snapshot then a firehose of incremental deltas; the live book is the snapshot with every delta folded in, in order. - A delta gives the new absolute size at a price (insert, update, or delete), not a relative change, so you apply it by overwriting or removing that level. - Use two paired structures: a hashmap (price to size) for constant-time edits, and a sorted tree (B-tree or red-black) for ordered access to best bid/ask and top-N levels. Maintain both on every delta, one pair per side. - Sequence numbers are the whole game. A gap means you dropped a message and your book is silently wrong; the only safe response is to discard, request a fresh snapshot, and resync. - A quietly-wrong book is worse than no book because it produces confident garbage features that poison every downstream model. - Build it as a separate hot component (often a fast-language data server feeding a slower strategy), and validate by comparing your reconstruction against the exchange's periodic snapshots, they must match exactly. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [The Art of Currency Trading - Brent Donnelly (Amazon)](https://www.amazon.com/s?k=The+Art+of+Currency+Trading+Brent+Donnelly&ref=aligrithm.com) ### 5.35 Market Impact and the Square-Root Law: Walking the Book to Price Your Slippage URL: https://aligrithm.com/market-impact-and-the-square-root-law-walking-the-book-to-price-your-slippage/ Last updated: 2026-07-27T13:25:21.000Z The old article ["Why Transaction Costs Should Be Added Before You Fall in Love"](https://aligrithm.com/why-transaction-costs-should-be-added-before-you-fall-in-love/) listed market impact as one of the five cost categories and warned that a strategy with a pre-cost Sharpe of 2.4 can collapse to negative once thin-instrument costs land. It named the cost. It did not tell you how to measure it. This article does, because impact is the one cost component you cannot read off a fee schedule, you have to compute it from the book by simulating what your order would actually do. Impact is what happens when your order is bigger than the liquidity at the best price. You do not get filled at the mid. You eat the best level, then the next, then the next, paying a worse price at each step, and the average price you end up with is worse than where the market started. That gap is your slippage, and it grows with size in a specific, measurable shape. ## Walk the book level by level The measurement is a simulation against a real book snapshot. Take the order book at a timestamp, grab the top levels, sort by price, and find the mid. Then pretend to execute your order: consume the size at the best price, and if your order is larger, move to the next level and consume that, continuing until your full size is filled. You are walking the book, removing liquidity level by level the way a real market order would. Two numbers come out. The average execution price is the size-weighted average of every level you ate, total cash paid divided by total quantity filled. The slippage is how far that average sits from the mid you started at, expressed in basis points. $$ \\text{avg price} = \\frac{\\sum\_i q\_i \\cdot p\_i}{\\sum\_i q\_i} \\qquad \\text{slippage (bps)} = \\frac{|\\,\\text{avg price} - \\text{mid}\\,|}{\\text{mid}} \\times 10{,}000 $$ Read the first as: for each level you consumed, multiply the quantity taken there by that level's price, sum across all levels, divide by total quantity, and you have your blended fill price. Read the second as: take the absolute distance from that blended price to the starting mid, divide by mid to get a fraction, multiply by ten thousand to put it in basis points. A 50-cent slip on a 100-dollar mid is 0.50 / 100 × 10,000 = 50 bps. That is the cost of clearing the book to your size, and it is invisible until you walk it. ## The square-root law Run the walk for many sizes and plot slippage against size. The curve is not a straight line, big orders do not cost proportionally more than small ones, they cost less per unit than a linear model would predict. The shape that fits is a power law, and the exponent lands near one-half, which is why it is called the square-root law of market impact. $$ \\text{impact}(x) = a \\cdot x^{\\,b}, \\qquad b \\approx 0.5 $$ Read it as: impact scales with order size x raised to a power b, where b comes out around 0.5, so doubling your size multiplies your impact by roughly the square root of two, about 1.41, not by two. The constant a is the instrument's sensitivity, thin books have a large a, deep liquid books a small one. You fit a and b by running the book-walk at a range of sizes to get empirical impact points, then regressing log-impact on log-size, the slope is b and the intercept gives a. The square-root shape has a practical payoff: it lets you extrapolate. You can only walk the book as deep as your snapshot has levels, so for sizes larger than the visible book you have no empirical point. The fitted power law extends past the data, giving you an impact estimate for sizes bigger than anything the book shows, which is exactly what you need for capacity planning, the cost-versus-volume relationship the transaction-cost article said you must model. ## Tying it to fill probability Impact and fill probability are the same book read from two directions, which is why this extends the old article ["Fill Probability from Trade Size CDFs."](https://aligrithm.com/fill-probability-from-trade-size-cdfs/) That article asked, for a resting quote, how likely a taker order is big enough to reach you, and built the answer from the distribution of taker sizes. Impact asks the mirror question: when you are the taker, how far up the book does your own size reach, and what does it cost. Both come from the same level-by-level structure of the book; one prices your passive fills, the other prices your aggressive ones. For a maker this matters because you do both. You quote passively most of the time, but when inventory gets dangerous you cross to flatten, and the impact curve tells you what that exit costs. A position you cannot exit without paying 40 bps of impact is more toxic than its markout alone suggests, because the cost of getting out is part of the risk. ## A worked number Take an alt at a 100.00 mid with 2 BTC resting at 100.05, 3 BTC at 100.12, and 5 BTC at 100.20\. You need to buy 4 BTC. You eat all 2 at 100.05 and 2 of the 3 at 100.12\. Your average price is (2·100.05 + 2·100.12) / 4 = 100.085\. Slippage is |100.085 − 100.00| / 100.00 × 10,000 = 8.5 bps. Now walk 8 BTC: 2 at 100.05, 3 at 100.12, 3 at 100.20, average (200.10 + 300.36 + 300.60) / 8 = 100.1325, slippage 13.25 bps. Doubling size from 4 to 8 BTC raised impact from 8.5 to 13.25 bps, a factor of about 1.56, in the neighborhood of the square-root's 1.41 rather than a linear 2.0\. Fit a·x^b through these and more points and you have a function that prices any size, including ones deeper than the book you can see. ## Cautions The book-walk assumes the displayed book is the book, which the spoofing thread throughout this pillar says is dangerous in crypto, size that vanishes when you reach for it makes your simulated impact optimistic. It also assumes you take everything instantly in one sweep; spreading the order over time changes the cost, usually lowering it, because the book refills between your child orders, so the single-sweep walk is a worst-case snapshot, not the cost of a patient execution. And the fitted a is non-stationary, it tracks current liquidity, so a square-root law fit last week understates impact in this week's thinner book. Refit on recent snapshots, and treat the extrapolated tail as a planning estimate, not a promise. ## Visualizing market impact ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_304-visual-0.png) ## KEY POINTS - Market impact is the cost you cannot read off a fee schedule; you compute it by walking the book, simulating your order eating liquidity level by level until filled. - Two outputs: the size-weighted average execution price, and slippage = absolute distance from the starting mid over mid, times ten thousand, in basis points. - Plot slippage against size and it bends: impact follows a power law a·x^b with b near 0.5, so doubling size multiplies impact by about 1.41, not 2\. Fit a and b by regressing log-impact on log-size. - The fitted square-root law lets you extrapolate impact for sizes deeper than the visible book, which is what capacity and cost-versus-volume planning need. - Impact is the mirror of fill probability: same book, read as how far your aggressive order reaches instead of whether a taker reaches your passive quote. A maker needs both because crossing to flatten toxic inventory pays the impact cost. - The walk assumes the displayed book is real (spoofing makes it optimistic) and assumes a single instant sweep (patient execution costs less as the book refills). The constant a is non-stationary, so refit on recent snapshots. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [The Art of Currency Trading - Brent Donnelly (Amazon)](https://www.amazon.com/s?k=The+Art+of+Currency+Trading+Brent+Donnelly&ref=aligrithm.com) ### 5.33 Positional Market Making and the Thousands-of-Alphas Ensemble URL: https://aligrithm.com/positional-market-making-and-the-thousands-of-alphas-ensemble/ Last updated: 2026-07-27T13:25:20.000Z Ask where a large market maker's money comes from and the textbook answer is the spread: post a bid, post an ask, collect the difference, repeat. That answer describes maybe the smallest part of the business. The bulk of the PnL at a serious desk comes from positional market making, using the quoting machine as a delivery mechanism for medium-frequency alpha, and running thousands of small skews at once rather than a handful of clever ones. This is the synthesis of two threads. The old article ["Why Small Alphas Matter More for Makers Than Takers"](https://aligrithm.com/why-small-alphas-matter-more-for-makers-than-takers/) established that a signal worth half a basis point is garbage to a taker and money to a maker, because the maker collects the spread instead of paying it. The old article ["Maker vs Taker Edge: Same Signal, Different Economics"](https://aligrithm.com/maker-vs-taker-edge-same-signal-different-economics/) showed the same signal becomes two different businesses depending on which side of the spread you sit. This article is what you build once you accept both: an ensemble of weak alphas skewed into passive quotes across a wide universe. ## Market making as a medium-frequency alpha delivery system Reframe the quote. A maker's fair value is the center, skew leans it toward where you want inventory, and spread sets the width. Skew is usually described as inventory management, but it is also a steering wheel: skewing your quotes toward the long side accumulates a long position passively, by being filled rather than by crossing the spread. So any signal that says "this name should go up" can be expressed as a skew, and you end up long without ever paying to take. That is positional market making, market making bent into a medium-frequency alpha. You are not scalping the spread in milliseconds; you are using passive fills to build positions a signal wants you to hold for seconds to minutes, collecting spread on the way in instead of paying it. The HFT spread-scalping book still runs, but the positional layer on top is where the size and the PnL live. ## Why zero cost changes everything The hinge is transaction cost, and for a maker it is effectively zero or negative, because you earn the spread rather than pay it. That single fact rewrites which signals are tradeable. A taker contemplating a half-basis-point signal does the math: 0.5 bps of edge against 4 bps of round-trip cost is a guaranteed loser, so the signal is worthless and gets thrown away. The maker does the same math with the cost flipped to income: the half-bp signal sits on top of captured spread, so the break-even bar drops to the floor and the signal only has to beat the adverse selection it offsets. Signals that are trash to everyone paying costs are tradeable to the one party collecting them. This is why expanding the universe helps so much. The resource puts it directly: zero cost to trade makes it easy to find edge, and pushing into a larger, less-efficient universe makes the same alphas perform better, because small caps carry more edge. Liquid large caps are picked clean; the thin, awkward, illiquid names are where weak signals survive, and a maker is exactly the participant who can quote them, because the wide spread there is income, not an obstacle. The old maker-vs-taker article made this point: cost pulls the maker toward illiquid wide-spread names while pushing the taker toward liquid tight ones. ## The ensemble: thousands of skews One half-bp alpha is not a business. The move is to find many of them and skew into all of them at once, ensembling the skews into the quote. The reason this works is the same noise-cancellation logic that powers signal ensembling generally: each weak alpha is true signal plus noise, the true components correlate with future returns while the noise components do not, so summing many of them grows the signal and averages down the noise. None of the individual alphas survives standalone, but stacked into one skewed quote they compound into the bulk of the PnL. A desk running a thousand skews is not running a thousand strategies; it is running one ensembled fair-value lean built from a thousand weak inputs. A concrete demonstration the resource suggests: backtest a plain ranking strategy on the log of past return times minus one, the one-hour reversal, with no fees. With costs it dies; without costs it works. Now imagine stacking dozens of effects like it, each individually marginal, all expressed as passive skews. That picture is how the positional PnL is generated, and the maker's zero-cost seat is the only one from which the picture is profitable. $$ \\text{skew}\_{\\text{total}} = \\sum\_{k=1}^{N} \\alpha\_k, \\qquad \\text{net edge}\_k = \\underbrace{\\alpha\_k}\_{\\text{signal}} + \\underbrace{s}\_{\\text{spread captured}} - \\underbrace{a\_k}\_{\\text{adverse selection}} $$ Read the first sum as the total inventory lean being the sum of many small alpha skews. Read the second as a maker's per-alpha economics: the edge is the raw signal plus the spread you collect minus the adverse selection you pay, and because the spread term is positive income rather than negative cost, even a tiny alpha clears the bar. Stack N of them and the signal grows while the noise in the skews cancels. ## A worked number Say you have one momentum alpha worth 0.5 bps of edge per fill. Standalone, against a taker's 4 bps cost, it loses 3.5 bps a round trip, dead on arrival. As a maker capturing a 3 bps half-spread, that same alpha nets 0.5 + 3 − (some adverse selection); if adverse selection runs 2 bps, you net 1.5 bps per fill on a signal a taker had to discard. Now find forty more uncorrelated half-bp alphas. The signal in the combined skew grows roughly with the number of alphas while the noise grows with its square root, so forty alphas give you something like a six-to-seven-fold improvement in signal-to-noise over one, turning a pile of individually-marginal effects into a robust positional lean. The catch the small-alpha article flagged: you swapped explicit spread and fee cost for implicit adverse-selection cost, so if your defense, fair value and skew against toxic loading, is wrong, the adverse-selection term blows up and eats all forty alphas at once. ## Where it sits Positional market making is the bridge between the microstructure pillar and portfolio construction, which is why this article straddles Pillar 5 and 6\. Each alpha is a microstructure or cross-sectional signal; the ensembling and universe expansion are portfolio decisions. The practical build order: get fair value right first because everything skews around it, get the toxic-flow defense right because it caps the adverse-selection term, then start bolting on weak alphas one at a time and let the ensemble do the work. Do not chase one brilliant signal. Collect many mediocre ones and let zero cost and the law of large numbers turn them into a business. ## Visualizing the ensemble ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_302-visual-0.png) ## KEY POINTS - Most of a large maker's PnL is positional, not spread-scalping: using passive fills to build positions a medium-frequency signal wants to hold, collecting spread on the way in instead of paying it. - Skew is a steering wheel, not just inventory management. Any "this should go up" signal becomes a skew toward the long side, accumulating the position passively. - Zero (or negative) transaction cost rewrites which signals are tradeable. A half-bp alpha is a guaranteed loser to a taker paying 4 bps and a winner to a maker collecting spread, so the break-even bar drops to the floor. - Expand the universe into illiquid small caps, where weak alphas survive and the wide spread is income. The maker is the only participant who can profitably quote those names. - Ensemble thousands of weak skews: each is signal plus noise, the signals stack and the noise averages down, so individually-marginal alphas compound into the bulk of PnL. Signal grows with N, noise with root N. - You traded explicit cost for implicit adverse-selection cost, so get fair value and toxic-flow defense right first, a broken defense blows up the adverse-selection term and eats every alpha at once. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [The Art of Currency Trading - Brent Donnelly (Amazon)](https://www.amazon.com/s?k=The+Art+of+Currency+Trading+Brent+Donnelly&ref=aligrithm.com) ### 5.32 Volatility-Regime Quoting: Discrete Steps vs Continuous Widths URL: https://aligrithm.com/volatility-regime-quoting-discrete-steps-vs-continuous-widths/ Last updated: 2026-07-27T13:25:20.000Z The old article ["Spread Widening During Volatility Expansion"](https://aligrithm.com/spread-widening-during-volatility-expansion/) set the principle: your spread has to track volatility, because a markup that is safe in a calm tape is a gift to informed traders in a moving one. Principle is easy. The implementation question nobody resolves cleanly is the mapping itself. You have a volatility number coming in and a spread width going out, and the function connecting them is a design choice with real consequences. Get it crude and you either over-quote and miss fills or under-quote and bleed to adverse selection at the regime boundary. There are two ways to build that mapping. The discrete one buckets volatility into a few regimes and assigns each a fixed width. The continuous one interpolates between those buckets so the width moves smoothly. Both start from the same place, and the second is a refinement of the first, not a different idea. ## Step one: define the regimes by quantile Do not pick volatility thresholds by eye. Use quantiles of the volatility distribution itself, which makes the regimes self-calibrating to whatever instrument and period you are in. The simplest cut is three regimes by tertile: the bottom third of volatility readings is the calm regime, the middle third is normal, the top third is stressed. You can also cut at extreme quantiles, the 0.9 say, if you only want to react to genuine tail volatility and quote a constant width the rest of the time. The quantile approach matters because volatility is non-stationary, the same absolute volatility number is calm in one month and stressed in another, so a fixed numeric threshold rots. Quantiles ride the distribution. Re-estimate the quantile boundaries on a rolling window and your regime labels stay honest as the market's overall volatility drifts. ## The discrete model: three regimes, three spreads The blunt version assigns one spread width to each regime and switches between them. $$ \\text{spread} = \\begin{cases} w\_1 & \\text{if } \\sigma \\le q\_{1/3} \\quad (\\text{calm})\\\\\[4pt\] w\_2 & \\text{if } q\_{1/3} < \\sigma \\le q\_{2/3} \\quad (\\text{normal})\\\\\[4pt\] w\_3 & \\text{if } \\sigma > q\_{2/3} \\quad (\\text{stressed}) \\end{cases} $$ Read it as a lookup: measure current volatility sigma, see which quantile band it falls in, quote the width for that band, with the three widths hand-set so w1 is tightest and w3 widest. It is trivial to build and trivial to reason about, which is why it is the right first version. The flaw is the cliff at the boundary. The moment volatility ticks from the 66th to the 67th percentile, your spread jumps discontinuously from w2 to w3\. Nothing real happened, volatility moved a hair, but your quote lurched, which churns your orders (every boundary crossing is a requote, eating rate limits) and makes your fill behavior jumpy right where the market is already getting jumpy. With three regimes you have two cliffs; with finer buckets you have more, each a little discontinuity the market can catch you on. ## The continuous model: interpolate the widths The fix is to treat the three regime widths as anchor points and draw a smooth curve through them, so the spread moves continuously with volatility instead of stepping. The simplest curve is linear interpolation between adjacent anchors. $$ w = \\frac{w\_0 (x\_1 - x) + w\_1 (x - x\_0)}{x\_1 - x\_0} $$ Read it as a weighted blend: x is your current volatility sitting between two anchor points x0 and x1 whose widths are w0 and w1, and the formula returns a width that is w0 when x sits on the lower anchor, w1 when it sits on the upper, and a proportional mix in between. Apply it on the calm-to-normal interval for low volatility and on the normal-to-stressed interval for high volatility, and the cliffs vanish. The spread now glides from w1 toward w3 as volatility climbs, with no discontinuity to churn your book. Linear is the floor, not the ceiling. The resource notes you can run the same interpolation with a polynomial, a spline, or a sigmoid through the anchors. The shape encodes a view about how spread should respond: a sigmoid stays flat through normal volatility and then ramps hard into the tail, which matches the intuition that adverse selection is benign until volatility gets genuinely large and then bites nonlinearly. A spline gives you a smooth curve obeying all three anchors at once instead of two straight segments meeting at a kink. Pick the shape, then let the three quantile anchors pin it to the data. ## A worked number Run three tertile regimes on an alt perp with hand-set half-widths of 4, 8, and 16 basis points for calm, normal, and stressed, with the regime anchors at volatility readings of 10, 20, and 35 bps of fair-price standard deviation. Current volatility prints 27 bps, sitting between the normal anchor (20) and the stressed anchor (35). The discrete model still calls this "normal" until volatility crosses 35, so it quotes 8 bps, half the width the move deserves, and you eat adverse selection on the way up. The linear continuous model interpolates: w = (8·(35−27) + 16·(27−20)) / (35−20) = (64 + 112) / 15 ≈ 11.7 bps. The continuous version already widened you most of the way toward stressed because volatility is most of the way there, instead of waiting for a threshold to trip. ## Which to use Build the discrete model first, because it is the honest baseline and lets you sanity-check that wider volatility maps to wider spreads at all. Move to continuous interpolation once the boundary churn shows up in your requote counts or your markouts spike right at regime crossings. Choose the interpolation shape by what you believe about the tail: linear if you have no strong view, sigmoid if you think the danger is concentrated in extreme volatility, spline if you want a single smooth fit through all anchors. Calibrate the widths the only way that means anything: markouts conditional on regime, the truth serum from the old article ["Markouts: The Truth Serum of Market Making."](https://aligrithm.com/markouts-the-truth-serum-of-market-making/) Set each width so the markout in that regime is flat, not negative; if your stressed-regime markouts still trough, w3 is too tight regardless of how clever the interpolation is. The whole apparatus sits on top of the fair value from the old article ["Why Fair Value Is the Core of Market Making,"](https://aligrithm.com/why-fair-value-is-the-core-of-market-making/) because spread is the second decision after the center, and a beautifully interpolated spread around a wrong center is still a losing quote. ## Visualizing discrete vs continuous ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_301-visual-0.png) ## KEY POINTS - Spread must track volatility; the open question is the mapping from a volatility number to a width, and a crude mapping costs you fills or markouts at the regime boundary. - Define regimes by quantiles of the volatility distribution, not fixed numeric thresholds, because volatility is non-stationary and a fixed threshold rots. Re-estimate the quantile cuts on a rolling window. - Discrete model: three quantile regimes, three hand-set widths, a lookup. Trivial to build and the right first version, but it has a cliff at each boundary that churns your book and jumps your quote when volatility barely moved. - Continuous model: treat the three widths as anchors and interpolate between them so the spread glides. Linear is the floor; polynomial, spline, or sigmoid encode a view about how spread should respond into the tail. - Worked size: with widths 4/8/16 bps at volatility anchors 10/20/35, a 27 bps reading still gets the discrete "normal" 8 bps while linear interpolation already quotes about 11.7 bps, widening with the move instead of waiting for a threshold. - Calibrate widths with markouts conditional on regime, set each so its markout is flat, and remember the spread sits on top of fair value, a clever spread around a wrong center still loses. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [The Art of Currency Trading - Brent Donnelly (Amazon)](https://www.amazon.com/s?k=The+Art+of+Currency+Trading+Brent+Donnelly&ref=aligrithm.com) ### 5.31 Limit Order Book Behavior: Negative Spreads, Wipeouts, and Why Size Tightens the Market URL: https://aligrithm.com/limit-order-book-behavior-negative-spreads-wipeouts-and-why-size-tightens-the-market/ Last updated: 2026-07-27T13:25:19.000Z The bid-ask spread is the first thing you read off a book and the most abused. Most people treat it as a clean number that goes up when things get scary and down when they calm, then build features on top of that cartoon. Crypto books do three things that break the cartoon: the spread sometimes goes negative, it blows out when a single large order eats through levels, and it gets tighter, not wider, when there is more size resting. Each one tells you something about the matching engine and the participants underneath, and a maker who does not account for them quotes against a fiction. Start with the definition, because the quirks all come from it. The best bid is the first order you hit on the buy side, the best ask the first on the sell side, and the spread is the gap between them. That is all it is, the distance between the two innermost resting orders. Every odd behavior below is what happens when that distance does something a textbook says it cannot. ## Negative spreads: the engine is behind A negative spread means the best bid is higher than the best ask, someone is willing to buy at a price above where someone else is willing to sell. In a sane market that cannot persist for a microsecond, because those two orders should cross and trade instantly. Yet crypto L2 data shows negative spreads, and you have to know why before you let one into a feature. Two causes. The first is matching-engine lag: the engine has not caught up to the book yet. The crossing orders exist in the data you received, but the engine has not processed the match that will annihilate them, so for a brief window your snapshot shows a state that is about to be resolved. The second is plain data error, dropped messages, out-of-order deltas, a feed that reconstructed the book wrong. Either way a negative spread in your data is not an arbitrage waiting for you. It is a measurement artifact, and a model that ingests it as a real price relationship will generate phantom signals. Filter negative spreads out or flag them; do not feed them to a fair-value model as if the book were genuinely crossed in your favor. ## Wipeouts: where the spread blowout comes from The second behavior is the spread suddenly widening, and the cause is mechanical. A large market order does not stop at the best level. It eats the first level, then the next, then the next, walking up the book until its size is exhausted. After it clears several levels, the new best ask is wherever the order ran out, far from where it started, and the spread is now wide because the inner levels are gone. This is the same level-walking that drives the square-root market impact law, and it is the dynamic behind the "disappearance of size is a price-moving event" point from the old article ["Spoofing, Sturdy Liquidity, and Book Pressure."](https://aligrithm.com/spoofing-sturdy-liquidity-and-book-pressure/) A wipeout is not the market deciding to be volatile in the abstract. It is one participant removing the resting liquidity that was holding the spread tight, and the widened spread is the crater left behind. For a maker this matters in two directions: your own quotes inside that range get filled as the order walks through you, and the post-wipeout wide spread is a transient, the book refills, so reading the blown-out spread as the new normal and quoting that wide leaves money on the table while liquidity comes back. ## Why more size means a tighter spread The third behavior is the counterintuitive one: the more size in the book, the smaller the spread. Naively you might think a big book and a wide spread go together, lots of size means lots of caution. The opposite holds, and it falls out of competition between makers. $$ \\text{size resting} \\uparrow \\;\\Longrightarrow\\; \\text{spread} \\downarrow $$ When many makers want to provide liquidity on a name, they queue up, and the way you get to the front of the queue is to quote a better price, a higher bid or a lower ask, than the maker beside you. That competition compresses the gap between best bid and best ask. A deep book is a crowded book, and a crowded book is a competitive one, so the spread tightens. A thin book is the reverse: few makers, no one undercutting anyone, so the lone maker present charges a wide spread because nothing forces them tighter. The old article ["How to Use Order Book Density for Better Limit Orders"](https://aligrithm.com/how-to-use-order-book-density-for-better-limit-orders/) treated density as a landscape of walls and gaps that changes fill speed and toxicity; this is the price-level consequence of that density, dense means tight, thin means wide. This also tells you which markets to make. Liquid, deep books offer thin spreads but fierce competition for the queue; thin books offer fat spreads but you carry the inventory alone with no one to share the flow, which is exactly the volatility-liquidity tradeoff a maker has to price. ## A worked number Take an alt at the best bid 100.00 and best ask 100.10, a 10 bps spread, with about 5 BTC resting in the top few levels each side. A market buy of 8 BTC arrives. It clears the 100.10 level, the 100.13 level, the 100.17 level, and stops at 100.22 where its size runs out. The new best ask is 100.22, the bid is still 100.00, and the spread just went from 10 bps to 22 bps on a single order, a wipeout, not a volatility regime change. Seconds later, makers seeing the gap and the now-attractive wide spread repost inside it, size rebuilds toward its old 5 BTC, and the spread compresses back toward 10 bps as the queue refills and competition resumes. A feature that logged the 22 bps reading as the prevailing spread, instead of as a transient wipeout that more size will heal, mis-prices the next several quotes. ## Visualizing the three behaviors ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_300-visual-0.png) ## KEY POINTS - The spread is only the gap between the two innermost resting orders, the best bid and best ask. Every odd behavior comes from that definition, not from some separate volatility dial. - Negative spreads in crypto data are artifacts, either the matching engine has not processed the crossing orders yet, or the feed is wrong. Filter or flag them; do not feed a "crossed in my favor" book to a fair-value model. - Spread blowouts come from wipeouts: a large market order walks up through multiple levels, and the widened spread is the crater of removed liquidity, not a regime change. It is transient and refills, so do not quote the blown-out width as the new normal. - More resting size means a tighter spread, because a deep book is a crowded book and makers compete for the queue by undercutting each other. A thin book is wide because no one forces the lone maker tighter. - Worked size: an 8 BTC buy clearing three levels can push a 10 bps spread to 22 bps on one order, then compress back toward 10 bps as makers repost; a feature that logs 22 bps as prevailing mis-prices the next quotes. - Read deep-tight versus thin-wide as the volatility-liquidity tradeoff: liquid books pay thin spreads with queue competition, thin books pay fat spreads but you hold the inventory alone. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [The Art of Currency Trading - Brent Donnelly (Amazon)](https://www.amazon.com/s?k=The+Art+of+Currency+Trading+Brent+Donnelly&ref=aligrithm.com) ### 5.30 Order-Flow Autocorrelation: Why Buys Follow Buys URL: https://aligrithm.com/order-flow-autocorrelation-why-buys-follow-buys/ Last updated: 2026-07-27T13:25:19.000Z Sign every market order, plus for a taker buy, minus for a taker sell, and plot the series. You do not get random noise around zero. You get long stretches sitting near zero punctuated by sudden clusters of same-signed prints: a run of buys, then quiet, then a run of sells. Market order flow is autocorrelated, and the cleanest model for it is an AR(1) process, the same first-order autoregression that shows up everywhere a value depends mostly on its own immediate past. The old article ["Using Trade Flow to Predict Short-Term Price Movement"](https://aligrithm.com/using-trade-flow-to-predict-short-term-price-movement/) leaned on this fact without dwelling on it: trade flow predicts because market orders are roughly AR(1), so net signed flow over a window carries forward. This article is about why that is true, what the AR(1) structure actually implies, and the one trap that makes you systematically underestimate how long a run will last. ## Why the flow clusters The mechanics are not mysterious. A large order rarely executes in one print. A fund unwinding size slices it into children, so one parent buy becomes twenty taker buys in a row. Momentum chasers pile onto a move that is already happening, adding buys on top of buys. And the makers absorbing all of it react: when a maker gets lifted and goes short inventory, the old article ["Order Book Imbalance: The First Microstructure Feature to Test"](https://aligrithm.com/order-book-imbalance-the-first-microstructure-feature-to-test/) described how a thinned ask becomes the path of least resistance, so the next taker finds it even easier to buy. Each buy makes the following buy slightly more likely. That feedback is the autocorrelation. The shape is distinctive: the flow clusters at a low value most of the time, then spikes. It is not a gentle sine wave of buying and selling. It is long quiet flat with occasional violent same-signed bursts, which matters because the bursts are where the price moves and where your inventory gets dangerous. ## The AR(1) model and what it says Write signed flow as a value that is mostly a fraction of its previous value plus a fresh shock. $$ m\_t = \\phi \\, m\_{t-1} + \\varepsilon\_t, \\qquad \\varepsilon\_t \\sim N(0, \\sigma^2), \\qquad 0 < \\phi < 1 $$ Read it as: this instant's flow equals phi times last instant's flow plus random noise. The phi is the memory coefficient. At phi near zero the flow is memoryless and yesterday tells you nothing; at phi near one the flow has long memory and a buy run takes a long time to die. The autocorrelation at lag k is phi to the power k, so a phi of 0.8 means flow one step ahead is still 80 percent correlated, two steps 64 percent, and the influence decays geometrically. For order flow phi is positive and often high at the sub-second scale, which is the formal statement of "buys follow buys." ## The trap: OLS underestimates the memory Here is the part that costs people money, and it is subtle. When you estimate phi by regressing the flow on its own lag with ordinary least squares, the estimate is biased low. The bias is structural, not a sample-size accident. The reason is that in an autocorrelated series the regressor and the error term are entangled. Both m\_t and m\_{t-1} contain the same past shocks, epsilon at t-1, t-2, and so on, so the numerator and denominator of the OLS slope estimate share error terms. That shared dependence drags the estimated slope below the true one. In the positive-autocorrelation case the bias is negative, so your fitted phi comes out smaller than reality. The consequence is direct and dangerous for a maker. Your model thinks there is structurally less memory in the flow than there really is. It expects a buy run to decay faster than it does, so it expects the pressure to fade, the price to settle, and your short inventory to become safe sooner than it will. You lean back into quoting offers too early, get filled again into the run that has not actually finished, and your inventory keeps climbing. The model underestimates both how long the run lasts and how far it can push, because the same bias that shrinks phi also makes the model underestimate the variance of the move. ## The practical rule The actionable takeaway needs no estimation at all: when you see a cluster of buy taker orders coming through, do not post sell limits into it. The resource says it plainly, and it is the entire trade. Selling into a buy-taker run hands those takers exactly the liquidity they want and leaves you holding the inventory while the run keeps going against you, the worst kind of fill, the one the old article on not getting filled warns is where nearly all of a maker's losses come from. Concretely, this is a skew. When signed flow over your window is strongly positive, you skew your quotes up: pull your offer back or widen it so you are less likely to sell into the buying, and lean your bid in if you want to participate on the side the flow favors. This is the inventory-and-flow skew, now driven by the recognition that the flow is autocorrelated, so a positive reading is not a one-off, it is the start of a likely run. ## A worked number Suppose your true flow phi is 0.85 but OLS hands you 0.70\. You see a buy run with current pressure at some level and you want to know how long until it decays to a tenth of its strength. With the fitted 0.70, the run decays as 0.70 to the power k, hitting one-tenth at about k = 6.5 steps, so your model says the pressure is gone in roughly 7 ticks and clears you to quote offers again. With the true 0.85, decay is 0.85 to the power k, reaching one-tenth at about k = 14 steps. The run lasts more than twice as long as your model believes. Quote offers at tick 7 and you are selling into seven more ticks of buying you did not price for. The biased phi did not look dangerous; it just quietly told you the all-clear half a run too early. ## Visualizing the clustered flow ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_299-visual-0.png) ## KEY POINTS - Signed market-order flow is autocorrelated and well modeled as an AR(1) process: this instant's flow is mostly a fraction phi of the last instant's flow plus a shock, so buys follow buys and sells follow sells. - The flow clusters at a low value with sudden same-signed spikes, not a smooth oscillation. The spikes are where price moves and where inventory gets dangerous. - Phi is the memory: autocorrelation at lag k is phi to the k. At the sub-second scale phi is positive and often high, the formal version of "buys follow buys." - OLS estimates of phi are biased low because the regressor and error share past shocks. Your model then thinks runs decay faster than they do and clears you to requote too early. - The practical rule needs no estimation: do not post sell limits into a cluster of buy takers. Selling into a run hands takers liquidity and leaves you holding inventory while price runs against you. - Act on it as a skew: when windowed signed flow is strongly positive, pull or widen your offer and lean your bid, treating the reading as the start of a likely run rather than a one-off. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [The Art of Currency Trading - Brent Donnelly (Amazon)](https://www.amazon.com/s?k=The+Art+of+Currency+Trading+Brent+Donnelly&ref=aligrithm.com) ### 5.29 Microstructural Volatility: Three Ways to Measure It URL: https://aligrithm.com/microstructural-volatility-three-ways-to-measure-it/ Last updated: 2026-07-27T13:25:18.000Z Volatility is the single most important input to a spread. The old article ["Spread Widening During Volatility Expansion"](https://aligrithm.com/spread-widening-during-volatility-expansion/) laid down the rule: blow your quotes out when the market's volatility does, because a fixed markup that is safe in a calm tape is a free gift to informed traders in a moving one. The whole argument rests on having a volatility number to react to. Which raises the question nobody answers cleanly: at the microstructure scale, where you are repricing every few hundred milliseconds, what volatility do you even measure? Standard realized volatility, the standard deviation of returns over some bars, is built for a slower world. By the time you have enough one-minute bars to estimate it, the burst that would have picked you off is over. You need a volatility that lives at the tick scale and updates as fast as the book does. There are three honest ways to build one, and they measure genuinely different things. ## Way one: the standard deviation of an observable The most direct route is to take something the market hands you tick by tick and compute its rolling standard deviation. Three candidates work, and each has a different bias. $$ \\sigma\_{\\text{micro}} = \\operatorname{std}(X) \\qquad X \\in \\{\\text{traded prices},\\ \\text{volume},\\ \\text{price impact}\\} $$ Standard deviation of traded prices over the last N trades is the obvious one: when prints are scattered widely, the market is volatile. Standard deviation of volume is a stranger choice that works because volume and volatility are nearly the same variable in disguise, swap one for the other in most microstructure formulas and the feature barely changes, so a jumpy volume series flags a jumpy price series. Standard deviation of price impact, the per-trade move from first to last fill, measures how violently each trade is shoving the mid around. The trade-price version is the most interpretable and the one to start with. The volume version is useful when the tape is thin and prices are sparse, because volume keeps printing even when the price sits still. They will disagree, and the disagreement is informative: high volume-vol with low price-vol means a lot of size is changing hands without moving the market, which is a thick, absorptive book, the opposite of fragile. ## Way two: derive it from the flow rates The second route skips price entirely and reads volatility off the book's churn. The previous article in this cluster, ["The Market-Maker Feature Catalog: Arrival, Cancellation, and Update Rates,"](https://aligrithm.com/the-market-maker-feature-catalog-arrival-cancellation-and-update-rates/) built three intensity features, and those rates are themselves a volatility proxy. When orders arrive and cancel at high speed, the book is being rewritten violently, and a violently rewritten book produces volatile prices. The logic: arrival rate plus cancellation rate is how fast the resting liquidity is turning over, and fast turnover is the precondition for a fast price move. You can read the update rate directly as a microstructural volatility, or combine arrival and cancellation into one churn number. The advantage over the price-based estimators is latency. The flow rates lead price, because the book churns before the print moves, so a vol estimate built from arrival and cancellation rates reacts a beat earlier than one built from traded prices. The cost is that it is a proxy, not the thing itself: a frantically churning book usually means a volatile price, but not always, and you are trading directness for speed. ## Way three: the volatility of your own fair price The third route is the one a real maker should reach for, and it only exists if you already run a pricing model. If you quote around a fair value, the old article ["Order Book Imbalance: The First Microstructure Feature to Test"](https://aligrithm.com/order-book-imbalance-the-first-microstructure-feature-to-test/) pushed imbalance as the core input to that fair value, then take the standard deviation of your own fair price over a rolling window. This is the cleanest of the three for one reason: it measures volatility in exactly the quantity you care about. You do not get adversely selected on traded prices or on volume; you get adversely selected when your fair value moves between the moment you post a quote and the moment someone fills it. The volatility of your fair price is the direct measure of that risk. It also automatically folds in everything your fair-value model already knows, imbalance, microprice, trade flow, so it is a richer estimate than any single raw observable. The catch is circularity and noise: if your fair-value model is jumpy or mis-specified, its volatility is your model's noise, not the market's, so this estimator is only as good as the price it is measuring. ## A worked number Run a 200-trade rolling window on an alt perp. Traded prices over that window have a standard deviation of 12 basis points around their mean. Your base spread in the calm regime was set to a 6 bps half-width, on the assumption that fair value would not move more than a few bps before you reprice. The micro-vol estimate just told you fair value is scattering twice that far, so a 6 bps half-width means you are quoting inside one standard deviation of where the price is wandering, getting picked off on the moves. Scaling the half-width by current-vol over baseline-vol, roughly 12 over 6, doubles your quote to a 12 bps half-width and restores the cushion. Same mechanism as the discrete and continuous regime models from the spread-widening article, now driven by a tick-scale number instead of a one-minute one. ## Which to use Start with the standard deviation of traded prices because it is honest and interpretable. Add the flow-rate version when you need the vol estimate to lead price rather than lag it, accepting that it is a proxy. Move to the volatility of your own fair price once you trust your pricing model, because it measures adverse-selection risk in its native units. The three are complementary, and a robust desk blends them: the fair-price vol for the core spread, the flow-rate vol as an early-warning overlay that widens you before the first violent print lands. The usual caution applies harder here than anywhere. Microstructural volatility is wildly non-stationary, it spikes and collapses in seconds, so any window you pick is a compromise between responsiveness and noise. Too short and the estimate jitters and whipsaws your spread; too long and it lags the burst you were trying to catch. There is no correct window, only a tradeoff you tune against markouts, and you should expect to run a fast estimate and a slow one and take the larger when they disagree. ## Visualizing the three estimators ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_298-visual-0.png) ## KEY POINTS - Standard realized volatility from one-minute bars is too slow for a maker repricing every few hundred milliseconds; microstructural volatility lives at the tick scale and updates with the book. - Way one: rolling standard deviation of an observable, traded prices (most interpretable), volume (works on thin tapes because volume tracks volatility), or price impact. Disagreement between price-vol and volume-vol is itself informative. - Way two: derive volatility from arrival plus cancellation rates. The book churns before the price moves, so this estimate leads, at the cost of being a proxy rather than the thing itself. - Way three: standard deviation of your own fair price. The best option once you trust your model, because adverse selection happens in fair-value space, so this measures the actual risk in its native units, and it inherits everything the fair-value model already knows. - Worked size: if traded-price std is 12 bps and your half-width is 6 bps, you are quoting inside one standard deviation of price wander and getting picked off; scale the half-width by current vol over baseline vol to restore the cushion. - Microstructural volatility is violently non-stationary, so every window is a responsiveness-versus-noise compromise; run a fast and a slow estimate, take the larger when they disagree, and tune against markouts. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [The Art of Currency Trading - Brent Donnelly (Amazon)](https://www.amazon.com/s?k=The+Art+of+Currency+Trading+Brent+Donnelly&ref=aligrithm.com) ### 5.28 The Market-Maker Feature Catalog: Arrival, Cancellation, and Update Rates URL: https://aligrithm.com/the-market-maker-feature-catalog-arrival-cancellation-and-update-rates/ Last updated: 2026-07-27T13:25:18.000Z Order book imbalance gets all the attention, and the old article ["Order Book Imbalance: The First Microstructure Feature to Test"](https://aligrithm.com/order-book-imbalance-the-first-microstructure-feature-to-test/) argued it should be your first build because it does most of the work centering fair value. But imbalance is a snapshot of how much size is resting right now. It says nothing about how fast that size is churning. A book sitting at +0.4 imbalance with nobody touching it is a different animal from a book at +0.4 where limits are pouring in and getting yanked twice a second, and a maker who treats them the same gets run over by the second one. The fix is to stop looking at the level of the book and start measuring its flow rates. Three of them carry most of the information: how fast new orders arrive, how fast they get cancelled, and how fast existing levels get rewritten. Each is a count divided by time, which makes them cheap to compute and easy to bolt onto whatever you already feed your quoting engine. ## The three rates Each rate is the same shape: count an event type over a window, divide by the window length, get a per-second intensity. The events come from two different data feeds, so you need both an L2 (order book delta) stream and a trade stream to build the full set. Order arrival rate counts new limit orders landing in the book per unit time. Cancellation rate counts orders pulled per unit time. Update rate counts deltas, every change to a level, which folds arrivals, cancels, and partial fills into one number that tells you how violently the book is being rewritten. $$ \\text{arrival} = \\frac{\\#\\,\\text{new limit orders}}{\\Delta t} \\qquad \\text{cancellation} = \\frac{\\#\\,\\text{cancels}}{\\Delta t} \\qquad \\text{update} = \\frac{\\#\\,\\text{deltas}}{\\Delta t} $$ Read those as three counters over the same clock. New limits hitting the book go in the first. Orders disappearing without a trade to explain them go in the second. Every book mutation of any kind goes in the third. The update rate is roughly the sum of the other two plus fills, so if you only have bandwidth for one number, it is the densest, but the split between arrival and cancellation is where the behavioral signal lives. ## Cancellation is the hard one to measure Arrival is easy: a new order ID shows up in the feed, you increment. Cancellation is where people cut corners and get it wrong, because a level shrinking does not always mean someone cancelled. Size at the best bid can drop because a market sell ate it, which is a fill, not a cancel. Lump those together and your "cancellation rate" is really a trade-activity rate wearing a disguise. To separate them you need the trade feed alongside the L2 feed. The rule is mechanical: when the size at a level decreases, check whether a trade printed at that price for that amount in the same instant. If a trade explains the decrease, it is a fill. If the size dropped with no trade to account for it, it is a cancellation. This is why the catalog needs both streams, and why a cancellation rate built from book data alone is contaminated. The cancellation rate is the feature that smells out spoofing and flighty liquidity. The old article ["Spoofing, Sturdy Liquidity, and Book Pressure"](https://aligrithm.com/spoofing-sturdy-liquidity-and-book-pressure/) made the case that a wall you can rest in front of is only worth anything if it is real, and that telling sturdy size from spoofed size comes down to order lifetime and cancel-on-approach behavior. A high cancellation rate is that behavior measured in aggregate: when cancels per second spike, the book is full of orders nobody intends to honor, and any placement edge you thought you had behind a large resting order is built on sand. ## Pressure: the signed cousin The rate features above are unsigned counts, they tell you intensity but not direction. The directional version comes from the trade feed by signing each execution and summing over time, which gives you flow pressure. $$ \\text{pressure} = \\frac{\\sum\_i \\text{side}\_i \\cdot \\text{amount}\_i}{\\Delta t} \\qquad \\text{side}\_i = +1 \\ (\\text{taker buy}), \\ -1 \\ (\\text{taker sell}) $$ Each trade contributes its size with a plus if a taker lifted the offer and a minus if a taker hit the bid, and you divide the running sum by the window. This is the per-time form of the net signed flow from the old article ["Using Trade Flow to Predict Short-Term Price Movement,"](https://aligrithm.com/using-trade-flow-to-predict-short-term-price-movement/) and it predicts for the same reason: takers move price, and their flow is autocorrelated, so a stretch of net buying pressure tends to be followed by more of it. Pressure is the direction; the arrival and cancellation rates are the context that tells you whether to trust it. ## A worked number Say you run a 10-second window on BTC perp. In that window you log 40 new limit orders, 32 cancellations, and 18 fills. Arrival rate is 4.0 orders per second, cancellation rate is 3.2 per second, and the update rate is roughly (40 + 32 + 18) / 10 = 9.0 deltas per second. Now the same window during a calm patch: 8 new limits, 5 cancels, 2 fills, giving 0.8 arrival, 0.5 cancellation, 1.5 update. The book in the first window is churning six times faster. A spread calibrated to the second window is dangerously tight in the first, because the same imbalance reading is being driven by transient orders that will not be there when you need them. ## Where it plugs in These rates are inputs, not signals on their own. You feed them into the decisions you already make. The cancellation rate weights how much you trust a resting wall for placement, extending the order-placement logic from the old article ["Order Placement Alpha: The Forgotten Edge,"](https://aligrithm.com/order-placement-alpha-the-forgotten-edge/) where the whole game is resting in front of size that will actually protect your fill. The arrival and update rates feed your volatility estimate, which is the next article in this cluster. Pressure skews your fair value. None of them replaces imbalance; they tell you how seriously to take it. Two cautions. First, the rates depend on how the exchange reports its feed: some venues aggregate updates, some send every individual order, so a "delta" on Binance is not the same unit as a "delta" on a venue that batches, and you cannot compare update rates across exchanges without normalizing. Second, like everything in this market the rates are non-stationary, so the absolute level of "high churn" drifts with the venue and the regime; use a rolling baseline and read the rates relative to their own recent distribution rather than against a fixed threshold. ## Visualizing the feature catalog ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_297-visual-0.png) ## KEY POINTS - Imbalance measures the level of resting size; arrival, cancellation, and update rates measure how fast the book churns, and a maker needs both to know whether a given imbalance is trustworthy. - Each rate is a count over a window: arrival = new limits / time, cancellation = cancels / time, update = deltas / time. The update rate is the densest single number but the arrival-versus-cancellation split carries the behavioral signal. - Cancellation rate is the one people measure wrong. A shrinking level can be a fill, not a cancel, so you need the trade feed alongside L2 to separate them, otherwise you are measuring trade activity in disguise. - A high cancellation rate is spoofing and flighty liquidity in aggregate; use it to discount any placement edge that assumes a resting wall is real. - Pressure is the signed, directional cousin, signed trade size over time, the per-time form of net trade flow that skews fair value. - Feed the rates as inputs, not standalone signals: cancellation weights placement trust, arrival and update feed volatility, pressure skews the center. Use a rolling baseline because the rates are non-stationary and venue-dependent. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [The Art of Currency Trading - Brent Donnelly (Amazon)](https://www.amazon.com/s?k=The+Art+of+Currency+Trading+Brent+Donnelly&ref=aligrithm.com) ### 5.27 NYSE-Open Volume Momentum URL: https://aligrithm.com/nyse-open-volume-momentum/ Last updated: 2026-07-27T13:25:17.000Z Crypto trades around the clock, but it does not trade evenly. When New York equities open, a wave of attention and flow hits the crypto book, and for the next ninety minutes the tape behaves differently than it does at 04:00 UTC. The specific claim is narrow and testable: from roughly 13:30 to 15:00 UTC, if volume keeps rising, price tends to trend, and the direction of the first half hour sets the direction for the rest of the window. This is a time-of-day strategy, which means the old article ["How to Choose the Right Timeframe for a Strategy"](https://aligrithm.com/how-to-choose-the-right-timeframe-for-a-strategy/) is the governing constraint. That piece argued the timeframe decides whether you see trend or chop, because noise itself is a function of timeframe and the window you pick is upstream of the trend-versus-reversion choice. Here the window is fixed by an external clock, the NYSE open, and the bet is that this particular window sits on the trending side of the noise threshold while the rest of the day does not. ## The setup The mechanism is the volume conditioning from the old article ["The Dance of Volume and Price,"](https://aligrithm.com/the-dance-of-volume-and-price/) pinned to a specific clock window. Rising volume signals continuation; the NYSE open is a reliable, recurring volume surge. Combine them and you get a window where the continuation regime is likely to be switched on, so a directional move that starts the window has backing to carry it. Two conditions have to hold together. The window is 13:30 to 15:00 UTC. First, volume must be rising through the window, confirming the continuation regime rather than a fading one. Second, the sign of the early move sets the direction. $$ \\text{sign}\\big(r\_{13{:}30 \\to 14{:}00}\\big) > 0 \\;\\Rightarrow\\; \\text{long until } 15{:}00 $$ $$ \\text{sign}\\big(r\_{13{:}30 \\to 14{:}00}\\big) < 0 \\;\\Rightarrow\\; \\text{short until } 15{:}00 $$ Read it straight: take the return over the first half hour of the window, 13:30 to 14:00 UTC. If it is positive, the open is pushing up, so go long and hold to 15:00\. If it is negative, the open is pushing down, so go short and hold to 15:00\. The early move is your direction read; the rising volume is your permission to treat it as momentum instead of fading it. Without the volume confirmation you are buying a thirty-minute move with no reason to think it continues, which is the chop the timeframe article warns the wrong window hands you. ## Why this connects to flow The deeper reason the open trends is order flow, the subject of the old article ["Using Trade Flow to Predict Short-Term Price Movement."](https://aligrithm.com/using-trade-flow-to-predict-short-term-price-movement/) The NYSE open brings a burst of takers into crypto, and takers move price and cluster, buys following buys. A surge of one-sided taking on the book at a predictable time is the cleanest version of the continuation signal: you know roughly when the flow arrives (the open), you confirm it is arriving (rising volume), and you read its direction from the first thirty minutes. The strategy is a structured way to ride a recurring, time-stamped burst of taker flow. ## Where it breaks Be blunt about the soft spots. The "NYSE open" is not a fixed UTC time across the year, because US daylight-saving shifts the open between 13:30 and 14:30 UTC, so a hard-coded window will be an hour off for part of the year and you must track the actual open, not a frozen timestamp. The effect is also a crowded, well-documented intraday pattern, so the edge net of fees on a ninety-minute hold is thin and decays as more desks trade it. And it is a directional bet inside a single narrow window, so the sample of independent days accumulates slowly, which the timeframe article flags as the cost constraint: few trades means slow confirmation and a long wait to know whether the edge is real or you are reading noise from a handful of strong opens. Confirm the rising-volume condition is doing real work by testing the rule with and without it; if the volume filter does not improve the result, you are trading a plain open-momentum rule and should admit it. ## Visualizing the window ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_296-visual-0.png) ## KEY POINTS - From about 13:30 to 15:00 UTC, when volume keeps rising, crypto price tends to trend, and the sign of the first half hour (13:30 to 14:00) sets the direction held to 15:00. - This is the volume-continuation switch from ["The Dance of Volume and Price"](https://aligrithm.com/the-dance-of-volume-and-price/) pinned to the recurring volume surge at the NYSE open: rising volume turns on the continuation regime, so the early move has backing to carry. - The rule needs both conditions: volume rising through the window (regime confirmation) and the early-move sign (direction). Without the volume filter you are fading or chasing a thirty-minute move with no reason it continues. - It works because the open brings a burst of clustered taker flow, the continuation logic of ["Using Trade Flow to Predict Short-Term Price Movement,"](https://aligrithm.com/using-trade-flow-to-predict-short-term-price-movement/) at a predictable, time-stamped moment. - Time-of-day strategy, so ["How to Choose the Right Timeframe for a Strategy"](https://aligrithm.com/how-to-choose-the-right-timeframe-for-a-strategy/) governs: the bet is that this window sits on the trending side of the noise threshold while the rest of the day does not. - Breaks: the open shifts 13:30 vs 14:30 UTC with daylight saving (track the real open), the pattern is crowded and decays, and one narrow window per day accumulates independent samples slowly. Test with and without the volume filter to prove it earns its keep. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [The Art of Currency Trading - Brent Donnelly (Amazon)](https://www.amazon.com/s?k=The+Art+of+Currency+Trading+Brent+Donnelly&ref=aligrithm.com) ### 5.26 The Dance of Volume and Price URL: https://aligrithm.com/the-dance-of-volume-and-price/ Last updated: 2026-07-27T13:25:17.000Z Volume is the cheapest piece of information on the tape and the most underused. It tells you not where price is going but whether the next move is likely to stick or snap back. Predict volume and you get a conditional read on price: high volume tends to carry a move forward, low volume tends to let it revert. That conditional split is the trade. Start with what volume is. It is the sum of trade sizes in a period, so it lives downstream of trades, the executed liquidity the old article ["Using Trade Flow to Predict Short-Term Price Movement"](https://aligrithm.com/using-trade-flow-to-predict-short-term-price-movement/) treats as the maker's core signal. Trade flow signs that activity into buys and sells; raw volume throws the sign away and keeps the magnitude. The magnitude alone, it turns out, already carries a usable continuation-versus-reversion signal. ## How volume behaves Three statistical facts set up everything else, and none of them are exotic. Volume clusters and has fat tails, the same shape as volatility. Quiet stretches sit near a low baseline, then a burst arrives and drags a run of high-volume bars behind it. So volume is forecastable in the weak sense that a high bar makes the next bar more likely high, the persistence you would model with the same machinery as vol. Volume tracks the number of trades, but not linearly. For large trade counts the relationship is roughly linear; below that it bends, with positive convexity, because a handful of large trades can post big volume on few executions while many tiny trades post modest volume on high count. So volume and trade count are strongly correlated and not interchangeable, and which exchange's volume you use depends on what you are predicting, since the books differ. Volume mean-reverts to its mode. Regress this bar's volume on last bar's and the slope pulls toward the center: a high bar tends to be followed by a lower one, a low bar by a higher one, both drifting back to the typical value. Persistence in the clusters, reversion in the level. Both are true at once because they describe different horizons. ## The price link The bridge to price is one observed regularity: price is more volatile when volume is high. More executed size means more information and more impact hitting the book, so the move is larger and, more usefully, more likely to be real rather than noise. The old article ["Noise Is Not Volatility"](https://aligrithm.com/noise-is-not-volatility/) is the tool that makes this precise: volatility is how far price travels, noise is how much of that travel cancels before producing net direction. High volume tends to come with cleaner, lower-noise travel, a move that nets out somewhere; low volume tends to come with high-noise chop that cancels. That gives the conditional rule. $$ \\text{Volume} \\uparrow \\;\\Rightarrow\\; \\text{continuation (price keeps going)} $$ $$ \\text{Volume} \\downarrow \\;\\Rightarrow\\; \\text{reversion (price snaps back)} $$ Read it as a regime switch on the magnitude of volume. When volume is rising, treat the current move as a trend to ride, because the size behind it says the move has backing and lower noise. When volume is falling, treat the current move as overextension to fade, because thin participation says the travel is noise that cancels. Volume is not the direction signal; it is the switch that tells you whether to apply a momentum rule or a reversion rule to whatever direction price is already showing. ## Turning it into something tradable The rule needs a volume forecast to act ahead of the move rather than after it. Use the two facts above: persistence says a high recent bar predicts a high next bar, reversion says an extreme bar predicts a pull toward the mode. A simple forecast blends them, predicting next-bar volume from recent bars while pulling extremes back toward the typical level. Then condition: if forecast volume is rising relative to its mode, run the continuation leg in the direction of the recent return; if forecast volume is falling, run the reversion leg against it. Be honest about the limits. This is a coarse, two-state switch sitting on top of a noisy volume forecast, and volume's exchange-dependence means a signal fit on one venue's tape can evaporate on another. The continuation-reversion split is a real conditioning variable, not a standalone alpha; it sharpens a directional rule you already have by telling you which regime you are in, the same way the old article ["Noise Is Not Volatility"](https://aligrithm.com/noise-is-not-volatility/) uses the noise axis to pick the game while volatility only sizes the bet. Treat volume as the column that picks momentum versus reversion, and your existing signal as the bet inside it. ## Visualizing the switch ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_295-visual-0.png) ## KEY POINTS - Volume is the sum of trade sizes, downstream of the trade flow from ["Using Trade Flow to Predict Short-Term Price Movement,"](https://aligrithm.com/using-trade-flow-to-predict-short-term-price-movement/) but it keeps magnitude and drops the buy/sell sign. The magnitude alone carries a continuation-versus-reversion read. - Volume clusters with fat tails like volatility (persistence), tracks trade count nonlinearly with convexity at low counts, and mean-reverts to its mode (a high bar tends to be followed by a lower one). Persistence and reversion describe different horizons and both hold. - Price is more volatile when volume is high, and high volume tends to mean lower-noise travel. ["Noise Is Not Volatility"](https://aligrithm.com/noise-is-not-volatility/) makes this exact: high volume carries cleaner net direction, low volume carries chop that cancels. - The conditional rule: volume up means continuation (ride the move), volume down means reversion (fade it). Volume is the regime switch applied to whatever direction price already shows, not the direction itself. - To trade it, forecast next-bar volume by blending persistence (high follows high) with mean reversion (extremes pull to the mode), then pick the momentum or reversion leg from that forecast. - Limits: a coarse two-state switch on a noisy, exchange-dependent volume forecast. It sharpens an existing directional signal by naming the regime; it is not a standalone alpha. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [The Art of Currency Trading - Brent Donnelly (Amazon)](https://www.amazon.com/s?k=The+Art+of+Currency+Trading+Brent+Donnelly&ref=aligrithm.com) ### 5.25 The 24-Hour Rolling-Return Artifact URL: https://aligrithm.com/the-24-hour-rolling-return-artifact/ Last updated: 2026-07-27T13:25:16.000Z Look at the "24h change" number on any crypto exchange ticker. It says plus 3 percent, and a retail trader reads that as momentum and buys. Half the time that number moved without the price moving at all. The window is rolling, not anchored to midnight, so every hour the oldest hour falls out of the trailing 24, and when a big move drops off the back the headline number jumps even though the current price sat still. The display is reacting to old information leaving, not new information arriving, and that mechanical jump is something you can front-run. This is the same lesson as the old article ["Predictive Power vs Long Bias: The Hidden Trap in Backtests,"](https://aligrithm.com/predictive-power-vs-long-bias-the-hidden-trap-in-backtests/) moved from the backtest to the ticker. There a positive return looked like edge but decomposed into exposure the trader never chose. Here a positive 24h change looks like momentum but decomposes into an arithmetic artifact of the rolling window. The trader chasing it owns a signal with no predictive content. ## The mechanism The exchange shows the return from now back to 24 hours ago. That window slides forward continuously. The newest point is the current price, which may be flat; the oldest point is whatever traded 24 hours ago, which is about to be discarded as the clock ticks one more hour. Walk the case where price has been flat for the last hour but moved hard down 24 hours ago. The return measured from the about-to-drop oldest bar up to now is negative, that old crash is still inside the window dragging the headline down. One hour later that crash falls out of the trailing 24\. The window now starts an hour later, after the crash, so the same flat current price now sits above the new window start, and the displayed 24h return jumps up. Price did nothing. The number rose because a negative print left the calculation. $$ R^{24h}\_t = \\frac{P\_t - P\_{t-24}}{P\_{t-24}} \\qquad \\Delta R^{24h} \\approx -\\,r\_{t-24} $$ The R-24h is the displayed rolling return: current price P\_t over the price 24 hours ago P\_{t-24}, minus one. The change in that headline as the window rolls one hour, with current price flat, is approximately the negative of r\_{t-24}, the single hour's return that just dropped off the back. A negative hour leaving (a crash rolling off) pushes the headline up; a positive hour leaving pushes it down. The sign flip is the whole story: the display moves opposite to the return that exits the window. ## Who chases it and how you front-run Uninformed traders read the rising 24h number as positive momentum and buy, hoping the trend continues. They are reacting to a quantity with no forward information, the removal of a stale crash. That predictable buying is order flow you can see coming, which connects straight to the old article ["Using Trade Flow to Predict Short-Term Price Movement"](https://aligrithm.com/using-trade-flow-to-predict-short-term-price-movement/): uninformed takers about to hit the book are exactly the flow a maker wants to anticipate and lean into. Here you know the flow is coming because you know which old return is about to roll off. So position ahead of the crowd. When the return 24 hours ago was negative, that print is about to leave the window, the headline is about to rise, the uninformed are about to buy, so go long before them. When the return 24 hours ago was positive, the headline is about to fall, the uninformed are about to sell, so go short. You are front-running a mechanical display, not predicting fundamentals. $$ \\text{Long if } r\_{t-24} < 0, \\qquad \\text{Short if } r\_{t-24} > 0 $$ Read it plainly: r\_{t-24} is the return over the hour that is about to exit the trailing window. Negative means a down-print is leaving, the headline will rise, so be long. Positive means an up-print is leaving, the headline will fall, so be short. The signal is the sign of the oldest hour, nothing more. ## Confirming the spike before you trade it Do not trust the story without the regression. Test whether returns at lag 24 actually predict the next return by regressing the one-hour return that happened x hours ago against the next one-hour return, sweeping x across a hundred lags, and looking for a spike at x equals 24. ```python slopes = [] for lag in range(1, 100): slope, intercept = np.polyfit(rets[:-lag], rets[lag:], 1) slopes.append(slope) # inspect slopes[24]: a clear spike there is the rolling-window artifact ``` A clean strategy shows a noticeable bump at lag 24 and noise everywhere else. That bump is the artifact leaking into realized returns, the uninformed buying lifting price as the headline rises. If the bump is not there in your data and your fee schedule, you do not have the trade, you have a plausible story, and the old article ["Predictive Power vs Long Bias: The Hidden Trap in Backtests"](https://aligrithm.com/predictive-power-vs-long-bias-the-hidden-trap-in-backtests/) is the standing warning that a plausible story plus a flattering line is how good narratives become live losses. Size it small: the edge is a few uninformed traders chasing a number, which is thin, regime-dependent, and competed away the moment it is widely known. ## Visualizing the artifact ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_294-visual-0.png) ## KEY POINTS - Exchange "24h change" is a rolling trailing-window return, not anchored 00:00 to 00:00\. Every hour the oldest hour drops out, so the headline can move while price sits still. - The change in the headline as the window rolls (with price flat) is approximately the negative of the hour's return that just exited. A crash rolling off pushes the number up; a rally rolling off pushes it down. - Uninformed traders read the rising number as momentum and buy. That predictable flow is what a maker leans into, the same trade-flow logic from ["Using Trade Flow to Predict Short-Term Price Movement,"](https://aligrithm.com/using-trade-flow-to-predict-short-term-price-movement/) except here you know the flow is coming. - Rule: long when the return 24 hours ago was negative (headline about to rise, crowd about to buy), short when it was positive. The signal is the sign of the oldest exiting hour. - Confirm with a lag sweep: regress the return x hours ago against the next return for x up to 100 and look for a clean spike at x equals 24\. No spike, no trade. - This is a momentum mirage, not predictive power, the live version of the bias trap in "Predictive Power vs Long Bias." Size it small; it is thin, regime-dependent, and competed away once known. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [The Art of Currency Trading - Brent Donnelly (Amazon)](https://www.amazon.com/s?k=The+Art+of+Currency+Trading+Brent+Donnelly&ref=aligrithm.com) ### 5.24 Harvesting the Volatility Risk Premium by Hour URL: https://aligrithm.com/harvesting-the-volatility-risk-premium-by-hour/ Last updated: 2026-07-27T13:25:16.000Z The old article ["Crypto Volatility Seasonality"](https://aligrithm.com/crypto-volatility-seasonality/) established that vol runs on a 24-hour clock. The same group-by that gave you the variance profile also gives you the average return profile, and the two together suggest a trade: if certain UTC hours reliably carry more risk and certain hours reliably pay for it, hold the coin during the paid hours and step out, or short, during the hours that bleed. That is the volatility risk premium sliced by time of day. The logic behind a risk premium is old and not crypto-specific: bearing risk should pay, so a high-volatility window that investors want to avoid should offer extra expected return to whoever sits through it. Whether crypto actually pays that premium cleanly, hour by hour, is an empirical question, and the honest answer is "sometimes, weakly, and not stably." This is a real effect worth measuring and a fragile one worth distrusting in equal measure. ## Build the two profiles Same machinery as the seasonality article, run twice. Bucket every hourly bar by its UTC hour, then within each bucket compute two averages: the mean return and the standard deviation of returns. The first tells you which hours tend to drift up or down. The second tells you which hours are risky. $$ \\bar{r}\_h = \\frac{1}{N\_h} \\sum\_{t \\,:\\, \\text{hour}(t)=h} r\_t \\qquad \\sigma\_h = \\sqrt{\\frac{1}{N\_h} \\sum\_{t \\,:\\, \\text{hour}(t)=h} (r\_t - \\bar{r}\_h)^2} $$ The r-bar\_h is the average return for clock hour h, the simple mean of every hourly return that landed in that hour. The sigma\_h is the standard deviation within the same bucket, the spread of those returns around their mean. N\_h is the count of bars in the bucket. Twenty-four pairs of numbers, return and risk for each hour, and the trade reads straight off them: be long during hours whose average return is reliably positive, be flat or short during hours whose average return is reliably negative, and weight your conviction by how clean the signal is relative to sigma\_h. ## A concrete rule The source profile points at a specific pattern: BTC tends to drift up over the 00:00 to 01:00 UTC hour and tends to drift down over the 01:00 to 03:00 window. So the toy rule is two trades. Buy BTC at 00:00 UTC and sell at 01:00\. Short BTC at 01:00 and cover at 03:00. Put a number on it. Suppose the 00:00 to 01:00 bucket averages plus 8 basis points and the 01:00 to 03:00 window averages minus 6 basis points per the two hours combined. A round trip that captures both legs nets roughly 14 basis points gross per day, before costs. Now subtract costs honestly. Two entries and two exits is four taker fills; at even 1.5 basis points per side that is 6 basis points of fees, plus spread paid and slippage on each. The 14 bps gross can become 4 to 6 bps net on a good day and negative on a normal one. A 14 bps edge that survives as a few bps after costs is exactly the kind of thin, fragile alpha that looks great in a frictionless backtest and dies on contact with a real fee schedule. ## Why to distrust it Three things will eat this faster than you expect. First, the per-hour averages are estimated on a small slice of your data, the same 24-way split as the seasonality article, so a couple of violent days can manufacture an entire "premium" that is not there. Confirm the hourly returns survive dropping the largest moves, and look at the distribution, not just the mean: an 8 bps average built from one 400 bps day and a row of small negatives is not a premium, it is a single event. Second, this is a directional bet, and the old article ["Why Volatility Is More Non-Stationary Than Trend"](https://aligrithm.com/why-volatility-is-more-non-stationary-than-trend/) is blunt about why that is hard: returns are near-random-walk with tiny autocorrelation, so a directional edge this small needs a large, stable sample to even confirm, and crypto does not give you a stable directional process. Vol seasonality is robust; return seasonality is far flimsier, and you should size accordingly. Third, anything this mechanical and well-known gets arbitraged. If a clean "long BTC at midnight" effect existed and persisted, the funding and rebalancing flows that cause the 00:00 vol bump would be front-run until the return edge flattened. The vol seasonality persists because it comes from real recurring plumbing; the return premium is the part most likely to already be competed away. ## Where it fits Treat the hourly return profile as a tilt, not a standalone strategy. The cleaner use is to let it nudge a position you already hold for another reason: skip adding risk in the hours that historically bled, lean in slightly during the hours that historically paid. The old article ["Volatility Regimes and Strategy Survival"](https://aligrithm.com/volatility-regimes-and-strategy-survival/) framed the general move of conditioning exposure on regime; time of day is just the cheapest regime axis to compute. And the old article ["Spread Widening During Volatility Expansion"](https://aligrithm.com/spread-widening-during-volatility-expansion/) is the cousin trade on the maker side: the same 00:00 and 14:00 windows that carry the premium are the ones where you widen quotes, so the time-of-day map serves both the taker tilt and the maker spread. ## Visualizing the two profiles ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_293-visual-0.png) ## KEY POINTS - The same hourly group-by from the seasonality work, run for mean return and for standard deviation, gives a 24-hour return profile and a 24-hour risk profile. The trade reads off them: long the paid hours, flat or short the bleeding hours. - Source pattern: long BTC 00:00 to 01:00 UTC, short BTC 01:00 to 03:00\. The midnight hour that carries the vol spike is also the one that tends to pay. - Worked costs: a \~14 bps gross round trip is four taker fills. At 1.5 bps per side plus spread and slippage, it nets a few bps on a good day and negative on a normal one. Thin, fragile alpha. - Distrust it on three counts: tiny per-bucket samples let a few violent days fake a premium; directional edges are hard to confirm because returns are near-random-walk; and a mechanical, well-known effect gets arbitraged. - The vol seasonality is robust because it comes from real recurring plumbing. The return premium is the flimsier sibling and the part most likely already competed away. - Use it as a tilt, not a standalone strategy: skip risk in historically bleeding hours, lean in during historically paid ones, and let the same time map drive maker spread widening. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [The Art of Currency Trading - Brent Donnelly (Amazon)](https://www.amazon.com/s?k=The+Art+of+Currency+Trading+Brent+Donnelly&ref=aligrithm.com) ### 5.23 SAR: Seasonal Autoregressive Volatility Forecasting URL: https://aligrithm.com/sar-seasonal-autoregressive-volatility-forecasting/ Last updated: 2026-07-27T13:25:15.000Z The old article ["Crypto Volatility Seasonality"](https://aligrithm.com/crypto-volatility-seasonality/) showed the bumps: vol clusters at 14:00 and 00:00 UTC, and the magnitude-only models walk right past them. This is the fix. Add lagged terms at the seasonal periods so the forecast can lean on "what did vol do at this same hour yesterday, and this same hour last week," not only "what did vol do an hour ago." SAR stands for Seasonal Autoregressive. It is the plain AR model with extra lags placed at the periods where the series repeats. For hourly crypto vol the repeats are daily (24 hours) and weekly (168 hours), so those are the lags you bolt on. The model is a linear regression you fit by ordinary least squares, which means no special library and no excuse. ## The model Use squared hourly returns as the volatility proxy, the same proxy from the seasonality article. The forecast for the next hour's squared return is a constant plus three lagged squared returns: one hour back, 24 hours back, and 168 hours back. $$ \\hat{y}^2\_t = \\alpha + \\beta\_1 \\epsilon^2\_{t-1} + \\beta\_2 \\epsilon^2\_{t-24} + \\beta\_3 \\epsilon^2\_{t-168} $$ Read each term in plain language. The epsilon-squared is the squared hourly return, the variance proxy. The alpha is the baseline variance the model falls back to when the lags carry no signal. Beta-1 on the one-hour lag captures the short-memory clustering that GARCH already gets: a violent last hour means a violent next hour. Beta-2 on the 24-hour lag is the daily seasonal term: it asks whether this same hour yesterday was loud, which is how the 00:00 and 14:00 bumps get into the forecast. Beta-3 on the 168-hour lag is the weekly term: weekends and specific weekdays trade differently, so the same hour one week ago carries its own information. The hat on the left is the predicted variance for hour t. Fitting is multiple regression. Build a design matrix whose columns are the three lagged squared-return series, line up the target as the current squared return, and solve for alpha and the three betas by OLS. That is it. The seasonal lags are not a different estimator, they are extra columns in a linear regression. ## A worked fit Concrete numbers make the betas legible. Suppose OLS hands you alpha equal to a baseline variance matching a 30 bps hourly move, beta-1 of 0.25, beta-2 of 0.40, and beta-3 of 0.15\. Now forecast the 00:00 UTC bar. The one-hour lag (23:00) was quiet, the 24-hour lag (last midnight) was a loud 70 bps move, and the 168-hour lag (midnight a week ago) was a 60 bps move. The forecast loads heavily on the 0.40 daily term hitting last midnight's large value, so the predicted vol for this midnight comes out well above the all-day baseline even though the immediately preceding hour was calm. A plain AR(1) sees only the calm 23:00 hour and forecasts low, then gets run over. The seasonal lags are what let SAR see midnight coming. Note the betas usually rank the way intuition says: the one-hour term is strong (clustering is real), the daily term is the one that earns its keep on this series, and the weekly term is smaller and noisier. If beta-2 comes out near zero on your data, you do not have exploitable daily seasonality, and you should not force the term in because the seasonality article's plot looked suggestive. Let the regression tell you. ## The other way to do it There are two routes to the same place. One is this single regression with seasonal lags baked in. The other is a two-stage build: fit a plain AR(1) forecast on recent vol, then adjust that forecast up or down by a fixed seasonal factor for the current hour of day or day of week, the 24-point profile from the seasonality article. The two-stage version is easier to read and debug, because the seasonal adjustment is a separate multiplier you can inspect and the AR(1) part stays familiar. The single regression is cleaner statistically, because OLS sets all the weights jointly instead of you hand-tuning the split between "recent vol" and "seasonal factor." Pick the two-stage version when you want transparency, the joint regression when you want the fit to allocate the weight for you. Either way this is the toy version. In production you would reach for S-GARCH, periodic GARCH, or a model that lets the conditional variance dynamics themselves change by season rather than just adding linear lags of squared returns. SAR is the honest minimum: it captures the seasonality the old article ["Volatility Regimes and Strategy Survival"](https://aligrithm.com/volatility-regimes-and-strategy-survival/) said you must respect, with a regression simple enough to fit in a few lines and audit by eye, and the old article ["Why Volatility Is More Non-Stationary Than Trend"](https://aligrithm.com/why-volatility-is-more-non-stationary-than-trend/) is the reason this is worth doing at all, since vol's long memory and structure are what the seasonal lags are pulling on. ## Visualizing the lag structure ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_292-visual-0.png) ## KEY POINTS - SAR is plain AR with extra lags placed at the seasonal periods. For hourly crypto vol those are 24 hours (daily) and 168 hours (weekly). - Forecast next hour's squared return as a constant plus three lagged squared returns: t-1, t-24, t-168\. Fit alpha and the three betas by ordinary least squares; the seasonal lags are just extra columns. - The 24-hour lag is the term that injects the 00:00 and 14:00 bumps into the forecast. A plain AR(1) sees only the immediately prior hour, forecasts low into a calm 23:00, and gets run over at midnight. - Read the betas: clustering (t-1) strong, daily (t-24) the earner on this series, weekly (t-168) smaller and noisier. If the daily beta fits near zero, you have no exploitable daily seasonality, so do not force it in. - Alternative two-stage build: AR(1) forecast times a per-hour seasonal factor. Easier to debug; the joint regression is cleaner because OLS sets the weights together. - This is the toy version. Production uses S-GARCH or periodic GARCH that let the variance dynamics change by season, not just linear lags of squared returns. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [The Art of Currency Trading - Brent Donnelly (Amazon)](https://www.amazon.com/s?k=The+Art+of+Currency+Trading+Brent+Donnelly&ref=aligrithm.com) ### 5.22 Crypto Volatility Seasonality URL: https://aligrithm.com/crypto-volatility-seasonality/ Last updated: 2026-07-27T13:25:15.000Z Pull up an hourly chart of realized BTC volatility, average each UTC hour over a few months, and the line is not flat. Two bumps stick out: one around 14:00 UTC and a bigger one at 00:00 UTC. Volatility is not arriving at random through the day. It clusters at specific clock times, and the clock is doing work your model probably throws away. This matters to anyone quoting a spread or sizing a position by volatility, because the standard tools, GARCH and HAR, are built to miss exactly this. They treat the volatility process as a function of its own recent magnitude and nothing else. They have no slot for "it is midnight UTC, so expect a spike." So they under-forecast vol right before the predictable bursts and over-forecast it right after, and a market maker running on that forecast quotes too tight into the 00:00 burst and too wide in the dead hours after. ## Where the bumps come from The 00:00 UTC spike is the loud one, and the mechanism is plumbing, not sentiment. A lot of crypto activity is pinned to midnight UTC: funding-rate settlements on perps, daily mark and rebalancing flows from funds and structured products, accounting cutoffs that force end-of-day trades. When a pile of mechanical orders lands in the same minute every day, you get a repeatable jump in traded volume and a repeatable jump in price variance around it. The 14:00 UTC bump lines up with the US session waking up, roughly the cash-equity and macro-data window in the morning New York hours, when discretionary flow and news reactions stack on top of the always-on crypto book. The point is not the exact story behind each bump. The point is that the bumps repeat on a 24-hour cycle, which means they are forecastable, which means a model that ignores the cycle is leaving information on the floor. ## Why GARCH and HAR are blind to it GARCH forecasts tomorrow's variance from a constant, yesterday's squared shock, and yesterday's variance. HAR does something similar with daily, weekly, and monthly realized-vol averages. Both are autoregressive on magnitude alone. Neither has a term that says "this observation happened at hour h of the day," so neither can lift its forecast for the 00:00 bar and lower it for the 04:00 bar. The structure that would let it do that simply is not in the equation. You can see the failure as a residual pattern. Fit a plain GARCH to hourly crypto returns, then average the model's squared forecast error by hour of day. The errors are not flat. They are large and positive (model under-predicted) around 00:00 and 14:00, and negative in the quiet hours. A residual that is predictable from the clock is a feature you forgot to add. The old article ["Why Volatility Is More Non-Stationary Than Trend"](https://aligrithm.com/why-volatility-is-more-non-stationary-than-trend/) made the general case that vol carries long memory and structure worth modeling; seasonality is a slice of that structure sitting in plain sight on the time axis. ## Measuring it before you trust it The estimate is a simple group-by. Take a long run of hourly returns, square them as your variance proxy, tag each by its UTC hour 0 through 23, and average within each bucket. That gives you a 24-point seasonal profile: average variance at each hour. Plot it, and the two bumps appear if your sample is honest. $$ \\bar{v}\_h = \\frac{1}{N\_h} \\sum\_{t \\,:\\, \\text{hour}(t) = h} \\epsilon\_t^2 \\qquad h = 0, 1, \\dots, 23 $$ Read this as: for each clock hour h, collect every hourly squared return whose timestamp falls in that hour, and average them. The epsilon-squared is the squared hourly return standing in for realized variance, N\_h is the count of bars in bucket h, and v-bar\_h is the seasonal variance estimate for that hour. Twenty-four numbers, one per hour, and the shape of those numbers is the seasonality. A worked number makes the size concrete. Say average hourly variance across all hours corresponds to a typical hourly move of 40 basis points. If the 00:00 bucket averages roughly twice the variance, that hour's typical move is about 40 times the square root of 2, near 57 basis points. A spread or stop calibrated to the 40 bps all-day average is roughly 30 percent too tight in the one hour that reliably moves most. That is the cost of treating the clock as noise. Two cautions before you lean on the profile. First, it is non-stationary like everything else in this market: the 00:00 mechanism is tied to current funding schedules and product flows, and those drift, so re-estimate the profile on a rolling window rather than freezing a year-old shape. Second, watch your sample size per bucket. Twenty-four buckets split your data 24 ways, and a noisy bump from a handful of violent days is not a seasonal effect, it is a small sample. Confirm the bumps survive when you drop the largest few days. ## What to do with it The direct use is to make any volatility input clock-aware. Wherever you feed a volatility estimate, scale it by the seasonal factor for the current hour: multiply your base vol forecast by v-bar\_h divided by the all-hours average. The old article ["Spread Widening During Volatility Expansion"](https://aligrithm.com/spread-widening-during-volatility-expansion/) set the rule that quote width should track volatility; seasonality says part of that volatility is on a timetable, so widen ahead of 00:00 and 14:00 by the clock, not after the burst has already picked you off. The next step, folding the lagged seasonal term straight into the forecast model, is the SAR build in the old article ["SAR: Seasonal Autoregressive Volatility Forecasting."](https://aligrithm.com/sar-seasonal-autoregressive-volatility-forecasting/) ## Visualizing the seasonal profile ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_291-visual-0.png) ## KEY POINTS - Crypto realized volatility has a repeatable 24-hour shape: a medium bump near 14:00 UTC (US session) and the largest at 00:00 UTC (midnight rebalancing, funding settlement, accounting cutoffs). - GARCH and HAR forecast vol from its own recent magnitude only. They have no clock term, so they under-forecast the predictable bursts and over-forecast the quiet hours. - Measure the profile with a group-by: square hourly returns, bucket by UTC hour, average within each bucket. The result is 24 numbers whose shape is the seasonality. - Worked size: if the 00:00 bucket runs roughly double the average variance, that hour's typical move is about 40 percent larger, so an all-day-average spread is roughly 30 percent too tight at midnight. - Re-estimate on a rolling window (the mechanism drifts with funding schedules) and confirm bumps survive dropping the largest few days, or you are fitting a small sample, not a season. - Use it by scaling any volatility input by the current hour's seasonal factor, which lets you widen ahead of 00:00 and 14:00 by the clock instead of after the burst. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [The Art of Currency Trading - Brent Donnelly (Amazon)](https://www.amazon.com/s?k=The+Art+of+Currency+Trading+Brent+Donnelly&ref=aligrithm.com) ### 1.24 Alpha Decay Is Just Competition (and Papers Lie) URL: https://aligrithm.com/alpha-decay-is-just-competition-and-papers-lie/ Last updated: 2026-07-27T13:23:00.000Z Your strategy stops working and you reach for a physics metaphor. The edge "decayed," like a radioactive isotope obeying some law of nature. The metaphor is comforting and wrong. Nothing about your edge half-lifed on its own. Other people found the same trade and competed the return out of it. Alpha decay is not entropy. It is a crowd. Once you see decay as competition rather than decay as physics, the defenses change. You stop treating a dying curve as bad luck and start treating it as a signal that the inefficiency you found is no longer yours alone. The old article ["Why Systems Work Until They Don't"](https://aligrithm.com/why-systems-work-until-they-dont/) mapped the lifecycle (discovery, deployment, decay, death) and named crowding as a decay mechanism. This is the mechanism running the whole show. ## There are more funds than strategies Start with the supply-demand picture. Equities is a developed, picked-over market, and there are still opportunities in it. There are also far more hedge funds than there are distinct strategies. Do the arithmetic on that and the conclusion is forced: the same handful of ideas is being run by many teams at once. Novel ideas are mostly a myth. Whatever you discover, someone else has it or will have it soon. The only real questions are how many people share your idea and how good their version is. You are not racing to find something nobody has seen. You are estimating how crowded a known room already is. $$ \\alpha\_{\\text{realized}} \\approx \\frac{\\alpha\_{\\text{raw}}}{1 + N\_{\\text{competitors}}} $$ This is a heuristic, not a measured law, so treat it as intuition. The raw edge of an inefficiency gets split across everyone trading it. One participant keeps most of it. Ten participants chasing the same signal each get a sliver, and the act of all of them trading it pushes the price to where the signal no longer fires. The edge did not evaporate. It got divided until each slice fell below costs. That is what a decaying equity curve is telling you: the room filled up. ## The exponential shape, and why it is competition Crowded edges fade with a recognizable curve. Model the surviving edge as exponential decay with a half-life. $$ \\alpha(t) = \\alpha\_0 \\cdot \\left(\\tfrac{1}{2}\\right)^{t / h} $$ The half-life h is the time for the edge to fall by half. The old article ["Why Systems Work Until They Don't"](https://aligrithm.com/why-systems-work-until-they-dont/) put rough class-specific values on it: statistical arbitrage at two to three years, factor strategies at five to ten, trend at ten to twenty, market-making at half a year to two. Plug in h of two years and read the schedule: full edge today, half in two years, a quarter in four, an eighth in six. A strategy you deploy at a 2.0 Sharpe is a 1.0 Sharpe strategy in two years if it is the crowded kind, and you should size and plan as if that schedule is real. The half-life is short precisely when the edge is easy to find and copy. Stat-arb decays fast because it is a known game with low barriers. Trend decays slowly because it is uncomfortable to hold and capacity-hungry, so fewer participants pile in. The decay rate is a competition rate. The barriers from the theory-of-edge framework (operational pain, capacity limits, the discomfort of fat-tailed payoffs) are the things that slow the crowd down and stretch the half-life. ## The data edge, and what it costs One way to stay ahead of the crowd is to see what they cannot. That is the data edge, and it is expensive and real. A firm like QRT spends on the order of $100 million a year on data. The most expensive feeds are not satellite images or credit-card panels. They are traders bought off bank execution desks, human knowledge of flow that does not exist in any dataset you can buy. For an individual, the data edge is mostly out of reach, and the lesson is to know that, not to pretend otherwise. The number matters because it sets a breakeven you will never meet at that scale. $$ \\text{data edge pays} \\iff \\big(\\alpha\_{\\text{with data}} - \\alpha\_{\\text{without}}\\big) \\cdot \\text{AUM} \\; > \\; \\text{data cost} $$ To justify $100M of annual data spend you need the incremental alpha from that data, times the capital you run it on, to clear $100M. That math only closes for very large books, which is why the data edge concentrates at the top. Trying to win on data against firms spending nine figures is the same mistake as trying to out-compute them: you picked the one game where your disadvantage is largest. The way around it is not better data. It is a constraint they decline to take. ## Papers lie The cheapest place to source ideas is published research, and it is also the place most likely to hand you a corpse. You cannot read a paper, implement the strategy, and expect alpha to fall out. Papers lie to you, and the lying is structural, not malicious. The mechanism is survivorship and incentive. A published backtest is the one configuration out of many that cleared the journal's bar, which is the undocumented parameter sweep the old article "Why Quantitative Does Not Automatically Mean Scientific" warned about. By the time an edge is in print, it has been read by thousands of the exact people who would trade it, which crowds it before you finish reading the abstract. And the author has no incentive to publish the version that still works and every incentive to publish the version that gets cited. The defense is to verify everything yourself and trust nothing on the page. Treat a paper as a hypothesis to be tested on your own data with your own costs, not as a result to be deployed. Expect that the effect is weaker than reported, already crowded, or gone. The same applies to blogs and books and any public idea: exposure to many sources is how you generate candidates, and independent verification is the only thing that separates a live edge from a retold story. People steal alpha constantly in this industry, and the published edge is the most stolen of all. ## Reading the death versus the dip Competition-driven decay needs a different response than a normal drawdown, and confusing them is expensive in both directions. The old article ["When to Switch Off a Trading System"](https://aligrithm.com/when-to-switch-off-a-trading-system/) drew the line between a statistical breach (recent behavior stops matching history) and a structural breach (the inefficiency provably changed). Crowding is a structural breach. It deserves more weight than a price-based drawdown alone, because it kills the edge before the equity curve has finished confirming the damage. Practical read. If your edge is the crowded kind and the return is grinding lower while the trade gets more popular, more discussed, more obviously known, that is competition eating the alpha and it is not coming back by waiting. If the edge sits behind a real barrier and the drawdown looks like one of its historical drawdowns, you are inside the normal envelope and switching off is the expensive error. Size the kill decision to the cause. Decay-as-competition does not reverse; decay-as-noise does. ## Visualizing the crowd ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_311-visual-0.png) ## KEY POINTS - Alpha decay is competition, not physics. The edge did not half-life on its own. Other people found the same trade and split the return until each slice fell below costs. - There are more funds than distinct strategies. Novel ideas are mostly a myth; the real questions are how many share your idea and how good their version is. - Crowded edges fade exponentially. With a two-year half-life, a 2.0 Sharpe is a 1.0 Sharpe in two years. The half-life is short when the edge is easy to copy and long when a real barrier slows the crowd. - The data edge is real and concentrated. Spending on the order of $100M a year, mostly on humans from bank execution desks, only breaks even on a very large book, so it is out of reach for an individual. - Papers lie by construction: published results are the surviving configuration of a parameter sweep, they are crowded the moment they print, and authors are not incentivized to publish what still works. Verify on your own data and costs. - Treat crowding as a structural breach, which kills the edge before the curve confirms it. A crowded edge grinding lower while the trade gets more popular is not coming back by waiting; a barriered edge in a normal-looking drawdown is. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Evidence-Based Technical Analysis - David Aronson (Amazon)](https://www.amazon.com/s?k=Evidence-Based+Technical+Analysis+David+Aronson&ref=aligrithm.com) - [Systematic Trading - Robert Carver (Amazon)](https://www.amazon.com/s?k=Systematic+Trading+Robert+Carver&ref=aligrithm.com) ### 1.23 The Theory of Edge: Why the Market Pays You URL: https://aligrithm.com/the-theory-of-edge-why-the-market-pays-you/ Last updated: 2026-07-27T13:23:00.000Z Most traders look for edge in the wrong place. They hunt for an insight, a pattern, a signal nobody else has found. If that is the game, the opponents are firms with more data, more compute, and more capital than you will ever assemble. You lose that game before you sit down. The reachable edge has nothing to do with knowing something others do not. You get paid for doing something others will not. The question to carry around is not "what do I know that others don't." It is "what can I do that others won't." That reframing is the whole article. The rest is the taxonomy of the constraints that pay, and how to tell which one is actually cutting you a check. ## Getting paid is a transaction, not a reward Decompose any gross return into the pieces that produced it. $$ R = R\_f + \\beta \\cdot R\_{\\text{market}} + \\sum\_k \\lambda\_k \\, x\_k + \\alpha $$ Read it left to right. R\_f is the risk-free rate you earn for doing nothing. The beta term is the market drift you capture for holding the index, which the old article "Why Quantitative Does Not Automatically Mean Scientific" called out as the free long-bias return that masquerades as skill. The sum is a set of risk premia: each x\_k is an exposure (carry, illiquidity, volatility selling, credit) and each lambda\_k is the price the market pays you to hold that exposure through the discomfort. Alpha is the leftover, the part you take from another participant who priced something wrong. Retail content sells the last term and ignores the middle one. The middle one is where most reachable money sits. Risk premia are not edge in the clever sense. They are payment for sitting in a position other people want to get out of. You provide liquidity to the person who wants out, you absorb the volatility they cannot stomach, and you collect the premium for being the buyer at the moment everyone else is a seller. You are not outsmarting that seller. You are out-tolerating them. ## The four constraints that pay The constraints worth building a business on fall into four buckets. Each one is a reason a better-resourced participant declines the trade and leaves it for you. **Operational pain.** Some edges require work that is annoying enough that capable firms pass. Reconciling messy data feeds, handling physical delivery, managing a dozen exchange integrations with bad APIs, dealing with tax or settlement plumbing. None of it is intellectually hard. All of it is a grind, and the grind is the moat. The firm that could out-model you decides the operational drag is not worth the capacity, so the trade stays open. **Capacity versus skill.** The most valuable of the four. A niche trade has a hard capacity ceiling, and that ceiling makes it invisible to size. The people who know how to run it do not care, because it cannot move the needle on a large book, and the people who would care cannot find or build it. Those who know don't care; those who care don't know. That gap is where a small trader lives. **Risk premia.** Covered above. You take a risk off someone's hands and get paid for the transfer. Carry, insurance-like volatility selling, illiquidity. The payment is real and recurring, and the price of it is that the risk occasionally arrives and hurts. **Small margins.** Edges too thin for a big cost base to bother with. A trade that nets a few basis points per turn is worthless to a fund carrying large fixed costs and answering to investors who want a headline Sharpe. Run lean, and a few basis points compounded across many turns is a business. ## The capacity math, worked The capacity-versus-skill edge is worth a number, because the number is why it survives. Take a niche trade with $5 million of capacity that returns 40% a year at the size it can hold. That is $2 million of profit. Real money to an individual. Now hand it to a $20 billion fund. $$ \\text{contribution} = \\frac{\\text{capacity} \\times \\text{return}}{\\text{AUM}} = \\frac{5\\text{M} \\times 0.40}{20{,}000\\text{M}} = 0.01\\% $$ The trade adds one basis point to the fund's annual return at full capacity, before the cost of the team, the infrastructure, and the risk committee's attention. The fund passes. Not because they cannot run it, but because it is not worth the seat it would take up. The same trade that is beneath their notice is a third of your year. The ceiling that makes it useless to them is exactly what keeps it uncrowded for you. This is the structural reason a small operator can survive in a market dominated by larger, smarter teams. You are not competing with them on the trades they want. You are taking the trades they decline. ## Know which check you are cashing Edge that you cannot name is edge you cannot defend. Two questions force the issue. First, why do you make money? The honest answers are short: you harvest a risk premium, you have a model edge, you have a data or information edge, you have superior access to cost or rebates, or you do not have an edge at all. Most traders cannot pick one. If you cannot, assume the last option until proven otherwise. Second, why do you keep the money? An edge with no barrier gets competed to zero. The barriers are specific: a fat-tailed payoff that scares competitors out (the kind that kills people who size it wrong), an infrastructure barrier, a knowledge barrier, a psychological barrier, an opportunistic special-situation nature with nothing to fit, or a structural barrier from unfair access. Every one of these is measured against one benchmark: the dollar-volume-weighted participant on the other side of your trades. You do not need to beat everyone. You need to be doing something the average dollar across from you will not do. $$ \\text{edge exists} \\iff \\big(\\text{your willingness} \\;-\\; \\text{marginal participant's willingness}\\big) > \\text{cost} $$ The marginal participant is the dollar-weighted average of who you trade against. If your only advantage is that you are willing to hold the position they are dumping, that is a real edge as long as the premium clears your costs. If you have no such gap, you are the marginal participant, and the expected value of the game is your cost base running in reverse. ## Mixing and stacking The constraints are not mutually exclusive, and the good operators run several at once and rotate them with conditions. When alts are running, add depth to alts and collect the premium for providing it. When funding is rich on perps, run the funding trade. When one market goes wide and illiquid, the operational-pain and small-margin edges fatten. The mix is the point. Each individual constraint is a thin, decaying source, which is the same lesson the old article ["The Death of the Single-System Trader"](https://aligrithm.com/the-death-of-the-single-system-trader/) made about rules: one source of edge is one regime change from zero, and the defense is several uncorrelated ones running in parallel. There is also a compounding structure. New alpha sources are additive. Better execution is a multiplier on the edge you already have. A firm like Citadel does not win by inventing every edge it runs. It wins from scale, from being the largest participant with the best cost structure, and it multiplies whatever raw edges it touches through superior execution. You will not have that multiplier. You compete on the additive side, by finding constraints the multiplier-firms decline to bother with. ## Visualizing the trade-off ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_310-visual-0.png) ## KEY POINTS - The reachable edge is not knowing what others don't. It is doing what others won't: absorbing volatility, providing liquidity to the person who wants out, out-tolerating rather than outsmarting. - Gross return splits into the risk-free rate, market beta drift, a sum of risk premia, and alpha. Retail content sells the alpha term and ignores the risk premia, which is where most reachable money sits. - Four constraints pay: operational pain, capacity-versus-skill, risk premia, and small margins. Each is a reason a better-resourced firm declines the trade. - Capacity-versus-skill is the strongest. A $5M-capacity trade returning 40% adds one basis point to a $20B fund, so they pass, and a third of your year stays uncrowded. - Name your edge with two questions: why you make money (risk premium, model, data, cost access, or nothing) and why you keep it (a specific barrier measured against the dollar-volume-weighted participant). - One constraint is one source of edge, and it decays. Run several and rotate them with market conditions, the same diversification logic that keeps a multi-system trader alive. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Evidence-Based Technical Analysis - David Aronson (Amazon)](https://www.amazon.com/s?k=Evidence-Based+Technical+Analysis+David+Aronson&ref=aligrithm.com) - [Systematic Trading - Robert Carver (Amazon)](https://www.amazon.com/s?k=Systematic+Trading+Robert+Carver&ref=aligrithm.com) ### 6.47 Building a Trend Follower, Component by Component URL: https://aligrithm.com/building-a-trend-follower-component-by-component/ Last updated: 2026-07-27T13:25:53.000Z Trend following is the most straightforward systematic strategy to describe: buy what has been going up, sell what has been going down. It is also a roughly 0.2-Sharpe strategy on any single market, which is to say a bad one, and the gap between that description and a fund that runs billions is entirely construction. The signal is the part everyone fixates on and the part that matters least. The work lives in the six components wrapped around it. Walk them in order and the strategy assembles like a machine where every part earns its place. The pipeline has seven stages: universe selection, trend detection, mapping trend strength to a target position, sector and asset-class weights, portfolio risk targeting, trading rules, and execution. None is optional, and the order is not arbitrary. ## Universe selection: breadth is the whole game Single-market trend following posts about 0.2 Sharpe, which nobody would trade. The strategy is rescued by breadth, and breadth scales in a specific way. $$ \\text{Sharpe}\_{\\text{portfolio}} \\;\\propto\\; \\sqrt{\\text{breadth}} $$ $$ \\text{breadth} = \\text{number of independent markets you trade} $$ Read that as: the portfolio Sharpe grows with the square root of the number of genuinely independent markets you run the strategy across. The word doing the work is independent. If you already trade S&P 500 and Nasdaq futures, adding Dow futures buys you almost nothing, because the Dow is the same bet you already hold, and the square-root law counts independent markets, not tickers. Add natural gas instead and you add a market whose trends fire on their own schedule, and the breadth count actually rises. This is the old article ["The Death of the Single-System Trader"](https://aligrithm.com/the-death-of-the-single-system-trader/) applied to markets rather than systems: correlated components give you almost no diversification, so the count that matters is the effective independent count, not the raw one. A typical futures trend follower trades 60 to 80 markets across the major exchanges. A larger and more sophisticated shop pushes into 300 to 400, reaching for niche futures like power, freight, and crypto, plus FX forwards, rate swaps, and CDS indices, because each genuinely different market adds another independent draw to the average. Expanding the universe this way can lift the strategy's Sharpe by a factor of one and a half to two, which is a larger improvement than any amount of signal tuning will ever give you. The universe is the highest-leverage decision in the build, and it comes first for that reason. ## Trend detection: a vol-normalized low-pass filter The signal matters less than its reputation, because it is usually obvious whether a market is trending, and the only job is to capture that quantitatively without overfitting. The standard tool is a moving-average crossover: a fast moving average of log price minus a slow one, normalized by volatility. $$ \\text{raw trend} \\;=\\; \\text{MA}\_{\\text{fast}}(\\log P) \\;-\\; \\text{MA}\_{\\text{slow}}(\\log P) $$ $$ \\text{signal} \\;=\\; \\frac{\\text{raw trend}}{\\sigma} $$ Read that as: take a fast and a slow moving average of the log price, subtract to get the raw trend, then divide by the market's volatility so the signal is in risk units and comparable across a fast market and a slow one. The vol normalization is the same logic as the old article ["Why ATR Normalization Is More Than a Volatility Trick"](https://aligrithm.com/why-atr-normalization-is-more-than-a-volatility-trick/): without it, a high-vol market dominates the book for no good reason. The crossover survives because it is simple, hard to overfit, and acts as a low-pass filter, stripping the high-frequency return noise and leaving the slow drift, which is exactly the thing the old article ["Efficiency Ratio Explained for Traders"](https://aligrithm.com/efficiency-ratio-explained-for-traders/) measures from the other direction. Trend followers typically detect on horizons from one month to one year, centered around three to six months. Shorter than that and you are a different, smaller business; longer than a year is rare. ## Mapping trend to position: cap the conviction A raw signal of arbitrary size is not a position. You pass it through a function that maps trend strength to a target risk allocation, and the point of the function is to stop you from building forever as the trend gets stronger. Two shapes are common. $$ \\text{position} = \\text{sigmoid}(\\text{signal}) \\qquad\\text{or}\\qquad \\text{position} = \\text{signal}\\cdot e^{-\\,\\text{signal}^2} $$ Read that as: either a sigmoid, which saturates so the position flattens out at a maximum once the trend is strong enough, or a function like signal times e to the minus signal-squared, which rises, peaks, then actually shrinks the position for extreme signal values. The second shape encodes overextension: a trend that has run absurdly far is more likely to snap back, so you trim rather than press. Both refuse to let a single roaring market take over the whole risk budget, which is the same discipline as capping any one bet, and the choice between them is a view on whether very strong trends keep paying or tend to reverse. There is a tempting addition here that deserves a skeptical flag. You can add a long bias to this function for markets that drift up on average, equities, credit, fixed income, and it does tend to improve backtested Sharpe. It also quietly guts the best property the strategy has, its ability to make money in an equity or bond bear market, because you have bolted a permanent long onto a strategy whose whole selling point is that it does not care which way the market goes. Many managers do it anyway for the Sharpe bump. The honest cost is that you are no longer running a pure trend follower, you are running a trend follower plus a hidden long, and the hidden long is the part that will hurt you in the crisis you bought the strategy to survive. ## Sector and asset-class weights: undo the accidental tilt Equal-weighting every market sounds neutral and is not. If you trade twice as many equity-index futures as rate futures, equal weighting leaves you structurally overweight equities, so a single bad month for stocks dominates a portfolio that was supposed to be balanced. Fix it by weighting up the under-represented sector, doubling the rate positions in that example, so risk is balanced across asset classes rather than across ticker counts. The same correction applies within a sector: commodity markets are less correlated with each other than rate markets are, so a basket of commodities carries more independent breadth per market than a basket of rates, and the weights should reflect that. This stage is housekeeping, and skipping it means your carefully chosen universe quietly collapses back into a concentrated bet. ## Portfolio risk targeting: size inverse to volatility Now size the positions. The standard rule makes each market's position inversely proportional to a forecast of that market's volatility. $$ \\text{size}\_i \\;\\propto\\; \\frac{\\text{target position}\_i}{\\sigma\_i^{\\text{forecast}}} $$ $$ \\sigma\_i^{\\text{forecast}} \\;=\\; w\\,\\sigma\_i^{\\text{long-run}} \\;+\\; (1-w)\\,\\sigma\_i^{\\text{short-run}} $$ Read the top line as: for a given trend strength, you size the position down in proportion to how volatile the market is, so every market contributes about the same amount of risk to the book and no single one can dominate. That equal-risk-contribution idea is the old article ["Why Volatility-Adjusted Position Sizing Matters"](https://aligrithm.com/why-volatility-adjusted-position-sizing-matters/) applied across a whole portfolio. The volatility forecast in the denominator can be as elaborate as you like, but the bottom line is the practical truth: you capture most of the benefit with a simple blend of a long-run volatility estimated over years and a short-run realized volatility over the last 30 to 60 days. The long-run term keeps the estimate steady, the short-run term lets it react when a market wakes up, and the blend is robust precisely because it is not trying to be clever. ## Trading rules and execution: stop trading the noise The last two stages are where a clean backtest meets its costs. A naive implementation re-sizes to the exact target every day, and the target jiggles constantly because the volatility forecast and the signal both wobble, so you pay round-trip costs on changes that carry no information. The fix is a buffer: leave the position alone while it sits inside a no-trade band around the target, and only trade when it drifts outside the band. That converts a stream of tiny noise trades into occasional meaningful ones, which is the cost-side version of the same noise problem the old article ["Signal Averaging: Killing Noise with Redundant Alphas"](https://aligrithm.com/signal-averaging-killing-noise-with-redundant-alphas/) attacks on the signal side. Execution then spreads the resulting trades over time to keep market impact small, because a trend follower's edge is measured in fractions of a percent and clumsy execution gives all of it back. These stages produce no alpha. They decide how much of the alpha from the first five stages actually reaches your account. ## Visualizing the pipeline ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_336-visual-0.png) ## KEY POINTS - Trend following on a single market is about 0.2 Sharpe. Everything good about it comes from the six components wrapped around the signal, not the signal. - Universe selection comes first and matters most. Sharpe grows like the square root of the number of independent markets, so adding a correlated market buys nothing and adding an uncorrelated one buys a lot. Expanding the universe can lift Sharpe one and a half to two times. - The trend signal is a fast minus slow moving average of log price, normalized by volatility. It is a low-pass filter, simple and hard to overfit, run on one-month to one-year horizons centered on three to six months. - Map trend strength to a target through a saturating or overextension-trimming function so no single roaring market eats the risk budget. Adding a long bias for equities and bonds lifts backtested Sharpe but destroys the strategy's bear-market hedge; that is a real cost, not a free lunch. - Weight sectors and asset classes to undo the accidental tilt from trading more markets in one sector than another. Equal-weighting tickers is not equal-weighting risk. - Size positions inversely to a volatility forecast so every market contributes equal risk. A simple blend of long-run and 30 to 60 day realized vol captures most of the benefit. - Trading rules and execution decide how much alpha survives. Use a no-trade buffer to stop trading the noise, and spread execution to keep impact small. These stages make no alpha; they protect it. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Systematic Trading - Robert Carver (Amazon)](https://www.amazon.com/s?k=Systematic+Trading+Robert+Carver&ref=aligrithm.com) - [Trading Systems - Urban Jaekle Emilio Tomasini (Amazon)](https://www.amazon.com/s?k=Trading+Systems+Urban+Jaekle+Emilio+Tomasini&ref=aligrithm.com) ### 9.9 Arbitrage Is Just Projection: One Calculation Gives Trade, Profit, and Target Prices URL: https://aligrithm.com/arbitrage-is-just-projection-one-calculation-gives-trade-profit-and-target-prices/ Last updated: 2026-07-27T13:26:09.000Z The article "The Marginal Polytope" gave you a yes/no test: are the prices inside the arbitrage-free shape or outside it. This article answers the three questions you actually care about once the answer is "outside." What is the best possible trade. How much profit does it guarantee. What should the prices be. All three fall out of a single calculation: find the closest point in the shape to the current prices. That closest-point operation is a projection, and the entire structural edge in this pillar reduces to it. This is the keystone. Every earlier article is a prerequisite for it and every later article implements or defends it. ## From "best trade" to "closest point" Set up the problem the way a trader would. You hold a portfolio, call it delta, where a positive entry buys a contract and a negative entry sells it. The market maker charges you for the trade through a cost function. Against that, the worst-case payout is the least the portfolio collects across all outcomes. Guaranteed profit is worst-case payout minus what you paid. $$ \\Pi(\\delta) = \\min\_{z \\in Z}\\, \\delta \\cdot z \\;-\\; \\big\[\\, C(\\theta + \\delta) - C(\\theta) \\,\\big\] $$ The first term is the worst payout, the minimum over valid outcomes z of the portfolio's payout. The bracket is the cost of moving the market state from theta to theta plus delta, where C is the market maker's cost function. If that difference is positive under every outcome, you have risk-free profit. The best trade maximizes it, and the structure is a fight: you pick the portfolio to maximize, nature picks the worst outcome to minimize. $$ \\max\_{\\delta}\\; \\min\_{z \\in Z}\\; \\big\[\\, \\delta \\cdot z - C(\\theta + \\delta) + C(\\theta) \\,\\big\] $$ You choose delta. Nature, seeing your choice, chooses the outcome that hurts you most. You want the portfolio that still profits after nature does its worst. A min-max fight like this looks intractable, because your best move depends on nature's response, which depends on your move. Convex duality cuts the knot. ## The duality swap, and why it turns into a projection Strong duality is a result that lets you swap the order of the min and the max without changing the answer. On the left, "pick the best portfolio, assuming nature picks the worst outcome." On the right, "pick the closest fair-price vector, then find the best portfolio for that target." Same number, different problem, and the right-hand version is one you can solve. $$ \\max\_{\\delta}\\, \\min\_{p \\in M} \\big\[\\cdots\\big\] \\;=\\; \\min\_{p \\in M}\\, \\max\_{\\delta} \\big\[\\cdots\\big\] $$ Work the inner maximization over delta with a target price p held fixed. Setting the gradient to zero gives the trade that moves the market to that target, and the profit from it turns out to equal a specific distance between the target and the current state. The distance is the Bregman divergence, and after the dust settles the whole min-max reduces to this. $$ \\Pi^\* = \\min\_{p \\in M}\\, D(p \\,\\|\\, \\theta) = D(p^\* \\,\\|\\, \\theta), \\qquad p^\* = \\arg\\min\_{p \\in M} D(p \\,\\|\\, \\theta) $$ Read it slowly. D is a distance, the Bregman divergence, measuring how far a candidate fair price p is from the current market state theta. You minimize that distance over all fair prices p in the polytope M. The minimizing point p-star is the projection of the current state onto the arbitrage-free set, the closest legal price to the illegal one you observe. And the punchline: the maximum guaranteed profit equals that minimum distance. Closer prices, less profit. Prices already inside M, distance zero, no arbitrage. The best trade and the best profit are the same object seen two ways. ## One projection, three answers The projection does not give you a number. It gives you the complete trade. $$ \\text{Trade: } \\delta^\* = \\nabla C^\*(p^\*) - \\theta, \\qquad \\text{Profit: } D(p^\* \\,\\|\\, \\theta), \\qquad \\text{Target: } p^\* $$ First, what to trade: the portfolio delta-star that moves the state to the projected price. The gradient of the conjugate function C-star maps a target price back to the market state that produces it, an inverse lookup, and the trade is the difference between that state and where the market is now. Second, how much: the divergence, the distance you projected across, is the guaranteed profit in dollars. Third, where prices should be: p-star itself is what a coherent market would show. If the projection lands exactly on the current prices, the distance is zero and there is nothing to do. One calculation, three outputs, no separate steps for detection, sizing, and pricing. A concrete anchor. Binary market, YES at 40 cents and NO at 40 cents, summing to 80\. The arbitrage-free set requires the sum to be one. Project the (0.40, 0.40) point onto that line and you land at (0.50, 0.50). The distance you crossed is the profit: buy both for 80 cents total, collect a dollar at resolution, keep 20 cents. The projection produced the trade (buy one of each) and the profit (20 cents) in the same motion. ## Why the distance is not a ruler you'd expect The obvious way to measure "closest" is straight-line distance, the Euclidean one. It is wrong here, and the reason matters because using the wrong ruler produces the wrong trade. Prices in these markets are probabilities, and probabilities do not move on a linear scale. A shift from 1 percent to 11 percent is an enormous change in what the market believes; in log-odds it moves roughly from minus 4.6 to minus 2.1\. A shift from 50 percent to 60 percent is a small change; in log-odds, from 0 to about 0.4\. Straight-line distance calls both of them the same 10-cent move. They are not the same, and a trade sized as if they were will misprice the tails badly. The right ruler for a log-scoring market maker is the Kullback-Leibler divergence, which measures distance in the same information units the market maker itself uses. $$ D(p \\,\\|\\, \\theta) = \\sum\_i \\mu\_i \\ln \\frac{\\mu\_i}{p\_i(\\theta)} $$ The sum runs over securities. Each term weights the log-ratio of the target probability to the current implied probability. Near the boundaries, where prices approach 0 or 1, this ruler stretches, so a 10-cent move at the extremes counts for far more than a 10-cent move in the middle. That is the correct accounting, because that is how the market maker prices risk. Measure with KL and you get the optimal trade. Measure with Euclidean distance and you leave money on the table or overtrade the tails. The stretching near the boundaries also causes a numerical problem that the article "Why Your Arbitrage Solver Crashes at 99 Cents" is entirely about, so file that away. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_351-visual-0.png) ## The guarantee is real, but it leaks Two honesty checks, because the word "guaranteed" invites carelessness. First, you will not compute the exact projection in practice. The article ["Frank-Wolfe: Solving a Quintillion-Vertex Problem in 100 Steps"](https://aligrithm.com/frank-wolfe-solving-a-quintillion-vertex-problem-in-100-steps/) returns an approximate projection with a measured gap between where it stopped and the true optimum. The profit you can count on is the approximate distance minus that gap. $$ \\Pi \\;\\ge\\; D(\\hat{p} \\,\\|\\, \\theta) - g(\\hat{p}) \\;\\ge\\; 0 $$ The first distance is the profit you'd get if the approximation were exact. The gap g is what you are leaving on the table by stopping early. The difference is what you keep no matter what, and convex duality guarantees it is never negative. An approximate solution can capture less than the maximum, but it never manufactures a fake profit that turns into a loss. That property is what makes stopping early safe. Which sets up the second check: when to stop and trade. Three rules. Alpha-extraction: stop when the gap is small enough that you have captured at least 90 percent of the available profit, because chasing the last 10 percent costs time and the market moves. Near-free: if the whole guaranteed profit is under 5 cents, do not trade at all, because execution risk and fees will eat it. Forced stop: a time limit or an incoming order forces you out with the best iterate you have. The 5-cent floor is not a rounding convenience. It is the line below which the theoretical guarantee stops surviving the real costs cataloged in the article "Execution Is Part of Expected Value." The reason the discipline matters shows up in the data. The top account in the 2024 to 2025 Polymarket study cleared $2,009,631 across 4,049 trades at a win rate above 90 percent. A 90-plus percent win rate is what this machinery produces when you only pull the trigger on a large, established profit guarantee and skip everything below the floor. It is not a forecasting record. It is the signature of a projection that refused to trade the marginal cases. ## KEY POINTS - The best arbitrage trade is the projection of the current market state onto the arbitrage-free set: find the closest legal price to the illegal one you observe. Maximum guaranteed profit equals the distance you project across. - A min-max fight (you pick the portfolio, nature picks the worst outcome) becomes a solvable projection through strong duality, which swaps the order of min and max without changing the answer. - One projection returns three things at once: the trade (the portfolio that moves the state to the projected price), the profit (the divergence in dollars), and the target prices (what a coherent market would show). Distance zero means no arbitrage. - Measure distance with KL divergence, not straight-line distance. Probabilities move on a log scale, so a move from 0.01 to 0.11 dwarfs a move from 0.50 to 0.60; the KL ruler stretches near the boundaries to match how the market maker prices, and Euclidean distance gives the wrong trade. - The guarantee survives approximation: guaranteed profit is the approximate distance minus the solver gap, and convex duality keeps that quantity non-negative, so stopping early never fabricates a loss. - Stop on three rules: capture at least 90 percent of the profit, skip anything under a 5-cent guarantee, and bail out on a time limit or incoming order. The 90-plus percent win rate of the top real account is what this discipline looks like, not a forecasting skill. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Arbitrage-Free Combinatorial Market Making via Integer Programming - Kroer et al. (2016)](https://arxiv.org/abs/1606.02825?ref=aligrithm.com) - [An Optimization-Based Framework for Automated Market-Making - Abernethy et al. (2011)](https://scholar.google.com/scholar?q=%22An+Optimization-Based+Framework+for+Automated+Market-Making%22+Abernethy+et+al.+2011&ref=aligrithm.com) - [Logarithmic Market Scoring Rules for Modular Combinatorial Information Aggregation - Hanson (2007)](https://scholar.google.com/scholar?q=%22Logarithmic+Market+Scoring+Rules+for+Modular+Combinatorial+Information+Aggregation%22+Hanson+2007&ref=aligrithm.com) - [Convex Optimization - Boyd and Vandenberghe (2004)](https://scholar.google.com/scholar?q=%22Convex+Optimization%22+Boyd+and+Vandenberghe+2004&ref=aligrithm.com) ### 9.8 The Marginal Polytope: One Shape That Contains Every Fair Price URL: https://aligrithm.com/the-marginal-polytope-one-shape-that-contains-every-fair-price/ Last updated: 2026-07-27T13:26:08.000Z Every arbitrage-detection question you can ask about a set of prediction markets collapses into one geometric question. Is the price vector inside a certain shape, or outside it. Inside, no structural trade exists and you should stop looking. Outside, a guaranteed-profit portfolio exists and the boundary of the shape hands you the exact trade. The shape has a name, the marginal polytope, and learning to think in it replaces a pile of ad-hoc coherence checks with a single test. The article "The Dutch Book Theorem, Plainly" handled the toy case, prices that fail to sum to one. That rule misses most of what happens in real markets: implications, mutual exclusions, conditionals, tangled combinatorial dependencies. This article generalizes all of it into one object and one theorem, and it is the intellectual center of the structural half of this pillar. ## Building the shape from the valid outcomes Start where the article "Outcome Geometry" left off, with the valid payoff set, the small collection of ones-and-zeros rows that describe what the contracts pay under each outcome that can actually happen. Those rows are corners. The marginal polytope is the solid shape you get by filling in everything between them. $$ M = \\text{conv}(Z) = \\Big\\{ \\sum\_{\\omega} \\lambda\_\\omega\\, \\varphi(\\omega) \\;:\\; \\lambda\_\\omega \\ge 0,\\; \\sum\_\\omega \\lambda\_\\omega = 1 \\Big\\} $$ The word conv is convex hull, the smallest shape with no dents that wraps a set of points. The formula spells it out: take the valid payoff vectors phi, weight them with numbers lambda that are all non-negative and sum to one, and add them up. Weights that are non-negative and sum to one are a probability distribution. So every point in M is a weighted blend of valid payoffs, and the weights are a probability over outcomes. That gives the reading that makes the shape worth caring about. A price vector sits inside M exactly when there is some probability distribution over the outcomes that produces those prices as its implied payout probabilities. Inside M means "these prices are consistent with some coherent view of the world." Outside M means "no probability distribution, however contrived, could ever price things this way." The second case is the mispricing, and it is guaranteed, not a matter of taste. ## The characterization theorem, and why the boundary is the trade One theorem ties membership to arbitrage with no gap between them. $$ p \\text{ is arbitrage-free} \\iff p \\in M $$ The double arrow is "if and only if." Arbitrage-free is the same statement as inside the polytope. No arbitrage exists precisely when the prices are in the shape, and arbitrage exists precisely when they are out of it. This subsumes the Dutch book: prices failing to sum to one are just one way to fall outside M, the flat face where the sum-to-one constraint lives. The proof is where the trade comes from, so it is worth walking, not skipping. Suppose the prices are outside M. There is a theorem in convex geometry, the separating hyperplane theorem, that says a point outside a convex shape can always be split off from it by a flat wall. Everything in the shape sits on one side of the wall; the outside point sits on the other. $$ p \\notin M \\;\\Rightarrow\\; \\exists\\, \\delta \\text{ such that } \\delta \\cdot p < \\min\_{z \\in Z}\\, \\delta \\cdot z $$ Read delta as a portfolio again. The left side, delta dot p, is what the portfolio costs at current prices. The right side, the minimum over valid outcomes of delta dot z, is the worst payout that portfolio can ever return. The wall guarantees the cost is strictly below the worst-case payout. You pay less than the least you will ever collect. That is a locked profit, and the direction of the wall is the direction of the trade. The separating hyperplane is not an abstraction. It is the trading strategy, written as a vector. The other direction is the honesty check. Suppose the prices are inside M. Then they are a probability blend of valid payoffs, and any portfolio's cost is that same blend of the portfolio's payouts across outcomes. A blend sits between the minimum and the maximum of the things blended. So the cost lands between the worst and best payout, and no portfolio can guarantee it collects more than it paid. Inside the shape, the free money is gone. ## The worked case you can hold in your head Two securities, YES and NO, on one event. The valid payoffs are (1,0) if YES resolves and (0,1) if NO resolves. Two corners. The convex hull of two points is the line segment between them, and every point on that segment has its coordinates summing to one. So M here is the line "YES price plus NO price equals one." Simple, and it is the whole shape. Now price the market at (0.40, 0.40). The sum is 0.80, below one, so the point sits below the segment, outside M. The separating wall is the portfolio (1, 1): buy one of each. Cost is 0.40 plus 0.40, which is 0.80\. Worst-case payout is the smaller of (1,1) dotted with each corner, which is the smaller of 1 and 1, so 1.00\. Cost 0.80, worst payout 1.00, guaranteed profit 20 cents. That is the Dutch book from the previous article, recovered as a point-outside-a-line-segment. Price it instead at (0.55, 0.45). Sum is 1.00, so the point sits on the segment, inside M. Try to build a winning portfolio and you cannot; every combination that wins if YES resolves loses if NO resolves. The 20 cents is gone because the prices are now consistent with a real probability, namely 55 percent. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_350-visual-0.png) ## Why you cannot just check membership directly The two-security case fits on a napkin. Real markets do not. The polytope has at most as many corners as there are valid outcomes, and that count is the runaway number from the article "Outcome Geometry." | Market | Securities | Number of outcomes | | -------------------------- | ---------- | --------------------- | | Single binary | 2 | 2 | | Ten independent binaries | 20 | 1,024 | | NCAA tournament (63 games) | over 5,000 | about 9.2 quintillion | Testing membership by listing the corners is hopeless when there are 9.2 quintillion of them. You cannot write down the shape, let alone check whether a point is inside it. This is the wall the rest of the structural pillar spends its time getting around: the article "Trillions of Outcomes, 200 Constraints" describes the shape with a handful of linear rules instead of a list of corners, and the article ["Frank-Wolfe: Solving a Quintillion-Vertex Problem in 100 Steps"](https://aligrithm.com/frank-wolfe-solving-a-quintillion-vertex-problem-in-100-steps/) tests membership and finds the trade by touching only about a hundred of those corners. The polytope is the right object. Enumerating it is not the right method. ## The hidden case, and the empirical scale The dangerous mispricings are not the ones where a single market fails to sum to one. Those are visible. The dangerous ones hide across markets that each look fine. Take Market A, "candidate wins State X," at 0.52 and 0.48, summing to a clean dollar. Take Market B, "wins X by more than 5 points," at 0.32 and 0.68, also a clean dollar. Each is coherent alone. But winning by more than 5 implies winning, so the joint outcome space has three corners, not four, and the joint price vector (0.52, 0.48, 0.32, 0.68) can sit outside the three-corner polytope even though every individual book passes the sum-to-one test. That is where the money actually was. The 2024 to 2025 Polymarket study found 1,576 logically dependent market pairs in the presidential race, of which 13 were manually confirmed as exploitable for combinatorial arbitrage. Zoom out to single markets and the numbers get larger: across 17,218 conditions examined, 7,051 (41 percent) had single-market arbitrage, with typical mispricing around 60 cents on the dollar of payout. Read that last figure carefully. Buying a dollar of guaranteed payout for roughly 60 cents was a regular occurrence for anyone with the infrastructure to detect it and the speed to act. The edge was real and large. It was also gated behind exactly the computational machinery the next three articles build, which is why 41 percent of conditions being mispriced did not translate into 41 percent of participants making money. ## KEY POINTS - The marginal polytope M is the convex hull of the valid payoff vectors: every point inside it is a probability blend of outcomes, so a price vector is inside M exactly when some coherent view of the world produces those prices. - The characterization theorem is exact: prices are arbitrage-free if and only if they lie in M. This absorbs the Dutch book as the single flat face where prices sum to one. - When prices sit outside M, the separating hyperplane theorem gives a wall between the point and the shape, and that wall is the guaranteed-profit portfolio. The geometry hands you the trade, not just a yes/no. - Worked case: with YES and NO, M is the line "prices sum to one." Prices (0.40, 0.40) sit below it, and the portfolio (1,1) locks 20 cents; prices (0.55, 0.45) sit on it and no winning portfolio exists. - You cannot test membership by listing corners: an NCAA-scale market has about 9.2 quintillion of them. The polytope is the right object; enumeration is the wrong method, which is why later articles use linear constraints and Frank-Wolfe. - The costly mispricings hide across markets that each look fair alone. A 2024 to 2025 study found 1,576 dependent pairs in one presidential race and 41 percent of 17,218 conditions carrying single-market arbitrage, typical mispricing near 60 cents on the dollar, all reachable only with the right computational infrastructure. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Convex Optimization - Boyd and Vandenberghe (2004)](https://scholar.google.com/scholar?q=%22Convex+Optimization%22+Boyd+and+Vandenberghe+2004&ref=aligrithm.com) - [Arbitrage-Free Combinatorial Market Making via Integer Programming - Kroer et al. (2016)](https://arxiv.org/abs/1606.02825?ref=aligrithm.com) - [Unravelling the Probabilistic Forest: Arbitrage in Prediction Markets - Saguillo et al. (2025)](https://arxiv.org/abs/2508.03474?ref=aligrithm.com) ### 9.7 The Dutch Book Theorem, Plainly, and Why Polymarket Will Always Have Arbitrage URL: https://aligrithm.com/the-dutch-book-theorem-plainly-and-why-polymarket-will-always-have-arbitrage/ Last updated: 2026-07-27T13:26:08.000Z A binary market shows YES at 60 cents and NO at 50 cents. Sum them: $1.10\. Exactly one of the two will pay a dollar at resolution. Sell one of each, collect $1.10 now, pay out $1.00 later, keep 10 cents no matter what happens. No forecast. No opinion about the event. The prices contradicted themselves and the contradiction was money. That trade has a name from 1937, and the theorem behind it explains why a platform like Polymarket cannot stop producing these opportunities without breaking the thing that makes it fast. The article "Outcome Geometry" set up the machinery: contracts, the outcome space, the valid payoff set. This article uses it to answer one decision. Are the prices in front of you logically consistent, or is there a portfolio that wins under every outcome. If the second, the prices are incoherent, and incoherent prices are exploitable by definition. ## What coherent prices actually mean Coherent is a precise word here, not a compliment. A set of prices is coherent if you can read them as probabilities without contradicting yourself. Three conditions have to hold at once. First, no negative prices. A contract cannot cost less than nothing. Second, normalization: across a set of outcomes that are mutually exclusive and cover every case, the prices sum to one. Exactly one outcome happens, so the implied probabilities have to add to a whole. Third, logical consistency: if event A forces event B to be true, then A cannot be more likely than B, so its price cannot be higher. "Wins by more than 5 points" implies "wins," so it cannot trade above "wins." Pass all three and the prices correspond to some genuine probability distribution over the outcomes. Fail any one and the prices are incoherent. The 60-and-50 market fails the second condition, and that failure is not a rounding quirk. It is a guaranteed loss for whoever is on the wrong side, which means a guaranteed profit for whoever notices. ## The theorem, stated as a trader would use it The de Finetti result is short and it is the mathematical definition of arbitrage. $$ \\text{If } p \\text{ is not a coherent probability measure, then there exists a portfolio } \\delta \\in \\mathbb{R}^{|\\mathcal{I}|} \\text{ with } \\delta \\cdot z > 0 \\text{ for every outcome } z. $$ Read it in plain terms. The letter p is the price vector, the list of all the prices. The delta is a portfolio, a list of how many units of each contract you buy (positive) or sell (negative). The dot with z is the profit under a specific outcome. The claim: if the prices are incoherent, some combination of buys and sells profits under every single way the world can end. The converse holds too, and it is the half that keeps you honest. If the prices are coherent, no such portfolio exists. You cannot squeeze a guaranteed profit out of prices that already behave like probabilities, no matter how clever the combination. Two worked books, because the theorem is only useful when you can build the trade. The over-round case, prices summing above one. YES at 60, NO at 50, sum $1.10\. Sell one YES for 60 cents, sell one NO for 50 cents, take in $1.10\. One of them resolves true and you pay the holder a dollar. Profit: $0.10, locked, regardless of outcome. $$ \\text{Revenue} = 0.60 + 0.50 = 1.10, \\qquad \\text{Payout} = 1.00, \\qquad \\text{Profit} = 0.10 $$ The under-round case, prices summing below one. YES at 30, NO at 33, sum $0.63\. Now you buy instead of sell. Buy one YES for 30, buy one NO for 33, pay out $0.63\. Exactly one pays you a dollar. Profit: $0.37. $$ \\text{Cost} = 0.30 + 0.33 = 0.63, \\qquad \\text{Payout} = 1.00, \\qquad \\text{Profit} = 0.37 $$ Notice the direction flips with the sign of the error. Prices too high, you sell the overpriced bundle. Prices too low, you buy the underpriced bundle. The theorem does not tell you which; the arithmetic does. And notice what is absent from both trades: any view on the event. This is the cleanest edge in the pillar precisely because it survives being completely wrong about the world. ## Why the mispricing does not get competed away The obvious objection: if this is free money, arbitrage should erase it in seconds and the opportunity should not persist. It persists. A 2024 to 2025 study of Polymarket estimated roughly $40 million of arbitrage profit available over a one-year sample. That is not a market that occasionally slips. That is a market that generates incoherence as a structural byproduct of how it is built. Three reasons, and each one is a design choice the platform cannot easily reverse. First, the speed-accuracy tradeoff. Enforcing coherence across many linked markets means solving integer programs that are NP-hard, the class of problems with no known fast algorithm. A platform that wants to confirm your order in milliseconds cannot run that solve on every incoming trade. Polymarket runs fast, largely independent books instead, and the $40 million is the price of that speed. The later articles in this pillar, the article ["Frank-Wolfe: Solving a Quintillion-Vertex Problem in 100 Steps"](https://aligrithm.com/frank-wolfe-solving-a-quintillion-vertex-problem-in-100-steps/) and its companions, exist because that same hard solve is what an arbitrageur has to run to find the trade the platform skipped. Second, independent listing. Each market keeps its own order book and its own liquidity. "A wins State X" and "A wins X by more than 5" sit in separate books with separate flow, and no real-time mechanism reconciles them. The books do not talk to each other, so nothing forces their prices to agree. Third, human local reasoning. A trader prices one market with the information in front of them. Nobody holds hundreds or thousands of linked markets in their head and checks every implication against every other. The same study identified 46,360 candidate market pairs in the 2024 US election alone. No participant, and no market maker chasing latency, evaluates that many relationships by hand. $$ \\text{Candidate market pairs, 2024 US election} = 46{,}360 $$ Put the three together and the conclusion is not that Polymarket is badly run. It is that any market prioritizing speed and accessibility over global consistency will leak arbitrage continuously, and the leak regenerates as fast as it is captured. Incoherence here is a permanent feature, not a bug someone will patch. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_349-visual-0.png) ## The gap between this and the real world Two cautions, stated flatly because the arithmetic above is seductive and the losses are real. The 10 cents in the over-round example is the profit if both legs fill at the quoted prices at the same instant. They will not. On a central limit order book you place the legs one at a time, your first fill moves the price, and the second leg can slip past the point where the trade still works. A textbook Dutch book can turn into a directional bet the moment execution is imperfect. The article "Execution Is Part of Expected Value" is where that leak gets measured, and the short version is that a large fraction of theoretical arbitrage never survives contact with the book. The second caution is scope. The Dutch book handles the simplest incoherence, prices on exclusive outcomes that fail to sum to one. Real markets are richer than that: one event implying another, mutual exclusions across markets, conditional relationships, whole webs of combinatorial dependency. The 60-and-50 trade is the toy case. The general test is not "do the prices sum to one" but "do the prices lie inside the set of all coherent price vectors," a geometric object called the marginal polytope. That object subsumes the Dutch book as one flat face and covers everything the simple sum-to-one rule misses. The article ["The Marginal Polytope: One Shape That Contains Every Fair Price"](https://aligrithm.com/the-marginal-polytope-one-shape-that-contains-every-fair-price/) builds it, and from there the pillar stops checking prices by hand and starts projecting onto the shape. ## KEY POINTS - Prices are coherent when you can read them as probabilities without contradiction: no negatives, exclusive outcomes sum to one, and an implication cannot cost more than the thing it implies. Fail any condition and the prices are incoherent. - The Dutch book theorem (de Finetti, 1937) says incoherent prices always admit a portfolio that profits under every outcome, and coherent prices never do. That is the mathematical definition of arbitrage, and it needs no forecast. - Over-round prices (sum above one): sell the bundle. YES 60 plus NO 50 gives $1.10 in, $1.00 out, 10 cents locked. Under-round (sum below one): buy the bundle. YES 30 plus NO 33 costs $0.63, pays $1.00, 37 cents locked. The sign of the error picks the direction. - The edge persists because of design, not oversight: enforcing coherence is NP-hard and too slow for a low-latency platform, each market is an independent book, and no human tracks the 46,360 linked pairs a single election generates. - A 2024 to 2025 Polymarket study estimated about $40 million of arbitrage available over a year. Incoherence regenerates as fast as it is captured, so it is a permanent structural feature of any speed-first market. - The quoted profit assumes both legs fill at once; sequential fills on an order book can erase it. And sum-to-one is only the toy case. The general coherence test is membership in the marginal polytope, which the next articles build. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [La prevision: ses lois logiques, ses sources subjectives - de Finetti (1937)](https://scholar.google.com/scholar?q=%22La+prevision%3A+ses+lois+logiques%2C+ses+sources+subjectives%22+de+Finetti+1937&ref=aligrithm.com) - [Foundations of the Theory of Probability - Kolmogorov (1933)](https://scholar.google.com/scholar?q=%22Foundations+of+the+Theory+of+Probability%22+Kolmogorov+1933&ref=aligrithm.com) - [Arbitrage-Free Combinatorial Market Making via Integer Programming - Kroer et al. (2016)](https://arxiv.org/abs/1606.02825?ref=aligrithm.com) - [Unravelling the Probabilistic Forest: Arbitrage in Prediction Markets - Saguillo et al. (2025)](https://arxiv.org/abs/2508.03474?ref=aligrithm.com) ### 9.6 Outcome Geometry: Why Two Fair Markets Can Be Jointly Unfair URL: https://aligrithm.com/outcome-geometry-why-two-fair-markets-can-be-jointly-unfair/ Last updated: 2026-07-27T13:26:08.000Z Take two prediction markets that each look priced correctly. "Candidate A wins State X" trades at 52 cents for YES, 48 for NO, and the two add to a dollar. "Party P wins State X by more than 5 points" trades at 32 and 68, also a clean dollar. Read either book on its own and you find nothing to trade. Read them together and there can be a guaranteed profit sitting in plain sight, because the two markets describe the same world and the platform priced them as if they didn't. That gap is the entire subject of this pillar, and it starts before any probability, any forecast, any opinion about who wins. It starts with the geometry of what these contracts pay. The first question a trader has to answer about any contract is mechanical: what does it pay in every possible state of the world? Not "what do I think will happen." What does the ticket pay if the world turns out one way, and what does it pay if it turns out another. Answer that for a set of related contracts and the shape of the opportunity is already fixed, whether or not anyone has looked. ## The outcome space is the list of ways the world can end Call the set of every possible joint resolution the outcome space. For one binary market, "Will A win X?", there are two ways the world lands: A wins, A loses. Two outcomes. For a single multi-candidate primary with four names on the ballot, four outcomes. Nothing subtle yet. The subtlety arrives the moment markets multiply. Put three binary markets side by side and the naive count of joint outcomes is 2 times 2 times 2, eight combinations. Ten binary markets give a thousand. This is the count that matters, and it grows the way compounding grows. $$ |\\Omega|\_{\\text{naive}} = 2^{n} $$ Here the vertical bars mean "the size of," the Greek omega is the outcome space, and n is the number of binary markets. Each market doubles the count. Ten markets, 1,024 joint outcomes; twenty markets, over a million; the 63 games of an NCAA bracket, about 9.2 quintillion. The number runs away from you fast, and most of this pillar is about taming it. Hold that thought. The count above is the naive count, and the word naive is doing real work. It assumes the markets are logically independent, that any combination of results can happen. Reality does not grant that. ## Payoff vectors turn a contract into a row of ones and zeros To reason about a bundle of contracts at once, stop thinking in prose and start thinking in a table. For each way the world can end, write down what each contract pays: a 1 if it pays out, a 0 if it does not. That row of ones and zeros is the payoff vector for that outcome. $$ \\varphi\_i(\\omega) = \\begin{cases} 1 & \\text{if security } i \\text{ pays out under outcome } \\omega,\\\\ 0 & \\text{otherwise.} \\end{cases} $$ Read it as a lookup, not a formula. The symbol phi with subscript i is the payoff of contract i; omega is one specific way the world ends. Feed in a world and a contract, get back a 1 or a 0\. Do this for every contract and you have one row. Do it for every possible world and you have the whole table. A programmer would call each row a boolean array indexed by contract, and that is exactly the right mental model. Now the example that carries the whole idea. Two markets on State X. The first has YES (A wins) and NO (A loses). The second has one security, YES on "Party P wins X by more than 5 points." Three tradeable securities. Write the table. | Outcome | A wins (YES) | A loses (NO) | Margin > 5 | | --------------------- | ------------ | ------------ | ---------- | | A wins by more than 5 | 1 | 0 | 1 | | A wins by 5 or less | 1 | 0 | 0 | | A loses | 0 | 1 | 0 | Count the rows. Three, not four. The missing row is the one where A loses and the margin is greater than 5, and it is missing because it cannot happen. You cannot win by more than 5 points and also lose. The logic of the contracts deleted a row that a naive count would have kept. ## The valid payoff set, and where the money hides Collect the rows that actually survive and you get the valid payoff set. Every row is a corner of a cube whose dimension equals the number of securities, but most corners of that cube never appear, because logic forbids them. $$ Z = \\{\\varphi(\\omega) : \\omega \\in \\Omega\\} \\subset \\{0,1\\}^{|\\mathcal{I}|}, \\qquad |Z| \\ll 2^{|\\mathcal{I}|} $$ Z is the set of valid rows. The curly-brace expression on the right is the full cube of all possible ones-and-zeros rows across the securities; the double-less-than sign says Z is far smaller than that cube. Three securities give a cube with eight corners. Our table used three of them. Five corners of the cube are logically impossible and simply do not exist as tradeable outcomes. That collapse is the point. The number of valid outcomes sits far below the number of ways you could arrange the securities on paper, and the size of the collapse measures how tightly the contracts are wired together. When two contracts share no logic, nothing collapses and there is no structural edge to find. When they are wired together, the valid set shrinks, and the shrinkage is precisely the room in which a platform can misprice one contract relative to another without any single book looking wrong. Here is the mechanism stated plainly. Polymarket lists and prices each contract in its own book, with its own order flow, as if it were alone in the universe. "Party P wins by more than 5" implies "A wins," but the market maker running the first book does not consult the second. So the joint set of prices can drift into a combination that no probability distribution over the three real outcomes could ever produce. The individual books stay coherent. The pair does not. Two fair markets, jointly unfair. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_348-visual-0.png) ## The structural pre-check Before you form any view about who wins anything, run four questions on any set of related markets. This is the whole chapter reduced to a checklist you can execute in a minute. First, list the contracts. What can you actually buy and sell. Second, describe the possible worlds, the outcome space, either by listing them or by writing down the constraints that link them. Third, compute the valid payoff set, the surviving rows. Fourth, compare the size of that set to the naive cube. If the valid set is smaller than the cube, dependencies exist, and structural edge may be present. If it equals the cube, the markets are genuinely independent and structural arbitrage across them is impossible, no matter how wrong the prices feel. That last clause matters more than it looks. A market can be badly mispriced in your opinion and still offer no structural trade, because your opinion is a forecast, and forecasting is the dangerous, low-ranked source of edge this pillar treats last. Structural edge does not care what you believe. It only asks whether the prices are internally consistent with the geometry of what the contracts pay. If you cannot answer the four questions, you cannot know whether an edge exists, and trading on the feeling that something is off is how accounts donate money to the people who ran the check. The empirical scale is worth stating so the geometry does not stay abstract. A 2024 to 2025 study of the US election markets on Polymarket found 1,576 logically dependent market pairs in the presidential race alone, and that was one race in one cycle. No human prices thousands of pairs against each other in their head. The collapse of the valid set is happening constantly, across far more contracts than anyone can track by hand, which is exactly why the edge is mechanical rather than a matter of insight. ## What this buys you, and what it does not Outcome geometry hands you a verdict on whether a trade can exist, not on whether you can capture it. The three-row table tells you the shape of the opportunity. It says nothing about whether the second leg will still be there at the price you need when your first leg fills, nothing about the fee to move on-chain, nothing about the two other bots watching the same pair. Those live in later articles, and they are where most of the theoretical profit leaks away. What you have after this article is the right first question. Not "what will happen," but "what does each contract pay in every state, and is the joint set of prices consistent with the small set of states that can actually occur." The next article, the article "The Dutch Book Theorem, Plainly," turns the smallest version of that inconsistency, prices that fail to sum to one, into a guaranteed-profit portfolio you can write down. The one after that, the article "The Marginal Polytope," generalizes the whole thing into a single shape that contains every fair price and a single test for whether you are outside it. ## KEY POINTS - The first question about any contract is mechanical, not predictive: what does it pay in every possible state of the world. Answer that for a bundle of related contracts and the shape of the opportunity is already fixed. - The naive count of joint outcomes for n binary markets is 2 to the n, which explodes fast (ten markets, 1,024; an NCAA bracket, about 9.2 quintillion), but that count assumes the markets are logically independent. - Write each outcome as a payoff vector, a row of ones and zeros over the securities. Logical links between contracts delete impossible rows, so the valid payoff set is far smaller than the full cube of arrangements. - The size of that collapse is where structural edge lives. Platforms price each contract in its own book; when contracts share logic that no single book enforces, the joint prices can land outside the small set of states that can actually happen. - Run the structural pre-check: list contracts, describe the worlds, compute the valid set, compare it to the naive cube. Smaller than the cube means dependencies and possible edge; equal to the cube means no structural trade exists regardless of how mispriced it feels. - A 2024 to 2025 Polymarket study found 1,576 logically dependent market pairs in the presidential race alone. No human tracks that many pairs, which is why this edge is mechanical, not a matter of having a better opinion. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Arbitrage-Free Combinatorial Market Making via Integer Programming - Kroer et al. (2016)](https://arxiv.org/abs/1606.02825?ref=aligrithm.com) - [Unravelling the Probabilistic Forest: Arbitrage in Prediction Markets - Saguillo et al. (2025)](https://arxiv.org/abs/2508.03474?ref=aligrithm.com) ### 9.5 Match Your Edge to Your Resources URL: https://aligrithm.com/match-your-edge-to-your-resources/ Last updated: 2026-07-27T13:26:07.000Z The article on the nine sources of edge ranks structural arbitrage first and forecasting fifth. That ranking is not a personal instruction. It describes which edge is cleanest in the abstract, not which edge you should chase with the capital and infrastructure you actually have. A trader with $5,000 and a laptop who reads "structural is best" and tries to run an integer-programming arbitrage engine against bots colocated near the Polygon sequencer is about to donate money. The right edge is a function of your resources, and choosing wrong is its own way to lose. Structural edge is the cleanest and also the most capital- and infrastructure-hungry. The two facts do not conflict. Cleanest means the profit is guaranteed once you capture it. Capturing it means winning a latency race that costs real money to enter. Match the edge to what you can afford to build. ## The selection table Read your row by the resources you have, then read what to pursue and what to avoid. $$ \\begin{array}{lll} \\textbf{Resources} & \\textbf{Pursue} & \\textbf{Avoid}\\\\ <\\$10\\text{K, no infra} & \\text{Informational + Regime} & \\text{Structural (infra cost} > \\text{capital)}\\\\ \\$10\\text{K--}\\$100\\text{K, basic infra} & \\text{Microstructure + Execution} & \\text{Adversarial (too little data)}\\\\ >\\$100\\text{K, production infra} & \\text{Structural + Execution} & \\text{Informational alone (not scalable)}\\\\ \\text{Strong ML / stats} & \\text{Informational + Temporal} & \\text{Structural (wrong skill set)}\\\\ \\text{Strong systems eng.} & \\text{Structural + Execution} & \\text{Informational (play to strengths)} \\end{array} $$ Read that as: a small account with no infrastructure should trade informational and regime edges and stay away from structural arbitrage, because the infrastructure to compete costs more than the account holds; a mid-size account should chase microstructure and execution; a large account with production infrastructure should run structural and execution and avoid informational trading alone because it does not scale; and if your strength is modeling, lean into informational and temporal, while if your strength is systems engineering, lean into structural and execution. The counterintuitive rows are the small account and the strong-modeler. The cleanest edge, structural, is the wrong target for the trader with the least money, because the guarantee is worthless if the two or three fastest bots capture it before your order confirms. The small account's realistic edge is informational, exactly the source the ranking called dangerous, because that trader has no faster option and must lean on judgment plus the discipline to abstain during bad regimes. A strong quant is told to avoid structural not because it is bad but because it rewards a different skill: winning structural arbitrage is a systems-engineering problem, low-latency I/O and solver speed, not a statistics problem. ## Why the small account cannot buy the clean edge Structural arbitrage on Polymarket closes fast. The opportunity can vanish inside one Polygon block, and the decision-to-mempool window that decides who wins is on the order of tens of milliseconds. Winning that race requires monitoring infrastructure, a fast integer-programming solver, and order routing tuned to submit all legs inside one block. That is a fixed cost measured in tens of thousands of dollars and ongoing engineering, and profits are concentrated: the fastest two or three systems on any given opportunity take the large majority, and the slower entrants lose on adverse fills. The article on regimes and crowding quantifies the winner-take-most math. A $5,000 account entering that race is exit liquidity. The same account can trade informational edge because that edge decays over hours or days, not milliseconds, so latency stops being the binding constraint. The binding constraint becomes calibration and discipline, which cost time rather than capital. The old article ["Why Trading Is a Probability Business, Not a Certainty Business"](https://aligrithm.com/why-trading-is-a-probability-business-not-a-certainty-business/) and the article on the six ways to lose spell out the calibration bar that edge demands. ## The six-stage life cycle applies to every row Whatever edge your row points to, the trade still passes through the same six stages, and skipping any stage is the same failure regardless of resources. $$ \\text{Detection} \\to \\text{Quantification} \\to \\text{Sizing} \\to \\text{Execution} \\to \\text{Attribution} \\to \\text{Adaptation} $$ Read that as: name and quantify the edge, size it for error, execute all legs, measure realized against guaranteed profit, and adapt as the regime shifts, in that order, every time. A small informational trader skips execution modeling at the same peril as a large structural desk. The stages do not scale with account size. Only the edge you feed into them does. ## Attribution keeps you honest about your row Pick a row, trade it, then check whether the edge you claimed is the edge you got. Tag every trade by source and track execution efficiency, realized profit over guaranteed profit, per source. $$ \\eta = \\frac{\\Pi\_{\\text{realized}}}{\\Pi\_{\\text{guaranteed}}}, \\qquad \\eta > 0.85 \\text{ excellent}, \\quad 0.60\\text{--}0.85 \\text{ fix infra first}, \\quad <0.60 \\text{ stop} $$ Read that as: efficiency eta is what you kept divided by what the setup promised, above 0.85 is excellent, between 0.60 and 0.85 means improve your infrastructure before adding capital, and below 0.60 means execution is destroying edge faster than the math creates it, so stop and fix it before deploying more. A large account whose structural eta keeps printing below 0.60 is not out-resourced on paper, it is losing the latency race in practice and should reconsider its row. The article on execution efficiency and regret builds the full attribution dashboard. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_347-visual-0.png) ## KEY POINTS - The nine-source ranking says which edge is cleanest, not which edge you should trade. The right edge depends on your capital, infrastructure, and skill set. - Structural arbitrage is the cleanest and the most infrastructure-hungry at once. The guarantee is worthless if faster systems capture it first. - Small accounts with no infrastructure should trade informational and regime edges, because those decay over hours not milliseconds, so discipline and calibration matter more than latency. - Mid-size accounts should pursue microstructure and execution; large accounts with production infrastructure should run structural and execution and avoid informational-alone, which does not scale. - Match the edge to your strength: modelers to informational and temporal, systems engineers to structural and execution. Winning structural arbitrage is an engineering problem, not a statistics one. - Every row runs the same six-stage life cycle. The stages do not scale with account size; only the edge you feed them does. - Track execution efficiency per source. Above 0.85 is excellent, 0.60 to 0.85 means fix infrastructure before scaling, below 0.60 means stop, and a persistently low structural efficiency is a signal you are in the wrong row. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Unravelling the Probabilistic Forest: Arbitrage in Prediction Markets - Saguillo et al. (2025)](https://arxiv.org/abs/2508.03474?ref=aligrithm.com) ### 9.4 The Nine Sources of Edge, Ranked Cleanest to Dirtiest URL: https://aligrithm.com/the-nine-sources-of-edge-ranked-cleanest-to-dirtiest/ Last updated: 2026-07-27T13:26:07.000Z A source of edge is a structural reason your expected value is positive before you size the trade and before you execute it. If you cannot name the source, you do not have an edge, you have a position. Prediction markets support nine sources, and they are not equal. Some hand you a mathematical guarantee. Some hand you a fleeting statistical tilt that is lethal to oversize. Ranking them, and knowing which one you are actually exploiting, is the map this whole pillar builds toward. The ordering below runs cleanest to dirtiest, where clean means the profit is close to guaranteed and the edge is hard to erase, and dirty means the profit is uncertain and the edge decays or blows up if you misjudge it. Most participants start at number five and wonder why they struggle. The correct developmental order runs the other way. ## The ranking Each source answers one question, offers one grade of certainty, persists for one kind of horizon, and gets formalized in specific later articles of this pillar. $$ \\begin{array}{clll} \\# & \\textbf{Source} & \\textbf{Core question} & \\textbf{Certainty / persistence}\\\\ 1 & \\text{Structural} & \\text{Is the market internally inconsistent?} & \\text{Guaranteed / permanent}\\\\ 2 & \\text{Execution} & \\text{Do I fill when others do not?} & \\text{Measurable / permanent}\\\\ 3 & \\text{Microstructure} & \\text{Is the order book mispriced?} & \\text{Small, repeatable / medium}\\\\ 4 & \\text{Temporal} & \\text{Are prices lagging reality?} & \\text{Time-decaying / short}\\\\ 5 & \\text{Informational} & \\text{Is my belief better than the market's?} & \\text{Uncertain / short--medium}\\\\ 6 & \\text{Regime} & \\text{Am I trading only when conditions favor me?} & \\text{Meta-edge / long}\\\\ 7 & \\text{Portfolio} & \\text{Am I combining trades better than others?} & \\text{Meta-edge / long}\\\\ 8 & \\text{Adversarial} & \\text{Do I understand how others behave?} & \\text{Rare / medium}\\\\ 9 & \\text{Operational} & \\text{Do I avoid catastrophic mistakes?} & \\text{Prevents loss / permanent} \\end{array} $$ Read that as: a nine-row table where each edge names the question it answers, how certain the profit is, and how long it survives, running from structural inconsistency at the top, where the math guarantees a permanent edge, down through information and portfolio construction to operational discipline, whose payoff is avoided disaster rather than gained profit. ## Why structural sits at the top Structural edge exists when prices contradict each other, when the price point sits outside the marginal polytope from the article on convex geometry. The math guarantees profit and no forecast is required. The contradiction is not a transient inefficiency competition will erase, it is a permanent feature of any venue that prices markets fast and independently, because enforcing full cross-market consistency in real time is an NP-hard problem no low-latency exchange will solve per order. The edge regenerates continuously. That is why it ranks first: guaranteed and permanent. Execution ranks second because infrastructure, not prediction, guarantees you realize the theoretical edge. If your system fills both legs of an arbitrage inside one block while slower systems drift, the profit is measurable and the advantage does not decay as long as you stay fast. Microstructure, reading a mispriced order book, ranks third: small per trade but reliably repeatable. Temporal edge, trading a price that lags reality, ranks fourth: real but fleeting, and it demands speed. ## Why informational sits in the middle, not the top Forecasting is number five, below four sources that require no view on the outcome at all. This ranking offends most traders, who arrive convinced their opinion is the product. The demotion is deliberate. Informational edge is real but dangerous to size, because the edge estimate carries large uncertainty and the penalty for overestimating is asymmetric and severe. The article on the six ways to lose shows how a felt 85% against a market's 52% turns into a 6.6-times overbet and negative growth. A structural arbitrage cannot do that to you. A forecast can. Uncertain profit, short-to-medium persistence, high blow-up risk: middle of the pack, and the most seductive trap in the market. ## The meta-edges and the floor Regime, number six, is a meta-edge: it improves every source above it by telling you when to trade and when to abstain. Not trading during a crowded or thin-liquidity regime beats detecting one more marginal opportunity. Portfolio, number seven, is the other meta-edge: combining trades so a hidden correlation does not blow up the book, the failure the article on tail dependence dissects. Adversarial, number eight, understanding how other participants behave, is rare and powerful but hard to formalize. Operational, number nine, is the floor: avoiding catastrophic mistakes, the kill switches and position caps whose payoff is a loss you never took. Survivorship is the ultimate edge, and the old article ["Why Loss Control Is the Only Thing You Fully Control"](https://aligrithm.com/why-loss-control-is-the-only-thing-you-fully-control/) is the same claim in a different market. ## The correct developmental order The ranking is not the order you learn them. Most participants begin at informational, hold beliefs about outcomes, and lack the detection, infrastructure, and regime awareness to survive. Build the other way. $$ \\text{structural} \\;\\to\\; \\text{execution} \\;\\to\\; \\text{sizing} \\;\\to\\; \\text{regime} \\;\\to\\; \\text{informational} $$ Read that as: master detecting guaranteed structural edge first, then the execution infrastructure to capture it, then the sizing math to survive error, then the regime awareness to know when to act, and only last take on the dangerous informational edge, once you have the machinery to size and evaluate it safely. ## The life cycle every good trade follows Whatever the source, a profitable trade passes through six stages, and skipping any one leaves a specific failure mode unguarded, the exact failures catalogued in the article on the six ways to lose. $$ \\begin{aligned} &\\text{1\. Detection: name the source, quantify the edge}\\\\ &\\text{2\. Quantification: } \\Pi = D(\\hat{p}\\,\\|\\,\\text{state}) - g - \\mathbb{E}\[\\text{slippage}\] - \\text{fees}\\\\ &\\text{3\. Sizing: } f = f^\* \\times P(\\text{fill}) \\times (1 - s) \\times \\rho\_{\\text{regime}}\\\\ &\\text{4\. Execution: parallel submission, fill monitoring, partial handling}\\\\ &\\text{5\. Attribution: } \\eta = \\Pi\_{\\text{realized}} / \\Pi\_{\\text{guaranteed}}, \\text{ tag by source}\\\\ &\\text{6\. Adaptation: update regime, recalibrate, adjust} \\end{aligned} $$ Read that as: detect and name the edge, quantify the profit net of the approximation gap, slippage and fees, size it by scaling the growth-optimal fraction down for fill probability, slippage, and regime, execute all legs in parallel while watching fills, attribute the realized-to-guaranteed ratio and tag it by source, then adapt as the regime shifts. Omit detection and you are gambling. Omit quantification and you cannot size. Omit sizing and you are guessing amounts. Omit execution modeling and you own theoretical profits only. Omit attribution and you cannot improve. Omit adaptation and you are static in a game that is not. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_346-visual-0.png) ## KEY POINTS - A source of edge is a named structural reason expected value is positive before sizing and execution. No name, no edge. - Nine sources exist, ranked cleanest to dirtiest by how guaranteed the profit is and how hard the edge is to erase. - Structural edge ranks first: a mathematical guarantee from price contradictions, permanent because fast, independent pricing cannot enforce consistency in real time. - Execution, microstructure, and temporal edges follow, none requiring a forecast. - Informational edge ranks fifth, not first. It is real but dangerous to size, and overconfidence turns it into ruin, which is why most participants who start there struggle. - Regime and portfolio are meta-edges that improve or protect everything above them; operational discipline is the floor, where the payoff is disaster avoided. - Learn them in reverse of most people's instinct: structural, then execution, then sizing, then regime, then informational. - Every good trade passes six stages: detection, quantification, sizing, execution, attribution, adaptation. Each skipped stage is a named way to lose. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Unravelling the Probabilistic Forest: Arbitrage in Prediction Markets - Saguillo et al. (2025)](https://arxiv.org/abs/2508.03474?ref=aligrithm.com) ### 9.3 The Six Ways to Lose Money on Polymarket URL: https://aligrithm.com/the-six-ways-to-lose-money-on-polymarket/ Last updated: 2026-07-27T13:26:06.000Z Most participants in prediction markets lose, and they lose in a small number of recognizable ways. Each way is a skipped step in the life of a trade, and each maps to one of the five invariants from the article on what a good trade is. Learn the six failure modes as a diagnostic. Before you enter, ask which one you might be committing. After you lose, the autopsy is faster because the suspects are named. Every anti-pattern below follows the same shape: how it looks, why it kills, which invariant it breaks, and the fix. The numbers are worked, because a failure mode you cannot quantify is a failure mode you will repeat. ## 1\. Belief without calibration You assign 85% to an event the market prices at 52%, and you bet big. The 85% is a feeling. You never computed a likelihood ratio, never tracked whether your past "85%" calls resolved 85% of the time, never shrank the estimate toward the market. Kelly sizes the bet for 33 points of edge. If your true edge is 5 points, you are roughly 6.6 times too large, and the growth penalty is quadratic in that error. $$ G(f^\* + \\Delta) \\approx G(f^\*) - \\frac{\\Delta^2}{2\\sigma^2}, \\qquad 6.6\\times \\text{ overbet} \\;\\Rightarrow\\; G < 0 $$ Read that as: deviating from the optimal fraction by delta drops your long-run growth by about delta-squared over twice the variance, and at 6.6 times the correct size the growth rate crosses below zero, which is a path to certain ruin even when your direction was right. Breaks invariant 1 (name and quantify the edge) and invariant 4 (size for error). Fix: build a reliability diagram from your last 50-plus predictions. If your 80% bin resolves at 60%, you are overconfident. Shrink, or stop trading on beliefs. The old article ["Why Trading Is a Probability Business, Not a Certainty Business"](https://aligrithm.com/why-trading-is-a-probability-business-not-a-certainty-business/) is the general form of this discipline. ## 2\. Arbitrage without execution modeling You see YES plus NO priced at $0.92 and buy both, expecting $0.08 risk-free. Leg one, YES at $0.30, fills. That fill moves the market, others update, and when you submit leg two the best ask on NO has climbed from $0.62 to $0.71\. Total cost $1.01 against a $1.00 payout. You detected real arbitrage and still lost a cent. $$ \\text{leg 1: } \\$0.30 \\text{ fills} \\;\\to\\; \\text{leg 2 ask } \\$0.62 \\to \\$0.71 \\;\\Rightarrow\\; \\text{cost } \\$1.01 > \\$1.00 \\text{ payout} $$ Read that as: the first fill is not free information; it tips the market, the second leg reprices against you, and the arithmetic that looked like an $0.08 lock becomes an $0.01 loss once both legs are priced at what you actually pay. Breaks invariant 3 (executable under real liquidity). Fix: never compute profit without VWAP at target size across all legs, and require the residual edge after slippage, gas, and partial-fill cost to clear $0.05\. Below that, skip. The article on execution as part of expected value formalizes the threshold. ## 3\. Kelly without error bounds Kelly returns a fraction of 0.20 and you deploy the full 20% of capital. Kelly assumes your edge estimate is exactly right. It never is. If the truth supports 0.04, you are five times too large. $$ \\text{true } f^\* = 0.04, \\quad \\text{deployed } 0.20 \\;\\Rightarrow\\; 5\\times \\text{ overbet} \\;\\Rightarrow\\; P(\\text{ruin over 100 trades}) > 30\\% $$ Read that as: betting five times the fraction the real edge justifies pushes the probability of ruin over a hundred trades above 30%, because the error compounds multiplicatively in a way underbetting never does. The asymmetry is the whole lesson: underbetting slows growth but preserves capital, overbetting past a threshold destroys it. Breaks invariant 4\. Fix: fractional Kelly. Half-Kelly keeps about 75% of the growth at roughly half the variance; quarter-Kelly is robust to large estimation error. The article on Kelly under model error works this in detail, and the old article ["Why Volatility-Adjusted Position Sizing Matters"](https://aligrithm.com/why-volatility-adjusted-position-sizing-matters/) is the continuous-market cousin. ## 4\. Correlation blindness You hold arbitrage positions across five US state-election markets and call it diversified. It is not. A national shock, a scandal or a health event, moves all five at once. Linear correlation of about 0.75 hides a tail dependence near 0.85 to 0.95, and that gap is where portfolios die. $$ \\text{normal P\\&L} \\approx \\pm\\$0.10, \\qquad \\text{joint-tail loss} \\approx \\$2.50 \\;\\Rightarrow\\; \\text{ratio } 25{:}1, \\text{ not the } 5{:}1 \\text{ independence predicts} $$ Read that as: under normal conditions the five positions bob around plus or minus ten cents as if roughly independent, but in a joint tail event they all lose together for a combined $2.50, a 25-to-1 ratio versus the 5-to-1 that assuming independence would give. Breaks invariant 2, because the joint downside was never bounded. Fix: model dependence with a copula, use a Student-t with four to six degrees of freedom to capture the fat joint tails, and cap correlated exposure. The article on tail dependence and Student-t copulas builds this; the old article ["Fat Tails: Why Gaussian Thinking Breaks Trading Systems"](https://aligrithm.com/fat-tails-why-gaussian-thinking-breaks-trading-systems/) is the foundation. ## 5\. Equating backtest and live performance Your backtest shows $500 a day at a 94% win rate, so you deploy immediately. Backtests assume perfect fills at quoted prices. Live trading pays VWAP slippage, suffers partial fills, and eats adverse selection. Execution efficiency, the ratio of realized profit to the profit the math guaranteed, drops from 1.00 to somewhere around 0.65 to 0.80. $$ \\eta = \\frac{\\Pi\_{\\text{realized}}}{\\Pi\_{\\text{guaranteed}}}, \\qquad \\text{backtest } \\eta = 1.00 \\;\\to\\; \\text{live } \\eta \\approx 0.65\\text{--}0.80 $$ Read that as: efficiency eta is what you actually kept divided by what the theory promised, it sits at one in a backtest that assumes flawless fills, and it falls to roughly two-thirds or three-quarters in live trading once slippage and adverse selection bite. The backtest also omits competition, since the other bots running your strategy were not in the historical tape, and it omits regime shift, since a 2024 edge can be gone by 2025\. Breaks invariant 5\. Fix: discount backtest profit by at least 30%, paper-trade two weeks, track eta from day one, and halt if it falls below 0.60\. The old article ["When to Switch Off a Trading System"](https://aligrithm.com/when-to-switch-off-a-trading-system/) covers the shutdown discipline. ## 6\. Trading without a named edge "The market feels wrong at $0.52\. I am buying." "Feels wrong" is not a source. Without a named mechanism you cannot quantify the edge, cannot size the trade, and cannot tell skill from luck afterward. You are handing capital to the participants who can name their edge. Breaks invariant 1\. Fix: before every trade, finish the sentence "my edge is \_\_\_ because \_\_\_." If you cannot, do not trade. This one discipline removes a large fraction of losing activity. ## The common structure Every anti-pattern is a skipped step in the trade life cycle. The map is exact. $$ \\begin{aligned} &\\text{1\. Belief w/o calibration} \\to \\text{skipped: Quantification}\\\\ &\\text{2\. No execution modeling} \\to \\text{skipped: Execution feasibility}\\\\ &\\text{3\. Kelly w/o error bounds} \\to \\text{skipped: Sizing-error analysis}\\\\ &\\text{4\. Correlation blindness} \\to \\text{skipped: Portfolio dependency check}\\\\ &\\text{5\. Backtest = live} \\to \\text{skipped: Execution reality check}\\\\ &\\text{6\. No named edge} \\to \\text{skipped: Detection} \\end{aligned} $$ Read that as: each of the six failures corresponds to exactly one omitted stage in the life of a trade, from detection through quantification, sizing, execution, and the reality check against live conditions. The life cycle exists because each step guards one failure mode. Skip the step and that mode goes unguarded. The article on the nine sources of edge lays out the full life cycle these steps belong to. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_345-visual-0.png) ## KEY POINTS - Losses on prediction markets come from six recognizable anti-patterns, each a skipped step in the trade life cycle and each breaking a specific invariant. - Belief without calibration: a felt probability, not a computed one, oversizes the bet; at 6.6 times too large the growth rate goes negative. Build a reliability diagram and shrink. - Arbitrage without execution modeling: the first fill moves the price, the second leg reprices, and a detected lock becomes a loss. Require residual edge over $0.05 after VWAP, gas, and partial fills. - Kelly without error bounds: deploying full Kelly on a wrong estimate risks ruin above 30% over 100 trades. Use fractional Kelly. - Correlation blindness: five "diversified" election bets share a joint tail near 0.85 to 0.95, turning a ten-cent wobble into a $2.50 loss. Model dependence with a Student-t copula and cap correlated exposure. - Backtest equals live: perfect-fill backtests ignore slippage, competition, and regime shift; efficiency drops to two-thirds. Discount 30%, paper-trade, and halt below 0.60. - Trading without a named edge: "feels wrong" cannot be sized or evaluated. Finish "my edge is \_\_\_ because \_\_\_" or do not trade. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Convex Optimization - Stephen Boyd Lieven Vandenberghe (Amazon)](https://www.amazon.com/s?k=Convex+Optimization+Stephen+Boyd+Lieven+Vandenberghe&ref=aligrithm.com) - [An Introduction to Copulas - Roger Nelsen (Amazon)](https://www.amazon.com/s?k=An+Introduction+to+Copulas+Roger+Nelsen&ref=aligrithm.com) ### 9.2 A Good Trade Is Not a Correct Prediction — The Five Invariants URL: https://aligrithm.com/a-good-trade-is-not-a-correct-prediction-the-five-invariants-2/ Last updated: 2026-07-27T13:26:06.000Z A trader buys "Candidate A wins" at $0.62, the candidate wins, and the trader collects. Good trade? You cannot tell. The outcome says nothing about whether the decision had an edge, because a single resolution is one draw from a distribution, and one draw carries almost no information. The old article ["Why Trading Is a Probability Business, Not a Certainty Business"](https://aligrithm.com/why-trading-is-a-probability-business-not-a-certainty-business/) made this point for price series. Prediction markets sharpen it: the contract either pays $1 or $0, so the temptation to grade yourself by the result is stronger, and the error is more expensive. Define a good trade by its structure before resolution, not by its payout after. A good trade satisfies five invariants at the same time. Miss one and the trade is not good, regardless of how it resolves. The five double as a pre-trade checklist and as an index to the rest of this pillar, because each later article builds the tool that answers one of them. ## Invariant 1: a named source of edge State why expected value is positive before you size or execute, in one sentence, naming a mechanism. "I believe A will win" is not a mechanism. "The YES and NO prices sum to $0.92, so buying both locks $0.08 against a $1.00 payout" is a mechanism. "The margin market and the winner market price a combination that lies outside the marginal polytope" is a mechanism. The mechanisms in this pillar are structural inconsistency, execution advantage, microstructure mispricing, temporal lag, and informational edge, ranked from cleanest to most dangerous in the article on the nine sources of edge. A conviction is not on the list. If you cannot name the source, you are not trading an edge, you are donating to someone who can name theirs. ## Invariant 2: bounded downside Know your worst case in dollars before entry. For a structural arbitrage the worst case is not the market moving against you, because a completed arbitrage cannot. The worst case is execution failure on one leg: you buy YES, the market moves, and you never fill NO at the price you needed, so a locked profit becomes a naked directional bet. For a directional position the worst case is the full loss of what you deployed. If you cannot write the maximum loss as a number, the trade is not defined well enough to take. This is the same discipline the old article ["The Trader's Real Job: Control Losses, Not Predict Everything"](https://aligrithm.com/the-traders-real-job-control-losses-not-predict-everything/) argues for in ordinary markets, ported to a venue where the payoff is binary and the leg risk is specific. ## Invariant 3: executable under real liquidity Theoretical profit is computed at the quoted price. You do not trade at the quoted price. You trade against the depth of the book, and sweeping the book moves the average price you pay. Check three things before believing any edge number. $$ \\text{VWAP}(S) = \\frac{\\sum\_{k} q\_k \\, p\_k}{\\sum\_k q\_k}, \\qquad \\text{filling size } S \\text{ up the ask ladder } (q\_k \\text{ at price } p\_k) $$ Read that as: the volume-weighted average price for an order of size S is the total dollars paid divided by total shares bought, summed as you eat successive levels of the book at rising prices. If the book is 500 shares at $0.32, 300 at $0.34, and 200 at $0.38, then buying 900 shares costs 500 times 0.32 plus 300 times 0.34 plus 100 times 0.38, which is $300, a VWAP of $0.333, not the $0.32 top-of-book quote. The 1.3-cent slippage is real money that never appeared in the theoretical edge. Model VWAP at your target size on every leg, confirm the depth exists, and confirm partial-fill risk is acceptable. The article on execution as part of expected value builds this out, and it connects to the old article ["TWAP and VWAP Are Execution Models, Not Just Indicators."](https://aligrithm.com/twap-and-vwap-are-execution-models-not-just-indicators/) ## Invariant 4: sized to survive error Your edge estimate is wrong by some amount, always. Size for that. The position must account for estimation error in the edge, the probability that execution fails, correlation with what you already hold, and a drawdown limit. A workable ceiling: no single trade should be able to cause more than a 5% portfolio drawdown. The reason sizing dominates is the asymmetry of the growth rate around the optimal fraction. $$ G(f^\* + \\Delta) \\approx G(f^\*) - \\frac{\\Delta^2}{2\\sigma^2} $$ Read that as: if you deviate from the growth-optimal bet fraction f-star by an amount delta, your long-run growth rate drops by roughly delta-squared over twice the variance, so the penalty grows with the square of the sizing error. Underbetting slows you down but keeps you alive. Overbetting past a threshold turns the growth rate negative, which is certain ruin no matter how good the edge was. The article on Kelly under model error works the numbers; the old article ["Why Volatility-Adjusted Position Sizing Matters"](https://aligrithm.com/why-volatility-adjusted-position-sizing-matters/) makes the parallel case for continuous markets. ## Invariant 5: robust to competition An edge you cannot capture before it disappears is not your edge. Structural arbitrage on Polymarket can close inside one Polygon block, roughly two seconds, and the decision-to-mempool window that matters is closer to 30 milliseconds. A system with 500-millisecond latency chasing a 30-millisecond window is not trading, it is providing exit liquidity to the two or three bots that got there first. Assess your latency against the opportunity's lifetime, and assess crowding: the more systems on the same trade, the faster the edge decays and the worse your fills. The article on regimes and crowding quantifies the winner-take-most concentration. ## The pre-trade checklist The five invariants collapse into five questions. Answer all five before execution. Any unknown is a fail. $$ \\begin{aligned} &\\text{1\. Source of edge: what is it, how is it quantified?}\\\\ &\\text{2\. Worst case: maximum loss in dollars?}\\\\ &\\text{3\. Fill probability: depth checked, VWAP modeled?}\\\\ &\\text{4\. Size: from Kelly or a constrained variant?}\\\\ &\\text{5\. Competition: latency assessed, crowding manageable?} \\end{aligned} $$ Read that as: five plain questions, one per invariant, each demanding a concrete answer rather than a feeling. If any answer is "I do not know," the trade fails the checklist. This is not bureaucracy. Each question guards a specific way traders lose money, catalogued in the article on the six ways to lose on Polymarket. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_344-visual-0.png) ## What a good trade is not Four things masquerade as good trades and are not. A prediction that resolved correct, which may be luck. A position that made money, which may be survivorship bias over a short horizon. A bet sized by conviction rather than a framework. A trade whose edge you cannot name. The old article ["Trading Systems Are Recipes, Not Predictions"](https://aligrithm.com/trading-systems-are-recipes-not-predictions/) makes the general version of this case; the binary payoff of prediction markets makes each failure cleaner to see and harder to excuse. Grade the decision, not the draw. Over a hundred trades the draws average out and the structure shows through. The trader who logs whether each entry passed all five invariants, and reviews that log instead of the equity curve, learns something. The trader who counts wins learns noise. ## KEY POINTS - A good trade is defined before resolution by its structure, not after by its payout. One binary resolution is a single draw and grades nothing. - Invariant 1, named edge: state the mechanism in one sentence. A conviction is not a mechanism. No name means no edge. - Invariant 2, bounded downside: know the worst case in dollars. For arbitrage the worst case is a failed leg turning a lock into a directional bet. - Invariant 3, executable: compute VWAP at your real size on every leg. Top-of-book quotes lie once you sweep depth; the slippage is money missing from your theoretical edge. - Invariant 4, sized for error: the growth penalty is quadratic in sizing error, and overbetting past a threshold guarantees ruin while underbetting only slows you. Cap any trade at a 5% portfolio drawdown. - Invariant 5, robust to competition: an edge that closes in one Polygon block is not yours if your latency is half a second. Assess speed and crowding. - The five invariants are a pre-trade checklist and an index to the pillar. Any unknown answer is a rejection. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Convex Optimization - Stephen Boyd Lieven Vandenberghe (Amazon)](https://www.amazon.com/s?k=Convex+Optimization+Stephen+Boyd+Lieven+Vandenberghe&ref=aligrithm.com) - [An Introduction to Copulas - Roger Nelsen (Amazon)](https://www.amazon.com/s?k=An+Introduction+to+Copulas+Roger+Nelsen&ref=aligrithm.com) ### 9.18 Fast Fills Are Bad Fills: Make vs Take, Adverse Selection, and Why the Spread Is Posterior Variance URL: https://aligrithm.com/fast-fills-are-bad-fills-make-vs-take-adverse-selection-and-why-the-spread-is-posterior-variance/ Last updated: 2026-07-27T13:26:13.000Z You post a limit order to buy NO at 41 cents. Two seconds later it fills. Feels great. It is a warning. A limit order that fills in seconds almost always filled because the market moved through your price on its way somewhere, and the somewhere is against you. The counterparty who hit your order knew something you did not, or was faster to the same news, and you just bought the thing that is about to be worth less. Slow fills are the good ones. Fast fills are the market telling you it agreed with your price for a reason you will not like. That inversion, fast fills are bad fills, sits under every decision to use a market order or a limit order in a prediction market. The old article ["Adverse Selection Explained for Traders"](https://aligrithm.com/adverse-selection-explained-for-traders/) built the general case for order books. This is the prediction-market cut, where the contract pays a dollar or zero and the information asymmetry is sharp. ## Two ways to touch the book There are exactly two modes, and they trade opposite risks. Taking liquidity means sending a market order or an aggressive limit that crosses the spread and fills immediately against resting orders. You get certainty of execution and you pay the spread, and you expose yourself to adverse selection because you are trading right now, against whoever is resting there, for reasons you have not checked. Providing liquidity means posting a limit order at or inside the spread and waiting. You earn the spread when someone trades against you, and you carry the opposite risk: informed counterparties pick off your quote when the fair value has moved and you have not updated yet. The old article ["Market Making Is Not Just Collecting the Spread"](https://aligrithm.com/market-making-is-not-just-collecting-the-spread/) makes the point that the spread is not free money, it is payment for taking that risk. Structural arbitrage from "Arbitrage Is Just Projection" is a taking decision, because the edge disappears in a block and you cannot wait. Most other edges are not, and treating them as taking decisions burns the spread for no reason. ## Adverse selection, measured When your market order fills, someone took the other side. Ask why. Most of the time the answer is that they had better information, and you can measure how badly that hurt with a single ratio: fill quality. $$ \\mathrm{FQ} \\;=\\; \\frac{p\_{\\text{exec}} - p\_{\\text{mid}}}{p\_{\\text{ask}} - p\_{\\text{bid}}} $$ For a buy, the numerator is how far above the midpoint you actually executed, and the denominator is the spread. A negative fill quality means you bought above where the market thinks fair value sits, which is the fingerprint of adverse selection. Track it across order types and the pattern is stark. | Order type | Average fill quality | | -------------------------------------- | -------------------- | | Market orders | \-0.72 | | Limit orders filled in under 1 minute | \-0.31 | | Limit orders filled in over 10 minutes | +0.43 | Read top to bottom. Market orders are badly adversely selected, because you cross the spread and trade against resting flow that had every chance to pull if the price was wrong. Limit orders that fill fast are still negative, because a quick fill means the market came to your price on its way through. Limit orders that sit for ten minutes before filling come out positive, because a slow fill means someone traded against you for liquidity reasons, not information, and you earned the spread you posted for. The metric confirms the slogan: the faster the fill, the worse the adverse selection. The companion piece ["Fast Fills Are Bad Fills: The Numbers"](https://aligrithm.com/fast-fills-are-bad-fills-the-numbers/) digs into just this table if you want the narrow version. ## Market making is selling insurance, not forecasting A market maker in a prediction market is not predicting the election. They are selling short-term insurance against price fluctuation and getting paid the spread for it. The profit model has two terms, and the second is the one that kills amateurs. $$ \\text{Profit} \\;=\\; \\text{spread} \\times \\text{volume} \\;-\\; \\text{adverse selection} \\times \\text{informed volume} $$ The first term is the spread you earn across all the uninformed flow that trades against you for liquidity. The second term is what you pay when informed traders pick you off. Market making is profitable only when the spread you collect from noise traders exceeds what the informed ones extract. That is the entire business, and it is why "just collect the spread" is a way to go broke: you collect the spread and pay it back to the people who knew more, and your edge is the difference. Two levers manage the second term. Inventory skew: when you are already long a contract, tighten your ask and widen your bid to encourage flow that flattens you, because a lopsided book is a directional bet you did not intend to make. This is the same skew logic as the old article ["Why Skewing Is Simpler Than People Think,"](https://aligrithm.com/why-skewing-is-simpler-than-people-think/) ported to binary contracts. And the truth serum for whether you are actually making money as opposed to accumulating toxic inventory is markouts, the subject of the old article ["Markouts: The Truth Serum of Market Making"](https://aligrithm.com/markouts-the-truth-serum-of-market-making/): look at where the price is a few minutes after your fill, not at the spread you nominally captured. ## The spread is posterior variance Here is the part specific to prediction markets, and it ties this chapter to the Bayesian machinery. How wide should you quote? As wide as your uncertainty about the true probability. The fair value of a binary contract is a probability, and you do not know it exactly. You hold a posterior distribution over it, and the width of that posterior is your uncertainty. The optimal spread should track that posterior variance. When you are confident about the true probability, quote tight and capture volume. When you are uncertain, because news is breaking or the market is thin or resolution is far off, quote wide, because a tight quote in high uncertainty is an invitation for informed traders to pick the side you got wrong. This is the direct link to the article on Bayesian edge in log-odds: greater uncertainty about the event probability means wider optimal spreads, and the spread is not a fee schedule, it is a readout of how much you know. The old article ["Spread Widening During Volatility Expansion"](https://aligrithm.com/spread-widening-during-volatility-expansion/) shows the same reflex on continuous instruments: uncertainty up, spread up. ## The decision table Put edge type, magnitude, and time decay together and the make-versus-take choice falls out. | Situation | Mode | | ----------------------------------------------------------- | --------------------------------------------------- | | Structural arbitrage detected (prices outside the fair set) | Take. Speed matters, the edge dies in a block. | | Large informational edge | Take. Capture it before it diffuses into the price. | | Small informational edge | Provide. Earn the spread on top of the edge. | | No edge, heavy flow | Provide. Pure market making, sell the insurance. | | Thin book, no flow | Neither. Abstain. | The logic is edge magnitude against time decay. A structural arb that vanishes in one Polygon block demands aggressive taking, and paying the spread is worth it because the alternative is not getting the trade. A small informational edge that persists for hours is wasted on a market order, because you pay the spread to capture something you had time to earn passively. The old article ["Maker vs Taker Edge: Same Signal, Different Economics"](https://aligrithm.com/maker-vs-taker-edge-same-signal-different-economics/) makes the general version: the same signal is a different trade depending on which side of the spread you are on. The bottom row matters most for discipline. Thin book, no flow, no edge: do nothing. Abstaining is a position, and it beats manufacturing trades to feel busy. ## Visualizing make vs take ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_360-visual-0.png) ## KEY POINTS - Fast fills are bad fills. A limit order that fills in seconds filled because the market moved through your price toward something bad. Slow fills earn the spread from liquidity traders; fast fills hand you adverse selection. - Two modes only: take liquidity (market order, instant, pay the spread, exposed to adverse selection) or provide it (limit order, earn the spread, get picked off by informed flow). Structural arbitrage is a take; most edges are not. - Fill quality is the execution price minus the midpoint over the spread. Negative means adverse selection. Market orders average -0.72, sub-minute limit fills -0.31, ten-minute-plus limit fills +0.43\. Speed and fill quality move opposite. - Market making is selling insurance against price fluctuation, not forecasting. Profit is spread times volume minus adverse selection times informed volume, and it is positive only when noise-trader spread exceeds informed-trader losses. Skew quotes to manage inventory, and check markouts, not nominal spread, to know if you are winning. - The spread should equal your posterior variance over the true probability. Confident, quote tight and capture volume. Uncertain, quote wide, because a tight quote under uncertainty invites informed traders to pick your wrong side. This is the Bayesian-edge link. - Make-versus-take follows edge magnitude and time decay: take a structural arb (dies in a block) or a large informational edge (before it diffuses); provide on a small persistent edge or pure flow; abstain on a thin book with no edge. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Bid, Ask and Transaction Prices in a Specialist Market with Heterogeneously Informed Traders - Glosten and Milgrom (1985)](https://scholar.google.com/scholar?q=%22Bid%2C+Ask+and+Transaction+Prices+in+a+Specialist+Market+with+Heterogeneously+Informed+Traders%22+Glosten+and+Milgrom+1985&ref=aligrithm.com) - [Continuous Auctions and Insider Trading - Kyle (1985)](https://scholar.google.com/scholar?q=%22Continuous+Auctions+and+Insider+Trading%22+Kyle+1985&ref=aligrithm.com) ### 9.17 Execution Is Part of Expected Value: VWAP Slippage, the Five-Cent Threshold, and Why Copytrading an Arb Loses URL: https://aligrithm.com/execution-is-part-of-expected-value-vwap-slippage-the-five-cent-threshold-and-why-copytrading-an-arb-loses/ Last updated: 2026-07-27T13:26:13.000Z The companion article "Arbitrage Is Just Projection" hands you a number: the guaranteed profit of the optimal trade, say 40 cents, proven risk-free across every outcome. That number is a lie until you fill both legs. A 40-cent arbitrage where one leg fills and the other does not is not a 40-cent trade. It is a directional bet you never meant to place, and it can lose. The whole guarantee lives or dies on execution, and execution is not a footnote to the math. It is a term in the expected value, and usually the term that decides the sign. This is the same lesson from the old article ["Why Traders Lose Even When Their Ideas Are Good,"](https://aligrithm.com/why-traders-lose-even-when-their-ideas-are-good/) where a Sharpe-1 rule turns negative live because the seven execution layers between signal and P&L each skim their cut. Prediction-market arbitrage has fewer layers and sharper teeth. Miss a leg on a binary and you own a naked position that pays a dollar or zero. ## The trades do not fill together Polymarket runs a central limit order book, and multi-leg arbitrage fills sequentially, one order at a time. There is no atomic "buy the whole bundle" button. You send leg one, it fills, and that fill is public information that moves the book before you send leg two. Watch it break. Your plan says buy YES at 30 cents and NO at 30 cents, total 60 cents, guaranteed dollar back, 40 cents locked. - Leg one, YES at 30 cents, fills. - The fill moves the market. Other participants see it and react. - Leg two, NO, now shows a best ask of 78 cents. - Total cost 30 plus 78, which is 108 cents, against a dollar payout. You lose 8 cents. You detected a real, provable arbitrage and still lost money, because the arbitrage existed at prices you could only get by filling both legs at once, and you could not fill both legs at once. The pre-execution guarantee became a post-execution directional bet the instant leg one printed. ## Chain the fill probabilities, and watch them rot Model execution as a stochastic process. For a trade with k legs, let the fill of each leg be an event, and the probability that all of them fill is the product of conditional probabilities. $$ \\Pr(\\text{all fill}) \\;=\\; \\Pr(F\_1)\\,\\Pr(F\_2 \\mid F\_1)\\,\\cdots\\,\\Pr(F\_k \\mid F\_1,\\dots,F\_{k-1}) $$ Read the conditioning bars, because they are the whole point. The probability that leg two fills is conditioned on leg one having filled, and that condition makes it worse, not better, because leg one's fill moved the price against you. These probabilities are not independent, and they do not stay flat. Each fill degrades the next. A four-leg combinatorial arb needs four dependent events to all break your way, and the joint probability collapses faster than intuition expects. So the honest expected value discounts the guaranteed profit by whether you actually get the whole trade on. $$ \\mathbb{E}\[\\Pi\_{\\text{realised}}\] \\;=\\; \\Pr(\\text{full}) \\cdot \\bigl(\\Pi^\* - \\mathbb{E}\[\\text{slippage}\]\\bigr) \\;+\\; \\Pr(\\text{partial}) \\cdot \\mathbb{E}\[\\Pi\_{\\text{partial}}\] $$ The first term is the good case: everything fills, and you keep the theoretical profit minus slippage. The second term is the trap: a partial fill leaves you holding some legs and not others, and that stranded position usually loses. The expected partial profit is negative, and when the full-fill probability is low, the second term dominates and the whole expression goes negative. A "guaranteed 40 cents" with a 50 percent chance of stranding you on a leg that loses 60 cents is a losing trade, and the projection math never told you that. Execution did. ## VWAP: what you actually pay when you eat the book The theoretical profit assumes you buy at the quoted price. You do not. You buy at the volume-weighted average price across every level your order eats, and the gap between that average and the top quote is slippage. $$ \\mathrm{VWAP}(S) \\;=\\; \\frac{1}{S}\\sum\_{j=1}^{L} \\min\\!\\bigl(w\_j,\\, (S - \\textstyle\\sum\_{k The S is your order size, the pairs of price and size are the levels of the ask book sorted cheapest first, and the formula fills as much as it can at each level before climbing to the next. Slippage is that average minus the best quote. Code the sweep once and it is clearer than the formula. ```python def vwap_and_slippage(book, size): # book: list of (price, size_available), sorted by ascending price spent, filled = 0.0, 0 for price, avail in book: take = min(avail, size - filled) # eat this level, up to what's left spent += take * price filled += take if filled >= size: break if filled < size: raise ValueError("book too thin to fill the order") # liquidity failure vwap = spent / size slippage = vwap - book[0][0] # avg fill minus the best quote return vwap, slippage, spent ``` Run it on a real-looking book. The ask side has 500 shares at 32 cents, 300 at 34 cents, and 200 at 38 cents. You want 900 shares. | Level | Price | Available | You take | Cost | | ----- | ----- | --------- | -------- | ------ | | 1 | 32c | 500 | 500 | 160.00 | | 2 | 34c | 300 | 300 | 102.00 | | 3 | 38c | 200 | 100 | 38.00 | Total spend is 300 dollars for 900 shares, a VWAP of 33.3 cents. The best quote was 32 cents. You paid 1.3 cents more per share on average, and on 900 shares that is 11.70 dollars of execution cost that never appeared in the theoretical profit. If the projection said the edge was 15 dollars and slippage across both legs eats 12, your real edge is 3 dollars, and it should not be traded. ## The five-cent threshold is not arbitrary Put the pieces together and you get the rule practitioners actually use: skip any arbitrage whose guaranteed profit is under about five cents. That floor is where the execution terms above stop being noise and start being the whole trade. Below five cents, the slippage on two legs, the gas to submit on-chain, and the negative expected value of a partial fill reliably exceed the edge. The projection can hand you a two-cent "guarantee" all day. Every one of them loses after execution. The maximum you can extract is also capped by the thinnest leg, not the fattest. $$ \\Pi\_{\\max} \\;=\\; \\text{price deviation} \\times \\min(\\text{depth across all legs}) $$ You cannot size the trade to the deep leg and hope the thin one keeps up. The thin leg sets the ceiling, and for a multi-condition arbitrage the depth has to exist simultaneously on every required position, at the same instant, or the trade is not there at the size you want. The old article ["Why Transaction Costs Should Be Added Before You Fall in Love"](https://aligrithm.com/why-transaction-costs-should-be-added-before-you-fall-in-love/) makes the general version of this point: apply costs at the prototype stage, before you have decided the strategy works. ## Latency: you are not fast enough, and the numbers say so Total time from spotting the opportunity to on-chain confirmation breaks into four pieces: detect, compute, submit, confirm. The retail-versus-professional gap is not a margin. It is three orders of magnitude. | Component | Retail | Professional | | ------------ | --------------- | -------------- | | Detection | 30,000 ms | 5 ms | | Computation | 10,000 ms | 10 ms | | Submission | 8,000 ms | 15 ms | | Confirmation | 2,000 ms | 2,000 ms | | Total | about 50,000 ms | about 2,030 ms | The confirmation time is shared, because that is the chain. Everything above it is you, and a retail stack spends 50 seconds where a professional spends two. The critical window is the roughly 30-millisecond gap between decision and getting into the mempool, and the goal is to land all legs inside a single Polygon block, about two seconds, so no price moves between legs. Same-block execution is the only version of this trade where the sequential-fill problem from the top of the article disappears. Miss the block and you are back to legging in against a moving book. ## Why copytrading an arbitrage is exit liquidity This kills the obvious retail shortcut. Someone watches a winning arbitrage bot, sees it capture an opportunity in block N, and copies the trade. By the time you can see block N, the arbitrage in it is gone, because the bot's fills are what closed it. You submit at block N plus one, buying at post-arbitrage prices, which means you are the counterparty the smart flow needed to exit. You are not capturing the arb. You are providing the liquidity that lets the winner realize it. The empirical realization rates tell the same story from the other side. Single-condition arbitrages, the simple two-leg kind, get realized most of the time, on the order of 87 percent, and they fail mostly on liquidity. The complex trades fail far more. | Type | Detected | Realized | Main failure | | ------------------ | -------- | --------- | ---------------------- | | Single condition | 7,051 | about 87% | liquidity | | Market rebalancing | 662 | about 42% | price movement | | Combinatorial | 13 | about 45% | simultaneous liquidity | Read the second and third rows as a warning. The trades the earlier articles in this pillar make possible, the multi-leg combinatorial ones from the projection and Frank-Wolfe machinery, are exactly the ones that fail on execution more than half the time, because they need many fills to land together against books that move the instant you touch them. The math is elegant and the execution is brutal, and only the execution shows up in your account. ## Visualizing execution risk ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_359-visual-0.png) ## KEY POINTS - A guaranteed arbitrage profit is fiction until both legs fill. Multi-leg trades on a central limit order book fill sequentially, and the first fill moves the book against the second, so a proven arb can become a losing directional bet. - The probability all legs fill is a product of conditional probabilities that get worse as you go, because each fill degrades the next. The expected value has a partial-fill term that is negative and often dominates. - You pay the volume-weighted average price across every level your order eats, not the top quote. On a 900-share order sweeping 32c/34c/38c you pay 33.3c average, and the 1.3c slippage is real money the theoretical profit ignored. - The five-cent minimum profit threshold is not arbitrary. Below it, two legs of slippage, gas, and negative partial-fill expected value reliably exceed the edge. Extractable size is capped by the thinnest leg, and combinatorial arbs need depth on every leg at the same instant. - Latency is three orders of magnitude between retail (about 50 seconds) and professional (about 2 seconds). The goal is all legs inside one Polygon block so no price moves between them. Miss the block and you leg in against a moving market. - Copytrading an arb is exit liquidity. By the time you see the block that captured it, the opportunity is gone, and your copy buys at post-arb prices. Single-condition arbs realize around 87 percent of the time; combinatorial ones fail more than half. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Unravelling the Probabilistic Forest: Arbitrage in Prediction Markets - Saguillo et al. (2025)](https://arxiv.org/abs/2508.03474?ref=aligrithm.com) ### 9.16 Kelly Isn't Enough: Drawdown- and CVaR-Constrained Sizing and the Portfolio QP URL: https://aligrithm.com/kelly-isnt-enough-drawdown-and-cvar-constrained-sizing-and-the-portfolio-qp/ Last updated: 2026-07-27T13:26:12.000Z Kelly maximizes growth. It does not promise you survive the path. The article "Kelly Rejects Bad Trades Automatically" ended on the variance buried inside the growth-optimal fraction: a real edge at full Kelly still draws down past 50 percent one time in ten. Growth-optimal and survivable are different objectives, and a trader who optimizes the first without constraining the second gets margin-called out of a winning strategy. This article adds the two constraints that keep you in the game, drawdown and tail risk, and then combines many trades into one allocation that accounts for the way arbitrage positions secretly move together. ## Maximum drawdown, and why even optimal strategies bleed Drawdown is the peak-to-trough drop in your capital, measured as a fraction of the peak. $$ \\mathrm{MDD} = \\max\_{t\_1 < t\_2} \\frac{W(t\_1) - W(t\_2)}{W(t\_1)} $$ The MDD is the worst such drop over the whole path: take every peak, find the deepest valley that follows it, keep the largest percentage fall. A mathematically optimal strategy still produces ugly ones, because execution fails on some legs, correlated events cluster losses into the same week, and edges decay as competition arrives. None of those show up in the expected growth rate, and all of them show up in the equity curve. The article "Execution Is Part of Expected Value" is where the first cause gets quantified; here the concern is bounding the damage before it happens. ## Size to a drawdown budget, by simulation State the goal as a constraint on Kelly, not a replacement for it. Maximize growth subject to keeping the odds of a drawdown worse than your budget below some tolerance. $$ \\max\_f\\; G(f) \\quad \\text{subject to} \\quad P(\\mathrm{MDD} > D\_{\\max}) \\le \\alpha $$ The D-max is the worst drawdown you will accept, say 15 percent. The alpha is how often you will tolerate breaching it. There is no clean formula for the f that satisfies this, so you simulate. Pick a candidate f, generate many wealth paths of realistic length, compute the drawdown of each, and check whether the fraction of paths breaching D-max stays under alpha. Too many breaches, cut f and repeat. Binary-search your way down to the largest f that respects the budget. The heuristic that falls out of running this against arbitrage-style profit-and-loss profiles is worth carrying. $$ f\_{\\text{constrained}} \\approx 0.3\\text{ to }0.5 \\times f^\* $$ A 15 percent drawdown budget lands you around a third to a half of full Kelly. Notice this sits right in the fractional-Kelly range from the previous article, reached by a different road. There the argument was estimation error; here it is path survival. Two independent reasons point at the same answer: bet a fraction of Kelly, not the whole thing. ## VaR tells you the boundary; CVaR tells you how bad it gets past it Drawdown constrains the sequence. Tail risk constrains the single catastrophic outcome, and the standard tool most people reach for hides exactly the number that matters. Value at Risk answers "I will not lose more than this on 95 percent of days." Useful, and dangerous, because it says nothing about the other 5 percent. Conditional Value at Risk answers the question that actually kills you: when you are in that worst 5 percent, how bad is it on average. $$ \\mathrm{CVaR}\_\\alpha = \\mathbb{E}\\big\[\\, \\text{Loss} \\;\\big|\\; \\text{Loss} > \\mathrm{VaR}\_\\alpha \\,\\big\] $$ CVaR is the expected loss conditional on being past the VaR boundary. Work the example. Run 100 trades, sort by profit, and the worst five lose 80, 60, 45, 30, and 20 dollars. VaR at 95 percent is 20 dollars, the boundary of the worst five. CVaR at 95 percent is the average of those five, which is 47 dollars. $$ \\mathrm{VaR}\_{95\\%} = \\$20, \\qquad \\mathrm{CVaR}\_{95\\%} = \\frac{80 + 60 + 45 + 30 + 20}{5} = \\$47 $$ VaR reports 20 and calls it a day. CVaR reports 47, more than double, and that is the honest number. The gap is the point: a strategy with a VaR of 20 but a CVaR of 200 has a fat tail that VaR completely conceals, and you find out about it the hard way. Constrain CVaR, not VaR. For a single arbitrage trade the loss is not a smooth distribution; it has three branches, and the middle one is the dangerous one. $$ \\text{Loss} = \\begin{cases} -(\\Pi^\* - \\Delta) & \\text{full execution (a small win, or the intended profit)}\\\\ -(\\text{partial cost} - \\text{recovery}) & \\text{partial execution (the bad branch)}\\\\ 0 & \\text{no execution} \\end{cases} $$ Full execution delivers the guaranteed profit. No execution costs nothing but opportunity. Partial execution is where a "risk-free" arb becomes a naked directional position: one leg filled, the other stranded, and you are exposed to the outcome you thought you had hedged. A CVaR constraint exists to stop those partial-execution disasters from dominating your tail, because they are correlated across trades in exactly the moments when everything else is going wrong too. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_358-visual-0.png) ## Many trades, one allocation: the portfolio QP Sizing trades one at a time misses the thing that blows up arbitrage books: the positions are not independent. Five election arbs that each look clean can all lose together on one national shock. Allocate across all of them at once with a single optimization. $$ \\max\_{f\_1, \\ldots, f\_k}\\; \\sum\_i f\_i e\_i - \\tfrac{1}{2}\\, \\mathbf{f}^\\top \\Sigma\\, \\mathbf{f} \\quad \\text{s.t.} \\quad \\sum\_i f\_i \\le F\_{\\max},\\;\\; 0 \\le f\_i \\le f\_i^{\\max},\\;\\; \\mathrm{CVaR} \\le C\_{\\max} $$ Read it piece by piece. The first sum is total expected profit, each fraction times its edge. The second term, with the covariance matrix sigma, is a penalty for correlated risk: when two trades tend to lose together, sigma grows and the penalty pushes the allocator to hold less of both, forcing diversification the trader would not do by eye. The constraints cap total deployed capital, cap each individual position, and cap the portfolio's tail loss. This shape is a quadratic program, a squared objective with linear constraints, and standard solvers like CVXPY, Gurobi, or OSQP handle hundreds of trades in milliseconds. The article "Correlation Lies; Tail Dependence Tells the Truth" is where the covariance matrix gets replaced with something that captures joint tails properly, because plain correlation understates exactly the co-movement that matters here. This is the same QP the production allocator reuses. The article "The Central Allocator and Non-Negotiable Risk Governors" wraps it in a live loop with kill switches; here it is the sizing math underneath. ## Priority when capital runs out Often you cannot fund every good trade. Rank them by profit and confidence per unit of execution risk, and fund from the top. $$ \\text{Priority}\_i = \\frac{\\text{profit}\_i \\times \\text{confidence}\_i}{\\text{execution\\\_risk}\_i} $$ A high-profit trade with shaky execution ranks below a modest one that fills reliably, which is the correct instinct: guaranteed capture of a smaller edge beats a lottery ticket on a larger one. Allocate down the priority list and keep 20 percent of capital in reserve, because the point of surviving is to have dry powder when the next mispricing appears and the book is not already full. The through-line across both survival articles is one idea stated twice. Growth-optimal sizing assumes a world with perfect inputs and infinite runway. You have neither. Estimation error argues for a fraction of Kelly, path survival argues for the same fraction, CVaR stops the fat tail you cannot see, and the portfolio QP stops the correlated blowup you did not price. None of it increases your edge. All of it keeps you alive long enough to collect the edge you already have, which the article ["Match Your Edge to Your Resources"](https://aligrithm.com/match-your-edge-to-your-resources/) argues is the whole job once the math is done. ## KEY POINTS - Kelly maximizes growth, not survival. A real edge at full Kelly still draws down past 50 percent one time in ten, so you constrain the path, not just the growth rate. - Size to a drawdown budget by simulation: pick f, simulate many paths, keep the largest f whose chance of breaching your drawdown limit stays under tolerance. A 15 percent budget lands around 0.3 to 0.5 times Kelly, the same range fractional Kelly reaches from the estimation-error argument. - VaR reports the boundary of the bad region and hides everything past it. CVaR reports the average loss once you are past it. In the worked example VaR is 20 dollars and CVaR is 47; a VaR of 20 with a CVaR of 200 has a fat tail VaR conceals. Constrain CVaR. - An arbitrage trade's loss has three branches, and partial execution, one leg filled and the other stranded, turns a risk-free arb into a naked directional bet. CVaR constraints exist to stop that branch from dominating the tail. - Allocate across all trades at once with the portfolio QP: maximize edge minus a correlated-risk penalty, subject to caps on total capital, per-trade size, and portfolio CVaR. The covariance penalty forces diversification, and QP solvers handle hundreds of trades in milliseconds. - When capital is scarce, rank trades by profit times confidence over execution risk, fund from the top, and hold 20 percent in reserve. None of this raises your edge; all of it keeps you solvent enough to collect it. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Optimization of Conditional Value-at-Risk - Rockafellar and Uryasev (2000)](https://scholar.google.com/scholar?q=%22Optimization+of+Conditional+Value-at-Risk%22+Rockafellar+and+Uryasev+2000&ref=aligrithm.com) - [Convex Optimization - Boyd and Vandenberghe (2004)](https://scholar.google.com/scholar?q=%22Convex+Optimization%22+Boyd+and+Vandenberghe+2004&ref=aligrithm.com) ### 9.15 Kelly Rejects Bad Trades Automatically: Why Overbetting Kills and Underbetting Doesn't URL: https://aligrithm.com/kelly-rejects-bad-trades-automatically-why-overbetting-kills-and-underbetting-doesnt/ Last updated: 2026-07-27T13:26:12.000Z You found a real arbitrage. The article "Arbitrage Is Just Projection" handed you the trade, the guaranteed profit, and the target prices. None of that tells you how much to bet, and the sizing decision destroys more accounts than bad trade selection. Size too small and you leave growth on the table, survivable and boring. Size too large and you go broke holding a positive edge, which is the outcome nobody expects and plenty of people reach. Kelly gives the growth-optimal fraction, and read correctly it does two things most sizing rules do not: it tells you when a trade is not worth taking at all, and it makes the penalty for guessing wrong asymmetric in a way that should terrify you into betting less. ## The formula, and where every input comes from Bet a fraction of your bankroll each round on a bet that pays b per dollar risked when you win, with win probability p. Your wealth after a round is either multiplied up or cut down, and over many rounds what compounds is the log of those multipliers. Maximize the expected log-growth. $$ G(f) = p \\ln(1 + fb) + q \\ln(1 - f), \\qquad q = 1 - p $$ G is the long-run growth rate as a function of the fraction f you bet. The first term is the win branch weighted by its probability, the second the loss branch. Set the derivative to zero and the maximizer falls out clean. $$ f^\* = \\frac{pb - q}{b} = p - \\frac{q}{b} $$ Three inputs, nothing hidden. The p is your estimated win probability, q is one minus that, and b is the profit per dollar risked on a win. Everything in the sizing decision reduces to those three numbers, which means every sizing error reduces to an error in one of them. Three worked cases show the range of what the formula says. Even-money bet, b equals 1, so the formula collapses to 2p minus 1\. Estimate a 55 percent win rate and f-star is 0.10\. Bet 10 percent of capital per round. Standard textbook result. Prediction-market arbitrage, guaranteed profit of 8 cents per dollar deployed, so b is 0.08, and execution reliability pulls the effective win probability to 0.90\. Now f-star is 0.90 times 0.08 minus 0.10, all over 0.08, which is negative 0.35. $$ f^\* = \\frac{0.90 \\times 0.08 - 0.10}{0.08} = \\frac{0.072 - 0.10}{0.08} = -0.35 $$ Negative. A negative Kelly fraction is not a short signal; it is a refusal. The execution risk is too high for the thin margin, and the formula says do not put money on this. Most sizing tutorials never mention that Kelly outputs "no trade," and it is the most useful thing it does. The article "The Six Ways to Lose Money on Polymarket" catalogs traders who took this exact trade anyway because 8 cents looked free. Same 8-cent profit, but execution is nearly certain, p equals 0.99\. Now f-star is 0.865: bet 86.5 percent of capital. Correct, and insane. Nobody sane bets 86 percent of their bankroll on one trade, which is the entire reason fractional Kelly exists and full Kelly does not get deployed raw. ## What Kelly promises, and the variance it hides The Breiman results are the reason Kelly is the reference point. Among all fixed-fraction strategies, f-star maximizes the long-run growth rate almost surely, reaches any wealth target in the least expected time, and never risks total ruin, because f-star stays below 1 whenever you have a positive edge. Optimal growth, minimum time, no ruin. It sounds like a free lunch. The bill is variance. A 55 percent even-money bettor sizing at the full f-star of 0.10 sees drawdowns past 50 percent at the 10th percentile over a thousand rounds. Half your capital gone, one time in ten, running the mathematically optimal strategy with a genuine edge. And that assumes you know p and b exactly, which you never do. Full Kelly is the growth-optimal size for a world where your inputs are perfect. That world does not exist. ## The penalty is quadratic, and it only bites in one direction Bet the wrong fraction, off from optimal by an amount delta, and the growth rate drops. $$ G(f^\* + \\Delta) \\approx G(f^\*) - \\frac{\\Delta^2}{2\\sigma^2} $$ The loss in growth is proportional to the square of the sizing error. Small misses barely register: a delta of 0.02 costs almost nothing. Large misses compound: a delta of 0.15 can drive the growth rate below zero, and below zero means you go broke over time even with a real edge. The square is the whole story, because it means being twice as wrong is four times as costly. The asymmetry is what turns a math result into a survival rule. Overbetting and underbetting are not mirror images. Underbet, f below f-star, and growth slows but stays positive. You compound more slowly, you never blow up. Overbet, f above f-star, and growth falls, then past a threshold turns negative, and negative growth is certain ruin. One direction is slow. The other is fatal. The numbers make it concrete. True edge supports f-star of 0.04, but you estimate 0.20, a 5-times overbet. That is not an exotic error; it is what happens when you take a noisy edge estimate at face value. Over 100 trades at that size, the probability of ruin exceeds 30 percent. You had a real, positive edge and a one-in-three chance of being wiped out, entirely from sizing. $$ \\text{true } f^\* = 0.04, \\quad \\hat{f}^\* = 0.20 \\;\\Rightarrow\\; 5\\times \\text{ overbet} \\;\\Rightarrow\\; P(\\text{ruin over 100 trades}) > 30\\% $$ ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_357-visual-0.png) ## Fractional Kelly, and the arbitrage-specific version Because you never know your edge exactly, you bet a fraction of the fraction. The tradeoff table is the practical core of the chapter. $$ \\begin{array}{lccl} \\textbf{Fraction} & \\textbf{Growth vs full} & \\textbf{Variance} & \\textbf{When to use}\\\\ 1.0 \\times f^\* & 100\\% & \\text{high} & \\text{edge is precisely known (it never is)}\\\\ 0.5 \\times f^\* & 75\\% & \\sim 50\\% \\text{ lower} & \\text{moderate estimation error}\\\\ 0.25 \\times f^\* & \\sim 44\\% & \\sim 75\\% \\text{ lower} & \\text{edge estimate is uncertain} \\end{array} $$ Read the middle row. Half-Kelly keeps 75 percent of the growth while cutting variance roughly in half. You give up a quarter of your growth to halve the chance of a brutal drawdown, and given the asymmetry above, that is a trade worth making every time. The rule of thumb: structural arbitrage, where the edge is derived from the geometry rather than forecast, can carry 0.5 to 0.75 times Kelly because the edge estimate is tighter; informational edge, where you are guessing at a probability, gets 0.25 to 0.5 times because the estimate is soft. The article "Bayesian Edge in Log-Odds" is where that softness gets measured, and the shrinkage it prescribes is the same instinct as fractional Kelly applied one layer up. Structural arbitrage needs one more adjustment, because the guaranteed profit is only guaranteed if you actually fill both legs. $$ f\_{\\text{arb}} = f^\* \\times P(\\text{full execution}) \\times (1 - s) $$ Take the Kelly fraction, scale it by the probability that all legs fill, then scale again by one minus the slippage as a fraction of the edge. A trade with a clean geometric edge but a 60 percent full-fill probability and slippage eating a fifth of the margin gets sized far below its naive Kelly number. And cap the position at half the order-book depth regardless, because a position larger than that moves the market against you as you build it, which is its own tax measured in the article "Execution Is Part of Expected Value." The real accounts confirm the shape. Reconstructing the top Polymarket account from its profit and win-rate statistics puts per-trade notionals in the low thousands of dollars against a capital base in the hundreds of thousands, well under 1 percent of capital per trade. That is not timidity. That is fractional Kelly plus aggressive diversification across many small, high-certainty structural bets, which is how a 90-plus percent win rate compounds into seven figures without a ruinous drawdown. The edge was never in betting big. It was in betting small, often, and never taking the negative-f-star trades. ## KEY POINTS - Kelly sizes for growth-optimal compounding: f-star equals p minus q over b, using your win probability p and the profit per dollar b. Every sizing error is an error in one of those inputs. - A negative f-star is a refusal, not a short. The arb example with 8-cent profit and 90 percent execution gives f-star of negative 0.35, which means do not trade. Kelly outputting "no trade" is its most useful and most ignored feature. - Breiman: f-star maximizes long-run growth, reaches targets fastest, and never fully ruins you. The cost is variance, with 50-percent-plus drawdowns at the 10th percentile even at full Kelly with a real edge, and it assumes inputs you never have exactly. - The growth penalty is quadratic in the sizing error, so twice as wrong is four times as costly. And it is asymmetric: underbetting slows growth but survives, overbetting past a threshold turns growth negative, which is certain ruin. - A 5-times overbet (true f-star 0.04, estimated 0.20) carries over 30 percent ruin probability across 100 trades, with a genuine positive edge. This asymmetry is the entire case for fractional Kelly. - Use half-Kelly to keep 75 percent of growth at half the variance. Structural edge takes 0.5 to 0.75 times, informational edge 0.25 to 0.5 times. For arbitrage, scale further by fill probability and slippage, and cap at half the book depth. The top real account sized well under 1 percent per trade and diversified hard. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [A New Interpretation of Information Rate - Kelly (1956)](https://scholar.google.com/scholar?q=%22A+New+Interpretation+of+Information+Rate%22+Kelly+1956&ref=aligrithm.com) - [Optimal Gambling Systems for Favorable Games - Breiman (1961)](https://scholar.google.com/scholar?q=%22Optimal+Gambling+Systems+for+Favorable+Games%22+Breiman+1961&ref=aligrithm.com) - [The Kelly Criterion in Blackjack, Sports Betting, and the Stock Market - Thorp (2006)](https://scholar.google.com/scholar?q=%22The+Kelly+Criterion+in+Blackjack%2C+Sports+Betting%2C+and+the+Stock+Market%22+Thorp+2006&ref=aligrithm.com) - [Unravelling the Probabilistic Forest: Arbitrage in Prediction Markets - Saguillo et al. (2025)](https://arxiv.org/abs/2508.03474?ref=aligrithm.com) ### 9.14 Bayesian Edge in Log-Odds: Better, Earlier, Calibrated — or It's Noise URL: https://aligrithm.com/bayesian-edge-in-log-odds-better-earlier-calibrated-or-its-noise/ Last updated: 2026-07-27T13:26:11.000Z You passed the Brier Skill Score gate from the article on what a market price is. Your forecasts beat the market on historical resolutions. Now you want to turn a specific piece of news into a specific edge, and the tool is Bayes' rule. It is a clean update formula, and it is also where informational traders quietly destroy themselves, because the whole calculation hangs on a number you have to estimate and usually estimate too confidently. This article shows the update in the form that makes the danger visible, ties it to the market maker's own internals, and then spends most of its length on the three tests a belief must pass before it counts as edge. The market price is your prior. Evidence moves it. The question is by how much, and the honest answer is smaller than you think. ## Bayes as an update, worked Start with the market price as your prior and let a poll arrive. Candidate A trades at $0.52, so the prior is 0.52\. A poll shows A leading. You judge that poll is likely if A will win and less likely if A will lose. $$ P(H \\mid E) = \\frac{P(E \\mid H)\\, P(H)}{P(E \\mid H)\\, P(H) + P(E \\mid \\neg H)\\, P(\\neg H)} = \\frac{0.70 \\times 0.52}{0.70 \\times 0.52 + 0.25 \\times 0.48} \\approx 0.752 $$ Read that as: the posterior probability that A wins given the poll equals the likelihood of the poll under a win times the prior, divided by the total probability of seeing that poll either way, and plugging in a 0.70 likelihood under a win and 0.25 under a loss lifts the belief from 0.52 to about 0.752\. That is a 23-point edge over the market, if the two likelihood numbers are right. They are the whole trade. ## The odds form makes the danger visible The fraction hides where the action is. Rewrite the update in odds and it becomes a single multiplication. $$ \\text{new odds} = \\text{old odds} \\times \\text{likelihood ratio}, \\qquad \\frac{0.52}{0.48} \\times \\frac{0.70}{0.25} = 1.083 \\times 2.8 = 3.033 \\;\\Rightarrow\\; p = \\frac{3.033}{1+3.033} \\approx 0.752 $$ Read that as: convert the prior to odds, multiply by the likelihood ratio (how many times more likely the evidence is under the hypothesis than against it), and convert back, so the poll's entire effect is the factor 2.8\. The edge lives in that ratio. Set the likelihood under a win to 0.70 when the truth is 0.55 and the ratio drops from 2.8 to 2.2, the posterior falls, and the edge you thought you had shrinks or vanishes. You did not misread the market. You misjudged one conditional probability, and the position sized on the inflated posterior is the belief-without-calibration failure from the article on the six ways to lose. ## Sequential evidence adds in log-odds Evidence rarely arrives once. In log-odds the update stops being a multiplication and becomes an addition, which is why this form scales. $$ \\operatorname{logit}\\bigl(P(H \\mid E\_1,\\ldots,E\_n)\\bigr) = \\operatorname{logit}(P(H)) + \\sum\_{k=1}^{n} \\ln\\frac{P(E\_k \\mid H)}{P(E\_k \\mid \\neg H)} $$ Read that as: take the log-odds of the prior and add, for each piece of independent evidence, the log of its likelihood ratio, so every new item just adds a number to a running total and you never reprocess old evidence. Work the example forward: the prior log-odds are the log of 0.52 over 0.48, about 0.080\. The poll adds the log of 0.70 over 0.25, about 1.030\. An endorsement with likelihood ratio 1.5 adds the log of 1.5, about 0.405\. The total is 0.080 plus 1.030 plus 0.405, which is 1.515, and converting back gives about 0.820\. Each item is one addition. This is not just convenient. The log-market-scoring-rule maker holds its state as exactly this log-odds vector and prices it with a softmax, the point the article on convex geometry makes about softmax being the price formula. Your Bayesian update and the market maker's pricing run on the same substrate. You are playing the maker's own game in the maker's own coordinates. ## The true probability drifts: a filter on price The event's real probability is not fixed. Campaigns, debates, and shocks move it, and the observed price is that moving target plus noise. A state-space model separates the two. $$ x\_t = f(x\_{t-1}) + w\_t \\quad (\\text{hidden true probability}), \\qquad y\_t = x\_t + v\_t \\quad (\\text{observed price}) $$ Read that as: the hidden true probability x evolves from its past value plus process noise w that represents genuine drift, and the price y you observe is that hidden value plus observation noise v from thin liquidity and noise trading. A Kalman filter estimates the hidden state and its residual, the filter's estimate minus the market price, is your trading signal: when the filter says the true probability has moved but the price has not caught up, that gap is temporal edge. For non-linear or non-Gaussian dynamics a particle filter does the same job with more flexibility. The old article ["Volatility Regimes and Strategy Survival"](https://aligrithm.com/volatility-regimes-and-strategy-survival/) frames the same idea of separating a slow-moving signal from fast noise; here the signal is a probability and the noise is microstructure. ## When belief is not edge: three tests A different posterior from the market is not automatically an edge. It counts only if it passes three tests at once. Better: your likelihood model is more accurate than the market's implied one. If the crowd's conditional probabilities are sharper than yours, your update is worse than theirs and your "edge" is negative. Earlier: you process the evidence before the price moves. A correct update that arrives after the market already repriced is worth nothing, the exit-liquidity trap the article on execution covers. Calibrated: when you say 0.70 it happens about 70% of the time, the gate from the article on what a market price is. Fail any one and the perceived edge is noise. The danger is asymmetric and it compounds through sizing. $$ \\text{true edge } 4\\text{ pts, estimated } 20\\text{ pts} \\;\\Rightarrow\\; 5\\times \\text{ Kelly overbet} \\;\\Rightarrow\\; \\text{growth turns negative past a threshold} $$ Read that as: if the real edge is 4 percentage points but you estimate 20, Kelly sizes you five times too large, and because the growth penalty is quadratic in sizing error, overbetting past a threshold drives long-run growth below zero. This is why the article on the nine sources of edge ranks informational trading fifth and calls it the dangerous one: the edge estimate carries large uncertainty, and the penalty for overestimating is severe and one-sided. The old article ["Ranking Beats Forecasting for Many Trading Problems"](https://aligrithm.com/ranking-beats-forecasting-for-many-trading-problems/) is worth holding in mind here, since a relative view is often more robust than an absolute probability estimate. ## Shrinkage: the math of being less sure When you suspect overconfidence, pull the forecast toward maximum uncertainty before sizing. $$ p\_{\\text{shrunk}} = (1-\\lambda)\\, p\_{\\text{model}} + \\lambda \\cdot 0.5, \\qquad \\lambda = 0.3:\\; 0.80 \\to 0.7(0.80) + 0.3(0.5) = 0.71 $$ Read that as: blend your model's probability with a coin flip, weighting the coin flip by lambda, so a shrinkage of 0.3 turns an 80% forecast into 71%, a mechanical way to be less sure. The equivalent log-odds form divides your log-odds by one plus a regularization constant, collapsing toward 0.5 as the constant grows. Tune lambda to minimize your Brier score on a held-out set, and when in doubt shrink more, because underbetting is survivable and overbetting is not. Shrinkage is the direct antidote to the inflated-likelihood danger the odds form exposed: it caps how far a single overconfident conditional probability can push your size. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_356-visual-0.png) ## KEY POINTS - Bayes updates the market prior with evidence, but the whole edge lives in the likelihood ratio, a number you estimate. Overstate one conditional probability and the edge inflates or vanishes. - The odds form makes it plain: new odds equal old odds times the likelihood ratio. That single factor is the trade. - In log-odds the update is additive: each independent evidence item adds its log-likelihood ratio to a running total, no reprocessing needed. This is the exact substrate the log-market-scoring-rule maker uses, so your update and its pricing share coordinates. - The true probability drifts. A Kalman or particle filter on price separates signal from microstructure noise; the residual between filter estimate and price is temporal edge. - A different posterior is edge only if it is better, earlier, and calibrated at once. Fail any one and it is noise. - Overconfidence is lethal through sizing: a 4-point true edge estimated at 20 points is a 5x Kelly overbet, and the quadratic penalty turns growth negative. This is why informational edge ranks fifth and dangerous. - Shrink toward 0.50 before sizing, tune the shrinkage to minimize held-out Brier score, and when in doubt shrink more, because underbetting survives and overbetting does not. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Logarithmic Market Scoring Rules for Modular Combinatorial Information Aggregation - Hanson (2007)](https://scholar.google.com/scholar?q=%22Logarithmic+Market+Scoring+Rules+for+Modular+Combinatorial+Information+Aggregation%22+Hanson+2007&ref=aligrithm.com) ### 9.13 What a Market Price Actually Is: Capital-Weighted Consensus and the Brier Skill Score Gate URL: https://aligrithm.com/what-a-market-price-actually-is-capital-weighted-consensus-and-the-brier-skill-score-gate/ Last updated: 2026-07-27T13:26:11.000Z You checked the price point against the marginal polytope from the article on convex geometry and it sits inside. No structural edge. The prices are coherent, which means they contradict no probability law. The next question is whether coherent prices can still be wrong, and the answer is yes, but the trade that exploits it is the dangerous one this pillar ranks fifth of nine. Before you take it, you owe yourself two things: an honest account of what the $0.60 on your screen means, and a hard numerical gate that tells you whether your forecast is allowed to trade at all. Most participants skip both. They read $0.60 as "the market says 60%," treat that as a fact to disagree with, and bet their disagreement. The price is not a fact. It is the settling point of a capital-weighted argument, and beating it requires proving you are more accurate, not merely more confident. ## A price is not a probability A $0.60 binary contract does not carry a 60% frequentist probability the way a fair die carries one-sixth. It is the price at which the marginal trader stops caring which side to take. $$ p \\;\\approx\\; \\frac{\\sum\_i W\_i \\, P\_i(H)}{\\sum\_i W\_i} $$ Read that as: the price is roughly a weighted average of every participant's private belief P-i about the event H, where each belief is weighted by that participant's capital W-i, so a rich trader with a wrong view drags the price more than a poor trader with a right one. Each trader bets in proportion to how far their belief sits from the market, and over time capital drains from the inaccurate toward the accurate. The price is the residue of that flow. Call it Darwinian information filtering: the selection pressure acts on capital, not on opinions, and a wrong opinion attached to a large bankroll survives longer than it should. This reframing matters because it tells you when to trust the number. The old article ["The Difference Between Explanation and Prediction in Markets"](https://aligrithm.com/the-difference-between-explanation-and-prediction-in-markets/) makes the general case that a confident narrative is not a forecast. Here the price is a genuine forecast, aggregated across money, and the empirical record is good: across 964 polls over five US presidential cycles, prediction-market prices beat the polls about 74% of the time, and the gap widened for forecasts made more than 100 days out. Betting against that aggregate is not free. ## When the aggregate breaks The capital-weighted average is only as good as the capital and the time behind it. Five conditions turn a coherent price into an exploitable one. Capital concentration: one wealthy, wrong trader moves the price where the crowd would not. Thin liquidity: small trades swing the price out of proportion, so the "consensus" is a handful of orders. Short horizons: selection pressure has not had time to move capital away from the inaccurate, so early prices are noisier. Strategic behavior: manipulation and bluffing distort the signal, especially on high-visibility events. Extreme probabilities: above 0.95 or below 0.05 the payout structure makes betting against a near-certainty expensive even when it is mispriced, which is why edges cluster in the middle of the range and the tails stay sticky. Notice what every condition has in common. Each is a reason the averaging mechanism failed, not a reason your opinion is right. Your edge, if it exists, lives in the gap the mechanism left open, not in your conviction. The old article ["Volatility Regimes and Strategy Survival"](https://aligrithm.com/volatility-regimes-and-strategy-survival/) makes the parallel point that conditions, not confidence, decide whether a strategy works. ## Calibration and sharpness: two different virtues A forecast has two qualities, and traders confuse them constantly. Calibration: among all the times you said 70%, the event happened about 70% of the time. Sharpness: your forecasts commit, clustering near 0 and 1 rather than hedging at 0.5. You want both. Calibration without sharpness is useless: a forecaster who says 0.5 to everything is perfectly calibrated and tells you nothing. Sharpness without calibration is dangerous: a forecaster who confidently says 0.9 to things that happen 60% of the time will get sized into ruin by any Kelly rule. The old article ["Why Trading Is a Probability Business, Not a Certainty Business"](https://aligrithm.com/why-trading-is-a-probability-business-not-a-certainty-business/) is the foundation here; prediction markets make the payoff binary, so miscalibration shows up faster and costs more. ## The Brier score and its decomposition The Brier score measures forecast quality as the squared error between your probability and the outcome, and it splits into three parts that map onto the two virtues plus the luck you cannot control. $$ \\text{BS} = \\underbrace{\\text{Reliability}}\_{\\text{calibration error}} \\;-\\; \\underbrace{\\text{Resolution}}\_{\\text{discriminative ability}} \\;+\\; \\underbrace{\\text{Uncertainty}}\_{\\text{base-rate entropy}} $$ Read that as: your Brier score is your calibration error (reliability, lower is better), minus how decisively you separate winners from losers (resolution, higher is better, so it subtracts), plus the event's inherent randomness (uncertainty, which you cannot change). Two of the three terms are yours to improve. The third is the noise floor. A trader who improves resolution without wrecking reliability is building real forecast skill; a trader who just gets louder raises resolution and reliability error together and nets nothing. ## The Brier Skill Score gate None of that answers the only question that gets you to trade: are you more accurate than the market? The Brier Skill Score compares your Brier score against the market's directly. $$ \\text{BSS} = 1 - \\frac{\\text{BS}\_{\\text{model}}}{\\text{BS}\_{\\text{market}}}, \\qquad \\text{BSS} > 0 \\iff \\text{you beat the market} $$ Read that as: the skill score is one minus the ratio of your Brier score to the market's, so it is positive only when your score is lower, meaning your forecasts are more accurate than the capital-weighted consensus. A positive score is a necessary condition for profitable informational trading, not a sufficient one, because execution and sizing can still lose money on a real edge. But a non-positive score is a full stop. If you cannot beat the market on historical resolutions, you have no informational edge, and every trade you place on your "view" is the belief-without-calibration anti-pattern from the article on the six ways to lose. Compute the score on your own logged predictions before you scale. The old article "Why Forecast Accuracy Is Not Enough in Market Making" makes the companion point that raw accuracy without the right loss function misleads; the skill score against the market is the loss function that matters here. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_355-visual-0.png) ## KEY POINTS - A prediction-market price is not a frequentist probability. It is the capital-weighted consensus, the point where the marginal trader is indifferent, and rich wrong traders move it more than poor right ones. - Capital flows from inaccurate to accurate participants over time, a Darwinian filter on money, not opinions. The aggregate is empirically strong, beating polls about 74% of the time across five presidential cycles. - Coherent prices can still be wrong under five conditions: capital concentration, thin liquidity, short horizons, strategic manipulation, and extreme probabilities. Each is a broken averaging mechanism, not proof your opinion is right. - Calibration means your 70% happens 70% of the time; sharpness means you commit near 0 and 1\. You need both. Calibrated-but-flat is useless; sharp-but-miscalibrated gets you sized into ruin. - The Brier score decomposes into reliability (yours to fix), resolution (yours to fix), and uncertainty (the noise floor). - The Brier Skill Score against the market is the gate. Positive means you beat the market and may trade on a view; non-positive is a full stop. Compute it on your logged predictions before scaling. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Interpreting the Predictions of Prediction Markets - Manski (2006)](https://scholar.google.com/scholar?q=%22Interpreting+the+Predictions+of+Prediction+Markets%22+Manski+2006&ref=aligrithm.com) - [If You're So Smart, Why Aren't You Rich? Belief Selection in Complete and Incomplete Markets - Blume and Easley (2006)](https://scholar.google.com/scholar?q=%22If+You%27re+So+Smart%2C+Why+Aren%27t+You+Rich%3F+Belief+Selection+in+Complete+and+Incomplete+Markets%22+Blume+and+Easley+2006&ref=aligrithm.com) - [Prediction Market Accuracy in the Long Run - Berg et al. (2008)](https://scholar.google.com/scholar?q=%22Prediction+Market+Accuracy+in+the+Long+Run%22+Berg+et+al.+2008&ref=aligrithm.com) - [Verification of Forecasts Expressed in Terms of Probability - Brier (1950)](https://scholar.google.com/scholar?q=%22Verification+of+Forecasts+Expressed+in+Terms+of+Probability%22+Brier+1950&ref=aligrithm.com) - [Prediction Markets - Wolfers and Zitzewitz (2004)](https://scholar.google.com/scholar?q=%22Prediction+Markets%22+Wolfers+and+Zitzewitz+2004&ref=aligrithm.com) ### 9.12 Why Your Arbitrage Solver Crashes at 99 Cents: Barrier Frank-Wolfe URL: https://aligrithm.com/why-your-arbitrage-solver-crashes-at-99-cents-barrier-frank-wolfe/ Last updated: 2026-07-27T13:26:10.000Z The projection converges in theory. The article ["Frank-Wolfe: Solving a Quintillion-Vertex Problem in 100 Steps"](https://aligrithm.com/frank-wolfe-solving-a-quintillion-vertex-problem-in-100-steps/) ended on a caveat that sounds academic and is not: the convergence guarantee needs a finite Lipschitz constant, and on the log-scoring market maker Polymarket runs, that constant is infinite. Point the clean algorithm at a real market and it works fine until a price drifts toward 99 cents or 1 cent, then it overflows or oscillates and never returns a trade. Prediction markets live at those extremes, near-certain events and near-impossible ones, which is exactly where the naive solver dies. The fix is small in concept and mandatory in practice. ## The gradient explodes at the boundary The market maker's cost function, for a log-scoring rule, has a conjugate equal to negative entropy. Its gradient and its curvature are what the solver leans on, and both misbehave as any price approaches zero. $$ \\nabla C^\*(p) = \\big(\\ln \\mu\_1 + 1,\\; \\ldots,\\; \\ln \\mu\_n + 1\\big), \\qquad \\nabla^2 C^\*(p) = \\text{diag}\\!\\left(\\tfrac{1}{\\mu\_1},\\; \\ldots,\\; \\tfrac{1}{\\mu\_n}\\right) $$ The gradient carries a natural log of each price. As a price mu approaches zero, the log runs to negative infinity. The curvature carries one over each price, which runs to positive infinity. And the arbitrage-free shape has corners where some coordinates are exactly zero, because those are outcomes where a security does not pay. March the iterates toward such a corner and the gradient and curvature both blow up. The Lipschitz constant, the ceiling on how fast the gradient can change, is the largest curvature anywhere in the shape, and near those corners it is unbounded. $$ L = \\sup\_{p \\in M} \\|\\nabla^2 C^\*(p)\\| = +\\infty $$ Once L is infinite, the convergence bound from the previous article is void. It divided by nothing useful. There is no promise the iterates settle, and in practice they do not: a price near zero behaves like dividing by zero, and the solver either throws a floating-point overflow or bounces between wild step sizes without homing in. This is not a rare pathology. It fires every time a security resolves toward 0 or 1, which on a prediction market is a daily event, not a corner case. ## Do not let prices touch the wall The fix reads like risk management, because it is. Do not let prices get arbitrarily close to 0 or 1\. Shrink the feasible shape slightly inward, toward a point known to sit safely in the interior, so the smallest price the solver can reach is bounded away from zero. $$ M\_\\varepsilon = (1 - \\varepsilon)\\, M + \\varepsilon\\, u, \\qquad u \\in \\text{int}(M) $$ The contracted shape M-epsilon is the original shape pulled a fraction epsilon of the way toward an interior point u. Every point in it is a blend, mostly the original shape, a sliver of the safe interior point. That sliver is enough to keep every coordinate off zero. The numbers make it concrete. Binary market, corners at (1,0) and (0,1), interior point u at (0.5, 0.5), contraction epsilon of 0.1\. The corner (1,0) gets pulled to 0.9 times (1,0) plus 0.1 times (0.5,0.5), which is (0.95, 0.05). The corner (0,1) moves to (0.05, 0.95). The contracted shape is the segment between those, and its smallest coordinate is 0.05, never zero. The gradient there is the log of 0.05 plus 1, about 2.0 in magnitude, a normal finite number instead of negative infinity. The curvature ceiling becomes finite too. $$ L\_\\varepsilon \\le \\frac{1}{\\varepsilon \\cdot \\min\_j u\_j} = O(1/\\varepsilon) $$ The bound on curvature is now one over epsilon times the smallest interior coordinate. Finite for any positive epsilon, so the convergence guarantee from the previous article applies again on the contracted shape. The interior point u is not something you invent; the initialization routine from the article ["Frank-Wolfe: Solving a Quintillion-Vertex Problem in 100 Steps"](https://aligrithm.com/frank-wolfe-solving-a-quintillion-vertex-problem-in-100-steps/) already produced it as the average of the starting corners, with every unsettled coordinate strictly between 0 and 1\. This is why that routine bothered to compute an interior point instead of just a starting vertex. ## The tradeoff, and the schedule that resolves it Contraction is not free. Shrink too hard and you converge fast to the wrong answer; shrink too little and you converge slowly to the right one. A large epsilon, say 0.1, gives a small curvature ceiling and fast convergence, but the contracted shape is a poor stand-in for the real one, so the trade you compute is off. A small epsilon, say 0.001, keeps the contracted shape almost identical to the real one, so the target is accurate, but the curvature ceiling is large and progress crawls. Pick one fixed epsilon and you are stuck choosing between fast-and-wrong or slow-and-right. The resolution is to move epsilon over time: start large for speed, shrink toward zero as the algorithm homes in, so early iterations move fast on a rough target and late iterations refine on an accurate one. $$ \\varepsilon\_t = \\min\\!\\left\\{ \\frac{g(p\_t)}{-4\\,g\_u},\\; \\frac{\\varepsilon\_{t-1}}{2} \\right\\} \\text{ if that ratio } < \\varepsilon\_{t-1}, \\text{ else hold } \\varepsilon\_{t-1} $$ The rule ties the shrink to progress. The g is the Frank-Wolfe gap from the previous article, the profit-still-on-the-table number. The g-u is the same gap measured toward the interior point, always negative. Their ratio measures how well the algorithm is converging relative to that interior direction. When the ratio is small, meaning good progress, epsilon halves. When progress stalls, epsilon holds. The schedule is automatic; nobody hand-tunes it per market. | Phase | Iterations | Epsilon | What happens | | ------ | ---------- | ------------ | ---------------------------------- | | Early | 1 to 10 | around 0.1 | fast progress toward a rough trade | | Middle | 10 to 50 | 0.01 to 0.05 | refinement | | Late | 50 to 150 | around 0.001 | high-precision convergence | As epsilon goes to zero the contracted shape converges to the real one and the iterates converge to the true projection. The overall rate is 1 over the square root of t, slower than the 1 over t of the standard method. Slower and it converges beats faster and it crashes. The standard method on this class of problem does not converge at all. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_354-visual-0.png) ## The full pipeline, and the honest cost Assemble the three computational articles and the structural arbitrage engine is complete. Initialization runs in under a second and hands over the starting corners and the interior point. Barrier Frank-Wolfe then runs 50 to 150 iterations; each iteration is a millisecond convex subproblem, a 1-to-30-second integer-program oracle call, and a negligible epsilon update and stop check. Total wall-clock, minutes to about half an hour. Then the profit check: if the guaranteed profit, the divergence minus the gap, clears 5 cents, execute the trade the projection produced; otherwise there is no arbitrage worth the execution risk. Read the timing again and the strategic picture sharpens. Minutes to thirty minutes to compute a trade, on a platform where prices move within a single Polygon block, roughly two seconds. The barrier fix is what makes the answer correct. It does nothing to make it fast enough to beat the fastest bots, and it does nothing about whether both legs fill once you commit. The 5-cent floor exists because the theoretical profit that clears it still has to survive the slippage, fees, and partial-fill risk cataloged in the article "Execution Is Part of Expected Value," and a large fraction of trades that pass the math fail there. Barrier Frank-Wolfe closes the numerical gap in the theory. The gap between a correct trade and a captured one stays wide open, and it is where most of this pillar's later warnings live. ## KEY POINTS - On a log-scoring market maker the gradient carries a natural log of price and the curvature carries one over price, so both blow up as a price nears zero. The arbitrage-free shape has corners at exactly zero, so naive Frank-Wolfe hits an infinite Lipschitz constant and its convergence guarantee is void. - The failure is not rare: a price near zero behaves like dividing by zero, causing overflow or oscillation, and it fires every time a security resolves toward 0 or 1, which is constant on prediction markets. - The fix contracts the shape a fraction epsilon toward a safe interior point, so the smallest reachable price is bounded away from zero. With epsilon 0.1 a binary market's corners move to (0.95, 0.05), the gradient becomes about 2.0 instead of negative infinity, and the curvature ceiling becomes finite. - The interior point comes free from the initialization routine, which is why that routine computed an interior point rather than just a starting vertex. - A fixed epsilon forces a choice between fast-and-wrong and slow-and-right, so epsilon shrinks on a schedule tied to convergence progress: around 0.1 early, 0.001 late. The rate is 1 over root t, slower than standard Frank-Wolfe but it converges, which the standard method does not. - The full pipeline (init under a second, 50 to 150 barrier iterations, minutes to 30 minutes, then a 5-cent profit floor) produces a correct trade. It does not make the trade fast enough to beat the quickest bots or guarantee both legs fill; those leaks live in the execution articles. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Barrier Frank-Wolfe for Marginal Inference - Krishnan et al. (2015)](https://scholar.google.com/scholar?q=%22Barrier+Frank-Wolfe+for+Marginal+Inference%22+Krishnan+et+al.+2015&ref=aligrithm.com) - [Arbitrage-Free Combinatorial Market Making via Integer Programming - Kroer et al. (2016)](https://arxiv.org/abs/1606.02825?ref=aligrithm.com) - [Logarithmic Market Scoring Rules for Modular Combinatorial Information Aggregation - Hanson (2007)](https://scholar.google.com/scholar?q=%22Logarithmic+Market+Scoring+Rules+for+Modular+Combinatorial+Information+Aggregation%22+Hanson+2007&ref=aligrithm.com) ### 9.11 Frank-Wolfe: Solving a Quintillion-Vertex Problem in 100 Steps URL: https://aligrithm.com/frank-wolfe-solving-a-quintillion-vertex-problem-in-100-steps/ Last updated: 2026-07-27T13:26:10.000Z The projection from the article "Arbitrage Is Just Projection" is the whole game: find the closest fair price to the current unfair one, and you have the trade, the profit, and the target. The problem is that the arbitrage-free shape has up to 9.2 quintillion corners for an NCAA-scale market, and every standard projection method wants to see all of them. Frank-Wolfe does the projection by looking at about a hundred corners out of the quintillion, adding one per step, and printing a number at each step that tells you how much profit you might still be leaving behind. If you have ever leaned on lazy evaluation to avoid computing something you did not need, this is the same instinct applied to arbitrage. ## Why the direct approaches are dead on arrival Projected gradient descent and interior-point methods, the usual tools, both need something you cannot afford. They either enumerate the corners of the shape or repeatedly check feasibility against the full constraint set at once. For a shape with quintillions of corners, neither finishes this century. The article "The Marginal Polytope" is where that wall first appears, and it is a hard wall, not a slow one. Frank-Wolfe sidesteps it by never asking for the whole shape. It replaces the projection with a sequence of much simpler questions, each answered by one call to the oracle from the article "Trillions of Outcomes, 200 Constraints": hand it a direction, get back the single cheapest valid outcome. Each answer is one corner. The method builds its solution out of the few corners it actually needed, and the count of corners it needs to reach an accurate answer is small, on the order of a hundred, regardless of whether the shape has a thousand corners or a quintillion. ## The loop, in the order a trader would run it The algorithm is short. Strip the notation and it is five steps repeated. ``` start with a few known outcomes (the initial working set) repeat: subproblem: best trade using only the outcomes found so far oracle: ask for the worst outcome you have NOT yet considered update: add that outcome to the working set gap: compute how much profit you might still be missing stop: if the gap is small enough, trade; else loop ``` Read each step for what it means, not what it computes. The subproblem is "given the handful of outcomes I have discovered, what is my best trade." That is a small convex problem over a few hundred corners, quick. The oracle call asks the integer-program solver the sharp question: among all valid outcomes, which one does my current trade handle worst. That worst offender is the corner worth adding, because it is the one most likely to break the trade if the world lands there. Add it, and the working set grows by one. The gap is the part that makes this usable rather than merely convergent. $$ g(p\_t) = \\nabla F(p\_t) \\cdot (p\_t - z\_t), \\qquad D(p\_t \\,\\|\\, \\theta) - D(p^\* \\,\\|\\, \\theta) \\le g(p\_t) $$ The gap g at step t is the current iterate scored against the worst outcome the oracle just returned. The inequality is the reason it matters: the true distance to the real optimum is at most the gap. So the gap is a certified upper bound on how far you are from the best possible trade, computed for free at every step. It is a profit thermometer. When it is small relative to the profit you have already secured, you are done. The stopping condition is one line of code. ``` if gap <= (1 - alpha) * divergence: break ``` With alpha set to 0.9, you break the moment your guaranteed profit reaches 90 percent of the theoretical maximum. The last 10 percent is not worth the iterations, because the market moves while you compute, and a trade captured now beats a marginally better trade captured too late. This connects straight to the profit guarantee from the article "Arbitrage Is Just Projection": the guaranteed profit is the divergence minus the gap, and convex duality keeps that non-negative, so stopping early is safe by construction. ## The lazy-evaluation analogy, made concrete Brute force enumerates all 9.2 quintillion brackets, computes the optimal trade over every one, then executes. It never finishes. Frank-Wolfe says: I will only look at the brackets that threaten my current trade. Each iteration, it asks the solver for the single bracket that most endangers where it stands. If no threatening bracket exists, the current trade is optimal or close enough, and it stops. After a hundred iterations you have examined a hundred brackets out of 9.2 quintillion and you hold a near-optimal trade with a mathematical profit guarantee attached. A programmer sees this immediately. You do not evaluate the whole list. You pull elements one at a time, driven by need, and you stop when the next element cannot change your answer. The oracle is the generator; the gap is the "have I seen enough" check. The quintillion outcomes that never threatened your trade never get touched, and touching them would have added nothing. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_353-visual-0.png) ## How fast, and how many corners The convergence rate is honest about being slow, and it does not matter as much as it sounds. $$ F(p\_t) - F(p^\*) \\;\\le\\; \\frac{2L \\cdot \\text{diam}(M)^2}{t + 2} $$ The error shrinks like 1 over t, where t is the iteration count. After 100 iterations you are roughly 100 times more accurate than after one. That is slow next to gradient descent, but each iteration costs a single solver call, and you need on the order of tens to a hundred of them. The active set after t iterations holds at most its starting size plus t corners, so even 150 iterations on an NCAA-scale market leaves you working with around 150 corners, not a quintillion. Slow-per-step convergence over a tiny working set beats fast convergence you can never start because you cannot load the shape. One term in that bound is a landmine, and it is labeled L, the Lipschitz constant, a measure of how fast the gradient can change. The whole rate is multiplied by it, and the guarantee only holds if L is finite. Note that word: finite. ## The starting point, and the caveat that breaks everything Frank-Wolfe needs a feasible place to begin. The initialization routine builds one by asking, for each unsettled security, two feasibility questions: is there a valid outcome where this security pays, and is there one where it does not. Three cases follow. Both feasible means the security is genuinely undecided, so both possibilities go into the starting set. Only one feasible means the security is already logically settled, so its price gets locked. The routine produces the initial working set, an interior point sitting strictly inside the shape (the average of the starting corners), and a record of anything it settled along the way. It runs in under a second even for a full bracket, and the interior point it produces is not a throwaway. The next article needs it. The caveat. The convergence guarantee assumed L is finite, and on a log-scoring market maker, the kind Polymarket runs, L is infinite. The gradient of the market maker's cost function blows up as any price approaches zero, and the shape has corners where prices are exactly zero, because those are outcomes where a security does not pay. Approach one of those corners and the gradient runs to negative infinity, the rate bound becomes meaningless, and the algorithm either crashes or oscillates instead of converging. This is not an edge case you can ignore. It happens every time a security resolves, which is constantly. The clean loop in this article, run naively on a real prediction market, fails. The fix is the entire subject of the article "Why Your Arbitrage Solver Crashes at 99 Cents," and it is a necessity, not a refinement. ## KEY POINTS - Frank-Wolfe does the projection by adding one corner of the arbitrage-free shape per step, needing about a hundred corners out of up to 9.2 quintillion, because it never asks to see the whole shape at once. - The loop is five steps: best trade over the corners found so far, ask the oracle for the worst unconsidered outcome, add it, compute the gap, stop if the gap is small. Each oracle call is one integer-program solve. - The gap is a certified upper bound on how far you are from the optimal trade, printed free at every step. It is a profit thermometer, and the stop condition "gap at most (1 minus alpha) times divergence" with alpha 0.9 breaks at 90 percent of max profit. - The analogy is lazy evaluation: pull outcomes one at a time driven by which one most threatens the current trade, and stop when the next one cannot change the answer. The untouched quintillion never mattered. - Convergence is 1 over t, slow per step but cheap per step, and the working set stays around 150 corners even for a full bracket. Slow convergence over a tiny set beats fast convergence you cannot start. - The rate multiplies by the Lipschitz constant L and only holds when L is finite. On a log-scoring market maker L is infinite because the gradient explodes near zero prices, so naive Frank-Wolfe crashes on real markets and needs the barrier fix. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Revisiting Frank-Wolfe: Projection-Free Sparse Convex Optimization - Jaggi (2013)](https://scholar.google.com/scholar?q=%22Revisiting+Frank-Wolfe%3A+Projection-Free+Sparse+Convex+Optimization%22+Jaggi+2013&ref=aligrithm.com) - [Arbitrage-Free Combinatorial Market Making via Integer Programming - Kroer et al. (2016)](https://arxiv.org/abs/1606.02825?ref=aligrithm.com) ### 9.10 Trillions of Outcomes, 200 Constraints: Reading Tournament Rules as Assert Statements URL: https://aligrithm.com/trillions-of-outcomes-200-constraints-reading-tournament-rules-as-assert-statements/ Last updated: 2026-07-27T13:26:09.000Z An NCAA bracket has about 9.2 quintillion possible outcomes. You cannot store that list, cannot loop over it, cannot project onto a shape whose corners you are unable to write down. The article "The Marginal Polytope" left you with exactly this wall: the right geometric object, and no way to touch it. The escape is to stop listing outcomes and start describing them with rules. A few hundred linear inequalities can pin down which of the quintillion combinations are legal, and a solver can then hand you valid outcomes on demand without ever enumerating the set. If you have written a function full of assert statements, you already understand the trick. ## Describe the set, do not list it The valid payoff set is a collection of ones-and-zeros vectors. Instead of storing them, write down the conditions every valid vector must satisfy. $$ Z = \\{\\, z \\in \\{0,1\\}^{|\\mathcal{I}|} : A^\\top z \\ge b \\,\\} $$ Read it as a membership rule, not a formula to evaluate. The z is a candidate outcome, a vector of ones and zeros over the securities. The matrix A and vector b encode the logical dependencies; the inequality is a stack of linear rules. Any z that satisfies all of them is a valid outcome, and any z that violates one is impossible. The whole exponential set is now the solution set of a small system of inequalities. A solver like Gurobi searches that system directly and never builds the list. The compression this buys is not marginal. | Market | Number of outcomes | Constraints | Compression | | ------------------------------- | --------------------- | ----------------------------- | ---------------- | | Duke vs Cornell (bracket slice) | 16,384 | 3 | about 5,461 to 1 | | NCAA tournament | about 9.2 quintillion | around 200 | astronomical | | 2024 US election | 17,218 conditions | around 1,500 dependency pairs | very large | Three rules stand in for 16,384 combinations. Around 200 rules stand in for 9.2 quintillion. The market looks exponential and the constraint structure collapses it to something polynomial, and that collapse is what makes real-time arbitrage detection possible at all. ## The worked case: two teams, three lines of logic Take a bracket slice, Duke against Cornell. Each team has seven securities, one for each possible number of wins from 0 through 6\. Fourteen binary variables, so a naive space of 2 to the 14, which is 16,384 combinations. Three constraints carve out the legal ones. $$ \\sum\_{k=0}^{6} z\_{\\text{Duke},k} = 1, \\qquad \\sum\_{k=0}^{6} z\_{\\text{Cornell},k} = 1, \\qquad z\_{\\text{Duke},5} + z\_{\\text{Duke},6} + z\_{\\text{Cornell},5} + z\_{\\text{Cornell},6} \\le 1 $$ The first says Duke finishes with exactly one win total, some single number between 0 and 6\. The second says the same for Cornell. The third says both teams cannot reach five or more wins, because to do that they would have to meet in the semifinals and one must lose. Read those three the way you would read code. ``` assert sum(duke_wins[0:7]) == 1 # Duke lands on exactly one win total assert sum(cornell_wins[0:7]) == 1 # Cornell lands on exactly one win total assert duke_wins[5] + duke_wins[6] + cornell_wins[5] + cornell_wins[6] <= 1 # at most one of them reaches the semifinals ``` Any assignment of the fourteen booleans that passes all three assertions is a legal tournament outcome. Any assignment that trips one is a fantasy the market cannot pay out on. The solver's job is to search the passing assignments efficiently, and it does that without trying all 16,384\. You wrote three lines of logic and eliminated 16,381 combinations you never had to look at. ## A vocabulary of dependencies Most real market relationships come from a short list of constraint shapes, and learning the four covers the overwhelming majority of cases. | Dependency | Constraint | Example | | -------------------- | -------------------------------------- | ---------------------------------------------------- | | Subset / implication | z\_A less than or equal to z\_B | "margin greater than 5" implies "candidate wins" | | Mutual exclusion | z\_A plus z\_B less than or equal to 1 | two teams cannot both win the final | | Partition | sum of z\_k equals 1 | exactly one outcome per group | | Conditional | z\_A less than or equal to z\_B | "wins championship" implies "reaches the final four" | Subset and conditional share the same algebra, an implication written as an inequality: A can only be 1 if B is also 1\. Mutual exclusion caps a pair at one. Partition forces exactly one member of a group to fire. Encode a market's rulebook with these four and you have described a polytope with quintillions of corners in a page of inequalities. The article "Outcome Geometry" showed why the valid set collapses below the naive cube; this is the collapse written in a form a solver can read. ## The one thing the solver actually needs The projection machinery from the article "Arbitrage Is Just Projection" does not need the full set Z. It needs one function, called repeatedly. Give it a cost vector, and it returns the single cheapest valid outcome under that cost. $$ \\text{oracle}(c) = \\arg\\min\_{z \\,:\\, A^\\top z \\ge b}\\; c \\cdot z $$ The c is a direction handed in by the projection algorithm. The oracle returns the valid outcome z that scores lowest against it. That is a single integer linear program, one solve, and a modern solver returns one optimal corner without walking the rest. This is the entire interface between the geometry and the computation: a black box you query with a direction and that answers with a corner. The article ["Frank-Wolfe: Solving a Quintillion-Vertex Problem in 100 Steps"](https://aligrithm.com/frank-wolfe-solving-a-quintillion-vertex-problem-in-100-steps/) is built on nothing but repeated calls to this oracle, roughly a hundred of them, to find a near-optimal trade over a set with quintillions of members. Solve times are not uniform, and the pattern is worth knowing. Early in a tournament, with few games settled, a call runs on the order of a second. Mid-tournament, with 30 to 40 games decided, seconds to tens of seconds. Late, with 50-plus settled, a few seconds or less. The acceleration is not luck: every settled game locks a variable, which shrinks the free space the solver searches. The problem gets easier as the world resolves, which is convenient, because that is also when prices move fastest. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_352-visual-0.png) ## Where this helps and where it does not The compression is genuine and it is the reason any of this runs in production. It also does nothing for the two problems that come next. It does not make the projection converge on a log-scoring market. The oracle answers questions; it does not run the search. The search is Frank-Wolfe, and on a log market maker the standard version fails to converge because the gradient blows up at the boundary, which is why the article "Why Your Arbitrage Solver Crashes at 99 Cents" exists. Encoding the set correctly and still watching the solver diverge is a real and common failure. And it does nothing for execution. A perfectly encoded, perfectly solved trade is a set of target orders, not filled positions. The 5,461-to-1 compression buys you the ability to find the trade in time. Whether the second leg is still there when your first fills, whether the fees leave anything behind, whether two faster bots already took it, all live in the article "Execution Is Part of Expected Value," and that is where most of the theoretical profit from a correctly encoded market quietly disappears. ## KEY POINTS - Do not list outcomes, describe them. The valid set is the solution set of a small system of linear inequalities, A-transpose z greater than or equal to b, and a solver finds valid outcomes on demand without enumerating the exponential list. - Three constraints replace 16,384 combinations in the Duke-Cornell slice; around 200 replace 9.2 quintillion for a full NCAA bracket. The market looks exponential and the constraint structure collapses it to polynomial size. - Read the constraints as assert statements: each valid outcome is a boolean assignment that passes every assertion, and the solver searches only the passing assignments. - Four dependency shapes cover most markets: implication (z\_A at most z\_B), mutual exclusion (z\_A plus z\_B at most 1), partition (a group sums to 1), and conditional (same algebra as implication). - The projection needs one thing from all this: a linear-minimization oracle that takes a direction and returns the single cheapest valid outcome, one integer-program solve per call. Frank-Wolfe is roughly a hundred such calls. - Encoding solves the size problem, not the numerical one or the execution one. The solver still diverges on log-market boundaries without the barrier fix, and a solved trade is target orders, not filled positions. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Arbitrage-Free Combinatorial Market Making via Integer Programming - Kroer et al. (2016)](https://arxiv.org/abs/1606.02825?ref=aligrithm.com) - [Unravelling the Probabilistic Forest: Arbitrage in Prediction Markets - Saguillo et al. (2025)](https://arxiv.org/abs/2508.03474?ref=aligrithm.com) ### 9.1 Convex Geometry for Traders: Polytopes, Hyperplanes, Projection, and Why KL Beats Euclidean URL: https://aligrithm.com/convex-geometry-for-traders-polytopes-hyperplanes-projection-and-why-kl-beats-euclidean/ Last updated: 2026-07-27T13:26:05.000Z Most traders reach for a forecast the moment they open a prediction market. They read the news, form a view, and buy the side they like. That instinct is the one this whole pillar argues against, and the argument is geometric before it is financial. The cleanest money on Polymarket and Kalshi comes from prices that contradict each other, not from prices that turn out wrong. To see those contradictions you need a small vocabulary of convex geometry: convex sets, hulls, polytopes, hyperplanes, gradients, conjugates, and two ways of measuring distance. Learn the vocabulary here, on toy examples, and the heavy machinery of the later articles stops looking like a wall. This is a gateway article. It teaches the words. Nothing here makes you money on its own, but every profitable idea in the pillar is written in this language, and traders who skip it end up trusting a solver they cannot read. ## A price vector is a point in space Fix a set of contracts. A US-election market might have "Candidate A wins" and "Candidate A loses" as two YES contracts, each paying $1 if its event resolves true. The prices, say $0.40 and $0.40, form a pair of numbers. Write that pair as a single point in a plane: the horizontal axis is the price of the first contract, the vertical axis the second. A market with three contracts is a point in three-dimensional space, and a market with a thousand contracts is a point in thousand-dimensional space you cannot draw but can still reason about. That reframing is the whole trick. Once prices are points, "is this market internally consistent?" becomes "does this point sit inside a particular region?" The region has a name and a shape, and the shape is convex. ## Convex sets: the shape with no dents A set is convex when a straight line between any two points inside it stays inside it. A filled disk is convex. A crescent moon is not, because a chord between the two horns leaves the shape. A solid cube is convex; a cube with a bite taken out is not. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_343-visual-0.png) Convexity matters because the set of all sensible prices is convex, and because convex problems are the ones optimizers can solve without getting stuck in false optima. Every guarantee later in the pillar leans on it. ## Convex combinations and the hull A convex combination is a weighted average where the weights are non-negative and add to one. Two-thirds of point A plus one-third of point B lands on the segment between them, closer to A. Stretch this to many points and any weighted average with non-negative weights summing to one lands somewhere in the solid region they span. $$ x = \\lambda\_1 x\_1 + \\lambda\_2 x\_2 + \\cdots + \\lambda\_n x\_n, \\qquad \\lambda\_i \\ge 0, \\quad \\sum\_i \\lambda\_i = 1 $$ Read that as: pick some anchor points, assign each a non-negative weight, force the weights to sum to one, and the result is a legal blended point. The lambdas behave exactly like probabilities, because they are non-negative and sum to one. That is not a coincidence, and it is the bridge to markets: a fair price vector is a probability-weighted average of what the contracts pay in each possible outcome. The convex hull is the smallest convex set containing a group of points. Hammer nails into a board at each point, stretch a rubber band around the outside, let it snap tight, and the shape it encloses is the hull. Every point inside is some convex combination of the corner nails. ## Polytopes: hulls with flat sides and corners A polytope is a convex hull of finitely many points. In two dimensions it is a polygon, in three a polyhedron, in higher dimensions a polytope you name but do not sketch. It has flat faces and a finite count of sharp corners called vertices. The object that runs this pillar is the marginal polytope. Its corners are the payoff vectors of the market, one corner per distinct way the world can resolve. For a single binary market the payoff vectors are (1, 0) if YES pays and (0, 1) if NO pays, so the marginal polytope is the line segment joining those two points. Every point on that segment has coordinates that sum to one. $$ \\mathcal{M} = \\operatorname{conv}(\\mathcal{Z}) = \\Bigl\\{ \\sum\_{\\omega} \\lambda\_\\omega \\, \\phi(\\omega) \\;:\\; \\lambda\_\\omega \\ge 0,\\; \\sum\_\\omega \\lambda\_\\omega = 1 \\Bigr\\} $$ Read that as: script-Z is the set of payoff vectors, one per outcome, and script-M is their convex hull. A price vector inside script-M corresponds to some honest probability distribution over outcomes. A price vector outside script-M corresponds to no probability distribution at all, which means the prices contradict themselves, which means guaranteed profit sits on the table. The entire structural half of the pillar is one membership question: is the current price point inside this polytope? ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_343-visual-1.png) ## Hyperplanes: the boundary that is the trade A hyperplane is a flat cut that splits space into two halves. In two dimensions it is a line, in three a plane, in higher dimensions a flat sheet of one dimension less than the space. Define it by a direction vector and a threshold. $$ \\mathbf{a} \\cdot \\mathbf{x} = c $$ Read that as: pick a direction a, and every point x whose dot product with a equals the constant c lies on the flat cut; points with a larger dot product sit on one side, points with a smaller dot product on the other. The separating hyperplane theorem says something a trader should tape to the wall: if a point sits outside a convex set, you can always slide a flat cut between the point and the set. That cut is not an abstraction. Its direction vector is the arbitrage portfolio. Return to prices (0.40, 0.40) outside the segment. The separating direction is (1, 1): buy one YES and one NO. Cost is 0.40 plus 0.40, which is $0.80\. Whatever happens, exactly one contract pays $1, so the worst-case payout is $1.00\. You paid $0.80 for a guaranteed $1.00\. The 20-cent gap is the separating hyperplane made cash. The direction told you what to buy; the gap told you how much you win. ## Gradients and Hessians: slope and curvature A gradient is the vector of slopes of a function, one slope per input direction. It points uphill, toward steepest increase. In a market, the gradient of the market maker's cost function gives the current prices, so "read the prices" and "read the gradient" are the same act. A Hessian is the matrix of second derivatives. It measures how fast the slope itself changes, which is curvature. Flat curvature means prices barely move when you trade; sharp curvature means a small order swings the price hard. For the log-market-scoring-rule maker the curvature for contract i scales as one over its price. $$ \\text{Hessian entry for security } i \\;=\\; \\frac{1}{\\mu\_i} $$ Read that as: when a contract trades near a price of $0.01, the curvature is about 100, and as the price crawls toward zero the curvature blows up toward infinity. That single fact is why arbitrage solvers detonate near 99-cent and 1-cent contracts, and why the later article on Barrier Frank-Wolfe exists. A price near zero behaves like a divide-by-zero waiting to happen. The Lipschitz constant, a bound on how fast the gradient can change, goes infinite exactly where real markets spend most of their time: at the edges, where events are nearly settled. ## Conjugates and duality: two coordinate systems for one market A market has two descriptions. One counts how many shares of each contract have been bought; call it the state, an unbounded vector where a trade just adds shares to a coordinate. The other reads the implied probabilities on the screen; call it the price, bounded between zero and one and summing to one. The convex conjugate, also called the Legendre-Fenchel transform, converts between them. $$ \\text{state} \\to \\text{price:}\\quad p = \\nabla C(\\theta), \\qquad\\qquad \\text{price} \\to \\text{state:}\\quad \\theta = \\nabla R^\*(p) $$ Read that as: apply the gradient of the cost function C to a state and you get prices; apply the gradient of its conjugate R-star to prices and you get back the state. These two moves are inverses. You need both because arbitrage math produces target prices, and an exchange only accepts orders in shares. The conjugate is the translator from "the market should be priced here" to "buy this many of these contracts." Duality is the deeper payoff. Strong duality says certain min-max problems can be flipped without changing the answer. The trader's problem, "choose the portfolio that maximizes my worst-case profit," is a brutal combinatorial search. Flip it with duality and it becomes "find the closest fair price to the current unfair price," a geometric projection. Same answer, tractable form. That flip is the engine of the keystone result in this pillar. ## Entropy, KL, and Bregman: measuring distance the right way Distance seems obvious until it costs you money. Euclidean distance treats a move from $0.50 to $0.60 as identical to a move from $0.01 to $0.11: both are ten cents. In probability terms they are nothing alike. The log-odds, the natural scale for beliefs, barely twitches from 0.50 to 0.60 but lurches from 0.01 to 0.11. $$ \\operatorname{logit}(p) = \\ln\\frac{p}{1-p}, \\qquad \\operatorname{logit}(0.50)=0,\\quad \\operatorname{logit}(0.60)\\approx 0.41,\\quad \\operatorname{logit}(0.01)\\approx -4.60,\\quad \\operatorname{logit}(0.11)\\approx -2.09 $$ Read that as: the logit turns a probability into a real number, zero at even odds and running off toward minus-infinity as the probability approaches zero. The 0.50-to-0.60 move shifts the logit by about 0.41\. The 0.01-to-0.11 move shifts it by about 2.5\. The second is roughly six times the information change, yet Euclidean distance calls them equal. Size a trade on that lie and you misprice the whole opportunity. The honest ruler is entropy-based. Entropy measures uncertainty and peaks when every outcome is equally likely. Its mirror, negative entropy, is the conjugate of the log-scoring-rule cost function. The distance built from it is the Kullback-Leibler divergence. $$ \\operatorname{KL}(p \\,\\|\\, q) = \\sum\_i p\_i \\ln\\frac{p\_i}{q\_i} \\;\\ge\\; 0, \\qquad \\text{zero only when } p = q $$ Read that as: KL sums, over every contract, the target probability times the log-ratio of target to current probability; it is always non-negative and hits zero only when the two price vectors match. KL is not symmetric and is not a true distance, but it is the distance the market maker itself uses, measured in the maker's own information units. KL is one instance of a Bregman divergence, the family of curved rulers built from any strictly convex function. $$ D\_R(p \\,\\|\\, q) = R(p) - R(q) - \\nabla R(q)\\cdot(p - q) $$ Read that as: the Bregman divergence takes the value of R at p, subtracts the value at q, then subtracts the linear approximation of R at q evaluated at p; what remains is the gap between the true curve and its tangent line, a curvature-aware distance. Plug in negative entropy and it collapses to KL. Plug in one-half the squared norm and it collapses to plain Euclidean distance. Choosing the ruler is choosing the market's geometry, and for these markets the ruler is KL. Use Euclidean and your "optimal" trade is provably not optimal. ## Softmax and logit: you already know the price formula If you have trained a classifier you have used softmax, and softmax is the price function. The log-market-scoring-rule maker sets prices by exponentiating the state and normalizing. $$ p\_i(\\theta) = \\frac{\\exp(\\theta\_i / b)}{\\sum\_j \\exp(\\theta\_j / b)} $$ Read that as: raise e to each state coordinate divided by a liquidity parameter b, then divide by the sum so the outputs are positive and sum to one. The state vector plays the role of logits, the prices are the softmax output, and buying shares of contract i adds to that coordinate, which pushes its price up and drags the rest down. The liquidity parameter b sets how hard prices move: small b means a tiny order swings the price, large b means the book absorbs size. The price impact of buying works out to roughly the order size times p times one-minus-p over b, largest at even odds and vanishing near the extremes. A machine-learning reader arrives already fluent; the market maker is a softmax layer with a bank account. ## Putting the vocabulary to work String the words together and the pillar's spine appears. Prices are a point. Fair prices form a convex polytope whose corners are payoff vectors. Ask whether the point sits inside the polytope. If it sits outside, a separating hyperplane exists, and its direction is the trade. The size of the win is the Bregman distance from the point to the polytope, measured with KL because the market maker measures in KL. Computing that distance is a projection, reachable only because duality flipped an ugly max-min into a clean minimization. Converting the target prices back into share orders uses the conjugate. Gradients read prices; Hessians warn you where the solver breaks. Nothing in that chain requires a forecast. It requires reading contradiction in a set of numbers and acting before the market repairs itself. The pillar treats forecasting as the fifth-cleanest edge of nine and the most dangerous to size, a point the article on the nine sources of edge makes in full. Before any of that, you needed the geometry, and now you have it. This gateway pairs with a working notebook that builds a polytope membership checker and emits the separating-hyperplane trade on the (0.40, 0.40) toy, so the words above turn into code you can run. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/07/article_343-visual-2.png) ## KEY POINTS - Turn a market into geometry: a price vector is a single point in a space with one axis per contract, and internal consistency becomes a question of whether that point sits inside a region. - A convex set has no dents, so a straight line between any two interior points stays inside. Convex combinations are probability-weighted averages, which is why fair prices behave like probabilities. - The marginal polytope is the convex hull of the payoff vectors. Inside it means no arbitrage; outside it means guaranteed profit. The structural half of the pillar is one membership test. - The separating hyperplane theorem turns geometry into a trade: if the price point sits outside the polytope, the separating direction is the portfolio to buy, and the gap it opens is the guaranteed profit. - Gradients of the cost function are the prices; Hessians are the curvature and scale as one over price, which is why solvers explode near 1-cent and 99-cent contracts. - The convex conjugate translates between shares (state) and prices, and duality flips the trader's worst-case-profit problem into a solvable projection onto the polytope. - Measure distance with KL, not Euclidean. A ten-cent move near even odds and a ten-cent move near the extremes carry different information, and Euclidean distance ignores that, yielding suboptimal trades. - Softmax is the price formula. The log-market-scoring-rule maker is a softmax layer, so a machine-learning reader already knows how prices are set. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Convex Optimization - Boyd and Vandenberghe (2004)](https://scholar.google.com/scholar?q=%22Convex+Optimization%22+Boyd+and+Vandenberghe+2004&ref=aligrithm.com) - [The Relaxation Method of Finding the Common Point of Convex Sets and Its Application to the Solution of Problems in Convex Programming - Bregman (1967)](https://scholar.google.com/scholar?q=%22The+Relaxation+Method+of+Finding+the+Common+Point+of+Convex+Sets+and+Its+Application+to+the+Solution+of+Problems+in+Convex+Programming%22+Bregman+1967&ref=aligrithm.com) - [Arbitrage-Free Combinatorial Market Making via Integer Programming - Kroer et al. (2016)](https://arxiv.org/abs/1606.02825?ref=aligrithm.com) ### 6.46 Signal Averaging: Killing Noise with Redundant Alphas URL: https://aligrithm.com/signal-averaging-killing-noise-with-redundant-alphas/ Last updated: 2026-07-27T13:25:53.000Z Most signals that look good in research die after costs. You build a momentum estimator, the in-sample curve climbs, and then you charge it spread and slippage on every position change and the edge evaporates. The reflex is to smooth the signal or trade it less, and those help, but they treat the symptom. The disease is that your signal is paying transaction costs to trade its own noise, and there is a technique that attacks the noise directly, costs almost nothing to implement, and works for anyone running more than one variant of an alpha. ## Your signal is two things stacked Write any signal as a true component plus a noise component. $$ \\text{signal}\_i \\;=\\; \\text{true}\_i \\;+\\; \\varepsilon\_i $$ $$ \\mathbb{E}\[\\varepsilon\_i\] = 0, \\qquad \\operatorname{Var}(\\varepsilon\_i) = \\sigma^2 $$ Read that as: the estimator you actually trade, signal\_i, is the thing you want, true\_i, which is positively correlated with future returns, plus a noise term epsilon\_i that has mean zero and some variance sigma-squared. The noise averages out to nothing over time, so it carries no edge, but it has variance, and variance is what jiggles your target position back and forth. Every jiggle is a round trip you pay for. The old article ["Why Forecast Accuracy Is Not Enough in Market Making"](https://aligrithm.com/why-forecast-accuracy-is-not-enough-in-market-making/) makes the cost-side version of the point: a forecast can be right on average and still bleed you if acting on it churns the book. So most of your transaction-cost bill is not the cost of expressing the true signal. It is the cost of trading the variance of epsilon. Strip the noise and the same true signal trades far less and keeps far more of its gross edge. ## Average many copies and the noise shrinks You cannot see true\_i and epsilon\_i separately, so you cannot subtract the noise directly. You can drown it. Build many estimators of the same type, ten or twenty momentum signals instead of one, and combine them. The bet you are making is structural: the true components of same-type signals are highly correlated with each other, because they are all reading the same underlying momentum, while their noise components are close to independent, because each variant's noise comes from its own particular lookback and sampling quirks. Average them and the two components behave completely differently under the sum. $$ \\bar{s} \\;=\\; \\frac{1}{N}\\sum\_{i=1}^{N}\\text{signal}\_i \\;=\\; \\underbrace{\\frac{1}{N}\\sum\_i \\text{true}\_i}\_{\\text{stays strong}} \\;+\\; \\underbrace{\\frac{1}{N}\\sum\_i \\varepsilon\_i}\_{\\text{collapses}} $$ $$ \\operatorname{Var}\\!\\Big(\\tfrac{1}{N}\\sum\_i \\varepsilon\_i\\Big) \\;=\\; \\frac{\\sigma^2}{N}\\quad\\text{when the }\\varepsilon\_i\\text{ are independent} $$ Read the top line as: the average of the signals splits into the average of the true parts and the average of the noise parts. Because the true parts are correlated, averaging them barely dilutes the signal, the mean of a pile of similar things is about as strong as each one. The bottom line is the payoff: the variance of the averaged noise falls by a factor of N when the noise terms are independent, so twenty signals cut the noise standard deviation by about a factor of four and a half. The true signal survives the average, the noise gets divided away, and the signal-to-noise ratio of the combination beats any single member by a wide margin. Fewer noise-driven jiggles means fewer round trips, which is where the after-cost edge comes from. ## Where the variants come from The usual objection is that coming up with twenty signals is hard. It is the opposite of hard once you stop trying to find the one best version. The standard workflow tunes a momentum lookback, tests 20, 30, 40, 60, 90 days, and ships whichever posted the best in-sample Sharpe. That is selection on noise: you picked the parameter whose particular noise draw happened to flatter the backtest, which is overfitting wearing a respectable hat. Flip it. Instead of choosing the winner, keep all of them and average. Each lookback is a legitimate variant of the same momentum idea, their true components track each other, their noise does not, and the average is both more robust and cheaper to trade than the cherry-picked single. You were already computing the whole sweep to pick a winner, so the averaged version costs you nothing extra to build. If the signal has a random component, as a machine-learning model does through its seed and its sampling, you get variants for free by changing the seed. Random-seed averaging trains the same model under several seeds and averages the predictions, and the logic is identical: the learned structure is shared across seeds, the seed-specific noise is not, so the average has a cleaner forecast. It is the cheapest variance reduction in the book, and it is the same averaging that the old article ["Why Volatility-Adjusted Position Sizing Matters"](https://aligrithm.com/why-volatility-adjusted-position-sizing-matters/) leans on at the portfolio layer, only applied one level down at the signal layer. ## The honest part: the noise is never fully independent The factor-of-N gain assumes independent noise, and same-type signals never deliver that cleanly. Two momentum estimators with lookbacks of 50 and 55 days are nearly the same signal, so their noise is correlated too, and correlated noise does not divide away at the full rate. The realized gain sits between no improvement, when the variants are identical, and the full factor-of-N, when the noise is independent, and where you land depends on how genuinely different the variants are. This is the same math as portfolio diversification: the old article ["The Death of the Single-System Trader"](https://aligrithm.com/the-death-of-the-single-system-trader/) shows that correlated systems give almost no diversification benefit, and correlated signal variants give almost no noise reduction, for the identical reason. So spread the variants out. Use lookbacks that actually differ, mix the construction where you can, and do not fool yourself that ten copies of nearly the same signal bought you ten-fold noise reduction. This is distinct from stacking different alpha types. The old article ["Positional Market Making and the Thousands-of-Alphas Ensemble"](https://aligrithm.com/positional-market-making-and-the-thousands-of-alphas-ensemble/) combines many genuinely different skews into one book, which is diversification across ideas. Signal averaging is narrower and more specific: it is combining many variants of one idea to clean the noise out of that one idea before it ever reaches the portfolio. You do both, at different layers, and the averaging here is the unglamorous first pass that makes each idea worth putting in the ensemble. ## Why averaging beats picking the best The whole move is a refusal to optimize. Parameter tuning to find the best single signal maximizes in-sample performance, which is the exact quantity that fails to survive out of sample, because the maximum is sitting on top of a favorable noise draw. Averaging across the parameter set throws away the in-sample peak on purpose and keeps the robust center, which is the part that survives. The old article ["Parameter Stability Beats Best Parameter"](https://aligrithm.com/parameter-stability-beats-best-parameter/) is the general statement and this is the concrete tool: you do not search the parameter space for a winner, you integrate over it and trade the average. The cost is a sliver of in-sample sharpness. The return is a signal that trades less, survives the regime change that moves the best lookback, and keeps its edge after costs. ## Visualizing the noise wash ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_335-visual-0.png) ## KEY POINTS - Any signal is a true component plus a zero-mean noise component. The true part carries the edge; the noise part carries only variance, and trading that variance is where most of your transaction-cost bill comes from. - You cannot subtract the noise directly, so drown it. Build ten to twenty variants of the same alpha type and average them. - The true components are correlated and survive the average; the noise components are close to independent and their variance falls by a factor of N. The combined signal-to-noise ratio beats any single member, so it trades less and keeps more edge after costs. - Get the variants for free: instead of tuning one lookback and shipping the best, keep the whole sweep and average it. For ML, average across random seeds. - The noise is never fully independent. Near-identical variants have correlated noise and give little reduction, the same way correlated systems give little diversification. Spread the variants out and do not overclaim the gain. - This is signal-layer cleaning, narrower than combining different alpha types into an ensemble. Do both, at different layers. - Averaging beats picking the best because the in-sample winner is sitting on a lucky noise draw. Integrate over the parameter set instead of searching it for a peak. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Systematic Trading - Robert Carver (Amazon)](https://www.amazon.com/s?k=Systematic+Trading+Robert+Carver&ref=aligrithm.com) - [Trading Systems - Urban Jaekle Emilio Tomasini (Amazon)](https://www.amazon.com/s?k=Trading+Systems+Urban+Jaekle+Emilio+Tomasini&ref=aligrithm.com) ### 2.82 Regularization from First Principles URL: https://aligrithm.com/regularization-from-first-principles/ Last updated: 2026-07-27T13:23:40.000Z The old article ["The Limits of Linear Models"](https://aligrithm.com/the-limits-of-linear-models/) left you with a clean little model that prices the next move off three order-book alphas, something like b\_1 times imbalance plus b\_2 times order-flow plus b\_3 times the microprice tilt. The first question is mechanical: where do b\_1, b\_2, b\_3 come from? You fit them on history. The second question is the one nobody asks until a live model starts drifting: why does the standard way of fitting them, the one every stats course teaches as optimal, fall apart on market data? Regularization is the answer, and you can build it from scratch with nothing fancier than the squared-error objective you already know. ## Ordinary least squares, and the theorem that blesses it Start with the fit itself. You have three noisy alphas x\_1, x\_2, x\_3 and a target, the future price change. You want the coefficients that make the model's predictions match history as closely as possible, so you pick the b\_i that minimize the sum of squared errors across all your historical rows. $$ L \\;=\\; \\sum\_{t}\\Big(\\,b\_1 x\_{1,t} + b\_2 x\_{2,t} + b\_3 x\_{3,t} - y\_t\\,\\Big)^2 $$ Read that as: for every timestamp t, take the model's prediction, subtract the actual future price change y\_t, square the miss so big errors dominate and the sign drops out, and add up the squares. The b\_i that drive this sum to its minimum are the ordinary-least-squares estimates. This is the workhorse, and there is a theorem that says it is not just convenient but optimal: the Gauss-Markov theorem. It states that among all unbiased estimators that are linear in the data, OLS has the smallest variance. The shorthand is BLUE, the best linear unbiased estimator. The theorem is real, and the trap is hidden in its fine print. BLUE holds only when three conditions about the model's errors are true: the errors average to zero, they all share one constant variance, and they are uncorrelated with each other. Grant those three and OLS is unbeatable in its class. The whole argument for using it rests on those three assumptions holding. ## The assumptions die in markets Markets violate all three, and not at the margin. The errors do not share a constant variance. Volatility clusters, so the model's misses are tiny in calm tape and enormous in a vol spike, which is heteroskedasticity by another name, and the old article ["Why Volatility Is More Non-Stationary Than Trend"](https://aligrithm.com/why-volatility-is-more-non-stationary-than-trend/) is the long version of why this is the rule rather than the exception. The errors are not even drawn from the same distribution over time, because the relationship you are fitting is non-stationary: the coefficient that priced imbalance well last quarter is wrong this quarter when the book regime shifts. And the tails are fat, so a handful of jump prints carry returns many standard deviations out, which a squared-error objective weights brutally because squaring an outlier makes it scream. The third violation is the one that matters most for regularization. Your alphas are correlated and redundant. Imbalance, order-flow, and microprice tilt all read the same pressure in the book from slightly different angles, so x\_1, x\_2, x\_3 move together. When the inputs are collinear, OLS cannot tell which one deserves the credit, and it copes by handing out enormous offsetting coefficients, a giant positive b\_1 nearly cancelled by a giant negative b\_2\. The fit looks fine in-sample and the coefficients are garbage: tiny changes in the data swing them wildly, and a coefficient vector that swings on noise prices the next trade on noise. So the estimates come out unbiased, exactly as the theorem promises, and useless, because their variance is gigantic the moment the inputs are correlated. Unbiased-but-high-variance is the worst trade you can be holding when you have to commit capital to the forecast. ## Buy variance back with a penalty The fix is to stop insisting on unbiasedness. Add a term to the objective that punishes large coefficients, so the optimizer has to weigh fitting the data against keeping the b\_i small. $$ \\text{Ridge (L2):}\\quad L \\;+\\; a\\,\\big(b\_1^2 + b\_2^2 + b\_3^2\\big) $$ $$ \\text{Lasso (L1):}\\quad L \\;+\\; a\\,\\big(|b\_1| + |b\_2| + |b\_3|\\big) $$ Read the ridge line as: minimize the same sum of squared errors as before, but add a tax equal to a constant a times the sum of the squared coefficients. The lasso line taxes the sum of the absolute coefficients instead. In both, a is a dial you set: at a equal to zero you are back to plain OLS, and as a grows the penalty drags every coefficient toward zero. Those offsetting giant coefficients that collinearity produced are now expensive, so the optimizer refuses to pay for them and settles on a smaller, steadier set. This estimator is biased. The Gauss-Markov theorem no longer applies and you would not have been allowed to use this before, because shrinking the coefficients pulls the fit away from the in-sample optimum on purpose. The trade is bias for variance: you give up a sliver of in-sample accuracy and buy back a large chunk of stability, so the coefficients barely move when the data wobbles and the out-of-sample forecast holds up. The old article ["Parameter Stability Beats Best Parameter"](https://aligrithm.com/parameter-stability-beats-best-parameter/) is the same discipline applied to a different knob, and it is the discipline that the whole machine-learning arc keeps coming back to. ## Ridge spreads, lasso selects The two penalties shrink differently, and the difference decides which to reach for. Ridge squares the coefficients, so the marginal tax on a coefficient grows as it grows, which makes ridge punish a few large weights far more than many small ones. Faced with three correlated alphas, ridge spreads the weight across all three rather than letting one blow up, which is what you want when you believe every input carries a little real signal and you want a smooth, stable blend. Lasso taxes the absolute value, and the geometry of that penalty drives weak coefficients exactly to zero rather than merely small. Lasso does feature selection: hand it ten redundant momentum variants and it tends to keep one and zero the rest. That is useful when you suspect most of your inputs are noise and you want a sparse model, and it is dangerous with correlated alphas because which one survives can flip from one sample to the next, so the selection itself becomes a source of instability. A common practical answer is to mix the two, an L1 plus L2 blend, which gets lasso's sparsity without its arbitrary tie-breaking among correlated inputs. The choice is not cosmetic: it encodes what you believe about your alphas, whether they are many weak real signals to be blended or a noisy pile to be pruned. ## Setting the dial without cheating The penalty strength a is a parameter, which means it is one more thing you can overfit if you pick it by staring at the in-sample fit. At a equal to zero the in-sample error is always lowest, because that is what OLS minimized, so the in-sample curve will always tell you to regularize less, which is exactly backwards. You set a out of sample. Cross-validate across folds, or walk it forward, and watch the held-out error as you raise a: it falls as shrinkage kills the variance, bottoms out, and climbs again once you have shrunk so hard the model can no longer fit the real structure. You want the floor of that held-out curve, not the floor of the in-sample one, and you want to read it as a plateau rather than a spike, the same way the old article ["The Hill, the Spike, and the Cliff: Reading Optimization Surfaces"](https://aligrithm.com/the-hill-the-spike-and-the-cliff-reading-optimization-surfaces/) tells you to read any optimization surface. ## Where this sits in the arc Regularization is the linear model's variance medicine, and it is the same medicine the next articles give to trees. The old article ["From One Tree to Forests to Boosting"](https://aligrithm.com/from-one-tree-to-forests-to-boosting/) stabilizes a fragile tree by averaging many of them, which is a variance cut by a different mechanism, and boosting's learning rate is a shrinkage knob in disguise, a per-step tax that keeps each tree from overcommitting. Ridge and lasso are where the idea is cleanest, because you can see the penalty sitting right next to the squared-error term and watch it trade bias for stability one coefficient at a time. Carry that picture forward: every model past the toy linear one is going to need some version of this brake, and the reason is always the same, that an unbiased fit on non-stationary, fat-tailed, collinear market data hands you a high-variance answer you cannot trade. ## Visualizing the trade ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_334-visual-0.png) ## KEY POINTS - Fitting a linear alpha model means minimizing squared error, and the Gauss-Markov theorem says that fit, OLS, is the best linear unbiased estimator. The blessing holds only under three assumptions: zero-mean errors, constant variance, and uncorrelated errors. - Markets break all three. Volatility clustering kills constant variance, regime shift kills stationarity, fat tails make squared error chase outliers, and correlated alphas make OLS hand out giant offsetting coefficients that swing on noise. - Unbiased but high-variance is a bad hand to hold when you have to trade the forecast. The fix is to drop unbiasedness on purpose. - Add a penalty: ridge taxes the sum of squared coefficients, lasso taxes the sum of absolute coefficients, with a dial a setting the strength. The estimator becomes biased but far lower variance and steadier out of sample. - Ridge spreads weight across correlated inputs for a smooth blend; lasso drives weak weights to zero for a sparse model but picks arbitrarily among correlated inputs. An L1+L2 blend gets sparsity without the arbitrary tie-break. - Set a out of sample, never in sample, where the error always favors a equal to zero. Cross-validate or walk forward and take the plateau of the held-out error. - This is the linear-model version of the variance cut that bagging and boosting give to trees. Every model past the toy linear one needs some version of this brake. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Statistically Sound Indicators for Financial Market Prediction - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Statistically+Sound+Indicators+for+Financial+Market+Prediction+Timothy+Masters&ref=aligrithm.com) - [Cycle Analytics for Traders - John Ehlers (Amazon)](https://www.amazon.com/s?k=Cycle+Analytics+for+Traders+John+Ehlers&ref=aligrithm.com) ### 8.8 Itô Calculus, Intuitively URL: https://aligrithm.com/ito-calculus-intuitively/ Last updated: 2026-07-27T13:26:04.000Z Stochastic calculus reads as a wall of unfamiliar symbols, and most traders bounce off it and decide the math behind every price model is beyond them. It is not. The central object, the Itô integral, is one small twist on the integral you already met in school, and you can get to it in three steps that each change exactly one thing. Build it that way and the symbols stop being a wall. They start telling you something concrete about a price that wanders. ## Step one: the Riemann integral you already know Start with the ordinary integral. You want the area under a curve f, so you chop the horizontal axis into thin rectangles, take the height of f on each slice, multiply by the width, and add them up. Make the rectangles thinner and add more of them, and the sum converges to the area. $$ \\int\_a^b f(x)\\,dx \\;\\approx\\; \\sum\_i f(x\_i)\\,\\Delta x \\qquad (\\Delta x \\to 0) $$ Read that as: every rectangle has the same kind of width, a plain step along the x axis written as delta-x, and the integral is the limit of the rectangle sum as those widths shrink to zero. The thing to notice, because it is the thing the next step changes, is that each rectangle is weighted by an equal step in x. Every slice counts the same amount of horizontal distance. ## Step two: weight the rectangles by another function The Riemann-Stieltjes integral changes one thing. Instead of weighting each rectangle by a fixed step in x, you weight it by the change in some second function g across that slice. $$ \\int\_a^b f(x)\\,dg(x) \\;\\approx\\; \\sum\_i f(x\_i)\\,\\Delta g\_i $$ $$ \\text{if } g \\text{ is smooth: } \\int\_a^b f(x)\\,dg(x) = \\int\_a^b f(x)\\,g'(x)\\,dx $$ Read the top line as: the same rectangle sum, but each height f is now multiplied by delta-g, how much g moved over that slice, rather than by a constant delta-x. The bottom line is the sanity check: when g is well behaved, this is just the ordinary integral of f times the derivative of g, so nothing exotic has happened yet. Two cases pin the intuition. If g is constant it never moves, so every delta-g is zero and the whole integral is zero, no matter what f does. If g is the identity, g of x equals x, then delta-g equals delta-x and you are back to the plain Riemann integral. So Riemann-Stieltjes is a strict generalization: it lets the weighting come from the behavior of g instead of from uniform horizontal steps. ## Step three: make the weighting function random Here is the only real leap, and it is small. Replace g with Brownian motion. The Itô integral weights each rectangle by the change in a Brownian path rather than the change in a deterministic function. $$ \\int\_0^t f \\,dW \\;\\approx\\; \\sum\_i f(t\_i)\\,\\Delta W\_i, \\qquad \\Delta W\_i = W\_{t\_{i+1}} - W\_{t\_i} $$ Read that as: the same construction one more time, but now each height f is multiplied by delta-W, the increment of a Brownian motion over that slice. Brownian motion is the random walk in continuous time, the canonical model of an unpredictable wandering path, and its increments delta-W are random, so the weights themselves are random. The consequence falls straight out: because the path W could have wandered any number of ways, the value of the integral is not a single number. It is a random variable, with its own distribution. You ran the same rectangle sum, but the answer now has a spread because the thing you weighted by has a spread. ## What you actually get: a process, not a number In trading you care about systems that evolve, so you take the integral from time zero up to a running time t and watch how it grows. $$ Y(t) = \\int\_0^t f \\,dW $$ Read that as: for every horizon t, the integral from zero to t gives you Y of t, and because the Brownian increments inside are random, Y of t is itself random, carrying a probability distribution at each point in time. Sweep t forward and you trace out a whole family of possible paths, which is a stochastic process. That is the payoff: the Itô integral does not output an area, it outputs a process whose value at each instant is a distribution rather than a point. And the integrator does not have to be plain Brownian motion. In full Itô calculus both the integrand f and the integrator can themselves be stochastic processes, which is what lets you write a price whose volatility is also random and wandering. ## Why this is the engine under your price models This machinery is the thing under the hood every time you see a price written as a drift term plus a noise term. The old article ["Log-Normal Prices and the Price/Volume Trick"](https://aligrithm.com/log-normal-prices-and-the-price-volume-trick/) models a price as geometric Brownian motion, and the noise piece of that model, the part written with a dW, is exactly the Itô integral built above: the accumulated effect of random Brownian kicks weighted by the current price. When a model says volatility scales the dW term, it is saying f in the integral is the volatility and the integrator is the Brownian path, so the price process Y of t inherits its randomness from the integral. The whole apparatus of diffusion that the old article ["Long-Range Dependence: Real Memory or Just Short-Range Echo?"](https://aligrithm.com/long-range-dependence-real-memory-or-just-short-range-echo/) leans on, the idea that price changes accumulate like a wandering walk, is this same integral run forward in time. You do not have to solve these integrals by hand to trade. Knowing what they mean tells you why a price model has a deterministic part you can reason about and a random part you can only describe by its distribution, and that split is the whole reason risk is a distribution and not a forecast. ## Visualizing the three steps ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_337-visual-0.png) ## KEY POINTS - The Itô integral is one small twist on the ordinary integral. Build it in three steps that each change exactly one thing. - Riemann integral: chop the axis into thin rectangles, weight each by an equal step in x, sum, and shrink the widths to zero. Every slice counts the same horizontal distance. - Riemann-Stieltjes integral: weight each rectangle by the change in a second function g instead of by a fixed step. If g is constant the integral is zero; if g equals x you recover the Riemann integral. For smooth g it is just the integral of f times g-prime. - Itô integral: replace g with Brownian motion, so each rectangle is weighted by a random Brownian increment. The weights are random, so the integral is a random variable with a distribution, not a single number. - Take the integral from zero to a running time t and you get a stochastic process Y of t, whose value at each instant is a distribution. Both the integrand and the integrator can themselves be stochastic processes. - This is the machinery under every price model written as drift plus a dW noise term, including the geometric Brownian motion of the old article ["Log-Normal Prices and the Price/Volume Trick."](https://aligrithm.com/log-normal-prices-and-the-price-volume-trick/) The random part of a price is exactly this integral run forward, which is why risk is a distribution, not a forecast. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Mathematical Techniques in Financial Markets - Don Mak (Amazon)](https://www.amazon.com/s?k=Mathematical+Techniques+in+Financial+Markets+Don+Mak&ref=aligrithm.com) - [Cycle Analytics for Traders - John Ehlers (Amazon)](https://www.amazon.com/s?k=Cycle+Analytics+for+Traders+John+Ehlers&ref=aligrithm.com) ### 7.3 Rust Data Server, Python Brain URL: https://aligrithm.com/rust-data-server-python-brain/ Last updated: 2026-07-27T13:25:59.000Z Profile a Python trading algo that feels slow and the result almost never points where you expect. The bottleneck is not your alpha, your portfolio logic, or your sizing math. It is decoding the JSON pouring in off the websocket, with sending orders a distant second. The interesting code, the part you spent months on, barely registers on the flame graph. That single fact dictates the architecture: put the dumb, hot, repetitive work in a fast language and keep the complicated, stateful, rarely-hot work in Python. You get HFT-grade ingest without paying HFT-grade development time, because you only rewrite the part that is both slow and simple. ## Where the time actually goes A streaming algo spends its life in a tight loop: receive a message, parse it, update state, maybe act. The receive-and-parse step runs millions of times and touches every byte on the wire, so it dominates. Reaching for `orjson` instead of the standard library buys you a real but bounded speedup, because you are still handing bytes to Python objects, allocating dicts, and paying the interpreter for every field. It is a band-aid on the right wound. The structural fix is to never let those bytes reach Python as JSON at all. $$ \\text{loop latency} \\;\\approx\\; \\underbrace{t\_{\\text{decode}}}\_{\\text{dominant}} \\;+\\; t\_{\\text{logic}} \\;+\\; \\underbrace{t\_{\\text{send}}}\_{\\text{runner-up}} $$ Read it as: the time around your loop is mostly the decode cost, plus a usually-small logic cost, plus the order-send cost, and the decode term is the one that scales with message volume and swamps the rest in a busy market. Optimize the dominant term and the loop gets fast. Optimize your logic, the term you enjoy working on, and you move a rounding error. ## The split: Rust ingests, Python decides Write the data plane as its own small program in a fast language, Rust, C, C++, or Java, whatever you can stand. Its only job is to connect to the exchange, decode the binary or JSON feed, package the result into a compact internal format, and forward it to the Python process. Because it is just streaming and reshaping data, it is a simple program you can finish in days, not a system. And since you are already in that program, push more work upstream: filter out updates that do not matter, downsample, even compute the spread or the signal on the server and forward the result instead of the raw book. The less the Python side has to decode, the less the dominant term costs. Python keeps the brain. Order management, portfolio state, position sizing, reporting, reconnection logic, unwinding, server controls, the trade database, inventory tracking, every messy edge case lives in Python where it is fast to write and fast to change. This is the part that takes months, and it takes months because of the fiddly stateful detail, not because of compute. In my experience that stateful logic runs at least twice as slow to write in Rust as in Python, and the gap widens on the gnarly edge cases, so building the brain in Rust would balloon your slowest, most error-prone work to speed up code that was never the bottleneck. You split the system exactly along the line where speed matters and complexity does not. ## Why this boundary and not "rewrite it all in Rust" The instinct after a slow profile is to rewrite the whole thing in the fast language. That trades a small, contained win for a large, ongoing cost. The hot path, ingest and forward, is a few hundred lines and changes rarely. The brain is thousands of lines and changes constantly as you add strategies, handle a new exchange quirk, or patch the way you unwind on a disconnect. Putting the volatile, complex code in the language that is slow to write means every change costs you several times more, forever, in exchange for speeding up logic that contributed almost nothing to latency. Keep the boundary at the JSON, where the fast side is simple and stable and the slow-to-write side is where all the real engineering lives. This is the infrastructure that makes event-driven trading practical. The old article ["Continuous, Event-Driven Trading vs Bar-Based Research"](https://aligrithm.com/continuous-event-driven-trading-vs-bar-based-research/) argued a maker must act on events as they land, not on a bar clock, and a pure-Python ingest loop chokes exactly when events cluster and the decode term spikes. The Rust front end absorbs the burst so the Python brain still sees every event in time to act. There is a backtest catch, the one the old article "Backtest Integrity Checklist as Code" keeps warning about: if your Rust server filters, downsamples, or precomputes features, it must do so identically to your research pipeline, or you have train-serve skew, a strategy validated on one transform and traded on another. The fast preprocessor is now part of your feature definition, so it has to match the backtest byte for byte. ## The boundary's own costs Two languages means two build systems, two sets of bugs, and a serialization layer between them, and that layer can quietly hand back the latency you saved. Forward the data as JSON over a socket and Python pays the decode cost anyway, so you gained nothing. Use a compact binary format, shared memory, or a zero-copy transport so the Python side reads structured fields without re-parsing text. Watch the failure modes that now span the boundary: if the Rust feed dies, the Python brain must notice and unwind rather than trade on a frozen book, so the reconnection and stale-data logic, already in Python, has to monitor the server too. And keep the feature parity discipline above front of mind, because a mismatch between the Rust preprocessor and the research code is the kind of bug that passes every test and loses money live. ## Visualizing the split ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_330-visual-0.png) ## KEY POINTS - Profiling a slow streaming algo points at JSON decode first and order send second; your alpha and portfolio logic are a rounding error, so optimize the decode term, not the code you enjoy. - `orjson` is a bounded band-aid because Python still allocates objects per field; the structural fix is to never let raw JSON reach Python. - Write the data plane as a small fast-language program that decodes, filters, packages, and even precomputes the spread or features, then forwards a compact format. It is simple and takes days. - Keep the brain in Python: OMS, portfolio, sizing, reporting, reconnection, unwinding, controls, trade DB, inventory. This is the months-long work, and that stateful logic runs at least twice as slow to write in Rust. - Split at the JSON boundary because the hot path is small and stable while the brain is large and volatile; rewriting the brain in Rust triples your hardest work to speed up code that was never slow. - This makes event-driven trading practical (the old article ["Continuous, Event-Driven Trading vs Bar-Based Research"](https://aligrithm.com/continuous-event-driven-trading-vs-bar-based-research/)), but any filtering or feature computation on the fast side must match your research pipeline exactly, or you get the train-serve skew the old article "Backtest Integrity Checklist as Code" warns about. Forward binary, not JSON, or you pay the decode cost twice. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Statistically Sound Indicators for Financial Market Prediction - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Statistically+Sound+Indicators+for+Financial+Market+Prediction+Timothy+Masters&ref=aligrithm.com) - [Systematic Trading - Robert Carver (Amazon)](https://www.amazon.com/s?k=Systematic+Trading+Robert+Carver&ref=aligrithm.com) ### 7.2 Dynamic Time Warping for Time-Series Alignment URL: https://aligrithm.com/dynamic-time-warping-for-time-series-alignment/ Last updated: 2026-07-27T13:25:58.000Z The lagged cross-correlation has a hidden assumption baked in. The old article ["Lead-Lag Relationships in Global Markets"](https://aligrithm.com/lead-lag-relationships-in-global-markets/) measured a lead by sliding one series against another by a fixed number of bars and finding the lag that lined them up best. That works when the delay between two markets is constant. It is not constant. The leader sometimes races ahead by two seconds, sometimes by ten, and a single fixed shift smears all of that into one blurry average. Dynamic time warping fixes the rigidity: instead of one shift for the whole series, it lets the alignment stretch and compress along the way so each point in A is matched to the point in B it actually corresponds to. The old article "Crypto Cross-Exchange Lead-Lag Study" built the fixed-lag version on crypto venues. This is the tool you reach for when that fixed lag stops fitting because the delay itself is moving. ## Euclidean matching is point-for-point and brittle The default way to compare two series is to line them up index-by-index: A's first point against B's first point, second against second, and sum the gaps. That is Euclidean matching, and it assumes both series march on the same clock. If B is the same shape as A but shifted or stretched in time, Euclidean matching reports a huge distance even though the two are obviously the same pattern, because it is comparing A's peak against B's trough purely on account of a timing offset. DTW throws out the one-to-one rule. It is allowed to match one point in A to several points in B, or skip ahead, whatever produces the cheapest total alignment. A point near A's peak gets matched to the point near B's peak even if they sit at different timestamps. The result is a distance that measures shape similarity after absorbing timing differences, which is exactly what you want when two markets trace the same move on a delay that wanders. ## The recurrence DTW is built from a cumulative cost matrix. Fill a grid where entry (i, j) is the cheapest total cost of aligning A up to point i with B up to point j, and build it with one short recurrence. $$ D\_{ij} = \\left|A\_i - B\_j\\right| \\; + \\; \\min\\!\\left(D\_{i-1,j-1},\\; D\_{i-1,j},\\; D\_{i,j-1}\\right) $$ Read it left to right. The first term, the absolute difference between A's value at i and B's value at j, is the local cost of pairing those two points. The second term is the cheapest way to have arrived at cell (i, j) from its three predecessors: the diagonal step (advance both series one point), the vertical step (hold A, advance B), and the horizontal step (hold B, advance A). The diagonal is a normal one-to-one match; the vertical and horizontal steps are where the warping happens, one series waiting while the other catches up. Adding the local cost to the minimum of the three accumulates the best alignment so far. The number in the bottom-right corner, after filling the whole grid, is the DTW distance, and tracing the minimizing path back through the grid gives you the actual alignment, which point in A maps to which point in B. That backtraced path is the lead-lag information you came for. When the path runs along the diagonal, the two series are moving in lockstep. When it veers horizontal or vertical, one is leading the other, and the size of the detour tells you by how much at that moment, a lag that varies through time instead of the single number the cross-correlation gives you. ## The recipe in a few lines The whole thing is a double loop over the cost grid. Code it once and you understand it better than any diagram. ```python import numpy as np def dtw(A, B): n, m = len(A), len(B) D = np.full((n + 1, m + 1), np.inf) D[0, 0] = 0.0 for i in range(1, n + 1): for j in range(1, m + 1): cost = abs(A[i - 1] - B[j - 1]) # local pairing cost D[i, j] = cost + min(D[i - 1, j - 1], # diagonal: match D[i - 1, j], # vertical: B waits D[i, j - 1]) # horizontal: A waits return D[n, m], D def warping_path(D): i, j = D.shape[0] - 1, D.shape[1] - 1 path = [(i, j)] while i > 1 or j > 1: step = np.argmin([D[i - 1, j - 1], D[i - 1, j], D[i, j - 1]]) i, j = [(i - 1, j - 1), (i - 1, j), (i, j - 1)][step] path.append((i, j)) return path[::-1] ``` The first function returns the DTW distance and the full cost matrix; the second walks the matrix backward from the corner, at each step taking whichever predecessor was cheapest, to recover the alignment. Run `warping_path` and plot the (i, j) pairs and you see the alignment bend wherever the lead-lag delay shifted. ## Where it earns its keep, and where it costs you The honest use cases are narrow and real. DTW gives you a similarity score between two assets that is robust to a wandering delay, which is a better input than raw correlation when you are clustering instruments by behavior or picking which pairs to even run a lead-lag study on, the graph-of-assets step the old lead-lag work used to choose what to trade. It also recovers a time-varying lag, telling you not just that A leads B but that the lead was short during calm and stretched during a fast move. The costs are not small. The double loop is order n times m, quadratic in series length, so naive DTW on long high-frequency series is slow and you need a band constraint (forbid the path from straying too far off the diagonal) to make it tractable and to stop it from inventing absurd alignments. That band is also a guard against the deeper failure: unconstrained DTW will happily warp noise into a beautiful match, the same overfitting the old lead-lag article warned about when you scan too many pairs and lags, except now the model has even more freedom to find a pattern that is not there. And the warping path itself is not causal. It uses the whole series, future included, to decide the alignment, so it is a research and diagnostic tool for understanding a relationship, not something you drop straight into a live signal without a causal, online reformulation. Treat the distance as a similarity measure you trust and the path as a hypothesis about a varying lag you then verify out of sample, not as a tradable signal on its own. ## Visualizing warping vs straight matching ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_309-visual-0.png) ## KEY POINTS - Lagged cross-correlation (the old article ["Lead-Lag Relationships in Global Markets"](https://aligrithm.com/lead-lag-relationships-in-global-markets/)) assumes a constant delay between two series; real lead-lag delays wander, and a single fixed shift averages them into a blur. - Euclidean matching compares two series index-by-index and reports a large distance for the same shape on a timing offset; DTW drops the one-to-one rule and matches each point to the corresponding point regardless of timestamp. - DTW fills a cumulative cost matrix with D\[i,j\] = |A\_i - B\_j| + min of the three predecessors (diagonal = match, vertical/horizontal = one series waits); the corner value is the distance and the backtraced path is the alignment. - The warping path is the payoff: diagonal stretches mean lockstep, horizontal or vertical detours mean one series leads, and the detour size is a lag that varies through time rather than one number. - Good uses are a delay-robust similarity score for clustering assets and choosing pairs to study, plus recovering a time-varying lag, extending the lead-lag study from the old article "Crypto Cross-Exchange Lead-Lag Study." - The costs are real: it is quadratic (needs a band constraint for long series), it overfits noise into pretty matches without that constraint, and the path is non-causal, so use it for research and diagnosis, not as a drop-in live signal. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Statistically Sound Indicators for Financial Market Prediction - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Statistically+Sound+Indicators+for+Financial+Market+Prediction+Timothy+Masters&ref=aligrithm.com) - [Systematic Trading - Robert Carver (Amazon)](https://www.amazon.com/s?k=Systematic+Trading+Robert+Carver&ref=aligrithm.com) ### 7.1 Synthetic Prices: EMA of Random Numbers as a Market Model URL: https://aligrithm.com/synthetic-prices-ema-of-random-numbers-as-a-market-model/ Last updated: 2026-07-27T13:25:58.000Z The old article ["The Problem with One Sample of Market History"](https://aligrithm.com/the-problem-with-one-sample-of-market-history/) made the uncomfortable point that you only ever get one realized price history, one draw from a meta-distribution you cannot resample, so every backtest conclusion rests on a sample size of one universe. The old article ["Monte Carlo for Trading Systems"](https://aligrithm.com/monte-carlo-for-trading-systems/) offered synthetic-path generation as one partial workaround, while flagging that it depends on a model that might be wrong. This article gives you the simplest synthetic-price model that captures the one property most generators miss: memory. It is almost trivial to code, and that simplicity is the feature. ## Prices are random numbers with memory The naive synthetic price is a pure random walk: draw independent random returns, add them up. The problem is that real prices are not a fresh coin flip each bar. They carry memory, the persistence the old noise-color and Hurst articles measured, and a pure random walk has none of the right kind. Ehlers' model fixes this with one move: take an exponential moving average of random numbers. The random numbers supply the unpredictability; the EMA supplies the memory, because each output blends today's random draw with a fading echo of everything before it. $$ P\[t\] \\;=\\; \\alpha\\, R\[t\] \\;+\\; (1-\\alpha)\\, P\[t-1\], \\qquad R\[t\]\\sim \\text{random}, \\quad \\alpha=\\frac{2}{M+1} $$ R\[t\] is a fresh random number each bar (uniform or Gaussian, it barely matters), alpha sets how much weight the new draw gets versus the carried-over history, and M is the EMA length that controls how long the memory lasts. A larger M means a smaller alpha, a longer fading echo, and a synthetic series that wanders with more persistence. That is the whole model: a low-pass filter fed noise. It is the mirror image of the roofing-filter idea, where you filtered noise out of price; here you filter noise into something that looks like price, and the result is a reasonable approximation of a real chart precisely because real prices behave like random numbers run through a memory. ## The recipe in a few lines There is nothing to it once you see the equation. Generate noise, run the EMA, and you have a price series with tunable memory. ```python import numpy as np def synthetic_prices(n=2000, M=20, seed=None): rng = np.random.default_rng(seed) alpha = 2.0 / (M + 1) r = rng.standard_normal(n) # the unpredictable part p = np.zeros(n) for t in range(1, n): p[t] = alpha * r[t] + (1 - alpha) * p[t - 1] # the memory part return np.cumsum(p) # integrate to a price-like level ``` The cumulative sum at the end turns the smoothed-noise increments into a wandering level that reads like a price chart, the integration step that, per the old noise-color article, pushes the spectral slope toward the brownian end where price lives. Change M and you change the cycle character: short M gives a jittery, fast-reverting series, long M gives long lazy swings. Change the seed and you get a different history from the same process, which is the entire point. ## What synthetic prices are good for, and what they are not The reason to bother is the one-sample problem. You cannot rerun 2008, but you can generate ten thousand histories from a process with the same memory structure and watch how your strategy behaves across all of them, turning a single fragile backtest into a distribution. That is the synthetic-path use the old Monte Carlo article described: confidence intervals on drawdown, sequence-of-returns stress tests, and kill-switch thresholds set at a percentile of the synthetic distribution rather than at the single worst day you happened to observe. It is also the cleanest way to test a cycle measurement tool, because you can build a synthetic series with a known injected cycle and check whether your periodogram recovers it. The honest limits are exactly the ones the Monte Carlo article pressed, and they are not small. An EMA of Gaussian noise has thin tails, so it will understate crash risk badly; if you size or stress-test off this model you will be blindsided by the fat tails real markets carry. It has stationary statistics by construction, so it cannot reproduce regime shifts, volatility clustering, or the non-stationarity that breaks live strategies. And it has none of the participant feedback that makes real markets adapt to the very edges you publish. So use synthetic prices to study a tool's behavior, to build intuition, and to put error bars on metrics, never to validate that a strategy has edge. Synthetic data cannot replace out-of-sample testing, cannot correct for search-width bias, and cannot fill regime-coverage gaps. It is a sandbox with the right kind of memory and the wrong kind of tails, useful exactly as far as you remember which is which. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_249-visual-0.png) ## KEY POINTS - You only get one realized market history (the old article ["The Problem with One Sample of Market History"](https://aligrithm.com/the-problem-with-one-sample-of-market-history/)), so synthetic prices let you turn a single fragile backtest into a distribution, the synthetic-path workaround from the old article ["Monte Carlo for Trading Systems."](https://aligrithm.com/monte-carlo-for-trading-systems/) - A pure random walk lacks the right memory; Ehlers' model is an EMA of random numbers, where the random draws give unpredictability and the EMA gives a fading memory of the past. - The recipe is P\[t\] = alpha*R\[t\] + (1-alpha)*P\[t-1\] with alpha = 2/(M+1), then a cumulative sum to get a price-like level; M tunes how long the memory lasts and what cycle character the series has. - It is the mirror of the roofing filter: there you filter noise out of price, here you filter noise into something that looks like price, which works because real prices behave like random numbers with memory. - Use it for confidence intervals on risk metrics, sequence-of-returns stress tests, percentile-based kill switches, and testing whether a cycle tool recovers a known injected cycle. - Its limits are severe and built in: Gaussian-EMA output has thin tails (understates crashes), is stationary (no regime shifts or volatility clustering), and has no participant feedback, so it is a sandbox for studying tools, never a substitute for out-of-sample validation. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Statistically Sound Indicators for Financial Market Prediction - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Statistically+Sound+Indicators+for+Financial+Market+Prediction+Timothy+Masters&ref=aligrithm.com) - [Systematic Trading - Robert Carver (Amazon)](https://www.amazon.com/s?k=Systematic+Trading+Robert+Carver&ref=aligrithm.com) ### 6.45 Don't Trust Your Strategy-Weighting Scheme URL: https://aligrithm.com/dont-trust-your-strategy-weighting-scheme/ Last updated: 2026-07-27T13:25:52.000Z You hold five strategies and a fixed pool of capital, and you need a rule for splitting it. A tempting rule scores each strategy on a handful of metrics, converts each metric into a weight vector, then averages those vectors with a fixed blend like 0.3, 0.3, 0.2, 0.2\. It reads as rigorous because it mixes return and risk, and it reads as safe because four numbers beat one. Both readings fail. The rule stacks several noisy in-sample estimates, secretly double-counts the same two quantities, and invites you to tune the blend until the backtest smiles. The old article ["Why Portfolio Construction Is Part of the Signal"](https://aligrithm.com/why-portfolio-construction-is-part-of-the-signal/) showed that how you weight changes what the book predicts, so a weighting rule this brittle is a second model riding on top of your strategies, and you never validated it. ## The scheme, written out Take four scores. Normalize each into a weight vector by dividing every strategy's score by the sum across strategies. $$ \\text{Return}\_i = \\text{CAGR}\_i \\cdot \\sqrt{N\_i}, \\qquad \\text{Sortino}\_i = \\frac{\\text{CAGR}\_i}{\\sigma\_{\\text{down},i}} $$ $$ \\text{Calmar}\_i = \\frac{\\text{CAGR}\_i}{|\\text{MaxDD}\_i|}, \\qquad R^2\_i = \\text{fit of strategy } i\\text{'s equity curve to a straight line} $$ $$ \\text{FinalWeight}\_i = 0.3\\,\\text{Return} + 0.3\\,\\text{Sortino} + 0.2\\,\\text{Calmar} + 0.2\\,R^2 $$ Read it as: the first score rewards compounded growth, CAGR scaled by the square root of the trade count so a strategy with more trades earns more trust. The second rewards return per unit of downside deviation, Sortino. The third and fourth reward a clean ride: Calmar punishes the deepest pit relative to growth, and the equity-curve R-squared rewards a strategy whose cumulative P&L tracks a straight line instead of lurching. The final line averages the four normalized vectors with a 60 percent tilt to return and a 40 percent tilt to path quality. It looks like a balanced panel of four judges. It is three problems wearing one robe. ## Pitfall one: every input is a noisy in-sample point estimate CAGR, downside deviation, max drawdown, and equity-curve R-squared all come off one backtest, so all four are sample statistics with fat standard errors, not the true values. A sample CAGR is one realized path out of a distribution you cannot see, and a measured Sortino of 2.0 over a few hundred trades is consistent with a true Sortino near 0.5\. Max drawdown is the worst case: it is an extreme-order statistic, the single deepest point on one path, so it carries enormous sampling variance and would land somewhere else on a fresh draw of the same process. The old article ["Average Drawdown vs Extreme Drawdown"](https://aligrithm.com/average-drawdown-vs-extreme-drawdown/) argued exactly this, that the worst point is the least stable number you can hang a decision on, and Calmar puts that number straight in the denominator. The square-root-of-trades multiplier pretends to fix the reliability problem by trusting busier strategies more, but it conflates count with information. A few hundred tightly autocorrelated trades from one regime carry far less independent evidence than the raw count implies, so the multiplier inflates confidence in strategies that churned inside a single market state. You weight on numbers whose error bars overlap the gaps between them, and the rule treats them as exact. ## Pitfall two: the four metrics are two metrics Average four scores and you picture four independent views voting. They are not independent. They collapse into two families. CAGR-times-root-N and Sortino both ride mean return: a strategy with high compounded growth posts a high Sortino too, since Sortino is that same growth divided by a downside-deviation term that moves slowly across your strategy set. Calmar and equity-curve R-squared both read the shape of the cumulative P&L: a strategy whose equity climbs in a straight line has shallow drawdowns and a high R-squared, while a lumpy one digs a deep MaxDD and bends away from the line. Calmar carries a CAGR in its numerator, but across strategies its spread is dominated by the MaxDD denominator, so it tracks the path family, not the return family. Two correlated return metrics and two correlated path metrics. So the 0.3, 0.3, 0.2, 0.2 blend is not a four-factor model. It is a 60 percent return, 40 percent path-quality weighting with two of the four columns adding the appearance of breadth and almost none of the substance. Averaging two near-copies of one quantity does not shrink its error, it doubles its vote, the same double-counting that poisons a feature set when you feed it two versions of one signal. If you wanted a 60/40 return-to-path tilt, you could write it in one line. The four-metric robe adds confidence without adding information. ## Pitfall three: weighting the weights The usual refinement, tune the 0.3/0.2 blend instead of leaving it flat, is the worst move on the board. The 60/40 split was already an arbitrary choice with no out-of-sample support. Optimize it and you convert a free constant into a fitted parameter, and fitting it on the same backtest that produced the scores means you are tuning the optimizer against in-sample noise. Every researcher degree of freedom you bolt onto a weighting rule is one more knob the backtest sets to flatter itself, and the bill arrives in live trading, where the tuned blend drifts like any other overfit parameter. ## What to actually do Reach for the rule that estimates the fewest inputs. Equal weight estimates nothing and is brutally hard to beat out of sample, which is why it is the honest baseline every fancier rule has to clear. Inverse-volatility or volatility-parity sizing estimates only a volatility, far more stable than CAGR or max drawdown, and it equalizes risk contribution directly, the thing the four-metric rule was groping toward. Go further and you go to correlation-aware construction from the old article ["Why Portfolio Construction Is Part of the Signal,"](https://aligrithm.com/why-portfolio-construction-is-part-of-the-signal/) because the diversification gain lives in how correlated the strategies are, not in how many ways you scored their past returns. The deeper lesson runs through the old article "Portfolio of Systems Simulator" and the old article ["The Death of the Single-System Trader"](https://aligrithm.com/the-death-of-the-single-system-trader/): you survive on several genuinely uncorrelated systems, and no clever blend of correlated metrics manufactures independence the strategies lack. Validate any rule the one honest way, on out-of-sample performance against equal weight, and if it cannot clear equal weight after costs, keep the baseline. ## Visualizing the collapse ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_329-visual-0.png) ## KEY POINTS - The four-score, average-the-weights scheme stacks four in-sample point estimates with fat standard errors; the max drawdown buried inside Calmar is an extreme-order statistic with the largest sampling variance of all, the weak foundation flagged by the old article ["Average Drawdown vs Extreme Drawdown."](https://aligrithm.com/average-drawdown-vs-extreme-drawdown/) - The square-root-of-trades multiplier conflates trade count with independent information and over-trusts strategies that churned inside one regime. - The four metrics collapse into two correlated families: CAGR-times-root-N and Sortino both ride mean return, Calmar and equity-curve R-squared both read the same path shape, so the blend is a 60/40 return-to-path tilt with two redundant columns faking breadth. - Averaging two near-copies of one quantity doubles its vote without shrinking its error, the same double-counting trap that wrecks a redundant feature set. - Weighting the weights turns an arbitrary blend into a fitted parameter optimized against the same in-sample noise, one more researcher degree of freedom the backtest will overfit. - Prefer the rule with the fewest estimated inputs: equal weight as the baseline, then inverse-vol, then correlation-aware construction from the old article ["Why Portfolio Construction Is Part of the Signal"](https://aligrithm.com/why-portfolio-construction-is-part-of-the-signal/); real diversification comes from uncorrelated systems, the lesson of the old article "Portfolio of Systems Simulator" and the old article ["The Death of the Single-System Trader."](https://aligrithm.com/the-death-of-the-single-system-trader/) Validate against equal-weight out of sample. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Systematic Trading - Robert Carver (Amazon)](https://www.amazon.com/s?k=Systematic+Trading+Robert+Carver&ref=aligrithm.com) - [Trading Systems - Urban Jaekle Emilio Tomasini (Amazon)](https://www.amazon.com/s?k=Trading+Systems+Urban+Jaekle+Emilio+Tomasini&ref=aligrithm.com) ### 5.42 Why Market-Making Simulations Don't Work URL: https://aligrithm.com/why-market-making-simulations-dont-work/ Last updated: 2026-07-27T13:25:26.000Z You can't simulate a limit fill that never happened. Maker/taker on historical trades works only on thin venues; pure making is just prod tuning. And Avellaneda-Stoikov is risk times root holding time. _This post is for paying subscribers only._ ### 5.41 Quotes From the Future: Clock Drift and the Latency Floor URL: https://aligrithm.com/quotes-from-the-future-clock-drift-and-the-latency-floor/ Last updated: 2026-07-27T13:25:25.000Z You are quoting a perp in Frankfurt, blending feeds from Kraken, Deribit, and OKX into one fair value. An OKX quote lands stamped 12 milliseconds ago. OKX's matching engine sits in Singapore. Light through fiber from Singapore to Frankfurt takes roughly 90 milliseconds one way, and the packet still has to reach you, so a 12ms-old quote cannot exist. Either physics broke or your clocks disagree. They disagree. That timestamp predates its own flight, and if you feed it into the cross-exchange fair value from the old article ["Cross-Exchange Fair Value for Crypto Perps"](https://aligrithm.com/cross-exchange-fair-value-for-crypto-perps/) you are pricing off a number that lies about when it was true. This bites the moment you run more than one geo-distributed feed. The old article ["Microprice: Better Than Mid Price?"](https://aligrithm.com/microprice-better-than-mid-price/) assumed the book you read is the book that exists right now. Across venues separated by continents, "now" is a negotiated quantity, and getting it wrong adds noise to fair value exactly when the feeds disagree most, which is during fast markets. ## First, fix your own clock Before you blame the exchange, look at your own machine. A trading box with a sloppy time service drifts, and a drifting local clock makes every remote timestamp look wrong by the same offset. If you see impossibly fresh timestamps across all venues at once, the common factor is you. _This post is for subscribers only._ ### 3.36 The NATGAS 20-25h Cycle: Real or Folklore? URL: https://aligrithm.com/the-natgas-20-25h-cycle-real-or-folklore/ Last updated: 2026-07-27T13:23:58.000Z A claim makes the rounds among natural gas daytraders, passed along like a secret: the NATGAS market runs a daily sell wave. Every 20 hours the probability of another wave climbs, and you can short after the 20th one-hour candle from the last peak for a 0.7% to 3.5% drop over the following three to four hours. The intervals run between 20 and 25 hours, almost never shorter or longer. Do it on both peaks and troughs and you get 2 times 22, or 44 trades per month, described as guaranteed profit. Many small wins add up. DYOR. I am not going to republish that as a strategy, because the word "guaranteed" next to a round 44 is the loudest sell-side tell in trading. But the claim is testable, and a testable claim deserves the gauntlet rather than a sneer. This article runs the NATGAS cycle through the same machinery the publication built for any seasonal: the cause test from the old article ["The Three Engines of Seasonality: Fixed-Date, Floating-Date, and Behavioral"](https://aligrithm.com/the-three-engines-of-seasonality-fixed-date-floating-date-and-behavioral/), the reliability checklist from the old article ["Is Your Seasonal Real or Curve-Fit? A Reliability Checklist"](https://aligrithm.com/is-your-seasonal-real-or-curve-fit-a-reliability-checklist/), and the null-distribution discipline from the old article ["Permutation Tests for Indicator Significance"](https://aligrithm.com/permutation-tests-for-indicator-significance/). Let the data confirm it or kill it. ## First, the red flags Before a single backtest, the claim trips four alarms that any seasonal should survive. It names no cause. Real natural gas seasonality has physical engines: heating demand in winter, injection season in summer, the EIA storage report every Thursday at 10:30 ET. A cycle that fires "every 20 hours from the last peak" anchors to nothing physical. No furnace runs on a 20-hour clock, no report drops on a drifting schedule. It drifts. A 20-to-25-hour interval is not a cycle, it is a band. A genuine periodicity has a frequency you can name; a band wide enough to slide an hour or five each day is a moving target that fits any data after the fact. The drift is the tell, not the feature. It promises a guaranteed count. 44 trades per month with guaranteed profit is the language of a course, not a researcher. Real edges come with win rates below 100%, drawdowns, and a cost line. It sits suspiciously close to 24 hours. A "20-to-25-hour cycle" lives one alias away from the daily cycle every market already has (the daily open, settlement, and report rhythm). A pattern that drifts around 24 hours is the signature of aliasing against the true daily structure, not evidence of a separate 20-hour engine. ## Step one: operationalize the claim into a precise rule A vague claim cannot be falsified. Pin it down. Detect the last swing peak on hourly NATGAS bars; count forward; at the 20th hourly candle open a short; hold for four hours; book the result. The mirror rule on troughs goes long. Already the operational choices pile up: how do you define a "peak"? A rolling-window maximum of what length? Does the 20th candle reset on every new peak or only on confirmed ones? Each choice is a free parameter, and free parameters are where curve-fits hide. The honest move is to enumerate the parameters you searched. Peak-window length (say 6, 8, 12 hours), the entry candle (18 through 25), the hold length (2, 3, 4 hours), the side rule. That is already on the order of 4 times 8 times 3 times 2, or 192 candidate strategies. The "20-25h" range alone is six candidate entry lags. You did not test one hypothesis. You tested a family, and the best member of a family always looks good. ## Step two: the cause test The old article "The Three Engines of Seasonality" gives three legitimate sources: fixed-date (a calendar anchor like the Thursday EIA report), floating-date (an anchor that moves on a known rule, like a holiday or an options expiry), and behavioral (a recurring human pattern, like end-of-quarter rebalancing). The NATGAS cycle fits none. It has no fixed calendar slot, no rule-based float, and no named behavioral driver. It is a pattern in search of an engine. This matters because a seasonal with no cause is a curve-fit until proven otherwise. The burden is on the claim to name the mechanism that would make a 20-hour sell wave recur, and "the chart shows it" is not a mechanism. ## Step three: the multiple-comparisons correction Searching 192 variants and reporting the best one inflates significance. If each variant has even a small chance of looking good by luck, the maximum over 192 of them looks great by luck alone. $$ p\_{\\text{adj}} = 1 - \\left(1 - p\_{\\text{single}}\\right)^{M}, \\qquad M = 192 $$ The adjusted p-value is one minus the chance that all M independent tests fail to reach the single-test threshold. A variant that reads a respectable p of 0.01 on its own, once you account for 192 searched variants, carries an adjusted p of 1 minus 0.99 to the 192nd power, which is about 0.85\. That is not significance. That is the near-certainty of finding something in 192 tries. The Bonferroni-style correction is crude, but it makes the point: the search width eats the edge. ## Step four: the permutation gauntlet The clean way to test the surviving rule is the permutation method from the old article ["Permutation Tests for Indicator Significance"](https://aligrithm.com/permutation-tests-for-indicator-significance/). Keep the rule fixed. Destroy the structure it claims to exploit while preserving everything else about the series. $$ \\hat{p} = \\frac{1 + \\#\\{\\, b : \\text{PnL}\_b^{\\text{perm}} \\ge \\text{PnL}^{\\text{real}} \\,\\}}{1 + B} $$ The permutation p-value counts how often a shuffled series produces a backtest PnL at least as good as the real one, across B shuffles, with the plus-one guarding against a zero count. For a cycle claim, the right shuffle is a circular block rotation or a phase scramble of the hourly returns: it keeps the daily volatility shape and the fat tails of NATGAS intact while breaking any fixed 20-hour phase relationship. Run B equal to 2000\. If the real rule's PnL sits in the fat middle of the permuted distribution, the cycle is folklore. If it lands beyond the 99th percentile and the cause test and the cost test also pass, you have something worth a small live allocation. The reliability checklist from "Is Your Seasonal Real or Curve-Fit?" adds the rest: does the pattern hold across sub-periods (split the sample by year), across both peaks and troughs symmetrically, and out of sample on data you never touched while searching? A real seasonal survives all three. A curve-fit collapses on the first cut it did not see during fitting. ## Step five: the cost test that kills most of these Grant the claim its gross edge for a moment and run the arithmetic, because NATGAS is one of the most expensive instruments retail touches. Front-month NG futures move in ticks of 0.001, worth 10 dollars per contract. A 0.7% move on a 3.00 contract is 0.021, or 21 ticks, about 210 dollars gross per contract on the small wins the claim leans on. Now the costs. NATGAS is notorious for wide spreads and gappy fills around the volatility the claim is trying to harvest. Budget one tick of spread plus a tick or two of slippage on entry and the same on exit, call it four to six ticks round trip, 40 to 60 dollars, before commissions. On the small end of the claimed move that is a quarter of the gross gone to friction. Multiply by 44 trades a month and the friction compounds into a number the "guaranteed profit" framing never mentions. And this assumes the win rate is high; the claim never states one, never shows a losing trade, and never shows the drawdown when the 21st candle keeps rising instead of falling. A strategy with no stated win rate and no drawdown is not a strategy, it is a brochure. ## The verdict and what would change it On the evidence available, the 20-25h NATGAS cycle is folklore: no named cause, a drifting band rather than a frequency, a search wide enough to manufacture significance, and a cost line the pitch hides. The aliasing against the true 24-hour structure is the most likely source of whatever pattern people see. What would move me: a single pre-registered rule (one peak-window, one entry lag, one hold), a named physical or behavioral mechanism for a sub-daily wave, a permutation p below 0.01 against a phase-scrambled null, survival across yearly sub-samples and on untouched out-of-sample data, and a net-of-cost equity curve that includes the losers. Produce those and I will trade it. Until then, the only guaranteed 44 per month is the number of times you pay the spread. ## Visualizing the teardown ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_333-visual-0-1.png) ## KEY POINTS - The claim: short NATGAS at the 20th hourly candle after a peak (mirror on troughs), 0.7% to 3.5% over three to four hours, intervals of 20 to 25 hours, 44 trades a month, guaranteed. Treat it as a hypothesis, not a strategy. - Four red flags before any test: no named cause, a drifting 20-25h band instead of a real frequency, a guaranteed round-number trade count, and a period sitting one alias away from the daily 24h structure. - Operationalize before falsifying: peak-window length, entry candle, hold length, and side rule give roughly 192 candidate variants. Reporting the best of 192 is not a single test. - Multiple-comparisons correction: a per-variant p of 0.01 across 192 searched variants becomes an adjusted p near 0.85, the near-certainty of finding something by luck. - The cause test from the old seasonality engines article: the cycle is neither fixed-date, floating-date, nor behavioral. A seasonal with no engine is a curve-fit until a mechanism is named. - The permutation gauntlet: keep the rule fixed, phase-scramble or block-rotate the hourly returns to break the 20-hour phase while preserving NATGAS volatility and tails, run 2000 shuffles, and check whether the real PnL clears the 99th percentile. - Reliability checklist: it must hold across yearly sub-samples, symmetrically on peaks and troughs, and on out-of-sample data untouched during the search. - The cost test: NG ticks are 10 dollars; a 0.7% move on a 3.00 contract is about 210 dollars gross, but four to six ticks of round-trip spread and slippage plus 44 trades a month, with no stated win rate and no drawdown, is how "guaranteed" dies. - Verdict: folklore on current evidence. It flips to tradeable only with a pre-registered single rule, a named mechanism, a permutation p below 0.01, sub-sample and out-of-sample survival, and a net-of-cost curve that includes the losers. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Testing and Tuning Market Trading Systems - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Testing+and+Tuning+Market+Trading+Systems+Timothy+Masters&ref=aligrithm.com) - [Data Mining Algorithms in C++ - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Data+Mining+Algorithms+in+C%2B%2B+Timothy+Masters&ref=aligrithm.com) ### 3.35 Trend and Reversion Are the Same, and OLS Understates Both URL: https://aligrithm.com/trend-and-reversion-are-the-same-and-ols-understates-both/ Last updated: 2026-07-27T13:23:57.000Z Trend and mean reversion look like opposites, and they are the same process with a sign flipped. Both describe a series whose next value depends on its last value, the only difference being whether the dependence pulls in the same direction or the opposite one. Write the process once and you cover both cases. $$ m\_t = \\beta \\, m\_{t-1} + \\varepsilon\_t, \\qquad \\varepsilon\_t \\sim N(0, \\sigma^2) $$ A positive beta means a high value tends to be followed by another high value: moves reinforce, the series trends. A negative beta means a high value tends to be followed by a low one: moves reverse, the series mean-reverts. The variance ratio and the efficiency ratio from the old articles ["Variance Ratio Tests for Traders"](https://aligrithm.com/variance-ratio-tests-for-traders/) and ["Efficiency Ratio Explained for Traders"](https://aligrithm.com/efficiency-ratio-explained-for-traders/) are both reading this one number from different angles. One process, one parameter, and the sign of beta is the whole distinction between the two strategy families people treat as separate worlds. Here is the problem. When you estimate beta from data with ordinary least squares, the estimate is biased toward zero in magnitude, in both the trend case and the reversion case, and the bias is not a quirk of bad data. It is built into the estimator the moment the series has any memory at all. ## Where the bias comes from OLS estimates beta with the same ratio it always uses, the covariance of the series with its own lag over the variance of the lag. $$ \\hat{\\beta}\_{\\text{OLS}} = \\frac{\\operatorname{Cov}\\!\\big(m\_t,\\, m\_{t-1}\\big)}{\\operatorname{Var}\\!\\big(m\_{t-1}\\big)} $$ For this ratio to be unbiased, the regressor has to be independent of the error. In a plain cross-sectional regression it is. Here it is not, and the reason is that the regressor is a lagged copy of the thing you are predicting. Expand the series and m\_t carries every past shock: epsilon\_t, then beta times epsilon\_{t-1}, then beta squared times epsilon\_{t-2}, and so on back. But m\_{t-1} carries the same shocks one step shifted: epsilon\_{t-1}, beta times epsilon\_{t-2}, and so on. The numerator and the denominator of the OLS ratio share these error terms, because both m\_t and m\_{t-1} are built out of epsilon\_{t-1}, epsilon\_{t-2}, and the rest of the history. That shared dependence is exactly what breaks the unbiasedness condition. In data with no autocorrelation the regressor and error are clean, the ratio is unbiased, and there is no issue. The autocorrelation that makes the series interesting is the same autocorrelation that contaminates the estimator. So the OLS estimate picks up an extra term. $$ \\hat{\\beta}\_{\\text{OLS}} = \\beta + \\kappa $$ In the trend case, positive beta, the bias kappa is negative, dragging the estimate down. In the strongly reversion case, negative beta, the bias kappa is positive, dragging the estimate up. Both pulls point the same way: toward zero. The estimator absorbs some of the error structure into the coefficient and reports a beta smaller in magnitude than the truth. ## What the understatement does to you The estimate is closer to zero than reality, so the model you build on it believes the series has less memory than it actually has. That single misbelief has two consequences, and they hit trend and reversion alike. The series converges slower than your model expects. For a trending series, your model expects a shock to decay faster than it truly does, so it underestimates how long a trend persists and exits early. For a reverting series, your model expects the pull back to the mean to happen faster than it truly does, so it underestimates how long the deviation lingers and re-enters too soon. In both cases you have priced in more speed than the process has, because a beta biased toward zero looks like weaker memory, and weaker memory means faster settling. The model underestimates variance. The magnitude of the next move away from the current level, the size of the swing the process can produce, comes out larger than the biased beta predicts. The old article ["Long-Range Dependence: Real Memory or Just Short-Range Echo?"](https://aligrithm.com/long-range-dependence-real-memory-or-just-short-range-echo/) warns about the opposite failure, seeing memory that is not there. This is the mirror: real memory you have measured too small. Underestimating beta's magnitude means underestimating how far the series ranges, so your model predicts weaker reversion than truly happens and shorter trends than truly happen, and it sizes risk against a process calmer than the one you are trading. ## Why this matters before you trade either The practical damage is that both strategy families inherit the same understatement from the same estimator. A trend follower fit with OLS will set its holding period too short and its risk too low. A mean-reversion strategy fit with OLS will set its entry and exit too tight and its variance too low. Neither error announces itself, because the biased beta still fits the in-sample data and still produces a plausible model. The old article ["Parameter Stability Beats Best Parameter"](https://aligrithm.com/parameter-stability-beats-best-parameter/) makes the broader case that the headline estimate is rarely the number to trust, and this is a clean instance: the point estimate of beta is systematically wrong in a known direction, and you can correct for it. The fixes are standard once you know the bias is there. Small-sample bias corrections for the autoregressive coefficient exist and push the estimate back out toward the truth. Resampling the series in blocks, the same block bootstrap the old article ["Variance Ratio Tests for Traders"](https://aligrithm.com/variance-ratio-tests-for-traders/) uses, gives you a distribution for beta that exposes how far the point estimate sits from center. The first move is the recognition: when you regress a series on its own lag, the number that comes back understates the memory, and you are believing in less trend and less reversion than the market actually carries. ## Visualizing the understatement ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_321-visual-0.png) ## KEY POINTS - Trend and mean reversion are the same first-order process, the next value depending on the last, separated only by the sign of beta. Positive beta reinforces and trends; negative beta reverses and mean-reverts. - OLS estimates beta as the covariance of the series with its lag over the variance of the lag. That ratio is unbiased only when the regressor is independent of the error. - The regressor is a lagged copy of the target, so both share the same past shocks. That shared dependence breaks the unbiasedness condition, and only autocorrelated data has it. The autocorrelation that makes the series tradable is what contaminates the estimate. - The estimate comes out as the true beta plus a bias term. For trends the bias is negative, for strong reversion the bias is positive, and both pull the magnitude toward zero. OLS understates the memory. - A beta too close to zero looks like weaker memory, so the model expects faster settling: trends exit too early, reversion re-enters too soon, and the series actually converges slower than predicted. - The same bias makes the model underestimate variance, predicting weaker reversion and shorter trends than reality and sizing risk against a calmer process. Correct it with small-sample bias adjustments and a block bootstrap, and recognize the understatement before trading either family. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Testing and Tuning Market Trading Systems - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Testing+and+Tuning+Market+Trading+Systems+Timothy+Masters&ref=aligrithm.com) - [Data Mining Algorithms in C++ - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Data+Mining+Algorithms+in+C%2B%2B+Timothy+Masters&ref=aligrithm.com) ### 2.81 Volume and Volatility Are the Same Feature URL: https://aligrithm.com/volume-and-volatility-are-the-same-feature/ Last updated: 2026-07-27T13:23:39.000Z Take an alpha you trust where momentum sits over a volatility denominator. Drop volume into that denominator instead and rerun the backtest. The two equity curves land almost on top of each other. Run it again on a different alpha and a different instrument and the pattern holds: a surprising number of formulas survive the substitution with barely a scratch on the Sharpe. They survive because volume and volatility are close to the same measurement wearing two labels, and a model that takes both as independent inputs is counting one effect twice. This is a small point with a sticky consequence. The old article ["Why ATR Normalization Is More Than a Volatility Trick"](https://aligrithm.com/why-atr-normalization-is-more-than-a-volatility-trick/) leaned on volatility as the structurally correct denominator for price-unit features, and the old article ["The Price Change Oscillator"](https://aligrithm.com/the-price-change-oscillator/) built a feature out of absolute movement. Both quietly assume volatility is its own thing. For most of what you build, volume would have done a near-identical job, and that overlap is the trap. ## Why they move together Both volume and volatility are driven by the same underlying clock: the rate at which information arrives and forces participants to act. On a quiet bar few traders need to do anything, so you get little volume and small moves. On a busy bar information hits, everyone repositions at once, and you get heavy volume and a large absolute move in the same breath. The mixture-of-distributions idea formalizes it: returns and volume are both subordinated to a latent information-flow process, so they rise and fall together rather than independently. $$ |r\_t| \\;\\propto\\; \\sqrt{I\_t}, \\qquad V\_t \\;\\propto\\; I\_t \\qquad\\Longrightarrow\\qquad |r\_t| \\;\\propto\\; \\sqrt{V\_t} $$ Read it as: the absolute return on a bar scales with the square root of the information arriving that bar, trading volume scales with that same information count, so absolute return scales with the square root of volume. The constant differs by instrument, but the link is mechanical, not coincidental, and it shows up as a high positive correlation between absolute returns and volume in almost every liquid market you measure. ## The double-counting trap Feed volume and volatility into the same model as if they were two separate signals and you have put the same factor in twice. In a linear model this is textbook collinearity: the two columns are nearly the same vector, the regression cannot tell which one earns the coefficient, and it splits the weight between them in an unstable way that flips sign across refits. You think you have two predictors holding the model up. You have one predictor leaning on itself. The damage is quieter in an ensemble. The old turnover-reduction logic, averaging many variants of the same alpha so the correlated signal survives and the uncorrelated noise cancels, assumes the variants share signal but not noise. Stack a volume-normalized momentum and a volatility-normalized momentum into that average and you have not added breadth. You have doubled the weight on one effect and fooled yourself into thinking the ensemble is more diversified than it is. Check the correlation before you treat two features as independent, or the regularization and the averaging both silently misprice them. ```python import numpy as np import pandas as pd # bars with log returns and volume r = np.log(df["close"]).diff() abs_ret = r.abs() vol = df["volume"] # the overlap: absolute return vs volume, and vs sqrt(volume) print("corr(|r|, V) :", abs_ret.corr(vol)) print("corr(|r|, sqrt V) :", abs_ret.corr(np.sqrt(vol))) # the orthogonal residual that is NOT shared: Amihud illiquidity illiq = abs_ret / vol # move per unit of volume ``` The first two correlations come back high and positive on liquid data, which is the whole point: the absolute move and the volume are mostly the same column. The interesting line is the last one. ## The residual is where the real signal lives Correlated is not identical, and the gap between them is the feature worth keeping. The part of volume that is orthogonal to volatility, big volume with a small price move, is absorption: someone soaked up a flood of trading without letting price run. The mirror case, a large move on thin volume, is a fragile book that gapped on almost nothing. Neither of those shows up if you carry volume and volatility as redundant twins. Both show up the instant you look at their ratio. $$ \\text{ILLIQ}\_t = \\frac{|r\_t|}{V\_t} $$ Read it as: Amihud illiquidity is the absolute return divided by volume, the price move you got per unit of volume traded. Because the numerator and denominator carry the shared information-flow component, dividing one by the other cancels most of it and leaves the residual: how much this market moves for a given amount of trading, which is liquidity itself. That residual is genuinely new information next to plain momentum, where two raw copies of the volatility regime are not. The practical rule that falls out: pick one of volume or volatility as your scale, the way the old ATR-normalization article picked ATR, then add their ratio as a second feature rather than adding the other raw level. One regime column plus one liquidity column beats two regime columns pretending to be different. ## Visualizing the overlap ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_328-visual-0.png) ## KEY POINTS - Drop volume in for the volatility denominator in most alpha formulas and the backtest barely changes, because volume and absolute returns are both driven by the same information-arrival clock and move together. - The mixture-of-distributions link is mechanical: absolute return scales with the square root of volume, so the two carry a high positive correlation in liquid markets. - Feeding both into one model double-counts the factor: collinearity destabilizes a linear model's coefficients, and stacking volume- and volatility-normalized variants into an ensemble fakes diversification the old signal-averaging logic assumed it had. - Check the correlation before treating two features as independent, or both regularization and ensemble averaging silently misprice them. - The residual is the keeper: the part of volume orthogonal to volatility is absorption (big volume, small move) or fragility (big move, thin volume), neither visible if you carry the two as redundant twins. - Amihud illiquidity, |r| over volume, cancels the shared component and leaves liquidity; pick one scale the way the old ATR-normalization article picked ATR, then add their ratio rather than the other raw level. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Statistically Sound Indicators for Financial Market Prediction - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Statistically+Sound+Indicators+for+Financial+Market+Prediction+Timothy+Masters&ref=aligrithm.com) - [Cycle Analytics for Traders - John Ehlers (Amazon)](https://www.amazon.com/s?k=Cycle+Analytics+for+Traders+John+Ehlers&ref=aligrithm.com) ### 2.80 Ridge Above 1h, XGBoost Below 5min URL: https://aligrithm.com/ridge-above-1h-xgboost-below-5min/ Last updated: 2026-07-27T13:23:38.000Z Pick the model by how strong the interactions are, not by fashion. At longer, near-additive horizons, ridge regression is the right tool. At shorter, interaction-dominated horizons, gradient-boosted trees win, and they win by enough to matter. The crossover is not a fixed clock time you read off a tweet; it is the horizon where the conditional structure starts to dominate the additive structure, and which side of it you are on should decide your model before you tune a single hyperparameter. The reason is the same conditional structure the rest of this arc has been circling. The old article ["The Limits of Linear Models"](https://aligrithm.com/the-limits-of-linear-models/) showed that signal relationships are interactions: momentum pays in low-vol clean-trend regimes, carry crashes when vol spikes. Those interactions are strongest at the shortest horizons, where the next move depends on the regime you are sitting in. As you stretch the horizon out, the regime conditioning washes out, the surviving signal flattens into something close to additive, and a regularized linear model captures it with less variance than a tree ever could. ## The thing that decides it Score the forecasts with the information coefficient, the correlation between the forecast and the realized return. It is the standard yardstick because it survives the near-zero signal-to-noise the old article ["Why Volatility Is More Non-Stationary Than Trend"](https://aligrithm.com/why-volatility-is-more-non-stationary-than-trend/) describes: you are not trying to nail the return, you are trying to be correlated with it across many predictions. Where the signal is near-additive, ridge and a tuned XGBoost land in roughly the same place on IC, so you take the simpler, cheaper, more stable model. Where interactions dominate, the tree pulls meaningfully ahead, and it needs careful tuning to do it. The point is not a precise IC delta you can quote; it is the direction of the gap. $$ \\text{IC}\_{\\text{tree}} > \\text{IC}\_{\\text{ridge}} \\;\\;\\text{(interactions dominate)} \\qquad \\text{IC}\_{\\text{tree}} \\approx \\text{IC}\_{\\text{ridge}} \\;\\;\\text{(near-additive)} $$ The edge, when it shows up, is paid purely for using a better model on the same features. Call it a few IC points if you need an illustrative figure, but treat that as a stand-in, not a promise; the real number depends on your features and horizon. The tree is not seeing different data. It is extracting the interactions out of the same columns that ridge has to treat additively, and at the horizon where interactions dominate, that extraction is the whole game. ## XGBoost eats NaNs, and that is not cosmetic The second reason to reach for XGBoost at short horizons has nothing to do with accuracy on clean data. It handles missing values natively, and missing values are the normal state of a real feature set. Your features come from different sources with different histories. One data provider has ten years of a given feed, another has two. One exchange feed has a gap. A feature with a long lookback is undefined for the first part of any window. So at any given row, some features are present and some are NaN, and which ones are missing changes over time. A linear model cannot multiply a weight by a NaN. You are forced to do something about the holes before the model runs. The obvious move is imputation: fill each NaN with the feature's mean or median. That move has a subtle lookahead the old article ["How to Test Indicator Thresholds Without Fooling Yourself"](https://aligrithm.com/how-to-test-indicator-thresholds-without-fooling-yourself/) would flag on sight. The mean or median is computed over data that includes the future relative to the row you are filling, so you are leaking the feature's future distribution into a past prediction. You can compute a lookahead-free running mean instead, expanding-window only, and it is better, and it still bothers me, because the running statistic is unstable early and you have baked an estimate into the feature where there was no information. XGBoost sidesteps all of it. At each split, it learns a default direction for missing values, sending NaN rows down whichever branch improves the fit, treating "this feature is absent" as information rather than a hole to plug. No imputation, no leak, no fabricated values. For a feature set stitched from uneven sources, that alone can justify the tree at any horizon, and it is the unglamorous reason boosting keeps winning in production. ## When you are stuck with linear and have NaNs At longer horizons you often cannot afford the tree, not because of accuracy but because of fitting cost, retraining cadence, or interpretability requirements. You still have the NaN problem, and imputation still leaks. The clean answer is an IC-weighted ensemble of single-feature or small-group forecasts, with the weights recomputed whenever the set of available features changes. Score each feature by its own information coefficient, weight each feature's contribution by its IC, and combine. $$ \\text{forecast} = \\frac{\\sum\_{i \\in \\text{available}} \\text{IC}\_i \\cdot x\_i}{\\sum\_{i \\in \\text{available}} \\lvert \\text{IC}\_i \\rvert} $$ When a feature goes missing, drop it from both sums and renormalize. No imputation, because an absent feature just leaves the weighted average instead of getting filled with a guess. Recomputing the weights is cheap: you already have each feature's IC and you know which features are in the set, so the new weights are one pass of arithmetic. This gives you a linear model that degrades gracefully as features come and go, which is the behavior you wanted from imputation without the lookahead it smuggles in. ## Pick the model from the horizon first The decision order matters. Settle the horizon, judge whether the signal is near-additive or interaction-dominated, then the model class follows, then you tune. At the longer, near-additive horizons, start with ridge, and reach for the IC-weighted ensemble when missing features force your hand. At the shorter, interaction-dominated horizons, start with XGBoost and budget the time to tune it, because the interaction edge and the native NaN handling are both real there. The old article ["From One Tree to Forests to Boosting"](https://aligrithm.com/from-one-tree-to-forests-to-boosting/) covers why boosting in particular, and the caution from ["Linear Models' Hidden Symmetry Advantage"](https://aligrithm.com/linear-models-hidden-symmetry-advantage/) still applies: the tree gives up the long-short symmetry the linear model held for free, so engineer it back with mirrored data before you trust the short-horizon model in production. ## Visualizing the crossover ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_320-visual-0.png) ## KEY POINTS - Choose the model by how strong the interactions are. Ridge regression where the signal is near-additive, gradient-boosted trees where interactions dominate. The crossover is a property of the signal, not a clock time, and it should decide the model before any tuning. - Signal relationships are interactions, and interactions dominate at the shortest horizons where the next move depends on the regime. Stretch the horizon out and the conditioning washes out into something close to additive, which ridge captures with less variance. - Score with the information coefficient, the correlation of forecast to realized return. Where the signal is near-additive ridge and XGBoost tie on IC, so take the simpler model. Where interactions dominate the tree pulls meaningfully ahead on the same features. Any IC figure is illustrative, not a promise. - XGBoost handles missing values natively, learning a default split direction for NaN rows. Feature sets stitched from uneven data sources are full of NaNs, so this is a production reason to use it, separate from accuracy. - Imputing NaNs with a mean or median leaks the future into the past. A lookahead-free running mean is better and still bakes an unstable estimate into the feature where there was no information. - When you must stay linear and still face NaNs, use an IC-weighted ensemble: weight each available feature by its IC, drop missing features, and renormalize. It degrades gracefully with no imputation and the weights recompute in one cheap pass. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Statistically Sound Indicators for Financial Market Prediction - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Statistically+Sound+Indicators+for+Financial+Market+Prediction+Timothy+Masters&ref=aligrithm.com) - [Cycle Analytics for Traders - John Ehlers (Amazon)](https://www.amazon.com/s?k=Cycle+Analytics+for+Traders+John+Ehlers&ref=aligrithm.com) ### 2.79 How a Decision Tree Engineers a New Alpha URL: https://aligrithm.com/how-a-decision-tree-engineers-a-new-alpha/ Last updated: 2026-07-27T13:23:37.000Z The old article ["The Limits of Linear Models"](https://aligrithm.com/the-limits-of-linear-models/) ended on a rule instead of an equation: if realized vol is high and the trend is weak, fade momentum and lean mean-reversion. That rule is a decision tree with one split. This article builds the machine that finds rules like it on its own, because hand-writing them does not scale. You have momentum, value, realized vol, dispersion, trend strength, and a dozen more, and the conditional combinations run into the thousands. You need a procedure that mines the conditions for you. Start with the training data. Each row is a moment in time: the values of every signal at that instant, paired with the future return that followed. The target is the future return, the features are the cross-sectional signals, and you have millions of rows. The job is to carve this data into regions where the future return behaves the same way inside each region. ## The split is a question that sorts the data A tree asks one yes-or-no question and splits the rows into two groups by the answer. ``` realized_vol > 80th pct ? momentum < 0 ? ``` Each question puts some rows on the left and the rest on the right. A good question puts rows with similar future returns together: outperformers on one side, underperformers on the other. A useless question scatters them evenly across both sides, so neither group tells you anything. The whole game is finding the question that separates best, then repeating it on each of the two groups, and again on their children, until you stop. This is the same instinct as fitting a linear regression, where you pick the line that minimizes squared error. A tree picks the split that minimizes squared error too, just within the groups instead of along a line. The standard CART formulation writes this as maximizing a similarity gain, which is the same arithmetic seen from the other side, and it is worth writing out because the formula is the entire engine. ## The similarity score and the gain Score a group of rows by how spread out their future returns are. Take each row's return, subtract the group's mean return, square the difference, sum across the rows, and divide by the count. $$ \\text{similarity(node)} = \\frac{\\sum\_{i \\in \\text{node}} \\big(y\_i - \\bar{y}\\big)^2}{n\_{\\text{node}}} $$ A low score means the returns in that group sit tight around their mean, so the group is internally consistent and its mean is a confident prediction. A high score means the returns are all over the place and the mean tells you little. You want splits that drive this score down in the children. Measure that with the gain: the parent's score minus the sum of the two children's scores. $$ \\text{gain} = \\text{similarity(parent)} - \\big(\\text{similarity(left)} + \\text{similarity(right)}\\big) $$ To pick the split at a node, the tree scans every feature and every candidate threshold on that feature, computes the gain for each, and keeps the one with the largest gain. "Momentum below zero" wins if cutting the data there leaves outperformers piled on one side and underperformers on the other better than any other cut. Then the procedure recurses: each child node runs the same scan on its own rows and finds its own best split, building the tree one question at a time. ## The prediction is a flat number per region When a node stops splitting, it becomes a leaf, and the leaf predicts the mean future return of the rows that landed in it. $$ \\text{prediction(leaf)} = \\bar{y}\_{\\text{leaf}} = \\frac{1}{n\_{\\text{leaf}}} \\sum\_{i \\in \\text{leaf}} y\_i $$ Trace a new market state down the tree, answering each question, and you land in one leaf and read off its mean. That number is the forecast. Notice what the tree has built: it has carved the feature space into boxes, and inside each box it predicts a single constant. "When realized vol is high and the trend is weak and momentum is negative, the next return averages slightly negative" is one box. The tree discovered that conjunction by splitting, which is the interaction the linear model could not represent. This is the sense in which a tree engineers a new alpha. It manufactures conditional signals out of raw features by finding the conditions where the raw features mean something. ## Three problems this opens, on purpose The construction is simple, and the simplicity hides three questions that decide whether the tree makes money or fits noise. When do you stop splitting? Keep splitting and you can drive every leaf down to a single row, where the similarity score is zero and the prediction is that one row's return. That tree has memorized the training data and learned nothing, the textbook overfit. You stop early, with a minimum number of rows per leaf or a maximum depth or a minimum gain to bother splitting, and where you set those knobs is the same plateau-versus-peak choice the old article ["Parameter Stability Beats Best Parameter"](https://aligrithm.com/parameter-stability-beats-best-parameter/) makes for any parameter. The old article ["How to Test Indicator Thresholds Without Fooling Yourself"](https://aligrithm.com/how-to-test-indicator-thresholds-without-fooling-yourself/) applies directly too, because a tree's split search is a threshold search run thousands of times, and an unconstrained one finds thresholds that work in-sample and die out. Why are the predictions discrete and bounded? A leaf predicts an average of training returns, so the tree can only ever output one of a finite set of leaf means, and none of them can exceed the most extreme average it saw. A linear model extrapolates past its training range; a tree cannot. Hand it a state more extreme than anything in the data and it returns the nearest leaf's tame average, which caps the upside and the downside of the forecast. How do you stop one tree from being fragile? A single tree's early splits are chosen from noisy data, and a small change in the sample can flip the top split and rebuild the whole tree differently. That instability is the subject of the next step in this arc, ["From One Tree to Forests to Boosting,"](https://aligrithm.com/from-one-tree-to-forests-to-boosting/) which trades one fragile tree for many and gets a stable forecast back. The long-short symmetry a single tree breaks, the subject of ["Linear Models' Hidden Symmetry Advantage,"](https://aligrithm.com/linear-models-hidden-symmetry-advantage/) has to be engineered back here too, because each split learns its threshold from one side of the mirror. ## Visualizing the carve ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_319-visual-0.png) ## KEY POINTS - A decision tree carves the training data into regions by asking yes-or-no questions on features, so that future returns inside each region behave alike. Each question is a split like "momentum below zero?". - Score a group by how tightly its returns cluster around their mean: the summed squared deviations divided by the count. Low score means a confident prediction, high score means the mean tells you little. - Pick the split with the largest gain, the parent's score minus the two children's scores. The tree scans every feature and threshold, keeps the best cut, then recurses on each child. - A leaf predicts the mean future return of the rows in it. The tree carves feature space into boxes and predicts a constant per box, which is the interaction a linear model could not represent. That is how a tree engineers a conditional alpha. - Stopping is the overfitting control. Split all the way and every leaf memorizes one row. Stop early with minimum leaf size, max depth, or a minimum gain, and choose those knobs on a stable plateau, not an in-sample peak. - Tree predictions are discrete and bounded: only leaf means, none more extreme than the most extreme average seen. A tree does not extrapolate. A single tree is also fragile, which the move to forests and boosting fixes. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Statistically Sound Indicators for Financial Market Prediction - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Statistically+Sound+Indicators+for+Financial+Market+Prediction+Timothy+Masters&ref=aligrithm.com) - [Cycle Analytics for Traders - John Ehlers (Amazon)](https://www.amazon.com/s?k=Cycle+Analytics+for+Traders+John+Ehlers&ref=aligrithm.com) ### 2.78 From One Tree to Forests to Boosting URL: https://aligrithm.com/from-one-tree-to-forests-to-boosting/ Last updated: 2026-07-27T13:23:37.000Z The old article ["How a Decision Tree Engineers a New Alpha"](https://aligrithm.com/how-a-decision-tree-engineers-a-new-alpha/) built a single tree that mines conditional signals out of cross-sectional features. It works, and it has one fault that sinks it on its own: it is fragile. Change the training sample a little, drop a few thousand rows or shift the window a month, and the top split can flip from a vol-regime cut to a momentum cut, and once the top split moves, every split beneath it is chosen on different data, so the whole tree rebuilds into a different model. A forecast that swings that much on a small data change is fitting the sample, not the market. The old article ["Why Volatility Is More Non-Stationary Than Trend"](https://aligrithm.com/why-volatility-is-more-non-stationary-than-trend/) frames why this bites so hard: the signal-to-noise ratio is brutal, and a high-variance model trained on it locks onto the noise. The fix is the same one that stabilizes a noisy linear model. Train many models on resampled data and average them. ## Bagging, borrowed from the linear case Bagging started as a trick for linear models, and it is worth seeing there first because the logic carries over unchanged. Take the noisy three-feature model from the old article ["Feature Engineering Before Machine Learning"](https://aligrithm.com/feature-engineering-before-machine-learning/) and suppose you want a stable live prediction instead of the best in-sample fit. You could split the history into pieces and fit a model on each, but each piece is too small to fit well. So you manufacture more samples instead. Draw rows from your history at random, with replacement, until you have a new dataset the size of the original. Some rows appear twice, some not at all. This bootstrap sample stands in for "going back out and collecting fresh data," which is what you would do if you could. Fit a model on it. Do it again on a fresh draw, and again, and you get a stack of fitted models. $$ \\hat{f}\_1, \\; \\hat{f}\_2, \\; \\dots, \\; \\hat{f}\_B \\quad\\Longrightarrow\\quad \\hat{f}\_{\\text{bag}}(x) = \\frac{1}{B}\\sum\_{b=1}^{B} \\hat{f}\_b(x) $$ Each model carries the same average error, the same bias, so averaging them does not fix bias. What averaging fixes is variance. The models disagree because each saw a different resample, and averaging their disagreement cancels a chunk of it, the way the old article on factors and regularization leans on shrinkage to stabilize estimates. You trade a sharper in-sample fit for a forecast that barely moves when the data wobbles. ## Bagging trees gives a random forest Run the same procedure on trees and you get a random forest. Each tree trains on its own bootstrap sample, so each tree's fragile top split is chosen from a different draw, and where one tree splits on the vol regime another splits on momentum. Average their predictions and the per-tree fragility washes out, because the thing that made one tree swing, its sensitivity to the exact sample, is exactly what averaging across many samples removes. ``` data |-- bootstrap #1 --> tree T_1 -. |-- bootstrap #2 --> tree T_2 |--> average --> predicted return |-- bootstrap #3 --> tree T_3 -' '-- ... --> tree T_B ``` Forests add one more shuffle: at each split, a tree is allowed to consider only a random subset of the features. That stops every tree from latching onto the single strongest feature and making the trees near-copies of each other. The old article ["The Death of the Single-System Trader"](https://aligrithm.com/the-death-of-the-single-system-trader/) makes the portfolio version of this point, that correlated components give you almost no diversification, and it applies here unchanged. Trees that all split on momentum first are correlated and average to nothing better than one tree. Forcing feature subsets decorrelates them, and decorrelated trees average to a steadier forecast. ## Boosting fits the next tree to the leftover error Step back and bagging looks wasteful. Each tree starts from scratch and takes its own swing at the full target, then you throw away their individual structure by averaging. Every tree redoes work the others already did. What if each new tree only worked on the part the current model still gets wrong? Start with the simplest possible forecast, the overall mean of the target. $$ \\text{pred}\_0 = \\overline{y} $$ Compute what that leaves on the table, the residuals, which are the actual returns minus the current prediction. Fit a tree to those residuals, not to the original target. That tree learns the structure the current model missed. Add a shrunken piece of it to the running prediction, and repeat. $$ \\text{pred}\_n = \\text{pred}\_{n-1} + \\eta \\cdot T\_n\\big(\\,y - \\text{pred}\_{n-1}\\,\\big) $$ Each tree corrects the last one's mistakes, and the running sum creeps toward a good forecast one small step at a time. This is boosting, and the popular implementations are LightGBM and XGBoost. The learning rate eta is the step size, and it does real work: a small eta takes tiny steps and needs many trees but generalizes better, which is the same plateau-over-peak discipline the old article ["Parameter Stability Beats Best Parameter"](https://aligrithm.com/parameter-stability-beats-best-parameter/) argues for everywhere. Too large a step and the model chases the residual noise, which at this signal-to-noise is most of the residual. ## Bagging and boosting are not the same medicine The two reduce different errors, and confusing them costs you. Bagging trains its trees independently and in parallel, all on the same target, and averages to cut variance. It rarely overfits as you add trees, because more trees just average more. Boosting trains its trees in sequence, each on the previous residual, and sums them to cut bias by grinding down the error the model has not yet explained. It will overfit as you add trees, because eventually it starts fitting the noise in the residual, so the number of rounds is a parameter you have to control with the same out-of-sample care you give any threshold. Where interactions dominate, boosting usually wins on accuracy, which is why the next article in this arc, ["Ridge Above 1h, XGBoost Below 5min,"](https://aligrithm.com/ridge-above-1h-xgboost-below-5min/) ends up recommending it at the shorter, interaction-dominated horizons. It wins because the conditional structure of cross-sectional returns is deep there, and stage-by-stage residual fitting digs into that structure further than averaging independent trees does. It costs more tuning and more discipline to keep from overfitting, and it still breaks the long-short symmetry that the old article ["Linear Models' Hidden Symmetry Advantage"](https://aligrithm.com/linear-models-hidden-symmetry-advantage/) says you have to engineer back in, because every tree in the ensemble inherits that flaw from the single tree underneath it. ## Visualizing the two ensembles ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_318-visual-0.png) ## KEY POINTS - A single decision tree is fragile. A small change in the training sample can flip the top split and rebuild the whole tree, so the forecast swings on noise. At this signal-to-noise this is fatal. - Bagging stabilizes it. Train many models on bootstrap resamples, drawn with replacement, and average them. Averaging leaves bias alone and cuts variance, trading in-sample sharpness for a forecast that barely moves when the data wobbles. - Bagging trees gives a random forest. Each tree trains on its own resample, and at each split a tree sees only a random subset of features, which decorrelates the trees so their average actually helps. Correlated trees average to nothing better than one. - Boosting takes a different route. Start at the mean, fit each new tree to the current residual, and add a shrunken piece. The running sum corrects its own mistakes step by step. The implementations are LightGBM and XGBoost. - The learning rate eta is the step size. Small steps need more trees and generalize better; large steps chase residual noise. The number of boosting rounds overfits if uncontrolled, unlike bagging. - Bagging cuts variance in parallel, boosting cuts bias in sequence. Boosting usually wins on accuracy where interactions dominate because the conditional structure of cross-sectional returns is deep, at the cost of more tuning. Every tree in either ensemble still breaks long-short symmetry. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Statistically Sound Indicators for Financial Market Prediction - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Statistically+Sound+Indicators+for+Financial+Market+Prediction+Timothy+Masters&ref=aligrithm.com) - [Cycle Analytics for Traders - John Ehlers (Amazon)](https://www.amazon.com/s?k=Cycle+Analytics+for+Traders+John+Ehlers&ref=aligrithm.com) ### 2.77 Linear Models' Hidden Symmetry Advantage URL: https://aligrithm.com/linear-models-hidden-symmetry-advantage/ Last updated: 2026-07-27T13:23:36.000Z Take every position in a cross-sectional book and flip it. Every long becomes a short, every short becomes a long. The winners you were buying turn into names you sell, a positive momentum reading turns negative, the whole signed feature set reverses. You have built the mirror world, the same strategy reflected across the line between long and short. Ask what a correct return model should do in that mirror world. If the original model predicted a name was about to outperform, the mirrored model should predict it underperforms by the same amount. The magnitude stays, the sign flips. $$ \\text{pred\\\_return} \\;\\longrightarrow\\; -\\,\\text{pred\\\_return} $$ This is not a nice-to-have. It is a law of the trading world. Going long and going short are symmetric operations on a return, so a model that prices the next move must respect the symmetry, or it is making different claims about winners and losers for no reason the market gives it. The old article ["The Limits of Linear Models"](https://aligrithm.com/the-limits-of-linear-models/) spent its length attacking linear models for being unable to capture interactions. This article pays the debt back. The same additivity that makes them blind to interactions hands them this symmetry for free, and the model people are running to instead, the tree, does not get it. ## Why the linear model gets it free Write the linear prediction as the weighted sum it is. Every feature is centered and linearly correlated with the return, so the prediction is the dot product of weights and features. $$ \\text{pred} = \\sum\_i b\_i \\, x\_i $$ Now flip the world. Mirroring sends every feature to its negative, because momentum, carry, value, and the rest are all signed quantities that reverse when you relabel every long as a short. Feed the negated features through the same fitted weights. $$ \\sum\_i b\_i \\,(-x\_i) = -\\sum\_i b\_i \\, x\_i = -\\,\\text{pred} $$ The prediction flips sign on its own, exactly as the trading world demands, and you did nothing. You did not retrain, you did not add a constraint, you did not check it. The structure of the model guarantees it. A model with one set of weights describes the long-case and the short-case with the same parameters, so it cannot get one right and the other wrong. It treats them as the single symmetric thing they are. ## Why the tree does not A tree makes its decisions by splitting on feature values. It learns a rule like "is momentum above alpha?" and assigns a prediction to each side. $$ x\_i > \\alpha \\;? $$ In the mirror world, the feature that was x\_i is now its negative, and the split that should fire is the reflection of the original. $$ x\_i < -\\alpha \\;? $$ A linear model already knew this, because the negative sign carried through the weight. A tree knows nothing of the sort. The split at alpha and the split at minus alpha are two separate questions to a tree, learned from two separate subsets of the data, fit independently with their own samples and their own noise. The tree has to discover both halves of the symmetry on its own, and it has half the data to learn each half with. That doubles the surface for error. Every place the linear model had one parameter doing symmetric duty, the tree has two estimates that should be mirror images and usually are not, because each was fit on a different slice of a low signal-to-noise sample. The tree can decide momentum above alpha predicts a strong outperformance while momentum below minus alpha predicts a weak underperformance, an asymmetry the market does not contain and the model invented out of sampling noise. In a domain where the old article ["Feature Engineering Before Machine Learning"](https://aligrithm.com/feature-engineering-before-machine-learning/) pegs the real signal at a thin slice of total return variance, inventing structure that is not there is exactly the failure you cannot afford. ## You have to engineer the symmetry back in The tree's flexibility is the reason you reach for it. The previous article in this arc made that case: cross-sectional returns are conditional, and a tree conditions natively where a linear model is stuck adding. You do not give that up. You pay for it by re-engineering the symmetry the linear model gave you free. The clean way is data augmentation. For every training example, add its mirror: negate the signed features and negate the target. The tree now sees both halves of every situation, the long-case and its reflected short-case, in equal measure, so it has no excuse to fit them asymmetrically. You have doubled the effective sample for learning each symmetric rule and removed the source of the invented asymmetry. The cost is real, more data to carry and a constraint to enforce on every feature you add, since you have to know which features are signed and flip them correctly. Get the sign bookkeeping wrong on one feature and you have taught the model a lie. This is the trade the whole arc keeps returning to. The linear model is rigid, and the rigidity buys you a guarantee. The tree is flexible, and the flexibility costs you the guarantee unless you rebuild it by hand. Neither is free. The mistake is reaching for the tree because it is more powerful and forgetting that some of the linear model's power came from what it could not do. ## Visualizing the mirror ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_317-visual-0.png) ## KEY POINTS - Flip every long and short in a book and a correct return model should flip the sign of its prediction while keeping the magnitude. Long-short symmetry is a law of the trading world, not an option. - A linear model satisfies it for free. Because the prediction is a weighted sum of signed features, negating the features negates the prediction through the same weights, with no retraining and no constraint. - A tree does not. Its split "is momentum above alpha?" and the mirror split "is momentum below minus alpha?" are two questions learned independently from two halves of the data, so they rarely come out as mirror images. - The asymmetry a tree invents is sampling noise dressed as structure. In a near-zero signal-to-noise domain, inventing structure that the market does not contain is a direct path to overfitting. - Recover the symmetry by augmenting the data: for every training row, add its mirror with signed features and target negated. The tree then sees both halves equally and cannot fit them asymmetrically. - The cost is more data and careful sign bookkeeping on every signed feature. The arc's recurring lesson holds: the tree's flexibility is real, and part of the linear model's strength was a guarantee that came from what it could not do. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Statistically Sound Indicators for Financial Market Prediction - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Statistically+Sound+Indicators+for+Financial+Market+Prediction+Timothy+Masters&ref=aligrithm.com) - [Cycle Analytics for Traders - John Ehlers (Amazon)](https://www.amazon.com/s?k=Cycle+Analytics+for+Traders+John+Ehlers&ref=aligrithm.com) ### 1.26 Large Trades Are Insider Trades by Definition URL: https://aligrithm.com/large-trades-are-insider-trades-by-definition/ Last updated: 2026-07-27T13:23:01.000Z A large market order moves the price whether or not the trader knows anything. It takes out the resting bids or offers, the book reprices around the hole it punched, and by the time the order finishes it has manufactured the very move it looked like it predicted. The trader did not need a secret. Size alone did the work. So the clean question is not whether a big trade was informed. It is whether the distinction even survives contact with the order book, and mostly it does not. This piece sits at the seam between markets philosophy and microstructure, because the argument is half definitional and half mechanical. The mechanics come from the old article ["Market Impact and the Square-Root Law: Walking the Book to Price Your Slippage."](https://aligrithm.com/market-impact-and-the-square-root-law-walking-the-book-to-price-your-slippage/) The philosophy is what those mechanics imply about who counts as an insider. ## Size moves price by construction The old square-root article gave the shape. Walk a real order book with an order of size x and the slippage you eat follows a power law, impact equal to a times x to the power b, with b landing near one-half. The constant is the instrument's sensitivity, but the part that matters here is that the impact is non-zero for any order big enough to clear more than the top level. $$ \\text{impact}(x) = a \\cdot x^{\\,b}, \\quad b \\approx 0.5 \\qquad\\Longrightarrow\\qquad \\text{impact}(x) > 0 \\text{ for large } x $$ Read it as: impact scales with order size raised to a power near one-half, and for a large order that quantity is strictly positive. There is no size large enough to be informed and small enough to be free. To move size, you move the price. The trade is not predicting the move. It is the move. ## Why that makes the trader quasi-informed by definition Size has a cost of entry. Committing real capital takes deep pockets and a view hard enough to risk them, because no one stakes that much on a thesis they hold loosely. From the tape's side, a participant with capital and a firm view behaves like someone trading on private information, and the book cannot separate the two while the order is live. Both leave the same footprint: a sweep through several levels and a book that reprices and does not snap all the way back. The permanent part of that footprint is the tell. Split any impact into the piece that reverts and the piece that sticks. $$ \\Delta P = \\underbrace{\\text{temporary}}\_{\\text{reverts as book refills}} \\;+\\; \\underbrace{\\text{permanent}}\_{\\text{the market's belief revision}} $$ Read it as: the price change from a large trade has a temporary component that fades as makers refill the levels you ate, and a permanent component that the rest of the market keeps as a revised belief about value. The permanent component is information by behavior, regardless of intent. If a whale buys a billion because of genuine conviction or because of a forced short cover, the market still reads the residual as a signal and adjusts, because there is no way to query the whale's reasons and the safe assumption is that money that large knows something. Treating large flow as uninformed is the expensive assumption. Treating it as informed is the cheap one. ## The maker's seat makes this concrete From the market maker's chair this stops being philosophy. The old article ["Using Trade Flow to Predict Short-Term Price Movement"](https://aligrithm.com/using-trade-flow-to-predict-short-term-price-movement/) built the signal: sign each trade by aggressor side, sum it over a decaying window, and the net signed flow predicts the next move because taker orders are autocorrelated, buys follow buys. Large flow is the strongest, most autocorrelated, hardest-to-fake version of that signal. You cannot spoof a trade that actually printed. So a maker who gets run over by size does not waste time wondering whether the taker was a genuine insider. The maker skews the quote away from the flow and widens, because pricing large flow as toxic is the only assumption that survives a P&L review. Adverse selection does not care about the counterparty's motives. It cares about the footprint, and a large trade leaves the footprint of someone who knows. The market's revealed definition of an insider is whoever moves the price and makes it stick, and large traders do that by construction. ## The caution: forced is not the same as informed The argument has a real failure mode, and ignoring it costs money. Large does not always mean smart. A liquidation, a margin call, an index rebalance, or a fund unwinding a book all generate enormous signed flow with zero predictive information, and the permanent-versus-temporary split is exactly how you tell them apart. Forced flow tends to be mostly temporary impact: a violent move that reverts once the forced seller is done, because there was never a belief revision underneath it. Informed flow leaves a permanent mark because the rest of the market updates. This also ties straight back to the old article ["Who Is the Marginal Buyer?"](https://aligrithm.com/who-is-the-marginal-buyer/) A large buyer is not just reacting to the marginal price, they are setting it, becoming the marginal buyer for every unit they sweep. That is the same coin from the other face: the marginal trade sets the price, and a large trade is a marginal trade with enough size to drag the print somewhere new and leave it there. Follow the permanent component, not the headline size, and you are following information rather than someone else's forced exit. ## Visualizing the large trade as information ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_327-visual-0.png) ## KEY POINTS - A large market order moves price by construction: it clears multiple book levels, so its impact (a·x^b with b near 0.5, from the old square-root-law article) is strictly positive for any real size. The trade is the move, not a forecast of it. - Putting real size on takes capital and a firm view, and from the tape's side that behaves like trading on information; the market cannot tell a true insider from a large uninformed trader in real time because both leave the same footprint. - Split impact into temporary (reverts as makers refill) and permanent (the market's belief revision); the permanent component is information by behavior, regardless of the trader's intent. - From the maker's seat this is forced: large signed flow is the strongest, unspoofable version of the trade-flow signal from the old article ["Using Trade Flow to Predict Short-Term Price Movement,"](https://aligrithm.com/using-trade-flow-to-predict-short-term-price-movement/) so makers skew away and price it as toxic. - The failure mode is forced flow: liquidations, margin calls, and rebalances move size with no information, and they show up as mostly temporary impact that reverts. - A large trade is a marginal trade with enough size to reset the print, the other face of the old article ["Who Is the Marginal Buyer?"](https://aligrithm.com/who-is-the-marginal-buyer/); follow the permanent impact, not the headline size. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Evidence-Based Technical Analysis - David Aronson (Amazon)](https://www.amazon.com/s?k=Evidence-Based+Technical+Analysis+David+Aronson&ref=aligrithm.com) - [Systematic Trading - Robert Carver (Amazon)](https://www.amazon.com/s?k=Systematic+Trading+Robert+Carver&ref=aligrithm.com) ### 1.25 Who Is the Marginal Buyer? URL: https://aligrithm.com/who-is-the-marginal-buyer/ Last updated: 2026-07-27T13:23:01.000Z A thinly traded name gives back a quarter of its value before lunch, and the explanations land before the tape settles. A big holder dumped. The float was always fake. Somebody knows something the rest of us don't. Each of these talks about the seller, and each one feels right because you can see the seller in your head. People sell to pay for things, so a wave of exits is easy to picture. Almost nobody asks the question that actually fixed the number on the screen: who was willing to pay $40 a share for a speculative position, and why did they walk away? Price is set at the margin. The seller's story tells you nothing about where price goes next, because the move was never about the seller alone. It was about the buyer who stood there yesterday and is gone today. ## Price is set by the marginal trade, not the average holder A market clears where the most desperate seller meets the least reluctant buyer. The number on the screen is the last price at which one marginal buyer and one marginal seller agreed, and that trade is a sliver of the float. Everyone else holding the position never transacted at that price. They are along for the ride on a valuation set by a tiny fraction of supply. $$ \\text{paper value} = P\_{\\text{last}} \\times \\text{supply}, \\qquad P\_{\\text{last}} = \\text{price of the marginal trade} $$ Read it as: the headline valuation multiplies the last traded price by the entire supply, but that last price came from the marginal trade alone, often a rounding error against the float. Mark ten million shares to a price set by a few hundred changing hands and you get a paper number nobody could realize by selling. The valuation holds only at the margin and only for the next small clip. That is why a position with a half-billion paper cap can shed a third of it in a session without anything close to a half-billion of selling. The marginal buyer stepped back, and the whole stack reprices to wherever the next willing buyer sits. ## A market falls because it runs out of buyers, not because someone sells The cleanest way to feel this: a market does not need a seller to fall. It needs the bids to disappear. With no bids, the position goes to zero on the spot, and nobody had to sell a single share to make that happen. You ran out of buyers. The recurring puzzle, how does price drop if no one is selling, is the same confusion in a different costume. Selling is not the cause. The withdrawal of the marginal buyer is the cause, and selling is the visible symptom that earns the headline. Turn your attention to the bid side and the down move stops being a mystery. Somebody was absorbing supply at the old level. They ran out of capital, ran out of conviction, or marked their reservation value lower. The size of the drop is the distance to the next buyer standing firm. ## The seller's story is explanation; the buyer's question is prediction The old article ["The Difference Between Explanation and Prediction in Markets"](https://aligrithm.com/the-difference-between-explanation-and-prediction-in-markets/) drew the line that matters here. Narrating the seller is explanation: a tidy after-the-fact account that feels like understanding and carries no forecast. You can pin "a big holder dumped" to any red candle, which is the reason it predicts nothing. An explanation that fits every outcome constrains none of them. Asking who the marginal buyer is forces you into prediction, because you have to name a cohort and a price at which they will or will not show up. Leveraged longs adding on strength. New inflows from a fresh allocation. Dealers covering a short they were forced into. Index or fund flows that buy regardless of view. Each is a buyer with a different trigger and a different reservation price, and each can be modeled, falsified, and watched. The moment you switch from "who sold" to "who buys here," you stop collecting stories and start building a demand curve. ## What the question buys you If you cannot name a willing buyer at the current price, the price has nothing under it. That is the practical payoff. A level holds only because of the buyers who will defend it, and an inventory of those buyers tells you where the air pockets sit. When the only marginal buyer you can identify is the leveraged long who needs the trend to continue, you are one liquidation cascade from learning how far down the next real bid lives. This is where the old article ["The Death of the Single-System Trader"](https://aligrithm.com/the-death-of-the-single-system-trader/) rhymes. A trader who leans on one rule is one regime change from broke, and a price that leans on one cohort of buyers wears the same fragility under a different hat. If every marginal buyer at $40 is the same participant running the same playbook, their demand is one correlated bet, and it vanishes at once when the regime turns. A market with diverse marginal buyers, inflows and short-covering and value bids that activate at different prices, has structural support a single-buyer market lacks. Counting the kinds of buyer under a price is the demand-side version of counting uncorrelated systems in a portfolio. ## Visualizing the marginal buyer ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_326-visual-0.png) ## KEY POINTS - Price is set by the marginal trade, a sliver of the float, not by the average holder; the headline valuation marks the entire supply to a price only the next small clip could realize. - A market does not need a seller to fall, it needs the bids to disappear; running out of buyers, not the act of selling, is the cause of a down move. - Narrating the seller is explanation that fits any outcome and forecasts nothing, the failure the old article ["The Difference Between Explanation and Prediction in Markets"](https://aligrithm.com/the-difference-between-explanation-and-prediction-in-markets/) named. - Asking who the marginal buyer is forces a prediction: name the cohort (leveraged longs, inflows, dealer short-covering, fund flows) and the price at which they show up or step back. - If you cannot name a willing buyer at the current price, the level has no support; an inventory of marginal buyers tells you where the air pockets are. - A price propped by one buyer type is one regime change from collapse, the demand-side echo of the old article ["The Death of the Single-System Trader"](https://aligrithm.com/the-death-of-the-single-system-trader/); diverse marginal buyers are structural support. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Evidence-Based Technical Analysis - David Aronson (Amazon)](https://www.amazon.com/s?k=Evidence-Based+Technical+Analysis+David+Aronson&ref=aligrithm.com) - [Systematic Trading - Robert Carver (Amazon)](https://www.amazon.com/s?k=Systematic+Trading+Robert+Carver&ref=aligrithm.com) ### 5.34 Layering Forecasts Across Horizons: Blending on the Markout Curve URL: https://aligrithm.com/layering-forecasts-across-horizons-blending-on-the-markout-curve/ Last updated: 2026-07-27T13:25:21.000Z You rarely have one forecast. You have several, and they disagree about time. One model reads top-of-book imbalance and predicts the next second. Another reads slower trade-flow structure and predicts the next minute. Both are real edge, but they live on different clocks, and the naive move, picking the one with the bigger number, throws away the other. A 30 basis point forecast over a minute looks far stronger than a 2 basis point forecast over a second until you notice they are not the same unit. The fix is to put every forecast on a common time basis and blend them there. The common basis is the markout curve measured per second. The old article ["Markouts: The Truth Serum of Market Making"](https://aligrithm.com/markouts-the-truth-serum-of-market-making/) established the markout curve as the only honest measure of a quote's edge, the signed price change after a fill read across horizons. This article uses that same curve as the conversion factor that makes forecasts at different horizons comparable, so you can layer them instead of choosing between them. _This post is for paying subscribers only._ ### 5.43 Predictive but Uncorrelated: Alphas That Only Work as Interactions URL: https://aligrithm.com/predictive-but-uncorrelated-alphas-that-only-work-as-interactions/ Last updated: 2026-07-27T13:25:26.000Z Here is a feature that knows the future and cannot trade. Volatility. When volatility moves from low to medium, the size of the next price move tends to go up. You have genuine information: the move is coming. You still have no idea which way. The correlation between volatility and the signed future return is about zero, so a model that takes volatility as a standalone input does nothing with it. The information is real and the regression coefficient is zero, and both statements are true at once. This trap catches a lot of careful people. You find a feature with obvious predictive content, drop it into a linear model expecting it to lift the fit, and watch it contribute nothing. The feature predicts the magnitude of the move, written as the absolute future return, not its sign. A linear model fits signed returns. The two never meet. $$ \\text{corr}\\big(\\text{vol},\\; r\_{t+1}\\big) \\approx 0 \\qquad\\text{while}\\qquad \\text{corr}\\big(\\text{vol},\\; \\lvert r\_{t+1}\\rvert\\big) > 0 $$ Read it as two separate facts. The left correlation, volatility against the signed next return, sits near zero, which is why volatility alone earns no weight in a model of direction. The right correlation, volatility against the absolute next return, is positive, which is the predictive content you can see but cannot directly monetize. A directional model only ever consumes the left one. ## The fix is multiplication, not addition _This post is for paying subscribers only._ ### 5.45 Detecting Wash Trading with an FFT, and Why Your TWAP Needs Random Timestamps URL: https://aligrithm.com/detecting-wash-trading-with-an-fft-and-why-your-twap-needs-random-timestamps/ Last updated: 2026-07-27T13:25:27.000Z The old article ["The Autocorrelation Periodogram"](https://aligrithm.com/the-autocorrelation-periodogram/) spent its whole length explaining why the Fourier transform is the wrong tool for price: price is non-stationary, the window is never a representative infinite sample, and a real market cycle is never a clean integer number of periods in your window. All true. But flip the target from price to the timing of trades and the objection inverts. A wash-trading bot or a naive execution algo is an artificial, stationary, periodic process by construction, the one signal that satisfies every assumption the DFT demands. The transform that fails on price nails a robot cold, because a robot is the textbook clean sinusoid that price never is. Two uses fall out of the same observation. You can run an FFT over trade timing to catch a counterparty trading on a clock, and you can run it over your own execution to make sure you are not the one getting caught. ## Build a count series and transform it Trade events arrive at irregular timestamps, so you cannot FFT them directly. Bin them. Slice time into fixed buckets, count the trades in each bucket, and you have a regularly-sampled series of trade intensity. Run the FFT on that series and read the power spectrum. A market driven by genuine, unscheduled flow gives a broad, flat-ish spectrum with no special period. A market with a bot firing on a fixed cadence gives a sharp spike at the bot's frequency, because the count series literally repeats every so many seconds. $$ n\[k\] = \\#\\{\\text{trades in bin } k\\}, \\qquad P(f) = \\big|\\,\\text{FFT}(n)\\,\\big|^2 $$ $$ \\text{spike at } f^\\star \\;\\Longrightarrow\\; \\text{periodic process with period } T = 1/f^\\star $$ _This post is for paying subscribers only._ ### 5.44 MFT Execution Is Built on HFT Market-Making URL: https://aligrithm.com/mft-execution-is-built-on-hft-market-making/ Last updated: 2026-07-27T13:25:27.000Z A mid-frequency shop wants to get long 250k of something. The naive move is to cross the spread and take it. The cheaper move, the one the good execution stacks actually use, is to become a market maker for the duration of the entry. Market makers have the best execution in the world for orders of trivially small size, because providing liquidity at a tight spread is their entire business. So the lowest-cost execution systems for MFT shops get built on the HFT side, by borrowing the market-making machinery and pointing it at a single goal: putting on the position instead of staying flat. "Trivially small size" is the load-bearing qualifier. Once your order is large relative to the venue, a new problem appears, disguising your flow so the market does not front-run your remaining size, and that is a different discipline. Below that threshold, the cheapest way in is to act like the maker who would have quoted you, and capture the spread you would otherwise have paid. ## How a maker changes position: skew To see why this works, recall how a market maker moves its own inventory. A maker quotes both sides. When it wants to shed inventory or pick some up, it skews: one side of the quote sits further from midprice than the other, so the near side fills more often and the far side fills less. The old article ["Why Skewing Is Simpler Than People Think"](https://aligrithm.com/why-skewing-is-simpler-than-people-think/) laid out the mechanics, and the old article ["Maker vs Taker Edge: Same Signal, Different Economics"](https://aligrithm.com/maker-vs-taker-edge-same-signal-different-economics/) laid out why the maker and the taker run opposite businesses off the same view. The maker skews to drag its position back toward zero: the more inventory it holds, the harder it skews to offload it. Now run that machinery in reverse for execution. You want to put on 250k long. So you tell your market-making engine that it currently holds 250k short. It does not, it holds zero, but the engine does not know that. Believing it is 250k short, it skews hard to its bid side to "get back to zero," resting aggressive passive bids and weak offers. It fills on the buy side, accumulates the long, and as far as it is concerned it is dutifully flattening a short. When it thinks it has reached zero, you are actually long 250k. You have put on the entire position as a maker, collecting spread on every fill instead of paying it. _This post is for paying subscribers only._ ### 4.70 Combining Three Weak Alphas on Cointegrated Futures URL: https://aligrithm.com/combining-three-weak-alphas-on-cointegrated-futures/ Last updated: 2026-07-27T13:24:50.000Z Take CME's WTI crude contract, ticker CL, a contract for physical delivery of light sweet crude at Cushing, Oklahoma. Call the front month CL0 and the next expiry CL1\. Right now CL0 trades at 78.40 and CL1 at 78.95, a 0.55 contango. You want a fair value for CL1, the less liquid leg, that is sharper than its own wide, slow quote. Three separate ideas each take a swing at this, and each one, on its own, is too weak to trade. Combine all three and you can price the contract adjacent to the active month well enough to make markets in it. That is the whole exercise: three losers that add up to a winner. This piece picks up where the old article ["Using Ratios as Trading Signals"](https://aligrithm.com/using-ratios-as-trading-signals/) left off. A ratio nets two correlated series into one rotation signal. Here we go the other way and fuse several noisy estimators of the same fair price into one estimate that is more precise than any input. The intermarket gating from the old article ["Intermarket Analysis for System Traders"](https://aligrithm.com/intermarket-analysis-for-system-traders/) is a cousin of this, but gating filters; fusion averages. ## The setup: why adjacent futures are cointegrated CL0 and CL1 are claims on the same barrel of oil, separated by one month of carry. The price difference between them cannot wander freely. If CL1 minus CL0 climbs far above the cost of storing oil for a month (warehouse, insurance, financing), a physical trader buys CL0, takes delivery, stores the barrel, and sells CL1 against it, pocketing the gap above storage cost. If the gap collapses too far, the reverse trade pays. Those trades carry frictions, so the spread is not pinned to a single number. It floats inside a band, and inside that band the two prices are cointegrated: they drift apart, then a physical arbitrageur drags them back. Cointegration is the structural reason any of this works. It is also why the example needs a contract with several expiries trading real volume at once. CL gives you that overlap window. A market where only the front month trades leaves you nothing to fuse. ## Alpha one: price cointegration via the spread EMA _This post is for paying subscribers only._ ### 2.76 The Limits of Linear Models URL: https://aligrithm.com/the-limits-of-linear-models/ Last updated: 2026-07-27T13:23:36.000Z Write down the model every quant writes in their first month predicting cross-sectional returns. You take three signals, momentum, carry, and value, and you weight them. $$ \\text{fut\\\_return} = b\_1 \\cdot \\text{momentum} + b\_2 \\cdot \\text{carry} + b\_3 \\cdot \\text{value} $$ You fit the three weights on history, you get a positive in-sample R-squared, and you feel like you have found something. The model is clean, the algebra is honest, and the fit looks real. It is also wrong in a way that no amount of data will fix, and the reason it is wrong is the whole subject of this article. Quant equity desks moved off plain linear factor models as their final word years ago. They still run them as baselines and as components. They stopped trusting them on their own, because the relationship between a signal and the next return is not a fixed number. It is a number that changes with the regime. ## The weights are not constants Momentum, the trend signal, does not pay equally in all conditions. The same context dependence shows up at every frequency: the old article ["Order Book Imbalance: The First Microstructure Feature to Test"](https://aligrithm.com/order-book-imbalance-the-first-microstructure-feature-to-test/) makes the point for a microstructure feature, and momentum behaves the same way at medium frequency. In a low-volatility regime with a clean, persistent trend, momentum is a strong signal: the move has direction and it keeps going. In a high-volatility chop, the same momentum number means almost nothing, and past some level of stress it reverses, because crowded trends unwind hardest when vol spikes. So the true coefficient on momentum is not the constant b1\. It behaves more like momentum gated by the volatility regime, or momentum scaled by trend strength. The honest model is closer to this. _This post is for paying subscribers only._ ### 5.37 Continuous, Event-Driven Trading vs Bar-Based Research URL: https://aligrithm.com/continuous-event-driven-trading-vs-bar-based-research/ Last updated: 2026-07-27T13:25:23.000Z Most research runs on bars. You resample the tape to one-minute or one-hour candles, compute features on the close, decide a position, and assume you execute at that close. It is fast, it vectorizes, and the old article "Backtest Integrity Checklist as Code" showed how to keep it honest with shift discipline so you never trade on a bar you could not have seen. Bars are the right tool for screening ideas. But they encode an assumption about time that breaks at the microstructure scale, and a maker who carries the bar habit into live quoting leaves money on the table and hands an edge to anyone watching. The assumption is that the clock drives the market. Sample every minute, act every minute, as if the interesting moments arrive on a fixed schedule. They do not. Markets move through events, a quote changes, a large order prints, liquidity vanishes, and those events do not wait for your bar to close. This is the intrinsic-time idea: the meaningful clock is the sequence of events, not the wall clock. Bar research forces event-driven reality onto a clock-time grid, and two specific things go wrong. ## Problem one: fixed rebalance times are front-runnable A bar strategy acts at bar boundaries. If you rebalance on the one-hour candle, you trade at the top of every hour. So does a large slice of the market running the same convention. Your order timing is public knowledge before you place it, because everyone knows the hour is about to turn and a wave of mechanical rebalancing is coming. _This post is for subscribers only._ ### 6.39 Why Traders Should Analyze Indicators Mathematically URL: https://aligrithm.com/why-traders-should-analyze-indicators-mathematically/ Last updated: 2026-07-27T13:25:49.000Z Traders use moving averages, oscillators, and momentum indicators every day, and almost none of them can tell you what those indicators actually do to the data. They know an indicator's recipe, the formula, the lookback, and they treat its output as a black box that produces buy and sell hints, without ever asking what mathematical operation it performs. This is a strange gap, because every common indicator is a well-understood signal-processing operation in disguise, and seeing it as one explains the indicator's behavior, its lag, and its failures far better than any amount of charting. No trader seems to analyze their indicators mathematically or characterize them, and that refusal leaves obvious knowledge on the table. This is the methodological close of the pillar, the habit of mind that the whole econophysics run has been arguing for. ## A moving average is a filter Start with the most common indicator. A trending indicator, any moving average or smoother, takes a noisy price series and produces a smoother one by averaging recent values, and that operation has a precise name in signal processing: it is a low-pass filter. It passes the low-frequency component of the price, the slow trend, and attenuates the high-frequency component, the fast noise. Seeing a moving average as a low-pass filter is not a metaphor; it is exactly what the math does, and it immediately explains why moving averages behave the way they do. Once you see the filter, the indicator's properties stop being mysterious. The reason a moving average lags is that filtering introduces phase delay: separating the slow trend from the fast noise takes information from the past, so the filtered output necessarily arrives later than the price, and a smoother filter, more lag, while a more responsive filter passes more noise. The eternal trade-off every trader feels between a moving average that is smooth-but-slow and one that is fast-but-noisy is not a tuning annoyance, it is a fundamental property of filters, the inescapable relationship between smoothing and delay. You cannot tune it away because it is built into what filtering is. ## Causality is the constraint that creates the lag The signal-processing view also explains why the lag cannot be removed, and the reason matters. An ideal filter that removed noise with no phase lag would need to use future data, because separating signal from noise without delay requires seeing what comes next. Traders cannot do that; an indicator must be causal, using only past and present data, since no future prices are available. That causality constraint is exactly what forces the lag: a causal filter cannot know the future, so it cannot anticipate the trend, so its output trails the price. $$ y\_t = \\sum\_{k=0}^{N} h\_k \\, x\_{t-k} $$ Every causal indicator has this form: the output is a weighted sum of current and past prices, with the weights defining the filter. A simple moving average makes all the weights equal; an exponential moving average makes them decay geometrically into the past; a momentum indicator takes a difference of past values. The weights are the indicator, and they completely determine its behavior, its lag, its smoothing, its response to a sudden move. Reading an indicator as its weight pattern, rather than as a chart line, tells you what it will do before you ever plot it, because the frequency response, how it treats slow trends versus fast noise, follows directly from the weights. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## What this habit buys you Analyzing indicators mathematically changes how you build and combine them. Instead of bolting together indicators by trial and error and keeping whatever backtests well, the overfitting trap this whole series warns against, you can reason about what each one does and choose them for their actual properties. If you want to extract a trend at a certain timescale, you choose or design a filter with the right frequency response, rather than sweeping moving-average lengths until one looks good. If two indicators backtest similarly, the math tells you whether they are doing nearly the same operation, in which case combining them adds nothing, or genuinely different ones, in which case they might diversify. This is the indicator-level version of ["Why Portfolio Construction Is Part of the Signal"](https://aligrithm.com/why-portfolio-construction-is-part-of-the-signal/): understanding the operation lets you compose deliberately instead of by accident. The honest limit keeps this from becoming its own delusion. Understanding an indicator mathematically tells you precisely what it does to the data; it does not tell you whether what it does has any predictive value, because a perfectly characterized filter applied to a series with no exploitable structure produces a perfectly characterized worthless signal. The math is necessary and not sufficient, the same refrain as the rest of this pillar: knowing your indicator is a low-pass filter with a given lag is real knowledge, and it still has to be paired with honest out-of-sample testing and cost accounting to become an edge. The value of the mathematical habit is that it stops you from fooling yourself about what an indicator is, removes the mystery that lets bad indicators hide, and lets you build on understanding rather than on the accidents of a backtest. It is the discipline of knowing your tools, which is the least a serious trader owes the work. ## Visualizing indicators as filters ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_179-visual-0.png) ## KEY POINTS - Traders use indicators daily without knowing what mathematical operation they perform. Every common indicator is a well-understood signal-processing operation, and seeing it as one explains its behavior better than charting. - A moving average is a low-pass filter: it passes the slow trend and attenuates the fast noise. That is exactly what the math does, not a metaphor. - The filter view explains the lag. Filtering introduces phase delay, so a smoother filter lags more and a faster one passes more noise. The smooth-but-slow versus fast-but-noisy trade-off is a fundamental property of filters, not a tuning annoyance. - Causality creates the lag. An indicator must use only past and present data, and a causal filter cannot see the future, so it cannot anticipate the trend and its output trails the price. - Every causal indicator is a weighted sum of current and past prices, and the weights are the indicator. A simple average uses equal weights, an exponential one uses decaying weights, momentum takes a difference, and the weights fully determine the behavior. - The habit lets you choose and combine indicators by their actual properties instead of by trial and error. The math tells you what an indicator does, not whether it predicts anything, so it is necessary but not sufficient and still needs honest testing. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Systematic Trading - Robert Carver (Amazon)](https://www.amazon.com/s?k=Systematic+Trading+Robert+Carver&ref=aligrithm.com) - [Trading Systems - Urban Jaekle Emilio Tomasini (Amazon)](https://www.amazon.com/s?k=Trading+Systems+Urban+Jaekle+Emilio+Tomasini&ref=aligrithm.com) ### 4.69 Why a Weak Dollar Means Strong Commodities URL: https://aligrithm.com/why-a-weak-dollar-means-strong-commodities/ Last updated: 2026-07-27T13:24:50.000Z Crude, copper, gold, wheat, all of them carry a price tag written in dollars. That single fact is the reason commodity prices and the dollar lean against each other, and it is more mechanical than most intermarket links in this pillar. The old article ["Gold, Dollar, and Rates: A Practical Intermarket Map"](https://aligrithm.com/gold-dollar-and-rates-a-practical-intermarket-map/) worked one corner of this with gold; the same logic runs across the whole commodity complex, and it feeds straight back into the commodity currencies. Start with the arithmetic, because it is cleaner than any story. A barrel of oil has some real value to a refiner in Europe or Japan, set by supply and demand for energy. The dollar price of that barrel is the real value translated through the dollar's exchange rate. Hold the real value fixed and weaken the dollar, and the dollar price has to rise to represent the same real worth. Nothing about oil changed. The measuring stick shrank, so the number printed in dollars got bigger. ## The mechanism: a cheaper dollar makes commodities cheaper abroad, so demand lifts the price The arithmetic has a behavioral engine behind it that turns a measurement effect into a real price move. When the dollar weakens, a commodity priced in dollars instantly becomes cheaper for anyone holding euros, yen, or yuan, because their currency now buys more dollars and therefore more of the dollar-priced commodity. Cheaper goods draw more buyers, foreign demand rises, and that real demand pushes the actual price up rather than just rescaling the quote. $$ P^{\\text{foreign}} = P^{\\text{USD}} \\times \\frac{1}{\\text{USD strength}} $$ The price a foreign buyer pays in their own currency equals the dollar price times the inverse of the dollar's strength against that currency. Weaken the dollar, the inverse term rises, and for an unchanged dollar price the foreign buyer would pay more, but the chain runs the other way in practice: the foreign buyer was about to pay less, so demand picks up, and the extra demand pulls the dollar price up until the foreign-currency cost settles back. A worked feel: crude sits at 80 dollars, and a European buyer with the euro at 1.10 pays about 72.7 euros. The dollar drops 5% against the euro (euro to 1.155), and if crude held at 80 the European pays only 69.3 euros, a 5% discount on the same barrel. That discount is the trigger; European demand lifts, and crude's dollar price drifts up toward the level that restores the euro cost. The weak dollar did not change the oil. It changed who could afford it and at what dollar number. This is why the link shows up as a broad, persistent inverse correlation between the dollar index and commodity indices, not a tick-for-tick mirror. The old article ["Gold, Dollar, and Rates: A Practical Intermarket Map"](https://aligrithm.com/gold-dollar-and-rates-a-practical-intermarket-map/) used exactly this for gold, where gold-up is a direct dollar-down read; the same reasoning generalizes to the whole complex because they all share the dollar denominator. ## From commodities to the commodity currencies The payoff for an FX trader is the second link in the chain, because a weak dollar lifting commodities lifts the currencies of the countries that export them. The old article ["Why FX Traders Must Watch Gold, Rates, and Equities"](https://aligrithm.com/why-fx-traders-must-watch-gold-rates-and-equities/) laid out the per-currency commodity map, and the dollar sits underneath all of it. A weak dollar pushes crude up, and a higher crude price improves Canada's terms of trade and feeds the loonie, the chain the old article ["Crude Oil, Inflation, and FX"](https://aligrithm.com/crude-oil-inflation-and-fx/) worked through the trade balance and the inflation-then-rates channel. The same weak dollar pushes metals up and feeds the Aussie through copper, gold, and iron ore. This produces a double-counting trap worth naming. A long-AUDUSD trade is partly a short-dollar trade by construction, since the dollar is the quote leg, and it is also a long-commodity trade through the Aussie's exports, and the commodities are themselves moving on the dollar. The dollar, the commodity, and the currency are three readings of an overlapping force, not three independent confirmations. Counting a weak dollar, a copper rally, and a rising Aussie as three votes for the same position is counting the dollar three times, the same correlated-confirmation warning the old article ["Why FX Traders Must Watch Gold, Rates, and Equities"](https://aligrithm.com/why-fx-traders-must-watch-gold-rates-and-equities/) attached to its confirmation count. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_290-visual-0.png) ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Each currency to its own commodity The chain is general but the wiring is specific, and watching the wrong commodity for a given currency adds noise. Each commodity currency links to the input that dominates its export economy, the per-currency map the old article ["Why FX Traders Must Watch Gold, Rates, and Equities"](https://aligrithm.com/why-fx-traders-must-watch-gold-rates-and-equities/) built out. | Currency | Primary commodity link | Why | | -------- | ---------------------- | -------------------------------------------------------------- | | CAD | crude oil | oil is Canada's dominant export, the loonie tracks crude | | AUD | gold, copper, iron ore | Australia exports metals, the Aussie tracks the metals complex | | NZD | dairy | New Zealand's export base is agricultural, dairy prices lead | | NOK | crude oil | Norway is an oil exporter, the krone tracks crude | The dollar denominator sits under every row, so a broad dollar move tends to lift all the commodity currencies together (the bloc behavior from the old article "Correlation Regimes: Positive Feedback, Breakdowns, and 'USDCAD Is the Truth'"), while a commodity-specific move (an oil supply shock, a dairy auction surprise) separates them and is what lets you trade one commodity currency against another. The cross between two commodity currencies is largely a bet on their two commodities' relative move, with the shared dollar cancelled out, the same cancellation the old article ["Cross-Pair Signals: Can EUR Predict GBP?"](https://aligrithm.com/cross-pair-signals-can-eur-predict-gbp/) used. ## Where it earns its keep and where it lies Used as a directional cross-check, the link is one of the more reliable in intermarket FX because the denominator mechanism is real rather than a fitted correlation. A long-commodity-currency trade taken while the dollar is broadly strengthening is fighting the denominator, and a short-dollar view gets confirmation from a rallying commodity complex. It is a clean read precisely because it is arithmetic before it is behavioral. The honest failures are specific. The dollar is not the only thing moving a commodity, and a genuine supply shock swamps the dollar link entirely: an OPEC cut or a mine strike can rally crude or copper while the dollar is also rising, the supply-driven case the old article ["Crude Oil, Inflation, and FX"](https://aligrithm.com/crude-oil-inflation-and-fx/) flagged, and reading that rally as a dollar-weakness signal gets the dollar backwards. The correlation is non-stationary and can invert in a risk-off panic, when the dollar rallies as a safe haven while commodities and the commodity currencies all fall together, the dollar-smile and correlation-to-one regimes from the old articles ["Why FX Traders Must Watch Gold, Rates, and Equities"](https://aligrithm.com/why-fx-traders-must-watch-gold-rates-and-equities/) and "Correlation Regimes." Gold carries an extra wrinkle the old article ["Gold, Dollar, and Rates: A Practical Intermarket Map"](https://aligrithm.com/gold-dollar-and-rates-a-practical-intermarket-map/) insisted on, since it tracks real yields and the dollar at once, so a gold move is not a pure dollar read. And the double-counting trap is the easiest way to fool yourself into thinking a single dollar bet is a diversified, multiply-confirmed thesis. Use the link as a directional gate and a sanity check on the dollar leg, separate dollar-driven commodity moves from supply-driven ones before trusting it, and count the dollar once. ## KEY POINTS - Commodities are priced in dollars, so the dollar is the measuring stick. Weaken the stick and the dollar price of an unchanged commodity rises, before any real supply or demand changes. - The behavioral engine: a weak dollar makes dollar-priced commodities instantly cheaper for foreign buyers, foreign demand rises, and the real demand pushes the actual dollar price up. A 5% dollar drop hands a European crude buyer a 5% discount, which lifts demand and the price. - The link is a broad, persistent inverse correlation between the dollar and commodity indices, not a tick-for-tick mirror, and it generalizes the gold-dollar relationship to the whole complex because they share the dollar denominator. - The FX payoff is the second link: a weak dollar lifts commodities, which lifts the exporters' currencies (crude to the loonie, metals to the Aussie). The dollar sits under the whole per-currency map. - Double-counting trap: a long-AUDUSD trade is a short-dollar trade, a long-commodity trade, and the commodity is moving on the dollar. Those are three views of one force, not three independent confirmations. Count the dollar once. - Wire each currency to its dominant export (CAD-crude, AUD-metals, NZD-dairy, NOK-crude). A broad dollar move lifts them together; a commodity-specific shock separates them and lets you trade one against another. - Limits: a real supply shock swamps the dollar link and inverts the read, the correlation flips in a risk-off panic (safe-haven dollar rallies while commodities fall), gold tracks real yields too, so separate dollar-driven from supply-driven commodity moves and use the link as a directional gate, not a standalone trigger. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Trading Systems and Methods - Perry Kaufman (Amazon)](https://www.amazon.com/s?k=Trading+Systems+and+Methods+Perry+Kaufman&ref=aligrithm.com) - [Cybernetic Trading Strategies - Murray Ruggiero (Amazon)](https://www.amazon.com/s?k=Cybernetic+Trading+Strategies+Murray+Ruggiero&ref=aligrithm.com) - [The Art of Currency Trading - Brent Donnelly (Amazon)](https://www.amazon.com/s?k=The+Art+of+Currency+Trading+Brent+Donnelly&ref=aligrithm.com) ### 4.68 Trading Crosses with Relative Equity-Index Ratios URL: https://aligrithm.com/trading-crosses-with-relative-equity-index-ratios/ Last updated: 2026-07-27T13:24:49.000Z A cross like AUDCAD has no dollar in it, which is the point and the problem. Cancel the dollar out of two USD pairs and you are left with the pure relative story, Australia against Canada, with no common factor to lean on. So what drives it? The textbook reaches for the rate differential and the commodity split (the Aussie's metals against the loonie's crude), and those matter. The faster, more tradable read is sitting on a screen most FX traders never open: the relative performance of the two countries' stock markets. The link is direct. A country's equity index is the market's real-time vote on that economy's growth, earnings, and risk appetite, and a currency is a claim on the same economy. When Australian stocks are crushing Canadian stocks, capital is voting for Australian growth over Canadian, and that vote tends to show up in AUDCAD before the slower macro data confirms it. The ratio of the two indices packages this into one number. The old article ["Using Ratios as Trading Signals"](https://aligrithm.com/using-ratios-as-trading-signals/) built the ratio tool for any two markets; this applies it to the specific case where the two markets are national equity indices and the thing you trade is the cross between their currencies. ## Why a ratio and not the two indices separately Watching both indices side by side tells you less than dividing one by the other, because most of what moves any equity index on a given day is global, not local. A risk-off session sells Sydney and Toronto together; a risk-on session buys both. That shared global move is exactly the part that does not help you pick AUDCAD over CADAUD, and the ratio nets it out. $$ R\_t = \\frac{\\text{AU equity index}\_t}{\\text{CA equity index}\_t} $$ Dividing the Australian index by the Canadian one cancels the move both share and keeps only the part where one outperformed the other. If both rise 2% on a global risk-on day, the ratio is flat and tells you correctly that nothing changed between the two economies. If Australian stocks rise 2% while Canadian stocks are flat, the ratio rises and isolates the Australia-specific outperformance, which is the signal that should map to a stronger Aussie against the loonie. The ratio moves only on relative performance, the same netting-out the old article ["Using Ratios as Trading Signals"](https://aligrithm.com/using-ratios-as-trading-signals/) used to isolate rotation while ignoring the overall direction of either leg. One detail decides whether the ratio is well-behaved: work in logs. A raw ratio is asymmetric, a 10% Australian outperformance and a 10% Canadian outperformance do not produce mirror-image moves, and a near-zero denominator can spike it. Take the log and the ratio becomes the difference of two log indices, symmetric and additive, the same fix the old article ["Using Ratios as Trading Signals"](https://aligrithm.com/using-ratios-as-trading-signals/) applied to keep ratio signals honest. $$ r\_t = \\ln(\\text{AU index}\_t) - \\ln(\\text{CA index}\_t) $$ The log ratio is just the log of one index minus the log of the other, and its changes are the difference in the two indices' returns. A rising log ratio means Australian equities are out-returning Canadian equities, full stop, with the global component already removed. ## Turning the ratio into a cross signal Signal off the ratio the way you would signal off any price: compare it to its own recent average and read direction from which side it sits. Above its moving average, Australian equities are in a relative uptrend, favor long AUDCAD; below, favor short. The old article ["Using Ratios as Trading Signals"](https://aligrithm.com/using-ratios-as-trading-signals/) framed this as a ratio oscillator, where the sign of the gap gives direction and the distance gives stretch. $$ \\text{Osc}\_t = r\_t - \\frac{1}{N}\\sum\_{i=t-N+1}^{t} r\_i $$ The oscillator is the current log ratio minus its N-bar average, positive when Australia is outperforming relative to its own recent norm and negative when it is lagging. A worked feel: say the AU/CA log ratio sits 0.8% above its 50-day average and rising, while AUDCAD has not yet moved. That is the equity market voting for the Aussie ahead of the cross, a long-AUDCAD candidate where the confirmation comes from a market structurally independent of FX. The same logic runs the AUDNZD case the source driver list points to, the Australia-New-Zealand equity performance ratio against the Australia-New-Zealand cross. There is a lead-lag premise underneath this, and it has to be tested rather than assumed. The signal pays only if the equity ratio leads the currency cross, the way the old article ["Lead-Lag Relationships in Global Markets"](https://aligrithm.com/lead-lag-relationships-in-global-markets/) demanded a clean peak at a positive lag that survives out of sample. Measure the lagged cross-correlation between the equity ratio and the cross and confirm the equity ratio actually moves first; if the currency leads the equities, you have the relationship backwards and the signal is a coincident readout, not a forecast. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_289-visual-0.png) ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Where it earns its keep and where it lies The reason this beats staring at the rate differential is timeliness and independence. Equity indices trade continuously and reprice growth and risk in real time, so the relative-performance ratio updates faster than the macro data that drives the textbook commodity-and-rates story for these crosses. It is also a structurally independent source from the FX chart, the property the old article ["Why Cross-Asset Signals Beat Isolated Chart Reading"](https://aligrithm.com/why-cross-asset-signals-beat-isolated-chart-reading/) identified as the whole point of intermarket work: a second information source that is imperfectly correlated with the first carries off-chart information and can lead. For a cross where the dollar has been cancelled away, the equity ratio is one of the cleaner remaining independent reads. The failure modes are specific and they stack. The lead-lag is fragile and drifts, the standing warning from ["Lead-Lag Relationships in Global Markets,"](https://aligrithm.com/lead-lag-relationships-in-global-markets/) so the equity ratio that led AUDCAD last year may move coincidentally or lag this year, and a fixed rule built on a stale lead bleeds. Trading hours misalign, because Sydney, Toronto, and the FX market are open at different times, so a daily ratio splices non-synchronous closes and the apparent lead can be a timing artifact rather than information, the synchronization problem the old article ["Cross-Pair Signals: Can EUR Predict GBP?"](https://aligrithm.com/cross-pair-signals-can-eur-predict-gbp/) flagged for crosses. The equity-currency link inverts in regimes: in a commodity-driven episode the loonie may track crude and the Aussie may track metals while their stock indices say something different, and in a full risk-off panic both indices and the cross collapse onto global risk appetite and the relative story vanishes, the correlation-to-one regime from the old article ["Why FX Traders Must Watch Gold, Rates, and Equities."](https://aligrithm.com/why-fx-traders-must-watch-gold-rates-and-equities/) The moving-average length is the usual arbitrary dial trading reactivity against whipsaw, and crosses like AUDCAD are less liquid with wider spreads than the USD majors, so the per-trade edge has to clear a real cost before any of this is worth trading. Use the equity ratio as a confirming tilt on a cross you have an independent reason to trade, gate it on the rolling lead-lag and the rolling correlation, and treat it as one read among the rates and commodity drivers rather than a standalone trigger. ## KEY POINTS - A cross has no dollar in it, so the question is what drives the pure relative story. The relative performance of the two countries' equity indices is a fast, independent read, because a stock index is the real-time vote on an economy and a currency is a claim on the same economy. - Use the ratio of the two indices, not the two indices separately, because most of any index's daily move is global and shared; dividing nets out the shared move and keeps only the relative outperformance that should map to the cross. - Work in logs, so the ratio becomes the difference of two log indices: symmetric, additive, and immune to near-zero-denominator blow-ups. - Signal off the ratio like a price: above its moving average favor long the cross (e.g. long AUDCAD when AU equities outperform), below favor short, with the oscillator's sign giving direction and distance giving stretch. AUDNZD against the AU/NZ equity ratio is the same construction. - The signal pays only if the equity ratio leads the cross. Measure the lagged cross-correlation and confirm a clean lead that survives out of sample; if the currency leads, the signal is coincident, not predictive. - It beats watching the rate differential on timeliness (equities reprice continuously) and independence (a second source uncorrelated with the FX chart), the core intermarket payoff. - Limits stack: the lead-lag drifts, trading hours misalign so daily ratios splice non-synchronous closes, the link inverts in commodity-driven and risk-off regimes, the MA length is arbitrary, and crosses carry wider spreads. Use it as a gated confirming tilt, not a standalone trigger. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Trading Systems and Methods - Perry Kaufman (Amazon)](https://www.amazon.com/s?k=Trading+Systems+and+Methods+Perry+Kaufman&ref=aligrithm.com) - [Cybernetic Trading Strategies - Murray Ruggiero (Amazon)](https://www.amazon.com/s?k=Cybernetic+Trading+Strategies+Murray+Ruggiero&ref=aligrithm.com) - [The Art of Currency Trading - Brent Donnelly (Amazon)](https://www.amazon.com/s?k=The+Art+of+Currency+Trading+Brent+Donnelly&ref=aligrithm.com) ### 4.67 Correlation Regimes: Positive Feedback, Breakdowns, and "USDCAD Is the Truth" URL: https://aligrithm.com/correlation-regimes-positive-feedback-breakdowns-and-usdcad-is-the-truth/ Last updated: 2026-07-27T13:24:48.000Z In 2008 the correlation between FX and every other asset class ran to one. Stocks, credit, commodities, the high-beta currencies all sold together, and the safe havens all caught the bid together, and the individual currency stories that fill the rest of this pillar stopped mattering for months. A trader who had spent years learning that the Aussie tracks copper and the loonie tracks crude watched both pairs move on a single variable: are we risk-on or risk-off today. The drivers had not disappeared. They had been swamped by a correlation regime, and the regime was the only thing worth knowing. Correlation in FX is not a constant you measure once and bolt into a model. It is a regime that builds, reinforces itself, and then snaps. The old article ["Intermarket Analysis for System Traders"](https://aligrithm.com/intermarket-analysis-for-system-traders/) named non-stationary, sign-flipping correlation as the central trap of the whole discipline, and this article works the specific shape that non-stationarity takes in currencies: correlations rise through positive feedback until a regime shift breaks them, USD pairs move as a bloc, and one pair (USDCAD) gets used as the cleanest read on the bloc. ## Correlation feeds on itself until it breaks A rising correlation is not a passive measurement. It is a process that reinforces its own cause. When traders notice that the S&P 500 and AUDUSD have been moving together, they start trading one off the other: a down day in stocks becomes a reason to sell the Aussie, which makes the Aussie fall when stocks fall, which tightens the very correlation that prompted the trade. The belief in the correlation manufactures more of it. Picture global capital as one connected organism, where a move in one limb propagates to the others through the traders who arbitrage the links, and the propagation gets faster as more participants act on it. This positive feedback is why correlations do not drift gently. They ratchet toward one in stress, because stress is when everyone reaches for the same simple variable (risk appetite) and trades every instrument as a proxy for it. The old article ["Why FX Traders Must Watch Gold, Rates, and Equities"](https://aligrithm.com/why-fx-traders-must-watch-gold-rates-and-equities/) caught the endpoint of this process in the 2008 jump to one; the mechanism is the feedback loop that gets it there. The trade that exploits a tight correlation is the trade that tightens it further, right up until the regime shifts. And the regime does shift. A central bank surprise, a change in the dominant macro story, a single currency suddenly trading on its own domestic event, and the bloc fractures. The correlation that was 0.9 for six months is 0.2 for the next three, or flips sign entirely. The danger is structural: the feedback that built the correlation gives no warning before it breaks, because the same crowd that reinforced it unwinds it, and the unwind is faster than the build. A strategy that earned its edge from a tight correlation is short a regime change it cannot see coming, the same acceptance the old article ["Why Systems Work Until They Don't"](https://aligrithm.com/why-systems-work-until-they-dont/) built into every relationship in this project. ## Measuring the regime instead of assuming it Read the correlation as a rolling, time-varying number, never a single historical figure. Compute it over a trailing window of returns and watch the path, not the point. $$ \\rho\_t = \\frac{\\sum\_{i=t-N+1}^{t} (x\_i - \\bar{x})(y\_i - \\bar{y})}{\\sqrt{\\sum (x\_i - \\bar{x})^2}\\,\\sqrt{\\sum (y\_i - \\bar{y})^2}} $$ This is the ordinary correlation between two return series x and y, computed only over the last N bars and recomputed every bar, with the bars sliding the window forward. Use returns, not prices, or you measure a shared trend and call it correlation. A worked feel: with N = 60 daily bars, the S&P-AUDUSD correlation might read 0.35 in a calm spring, climb to 0.80 through a summer growth scare, and sit near 0.85 in an autumn panic. The level is the regime, and the slope is the warning; a correlation marching toward one is telling you the individual stories are being swamped, and a correlation rolling over from a high level is telling you the bloc is starting to fracture. The window length is the usual arbitrary dial. Short windows react fast and jump around on noise; long windows are stable and blind to the regime change you most want to catch. There is no correct N, only a trade between reactivity and stability, and the honest move is to watch two windows (a fast one for the warning, a slow one for the state) rather than pretend one number is the truth. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_288-visual-0.png) ## USD pairs move as a bloc, and EURUSD speaks for it Most of what looks like a hundred separate currency markets is one variable wearing different clothes. Every USD-based pair shares the dollar leg, so when the dollar moves, EURUSD, AUDUSD, GBPUSD, and the rest move together by construction, the same shared-USD mechanism the old article ["Cross-Pair Signals: Can EUR Predict GBP?"](https://aligrithm.com/cross-pair-signals-can-eur-predict-gbp/) used to cancel the dollar out of a cross. The bloc behavior is not a discovered correlation; it is arithmetic. A dollar move is common to all of them. EURUSD is the loudest voice in the bloc because it is the most liquid and the pair most speculators use to express a dollar view. A large EURUSD move is the cleanest signal that the dollar itself is moving, and that move drags the other USD pairs with it. Watching EURUSD is watching the common factor, which is why a trader holding AUDUSD or GBPUSD ignores EURUSD at their peril: a sharp EURUSD move says the thing your pair shares with every other USD pair just changed. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## "USDCAD is the truth" Some traders trust USDCAD above every other pair as the read on the dollar, to the point it has earned the nickname "the truth." The logic is that the Canadian dollar is the G10 currency closest to the United States in economy, trade, and time zone, so USDCAD strips out a lot of the foreign-side noise that muddies the other pairs and leaves a comparatively clean dollar signal. EURUSD carries European politics, Italian yields, and ECB policy on its non-dollar leg; USDJPY carries the entire risk-haven and carry complex on the yen leg; USDCAD's loonie leg is the quietest of the majors when oil is not the story, so the pair moves on the dollar more purely. Treat the nickname as a heuristic, not a law. USDCAD is a clean dollar read only when crude is dormant, because the old article ["Crude Oil, Inflation, and FX"](https://aligrithm.com/crude-oil-inflation-and-fx/) showed the loonie is an oil currency, and a sharp move in crude makes USDCAD a Canada story rather than a dollar story. The honest use of "the truth" is conditional: when the USD pairs disagree and you need a tiebreak on whether the dollar is really moving, USDCAD is a good arbiter as long as oil is not the thing that moved. When oil is moving, USDCAD is lying to you about the dollar, and EURUSD is the better common-factor read. ## Trading the regime, and the honest limits The operational use has two sides. First, defense: do not run an intermarket or cross-pair strategy as if its correlations are fixed. Gate it on the rolling correlation the way the old article ["Using Bonds to Filter Equity Signals"](https://aligrithm.com/using-bonds-to-filter-equity-signals/) gated on the rolling stock-bond correlation, suspending or resizing the strategy when the regime moves against its assumptions. A pairs trade or a relative-value cross built in a 0.3-correlation regime behaves like a different instrument at 0.9, and a market-neutral book stops being neutral when everything correlates to one. Second, offense, with heavy caveats. A divergence inside a tight bloc is a candidate signal: if USD pairs are moving together and one lags the move EURUSD or USDCAD just made, that laggard is a catch-up candidate, the bloc version of the divergence logic from the old article ["Intermarket Divergence as a Trading Filter."](https://aligrithm.com/intermarket-divergence-as-a-trading-filter/) The edge is thin and the catch comes with the usual lead-lag fragility from ["Lead-Lag Relationships in Global Markets"](https://aligrithm.com/lead-lag-relationships-in-global-markets/): the lag may be the laggard pricing in a genuine domestic difference rather than waiting to catch up, in which case fading the divergence is fighting real information. The limits are the same ones that run through every correlation in this pillar, sharpened by the feedback mechanism. Correlations are non-stationary and the breakdown comes without warning because the crowd that built it unwinds it. A high correlation is not three independent confirmations; when the bloc moves to one, your "diversified" FX book is one position. The rolling window is an arbitrary choice trading reactivity against stability. And the regime that makes correlation-based filters most attractive (everything tightly linked, signals clean) is exactly the regime sitting closest to the break, so the filter works beautifully right up to the moment it costs you the most. ## KEY POINTS - Correlation in FX is a regime, not a constant. It rises through positive feedback (traders trade one market off another, which tightens the link that prompted the trade) until a regime shift snaps it, and the snap comes without warning because the crowd that built it unwinds it. - In 2008 FX-asset correlations ran to one and individual currency stories stopped mattering for months. Stress drives every instrument toward a single variable (risk appetite). - Measure correlation rolling on returns over a trailing window, never as one historical number. The level is the regime, the slope toward one is the warning. The window length is an arbitrary reactivity-versus-stability dial; watch a fast and a slow window. - USD pairs move as a bloc by arithmetic, because they share the dollar leg. EURUSD is the loudest voice and the cleanest read that the dollar itself is moving, so a sharp EURUSD move warns every other USD pair. - "USDCAD is the truth" because the loonie leg is the quietest major when oil is dormant, leaving a comparatively pure dollar signal. The nickname is conditional: when crude moves, USDCAD becomes a Canada story and lies about the dollar, so use EURUSD instead. - Defense: gate intermarket and cross strategies on the rolling correlation, since a market-neutral book stops being neutral when everything correlates to one. Offense: a laggard inside a tight bloc is a thin catch-up candidate, with the usual lead-lag fragility. - The regime that makes correlation filters most attractive (clean, tightly linked) sits closest to the break, so the filter works best right before it hurts most. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Trading Systems and Methods - Perry Kaufman (Amazon)](https://www.amazon.com/s?k=Trading+Systems+and+Methods+Perry+Kaufman&ref=aligrithm.com) - [Cybernetic Trading Strategies - Murray Ruggiero (Amazon)](https://www.amazon.com/s?k=Cybernetic+Trading+Strategies+Murray+Ruggiero&ref=aligrithm.com) - [The Art of Currency Trading - Brent Donnelly (Amazon)](https://www.amazon.com/s?k=The+Art+of+Currency+Trading+Brent+Donnelly&ref=aligrithm.com) ### 4.66 Granger Causality: Finding What's Driving Your Currency Right Now URL: https://aligrithm.com/granger-causality-finding-whats-driving-your-currency-right-now/ Last updated: 2026-07-27T13:24:48.000Z A trader is long AUDUSD and wants to know what to watch on the second screen. The textbook answer is a fixed list: gold, copper, iron ore, the S&P 500, the Australia-US rate differential. The old article ["Why FX Traders Must Watch Gold, Rates, and Equities"](https://aligrithm.com/why-fx-traders-must-watch-gold-rates-and-equities/) built that list and turned it into a confirmation count. The list is correct and useless at the same time, because it names every driver that has ever mattered for the Aussie and stays silent on the one that matters this week. Risk appetite drives it in a panic, the rate differential drives it in a calm tightening cycle, copper drives it when China is the story. The driver rotates, and a static list does not tell you where you are in the rotation. Granger causality is the tool that picks the live driver out of the candidate set. It does not prove that copper causes AUDUSD in any physical sense. It asks a narrower, testable question: does knowing copper's recent moves help me predict AUDUSD better than AUDUSD's own past alone? If the answer is yes, copper "Granger-causes" AUDUSD over that window, and that is exactly the relationship a confirmer or a lead-lag tilt wants to exploit. The old article "Lead-Lag Relationships in Global Markets" measured a single lead with a lagged cross-correlation; Granger generalizes it to a regression that handles several lags at once and, crucially, lets you condition on the other candidates so you do not get fooled by a hidden third variable. ## The bivariate test: does X improve the forecast of Y? Start with two series, candidate driver x (say copper returns) and your currency y (AUDUSD returns). Fit two forecasting models of y. The first uses only y's own past, the restricted model. The second adds lagged values of x, the unrestricted model. $$ \\text{Restricted:}\\quad y\_t = a\_0 + \\sum\_{i=1}^{p} a\_i\\, y\_{t-i} + e\_t $$ $$ \\text{Unrestricted:}\\quad y\_t = a\_0 + \\sum\_{i=1}^{p} a\_i\\, y\_{t-i} + \\sum\_{j=1}^{p} b\_j\\, x\_{t-j} + u\_t $$ Both models predict today's return on the currency from the last p bars. The restricted model is allowed to use only the currency's own history. The unrestricted model also gets the last p bars of the candidate driver, with the coefficients b\_j measuring how much each lag of x pulls on y. The whole test reduces to one question about those b coefficients: are they jointly different from zero, or is the candidate adding nothing the currency's own past did not already contain? Measure it by how much the added series shrinks the prediction error. Collect the sum of squared residuals from each model and compare. $$ F = \\frac{(\\text{RSS}\_R - \\text{RSS}\_U)/p}{\\text{RSS}\_U/(N - 2p - 1)} $$ RSS\_R is the squared error of the restricted model and RSS\_U the squared error of the unrestricted one. Adding x can only lower the error or leave it flat, so RSS\_R is always at least RSS\_U; the numerator is the chunk of error the candidate driver killed, spread over the p lags it cost you. The denominator is the error that remains, per remaining degree of freedom, with N the number of bars. A worked feel for the number: say the currency's own past leaves RSS\_R = 1.000 over a window, and adding copper drops it to RSS\_U = 0.940 with p = 3 lags and N = 250 bars. The numerator is (1.000 - 0.940)/3 = 0.020, the denominator is 0.940/243 = 0.00387, so F is about 5.2\. Against an F distribution with 3 and 243 degrees of freedom that clears the 1% threshold, so copper Granger-causes AUDUSD over that window. The driver is live. Flip x and y and run it again. Granger causality is directional, and you want the direction where the external market leads the currency, not the one where the currency leads the external market. Often both directions test positive, which is its own warning, covered below. ## Multivariate: condition on the other suspects to kill the third-variable trap The bivariate test has the same hole as every correlation in this pillar. Two series can both be driven by a third, and the pairwise test will happily report that one causes the other. The old article ["Lead-Lag Relationships in Global Markets"](https://aligrithm.com/lead-lag-relationships-in-global-markets/) called this the ice-cream-and-drowning problem: summer drives both ice cream sales and pool deaths, so they predict each other while neither moves the other. In FX the hidden summer is usually the dollar itself or global risk appetite. Gold predicts AUDUSD and copper predicts AUDUSD, but a chunk of that is just the dollar moving all three, the same shared-USD confound the old article ["Cross-Pair Signals: Can EUR Predict GBP?"](https://aligrithm.com/cross-pair-signals-can-eur-predict-gbp/) dismantled. The fix is to test the candidate inside a model that already contains the other suspects. Put every candidate driver into one system and ask whether x still improves the forecast of y after the others have had their say. $$ y\_t = a\_0 + \\sum\_{i=1}^{p} a\_i\\, y\_{t-i} + \\sum\_{j=1}^{p} b\_j\\, x\_{t-j} + \\sum\_{k=1}^{p} \\sum\_{m} c\_{k,m}\\, z^{(m)}\_{t-k} + u\_t $$ The new term is the block of other candidates z^(m), each with its own lags, sitting in the model alongside x. The conditional Granger test now asks whether the b coefficients on x are jointly nonzero given that gold, the rate differential, and risk appetite are already in the regression. A candidate that tested positive on its own and dies once the dollar proxy is included was never a driver; it was a passenger riding the same factor. The ones that survive conditioning are the genuinely informative drivers, and the count of them tells you something the static list never could: in a calm week one or two survive, and in a 2008-style panic everything collapses onto risk appetite and only that survives, the regime the old article ["Why FX Traders Must Watch Gold, Rates, and Equities"](https://aligrithm.com/why-fx-traders-must-watch-gold-rates-and-equities/) flagged when FX-asset correlations jumped toward one. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_287-visual-0.png) ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## The "before I trust this" checklist A surviving F-statistic is a license to investigate, not a signal. Three checks stand between the test result and putting on a trade, and they are the difference between using Granger and getting wrecked by it. First, is there an economic reason. Copper leading the Aussie has a story: China's industrial demand prices copper and also drives Australia's terms of trade, so copper moves first and the currency follows. A surviving F-statistic with no story behind it is a data-mined coincidence, and with enough candidates and lags you will find dozens. This is the same defense the old article ["Intermarket Analysis for System Traders"](https://aligrithm.com/intermarket-analysis-for-system-traders/) insisted on, an economically motivated candidate set chosen before the test, not a scan over everything. Second, will it persist. Granger causality is estimated over a window, and the window is a snapshot of a relationship that drifts. The old article ["Lead-Lag Relationships in Global Markets"](https://aligrithm.com/lead-lag-relationships-in-global-markets/) warned that a real lead gets arbitraged toward zero as markets speed up, and the same decay hits a Granger link. Re-estimate on a rolling window and watch whether the F-statistic holds up out of sample or collapses the moment you stop looking at the data that produced it. Third, is a third variable moving both. The conditional test handles the suspects you thought to include. It says nothing about the one you left out. If AUDUSD and your candidate both load on a macro factor you never put in the regression, the conditional test still reports a spurious link, because you cannot condition on a variable that is not in the model. Treat a surviving link as conditional on your candidate set being complete, which it never fully is. ## Where it earns its keep and where it lies Used honestly, Granger turns the static driver map into a live one. Run the conditional test over a rolling window across your economically motivated candidates, take the survivors as this period's active drivers, and feed those into the confirmation logic from the old article ["Why FX Traders Must Watch Gold, Rates, and Equities"](https://aligrithm.com/why-fx-traders-must-watch-gold-rates-and-equities/) or the tilt from ["From Intermarket Analysis to Network Momentum"](https://aligrithm.com/from-intermarket-analysis-to-network-momentum/) instead of confirming against a fixed list that is half stale. The payoff is knowing that this week the Aussie is a risk-appetite trade and copper is noise, which a constant list cannot tell you. The honest failure modes are specific and they are mostly about the word "causality," which the test does not deliver. Granger causality is predictive improvement, nothing more; a link can be strong and still reverse the day you trade it, because prediction over a past window is not a promise about the next bar. The test assumes a linear relationship at fixed lags, and FX drivers act non-linearly and at shifting horizons, so a real but non-linear link can hide from it. It assumes the series are stationary, which the old article on stationarity in this project beat to death, so you run it on returns and not raw prices or you get a confident number about a trend. Bidirectional results, where x predicts y and y predicts x, usually mean a shared driver you have not modeled rather than a feedback loop you can trade. And the multiple-testing problem is brutal: scan ten candidates at four lag lengths in two directions and you have eighty tests, so a few will clear any threshold by chance. Pre-register the candidates, fix the lag length on economic grounds, re-estimate rolling, and treat the survivors as suspects worth watching rather than triggers worth trading. ## KEY POINTS - The driver of a currency rotates by regime, so a fixed driver list names everything that has ever mattered and stays silent on what matters this week. Granger causality picks the live driver out of the candidate set. - The test is a forecast comparison: fit the currency's return on its own past (restricted), then add the candidate driver's lags (unrestricted), and check whether the candidate's coefficients are jointly nonzero via an F-test on how much the prediction error shrank. - It is directional. Test both ways and keep the direction where the external market leads the currency, not the reverse. - Run it multivariate, not pairwise. Put every candidate in one regression and test each conditional on the others, which kills the hidden-third-variable trap where the dollar or risk appetite drives several "drivers" at once. The count of survivors reads the regime: many in calm, one (risk appetite) in a panic. - Before trusting a survivor: confirm an economic reason (not a data-mined fluke), confirm it persists out of sample on a rolling re-estimate, and remember the conditional test cannot control for a factor you left out of the model. - It is predictive improvement, not physical causality. The link can reverse the day you trade it. It assumes linear, fixed-lag, stationary relationships (run it on returns), bidirectional results usually mean a shared driver, and scanning many candidates and lags manufactures false positives. - Use the survivors as the live driver set feeding the confirmation count from the old article ["Why FX Traders Must Watch Gold, Rates, and Equities"](https://aligrithm.com/why-fx-traders-must-watch-gold-rates-and-equities/) or the network tilt from ["From Intermarket Analysis to Network Momentum,"](https://aligrithm.com/from-intermarket-analysis-to-network-momentum/) not as standalone entries. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Trading Systems and Methods - Perry Kaufman (Amazon)](https://www.amazon.com/s?k=Trading+Systems+and+Methods+Perry+Kaufman&ref=aligrithm.com) - [Cybernetic Trading Strategies - Murray Ruggiero (Amazon)](https://www.amazon.com/s?k=Cybernetic+Trading+Strategies+Murray+Ruggiero&ref=aligrithm.com) - [The Art of Currency Trading - Brent Donnelly (Amazon)](https://www.amazon.com/s?k=The+Art+of+Currency+Trading+Brent+Donnelly&ref=aligrithm.com) ### 4.65 Monetary Policy and Central-Bank Bias as the #1 FX Driver URL: https://aligrithm.com/monetary-policy-and-central-bank-bias-as-the-1-fx-driver/ Last updated: 2026-07-27T13:24:47.000Z The central bank cuts rates and the currency rallies. New traders stare at the screen and conclude the market is broken. The market is not broken. The cut was 25 basis points and the market had already priced 50, so a "cut" that undershoots expectations is a hawkish surprise, and the currency rallies on the gap between what happened and what was already in the price. Monetary policy is the single most consistent driver of currencies most of the time, and it works through expectations, not through the level. Miss that and you trade every meeting backwards. Monetary policy tops the domestic-driver list because the relative rate setting is what a floating currency is, at bottom: a claim on a yield differential that the market reprices the instant the expected path of that differential moves. The old article ["Global vs Domestic Currency Drivers"](https://aligrithm.com/global-vs-domestic-currency-drivers/) put it at the head of the three domestic engines and noted it drives most of the time, when the world is calm enough for domestic factors to be in charge. This is the mechanism underneath that ranking. ## Two levers, not one A central bank has two tools, and the second one used to be invisible to traders. The interest rate is the primary lever and the one everybody watches: raise it and the currency's yield advantage grows, cut it and the advantage shrinks. The balance sheet is the second lever, the quantitative-easing and balance-sheet-management toolkit that central banks leaned on after the rate lever hit zero. Expanding the balance sheet floods the system with the currency and acts like an easing even when the headline rate does not move; shrinking it does the reverse. For most of the modern era rates were the whole story, but a trader who only watches the policy rate now misses half the stance, because a bank holding rates flat while expanding its balance sheet is easing through the back door. The two levers combine into one thing the market actually trades: the stance, hawkish, dovish, or neutral. Hawkish means leaning toward tighter policy, higher rates or a shrinking balance sheet, which supports the currency. Dovish means leaning toward easier policy, which pressures it. Neutral means on hold. The stance is a direction, and the currency trades the change in direction more than the current setting. ## The market forward-discounts the path The core mechanic, the one that makes "cut equals rally" sane: the market does not wait for the central bank to act. It forward-discounts the anticipated path of rates, pricing expected future moves into the currency today. $$ \\Delta\\text{FX} \\;\\propto\\; \\big(\\text{actual policy} - \\text{expected policy}\\big) \\;=\\; \\text{surprise} $$ The currency move is driven by the surprise, the gap between what the bank does (or signals) and what the market had already priced, not by the action in isolation. A 25 basis point hike into a market expecting 25 is a non-event; the move already happened when the expectation formed. A 25 hike into a market expecting nothing is a hawkish shock and the currency jumps. A 25 cut into a market braced for 50 is a hawkish surprise and the currency rallies on a cut. The level is old news the moment the market agrees on it. The trade lives in the revision of expectations, which is why currencies react violently to the unexpected and shrug at the expected. This is also why the stance and the central bank's communication matter as much as the action. A bank can move the currency without touching rates at all, by shifting the expected path: a hawkish tilt in the statement, a dot-plot revision, a press conference that walks back a planned cut. The market reprices the whole forward path off the new information, and the spot currency moves now to reflect rates that have not happened yet and may never. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Banks carry a bias, and the bias is tradeable context Central banks are not blank reaction functions. Each carries a historical bias baked in by its institutional memory, and the bias tilts how it responds to the same data. The Bundesbank's lineage runs hawkish, a hard preference for tight policy that traces directly to the long German memory of hyperinflation. The Federal Reserve has leaned toward keeping rates low for long stretches, a tilt that reflects the Keynesian center of gravity in US policy thinking. Same inflation print, different institutional reflex, different currency consequence. The bias is not a signal by itself; it is prior context that shapes how you read the surprise. Knowing a bank runs dovish tells you it will take more inflation to provoke a hike than a hawkish bank would need, so the bar for a hawkish surprise is higher and the bar for a dovish confirmation is lower. It frames the distribution of likely outcomes before the meeting, which is exactly the kind of slow structural read that belongs in the regime layer. It does not tell you what prints tomorrow. ## Trading it without trading it by hand Monetary policy as the number-one driver does not mean trading central-bank meetings discretionarily, which is where the edge gets donated back. The honest structure is the one from the old article ["Why FX Traders Need Macro but Should Trade Systematically"](https://aligrithm.com/why-fx-traders-need-macro-but-should-trade-systematically/): the policy read sets the gate, a rule-based system pulls the trigger. The stance and the rate-differential trajectory define the allowed direction, hawkish bank, widening differential, currency-supportive regime, and the systematic layer takes its signals inside that allowance. The reason to keep hands off the trigger is the forward-discounting itself. By the time a policy shift is obvious enough to trade by gut, it is priced, and the discretionary trader ends up buying the currency at the top of the expectations move and holding through the "buy the rumor, sell the fact" reversal. The measurable inputs make the gate encodeable: the rate differential and its trend, the market-implied path from rate futures, the surprise index against consensus, all of which can be turned into a rule rather than a feeling. Encode the gate and you can backtest the policy overlay instead of trusting your read of a press conference. The residual that resists encoding, a genuine regime shift in a bank's reaction function, a credibility break, stays a slow human regime call, never a fast in-trade override. Policy chooses the direction; rules trade the path. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_286-visual-0.png) ## KEY POINTS - Monetary policy is the single most consistent domestic driver of currencies most of the time, the mechanism behind its top ranking in the old article ["Global vs Domestic Currency Drivers."](https://aligrithm.com/global-vs-domestic-currency-drivers/) It works through expectations, not the rate level. - Two levers: the interest rate (primary, watched by all) and the balance sheet (QE and balance-sheet management). A bank holding rates flat while expanding its balance sheet is easing through the back door, so watching only the policy rate misses half the stance. - The levers combine into a stance, hawkish, dovish, or neutral, and the currency trades the change in direction more than the current setting. - The market forward-discounts the anticipated path, so the currency move is driven by the surprise (actual minus expected), not the action. A cut smaller than priced is a hawkish surprise and the currency rallies on a cut. Communication moves the currency without any rate change by shifting the expected path. - Central banks carry historical biases (the Bundesbank hawkish from hyperinflation memory, the Fed tilted to low rates from Keynesian dominance). The bias is prior context that frames the distribution of outcomes and shapes how you read a surprise, not a signal by itself. - Do not trade meetings by hand, since by the time a shift is obvious it is priced. Let the policy read set the gate (allowed direction from the stance and rate differential) and a rule-based system pull the trigger, the split from the old article ["Why FX Traders Need Macro but Should Trade Systematically."](https://aligrithm.com/why-fx-traders-need-macro-but-should-trade-systematically/) The inputs are measurable, so the gate is encodeable and backtestable. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Trading Systems and Methods - Perry Kaufman (Amazon)](https://www.amazon.com/s?k=Trading+Systems+and+Methods+Perry+Kaufman&ref=aligrithm.com) - [Cybernetic Trading Strategies - Murray Ruggiero (Amazon)](https://www.amazon.com/s?k=Cybernetic+Trading+Strategies+Murray+Ruggiero&ref=aligrithm.com) - [The Art of Currency Trading - Brent Donnelly (Amazon)](https://www.amazon.com/s?k=The+Art+of+Currency+Trading+Brent+Donnelly&ref=aligrithm.com) ### 4.64 Global vs Domestic Currency Drivers URL: https://aligrithm.com/global-vs-domestic-currency-drivers/ Last updated: 2026-07-27T13:24:47.000Z Same currency, same week, two opposite explanations. On Monday the Australian dollar rallies and the desk says iron ore and a risk-on tape. On Friday it sells off and the desk says soft local jobs data and a dovish RBA. Both are right. What changed is not the currency. What changed is which set of drivers the market decided to price. A currency is pushed by global forces and domestic forces at the same time, and the market oscillates over which set is in charge. Read the wrong set on the wrong day and your correct analysis gets the direction backwards. The drivers split into two buckets. Four are global, shared across currencies and set by the state of the world. Three are domestic, specific to the country. They are tangled together, but the practical skill is knowing which bucket holds the steering wheel right now, because the answer rotates and the rotation is the part that breaks naive models. ## The four global drivers Global growth comes first because it moves money around the planet. When growth is strong, capital flows toward exporters and the currencies levered to expansion, the Chinas and Koreas and Brazils of the world. When growth weakens, money retreats to safe havens, the Swiss franc and often the dollar, regardless of what those economies are doing domestically. Growth is the tide, and the tide sets the baseline direction for a whole class of currencies at once. Commodity prices are the second, driven mostly by global growth but kicked around by weather, supply, production, and the fashion of treating a commodity as an asset class. Exporters benefit when prices rise: Brazil, Canada, Australia carry strong terms of trade when their exports get dear. Importers get hurt on the same move: Turkey and India bleed when energy and food climb. One commodity move splits the currency universe into winners and losers along the export-import line. Risk appetite is the third and the most violent. Global risk-on and risk-off moves the currencies regardless of any domestic fundamental. In a risk-off panic, nobody is pricing the Australian jobs report; they are selling the high-beta currencies and buying havens because the world turned scared, and the local data sheet goes in the bin. Geopolitics is the fourth, often working through risk appetite, sometimes hitting affected countries directly. Instability in Russia sells the ruble, the zloty, and the lira together as the market dumps regional risk, not because three central banks moved but because one map got more dangerous. ## The three domestic drivers Monetary policy is the number-one domestic driver most of the time, the relative interest-rate setting and the central-bank stance. It gets its own treatment in the old article ["Monetary Policy and Central-Bank Bias as the #1 FX Driver,"](https://aligrithm.com/monetary-policy-and-central-bank-bias-as-the-1-fx-driver/) so the short version here: rate differentials and the expected path of policy are the most consistent domestic engine, and currencies react hard to surprises in them. Interest rates are the second, closely tied to policy but worth separating because the market trades the relative differential and its changes, not the level. The third is politics, the elections, fiscal fights, and institutional shocks that move a currency through its own internal story. Domestic drivers are the engine when the world is calm. Stable global backdrop, no panic, no commodity shock, and the market goes back to pricing local rates and local data, which is when fundamental analysis of a single economy actually pays. ## The oscillation is the whole game The drivers do not add up to a fixed weighting. The market swings between regimes where global factors dominate and regimes where domestic factors dominate, and the swing is what wrecks a model built on one bucket. $$ \\Delta\\text{FX}\_t \\;=\\; w\_t \\cdot (\\text{global drivers}) \\;+\\; (1 - w\_t)\\cdot(\\text{domestic drivers}), \\qquad w\_t \\in \[0,1\] $$ Read the weight, not the symbols: a currency's move on any day is a blend of the global bucket and the domestic bucket, and the blend weight shifts over time. In a calm world the weight on global sits low and local rates and data drive the currency, so a hawkish surprise from the local central bank sends it up cleanly. In a risk-off episode the weight on global swings toward one and domestic drivers go to irrelevant, so that same hawkish surprise gets steamrolled by a flight to havens and the currency falls anyway. The weight is not constant and it is not observable on a fixed schedule, which is why a regression with static coefficients keeps getting the regime wrong precisely when the regime matters most. This is the same non-stationarity that runs through the old article ["Why FX Traders Must Watch Gold, Rates, and Equities."](https://aligrithm.com/why-fx-traders-must-watch-gold-rates-and-equities/) That article said to watch gold, rates, and equities together and count confirmations; the driver framework here says why the count rotates. Gold and the risk tape carry weight when the global bucket is in charge; the rate differential carries weight when the domestic bucket is in charge. The confirmation that mattered last month can be noise this month because the weight moved underneath it. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Trading the regime, not the average The usable discipline is to first identify which bucket is driving, then read the matching signals, and to treat that identification as the slow regime call it is. In a risk-off or China-obsessed tape, stop pricing local data and trade the currency as a risk proxy: where does it sit in the high-beta-to-haven spectrum, and which way is global risk leaning. In a calm tape, switch back to the domestic engine and trade the rate differential and the local data flow. The error is running one playbook through both regimes, which guarantees you fight the tape on every transition. Detecting the switch is the hard part, and it is mostly a context read rather than a clean indicator: a spike in cross-asset correlation, equities and FX and credit all moving as one bloc, is the tell that the global bucket has taken over, a pattern the intermarket articles return to. The honest limit is that the weight is latent and you infer it after it has already started moving, so this framework sets the regime gate and never the trigger. It tells you which set of inputs deserves your attention this week. Pair it with the systematic execution from the old article ["Why FX Traders Need Macro but Should Trade Systematically,"](https://aligrithm.com/why-fx-traders-need-macro-but-should-trade-systematically/) where the driver read sets the allowed direction and a rule-based system pulls the trigger inside it. Macro picks the bucket; rules trade it. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_285-visual-0.png) ## KEY POINTS - A currency is pushed by four global drivers and three domestic drivers at once, and the market oscillates over which bucket is in charge. Reading the wrong bucket on the wrong day gets your direction backwards. - Global drivers: global growth (the tide that moves capital toward exporters or havens), commodity prices (splits the universe into exporter winners and importer losers), risk appetite (moves currencies regardless of domestic fundamentals), and geopolitics (often via risk appetite, sometimes direct). - Domestic drivers: monetary policy (number one most of the time), interest rates (the traded relative differential), and politics. These drive when the world is calm. - The blend weight between buckets is not fixed and not observable on schedule, which is why static-coefficient models get the regime wrong exactly when it matters. In risk-off, domestic drivers go to irrelevant and the currency trades as a risk proxy. - This is the non-stationarity behind the old article ["Why FX Traders Must Watch Gold, Rates, and Equities"](https://aligrithm.com/why-fx-traders-must-watch-gold-rates-and-equities/): gold and the risk tape carry weight in the global regime, the rate differential in the domestic regime, and the confirmation that mattered last month can be noise now. - Trade the regime, not the average: identify the bucket first, then read the matching signals. A cross-asset correlation spike (equities, FX, credit moving as one) signals the global bucket has taken over. The weight is latent, so use this as a regime gate and pair it with systematic execution, the way the old article ["Why FX Traders Need Macro but Should Trade Systematically"](https://aligrithm.com/why-fx-traders-need-macro-but-should-trade-systematically/) splits the gate from the trigger. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Trading Systems and Methods - Perry Kaufman (Amazon)](https://www.amazon.com/s?k=Trading+Systems+and+Methods+Perry+Kaufman&ref=aligrithm.com) - [Cybernetic Trading Strategies - Murray Ruggiero (Amazon)](https://www.amazon.com/s?k=Cybernetic+Trading+Strategies+Murray+Ruggiero&ref=aligrithm.com) - [The Art of Currency Trading - Brent Donnelly (Amazon)](https://www.amazon.com/s?k=The+Art+of+Currency+Trading+Brent+Donnelly&ref=aligrithm.com) ### 4.63 PPP and the Big Mac Index: Why Valuation Only Matters Long-Term URL: https://aligrithm.com/ppp-and-the-big-mac-index-why-valuation-only-matters-long-term/ Last updated: 2026-07-27T13:24:46.000Z A Big Mac costs about 30 percent more in dollars when you buy it in Zurich than in New York. Purchasing power parity reads that gap and announces the franc is overvalued. The conclusion is correct and the trade built on it is a grave. You can short the franc against that 30 percent for three years and bleed out long before the gap closes, because the gap closes on a clock that has nothing to do with your margin call. Valuation tells you where the currency should be. It says nothing about when, and in FX the when is the whole problem. Purchasing power parity is the most common framework for valuing a currency, and the Big Mac index is its cartoon version. Both rest on one idea: a basket of goods should cost the same across countries once you convert through the exchange rate, and relative inflation should be the thing that keeps the prices aligned. If a good is cheaper in one country, buyers shift demand there, the demand pulls on the currency, and over a long enough horizon the prices reconverge. That is the law of one price stretched across borders. ## The arithmetic, and why it disappoints The mechanics are simple enough to write on a napkin. $$ S\_{\\text{PPP}} = \\frac{P\_{\\text{domestic}}}{P\_{\\text{foreign}}} \\qquad\\qquad \\%\\Delta S \\;\\approx\\; \\pi\_{\\text{domestic}} - \\pi\_{\\text{foreign}} $$ The left equation is absolute PPP: the fair exchange rate equals the ratio of the two countries' price levels, so if the same basket costs 6.50 in Switzerland and 5.00 in the United States, the franc's fair value against the dollar is 6.50 divided by 5.00, about 1.30\. The Big Mac index just uses one burger as the basket. The right equation is relative PPP: the exchange rate should drift by the inflation differential over time, so if Swiss inflation runs two points below US inflation, the franc should appreciate about two percent a year against the dollar to keep the burger prices in line. Both are clean. Both are nearly useless for trading. The reason is buried in the assumptions. The basket is not actually traded across borders (you cannot arbitrage a haircut or a Zurich rent), goods carry different tax and labor and transport costs in each country, and the convergence force is real but glacial. Orthodox economics piles on enough faulty premises that by the time you try to put PPP on a trade, the edge has leaked out of every joint. PPP gives you a starting point to call a currency cheap or rich. It does not give you a timer, and a valuation signal without a timer is a thesis, not a trade. ## Mispricing is not a countdown The trap is treating "overvalued" as "about to fall." A currency can stay overvalued or undervalued far longer than you can stay short, and the FX graveyard is full of traders who were right about fair value and wrong about their financing. The mispricing is not a coiled spring. It is a slope that the market climbs and descends on its own schedule, driven by flows and positioning that ignore the burger entirely. The cleanest way to hold this is the old Graham line: in the short run the market is a voting machine, in the long run a weighing machine. PPP is the scale in the long run. Day to day, week to week, even year to year, the currency is being voted on by speculators whose decisions run on momentum, carry, risk appetite, and emotion, none of which care that the franc is 30 percent rich. The market oscillates above and below fair value because the voting dominates, and only over the super long term does the weighing pull price back toward the equilibrium the napkin math predicts. Estimates of PPP half-lives in floating rates run to several years, which is to say the reconvergence is real and slower than almost any position you would put on. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## PPP explains, it does not predict This is the same distinction the old article ["The Difference Between Explanation and Prediction in Markets"](https://aligrithm.com/the-difference-between-explanation-and-prediction-in-markets/) drew, applied to valuation. A prediction commits before the outcome and gets dated and scored. An explanation is assembled after, with the answer already visible. PPP is built for explanation. When a currency has spent five years grinding back toward fair value, PPP narrates the move with total confidence and zero risk, because it had the answer the whole time. Run it forward as a dated signal ("the franc is rich, so short it now") and it produces no usable timing and a string of stopped-out shorts. That is why PPP belongs in the regime layer, not the trigger layer. It is the kind of slow, structural context the old article ["Why FX Traders Need Macro but Should Trade Systematically"](https://aligrithm.com/why-fx-traders-need-macro-but-should-trade-systematically/) said macro is for: it tells you which direction the multi-year wind leans, so you do not fight a long-run reconvergence with a structural short forever. It never sets an entry. Valuation chooses the backdrop; positioning and momentum choose the trade. Confuse the two and you have converted a true statement about the next decade into a losing position this quarter. ## Where it earns its keep PPP is not worthless. It is a horizon tool used at the right horizon. For a multi-year allocation, a currency trading 30 percent above its PPP estimate carries a structural headwind that a one-week momentum signal cannot see, and ignoring it means sizing a long-term position against a slow but persistent drag. The honest use is asymmetric: let extreme PPP mispricing veto a structural position in the wrong direction, and let it lean a long-horizon allocation toward the cheap side, but never let it pull a trigger or set a stop. The further price sits from fair value, the more the long-run odds favor reconvergence, and the less that tells you about the next hundred trading days. The discipline is to keep the timeframes from contaminating each other. A trader who lets a multi-year valuation view leak into a short-term book starts holding losers "because the franc is rich anyway," which is the cognitive defect of the voting-machine world wearing the costume of the weighing-machine truth. Valuation matters in the super long term. Positioning and momentum matter for everything shorter, which is everything you will actually trade. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_284-visual-0.png) ## KEY POINTS - Purchasing power parity (and its cartoon, the Big Mac index) values a currency by saying the same basket of goods should cost the same across borders, with relative inflation as the equalizer. Absolute PPP sets fair value as the ratio of price levels; relative PPP says the rate should drift by the inflation differential. - The math is clean and nearly useless for trading. The basket is not arbitrageable across borders, costs and taxes differ by country, and convergence is glacial, with PPP half-lives running to years in floating rates. - A currency can stay mispriced far longer than you can stay short. Mispricing is a slow slope, not a coiled spring, so "overvalued" does not mean "about to fall." - Graham's frame: short run the market is a voting machine (momentum, positioning, emotion), long run a weighing machine (fair value). PPP is the long-run scale; the voting dominates every horizon you actually trade. - PPP explains beautifully after the fact and predicts nothing on a dated basis, the explanation-versus-prediction split from the old article ["The Difference Between Explanation and Prediction in Markets."](https://aligrithm.com/the-difference-between-explanation-and-prediction-in-markets/) Keep it in the regime layer, never the trigger layer. - Honest use is asymmetric and slow: let extreme mispricing veto a wrong-way structural position or lean a multi-year allocation toward the cheap side, but never set an entry or stop with it. Valuation chooses the backdrop, positioning and momentum choose the trade. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Trading Systems and Methods - Perry Kaufman (Amazon)](https://www.amazon.com/s?k=Trading+Systems+and+Methods+Perry+Kaufman&ref=aligrithm.com) - [Cybernetic Trading Strategies - Murray Ruggiero (Amazon)](https://www.amazon.com/s?k=Cybernetic+Trading+Strategies+Murray+Ruggiero&ref=aligrithm.com) - [The Art of Currency Trading - Brent Donnelly (Amazon)](https://www.amazon.com/s?k=The+Art+of+Currency+Trading+Brent+Donnelly&ref=aligrithm.com) ### 4.62 Trading Non-USD Crosses with Vol-Weighted USD Legs URL: https://aligrithm.com/trading-non-usd-crosses-with-vol-weighted-usd-legs/ Last updated: 2026-07-27T13:24:46.000Z Ask a macro trader about CAD/MXN and they will tell you a story about oil versus Mexican rates, about Canada's terms of trade versus Banxico's hiking cycle. The story is not wrong, but it is mostly beside the point on any horizon shorter than a quarter. CAD/MXN is a synthetic cross with no native market of its own. What you actually trade when you click "buy CAD/MXN" is two USD legs stapled together, and the thing that moves your P&L day to day is not the relative macro narrative, it is the relative volatility of those two legs. USD/MXN swings several times harder than USD/CAD, so a naive cross position is dominated by the peso leg whether you meant it to be or not. The old article ["Cross-Pair Signals: Can EUR Predict GBP?"](https://aligrithm.com/cross-pair-signals-can-eur-predict-gbp/) made the structural point that two instruments move together because they share a driver. Every non-USD cross shares the dollar. CAD/MXN is CAD priced through the dollar against MXN priced through the dollar, which means it is a position in two USD pairs at once. $$ \\frac{\\text{CAD}}{\\text{MXN}} \\;=\\; \\frac{\\text{USD/MXN}}{\\text{USD/CAD}} $$ The cross equals the ratio of its two dollar legs. Going long CAD/MXN, meaning long CAD and short MXN, decomposes into being short USD/CAD (long CAD against the dollar) and long USD/MXN (short MXN against the dollar). The dollar exposure in the two legs offsets and what is left is the clean CAD-versus-MXN bet you wanted. The decomposition is exact. The interesting question is how much of each leg to hold. ## Equal notional is not equal risk If you put on the two legs in equal notional size, you do not get a balanced CAD-versus-MXN position. You get a position whose risk is almost entirely the peso, because USD/MXN moves far more per day than USD/CAD. This is the same error the old article ["Why Volatility-Adjusted Position Sizing Matters"](https://aligrithm.com/why-volatility-adjusted-position-sizing-matters/) attacked at the portfolio level: equal dollars is not equal bets, and the loudest instrument hijacks your P&L. Here the two instruments are the two legs of a single cross, and the fix is identical. Size each leg inversely to its own volatility so both contribute the same risk. $$ N\_i \\;\\propto\\; \\frac{1}{\\sigma\_i} \\qquad\\Longrightarrow\\qquad \\frac{N\_{\\text{USD/MXN}}}{N\_{\\text{USD/CAD}}} \\;=\\; \\frac{\\sigma\_{\\text{USD/CAD}}}{\\sigma\_{\\text{USD/MXN}}} $$ The notional `N_i` of each leg scales as one over that leg's volatility `σ_i`, so the risk contribution `N_i · σ_i` comes out equal across the two legs. The ratio of the two notionals is the inverse ratio of the two volatilities: the more volatile leg gets the smaller notional. Because USD/MXN is the louder of the two, it carries less notional than USD/CAD, not more. Put numbers on it the way Donnelly's source does. Suppose you would naively size the direct cross at 130 units. Measure the recent daily volatility of each leg and find USD/MXN running about a third hotter than USD/CAD, so the vol ratio `σ_USD/CAD / σ_USD/MXN` is roughly 0.75\. Apply it and you hold the legs at long 75 USD/MXN against short 100 USD/CAD. The peso leg, the volatile one, takes the smaller notional precisely so it stops dominating the position. Now each leg contributes the same daily risk, and the trade delivers the relative-performance return you were actually betting on instead of a thinly disguised short-peso punt. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_283-visual-0.png) ## Why bother instead of clicking the cross Two reasons, one structural and one practical. The structural one is control: when you build the cross from weighted legs you decide the risk split, instead of inheriting whatever split the volatility ratio happens to impose on a single ticket. You can run the legs at equal risk, or deliberately tilt toward the leg you have more conviction on, or hedge out one leg when its USD pair gets driven by something idiosyncratic. The decomposition turns one opaque bet into two dials you can set. The practical one is liquidity. The direct cross CAD/MXN is thin, and the old FX-clock and bid/ask reality means a thin pair costs you a wider spread and more impact. Both USD legs, USD/CAD and USD/MXN, are far more liquid than the cross built from them, so executing the two legs separately often costs less than the single synthetic ticket your platform would have quoted anyway, since the platform builds it from those same legs and charges you for the construction. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Where this quietly breaks The volatility you weight by is estimated, and the old article on volatility-adjusted sizing already flagged the catch: vol lags. When peso volatility jumps, your weights are stale, the MXN leg is briefly larger in risk terms than you intended, and you are oversized into exactly the move you were trying to balance against. Faster estimators and floors help, but the estimate is never current. Rebalance the leg weights as volatility shifts, and accept that you are always slightly behind. The decomposition is exact in price but not in risk, because the two USD legs are correlated, and that correlation is not constant. When USD/CAD and USD/MXN both spike on a broad dollar move, the dollar exposure you thought cancelled does not fully cancel and you are left holding residual long or short dollar. The cleaner your equal-and-opposite dollar notionals, the smaller that residual, but a correlation-driven mismatch in the moves themselves can still leak USD beta into a position you booked as USD-neutral. Watch the residual dollar exposure, do not assume it is zero. You also now pay two spreads and carry two financing legs instead of one. For a position you hold for weeks the construction cost is trivial against the risk control you gain. For something you flip intraday, two sets of round-trip costs can eat the edge, so the vol-weighted decomposition is a tool for holding a relative-value view, not for scalping a cross. And the whole approach assumes the dollar legs really are the dominant driver, which is true most of the time and false exactly when a country-specific shock hits. A Banxico surprise or a Canadian oil shock drives one leg on its own news, the relative-beta framing stops describing the move, and you are back in the macro story the trade was built to sidestep. That is not a flaw in the method, it is the boundary of where it applies: vol-weighted legs capture the relative-volatility regime that governs the cross on normal days, and a genuine idiosyncratic shock is the regime change that ends it. ## Building it The recipe is short. Measure each leg's recent return volatility, set the leg notionals inversely proportional to those vols so risk contributions match, and orient the legs to express the cross direction you want. Rebalance when vols drift. ```python import numpy as np import pandas as pd def vol_weighted_legs(usd_quote_a: pd.Series, usd_quote_b: pd.Series, gross_notional: float, lookback: int = 20) -> dict: # usd_quote_a, usd_quote_b: e.g. USDCAD and USDMXN price series # cross A/B (long A, short B) = short USD/A leg + long USD/B leg sig_a = np.log(usd_quote_a).diff().rolling(lookback).std().iloc[-1] sig_b = np.log(usd_quote_b).diff().rolling(lookback).std().iloc[-1] inv = np.array([1.0 / sig_a, 1.0 / sig_b]) w = inv / inv.sum() return { "short_usd_a_notional": gross_notional * w[0], # short USD/A -> long A "long_usd_b_notional": gross_notional * w[1], # long USD/B -> short B "vol_a": float(sig_a), "vol_b": float(sig_b), } ``` The function returns the two leg notionals split by inverse volatility. The lookback is an arbitrary dial trading reactivity against noise in the vol estimate, the same trade-off every rolling volatility measure carries, and you should sweep it rather than trust the default. Orient `usd_quote_a` as the leg whose currency you want to be long in the cross and `usd_quote_b` as the leg you want to be short, and the signs follow from the decomposition above. ## KEY POINTS - A non-USD cross like CAD/MXN has no native market. It is two USD legs (USD/CAD and USD/MXN) stapled together, and on short horizons relative volatility, not the macro story, drives the P&L. - Long CAD/MXN decomposes exactly into short USD/CAD plus long USD/MXN. The dollar exposure offsets and the clean CAD-versus-MXN bet remains. - Equal notional on the two legs is not equal risk, the same error from the old article ["Why Volatility-Adjusted Position Sizing Matters."](https://aligrithm.com/why-volatility-adjusted-position-sizing-matters/) The louder leg dominates, so size each leg inversely to its volatility: the more volatile USD/MXN gets the smaller notional (long 75 USD/MXN against short 100 USD/CAD in the worked case). - Building the cross from weighted legs gives you two dials instead of one opaque ticket, and the liquid USD legs often execute cheaper than the thin direct cross, which the platform synthesizes from those same legs anyway. - It breaks where vol estimates lag, where leg correlation leaks residual USD beta into a position booked as dollar-neutral, where double round-trip costs kill an intraday edge, and where a country-specific shock drives one leg on its own news and ends the relative-beta regime. This is a tool for holding a relative-value view, not for scalping. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Trading Systems and Methods - Perry Kaufman (Amazon)](https://www.amazon.com/s?k=Trading+Systems+and+Methods+Perry+Kaufman&ref=aligrithm.com) - [Cybernetic Trading Strategies - Murray Ruggiero (Amazon)](https://www.amazon.com/s?k=Cybernetic+Trading+Strategies+Murray+Ruggiero&ref=aligrithm.com) - [The Art of Currency Trading - Brent Donnelly (Amazon)](https://www.amazon.com/s?k=The+Art+of+Currency+Trading+Brent+Donnelly&ref=aligrithm.com) ### 4.61 Intraday Session Momentum: London Trends, NY Extends and Reverses URL: https://aligrithm.com/intraday-session-momentum-london-trends-ny-extends-and-reverses/ Last updated: 2026-07-27T13:24:45.000Z The London open at 07:00 GMT is a reflection point. Through the Asia session the market drifts on thin flow, and then the largest risk-takers and market-makers sit down at their London desks and price in everything that has accumulated since the previous daily close: the overnight headlines, the Asian data, the positioning that built up while the deep book was asleep. That repricing is not gentle. It often kicks off an aggressive trending move in the first hour or two of London, because a lot of decisions get made at once and shoved through a book that has just gotten deep enough to absorb them. New York opens around 12:00 GMT into whatever London has already built. If London printed a clean directional move, New York desks tend to hop on it and extend it through the start of the overlap. The pattern Donnelly's source material describes, and that you can see on enough days to take seriously, is a three-act structure: London initiates the trend, New York extends it into the overlap, and then later in the New York afternoon the move runs out of fresh participants and reverses as London books square up and the day's flow exhausts itself. A session-based intraday bias, not a chart pattern. This is a claim about conditional behavior tied to the wall clock, and it lives or dies on the timeframe you measure it at. The old article ["How to Choose the Right Timeframe for a Strategy"](https://aligrithm.com/how-to-choose-the-right-timeframe-for-a-strategy/) made the point that noise is a function of timeframe and that the right resolution is upstream of the trend-versus-reversion choice. Session momentum is exactly that idea applied to the time-of-day axis: the continuation edge is a London-into-overlap phenomenon, and the reversal edge is a late-New-York phenomenon, and if you measure on a timeframe or a window that smears the two together you will conclude there is nothing there. ## Decomposing the day into session returns Stop thinking of the day as one return and split it into the pieces the clock actually produces. Define the London session return as the move from the London open to the New York open, and the New York session return as the move from the New York open to the New York close. The bias claim is that these two are not independent: the sign of the London move carries information about what New York does next. $$ r\_{\\text{LDN}} = \\frac{P\_{\\text{NY open}}}{P\_{\\text{LDN open}}} - 1 \\qquad r\_{\\text{NY}} = \\frac{P\_{\\text{NY close}}}{P\_{\\text{NY open}}} - 1 $$ Both are just simple percentage moves over a fixed slice of the clock: `r_LDN` is the London-session return measured from the 07:00 GMT open to the 12:00 GMT New York open, and `r_NY` is the New-York-session return from the 12:00 open to the New York close. The continuation hypothesis says that when `r_LDN` is positive, the early part of `r_NY` is positive too, because New York extends the London trend. The reversal hypothesis says the late part of `r_NY` tends to fade back against `r_LDN`. To test either, you condition the New York move on the sign of the London move and look at the average. $$ \\text{continuation edge} = E\\!\\left\[\\, r\_{\\text{NY}} \\cdot \\operatorname{sign}(r\_{\\text{LDN}}) \\,\\right\] $$ This single number is the whole test. You take each day's New York return, flip its sign so that an up-London day and a down-London day both count "in the direction London was going," and average across all days. If the average is reliably positive, New York extends London on balance. If it is negative, New York fades London on balance. Split the New York session into an early window and a late window and compute the same number for each, and you can see the extend-then-reverse shape directly: positive in the early window, negative in the late one. No chart-reading, no eyeballing, one conditional expectation per window. The honest version of this test reports the hit rate alongside the average, because one or two violent trend days can manufacture a positive expectation that almost never repeats. A session bias that is real shows up as a modest edge on a majority of days, not a huge edge on a handful. That is the same discipline the rest of this work keeps returning to: a few outlier sessions are not a strategy. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_282-visual-0.png) ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Why the edge is fragile and easy to fake The seductive part of a session bias is that it backtests cleanly on a naive engine, and that is exactly where it lies to you. The old article ["Why Bid/Ask Bounce Matters for Intraday FX Systems"](https://aligrithm.com/why-bid-ask-bounce-matters-for-intraday-fx-systems/) is the trap here. If you measure the London and New York session returns off last-trade prices, the bid/ask bounce contaminates both endpoints, and worse, it injects negative autocorrelation between consecutive measurements that a session-momentum test can misread as a real reversal signal. The "New York reverses London" finding can be partly manufactured by the spread oscillation at the session boundaries rather than by any flow. Compute the session returns on mid-price, sample at the open and close marks rather than the noisiest ticks, and require the conditional edge to clear a multiple of the spread before you believe it. The session boundaries also sit near the fixes. The New York reversal window overlaps the 16:00 GMT WMR fix and the late-overlap option expiries, so part of what looks like a clean session reversal is forced fixing flow that mean-reverts on its own schedule. That is not a reason to discard the bias, but it means the reversal you are trading may be a fix artifact wearing a session costume, and a fresh entry into the late New York window can collide with that flow. Then there is the obvious problem: this is a static time-of-day rule, and time-of-day rules decay as the structure that produced them changes. The London-trends, New-York-extends pattern came out of a particular era of who traded when. Algorithmic execution has flattened a lot of intraday seasonality since, so the edge is almost certainly smaller now than the description implies, and you should expect it to keep shrinking. Treat the magnitude as a thing to re-measure on recent data, not a constant. ## Building it without fooling yourself The construction is a per-session conditional bias estimator with the windows pinned to GMT and the prices taken at mid. Tag every bar with its session, collapse each day into the London and New York session returns, and accumulate the sign-conditioned New York return separately for an early and a late window. Report the average and the hit rate for each, and only then decide whether either window clears costs. ```python import numpy as np import pandas as pd def session_bias(mid: pd.Series, tz_gmt_index: bool = True) -> dict: # mid: minute-bar mid-price series, DatetimeIndex in GMT g = mid.groupby(mid.index.normalize()) rows = [] for day, s in g: def at(hour): w = s.between_time(f"{hour:02d}:00", f"{hour:02d}:05") return w.iloc[0] if len(w) else np.nan ldn_open, ny_open = at(7), at(12) ny_mid, ny_close = at(16), s.iloc[-1] if np.isnan([ldn_open, ny_open, ny_mid]).any(): continue r_ldn = ny_open / ldn_open - 1.0 r_ny_early = ny_mid / ny_open - 1.0 r_ny_late = ny_close / ny_mid - 1.0 rows.append((np.sign(r_ldn), r_ny_early, r_ny_late)) a = np.array(rows) sgn, early, late = a[:, 0], a[:, 1], a[:, 2] return { "early_edge": float(np.mean(sgn * early)), "early_hit": float(np.mean((sgn * early) > 0)), "late_edge": float(np.mean(sgn * late)), "late_hit": float(np.mean((sgn * late) > 0)), "n_days": len(a), } ``` The two edge numbers are the sign-conditioned average New York return in each window, and the two hit rates tell you whether the edge is broad or driven by a few days. The windows here are arbitrary dials, the 07:00 and 12:00 opens and the 16:00 split, and you should treat them as such: shift them, sweep them, and see whether the result is a knife-edge or a plateau. A bias that only appears at one exact minute boundary is curve fit. A bias that survives a half-hour wobble in the window edges has a chance of being real. ## KEY POINTS - The day has a structure tied to the clock: London at 07:00 GMT reprices the overnight and initiates a trend, New York at 12:00 extends it into the overlap, and the late New York afternoon reverses it as books square and flow exhausts. - Test it by splitting each day into a London session return and a New York session return, then averaging the New York return conditioned on the sign of the London return. Positive in the early window means continuation, negative in the late window means reversal. - Always report the hit rate next to the average. A real session bias is a modest edge on most days, not a huge edge on a few trend days. - The reversal finding is easy to fake. Bid/ask bounce off last-trade prices injects negative autocorrelation that masquerades as a session reversal (the old article ["Why Bid/Ask Bounce Matters for Intraday FX Systems"](https://aligrithm.com/why-bid-ask-bounce-matters-for-intraday-fx-systems/)), so compute on mid-price and require the edge to clear a multiple of the spread. - The late New York window overlaps the 16:00 WMR fix, so part of the "reversal" can be forced fixing flow, and the whole pattern is a static time-of-day rule that algos have eroded. Re-measure the magnitude on recent data, and sweep the session windows to confirm the edge is a plateau, not a knife-edge (the timeframe discipline from the old article ["How to Choose the Right Timeframe for a Strategy"](https://aligrithm.com/how-to-choose-the-right-timeframe-for-a-strategy/)). --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Trading Systems and Methods - Perry Kaufman (Amazon)](https://www.amazon.com/s?k=Trading+Systems+and+Methods+Perry+Kaufman&ref=aligrithm.com) - [Cybernetic Trading Strategies - Murray Ruggiero (Amazon)](https://www.amazon.com/s?k=Cybernetic+Trading+Strategies+Murray+Ruggiero&ref=aligrithm.com) - [The Art of Currency Trading - Brent Donnelly (Amazon)](https://www.amazon.com/s?k=The+Art+of+Currency+Trading+Brent+Donnelly&ref=aligrithm.com) ### 4.60 The FX Clock: Sessions, Overlaps, and Fixes URL: https://aligrithm.com/the-fx-clock-sessions-overlaps-and-fixes/ Last updated: 2026-07-27T13:24:45.000Z A EURUSD chart looks like one continuous market. It is not. The same pair costs you a different spread at 03:00 GMT than it does at 14:00 GMT, prints fake volatility when Tokyo is the only desk awake, and lurches on a schedule that has nothing to do with anything you would find on a chart. The price runs on a clock, and the clock is set by which humans are at their desks and which mechanical orders are forced through at fixed times. Trade without knowing where you are on that clock and you pay the spread of a thin hour to capture an edge you only measured in a thick one. The clock has two kinds of marks on it: sessions, which are slow tidal shifts in who is trading, and fixes, which are sharp spikes of forced flow at exact minutes. They matter for opposite reasons. Sessions decide your baseline cost and how much of a candle is signal versus noise. Fixes decide when you should not be holding a fresh position into a wall of price-insensitive orders. ## The three desks and the one window that matters Liquidity in FX is a relay race between three regional hubs. Asia (Tokyo, then later China and Singapore) runs the overnight book, thin and prone to air pockets. London opens at 07:00 GMT and is the largest single pool of FX turnover on the planet. New York opens around 12:00 GMT and overlaps London until London winds down in the late afternoon. The window where both London and New York are live, roughly 12:00 to 16:00 GMT, is where the deepest book and the tightest spreads sit. The book is deepest precisely when the two biggest desks are both quoting. That overlap is the only part of the day a price-taker should treat as cheap. Everything outside it costs more, and the further you drift toward the Asia handover the worse it gets. The reason is mechanical, not mystical: spread and market impact both scale inversely with the depth of resting liquidity, and depth is just a headcount of who is quoting. $$ \\text{round-trip cost} \\;\\approx\\; \\text{spread} \\;+\\; k\\,\\frac{Q}{L} $$ Read it plainly. Your cost to get in and out is the quoted spread plus an impact term, where `Q` is your order size, `L` is the resting liquidity available at the top of the book, and `k` is a constant that depends on the pair. When London and New York are both on, `L` is large, so the same order `Q` barely moves the price and the spread term is small too. Drag the same trade into the Asia handover and `L` collapses, so both terms inflate and an edge that looked real at 14:00 GMT quietly evaporates at 23:00 GMT. This is the same machinery the old article ["TWAP and VWAP Are Execution Models, Not Just Indicators"](https://aligrithm.com/twap-and-vwap-are-execution-models-not-just-indicators/) leaned on: you slice a large order into the thick part of the day because that is where `L` is biggest and your footprint is smallest. The Asia book deserves its own warning. Thin liquidity does not just widen the spread, it manufactures motion. The old article ["Why Bid/Ask Bounce Matters for Intraday FX Systems"](https://aligrithm.com/why-bid-ask-bounce-matters-for-intraday-fx-systems/) showed how alternating buyer and seller prints make a spread-sized move every tick with no information behind it. That effect is worst exactly where the book is thinnest, so overnight candles carry the most bounce per unit of real movement. A volatility reading or an indicator crossing computed on the Asia session is mostly microstructure artifact, and any intraday system that does not condition on the session is averaging a clean overlap regime together with a junk overnight one. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_281-visual-0.png) ## The fixes: scheduled walls of price-insensitive flow A fix is a benchmark price struck at a fixed time, and around that time a crowd of orders that do not care about price gets pushed through to be filled at the benchmark. Index funds, corporates hedging cash flows, and option desks all reference these marks, so the flow is large, one-sided, and indifferent to where the price already is. That combination produces a short, sharp distortion that mean-reverts once the forced orders clear. The fix is not a trade signal. It is a hazard you schedule around. The ones that move the G10 book, in GMT order through the European-into-New-York day: - **ECB fix, 13:15 GMT.** The European Central Bank reference rate. Matters most for EUR crosses. - **US economic data, 13:30 GMT.** Not a fix, but the heaviest scheduled news slot of the day. Payrolls, CPI, and the rest land here and the overlap is liquid enough to absorb the reaction. - **Options expiry and second-tier data, \~15:00 GMT.** Large option strikes expiring pin or repel the spot price as dealers hedge their books into the cut. - **WMR benchmark fix, 16:00 GMT.** The 4pm London WM/Reuters fix is the single most important mark in FX. Vast passive flows reference it, so the minutes around 16:00 GMT routinely show a sharp directional push that reverses afterward. - **Bank of Canada noon fix, \~17:00 GMT.** A CAD-specific mark. - **Tokyo and China fixes, around the 22:00 GMT Asia open / NY close handover.** The marks that govern the JPY and CNY books as the day rolls back to Asia. The WMR 16:00 GMT fix is the one that catches systematic traders out. It sits at the tail end of the overlap, the book is still deep enough to look safe, and then a wave of passive month-end or index rebalancing flow shoves the price in one direction for a few minutes before it snaps back. A momentum system that reads the push as a breakout buys the top of a move that was never about value. A mean-reversion system can sometimes fade it, but only if it has sized for the fact that the reversal arrives on no fixed timetable. The practical rule is the same across all of them: know the fix calendar in GMT, and do not let an entry signal fire you into a fresh position in the minutes straddling a major fix unless your strategy is explicitly built to trade that flow. The fixes are predictable in timing and unpredictable in direction, which is the worst combination to be surprised by. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_281-visual-1.png) ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## How to actually use the clock Treat the session as a regime label on every bar before you compute anything. Tag each timestamp as Asia, London-only, the overlap, or New York-only, and let your backtest report performance per session instead of blending them. Most intraday FX edges live in the overlap and die in Asia, and a single aggregate number hides that. If a signal only works in the overlap, that is fine, but you need to see it so you do not trade it overnight where the cost term and the bounce eat the edge. Schedule execution, not just signals. If you have a large order, the old TWAP and VWAP logic says push the bulk of it through the overlap where `L` is largest. If your signal fires in the Asia session, the honest options are to widen your cost assumptions, wait for London, or skip it. Put the fix calendar in your code as a hard filter. A simple no-trade buffer of a few minutes on either side of the WMR, ECB, and Asia fixes removes a class of losses that has nothing to do with whether your idea is right. You are not predicting the fix, you are declining to be standing in front of it. The clock does not give you an edge by itself. It tells you when the edge you already have is cheap to harvest and when it is a trap dressed as a candle. ## KEY POINTS - FX is not one continuous market. It is a relay between Tokyo, London, and New York, and the London-New York overlap (roughly 12:00 to 16:00 GMT) is the deepest book and cheapest spread of the day. - Round-trip cost is spread plus an impact term that scales as order size over resting liquidity. Liquidity is largest in the overlap and collapses into the Asia handover, so the same trade costs far more overnight. - The thin Asia book manufactures fake movement through the bid/ask bounce covered in the old article ["Why Bid/Ask Bounce Matters for Intraday FX Systems,"](https://aligrithm.com/why-bid-ask-bounce-matters-for-intraday-fx-systems/) so volatility and indicator readings on overnight candles are mostly artifact. - Fixes (ECB 13:15, options expiry \~15:00, WMR 16:00, BoC 17:00, Tokyo/China around 22:00 GMT) are scheduled walls of price-insensitive flow with predictable timing and unpredictable direction. The 16:00 WMR fix is the one that traps systematic traders. - Tag every bar with its session and report backtest performance per session, push large orders through the overlap as in the old article ["TWAP and VWAP Are Execution Models, Not Just Indicators,"](https://aligrithm.com/twap-and-vwap-are-execution-models-not-just-indicators/) and hard-code a no-trade buffer around the major fixes. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Trading Systems and Methods - Perry Kaufman (Amazon)](https://www.amazon.com/s?k=Trading+Systems+and+Methods+Perry+Kaufman&ref=aligrithm.com) - [Cybernetic Trading Strategies - Murray Ruggiero (Amazon)](https://www.amazon.com/s?k=Cybernetic+Trading+Strategies+Murray+Ruggiero&ref=aligrithm.com) - [The Art of Currency Trading - Brent Donnelly (Amazon)](https://www.amazon.com/s?k=The+Art+of+Currency+Trading+Brent+Donnelly&ref=aligrithm.com) ### 4.59 The Volatility–Liquidity Tradeoff in FX URL: https://aligrithm.com/the-volatility-liquidity-tradeoff-in-fx/ Last updated: 2026-07-27T13:24:44.000Z The two things a trader wants from a market pull against each other. You want volatility, because volatility is the movement your edge feeds on; a dead-flat currency offers nothing to capture. You want liquidity, because liquidity is what lets you get in and out without paying for the privilege; a thin currency charges you a fortune in slippage to take a position. In FX these two run inverse to each other. The pairs and the moments with the most movement have the least depth, and the pairs and moments with the most depth have the least movement. There is no corner of the market that hands you both at once, and the trader's job is to choose where on the tradeoff to sit rather than to pretend the corner exists. This matters because the inverse relationship is not a static feature you can calibrate once. Both volatility and liquidity move over time and across regimes, and they move together in the worst way: when volatility spikes, liquidity evaporates in the same instant, so the cost of trading explodes exactly when your signals are firing hardest. A system that measures volatility and liquidity in a calm sample and assumes those numbers hold is mispricing its own execution in the only environment that decides whether it survives. This article works the tradeoff, the regime dependence, and the two defenses that follow: volatility-normalized sizing and a strong default toward the liquid majors. ## The inverse relationship Volatility measures how hard a price moves, computed from historical data, and higher volatility means higher risk by construction. Liquidity is harder to pin down but amounts to how much slippage you eat when you transact, and it tracks volume closely; high-volume currencies carry deep books and small slippage. The cleanest statement of the cost side is that expected slippage scales with the size you push through divided by the depth available to absorb it. $$ \\text{expected slippage} \\;\\approx\\; \\frac{\\text{order size}}{\\text{available depth}}, \\qquad \\text{and depth falls as volatility rises} $$ Slippage is roughly your order size over the depth in the book, the cost framing from the old article ["The Real Heart of FX Liquidity."](https://aligrithm.com/the-real-heart-of-fx-liquidity/) The tradeoff lives in the second clause: depth is not constant, it shrinks as volatility rises. Market makers widen spreads and pull size when the price is moving fast, because quoting tight in a fast market is how they get run over. So the denominator collapses precisely when volatility, the thing in your numerator's interest, is highest. A fixed-size order suffers its worst slippage in the high-volatility window, which is the window a mid-price backtest quietly assumes away by filling you at the midpoint in any size. The best currencies to trade are the ones with both high liquidity and high volatility, and the reason that phrase contains a "tradeoff" is that the two rarely coexist. You pick liquid majors and accept calmer movement, or you chase the movement in thinner crosses and pay for it in slippage. Pretending you can have deep books and big clean moves in the same instrument is the assumption that kills small per-trade edges. ## It changes by regime Neither side of the tradeoff sits still. All historical measures of volatility are backward-looking, and the future can differ sharply from the past, so a volatility number is a statement about yesterday, not a guarantee about tomorrow. FX volatility tends to move in sync with global asset-market volatility, though not always; there are stretches where equities are wild and FX is calm, and the reverse. The characteristic shape is the dangerous part: volatility drifts lower in slow motion during calm regimes, then skyrockets when a disturbance hits. It leaks down and explodes up. That asymmetry is why you cannot get comfortable in one environment. A system tuned to the slow calm drift, sized and costed for the depth available in that regime, meets a volatility spike that arrives in a single session and finds its execution assumptions inverted: wider spreads, thinner books, larger slippage, all at once. The instruction that follows is to stay alert to changes in market microstructure so that when the regime shifts you are already positioned for it rather than discovering it through fills. The same regime non-stationarity runs through every relationship in this pillar, the acceptance that systems work until they don't. ## Liquidity also varies by pair and by clock The tradeoff has a cross-sectional version and an intraday version on top of the regime version. Cross-sectionally, liquidity concentrates in the major pairs and thins out in the exotics, which are jumpy and expensive to trade in size. Intraday, liquidity follows the clock: the overlap of the London and New York sessions is the deepest window by a wide margin, because the two largest trading hubs are both open and the major events, US economic data, options expiries, and the benchmark fixes, cluster in that period. The thin Asian hours carry wider spreads and shallower books, so the same order costs more and moves the market more outside the overlap. For a system this means execution quality is a function of which pair and which hour, not a constant. A signal that fires in the liquid majors during the London–New York overlap is cheap to act on; the same signal in an exotic cross during thin Asian hours can cost several times as much to fill, enough to flip a positive expected edge negative. Defaulting to the liquid pairs and the deep hours is not timidity, it is refusing to donate the edge to slippage. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## The two defenses The first defense is volatility-normalized sizing. Scale each position inversely to the instrument's volatility so that a wild pair gets a small position and a calm pair gets a larger one, equalizing the risk each contributes, the discipline from the old article ["Why Volatility-Adjusted Position Sizing Matters."](https://aligrithm.com/why-volatility-adjusted-position-sizing-matters/) $$ \\text{position size} \\;=\\; \\frac{\\text{risk target} \\times \\text{capital}}{\\sigma\_{\\text{instrument}}} $$ The position is the risk you are willing to take times your capital, divided by the instrument's volatility, so doubling the volatility halves the size and the dollar risk stays put. This is what lets a thin, volatile cross and a deep, calm major sit in the same book without the volatile one dominating the P&L. The catch, the same one the sizing article and the carry article flagged, is that volatility is estimated and lags, so when it jumps you are briefly oversized into the spike, which is the same moment liquidity has vanished. Faster estimators and floors soften it; nothing removes it. The second defense is the default that most traders should hold to the liquid majors, the G10\. The thin pairs offer more apparent movement, but the slippage and the gap risk in their low-liquidity windows take back more than the extra movement gives, especially for anything trading at higher frequency where the cost is paid often. The honest version of "trade where the edge is" in FX is "trade where the edge survives execution," and for most strategies that is the deep, liquid pairs in the deep, liquid hours, sized by volatility, with the acceptance that the calm you measured can skyrocket without notice. ## Visualizing the tradeoff ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_280-visual-0.png) ## KEY POINTS - Volatility (the movement your edge feeds on) and liquidity (the depth that keeps slippage small) run inverse to each other in FX. No instrument or moment hands you both at once. - Expected slippage is roughly order size over available depth, and depth shrinks as volatility rises, so a fixed-size order suffers its worst slippage in exactly the high-volatility window a mid-price backtest assumes away. - Both sides move by regime. Volatility is backward-looking, tends to track global asset volatility, and drifts down slowly then skyrockets, so calm-regime execution assumptions invert in a single session when a disturbance hits. - Liquidity varies cross-sectionally (concentrated in majors, thin and jumpy in exotics) and intraday (deepest in the London–New York overlap, thin in Asian hours), so execution quality depends on which pair and which hour, not a constant. - The same signal can be cheap to fill in a liquid major during the overlap and expensive enough in a thin cross during Asian hours to flip its expected edge negative. - First defense: volatility-normalized sizing, position equal to risk target times capital over instrument volatility, so a wild pair gets a small position. The catch is that volatility lags, so you are briefly oversized into a spike when liquidity has vanished. - Second defense: default to the liquid majors in the deep hours. The extra movement in thin pairs is taken back by slippage and gap risk, so "trade where the edge survives execution" beats "trade where the movement is." --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Trading Systems and Methods - Perry Kaufman (Amazon)](https://www.amazon.com/s?k=Trading+Systems+and+Methods+Perry+Kaufman&ref=aligrithm.com) - [Cybernetic Trading Strategies - Murray Ruggiero (Amazon)](https://www.amazon.com/s?k=Cybernetic+Trading+Strategies+Murray+Ruggiero&ref=aligrithm.com) - [The Art of Currency Trading - Brent Donnelly (Amazon)](https://www.amazon.com/s?k=The+Art+of+Currency+Trading+Brent+Donnelly&ref=aligrithm.com) ### 4.58 The Carry Trade: Up the Escalator, Down the Elevator URL: https://aligrithm.com/the-carry-trade-up-the-escalator-down-the-elevator/ Last updated: 2026-07-27T13:24:43.000Z Borrow yen at half a percent, buy the Australian dollar at five, pocket the four-and-a-half-percent gap, and watch the Australian dollar slowly appreciate on top of the yield you are already collecting. The position pays you every day to hold it. The equity curve climbs a smooth, gentle slope for two or three years, the Sharpe ratio looks like something you would frame, and then over four sessions in a risk-off panic the whole thing gives back eighteen months of gains. Carry goes up the escalator and down the elevator. The escalator is the slow grind of the yield pickup and the structural appreciation it drives. The elevator is the unwind, and it does not stop at your floor. The carry trade is the most important structural force in FX and the most dangerous payoff profile a systematic trader can hold without understanding it. The danger is not that it loses money on average. It makes money on average, which is exactly the trap: a strategy with a high hit rate, a smooth ride, and a fat negative tail looks like a great strategy right up until the tail arrives, and the standard tools that should size it down actually size it up at the worst moment. This article works the mechanics, the skew, and the specific way volatility-targeting betrays the carry trader. ## What the trade actually earns The return on a carry position has two parts: the interest differential you collect for holding the high-yielder against the low-yielder, and the change in the spot rate over your holding period. $$ r\_{\\text{carry}} \\;=\\; \\underbrace{(r\_{\\text{high}} - r\_{\\text{low}})}\_{\\text{interest differential (the carry)}} \\;+\\; \\underbrace{\\Delta S}\_{\\text{spot change}} $$ The first term is the carry itself, the yield gap between the currency you are long and the currency you funded with, and it accrues steadily as long as you hold. The second term is the spot move. Theory says the spot should move against you by roughly the differential (the high-yielder should depreciate enough to cancel the yield advantage), and in the long run and across the cleanest samples it partly does. In practice, for long stretches, it does not. The high-yielding currency appreciates instead of depreciating, so the spot term adds to the carry term rather than offsetting it. You get paid the differential and the currency goes up. Both terms positive, month after month. The reason is flow, not fundamentals. The market prefers owning high-yielders and shorting low-yielders, so capital piles into the trade, and that buying pressure pushes the high-yielder up. The appreciation is structural, a crowd leaning the same way, and it persists because the carry keeps paying people to keep leaning. That is the escalator: the yield pickup plus the self-reinforcing appreciation the yield pickup creates. ## The elevator The structural appreciation is the same thing as crowded positioning, and crowded positioning unwinds violently. When risk appetite turns, traders cut the carry trade to reduce risk, which means selling the high-yielder and buying back the low-yielding funding currency. Everyone is on the same side, so the exit is a stampede through one door, and the spot term that spent two years quietly adding to your return flips to a large negative number in days. The low-yielders that funded the trade (the euro, the franc, the yen in recent cycles) are exactly the safe havens that get bought back in a panic, so the funding side and the risk-off flow reinforce each other. Down the elevator. This produces the defining feature of carry: negative skew. The trade wins small and often and loses big and rarely. Picking up the differential day after day is collecting nickels; the unwind is the steamroller. Picking up nickels in front of a steamroller is a fine living until the day you misjudge the steamroller, and the strategy's smoothness is what lulls you into standing in front of it with size. A high Sharpe computed over a sample that has not yet contained a crash is measuring the escalator and pricing the elevator at zero. ## Why volatility-targeting makes it worse Here is the part that catches careful traders. The instinct from the old article ["Why Volatility-Adjusted Position Sizing Matters"](https://aligrithm.com/why-volatility-adjusted-position-sizing-matters/) is to scale the position inversely to volatility, sizing up when the currency is calm and down when it is wild, so that every position contributes equal risk. For most strategies that discipline is the foundation. For carry it is a trap, because of when carry's volatility is low. Carry's volatility is lowest during the long calm grind up the escalator, and it spikes only when the unwind has already begun. Inverse-volatility sizing therefore tells you to hold the largest position during the calm appreciation, which is precisely when the crowd is most leaned-in and the elevator is most loaded, and to cut only after volatility has jumped, which is after the crash has started. The sizing rule maximizes exposure into the steamroller and trims after the first wheel has passed. $$ \\text{position} \\;\\propto\\; \\frac{1}{\\hat{\\sigma}\_t} \\quad\\Rightarrow\\quad \\hat{\\sigma}\_t \\text{ low during the calm grind} \\;\\Rightarrow\\; \\text{position largest right before the unwind} $$ The position is set proportional to one over the estimated volatility, so when the estimate is low the position is large, and because carry's volatility is lowest just before the unwind, the rule puts on its biggest size at the worst possible moment. This is the lag problem the sizing article warned about, sharpened by carry's specific structure: volatility is an estimate that trails the truth, and for a negatively-skewed trade the gap between the calm estimate and the coming reality is the whole risk. The honest fixes are to cap carry exposure regardless of how calm volatility looks, to size off a stressed or floored volatility rather than the realized calm, and to treat the carry differential as the signal while letting a risk-off filter (the macro regime gate from ["Why FX Traders Need Macro but Should Trade Systematically"](https://aligrithm.com/why-fx-traders-need-macro-but-should-trade-systematically/)) pull exposure down before the volatility spike confirms what the macro already said. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Is it a strategy worth running Carry has a real, persistent edge: it is compensation for bearing crash risk, a risk premium, and risk premia do not arbitrage away because someone has to be paid to hold the tail. Against this kind of strategy, run with naive sizing and no regime gate, I am skeptical, because the backtest that has not eaten a crash overstates the Sharpe and understates the drawdown, and the standard volatility-targeting that would protect a normal strategy actively harms this one. Run with crash awareness baked in (capped size, stressed-volatility sizing, a risk-off gate, diversification across funding and target currencies so the whole book is not one risk-appetite bet), it is a legitimate premium harvest. The difference between the two is not the signal. It is whether you respected the elevator while you were riding the escalator. ## Visualizing the escalator and the elevator ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_279-visual-0.png) ## KEY POINTS - A carry trade borrows a low-yielding currency to buy a high-yielding one, earning the interest differential plus whatever the spot rate does over the holding period. - Theory says the high-yielder should depreciate to cancel the yield edge. For long stretches it appreciates instead, because crowded positioning into the trade pushes it up, so both the carry term and the spot term pay you. That is the escalator. - The structural appreciation is crowded positioning, and it unwinds violently when risk appetite turns. Selling the high-yielder and buying back the safe-haven funding currency happens through one door, so the spot term flips to a large loss in days. That is the elevator. - Carry has negative skew: it wins small and often, loses big and rarely. Picking up the differential is collecting nickels in front of a steamroller, and the smooth ride is what lulls you into standing there with size. - A high Sharpe measured over a sample with no crash is pricing the escalator and ignoring the elevator. - Volatility-targeting betrays carry. Carry's volatility is lowest during the calm grind and spikes only after the unwind starts, so inverse-vol sizing holds the largest position right before the crash and cuts only after it begins. - Run carry with crash awareness: cap exposure regardless of calm, size off stressed or floored volatility, use a risk-off macro gate to cut before the volatility spike confirms, and diversify funding and target currencies so the book is not one risk-appetite bet. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Trading Systems and Methods - Perry Kaufman (Amazon)](https://www.amazon.com/s?k=Trading+Systems+and+Methods+Perry+Kaufman&ref=aligrithm.com) - [Cybernetic Trading Strategies - Murray Ruggiero (Amazon)](https://www.amazon.com/s?k=Cybernetic+Trading+Strategies+Murray+Ruggiero&ref=aligrithm.com) - [The Art of Currency Trading - Brent Donnelly (Amazon)](https://www.amazon.com/s?k=The+Art+of+Currency+Trading+Brent+Donnelly&ref=aligrithm.com) ### 4.56 A Short History of Floating FX and Why Central Banks Intervene URL: https://aligrithm.com/a-short-history-of-floating-fx-and-why-central-banks-intervene/ Last updated: 2026-07-27T13:24:42.000Z A dollar is worth something because the United States government says so, and enough of the world agrees to keep using it. There is no gold in a vault behind it, no commodity you can redeem it for, no intrinsic floor. The price of a currency is the market's running opinion of one country's relative value against another, and that opinion is the only thing holding the number up. A trader who forgets this builds models that assume a currency behaves like an asset with fundamentals. It does not. It behaves like a vote that gets re-counted every second, and occasionally the vote-counter steps onto the floor and starts buying. This is worth understanding before you trade FX systematically, because two features of the modern currency system create risks that equity and futures models do not carry: the value is faith-based and floats freely, and a small set of very large players can intervene to move it on purpose. The first means there is no valuation anchor to lean on in the short run. The second means the price process has a jump risk that no historical volatility estimate will warn you about, because the jump is a policy decision and not a market event. ## Faith, not gold The currencies you trade run on a fiat system. "Fiat" means the money is legal tender by government decree, backed by confidence in the issuing country rather than by any redeemable commodity. Before the 1970s the major currencies were pegged through a gold-anchored arrangement, and the exchange rate was a fixed administrative number. When that broke down the majors began to float, and their value has been set by the market ever since, rising and falling on the market's perception of relative value between two economies. The practical consequence for a system is that a currency has no standalone fundamental to mean-revert toward on any tradeable horizon. A stock has earnings and a book value. A bond has a coupon and a face value. A currency pair is one economy priced against another, and the "fair value" anchors that exist, purchasing power parity and the like, operate on horizons of years and get overwhelmed by positioning and flow in the short run. A model that treats a currency level as cheap or rich the way it would treat a stock is fighting the float. The level can stay "wrong" far longer than any position can survive, which is the macro-but-systematic discipline argued in the old article "Why FX Traders Need Macro but Should Trade Systematically": use macro to read the regime, not to call a currency mispriced and wait for justice. ## The players who can move it on purpose The float is not a free-for-all. The G7 and the wider G-20 monitor currency levels, and when they judge a currency badly misaligned they intervene, either verbally or directly. This is the decentralized structure described in the old article ["FX Is Not One Market: Retail vs Wholesale Structure"](https://aligrithm.com/fx-is-not-one-market-retail-vs-wholesale-structure/) with one extra wrinkle: among the wholesale players sits a participant who is not trying to make money and does not face a risk limit. A central bank intervening to defend or weaken its currency will trade size that no profit-seeking desk would, in the direction it has chosen, for reasons that have nothing to do with your signal. Intervention comes in two strengths. Verbal intervention, often called jawboning, is an official statement that a currency is too strong or too weak. It moves the price with no trade at all, because the market front-runs the threat of action. Direct intervention is the central bank actually buying or selling its currency in the market, sometimes alone, sometimes coordinated across several central banks at once. Coordinated intervention is the dangerous version, because the combined size can move a major pair several percent in minutes and reverse a trend that every momentum model was happily riding. ## Why a system should treat intervention as jump risk Model a currency price as a normal diffusion most of the time, with an occasional jump bolted on: $$ \\Delta P\_t = \\underbrace{\\mu\\,\\Delta t + \\sigma\\,\\varepsilon\_t}\_{\\text{ordinary market}} \\;+\\; \\underbrace{J\_t}\_{\\text{intervention jump}} $$ The first part is the ordinary market: a drift term and a volatility term driven by normal flow, the part your historical volatility estimate measures. The second part is the jump, a sudden displacement that fires when a central bank acts or threatens to act. The jump is the problem, because it is absent from almost all of your sample (interventions are rare), so your estimated volatility carries almost no information about it, and it arrives precisely in the pairs and moments where a central bank has a policy reason to care, which is not random with respect to your positions. A momentum system long a currency that has run far enough to anger its central bank is sitting in the exact spot where the jump risk is highest, and the historical vol it used to size the position never saw the jump coming. This is why intervention belongs in the risk model and not the signal model. You will not predict the jump, because it is a discretionary policy choice, so do not try to trade it as a forecast. You can defend against it. Cap position size in pairs with a history of intervention or an active policy band (the yen, the franc in its peg years, several emerging-market currencies), respect that a stop-loss is a market order that slips badly through a gap, the execution reality from the order-type discussion in this pillar, and treat any currency that has appreciated or depreciated into a level officials keep complaining about as carrying a fat left tail that your Gaussian sizing understates. The minor, optional payoff is that a credible verbal threat sometimes marks a usable turning point, but that is a discretionary macro read, not a systematic edge, and it sits inside the same regime gate the macro article insisted on. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Visualizing the float and the jump ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_277-visual-0.png) ## KEY POINTS - Modern currencies run on a fiat system: their value is backed by confidence in the issuing country, not by gold or any redeemable commodity, and the majors have floated freely since the 1970s. - A floating currency has no standalone fundamental to mean-revert toward on a tradeable horizon. Valuation anchors like purchasing power parity work over years and get overwhelmed by positioning and flow in the short run. - Use macro to read the regime, not to declare a currency mispriced and wait, because a floating level can stay "wrong" far longer than a position can survive. - The G7 and G-20 intervene when they judge a currency badly misaligned, either verbally (jawboning, which moves price with no trade) or directly (the central bank buying or selling its own currency). - Coordinated intervention across several central banks can move a major pair several percent in minutes and reverse a trend every momentum model was riding. - Treat intervention as a jump risk in the risk model, not a signal. It is absent from most of your sample, so historical volatility understates it, and it fires in the pairs and moments where officials have a policy reason to act. - Defend against it: cap size in intervention-prone pairs, respect that stops slip through gaps, and treat a currency that has run into a level officials keep complaining about as carrying a fat tail your Gaussian sizing misses. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Trading Systems and Methods - Perry Kaufman (Amazon)](https://www.amazon.com/s?k=Trading+Systems+and+Methods+Perry+Kaufman&ref=aligrithm.com) - [Cybernetic Trading Strategies - Murray Ruggiero (Amazon)](https://www.amazon.com/s?k=Cybernetic+Trading+Strategies+Murray+Ruggiero&ref=aligrithm.com) - [The Art of Currency Trading - Brent Donnelly (Amazon)](https://www.amazon.com/s?k=The+Art+of+Currency+Trading+Brent+Donnelly&ref=aligrithm.com) ### 4.55 Currency Quoting Conventions: Numerator, Denominator, and the Priority Order URL: https://aligrithm.com/currency-quoting-conventions-numerator-denominator-and-the-priority-order/ Last updated: 2026-07-27T13:24:42.000Z Two traders look at the euro. One has it at 1.0850, the other at 0.9217, and both are right. The first quotes EURUSD (dollars per euro), the second quotes USDEUR (euros per dollar), and the two numbers are reciprocals of the same exchange rate. A stock has one price. A currency pair has two ways to write the same price, and which way you write it decides where your profit lands, what your pip is worth, and whether your backtest computed a long or a short. Getting the convention wrong does not throw an error. It silently flips the sign on half your trades and you find out in the equity curve. Every exchange rate is a ratio of two currencies, a numerator over a denominator, and the market settled on a fixed rule for which currency goes on top. The rule is not arbitrary taste. It exists so that one bank quoting to another never has to ask "which way are we quoting this," because the answer is already fixed by convention. This article makes that convention explicit, shows the priority order that decides every major pair, and connects it to the two places it bites a systematic trader: the sign of the position and the currency the PnL arrives in. ## A pair is a ratio, and the order is not free Write a pair as X/Y. The price is how many units of Y it takes to buy one unit of X. EURUSD at 1.0850 means one euro costs 1.0850 dollars. X (the euro) is the base currency, the thing you are pricing. Y (the dollar) is the quote currency, the thing you are pricing it in. Buy the pair and you are long X, short Y; you own euros financed by selling dollars. The decision of which currency becomes X and which becomes Y is fixed by a priority order. The stronger-ranked currency goes in the numerator. The market convention runs: $$ \\text{EUR} \\succ \\text{GBP} \\succ \\text{AUD} \\succ \\text{NZD} \\succ \\text{USD} \\succ \\text{CAD} \\succ \\text{CHF} \\succ \\text{JPY} $$ Read the ranking as "outranks." When you form a pair from two currencies, the one further left becomes the base (numerator) and the one further right becomes the quote (denominator). The euro outranks everything, so it is always the base: EURUSD, EURGBP, EURJPY. The yen sits at the bottom, so it is always the quote: USDJPY, GBPJPY, AUDJPY. The dollar sits in the middle, which is why it shows up as the base against the currencies below it (USDJPY, USDCAD, USDCHF) and as the quote against the currencies above it (EURUSD, GBPUSD, AUDUSD). The convention is what makes a trader say "cable" and know without asking that it means GBPUSD and not USDGBP. This is also the origin of the American-versus-European terms distinction. A pair quoted with the dollar in the denominator (EURUSD, GBPUSD) is in American terms: the price is dollars per unit of foreign currency. A pair quoted with the dollar in the numerator (USDJPY, USDCHF) is in European terms: the price is foreign currency per dollar. Same dollar, two sides of the ratio, and the priority order is what decides which side it lands on for any given pair. ## The sign trap in a backtest The convention matters to a system because the base currency defines the direction of a long. Long EURUSD is long euros against dollars. Long USDJPY is long dollars against yen. If your signal generator decides "buy the dollar" and you express it by going long whatever pair the dollar appears in, you will be correct on USDJPY (dollar is the base, long the pair is long the dollar) and exactly backward on EURUSD (dollar is the quote, long the pair is short the dollar). The same dollar view produces opposite positions depending on where the priority order parked the dollar in each pair. This is the mechanical reason a currency-strength view has to be translated through the quoting convention before it becomes an order, the same decoupling problem worked in the old article ["Currency Strength Models from Pair Decomposition."](https://aligrithm.com/currency-strength-models-from-pair-decomposition/) A strength model hands you a view on the currency. The convention tells you the sign to trade in each pair that contains it. Skip the translation step and the backtest runs clean, throws no error, and reports a sign-scrambled equity curve that looks like noise because half the trades are inverted. The defense is boring and non-negotiable: store every pair in a canonical orientation, decide the sign of the position in terms of the base currency, and never let the signal code touch a pair without knowing which currency sits in the numerator. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Where the PnL lands The second consequence is that profit accrues in the quote currency, the denominator, which is not always your account currency. This is the mechanics worked in full in the old article ["How FX PnL Actually Works,"](https://aligrithm.com/how-fx-pnl-actually-works/) and the quoting convention is what determines it. PnL comes out in units of Y, and you may have to convert. $$ \\text{PnL} = \\text{Position size} \\times \\Delta P \\quad \\text{(in units of the quote currency Y)} $$ Take EURUSD for a dollar-denominated account. A one-million-euro position that moves 20 pips earns 0.0020 times one million, or 2,000 dollars, and because the quote currency is the dollar the profit is already in your account currency. One step. Now take USDJPY for the same dollar account. A ten-million-dollar position that moves five pips earns 0.05 times ten million, or 500,000 yen, because the quote currency is the yen. To know that in dollars you divide by the current rate: 500,000 divided by roughly 101.65 is about 4,956 dollars. Two steps, and the pip is not even worth a constant number of dollars over the life of the trade, because the conversion rate moves while you hold. The systematic consequence is that "dollars per pip" is not a property of the pair, it is a property of the pair times your account currency times the current rate. A book that sizes every pair off one fixed dollars-per-pip number is not running equal risk across pairs, it is running whatever risk the quoting convention and the conversion rate happen to hand it. Equal-risk sizing has to convert each pair's pip value into the account currency first, which is the input the volatility-scaling in the old article ["Why Volatility-Adjusted Position Sizing Matters"](https://aligrithm.com/why-volatility-adjusted-position-sizing-matters/) assumes you already computed correctly. ## Visualizing the priority ladder ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_276-visual-0.png) ## KEY POINTS - Every exchange rate is a ratio X/Y, read as units of the quote currency Y per one unit of the base currency X. EURUSD at 1.0850 means one euro costs 1.0850 dollars. - Which currency becomes the base is fixed by a priority order, EUR over GBP over AUD over NZD over USD over CAD over CHF over JPY, with the higher-ranked currency taking the numerator. - The dollar sits in the middle of the ladder, so it is the base against the currencies below it (USDJPY, USDCAD, USDCHF) and the quote against the currencies above it (EURUSD, GBPUSD, AUDUSD). That split is the American-versus-European terms distinction. - Long a pair means long the base, short the quote. A single "buy the dollar" view produces a long in USDJPY and a short in EURUSD, so a strength view must be translated through the convention before it becomes an order. - The sign trap is silent: trading the dollar long in every dollar pair inverts half your positions, runs without error, and shows up only as a scrambled equity curve. - PnL accrues in the quote currency. EURUSD profit lands in dollars in one step; USDJPY profit lands in yen and needs a second conversion step, and the pip value drifts with the conversion rate. - "Dollars per pip" is not a constant of the pair. Equal-risk sizing has to convert each pair's pip value into the account currency before volatility scaling, or the book quietly over-weights whatever pair the convention favored. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Trading Systems and Methods - Perry Kaufman (Amazon)](https://www.amazon.com/s?k=Trading+Systems+and+Methods+Perry+Kaufman&ref=aligrithm.com) - [Cybernetic Trading Strategies - Murray Ruggiero (Amazon)](https://www.amazon.com/s?k=Cybernetic+Trading+Strategies+Murray+Ruggiero&ref=aligrithm.com) - [The Art of Currency Trading - Brent Donnelly (Amazon)](https://www.amazon.com/s?k=The+Art+of+Currency+Trading+Brent+Donnelly&ref=aligrithm.com) ### 4.50 Measuring Fractal Dimension Directly from Price URL: https://aligrithm.com/measuring-fractal-dimension-directly-from-price/ Last updated: 2026-07-27T13:24:39.000Z The old article ["The Hurst Exponent from Fractal Dimension"](https://aligrithm.com/the-hurst-exponent-from-fractal-dimension/) left you with a promise: H equals two minus D, so if you can measure the fractal dimension D of the price curve you get the persistence reading for free, no spectrum required. This article cashes that promise. The old article ["Price Density: A Visual Way to Measure Market Choppiness"](https://aligrithm.com/price-density-a-visual-way-to-measure-market-choppiness/) already gave you the intuition, counting how many times price covers its own box, and fractal dimension is that same intuition made into a number you can compute in three lines on any bar series. ## Fractal dimension by covering a shape with boxes The textbook way to find a fractal dimension is box-counting. Cover a shape with boxes of size S, count how many boxes N it takes, then shrink the boxes and count again. The dimension is how fast the count grows as the boxes shrink. $$ \\frac{N\_2}{N\_1} = \\left(\\frac{S\_1}{S\_2}\\right)^{D} \\qquad\\Longrightarrow\\qquad D = \\frac{\\log(N\_2/N\_1)}{\\log(S\_1/S\_2)} $$ Check it on shapes you know. A straight line one meter long: with one-meter boxes you need 10 to cover a ten-meter span, with tenth-of-a-meter boxes you need 100, so D equals log(100/10) over log(1/0.1), which is one. A line is one-dimensional. Do the same on a filled square and you need 10,000 of the small boxes against 100 of the large, giving D equal to two. A square is two-dimensional. A price curve is neither: it is a wiggly line stuck somewhere between dimension one and dimension two, and where it sits between them is exactly its choppiness. ## The shortcut that works because bars are evenly spaced Covering a real price curve with literal boxes is fiddly, but price has a feature that hands you a shortcut: the bars are uniformly spaced in time. With even spacing, the box count over any interval is approximately the curve's average slope over that interval, which is just the price range divided by the length of the interval. $$ N\_i = \\frac{\\text{Highest} - \\text{Lowest}}{\\text{Interval}} $$ Now build the estimate from two equal windows so you have a small box and a large box to compare, the two scales box-counting needs. Take a lookback of 2T bars and split it. Interval 1 covers the most recent T bars (now back to T bars ago), interval 2 covers the older T bars (T back to 2T bars ago), and interval 3 covers the whole 2T span at once. Compute a range-over-length count for each, then combine them. $$ N\_1 = \\frac{H\_1 - L\_1}{T}, \\quad N\_2 = \\frac{H\_2 - L\_2}{T}, \\quad N\_3 = \\frac{H\_3 - L\_3}{2T} \\qquad\\Longrightarrow\\qquad D = \\frac{\\log(N\_1 + N\_2) - \\log(N\_3)}{\\log 2} $$ The numerator compares the fine-scale count (the two short windows added) against the coarse-scale count (the one long window), and the log-base-two is there because the long window is twice the length of each short one, so S1 over S2 is two. This is Ehlers' practical fractal dimension index, and it is cheap enough to run on every bar. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## A worked reading and how to interpret it Put numbers on it. Take T equal to 30, so a 60-bar lookback. Suppose the recent 30 bars ran from a low of 100 to a high of 106, the prior 30 bars from 101 to 105, and the full 60 bars from 100 to 106\. Then N1 is 6 over 30, which is 0.20, N2 is 4 over 30, which is about 0.133, and N3 is 6 over 60, which is 0.10\. The sum N1 plus N2 is 0.333\. D is log(0.333) minus log(0.10), all over log(2): that is log(3.33) over log(2), roughly 1.20 over 0.693, about 1.74\. A dimension near 1.74 is high, close to the box-filling limit of two, so this market is choppy, and through H equals two minus D it implies a Hurst near 0.26, deep in anti-persistent territory. The reading and the geometry agree: a market whose price retraces enough to nearly fill its box is mean-reverting. Flip the picture. If both short windows trend cleanly in the same direction, their combined range barely exceeds the full window's range, so N1 plus N2 sits close to N3, the log difference shrinks toward zero, and D drops toward one. A dimension near one is a clean directional line, H near one, the trend environment. This is the price-density measure from the old article turned into a single bounded number: high D is the wall-to-wall box of chop, low D is the white box with a diagonal smear of a trend, and you no longer have to read it by eye. Two failure modes carry over from the old price-density and Hurst articles and deserve a flag. The estimate is jumpy on short windows, so smooth it before you act on it, and a single wide bar or a gap inflates a range and can spike D for one reading, so prefer true range and treat sudden jumps as suspect rather than as a regime change. As always, this is a measurement of the recent sample, not a forecast: it tells you what the last 2T bars looked like, and it carries the same length-dependence warning the Hurst article hammered. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_244-visual-0.png) ## KEY POINTS - Box-counting defines fractal dimension by how fast the box count grows as boxes shrink, D = log(N2/N1) / log(S1/S2); a line gives D = 1, a filled square gives D = 2, and a price curve sits between. - Because bars are evenly spaced, the box count over an interval is approximately range over length, N\_i = (Highest - Lowest) / Interval, which turns the abstract method into a cheap per-bar computation. - Split a 2T lookback into two T-bar halves plus the full span and combine: D = \[log(N1 + N2) - log(N3)\] / log(2). This is Ehlers' practical fractal dimension index. - High D (near 2) means the curve nearly fills its box: choppy, mean-reverting, and via H = 2 - D it maps to low Hurst. Low D (near 1) is a clean trend, high Hurst. This is the old price-density measure turned into one bounded number. - A worked T=30 example with N1=0.20, N2=0.133, N3=0.10 gives D = 1.74 and H = 0.26, deep anti-persistent territory, matching the choppy geometry. - It is jumpy on short windows (smooth it), distorted by single wide bars and gaps (use true range, distrust spikes), and a measurement of the recent sample rather than a forecast, carrying the same length-dependence as the Hurst estimate. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Trading Systems and Methods - Perry Kaufman (Amazon)](https://www.amazon.com/s?k=Trading+Systems+and+Methods+Perry+Kaufman&ref=aligrithm.com) - [Cybernetic Trading Strategies - Murray Ruggiero (Amazon)](https://www.amazon.com/s?k=Cybernetic+Trading+Strategies+Murray+Ruggiero&ref=aligrithm.com) - [The Art of Currency Trading - Brent Donnelly (Amazon)](https://www.amazon.com/s?k=The+Art+of+Currency+Trading+Brent+Donnelly&ref=aligrithm.com) ### 4.49 The Hurst Exponent from Fractal Dimension URL: https://aligrithm.com/the-hurst-exponent-from-fractal-dimension/ Last updated: 2026-07-27T13:24:39.000Z The old article ["Efficiency Ratio Explained for Traders"](https://aligrithm.com/efficiency-ratio-explained-for-traders/) gave you one number on a zero-to-one scale that says how much of a market's motion became net direction. The old article ["Variance Ratio Tests for Traders"](https://aligrithm.com/variance-ratio-tests-for-traders/) gave you another, built from how variance grows with horizon, that says whether moves reinforce or reverse. The Hurst exponent is a third member of the same family, and the three agree more often than not because they all measure persistence. What makes Hurst worth knowing is its plumbing: it connects the spectral slope from the noise-color view to the fractal dimension of the price curve, three different measurements of one property tied together by two short equations. ## Hurst lives on the same scale as the efficiency ratio The Hurst exponent, written H, runs between zero and one, and the reading splits at one half. $$ H < 0.5 \\;\\;\\text{anti-persistent (mean-reverting)} \\qquad H = 0.5 \\;\\;\\text{random walk} \\qquad H > 0.5 \\;\\;\\text{persistent (trending)} $$ A market with H below one half is anti-persistent: an up move is more likely to be followed by a down move, motion cancels, and this is the fade environment the old efficiency-ratio article flagged with a low reading. A market with H above one half is persistent: moves reinforce, the trend environment. H at exactly one half is the random walk, no memory either way. This is the variance-ratio result wearing a different number; a variance ratio above one (moves reinforcing) corresponds to H above one half, and a ratio below one to H below one half. Same axis, three instruments. ## Two equations connect Hurst to slope and to shape The reason Hurst is more than a fourth redundant indicator is that it bridges the two views from the noise-color article: the spectral slope and the geometric roughness of the curve. The first bridge ties H to the spectral exponent alpha. $$ H \\;=\\; 1 - \\frac{\\alpha}{2} $$ Recall from the old article on noise colors that alpha is the slope of the log power spectrum. White noise (alpha = 0) gives H = 1, brownian noise (alpha = 2) gives H = 0; the often-quoted "H = 0.5 is a random walk" applies to the price increments, where the bookkeeping lands at one half. The point is that you do not need a separate machine for Hurst, you can read it off the spectral slope you already estimated. The second bridge is more practical, because it ties H to the fractal dimension D of the price curve, a quantity you can measure directly off the chart without any spectrum at all. $$ H \\;=\\; 2 - D $$ The fractal dimension D measures how thoroughly the wiggly price line fills the plane: a clean directional line has D near 1, a series so choppy it nearly fills its box has D near 2\. So a smooth trending curve (low D, near 1) gives high H (near 1, persistent), and a jagged choppy curve (high D, near 2) gives low H (near 0, anti-persistent). The geometry of the line and its memory are the same fact. The old article on noise colors gave you the spectral route, H equals one minus alpha over two; this gives you the geometric route, H equals two minus D, and the next article shows how to get D straight off the bars. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## The catch: it is estimated, not computed Here is where most write-ups stop and most traders get burned. The efficiency ratio is computed: feed it a window and it returns an exact number. The Hurst exponent is estimated: there is no closed-form value, only a number you fit, and the fit depends on choices that move it around. Worst of all, H changes dramatically with the length of the data used. Run the estimate over 200 bars and over 2000 bars of the same instrument and you can get materially different exponents, not because the market changed but because the estimator is sensitive to scale and to the range of horizons you let it see. This is the same warning the old variance-ratio article gave from its own angle: the persistence reading is a property of the sample and the window, noisy on short samples, distorted by fat tails, and capable of vanishing out of sample. So treat H as a regime label with error bars, never a precise constant. Report it as a profile across data lengths rather than a single figure, the way the old efficiency-ratio article insisted on a short/horizon/long reading instead of one number. If H is comfortably above one half across several window lengths, the persistence is probably real and a trend approach has structural support; if it hovers near one half or flips with window length, you are looking at a random walk dressed up by estimation noise, and any strategy keyed to it is fitting the estimator, not the market. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_243-visual-0.png) ## KEY POINTS - The Hurst exponent H runs 0 to 1: below 0.5 is anti-persistent (mean-reverting, the fade environment), 0.5 is a random walk, above 0.5 is persistent (trending). It sits on the same persistence axis as the efficiency ratio and the variance ratio. - Two equations connect it to the noise-color view: H = 1 - alpha/2 ties it to the spectral slope alpha, and H = 2 - D ties it to the fractal dimension D of the price curve. - A smooth directional curve has low fractal dimension (D near 1) and high H (persistent); a jagged box-filling curve has high D (near 2) and low H (anti-persistent). Geometry and memory are the same fact. - H is estimated, not computed: there is no exact value, only a fit, and it changes dramatically with the length of data used, echoing the variance-ratio warning that persistence readings are sample properties. - Use it as a regime label with error bars: report a profile across window lengths, trust a trend read only when H stays above 0.5 across several lengths, and distrust any reading that flips with window size as estimation noise. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Trading Systems and Methods - Perry Kaufman (Amazon)](https://www.amazon.com/s?k=Trading+Systems+and+Methods+Perry+Kaufman&ref=aligrithm.com) - [Cybernetic Trading Strategies - Murray Ruggiero (Amazon)](https://www.amazon.com/s?k=Cybernetic+Trading+Strategies+Murray+Ruggiero&ref=aligrithm.com) - [The Art of Currency Trading - Brent Donnelly (Amazon)](https://www.amazon.com/s?k=The+Art+of+Currency+Trading+Brent+Donnelly&ref=aligrithm.com) ### 2.74 Legendre-Polynomial Trend and Trend Relative to Local Variation URL: https://aligrithm.com/legendre-polynomial-trend-and-trend-relative-to-local-variation/ Last updated: 2026-07-27T13:23:32.000Z Ask most indicators whether the market is trending and they answer with direction: price is above its average, so up. The old article ["CMMA: A Better Momentum Primitive Than Price-minus-MA Alone"](https://aligrithm.com/cmma-a-better-momentum-primitive-than-price-minus-ma-alone/) already sharpened that into a volatility-scaled distance, and the old article ["Decyclers: Extracting Trend by Removing Cycle Energy"](https://aligrithm.com/decyclers-extracting-trend-by-removing-cycle-energy/) pulled trend out by deleting the cyclic part of the signal. Both still describe where price sits. Neither tells you whether the move was a clean, straight climb or a drunk stagger that happened to end higher. A grinding, low-noise advance and a violent chop with the same net displacement are different trades, and you want an indicator that scores them differently. Fitting a line with Legendre polynomials, then dividing by the local scatter, does exactly that. ## Trend as a least-squares slope, not a difference Price-minus-average compares two points and calls the gap a trend. A regression uses every bar in the window. Fit a straight line to the recent log prices by least squares and its slope is a far more stable measure of direction than any single difference, because it pools all the bars instead of leaning on the endpoints. The trick is how you fit it cheaply and cleanly, and that is what Legendre polynomials buy you. The first-degree Legendre polynomial over a lookback window is a straight ramp that runs symmetrically from negative to positive, sums to zero across the window, and is scaled to unit length. Call it the linear basis vector. Because it is centered and normalized, projecting the window of log prices onto it (the dot product) returns the slope of the best-fit line directly, with no matrix inversion and no intercept to estimate. $$ c\_1(i) = \\frac{i - \\bar{\\imath}}{\\sqrt{\\sum\_{j}\\left(j - \\bar{\\imath}\\right)^2}}, \\qquad \\text{Trend} = \\sum\_{i=1}^{L} c\_1(i)\\, \\ln P\_i $$ The term c-one is the normalized linear basis: i indexes the bars in the window, i-bar is the middle bar, so the numerator is just each bar's distance from center, and the denominator rescales the whole ramp to unit length. The trend is the dot product of that ramp with the log prices in the window. A steadily rising series lines up with the ramp and produces a large positive number; a falling series produces a large negative one; a flat or symmetric series cancels against the centered ramp and produces near zero. Use log prices so the slope reads as a compounding rate rather than a dollar amount, the same reason the old CMMA primitive worked in returns rather than levels. The basis does not stop at a line. The second-degree Legendre polynomial is a centered parabola, and projecting onto it measures curvature, whether the trend is accelerating or rolling over. The third-degree one is a cubic that captures an S-shape. Each higher polynomial is orthogonal to the ones below it, so the linear, quadratic, and cubic coefficients carry independent pieces of the move and you can read slope, acceleration, and reversal as separate signals from the same fit. ## Dividing by the gyrations The slope alone still cannot tell the clean climb from the lucky stagger, because both can share the same slope. The information you are missing is how far the prices strayed from the fitted line along the way. A tight fit means the move tracked the line bar by bar, a real trend. A loose fit means the prices wandered all over and the line is a polite fiction. Measure that scatter as the root-mean-square of the residuals around the fit, the within-window gyrations, and divide the trend by it. $$ \\text{TrendRel} = \\frac{\\text{Trend}}{\\sqrt{\\dfrac{1}{L}\\sum\_{i=1}^{L}\\left(\\ln P\_i - \\widehat{\\ln P\_i}\\right)^2}} $$ The numerator is the slope from before. The denominator is the typical distance between the actual log price and the line you fitted to it, averaged over the window. Their ratio is a signal-to-noise reading of the trend: the move scaled by how much of the window was gyration rather than progress. A clean, low-residual advance produces a large ratio even at a modest slope, while a steep but jagged move with the same displacement gets discounted by its own noise. This is the same shape as a regression t-statistic, slope over the spread of the residuals, and it answers a sharper question than "which way" — it answers "how much of this move should I believe." Two practical notes. Raw ratios like this grow heavy tails when a near-flat window drives the denominator toward zero, so pass the result through a normal cumulative distribution function to compress the extremes into a bounded, well-behaved range, the same taming step the CMMA and moving-average-difference indicators use. And the window length is the one real choice: a short window reacts fast and reports local trend with lots of variance, a long window is stable but laggy, and the right length is the horizon you actually trade rather than a number an optimizer liked in-sample. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Where it fits against what you already have The decyclers article got trend by subtracting the cyclic energy, a filter-side answer. This is the regression-side answer to the same question, and the two are complementary: the decycler cleans the series, the Legendre fit measures the slope and, more usefully, the conviction behind it. Against CMMA, the Legendre trend is the multi-bar generalization. CMMA scales a single displacement by volatility; the Legendre-relative trend scales a whole-window slope by the window's own scatter, so it rewards persistence and punishes chop in a way a two-point distance cannot. Use it as a regime gauge, gate other signals on it, or feed the linear, quadratic, and relative-trend values straight into a model as a compact description of the recent move that says direction, acceleration, and trustworthiness in three numbers. ## Visualizing trend relative to variation ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_275-visual-0.png) ## KEY POINTS - Direction indicators say where price sits; they cannot tell a clean straight climb from a jagged stagger with the same net displacement. Those are different trades. - Fit a line to the window of log prices by least squares: the slope pools every bar and is far more stable than any single price-minus-average difference. - The first-degree Legendre polynomial is a centered, unit-length linear ramp. Its dot product with the log prices returns the best-fit slope directly, with no matrix inversion and no intercept to estimate. - Higher Legendre polynomials are orthogonal: the second-degree parabola measures curvature (acceleration or roll-over) and the third-degree cubic captures an S-shaped reversal, three independent signals from one fit. - Divide the slope by the root-mean-square of the residuals around the fit, the within-window gyrations. The ratio is a signal-to-noise reading: a clean low-residual move scores high even at a modest slope; a steep jagged move with the same displacement gets discounted by its own noise. - The construction is the shape of a regression t-statistic, slope over residual spread, and it answers "how much of this move should I believe," not just "which way." - Compress the ratio through a normal CDF to bound heavy tails when a flat window drives the denominator toward zero, and pick the window length as the horizon you trade. - It is the regression-side complement to the filter-side trend of the old article ["Decyclers: Extracting Trend by Removing Cycle Energy,"](https://aligrithm.com/decyclers-extracting-trend-by-removing-cycle-energy/) and the multi-bar generalization of the old article ["CMMA: A Better Momentum Primitive Than Price-minus-MA Alone."](https://aligrithm.com/cmma-a-better-momentum-primitive-than-price-minus-ma-alone/) --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Statistically Sound Indicators for Financial Market Prediction - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Statistically+Sound+Indicators+for+Financial+Market+Prediction+Timothy+Masters&ref=aligrithm.com) - [Cycle Analytics for Traders - John Ehlers (Amazon)](https://www.amazon.com/s?k=Cycle+Analytics+for+Traders+John+Ehlers&ref=aligrithm.com) ### 4.57 Currency Personality and Correlation Blocs URL: https://aligrithm.com/currency-personality-and-correlation-blocs/ Last updated: 2026-07-27T13:24:43.000Z A trader builds what looks like a diversified short-euro book: short EURUSD, short USDSEK turned into a euro short, a position in the Norwegian krone, a little Swiss franc. Five tickets, one view. When EURUSD moves, all five move together, because the Swedish krona, the Norwegian krone, the Danish krone, and the Swiss franc are not independent currencies in any tradeable sense; they are satellites whose dollar exchange rate is mostly set by EURUSD. The trader thought the book held five bets. It held one bet wearing five tickets, sized as if it were five, which means it carried roughly the risk of a single position five times too large. Currencies have personalities, and currencies with similar personalities correlate hard enough that treating them as separate names is a sizing error. The personalities are not folklore. They come from what actually drives each currency: a shared regional anchor, an exported commodity, a yield level that makes the currency a haven or a risk asset. Group the currencies by driver and you get a small number of blocs, and inside a bloc the correlation is high enough that the bloc, not the individual currency, is the real unit of risk. This article maps the blocs, explains the driver behind each, and shows why the bloc structure has to enter the risk model before a currency book can call itself diversified. ## The European bloc The cleanest bloc is European. The euro, the Swiss franc, the Swedish krona, the Norwegian krone, and the Danish krone trade as a cluster, and their exchange rate against the dollar is mostly determined by EURUSD. Geography and trade integration tie these economies to the euro area, so when EURUSD moves on a dollar story or a euro story, the satellites follow. The Danish krone is the extreme case, held in a tight band against the euro by policy, so DKKUSD is almost a scaled copy of EURUSD. The frank and the Scandinavian currencies have more of their own personality but still take their primary cue from the euro. The British pound sits at the edge of this bloc. It gets lumped into the European basket because of trade and proximity, but its links are looser than the others, and it carries enough of its own domestic story that treating it as a pure euro satellite is wrong. Sterling is in the room, not in the cluster. The systematic point: a position in two or more European-bloc currencies against the dollar is mostly the same EURUSD bet repeated. Decompose the pairs into per-currency strength and the redundancy is obvious, the matrix view from the old article ["Currency Strength Models from Pair Decomposition."](https://aligrithm.com/currency-strength-models-from-pair-decomposition/) The bloc is what the strength model recovers as a shared factor, and trading several members of it is loading that one factor several times. ## The commodity bloc The second bloc is the commodity currencies: the Australian dollar, the New Zealand dollar, and the Canadian dollar among the majors, with the Brazilian real, the South African rand, and the Chilean peso extending the idea into emerging markets. These come from commodity-exporting economies, and their currencies move with the prices of what they sell. The links are specific, not generic: - The Australian dollar tracks copper, gold, and iron ore. - The Canadian dollar tracks crude oil and natural gas. - The New Zealand dollar tracks soft commodities, dairy above all. This is the per-currency driver map from the old article ["Why FX Traders Must Watch Gold, Rates, and Equities"](https://aligrithm.com/why-fx-traders-must-watch-gold-rates-and-equities/) read from the currency's side: the right external market for the Canadian dollar is crude, and watching gold for the Canadian dollar adds noise where watching crude adds signal. The commodity link is also why these currencies do not move as one bloc as tightly as the European cluster does. The Australian dollar and the Canadian dollar share the "commodity exporter, risk-sensitive" personality, but iron ore and crude can diverge, so the intra-bloc correlation is real but looser than the EURUSD-anchored cluster. ## The yield personality: havens and risk currencies Cutting across the regional and commodity blocs is a yield axis that sets risk-on and risk-off behavior. Low-yielding currencies behave as safe havens. In recent cycles the euro, the franc, and the yen have carried the lowest yields, and they strengthen when markets are frightened because they are the funding currencies traders buy back when they cut risk. High-yielding currencies, the Australian and New Zealand dollars among the majors and the emerging-market currencies beyond them, are the risk currencies; they pay you to hold them in calm times and sell off hard during stress. This yield personality is what makes the commodity bloc and the risk axis overlap. The same Australian dollar that tracks iron ore is also a high-yielder that dumps in a panic, so it carries two correlated reasons to fall together with other risk assets when fear hits. That overlap is the carry behavior worked separately in the article on the carry trade, and it is the reason a "diversified" basket of high-yielders is really one concentrated risk-appetite bet, the correlation-to-one problem the macro article flagged for 2008. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Why the bloc structure has to enter the risk model Treating correlated currencies as independent names breaks the sizing arithmetic. The risk of a basket is not the sum of the individual risks; it depends on the correlations, and inside a bloc those correlations are high. $$ \\sigma\_{\\text{book}}^2 = \\sum\_{i} w\_i^2 \\sigma\_i^2 \\;+\\; \\sum\_{i \\neq j} w\_i w\_j \\,\\rho\_{ij}\\, \\sigma\_i \\sigma\_j $$ The variance of the book is the sum of each position's own variance (the first term) plus a cross term for every pair of positions, scaled by their correlation. When the correlations between members of a bloc are near one, the cross terms are almost as large as the variance terms, and a five-position European book has a variance close to that of one position five times the size. The naive calculation, which assumes the correlations are zero and keeps only the first term, reports a fifth of the true risk. The book is concentrated and the risk number says it is diversified. The fix is to size at the bloc level. Estimate the correlation structure, recognize that several European-bloc shorts are one factor exposure, and budget risk to the factor rather than to the ticket count. This is the same currency-strength decomposition turned into a risk control: solve for the shared drivers, size against the drivers, and stop counting correlated tickets as independent bets. The honest caveat is that the correlations are not stable; the European bloc holds tighter than the commodity bloc, and every bloc tightens toward one in a crisis, so the structure you measured in calm markets understates the concentration in the panic where it matters most. ## Visualizing the blocs ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_278-visual-0.png) ## KEY POINTS - Currencies have personalities set by their drivers, and currencies with similar drivers correlate hard enough that the bloc, not the individual currency, is the real unit of risk. - The European bloc (euro, franc, Swedish krona, Norwegian krone, Danish krone) takes its dollar exchange rate mostly from EURUSD. The Danish krone is held in a band and is almost a scaled EURUSD; sterling is loosely attached, not in the cluster. - The commodity bloc tracks specific exports: the Australian dollar on copper, gold, and iron ore; the Canadian dollar on crude and natural gas; the New Zealand dollar on dairy. Watch the right commodity per currency, not a generic one. - A yield axis cuts across both blocs. Low-yielders (euro, franc, yen) act as safe havens and strengthen in fear; high-yielders (Australian, New Zealand, emerging-market currencies) are risk currencies that sell off in stress. - The commodity and yield personalities overlap: the Australian dollar is both an iron-ore proxy and a high-yielder, so it carries two correlated reasons to fall together with risk assets in a panic. - Correlated currencies break the sizing arithmetic. Book variance includes cross terms scaled by correlation; when intra-bloc correlation is near one, a five-position bloc book carries the risk of one position five times the size while a naive calculation reports a fifth of it. - Size at the bloc level using a currency-strength decomposition, budget risk to the shared factor, and remember the correlations tighten toward one in a crisis, so the calm-market structure understates the concentration when it matters. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Trading Systems and Methods - Perry Kaufman (Amazon)](https://www.amazon.com/s?k=Trading+Systems+and+Methods+Perry+Kaufman&ref=aligrithm.com) - [Cybernetic Trading Strategies - Murray Ruggiero (Amazon)](https://www.amazon.com/s?k=Cybernetic+Trading+Strategies+Murray+Ruggiero&ref=aligrithm.com) - [The Art of Currency Trading - Brent Donnelly (Amazon)](https://www.amazon.com/s?k=The+Art+of+Currency+Trading+Brent+Donnelly&ref=aligrithm.com) ### 2.60 The Ehlers Modified Hilbert Transformer URL: https://aligrithm.com/the-ehlers-modified-hilbert-transformer/ Last updated: 2026-07-27T13:23:22.000Z The old article ["Dominant Cycle Estimation Without Astrology"](https://aligrithm.com/dominant-cycle-estimation-without-astrology/) listed instantaneous frequency as one way to measure the cycle, and quietly leaned on a tool it never built: the thing that turns a single price series into an amplitude and a phase you can read bar by bar. That tool is the Hilbert transformer, and it is the engine under every "instantaneous period" and "instantaneous phase" claim in cycle analysis. The textbook version is also unusable for trading, because the honest one needs an infinite filter and the short one lags too much to act on. The practical answer is a deliberately compromised version that trades a little accuracy for usable lag, and knowing where that compromise sits is the difference between trusting the phase readout and being fooled by it. ## Why one number is not enough A price series gives you one value per bar, the level, and from one value you cannot separate how big the swing is from where in the swing you are. A sine wave at its peak and a flat market can momentarily sit at the same level; the level alone does not tell them apart. What you want is the pair: amplitude (how strong the cycle is right now) and phase (where in the cycle you are right now). Engineers get that pair by building the analytic signal, a two-part object whose first part is the price and whose second part is the same price shifted ninety degrees in phase, the quadrature component. $$ x\_a(t) = I(t) + j\\,Q(t), \\qquad A(t) = \\sqrt{I^2 + Q^2}, \\qquad \\phi(t) = \\arctan\\!\\left(\\frac{Q}{I}\\right) $$ I is the in-phase part, the price itself, and Q is the quadrature part, the price delayed by a quarter cycle. Treat the pair as a point in a plane: its distance from the origin is the instantaneous amplitude A, and the angle it makes is the instantaneous phase phi. As bars advance the point rotates, and the speed of rotation is the instantaneous frequency, so the rate of change of phase gives you the period directly, bar by bar, with far less delay than counting zero crossings. This is the generalization the old article gestured at: a phasor is a point rotating at a fixed speed and amplitude, and the analytic signal is the same idea allowed to change its speed and amplitude every bar, which is exactly what an evanescent market cycle does. ## The transformer, and why the textbook one is useless Everything above needs Q, the price shifted ninety degrees, and producing that shift is the Hilbert transformer's only job. A ninety-degree phase shift that is flat across all frequencies is, in theory, a specific filter, and that filter is a non-recursive (FIR) one of infinite length, with coefficients that decay slowly on both sides of center. Two problems make the textbook version dead on arrival for trading. First, infinite length is not buildable, so you truncate it, and truncation wrecks the phase accuracy at exactly the long periods traders care about. Second, and fatal, the ideal quadrature filter is centered, it uses future bars as well as past ones, and a centered FIR has a lag equal to half its length. Make it long enough to be accurate and its lag swallows any signal you could trade; make it short enough to be timely and its phase shift is wrong. That is the same FIR length-versus-lag bind from the old article ["Why Moving Averages Can Lie at Turning Points"](https://aligrithm.com/why-moving-averages-can-lie-at-turning-points/), and here it has teeth, because a wrong phase does not just arrive late, it points the wrong way. So the raw Hilbert transformer is a correct idea you cannot run live. You need a version that gives an approximate ninety-degree shift over the band of periods you actually trade, with a lag small enough to act on, and accepts being wrong outside that band. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## The modified version: accurate where it counts, wrong where it does not The practical transformer drops the demand for a flat ninety-degree shift at every frequency and asks for it only across the cycle band, the periods a roofing filter has already let through. Inside that band the modified transformer holds the phase shift close to ninety degrees with only a few coefficients and a lag of a bar or two; outside it the shift drifts off, which costs nothing because you band-limited those frequencies away before the transformer ever saw them. This is engineering by priorities, spend your filter budget where the cycle lives and abandon the frequencies you do not trade, and it is the same logic as design-by-band that runs through this whole pillar. The reward is a usable I and Q, hence a usable amplitude and phase, hence an instantaneous period with a one-to-two-bar lag instead of the twelve-bar lag of the periodogram or the structural lag of zero crossings. The catch is the one stamped on every cycle tool here. The modified transformer is accurate only inside the band you tuned it for, so feed it through a roofing filter first or its phase readout is noise. Its low lag comes with high variance: instantaneous frequency from a short transformer jumps around violently bar to bar, so gate it on amplitude, when A is small the phase is meaningless because there is no cycle to have a phase, and the old article ["Why Market Cycles Are Evanescent"](https://aligrithm.com/why-market-cycles-are-evanescent/) is the reason A keeps collapsing. Use it as the low-latency cycle sensor it is, band-limit the input, smooth the raw frequency, ignore the phase whenever amplitude is below the noise floor, and never forget that you are reading an approximation that is correct in one octave and lying everywhere else. The textbook transformer is right and unusable; the modified one is usable and right only where you aimed it, which is the trade you always make in causal filtering. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_253-visual-0.png) ## KEY POINTS - One price value per bar cannot separate how big the swing is from where in the swing you are; you need amplitude and phase together. - The analytic signal supplies both: pair the price I with its quarter-cycle-shifted copy Q, then amplitude is sqrt(I squared plus Q squared) and phase is arctan(Q over I), with the rotation speed giving instantaneous frequency, the low-lag period the old article ["Dominant Cycle Estimation Without Astrology"](https://aligrithm.com/dominant-cycle-estimation-without-astrology/) wanted. - Producing Q is the Hilbert transformer's only job: a ninety-degree phase shift across frequency. - The textbook transformer is an infinite-length, centered FIR, so making it accurate forces a lag of half its length, which the old article ["Why Moving Averages Can Lie at Turning Points"](https://aligrithm.com/why-moving-averages-can-lie-at-turning-points/) warned about; it is correct but unusable live. - The Ehlers modified transformer holds the ninety-degree shift only across the cycle band with a few taps and a one-to-two-bar lag, and accepts being wrong at frequencies you band-limited away anyway. - It demands the usual cycle-tool discipline: roofing-filter the input first, smooth the high-variance frequency, and gate the phase on amplitude, because when amplitude collapses (the old article ["Why Market Cycles Are Evanescent"](https://aligrithm.com/why-market-cycles-are-evanescent/)) the phase is meaningless. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Statistically Sound Indicators for Financial Market Prediction - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Statistically+Sound+Indicators+for+Financial+Market+Prediction+Timothy+Masters&ref=aligrithm.com) - [Cycle Analytics for Traders - John Ehlers (Amazon)](https://www.amazon.com/s?k=Cycle+Analytics+for+Traders+John+Ehlers&ref=aligrithm.com) ### 2.59 Convolution: Detecting Reversals by Folding Price URL: https://aligrithm.com/convolution-detecting-reversals-by-folding-price/ Last updated: 2026-07-27T13:23:22.000Z The whole point of trend trading is to be in the market the right way around after a major turn, and the old article ["Why Moving Averages Can Lie at Turning Points"](https://aligrithm.com/why-moving-averages-can-lie-at-turning-points/) showed why the usual tools find that turn late: a moving average at a reversal reports a blend of the regime that just ended. So skip the average and ask a sharper question directly. A market bottom has a shape: prices fall into it and rise out of it, roughly mirror images around the low. If a stretch of price is symmetric about some bar, that bar is a turn. Folding the price around a candidate point and measuring how well the two halves match is a reversal detector that owes nothing to a moving average, and it lights up at the turn instead of after it. ## What a turn actually looks like Consider an idealized bottom. Prices decrease in a clean line down to the low, then increase in a clean line away from it. Pick the low as a hinge and fold the chart along it, so the bars before the low land on top of the bars after it. The down-leg and the up-leg lie on top of each other almost perfectly, because a reversal is a reflection in time. That mirror symmetry is the signature of a turn, and a top is the same picture upside down. No oscillator, no lookback tuning, just a geometric fact: turns are symmetric, trends are not. A trending stretch folded about any of its bars produces two halves going the same direction, which do not match when one is reversed. Convolution is the operation that measures this folding. It is the close cousin of correlation, with one twist, you reverse one of the two series before you slide and multiply, and that reversal is exactly the fold. Cross-correlation slides two series past each other and asks how similar they are at each offset; convolution does the same after flipping one in time. Flip price about a candidate bar and correlate it with itself and you are asking, quantitatively, how symmetric is the price around this point. ## Folding price and scoring the match For a candidate center bar, take a window of length N on each side and measure the correlation between the forward half and the time-reversed back half. The cleanest score is the Pearson correlation coefficient between the two folded segments. $$ C(n) = \\frac{\\sum\_{k=1}^{N}\\big(x\_{n-k}-\\bar{x}\\big)\\big(x\_{n+k}-\\bar{x}\\big)}{\\sqrt{\\sum\_{k=1}^{N}\\big(x\_{n-k}-\\bar{x}\\big)^2}\\;\\sqrt{\\sum\_{k=1}^{N}\\big(x\_{n+k}-\\bar{x}\\big)^2}} $$ Here n is the candidate center bar and k steps outward to each side. The numerator pairs the bar k steps before the center with the bar k steps after it, for every k out to N, and sums the products of their deviations from the window mean; the denominator normalizes so the score lands between minus one and plus one. A score near plus one at bar n means the price k bars before n mirrors the price k bars after n for the whole window, a clean reflection, a turn. A score near zero means no symmetry, a trend or noise. You can compute this for a range of window lengths N at once and stack the results into a heatmap: candidate bars on one axis, fold half-length on the other, brightness equal to the correlation. A real turn shows up as a bright vertical stripe at the reversal bar that persists across many window lengths, because a genuine reflection is symmetric whether you fold two bars or twenty. The heatmap is the honest way to read it, because a strong symmetry score at only one window length is probably an accident, while a stripe that holds across scales is structure. Treat the brightest, most scale-persistent column as the detected reversal. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## The catch every fold hides The fold has a built-in delay, and pretending otherwise is the trap. To know that bar n is a symmetric center you need the bars after it, all N of them, because symmetry is a statement about both sides. The correlation at the true low cannot be computed until N bars after the low has printed, so a fold half-length of 20 means a 20-bar wait before the bottom can confirm. This is not a flaw in the method, it is the price of asking a question about both sides of a point, and any tool claiming to confirm a reversal in real time with no lag is lying or peeking. Convolution is honest about the lag, which is more than the moving-average crossover from the old article ["Why Moving Averages Can Lie at Turning Points"](https://aligrithm.com/why-moving-averages-can-lie-at-turning-points/) manages, but honest lag is still lag. Short windows confirm faster and false-trigger on noise that happens to look symmetric; long windows are reliable and slow. Pick the half-length for the swing you trade, no shorter. The deeper limit is that symmetry assumes the turn is reflection-like, and real bottoms are messy, sharp V on one side, slow rounding on the other, gaps, noise. The correlation degrades exactly when the turn is asymmetric, which is often. The old article ["Dominant Cycle Estimation Without Astrology"](https://aligrithm.com/dominant-cycle-estimation-without-astrology/) gives the natural setting for the fold length, scale it to the measured cycle so you are folding over a swing that actually exists, and gate the detector on cycle quality the same way every cycle-mode tool in this pillar must be gated. Read the convolution heatmap as evidence of symmetry, weigh it against its unavoidable confirmation lag, and never sell it as a leading reversal call, because the bars it needs to declare the turn arrive after the turn. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_252-visual-0.png) ## KEY POINTS - A market turn is a reflection in time: price falls into a bottom and rises out of it roughly symmetrically, so folding price about a candidate bar and measuring how well the halves match detects the turn. - Convolution is correlation with one series reversed in time, and that reversal is exactly the fold, so it quantifies symmetry about a point. - Score a candidate center bar with the Pearson correlation between its forward half and time-reversed back half; near plus one means a clean reflection (a turn), near zero means a trend or noise. - Compute across many fold half-lengths and read it as a heatmap. A real reversal is a bright vertical stripe that persists across window lengths; a one-window hit is probably an accident. - The method has unavoidable confirmation lag: knowing bar n is a symmetric center needs the N bars after it, so a 20-bar fold confirms 20 bars after the low. It is honest lag, not zero lag. - Symmetry degrades on asymmetric real turns. Scale the fold length to the measured dominant cycle from the old article ["Dominant Cycle Estimation Without Astrology"](https://aligrithm.com/dominant-cycle-estimation-without-astrology/), gate on cycle quality, and never sell convolution as a leading reversal call. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Statistically Sound Indicators for Financial Market Prediction - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Statistically+Sound+Indicators+for+Financial+Market+Prediction+Timothy+Masters&ref=aligrithm.com) - [Cycle Analytics for Traders - John Ehlers (Amazon)](https://www.amazon.com/s?k=Cycle+Analytics+for+Traders+John+Ehlers&ref=aligrithm.com) ### 8.7 Mutual Information as a Regime / Noise Filter URL: https://aligrithm.com/mutual-information-as-a-regime-noise-filter/ Last updated: 2026-07-27T13:26:03.000Z Entropy from the old article ["Entropy as a Market Choppiness Gauge"](https://aligrithm.com/entropy-as-a-market-choppiness-gauge/) tells you whether the recent tape is choppy or structured, but it answers a one-sided question: how surprising is the next move on its own. It never asks whether the next move is connected to what just happened. Mutual information closes that gap. It measures how much knowing the recent pattern of price changes tells you about the direction of the very next one, and that is exactly the quantity a trading system lives or dies on. When recent direction is statistically tied to the prior pattern, your rules have something to grip. When the tie goes to zero, the next move is independent of the last few, the noise the old article ["Noise Is Not Volatility"](https://aligrithm.com/noise-is-not-volatility/) described has taken over, and the honest call is to pull out of the market. ## Two quantities, one relationship Mutual information is a general measure of how related two discrete quantities are, and the build here picks the two that matter for a trader. The first quantity is the direction of the most recent price change, a single bit: did the last bar close up or not. The second quantity is the arrangement of the prior recent moves, the same up-or-down word from the choppiness gauge, one of B possible bins. Mutual information asks how much the recent direction and the prior pattern share. If up almost always follows the pattern up-up and down almost always follows down-down, the two are tightly linked and there is momentum to trade. If the recent direction lands the same way no matter what preceded it, they share nothing. $$ \\text{MI} = \\sum\_{b=1}^{B} \\left\[ p(0,b)\\,\\log\\frac{p(0,b)}{p(0)\\,p(b)} + p(1,b)\\,\\log\\frac{p(1,b)}{p(1)\\,p(b)} \\right\] $$ Read it as a comparison between what happens and what would happen by chance. The index b runs over the B prior-pattern bins, the labels 0 and 1 mark the recent direction (not-up and up), and p(d,b) is the joint probability of seeing direction d together with pattern b. The marginals are the standalone odds: p(d) is how often the recent move is up or not regardless of the pattern, and p(b) is how often pattern b shows up regardless of what came next. The ratio inside each log is the tell. If direction and pattern were independent, the joint p(d,b) would equal the product p(d) times p(b), the ratio would be 1, the log would be 0, and the whole sum would collapse to zero MI. Every bit the joint pulls away from that product is structure, and MI adds it all up. The marginals are not free parameters, they fall out of the joint table. $$ p(0) = \\sum\_{b=1}^{B} p(0,b), \\qquad p(1) = \\sum\_{b=1}^{B} p(1,b), \\qquad p(b) = p(0,b) + p(1,b) $$ Sum the joint across patterns to get how often each direction occurs, and sum across the two directions to get how often each pattern occurs. Everything comes from one count table over the window: tally how often each pattern was followed by an up versus a not-up, divide by the total, and the rest is arithmetic. ## A worked number Take the simplest case, a prior pattern that is just the single previous move, so two bins, down or up, and pair it with the next move. Suppose over the window the joint probabilities come out as up-then-up 0.4, down-then-up 0.1, up-then-down 0.1, down-then-down 0.4\. The market is showing momentum: a move tends to repeat. The marginals are all 0.5, since up-now totals 0.5 and up-prior totals 0.5\. Plug in with natural logs. Each of the two heavy cells contributes 0.4 times the log of 0.4 over 0.25, and 0.4 over 0.25 is 1.6, whose log is about 0.47, so each heavy cell adds about 0.188\. Each light cell contributes 0.1 times the log of 0.1 over 0.25, and 0.1 over 0.25 is 0.4, whose log is about minus 0.92, so each light cell subtracts about 0.092\. The sum is roughly 0.188 plus 0.188 minus 0.092 minus 0.092, about 0.19 nats of mutual information: real, tradeable structure. Now flatten it to pure noise. Make all four cells 0.25, meaning the next move is a coin flip no matter what came before. Every joint equals the product of its marginals, 0.25 equals 0.5 times 0.5, every ratio is 1, every log is 0, and MI is exactly zero. Same volatility, same average move size, but the link between past and future is gone. That zero is the signal to stand down. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Why this beats a volatility reading The old article ["Noise Is Not Volatility"](https://aligrithm.com/noise-is-not-volatility/) made the case that how far price travels and how much of that travel survives as net direction are two different things, driven independently. Mutual information is the second of those measured directly. A market can double its volatility, throw bigger bars, and keep MI exactly where it was, because the bigger moves still follow the same patterns. Another market can keep volatility flat while MI drains to zero, the moves staying the same size but losing all connection to what preceded them. A volatility gauge cannot tell these apart; it reports the size of the bars and stays silent on whether they are predictable. MI reports the predictability and stays silent on the size, which is the half a trader actually needs to know whether the edge is live. It also catches what linear tools miss. The old article ["Entropy as a Market Concept"](https://aligrithm.com/entropy-as-a-market-concept/) pointed out that a series with zero linear autocorrelation can still be predictable through nonlinear dependence, and MI does not care whether the relationship is linear. If the next direction depends on the prior pattern in any way, even one a correlation coefficient reads as zero, MI sees it, because it compares full joint probabilities against the independence baseline rather than fitting a straight line. The price is a harder estimate that needs more data to pin down than a correlation does, which is why the window has to be long enough to fill the bins. ## Using it as a kill switch Treat MI as a go or no-go gate on whatever rule you already trust, not as the rule itself. Compute it on a rolling window, and when it sits comfortably above zero the structure your system was built on is present, so let the system run. When MI decays toward zero the market has gone to noise, the next move has come unhooked from the recent ones, and the right move is to cut size or step out until it recovers, exactly the "pull out of the market" call this indicator exists to make. The noise level of markets rises and falls over time, so this is not a one-time check; it is a continuous read on whether conditions still match the ones your edge was measured in. Two cautions keep it from lying to you. First, MI is always non-negative and any finite sample produces a small positive value even from genuinely independent data, because random counts never land exactly on the independence product, so a tiny MI is indistinguishable from zero and you should set your floor above that sampling-noise level rather than at a literal zero. Second, MI measures that a relationship exists, not what it is or which way it points; a high reading on a momentum window and a high reading on a mean-reversion window look identical, so MI tells you the edge is present and something else has to tell you how to take it. And as with entropy, every reading is a window average, so it lags the regime change that triggered it, and you cannot fix that by shrinking the window without starving the bins and turning the estimate into noise of its own. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_268-visual-0.png) ## KEY POINTS - Mutual information measures how much the prior pattern of price moves tells you about the direction of the next one, the exact quantity a system needs: pair the recent direction (up or not, one bit) with the prior up-or-down word (one of B bins). - The formula sums, over every cell of the joint table, the joint probability times the log of joint over the product of marginals; when direction and pattern are independent the joint equals that product, every log is zero, and MI collapses to zero. - Worked case: a momentum window (up-up and down-down at 0.4, off-diagonals at 0.1) gives about 0.19 nats of structure, while flattening every cell to 0.25 gives exactly zero, same volatility but no link between past and future. - It is the predictability half of the old article ["Noise Is Not Volatility"](https://aligrithm.com/noise-is-not-volatility/) measured directly: volatility can double while MI holds, or stay flat while MI drains to zero, and a volatility gauge cannot tell those apart. - It catches nonlinear dependence a correlation coefficient reads as zero, the point the old article ["Entropy as a Market Concept"](https://aligrithm.com/entropy-as-a-market-concept/) raised, because it compares full joint probabilities to the independence baseline instead of fitting a line. - Use it as a go/no-go gate: above a noise floor, run the system; decaying toward zero, cut size or pull out. Set the floor above the small positive value finite samples always produce, remember MI shows that structure exists but not its direction, and accept that it lags because it averages a window. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Mathematical Techniques in Financial Markets - Don Mak (Amazon)](https://www.amazon.com/s?k=Mathematical+Techniques+in+Financial+Markets+Don+Mak&ref=aligrithm.com) - [Cycle Analytics for Traders - John Ehlers (Amazon)](https://www.amazon.com/s?k=Cycle+Analytics+for+Traders+John+Ehlers&ref=aligrithm.com) ### 2.58 The Even Better Sine Wave: Advancing Phase to Predict URL: https://aligrithm.com/the-even-better-sine-wave-advancing-phase-to-predict/ Last updated: 2026-07-27T13:23:21.000Z Every indicator in this pillar reshapes the past, and the old article ["Why Moving Averages Can Lie at Turning Points"](https://aligrithm.com/why-moving-averages-can-lie-at-turning-points/) showed the bill that comes due for it: signals arrive after the turn, never before. There is one trick that looks like it cheats this, and it almost does. If you can convert the market's swing into a clean sine wave, you know something a price chart never gives you, the exact shape of what comes next, because a sine wave is perfectly predictable once you know where in its cycle you are. So you take the phase you measured and push it forward. The indicator now reports where the cycle will be a few bars from now instead of where it is. That is real prediction, and it is valid for exactly as long as the market keeps behaving like a sine wave, which is the whole catch. ## Turn the swing into a sine wave first Raw price is trend plus cycle plus noise, and none of those three is a sine wave. To get one, band-limit the price into a roofing filter: a high-pass that kills the slow trend and a smoother that kills the fast noise, leaving the cycle band. The old article on isolating the cycle band covered why this matters; here it is the mandatory first step, because the phase-advance trick only works on a clean oscillation. Feed it raw price and you are advancing the phase of trend lag and noise, which is meaningless. Once you have a clean cyclic waveform, normalize it hard so its amplitude stays fixed regardless of how strong the swing is. Compute the average wave amplitude and the average power over a short window, where power is proportional to the square of the amplitude, then divide the amplitude by the square root of the power. $$ \\text{Power} = \\overline{\\text{Wave}^2}, \\qquad \\text{SineWave} = \\frac{\\overline{\\text{Wave}}}{\\sqrt{\\text{Power}}} $$ Dividing amplitude by the square root of average power forces the output to swing between roughly minus one and plus one no matter how big or small the raw cycle is, which is the same amplitude-stabilizing logic as the old article ["Automatic Gain Control for Trading Indicators"](https://aligrithm.com/automatic-gain-control-for-trading-indicators/). The result is a waveform whose height you can trust, so a reading near plus one always means the same thing (near a cyclic top) and near minus one always means a cyclic bottom, across calm and violent markets alike. ## Advance the phase to look forward Now the prediction. A sine wave is fully described by its phase, the angle that says where in the cycle you are right now, and the wave's value is the sine of that phase. $$ \\text{now} = \\sin(\\phi), \\qquad \\text{lead} = \\sin\\!\\left(\\phi + \\Delta\\phi\\right), \\qquad \\Delta\\phi = 2\\pi\\,\\frac{k}{T} $$ To see the wave k bars into the future, add the phase that k bars represents, which is 2 pi times k over the period T, and take the sine of the advanced angle. If the period is 20 bars and you advance by 3 bars, you add 2 pi times 3 over 20, about 54 degrees, and the lead curve crosses its peaks and zeros three bars before the present curve does. Plot both: the present sine wave and the advanced one. The advanced curve calls the top before the present one reaches it, and that lead is genuine prediction rather than lag reduction, because for a pure sine wave the future is determined and you are computing it, not estimating it. This is why it earns the name. The original sine wave indicator hunted for the dominant cycle phase by correlating price against a theoretical sine, an expensive measurement. The even better version leans on the strong normalization instead, so the waveform is clean enough to advance directly. You get a forward-looking line from a backward-looking series, which feels like a free lunch. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## It is not a free lunch, and the bill is regime Read the assumption in the phase-advance equation: it computes the future under the premise that the wave continues as a sine of the same period. That premise is true only in cycle mode. The old article ["Why Market Cycles Are Evanescent"](https://aligrithm.com/why-market-cycles-are-evanescent/) is the entire risk here: the period drifts, the amplitude decays, the phase loses coherence, and the moment the market stops oscillating the advanced curve is confidently predicting a swing that will never happen. In a trend, a phase-advanced sine wave will scream reversal at every minor pause and be wrong each time, because there is no cycle to advance. Prediction of a cyclic swing while the market is trending is not just useless, it is a sequence of high-conviction wrong calls, which is worse than a lagging indicator that at least stays quiet. So the discipline is the same gate as every cycle-mode tool. Advance the phase only when the dominant cycle is real and persistent, a sharp periodogram peak with stable period and amplitude above the noise floor, and the further you advance the more fragile the call, because a small error in the measured period compounds into a large phase error several bars out. Advance by a fraction of the cycle, not half of it. Treat the lead curve as a hypothesis that the current swing continues, gate it on cycle quality, and kill it the instant the market trends. Used inside its regime it is the closest thing to legitimate prediction this pillar offers. Used outside it, it is a confident liar. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_251-visual-0.png) ## KEY POINTS - The phase-advance trick is the closest thing to genuine prediction in this pillar: convert the swing to a sine wave, then push its phase forward to read where the cycle will be a few bars from now. - It requires a clean oscillation first, so band-limit price through a roofing filter (kill trend and noise) before doing anything else. - Normalize hard: divide average wave amplitude by the square root of average power so the output swings a fixed minus-one to plus-one regardless of swing strength, the same amplitude logic as the old article ["Automatic Gain Control for Trading Indicators"](https://aligrithm.com/automatic-gain-control-for-trading-indicators/). - Advance by adding the phase that k bars represents, 2 pi times k over the period, and take the sine of the advanced angle; the lead curve calls peaks and troughs k bars before the present curve. - This is real prediction, not lag reduction, because a pure sine wave's future is determined, but it assumes the wave continues at the same period. - That assumption holds only in cycle mode. The old article ["Why Market Cycles Are Evanescent"](https://aligrithm.com/why-market-cycles-are-evanescent/) is the risk: in a trend the advanced curve fires high-conviction false reversals. Gate on cycle quality, advance by a small fraction of the cycle, and kill it when the market trends. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Statistically Sound Indicators for Financial Market Prediction - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Statistically+Sound+Indicators+for+Financial+Market+Prediction+Timothy+Masters&ref=aligrithm.com) - [Cycle Analytics for Traders - John Ehlers (Amazon)](https://www.amazon.com/s?k=Cycle+Analytics+for+Traders+John+Ehlers&ref=aligrithm.com) ### 8.6 Entropy as a Market Choppiness Gauge URL: https://aligrithm.com/entropy-as-a-market-choppiness-gauge/ Last updated: 2026-07-27T13:26:03.000Z The old article ["Relative Entropy as an Indicator Quality Score"](https://aligrithm.com/relative-entropy-as-an-indicator-quality-score/) used entropy to grade an indicator: bin its values, measure how evenly it fills its range, and reject the ones that pile up in a corner. Same formula, opposite job here. Point the entropy measurement at the price itself instead of at an indicator, and you stop grading a tool and start reading the market. A choppy stretch where price jumps up and down with no follow-through is high entropy, maximum surprise about the next move. A slow steady drift, where each bar mostly continues the last, is low entropy. The old article ["Entropy as a Market Concept"](https://aligrithm.com/entropy-as-a-market-concept/) argued an efficient market is high-entropy with the predictable structure squeezed out. This article builds the gauge that puts a number on how much structure is left in the last few weeks of price. ## Turning price into words Entropy needs discrete outcomes, and a raw price is continuous, so the first move is to chop recent price action into a finite alphabet. Compare each close to the one before it and ask a single yes-or-no question: did price go up? That gives one bit. Stack K of these consecutive comparisons and you get a pattern of K bits, what information theory calls a word of length K. A word of length 3 might read up-up-down, or down-up-up, and there are exactly two-to-the-K such words. $$ B = 2^{K} $$ B is the number of possible words, the size of the alphabet, and K is the word length, the number of consecutive up-or-down comparisons that make one word. Length 2 gives 4 words, length 3 gives 8, length 4 gives 16\. Keep K small. A word length of 2 already carries usable information about market behavior, and pushing K up multiplies the bins faster than your window can fill them, so the estimate degrades into noise. Each bin needs enough visits to estimate its probability, so the window length should be a multiple of B, not a multiple of bars you picked because they felt round. Slide a window over the series, read off the word at each bar, and tally how often each of the B words shows up. Divide each count by the number of words in the window and you have a probability histogram: p(b) is the fraction of the time the recent market spelled out word b. ## From the histogram to one number Entropy collapses that whole histogram into a single value, the average surprise of seeing a word. $$ H = -\\sum\_{b=1}^{B} p(b)\\,\\log p(b) $$ H is the entropy, the sum runs over all B words, p(b) is the probability of word b from the histogram, and the log base is your choice because it only rescales the result. Read the formula through its extremes. If one word dominates and the rest almost never appear, the market keeps repeating the same pattern, p(b) is near 1 for that word and near 0 for the others, and H collapses toward 0: low surprise, high predictability. If every word is equally likely, the market spells out a different pattern every time with no preference, p(b) equals 1/B for all of them, and H hits its ceiling at log(B): maximum surprise, no structure. That ceiling is the trick for making the number portable. Raw H depends on B, so an H of 1.2 means nothing until you know whether the alphabet had 4 words or 64\. Divide by the maximum and the scale problem disappears. $$ \\text{RE} = \\frac{H}{\\log B} $$ RE is the relative entropy, H over its theoretical maximum log(B), and it lands in a clean zero-to-one band regardless of word length. RE near 1 is a choppy, unpredictable market with the patterns spread evenly across the alphabet. RE near 0 is a market locked into a few repeating patterns, the low-entropy pocket the old article ["Entropy as a Market Concept"](https://aligrithm.com/entropy-as-a-market-concept/) called an edge. Work a length-2 case to see the gauge move. Four words, so log(B) is log(4). Suppose the window splits evenly, each word at probability 0.25\. Then H is the sum of four terms of minus 0.25 times log(0.25), which equals log(4), and RE is exactly 1.0: the market filled its alphabet uniformly, peak choppiness. Now skew it so one word carries 0.7 of the visits and the other three split 0.1 each. Using natural logs, H is minus the quantity 0.7 times ln(0.7) plus three times 0.1 times ln(0.1), which works out to about 0.94, and RE is 0.94 over ln(4), about 0.68\. The market leaned on one pattern, entropy dropped, and the gauge reported a more structured tape without you eyeballing a single candle. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Why this is not the old quality score The formula matches the old article ["Relative Entropy as an Indicator Quality Score"](https://aligrithm.com/relative-entropy-as-an-indicator-quality-score/) down to the symbol, and that is exactly where traders trip. There, you bin one indicator's output and you want RE high, because a high score means the indicator uses its full range and feeds a model resolution instead of clumping into one bin. The high reading is the good outcome, the sign of a well-behaved tool. Here you bin patterns of price moves, and a high reading is the market telling you it has no exploitable structure. High RE is the thing you wanted from an indicator and the thing you fear from the market. The inputs are different too, and that matters more than the shared equation. The quality score eats a continuous indicator and bins its values by magnitude. The choppiness gauge eats the sign of price changes and bins their arrangement, the words. One judges whether your measuring stick is any good; the other judges whether the thing you are measuring can be traded at all. Run both and you can be in the worst spot without noticing from either alone: a high-quality indicator, RE above 0.8 on its own distribution, pointed at a high-entropy market, RE near 1 on its words, a sharp instrument reading a surface with nothing on it. ## Trading the gauge, and what it will not tell you Use it as a state switch, not a signal. High RE flags a regime where the next move barely depends on the last, the choppy churn where trend rules get whipsawed and where mean-reversion or staying flat is the honest play. Low RE flags a stretch with repeating structure, where directional rules have something to grip. This is the noise axis from the old article ["Noise Is Not Volatility"](https://aligrithm.com/noise-is-not-volatility/) reached through information theory instead of path length, and the two agree on the part that matters: a quiet, low-volatility market can still be pure noise with RE near 1, and a violent one can carry real structure with RE well under it. The gauge says nothing about direction. A low-entropy market repeating down-down-down scores the same as one repeating up-up-up; entropy counts patterns, not their sign, so pair it with something that reads which way the structure points before you take a side. It also lags by construction, because every reading is an average over a window, so a regime that just flipped will not show up until enough new words refill the histogram, and shrinking the window to react faster starves the bins and corrupts the estimate. Pick the window for stable bin counts first and accept the lag, rather than chasing responsiveness into garbage. And do not over-read small wiggles in RE near the top of its range; the difference between 0.95 and 0.98 is mostly estimation noise, while the drop from 0.9 to 0.6 is the market actually changing character. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_267-visual-0.png) ## KEY POINTS - Chop recent price into a finite alphabet: compare each close to the prior one for an up-or-down bit, stack K of them into a word, giving two-to-the-K possible words; keep K small (2 is enough) so the bins fill. - Build a probability histogram of the words over a sliding window, then collapse it with entropy H = minus the sum of p(b) log p(b), which is near 0 when one pattern repeats and at its maximum log(B) when every pattern is equally likely. - Normalize to relative entropy RE = H over log(B) so the reading sits in zero-to-one regardless of word length; RE near 1 is a choppy unpredictable market, RE near 0 is the structured pocket the old article ["Entropy as a Market Concept"](https://aligrithm.com/entropy-as-a-market-concept/) called an edge. - The formula matches the old article ["Relative Entropy as an Indicator Quality Score"](https://aligrithm.com/relative-entropy-as-an-indicator-quality-score/) but the meaning flips: there high RE means a good indicator, here high RE means a market with no structure to trade, and the inputs differ (indicator magnitudes versus arrangements of price moves). - It is the noise axis of the old article ["Noise Is Not Volatility"](https://aligrithm.com/noise-is-not-volatility/) reached through information theory: a calm low-volatility market can still read RE near 1 (pure noise) and a violent one can carry real structure. - Use it as a regime switch, not a signal: it ignores direction, it lags because every reading averages a window, and shrinking the window to react faster starves the bins and corrupts the estimate. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Mathematical Techniques in Financial Markets - Don Mak (Amazon)](https://www.amazon.com/s?k=Mathematical+Techniques+in+Financial+Markets+Don+Mak&ref=aligrithm.com) - [Cycle Analytics for Traders - John Ehlers (Amazon)](https://www.amazon.com/s?k=Cycle+Analytics+for+Traders+John+Ehlers&ref=aligrithm.com) ### 2.57 Adaptive Indicators: Tuning RSI to the Measured Cycle URL: https://aligrithm.com/adaptive-indicators-tuning-rsi-to-the-measured-cycle/ Last updated: 2026-07-27T13:23:21.000Z The RSI you trade has a fixed lookback. You picked 14 because someone printed 14 in a book in 1978, and you have used 14 on a 5-minute chart and a weekly chart and a tick chart without ever asking whether 14 bars is the right amount of history for the swing in front of you. It usually is not. A 14-bar oscillator on a market whose dominant swing runs 30 bars is half-tuned, too twitchy to catch the cycle and too slow to call its turn. The old article ["Dominant Cycle Estimation Without Astrology"](https://aligrithm.com/dominant-cycle-estimation-without-astrology/) gave us a number for that swing, the dominant cycle period, measured from the data and updated every bar. This article does the obvious thing with it: stop guessing the lookback and let the measured cycle set it. ## A fixed lookback is a bet you keep losing Every oscillator carries an implicit claim about how long the market's relevant memory is. RSI(14) claims 14 bars. A stochastic with a 20-bar window claims 20\. When the real swing is longer than your window the oscillator pins at the extremes and chops; when the real swing is shorter your oscillator smooths it into mush and reports the turn late. The old article ["Why Market Cycles Are Evanescent"](https://aligrithm.com/why-market-cycles-are-evanescent/) is the reason this hurts: the dominant period is not a constant you can hardcode, it drifts through regimes, sitting at 22 bars for months and then collapsing to 8 in a panic. A fixed-lookback indicator is right for the brief window when its period happens to match the market and wrong the rest of the time, and it never tells you which case you are in. The fix is to make the lookback a variable fed by the measurement. Measure the dominant cycle, set the indicator's window to it, recompute every bar. The indicator now breathes with the market: it widens its memory when swings stretch out and tightens it when they compress. The old article ["Automatic Gain Control for Trading Indicators"](https://aligrithm.com/automatic-gain-control-for-trading-indicators/) did this for amplitude, rescaling an indicator so its swings stay readable as volatility changes. This is the same move applied to time instead of size, adapt the lookback, not the gain. ## Building the adaptive RSI Start with RSI written cleanly. Over a lookback of N bars, sum the up closes and the down closes, then take the ratio of up movement to total movement. $$ \\text{CU} = \\sum\_{i=1}^{N}\\max(x\_i - x\_{i-1},\\,0), \\qquad \\text{CD} = \\sum\_{i=1}^{N}\\max(x\_{i-1} - x\_i,\\,0) $$ $$ \\text{RSI} = \\frac{\\text{CU}}{\\text{CU} + \\text{CD}} $$ This is the same RSI you know, written so the output runs 0 to 1 instead of 0 to 100 (multiply by 100 if you miss the old scale). CU is the total up movement over the window, CD the total down movement, and the ratio is the fraction of motion that was upward. Pinned near 1 means almost all motion was up, near 0 means almost all down. Two changes make it adaptive. First, set the lookback to half the measured dominant cycle rather than a constant. $$ N = \\frac{\\text{DominantCycle}}{2} $$ Half a cycle is the natural window for an oscillator, because one full swing from peak to trough takes half a period, so N bars of half-cycle memory is exactly enough to span the move you are trying to time without dragging in the previous swing. When the periodogram reports a 30-bar dominant cycle, RSI runs on a 15-bar window; when the market compresses to a 16-bar cycle, the same RSI tightens to 8 bars on its own, no reoptimization, no curve-fit. Second, do not feed it raw closing prices. Feed it the output of a band-pass filter centered on the dominant cycle, the FILT in the source, a two-pole band-pass that strips the trend below the cycle and the noise above it. The old article ["Why Moving Averages Can Lie at Turning Points"](https://aligrithm.com/why-moving-averages-can-lie-at-turning-points/) showed that raw price near a turn is dominated by trend lag and noise; band-limiting the input first means the RSI sees a clean cycle, so its overbought and oversold readings line up with the actual swing extremes instead of with whatever the trend and the jitter happened to be doing. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Why this can lead instead of lag, and when it cannot Every indicator built from past data lags, this pillar has hammered that since the start, and adapting the lookback does not repeal it. What adaptation buys is something narrower and real: when a cycle persists for several swings, an indicator tuned to that cycle is at its sharpest exactly where the swing turns, because the band-pass input and the matched window put the oscillator's most responsive region on the live cycle. The oscillator crosses its midline and hits its extremes in step with the swing rather than a quarter-cycle late, and during a persistent cycle the next swing looks like the last, so a correctly tuned reading of this swing carries information about the next one. That is the predictive characteristic, and it is conditional, not free. The condition is cycle mode, and the old article ["Why Market Cycles Are Evanescent"](https://aligrithm.com/why-market-cycles-are-evanescent/) is the gate. The adaptive RSI is only trustworthy when the periodogram shows a real, persistent peak: a sharp dominant cycle with stable period and amplitude above the noise floor. When the market trends, the periodogram flattens, the dominant cycle measurement turns to noise, and the lookback it feeds the RSI starts thrashing on garbage, at which point your adaptive oscillator is worse than the fixed one because at least the fixed one was stable. So gate it. Run the adaptive RSI only when cycle quality is high, stand down to a trend-mode tool when the peak collapses, and never let a low-confidence cycle estimate drive the window. Tune the indicator to the cycle when there is a cycle; when there is not, the honest reading is that this tool has nothing to say. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_250-visual-0.png) ## KEY POINTS - A fixed-lookback oscillator like RSI(14) hardcodes a claim about the market's swing length that is right only for the brief window when the dominant cycle matches it, and the old article ["Why Market Cycles Are Evanescent"](https://aligrithm.com/why-market-cycles-are-evanescent/) guarantees that window keeps moving. - The adaptive RSI sets its lookback to half the dominant cycle measured by the autocorrelation periodogram from the old article ["Dominant Cycle Estimation Without Astrology"](https://aligrithm.com/dominant-cycle-estimation-without-astrology/), recomputed every bar, so the window widens and tightens with the market. - Half a cycle is the natural oscillator window because one peak-to-trough swing spans half a period, so N = DominantCycle/2 holds exactly one swing of memory. - Feed the RSI a band-pass (FILT) output centered on the cycle, not raw price, so its extremes align with the real swing instead of with trend lag and noise, the failure detailed in the old article ["Why Moving Averages Can Lie at Turning Points"](https://aligrithm.com/why-moving-averages-can-lie-at-turning-points/). - The predictive edge is conditional: when a cycle persists, a correctly tuned reading of this swing informs the next, but only in cycle mode. - Gate it on cycle quality. When the periodogram flattens (trend mode) the dominant-cycle estimate is noise and the adaptive window thrashes, so stand the tool down rather than trust it. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Statistically Sound Indicators for Financial Market Prediction - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Statistically+Sound+Indicators+for+Financial+Market+Prediction+Timothy+Masters&ref=aligrithm.com) - [Cycle Analytics for Traders - John Ehlers (Amazon)](https://www.amazon.com/s?k=Cycle+Analytics+for+Traders+John+Ehlers&ref=aligrithm.com) ### 4.54 Order Statistics Across Markets: Median and Range as Breadth URL: https://aligrithm.com/order-statistics-across-markets-median-and-range-as-breadth/ Last updated: 2026-07-27T13:24:41.000Z The old article ["Cross-Sectional Percentile Rank Within a Universe"](https://aligrithm.com/cross-sectional-percentile-rank-within-a-universe/) asked where one market sits in the pack. This asks a different question about the same panel: what is the pack itself doing? Take an indicator computed identically across every market in a universe, and on each bar you have a cloud of values, one per market. Summarize that cloud with order statistics, the numbers you get by sorting the values and reading off positions, and you have a breadth gauge: a measure of the general market state and how tightly or loosely the markets agree. The median of the cloud is the state. The spread of the cloud is the dispersion. Both come straight out of sorting, no model required. The choice of order statistics over the obvious alternatives is the same argument the old article ["Why the Median Often Beats the Mean in Trading Features"](https://aligrithm.com/why-the-median-often-beats-the-mean-in-trading-features/) made. A mean across markets and a standard deviation across markets are both wrecked by one berserk member, and in a panel of markets you will have a berserk member, a single name gapping on a buyout or a halt. The median and the interquartile range do not care. One market doing something insane shifts the median by at most one slot and barely touches the IQR, so the breadth reading reflects the group, not the loudest outlier in it. ## Median of the cloud is the market state Compute a stationary trend indicator for every market, the same recipe everywhere. The natural primitive is the CMMA, the close minus a moving average normalized by volatility, from the old article ["CMMA: A Better Momentum Primitive Than Price-minus-MA Alone,"](https://aligrithm.com/cmma-a-better-momentum-primitive-than-price-minus-ma-alone/) because it is already comparable across instruments. On each bar you have M of these values. Take their median. $$ \\text{State}\_t = \\operatorname{median}\\big(\\,\\text{CMMA}\_t(1),\\, \\text{CMMA}\_t(2),\\, \\ldots,\\, \\text{CMMA}\_t(M)\\,\\big) $$ The median CMMA is the trend of the typical market. Positive means more than half the universe is above its own moving average, a broad uptrend. Negative means the typical market is below, a broad downtrend. Near zero means the universe is split. This is breadth in the honest sense, not a cap-weighted index that two giant names can drag around, but the experience of the median market, which is closer to what a diversified book actually feels. A worked read: across 30 markets the median CMMA sits at +0.8 in a healthy trend, then over two weeks slides to -0.3 while the headline index is still green, because the index is being held up by a handful of leaders while the median market has already rolled over. That divergence, breadth deteriorating under a rising index, is the classic warning the median catches and the index hides. ## Median over IQR is a non-parametric breadth z-score The state alone is half the picture. A median CMMA of +0.8 means something different when every market is clustered near +0.8 than when the markets are scattered from -3 to +5 and +0.8 is just the midpoint of chaos. Measure the scatter with the interquartile range, the distance from the 25th to the 75th percentile of the cloud, the robust cousin of standard deviation. Then divide the state by it. $$ \\text{Breadth}\_t = \\frac{\\operatorname{median}\_t}{\\text{IQR}\_t}, \\qquad \\text{IQR}\_t = Q\_{75,t} - Q\_{25,t} $$ The IQR is the width of the middle half of the universe, the typical spread among markets, and dividing the median by it builds a non-parametric z-score: how far the typical market has departed from neutral, measured in units of how spread out the markets normally are. A large positive Breadth means the markets are not only mostly up but tightly agreed about it, a coherent broad trend. A small Breadth, even with a positive median, means the agreement is weak and the trend is a wide disagreement that happens to lean up. The construction exaggerates the readings that matter, a unified market, and damps the readings that do not, a market pulling in every direction at once, exactly the behavior you want from a regime gauge. Because it is built from order statistics it carries the outlier immunity of the median straight through to the dispersion term, where a standard z-score would have its denominator blown out by the same outlier that spares the median. A number to anchor it. Median CMMA +0.8, IQR 1.0, Breadth 0.8, a solid coherent uptrend. Same median +0.8 but the universe has fractured and IQR widens to 4.0, Breadth drops to 0.2, and the gauge correctly reports that the broad up-lean is not to be trusted because the markets are all over the place. Same state, opposite confidence, decided by dispersion. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_272-visual-0.png) ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Building the breadth panel Lay the per-market indicator out as a DataFrame, one column per market, then collapse across columns on each bar with the robust order statistics. Every operation is a row-wise quantile, so it uses only the markets' values on that bar, and the only history involved is inside the per-market CMMA window. No look-ahead, no cap weighting, no single market dominating. ```python import numpy as np import pandas as pd def cmma(prices: pd.DataFrame, ma_window: int = 20, atr_window: int = 20) -> pd.DataFrame: logp = np.log(prices) ma = logp.rolling(ma_window).mean() vol = logp.diff().rolling(atr_window).std() return (logp - ma) / (vol * np.sqrt(ma_window)) def market_breadth(prices: pd.DataFrame, ma_window: int = 20) -> pd.DataFrame: c = cmma(prices, ma_window) state = c.median(axis=1) iqr = c.quantile(0.75, axis=1) - c.quantile(0.25, axis=1) breadth = state / iqr.replace(0, np.nan) return pd.DataFrame({"state": state, "iqr": iqr, "breadth": breadth}) ``` The CMMA function normalizes each market's distance from its own moving average by its own volatility, scaled by the square root of the window so the value does not grow with lookback, which is what makes the per-market values comparable enough to pool. The breadth function then reports three lines: the median state, the IQR dispersion, and the non-parametric z-score that combines them. Use the state as a market-regime variable, gating or sizing the rest of your book, and use the breadth z-score as the confidence weight on that state. ## What it is and what it is not This is a regime and confidence gauge, not a timing trigger, and the line between those is where people misuse it. The median state tells you the broad posture of the universe, which is exactly the absolute regime filter the old article ["Cross-Sectional Percentile Rank Within a Universe"](https://aligrithm.com/cross-sectional-percentile-rank-within-a-universe/) said you needed to pair with a relative ranking, because a 90th-percentile market in a collapsing universe is still inside a collapsing universe. Run the two together: percentile rank picks which markets, breadth state tells you whether the universe as a whole is worth being long at all. The honest limits are the usual ones. The universe definition decides everything, and a survivorship-biased or mismatched basket produces a breadth reading about a fiction. The breadth gauge lags by exactly the CMMA window underneath it, so it confirms regimes rather than calling tops, and treating a breadth rollover as a precise sell signal will have you late every time. Order statistics are robust but coarse, the median ignores how far above zero the leaders are and the IQR ignores the tails entirely, so a market with healthy breadth can still carry concentrated tail risk in a few names the gauge cannot see. And like everything cross-sectional it is silent on direction conviction for any single instrument; it describes the forest, and you still need the per-market work to choose the trees. Use it to set the weather, not to place the trade. ## KEY POINTS - Compute one stationary indicator (CMMA is the natural primitive) identically across a universe, and on each bar summarize the cloud of values with order statistics: the median is the market state, the spread is the dispersion. No model, just sorting. - Use median and IQR, not mean and standard deviation, because one berserk market wrecks the mean and inflates the standard deviation while barely moving the robust pair, the outlier argument from the old article ["Why the Median Often Beats the Mean in Trading Features."](https://aligrithm.com/why-the-median-often-beats-the-mean-in-trading-features/) - Median CMMA is honest breadth: the trend of the typical market, not a cap-weighted index two giants can drag. Watch for the median rolling over while the headline index still rises, the classic breadth-deterioration warning. - Divide median by IQR for a non-parametric breadth z-score: the typical market's departure from neutral measured in units of how spread out markets normally are. It exaggerates coherent moves and damps fractured ones. Same state, opposite confidence, decided by dispersion. - Build it as row-wise quantiles across a panel, no look-ahead and no cap weighting. Report state, IQR, and breadth as three lines. - It is a regime and confidence gauge, not a trigger. Pair the state with the cross-sectional percentile rank from the old article ["Cross-Sectional Percentile Rank Within a Universe"](https://aligrithm.com/cross-sectional-percentile-rank-within-a-universe/) so you know whether the universe is worth being long at all. It lags by the CMMA window, depends on an honest universe, and is coarse to the tails. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Trading Systems and Methods - Perry Kaufman (Amazon)](https://www.amazon.com/s?k=Trading+Systems+and+Methods+Perry+Kaufman&ref=aligrithm.com) - [Cybernetic Trading Strategies - Murray Ruggiero (Amazon)](https://www.amazon.com/s?k=Cybernetic+Trading+Strategies+Murray+Ruggiero&ref=aligrithm.com) - [The Art of Currency Trading - Brent Donnelly (Amazon)](https://www.amazon.com/s?k=The+Art+of+Currency+Trading+Brent+Donnelly&ref=aligrithm.com) ### 2.56 Reading Reversals from Autocorrelation URL: https://aligrithm.com/reading-reversals-from-autocorrelation/ Last updated: 2026-07-27T13:23:20.000Z The old article ["The Autocorrelation Periodogram"](https://aligrithm.com/the-autocorrelation-periodogram/) used autocorrelation to answer one question: what is the dominant cycle period right now. The same autocorrelation data answers a second question for free, and it is the one traders actually care about: did the market just turn. The old article ["Why Market Cycles Are Evanescent"](https://aligrithm.com/why-market-cycles-are-evanescent/) warned that the dominant period drifts and a cycle you measured may already be gone, which makes a turn detector that does not depend on knowing the period valuable. Autocorrelation gives you exactly that. ## A reversal moves the correlation at every lag at once Picture the autocorrelation function as a row of readings, one per lag: the correlation of price with itself one bar back, two bars back, on out to some maximum lag. Plot it as a colored strip, green where correlation is high, red where it is low, and stack a new strip every bar. While the market grinds along in a stable cycle, the strip barely changes from bar to bar; the correlation at each lag holds roughly steady because the relationship between now and the recent past is stable. At a genuine cyclic reversal, the relationship between now and the past inverts, and the autocorrelation shifts at every lag simultaneously, the whole strip flips from green to red across all values of lag at once. That synchronized flip across the entire lag spectrum is the fingerprint of a turn, and it is far harder to fake than any single-lag wiggle. This is why the method is robust. A change at one lag could be noise. A coordinated change at every lag at the same bar is structural, the signature of the underlying relationship reorganizing rather than one correlation reading jittering. ## Sum the bar-to-bar changes across all lags Turn that observation into a number. For each lag, take the absolute change in its autocorrelation from the previous bar to this one. Then sum those changes across all lags into a single reversal indicator. $$ R(t) \\;=\\; \\sum\_{\\text{lag}} \\left| \\,\\text{AutoCorr}(\\text{lag},\\, t) \\;-\\; \\text{AutoCorr}(\\text{lag},\\, t-1)\\, \\right| $$ Read it piece by piece. The inner term is how much the correlation at one lag moved from yesterday to today; the absolute value keeps a rise and a fall from cancelling; the sum adds up that movement over every lag. When the market is holding its structure, each lag barely moves, the terms are small, and R(t) stays low. When the market reverses, every lag jumps together, the terms pile up in the same direction of magnitude, and R(t) spikes. So a large value of R(t) flags that a turning point has been identified, without R(t) ever needing to know what the cycle period is. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## What it does and does not tell you The strength of this indicator is precisely that it is period-agnostic. The old dominant-cycle and evanescent-cycles articles both ran into the same wall: to trade a cycle you usually need its period, and the period keeps drifting. The autocorrelation reversal indicator steps around that wall because the synchronized flip happens at a turn regardless of which period the market was running, so it keeps working through the regime changes that break a period-locked strategy. It is a clean companion to the periodogram: the periodogram tells you whether a tradeable cycle exists and how long it is, while the reversal indicator tells you when that cycle is rolling over. The limits are the usual ones, stated honestly. R(t) detects that a turn has occurred, it does not tell you the direction, so it is a timing alarm that still needs a directional read from price or trend context to become a trade. It will fire on any large reorganization of structure, which includes the transition from cycle mode into trend mode, so a spike is not always a clean reversal to fade. And like everything built from a trailing window, it confirms a turn slightly after the fact rather than predicting it; the synchronized flip is visible once the new relationship has started forming, not before. Use it to flag that the recent structure just broke, then let the periodogram and the price context decide whether there is anything worth trading on the other side. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_248-visual-0.png) ## KEY POINTS - The autocorrelation function is a row of correlations, one per lag; stacked bar by bar it forms a heatmap that stays stable while the market holds its structure. - At a genuine cyclic reversal the relationship between now and the past inverts, so the autocorrelation shifts at every lag simultaneously, the whole strip flips from green to red at once, a fingerprint far harder to fake than a single-lag move. - Quantify it by summing the absolute bar-to-bar change in autocorrelation across all lags into one indicator R(t); it stays low during stable structure and spikes at a turn. - The method is period-agnostic, which is its main advantage: it keeps working through the period drift that breaks the cycle-locked strategies the old dominant-cycle and evanescent-cycles articles described. - It pairs with the autocorrelation periodogram: the periodogram says whether a cycle exists and how long, the reversal indicator says when it is rolling over. - It flags that a turn happened, not its direction, fires on any large structural reorganization including cycle-to-trend transitions, and confirms slightly after the fact rather than predicting; combine it with directional context before trading. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Statistically Sound Indicators for Financial Market Prediction - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Statistically+Sound+Indicators+for+Financial+Market+Prediction+Timothy+Masters&ref=aligrithm.com) - [Cycle Analytics for Traders - John Ehlers (Amazon)](https://www.amazon.com/s?k=Cycle+Analytics+for+Traders+John+Ehlers&ref=aligrithm.com) ### 4.53 Cross-Sectional Percentile Rank Within a Universe URL: https://aligrithm.com/cross-sectional-percentile-rank-within-a-universe/ Last updated: 2026-07-27T13:24:41.000Z The old article ["Ranking Beats Forecasting for Many Trading Problems"](https://aligrithm.com/ranking-beats-forecasting-for-many-trading-problems/) made the case from the portfolio side: for selecting among instruments you do not need to predict returns, you need to order instruments, because the order survives the miscalibration that wrecks point forecasts. This builds the same idea from the indicator side. Instead of asking what a single market is doing on its own absolute scale, ask where it sits among its peers right now. The output is a percentile, a number from 0 to 100 that says how this market's trend ranks against the universe it belongs to, and that number is comparable across instruments in a way a raw indicator never is. The motivation is the scale problem the old article ["Why Z-Scoring Makes Ranking Cleaner"](https://aligrithm.com/why-z-scoring-makes-ranking-cleaner/) diagnosed. A raw trend reading of 0.04 means one thing on a quiet bond future and something else entirely on a volatile small-cap, so sorting raw values lets the loudest instruments dominate for no reason but their volatility. Percentile rank dissolves the problem by construction. Every market's score becomes its position in the line, 0 for the weakest trend in the universe, 100 for the strongest, and a market at the 90th percentile is at the 90th percentile whether it is gold or a penny stock. ## Pick a per-market property, then rank it across the universe The recipe has two stages. First compute one number per market that captures the property you care about, on each bar. The natural choice is a normalized linear trend over a short window, the slope of a line fit through recent prices, scaled by volatility so it is comparable in spirit before ranking even starts. Any stationary per-market indicator works here, a CMMA, a momentum, an efficiency ratio, the percentile machinery does not care what you feed it as long as it is the same recipe for every market. Second, on each bar, line up that property across all markets in the universe and find where the market of interest falls. $$ \\text{Percentile}\_t(m) = 100 \\times \\frac{\\#\\{\\,j : v\_t(j) < v\_t(m)\\,\\}}{M - 1} $$ Read it directly. The term v\_t(m) is the property, say the normalized trend, of the market m you care about on bar t, and v\_t(j) runs over all M markets in the universe on that same bar. Count how many peers have a strictly smaller value than your market, divide by the M minus one other markets, and scale to 100\. A market stronger than 90 percent of its peers lands near 90\. A market in the middle of the pack lands near 50\. The denominator uses M minus one so the strongest market hits 100 and the weakest hits 0, the full range. Ties get handled by averaging their ranks, the standard convention, so two markets with identical trend share the percentile slot between them. A worked case. Ten sector ETFs, and you compute each one's 20-day normalized trend on today's bar. Energy reads strongest, financials weakest, and the tech ETF you are tracking beats six of the other nine. Its percentile is 100 times six over nine, about 67, so tech is trending in the upper third of the sector board today. Tomorrow energy fades and tech now beats eight of nine, percentile jumps to 89\. The absolute trend value moved too, but the percentile tells you the thing you actually want for cross-sectional work: tech climbed the leaderboard. ## Why the cross-section is doing the normalizing Notice what the ranking step replaced. A z-score, from the old article ["Why Z-Scoring Makes Ranking Cleaner,"](https://aligrithm.com/why-z-scoring-makes-ranking-cleaner/) normalizes a market against its own history, subtracting a trailing mean and dividing by a trailing standard deviation, so a +2 means two of this market's own standard deviations. Percentile rank normalizes against the contemporaneous cross-section instead, so a 90 means stronger than 90 percent of the peer group today. The two answer different questions. The z-score asks whether this market is unusual for itself; the percentile asks whether it is unusual for the group right now. For rotation strategies, longing the strongest names and shorting the weakest, the cross-sectional question is the one that pays, because you are choosing between instruments on the same bar, not deciding whether one instrument is having a strange day. Percentile rank is also the most aggressive normalizer available, and that cuts both ways. It throws away magnitude entirely, keeping only order, so it is maximally robust to outliers and scale, a berserk reading just becomes rank M and stops there, the same outlier immunity the rank approach bought in the old article ["Why the Median Often Beats the Mean in Trading Features."](https://aligrithm.com/why-the-median-often-beats-the-mean-in-trading-features/) The cost is that it cannot tell a runaway leader from a market barely ahead of the pack. If the top three sectors are in a dead heat and the fourth is far behind, the percentiles still read 100, 89, 78, 67, evenly spaced, hiding the cluster at the top. Pure rank discards exactly the spacing that a z-score keeps. The old article ["Why Z-Scoring Makes Ranking Cleaner"](https://aligrithm.com/why-z-scoring-makes-ranking-cleaner/) argued you fall back to pure rank only when magnitudes are too unreliable to trust; percentile rank is that fallback, chosen deliberately when you want robustness over resolution. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_271-visual-0.png) ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Building it across a panel With a panel of markets in a DataFrame, one column per market, compute the per-market property column by column, then rank across columns on each bar. The cross-sectional rank is a single pandas call, and because each bar ranks only the values known on that bar, there is no look-ahead from the ranking step itself; the only history used is inside the per-market trend window. ```python import numpy as np import pandas as pd def normalized_trend(prices: pd.DataFrame, window: int = 20) -> pd.DataFrame: # slope of a line fit through log prices, scaled by return volatility logp = np.log(prices) x = np.arange(window) xc = x - x.mean() denom = (xc ** 2).sum() slope = logp.rolling(window).apply(lambda y: np.dot(xc, y - y.mean()) / denom, raw=True) vol = np.log(prices).diff().rolling(window).std() return slope / vol def cross_sectional_percentile(prices: pd.DataFrame, window: int = 20) -> pd.DataFrame: trend = normalized_trend(prices, window) # rank each row (bar) across markets, scale to 0..100 return trend.rank(axis=1, pct=True) * 100.0 ``` The first function turns each market's price history into a volatility-scaled trend, the per-market property. The second ranks that property across the universe on every bar and scales to a percentile. The output is a panel of the same shape, every cell a 0-to-100 standing of that market against its peers on that bar, ready to drive a rotation rule: long the cells above some high percentile, short the cells below some low one, the indicator-side feed for the ranked long/short machinery the portfolio articles built. ## The honest caveats Percentile rank inherits every limit of ranking and adds one of its own. It says nothing about how hard to bet. A market at the 95th percentile might be a screaming standout or a hair ahead of the 94th, and the percentile cannot tell you, so the old article ["Ranking Beats Forecasting for Many Trading Problems"](https://aligrithm.com/ranking-beats-forecasting-for-many-trading-problems/) was right that ranking solves selection and is silent on sizing; you still need a separate volatility-based sizing layer downstream. The universe definition is a real choice that the math hides: a market is only as meaningful as the peer group you rank it against, and a badly chosen universe, mixing instruments that do not belong together or quietly dropping survivors, produces confident percentiles about a meaningless comparison. The per-market property has its own lookback, an arbitrary dial trading reactivity against noise, and the percentile is only as good as that underlying number, so garbage trend in means garbage rank out. And the whole thing is relative, which means in a universe where everything is falling, the 90th-percentile market is the best of a bad lot, not a long, so the cross-section tells you who is winning the race without telling you whether the race is worth running. Pair it with an absolute trend or regime filter if that distinction matters to your strategy. ## KEY POINTS - Instead of reading a single market's indicator on its own absolute scale, rank that indicator across the universe and report a 0-to-100 percentile. The percentile is comparable across instruments in a way a raw reading never is. - Two stages: compute one stationary per-market property on each bar (a volatility-scaled linear trend is the natural choice), then on each bar count how many peers it beats and scale to 100. - Percentile rank normalizes against the contemporaneous cross-section, answering "unusual for the group today?", while a z-score normalizes against a market's own history, answering "unusual for itself?". For rotation, the cross-sectional question is the one that pays. - It is the most aggressive normalizer: maximally robust to scale and outliers because it keeps only order, at the cost of discarding spacing, so it cannot tell a runaway leader from one barely ahead. It is the pure-rank fallback from the old article ["Why Z-Scoring Makes Ranking Cleaner,"](https://aligrithm.com/why-z-scoring-makes-ranking-cleaner/) chosen when you want robustness over resolution. - Compute it on a panel by ranking each bar across markets (pandas rank with pct), with no look-ahead from the ranking step. Feed the result to a ranked long/short rule. - It is silent on sizing (pair with volatility-based sizing), depends entirely on an honestly built universe, inherits its lookback's reactivity-versus-noise tradeoff, and is purely relative, so in a falling universe the top percentile is the best of a bad lot, not a long. Gate it with an absolute regime filter if that matters. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Trading Systems and Methods - Perry Kaufman (Amazon)](https://www.amazon.com/s?k=Trading+Systems+and+Methods+Perry+Kaufman&ref=aligrithm.com) - [Cybernetic Trading Strategies - Murray Ruggiero (Amazon)](https://www.amazon.com/s?k=Cybernetic+Trading+Strategies+Murray+Ruggiero&ref=aligrithm.com) - [The Art of Currency Trading - Brent Donnelly (Amazon)](https://www.amazon.com/s?k=The+Art+of+Currency+Trading+Brent+Donnelly&ref=aligrithm.com) ### 2.55 The Autocorrelation Periodogram URL: https://aligrithm.com/the-autocorrelation-periodogram/ Last updated: 2026-07-27T13:23:19.000Z The old article ["Dominant Cycle Estimation Without Astrology"](https://aligrithm.com/dominant-cycle-estimation-without-astrology/) laid out four ways to measure the dominant cycle and named one the production default: the autocorrelation periodogram, lowest variance, fine resolution, robust. This article is the why. The Fourier transform is the textbook tool for measuring frequency, but applied to price it carries three assumptions price refuses to satisfy, and the autocorrelation periodogram is the workaround that quietly fixes all three at once. ## What the Fourier transform demands and price denies The discrete Fourier transform is nearly synonymous with frequency measurement, and it rests on three requirements. The data window must be representative of an infinitely long series, the data must be stationary, and only a whole number of cycles can sit in the window or the edges smear the estimate. Price violates each one. A 60-bar window is not a sample of some eternal stationary process; it is a slice of a market whose statistics drift bar to bar, and the dominant cycle inside it is rarely an exact integer divisor of the window length. The old article ["Why Market Cycles Are Evanescent"](https://aligrithm.com/why-market-cycles-are-evanescent/) made the structural version of this point: cycles come and go and change period, so a tool that assumes a stable, infinitely repeating cycle is mismatched to the substrate. The standard patch is a moving DFT. Instead of one transform over all history, compute a block DFT over a sliding window and repeat it every bar, and dodge the integer-cycle problem by computing in terms of cycle period rather than frequency intervals. A useful trick on its own, the moving DFT, but it still inherits the spectral dilation from the previous article and still needs historical averaging to stabilize, both of which add latency. ## Autocorrelation builds the spectrum without the raw amplitude The autocorrelation periodogram takes a different route to the same destination. Start from autocorrelation: correlate the price series with a lagged copy of itself. At lag zero the correlation is perfect; lag it by one bar and the correlation drops; keep lagging and the correlation rises and falls in step with whatever cycle the data carries. $$ \\text{AutoCorr}(\\text{Lag}) \\;=\\; C \\cdot \\text{Lag}^{-\\alpha} \\quad \\text{(efficient-market, long-memory case)} $$ For a pure long-memory random series the autocorrelation decays as a clean power law of the lag, the same alpha exponent from the old noise-color article. Real market data refuse to follow that power law, and the deviation is the point: the partial correlation structure that does not fit the efficient-market decay is exactly the cyclic information you want to extract. The autocorrelation periodogram computes the autocorrelation across a span of lags, then for each candidate cycle period measures how strongly the autocorrelation profile resonates at that period, and the period with the strongest resonance is the dominant cycle. ## Why it beats the DFT and the comb filter The advantage is that working from correlation rather than raw amplitude dissolves the problems that plagued the other methods. $$ \\text{less latency} \\;+\\; \\text{wider amplitude swing} \\;+\\; \\text{no historical averaging} \\;+\\; \\text{no dilation compensation} $$ Less latency, because it does not need a long averaging window to stabilize. A wider range of amplitude swings, so the dominant cycle stands out sharply instead of blurring into its neighbors. No historical averaging, so the estimate reacts when the cycle shifts. And no spectral dilation compensation, because correlation is already normalized, so the amplitude tilt from the previous article never appears and there is nothing to undo. Set against the comb-filter approach, a bank of band-pass filters each tuned to a different period with the output amplitudes read as the spectrum, the autocorrelation periodogram wins on every count: the comb filter has more latency, narrower amplitude swings, and needs the dilation correction the autocorrelation method renders unnecessary. That combination is why the old dominant-cycle article ranked it the production default at roughly 12-bar lag with the lowest variance of the four methods. Keep the honest boundary in view. The autocorrelation periodogram measures the dominant cycle of the recent window cleanly, but it is still a measurement, not a prediction; the evanescent-cycles warning stands, and the number it reports can drift or vanish out of sample. It also still requires a window long enough to contain a few repetitions of the cycle you are chasing, so it is fast relative to the DFT, not instantaneous. Use it as the front end that feeds adaptive indicators their lookback, gate cycle-mode strategies on the sharpness of its peak, and treat a flat, peakless periodogram as the market telling you there is no cycle to trade right now. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_247-visual-0.png) ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## KEY POINTS - The DFT is the standard frequency tool but demands a representative infinite series, stationarity, and a whole number of cycles in the window; price violates all three, echoing the old article ["Why Market Cycles Are Evanescent."](https://aligrithm.com/why-market-cycles-are-evanescent/) - The usual patch is a moving DFT computed every bar in terms of cycle period, but it inherits spectral dilation and needs historical averaging, both of which add latency. - The autocorrelation periodogram works from autocorrelation: correlate price with lagged copies of itself, find which cycle period the correlation profile resonates at, and call that the dominant cycle. - A pure long-memory series gives autocorrelation that decays as a power law of lag; real markets deviate from it, and that deviation is the cyclic information worth extracting. - Because it works from normalized correlation rather than raw amplitude, it has less latency, wider amplitude swings, no need for historical averaging, and no spectral-dilation compensation, beating both the DFT and the comb filter. - It is still a measurement, not a forecast, and needs a window holding a few cycle repetitions; use it to feed adaptive lookbacks and gate cycle-mode strategies, treating a flat periodogram as no tradeable cycle. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Statistically Sound Indicators for Financial Market Prediction - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Statistically+Sound+Indicators+for+Financial+Market+Prediction+Timothy+Masters&ref=aligrithm.com) - [Cycle Analytics for Traders - John Ehlers (Amazon)](https://www.amazon.com/s?k=Cycle+Analytics+for+Traders+John+Ehlers&ref=aligrithm.com) ### 4.52 Deviation from Index Prediction, Weighted by Fit Quality URL: https://aligrithm.com/deviation-from-index-prediction-weighted-by-fit-quality/ Last updated: 2026-07-27T13:24:40.000Z The old article ["Spearman Coupling: When a Stock Decouples from Its Index"](https://aligrithm.com/spearman-coupling-when-a-stock-decouples-from-its-index/) answered one question: is the index still a valid reference for this stock? This answers the next one. Given that the index is a reference, what does it predict the stock should be doing right now, and how far is the stock from that prediction? The gap between the index-implied price and the actual price is the signal. A member tightly bound to the market moves when the market moves, so the index becomes a one-variable forecast of the member, and the residual, the part the index cannot explain, is where anything stock-specific shows up. The old article ["Intermarket Divergence as a Trading Filter"](https://aligrithm.com/intermarket-divergence-as-a-trading-filter/) worked the qualitative version of this: a market moving against what its partner predicts is the tradeable event. This builds the quantitative version, a regression that produces a number instead of an eyeball judgment. The trap is treating every residual as equally meaningful. A two-dollar gap from a prediction built on a tight fit means something. The same two-dollar gap from a prediction built on a loose, scattered fit means nothing, because the model was never accurate enough to call two dollars a deviation. The fix is to weight the deviation by the quality of the fit that produced it, so a residual only counts when the regression had earned the right to be believed. ## Regress the member on the index, in logs, including now Use ordinary linear regression to predict the log price of the member from the log price of the index over a lookback window. Logs, not raw prices, for the reason this pillar keeps repeating: a ratio relationship between two prices becomes a linear one in logs, and proportional moves get treated equally across price levels. Do not reach for anything fancier than a straight line. Both series are soaked in noise, and a flexible model just fits more of that noise, so the simplest fit is the honest one. Center both log-price windows, compute the slope from the cross term over the index's own variation, and you have the line. $$ \\text{slope} = \\frac{\\sum\_{i} (x\_i - \\bar{x})(y\_i - \\bar{y})}{\\sum\_{i} (x\_i - \\bar{x})^2}, \\qquad \\hat{y}\_t = \\bar{y} + \\text{slope}\\,(x\_t - \\bar{x}) $$ Here x is the log index price and y is the log member price over the window, the bars are the windowed means, and the slope is the familiar least-squares estimate, the cross-variation of the two divided by the index's variation with itself. The predicted log member price y-hat at the current bar t is the member's average nudged by the slope times how far the index sits from its own average. The deviation is what is left over. $$ \\text{Dev}\_t = y\_t - \\hat{y}\_t $$ One detail decides whether this is honest: include the current bar in the regression. It sounds like cheating, fitting the line on the same bar you then measure against, but it is the opposite of cheating here. Leaving the current bar out lets a single fresh outlier blow the deviation up to nonsense, and it allows the absurd state of a fit that was perfect over the lookback yet shows a deviation on the new bar, which cannot happen if the new bar helped build the line. Including it anchors the deviation to a line the current bar actually belongs to, and shrinks the influence of any one print. ## Devalue the deviation by the RMS error The slope alone is not enough. A line can have a sensible slope and still fit terribly, with the points scattered far from it, and a deviation measured against a bad line is just noise dressed as signal. Quantify the fit with the root-mean-square error: predict the log member price at every bar in the lookback, accumulate the squared misses, divide by the number of bars, take the square root. $$ \\text{RMS} = \\sqrt{\\frac{1}{N}\\sum\_{i=1}^{N} \\left(y\_i - \\hat{y}\_i\\right)^2}, \\qquad \\text{Score}\_t = \\frac{\\text{Dev}\_t}{\\text{RMS}} $$ The RMS error is the typical size of a prediction miss over the window, in log-price units. Dividing the current deviation by it converts a raw gap into a count of typical misses, which is the whole point: a deviation gets emphasized when the fit was tight, because RMS is small and the ratio blows up, and gets buried when the fit was loose, because RMS is large and the ratio shrinks toward zero. A two-typical-miss deviation off a clean fit outranks a two-dollar deviation off a messy one, automatically. When the fit was excellent the model has earned the right to flag the gap as meaningful; when the fit was poor the same gap is correctly demoted to noise. A worked number. Say the current log deviation is 0.018 and the window RMS error is 0.006, so the score is 3.0, a deviation three times the size of a normal miss, a genuine standout. Now suppose the index explained the member poorly that month and RMS is 0.020\. The same 0.018 deviation scores 0.9, less than one typical miss, unremarkable. Same gap, opposite verdict, decided entirely by how trustworthy the prediction was. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Compress and smooth, then read it as a filter The raw score has fat tails, so squash it into a bounded range with a normal CDF, the same compression the rest of the indicator pillar uses to tame outliers, scaled so the bulk of values land in a usable band. Then apply a short exponential smoothing to kill single-bar chatter, accepting the small lag that buys. ```python import numpy as np import pandas as pd from scipy.stats import norm def index_deviation(member: pd.Series, index: pd.Series, lookback: int = 60, smooth: int = 3) -> pd.Series: y = np.log(member) x = np.log(index) out = pd.Series(np.nan, index=member.index) for t in range(lookback, len(member)): xw = x.iloc[t - lookback + 1 : t + 1].to_numpy() # includes current bar yw = y.iloc[t - lookback + 1 : t + 1].to_numpy() xc, yc = xw - xw.mean(), yw - yw.mean() xss = (xc ** 2).sum() slope = (xc * yc).sum() / xss if xss > 0 else 1.0 pred = yw.mean() + slope * (xw - xw.mean()) rms = np.sqrt(((yw - pred) ** 2).mean()) dev = yw[-1] - pred[-1] score = dev / rms if rms > 0 else 0.0 out.iloc[t] = 100.0 * norm.cdf(score) - 50.0 # compress to [-50, 50] return out.ewm(span=smooth).mean() ``` The window ends on the current bar, so there is no look-ahead despite the current bar being inside the regression, because every value used was known at time t. The output is a bounded oscillator centered at zero: positive when the member sits above what the index predicts after accounting for fit quality, negative when it sits below, and near zero when either the member is tracking the prediction or the fit was too poor to trust the gap. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_270-visual-0.png) ## Where it earns its keep and where it lies Read this as a filter or a flag, the way the old article ["Intermarket Divergence as a Trading Filter"](https://aligrithm.com/intermarket-divergence-as-a-trading-filter/) framed divergence, not as a standalone entry. A large positive score says the member has run ahead of what the index justifies, which feeds two opposite theses depending on your model: mean-reversion, if you expect the member to snap back toward the index-implied level, or confirmation, if you read the breakaway as stock-specific strength the index has not caught up to. The indicator does not pick between those for you. It only quantifies and ranks the gap, and which interpretation pays is a property of your instrument and horizon that you have to test. The honest failure modes are specific. The regression assumes a stable linear relationship over the lookback, and that relationship is exactly what breaks during the regime shifts you most want to catch, so the slope you fit can already be stale, which is why the coupling check from the old article ["Spearman Coupling: When a Stock Decouples from Its Index"](https://aligrithm.com/spearman-coupling-when-a-stock-decouples-from-its-index/) belongs upstream of this: do not trust an index-implied prediction for a stock that has decoupled. The fit-quality weighting helps but does not save you, because a high RMS correctly demotes the signal but also means you are flying blind exactly when the stock has gone idiosyncratic. The lookback is an arbitrary dial trading reactivity against stability, the CDF scaling is a cosmetic choice about where values land, and the whole construction is a relative statement, the member versus this index, so feeding it the wrong index produces a confident number about nothing. Choose the reference deliberately, gate it behind a coupling check, and treat the score as one input among several rather than a trigger. ## KEY POINTS - Regress the member's log price on the index's log price over a lookback, and read the residual: the index is a one-variable forecast of the member, and the gap between predicted and actual is where stock-specific information shows up. - Use logs (proportional moves treated equally) and a plain straight line (both series are noisy, so a flexible model just fits noise). Include the current bar in the regression to stop a single fresh print from blowing up the deviation and to forbid the impossible perfect-fit-with-deviation state. - Weight the deviation by fit quality: divide it by the window RMS error. A two-typical-miss gap off a clean fit outranks the same dollar gap off a messy fit. Same deviation, opposite verdict, decided by how trustworthy the prediction was. - Compress the score with a normal CDF and apply short smoothing to get a bounded zero-centered oscillator, with the window ending on the current bar so there is no look-ahead. - It is a filter, not a trigger. A large gap supports either mean-reversion or breakaway confirmation, and the indicator does not choose; your instrument and horizon do. - Gate it behind the coupling check from the old article ["Spearman Coupling: When a Stock Decouples from Its Index."](https://aligrithm.com/spearman-coupling-when-a-stock-decouples-from-its-index/) The linear fit assumes a stable relationship, which is exactly what breaks in the regimes you care about, and a high RMS demotes the signal precisely when the stock has gone idiosyncratic. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Trading Systems and Methods - Perry Kaufman (Amazon)](https://www.amazon.com/s?k=Trading+Systems+and+Methods+Perry+Kaufman&ref=aligrithm.com) - [Cybernetic Trading Strategies - Murray Ruggiero (Amazon)](https://www.amazon.com/s?k=Cybernetic+Trading+Strategies+Murray+Ruggiero&ref=aligrithm.com) - [The Art of Currency Trading - Brent Donnelly (Amazon)](https://www.amazon.com/s?k=The+Art+of+Currency+Trading+Brent+Donnelly&ref=aligrithm.com) ### 2.54 Spectral Dilation: Why Long Cycles Drown Out Short Ones URL: https://aligrithm.com/spectral-dilation-why-long-cycles-drown-out-short-ones/ Last updated: 2026-07-27T13:23:18.000Z The old article ["The Frequency Response of Trading Indicators"](https://aligrithm.com/the-frequency-response-of-trading-indicators/) taught you to read a filter's magnitude curve, the gain at every cycle band, and to pick indicators by which cycles they keep rather than by their name. That curve describes what the filter does to a signal of fixed amplitude. It quietly assumes every cycle arrives at the same size. In real price data they do not, and the gap between that assumption and reality has a name: spectral dilation. Ignore it and your spectrum estimate will tell you the market is dominated by a long cycle when the truth is that the long cycle is just louder. ## Long cycles are bigger, not just slower Pull up any instrument across timescales and the pattern is impossible to miss: the annual swing is larger than the monthly swing, and the monthly swing is larger than the daily swing. Amplitude grows with the period of the cycle. This is not a quirk of one market; it is the structural consequence of price being close to a random walk, where the expected size of a move scales with the square root of the horizon, so longer cycles accumulate larger excursions. Ehlers' working approximation is even blunter and good enough for the trader: amplitude is roughly proportional to period. $$ \\text{Amplitude} \\;\\propto\\; \\text{Period} $$ Read what that does to a spectrum. A spectrum measures power, the squared amplitude, at each cycle. If amplitude rises with period, power rises faster still, so a raw spectral estimate of price is tilted: the slow end towers over the fast end before you have learned anything about which cycle actually carries tradeable structure. This is the same fact the old article on noise colors stated from the other direction, that price sits near the brownian spectrum with power piled at the slow cycles. Spectral dilation is that tilt seen as a measurement problem. ## When you can ignore it and when it ruins the estimate The fix depends entirely on how wide a span of cycles you are looking across, measured in octaves (an octave is a doubling of period: 10 to 20 bars is one octave, 10 to 40 is two). $$ \\text{narrow span (under \~1 octave)} \\;\\Rightarrow\\; \\text{dilation negligible} \\qquad \\text{several octaves} \\;\\Rightarrow\\; \\text{must compensate} $$ If you are hunting the dominant cycle inside a narrow band, say somewhere between 18 and 30 bars, the amplitude tilt across that small range is mild and you can ignore it; every candidate cycle is roughly the same size, so whichever shows the most power is genuinely dominant. The trouble starts when you let the estimator see several octaves at once, from a 10-bar cycle out to an 80-bar cycle. Now the amplitude tilt spans a factor of eight, the power tilt far more, and the slow cycles drown out the fast ones by construction. Your periodogram will report a long dominant cycle every time, not because the market is trending on that period but because the long cycle was handed an unfair amplitude advantage before the contest began. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Compensate, or pick a method that never tilts There are two honest responses. The first is to compensate explicitly: before reading the spectrum, scale each cycle's measured amplitude back down in proportion to its period, flattening the tilt so cycles compete on equal footing. This is the correction the dominant-cycle estimators apply when they have to work across a wide range, and skipping it is the single most common reason a homemade spectrum points at the wrong cycle. The second response, which the next article develops, is to use a method that does not produce the tilt in the first place. The autocorrelation periodogram, the old article ["Dominant Cycle Estimation Without Astrology"](https://aligrithm.com/dominant-cycle-estimation-without-astrology/) flagged as the production default, sidesteps dilation entirely because it works from correlation rather than raw amplitude, so there is no amplitude tilt to undo. Carry the caveat the frequency-response article and the evanescent-cycles article both pressed. A clean dominant-cycle number, even a dilation-corrected one, describes the recent sample, not the future; the cycle it names may already be drifting or gone. Spectral dilation is a measurement correction, not a source of edge. It keeps your estimator honest about which cycle is loudest right now, and nothing more. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_246-visual-0.png) ## KEY POINTS - A filter's frequency-response curve assumes every cycle arrives at the same amplitude; real price violates this because longer cycles are larger, not just slower. - Ehlers' working rule is amplitude proportional to period (a random-walk consequence), so a raw spectrum of price is tilted with power piled at the slow end, the same fact the noise-color article stated as price sitting near the brownian spectrum. - The tilt is measured in octaves of cycle period. Across a narrow span (under about one octave) it is negligible and you can ignore it; across several octaves it dominates and your periodogram reports a long cycle by construction. - Fix it either by compensating explicitly (scale each cycle's amplitude down in proportion to its period before reading the spectrum) or by using a method that never tilts. - The autocorrelation periodogram, the production default from the old article ["Dominant Cycle Estimation Without Astrology,"](https://aligrithm.com/dominant-cycle-estimation-without-astrology/) sidesteps dilation entirely because it works from correlation, not raw amplitude. - Dilation correction is a measurement fix, not edge; a clean dominant-cycle number still describes the recent sample and may already be drifting, per the old article ["Why Market Cycles Are Evanescent."](https://aligrithm.com/why-market-cycles-are-evanescent/) --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Statistically Sound Indicators for Financial Market Prediction - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Statistically+Sound+Indicators+for+Financial+Market+Prediction+Timothy+Masters&ref=aligrithm.com) - [Cycle Analytics for Traders - John Ehlers (Amazon)](https://www.amazon.com/s?k=Cycle+Analytics+for+Traders+John+Ehlers&ref=aligrithm.com) ### 4.51 Spearman Coupling: When a Stock Decouples from Its Index URL: https://aligrithm.com/spearman-coupling-when-a-stock-decouples-from-its-index/ Last updated: 2026-07-27T13:24:40.000Z A single stock spends most of its life chained to its index. SPY twitches and AAPL twitches with it, because the market factor is the largest piece of any large-cap return. That chain is not a constant. It loosens during earnings, breaks when a company-specific shock lands, and occasionally inverts when the stock trades on its own news while the tape does the opposite. The coupling itself is a regime variable, and you can measure it directly instead of guessing from a chart. The old article ["Cross-Pair Signals: Can EUR Predict GBP?"](https://aligrithm.com/cross-pair-signals-can-eur-predict-gbp/) made the same point in FX: two instruments move together because they share a driver, and the interesting information lives in the moments they stop. Here the shared driver is the index, the member is the stock, and the quantity worth tracking is how tightly the two are still linked right now. The wrong tool for this is Pearson correlation, and the old article ["Why the Median Often Beats the Mean in Trading Features"](https://aligrithm.com/why-the-median-often-beats-the-mean-in-trading-features/) already explained why. One earnings gap, one fat-tailed day, and Pearson's correlation swings on the strength of a single outlier, because it is built on squared deviations that the tail dominates. Stock returns are heavy-tailed by nature. Spearman's rank correlation fixes this by throwing away the magnitudes and keeping only the order. ## Rank correlation, and why it survives the tail Spearman correlation is Pearson correlation computed on ranks instead of raw values. Replace each return in the window with its rank (1 for the smallest, n for the largest), do the same for the index, and correlate the two rank vectors. Because a wild outlier becomes rank n no matter how wild, its leverage is capped. The closed form, when there are no ties, is a count of how far the two rank orders disagree. $$ \\rho = 1 - \\frac{6\\sum\_{i=1}^{n} d\_i^2}{n\\,(n^2 - 1)} $$ Read it piece by piece. The term d\_i is the difference between a day's rank in the stock and its rank in the index, so a day that is the stock's 3rd-largest move and the index's 3rd-largest move contributes a d\_i of zero. Square those rank gaps, sum them, scale by the window size n, and subtract from one. When the two orderings match perfectly, every d\_i is zero and rho is +1\. When they are reversed, the sum hits its maximum and rho is -1\. The output lives on the same minus-one-to-plus-one scale as Pearson, so it drops into any pipeline that expected a correlation, but it cannot be hijacked by one violent session. A worked case makes the outlier immunity concrete. Take five days where the stock and index agree on direction every day, so their Spearman rho is +1\. Now replace one of the stock's days with a 12-sigma earnings gap that the index did not share. Pearson correlation collapses toward zero because that one squared term swamps the others. Spearman barely moves, because the gap is still just the largest rank in the window, position n, and it only disagrees with the index by however many rank slots separate them. The rank machinery refuses to let one day rewrite the relationship. ## What a falling rho actually tells you The level of rho is the coupling state. A large-cap that normally rides the index sits at a high baseline, often 0.7 to 0.9 over a two- or three-month window. The signal is not the absolute number, it is the departure from that stock's own baseline. When rho slides from 0.85 toward 0.2, the stock has stopped taking its orders from the market and started trading on something idiosyncratic, an acquisition rumor, a guidance cut, a sector rotation that singled it out. When rho goes negative, the stock is fighting the tape, which is rare and usually news-driven. This is a regime flag, not a direction signal, and conflating the two is the standard way to lose money with it. Spearman coupling says nothing about whether the stock goes up or down. It says whether the index is still a valid reference for this stock today. That distinction drives two honest uses. The first is as a kill switch for any index-based model: if you trade the stock using the index as a lead or a hedge or a beta-neutral leg, a collapsed rho means the assumption underneath that model just expired, so suppress those signals until coupling returns. The old article ["Intermarket Divergence as a Trading Filter"](https://aligrithm.com/intermarket-divergence-as-a-trading-filter/) built exactly this caution case, where a broken relationship is a reason to stand down rather than a reason to act. The second use is the opposite posture for a different strategy: a decoupling event is the entry condition for stock-specific, event-driven trades, because the stock has announced that it is now moving on its own information. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_269-visual-0.png) ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Building it without lying to yourself The computation is a rolling Spearman of log returns, with the window ending at the current bar so nothing leaks from the future. Resist the temptation to rank the returns once over the whole history and roll a Pearson on those global ranks, because that smuggles in look-ahead: a day's global rank depends on days that have not happened yet. Rank inside each window instead. ```python import numpy as np import pandas as pd from scipy.stats import spearmanr def spearman_coupling(member: pd.Series, index: pd.Series, lookback: int = 60) -> pd.Series: rm = np.log(member).diff() ri = np.log(index).diff() out = pd.Series(np.nan, index=member.index) for t in range(lookback, len(member)): wm = rm.iloc[t - lookback + 1 : t + 1] wi = ri.iloc[t - lookback + 1 : t + 1] out.iloc[t] = spearmanr(wm, wi).correlation return out def decoupling_flag(rho: pd.Series, baseline_window: int = 250, drop: float = 0.3) -> pd.Series: baseline = rho.rolling(baseline_window).median() return (rho < baseline - drop).astype(int) ``` The first function returns the coupling line. The second turns it into a flag by comparing today's rho to a slow median baseline, the stock's own normal, computed with the median rather than the mean for the same outlier reason the rest of this pillar keeps returning to. A flag fires when coupling drops a fixed amount below baseline. Both numbers, the lookback and the drop threshold, are arbitrary dials, and you should treat them as such: the lookback sets how fast the coupling estimate reacts versus how noisy it is, and the drop threshold trades false alarms against missed decouplings. Neither has a correct value, only a value you have tested on your instrument. ## The limits worth stating out loud Rank correlation buys robustness by discarding magnitude, and that is also its blind spot. A stock can stay perfectly rank-coupled to the index while its volatility doubles, because order is preserved even as amplitudes blow out, so Spearman alone will not catch a pure volatility-regime shift. Pair it with a separate volatility measure if that case matters to you. The estimate is also noisy on short windows, and a short window is exactly what you want for catching a fast decoupling, so you sit on the same lag-versus-noise frontier every rolling statistic lives on. Coupling can revert as fast as it broke, since the news that decoupled the stock gets digested and the index reasserts itself, which means a decoupling flag is a short-lived state, not a trend to ride. And the relationship is built on the assumption that this index is the stock's real reference, so a member that has quietly migrated to a different driver, a semiconductor name that now tracks an industry ETF more than the broad market, will show a misleadingly low baseline that is structure, not signal. Measure the coupling against the reference that actually matters, and recheck which reference that is. ## KEY POINTS - A stock's coupling to its index is a regime variable, not a constant. It loosens on earnings, breaks on company-specific shocks, and rarely inverts, and you can measure it directly rather than reading it off a chart. - Use Spearman rank correlation, not Pearson. Ranks cap the leverage of any single outlier, so a 12-sigma earnings gap barely moves rho while it collapses a Pearson estimate, the heavy-tail problem from the old article ["Why the Median Often Beats the Mean in Trading Features."](https://aligrithm.com/why-the-median-often-beats-the-mean-in-trading-features/) - Rho is the coupling state, never a direction. Track its departure from the stock's own baseline, not its absolute level: a slide from 0.85 toward 0.2 means the index has stopped being a valid reference for this stock. - Two honest uses, opposite in posture: kill index-based signals when coupling collapses (the caution case from the old article ["Intermarket Divergence as a Trading Filter"](https://aligrithm.com/intermarket-divergence-as-a-trading-filter/)), or arm stock-specific event trades because the stock now moves on its own information. - Compute it as a rolling Spearman with the window ending on the current bar, ranking inside each window. Ranking once over all history leaks the future. - It discards magnitude, so it misses pure volatility-regime shifts, is noisy on the short windows that catch fast decouplings, reverts quickly, and assumes the chosen index is the stock's true reference. Verify that reference and pair rho with a volatility measure. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Trading Systems and Methods - Perry Kaufman (Amazon)](https://www.amazon.com/s?k=Trading+Systems+and+Methods+Perry+Kaufman&ref=aligrithm.com) - [Cybernetic Trading Strategies - Murray Ruggiero (Amazon)](https://www.amazon.com/s?k=Cybernetic+Trading+Strategies+Murray+Ruggiero&ref=aligrithm.com) - [The Art of Currency Trading - Brent Donnelly (Amazon)](https://www.amazon.com/s?k=The+Art+of+Currency+Trading+Brent+Donnelly&ref=aligrithm.com) ### 2.53 The Roofing Filter: Band-Limit Before You Build Any Indicator URL: https://aligrithm.com/the-roofing-filter-band-limit-before-you-build-any-indicator/ Last updated: 2026-07-27T13:23:18.000Z The old article ["Feature Engineering Before Machine Learning"](https://aligrithm.com/feature-engineering-before-machine-learning/) made the case that what you feed an indicator decides its ceiling, and that most of the achievable edge is won or lost before any model sees the data. The old article ["High-Pass Filters for Traders"](https://aligrithm.com/high-pass-filters-for-traders/) and the old article ["The Trader's Guide to Low-Pass Filters"](https://aligrithm.com/the-traders-guide-to-low-pass-filters/) gave you the two tools to control that input. The roofing filter is what you get when you put them in series, and it answers a question almost no retail trader asks before slapping an RSI on a chart: what cycles am I even letting into this thing? ## Every indicator runs on whatever spectrum you hand it Linear systems theory frames any indicator as three boxes: input, transfer, output. Traders obsess over the transfer box, the indicator formula itself, and ignore the input box entirely, which is where the damage happens. Closing prices contain everything at once: a slow multi-month trend, a tradeable cycle in the middle, and fast bar-to-bar noise. Feed that raw stew into an oscillator and the oscillator dutifully responds to all three. An RSI computed on raw price drifts because the slow trend is dragging its baseline, and it whipsaws because the fast noise is jiggling its numerator. The oscillator was never broken; you handed it garbage at both ends of the spectrum. The roofing filter fixes the input box before the transfer box ever runs. It strips the two ends you do not want and passes only the band in the middle where tradeable cycles live. $$ \\text{prices} \\;\\longrightarrow\\; \\underbrace{\\text{High-Pass}}\_{\\text{kill slow trend}} \\;\\longrightarrow\\; \\underbrace{\\text{SuperSmoother}}\_{\\text{kill fast noise}} \\;\\longrightarrow\\; \\text{indicator (RSI, etc.)} $$ ## Two filters, two cutoffs, one clean band The first stage is the high-pass filter from the old high-pass article. A high-pass passes fast cycles and removes slow ones, so set its cutoff at a long period, around 40 to 48 bars, and it strips out everything slower than that: the multi-month drift, the wandering baseline, the brownian trend component. What survives is detrended, centered, free of the slow drag that was tilting your oscillator. The second stage is the SuperSmoother, the two-pole low-pass from the old low-pass article. A low-pass passes slow cycles and removes fast ones, so set its cutoff at a short period, around 10 bars, and it removes everything faster than that: the single-bar jitter, the microstructure noise, the high-frequency garbage that produces false oscillator crossings. The SuperSmoother matters specifically because it is the critically-damped two-pole design that gives a clean 12 dB-per-octave rolloff without the overshoot and ringing a naive smoother adds, so it kills noise without inventing wiggles of its own. Cascade the two and the high-pass sets the roof while the SuperSmoother sets the floor. What passes is a band, roughly 10 to 48 bars in this example, which is why a roofing filter is a wide-band band-pass built from a high-pass and a low-pass stacked together. $$ \\text{High-Pass cutoff (e.g. 48 bars)} \\;=\\; \\text{upper period limit (roof)} \\qquad \\text{SuperSmoother cutoff (e.g. 10 bars)} \\;=\\; \\text{lower period limit (floor)} $$ The two cutoffs are choices, not constants. Set the roof to the longest cycle you consider tradeable and the floor just above the noise you cannot trade, and recognize that picking them is itself a modeling decision you should make deliberately rather than inheriting from a default. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Why this beats reaching for a different oscillator The output of the roofing filter is useful two ways. You can read it directly as a detrended, denoised oscillator that crosses zero at cycle turns, or you can use it as the input to a conventional indicator. Feed roofed prices into an RSI and the RSI behaves: its baseline stops drifting because the trend is gone, and its crossings stop chattering because the noise is gone, so the same formula that looked useless on raw price becomes a clean cycle reader. You did not change the indicator. You fixed what you fed it. This is the old feature-engineering article's thesis made concrete. When an oscillator misbehaves, the reflex is to try a different oscillator, or to tune its length, or to add a confirming indicator on top. The roofing filter says the problem is upstream: you never controlled the input spectrum, so you were asking the indicator to separate trend, cycle, and noise all at once, a job it was never built to do. Band-limit first, and every downstream indicator gets easier. Keep the honest caveats from the filter articles in view. Band-limiting reshapes history, it does not predict; the roofing filter makes a cleaner feature, not a forecast. Both stages cost lag, and a sharper roof or floor costs more of it, so you are trading responsiveness for cleanliness exactly as the low-pass article warned. And a beautifully band-limited input still has to earn its keep: a clean cycle feature that carries no predictive value on your instrument is a clean feature that loses money, which is the empirical test the feature-engineering article insisted you run before trusting any of this. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_245-visual-0.png) ## KEY POINTS - Every indicator is input-transfer-output; traders tune the transfer (the formula) and ignore the input. Raw price carries slow trend, a middle cycle, and fast noise all at once, so an oscillator fed raw price drifts on the trend and whipsaws on the noise. - The roofing filter fixes the input first: a high-pass stage removes the slow trend (the roof) and a SuperSmoother low-pass stage removes the fast noise (the floor), passing only the tradeable cycle band in between. - Set the high-pass cutoff at a long period (around 40-48 bars) and the SuperSmoother at a short one (around 10 bars); the result is a wide-band band-pass. The cutoffs are deliberate modeling choices, not defaults. - The SuperSmoother is the critically-damped two-pole low-pass from the old low-pass article, chosen because it rolls off noise cleanly without the overshoot and ringing a naive smoother adds. - Use the output directly as a detrended denoised oscillator, or feed roofed prices into a conventional indicator like RSI; the same formula that looked useless on raw price becomes a clean cycle reader because its baseline stops drifting and its crossings stop chattering. - It reshapes history, it does not predict; both stages cost lag, and a band-limited input still has to pass the empirical test from the old feature-engineering article that the band you kept actually carries predictive value. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Statistically Sound Indicators for Financial Market Prediction - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Statistically+Sound+Indicators+for+Financial+Market+Prediction+Timothy+Masters&ref=aligrithm.com) - [Cycle Analytics for Traders - John Ehlers (Amazon)](https://www.amazon.com/s?k=Cycle+Analytics+for+Traders+John+Ehlers&ref=aligrithm.com) ### 3.34 The Trade-Frequency Floor: Choosing a Threshold Honestly URL: https://aligrithm.com/the-trade-frequency-floor-choosing-a-threshold-honestly/ Last updated: 2026-07-27T13:23:57.000Z You have an indicator that correlates with next-bar returns. The old article ["How to Test Indicator Thresholds Without Fooling Yourself"](https://aligrithm.com/how-to-test-indicator-thresholds-without-fooling-yourself/) argued that finding such an indicator is most of the battle. The rest of the battle is picking the threshold that turns it into a rule, and that step quietly destroys more strategies than the indicator selection ever does. Set the threshold too loose and you take a flood of marginal trades that drown the edge in costs. Set it too tight and you take a handful of trades with a gorgeous profit factor that is pure luck. The optimizer, left alone, walks straight into the tight trap, because the tightest thresholds always look best in-sample. The defense is a floor on how often the rule fires. ## The two traps a threshold optimizer faces Sort your indicator values ascending and keep each one paired with the return that followed it. Now slide a candidate threshold up through the sorted values. For each candidate, everything above is the long region and everything below is the flat or short region, and you can read off the win count, loss count, and profit factor on each side. $$ \\text{Frac}\_{>} = \\frac{\\#\\{\\text{cases with indicator} \\ge \\text{threshold}\\}}{N}, \\qquad \\text{Frac}\_{<} = 1 - \\text{Frac}\_{>} $$ Frac-above is the fraction of all bars that clear the threshold, and Frac-below is the rest. Push the threshold to the very top of the sorted values and Frac-above shrinks toward zero: you are now selecting the two or three most extreme indicator readings in the whole sample. Those few bars almost always show a spectacular profit factor, because with three observations you can get any number you like by chance. That is the tight trap. Pull the threshold down toward the median and Frac-above grows, the profit factor regresses toward the indicator's true thin edge, and transaction costs start eating the trades that barely cleared the line. That is the loose trap. An unconstrained optimizer maximizing profit factor will pick the tight end almost every time, because a sample of three lucky bars beats a sample of three thousand honest ones on any in-sample metric. The result is a threshold tuned to noise, and it has the same disease the old article ["Why 30 Trades Is Not a Strategy"](https://aligrithm.com/why-30-trades-is-not-a-strategy/) diagnosed: a performance number computed from too few events to mean anything. ## The floor Place a minimum on Frac-above before you let the optimizer choose. Require that any acceptable threshold trades on at least some fraction of the bars, say five or ten percent, and search for the best profit factor only among thresholds that clear the floor. The floor walls off the region where the optimizer manufactures fantasy results from a handful of observations, and forces it to find an edge that survives across a statistically meaningful slice of the data. Two accounting details keep the search honest. When several bars share the same indicator value, a trial threshold can land on a run of ties, so the wins and losses for that trial are valid only at the first occurrence of the value; evaluate each distinct threshold once and do not let duplicates inflate your count. And keep "trades" and "bars" straight: a trade is one open-and-close of a position, while the bars are the individual returns inside it. The frequency floor is a floor on bars that satisfy the threshold, which is the unit that actually carries your statistical power, not on the smaller number of round trips. ## Proving the floor is not enough on its own A frequency floor stops you from overfitting to three bars, but it does not prove the surviving threshold has real predictive power. The optimizer is still searching, and a powerful enough search can split returns into wins and losses well even when the indicator carries no information at all. The test for that is permutation. Shuffle the returns to break their pairing with the indicator, re-run the full threshold optimization on the shuffled data, and record the best profit factor it finds. Repeat many times. If the indicator's edge is real, the original pairing should beat the shuffled ones; if the optimizer gets comparable profit factors from randomly paired returns, your threshold was an artifact of the search, not a property of the indicator. $$ p \\approx \\frac{k + 1}{m + 1} $$ Here m is the number of shuffles you ran and k is how many of them produced a profit factor at least as good as the original, unshuffled result. The plus-ones keep the estimate honest by counting the original arrangement as one of the possible outcomes. A small p means the optimizer rarely matched your result by chance, so the indicator-return pairing did the work. A p near a half means correct pairing bought you nothing over random pairing, and the threshold, floor or no floor, is worthless. Run as many shuffles as you can afford, because the p-value is itself a noisy estimate, and one shuffle proves nothing on a coin that lands the right way half the time. The honest workflow is both steps together: a frequency floor so the optimizer cannot hide in a corner of three lucky bars, and a permutation test so you can see whether the threshold that cleared the floor reflects the indicator or reflects the search. Skip the floor and you overfit to extremes; skip the permutation and you trust a search that can fool you on noise. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Visualizing the threshold sweep ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_274-visual-0.png) ## KEY POINTS - Finding an indicator that correlates with returns is most of the battle; choosing the threshold that turns it into a rule is the step that quietly kills strategies. - Sort indicator values ascending, keep each paired with its following return, and slide a candidate threshold up through them, reading win count, loss count, and profit factor on each side. - The loose trap (threshold near the median) floods you with marginal trades that costs eat. The tight trap (extreme threshold) selects a handful of bars whose spectacular profit factor is luck. - An unconstrained optimizer maximizing profit factor walks into the tight trap, because three lucky bars beat thousands of honest ones on any in-sample metric, the same too-few-events disease from the old article ["Why 30 Trades Is Not a Strategy."](https://aligrithm.com/why-30-trades-is-not-a-strategy/) - The defense is a trade-frequency floor: require any acceptable threshold to fire on at least some fraction of the bars (say 5-10%), and search for the best profit factor only above the floor. - Evaluate each distinct threshold once so tied indicator values do not inflate counts, and floor the bars that satisfy the threshold, not the smaller number of round-trip trades, because bars carry the statistical power. - The floor stops overfitting to extremes but does not prove predictive power. Permute: shuffle returns to break the pairing, re-optimize, and compute p as (k+1)/(m+1) where k of m shuffles matched the original. A small p means the indicator did the work; p near a half means the threshold is an artifact of the search. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Testing and Tuning Market Trading Systems - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Testing+and+Tuning+Market+Trading+Systems+Timothy+Masters&ref=aligrithm.com) - [Data Mining Algorithms in C++ - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Data+Mining+Algorithms+in+C%2B%2B+Timothy+Masters&ref=aligrithm.com) ### 2.52 Noise Colors: White, Pink, and Brownian Markets URL: https://aligrithm.com/noise-colors-white-pink-and-brownian-markets/ Last updated: 2026-07-27T13:23:17.000Z The old article ["Noise Is Not Volatility"](https://aligrithm.com/noise-is-not-volatility/) split two quantities traders constantly confuse: how far price travels versus how much of that travel cancels. The old article ["Efficiency Ratio Explained for Traders"](https://aligrithm.com/efficiency-ratio-explained-for-traders/) turned the second one into a single number between zero and one. Both measure the same underlying thing from the time side. There is a second way to look at it, from the frequency side, and it comes with a vocabulary borrowed from physics: noise has a color, and the color tells you whether a market remembers its own past. ## One slope describes the whole spectrum Take any series, compute its power spectrum (how much energy sits at each cycle speed), and most natural signals show power falling off as a simple power of frequency. $$ S(f) \\;\\propto\\; \\frac{1}{f^{\\alpha}} $$ Read it plainly: S(f) is the power at frequency f, and alpha is the exponent that sets how fast power drops as you move from slow cycles to fast ones. A single number, alpha, fixes the shape. The reason this is useful is that the relationship turns into a straight line the moment you take logs of both sides, so you never have to fit a curve, you fit a slope. $$ \\log S(f) \\;=\\; -\\alpha \\,\\log f \\;+\\; \\text{constant} $$ Plot log power against log frequency, draw the best straight line through it, and the slope is minus alpha. That slope is the color of the noise. ## Three colors, three market personalities White noise has alpha equal to zero. The spectrum is flat, every cycle speed carries the same power, and there is zero correlation in time. Each sample is independent of the last. A market that were pure white noise at your trading horizon would have no exploitable memory at all. Brownian noise (also called red noise) has alpha equal to two. Power piles up at the slow end and collapses at the fast end. This is the spectrum of a random walk, which is what you get when you add up independent white-noise increments. Price itself usually lives near here, because price is the running sum of its returns, and summing is integration. Pink noise sits between them at alpha equal to one. This is the interesting one. Pink noise is synonymous with long memory, what physicists call long-range dependence: the series is neither a fresh coin flip each bar nor a pure random walk, but something with persistent structure that decays slowly. The ubiquity of pink noise across physics, biology, and economics is one of the genuinely unsolved puzzles in science, and markets show it at many horizons. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Why returns and prices have different colors Calculus moves the color, and the rule is clean: integrating a signal raises alpha by two, differentiating it lowers alpha by two. $$ \\text{integrate:}\\;\\; \\alpha \\rightarrow \\alpha + 2 \\qquad\\qquad \\text{differentiate:}\\;\\; \\alpha \\rightarrow \\alpha - 2 $$ This explains a fact every quant half-knows. Returns are the difference (the derivative) of log price; price is the running sum (the integral) of returns. If returns were close to white (alpha near zero), integrating them gives price an alpha near two, which is the brownian spectrum of a random walk. Difference the price back into returns and you drop alpha back toward zero. So the same market wears different colors depending on whether you look at price or at returns, and the gap between the two slopes is pinned near two by construction. The corollary matters: you cannot turn brownian into pink with simple differencing, because differencing moves alpha in jumps of two, never by one. Genuine pink behavior, alpha near one, is a property of the data, not an artifact you can manufacture by transforming it. ## What the slope buys a trader The slope is a regime classifier that lives on the same axis as the efficiency ratio, approached from the frequency side instead of the time side. A return series whose slope sits near zero is close to white, motion cancels, and the old article's efficiency-ratio reading would be low, the fade environment. A return series with a slope pushed toward one carries long memory, moves reinforce across many lags, and the efficiency ratio would read higher, the trend environment. The two diagnostics agree because they are measuring the same persistence, one by net-distance-over-path, the other by where energy sits in the spectrum. Two cautions before you build on it. The slope is a sample estimate, not a constant of the market, and it drifts as participants and volatility regimes change, the same non-stationarity that haunts every other indicator. And a slope estimate is only as honest as your frequency range: fit it over too few octaves of cycle period and the line you draw is mostly noise about the noise. Measure it over a wide span of periods, report it at the horizon you actually trade, and treat a shift in color as a regime change worth respecting rather than a number to optimize. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_242-visual-0.png) ## KEY POINTS - Any series' power spectrum tends to fall off as one over frequency to a power, S(f) proportional to 1/f^alpha, so a single slope describes the whole spectrum once you plot log power against log frequency. - White noise has slope alpha = 0 (flat spectrum, no time memory), brownian/red noise has alpha = 2 (the random-walk spectrum, power piled at slow cycles), and pink noise has alpha = 1, the long-memory case between them. - Pink noise means long-range dependence: the series is neither independent each bar nor a pure random walk, but carries slowly decaying persistent structure. - Integrating a signal raises alpha by 2, differentiating lowers it by 2\. Price is the integral of returns, so price reads near brownian (alpha 2) while returns read near white (alpha 0); the two-step jump means you cannot manufacture genuine pink behavior by differencing. - The slope is a regime classifier on the same axis as the efficiency ratio from the old article ["Efficiency Ratio Explained for Traders,"](https://aligrithm.com/efficiency-ratio-explained-for-traders/) approached from frequency instead of time: slope near zero is the fade environment, slope pushed toward one is the trend environment. - The estimate is non-stationary and only as good as the frequency range you fit it over. Measure across many octaves of cycle period, report it at your trading horizon, and treat a color change as a regime shift, not a parameter to optimize. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Statistically Sound Indicators for Financial Market Prediction - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Statistically+Sound+Indicators+for+Financial+Market+Prediction+Timothy+Masters&ref=aligrithm.com) - [Cycle Analytics for Traders - John Ehlers (Amazon)](https://www.amazon.com/s?k=Cycle+Analytics+for+Traders+John+Ehlers&ref=aligrithm.com) ### 3.33 Why You Compute Profit Factor Per Bar, Not Per Trade URL: https://aligrithm.com/why-you-compute-profit-factor-per-bar-not-per-trade/ Last updated: 2026-07-27T13:23:56.000Z The old article ["Why Profit Factor Can Lie"](https://aligrithm.com/why-profit-factor-can-lie/) showed how three lucky winners inflate the headline number. There is a second, quieter way profit factor lies, and it shows up the moment you test an indicator with a threshold rule rather than discrete entries and exits. Hold a position for as long as the indicator stays above some level, and the question of what counts as a "trade" stops having an obvious answer. Answer it the convenient way, by grouping every consecutive bar of a hold into one trade, and you can manufacture a profit factor of infinity out of an indicator that is barely better than a coin. The fix is to stop counting trades and start counting bars. ## The grouping artifact Threshold testing works like this: you stay long while the indicator sits above its threshold and flat otherwise, so a single "position" can run for one bar or forty. Take one such position that ran five bars, with per-bar log returns of 1, 3, -2, -2, 2, then later another that ran three bars at 2, 3, 1\. The honest accounting treats each bar as its own outcome. Sum the up-bars, sum the down-bars, divide. $$ \\text{PF}\_{\\text{bar}} = \\frac{\\sum\_{i:\\, r\_i > 0} r\_i}{\\sum\_{i:\\, r\_i < 0} |r\_i|} = \\frac{1+3+2+2+3+1}{2+2} = \\frac{12}{4} = 3 $$ Each term is a single bar's log return, the up-bars on top and the absolute value of the down-bars on the bottom. Now do it the lazy way. Net each hold into one number: the first position nets 1+3-2-2+2 = 2, the second nets 2+3+1 = 6\. Both holds finished positive, so there are two winning trades and zero losing trades. $$ \\text{PF}\_{\\text{trade}} = \\frac{2 + 6}{0} = \\infty $$ Zero in the denominator, profit factor of infinity, and the indicator looks flawless. It is not. Inside those holds were four losing bars that the netting quietly absorbed. The per-bar profit factor of 3 is the truth; the per-trade infinity is an artifact of where you happened to draw the trade boundaries. ## Why grouping is the wrong unit here A discrete strategy has natural trade boundaries: you decide to open, you decide to close, and the round trip is a real event you could point to on a chart. A threshold rule has no such events. You are not making entry and exit decisions; you are continuously in or out as a function of one indicator, and the bars are the only natural unit of P&L. Grouping consecutive in-bars into a "trade" is an arbitrary choice, and the arbitrariness is the problem. The longer your average hold, the more down-bars get buried inside winning holds, and the more the per-trade profit factor drifts above the per-bar truth, all the way to infinity once no single hold happens to net negative. The infinity is the tell, but the distortion is there long before you hit it. An indicator with a per-bar profit factor of 1.2, a thin but real edge, can show a per-trade profit factor of 4 or 5 simply because its holds are long enough to net positive most of the time. You would chase that 4 into production and discover the 1.2 the hard way. One more reason to work in bars: log returns add. A bar's log return plus the next bar's log return equals the two-bar log return, so summing up-bars and down-bars across the whole test is exact, with no compounding error and no dependence on how you slice the holds. Raw percentage returns do not add cleanly, so use logs and the accounting stays honest no matter how long the position runs. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## What this changes in practice Compute profit factor per bar whenever the position is driven by an indicator threshold, an indicator-and-model output, or anything that keeps you in for a variable, signal-determined number of bars. Reserve per-trade profit factor for genuinely discrete strategies with explicit, rule-based entries and exits, where a round trip is a real decision and not an accident of where a threshold lapsed. Mixing the two is how a methodology paper reports a profit factor that no live account ever reproduces. This sits next to the lesson from ["Why Profit Factor Can Lie."](https://aligrithm.com/why-profit-factor-can-lie/) That article warned that a few outlier trades dominate the ratio; this one warns that the act of forming trades at all can dominate the ratio. Both collapse to the same discipline: define your unit of P&L before you compute anything, prefer the smallest natural unit, and distrust any profit factor that depends on how you grouped the data rather than on the data. ## Visualizing the artifact ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_273-visual-0.png) ## KEY POINTS - A threshold rule keeps you in a position for a variable, signal-determined number of bars, so there is no natural definition of a "trade." The bars are the only natural unit of P&L. - Group consecutive in-bars into one trade and net them, and losing bars get absorbed into winning holds. Once no single hold nets negative, the denominator is zero and per-trade profit factor goes to infinity. - The same data computed per bar (sum of up-bars over the absolute sum of down-bars) gives the honest, finite number. In the worked example, per bar is 3.0 while per trade is infinity. - The distortion exists long before infinity: a per-bar edge of 1.2 can show a per-trade profit factor of 4 or 5 purely because the holds are long enough to net positive. - Use log returns so per-bar P&L adds exactly across any hold length, with no dependence on how you sliced the position. - Compute per-bar profit factor for indicator-threshold and model-output rules; reserve per-trade profit factor for genuinely discrete strategies with explicit rule-based entries and exits. - This is the second way profit factor lies, alongside the outlier domination from the old article ["Why Profit Factor Can Lie."](https://aligrithm.com/why-profit-factor-can-lie/) Both reduce to one rule: fix your unit of P&L before computing, and distrust any profit factor that depends on grouping rather than on the data. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Testing and Tuning Market Trading Systems - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Testing+and+Tuning+Market+Trading+Systems+Timothy+Masters&ref=aligrithm.com) - [Data Mining Algorithms in C++ - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Data+Mining+Algorithms+in+C%2B%2B+Timothy+Masters&ref=aligrithm.com) ### 2.51 Measuring the Dominant Cycle with Band-Pass Zero Crossings URL: https://aligrithm.com/measuring-the-dominant-cycle-with-band-pass-zero-crossings/ Last updated: 2026-07-27T13:23:17.000Z The old article ["Dominant Cycle Estimation Without Astrology"](https://aligrithm.com/dominant-cycle-estimation-without-astrology/) surveyed the heavy machinery for measuring a cycle period: DFT periodograms and autocorrelation estimators that resolve a frequency cleanly but lag 12 to 32 bars. This article gives the cheap, fast alternative. Run price through a band-pass filter, count the bars between the times the output crosses zero, and double it. That is the dominant cycle period, measured with a few bars of lag and no spectral math. It works, it is the lowest-latency cycle estimator in the pillar, and it carries a specific bias that will fool you if you tune the filter wrong. ## A cycle is two zero crossings The method rests on one geometric fact about an oscillation. A clean cycle, centered on zero, crosses zero twice per full period: once going up, once going down. So the gap between two successive zero crossings is half a cycle. $$ T\_{\\text{dominant}} = 2 \\times (\\text{bars between successive zero crossings}) $$ Each successive zero crossing of the band-pass output marks a half-cycle boundary, and twice the spacing between them is the full period. Feed price into a band-pass, watch its output swing above and below zero, time the crossings, double the spacing, and you have the dominant cycle. The band-pass is doing the essential work the raw price cannot: price has a trend and noise that smear its crossings, while the band-pass strips both away and leaves a centered oscillation whose crossings are meaningful. This is why you measure on the band-pass output, never on raw price. ## Widen the passband, or the filter measures itself Here is the trap, and it comes straight from the previous article on Q. A band-pass filter has a center period it is tuned to and a selectivity that decides how hard it rejects neighbors. A narrow, high-Q band-pass resonates at its own tuned period, so its output oscillates near that period almost regardless of what the market is doing. Measure the zero crossings of a narrow band-pass and you largely recover the period you tuned the filter to, not the period the market is running. The filter answers your question with your own assumption. That is resonance bias, the reason the old article ["Dominant Cycle Estimation Without Astrology"](https://aligrithm.com/dominant-cycle-estimation-without-astrology/) listed band-pass zero-crossing as fast but resonance-biased. The fix is to widen the passband, lowering the Q. A broad band-pass passes a wide range of cycles, so its output is free to oscillate at whatever period dominates the market within that range, rather than being dragged to a single tuned tone. You give up selectivity and accept a noisier, less pure output, and in exchange the zero crossings reflect the market's cycle instead of the filter's. So the measurement setting is the opposite of the isolation setting: when you want to isolate a known cycle you go narrow, but when you want to measure an unknown cycle you go wide, so the instrument does not bias the reading toward its own tuning. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Fast and cheap, noisy and fragile: where it fits This estimator sits at one end of the speed-versus-stability trade the old article ["Dominant Cycle Estimation Without Astrology"](https://aligrithm.com/dominant-cycle-estimation-without-astrology/) mapped, the mirror image of the periodogram. The zero-crossing method has very low lag, only a few bars, because it reacts as soon as the output crosses zero, with no long analysis window. That speed is its entire reason to exist. The price is variance: a single noisy crossing, or a half-cycle distorted by a passing shock, throws the estimate off, and with no averaging window to lean on the reading jumps around bar to bar. It is the low-lag, high-variance corner, where the periodogram is the high-lag, low-variance corner. So deploy it accordingly. Smooth the raw period estimate over several crossings before trusting it, which gives back a little of the lag you saved but tames the jumpiness. Keep the passband wide to avoid resonance bias, and gate the whole thing on amplitude, because when the band-pass output is small there is no cycle to cross zero meaningfully and the spacing is noise. And hold the standing skepticism of this pillar: the period you measure describes the recent past, and the cycle it found may already be drifting or dead, so a zero-crossing reading is a fast hypothesis about current structure, not a forecast. Use it where you need a quick cycle estimate to feed an adaptive filter and can tolerate variance, and reach for the autocorrelation periodogram when you need a stable number and can pay the lag. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_241-visual-0.png) ## KEY POINTS - A centered oscillation crosses zero twice per cycle, so the dominant period equals twice the bar spacing between successive zero crossings of a band-pass output. - Measure on the band-pass output, not raw price: the band-pass strips the trend and noise that smear price's crossings, leaving a centered wave whose crossings are meaningful. - A narrow, high-Q band-pass resonates at its own tuned period, so its zero crossings recover the period you assumed, not the market's. This is resonance bias, the cost of measuring with a selective filter. - The fix is a wide, low-Q passband, the opposite of the isolation setting: wide lets the output oscillate at the market's dominant period instead of the filter's tuning, trading purity for an honest reading. - The method is the low-lag, high-variance corner of the speed-stability trade from the old article ["Dominant Cycle Estimation Without Astrology,"](https://aligrithm.com/dominant-cycle-estimation-without-astrology/) reacting in a few bars but jumping around on noisy crossings, the mirror of the high-lag periodogram. - Deploy it by smoothing the estimate over several crossings, keeping the passband wide, and gating on amplitude; treat the reading as a fast hypothesis about current structure, not a forecast, since the cycle may already be drifting or dead. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Statistically Sound Indicators for Financial Market Prediction - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Statistically+Sound+Indicators+for+Financial+Market+Prediction+Timothy+Masters&ref=aligrithm.com) - [Cycle Analytics for Traders - John Ehlers (Amazon)](https://www.amazon.com/s?k=Cycle+Analytics+for+Traders+John+Ehlers&ref=aligrithm.com) ### 2.73 Volume Momentum URL: https://aligrithm.com/volume-momentum/ Last updated: 2026-07-27T13:23:31.000Z Every other volume indicator in this group ties volume to price, the close position, the direction of the change, the price level. This one ignores price entirely. Volume Momentum asks one question: is the market more active now than it has been, and by how much. That sounds trivial until you remember why activity matters. Volume surges precede and accompany regime changes, breakouts run on participation, and dead tape kills mean-reversion edges that need turnover to work. A clean reading of recent volume against its own recent baseline is a regime gauge, and like every raw volume quantity it is unusable until you normalize it. This article builds it from the same recipe the rest of the group uses: a short window over a long window, a log, and a CDF squash. ## Recent activity against its own baseline The construction borrows the structure of a price momentum oscillator and swaps price for volume. Take the average volume over a short recent window and divide it by the average volume over a longer window that sets the baseline, the way a fast moving average over a slow moving average reads price momentum. The longer window is a multiple of the shorter one, so a single lookback and a multiplier define both. Take the log of the ratio so the result centers on zero and the spread between an activity surge and an activity drought becomes symmetric. $$ \\text{Raw} = \\log\\!\\left(\\frac{\\frac{1}{1 + m \\cdot LB}\\sum\_{k=0}^{LB} V\_k}{\\sum\_{k=0}^{m \\cdot LB - 1} V\_k}\\right) $$ The numerator is recent average volume over the short window of length LB, the denominator the broader volume over the longer window scaled by the multiplier m. Their ratio reads above one when recent volume runs hot relative to the baseline and below one when the tape has gone quiet. The log does the same job it did in the volume-weighted MA ratio: it turns the multiplicative center of one into an additive center of zero and makes a doubling of activity and a halving of activity equal distances from the middle. The old article ["Why Most Indicators Should Be Transformed Before Modeling"](https://aligrithm.com/why-most-indicators-should-be-transformed-before-modeling/) is the reason every one of these constructions ends with a log and not a raw ratio; a ratio bunched around one with a heavy upper tail is the textbook case for it. ## Normalize for the multiplier, then squash The log ratio still has a spread that depends on how far apart the two windows sit, governed by the multiplier. A wide gap between short and long windows produces a larger swing than a narrow gap, so to keep the indicator comparable across settings you scale by the square root of the multiplier before compressing. Then run it through a normal-CDF squash to bound the output and rescale to plus or minus fifty. $$ V\_{\\text{norm}} = 100 \\cdot \\Phi\\!\\left(\\frac{3 \\cdot \\text{Raw}}{\\sqrt{m}}\\right) - 50 $$ The Raw log ratio sits inside, divided by the square root of the multiplier to normalize for the window gap, times a constant of three to spread the bulk of the distribution across the linear region of the CDF rather than letting it bunch in the middle. Phi is the standard normal CDF, the bounded squashing transform from the old article ["Taming Indicator Tails with Sigmoid Transforms"](https://aligrithm.com/taming-indicator-tails-with-sigmoid-transforms/), which maps the whole real line into a fixed interval while keeping the body of the distribution roughly linear and pulling rare volume spikes back inside the range. The minus fifty recenters the output to swing around zero. The result is a volume-activity oscillator on a fixed minus-fifty to plus-fifty axis, stationary, bounded, and comparable across instruments and across decades despite the relentless growth in raw volume. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## A regime gauge, not a signal Volume Momentum does not tell you which way to trade, because it knows nothing about price. It tells you whether the conditions your other tools assume are present. A reading near plus fifty says the tape is far more active than its baseline, which is when breakouts and trend continuations tend to have fuel and when the high-noise mean-reversion regime can flip; a reading near minus fifty says activity has collapsed, which is when momentum signals tend to fail for lack of participation and when illiquidity makes every fill expensive. Use it as a filter and a context variable, the input that decides which of your other indicators to trust right now, rather than as a thing you act on directly. This is the same role volume plays throughout the group, with the noise warning from the old article ["Why Median Filters Are Useful for Volume and Outliers"](https://aligrithm.com/why-median-filters-are-useful-for-volume-and-outliers/) still in force: volume is the most spike-prone series you will handle, so the smoothing built into the two windows is doing real work before the log and the squash ever see the data. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_266-visual-0.png) ## KEY POINTS - Volume Momentum ignores price and measures only whether current activity runs hot or cold against its own recent baseline, a regime gauge rather than a directional signal. - It divides average volume over a short window by average volume over a longer window set by a multiplier, the same short-over-long structure a price momentum oscillator uses, then takes the log to center on zero and symmetrize surges against droughts. - The log is mandatory because a raw ratio bunched around one with a heavy upper tail is the textbook defect from the old article ["Why Most Indicators Should Be Transformed Before Modeling"](https://aligrithm.com/why-most-indicators-should-be-transformed-before-modeling/). - The log ratio's spread depends on the window gap, so dividing by the square root of the multiplier keeps it comparable across settings before a normal-CDF squash bounds it to plus or minus fifty, the transform from the old article ["Taming Indicator Tails with Sigmoid Transforms"](https://aligrithm.com/taming-indicator-tails-with-sigmoid-transforms/). - The two-window averaging does heavy smoothing before any transform sees the data, which matters because volume is the most spike-prone series you handle, the warning from the old article ["Why Median Filters Are Useful for Volume and Outliers"](https://aligrithm.com/why-median-filters-are-useful-for-volume-and-outliers/). - Use it as a context filter: high activity is when breakouts and trend continuations have fuel, dead tape is when momentum fails and fills get expensive, so it decides which of your other indicators to trust now. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Statistically Sound Indicators for Financial Market Prediction - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Statistically+Sound+Indicators+for+Financial+Market+Prediction+Timothy+Masters&ref=aligrithm.com) - [Cycle Analytics for Traders - John Ehlers (Amazon)](https://www.amazon.com/s?k=Cycle+Analytics+for+Traders+John+Ehlers&ref=aligrithm.com) ### 2.50 Band-Pass Q and Selectivity URL: https://aligrithm.com/band-pass-q-and-selectivity/ Last updated: 2026-07-27T13:23:16.000Z The old article ["Band-Pass Filters: The Most Underused Tool in Technical Analysis"](https://aligrithm.com/band-pass-filters-the-most-underused-tool-in-technical-analysis/) built the band-pass and named its two dials: a center period and a bandwidth. This article is about the second dial, because it is the one traders set badly. Bandwidth has a standard summary number, the quality factor Q, and Q is the single knob that controls the tradeoff every cycle tool lives or dies on: how cleanly the filter isolates one cycle versus how fast and stable it responds. Set Q without understanding it and you either pass mud or chase a ringing ghost. Understand it and you can pick the right narrowness for the job. ## Selectivity, and the number that measures it Selectivity is the degree to which a filter rejects the frequencies you did not want. A highly selective band-pass passes a narrow slice of cycles around its center and crushes everything else; a poorly selective one passes a wide smear. The quality factor Q puts a number on selectivity, defined as the center frequency divided by the bandwidth measured at the half-power points. $$ Q = \\frac{\\text{center frequency}}{\\text{bandwidth at half-power}} = \\frac{1}{\\text{fractional bandwidth}} $$ The bandwidth at the half-power points is the width of the band between the two frequencies where the filter's power has fallen to half, the 0.707-amplitude points from the critical-period article. Express that width as a fraction of the center frequency and Q is its reciprocal. So a band-pass set to a 30 percent bandwidth has a Q of about one over 0.3, roughly 3.33\. A narrow 10 percent bandwidth gives Q of 10; a wide 50 percent gives Q of 2\. Higher Q means narrower, more selective, more sharply tuned. There is a second, equivalent definition that exposes what Q physically is, borrowed from resonant systems. $$ Q = 2\\pi \\, \\frac{\\text{energy stored}}{\\text{energy dissipated per cycle}} $$ A high-Q filter stores a lot of energy relative to what it loses each cycle, which is precisely why it rings: energy you put in keeps sloshing around for many cycles before it dies. That is the same fact as sharp selectivity, seen from the time domain instead of the frequency domain. ## Q is the selectivity-versus-responsiveness dial Now the tradeoff, which is the reason Q matters. The two definitions are two views of one thing, and they pull in opposite directions for a trader. High Q buys you a narrow, sharply selective band: the filter isolates one cycle period and rejects neighbors hard, so the output is a clean single cycle. But the same narrowness means the filter stores energy for many cycles, so it responds slowly to changes, rings after a shock, and carries more lag. Tune Q high and you get a beautiful clean cycle that arrives late and keeps oscillating after the real cycle has died. Low Q buys you the opposite: a wide, gently selective band that responds fast, rings little, and lags less, at the cost of passing a broader smear of cycles, so the output is not a clean single tone. Tune Q low and you get a fast, honest, but blurry read. This maps directly onto the Q ranges the old article ["Band-Pass Filters: The Most Underused Tool in Technical Analysis"](https://aligrithm.com/band-pass-filters-the-most-underused-tool-in-technical-analysis/) laid out. Low Q, under about 3, is broad and fast with little ringing, the right choice when you need a quick, stable read more than a pure tone. Medium Q, roughly 3 to 8, is the standard cycle-isolation setting, narrow enough to pull out one cycle without ringing badly. High Q, above 8, is narrow and pure but slow and prone to ringing, useful only when you genuinely need a single isolated tone and can tolerate the lag. There is no universally correct Q, only a Q matched to whether your job rewards selectivity or responsiveness, and the two are a strict trade. Pick Q from the job, then check the lag and ringing it costs before you rely on the output, because a high-Q filter's clean appearance is exactly what tempts traders into paying lag they did not budget for. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_240-visual-0.png) ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## KEY POINTS - Selectivity is how hard a band-pass rejects the cycles you did not want. The quality factor Q measures it: center frequency divided by the half-power bandwidth, which equals one over the fractional bandwidth. - A 30 percent bandwidth gives Q of about 3.33; 10 percent gives Q of 10; 50 percent gives Q of 2\. Higher Q means narrower and more sharply tuned. - The energy definition, Q equals two pi times energy stored over energy dissipated per cycle, shows what Q physically is: a high-Q filter hoards energy, which is why it rings. - Q is one dial with two opposing effects. High Q isolates a pure cycle but responds slowly, rings, and lags more. Low Q responds fast with little ringing but passes a blurry smear of cycles. - The ranges from the old article ["Band-Pass Filters: The Most Underused Tool in Technical Analysis"](https://aligrithm.com/band-pass-filters-the-most-underused-tool-in-technical-analysis/): low Q (under 3) for fast stable reads, medium Q (3 to 8) for standard cycle isolation, high Q (over 8) for a pure tone you can wait for. - There is no universally right Q, only one matched to whether the job rewards selectivity or responsiveness. Pick from the job, then check the lag and ringing before trusting the clean-looking output. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Statistically Sound Indicators for Financial Market Prediction - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Statistically+Sound+Indicators+for+Financial+Market+Prediction+Timothy+Masters&ref=aligrithm.com) - [Cycle Analytics for Traders - John Ehlers (Amazon)](https://www.amazon.com/s?k=Cycle+Analytics+for+Traders+John+Ehlers&ref=aligrithm.com) ### 2.72 The Volume-Weighted MA Ratio URL: https://aligrithm.com/the-volume-weighted-ma-ratio/ Last updated: 2026-07-27T13:23:30.000Z Intraday Intensity and On-Balance Volume both ask the same question, whether volume lined up with price changes. This indicator asks a different one: did volume line up with price levels. The distinction matters. Knowing that volume clustered on the up bars tells you about momentum; knowing that volume clustered on the high-priced bars tells you where the crowd was willing to transact, which is a statement about value, not direction. The tool is a ratio of two averages, a volume-weighted one against a plain one, and the ratio reads above one when the expensive bars carried the volume and below one when the cheap bars did. The raw ratio is badly behaved. This article builds it and then applies the three transforms that turn it into a feature you can model. ## Two averages, one weighted by volume A plain simple moving average treats every bar in the window as equally important. Sum the closes, divide by the count. $$ \\text{SMA} = \\frac{1}{LB}\\sum\_{k=0}^{LB-1} C\_k $$ A weighted average lets some bars count more than others. Multiply each close by a positive weight, sum those products, and divide by the sum of the weights; when the weights are all equal you get the plain average back. Use volume as the weight and you get the volume-weighted moving average, an average price where heavily traded bars pull the result toward their level. $$ \\text{VWMA} = \\frac{\\sum\_{k=0}^{LB-1} V\_k \\cdot C\_k}{\\sum\_{k=0}^{LB-1} V\_k} $$ The VWMA answers "what was the average price, counting each bar by how much traded there." The SMA answers "what was the average price, counting each bar once." Their ratio is the whole indicator. Divide the volume-weighted average by the plain average and you measure how price and volume associated over the window. $$ R = \\frac{\\text{VWMA}}{\\text{SMA}} $$ When the higher-priced bars in the window also carried the higher volume, the VWMA gets dragged above the SMA and the ratio reads above one. When the heavy volume landed on the cheaper bars, the VWMA sinks below the SMA and the ratio reads below one. A ratio near one says volume and price level had no particular relationship. This is genuinely different information from a momentum-style volume indicator, and it pairs well with Intraday Intensity rather than duplicating it. ## Three transforms to make the ratio usable The raw ratio R is no good as a model input, for the usual reasons a ratio bunched around one is no good: it is centered on an awkward value, it is asymmetric because a ratio can run to large multiples on the upside but is floored near zero on the downside, and its spread depends on the lookback. The old article ["Why Most Indicators Should Be Transformed Before Modeling"](https://aligrithm.com/why-most-indicators-should-be-transformed-before-modeling/) listed exactly these defects and the transforms that fix them. Apply three in sequence. First, take the log of the ratio. A log recenters the awkward multiplicative center of one onto an additive center of zero, since the log of one is zero, and it symmetrizes the asymmetry, mapping a doubling and a halving to equal and opposite distances from zero. The log also pulls in the heavy upper tail that a raw ratio carries. Second, divide the log ratio by the square root of the lookback. The spread of the log ratio shrinks as the window grows, the same inverse relationship seen across these volume indicators, so the square-root-of-lookback divisor makes the standard deviation roughly independent of the window and lets a single downstream compression setting work across lookbacks. The old article ["How to Build Stationary Indicators from Non-Stationary Prices"](https://aligrithm.com/how-to-build-stationary-indicators-from-non-stationary-prices/) is the principle behind both moves: log to fix the mean and center, scale to fix the variance. Third, apply a nonlinear compression to tame whatever outliers survive. Putting the pieces together gives the volume-weighted ratio indicator. $$ \\text{VWR} = 100 \\cdot \\frac{1000 \\cdot \\log\\!\\left(\\text{VWMA}/\\text{SMA}\\right)}{\\sqrt{2 \\cdot LB}} $$ The constants are cosmetic, not structural. The log of VWMA over SMA does the real work, centering and symmetrizing; the square root of twice the lookback in the denominator does the variance normalization; the thousand and the hundred are scale factors that lift a tiny log ratio into human-readable numbers. After this you would still pass the output through a bounded compression for the final outliers, the normal-CDF or sigmoid squash from the old article ["Taming Indicator Tails with Sigmoid Transforms"](https://aligrithm.com/taming-indicator-tails-with-sigmoid-transforms/), but the heavy lifting is done by the log and the lookback scaling. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## What it adds to a volume toolkit The reason to carry this alongside money-flow indicators is that it answers a question they cannot. Intraday Intensity and Normalized OBV both tie volume to the direction of the move. The volume-weighted MA ratio ties volume to the price level, so it can flag situations the others miss: a market grinding sideways where the crowd is quietly transacting at the upper end of the range reads above one here while a money-flow indicator sees no clean directional bias to report. Read above one as the volume concentrating at higher prices, a sign the crowd accepted the elevated levels, and below one as volume concentrating at lower prices, a sign the heavy transacting happened on the discount bars. As with every indicator in this group, the clean construction makes the relationship testable and comparable across instruments; whether the relationship predicts anything is a question for the data, not the formula. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_265-visual-0.png) ## KEY POINTS - The volume-weighted MA ratio measures whether volume lined up with price level, a different question from money-flow indicators that measure whether volume lined up with price direction. - The VWMA is an average close weighted by each bar's volume, so heavily traded bars pull it toward their price; the SMA weights every bar once, and the ratio VWMA over SMA reads above one when the higher-priced bars carried the volume. - The raw ratio is unusable: centered on the awkward value of one, asymmetric because ratios run large up and floored down, and lookback-dependent in spread, the defects catalogued in the old article ["Why Most Indicators Should Be Transformed Before Modeling"](https://aligrithm.com/why-most-indicators-should-be-transformed-before-modeling/). - Take the log to recenter on zero and symmetrize, divide by the square root of the lookback to make the spread window-independent, then compress, the two-step mean-then-variance fix from the old article ["How to Build Stationary Indicators from Non-Stationary Prices"](https://aligrithm.com/how-to-build-stationary-indicators-from-non-stationary-prices/). - The constants in the final formula are cosmetic scaling; the log and the square-root-of-lookback divisor do the real normalization, and a final bounded squash from the old article ["Taming Indicator Tails with Sigmoid Transforms"](https://aligrithm.com/taming-indicator-tails-with-sigmoid-transforms/) handles residual outliers. - It complements rather than duplicates Intraday Intensity and Normalized OBV, flagging crowds transacting at the high or low end of a range even when there is no clean directional flow to report. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Statistically Sound Indicators for Financial Market Prediction - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Statistically+Sound+Indicators+for+Financial+Market+Prediction+Timothy+Masters&ref=aligrithm.com) - [Cycle Analytics for Traders - John Ehlers (Amazon)](https://www.amazon.com/s?k=Cycle+Analytics+for+Traders+John+Ehlers&ref=aligrithm.com) ### 2.49 The Decycler Oscillator: Spotting Trend Transitions URL: https://aligrithm.com/the-decycler-oscillator-spotting-trend-transitions/ Last updated: 2026-07-27T13:23:15.000Z The old article ["Decyclers: Extracting Trend by Removing Cycle Energy"](https://aligrithm.com/decyclers-extracting-trend-by-removing-cycle-energy/) built the decycler: instead of smoothing price until the cycle fades, subtract the cycle out directly, leaving a clean trend line at low lag. This article uses the same subtraction trick twice over to build something different, a regime detector. Take two high-pass filters at different cutoffs, subtract one from the other, and you get an oscillator that crosses zero when the market flips between trending up and trending down. It is a useful classifier and a dangerous signal, and the difference between those two uses is the whole point of the article. ## A reminder: the decycler subtracts the cycle away The decycler comes from subtracting a high-pass filter's output from the input. The high-pass keeps the fast cycle and noise; remove it from the price and what survives is the slow trend. $$ H\_{\\text{decycler}}(z) = 1 - H\_{\\text{HPF}}(z) = \\frac{(\\alpha/2)(1 + z^{-1})}{1 - (1-\\alpha)z^{-1}} $$ Algebraically that is a one-pole low-pass, so the decycler is not a new filter, it is a low-pass wearing better clothes. The clothes matter, as the old article ["Decyclers: Extracting Trend by Removing Cycle Energy"](https://aligrithm.com/decyclers-extracting-trend-by-removing-cycle-energy/) argued: the dial is an interpretable cycle-cutoff period, remove everything shorter than 30 bars, instead of an opaque smoothing constant. The decycler gives you a trend line whose meaning you can state in plain language. ## The oscillator: difference two high-pass filters The decycler oscillator is built by subtracting the output of a high-pass filter with a short cutoff period from the output of a high-pass filter with a longer cutoff period. $$ \\text{decycler oscillator} = H\_{\\text{HPF, short cutoff}} - H\_{\\text{HPF, long cutoff}} $$ Think about what each high-pass keeps. The short-cutoff high-pass keeps everything faster than a short period, a wide band of cycles. The long-cutoff high-pass keeps only everything faster than a longer period, a narrower band. Subtract the narrow band from the wide band and what remains is the slice in between, the band of cycles between the two cutoffs. So differencing two high-pass filters is a band-pass, and the decycler oscillator passes a chosen middle band of trend-transition speeds while rejecting both the very slow drift and the very fast noise. The output oscillates around zero, and its zero crossings are the event you care about. When the oscillator crosses from below zero to above, the medium-term motion has tipped from down to up; the reverse crossing marks the flip from up to down. It is built to mark the transition between uptrends and downtrends, which a single moving average reports late and a single decycler line does not flag at all. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Classifier, not signal: where this earns its keep and where it kills you Here is the line that decides whether the decycler oscillator helps or hurts. It is a regime classifier, not a tradeable entry signal, and treating it as the latter is how you lose money with a tool that works. As a classifier it answers a coarse, honest question: is the medium-term structure currently leaning up or down? That label is useful upstream, gating a trend-following system on so it stands aside when the oscillator says the regime is against it, or switching a strategy between trend-mode and cycle-mode logic. Used that way, its lag and its bluntness are tolerable, because you are reading a slow-changing state, not timing a fill. As a signal it fails the way every cycle tool fails. The decycler oscillator is a band-pass, so it inherits the band-pass's habits: it lags the actual turn, it rings, and it manufactures false crossings when the market is trending cleanly with no cycle in the band it watches, the exact failure the old article ["Decyclers: Extracting Trend by Removing Cycle Energy"](https://aligrithm.com/decyclers-extracting-trend-by-removing-cycle-energy/) flagged when it said the oscillator wins only when a clear cycle band exists. Trade its zero crossings directly and you buy late, sell late, and whipsaw through every trend that does not happen to oscillate at its tuned band. And like everything in this pillar it is backward-looking: a zero crossing reports that the regime already changed, it does not predict the next one. Use it to label the regime and gate other logic, never as the trigger that pulls the trigger. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_239-visual-0.png) ## KEY POINTS - The decycler subtracts a high-pass filter's output from the input, leaving the trend; algebraically it is a one-pole low-pass with an interpretable cycle-cutoff dial, from the old article ["Decyclers: Extracting Trend by Removing Cycle Energy."](https://aligrithm.com/decyclers-extracting-trend-by-removing-cycle-energy/) - The decycler oscillator subtracts a long-cutoff high-pass from a short-cutoff high-pass. Differencing two high-pass filters keeps the band in between, so the oscillator is a band-pass on the trend-transition band. - Its zero crossings mark regime flips: below-to-above signals the medium-term motion tipping from down to up, and the reverse marks up to down, a transition a single moving average reports late. - It is a regime classifier, not a tradeable signal. As a classifier it gives a coarse up-or-down state useful for gating trend-following or switching trend-mode and cycle-mode logic. - As a signal it fails like any band-pass: it lags the turn, rings, and whipsaws when the market trends cleanly with no cycle in its band, so it wins only when a clear cycle band exists. - It is backward-looking: a zero crossing reports the regime already changed. Use it to label the regime and gate other logic, never as the entry trigger. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Statistically Sound Indicators for Financial Market Prediction - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Statistically+Sound+Indicators+for+Financial+Market+Prediction+Timothy+Masters&ref=aligrithm.com) - [Cycle Analytics for Traders - John Ehlers (Amazon)](https://www.amazon.com/s?k=Cycle+Analytics+for+Traders+John+Ehlers&ref=aligrithm.com) ### 2.71 Normalized On-Balance Volume URL: https://aligrithm.com/normalized-on-balance-volume/ Last updated: 2026-07-27T13:23:29.000Z On-Balance Volume has survived since the early 1960s on a single idea that sounds reasonable: when price rises, add the day's volume to a running total, and when it falls, subtract it. The running total is supposed to reveal whether volume is quietly accumulating behind up moves or down moves before price confirms it. Joseph Granville built a following on this, and the indicator still ships in every charting package. As a number you feed a model, it is broken. The running sum wanders like a random walk, its level depends on how many years ago you started counting, and its variance balloons as market volume grows. This article keeps Granville's signed-volume insight and throws away the running sum, rebuilding OBV into a bounded oscillator that means the same thing on any instrument in any decade. ## The original, and why the running sum poisons it The construction is one line. Look at the sign of the close-to-close change, then add or subtract the full bar volume from a cumulative total. $$ \\text{OBV}\_t = \\text{OBV}\_{t-1} + \\operatorname{sign}(C\_t - C\_{t-1}) \\cdot V\_t $$ The signed-volume idea is sound. Volume on up bars and volume on down bars are different evidence, and tracking which side the volume lands on is a real question. The cumulative wrapper destroys it. A running sum of signed volume has no fixed level, so its current value depends entirely on the arbitrary date you began accumulating; start in 2010 and start in 2015 and you get two completely different OBV levels on the same bar today. Worse, because volume grows over the years, both the local mean and the local variance of the sum drift upward without bound, and the path itself wanders like a near-random walk with no tendency to return to any reference. The old article ["The Case Against Raw Price Indicators"](https://aligrithm.com/the-case-against-raw-price-indicators/) condemned raw price for being non-stationary in mean and variance and incomparable across instruments; cumulative OBV fails all three the same way, and a momentary OBV reading carries no usable information because there is nothing to compare it against. ## Cage it in a window, then divide out the volume Two changes fix it. First, stop accumulating forever and compute the signed volume only over a rolling lookback window. A windowed sum cannot wander off to infinity; old bars fall out the back as new ones enter, so the quantity behaves like an oscillator instead of a random walk. Second, kill the variance non-stationarity by dividing the signed volume in the window by the total volume in the same window. The growing volume scale sits in both numerator and denominator, so it cancels, and the result is a ratio that lives on a fixed axis. $$ \\text{Ratio} = \\frac{\\sum\_{k=0}^{LB-1} \\operatorname{sign}(C\_k - C\_{k+1}) \\cdot V\_k}{\\sum\_{k=0}^{LB-1} V\_k} $$ Read the bounds and the meaning falls out. If every bar in the window closed up, every term in the numerator carries a positive sign and the numerator equals the denominator, so the ratio pins at plus one. If every bar closed down the ratio pins at minus one. Anything in between measures how cleanly volume aligns with direction: near plus one means the up bars carried the volume, near minus one the down bars did, near zero means volume split evenly across up and down bars with no net flow. This is the same cancellation that built Chaikin Money Flow in the old article ["Intraday Intensity and Chaikin Money Flow, Made Stationary"](https://aligrithm.com/intraday-intensity-and-chaikin-money-flow-made-stationary/), applied to signed volume instead of close-position-weighted volume. The old article ["How to Build Stationary Indicators from Non-Stationary Prices"](https://aligrithm.com/how-to-build-stationary-indicators-from-non-stationary-prices/) is the general principle: divide two quantities that grow together and the growth disappears. ## Make it independent of the lookback, then tame the tails The bounded ratio is already a decent indicator, and it has one more flaw worth removing. Its variance depends on the window length. A short lookback samples few bars, so the ratio swings hard toward the extremes; a long lookback averages more bars, so it hugs zero. That means a reading of 0.4 from a 10-bar window and a reading of 0.4 from a 100-bar window are not the same event, which breaks comparability across settings. The variance of the ratio runs inversely with the number of bars, so multiplying by the square root of the lookback rescales every window to a common spread. The old article ["Why ATR Normalization Is More Than a Volatility Trick"](https://aligrithm.com/why-atr-normalization-is-more-than-a-volatility-trick/) used the same square-root-of-lookback divisor to cancel horizon dependence in price momentum; here it cancels horizon dependence in volume flow. After the square-root rescaling, pass the result through a normal-CDF compression to pull rare outliers into a bounded range and rescale to a clean plus or minus fifty. $$ \\text{NormOBV} = 100 \\cdot \\Phi\\!\\left(\\sigma\_0 \\sqrt{LB} \\cdot \\text{Ratio}\\right) - 50 $$ The term inside is the lookback-normalized ratio, scaled by a constant so the bulk of the distribution lands in the linear region of the CDF rather than saturating against its knees. Phi is the standard normal CDF, the same bounded squashing transform the old article ["Taming Indicator Tails with Sigmoid Transforms"](https://aligrithm.com/taming-indicator-tails-with-sigmoid-transforms/) recommended, chosen here because it maps the whole real line into a fixed interval while leaving the body of the distribution roughly linear. The minus fifty recenters the output so it swings symmetrically around zero. The result is a volume-flow oscillator on a fixed minus-fifty to plus-fifty axis whose meaning does not shift with the lookback, the instrument, or the year. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## The change matters more than the level One operational note that applies more to the original than the rebuild. With cumulative OBV, traders watched the slope, the bar-to-bar change, far more than the absolute level, because the level was meaningless and only the direction of accumulation carried any signal. That habit was a workaround for a broken indicator. With the normalized version the level itself is meaningful, since it is a bounded, stationary reading of how volume aligns with direction over the window, so you can use the value directly. The delta still carries information, a sharp move in the normalized OBV flags a regime shift in who is carrying the volume, but you are no longer forced to read the change because the level finally means something on its own. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_264-visual-0.png) ## KEY POINTS - Original OBV adds the bar's volume to a running total on up closes and subtracts it on down closes; the signed-volume idea is sound but the cumulative wrapper makes the level depend on an arbitrary start date and wander like a random walk. - Cumulative OBV is non-stationary in mean and variance and incomparable across instruments, the exact failures the old article ["The Case Against Raw Price Indicators"](https://aligrithm.com/the-case-against-raw-price-indicators/) warned about, so a momentary reading carries no usable information. - Replacing the endless sum with a rolling-window sum stops the wandering and turns the indicator into an oscillator; dividing the windowed signed volume by the windowed total volume cancels the growing volume scale, the trick from the old article ["How to Build Stationary Indicators from Non-Stationary Prices"](https://aligrithm.com/how-to-build-stationary-indicators-from-non-stationary-prices/). - The resulting ratio is bounded in minus one to plus one: plus one when every bar in the window closed up, minus one when every bar closed down, zero when volume splits evenly across direction. - The ratio's variance runs inversely with the lookback, so multiplying by the square root of the lookback makes it independent of the window, the same horizon-cancelling move as the old article ["Why ATR Normalization Is More Than a Volatility Trick"](https://aligrithm.com/why-atr-normalization-is-more-than-a-volatility-trick/). - A normal-CDF compression tames outliers and rescales to plus or minus fifty, following the old article ["Taming Indicator Tails with Sigmoid Transforms"](https://aligrithm.com/taming-indicator-tails-with-sigmoid-transforms/); the normalized level is meaningful on its own, so you no longer have to read only the slope. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Statistically Sound Indicators for Financial Market Prediction - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Statistically+Sound+Indicators+for+Financial+Market+Prediction+Timothy+Masters&ref=aligrithm.com) - [Cycle Analytics for Traders - John Ehlers (Amazon)](https://www.amazon.com/s?k=Cycle+Analytics+for+Traders+John+Ehlers&ref=aligrithm.com) ### 2.48 The Weighted Moving Average Was a Mistake URL: https://aligrithm.com/the-weighted-moving-average-was-a-mistake/ Last updated: 2026-07-27T13:23:15.000Z The old article ["Why the SMA Is Often a Terrible Smoother"](https://aligrithm.com/why-the-sma-is-often-a-terrible-smoother/) showed the simple moving average leaks noise through its sidelobes, and the old article ["EMA vs SMA: Why Simplicity Still Matters"](https://aligrithm.com/ema-vs-sma-why-simplicity-still-matters/) defended the EMA for its recursive efficiency. The weighted moving average sits between them in popularity and below both in merit. It weights recent bars more and older bars less, in a straight linear ramp, on the theory that newer data deserves more say. That theory came from someone reasoning about trading, not about filters, and the filter view shows the WMA loses on the axes that matter. It is a window function chosen by intuition, and intuition picked a bad window. ## What the WMA is, structurally The WMA is a non-recursive filter, a finite impulse response, like the SMA. The difference is the shape of its weights. The SMA weights every bar in the window equally; the WMA ramps the weights down linearly from the most recent bar to the oldest, so the newest close gets the largest coefficient and the weight decays to near zero at the far end of the window. The coefficients are asymmetric about the center of the window, heavy on the recent side and light on the old side. That asymmetry is the whole story, because two of a filter's most important properties depend on whether its coefficients are symmetric, and the WMA throws symmetry away. ## Asymmetry breaks linear phase The old article ["Recursive vs Non-Recursive: The Two Families of Every Indicator"](https://aligrithm.com/recursive-vs-non-recursive-the-two-families-of-every-indicator/) stated the rule: a non-recursive filter whose coefficients are symmetric about the center has a lag equal to its degree over two, and that lag is the same at every frequency. Constant lag across frequency is linear phase, and it is precious, because it means the filter delays every cycle by the same number of bars and does not distort the shape of a waveform passing through. The SMA has it. The WMA does not. Because the WMA's coefficients are asymmetric, its phase response is nonlinear: different cycles get delayed by different amounts. A slow swing and a fast wiggle come out shifted by unequal lags, so the filter does not just delay the waveform, it warps its shape. Peaks and troughs land in the wrong relative positions. This is the same defect that makes recursive filters tricky, except the WMA pays it without buying the recursive family's efficiency, since it still needs all N taps. You get phase distortion and you still store the whole window. ## It loses the comparison that matters The WMA's marketing rests on lag. At equal length it does react faster than the SMA, its weight sits closer to the present, so the naive comparison favors it. But equal length is the wrong comparison. The honest comparison holds noise rejection constant and asks which filter lags less while attenuating the same amount of high-frequency junk, and there the WMA falls apart. Its stopband attenuation is poor, worse than the SMA's, so to match the SMA's actual noise rejection you must lengthen the WMA, and the lengthened WMA lags more than the SMA you were comparing against. The faster response was bought by doing less filtering, which is not a free lunch, it is no lunch. Set the WMA against a properly designed window and the verdict gets blunter. A tapered window like the Hann or Blackman, covered in the old article ["Why the SMA Is Often a Terrible Smoother,"](https://aligrithm.com/why-the-sma-is-often-a-terrible-smoother/) gives an order of magnitude better stopband attenuation at the same lag as the SMA, and it keeps symmetric coefficients, so it preserves linear phase. The WMA delivers worse attenuation than the SMA and worse phase behavior than a tapered window, landing behind both on the two axes that decide a smoother's quality. The linear ramp was a guess at what good weighting looks like, and it guessed wrong: real window design tapers smoothly toward zero at both ends, symmetrically, which is precisely what the WMA's one-sided ramp refuses to do. The practical conclusion is unsentimental. If you want lower lag than the SMA, use an EMA for its recursive cheapness or a tapered window for its clean stopband, both of which beat the WMA. The WMA survives on familiarity and a plausible-sounding story, not on performance, and there is no filtering job where it is the right answer. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_238-visual-0.png) ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## KEY POINTS - The WMA is a non-recursive (FIR) filter like the SMA, but it ramps weights down linearly from the newest bar to the oldest, so its coefficients are asymmetric about the center. - Asymmetric coefficients break linear phase. By the rule from the old article ["Recursive vs Non-Recursive: The Two Families of Every Indicator,"](https://aligrithm.com/recursive-vs-non-recursive-the-two-families-of-every-indicator/) only symmetric FIR filters have constant lag across frequency. - Nonlinear phase means different cycles are delayed by different amounts, so the WMA warps waveform shape, not just delays it, putting peaks and troughs in the wrong relative places. - At equal length the WMA reacts faster than the SMA, but that is the wrong comparison. Its stopband attenuation is worse, so to match the SMA's noise rejection you must lengthen it, and then it lags more. - Against a tapered window (Hann, Blackman) from the old article ["Why the SMA Is Often a Terrible Smoother,"](https://aligrithm.com/why-the-sma-is-often-a-terrible-smoother/) the WMA loses twice: worse attenuation than the SMA and worse phase than the symmetric window. - For lower lag than the SMA, use an EMA (recursive, cheap) or a tapered window (clean stopband, linear phase). The WMA is the right answer to no filtering job. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Statistically Sound Indicators for Financial Market Prediction - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Statistically+Sound+Indicators+for+Financial+Market+Prediction+Timothy+Masters&ref=aligrithm.com) - [Cycle Analytics for Traders - John Ehlers (Amazon)](https://www.amazon.com/s?k=Cycle+Analytics+for+Traders+John+Ehlers&ref=aligrithm.com) ### 2.70 Intraday Intensity and Chaikin Money Flow, Made Stationary URL: https://aligrithm.com/intraday-intensity-and-chaikin-money-flow-made-stationary/ Last updated: 2026-07-27T13:23:29.000Z Volume indicators carry a promise that the price-only crowd never gets: the chance to see whether institutions, not retail, drove the bar. The folklore says big money executes in size and big money tends to be right, so a price move on heavy volume means something a price move on thin volume does not. Intraday Intensity tries to capture exactly that, pairing where the bar closed inside its range with how much volume traded. The raw version is unusable. It jumps with every volume spike, drifts upward for years as market volume grows, and feeds your model a number whose meaning changes from decade to decade. This article takes the raw idea apart, shows why the old article ["The Case Against Raw Price Indicators"](https://aligrithm.com/the-case-against-raw-price-indicators/) applies word for word to raw volume too, and rebuilds it into Chaikin Money Flow, a quantity you can drop into a model with no further work. ## Where did the bar close inside its range Start with the price half, no volume yet. Each bar has a high, a low, and a close, and the one fact worth extracting is whether the close landed near the top or near the bottom of the range. Compare the distance from close to low against the distance from high to close. Their difference collapses to a clean expression. $$ (C - L) - (H - C) = 2C - H - L $$ When the close sits at the high the term equals the full range and reads positive; when the close sits at the low it equals the negative range. So 2C minus H minus L is a raw conviction reading: large and positive means buyers closed the bar on their highs, large and negative means sellers slammed it shut on the lows. The problem is scale. That number is in price units, so a 2-dollar close-off-the-low on a 200-dollar stock and a 2-tick close-off-the-low on a currency are not comparable, and the same instrument changes its dollar range as its price level and volatility drift. Divide by the bar range to strip the units and you get a dimensionless multiplier bounded between minus one and plus one. $$ m = \\frac{2C - H - L}{H - L} $$ This multiplier is the heart of every money-flow indicator. It says nothing about magnitude in dollars, only about position inside the range: plus one is a close exactly at the high, minus one a close exactly at the low, zero a close in the dead center. The old article ["Why Most Indicators Should Be Transformed Before Modeling"](https://aligrithm.com/why-most-indicators-should-be-transformed-before-modeling/) makes the case for exactly this kind of move, trading a scale-bound raw number for a bounded ratio that means the same thing on every instrument. ## Adding volume, and breaking the indicator in the process Now weight that conviction by participation. A close on the high means more when a million shares traded than when a thousand did, so multiply the multiplier by volume. Masters scales by the square root of volume rather than volume itself, because raw volume's tail is so heavy that the indicator would be dominated by a handful of monster bars; the square root pulls those spikes back toward the body without erasing them. Times one hundred for human-readable numbers gives raw Intraday Intensity. $$ \\text{II} = 100 \\cdot \\frac{2C - H - L}{H - L} \\cdot \\sqrt{V} $$ Read it straight and it is unusable, for three reasons that stack. First, bar-to-bar volume swings wildly, so the square-root-of-volume weight whipsaws the output even when the close position barely moves. Second, the close lands near the high on one bar and near the low on the next, flipping the multiplier's sign, so the product fluctuates with no stable level to read. Third, and this is the deal killer, market volume grows year over year as the asset matures and more participants pile in, so the absolute size of II ratchets upward over time with nothing to do with price behavior. The old article ["The Case Against Raw Price Indicators"](https://aligrithm.com/the-case-against-raw-price-indicators/) laid out why a feature whose mean and variance march upward over the years poisons every model class downstream: trees send out-of-range bars to one leaf, linear models extrapolate a slope they never saw, and your in-sample fit collapses live. Raw II fails that test the same way raw price does. Volume is even worse behaved than price, which is why the old article ["Why Median Filters Are Useful for Volume and Outliers"](https://aligrithm.com/why-median-filters-are-useful-for-volume-and-outliers/) singled volume out as the worst-case input for any linear filter. ## Dividing the flow by the volume that carried it The cure is to stop reading II as an absolute number and start reading it as a fraction of the volume that produced it. Sum the raw money flow over a rolling window, sum the volume over the same window, and divide. The ever-growing volume scale cancels top and bottom, and the result is Chaikin Money Flow. $$ \\text{CMF} = 100 \\cdot \\frac{\\sum\_{i=0}^{K-1} \\text{II}(i)}{\\sum\_{i=0}^{K-1} V(i)} $$ Both the wild bar-to-bar jumps and the multi-decade drift die in one operation. The numerator carries the conviction-weighted volume, the denominator carries the total volume, and their ratio is a stationary oscillator that means the same thing in 1995 and 2025\. This is what the old article ["How to Build Stationary Indicators from Non-Stationary Prices"](https://aligrithm.com/how-to-build-stationary-indicators-from-non-stationary-prices/) calls dividing two quantities that share the same non-stationary driver: volume grows on top and bottom together, so the ratio holds still. CMF needs no further transform, no rolling z-score, no sigmoid; it lands ready for predictive modeling, which is rare and worth respecting. A variant smooths each piece on its own clock instead of summing over a single window. Run a short simple moving average on the money flow and divide by a long exponential moving average of volume. $$ \\text{II}\_{\\text{smoothed}} = 100 \\cdot \\frac{\\text{SMA}(\\text{II})}{\\text{EMA}(V)} $$ The only number that matters here is the length of the volume EMA. Volume is so erratic that you need several dozen bars of smoothing in the denominator before the ratio settles into something stationary; a short volume average leaves the spikes in and the indicator goes back to whipsawing. Pick the volume EMA long, longer than instinct suggests, and the money-flow numerator can stay short and responsive. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## What it tells you and what it does not CMF is a conviction gauge, not a direction oracle. A persistently positive reading says closes have been landing in the upper range on the heavier-volume bars, which the folklore reads as accumulation; persistently negative says distribution. Treat the folklore as a hypothesis, not a law. The claim that institutions know more is exactly that, a claim, and the old article on raw price indicators is a reminder that a clean construction does not make the underlying edge real, only testable. What the construction buys you is a feature that is stationary, bounded, and cross-instrument comparable, so when you test whether money flow predicts forward returns you are testing the signal and not an artifact of growing volume. That is the whole point of the rebuild: get the indicator honest first, then let the data say whether it works. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_263-visual-0.png) ## KEY POINTS - The money-flow multiplier (2C minus H minus L) over the range is a bounded, unit-free reading of where the bar closed inside its range, plus one at the high and minus one at the low, comparable across every instrument. - Intraday Intensity weights that multiplier by the square root of volume, scaling conviction by participation while the square root pulls the heavy volume tail back toward the body. - Raw II is unusable on three counts: bar-to-bar volume swings whipsaw it, the close position flips sign every bar, and market volume drifts upward for years so the absolute level is non-stationary, the deal killer flagged in the old article ["The Case Against Raw Price Indicators"](https://aligrithm.com/the-case-against-raw-price-indicators/). - Chaikin Money Flow divides summed II by summed volume over a rolling window, canceling the growing volume scale top and bottom, the exact trick from the old article ["How to Build Stationary Indicators from Non-Stationary Prices"](https://aligrithm.com/how-to-build-stationary-indicators-from-non-stationary-prices/), and lands stationary with no further transform. - The smoothed variant SMA of II over EMA of volume works only if the volume EMA runs several dozen bars, because volume is erratic enough to demand heavy denominator smoothing, the worst-case input from the old article ["Why Median Filters Are Useful for Volume and Outliers"](https://aligrithm.com/why-median-filters-are-useful-for-volume-and-outliers/). - CMF measures conviction, not direction; the construction makes the institutional-accumulation story testable, it does not make it true. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Statistically Sound Indicators for Financial Market Prediction - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Statistically+Sound+Indicators+for+Financial+Market+Prediction+Timothy+Masters&ref=aligrithm.com) - [Cycle Analytics for Traders - John Ehlers (Amazon)](https://www.amazon.com/s?k=Cycle+Analytics+for+Traders+John+Ehlers&ref=aligrithm.com) ### 2.47 Critical Period and the Half-Power Point: How to Pick Filter Length URL: https://aligrithm.com/critical-period-and-the-half-power-point-how-to-pick-filter-length/ Last updated: 2026-07-27T13:23:14.000Z The old article ["How to Think About Indicator Lag Before Backtesting"](https://aligrithm.com/how-to-think-about-indicator-lag-before-backtesting/) demanded you compute an indicator's lag before you trust it, and the old article ["The Hidden Cost of Every Moving Average: Lag"](https://aligrithm.com/the-hidden-cost-of-every-moving-average-lag/) priced that lag in money and missed entries. Both leave a question dangling: when you set a filter's length, what cycle are you actually choosing to keep? Most traders answer wrong. They think an SMA of length 50 filters out the 50-bar cycle. It does not. The honest answer comes from one number, the critical period, defined at the half-power point, and once you can compute it you set filter length on purpose instead of by superstition. ## Half-power means amplitude 0.707, not 0.5 The critical period is the cycle length at which the filter's output power is half the input power at that frequency. The word power is doing real work here. Power is proportional to the square of amplitude, so cutting power in half does not mean cutting amplitude in half. $$ \\frac{\\text{output power}}{\\text{input power}} = \\frac{1}{2} \\;\\Longrightarrow\\; \\frac{\\text{output amplitude}}{\\text{input amplitude}} = \\frac{1}{\\sqrt{2}} \\approx 0.707 $$ So at the critical period the wave comes out at about 70.7 percent of the size it went in, not 50 percent. This is the same point engineers call the minus-three-decibel point, and it is the universal convention for where a filter's passband ends and its stopband begins. The dividing line is not where the filter kills a cycle; it is where the filter has taken the first real bite out of it, halving its power. Everything slower than the critical period passes mostly intact; everything faster gets progressively crushed. ## For an SMA, the critical period is twice the length Now the number that corrects the common error. For a simple moving average, the critical period is about twice the filter length. $$ T\_{\\text{critical}} \\approx 2N \\quad \\text{(for an SMA of length } N\\text{)} $$ An SMA of length 50 has its half-power point at a cycle of roughly 100 bars. It only takes the first real bite out of the 100-bar cycle, and it barely touches the 50-bar cycle a trader assumes it targets. Worse, the old article ["Why the SMA Is Often a Terrible Smoother"](https://aligrithm.com/why-the-sma-is-often-a-terrible-smoother/) showed the SMA's first response zero sits at the period equal to its length, so the SMA(50) annihilates the 50-bar cycle outright while leaving everything between 50 and 100 bars mostly alive. The "50-day average" is neither a clean filter at 50 days nor at any single period; its honest single-number summary is the half-power point near 100 bars. If you want the boundary between trend and cycle to sit at a 40-bar period, you do not use an SMA(40), you use an SMA(20), because the critical period is twice the length. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Picking length is picking lag, so pick both at once This is the practical procedure the lag articles were building toward. Decide the cycle period you want at the pass-stop boundary, then back out the length: for an SMA, length equals half the critical period. That single decision is honest in a way that "I use a 50-day average because everyone does" is not, because it ties the filter to a cycle you can name. But the same choice sets your lag, and you do not get to choose them independently. A longer critical period means more smoothing, which means more bars averaged, which means more lag, the structural delay from the old article ["The Hidden Cost of Every Moving Average: Lag."](https://aligrithm.com/the-hidden-cost-of-every-moving-average-lag/) So setting the critical period is setting the lag budget the old article ["How to Think About Indicator Lag Before Backtesting"](https://aligrithm.com/how-to-think-about-indicator-lag-before-backtesting/) told you to compute, in the same motion. Push the critical period out to keep slower cycles and you pay with more lag; pull it in for responsiveness and you let more cycle and noise through. There is no length that escapes the trade. The discipline: name the boundary cycle you want, convert it to a length, read off the lag that length forces, and check that lag against your holding period before you run anything. If the half-power point you need carries more lag than your strategy can absorb, the SMA is the wrong tool and you reach for a sharper filter, not a different length. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_237-visual-0.png) ## KEY POINTS - The critical period is the cycle length where the filter's output power is half the input power. Because power is amplitude squared, half power means amplitude 0.707, not 0.5, the same as the engineer's minus-three-decibel point. - The critical period is the honest single-number summary of what a filter targets: the boundary where it takes the first real bite, with slower cycles mostly passing and faster ones progressively crushed. - For an SMA the critical period is about twice the length. So SMA(50)'s half-power point is near a 100-bar cycle, and it does not target the 50-day cycle traders assume. - The SMA's first response zero sits at the period equal to its length, so SMA(50) annihilates the 50-bar cycle outright (the old article ["Why the SMA Is Often a Terrible Smoother"](https://aligrithm.com/why-the-sma-is-often-a-terrible-smoother/)) while cycles between 50 and 100 bars mostly survive. - To set length on purpose: name the boundary cycle you want, then use length equals half the critical period for an SMA, instead of copying a default. - Critical period and lag are one choice, not two: a longer critical period means more smoothing means more lag. Read off the lag the length forces and check it against your holding period before trading, per the old article ["How to Think About Indicator Lag Before Backtesting."](https://aligrithm.com/how-to-think-about-indicator-lag-before-backtesting/) --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Statistically Sound Indicators for Financial Market Prediction - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Statistically+Sound+Indicators+for+Financial+Market+Prediction+Timothy+Masters&ref=aligrithm.com) - [Cycle Analytics for Traders - John Ehlers (Amazon)](https://www.amazon.com/s?k=Cycle+Analytics+for+Traders+John+Ehlers&ref=aligrithm.com) ### 2.69 Reactivity: Momentum Times Aspect Ratio URL: https://aligrithm.com/reactivity-momentum-times-aspect-ratio/ Last updated: 2026-07-27T13:23:28.000Z A price move on heavy volume and the same move on thin volume are not the same event, and most momentum indicators treat them as identical. Reactivity, Al Gietzen's construction, refuses to. It multiplies a plain momentum reading by an aspect ratio that asks how much range the market produced per unit of volume, so a big move that took little volume to produce scores high, the market reacting hard to a light push. It is a sharp idea wrapped around an unstable core, and it demands the same disciplined normalization the old article ["CMMA: A Better Momentum Primitive Than Price-minus-MA Alone"](https://aligrithm.com/cmma-a-better-momentum-primitive-than-price-minus-ma-alone/) used to make a noisy primitive usable. ## Two pieces: momentum and aspect ratio The first piece is momentum in its crudest form, the price change from a lookback ago to now. $$ \\text{PM}\_t = P\_t - P\_{t-\\text{LB}} $$ The terms are the current price and the price LB bars ago, so this is raw directional movement over the window, nothing clever. The cleverness is the second piece, the aspect ratio, which Gietzen defines as price range over a period divided by total volume over that period. The intuition is a measure of how reactive the market is: a lot of range produced on little volume means price moves easily, the book is thin and the market is twitchy; a little range on heavy volume means price is sticky, absorbing trade without moving. In its raw form that ratio is far too unstable to use, because both range and volume swing wildly bar to bar and their quotient explodes. ## Normalize both halves before you divide The fix is to normalize the range and the volume separately against their own smoothed baselines before forming the ratio, so each half becomes a dimensionless "relative to recent normal" reading instead of a raw quantity in pips or contracts. $$ \\text{SR}\_t = \\alpha\\,(\\text{HH}\_t - \\text{LL}\_t) + (1-\\alpha)\\,\\text{SR}\_{t-1} \\qquad \\text{SV}\_t = \\alpha\\,V\_t + (1-\\alpha)\\,\\text{SV}\_{t-1} $$ $$ \\text{AR}\_t = \\frac{(\\text{HH}\_t - \\text{LL}\_t)\\,/\\,\\text{SR}\_t}{V\_t\\,/\\,\\text{SV}\_t} $$ The SR and SV terms are the exponentially smoothed price range (highest high minus lowest low) and smoothed volume, each an EMA with weight alpha. The aspect ratio is then current range over its own smoothed baseline, divided by current volume over its own smoothed baseline. Normalizing both halves this way is what tames the explosion: a range of "1.4 times its recent normal" against a volume of "0.7 times its recent normal" is a stable, dimensionless reading no matter the instrument or the era, where the raw range-over-volume was not. The smoothing length for these baselines runs a multiple of the momentum lookback, long enough that the baselines are steady references rather than chasing the same moves the numerators carry. ## Scale, compress, and accumulate Multiply momentum by the aspect ratio and you have raw reactivity, big when a strong move coincided with a high range-per-volume reading, the market lurching on light trade. That product still carries the price units of the momentum term and the volatility regime, so divide it by the smoothed price range to get scaled reactivity, which is dimensionless and normalized against market and volatility regimes, the same divide-by-a-range-baseline move that ATR normalization performs in the old ATR article. Then compress and accumulate into the final oscillator. $$ \\text{Scaled Reactivity}\_t = \\frac{\\text{PM}\_t \\cdot \\text{AR}\_t}{\\text{SR}\_t} \\qquad \\text{Reactivity}\_t = 100 \\cdot \\Phi\\!\\Big(\\textstyle\\sum 0.6 \\cdot \\text{Scaled Reactivity}\\Big) - 50 $$ The capital phi is the normal CDF doing the tail compression, the running sum with its decay weight accumulates the scaled reactivity into a smoother series, and the final scaling bounds and recenters it at zero. The whole chain echoes the CMMA build pattern the old article laid out: take a crude momentum primitive, divide by a volatility-scaled range to kill regime and units, then squash the tails into a bounded model-ready feature. Reactivity just inserts the aspect ratio in the middle, so the momentum is weighted by how easily the market was moving when it moved. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_262-visual-0.png) ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Handle with care Gietzen flagged reactivity as an indicator that needs caution and sophistication, and that warning is the honest part to keep. It is a product of two separately noisy quantities, momentum and an aspect ratio, and multiplying noisy things compounds their instability, which is the entire reason every half gets normalized against a smoothed baseline before they meet. Get the smoothing lengths wrong and the aspect ratio still misbehaves, throwing reactivity readings that track artifacts of the normalization rather than the market. It was built for trading price cycles, reading the moments a market moves easily on light volume as the reactive phases of a cycle, so it is a specialist tool, not a general trend or momentum replacement. It depends on volume, which means it inherits volume's problems, the year-over-year drift and the bar-to-bar erraticness that force the heavy smoothing in the first place, and on instruments with unreliable volume the aspect ratio is built on sand. Every smoothing length and the momentum lookback are parameters that interact, the normalization must be causal, and a reactivity reading should be confirmed against a cleaner indicator before it sizes anything. ## KEY POINTS - Reactivity multiplies plain momentum (price now minus price a lookback ago) by an aspect ratio, range over volume, so a big move produced on light volume scores high: the market reacting hard to a small push. - The raw aspect ratio is too unstable to use; normalize range and volume each against their own smoothed EMA baselines first, turning both into dimensionless "relative to recent normal" readings before dividing. - Multiply momentum by the normalized aspect ratio for raw reactivity, divide by the smoothed range to kill units and volatility regime (the ATR-normalization move from the old ATR article), then compress with the normal CDF and accumulate into a bounded oscillator. - The build mirrors the old CMMA pattern: crude momentum primitive, divide by a volatility-scaled range, squash the tails, with the aspect ratio weighting momentum by how easily the market was moving. - It is powerful but touchy: a product of two noisy parts, dependent on volume, built specifically for trading cycles. The smoothing lengths interact, the normalization must be causal, and it should confirm against a cleaner indicator before sizing a trade. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Statistically Sound Indicators for Financial Market Prediction - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Statistically+Sound+Indicators+for+Financial+Market+Prediction+Timothy+Masters&ref=aligrithm.com) - [Cycle Analytics for Traders - John Ehlers (Amazon)](https://www.amazon.com/s?k=Cycle+Analytics+for+Traders+John+Ehlers&ref=aligrithm.com) ### 2.46 The SMA Is a Least-Squares Straight-Line Fit URL: https://aligrithm.com/the-sma-is-a-least-squares-straight-line-fit/ Last updated: 2026-07-27T13:23:14.000Z The old article ["Why the SMA Is Often a Terrible Smoother"](https://aligrithm.com/why-the-sma-is-often-a-terrible-smoother/) tore into the simple moving average for its leaky sidelobes and its blind spots. This article comes at it from the other direction, and the result is more damning, not less. The SMA is not a crude hack. It is the provably optimal answer to a precise statistical question: what single number best represents this window of data if you assume the data is a flat line plus noise? That is the catch hiding inside the praise. The SMA is optimal for a model of the market that is almost never true, and pinning down which model it is optimal for tells you precisely when it lies. ## Set up the fit, and the average falls out Suppose you try to design a filter by fitting a straight line to the data in the window. A straight line is y equals m x plus b, and the best fit minimizes the sum of squared differences between the line and the data points. $$ F = \\sum \\big\[\\,y - (mx + b)\\,\\big\]^2 $$ To minimize, take the derivative of F with respect to the two unknowns, the slope m and the intercept b, and set each to zero. Solve, and the intercept b, the value of the fitted line at the center of the window, comes out as one specific quantity. $$ \\frac{\\partial F}{\\partial b} = 0 \\;\\Longrightarrow\\; b = \\frac{1}{N}\\sum y = \\text{the average of the data points} $$ The least-squares intercept is the plain average. So the simple moving average is not an approximation to a best fit; it is the best fit, in the least-squares sense, of a constant to the window. It is also the special case of a non-recursive filter with all coefficients equal to one over N, and its impulse response is finite, so it is a finite impulse response filter. The arithmetic that traders treat as the most basic thing in technical analysis is a solved optimization problem. ## Optimal, but for a model markets violate Here is where the endorsement turns into the indictment. The SMA is the optimal estimator of the level only if the data in the window is genuinely a constant corrupted by zero-mean noise. That is the model least squares assumed when it handed you the average. Ask whether markets look like a constant plus noise over any window long enough to matter, and the answer is no. Over a trend, the data is a sloped line, and the best constant fit to a sloped line sits at the center of the window, which means the SMA reports the trend as it was N over two bars ago. That is the lag from the old article ["The Hidden Cost of Every Moving Average: Lag,"](https://aligrithm.com/the-hidden-cost-of-every-moving-average-lag/) now derived from first principles: the SMA lags because it is fitting a flat line to data that is not flat, and the flat line's best guess for "now" is the middle of the past. Over a cycle it is worse than lag. The least-squares constant averages a rising half against a falling half, and when the window length matches the cycle the two halves cancel, which is why the SMA has a response zero at the period equal to its length. The SMA does not just smooth the cycle; it can erase it. Both failures, the lag in trends and the annihilation of matched cycles, are not defects in the SMA. They are correct behavior for the constant-plus-noise model, applied to data that is neither constant nor only noise. The tool is doing exactly what it was optimized to do, against a world that does not match its assumption. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## What to do with a tool that is optimal for the wrong question The lesson generalizes past the SMA and is the real payoff. Every smoother is the optimal estimator for some implicit model of the underlying. The SMA assumes a local constant. Fit a straight line and keep the slope and you get a linear-trend estimator that does not lag a trend, a least-squares moving average, at the cost of more noise. Fit a local polynomial and you get a Savitzky-Golay smoother that preserves curvature, peaks, and turns the SMA flattens. The question is never "is this smoother good," it is "does this smoother's hidden model match what my data does locally." When you choose the SMA you are betting the market is locally flat, and that bet is wrong precisely in trends and cycles, the two regimes you most want to read. So the practical rules are sharp. Use the SMA when you actually want the local mean and believe the underlying is roughly flat over the window, for a noise-floor estimate or a slow reference level, not for tracking a trend or timing a cycle. When you want the trend, fit the line and keep the slope rather than averaging it away. When you want to preserve turns, fit a polynomial. And whenever someone defends the SMA as "optimal in the least-squares sense," finish the sentence they left off: optimal for a constant, which the market is not. The old article ["Why the SMA Is Often a Terrible Smoother"](https://aligrithm.com/why-the-sma-is-often-a-terrible-smoother/) said the SMA leaks; this says why even its optimality is a trap, because it is the right answer to a question the market refuses to ask. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_236-visual-0.png) ## KEY POINTS - Fit a straight line to a window by minimizing the sum of squared errors, and the least-squares intercept (the line's value at the window center) works out to the plain average. The SMA is the provably best least-squares fit of a constant. - It is also a non-recursive (FIR) filter with equal one-over-N coefficients. The most basic indicator in technical analysis is a solved optimization problem. - That optimality holds only under one model: the data is a constant plus zero-mean noise. Markets over any meaningful window are not constant. - Over a trend, the best flat fit sits at the window center, so the SMA reports the level N over two bars ago. The lag of the old article ["The Hidden Cost of Every Moving Average: Lag"](https://aligrithm.com/the-hidden-cost-of-every-moving-average-lag/) is derived: it lags because it fits a flat line to non-flat data. - Over a cycle whose length matches the window, the flat fit averages the rising and falling halves to zero, so the SMA can erase the cycle, the response zero at period N. Both failures are correct behavior for the wrong model. - Every smoother is optimal for some implicit model. Want the trend, fit a line and keep the slope (least-squares moving average); want to preserve turns, fit a polynomial (Savitzky-Golay). Use the SMA only when you truly want the local mean of roughly flat data. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Statistically Sound Indicators for Financial Market Prediction - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Statistically+Sound+Indicators+for+Financial+Market+Prediction+Timothy+Masters&ref=aligrithm.com) - [Cycle Analytics for Traders - John Ehlers (Amazon)](https://www.amazon.com/s?k=Cycle+Analytics+for+Traders+John+Ehlers&ref=aligrithm.com) ### 2.68 The Price Change Oscillator URL: https://aligrithm.com/the-price-change-oscillator/ Last updated: 2026-07-27T13:23:28.000Z Volatility is not one number. The volatility of the last five bars and the volatility of the last hundred are different things, and the gap between them carries information that neither level carries alone. The price change oscillator measures that gap: short-term movement minus a long-term baseline, so a reading tells you whether the market is moving faster or slower than it has been. The trap waiting inside it is the one the old article ["When Forcing Stationarity Destroys Information"](https://aligrithm.com/when-forcing-stationarity-destroys-information/) warned about, because the obvious way to clean this indicator up also throws away the exact signal you built it to capture. ## Short-term movement against a long-term baseline Start with a measure of absolute volatility that does not care about direction. Take the log price changes, take their absolute values, and average them over a window. That mean absolute log change is how much the market has been moving per bar, up or down treated alike. Compute it twice, once over a short window and once over a long one, and subtract. $$ v\_K = \\overline{\\,\\lvert \\Delta \\log P \\rvert\\,}\_{\\text{over } K \\text{ bars}} \\qquad \\text{Raw} = \\frac{v\_{\\text{short}} - v\_{\\text{long}}}{c \\cdot \\text{ATR}} $$ The term v is the mean absolute log change over its window, so v-short is recent movement and v-long is the baseline, and the raw oscillator is their difference. Read the difference directly. Positive means the market is moving faster than its baseline, a volatility expansion; negative means it has gone quieter than usual, a contraction. The long-term measure is doing the work of context: a mean absolute move of half a percent means nothing until you know whether the baseline is a quarter percent or a full percent, and subtracting the baseline puts the short-term reading in that context. ## Why ATR has to be the denominator The difference of two absolute-volatility measures is still an absolute quantity, and that is a problem. Double every price change in the series and the difference doubles too, so the raw subtraction is contaminated by the overall scale and the volatility regime, the exact non-stationarity that makes a feature read differently across years and across instruments. The old article ["Why ATR Normalization Is More Than a Volatility Trick"](https://aligrithm.com/why-atr-normalization-is-more-than-a-volatility-trick/) gave the structural fix, and it applies here without modification. Divide by ATR. ATR is in the instrument's own units and scales with the same movement the numerator measures, so dividing one by the other cancels the scale and leaves a dimensionless number that means the same thing on a quiet bond and a frantic crypto pair. There is one more cleanup. Different window lengths and different short-to-long ratios produce difference values on different scales, which hurts a model that wants its features in similar ranges and wrecks any attempt to compress outliers, because you cannot place the usual values in the linear part of a squashing transform if you do not know the usual range in advance. So a self-adjusting scaling constant, set from the lookback and the window ratio, normalizes the range before compression, and then a gentle transform bounds the tails. $$ \\text{PCO} = 100 \\cdot \\Phi\\big(4 \\cdot \\text{Raw}\\big) - 50 $$ The capital phi is the normal cumulative distribution function, the factor of four sets the compression strength so the bulk of the values land in its near-linear middle, and the final scaling recenters the bounded output at zero. The result is a tidy oscillator that reads volatility expansion above zero and contraction below. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## The stationarity-versus-information trade-off Here is where the old article on forcing stationarity earns its place on the page. Dividing by ATR makes the oscillator stationary by removing the absolute volatility level, and removing that level is sometimes exactly wrong. If your strategy's edge lives in the absolute volatility regime itself, in the fact that the market is in a high-vol or low-vol world right now, then normalizing that information away leaves you with a clean, stationary feature that no longer knows the thing you needed it to know. The old article called this violating-signal non-stationarity: the non-stationary component *is* the signal, and aggressive stationarization destroys it. The price change oscillator hands you the dial for this trade-off rather than hiding it. The short-to-long window ratio controls how much absolute information survives. A short baseline relative to the long one keeps the oscillator close to a pure regime reading that still carries some level information; a long, slow baseline strips more of it out in exchange for a more stationary feature. You decide based on what your strategy trades, the same prescription the old article gave: stationarize the part of the feature that is orthogonal to your edge, and leave the part your edge lives in alone. There is no universally correct setting, only a setting correct for what you are trying to predict. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_261-visual-0.png) ## Where it sits and how to set it The price change oscillator is a volatility-regime feature, which is why it leans toward the portfolio and risk side as much as the indicator side: it tells you the market's character has shifted, not where price is going. Use it to switch behavior between calm and turbulent regimes, to widen or tighten risk, or as a conditioning variable that tells a directional signal which world it is operating in. The windows and the ratio are the parameters that matter, and they are not free choices, because they set both the timescale of the regime you detect and how much absolute volatility information you keep, so tie them to the strategy and test the raw-versus-normalized feature the way the old stationarity article prescribed, comparing performance to confirm you did not normalize away your edge. ATR must be computed without peeking at the current bar's full information in a way that leaks, and the compression must be calibrated on training data only, or the bounded, well-mannered oscillator is quietly lookahead-poisoned. ## KEY POINTS - The price change oscillator measures short-term mean absolute log movement minus a long-term baseline, so it reads above zero on volatility expansion and below zero on contraction. - The long-term measure supplies context: a given amount of recent movement is only fast or slow relative to the baseline the market has been running. - The difference is still an absolute quantity that doubles if all moves double, so divide by ATR to cancel scale and volatility regime, the structural fix from the old ATR-normalization article, then compress with the normal CDF to bound tails. - ATR normalization can destroy the signal: if your edge lives in the absolute volatility regime, stationarizing it away leaves a clean feature that no longer knows what you need, the violating-signal case from the old forcing-stationarity article. - The short-to-long window ratio is the trade-off dial between stationarity and retained absolute information; set it from what your strategy trades and test raw versus normalized to confirm you kept the edge. - It is a volatility-regime conditioning feature, not a directional signal; tie the windows to your strategy and keep the construction causal. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Statistically Sound Indicators for Financial Market Prediction - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Statistically+Sound+Indicators+for+Financial+Market+Prediction+Timothy+Masters&ref=aligrithm.com) - [Cycle Analytics for Traders - John Ehlers (Amazon)](https://www.amazon.com/s?k=Cycle+Analytics+for+Traders+John+Ehlers&ref=aligrithm.com) ### 2.45 Nyquist and Aliasing: The Hard Limit on What Price Data Can Show URL: https://aligrithm.com/nyquist-and-aliasing-the-hard-limit-on-what-price-data-can-show/ Last updated: 2026-07-27T13:23:13.000Z The old article ["The Frequency Response of Trading Indicators"](https://aligrithm.com/the-frequency-response-of-trading-indicators/) taught you to read which cycles a filter passes, and the old article ["Dominant Cycle Estimation Without Astrology"](https://aligrithm.com/dominant-cycle-estimation-without-astrology/) measured which cycle the market is running. Both assume the cycle you care about is even visible in your data. This article is about the cycles that are not, and can never be, because of how you sampled. Take one price per bar and there is a hardest, fastest cycle your data can physically represent. Anything faster does not just get missed. It disguises itself as a slower cycle that was never there, and you cannot tell the fake from the real after the fact. This is the one limit in the pillar that no filter, no indicator, and no cleverness can beat. ## The Nyquist limit falls out of the SMA's own math You do not need to import the sampling theorem on faith; it drops out of a filter you already know. The simple moving average's transfer response, factored, contains a term that is a sum of consecutive samples, and that term hits zero at a specific frequency. $$ e^{-j2\\pi f} + 1 = 0 \\;\\Longrightarrow\\; \\cos(\\pi f) = 0 \\;\\Longrightarrow\\; f = 0.5 $$ The frequency f here is measured relative to the sampling rate, so f equals 0.5 means a cycle at half the sampling frequency. With one sample per bar, half the sampling rate is a cycle that completes every two bars: up on one bar, down on the next. That two-bar cycle is the fastest thing sampled data can hold, and it has a name, the Nyquist frequency, half the sampling rate. $$ \\text{shortest period your data can represent} = 2\\ \\text{bars} $$ This is why every indicator period has a floor and a "3-bar cycle" is already scraping the bottom of what exists. It is not a tuning preference or a noise problem. It is arithmetic forced by sampling once per bar, and it holds for daily closes, five-minute bars, and ticks alike, each relative to its own bar. ## Where the zeros sit, and why the SMA is blind on a grid The same factoring shows the full pattern of what an SMA can and cannot see. The N-bar simple moving average has an amplitude response that is a ratio of sines, and it goes to zero at evenly spaced frequencies. $$ H(f) = \\frac{\\sin(N\\pi f)}{N\\sin(\\pi f)}, \\qquad \\text{first zero at } f = \\frac{1}{N}\\ (\\text{period } N) $$ The first zero sits at the frequency whose period equals the filter length, so an SMA of length 50 completely annihilates the 50-bar cycle, the one a naive trader might think it was built to track. More zeros follow at the harmonics. This is the deeper reading of the old article ["Why the SMA Is Often a Terrible Smoother"](https://aligrithm.com/why-the-sma-is-often-a-terrible-smoother/): the SMA is not merely leaky, it is stone blind at a grid of specific periods, and the Nyquist zero at the two-bar period is just the last member of that grid. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Aliasing: the fast cycle that comes back wearing a disguise The Nyquist limit would be harmless if too-fast cycles simply vanished. They do not. A cycle faster than the two-bar floor folds down and reappears as a slower cycle that was never in the market. This is aliasing, and you have seen it outside trading: wagon wheels that spin backward in old films, helicopter rotors frozen on video. The camera samples too slowly, so a fast rotation shows up as a slow or reversed one. Your daily close does the same to any intraday cycle faster than two days. It folds the fast content down into the visible band, where it masquerades as a multi-day cycle indistinguishable from a genuine one. The damage is concrete and the dominant-cycle tools are exposed to it. A periodogram or an autocorrelation estimator can report a clean, convincing 8-day cycle that is an artifact, a fast intraday oscillation folded down by sampling, not a feature of the market. Because the fake is structurally identical to a real cycle in your sampled series, no amount of staring at the daily data can separate them. The defense has to happen before or at sampling: low-pass filter to kill the too-fast content before you downsample it, so it cannot fold, and distrust any measured cycle sitting near the two-bar floor, where aliased junk and real signal pile up together. The honest stance is the one the old article ["Why Market Cycles Are Evanescent"](https://aligrithm.com/why-market-cycles-are-evanescent/) set: a measured cycle is a hypothesis about structure, and a cycle near Nyquist is a hypothesis you should suspect of being a sampling ghost. The filter machinery in this pillar is powerful, but it operates only on what survived sampling, and what sampling discarded or disguised is gone for good. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_235-visual-0.png) ## KEY POINTS - Sampling once per bar sets a hard ceiling: the fastest cycle your data can represent completes in two bars, at half the sampling rate. This is the Nyquist frequency, and it drops out of the SMA's own factored response (cos(pi f) = 0 at f = 0.5). - The limit is arithmetic, not a tuning choice. Every indicator period has a two-bar floor, so a "3-bar cycle" is already at the edge of what physically exists, for daily, intraday, or tick bars alike. - An SMA of length N has its first response zero at the N-bar period and more at the harmonics, so it is stone blind at a grid of specific periods (SMA(50) annihilates the 50-bar cycle), the deeper failure behind the old article ["Why the SMA Is Often a Terrible Smoother."](https://aligrithm.com/why-the-sma-is-often-a-terrible-smoother/) - Cycles faster than the two-bar floor do not vanish; they alias, folding down to masquerade as slower cycles that were never in the market, the wagon-wheel effect. - This poisons cycle measurement: a periodogram can report a clean cycle that is an aliased artifact, structurally indistinguishable from a real one in the sampled series, the old article ["Dominant Cycle Estimation Without Astrology"](https://aligrithm.com/dominant-cycle-estimation-without-astrology/) included. - Defense happens at or before sampling: low-pass to kill too-fast content before downsampling so it cannot fold, and distrust any measured cycle near the two-bar floor. Filters work only on what survived sampling. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Statistically Sound Indicators for Financial Market Prediction - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Statistically+Sound+Indicators+for+Financial+Market+Prediction+Timothy+Masters&ref=aligrithm.com) - [Cycle Analytics for Traders - John Ehlers (Amazon)](https://www.amazon.com/s?k=Cycle+Analytics+for+Traders+John+Ehlers&ref=aligrithm.com) ### 2.67 Deviation from Expectation: Trend Projection as a Signal URL: https://aligrithm.com/deviation-from-expectation-trend-projection-as-a-signal/ Last updated: 2026-07-27T13:23:27.000Z A moving average tells you where price has been. Fit a trend and extend it one bar into the future, and you get something a trailing average cannot give you: a number for where price *should* be right now if the trend that has been running simply continued. The gap between that projection and the actual close is the indicator, and it does a specific job no trend-following tool does. It flags the moment the trend breaks. The old article ["CMMA: A Better Momentum Primitive Than Price-minus-MA Alone"](https://aligrithm.com/cmma-a-better-momentum-primitive-than-price-minus-ma-alone/) built its edge on a residual, price minus a trailing baseline; this swaps the trailing baseline for a forward-projected one, and the residual now measures surprise rather than stretch. ## Project the trend, then subtract it The construction has two steps. First, quantify the trend over a lookback window with a fit that has a slope you can extend, a linear regression line or a low-order polynomial through the recent prices. Then evaluate that fitted curve at the current bar, treating it as the price the trend predicts, and subtract it from what actually printed. $$ \\widehat{P}\_t = \\text{fit}\\big(P\_{t-n},\\dots,P\_{t-1}\\big)\\ \\text{projected to } t \\qquad \\text{Dev}\_t = \\frac{\\log P\_t - \\log \\widehat{P}\_t}{\\text{ATR}} $$ The hatted term is the trend's projection for the current bar, fit over the n prior bars and extrapolated forward; the deviation is the actual log price minus that projection, divided by ATR. Read the sign and size together. A deviation near zero means price landed where the trend expected, the trend is intact, nothing surprising happened. A large positive or negative deviation means price jumped out of line with its own established trajectory, which is exactly the event you want flagged: a regime break, a shock, a trend that just stopped behaving like a trend. This is the old CMMA logic with the baseline turned around. CMMA asks how far price has stretched from a trailing average; deviation from expectation asks how far price missed a forward projection, and the miss is a cleaner regime-break detector because a smooth continuing trend produces a small deviation no matter how steep it is. ## Why you cannot do this on raw price The old article ["The Case Against Raw Price Indicators"](https://aligrithm.com/the-case-against-raw-price-indicators/) spelled out why raw price is unusable: non-stationary in mean, non-stationary in variance, incomparable across instruments. Every one of those defects would poison this indicator if you were careless, and the construction dodges them deliberately. Fit on log prices, not prices, so the trend is a constant-percentage growth line rather than a dollar line that means different things at 300 and 5500\. Take the deviation as a difference of logs, which is stationary in mean because it is centered on the projection rather than on an absolute level. Divide by ATR, including the current bar, so the deviation is in volatility units and a two-ATR miss reads as the same surprise whether the market is calm or wild and whether the instrument is a quiet bond or a frantic crypto. The deviation itself is the cure for raw price's disease. Raw close drifts with the level; the deviation from a projected trend has no level, only a surprise measured in volatility units, so it stays stationary and comparable across the regimes that destroy a raw-price feature. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_260-visual-0.png) ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## A break detector, not an entry signal The deviation from expectation earns its keep as a regime-change flag, and that is a different role from most indicators. A large deviation does not tell you which way to trade; it tells you that the trend your other tools are riding may have just ended, which makes it a natural kill switch for a trend-following system rather than a standalone signal. When the deviation blows out, the assumption that the trend continues has failed, and a trend system leaning on that assumption should stand down or tighten until price settles back onto a projectable path. The fit type and the lookback are real parameters: a short window projects a twitchy trend that throws large deviations on noise, a long one projects a stale trend that misses genuine breaks late, and the choice of linear versus polynomial decides whether you assume straight-line or curving continuation. Like every rolling fit, the projection must be computed from bars strictly before the current close, or the deviation is contaminated by the very bar it is supposed to be surprised by. And the deviation only flags that something changed, never what comes next, so it belongs alongside a directional view, not in place of one. ## KEY POINTS - Fit a trend over a lookback (linear or low-order polynomial), project it one bar forward, and subtract that projection from the actual close. The residual is the indicator. - A near-zero deviation means price landed where the trend expected; a large one means price broke from its own trajectory, which makes this a regime-break detector rather than a stretch gauge. - It is the old CMMA residual logic with a forward-projected baseline instead of a trailing average, so a smooth continuing trend produces a small deviation no matter how steep. - Avoid the raw-price traps from the old raw-price article: fit on log prices, take the deviation as a log difference, and divide by ATR (including the current bar) so the surprise is stationary and comparable across regimes and instruments. - Use it as a kill switch for a trend system, not an entry signal; it flags that the trend may have ended without saying which way price goes next. The fit type and lookback are tunable, and the projection must be causal. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Statistically Sound Indicators for Financial Market Prediction - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Statistically+Sound+Indicators+for+Financial+Market+Prediction+Timothy+Masters&ref=aligrithm.com) - [Cycle Analytics for Traders - John Ehlers (Amazon)](https://www.amazon.com/s?k=Cycle+Analytics+for+Traders+John+Ehlers&ref=aligrithm.com) ### 2.44 The Filter Coefficient Cookbook: One Equation for EMA, LPF, HPF, BPF URL: https://aligrithm.com/the-filter-coefficient-cookbook-one-equation-for-ema-lpf-hpf-bpf/ Last updated: 2026-07-27T13:23:13.000Z The old article ["The Trader's Guide to Low-Pass Filters"](https://aligrithm.com/the-traders-guide-to-low-pass-filters/) built smoothers, the old article ["High-Pass Filters for Traders"](https://aligrithm.com/high-pass-filters-for-traders/) built detrenders, and the old article ["Band-Pass Filters: The Most Underused Tool in Technical Analysis"](https://aligrithm.com/band-pass-filters-the-most-underused-tool-in-technical-analysis/) built cycle isolators. They look like different machines. They are not. Every one of them is the same second-order equation with different numbers plugged in. This article gives you that equation and the table of numbers, so you can stop installing a separate indicator for every job and instead run one filter engine and swap six coefficients. Once you see the cookbook, the named-indicator zoo collapses into one recipe card. ## One engine: the second-order transfer function Take the general filter from the recursive-versus-non-recursive split and keep terms up to two bars back. That second-order form is enough to build every standard trading filter. $$ \\text{output} = \\frac{b\_0 + b\_1 z^{-1} + b\_2 z^{-2}}{1 + a\_1 z^{-1} + a\_2 z^{-2}}\\;\\text{input} $$ Rearranged for code, the output is b0 times input, plus b1 times input one bar ago, plus b2 times input two bars ago, minus a1 times output one bar ago, minus a2 times output two bars ago. That is the whole engine: three input taps, two feedback taps, one line of arithmetic. A low-pass, a high-pass, a band-pass, a band-stop, and the plain EMA are all this same line with different b and a values. Build it once and you never write another filter from scratch; you look up coefficients. ## The two constants every recipe is built from The coefficients are not arbitrary. They all derive from the critical period you want, the cycle length at the boundary between pass and stop, through one smoothing constant alpha. $$ \\alpha = \\frac{\\cos\\omega + \\sin\\omega - 1}{\\cos\\omega}, \\qquad \\omega = \\frac{2\\pi}{\\text{period}} $$ You pick the period in bars, convert it to an angle omega, and out comes alpha, a number between zero and one that sets how aggressively the filter smooths. The low-pass and high-pass recipes need only alpha. The band-pass and band-stop need one more input, a bandwidth, which produces a second constant that controls how narrow the passed or rejected band is and a center term tied to the period. So the user-facing dials are honest and few: a period for everything, plus a bandwidth for the band filters. Everything else is bookkeeping the cookbook does for you. ## The recipe card Here is the table. Pick the filter you want, compute alpha from your period, and fill in the coefficients. The leading denominator coefficient a0 is always one. | Filter | b0 | b1 | b2 | a1 | a2 | | --------------------- | -------- | ---------- | -------- | ------- | ------ | | EMA (1-pole low-pass) | α | 0 | 0 | −(1−α) | 0 | | 2-pole low-pass | α² | 0 | 0 | −2(1−α) | (1−α)² | | High-pass (1-pole) | 1−α/2 | −(1−α/2) | 0 | −(1−α) | 0 | | 2-pole high-pass | (1−α/2)² | −2(1−α/2)² | (1−α/2)² | −2(1−α) | (1−α)² | | Band-pass | (1−σ)/2 | 0 | −(1−σ)/2 | −λ(1+σ) | σ | | Band-stop | (1+σ)/2 | −λ(1+σ) | (1+σ)/2 | −λ(1+σ) | σ | Read the structure and the families from the earlier article jump out. The low-pass recipes have all their weight in the numerator's b0 with feedback in the denominator, so they pass the slow trend. The high-pass recipes put a minus sign between b0 and b1, a difference that cancels the DC level and keeps the fast deviation, which is why a one-pole high-pass is close to taking the difference of consecutive bars. The band-pass and band-stop use the extra constants lambda, the cosine of the center period angle, and sigma, set by the bandwidth, to carve out or notch a middle band. Doubling a pole, going from one-pole to two-pole, squares the relevant terms and steepens the rolloff at the cost of more lag, exactly the trade the low-pass article priced. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## What the cookbook is for, and what it is not The practical win is real. You implement one second-order section, expose a period dial and a filter-type selector, and you have replaced a drawer full of separate indicators with one tested routine. Need a steeper cut? Cascade two sections and the orders add. Need a roofing filter? Run a high-pass into a low-pass. The composition is the polynomial multiplication from the transfer-function article, done with coefficient blocks instead of named products. This is the engineering posture the old article ["Why Traders Should Analyze Indicators Mathematically"](https://aligrithm.com/why-traders-should-analyze-indicators-mathematically/) was pushing toward: you stop collecting indicators and start configuring one. The cookbook does not, however, tell you which filter to want, and that is the part that decides whether you make money. Choosing the period sets which cycles you keep, choosing the type sets the job, and both are bets about your instrument that the table cannot make for you. A perfectly implemented two-pole low-pass tuned to the wrong period is a clean answer to the wrong question. And every recipe here is linear, so it carries the lag its structure implies; the cookbook makes lag easy to compute, not easy to avoid. Use it to build filters fast and correctly, then spend your real effort on the only hard questions it leaves open: which band carries edge on your data, and whether the lag fits your holding period. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_234-visual-0.png) ## KEY POINTS - Every standard trading filter is the same second-order transfer function with different coefficients: three input taps (b0, b1, b2) and two feedback taps (a1, a2), one line of arithmetic. - All the coefficients derive from the critical period through one smoothing constant alpha equals (cos w plus sin w minus 1) over cos w, with w equal to two pi over period. Band filters need one extra bandwidth constant. - The recipe table gives EMA, 1- and 2-pole low-pass, 1- and 2-pole high-pass, band-pass, and band-stop. Low-pass weight sits in b0; high-pass uses a minus between b0 and b1 to cancel DC; band filters use lambda (center) and sigma (bandwidth). - Doubling a pole (1-pole to 2-pole) squares the relevant terms and steepens the rolloff at the cost of more lag, the trade from the old article ["The Trader's Guide to Low-Pass Filters."](https://aligrithm.com/the-traders-guide-to-low-pass-filters/) - The payoff: implement one second-order section, expose a period dial and a type selector, and replace a drawer of separate indicators. Cascade sections for steeper cuts; compose by multiplying transfer functions. - The cookbook cannot tell you which filter to want. Period and type are bets about your instrument; a perfect filter on the wrong period answers the wrong question, and every recipe carries the lag its structure implies. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Statistically Sound Indicators for Financial Market Prediction - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Statistically+Sound+Indicators+for+Financial+Market+Prediction+Timothy+Masters&ref=aligrithm.com) - [Cycle Analytics for Traders - John Ehlers (Amazon)](https://www.amazon.com/s?k=Cycle+Analytics+for+Traders+John+Ehlers&ref=aligrithm.com) ### 2.66 The Aroon Difference Oscillator URL: https://aligrithm.com/the-aroon-difference-oscillator/ Last updated: 2026-07-27T13:23:26.000Z Almost every trend indicator measures the size of price moves: how far price traveled, how big the gap between two averages grew, how steep the slope is. Aroon ignores size entirely and measures time. It asks one question, how long since the market last printed a new high or a new low, and that single shift in framing produces an oscillator that is bounded, stationary, and comparable across instruments by construction, before you apply a single transform. The old article ["Why Most Indicators Should Be Transformed Before Modeling"](https://aligrithm.com/why-most-indicators-should-be-transformed-before-modeling/) listed the four geometric properties a model needs from a feature. Aroon arrives with most of them already satisfied, which is rare. ## Counting bars, not dollars Pick a lookback window of n bars including the current one. Find the bar holding the highest high in that window and count how many bars ago it was. If the current bar just made the highest high, that count is zero; if the highest high sits at the oldest bar in the window, the count is n. Aroon Up turns that count into a 0-to-100 reading by linear interpolation, and Aroon Down does the identical thing for the lowest low. $$ \\text{AroonUp} = 100 \\cdot \\frac{n - (\\text{bars since highest high})}{n} \\qquad \\text{AroonDown} = 100 \\cdot \\frac{n - (\\text{bars since lowest low})}{n} $$ The term n is the lookback, and the bracketed count is how many bars back the extreme occurred. A fresh high today gives Aroon Up of 100; a high that has not been touched for the full window gives 0; anything between scales linearly with recency. Read the two lines together and the logic is immediate. Aroon Up near 100 with Aroon Down near 0 means the market keeps making new highs and has not made a new low in ages, a clean uptrend. Both lines low means neither extreme has printed recently, a market gone quiet inside its range. ## The difference is the well-behaved oscillator Watching two lines is clumsy for a model, so subtract them. $$ \\text{AroonDiff} = \\text{AroonUp} - \\text{AroonDown} $$ The Aroon difference lives in minus 100 to plus 100, centered at zero, with positive values signaling recent highs dominating and negative values signaling recent lows dominating. This is a well-behaved oscillator in the precise sense the old transform article meant. It has a stable scale across time because it is built from bar counts, not prices, so it reads the same in 1995 and 2025 regardless of whether the index sits at 300 or 5500\. It is bounded, so no outlier can blow out its range and saturate a model. It is comparable across instruments, because a count of bars-since-new-high means the same thing on a stock, a currency, and a futures contract. Raw price fails all three of those, which is why the old article on raw price indicators dismissed it; Aroon clears them for free because it never touches the price level, only the timing of extremes. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Read the histogram before you trust it The old article ["Why Indicator Histograms Matter"](https://aligrithm.com/why-indicator-histograms-matter/) insisted that a scalar summary hides the shape, and the shape decides the action. Aroon rewards that habit, because its distribution exposes a quirk you have to know about. The indicator is built on the passage of time, not the magnitude of price change, and those two can decouple hard. A market can grind to a strong directional Aroon reading while barely moving in price, because it keeps nudging out marginal new highs on tiny increments. Another market can run a violent, profitable trend yet show a weak Aroon, because one early spike set a high that later bars never quite beat, so the bars-since-new-high count climbs even as price trends. Plot the Aroon difference against forward returns and the histogram tells you whether, on your instrument, the time-based reading actually lines up with tradeable moves or fires on flat drift. That is why Aroon is a companion, not a standalone. It quantifies the timing structure of a trend cleanly, and it stays silent on whether the trend carries enough price movement to trade, so it pairs with a magnitude-based trend or volatility indicator that supplies the piece Aroon deliberately throws away. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_259-visual-0.png) ## What you get and what you gave up The Aroon difference is a bounded, stationary, cross-instrument timing oscillator that needs no rescaling to enter a model, which makes it cheap and robust as a feature. The cost is baked into the design: by counting bars instead of measuring moves, it is blind to how big those moves are, so it can shout trend on a flat tape and whisper on a runaway one. The lookback n is a real parameter that sets which timescale's extremes you are tracking, and a length that suits a swing horizon will misread an intraday one, so set it from the cycle you intend to trade rather than a default. Treat a strong Aroon reading as evidence about the timing of new extremes, confirm it with something that measures price magnitude, and run the histogram against forward returns on your own instrument before you let it size a position. ## KEY POINTS - Aroon measures time, not price size: how many bars since the last new high (Aroon Up) and new low (Aroon Down), each interpolated linearly to a 0-to-100 reading. - The Aroon difference, Up minus Down, is an oscillator bounded in minus 100 to plus 100 and centered at zero, positive when recent highs dominate and negative when recent lows do. - It satisfies the old transform article's feature requirements for free: stable scale across time, bounded range, and cross-instrument comparability, because it never touches the price level, only the timing of extremes. Raw price fails all three. - Timing and magnitude decouple: Aroon can read strong on a flat market that keeps nudging marginal highs and weak on a violent trend whose early spike was never beaten. Read the histogram against forward returns, as the old histograms article demanded. - Use it as a companion to a magnitude-based trend or volatility indicator, not alone; the lookback sets the timescale and must match the cycle you trade. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Statistically Sound Indicators for Financial Market Prediction - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Statistically+Sound+Indicators+for+Financial+Market+Prediction+Timothy+Masters&ref=aligrithm.com) - [Cycle Analytics for Traders - John Ehlers (Amazon)](https://www.amazon.com/s?k=Cycle+Analytics+for+Traders+John+Ehlers&ref=aligrithm.com) ### 2.43 Recursive vs Non-Recursive: The Two Families of Every Indicator URL: https://aligrithm.com/recursive-vs-non-recursive-the-two-families-of-every-indicator/ Last updated: 2026-07-27T13:23:12.000Z The old article ["Why Traders Should Analyze Indicators Mathematically"](https://aligrithm.com/why-traders-should-analyze-indicators-mathematically/) argued that every indicator is a signal-processing operation, and the old article ["The Transfer Function View: Characterizing Any Indicator with the Z-Transform"](https://aligrithm.com/the-transfer-function-view-characterizing-any-indicator-with-the-z-transform/) showed how to write any linear one as a ratio of two polynomials. This article makes the split that ratio implies. There are exactly two families of linear filter, and every moving average, oscillator, and smoother you have ever used belongs to one or the other. Which family a filter is in determines its lag behavior, its stability, and its cost, before you know anything else about it. Learn the dividing line and you can predict a filter's worst habits from its structure alone. ## The line: does the filter eat its own output? Write any linear filter as output equals numerator over denominator times input, with the leading denominator coefficient set to one. $$ \\text{output} = \\frac{b\_0 + b\_1 z^{-1} + b\_2 z^{-2} + \\cdots}{1 + a\_1 z^{-1} + a\_2 z^{-2} + \\cdots}\\;\\text{input} $$ The numerator uses only input data, past and present prices. The denominator uses previously computed outputs, the filter's own past results. That single distinction names the two families. If the denominator is trivial, the filter touches only inputs, and it is non-recursive. If the denominator carries real terms, the filter feeds its own output back in, and it is recursive. Rearranged for computation, a recursive filter reads output equals the input terms minus the fed-back output terms, which is why it can remember forever from one or two stored numbers. Every indicator sorts into one of these two bins by this one question: does it eat its own output. ## Non-recursive: finite memory, honest lag, no surprises A non-recursive filter is a weighted sum of past inputs and nothing else. Its impulse response is finite, the family name FIR, finite impulse response, because once a price falls out of the window it is gone. The simple moving average is the plain case, equal weights summed over N bars. With only a numerator, the transfer function is a single polynomial, and you can factor that polynomial into roots called the zeros of the response, the cycle speeds the filter kills dead. The property that makes this family worth using is its lag behavior. A non-recursive filter whose coefficients are symmetric about the center has a lag equal to its degree divided by two, and that lag is the same at every frequency. $$ \\text{lag} = \\frac{\\text{degree}}{2}\\ \\text{bars, constant across all frequencies} $$ A degree-six symmetric filter lags three bars, whether the cycle is fast or slow. That is linear phase, and it matters because the filter delays every component by the same amount, so it does not distort the shape of a waveform passing through, it only shifts it. The cost is bluntness: a sharp non-recursive filter needs many taps, which means more lag and more computation, and the simple moving average pays for its equal weights with leaky sidelobes, the failure detailed in the old article ["Why the SMA Is Often a Terrible Smoother."](https://aligrithm.com/why-the-sma-is-often-a-terrible-smoother/) What you get in return is predictability. A non-recursive filter cannot go unstable and holds no nasty resonances; what you see in its coefficients is what you get. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Recursive: infinite memory, cheap and sharp, occasionally feral A recursive filter feeds its own output back, so its impulse response is infinite, the family name IIR, because a piece of every past bar lives on forever in the feedback. The exponential moving average is the canonical case: today's value is a slice of today's price plus the rest of yesterday's output, one stored number holding the entire history. With a real denominator, the transfer function's denominator polynomial factors into roots called poles, the feedback resonances, and the EMA has exactly one. The payoff is efficiency. A recursive filter gets a sharp response from one or two stored values, where a non-recursive filter would need dozens of taps for the same steepness. That O(1) or O(2) state is decisive for streaming and intraday work. The price is that feedback comes with teeth. A recursive filter does not have linear phase, so its lag varies with frequency and it distorts waveform shape, not just timing. A pole placed badly makes the filter ring, overshoot, or in the worst case run away unstable, none of which a non-recursive filter can do. So the families trade cleanly: non-recursive buys you stability, linear phase, and transparency at the cost of many taps and lag; recursive buys you sharpness and near-zero storage at the cost of nonlinear phase, possible ringing, and the chance of instability. ## Why the label is the first thing to check Knowing a filter's family tells you its failure modes before you test it. Hand me a non-recursive filter and I know its lag from its degree, I know it is stable, and I know its main risk is sidelobe leakage from a poor window. Hand me a recursive filter and I know it is cheap and can be sharp, and I know to check it for ringing, frequency-dependent lag, and stability of its poles. Most of the smoothers in this pillar, the EMAs, the super-smoother, the Butterworth, the band-pass recursions, are recursive, which is why poles and ringing keep coming up. Most of the windowed averages and the difference-based detrenders are non-recursive, which is why their lag is a clean degree-over-two. When you reach for an indicator, read its structure first: the family it belongs to has already decided half of how it will behave and how it will fail, and the old article ["The Trader's Guide to Low-Pass Filters"](https://aligrithm.com/the-traders-guide-to-low-pass-filters/) picks pole counts inside exactly this recursive family. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_233-visual-0.png) ## KEY POINTS - Write any linear filter as numerator over denominator times input. The numerator uses inputs only; the denominator uses the filter's own past outputs. That split defines the two families. - Non-recursive (FIR) filters use inputs only, have finite memory, and their numerator factors into zeros (cycles killed). The SMA is the plain case. - A symmetric non-recursive filter lags by degree over two, the same at every frequency (linear phase), so it shifts a waveform without distorting its shape. It is stable and transparent but needs many taps for sharpness and can leak through sidelobes. - Recursive (IIR) filters feed their output back, have infinite memory from one or two stored numbers, and their denominator factors into poles (resonances). The EMA is the canonical case, with one pole. - Recursive filters are cheap and sharp but have nonlinear (frequency-dependent) lag, can ring or overshoot, and can go unstable if a pole is misplaced, none of which a non-recursive filter can do. - The family is the first thing to check: it predicts the failure modes. Most smoothers in this pillar are recursive (watch for ringing and unstable poles); most windowed averages and difference detrenders are non-recursive (clean degree-over-two lag). --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Statistically Sound Indicators for Financial Market Prediction - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Statistically+Sound+Indicators+for+Financial+Market+Prediction+Timothy+Masters&ref=aligrithm.com) - [Cycle Analytics for Traders - John Ehlers (Amazon)](https://www.amazon.com/s?k=Cycle+Analytics+for+Traders+John+Ehlers&ref=aligrithm.com) ### 2.65 ADX Done Right: Two-Level Smoothing URL: https://aligrithm.com/adx-done-right-two-level-smoothing/ Last updated: 2026-07-27T13:23:25.000Z Everyone knows the ADX as the line that tells you whether a market is trending or chopping, and almost nobody who plots it can tell you what it actually computes or why it has two separate smoothing stages buried inside it. That ignorance is fine until you try to use it as a model feature, at which point the construction details decide whether you have a stationary, comparable signal or a volatility artifact. The old article ["Why ATR Normalization Is More Than a Volatility Trick"](https://aligrithm.com/why-atr-normalization-is-more-than-a-volatility-trick/) gave the key: the ADX is not a price indicator at all, it is a ratio of ATR-normalized range expansions, and once you see it that way the whole thing makes sense. ## What the ADX is trying to measure Wilder built the ADX to answer one question, how strong is the trend, with no opinion on direction. It is a strength gauge, so it pairs with a directional signal rather than standing alone. The raw material is not the close. It is the bar-to-bar change in the highs and the lows, treated separately, on the logic that a market making higher highs is expressing upward pressure and one making lower lows is expressing downward pressure, and the ADX cares about how lopsided that pressure is. For each bar, measure how much the current high exceeds the prior high, and how much the current low undercuts the prior low. Compare the two and keep only the larger as that bar's directional movement, setting the other to zero, and if even the larger one is negative, set both to zero. The rule sounds fussy but the intent is clean: count a bar only when the price range expanded past the prior bar, a higher high or a lower low, and ignore inside bars where the range contracted and neither extreme was beaten. $$ \\text{DM}^{+} = H\_t - H\_{t-1}, \\qquad \\text{DM}^{-} = L\_{t-1} - L\_t $$ $$ \\text{keep the larger, zero the other; if the larger} < 0,\\ \\text{both} = 0 $$ The terms with H and L are this bar's and the prior bar's high and low, so the plus movement is how far the high pushed up and the minus movement is how far the low pushed down. Keeping only the dominant one per bar is what makes the eventual oscillator read range expansion rather than mere wiggle. ## Two levels of smoothing, and the ATR denominator These raw movements are too volatile to use directly, a single dramatic bar swamps them, so the first smoothing pass, the inner level, averages each of the three quantities separately: the plus movement, the minus movement, and the ATR. The smoothing is the same EMA-style recursion the RSI and the rest of Wilder's indicators use. $$ S\_t = \\frac{(L-1)\\,S\_{t-1} + X\_t}{L} $$ The term X is the quantity being smoothed on the current bar, S is its running smoothed value, and L is the lookback. Now the ATR normalization does the real work. Divide each smoothed directional movement by the smoothed ATR and you convert raw price-unit movements into dimensionless directional indicators that mean the same thing on any instrument in any volatility regime. $$ \\text{DI}^{+} = \\frac{\\text{DM}^{+}}{\\text{ATR}}, \\qquad \\text{DI}^{-} = \\frac{\\text{DM}^{-}}{\\text{ATR}} $$ $$ \\text{Raw ADX} = 100 \\cdot \\frac{\\lvert\\, \\text{DI}^{+} - \\text{DI}^{-}\\,\\rvert}{\\text{DI}^{+} + \\text{DI}^{-}} $$ This is the old ATR article's thesis made literal: ATR is the structurally correct denominator because the directional movements are price-unit quantities generated by the same volatility, so dividing by ATR removes the regime and leaves a pure shape. Read the ratio's two pieces. The denominator, plus-DI plus minus-DI, is the total range expansion the market produced, up moves and down moves added together. The numerator, the absolute difference, is how one-sided that expansion was. Their ratio is the fraction of total expansion that pointed one way, which is exactly what "trend strength regardless of direction" should mean: a market expanding hard but evenly in both directions scores low, and one expanding all in one direction scores near 100. The outer smoothing is the second level. The raw ADX itself is still jumpy, so smooth it again with the same recursion to get the line you plot. Two stages, inner on the components and outer on the ratio, which is the detail the indicator's name hides and the reason it behaves as a stable trend-strength reading instead of a twitchy one. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_258-visual-0.png) ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## The lag it carries, and how to use it The ADX done this way is a bounded, stationary, cross-instrument trend-strength feature, which is what you want, and it comes with a built-in delay you must respect. Two smoothing passes mean the line reacts late, and the lag bites hardest at the end of sharp trends: a strong move that suddenly dies leaves the ADX elevated for several bars, still reporting strength that has already evaporated. So read a high ADX as confirmation that a trend has been strong, not as a promise it will continue, and never use the ADX to time exits on fast reversals where its lag will leave you holding. Because it is strength without direction, it is a filter and a regime switch, telling a trend system to engage and a mean-reversion system to stand down, but it needs a separate signal for the which-way call. The two lookbacks, inner and outer, are real parameters that trade responsiveness against stability, and the ATR window must be set and lagged so it does not peek at the current bar, the same causality discipline the old ATR article demanded, or the tidy bounded line is quietly lookahead-poisoned. ## KEY POINTS - The ADX measures trend strength, not direction, so it pairs with a directional signal. Its raw material is the bar-to-bar change in highs and lows treated separately, not the close. - Per bar, keep the larger of the up-movement and down-movement, zero the other, and zero both if the range did not expand, so the indicator reads range expansion rather than mere wiggle. - It has two smoothing levels: an inner pass on the plus-movement, minus-movement, and ATR separately, then an outer pass on the final ratio. The name hides this and it is why the line is stable. - Dividing each smoothed movement by ATR turns price-unit movements into regime-free, cross-instrument directional indicators, the literal payoff of the old ATR-normalization article. - The ratio is total range expansion (DI+ plus DI-) in the denominator and how one-sided it was (absolute DI+ minus DI-) in the numerator, so it reads the fraction of expansion that pointed one way. - It lags, worst at the end of sharp trends where it stays elevated after the move dies; use it as a strength filter and regime switch, not an exit timer, and keep the ATR window causal. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Statistically Sound Indicators for Financial Market Prediction - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Statistically+Sound+Indicators+for+Financial+Market+Prediction+Timothy+Masters&ref=aligrithm.com) - [Cycle Analytics for Traders - John Ehlers (Amazon)](https://www.amazon.com/s?k=Cycle+Analytics+for+Traders+John+Ehlers&ref=aligrithm.com) ### 2.42 Designing an Indicator by Specifying Its Phase First URL: https://aligrithm.com/designing-an-indicator-by-specifying-its-phase-first/ Last updated: 2026-07-27T13:23:12.000Z Every article in this pillar so far took an indicator and read off its behavior: write the filter, derive its frequency response, see what it keeps and how late it reports. The old article ["The Frequency Response of Trading Indicators"](https://aligrithm.com/the-frequency-response-of-trading-indicators/) made reading that response routine. This article runs the machine backwards. A trader does not care about coefficients; a trader cares about lag, and lag is phase. So why not specify the phase response you want, low lag in the band that matters, and solve for the indicator that delivers it? You can, and the attempt teaches the deepest constraint in causal filtering: you are not allowed to choose phase freely, because causality has already spent half your freedom. ## The honest goal: pick the lag, then find the filter The old article ["How to Think About Indicator Lag Before Backtesting"](https://aligrithm.com/how-to-think-about-indicator-lag-before-backtesting/) framed lag as a budget you must meet, and phase is where lag lives, phase divided by frequency is lag in bars. The forward way to hit a lag target is trial and error: build a filter, measure its phase, adjust, repeat. The backward way is to state the target phase response across the frequency band as the design specification and solve for the filter that realizes it. That is design by phase: decide that you want near-zero lag from period 20 to period 40 and tolerable lag elsewhere, then ask what indicator has that phase. It is the right question because it starts from the thing you actually care about instead of from coefficients you do not. ## Causality couples the real and imaginary parts Here is where the attempt collides with a law. A filter's frequency response is a complex number at each frequency, a real part and an imaginary part, and together they encode both the magnitude (how much each cycle survives) and the phase (how late it arrives). You might hope to dial magnitude and phase independently. You cannot, if the filter is causal, meaning it uses only past and present data, which every tradeable indicator must. Causality forces a fixed relationship between the real and imaginary parts: fix one and the other is determined. $$ H\_I(\\omega) = -\\frac{1}{2\\pi}\\int\_{-\\pi}^{\\pi} H\_R(\\lambda)\\,\\cot\\!\\left(\\frac{\\omega - \\lambda}{2}\\right) d\\lambda $$ The imaginary part is a fixed integral transform of the real part, a Hilbert relation. This is the discrete cousin of the Kramers-Kronig relations that physicists use, and it is not a guideline, it is a consequence of the filter not being allowed to respond before its input arrives. The phase is then the angle of that complex response. $$ \\tan\\varphi(\\omega) = \\frac{H\_I(\\omega)}{H\_R(\\omega)} $$ So the moment you specify a phase response, the coupling pins down the magnitude response too, and vice versa. You get to design one freely; the other is handed to you by causality. There is no causal filter with an arbitrary phase and an arbitrary magnitude. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## What this means for the dream of zero lag This is the rigorous burial of every zero-lag fantasy, and it is more fundamental than the lag-reduction caveats in earlier articles. Suppose you specify the dream: gain of one across your cycle band, like a perfect bandpass, and zero phase everywhere, no lag at all. The coupling says that pairing is impossible for a causal filter. Demand zero lag in your band and the real-imaginary relation forces a magnitude response you did not ask for and probably do not want, gain that overshoots, rings, or amplifies noise where you wanted it flat. Push the phase down in one band and it pops up elsewhere, because the total is conserved by the transform. You are moving lag around the spectrum, never deleting it. The old articles said zero-lag claims are lag reduction in disguise; this says why, from the structure of causality itself. That makes design-by-phase a real tool with an honest ceiling. You can shape the phase to put your low lag where your strategy trades and accept worse lag and a compromised magnitude in bands you do not use, which is a legitimate and powerful optimization. What you cannot do is wish lag away across the board. The discipline that follows is concrete. Specify the phase only in the band that matters, and inspect the magnitude the coupling forces on you before committing, because a beautiful phase with a ringing, noise-amplifying magnitude is a worse indicator than an honest laggy one. Read both halves of the complex response, always, because causality will not let you ignore the half you did not design. The reward for respecting the constraint is an indicator tuned to your actual lag budget. The punishment for ignoring it is a filter that meets your phase spec and feeds your model garbage. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_232-visual-0.png) ## KEY POINTS - Design-by-phase runs the frequency-response reading of the old article ["The Frequency Response of Trading Indicators"](https://aligrithm.com/the-frequency-response-of-trading-indicators/) backwards: specify the phase (hence lag) you want across the band, then solve for the indicator that delivers it. - It starts from what traders actually care about, lag, which is phase divided by frequency, the budget framed in the old article ["How to Think About Indicator Lag Before Backtesting."](https://aligrithm.com/how-to-think-about-indicator-lag-before-backtesting/) - A causal filter's frequency response is complex (real and imaginary parts), and causality forces the imaginary part to be a fixed Hilbert transform of the real part. Fix one and the other is determined. - Therefore magnitude and phase cannot be chosen independently. The phase is the angle of the coupled response (tan phi equals imaginary over real), so specifying phase pins down magnitude and vice versa. - The zero-lag dream (flat gain plus zero phase) is impossible for a causal filter. Demanding low lag in one band forces an unwanted magnitude (overshoot, ringing, noise gain) and pushes lag elsewhere. You move lag around the spectrum, never delete it. - Used honestly it is powerful: shape low lag into the band you trade, accept worse lag where you do not, and always inspect the magnitude causality forces before committing. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Statistically Sound Indicators for Financial Market Prediction - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Statistically+Sound+Indicators+for+Financial+Market+Prediction+Timothy+Masters&ref=aligrithm.com) - [Cycle Analytics for Traders - John Ehlers (Amazon)](https://www.amazon.com/s?k=Cycle+Analytics+for+Traders+John+Ehlers&ref=aligrithm.com) ### 2.64 Price Intensity: Reading Intrabar Conviction URL: https://aligrithm.com/price-intensity-reading-intrabar-conviction/ Last updated: 2026-07-27T13:23:24.000Z Most indicators throw away three quarters of every bar. They take the close and discard the open, the high, and the low, which means they ignore the entire story of what happened between the bell rings. Price intensity, David Bostian's construction, reads that discarded story: it asks how decisively the bar traveled from open to close relative to how far it sprawled in total. The old article ["Why Most Indicators Should Be Transformed Before Modeling"](https://aligrithm.com/why-most-indicators-should-be-transformed-before-modeling/) insisted that same-bar OHLC differences are some of the few near-raw quantities that survive as features, and intensity is the sharpest of them, a pure read on intrabar conviction. ## Open-to-close, scaled by the true range The raw intensity is the net distance price covered from open to close, divided by the largest of three spans that capture the bar's full reach including its gap from the prior close. $$ \\text{Raw PI} = \\frac{C - O}{\\max\\big(H - L,\\; H - C\_1,\\; C\_1 - L\\big)} $$ The terms C and O are this bar's close and open, H and L its high and low, and the C with the subscript is the prior bar's close. The numerator is the directional payload: how far, and which way, price ended up versus where it started the session. The denominator is the true range, the widest of the bar's own high-low span and the two spans that reach back to the prior close, so a gap that opened the bar away from yesterday is counted as part of the territory covered. Divide one by the other and intensity lands in minus one to plus one. A reading near plus one means the bar opened low, closed high, and spent its whole range pushing in that one direction, pure buying conviction. Near zero means price thrashed across a wide range and ended where it began, a lot of motion and no resolve. ## The volume-free version is the useful one Bostian's original intensity multiplied this by volume, on the logic that a conviction move backed by heavy trade means more than the same move on thin trade. Strip the volume out and you get a cleaner indicator. The volume-free intensity still captures the intrabar conviction, it is far easier to normalize because it is already bounded, and it turns out to be effective for mean reversion: a string of high-conviction bars in one direction is often the setup for a snap back, not a reason to chase. Dropping volume also sidesteps the non-stationarity that volume drags in, since traded volume trends, spikes, and shifts its whole distribution across years in ways that contaminate any indicator built on it. The raw intensity has one rough edge: it swings wildly from bar to bar, because a single indecisive bar can flip it from near plus one to near minus one regardless of the larger drift. So smooth it with an exponential average, which is the standard tool here, to expose the underlying tendency of intrabar trend rather than the per-bar jitter. $$ \\text{NPI} = 100 \\cdot \\Phi\\!\\big(\\sqrt{K}\\cdot \\text{EMA}(\\text{Raw PI})\\big) - 50 $$ The smoothed intensity then gets normalized. The EMA term is the exponentially smoothed raw intensity, K is the effective smoothing length, and multiplying the smoothed value by the square root of that length restores a consistent scale, the same random-walk root-length correction the moving-average difference used, so a longer smoother does not silently shrink the indicator's range. The normal CDF, written here as capital phi, compresses the result into a bounded band and the final scaling recenters it at zero. The whole chain is the old transform article's recipe applied to one clean primitive: take a same-bar OHLC quantity, smooth the jitter, rescale for lookback independence, and squash to a bounded model-ready feature. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_257-visual-0.png) ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Where it helps and where it lies Price intensity is an intrabar-conviction read, not a trend or a price level, and that narrowness is its strength and its limit. It tells you how hard a bar pushed relative to its own range, so it is most useful as a mean-reversion feature, flagging when conviction has run hot enough to invite a reversal, and as a confirmation overlay on a separate directional signal. It does not know where price is or where the trend points, so leaning on it alone is asking a conviction gauge to do a job it was never built for. The smoothing length is a real parameter that sets how much per-bar jitter you suppress against how much lag you accept, and the normalization must be computed causally or the bounded, tidy-looking feature is quietly lookahead-poisoned. Same-bar OHLC construction keeps it honest and stationary, but stationary is not the same as predictive, and intensity still has to clear that bar on your instrument before it earns a slot in the feature set. ## KEY POINTS - Price intensity reads the part of the bar most indicators discard: open-to-close net travel divided by the true range, where the true range is the widest of the high-low span and the two spans back to the prior close, so gaps count. - It lands in minus one to plus one. Near plus one is a bar that pushed one direction across its whole range (conviction); near zero is wide thrashing that ended where it began (no resolve). - The volume-free version is the better feature: it stays bounded and easy to normalize, works well for mean reversion, and avoids the non-stationarity that volume drags in. - Raw intensity is jumpy, so smooth it with an EMA to expose the intrabar tendency, then multiply by the square root of the smoothing length for scale consistency and compress with the normal CDF, the same transform chain from the old indicator-transform article. - It is an intrabar-conviction gauge only, best as a mean-reversion or confirmation feature; it carries no trend direction. The smoothing length is a real lag-versus-noise knob and the normalization must be causal. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Statistically Sound Indicators for Financial Market Prediction - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Statistically+Sound+Indicators+for+Financial+Market+Prediction+Timothy+Masters&ref=aligrithm.com) - [Cycle Analytics for Traders - John Ehlers (Amazon)](https://www.amazon.com/s?k=Cycle+Analytics+for+Traders+John+Ehlers&ref=aligrithm.com) ### 2.41 Momentum Is a High-Pass Filter URL: https://aligrithm.com/momentum-is-a-high-pass-filter/ Last updated: 2026-07-27T13:23:11.000Z Traders treat momentum as a strength gauge: price now minus price N bars ago, big when the market is moving, small when it is not. The old article ["High-Pass Filters for Traders"](https://aligrithm.com/high-pass-filters-for-traders/) built detrenders that keep the fast deviations and throw away the slow trend. This article collapses the distinction, because they are the same thing. Momentum is a high-pass filter, exactly and provably, and writing out its frequency response tells you precisely which cycles it amplifies, which it kills, and how many bars late it reports them. The strength-gauge framing hides all of that. The filter framing exposes it. ## Momentum is a first-degree causal high-pass The momentum indicator is one subtraction: today's price minus the price from N minus one bars ago. $$ y(n) = x(n) - x\\big(n - (N-1)\\big), \\qquad N > 2 $$ That is a high-pass filter, structurally, the simplest member of the family from the old high-pass article. It subtracts a delayed copy of the price from the present price. Whatever is slow, the trend, is nearly the same value now as N minus one bars ago, so it cancels in the subtraction. Whatever is fast, the recent change, survives. Subtracting a delayed copy is the definition of a first-degree, polynomial-of-degree-one detrender, so momentum kills the trend and keeps the deviation. The strength-gauge intuition is downstream of this: momentum looks big during fast moves because a high-pass passes fast content. ## The frequency response: what it really keeps and kills Take the transfer-function view and the behavior is no longer a story, it is a curve. Put the delay operator on the unit circle and the frequency response of momentum is a complex number you can split into magnitude and phase. $$ H(\\omega) = 1 - e^{-i(N-1)\\omega} = 2\\sin\\!\\left\[\\tfrac{(N-1)\\omega}{2}\\right\] e^{\\,i\\left(\\frac{\\pi}{2} - \\frac{(N-1)\\omega}{2}\\right)} $$ The magnitude is twice the sine of (N minus one) times omega over two. Read it at the ends. At zero frequency, the trend, the magnitude is zero: momentum annihilates the DC level, which is what makes it a detrender. As frequency rises the magnitude climbs, so momentum amplifies faster cycles, up to a peak. Past that peak the sine turns over and the response develops nulls and lobes, the same sinc-shaped sidelobe structure the old article ["Why the SMA Is Often a Terrible Smoother"](https://aligrithm.com/why-the-sma-is-often-a-terrible-smoother/) warned about, because momentum is an N-tap difference, not a clean filter. So momentum is not a uniform strength gauge. It has zeros at specific cycle periods, where it is blind, and a passband centered on a particular speed set by N. A momentum reading near a null is not weak strength; it is a cycle the filter cannot see. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## The phase lag is structural and computable The exponential factor carries the timing, and it splits into two parts. There is a constant phase lead of pi over two, the quarter-cycle advance a differencer gives, and a frequency-dependent phase lag. $$ \\text{phase lag} = \\frac{(N-1)\\,\\omega}{2} $$ That lag grows with the lookback N and with the cycle speed omega, so a longer momentum reports turns later, and it reports fast cycles later than slow ones. This is not an accident of implementation, it is in the transfer function, computable before you ever run it, which is exactly what the old article ["How to Think About Indicator Lag Before Backtesting"](https://aligrithm.com/how-to-think-about-indicator-lag-before-backtesting/) demanded: know each indicator's lag from the formula and add it into the budget. Momentum's contribution is (N minus one) times omega over two at the cycle you care about. Two practical consequences fall out. First, choosing the momentum lookback N is not choosing a sensitivity, it is choosing a passband center and a lag together, and you cannot move one without the other. A short N passes fast cycles with little lag but rejects the slow swings you may want; a long N reaches slower cycles but lags them more and plants its blind nulls at periods that may matter. Second, stacking momentum with another high-pass detrender is not confirmation; if they share a band they are one filter counted twice, the redundancy the frequency-response article kept flagging. Pick the momentum lookback by the cycle band you want to isolate and the lag you can afford, both of which the frequency response hands you in closed form, not by which number looked good on a chart. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_231-visual-0.png) ## KEY POINTS - Momentum, price now minus price N minus one bars ago, is a first-degree causal high-pass filter: it subtracts a delayed copy of price, so the slow trend cancels and the fast deviation survives. It is the same object as the detrenders in the old article ["High-Pass Filters for Traders."](https://aligrithm.com/high-pass-filters-for-traders/) - Its frequency-response magnitude is 2*sin\[(N-1)*omega/2\]: zero at DC (it kills the trend), rising to a passband peak, then nulls and sidelobes, the sinc-style leakage of any N-tap difference. - So momentum is not a uniform strength gauge. It has blind spots (zeros) at specific cycle periods; a reading near a null is a cycle the filter cannot see, not weakness. - The phase lag is (N-1)\*omega/2, computable from the formula, growing with the lookback N and the cycle speed. This is the lag-before-backtest discipline of the old article ["How to Think About Indicator Lag Before Backtesting"](https://aligrithm.com/how-to-think-about-indicator-lag-before-backtesting/) applied to momentum. - Choosing N is choosing a passband center and a lag together; you cannot tune sensitivity without moving both, and you plant blind nulls at periods set by N. - Stacking momentum with another high-pass that shares its band is one filter counted twice, not confirmation. Pick N by the target cycle band and affordable lag, both given in closed form. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Statistically Sound Indicators for Financial Market Prediction - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Statistically+Sound+Indicators+for+Financial+Market+Prediction+Timothy+Masters&ref=aligrithm.com) - [Cycle Analytics for Traders - John Ehlers (Amazon)](https://www.amazon.com/s?k=Cycle+Analytics+for+Traders+John+Ehlers&ref=aligrithm.com) ### 2.63 The Normalized Moving-Average Difference URL: https://aligrithm.com/the-normalized-moving-average-difference/ Last updated: 2026-07-27T13:23:24.000Z Subtract a long moving average from a short one and you get the oldest trend indicator there is, the bones of the MACD and a hundred crossover systems. You also get a number whose variance swings by an order of magnitude between calm and volatile periods, which makes it nearly worthless as a model feature in its raw form. The old article ["Why ATR Normalization Is More Than a Volatility Trick"](https://aligrithm.com/why-atr-normalization-is-more-than-a-volatility-trick/) showed that the right denominator is not a cosmetic rescale but a structural match to how the numerator was generated. The moving-average difference is where that idea pays off most cleanly, because the noise contaminating it comes from a random walk whose variance you can write down. ## Why the raw difference is unusable The short MA minus the long MA measures how far recent price has pulled away from its slower baseline. The trouble is the size of that gap depends on the volatility regime far more than on the trend. In a quiet market the two averages hug each other and the difference is tiny; in a violent one the same trend strength produces a difference five or ten times larger. Feed that into a model and the feature spends most of its variance encoding which year it is, not what the trend is doing, which is the non-stationarity the old ATR article was built to kill. There is a second, subtler source of noise. The two moving averages have centers of mass at different points in the past, and the price has to random-walk from one center to the other. That walk injects error variance into the difference that has nothing to do with trend, and crucially, the amount of it scales with the distance between the two centers. ## Normalize by the random walk that contaminates you The variance of a random walk's net change from start to finish equals the number of steps times the per-step variance. So if you know how many bars separate the center of the long MA from the center of the short MA, you know how much walk-noise is baked into the difference, and the square root of that distance is the right multiplier on a per-bar volatility unit to cancel it. ATR is that per-bar unit, the structurally correct one from the old ATR article because it lives in the instrument's own price units and scales with the same noise the numerator does. $$ \\text{NormDiff} = \\frac{\\text{SMA}\_{\\text{short}} - \\text{SMA}\_{\\text{long}}}{\\text{ATR}\\,\\sqrt{\\big(0.5(L-1) + \\text{lag}\\big) - 0.5(S-1)}} $$ Read the denominator carefully because every piece earns its place. The terms L and S are the long and short lookbacks, and the center of mass of a simple moving average of length k sits at 0.5(k-1) bars back. The expression inside the square root is the distance between the two centers: the long MA's center, pushed back by its lag term, minus the short MA's center. ATR supplies the per-bar volatility, and the square root of the inter-center distance scales it to the exact amount of random-walk noise the difference carries. Divide by that product and the result has roughly constant variance across regimes, which is the entire goal. ## Lag the long average so the two windows do not overlap The lag term in that formula is the third parameter, and it is the part most people leave at zero. By default both moving averages include the current bar as their most recent point, so the long MA is partly tracking the same recent movement the short MA is built to capture. The two windows overlap, and the difference double-counts the present. Lag the long MA backward by an amount equal to the short MA's lookback and the windows become contiguous but non-overlapping: the short MA covers the recent stretch, the long MA covers the stretch before it, and the difference cleanly compares now against before. $$ \\text{MADiff} = 100 \\cdot \\Phi\\big(1 + c\\,\\text{NormDiff}\\big) - 50 $$ The last step tames the tails. Even after volatility normalization a huge price swing can throw the difference into a fat-tailed extreme, so push it through the normal cumulative distribution function, written here as the capital phi. The constant c sets how hard you compress, the normal CDF maps the bell-shaped NormDiff onto a bounded range while leaving its near-linear middle mostly intact, and the final scaling recenters the output around zero. This mirrors the construction logic of the old CMMA article almost exactly: log-clean the numerator, divide by ATR times the square root of the relevant lookback distance to cancel both volatility regime and lookback dependence, then optionally squash the tails. CMMA does it for close-minus-average; this does it for short-average-minus-long-average. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_256-visual-0.png) ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## What you traded away The normalized difference is a stationary, cross-instrument-comparable trend feature, and the normalization is not free. Dividing by ATR times the root-distance removes the volatility regime and the lookback dependence, but it also strips the indicator of any sense of absolute move size, so a two-ATR pull in a calm market and a two-ATR pull in chaos read the same, which is the point and also a real information loss you accepted. The lag parameter, the compression constant c, and the two lookbacks are all knobs, and every knob is a chance to fit noise, so set them from the timescale you intend to trade rather than from whatever maximizes the backtest. The ATR window must be much longer than the lookbacks and must exclude the current bar, or you reintroduce the lookahead the old ATR article warned about and inflate every metric that follows. A beautifully normalized difference is still a trend feature that has to prove predictive value on your instrument before it is worth the trouble. ## KEY POINTS - Short MA minus long MA is the classic trend gap, but its raw variance swings tenfold between calm and volatile regimes, so the feature encodes the era instead of the trend. - A second noise source is the random walk price takes between the two MA centers; its variance equals the bar-distance between the centers, so the square root of that distance is the correct multiplier on ATR. - Divide the difference by ATR times the square root of the inter-center distance to get a roughly variance-stationary measure, using ATR as the structurally correct per-bar denominator from the old ATR-normalization article. - Lag the long MA backward by the short lookback so the two windows are contiguous but non-overlapping, which stops the difference from double-counting the current bar. - Compress the result with the normal CDF to bound the tails and recenter at zero. This is the same build pattern as the old CMMA article: clean numerator, cancel volatility and lookback with ATR times a root-length, then squash tails. - Normalization discards absolute move size; the lag, compression, and lookbacks are tunable knobs to set from the trade timescale, and the ATR window must exclude the current bar to avoid lookahead. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Statistically Sound Indicators for Financial Market Prediction - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Statistically+Sound+Indicators+for+Financial+Market+Prediction+Timothy+Masters&ref=aligrithm.com) - [Cycle Analytics for Traders - John Ehlers (Amazon)](https://www.amazon.com/s?k=Cycle+Analytics+for+Traders+John+Ehlers&ref=aligrithm.com) ### 2.40 Wave Velocity and Acceleration: Reading When the Market Runs Out of Gas URL: https://aligrithm.com/wave-velocity-and-acceleration-reading-when-the-market-runs-out-of-gas/ Last updated: 2026-07-27T13:23:11.000Z The old article ["Why Moving Averages Can Lie at Turning Points"](https://aligrithm.com/why-moving-averages-can-lie-at-turning-points/) showed that a moving average reports the turn late, because its slope flips bars after the price does. The old article ["The Hidden Cost of Every Moving Average: Lag"](https://aligrithm.com/the-hidden-cost-of-every-moving-average-lag/) priced that delay. Both leave a question open: is there a way to read the turn early, from the shape of the move itself? If you accept the local sine model from the previous article, the answer is yes. The sine has an exact derivative, so its velocity and acceleration are available with no smoothing and no lag, and the acceleration is the cleanest early read on a market running out of gas. The catch is that all of it inherits the sine model's fragility. ## Velocity and acceleration come free from the sine model Once you model price locally as a sine, you do not estimate its slope numerically; you differentiate the model. The first derivative is the velocity, the rate price is moving. $$ v = A\\,\\omega \\cos(\\omega t + \\varphi) $$ The second derivative is the acceleration, the rate the velocity itself is changing. $$ a = -A\\,\\omega^2 \\sin(\\omega t + \\varphi) = -\\omega^2 (x - D) $$ Read what the acceleration says. It is the price swing flipped in sign and scaled by frequency squared. When price is stretched far above its center, acceleration points hard down, pulling back; when price is far below, it points hard up. Acceleration is largest in magnitude at the extremes and zero as price crosses its center at full speed. That is the signature of a market losing momentum at the edges of a swing, which is the early-warning the lagging moving average could never give. ## No phase lag, unlike a numerical slope Here is the property that makes this worth the trouble. Differentiating an analytic model introduces no phase lag. The velocity and acceleration are evaluated at the current bar from the fitted sine, not built by differencing past bars, so they do not trail the price the way a numerical derivative or a polynomial-regression slope does. Don Mak's point is direct: computing velocity and acceleration from the sine model gives a reading with no phase or lag shift, which makes it more accurate than fitting a polynomial and differentiating that. The contrast matters. A polynomial-regression slope, or any finite-difference derivative, is a weighted sum of past bars, so it carries the same structural lag as every causal filter and it amplifies noise on top. The sine model sidesteps the lag by committing to a functional form and reading its exact derivative at the present bar. You trade a model assumption for the elimination of differencing lag. When the assumption holds, that is a genuinely better derivative. When it does not, you have an exact derivative of the wrong function. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Acceleration as an exhaustion signal, with the model's fine print The practical payoff is an exhaustion read. The market can turn down, catch a second wind, and charge back up, and price alone or a lagging average will not tell you which is happening until it is over. Acceleration can, because it measures whether the move is still gaining or already fighting itself. Acceleration opposing velocity, growing in magnitude while price extends, says the swing is decelerating into exhaustion, a turn forming before the price confirms it. Acceleration aligned with velocity says the move still has fuel. This is a leading read on the turn, the thing the old turning-point article said moving averages structurally cannot deliver. The fine print is the whole sine model, and it does not get smaller because we took derivatives. Every value here depends on omega, A, and phi estimated locally, and the previous article showed that local frequency estimate is high-variance and explodes near turns, which is exactly where you want to read acceleration. So the exhaustion signal is sharpest in theory at the moment it is least reliable in practice. Differentiating twice multiplies the sensitivity: acceleration carries the factor omega squared, so any error in the frequency estimate is amplified hard. The discipline is the same as before. Gate the signal on a real, persistent cycle with adequate amplitude, smooth the inputs enough to survive the noise without giving back all the lead, and treat a clean acceleration reversal in a trending or choppy regime as an artifact of a model that does not fit, not as a signal. Used inside its assumptions it is a rare leading indicator. Used outside them it is confident nonsense. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_230-visual-0.png) ## KEY POINTS - If you accept the local sine model, velocity and acceleration are exact derivatives of the model, not numerical differences of past bars, so they come with no smoothing and no differencing lag. - Velocity is A*omega*cos: largest as price crosses its center at full speed. Acceleration is minus omega-squared times the swing (price minus center): largest at the extremes, opposing the move. - Differentiating an analytic model adds no phase lag, unlike a polynomial-regression slope or any finite-difference derivative, which lag and amplify noise. You trade a model assumption for the removal of differencing lag. - Acceleration is an exhaustion read: opposing and growing while price extends means the swing is decelerating into a turn, a leading signal the old article ["Why Moving Averages Can Lie at Turning Points"](https://aligrithm.com/why-moving-averages-can-lie-at-turning-points/) said averages cannot give. - The whole thing rides on the sine fit's local frequency estimate, which the previous article showed is high-variance and worst near turns, the exact place you want acceleration. The omega-squared factor amplifies any frequency error. - Use it gated on a persistent cycle with real amplitude, smoothed enough to survive noise; treat a clean acceleration reversal in a trend or chop as a model artifact, not a signal. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Statistically Sound Indicators for Financial Market Prediction - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Statistically+Sound+Indicators+for+Financial+Market+Prediction+Timothy+Masters&ref=aligrithm.com) - [Cycle Analytics for Traders - John Ehlers (Amazon)](https://www.amazon.com/s?k=Cycle+Analytics+for+Traders+John+Ehlers&ref=aligrithm.com) ### 2.62 A Statistically Sound Stochastic and Stochastic RSI URL: https://aligrithm.com/a-statistically-sound-stochastic-and-stochastic-rsi/ Last updated: 2026-07-27T13:23:23.000Z The raw stochastic oscillator has a defect almost nobody who uses it has looked at, and it is not in the formula. The formula is fine. The problem is the denominator quietly jumps every time the lookback window rolls forward, and that jump injects noise that has nothing to do with where the close actually sits. The old article ["How to Build Stationary Indicators from Non-Stationary Prices"](https://aligrithm.com/how-to-build-stationary-indicators-from-non-stationary-prices/) argued that most indicator failures are a moving reference frame masquerading as signal. The stochastic is a textbook case, and the repair is two cheap smoothing passes. ## What the stochastic actually measures George Lane's idea is simple and worth keeping: look back a fixed number of bars including the current one, find the highest high and lowest low in that window, and report where the current close sits inside that range. $$ \\text{Raw Stochastic} = 100 \\cdot \\frac{\\text{close} - \\text{low}\_n}{\\text{high}\_n - \\text{low}\_n} $$ The terms low and high are the lowest and highest prices over the n-bar window, so a close at the bottom of the range reads 0, a close at the top reads 100, and everything else lands in between by position. As a description of "where are we in the recent range" it is honest and bounded, which is more than raw price offers. ## The rolling-range jump is the bug Watch the denominator as the window advances one bar. You drop the oldest bar and add a new one. If the bar that fell off the back happened to be the window's high or low, the entire range snaps to a new value in a single step, and the raw stochastic lurches even when the current close barely moved. The oscillator is reacting to a bookkeeping event, the expiry of an old extreme, not to anything price did on this bar. That is the non-stationarity the old stationary-indicators article warned about, sitting inside an indicator most people trust as-is. The reference frame, the high-low range, shifts discontinuously under the measurement, so any genuine information about the close's position gets drowned by the noise of a denominator that keeps redefining itself. Read the raw stochastic on a fast lookback and you see it: a jagged line slamming between extremes, far more violent than the underlying price warrants. ## Smooth once, smooth twice, center at fifty The repair is to smooth the raw output with an EMA-style pass, and if it is still too jumpy, smooth the smoothed version again. $$ S\_t = 0.66667\\,S\_{t-1} + 0.33333\\,X\_t $$ The term X is the raw stochastic on the current bar, S is the smoothed value, and the two coefficients are a fixed exponential blend, roughly a three-bar effective average. One pass absorbs most of the range-expiry jumps because a single discontinuous step in the input becomes a gentle ramp in the output. A second pass, applied to the first pass's output, kills what survives, at the cost of a little more lag. As with every smoother, you are buying steadiness with delay, so stop at two passes unless you have a reason to pay more. For a predictive model, subtract 50 so the indicator centers at zero, the same recentering the RSI wanted, because a feature that lives on 0 to 100 with a midpoint of 50 hands a linear model a baseline it has to learn instead of one you can give it for free. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Stochastic RSI: stack one bounded oscillator on another The stochastic RSI is the hybrid you get by running the stochastic formula not on price but on the RSI series. Compute the RSI, then treat the RSI values as the input to the stochastic: find the highest and lowest RSI over the lookback and locate the current RSI inside that band. $$ \\text{StochRSI}\_t = 100 \\cdot \\frac{\\text{RSI}\_t - \\min\_n(\\text{RSI})}{\\max\_n(\\text{RSI}) - \\min\_n(\\text{RSI})} $$ The terms are the same as the price stochastic with RSI swapped in for price. The payoff is responsiveness: RSI already compresses price into a bounded range, and asking where today's RSI sits within its own recent range amplifies the moments when RSI itself is stretching to a local extreme, so the StochRSI reacts faster to changing conditions than either parent alone. It inherits the same disease, though. The RSI's rolling min and max jump on window expiry exactly like price's do, so the raw StochRSI gyrates just as wildly and needs the same one-or-two-pass smoothing before it is fit to use. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_255-visual-0.png) ## Use it knowing what you smoothed away The smoothed stochastic is a cleaner cycle-position reader than the raw version, and it is still a bounded oscillator, not a forecast. Each smoothing pass adds lag, so the two-pass version will confirm a turn later than the raw output that whipsawed you, which is the trade you chose when you stopped the noise. The lookback sets the cycle length the oscillator is tuned to, and a length that fits one regime will mismatch another, so it is a parameter to set deliberately and re-examine, not a constant. The StochRSI buys faster response by compounding two bounded transforms, and it pays for that with a sharper temper, so the smoothing is not optional there, it is the price of admission. Bounded and centered does not mean predictive: a clean stochastic still has to earn an edge on your instrument before you trust it, which is the empirical test no amount of smoothing can pass on its behalf. ## KEY POINTS - The stochastic reports where the close sits inside the recent high-low range, bounded 0 to 100\. The formula is sound; the denominator is the problem. - As the window rolls, dropping an old high or low snaps the range to a new value in one step, so the raw stochastic lurches on a bookkeeping event rather than on price. This is the non-stationary reference frame from the old stationary-indicators article. - Fix it by smoothing the raw output once with an EMA-style blend, twice if it is still jumpy; a discontinuous input step becomes a gentle ramp. Subtract 50 to center it for modeling. - The stochastic RSI runs the stochastic formula on the RSI series instead of price, giving faster response because it flags when RSI itself stretches to a local extreme. It inherits the same range-expiry jumpiness and needs the same smoothing. - Each smoothing pass costs lag, the lookback is a deliberate parameter tied to cycle length, and a clean bounded oscillator still has to prove a real edge before you trade it. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Statistically Sound Indicators for Financial Market Prediction - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Statistically+Sound+Indicators+for+Financial+Market+Prediction+Timothy+Masters&ref=aligrithm.com) - [Cycle Analytics for Traders - John Ehlers (Amazon)](https://www.amazon.com/s?k=Cycle+Analytics+for+Traders+John+Ehlers&ref=aligrithm.com) ### 2.39 Modeling Price as a Sine Wave: Instantaneous Frequency from 4 Points URL: https://aligrithm.com/modeling-price-as-a-sine-wave-instantaneous-frequency-from-4-points/ Last updated: 2026-07-27T13:23:09.000Z The old article ["Dominant Cycle Estimation Without Astrology"](https://aligrithm.com/dominant-cycle-estimation-without-astrology/) measured the dominant cycle with autocorrelation and periodograms, methods that carry 12 to 32 bars of lag because they need a long window to resolve a frequency. This article shows the opposite extreme: assume the price is locally one sine wave and you can solve for its frequency from just four bars, with almost no lag. The math is exact and the appeal is obvious. The danger is equally obvious once you remember the old article ["Why Market Cycles Are Evanescent"](https://aligrithm.com/why-market-cycles-are-evanescent/): a four-bar frequency estimate is only as real as the assumption that four bars actually contain a clean cycle, and most of the time they do not. ## One dominant cycle, four unknowns The model is the simplest thing that can oscillate: a single sine wave sitting on a constant level. $$ x\_t = A \\sin(\\omega t + \\varphi) + D $$ Four unknowns live here: the amplitude A, the frequency omega, the phase phi, and the DC offset D. Four unknowns need four equations, which means four data points, the last four bars. Solve the system and you get omega, and from omega the period falls out as two pi over omega. The frequency you recover this way is the instantaneous frequency: how fast the market is oscillating right now, measured locally, not averaged over a long window. That locality is the whole selling point, because a long-window estimate tells you what the cycle was, while this tells you what it is, if the assumption holds. The classical route to instantaneous frequency is the Hilbert transform, which builds in-phase and quadrature components and reads the rate of phase change. It works but it smooths, so it carries delay. The four-point solve is the low-lag alternative. ## Difference away the offset, and the frequency falls out The clean way to solve the system is to kill the unknowns you do not want. The DC offset D is a nuisance, so difference the series: consecutive differences of a sine are themselves a sine with the same frequency and no constant term. A pure sinusoid obeys a three-term recurrence, each sample is a fixed multiple of the average of its two neighbors, and that multiple is the cosine of the frequency. Apply it to the differenced series and the frequency comes out in closed form from four raw bars. $$ \\cos\\omega = \\frac{(x\_1 - x\_0) + (x\_3 - x\_2)}{2\\,(x\_2 - x\_1)}, \\qquad T = \\frac{2\\pi}{\\omega} $$ The numerator and denominator are built from the three first-differences of the four bars, which is why D drops out: differencing removed it, and the ratio removed the amplitude A. What survives is the frequency alone. Take the arccosine, convert to a period, and you have measured the local cycle from four bars with effectively one bar of lag. No autocorrelation window, no periodogram, no 30-bar warmup. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Why this is fragile, and how it fails The elegance hides a brutal sensitivity, and it lives in the denominator. The divisor is x2 minus x1, a single one-bar difference. When price is near a turning point or moving slowly, that difference is tiny, and dividing by a tiny noisy number blows the estimate up: the recovered frequency swings wildly from bar to bar on what is mostly measurement noise. A four-point estimator has no averaging to lean on, so every tick of noise lands directly in omega. The methods from the old dominant-cycle article paid 12 to 32 bars of lag precisely to buy the stability that this one throws away. So the four-point frequency is high-variance by construction, the mirror image of the low-variance, high-lag periodogram. That is not a flaw to patch, it is the trade. The honest deployment follows from it. Smooth the raw four-point estimate before you trust it, which gives back some of the lag you saved but tames the explosions. Gate it on amplitude: when A is small there is no cycle to measure and the number is garbage, so suppress it. And hold the deeper skepticism from the old article ["Why Market Cycles Are Evanescent"](https://aligrithm.com/why-market-cycles-are-evanescent/): the model assumes one dominant clean cycle, and when the market is trending or thrashing there is no such cycle, so a confident four-bar frequency is then a precise measurement of something that is not there. Use it where a cycle demonstrably exists, distrust it everywhere else. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_229-visual-0.png) ## KEY POINTS - Model price locally as one sine on a constant level, with four unknowns (amplitude, frequency, phase, offset). Four unknowns need four bars, so you can solve for the frequency from the last four points. - The result is instantaneous frequency: the cycle speed right now, measured locally, versus the old article ["Dominant Cycle Estimation Without Astrology"](https://aligrithm.com/dominant-cycle-estimation-without-astrology/) which averages over a long, lagging window. - Difference the series to kill the offset D and form a ratio to kill the amplitude A; a sinusoid's three-term recurrence then gives cos(omega) from the four bars, and the period is two pi over omega, with about one bar of lag. - The denominator is a single one-bar difference. Near turns or in slow markets it goes tiny and noisy, so the frequency estimate explodes. The estimator is high-variance by construction, the exact opposite trade from the high-lag periodogram. - Deploy it by smoothing the raw estimate (giving back some lag for stability) and gating on amplitude, since a small A means no cycle and a garbage number. - The old article ["Why Market Cycles Are Evanescent"](https://aligrithm.com/why-market-cycles-are-evanescent/) still binds: the one-clean-cycle assumption fails in trends and chop, where a confident four-bar frequency precisely measures a cycle that is not there. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Statistically Sound Indicators for Financial Market Prediction - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Statistically+Sound+Indicators+for+Financial+Market+Prediction+Timothy+Masters&ref=aligrithm.com) - [Cycle Analytics for Traders - John Ehlers (Amazon)](https://www.amazon.com/s?k=Cycle+Analytics+for+Traders+John+Ehlers&ref=aligrithm.com) ### 2.61 The Detrended RSI: Predicting RSI(2) from RSI(20) URL: https://aligrithm.com/the-detrended-rsi-predicting-rsi-2-from-rsi-20/ Last updated: 2026-07-27T13:23:23.000Z A two-bar RSI is one of the sharpest sudden-move detectors on a chart, and on its own it is close to useless for a model. It fires on every shock, swings to its extremes constantly, and piles its mass at 0 and 100 instead of spreading out like a well-behaved feature. The old article ["CMMA: A Better Momentum Primitive Than Price-minus-MA Alone"](https://aligrithm.com/cmma-a-better-momentum-primitive-than-price-minus-ma-alone/) made the case that the fix for a noisy primitive is not to abandon it but to measure it against the right expectation and keep the residual. The detrended RSI does exactly that to the RSI family: it asks what the fast RSI *should* be given the slow one, and trades the gap. ## Why two RSIs at different lengths carry different information Start with the indicator itself. The RSI compares the average size of recent up-closes against recent down-closes, smoothed with an EMA-style recursion rather than a flat average. $$ S\_t = \\frac{(L - 1)\\,S\_{t-1} + \\text{Value}\_t}{L} \\qquad \\text{RSI} = 100 \\cdot \\frac{\\text{UpSum}}{\\text{UpSum} + \\text{DnSum}} $$ The term L is the lookback, S is the smoothed up- or down-move average, UpSum and DnSum are those smoothed averages of gains and losses, and the ratio lands in 0 to 100\. If you plan to feed RSI into a predictive model, subtract 50 first so it centers at zero. None of that is new. What matters is the lookback. A long RSI, say 20 bars, is a slow gauge of where the market has been leaning over the last month. A short RSI, two bars, reacts to the last two closes and almost nothing else, so it spikes the instant price jumps out of line. The two are correlated because they read the same price stream, and that correlation is the opening the indicator exploits. When a market is marching along a trend and a shock knocks price out of line, the short RSI lurches while the long RSI barely moves. That divergence is the imbalance, and imbalances of this kind tend to revert: price snaps back toward where the slower measure says it belongs. Quantify how far out of line the fast RSI is relative to what the slow RSI predicts, and you have an indicator that runs negatively correlated with short-term returns. ## Regress the fast RSI on the slow one, keep the residual The mechanism is a rolling linear regression. Over an extended window you fit the short RSI as a linear function of the long RSI, then use that fitted line to predict what the short RSI should be on the current bar given the current long RSI. The detrended RSI is the gap between the prediction and reality. $$ \\widehat{\\text{RSI}}\_{2,t} = a + b\\,\\text{RSI}\_{20,t} \\qquad \\text{DetrendedRSI}\_t = \\widehat{\\text{RSI}}\_{2,t} - \\text{RSI}\_{2,t} $$ Here a and b are the intercept and slope fit by ordinary least squares over the rolling window, the hatted term is the short RSI the slow trend implies, and the indicator is prediction minus actual. Read the sign directly. When the actual fast RSI sits far above its prediction, price has jumped up harder than the slow lean justifies, the residual goes negative, and you lean for a reversion down. When the fast RSI is far below prediction, price has been knocked down out of line and the residual goes positive. This is the same move as the old CMMA article: do not trade the raw quantity, trade its departure from a baseline expectation, because the departure is where the conditional edge lives. ## The U-shape problem, and the inverse-logistic fix There is a distributional trap that wrecks the regression if you ignore it, and the old article ["Why Most Indicators Should Be Transformed Before Modeling"](https://aligrithm.com/why-most-indicators-should-be-transformed-before-modeling/) named the exact defect. For a lookback of three or more, RSI has a pleasant bell-ish distribution, a central peak tailing off at both ends. At a lookback of two the shape inverts into a U: the extreme values near 0 and 100 are *more* common than the middle. Regressing a U-shaped variable on a roughly bell-shaped one is a broken setup, because least squares assumes the response spreads symmetrically around its conditional mean, and a U-shaped response violates that on every bar. The fix is to reshape the two-bar RSI before it enters the regression. The inverse-logistic transform stretches the crowded extremes back out toward the middle so the variable behaves more like the normal target the regression wants. $$ \\text{IL}(x) = \\log\\!\\left(\\frac{x}{1-x}\\right) \\qquad \\text{ILRSI}\_2 = -\\log\\!\\left(\\frac{2}{\\,1 + 0.00049\\,(2\\,\\text{RSI}\_2 - 100)\\,}\\right) $$ The first expression is the plain inverse logistic, the inverse of the squashing sigmoid, which pushes values near the 0 and 1 boundaries outward. The second is that transform shifted and scaled so the 0-to-100 RSI lands in the transform's usable domain. The full recipe is short: compute RSI(2) and RSI(20), run the two-bar RSI through the inverse logistic, regress the transformed fast RSI on the slow RSI over the rolling window, and keep predicted minus actual as the signal. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_254-visual-0.png) ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## What it is good for, and what it is not Keep the indicator inside its lane. The detrended RSI is a short-horizon mean-reversion feature, built to flag when price has shocked away from its own recent trend and to bet on the snap back. It says nothing about the direction of the underlying trend, which is the long RSI's job and not this construction's. The regression window is a real parameter with real risk: too short and the fit is noise, too long and it drags stale relationships into the current bar, and like every rolling fit it must be computed from data strictly before the current bar or you have manufactured a lookahead that will not survive live. The inverse logistic is the right transform here only because the two-bar RSI is U-shaped; copy it onto a bell-shaped indicator and you would do harm, which is the whole point of diagnosing distribution shape before transforming rather than reaching for a default. ## KEY POINTS - The RSI compares smoothed average up-moves against down-moves; subtract 50 before modeling. A long RSI (20) is a slow trend baseline, a short RSI (2) reacts to the last two closes and spikes on any shock. - The two are correlated. When a shock knocks the fast RSI far off what the slow RSI implies, that gap tends to revert, giving an indicator negatively correlated with short-term returns. - Build it by rolling-regressing the fast RSI on the slow RSI and keeping predicted minus actual. This is the old CMMA move: trade the departure from a baseline expectation, not the raw quantity. - The two-bar RSI has a U-shaped distribution (extremes more common than the center), which breaks linear regression. Fix it with an inverse-logistic transform that spreads the crowded extremes back toward a bell, exactly the diagnose-then-transform logic from the old article on transforming indicators. - It is a short-horizon mean-reversion feature only; it carries no trend direction. The regression window is a tunable parameter and must be fit causally to avoid lookahead. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Statistically Sound Indicators for Financial Market Prediction - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Statistically+Sound+Indicators+for+Financial+Market+Prediction+Timothy+Masters&ref=aligrithm.com) - [Cycle Analytics for Traders - John Ehlers (Amazon)](https://www.amazon.com/s?k=Cycle+Analytics+for+Traders+John+Ehlers&ref=aligrithm.com) ### 2.38 Causal Wavelet Filters and the Mexican Hat URL: https://aligrithm.com/causal-wavelet-filters-and-the-mexican-hat/ Last updated: 2026-07-27T13:23:09.000Z The old article ["Band-Pass Filters: The Most Underused Tool in Technical Analysis"](https://aligrithm.com/band-pass-filters-the-most-underused-tool-in-technical-analysis/) built band-pass filters with fixed-period recursions to isolate one cycle. The old article ["Dominant Cycle Estimation Without Astrology"](https://aligrithm.com/dominant-cycle-estimation-without-astrology/) measured which cycle to isolate. Wavelets attack the same problem from a different angle: instead of one band-pass tuned to one period, a wavelet is a band-pass whose width scales with the cycle it examines, so a single family of filters dissects the market at every scale at once. The price for that elegance is the same Fourier resolution tradeoff in new clothing, and the Mexican Hat is the cleanest place to see both the power and the cost. ## A wavelet is a band-pass; its scaling function is the low-pass father Wavelet analysis was built for signals of short duration, the bursts and transients that Fourier analysis, which assumes long stationary signals, handles badly. That makes it a natural fit for market data, where structure appears and dissolves rather than ringing forever like a tuning fork. The key fact for a filter-minded trader is plain: a wavelet is a band-pass filter. It passes a band of cycle speeds and rejects the rest, exactly like the recursions from the old band-pass article, but with a shape that stretches and shrinks. Every wavelet has a partner called its scaling function, and the scaling function is a low-pass filter, the father of the family. It keeps everything below a chosen cutoff. The signals it removes, the faster content above the cutoff, are precisely what the wavelets analyze. Think of the price as an onion of twelve layers. Choose a cutoff and you split it into a core of, say, four layers and the eight outer layers. The core is the scaling function, the slow trend you decided to set aside. The outer layers are the wavelets, each a band-pass capturing one scale of cycle. Together, father and wavelets reconstruct the whole onion and tell you which scales carry the action. ## The Mexican Hat, the second derivative of a Gaussian The sinc from the old low-pass work was the ideal brick-wall filter, and its fatal flaw was infinite length: its tails never reach zero, so it needs infinitely many bars. The Mexican Hat is the deliberate opposite, a wavelet built to have compact support, finite reach in time. $$ \\Psi(t) = (1 - 2t^2)\\, e^{-t^2} $$ It is the second derivative of a Gaussian bump, which gives it its shape: a central peak with two symmetric dips on either side, the silhouette that earns the name. Because the Gaussian decays fast, the wavelet effectively dies out over a finite stretch, unlike the sinc that rings on forever. That compact support is the practical advantage: a Mexican Hat filter uses a bounded window of bars, so you can run it without the infinite-tail problem. Its frequency response is a band-pass, a hump that passes a middle band and rejects both the slow trend and the fast noise, with the band's center set by how much you stretch the wavelet. ## Scale is the dial, and it does the cycle scan for you The reason a wavelet beats a single fixed band-pass is the stretch parameter, the scale. Stretch the Mexican Hat wide and it becomes a band-pass centered on a slow cycle. Squeeze it narrow and the same shape becomes a band-pass centered on a fast cycle. Run a whole range of scales and you get a band-pass at every period at once, a scan across the cycle spectrum from one wavelet shape. Where the old band-pass article fixed the center period by hand and the dominant-cycle article measured one period to target, the wavelet sweeps all of them and shows you where the energy sits across scales and across time together. That is the genuine edge for market data: a wavelet scan reveals that a 20-bar cycle was strong last month and a 40-bar cycle dominates now, localized in time, which a single Fourier spectrum smears together. For the econophysics and geometry questions this sits next to, that scale-by-scale view is the right lens for self-similar, multi-scale price structure. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## The cost is the resolution tradeoff, again None of this repeals the law that the dominant-cycle and evanescent-cycle articles kept enforcing. A wavelet faces the same uncertainty tradeoff as every spectral tool: a filter narrow in time is wide in frequency, and a filter narrow in frequency is wide in time. The Mexican Hat's compact support, the thing that makes it usable, is exactly what blurs its frequency selectivity, so it tells you when a cycle burst happened with decent timing but smears which precise period it was. Squeeze it for sharp timing and the band gets broad; stretch it for a sharp band and the timing smears. You do not escape the tradeoff by switching from a recursion to a wavelet; you relabel it. And the deeper caution from the old article ["Why Market Cycles Are Evanescent"](https://aligrithm.com/why-market-cycles-are-evanescent/) applies in full. A wavelet scan can show you a beautiful localized cycle that was real and is already gone, because market cycles drift in period, decay in amplitude, and lose phase coherence. The wavelet is a sharper microscope for a substrate that keeps dissolving under the lens. Use it to characterize structure that exists, gate any cycle-mode logic on the structure persisting, and never mistake a clean wavelet ridge for a promise about the next bar. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_228-visual-0.png) ## KEY POINTS - A wavelet is a band-pass filter, like the recursions in the old article ["Band-Pass Filters: The Most Underused Tool in Technical Analysis,"](https://aligrithm.com/band-pass-filters-the-most-underused-tool-in-technical-analysis/) but its width scales, so one family dissects the market at every cycle scale at once. - Every wavelet has a scaling function, its low-pass father, which keeps the slow trend below a chosen cutoff; the content it removes is what the wavelets analyze. The onion picture: core equals scaling function, outer layers equal wavelets. - The Mexican Hat, the second derivative of a Gaussian, has compact support (finite reach in time), the deliberate opposite of the infinite-length sinc, so it is runnable as a bounded-window band-pass. - Scale is the dial: stretch the wavelet for a slow-cycle band, squeeze it for a fast-cycle band. Sweeping scales scans all periods at once and localizes when each cycle was strong, which a single Fourier spectrum smears. - The cost is the resolution tradeoff in new clothing: sharp timing means a broad band, sharp band means smeared timing. Compact support buys usability by blurring frequency selectivity. - The old article ["Why Market Cycles Are Evanescent"](https://aligrithm.com/why-market-cycles-are-evanescent/) still rules: a clean wavelet ridge can be a real cycle that already died. Gate cycle-mode logic on persistence; never read a ridge as a forecast. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Statistically Sound Indicators for Financial Market Prediction - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Statistically+Sound+Indicators+for+Financial+Market+Prediction+Timothy+Masters&ref=aligrithm.com) - [Cycle Analytics for Traders - John Ehlers (Amazon)](https://www.amazon.com/s?k=Cycle+Analytics+for+Traders+John+Ehlers&ref=aligrithm.com) ### 2.37 Cubic-Velocity Modified EMA and Skipped Convolution URL: https://aligrithm.com/cubic-velocity-modified-ema-and-skipped-convolution/ Last updated: 2026-07-27T13:23:08.000Z The old article ["Zero-Lag EMA: The Kalman Filter, Simplified"](https://aligrithm.com/zero-lag-ema-the-kalman-filter-simplified/) cut the EMA's lag by adding a velocity term to the price before smoothing, where velocity was a simple slope. This article pushes the same idea harder and then shows the wall it hits. Estimate velocity from a cubic fit instead of a straight line, and you get a filter that turns faster and lags less than a plain EMA. But the same aggressiveness that buys responsiveness destroys the one property a trend filter is supposed to have: this thing is no longer a low-pass, so its output is not smooth, and pretending otherwise is how you end up trading noise. ## The modified EMA: smooth a velocity-corrected price Start from the modified EMA, the structure behind the zero-lag family. It is an ordinary EMA fed a price that has been nudged forward by an estimate of its own velocity. $$ \\text{MEMA}\[n\] = \\alpha\\,\\big(x\[n\] + V\[n\]\\big) + (1-\\alpha)\\,\\text{MEMA}\[n-1\], \\qquad \\alpha = \\frac{2}{M+1} $$ The term V is the estimated velocity, the local slope of price, and adding it pushes the input toward where the slope says the next bar is heading. The old article ["The Hidden Cost of Every Moving Average: Lag"](https://aligrithm.com/the-hidden-cost-of-every-moving-average-lag/) explained why a plain average trails price: its weight sits behind the present. The velocity term offsets that backward pull, so the more accurate V is, the more lag cancels. The whole game now reduces to one question, how do you measure V, and the answer is where the cubic comes in. ## Cubic velocity: fit four points, take the slope A straight-line velocity assumes price moves at constant speed over the lookback, which is wrong whenever the trend bends, which is when you care. Fit the last four bars with a cubic polynomial instead and read its slope, and you capture curvature. The slope of a four-point cubic fit is a fixed weighted combination of those four bars. $$ V\[n\] = \\tfrac{11}{6}\\,x\[n\] - 3\\,x\[n-1\] + \\tfrac{3}{2}\\,x\[n-2\] - \\tfrac{1}{3}\\,x\[n-3\] $$ These coefficients are the standard backward-difference weights for a first derivative, accurate to a higher order than a two-point slope. Feed this V into the modified EMA and the result, the cubic-velocity modified EMA, responds faster than the plain EMA and carries less phase lag, because the cubic tracks a bending trend that a linear slope misreads. So far this looks like a clean win. ## Skipped convolution: spread the stencil to tame the jitter The cubic velocity has a vice: it is computed from four adjacent bars, so it amplifies high-frequency noise, the same way any tight differencing stencil does. Don Mak's fix is skipped convolution, which estimates velocity from past prices spaced apart rather than packed together. Skip one bar between taps and the velocity stencil spans more time for the same number of points, which smooths the slope estimate. Skip two bars and it smooths further. The idea comes straight from convolution: a filter is a weighted sum of past prices, and spacing the taps changes the filter's frequency response without adding taps or cost. Skipping trades a little timing precision for a steadier velocity, which steadies the whole modified EMA. The dial is the skip distance: skip zero is the raw, jittery cubic; skip one calms it; skip two calms it more at the price of reaching further back. You tune the skip to the noise of your instrument, not to a backtest's flattering curve. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## The catch: this is not a low-pass, so it is not smooth Here is the property that the responsiveness costs you, and it is not negotiable. A trend filter is supposed to be a low-pass: it keeps the slow trend and kills the fast noise. The cubic-velocity modified EMA is not a low-pass. The velocity correction, being a derivative, has high gain at high frequencies, so it lets fast content through, and the output is not smooth. You bought a faster turn and a lower lag by giving up the noise rejection that was the point of smoothing in the first place. That trade is fine if you know you made it and fatal if you do not. Used as a fast trend-state feature with the noise handled downstream, the cubic-velocity modified EMA earns its lower lag. Used as a clean smoother, it feeds your strategy exactly the high-frequency jitter a smoother exists to remove, and the lower lag is cold comfort when the line is rough enough to whipsaw you. And like every velocity-based lag reducer, the derivative term extrapolates the recent slope, so it overshoots at reversals, the failure the turning-point literature keeps returning to. Treat it as what it is: a reduced-lag, reduced-smoothness filter, not a free upgrade to the EMA. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_227-visual-0.png) ## KEY POINTS - The modified EMA smooths a velocity-corrected price (price plus estimated slope), the same lag-cancelling trick as the old article ["Zero-Lag EMA: The Kalman Filter, Simplified."](https://aligrithm.com/zero-lag-ema-the-kalman-filter-simplified/) The accuracy of the velocity estimate is the whole game. - Cubic velocity fits the last four bars with a cubic and reads its slope, with fixed weights 11/6, -3, 3/2, -1/3\. It captures a bending trend that a straight-line slope misreads, so it turns faster and lags less than a plain EMA. - A tight four-bar derivative amplifies noise. Skipped convolution spaces the velocity taps apart (skip 1, skip 2) to smooth the slope estimate without adding taps or cost; the skip distance is the dial. - The fatal property: a derivative has high gain at high frequency, so the cubic-velocity modified EMA is not a low-pass and its output is not smooth. The lower lag is bought by surrendering noise rejection. - It is useful as a fast trend-state feature with noise handled downstream, and dangerous if mistaken for a clean smoother, because then it feeds your model the jitter a smoother exists to remove. - Like every velocity-based reducer it extrapolates the recent slope, so it overshoots reversals. Treat it as a reduced-lag, reduced-smoothness filter, not a free EMA upgrade. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Statistically Sound Indicators for Financial Market Prediction - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Statistically+Sound+Indicators+for+Financial+Market+Prediction+Timothy+Masters&ref=aligrithm.com) - [Cycle Analytics for Traders - John Ehlers (Amazon)](https://www.amazon.com/s?k=Cycle+Analytics+for+Traders+John+Ehlers&ref=aligrithm.com) ### 2.36 Zero-Lag EMA: The Kalman Filter, Simplified URL: https://aligrithm.com/zero-lag-ema-the-kalman-filter-simplified/ Last updated: 2026-07-27T13:23:08.000Z The old article ["The Hidden Cost of Every Moving Average: Lag"](https://aligrithm.com/the-hidden-cost-of-every-moving-average-lag/) established the rule no smoother escapes: averaging the past reports the past, so the output trails price by a structural delay you pay in money and missed entries. The old article ["Why Moving Averages Can Lie at Turning Points"](https://aligrithm.com/why-moving-averages-can-lie-at-turning-points/) showed where that lag does the most damage, at the turn, when a moving average is still reporting the regime that just ended. The zero-lag EMA is the most honest attempt to cut that delay, and it works by stealing one idea from the Kalman filter: do not just smooth the price you have, add a guess about where it is going next. ## Smooth the forecast, not the price A plain EMA smooths the current price. The Kalman filter, the optimal estimator Kalman built for navigation, does more: it tracks both position and velocity, and uses the velocity to predict the next position before the next measurement arrives. Ehlers stripped that idea down to a single line that keeps the prediction and drops the heavy machinery. $$ \\text{ZEMA}\[n\] = \\alpha\\,(\\text{Price} + \\kappa V) + (1-\\alpha)\\,\\text{ZEMA}\[n-1\], \\qquad \\alpha = \\frac{2}{M+1} $$ The structure is a normal EMA, with one substitution. Where a plain EMA feeds in the raw price, the zero-lag EMA feeds in the price plus a velocity term, kappa times V, where V is the estimated velocity, the recent slope, and kappa is a tunable gain that scales how far ahead you lean. The quantity Price plus kappa-V is the predicted next price: take where you are, add where the slope says you are going. Smooth that forecast instead of the bare price and the output sits closer to the live price than a same-length EMA, because the velocity term cancels part of the delay that the averaging introduces. It is the Kalman filter's predict step without the covariance bookkeeping. ## Why the velocity term cancels lag Lag exists because an average is weighted toward older bars, so its center of mass sits behind the present. Adding kappa times the slope nudges the input forward by an amount proportional to how fast price is moving, which offsets the backward pull of the average. In a steady trend the slope is roughly constant, the forward nudge roughly matches the backward lag, and the two cancel, so the zero-lag EMA hugs the trend where a plain EMA trails it. Turn kappa up and you lean further ahead and cut more lag; turn it down toward zero and the whole thing collapses back into an ordinary EMA. The name oversells it, the way every zero-lag name does: this is lag reduction, not lag elimination, and the residual delay is real, just smaller. It is worth naming what the filter is, structurally. The zero-lag EMA is recursive, an infinite impulse response filter, because it feeds its own past output back in, so it carries infinite memory from one stored number like any EMA. That makes it cheap, one state variable, a few multiplies per bar, which is why it can run on streaming data without the matrix work a full Kalman filter demands. You get most of the Kalman predict benefit at a fraction of the Kalman cost. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## The trap is exactly the turning point The velocity term is a blessing in a trend and a curse at a turn, and the old article ["Why Moving Averages Can Lie at Turning Points"](https://aligrithm.com/why-moving-averages-can-lie-at-turning-points/) already told you why. The forward nudge assumes the recent slope continues. While price trends, that assumption holds and the lag cancels. At a reversal the slope is about to flip, so the velocity term points the prediction the wrong way, leaning the filter further into the old direction at the precise moment the market changes its mind. The zero-lag EMA overshoots the turn and can fire a confident signal in the dying direction. This is the same failure that haunts every prediction-based lag reducer, including the Kalman models it imitates: they buy responsiveness in trends by extrapolating slope, and they pay for it at reversals, which is when you most wanted to be right. A larger kappa cuts more lag in trends and overshoots harder at turns, so the gain is not a free dial, it is a slider between trend-tracking and turn-safety. The practical rules follow directly. Tune kappa to the smallest value that gives acceptable trend lag, not the largest that looks impressive on a trending backtest. Do not use the zero-lag EMA as a turning-point detector, because that is its worst job. And whatever lag you measured on trending history, expect it to reappear as overshoot at the next reversal, because the same velocity term produces both. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_226-visual-0.png) ## KEY POINTS - The zero-lag EMA cuts the structural delay from the old article ["The Hidden Cost of Every Moving Average: Lag"](https://aligrithm.com/the-hidden-cost-of-every-moving-average-lag/) by stealing the Kalman filter's predict step: smooth a forecast of the next price, not the current price. - The forecast is Price plus kappa times V, where V is estimated velocity (recent slope) and kappa is a tunable gain. Feed that into a normal EMA recursion and the output hugs price closer than a same-length EMA. - The velocity term cancels lag because the forward nudge offsets the average's backward center of mass. In a steady trend the two roughly cancel; kappa scales how far ahead you lean. - It is a recursive (infinite impulse response) filter: one stored state, a few multiplies per bar, which delivers most of the Kalman predict benefit without the covariance matrix machinery. - "Zero-lag" oversells it. This is lag reduction, not elimination, and the residual delay is real, just smaller. Set kappa to zero and it becomes a plain EMA. - The slope assumption breaks at reversals, so the filter overshoots turns and can signal in the dying direction, the exact failure from the old article ["Why Moving Averages Can Lie at Turning Points."](https://aligrithm.com/why-moving-averages-can-lie-at-turning-points/) Tune kappa small, never use it as a turn detector, and expect trend lag savings to return as turn overshoot. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Statistically Sound Indicators for Financial Market Prediction - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Statistically+Sound+Indicators+for+Financial+Market+Prediction+Timothy+Masters&ref=aligrithm.com) - [Cycle Analytics for Traders - John Ehlers (Amazon)](https://www.amazon.com/s?k=Cycle+Analytics+for+Traders+John+Ehlers&ref=aligrithm.com) ### 2.35 Frequency-Adaptive EMA: Smoothing That Reacts to Noise URL: https://aligrithm.com/frequency-adaptive-ema-smoothing-that-reacts-to-noise/ Last updated: 2026-07-27T13:23:07.000Z The old article ["EMA vs SMA: Why Simplicity Still Matters"](https://aligrithm.com/ema-vs-sma-why-simplicity-still-matters/) left the EMA's smoothing fixed: you pick alpha once, and the filter smooths the same amount whether the market is calm or thrashing. That is the wrong behavior. When the data is quiet you want a light touch that tracks closely; when the data is noisy you want heavy smoothing to reject the thrash. The old article ["Automatic Gain Control for Trading Indicators"](https://aligrithm.com/automatic-gain-control-for-trading-indicators/) made an indicator's output adapt to changing amplitude. This makes the smoothing itself adapt, by letting alpha depend on the cycle content of the data: smooth hard when the signal is fast and noisy, smooth lightly when it is slow and clean, and pay at most one bar of lag for it. ## Make alpha a function of cycle speed The standard EMA fixes alpha through the length M, and a fixed M means fixed smoothing forever. $$ y\[n\] = \\alpha\\, x\[n\] + (1-\\alpha)\\, y\[n-1\], \\qquad \\alpha = \\frac{2}{M+1} $$ The adaptive version turns alpha into a variable that depends on the cycle speed omega present in the data. The rule is a clamp with a ramp: alpha stays at a high ceiling c, below 1, while the cycle speed is at or below a threshold omega-zero, then ramps down as the speed climbs toward the fast end at pi. $$ \\alpha = \\begin{cases} c, & \\omega \\le \\omega\_0 \\\\\[4pt\] \\dfrac{a\\,\\omega + b}{\\omega\_0}, & \\omega\_0 < \\omega \\le \\pi \\end{cases} $$ Read what the clamp does in plain terms. When the data is slow, omega small, alpha sits at its high ceiling, the filter leans on the new bar and tracks closely with little smoothing. When the data is fast, omega large, alpha shrinks, which is the same as a large M, so the filter smooths hard. The logic is exactly backwards from a fixed EMA's helplessness: the noisier the data, the larger the effective M, the more it gets smoothed. A fixed EMA cannot do this because its alpha has no idea what the market is doing. ## The payoff: heavy smoothing without the usual lag bill A fixed EMA forces an ugly choice. Set M large and you smooth noise well but lag badly in calm trends; set M small and you track calm trends well but let noise through when the market thrashes. You cannot have both with one alpha. The adaptive EMA escapes the choice by spending smoothing only where it is needed. In the slow, clean stretches it keeps alpha high and tracks tight; in the fast, noisy stretches it drops alpha and smooths hard. Don Mak's measurement on this construction is the headline: the adaptive EMA carries a lag of at most one bar, better than the fixed original at comparable smoothing. You get the noise rejection of a long EMA in the regions that need it without dragging that long-EMA lag through the regions that do not. This is the same instinct as the old AGC article, applied to a different knob. AGC adapts the output amplitude so a fixed threshold keeps working across regimes; the adaptive EMA adapts the smoothing strength so a fixed filter keeps working across regimes. Both replace a constant that was secretly a regime bet with a quantity that tracks the regime. Both are one extra stage that makes the rest of the pipeline regime-independent. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## The catch: omega has to be estimated, and badly estimated omega poisons it The whole scheme rests on knowing the cycle speed omega, and omega is not handed to you. You have to estimate it from the data, in real time, before you can set alpha. Don Mak is blunt that this is the weak point: the adaptive EMA's one-bar-lag advantage holds only when omega is estimated accurately, and a bad estimate feeds the wrong alpha, which smooths the wrong amount at the wrong time. Estimate omega too fast and the filter under-smooths a noisy patch; estimate it too slow and it over-smooths a clean trend and reintroduces the lag you were trying to avoid. This is where the cost moved, not vanished. The fixed EMA's problem was a frankly wrong constant. The adaptive EMA's problem is the accuracy and lag of the cycle-speed estimator sitting upstream of it. Dominant-cycle estimation is its own hard problem, and any lag or error in that estimate flows straight into the smoothing decision. The honest reading: the adaptive EMA is a real improvement when you have a trustworthy cycle-speed estimate, and it is a liability when you do not, because it will confidently smooth by the wrong amount. Before deploying one, prove that your omega estimator is good enough on your instrument, because the filter is only as good as the speed it is told. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_225-visual-0.png) ## KEY POINTS - A fixed EMA smooths the same amount in calm and chaos, because alpha is set once through M. That is a regime bet baked into a constant, the limitation left open by the old article ["EMA vs SMA: Why Simplicity Still Matters"](https://aligrithm.com/ema-vs-sma-why-simplicity-still-matters/). - The adaptive EMA makes alpha a function of the cycle speed omega: a high ceiling c when omega is small (slow, clean data, track closely) ramping down as omega rises (fast, noisy data, smooth hard). - The behavior is the right way round: the noisier the data, the smaller alpha, the larger the effective M, the heavier the smoothing, automatically. - It escapes the fixed EMA's forced tradeoff between tracking calm trends and rejecting noisy thrash, spending smoothing only where needed, with a lag of at most one bar per Don Mak. - It is the same move as the old article ["Automatic Gain Control for Trading Indicators"](https://aligrithm.com/automatic-gain-control-for-trading-indicators/): replace a regime-blind constant with a quantity that tracks the regime, as one extra pipeline stage. - The whole scheme depends on estimating omega accurately in real time. A bad cycle-speed estimate feeds the wrong alpha and smooths by the wrong amount, so prove your omega estimator before trusting the filter. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Statistically Sound Indicators for Financial Market Prediction - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Statistically+Sound+Indicators+for+Financial+Market+Prediction+Timothy+Masters&ref=aligrithm.com) - [Cycle Analytics for Traders - John Ehlers (Amazon)](https://www.amazon.com/s?k=Cycle+Analytics+for+Traders+John+Ehlers&ref=aligrithm.com) ### 2.34 Sinc / Scaling Functions: The Closest Thing to a Brick-Wall Filter URL: https://aligrithm.com/sinc-scaling-functions-the-closest-thing-to-a-brick-wall-filter/ Last updated: 2026-07-27T13:23:07.000Z The old article ["Why the SMA Is Often a Terrible Smoother"](https://aligrithm.com/why-the-sma-is-often-a-terrible-smoother/) showed that the simple moving average leaks 22 percent of the high-frequency noise it was supposed to block, because its frequency response is a ragged sinc with sidelobes that never settle. The old article ["The Trader's Guide to Low-Pass Filters"](https://aligrithm.com/the-traders-guide-to-low-pass-filters/) gave you the recursive smoothers that do better with a gentle rolloff. This article reaches for the opposite extreme, the filter whose frequency response is as close to a vertical wall as a low-pass can get: the sinc function. It passes everything below a cutoff at full size and kills everything above it, the ideal a trader keeps wishing the SMA were. The price for that near-perfect cut is the reason you cannot run the ideal version live, and understanding why is half the lesson. ## The sinc is the ideal low-pass, by definition A brick-wall filter passes every cycle below a cutoff untouched and stops every cycle above it dead, with a vertical edge between the two. Ask what impulse response produces that perfectly rectangular frequency response and the math hands back exactly one answer: the sinc function. The trading form Don Mak uses indexes it by a smoothing parameter n. $$ h\_n(k) = \\frac{\\sin(\\pi k / n)}{\\pi k}, \\qquad k = \\ldots, -2, -1, 0, 1, 2, \\ldots $$ These coefficients are the unit impulse response of a low-pass filter, and their Fourier transform is the rectangular wall: flat gain up to the cutoff, then nothing. This is not an approximation that happens to look good. The sinc is the rectangle's exact partner, because a rectangle in the frequency domain and a sinc in the time domain are a Fourier-transform pair. Where the SMA's response wobbles with sidelobes that leak noise at minus 13 decibels forever, the sinc's response is the clean step the SMA only pretended to be. These coefficients are also called scaling functions, and they are the father of wavelets. A scaling function is a low-pass; the wavelets built on top of it are band-pass. So the sinc sits at the root of the whole wavelet machinery, the smooth foundation that the band-pass detail filters are constructed against. ## Less phase lag than EMA(M=3), until you push the cutoff The sinc's appeal for traders is not only the sharp cut. Over the slow part of the band it is also light on lag. Don Mak's measurement is concrete: the two-term sinc holds a phase lag under 0.41 radians for cycle speeds below about 1.13 radians per bar, which is less phase lag than an EMA with M equal to 3 over the same range. So in the band where most tradable structure lives you get a sharper cut and a lighter delay than a fast EMA, a rare combination. The warning sits right next to the measurement. As the cycle speed climbs toward the fast end, near half the sampling rate, the phase lag increases dramatically. The filter is gentle on lag only in its passband; push the cutoff up toward the noise floor and the delay grows fast. The practical reading is that the sinc is excellent when you want to keep a clearly slow band and reject a clearly fast one, and it degrades when you ask it to cut close to the Nyquist limit, which is the same region the old article on the SMA warned was physically fragile anyway. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Why you truncate, and what truncation costs Here is the catch that keeps the ideal sinc off your live chart. The sinc function has infinite length. Its tails ripple out forever and never reach zero, so the exact filter would need infinitely many past and future bars to compute one output. Future bars make it non-causal, which alone disqualifies it for real-time trading, and infinite length makes it impossible regardless. In practice the tails are tiny far out, so you truncate: Don Mak notes that for the two-term sinc the coefficients past about k of 120 are effectively zero, so you keep a finite window and drop the rest. Truncation is not free. Chopping the sinc's tails reintroduces ripple in the frequency response, the wall stops being perfectly vertical and the passband and stopband pick up small oscillations, the Gibbs phenomenon. Cutting the window shorter narrows the cost in computation and lag but widens the ripple; keeping it longer tightens the wall but costs more lag and more bars of warmup. The standard repair, beyond raw truncation, is to taper the window rather than chop it square, which trades a slightly less vertical wall for far smaller ripple, the same windowing logic that fixed the SMA's sidelobes in the old smoother article. The honest summary: the sinc is the closest thing to a brick wall you can build, you can run a truncated, windowed, delayed version on history, and you can never run the ideal one at the live right edge. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_224-visual-0.png) ## KEY POINTS - The sinc function is the exact impulse response of an ideal brick-wall low-pass: flat gain below the cutoff, zero above, a vertical edge. It is the clean response the SMA from the old article ["Why the SMA Is Often a Terrible Smoother"](https://aligrithm.com/why-the-sma-is-often-a-terrible-smoother/) only imitated. - A rectangle in frequency and a sinc in time are a Fourier-transform pair, so the rectangular wall and the sinc coefficients are two views of one filter, not an approximation. - Sinc coefficients are scaling functions, the father of wavelets: the scaling function is the low-pass foundation, the wavelets built on it are the band-pass details. - In its passband the two-term sinc carries less phase lag than EMA(M=3) (under 0.41 radians below about 1.13 radians per bar), but the lag rises sharply as the cutoff approaches the fast Nyquist end. - The ideal sinc has infinite length and needs future bars, so it is non-causal and unrealizable live. You truncate (coefficients past about k=120 are negligible) and run a delayed version on history only. - Truncation reintroduces ripple (the Gibbs phenomenon). A tapered window beats a square chop, trading a slightly softer wall for far less ripple, the same windowing fix that tames the SMA's sidelobes. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Statistically Sound Indicators for Financial Market Prediction - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Statistically+Sound+Indicators+for+Financial+Market+Prediction+Timothy+Masters&ref=aligrithm.com) - [Cycle Analytics for Traders - John Ehlers (Amazon)](https://www.amazon.com/s?k=Cycle+Analytics+for+Traders+John+Ehlers&ref=aligrithm.com) ### 2.33 The Butterworth Filter for Traders URL: https://aligrithm.com/the-butterworth-filter-for-traders/ Last updated: 2026-07-27T13:23:06.000Z The old article ["The Trader's Guide to Low-Pass Filters"](https://aligrithm.com/the-traders-guide-to-low-pass-filters/) gave you the recipe for the one- and two-pole smoothers that do most of the trend-extraction work, and noted in passing that the super-smoother is a critically damped two-pole Butterworth. This is the Butterworth itself, the filter electrical engineers reach for first when they want a smoother whose passband has no ripple. It buys you one specific property the EMA cannot give: a maximally flat passband, so the slow cycles you want to keep come through at full size with no bumps, while everything past the cutoff falls away faster the more poles you add. The catch is the same tax every causal smoother pays, and the Butterworth pays it harder with each pole. ## Maximally flat, by construction The Butterworth is defined in the continuous domain by a transfer function whose denominator is a polynomial of order N, with the order setting the number of poles. $$ H(s) = \\frac{1}{s^N + a\_{1}s^{N-1} + \\cdots + a\_{N-1}s + a\_N}, \\qquad s = \\frac{j\\omega}{\\omega\_c} $$ The variable s is the cycle speed scaled by the cutoff speed omega-c, the point where the filter has dropped to the half-power level. The coefficients a are fixed numbers that depend only on the order N, tabulated once and reused. The defining feature is what the magnitude does: it decreases monotonically as the cycle speed rises, with no ripple anywhere in the passband. The slow cycles you want come through flat and untouched, which matters for a trend filter because ripple in the passband would distort the very trend you are trying to read. Among the standard filter families this flatness is the Butterworth's signature, and it is the reason Ehlers ported the two- and three-pole versions to price data. ## More poles, sharper cut, more lag The order N is the dial, and it controls the sharpness of the boundary between what passes and what dies. $$ |H(\\omega)| = \\frac{1}{\\sqrt{1 + (\\omega/\\omega\_c)^{2N}}} $$ Read the exponent. With N equal to 1 the magnitude falls off gently, the same six-decibels-per-octave slope as a single EMA. Raise N to 2 and the falloff doubles to twelve per octave; raise it to 3 and the transition band, the stretch of cycle speeds between full pass and full stop, narrows again. A higher order means a cleaner separation, a filter that holds the trend at full gain closer to the cutoff and then drops the noise harder past it. For cycle-detection pipelines where a sloppy cutoff lets the wrong band leak through, that sharpness is worth paying for. The payment is lag, and it climbs with every pole. Each pole adds roughly one EMA's worth of delay at the same critical period, so a two-pole Butterworth lags about twice a one-pole EMA tuned to the same cutoff, and a three-pole lags about three times. This is the same structural delay the old article ["The Trader's Guide to Low-Pass Filters"](https://aligrithm.com/the-traders-guide-to-low-pass-filters/) priced for the pole count, and it does not go away because the magnitude response looks elegant. Past a few poles the lag makes the filter useless for real-time trading: the cut gets beautiful and the signal arrives too late to act on. For most daily-frequency work two poles is the sweet spot, and three is the ceiling. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## The critically damped variant avoids the ringing A plain Butterworth above one pole has a subtle vice. The poles sit at locations that produce a small overshoot and ringing in the step response, so when price gaps the filter output briefly swings past the new level and oscillates before settling. On a step-sensitive job like breakout or regime-change detection, that overshoot manufactures false signals. The super-smoother from the old low-pass article is the repair: it moves the two poles to conjugate, critically damped locations that give the same twelve-per-octave rolloff with no overshoot and a monotonic settle. So the two-pole Butterworth and the super-smoother share a rolloff slope and differ in step behavior, and for trading you usually want the critically damped version because clean settling beats a marginally different passband shape. This leaves a short rule for choosing a Butterworth on price data. Use two poles as the default when you want a sharper cut than an EMA gives and can absorb double the lag. Reach for three only in cycle-detection pipelines where the cleanliness of the boundary earns its keep and the extra lag is acceptable. Stop there, because beyond three poles the lag dominates. And prefer the critically damped coefficients whenever the downstream logic reacts to steps, so a gap does not ring its way into a false trade. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_223-visual-0.png) ## KEY POINTS - The Butterworth is the engineer's default low-pass: its passband is maximally flat, so the slow cycles you keep pass at full size with no ripple, which is what a trend filter needs. - It is defined by an order-N denominator polynomial with fixed tabulated coefficients; the magnitude falls off monotonically with cycle speed and has no bumps. - The order N is the dial. Each pole steepens the rolloff (6 dB/octave per pole) and narrows the transition band, giving a cleaner split between the trend kept and the noise killed. - The price is lag, climbing roughly one EMA's worth per pole, the same structural delay the old article ["The Trader's Guide to Low-Pass Filters"](https://aligrithm.com/the-traders-guide-to-low-pass-filters/) priced. Two poles is the daily sweet spot, three the ceiling, beyond that the signal arrives too late. - A plain multi-pole Butterworth overshoots and rings on a price gap, manufacturing false signals on step-sensitive logic. The critically damped super-smoother gives the same rolloff with a clean monotonic settle. - Choose two poles by default, three only for cycle-detection pipelines that need the sharp boundary, and prefer the critically damped coefficients whenever downstream logic reacts to steps. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Statistically Sound Indicators for Financial Market Prediction - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Statistically+Sound+Indicators+for+Financial+Market+Prediction+Timothy+Masters&ref=aligrithm.com) - [Cycle Analytics for Traders - John Ehlers (Amazon)](https://www.amazon.com/s?k=Cycle+Analytics+for+Traders+John+Ehlers&ref=aligrithm.com) ### 2.32 The Transfer Function View: Characterizing Any Indicator with the Z-Transform URL: https://aligrithm.com/the-transfer-function-view-characterizing-any-indicator-with-the-z-transform/ Last updated: 2026-07-27T13:23:06.000Z The old article ["Why Traders Should Analyze Indicators Mathematically"](https://aligrithm.com/why-traders-should-analyze-indicators-mathematically/) made the claim that every common indicator is a known signal-processing operation, and the old article ["The Frequency Response of Trading Indicators"](https://aligrithm.com/the-frequency-response-of-trading-indicators/) showed that the magnitude curve tells you which cycles an indicator keeps or kills. Neither showed you the machine that produces that curve. The transfer function is that machine. Write any linear indicator as a transfer function and you can read its entire behavior, the gain at every cycle, the lag at every cycle, and the resonances that make it ring, straight off one compact fraction, before you ever plot a single bar. ## The delay operator: one symbol does the bookkeeping Start with the exponential moving average, because its recursion contains the one moving part you need. Today's EMA is a slice of today's price plus the rest of yesterday's EMA. $$ y\[t\] \\;=\\; \\alpha\\, x\[t\] \\;+\\; (1-\\alpha)\\, y\[t-1\], \\qquad \\alpha = \\frac{2}{M+1} $$ The alpha sets how hard the filter leans on the new bar versus its own history, and the length M maps to alpha through the usual relation. Writing "yesterday's value" as y\[t-1\] over and over gets clumsy once filters reach across several bars, so engineers compress it into one symbol, the delay operator z to the minus one, which means nothing more than shift back one bar. $$ z^{-1}\\, x\[t\] = x\[t-1\], \\qquad z^{-2}\\, x\[t\] = x\[t-2\] $$ Multiply a series by z to the minus one and you get its value one bar ago. That is the whole meaning of z at this stage, a relabeling that turns recursions into algebra. The old article ["DSP and Digital Filters for Traders: The Primer Nobody Wrote First"](https://aligrithm.com/dsp-and-digital-filters-for-traders-the-primer-nobody-wrote-first/) introduced this button; here it earns its keep. ## Divide output by input and you get the indicator's DNA Rewrite the EMA with the delay operator, gather the output terms on one side, and divide output by input. The result is the transfer function, written H of z. $$ y\\,\\big(1 - (1-\\alpha) z^{-1}\\big) = \\alpha\\, x \\qquad\\Longrightarrow\\qquad H(z) = \\frac{y}{x} = \\frac{\\alpha}{1 - (1-\\alpha)\\, z^{-1}} $$ That fraction is the complete EMA. The numerator carries the input terms, the denominator carries the feedback terms, and every linear indicator in this pillar reduces to a ratio of two short polynomials in z. The values of z that zero the numerator are the zeros, the cycle speeds the filter kills. The values that zero the denominator are the poles, the feedback resonances that let a recursive filter get a sharp response from almost no storage. The EMA has one pole, sitting at z equal to one minus alpha. A pole near the unit circle makes the filter ring at the matching speed, which you want in a band-pass and nowhere else. ## Put z on the unit circle and the frequency response falls out The transfer function becomes the frequency response the instant you restrict z to the unit circle, setting z equal to e to the j-omega, where omega is the cycle speed in radians per bar. Substitute and split into size and angle. $$ |H(\\omega)| = \\frac{\\alpha}{\\sqrt{1 - 2(1-\\alpha)\\cos\\omega + (1-\\alpha)^2}}, \\qquad \\phi(\\omega) = \\tan^{-1}\\!\\frac{-(1-\\alpha)\\sin\\omega}{1 - (1-\\alpha)\\cos\\omega} $$ The magnitude is the gain curve from the old frequency-response article, derived rather than asserted: it sits near 1 for slow cycles where omega is small, and falls toward alpha as omega climbs to pi, the textbook low-pass shape. The phase is the delay each cycle suffers. Turn the phase into the number traders care about, lag in bars, by dividing the phase by the cycle speed. $$ \\text{lag in bars}(\\omega) = \\frac{\\phi(\\omega)}{\\omega} $$ Now you can answer, for one filter, the two questions that decide whether to use it: how much of each cycle survives, and how many bars late each cycle arrives. Both come from the same fraction, with no backtest and no plotting. That is the payoff of the transfer-function view: the indicator's full character is in H of z, and the unit circle reads it out. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## What this buys you, and where it stops The real leverage shows up when you compose filters. Cascade two filters and you multiply their transfer functions; subtract one output from another, as a moving-average difference does, and you add or subtract the transfer functions. Build a band-pass by placing poles near the cycle you want and zeros at zero and at the Nyquist speed, and the magnitude curve obeys before you write a line of code. Stacking indicators that share a pole-zero layout stacks the same filter, which is the structural reason the redundancy the old frequency-response article warned about exists: confluence between two indicators with the same H of z is one indicator counted twice. The method has a hard boundary. The transfer function exists only for linear filters, the ones that are a weighted sum of past inputs and past outputs. RSI, stochastics, and anything with a division or a range-normalization are nonlinear, so they have no single H of z, and you can only characterize their linear core before the nonlinear squashing. And a clean transfer function is a description of behavior, not evidence of edge. A filter can pass exactly the band you wanted, with the lag you accepted, and still feed your model a band that carried no predictive value on your instrument. The transfer function tells you what the indicator does. Whether what it does pays is a separate question you answer with data. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_222-visual-0.png) ## KEY POINTS - The old article ["Why Traders Should Analyze Indicators Mathematically"](https://aligrithm.com/why-traders-should-analyze-indicators-mathematically/) said every indicator is a signal-processing operation; the transfer function is the machine that produces the frequency response the old article ["The Frequency Response of Trading Indicators"](https://aligrithm.com/the-frequency-response-of-trading-indicators/) told you to read. - The delay operator z to the minus one means shift back one bar. It turns a recursion into algebra so you can manipulate filters as fractions. - Divide output by input and you get the transfer function H of z, a ratio of two short polynomials. The numerator's roots are zeros (cycles killed); the denominator's roots are poles (feedback resonances). The EMA has one pole at z equal to one minus alpha. - Restrict z to the unit circle (z equals e to the j-omega) and the magnitude gives the gain at each cycle while the phase gives the delay. Divide phase by cycle speed to read lag directly in bars. - Composition is algebra: cascade filters by multiplying transfer functions, subtract outputs by subtracting them. Two indicators with the same pole-zero layout are one filter, which is why stacking them is redundancy, not confirmation. - The method covers linear filters only. RSI and stochastics are nonlinear and have no single H of z. And a clean transfer function describes behavior, never proves edge; whether the passed band pays is an empirical question. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Statistically Sound Indicators for Financial Market Prediction - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Statistically+Sound+Indicators+for+Financial+Market+Prediction+Timothy+Masters&ref=aligrithm.com) - [Cycle Analytics for Traders - John Ehlers (Amazon)](https://www.amazon.com/s?k=Cycle+Analytics+for+Traders+John+Ehlers&ref=aligrithm.com) ### 8.5 Markets Are Getting More Random: Non-Stationarity of Randomness Itself URL: https://aligrithm.com/markets-are-getting-more-random-non-stationarity-of-randomness-itself/ Last updated: 2026-07-27T13:26:02.000Z Most traders worry about the wrong kind of non-stationarity. They watch the mean drift, the volatility cluster, the correlations break, all the things the old article ["Stationarity: The Word Every Trader Ignores Until It Kills the Strategy"](https://aligrithm.com/stationarity-the-word-every-trader-ignores-until-it-kills-the-strategy-2/) warned about. There is a deeper one. The degree of randomness itself drifts. Run the same random-walk test on old market data and on recent data and you get different answers: recent prices conform more to a random walk than older prices did. The market does more than shift its parameters; it changes how predictable it is, and the direction of that change is toward less predictability. The edge that existed in the data you studied may be smaller in the data you trade, because the market evolved while you were calibrating. ## Randomness as a measured quantity, not a fixed property Treat predictability as something you measure on a window rather than a permanent label. The old article ["Random Walk and Efficient Markets Are Not the Same Thing"](https://aligrithm.com/random-walk-and-efficient-markets-are-not-the-same-thing/) made the case that a random walk is a testable statistical claim about price changes, distinct from the economic claim of efficiency. The variance ratio is one such measurement: a value of 1 means the increments scale like a random walk, departures from 1 flag the autocorrelation that a tradable structure leaves behind. $$ VR\_q(\\text{epoch}) = \\frac{\\operatorname{Var}(r\_t^{(q)})}{q\\,\\operatorname{Var}(r\_t^{(1)})}, \\qquad VR\_q \\to 1 \\text{ as the epoch gets more recent} $$ The variance ratio for horizon q is the variance of the q-period return divided by q times the one-period variance, which equals 1 under a pure random walk. Compute it per epoch instead of once over the whole history, the 1960s, the 1980s, the 2010s, and the finding is that the number drifts toward 1 as the epoch gets more recent. The same is true of first-order autocorrelation and the other departure-from-randomness statistics: they were larger in magnitude in older data and have shrunk toward the random-walk values over the decades. Randomness is not a constant of the market. It is a time series in its own right, and it has been trending up. ## Why the market got more random This is not mysterious once you stop treating the market as a fixed object and start treating it as a population of participants that changes. The old finding behind this is concrete: when researchers asked why recent data looked more random, they traced it to the rise of day trading and the growth in the number of active short-horizon participants who altered the return distribution. More traders hunting the same short-term patterns means each pattern gets arbitraged away faster, so the autocorrelation that one generation of traders fed on is eaten by the next generation competing to exploit it. The structure does not vanish because the market became wise in some abstract sense. It vanishes because more money chased it, which is the mechanism, not a metaphor. That makes the market a complex evolving system, the framing the econophysics thread keeps returning to. Efficiency is an ecological variable that rises and falls with how crowded the relevant strategies are, so the careful version of this story is a cyclical, context-dependent efficiency that tightens when a strategy gets crowded and loosens again when participants leave, a new instrument launches, or a regime scares the competition out. The trend in the data has been toward more randomness in the most-traded, mature markets, and the mechanism, competition eroding exploitable structure, tells you where to look for the exceptions. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## What an evolving randomness does to your edge This is a survival argument, the core concern of this pillar, so treat it as one. First, your backtest's edge is an upper bound, not an estimate, because the structure you measured on historical data has been getting arbitraged away and the same statistic computed on next year's data tends to be weaker, so discount the historical Sharpe before you trust it and assume decay rather than persistence. Second, the more crowded and liquid the market, the faster the erosion, so the deepest, most-studied markets are where randomness has advanced furthest and where a pattern you discover today has most often been arbitraged away already. Third, hunt the exceptions the mechanism predicts: newer instruments, less-trafficked markets, and strategies that are hard or uncomfortable for the crowd to run are where exploitable structure survives longest, because fewer participants have competed it away. Fourth, monitor the randomness of your own market in real time, not once, tracking whether the variance ratio or autocorrelation your edge depends on is itself drifting toward the random-walk value, because that drift is the early warning of the system death the old article on stationarity told you to budget for. The market you backtested is not the market you are trading, and the specific way it changed is that it got harder to beat. ## Visualizing randomness on the move ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_221-visual-0.png) ## KEY POINTS - The deepest non-stationarity is in randomness itself. Run the same random-walk test on old versus recent data and recent prices conform more to a random walk, so the market's predictability drifts toward less predictability. - Predictability is a measured quantity, not a fixed label. Compute the variance ratio or first-order autocorrelation per epoch and the departures from the random-walk values shrink toward 1 over the decades, building on the old article ["Random Walk and Efficient Markets Are Not the Same Thing."](https://aligrithm.com/random-walk-and-efficient-markets-are-not-the-same-thing/) - The mechanism is competition: more day traders and short-horizon participants altered the return distribution and arbitraged away the autocorrelation earlier traders fed on. Structure vanishes because more money chased it. - That makes the market a complex evolving system. Efficiency is an ecological variable tied to how crowded the relevant strategies are, so the careful version is cyclical, context-dependent efficiency, not a steady march to perfect randomness. - Your backtest edge is an upper bound, not an estimate. Discount the historical Sharpe, assume decay, and expect the fastest erosion in the deepest, most-studied markets where randomness has advanced furthest. - Hunt the exceptions the mechanism predicts (newer instruments, thin markets, strategies the crowd finds uncomfortable) and monitor your own market's randomness in real time, because drift toward the random-walk value is the early warning of the system death the old article ["Stationarity: The Word Every Trader Ignores Until It Kills the Strategy"](https://aligrithm.com/stationarity-the-word-every-trader-ignores-until-it-kills-the-strategy-2/) said to budget for. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Mathematical Techniques in Financial Markets - Don Mak (Amazon)](https://www.amazon.com/s?k=Mathematical+Techniques+in+Financial+Markets+Don+Mak&ref=aligrithm.com) - [Cycle Analytics for Traders - John Ehlers (Amazon)](https://www.amazon.com/s?k=Cycle+Analytics+for+Traders+John+Ehlers&ref=aligrithm.com) ### 8.3 Log-Normal Prices and the Price/Volume Trick URL: https://aligrithm.com/log-normal-prices-and-the-price-volume-trick/ Last updated: 2026-07-27T13:26:01.000Z Bachelier started everyone off on the wrong distribution. In 1900 he modeled the stock price itself as Brownian motion, a random walk with normally distributed changes, which is elegant and has one fatal flaw: it lets the price go negative. A stock cannot trade below zero, so a model that assigns real probability to negative prices is broken at the boundary before you even reach the fat-tail problems the old article ["Fat Tails: Why Gaussian Thinking Breaks Trading Systems"](https://aligrithm.com/fat-tails-why-gaussian-thinking-breaks-trading-systems/) laid out. The fix that the field settled on is to put the random walk on the logarithm of price instead of the price, which makes returns normal and prices log-normal. That single change repairs the boundary, and it sets up a less obvious result: the thing that fits a clean distribution is not price at all, it is price divided by volume. ## From Bachelier to geometric Brownian motion Move the Brownian motion onto log price and the model becomes geometric Brownian motion, the workhorse behind most of continuous-time finance. The log of price does a random walk, so the price is the exponential of a normal variable, which is the definition of a log-normal distribution. $$ \\ln P\_t - \\ln P\_0 \\sim \\mathcal{N}(\\mu t,\\ \\sigma^2 t) \\quad\\Longrightarrow\\quad P\_t = P\_0\\, e^{\\,\\mu t + \\sigma W\_t} $$ The left side says the change in log price over an interval is normally distributed with a mean and variance that grow with time, which is the random-walk-on-returns statement. The right side is the same thing solved for the price: the starting price multiplied by the exponential of a drift term plus a random term, where W is the Brownian noise. Because price is an exponential of a normal variable it can never go negative, which fixes Bachelier's boundary problem, and the multiplicative form matches how returns compound, a 1% move on a $10 stock and on a $1000 stock being the same percentage rather than the same dollar amount. This is the price process that has to hold, up to the structure, for the random-walk half of the old article ["Random Walk and Efficient Markets Are Not the Same Thing"](https://aligrithm.com/random-walk-and-efficient-markets-are-not-the-same-thing/) to even be stated. The log-normal is a genuine improvement over the raw Gaussian and still not the truth. Real log returns have the fat tails and the peaked center the old article on fat tails cataloged, so the log-normal price model gets the no-negative-price boundary right while still understating the extremes. It is the correct skeleton, not the correct flesh. ## The price/volume trick Here is the part worth stealing. Fit a log-normal to raw price changes and the fit is mediocre, the tails are wrong and the parameters drift. Antoniou's finding is that if you normalize price by volume first, dividing price by the volume traded, the resulting quantity fits a log-normal cleanly across most stocks. $$ f(x) = \\frac{A}{\\sqrt{2\\pi}\\,\\sigma\\, x}\\, \\exp\\!\\left(-\\frac{(\\ln x - \\mu)^2}{2\\sigma^2}\\right), \\qquad x = \\frac{\\text{price}}{\\text{volume}} $$ The function is the standard log-normal density: A is a normalizing constant, mu and sigma are the mean and dispersion of the logarithm, and the variable x is not price but price divided by volume. The reason this works points at something real about market structure. Volume is the denominator of price discovery, the amount of trading it took to hold the price where it is, so price-per-unit-volume is closer to a measure of value adjusted for the effort of trading than price alone. Dividing the two strips out a chunk of the messiness that makes raw price changes deviate from any clean distribution, and what survives is the well-behaved log-normal shape. Volume carries information that price alone hides, which is the same instinct behind every volume-weighted indicator, stated as a distributional fact rather than a charting habit. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## What to use this for Resist the urge to build a system that bets on price/volume reverting to its log-normal mean, because a clean static fit on historical data is a description, not a tradable edge, and the same fat-tail and non-stationarity warnings from the rest of this pillar apply the moment you try to trade it. The honest uses are narrower and more useful. First, model returns and risk in log space, not price space, so your volatility scaling and position sizing operate on the quantity that is closer to normal and your downside math never relies on a price that the model says could go negative. Second, treat the price/volume result as a reason to engineer features in price/volume units rather than raw price, because a quantity that fits a known distribution is far easier to standardize, to z-score, and to compare across instruments than one that does not. Third, hold the boundary lesson: any model that can produce a negative price, a negative volatility, or a probability that ignores the fat tail is telling you about its own assumptions, not about the market, and the log-normal is the minimum repair, not the finish line. ## Visualizing the log-normal repair ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_219-visual-0.png) ## KEY POINTS - Bachelier modeled price itself as Brownian motion, which lets price go negative, a boundary failure that breaks the model before fat tails even enter. - Putting the random walk on log price gives geometric Brownian motion: log returns are normal and prices are log-normal, so price is an exponential of a normal variable and can never go negative, matching how returns compound multiplicatively. - The log-normal price model is the right skeleton, not the right flesh. Real log returns still carry the fat tails and peaked center from the old article ["Fat Tails: Why Gaussian Thinking Breaks Trading Systems,"](https://aligrithm.com/fat-tails-why-gaussian-thinking-breaks-trading-systems/) so it fixes the boundary while understating the extremes. - Antoniou's result: raw price changes fit a log-normal poorly, but price divided by volume fits one cleanly across most stocks, because volume is the effort of price discovery and dividing it out strips messiness that price alone carries. - The honest uses are modeling returns and risk in log space so sizing never relies on an impossible negative price, and engineering features in price/volume units because a quantity with a known distribution is easy to standardize and compare across instruments. - A clean static fit is a description, not an edge. Do not trade price/volume reverting to its mean; the fat-tail and non-stationarity warnings of this pillar still apply. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Mathematical Techniques in Financial Markets - Don Mak (Amazon)](https://www.amazon.com/s?k=Mathematical+Techniques+in+Financial+Markets+Don+Mak&ref=aligrithm.com) - [Cycle Analytics for Traders - John Ehlers (Amazon)](https://www.amazon.com/s?k=Cycle+Analytics+for+Traders+John+Ehlers&ref=aligrithm.com) ### 8.4 Long-Range Dependence: Real Memory or Just Short-Range Echo? URL: https://aligrithm.com/long-range-dependence-real-memory-or-just-short-range-echo/ Last updated: 2026-07-27T13:26:02.000Z Traders love the idea that markets have long memory. It would mean a move today still nudges prices months or years out, that business cycles leave a persistent fingerprint you can lean on, that the Hurst exponent you computed and got 0.6 from is real persistence you can trade. The old article ["Variance Ratio Tests for Traders"](https://aligrithm.com/variance-ratio-tests-for-traders/) already showed how to measure whether moves reinforce or reverse over a horizon. Long-range dependence is the grander claim: memory so persistent it never dies, beyond the few-day reinforcement the variance ratio already covers. The claim is mostly an artifact. When you correct for ordinary short-range autocorrelation, the long memory tends to evaporate, and most of the famous Hurst estimates were measuring something they were not designed to separate. ## The R/S statistic and the Hurst exponent The classic tool for long memory is rescaled range analysis, the R/S statistic that Hurst built studying Nile river floods. Take a window of length n, compute the running sum of deviations from the mean, measure the range of that running sum (the spread between its high and low), and divide by the standard deviation over the window. Do it across many window sizes and see how the rescaled range grows with n. $$ \\left(\\frac{R}{S}\\right)\_n \\sim c\\, n^{H}, \\qquad H = 0.5 \\text{ (no memory)}, \\quad H > 0.5 \\text{ (persistent)}, \\quad H < 0.5 \\text{ (anti-persistent)} $$ The rescaled range scales as the window size raised to the power H, where H is the Hurst exponent. Read the three cases. At H equal to 0.5 the range grows as the square root of the window, which is what a pure random walk with no memory does, the same square-root-of-time benchmark behind the variance ratio. Above 0.5 the range grows faster, which says the series is persistent, with moves tending to be followed by moves in the same direction over long stretches. Below 0.5 it grows slower, the anti-persistent case where moves tend to reverse. Find H well above 0.5 on stock returns and the temptation is to declare long memory and a tradable trend at long horizons. ## The trap: short-range dependence inflates H The problem is that the plain R/S statistic cannot tell long memory apart from ordinary short-range autocorrelation. A series with no long memory at all, holding only a little week-to-week persistence of the kind every market has, will still produce an R/S slope above 0.5 and hand you a Hurst exponent that looks like long-range dependence. The statistic conflates the two, and the famous early findings of long memory in stock returns were mostly this confusion: short-range dependence wearing a long-memory costume. Lo's fix was a modified R/S statistic that normalizes not by the plain standard deviation but by one that includes the autocovariances out to a chosen number of lags, deliberately soaking up the short-range structure before measuring the range. $$ S\_q^2 = \\hat{\\sigma}^2 + 2\\sum\_{j=1}^{q}\\left(1 - \\frac{j}{q+1}\\right)\\hat{\\gamma}\_j $$ The modified denominator takes the ordinary variance and adds a weighted sum of the first q autocovariances, the gamma terms, with weights that taper as the lag grows. By folding the short-range correlations into the normalizer, the statistic discounts the echo that was inflating H. Apply it to daily, weekly, monthly, and annual stock index returns across many sample periods and the result is blunt: once short-term autocorrelation is accounted for, the evidence for long-range dependence disappears. The long memory was the short memory, measured with a tool that could not separate them. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## The methodology lesson that outlives the result This is a robustness cautionary tale before it is a fact about markets, and the lesson generalizes far past the Hurst exponent. A statistic that produces a striking number is worthless until you know what else could produce that same number. The plain R/S looked like it was measuring deep memory while part of what it measured was ordinary autocorrelation, so the headline finding was an artifact of the estimator, not a property of the data, the same failure mode as any indicator that looks predictive because it shares regime structure with its target. Note also that the debate did not close: later work argued Lo's modified statistic leans too hard the other way, biased toward accepting no long memory as the lag count grows, with some studies still finding weak long-range dependence at Hurst values near 0.6 in individual stocks. So the honest position is the uncomfortable one. Long memory in returns is weak at best and mostly an artifact of short-range dependence, the effect that survives correction is too small and too unstable to build a long-horizon system on, and any Hurst exponent you compute should be reported as an estimate that changes with the window length and the correction method, not as a constant of the market. Before you trade a number, try to kill it with a simpler explanation, and the simplest explanation here, short-range autocorrelation, kills most of it. ## Visualizing memory versus echo ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_220-visual-0.png) ## KEY POINTS - Long-range dependence is the claim that market memory is so persistent it never dies, a grander version of the horizon question the old article ["Variance Ratio Tests for Traders"](https://aligrithm.com/variance-ratio-tests-for-traders/) measured. - The classic R/S (rescaled range) statistic estimates a Hurst exponent H from how the rescaled range grows with window size: H of 0.5 is a memoryless random walk, above 0.5 is persistent, below 0.5 is anti-persistent. - Plain R/S cannot separate long memory from ordinary short-range autocorrelation. A series with only week-to-week persistence still yields H above 0.5, so the famous early long-memory findings were short-range dependence in disguise. - Lo's modified R/S normalizes by a standard deviation that includes the first q autocovariances, soaking up the short-range structure. Applied across daily to annual index returns, the evidence for long-range dependence disappears once short-term autocorrelation is accounted for. - The debate is not closed: later work found Lo's statistic biased toward rejecting long memory as the lag grows, with weak Hurst values near 0.6 surviving in some individual stocks, so the surviving effect is too small and unstable to trade. - The robustness lesson outlives the result: a striking statistic is worthless until you know what else produces the same number, and any Hurst estimate should be reported as window- and method-dependent, not a constant of the market. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Mathematical Techniques in Financial Markets - Don Mak (Amazon)](https://www.amazon.com/s?k=Mathematical+Techniques+in+Financial+Markets+Don+Mak&ref=aligrithm.com) - [Cycle Analytics for Traders - John Ehlers (Amazon)](https://www.amazon.com/s?k=Cycle+Analytics+for+Traders+John+Ehlers&ref=aligrithm.com) ### 6.42 Stable Paretian / Fractal Distributions: Infinite Variance Markets URL: https://aligrithm.com/stable-paretian-fractal-distributions-infinite-variance-markets/ Last updated: 2026-07-27T13:25:51.000Z The old article ["Lévy Distributions and Market Extremes"](https://aligrithm.com/levy-distributions-and-market-extremes/) introduced the fatter-tailed cousin of the Gaussian and showed where it breaks. Here is the part worth taking seriously, the version Mandelbrot put forward back in the early 1960s when he looked at cotton prices and refused to round the ugly tails off into a bell curve. He proposed that speculative price changes follow a family he called stable Paretian: a sharper peak than the Gaussian, genuine power-law tails, a built-in tendency to trend and to cycle, discontinuous jumps, and one property that quietly demolishes most of the risk math people run. The variance is infinite. Not large. Not hard to estimate. Mathematically undefined. The Lévy stable distribution from the old article is this same family under a different name, and the trading consequence of believing it is that half your toolbox stops being valid. ## Mandelbrot's family: peaked center, power-law tails, one knob The stable Paretian distributions generalize the Gaussian by adding a single shape parameter, the characteristic exponent, usually written alpha, that controls how fat the tails are. The Gaussian sits at one end of the family with the thinnest possible tails. Turn the knob down and the tails thicken into power laws while the center grows a sharper peak, the leptokurtotic shape real returns show. $$ P(|r| > x) \\sim x^{-\\alpha}, \\qquad 0 < \\alpha \\le 2, \\qquad \\alpha = 2 \\Rightarrow \\text{Gaussian} $$ The tail probability falls off as the move size raised to a negative power, the slow power-law decay rather than the ferocious exponential decay of the bell curve. A smaller alpha means fatter tails and more frequent extremes. Set alpha to 2 and you recover the normal distribution as the special, thin-tailed corner of the family. So this does not reject the Gaussian; it is a wider net that contains the Gaussian and also contains the peaked, heavy-tailed shapes the market draws. The family carries two more properties Mandelbrot needed and that matter for trading. It can be skewed, tilted to assign more mass to crashes than to melt-ups or the reverse, which the symmetric Gaussian cannot do. And it allows discontinuous changes, price jumping from one level to another with nothing in between, which is what a gap or an overnight repricing looks like and what no continuous diffusion produces. ## Self-similar across scales: why these are called fractal The reason this family later got the name fractal is that it scales. The shape of the distribution at one timescale looks like the shape at another, up to a stretch factor, instead of smoothing toward a bell curve the way independent Gaussian increments do. $$ r(\\lambda t) \\;\\overset{d}{=}\\; \\lambda^{1/\\alpha}\\, r(t) $$ The return measured over a stretched window of length lambda times t has the same distribution as the original return scaled by lambda raised to one over alpha. The equals sign with the d over it means equal in distribution, the same statistical shape, not the same number. The practical reading: zoom out from daily to weekly to monthly and you do not see the tails politely shrink into normality, you see the same fat-tailed, peaked, jumpy shape rescaled. For the Gaussian, one over alpha is one half, the familiar square-root-of-time scaling. For a fat-tailed alpha below 2, the exponent is larger, so the extremes grow faster than the square root of time as you aggregate. Trends and cycles persist across timescales rather than washing out, which is the same self-similarity the trend-following edge in the old article ["Why Fat Tails Are the Reason Trend Following Works"](https://aligrithm.com/why-fat-tails-are-the-reason-trend-following-works/) relies on, now stated as a scaling law. ## Infinite variance, and what it breaks The property that earns this distribution its danger label: for any alpha below 2, the variance is infinite. $$ \\text{Var}(r) = \\int\_{-\\infty}^{\\infty} x^2\\, p(x)\\, dx \\;=\\; \\infty \\quad \\text{for } \\alpha < 2 $$ Variance is the integral of the squared deviation weighted by its probability. With power-law tails that decay slowly, the squared term grows faster than the probability shrinks far out in the tail, so the integral does not converge to a finite number. It diverges. This is not a measurement problem you fix with more data. The population quantity does not exist, and the more data you collect the more your sample variance jumps around instead of settling, because the next observation can be large enough to dominate everything before it. Sit with what that destroys. Any tool that assumes a finite, stable variance is operating on a quantity the distribution refuses to provide. The volatility scaling behind position sizing, the standard deviation in a Sharpe ratio, the variance ratio test from the old article ["Variance Ratio Tests for Traders"](https://aligrithm.com/variance-ratio-tests-for-traders/), the confidence bands you draw around an estimate, all of them rest on a second moment that, under a true stable Paretian world, is not there. Your sample standard deviation still computes a number. It estimates something that does not converge, so it understates the risk during calm stretches and lurches when a tail event lands, which is the same instability the old article on fat tails warned makes volatility estimates untrustworthy. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## How to use a distribution you cannot trust You do not adopt the stable Paretian distribution as your working model and start sizing positions off it, for the same reason the old article on Lévy distributions gave: it fits the body and shoulders well and still misses the genuine crashes, and a model with infinite variance is awkward to plug into any sizing formula. Its value is diagnostic, not predictive. It tells you that the trending-and-cyclical, jumpy, scale-invariant character of markets is the baseline, not an anomaly, so you should expect persistence across timescales and stop being surprised by gaps. And it tells you, hard, to distrust every statistic that assumes a finite variance without admitting it. When your risk number is a multiple of a standard deviation, you are leaning on a quantity that may not exist, so treat it as a floor on risk rather than a measure of it, and stress-test against moves larger than any variance-based bound would suggest. The honest takeaway from Mandelbrot's family is not a better curve to fit. It is a warning that the second moment, the thing half of quantitative finance is built on, may be a fiction in the markets you actually trade. ## Visualizing infinite variance ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_217-visual-0.png) ## KEY POINTS - Mandelbrot proposed in the early 1960s that speculative prices follow a stable Paretian family: a sharper peak than the Gaussian, genuine power-law tails, skewness, and discontinuous jumps. It is the same family as the Lévy stable distribution from the old article ["Lévy Distributions and Market Extremes."](https://aligrithm.com/levy-distributions-and-market-extremes/) - One shape parameter, the characteristic exponent alpha, controls tail fatness. At alpha equal to 2 you recover the Gaussian as the thin-tailed corner; turn it down and the tails fatten into slow power laws while the center peaks. - The family is self-similar across timescales, which is why it is called fractal: aggregate from daily to weekly to monthly and the fat, peaked, jumpy shape rescales instead of smoothing into a bell curve, so trends and cycles persist across scales. - For any alpha below 2 the variance is infinite, not large but mathematically undefined, because the slow power-law tail makes the integral of squared deviations diverge. More data makes the sample variance jump around, not settle. - Infinite variance breaks every tool that assumes a finite second moment: volatility-based position sizing, the Sharpe denominator, the variance ratio test, and any confidence band. The sample standard deviation still prints a number, but it estimates a quantity that does not converge. - The distribution is diagnostic, not a working model. It fits the body and misses the crashes like the old article on Lévy showed, so use it to expect persistence and jumps, distrust variance-based risk numbers, and stress-test beyond any bound built on a standard deviation. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Systematic Trading - Robert Carver (Amazon)](https://www.amazon.com/s?k=Systematic+Trading+Robert+Carver&ref=aligrithm.com) - [Trading Systems - Urban Jaekle Emilio Tomasini (Amazon)](https://www.amazon.com/s?k=Trading+Systems+Urban+Jaekle+Emilio+Tomasini&ref=aligrithm.com) ### 8.2 The Omori Law: Why Aftershocks Follow Market Crashes URL: https://aligrithm.com/the-omori-law-why-aftershocks-follow-market-crashes/ Last updated: 2026-07-27T13:26:01.000Z Seismologists noticed something over a century ago that traders rediscover the hard way after every crash: the big shock starts a decaying sequence of smaller shocks. After a major earthquake the aftershocks arrive fast at first, then taper off along a fixed mathematical curve. Markets do the same thing. The crash hits, and then the violent days keep coming, dense right after the break and thinning out over weeks, following the same curve geologists use for aftershocks. The old article ["Why Financial Markets Are Complex Systems"](https://aligrithm.com/why-financial-markets-are-complex-systems/) argued that crashes are critical phenomena, not random draws. This is the cleanest evidence for that claim: the aftermath of a crash is structured, predictable in its decay, and impossible if returns were independent. ## The aftershock curve, borrowed from geology The Omori law describes how the rate of aftershocks decays after a big earthquake. Count the number of aftershocks per unit time above some size threshold, and that rate falls off as a power of the time since the main shock. $$ m(t) = K\\,(1 + t/\\tau)^{-p} $$ Read it piece by piece. The term m(t) is the rate of aftershocks at time t after the main event, K and tau are constants that set the scale and the early plateau, and p is the exponent that controls how fast the rate decays, usually near 1\. The shape is the point: a fast initial burst that bends into a slow power-law tail, so aftershocks stay more frequent than background for a long stretch after the shock. Now swap earthquakes for a market index and aftershocks for days where the absolute return exceeds a large threshold. Lillo and Mantegna ran that substitution and found the count of these threshold-exceeding days after a crash follows the same Omori curve, with volatility relaxing as a power law rather than snapping back or fading on an exponential. Integrate that rate from the crash to time t and you get the cumulative number of big-move days, the thing you can watch tick up in real time. $$ N(t) = \\frac{K\\left\[(1 + t/\\tau)^{1-p} - 1\\right\]}{1-p}\\quad (p \\neq 1), \\qquad N(t) = K\\,\\ln\\!\\left(1 + t/\\tau\\right)\\quad (p = 1) $$ The cumulative count N(t) grows fast at first and then bends over, the integral of a decaying rate. For a trader the formula matters less than what it rules out. A logarithmic or power-law cumulative count is what you see when the post-crash period has no fixed scale, when each day's chance of another violent move depends on how long ago the crash was. Independent returns cannot produce it. ## Why this kills the independence assumption Return the argument to the heart of the old article ["Anomalous Diffusion in Financial Markets,"](https://aligrithm.com/anomalous-diffusion-in-financial-markets/) which showed that price changes over successive intervals are correlated, not independent, because traders react to earlier moves. The Omori law is that correlation caught in its most dramatic form. If each day's return were an independent draw from some fixed distribution, the gap between violent days would itself be memoryless and the rate of big moves would be flat, not a decaying power law. The crash would carry no information about tomorrow. Markets refuse this. After a crash the conditional probability of another large move is elevated and comes down along a slow tail, so the days cluster, and that clustering is the signature of a system relaxing from an extreme state rather than a coin reshuffling its outcomes. The structure goes deeper than a single crash. The aftershock sequence is self-similar: inside the relaxation period after a major break, smaller shocks set off their own miniature Omori sequences, aftershocks within aftershocks, the same power-law decay nested at finer scales. That self-similarity is the fingerprint of the complex, critical system the old article on complex markets described, and it is why a standard volatility model that assumes a single mean-reverting variance, like a plain GARCH specification, fails to reproduce the post-crash relaxation: the decay has no characteristic timescale to mean-revert toward. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## What to do with it Do not read this as a forecast of direction, because Omori says nothing about which way the aftershocks point, only that more violence is coming and roughly how its frequency will fade. The trading consequences are about risk, not entry signals. After a crash, treat elevated volatility as the base case for weeks, not days, because the power-law tail keeps the big-move rate high long after the headlines move on, and a volatility estimate that assumes a quick return to calm will understate your risk through the entire relaxation. Size down and keep sizing down until the aftershock rate has decayed, not for the first quiet session that tempts you back to full size. Expect the secondary shocks, the failed bounces and the second-leg-down days, as the normal anatomy of a post-crash tape rather than fresh surprises. And carry the lesson from the old article on complex systems into position management: the crash is a transition into a different regime with its own statistics, so the rules and volatility assumptions you calibrated in the calm period do not apply until the system has finished relaxing. ## Visualizing the aftershock decay ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_218-visual-0.png) ## KEY POINTS - The Omori law from seismology says the rate of aftershocks after a big earthquake decays as a power law of time since the shock, a fast initial burst bending into a slow tail. - Lillo and Mantegna showed market index returns obey the same law: after a crash, the count of days where the absolute return exceeds a large threshold decays along the Omori curve, with volatility relaxing as a power law. - This is the strongest form of the correlation the old article ["Anomalous Diffusion in Financial Markets"](https://aligrithm.com/anomalous-diffusion-in-financial-markets/) described. Independent returns would make big-move days memoryless and their rate flat, not a decaying power law, so the crash carries information about tomorrow. - The aftershock sequence is self-similar: smaller shocks inside the relaxation period spawn their own nested Omori decays, the fingerprint of the critical system from the old article ["Why Financial Markets Are Complex Systems."](https://aligrithm.com/why-financial-markets-are-complex-systems/) - A single mean-reverting volatility model like plain GARCH cannot reproduce the post-crash relaxation, because the power-law decay has no characteristic timescale to revert toward. - Use it for risk, not direction. After a crash, treat high volatility as the base case for weeks, size down through the entire decay rather than the first calm session, and expect secondary shocks as the normal anatomy of a post-crash tape. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Mathematical Techniques in Financial Markets - Don Mak (Amazon)](https://www.amazon.com/s?k=Mathematical+Techniques+in+Financial+Markets+Don+Mak&ref=aligrithm.com) - [Cycle Analytics for Traders - John Ehlers (Amazon)](https://www.amazon.com/s?k=Cycle+Analytics+for+Traders+John+Ehlers&ref=aligrithm.com) ### 6.41 Why Fat Tails Are the Reason Trend Following Works URL: https://aligrithm.com/why-fat-tails-are-the-reason-trend-following-works/ Last updated: 2026-07-27T13:25:50.000Z The old article ["Fat Tails: Why Gaussian Thinking Breaks Trading Systems"](https://aligrithm.com/fat-tails-why-gaussian-thinking-breaks-trading-systems/) told you the bad news: real returns have fat tails, the bell curve calls crashes impossible, and Gaussian risk models blow up in the one regime they exist to survive. Same fact, flip the sign, and it becomes the entire reason a whole industry makes money. Markets produce big moves far more often than a normal distribution allows, those big moves arrive in directional runs rather than as isolated spikes, and a strategy that climbs aboard a move and holds it until it dies is harvesting the part of the distribution the Gaussian swears does not exist. Fat tails do more than threaten the careless trader. They hand the trend follower an edge. ## Leptokurtosis is where the trends live A return distribution is leptokurtotic when it has a sharper peak and fatter tails than the Gaussian: most days cluster tighter than the bell curve near zero, and the days that escape escape much further than the bell curve permits. The single number that flags it is kurtosis, the fourth moment scaled by variance. $$ \\text{kurtosis} = \\frac{E\\!\\left\[(r-\\mu)^4\\right\]}{\\sigma^4}, \\qquad \\text{Gaussian} = 3, \\qquad \\text{markets} \\gg 3 $$ The fourth power means this statistic weights the biggest deviations from the mean far more than the rest, so it is a magnifying glass pointed straight at the extremes. A Gaussian sits at 3\. Equity index returns, commodity returns, and currency returns run well above that, into the teens and higher on daily data. Excess kurtosis, the amount above 3, is the part the normal distribution cannot produce, and it is not a measurement error. It is the market telling you that large moves carry more of the total variation than a random walk would assign them. Read that operationally and the trading consequence falls out. If price changes were independent Gaussian draws, the big moves would scatter at random and you could never lean on them, because tomorrow's draw would forget today's. Markets do not forget. The fat tail shows up alongside positive autocorrelation at the horizons trend followers care about, the same property the old article ["Variance Ratio Tests for Traders"](https://aligrithm.com/variance-ratio-tests-for-traders/) measured as a variance ratio above 1: moves reinforce, variance compounds faster than it would under independence, and the extreme observations group into sustained directional runs instead of one-day stabs. The fat tail is the footprint of trends. No fat tail, no trend, no edge. ## The payoff that only fat tails can pay Trend following has an ugly win rate. Most trades lose a little, the stops get hit, the breakouts fail, and the equity curve grinds sideways or down for long stretches. The strategy survives because of the shape of the payoff, not the frequency of the wins. $$ E\[\\text{PnL}\] = p\_{\\text{loss}} \\cdot (-L) \\; + \\; p\_{\\text{tail}} \\cdot G, \\qquad p\_{\\text{tail}} \\text{ small}, \\quad G \\gg L $$ Read it plainly. The expected profit is the probability of the many small losses times their size, which is negative, plus the probability of the rare large win times its size. The win probability is low and the loss term drags, so the only thing that pulls the whole expression positive is the second piece: the gain G being so much larger than the loss L that a handful of them outweighs the long tail of small bleeds. That large G has to come from somewhere. It comes from the fat tail. A Gaussian world caps the size of the winners at a few standard deviations, the asymmetry collapses, G stops dwarfing L, and the strategy's expectation goes to zero or below once costs bite. The positively skewed return stream that trend followers live on, many small losses funding occasional outsized gains, is a direct print of the leptokurtosis in the underlying. The convexity people praise as crisis alpha is the same thing seen from the portfolio level: the big tail moves tend to cluster in stressed regimes, the strategy is positioned to ride them, and the rare giant winner shows up when everything else is bleeding. So the causal chain is short. Fat tails create large moves, large moves come in runs, runs are trends, trends produce the oversized winners, and the oversized winners are the only reason a low-win-rate strategy clears its costs. Strip the fat tail and you strip the edge. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## The same tail that pays you can also gap through your stop Do not mistake this for a free lunch, because the fat tail is double-edged and the old article on Gaussian failure spelled out the dangerous side. The leptokurtosis that delivers your giant winner is the same leptokurtosis that delivers a gap straight through your stop, and trend followers eat those too: an overnight reversal that the bell curve called impossible takes back weeks of a position before you can act. The asymmetry stays in your favor only if your exits cut the losers fast and let the winners run, which is what converts a symmetric fat tail into a positively skewed P&L. Get the exit logic wrong, hold the losers and clip the winners, and you invert the skew and feed the tail instead of harvesting it. There is also the cost of waiting. The tail events are rare by definition, so most of the time the strategy is paying small losses while waiting for a move that may be months away, and that stretch of underperformance is where most people quit right before the acceleration that would have paid for everything. The fat tail is the source of the edge and the source of the pain in the same distribution. Respect it the way the old article said: size for the move you cannot model, because the tail that funds you does not announce which direction it will fire. ## Visualizing the edge inside the tail ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_216-visual-0.png) ## KEY POINTS - Fat tails are the same fact the old article ["Fat Tails: Why Gaussian Thinking Breaks Trading Systems"](https://aligrithm.com/fat-tails-why-gaussian-thinking-breaks-trading-systems/) framed as a threat, read from the other side: the big moves the Gaussian forbids are what trend following harvests. - Leptokurtosis (kurtosis far above the Gaussian value of 3) is a magnifying glass on the extremes. It says large moves carry more of the total variation than a random walk allows, and they arrive in directional runs, not isolated spikes. - The runs are trends. The fat tail shows up alongside the positive autocorrelation the old article ["Variance Ratio Tests for Traders"](https://aligrithm.com/variance-ratio-tests-for-traders/) measured as a variance ratio above 1\. No fat tail, no trend, no edge. - Trend following has a low win rate and survives only on payoff shape: many small losses funded by rare oversized winners, where the winner G dwarfs the loss L. That oversized G can only come from the fat tail. - A Gaussian world caps the winners at a few sigma, collapses the asymmetry, and kills the strategy's expectation once costs bite. The positive skew and the crisis-alpha convexity are direct prints of the underlying leptokurtosis. - The tail is double-edged. The same fat tail that funds you can gap through your stop, the edge survives only if exits cut losers fast and let winners run, and the long waits between tail events are where most traders quit right before the move that pays for everything. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Systematic Trading - Robert Carver (Amazon)](https://www.amazon.com/s?k=Systematic+Trading+Robert+Carver&ref=aligrithm.com) - [Trading Systems - Urban Jaekle Emilio Tomasini (Amazon)](https://www.amazon.com/s?k=Trading+Systems+Urban+Jaekle+Emilio+Tomasini&ref=aligrithm.com) ### 4.46 Why Commercials Are Counter-Trend (and Lead by 2 Weeks) URL: https://aligrithm.com/why-commercials-are-counter-trend-and-lead-by-2-weeks/ Last updated: 2026-07-27T13:24:37.000Z A corn farmer does not sell futures because he thinks corn is going down. He sells because the price on offer is good enough to lock in a profit on a crop he has to grow anyway. When prices spike, more producers reach for that lock, so commercial selling rises into strength. When prices collapse, hedgers lift shorts and consumers buy forward to secure cheap supply, so commercial buying rises into weakness. The smart money in futures is mechanically counter-trend, and it is counter-trend for a business reason, not a market opinion. That single fact is what makes commercial positioning a leading indicator instead of a lagging one. The ["Reading the COT Report: Three Trader Groups, One Edge"](https://aligrithm.com/reading-the-cot-report-three-trader-groups-one-edge/) article set up the three groups and flagged the commercials as the ones to watch. This article explains the mechanism behind why they lead, because a lead-lag relationship you cannot explain is a lead-lag relationship that will betray you, the warning that runs through ["Lead-Lag Relationships in Global Markets."](https://aligrithm.com/lead-lag-relationships-in-global-markets/) The commercial lead has a clean causal story, which is why it has held up. ## Accumulation and distribution Commercials move size, and size cannot be deployed in a day without moving the market against yourself. So they work positions through a slow process of accumulation and distribution: buying patiently over weeks while prices are low and supply is cheap, selling patiently over weeks while prices are high. They are scaling in and out against the prevailing move, absorbing the futures that trend-followers and retail are dumping or chasing. This is the opposite of how a speculator behaves. A trend-following fund adds to longs as price rises and the strength confirms its model. A commercial does the reverse: rising price is rising hedging opportunity, so it sells more. The two groups are structurally on opposite sides, which the zero-sum constraint from the previous article guarantees, but the point here is causal, not just accounting. Commercials buy as prices fall and sell as prices rise because their incentive is tied to the physical business, and that incentive points against the trend by construction. $$ \\text{price} \\uparrow \\;\\Rightarrow\\; \\text{commercial selling} \\uparrow \\quad(\\text{distribution}) $$ $$ \\text{price} \\downarrow \\;\\Rightarrow\\; \\text{commercial buying} \\uparrow \\quad(\\text{accumulation}) $$ As price rises, commercials distribute, selling more into the strength. As price falls, they accumulate, buying more into the weakness. Read together, these say the commercial net position moves inverse to price: heavy net short near highs, heavy net long near lows. Their positioning is a contrarian gauge built by people who are not trying to be contrarian, which is what makes it trustworthy. ## Why the lead is about two weeks The accumulation process is why the positioning leads price. Commercials are buying near a bottom before the bottom is visible on the chart, because they buy on cheap physical value, not on a price signal. By the time the market turns up and the trend becomes obvious, the commercials are already loaded long; they did their buying during the decline. Their net position swings to an extreme before the price reverses, so reading their extreme is reading the setup for a turn that has not happened yet. Empirically, that head start has run on the order of two weeks. Commercial positioning tends to reach its bullish extreme roughly two weeks before price bottoms, and its bearish extreme roughly two weeks before price tops. The exact figure drifts by market and era, so treat "two weeks" as the order of magnitude, not a stopwatch. The mechanism is what matters: they lead because they act on physical value while the market is still acting on price momentum, and value leads momentum at turns. This is the rare leading indicator with a reason attached. Most claimed leads are curve-fits that decay the moment you trade them, the graveyard ["Lead-Lag Relationships in Global Markets"](https://aligrithm.com/lead-lag-relationships-in-global-markets/) catalogs. The commercial lead persists because the cause persists: producers and consumers will always hedge against the trend, and they will always do it slowly enough to leave a trail in the positioning data before the price catches up. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## How the two-week lead survives a three-week lag There is an obvious objection. COT data reaches you about three weeks old, and the lead is only about two weeks, so on paper the lag eats the lead and the signal is stale before you see it. If commercials lead price by two weeks and you learn their position three weeks after the fact, you are looking at a setup that already resolved. The resolution is that the commercial extreme is a level, not a one-day event. Commercials sit at a bullish extreme for weeks while they finish accumulating, not for a single Tuesday. So even reading the data with a multi-week delay, you can still catch the market while the commercials remain at the extreme and before price has fully turned, because the extreme persists longer than the lag. The signal is the sustained positioning state, not the instantaneous reading, which is why COT works as a slow weekly gauge despite the delay and fails as a fast trigger. A multi-week setup tolerates a multi-week lag; a one-bar setup would not. The trade this sets up is a contrarian one, taken with the commercials and against the crowd. When commercial net position reaches a bullish extreme relative to its own history, you lean long into the weakness the commercials are buying; when it reaches a bearish extreme, you lean short into the strength they are selling. Trading the raw net level is clumsy, though, because "extreme" only means something relative to this market's own range. Turning the commercial net into a normalized, bounded signal is the job of the next article, "Building a COT Index System." ## Visualizing the commercial lead ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_213-visual-0.png) ## KEY POINTS - Commercial hedgers are mechanically counter-trend for a business reason: they sell futures into rising prices to lock in good selling levels and buy into falling prices to secure cheap supply. Their incentive points against the trend by construction. - They move size through slow accumulation and distribution, scaling in over weeks while prices are low and out while prices are high, taking the other side of what trend-followers and retail chase. - Commercial net position moves inverse to price: heavy net short near highs, heavy net long near lows. It is a contrarian gauge built by people who are not trying to be contrarian, which is why it is trustworthy. - The positioning leads price because commercials act on physical value before the chart turns. They are already loaded long near a bottom, having bought during the decline. - The lead has run on the order of two weeks, reaching a bullish extreme about two weeks before price bottoms and bearish about two weeks before tops. Treat two weeks as an order of magnitude that drifts by market. - This is a leading indicator with a real cause, which is why it persists where curve-fit leads decay: producers and consumers will always hedge against the trend, leaving a trail before price catches up. - The two-week lead survives the three-week data lag because the commercial extreme is a level held for weeks, not a one-day event. A multi-week setup tolerates a multi-week delay; trade it as a slow gauge, not a trigger. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Trading Systems and Methods - Perry Kaufman (Amazon)](https://www.amazon.com/s?k=Trading+Systems+and+Methods+Perry+Kaufman&ref=aligrithm.com) - [Cybernetic Trading Strategies - Murray Ruggiero (Amazon)](https://www.amazon.com/s?k=Cybernetic+Trading+Strategies+Murray+Ruggiero&ref=aligrithm.com) - [The Art of Currency Trading - Brent Donnelly (Amazon)](https://www.amazon.com/s?k=The+Art+of+Currency+Trading+Brent+Donnelly&ref=aligrithm.com) ### 4.47 Building a COT Index System URL: https://aligrithm.com/building-a-cot-index-system/ Last updated: 2026-07-27T13:24:38.000Z The raw commercial net position is unusable as a signal, and the reason is the one that kills most positioning data. "Commercials are net short 60,000 contracts" means nothing on its own. Is 60,000 a lot? In a market that routinely runs net short 200,000, it is nothing. In a market that has never been more short than 50,000, it is an off-the-charts extreme. The number only has meaning against the market's own history, and that is exactly the problem normalization solves. To trade commercial positioning you stop trading the level and start trading where the level sits inside its own historical range. This is the stationarity problem from ["How to Build Stationary Indicators from Non-Stationary Prices,"](https://aligrithm.com/how-to-build-stationary-indicators-from-non-stationary-prices/) applied to positioning instead of price. The raw net wanders with the size of the market, the number of participants, and the era. A bounded oscillator that rescales the net against its recent extremes gives you a number that means the same thing in every market and every decade, which is the only kind of number you can put a threshold on. ## The COT index The standard fix is a range-normalization that maps the current net position onto a 0-to-100 scale defined by its own highest and lowest values over a lookback window. $$ \\text{COT Index} \\;=\\; 100 \\times \\frac{\\text{Net} - \\text{Lowest}(\\text{Net},\\, N)}{\\text{Highest}(\\text{Net},\\, N) - \\text{Lowest}(\\text{Net},\\, N)} $$ Take the current net position, subtract the lowest net seen over the last N periods, and divide by the full range between the highest and lowest net over that window. Multiply by 100\. The result sits between 0 and 100 by construction: it reads 100 when the current net is the most net long it has been in N periods, 0 when it is the most net short, and 50 at the midpoint of its recent range. A reading near 100 for the commercials means they are as bullishly positioned as they get; near 0 means as bearish as they get. The wandering, market-specific net becomes a clean contrarian oscillator you can compare across silver, corn, and crude on the same axis. This is a min-max rescaling, the same family as the standardized seasonal and a cousin of the z-score ranking in ["Why Z-Scoring Makes Ranking Cleaner."](https://aligrithm.com/why-z-scoring-makes-ranking-cleaner/) The difference from a z-score is that this uses the observed high and low of the range rather than the mean and standard deviation, which makes it bounded and sensitive to the extremes, exactly what you want when the whole signal is about reaching an extreme. The lookback N is the one real parameter, and it is a genuine tradeoff: a short N makes the index reach 0 and 100 constantly and cry wolf, a long N makes extremes rare but meaningful. Pick it and then test it honestly, because it is as overfittable as any threshold. ## Run it on the hedgers The index works best on the commercial hedgers, and that is not arbitrary. The previous article, ["Why Commercials Are Counter-Trend (and Lead by 2 Weeks),"](https://aligrithm.com/why-commercials-are-counter-trend-and-lead-by-2-weeks/) established that commercials are the group with the causal lead, so their normalized extreme is the one that means something. Build the index on the commercial net and the most predictive signals come out of it. You can compute the same index on the large speculators or small traders, but those groups follow the trend, so their extremes confirm the move rather than anticipating the turn. The signal is contrarian and you take it with the commercials. A high COT index, commercials at a bullish extreme, says lean long, because the smart money has been accumulating into weakness and the turn up tends to follow. A low COT index, commercials at a bearish extreme, says lean short, because they have been distributing into strength. You are fading the speculators and the retail crowd who sit on the other side of the zero-sum book. $$ \\text{COT Index}\_{\\text{commercial}} > 80 \\;\\Rightarrow\\; \\text{bullish lean} $$ $$ \\text{COT Index}\_{\\text{commercial}} < 20 \\;\\Rightarrow\\; \\text{bearish lean} $$ Above 80, commercials are near the top of their historical net-long range and you favor long setups; below 20, near the bottom and you favor short setups. The 80 and 20 are conventional and arbitrary, the same kind of threshold ["How to Build Stationary Indicators from Non-Stationary Prices"](https://aligrithm.com/how-to-build-stationary-indicators-from-non-stationary-prices/) insisted you test out-of-sample rather than eyeball. Some markets need 90/10 to mean anything; some give plenty of signal at 75/25\. The bands are a starting point, not a law. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Where it works and how to use it The COT index earns its keep in commodities and falls apart elsewhere, a limitation important enough that the next article, ["Why COT Fails in Currencies"](https://aligrithm.com/why-cot-fails-in-currencies/), is devoted to it. In the physical commodity markets, the futures market is where the real hedging happens, so the commercial position genuinely reflects the smart money. That is the precondition for the whole index to mean anything. Two practical cautions before you trade it. First, this is a slow gauge with a multi-week reporting lag and a multi-week signal horizon, so the COT index is a positioning filter and a bias-setter, not an entry trigger. It tells you which direction to favor over the coming weeks; it does not tell you the bar to buy. Used the way ["Seasonality as a Filter, Not a Standalone System"](https://aligrithm.com/seasonality-as-a-filter-not-a-standalone-system/) used a weak seasonal, the COT index belongs as a gate or a tilt on a faster system that handles timing, not as a standalone signal generator. Second, normalize honestly. The Highest and Lowest in the index look back over N periods, and if you compute them over the whole sample including future data you have built look-ahead straight into the index, the current reading scaled against a range it could not have known. Use only data available up to the current bar to define the range, or your backtest is reading a ruler from the future. With that discipline, the COT index turns the commercials' causal lead into a bounded, comparable, tradable contrarian signal, and the only question left is which markets it survives in. ## Visualizing the COT index ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_214-visual-0.png) ## KEY POINTS - Raw commercial net position is unusable as a signal because its meaning depends entirely on the market's own history. The same contract count is nothing in one market and an extreme in another. - The COT index normalizes the net onto a 0-to-100 scale using the highest and lowest net over a lookback N: 100 at the most net long in N periods, 0 at the most net short, 50 at the midpoint. - This is a bounded min-max rescaling, a cousin of z-score ranking but built on observed extremes rather than mean and standard deviation, which is what you want when the signal is about reaching an extreme. - The lookback N is the key parameter and a real tradeoff: short N cries wolf, long N makes extremes rare but meaningful. Test it out-of-sample; it overfits like any threshold. - Run the index on the commercial hedgers, the group with the causal lead. Their normalized extreme anticipates turns; speculator and small-trader extremes only confirm the existing trend. - Trade it contrarian with the commercials: a high index (commercials bullish, conventionally above 80) leans long, a low index (below 20) leans short. The 80/20 bands are arbitrary starting points to be tested. - It works in commodities, where the futures market is where real hedging happens, and fails elsewhere. It is a slow positioning filter and bias-setter with a multi-week lag, not an entry trigger, and the lookback range must use past data only to avoid look-ahead. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Trading Systems and Methods - Perry Kaufman (Amazon)](https://www.amazon.com/s?k=Trading+Systems+and+Methods+Perry+Kaufman&ref=aligrithm.com) - [Cybernetic Trading Strategies - Murray Ruggiero (Amazon)](https://www.amazon.com/s?k=Cybernetic+Trading+Strategies+Murray+Ruggiero&ref=aligrithm.com) - [The Art of Currency Trading - Brent Donnelly (Amazon)](https://www.amazon.com/s?k=The+Art+of+Currency+Trading+Brent+Donnelly&ref=aligrithm.com) ### 4.48 Why COT Fails in Currencies URL: https://aligrithm.com/why-cot-fails-in-currencies/ Last updated: 2026-07-27T13:24:38.000Z The COT index works beautifully on corn and silver and dies quietly on the euro, and traders who do not understand why keep trying to trade it there anyway. The failure is not a bad parameter or a weak sample. It is structural. COT measures positioning in the futures market, and the entire edge depends on the futures market being where the smart money actually trades. In currencies, it is not, so the data you are reading is a tiny, unrepresentative corner of the real market, and a contrarian signal built on it is noise wearing the costume of a signal. The previous articles in this arc built the COT system on a single assumption: the commercial position in futures reflects the genuine hedging activity of the people who know the physical market. That assumption holds in commodities. It collapses in FX, and the reason ties directly back to ["FX Is Not One Market: Retail vs Wholesale Structure."](https://aligrithm.com/fx-is-not-one-market-retail-vs-wholesale-structure/) ## Futures are the whole market in commodities, a sliver in FX In a physical commodity, the regulated futures exchange is the center of gravity for hedging. A corn producer who wants to lock in a price uses corn futures, because that is where the liquidity and the standardized contracts are. The futures open interest captures a large and representative share of the real hedging, so the commercial net position in the COT report is a faithful read on what the smart money is doing. Currencies are the opposite. Currency futures exist, but they are a rounding error next to the over-the-counter spot, forward, and swap market where the actual volume lives. The real FX market is the interbank and institutional OTC world described in ["The Real Heart of FX Liquidity,"](https://aligrithm.com/the-real-heart-of-fx-liquidity/) and it dwarfs the listed futures by orders of magnitude. The futures represent only a small slice of the total currency market, so the positions reported in COT are not the smart money's book; they are a fraction of it, and not even a random fraction. $$ \\text{COT signal quality} \\;\\propto\\; \\frac{\\text{hedging captured by futures}}{\\text{total hedging in the market}} $$ $$ \\text{commodities: ratio} \\approx \\text{high} \\qquad \\text{currencies: ratio} \\approx \\text{tiny} $$ The usefulness of the COT data scales with how much of the market's real hedging actually shows up in futures. In commodities that fraction is high, so the signal is informative. In currencies it is tiny, so the commercial net in FX futures tells you about the positioning of a small, self-selected group of futures users, not about the institutions moving the currency in the OTC market. A clean formula computed on the wrong dataset is still the wrong answer. ## Why this is a sample problem, not a tuning problem The temptation is to assume the FX failure is fixable: try a different lookback, a different threshold, a different currency. It is not, and recognizing why saves you a lot of wasted backtesting. The futures positioning in FX is an unrepresentative sample of the true positioning. No amount of normalizing an unrepresentative sample makes it representative. You can compute a gorgeous COT index on euro futures, and it will have all the statistical polish of the commodity version and none of the predictive content, because the thing it measures is disconnected from the thing that moves the price. This is the difference between a parameter you can tune and a precondition you cannot. The COT system has one hard precondition: the futures market must capture the smart money. Tune all you want inside markets that meet it. In markets that fail it, tuning is just fitting noise, and a great backtest there is a warning, not a green light, the same "works everywhere is a red flag" instinct that should fire whenever a method posts results in a market where its mechanism cannot hold. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## The general rule this teaches The currency failure generalizes into a check you should run before applying any positioning-based system to a new market: does the dataset you are measuring capture the participants who actually drive the price? COT works in commodities and fails in FX for the same reason a microstructure signal works on a lit exchange and fails when most volume is in dark pools or OTC. If the venue you can see is not where the decisive flow happens, the data from that venue is structurally blind to the thing you care about. So the practical boundary is clean. Use COT in the physical commodity markets, grains, metals, energy, softs, livestock, where futures are the hedging venue and the commercial position is the real smart-money book. Do not use it in currencies, where the listed futures are a sideshow to the OTC market. The same caution extends to any market where exchange-listed futures are a small fraction of total activity; the commodity-versus-currency split is the clearest case, not the only one. That closes the Commitment of Traders arc. The report splits the market into hedgers, speculators, and retail; the commercials lead by about two weeks because they trade physical value against the trend; the COT index normalizes their position into a bounded contrarian signal; and the whole apparatus only works where futures are the real market, which means commodities yes, currencies no. A powerful tool with a sharp edge and a clearly marked boundary is worth more than a tool you believe works everywhere, because the second kind is the one that quietly loses your money in the markets where it never could have worked. ## Visualizing the structural failure ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_215-visual-0.png) ## KEY POINTS - The COT index works in commodities and fails in currencies for a structural reason, not a tuning reason. COT measures futures positioning, and the edge requires the futures market to be where the smart money actually trades. - In physical commodities, the regulated futures exchange is the center of gravity for hedging, so the commercial net position is a faithful read on the smart money. - In currencies, listed futures are a rounding error next to the OTC spot, forward, and swap market. Futures capture only a tiny slice of total FX activity, so the reported positions are an unrepresentative fraction of the real book. - COT signal quality scales with the fraction of real hedging that shows up in futures: high in commodities, tiny in currencies. A clean formula on the wrong dataset is still the wrong answer. - This is a sample problem, not a tuning problem. No lookback or threshold fixes an unrepresentative sample, and a polished COT backtest on FX futures is a warning, not a green light. - The general check before applying any positioning system to a new market: does the dataset capture the participants who actually drive the price? If the visible venue is not where the decisive flow happens, the data is structurally blind. - Use COT in grains, metals, energy, softs, and livestock; avoid it in currencies and any market where listed futures are a small fraction of total activity. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Trading Systems and Methods - Perry Kaufman (Amazon)](https://www.amazon.com/s?k=Trading+Systems+and+Methods+Perry+Kaufman&ref=aligrithm.com) - [Cybernetic Trading Strategies - Murray Ruggiero (Amazon)](https://www.amazon.com/s?k=Cybernetic+Trading+Strategies+Murray+Ruggiero&ref=aligrithm.com) - [The Art of Currency Trading - Brent Donnelly (Amazon)](https://www.amazon.com/s?k=The+Art+of+Currency+Trading+Brent+Donnelly&ref=aligrithm.com) ### 4.44 Long-Term Market Timing from Fundamentals, Not Charts URL: https://aligrithm.com/long-term-market-timing-from-fundamentals-not-charts/ Last updated: 2026-07-27T13:24:36.000Z A moving average crossover cannot tell you whether stocks are entering a decade-long bull or a decade-long grind. It tells you what price did last week. For the slowest timing decisions, the multi-year tilt between owning stocks, owning bonds, or hiding in cash, the chart is the wrong instrument, because the forces that set those long swings are fundamental, and they move on a timescale the chart was never built to see. This article closes the macro-timing arc by stepping back from the individual signals and using the whole fundamental toolkit for the job it is actually suited to: timing trends measured in years. The case for fundamentals here is not aesthetic. Over short horizons, price-based signals win because the fundamental data is too slow and too noisy to react in time. Over multi-year horizons, that flips. Fundamentals lead, price follows, and a chart-only trader is reading the smoke while the fundamentals are the fire. The inflation, money-supply, and rate signals built in the rest of this chapter are slow precisely because the trends they predict are slow, and that slowness is a feature when the holding period is measured in years. ## What the fundamentals are timing The two big slow trends are interest rates and the stock market, and they are linked. Stocks and long bonds are both discounted by the same long rate, the relationship ["Using Bonds to Filter Equity Signals"](https://aligrithm.com/using-bonds-to-filter-equity-signals/) used to gate equity trades, so a fundamental read on the rate trend is also a read on the equity backdrop. Falling rates lift the present value of future earnings and tend to support stocks; rising rates do the reverse. Time the rate trend and you have timed a large part of the equity trend for free. The inputs are the ones already on the bench. The Real-Rate Ratio from ["Predicting Interest Rates from Inflation: The Real-Rate Ratio"](https://aligrithm.com/predicting-interest-rates-from-inflation-the-real-rate-ratio/) reads the inflation pressure on rates. The money-supply, confidence, and unemployment-duration signals from ["Money Supply, Confidence, and Unemployment Duration as Rate Predictors"](https://aligrithm.com/money-supply-confidence-and-unemployment-duration-as-rate-predictors/) read the growth pressure. And the maturity split from ["Short Rates Price the Present, Long Rates Price the Future"](https://aligrithm.com/short-rates-price-the-present-long-rates-price-the-future/) tells you which of those inputs to weight for which part of the curve. For multi-year timing you lean on the long-end drivers, money supply and expectations, because those are the ones pricing the future you are trying to position for. $$ \\text{long-term equity backdrop} \\;\\sim\\; -\\,\\Delta(\\text{long rate trend}),\\qquad \\Delta(\\text{long rate trend}) \\;\\sim\\; f\\big(\\text{money supply, inflation regime, growth}\\big) $$ Read it as a chain, not a precise model. The equity backdrop improves when the long-rate trend falls, and the long-rate trend is driven by the money-supply, inflation, and growth fundamentals. So a fundamental read that points to falling long rates points, secondhand, to a supportive equity backdrop, and the reverse for rising rates. The arrows are tendencies with long and variable lags, which is the whole challenge. ## Why this is timing, not prediction The honest framing matters most here, because long-horizon fundamental timing is where overconfidence does the most damage. A fundamental setup that says "rates should fall over the next few years" is an explanation of the pressure on rates, not a prediction of the path. ["The Difference Between Explanation and Prediction in Markets"](https://aligrithm.com/the-difference-between-explanation-and-prediction-in-markets/) drew this line, and it is sharpest at multi-year horizons: the explanation can be correct and the market can take three years to agree, during which you are early, which is indistinguishable from wrong on a brokerage statement. So fundamental timing buys you direction and backdrop, not entry precision. It tells you which way the slow tide is running and therefore which trades to favor and which to fade. It does not tell you the week to put the position on. The "long rates can stay artificially low for years" warning from the maturity article applies in full: a correct fundamental thesis can be punished for a long time before it pays, and sizing has to assume you will be early. That is why long-term fundamental timing belongs in the role of a tilt, not a trigger. Use it to set the strategic lean, overweight equities and bonds when the fundamental backdrop is supportive, lighten up and raise cash when it turns hostile, and let faster, price-based systems handle the actual entries and exits inside that lean. The fundamental view is the wind; the price system is the sail. Trading the wind directly, with no regard for what price is doing, is how you end up short a market for three years because your macro model was right about a move that had not started. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## The discipline this requires Two failure modes kill fundamental timers, and both are avoidable. The first is data that lies about what you knew. Macro series are published late and revised heavily, so a backtest of a fundamental timing model on final, revised data is fiction. You have to use the data vintage available on the decision day, the same real-time discipline from ["How to Test Indicator Thresholds Without Fooling Yourself,"](https://aligrithm.com/how-to-test-indicator-thresholds-without-fooling-yourself/) or your beautiful multi-year timing model is reading numbers it could not have had. At long horizons this is worse, not better, because revisions accumulate and a few mis-dated releases can flip the whole signal. The second is the small sample problem. Multi-year trends are rare. A 50-year history contains only a handful of complete rate-and-equity cycles, so any timing rule tuned on that history is fit to a tiny number of independent events, and "it worked across every major cycle" might mean it worked across four. That is not a sample, it is an anecdote with good production values. A fundamental timing rule has to rest on a mechanism robust enough to trust on causal grounds, because the statistics will never be strong enough to carry it alone, the exact bind this whole pillar keeps returning to: at the slowest horizons, the cause does more of the work than the backtest can. The verdict is narrow and useful. Fundamentals are the right instrument for multi-year timing, where they lead and the chart lags, but they deliver a strategic tilt with a wide error bar, not a dated signal. Treat the fundamental backdrop as the slow context that decides which way to lean, run faster systems for execution inside that lean, respect the lag, and never trust a multi-year rule that has only a handful of cycles and no mechanism behind it. ## Visualizing fundamental timing ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_211-visual-0.png) ## KEY POINTS - For multi-year timing decisions, the strategic tilt between stocks, bonds, and cash, the chart is the wrong instrument. The forces setting those long swings are fundamental and move on a timescale price signals cannot see. - The slowness of fundamental data is a feature at long horizons. Over years, fundamentals lead and price follows; the chart-only trader reads the smoke while fundamentals are the fire. - Rates and stocks are linked through the same long discount rate, so a fundamental read on the rate trend is also a read on the equity backdrop. Falling rates support stocks, rising rates pressure them. - The inputs are the chapter's toolkit: the Real-Rate Ratio for inflation pressure, money supply/confidence/duration for growth pressure, and the maturity split to weight long-end drivers for multi-year work. - This is timing, not prediction. A correct fundamental thesis gives direction and backdrop, not entry precision, and can be early for years, which looks identical to wrong on a statement. - Use fundamentals as a strategic tilt, not a trigger: set the lean with the backdrop, let faster price-based systems handle entries and exits inside it. The fundamental view is the wind, the price system is the sail. - Two failure modes: backtesting on revised data instead of real-time vintages, and tuning a rule to a tiny sample of multi-year cycles. At the slowest horizons the mechanism must carry the rule, because the statistics never will. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Trading Systems and Methods - Perry Kaufman (Amazon)](https://www.amazon.com/s?k=Trading+Systems+and+Methods+Perry+Kaufman&ref=aligrithm.com) - [Cybernetic Trading Strategies - Murray Ruggiero (Amazon)](https://www.amazon.com/s?k=Cybernetic+Trading+Strategies+Murray+Ruggiero&ref=aligrithm.com) - [The Art of Currency Trading - Brent Donnelly (Amazon)](https://www.amazon.com/s?k=The+Art+of+Currency+Trading+Brent+Donnelly&ref=aligrithm.com) ### 4.45 Reading the COT Report: Three Trader Groups, One Edge URL: https://aligrithm.com/reading-the-cot-report-three-trader-groups-one-edge/ Last updated: 2026-07-27T13:24:37.000Z Every Friday the regulator publishes a census of who owns the futures market. Not price, not volume, but positions: how many contracts each kind of trader is long and short in every major futures market. Most traders never open it. The ones who do are reading the only public dataset that tells you what the people with the deepest pockets and the best information are actually doing with their money, instead of what they say on television. The Commitment of Traders report is that census, and the edge in it comes from splitting the market into three groups that behave in completely different ways. This is intermarket analysis turned inward. The ["Intermarket Analysis for System Traders"](https://aligrithm.com/intermarket-analysis-for-system-traders/) article gated one market's signals on the state of another market; COT gates a market's signals on the positioning inside that same market. Instead of asking "what are bonds doing while I trade stocks," you ask "who is long this contract, and are they the smart money or the dumb money." Same discipline, different axis. The data is positional, not price-derived, which is exactly why it adds something a chart cannot. ## The three groups The report sorts every reportable position into three buckets, and the whole value of COT lives in keeping them separate. Commercial hedgers are the producers and consumers of the physical commodity. The farmer who grows the corn, the cereal company that buys it, the miner who digs the metal, the refiner who processes the crude. They trade futures to hedge a real-world business exposure, not to speculate, and they are the group that knows the physical supply and demand cold because they live in it. This is the smart money, and they are the group you watch. Large speculators are the non-commercial reportable traders: the funds, the managed-money accounts, the big trend-followers carrying positions large enough to clear the reporting threshold. They are speculating, not hedging, and as a group they tend to be trend-followers, long after a market has risen and short after it has fallen. Useful to track, but they are riding the move, not anticipating it. Small traders are everyone below the reporting threshold, the non-reportable positions backed out as a residual. This is the retail crowd, and the group's historical reputation is the unflattering one: small traders tend to be heaviest long at tops and heaviest short at bottoms. The connection to the retail-versus-institutional split in ["FX Is Not One Market: Retail vs Wholesale Structure"](https://aligrithm.com/fx-is-not-one-market-retail-vs-wholesale-structure/) is direct, the small trader here is the futures-market version of the retail flow that gets run over. $$ \\text{Net}\_{\\text{group}} \\;=\\; \\text{long contracts}\_{\\text{group}} - \\text{short contracts}\_{\\text{group}} $$ $$ \\text{Net}\_{\\text{commercial}} + \\text{Net}\_{\\text{large spec}} + \\text{Net}\_{\\text{small}} \\;=\\; 0 $$ For each group, the net position is its long contracts minus its short contracts. The first line is the single number you build everything on. The second line is the constraint that makes COT a closed system: futures are a zero-sum instrument, every long is somebody's short, so the three groups' net positions sum to zero. When commercials are heavily net short, the speculators and small traders must be net long by exactly that amount on the other side. You are never reading one group in isolation; you are reading who is on the opposite side of the smart money. ## What the report actually contains Beyond the headline long and short for each group, the report gives you the number of traders in each group, the number of spread positions, and the net and percent change in the open interest each group holds. The first thing you compute is the net long position per group, longs minus shorts, because the raw long and short totals move with the overall size of the market and the net strips that out. The catch that traps newcomers: the absolute level of net commercial position is close to meaningless on its own. Commercials in some markets are structurally always net short, because producers always have product to hedge, and in a market like silver the commercials have effectively never been net long in the modern record. So "commercials are net short 40,000 contracts" tells you nothing until you know whether 40,000 is high or low for this market's own history. The signal is the position relative to its own past extremes, not the raw number, which is the differencing-and-normalizing instinct from ["How to Build Stationary Indicators from Non-Stationary Prices"](https://aligrithm.com/how-to-build-stationary-indicators-from-non-stationary-prices/) and the reason the next article in this arc builds an index instead of trading the level. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## The three-week catch COT is not real-time, and pretending it is will hurt you. Positions are tallied after the market close on Tuesday. Auditing the data takes time, so the report is released later that week, and the practical reality across the publication schedule is that by the time the numbers are in your hands, they can be around three weeks old. You are looking at a photograph of where the big players stood, taken weeks before you got to see it. That lag is the central design constraint of any COT system, and it cuts two ways. It kills COT as a short-term timing tool outright; you cannot react to a position shift you learn about three weeks late. But it barely dents COT as a slow positioning gauge, because the commercial accumulation it tracks plays out over weeks and months, not days. A signal that itself leads the market by about two weeks, the property ["Why Commercials Are Counter-Trend (and Lead by 2 Weeks)"](https://aligrithm.com/why-commercials-are-counter-trend-and-lead-by-2-weeks/) builds on, survives a multi-week reporting lag where a fast signal would be destroyed by it. Match the data's frequency to the strategy's horizon or do not use it. Why this group of three matters at all comes down to information asymmetry. Commercials have the money and the inside view of physical supply and demand that the public never sees, so their aggregate positioning carries information. The report is valuable because it tells you how the large commercial traders are working, and the rest of this arc turns that into a tradable system: why the commercials lead, how to normalize their position into an index, and the one market family where the whole approach falls apart. ## Visualizing the three groups ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_212-visual-0.png) ## KEY POINTS - The Commitment of Traders report is a weekly census of futures positioning: how many contracts each kind of trader is long and short in every major market. It is positional data, not price-derived, so it adds what a chart cannot. - It sorts positions into three groups that behave differently. Commercial hedgers (producers and consumers) are the smart money with the physical-market view; large speculators are trend-following funds riding the move; small traders are the retail residual, long at tops and short at bottoms. - Build everything on net position: long contracts minus short contracts per group. Futures are zero-sum, so the three groups' nets sum to zero, you are always reading who sits opposite the smart money. - The report also gives trader counts, spreads, and changes in open interest, but the net per group is the workhorse number. - The absolute level of net commercial position is close to meaningless alone. Commercials are structurally net short in many markets (effectively always in silver), so the signal is the position relative to this market's own past extremes, not the raw number. - COT is roughly three weeks old by publication: tallied Tuesday's close, audited, released later. It is a slow positioning gauge, not a short-term timing tool. - The lag survives because commercial accumulation plays out over weeks and the signal leads by about two weeks. Match the data frequency to a multi-week horizon or do not use it. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Trading Systems and Methods - Perry Kaufman (Amazon)](https://www.amazon.com/s?k=Trading+Systems+and+Methods+Perry+Kaufman&ref=aligrithm.com) - [Cybernetic Trading Strategies - Murray Ruggiero (Amazon)](https://www.amazon.com/s?k=Cybernetic+Trading+Strategies+Murray+Ruggiero&ref=aligrithm.com) - [The Art of Currency Trading - Brent Donnelly (Amazon)](https://www.amazon.com/s?k=The+Art+of+Currency+Trading+Brent+Donnelly&ref=aligrithm.com) ### 4.42 Money Supply, Confidence, and Unemployment Duration as Rate Predictors URL: https://aligrithm.com/money-supply-confidence-and-unemployment-duration-as-rate-predictors/ Last updated: 2026-07-27T13:24:35.000Z Inflation is the loudest driver of interest rates, but it is not the only one, and in the stretches where inflation sits quiet, rates still move. Three slower fundamentals do the moving: how much money is sloshing through the system, how confident consumers feel, and how long the average jobless person stays jobless. Each one is a window into the speed of the economy, and the speed of the economy sets the demand for money, which sets rates. The ["Predicting Interest Rates from Inflation: The Real-Rate Ratio"](https://aligrithm.com/predicting-interest-rates-from-inflation-the-real-rate-ratio/) article handled the inflation spoke; this one adds the activity spokes that fire when inflation is not the story. The common thread is that rates follow growth. A faster economy wants more credit, bids up the price of money, and pushes rates higher. A slowing economy does the reverse. So any indicator that leads the growth rate is, secondhand, an indicator that leads rates. That is the logic behind treating copper as an activity proxy in ["Copper as an Economic Activity Indicator,"](https://aligrithm.com/copper-as-an-economic-activity-indicator/) and these three series are the same idea sourced from monetary and labor data instead of an industrial metal. ## Money supply: M1, M2, M3 Money supply is measured in widening circles, and you should know which circle you are quoting. M1 is the narrow one: money held as cash or in checking accounts, the stuff you can spend right now. M2 adds the next ring out, time deposits and CDs, money that is one small step from spendable. M3 is the widest, M2 plus the assets and liabilities of financial institutions that can be converted into spendable form. $$ \\text{M1} \\;=\\; \\text{cash} + \\text{checking} $$ $$ \\text{M2} \\;=\\; \\text{M1} + \\text{time deposits, CDs} $$ $$ \\text{M3} \\;=\\; \\text{M2} + \\text{convertible institutional assets/liabilities} $$ Each layer adds money that is a little harder to spend on a moment's notice. The mechanism that connects this to rates is direct: the more money in the system, the more the economy grows, and that growth translates into higher rates. M2 is the workhorse for this. M2 and rates are correlated, and money supply is predictive for rates, with a stronger pull on long-term rates than on short. More money in the system today plants the growth that long rates are trying to price years out, which is why the money-supply signal reaches further down the curve than the inflation signal does. Use the growth rate of M2, not its level. The absolute dollar amount of M2 climbs forever; what matters is whether it is accelerating or decelerating relative to its own history, the same stationarizing instinct from ["How to Build Stationary Indicators from Non-Stationary Prices."](https://aligrithm.com/how-to-build-stationary-indicators-from-non-stationary-prices/) A rising M2 growth rate is fuel for future activity and future rates. ## Consumer confidence Confidence is the psychology spoke. If consumers feel positive, they spend and borrow, growth increases, and rates follow growth up. If they turn cautious, they save, growth cools, and rates ease. The series is soft and sentiment-driven, so it is noisier than money supply, but it leads because spending decisions get made on feeling before they show up in hard activity data. The honest caveat is that confidence is the weakest of the three. Sentiment surveys whip around, get revised, and sometimes diverge from what people actually do with their wallets. Treat confidence as a confirming vote, not a standalone trigger. When confidence agrees with money supply and labor data, you have a coherent growth story; when it fights them, trust the harder series and wait. ## Unemployment duration The labor spoke is the sharpest of the three, and it is not the unemployment rate. It is the average duration that someone stays unemployed. Duration measures the texture of the labor market in a way the headline rate misses. When people who lose jobs find new ones fast, the economy is humming, demand for money is firm, and rates hold up or rise. When the average jobless spell stretches out, the economy is slow, hiring is weak, and rates drift down. The logic is clean: the longer someone is out of work, the slower the economy is running, and the lower rates will go. Duration is a better read than the raw unemployment rate because the rate can fall for ugly reasons, like discouraged workers leaving the labor force, while rising duration unambiguously says the people who are unemployed are stuck, which only happens when hiring has stalled. Duration lengthening is a slowdown signal, and a slowdown signal is a rates-down signal. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Putting the three together These are slow, low-frequency series, and the right way to use them is as a composite read on the direction of the economy, which then conditions your rate view. $$ \\text{growth pressure} \\;\\sim\\; \\big(\\text{M2 growth} \\uparrow\\big)\\;+\\;\\big(\\text{confidence} \\uparrow\\big)\\;+\\;\\big(\\text{unemployment duration} \\downarrow\\big) \\;\\Longrightarrow\\; \\text{rates bias up} $$ Read it as a vote, not an equation to evaluate to a decimal. Rising M2 growth, rising confidence, and shortening unemployment duration all point the same way: the economy is speeding up, and rates have an upward bias. Flip all three and rates have a downward bias. The value of stacking them is that they fail at different times. Money supply can mislead during a credit crunch when money exists but does not circulate; confidence can spike on a headline and fade; duration lags the turn. No single one is trustworthy alone, so you want agreement across the set before leaning on the macro view, and you treat disagreement as a signal to wait rather than to average. Mind the publication lag and revisions. These series arrive late and get revised, sometimes heavily, so a real-time backtest has to use the vintage of the data you would have actually had on the day, not the polished number in today's database. Backtesting macro signals on revised data is a quiet form of look-ahead, and it flatters every fundamental system that has ever been built. This is the kind of subtle leakage ["How to Test Indicator Thresholds Without Fooling Yourself"](https://aligrithm.com/how-to-test-indicator-thresholds-without-fooling-yourself/) warned about, transplanted from price indicators to economic releases. Where this lands: money supply, confidence, and unemployment duration give you the non-inflation read on rates, strongest on the slow end of the curve. They pair with the inflation signal from the Real-Rate Ratio, and they set up the next distinction in this arc, ["Short Rates Price the Present, Long Rates Price the Future"](https://aligrithm.com/short-rates-price-the-present-long-rates-price-the-future/), which explains why money supply reaches the long end while inflation grips the short end. ## Visualizing the three rate predictors ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_209-visual-0.png) ## KEY POINTS - When inflation is quiet, rates still move on the speed of the economy. Three slow fundamentals read that speed: money supply, consumer confidence, and unemployment duration. Rates follow growth. - Money supply comes in widening circles: M1 (cash and checking), M2 (M1 plus time deposits and CDs), M3 (M2 plus convertible institutional assets). More money means more growth means higher rates. - M2 is the workhorse: correlated with rates and predictive, with a stronger pull on long-term rates than short. Use M2 growth rate, not its ever-rising level. - Consumer confidence is the psychology spoke and the weakest of the three. Positive consumers spend and borrow, lifting growth and rates. Noisy and revision-prone; use as a confirming vote, not a standalone trigger. - Unemployment duration, not the headline rate, is the sharpest labor signal. Longer average jobless spells mean a slower economy and lower rates. Duration avoids the trap of a rate that falls because workers left the labor force. - Use the three as a composite vote on economic direction. They fail at different times, so require agreement before leaning on the view and treat disagreement as a reason to wait. - These series are published late and revised. Backtest on the data vintage you would have had that day, or you import look-ahead bias into every fundamental signal. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Trading Systems and Methods - Perry Kaufman (Amazon)](https://www.amazon.com/s?k=Trading+Systems+and+Methods+Perry+Kaufman&ref=aligrithm.com) - [Cybernetic Trading Strategies - Murray Ruggiero (Amazon)](https://www.amazon.com/s?k=Cybernetic+Trading+Strategies+Murray+Ruggiero&ref=aligrithm.com) - [The Art of Currency Trading - Brent Donnelly (Amazon)](https://www.amazon.com/s?k=The+Art+of+Currency+Trading+Brent+Donnelly&ref=aligrithm.com) ### 4.43 Short Rates Price the Present, Long Rates Price the Future URL: https://aligrithm.com/short-rates-price-the-present-long-rates-price-the-future/ Last updated: 2026-07-27T13:24:36.000Z A 90-day T-bill and a 30-year bond are both loans to the same government, but they are pricing two different things. The bill prices what the economy is doing right now. The bond prices what the economy might do for the next three decades. That single distinction explains a fact that trips up traders who treat "interest rates" as one number: the inflation signal that works so well on short rates gets weaker and less reliable the further out the curve you go. The "Predicting Interest Rates from Inflation: The Real-Rate Ratio" article built a clean inflation-to-rates rule; this one explains where that rule applies and where it loses its grip. Short rates are anchored to current economic activity. Long rates are anchored to the perceived effects of future activity. Today's inflation is a hard fact, so it grips the short end tightly. Future inflation is a forecast, a hope, a fear, and a hundred other things, so it grips the long end loosely. Same mechanism, different time horizon, very different reliability. ## Why inflation grips the short end The short end has nowhere to hide from current conditions. A 90-day bill matures in three months, so the only inflation that matters to it is the inflation happening now and over the next quarter, which is observable. A lender pricing a three-month loan looks at today's inflation, demands a real return over it, and sets the yield. There is no room for a long story to override the present fact. That is why the Real-Rate Ratio works best on short rates. The inputs, current inflation and current yield, are both about now, and the relationship between them is tight because neither side gets to argue about a distant future. When inflation runs hot relative to short yields, the squeeze is real and immediate, and short rates correct. The negative-real-rate setup that preceded the 1993 bond bear started as a short-end signal precisely because the short end has the cleanest read on present inflation. ## Why the link loosens on the long end The long bond has to price decades it cannot see. A 30-year yield is a bet on the average short rate over 30 years, plus a premium for the risk of being wrong, and that average depends on inflation, growth, policy, and politics that have not happened yet. Today's inflation is one input among many, and it gets diluted. The relationship between long rates and inflation is real but not as reliable as the short-rate version, because the long end is forecasting and forecasts are loose. $$ \\text{short rate} \\;\\approx\\; f\\big(\\text{current inflation, current activity}\\big) $$ $$ \\text{long rate} \\;\\approx\\; f\\big(\\text{expected average short rate over the term, term premium}\\big) $$ The short rate is a function of things you can measure today. The long rate is a function of an expectation about the future path of short rates plus a term premium for uncertainty. Because the long rate's main input is an expectation, it responds to changes in the story, not changes in the present, which is why a long-end model built on current inflation alone underperforms the short-end version. Two consequences fall out of this. First, long rates can be kept artificially low for extended stretches. During long periods of economic growth with low inflation, the market believes the calm will continue and prices the long end below where current data alone would put it. The term premium compresses because nobody is demanding much compensation for a future that looks placid. That belief can persist for years, and a trader shorting bonds on a current-inflation model will bleed the whole time. Second, when long rates do respond to inflation, the response is amplified. A 1% increase in inflation has historically produced roughly a 2.5% increase in long-term rates. The long end sits still while the market believes the future is calm, then moves more than the inflation change itself once belief shifts, because a change in current inflation gets extrapolated across the whole remaining term. Long rates are stickier and then jumpier than the short end. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## What this means for trading rate markets Match the signal to the maturity. The inflation-based tools, the Real-Rate Ratio and its threshold rule, belong on the short end where the link is tight and the inputs are observable. Do not port a short-rate inflation model onto the 30-year and expect the same hit rate; you are asking a present-tense signal to forecast a future-tense instrument. For the long end, weight the forward-looking and growth fundamentals more heavily. Money supply reaches the long end better than inflation does, the point from ["Money Supply, Confidence, and Unemployment Duration as Rate Predictors,"](https://aligrithm.com/money-supply-confidence-and-unemployment-duration-as-rate-predictors/) because money supply plants the future growth the long bond is trying to price. Expectations, policy stance, and the slow growth fundamentals carry more of the load out there, and you accept that the long end will be wrong-footed for long stretches when the market is committed to a calm-future narrative. This also reframes the yield curve itself as information. The gap between long and short rates is the gap between the future the market expects and the present it can measure. When that gap compresses or inverts, the market is saying its forecast of the future diverges sharply from current conditions, and that divergence is a signal in its own right, the kind of cross-instrument read that ["Using Bonds to Filter Equity Signals"](https://aligrithm.com/using-bonds-to-filter-equity-signals/) used to gate equity trades. Short and long rates disagreeing is not noise; it is the present and the future pricing different things, which is exactly what they are built to do. The takeaway is a placement rule, not a new indicator. Inflation signals go on the short end. Growth and expectation signals go on the long end. The curve between them is the market's forecast error waiting to be read. The next article, ["Long-Term Market Timing from Fundamentals, Not Charts"](https://aligrithm.com/long-term-market-timing-from-fundamentals-not-charts/), takes this whole fundamental toolkit and uses it to time multi-year trends instead of quarters. ## Visualizing the maturity split ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_210-visual-0.png) ## KEY POINTS - A short bill and a long bond price two different things: the bill prices current economic activity, the bond prices the expected effects of future activity. Treating "interest rates" as one number hides this. - Inflation grips the short end tightly because a 90-day bill only cares about inflation it can observe now. That is why the Real-Rate Ratio works best on short rates. - The long end prices an expectation: the average short rate over the term plus a term premium. Current inflation is one diluted input among many, so the inflation-to-long-rate link is real but less reliable. - Long rates can stay artificially low for years when the market believes in a calm, low-inflation future and compresses the term premium. A short-rate model shorted against that belief bleeds the whole time. - When long rates do respond, they amplify: roughly a 2.5% long-rate move per 1% inflation change, because the change gets extrapolated across the whole term. The long end is stickier, then jumpier. - Match signal to maturity: inflation tools on the short end, growth and expectation fundamentals (money supply, policy) on the long end. Do not port a short-rate inflation model onto the 30-year. - The yield curve is the gap between the expected future and the measured present. Compression or inversion signals that the market's forecast diverges from current conditions. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Trading Systems and Methods - Perry Kaufman (Amazon)](https://www.amazon.com/s?k=Trading+Systems+and+Methods+Perry+Kaufman&ref=aligrithm.com) - [Cybernetic Trading Strategies - Murray Ruggiero (Amazon)](https://www.amazon.com/s?k=Cybernetic+Trading+Strategies+Murray+Ruggiero&ref=aligrithm.com) - [The Art of Currency Trading - Brent Donnelly (Amazon)](https://www.amazon.com/s?k=The+Art+of+Currency+Trading+Brent+Donnelly&ref=aligrithm.com) ### 4.39 Day-of-Week Effects That Actually Have a Cause URL: https://aligrithm.com/day-of-week-effects-that-actually-have-a-cause/ Last updated: 2026-07-27T13:24:34.000Z Day-of-week effects are where seasonal research goes to embarrass itself. Five trading days, two directions, and a long history give you ten obvious buckets to mine, and you will always find a few that look significant. The Monday effect in stocks got published, traded, written into folklore, and then half of it decayed. So the bar for taking a day-of-week pattern seriously has to be high, and the thing that clears the bar is never the backtest. It is a cause. A day-of-week effect is worth a look only when you can name why that day, in that market, and the why has to be a mechanism, not a story you reverse-engineered after seeing the chart. This is the cleanest possible test of the principle from ["The Difference Between Explanation and Prediction in Markets."](https://aligrithm.com/the-difference-between-explanation-and-prediction-in-markets/) A day-of-week bias is the kind of pattern that is trivial to explain after the fact and brutal to predict forward. The ones below survive not because their stats are unbeatable but because the mechanism behind each is concrete enough to say when it should hold and when it should break. ## S&P 500: Monday up, Friday down The flagship behavioral seasonal. Stocks have shown an upward bias on Mondays and a softer tone into Friday, and the standard mechanism is weekend positioning. Traders flatten or lighten risk before the weekend, because two days of headline exposure with no ability to react is a real risk, and they re-establish positions at the start of the week. Selling pressure into Friday, buying pressure on Monday. The flows are the cause; the calendar is just where they land. The honest part is that this one moved. The Monday effect was magnified after the 1987 crash, "Black Monday," which permanently changed how the market thinks about Monday risk, and the raw edge has weakened since the era when it was first documented. That is the signature of a behavioral seasonal, the fragile engine from ["The Three Engines of Seasonality: Fixed-Date, Floating-Date, and Behavioral."](https://aligrithm.com/the-three-engines-of-seasonality-fixed-date-floating-date-and-behavioral/) A pattern driven by habit gets watched, gets traded against, and shifts. The mechanism is real, which is why some version of the weekend-positioning bias persists in places, but the magnitude is not stable and you cannot trade it as if it were a constant. Treat the S&P day-of-week effect as the cautionary case study, not the moneymaker. ## Silver: Thursday up, conditioned on bonds Silver is the interesting one, because its day-of-week bias is not freestanding. It is conditional, and the condition is what makes it credible. Silver has carried an upward bias on Thursdays, but the better way to read it is through silver's role as an industrial-demand and economic-strength proxy. Silver is half precious metal, half industrial input, so it tends to firm when the economy looks strong. Strong economy also means rising rates and falling bond prices. So the silver bias shows up most on days when T-bonds are falling, and that relationship has held since around 1980. $$ \\text{Silver bias active} \\;\\Longleftrightarrow\\; \\big(\\text{day} = \\text{Thursday}\\big)\\;\\wedge\\;\\big(\\text{T-bonds falling}\\big) $$ Read it as a conditional seasonal: the day-of-week tilt is gated by an intermarket state. The Thursday effect alone is a weak, mineable number. The Thursday effect filtered to days when bonds are falling has a mechanism behind it, because falling bonds signal the economic strength that bids silver as an industrial metal. The cause is not "Thursday." The cause is "silver rises with economic strength, and the day-of-week pattern is the visible residue of that." This is the bridge from a behavioral curiosity to an intermarket signal, and it points straight at the approach in ["Intermarket Analysis for System Traders"](https://aligrithm.com/intermarket-analysis-for-system-traders/): a second market gates the first. A bare day-of-week bias becomes defensible the moment you can attach it to a cross-asset cause. ## Coffee: Thursday up, Monday down Coffee has shown an upward bias on Thursdays and a downward bias on Mondays. The mechanism is murkier than silver's and softer than the S&P's weekend story, which is itself the lesson. Coffee is a thinner, supply-shock-driven market, and weekly patterns there plausibly tie to the rhythm of physical trade, shipping schedules, weekend weather risk in growing regions building into the week, and dealer positioning around it. None of that is as clean as "the earth tilts" or "bonds price economic strength." So coffee sits in the suspicious tier. The pattern is real in the sample, but the cause is a hand-wave, and a hand-wave fails test four of the reliability checklist in ["Is Your Seasonal Real or Curve-Fit? A Reliability Checklist."](https://aligrithm.com/is-your-seasonal-real-or-curve-fit-a-reliability-checklist/) The right move is not to trade it on faith and not to dismiss it outright. The right move is to keep digging for the actual mechanism, weather risk into the week, a scheduled report, a roll or delivery rhythm, and only graduate it to tradable once the cause is named. Until then it is a flagged candidate, not a signal. Listing it here is the point: most day-of-week effects look like coffee, plausible and unproven, not like silver. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## How to handle any day-of-week claim Three rules, learned from the three cases above. First, demand the mechanism before the magnitude. The S&P, silver, and coffee differ not in how good their backtests look but in how solid their causes are. Rank by cause strength: silver's intermarket gating is the strongest, the S&P's weekend flows are real but decaying, coffee's is a guess. Trade in that order of confidence, not in order of backtest p-value. Second, look for the conditioning variable. The silver case shows the upgrade path: a weak standalone day-of-week bias often becomes a real signal once you find the state that gates it. The market does not repeat, it rhymes, the point of ["Why the Market Does Not Repeat, But Still Rhymes,"](https://aligrithm.com/why-the-market-does-not-repeat-but-still-rhymes/) and the rhyme in a day-of-week effect is usually some underlying driver that happens to express itself on certain days. Find the driver and you have something more stable than the day. Third, assume decay on the behavioral ones. Any day-of-week effect that runs on pure trader habit, the S&P Monday bias being the archetype, is competing with everyone else who read the same research. Behavioral edges erode as they spread. Size them small, monitor them, and do not be surprised when they fade or invert, because that is what the engine does. The deeper takeaway is that the best day-of-week effects stop being about the day. Silver's Thursday bias is really an economic-strength signal that leaks through on Thursdays; the day is the symptom, the intermarket state is the disease. That reframing is exactly what the next article in the arc, ["Seasonality as a Filter, Not a Standalone System"](https://aligrithm.com/seasonality-as-a-filter-not-a-standalone-system/), is built on: a seasonal earns its keep by combining with the cross-asset signals it is secretly a proxy for, not by standing alone on a calendar. ## Visualizing the three cases ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_206-visual-0.png) ## KEY POINTS - Day-of-week effects are trivial to mine: five days, two directions, long history. The bar for taking one seriously is a named mechanism, not a good backtest. - S&P 500 (Monday up, Friday down): driven by weekend positioning, traders flatten into Friday and re-enter Monday. Real mechanism, but magnified then destabilized by the 1987 crash and decayed since. The cautionary case, not the moneymaker. - Silver (Thursday up): the strongest case because it is conditional. Silver is an economic-strength proxy, so its bias shows most on days when T-bonds are falling (since \~1980). The cause is economic strength; the day is the residue. - Coffee (Thursday up, Monday down): real in-sample but the mechanism is a hand-wave. Sits in the suspicious tier; keep digging for the actual cause before trading it. - Rank day-of-week claims by cause strength, not by p-value: silver (intermarket-gated) over S&P (real but decaying) over coffee (guess). - Look for the conditioning variable. A weak standalone day-of-week bias often becomes a real signal once you find the state that gates it, as falling bonds gate silver. - Assume decay on behavioral effects. Pure-habit edges like the Monday bias erode as the research spreads. Size small, monitor, expect fade or inversion. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Trading Systems and Methods - Perry Kaufman (Amazon)](https://www.amazon.com/s?k=Trading+Systems+and+Methods+Perry+Kaufman&ref=aligrithm.com) - [Cybernetic Trading Strategies - Murray Ruggiero (Amazon)](https://www.amazon.com/s?k=Cybernetic+Trading+Strategies+Murray+Ruggiero&ref=aligrithm.com) - [The Art of Currency Trading - Brent Donnelly (Amazon)](https://www.amazon.com/s?k=The+Art+of+Currency+Trading+Brent+Donnelly&ref=aligrithm.com) ### 4.40 Seasonality as a Filter, Not a Standalone System URL: https://aligrithm.com/seasonality-as-a-filter-not-a-standalone-system/ Last updated: 2026-07-27T13:24:34.000Z A seasonal that passed every test in the reliability checklist is still a weak signal. That is not a failure of your research, it is the nature of the thing. A well-founded seasonal bias might shift the odds from 50/50 to 56/44 over a calendar window. Real, repeatable, and nowhere near enough to trade on its own once you subtract slippage, commission, and the years it goes the wrong way. Traders who build standalone seasonal systems learn this the expensive way: the edge is too thin to survive contact with costs, and the variance year to year is too high to stomach. The fix is to stop asking the seasonal to be the system. Use it as a filter. A modest, well-understood bias is exactly the kind of input that improves a system it does not have to carry. It gates other signals, tilts position size, or confirms a setup you already had a reason to take. This is the same move the whole intermarket arc is built on, the logic from ["Intermarket Analysis for System Traders"](https://aligrithm.com/intermarket-analysis-for-system-traders/): a second source of information decides whether the first source's signals are live. A seasonal is just another such source, and a cheap one, because the calendar costs nothing to compute. ## Why standalone seasonality fails Run the math on a clean seasonal and the problem is obvious. Suppose your best, fully-vetted seasonal wins 56% of years in a 15-day window with a small average edge. Across 20 years that is 11 or 12 winning years. The dispersion around that is large, the average trade is small, and once you charge realistic costs for entering and exiting, the thin per-trade edge can flip negative. You are taking real variance for a sliver of expected return. $$ E\[\\text{net}\] \\;=\\; \\underbrace{p\\cdot W - (1-p)\\cdot L}\_{\\text{gross seasonal edge}} \\;-\\; \\underbrace{c}\_{\\text{costs}} $$ $$ \\text{thin gross edge} - \\text{costs} \\;\\approx\\; 0 \\quad\\text{or worse} $$ The gross edge is the win probability times the average win minus the loss probability times the average loss. For a 56/44 seasonal with comparable win and loss sizes, that gross number is small, and subtracting round-trip costs c eats most or all of it. The seasonal is not wrong; it is just too small to stand alone. A 56% directional bias is genuinely useful, but only as a thumb on the scale of a decision you were already going to make, not as the decision itself. ## The filter construction The clean way to use a seasonal is as an AND condition on a signal you already trust. Your base system generates a long signal; the seasonal says the calendar bias is also long; you take the trade only when both agree. When they disagree, you stand down or size down. $$ \\text{Trade}\_{\\text{long}} \\;=\\; \\big(\\text{base signal} = \\text{long}\\big)\\;\\wedge\\;\\big(\\text{seasonal bias} \\geq 0\\big) $$ $$ \\text{size} \\;=\\; \\text{base size} \\times \\big(1 + k\\cdot \\text{seasonal strength}\\big) $$ The first line is the gate: take the long only when your primary system and the seasonal point the same way. The second line is the softer version, a tilt instead of a switch, where the seasonal scales position size up when its bias agrees and down when it does not, with k controlling how much you let the calendar move your bet. Both are ways of letting a weak-but-honest signal improve a stronger one without being asked to generate trades by itself. The seasonal contributes where it is good, narrowing the field, and never has to be right on its own. This is structurally identical to the divergence filter in ["Intermarket Divergence as a Trading Filter."](https://aligrithm.com/intermarket-divergence-as-a-trading-filter/) There, a second market's disagreement with the traded market flagged or vetoed a trade. Here, the calendar's agreement or disagreement does the same job. The seasonal is one more gate in the same machine, and it composes with the others. You can require your base signal, the intermarket state, and the seasonal bias to all line up, each one independently thinning the trade set toward the cleaner outcomes. ## Stacking the seasonal with intermarket filters The strongest use combines the seasonal with the cross-asset filters from the intermarket arc, because the best seasonals are often proxies for those same forces anyway. The silver case from ["Day-of-Week Effects That Actually Have a Cause"](https://aligrithm.com/day-of-week-effects-that-actually-have-a-cause/) is the template: silver's Thursday bias is really an economic-strength signal that the bond market also reports. Stacking the silver seasonal with a "bonds falling" filter is not two independent edges, it is the same cause confirmed two ways, which is a legitimate way to raise conviction on the days the story actually holds. The general pattern is a layered gate. The intermarket state sets the macro backdrop (is the external market confirming?), the seasonal sets the calendar backdrop (is the bias with us this week?), and the base signal sets the entry. Each layer removes trades that the layer above would have taken blindly. The result trades less and trades cleaner. You give up frequency, which is the correct trade when each filter genuinely improves the forward-return distribution and the alternative is a standalone seasonal that costs you money. Two cautions before you stack everything onto everything. First, every filter you add is another degree of freedom and another chance to overfit, the warning from the reliability checklist in ["Is Your Seasonal Real or Curve-Fit? A Reliability Checklist."](https://aligrithm.com/is-your-seasonal-real-or-curve-fit-a-reliability-checklist/) Adding the seasonal as a filter has to clear the same bar as adding any condition: does it improve out-of-sample results, or did you just find a combination that flattered the backtest? Count the seasonal against your degrees of freedom like any other knob. Second, do not stack correlated filters and pretend they are independent confirmation. Silver's seasonal and the bond filter share a cause, so requiring both is fine as a conviction check but does not give you two separate edges. Stacking three filters that are all secretly the same macro signal narrows your trade count to nearly zero and gives you false confidence in the handful that survive. Know which of your filters are independent and which are the same force wearing different clothes. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Where this leaves seasonality The honest verdict closes the loop the whole arc has been building toward. Seasonality is real, it is mostly weak, and its correct job is supporting, not starring. A seasonal that survives the reliability checklist is a modest, well-founded bias, and a modest well-founded bias is worth exactly what a good filter is worth: it cleans up the distribution of a system that can already stand on its own. Ask it to be the system and it dies to costs. Ask it to gate or tilt a system, and it earns its keep. That reframing, from standalone pattern to one input among several, is the through-line of this pillar. No single market trades alone, and no single signal carries a book. The seasonal takes its place next to the intermarket filters as one more way to condition the trades you take on information the price chart in front of you does not show. ## Visualizing the filter stack ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_207-visual-0.png) ## KEY POINTS - A seasonal that passes every reliability test is still a weak signal, maybe a 56/44 bias. That is the nature of seasonality, not a research failure. - Standalone seasonal systems die to costs. A thin gross edge minus realistic round-trip costs lands near zero or negative, and the year-to-year variance is high for a small expected return. - Use the seasonal as a filter, not the system: an AND gate on a signal you already trust, or a tilt that scales position size up when the bias agrees and down when it disagrees. - This is the intermarket logic applied to the calendar: a second source of information decides whether the first source's signals are live. The seasonal is a cheap extra gate that composes with the others. - The strongest use stacks the seasonal with cross-asset filters, because the best seasonals are proxies for the same forces. Silver's Thursday bias plus a falling-bonds filter is one cause confirmed two ways. - Every added filter is another degree of freedom. Adding a seasonal gate must clear the same overfitting bar as any condition: does it improve out-of-sample results, counted against your degrees of freedom. - Do not treat correlated filters as independent confirmation. Filters that share a cause raise conviction but are not two separate edges; stacking several of the same hidden signal manufactures false confidence. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Trading Systems and Methods - Perry Kaufman (Amazon)](https://www.amazon.com/s?k=Trading+Systems+and+Methods+Perry+Kaufman&ref=aligrithm.com) - [Cybernetic Trading Strategies - Murray Ruggiero (Amazon)](https://www.amazon.com/s?k=Cybernetic+Trading+Strategies+Murray+Ruggiero&ref=aligrithm.com) - [The Art of Currency Trading - Brent Donnelly (Amazon)](https://www.amazon.com/s?k=The+Art+of+Currency+Trading+Brent+Donnelly&ref=aligrithm.com) ### 4.41 Predicting Interest Rates from Inflation: The Real-Rate Ratio URL: https://aligrithm.com/predicting-interest-rates-from-inflation-the-real-rate-ratio/ Last updated: 2026-07-27T13:24:35.000Z Lenders are not charities. Hand someone money for 90 days and you want back more purchasing power than you gave up, which means the rate you charge has to clear inflation with room to spare. That single piece of self-interest is why short-term interest rates track inflation, and it is the most reliable fundamental link in macro markets. Over roughly five decades, T-bill yields ran about twice the inflation rate. Lenders demanded a real return, and that real return showed up as yields sitting at a multiple of inflation rather than a hair above it. The link is real but not constant, and the interesting trading information lives in the moments it stretches too far. When yields fall too close to inflation, or below it, the lender is being paid nothing or less than nothing in real terms, and that situation does not last. Rates snap back up. The ["Gold, Dollar, and Rates: A Practical Intermarket Map"](https://aligrithm.com/gold-dollar-and-rates-a-practical-intermarket-map/) article laid out the web connecting inflation, rates, gold, and the dollar; this one turns the inflation-to-rate spoke of that web into a single number you can compute and threshold. ## The negative-real-rate warning Strip out the precise multiple and look at the danger zone. A negative real rate means inflation is running hotter than the T-bill yield. You lend at 3%, inflation is 4%, you lose 1% of purchasing power for the privilege of lending. That is unstable. The market corrects it by pushing yields up, and the correction tends to be violent because it has been suppressed. The cleanest historical example is 1993\. The real rate went negative, inflation outran the T-bill yield, and it happened right before a severe bear market in bonds. Bond prices move inverse to yields, so "rates about to rise" and "bond bear market" are the same sentence. The negative real rate was the tell, and it fired before the damage. This matters because the gold standard mattered: once the US dropped gold convertibility, inflation became a better measure of interest rates, since there was no longer a metal anchor competing with inflation expectations to set the price of money. So the first, crude signal is just the sign of the real rate. Inflation above yield is a loaded spring. It does not tell you the day rates rise, but it tells you the configuration that precedes rate rises, and that is a backdrop worth respecting before you go long bonds. ## Turning it into a ratio The sign of the real rate is binary and throws away information. A ratio keeps the magnitude. Build the relationship between inflation and yield as a single bounded number that reads near zero when the lender is getting squeezed and near one when the lender is fat. $$ \\text{Ratio} \\;=\\; 1 - \\frac{\\text{inflation}}{\\text{yield}} $$ Read it slowly. The fraction inflation over yield is how much of your nominal yield gets eaten by inflation. If inflation is 2% and yield is 4%, inflation eats half your yield, the fraction is 0.5, and the Ratio is 0.5\. If inflation equals yield, the fraction is 1 and the Ratio is 0, the break-even point where the real rate is exactly zero. If inflation exceeds yield, the fraction goes above 1 and the Ratio goes negative, the negative-real-rate danger zone. A high Ratio means the lender keeps most of the yield in real terms; a low or negative Ratio means inflation is winning and rates are under pressure to rise. One number now carries both the sign and the severity. The Ratio drifts slowly with the rate regime, so an absolute reading is more useful than a level you have to eyeball. The fix is to measure it against its own recent average, the stationarizing move from ["How to Build Stationary Indicators from Non-Stationary Prices."](https://aligrithm.com/how-to-build-stationary-indicators-from-non-stationary-prices/) $$ \\text{RatioOsc} \\;=\\; \\text{Ratio} - \\text{Avg}(\\text{Ratio},\\, 20) $$ Subtract the 20-period average of the Ratio from the current Ratio. Now you have an oscillator centered on zero: positive when the lender's real cushion is better than its recent norm, negative when it is deteriorating. The oscillator catches the change in the inflation-yield balance even when the absolute Ratio is still in a comfortable-looking range, because a falling real cushion is the early part of the move that ends in higher rates. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## The combined rule Put the level and the change together with a confirmation, and you get a concrete signal for short rates rising. $$ \\big(\\text{Ratio} < 0.2 \\;\\;\\text{OR}\\;\\; \\text{RatioOsc} < 0\\big) \\;\\;\\text{AND}\\;\\; \\big(\\text{yield} > \\text{yield}\[3\]\\big) \\;\\Longrightarrow\\; \\text{90-day rate likely to rise} $$ Three conditions. Ratio below 0.2 means the lender's real cushion is thin, inflation is eating most of the yield, the loaded-spring state. Or RatioOsc below zero means the cushion is deteriorating relative to its recent norm even if the level is not yet extreme. Either of those raises the flag. Then the AND clause demands confirmation: the current yield is above the yield three periods ago, so rates are already turning up, not just theoretically under pressure. The fundamental condition says "rates should rise," and the yield-versus-yield-three-bars-back check says "and they have started to." Requiring both keeps you from fighting a market that has not yet agreed with your macro thesis. The 0.2 threshold and the 20-period lookback and the 3-bar confirmation are arbitrary, and you should treat them as such. They are starting points, not laws, and they have to survive the same threshold testing as any other parameter, the discipline from ["How to Test Indicator Thresholds Without Fooling Yourself."](https://aligrithm.com/how-to-test-indicator-thresholds-without-fooling-yourself/) Pick them on one stretch of history and trade them blind on the next and you will learn what overfitting a macro rule costs. ## What this is and is not This is a fundamental filter for the direction of short rates, not a precision timing tool. It tells you the configuration that precedes rising rates and a falling bond market, with one confirmation that the move has begun. It does not tell you the magnitude or the exact week. The negative-real-rate setup in 1993 preceded a bond bear market, but "preceded" did some heavy lifting; the spring can stay loaded longer than your position can stay solvent. The honest framing is the one from ["The Difference Between Explanation and Prediction in Markets."](https://aligrithm.com/the-difference-between-explanation-and-prediction-in-markets/) The inflation-to-rates mechanism is a clean explanation, lenders demand a real return, and a clean explanation earns the rule a place in your toolkit. It does not promise the rule predicts on a schedule. Use the Real-Rate Ratio as a slow backdrop filter on bond and short-rate positions, stack it with the intermarket map it came from, and the next article in this arc, ["Money Supply, Confidence, and Unemployment Duration as Rate Predictors"](https://aligrithm.com/money-supply-confidence-and-unemployment-duration-as-rate-predictors/), adds the non-inflation drivers that move rates when the inflation signal is quiet. ## Visualizing the Real-Rate Ratio ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_208-visual-0.png) ## KEY POINTS - Short-term rates track inflation because lenders demand a real return. Over roughly five decades, T-bill yields ran about twice the inflation rate, sitting at a multiple of inflation rather than just above it. - The link is real but not constant. The tradable information is in the stretch: when yields fall too close to or below inflation, the real rate is negative and rates tend to snap back up. - A negative real rate (inflation above yield) is a loaded spring. The 1993 example went negative right before a severe bond bear market, because rising yields mean falling bond prices. - Dropping the gold standard made inflation a better measure of rates, since no metal anchor competes to price money. - The Real-Rate Ratio, 1 minus inflation-over-yield, is one bounded number: near 1 when the lender keeps most of the yield in real terms, 0 at break-even, negative in the danger zone. RatioOsc subtracts its 20-period average to make it a zero-centered oscillator. - The combined rule: Ratio below 0.2 or RatioOsc below zero (fundamental pressure), AND yield above yield three bars back (confirmation the move started), flags short rates likely to rise. - The thresholds (0.2, 20, 3) are arbitrary and must survive out-of-sample threshold testing. This is a slow directional backdrop filter for bonds and short rates, not a precision timing tool. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Trading Systems and Methods - Perry Kaufman (Amazon)](https://www.amazon.com/s?k=Trading+Systems+and+Methods+Perry+Kaufman&ref=aligrithm.com) - [Cybernetic Trading Strategies - Murray Ruggiero (Amazon)](https://www.amazon.com/s?k=Cybernetic+Trading+Strategies+Murray+Ruggiero&ref=aligrithm.com) - [The Art of Currency Trading - Brent Donnelly (Amazon)](https://www.amazon.com/s?k=The+Art+of+Currency+Trading+Brent+Donnelly&ref=aligrithm.com) ### 4.37 Calculating a Seasonal: Raw Change vs Detrended vs Standardized URL: https://aligrithm.com/calculating-a-seasonal-raw-change-vs-detrended-vs-standardized/ Last updated: 2026-07-27T13:24:32.000Z You decided corn has a seasonal. Now you have to turn that hunch into a number, and the number you get depends entirely on how you compute it. Three traders can look at the same twenty years of corn and produce three different seasonal curves, all "correct," all useless to each other, because one averaged raw price changes, one detrended first, and one standardized to a yearly range. The method is not a footnote. It decides what the seasonal even means and whether you can compare it across years or across markets. The problem underneath is the one the article ["How to Build Stationary Indicators from Non-Stationary Prices"](https://aligrithm.com/how-to-build-stationary-indicators-from-non-stationary-prices/) hammered on. Price is non-stationary. Corn at \\$3 a bushel in one decade and \\$7 in another is the same percentage wiggle in dollars that are twice as big. Average raw dollar moves across twenty years and the high-price years drown out the low-price years, so your seasonal is really a chart of which years happened to be expensive. Every method below is an attempt to get a seasonal that says "what the calendar does" instead of "which years had big absolute prices." Three methods, increasing in how much they correct for. Pick based on what you need the seasonal for. ## Method one: raw price change The simplest seasonal averages the price change at each calendar slot across all years in your sample. Slot can be a day-of-year, a week, or a month. For each slot you collect the change observed in every year and take the mean. $$ \\text{Seasonal}\_{\\text{raw}}(d) \\;=\\; \\frac{1}{Y}\\sum\_{y=1}^{Y} \\big( P\_{y,d} - P\_{y,d-1} \\big) $$ Read it plainly: for calendar slot d (say, the 5th trading day of July), look at every year y, take that day's price change in dollars, and average over all Y years. Do it for every slot and you have a raw seasonal curve. It is fast, it needs nothing but prices, and for a single market over a short, price-stable window it is fine. The flaw shows up the moment your sample spans a big change in price level. A \\$0.10 move in \\$7 corn and a \\$0.10 move in \\$3 corn count the same in dollars but are not the same event. Worse, if corn spent the back half of your sample at double the price, the recent years contribute bigger dollar swings and dominate the average. Your "seasonal" is partly a seasonal and partly a record of the price trend over the sample. That contamination is exactly why the next method exists. ## Method two: detrended seasonal Detrending removes the slow drift in price level so what is left is the wiggle around the trend. Subtract a long-term average from each price before you measure the seasonal. The long-term average is the trend; the price minus the trend is the deseasonalized-of-trend residual, and averaging that residual by calendar slot gives you a seasonal that is not riding the bull or bear market underneath. $$ \\tilde{P}\_{y,d} \\;=\\; P\_{y,d} - \\overline{P}\_{y}\\,,\\qquad \\overline{P}\_{y} \\;=\\; \\frac{1}{N}\\sum\_{i=1}^{N} P\_{y,i} $$ $$ \\text{Seasonal}\_{\\text{detrended}}(d) \\;=\\; \\frac{1}{Y}\\sum\_{y=1}^{Y} \\tilde{P}\_{y,d} $$ The first line subtracts a long-term average from each price. Here the average runs over the year (or a rolling long window), so each year's prices are expressed as a deviation from that year's own level. A price above its yearly average is positive, below is negative. The second line averages those deviations by calendar slot. The result tells you where in the year price tends to sit above or below its own trend, which is the actual seasonal shape with the trend stripped out. This is the same move ATR normalization makes for volatility, described in ["Why ATR Normalization Is More Than a Volatility Trick"](https://aligrithm.com/why-atr-normalization-is-more-than-a-volatility-trick/): divide out the part that contaminates the comparison so the remainder is comparable across regimes. Detrending kills the level contamination. It does not yet kill the scale contamination. A detrended seasonal in dollars is still in dollars, so a volatile high-priced year still contributes bigger deviations than a calm cheap year. If you want to average across markets, or across decades with very different volatility, you need one more step. ## Method three: standardized seasonal Standardizing rescales each year onto a common ruler so every year contributes equally regardless of its price level or its range. The straightforward version uses the year's own high and low to map price into a 0-to-1 band. $$ S\_{y,d} \\;=\\; \\frac{P\_{y,d} - \\text{Low}\_{y}}{\\text{High}\_{y} - \\text{Low}\_{y}} $$ $$ \\text{Seasonal}\_{\\text{std}}(d) \\;=\\; \\frac{1}{Y}\\sum\_{y=1}^{Y} S\_{y,d} $$ For each year, find that year's highest and lowest price. The first line scales every price in the year to where it sits between them: 0 at the year's low, 1 at the year's high, 0.5 halfway up the range. Now a year that traded \\$2 to \\$3 and a year that traded \\$6 to \\$9 are on the same 0-to-1 scale, and neither dominates. The second line averages the scaled values by calendar slot. The seasonal now reads as "on this date, price tends to sit at, say, 0.7 of the year's range," which is comparable across years and across markets. Standardizing buys you cross-comparability at a cost you should name out loud. Using the full-year high and low means you are peeking at the whole year to scale each day, which is look-ahead if you ever try to compute it in real time on the current, unfinished year. For research on completed historical years it is fine; for a live signal you have to scale with data available up to that bar, not the year's eventual high and low. This is the kind of subtle in-sample leakage the article ["How to Build Stationary Indicators from Non-Stationary Prices"](https://aligrithm.com/how-to-build-stationary-indicators-from-non-stationary-prices/) flagged, and it is easy to commit by accident when you copy a textbook seasonal formula that quietly assumes you already know the year's range. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Which one to use Match the method to the job, and do not pretend the choice is free. $$ \\begin{array}{ll} \\text{single market, short stable window} & \\rightarrow \\;\\text{raw change}\\\\\[4pt\] \\text{single market, long sample with trend} & \\rightarrow \\;\\text{detrended}\\\\\[4pt\] \\text{cross-year or cross-market comparison} & \\rightarrow \\;\\text{standardized} \\end{array} $$ Raw change for a quick look at one market where price did not move much over the sample. Detrended once your sample is long enough that the bull or bear trend underneath would otherwise leak into the seasonal. Standardized when you need years or markets on a common ruler, accepting the look-ahead caveat for live use. The three are not competing answers to one question; they answer three different questions, and the mistake is computing one while believing you measured another. Whatever you pick, the calendar slot also matters, and it ties back to the engine. The article ["The Three Engines of Seasonality: Fixed-Date, Floating-Date, and Behavioral"](https://aligrithm.com/the-three-engines-of-seasonality-fixed-date-floating-date-and-behavioral/) made the point that a floating-date seasonal has to be aligned to its event before averaging. None of these formulas fix that for you. If you average an unemployment-release effect by raw calendar date, the cleanest detrending and standardizing in the world still gives you noise, because you binned the data on the wrong axis. Get the alignment right first, then choose your computation. A computed seasonal is an average, and an average hides its own reliability. A curve that says "up 0.7 of the range on this date" could be ten years agreeing or two huge years dragging eight flat ones. The number on the chart does not tell you which. That is the next problem, and the article ["Is Your Seasonal Real or Curve-Fit? A Reliability Checklist"](https://aligrithm.com/is-your-seasonal-real-or-curve-fit-a-reliability-checklist/) is where the seasonal stops being a pretty curve and has to prove it is real. ## Visualizing the three computations ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_204-visual-0.png) ## KEY POINTS - The method you use to compute a seasonal decides what it means. Raw change, detrended, and standardized answer three different questions; computing one while believing you measured another is the core mistake. - Raw price change averages the dollar change at each calendar slot across years. Fine for a single market over a short, price-stable window. Contaminated once the sample spans a big change in price level, because high-price years dominate the average. - Detrended subtracts a long-term average from each price before averaging by slot, so the seasonal does not ride the bull or bear trend underneath. Removes level contamination, not scale contamination. - Standardized rescales each year to a 0-to-1 band using that year's high and low, putting every year and market on a common ruler. Buys cross-comparability at the cost of look-ahead: using the full-year high/low peeks at the future if you compute it live. - Match method to job: raw for a quick single-market look, detrended for long samples with a trend, standardized for cross-year or cross-market comparison. - The formulas do not fix alignment. A floating-date seasonal must be aligned to its event before averaging, or the cleanest detrending still produces noise. - A computed seasonal is an average that hides its own reliability. Ten years agreeing and two outlier years dragging the rest produce the same curve; the curve alone cannot tell you which. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Trading Systems and Methods - Perry Kaufman (Amazon)](https://www.amazon.com/s?k=Trading+Systems+and+Methods+Perry+Kaufman&ref=aligrithm.com) - [Cybernetic Trading Strategies - Murray Ruggiero (Amazon)](https://www.amazon.com/s?k=Cybernetic+Trading+Strategies+Murray+Ruggiero&ref=aligrithm.com) - [The Art of Currency Trading - Brent Donnelly (Amazon)](https://www.amazon.com/s?k=The+Art+of+Currency+Trading+Brent+Donnelly&ref=aligrithm.com) ### 4.38 Is Your Seasonal Real or Curve-Fit? A Reliability Checklist URL: https://aligrithm.com/is-your-seasonal-real-or-curve-fit-a-reliability-checklist/ Last updated: 2026-07-27T13:24:33.000Z The standard objection to seasonal trading is the correct one: most of it is curve-fitting dressed up as a pattern. Slice any market by enough calendar windows and you will find a stretch that was up 80% of the time over your sample. It means nothing. With 250 trading days and the freedom to pick start and end dates, you can manufacture a gorgeous seasonal out of pure noise, the same data-snooping trap the article ["How to Test Indicator Thresholds Without Fooling Yourself"](https://aligrithm.com/how-to-test-indicator-thresholds-without-fooling-yourself/) laid out for indicator thresholds. The calendar just gives you more knobs to overfit with. So the burden is on the seasonal to prove it is real. A pretty backtest is the entry fee, not the proof. Most computed seasonals will not survive the future, and the job of this checklist is to throw out the ones that were never going to. Four tests, and a seasonal has to clear all four. Fail one and you do not have a seasonal, you have a coincidence with good marketing. ## Test one: beat the same-period bias A seasonal return is only interesting if it beats what the market was doing anyway during that window. If stocks drift up over the whole sample, then "long stocks in November" looks profitable, but so does long stocks in any month. The seasonal has to outperform the baseline drift over the same calendar period, not just be positive. $$ \\text{Edge}\_{\\text{seasonal}} \\;=\\; R\_{\\text{seasonal}} \\;-\\; R\_{\\text{baseline,\\,same period}} $$ The seasonal return minus the average return of the market over the same span. If you trade long during a 10-day window, compare it to the average 10-day return of just holding the market, not to zero. A seasonal that makes 2% while the market averaged 1.8% over that window is barely a seasonal; you captured the drift, not a pattern. The bias-relative return is what has to be positive and meaningful, and it is the number most seasonal pitches quietly skip, because the raw return looks better than the edge. This is the seasonal version of a benchmark. The honest question is never "did this make money," it is "did this make money beyond the dumbest thing I could have done over the same dates." ## Test two: no single trade carries the profit A seasonal that worked 19 years out of 20 is a pattern. A seasonal that lost or broke even 19 years and got rescued by one monster year is an accident wearing a costume. Pull up the year-by-year contribution and check whether the total profit survives removing the single best trade. $$ \\text{Concentration} \\;=\\; \\frac{\\max\_{y}\\, \\pi\_{y}}{\\sum\_{y} \\pi\_{y}}\\,,\\qquad \\text{drop the top year:}\\;\\; \\sum\_{y \\neq y^{\*}} \\pi\_{y} \\;\\overset{?}{>}\\; 0 $$ The first piece is the share of total profit from the single best year; the second checks whether the seasonal is still profitable after you delete that year. If one year out of twenty supplies half the profit, the concentration is 0.5 and the pattern is fragile. If removing the best year flips the seasonal to a loss, you do not have a seasonal, you have one good trade and nineteen years of filler. This is the same fragility test you would run on any system: a strategy whose edge lives in one or two observations is indistinguishable from luck, the concern the article ["Permutation Tests for Indicator Significance"](https://aligrithm.com/permutation-tests-for-indicator-significance/) formalized for indicators. ## Test three: a significant share of years follows the bias Direction matters more than dollars here. If the seasonal says "up in this window," you want a statistically significant majority of years to actually go up in that window, not just a high average dragged around by magnitude. Count the hit rate and ask whether it could have come from a fair coin. $$ \\text{under } H\_0\\,(p=0.5):\\quad P(\\text{at least }k\\text{ up in }Y\\text{ years}) \\;=\\; \\sum\_{i=k}^{Y} \\binom{Y}{i} (0.5)^{Y} $$ $$ \\text{example: } Y=20,\\; k=16 \\;\\Rightarrow\\; P \\approx 0.006 $$ The null hypothesis is a coin: with no real bias, each year is up or down with probability one half. The formula gives the chance of seeing at least k up-years out of Y purely by luck. With 20 years, getting 16 or more up in the window has about a 0.6% chance under the coin, so 16/20 in the right direction is hard to dismiss as noise. Getting 12/20 has a probability around 25%, which a coin produces all the time, so 12/20 proves nothing no matter how good the average return looks. Pair this with the magnitude test from before: you want both a directional hit rate that beats the coin and profit that does not hinge on one year. Either alone is gameable. This is the falsifiability discipline from ["How to Make Technical Analysis Falsifiable"](https://aligrithm.com/how-to-make-technical-analysis-falsifiable/) applied to the calendar: state the coin-flip null out loud and make the seasonal reject it. One warning that the test cannot enforce for you. The more calendar windows you searched to find this seasonal, the more your p-value lies. A 0.6% result is only a 0.6% result if you tested one window. Test 100 windows and you expect several to clear that bar by chance alone. Count the windows you tried, not just the one you kept. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Test four: it has to make sense The last test is not statistical, and it is the one that matters most for the future. The seasonal must have a fundamental cause you can name before you trust it forward. Corn falls after the danger of crop damage from drought has passed, because the weather premium that justified the higher price is no longer needed. That is a reason. "Soybeans are up the second week of March" with no mechanism is not a reason, it is a number. The cause is what makes the pattern likely to persist. A seasonal with a real driver behind it, a harvest cycle, a tax deadline, a scheduled release, will tend to keep working until that driver changes, and when it stops you will often be able to see why. A seasonal with no cause has nothing holding it in place, so it can evaporate the year after you find it and you will never know whether it was ever real. This connects to the engine framing in ["The Three Engines of Seasonality: Fixed-Date, Floating-Date, and Behavioral"](https://aligrithm.com/the-three-engines-of-seasonality-fixed-date-floating-date-and-behavioral/): the engine is the cause, and a seasonal with no identifiable engine fails test four by definition. The order matters. Ideally you form the fundamental hypothesis first, then test it, the way you would design any honest experiment. Finding the pattern by data-mining and then inventing a story to fit it is the cardinal sin, because human beings can rationalize any number after the fact. A plausible story attached to a mined pattern is more dangerous than no story, because it manufactures false confidence. ## Running the checklist All four, in order, before a seasonal earns a place in a system. $$ \\text{Trust} \\;=\\; \\big(\\text{edge} > 0\\big)\\;\\wedge\\; \\big(\\text{survives dropping best year}\\big)\\;\\wedge\\; \\big(\\text{hit rate beats coin}\\big)\\;\\wedge\\; \\big(\\text{nameable cause}\\big) $$ It is a logical AND, not a weighted score. A seasonal does not get to be strong on three and weak on one. A pattern with a great hit rate, great edge, and no cause is a curve-fit you have not caught yet. A pattern with a perfect story but a coin-flip hit rate is a story, not a trade. The whole point of the checklist is that each test catches a different way to fool yourself, and skipping any one leaves that door open. What survives the checklist is rarely a standalone system, and that is fine. A seasonal that clears all four is usually a modest bias, not a money machine, and the right use for a modest, well-founded bias is as a filter or a tilt on top of other signals. The article ["Seasonality as a Filter, Not a Standalone System"](https://aligrithm.com/seasonality-as-a-filter-not-a-standalone-system/) takes the seasonals that pass this gate and puts them to work without asking them to carry the whole book. ## Visualizing the checklist ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_205-visual-0.png) ## KEY POINTS - The burden is on the seasonal to prove it is real. With 250 days and free choice of start and end dates, you can mine a beautiful seasonal out of pure noise; a good backtest is the entry fee, not the proof. - Test one, beat the same-period bias: the seasonal return must exceed the market's baseline drift over the same calendar window, not just be positive. Capturing the drift is not a pattern. - Test two, no single trade carries the profit: check the concentration and whether the seasonal stays profitable after deleting the single best year. One monster year rescuing nineteen flat ones is luck in costume. - Test three, a significant share of years follows the bias: count the directional hit rate and test it against a fair-coin null. 16 of 20 up has about a 0.6% chance by luck; 12 of 20 has about 25% and proves nothing. - The hit-rate p-value lies if you searched many windows. Count every window you tested, not just the one you kept. - Test four, it has to make sense: the seasonal needs a nameable fundamental cause (harvest, tax deadline, scheduled release). The cause is what makes it likely to persist. Mining first and inventing a story after is the cardinal sin. - Run all four as a logical AND. Each test catches a different self-deception; skipping one leaves that door open. What survives is usually a modest bias best used as a filter, not a standalone system. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Trading Systems and Methods - Perry Kaufman (Amazon)](https://www.amazon.com/s?k=Trading+Systems+and+Methods+Perry+Kaufman&ref=aligrithm.com) - [Cybernetic Trading Strategies - Murray Ruggiero (Amazon)](https://www.amazon.com/s?k=Cybernetic+Trading+Strategies+Murray+Ruggiero&ref=aligrithm.com) - [The Art of Currency Trading - Brent Donnelly (Amazon)](https://www.amazon.com/s?k=The+Art+of+Currency+Trading+Brent+Donnelly&ref=aligrithm.com) ### 4.36 The Three Engines of Seasonality: Fixed-Date, Floating-Date, and Behavioral URL: https://aligrithm.com/the-three-engines-of-seasonality-fixed-date-floating-date-and-behavioral/ Last updated: 2026-07-27T13:24:32.000Z A corn farmer in Iowa does not care about your moving average. He cares about whether it rains in the second week of July, because that is when the crop pollinates, and a dry spell then decides the size of the harvest. Every year the futures market prices that anxiety into the same six-week window, and every year, once the weather risk passes, the premium bleeds out. That bleed is a seasonal. It shows up at roughly the same calendar spot, it runs in the same direction, and it traces back to a physical cause you can name. Strip the cause away and you have a curve-fit. Keep the cause and you have something worth testing. Seasonality is the study of a market's behavior as a function of the calendar instead of as a function of price. You are no longer asking "what has price done lately." You are asking "what does this market tend to do during this stretch of the year, across many years, and is there a reason." That second clause is the whole game. The article ["Why the Market Does Not Repeat, But Still Rhymes"](https://aligrithm.com/why-the-market-does-not-repeat-but-still-rhymes/) set up the general principle that history rhymes through recurring structure rather than identical paths; seasonality is the cleanest rhyme there is, because the calendar itself is the recurring structure. There are three distinct machines that produce a seasonal pattern, and they are not interchangeable. Confusing them is how traders end up trading noise. A fixed-date seasonal, a floating-date seasonal, and a behavioral seasonal each have a different cause, a different reliability profile, and a different way of breaking. Treat them separately or you will average them into mush. ## Engine one: the fixed-date seasonal The first engine fires on events nailed to a specific calendar date, year after year. The pollination of corn in late June and early July. The filing deadline for US federal taxes on April 15th. Heating-oil demand that builds ahead of winter. These are the easiest seasonals to trust, because the event does not wander. The earth tilts the same way every year, and the tax code does not move April 15th to suit your backtest. The corn case is the textbook one. The plant is most vulnerable to heat and drought during pollination, a tight window in the back half of June into early July. Traders bid in a weather premium ahead of that window because a bad pollination can wreck yield. Once the danger passes without disaster, which is most years, the premium unwinds and corn tends to fall into harvest. The pattern is not "corn goes up in summer." The pattern is "corn carries a fear premium into pollination and sheds it after, because the physical risk that justified the premium is gone." Name the cause and the trade has a spine. Tax season is the same shape with a different driver. The April 15th deadline forces a known, dated cash-flow event across the entire economy, and the flows around it leave fingerprints in equity and short-rate markets. The date is legislated, so the seasonal does not drift unless Congress moves the deadline, which it occasionally does, and when it does, the seasonal moves with it. That is the tell of a real fixed-date pattern: change the date of the cause and the pattern follows. ## Engine two: the floating-date seasonal The second engine fires on events that recur but whose date moves. The clearest example is a scheduled economic release. The monthly US unemployment report does not land on the same calendar date each month, but it lands on a knowable schedule, and the market's behavior around it is patterned. The Thursday before the unemployment number is released has historically carried a downward bias, as positioning gets defensive ahead of a high-variance event. Floating-date seasonals are anchored to an event, not a date, so you cannot find them by averaging the 15th of every month. You have to align your data to the event first, then measure the behavior in the days around it. Line up every unemployment release, look at the bar before, the bar of, and the bar after, and the pattern emerges. Average by raw calendar date and you smear it into nothing, because the event sat on the 3rd one month and the 7th the next. This is the engine most traders get wrong, because the analysis is harder. You are not slicing by date. You are slicing by "days relative to the event," which means you need the event calendar, you need to handle months where the schedule shifted, and you need enough releases in your sample to say anything. The reward for the extra work is a cleaner signal, because event-driven flows are more concentrated than diffuse calendar drift. ## Engine three: the behavioral seasonal The third engine has no physical or scheduled cause. It runs on human habit. The most cited case is the Monday bias in stocks: equities have shown an upward tendency on Mondays, attributed to traders who flatten positions before the weekend and re-enter at the start of the week. There is no harvest, no release, no deadline. The pattern exists because a large number of people behave the same way at the same point in the week, and that collective habit moves price. Behavioral seasonals are the most fragile of the three, because habits change and because they get arbitraged once they are well known. The Monday effect is the cautionary tale. It was real enough to publish on, then it shifted after the October 1987 crash, when "Black Monday" rewired how people thought about Monday risk. A pattern that was a mild upward bias became something traders watched and second-guessed, and watched patterns decay. The cause here is psychology, and psychology is the least stable input you can build on. That instability is the point, not a footnote. A behavioral seasonal can be tradable for a stretch and then quietly invert, and you will not get a weather report telling you it changed. This is the difference the article ["The Difference Between Explanation and Prediction in Markets"](https://aligrithm.com/the-difference-between-explanation-and-prediction-in-markets/) drew: you can explain the Monday effect with a tidy story about weekend positioning, and that explanation can be true, and the pattern can still stop predicting. Explanation is not a guarantee of forward edge. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Why the engine matters more than the pattern Sort any seasonal you find by its engine before you trade it, because the engine tells you how it will fail. $$ \\text{Reliability ranking (typical):}\\quad \\text{fixed-date} \\;>\\; \\text{floating-date} \\;>\\; \\text{behavioral} $$ $$ \\text{because cause stability:}\\quad \\underbrace{\\text{physics, law}}\_{\\text{rigid}} \\;>\\; \\underbrace{\\text{schedule}}\_{\\text{semi-rigid}} \\;>\\; \\underbrace{\\text{habit}}\_{\\text{soft, arbitraged}} $$ Read that as a rough ordering, not a hard rule. A fixed-date seasonal rests on something rigid, like the tilt of the earth or a line in the tax code, so it tends to persist and to break only when the physical or legal cause changes. A floating-date seasonal rests on a schedule, which is stable until the schedule or the market's reaction to the event changes. A behavioral seasonal rests on a habit, which is the softest input and the first to get competed away once a crowd notices it. None of this tells you the pattern is real; the next article in this arc, ["Is Your Seasonal Real or Curve-Fit? A Reliability Checklist"](https://aligrithm.com/is-your-seasonal-real-or-curve-fit-a-reliability-checklist/), handles the testing. The engine tells you what kind of failure to watch for and how much trust the pattern has earned before you even run the test. Two practical consequences. First, when a seasonal stops working, ask whether its engine changed. Corn pollination seasonality weakening would point you at drought-resistant seed or shifted planting, a change in the physical cause. A behavioral seasonal weakening usually means the crowd found it. Second, the engine decides your analysis method. Fixed-date patterns you find by calendar slicing. Floating-date patterns you find by aligning to the event. Behavioral patterns you find by day-of-week or day-of-month slicing, and you treat every result with suspicion, because that bin is where pure data-mining noise hides best. This article defines the object and its three sources. The rest of the seasonal arc builds on it: how to compute a seasonal so the number means something, how to test whether it is real, which day-of-week effects survive scrutiny, and how to use a seasonal as a filter rather than a standalone system. The engine you are dealing with sets the priors for all of it. ## Visualizing the three engines ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_203-visual-0.png) ## KEY POINTS - Seasonality studies a market as a function of the calendar instead of price: what this market tends to do during this stretch of the year, across many years, and whether there is a nameable cause. - A real seasonal needs a cause. Strip the cause away and you are trading a curve-fit; keep it and the pattern has a spine that tells you when it should hold and when it should break. - Engine one, fixed-date: events nailed to a date (corn pollination late June to early July, the April 15 tax deadline). Most reliable, because the cause is physics or law and does not wander. The tell: move the date of the cause and the pattern moves with it. - Engine two, floating-date: recurring events whose date moves (the unemployment release; a downward bias the Thursday before). Found by aligning data to the event, not by slicing raw calendar dates. - Engine three, behavioral: pure habit with no physical or scheduled cause (the Monday up-bias from weekend positioning). The most fragile, because habits change and get arbitraged once known; the Monday effect shifted after the 1987 crash. - Sort every seasonal by engine before trading it. The engine sets your prior on reliability and tells you the failure mode to watch: physical/legal change, schedule change, or crowd arbitrage. - The engine also dictates the analysis method: calendar slicing for fixed-date, event alignment for floating-date, day-of-week or day-of-month slicing for behavioral, with the most suspicion reserved for behavioral. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Trading Systems and Methods - Perry Kaufman (Amazon)](https://www.amazon.com/s?k=Trading+Systems+and+Methods+Perry+Kaufman&ref=aligrithm.com) - [Cybernetic Trading Strategies - Murray Ruggiero (Amazon)](https://www.amazon.com/s?k=Cybernetic+Trading+Strategies+Murray+Ruggiero&ref=aligrithm.com) - [The Art of Currency Trading - Brent Donnelly (Amazon)](https://www.amazon.com/s?k=The+Art+of+Currency+Trading+Brent+Donnelly&ref=aligrithm.com) ### 6.44 Predict Residual Returns, Not Gross URL: https://aligrithm.com/predict-residual-returns-not-gross/ Last updated: 2026-07-27T13:25:52.000Z Most people point their model at the wrong target and never notice. They take a predictor, build a forecast of where the price goes next, and call the gross return the thing to predict. The gross return is the worst possible target, because it secretly bundles a bet you did not mean to make and are bad at: timing the market. Forecast the residual instead, the part of the return that is the instrument being itself, and the job gets easier and more honest in the same move. ## Decompose the return before you forecast it An instrument's gross return is not one thing. It is a sum of pieces, and most of those pieces have nothing to do with the edge you think you have. $$ R = R^{\\text{idio}} + R^{\\text{market}} + R^{\\text{other exposures}} $$ The gross return R splits into the idiosyncratic part, the part driven by market exposure, and the part driven by other exposures like sector, size, or whatever factors your instrument loads on. The idiosyncratic piece is the instrument doing its own thing. The market piece is the instrument getting dragged along by the index through its beta. The other-exposures piece is the same drag from every other common factor. When you build a model of gross R, you are building a model of all three at once whether you wanted to or not. This is the same decomposition behind the factor work in ["What Is a Factor, Really,"](https://aligrithm.com/what-is-a-factor-really/) seen from the forecasting side instead of the attribution side. There the point was that a factor is an effect you keep re-finding and should subtract from returns. Here the point is what you should do with the leftover: forecast it, and only it. ## Predicting gross secretly forecasts the market Here is the trap, stated as arithmetic. If your forecast targets gross R, then your forecast is implicitly a forecast of every component. $$ \\hat{y}\_{\\text{gross}} = \\hat{y}^{\\text{idio}} + \\beta \\, \\hat{y}^{\\text{market}} + \\hat{y}^{\\text{other}} $$ To predict gross return well, you now need to predict the market return too, scaled by beta, because that term sits inside your target. You have signed up for market timing without deciding to. Most desks have no consistent edge timing the index, and the ones that think they do usually have a few good years and a blowup, because market direction is dominated by macro regime shifts that are hard to call and brutal when you are wrong. Active managers fold this in by accident constantly: a stock-selection model that looks like it picks winners is often half a long-the-market bet riding a bull run, and it falls apart the moment the index stops cooperating. Forecasting residual returns is the consistent job. Forecasting the market is the inconsistent one. Bundle them and you contaminate the part you are good at with the part you are not, and your live performance inherits the volatility and inconsistency of the market-timing bet you never meant to place. ## Strip the exposures, forecast what is left The repair is to subtract the exposures before forecasting and point the model at the residual. $$ R^{\\text{idio}} = R - \\beta \\, R^{\\text{market}} - R^{\\text{other exposures}}, \\qquad \\hat{y} = f\\!\\left(x\\right) \\approx R^{\\text{idio}} $$ Estimate the betas, remove beta times the market return and the contributions of the other factor exposures, and what remains is the idiosyncratic return. Train the model to predict that. The forecast no longer carries a hidden market call, so its success or failure is about the edge you actually have, and its return stream stops swinging with the index for reasons unrelated to your signal. This is the same instinct that powers using one asset to clean another, the move in ["Using Bonds to Filter Equity Signals"](https://aligrithm.com/using-bonds-to-filter-equity-signals/): you remove a known common driver so the equity signal you keep is the part bonds did not already explain. Residualizing generalizes it. Pick the exposures you do not have an edge on, subtract them, and forecast the remainder. Your model gets a cleaner target, your backtest stops rewarding you for accidental market beta, and the thing you ship predicts what you meant to predict. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## A caution on the residual itself Subtracting exposures is not free, and the residual can mislead in its own way. The betas are estimated, so a noisy or unstable beta leaves market return leaking into your "idiosyncratic" target, and you are back to partial market timing without knowing it. Over-residualizing is the opposite failure: regress out so many factors that you strip the genuine edge along with the noise and forecast a flat, dead series. And the relationship between your residual predictor and residual return need not be monotone, the trap ["When an Alpha Metric Is U-Shaped"](https://aligrithm.com/when-an-alpha-metric-is-u-shaped/) warns about, so check the shape of the residual signal before trusting a linear forecast of it. Residualizing fixes the target. It does not exempt you from validating everything you would validate on a raw signal. ## Visualizing the residual target ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_314-visual-0.png) ## KEY POINTS - Gross return is the wrong forecast target. It splits into idiosyncratic return plus market exposure plus other factor exposures, and a model of gross R models all three at once. - Predicting gross secretly forecasts the market, scaled by beta, because that term sits inside the target. You sign up for market timing, which is hard and inconsistent, without deciding to. - Active managers fold this in by accident: a stock-selection model is often half a long-the-market bet that works in a bull run and falls apart when the index stops cooperating. - The fix is to estimate the exposures, subtract beta times the market return and the other factor contributions, and forecast only the idiosyncratic residual that remains. - This generalizes filtering one asset with another. Pick the exposures you have no edge on, strip them, forecast the remainder, and your backtest stops rewarding accidental market beta. - Residualizing fixes the target but not the validation. Noisy betas leak the market back in, over-residualizing strips the real edge, and the residual signal can be U-shaped, so check it like any other. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Systematic Trading - Robert Carver (Amazon)](https://www.amazon.com/s?k=Systematic+Trading+Robert+Carver&ref=aligrithm.com) - [Trading Systems - Urban Jaekle Emilio Tomasini (Amazon)](https://www.amazon.com/s?k=Trading+Systems+Urban+Jaekle+Emilio+Tomasini&ref=aligrithm.com) ### 5.40 QLike: The Right Loss Function for Vol Forecasting URL: https://aligrithm.com/qlike-the-right-loss-function-for-vol-forecasting/ Last updated: 2026-07-27T13:25:24.000Z Before you ask which volatility model is good, ask a smaller question that almost nobody answers honestly: how do you score a vol forecast at all? You compare predicted vol to realized vol, fine, but the comparison rule is a modeling choice with real money attached, and the default choice is wrong. Mean squared error on variance is the lazy answer, and it is poorly motivated for the thing you are measuring. The right answer, QLike, also happens to match the asymmetric way a market maker actually loses money, which is the part that makes it worth the bother. ## MSE on variance is the wrong ruler Mean squared error is not a neutral default. It is the loss you get from maximum likelihood when you assume the errors are Gaussian, so using it on a quantity quietly asserts that quantity is roughly normally distributed. Variance is not. Variance is bounded below at zero and heavily right-skewed: long stretches of small numbers punctuated by violent spikes. Run MSE on variance and the squared term lets a handful of spike days dominate the entire fit, so your model spends all its effort chasing the tail and ignores the regime you sit in most of the time. The cheap repair is to change what you take the error of. The log of variance is roughly Gaussian, not skewed, so MSE on log-variance is the loss that maximum likelihood actually justifies here. $$ \\text{Loss} = \\frac{1}{T} \\sum\_{t} \\left( \\log \\sigma^2\_t - \\log h\_t \\right)^2 $$ Here h\_t is your predicted variance and sigma-squared\_t is the realized variance. Because you took logs, a forecast that is half the realized vol and one that is double it get penalized symmetrically, which is sane behavior on a quantity that lives on a multiplicative scale. This single change, logging before squaring, fixes most of what is wrong with the naive ruler and costs you nothing. ## QLike: optimal without knowing the distribution You can do better than log-MSE, because log-variance still carries skew and kurtosis. Any trader knows this without a plot: vol does not just spike, it spikes enormously, then sits in unusually quiet stretches, then drifts through the middle, so the distribution is a lumpy mixture, not a clean bell. The loss best motivated under that mess is QLike. $$ \\text{Loss} = \\frac{1}{T} \\sum\_{t} \\left\[ \\frac{\\sigma^2\_t}{h\_t} - \\log\\!\\left(\\frac{\\sigma^2\_t}{h\_t}\\right) - 1 \\right\] $$ The term in brackets depends only on the ratio of realized to predicted variance. When the forecast is perfect that ratio is 1, the bracket is zero, and the loss is at its floor. The strength of QLike is that you can show it is optimal under extremely relaxed assumptions about the underlying distribution. You do not need to know what generates the vol. It can be a mixture of regimes, a latent variable flipping between calm and chaos, anything, and QLike still ranks forecasts correctly. That robustness is the whole point: you are scoring a process you cannot fully specify, so you want a ruler that does not depend on specifying it. ## Why the asymmetry is exactly the maker's cost The practical reason to care sits in the shape of QLike, not its derivation. Plot the loss against the forecast for a fixed realized vol and it is not symmetric. It punishes underestimating vol, h smaller than sigma, harder than it punishes overestimating, h larger than sigma. That asymmetry is not a quirk to tolerate. It is the exact shape of a market maker's payoff. Underestimate vol and you quote too tight. Tight quotes in a market that is moving more than you think get picked off by informed flow, and you wear the adverse selection, the bleeding ["Why Forecast Accuracy Is Not Enough in Market Making"](https://aligrithm.com/why-forecast-accuracy-is-not-enough-in-market-making/) describes when a technically accurate forecast still loses money. Overestimate vol and you quote too wide, and the cost is mild: you get fewer fills, you sit out some trades, you make less. Missing low is a loss, missing high is a missed opportunity, and those are not the same size. QLike bakes that exact preference into the loss, so the model you fit with it is tuned to fear the error that actually hurts you. Be honest about the size of the win, though. Given a decent underlying vol model, QLike versus log-MSE is rarely a game changer in raw forecast numbers. It is one of those quiet optimizations that aligns the objective with the real cost rather than buying you a large jump in accuracy. The reason to use it is correctness and the asymmetry match, not a dramatic backtest bump, and a weak vol model fit with the perfect loss is still a weak vol model. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Where it fits with the vol models The loss function is upstream of the model and separate from it. The seasonal autoregressive build in ["SAR: Seasonal Autoregressive Volatility Forecasting"](https://aligrithm.com/sar-seasonal-autoregressive-volatility-forecasting/) is one model you could score this way; the seasonality in ["Crypto Volatility Seasonality"](https://aligrithm.com/crypto-volatility-seasonality/) is structure any of these losses will reward you for capturing. QLike is the referee, not a player. Pick it first, because the referee decides which model wins your selection, and a bad referee will happily crown a model that quotes too tight and bleeds to adverse selection while posting a flattering MSE. ## Visualizing the loss asymmetry ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_315-visual-0.png) ## KEY POINTS - Pick the loss function before the model. Scoring a vol forecast is a modeling choice with money attached, and the default, MSE on variance, is the wrong ruler. - MSE assumes Gaussian errors via maximum likelihood, but variance is bounded at zero and heavily right-skewed, so a few spike days dominate the fit and the model ignores the common regime. - Log-variance is roughly Gaussian, so MSE on log-variance is the maximum-likelihood-justified repair. It penalizes half and double the true vol symmetrically and costs nothing. - QLike depends only on the ratio of realized to predicted variance and is optimal under extremely relaxed distributional assumptions. You do not need to know the generating process, even a regime mixture. - QLike is asymmetric: it punishes underestimating vol harder than overestimating, which matches the maker's payoff. Quote too tight and adverse selection bleeds you; quote too wide and you just miss fills. - Be honest about the size: with a decent model, QLike versus log-MSE is a quiet correctness win, not a large accuracy jump. The loss is the referee that picks the model, so a bad referee crowns a model that quotes too tight. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [The Art of Currency Trading - Brent Donnelly (Amazon)](https://www.amazon.com/s?k=The+Art+of+Currency+Trading+Brent+Donnelly&ref=aligrithm.com) ### 6.43 What Is a Factor, Really URL: https://aligrithm.com/what-is-a-factor-really/ Last updated: 2026-07-27T13:25:51.000Z I never understood factors from the textbook definition. The understanding came from trading them, watching the same effect show up in alpha after alpha until I stopped treating each rediscovery as a new edge. A factor is not a fundamental force of markets, not a risk premium handed down from theory, not anything special. A factor is an alpha that explains a large chunk of return variance, important enough that you keep accidentally finding it, so you regress it out of returns and stop paying yourself for the same bet twice. Nothing more. ## A factor is just an alpha you got tired of re-finding Say you research crypto momentum. You build a 24-hour momentum signal, it works, you ship it. Three weeks later you build a "new" signal off funding rates and it also works, and a month after that a volume-breakout thing works too. Run the correlations and the three return streams are 0.7 with each other, because all three are momentum wearing different clothes. You did not find three alphas. You found one alpha three times and congratulated yourself twice for free. The fix is to name the dominant effect, call it a factor, and remove it from returns before you test anything new. Cross-sectionally regress returns on your momentum feature, keep the residual, and test future signals against that residual instead of raw returns. The residual is returns minus the part momentum already explains. Now a signal only scores if it predicts something momentum did not, which is the only thing worth paying for. So the word "factor" is doing far less work than people think. A factor is an alpha you believe in enough, and that explains enough variance, that you are willing to subtract it from the world before looking for the next thing. The promotion from "alpha" to "factor" is a decision about variance and recurrence, not a discovery about the deep structure of markets. ## The market is always factor number one The first factor, in any equity book, is the market. You take an asset's beta to the S&P 500 and remove the index return scaled by that beta, and what is left is the idiosyncratic return. $$ r^{\\text{idio}}\_{i,t} = r\_{i,t} - \\beta\_i \\, r^{\\text{mkt}}\_{t} $$ Read it plainly: the asset's own return minus its share of whatever the market did. If a stock has a beta of 1.3 and the S&P returned 1% on the day, you attribute 1.3% of the stock's move to the market and treat the rest as the stock being itself. From there you keep going, peeling off a size factor, a value factor, a momentum factor, each one another column you regress out, each one another piece of "this is just exposure I already understand" stripped away until the residual is the part that is genuinely the stock. This is the same move as the crypto momentum case, scaled up. The market is factor number one because it explains the most variance of anything you will ever measure, so it is the effect most likely to contaminate every other signal you test. You remove it first for the same reason you remove momentum first in crypto: it is the biggest liar in the room about whether you have found something new. ## Factors are relative to your horizon, and HFT has them too People assume factors are a slow, low-frequency, equities-only idea. They are not. A factor is whatever explains a lot of variance at the horizon you trade, so the factors change when the horizon changes. Predicting one minute ahead in an order book, the canonical factor is order book imbalance, the same effect ["Order Book Imbalance: The First Microstructure Feature to Test"](https://aligrithm.com/order-book-imbalance-the-first-microstructure-feature-to-test/) treats as the first feature anyone should check. At a one-hour horizon the dominant factor is closer to short-term reversion, past return times negative one, because that is what explains most of the variance there. Order book imbalance at HFT horizons is worth a careful caveat, because it is not alpha in the risk-adjusted-return sense. You do not get paid a clean Sharpe for being long imbalance. It is a factor in the variance-and-recurrence sense: it explains a large fraction of one-minute price movement, and it leaks into almost every other microstructure signal you build, so you will keep finding it inside your "new" alphas whether you meant to or not. That is exactly why you factor it out of returns. If you do not, you will confuse factor exposure, which is an effect you already found, for fresh edge. Imbalance is the order-book version of the market beta: too big and too pervasive to leave in the residual. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## The test that tells you new alpha from old exposure Here is the part that made factors click for me, and it is a single line of arithmetic. Take your candidate strategy's position vector, the signed size it wanted to hold over time, and dot it with the specific-factor returns, which are the returns left after the factors are removed. $$ \\text{PnL}^{\\text{specific}} = \\sum\_{t} \\text{position}\_{t} \\cdot r^{\\text{specific}}\_{t}, \\qquad r^{\\text{specific}}\_{t} = r\_{t} - \\big(\\text{returns explained by factors}\\big)\_t $$ If your alpha is really just order book imbalance in disguise, this curve flatlines, because the thing your positions were tracking has already been subtracted out of the returns you are paying them against. The positions still move, but they move against returns that no longer contain the effect they were exploiting, so the cumulative line goes nowhere. If your alpha carries genuine idiosyncratic edge, the same curve slopes up, because the positions are aligned with the part of returns the factors do not explain. One dot product separates "I found something new" from "I re-found the factor," and it shows you performance strictly in excess of what you already know. Run this before you get excited about any signal. A backtest against raw returns will happily reward you for reloading the market beta or the imbalance factor, and the equity curve looks real because the returns are real, they are just not new. The specific-factor curve is the honest version of the same backtest, and a flat one is the cheapest bad news you will ever buy. ## Visualizing the factor test ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_313-visual-0.png) ## Where this connects Removing a factor before forecasting is the same instinct as predicting the residual rather than the gross return, which ["Predict Residual Returns, Not Gross"](https://aligrithm.com/predict-residual-returns-not-gross/) takes as its whole thesis: forecast the part that is not already explained, not the contaminated whole. It also sits underneath cross-sectional work, because ranking a market's normalized signal against its peers, the move in ["Cross-Sectional Percentile Rank Within a Universe,"](https://aligrithm.com/cross-sectional-percentile-rank-within-a-universe/) is a quiet way of removing the common factor and keeping the relative residual. And fair value, the spine of ["Why Fair Value Is the Core of Market Making,"](https://aligrithm.com/why-fair-value-is-the-core-of-market-making/) is itself a factor at the tick level: the dominant explainer of the next price that you subtract before you go hunting for anything subtler. Same operation everywhere. Name the big effect, remove it, and test against what survives. The last use is portfolio construction, where factors act as a regularization tool: constraining factor exposures keeps a book from quietly piling into one effect across positions that looked independent. Same idea from the other side. If a factor is the alpha you already own, controlling exposure to it stops you from owning it ten more times by accident. ## KEY POINTS - A factor is not fundamental or special. It is an alpha that explains a large share of return variance, important enough that you keep re-finding it, so you regress it out of returns before testing anything new. - Promotion from "alpha" to "factor" is a decision about variance and recurrence, not a discovery. You subtract the effect from the world so you stop paying yourself twice for the same bet. - The market is always factor number one. Remove beta times the index return to get the idiosyncratic return, then peel off further factors. It is removed first because it explains the most variance and contaminates everything. - Factors are relative to horizon. Order book imbalance is the canonical one-minute factor and short-term reversion dominates at one hour. Imbalance is not risk-adjusted alpha at HFT horizons, but it leaks into other signals, so factor it out. - The test: dot your position vector with the specific-factor returns (returns minus what factors explain). A flat curve means the alpha was known exposure; an upward curve means genuine new edge. It shows performance in excess of what you already know. - Run the specific-factor backtest before trusting any signal. A raw-returns backtest rewards reloading the factor and the curve looks real, just not new. Factors also regularize portfolio construction by capping exposure to effects you already own. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Systematic Trading - Robert Carver (Amazon)](https://www.amazon.com/s?k=Systematic+Trading+Robert+Carver&ref=aligrithm.com) - [Trading Systems - Urban Jaekle Emilio Tomasini (Amazon)](https://www.amazon.com/s?k=Trading+Systems+Urban+Jaekle+Emilio+Tomasini&ref=aligrithm.com) ### 2.75 DSP and Digital Filters for Traders: The Primer Nobody Wrote First URL: https://aligrithm.com/dsp-and-digital-filters-for-traders-the-primer-nobody-wrote-first/ Last updated: 2026-07-27T13:23:35.000Z Open any serious book on cycles and you hit a wall by page three. The author talks about transfer functions, frequency response, phase lag, and Nyquist, and assumes you already know what a signal is. You don't, because nobody taught you, and the book moves on without you. The articles in this series on low-pass filters, high-pass filters, band-pass filters, and dominant cycle estimation carry the same assumption. A trader who knows arithmetic and "a sine wave is a wiggly line" deserves a way in. This is that way in. This article ships no new filter. The filter builds live in their own places: the super-smoother in ["The Trader's Guide to Low-Pass Filters"](https://aligrithm.com/the-traders-guide-to-low-pass-filters/), the detrending recursion in ["High-Pass Filters for Traders"](https://aligrithm.com/high-pass-filters-for-traders/), the cycle isolation in ["Band-Pass Filters: The Most Underused Tool in Technical Analysis"](https://aligrithm.com/band-pass-filters-the-most-underused-tool-in-technical-analysis/). This one gives you the vocabulary and the mental model underneath all of them. Read it once and those articles stop being a wall. Two facts up front, because the rest of the article keeps returning to them and you should resent any source that hides them. First, no causal filter predicts the future; it reshapes the past. Second, smoothing costs lag, and in real-time trading that lag is a tax you pay, never zero, no matter what "zero-lag" marketing claims. Hold those two and you already think more clearly about indicators than most of the people selling them. ## What you will be able to do after this You will be able to say why an indicator lags and whether that lag can be reduced. You will read an amplitude-and-phase response chart without flinching. You will know what low-pass, high-pass, and band-pass mean and which trading job each one does. You will know why a cycle shorter than two bars cannot exist in your data. And you will open any cycles chapter and recognize every term in it. No math beyond arithmetic and the idea that a sine wave goes up and down. ## The one-sentence thesis Every indicator you have used is a filter: a machine that takes the price series in and puts a reshaped series out. A moving average is a filter. RSI is a filter wrapped in a nonlinear squashing step. MACD is two filters subtracted. Momentum is a filter. Once you see indicators as filters instead of as named recipes, you stop collecting them and start engineering them, because you can ask the only question that matters: which part of the price did this machine keep, and which part did it throw away? That reframe is the whole point of this series. A trader who collects indicators owns forty boxes and cannot say what any of them keeps or discards. A trader who thinks in filters owns one idea (reshape the signal) and a handful of dials. The forty boxes turn out to be four machines at different settings. ## Price is a signal A signal is a list of numbers in time order. That is the entire definition. The closing price of the S&P 500, one number per day across years, is a signal. Engineers who design audio gear, radios, and medical scanners work with the same object. They have spent seventy years building tools to reshape those numbers, and almost none of that toolkit was invented for trading, which is why it works on trading; it never saw your backtest and cannot be overfit to it. Each closing price is a sample. You take one sample per bar, so your sample rate is one sample per bar. On a daily chart that is one sample per day; on a 5-minute chart, one sample every five minutes. The sample rate matters more than beginners expect, and the section on Nyquist below shows why: it sets a hard floor on the fastest wiggle your data can even contain. Note that closes are not the only choice. You could sample the midpoint, the typical price (high plus low plus close over three), or the volume-weighted price. Each is a different signal built from the same market. For this primer, picture the close, one number per bar, marching left to right. ## Signal equals trend plus cycle plus noise Take any price chart and you can split it into three parts that add back up to the original. A slow drift that bends across the whole window: the trend. A medium wiggle that rises and falls every few dozen bars: the cycle. A fast jitter that changes every bar and looks like static: the noise. Stack those three and you get the price back. Pull them apart and you can see what you are actually trading. $$ \\text{price}\[t\] \\;=\\; \\text{trend}\[t\] \\;+\\; \\text{cycle}\[t\] \\;+\\; \\text{noise}\[t\] $$ This decomposition is the whole game. A filter is a machine that keeps some of those three parts and kills the rest. A trend follower wants the slow drift and treats the cycle and noise as enemies. A mean-reversion trader on daily equities wants the deviations around the trend and treats the slow drift as the enemy, which is the argument made in detail in ["High-Pass Filters for Traders"](https://aligrithm.com/high-pass-filters-for-traders/). A cycle trader wants the medium wiggle and treats both the drift and the static as enemies. Same price, three different jobs, three different filters. The split is not exact and it is not unique. Where the trend ends and the cycle begins is a choice you make by setting a dial (the cutoff), and reasonable people set it differently. That arbitrariness is real and the article on dominant cycle estimation, ["Dominant Cycle Estimation Without Astrology"](https://aligrithm.com/dominant-cycle-estimation-without-astrology/), spends its length on how to set the cycle dial from the data instead of by taste. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_312-visual-0.png) ## The wave recipe Here is the big idea, and it costs you no math to accept it: any wiggly line, however messy, can be rebuilt by stacking simple sine waves of different speeds and sizes. A slow sine that goes up and down once across the chart, plus a faster one cycling every fifty bars, plus a fast one cycling every three bars, and so on. Add enough of them with the right sizes and you reproduce the original line exactly. This is a mathematical fact, proven two hundred years ago, and you do not need the proof to use it. Think of the line as a chord played on a piano. The chord sounds like one thing, but it is built from separate notes: a low bass note, some middle notes, a high treble note. Your ear hears the blend; a tuner hears the recipe. The slow sines are the bass (the trend). The fast sines are the treble (the noise). The middle notes are the tradable cycles. The price chart is the chord; the recipe of sines is what filters actually work on. That recipe has a name: the frequency domain. Looking at the price line itself, value against time, is the time domain, the normal chart you stare at all day. Looking at the recipe instead, how much of each wave speed is present, is the frequency domain. Same information, two views. The time domain answers "what was the price on Tuesday." The frequency domain answers "how much slow drift versus fast noise is in this stretch." A filter is easiest to understand in the frequency domain, because all a filter does is turn the volume up or down on each wave speed. Turn the bass up and the treble down and you have a trend filter. Turn the bass down and the treble up and you have a detrender. That is the entire trick, and the rest of this article makes it precise. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_312-visual-1.png) ## Anatomy of a sine wave Four words describe a sine wave completely, and they are the entire vocabulary of cycles. Learn these four and the cycle literature opens up. Amplitude: how big the wave is, the height from the middle to the peak. A high-amplitude cycle swings the price a lot; a low-amplitude one barely moves it. Period: how many bars one full cycle takes, from one peak to the next peak. A 20-bar period means the wave repeats every 20 bars. Frequency: how many cycles fit in one bar, which is just one divided by the period. It is the same fact as period, stated upside down. $$ \\text{frequency} \\;=\\; \\frac{1}{\\text{period}} $$ A 20-bar period is a frequency of 1/20 = 0.05 cycles per bar. A 4-bar period is 0.25 cycles per bar. Short period means high frequency (fast wiggle); long period means low frequency (slow drift). Traders think in periods because "20-bar cycle" is intuitive; engineers think in frequency because the math is cleaner. You will see both, and the conversion is always this one division. When a chart's x-axis is labeled in frequency from 0 to 0.5, just remember the left side is slow and the right side is fast. Phase: where in its up-and-down journey the wave sits right now. Two waves with the same period can be out of step, one cresting while the other troughs. Phase measures that offset. Phase matters because lag, the thing you will learn to hate, is a phase shift: a filter that delays the cycle by a quarter of its period has shifted its phase, and that shift is the cost you pay for smoothing. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_312-visual-2.png) ## Sampling, Nyquist, and aliasing You take one sample per bar. That single fact puts a hard floor on the fastest cycle your data can contain, and the floor is two bars. To see a cycle you need at least one sample on the way up and one on the way down; with one sample per bar, the fastest thing you can resolve goes up on one bar and down on the next, which is a two-bar period. Anything faster than that is invisible to you no matter how good your indicator is. This limit has a name, the Nyquist limit, and for trading it reduces to one rule. $$ \\text{shortest period you can see} \\;=\\; 2 \\text{ bars} $$ This is why every indicator period has a floor and why a "3-bar RSI" is already scraping the bottom of what the data physically holds. It is not a tuning preference; it is a property of sampling once per bar. Worse than invisibility is disguise. A cycle faster than the two-bar floor does not vanish cleanly when you sample it. It masquerades as a slower cycle that was never there. This is aliasing, and you have seen it: the wagon wheels in old movies that appear to spin backward, or a helicopter rotor that looks frozen on video. The camera samples too slowly, so a fast rotation shows up as a slow or backward one. Your daily close does the same to intraday cycles faster than two days: they fold down and appear as fake multi-day cycles, and you cannot tell them from real ones after the fact. The practical damage is that a cycle detector can report a clean 8-day cycle that is actually a folded-down intraday cycle, an artifact of sampling, not a feature of the market. The defense is to low-pass filter before you downsample (kill the too-fast content before it can fold), and to distrust any measured cycle near the two-bar floor. The article on why cycles are slippery, ["Why Market Cycles Are Evanescent"](https://aligrithm.com/why-market-cycles-are-evanescent/), carries this skepticism further: even a real measured cycle may have already died by the time you act on it. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_312-visual-3.png) ## The lag tax This is the most important section in the article. Read it twice. To smooth a series you average over the past. A 10-bar simple moving average at today's bar is the mean of the last ten closes. The output is smoother than the input because the jitter partly cancels when you average. The catch is that the output describes not today but the center of the window you averaged over. Average the last ten bars and the "center of mass" of that window sits 4.5 bars back, so your smooth line reports, on average, where price was four and a half bars ago. That delay is lag, and it is not a bug you can patch. It is the price of averaging. $$ \\text{lag of an SMA of length } N \\;=\\; \\frac{N-1}{2} \\text{ bars} $$ For N = 10 that is 4.5 bars. For N = 50 it is 24.5 bars. The arithmetic is brutal and unavoidable: the more you smooth, the more bars you average, the further back the center of mass sits, the more your indicator lags. Smoothness and responsiveness pull in opposite directions, and on a causal filter (one that only sees the past, which is the only kind you can trade) you cannot have both. There is no free lunch. This is the lie detector for the entire indicator industry. Every "zero-lag" indicator you will ever see is doing lag reduction, not lag elimination. The clever ones extrapolate the recent slope forward to partly cancel the delay, which works when the slope holds and fails exactly when it does not, at turning points, which is when you most wanted the indicator to be right. The article ["Why Moving Averages Can Lie at Turning Points"](https://aligrithm.com/why-moving-averages-can-lie-at-turning-points/) is the full autopsy. For now, hold the rule: in real-time causal filtering, lag is a tax. You can choose how much to pay by trading it against smoothness, but you cannot get the bill to zero. Note the one honest escape, and why you cannot use it live. A centered filter that averages five bars back and five bars forward has no lag, because its center of mass is today. It also requires knowing five bars of the future, so you can only run it on history, in a chart study, never on the live right edge. Every backtest that quietly uses a centered smoother is cheating, and the cheat is invisible unless you go looking. The non-causal ideal filter (the sinc) is the extreme version of this, and the rigorous DSP cluster covers why you cannot run it in real time. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_312-visual-4.png) ## The four jobs filters do There are four filter types and each maps to one trading job. Learn the map and you can pick a filter by what you want to keep, instead of by its brand name. Low-pass: keeps the slow waves, kills the fast ones. The job is trend extraction. Every moving average is a low-pass filter. You use one when the slow drift is the thing you want to trade and the wiggle is in your way. The construction recipe for the good ones lives in ["The Trader's Guide to Low-Pass Filters"](https://aligrithm.com/the-traders-guide-to-low-pass-filters/). High-pass: keeps the fast waves, kills the slow ones. The job is detrending, which gives you momentum and oscillators. You subtract the trend and keep the deviations around it. On daily equities those deviations are where the mean-reversion edge lives, which is the argument in ["High-Pass Filters for Traders"](https://aligrithm.com/high-pass-filters-for-traders/). The simplest high-pass filter is today's close minus yesterday's close, and the article ["No Filter Is Predictive: What Traders Misunderstand About Smoothing"](https://aligrithm.com/no-filter-is-predictive-what-traders-misunderstand-about-smoothing/) shows why that one-bar difference is the root of every momentum feature. Band-pass: keeps a chosen middle band of speeds, kills both the slow and the fast. The job is isolating one cycle. You set a center period and a width, and the filter passes wiggles near that period while rejecting both the trend and the noise. This is the core tool for cycle work, built in ["Band-Pass Filters: The Most Underused Tool in Technical Analysis"](https://aligrithm.com/band-pass-filters-the-most-underused-tool-in-technical-analysis/). Band-stop, the decycler version: removes a chosen band and keeps everything else. The job is stripping cycle energy out so the clean trend shows through. Take the price, remove the cycle band, and what remains is a smooth trend line that did not lag the way a moving average would. The construction is in ["Decyclers: Extracting Trend by Removing Cycle Energy"](https://aligrithm.com/decyclers-extracting-trend-by-removing-cycle-energy/). Four jobs, one price. Feed the same chart into all four and you get four different outputs: a smooth trend, a jittery detrended oscillator, a clean isolated cycle, and a decycled trend. None of them predicts anything. Each one keeps a different slice of what was already there. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_312-visual-5.png) ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Frequency response, the filter's fingerprint Cycle authors put up the same chart constantly and never slow down to teach you how to read it. The chart is the frequency response, and it is the single most useful picture in filter design because it tells you exactly what a filter keeps and what it kills, before you ever run it on data. The x-axis is wave speed, labeled either as frequency (0 on the left for slow, up to 0.5 on the right for the fastest two-bar wiggle) or as period (long periods on the left, short on the right). Same axis, two labelings. The y-axis is gain: how much of each speed survives the filter. A gain of 1 means that speed passes untouched. A gain of 0 means that speed is killed dead. A gain of 0.5 means it comes out half as big as it went in. This curve is the amplitude response, and it is the filter's fingerprint. Read a low-pass filter's fingerprint and the story is plain: gain near 1 on the left (slow waves pass), falling toward 0 on the right (fast waves die). Read a high-pass and it is the mirror: near 0 on the left, near 1 on the right. A band-pass shows a hump in the middle and near 0 on both ends. You can identify any filter's job from the shape of this one curve without knowing a single equation behind it. Two pieces of standard vocabulary live on this chart. The cutoff, also called the half-power point or the 0.707 point, is the speed where the gain has dropped to about 0.707 of its peak (that number is one over the square root of two, and it marks where the filter has cut the wave's power in half). To the slow side of the cutoff the low-pass mostly passes; to the fast side it mostly blocks; the cutoff is the dividing line you set with the filter's period dial. The rolloff is how steeply the curve falls past the cutoff. A gentle rolloff lets some unwanted content leak through; a steep one cuts harder but, as the lag section warned, costs more lag. You are always trading sharpness against lag, and the frequency response lets you see the trade before you pay for it. There is a second curve people show alongside the gain: the phase response, how much each speed gets delayed. This is the lag tax drawn as a curve. A filter can pass a cycle at full size (gain 1) while still shifting it late in time (phase delay), which is exactly how a moving average keeps the trend but reports it late. When you read these charts in the cycles literature, gain answers "how much survives" and phase answers "how late does it arrive." The full treatment of reading these on real indicators is ["The Frequency Response of Trading Indicators"](https://aligrithm.com/the-frequency-response-of-trading-indicators/). ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_312-visual-6.png) ## FIR versus IIR, in plain words Two ways exist to build any filter, and the difference shows up in how the filter remembers the past. The first kind blends only past inputs. To make today's output you take a weighted mix of the last several closes and nothing else. The simple moving average is the plain example: add the last ten closes, divide by ten, done. This kind is called FIR, finite impulse response, or non-recursive. "Finite" because its memory has a hard end: an SMA of length ten forgets a bar completely once it falls out of the ten-bar window. The weights are out in the open and easy to reason about, which makes FIR filters predictable and stable. The cost is that sharp filtering needs many taps (many past bars in the blend), and many taps means more lag and more computation. The second kind feeds its own past outputs back in. To make today's output you mix some of today's input with a piece of yesterday's output. The exponential moving average is the example: today's EMA is a fraction of today's close plus the rest of yesterday's EMA. Because yesterday's output already contained a piece of the day before, and so on back forever, this kind has infinite memory from very little storage. It is called IIR, infinite impulse response, or recursive. The payoff is efficiency: an IIR filter gets a sharp response from one or two stored numbers instead of fifty. The risk is that feedback can ring or overshoot, like a pushed swing that keeps swinging, and a badly designed one can go unstable. A teaser for the math section: the past-input taps of a filter are called its zeros (speeds it kills), and the feedback terms are called its poles (the resonance engine that makes IIR filters efficient and occasionally twitchy). You do not need to compute either one to use this section. Hold the picture: FIR blends the past inputs and forgets cleanly; IIR also recycles its own outputs and remembers forever on the cheap. Almost every Ehlers-style smoother you will meet is IIR, which is why "poles" keeps coming up. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_312-visual-7.png) ## From averages to the z-transform This is the only section that gets mathy, and it is built one small step at a time. Nothing here is harder than algebra plus one new symbol. The payoff is that you will recognize the transfer function notation that every rigorous filter article uses, even if you never solve one by hand. Step one, write the filters you already know as equations. The simple moving average of length 3 is just the mean of the last three closes. $$ y\[t\] \\;=\\; \\frac{x\[t\] + x\[t-1\] + x\[t-2\]}{3} $$ The exponential moving average mixes a fraction of today's close with the rest of yesterday's output. Call the fraction alpha, a number between 0 and 1. $$ y\[t\] \\;=\\; \\alpha \\, x\[t\] \\;+\\; (1 - \\alpha)\\, y\[t-1\] $$ A small alpha (say 0.1) leans hard on the old output and smooths a lot with heavy lag; an alpha near 1 barely smooths at all. That one equation, with one stored number, is the whole EMA. Notice the EMA feeds its own past output back in (the y\[t-1\] term), which is what made it IIR in the last section. Step two, the one new symbol. Writing x\[t-1\] for "yesterday's value" gets clumsy fast, so engineers use a shorthand: the delay operator, written z to the minus one. It means nothing more than "shift back one bar." Multiply a signal by z-to-the-minus-one and you get yesterday's version of it. Multiply by z-to-the-minus-two and you get the value from two bars ago. That is the entire meaning of z at this stage. Do not let the letter scare you; it is a "yesterday" button. $$ z^{-1} \\,x\[t\] \\;=\\; x\[t-1\] \\qquad\\qquad z^{-2}\\,x\[t\] \\;=\\; x\[t-2\] $$ Step three, rewrite the EMA with the delay operator. Replace y\[t-1\] with z-to-the-minus-one times y, gather the y terms on one side, and you get a compact form. $$ y \\;=\\; \\alpha\\, x \\;+\\; (1-\\alpha)\\, z^{-1} y \\qquad\\Longrightarrow\\qquad y \\,\\big(1 - (1-\\alpha) z^{-1}\\big) \\;=\\; \\alpha\\, x $$ Step four, define the transfer function. Divide output by input and you get a single expression that is the filter's complete DNA. It is called the transfer function, written H of z, and it captures everything the filter does to every wave speed at once. $$ H(z) \\;=\\; \\frac{y}{x} \\;=\\; \\frac{\\alpha}{1 - (1-\\alpha)\\, z^{-1}} $$ That fraction is the EMA, fully described. The top (numerator) is built from past-input terms. The bottom (denominator) is built from feedback terms. Every linear filter in this entire series can be written this way: a ratio of two short polynomials in the "yesterday" button. When a cycles author writes H(z) and starts factoring it, this is all it is. Step five, poles and zeros, intuition only. The values of z that make the numerator zero are the zeros: the wave speeds the filter kills dead. The values that make the denominator zero are the poles: the feedback resonances, the engine that lets an IIR filter get a sharp response from almost no storage. A pole sitting close to a cycle speed makes the filter ring at that speed (helpful for a band-pass, dangerous everywhere else). You will leave this section able to look at H(z), point at the numerator and say "those are the zeros," point at the denominator and say "those are the poles," and know which one is the resonance engine. That recognition, not the algebra of solving them, is what unlocks the rigorous articles. ## Three worked filters, end to end Theory earns nothing until you can build it. Here are the three simplest real filters, each as an equation, a one-line description of what it does, and runnable code. The code uses plain Python so you can read it as pseudocode if you do not run it. The exponential moving average, the 1-pole low-pass. One stored number, one alpha, smooths the series and keeps the trend. This is the filter from the math section above. ```python def ema(x, alpha): y = [x[0]] for t in range(1, len(x)): y.append(alpha * x[t] + (1 - alpha) * y[t - 1]) return y ``` Pass a small alpha for heavy smoothing and heavy lag, a larger alpha for a twitchier, faster line. One pole, gentle rolloff, the cheapest trend filter that exists. Frequency response: gain near 1 for slow waves, sliding down toward 0 for fast ones, a textbook low-pass that fades rather than cuts. The construction details and why traders should prefer it to the SMA in some cases live in ["EMA vs SMA: Why Simplicity Still Matters"](https://aligrithm.com/ema-vs-sma-why-simplicity-still-matters/). Simple momentum, today minus N bars ago, which is secretly a high-pass filter. People think of momentum as a strength gauge; structurally it subtracts the slow level and keeps the change, which is exactly what a detrender does. ```python def momentum(x, n): return [None] * n + [x[t] - x[t - n] for t in range(n, len(x))] ``` The output is large when price has moved a lot over the last N bars and near zero when it has not. Because it subtracts a delayed copy of the price, the slow drift cancels and the fast change survives: a high-pass filter wearing a momentum label. Frequency response: gain near 0 for slow waves (the trend gets killed) rising for faster ones, the mirror image of the EMA. This is the seed that grows into the full detrending toolkit in ["High-Pass Filters for Traders"](https://aligrithm.com/high-pass-filters-for-traders/). A 2-pole smoother, the super-smoother flavor, to show why more poles cut sharper but cost more lag. Two stored numbers, a steeper rolloff than the EMA, much cleaner rejection of fast noise. ```python import math def two_pole_smoother(x, period): a = math.exp(-1.414 * math.pi / period) b = 2 * a * math.cos(1.414 * math.pi / period) c2, c3 = b, -a * a c1 = 1 - c2 - c3 y = list(x[:2]) for t in range(2, len(x)): y.append(c1 * (x[t] + x[t - 1]) / 2 + c2 * y[t - 1] + c3 * y[t - 2]) return y ``` The two feedback terms (the y\[t-1\] and y\[t-2\]) are the two poles. Frequency response: the same low-pass shape as the EMA but with a steeper rolloff past the cutoff, so it holds the trend at full gain longer and then drops fast noise harder. Compared to the 1-pole EMA, this filter passes the trend almost untouched while killing fast noise far harder, at the cost of a bit more lag, exactly the sharpness-versus-lag trade the frequency-response section promised. This is the workhorse low-pass behind the more rigorous smoothers, built out fully in ["The Trader's Guide to Low-Pass Filters"](https://aligrithm.com/the-traders-guide-to-low-pass-filters/), and the same two-pole structure reappears in the rigorous DSP cluster on Butterworth and higher-order designs. Run all three on the same price series and watch the lag tax appear on its own: the heavier the smoothing, the later the line turns. No new theory needed to see it. Reuse the frequency-response idea from earlier and you can predict the shape of each output before you plot it. ## Bridge to the rest of the series You now hold the whole vocabulary. Every term in the cycles literature maps to something you learned above. Here is the payoff map, so the downstream syllabus snaps into place. | Concept you just learned | Where it gets used | | -------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Lag, center of mass | ["The Hidden Cost of Every Moving Average: Lag"](https://aligrithm.com/the-hidden-cost-of-every-moving-average-lag/), and ["How to Think About Indicator Lag Before Backtesting"](https://aligrithm.com/how-to-think-about-indicator-lag-before-backtesting/) | | Filters do not predict | ["No Filter Is Predictive: What Traders Misunderstand About Smoothing"](https://aligrithm.com/no-filter-is-predictive-what-traders-misunderstand-about-smoothing/) | | SMA weaknesses | ["Why the SMA Is Often a Terrible Smoother"](https://aligrithm.com/why-the-sma-is-often-a-terrible-smoother/) | | EMA construction | ["EMA vs SMA: Why Simplicity Still Matters"](https://aligrithm.com/ema-vs-sma-why-simplicity-still-matters/) | | Low-pass = trend | ["The Trader's Guide to Low-Pass Filters"](https://aligrithm.com/the-traders-guide-to-low-pass-filters/) | | High-pass = detrend | ["High-Pass Filters for Traders"](https://aligrithm.com/high-pass-filters-for-traders/) | | Band-pass = isolate a cycle | ["Band-Pass Filters: The Most Underused Tool in Technical Analysis"](https://aligrithm.com/band-pass-filters-the-most-underused-tool-in-technical-analysis/) | | Band-stop / decycler | ["Decyclers: Extracting Trend by Removing Cycle Energy"](https://aligrithm.com/decyclers-extracting-trend-by-removing-cycle-energy/) | | Lag at turning points | ["Why Moving Averages Can Lie at Turning Points"](https://aligrithm.com/why-moving-averages-can-lie-at-turning-points/) | | Reading the frequency response | ["The Frequency Response of Trading Indicators"](https://aligrithm.com/the-frequency-response-of-trading-indicators/) | | Spikes and outliers (nonlinear filters) | ["Why Median Filters Are Useful for Volume and Outliers"](https://aligrithm.com/why-median-filters-are-useful-for-volume-and-outliers/) | | Amplitude normalization | ["Automatic Gain Control for Trading Indicators"](https://aligrithm.com/automatic-gain-control-for-trading-indicators/) | | Dominant cycle, period measurement | ["Dominant Cycle Estimation Without Astrology"](https://aligrithm.com/dominant-cycle-estimation-without-astrology/) | | Cycles decay | ["Why Market Cycles Are Evanescent"](https://aligrithm.com/why-market-cycles-are-evanescent/) | | Nyquist, transfer function, z, poles and zeros, roofing filter | the rigorous DSP cluster in the spectral-analysis pillar | Read down that column and you have the reading order. Every one of those articles assumes the vocabulary you now own. ## Honest caveats Five closing truths, because a primer that left you optimistic would have failed you. No filter predicts. A filter reshapes the history it has seen; it has no term for the future, as ["No Filter Is Predictive: What Traders Misunderstand About Smoothing"](https://aligrithm.com/no-filter-is-predictive-what-traders-misunderstand-about-smoothing/) proves with the algebra. When a smoothed line turns up, it is reporting that the past already turned up, late. Trade the output as a feature, never as a forecast. Lag is a tax, never zero. On any causal filter you can trade smoothness against responsiveness, but you cannot get the lag to zero. Every "zero-lag" claim is lag reduction that works until the slope it extrapolates breaks, which tends to be the turning point you cared about. There is no perfect brick-wall filter you can run live. The ideal filter that passes everything below a cutoff and kills everything above, with no transition and no lag, needs future data to build (its shape, the sinc, extends infinitely in both directions). You can run it on history and feel clever; you cannot run it at the live right edge. The rigorous DSP cluster covers why. Cycles are slippery. A cycle you measure today may already be gone, because market cycles fade in and out rather than persisting like a clock, the argument in ["Why Market Cycles Are Evanescent"](https://aligrithm.com/why-market-cycles-are-evanescent/). Measuring a 20-bar cycle does not promise it survives to the next swing. A filter with a beautiful response can still be a terrible trading feature. Clean frequency response and good predictive value are different questions. A filter can pass exactly the band you wanted and still feed your model noise, because that band carried no edge on your instrument. Whether a feature predicts is an empirical question you answer with the indicator-quality tests in the pillar on building features that actually work, not with the shape of a response curve. ## KEY POINTS - Every indicator is a filter: a machine that takes price in and puts a reshaped series out. Stop collecting indicators, start asking which part of the price each one keeps and kills. - A signal is numbers in time order. Closes are samples at one sample per bar. The sample rate sets a hard floor on the fastest cycle you can see. - Any price line splits into trend plus cycle plus noise, and those three add back up. A filter keeps some parts and kills the rest. The split is a choice you set with a dial. - Any wiggly line rebuilds from stacked sine waves. The recipe of waves is the frequency domain; the price line is the time domain. A filter turns the volume up or down on each wave speed. - Four words describe any cycle: amplitude (size), period (bars per cycle), frequency (1 over period), phase (where in the cycle it sits now). Lag is a phase shift. - The shortest cycle your data can hold is 2 bars (Nyquist). Anything faster either vanishes or, worse, masquerades as a slow fake cycle (aliasing, the wagon-wheel effect). - Lag is the tax on smoothing. An SMA of length N lags by (N-1)/2 bars because its center of mass sits that far back. On a causal filter the lag is never zero; every "zero-lag" claim is lag reduction. - Four filter jobs: low-pass keeps the trend, high-pass keeps the deviations (momentum, mean reversion), band-pass isolates one cycle, band-stop (decycler) strips the cycle to leave a clean trend. - The frequency response is the filter's fingerprint: x-axis is wave speed, y-axis is gain (how much survives). The 0.707 point is the cutoff; the rolloff slope trades sharpness against lag. - FIR (non-recursive, like the SMA) blends past inputs and forgets cleanly. IIR (recursive, like the EMA) recycles its own outputs for infinite memory from almost no storage, cheaper but can ring. - The transfer function H(z) is a filter's complete DNA: a ratio of two short polynomials in the delay operator z (the "yesterday" button). Numerator terms are zeros (speeds killed); denominator terms are poles (the resonance engine). - No filter predicts; it reshapes history. Cycles fade. A clean response does not guarantee a profitable feature. Prove predictive value empirically, not from the shape of a curve. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Statistically Sound Indicators for Financial Market Prediction - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Statistically+Sound+Indicators+for+Financial+Market+Prediction+Timothy+Masters&ref=aligrithm.com) - [Cycle Analytics for Traders - John Ehlers (Amazon)](https://www.amazon.com/s?k=Cycle+Analytics+for+Traders+John+Ehlers&ref=aligrithm.com) ### 4.35 Currency Strength from Pair Decomposition: One Matrix, Every Currency URL: https://aligrithm.com/currency-strength-from-pair-decomposition-one-matrix-every-currency/ Last updated: 2026-07-27T13:25:00.000Z Every FX pair is a quotient. EURUSD is the price of the euro measured in dollars, so a move in EURUSD tells you nothing on its own about whether the euro rose or the dollar fell. Both produce the same green candle. Traders stare at a chart of one pair and narrate it as a story about one currency, when the number in front of them mixes two. Currency strength models try to split that quotient back into its parts, one line per currency, so you can ask the question the pair refuses to answer: who actually moved? This is the cleanest idea in the currency-strength world and the one most often dressed up as a signal it is not. The decomposition itself comes from the [Dekalog Blog post "Currency Strength Revisited"](https://dekalogblog.blogspot.com/2023/08/currency-strength-revisited.html?ref=aligrithm.com), which sets up the algebra and solves it bar by bar with a numerical optimizer. The version here keeps the algebra and throws out the optimizer. We will build it, check that it reconstructs the original pairs, and then say plainly what it does and does not give you. ## The identity hiding in every pair Start with three currencies and the prices that connect them. Call the dollar value of one euro EURUSD, the dollar value of one pound GBPUSD, and the pound value of one euro EURGBP. These three are not independent. If you could buy euros with dollars, sell those euros for pounds, and sell the pounds back to dollars, any mismatch would be free money, and the interbank desks closed that gap a long time ago. So the cross holds: $$ \\text{EURGBP} \\;=\\; \\frac{\\text{EURUSD}}{\\text{GBPUSD}} $$ Read it slowly. The pound-price of a euro equals the dollar-price of a euro divided by the dollar-price of a pound. The dollar cancels. That cancellation is the whole game: each pair is a ratio of two underlying currency "values," and if we take logs, ratios turn into differences. Define a hidden log-value for each currency and call it strength. Then the log return of any pair over one bar is the strength change of the base currency minus the strength change of the quote currency: $$ r\_{\\text{base/quote}} \\;=\\; s\_{\\text{base}} - s\_{\\text{quote}} $$ For EURUSD that reads as the euro's strength change minus the dollar's. For GBPUSD, the pound's minus the dollar's. The strengths are unobserved; the pair returns are observed. We have a system of equations, one per pair, and we want to solve for the per-currency strengths underneath. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_202-visual-0.png) ## Why one currency has to be the anchor Count the unknowns. With 8 currencies you have 8 strengths, but the pairs only ever measure differences between them. Add a constant to every currency's strength and all pair returns stay identical, because the constant cancels in every subtraction. The system is underdetermined by exactly one degree of freedom. The Dekalog Blog write-up that this method comes from fixes it by pinning one currency (the dollar) as the reference, which is the same move as setting a zero on a thermometer. Strength is relative by construction. There is no absolute "the euro is strong," only "the euro is strong against this basket right now." That basket framing is the link to the dollar index. Dean Markwick's post ["Making Sense of the DXY"](https://dm13450.github.io/2024/03/11/Making-Sense-of-the-DXY.html?ref=aligrithm.com) shows the DXY is a fixed-weight geometric average of the dollar against six currencies, which is one hand-picked anchor basket. The decomposition here generalizes that: instead of one currency measured against a frozen 1970s weighting, solve for every currency's strength against the whole board at once, weights and all, on the data you actually have. That single fact kills most of the mysticism. A currency-strength meter is a relative ranking with an arbitrary origin, not a measurement of intrinsic value. ## The slow way and the fast way The original Dekalog Blog implementation solves the system one bar at a time. For each bar it hands the residual to a numerical optimizer (Octave's `fminunc`) and lets it search for the strengths that best fit that bar's pair returns. It works, and it is slow, because you re-run an optimization thousands of times, once per row, to solve what is the same linear problem with the same structure on every bar. The structure never changes. Which currency is the base and which is the quote is fixed by the pair names, not by the data. So write that structure once as a matrix and reuse it. Build a design matrix `A` with one row per pair and one column per currency. Put `+1` in the base column, `-1` in the quote column, zero everywhere else. EURUSD becomes a row with `+1` under EUR and `-1` under USD. That matrix encodes the entire "base minus quote" relationship for the whole board. Now every bar is the same linear equation `r = A s`, and we solve all bars in one shot. Because the system is rank-deficient (the anchor problem above), a raw inverse blows up. Ridge regularization handles it: add a small `lam` on the diagonal, which both fixes the singularity and pins the solution to the minimum-norm strengths, the symmetric choice where the strengths sum to roughly zero instead of drifting off with the free constant. $$ \\hat{s} \\;=\\; \\left(A^{\\top}A + \\lambda I\\right)^{-1} A^{\\top}\\, r $$ The term `AᵀA + λI` is small, square, and constant, so you invert it once. Everything else is matrix multiplication. The optimizer disappears, the result is deterministic, and there is no per-bar search to converge or fail. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## The code Two functions. The first builds the constant matrix from the pair names so you never hand-encode `+1`/`-1`. The second solves every bar at once and cumulates the per-bar strength changes into an index. ```python import numpy as np import pandas as pd def build_design_matrix(pairs: list[str]) -> tuple[np.ndarray, list[str]]: currencies = sorted({c for p in pairs for c in (p[:3], p[3:])}) col = {c: i for i, c in enumerate(currencies)} A = np.zeros((len(pairs), len(currencies))) for row, p in enumerate(pairs): A[row, col[p[:3]]] = 1.0 # base A[row, col[p[3:]]] = -1.0 # quote return A, currencies def build_currency_strength(closes: pd.DataFrame, lam: float = 1.0) -> pd.DataFrame: pairs = list(closes.columns) # ["EURUSD", "GBPUSD", ...] A, currencies = build_design_matrix(pairs) # observed: per-bar log returns of every pair, shape (T, n_pairs) r = np.log(closes).diff().to_numpy() # solve all bars at once: (AᵀA + λI)⁻¹ Aᵀ applied to every row of r solver = np.linalg.inv(A.T @ A + lam * np.eye(len(currencies))) @ A.T s = r @ solver.T # shape (T, n_currencies) out = closes.copy() for c, i in zip(currencies, range(len(currencies))): out[f"{c}_strength"] = np.nancumsum(s[:, i]) # cumulate to an index return out ``` Two things matter here and both are about honesty, not speed. The strengths on bar `t` use only pair returns from bar `t`, so there is no lookahead. And the index is a plain `cumsum` of those one-bar changes, which means a value at any point depends only on the past. Whatever this chart shows, it does not borrow from the future. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/cur_stren.png) ## Does it actually reconstruct the pairs? A decomposition you cannot invert is a decoration. The test is direct: take two solved strengths, subtract them, cumulate, and you should recover the real pair. If the euro and pound strengths are right, their difference has to trace EURGBP, the pair we never fed in as a primary leg. ```python df = build_currency_strength(closes, lam=1.0) synthetic = (df["EUR_strength"] - df["GBP_strength"]) # implied EURGBP path real = np.log(closes["EURGBP"]).diff().fillna(0).cumsum() # actual EURGBP path print(real.corr(synthetic)) # ~0.999 ``` A correlation near 0.999 says the linear algebra works. The hidden strengths, recombined, reproduce a pair the model never solved for directly. That is the strongest claim this method can make, and it is a claim about internal consistency, not about prediction. ## What the 0.999 does not buy you Here is where most currency-strength content stops, takes a victory lap on that correlation, and pivots to selling a meter. The reconstruction being near-perfect tells you the decomposition is arithmetic done correctly. It says nothing about whether ranking currencies by strength makes money. I ran the obvious strategies on the strength indices and they died. Cross-sectional momentum (long the strongest currencies, short the weakest, rebalanced) printed a Sharpe around -1.4 over the sample. A time-series trend filter on a single strength line scraped a Sharpe near +0.3, inside the range you get from noise plus a few lucky trends, before costs. Mean-reversion on the strength z-score landed slightly negative. None of this survives transaction costs or out-of-sample testing, and I would not trade any of it. The reason is structural, not a tuning failure. The strength index is a recombination of the same pair returns you already had. It adds no information; it reorganizes information. A perfect attribution of where a move came from carries no claim about where the next move goes. The cumsum is a clean accounting of the past, and the past, accounted for cleanly, is still the past. If you want a cross-currency model that reaches for actual return drivers rather than a relabeling, Dean Markwick's ["A Fundamental FX Factor Model"](https://dm13450.github.io/2023/06/26/A-Fundamental-FX-Factor-Model.html?ref=aligrithm.com) goes the other direction: regress currency returns on macro factor proxies (rates, equities, commodities) to explain why a currency moved, not just to restate that it did. That is the harder, more honest path when the goal is prediction. The decomposition in this article is the diagnostic you run first, before you go looking for factors. So treat this for what it is: an attribution tool. When EURUSD drops, the decomposition tells you whether the euro sagged across the board or the dollar bid up everywhere, which changes how you read the rest of your screen and whether a "EUR trade" is really a "USD trade" wearing a different ticker. That is genuinely useful for understanding a session. It is not an edge, and the moment someone charts these lines and draws trend lines on them as if they were a leading indicator, they have wandered back into the same overfit the rest of this pillar warns about, the one examined in ["Cross-Pair Signals: Can EUR Predict GBP?"](https://aligrithm.com/cross-pair-signals-can-eur-predict-gbp/). ## KEY POINTS - Every FX pair is a ratio of two hidden currency values, so a single pair's move cannot tell you which currency actually moved. In logs, the pair return equals base strength minus quote strength. - The system is underdetermined by one degree of freedom. Pin one currency as the anchor (or use ridge, which picks the minimum-norm, sum-to-zero solution). Strength is always relative, never absolute. - Replace the per-bar optimizer with one constant design matrix: rows are pairs, columns are currencies, `+1` base and `-1` quote. Solve every bar at once with a single ridge inverse. Deterministic, fast, no lookahead. - The index is a plain cumsum of one-bar strength changes, so every value depends only on the past. - Validate by reconstruction: the difference of two solved strengths must retrace a real pair. Around 0.999 correlation confirms the algebra, nothing more. - It is attribution, not prediction. Cross-sectional momentum, trend, and mean-reversion on the strength lines lost or broke even in testing. Use it to see where a move came from, never to forecast where it goes. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Trading Systems and Methods - Perry Kaufman (Amazon)](https://www.amazon.com/s?k=Trading+Systems+and+Methods+Perry+Kaufman&ref=aligrithm.com) - [Cybernetic Trading Strategies - Murray Ruggiero (Amazon)](https://www.amazon.com/s?k=Cybernetic+Trading+Strategies+Murray+Ruggiero&ref=aligrithm.com) - [The Art of Currency Trading - Brent Donnelly (Amazon)](https://www.amazon.com/s?k=The+Art+of+Currency+Trading+Brent+Donnelly&ref=aligrithm.com) ### 8.1 Market Data Is Social Data: From Fractals to Topological Data Analysis URL: https://aligrithm.com/market-data-is-social-data-from-fractals-to-topological-data-analysis/ Last updated: 2026-07-27T13:26:00.000Z A price is a number two strangers agreed on for one instant before they disagreed again. Stack those instants into a chart and you are not looking at a physical process like temperature or rainfall. You are looking at the residue of people arguing about value, each with a different horizon, a different position, and a different fear. The participants disagree by construction, because a buyer and a seller hold opposite views and trade anyway, and that disagreement is what moves the tape. So the first claim of this whole pillar is blunt: market data is social data, and any model that forgets the humans gets the shape of the data wrong. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/5.jpg) The shape it gets wrong is the tail. A century of price data shows violent moves far too often for the comfortable bell curve, with kurtosis that fat-tail thinking demands and the Gaussian forbids. The distribution of daily index returns is not normal, never has been, and the gap is not a rounding error. It is the difference between a model that says a 1987 or a 2008 happens once in the lifetime of the universe and a market that produces them every couple of decades. ## The random walk, and why it breaks Start with the model markets are usually taught through. Louis Bachelier put Brownian motion under prices in 1900, and the logic is clean: if a price showed an exploitable pattern, traders would arbitrage it away, so what remains is a Gaussian process with independent increments, a martingale wandering with no memory. The appeal is that it is tractable. You get closed forms, you get option prices, you get a tidy lattice of probabilities. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/2.png) The trouble is the data refuses to live on that lattice. Four failures show up the moment you look at real series. The Gaussian underweights the size of typical moves, so real markets are wilder than it predicts. Large moves cluster in time rather than scattering independently, which a memoryless process cannot produce. Price changes carry identifiable structure that does not behave like a fair coin, which is why technical traders, for all the mysticism, are not chasing nothing. And the distribution itself drifts across decades, so the process is not even stationary. Mandelbrot made the point in 1963 and it still holds: the Brownian model is a beautiful object that does not describe the thing it claims to describe. ## Mandelbrot's fractal turn ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/6.jpg) Mandelbrot's fix was to keep the randomness and change the geometry. A fractal is a shape that looks the same as you zoom, rough at every scale, with no length where it suddenly turns smooth. Coastlines do this. So do price charts: hide the axis labels and you cannot tell an hourly chart from a daily one, because the roughness repeats across scales. That self-similarity is not a metaphor here. It is a measurable property of the increments. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/3.jpg) The 1997 Multifractal Model of Asset Returns, which Mandelbrot built with Fisher and Calvet, deforms clock time to capture two facts the Gaussian misses: returns show long memory, so calm and storm arrive in runs, and the tails stay heavy at every horizon. The way to test it is to measure how the moments of price changes scale with the time interval. Take the average of the absolute return raised to a power q, and watch how it grows as you stretch the interval. $$ \\mathbb{E}\\big\[\\,|X(t+\\Delta t) - X(t)|^{\\,q}\\,\\big\] \\;\\propto\\; (\\Delta t)^{\\,\\zeta(q)} $$ Read it plainly. The left side is the typical size of a q-th power of a move over a window of length delta-t. The right side says that size grows as the window raised to some exponent, zeta of q. We sweep q from small to large to probe the small moves and the extreme moves separately, and the whole question is the shape of zeta. For a plain random walk zeta is a straight line, equal to q over two, because one exponent governs all scales. For real markets zeta bends into a curve, and that curvature is the entire content of the word multifractal: it means no single Hurst exponent describes the series, the small moves and the large moves scale by different rules, and the heavy tails are baked into the scaling rather than glued on afterward. Mandelbrot's own summary was that price is a multiscaling process with long tails and long memory. Note what it does not claim. It describes the statistics of wildness. It does not forecast tomorrow. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## The social-media era made the cause visible Mandelbrot's work sat mostly ignored until 2008, when the Lehman collapse sent people hunting for any framework that took crashes seriously, and his book on the misbehavior of markets sold out. At the same moment a separate group stopped modeling the symptom and went after the cause: if prices are social data, measure the society directly. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/4.jpg) Bollen and co-authors published "Twitter Mood Predicts the Stock Market" in 2010 and reported 86.7 percent accuracy calling the daily direction of the Dow from public mood. Treat that number with the skepticism it earns. It is one sample of one period, a short window, a single market, and a headline accuracy figure with no transaction costs, no out-of-sample wall, and every degree of freedom that sentiment pipelines quietly burn. The result is suggestive, not bankable, and the honest reading is that a signal showed up in one slice of history, which is where every overfit also shows up. What survives the skepticism is the mechanism, not the percentage. Khurshid Ahmad's framing is the durable part: the volatility clustering Mandelbrot spotted in 1963 has a source, and the source is information arrival, which in the modern market is a social conversation about the instrument. Volatility clusters because attention clusters. ## Topology enters: shapes instead of distributions The 2010s added a third lens. Topological data analysis treats a stretch of multivariate price history as a cloud of points in high-dimensional space and asks about its shape, the loops and voids and connected pieces, rather than its distribution. Persistent homology records which of those features survive as you blur the cloud at coarser and coarser resolution, so noise dies fast and genuine structure persists. The selling point is robustness: the topology barely moves under small perturbations, which makes it a filter that passes structure and rejects jitter. ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/1-1.png) The headline result is that the loop-structure of the market grows agitated before a crash. Gidea and Katz showed persistence landscapes spiking ahead of the 2000 dot-com and 2008 collapses, and the giotto-tda library that Tauzin and colleagues built reproduces the exercise in Python. Hold the same line here as with the Twitter number. Calling two famous crashes after the fact, on data that contains those crashes, is an in-sample demonstration, and an early-warning index that lit up before the two events everyone already studies has not yet proven it will light up before the next one without crying wolf in between. The original telling of this story claims TDA plus machine learning can accurately predict future market activity. That oversells it. TDA gives a turbulence indicator, a measure that tends to rise as the correlation structure deforms toward a critical state, and a turbulence indicator is a warning, not a forecast. It belongs next to the variance-ratio and entropy diagnostics in this pillar, not above them. ## What every layer agrees on Schadner's 2021 multifractal fluctuation analysis closes the loop by measuring sentiment itself the way Mandelbrot measured prices, and it finds the same fingerprint: sentiment carries long memory, the multifractality comes from nonlinear correlations and fat tails, and the persistence is strongest when fear dominates. His split is greed against fear, and the asymmetry is the useful bit. When fear wins the market overreacts and the structure gets sharply persistent, and when optimism wins the series drifts back toward a random walk. Crashes are more structured than rallies, which is why the tools in this pillar earn their keep on the downside. Line up the four lenses and they point at one thing. Mandelbrot says the geometry is rough and self-similar. Bollen says the roughness tracks a measurable social mood. Schadner says the mood is itself multifractal. The topologists say the same structure shows up as shape. None of them is selling a forecast. Mandelbrot, asked about prediction, said the first job is to describe, and his model delivers exactly that and stops. The geometry and the topology tell you the kind of system you are trading, one that is social, fat-tailed, memory-laden, and prone to phase transitions. They do not tell you the price on Friday, and the moment a vendor says they do, you are back in the overfit you started in. Treat the fractal-to-topology arc as the reason to respect the tails and size for them, which is the through-line of ["Fat Tails: Why Gaussian Thinking Breaks Trading Systems"](https://aligrithm.com/fat-tails-why-gaussian-thinking-breaks-trading-systems/) and ["Why Financial Markets Are Complex Systems"](https://aligrithm.com/why-financial-markets-are-complex-systems/), not as a crystal ball. ## KEY POINTS - Market data is social data, the residue of participants with opposite views trading anyway, so any model that drops the humans gets the tail shape wrong. - The Brownian random walk fails on four counts: moves are larger than Gaussian, they cluster in time, they carry structure unlike a coin toss, and the distribution drifts across decades. - Mandelbrot's multifractal model keeps the randomness and fixes the geometry. Measure how moments scale with the time window, and the scaling exponent bends into a curve, which means heavy tails and long memory are built into the scaling, not added on. - The Twitter-mood and TDA-crash results are in-sample demonstrations on famous events. The durable lesson is the mechanism, that volatility clusters because attention and information arrival cluster, not the headline accuracy figure. - TDA produces a turbulence warning as the correlation structure deforms toward a critical state. That is a diagnostic alongside variance-ratio and entropy, not a forecast, and the "accurately predict" framing oversells it. - Every lens agrees the market is social, fat-tailed, and memory-laden, and crashes are more structured than rallies. Use that to respect and size for the tails, not to claim you can time them. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [A Multifractal Model of Asset Returns - Mandelbrot, Fisher, Calvet (1997)](https://users.math.yale.edu/~bbm3/web%5Fpdfs/Cowles1164.pdf?ref=aligrithm.com) - [Twitter Mood Predicts the Stock Market - Bollen, Mao, Zeng (2010)](https://arxiv.org/abs/1010.3003?ref=aligrithm.com) - [The Wisdom of Twitter Crowds - Azar, Lo](https://papers.ssrn.com/sol3/papers.cfm?abstract%5Fid=2756815&ref=aligrithm.com) - [Topological Data Analysis of Financial Time Series: Landscapes of Crashes - Gidea, Katz (2018)](https://arxiv.org/abs/1703.04385?ref=aligrithm.com) - [giotto-tda: A Topological Data Analysis Toolkit - Tauzin et al. (2020)](https://arxiv.org/abs/2004.02551?ref=aligrithm.com) - [On the Persistence of Market Sentiment: A Multifractal Fluctuation Analysis - Schadner (2021)](https://www.sciencedirect.com/science/article/abs/pii/S0378437121005343?ref=aligrithm.com) ### 6.9 The Difference Between Signal Quality and Portfolio Quality URL: https://aligrithm.com/the-difference-between-signal-quality-and-portfolio-quality/ Last updated: 2026-07-27T13:25:34.000Z A signal can be excellent and the portfolio built on it can be garbage, and a mediocre signal can power a fine portfolio. These are two different quality dimensions, measured with different numbers, and conflating them is how traders fall in love with a high-hit-rate indicator that produces a flat, expensive equity curve. ["Why Portfolio Construction Is Part of the Signal"](https://aligrithm.com/why-portfolio-construction-is-part-of-the-signal/) argued that construction is part of the alpha; this draws the line between the two qualities so you know which one you are actually measuring when you look at a backtest. ## Signal quality is about the prediction Signal quality asks one question: does the indicator predict the thing it claims to predict? You measure it on the prediction itself, before any portfolio exists. The information coefficient, the correlation between the forecast and the subsequent return, is the canonical number. Hit rate, the fraction of calls that point the right way, is another. Rank correlation, from ["Ranking Beats Forecasting for Many Trading Problems"](https://aligrithm.com/ranking-beats-forecasting-for-many-trading-problems/), is the right one for a ranking signal. All of these grade the signal in isolation: given the forecast, how well does reality follow it? A high information coefficient says the signal carries genuine information. It says nothing about whether you can make money on it. The trap is that signal quality is necessary and nowhere near sufficient. An indicator with a beautiful information coefficient can still lose money, because the question of whether the prediction is right is separate from the question of whether you can harvest it. Plenty of statistically real signals are uncapturable: they predict moves smaller than the cost of trading them, or they fire only in names too illiquid to size, or they cluster their bets into one correlated factor so the apparent breadth is an illusion. ## Portfolio quality is about the harvested return Portfolio quality asks the other question: after construction, costs, and correlation, what does the actual return stream look like? You measure it on the realized P&L, with Sharpe ratio, drawdown, turnover, and the cost-adjusted return. A good portfolio has a high Sharpe because its bets are genuinely diversified, a tolerable drawdown because risk is not concentrated, and a return that survives realistic costs because turnover is controlled. These numbers depend on the construction as much as the signal, which is the whole point of separating them. $$ \\text{SR}\_{\\text{portfolio}} \\approx \\text{IC} \\times \\sqrt{\\text{breadth}} \\times (\\text{construction efficiency}) $$ A rough decomposition makes the relationship concrete. The portfolio Sharpe is roughly the information coefficient (signal quality) times the square root of breadth (how many genuinely independent bets you make) times a construction efficiency factor (how well the build captures the signal without leaking it to costs and correlation). The fundamental law of active management is the first two terms; the third is everything this pillar has been about. A high IC with low breadth, ten correlated bets pretending to be ten independent ones, gives a poor Sharpe. A modest IC spread across many independent bets and built efficiently gives a strong one. The signal sets the ceiling; the construction decides how much of the ceiling you reach. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Why you measure both, separately Keep the two scorecards apart because they fail for different reasons and you fix them in different places. A weak portfolio with a strong signal is a construction problem: your bets are correlated, your costs are eating the edge, your sizing is concentrated. You fix it in the portfolio layer, not by hunting for a better signal. A strong-looking portfolio with a weak signal is a different and more dangerous case, usually a sign that the backtest's return came from a few lucky concentrated positions rather than from the signal's breadth, and it will not repeat. Reporting only the blended backtest hides which of these you have. Report the signal's information coefficient and breadth alongside the portfolio's Sharpe and costs, and you can see whether the strategy works because the prediction is good, because the construction is good, or because you got lucky in-sample. ## Visualizing the two qualities ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_149-visual-0.png) ## KEY POINTS - Signal quality and portfolio quality are different dimensions measured with different numbers. A great signal can power a garbage portfolio, and a mediocre signal can power a fine one. - Signal quality grades the prediction in isolation with information coefficient, hit rate, or rank correlation. A high information coefficient says the signal carries information, not that you can make money on it. - Many statistically real signals are uncapturable: they predict moves smaller than trading costs, fire only in illiquid names, or cluster their bets into one correlated factor so the breadth is an illusion. - Portfolio quality grades the realized return with Sharpe, drawdown, turnover, and cost-adjusted return. These depend on construction as much as on the signal. - Portfolio Sharpe decomposes roughly into information coefficient times the square root of breadth times construction efficiency. The signal sets the ceiling; the construction decides how much of it you reach. - Measure both separately because they fail for different reasons. A weak portfolio with a strong signal is a construction problem; a strong portfolio with a weak signal is usually lucky concentrated bets that will not repeat. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Systematic Trading - Robert Carver (Amazon)](https://www.amazon.com/s?k=Systematic+Trading+Robert+Carver&ref=aligrithm.com) - [Trading Systems - Urban Jaekle Emilio Tomasini (Amazon)](https://www.amazon.com/s?k=Trading+Systems+Urban+Jaekle+Emilio+Tomasini&ref=aligrithm.com) ### 6.8 Why Portfolio Construction Is Part of the Signal URL: https://aligrithm.com/why-portfolio-construction-is-part-of-the-signal/ Last updated: 2026-07-27T13:25:33.000Z Traders draw a clean line between the signal and the portfolio: the signal decides what to trade, the portfolio is just plumbing that turns the decision into positions. That line is fake. How you combine, weight, and net your positions changes what the strategy actually predicts, so portfolio construction is not a downstream wrapper around the signal; it is part of the signal. Two desks running the identical raw forecast and different construction end up with different strategies, different return streams, and different risk, which is why ["From Indicator Value to Expected Value"](https://aligrithm.com/from-indicator-value-to-expected-value/) and the sizing articles in this pillar are not separate from the alpha. They are the alpha. ## The same forecast, two portfolios, two strategies Take one forecast: a momentum score across forty futures. Construct it as an equal-weight long/short and you have a diversified momentum strategy whose return is the cross-sectional spread. Construct the identical forecast with volatility weighting and a correlation-aware optimizer and you have shifted risk away from the crowded, correlated names toward the diversifying ones, and your return stream now looks different, often with a higher Sharpe and a different drawdown profile, from the same underlying score. Nothing about the signal changed. The construction changed what the signal expresses, because the construction decided which of the forecast's many bets actually carry size. This is the part retail skips and the part that separates a backtest from a strategy. You can have a genuinely predictive forecast and destroy it in construction by concentrating risk in a few correlated positions, or by letting one volatile instrument dominate, or by netting two sub-signals that should have been kept separate. The forecast set the direction; the construction set how much of each direction you actually own, and that second decision moves the result as much as the first. ## Correlation is where construction earns its keep The clearest case is correlation. A forecast that says "long ten energy futures" looks like ten bets and is closer to one, because the energy complex moves together, so equal-weighting them stacks ten times the intended risk into a single factor. Construction that accounts for the correlation structure recognizes the ten as nearly one bet and sizes them down accordingly, freeing risk for genuinely independent positions elsewhere. The raw forecast cannot see this; it scores each instrument in isolation. The portfolio step is where the relationships between instruments enter the strategy, and those relationships carry real information about risk that the per-instrument signal never contained. A proper systematic framework builds this in deliberately by optimizing instrument weights and forecast weights with correlations in view, rather than sizing each position as if it were alone. The weights are not administrative bookkeeping; they are a second model, layered on the first, that says how independent these bets really are. Get the weights wrong and a diversified-looking book is secretly a concentrated one, and the realized risk overshoots the target by the amount of hidden correlation you ignored. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Construction also encodes your costs and constraints Beyond correlation, construction is where every real-world constraint enters and reshapes the signal. The no-trade buffers from ["Cost-Aware Ranking: The Missing Step in Cross-Sectional Strategies"](https://aligrithm.com/cost-aware-ranking-the-missing-step-in-cross-sectional-strategies/) live here, and they change the effective signal by holding stale positions longer than the raw forecast wants. Position limits, capital limits, and gross-exposure caps all live here, and each one bends the realized strategy away from the unconstrained ideal. A forecast tested without these constraints is a forecast you will never actually trade, so the construction layer is where the strategy becomes real and where the difference between the paper return and the live return is decided. Treat construction as part of the research, not as an afterthought bolted on once the signal looks good, because the construction choices are themselves alpha decisions that you can get right or wrong. ## Visualizing construction as signal ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_148-visual-0.png) ## KEY POINTS - The line between signal and portfolio is fake. How you combine, weight, and net positions changes what the strategy predicts, so portfolio construction is part of the signal, not downstream plumbing. - The identical forecast under two constructions becomes two strategies. Equal-weight long/short and a correlation-aware vol-weighted build produce different return streams, Sharpes, and drawdowns from the same score. - You can have a predictive forecast and destroy it in construction by concentrating risk in correlated names, letting one volatile instrument dominate, or netting sub-signals that should stay separate. - Correlation is where construction earns its keep. Ten correlated energy futures are closer to one bet than ten; equal-weighting stacks the risk, and correlation-aware sizing frees risk for independent positions. - Instrument and forecast weights are a second model layered on the first, encoding how independent the bets really are. Get them wrong and a diversified-looking book is secretly concentrated, overshooting the risk target. - Construction encodes costs and constraints too: no-trade buffers, position limits, capital limits, gross caps. A forecast tested without them is one you will never trade, so treat construction as research, not an afterthought. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Systematic Trading - Robert Carver (Amazon)](https://www.amazon.com/s?k=Systematic+Trading+Robert+Carver&ref=aligrithm.com) - [Trading Systems - Urban Jaekle Emilio Tomasini (Amazon)](https://www.amazon.com/s?k=Trading+Systems+Urban+Jaekle+Emilio+Tomasini&ref=aligrithm.com) ### 6.7 From Indicator Value to Expected Value URL: https://aligrithm.com/from-indicator-value-to-expected-value/ Last updated: 2026-07-27T13:25:33.000Z An indicator outputs a number. A position needs a different number. The gap between them is where most systems quietly break, because traders take the raw output of an RSI or a moving-average crossover and shove it straight into a position size, as if an RSI of 70 told them how much to bet. It does not. The indicator value is a reading on a scale nobody calibrated against money; the expected value is what that reading is worth in risk-adjusted return. Turning one into the other is a real step, and skipping it means your position sizes are an accident of whatever scale the indicator happened to use. This is the translation layer underneath ["Why Volatility-Adjusted Position Sizing Matters"](https://aligrithm.com/why-volatility-adjusted-position-sizing-matters/): before you size a forecast, you have to make the forecast mean something. ## The indicator scale is arbitrary Every indicator carries a scale that came from its construction, not from any relationship to returns. RSI lives between 0 and 100 because of how it normalizes gains against losses. A moving-average crossover outputs a price difference in whatever units the instrument trades. A z-score lives in standard deviations. None of these scales knows anything about how much money the signal predicts, so a "strong" reading on one indicator and a "strong" reading on another are strong in different, incomparable currencies. Feed the raw value into a position and you are letting the indicator's internal arithmetic decide your bet size, which is arbitrary. ## Calibrate the reading against realized returns The fix is to map the indicator value to the expected risk-adjusted return that historically followed it. You bucket the indicator's range, measure the forward return in each bucket, express it in volatility units so it is comparable across instruments, and build the mapping from indicator reading to expected risk-adjusted return. $$ f(x) = \\frac{\\mathbb{E}\[\\,\\text{return} \\mid \\text{indicator} = x\\,\]}{\\sigma} $$ The calibrated forecast is the expected return conditional on the indicator reading, divided by volatility to put it in risk-adjusted units. This is the number you size on, not the raw reading. It answers the question the indicator never did: given this reading, how much return per unit of risk do I actually expect? An RSI of 30 and an RSI of 70 might map to expected values of opposite sign and unequal magnitude, and the mapping, not the 30 and 70, is what drives the position. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Scale it to a common target Once you have an expected value, normalize every signal to a common scale so different indicators can be combined and compared. A disciplined forecast framework standardizes each forecast to a target average absolute value, around 10 on a scale where 20 is a strong conviction, by scaling the calibrated forecast by whatever constant brings its long-run average magnitude to that target. After this step, a 10 from the RSI system and a 10 from the crossover system mean the same thing, an average-strength bet, and they can be averaged into a combined forecast with sensible weights. Without the common scale, combining indicators is adding apples to furlongs, and the loudest-scaled indicator silently dominates the blend. The honest caveat: the mapping is fit to history, so it inherits every bias of the sample. If the relationship between the indicator and forward returns was a regime artifact, the calibration encodes the artifact and hands you a confident expected value that does not hold out of sample. Estimate the mapping on enough data, keep it smooth rather than letting it chase every bucket's noise, and treat a too-good calibration as a warning, not a prize. ## Visualizing the translation ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_147-visual-0.png) ## KEY POINTS - An indicator outputs a number on an arbitrary scale; a position needs a number that means risk-adjusted return. Shoving raw indicator values into position sizes lets the indicator's internal arithmetic decide your bet size. - Indicator scales come from construction, not from returns. RSI's 0-to-100, a crossover's price units, and a z-score's standard deviations are incomparable currencies, so "strong" on one is not "strong" on another. - Calibrate by bucketing the indicator's range, measuring forward return in each bucket, and expressing it in volatility units. The mapping from reading to expected risk-adjusted return is what you size on, not the raw value. - The calibrated forecast is the expected return conditional on the indicator reading, divided by volatility. An RSI of 30 and 70 can map to expected values of opposite sign and unequal magnitude. - Normalize every calibrated forecast to a common target scale, around 10 average with 20 as strong conviction, so different indicators combine sensibly. Without it, the loudest-scaled indicator dominates the blend. - The mapping is fit to history and inherits its biases. Use enough data, keep the mapping smooth instead of chasing bucket noise, and treat a too-good calibration as a warning of a regime artifact. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Systematic Trading - Robert Carver (Amazon)](https://www.amazon.com/s?k=Systematic+Trading+Robert+Carver&ref=aligrithm.com) - [Trading Systems - Urban Jaekle Emilio Tomasini (Amazon)](https://www.amazon.com/s?k=Trading+Systems+Urban+Jaekle+Emilio+Tomasini&ref=aligrithm.com) ### 6.6 Why Z-Scoring Makes Ranking Cleaner URL: https://aligrithm.com/why-z-scoring-makes-ranking-cleaner/ Last updated: 2026-07-27T13:25:32.000Z Raw signals are not comparable. A momentum reading on Bitcoin and a momentum reading on a Treasury future come out of the same formula and live on completely different scales, because one instrument moves in whole percent a day and the other in basis points. Sort a universe by a raw signal and the loud, high-volatility names dominate the top and bottom of the ranking for no reason other than their scale, drowning the quiet names that may carry the same signal strength. Z-scoring fixes the scale problem before you ever sort, and it is the cheapest piece of hygiene that separates a clean cross-sectional system from a noisy one. This sits between ["Ranked Long/Short Systems Explained"](https://aligrithm.com/ranked-long-short-systems-explained/) and ["How to Build a Cost-Aware RSI Ranking System"](https://aligrithm.com/how-to-build-a-cost-aware-rsi-ranking-system/): the preprocessing step that makes a ranking mean what you think it means. ## The transform: subtract the mean, divide by the spread $$ z\_i = \\frac{x\_i - \\mu}{\\sigma} $$ The z-score of an instrument is its raw signal value minus the mean of the signal across the universe, divided by the standard deviation of the signal across the universe. The subtraction recenters every signal on zero, so positive means above-average and negative means below-average regardless of the raw units. The division rescales so that one z-score unit means the same thing for Bitcoin and for the bond: one standard deviation of that signal's own dispersion. After z-scoring, a +2 on copper and a +2 on gold are genuinely comparable; both say "two standard deviations rich on this signal," and the units they started in are gone. ## Why this is more than just ranking A fair question: if you are going to sort anyway, why not skip z-scoring and rank the raw values directly, since ranking already discards scale? Because rank throws away too much. Ranking compresses the universe to its order and forgets the spacing, so a name that is far and away the best looks identical to a name that barely edges out the next one. Z-scoring keeps the spacing, the magnitude of how far each name sits from the crowd, which carries real information about conviction. A z-score of +3 is a genuine outlier worth a bigger position than a +0.5, and the weighting schemes from ["Ranked Long/Short Systems Explained"](https://aligrithm.com/ranked-long-short-systems-explained/) can use that spacing directly. You z-score when you want comparability and conviction; you fall back to pure rank only when the signal's magnitudes are too unreliable to trust and you only believe the order. ## The robust version, because outliers wreck the mean Plain z-scoring has a weakness: the mean and standard deviation are themselves wrecked by outliers. One instrument with a berserk signal value drags the mean and inflates the standard deviation, which then squashes every other name's z-score toward zero, so a single crazy reading quietly mutes the whole rest of the universe. The robust fix swaps the mean for the median and the standard deviation for the median absolute deviation, both of which shrug off extreme values. You compute the median of the signal, the median of the absolute distances from that median, and z-score against those instead. The outlier still scores as extreme, which is correct, but it no longer distorts the scores of everyone else. You also cap the z-scores at some bound, the same way a disciplined forecast framework caps a forecast at twice its average value, so that a 10-sigma data glitch cannot demand a 10x position. Recompute the mean and spread cross-sectionally each rebalance, because the right scale this month is not the right scale next month, and a z-score against a stale dispersion is just a raw signal wearing a costume. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Visualizing z-scoring ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_146-visual-0.png) ## KEY POINTS - Raw signals across instruments are not comparable because they live on different scales. Sort the raw values and high-volatility names dominate the ranking for no reason but their scale, drowning quiet names with equal signal strength. - Z-scoring subtracts the cross-sectional mean and divides by the cross-sectional standard deviation, so every signal is recentered on zero and rescaled into standard-deviation units. A +2 then means the same thing on every instrument. - Z-scoring beats pure rank because it keeps the spacing. Ranking forgets how far each name sits from the crowd, but a +3 is a real outlier deserving a bigger position than a +0.5, and weighting schemes can use that conviction. - Use pure rank only when the magnitudes are too unreliable to trust and you believe only the order. - Plain z-scoring is wrecked by outliers, since one berserk value drags the mean and inflates the standard deviation, squashing everyone else toward zero. Use the median and median absolute deviation for a robust version. - Cap the z-scores so a data glitch cannot demand a giant position, and recompute the mean and spread each rebalance, because a z-score against stale dispersion is a raw signal in disguise. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Systematic Trading - Robert Carver (Amazon)](https://www.amazon.com/s?k=Systematic+Trading+Robert+Carver&ref=aligrithm.com) - [Trading Systems - Urban Jaekle Emilio Tomasini (Amazon)](https://www.amazon.com/s?k=Trading+Systems+Urban+Jaekle+Emilio+Tomasini&ref=aligrithm.com) ### 6.5 When an Alpha Metric Is U-Shaped URL: https://aligrithm.com/when-an-alpha-metric-is-u-shaped/ Last updated: 2026-07-27T13:25:32.000Z Ranking assumes your signal is monotone: more of the metric means more expected return, all the way up. Sort by it, buy the top, short the bottom, done. That assumption is a trap, because plenty of real signals are not monotone at all. Some are U-shaped, where both extremes predict the same thing and the middle predicts the opposite, and if you feed a U-shaped metric into a linear ranking you will systematically buy the worst names and call it alpha. ["Ranked Long/Short Systems Explained"](https://aligrithm.com/ranked-long-short-systems-explained/) took monotonicity for granted; this is what you check before you trust it. ## The shape you assume versus the shape you have A monotone alpha metric draws a straight-ish upward line: low metric, low return; high metric, high return. Ranking is the right tool because the order of the metric is the order of expected return. A U-shaped metric draws a valley: the lowest values and the highest values both sit at high expected return, and the middle sits at low expected return. Value can look like this. A moderately cheap stock outperforms, but the very cheapest names are often cheap because they are distressed, so the relationship bends back down at the extreme and the deepest-value bucket underperforms the merely-cheap one. Volatility can look like this too, and so can many sentiment and flow metrics where both crowding extremes mean something and the middle is noise. Rank a U-shaped metric naively and the system does exactly the wrong thing at one end. It puts the distressed extreme in your long slice because the raw metric is highest there, when that extreme is the part you wanted to avoid. The ranking is faithfully sorting a number whose relationship to return reverses inside the range, so the top of your sort is a mix of genuine alpha and a pocket of names that predict the opposite. ## See the shape before you rank The discipline is to plot expected return against the metric before assuming a direction, in buckets. Slice the universe into deciles of the raw metric, compute the realized forward return in each decile, and look at the shape. A clean upward staircase says monotone, rank away. A valley says U-shaped, and a naive rank is broken. A hump (inverted U) says the middle is best and both extremes are bad, which breaks ranking the other way. You cannot know which case you are in from the name of the signal; you have to look, and the bucket plot is the cheapest way to look. ## Folding the metric back into something monotone Once you see a U, the fix is to transform the metric so it becomes monotone in expected return again. $$ g(x) = |x - x^\*| \\quad\\text{where } x^\* = \\arg\\min\_x \\, \\mathbb{E}\[\\,\\text{return} \\mid x\\,\] $$ If the expected return is lowest at some interior value and rises toward both ends, fold the metric around that low point by taking the distance from it. The transformed metric is now monotone, both extremes map to a large value, the dead middle maps to a small value, and ranking on it puts the genuinely high-return names at the top. For a hump you flip the sign, ranking on negative distance from the peak. The transform is the operational repair: you keep the ranking machinery, you just feed it a metric that actually increases with expected return instead of one that bends. Estimate the turning point carefully, because a turning point fit to noise is one more way to overfit, and a U that only exists in-sample will hand you a confident, wrong transform. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Visualizing the U-shape ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_145-visual-0.png) ## KEY POINTS - Ranking assumes the metric is monotone in expected return. Many real signals are not, and a U-shaped metric, where both extremes are high-return and the middle is low, breaks the assumption. - Feed a U-shaped metric into a linear ranking and it buys the wrong extreme. The distressed deep-value or deep-vol names sit at the top of the raw sort while predicting the opposite of what you want. - Examples: value bends back down at the distressed extreme, and volatility, sentiment, and flow metrics often carry information at both crowding extremes with a noisy middle. - See the shape before ranking. Bucket the universe into deciles of the raw metric, plot realized forward return per decile, and look. An upward staircase is monotone; a valley is U-shaped; a hump is inverted. - Fix a U by folding the metric around its turning point, ranking on distance from the low point so both extremes map high and the dead middle maps low. Flip the sign for a hump. - Estimate the turning point honestly. A U that exists only in-sample hands you a confident, wrong transform, so treat the fold as another place overfitting hides. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Systematic Trading - Robert Carver (Amazon)](https://www.amazon.com/s?k=Systematic+Trading+Robert+Carver&ref=aligrithm.com) - [Trading Systems - Urban Jaekle Emilio Tomasini (Amazon)](https://www.amazon.com/s?k=Trading+Systems+Urban+Jaekle+Emilio+Tomasini&ref=aligrithm.com) ### 6.40 From Econophysics to Practical Trading Signals URL: https://aligrithm.com/from-econophysics-to-practical-trading-signals/ Last updated: 2026-07-27T13:25:50.000Z Econophysics gives you a beautiful understanding of markets and almost no money directly. The fat tails, the anomalous diffusion, the power laws, the entropy, the complex-systems framing of this pillar's final run, all of it explains why markets behave as they do with a depth that charting and folklore never approach, and none of it is a trading signal on its own. The gap between understanding a market and profiting from one is wide, and it is where most of the elegant physics dies, because a model that describes price beautifully is not the same as a rule that extracts money from it after costs. This closes Pillar 6 by being honest about what the physics buys you and what it does not. ## Understanding is not an edge The seduction of econophysics is that the models are genuinely true in a way most trading lore is not. A Lévy distribution really does fit the body of returns better than a Gaussian; markets really do diffuse anomalously; crashes really do follow power laws. The trouble is that being a correct description and being a profitable signal are different properties, and the physics delivers the first while staying silent on the second. Knowing that returns are fat-tailed tells you to size for catastrophe; it does not tell you when the catastrophe comes. Knowing that a series superdiffuses tells you it trended; it does not tell you it will keep trending. The descriptive truth is real and the predictive payoff has to be extracted separately, against all the obstacles the rest of this pillar catalogs. This is why the econophysics literature is full of models that explain everything and predict nothing tradable. The log-periodic crash precursor, the Tsallis-entropy return distribution, the multifractal volatility model, each captures a real feature of markets and each, when you try to trade it, runs into the same wall: the structure it describes is either too weak to beat costs, too unstable to persist, or already priced by everyone who read the same papers. The elegance of the description is not evidence of tradability, and the temptation to assume it is, to believe that a model this beautiful must pay, is exactly the trap ["The Three Stages of a Trading Idea: Absurd, Familiar, Inevitable"](https://aligrithm.com/the-three-stages-of-a-trading-idea-absurd-familiar-inevitable/) warned about, mistaking your own excitement for an edge. ## The bridge is the same discipline as everything else Crossing from understanding to a signal requires the unglamorous machinery this entire series has built. The physics tells you where to look, the departures from the random walk, the regimes where diffusion is anomalous, the conditions where structure appears, and finding where to look is a genuine contribution, because it beats searching blindly. But turning a promising place into a tradable signal demands the same steps as any other strategy: measure the structure honestly with proper error bands, as in ["Variance Ratio Tests for Traders"](https://aligrithm.com/variance-ratio-tests-for-traders/); charge realistic costs and confirm the edge survives them, as in ["Cost-Aware Ranking: The Missing Step in Cross-Sectional Strategies"](https://aligrithm.com/cost-aware-ranking-the-missing-step-in-cross-sectional-strategies/); size by volatility and respect the fat tail; check it out-of-sample with the skepticism about overfitting that haunts every article here; and build the portfolio and the loss control that turn a raw signal into something you can actually run. The physics, in other words, is an input to the front of the pipeline, not a replacement for the pipeline. It supplies hypotheses with better motivation than a data-mined pattern, an idea grounded in a mechanism rather than an accident of the backtest, which matters because a mechanistically motivated signal is less likely to be pure overfit. But a well-motivated hypothesis still has to survive the same testing as a poorly-motivated one, and many do not. The value of econophysics is that it raises the quality of your hypotheses and your understanding of risk; the work of converting a hypothesis into a strategy is unchanged, and skipping it because the physics is elegant is how smart people lose money with beautiful models. ## What the physics is actually for The honest conclusion is that econophysics earns its place not as a signal factory but as a way of seeing the market correctly, which protects you and occasionally points you somewhere useful. It protects you by making the fat tails, the regime shifts, and the complex-systems fragility vivid and permanent in your thinking, so you size and build for the market that exists rather than the Gaussian one that is convenient, which is the difference between surviving and not. It points you by suggesting where exploitable structure might live and why, giving your search a mechanism to chase instead of a blind sweep. And it disciplines your expectations by explaining why edges decay and why crashes are inevitable, so you neither over-trust a model nor get blindsided by the tail. That is a substantial return on understanding, and it is not the same as a return on capital, which only the full discipline of the pillar produces. The trader who takes econophysics as a worldview, sizes for its tails, hunts where it points, and then runs every candidate through the same ruthless testing and cost accounting as everything else, gets the best of it. The trader who takes it as a shortcut, who believes the beauty of the model substitutes for the verification, joins the long line of physicists who arrived in finance certain that markets would yield to elegant mathematics and learned, expensively, that the market does not care how beautiful your model is, only whether your edge survives contact with costs, capacity, and the tail. The whole of this pillar has been the bridge from understanding to survival, and the physics is the far bank, worth reaching and not, by itself, worth standing on. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Visualizing the bridge ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_180-visual-0.png) ## KEY POINTS - Econophysics gives a deep, true understanding of markets and almost no money directly. The fat tails, anomalous diffusion, power laws, and complex-systems framing explain market behavior without being trading signals. - A correct description and a profitable signal are different properties. The physics delivers the description and stays silent on the prediction: knowing returns are fat-tailed says size for catastrophe, not when it comes. - The literature is full of models that explain everything and predict nothing tradable, because the structure they describe is too weak to beat costs, too unstable to persist, or already priced by everyone who read the papers. - The elegance of a model is not evidence of tradability. Believing a beautiful model must pay is mistaking your excitement for an edge, the absurd-idea trap. - The physics is an input to the front of the pipeline, supplying better-motivated hypotheses, not a replacement for it. A well-motivated hypothesis still needs the same honest testing, cost accounting, sizing, and out-of-sample validation as any other. - Econophysics earns its place as a way of seeing the market correctly: it makes the tails and fragility permanent in your thinking, points your search at mechanisms, and disciplines your expectations, which is a return on understanding, not a return on capital. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Systematic Trading - Robert Carver (Amazon)](https://www.amazon.com/s?k=Systematic+Trading+Robert+Carver&ref=aligrithm.com) - [Trading Systems - Urban Jaekle Emilio Tomasini (Amazon)](https://www.amazon.com/s?k=Trading+Systems+Urban+Jaekle+Emilio+Tomasini&ref=aligrithm.com) ### 6.4 Cost-Aware Ranking: The Missing Step in Cross-Sectional Strategies URL: https://aligrithm.com/cost-aware-ranking-the-missing-step-in-cross-sectional-strategies/ Last updated: 2026-07-27T13:25:31.000Z A ranked long/short system has a hidden expense that never shows up in the signal research and wrecks the strategy in production: every time an instrument crosses a slice boundary, you trade. The signal updates, copper slips from rank 3 to rank 5, falls out of your long slice, and now you sell copper and buy whatever climbed into its place. Each of those crossings is a round trip with a cost, and a ranked book that rebalances on every twitch of the ranking can churn its entire portfolio many times over before the spread it was chasing ever pays off. ["Ranked Long/Short Systems Explained"](https://aligrithm.com/ranked-long-short-systems-explained/) built the book; this is the step almost everyone skips when they put it into production. ## The turnover is in the boundaries, not the signal The cost of a ranked system lives at the edges of the slices. Names deep in the long slice or deep in the short slice stay put; the churn comes from the marginal names hovering at the cutoff, flickering in and out as the score wobbles. A signal that reorders the universe gently still produces constant boundary crossings if the cutoff is a hard line, because two names near the threshold will swap rank back and forth on noise alone, and each swap is a pair of trades you pay for and earn nothing from. $$ \\text{net edge}\_i = \\underbrace{(\\hat{r}\_i - \\hat{r}\_{\\text{boundary}})}\_{\\text{rank-spread gain}} - \\underbrace{c\_i \\cdot \\Delta w\_i}\_{\\text{cost to switch}} $$ Before you act on a ranking change, compare the gain to the cost. The rank-spread gain is how much better the new name's expected return is than the one it would replace at the boundary; the cost to switch is the per-unit trading cost times the position change the swap requires. If a name is only marginally better than the one already in your slice, the rank-spread gain is small, and the cost to switch eats it. You should only make the trade when the gain clears the cost, which means a name has to be meaningfully better, not trivially better, to justify swapping into it. ## The fix: a buffer around the boundary The practical tool is a no-trade buffer, a band around each slice cutoff where you do nothing. Once a name is in your long slice it stays in until its rank deteriorates past the lower edge of the buffer, not merely past the cutoff itself; a new name only enters once it climbs past the upper edge. The buffer turns the hard boundary into a sticky zone, so the marginal names flickering on noise no longer trigger trades. You give up a little selection sharpness, holding a name a bit longer than the raw ranking would prefer, in exchange for killing the noise-driven turnover that was the real cost. This is the same logic as a position buffer in a single-instrument system, where you do not adjust a position until it drifts far enough from target to be worth the trade. The wider the buffer, the lower the turnover and the staler the book; the narrower, the fresher and the more expensive. You set the width from the actual cost of trading the universe, not from whatever makes the backtest prettiest, because the buffer is exactly the kind of knob that overfits if you sweep it for performance. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Costs decide which signals are even tradable Cost-aware ranking changes more than turnover; it changes which signals survive at all. A fast signal that reorders the universe daily might have a real gross edge and a negative net edge once you pay to chase it, while a slower signal with half the gross edge but a tenth the turnover clears costs comfortably. You cannot judge a ranking signal on its gross rank correlation alone. The right test runs the signal through the buffer, charges realistic costs on the trades that survive, and reports the net, because a strategy that looks brilliant before costs and bleeds after them is the most common way a cross-sectional backtest lies to you. ## Visualizing cost-aware ranking ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_144-visual-0.png) ## KEY POINTS - A ranked book's hidden cost is turnover. Every time an instrument crosses a slice boundary you trade, and a book that rebalances on every twitch can churn its whole portfolio before the spread pays off. - The churn lives at the slice boundaries, not in the signal. Marginal names hovering at the cutoff flicker in and out on noise alone, and each swap is a paid round trip that earns nothing. - Before acting on a ranking change, weigh the rank-spread gain (how much better the new name is) against the cost to switch. Only trade when a name is meaningfully better, not trivially better. - The fix is a no-trade buffer: a band around each cutoff where you hold. Names stay in until they deteriorate past the band's far edge, and new names enter only once they climb past it, which kills noise-driven turnover. - Buffer width trades turnover against staleness. Set it from the real cost of trading the universe, not from whatever makes the backtest prettiest, because it overfits if swept for performance. - Costs decide which signals are tradable at all. A fast signal can have a real gross edge and a negative net edge, so judge ranking signals on net performance after realistic costs through the buffer, never on gross rank correlation. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Systematic Trading - Robert Carver (Amazon)](https://www.amazon.com/s?k=Systematic+Trading+Robert+Carver&ref=aligrithm.com) - [Trading Systems - Urban Jaekle Emilio Tomasini (Amazon)](https://www.amazon.com/s?k=Trading+Systems+Urban+Jaekle+Emilio+Tomasini&ref=aligrithm.com) ### 6.38 The Three Stages of a Trading Idea: Absurd, Familiar, Inevitable URL: https://aligrithm.com/the-three-stages-of-a-trading-idea-absurd-familiar-inevitable/ Last updated: 2026-07-27T13:25:48.000Z Every trading idea that ever made money passed through three stages. First it sounded absurd, so strange that people wondered why anyone would suggest it. Then it became familiar, as evidence accumulated and the idea entered the conversation. Finally it became inevitable, so obvious in hindsight that everyone agreed they always knew it. The cruel part for a trader is that the money is in the first stage, and by the third stage the edge is gone, competed away by the same consensus that made the idea obvious. Understanding this arc tells you where to hunt and why the comfortable, well-accepted ideas are the worst place to look. This sits near the end of the pillar because it reframes the whole enterprise: the goal is to find ideas while they are still absurd. ## The arc of an idea Stage one is absurdity. A new idea about how markets work, that price increments are not independent, that a physical diffusion model describes returns, that you should rank instruments rather than forecast them, lands as nonsense to the established view, because it contradicts what everyone knows. The reaction is dismissal: that cannot be right, why would anyone think that. The idea has almost no adherents, no respectability, and, crucially, almost no capital deployed against it, which is precisely why the edge is largest here. The inefficiency the idea exploits is wide open, because the consensus has not only failed to close it, it denies the inefficiency exists. Stage two is familiarity. Evidence accumulates, a few practitioners demonstrate the idea works, papers appear, and the idea moves from absurd to plausible to discussed. More capital flows in as the respectability grows, and the edge begins to narrow, because each new entrant trades away a piece of the inefficiency. The idea is still profitable here but less so than at stage one, and the window is closing as familiarity spreads. This is where most traders discover an idea, which is the problem: by the time something is familiar enough to find easily, the easy money has already been taken. Stage three is inevitability. The idea is now obvious, taught, assumed, built into everyone's models. In hindsight it seems absurd that anyone ever doubted it, and that universal acceptance is exactly what kills the edge, because when everyone trades an idea the inefficiency it exploited is fully arbitraged and the returns vanish. The idea is true, more established than ever, and worthless as a source of edge, because being right is not the same as being early, and at stage three everyone is right together. ## The money is in the discomfort The arc has a brutal implication: the profitability of an idea is inversely related to its acceptance. The maximum edge sits at maximum absurdity, when the idea is most uncomfortable to hold and most likely to make you look foolish, and the edge decays monotonically as the idea gains acceptance. This is the structural reason that comfortable, consensus ideas do not pay, the ones everyone agrees on are at stage three, fully priced, and the only ideas with real edge are the ones that still feel wrong, that you can barely defend at a dinner party without being laughed at. This connects to the discipline premium from ["The Discipline Premium in Trading"](https://aligrithm.com/the-discipline-premium-in-trading/), but at the level of ideas rather than execution. Just as the discipline premium pays you for tolerating the discomfort of following rules, the absurdity premium pays you for tolerating the discomfort of holding a view the consensus rejects. Both are paid for enduring something unpleasant that most people will not endure, and both are durable for the same reason: human nature reliably avoids discomfort, so the supply of people willing to hold absurd-but-correct ideas stays small. The trader hunting for edge should be uncomfortable, because comfort means consensus and consensus means stage three. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## The trap: most absurd ideas are just wrong The arc is seductive and the seduction is dangerous, because it can be read as a license to believe any absurd idea, on the theory that absurdity signals edge. It does not. The overwhelming majority of absurd ideas are absurd because they are wrong, not because they are early, and the graveyard of trading is full of people who held a ridiculous view to the end, convinced they were at stage one of a three-stage arc that was actually a one-stage delusion. Absurdity is necessary for a large edge and nowhere near sufficient, and distinguishing the absurd-but-correct idea from the merely-absurd one is the entire difficulty, the thing that no framework solves for you. What separates the two is evidence, held to the same standard as everything else in this pillar. An absurd idea worth trading is one that is absurd to the consensus and supported by the data when you test it honestly, with out-of-sample validation, cost accounting, and the skepticism about overfitting that every signal demands. The arc tells you to look in uncomfortable places and to take seriously an idea that tests well even though it sounds wrong; it does not tell you to abandon the testing because the idea is exciting. The discipline is to be open to the absurd and ruthless in verifying it, holding the contrarian idea and the rigorous test at the same time, because the absurd idea that survives honest testing is the rare one worth the discomfort, and the absurd idea that you believe because it is absurd is just a bias wearing a contrarian costume. ## Visualizing the three stages ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_178-visual-0.png) ## KEY POINTS - Every profitable trading idea passes through three stages: absurd (people wonder why anyone would suggest it), familiar (evidence accumulates), and inevitable (obvious in hindsight). The money is in the first stage. - At the absurd stage the idea has almost no capital deployed against it, so the inefficiency is wide open and the edge is largest, precisely because the consensus denies the inefficiency exists. - At the familiar stage capital flows in and the edge narrows. Most traders discover an idea here, which is the problem: by the time it is easy to find, the easy money is taken. - At the inevitable stage universal acceptance fully arbitrages the inefficiency, so the idea is more established than ever and worthless as edge. Being right is not the same as being early. - Profitability is inversely related to acceptance. The maximum edge sits at maximum absurdity, when the idea is most uncomfortable to hold, which is the structural reason consensus ideas do not pay. - The trap is that most absurd ideas are wrong, not early. Absurdity is necessary for a large edge and nowhere near sufficient, so be open to the absurd and ruthless in verifying it with honest out-of-sample, cost-aware testing. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Systematic Trading - Robert Carver (Amazon)](https://www.amazon.com/s?k=Systematic+Trading+Robert+Carver&ref=aligrithm.com) - [Trading Systems - Urban Jaekle Emilio Tomasini (Amazon)](https://www.amazon.com/s?k=Trading+Systems+Urban+Jaekle+Emilio+Tomasini&ref=aligrithm.com) ### 6.37 Why Financial Markets Are Complex Systems URL: https://aligrithm.com/why-financial-markets-are-complex-systems/ Last updated: 2026-07-27T13:25:48.000Z A market is not a machine with a few dials you can turn to predict its output, and it is not a casino where every spin is independent. It is a complex system, a large collection of interacting agents whose collective behavior produces patterns that no individual agent intended and no simple equation captures. Calling markets complex is not hand-waving; it is a specific claim with testable consequences, the power-law distributions, the clustering of volatility, the crashes that behave like avalanches, all of which follow from the structure of a complex system and none of which fit the tidy random-walk picture. This pulls together the fat tails, the anomalous diffusion, and the entropy of the preceding articles into one frame, because they are all symptoms of the same complex-systems nature. ## Emergence: the whole is not the sum The defining feature of a complex system is emergence, where the interactions between many simple components produce collective behavior that you cannot predict from the components alone. No single trader decides the market will crash; the crash emerges from millions of individual decisions, each locally reasonable, interacting through prices and feedback loops into a collective cascade. This is why reductionism fails on markets: you can understand every participant's strategy perfectly and still not predict the aggregate, because the behavior lives in the interactions, not in the parts. The herding from the behavioral articles in this pillar is a microscopic input; the emergent output is a price bubble, and the bubble has properties the individual herding instinct does not. Emergence explains why markets resist the clean models that work in physics-with-few-bodies. A complex system has too many interacting degrees of freedom and too much feedback to collapse into a simple law, and the attempts to force one, the Gaussian returns, the efficient-market equilibrium, the single fitted distribution, all break on the parts of behavior that emerge from the interactions rather than from the average agent. The fat tails of ["Fat Tails: Why Gaussian Thinking Breaks Trading Systems"](https://aligrithm.com/fat-tails-why-gaussian-thinking-breaks-trading-systems/) are emergent: they come from the way agents' decisions correlate and cascade, not from any single agent's return distribution. ## Power laws are the fingerprint of complexity Complex systems leave a characteristic signature, and it is the power law. Across wildly different complex systems, earthquakes, avalanches, city sizes, word frequencies, the distribution of event sizes follows a power law, where events occur with a frequency proportional to their size raised to a negative power, so there is no typical scale and the rare giant event is a natural part of the same distribution as the common small ones. Markets show this fingerprint everywhere: the distribution of price moves, the distribution of trade sizes, the distribution of drawdowns. The power law is why the fat tail exists, why there is no characteristic size of a crash, and why the same statistical law governs a tiny fluctuation and a market-wide collapse. The earthquake analogy runs deep enough to be useful. After a large earthquake, aftershocks follow the Omori law, their rate decaying as a power of the time since the main shock, and market volatility after a crash relaxes the same way, with elevated turbulence decaying as a power law rather than snapping back to calm. Some researchers go further, proposing that crashes have a precursor signature, a log-periodic oscillation superimposed on a power-law acceleration as the system approaches a critical point, the bubble building toward a most-probable crash time. Whether or not that specific signature is reliably tradable, the framing matters: a crash is a critical phenomenon, a phase transition in a complex system, not a random bad draw, which is a completely different thing to prepare for than a Gaussian tail event. ## What complexity means for a trader The complex-systems view is a stance, not a strategy, and the stance has concrete consequences for how you trade. It tells you to expect the fat tails and the volatility clustering as permanent structural features rather than anomalies, which reinforces every survival argument in this pillar: the catastrophes are built into the system's nature, so you size for them always. It tells you that the structure you exploit, the trending, the reversion, emerges from agent interactions that can shift when the agents or their incentives change, which is why edges decay and systems die, the subject of ["When to Switch Off a Trading System"](https://aligrithm.com/when-to-switch-off-a-trading-system/). And it tells you that prediction has hard limits, because emergence means the aggregate is not deducible from the parts, so the honest goal is not to forecast the complex system but to find the statistical regularities it throws off and harvest them while they last. The danger of the complex-systems framing is the opposite of the Gaussian one, and worth naming. Where Gaussian thinking lulls you into ignoring the tail, complexity thinking can seduce you into over-claiming, dressing ordinary curve-fitting in the language of critical phenomena and log-periodic precursors to sell a crash-prediction model that does not work out of sample. The same skepticism applies here as everywhere in this pillar: the complex-systems lens is valuable for understanding why markets behave as they do and why the tails are permanent, and it is not a license to believe you can time the next crash because you fit a power law to the last one. Use it to respect the system's nature, not to claim mastery over it. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Visualizing markets as complex systems ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_177-visual-0.png) ## KEY POINTS - A market is a complex system: many interacting agents whose collective behavior produces patterns no individual intended and no simple equation captures. This is a specific, testable claim, not hand-waving. - The defining feature is emergence. A crash emerges from millions of locally reasonable decisions interacting through feedback, so you can understand every participant and still not predict the aggregate, because the behavior lives in the interactions. - Emergence is why clean models break on markets. The fat tails are emergent, coming from how agents' decisions correlate and cascade, not from any single agent's return distribution. - Power laws are the fingerprint of complexity. Price moves, trade sizes, and drawdowns all follow them, so there is no typical event size and the rare crash belongs to the same distribution as the common fluctuation. - A crash behaves like a critical phenomenon, a phase transition. Volatility after a crash relaxes as a power law like earthquake aftershocks, which is a different thing to prepare for than a Gaussian tail event. - The view is a stance, not a strategy: expect fat tails and clustering as permanent, expect edges to decay as agents shift, and accept hard limits on prediction. The danger is over-claiming, dressing curve-fitting as crash prediction, so respect the system rather than claiming to time it. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Systematic Trading - Robert Carver (Amazon)](https://www.amazon.com/s?k=Systematic+Trading+Robert+Carver&ref=aligrithm.com) - [Trading Systems - Urban Jaekle Emilio Tomasini (Amazon)](https://www.amazon.com/s?k=Trading+Systems+Urban+Jaekle+Emilio+Tomasini&ref=aligrithm.com) ### 6.36 Anomalous Diffusion in Financial Markets URL: https://aligrithm.com/anomalous-diffusion-in-financial-markets/ Last updated: 2026-07-27T13:25:48.000Z A drop of ink in still water spreads at a predictable rate, the cloud's width growing as the square root of time. That is normal diffusion, the physics of a random walk, and if prices were a true random walk they would diffuse the same way, with their spread growing as the square root of time and their variance growing linearly. They do not, quite. Markets exhibit anomalous diffusion, where the spread grows faster or slower than the square-root law, and that anomaly is the same trending-or-reverting structure the variance ratio measures, seen through the lens of physics. ["Variance Ratio Tests for Traders"](https://aligrithm.com/variance-ratio-tests-for-traders/) gave you the statistical version; this is the physical picture, which makes the structure intuitive and connects it to the fat tails and long-range correlations of the surrounding articles. ## Normal diffusion and its signature Watch any quantity driven by independent random kicks, a pollen grain in water, a drunkard's walk, a true random-walk price, and it spreads according to one law: the typical distance from the start grows as the square root of elapsed time. Double the time and the spread grows by the square root of two, not by two, because the random kicks partly cancel. This is the physical meaning of the random walk's linear variance growth, since variance is the square of the spread, and it is the benchmark against which markets are measured. $$ \\langle x^2(t) \\rangle \\sim t^{\\alpha} $$ The general law for how a system spreads is that the mean-squared displacement grows as time raised to a power, call it alpha. Normal diffusion, the random walk, has alpha equal to 1: variance grows linearly with time. Anomalous diffusion is any alpha other than 1\. When alpha is above 1 the system spreads faster than a random walk, called superdiffusion, and when alpha is below 1 it spreads slower, called subdiffusion. The single exponent alpha captures the entire departure from randomness, and measuring it on a price series tells you which regime the market is in. ## Superdiffusion is trending, subdiffusion is reverting The two anomalous regimes map directly onto the trading concepts from earlier in this pillar. Superdiffusion, spreading faster than the square-root law with alpha above 1, happens when successive price changes are positively correlated, when moves reinforce each other so the price travels farther than independent kicks would carry it. That is trending, the physical face of momentum, and it is the same thing the variance ratio reports as a value above 1\. The ink, in this analogy, is being carried by a current, not just diffusing, so it spreads downstream faster than random mixing alone. Subdiffusion, spreading slower than the square-root law with alpha below 1, happens when successive changes are negatively correlated, when moves partly reverse so the price stays closer to where it started than independent kicks would leave it. That is mean reversion, the physical face of the kind of signal the RSI ranking system harvests, and the variance ratio reports it as a value below 1\. The ink is being pulled back toward its origin, held near the center by a restoring force, so it spreads less than random. The diffusion exponent and the variance ratio are two names for the same measurement, one from physics and one from statistics, and they agree because they describe the same underlying correlation in the increments. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Why the physical picture is worth having The diffusion framing buys intuition and connection, not a separate strategy. Its value is that it places the trending-and-reverting structure inside a larger physics of markets that explains the other phenomena in this run. Anomalous diffusion arises from correlations in the price increments, and those same long-range correlations are the constraints that, through the maximum-entropy argument of ["Entropy as a Market Concept"](https://aligrithm.com/entropy-as-a-market-concept/), produce the fat tails of ["Fat Tails: Why Gaussian Thinking Breaks Trading Systems"](https://aligrithm.com/fat-tails-why-gaussian-thinking-breaks-trading-systems/). The trending, the fat tails, and the failure of the Gaussian are not separate facts; they are different consequences of the same thing, that price increments are correlated rather than independent, and the diffusion exponent is the cleanest single number that captures the departure from independence. The caveats carry over unchanged. The diffusion exponent is estimated from a finite sample and is non-stationary, so a market that superdiffuses over one period can revert to normal or subdiffuse over the next, and an exponent measured in-sample is no promise about the future, the same fragility that undermines every signal in this pillar. The exponent is also distorted by the fat tails, because a few extreme moves dominate the mean-squared displacement and can swing the estimate, exactly as they distort the variance ratio. So use the diffusion picture to understand why markets trend and revert and why those facts connect to fat tails, measure the exponent across horizons with appropriate skepticism, and treat it as a description of where structure has lived, not a guarantee of where it will live. The physics makes the structure intuitive; it does not make it permanent. ## Visualizing anomalous diffusion ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_176-visual-0.png) ## KEY POINTS - Normal diffusion, the physics of a random walk, spreads as the square root of time, so variance grows linearly. If prices were a true random walk they would diffuse this way. - Markets show anomalous diffusion: mean-squared displacement grows as time to the power alpha. Normal diffusion is alpha equals 1, and any other value is anomalous. - Superdiffusion (alpha above 1) spreads faster than random because moves reinforce. That is trending, the physical face of momentum, and the same thing the variance ratio reports above 1. - Subdiffusion (alpha below 1) spreads slower because moves partly reverse. That is mean reversion, and the variance ratio reports it below 1\. The diffusion exponent and the variance ratio are two names for the same measurement. - The physical picture connects the phenomena: anomalous diffusion comes from correlated increments, and those same long-range correlations produce the fat tails through the maximum-entropy argument. Trending, fat tails, and Gaussian failure share one cause. - The exponent is estimated from a finite, non-stationary sample and distorted by fat tails, so it describes where structure has lived, not where it will. The physics makes the structure intuitive, not permanent. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Systematic Trading - Robert Carver (Amazon)](https://www.amazon.com/s?k=Systematic+Trading+Robert+Carver&ref=aligrithm.com) - [Trading Systems - Urban Jaekle Emilio Tomasini (Amazon)](https://www.amazon.com/s?k=Trading+Systems+Urban+Jaekle+Emilio+Tomasini&ref=aligrithm.com) ### 6.35 Entropy as a Market Concept URL: https://aligrithm.com/entropy-as-a-market-concept/ Last updated: 2026-07-27T13:25:47.000Z Entropy is a measure of disorder, of how much you do not know about a system, and that makes it a natural fit for markets, where the central problem is uncertainty. Borrowed from physics and information theory, entropy gives you a way to quantify how unpredictable a return series is, how much information a price move carries, and why the fat-tailed distributions from the last two articles show up at all. It is one of the more abstract tools in this pillar, and it pays off concretely: maximizing the right kind of entropy reproduces the heavy tails that the Gaussian misses, which tells you the tails are not an anomaly but the most natural state of a system under the constraints markets impose. ["Lévy Distributions and Market Extremes"](https://aligrithm.com/levy-distributions-and-market-extremes/) left us needing a reason the tails are fat; entropy supplies it. ## Entropy measures what you cannot predict In information theory, entropy measures the average surprise of an outcome. A coin that always lands heads has zero entropy, because every flip is certain and carries no information. A fair coin has maximum entropy for two outcomes, because each flip is maximally surprising. Applied to returns, entropy quantifies how much uncertainty remains about the next move given what you know: a perfectly predictable series has low entropy, and a series where the next return is anyone's guess has high entropy. This reframes the efficient-market question from this pillar in physical terms, an efficient market is a high-entropy one where prices have squeezed out the predictable structure, and an edge is a pocket of lower-than-expected entropy, a place where the next move is more predictable than the market average. The appeal of entropy over the autocorrelation-based tools like the variance ratio is that it captures nonlinear dependence. The variance ratio sees linear autocorrelation; a series can have zero linear autocorrelation and still be predictable through a nonlinear relationship, and entropy-based measures can detect that structure where a variance ratio reports randomness. So entropy is a more general detector of predictability, catching dependencies that the linear tools miss, at the cost of being harder to estimate and interpret. ## Why maximum entropy gives fat tails The deeper use of entropy explains where the heavy tails come from. The maximum-entropy principle says that, given some constraints on a system, the least-biased distribution to assume is the one with the highest entropy consistent with those constraints, because it assumes nothing beyond what the constraints force. Under the constraint of a fixed variance, the maximum-entropy distribution is the Gaussian, which is why the bell curve is everywhere. But markets are not constrained by a simple fixed variance; they have long-range correlations and the anomalous behavior of ["Anomalous Diffusion in Financial Markets"](https://aligrithm.com/anomalous-diffusion-in-financial-markets/), and under those richer constraints the maximum-entropy distribution changes. $$ S\_q = \\frac{1 - \\sum\_i p\_i^{\\,q}}{q - 1} $$ The Tsallis entropy generalizes the ordinary entropy with a parameter q that controls how much weight the measure puts on rare events. When q is 1 it reduces to the standard entropy, whose maximization under fixed variance gives the Gaussian. When q is not 1, maximizing the entropy produces distributions with power-law tails, the fat-tailed shapes that actually fit market returns. The parameter q tunes the tail heaviness, and the value that fits the data tells you how far the market sits from the simple Gaussian regime. The fat tails, in this light, are not a deviation from a natural order; they are the natural maximum-entropy state of a system with the kind of long-range structure markets have, which is why every thin-tailed model keeps failing against them. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## What entropy buys you, and what it does not The practical value of entropy is as a measurement of predictability and a justification for taking the tails seriously, not as a strategy in itself. As a measurement, an entropy estimate on a return series tells you whether there is exploitable structure, including the nonlinear structure the variance ratio cannot see, and a series whose entropy is meaningfully below the random benchmark is one worth investigating for an edge. As a justification, the maximum-entropy result explains why fat tails are unavoidable and stable, which reinforces every survival argument in this pillar: the tails are baked into the system, not a passing feature you can wait out. The honest limits are real. Entropy is hard to estimate from finite, noisy financial data, because it requires estimating a whole distribution, and the estimate is sensitive to how you bin the data and how much data you have, so an entropy edge measured in-sample is as prone to overfitting as any other, maybe more, given the estimation difficulty. Low entropy tells you structure exists; it does not tell you the structure is tradable after costs or that it will persist, the same caveats that apply to every signal here. And the Tsallis machinery, while it fits the data, is a description, not a mechanism, it tells you the distribution that emerges, not why the market produces those constraints. Use entropy to detect nonlinear predictability and to understand why the tails are permanent, and keep the same skepticism you apply to every other measurement: a structure found is not yet an edge earned. ## Visualizing entropy ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_175-visual-0.png) ## KEY POINTS - Entropy measures disorder, the average surprise of an outcome, so it quantifies how unpredictable a return series is. A predictable series has low entropy; one where the next move is anyone's guess has high entropy. - In entropy terms, an efficient market is high-entropy with the predictable structure squeezed out, and an edge is a pocket of lower-than-expected entropy where the next move is more predictable than average. - Entropy captures nonlinear dependence that the variance ratio misses. A series with zero linear autocorrelation can still be predictable, and entropy-based measures detect it, at the cost of being harder to estimate. - The maximum-entropy principle says to assume the highest-entropy distribution consistent with the constraints. Under fixed variance that is the Gaussian; under the long-range correlations markets have, it changes. - Tsallis entropy generalizes the standard measure with a parameter q that weights rare events. Maximizing it for q not equal to 1 produces the power-law fat tails that fit real returns, so the tails are the natural max-entropy state, not a deviation. - Entropy is a measurement and a justification, not a strategy. It is hard to estimate from noisy data and prone to overfitting, and low entropy means structure exists, not that it is tradable after costs or will persist. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Systematic Trading - Robert Carver (Amazon)](https://www.amazon.com/s?k=Systematic+Trading+Robert+Carver&ref=aligrithm.com) - [Trading Systems - Urban Jaekle Emilio Tomasini (Amazon)](https://www.amazon.com/s?k=Trading+Systems+Urban+Jaekle+Emilio+Tomasini&ref=aligrithm.com) ### 6.34 Lévy Distributions and Market Extremes URL: https://aligrithm.com/levy-distributions-and-market-extremes/ Last updated: 2026-07-27T13:25:47.000Z If the Gaussian is too thin-tailed to describe returns, the natural next move is to find a distribution with fatter tails that fits better, and the Lévy stable distribution is the classic candidate. It has genuine power-law tails, it captures the peaked center and heavy shoulders of real return distributions that the bell curve misses, and for a long stretch of econophysics it looked like the answer. It is a better description, and it still fails at the exact place you most need it to work, the crashes, which makes it a useful lesson in why no clean distribution rescues you from the tail. ["Fat Tails: Why Gaussian Thinking Breaks Trading Systems"](https://aligrithm.com/fat-tails-why-gaussian-thinking-breaks-trading-systems/) said returns are fat-tailed; this is what happens when you try to pin down the fat tail with a specific model. ## A better fit for the body and shoulders The Lévy stable family generalizes the normal distribution by adding a parameter that controls tail thickness. The Gaussian is a special case with the thinnest tails; dial the stability parameter down and the tails fatten into power laws, so the distribution starts assigning real probability to the large moves the Gaussian forbids. Fit a Lévy distribution to daily returns and it tracks them far better than a normal curve: it reproduces the sharp peak at the center, where calm days cluster more tightly than a bell curve, and the heavy shoulders, where moderate-to-large moves happen more than normality allows. $$ P(|r| > x) \\sim x^{-\\alpha}, \\quad 0 < \\alpha < 2 $$ The Lévy tail decays as a power law with a stability index between 0 and 2\. A smaller index means fatter tails and more extreme events; at the value 2 you recover the Gaussian. The power-law form is the right shape for the bulk of the return distribution, which is why Lévy and related stable laws describe the central part of market price variation quite well. For the everyday and even the moderately unusual move, the Lévy distribution is a real improvement, and if all you needed was a better model of normal-to-rough days it would be enough. ## Where it breaks: the crashes The trouble is that the rarest, most consequential events do not obey the same law as the body of the distribution. The Lévy distribution that fits the center and shoulders does not match the real crashes; the genuine extremes, the 1987s and 2008s, deviate even from the fat-tailed Lévy fit, sitting in their own regime that the single fitted distribution does not capture. The market behaves as if the everyday fluctuations and the catastrophic ones are generated by different processes, so a distribution calibrated to the common moves systematically misprices the rarest ones, and the rarest ones are the ones that determine whether you survive. This is a deeper problem than picking the wrong curve. The Lévy stable distribution also has theoretical baggage that makes it awkward for trading: for the tail-fattening values of the stability index, the variance is infinite, which breaks every tool that relies on a finite variance, including the volatility scaling from ["Why Volatility-Adjusted Position Sizing Matters"](https://aligrithm.com/why-volatility-adjusted-position-sizing-matters/) and the variance ratio from ["Variance Ratio Tests for Traders"](https://aligrithm.com/variance-ratio-tests-for-traders/). A model whose variance does not exist is hard to use for position sizing, even if it describes the data well, so you are caught between a Gaussian that is too thin and a Lévy that is too pathological, with the real distribution sitting uncomfortably between them and refusing to be either. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## The lesson: no curve saves you from the tail The practical takeaway is not to adopt the Lévy distribution and move on, but to absorb why even the better distribution fails. The tail of market returns is not a clean mathematical object that the right closed-form distribution captures; it is a mixture, the product of normal fluctuations most of the time and rare regime-driven catastrophes that follow their own logic, and no single elegant distribution describes both. Every attempt to nail the tail with one curve, Gaussian, Lévy, or anything else, succeeds on the part of the distribution it was fit to and fails on the part it was not, and the part it fails on is the part that kills you. So the response from the fat-tails article still holds: do not trust any model in the deep tail, size for moves beyond what any fitted distribution predicts, and treat the crashes as their own category, studied directly through stress tests and the complex-systems lens of ["Why Financial Markets Are Complex Systems"](https://aligrithm.com/why-financial-markets-are-complex-systems/), not read off a probability curve. The Lévy distribution is the best illustration of the trap, because it is good enough to tempt you into trusting it and wrong enough, exactly at the extremes, to ruin you if you do. ## Visualizing the Lévy fit and its failure ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_174-visual-0.png) ## KEY POINTS - The Lévy stable distribution generalizes the Gaussian with a parameter controlling tail thickness, producing genuine power-law tails. It fits the sharp peak and heavy shoulders of real returns far better than a bell curve. - Its tail decays as a power law with a stability index between 0 and 2; smaller means fatter tails, and at 2 you recover the Gaussian. The power-law form is right for the bulk of the return distribution. - It fails at the crashes. The genuine extremes deviate even from the fat-tailed Lévy fit, behaving as if everyday and catastrophic moves come from different processes, so a distribution fit to common moves misprices the rarest ones. - The Lévy distribution also has infinite variance for its fat-tail values, which breaks every tool relying on finite variance, including volatility scaling and the variance ratio, making it awkward for position sizing. - The deeper lesson is that the tail is a mixture of normal fluctuations and rare regime-driven catastrophes, and no single elegant distribution describes both. Every curve succeeds where it was fit and fails on the extremes that kill you. - The response holds: distrust any model in the deep tail, size for moves beyond any fitted distribution, and treat crashes as their own category studied through stress tests and complex-systems thinking, not read off a curve. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Systematic Trading - Robert Carver (Amazon)](https://www.amazon.com/s?k=Systematic+Trading+Robert+Carver&ref=aligrithm.com) - [Trading Systems - Urban Jaekle Emilio Tomasini (Amazon)](https://www.amazon.com/s?k=Trading+Systems+Urban+Jaekle+Emilio+Tomasini&ref=aligrithm.com) ### 6.33 Fat Tails: Why Gaussian Thinking Breaks Trading Systems URL: https://aligrithm.com/fat-tails-why-gaussian-thinking-breaks-trading-systems/ Last updated: 2026-07-27T13:25:46.000Z The normal distribution is the most useful wrong assumption in finance. It makes the math tractable, it underlies most risk models, and it is false about markets in the one way that matters: it radically underestimates how often extreme moves happen. Real returns have fat tails, the rare giant move occurs far more often than a bell curve allows, and a system built on Gaussian assumptions is calibrated for a world where its worst day cannot happen, right up until that day arrives. ["Variance Ratio Tests for Traders"](https://aligrithm.com/variance-ratio-tests-for-traders/) already noted that fat tails distort variance; this is the full problem and why it kills systems that ignore it. ## The bell curve says the crash is impossible The normal distribution has thin tails that decay fast, so the probability of a move many standard deviations from the mean drops to essentially zero. Under a Gaussian, a five-standard-deviation daily move should happen roughly once every several thousand years, and a ten-sigma move never in the history of the universe. Markets produce moves of that magnitude every few years. October 1987, the 2008 cascade, the 2010 flash crash, the 2020 March selloff, each was a move the Gaussian model assigns a probability so small it is indistinguishable from impossible, and they keep happening, which means the model is not slightly off, it is wrong about the part of the distribution that determines survival. $$ P(|r| > x) \\sim x^{-\\alpha} \\quad \\text{(fat tail, power law)} \\qquad \\text{vs.} \\qquad P(|r| > x) \\sim e^{-x^2/2} \\quad \\text{(Gaussian)} $$ The difference is in how the tail decays. A fat-tailed distribution's tail falls off as a power law, a negative power of the move size, which declines slowly, so even far out in the tail there is meaningful probability mass. The Gaussian tail falls off as the exponential of the negative squared move, which declines ferociously fast, crushing the probability of extremes to nothing. The gap between a slow power-law decay and a fast exponential one is the gap between a distribution that expects occasional catastrophes and one that forbids them, and that gap is exactly where trading systems die. Excess kurtosis, the fourth moment running far above the Gaussian value of 3, is the summary statistic that flags this, but the tail behavior is the thing that hurts you. ## Why Gaussian risk models fail when it counts A risk model built on the normal distribution does its job well in calm markets and fails precisely in the crises it exists to protect against. Value-at-risk computed under normality tells you the most you can lose on a normal day, and a normal day is not when you blow up. The model sizes your positions for a world where the worst plausible move is four or five sigma, and then a ten-sigma move arrives and the loss is many times what the model said was possible. The failure is structural, not a calibration error: the model assumed the tail was thin, the tail was fat, and no amount of tuning a thin-tailed model captures a fat tail. This connects directly to the survival arguments in this pillar. The position sizing from ["Why Volatility-Adjusted Position Sizing Matters"](https://aligrithm.com/why-volatility-adjusted-position-sizing-matters/) scales by volatility, a measure that fat tails make unstable, because a few extreme moves dominate the variance and a calm stretch understates the true risk. The drawdown analysis from ["Average Drawdown vs Extreme Drawdown"](https://aligrithm.com/average-drawdown-vs-extreme-drawdown/) warned that the worst case lives in the tail the permutation shuffle underweights, and fat tails are why: the real distribution of moves has more extreme events than any thin-tailed model or naive resampling reproduces. Loss control from ["Why Loss Control Is the Only Thing You Fully Control"](https://aligrithm.com/why-loss-control-is-the-only-thing-you-fully-control/) is necessary precisely because the tail can deliver a loss your Gaussian-calibrated stop never contemplated. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Trade as if the tail is real The practical response is not to find the perfect distribution, because no closed-form distribution captures the tail exactly, and the next article on Lévy distributions shows even the better-fitting ones miss the rarest events. The response is to stop trusting any model in the tail and to build for the move you cannot model. Size positions so that a move several times larger than your worst historical day does not end you, not just so that a normal day is comfortable. Stress-test against extreme scenarios directly, asking what a 20% overnight gap does to the book, rather than reading a probability off a curve that says the gap cannot happen. Treat volatility estimates as understated, because the calm periods that dominate the estimate are exactly when the tail is being underweighted. And accept that the tail is not a rare nuisance to be smoothed away but a permanent feature of markets that determines whether you survive long enough for your edge to matter. The trader who respects the fat tail sizes smaller and lasts; the one who trusts the bell curve sizes for the world the model describes and gets carried out in the world that exists. ## Visualizing fat tails ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_173-visual-0.png) ## KEY POINTS - The normal distribution is the most useful wrong assumption in finance. It is tractable and underlies most risk models, and it radically underestimates how often extreme moves happen. - Real returns have fat tails: the rare giant move occurs far more often than a bell curve allows. Five- and ten-sigma moves that the Gaussian calls essentially impossible happen every few years. - The difference is tail decay. Fat tails fall off as a slow power law, leaving real probability mass far out, while the Gaussian falls off exponentially, crushing extremes to nothing. Excess kurtosis flags it. - Gaussian risk models work in calm markets and fail in the crises they exist for. Value-at-risk sizes you for a normal day, and a normal day is not when you blow up. The failure is structural, not a calibration error. - Fat tails destabilize volatility estimates (a few extremes dominate variance), are why the worst-case drawdown lives in the tail resampling underweights, and are why loss control must cover moves no Gaussian stop contemplated. - The response is not a perfect distribution but distrusting any model in the tail: size for a move several times your worst day, stress-test extremes directly, treat volatility as understated, and respect the tail as a permanent feature that decides survival. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Systematic Trading - Robert Carver (Amazon)](https://www.amazon.com/s?k=Systematic+Trading+Robert+Carver&ref=aligrithm.com) - [Trading Systems - Urban Jaekle Emilio Tomasini (Amazon)](https://www.amazon.com/s?k=Trading+Systems+Urban+Jaekle+Emilio+Tomasini&ref=aligrithm.com) ### 6.32 Variance Ratio Tests for Traders URL: https://aligrithm.com/variance-ratio-tests-for-traders/ Last updated: 2026-07-27T13:25:46.000Z If you want to know whether a price series trends, mean-reverts, or wanders randomly, you do not need a fancy model. You need one number, the variance ratio, and it comes from a property so simple it is almost embarrassing: for a random walk, variance grows linearly with time. Measure how the variance actually grows in your series, compare it to the linear benchmark, and the comparison tells you whether the increments are trending, reverting, or independent. ["Random Walk and Efficient Markets Are Not the Same Thing"](https://aligrithm.com/random-walk-and-efficient-markets-are-not-the-same-thing/) said the random walk is the testable claim; the variance ratio is how you test it. ## The linear-variance property The whole test rests on one fact. If price changes are independent, the variance of a return measured over a long interval equals the variance over a short interval times the number of short intervals in it. Returns over a month should have four times the variance of returns over a week, because a month is roughly four weeks and the variances of independent increments add. This linear scaling of variance with the sampling interval is the signature of a random walk, and any departure from it is a departure from independence. $$ \\text{VR}(q) = \\frac{\\text{Var}(r\_t^{(q)})}{q \\cdot \\text{Var}(r\_t^{(1)})} $$ The variance ratio compares the actual scaling to the linear benchmark. The numerator is the variance of returns measured over q periods, and the denominator is what that variance should be if the series were a random walk, the single-period variance scaled up linearly by q. Divide the two. If the series is a true random walk, the ratio is 1: the long-horizon variance is exactly q times the short-horizon variance, and you have confirmed independence. The departures from 1 are the interesting part. ## Reading the ratio A variance ratio above 1 means the long-horizon variance is larger than the random-walk benchmark, which happens when the increments are positively autocorrelated, when a move in one direction tends to be followed by another move in the same direction. The variance compounds faster than linearly because the moves reinforce each other. That is trending, the statistical fingerprint of momentum, and a series with a variance ratio persistently above 1 is one where trend-following has raw material to work with. A variance ratio below 1 means the long-horizon variance is smaller than the benchmark, which happens when increments are negatively autocorrelated, when a move in one direction tends to be partly reversed. The reversals cancel some of the variance, so it grows slower than linearly. That is mean reversion, the fingerprint of the kind of signal ["How to Build a Cost-Aware RSI Ranking System"](https://aligrithm.com/how-to-build-a-cost-aware-rsi-ranking-system/) tries to harvest. A ratio near 1 says the series is close to a random walk at that horizon and neither trend nor reversion has much to grip. The horizon matters: a series can mean-revert at short horizons (ratio below 1 for small q) and trend at longer ones (ratio above 1 for large q), so you compute the ratio across a range of horizons and read the term structure, not a single number. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## What the test does not tell you The variance ratio is a measurement, not a strategy, and treating it as one is the trap. A ratio above 1 says the series trended over the sample you measured; it does not promise the trending will continue, because the autocorrelation that produced it can be non-stationary and vanish out of sample, the same fragility that haunts every signal in this pillar. The test also has real statistical limits: variance ratios are noisy on short samples, so a ratio of 1.1 might be indistinguishable from 1 once you account for the sampling error, and you need the confidence bands, not just the point estimate, before claiming you found structure. Fat tails, the subject of ["Fat Tails: Why Gaussian Thinking Breaks Trading Systems"](https://aligrithm.com/fat-tails-why-gaussian-thinking-breaks-trading-systems/), also distort the variance estimate, because variance is dominated by the extreme moves and a few outliers can swing the ratio. So use the variance ratio to decide whether a series is worth building a trend or reversion strategy on, compute it across horizons with proper error bands, and remember it describes the past sample, not a guarantee about the future. It points you at where structure lived; whether the structure survives costs and persists is the work that follows. ## Visualizing the variance ratio ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_172-visual-0.png) ## KEY POINTS - The variance ratio tells you whether a series trends, mean-reverts, or wanders, from one property: for a random walk, variance grows linearly with time, so monthly variance should be about four times weekly variance. - The variance ratio is the actual q-period variance divided by q times the one-period variance. For a true random walk it equals 1, confirming independence. The departures from 1 are the signal. - Above 1 means positive autocorrelation: moves reinforce, variance compounds faster than linearly. That is trending, the fingerprint of momentum. - Below 1 means negative autocorrelation: moves partly reverse, variance grows slower than linearly. That is mean reversion. - Compute the ratio across a range of horizons and read the term structure, because a series can mean-revert at short horizons and trend at long ones. - The variance ratio measures the past sample, not a guarantee. It is noisy on short samples so you need confidence bands, fat tails distort the variance estimate, and the autocorrelation can vanish out of sample. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Systematic Trading - Robert Carver (Amazon)](https://www.amazon.com/s?k=Systematic+Trading+Robert+Carver&ref=aligrithm.com) - [Trading Systems - Urban Jaekle Emilio Tomasini (Amazon)](https://www.amazon.com/s?k=Trading+Systems+Urban+Jaekle+Emilio+Tomasini&ref=aligrithm.com) ### 6.31 Random Walk and Efficient Markets Are Not the Same Thing URL: https://aligrithm.com/random-walk-and-efficient-markets-are-not-the-same-thing/ Last updated: 2026-07-27T13:25:45.000Z Traders use "random walk" and "efficient market" as synonyms, and they are not. The two ideas get welded together in textbooks into a single discouraging slogan, the market is efficient so prices are a random walk so you cannot win, and the slogan is wrong on the logic. A random walk is a statistical statement about the sequence of price changes. Market efficiency is an economic statement about information. Neither implies the other, and untangling them is the difference between believing no edge can exist and knowing where edges actually hide. This opens the final run in this pillar, the one that treats price as a physical process to be measured rather than a chart to be read. ## Two different claims about prices The random walk hypothesis says that successive price changes are statistically independent, that tomorrow's return has no predictable relationship to today's, so the increments are uncorrelated and the variance of returns grows linearly with time. It is a claim you can test directly on the price series, with no reference to information, value, or rationality. You take the returns and you measure whether they are independent. That is all the random walk is: a property of the numbers. The efficient market hypothesis says something else entirely, that prices fully reflect available information, so you cannot earn excess risk-adjusted returns using that information because it is already in the price. This is a claim about economics and information, not about the statistical pattern of increments. An efficient market can have predictable price changes, as long as the predictability corresponds to risk premia or costs that prevent anyone from profiting, and an inefficient market can look statistically random if the inefficiencies are complex enough to evade simple tests. The two hypotheses live in different domains, and the point has been made sharply: random walk and efficient markets are not equivalent statements, one does not imply the other and vice versa. ## Why the distinction matters for finding edges The practical payoff is that you can reject the random walk without claiming the market is inefficient, and that rejection is where systematic edges come from. If you test a price series and find that the increments are not independent, that there is autocorrelation, that variance does not grow linearly with time, you have falsified the random walk for that series. You have not necessarily found free money, because the predictability might be a risk premium you are being paid to bear, but you have found a statistical structure, and statistical structure is the raw material of every quantitative strategy. The momentum and mean-reversion signals throughout this pillar all live on departures from the random walk. Conflating the two hypotheses talks you out of looking. A trader who believes random walk and efficiency are the same thing, and believes the market is efficient, concludes that prices must be a random walk and stops testing, when the testable claim, the random walk, is frequently false in ways the economic claim, efficiency, does not forbid. The honest position is to test the statistical claim directly, because it is testable, and to stay agnostic about the economic claim, because it is nearly untestable, you can never observe the full information set or the true risk premia. Test what you can measure, the independence of increments, and let the economics argue with itself. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## The structure is real but not necessarily free The caveat that keeps this honest: finding a departure from the random walk is necessary for an edge and nowhere near sufficient. The autocorrelation you measure might be entirely consumed by trading costs, so the structure exists statistically and cannot be harvested, the exact trap from ["Cost-Aware Ranking: The Missing Step in Cross-Sectional Strategies"](https://aligrithm.com/cost-aware-ranking-the-missing-step-in-cross-sectional-strategies/). It might be a risk premium, real predictability that pays you precisely because it exposes you to a risk that occasionally bites hard, in which case the market is efficient and the random walk is still false and you still might want the trade for the premium. Or it might be a genuine inefficiency that real money can extract, which is the rarest and most competed-for case. Rejecting the random walk tells you the structure is there; it does not tell you which of these three it is, and sorting that out is the work. The point is only that the work is worth starting, which the efficiency slogan tells you it is not. ## Visualizing the two hypotheses ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_171-visual-0.png) ## KEY POINTS - "Random walk" and "efficient market" are not synonyms. A random walk is a statistical claim about price changes; efficiency is an economic claim about information. Neither implies the other. - The random walk says successive price changes are independent and variance grows linearly with time. It is testable directly on the price series, with no reference to information. - The efficient market hypothesis says prices reflect available information so you cannot earn excess risk-adjusted returns. An efficient market can still have predictable changes if they correspond to risk premia or costs. - You can reject the random walk without claiming the market is inefficient, and that rejection, finding autocorrelation or nonlinear variance growth, is where systematic edges come from. - Conflating the two talks you out of looking. Believing they are the same and believing in efficiency makes you conclude prices must be random and stop testing the claim that is frequently false. - A departure from the random walk is necessary but not sufficient for an edge. The structure might be eaten by costs, paid as a risk premium, or a true inefficiency, and sorting which is the work worth starting. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Systematic Trading - Robert Carver (Amazon)](https://www.amazon.com/s?k=Systematic+Trading+Robert+Carver&ref=aligrithm.com) - [Trading Systems - Urban Jaekle Emilio Tomasini (Amazon)](https://www.amazon.com/s?k=Trading+Systems+Urban+Jaekle+Emilio+Tomasini&ref=aligrithm.com) ### 6.30 How to Remove Yourself from Your Trading System URL: https://aligrithm.com/how-to-remove-yourself-from-your-trading-system/ Last updated: 2026-07-27T13:25:45.000Z Every article in this behavioral run reaches the same conclusion: your in-the-moment judgment is the problem, and the fix is to take it out of the loop. This is the how. Removing yourself from your trading system is a concrete engineering task, not a matter of resolve, and it is done by closing off the specific points where your corrupted judgment can re-enter the process. The goal is a system that runs its decisions without consulting the version of you that feels loss aversion, near-miss revenge, and the illusion of control, because that version of you, as the whole pillar has shown, decides worse than the rule does. ["The Discipline Premium in Trading"](https://aligrithm.com/the-discipline-premium-in-trading/) said discipline is the edge; this is how you build the machine that enforces it. ## Find every point where you can intervene Start by mapping the decisions in your trading and marking which ones you currently make by hand in the moment, because each of those is a door the biases walk through. The entry timing, the position size, the stop placement, the exit, the re-entry after a stop, the decision to skip a signal, the decision to add to a position, the decision to turn the system off in a drawdown, every manual intervention point is a place where get-even-itis, the disposition effect, or revenge can corrupt the decision. You cannot remove yourself from a system whose intervention points you have not named, so the first step is the inventory: list every moment where the outcome depends on a choice you make while the trade is live. Most traders discover they have far more intervention points than they thought, and that the points cluster exactly where the biases are worst, the exits and the sizing, because those are the decisions that feel like they need judgment and are actually where judgment fails. The map itself is diagnostic: a system with many live-decision points is a system mostly run by your biases, no matter how systematic the entry signal looks. ## Close the doors in order of damage Remove yourself by converting intervention points into pre-committed rules, starting with the ones the behavioral articles flagged as most damaging. Sizing first, because the urge to oversize after a loss is revenge and to undersize after a win is fear, so a fixed volatility-based sizing rule from ["Why Volatility-Adjusted Position Sizing Matters"](https://aligrithm.com/why-volatility-adjusted-position-sizing-matters/) closes the door anger and fear came through. Exits next, because that is the home of get-even-itis and the disposition effect, so a mechanical stop and exit decided before entry removes the decision your brain is guaranteed to get wrong. Re-entry next, because that is where the near-miss revenge loop runs, so a rule requiring a fresh valid signal before re-entering closes it. The on-off decision last and most carefully, because quitting a system in a drawdown is the most expensive override of all, so the kill rule from ["When to Switch Off a Trading System"](https://aligrithm.com/when-to-switch-off-a-trading-system/), written in advance and tied to the drawdown envelope, replaces the panic decision with a statistical one. The strongest version of closing a door is automation: code the rule so the decision executes without you, because a rule you have to manually obey is a rule you can manually break, and you will break it at the worst moment. Automation is not about speed or sophistication; it is about removing your hand from the lever entirely, so that following the rule requires no willpower because there is no opportunity to deviate. Where full automation is not possible, the next best thing is making deviation costly and slow, a written rule you must consult, a cooling-off delay, a second person who has to approve an override, anything that puts friction between your in-the-moment impulse and the action. ## Engineer your contact with the system Removing yourself is also about reducing exposure to the temptation, not just blocking the action. The more you watch a live system, the more chances the urge to intervene gets, so check it on a schedule rather than continuously, the way ["Why Systematic Trading Feels Emotionally Unsatisfying"](https://aligrithm.com/why-systematic-trading-feels-emotionally-unsatisfying/) prescribed, denying the discomfort the constant contact it needs to drive an override. Separate your research time from your monitoring time, so the engagement you crave goes into building and testing in a calm state, where your judgment is actually useful, rather than into meddling with the running strategy, where it is not. And design the moments you do interact with the system to be decision-free by default, a dashboard you review, not a console you operate, so that the normal state of contact is observation, and intervention requires a deliberate, friction-laden departure from it. The deepest version of removing yourself is accepting that the disciplined version of you, the one who designed the rules in a calm moment, should overrule the live version of you every time they disagree, because the calm version had access to your skill without your biases. Removing yourself from your trading system is really transferring authority from the live you to the calm you, and building the structure that makes the transfer stick when the live you, deep in a drawdown or a near-miss rage, tries to take the authority back. That structure is the final product of this pillar: not a better signal, but a process that protects a sound signal from the one person most likely to wreck it. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Visualizing removing yourself ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_170-visual-0.png) ## KEY POINTS - Removing yourself from your trading system is an engineering task, not a matter of resolve. You close off the specific points where your corrupted in-the-moment judgment can re-enter the process. - First, inventory every intervention point, every decision you currently make by hand while the trade is live, because each is a door the biases walk through. Most traders find far more than they expected, clustered at the exits and sizing. - A system with many live-decision points is mostly run by your biases no matter how systematic the entry looks. The map itself is diagnostic. - Close the doors in order of damage: sizing first (anger oversizes, fear undersizes), exits next (get-even-itis and the disposition effect), re-entry next (near-miss revenge), and the on-off decision last and most carefully, with a pre-written kill rule. - Automation is the strongest close, because a rule you must manually obey is one you can manually break at the worst moment. Where automation is impossible, make deviation slow and costly with friction. - Reduce contact with the live system: check on a schedule, separate research from monitoring, and make interaction observation by default. The deepest version is transferring authority from the live you to the calm you who designed the rules, and making the transfer stick. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Systematic Trading - Robert Carver (Amazon)](https://www.amazon.com/s?k=Systematic+Trading+Robert+Carver&ref=aligrithm.com) - [Trading Systems - Urban Jaekle Emilio Tomasini (Amazon)](https://www.amazon.com/s?k=Trading+Systems+Urban+Jaekle+Emilio+Tomasini&ref=aligrithm.com) ### 6.3 Why Volatility-Adjusted Position Sizing Matters URL: https://aligrithm.com/why-volatility-adjusted-position-sizing-matters/ Last updated: 2026-07-27T13:25:31.000Z Put the same dollar amount into a Treasury future and a Bitcoin perp and you have not made two equal bets. You have made one tiny bet and one enormous one, because the Bitcoin position swings ten or twenty times harder per day than the bond. Equal dollars is the most common sizing mistake in retail trading, and it guarantees that your risk, your profit, and your pain all come from whichever instrument happens to be the most volatile, regardless of which one you actually had a view on. Volatility-adjusted sizing fixes this by sizing in units of risk instead of units of cash. This is the sizing layer that ["Ranking Beats Forecasting for Many Trading Problems"](https://aligrithm.com/ranking-beats-forecasting-for-many-trading-problems/) deliberately left out. Ranking picks what to hold; this decides how much. ## Equal risk, not equal cash The fix is to scale each position inversely to its volatility, so a more volatile instrument gets a smaller position and a calmer one gets a larger position, and both end up contributing the same risk to the book. You decide once how much risk you want each position to carry, expressed as a target volatility, and then you back out the position size that delivers it. $$ N\_i = \\frac{\\tau \\cdot \\text{Capital}}{\\sigma\_i \\cdot \\text{(price} \\times \\text{point value)}\_i} $$ The position size in contracts is the risk target (an annualized volatility you want the position to run, say 20%) times your capital, divided by the instrument's annualized price volatility times the cash value of one point of price movement. The numerator is how much risk in dollars you are willing to spend on this position; the denominator is how much risk one contract delivers. Divide and you get the number of contracts that hits your target. A high-volatility instrument has a large volatility, which shrinks the position; a low-volatility one has a small volatility, which inflates it. Every position, after the scaling, contributes the same expected swing. ## Why this is the foundation, not a refinement Without volatility scaling, every other part of your system is quoting into the dark. A ranked long/short book that sizes by equal cash is not actually market-neutral, because the high-volatility names dominate both the long and short legs and the supposed hedge is lopsided. A forecast that says "lean twice as hard on copper as on gold" means nothing in dollar terms until you express both in the same risk units, which is exactly why a disciplined framework standardizes every forecast to a common volatility scale before combining them. Volatility scaling is the common currency that lets you compare a bond signal to a crypto signal to an equity signal on one desk. Skip it and the comparisons are meaningless, no matter how good the underlying signals are. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## The catch: volatility is estimated, and it moves Volatility scaling rests on the volatility estimate, and you never know the true volatility; you estimate it from recent history, and the estimate is always a little stale and a little wrong. When volatility jumps, your estimate lags, so for a window you are running larger positions than your target intends, right when the market is most dangerous. When volatility collapses, you lever up into calm that may be the eye of a storm. The standard mitigations, an exponentially weighted estimate that reacts faster, a blend of recent and long-run volatility, a floor under the estimate so a quiet patch does not let you take grotesque size, all help and none of them make the estimate correct. You are sizing off a moving target, and the discipline is to respect that the number is an estimate, not to pretend a fancier estimator makes it exact. ## Visualizing volatility scaling ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_143-visual-0.png) ## KEY POINTS - Equal dollars is not equal bets. A high-volatility instrument swings many times harder than a calm one, so equal cash makes your risk and P&L come from whichever instrument is loudest, not the one you had a view on. - Volatility-adjusted sizing scales each position inversely to its volatility so every position contributes the same risk. You pick a volatility target once and back out the position size that delivers it. - The size in contracts is the risk target times capital, divided by the instrument's price volatility times the cash value of a one-point move. Numerator is risk you will spend; denominator is risk one contract delivers. - This is the foundation, not a refinement. Without it a ranked book is not really neutral, and forecasts that say "lean twice as hard here" are meaningless until expressed in common risk units. - Volatility scaling is the common currency that lets you compare a bond, an equity, and a crypto signal on one desk. Skip it and the comparisons are meaningless. - The catch: volatility is estimated and lags. When it jumps you are briefly oversized into danger. Faster estimators, blends, and floors help but never make the estimate exact, so respect that it is an estimate. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Systematic Trading - Robert Carver (Amazon)](https://www.amazon.com/s?k=Systematic+Trading+Robert+Carver&ref=aligrithm.com) - [Trading Systems - Urban Jaekle Emilio Tomasini (Amazon)](https://www.amazon.com/s?k=Trading+Systems+Urban+Jaekle+Emilio+Tomasini&ref=aligrithm.com) ### 6.29 The Discipline Premium in Trading URL: https://aligrithm.com/the-discipline-premium-in-trading/ Last updated: 2026-07-27T13:25:44.000Z Most trading edges decay as more people discover them. The edge from discipline does the opposite: it persists, because it comes from doing the boring, uncomfortable thing that most people are constitutionally unable to do, and human nature does not get arbitraged away. While everyone hunts for the next clever signal, the most durable edge available is the willingness to follow a mediocre rule consistently when everyone around you is breaking theirs. Call it the discipline premium, the return you collect simply for not making the behavioral mistakes the rest of the market makes. ["Why Simple Algorithms Beat Smart Humans"](https://aligrithm.com/why-simple-algorithms-beat-smart-humans/) mentioned this second effect; this is the full case for discipline as a tradable edge. ## Discipline is an edge because indiscipline is everywhere A trading edge needs an inefficiency to feed on, and the behavioral biases from this pillar are a vast, permanent inefficiency. Every discretionary trader holding losers too long, selling winners too early, revenge trading after a near miss, and panic-selling the bottom is leaving money on the table in a patterned, predictable way. Those patterns show up in prices, the over-extension from herding, the mean reversion from forced liquidations, the momentum from slow reaction, and a disciplined system can be built to harvest them. The other side of every behavioral mistake is a profit for whoever is positioned to take it, and the disciplined trader is that whoever. This edge is unusually durable because its source cannot be competed away. A clever signal stops working when enough people trade it, but the discipline premium feeds on human instinct, and human instinct does not learn. The biases are firmware, as ["The Flawed Human Brain in Trading"](https://aligrithm.com/the-flawed-human-brain-in-trading/) argued, so each new generation of traders arrives with the same loss aversion and the same disposition effect, makes the same mistakes, and leaves the same patterns. The supply of indiscipline is permanent, which makes the premium for harvesting it permanent too, in a way almost no other edge is. ## The premium is paid for discomfort, not cleverness The discipline premium is not collected by being smarter; it is collected by tolerating discomfort that others will not. Following a rule through a drawdown, when every instinct screams to override it, is uncomfortable, and the discomfort is exactly why the premium exists, because most people pay to make the discomfort stop. They override the rule to feel in control, they take the early profit to relieve the anxiety, they cut the system in a drawdown to escape the pain, and every one of those comfort-seeking actions forfeits a piece of the premium to whoever endured the discomfort instead. This reframes the emotional emptiness of systematic trading from ["Why Systematic Trading Feels Emotionally Unsatisfying"](https://aligrithm.com/why-systematic-trading-feels-emotionally-unsatisfying/) as the literal source of the return. You are not enduring the boredom and the helplessness for nothing; you are being paid for them, because they are the price of admission to an edge that requires doing the unpleasant thing consistently. The trader who finds systematic trading unbearable and quits is not just failing to capture an edge, he is becoming the counterparty, joining the pool of indiscipline that funds the premium for those who stayed. There is no neutral position: you either pay the discomfort and collect the premium, or you avoid the discomfort and pay the premium to someone else. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## You still need a real edge underneath The honest limit: discipline is a multiplier on an edge, not an edge by itself. Disciplined execution of a genuinely negative-expectancy system loses money with great consistency, and following a bad rule faithfully just means you lose in an orderly fashion. The discipline premium is the gap between a sound system's backtested return and the worse return an undisciplined trader actually realizes running it, the leakage from overrides, early exits, and abandonment. That gap is real and often large, frequently the difference between a profitable strategy on paper and a losing one in practice, but it is a gap relative to a system that had an edge to begin with. So the discipline premium is what you collect for executing a sound process cleanly, and it presupposes the process is sound, which is the work of every other article in this series. Build the edge first, then collect the discipline premium by not sabotaging it, and recognize that for most traders the second part is harder than the first. ## Visualizing the discipline premium ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_169-visual-0.png) ## KEY POINTS - The discipline premium is the return collected simply for not making the behavioral mistakes the rest of the market makes. Unlike a clever signal, it persists because it feeds on human nature. - Discipline is an edge because indiscipline is everywhere. Held losers, early-sold winners, revenge trades, and panic-selling leave patterned, predictable footprints in prices that a disciplined system can harvest. - The edge is durable because its source cannot be competed away. The biases are firmware, so each new generation arrives with the same loss aversion and makes the same mistakes, keeping the supply of indiscipline permanent. - The premium is paid for tolerating discomfort, not for being smarter. Most people override the rule to feel in control or quit in a drawdown to escape the pain, forfeiting the premium to whoever endured it instead. - The emotional emptiness of systematic trading is the literal source of the return. There is no neutral position: you pay the discomfort and collect the premium, or you avoid it and pay the premium to someone else. - Discipline is a multiplier on an edge, not an edge by itself. Faithfully following a negative-expectancy system loses money in an orderly way, so build a sound process first, then collect the premium by not sabotaging it. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Systematic Trading - Robert Carver (Amazon)](https://www.amazon.com/s?k=Systematic+Trading+Robert+Carver&ref=aligrithm.com) - [Trading Systems - Urban Jaekle Emilio Tomasini (Amazon)](https://www.amazon.com/s?k=Trading+Systems+Urban+Jaekle+Emilio+Tomasini&ref=aligrithm.com) ### 6.28 Why Discretionary Traders Need Rules Even If They Hate Systems URL: https://aligrithm.com/why-discretionary-traders-need-rules-even-if-they-hate-systems/ Last updated: 2026-07-27T13:25:44.000Z Plenty of skilled discretionary traders recoil at the word systematic. They believe their edge lives in judgment, in reading the market in ways no rule can capture, and handing the decision to an algorithm would throw that edge away. Some of them are right that they have judgment worth keeping. All of them are wrong that judgment means no rules, because the same biases that wreck a novice wreck an expert too, and a discretionary trader without rules is not expressing pure skill, he is expressing skill contaminated by get-even-itis, the disposition effect, and revenge trading. Rules do not replace the discretionary trader's judgment; they fence off the parts of the decision where judgment is known to fail. This article is for the trader who will never fully systematize and still needs the structure. ## Judgment and bias arrive together The discretionary trader's mistake is to treat his judgment as a single thing, all of it valuable, when it is actually a mix of genuine skill and the cognitive flaws from ["The Flawed Human Brain in Trading"](https://aligrithm.com/the-flawed-human-brain-in-trading/), and the two are tangled together in the moment. His read on which markets are setting up, where the risk is, what the flow is telling him, may be real and hard to replicate. His decision to hold a loser past his plan because he cannot stand crystallizing it is not skill, it is loss aversion, and it feels identical from the inside. Both are his judgment; only one is worth keeping. Without rules, he expresses both, and the bias half quietly drains the edge the skill half generates. This is why a discretionary trader can have genuine alpha and still lose money. The skill produces good entries and good reads; the unconstrained biases produce bad exits, oversized revenge trades, and held losers, and the second set eats the first. He blames variance or a bad market, when the actual problem is that his process let the biases into the decisions where they do the most damage, the exits and the sizing, while his real skill, the reads, went underexploited. ## Rules where bias is worst, freedom where skill is real The resolution is not all-or-nothing. The discretionary trader does not have to choose between pure discretion and a fully automated system; he can rule-bind the specific decisions where bias is known to dominate and keep discretion where his skill is genuine. The decisions to systematize are the ones the behavioral articles in this pillar flagged: position sizing, because the urge to oversize after a loss is revenge and the urge to undersize after a win is fear, neither of them skill; exits and stops, because that is where get-even-itis and the disposition effect operate; and re-entry, because that is where the near-miss compulsion drives revenge trades. Hand those to rules. Keep discretion for the part that might be real: which markets to look at, when a setup is forming, the qualitative read that no rule captures. This hybrid is a partial systematization, and it captures most of the benefit. The trader keeps the judgment that may be his edge and removes his hands from the levers where his judgment is reliably corrupted. A pre-set stop does not insult his market read; it protects the trade his read identified from the exit instinct that would mismanage it. A position-size rule does not override his conviction; it stops his conviction from being inflated by anger or deflated by fear. The rules constrain the failure modes and leave the skill alone, which is the most a trader who refuses full systematization can do, and it is a great deal better than nothing. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## The honest part: some of the judgment is not real either The hard truth the discretionary trader resists is that some of what he calls skill is also bias wearing a costume, and the rules help him find out which. A trader convinced his discretionary exits add value can test it: follow a mechanical exit rule on half his trades and his feel on the other half, and compare. Often the mechanical exits win, which is evidence that the discretion he was defending was the disposition effect all along. Rules are not just a constraint, they are a measurement, a way to separate the judgment that survives a fair test from the judgment that only felt like skill. The discretionary trader who is honest enough to run that test usually ends up rule-binding more of his process than he expected, because more of his judgment than he believed turns out to be bias, and the part that is genuinely skill is happy to be freed from the part that was not. ## Visualizing the hybrid ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_168-visual-0.png) ## KEY POINTS - A discretionary trader without rules is not expressing pure skill; he is expressing skill contaminated by get-even-itis, the disposition effect, and revenge trading, which feel identical to skill from the inside. - Judgment and bias arrive tangled together in the moment. A real read on which markets are setting up is valuable; holding a loser past plan because crystallizing hurts is loss aversion. Both are his judgment, only one is worth keeping. - This is why a discretionary trader can have genuine alpha and still lose: good reads produce good entries, unconstrained biases produce bad exits and oversized revenge trades, and the second set eats the first. - The fix is partial, not all-or-nothing. Rule-bind the decisions where bias dominates, position sizing, exits and stops, re-entry, and keep discretion for the qualitative reads where skill might be real. - A pre-set stop protects the trade the read identified from the exit instinct that would mismanage it; a size rule stops conviction from being inflated by anger or deflated by fear. Rules constrain failure modes and leave skill alone. - Rules also measure. Test mechanical exits against your feel on split samples; the mechanical version often wins, revealing that the defended discretion was the disposition effect, so honest traders end up rule-binding more than they expected. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Systematic Trading - Robert Carver (Amazon)](https://www.amazon.com/s?k=Systematic+Trading+Robert+Carver&ref=aligrithm.com) - [Trading Systems - Urban Jaekle Emilio Tomasini (Amazon)](https://www.amazon.com/s?k=Trading+Systems+Urban+Jaekle+Emilio+Tomasini&ref=aligrithm.com) ### 6.27 Near Misses and Revenge Trading URL: https://aligrithm.com/near-misses-and-revenge-trading/ Last updated: 2026-07-27T13:25:43.000Z A slot machine that showed only losses would empty the casino in an hour. The machines pay in near misses, two cherries and a blank, the jackpot symbols landing just above or just below the line, because an almost-win is more compelling than a clear loss and keeps the player feeding coins. Trading has the same mechanism, and it produces the same compulsive behavior: the trade that almost worked, the stop tagged a tick before the move, breeds the urge to immediately get back in and make it right. That urge is revenge trading, and it is the near-miss factor weaponized against your account. ["The Illusion of Control in Active Trading"](https://aligrithm.com/the-illusion-of-control-in-active-trading/) named near misses as fuel; this is what they ignite. ## A near miss is not a loss to the brain A clear loss and a near miss are the same outcome on your P&L and completely different events in your head. A trade that was wrong from the start and lost feels like a loss, and the brain files it as such. A trade that was right, that would have worked, that got stopped out a tick before the move you correctly predicted, does not feel like a loss; it feels like a theft, like a win that was taken from you by bad luck or a tick of slippage. The near miss carries the message you were right, which is far more motivating than the message you were wrong, because being-almost-right invites you to try again immediately and claim the win you were owed. This is why near misses drive activity that clear losses do not. A string of clean losses eventually discourages a trader into stopping. A string of near misses energizes him, each one whispering that the edge is real and the next attempt will land it, so he keeps trading, keeps re-entering, keeps feeding coins into the machine. The motivational asymmetry is the engine of compulsion, and markets, especially fast ones, manufacture near misses constantly, because the difference between a winning and losing scalp is often a single tick. ## Revenge trading is the near miss acting out Revenge trading is what the near-miss urge produces when you obey it. Stopped out just before the move, you re-enter immediately to capture the gain you feel was stolen, except now you are trading from emotion rather than signal: the size is bigger because you want to make the loss back faster, the timing is worse because you are reacting to the sting rather than to a setup, and the original edge, if there ever was one, is gone, replaced by the need to be made whole. The revenge trade is sized by anger and timed by impatience, the two worst inputs to a position, and it tends to lose, which produces another near miss or another loss, which produces another revenge trade. The loop tightens with each turn. The rapidity factor makes it worse, because fast markets serve the next opportunity for revenge before the emotion of the last one has cooled. A day trader stopped out at 10:03 can be back in by 10:04, while the sting is fresh and the judgment is hot, which is exactly the condition under which the revenge loop runs fastest. Slower trading has a built-in cooling period; the trade is over and the next setup is days away, so the emotion dissipates before it can drive the next decision. Speed removes the cooling and lets the near-miss compulsion compound, which is one more reason the fast styles that feel most engaging are the most dangerous. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Break the loop with structure, not willpower You do not beat revenge trading by resolving to stay calm, because the near-miss urge fires before your resolve gets a vote, and "just this once, to make it back" is the rationalization that always sounds reasonable in the heat of it. The defense is structural. Re-entry rules decided in advance, so that getting back in requires a fresh valid signal, not a feeling, removes the discretion the revenge trade needs. A cooling-off rule, no new position for a set period after a stop-out, manufactures the pause that fast markets deny you. Position-size limits cap how much damage a revenge trade can do even if you take one, since the revenge instinct's signature move is oversizing. And the deepest fix, from ["How to Remove Yourself from Your Trading System"](https://aligrithm.com/how-to-remove-yourself-from-your-trading-system/), is to automate the re-entry decision entirely, so the system, which feels no sting and seeks no revenge, decides when you get back in. The near miss is a feeling the machine does not have, which is exactly why the machine should be making the decision the feeling wants to corrupt. ## Visualizing the revenge loop ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_167-visual-0.png) ## KEY POINTS - Slot machines pay in near misses because an almost-win is more compelling than a clear loss. Trading has the same mechanism, and it breeds the urge to immediately get back in. - A near miss and a clear loss are identical on the P&L and opposite in the brain. The near miss carries the message you were right, which is far more motivating than you were wrong and invites you to try again at once. - Near misses energize trading where clear losses discourage it. Fast markets manufacture them constantly, because the difference between a winning and losing scalp is often one tick. - Revenge trading is the near-miss urge acting out: you re-enter from emotion, sized by anger and timed by impatience, the edge gone, replaced by the need to be made whole, and it tends to lose, feeding the loop. - The rapidity factor tightens the loop, serving the next revenge before the last one's emotion cools. Slower trading has a built-in cooling period that speed removes. - Beat it with structure, not willpower: re-entry rules requiring a fresh signal, a cooling-off period after a stop, position-size caps, and automating the re-entry so the machine, which feels no sting, decides. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Systematic Trading - Robert Carver (Amazon)](https://www.amazon.com/s?k=Systematic+Trading+Robert+Carver&ref=aligrithm.com) - [Trading Systems - Urban Jaekle Emilio Tomasini (Amazon)](https://www.amazon.com/s?k=Trading+Systems+Urban+Jaekle+Emilio+Tomasini&ref=aligrithm.com) ### 6.26 Why Systematic Trading Feels Emotionally Unsatisfying URL: https://aligrithm.com/why-systematic-trading-feels-emotionally-unsatisfying/ Last updated: 2026-07-27T13:25:43.000Z A working systematic strategy is boring to run. It makes its decisions without asking you, executes while you watch, and gives you nothing to do but follow it, and that nothing is unbearable to most people. The dirty secret of systematic trading is not that it is hard to build; it is that it is hard to tolerate, because it deprives you of every emotional reward that discretionary trading provided. The system's emotional emptiness is the price of its edge, and most traders are unwilling to pay it. ["The Illusion of Control in Active Trading"](https://aligrithm.com/the-illusion-of-control-in-active-trading/) explained the feeling that goes missing; this is about living without it and why that is the real barrier. ## The rewards you lose Discretionary trading paid you in feelings, and a good system stops the payments. You lose the feeling of control, the sense that you chose the moment and steered the outcome, replaced by a rule that chose for you. You lose the engagement, the constant involvement of watching, deciding, intervening, replaced by long stretches where the correct action is to do nothing. You lose the near-miss thrill and the fast-feedback rush that made active trading feel alive. You lose the story, the satisfying narrative of having read the market right, because the system has no narrative, only a rule that fired. What remains is a procedure you follow, and following a procedure delivers none of the dopamine that discretion did. The absence is felt as a specific set of discomforts. Boredom, because there is nothing to do. Anxiety, because doing nothing feels irresponsible when money is at stake, as though you should be managing the position. Disconnection, because the wins do not feel earned, the system made them, not you, and the losses feel worse, suffered passively rather than fought. A systematic trader in a drawdown cannot even soothe himself with action; the system says hold, and holding while losing, with no intervention permitted, is one of the most psychologically punishing positions in trading. ## The emptiness is the feature The discomfort is not a flaw in systematic trading to be fixed; it is the direct consequence of the thing that makes it work. The system removes your in-the-moment judgment precisely because that judgment, as the rest of this pillar has shown, is corrupted by biases the moment money and emotion are involved. The emotional rewards of discretionary trading, the control, the engagement, the thrill, are the exact channels through which those biases operate, so a system that delivered those rewards would have to re-admit the biases. The emptiness and the edge are the same thing viewed from two sides: the system is unsatisfying because it has closed the doors the biases came through. This reframes the discomfort. The boredom is the feeling of not overtrading. The anxiety of doing nothing is the feeling of not meddling with a position your rule says to hold. The disconnection from the wins is the feeling of not taking credit for luck, and from the losses, the feeling of not panic-selling the bottom. Every uncomfortable absence corresponds to a specific mistake you are not making. The feeling that you should be doing something is the bias asking to be let back in, and the discomfort of refusing it is the cost of keeping it out. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Surviving the emptiness Knowing the emptiness is a feature does not make it comfortable, and the practical problem remains: traders abandon working systems because they cannot stand the lack of engagement, then return to discretionary trading and lose. The defenses are structural, not emotional. Reduce your contact with the system, because watching it tick invites the urge to intervene; check it on a schedule, not continuously, so there is less opportunity for the discomfort to drive an override. Find the engagement you crave somewhere other than the live position, in research, in building the next system, in improving the process, channeling the need to be active into work that does not corrupt the running strategy. And accept, going in, that the satisfaction of systematic trading is deferred and abstract, the equity curve over years, not the thrill of the individual trade, which is a worse emotional deal in the moment and a better financial one over time. The traders who last are the ones who stopped needing the trade to feel good, covered next in ["Near Misses and Revenge Trading"](https://aligrithm.com/near-misses-and-revenge-trading/) and resolved in ["How to Remove Yourself from Your Trading System"](https://aligrithm.com/how-to-remove-yourself-from-your-trading-system/). ## Visualizing the emptiness ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_166-visual-0.png) ## KEY POINTS - A working systematic strategy is boring to run. It decides without you and leaves you nothing to do, and that nothing is the real barrier, not the difficulty of building it. - The system stops paying you in feelings. You lose the control, the engagement, the near-miss thrill, and the story of having read the market right, and following a procedure delivers none of that dopamine. - The absence shows up as boredom (nothing to do), anxiety (doing nothing feels irresponsible), and disconnection (wins feel unearned, losses feel suffered passively). Holding while losing with no intervention allowed is especially punishing. - The discomfort is not a flaw to fix; it is the consequence of the thing that works. The emotional rewards of discretion were the channels the biases used, so a system that restored them would re-admit the biases. - Every uncomfortable absence corresponds to a mistake you are not making: boredom is not overtrading, the urge to act is not meddling. The feeling that you should do something is the bias asking to be let back in. - Survive it structurally: check the system on a schedule instead of continuously, channel your need for engagement into research rather than the live position, and accept that the satisfaction is deferred and abstract. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Systematic Trading - Robert Carver (Amazon)](https://www.amazon.com/s?k=Systematic+Trading+Robert+Carver&ref=aligrithm.com) - [Trading Systems - Urban Jaekle Emilio Tomasini (Amazon)](https://www.amazon.com/s?k=Trading+Systems+Urban+Jaekle+Emilio+Tomasini&ref=aligrithm.com) ### 6.25 The Illusion of Control in Active Trading URL: https://aligrithm.com/the-illusion-of-control-in-active-trading/ Last updated: 2026-07-27T13:25:42.000Z Active trading feels like control. You watch the screen, you decide the moment, you click the button, and the position is yours, chosen, timed, owned. That feeling is mostly an illusion, because the part you control, the click, is trivial, and the part that determines the outcome, whether the market moves your way, you do not control at all. The illusion is pleasant and expensive: it makes people trade more than they should, because trading scratches a psychological itch that has nothing to do with making money. ["The Flawed Human Brain in Trading"](https://aligrithm.com/the-flawed-human-brain-in-trading/) listed this among the flaws; this is why it specifically sabotages active traders and why systematic trading feels so unsatisfying by comparison. ## The feeling of agency is not the same as agency Humans crave control, and the brain manufactures a sense of it from actions that change nothing about the odds. Roll dice harder for a high number, pick your own lottery numbers, press the elevator button twice, all give a feeling of influence over outcomes that are indifferent to your effort. Active trading is the same illusion with money attached. Choosing the exact entry, adjusting the position, timing the exit by feel, these acts deliver a vivid sense of agency, and that sense is largely disconnected from whether they improve your results. The market does not move because you chose your moment with conviction; it moves for reasons that have nothing to do with your sense of control. The trap is that the feeling is rewarding on its own, independent of the P&L. Placing a trade, watching it tick, intervening to manage it, all engage the brain's reward system the way any sense of control does, so trading becomes self-reinforcing regardless of whether it makes money. The trader is being paid in the feeling of control, a currency the brain values, while losing the actual currency to costs and bad timing. He thinks he trades to profit; he partly trades because the activity itself feels good, and the feeling does not care about the result. ## Near misses and speed pour fuel on it Two features make the illusion especially potent in active trading, and both are strongest in fast, frequent trading. The near-miss factor: a trade that almost worked, stopped out by a tick before the move you predicted, feels like a near win, and near wins are more motivating than clear losses, the same mechanism that keeps gamblers feeding slot machines on outcomes that just missed. The rapidity factor: the faster the feedback, the more potent the engagement, because quick outcomes deliver the reward loop again and again in a short time. Day trading maximizes both, a stream of near misses arriving fast, which is why it is the most psychologically addictive and least systematically profitable way to trade. The structure that feels most engaging is the structure most engineered to exploit the illusion of control. This connects the behavioral flaw to a concrete warning from ["Near Misses and Revenge Trading"](https://aligrithm.com/near-misses-and-revenge-trading/): the trading styles that feel the most alive, that give the strongest sense of being in control, in the action, on top of every tick, are the styles where the illusion is doing the most work and the edge is doing the least. The engagement is the product being sold to you, and you are paying for it in costs and overtrading. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Systematic trading removes the illusion, which is why people hate it The honest reason many people cannot stick to systematic trading is that it strips the illusion of control away, and they miss it. A rule-based system makes the decisions, executes on its own logic, and gives you nothing to do in the moment, no entry to time by feel, no position to nurse, no click that feels like agency. The feeling of control that active trading manufactured is simply absent, and its absence feels like boredom, helplessness, even anxiety, the sense that you should be doing something. That void is the subject of ["Why Systematic Trading Feels Emotionally Unsatisfying"](https://aligrithm.com/why-systematic-trading-feels-emotionally-unsatisfying/), and it is the single biggest reason traders abandon working systems to go back to losing discretionary trading: not because the system failed, but because it deprived them of the feeling they were really trading for. Recognizing that the feeling of control was always an illusion is the first step to not needing it, and to letting the rule do the part you were never actually controlling anyway. ## Visualizing the illusion ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_165-visual-0.png) ## KEY POINTS - Active trading feels like control, but the part you control (the click) is trivial and the part that decides the outcome (whether the market moves your way) you do not control at all. - The brain manufactures a sense of control from actions that do not change the odds, like rolling dice harder or picking lottery numbers. Active trading is the same illusion with money attached. - The feeling is rewarding on its own, independent of P&L. Trading engages the brain's reward system the way any sense of control does, so you get paid in the feeling of control while losing the actual currency to costs and bad timing. - Two features amplify the illusion: the near-miss factor (an almost-win is more motivating than a clear loss) and the rapidity factor (faster feedback fires the reward loop more often). Day trading maximizes both. - The styles that feel most alive and in-control are where the illusion does the most work and the edge does the least. The engagement is the product being sold to you. - Systematic trading removes the illusion, leaving nothing to do in the moment, which feels like boredom or anxiety. That void, not system failure, is the biggest reason traders abandon working systems. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Systematic Trading - Robert Carver (Amazon)](https://www.amazon.com/s?k=Systematic+Trading+Robert+Carver&ref=aligrithm.com) - [Trading Systems - Urban Jaekle Emilio Tomasini (Amazon)](https://www.amazon.com/s?k=Trading+Systems+Urban+Jaekle+Emilio+Tomasini&ref=aligrithm.com) ### 6.24 Why Traders Take Profits Too Early and Losses Too Late URL: https://aligrithm.com/why-traders-take-profits-too-early-and-losses-too-late/ Last updated: 2026-07-27T13:25:42.000Z The same trader does two opposite things with winners and losers, and both are wrong. He grabs a small profit the moment it appears, terrified it will evaporate, and he clings to a loss long after it should have been cut, certain it will come back. Sell the winners early, ride the losers late: the disposition effect, and it is the exact reverse of the rule that makes money, which is to cut losses and let winners run. The disposition effect is not two separate mistakes; it is one asymmetry in how the brain treats gains and losses, producing two symmetric errors. ["Get-Even-Itis: The Most Expensive Disease in Trading"](https://aligrithm.com/get-even-itis-the-most-expensive-disease-in-trading/) covered the losing half; this is the full pattern and why it flips your edge upside down. ## One asymmetry, two errors Prospect theory explains both halves with a single curved value function. In the domain of gains, people are risk-averse: a sure gain is preferred to a gamble with a higher expected value, because the pleasure of gains diminishes as they grow, so a bird in the hand beats two in the bush. In the domain of losses, people flip to risk-seeking: a sure loss is rejected in favor of a gamble that might avoid the loss entirely, because the pain of losses also diminishes with size, so a certain small loss feels worse than a coin-flip at a larger one. $$ v(x) = \\begin{cases} x^{\\alpha} & x \\geq 0 \\quad (\\text{gains: concave, risk-averse}) \\\\ -\\lambda(-x)^{\\beta} & x < 0 \\quad (\\text{losses: convex, risk-seeking, } \\lambda > 1) \\end{cases} $$ The value function bends differently above and below the reference point. For gains it is concave, so each additional dollar of profit feels less exciting than the last, which makes you want to lock in the sure thing and sell the winner. For losses it is convex and steeper, the steepness making losses loom larger and the curvature making each additional dollar of loss hurt less than the first, which makes you willing to gamble on the loser recovering rather than accept the certain loss. The same curvature that sells winners early holds losers late. It is one shape producing both errors. ## This is exactly backwards from what works The disposition effect flips the rule that actually generates returns. Profitable trading, especially trend following, depends on a few large winners paying for many small losers, the positive-skew payoff from ["Why Percent Profitable Is Overrated"](https://aligrithm.com/why-percent-profitable-is-overrated/). That payoff requires letting winners run far past the point where the bird-in-the-hand instinct screams to sell, and cutting losers fast, before the get-even instinct can talk you into holding. The disposition effect does the precise opposite: it amputates the large winners before they grow large, destroying the right tail that the whole strategy depends on, and it lets the small losers grow into large ones, fattening the left tail you were trying to keep thin. A trader running on the disposition effect inverts his own edge. He turns a positive-skew strategy, many small losses and rare huge wins, into a negative-skew one, many small wins and rare huge losses, which is the shape that blows up accounts. Every time he sells a winner early he caps a gain; every time he holds a loser late he uncaps a loss; and the sum of those two habits is a system engineered to lose, built on top of signals that might have won. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## You cannot feel your way out of it The disposition effect is not a discipline problem you solve by trying harder, because the asymmetry operates below conscious control and feels like prudence from the inside. Selling a winner feels responsible, taking the profit, not being greedy. Holding a loser feels patient, giving the trade room, not panicking. Both feel like virtues, which is why willpower fails: you are not fighting an obvious vice, you are fighting an instinct that has disguised itself as good judgment. The fix is mechanical. Set exits by rule before the trade, both the stop that cuts the loser and the logic that lets the winner run, so neither decision is made in the moment when the asymmetry is loudest. A disciplined framework does this by forbidding changes to a forecast once a bet is open, because the urge to take the profit or give the loser room is the disposition effect, and the only reliable defense is to have decided the exits when you were calm and then refuse to override them. Let the rule run the winner past your comfort and cut the loser against your hope, because your comfort and your hope are the two signals pointing exactly the wrong way. ## Visualizing the disposition effect ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_164-visual-0.png) ## KEY POINTS - Traders sell winners too early and hold losers too late, the disposition effect, which is the exact reverse of the rule that makes money: cut losses, let winners run. - Both errors come from one asymmetry. Prospect theory's value function is concave in gains (risk-averse, so you lock in the sure profit) and convex and steeper in losses (risk-seeking, so you gamble on the loser recovering). - The same curvature that sells winners early holds losers late. It is one shape producing two symmetric errors, not two separate problems. - This inverts the payoff that profitable trading depends on. Trend following needs a few large winners paying for many small losers, and the disposition effect amputates the winners and fattens the losers. - A trader on the disposition effect turns a positive-skew strategy into a negative-skew one, the shape that blows up accounts, on top of signals that might have won. - Willpower fails because selling winners feels responsible and holding losers feels patient. Both feel like virtues. Set exits by rule before the trade and refuse to override them, because your comfort and hope point the wrong way. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Systematic Trading - Robert Carver (Amazon)](https://www.amazon.com/s?k=Systematic+Trading+Robert+Carver&ref=aligrithm.com) - [Trading Systems - Urban Jaekle Emilio Tomasini (Amazon)](https://www.amazon.com/s?k=Trading+Systems+Urban+Jaekle+Emilio+Tomasini&ref=aligrithm.com) ### 6.23 Get-Even-Itis: The Most Expensive Disease in Trading URL: https://aligrithm.com/get-even-itis-the-most-expensive-disease-in-trading/ Last updated: 2026-07-27T13:25:41.000Z A position goes against you. The thesis that justified it has quietly broken, the reasons to be in the trade are gone, and you hold anyway, because selling now means admitting the loss, and admitting the loss hurts in a way that holding does not. You tell yourself you are waiting to get back to even, just to break even, then you will sell. That is get-even-itis, and it is the single most expensive behavioral disease in trading, because it converts small manageable losses into account-ending ones, one rationalization at a time. ["The Flawed Human Brain in Trading"](https://aligrithm.com/the-flawed-human-brain-in-trading/) named loss aversion as the root; this is the specific way it bankrupts people. ## A paper loss does not feel real The mechanism is a quirk of how the brain registers losses. A paper loss, a position underwater but not yet sold, does not feel fully real, because the brain treats the loss as hypothetical until it is crystallized by the sale. Hold the loser and you preserve the fiction that you have not really lost; sell it and the loss becomes a fact, a number in the realized column, a thing that actually happened to you. The aversion to taking losses is powerful enough that people will accept a worse expected outcome, holding a position likely to fall further, in exchange for postponing the moment the loss becomes real. This is why the trader fixates on getting back to even rather than on the trade's actual prospects. The break-even point has no significance to the market; the market does not know or care what price you paid. It has enormous significance to your brain, because reaching it would let you exit without crystallizing a loss, escaping the pain entirely. So you anchor on your entry price, a number that is meaningful only to you, and you make the hold-or-sell decision based on that anchor instead of on whether the position is still worth holding. The anchor is the disease, and the market punishes anchoring to a price it has already left behind. ## The math turns the disease fatal $$ \\text{gain to recover} = \\frac{1}{1 - L} - 1 $$ Get-even-itis is lethal because the recovery you are waiting for gets harder the longer you wait, in a viciously nonlinear way. A position down 20% needs a 25% gain to get back to even. Down 50%, it needs 100%, a double. Down 80%, it needs 400%. Every increment you hold a deteriorating loser, hoping for the round trip back to your entry, the round trip you need grows faster than the loss, so the break-even you are clinging to recedes faster than the price can plausibly chase it. The trader waiting to get even is waiting for an outcome that becomes less likely precisely because he waited, and the disease feeds itself: the deeper the loss, the more unbearable crystallizing it becomes, the longer he holds, the deeper the loss. ## The cure is a rule that sells for you You cannot beat get-even-itis with willpower in the moment, because the moment is when loss aversion is strongest and your reasoning is weakest. A trader staring at a loser, feeling the lurch of the unrealized loss, will rationalize holding every time, and "just to break even" is the rationalization that always sounds reasonable. The cure is to take the decision out of that moment entirely. Set the exit before the trade, as a stop or a rule tied to the thesis, when you are calm and the loss is hypothetical and your judgment is clear. Then let the rule execute without consulting your in-the-moment feelings, because consulting them is how the disease re-enters. The rule does not care about your entry price, does not feel the paper loss as unreal, does not anchor on break-even; it sells when the condition is met, crystallizing the small loss that keeps you on the gentle part of the recovery curve from ["Why Loss Control Is the Only Thing You Fully Control"](https://aligrithm.com/why-loss-control-is-the-only-thing-you-fully-control/). This is the whole reason a disciplined framework forbids changing your forecast once a bet is open: the change is almost always get-even-itis wearing the costume of a fresh analysis. Pre-commit the exit, and remove yourself from the one decision your brain is guaranteed to get wrong. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Visualizing get-even-itis ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_163-visual-0.png) ## KEY POINTS - Get-even-itis is holding a losing position whose thesis has broken, just to get back to even, because selling crystallizes a loss the brain treats as not-yet-real while it stays on paper. - A paper loss does not feel fully real because the brain treats it as hypothetical until the sale makes it a fact. People accept a worse expected outcome to postpone that moment. - The trader anchors on his entry price, a number meaningful only to him and irrelevant to the market, and decides hold-or-sell based on the anchor instead of the position's actual prospects. - The recovery math is viciously nonlinear: down 20% needs 25% back, down 50% needs 100%, down 80% needs 400%. The break-even recedes faster than price can chase it. - The disease feeds itself: the deeper the loss, the more unbearable crystallizing it becomes, so the trader holds longer and the loss deepens. - Willpower fails in the moment because loss aversion is strongest then. Set the exit before the trade when the loss is hypothetical, let the rule sell, and never change a forecast once a bet is open, since the change is usually the disease in disguise. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Systematic Trading - Robert Carver (Amazon)](https://www.amazon.com/s?k=Systematic+Trading+Robert+Carver&ref=aligrithm.com) - [Trading Systems - Urban Jaekle Emilio Tomasini (Amazon)](https://www.amazon.com/s?k=Trading+Systems+Urban+Jaekle+Emilio+Tomasini&ref=aligrithm.com) ### 6.22 The Flawed Human Brain in Trading URL: https://aligrithm.com/the-flawed-human-brain-in-trading/ Last updated: 2026-07-27T13:25:41.000Z Your brain was built to keep a primate alive on a savanna, and almost none of what it was tuned for helps you trade. The instincts that made your ancestors survive, react fast to threats, avoid losses at all costs, see patterns in noise, follow the herd, are precisely the instincts that lose money in markets, and they are not bugs you can debug. They are the firmware, deep-wired by evolution, strong enough to override the rational analysis you are convinced you are doing. ["Why Simple Algorithms Beat Smart Humans"](https://aligrithm.com/why-simple-algorithms-beat-smart-humans/) claimed the human loses to the rule; this is the catalog of why, the specific flaws that make trading on instinct a losing proposition. ## The biases are features that misfire Behavioral finance grew out of one observation: real people do not make the cold, rational choices that classical economics assumes. They make systematically biased ones, and the biases are consistent enough to have names and to be reproduced in experiment after experiment. The unifying model is prospect theory, which says people do not evaluate outcomes against zero, they evaluate them as gains and losses against a reference point, and they weight losses far more heavily than equivalent gains. A loss of a hundred dollars hurts roughly twice as much as a gain of a hundred dollars pleases, and that single asymmetry drives a cascade of bad trading behavior. These instincts were adaptive once. Loss aversion kept you from gambling away the food that kept you alive through winter. Fast threat response kept you from being eaten. Pattern-seeing helped you find the predator in the grass, even at the cost of seeing predators that were not there. Herding kept you with the group that protected you. Every one of these was a survival advantage in an environment of scarcity and physical danger, and every one of them is a handicap in an environment of probabilistic payoffs and abstract risk, which is the environment of trading. The brain does not know the environment changed. It runs the old firmware on the new problem. ## How each flaw shows up at the screen The savanna instincts translate directly into trading errors, each with a specific cost. Loss aversion becomes the refusal to take losses, holding a loser long past the point the thesis broke, because crystallizing the loss makes it real and the brain will do almost anything to postpone that pain, the get-even-itis from ["Get-Even-Itis: The Most Expensive Disease in Trading"](https://aligrithm.com/get-even-itis-the-most-expensive-disease-in-trading/). The same asymmetry becomes taking winners too early, grabbing the sure small gain because a paper profit feels fragile, detailed in ["Why Traders Take Profits Too Early and Losses Too Late"](https://aligrithm.com/why-traders-take-profits-too-early-and-losses-too-late/). Pattern-seeing becomes finding trends and signals in random price noise, trading setups that are not there. Herding becomes buying what everyone is buying at the top and selling what everyone is selling at the bottom. The need for control becomes overtrading, the illusion from ["The Illusion of Control in Active Trading"](https://aligrithm.com/the-illusion-of-control-in-active-trading/). None of these feels like a bias from the inside; each feels like a reasonable decision, which is exactly what makes them dangerous. ## You cannot fix the brain, so you route around it The conclusion behavioral finance points to is uncomfortable: you cannot debug these flaws by knowing about them. Awareness of a bias does not remove it, because the bias operates below the level of conscious analysis and fires before your reasoning gets a vote. A trader who has read every book on cognitive bias still feels the same lurch of loss aversion when a position goes against him, and still, often, holds the loser. Knowledge is not the cure. The cure is structural: build a rule that makes the decision for you, and remove your in-the-moment judgment from the loop, which is what the rest of this pillar is about. The rule does not feel loss aversion, does not see patterns in noise, does not herd. It is the prosthetic for a brain that was never designed for this task. And the same flaws that you are routing around in yourself persist in every other human in the market, so the rule that immunizes you also lets you harvest their mistakes, which is the only consolation evolution offers the systematic trader. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Visualizing the flawed brain ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_162-visual-0.png) ## KEY POINTS - The human brain was tuned by evolution for survival on a savanna, and the instincts that kept your ancestors alive are the same ones that lose money in markets. They are firmware, not bugs you can debug. - Behavioral finance starts from the observation that real people make systematically biased choices, not the rational ones economics assumes. The unifying model is prospect theory. - Prospect theory says people judge outcomes as gains and losses against a reference point and weight losses about twice as heavily as equal gains. That asymmetry drives a cascade of bad trading. - The savanna instincts misfire at the screen: loss aversion becomes holding losers, the same asymmetry becomes selling winners early, pattern-seeing becomes trading noise, herding becomes buying tops, the need for control becomes overtrading. - None of these feels like a bias from the inside; each feels like a reasonable decision, which is what makes them dangerous. - Awareness does not cure a bias, because it fires below conscious analysis before your reasoning votes. The cure is structural: a rule that makes the decision and removes your in-the-moment judgment, which also lets you harvest everyone else's biases. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Systematic Trading - Robert Carver (Amazon)](https://www.amazon.com/s?k=Systematic+Trading+Robert+Carver&ref=aligrithm.com) - [Trading Systems - Urban Jaekle Emilio Tomasini (Amazon)](https://www.amazon.com/s?k=Trading+Systems+Urban+Jaekle+Emilio+Tomasini&ref=aligrithm.com) ### 6.21 Why Simple Algorithms Beat Smart Humans URL: https://aligrithm.com/why-simple-algorithms-beat-smart-humans/ Last updated: 2026-07-27T13:25:40.000Z A trader with twenty years of experience, a deep understanding of the markets, and a genuine feel for price action will, on average, be beaten by a simple rule he could write on an index card. This is not an insult to the trader; it is a finding that holds across domains, from medical diagnosis to parole decisions to crop yields, and it holds because the human brain carries flaws that no amount of expertise removes. The expert knows more than the rule and trades worse than the rule, because knowledge is not the bottleneck. Consistency is, and the rule is consistent in a way the human cannot be. This opens the behavioral run in this pillar, and ["The Flawed Human Brain in Trading"](https://aligrithm.com/the-flawed-human-brain-in-trading/) catalogs the specific flaws that make it true. ## Knowledge is not the problem The intuitive objection is that a simple rule must be inferior to an expert, because the expert knows so much more, the context, the nuance, the special cases the rule ignores. That extra knowledge is real, and it does not help, because the expert cannot apply it consistently. The same trader, given the same setup on two different days, will trade it differently depending on his mood, his recent results, whether the last trade won or lost, what he read that morning. The knowledge is there; the application of it wobbles, and the wobble costs more than the knowledge gains. A simple rule has no moods. Given the same input it produces the same output every time, which sounds like a limitation and is the entire advantage. The rule's consistency means its edge, however modest, gets applied in full on every occasion, while the expert's larger edge gets applied erratically, eroded by every inconsistency between what he knows and what he does under pressure. Across many decisions, the consistent small edge beats the erratic large one, because the erosion compounds. ## The expert adds noise, not signal The deeper finding is that human judgment laid on top of a rule usually makes it worse, not better. Give an expert a good statistical model and let him override it when his experience says the model is wrong, and his overrides, on average, subtract value. He overrides on the cases that feel exceptional, and feeling exceptional is exactly when his biases are loudest, so his interventions are concentrated in the moments he is least reliable. The rule was right and the override was the bias talking, dressed up as expertise. This is why the recommendation is not to use the rule as a starting point for human refinement, but to use the rule and resist refining it. The whole case for systematic trading rests on an admission that even people who know finance deeply make bad financial decisions because they are human, and the fix is to systematize the decision and then stay out of its way. The expert's job is to design the rule, using all his knowledge, in a calm moment. The expert's job is not to second-guess the rule in the heat of the trade, because the second-guessing is where the flaws re-enter the system he built to escape them. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## You also get to exploit the other humans There is a second reason the simple rule wins, beyond your own consistency: everyone else is human too. The cognitive biases that wreck your judgment wreck the judgment of every discretionary trader in the market, and their biased decisions leave patterns in prices, the systematic over-holding of losers, the premature selling of winners, the overreaction and the herding. A rule that is immune to those biases can be built to exploit them in others. So the rule helps you twice: it stops you from making the mistakes, and it positions you to profit from everyone who still does. The discipline premium from ["The Discipline Premium in Trading"](https://aligrithm.com/the-discipline-premium-in-trading/) is exactly this second effect, the money left on the table by the undisciplined for the disciplined to collect. ## Visualizing rule vs expert ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_161-visual-0.png) ## KEY POINTS - A simple rule beats an experienced expert on average, across many domains, because knowledge is not the bottleneck. Consistency is, and the rule is consistent in a way the human cannot be. - The expert's extra knowledge is real and does not help, because he cannot apply it consistently. The same setup gets traded differently depending on mood, recent results, and what he read that morning. - A rule has no moods, so its modest edge gets applied in full every time, while the expert's larger edge gets eroded by the wobble between what he knows and what he does under pressure. - Human judgment laid on top of a rule usually subtracts value. Overrides cluster on cases that feel exceptional, which is exactly when biases are loudest and the human is least reliable. - The fix is to design the rule with all your knowledge in a calm moment, then stay out of its way. Second-guessing in the heat of the trade re-introduces the flaws the rule was built to escape. - The rule wins twice: it stops your mistakes and positions you to exploit the biases of every discretionary trader still making theirs. That is the discipline premium. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Systematic Trading - Robert Carver (Amazon)](https://www.amazon.com/s?k=Systematic+Trading+Robert+Carver&ref=aligrithm.com) - [Trading Systems - Urban Jaekle Emilio Tomasini (Amazon)](https://www.amazon.com/s?k=Trading+Systems+Urban+Jaekle+Emilio+Tomasini&ref=aligrithm.com) ### 6.20 Why Loss Control Is the Only Thing You Fully Control URL: https://aligrithm.com/why-loss-control-is-the-only-thing-you-fully-control/ Last updated: 2026-07-27T13:25:40.000Z You do not control whether a trade wins. You do not control how far the market moves, when the trend comes, or whether your edge shows up this month. The one thing entirely in your hands is how much you lose when you are wrong, and that single controllable lever, position size, stops, and exits, determines survival more than any signal does. Traders pour all their effort into the part they cannot control, the prediction, and neglect the part they own completely, the loss. This closes the system-death run in this pillar, because the systems that die are the ones that lost control of their losses, not the ones with the worst signals. ## The asymmetry of control Split a trade into what you decide and what the market decides. You decide the entry, the size, where you cut a loser, and when you take a winner. The market decides everything else: whether the move comes, how big it is, how long it takes. Of your decisions, the entry gets all the attention and matters least, because once you are in the position the outcome is in the market's hands. The exit and the size are where your control actually lives, because they cap the damage regardless of what the market does. You cannot make a trade win, but you can guarantee that a loss stays within a bound you chose in advance. This is why loss control, not forecasting, is the load-bearing skill. A great forecast with no loss control blows up on the one trade that goes wrong before the edge plays out, because an uncapped loss can erase a year of capped gains in a single position. A mediocre forecast with strict loss control survives, compounds its small edge, and stays in the game long enough for the edge to matter. Survival is a prerequisite for everything else, and survival is a function of the losses you allow, which is the one input you set yourself. ## The math is brutally asymmetric $$ \\text{gain needed to recover} = \\frac{1}{1 - L} - 1 $$ Loss control matters more than gain because recovery from a loss is harder than the loss itself. A drawdown of a given fraction requires a gain of one divided by one minus that fraction, minus one, just to get back to even. Lose 10% and you need 11% to recover, an almost-fair trade. Lose 50% and you need 100%, a doubling, to break even. Lose 90% and you need 900%. The curve steepens viciously as losses grow, so a large loss is not just twice as bad as a medium one, it is many times harder to undo. This asymmetry is the entire mathematical case for controlling losses tightly: keeping every loss small keeps you on the gentle part of the recovery curve, where getting back to even is plausible, and one uncontrolled loss throws you onto the steep part, where recovery requires returns you will never reliably produce. ## Control it where you actually can Loss control is implemented in the levers you set, not in hope. Position size is the first and largest, because it scales every loss directly; the volatility-based sizing from ["Why Volatility-Adjusted Position Sizing Matters"](https://aligrithm.com/why-volatility-adjusted-position-sizing-matters/) is loss control, capping how much any single position can swing against you. The stop or exit is the second, bounding the loss on a position that moves against you before your thesis resolves. The portfolio construction from ["Why Portfolio Construction Is Part of the Signal"](https://aligrithm.com/why-portfolio-construction-is-part-of-the-signal/) is loss control at the book level, stopping correlated positions from turning into one giant concentrated loss. Each of these is something you decide and the market cannot touch, which is exactly why they are where your effort belongs. The honest limit: loss control bounds the loss you planned for, not the loss the market can deliver. A gap through your stop, a limit-down market you cannot exit, a position that moves more between the close and the open than your stop ever contemplated, these break the cap, and they are the reason loss control is necessary but not sufficient. You set the size assuming a worst case, and the market occasionally serves a worse one, which is why the sizing leaves margin beyond the planned loss and why the tail articles in this pillar matter. Control what you can to the limit of your ability, and size for the days when even your controls slip, because the only thing more dangerous than no loss control is loss control you trust completely. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Visualizing loss control ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_160-visual-0.png) ## KEY POINTS - You do not control whether a trade wins or how far the market moves. The one thing fully in your hands is how much you lose when you are wrong, and that lever decides survival more than any signal. - The entry gets all the attention and matters least, because once you are in, the outcome is the market's. The exit and the size are where your control lives, capping damage regardless of what the market does. - A great forecast with no loss control blows up on the one trade that goes wrong before the edge plays out. A mediocre forecast with strict loss control survives and compounds. Survival is a prerequisite for everything. - Recovery is asymmetric: a 10% loss needs an 11% gain, a 50% loss needs 100%, a 90% loss needs 900%. The curve steepens viciously, so a big loss is many times harder to undo than a medium one. - Implement loss control in the levers you set: position size (scales every loss), stops and exits (bound a single position), and portfolio construction (stops correlated positions becoming one giant loss). Each is yours and the market cannot touch it. - The cap holds for planned losses, not for gaps, limit-down markets, or overnight moves past your stop. Loss control is necessary but not sufficient, so size with margin beyond the planned loss and never trust the controls completely. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Systematic Trading - Robert Carver (Amazon)](https://www.amazon.com/s?k=Systematic+Trading+Robert+Carver&ref=aligrithm.com) - [Trading Systems - Urban Jaekle Emilio Tomasini (Amazon)](https://www.amazon.com/s?k=Trading+Systems+Urban+Jaekle+Emilio+Tomasini&ref=aligrithm.com) ### 6.2 Ranked Long/Short Systems Explained URL: https://aligrithm.com/ranked-long-short-systems-explained/ Last updated: 2026-07-27T13:25:30.000Z Take a basket of instruments, score each one with a signal, sort them, buy the top slice and short the bottom slice in equal risk. That sentence is the entire architecture of a ranked long/short system, and once you internalize it you stop thinking about individual trades and start thinking about a portfolio that rebalances itself every period. ["Ranking Beats Forecasting for Many Trading Problems"](https://aligrithm.com/ranking-beats-forecasting-for-many-trading-problems/) argued why the ordering is the robust thing to trade; this is how you turn the ordering into actual positions. ## The mechanism: sort, slice, offset You run one number per instrument, the score, whatever your signal produces, a momentum reading, a carry, a mean-reversion z-score. Sort the universe by that score. The instruments at the top are the ones the signal likes most, the bottom are the ones it likes least. Go long a slice of the top, say the top fifth or top third, and short the matching slice at the bottom. Size the long book and the short book to carry the same risk, so the position is a bet on the spread between the winners and losers, not on the market itself. That equal-and-offsetting construction is what makes the system market-neutral by design. When the whole universe rallies, your longs and shorts both move and the broad move cancels. What survives is the differential: did the instruments you ranked high outperform the ones you ranked low? The market direction, the thing you have no edge predicting, gets subtracted out, and you are left holding only the part you actually have a view on. ## Why the spread, not the level A ranked book is structurally a relative bet, and that is its strength and its limit at once. The strength: you are immunized against the factor you cannot forecast. You never have to be right about whether commodities go up or down this month; you only have to be right that your top names beat your bottom names. The limit: if your whole universe is junk, ranking the junk still leaves you holding junk on both sides, and the spread can stay flat or bleed costs while you wait for a differential that never shows up. $$ w\_i = \\frac{r\_i - \\bar{r}}{\\sum\_{j} |r\_j - \\bar{r}|} $$ One clean way to turn ranks into weights: center the ranks and normalize. Each instrument's rank, compared against the average rank, gives a centered value, and the denominator sums the absolute deviations so the weights add to a fixed gross exposure. Instruments above the median rank get positive weights (longs), those below get negative (shorts), and the size scales with how far from the middle they sit. The top-ranked and bottom-ranked names carry the biggest weights; the ones near the median carry almost nothing, which is correct, because the middle of a ranking is where the signal is weakest and least worth betting on. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## The slice width is a real decision How wide you cut the slices trades breadth against conviction. Top-and-bottom fifths give you a sharp, concentrated bet on the extremes where the signal is strongest, and more turnover and more idiosyncratic risk per name. Top-and-bottom halves spread the bet across the whole universe, diluting conviction but smoothing the ride and lowering turnover. Neither is right in the abstract. A noisy signal where only the extremes carry information wants narrow slices; a broad, gentle signal that orders the whole universe meaningfully wants wide ones. This is a tuning choice, and like every tuning choice it is a place to overfit, so you decide it on the logic of the signal, not by sweeping it for the best backtest. ## Visualizing the ranked book ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_142-visual-0.png) ## KEY POINTS - A ranked long/short system scores each instrument, sorts the universe, and goes long the top slice while shorting the bottom slice in matched risk. The position is a bet on the spread between winners and losers. - Matching long and short risk makes the system market-neutral by construction. The broad move cancels, and what survives is the differential between the high-ranked and low-ranked names. - The strength is immunity to the factor you cannot forecast; you never bet on market direction. The limit is that ranking a junk universe leaves you holding junk on both sides while costs bleed. - Turn ranks into weights by centering and normalizing, so weight scales with distance from the median. Extreme names carry the most; names near the middle carry almost nothing, which is correct because the signal is weakest there. - Slice width trades breadth against conviction. Narrow extremes suit a noisy signal that only informs at the tails; wide slices suit a gentle signal that orders the whole universe. - Slice width is a tuning choice and a place to overfit. Decide it from the logic of the signal, not by sweeping for the best backtest. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Systematic Trading - Robert Carver (Amazon)](https://www.amazon.com/s?k=Systematic+Trading+Robert+Carver&ref=aligrithm.com) - [Trading Systems - Urban Jaekle Emilio Tomasini (Amazon)](https://www.amazon.com/s?k=Trading+Systems+Urban+Jaekle+Emilio+Tomasini&ref=aligrithm.com) ### 6.19 When to Switch Off a Trading System URL: https://aligrithm.com/when-to-switch-off-a-trading-system/ Last updated: 2026-07-27T13:25:39.000Z Knowing when to kill a system is harder than building one, and most traders are terrible at it in both directions. They switch off a good system in a normal drawdown out of fear, locking in the loss right before the recovery, and they keep a dead system running long past the point its edge vanished, hoping. Both errors come from the same root: deciding in the moment, when fear and hope are loudest, instead of by a rule fixed in advance. ["When a Drawdown Means the System Is Broken"](https://aligrithm.com/when-a-drawdown-means-the-system-is-broken/) gave you the statistical signal; this is the full decision of when to pull the plug and how to avoid pulling it for the wrong reason. ## The two ways to get it wrong Switching off too early is the more common error and the more expensive one. A system enters a drawdown that sits comfortably inside its historical and Monte Carlo envelope, the trader feels the pain, decides the magic is gone, and turns it off. The drawdown was the cost of doing business, the recovery comes, and the trader either misses it or, worse, switches the system back on at the next peak and catches the next drawdown, doing the exact opposite of what the system needed. This is buying high and selling low applied to your own strategies, and it converts a profitable system into a realized loss through nothing but bad timing of the on-off switch. Switching off too late is rarer and more lethal. The edge genuinely decayed, the world changed, the inefficiency got arbitraged away, and the trader keeps running the system because admitting it is dead means admitting the research was wasted and the recent losses were not bad luck. Hope substitutes for evidence, the drawdown grinds past every historical precedent, and the account bleeds while the trader waits for a mean reversion that will not come because the mean itself moved. This is "Get-Even-Itis" from the behavioral articles in this pillar, applied to a strategy instead of a trade. ## Decide by rule, before the pain The only defense against both errors is a kill rule written before the drawdown, when you are calm. Tie it to the statistical envelope: keep running while the drawdown stays inside the permutation distribution, investigate and cut risk when it breaches a high percentile of depth or duration, stop when the breach is confirmed and the edge logic no longer holds. Writing it in advance does two things. It stops you from switching off a system that is merely having a statistically normal bad run, because the rule says hold while you are inside the envelope. And it forces you to switch off a system that has objectively broken, because the rule fires whether or not your hope agrees. The rule should also separate two distinct kill reasons, because they demand different responses. A statistical breach, a drawdown beyond the envelope, says the system's recent behavior no longer matches its history, which might be a dead edge or a rare tail. A structural breach is different and more decisive: the specific market inefficiency the system exploited has provably changed, a rule change, a new participant, a spread that closed, an instrument that stopped trading the way it did. A structural breach kills a system even before the drawdown confirms it, because you know the edge is gone for a reason, not just because the equity fell. Watch for both, and weight the structural reason more heavily, because it is evidence about the cause rather than the symptom. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Switching off is not all-or-nothing The decision is not a binary switch, and treating it as one drives the panic. The graduated version is to cut size as confidence in the edge falls and restore it as confidence returns, so a breach of the first envelope threshold halves the allocation rather than zeroing it, and a confirmed structural break zeroes it. Sizing down buys you information cheaply: you keep the system live enough to see whether it recovers, while limiting the damage if it does not, which is the right posture under genuine uncertainty about whether an edge is dead or merely dormant. The trader who can only fully on or fully off is forced into a high-stakes guess at the worst moment; the trader who sizes down turns the kill decision into a dial and lets the evidence accumulate. The discipline is the same as the rest of this pillar: pre-commit the rule, act on the statistics, and keep your in-the-moment judgment, the part most corrupted by fear and hope, out of the decision. ## Visualizing the kill decision ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_159-visual-0.png) ## KEY POINTS - Knowing when to kill a system is harder than building one. Traders switch off good systems in normal drawdowns out of fear and keep dead systems running out of hope, both from deciding in the moment. - Switching off too early is common and expensive: a drawdown inside the normal envelope feels fatal, the trader quits, and misses the recovery or re-enters at the next peak, buying high and selling low on their own strategy. - Switching off too late is rarer and lethal: the edge genuinely decayed but admitting it means admitting wasted research, so hope replaces evidence and the account bleeds past every precedent. - Defend against both with a kill rule written before the drawdown: hold inside the permutation envelope, cut risk on a high-percentile breach of depth or duration, stop when the breach is confirmed and the edge logic fails. - Separate a statistical breach (recent behavior no longer matches history, could be a dead edge or a tail) from a structural breach (the exploited inefficiency provably changed). A structural break kills a system even before the drawdown confirms it. - The decision is a dial, not a switch. Cut size as confidence falls and restore it as confidence returns, so a breach halves the allocation and a confirmed structural break zeroes it, letting evidence accumulate cheaply. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Systematic Trading - Robert Carver (Amazon)](https://www.amazon.com/s?k=Systematic+Trading+Robert+Carver&ref=aligrithm.com) - [Trading Systems - Urban Jaekle Emilio Tomasini (Amazon)](https://www.amazon.com/s?k=Trading+Systems+Urban+Jaekle+Emilio+Tomasini&ref=aligrithm.com) ### 6.18 Smooth Equity Curves Are Built, Not Found URL: https://aligrithm.com/smooth-equity-curves-are-built-not-found/ Last updated: 2026-07-27T13:25:39.000Z Traders hunt for the one system with the clean, gently rising equity curve, the holy grail that never has an ugly drawdown. It does not exist, and the hunt is the wrong activity. Smooth equity curves are not discovered in a single strategy; they are assembled from several rougher ones whose drawdowns happen at different times. The smoothness is a property of the combination, not of any component, which means the work is in the portfolio, not in finding a perfect signal. ["The Hidden Importance of Time in Market"](https://aligrithm.com/the-hidden-importance-of-time-in-market/) set up why systems have to overlap to combine; this is what the combining buys you. ## Every single system is lumpy A standalone trading system has a jagged equity curve because its edge depends on a particular kind of market, and markets cycle through conditions that suit it and conditions that punish it. A trend follower soars in trending regimes and bleeds in chop. A mean-reverter does the reverse. Each one, run alone, delivers stretches of gains broken by drawdowns that can last quarters or years, and no amount of parameter tuning removes the drawdowns, because the drawdowns are the system being out of its favorable regime, not a flaw you can optimize away. Chasing a single smooth curve is chasing a system with no unfavorable regime, which is a system that does not exist. The drawdowns of a single system are also the thing most likely to make you quit it, and quitting a good system in its drawdown is how traders convert a profitable strategy into a realized loss. The roughness is not just uncomfortable; it is the behavioral trap that ["When to Switch Off a Trading System"](https://aligrithm.com/when-to-switch-off-a-trading-system/) warns about, because a long drawdown in a standalone system is indistinguishable, in the moment, from a dead one. ## Smoothness comes from offsetting drawdowns The fix is to run several systems whose drawdowns do not coincide. When the trend follower is bleeding in a chop, the mean-reverter is feeding, and the combined equity curve barely notices the chop because one system's loss is filled by the other's gain. Stack enough genuinely different systems, different strategies, different markets, different holding periods, and the combined curve smooths out, not because any component improved, but because their rough patches fall in different places and average away. $$ \\sigma\_{\\text{portfolio}}^2 = \\sum\_i w\_i^2 \\sigma\_i^2 + \\sum\_{i \\neq j} w\_i w\_j \\, \\rho\_{ij}\\, \\sigma\_i \\sigma\_j $$ The portfolio variance has two parts: the sum of each system's own variance and the cross terms that depend on the correlations between systems. The first part you cannot escape; the second is where the smoothing lives. When the correlations are low or negative, the cross terms add little or subtract, so the combined volatility falls well below the average of the individual volatilities. That gap, the combined volatility coming in under the components' average, is the entire benefit of diversification across systems, and it is the only free lunch in trading, available solely to the trader who builds the combination rather than hunting for a single perfect curve. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## What it takes to build it Building a smooth curve is harder than finding a good system, because you need several systems that are each independently worth running and genuinely uncorrelated with each other. Two trend followers on correlated markets are nearly the same system and smooth nothing; their drawdowns coincide and the cross terms stay high. You need real diversity of edge, a trend system and a mean-reversion system and a carry system, across diverse markets, with enough overlapping time in market from ["The Hidden Importance of Time in Market"](https://aligrithm.com/the-hidden-importance-of-time-in-market/) for the offsetting to actually occur. And you need the correlations to be low out of sample, not just in the backtest, which is the catch, because correlations between strategies rise exactly in crises, when several supposedly independent systems all draw down together as liquidity vanishes and everything correlates to one. The smooth curve you built in calm conditions can turn jagged in the one environment where you needed the smoothing most, so you build for diversification and still size for the day the diversification fails. ## Visualizing the built curve ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_158-visual-0.png) ## KEY POINTS - The single smooth-equity-curve system does not exist. Smoothness is a property of a combination of rougher systems, so the work is in the portfolio, not in finding a perfect signal. - Every standalone system is lumpy because its edge depends on a regime, and markets cycle through conditions that suit it and conditions that punish it. The drawdowns are the system being out of favor, not a flaw you can tune away. - A single system's drawdowns are the behavioral trap that makes traders quit good strategies, because a long drawdown is indistinguishable in the moment from a dead edge. - Smoothness comes from running systems whose drawdowns do not coincide. When the trend follower bleeds in a chop, the mean-reverter feeds, and the combined curve barely notices. - Portfolio variance has a cross-term driven by the correlations between systems. Low or negative correlations shrink it, so combined volatility falls below the components' average. That gap is the only free lunch in trading. - Building it needs several independently worthwhile, genuinely uncorrelated systems with overlapping time in market. Correlations rise in crises, so the smoothing fails when you need it most, and you still size for that day. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Systematic Trading - Robert Carver (Amazon)](https://www.amazon.com/s?k=Systematic+Trading+Robert+Carver&ref=aligrithm.com) - [Trading Systems - Urban Jaekle Emilio Tomasini (Amazon)](https://www.amazon.com/s?k=Trading+Systems+Urban+Jaekle+Emilio+Tomasini&ref=aligrithm.com) ### 6.17 The Hidden Importance of Time in Market URL: https://aligrithm.com/the-hidden-importance-of-time-in-market/ Last updated: 2026-07-27T13:25:38.000Z Two systems can earn the same profit per trade and have wildly different value, and the difference is time. One system holds each position for two days; the other holds for two months. The first recycles its capital thirty times while the second holds one bet, so even at equal per-trade expectancy the fast system can put its edge to work far more often, or tie up capital far less, than the slow one. Time in market is the dimension that profit factor, win rate, and expectancy all ignore, and it decides both how much you earn per unit of capital and whether two systems can be combined at all. The metrics in ["Expectancy: The Most Important Formula in Trading"](https://aligrithm.com/expectancy-the-most-important-formula-in-trading/) are all per-trade; this is what happens when you put them on a clock. ## Per-trade is not per-year Expectancy tells you what a trade is worth. It says nothing about how many trades you get or how long each one ties up your capital, and those determine your actual return on capital. A system with a modest per-trade expectancy that turns over weekly compounds that edge many times a year. A system with a large per-trade expectancy that holds for months collects the edge a few times a year and leaves the capital committed in between. Comparing the two on per-trade numbers is comparing a sprinter's stride length without asking how many strides they take, and the answer changes which system you would rather run. Average time in trade is the bridge between the per-trade metrics and the per-year reality. Multiply expectancy by how many non-overlapping trades the holding period allows in a year and you get the annual edge from that capital, which is the number that actually matters for allocation. A short holding period is a multiplier on your edge, and it is also a cost multiplier, because faster turnover pays more spread, which is why the cost discipline from ["Cost-Aware Ranking: The Missing Step in Cross-Sectional Strategies"](https://aligrithm.com/cost-aware-ranking-the-missing-step-in-cross-sectional-strategies/) bites hardest on the fastest systems. ## Time decides whether systems combine The deeper reason time matters shows up when you build a portfolio of systems. The whole value of running several systems together, covered in ["Smooth Equity Curves Are Built, Not Found"](https://aligrithm.com/smooth-equity-curves-are-built-not-found/), is that their equity curves are uncorrelated, so one is winning while another draws down and the combined ride is smoother than any single one. That smoothing only works if the systems are actually in the market at overlapping times. Two beautifully uncorrelated systems that are never in a position simultaneously cannot offset each other, because when one is drawing down the other is flat, sitting in cash, doing nothing to cushion the dip. So the average time in trade is not just an efficiency number, it is a compatibility number. To exploit the negative correlation between two systems' returns, they need enough simultaneous market exposure for one to be winning while the other loses. A system that is in the market 5% of the time contributes almost nothing to smoothing a portfolio of systems that are always on, no matter how attractive its standalone statistics, because it is in cash during the drawdowns it was supposed to offset. When you assemble systems, you check their time in market and their overlap, not just their standalone returns and correlations, because a correlation between two return streams that rarely coincide is a correlation you cannot actually harvest. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Time also sets your opportunity cost The third edge of time is the capital you are not deploying. A system in the market 10% of the time leaves your capital idle 90% of the time, and idle capital earns nothing or earns the cash rate, which is an opportunity cost the per-trade metrics never charge you. A lower per-trade expectancy that keeps capital working can beat a higher one that leaves it parked, once you account for what the parked capital could have earned elsewhere. This is why time in market belongs on the report next to expectancy and profit factor, and why a system that looks excellent per trade can be a mediocre use of capital once you see how rarely it actually has a position on. ## Visualizing time in market ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_157-visual-0.png) ## KEY POINTS - Time in market is the dimension expectancy, profit factor, and win rate all ignore. Two systems with equal per-trade edge differ in value because one recycles capital many times a year and the other holds one bet. - Per-trade is not per-year. Annual edge is roughly expectancy times the number of non-overlapping trades the holding period allows, so a short holding period multiplies a modest edge and a long one collects a large edge rarely. - A short holding period multiplies both the edge and the cost, since faster turnover pays more spread, so cost discipline bites hardest on the fastest systems. - Time decides whether systems combine. The smoothing from uncorrelated systems only works when they are in the market at overlapping times; otherwise one sits in cash during the other's drawdown and cannot offset it. - Average time in trade is a compatibility number, not just an efficiency one. Check time in market and overlap when assembling systems, because a correlation between rarely-coinciding streams cannot be harvested. - Time sets opportunity cost. Idle capital earns nothing or the cash rate, so a lower per-trade expectancy that keeps capital working can beat a higher one that leaves it parked. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Systematic Trading - Robert Carver (Amazon)](https://www.amazon.com/s?k=Systematic+Trading+Robert+Carver&ref=aligrithm.com) - [Trading Systems - Urban Jaekle Emilio Tomasini (Amazon)](https://www.amazon.com/s?k=Trading+Systems+Urban+Jaekle+Emilio+Tomasini&ref=aligrithm.com) ### 6.16 Profit Factor, Expectancy, and the Shape of Returns URL: https://aligrithm.com/profit-factor-expectancy-and-the-shape-of-returns/ Last updated: 2026-07-27T13:25:38.000Z Profit factor and expectancy are the two headline numbers on every system report, and traders treat them as a verdict: high profit factor good, positive expectancy good, ship it. They are useful, and they are both blind to the same thing, which is the shape of the returns that produced them. Two systems with identical profit factor and identical expectancy can have completely different survival odds, because one earns its number from many balanced trades and the other from a handful of monster winners that may never repeat. ["Expectancy: The Most Important Formula in Trading"](https://aligrithm.com/expectancy-the-most-important-formula-in-trading/) gave you the average; this is about why the average hides the distribution that actually kills or saves you. ## Profit factor, and how it relates to expectancy $$ \\text{PF} = \\frac{\\text{gross profit}}{\\text{gross loss}} = \\frac{P\_{\\text{win}} \\times \\text{avg win}}{P\_{\\text{loss}} \\times \\text{avg loss}} $$ Profit factor is the total money won divided by the total money lost, the sum of all winners over the absolute sum of all losers. A profit factor above 1 means you made more than you lost; below 1 means you bled. It is the same ingredients as expectancy, win rate and the payoff sizes, arranged as a ratio instead of a difference. Expectancy tells you the average dollars per trade; profit factor tells you how many dollars you earn per dollar you lose. A profit factor of 1.5 means every dollar of losses bought you a dollar fifty of gains, which is a workable but not generous edge; values comfortably above that are strong, and values clinging just above 1 are fragile, because a small deterioration tips them under water. The two numbers are different views of one edge, and neither shows you the shape. Both collapse the entire return distribution into a single statistic, and a single statistic cannot distinguish a smooth stream of moderate results from a jagged one dominated by outliers. ## Same numbers, different shape, different survival Picture two systems, both with a profit factor of 2 and the same expectancy. System A earns it from a steady distribution: most winners cluster around the average, most losers around the average, and no single trade dominates the total. System B earns the identical profit factor from a lumpy distribution: a long tail of small losses and two enormous winners that, between them, account for most of the gross profit. On the report they look the same. In reality they are not remotely the same bet. System A is robust, because its edge is spread across many trades and the law of large numbers is working for you; remove any single trade and the numbers barely move. System B is fragile, because its entire edge rests on two outliers, and if those two trades were luck, a perfect entry that will not recur, a gap that happened to go your way, the real expectancy going forward is near zero or negative. The profit factor of 2 is real in the backtest and meaningless out of sample, because it was never an edge, it was two lottery tickets that hit. The shape is the difference between a system you can size with confidence and one that will betray you the moment the outliers stop arriving. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Read the distribution, not just the summary The defense is to look past the two headline numbers at the distribution that made them. Plot the trade returns and check whether the profit is broad-based or concentrated. Compute how much of the gross profit comes from the top few trades; if removing your three best winners turns a great system into a losing one, you do not have a system, you have three winners and a lot of noise. Check the loss tail the same way, because a profit factor built on a quiet loss distribution that has never yet seen its worst day is overstating your edge until that day arrives. This connects to ["Average Drawdown vs Extreme Drawdown"](https://aligrithm.com/average-drawdown-vs-extreme-drawdown/): the shape of the trade distribution and the shape of the drawdown distribution are the same information, and both warn you when a summary statistic is resting on a tail you cannot count on. Use profit factor and expectancy to size up the edge, and use the shape to decide whether the edge is real enough to trust. ## Visualizing the shape ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_156-visual-0.png) ## KEY POINTS - Profit factor is gross profit divided by gross loss, the dollars won per dollar lost. Above 1 you made money; around 1.5 is workable; comfortably higher is strong; just above 1 is fragile. - Profit factor and expectancy use the same ingredients arranged differently, a ratio versus a difference. Both collapse the whole return distribution into one number and are blind to its shape. - Two systems with identical profit factor and expectancy can have opposite survival odds. One earns its number from many balanced trades; the other from a few monster winners. - The broad-based system is robust: remove any single trade and the numbers barely move. The outlier-driven system is fragile: its edge rests on a handful of trades that may have been luck. - A profit factor built on two outliers is real in the backtest and meaningless out of sample. If removing your three best winners turns the system into a loser, you have three winners and noise, not a system. - Read the distribution behind the summary. Check how much profit comes from the top few trades, inspect the loss tail, and remember the trade-shape and drawdown-shape are the same warning. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Systematic Trading - Robert Carver (Amazon)](https://www.amazon.com/s?k=Systematic+Trading+Robert+Carver&ref=aligrithm.com) - [Trading Systems - Urban Jaekle Emilio Tomasini (Amazon)](https://www.amazon.com/s?k=Trading+Systems+Urban+Jaekle+Emilio+Tomasini&ref=aligrithm.com) ### 6.15 Why Percent Profitable Is Overrated URL: https://aligrithm.com/why-percent-profitable-is-overrated/ Last updated: 2026-07-27T13:25:37.000Z Win rate is the metric beginners worship and professionals barely glance at. A system that wins 80% of the time feels safe, smart, and worth bragging about, while one that wins 35% feels broken. Both feelings are usually wrong, because the percentage of profitable trades, on its own, tells you nothing about whether a system makes money. ["Expectancy: The Most Important Formula in Trading"](https://aligrithm.com/expectancy-the-most-important-formula-in-trading/) already showed the arithmetic; this article is about why the win rate is the most psychologically seductive and least informative number on the report, and why chasing it actively damages systems. ## The win rate is half a number The percentage of profitable trades measures one thing: how often you are right. It says nothing about how much you make when right or lose when wrong, and those sizes are the other half of the result. A win rate without the average win-to-loss ratio beside it is a sentence with the verb removed. You can read it and feel something, but you cannot act on it, because the same win rate sits on top of a great system and a terrible one depending on what the winners and losers are worth. The two numbers also trade off against each other in a way that makes a high win rate suspicious rather than reassuring. Systems engineered for a high percentage of winners usually get there by taking profits early and giving losers room, which produces many small wins and occasional large losses, the exact shape that quietly destroys accounts. Systems with low win rates often get there by cutting losers fast and letting winners run, producing many small losses and occasional large wins. The win rate is partly a readout of which of these two postures the system takes, and the comfortable-looking high number frequently marks the dangerous posture. ## The breakeven line depends entirely on the ratio $$ P\_{\\text{win}}^{\\text{breakeven}} = \\frac{1}{1 + R}, \\quad R = \\frac{\\text{avg win}}{\\text{avg loss}} $$ The win rate you need just to break even depends on the win-to-loss ratio. With a payoff ratio of 1, where winners and losers are the same size, you need to win more than half your trades. With a ratio of 6, the trend-following case where the average win is six times the average loss, the breakeven win rate is one divided by one plus six, about 14%, so a system winning 36% of the time is hugely profitable. With a ratio of 0.2, where winners are a fifth the size of losers, the breakeven win rate is one divided by 1.2, about 83%, so an 80% win rate is still a losing system. There is no win rate that is good or bad in isolation; there is only a win rate relative to the payoff ratio it sits next to. ## Chasing win rate breaks systems The seduction is not harmless, because optimizing for the win rate degrades real systems. Add a tight profit target to a trend follower and the win rate climbs, the metric you were chasing improves, and the expectancy collapses because you have amputated the large winners that paid for all the small losses. Move a stop further away to avoid getting tagged out and the win rate climbs again while the average loss balloons, trading a better-looking percentage for a worse actual result. Each of these is a move that makes the seductive number go up and the number that matters go down, which is why win-rate optimization is one of the most reliable ways to ruin a working strategy. Judge a system on expectancy and the payoff ratio, and treat the win rate as context, useful for understanding the system's posture and for knowing whether you can psychologically tolerate a long string of small losses, never as a goal. A trader who cannot sit through a 35%-win-rate system's frequent losses will abandon a profitable strategy, so the win rate matters for your behavior, covered in ["Why Systematic Trading Feels Emotionally Unsatisfying"](https://aligrithm.com/why-systematic-trading-feels-emotionally-unsatisfying/), and not for the system's edge. The edge lives in expectancy; the win rate just tells you how hard the edge will be to hold onto. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Visualizing the win-rate trap ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_155-visual-0.png) ## KEY POINTS - Win rate measures only how often you are right, never how much you make when right or lose when wrong. On its own it cannot tell you whether a system makes money. - A win rate without the average win-to-loss ratio beside it is unusable. The same percentage sits on a great system and a terrible one depending on the payoff sizes. - High win rates are often produced by taking profits early and giving losers room, the many-small-wins, occasional-huge-loss shape that destroys accounts. Low win rates often mark the safer cut-losers-let-winners-run posture. - The breakeven win rate is one over one plus the payoff ratio. At a ratio of 6 you break even at 14%, so 36% is hugely profitable; at a ratio of 0.2 you break even at 83%, so 80% still loses. - Optimizing for win rate breaks systems. A tight profit target raises the win rate and collapses expectancy by amputating the large winners; a wider stop raises the win rate and balloons the average loss. - Judge systems on expectancy and payoff ratio. Treat win rate as context for the system's posture and for whether you can psychologically tolerate long strings of losses, never as a goal. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Systematic Trading - Robert Carver (Amazon)](https://www.amazon.com/s?k=Systematic+Trading+Robert+Carver&ref=aligrithm.com) - [Trading Systems - Urban Jaekle Emilio Tomasini (Amazon)](https://www.amazon.com/s?k=Trading+Systems+Urban+Jaekle+Emilio+Tomasini&ref=aligrithm.com) ### 6.14 Expectancy: The Most Important Formula in Trading URL: https://aligrithm.com/expectancy-the-most-important-formula-in-trading/ Last updated: 2026-07-27T13:25:37.000Z Strip a trading system down to one number and that number is expectancy: the average amount you expect to win or lose per trade. Not the win rate, not the size of your biggest winner, not how it felt. Expectancy combines how often you win with how much you win and lose into a single figure that tells you whether the system makes money at all. Traders chase every metric except this one, and a system with a negative expectancy is a machine for converting capital into broker fees no matter how clever the entries look. This is the anchor for the next several articles. ["Why Percent Profitable Is Overrated"](https://aligrithm.com/why-percent-profitable-is-overrated/) and ["Profit Factor, Expectancy, and the Shape of Returns"](https://aligrithm.com/profit-factor-expectancy-and-the-shape-of-returns/) both orbit it, so it is worth getting the arithmetic exactly right first. ## The formula and what each piece does $$ E = \\big(P\_{\\text{win}} \\times \\text{avg win}\\big) - \\big(P\_{\\text{loss}} \\times \\text{avg loss}\\big) $$ Expectancy is the probability of a win times the average winning trade, minus the probability of a loss times the average losing trade. The first term is the money you make from winners weighted by how often they happen; the second is the money you give back to losers weighted by how often they happen. Subtract and you get the average result of a single trade, in dollars or in R-multiples. A positive expectancy means the average trade adds to your equity; a negative one means the average trade subtracts from it. Every other performance metric is a way of looking at one slice of this number; expectancy is the whole thing. ## A worked example, because the formula hides the trade-off Numbers make the point the formula obscures. Take a trend-following system that wins 36% of the time. That sounds like a losing proposition until you put in the sizes. Suppose the average winner is $206 and the average loser is $35. The expectancy is 0.36 times $206 minus 0.64 times $35, which is $74.16 minus $22.40, or $51.76 per trade. The system wins barely a third of the time and makes about $52 on every trade it takes, because the winners are roughly six times the size of the losers. Now flip it. Take a system that wins 70% of the time, a number that feels wonderful, with an average winner of $35 and an average loser of $206, the mirror image. Its expectancy is 0.70 times $35 minus 0.30 times $206, which is $24.50 minus $61.80, or negative $37.30 per trade. It wins twice as often and loses money on average, because the rare losses are large enough to swamp the frequent small wins. The win rate told you the opposite of the truth in both cases. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Why this is the number that decides everything Expectancy is the foundation because it is what you multiply by frequency to get your edge over time. A system's profit per period is roughly its expectancy times the number of trades it takes, so a small positive expectancy traded often can beat a large one traded rarely, and a large negative expectancy traded often is the fastest way to ruin. This is also where position sizing enters: expectancy in R-multiples tells you the edge per unit of risk, and the sizing articles in this pillar turn that edge into how much capital to put behind each trade. Without a positive expectancy there is nothing to size, because betting more on a negative-expectancy system loses faster, not slower. The caveat that protects you: expectancy is estimated from a sample, so the win probability, average win, and average loss are all noisy, and a backtest expectancy computed from a few dozen trades carries enormous uncertainty. A system whose entire positive expectancy rests on two outlier winners has an expectancy you cannot trust, because remove those two trades and the number flips. Compute expectancy on enough trades to mean something, look at how much of it depends on the tails, and treat a fragile expectancy the same way you treat any other overfit result. ## Visualizing expectancy ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_154-visual-0.png) ## KEY POINTS - Expectancy is the average win or loss per trade, the one number that says whether a system makes money. It combines how often you win with how much you win and lose. - The formula is the probability of a win times the average win, minus the probability of a loss times the average loss. Positive means the average trade adds to equity; negative means it subtracts. - A 36% win-rate trend system with a $206 average win and $35 average loss has an expectancy of about +$52 per trade, because winners are six times the size of losers. - The mirror image, a 70% win rate with a $35 average win and $206 average loss, has an expectancy of about -$37 per trade. It wins twice as often and loses money, because rare large losses swamp frequent small wins. - Profit per period is roughly expectancy times trade frequency, so a small positive expectancy traded often can beat a large one traded rarely, and a negative expectancy traded often is the fastest route to ruin. - Expectancy is estimated from a noisy sample. If two outlier winners carry the whole number, you cannot trust it, so compute it on enough trades and check how much rests on the tails. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Systematic Trading - Robert Carver (Amazon)](https://www.amazon.com/s?k=Systematic+Trading+Robert+Carver&ref=aligrithm.com) - [Trading Systems - Urban Jaekle Emilio Tomasini (Amazon)](https://www.amazon.com/s?k=Trading+Systems+Urban+Jaekle+Emilio+Tomasini&ref=aligrithm.com) ### 6.13 When a Drawdown Means the System Is Broken URL: https://aligrithm.com/when-a-drawdown-means-the-system-is-broken/ Last updated: 2026-07-27T13:25:36.000Z Every system draws down, so a drawdown by itself proves nothing. The hard question is whether this drawdown is the normal cost of the strategy or the moment the edge died, and traders get it wrong in both directions: they kill good systems in routine dips and ride dead ones into the ground hoping for a recovery that is never coming. The honest answer comes from comparing the drawdown you are living through against the drawdowns the system should produce if it still works. ["Average Drawdown vs Extreme Drawdown"](https://aligrithm.com/average-drawdown-vs-extreme-drawdown/) built that envelope; this is how you use it as a kill signal. ## The envelope tells you what normal looks like The Monte Carlo permutation distribution from the previous article is not just a capital-planning tool. It is a definition of normal. If the system's edge is intact, the drawdowns it produces should fall inside the distribution of drawdowns its own trades generate when reshuffled. A dip that lands in the body of that distribution, even a painful one near the median or up toward the 95th percentile, is consistent with a working system having a bad run. You expected drawdowns of that depth; one showed up; nothing is broken. The signal that something has changed is a drawdown that exits the envelope. When the depth or the duration runs past what the permutation distribution says should happen, deeper than the 99th percentile, or lasting far longer than any reshuffle of the historical trades produces, the working-system hypothesis is in trouble. A drawdown that the system's own statistics say should almost never happen, happening, is evidence that the trades generating it are no longer drawn from the same distribution as the backtest. The edge that produced the historical trades may be gone. $$ \\text{kill signal: } \\quad \\text{maxDD}\_{\\text{live}} > \\text{DD}\_{99} \\quad \\text{or} \\quad \\text{duration}\_{\\text{live}} > \\text{Dur}\_{99} $$ Set the threshold before the drawdown, not during it. Pick a percentile of the permutation distribution, the 99th depth or the 99th duration, and commit to treating a breach as a reason to investigate or stop. The point of fixing it in advance is to remove your judgment from the moment when your judgment is worst, deep in a losing streak, desperate for the recovery, every bias pulling you to hold. A pre-committed statistical line is the only thing that overrides the desperation. ## Depth alone is not enough, watch duration and character A breach of the depth envelope is the loud signal, and duration is the quiet one that often comes first. A system can stay inside its depth limits while drawing down for far longer than it ever did historically, grinding sideways-to-down for a period the backtest never produced. That extended flat-to-bleeding stretch, even without a record depth, is its own warning, because a working edge should recover within roughly the timeframes its history shows, and one that does not is behaving like a different system. Watch both, and watch the character of the trades from ["Drawdown Is Not Just a Number: It Is a Diagnosis"](https://aligrithm.com/drawdown-is-not-just-a-number-it-is-a-diagnosis/): losses arriving in the conditions that historically favored the system are worse news than the same depth arriving in a regime the system was never built for. The caveat that keeps you honest: the envelope assumes the future trade distribution matches the past, and a breach can mean either the edge died or you simply hit the genuine tail that the permutation underweighted because it broke up loss clustering. You cannot distinguish a dead edge from an unlucky-but-alive one with certainty, which is why a breach triggers investigation and a size reduction rather than instant, permanent shutdown. Cut risk when the line is crossed, dig into whether the edge logic still holds, and decide from the evidence, accepting that you will sometimes kill a system that would have recovered and sometimes hold one that is already dead. The line does not make you right; it makes you act instead of freeze. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Visualizing the kill signal ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_153-visual-0.png) ## KEY POINTS - A drawdown alone proves nothing, since every system draws down. The question is whether it is the normal cost of the strategy or the death of the edge. - The permutation distribution defines normal. Drawdowns inside the body of the distribution, even painful ones near the 95th percentile, are consistent with a working system having a bad run. - The kill signal is a drawdown that exits the envelope: deeper than the 99th percentile or lasting far longer than any reshuffle produces. A drawdown the system's own statistics say should almost never happen is evidence the trades changed distribution. - Set the threshold before the drawdown and commit to it, to remove your judgment from the moment it is worst, deep in a losing streak with every bias pulling you to hold. - Watch duration, not just depth. An extended flat-to-bleeding stretch longer than history ever produced is its own warning, and it often arrives before a record depth. - A breach can mean a dead edge or a genuine tail the shuffle underweighted. You cannot be certain, so a breach triggers a risk cut and investigation, not instant permanent shutdown. The line makes you act instead of freeze. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Systematic Trading - Robert Carver (Amazon)](https://www.amazon.com/s?k=Systematic+Trading+Robert+Carver&ref=aligrithm.com) - [Trading Systems - Urban Jaekle Emilio Tomasini (Amazon)](https://www.amazon.com/s?k=Trading+Systems+Urban+Jaekle+Emilio+Tomasini&ref=aligrithm.com) ### 6.12 Average Drawdown vs Extreme Drawdown URL: https://aligrithm.com/average-drawdown-vs-extreme-drawdown/ Last updated: 2026-07-27T13:25:36.000Z The max drawdown in your backtest is one number from one history, and traders treat it as the worst thing that can happen. It is not. It is the worst thing that did happen in a single sequence of trades that could have fallen in any order, and a different order would have produced a different max. Confusing the drawdown you will live with most of the time, the average drawdown, with the drawdown that defines your survival, the extreme, leads to two opposite mistakes: being spooked by routine dips and being blindsided by the tail you never simulated. ["Drawdown Is Not Just a Number: It Is a Diagnosis"](https://aligrithm.com/drawdown-is-not-just-a-number-it-is-a-diagnosis/) said to read the cause; this is about reading the distribution. ## Two different questions about the same equity curve The average drawdown answers a comfort question: how deep is the typical dip I have to stomach to run this system? It sets your emotional and operational baseline, the drawdowns you will see several times a year and must hold through without flinching. The extreme drawdown answers a survival question: how deep can the dip get before my capital, my risk limits, or my nerve gives out? These are different numbers serving different purposes, and a single backtest reports only one realized path through both. The depth-counting trap is to take the one max drawdown from the backtest and treat it as the extreme. That single number is a sample of size one from a distribution you have not measured. The trades happened to arrive in one particular order; the losers happened to cluster the way they did. Reshuffle the same trades and the max drawdown moves, sometimes a lot, because a drawdown is about the sequence, not just the trades. The realized max is neither the average you will usually feel nor the worst you should plan for. It is one draw. ## Reshuffle the trades to see the distribution The tool that turns one path into a distribution is the Monte Carlo permutation: keep your exact set of trades, shuffle their order thousands of times, and record the max drawdown of each shuffled equity curve. Because you are reordering the same trades without adding or removing any, every permuted curve ends at the identical final profit, but the drawdowns along the way differ, since the shape of the equity curve depends on the order the wins and losses arrive in. From the thousands of shuffles you get a full distribution of max drawdowns rather than a single observation. $$ \\text{DD}\_{95} = \\text{95th percentile of } \\{\\,\\text{maxDD}(\\pi) : \\pi \\in \\text{permutations}\\,\\} $$ Read the planning number off the distribution, not off the backtest. The 95th-percentile max drawdown across the permutations, the depth that only 5% of reorderings exceed, is a far more honest figure for your extreme than the single realized max, which might sit anywhere in the distribution. The median of the distribution gives you a sense of the typical max; the high percentiles give you the tail you size your capital against. Plan your risk limits around the high percentile, not the backtest's lucky or unlucky single draw. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## The honest caveat about the shuffle The permutation assumes the trades are exchangeable, that reordering them is fair, and that assumption has a real hole. Shuffling destroys any serial dependence in the returns, so if your losses genuinely cluster, a bad regime producing a run of losers that the random shuffle breaks apart, the permutation understates the true extreme, because the real world serves the losers in clumps the shuffle never reproduces. The Monte Carlo also assumes the future trade distribution looks like the past one, which fails exactly when a regime shifts. So treat the permutation distribution as a better estimate of the extreme than the single max, and still not the true worst case, because the worst case lives in correlated streaks and regime breaks that no reshuffle of historical trades will show you. Size with margin beyond even the 95th percentile. ## Visualizing the two drawdowns ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_152-visual-0.png) ## KEY POINTS - The backtest's max drawdown is one number from one history, a sample of size one. A different order of the same trades produces a different max, because a drawdown depends on the sequence, not just the trades. - Average drawdown answers a comfort question (the typical dip you stomach several times a year); extreme drawdown answers a survival question (how deep before capital or nerve fails). They are different numbers for different purposes. - The realized max is neither the average you usually feel nor the worst you should plan for. Treating it as the extreme is the depth-counting trap. - Monte Carlo permutation turns one path into a distribution: keep the same trades, shuffle the order thousands of times, record each curve's max drawdown. Every permuted curve ends at the same profit, but the dips differ. - Plan around a high percentile of the drawdown distribution, like the 95th, not the single realized max, which could sit anywhere in the distribution. - The shuffle destroys loss clustering, so if losers genuinely arrive in streaks the permutation understates the true extreme. It also assumes the future looks like the past. Size with margin beyond the 95th percentile. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Systematic Trading - Robert Carver (Amazon)](https://www.amazon.com/s?k=Systematic+Trading+Robert+Carver&ref=aligrithm.com) - [Trading Systems - Urban Jaekle Emilio Tomasini (Amazon)](https://www.amazon.com/s?k=Trading+Systems+Urban+Jaekle+Emilio+Tomasini&ref=aligrithm.com) ### 6.11 Drawdown Is Not Just a Number: It Is a Diagnosis URL: https://aligrithm.com/drawdown-is-not-just-a-number-it-is-a-diagnosis/ Last updated: 2026-07-27T13:25:35.000Z Most traders treat a drawdown as a wound to be bandaged. The equity curve dips, panic sets in, and the reflex is to tighten the stops, optimize the exits, add a filter, anything to stop the bleeding. That reflex treats the symptom and ignores the disease. A drawdown is the system telling you something about what it is doing and what the market is doing to it, and the useful response is to read the message, not to muffle it. Fighting a drawdown by tuning stops is like silencing a smoke alarm instead of looking for the fire. This opens the run of articles on system death in this pillar, and it sits against ["When a Drawdown Means the System Is Broken"](https://aligrithm.com/when-a-drawdown-means-the-system-is-broken/), which draws the line between a drawdown you sit through and one you act on. ## A drawdown has a cause, and the cause is the information Define the thing first. A drawdown is the drop in equity from a peak to the lowest point before a new peak is made, measured as a depth and a duration. The depth and duration are the visible numbers, and they are the least interesting part. What matters is why the equity fell. A trend-following system in a long chop draws down because there are no trends to catch, and the losses are the strategy paying its membership fee while it waits, exactly the behavior you signed up for. A mean-reversion system drawing down in a strong trend is being run over by the thing it bets against, also expected. Those drawdowns are the system working as designed in a hostile regime, and tightening the stops would only make you pay the fee faster. A different drawdown carries a different message. If the losses are coming from trades that should have worked under the system's own logic, fills far worse than modeled, the edge fading in conditions that historically favored it, the drawdown is reporting a problem with the strategy or its execution, not the market. Same depth on the equity curve, completely different diagnosis. You cannot tell which one you have by staring at the depth, because the number is identical in both cases. You have to look at the trades underneath it. ## Optimizing the exits treats the symptom The common error is to respond to any drawdown by reaching for the exit parameters, because exits feel like the lever closest to the loss. Tighten the stop, add a profit target, and the backtest drawdown shrinks. What you have done is fit the exits to the specific losing sequence you just lived through, which is curve-fitting dressed up as risk management. The next drawdown will arrive in a different shape, your tuned exits will not fit it, and you will have spent your robustness budget reducing the depth of a drawdown that already happened. The drawdown was a diagnosis; you treated it as a parameter to minimize, and minimized it on the one sample that no longer matters. Read the drawdown instead. Pull the trades inside it and ask whether they are losses the system is supposed to take in this regime or losses it should have avoided. If they are the expected fee, you hold, because the cure is to wait for the regime the system was built for. If they are unexpected, the exits are the wrong place to look, and you investigate the edge and the execution. The diagnosis decides the treatment, and the depth of the drawdown tells you almost nothing about which treatment you need. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Visualizing the diagnosis ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_151-visual-0.png) ## KEY POINTS - A drawdown is the drop from an equity peak to the trough before the next peak, measured as a depth and duration. The depth is the least informative part; the cause is the information. - The same drawdown depth can mean opposite things. A trend system drawing down in a chop is paying its expected fee while it waits; a system losing trades its own logic says should work is reporting a broken edge. - You cannot diagnose a drawdown from its depth, because the number is identical whether the system is working as designed or failing. You have to read the trades inside it. - Responding to any drawdown by tightening stops or optimizing exits treats the symptom. It curve-fits the exits to the specific losing sequence you just lived through. - The next drawdown arrives in a different shape, so tuned exits will not fit it, and you have spent your robustness budget shrinking a drawdown that already happened. - Diagnose first: are these losses the system is supposed to take in this regime, or losses it should have avoided? Expected fee means hold; unexpected means investigate the edge and execution, not the exits. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Systematic Trading - Robert Carver (Amazon)](https://www.amazon.com/s?k=Systematic+Trading+Robert+Carver&ref=aligrithm.com) - [Trading Systems - Urban Jaekle Emilio Tomasini (Amazon)](https://www.amazon.com/s?k=Trading+Systems+Urban+Jaekle+Emilio+Tomasini&ref=aligrithm.com) ### 6.10 How to Build a Cost-Aware RSI Ranking System URL: https://aligrithm.com/how-to-build-a-cost-aware-rsi-ranking-system/ Last updated: 2026-07-27T13:25:35.000Z RSI is the most abused indicator in retail trading, drawn on charts with 70 and 30 lines and traded as a binary overbought/oversold trigger that costs people money. Used that way it is noise. Used as a cross-sectional ranking signal, with the cost discipline this pillar has been building, it becomes a workable mean-reversion system. This article assembles the whole pipeline from ["Ranking Beats Forecasting for Many Trading Problems"](https://aligrithm.com/ranking-beats-forecasting-for-many-trading-problems/) through ["The Difference Between Signal Quality and Portfolio Quality"](https://aligrithm.com/the-difference-between-signal-quality-and-portfolio-quality/) into one concrete build, so the abstract pieces become a thing you could run. ## Step one: turn RSI into a cross-sectional score Compute RSI on every instrument in your universe over a chosen lookback. Do not trade the raw 0-to-100 value, and do not apply the 70/30 lines. Instead, z-score the RSI cross-sectionally each rebalance, as in ["Why Z-Scoring Makes Ranking Cleaner"](https://aligrithm.com/why-z-scoring-makes-ranking-cleaner/), so each instrument's RSI is expressed in standard deviations relative to the universe that day. A name with an RSI far above the cross-sectional average scores high; a name far below scores low. The z-score makes a high-volatility instrument's RSI comparable to a calm one's, and the cross-sectional framing turns an absolute overbought/oversold reading into a relative one, which is the only version that ranks cleanly. For a mean-reversion system you invert the score: high RSI (recently strong, overbought relative to peers) is the short candidate, low RSI (recently weak, oversold relative to peers) is the long candidate. You are betting the cross-sectional extremes snap back toward the middle. ## Step two: check the shape before you trust the direction Before assuming the relationship is monotone, do the bucket plot from ["When an Alpha Metric Is U-Shaped"](https://aligrithm.com/when-an-alpha-metric-is-u-shaped/). Slice the universe into deciles of the RSI z-score, measure the forward return in each decile, and look. Mean reversion predicts a downward slope, low RSI z-score leads to high forward return. If instead you see a valley or a hump, the naive long/short is mismapped and you fold the metric before ranking. Skipping this step is how people deploy a ranking system that is confidently long the wrong decile. The plot costs nothing and tells you whether the rest of the build is even pointed the right way. ## Step three: rank, slice, and size by risk Sort the universe by the inverted RSI z-score. Long the bottom slice (most oversold relative to peers), short the top slice (most overbought), with slice width chosen from the logic of the signal, narrow if only the extremes mean-revert, wider if the whole cross-section does. Then size each position by volatility, from ["Why Volatility-Adjusted Position Sizing Matters"](https://aligrithm.com/why-volatility-adjusted-position-sizing-matters/), so the calm and wild names contribute equal risk and the book is genuinely neutral rather than dominated by the loudest instrument. $$ w\_i = \\frac{-z\_i}{\\sum\_j |z\_j|} \\times \\frac{\\tau}{\\sigma\_i} $$ The weight on each instrument combines the two ideas: the normalized negative z-score gives the cross-sectional direction and conviction (negative, because we fade the score), and the risk-target-over-volatility term scales by the risk target over the instrument's volatility so each position carries comparable risk. The most oversold, that is, most negative z-score, names get the largest long weights, the most overbought get the largest shorts, and a volatile name gets trimmed relative to a calm one with the same z-score. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Step four: make it cost-aware, then judge it on the net A raw RSI ranking system rebalances constantly, because RSI is a fast, noisy indicator and the ranking churns every period. Without the cost discipline this whole thing is a turnover machine that pays the spread to chase a mean reversion smaller than the cost. Apply the no-trade buffer from ["Cost-Aware Ranking: The Missing Step in Cross-Sectional Strategies"](https://aligrithm.com/cost-aware-ranking-the-missing-step-in-cross-sectional-strategies/) around the slice boundaries so the marginal names stop flickering in and out, and consider slowing the RSI lookback to cut the churn at the source. Then judge the result the way ["The Difference Between Signal Quality and Portfolio Quality"](https://aligrithm.com/the-difference-between-signal-quality-and-portfolio-quality/) demands: report the signal's rank correlation and breadth, and separately the portfolio's Sharpe, drawdown, and turnover after realistic costs. An RSI ranking system that looks great gross and dies net is the default outcome, not the exception, so the cost-adjusted number is the only one that decides whether you trade it. A blunt caveat on expectations: cross-sectional RSI mean reversion is a real but thin edge, crowded and cost-sensitive, and the honest version of this build often clears costs by a modest margin or not at all depending on the universe. The value of the pipeline is not that RSI is special; it is that the same four steps, score, check the shape, rank and risk-size, control costs, apply to any indicator you want to turn into a cross-sectional system. ## Visualizing the pipeline ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_150-visual-0.png) ## KEY POINTS - RSI as a 70/30 binary trigger is noise. As a cross-sectional ranking signal with cost discipline it becomes a workable mean-reversion system. - Step one: compute RSI per instrument, then z-score it cross-sectionally each rebalance so readings are comparable across volatile and calm names. Invert it for mean reversion: low RSI is the long candidate, high RSI the short. - Step two: bucket the universe into deciles of the RSI z-score and plot forward return per decile. Confirm the downward mean-reversion slope before trusting the direction, and fold the metric if you see a valley or hump. - Step three: sort by the inverted score, long the oversold slice, short the overbought slice, and size every position by volatility so each contributes equal risk. The weight combines normalized negative z-score with a volatility scaler. - Step four: apply a no-trade buffer around the slice boundaries and consider a slower RSI lookback, because raw RSI ranking churns and pays the spread to chase a tiny mean reversion. - Judge the system on net performance after realistic costs, reporting signal rank correlation and breadth separately from portfolio Sharpe and turnover. Looking great gross and dying net is the default outcome. The four-step pipeline generalizes to any indicator. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Systematic Trading - Robert Carver (Amazon)](https://www.amazon.com/s?k=Systematic+Trading+Robert+Carver&ref=aligrithm.com) - [Trading Systems - Urban Jaekle Emilio Tomasini (Amazon)](https://www.amazon.com/s?k=Trading+Systems+Urban+Jaekle+Emilio+Tomasini&ref=aligrithm.com) ### 6.1 Ranking Beats Forecasting for Many Trading Problems URL: https://aligrithm.com/ranking-beats-forecasting-for-many-trading-problems/ Last updated: 2026-07-27T13:25:30.000Z Most traders reach for a forecast when the problem only needs an ordering. They build a model to predict that copper returns 4.2% over the next month, sweat over whether it is 4.2% or 3.1%, and then trade off a number that was never stable to begin with. For a large class of strategies you do not need the number at all. You need to know that copper looks better than gold this month, and gold looks better than silver. That is a ranking problem, and ranking is both easier to get right and harder to break than point forecasting. The distinction runs through the rest of this pillar, so it is worth nailing down before ["Ranked Long/Short Systems Explained"](https://aligrithm.com/ranked-long-short-systems-explained/) builds the machinery on top of it. ## A point forecast asks for more than you can deliver To forecast an absolute return you have to be right about two things: the direction and the magnitude. The magnitude is where forecasts go to die. Expected returns are tiny relative to the noise around them, so your estimate of "4.2%" carries a standard error wide enough to swallow the whole signal. You are committing to a precise number that the data cannot support, and every downstream sizing decision inherits that false precision. Ranking throws the magnitude away on purpose. You are not claiming copper returns 4.2%; you are claiming copper sits above gold in this month's ordering. That claim survives even when your absolute return model is badly miscalibrated, because a monotone error that shifts every estimate up or down leaves the ordering intact. The information you actually trade on is the part of the forecast that was robust anyway. ## Why the ordering is the stable part Consider what happens across a regime change. Volatility doubles, the dispersion of returns blows out, and every point forecast you made last month is now the wrong scale. A model trained to output return magnitudes is quoting numbers from a distribution that no longer exists. The cross-sectional ordering, though, tends to persist: the cheap-carry instrument is still cheaper-carry than the expensive one, the strong-momentum instrument still has stronger momentum than the weak one, even as the absolute numbers move. $$ \\rho\_s = 1 - \\frac{6 \\sum\_{i=1}^{n} d\_i^2}{n(n^2 - 1)} $$ This is the Spearman rank correlation, the natural way to score a ranking model. The d terms are the differences between each instrument's predicted rank and its realized-return rank, n is the number of instruments, and the score runs from -1 to +1\. It only cares about order, not distance, so a model that nails the ordering but mangles the magnitudes scores a perfect 1\. That is the right scoring rule for a ranking system, and the fact that it ignores magnitude is the whole point: you are grading the part of the forecast you can actually trust. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## When ranking is not enough Ranking is the right tool when you trade a cross-section, a basket where you can be long the top and short the bottom and you only care about relative performance. It is the wrong tool when the absolute level matters. A single-instrument timing system that decides whether to be in or out of the S&P cannot rank one thing against nothing; it needs a real forecast of whether the move is worth the cost. A volatility-targeting overlay needs the magnitude of expected return to size the bet, which is why ["Why Volatility-Adjusted Position Sizing Matters"](https://aligrithm.com/why-volatility-adjusted-position-sizing-matters/) treats sizing as a separate problem from selection. Ranking solves selection cleanly and says nothing about how hard to bet, so you pair it with a sizing layer rather than asking it to do both. ## Visualizing rank vs magnitude ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_141-visual-0.png) ## KEY POINTS - A point forecast commits you to direction and magnitude. Magnitude is where forecasts fail, because expected returns are tiny next to the noise around the estimate, so the precise number carries a standard error wide enough to swallow the signal. - Ranking discards magnitude on purpose. You claim only that one instrument sits above another, and that claim survives a miscalibrated return model, since a monotone error leaves the ordering intact. - The cross-sectional ordering is the stable part across regime changes. When volatility doubles, point forecasts are the wrong scale, but the relative ordering of carry, momentum, or value tends to persist. - Score a ranking model with Spearman rank correlation, which grades order and ignores distance. A model that nails the ordering but mangles the magnitudes still scores a perfect 1. - Ranking is for cross-sectional selection, where you go long the top and short the bottom and only relative performance matters. It says nothing about how hard to bet. - Pair ranking with a separate volatility-based sizing layer. Use a real forecast only where the absolute level matters, like single-instrument timing. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Systematic Trading - Robert Carver (Amazon)](https://www.amazon.com/s?k=Systematic+Trading+Robert+Carver&ref=aligrithm.com) - [Trading Systems - Urban Jaekle Emilio Tomasini (Amazon)](https://www.amazon.com/s?k=Trading+Systems+Urban+Jaekle+Emilio+Tomasini&ref=aligrithm.com) ### 5.21 Cross-Exchange Fair Value for Crypto Perps URL: https://aligrithm.com/cross-exchange-fair-value-for-crypto-perps/ Last updated: 2026-07-27T13:25:14.000Z Quote a perpetual on a mid-size exchange using that exchange's own mid price as your fair value, and you have started from a bad number. Crypto is fragmented across dozens of venues that do not move together, and the smaller venues follow the larger ones with a lag. A market maker on a follower exchange who treats its local mid as the truth is quoting around a stale price, and every fast trader who saw the leader move first picks them off. The fix is to build fair value from the whole cross-exchange picture, with the dominant venue as the leader that pulls the laggards. This article extends ["Why Fair Value Is the Core of Market Making"](https://aligrithm.com/why-fair-value-is-the-core-of-market-making/) to the multi-venue reality of crypto perpetuals, the hardest and most important place to get the center right. ## The leader-laggard structure Crypto venues form a hierarchy. The largest, Binance for most pairs, leads, and the smaller exchanges lag, tending to drift toward wherever the leader's price went. So the leader's price is information about where the laggard's price is about to go, and a market maker on a laggard venue should treat the leader as a forecast of their own near-future mid, not as a separate market. In a simplified model, take the Binance mid price as a proxy for the globally weighted mid, which works because its market share is large enough that its mid is close to the volume-weighted consensus. Then quantify the lead-lag: regress the basis, the difference between the Binance mid and your exchange's mid, against your exchange's future mid-price changes. $$ \\Delta \\text{mid}^{\\text{local}}\_{t \\to t+\\tau} = \\beta \\cdot \\big( \\text{mid}^{\\text{Binance}}\_t - \\text{mid}^{\\text{local}}\_t \\big) + \\varepsilon $$ The future change in your local mid over horizon tau is proportional, with coefficient beta, to the current basis between the Binance mid and your local mid. The basis is the gap, and the regression says your local price will close that gap, moving toward Binance. This relationship is highly predictive, because most smaller exchanges follow Binance, so the basis is a clean lead signal you fold straight into your local fair value: when Binance trades above your local mid, lean your fair value up, and when below, lean it down. ## The full perp fair value The simplified Binance-mid proxy is a starting point. A proper perpetual fair value blends several venue-level prices, because no single number captures it. The inputs a desk combines: the globally volume-weighted mid across exchanges, the globally weighted index price, the local index price, the local mid price, and the stablecoin rates needed to convert between contracts denominated in different quote currencies. The global mid leads and pulls the laggard exchanges, so you weight it heavily when forecasting where local prices will go. The local mid and local index anchor where you actually trade. The stablecoin rates matter because a perp quoted in USDT and a perp quoted in USDC are not directly comparable until you account for the USDT-USDC rate, and ignoring that conversion misprices the cross-venue basis. Combining these gives a fair value that respects both the global leader and the local microstructure of the venue you quote on. Layer the standard predictive factors on top of the cross-exchange basis: funding rates, the lead-lag between spot and perp, volatility, rate of trading, book imbalance from ["Order Book Imbalance: The First Microstructure Feature to Test"](https://aligrithm.com/order-book-imbalance-the-first-microstructure-feature-to-test/), and momentum. Each adds information about where the price is heading, and you tune their weights over time as you learn how the price evolves. The cross-exchange basis is usually the single strongest of these on a follower venue, because it is the most direct read on the leader's information advantage. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Spot has extra constraints The spot case is the same problem with two added constraints. Spot fair value must respect triangular arbitrage bounds, the requirement that the prices of related pairs, say a coin against USDT, against USDC, and the stablecoins against each other, stay consistent or an arbitrage opens. And it must respect cross-exchange arbitrage bounds, the requirement that the same spot asset cannot diverge too far across venues without inviting arbitrage. These bounds constrain how far your fair value can drift from the cross-venue consensus, because a fair value that violates them is a fair value the arbitrageurs will trade against you. For perps the funding mechanism and the index construction play a similar anchoring role, tying the perp to its underlying. ## The limits Cross-exchange fair value is the strongest center on a follower venue, and it carries real fragilities. The lead-lag relationship is non-stationary, so the beta drifts and the regression must be re-estimated continuously, and a stale beta quotes around a relationship that no longer holds. Latency decides whether the edge survives: the whole signal is the leader moving before the laggard, so if your feed from the leader is slow you are the laggard being picked off rather than the maker exploiting the lag, and the cross-exchange game is partly a latency race you can lose. The leader itself can be wrong during a venue-specific event, a Binance outage or a local liquidation cascade, when the usual hierarchy inverts and the follower leads for a moment. And the stablecoin rates that tie the contracts together can themselves dislocate, a USDC depeg being the obvious case, which breaks the conversion the whole fair value rests on. The cross-exchange center is the right one to build, and it is a moving, latency-sensitive estimate, not a constant truth. ## Visualizing cross-exchange fair value ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_140-visual-0.png) ## KEY POINTS - Quoting a perp on a mid-size venue using its own mid as fair value starts from a bad number, because crypto is fragmented and smaller venues follow the larger ones with a lag. - Crypto venues form a leader-laggard hierarchy. The dominant venue (Binance for most pairs) leads, and its mid proxies the globally weighted mid, so it forecasts the laggard's near-future price. - Quantify the lead-lag by regressing the basis (Binance mid minus local mid) against the local mid's future change. It is highly predictive, so lean your local fair value toward the leader. - A full perp fair value blends global weighted mid, global index, local index, local mid, and stablecoin rates (needed to compare USDT- and USDC-denominated contracts), weighting the leader heavily. - Layer standard factors on top: funding rates, spot-perp lead-lag, volatility, trading rate, book imbalance, and momentum, tuning weights over time. The cross-exchange basis is usually the strongest on a follower venue. - Spot adds triangular and cross-exchange arbitrage bounds that constrain how far fair value can drift from consensus. Perps are anchored by funding and the index. - Limits: the beta is non-stationary and needs continuous re-estimation; the edge is a latency race you can lose; the leader can invert in a venue-specific event; and the stablecoin rates can depeg, breaking the conversion. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [The Art of Currency Trading - Brent Donnelly (Amazon)](https://www.amazon.com/s?k=The+Art+of+Currency+Trading+Brent+Donnelly&ref=aligrithm.com) ### 5.20 Dynamic Symbol Selection for Market Makers URL: https://aligrithm.com/dynamic-symbol-selection-for-market-makers/ Last updated: 2026-07-27T13:25:13.000Z A market maker does not have to quote everything, and quoting the wrong names is how an otherwise sound system bleeds. Some symbols hand you good markouts and some feed you toxic flow all day, and which is which changes over time. The cheapest, highest-leverage improvement to a market making book is often not a better forecast or a tighter spread, it is refusing to quote the symbols where you have no edge right now. Dynamic selection of what to quote turns a system that lacks edge into a flat or profitable curve. This article applies the markout diagnostic from ["Markouts: The Truth Serum of Market Making"](https://aligrithm.com/markouts-the-truth-serum-of-market-making/) as a live filter on which instruments to quote, and as a control on how much size to put on each. ## Measure markouts before you quote You can judge a symbol's toxicity without ever posting a quote in it. Take the average markout, over a short horizon like five seconds, of every single trade that happens in the symbol, and track its EWMA over time. The markout of a trade is the price change after it, the same measure used on your own fills, applied here to the whole market's trades rather than just yours. $$ \\text{toxicity score}\_t = \\text{EWMA}\\big( \\text{markout}\_{5s} \\text{ of every trade} \\big) $$ The toxicity score is an exponentially weighted moving average of the five-second markout of every trade in the symbol. A symbol where trades are consistently followed by the price moving against the aggressor is non-toxic, the takers there have no edge, so quoting against them is safe. A symbol where trades are consistently followed by the price continuing in the aggressor's direction is toxic, the takers there are informed, and quoting against them bleeds. Because the score is an EWMA of market-wide trades, you get it before committing capital, and it updates continuously as conditions shift. ## Quote the non-toxic names The names that sustain genuinely good markouts are the super non-toxic pairs, stablecoin pairs like USDC against USDT being the cleanest example, where almost no one trading carries information and the flow is mechanical. Those are the symbols to quote. The dynamic part is that toxicity is not a fixed property of a symbol, it drifts, so you quote a name when its score says non-toxic and pull quotes when the score deteriorates, rather than quoting a fixed list forever. This is why selection alone can rescue a weak system. A forecast that is marginal on average is often marginal because it is mixing profitable fills on clean names with losing fills on toxic ones. Filter to only quote when the markout score says you have an edge, and the same forecast that looked flat across everything turns profitable on the subset where it works. You are not improving the alpha, you are stopping it from being spent on flow that was always going to take it back. ## Scale size by markout, too Selection is binary, quote or do not. The same signal also drives a continuous control: sizing. The default is a fixed size per instrument, but the advanced version scales your size by your markouts for that symbol, so you quote more on the names you are making money on and less on the names you are not. Markouts are responsive, which makes this safe. When your markouts on a symbol start deteriorating, the size scales down quickly, cutting your exposure to a name that is turning toxic before it does serious damage. The combination is a soft, continuous version of selection: full size on clean names, shrinking size as markouts worsen, and quotes pulled entirely when the score crosses into toxic. The book concentrates capital where it earns and starves the flow that bleeds, all driven off the one markout signal. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## The limits Selection is powerful and it narrows your universe, which is the trade-off. The cleanest non-toxic names, stablecoin pairs and the like, are also the ones with the thinnest spreads and the most competition, so the markout edge there is real but small, and a book that only quotes the safest names earns a small, safe number. Pushing for more requires quoting riskier names where the markout score is borderline, which reintroduces the toxicity you were filtering out. The EWMA also lags, so a name that turns toxic suddenly will bleed for the window it takes the score to react, and the responsiveness that protects you is bounded by how fast the average moves. Dynamic selection is a real edge and a defensive one, it improves the curve mostly by avoiding losses, not by manufacturing new alpha. ## Visualizing selection ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_139-visual-0.png) ## KEY POINTS - A maker should not quote everything. Quoting names where you have no edge is how a sound system bleeds, and refusing them is often the cheapest, highest-leverage improvement. - Judge a symbol before quoting it: track the EWMA of the short-horizon (five-second) markout of every market trade in the symbol. Trades followed by price continuing in the aggressor's direction mark toxic flow. - The non-toxic names that sustain good markouts (stablecoin pairs like USDC/USDT are cleanest) are the ones to quote, because the takers there carry no information. - Toxicity drifts, so selection is dynamic: quote a name when its score says clean, pull quotes when it deteriorates, rather than holding a fixed list. - Selection alone can rescue a weak system by separating profitable fills on clean names from losing fills on toxic ones, turning a flat forecast profitable on the subset where it works. - The same markout signal drives sizing: scale size up on names you make money on and down on names you do not. Markouts are responsive, so size cuts quickly as a name turns toxic. - Limits: the cleanest names have the thinnest spreads and most competition, so the safe edge is small; reaching for more reintroduces toxicity; and the EWMA lags, so a sudden turn bleeds until the score reacts. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [The Art of Currency Trading - Brent Donnelly (Amazon)](https://www.amazon.com/s?k=The+Art+of+Currency+Trading+Brent+Donnelly&ref=aligrithm.com) ### 5.19 Maker vs Taker Edge: Same Signal, Different Economics URL: https://aligrithm.com/maker-vs-taker-edge-same-signal-different-economics/ Last updated: 2026-07-27T13:25:13.000Z Take one signal, an RSI-based ranking, a momentum tilt, an order-flow lean, whatever, and hand it to two desks. One trades it as a taker, crossing the spread to enter. One trades it as a maker, posting passive quotes. The signal is identical, the predicted move is identical, and the two desks end up with completely different businesses. The taker needs a strong signal to overcome cost and trades a small universe at low frequency. The maker thrives on weak signals, trades a huge universe at high frequency, and earns the spread the taker pays. Same alpha, opposite economics. This article runs the comparison that ["Why Small Alphas Matter More for Makers Than Takers"](https://aligrithm.com/why-small-alphas-matter-more-for-makers-than-takers/) set up, by walking a single concrete signal through both roles. ## The same signal, two strategies Start with one alpha: rank a universe of assets by a metric, go long the top decile and short the bottom, the ranked long-short structure. Size by volatility, normalize, rebalance. The taker version crosses the spread to take each position. The maker version posts passive quotes skewed toward the same ranking, leaning the bid down on the names it wants long and the ask up on the names it wants short, so it accumulates the position by being filled rather than by lifting. The predicted edge per name is the same number in both. What differs is everything that happens around the predicted edge: the cost of entry, the frequency you can run it, and the breadth of the universe you can apply it to. ## Cost: the spread is the hinge The taker pays the spread plus fees on entry, so the signal's edge must clear that cost or the trade loses. The maker collects the spread, so the same edge sits on top of spread income. The entire difference in economics hinges on this sign flip, worked through in the previous article: a cost for one desk, a revenue for the other. The consequence on strength: a taker needs a strong signal, several basis points of edge, to clear the wall. The maker needs almost nothing, because the spread capture does the heavy lifting and the signal is a small additional tilt. A signal worth 0.5 bps is a taker's loser and a maker's tiny winner, the same 0.5 bps in both calculations. ## Frequency and universe The cost flip cascades into two more differences. Frequency: because each taker round trip pays the spread, the taker must hold long enough for the move to clear the cost, which caps how often it can trade. The maker pays no entry cost, so it can churn at the microstructure frequency, rebalancing on every quote-level update, harvesting tiny moves the taker cannot afford to chase. The maker's natural frequency is far higher because nothing penalizes turnover. Universe: the taker is pushed toward liquid names where spreads are tight enough to clear, while the maker is pulled toward the illiquid names where spreads are wide, because a wide spread is more income to capture and the smaller, less efficient names carry more edge per signal. The two desks want opposite ends of the liquidity spectrum from the same signal. The maker expanding its universe makes the same alphas perform better, the breadth amplifier from the small-alphas article, while the taker's universe shrinks to where costs are survivable. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## They are mirror images, and the maker is the adverse selector's prey The cleanest way to see the relationship: the maker and taker are two sides of the same fill. When the taker crosses to express the signal, someone makes the other side, and if that someone is a maker without the signal, the maker is being adversely selected, this is the adverse selection from ["Adverse Selection Explained for Traders"](https://aligrithm.com/adverse-selection-explained-for-traders/). So the maker does not just want weak signals because they are cheap to trade, the maker needs them to avoid being the uninformed side of the informed taker's trade. A maker running the same signal as the aggressive takers is no longer their prey, it is shading its quotes away from exactly the side they are about to hit. This closes the loop of the pillar. The maker's edge is not a different kind of alpha from the taker's, it is the same alpha plus the spread, minus the adverse selection, run across a wider universe at higher frequency. Identical signal, mirror-image business, and the spread is the line between them. ## Visualizing the two economics ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_138-visual-0.png) ## KEY POINTS - Hand one signal to a taker and a maker and you get two different businesses. The predicted edge is identical; the cost, frequency, and universe around it are opposite. - Run a ranked long-short the same way: the taker crosses the spread to take positions, the maker posts passive quotes skewed toward the same ranking and accumulates by being filled. - The spread is the hinge. The taker pays it on entry and needs a strong signal to clear the cost; the maker collects it and needs almost nothing, since spread capture does the heavy lifting. - A 0.5 bps signal is a taker's loser and a maker's tiny winner, the same number in both calculations. - Frequency cascades from cost: the taker must hold long enough to clear the spread, capping turnover; the maker pays no entry cost and churns at microstructure frequency. - Universe diverges too: the taker is pushed to liquid, tight-spread names; the maker is pulled to illiquid, wide-spread names where spread income and per-signal edge are larger. - Maker and taker are two sides of the same fill. A maker without the signal is the adversely-selected prey of the informed taker, so the maker needs the signal to shade quotes away from the side the taker will hit. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [The Art of Currency Trading - Brent Donnelly (Amazon)](https://www.amazon.com/s?k=The+Art+of+Currency+Trading+Brent+Donnelly&ref=aligrithm.com) ### 5.18 Why Small Alphas Matter More for Makers Than Takers URL: https://aligrithm.com/why-small-alphas-matter-more-for-makers-than-takers/ Last updated: 2026-07-27T13:25:12.000Z A signal that predicts the next move with half a basis point of edge is garbage to a taker. Cross the spread, pay the fees, and the cost swamps the edge before you start. The same signal in the hands of a market maker is money, because the maker is collecting the spread instead of paying it, so the cost that killed the signal for the taker is not just absent, it is reversed into income. The economics of who pays and who earns the spread is the whole reason small alphas matter far more on the maker side. This article works the cost arithmetic that ["Adverse Selection Explained for Traders"](https://aligrithm.com/adverse-selection-explained-for-traders/) leaned on when it said a maker does not need to predict big moves. It sets up ["Maker vs Taker Edge: Same Signal, Different Economics"](https://aligrithm.com/maker-vs-taker-edge-same-signal-different-economics/), which runs the same signal through both desks. ## The cost wall a taker faces A taker pays to enter every position: the spread crossed plus the fee. For the trade to make money, the predicted edge must clear that wall. $$ \\text{net edge}\_{\\text{taker}} = \\text{signal edge} - \\text{spread} - \\text{fee} $$ The taker's net edge is the raw signal edge minus the spread minus the fee. If the signal predicts 0.5 bps and the round-trip cost is 4 bps, the net is negative 3.5 bps, a guaranteed loser no matter how often the signal is right. The signal has to clear several basis points of cost just to break even, which throws out the entire population of weak, high-frequency edges, the small structural tilts that predict the next move by a sliver. For a taker, those are uninvestable. ## The maker earns what the taker pays The maker expresses a view by posting passive quotes, so the maker is on the other side of that cost. Instead of paying the spread, the maker collects it on each fill. Having zero cost to trade, earning the spread rather than paying it, makes it suddenly easy to find an edge, because the break-even bar drops to the floor. $$ \\text{net edge}\_{\\text{maker}} = \\text{signal edge} + \\text{spread captured} - \\text{adverse selection cost} $$ The maker's net edge is the signal edge plus the spread captured minus the adverse-selection cost from the fills the informed pick off. The spread term flipped sign relative to the taker: what was a cost is now income. A 0.5 bps signal that was hopeless for a taker becomes, for a maker, a small tilt layered on top of spread capture, and the only real cost left is adverse selection, which the signal itself helps reduce by shading quotes away from the informed side. The signal does not have to clear a cost wall, it only has to beat the adverse selection it offsets. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Why this expands the investable universe The consequence is that the maker's universe of usable signals is enormous compared to the taker's. Every weak edge that predicts the next move by a fraction of a basis point, uninvestable for anyone who pays the spread, is investable for the maker. This is why the large desks run thousands of small alphas rather than a few strong ones. None of them would survive transaction costs as a standalone taker strategy. Skewed into passive quotes and ensembled, they stack into the bulk of a market making book's PnL, the positional market making from ["Market Making Is Not Just Collecting the Spread"](https://aligrithm.com/market-making-is-not-just-collecting-the-spread/). The amplifier is universe size. With zero cost to trade, expanding the set of instruments you quote makes the same alphas perform better, because the smaller and less efficient names carry more edge per signal, and you can afford to harvest edge there that a taker paying spreads on illiquid books never could. Small alphas times many instruments times zero entry cost is the maker's structural advantage. ## The catch: adverse selection is the real cost The maker's break-even bar is low, not zero, and the thing that fills the gap is adverse selection from ["Toxic Flow vs Inventory Risk"](https://aligrithm.com/toxic-flow-vs-inventory-risk/). The spread is not free income, it is payment for providing liquidity, and the informed traders who pick off your quotes are the cost of earning it. A small alpha pays for a maker only when its edge plus the spread it captures exceeds the adverse selection on those fills. So the maker is not escaping cost, the maker is trading the taker's explicit spread-and-fee cost for an implicit adverse-selection cost, and the small alphas are valuable precisely because they reduce that implicit cost while the spread capture covers the rest. Get the adverse-selection defense wrong and the thousands of small alphas bleed through the same hole. ## Visualizing the cost flip ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_137-visual-0.png) ## KEY POINTS - A weak signal worth half a basis point is garbage to a taker, who pays the spread and fees on entry, but money to a maker, who collects the spread instead of paying it. - The taker's net edge is signal minus spread minus fee. A 0.5 bps signal against a 4 bps cost is a guaranteed loser, so the whole population of weak high-frequency edges is uninvestable for takers. - The maker's net edge is signal plus spread captured minus adverse selection. The spread term flips from cost to income, so the break-even bar drops to the floor. - A signal that was hopeless for a taker becomes a small tilt on top of spread capture for a maker, and it only has to beat the adverse selection it offsets, not clear a cost wall. - This is why desks run thousands of small alphas: none survive transaction costs as standalone taker strategies, but skewed into passive quotes and ensembled they stack into the bulk of the book's PnL. - Universe size amplifies it: with zero entry cost, expanding the instruments you quote makes the same alphas perform better because smaller, less efficient names carry more edge. - The catch: the maker trades explicit spread-and-fee cost for implicit adverse-selection cost. Small alphas pay only when edge plus spread beats adverse selection, so getting the defense wrong bleeds all of them. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [The Art of Currency Trading - Brent Donnelly (Amazon)](https://www.amazon.com/s?k=The+Art+of+Currency+Trading+Brent+Donnelly&ref=aligrithm.com) ### 5.17 TWAP and VWAP Are Execution Models, Not Just Indicators URL: https://aligrithm.com/twap-and-vwap-are-execution-models-not-just-indicators/ Last updated: 2026-07-27T13:25:11.000Z Most charting packages list TWAP and VWAP next to the moving averages, as if they were one more line to cross over for a buy signal. That framing misses what they are. TWAP and VWAP are execution benchmarks and the algorithms built to hit them, the machinery a desk uses to push a large order into the market without moving it. Treat VWAP as the most important indicator in quant finance only once you see it as the yardstick execution is measured against, not as a squiggle to trade. This article reframes two numbers retail traders use as oscillators. It connects to the maker side of this pillar, because the flip side of an execution algorithm working an order is the market maker on the other side of those slices. ## The two benchmarks TWAP is the time-weighted average price, the simple average of price over a period, every observation counted equally. $$ \\text{TWAP} = \\frac{1}{N} \\sum\_{i=1}^{N} P\_i $$ The TWAP over a period is the sum of the prices divided by the number of observations N. It weights every moment of the period equally, ignoring how much volume traded at each price. VWAP is the volume-weighted average price, each price weighted by the volume that traded at it. $$ \\text{VWAP} = \\frac{\\sum\_i P\_i V\_i}{\\sum\_i V\_i} $$ The VWAP is the sum of price times volume over the sum of volume, so a price where a lot of traded changes the average more than a price where little traded. VWAP answers the question execution actually cares about: what was the average price the market transacted at, weighted by where the trading happened. ## They are execution algorithms The reason these are execution models is that each benchmark has a matching algorithm designed to achieve it. A TWAP execution buys a fixed quantity every fixed interval: split a large parent order into equal child slices spaced evenly in time, and your average fill price tracks the time-weighted average. To hide the schedule from anyone watching, randomize the slice size so the pattern is not a clean metronome an observer can front-run. A VWAP execution makes the slice size depend on volume instead of time: trade more when the market trades more, less when it trades less, so your participation tracks the volume profile and your average fill lands near the VWAP. Over the execution window you end up at an average price of roughly the VWAP. The algorithm is leaning the size toward the high-volume parts of the period, because matching the volume profile is what keeps your average at the benchmark. The point of both: a desk with a large order to fill cannot just cross the spread for the whole size without moving the price against itself. Slicing the order to track TWAP or VWAP spreads the impact across time or volume, so the execution lands near a fair average rather than at a price the order itself blew out. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Why a market maker cares For the market maker side of this pillar, execution algorithms are the counterparty. A VWAP algorithm working a large buy is a stream of taker buy slices arriving in proportion to volume, which shows up in your trade flow as persistent signed pressure, the autocorrelated buying from ["Using Trade Flow to Predict Short-Term Price Movement"](https://aligrithm.com/using-trade-flow-to-predict-short-term-price-movement/). Recognizing execution flow matters because it is comparatively non-toxic: an algorithm mechanically working a parent order is not trading on a short-horizon price forecast against you, so the flow it generates is less likely to be the toxic, adversely-selecting flow from ["Toxic Flow vs Inventory Risk"](https://aligrithm.com/toxic-flow-vs-inventory-risk/). This is why distinguishing execution flow from informed flow is part of the maker's job. The same buy pressure can come from a trader who knows the price is about to jump (toxic, widen against it) or from a VWAP algorithm indifferent to the next tick (benign, you can provide liquidity into it and hold the inventory). Reading which is which, partly from trade size and pattern, is how a maker decides whether a stream of one-sided flow is a threat or a customer. ## The limits TWAP and VWAP as benchmarks have known weaknesses. VWAP is a backward-looking number, the true VWAP of a period is only known after the period ends, so a live VWAP algorithm forecasts the remaining volume profile and is wrong when the profile shifts. Both are gameable: a participant who knows a large VWAP order is working can trade ahead of its predictable slices. And hitting the benchmark is not the same as good execution, since the benchmark itself can be a bad price if the whole period drifted against you. As indicators to trade, a price crossing its VWAP carries far less information than the order-flow features in this pillar; their real value is as the language execution speaks, not as another oscillator. ## Visualizing the two algorithms ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_136-visual-0.png) ## KEY POINTS - TWAP and VWAP are execution benchmarks and the algorithms built to hit them, not indicators to cross over. They are the machinery for pushing a large order into the market without moving it. - TWAP is the equal-weighted average price over a period. VWAP weights each price by the volume traded there, answering what average price the market actually transacted at. - A TWAP algorithm buys a fixed quantity every fixed interval, randomizing slice size to hide the schedule. A VWAP algorithm sizes slices by volume, trading more when the market trades more, landing near the VWAP. - Both exist because a desk cannot cross the spread for a whole large order without moving price against itself. Slicing spreads the impact across time or volume. - For a maker, an execution algorithm is the counterparty: a VWAP buy shows up as persistent autocorrelated buy pressure in trade flow, and it is comparatively non-toxic because the algo is not trading a short-horizon forecast against you. - Distinguishing execution flow from informed flow (partly via trade size and pattern) decides whether one-sided flow is a threat to widen against or a customer to provide liquidity into. - Limits: true VWAP is known only after the period, so live algos forecast the remaining profile; both are gameable by anyone who knows the order is working; and hitting the benchmark is not the same as a good price. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [The Art of Currency Trading - Brent Donnelly (Amazon)](https://www.amazon.com/s?k=The+Art+of+Currency+Trading+Brent+Donnelly&ref=aligrithm.com) ### 5.16 Fill Probability from Trade Size CDFs URL: https://aligrithm.com/fill-probability-from-trade-size-cdfs/ Last updated: 2026-07-27T13:25:11.000Z A market maker's quote has two unknowns: how much you make if it fills, and how likely it is to fill. The first is easy, it is the distance from your quote to the fair value. The second is the hard one, and getting it wrong is what makes a desk quote too wide and never trade, or too tight and barely profit. The clean way to estimate fill probability uses the distribution of taker order sizes, the cumulative distribution function of how big incoming market orders tend to be, read against how much liquidity sits in front of your order. This article builds the fill-probability estimate that the EV-maximization quoting model depends on. It pairs with the improved market making strategy that optimizes expected PnL times probability of fill. ## The quote is an expected-value problem Quoting is maximizing profit per unit time, which is expected value. The profit side is simple: your PnL on a fill is the distance from your quote to the mid (or wherever you can exit), minus fees. $$ \\text{PnL}\_{\\text{buy}} = \\text{distance}\_{\\text{buy}} - \\text{fee}, \\qquad \\text{EV} = \\text{PnL} \\cdot P\[\\text{filled}\] $$ The PnL on a buy quote is its distance from fair value minus the fee, and the expected value of the quote is that PnL times the probability it fills. Quote too wide and the PnL per fill is large but the fill probability collapses to near zero. Quote too tight and you fill constantly but the PnL per fill is thin. The EV product peaks somewhere in between, and finding that peak requires the fill probability, the only hard term in the expression. ## Estimating fill probability from the size CDF The fill-probability estimate runs on the statistics of taker order sizes. The procedure has a few steps. Collect historical trades data over some window, an hour works. Build the cumulative distribution function of taker order sizes from that data, which tells you the probability that an incoming market order is at least any given size. Then read your fill probability off the CDF: the probability that the next trade is large enough to clear all the liquidity resting between the touch and your order, plus your own order size. $$ P\[\\text{fill on next trade}\] = P\[\\text{taker size} \\geq q\_{\\text{ahead}} + q\_{\\text{own}}\] $$ Your order fills on the next trade if that trade's size is at least the quantity resting ahead of you in the queue plus your own order size, and the CDF gives that probability directly. The model assumes one big order wipes out everything in front of you and fills you in the same sweep, which is the simplifying assumption that makes the estimate tractable. The deeper you place, the more liquidity sits ahead of you, the larger the order needed to reach you, and the lower the probability the CDF returns. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## From one-trade probability to time Fill probability per trade is not yet what the EV needs, because EV is per unit time. Convert it. Count the total number of takers per minute, which gives the arrival rate of trades. With the per-trade fill probability from the CDF and the trade arrival rate, compute the average number of trades you must wait for before one is large enough to fill you, then convert that count of trades into an expected wait time using the arrival rate. Now you have an expected time to fill, which turns the EV into profit per unit time and lets you compare quotes at different depths on the same footing. The full quoting model optimizes the PnL times fill probability on both sides of the book independently, a buy PnL with a buy fill probability and a sell PnL with a sell fill probability, since the quotes are not symmetric once skew enters. ## The moving-target caveat The estimate rests on assumptions that drift. The mid price wanders, so your distance to the mid constantly changes, which means your fill probability is not static, it moves as the price moves relative to your resting order, and a snapshot estimate goes stale. The one-big-order assumption overstates fills, since real fills often come from several smaller trades chipping at the queue rather than a single sweep, so the CDF read is an approximation. And the taker size distribution itself is non-stationary, clustering and spiking, so the hour of history you fit the CDF on may not describe the next hour. You can build an order book simulator from these statistical properties to stress the estimate, but the fill probability remains a model of a moving target, useful for placing quotes and wrong if you trust it as exact. ## Visualizing fill probability ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_135-visual-0.png) ## KEY POINTS - A quote has two unknowns: PnL if filled (easy, the distance to fair value minus fees) and probability of fill (hard). The EV is their product, and it peaks between quoting too wide and too tight. - Estimate fill probability from taker order sizes: collect an hour of trades, build the CDF of taker sizes, and read off the probability that the next trade is at least the liquidity ahead of you plus your own size. - The model assumes one big order wipes out everything in front of you and fills you in the same sweep. Deeper placement means more liquidity ahead, a larger order needed, and lower fill probability. - Convert per-trade probability to time: use the taker arrival rate (takers per minute) to get the average number of trades to wait, then the expected time to fill, so EV becomes profit per unit time. - The full model optimizes PnL times fill probability on each side independently, since skew makes the quotes asymmetric. - Caveats: the mid wanders so fill probability is not static; the one-big-order assumption overstates fills; and the taker size distribution is non-stationary, so an hour of history may not describe the next hour. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [The Art of Currency Trading - Brent Donnelly (Amazon)](https://www.amazon.com/s?k=The+Art+of+Currency+Trading+Brent+Donnelly&ref=aligrithm.com) ### 5.15 Using Trade Flow to Predict Short-Term Price Movement URL: https://aligrithm.com/using-trade-flow-to-predict-short-term-price-movement/ Last updated: 2026-07-27T13:25:10.000Z Order book imbalance reads the resting liquidity, the orders people have posted. Trade flow reads the opposite thing: the liquidity people are taking, the actual executed trades. As a market maker you ultimately do not care where others provide liquidity, you care where people are taking it and how much, because takers are the ones who move the price and the ones who fill your quotes. Signed trade flow is the second microstructure feature to build after imbalance, and it works because trades are autoregressive, which means past trades predict future trades, and future trades predict the price. This article builds the trade-based forecast that ["Why Fair Value Is the Core of Market Making"](https://aligrithm.com/why-fair-value-is-the-core-of-market-making/) listed alongside imbalance, and connects it to the skew logic that uses a predicted move. ## Sign the flow The raw input is the trade tape: each execution has a size and a side, a buy taker or a sell taker. Sign the size by the side, positive for taker buys and negative for taker sells, and sum over a window to get net signed flow, sometimes called pressure. $$ \\text{pressure}\_t = \\frac{\\sum\_i s\_i \\cdot \\text{size}\_i}{\\Delta t}, \\quad s\_i = +1 \\text{ for taker buy}, \\; -1 \\text{ for taker sell} $$ Pressure over a window is the sum of signed trade sizes divided by the window length, where s\_i is the side of trade i. Positive pressure means buyers are leaning on the book harder than sellers; negative means the reverse. This is a flow rate, dollars or units of net taking per unit time, and it is the cleanest summary of which side is consuming liquidity. ## Why it predicts: autocorrelation Trade flow predicts because market orders are autocorrelated and follow roughly an AR(1) process: buys tend to be followed by more buys and sells by more sells, clustering at a low baseline with sudden spikes. A large order is often a slice of a bigger parent order being worked through execution, so seeing taker buys now raises the odds of taker buys in the next moments. The chain to price is short. If enough taking happens on one side and the book is thin enough, the mid moves, because the takers consume the resting liquidity and walk the price. So predicting trades predicts price changes: forecast that buy pressure will persist, and you forecast the mid will drift up. The autocorrelation is the engine, the same statistical property that makes the feature actionable rather than instantaneous noise. There is an immediate defensive use too. If you see a burst of taker buys, you should not be sitting there with sell orders feeding those takers cheap liquidity, because you will be left short into a market that is about to climb. Trade flow tells you which side is dangerous to quote on right now. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Fold it into fair value The forecast enters the same way imbalance does: regress the signed-flow feature against future changes in your basic fair price, and skew the center toward the prediction. Positive pressure predicts an up move, so you lean fair value up, which tightens your bid and widens your ask, the skew from ["Why Skewing Is Simpler Than People Think"](https://aligrithm.com/why-skewing-is-simpler-than-people-think/), expressed through a forecast rather than through inventory. Trade flow and imbalance are complementary, not redundant. Imbalance reads intentions posted to the book, which can be spoofed; trade flow reads executions, which are real and cannot be faked the same way, since a trade is a completed transaction, not a displayed promise. Combining a resting-liquidity feature with a taken-liquidity feature gives a center that respects both what the book shows and what the tape does. ["Order Book Imbalance: The First Microstructure Feature to Test"](https://aligrithm.com/order-book-imbalance-the-first-microstructure-feature-to-test/) builds the first, and the microprice in ["Microprice: Better Than Mid Price?"](https://aligrithm.com/microprice-better-than-mid-price/) is where both can be folded into the center. ## The limits Trade flow is a short-horizon signal with the usual caveats. The autocorrelation decays fast, so the prediction is for the next seconds, not minutes, and stale flow is worthless. The signing can be wrong when the data does not cleanly mark the aggressor side, and a mis-signed tape produces a backwards feature. And the relationship between flow and price depends on book thickness: the same pressure moves a thin book and barely budges a thick one, so flow should be read together with the resting liquidity rather than alone. It is a strong, real feature, and it is one input to a center, not a standalone strategy. ## Visualizing trade flow ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_134-visual-0.png) ## KEY POINTS - Trade flow reads liquidity being taken (executed trades), the opposite of order book imbalance, which reads liquidity being posted. A maker cares where people take liquidity, because takers move price and fill quotes. - Sign each trade by its side, positive for taker buys and negative for taker sells, and sum over a window to get net signed flow (pressure), a rate of net taking per unit time. - It predicts because market orders are autocorrelated and follow roughly an AR(1) process: buys follow buys, sells follow sells. Predicting trades predicts price, since enough taking on a thin book moves the mid. - Defensive use: a burst of taker buys means you should not be feeding sell orders into it, or you end up short into a rising market. - Fold it into fair value by regressing signed flow against future fair-price changes and skewing the center toward the prediction, the same way imbalance enters. - Trade flow and imbalance are complementary: imbalance reads spoofable intentions, trade flow reads real executions that cannot be faked the same way. The microprice can carry both. - Limits: the autocorrelation decays fast (next seconds, not minutes), the aggressor signing can be wrong in messy data, and the flow-to-price link depends on book thickness, so read it with resting liquidity, not alone. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [The Art of Currency Trading - Brent Donnelly (Amazon)](https://www.amazon.com/s?k=The+Art+of+Currency+Trading+Brent+Donnelly&ref=aligrithm.com) ### 5.14 Microprice: Better Than Mid Price? URL: https://aligrithm.com/microprice-better-than-mid-price/ Last updated: 2026-07-27T13:25:09.000Z The mid price, the average of the best bid and the best ask, is the default center everyone reaches for. It throws away information sitting in plain view: the sizes resting at the bid and the ask. When a thousand units rest on the bid and ten on the ask, the mid treats the situation as balanced even though the price is far more likely to climb than fall. The weighted mid, and its refinement the microprice, fix that by leaning the center toward the side the book says price will move. This article builds on ["Order Book Imbalance: The First Microstructure Feature to Test"](https://aligrithm.com/order-book-imbalance-the-first-microstructure-feature-to-test/), which established that book lopsidedness predicts the next move. The microprice is the natural way to fold that prediction directly into your fair value. ## The weighted mid The plain mid weights the best bid and best ask equally. The weighted mid weights each price by the size resting on the opposite side, so the heavier side pulls the center toward the lighter side. $$ \\text{weighted mid} = \\frac{P\_a \\cdot V\_b + P\_b \\cdot V\_a}{V\_a + V\_b} $$ Here P\_a and P\_b are the best ask and best bid prices, and V\_a and V\_b are the sizes resting at the ask and the bid. The ask price is weighted by the bid volume and the bid price by the ask volume. The reasoning: if the size at the bid is much larger than the size at the ask, the ask side is the one likely to get cleared and the price is likely to move up toward the ask, so the center should sit closer to the ask. Weighting the ask price by the large bid volume does exactly that, pulling the weighted mid up toward the ask. The weights are crossed on purpose, the big side pushes the center away from itself, toward where the price is headed. A quick numerical check: with a bid at 100 holding 1000 units and an ask at 101 holding 10 units, the plain mid is 100.5, but the weighted mid lands near 100.99, almost at the ask, because the heavy bid says the price will climb. That shift, from 100.5 to nearly 101, is the information the plain mid discarded. ## Microprice as the improvement The weighted mid is a good improvement on the plain mid, and the microprice is a good improvement on the weighted mid. The weighted mid uses only the top-of-book sizes and a linear weighting. The microprice refines the idea by accounting for the fact that the imbalance itself predicts the future mid, and that the relationship is not linear, the same tails-matter pattern from the imbalance article. It estimates where the mid will actually be conditional on the current imbalance, rather than just taking a size-weighted average of the current quotes. In practice the microprice is the expected future mid given the current book state, built to respect that imbalance is most informative on the tails. Where the weighted mid moves linearly with the size ratio, the microprice bends, leaning hard toward one side only when the book is genuinely lopsided and staying near the plain mid when the book is roughly balanced. That curvature is the same spline-shaped relationship the imbalance article fit, now expressed as a price rather than a raw feature. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Why a maker wants this as the center For a market maker the microprice is not an academic nicety, it is the fair value you quote around. Center your spread on the plain mid when the book is lopsided and you are systematically late: the price is about to move toward the thin side, and you are quoting symmetrically around a center that ignores it, handing the informed trader the mispriced side from ["Adverse Selection Explained for Traders"](https://aligrithm.com/adverse-selection-explained-for-traders/). Center on the microprice and your quotes already lean toward the predicted move, so the option you hand the taker is no longer mispriced and your markouts improve. This is the most direct way book imbalance enters fair value. Rather than computing imbalance as a separate feature and bolting a skew onto the mid, the microprice bakes the imbalance prediction into the center itself. You can still add the other predictions, signed trade flow, volatility, cross-exchange leads, on top, but the microprice is the clean starting center that already respects what the resting sizes are telling you. ## The limits The microprice inherits every weakness of imbalance. Spoofed size distorts it, a fake stacked bid pulls the microprice up toward the ask exactly as a real one would, so the sizes feeding it should be filtered for sturdiness from ["Spoofing, Sturdy Liquidity, and Book Pressure"](https://aligrithm.com/spoofing-sturdy-liquidity-and-book-pressure/). It is a short-horizon estimate, useful for the next move, not a multi-minute fair value. And the choice of how many levels to include and how to fit the curvature are decisions you have to test, not constants handed to you. The microprice is a better center than the mid, not a perfect one, and it is only as honest as the book sizes you trust. ## Visualizing the microprice ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_133-visual-0.png) ## KEY POINTS - The plain mid averages the best bid and ask equally and throws away the resting sizes. When one side is far heavier, the price is likely to move toward the thin side, and the mid ignores that. - The weighted mid weights each price by the opposite side's volume, so the heavy side pulls the center toward the lighter side, where price is headed. The weights are crossed on purpose. - Worked case: bid 100 with 1000 units, ask 101 with 10 units gives a plain mid of 100.5 but a weighted mid near 100.99, almost at the ask. That shift is the information the plain mid discarded. - The microprice improves on the weighted mid by respecting that imbalance predicts the future mid non-linearly, leaning hard only when the book is genuinely lopsided and staying near the plain mid when balanced. - For a maker the microprice is the fair value to quote around. It bakes the imbalance prediction into the center, so quotes already lean toward the move and the option handed to takers is no longer mispriced. - Limits: spoofed size distorts it, so filter for sturdiness; it is a short-horizon estimate, not a multi-minute fair value; and the level count and curvature fit are choices to test. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [The Art of Currency Trading - Brent Donnelly (Amazon)](https://www.amazon.com/s?k=The+Art+of+Currency+Trading+Brent+Donnelly&ref=aligrithm.com) ### 5.13 Order Book Imbalance: The First Microstructure Feature to Test URL: https://aligrithm.com/order-book-imbalance-the-first-microstructure-feature-to-test/ Last updated: 2026-07-27T13:25:09.000Z If you build one microstructure feature, build this one. Order book imbalance compares the size resting on the bid against the size resting on the ask and turns it into a single number between minus one and plus one. It is the strongest simple predictor of the next short-horizon price move, the first feature any market making forecast should include, and the one that does most of the work in centering fair value. Top-of-book imbalance is by far the strongest of the basic features, ahead of trade-volume imbalance and volatility. This article builds the feature that ["Why Fair Value Is the Core of Market Making"](https://aligrithm.com/why-fair-value-is-the-core-of-market-making/) named as the workhorse of the forecast. It is the cleanest entry point into microstructure alpha. ## The formula Order book imbalance at time t compares bid liquidity against ask liquidity, normalized by their sum. $$ I(t) = \\frac{B(t) - A(t)}{B(t) + A(t)} $$ Here B(t) is the liquidity resting on the bid and A(t) the liquidity resting on the ask, at time t. The numerator is the raw excess of bid over ask; the denominator normalizes it so the result always lands between minus one and plus one regardless of the absolute size in the book. A value near plus one means the bid is stacked relative to the ask, near minus one means the ask is stacked, and zero means balanced. A worked value makes it concrete. Suppose three units rest on one side and four on the other across the levels you measure, so the difference is one unit and the total is seven. The imbalance is one over seven, roughly 0.14, a mild lean toward the heavier side. The sign tells you the direction of the pressure, the magnitude tells you how lopsided the book is. ## Why it predicts The intuition is mechanical. A heavily stacked bid against a thin ask means the ask side is easier to clear, so the price is more likely to move up toward the ask, because it takes less volume to push through the thin side than the thick one. Imbalance measures which side is the path of least resistance, and price tends to take that path over the next short interval. The relationship shows up cleanly in the data. Plot imbalance against the average mid-price change that follows, binning by imbalance level, and you get a steadily increasing curve: more positive imbalance, more positive expected next move. The feature also has autocorrelation, the imbalance now predicts the imbalance shortly ahead, which means it persists long enough to act on rather than vanishing the instant you measure it. ## Use it on the tails, fit a spline The predictive content is not uniform across the range. Imbalance is most predictive on the tails, when the book is heavily lopsided, and weakest in the middle, when it is roughly balanced. A book at 0.05 imbalance tells you almost nothing; a book at 0.85 imbalance is leaning hard and carries real signal. So the relationship between imbalance and the next move is not a straight line, and forcing a linear fit wastes the information sitting in the extremes. The fix is to fit a spline to imbalance versus mid-price change rather than a single slope. The spline captures the flat middle and the steep tails, giving a continuous prediction of the next move for any imbalance value. That continuous prediction feeds straight into fair value: skew the center toward the predicted move, which ["Microprice: Better Than Mid Price?"](https://aligrithm.com/microprice-better-than-mid-price/) formalizes as a specific weighting of the mid, and which ["Adverse Selection Explained for Traders"](https://aligrithm.com/adverse-selection-explained-for-traders/) needs to shade quotes away from the informed side. ## The honest limits Imbalance is the strongest simple feature, and it is still a weak signal on any single read, with the usual microstructure caveats. It can be gamed: a stacked bid that is a spoof, from ["Spoofing, Sturdy Liquidity, and Book Pressure"](https://aligrithm.com/spoofing-sturdy-liquidity-and-book-pressure/), shows as positive imbalance and then vanishes, so raw imbalance counts spoofed size as real pressure and a careful version weights levels by their sturdiness. It is short-lived: the autocorrelation that makes it tradable also decays fast, so the feature is for short-horizon forecasts, not multi-minute views. And how many levels you include in B and A changes the feature, top-of-book versus several levels deep behave differently, so the depth is a choice you have to make and test rather than a given. None of this disqualifies it. It remains the first feature to build and the one that earns the most per line of code. ## Imbalance is also a factor, not just a feature There is a second way to think about imbalance that changes how you test everything built after it. At HFT horizons, order book imbalance does not earn risk-adjusted return on its own, it is too short-lived and too well known for that. What it is, in the language of the old article ["What Is a Factor, Really,"](https://aligrithm.com/what-is-a-factor-really/) is a factor: an effect that explains a large chunk of the next-move variance and that you will keep accidentally re-finding inside your other alphas. Build a trade-flow signal, a microprice tilt, a queue-position model, and run their correlations, and you will find imbalance hiding inside all of them, the same way three value signals turn out to be one value factor wearing different clothes. The consequence is procedural. Before you get excited that a new microstructure alpha predicts the next move, regress imbalance out of returns and test the candidate against the residual, the part imbalance does not already explain. A signal that only scored because it was imbalance in disguise flatlines against the residual; a signal with genuine independent content still slopes up. Skip this and a backtest against raw next-move returns will happily reward you for re-discovering imbalance under a new name, the curve will look real because the moves are real, they are just not new. Imbalance is the first feature to build and the first factor to subtract. ## Visualizing imbalance ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_132-visual-0.png) ## KEY POINTS - Order book imbalance is the strongest simple microstructure feature and the first one to build, ahead of trade-volume imbalance and volatility. It does most of the work in centering fair value. - It is the bid liquidity minus the ask liquidity over their sum, always between minus one and plus one. Near plus one the bid is stacked, near minus one the ask is stacked, zero is balanced. - It predicts because a stacked bid against a thin ask makes the ask the path of least resistance, so price tends to move that way. Binned plots show a steadily increasing curve of next move versus imbalance. - Imbalance has autocorrelation, so it persists long enough to act on instead of vanishing the instant you measure it. - The signal lives in the tails: heavily lopsided books predict, balanced books do not. Fit a spline to imbalance versus next move to capture the flat middle and steep tails, and feed the continuous prediction into fair value. - Limits: spoofed size shows as fake imbalance, so weight levels by sturdiness; the signal decays fast and suits short horizons; and the number of levels included changes the feature, a choice to test rather than assume. - Imbalance is also a factor, not just a feature. At HFT horizons it does not earn risk-adjusted return on its own, but it explains a lot of next-move variance and you keep re-finding it inside other alphas. Regress it out of returns and test new microstructure signals against the residual, the move from the old article ["What Is a Factor, Really,"](https://aligrithm.com/what-is-a-factor-really/) so you do not reward yourself for re-discovering imbalance under a new name. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [The Art of Currency Trading - Brent Donnelly (Amazon)](https://www.amazon.com/s?k=The+Art+of+Currency+Trading+Brent+Donnelly&ref=aligrithm.com) ### 5.12 Spoofing, Sturdy Liquidity, and Book Pressure URL: https://aligrithm.com/spoofing-sturdy-liquidity-and-book-pressure/ Last updated: 2026-07-27T13:25:08.000Z Every placement trick in the last two articles rests on one assumption: that the large order you are sitting in front of is real. In crypto that assumption is dangerous. Spoofing is real and common, large orders posted with no intention of filling, designed to create the appearance of pressure and then yanked the instant someone reacts. Place in front of a spoof and the wall you thought was protecting your markouts vanishes the moment you need it, and the price turns toward exactly where the fake order was. This article handles the hazard that ["Order Placement Alpha: The Forgotten Edge"](https://aligrithm.com/order-placement-alpha-the-forgotten-edge/) and ["How to Use Order Book Density for Better Limit Orders"](https://aligrithm.com/how-to-use-order-book-density-for-better-limit-orders/) both flagged. The job is to tell sturdy liquidity from a spoof, and to understand what happens to the price when a large order disappears. ## Why a vanishing order moves the price Start with the mechanics of book pressure, because they explain both the danger and the opportunity. A large resting order is weight on one side of the book. While it sits there, it holds the price, because anyone wanting to push through that side has to consume it first. When that large order disappears, whether filled or cancelled, the weight is gone and the price suddenly turns toward where the order was, because the book pressure has shifted aggressively in that direction. Picture a large bid sitting below the price. It acts as support; the price rests above it. Pull that bid and the support is gone, and the price drops toward the level the bid was defending. A spoofer who never intended to buy posts that bid to make the book look supported, lets others lean on it, then cancels, and the price falls. The disappearance of size is itself a price-moving event, and a spoofer manufactures that event on purpose. ## The danger to your placement This is precisely why placing in front of a spoof is worse than useless. The whole point of sitting in front of a large order, from the order-placement article, is that a taker who fills you cannot push the price through the wall behind you, so your markout stays flat. If that wall is a spoof, it disappears the instant it would have protected you. You get filled, the spoofed order vanishes, the book pressure shifts toward where it was, and the price runs against your fill, the exact adverse markout you placed there to avoid. You took on the toxicity you were hedging against, because you trusted a wall that was never load-bearing. So placement that leans on resting size demands a filter: which orders are sturdy and which will evaporate quickly. Without that filter, density-based placement is leaning on liquidity that fails under load. ## Telling sturdy from spoofed The sturdiness logic watches how orders behave over time rather than trusting their displayed size. A few signals separate the two. Sturdy liquidity persists: a level that holds its size across the multi-minute average density from the placement article is more likely real than a wall that appears and vanishes in seconds. Spoofs flicker: they tend to post, sit briefly to be noticed, and pull as soon as the price approaches or as soon as flow reacts. Tracking the lifetime and the cancel behavior of large orders, how long they rest and whether they pull on approach, builds a probability that a given wall is sturdy versus spoofed, and you only lean your placement on the ones that score sturdy. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Turning the spoof against the spoofer The disappearance dynamic also cuts the other way, and there is an edge in it. If you have an order that is likely to be spoofed, you can have the opposite effect by flying with it, following it around. Identify a large order you judge to be a spoof, anticipate that it will be pulled and that the price will then turn toward where it sat, and position for that turn rather than leaning on the order as support. Instead of placing in front of the fake wall and getting run over when it vanishes, you trade the predictable move that its disappearance creates. The spoofer is manufacturing a price-moving event on a schedule you can read, and reading it turns their manipulation into your signal. ## Visualizing book pressure ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_131-visual-0.png) ## KEY POINTS - Every placement trick assumes the large order you sit in front of is real. In crypto, spoofing (large orders posted with no intent to fill, then pulled) makes that assumption dangerous. - A large resting order is weight that holds the price. When it disappears, filled or cancelled, the price turns toward where it was because book pressure has shifted. The disappearance of size is itself a price-moving event. - Placing in front of a spoof is worse than useless: the protective wall vanishes the instant it would help, you get filled, and the price runs against you, the exact adverse markout you placed there to avoid. - Placement on resting size requires a filter separating sturdy liquidity from orders that will evaporate. - Sturdy liquidity persists across the multi-minute average density; spoofs flicker, posting briefly and pulling on approach. Track order lifetime and cancel-on-approach behavior to score sturdiness, and lean only on sturdy orders. - The disappearance dynamic is also an edge: identify a likely spoof, anticipate it will be pulled and that price turns toward where it sat, and trade that turn instead of leaning on the order. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [The Art of Currency Trading - Brent Donnelly (Amazon)](https://www.amazon.com/s?k=The+Art+of+Currency+Trading+Brent+Donnelly&ref=aligrithm.com) ### 5.11 How to Use Order Book Density for Better Limit Orders URL: https://aligrithm.com/how-to-use-order-book-density-for-better-limit-orders/ Last updated: 2026-07-27T13:25:08.000Z The order book is not a flat ladder where every price level holds the same liquidity. It is a landscape of dense walls and thin gaps, and where you rest a limit order relative to that landscape changes both how fast you fill and how toxic the fill is. The previous article, ["Order Placement Alpha: The Forgotten Edge"](https://aligrithm.com/order-placement-alpha-the-forgotten-edge/), argued that placement is its own source of edge. This one builds the density analysis that drives the placement decision. The core move is to stop looking at the book as a snapshot and start looking at it as an average over time, then place where that average tells you liquidity reliably sits. ## Average density, not a snapshot A single book snapshot lies to you. Orders flicker in and out on millisecond timescales, layered and pulled by other participants and by spoofers, so the wall you see this instant may be gone before you can use it. Take the average density over the course of multiple minutes instead, and the picture stabilizes into where liquidity actually rests rather than where it momentarily appears. The average density at each price level near your target neighborhood is the input. A level that holds size consistently across minutes is a real shelf you can lean on; a level that shows size only in flickers is noise you should ignore. Build the average, find the optimal spot to place within your target neighborhood, and place there. ## The size-versus-tightness trade-off Placement is an optimization between two competing pulls. Place tighter, closer to the touch, and you fill faster but earn less edge per fill and sit in front of less protective size. Place behind a chunk of density, further from the touch, and you fill slower but with better markouts and more edge. The optimal placement balances the two. The mechanism for the trade-off is explicit. You weight the benefit of skipping in front of a large chunk of density against the cost of the increased aggression that skipping requires, and the optimal density to place against falls out of that weighting. Skipping past a dense level to get in front of it costs you a more aggressive price; the question is whether the markout improvement and edge from sitting in front of that wall justify the aggression. Assign weights to that trade-off and the placement is decided by the numbers rather than by feel. $$ \\text{score}(p) = \\text{edge}(p) + w \\cdot \\text{protective density behind } p - c \\cdot \\text{aggression}(p) $$ The score of placing at price p combines the edge you earn there, the protective resting density sitting behind that placement (weighted by w), minus the aggression cost of getting there (weighted by c). Place at the p that maximizes the score. The protective-density term is the markout benefit from ["Order Placement Alpha: The Forgotten Edge"](https://aligrithm.com/order-placement-alpha-the-forgotten-edge/): size behind you blocks takers from running the price through your fill. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Respect the rate limit For large-scale quoting the analysis collides with exchange rate limits. Continuously one-upping other participants, shifting your order a tick every time someone else does, burns your order-update budget and gains little. So you match these placement modifications infrequently. Apply the density-driven adjustment at a measured cadence rather than chasing every change in the book, because the average density you are placing against moves slowly anyway, and reacting to every flicker spends your update budget on noise. This is the practical reason the average-over-minutes view is not just cleaner but necessary: you cannot afford to re-place on every snapshot, so you place against the slow-moving average and update only when it has meaningfully shifted. ## The sturdiness caveat Density analysis assumes the density is real. The whole edge of placing in front of a large order depends on that order being there when you need it, and in crypto much of the visible size is spoofed and will vanish. Average density over minutes partly defends against this, since a persistent shelf is less likely to be a single spoof than an instantaneous wall is. But it is not a full defense, and density-based placement still needs the sturdiness logic from ["Spoofing, Sturdy Liquidity, and Book Pressure"](https://aligrithm.com/spoofing-sturdy-liquidity-and-book-pressure/) to avoid leaning on size that evaporates the moment it would have protected you. ## Visualizing density ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_130-visual-0.png) ## KEY POINTS - The order book is a landscape of dense walls and thin gaps, not a flat ladder. Where you rest relative to that landscape changes both fill speed and fill toxicity. - A single snapshot lies because orders flicker in and out. Average density over several minutes to see where liquidity reliably sits, then place within your target neighborhood against that average. - Placement is a trade-off: tighter fills faster with less edge and less protection; behind a density chunk fills slower with better markouts and more edge. - Decide it by weighting the benefit of getting in front of a large density chunk against the aggression cost of skipping to it. Score each placement and pick the maximum. - The protective-density term is the markout benefit: size behind you blocks takers from running the price through your fill. - Large-scale quoting hits exchange rate limits, so apply placement modifications infrequently at a measured cadence rather than one-upping every tick. The average density moves slowly, so this loses little. - Density analysis assumes the size is real. Averaging over minutes partly filters spoofs, but density-based placement still needs explicit sturdiness logic to avoid leaning on liquidity that evaporates. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [The Art of Currency Trading - Brent Donnelly (Amazon)](https://www.amazon.com/s?k=The+Art+of+Currency+Trading+Brent+Donnelly&ref=aligrithm.com) ### 5.10 Order Placement Alpha: The Forgotten Edge URL: https://aligrithm.com/order-placement-alpha-the-forgotten-edge/ Last updated: 2026-07-27T13:25:07.000Z Fair value, spread, and skew decide the levels you want to quote at. They do not decide the exact tick you rest your order on, and that last decision, where inside the small neighborhood of your target level you actually place, is its own source of edge. Order placement modification is more important than skewing, and almost nobody writes about it. The academic literature spends its time on inventory control while the practical money sits in an unglamorous adjustment made after the three pillars have already spoken. This article covers the fourth decision the three pillars articles deferred. It pairs with ["How to Use Order Book Density for Better Limit Orders"](https://aligrithm.com/how-to-use-order-book-density-for-better-limit-orders/), which works through the density analysis that drives the placement. ## What order placement is Once fair value, spread, and skew have set the level you want to quote, you make an ad hoc modification to the precise placement of the order around that level. The goal is to extract a little more edge for very little change in fill probability. You are not changing your view or your width. You are nudging the order a tick or two to a better spot within the neighborhood your strategy already chose, where "better" means improved markouts and hidden flow without meaningfully changing how often you fill. Three things the modification buys you: a bit more edge per fill, improved markouts, and concealment of your flow so other participants cannot read your intentions off the book. The first is the obvious one. The second and third are where the real value sits. ## Placing in front of size improves markouts The sharpest placement trick uses the resting liquidity around you. Place your order just in front of a large resting order and your markouts improve, for a mechanical reason. If a taker fills you, they then have to get through the large order sitting right behind you to push the price further against your fill. They usually cannot. The large order behind you absorbs the continuation, so the price does not run, and your markout stays flat instead of sinking into the adverse-selection trough from ["Markouts: The Truth Serum of Market Making"](https://aligrithm.com/markouts-the-truth-serum-of-market-making/). The large order is a wall at your back. Sitting just in front of it means anyone who hits you is immediately blocked from moving the price, so the fills you take in that spot are systematically less toxic than fills taken in open water with nothing behind them. ## Read density over minutes, not snapshots The placement decision runs on average book density, not the instantaneous snapshot. Take the average density over the course of several minutes and find the optimal spot to place within your target neighborhood, because a single snapshot is noise: orders flicker in and out, and a wall that is there this instant may be gone the next. The multi-minute average tells you where liquidity reliably sits. For large-scale quoting there is a rate-limit constraint. One-upping other participants tick by tick burns through your order-update budget, so you only want to apply these placement modifications infrequently, matching them at a measured cadence rather than chasing every flicker. The analysis also trades size against tightness: you weight the benefit of skipping past a chunk of density against the cost of the increased aggression that skipping requires, and the optimal placement falls out of that trade-off. ["How to Use Order Book Density for Better Limit Orders"](https://aligrithm.com/how-to-use-order-book-density-for-better-limit-orders/) builds this density analysis in full. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## The spoofing complication One hazard hangs over all of it. Placing in front of a large order assumes the large order is real. In crypto a lot of large orders are spoofs that vanish before they fill, and placing in front of a spoof is worthless: when the spoof disappears the wall at your back is gone, and the price turns toward where the order was because the book pressure has shifted. So order placement that leans on resting size has to be paired with logic that decides which orders are sturdy and which will evaporate, the subject of ["Spoofing, Sturdy Liquidity, and Book Pressure"](https://aligrithm.com/spoofing-sturdy-liquidity-and-book-pressure/). Place in front of sturdy size and your markouts improve; place in front of a spoof and you get the adverse move you were trying to avoid. ## Visualizing the placement ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_129-visual-0.png) ## KEY POINTS - Fair value, spread, and skew set the level you want to quote. Order placement is the separate decision of the exact tick to rest on within that level, and it is more important than skewing. - The modification extracts a little more edge for very little change in fill probability. Its main value is improved markouts and hidden flow, not just extra edge per fill. - Placing just in front of a large resting order improves markouts: a taker who fills you is then blocked by the large order behind you from pushing the price further, so your fill is less toxic. - Run placement on average book density over several minutes, not an instantaneous snapshot, because single snapshots are noise and walls flicker in and out. - Large-scale quoting hits rate limits, so apply placement modifications infrequently rather than one-upping every tick, and trade size against tightness when deciding how far to skip. - The hazard is spoofing: placing in front of a spoofed order is worthless because the wall vanishes before it protects you, and the price turns toward where the order was. Pair placement with logic that judges which orders are sturdy. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [The Art of Currency Trading - Brent Donnelly (Amazon)](https://www.amazon.com/s?k=The+Art+of+Currency+Trading+Brent+Donnelly&ref=aligrithm.com) ### 5.9 Spread Widening During Volatility Expansion URL: https://aligrithm.com/spread-widening-during-volatility-expansion/ Last updated: 2026-07-27T13:25:07.000Z The width of your quote is the dial that trades fill frequency against profit per fill, and the single most important thing that dial responds to is volatility. The rule is blunt: blow your spreads out when the market's volatility does. A fixed markup that was safe in a calm tape becomes a gift to informed traders the moment the market starts moving, because the same spread now covers a fraction of the distance the price can travel before you reprice. This article builds the spread pillar from ["The Three Pillars of Market Making: Fair Price, Spread, Skew"](https://aligrithm.com/the-three-pillars-of-market-making-fair-price-spread-skew/) and uses the markout diagnostic from ["Markouts: The Truth Serum of Market Making"](https://aligrithm.com/markouts-the-truth-serum-of-market-making/) to decide where the widening should bite. ## Start fixed, then add a volatility layer The most basic spread is a fixed markup on your skewed fair value. Quote a constant number of basis points on each side and leave it. It works in a stationary market and fails the instant volatility regime-shifts, which crypto does constantly. The next model adds layers of wider spread as volatility rises. Estimate short-horizon volatility, your own fair-value volatility is a clean choice, and map it to a spread width: calm regime, tight spread; elevated regime, wider; violent regime, widest. The mapping can be discrete, three regimes set by volatility quantiles each with its own width, or continuous, interpolating the width between regime anchors so the spread moves smoothly as volatility drifts. Either way the spread is now a function of volatility rather than a constant. $$ \\text{spread}\_t = \\text{base} \\cdot \\left( 1 + \\lambda \\cdot \\frac{\\sigma\_t}{\\bar{\\sigma}} \\right) $$ The spread at time t is a base width scaled up by the ratio of current volatility sigma-t to its average sigma-bar, with a sensitivity constant lambda. When current volatility sits at its average the ratio is one and the spread is near base; when volatility doubles, the spread expands by lambda times the excess. The exact functional form is less important than the direction: more volatility, more width. ## Why widen, in adverse-selection terms The reason is the adverse selection from ["Adverse Selection Explained for Traders"](https://aligrithm.com/adverse-selection-explained-for-traders/). Volatility is the size of the move the market can make between now and the moment you can reprice. A larger possible move means a larger chance an informed trader runs through your stale quote before you adjust, and a larger loss when they do. Wider spreads during volatility buy protection in the exact moments getting picked off costs the most, because the price can travel further against your fill. The markout tells you whether the widening is calibrated. If your markouts stay bad during high-volatility periods even after widening, you are not widening enough; if they are fine, the width is doing its job. Read the markout curve conditional on volatility regime and let it set how aggressively the spread should respond. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Different fair value per depth Advanced desks attach a different fair value to each quoting depth, because quotes at different depths worry about different horizons. A quote sitting at the top of the book is exposed to getting picked off in the next minutes, so it is priced against a short-horizon forecast and reacts fast to short-horizon volatility. A deep quote will only fill on a large move, so it is concerned with longer-term forecasts and longer-horizon volatility. Running one fair value and one volatility estimate across all depths misprices both ends; matching the horizon of the forecast and the volatility to the depth of the quote prices each level for the risk it actually faces. ## The news case: sometimes stop quoting Beyond the smooth volatility response, there is a discrete defense. You can add modifications that blow your spread out conditional on spread-specific alpha, signals in the order book that news has been released. The clearest case is scheduled high-impact news. Some firms stop quoting entirely around high-impact news, because the informed traders with fast news feeds will pick off any quote you leave up, and no reasonable spread is wide enough to cover the gap risk. Scheduled events at least give advance warning, so a desk can widen drastically or pull quotes from the start rather than reacting after the first informed fill has already landed. Widening is the continuous defense; pulling quotes is the discrete one for the moments when no width is safe. ## Visualizing the response ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_128-visual-0.png) ## KEY POINTS - Spread is the dial trading fill frequency against profit per fill, and its most important input is volatility. The rule: blow your spreads out when the market's volatility does. - A fixed markup is safe only in a stationary market. Add a volatility layer that maps short-horizon volatility (your own fair-value volatility is a clean choice) to width, discretely by regime or continuously by interpolation. - A simple form scales a base width by the ratio of current to average volatility. The exact function matters less than the direction: more volatility, more width. - The reason is adverse selection. Volatility is the size of the move possible before you reprice, so it raises both the chance and the cost of getting picked off. Wider spreads protect exactly when pick-offs cost most. - Calibrate with markouts conditional on volatility regime. Bad markouts in high-volatility periods after widening mean you are not widening enough. - Advanced desks attach a different fair value and volatility horizon to each depth: top-of-book quotes price a short horizon, deep quotes price a long one. - The discrete defense is stopping quotes around scheduled high-impact news, where informed fast-feed traders pick off any quote and no width is safe. Scheduled events give advance warning to pull or widen from the start. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [The Art of Currency Trading - Brent Donnelly (Amazon)](https://www.amazon.com/s?k=The+Art+of+Currency+Trading+Brent+Donnelly&ref=aligrithm.com) ### 5.8 Why Skewing Is Simpler Than People Think URL: https://aligrithm.com/why-skewing-is-simpler-than-people-think/ Last updated: 2026-07-27T13:25:06.000Z Skewing is the most published topic in academic market making and the most over-engineered idea on a real desk. The papers wrap it in stochastic-control equations, Hamilton-Jacobi-Bellman solutions, optimal inventory trajectories. The actual operation is this: make one side of your quote wider than the other so that side fills less, and the cheaper side fills more, which moves your inventory the way you want. That is the whole mechanism. You do not need a complex skewing model, and the elaborate machinery mostly polishes the least important of the three pillars. ["The Three Pillars of Market Making: Fair Price, Spread, Skew"](https://aligrithm.com/the-three-pillars-of-market-making-fair-price-spread-skew/) ranked skew last on purpose. This article explains why the simple version is enough, building on the toxicity argument from ["Toxic Flow vs Inventory Risk"](https://aligrithm.com/toxic-flow-vs-inventory-risk/). ## The mechanism in one move You have a center (fair value) and a spread on each side. To skew, you push one side out and pull the other in around that center. Short some inventory and want to get flatter? Tighten your bid and widen your ask. The tighter bid gets hit more often, rebuilding your long side; the wider ask gets lifted less often, slowing the bleed of more short. Over the next batch of fills your inventory drifts back toward your target. Reverse it to shed a long. The asymmetry of the two distances is the entire control. There is nothing to solve here that needs an optimizer. A skew proportional to your inventory, leaning harder the further you are from flat, does the job. The marginal gain from a theoretically optimal skew schedule over a simple proportional one is tiny next to the gain from getting the fair value right, which is why the effort ranking matters. ## Why simple is enough: the target is toxicity, not variance The reason the simple version suffices traces back to what skew actually defends against. If skew existed to minimize the variance of your inventory, the precise schedule would matter, because you would be solving a genuine optimization of risk against spread captured. But skew does not exist for that. It exists to stop informed traders from loading you with toxic inventory, and that job is coarse by nature. You do not need a finely tuned response to refuse a toxic loading. You need to lean away from aggression fast enough that an informed trader cannot pile a large position onto you cheaply. A blunt, simple skew does that as well as a precise one, because the thing you are stopping is itself blunt: someone hammering one side of your book. ## The one rule that matters: graduate with aggression The single piece of nuance worth keeping is matching your skew to how toxic the counterparty looks, and aggression is the proxy for toxicity. The more aggressive someone becomes, the more likely they are an informed player, so you widen against them gradually in line with their impact. A light, occasional taker on one side is probably non-toxic, and as the toxicity argument says you should not skew hard against benign flow, since you give up spread refusing inventory that was never going to hurt you. Hold some of it. A relentless aggressor running through one side and moving the price is probably informed, and you lean away from them harder the more they push. The skew is a graduated response to observed aggression, not a fixed inventory-control law. Aggression you can measure in real time; the optimal control law you cannot, and you do not need to. $$ \\text{ask offset} = \\text{base} + k \\cdot (\\text{inventory lean}) + g \\cdot (\\text{observed aggression on this side}) $$ The ask offset is a base spread plus a term proportional to your inventory (the standard inventory lean, with constant k) plus a term proportional to how aggressively that side is being taken (constant g). Both k and g are simple, hand-set constants. The second term is the toxicity defense; the first is the inventory nudge. No optimizer is involved, and that is the point. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Visualizing the skew ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_127-visual-0.png) ## KEY POINTS - Skewing is the most published and most over-engineered topic in market making. The real operation: make one side wider so it fills less and the other side fills more, moving your inventory the way you want. - A skew proportional to inventory, leaning harder the further you are from flat, does the job. There is nothing here that needs an optimizer, and the gain over a theoretically optimal schedule is tiny next to getting fair value right. - Simple suffices because skew defends against toxic loading, not inventory variance. Refusing a toxic loading is a coarse job, so a blunt skew works as well as a precise one. - The one rule worth keeping: graduate the skew with counterparty aggression, the proxy for toxicity. Light flow is probably benign and should not be skewed against hard; a relentless aggressor gets leaned away from harder. - A practical skew is a base spread plus an inventory-lean term plus an aggression term, all with simple hand-set constants. The aggression term is the toxicity defense, the inventory term is the nudge. - Aggression is measurable in real time; the optimal control law is not, and you do not need it. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [The Art of Currency Trading - Brent Donnelly (Amazon)](https://www.amazon.com/s?k=The+Art+of+Currency+Trading+Brent+Donnelly&ref=aligrithm.com) ### 5.7 Toxic Flow vs Inventory Risk URL: https://aligrithm.com/toxic-flow-vs-inventory-risk/ Last updated: 2026-07-27T13:25:06.000Z The textbook market making problem is framed as inventory risk. You hold a position, the position has variance, the variance is risk, so you build elaborate machinery to keep inventory near zero. That framing puts the effort in the wrong place. Holding some inventory is not a serious problem. Holding toxic inventory, and a lot of it, is the problem that kills the book, and the two are not the same thing at all. This article draws the line that ["Adverse Selection Explained for Traders"](https://aligrithm.com/adverse-selection-explained-for-traders/) set up and that ["Why Skewing Is Simpler Than People Think"](https://aligrithm.com/why-skewing-is-simpler-than-people-think/) relies on. Confuse toxic flow with inventory risk and you build a skewing model that fights the wrong enemy. ## Inventory risk is mostly benign A position you accumulate from uninformed flow is just a position. It has variance, and variance is uncomfortable, but the expected value of the path is roughly flat, because the people who filled you had no edge over your fair value. You are long some coin because balanced two-sided flow happened to leave you long. Sit on it, let the next round of flow pull you back toward flat, and the holding cost is the ordinary variance of a position centered on a fair price. The mistake is treating that benign variance as the central danger and skewing hard to flatten it. Skew too aggressively on non-toxic flow and you give up spread you did not need to give up, refusing inventory that was never going to hurt you. As a rule, if the flow is clearly non-toxic you should not skew aggressively and should be willing to hold some risk. Even a net imbalance can be absorbed when it is probably not an informed imbalance. ## Toxic flow is the real danger Toxic inventory is a position handed to you by someone who knows it is about to move against you. The danger is not the variance of the position. It is the negative drift, the informed move that the toxic counterparty was front of and you are now behind. And the danger scales with size: a small toxic position is a small bleed, but an informed trader who can load you with a large toxic position at a fixed price has handed you a large guaranteed loss. The markout curve from ["Markouts: The Truth Serum of Market Making"](https://aligrithm.com/markouts-the-truth-serum-of-market-making/) is how you tell the two apart. Non-toxic inventory shows a flat markout, the price goes nowhere special after the fill. Toxic inventory shows the decaying trough, the price keeps moving against you. Two positions of identical dollar size can carry completely different toxicity, and the loss rate on the inventory, not its size, is what measures it. ## What skew is actually defending This reframing changes the job of skew. Skew is not there to manage inventory risk in the textbook variance sense. Skew is there to stop informed traders from loading you up with toxic inventory. When the flow turns aggressive on one side, you lean your quotes away from that side so the aggressor cannot keep filling you cheaply, which caps how much toxic inventory they can build before your price has moved against them. The graduated response follows from this. The more aggressive a counterparty becomes, the more likely they are toxic, so you widen your quotes against them gradually in line with their impact. A timid taker gets near-normal quotes; an aggressor who keeps hitting one side and moving the price gets progressively worse prices from you, because their aggression is itself the evidence of toxicity. You are not flattening a position. You are refusing to be the cheap counterparty to someone who knows where the price is going. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## The hedging trap that follows The same confusion produces a second, expensive mistake on the risk-management side: hedging toxic inventory with a correlated instrument and believing you are now flat. The correlations in your model do not necessarily hold, and when they break you discover the hedge was toxic too. A book that trusts its correlation matrix will happily take on enormous toxic inventory believing it is fully hedged, then get adversely filled on both legs when the correlation fails. You can be adversely selected on nearly anything, and a hedge built on an assumed correlation is one more thing to be adversely selected on. The defense against toxic flow is pricing it out through skew and spread, not netting it against a correlation you do not control. ## Visualizing the distinction ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_126-visual-0.png) ## KEY POINTS - The textbook frames market making as inventory risk and builds machinery to flatten positions. That puts effort in the wrong place: holding some inventory is benign, holding a lot of toxic inventory is what kills the book. - Inventory from uninformed flow is just a position with variance and roughly flat expected path. Skewing hard to flatten it gives up spread you did not need to give up. - If flow is clearly non-toxic, do not skew aggressively; hold some risk and absorb even a net imbalance, since it is probably not an informed imbalance. - Toxic inventory carries negative drift, not just variance, and the danger scales with size: an informed trader loading you large at a fixed price hands you a large guaranteed loss. - The markout curve separates the two: flat for non-toxic, a decaying trough for toxic. Equal-size positions can differ entirely in toxicity, measured by loss rate, not size. - Skew's real job is stopping informed traders from loading you with toxic inventory, not textbook variance management. Widen against a counterparty gradually as their aggression (the evidence of toxicity) rises. - Hedging toxic inventory with a correlated instrument is a trap: when the assumed correlation breaks you get adversely filled on both legs. Price toxic flow out with skew and spread instead of netting it against a correlation you do not control. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [The Art of Currency Trading - Brent Donnelly (Amazon)](https://www.amazon.com/s?k=The+Art+of+Currency+Trading+Brent+Donnelly&ref=aligrithm.com) ### 5.6 Adverse Selection Explained for Traders URL: https://aligrithm.com/adverse-selection-explained-for-traders/ Last updated: 2026-07-27T13:25:05.000Z Post a bid and an ask around a price you believe is fair. You have no view on direction, so you sit symmetrically on both sides. Someone with a view does have a direction, and they are usually right, so they hit the side that the market is about to move toward. They lift your offer right before the price climbs, or hit your bid right before it falls, and you are left holding the inventory that loses. That is adverse selection, and it is the permanent battle of maker against taker that decides whether a market making book lives or dies. This article names the enemy that ["Market Making Is Not Just Collecting the Spread"](https://aligrithm.com/market-making-is-not-just-collecting-the-spread/) introduced and that ["Markouts: The Truth Serum of Market Making"](https://aligrithm.com/markouts-the-truth-serum-of-market-making/) measures. Every defense in this pillar exists to fight it. ## Where it comes from The maker quotes both sides because the maker has no forecast of direction, only an estimate of the current center. The taker who crosses the spread is choosing to pay it, and people do not pay the spread at random. They pay it when they expect the move to more than cover the cost, which means they carry information, or speed, or both, that the maker lacks at the moment of the fill. So the maker's two-sided quote is an offer to trade in either direction at a fixed center, and the informed counterparty exercises the side that is mispriced relative to the immediate future. The maker is not unlucky. The maker is structurally on the wrong side of the fills that come from informed flow, because the informed flow self-selects onto the losing side. This is why a symmetric quote with no prediction bleeds: it hands the taker a free option to pick the direction. ## You do not need to predict big moves The fix is not to out-forecast the market on large moves. For a maker the bar is much lower, because the maker pays no spread to enter. A signal too weak to be profitable for a taker, who must cross the spread and overcome fees, can be plenty for a maker, who is collecting the spread instead of paying it. A fraction of a basis point of directional edge, worthless to anyone who has to pay to act on it, lets the maker lean quotes away from the side the informed are about to hit. The relationship is direct: the better you can predict where the market is going, even slightly, the less adverse selection you suffer, because you shade your fair value and your quotes toward the predicted move and the informed trader no longer finds a mispriced side to exercise. ["Why Small Alphas Matter More for Makers Than Takers"](https://aligrithm.com/why-small-alphas-matter-more-for-makers-than-takers/) works through why weak signals are enough on the maker side. ## The defenses Three lines of defense fight adverse selection, each covered in its own article. Forecast the fair value. Move your center toward where price is heading so the option you are handing the taker is no longer mispriced. The microstructure features do this work: top-of-book imbalance, signed trade flow, short-horizon volatility. ["Why Fair Value Is the Core of Market Making"](https://aligrithm.com/why-fair-value-is-the-core-of-market-making/) builds the center and ["Order Book Imbalance: The First Microstructure Feature to Test"](https://aligrithm.com/order-book-imbalance-the-first-microstructure-feature-to-test/) builds the strongest single feature. Widen where the markouts are bad. Your markout curve marks exactly the conditions, instruments, and sides where adverse selection is eating you, so you widen spreads there and charge the informed more for the option. ["Spread Widening During Volatility Expansion"](https://aligrithm.com/spread-widening-during-volatility-expansion/) runs this on volatility. Skew against toxic loading. Lean your quotes so an aggressive informed trader cannot pile a large toxic position onto you at a fixed price. The point of skew is not ordinary inventory management, it is refusing to be loaded up by the informed. ["Toxic Flow vs Inventory Risk"](https://aligrithm.com/toxic-flow-vs-inventory-risk/) makes that distinction the center of its argument. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Visualizing adverse selection ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_125-visual-0.png) ## KEY POINTS - Adverse selection: a maker quotes both sides with no directional view, and informed takers hit whichever side the market is about to move toward, leaving the maker with the inventory that loses. - People do not pay the spread at random. They pay it when they expect the move to cover the cost, so paid spread carries information or speed the maker lacks at the moment of the fill. - A symmetric quote with no prediction hands the taker a free option to pick the direction. The maker is structurally on the wrong side of informed fills, not merely unlucky. - The fix is not out-forecasting large moves. A maker pays no spread to enter, so a signal too weak for a taker is enough to shade quotes away from the informed side. - The better you predict direction, even slightly, the less adverse selection you suffer, because the informed trader no longer finds a mispriced side to exercise. - Three defenses: forecast the fair value to remove the mispriced option, widen spreads where markouts are bad, and skew to refuse being loaded with toxic inventory. - Adverse selection is the permanent maker-versus-taker battle; every microstructure feature and quoting rule in this pillar exists to fight it. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [The Art of Currency Trading - Brent Donnelly (Amazon)](https://www.amazon.com/s?k=The+Art+of+Currency+Trading+Brent+Donnelly&ref=aligrithm.com) ### 5.5 Why Forecast Accuracy Is Not Enough in Market Making URL: https://aligrithm.com/why-forecast-accuracy-is-not-enough-in-market-making/ Last updated: 2026-07-27T13:25:05.000Z A taker who builds a price forecast measures it the obvious way: R-squared, mean squared error, directional accuracy across all the moves in the sample. A higher number is a better forecast. For a market maker that scoring is not just incomplete, it is measuring the wrong sample, and a model that looks excellent by it can lose money in production. The reason is brutal and simple: most of the moves your forecast gets right are moves you will never get filled on. This article is the warning that ["Why Fair Value Is the Core of Market Making"](https://aligrithm.com/why-fair-value-is-the-core-of-market-making/) deferred. It changes what "a good forecast" means once the forecast is attached to passive quotes rather than aggressive orders. ## The fill is conditional, and the conditioning is adverse A taker who likes a forecast crosses the spread and gets the trade. A maker posts a quote and waits, and whether the quote fills is decided by someone else, the trader who chooses to hit it. That someone tends to hit you precisely when they think the price is about to move against your fill. The fills you get are not a random sample of the moves your model predicted. They are a sample selected by counterparties to be the ones where you are wrong. So the relevant accuracy is not your accuracy across all moves. It is your accuracy conditional on getting filled, and the conditioning event is hostile. Optimizing R-squared over the full distribution optimizes performance on a population you mostly do not trade, while ignoring the population you do. ## The arithmetic of conditional accuracy Put numbers on it. Suppose your forecast is right 40% of the time across all moves, which sounds like a real edge for a directional model. Now look only at the fills you actually get. On those fills, the forecast is wrong 70% of the time, because the people hitting you selected the adverse ones. $$ \\text{accuracy}\_{\\text{unconditional}} = 40\\%, \\qquad \\text{accuracy}\_{\\text{on fills}} = 100\\% - 70\\% = 30\\% $$ Your real accuracy is 30%, not 40%, because being right on a move you never traded pays nothing. The full-sample number flattered you by ten points, and those ten points were exactly the moves you missed. A forecast that scores worse on R-squared but is right more often on the fills it actually receives is the better market making forecast, full stop. ## What to optimize instead Two changes follow. First, score the forecast on the conditional sample. Simulate or record which quotes would have filled, and evaluate accuracy and markout on that subset alone. The markout, the change in price after a fill from ["Markouts: The Truth Serum of Market Making"](https://aligrithm.com/markouts-the-truth-serum-of-market-making/), is the natural objective here because it is already conditioned on fills by construction. A forecast that produces good average markouts is by definition right on the moves you trade. Second, build forecasts you get filled into, not forecasts that are merely correct. The goal is a model whose predictions line up with the conditions under which your quotes actually trade, even at the cost of being blind to large moves you would never have caught passively anyway. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Blending horizons by their markout curves A practical consequence: you usually run several forecasts on different horizons, and you blend them by their markout contribution per unit time rather than by their headline accuracy. Take each forecast's markout curve, its edge measured second by second after a fill, and average the curves. A worked blend: one forecast carries 2 bps of edge but only fires on moves of at least 30 bps, which over a one-minute horizon averages to roughly 0.5 bps per second of usable edge; another forecast contributes around 2 bps per second on the first parcel of the move. Layer them and you stack many forecast horizons into one quoting decision, each weighted by the edge it delivers on fills per second rather than by how well it fits the full sample. The headline accuracy of any single model is not the input. The per-second markout it earns on real fills is. ## Visualizing the conditional sample ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_124-visual-0.png) ## KEY POINTS - Takers score a forecast on R-squared, MSE, or accuracy across all moves. For a maker that scores the wrong sample, because most moves you predict correctly are moves you never get filled on. - A maker's fills are not a random sample. The counterparty chooses to hit you when the price is about to move against your fill, so the fills you receive are selected to be the adverse ones. - The relevant metric is accuracy conditional on getting filled, and the conditioning event is hostile. Optimizing full-sample R-squared optimizes a population you do not trade. - Worked example: 40% accuracy across all moves can become 30% on the fills you actually get when the forecast is wrong 70% of the time on those fills. The ten-point gap is the moves you missed. - Score forecasts on the conditional (filled) sample, and use markout as the objective since it is already conditioned on fills by construction. - Build forecasts you get filled into, not forecasts that are merely correct across the full distribution, even at the cost of missing large moves you would never catch passively. - Blend multiple-horizon forecasts by their markout edge per second on real fills, not by headline accuracy. A 2 bps edge that fires only on 30 bps moves averages to about 0.5 bps/sec over a minute. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [The Art of Currency Trading - Brent Donnelly (Amazon)](https://www.amazon.com/s?k=The+Art+of+Currency+Trading+Brent+Donnelly&ref=aligrithm.com) ### 5.4 Markouts: The Truth Serum of Market Making URL: https://aligrithm.com/markouts-the-truth-serum-of-market-making/ Last updated: 2026-07-27T13:25:04.000Z A market maker can tell a flattering story about any fill. The spread was captured, the quote was tight, the volume was good. The markout ignores the story and asks one question: what did the price do right after the fill printed? Measure the change in price in the seconds following each of your trades and the flattering story collapses into a number that is either positive or negative. That number is the truth serum, because it does not care how clever your quote looked, only whether the market ran through it. This article defines the measurement that ["Why Fair Value Is the Core of Market Making"](https://aligrithm.com/why-fair-value-is-the-core-of-market-making/) leaned on and that "Toxicity and Adverse Selection" in the order-book material treats as the central diagnostic. Markouts sit under almost every later refinement in this pillar. ## What a markout is A markout is the change in price after a fill, measured at a chosen horizon. Record the price you filled at, look at a reference price (the mid is the usual choice) some interval later, and take the difference, signed by the side you traded so that a profitable outcome is positive. $$ \\text{markout}(\\tau) = s \\cdot \\left( \\text{mid}\_{t+\\tau} - p\_{\\text{fill}} \\right), \\quad s = +1 \\text{ if you bought}, \\; -1 \\text{ if you sold} $$ The sign s flips the difference so a buy that is followed by a rising mid scores positive and a sell followed by a falling mid scores positive. Compute this for every fill, average across many fills, and you have the average markout, the cleanest available estimate of whether your quotes are centered on the future or the past. ## Read it as a curve, not a point A single horizon hides the shape. Compute the markout at a sweep of horizons (50 ms, 100 ms, 250 ms, 750 ms, 2 s) and plot the average against the horizon. The curve is the story of what your inventory does to you over time. The classic adverse-selection signature is a curve that starts near zero and decays downward: right after the fill you look flat, and as the seconds pass the price keeps moving against you, the loss deepening as the informed move you were on the wrong side of plays out. The depth of the trough and the horizon at which it bottoms tell you how toxic the flow is and over what timescale the damage lands. That bottoming horizon is the same one that defines where you should center fair value, which is why the two articles share the 750 millisecond figure. ## Why it is the truth serum Markouts strip out the two things traders use to flatter themselves. They ignore the spread you collected, since the markout measures only the post-fill price path, so a fill that banked 2 bps of spread but lost 10 bps to an adverse move shows up as the loser it is. And they ignore intent, since the price does not know you meant to be clever. This makes the markout the validator that overrides the quoted spread. A book can show a healthy gross spread and a deeply negative average markout at the same time, which means it is collecting pennies and losing dollars to the people picking it off. No other single number catches that as fast. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## What you do with it Markouts are not only a scorecard, they are a control input. Three uses run through this pillar. First, fair-value placement. The horizon where the markout curve bottoms is the horizon your fair value should forecast to, so a desk reads the markout shape to decide where to center quotes. Second, defense. Widen spreads on the instruments, sides, or conditions where your markouts are worst, since a bad markout marks exactly the flow you should be charging more to interact with. ["Spread Widening During Volatility Expansion"](https://aligrithm.com/spread-widening-during-volatility-expansion/) runs this logic on the volatility dimension. Third, instrument selection. Track the rolling average markout (a five-second markout works) of every trade in a symbol as an EWMA, before you ever quote it, and the symbols that sustain genuinely good markouts are the non-toxic ones worth quoting. ["Dynamic Symbol Selection for Market Makers"](https://aligrithm.com/dynamic-symbol-selection-for-market-makers/) turns this into a live filter, and the same signal can scale your size up on names you make money on and cut it fast on names you do not. ## Visualizing the markout curve ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_123-visual-0.png) ## KEY POINTS - A markout is the change in price after a fill at a chosen horizon, signed by your side so a favorable outcome is positive. Averaged across many fills it estimates whether your quotes are centered on the future or the past. - Read markouts as a curve across horizons (50 ms to a few seconds), not a single point. The shape is the story of what your inventory does to you over time. - The adverse-selection signature is a curve starting near zero and decaying into a trough: the price keeps moving against you as the informed move plays out. Trough depth and timing measure toxicity. - Markouts are the truth serum because they ignore the spread you collected and the intent behind the quote. A book can show a healthy gross spread and a deeply negative markout at once. - The horizon where the markout bottoms is the horizon your fair value should forecast to, linking markouts directly to where you center quotes. - Use markouts as a control: place fair value at the trough horizon, widen spreads where markouts are worst, and select instruments by tracking the rolling EWMA markout of every trade before you quote. - The same signal scales size up on names you make money on and cuts it fast on names you lose on. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [The Art of Currency Trading - Brent Donnelly (Amazon)](https://www.amazon.com/s?k=The+Art+of+Currency+Trading+Brent+Donnelly&ref=aligrithm.com) ### 5.3 Why Fair Value Is the Core of Market Making URL: https://aligrithm.com/why-fair-value-is-the-core-of-market-making/ Last updated: 2026-07-27T13:25:04.000Z Ask a market maker what fair value is and the honest answer is not philosophical. Fair value is the price that, when you quote around it, gives you good markouts. Nothing about intrinsic worth, nothing about discounted cash flows. A number is fair if your fills around it do not bleed in the seconds after they print. That operational definition is the whole reason fair value sits above spread and skew in importance: it is the only one of the three pillars defined directly by the profit it produces. The previous article, ["The Three Pillars of Market Making: Fair Price, Spread, Skew"](https://aligrithm.com/the-three-pillars-of-market-making-fair-price-spread-skew/), ranked fair price first. This one builds it. ## Define fair value by markouts, not by belief Markout is the change in price after one of your fills, measured at some horizon. Fill a sell, look at the mid price 750 milliseconds later, and the difference is your markout on that trade. Good markouts mean the price did not run away from your fills. Bad markouts mean it did, and you were quoting around the wrong center. The horizon is where the definition gets sharp. Suppose your losses on adverse fills tend to fully realize about 750 milliseconds after you trade. Then fair value is not today's mid price. It is roughly where you expect the market to be 750 milliseconds from now, because that is the price your fills will be marked against once the dust settles. Quote around the present when your pain horizon is the near future, and you are systematically late. $$ \\text{markout}(\\tau) = \\text{mid}\_{t+\\tau} - p\_{\\text{fill}, t} $$ The markout at horizon tau is the mid price tau later minus the price you filled at. Average it across many fills and a center that produces a markout near zero (after accounting for the side you traded) is a fair center; a center that produces a persistently negative markout is mis-placed, and the size of that negative number is the rate you are bleeding per fill. ## A first fair value with no forecast The simplest workable fair value uses no forecast at all. Take the mid price of Binance, the venue with the dominant share, and quote around it. It costs nothing to compute and it beats quoting around your own small venue's mid by a wide margin. Improve it one step by replacing a single venue's mid with a volume-weighted mid across the top exchanges, so a thin or stale book on any one venue does not drag your center. A further refinement filters which moves to follow: some moves on other venues are informed and will pull the global fair price, others are noise that will not, and a careful version reacts to the first kind and ignores the second. None of this forecasts the future. It only builds a clean estimate of the present, and that alone is a reasonable starting center. The contrast that makes the point: quote on a small exchange and assume Bitcoin is worth whatever that exchange's mid says, and you have started from a bad number. The small venue follows the large one with a lag, so its mid is a stale shadow of the real center, and every fast trader who sees the large venue move first picks you off before your stale mid catches up. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Adding a forecast on top A real edge layers a forecast onto the present estimate. The center becomes the current price plus a prediction of where the price will sit a few hundred milliseconds ahead, so traders with a good short-horizon forecast cannot lift the side you have mis-priced. Building the forecast is a standard machine-learning problem. Take features that predict the near-future price and fit whatever maps them to a prediction: ridge regression, OLS, gradient boosting. Three simple features start the work. Top-of-book imbalance, the relative size resting on the bid versus the ask, is by far the strongest. Trade-volume imbalance and short-horizon volatility help but need more care to extract signal from. ["Order Book Imbalance: The First Microstructure Feature to Test"](https://aligrithm.com/order-book-imbalance-the-first-microstructure-feature-to-test/) builds the strongest of these, and ["Using Trade Flow to Predict Short-Term Price Movement"](https://aligrithm.com/using-trade-flow-to-predict-short-term-price-movement/) builds the trade-based one. One warning carries over from the forecasting articles in this pillar. The goal is not a forecast with a high R-squared across all moves. It is a forecast that is accurate conditional on the moves where you actually get filled, which is a different and harder target. ["Why Forecast Accuracy Is Not Enough in Market Making"](https://aligrithm.com/why-forecast-accuracy-is-not-enough-in-market-making/) handles that distinction. ## Visualizing fair value ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_122-visual-0.png) ## KEY POINTS - Fair value is operational, not philosophical: the price that, when you quote around it, gives good markouts. A number is fair if your fills do not bleed in the seconds after they print. - Markout is the change in price after a fill at some horizon. Average it across fills; a center near zero markout is fair, a persistently negative one is mis-placed and tells you the rate you are bleeding. - If your adverse losses realize about 750 ms after a fill, fair value is roughly where you expect price to be 750 ms ahead, not today's mid. Quoting around the present when your pain is in the near future leaves you late. - A first fair value needs no forecast: quote around the dominant venue's mid (Binance for crypto), then upgrade to a volume-weighted mid across top exchanges so a stale single book does not drag the center. - Treating a small venue's own mid as the value of the asset starts you from a bad number, because the small venue lags the large one and fast traders pick off your stale center. - A real edge adds a forecast: current price plus a prediction of price a few hundred ms ahead. It is a standard ML problem; top-of-book imbalance is the strongest simple feature, with trade imbalance and volatility behind it. - The forecast target is accuracy conditional on the moves you get filled on, not a high R-squared across all moves. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [The Art of Currency Trading - Brent Donnelly (Amazon)](https://www.amazon.com/s?k=The+Art+of+Currency+Trading+Brent+Donnelly&ref=aligrithm.com) ### 5.2 The Three Pillars of Market Making: Fair Price, Spread, Skew URL: https://aligrithm.com/the-three-pillars-of-market-making-fair-price-spread-skew/ Last updated: 2026-07-27T13:25:03.000Z Every two-sided quote a market maker posts comes out of three decisions stacked on top of each other. Pick a center price you believe is fair. Add a spread on each side to set how wide you sit. Lean the whole thing left or right depending on the inventory you already carry. Fair price, spread, skew. That is the entire machine, and the order matters because the three pillars carry wildly different weight. The previous article, ["Market Making Is Not Just Collecting the Spread"](https://aligrithm.com/market-making-is-not-just-collecting-the-spread/), argued the job is forecasting and survival rather than passive spread capture. This one names the three controls that do the work and ranks them by how much they decide your profit. ## Pillar one: fair price, and why it dominates Fair price is the center you quote around, and it carries more of the result than the other two combined. The working definition is operational: fair price is the price that, when you quote around it, gives you good markouts. Markout is the change in price after one of your fills, the measure of whether your quote was centered on the future or on the past. Quote around a number that the market drifts away from and every fill bleeds; quote around a number the market sits at and your fills break even before you collect any spread. A first version of fair price needs no forecast at all. Take the mid price of the most liquid venue, Binance for most crypto, and quote around that. Improve it by using a volume-weighted mid across the top exchanges instead of one venue's mid. A real edge layers a forecast on top: the current price plus a prediction of where the price will sit a few hundred milliseconds from now, so the fast traders with good forecasts cannot pick you off by quoting a better center than yours. The failure mode is concrete. Quote on a small exchange and treat that exchange's own mid as the value of Bitcoin, and you have started from a bad number, because the small venue lags the large one. ["Why Fair Value Is the Core of Market Making"](https://aligrithm.com/why-fair-value-is-the-core-of-market-making/) works through how to build the center properly, and ["Cross-Exchange Fair Value for Crypto Perps"](https://aligrithm.com/cross-exchange-fair-value-for-crypto-perps/) handles the multi-venue case. ## Pillar two: spread, the width dial Spread is how wide you sit around the fair price, and it controls the trade-off between how often you get filled and how much you make per fill. The most basic version is a fixed markup on the skewed fair value. The next version widens the markup when volatility rises. The rule is to blow your spreads out when the market's volatility does. Volatility means larger moves between now and whenever you can adjust, which means more chance an informed trader runs through your quote before you reprice. Wider spreads buy you protection during the exact moments when getting picked off costs the most. Advanced desks attach a different fair value to each quoting depth, because a quote sitting at the top of the book worries about getting picked off in the next minutes while a deep quote worries about longer-horizon forecasts. ["Spread Widening During Volatility Expansion"](https://aligrithm.com/spread-widening-during-volatility-expansion/) covers the width-control logic in full. ## Pillar three: skew, and why it is the trivial one Skew makes one side of your quote wider than the other so that one side fills more than the other, which lets you push inventory on or take it off. Short too much? Skew so your bid is more aggressive and your ask backs away, and the next fills rebuild your position toward flat. The academic literature treats skew as the hard problem, the place for elaborate inventory-control equations. In practice it is the most trivial of the three pillars and needs no complex model. Most people think skewing manages inventory risk. The risk of holding some inventory is not the real problem. Holding toxic inventory, and a lot of it, is the problem, and skew earns its keep by stopping informed traders from loading you up. You do not need a fancy skewing model. Keep it simple. ["Why Skewing Is Simpler Than People Think"](https://aligrithm.com/why-skewing-is-simpler-than-people-think/) and ["Toxic Flow vs Inventory Risk"](https://aligrithm.com/toxic-flow-vs-inventory-risk/) take this apart. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Stacking the three into a quote The construction runs in order. Start with the fair price, your best estimate of the center including any forecast. Apply skew around that center based on your current position, so your aversion to inventory leans the quotes. Add the spread on each side of the skewed center to get the actual buy and sell prices you post. Often a desk runs multiple quoters at once with different settings, a grid of quotes at different depths, each with its own fair value and width. The ranking is the lesson. Fair price first, spread second, skew a distant third. Effort spent perfecting the skew model while the fair price is mis-centered is effort spent polishing the least important pillar while the most important one leaks. ## Visualizing the stack ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_121-visual-0.png) ## KEY POINTS - Every quote is three stacked decisions: fair price (the center), spread (the width), and skew (the lean based on inventory). The order reflects how much each decides your profit. - Fair price dominates and carries more than the other two combined. Operationally it is the price that gives good markouts when you quote around it. - A first fair price is just the most-liquid venue's mid; a better one is a volume-weighted mid across top exchanges; a real edge adds a forecast of where price sits a few hundred milliseconds ahead. - Spread sets the fill-frequency versus profit-per-fill trade-off. Widen it when volatility rises, since that is when getting picked off costs the most. Advanced desks attach a different fair value to each depth. - Skew leans one side wider to push inventory on or off. The literature treats it as the hard problem; in practice it is trivial and needs no complex model. - Skew's real job is blocking informed traders from loading you with toxic inventory, not managing ordinary inventory risk. Keep it simple. - Construction order: start at fair price, apply skew for your position, then add spread on each side. Desks often run a grid of quoters at different depths, each with its own fair value and width. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [The Art of Currency Trading - Brent Donnelly (Amazon)](https://www.amazon.com/s?k=The+Art+of+Currency+Trading+Brent+Donnelly&ref=aligrithm.com) ### 5.1 Market Making Is Not Just Collecting the Spread URL: https://aligrithm.com/market-making-is-not-just-collecting-the-spread/ Last updated: 2026-07-27T13:25:03.000Z The picture most retail traders carry of a market maker is a patient clerk who posts a bid and an ask, sells to whoever crosses the spread, buys from whoever hits the bid, and pockets the difference forever. Quote 99 bid, 101 ask, collect two dollars per round trip, repeat a million times. That picture is wrong in the way that matters, and the gap between it and the real job is where the entire profit-and-loss of a market making desk lives. The clerk picture assumes the flow is symmetric and dumb. It is neither. The trader who lifts your offer often knows something you do not, and a second later the price has moved through your fill and left you holding inventory at a loss larger than the spread you collected. Posting a two-sided quote is the easy part. Surviving the people who choose which side to hit is the whole game. This article frames what market making actually is before the later articles in this pillar pull apart its pieces. It sits next to ["The Three Pillars of Market Making: Fair Price, Spread, Skew"](https://aligrithm.com/the-three-pillars-of-market-making-fair-price-spread-skew/), which names the components, and ["Adverse Selection Explained for Traders"](https://aligrithm.com/adverse-selection-explained-for-traders/), which names the enemy. ## The spread is a gross number, not a net one Start with the arithmetic that the clerk picture gets right and then ruins. You quote a bid and an ask around some center price. The full width between them is your gross edge per round trip: buy at the bid, sell at the ask, bank the difference. On a perfectly random walk with uninformed counterparties on both sides, you would earn half the spread per fill and never carry meaningful inventory. Real flow does not arrive at random. Suppose you quote a 4 bps spread and a fast trader with a genuine forecast lifts your ask right before the price climbs 10 bps. You collected 2 bps of spread on that fill and you are now short into a rising market, marked down 10 bps. One adversely selected fill erased five spreads of honest profit. The gross spread is real; the net spread after the people who pick you off is what you keep, and it can be negative even when the quoted spread looks generous. ## What the desk is actually optimizing A market making book is not collecting a spread. It is running a forecast of the fair price, posting quotes around that forecast, and shading those quotes to avoid the informed and attract the uninformed. Three things drive the whole result, in descending order of how much they matter: where you center your quotes (fair price), how wide you make them (spread), and how you lean them based on the inventory you already carry (skew). The academic literature inverts this ordering. Most published market making research obsesses over the skewing problem, the inventory-control equations that decide how to lean your quotes. In practice skew is the most trivial of the three. Get the fair price wrong and no amount of clever inventory math saves you, because you are quoting around the wrong number and every fill is a small donation to whoever quoted around the right one. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Most of the money is positional, not micro There is a second, larger thing the clerk picture misses. A great deal of market making profit is not made on the second-to-second spread capture at all. It is made by treating market making as a delivery mechanism for medium-frequency alphas, what the desks call positional market making. The mechanism is simple once you see it. When your cost to trade is zero, because you are earning the spread rather than paying it, signals that would never survive transaction costs as a taker suddenly become tradable. A weak ranking signal that predicts tomorrow's returns with a thin edge is worthless if you have to cross the spread to express it. Skew your passive quotes toward that signal and you express the same view while collecting spread instead of paying it. Stack thousands of these weak signals, skew into all of them, and ensemble the result. That stack, not the tick-by-tick spread, is how the large firms generate the bulk of the number. ["Why Small Alphas Matter More for Makers Than Takers"](https://aligrithm.com/why-small-alphas-matter-more-for-makers-than-takers/) works through the cost arithmetic that makes this possible, and ["Maker vs Taker Edge: Same Signal, Different Economics"](https://aligrithm.com/maker-vs-taker-edge-same-signal-different-economics/) shows the same signal paying off for a maker and dying for a taker. ## Visualizing the difference ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_120-visual-0.png) ## KEY POINTS - The clerk picture (post a bid and ask, collect the difference forever) assumes symmetric, uninformed flow. Real flow is chosen by counterparties, and the one who hits you often knows the price is about to move through your fill. - The quoted spread is a gross number. One adversely selected fill into a real move can erase several spreads of honest profit, so net spread after pick-offs is what you keep and it can be negative. - A desk optimizes three things in descending importance: fair price (where to center quotes), spread (how wide), and skew (how to lean given inventory). Academic research over-weights skew, the most trivial of the three. - Get the fair price wrong and no inventory math saves you; every fill is a donation to whoever quoted around the right number. - Much of the profit is positional, not micro: market making delivers medium-frequency alphas at zero trading cost, so weak signals that die for a taker become tradable for a maker. - The large firms run thousands of small alphas, skew passive quotes into all of them, and ensemble the result. That stack, not tick-by-tick spread capture, is the bulk of the number. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [The Art of Currency Trading - Brent Donnelly (Amazon)](https://www.amazon.com/s?k=The+Art+of+Currency+Trading+Brent+Donnelly&ref=aligrithm.com) ### 4.34 Currency Strength Models from Pair Decomposition URL: https://aligrithm.com/currency-strength-models-from-pair-decomposition/ Last updated: 2026-07-27T13:24:31.000Z The previous article cancelled the dollar out of two pairs to isolate EUR versus GBP. The natural next question: why stop at two. Every currency you care about appears in many pairs at once, and each pair is a tangle of two currencies moving against each other. If subtracting two pairs decouples three currencies, then solving the whole web of pairs simultaneously should decouple every currency, handing you a single strength number for each one with all the other currencies divided out. That is a currency strength model, and it gives you X-ray vision through the quoted prices: instead of EURUSD, GBPUSD, USDJPY, you see EUR, GBP, USD, JPY as separate, decoupled objects. The motivation is that a quoted pair never tells you which currency moved. EURUSD up could be EUR strong, USD weak, or both. A trader who wants to trade EUR (not EUR-against-some-specific-dollar) needs EUR's own strength, separated from whatever the dollar is doing, and that is exactly what pair decomposition produces. Once you have per-currency strength, you can build an index for each currency, rank them, and trade the strongest against the weakest. This article builds the currency strength model, generalizing the two-pair cancellation from ["Cross-Pair Signals: Can EUR Predict GBP?"](https://aligrithm.com/cross-pair-signals-can-eur-predict-gbp/) to the whole cross matrix. It is the FX endpoint of the decomposition idea and a cousin of the learned-network view in ["From Intermarket Analysis to Network Momentum"](https://aligrithm.com/from-intermarket-analysis-to-network-momentum/), both of which extract structure hidden inside a wall of correlated prices. ## One pair is underdetermined Start with why a single pair cannot be decomposed. Express a pair's change over a period as the product of two per-currency multipliers: the old EURUSD quote times an EUR multiplier over a USD multiplier equals the new quote. $$ \\left(\\frac{\\text{EUR}}{\\text{USD}}\\right)\\left(\\frac{E}{U}\\right) = \\frac{\\text{eur}}{\\text{usd}} $$ The old EURUSD quote times the ratio of the EUR multiplier E to the USD multiplier U gives the new quote. The problem is that only the ratio E over U is determined by the pair's move. A 0.8% rise in EURUSD could be E equal to 1.008 with U equal to 1 (all EUR strength), or E equal to 1 with U equal to 0.992 (all USD weakness), or any combination in between. One equation, two unknowns: the pair fixes the ratio and leaves the individual strengths free. This is the same identification problem the two-pair subtraction faced, and the solution is the same: add more pairs. ## Many pairs make it solvable Add more pairs and each one contributes another equation in the same shared set of currency multipliers. EURUSD constrains E over U; GBPUSD constrains G over U; EURGBP constrains E over G; and so on across every cross you have data for. $$ \\left(\\frac{\\text{EUR}}{\\text{USD}}\\right)\\!\\left(\\frac{E}{U}\\right) = \\frac{\\text{eur}}{\\text{usd}}, \\quad \\left(\\frac{\\text{GBP}}{\\text{USD}}\\right)\\!\\left(\\frac{G}{U}\\right) = \\frac{\\text{gbp}}{\\text{usd}}, \\quad \\left(\\frac{\\text{EUR}}{\\text{GBP}}\\right)\\!\\left(\\frac{E}{G}\\right) = \\frac{\\text{eur}}{\\text{gbp}} $$ Three pairs in three currency multipliers E, G, and U are now jointly constrained: the EURGBP equation ties E and G together, GBPUSD ties G and U, and EURUSD ties E and U, so the system pins down the individual strengths up to one overall normalization. Extend this to the whole market (dozens of crosses across all the currencies you trade) and you have many equations in relatively few currency multipliers, an over-determined system. Because the real market does not satisfy all equations exactly (quotes are noisy and not perfectly synchronous), you do not solve it by substitution. You hand it to a numerical optimizer: define an error function over a candidate set of currency multipliers that measures how badly they reproduce all the observed pair quotes, start every multiplier at 1 (no change), and minimize. The optimizer returns the set of per-currency multipliers that best explains the entire pair matrix at once. ## From multipliers to strength indices Once you have each currency's multiplier, the strength signal is one step away. Take the log of a currency's multiplier and it is that currency's decoupled log return over the period, its move with every other currency divided out. $$ \\log(E) = \\log\\!\\left(\\frac{\\text{eur}}{\\text{EUR}}\\right) = \\text{decoupled log return of EUR} $$ The log of the EUR multiplier equals the log return of EUR's own strength, independent of the dollar, the pound, or anything else it is normally quoted against. Accumulate these per-period log returns and you get a strength index for each currency, a clean price-like chart of EUR itself, GBP itself, USD itself. This is the X-ray view: under a EURUSD chart that rose, you can now see whether EUR climbed, the dollar sank, or both, and by how much each. With per-currency strength indices in hand, the toolbox opens. Apply ordinary trend and momentum analysis to a single currency's index. Rank all currencies by strength and trade the strongest against the weakest as a single pair, which concentrates the relative-strength edge the cross-pair article found between just two. Feed the decoupled returns into intermarket currency analysis or a model, the same structure-extraction the network-momentum article used on a learned graph. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## The honest limits The decomposition is powerful and it is an estimate, with the failure modes every estimate carries. It is only identified up to a normalization. The system fixes the relative strengths, not an absolute level, so you must impose a constraint (fix a numeraire, or require the average strength change across currencies to be zero) to pin the solution. Different normalizations give different-looking indices, and comparing two strength models built on different conventions is comparing apples to oraches. The choice is arbitrary and must be stated, the kind of acknowledged arbitrary choice the house style flags rather than hides. It depends on the basket of pairs. The strength of a currency is estimated from the pairs you fed in, so a model built on G10 crosses gives different numbers from one that includes emerging-market pairs, and a currency thinly represented in the basket gets a noisy, poorly identified strength. The decomposition is a property of the basket as much as of the currency. It is non-stationary and not free to trade. The relationships drift, so the model must be re-estimated continuously, and the strongest-versus-weakest trade it suggests still pays spreads and slippage on real pairs, often on the less liquid crosses where the strength extremes live, the cost constraint from ["Why Transaction Costs Should Be Added Before You Fall in Love"](https://aligrithm.com/why-transaction-costs-should-be-added-before-you-fall-in-love/). A strength index is a clean signal sitting on top of a messy, costly execution surface, and the cleanliness of the signal does not reduce the cost of acting on it. ## Visualizing the decomposition ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_119-visual-0.png) ## KEY POINTS - A quoted pair never tells you which currency moved: EURUSD up could be EUR strong, USD weak, or both. A currency strength model decouples each currency from the pairs it is quoted in, giving one strength number per currency. - A single pair is underdetermined: it fixes only the ratio of its two currencies' multipliers, leaving the individual strengths free. One equation, two unknowns. - Adding pairs adds equations in the same shared currency multipliers. Across the whole cross matrix you get an over-determined system that pins down each currency's strength up to one overall normalization. - Because real quotes are noisy and asynchronous, you do not solve by substitution. A numerical optimizer minimizes an error function measuring how badly a candidate set of multipliers reproduces all observed pair quotes, starting every multiplier at 1. - The log of a currency's multiplier is its decoupled log return, its move with every other currency divided out. Accumulating these gives a clean strength index per currency, X-ray vision through the quotes. - With per-currency strength you can apply trend and momentum to a single currency, rank all currencies and trade the strongest against the weakest, or feed decoupled returns into intermarket analysis or a model. - It is identified only up to a normalization, so you must impose a constraint (a numeraire or zero-average-change), and the choice is arbitrary and must be stated. Different conventions give different-looking indices. - It depends on the basket of pairs, is non-stationary and must be re-estimated continuously, and the strongest-versus-weakest trade still pays real spreads and slippage, often on the less liquid crosses where the extremes live. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Trading Systems and Methods - Perry Kaufman (Amazon)](https://www.amazon.com/s?k=Trading+Systems+and+Methods+Perry+Kaufman&ref=aligrithm.com) - [Cybernetic Trading Strategies - Murray Ruggiero (Amazon)](https://www.amazon.com/s?k=Cybernetic+Trading+Strategies+Murray+Ruggiero&ref=aligrithm.com) - [The Art of Currency Trading - Brent Donnelly (Amazon)](https://www.amazon.com/s?k=The+Art+of+Currency+Trading+Brent+Donnelly&ref=aligrithm.com) ### 4.33 Cross-Pair Signals: Can EUR Predict GBP? URL: https://aligrithm.com/cross-pair-signals-can-eur-predict-gbp/ Last updated: 2026-07-27T13:24:31.000Z EURUSD and GBPUSD move together. Watch them on the same screen and they look like two copies of one chart, rising and falling almost in lockstep. The obvious trade idea: EUR leads, so a move in EURUSD predicts the next move in GBPUSD, trade GBP off EUR. The idea is mostly wrong, and the reason it is wrong is the most important thing to understand about cross-pair signals. EURUSD and GBPUSD share the dollar. Both are "something divided by USD", so when the dollar moves, both pairs move the same way, and the correlation you see is the shared dollar, not EUR telling you anything about GBP. This is the third-variable confound in its purest form. Two series correlate because a common factor drives both, not because one causes the other, the ice-cream-and-drowning trap from ["Lead-Lag Relationships in Global Markets"](https://aligrithm.com/lead-lag-relationships-in-global-markets/). To get a real cross-pair signal you have to remove the shared dollar and ask whether what is left in EUR predicts what is left in GBP. Done right, that subtraction is one line, and it turns "can EUR predict GBP" into a clean, tradeable question about EUR-versus-GBP relative strength. This article builds the cross-pair signal correctly, using the CMMA momentum primitive from ["CMMA: A Better Momentum Primitive Than Price-minus-MA Alone"](https://aligrithm.com/cmma-a-better-momentum-primitive-than-price-minus-ma-alone/) to cancel the dollar. It follows the co-movement facts from ["Why FX Traders Must Watch Gold, Rates, and Equities"](https://aligrithm.com/why-fx-traders-must-watch-gold-rates-and-equities/) and sets up the full decomposition in ["Currency Strength Models from Pair Decomposition"](https://aligrithm.com/currency-strength-models-from-pair-decomposition/). ## Why the pairs move together Write each pair as what it is: a ratio of two currencies. EURUSD is EUR over USD; GBPUSD is GBP over USD. In log returns that becomes a difference: the return of EURUSD is the strength change of EUR minus the strength change of USD, and the return of GBPUSD is the strength change of GBP minus the strength change of USD. $$ r\_{\\text{EURUSD}} = \\Delta\\text{EUR} - \\Delta\\text{USD}, \\qquad r\_{\\text{GBPUSD}} = \\Delta\\text{GBP} - \\Delta\\text{USD} $$ The EURUSD return is the change in EUR strength minus the change in USD strength; the GBPUSD return is the change in GBP strength minus the change in USD strength. Both contain the same minus-USD term. When the dollar moves, that shared term moves both pairs together, which is the entire source of their correlation. A dollar rally drags EURUSD and GBPUSD down together while telling you nothing about whether EUR is stronger or weaker than GBP. The co-movement is real and it is uninformative, because it is the dollar talking, not EUR. ## Subtracting the dollar The fix falls straight out of the algebra. Subtract the two pair returns and the shared minus-USD term cancels, leaving the EUR-versus-GBP relative move with no dollar in it. $$ r\_{\\text{EURUSD}} - r\_{\\text{GBPUSD}} = \\big(\\Delta\\text{EUR} - \\Delta\\text{USD}\\big) - \\big(\\Delta\\text{GBP} - \\Delta\\text{USD}\\big) = \\Delta\\text{EUR} - \\Delta\\text{GBP} $$ Subtracting the GBPUSD return from the EURUSD return cancels the USD term exactly, leaving the change in EUR strength minus the change in GBP strength. That difference is the EUR-versus-GBP signal, and it is precisely what EURGBP measures, because EURGBP is EUR over GBP with no dollar in it at all. The whole "can EUR predict GBP" question reduces to: is there a tradeable relative-strength relationship between EUR and GBP, which you read off EURGBP or off the difference of the two USD pairs. The practical version uses a normalized momentum so the two legs are comparable. Compute CMMA (close minus moving average, divided by ATR times the square root of the lookback) on each pair, then take the difference: CMMA of EURUSD minus CMMA of GBPUSD. CMMA is volatility-normalized and lookback-invariant, so the subtraction is between two comparable, unitless momentum readings, and the result is a clean relative-strength signal with the dollar removed. $$ \\text{Signal}\_{\\text{EUR vs GBP}} = \\text{CMMA}\_{\\text{EURUSD}} - \\text{CMMA}\_{\\text{GBPUSD}} \\;\\approx\\; \\text{momentum}(\\text{EUR}) - \\text{momentum}(\\text{GBP}) $$ The cross-pair signal is the CMMA of EURUSD minus the CMMA of GBPUSD, which approximates the momentum of EUR minus the momentum of GBP. A positive value says EUR is gaining on GBP regardless of the dollar; a negative value says GBP is gaining on EUR. You trade it through EURGBP, going long EURGBP when EUR's normalized momentum leads and short when GBP's does. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## What the signal is and is not The honest read on what you have built and what it can do. It is a relative-strength rotation between two correlated currencies, the FX analogue of the ratio signal from ["Using Ratios as Trading Signals"](https://aligrithm.com/using-ratios-as-trading-signals/). It is not a prediction that EUR causes GBP; it is a measurement of which of the two is stronger right now with the common dollar driver stripped out. The edge, if any, comes from EUR and GBP responding at slightly different speeds or magnitudes to their own drivers (UK-specific rates and politics for GBP, euro-area rates and politics for EUR), so the relative-strength line trends when one currency's story is running ahead of the other's. The naive version (trade GBPUSD off EURUSD without subtracting the dollar) fails because it is mostly a dollar bet wearing a cross-pair costume. When it appears to work, it works because the dollar trended, and you would have done as well or better trading the dollar directly. The subtraction is what separates a genuine EUR-versus-GBP signal from a disguised dollar position, and skipping it is the single most common cross-pair mistake. ## The traps Where the clean signal still bites. EURGBP is less liquid and wider-spread than the two majors, so the relative-strength edge, which is small to begin with, faces a larger cost hurdle, the cost-versus-edge constraint from ["Why Transaction Costs Should Be Added Before You Fall in Love"](https://aligrithm.com/why-transaction-costs-should-be-added-before-you-fall-in-love/). A signal that looks profitable on mid-price differences can die on the EURGBP spread. The cancellation is exact only if the two pairs share the same dollar move over the same window, which holds for synchronous data but breaks across illiquid hours or asynchronous closes, leaving a residual dollar contamination in the difference. Use synchronous, liquid-hour data or the residual reintroduces the very thing you subtracted out. The relationship drifts. EUR and GBP are correlated because of overlapping European exposure, and that overlap changes with events (a UK-specific shock decouples GBP for a while). The relative-strength signal is non-stationary like every relationship in this pillar, so it needs the rolling re-estimation the article ["Slow Wandering: The Most Dangerous Type of Market Change"](https://aligrithm.com/slow-wandering-the-most-dangerous-type-of-market-change/) demanded. Two currencies that decoupled this year may not give the same signal next year. ## Visualizing the dollar cancellation ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_118-visual-0.png) ## KEY POINTS - EURUSD and GBPUSD move together because they share the dollar: both are something-over-USD, so a dollar move drags both the same way. The correlation is the dollar talking, not EUR predicting GBP. - This is the third-variable confound: two series correlate because a common factor drives both, not because one causes the other. A naive "trade GBP off EUR" signal is mostly a disguised dollar bet. - In log returns each pair is a difference: EURUSD is EUR strength minus USD strength, GBPUSD is GBP strength minus USD strength. Both carry the same minus-USD term. - Subtracting the two pair returns cancels the shared USD term exactly, leaving EUR strength minus GBP strength, which is what EURGBP measures with no dollar in it. - The practical signal: CMMA of EURUSD minus CMMA of GBPUSD. CMMA is volatility-normalized and lookback-invariant, so the difference is between two comparable momentum readings and yields a clean relative-strength signal. - Positive means EUR is gaining on GBP regardless of the dollar; negative means GBP is gaining. Trade it through EURGBP, long when EUR's normalized momentum leads and short when GBP's does. - It is a relative-strength rotation, not a causal prediction. The edge comes from EUR and GBP responding at different speeds to their own drivers, so the relative line trends when one story runs ahead of the other. - Traps: EURGBP is less liquid and wider-spread, raising the cost hurdle on a small edge; the cancellation needs synchronous liquid-hour data or residual dollar contamination returns; and the EUR-GBP overlap drifts, so the signal needs rolling re-estimation. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Trading Systems and Methods - Perry Kaufman (Amazon)](https://www.amazon.com/s?k=Trading+Systems+and+Methods+Perry+Kaufman&ref=aligrithm.com) - [Cybernetic Trading Strategies - Murray Ruggiero (Amazon)](https://www.amazon.com/s?k=Cybernetic+Trading+Strategies+Murray+Ruggiero&ref=aligrithm.com) - [The Art of Currency Trading - Brent Donnelly (Amazon)](https://www.amazon.com/s?k=The+Art+of+Currency+Trading+Brent+Donnelly&ref=aligrithm.com) ### 4.32 Using Gold as an FX Indicator URL: https://aligrithm.com/using-gold-as-an-fx-indicator/ Last updated: 2026-07-27T13:24:30.000Z Pull up the driver list for almost any major currency pair and gold is on it. EURUSD, USDJPY, USDCHF, AUDUSD, USDCAD: gold shows up again and again, which is strange at first glance, because gold is not a currency and most of these pairs have no direct gold connection. The reason gold belongs on all of them is that gold reads the one thing every USD pair shares: the dollar. Gold trades inverse to the dollar, so a move in gold is a move in the dollar seen from a different angle, and that makes gold a single instrument you can watch to read the common factor sitting inside every dollar pair at once. Gold carries a second signal on top of the dollar one. It anticipates inflation and tracks real yields, so gold strength flags either a weakening dollar or a rising inflation-and-easing backdrop, both of which matter for FX. For one currency, the Australian dollar, gold is also a direct export, so it acts through a second channel as a commodity input. One instrument, three overlapping readings: the dollar, the inflation-and-rate backdrop, and the commodity terms of trade for the gold exporters. This article turns gold into a concrete FX indicator, drawing the dollar link from ["Gold, Dollar, and Rates: A Practical Intermarket Map"](https://aligrithm.com/gold-dollar-and-rates-a-practical-intermarket-map/) and the driver framing from ["Why FX Traders Must Watch Gold, Rates, and Equities"](https://aligrithm.com/why-fx-traders-must-watch-gold-rates-and-equities/). It is the gold-specific deep cut of the FX-macro arc, and it leads into the cross-pair and currency-strength articles that close the pillar. ## Gold as the dollar barometer The cleanest use of gold in FX is as a dollar proxy. Gold is priced in dollars and competes with the dollar as a store of value, so the two trade inverse across most regimes. When gold rises, the dollar is usually weakening, and that weakening shows up across every dollar pair at once. $$ \\text{gold} \\uparrow \\;\\Rightarrow\\; \\text{USD} \\downarrow \\;\\Rightarrow\\; \\{\\,\\text{EURUSD} \\uparrow,\\;\\; \\text{AUDUSD} \\uparrow,\\;\\; \\text{USDJPY} \\downarrow\\,\\} $$ Rising gold signals a weaker dollar, which lifts the pairs where the dollar is the quote currency (EURUSD, AUDUSD rise) and lowers the pairs where the dollar is the base (USDJPY falls). The direction flips with the dollar's position in the pair, but the underlying read is one number: gold up means dollar down, applied to whichever side of the pair the dollar sits on. A trader watching gold is watching the dollar's common factor without having to average a basket of dollar pairs by hand. This makes gold a natural confirmer for any dollar trade, the confirmation logic from ["Cross-Asset Confirmation for Trend Systems"](https://aligrithm.com/cross-asset-confirmation-for-trend-systems/) applied to the dollar leg. A short-dollar signal taken while gold is breaking higher is backed by the gold tape; the same signal with gold falling is fighting it, the divergence read from ["Intermarket Divergence as a Trading Filter"](https://aligrithm.com/intermarket-divergence-as-a-trading-filter/). ## The inflation and real-yield reading Gold's second signal is the inflation-and-rate backdrop, and it carries the same subtlety the triangle article flagged: gold tracks real yields, not nominal ones. Rising real yields (nominal rates climbing faster than inflation) hurt gold, because the opportunity cost of holding a zero-yield asset rises. Falling or negative real yields help it. For FX this matters because real yields drive currencies through the rate-differential channel. A gold rally on falling real yields is consistent with a dovish, currency-weakening backdrop for the dollar; a gold selloff on rising real yields is consistent with a hawkish, dollar-strengthening one. The trap is the inflation-spike case: nominal yields can rise while real yields fall, and gold rallies against what a trader watching the nominal ten-year expects. A gold-based FX read built on nominal rates misfires in exactly the high-inflation regimes where the signal matters most, so the honest version uses a real-yield or breakeven proxy alongside gold. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## The commodity-currency channel For the Australian dollar, gold is not only a dollar barometer but a direct export, so it acts through a second, reinforcing channel. Rising gold improves Australia's terms of trade and supports the currency on top of the dollar-weakness effect, which is why AUDUSD is one of the more gold-sensitive pairs. The gold-miner equity index sits adjacent to this: miner stocks track gold and, historically, the European currency too, because they discount the future gold price the currency-and-commodity complex responds to. The practical caution is not to double-count. For AUDUSD a gold rally pushes the same direction through both the dollar channel and the export channel, so the signal is strong but the two channels are correlated, not independent, the confirmation-count caveat from ["Why FX Traders Must Watch Gold, Rates, and Equities"](https://aligrithm.com/why-fx-traders-must-watch-gold-rates-and-equities/). For a pair with no commodity link to gold (EURUSD), only the dollar channel operates, so gold is a pure dollar read there and nothing more. Knowing which channels apply to which pair is what keeps the indicator honest. ## Building the indicator The operational version is a normalized gold trend used as a dollar-direction input. Compute a volatility-normalized momentum on gold (the CMMA-style primitive from ["CMMA: A Better Momentum Primitive Than Price-minus-MA Alone"](https://aligrithm.com/cmma-a-better-momentum-primitive-than-price-minus-ma-alone/) works), and read its sign and strength as the dollar signal: positive gold momentum implies dollar weakness, scaled by how strong the gold trend is. Apply that dollar read to each USD pair with the correct sign for the dollar's position in the pair. Three disciplines keep this from becoming noise. Use a real-yield cross-check so the inflation-spike case does not flip the read. Re-estimate the gold-dollar correlation on a rolling window, because in a genuine crisis gold and the dollar can both rally as safe havens and the inverse link breaks, the regime caveat from the triangle article. And treat gold as a confirmer or tilt, not a standalone entry, because it explains the dollar's common factor and not the pair-specific move, the same modest-edge texture every cross-asset signal in this pillar carries. Gold tells you where the dollar is leaning; it does not tell you the entry price. ## Visualizing gold as the dollar's mirror ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_117-visual-0.png) ## KEY POINTS - Gold appears on nearly every major currency pair's driver list because it reads the one thing every dollar pair shares: the dollar. Gold trades inverse to the dollar, so a gold move is a dollar move seen from another angle. - Watching gold lets you read the dollar's common factor across all USD pairs at once, without averaging a basket by hand. - Gold up means dollar down, applied with the correct sign for the dollar's position in the pair: EURUSD and AUDUSD rise, USDJPY falls. This makes gold a natural confirmer for any dollar trade. - Gold's second signal is the inflation-and-real-yield backdrop. It tracks real yields, not nominal, so a gold rally on falling real yields is a dovish, dollar-weakening read. - The inflation-spike trap: nominal yields can rise while real yields fall and gold rallies against a nominal-rate expectation. Use a real-yield or breakeven proxy so the read does not flip in the regime that matters most. - For the Australian dollar gold is also a direct export, adding a second reinforcing channel, which makes AUDUSD gold-sensitive. The two channels are correlated, not independent, so do not double-count. - For a pair with no commodity link to gold, only the dollar channel operates and gold is a pure dollar read. Knowing which channels apply to which pair keeps the indicator honest. - Build it as a volatility-normalized gold momentum read as a dollar-direction input, cross-checked against real yields, with a rolling correlation check (gold and the dollar can both rally in a crisis), used as a confirmer or tilt rather than a standalone entry. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Trading Systems and Methods - Perry Kaufman (Amazon)](https://www.amazon.com/s?k=Trading+Systems+and+Methods+Perry+Kaufman&ref=aligrithm.com) - [Cybernetic Trading Strategies - Murray Ruggiero (Amazon)](https://www.amazon.com/s?k=Cybernetic+Trading+Strategies+Murray+Ruggiero&ref=aligrithm.com) - [The Art of Currency Trading - Brent Donnelly (Amazon)](https://www.amazon.com/s?k=The+Art+of+Currency+Trading+Brent+Donnelly&ref=aligrithm.com) ### 4.31 Why FX Traders Need Macro but Should Trade Systematically URL: https://aligrithm.com/why-fx-traders-need-macro-but-should-trade-systematically/ Last updated: 2026-07-27T13:24:30.000Z A trader reads everything: the central-bank statements, the rate-differential charts, the risk-appetite gauges, the gold and crude tape. The macro view is sharp and correct, the dollar should weaken. Then the trader sits at the screen and discretionarily trades that view, and loses money, because the view was right about the destination and wrong about the path, and the trader's brain closed the winners early and held the losers waiting for vindication. Macro told the truth and discretion squandered it. The resolution is not to drop the macro and not to trust the gut. It is to keep the macro as context and hand the execution to rules. FX sits in a hard spot. The market is non-stationary, the rules of the game keep changing, and a technique that works for a while stops working, so no fixed system is permanent. Macro understanding is what tells you which regime you are in and which way the structural forces lean. But macro understanding executed by hand collides with the same cognitive defects that wreck every discretionary trader. The synthesis is macro-informed, systematically-executed: let the macro set the context and the gates, and let a rule-based system pull the trigger. This article closes the FX run of Pillar 4 by reconciling the macro content of ["Why FX Traders Must Watch Gold, Rates, and Equities"](https://aligrithm.com/why-fx-traders-must-watch-gold-rates-and-equities/) with the systematic discipline of ["Trading Systems Are Recipes, Not Predictions"](https://aligrithm.com/trading-systems-are-recipes-not-predictions/). It is the bridge between the intermarket and microstructure halves of the pillar and the trader's actual workflow. ## Why you need the macro A currency is a relative macro object with no standalone fundamentals, so the forces that move it are macro by nature: rate differentials, risk appetite, commodity terms of trade, capital flows. Ignoring them and trading EURUSD off the chart alone is trading the shadow, the point the FX-drivers article made. The macro view tells you which way the structural wind blows and, more importantly, which regime you are in, because the driver that matters rotates: in a risk-off panic, domestic fundamentals become irrelevant and global risk appetite drives everything; in calm times, local rate expectations take over. Macro also sets the time horizon honestly. In the short run the market is a voting machine, driven by positioning and momentum and emotion; in the long run it is a weighing machine, pulled toward fundamental value. Valuation measures like purchasing power parity only matter over the super-long term and are nearly useless for trading horizons, because a currency can stay cheap or rich far longer than you can hold the position. Macro's job is not to time entries; it is to tell you which regime and horizon you are operating in, so you run the right system. ## Why you should not trade it by hand The macro view is necessary and discretionary execution of it is where the money leaks, for the reasons the article ["Trading Systems Are Recipes, Not Predictions"](https://aligrithm.com/trading-systems-are-recipes-not-predictions/) laid out. A correct macro thesis still has to survive a noisy path, and the human in the loop systematically mishandles the path: taking small profits to feel right, holding losers to avoid being wrong, oversizing the trades they feel strongest about (which are not the ones that pay best), and abandoning the thesis at the worst moment. The macro can be right and the discretionary P&L still negative, because the errors are in the execution, not the analysis. The non-stationarity makes it worse. Because the rules change and techniques decay, a discretionary trader is constantly tempted to override the plan with the latest narrative, and the override is usually the cognitive defect wearing a macro costume. "This time the dollar will ignore rates because of the new narrative" is exactly the kind of mid-trade rationalization that a systematic process exists to prevent. Adapt or die is true at the level of redesigning systems between sessions with data; it is a disaster as a license to improvise inside a live trade. ## The synthesis: macro sets the gate, rules pull the trigger The workable structure separates the two jobs cleanly. Macro analysis defines the regime and the allowed direction; a rule-based system generates the entries, exits, and sizing within that allowance. Macro is the gate; the system is the trigger. $$ \\text{Position}\_t = \\text{Gate}\_{\\text{macro}}(\\text{regime, direction}) \\times \\text{Signal}\_{\\text{rules}}(\\text{price, vol}) $$ The position is the macro gate (a regime-and-direction allowance, often just a plus-one, zero, or minus-one that says which way you are permitted to trade) multiplied by the systematic signal (the rule-based entry and size from price and volatility). When the macro gate says "dollar bearish regime," the system is allowed to take its short-dollar signals and forbidden from going long; when the macro is neutral or conflicted, the gate narrows or closes. The macro never sets the entry price or the stop; the rules never decide the regime. Each does the job it is good at, and neither is allowed to do the job it is bad at. This is the same two-layer structure as the confirmation count in ["Why FX Traders Must Watch Gold, Rates, and Equities"](https://aligrithm.com/why-fx-traders-must-watch-gold-rates-and-equities/) and the regime gate in ["Matching Strategy Families to Market Conditions"](https://aligrithm.com/matching-strategy-families-to-market-conditions/), generalized to the macro-versus-systematic split. The macro layer is slow, judgmental, and updated between sessions; the systematic layer is fast, mechanical, and runs without intervention. The discipline is that once the gate is set, the trader does not touch the trigger. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Making the gate itself systematic The honest extension: the more of the macro gate you can encode in rules, the fewer places discretion can leak in. The drivers from the FX-macro article (the rate differential, the risk-appetite proxy, gold) are measurable, so the gate can be a rule too: dollar-bearish regime when the rate differential is narrowing and gold is rising and equities are firm, for instance. Encoding the gate turns the whole stack systematic and lets you backtest the macro overlay instead of trusting it on faith, the testability the article ["Optimization Comes After Testing, Not Before"](https://aligrithm.com/optimization-comes-after-testing-not-before/) demanded. Some macro judgment resists encoding (a genuine geopolitical regime break, a central-bank stance shift the data has not yet shown), and there the human still sets the gate. That residual discretion is the smallest possible surface for error: a slow, between-sessions regime call, not a fast, in-trade override. The goal is not to eliminate the human, which is impossible in a non-stationary market that requires adaptation, but to confine the human to the slow, high-level decision and keep the fast, error-prone execution mechanical. Macro to choose the game, rules to play it. ## Visualizing the two-layer workflow ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_116-visual-0.png) ## KEY POINTS - A correct macro view executed by hand still loses money, because the human mishandles the noisy path: taking profits early, holding losers, oversizing favorite trades, and abandoning the thesis at the worst time. Macro can be right and discretionary P&L still negative. - A currency is a relative macro object, so you need the macro: rate differentials, risk appetite, commodities, and flows. The driver that matters rotates by regime, and macro's job is to tell you which regime and horizon you are in. - In the short run the market is a voting machine (positioning, momentum, emotion); in the long run a weighing machine (fundamental value). Valuation like purchasing power parity only matters over the super-long term and is useless for trade timing. - You should not trade the macro by hand. Non-stationarity tempts constant overrides, and the override is usually a cognitive defect in a macro costume. "Adapt or die" means redesigning systems between sessions with data, not improvising inside a live trade. - The synthesis: macro sets the gate (regime and allowed direction), and a rule-based system pulls the trigger (entry, exit, size). The position is the macro gate times the systematic signal. - Each layer does what it is good at: macro is slow, judgmental, updated between sessions; the system is fast, mechanical, and runs without intervention. Once the gate is set, the trader does not touch the trigger. - Encode as much of the gate as possible in rules, since the drivers are measurable, which makes the macro overlay backtestable instead of faith-based and shrinks the surface for discretionary error. - Residual judgment that resists encoding (a geopolitical break, an un-priced policy shift) stays human, but confined to the slow, high-level regime call. Macro to choose the game, rules to play it. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Trading Systems and Methods - Perry Kaufman (Amazon)](https://www.amazon.com/s?k=Trading+Systems+and+Methods+Perry+Kaufman&ref=aligrithm.com) - [Cybernetic Trading Strategies - Murray Ruggiero (Amazon)](https://www.amazon.com/s?k=Cybernetic+Trading+Strategies+Murray+Ruggiero&ref=aligrithm.com) - [The Art of Currency Trading - Brent Donnelly (Amazon)](https://www.amazon.com/s?k=The+Art+of+Currency+Trading+Brent+Donnelly&ref=aligrithm.com) ### 4.30 Why Bid/Ask Bounce Matters for Intraday FX Systems URL: https://aligrithm.com/why-bid-ask-bounce-matters-for-intraday-fx-systems/ Last updated: 2026-07-27T13:24:29.000Z Record the last traded price of EURUSD tick by tick on a quiet afternoon and it looks like the market is constantly moving: up a tick, down a tick, up two, down one, a restless little zig-zag. Almost none of that is information. It is the price bouncing between the bid and the offer as buyers and sellers alternate, with the true value sitting still in the middle the whole time. A trade prints at the offer, the next at the bid, and the recorded "price" jumps by the spread without the fair value changing at all. This is bid/ask bounce, and on intraday FX data it manufactures a huge fraction of the apparent movement, which means it manufactures a huge fraction of the apparent signals. For a system trading at short horizons, bounce is poison disguised as opportunity. It inflates measured volatility, it creates phantom mean-reversion (the price "reverts" simply because it bounced back across the spread), and it generates indicator crossings that are pure microstructure artifact. A strategy backtested on bounce-laden last-trade prices can show a beautiful short-term reversion edge that consists entirely of trying to capture the spread, and then it pays the spread to trade and dies. Understanding bounce is what separates a real intraday edge from an accounting illusion. This article isolates bid/ask bounce as a microstructure noise source, following the order mechanics in ["Market Orders vs Limit Orders in FX"](https://aligrithm.com/market-orders-vs-limit-orders-in-fx/) and the liquidity structure in ["The Real Heart of FX Liquidity"](https://aligrithm.com/the-real-heart-of-fx-liquidity/). It is the intraday-noise complement to the longer-horizon noise treated in ["Noise Is Not Volatility"](https://aligrithm.com/noise-is-not-volatility/). ## What bounce is Bid/ask bounce is the oscillation of the recorded transaction price between the bid and the offer when the underlying value is unchanged. With a bid at 1.1000 and an offer at 1.1001, trades print at one or the other depending on whether a buyer or a seller arrives. A run of alternating buys and sellers prints 1.1001, 1.1000, 1.1001, 1.1000, a one-pip "move" every tick, while the mid-price (1.10005) and the fair value never budge. The recorded return series therefore contains a spurious component that has nothing to do with information. Decompose the observed price into the true value plus a bounce term, and the observed return is the true return plus the change in the bounce term. $$ P^{\\text{obs}}\_t = P^{\\text{true}}\_t \\pm \\tfrac{1}{2}s, \\qquad r^{\\text{obs}}\_t = r^{\\text{true}}\_t + \\big(\\text{bounce}\_t - \\text{bounce}\_{t-1}\\big) $$ The observed price equals the true price plus or minus half the spread depending on whether the trade hit the bid or the offer; the observed return equals the true return plus the change in the bounce component between consecutive ticks. Because the bounce term flips sign as trades alternate between bid and offer, its change is large and negatively autocorrelated: an up-bounce is likely followed by a down-bounce. That negative autocorrelation is the fingerprint of bounce, and it is exactly what a naive mean-reversion indicator reads as a tradeable signal. ## Why it fakes a mean-reversion edge The damage is specific and predictable. Bounce makes consecutive observed returns negatively correlated even when the true returns are pure noise, because a print at the offer (an up-tick) tends to be followed by a print at the bid (a down-tick). A short-term mean-reversion system looks for exactly this pattern: price up, so sell; price down, so buy. On bounce-laden data it finds the pattern everywhere and backtests beautifully. The catch is that the "reversion" is the spread, and you cannot capture it. To trade the bounce-driven reversion you would have to buy at the bid and sell at the offer faster than the spread, which means you are trying to earn the spread, not a price move, and as a price-taker paying the spread you lose it instead. The apparent edge is the spread measured as a signal, and the cost of trading is the spread paid as a fee; they cancel, and a strategy that ignored bounce books the phantom profit in the backtest and the real spread loss live. This is the intraday version of the warning in ["Why Transaction Costs Should Be Added Before You Fall in Love"](https://aligrithm.com/why-transaction-costs-should-be-added-before-you-fall-in-love/): the edge and the cost are the same quantity, and only one of them shows up in a careless backtest. The negative autocorrelation that bounce induces is even usable in reverse, as a spread estimator: the size of the negative autocovariance in observed returns is proportional to the spread, so you can back out the effective spread from the price series itself. That is the honest use of bounce, as a measurement of cost, not as a source of signal. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Defending a system against bounce Concrete habits that keep bounce out of your signals. Compute signals on mid-price, not last-trade. The mid (average of bid and offer) removes most of the bounce by construction, because it does not jump when a trade alternates between bid and offer. Any intraday indicator fed last-trade prices is reading bounce; the same indicator on mid-price is reading something closer to the true value. This single change kills most phantom signals. Sample slower than the bounce. Bounce lives at the tick scale; the true signal you are after usually lives at a slower scale. Sampling at a coarser interval (bars rather than ticks) averages out the bounce while preserving the real move, the timeframe-matching logic from ["How to Choose the Right Timeframe for a Strategy"](https://aligrithm.com/how-to-choose-the-right-timeframe-for-a-strategy/). A system that must trade at the tick scale is trading mostly bounce. Require the signal to exceed the spread. A move smaller than the spread is indistinguishable from bounce, so any signal whose magnitude is on the order of the spread should be ignored. Setting a threshold of a multiple of the spread before acting filters the bounce-scale noise and keeps only moves large enough to be information, the price-density and noise logic applied at the micro scale. Backtest with the spread paid on every fill. The decisive test: if the intraday edge survives paying the realized spread on each round trip, it is real; if it evaporates, it was bounce. An edge that only exists on mid-to-mid fills with no spread paid is an edge you cannot trade, the tier-cost honesty from ["Why Retail FX Execution Is Not the Same as Interbank FX"](https://aligrithm.com/why-retail-fx-execution-is-not-the-same-as-interbank-fx/). ## Visualizing the bounce ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_115-visual-0.png) ## KEY POINTS - Bid/ask bounce is the recorded transaction price oscillating between the bid and the offer while the true value sits unchanged in the middle. A run of alternating buyers and sellers prints a spread-sized "move" every tick with no information behind it. - On intraday FX data bounce manufactures a large fraction of apparent movement, inflating measured volatility and generating indicator crossings that are pure microstructure artifact. - The observed return equals the true return plus the change in a bounce term that flips sign as trades alternate between bid and offer. That flip makes consecutive observed returns negatively autocorrelated, the fingerprint of bounce. - Bounce fakes a mean-reversion edge: a short-term system reads the negative autocorrelation as "price up, sell; price down, buy" and backtests beautifully. - The fake edge is the spread itself. Capturing it means earning the spread, but a price-taker pays the spread, so the apparent profit and the real cost cancel. A careless backtest books the phantom profit and the live account pays the spread loss. - The honest use of bounce is as a spread estimator: the negative autocovariance of observed returns is proportional to the effective spread, so you can back out the cost from the price series. - Defenses: compute signals on mid-price not last-trade, sample slower than the tick-scale bounce, require the signal to exceed a multiple of the spread before acting, and backtest with the realized spread paid on every fill. - The decisive test: an intraday edge that survives paying the spread on each round trip is real; one that evaporates was bounce. An edge that only exists on mid-to-mid fills with no spread paid is untradeable. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Trading Systems and Methods - Perry Kaufman (Amazon)](https://www.amazon.com/s?k=Trading+Systems+and+Methods+Perry+Kaufman&ref=aligrithm.com) - [Cybernetic Trading Strategies - Murray Ruggiero (Amazon)](https://www.amazon.com/s?k=Cybernetic+Trading+Strategies+Murray+Ruggiero&ref=aligrithm.com) - [The Art of Currency Trading - Brent Donnelly (Amazon)](https://www.amazon.com/s?k=The+Art+of+Currency+Trading+Brent+Donnelly&ref=aligrithm.com) ### 4.29 Market Orders vs Limit Orders in FX URL: https://aligrithm.com/market-orders-vs-limit-orders-in-fx/ Last updated: 2026-07-27T13:24:28.000Z A breakout system fires a buy. Two ways to act on it. Send a market order and you cross the spread immediately, paying the offer, certain of a fill but starting the trade already down the spread. Send a limit order at the bid and you wait for the market to come to you, paying nothing to cross, but you only get filled if price dips to your level, and if the breakout is real it never does, so you miss the trade entirely. The choice between the two is not a detail. For a system that trades often, the order type can be the difference between a live edge and a dead one, because it determines whether you pay the spread or earn it, and whether you get filled when you most need to. A market order takes liquidity and pays for immediacy; a limit order provides liquidity and gets paid for patience, at the cost of fill uncertainty. Every FX system makes this trade-off on every order, and the right answer depends on how urgent the signal is. Momentum needs to be in now and pays the spread; mean-reversion can afford to wait and can earn it. Picking the wrong order type for the signal is a quiet, recurring leak. This article works the two order types and the costs each carries, building on the PnL mechanics in ["How FX PnL Actually Works"](https://aligrithm.com/how-fx-pnl-actually-works/) and the execution tiers in ["Why Retail FX Execution Is Not the Same as Interbank FX"](https://aligrithm.com/why-retail-fx-execution-is-not-the-same-as-interbank-fx/). It leads into the microstructure noise of ["Why Bid/Ask Bounce Matters for Intraday FX Systems"](https://aligrithm.com/why-bid-ask-bounce-matters-for-intraday-fx-systems/). ## The market order: pay to be certain A market order executes immediately at the best available price, which means crossing the spread: a buy pays the offer, a sell hits the bid. You are guaranteed a fill (in normal liquidity), and you pay for that certainty with the spread plus, for larger orders, market impact as you consume depth and leave a footprint. $$ \\text{Cost}\_{\\text{market}} \\;\\approx\\; \\tfrac{1}{2}\\,s \\;+\\; \\text{impact}(\\text{size}), \\qquad \\text{fill probability} = 1 $$ The cost of a market order is about half the spread (the distance from mid to the offer you pay) plus a market-impact term that grows with order size, and the fill probability is one. You always trade, and you always pay. For a small order the cost is essentially half the spread; for a large order the impact term dominates and the real skill is working the order to minimize the footprint. The market order's defining feature is certainty: you will be in the trade, which is exactly what an urgent signal needs. ## The limit order: get paid to wait A limit order rests at a chosen price and executes only if the market trades to it. A buy limit at or below the bid does not cross the spread; if filled, you bought at your price rather than paying the offer, effectively earning the spread instead of paying it. The cost is not in the spread but in two other places: the order may never fill (opportunity cost), and the fills you do get are adversely selected. $$ \\text{Cost}\_{\\text{limit}} \\;\\approx\\; -\\tfrac{1}{2}\\,s \\;+\\; \\text{adverse selection}, \\qquad \\text{fill probability} < 1 $$ The nominal cost of a limit order is negative half the spread (you earn rather than pay it), offset by an adverse-selection term, and the fill probability is below one. Adverse selection is the catch that kills naive limit strategies: your buy limit fills when sellers are hitting it, which is disproportionately when the price is about to keep falling, so the fills you get are the ones you would rather not have, and the fills you wanted (when price rallies away) never happen. You earn the spread on the trades that go against you and miss the trades that go for you, which is the opposite of what the headline "earn the spread" suggests. ## Matching order type to signal The decision rule is to match the order type to the urgency and direction-dependence of the signal. Momentum and breakout signals need market orders. The whole premise is that price is about to move in your direction and keep going, so a limit order resting behind the move never fills, and you miss precisely the trades the system exists to catch. You pay the spread because the alternative is no position, and the expected move is large relative to the spread, so paying it is worth it. A breakout system that tries to save the spread with limit orders systematically captures only the failed breakouts (the ones that came back to fill the limit) and misses the winners, an adverse selection that inverts the edge. Mean-reversion and range signals suit limit orders. The premise is that price has overshot and will come back, so resting a limit at the extreme both earns the spread and expresses the thesis: you want to be filled exactly when price spikes to your level, because that is the entry you were waiting for. Here the adverse selection partly works for you, because the fill happens at the overshoot you intended to fade. A mean-reversion system using market orders pays the spread on every trade and gives back a chunk of a naturally small edge. This is the execution layer of the strategy-family map from ["Matching Strategy Families to Market Conditions"](https://aligrithm.com/matching-strategy-families-to-market-conditions/): the same noise axis that picks trend versus reversion also picks market versus limit, because trend needs immediacy and reversion needs patience. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## The traps Where the order-type choice goes wrong. Limit orders in fast markets give a false sense of cost saving. In a breakout, the limit that does not fill is not a saved spread; it is a missed trade, and the opportunity cost of missing the winners dwarfs the spread you saved on the losers. Counting only the spread saved on filled orders and ignoring the trades you missed is a measurement error that makes limit strategies look better than they are. Market orders in thin liquidity pay far more than the quoted spread. The cost formula assumes normal depth; in the thin hours or a volatility spike, a market order walks the book and pays slippage well beyond half the spread, the liquidity-volatility inverse from ["The Real Heart of FX Liquidity"](https://aligrithm.com/the-real-heart-of-fx-liquidity/). An urgent market order during a news shock can cost many times its calm-market spread. Stop orders are market orders in disguise. A stop-loss triggers a market order when hit, so it pays the spread and the slippage at the worst possible moment, when price is moving against you fast. Sizing and placing stops without accounting for that slippage understates the realized loss, the gap between a backtest's clean stop fill and a live stop's slipped fill. ## Visualizing the order-type trade-off ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_114-visual-0.png) ## KEY POINTS - A market order takes liquidity and pays for immediacy; a limit order provides liquidity and gets paid for patience, at the cost of fill uncertainty. Every FX order makes this trade-off. - A market order crosses the spread (buy pays the offer, sell hits the bid), costs about half the spread plus size-dependent market impact, and fills with certainty. Its defining feature is that you will be in the trade. - A limit order rests and fills only if price reaches it. It nominally earns half the spread instead of paying it, offset by adverse selection, and fills with probability below one. - Adverse selection is the limit-order catch: your buy fills when sellers are hitting it, disproportionately when price keeps falling. You earn the spread on the trades that go against you and miss the ones that go for you. - Match order type to signal. Momentum and breakout need market orders, because a resting limit never fills on a real breakout and captures only the failed ones, inverting the edge. - Mean-reversion and range signals suit limit orders, because you want to fill exactly at the overshoot you intend to fade, and the adverse selection partly works for you. - This is the execution layer of the strategy-family map: the noise axis that picks trend versus reversion also picks market versus limit, since trend needs immediacy and reversion needs patience. - Traps: limit orders in fast markets hide missed-trade opportunity cost as fake spread savings; market orders in thin liquidity pay far beyond the quoted spread; and stop-losses are disguised market orders that slip at the worst moment. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Trading Systems and Methods - Perry Kaufman (Amazon)](https://www.amazon.com/s?k=Trading+Systems+and+Methods+Perry+Kaufman&ref=aligrithm.com) - [Cybernetic Trading Strategies - Murray Ruggiero (Amazon)](https://www.amazon.com/s?k=Cybernetic+Trading+Strategies+Murray+Ruggiero&ref=aligrithm.com) - [The Art of Currency Trading - Brent Donnelly (Amazon)](https://www.amazon.com/s?k=The+Art+of+Currency+Trading+Brent+Donnelly&ref=aligrithm.com) ### 4.28 How FX PnL Actually Works URL: https://aligrithm.com/how-fx-pnl-actually-works/ Last updated: 2026-07-27T13:24:27.000Z A trader buys one million dollars of EURUSD and the pair moves 20 pips in their favor. The profit is 200 dollars. Wait, no: it is 2,000 dollars. The confusion is the whole reason this article exists. FX profit and loss is not "pips times a fixed dollar value" the way a futures contract is, because a currency pair is one currency priced in another, and the profit lands in the quote currency, which may not be the one your account is denominated in. Get the pip value wrong and you size every position wrong, which means you risk-manage every position wrong. The mechanics are not hard, but they are specific, and they differ between a pair like EURUSD where the dollar is the quote currency and a pair like USDJPY where the dollar is the base. A system that sizes positions by a single fixed "dollar per pip" across all pairs is mis-stating its risk on every pair where that assumption is false, which is most of them. This article works the FX PnL calculation end to end, building on the tier and execution structure from ["Why Retail FX Execution Is Not the Same as Interbank FX"](https://aligrithm.com/why-retail-fx-execution-is-not-the-same-as-interbank-fx/). It is the arithmetic foundation for position sizing, and it connects to the order-cost mechanics in ["Market Orders vs Limit Orders in FX"](https://aligrithm.com/market-orders-vs-limit-orders-in-fx/). ## The base case: quote currency is your account currency Start with the clean case. PnL is reported in the quote currency (the denominator of the pair), and the formula is position size times price change. $$ \\text{PnL} = \\text{Position Size} \\times \\Delta P $$ PnL equals the position size times the change in price. For a one-million-unit EURUSD position that moves 20 pips, the price change is 0.0020, so the PnL is 1,000,000 times 0.0020, equal to 2,000 dollars. The profit is in dollars because USD is the quote currency, and for a USD-denominated account that is the final answer. This is the case that matches intuition, and it is the only case where "pips times a simple value" works without a second step. The pip value here is fixed and clean: on a one-million EURUSD position, each pip (0.0001) is worth 100 dollars, so 20 pips is 2,000 dollars. The reason a 20-pip move is 2,000 and not 200 is that the position is a million units; the pip value scales with position size, and confusing notional with the round-number "lot" is how the factor-of-ten errors creep in. ## The two-step case: base currency is not your account currency Now the case that trips everyone. When the dollar is the base currency, like USDJPY, the PnL first lands in the quote currency (yen), and you have to convert it back to dollars. Take a ten-million USDJPY position bought at 101.60 and sold at 101.65, a five-pip move (for JPY pairs a pip is 0.01). The profit in yen is the position size times the price change: 10,000,000 times 0.05, equal to 500,000 yen. That is step one, and it is denominated in yen, not dollars. Step two converts it: divide the 500,000 yen by the current USDJPY rate of about 101.65, giving roughly 4,956 dollars. The profit a USD account actually books is 4,956 dollars, not 500,000 of anything and not a number you can read off the pip count alone. The general rule: PnL is computed in the quote currency first, then converted to the account currency at the prevailing rate if they differ. For pairs where the quote currency is your account currency, the conversion is trivial (multiply by one). For pairs where it is not, the conversion step is mandatory, and the pip value in your account currency changes as the exchange rate moves, so it is not even constant over the life of the trade. ## Why this matters for position sizing The payoff of getting the arithmetic right is correct risk per trade across pairs, which is the foundation of every sizing rule. A system that risks a fixed dollar amount per trade has to convert that dollar risk into a position size pair by pair, and the conversion depends on the pip value in account-currency terms. The same nominal position carries different dollar risk on EURUSD, USDJPY, and GBPUSD, because the pip values differ. Sizing all three with one fixed "dollars per pip" over-risks some and under-risks others, which silently distorts the risk weighting the article ["Why ATR Normalization Is More Than a Volatility Trick"](https://aligrithm.com/why-atr-normalization-is-more-than-a-volatility-trick/) was trying to achieve by scaling to volatility. Volatility normalization assumes you measured the dollar risk correctly in the first place, and the pip-value conversion is how you do that. The clean procedure is to express each pair's stop distance in pips, convert pips to account-currency risk using the correct pip value, then set the position size so that account-currency risk equals your fixed risk budget. Do this per pair, with the conversion step for the pairs that need it, and every position in the book carries the same intended risk. Skip it, and your "equal risk" book is quietly overweight the pairs where you guessed the pip value too low. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## The frictions that reduce the booked number The PnL formula gives the gross result; the booked result is smaller for the reasons the execution articles laid out. The spread is paid on the round trip, so the realized entry and exit are worse than the prices a clean calculation uses, and on a retail account the markup widens that further, the tier costs from ["Why Retail FX Execution Is Not the Same as Interbank FX"](https://aligrithm.com/why-retail-fx-execution-is-not-the-same-as-interbank-fx/). A 20-pip gross winner on a pair with a two-pip retail spread is an 18-pip net winner before any other friction. For carry positions held overnight, the swap or rollover (the interest-rate differential credited or debited for holding the position) adds or subtracts from PnL daily, a real cash flow a price-only PnL ignores. The discipline is to compute PnL net: gross price change converted correctly to the account currency, minus spread and markup on the round trip, plus or minus carry for the holding period. That net number is what hits the account, and it is the only number a backtest should report, the cost-honesty the article ["Why Transaction Costs Should Be Added Before You Fall in Love"](https://aligrithm.com/why-transaction-costs-should-be-added-before-you-fall-in-love/) demanded. ## Visualizing the two PnL paths ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_113-visual-0.png) ## KEY POINTS - FX PnL is not "pips times a fixed dollar value." A currency pair is one currency priced in another, and the profit lands in the quote currency, which may not be your account currency. - Base case (quote currency is your account currency, like EURUSD for a USD account): PnL equals position size times price change. A one-million EURUSD position moving 20 pips earns 1,000,000 times 0.0020, equal to 2,000 dollars, in one step. - The pip value scales with position size. A 20-pip move on a million units is 2,000 dollars because each pip is 100 dollars; confusing notional with a round-number lot causes factor-of-ten errors. - Two-step case (base currency is your account currency, like USDJPY): PnL lands in yen first (position times price change), then converts to dollars at the current rate. A ten-million position moving five pips is 500,000 yen, divided by about 101.65, equal to roughly 4,956 dollars. - The pip value in account-currency terms is not constant for pairs that need conversion; it moves with the exchange rate over the life of the trade. - Correct pip values are the foundation of position sizing. The same nominal position carries different dollar risk across pairs, so a single fixed "dollars per pip" over-risks some pairs and under-risks others. - The clean procedure: express the stop in pips, convert to account-currency risk with the correct pip value, then size so account-currency risk equals your fixed budget, per pair. - Report PnL net: gross change converted correctly, minus spread and markup on the round trip, plus or minus carry for the holding period. That net number is the only one a backtest should use. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Trading Systems and Methods - Perry Kaufman (Amazon)](https://www.amazon.com/s?k=Trading+Systems+and+Methods+Perry+Kaufman&ref=aligrithm.com) - [Cybernetic Trading Strategies - Murray Ruggiero (Amazon)](https://www.amazon.com/s?k=Cybernetic+Trading+Strategies+Murray+Ruggiero&ref=aligrithm.com) - [The Art of Currency Trading - Brent Donnelly (Amazon)](https://www.amazon.com/s?k=The+Art+of+Currency+Trading+Brent+Donnelly&ref=aligrithm.com) ### 4.27 Why Retail FX Execution Is Not the Same as Interbank FX URL: https://aligrithm.com/why-retail-fx-execution-is-not-the-same-as-interbank-fx/ Last updated: 2026-07-27T13:24:27.000Z A backtest fills every order at the mid-price, instantly, in any size, at any time. Interbank execution comes close to that ideal for a professional: tight firm spreads, deep liquidity, fills that stick. Retail execution does not come close. A retail trader faces a marked-up spread, a broker that may hold the order for a last look and reject it if the market moved, possible requotes, and a counterparty that is often the broker itself rather than the market. The same signal, the same pair, the same instant, produces a different realized price depending on which door you walk through, and the retail door costs more. Retail FX is enormous (in Japan alone there are on the order of a million active retail accounts, against far fewer in the US), and the entire retail apparatus is built on an execution model that differs from interbank in ways that directly determine whether a system is profitable. A strategy can clear interbank costs and fail retail costs, or be designed so its trade frequency and edge survive the retail markup. Knowing the difference is a prerequisite to backtesting honestly. This article details the execution gap between the two tiers, continuing from ["FX Is Not One Market: Retail vs Wholesale Structure"](https://aligrithm.com/fx-is-not-one-market-retail-vs-wholesale-structure/) and ["The Real Heart of FX Liquidity"](https://aligrithm.com/the-real-heart-of-fx-liquidity/). It sets up the order-mechanics detail in ["Market Orders vs Limit Orders in FX"](https://aligrithm.com/market-orders-vs-limit-orders-in-fx/) and ["How FX PnL Actually Works"](https://aligrithm.com/how-fx-pnl-actually-works/). ## What interbank execution looks like In the interbank market a dealer trades against firm, executable quotes on a primary venue. The spread is a fraction of a pip for the major pairs in normal conditions, the quotes are committed, and the main cost of trading in size is market impact: a large order moves the price as it consumes depth, leaving a footprint. The dealer's execution problem is to work a large order without revealing it and pushing the market away, a size problem, not a fairness problem. The price is real and the fill is honest; what you pay for is depth. The interbank trader's costs are therefore the spread (small) plus market impact (rising with size). For a system trading modest size, interbank costs are close to the mid-price assumption a backtest makes, which is why institutional FX backtests can use tight cost models and be roughly right. ## What retail execution adds The retail trader inherits the interbank price and then pays for a stack of frictions the interbank trader does not. The markup and wider spread. The broker quotes the interbank price plus a markup and a wider spread, so every round trip starts in a deeper hole than interbank. This is the tier cost from the structure article, paid on entry and exit. Last look. Many retail and aggregated venues hold a submitted order for a brief moment and reserve the right to reject it if the price moved against the liquidity provider in that instant. The asymmetry is the problem: fills that would have moved in your favor get rejected, fills that move against you stick. Last look quietly worsens your realized fill distribution in a way no mid-price backtest captures. B-book conflict. When the broker takes the other side of your trade (B-book) rather than passing it to the market (A-book), the broker profits when you lose. That conflict shows up as requotes on winning setups, slippage skewed against you, and spreads that widen at the worst moments. Not every broker B-books every client, but the system builder has to assume the execution terms are set by a counterparty whose interests may oppose theirs. $$ \\text{Edge}\_{\\text{net}} = \\text{Edge}\_{\\text{gross}} - s\_{\\text{tier}} - \\text{impact}\_{\\text{tier}} - c\_{\\text{tier}}, \\qquad c\_{\\text{retail}} \\gg c\_{\\text{interbank}} $$ Net edge is the gross edge minus the tier's spread, minus the tier's market impact, minus a tier-specific extra cost that bundles markup, last-look rejection, and adverse fill skew. That extra cost is small for interbank and large for retail. Two traders with the same gross signal edge end up with different net edges because the subtracted costs differ by tier, and a gross edge that survives interbank's small deductions can be wiped out by retail's large ones. ## What this does to a strategy The execution gap reshapes which strategies are viable at each tier, the same cost-versus-frequency logic the article ["Why Transaction Costs Should Be Added Before You Fall in Love"](https://aligrithm.com/why-transaction-costs-should-be-added-before-you-fall-in-love/) applied to all systems, sharpened for FX tiers. High-frequency and scalping strategies depend on capturing a tiny edge many times, and the retail markup plus last look devours that edge. These strategies are interbank-only; on retail they are a fee-generation machine for the broker. A scalping backtest that ignores retail execution is the most dangerous backtest in FX, because it looks the most profitable and fails the hardest. Lower-frequency swing and position strategies trade rarely enough that the retail cost per trade is small relative to the per-trade edge. A system that holds for days and targets a move many times the spread can absorb the retail markup and still be net positive. Matching trade frequency to tier cost is the design lever: the higher the per-trade cost, the larger and rarer the moves the strategy must target. The honest test is to backtest with the actual tier's costs, including a realistic last-look rejection rate and a markup, not the mid. If the edge only exists on mid-price fills, it is not an edge you can trade retail. Model the frictions, then decide. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## What you can and cannot control A closing dose of realism on what the retail system builder actually governs. You control pair choice, time of day, order type, and trade frequency, and these are the levers that keep you on the cheaper side of the retail cost structure: liquid majors in the London-New York overlap, order types that do not pay unnecessary spread, and a frequency that respects the markup. You do not control last look, the markup, or whether the broker B-books you. The system has to be robust to costs you cannot remove, which means leaving a margin between the gross edge and the worst plausible retail cost, not designing to the mid and hoping. The broker choice itself is part of the system. An honest A-book broker with tight markups and no last look changes the math; a wide B-book broker can make the same strategy unviable. Treating the broker as a fixed input and stress-testing the strategy against pessimistic execution is the discipline that keeps a retail FX system from dying on contact with live fills, the recipe-versus-reality gap the article ["Trading Systems Are Recipes, Not Predictions"](https://aligrithm.com/trading-systems-are-recipes-not-predictions/) flagged for every live system. ## Visualizing the execution gap ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_112-visual-0.png) ## KEY POINTS - A backtest fills at the mid instantly in any size. Interbank execution approximates that for a professional; retail execution does not, and the gap determines whether a system is profitable. - Interbank execution is firm, tight-spread, and honest. Its main cost is market impact: large orders move the price and leave a footprint. For modest size, interbank costs are close to the mid assumption. - Retail execution adds a stack of frictions: a markup and wider spread, last look (the right to reject a fill if the price moved, asymmetrically against you), and a possible B-book conflict where the broker profits when you lose. - Net edge is gross edge minus the tier's spread, impact, and a tier-specific extra cost bundling markup, last-look rejection, and adverse fill skew. That extra cost is small for interbank and large for retail. - High-frequency and scalping strategies depend on a tiny repeated edge that the retail markup and last look devour. They are interbank-only; a scalping backtest ignoring retail execution is the most dangerous in FX. - Lower-frequency swing strategies trade rarely enough that retail cost per trade is small relative to the per-trade edge. Match trade frequency to tier cost: higher cost demands larger, rarer target moves. - Backtest with the actual tier's costs, including a realistic last-look rejection rate and markup. An edge that only exists on mid-price fills is not tradeable retail. - You control pair, time of day, order type, and frequency; you do not control last look, markup, or B-booking. Leave a margin between gross edge and worst plausible retail cost, and treat the broker choice as part of the system. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Trading Systems and Methods - Perry Kaufman (Amazon)](https://www.amazon.com/s?k=Trading+Systems+and+Methods+Perry+Kaufman&ref=aligrithm.com) - [Cybernetic Trading Strategies - Murray Ruggiero (Amazon)](https://www.amazon.com/s?k=Cybernetic+Trading+Strategies+Murray+Ruggiero&ref=aligrithm.com) - [The Art of Currency Trading - Brent Donnelly (Amazon)](https://www.amazon.com/s?k=The+Art+of+Currency+Trading+Brent+Donnelly&ref=aligrithm.com) ### 4.26 The Real Heart of FX Liquidity URL: https://aligrithm.com/the-real-heart-of-fx-liquidity/ Last updated: 2026-07-27T13:24:26.000Z The price of EURUSD is not set in a retail app, on a charting platform, or in a data vendor's feed. It is set on a small number of primary interbank electronic venues where banks trade with each other, and historically two of them carried most of the flow: one that became the home of EUR and JPY liquidity, and another that anchored GBP, commodity, and emerging-market pairs. Every other price in the FX world (the retail quote, the broker feed, the platform chart) is a derivative of what happens on those primary venues. When a system trader talks about "the market," in FX that means these venues, and everything else is a copy with a delay and a markup. Liquidity is the depth and tightness available to transact, and in FX it is concentrated at the wholesale tier and thins out as you move away from it. A retail trader's liquidity is whatever the broker chooses to show, sourced from the primary venues and reshaped. Understanding where the real liquidity lives, and how it changes by pair, by time of day, and by regime, is what separates a system that executes at the prices it assumed from one that bleeds slippage it never modeled. This article locates the source of FX liquidity and the rules it follows. It continues from ["FX Is Not One Market: Retail vs Wholesale Structure"](https://aligrithm.com/fx-is-not-one-market-retail-vs-wholesale-structure/), which established the two tiers, and it underpins the execution choices in ["Market Orders vs Limit Orders in FX"](https://aligrithm.com/market-orders-vs-limit-orders-in-fx/) and ["Why Bid/Ask Bounce Matters for Intraday FX Systems"](https://aligrithm.com/why-bid-ask-bounce-matters-for-intraday-fx-systems/). ## Where the price is made The heart of the FX market is a handful of primary electronic communication networks, the venues where interbank dealers post and take two-sided prices. These are the primary markets, the place price discovery actually happens. A second ring of smaller venues has grown over time, but the primary venues remain the reference, and the price on every downstream screen is sourced from them. This matters for a system builder because it locates the real bid and offer. The tightest, deepest, most reliable price for a major pair is on the primary venue, and that price is the benchmark against which your execution should be measured. If your retail fills are consistently worse than the primary-venue mid by more than the expected spread and markup, you are paying for something (latency, last look, a wide B-book quote) that your backtest did not include, the tier gap the previous article made central. ## Liquidity is not constant The single most important property of FX liquidity is that it moves, and it moves inversely to volatility. When markets are calm, spreads are tight and depth is good; when volatility spikes, liquidity evaporates exactly when you most want to trade. The two are inversely related, which is the cruel arithmetic of execution: the moments your system most wants to act (a violent breakout, a news shock) are the moments liquidity is thinnest and slippage is worst. $$ \\text{liquidity} \\;\\propto\\; \\frac{1}{\\text{volatility}}, \\qquad \\text{expected slippage} \\;\\approx\\; \\frac{\\text{order size}}{\\text{available depth}} $$ Liquidity runs roughly inverse to volatility, so depth shrinks as volatility rises. Expected slippage is approximately your order size divided by the available depth, so the same order costs more to execute when depth has thinned. Put the two together and a fixed-size order suffers its worst slippage in the high-volatility, low-liquidity moments, which is precisely when momentum and breakout systems are firing. A backtest that assumes constant spread and instant fills is most wrong exactly where it matters most. ## Liquidity varies by pair and by clock Two more axes of variation a system has to respect. By pair. Liquidity concentrates in the major, high-volume pairs, and there is a strong relationship between volume and liquidity: high-volume currencies have high liquidity and tight spreads, while exotic and emerging-market pairs have thin liquidity and wide, jumpy spreads. The practical rule is to stick to the liquid majors for any system sensitive to execution cost, because the wide spreads and slippage on illiquid pairs will eat an edge that looks fine on mid-price data. The best pairs to trade combine high liquidity with enough volatility to move, and there is a trade-off between the two. By clock. FX liquidity follows the trading day. The overlap of the London and New York sessions is the deepest, most liquid window, when both major hubs are active and the major economic events, fixes, and option expiries cluster. The London open is a reflection point where big risk-takers process overnight information and aggressive trends often start; the New York open frequently extends a London move before it reverses. A system that trades the thin hours (the gap between the New York close and the Asian open) faces wider spreads and worse fills than the same system run in the London-New York overlap, the timeframe-and-timing point the article ["How to Choose the Right Timeframe for a Strategy"](https://aligrithm.com/how-to-choose-the-right-timeframe-for-a-strategy/) made about matching a strategy to its conditions. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## What this means for a system The execution model has to be liquidity-aware, not a flat assumption. Three concrete habits follow. Size to the available depth, not to a fixed lot. An order that is small relative to depth in the London-New York overlap can be large relative to depth in the thin Asian hours, and the slippage scales with that ratio. A liquidity-aware system shrinks size or stands aside when depth is thin. Trade the liquid pairs and the liquid hours unless the strategy explicitly earns its keep elsewhere. The default is the majors during the overlap, because that is where the assumptions in a mid-price backtest are closest to true. Model slippage as a function of volatility, not a constant. Since liquidity falls as volatility rises, the slippage your system pays should rise with volatility in the backtest, or you will systematically overstate the profitability of every signal that fires in turbulent conditions, the conditions a trend system depends on. ## Visualizing liquidity through the day ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_111-visual-0.png) ## KEY POINTS - The price of a currency is set on a small number of primary interbank electronic venues where banks trade with each other. Every retail, broker, and platform price is a derivative of those primary venues. - Locating the real bid and offer matters: the primary-venue price is the benchmark for your execution. Fills consistently worse than that mid beyond the expected spread and markup mean you are paying for latency, last look, or a wide B-book quote. - FX liquidity is not constant and runs inverse to volatility. Spreads tighten and depth improves in calm markets and evaporate when volatility spikes, exactly when momentum and breakout systems fire. - Expected slippage is roughly order size divided by available depth, so a fixed-size order suffers its worst slippage in the low-liquidity, high-volatility moments a mid-price backtest assumes away. - Liquidity concentrates in high-volume major pairs; exotic and emerging pairs have thin, jumpy liquidity. Stick to the liquid majors for any cost-sensitive system. The best pairs combine high liquidity with enough volatility to move. - Liquidity follows the clock. The London-New York overlap is the deepest window, where major events and fixes cluster. Thin hours between the New York close and the Asian open carry wider spreads and worse fills. - Build a liquidity-aware execution model: size to available depth rather than a fixed lot, default to liquid pairs and liquid hours, and model slippage as a function of volatility rather than a constant. - A backtest with constant spreads and instant fills is most wrong precisely where it matters most, in the turbulent conditions trend systems depend on. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Trading Systems and Methods - Perry Kaufman (Amazon)](https://www.amazon.com/s?k=Trading+Systems+and+Methods+Perry+Kaufman&ref=aligrithm.com) - [Cybernetic Trading Strategies - Murray Ruggiero (Amazon)](https://www.amazon.com/s?k=Cybernetic+Trading+Strategies+Murray+Ruggiero&ref=aligrithm.com) - [The Art of Currency Trading - Brent Donnelly (Amazon)](https://www.amazon.com/s?k=The+Art+of+Currency+Trading+Brent+Donnelly&ref=aligrithm.com) ### 4.25 FX Is Not One Market: Retail vs Wholesale Structure URL: https://aligrithm.com/fx-is-not-one-market-retail-vs-wholesale-structure/ Last updated: 2026-07-27T13:24:26.000Z Two traders buy EURUSD at the same instant. One is a bank dealer hitting a price on a primary interbank venue; the other is a retail trader clicking buy in a broker app. They are not trading in the same market, and they are not getting the same price. The dealer trades on a centralized electronic venue against other professional liquidity, at a spread of a fraction of a pip in size. The retail trader trades against a broker that quotes a derived price with a markup, may decline the fill on a last look, and shows a spread several times wider. The EURUSD chart they both stare at hides the fact that "the price of EURUSD" is not a single number. It is a stack of different prices at different tiers. FX is highly decentralized, with no single exchange, and it splits into two interacting segments: a wholesale interbank market where price discovery happens, and a retail market that consumes a derived version of that price. The two are connected, so they track each other, but they are structurally different markets with different prices, spreads, and execution rules. Treating them as one is the mistake behind most retail FX backtests that look profitable and lose money live. This article maps the two-tier structure and draws the consequences for system builders. It opens the FX microstructure run of Pillar 4 that continues through ["The Real Heart of FX Liquidity"](https://aligrithm.com/the-real-heart-of-fx-liquidity/) and ["Why Retail FX Execution Is Not the Same as Interbank FX"](https://aligrithm.com/why-retail-fx-execution-is-not-the-same-as-interbank-fx/), and it is the structural foundation the execution articles build on. ## The two tiers The wholesale tier is the interbank market, where banks and large institutions trade with each other and where the real price of a currency is discovered. Bank dealers act as market makers, quoting two-sided prices, and they also take directional risk in anticipation of client flow. This is where size trades, where spreads are tightest, and where the reference price that everything else hangs off is set. The participants are professionals trading in millions and billions; in dealer slang a "buck" is one million and a "yard" is one billion. The retail tier sits downstream. A retail broker takes the interbank price as a reference, adds a markup, and shows the client a quote. The broker either passes the client's trade through to the wholesale market (agency, or A-book) or takes the other side itself (principal, or B-book). The retail price is a function of the wholesale price, but it is not the wholesale price: it carries a markup, a wider spread, and execution terms the client does not control. $$ P\_{\\text{retail}} = P\_{\\text{interbank mid}} \\;\\pm\\; \\tfrac{1}{2}\\,s\_{\\text{retail}} \\;+\\; m, \\qquad s\_{\\text{retail}} > s\_{\\text{interbank}} $$ The retail price equals the interbank mid-price, plus or minus half the retail spread depending on whether you buy or sell, plus a markup m. The retail spread is wider than the interbank spread. The mid is shared (the two tiers track each other), but the spread and the markup are the retail trader's added cost, and they are the gap between the price on the chart and the price you actually transact at. ## Why this breaks naive backtests The structural split has one dominant consequence for systems: the data you backtest on and the prices you execute at come from different tiers, and the gap between them is where the edge dies. A backtest run on a clean mid-price series (or an interbank feed) measures an edge available in the wholesale market. Trade the same system through a retail broker and you pay the wider retail spread and the markup on every round trip, costs the backtest never saw. A strategy with a small per-trade edge, common for any high-frequency or mean-reversion FX system, can be net positive on interbank costs and net negative on retail costs, the threshold problem the article ["Why Transaction Costs Should Be Added Before You Fall in Love"](https://aligrithm.com/why-transaction-costs-should-be-added-before-you-fall-in-love/) made central. The tier mismatch is the single most common reason a profitable-looking FX backtest loses money in a retail account. The fix is to backtest at the tier you will trade at. If you execute retail, model the retail spread and markup, not the mid. If you execute interbank, you still pay the spread and face market impact for size. Either way, the costs are tier-specific, and a backtest that uses one tier's prices to estimate another tier's profits is measuring a market you cannot access. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## The same currency, different liquidity The two tiers also differ in how much you can trade and how the price reacts. In the wholesale market, large orders move the price (market impact), and the cost of size is the footprint your order leaves, handled in ["Market Orders vs Limit Orders in FX"](https://aligrithm.com/market-orders-vs-limit-orders-in-fx/). In the retail market, the broker absorbs small clicks easily but controls the terms: it can widen spreads in volatility, requote, or decline a fill on last look, so the retail trader's effective liquidity is whatever the broker chooses to offer at that moment. The practical upshot for a system builder is that the same EURUSD signal has different economics at different tiers, the FX-specific version of the point ["Why FX Traders Must Watch Gold, Rates, and Equities"](https://aligrithm.com/why-fx-traders-must-watch-gold-rates-and-equities/) made about a currency being a relative object. A scalping system that needs sub-pip spreads can only live in the wholesale tier; a slower swing system that trades a few times a week can survive retail spreads. Matching the strategy's cost sensitivity to the tier's cost structure is a design decision that comes before any signal work. ## Visualizing the two tiers ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_110-visual-0.png) ## KEY POINTS - FX has no single exchange. It splits into two interacting segments: a wholesale interbank market where price is discovered, and a retail market that consumes a derived version of that price. - In the wholesale tier, banks act as market makers and take directional risk anticipating client flow. This is where size trades, spreads are tightest, and the reference price is set. A "buck" is a million, a "yard" a billion. - The retail tier sits downstream: a broker takes the interbank price, adds a markup and a wider spread, and either passes the trade to the wholesale market (A-book) or takes the other side (B-book). - The retail price equals the shared interbank mid plus or minus half the wider retail spread plus a markup. The mid is common; the spread and markup are the retail trader's added, tier-specific cost. - The dominant consequence for systems: backtest data and execution prices come from different tiers, and the gap is where the edge dies. A small per-trade edge can be positive on interbank costs and negative on retail costs. - Backtest at the tier you will trade. Retail execution means modeling retail spread and markup, not the mid; interbank execution still pays the spread and faces market impact for size. - Liquidity differs too: the wholesale market charges size through market impact, while the retail broker controls the terms and can widen, requote, or decline on last look. - Match the strategy's cost sensitivity to the tier: sub-pip scalping can only live wholesale, while a slower swing system can survive retail spreads. Tier choice is a design decision before any signal work. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Trading Systems and Methods - Perry Kaufman (Amazon)](https://www.amazon.com/s?k=Trading+Systems+and+Methods+Perry+Kaufman&ref=aligrithm.com) - [Cybernetic Trading Strategies - Murray Ruggiero (Amazon)](https://www.amazon.com/s?k=Cybernetic+Trading+Strategies+Murray+Ruggiero&ref=aligrithm.com) - [The Art of Currency Trading - Brent Donnelly (Amazon)](https://www.amazon.com/s?k=The+Art+of+Currency+Trading+Brent+Donnelly&ref=aligrithm.com) ### 4.24 From Intermarket Analysis to Network Momentum URL: https://aligrithm.com/from-intermarket-analysis-to-network-momentum/ Last updated: 2026-07-27T13:24:25.000Z Classical intermarket analysis is a hand-drawn graph. A trader decides bonds link to stocks, gold links to the dollar, copper links to yields, and writes those edges in by hand from economic reasoning. The whole web in ["Intermarket Analysis for System Traders"](https://aligrithm.com/intermarket-analysis-for-system-traders/) is maybe a dozen edges chosen by a human who knew the macro story. The obvious question, once you have built a few of these by hand, is whether you can stop guessing the edges and learn the entire graph from the price data itself, then let a signal flow along it. That is network momentum, and it is the natural endpoint of everything in this pillar. The idea has one moving part. Momentum spills over between connected assets: when one asset trends, the assets linked to it tend to inherit a piece of that trend a little later. Classical intermarket already used this for single pairs (bonds leading stocks is momentum spilling from one to the other). Network momentum generalizes it from a pair to a whole graph, and from a hand-picked edge to a learned one. Instead of a trader asserting that bonds matter for stocks, an algorithm infers the connection strengths from how the assets' momentum features co-move, then builds each asset's signal from its neighbors. This article closes the intermarket arc by reframing the hand-built web of the earlier articles as a special, sparse, human-specified case of a learned network. It builds on the lead-lag measurement from ["Lead-Lag Relationships in Global Markets"](https://aligrithm.com/lead-lag-relationships-in-global-markets/) and the confirmation logic from ["Cross-Asset Confirmation for Trend Systems"](https://aligrithm.com/cross-asset-confirmation-for-trend-systems/), both of which become automatic once the graph is learned. ## The network momentum signal The core construction is simple enough to write in one line. For a target asset, its network momentum signal is the weighted average of its neighbors' momentum, where the weights are the learned connection strengths. $$ \\text{NetMom}\_i \\;=\\; \\sum\_{j \\neq i} A\_{ij}\\,\\text{Mom}\_j, \\qquad A\_{ij} = \\text{learned connection strength between assets } i \\text{ and } j $$ The network momentum of asset i is the sum over all other assets j of the connection strength between i and j times asset j's own momentum. The matrix A holds the graph: a large entry means a strong link, a zero means no link. Set A by hand with a dozen economic edges and you recover classical intermarket analysis. Learn A from the data across dozens of assets and you get network momentum. The equation is the same; the difference is who fills in the weights and how many there are. The momentum that spills is the standard time-series kind: an asset's own recent trend, normalized by its volatility. The network turns each asset's individual momentum into a second signal built from its neighbors, and the two can be combined. An asset whose own momentum is flat but whose well-connected neighbors are all trending up inherits an upward network signal that its own chart does not show. ## Learning the graph from prices alone The practical breakthrough is that the graph is learned from pricing data, with no fundamental database, no company linkages, no sector tags. Earlier momentum-spillover work needed an explicit relationship to exist (two stocks of the same company, firms in a supply chain), which confined it to within-equity links. Learning the network from how momentum features co-move removes that constraint and lets the graph span asset classes, the thing classical intermarket analysis reached for with a handful of hand-drawn cross-asset edges. A linear, interpretable graph-learning model is enough; the network does not require a black box. Run across 64 continuous futures spanning commodities, equities, fixed income, and currencies, the learned graph reproduces the macro intuition rather than contradicting it. The assets cluster into communities that match their asset classes. Commodities act as a bridge, connecting to the other three classes, which is the data's version of the copper-and-crude-feed-everything story from ["Copper as an Economic Activity Indicator"](https://aligrithm.com/copper-as-an-economic-activity-indicator/) and ["Crude Oil, Inflation, and FX"](https://aligrithm.com/crude-oil-inflation-and-fx/). Equities and fixed income show little direct spillover between them, consistent with the two being hedges that often move opposite, the same sign-instability the bond-equity filter had to manage. ## What the empirical result says The honest summary of what learning the graph buys you. A network momentum strategy built on the learned graph, after volatility scaling, posted a Sharpe ratio around 1.5 and an annual return near 22% over roughly 2000 to 2022\. An end-to-end version that learns the network and optimizes the trading signal together pushed the Sharpe toward 1.7 over a comparable window. Those are strong numbers for a systematic multi-asset strategy, and the more important point is where they come from: a signal source that is neither each asset's own momentum nor a hand-coded intermarket rule, but the propagation of momentum across a graph inferred from prices. The result also validates the whole premise of this pillar. Cross-asset information is real and tradeable, the claim ["Why Cross-Asset Signals Beat Isolated Chart Reading"](https://aligrithm.com/why-cross-asset-signals-beat-isolated-chart-reading/) made with a single bond-equity split, now demonstrated across an entire learned network of assets. The hand-built intermarket web was a low-resolution sketch of a structure that, drawn at full resolution from the data, carries a measurable risk premium. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Why this does not retire the hand-built web The skeptical balance, because a learned graph is not magic and the earlier articles still matter. It is still induction. The learned graph is estimated from history, and like every relationship in this pillar it drifts; the connection strengths in a turbulent year differ from a calm one. A network learned on one regime degrades in the next, the inductive limit ["Induction in Trading: Why Past Patterns Are Always Uncertain"](https://aligrithm.com/induction-in-trading-why-past-patterns-are-always-uncertain/) applies to graphs as much as to single rules. Re-learning the graph on a rolling basis is mandatory, and even then the strategy carries the regime-change cost every system carries. It has more parameters, so it overfits more easily. A graph over dozens of assets has far more free quantities than a dozen hand-drawn edges, and more free quantities means more ways to fit noise. The linear, interpretable, sparse formulation is what keeps this in check; a denser or more flexible model would fit the backtest beautifully and fail live, the warning from "Why Works on All Markets Is Usually a Red Flag" scaled up to a network. The economic reasoning behind the hand-built web is the sanity check on the learned one: when the learned graph reproduces the known macro communities, you trust it; when it asserts a strong link with no conceivable economic basis, you suspect overfit. It does not remove costs. The strategy trades many assets, and the turnover, spreads, and slippage across 64 futures are real, the constraint ["Why Transaction Costs Should Be Added Before You Fall in Love"](https://aligrithm.com/why-transaction-costs-should-be-added-before-you-fall-in-love/) puts on every multi-asset book. A Sharpe measured before costs is a different animal from one measured after, and the network's many small cross-asset tilts are exactly the kind of signal that can be eaten alive by execution if traded carelessly. ## Visualizing the learned network ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_109-visual-0.png) ## KEY POINTS - Classical intermarket analysis is a hand-drawn graph: a human writes in about a dozen economic edges. Network momentum learns the entire graph from price data and lets a signal flow along it. - The one moving part is momentum spillover: when one asset trends, its connected assets inherit a piece of that trend a little later. Network momentum generalizes this from a single pair to a whole graph. - The signal in one line: an asset's network momentum is the connection-weighted average of its neighbors' momentum. Fill the connection matrix by hand and you recover classical intermarket; learn it from data and you get network momentum. Same equation, different source and scale. - The graph is learned from prices alone, with no fundamental database, which lets it span asset classes where earlier spillover work was confined to within-equity company links. - A linear, interpretable model is enough. Across 64 futures the learned graph reproduces macro intuition: assets cluster by class, commodities bridge to the other three, and equities and fixed income show little spillover because they hedge each other. - Empirically, a vol-scaled network momentum strategy posted a Sharpe near 1.5 and about 22% annual return over roughly 2000 to 2022, with an end-to-end version near 1.7\. The signal source is neither own-momentum nor a hand-coded rule but propagation across a learned graph. - The result validates the pillar's premise: cross-asset information carries a real, tradeable risk premium, demonstrated across a whole network rather than a single bond-equity split. - It does not retire the hand-built web. It is still induction and the graph drifts; it has more parameters and overfits more easily, kept in check by a sparse interpretable form and economic sanity checks; and it trades many assets, so costs are real and can eat the edge. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Trading Systems and Methods - Perry Kaufman (Amazon)](https://www.amazon.com/s?k=Trading+Systems+and+Methods+Perry+Kaufman&ref=aligrithm.com) - [Cybernetic Trading Strategies - Murray Ruggiero (Amazon)](https://www.amazon.com/s?k=Cybernetic+Trading+Strategies+Murray+Ruggiero&ref=aligrithm.com) - [The Art of Currency Trading - Brent Donnelly (Amazon)](https://www.amazon.com/s?k=The+Art+of+Currency+Trading+Brent+Donnelly&ref=aligrithm.com) ### 4.23 Lead-Lag Relationships in Global Markets URL: https://aligrithm.com/lead-lag-relationships-in-global-markets/ Last updated: 2026-07-27T13:24:25.000Z Short-term interest-rate markets turn before the long bond at major inflection points, and the long bond turns before the broad equity market. Gold-miner stocks turn before gold. Oil-producer stocks turn with or slightly ahead of crude. Each of these is a lead-lag relationship: one market moves first and a related market follows, with a measurable delay. A leading market is the rarest and most valuable thing in intermarket analysis, because it is a forward-looking signal, a read on where the lagging market is headed before the lagging market shows it. The catch is that lead-lag is the most fragile relationship in the web. A correlation can be stable for decades; a lead can decay in a year, reverse without warning, or turn out to be an artifact of two markets sharing a slow common driver. The seductive part (knowing the future of one market by watching another) is exactly the part most likely to be a backtest illusion. This article treats lead-lag as a measurement problem with a high false-positive rate, not as a list of leads to trust. It follows directly from ["Cross-Asset Confirmation for Trend Systems"](https://aligrithm.com/cross-asset-confirmation-for-trend-systems/), which left open the question of which market should confirm and when, and it feeds ["From Intermarket Analysis to Network Momentum"](https://aligrithm.com/from-intermarket-analysis-to-network-momentum/), where the lead-lag structure becomes a learned graph rather than a hand-picked pair. ## Measuring a lead A lead-lag relationship is found by the lagged cross-correlation: correlate one market's returns today against another market's returns some number of periods later, and scan the lag that maximizes the correlation. If market X's returns at time t correlate most strongly with market Y's returns at time t plus k, then X leads Y by k periods. $$ \\rho\_{XY}(k) = \\text{corr}\\big(X\_t,\\; Y\_{t+k}\\big), \\qquad \\hat{k} = \\arg\\max\_{k} \\,\\rho\_{XY}(k) $$ The lagged cross-correlation is the correlation between market X at time t and market Y at time t plus k, computed across a range of lags k. The estimated lead is the lag that maximizes it. A positive peak at a positive k says X leads Y by that many periods; a peak at k equal to zero says they move together with no exploitable lead. The signal you want is a clear, single peak at a positive lag that survives out of sample. The signal you usually get is a noisy curve with several near-equal bumps, and picking the highest bump is how lead-lag becomes data-mining. ## The leads worth knowing The classical leads cluster around markets that sit closer to the economic cause than the markets they lead. Short rates lead long rates and equities, because short-rate markets price central-bank policy directly, and policy is upstream of both the long bond and the equity discount rate. Utility stocks have historically led the broad equity market and the bond market at major tops, because their rate sensitivity makes them an early casualty of rising yields. Gold-miner stocks lead gold, and oil-producer stocks lead or coincide with crude, because the equity of a producer discounts the future commodity price that the producer's earnings depend on, the relationship the article ["Crude Oil, Inflation, and FX"](https://aligrithm.com/crude-oil-inflation-and-fx/) used in reverse to read crude from the oil complex. The common thread: the leader is the market with the most forward-looking participants or the tightest link to the underlying driver. A market that prices expectations leads a market that prices the present. That is the only economically defensible reason to expect a lead, and a lead without such a reason is a coincidence waiting to break. ## Why most leads are illusions The honest core of the article. Three mechanisms manufacture fake leads, and all three are common. The shared slow driver. Two markets both driven by the same slow macro variable will show a lead-lag pattern in the data even when neither causes the other, purely because the common driver hits them with slightly different delays. This is the ice-cream-and-drowning problem: both rise in summer, neither causes the other. A lead found this way vanishes the moment the common driver behaves differently, and it gives you no real predictive power over the lagging market. The arbitraged-away lead. A genuine, stable, exploitable lead is a free forecast, and free forecasts get traded until they disappear. The most-cited leads from decades past decayed as markets grew more efficient and faster, the way the article ["Why Systems Work Until They Don't"](https://aligrithm.com/why-systems-work-until-they-dont/) framed every edge. A lead that was real in slow markets can be gone in fast ones. The data-mined lead. Scan enough market pairs and enough lags and you will find impressive lead-lag correlations by chance. With dozens of markets and dozens of candidate lags, the number of combinations is huge, and the best one looks tradeable on the sample and is noise out of sample. The defenses are the ones the article ["Induction in Trading: Why Past Patterns Are Always Uncertain"](https://aligrithm.com/induction-in-trading-why-past-patterns-are-always-uncertain/) insisted on: restrict to economically motivated pairs before testing, demand a single clean peak rather than the best of many bumps, and confirm the lead holds on data you did not use to find it. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Using a lead without getting burned If a lead survives the economic-reason test and out-of-sample validation, use it conservatively. Treat the leader as a confirmer or a tilt, not as a standalone signal, because even a real lead explains only a fraction of the lagging market's move. Size the bet to the strength of the lead, which the cross-correlation peak measures: a lead that explains a small slice of variance deserves a small tilt, not a full position. Re-estimate the lead on a rolling window and watch for the peak migrating toward zero lag or splitting, which is the early sign the lead is decaying or inverting. A lead that was three days last year and is one day this year is on its way to zero, and a system that hard-codes the old lag will be trading a relationship that no longer exists, the slow-drift failure the article ["Slow Wandering: The Most Dangerous Type of Market Change"](https://aligrithm.com/slow-wandering-the-most-dangerous-type-of-market-change/) warned about. The discipline is to trust the lead exactly as much as the current data supports and not one period more. ## Visualizing the cross-correlation scan ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_108-visual-0.png) ## KEY POINTS - A lead-lag relationship is one market moving first and a related market following with a measurable delay. A leading market is a forward-looking signal, the rarest and most valuable thing in intermarket analysis. - Lead-lag is also the most fragile relationship in the web: leads decay, reverse, or turn out to be artifacts of a shared slow driver. The most seductive part is the most likely to be an illusion. - Measure a lead with the lagged cross-correlation: correlate market X at time t against market Y at time t plus k, and find the lag that maximizes it. A real lead is a single clean peak at a positive lag that survives out of sample. - The defensible leads sit closer to the economic cause: short rates lead long rates and equities, utilities have led the market and bonds at tops, and producer stocks lead their commodity. A market pricing expectations leads a market pricing the present. - Three mechanisms manufacture fake leads: a shared slow driver hitting both markets with different delays (the ice-cream-and-drowning trap), a real lead getting arbitraged away as markets speed up, and a data-mined lead from scanning too many pairs and lags. - The defenses: restrict to economically motivated pairs before testing, demand a single clean peak rather than the best of many bumps, and validate on unused data. - Use a surviving lead conservatively, as a confirmer or tilt rather than a standalone signal, and size it to the variance the lead actually explains. - Re-estimate the lead on a rolling window and watch the peak migrate toward zero or split, the early sign of decay or inversion. Trust the lead exactly as much as current data supports. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Trading Systems and Methods - Perry Kaufman (Amazon)](https://www.amazon.com/s?k=Trading+Systems+and+Methods+Perry+Kaufman&ref=aligrithm.com) - [Cybernetic Trading Strategies - Murray Ruggiero (Amazon)](https://www.amazon.com/s?k=Cybernetic+Trading+Strategies+Murray+Ruggiero&ref=aligrithm.com) - [The Art of Currency Trading - Brent Donnelly (Amazon)](https://www.amazon.com/s?k=The+Art+of+Currency+Trading+Brent+Donnelly&ref=aligrithm.com) ### 4.22 Cross-Asset Confirmation for Trend Systems URL: https://aligrithm.com/cross-asset-confirmation-for-trend-systems/ Last updated: 2026-07-27T13:24:24.000Z A moving-average trend system on the S&P fires a long. Left alone, it takes the trade. Add one rule: take the long only if T-bonds are also in an uptrend, confirming the move from the rate side. Over a long sample the unconfirmed system takes 100 trades a year at a 51% hit rate; the confirmed system takes 60 trades a year at a 56% hit rate, and the 40 trades it skipped were the ones where the equity chart was trending up while the bond market was already pricing the move to end. Confirmation did not improve the system's logic. It removed the trades that the rest of the macro picture was voting against. Confirmation is the close cousin of the intermarket filter, with a different emphasis. A filter gates a whole strategy on or off by an external state, the way ["Using Bonds to Filter Equity Signals"](https://aligrithm.com/using-bonds-to-filter-equity-signals/) switched the equity book by the bond trend. Confirmation operates trade by trade: a signal in the traded market must be seconded by an agreeing signal in a related market before you act. The filter asks "is the regime friendly"; confirmation asks "does a second market agree with this specific trade". They overlap, and the distinction matters because confirmation is where trend systems win back their worst weakness, the whipsaw. This article takes the confirmation idea from the intermarket templates in ["Intermarket Analysis for System Traders"](https://aligrithm.com/intermarket-analysis-for-system-traders/) and applies it to the whipsaw problem that defines trend following. It pairs with ["Why FX Traders Must Watch Gold, Rates, and Equities"](https://aligrithm.com/why-fx-traders-must-watch-gold-rates-and-equities/), which built the same confirmation count for currencies, and it leads into ["Lead-Lag Relationships in Global Markets"](https://aligrithm.com/lead-lag-relationships-in-global-markets/), which asks which market should be doing the confirming. ## Why trend systems need a second opinion A trend follower's structural cost is the whipsaw: it enters on a breakout, the move fails, it exits at a loss, and it repeats this through every choppy stretch. The article ["Why Breakout Systems Need Low Noise Environments"](https://aligrithm.com/why-breakout-systems-need-low-noise-environments/) framed the noise side of this. Confirmation attacks it from the cross-asset side. The premise is that a real trend in one market usually shows up as a coordinated move across its related markets, while a false breakout is a local event that the related markets do not echo. A dollar rally that is real shows up in falling gold and a widening rate differential at the same time. A dollar rally that is noise shows up on the dollar chart alone, with gold and rates indifferent. Requiring the related market to confirm filters the second kind of move, because the noise breakout has no macro second. $$ \\text{Trade}\_t = \\text{Signal}\_{\\text{traded},t} \\;\\wedge\\; \\big\[\\,\\text{sign}(\\text{trend}\_{\\text{confirm},t}) = \\text{expected sign}\\,\\big\] $$ A trade fires only when the traded market's own signal and the confirming market's trend both point the agreed way (the logical AND of the two conditions). For a positively linked confirmer the expected sign matches the traded signal; for a negatively linked one it is flipped. The whole mechanism is this AND: two markets must agree, and the trades that die are the ones where the second market refused to second the first. ## The cost of confirmation Nothing in trading is free, and confirmation has three costs you pay for the cleaner trades. Fewer trades. Requiring agreement throws out every signal the confirmer disagrees with, and a thinner sample is a less reliable sample. Confirm too hard, with several markets that all must agree, and you arrive at a handful of trades a year and a hit rate you cannot trust, the small-sample fragility the article ["Trade-Count Thresholds for Backtest Reliability"](https://aligrithm.com/trade-count-thresholds-for-backtest-reliability/) warned about. Lag. The confirming market has to establish its own trend before it can confirm, so the confirmed entry arrives later than the raw signal. For a fast trend system that captures most of its profit in the first days of a move, the lag eats the edge. Confirmation suits slower trend systems where the move lasts long enough that a late entry still pays. More parameters to overfit. The confirmer adds a market choice and a trend-definition length, and every added knob is another way to fit the backtest. A confirmation rule found by searching which of thirty markets and which moving-average length best improves the equity curve is data-mined, the trap "Why Works on All Markets Is Usually a Red Flag" described. The confirmer must be chosen for an economic reason before the test, not selected for its backtest contribution. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## What to confirm with The confirming market should be the one with the tightest economic link to the traded market, not the one that scores best in-sample. For an equity trend, bonds and the rate complex are the natural confirmers, because they share the discount rate. For a currency trend, the rate differential, gold, and the risk-appetite proxy from ["Why FX Traders Must Watch Gold, Rates, and Equities"](https://aligrithm.com/why-fx-traders-must-watch-gold-rates-and-equities/) are the natural confirmers. For a commodity-currency trend, the underlying commodity confirms, the link worked in ["Crude Oil, Inflation, and FX"](https://aligrithm.com/crude-oil-inflation-and-fx/). A second design choice is how many confirmers and how strict. A single confirmer with a simple agree/disagree test is robust and cheap. A confirmation count across two or three related markets, taking the trade when a majority agree, is steadier but spends more degrees of freedom. The conservative default is one well-motivated confirmer, because the marginal confirmer adds correlation, not independence: related markets agree with each other, so the third confirmer rarely says anything the first two did not. ## When confirmation backfires The honest failure modes, because confirmation is not a free upgrade. The link inverts. Confirmation assumes the sign of the relationship, and when stock-bond or dollar-gold correlation flips regime, the confirmer starts vetoing the right trades and approving the wrong ones. A confirmation rule needs the same rolling correlation check that every relationship in this pillar needs, or it will confidently filter you into the losing half during the regime where the sign is backward. Both markets are wrong together. In a correlated panic every market moves the same way, so they all confirm each other into the same bad trade. Confirmation protects against idiosyncratic noise in one market; it offers nothing against a systemic move that drags every confirmer along. The diversification of signal source the intermarket overview promised is exactly the thing that fails when correlations spike to 1. The confirmer leads, so it has already moved. If the confirming market leads the traded market, by the time it confirms it may be near the end of its own move, and you are entering the traded market just as the driver exhausts. This is why confirmation and lead-lag are two sides of one question, and why the next article, ["Lead-Lag Relationships in Global Markets"](https://aligrithm.com/lead-lag-relationships-in-global-markets/), treats the timing of the confirmer as the real problem. ## Visualizing confirmation ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_107-visual-0.png) ## KEY POINTS - Confirmation requires a second related market to agree with a trade before you take it. A filter gates a whole strategy by regime; confirmation operates trade by trade. - The premise: a real trend shows up as a coordinated move across related markets, while a false breakout is a local event the related markets do not echo. Confirmation removes the local noise. - The mechanism is a logical AND: the traded market's signal and the confirmer's trend must both point the agreed way (matched sign for a positive link, flipped for a negative one). - Confirmation attacks the trend follower's core weakness, the whipsaw, from the cross-asset side, complementing the noise-based defenses. - It costs three things: fewer trades (a thinner, less reliable sample), lag (the confirmer must establish its own trend first, hurting fast systems), and more parameters to overfit (a market choice and a length). - Choose the confirmer for the tightest economic link before testing, not for its backtest contribution. Bonds confirm equities; rates, gold, and risk appetite confirm currencies; the underlying commodity confirms a commodity currency. - One well-motivated confirmer is the robust default. Extra confirmers add correlation, not independence, because related markets already agree with each other. - Failure modes: the link inverts across regimes and the confirmer vetoes the right trades; a correlated panic makes every confirmer agree into the same bad trade; and a leading confirmer may have already exhausted its move by the time it confirms. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Trading Systems and Methods - Perry Kaufman (Amazon)](https://www.amazon.com/s?k=Trading+Systems+and+Methods+Perry+Kaufman&ref=aligrithm.com) - [Cybernetic Trading Strategies - Murray Ruggiero (Amazon)](https://www.amazon.com/s?k=Cybernetic+Trading+Strategies+Murray+Ruggiero&ref=aligrithm.com) - [The Art of Currency Trading - Brent Donnelly (Amazon)](https://www.amazon.com/s?k=The+Art+of+Currency+Trading+Brent+Donnelly&ref=aligrithm.com) ### 4.21 Why FX Traders Must Watch Gold, Rates, and Equities URL: https://aligrithm.com/why-fx-traders-must-watch-gold-rates-and-equities/ Last updated: 2026-07-27T13:24:23.000Z A trader is short EURUSD on a clean chart signal: lower highs, price under the 50-day average, a tidy downtrend. The position bleeds for a week. Pull up three charts the trader never looked at and the reason is obvious. The US-Germany two-year rate differential narrowed (the rate edge moved against the dollar), gold rallied hard (a dollar-weakness tell), and global equities ripped higher (risk-on, which sells the funding currencies the dollar now sits among). Three external markets were screaming "dollar down, EURUSD up" while the EURUSD chart still looked short. The chart was the last thing to know. FX is the asset class where single-chart trading fails worst, because a currency price is a relative number with no standalone fundamentals. A stock has earnings; a bond has a coupon; a currency pair is one economy priced against another, and the thing that moves it lives outside the pair. Three external markets carry most of that information: rates, equities, and gold. Rates are the relative-return engine, equities are the risk-appetite gauge, and gold is the inflation-and-dollar barometer. An FX system that ignores all three is trading the shadow and not the object. This article makes the case for the three drivers, shows the per-currency driver map, and turns the drivers into a confirmation filter. It applies the gold-dollar-rates triangle from ["Gold, Dollar, and Rates: A Practical Intermarket Map"](https://aligrithm.com/gold-dollar-and-rates-a-practical-intermarket-map/) to the currency a trader actually holds, and it sets up the mechanical version in ["Cross-Asset Confirmation for Trend Systems"](https://aligrithm.com/cross-asset-confirmation-for-trend-systems/) and the gold-specific case in ["Using Gold as an FX Indicator"](https://aligrithm.com/using-gold-as-an-fx-indicator/). ## Driver one: rates, the most consistent mover The single most reliable driver of a currency is its relative interest rate, and currencies react violently to unexpected central-bank shifts. Money flows toward the higher risk-adjusted yield, so a pair tends to follow the rate differential between its two legs: when the base currency's rates rise relative to the quote currency's, the pair appreciates. The level matters less than the change and the surprise; the market has already discounted the expected path, so the move comes from the revision. $$ \\Delta\\text{Pair}\_t \\;\\approx\\; \\beta \\,\\Delta\\!\\left(r\_{\\text{base}} - r\_{\\text{quote}}\\right)\_t \\;+\\; \\gamma \\,\\Delta\\text{RiskAppetite}\_t \\;+\\; \\delta \\,\\Delta\\text{Gold}\_t $$ The change in the pair over a period is approximated by three terms: the change in the rate differential between the base and quote currencies (driver one), the change in global risk appetite proxied by equities (driver two), and the change in gold (driver three), each scaled by a sensitivity. This is a schematic, not an estimated model; the coefficients drift and the relevant terms rotate over time. The value of writing it down is that it names the three things to watch and says the pair moves on their changes, not their levels. Carry sits on top of the rate term. High-yield currencies structurally appreciate as the market piles into them to earn the differential, and that appreciation unwinds far faster than it built, the "up the escalator, down the elevator" pattern of carry. Low-yield currencies (the euro, the franc, the yen in recent cycles) become funding currencies and behave as safe havens, strengthening when carry trades unwind. A rate differential tells you the direction of the escalator; it does not tell you when the elevator drops, which is why the rate driver needs the risk-appetite driver beside it. ## Driver two: equities, the risk-appetite gauge Equities are the cleanest read on global risk appetite, and risk appetite drives FX regardless of domestic fundamentals. When equities rally, capital chases the high-beta and commodity currencies; when equities fall, capital flees to the safe havens. The pairings are well documented: the S&P 500 moves with AUDUSD, the Nikkei moves with USDJPY. A trader long a commodity currency is implicitly long global equities, whether or not the currency chart says so. The regime warning is sharp here. In the 2008 crisis the correlation between FX and other asset classes jumped toward 1: everything risky sold together and everything safe was bought together, and the individual currency stories stopped mattering. Correlation feeds on itself in stress, rising until a regime shift snaps it. A risk-appetite filter built in calm markets behaves differently in a panic, when it becomes the only thing that matters, the acceptance the article ["Why Systems Work Until They Don't"](https://aligrithm.com/why-systems-work-until-they-dont/) built into every relationship in this pillar. The dollar complicates the equity link, because the dollar does not behave like a simple risk asset. The dollar tends to rally when US growth is strong and again when the global economy is in recession, and to sell off only in the soft middle where US growth is moderate and global growth is fine. That U-shape, the "dollar smile", means the sign of the dollar-equity correlation depends on which side of the smile you are on: in a growth scare the dollar and equities fall together, but in a genuine global recession the dollar rallies as equities crash. A naive "stocks up, dollar down" rule gets the recession case backward. ## Driver three: gold, the inflation-and-dollar barometer Gold appears in almost every major currency's driver list, because it reads two things at once: the inflation-and-real-yield backdrop and the dollar itself. Gold trades inverse to the dollar, so gold strength is a direct dollar-weakness signal that shows up across every USD pair simultaneously. Gold also anticipates inflation, which feeds back into the rate driver through what it implies for central-bank policy. The gold-dollar-rates triangle is the reason this one instrument informs the whole currency complex, worked in full in ["Gold, Dollar, and Rates: A Practical Intermarket Map"](https://aligrithm.com/gold-dollar-and-rates-a-practical-intermarket-map/). For a currency trader the practical use is as a cross-check on the dollar leg of any pair. A long-dollar trade taken while gold is breaking higher is fighting a market that disagrees, the same divergence logic from ["Intermarket Divergence as a Trading Filter"](https://aligrithm.com/intermarket-divergence-as-a-trading-filter/) applied to the dollar. Gold tracks real yields rather than nominal ones, so the caveat from the triangle article carries over: in an inflation spike, nominal yields can rise while gold rallies, and a trader reading the nominal rate alone will be confused by gold's refusal to fall. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## The per-currency driver map Each currency has its own driver set, and watching the wrong external market for a given pair adds noise instead of signal. The map below is the working version, with gold and rates appearing almost everywhere and equities or commodities filling the third slot. | Pair | Rate driver | Equity / risk driver | Gold / commodity driver | | ------ | ------------------------------ | ---------------------------------- | ----------------------- | | EURUSD | US-Germany 2-year differential | European bank stocks, Italy yields | gold, crude | | USDJPY | US 2-10 year yields | Nikkei, S&P 500 | gold | | AUDUSD | Australia-US differential | S&P 500 (risk appetite) | gold, copper, iron ore | | USDCAD | US-Canada differential | Canadian oil equities | crude, gold | | GBPUSD | UK-US differential | global risk appetite | crude | The commodity currencies extend the gold idea to their own inputs: the Australian dollar tracks gold, copper, and iron ore; the Canadian dollar tracks crude, the link worked in ["Crude Oil, Inflation, and FX"](https://aligrithm.com/crude-oil-inflation-and-fx/). The principle from ["Why One Indicator Should Not Be Used on Every Market"](https://aligrithm.com/why-one-indicator-should-not-be-used-on-every-market/) applies directly: there is no universal FX driver, only the right external market for the specific pair. ## Turning three drivers into a filter The operational payoff is a confirmation count. For a given FX signal, check whether the rate differential, the equity/risk proxy, and gold each agree with the position, and take only the trades where a majority confirm. A short-dollar trade confirmed by a narrowing rate differential, rising equities, and rising gold is backed by three independent readings; the same trade with all three disagreeing is the chart fighting the macro, and the chart usually loses that fight. The confirmation count works for the same reason the bond filter worked in ["Using Bonds to Filter Equity Signals"](https://aligrithm.com/using-bonds-to-filter-equity-signals/): it splits the signals into a cleaner subset (drivers agree) and a dirtier subset (drivers disagree), and trading only the cleaner subset lifts the per-trade quality at the cost of fewer trades. The honest limit is the same too. The drivers are correlated with each other, so three agreeing readings are not three independent votes, and the relevant driver rotates over time, so a fixed confirmation rule needs the rolling re-estimation that ["Why Cross-Asset Signals Beat Isolated Chart Reading"](https://aligrithm.com/why-cross-asset-signals-beat-isolated-chart-reading/) insisted on. Counting which driver leads right now is the harder, more valuable problem, and it is the subject of ["Lead-Lag Relationships in Global Markets"](https://aligrithm.com/lead-lag-relationships-in-global-markets/). ## Visualizing the three-driver dashboard ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_106-visual-0.png) ## Visualizing the dollar smile ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_106-visual-1.png) ## KEY POINTS - A currency price is a relative number with no standalone fundamentals, so the force that moves it lives outside the pair. Single-chart trading fails worst in FX. - Three external markets carry most of the information: rates (the relative-return engine), equities (the risk-appetite gauge), and gold (the inflation-and-dollar barometer). - Rates are the most consistent driver. Pairs follow the change and the surprise in the rate differential, not its level. Carry rides on top: high-yield currencies climb the escalator and drop down the elevator, while low-yield funding currencies act as safe havens. - Equities read global risk appetite, which moves FX regardless of domestic fundamentals. The S&P 500 tracks AUDUSD, the Nikkei tracks USDJPY. In 2008 the FX-asset correlation jumped toward 1 and individual stories stopped mattering. - The dollar smile breaks the naive equity link: the dollar rallies on strong US growth and again in a global recession (safe haven), selling off only in the soft middle. The sign of the dollar-equity correlation depends on which side of the smile you are on. - Gold reads the inflation-and-real-yield backdrop and the dollar at once, and it appears in nearly every currency's driver list. Gold strength is a dollar-weakness signal across all USD pairs simultaneously. - Each pair has its own driver set: EURUSD on the US-Germany differential and crude, AUDUSD on gold, copper, and iron ore, USDCAD on crude. There is no universal FX driver, only the right external market for the pair. - The filter is a confirmation count: take the trade only when a majority of rate, equity, and gold readings agree. It splits signals into a cleaner subset and a dirtier one, lifting per-trade quality for fewer trades. - The limits: the three drivers are correlated, so agreement is not three independent votes; the relevant driver rotates, so the rule needs rolling re-estimation; and the correlations spike in crisis then break at regime shifts. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Trading Systems and Methods - Perry Kaufman (Amazon)](https://www.amazon.com/s?k=Trading+Systems+and+Methods+Perry+Kaufman&ref=aligrithm.com) - [Cybernetic Trading Strategies - Murray Ruggiero (Amazon)](https://www.amazon.com/s?k=Cybernetic+Trading+Strategies+Murray+Ruggiero&ref=aligrithm.com) - [The Art of Currency Trading - Brent Donnelly (Amazon)](https://www.amazon.com/s?k=The+Art+of+Currency+Trading+Brent+Donnelly&ref=aligrithm.com) ### 4.20 Intermarket Divergence as a Trading Filter URL: https://aligrithm.com/intermarket-divergence-as-a-trading-filter/ Last updated: 2026-07-27T13:24:23.000Z Two markets are supposed to move together. Bonds and stocks share a discount rate; gold and the dollar trade inverse; copper and yields rise together. Most of the time they honor those relationships, and the relationship is uninformative because everyone already prices it in. The signal lives in the moments they disagree. Bonds rallying hard while stocks keep falling, when the two normally move together, is an intermarket divergence: the traded market is doing the opposite of what its partner says it should. That disagreement is information, because one of the two markets is usually right, and the divergence flags which trades to trust and which to fade. This article defines intermarket divergence precisely, gives the two standard ways to measure it, and turns it into a filter using the divergence template from ["Intermarket Analysis for System Traders"](https://aligrithm.com/intermarket-analysis-for-system-traders/). It is the most subtle of the four templates because it fires on disagreement rather than state, and it closes the intermarket arc that ["Using Ratios as Trading Signals"](https://aligrithm.com/using-ratios-as-trading-signals/) and the gold-dollar-rates map opened. ## Defining divergence operationally Divergence is the traded market moving against what its intermarket partner predicts, given their normal correlation. The definition has two parts: the expected relationship, and the observed violation of it. For a positively correlated pair (stocks and bonds), divergence is the two moving in opposite directions: bonds up, stocks down, or bonds down, stocks up. For a negatively correlated pair (gold and the dollar), divergence is the two moving in the same direction: both rising or both falling, when they normally trade inverse. The sign of the expected correlation flips what counts as a divergence, which is why you must know the pair's normal sign before you can detect a violation of it. The trading interpretation rests on a lead-lag assumption: when the leading market and the lagging market disagree, the leading market tends to win, and the lagging market resolves toward it. Bonds leading stocks means a bond-stock divergence often resolves with stocks following bonds, so the divergence is a signal to position in the direction of the leader, the lead-lag idea the article ["Lead-Lag Relationships in Global Markets"](https://aligrithm.com/lead-lag-relationships-in-global-markets/) treats as a measurement problem rather than an assumption to take on faith. ## Two ways to measure it The two standard measures differ in what they compare, and they often agree, which lets them cross-check each other. Measure one, momentum divergence. Compute a momentum (rate of change over n bars) for both the traded market and its partner. Divergence is when the two momentums have signs that violate the expected correlation: opposite signs for a positive pair, same signs for a negative pair. This is fast and sensitive, and noisy, because short-window momentum flips easily. Measure two, moving-average divergence. Compare each market's price to its own moving average. Divergence is when the traded market is below its average while the partner (positive pair) is above its average, or the equivalent sign-flipped condition for a negative pair. This is slower and steadier than momentum divergence, because a price crossing its moving average is a higher bar than a momentum sign flip. $$ \\text{Divergence (positive pair)} = \\big\[\\,\\text{sign}(\\Delta A) \\neq \\text{sign}(\\Delta B)\\,\\big\], \\qquad \\Delta X = X\_t - X\_{t-n} \\;\\;\\text{or}\\;\\; X\_t - \\text{MA}\_n(X) $$ A divergence on a positively correlated pair fires when the change in market A and the change in market B have opposite signs, where the change is either an n-bar momentum or the gap between price and its moving average. For a negatively correlated pair you flip the condition to fire when the signs match. Using both the momentum and the moving-average versions and requiring them to agree filters out the weakest divergences, the cross-check discipline that runs through this pillar. ## The filter Divergence is used two ways, and they point in opposite directions, so you must decide which you believe for a given pair. Confirmation use (trade with the leader). When the leading market diverges from the lagging traded market, position the traded market in the direction the leader implies, expecting the lagger to catch up. Bonds rallying while stocks fall, read through a positive correlation, says stocks should turn up to follow bonds, so a divergence becomes a long-stock signal. This treats divergence as an early warning that the traded market is about to reverse toward its partner. Caution use (suppress signals during divergence). Treat any divergence as a regime where the normal relationship is broken and the traded market's own signals are less reliable, so reduce size or stand aside until the markets realign. This is the conservative reading: divergence means "the macro plumbing and the chart disagree, do not trust the chart until they reconcile." The two uses are not contradictory once you tie them to the lead-lag structure. If you have evidence the partner leads, use confirmation and trade toward the leader. If you do not know which leads, use caution and stand aside. A worked sketch on an equity trend system: suppressing long-equity entries while bonds are diverging bearishly (bonds falling, signaling rising yields, while the equity chart still says buy) removes a cluster of trades taken into a hostile macro backdrop, the same drawdown-trimming effect the bond filter produced in ["Using Bonds to Filter Equity Signals"](https://aligrithm.com/using-bonds-to-filter-equity-signals/), but triggered by disagreement rather than state. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## The traps with divergence Divergence is the trickiest template, and it fails in specific ways. You need to know who leads, and the lead is unstable. The confirmation use depends on the partner leading the traded market, and leadership shifts over time and even reverses. Betting that bonds lead stocks when in the current regime stocks are leading bonds means you fade the right market. Without a measured, current lead-lag estimate, the confirmation use is a guess, and the article ["Induction in Trading: Why Past Patterns Are Always Uncertain"](https://aligrithm.com/induction-in-trading-why-past-patterns-are-always-uncertain/) applies directly to the lead assumption. Divergences resolve both ways. A divergence can resolve by the lagger catching up to the leader (the assumption), or by the leader reverting to the lagger, or by the correlation simply having changed so the "divergence" is the new normal. Only the first resolution pays the confirmation trade. The honest expectation is a modest edge from the cases that resolve as assumed, diluted by the cases that do not, not a clean reversal signal. It is noisy and easy to over-detect. Short-window momentum divergences fire constantly, most of them meaningless. Requiring agreement between the momentum and moving-average measures, and restricting to economically linked pairs, cuts the false positives, but a divergence filter that fires too often is mostly trading noise, the over-detection problem the article "Why Works on All Markets Is Usually a Red Flag" warned about generalized to disagreement signals. The correlation sign can have flipped. A "divergence" detected against an assumed positive correlation is meaningless if the pair has quietly gone negatively correlated. The divergence detector must use the current measured correlation sign, re-estimated on a rolling window, or it will label normal behavior as a divergence and vice versa. ## Visualizing a divergence ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_105-visual-0.png) ## KEY POINTS - Markets that normally move together are uninformative when they honor the relationship, because it is already priced. The signal lives in the moments they disagree, which is an intermarket divergence. - Divergence is the traded market moving against what its partner predicts given their normal correlation: opposite directions for a positively correlated pair, the same direction for a negatively correlated pair. You must know the pair's normal sign before you can detect a violation. - The trading interpretation rests on lead-lag: when the leader and the lagger disagree, the leader tends to win and the lagger resolves toward it. - Two measures: momentum divergence (the two markets' rate-of-change signs violate the expected correlation, fast and noisy) and moving-average divergence (each price versus its own average, slower and steadier). Requiring both to agree filters the weakest signals. - Confirmation use: trade the traded market toward the direction the leader implies, expecting the lagger to catch up. Caution use: treat divergence as a broken-relationship regime and suppress the traded market's own signals until realignment. - The two uses tie to lead-lag: trade toward the leader if you have evidence the partner leads, stand aside if you do not know which leads. - Worked effect: suppressing long-equity entries while bonds diverge bearishly removes trades taken into a hostile macro backdrop, trimming drawdowns the way the bond filter did, but triggered by disagreement rather than state. - The traps: leadership is unstable and must be measured currently; divergences resolve both ways so the edge is modest and diluted; the signal is noisy and easy to over-detect; and the correlation sign can have flipped, so re-estimate it on a rolling window before labeling anything a divergence. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Trading Systems and Methods - Perry Kaufman (Amazon)](https://www.amazon.com/s?k=Trading+Systems+and+Methods+Perry+Kaufman&ref=aligrithm.com) - [Cybernetic Trading Strategies - Murray Ruggiero (Amazon)](https://www.amazon.com/s?k=Cybernetic+Trading+Strategies+Murray+Ruggiero&ref=aligrithm.com) - [The Art of Currency Trading - Brent Donnelly (Amazon)](https://www.amazon.com/s?k=The+Art+of+Currency+Trading+Brent+Donnelly&ref=aligrithm.com) ### 4.19 Using Ratios as Trading Signals URL: https://aligrithm.com/using-ratios-as-trading-signals/ Last updated: 2026-07-27T13:24:22.000Z Divide one market's price by another and you get a ratio, a single series that throws away the part of the move both markets share and keeps only their relative performance. Take the bond price divided by the stock price. When that ratio rises, bonds are outperforming stocks; when it falls, stocks are outperforming bonds. Now apply a plain moving-average rule to the ratio: above its average, favor bonds; below its average, favor stocks. You have converted two price series into one tradeable signal that captures rotation between them, and rotation is exactly what a pair of correlated markets does that you cannot see by watching either chart alone. This article covers the ratio, the fourth rule template from ["Intermarket Analysis for System Traders"](https://aligrithm.com/intermarket-analysis-for-system-traders/), as its own tool: how to build it, what it measures, how to signal off it, and the specific traps that make naive ratio systems blow up. The ratio is also the engine behind relative-strength and currency-strength work later in the pillar, including ["Currency Strength Models from Pair Decomposition"](https://aligrithm.com/currency-strength-models-from-pair-decomposition/). ## What a ratio actually measures A ratio is relative performance with the common move netted out. If two markets both rise 10%, their ratio is unchanged, because the shared move cancels in the division. The ratio only moves when one outperforms the other, which is why it isolates rotation and ignores the overall direction of the pair. $$ R\_t = \\frac{A\_t}{B\_t}, \\qquad \\text{Signal}\_t = \\begin{cases} \\text{favor } A & \\text{if } R\_t > \\text{MA}\_n(R)\\\\\[4pt\] \\text{favor } B & \\text{if } R\_t < \\text{MA}\_n(R) \\end{cases} $$ The ratio R at time t is market A's price divided by market B's price. The signal compares the ratio to its own n-period moving average: above the average, A is in a relative uptrend, so favor A; below, favor B. The rule is identical to a moving-average trend rule, but applied to relative performance instead of an outright price, so it generates a rotation signal rather than a directional one. This is what makes the ratio different from the directional gates in the earlier intermarket articles. A gate says "trade A only when B is in some state." A ratio signal says "hold whichever of A and B is winning the relative race." One filters; the other rotates. ## Two ways to use a ratio Use one, the rotation switch. Hold the outperforming asset and switch when the ratio crosses its average. The bond/stock ratio above its average puts you in bonds; below, in stocks. This is a long-only rotation that is always invested in the stronger of the two, which historically smooths the equity curve relative to holding either one, because the two assets tend to take turns leading. The cost is whipsaw when the ratio chops around its average, the same turnover problem every moving-average rule has. Use two, the spread trade. Go long the outperformer and short the underperformer, holding the ratio as a market-neutral position. This strips out the common market move entirely and bets only on the relative leg, which is the basis of pairs trading. The neutrality is the appeal and the danger: a market-neutral ratio position has no directional cushion, so if the relative relationship breaks, there is no broad market tailwind to bail you out, and the article ["When a Stop Loss Improves Risk but Destroys Edge"](https://aligrithm.com/when-a-stop-loss-improves-risk-but-destroys-edge/) applies because stopping a spread trade is its own hard problem. A ratio oscillator refines both uses. Instead of the raw ratio against its average, compute the ratio minus its average (or a normalized version), giving an oscillator centered on zero. Positive means A is outperforming, negative means B is, and the distance from zero measures how stretched the rotation is. The oscillator form makes overextension visible, which helps separate a fresh rotation from an exhausted one. ## A worked rotation Numbers, because a risk-adjusted claim is empty without them. Take a monthly bond/stock rotation using a ratio against its 10-month average. Hold bonds when the ratio is above its average, stocks when below. Over a long sample, the rotation tends to produce a return between the two assets' standalone returns but with a lower drawdown than either, because it is usually out of the asset having its worst stretch. Suppose stocks alone annualize 8% with a 50% max drawdown and bonds alone annualize 5% with a 15% max drawdown; the rotation might annualize 8% to 9% with a 25% max drawdown, beating both on risk-adjusted terms in the historical sample. The honest footnote: that result is the in-sample best case, and the rotation's edge depends entirely on the two assets continuing to take turns. In a regime where both fall together (the negative-diversification crash), the rotation is in one of two losers and the relative signal does nothing for you. The article ["How to Evaluate a Strategy Beyond Net Profit"](https://aligrithm.com/how-to-evaluate-a-strategy-beyond-net-profit/) applies: a rotation that looks great on risk-adjusted return in calm decades can still hand you both losers at once in the regime that matters. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## The traps that blow up ratio systems The ratio has failure modes that outright price systems do not, and they are the reason most naive ratio strategies disappoint. The denominator can approach zero or change scale. If B falls hard, the ratio A/B explodes upward for reasons that have nothing to do with A outperforming, purely an artifact of dividing by a small number. Ratios of series that can approach zero or that live on very different scales produce unstable, spiky signals. Working with log ratios (the log of A minus the log of B) tames this, because logs turn the ratio into a difference and treat proportional moves symmetrically. Both legs can move together and kill the rotation premium. The rotation switch earns its keep only when the two assets are imperfectly correlated and take turns. When the correlation spikes toward 1 in a crisis, the ratio flattens and the switch adds nothing while you sit in a falling asset. The diversification a ratio promises is exactly the diversification that fails when you need it, the recurring warning of this pillar. The signal whipsaws near the average. A ratio chopping around its moving average flips the rotation repeatedly, each flip a round-trip cost, and the article ["Why Transaction Costs Should Be Added Before You Fall in Love"](https://aligrithm.com/why-transaction-costs-should-be-added-before-you-fall-in-love/) applies with force because a rotation system trades both legs. A band around the average, or a longer smoothing, cuts the whipsaw at the cost of slower switches. It is easy to data-mine. With a universe of markets you can search thousands of ratio pairs and moving-average lengths and find one with a gorgeous backtest by luck, the multiple-comparisons trap. Restrict ratio pairs to economically linked markets (bonds and stocks, gold and the dollar, two related currencies) before testing, and count every pair you tried, the discipline from ["Optimization Comes After Testing, Not Before"](https://aligrithm.com/optimization-comes-after-testing-not-before/). ## Visualizing the ratio signal ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_104-visual-0.png) ## KEY POINTS - A ratio divides one market's price by another, netting out the move both share and keeping only relative performance. The ratio moves only when one asset outperforms the other, so it isolates rotation and ignores the pair's overall direction. - Signal off the ratio the same way you signal off a price: above its moving average, favor the numerator asset; below, favor the denominator asset. It is a moving-average trend rule applied to relative performance. - A gate filters ("trade A only when B is in some state"); a ratio rotates ("hold whichever of A and B is winning"). They are different tools built from the same parts. - Two uses: the rotation switch holds the outperformer and switches on the crossing (long-only, smoother than either asset); the spread trade goes long the winner and short the loser for a market-neutral position with no directional cushion. - A ratio oscillator (ratio minus its average, normalized) centers on zero, showing direction by sign and overextension by distance, which separates a fresh rotation from an exhausted one. - Worked rotation: a bond/stock switch on a 10-month ratio average tends to land between the two assets' returns with a lower drawdown than either, because it sits out the asset having its worst stretch. That is an in-sample best case that fails when both fall together. - Trap one: a denominator approaching zero or living on a different scale spikes the ratio. Use log ratios (log A minus log B) to tame it. - Trap two: when correlation spikes to 1 in a crisis the rotation premium vanishes. Trap three: chop near the average whipsaws both legs at double cost. Trap four: ratio pairs are easy to data-mine, so restrict to economically linked pairs and count every pair tested. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Trading Systems and Methods - Perry Kaufman (Amazon)](https://www.amazon.com/s?k=Trading+Systems+and+Methods+Perry+Kaufman&ref=aligrithm.com) - [Cybernetic Trading Strategies - Murray Ruggiero (Amazon)](https://www.amazon.com/s?k=Cybernetic+Trading+Strategies+Murray+Ruggiero&ref=aligrithm.com) - [The Art of Currency Trading - Brent Donnelly (Amazon)](https://www.amazon.com/s?k=The+Art+of+Currency+Trading+Brent+Donnelly&ref=aligrithm.com) ### 4.18 Crude Oil, Inflation, and FX URL: https://aligrithm.com/crude-oil-inflation-and-fx/ Last updated: 2026-07-27T13:24:21.000Z Crude oil is the price of energy, and energy is an input cost to almost everything that gets produced, shipped, or heated. When crude rises, transport costs, manufacturing costs, and headline inflation rise behind it within a few months. That makes crude a leading inflation gauge, which links it to two markets at once: bonds, because inflation erodes a fixed coupon, and the dollar, because the dollar is the currency oil is priced and settled in. Crude runs inverse to both bond prices and the dollar over most samples, and those two inverse links are the spine of every oil-based intermarket system. This article handles the inflation side of the commodity complex, the complement to ["Copper as an Economic Activity Indicator"](https://aligrithm.com/copper-as-an-economic-activity-indicator/), which handled the growth side. Together they split commodities into a growth signal (copper) and an inflation signal (crude), and this article also extends the map into currencies, feeding the FX articles later in the pillar including ["Why FX Traders Must Watch Gold, Rates, and Equities"](https://aligrithm.com/why-fx-traders-must-watch-gold-rates-and-equities/). ## Crude as a leading inflation gauge The transmission from crude to inflation is mechanical and lagged. A jump in crude raises the cost of fuel, freight, plastics, fertilizer, and heating, and those costs pass through to consumer prices over the following months. The official inflation print is therefore a delayed echo of where crude went earlier, which means crude leads the inflation data that bond markets and central banks react to. That lead is the whole reason crude is useful in a system. By the time the inflation print confirms a rise, bonds have often already moved. Crude gives a forward read on the inflation impulse, the same forward-looking property that made copper valuable, but pointed at inflation rather than activity. A trader who waits for the inflation release is reacting to old news; a trader watching crude is reading the input that release will eventually report. ## The two inverse links Crude against bonds. Rising crude feeds inflation, inflation erodes the real value of a fixed coupon, so bond yields rise and bond prices fall to compensate. Crude up, inflation up, yields up, bonds down. The chain is the inflation-side mirror of copper's activity chain, and both end at the same place (yields up, bonds down) through different mechanisms. Crude against the dollar. Oil is priced and settled in dollars worldwide. A stronger dollar means each dollar buys more oil, pushing the dollar price of crude down; a weaker dollar pushes it up. The two trade inverse. The relationship also runs the other way through trade balances and petrodollar flows, but the priced-in-dollars channel is the dominant, most reliable one. $$ \\text{Crude} \\;\\uparrow \\;\\Rightarrow\\; \\text{inflation} \\;\\uparrow \\;\\Rightarrow\\; \\text{bond price} \\;\\downarrow, \\qquad \\text{Dollar} \\;\\uparrow \\;\\Rightarrow\\; \\text{Crude (in USD)} \\;\\downarrow $$ The first chain reads crude up, inflation up, bond price down; the second reads dollar up, crude down. The oil-stock index (a basket of oil producer equities) tends to turn with or slightly ahead of crude itself, so it serves as a confirming or leading read on the crude move, the lead-lag idea applied within the energy complex. ## Crude as an FX signal The inflation and dollar links make crude a usable input for currency systems, especially for commodity-exporting and commodity-importing currencies. Commodity-exporter currencies. The currencies of major oil exporters tend to strengthen when crude rises, because higher oil revenue improves the country's trade balance and pulls capital in. Rising crude is a tailwind for an exporter currency and a headwind for a heavy importer currency that has to pay more for the same barrels. A crude trend can gate long signals on an exporter currency and short signals on an importer currency, template 1 and template 2 applied to FX. Inflation-and-rates channel. Rising crude raises inflation, which can force a central bank to tighten, which lifts that country's currency through the rate differential, the carry mechanism from ["Gold, Dollar, and Rates: A Practical Intermarket Map"](https://aligrithm.com/gold-dollar-and-rates-a-practical-intermarket-map/). Crude therefore feeds the FX picture twice: directly through the trade balance and indirectly through the inflation-to-rates-to-currency chain. The two channels usually reinforce for exporters and can conflict for importers, which is exactly the kind of cross-current the divergence test in ["Intermarket Divergence as a Trading Filter"](https://aligrithm.com/intermarket-divergence-as-a-trading-filter/) is built to flag. A worked sketch: gating a long-trade system on an oil-exporter currency to fire only when crude is above its moving average aligns the currency position with its dominant fundamental driver, raising the per-trade quality the same way the bond filter did for equities. The exporter currency fights an uphill battle when crude is falling, and the filter keeps you out of that fight. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## The traps with crude Where crude misleads. Supply shocks dominate the inflation read. Crude spikes driven by war, OPEC supply cuts, or pipeline disruptions raise the price without any underlying demand strength, and they raise inflation while activity is weakening, the "stagflation" combination. A crude-as-inflation filter behaves differently in a demand-driven oil rally than in a supply-shock rally, and treating the two as identical is a mistake. Cross-checking crude against copper helps: copper and crude rising together points to demand-driven reflation, while crude rising as copper falls points to a supply shock, the cross-asset confirmation logic again. The dollar link weakens for big oil moves. When crude moves on a large supply shock, it can swamp the dollar relationship entirely, moving far more than the dollar can explain. The inverse link is a normal-regime tendency, not a tight arbitrage. The FX links are country-specific and drift. Which currencies are "oil currencies" depends on each country's export mix, and that mix changes over years as economies diversify or as production shifts. An FX-crude relationship estimated a decade ago may not hold today, the slow-drift problem that applies to every edge in the web. Re-estimate the currency's sensitivity to crude on a rolling window rather than treating it as a fixed label. ## Visualizing the crude links ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_103-visual-0.png) ## KEY POINTS - Crude oil is an input cost to almost everything produced, shipped, or heated, so a rise in crude feeds headline inflation within a few months. Crude is a leading inflation gauge. - The lead matters: the official inflation print is a delayed echo of where crude already went, so watching crude reads the inflation impulse before the data confirms it and before bonds finish reacting. - Two inverse links form the spine. Crude up feeds inflation, lifts yields, and lowers bond prices; a stronger dollar lowers the dollar price of crude because oil is priced in dollars. The oil-stock index turns with or slightly ahead of crude. - Crude's activity-side mirror is copper. Both chains end at yields up and bonds down, but copper runs through real activity and crude runs through inflation. - Crude is an FX signal through two channels: directly via the trade balance (rising crude helps exporter currencies, hurts importer currencies) and indirectly via inflation forcing rate hikes that lift a currency through carry. The channels reinforce for exporters and can conflict for importers. - Gating a long-trade system on an exporter currency to fire only when crude is above its average aligns the position with its dominant driver and keeps you out of the uphill fight when crude is falling. - Trap one: supply-shock rallies (war, OPEC cuts) raise inflation while activity weakens (stagflation), behaving differently from demand-driven rallies. Cross-check against copper to tell them apart. - Trap two: large oil moves swamp the dollar link, and the FX relationships are country-specific and drift as export mixes change. Re-estimate a currency's crude sensitivity on a rolling window rather than treating it as a fixed label. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Trading Systems and Methods - Perry Kaufman (Amazon)](https://www.amazon.com/s?k=Trading+Systems+and+Methods+Perry+Kaufman&ref=aligrithm.com) - [Cybernetic Trading Strategies - Murray Ruggiero (Amazon)](https://www.amazon.com/s?k=Cybernetic+Trading+Strategies+Murray+Ruggiero&ref=aligrithm.com) - [The Art of Currency Trading - Brent Donnelly (Amazon)](https://www.amazon.com/s?k=The+Art+of+Currency+Trading+Brent+Donnelly&ref=aligrithm.com) ### 4.17 Copper as an Economic Activity Indicator URL: https://aligrithm.com/copper-as-an-economic-activity-indicator/ Last updated: 2026-07-27T13:24:21.000Z Copper goes into wiring, plumbing, motors, and construction, so the quantity of copper the world wants to buy this quarter is a direct read on how much building and manufacturing is happening. That physical fact gives copper a job no purely financial series can do: it measures real economic activity in close to real time, before the official statistics confirm it. Traders nicknamed it "Doctor Copper" for the joke that it holds an economics PhD, and the serious version of the joke is that copper's price leads the data that central banks and bond markets eventually react to. For an intermarket system, copper is the cleanest single proxy for the growth side of the macro picture. This article makes the case that copper beats the broad commodity basket as an activity gauge, then turns copper into filters on bonds and growth-sensitive assets using the templates from ["Intermarket Analysis for System Traders"](https://aligrithm.com/intermarket-analysis-for-system-traders/). It pairs with ["Crude Oil, Inflation, and FX"](https://aligrithm.com/crude-oil-inflation-and-fx/), which handles the inflation side of the commodity complex, so together they split the commodity world into its growth signal and its inflation signal. ## Why copper beats the broad commodity index The instinct is to use a broad commodity basket as the activity gauge, because a basket is diversified and stable. For measuring activity, the diversification works against you. A broad commodity index mixes things driven by demand (industrial metals) with things driven by supply shocks and weather (grains, livestock, natural gas) and things driven by monetary fear (gold). The index average blends a clean growth signal with a pile of unrelated noise, so it tracks activity loosely. Copper is almost pure industrial demand. It has no significant safe-haven bid, no weather-driven supply spikes on the scale of agriculture, and its consumption maps directly to construction and manufacturing. The narrower instrument carries the cleaner signal, the same reason the article ["Why One Indicator Should Not Be Used on Every Market"](https://aligrithm.com/why-one-indicator-should-not-be-used-on-every-market/) argued a sharp single-purpose tool beats a blunt general one. Lumber sits in the same category for the same reason: it is a direct read on housing and construction, and historically it has led activity and bond yields. The general principle is to pick the commodity whose demand is closest to the economic variable you want to measure, rather than averaging everything and hoping the signal survives the blend. ## The copper-bond link Copper's most useful intermarket relationship is its inverse link to bond prices, which is really a positive link to yields. Rising copper means rising industrial demand, which means a strengthening economy, which means rising inflation expectations and a central bank more likely to hold or raise rates. All of that pushes bond yields up, so bond prices down. Copper up, yields up, bonds down. The chain runs through real activity and inflation expectations, and copper sits at the front of it because the physical demand registers in the price before the inflation prints arrive. $$ \\text{Copper} \\;\\uparrow \\;\\Rightarrow\\; \\text{activity} \\;\\uparrow \\;\\Rightarrow\\; \\text{inflation expectations} \\;\\uparrow \\;\\Rightarrow\\; \\text{yields} \\;\\uparrow \\;\\Rightarrow\\; \\text{bond price} \\;\\downarrow $$ The chain reads left to right: rising copper signals rising activity, which lifts inflation expectations, which lifts yields, which lowers bond prices. Because copper sits at the front of the chain and bonds at the back, copper tends to lead bonds at turning points, which is what makes it a forward-looking filter rather than a coincident one, the lead-lag property the article ["Lead-Lag Relationships in Global Markets"](https://aligrithm.com/lead-lag-relationships-in-global-markets/) treats as its own measurement. ## Copper as a filter Two filters fall out of the chain, both using the simple state templates. Bond/rate filter. Suppress long-bond signals when copper is in a strong uptrend (copper above its moving average and rising), because rising copper is a forward warning of rising yields and falling bond prices. Allow long-bond signals when copper is weak or falling, because slowing activity points to falling yields and rising bond prices. Copper gates the bond system from outside the bond market. Growth-asset filter. Use copper as a risk-on/risk-off gate for growth-sensitive positions. When copper is trending up, the growth backdrop is supportive, so enable cyclical and equity-trend signals. When copper rolls over, the growth backdrop is deteriorating, so reduce or suppress them. This is the same logic as the bond filter pointed at the equity side, and it complements the bond-equity gate from ["Using Bonds to Filter Equity Signals"](https://aligrithm.com/using-bonds-to-filter-equity-signals/) by adding an activity reading the bond price alone does not give. A worked sketch on a long-bond trend system: gating out long-bond entries during strong copper uptrends removes a cluster of trades that fight a rising-yield backdrop, lifting the per-trade quality of the remaining long-bond signals. The filter does not change the bond system's logic; it removes the macro condition under which that logic is structurally disadvantaged. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## The traps with copper Where the "Doctor Copper" story misleads if you take it too literally. Supply shocks override demand. Copper is mined in a handful of countries, and a major mine strike, a political disruption in a key producer, or an export ban can spike the price for reasons that have nothing to do with global demand. During such a spike, copper is signaling supply scarcity, not booming activity, and reading it as an activity surge is wrong. The fix is to cross-check copper against a second activity proxy (lumber, an industrial-production nowcast) and distrust copper moves that no other activity series confirms, the cross-check discipline the article ["Why Cross-Asset Signals Beat Isolated Chart Reading"](https://aligrithm.com/why-cross-asset-signals-beat-isolated-chart-reading/) built in. Financialization adds noise. Copper is now a heavily traded financial asset, not just an industrial input, so speculative flows and dollar moves push it around on horizons shorter than the activity signal it is supposed to measure. The activity signal lives at the weekly-to-monthly scale; the daily wiggle is mostly financial noise. Reading copper on too short a timeframe is reading the noise, the timeframe-matching point from ["How to Choose the Right Timeframe for a Strategy"](https://aligrithm.com/how-to-choose-the-right-timeframe-for-a-strategy/). The link to yields is not constant. Like every intermarket relationship in this pillar, the copper-yield correlation drifts and can weaken when monetary policy is driven by factors other than activity. Re-estimate the relationship on a rolling window rather than assuming it holds, the slow-drift discipline that applies to every edge in the web. ## Visualizing copper as the lead ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_102-visual-0.png) ## KEY POINTS - Copper goes into wiring, plumbing, motors, and construction, so demand for it is a near real-time read on real economic activity, ahead of the official statistics. That earns it the "Doctor Copper" nickname. - Copper beats a broad commodity basket as an activity gauge because the basket blends demand-driven metals with weather-driven grains, supply-driven energy, and fear-driven gold. The narrow instrument carries the cleaner signal. Lumber works for the same reason on housing. - The core link is copper up, activity up, inflation expectations up, yields up, bond price down. Copper sits at the front of that chain and bonds at the back, so copper tends to lead bonds at turning points. - Bond filter: suppress long-bond signals during strong copper uptrends (a forward warning of rising yields) and allow them when copper is weak. Copper gates the bond system from outside the bond market. - Growth-asset filter: use copper as a risk-on/risk-off gate for cyclical and equity-trend signals, enabling them when copper trends up and reducing them when it rolls over. This adds an activity reading the bond price alone does not provide. - Trap one: supply shocks (mine strikes, export bans) spike copper for reasons unrelated to demand. Cross-check against a second activity proxy and distrust copper moves nothing else confirms. - Trap two: copper is financialized, so daily moves are mostly speculative and dollar-driven noise. The activity signal lives at the weekly-to-monthly scale; reading copper too short captures the noise. - Trap three: the copper-yield correlation drifts and weakens when policy is driven by non-activity factors. Re-estimate on a rolling window rather than assuming it holds. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Trading Systems and Methods - Perry Kaufman (Amazon)](https://www.amazon.com/s?k=Trading+Systems+and+Methods+Perry+Kaufman&ref=aligrithm.com) - [Cybernetic Trading Strategies - Murray Ruggiero (Amazon)](https://www.amazon.com/s?k=Cybernetic+Trading+Strategies+Murray+Ruggiero&ref=aligrithm.com) - [The Art of Currency Trading - Brent Donnelly (Amazon)](https://www.amazon.com/s?k=The+Art+of+Currency+Trading+Brent+Donnelly&ref=aligrithm.com) ### 4.16 Gold, Dollar, and Rates: A Practical Intermarket Map URL: https://aligrithm.com/gold-dollar-and-rates-a-practical-intermarket-map/ Last updated: 2026-07-27T13:24:20.000Z Three instruments sit at the center of every macro intermarket system: gold, the US dollar, and the interest-rate complex. They form a tight triangle. Gold runs inverse to real interest rates, because gold pays no coupon and loses its appeal when bonds and cash pay more. The dollar runs inverse to gold, because gold is priced in dollars and a stronger dollar buys more of it. Rates drive the dollar, because higher US rates pull capital toward dollar assets and lift the currency. Pin down two corners of that triangle and the third is largely determined, which is what makes it a practical map rather than a list of trivia. This article draws the gold-dollar-rates triangle, gives each edge its mechanism and its sign, and turns the map into concrete filters using the rule templates from ["Intermarket Analysis for System Traders"](https://aligrithm.com/intermarket-analysis-for-system-traders/). It feeds directly into the FX-focused articles later in the pillar, ["Why FX Traders Must Watch Gold, Rates, and Equities"](https://aligrithm.com/why-fx-traders-must-watch-gold-rates-and-equities/) and ["Using Gold as an FX Indicator"](https://aligrithm.com/using-gold-as-an-fx-indicator/), which apply this triangle to currency trading. ## The three edges Edge one, gold against rates. Gold is a zero-yield asset. When real yields rise, holding gold means giving up the yield you could earn in bonds or cash, so gold weakens. When real yields fall (or go negative), the opportunity cost of holding gold drops and gold strengthens. The link is to real yields (nominal yield minus inflation), not nominal yields, which matters because gold can rise alongside rising nominal rates if inflation is rising faster. This is the subtlety that trips up traders who watch the nominal ten-year and wonder why gold is not cooperating. Edge two, dollar against gold. Gold is quoted in dollars per ounce. Hold the global demand for gold roughly constant and a stronger dollar mechanically lowers the dollar price of gold, because each dollar is worth more gold. The two trade inverse over most samples. The inverse is not mechanical-only; the dollar and gold are also competing safe-haven and store-of-value assets, so capital rotating between them reinforces the negative correlation. Edge three, rates against the dollar. Higher US short rates, relative to the rest of the world, make dollar deposits more attractive and pull capital into the dollar, lifting it. This is the carry mechanism: money flows toward the higher risk-adjusted yield. Falling US rates do the reverse, weakening the dollar and, by edge two, supporting gold. The triangle closes: rates up, dollar up, gold down; rates down, dollar down, gold up. $$ \\text{Gold} \\;\\downarrow\\; \\text{as real rates} \\;\\uparrow, \\qquad \\text{Dollar} \\;\\uparrow\\; \\text{as US rates} \\;\\uparrow, \\qquad \\text{Gold} \\;\\downarrow\\; \\text{as Dollar} \\;\\uparrow $$ The three relations are consistent: a rise in real US rates lifts the dollar and weighs on gold through two reinforcing channels (the opportunity-cost channel directly on gold and the dollar-strength channel on gold's price). When two of the three corners agree, the third is confirmed; when they disagree, one of the edges is in a regime shift and the map is warning you, the divergence logic the article ["Intermarket Divergence as a Trading Filter"](https://aligrithm.com/intermarket-divergence-as-a-trading-filter/) formalizes. ## Turning the triangle into filters Each edge is a ready-made gate, using template 1 and template 2 from the intermarket overview. Gold long, gated by the dollar. Allow long-gold signals only when the dollar index is below its moving average (template 2, inverse pair). A falling dollar is the tailwind; a rising dollar fights every long-gold trade. The same gate inverted (short gold only when the dollar is above its average) handles the other side. Gold long, gated by rates. Allow long-gold signals only when the rate proxy is falling (bond price above its average, since bond price up means yields down). Rising real yields are gold's structural headwind, so suppressing long-gold trades into rising yields removes a predictable losing condition. Dollar long, gated by rates. Allow long-dollar signals only when the US short-rate trend is up relative to a foreign benchmark. The dollar is a carry vehicle, and trading it with the rate differential rather than against it aligns the position with the dominant flow. The most robust setups require two corners to agree. A long-gold trade taken only when the dollar is below its average and bonds are above theirs (yields falling) is gated by two reinforcing edges, which cuts the trade count and lifts the per-trade quality. Fewer trades, cleaner distribution, the same trade-off the article ["Using Bonds to Filter Equity Signals"](https://aligrithm.com/using-bonds-to-filter-equity-signals/) described for a single gate, doubled. ## Worked example: gold conditioned on two corners Numbers, because the map is useless without them. Take a long-gold trend system and three states. Unconditioned long-gold trades: suppose a hit rate of 48% and a small positive average trade. Gated on the dollar alone (dollar below its 50-day average): hit rate near 54%, larger average trade, roughly 60% of the original trade count retained. Gated on both corners (dollar below its average and bonds above theirs): hit rate near 58%, the largest average trade, but only about 35% of the original trade count retained. The double gate produces the best per-trade quality and the fewest trades. Whether that is the right setting depends on whether you have enough trades left for the result to be reliable, the sample-size constraint the article ["Trade-Count Thresholds for Backtest Reliability"](https://aligrithm.com/trade-count-thresholds-for-backtest-reliability/) insisted on. Gate too hard and you end up with twenty trades and a beautiful, meaningless hit rate. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Where the triangle breaks The honest caveats, because this triangle is more stable than the stock-bond link but still drifts. The real-rate subtlety dominates. Gold tracks real yields, and real yields require an inflation estimate that is itself noisy and lagged. In a sudden inflation spike, nominal yields can rise while real yields fall, and gold rallies against what a nominal-rate watcher expects. A system that gates gold on nominal bond prices will misfire in exactly those high-inflation regimes where gold matters most. Using a real-yield proxy (or a breakeven-inflation series) fixes this, at the cost of more data plumbing. The dollar is not one thing. "The dollar" usually means a trade-weighted index dominated by the euro. Gold's inverse relationship to the dollar index can weaken when the euro is moving for idiosyncratic European reasons unrelated to gold. The article ["FX Is Not One Market: Retail vs Wholesale Structure"](https://aligrithm.com/fx-is-not-one-market-retail-vs-wholesale-structure/) makes the broader point that a currency index hides as much as it shows. The safe-haven regime scrambles signs. In a genuine crisis, gold and the dollar can both rally as everything else is sold, breaking edge two for the duration. The triangle is a normal-regime map, and like every relationship in this pillar it gets switched off when the regime changes, the acceptance built into ["Why Systems Work Until They Don't"](https://aligrithm.com/why-systems-work-until-they-dont/). ## Visualizing the triangle ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_101-visual-0.png) ## KEY POINTS - Gold, the dollar, and rates form a tight triangle. Gold runs inverse to real yields (it pays no coupon), the dollar runs inverse to gold (gold is priced in dollars), and rates drive the dollar (capital chases yield). - The triangle is internally consistent: rates up, dollar up, gold down, and the reverse. Pin down two corners and the third is largely determined; when two corners disagree, one edge is shifting and the map is warning you. - Each edge is a filter. Gate long-gold on a falling dollar, gate long-gold on falling yields (bonds above their average), and gate long-dollar on a rising rate differential. The most robust setups require two corners to agree. - Worked example: an unconditioned long-gold system at 48% hit rate rises to 54% gated on the dollar alone and 58% gated on both the dollar and bonds, but the double gate retains only about a third of the trades. - The double gate trades quantity for quality. Gate too hard and you are left with too few trades for the result to be reliable; balance per-trade quality against sample size. - Gold tracks real yields, not nominal. In an inflation spike, nominal yields can rise while real yields fall and gold rallies against a nominal-rate watcher's expectation. Use a real-yield or breakeven proxy. - The dollar index is dominated by the euro and can move for European reasons unrelated to gold, weakening edge two. A currency index hides as much as it shows. - In a genuine crisis gold and the dollar can both rally, breaking the inverse edge. The triangle is a normal-regime map and gets switched off when the regime changes. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Trading Systems and Methods - Perry Kaufman (Amazon)](https://www.amazon.com/s?k=Trading+Systems+and+Methods+Perry+Kaufman&ref=aligrithm.com) - [Cybernetic Trading Strategies - Murray Ruggiero (Amazon)](https://www.amazon.com/s?k=Cybernetic+Trading+Strategies+Murray+Ruggiero&ref=aligrithm.com) - [The Art of Currency Trading - Brent Donnelly (Amazon)](https://www.amazon.com/s?k=The+Art+of+Currency+Trading+Brent+Donnelly&ref=aligrithm.com) ### 4.15 Using Bonds to Filter Equity Signals URL: https://aligrithm.com/using-bonds-to-filter-equity-signals/ Last updated: 2026-07-27T13:24:20.000Z The single most studied intermarket link is the one between long bonds and stocks, and it produces a filter you can write in one line: take long equity signals only when T-bonds are above their moving average. The mechanism is the discount rate. Both stocks and bonds are valued by discounting future cash flows at the long interest rate, so when the bond market rallies (yields falling), the discount rate on equities drops too, and the present value of every future earnings stream rises. When bonds sell off (yields rising), the same discounting works against stocks. Bonds and stocks are two prices set by one rate, which is why the bond trend is a usable gate on the equity trend. This article takes the bond-equity pair, the most reliable link in the intermarket web from ["Intermarket Analysis for System Traders"](https://aligrithm.com/intermarket-analysis-for-system-traders/), and builds the filter step by step with the distribution split from ["Why Cross-Asset Signals Beat Isolated Chart Reading"](https://aligrithm.com/why-cross-asset-signals-beat-isolated-chart-reading/). It also flags the regime where the whole thing inverts, because the stock-bond correlation is the textbook case of a relationship that changes sign across decades. ## The filter, stated precisely The rule has three parts: the external market, the state test, and the action. External market: the long-bond future (or a bond price proxy). State test: is the bond close above its moving average of length n. Action: when the bond is above its average, allow the equity system's long signals; when the bond is below its average, suppress them (sit in cash) or reduce size. $$ \\text{Position}\_{\\text{equity}} = \\begin{cases} \\text{equity signal} & \\text{if } B\_t > \\text{MA}\_n(B)\\\\\[4pt\] 0 & \\text{if } B\_t \\le \\text{MA}\_n(B) \\end{cases} $$ The equity position equals whatever the underlying equity system says, but only when the bond price B at time t sits above its n-period moving average. Below the average, the position is flattened. The moving-average length n is the one free parameter, and it should be chosen for stability, not for the value that maximizes backtest profit, the trap the article ["Trading Systems Are Recipes, Not Predictions"](https://aligrithm.com/trading-systems-are-recipes-not-predictions/) called curve-fit stops. A length anywhere from 30 to 80 days that gives similar results is a sign the filter is robust; a length where 50 wins and 45 and 55 both fail is a sign you have fit noise. ## Subdividing the days The clean way to see the filter work is to subdivide the equity sample by the bond state and compare the two sub-distributions to the whole, exactly the procedure that justifies any intermarket gate. Take the full set of equity trading days and their forward returns. Split into "bond up" days (bond above its average) and "bond down" days. Across long historical samples, the bond-up subset carries a materially higher mean forward return and a lower standard deviation than the full set, while the bond-down subset carries a near-zero or negative mean and a higher standard deviation. The two halves are different markets wearing the same ticker. A representative split on a few thousand equity days: the full sample annualizes to roughly 7% with a given drawdown profile; the bond-up subset annualizes closer to 11% with shallower drawdowns; the bond-down subset annualizes near 1% to 2% with deeper, faster drawdowns. The bond-down days are where most of the equity market's pain concentrates, because rising yields compress valuations and rising yields are exactly what "bond below its average" detects. The practical read: the filter does not improve the average day by a lot. It removes a slice of bad days that are disproportionately responsible for drawdowns, which improves the risk-adjusted return more than the raw return. That is the right way to judge it, the lesson from ["Why Profit Factor Can Lie"](https://aligrithm.com/why-profit-factor-can-lie/) about looking at the shape of the distribution rather than one summary number. ## What the filter costs you No filter is free, and this one has two clear costs. It sits out good days. Equities sometimes rally hard while bonds are below their average (a "good growth, rising rates" regime), and the filter has you in cash for those. You give up real upside to avoid the bad days. Whether that trade is worth it depends on whether your objective is return or risk-adjusted return, and for most leveraged or drawdown-sensitive books it is the latter. It adds turnover. The bond crossing its average flips the filter on and off, generating entries and exits that the unfiltered equity system would not make. Each flip is a round-trip cost. If the bond oscillates around its average for a stretch, the filter whipsaws, and the article ["Why Transaction Costs Should Be Added Before You Fall in Love"](https://aligrithm.com/why-transaction-costs-should-be-added-before-you-fall-in-love/) applies directly: a filter that improves gross returns can destroy net returns through whipsaw if the cross-asset series is choppy near the threshold. Smoothing the bond series or adding a band around the average reduces this, at the cost of slower reaction. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## The regime that inverts the sign The dangerous part, and the reason a hard-coded bond-equity filter is not safe to set and forget. The stock-bond correlation is not stable across decades. Through long inflationary stretches, stocks and bonds have moved together (both hurt by rising yields), and the filter as written works. Through other stretches, the correlation goes negative: bonds rally as a safe haven precisely when stocks crash, so "bonds above their average" coincides with equity stress rather than equity strength, and the filter's sign is backward. A filter built on the positive-correlation assumption and run through a negative-correlation regime will gate exactly wrong. The defense is to make the filter conditional on the measured correlation, not to assume it. Estimate the rolling correlation between stock and bond returns over a trailing window. When it is reliably positive, apply the filter as written. When it is negative or unstable, disable the filter or flip its sign. This turns a brittle hard-coded rule into a regime-aware one, the slow-drift discipline the article ["Slow Wandering: The Most Dangerous Type of Market Change"](https://aligrithm.com/slow-wandering-the-most-dangerous-type-of-market-change/) argued every structural relationship needs. A trader who skips this step is not running an intermarket system; he is betting that one historical correlation regime lasts forever. ## Visualizing the filtered equity curve ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_100-visual-0.png) ## KEY POINTS - Stocks and long bonds are two prices set by one rate. Falling yields (bonds rallying) lift equity valuations through a lower discount rate; rising yields compress them. That shared mechanism makes the bond trend a usable gate on the equity trend. - The filter in one line: take long equity signals only when bonds are above their moving average; flatten or reduce size when bonds are below. The moving-average length is the only free parameter and should be chosen for stability, not maximum backtest profit. - Subdividing equity days by bond state splits one market into two. The bond-up subset shows higher mean forward return and lower dispersion; the bond-down subset concentrates the drawdowns, because rising yields and falling equity valuations are the same event. - The filter's main value is risk-adjusted, not raw. It removes a slice of disproportionately bad days, improving drawdown more than average return. Judge it by the shape of the distribution, not one summary number. - It costs you the good days that occur while bonds are below their average (strong growth, rising rates), and it adds turnover. A bond series chopping around its average whipsaws the filter and can destroy net returns through round-trip costs. - The correlation inverts across regimes. In long inflationary stretches stocks and bonds fall together and the filter works; in safe-haven regimes bonds rally as stocks crash and the filter gates backward. - The fix is to condition the filter on the rolling stock-bond correlation: apply it when the correlation is reliably positive, disable or flip it when negative or unstable. A hard-coded filter bets one correlation regime lasts forever. - Smooth the bond series or add a band around the average to cut whipsaw, accepting slower reaction in exchange for fewer false flips. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Trading Systems and Methods - Perry Kaufman (Amazon)](https://www.amazon.com/s?k=Trading+Systems+and+Methods+Perry+Kaufman&ref=aligrithm.com) - [Cybernetic Trading Strategies - Murray Ruggiero (Amazon)](https://www.amazon.com/s?k=Cybernetic+Trading+Strategies+Murray+Ruggiero&ref=aligrithm.com) - [The Art of Currency Trading - Brent Donnelly (Amazon)](https://www.amazon.com/s?k=The+Art+of+Currency+Trading+Brent+Donnelly&ref=aligrithm.com) ### 4.14 Why Cross-Asset Signals Beat Isolated Chart Reading URL: https://aligrithm.com/why-cross-asset-signals-beat-isolated-chart-reading/ Last updated: 2026-07-27T13:24:59.000Z Run a simple experiment on twenty years of S&P 500 daily data. Compute every day's forward five-day return, then split those days into two buckets by a condition that uses no S&P price at all: was the T-bond closing above or below its own 50-day average that day. The full sample might show an average forward return of 0.05% over five days with a standard deviation of 2.0%. Split it, and the "bonds above their average" bucket shows 0.12% with lower dispersion, while the "bonds below" bucket shows 0.00% or slightly negative with higher dispersion. The chart of the S&P never changed. A variable from a different market sorted the same days into a better-expected set and a worse-expected set. That sort is the entire argument for cross-asset signals. Isolated chart reading uses one information source: the price history of the instrument you trade. Every moving average, breakout level, and oscillator is a transformation of that single series. Cross-asset analysis adds a second, structurally independent source, and the test of whether it helps is mechanical: does conditioning on the external market produce sub-samples with different forward-return distributions than the unconditioned whole. When it does, the external market is carrying tradeable information the chart cannot. This article makes the case with the distribution-splitting logic, then catalogs when the second source helps and when it is noise dressed as insight. It is the "why" behind the four rule templates from "Intermarket Analysis for System Traders", and it sets up the specific bond-equity filter worked in "Using Bonds to Filter Equity Signals". ## The distribution-splitting test A trading signal is worth something only if it changes the distribution of forward returns relative to the base rate. That standard, borrowed from the way the article ["How to Evaluate a Strategy Beyond Net Profit"](https://aligrithm.com/how-to-evaluate-a-strategy-beyond-net-profit/) insisted on looking past a single profit number, is the cleanest way to judge a cross-asset filter. Start with the unconditioned base rate. For a chosen instrument and horizon, compute the mean, the standard deviation, and the hit rate of forward returns over the whole sample. This is the null: what you get by trading with no filter at all. Then introduce the external condition and recompute the same statistics on each sub-sample. $$ \\text{Lift} = E\[r\_{t \\to t+h} \\mid X\_t \\in \\text{state}\] - E\[r\_{t \\to t+h}\] $$ The lift is the conditional expected forward return given the external market X is in some state, minus the unconditional expected forward return. A positive lift in the "enabled" state and a negative or zero lift in the "disabled" state means the external market is sorting the days into useful buckets. A lift of zero in both states means the external market is decorative, and you have spent degrees of freedom for nothing. The discipline this enforces: you never accept a cross-asset signal because the story is good. You accept it because the conditional distributions separate, and you reject it when they collapse back onto the base rate. The article ["Induction in Trading: Why Past Patterns Are Always Uncertain"](https://aligrithm.com/induction-in-trading-why-past-patterns-are-always-uncertain/) applies here in full, because a separation measured on one sample is an estimate, not a guarantee. ## Why two sources beat one, structurally The mathematical reason cross-asset signals can help is that two markets are not perfectly correlated, so the second carries information the first does not contain. If bonds and stocks were correlated 1.0, the bond filter would be a relabeled stock filter and add nothing. They are correlated somewhere around 0.3 to 0.6 over medium horizons, which leaves a large independent component, and that independent component is where the off-chart information lives. Think of it as two noisy measurements of the same underlying macro state (the level and direction of discount rates, the inflation impulse, the risk appetite of capital). The S&P price is one noisy readout of that state. The bond price is a second noisy readout with different noise. Averaging or cross-conditioning two imperfect readouts of the same latent driver gives a sharper estimate of the driver than either alone, the same logic that makes an ensemble of weak models beat a single one. The independence is also why the leads exist. Short-rate markets and utility stocks have historically turned ahead of the broad equity market at major inflection points, because they sit closer to the discount-rate driver than equities do. A market closer to the cause moves before a market further down the chain, which is a forward-looking filter rather than a coincident one, and the article ["Lead-Lag Relationships in Global Markets"](https://aligrithm.com/lead-lag-relationships-in-global-markets/) treats measuring that lead as its own problem. ## A worked split Concrete numbers, because the abstract claim does not land otherwise. Suppose 5,000 trading days of S&P forward five-day returns. Unconditioned: mean 0.05%, standard deviation 2.0%, share of positive days 53%. Condition on bonds above their 50-day average (roughly 2,800 days): mean 0.13%, standard deviation 1.8%, positive share 57%. Condition on bonds below their 50-day average (roughly 2,200 days): mean minus 0.05%, standard deviation 2.3%, positive share 48%. The lift in the "bonds up" state is 0.13% minus 0.05%, equal to 0.08% per five days, with lower dispersion and a higher hit rate. The "bonds down" state has negative lift and higher dispersion. A long-only S&P system that only takes signals while bonds are above their average is trading the better half of the sample and sitting out the worse half. The improvement is not enormous per trade, and that is the honest texture of intermarket edges: small, persistent shifts in a distribution, not a switch between winning and losing. The annualized arithmetic matters here. An 0.08% improvement per five-day holding period, compounded across the trades you actually take, is the kind of edge that survives only if costs are small relative to it, the constraint the article ["Why Transaction Costs Should Be Added Before You Fall in Love"](https://aligrithm.com/why-transaction-costs-should-be-added-before-you-fall-in-love/) hammered. A cross-asset filter that adds 0.08% and costs 0.10% in extra round-trips is a net loss dressed as an insight. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## When the second source is noise The failure modes, so you do not fool yourself. The correlation flipped. If your filter was estimated in a regime where stocks and bonds moved together and you trade it in a regime where they move opposite, the lift inverts and the filter actively hurts. The fix is a rolling re-estimate of the correlation sign, and a willingness to disable the filter when the sign is unstable, the slow-drift problem again. You tested too many external markets. Try thirty candidate markets and you will find one that splits the distribution beautifully on the sample by luck alone. This is the multiple-comparisons trap, and the only honest defense is to fix a small, economically motivated candidate set in advance and count every market you tried, the discipline the article ["Optimization Comes After Testing, Not Before"](https://aligrithm.com/optimization-comes-after-testing-not-before/) sequenced. The split is real but tiny relative to costs. A statistically detectable lift that is smaller than your transaction costs is a curiosity, not a strategy. Always net the lift against the extra trading the filter induces before believing it. ## Visualizing the split ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_99-visual-0.png) ## KEY POINTS - Isolated chart reading uses one information source: the price history of the traded instrument. Every indicator on the chart is a transformation of that single series. - Cross-asset analysis adds a structurally independent second source. The test of whether it helps is mechanical: does conditioning on the external market produce forward-return sub-samples that differ from the unconditioned base rate. - The measure is lift: the conditional expected forward return in a given external state minus the unconditional expected return. Positive lift in the enabled state and zero or negative in the disabled state means the external market sorts days usefully. - Two markets beat one because they are not perfectly correlated (roughly 0.3 to 0.6 over medium horizons), so the second carries a large independent component. Cross-conditioning two noisy readouts of the same macro driver sharpens the estimate. - The independence also produces leads: markets closer to the discount-rate driver (short rates, utilities) turn before equities, giving a forward-looking filter rather than a coincident one. - Worked split: S&P forward five-day return averages 0.05% unconditioned, 0.13% when bonds are above their average, and minus 0.05% when below. The lift is real, small (0.08% per five days), and only survives if costs are smaller. - Failure modes: the correlation sign flips between regimes, you test too many candidate markets and find a fluke, or the split is real but smaller than transaction costs. - The defense: fix a small economically motivated candidate set in advance, re-estimate the correlation sign on a rolling window, count every market tested, and always net the lift against the extra trading the filter induces. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Trading Systems and Methods - Perry Kaufman (Amazon)](https://www.amazon.com/s?k=Trading+Systems+and+Methods+Perry+Kaufman&ref=aligrithm.com) - [Cybernetic Trading Strategies - Murray Ruggiero (Amazon)](https://www.amazon.com/s?k=Cybernetic+Trading+Strategies+Murray+Ruggiero&ref=aligrithm.com) - [The Art of Currency Trading - Brent Donnelly (Amazon)](https://www.amazon.com/s?k=The+Art+of+Currency+Trading+Brent+Donnelly&ref=aligrithm.com) ### 4.13 Intermarket Analysis for System Traders URL: https://aligrithm.com/intermarket-analysis-for-system-traders/ Last updated: 2026-07-27T13:24:58.000Z Take a trader watching only the S&P 500 chart. Price is grinding higher, the moving averages are stacked bullish, the trend looks clean. Now hand that same trader a second screen showing T-bond futures rolling over hard, and a third showing the dollar firming while gold sags. The single chart says "long, stay long." The three charts together say "the move you are riding is fighting its own macro plumbing, and the bond market usually wins that argument." Intermarket analysis is the study of how markets move against each other, and for a system trader it is a way to generate signals from data that never appears on the chart you are trading. The core claim is narrow and testable. Markets do not float independently. Bonds, stocks, currencies, and commodities are wired together by inflation, interest rates, and capital flows, and those wires create lead-lag and correlation structure you can measure and rule-ify. The retail version of this idea is a wall of squiggly lines and a guru pointing at "confluence." The systematic version is a filter: a second market gates the signals of the first, and you measure whether the gate improves the distribution of forward returns. Same data, completely different discipline. This article sets up the intermarket arc of Pillar 4\. The articles that follow drill into specific pairs (bonds filtering equities, gold against the dollar, copper as an activity proxy, crude against FX), and later ones push the idea toward network momentum and currency-strength decomposition. This one defines the object, lays out the relationship web, and converts the web into the four rule templates every later article instantiates. ## What "interrelated" actually means Two markets are intermarket-related when the state of one shifts the forward-return distribution of the other. That is the operational definition, and it has nothing to do with whether the relationship "makes sense." A relationship earns its place by changing a distribution, not by sounding plausible on a macro podcast. The classic web, measured over decades of daily and weekly data, looks like this. Stocks and long bonds move together over the medium term, because both are discounted by the same long rate. Bonds run inverse to inflation, because inflation erodes a fixed coupon. Commodities run inverse to bonds, because rising commodity prices feed inflation a few months later. Gold runs inverse to interest-rate markets, because gold pays no coupon and suffers when real yields rise. The dollar runs inverse to gold and to the broad commodity basket, because commodities are priced in dollars. Foreign currencies run with bonds, because falling US rates make dollar assets less attractive and push capital abroad. None of these is a law. Each is a tendency that holds across long samples, weakens in some regimes, and inverts in others. The point of treating them systematically is exactly that you stop trusting the story and start measuring the current strength of each link, the same way the article ["Slow Wandering: The Most Dangerous Type of Market Change"](https://aligrithm.com/slow-wandering-the-most-dangerous-type-of-market-change/) treated every structural relationship as something that drifts. ## The four rule templates Every intermarket system in this pillar is one of four shapes. Write them once, reuse them everywhere. Template 1, directional state of the external market. The external market X is in an uptrend (price above its moving average), so take or allow signals in the traded market. Example: trade the S&P long only when T-bonds are above their 50-day average. The external market acts as an on/off switch. Template 2, inverse directional state. X is in a downtrend, so take signals in the traded market. Example: trade gold long only when the dollar index is below its average. This is template 1 with the sign flipped for a negatively correlated pair. Template 3, divergence between X and the traded market. X moves one way while the traded market moves the way the correlation says it should not. Example: bonds rising while stocks fall, when the two normally move together, flags a dislocation that often resolves in the direction of the leading market. The article ["Intermarket Divergence as a Trading Filter"](https://aligrithm.com/intermarket-divergence-as-a-trading-filter/) develops this template in full. Template 4, the ratio. Build the ratio of two markets and trade its position relative to its own moving average. Example: the T-bond / S&P ratio above its average favors bonds over stocks. The ratio collapses two price series into one signal series, and the article ["Using Ratios as Trading Signals"](https://aligrithm.com/using-ratios-as-trading-signals/) handles its mechanics and traps. $$ \\text{Signal}\_{\\text{traded}} = \\begin{cases} \\text{enabled} & \\text{if } X\_t > \\text{MA}\_n(X) \\quad \\text{(template 1, positive link)}\\\\\[4pt\] \\text{enabled} & \\text{if } X\_t < \\text{MA}\_n(X) \\quad \\text{(template 2, inverse link)} \\end{cases} $$ The external market X gates the traded market by sitting above or below its own moving average of length n. For a positively correlated pair you enable when X is in an uptrend; for a negatively correlated pair you enable when X is in a downtrend. Templates 3 and 4 replace the simple threshold with a divergence test or a ratio, but the skeleton is identical: a second market decides whether the first market's signals are live. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Why a system trader bothers Three reasons, each measurable, each tied to money. The external market carries information the price chart does not. A breakout on the S&P chart is a function of S&P prices alone. Whether bonds are confirming or fighting that breakout is information from outside the chart, and adding it can split the breakout's outcomes into a cleaner set (bonds confirming) and a dirtier set (bonds fighting). The article ["Why Cross-Asset Signals Beat Isolated Chart Reading"](https://aligrithm.com/why-cross-asset-signals-beat-isolated-chart-reading/) works this split with numbers. The external market often leads. Eurodollars and short-rate markets tend to turn before the S&P at major inflection points; utility stocks have historically turned before both bonds and the broad market at major tops. A leading series is a forward-looking filter, which is rare and valuable, and the article ["Lead-Lag Relationships in Global Markets"](https://aligrithm.com/lead-lag-relationships-in-global-markets/) treats lead-lag as its own measurement problem rather than an article of faith. The external market gives you diversification of signal source. A book built only on price-derived signals (moving averages, breakouts, oscillators) has one failure mode: it dies when price stops trending. A book that also conditions on cross-asset state has signals that fire for structurally different reasons, which is the only diversification that survives a regime where pure price systems all break at once. ## The traps that kill naive intermarket systems The honest part. Intermarket analysis is a data-mining minefield, and most published intermarket systems are overfit. The correlation is non-stationary. Stock-bond correlation has been strongly positive in some decades and outright negative in others (the "risk-on/risk-off" regime flipped the sign for years). A system that hard-codes "bonds up is good for stocks" will pay for the years the sign inverts. You either re-estimate the link on a rolling window or you accept that the system has a regime where it is structurally wrong. The search space is enormous. With dozens of candidate external markets, two directions, and a range of moving-average lengths, you can manufacture a beautiful intermarket filter from pure noise. This is the data-snooping problem the article "Why Works on All Markets Is Usually a Red Flag" warned about, multiplied by the number of external markets you let yourself test. The defense is to restrict the candidate set to economically motivated links before you touch the data, and to count every market you tried against your degrees of freedom. The relationships lag in the wrong places. A few of the most-cited intermarket leads (the Dow 20 Bond Index as a bond-timing tool, for one) decayed badly once their composition or the market structure around them changed. A lead-lag relationship is itself a system, and like every system it gets switched off at some point, the acceptance the article ["Why Systems Work Until They Don't"](https://aligrithm.com/why-systems-work-until-they-dont/) built into the framework. ## Visualizing the intermarket web ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_98-visual-0.png) ## KEY POINTS - Intermarket analysis studies how markets move against each other. Two markets are related when the state of one shifts the forward-return distribution of the other, whether or not the story sounds plausible. - The classic web: stocks and long bonds move together; bonds run inverse to inflation and commodities; gold runs inverse to rate markets; the dollar runs inverse to gold and commodities; foreign currencies run with bonds. Each link is a tendency that drifts, not a law. - For a system trader the payoff is signal from outside the traded chart. A second market gates the first, and you measure whether the gate cleans up the forward-return distribution. - Four rule templates cover almost everything: external market in an uptrend enables the traded market; external market in a downtrend enables it (inverse pair); divergence between the two against the expected correlation; and the ratio of two markets versus its own moving average. - The three reasons to bother: the external market carries off-chart information, it often leads the traded market, and it diversifies the signal source so the book does not have a single price-driven failure mode. - The traps: the correlations are non-stationary and the sign can invert for years; the candidate set of external markets makes data-snooping easy; and individual lead-lag relationships decay when market structure changes. - The defense is to restrict the candidate links to economically motivated ones before touching the data, re-estimate the relationship on a rolling window, and count every market tested against the degrees of freedom. - The rest of the intermarket arc instantiates the four templates on specific pairs: bonds filtering equities, gold against the dollar and rates, copper and crude as activity and inflation proxies, ratios, and divergence. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Trading Systems and Methods - Perry Kaufman (Amazon)](https://www.amazon.com/s?k=Trading+Systems+and+Methods+Perry+Kaufman&ref=aligrithm.com) - [Cybernetic Trading Strategies - Murray Ruggiero (Amazon)](https://www.amazon.com/s?k=Cybernetic+Trading+Strategies+Murray+Ruggiero&ref=aligrithm.com) - [The Art of Currency Trading - Brent Donnelly (Amazon)](https://www.amazon.com/s?k=The+Art+of+Currency+Trading+Brent+Donnelly&ref=aligrithm.com) ### 4.12 Matching Strategy Families to Market Conditions URL: https://aligrithm.com/matching-strategy-families-to-market-conditions/ Last updated: 2026-07-27T13:24:57.000Z A trader has four working systems: a breakout, a moving-average trend follower, a mean-reversion fade, and a grid. The question is not which one is best. The question is which one to run on which instrument right now, and the answer is a lookup, not a debate. Measure the instrument's efficiency ratio and volatility at the trading horizon, read the regime off a two-axis map, and the map tells you which family belongs there. The breakout and the trend follower go on the low-noise instruments; the fade and the grid go on the high-noise ones; volatility sets the size, not the family. There is no universally best system, only systems matched to conditions, and the matching is mechanical once you measure the right two numbers. This article assembles the whole Pillar 4 framework into one decision procedure. The preceding articles built the pieces: noise versus volatility, the efficiency ratio, trend-quality ranking, the trend-reversion split, the indicator-portability problem, timeframe selection, price density, the volatility-direction distinction, and the breakout and grid design cases. This article is the map they were all building toward, the operational complement to the article ["Market Personality: Why Gold, FX, Crypto, and Equities Need Different Systems"](https://aligrithm.com/market-personality-why-gold-fx-crypto-and-equities-need-different-systems/) from Pillar 3\. Where the personality article matched frameworks to market families by structural character, this one matches them to measured conditions that change over time, so the allocation updates as the market moves. ## The two axes that determine the family Every strategy family has two environmental requirements, and exactly two numbers measure them. The first is noise, measured by the efficiency ratio at the trading horizon, which decides the family. The second is volatility, measured by average true range or its normalized form, which decides the size. The articles in this pillar drove toward this separation: the article ["Noise Is Not Volatility"](https://aligrithm.com/noise-is-not-volatility/) established the two axes are independent, and ["The Difference Between Volatility Expansion and Directional Opportunity"](https://aligrithm.com/the-difference-between-volatility-expansion-and-directional-opportunity/) showed why conditioning on only one mis-sizes or mis-selects. Noise selects the family because it determines whether motion continues or reverses, and continuation versus reversal is the fundamental divide between strategy families. Low noise (high efficiency ratio) means continuation, which the trend family (breakout, moving average, momentum) monetizes. High noise (low efficiency ratio) means reversal, which the reversion family (fade, grid, mean-reversion band) monetizes. This is the trend-reversion split from ["Low Noise Markets Are Trend-Following Markets"](https://aligrithm.com/low-noise-markets-are-trend-following-markets/) and ["High Noise Markets Are Mean-Reversion Markets"](https://aligrithm.com/high-noise-markets-are-mean-reversion-markets/), and it is the first and largest decision. Volatility sets the size because it determines how far price moves per unit time, which determines how much capital a fixed risk budget supports. Higher volatility means smaller position for the same risk, the volatility-targeting logic. Volatility does not select the family; a low-noise market is a trend market whether its volatility is high (size down) or low (size up), and a high-noise market is a reversion market at any volatility. Confusing the two axes is the error that runs grids into trends (sizing to volatility, blind to noise) and the error that abandons breakouts in chop (blaming volatility for a noise problem). ## The master map Cross the two axes and place every family in its cell. This is the two-by-two from ["Noise Is Not Volatility"](https://aligrithm.com/noise-is-not-volatility/) populated with the design conclusions from the whole pillar. | | Low noise (ER high, continuation) | High noise (ER low, reversal) | | --------------- | ----------------------------------------------------- | ---------------------------------------------------------- | | Low volatility | Quiet trend: breakout/MA, size up, let winners run | Quiet range: fade/grid, size up, tight rungs | | High volatility | Violent trend: breakout/MA, size down, wide MAE stops | Violent chop: fade with caution or stand aside, wide stops | The columns are the strategy decision, the rows are the size decision. Three of the four cells are clean. The quiet-trend cell is the breakout's best environment from ["Why Breakout Systems Need Low Noise Environments"](https://aligrithm.com/why-breakout-systems-need-low-noise-environments/). The quiet-range cell is the grid's paradise from ["Why Grid Systems Need Noise, Not Just Volatility"](https://aligrithm.com/why-grid-systems-need-noise-not-just-volatility/). The violent-trend cell is the trend follower's high-conviction environment with reduced size. The fourth cell, violent chop, is the most dangerous and the one where the honest answer is often "stand aside", because high volatility with high noise is maximum motion with no directional opportunity, the trap the volatility-direction article isolated. The map's most valuable cell is the one that tells you not to trade. ## The procedure, end to end Six steps, in order, each drawing on a specific article in the pillar. Step 1: pick the timeframe. From ["How to Choose the Right Timeframe for a Strategy"](https://aligrithm.com/how-to-choose-the-right-timeframe-for-a-strategy/), choose the timeframe where the noise constraint, cost constraint, and sample constraint are jointly satisfied for your strategy. The timeframe is upstream of everything because noise is a function of timeframe. Step 2: measure noise. Compute the efficiency ratio at the chosen horizon, cross-checked with price density from ["Price Density: A Visual Way to Measure Market Choppiness"](https://aligrithm.com/price-density-a-visual-way-to-measure-market-choppiness/). Their agreement confirms the regime; their divergence flags a one-event move or a turbulent trend. Step 3: measure volatility. Compute the average true range or its normalized form at the same horizon. This sets size, not family. Step 4: select the family from the column. Low noise, deploy the trend family; high noise, deploy the reversion family; mixed, gate or stand aside. This is the largest decision and it precedes any parameter tuning, the sequencing the article ["Optimization Comes After Testing, Not Before"](https://aligrithm.com/optimization-comes-after-testing-not-before/) insisted on. Step 5: size from the row. Scale the position inversely to volatility for risk, then tilt by the efficiency ratio for opportunity (more conviction the further from the mixed band), the two-axis sizing from the volatility-direction article. Step 6: gate dynamically. Recompute the efficiency ratio on a rolling basis and switch or disable the family when the regime drifts across the column boundary, the defensive gate that appears in every design article in this pillar. The market changes column over time, and the allocation has to follow. $$ \\text{Family} = f(\\text{ER}\_t), \\qquad \\text{Size} = \\frac{\\text{risk budget}}{\\text{ATR}\_t} \\times g(\\text{ER}\_t) $$ The family is a function of the efficiency ratio alone (the column), while the size is the risk budget divided by volatility (the row) times an opportunity tilt that also depends on the efficiency ratio. Two numbers, the efficiency ratio and the average true range, drive both decisions. Everything else in the pillar is about measuring these two correctly and protecting against their drift. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Why a portfolio of opposite families beats one system The map also explains why running multiple families is not indecision but diversification of a specific, valuable kind. The trend family and the reversion family are hostile to opposite regimes: trend dies in chop, reversion dies in trend. A portfolio that runs both, each gated to its own regime and each on the instruments that suit it, has two halves that lose in different conditions, which is the only diversification that survives a correlated sell-off, the article "Market Personality" point about matching frameworks to conditions extended to a within-book hedge. When the trend half is bleeding through a choppy patch, the reversion half is in its element, and vice versa. The condition for this to work is that the gating is honest. A trend system that keeps trading in chop and a grid that keeps trading in a trend are not diversified; they are two ways of losing in the same regime, because both have leaked outside their cell. The diversification is real only if each family is actually confined to its column by the efficiency-ratio gate. The article ["Why Systems Work Until They Don't"](https://aligrithm.com/why-systems-work-until-they-dont/) framed the acceptance that every system has a hostile regime; the multi-family book's value is that the families' hostile regimes do not overlap, so the book as a whole has no single regime that kills all of it, provided each piece stays in its lane. ## What the map does not do Honesty about the framework's limits. The map tells you which family fits the current measured regime; it does not predict regime changes. A market in the low-noise column can cross into the high-noise column next week, and the efficiency-ratio gate reacts after the crossing begins, not before, so there is always a transition cost as the gate catches up to the regime shift. This is the inductive limit from ["Induction in Trading: Why Past Patterns Are Always Uncertain"](https://aligrithm.com/induction-in-trading-why-past-patterns-are-always-uncertain/): the recent efficiency ratio is the best available estimate of the current regime and a poor predictor of the next one. The map reduces the cost of being in the wrong family; it does not eliminate the lag at regime boundaries. The map also does not manufacture edge where none exists. Matching the right family to the right regime is necessary, not sufficient: a correctly-matched breakout in a low-noise market still has to clear costs, still has the low win rate, still needs enough trades to be reliable. The map solves the selection problem, which is the largest and most-ignored one, and leaves the executions, costs, and sample-size problems that the rest of the publication addressed. A perfectly matched family on an instrument where costs exceed the edge still loses, the point the article ["Why Transaction Costs Should Be Added Before You Fall in Love"](https://aligrithm.com/why-transaction-costs-should-be-added-before-you-fall-in-love/) made and the trend-quality ranking encoded in its cost-reach term. ## Decision summary | Measured condition | Family | Size | Note | | -------------------- | ------------------------------- | ------- | -------------------------------- | | ER high, vol low | Breakout / MA trend | Larger | Quiet trend, let winners run | | ER high, vol high | Breakout / MA trend | Smaller | Violent trend, wide MAE stops | | ER low, vol low | Fade / grid | Larger | Quiet range, capped ladder | | ER low, vol high | Fade with caution / stand aside | Smaller | Violent chop, the dangerous cell | | ER mixed (0.20-0.30) | Gate or stand aside | Minimal | Highest overfit risk | Two numbers select everything: the efficiency ratio picks the family, volatility sets the size. Measure them at the right timeframe, gate on the efficiency ratio as the regime drifts, run opposite families on the instruments that suit them, and accept that the map reduces but does not remove the cost of regime change. ## Visualizing the master map ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_97-visual-0.png) ## KEY POINTS - There is no universally best system, only systems matched to conditions. The matching is mechanical: measure two numbers at the trading horizon and read the family off a two-axis map. - Two axes determine everything. Noise (the efficiency ratio) selects the family because it decides continuation versus reversal. Volatility (average true range) sets the size because it decides how much capital a fixed risk budget supports. The axes are independent. - The master map: low noise is the trend family (breakout, moving average, momentum); high noise is the reversion family (fade, grid). Low volatility sizes up; high volatility sizes down. The family is the column decision, the size is the row decision. - Three of the four cells are clean (quiet trend, violent trend, quiet range). The fourth, violent chop (high volatility with high noise), is the most dangerous, and the honest answer is often to stand aside. The map's most valuable cell tells you not to trade. - The end-to-end procedure: pick the timeframe (jointly satisfy noise, cost, sample), measure noise (efficiency ratio cross-checked with price density), measure volatility, select the family from the column, size from the row with an opportunity tilt, and gate dynamically as the regime drifts. - The family is a function of the efficiency ratio alone; the size is the risk budget divided by volatility times an efficiency-ratio opportunity tilt. Two measured numbers drive both decisions. - Running opposite families is diversification of the only kind that survives a correlated move: trend and reversion are hostile to opposite regimes, so a book that gates each to its column has no single regime that kills all of it, provided each piece stays in its lane. - The map's limits: it does not predict regime changes (the gate reacts after the crossing, with a transition cost), and it does not manufacture edge where costs exceed it. It solves the selection problem, the largest and most-ignored one, and leaves the cost, execution, and sample-size problems to the rest of the framework. - This article closes the Pillar 4 noise-and-conditions arc. Combined with the per-market personality view, the operational rule is complete: measure the efficiency ratio and volatility at the right timeframe, deploy the family the column demands, size from the row, and gate as the market moves. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Trading Systems and Methods - Perry Kaufman (Amazon)](https://www.amazon.com/s?k=Trading+Systems+and+Methods+Perry+Kaufman&ref=aligrithm.com) - [Cybernetic Trading Strategies - Murray Ruggiero (Amazon)](https://www.amazon.com/s?k=Cybernetic+Trading+Strategies+Murray+Ruggiero&ref=aligrithm.com) - [The Art of Currency Trading - Brent Donnelly (Amazon)](https://www.amazon.com/s?k=The+Art+of+Currency+Trading+Brent+Donnelly&ref=aligrithm.com) ### 4.11 Why Grid Systems Need Noise, Not Just Volatility URL: https://aligrithm.com/why-grid-systems-need-noise-not-just-volatility/ Last updated: 2026-07-27T13:24:57.000Z A grid system places a ladder of buy orders below the current price and sell orders above it, harvesting a small profit each time price oscillates across a rung. Traders who run grids obsess over volatility, because more volatility means price travels farther and crosses more rungs, which feels like more profit. That instinct is half right and half fatal. A grid does need movement to cross rungs, but movement alone is not the requirement. The requirement is oscillation that comes back. A high-volatility market that travels far in one direction crosses every rung once and then strands the entire ladder of losing positions on the wrong side, never to be closed. The grid did not need volatility. It needed noise, and the two are not the same. This article is the grid-design consequence of the noise-versus-volatility distinction, the mirror of the previous article's breakout case. ["Why Breakout Systems Need Low Noise Environments"](https://aligrithm.com/why-breakout-systems-need-low-noise-environments/) showed the trend family needs low noise; the grid is the purest case of a family that needs high noise, and it is dangerous precisely because volatility, the thing grid traders watch, looks like the requirement while actually being a trap when it comes without noise. The article ["Noise Is Not Volatility"](https://aligrithm.com/noise-is-not-volatility/) drew the distinction; the grid is where confusing the two empties an account. ## A grid is a bet that price will come back across each rung The grid's profit mechanism is round-trips. Price falls to a buy rung, you buy; price rises back across that rung, you sell for a small gain; repeat. Each completed round-trip is a small profit, and a grid in a oscillating market completes many of them. The entire premise is that price crosses each rung in both directions, down to trigger the buy and back up to close it. That two-way crossing is the definition of high noise: motion that cancels out, returning to where it was. The article ["High Noise Markets Are Mean-Reversion Markets"](https://aligrithm.com/high-noise-markets-are-mean-reversion-markets/) established that fade systems harvest cancellation; a grid is a mechanized fade system, a ladder of small reversion bets, and it harvests the same noise. Volatility helps only to the extent it produces two-way crossings. Volatility that produces one-way travel is the grid's ruin. When price trends through the grid without returning, every buy rung below a falling price fills and never closes (the price keeps falling, the positions keep losing), or every sell rung above a rising price fills and never closes. The grid accumulates a stack of open losers in the trend direction while booking nothing, the exact opposite of its intended behavior. This is why a grid's risk is not measured by volatility but by the probability of a sustained one-way move, which is low noise, the grid's lethal environment. ## The math: round-trips need return, not range Quantify why volatility without noise is worthless to a grid. Suppose a grid with rungs spaced d apart over a window where price covers total path length S. The number of rung crossings is roughly the path length divided by the rung spacing, and each crossing is half a round-trip. But a crossing only becomes a profit if it is matched by a return crossing, and the fraction of crossings that get matched is governed by how much the motion cancels, the noise level. $$ \\text{Round-trips} \\approx \\frac{1}{2}\\left(\\frac{S}{d}\\right)\\cdot(1 - \\text{ER}), \\qquad \\text{Stranded positions} \\approx \\frac{|P\_{\\text{end}} - P\_{\\text{start}}|}{d} = \\frac{S \\cdot \\text{ER}}{d} $$ The number of completed round-trips scales with the total path S over the rung spacing d, multiplied by one minus the efficiency ratio: the more the motion cancels (low efficiency ratio, high noise), the more crossings get matched into round-trips. The number of stranded, unclosed positions scales with the net displacement, which is the path times the efficiency ratio: the more the motion accumulates in one direction (high efficiency ratio, low noise), the more positions get left open on the wrong side. The two terms split the path between profit and danger by the efficiency ratio. At efficiency ratio near 0 (pure noise), almost all crossings become round-trips and almost nothing is stranded: the grid's paradise. At efficiency ratio near 1 (pure trend), almost nothing round-trips and the displacement strands the whole ladder: the grid's grave. Volatility sets S, the total path, but the efficiency ratio decides how S is divided between the round-trip term and the stranded term, and only the round-trip term is profit. This is the precise sense in which a grid needs noise, not just volatility. Two markets with identical volatility (identical S) deliver opposite grid outcomes depending on their efficiency ratio. The grid trader who sizes the grid to volatility and ignores the efficiency ratio is sizing to S while blind to the split, and the split is the whole game. ## The hidden short-volatility-of-trend exposure A grid has the risk profile the article ["Why Profit Factor Can Lie"](https://aligrithm.com/why-profit-factor-can-lie/) warned about, in its most extreme form: a high win rate (almost every round-trip closes for a small gain) hiding a catastrophic left tail (the one-way move that strands the ladder and runs). The grid's equity curve looks immaculate in a ranging market, a steady climb of small wins, right up until a trend arrives and the stranded positions produce a loss larger than all the accumulated round-trips combined. This is the classic grid blow-up, and it is structural: the grid is implicitly short the trend, and a trend is its short-volatility-style tail event. Most grid blow-ups are misattributed to "too much volatility". The volatility was not the problem; a high-volatility ranging market is a grid's best environment. The problem was the efficiency ratio rising, the market converting its volatility from two-way oscillation into one-way displacement. The article "Volatility Expansion vs Directional Opportunity" separated these exactly: the grid survives a volatility expansion that stays noisy and dies on a volatility expansion that becomes directional. A grid trader watching only volatility cannot see the difference coming; a grid trader watching the efficiency ratio can. ## Designing a grid that respects noise If the grid needs high noise, the design follows from measuring and protecting against the efficiency ratio, not the volatility. Three rules, mirroring the breakout's three. Rule 1: select instruments by inverse trend quality. The grid belongs on the bottom of the trend-quality ranking from ["How to Rank Markets by Trend Quality"](https://aligrithm.com/how-to-rank-markets-by-trend-quality/), the high-noise, low-efficiency-ratio instruments (G10 FX, range-bound commodities) where two-way crossings dominate. Running a grid on a top-ranked trending instrument is a design error that the smooth early equity curve will hide until the trend arrives. Rule 2: gate on the rolling efficiency ratio, and gate hard. The grid's gate is more important than the breakout's because the grid's failure is catastrophic rather than gradual. When the rolling efficiency ratio at the grid's horizon rises toward trend territory, stop adding rungs in the trend direction and begin closing stranded positions, accepting the small loss to avoid the large one. The article ["When a Stop Loss Improves Risk but Destroys Edge"](https://aligrithm.com/when-a-stop-loss-improves-risk-but-destroys-edge/) noted that stops trade edge for risk control; for a grid the stop (a hard exit when the efficiency ratio signals a regime change) is non-negotiable, because the ungated grid's tail is unbounded. Rule 3: cap the ladder and never average into a trend indefinitely. A grid that keeps adding rungs as price runs away is a martingale, and a martingale meets a trend and dies. Cap the total number of rungs and the total exposure so that even a full one-way run through the entire grid is a survivable loss, not an account-ending one. The cap is the difference between a grid that has a bad month and a grid that has a last month. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## The asymmetry with breakouts The grid and the breakout are exact mirror images, and seeing the symmetry clarifies both. The breakout needs low noise and dies in high noise; the grid needs high noise and dies in low noise. The breakout has many small losers and rare large winners; the grid has many small winners and rare large losers. The breakout's equity curve is lumpy and looks bad in chop; the grid's equity curve is smooth and looks great in chop right until it does not. The breakout's danger is visible (a string of losses you feel); the grid's danger is hidden (a tail you do not feel until it arrives). The article ["Why Profit Factor Can Lie"](https://aligrithm.com/why-profit-factor-can-lie/) applies to both but in opposite directions, and a trader who runs both families on the right instruments has a portfolio whose two halves are hostile to opposite regimes, which is the diversification the next article, ["Matching Strategy Families to Market Conditions"](https://aligrithm.com/matching-strategy-families-to-market-conditions/), builds on. The grid is the more dangerous of the two for a retail trader precisely because its failure is hidden. A breakout that whipsaws tells you immediately it is in the wrong regime; you feel the losses and stop. A grid in a slowly trending market keeps booking small wins while the stranded losers quietly grow, so the feedback that you are in the wrong regime is delayed until the loss is already large. The efficiency-ratio gate exists to provide the feedback the equity curve withholds. ## Decision summary | Efficiency ratio at grid horizon | Two-way crossing fraction | Action | | -------------------------------- | ------------------------------------- | ------------------------------------------------ | | Below 0.15 | High (most crossings round-trip) | Grid's ideal environment, run with capped ladder | | 0.15 to 0.25 | Moderate | Grid with tighter rung cap, watch the gate | | Above 0.30 | Low (displacement strands the ladder) | Close the grid; a trend will strand it | A grid needs noise, not volatility. Volatility sets how far price travels; the efficiency ratio decides whether that travel is round-trips (profit) or displacement (stranded losers). Select high-noise instruments, gate hard on the efficiency ratio, and cap the ladder so the inevitable trend is survivable. ## Visualizing the grid's true requirement ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_96-visual-0.png) ## KEY POINTS - A grid harvests round-trips: price crosses a rung down to trigger a buy and back up to close it for a small gain. The premise is two-way crossing, which is the definition of high noise (motion that comes back). - Volatility helps only when it produces two-way crossings. Volatility that produces one-way travel is the grid's ruin: rungs fill and never close, accumulating a stack of stranded losers in the trend direction. - The math splits the path by the efficiency ratio: completed round-trips scale with one minus the efficiency ratio (the cancellation), and stranded positions scale with the efficiency ratio (the displacement). Volatility sets the total path; the efficiency ratio decides how it splits, and only the round-trip part is profit. - Two markets with identical volatility deliver opposite grid outcomes depending on their efficiency ratio. Sizing a grid to volatility while blind to the efficiency ratio is sizing to the path while ignoring the split that is the whole game. - A grid has a high win rate hiding a catastrophic left tail. The smooth equity curve of small wins persists until a trend strands the ladder and produces a loss larger than all accumulated round-trips. The grid is implicitly short the trend. - Most grid blow-ups are misattributed to "too much volatility". A high-volatility ranging market is a grid's best environment; the killer is the efficiency ratio rising, converting oscillation into displacement. - Three design rules: select high-noise (bottom-of-trend-quality) instruments, gate hard on the rolling efficiency ratio and close stranded positions when it rises, and cap the ladder so a full one-way run is survivable rather than account-ending. Never run an uncapped grid that averages into a trend (a martingale). - The grid is the exact mirror of the breakout and the more dangerous for retail because its failure is hidden: a whipsawing breakout signals the wrong regime immediately, while a grid books small wins as the stranded losers quietly grow. The efficiency-ratio gate supplies the feedback the equity curve withholds. - The next article, ["Matching Strategy Families to Market Conditions"](https://aligrithm.com/matching-strategy-families-to-market-conditions/), assembles the whole Pillar 4 framework: a map from measured market conditions to the strategy family that fits, with trend and grid as the two poles. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Trading Systems and Methods - Perry Kaufman (Amazon)](https://www.amazon.com/s?k=Trading+Systems+and+Methods+Perry+Kaufman&ref=aligrithm.com) - [Cybernetic Trading Strategies - Murray Ruggiero (Amazon)](https://www.amazon.com/s?k=Cybernetic+Trading+Strategies+Murray+Ruggiero&ref=aligrithm.com) - [The Art of Currency Trading - Brent Donnelly (Amazon)](https://www.amazon.com/s?k=The+Art+of+Currency+Trading+Brent+Donnelly&ref=aligrithm.com) ### 4.10 Why Breakout Systems Need Low Noise Environments URL: https://aligrithm.com/why-breakout-systems-need-low-noise-environments/ Last updated: 2026-07-27T13:24:56.000Z A breakout system buys when price exceeds the highest high of the prior 20 days. The signal itself is trivial; the price either makes a new 20-day high or it does not. What determines whether that signal makes money is not the breakout rule but the environment the breakout fires into. In a low-noise market, a new 20-day high is the leading edge of a sustained move, and the breakout captures most of it. In a high-noise market, a new 20-day high is the top of an oscillation that is about to reverse, and the breakout buys the exact worst price. The rule is identical. The noise environment decides whether the rule is an entry or a trap. This article is the design consequence of the trend-reversion split for the most popular trend-entry mechanism. ["Low Noise Markets Are Trend-Following Markets"](https://aligrithm.com/low-noise-markets-are-trend-following-markets/) established that trend systems need efficient markets; the breakout is the sharpest case because its entire premise, that a new extreme will be followed by continuation, is a direct bet on low noise. A breakout in a high-noise market is a contradiction in terms: you are betting on continuation in a market defined by reversal. The numbers make the dependence stark, which is why breakout systems are the cleanest illustration of why the noise environment is the first design parameter, not the last. ## A breakout is a bet that an extreme will not reverse The breakout logic assumes that when price pushes past a recent boundary, the push will continue rather than snap back. That assumption is the definition of low noise. In a low-noise market, extremes are made by directional flow that keeps going, so the new high is a waypoint, not a destination. In a high-noise market, extremes are made by oscillation that reverses, so the new high is a destination the price immediately leaves, in the wrong direction for your long. The whipsaw cost is the breakout's structural enemy, and it scales with noise. Every false breakout (a new high that reverses) costs the entry-to-stop distance. In a high-noise market, most breakouts are false because most extremes reverse, so the system pays the whipsaw cost over and over while waiting for the rare breakout that runs. The article ["Noise Is Not Volatility"](https://aligrithm.com/noise-is-not-volatility/) framed high noise as motion that cancels out; for a breakout system, every cancellation is a false signal and a small loss. The breakout does not fail gracefully in noise; it fails repeatedly. The source material on breakout design points at the same dependence from the other direction: the value of a breakout improves when the lookback is long enough to filter noise but short enough to catch the trend, and that balance only exists when there is a trend to catch. In a market with no sustained direction at any reasonable lookback, no breakout length works, because the thing the breakout is trying to catch is not present. ## The hit-rate math that low noise rescues A breakout system has a low win rate by design (the trend-payoff asymmetry from ["Low Noise Markets Are Trend-Following Markets"](https://aligrithm.com/low-noise-markets-are-trend-following-markets/): many small losers, few large winners). It survives only if the few winners are large enough to pay for the many losers. The noise level controls both the fraction of breakouts that are true and the size of the winners relative to the losers, so it controls whether the asymmetry closes. Write the breakeven condition. Let p be the fraction of breakouts that are true (lead to a real move), the average winner be W, and the average loser (a false breakout cut at the stop) be L. $$ E = p \\cdot W - (1 - p) \\cdot L > 0 \\quad\\Longleftrightarrow\\quad \\frac{W}{L} > \\frac{1 - p}{p} $$ The expectancy E is positive only when the win-to-loss size ratio exceeds the loss-to-win odds ratio. In a low-noise market, p is higher (more breakouts are true) and W is larger (the true ones run further before reversing), so both sides of the inequality move in your favor. Suppose low noise gives p equal to 0.40 and a win-to-loss ratio of 3.0: the condition needs W over L above (1 minus 0.40) over 0.40, equal to 1.5, and 3.0 clears it comfortably. Now suppose high noise gives p equal to 0.25 and a win-to-loss ratio of 1.5 (winners are smaller because the market reverses sooner): the condition needs W over L above 0.75 over 0.25, equal to 3.0, and 1.5 fails it badly. The same breakout rule has positive expectancy in low noise and negative expectancy in high noise, and the efficiency ratio predicts which regime you are in before you trade. ## Designing the breakout around the noise environment If the breakout depends on low noise, the design follows: measure the noise, deploy the breakout only where the noise is low, and gate it dynamically when the noise rises. Three design rules. Rule 1: select instruments by trend quality. The breakout belongs on the top of the trend-quality ranking from ["How to Rank Markets by Trend Quality"](https://aligrithm.com/how-to-rank-markets-by-trend-quality/), the instruments with high median efficiency ratio and high persistence. Deploying a breakout on a bottom-ranked choppy instrument is a design error no parameter tuning fixes. The selection is the first and largest design decision. Rule 2: gate entries on the rolling efficiency ratio. Even on a good instrument, the regime drifts. Disable or shrink breakout entries when the rolling efficiency ratio at the breakout horizon falls toward chop territory, the defensive gate from ["Low Noise Markets Are Trend-Following Markets"](https://aligrithm.com/low-noise-markets-are-trend-following-markets/). This keeps the breakout out of the choppy patches where its expectancy is negative, paying the small cost of missing the occasional breakout that fires during a low-efficiency reading. Rule 3: confirm the breakout with directional follow-through. From ["The Difference Between Volatility Expansion and Directional Opportunity"](https://aligrithm.com/the-difference-between-volatility-expansion-and-directional-opportunity/), a new high accompanied by a volatility expansion is not enough; require the post-breakout efficiency ratio or a close near the extreme to confirm direction before committing full size. This filters the violent-chop false breakouts (big bar through the high, immediate reversal) that pure price-level breakouts get trapped by. These three are layers of the same idea: the breakout rule is fixed and cheap; the value is entirely in restricting it to the low-noise environments where its continuation premise holds. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## The temptation to "fix" breakouts in noise, and why it fails When a breakout system whipsaws in a choppy market, the tempting fixes all make it worse. Widening the stop to avoid whipsaws turns the many small losers into fewer large losers without adding winners, because the problem is that the extremes reverse, not that the stop is too tight. Adding confirmation filters until the whipsaws stop usually filters out the real breakouts too, because in a choppy market the real and false breakouts look the same on the way in. Shortening the lookback to catch moves earlier increases the signal count and the whipsaw count together. Every one of these is an attempt to make a continuation strategy work in a reversal environment, and they fail for the same reason: the environment, not the rule, is wrong. The article ["The Difference Between Robustness and Optimization"](https://aligrithm.com/the-difference-between-robustness-and-optimization/) warned that piling on parameters to rescue a strategy in a hostile regime is how you overfit; the breakout-in-chop case is the textbook example. The correct response to a breakout that is whipsawing is not to fix the breakout. It is to recognize that the market has entered a high-noise regime and either step aside (the gate) or switch to the mean-reversion family that the high-noise regime favors (the family switch from ["Why One Indicator Should Not Be Used on Every Market"](https://aligrithm.com/why-one-indicator-should-not-be-used-on-every-market/)). A whipsawing breakout is the market telling you it has changed regime, and the information is more valuable than the lost trade. ## Decision summary | Efficiency ratio at breakout horizon | True-breakout fraction | Action | | ------------------------------------ | ---------------------------- | ---------------------------------------------------- | | Above 0.30 | High (most breakouts run) | Deploy breakout at full size, let winners run | | 0.20 to 0.30 | Mixed | Breakout with directional confirmation, reduced size | | Below 0.20 | Low (most breakouts reverse) | Disable breakout; switch to mean reversion | The breakout rule is trivial and the environment is everything. Select low-noise instruments, gate on the rolling efficiency ratio, confirm with follow-through, and never try to optimize a breakout into a choppy market. ## Visualizing the breakout's noise dependence ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_95-visual-0.png) ## KEY POINTS - A breakout signal is trivial; the price makes a new extreme or it does not. The noise environment the signal fires into decides whether it is an entry or a trap. - A breakout is a direct bet that an extreme will be followed by continuation, which is the definition of low noise. A breakout in a high-noise market bets on continuation in a market defined by reversal. - The whipsaw cost scales with noise. In high noise most extremes reverse, so most breakouts are false, and the system pays the whipsaw cost repeatedly while waiting for the rare runner. - Breakouts have a low win rate by design and survive only if the few winners pay for the many losers. The breakeven needs the win-to-loss size ratio above the loss-to-win odds ratio. - Low noise moves both sides of the inequality in your favor: more breakouts are true (higher p) and the true ones run further (larger W). The same rule has positive expectancy in low noise and negative in high noise. - Worked example: low noise (p 0.40, W/L 3.0) clears the 1.5 hurdle; high noise (p 0.25, W/L 1.5) fails the 3.0 hurdle. The efficiency ratio predicts which regime before you trade. - Three design rules: select instruments by trend quality, gate entries on the rolling efficiency ratio, and confirm breakouts with directional follow-through to filter violent-chop false signals. - The tempting fixes (wider stops, more filters, shorter lookback) all fail because they try to make a continuation strategy work in a reversal environment. A whipsawing breakout means the regime changed; step aside or switch to mean reversion rather than optimizing the breakout into the chop. - The next article, ["Why Grid Systems Need Noise, Not Just Volatility"](https://aligrithm.com/why-grid-systems-need-noise-not-just-volatility/), takes the opposite design case: the strategy family that needs exactly the high-noise environment a breakout cannot survive. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Trading Systems and Methods - Perry Kaufman (Amazon)](https://www.amazon.com/s?k=Trading+Systems+and+Methods+Perry+Kaufman&ref=aligrithm.com) - [Cybernetic Trading Strategies - Murray Ruggiero (Amazon)](https://www.amazon.com/s?k=Cybernetic+Trading+Strategies+Murray+Ruggiero&ref=aligrithm.com) - [The Art of Currency Trading - Brent Donnelly (Amazon)](https://www.amazon.com/s?k=The+Art+of+Currency+Trading+Brent+Donnelly&ref=aligrithm.com) ### 4.9 The Difference Between Volatility Expansion and Directional Opportunity URL: https://aligrithm.com/the-difference-between-volatility-expansion-and-directional-opportunity/ Last updated: 2026-07-27T13:24:56.000Z A volatility breakout system fires when the daily range expands past a multiple of its recent average true range, on the theory that a volatility burst signals the start of a move. On a day when crude oil's range triples after a supply headline and the price closes near its high, the system enters and rides a clean directional follow-through. On a day when the SPX range triples after a confusing macro print and the price whipsaws violently but closes near its open, the same system enters and gets stopped out by the next reversal. Both days had identical volatility expansion. Only one had directional opportunity. The system that cannot tell them apart will take both trades and wonder why half its volatility-breakout signals fail. This article separates two things that the word "volatility" routinely conflates and that the noise framework already gives us the tools to distinguish. The article "Noise Is Not Volatility" established that volatility (how far price moves) and noise (how much cancels out) are different axes. Volatility expansion is a change in the magnitude of motion. Directional opportunity is net displacement per unit of that motion. A market can expand its volatility without producing any net direction, and that combination, big motion with no displacement, is the most dangerous environment for a system that treats volatility as a signal. ## Two different events that look the same on a volatility chart Volatility expansion is a denominator event. It means the daily ranges got bigger, the price is covering more ground per bar, the average true range is rising. That is all it means. It says nothing about whether the ground covered is accumulating in a direction or canceling out. The article ["Why Volatility Is More Non-Stationary Than Trend"](https://aligrithm.com/why-volatility-is-more-non-stationary-than-trend/) documented how volatility clusters and spikes on its own clock; those spikes are denominator events, and they arrive in both trending and choppy regimes. Directional opportunity is a numerator-over-denominator event. It means the net displacement is large relative to the path, the efficiency ratio is high, the motion is going somewhere. This is the thing a directional strategy can actually monetize. It can coincide with a volatility expansion (a violent trending breakout) or with low volatility (a quiet steady grind), because direction and magnitude are close to independent, the two-axis point from the noise article. The confusion happens because the two events share a symptom: both can produce a large bar. A large bar in a trend is direction. A large bar in a chop is just a wide oscillation that will be reversed. The bar size cannot distinguish them, which is exactly why a volatility-magnitude signal alone is an incomplete trade trigger. You need the second axis, whether the expansion is converting to displacement, and that is the efficiency ratio or price density measured around the expansion. ## The four combinations Cross volatility (expanding or contracting) with direction (high or low efficiency ratio) and you get four regimes, each demanding different handling. This is the two-by-two from ["Noise Is Not Volatility"](https://aligrithm.com/noise-is-not-volatility/) applied to the dynamic case, where volatility is changing rather than fixed. | | Low noise (directional) | High noise (no direction) | | ---------------------- | ------------------------------------- | ---------------------------------------- | | Volatility expanding | Violent trend: best trend environment | Violent chop: worst environment, whipsaw | | Volatility contracting | Quiet grind: trend with small size | Quiet range: classic fade environment | The top-left, expanding volatility with direction, is the volatility-breakout system's dream: a real move with size behind it. The top-right, expanding volatility without direction, is its nightmare: maximum motion, zero displacement, every entry reversed. A system that triggers on volatility expansion alone takes both top cells and cannot tell which it is in until after the loss. The fix is to require a directional confirmation (a rising efficiency ratio, a close near the extreme, a follow-through bar) before treating a volatility expansion as a trade signal. Volatility expansion is a necessary condition for a violent trend and not a sufficient one. ## A worked separation Quantify the two days from the opening. Both expand from an average true range of 1.0 to a range of 3.0 on the signal day, identical volatility expansion. Crude oil day: the 3.0 range is covered mostly in one direction, the close is 2.7 above the open, and the next two bars add another 1.5 in the same direction. Measure the efficiency ratio over the three bars: net change about 4.2, total path about 5.0, efficiency ratio 0.84\. High. This was directional opportunity wearing a volatility expansion. SPX day: the 3.0 range is covered by a violent up-then-down whipsaw, the close is 0.2 above the open, and the next two bars oscillate another 4.0 of path while ending near where they started. Efficiency ratio over the three bars: net change about 0.3, total path about 7.0, efficiency ratio 0.04\. Near zero. This was volatility expansion with no directional opportunity at all. $$ \\text{ER}\_{\\text{event}} = \\frac{|P\_{\\text{end}} - P\_{\\text{start}}|}{\\sum |P\_i - P\_{i-1}|} \\quad\\Rightarrow\\quad \\text{crude: } \\frac{4.2}{5.0} = 0.84, \\quad \\text{SPX: } \\frac{0.3}{7.0} = 0.04 $$ Same volatility expansion, an efficiency ratio of 0.84 versus 0.04 around the event. The efficiency ratio measured over the bars following the expansion is what separates the tradeable expansion from the trap. A volatility-breakout system that adds this filter, only act on an expansion if the immediate efficiency ratio confirms direction, takes the crude trade and skips the SPX trade, which is the entire difference between the strategy working and not. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Why this matters for sizing, not just entries The distinction also breaks a common position-sizing error. Volatility-targeting sizing scales positions inversely to volatility, so it cuts size when volatility expands. That is correct for risk control and silent about opportunity. In the top-left cell (violent trend), cutting size because volatility expanded means under-betting the best directional opportunity available, the moment you most want exposure. In the top-right cell (violent chop), the volatility-targeting cut is appropriate because there is no opportunity to size into. Volatility targeting treats both cells identically, sizing down, when one deserves more conviction and the other deserves none. The richer sizing rule conditions on both axes: size to volatility for risk, then tilt by the efficiency ratio for opportunity. When volatility expands with high directional confirmation, the volatility cut and the directional tilt partly offset, leaving meaningful exposure into the violent trend. When volatility expands with no directional confirmation, both the volatility cut and the absent tilt point the same way, toward little or no exposure. The article "Market Personality" listed vol-conditioned sizing as a fit for several families; the two-axis refinement is to condition on direction as well as magnitude, because magnitude alone mis-sizes the violent-trend cell. ## The misdiagnosis this prevents The error this article kills is "my volatility breakout system has a low win rate, so volatility breakouts do not work". Often the truth is that the system traded both top cells indiscriminately, won in the violent-trend cell, lost in the violent-chop cell, and the blended low win rate hid a strong conditional edge. Split the historical signals by the post-expansion efficiency ratio and the picture usually clarifies: the high-efficiency-ratio expansions were profitable, the low-efficiency-ratio expansions were the losers, and the strategy was never broken, it was unconditioned. This is the same stratification discipline the article ["Regime Coverage: Why Your Backtest Needs Different Market States"](https://aligrithm.com/regime-coverage-why-your-backtest-needs-different-market-states/) applied to regimes, here applied to the volatility-direction split within the signal set. The forward caveat is the usual one. The post-expansion efficiency ratio confirms direction only after a few bars have printed, which means the confirmation costs you the first part of the move. You trade a slightly later, more reliable entry for the earlier, noisier one. That trade-off is real and there is no free version: requiring directional confirmation reduces whipsaw and also reduces the captured move, and the right balance depends on the instrument's noise structure, which is why you measure the efficiency ratio per instrument rather than assuming. ## Decision summary | Volatility | Post-event efficiency ratio | Regime | Action | | ----------- | ------------------------------- | ------------- | ------------------------------------------------ | | Expanding | High (above \~0.5 on the event) | Violent trend | Trade the breakout, tilt size up despite vol cut | | Expanding | Low (below \~0.2 on the event) | Violent chop | Skip; volatility without direction is a trap | | Contracting | High | Quiet grind | Trend with small size | | Contracting | Low | Quiet range | Fade environment | Volatility expansion is a necessary but not sufficient condition for a directional trade. Confirm with the efficiency ratio around the event before treating the expansion as a signal, and condition size on direction, not magnitude alone. ## Visualizing the two events ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_94-visual-0.png) ## KEY POINTS - Volatility expansion (how far price moves per bar) and directional opportunity (net displacement per unit of motion) are different events that share a symptom: both can produce a large bar. - A large bar in a trend is direction; a large bar in a chop is a wide oscillation that reverses. Bar size alone cannot distinguish them, which is why a volatility-magnitude signal is an incomplete trade trigger. - Crossing volatility (expanding/contracting) with direction (high/low efficiency ratio) gives four regimes. Expanding volatility with direction is the violent trend (best for breakouts); expanding volatility without direction is the violent chop (worst, pure whipsaw). - A system that triggers on volatility expansion alone takes both top cells and cannot tell which it is in until after the loss. Require directional confirmation (rising efficiency ratio, close near the extreme, follow-through) before treating an expansion as a signal. - Worked example: two days expand identically from ATR 1.0 to range 3.0; the crude day reads efficiency ratio 0.84 over the following bars (direction), the SPX day reads 0.04 (no direction). The event-window efficiency ratio separates the tradeable expansion from the trap. - Volatility-targeting sizing cuts size when volatility expands, which under-bets the violent-trend cell (the moment you most want exposure) and correctly cuts the violent-chop cell. It treats both identically. The fix: size to volatility for risk, then tilt by the efficiency ratio for opportunity. - The misdiagnosis this prevents: "volatility breakouts do not work" usually means the signals were unconditioned, winning in the trend cell and losing in the chop cell, with the blend hiding a strong conditional edge. Split signals by post-expansion efficiency ratio to see it. - Directional confirmation costs the first part of the move: a later, more reliable entry trades against an earlier, noisier one. The right balance depends on the instrument's noise structure, so measure it per instrument. - The next article, ["Why Breakout Systems Need Low Noise Environments"](https://aligrithm.com/why-breakout-systems-need-low-noise-environments/), applies this directly to breakout design: why a breakout signal is only as good as the noise environment it fires into. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Trading Systems and Methods - Perry Kaufman (Amazon)](https://www.amazon.com/s?k=Trading+Systems+and+Methods+Perry+Kaufman&ref=aligrithm.com) - [Cybernetic Trading Strategies - Murray Ruggiero (Amazon)](https://www.amazon.com/s?k=Cybernetic+Trading+Strategies+Murray+Ruggiero&ref=aligrithm.com) - [The Art of Currency Trading - Brent Donnelly (Amazon)](https://www.amazon.com/s?k=The+Art+of+Currency+Trading+Brent+Donnelly&ref=aligrithm.com) ### 4.8 Price Density: A Visual Way to Measure Market Choppiness URL: https://aligrithm.com/price-density-a-visual-way-to-measure-market-choppiness/ Last updated: 2026-07-27T13:24:55.000Z Draw a box around 10 days of price action: the top of the box is the highest high of the 10 days, the bottom is the lowest low. Now ask how much of that box the price actually filled in with its daily ranges. A market that trended cleanly through the 10 days drew one diagonal line corner to corner and left most of the box empty. A market that chopped sideways painted the whole box, its daily ranges stacking on top of each other to fill the space. The fraction of the box that got filled is price density, and a high number means a choppy market while a low number means a directional one. You can almost read it off the chart by eye, which is the point: it is the efficiency ratio's more visual cousin. The article ["Efficiency Ratio Explained for Traders"](https://aligrithm.com/efficiency-ratio-explained-for-traders/) gave the primary noise measure. This article gives the second measure that ["Noise Is Not Volatility"](https://aligrithm.com/noise-is-not-volatility/) promised as a cross-check, the one you can see rather than only compute. Price density and the efficiency ratio answer the same question (per unit of motion, how much net direction) through different geometry, and when they disagree it tells you something specific about the kind of choppiness present. Using two measures that can disagree is the same discipline as the article ["Permutation Tests for Indicator Significance"](https://aligrithm.com/permutation-tests-for-indicator-significance/) applied to significance: one estimate can fool you, two estimates that agree are harder to fool. ## The formula and the geometry Price density is the sum of the daily ranges divided by the total range of the window. $$ \\text{PD} = \\frac{\\displaystyle\\sum\_{i=1}^{n} (\\text{High}\_i - \\text{Low}\_i)}{\\max(\\text{High}) - \\min(\\text{Low})} $$ The numerator adds up each bar's individual high-to-low range, the total amount of vertical ground the price covered bar by bar. The denominator is the full height of the box, the highest high minus the lowest low over the whole window. The ratio is how many times over the price covered the box's height. A market that trended straight up covers the box height roughly once, so its density is near 1\. A market that chopped up and down covered the box height many times over, so its density is large, 3, 4, 5 or more, depending on how many bars it took and how much each bar overlapped the others. Note that price density is not bounded between 0 and 1 the way the efficiency ratio is; it runs from roughly 1 (a perfect trend covers the box once) upward with no fixed ceiling (the more bars and the more overlap, the higher it climbs). To compare it across windows of different length you normalize by the number of bars, because a 20-day window mechanically sums more ranges than a 10-day window. The cleaner cross-window form divides the density by the bar count, giving an average overlap per bar that is comparable across window lengths. The geometric reading is the value. The efficiency ratio asks "how straight was the path from start to end". Price density asks "how much of the available vertical space did the price scribble over". They are two ways of measuring the same scribbling. A clean trend is a straight path (high efficiency ratio) that fills little of the box (low density). A chop is a tangled path (low efficiency ratio) that fills all of the box (high density). The two measures move in opposite directions by construction, which is exactly what makes their agreement a meaningful cross-check. ## Why a second measure earns its keep If price density and the efficiency ratio always agreed perfectly, the second one would be redundant. They mostly agree, and the cases where they diverge are diagnostic, which is the reason to carry both. Agreement case: clean trend. Low density (price fills little of the box) and high efficiency ratio (straight path). Both say trend. Deploy trend systems. No ambiguity. Agreement case: pure chop. High density (price fills the whole box) and low efficiency ratio (tangled path). Both say reversion. Deploy fade systems. No ambiguity. Divergence case: the gap-and-drift. Consider a market that gaps once and then sits quietly: one big jump fills a lot of the box height in a single bar, and the efficiency ratio reads high (the net change is large relative to the small subsequent path), but the density reads low for a different reason (few bars, little overlap). Here both happen to say "trend-ish" but for different reasons, and the low trade frequency warns you the trend is one event, not a sustained move, the gap-distortion failure the efficiency-ratio article flagged. The density confirms the move was a single bar rather than a process. Divergence case: the volatile grind. Consider a market that trends up but with enormous daily ranges, each bar covering huge ground while still closing higher. The efficiency ratio can read moderate-to-high (net direction survives), but the density reads high (the large overlapping ranges fill the box). This divergence flags a trend that is real but expensive to hold, because the wide bars mean deep adverse excursions inside the trend, the kind of thing the article ["MAE/MFE Analysis: Seeing What Net Profit Hides"](https://aligrithm.com/mae-mfe-analysis-seeing-what-net-profit-hides/) tells you to size stops around. The density caught the within-trend turbulence the efficiency ratio's endpoint view missed. The general rule: when the two measures agree, trust the regime call. When they diverge, the divergence names the specific feature (one-event move, or turbulent-but-directional) that a single measure would have hidden. Two cheap measures buy you a third piece of information their disagreement reveals. ## Reading density off a chart by eye Price density's practical advantage is that you can estimate it visually, which the efficiency ratio you cannot. Put a box around the window and look at how much candle-body-and-wick ink fills it. A box that is mostly white space with a single diagonal smear of candles is low density: a trend. A box that is densely packed with overlapping candles wall to wall is high density: a chop. A box with packed candles at one end and white space at the other is a transition: a range that broke into a trend or a trend that died into a range. This visual read is a fast first-pass regime classifier you can do on any chart without a calculation, and then confirm with the computed efficiency ratio. The article ["Why Indicator Histograms Matter"](https://aligrithm.com/why-indicator-histograms-matter/) argued for looking at the distribution rather than a single number; price density is the chart-level version of that instinct, looking at how the price distributed itself through the box. The visual read also makes the timeframe dependence from ["How to Choose the Right Timeframe for a Strategy"](https://aligrithm.com/how-to-choose-the-right-timeframe-for-a-strategy/) tangible. Box the same market at the hourly scale and the box is densely packed (high density, choppy); box it at the weekly scale and the box opens up (lower density, more directional). You see the noise-by-timeframe relationship directly as the box filling and emptying as you change the bar size. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Failure modes and how to handle them Price density has its own ways of misleading, distinct from the efficiency ratio's, which is part of why pairing them helps. Failure 1: window-length sensitivity. Because the numerator sums more ranges on a longer window, raw density is not comparable across window lengths. Always normalize by bar count before comparing, and never compare a raw 10-day density to a raw 30-day density. Failure 2: a single wide bar. One enormous range bar inflates both numerator (its own large range) and denominator (it sets the box height), and the net effect on density is ambiguous. As with the efficiency ratio, winsorize or cap extreme single-bar ranges before computing, or read the measure on a window long enough that one bar does not dominate. Failure 3: trending-with-overlap ambiguity. A market can trend and still have high density if every bar is wide and overlapping, the volatile-grind divergence above. Do not read high density as automatically meaning chop; read it together with the efficiency ratio, and let the divergence tell you it is a turbulent trend rather than a range. Failure 4: gap markets. Markets that gap (overnight equity, weekend crypto, weekly FX opens) break the bar-to-bar range assumption, because the gap is movement that no single bar's high-low range captures. For gap-prone markets, include the gap in the range (use true range, which counts the gap) so the density reflects the actual ground covered, the same adjustment the article ["Why ATR Normalization Is More Than a Volatility Trick"](https://aligrithm.com/why-atr-normalization-is-more-than-a-volatility-trick/) made for true range. ## Decision summary | Density | Efficiency ratio | Regime call | Action | | ------------ | ------------------------ | ------------------------- | ---------------------------------- | | Low (near 1) | High (above 0.30) | Clean trend | Trend systems | | High (3+) | Low (below 0.20) | Pure chop | Mean-reversion systems | | Low | High, but few bars moved | One-event move (gap) | Treat as single event, not a trend | | High | Moderate-to-high | Turbulent but directional | Trend with wide, MAE-sized stops | The two measures together give a richer regime call than either alone. Agreement confirms the regime; divergence names the specific feature that changes how you trade it. ## Visualizing price density ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_93-visual-0.png) ## KEY POINTS - Price density is the sum of the daily ranges divided by the total range of the window: how many times over the price covered the height of its own high-low box. - A clean trend covers the box height about once (density near 1, empty box). A chop covers it many times over (density 3, 4, 5 or more, full box). It is the efficiency ratio's visual cousin and moves in the opposite direction by construction. - Price density is not bounded 0 to 1 like the efficiency ratio; it runs from about 1 upward with no ceiling. Normalize by bar count before comparing across window lengths. - Carrying both measures earns its keep through their disagreements. When they agree, trust the regime call. When they diverge, the divergence names a specific feature one measure alone would hide. - Divergence case, gap-and-drift: high efficiency ratio but low density and few bars warns the move was one event, not a sustained trend. Divergence case, volatile grind: moderate efficiency ratio but high density flags a real but turbulent trend with deep within-trend excursions. - The practical advantage is visual: box a window and read how much candle ink fills it. Mostly white with a diagonal smear is a trend; packed wall to wall is a chop. A fast first-pass regime classifier you confirm with the computed efficiency ratio. - The visual read makes timeframe dependence tangible: the same market's box is packed at the hourly scale and opens up at the weekly scale, showing the noise-by-timeframe relationship directly. - Failure modes: window-length sensitivity (normalize by bar count), single wide bars (winsorize), trending-with-overlap ambiguity (read alongside the efficiency ratio), and gap markets (use true range so gaps count toward the ground covered). - The next article, ["The Difference Between Volatility Expansion and Directional Opportunity"](https://aligrithm.com/the-difference-between-volatility-expansion-and-directional-opportunity/), separates a related confusion: a market filling its box faster (volatility expanding) is not the same as a market offering more net direction. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Trading Systems and Methods - Perry Kaufman (Amazon)](https://www.amazon.com/s?k=Trading+Systems+and+Methods+Perry+Kaufman&ref=aligrithm.com) - [Cybernetic Trading Strategies - Murray Ruggiero (Amazon)](https://www.amazon.com/s?k=Cybernetic+Trading+Strategies+Murray+Ruggiero&ref=aligrithm.com) - [The Art of Currency Trading - Brent Donnelly (Amazon)](https://www.amazon.com/s?k=The+Art+of+Currency+Trading+Brent+Donnelly&ref=aligrithm.com) ### 4.7 How to Choose the Right Timeframe for a Strategy URL: https://aligrithm.com/how-to-choose-the-right-timeframe-for-a-strategy/ Last updated: 2026-07-27T13:24:55.000Z Take EURUSD and compute the efficiency ratio at four timeframes on the same calendar period: hourly bars over the prior month, daily bars over the prior quarter, weekly bars over the prior two years, monthly bars over the prior decade. The readings climb as the timeframe lengthens: roughly 0.10 hourly, 0.18 daily, 0.24 weekly, 0.30 monthly. The same market is choppy at the hour and efficient at the month, because short-horizon flow mean-reverts and long-horizon flow carries a macro lean. If you build a trend system and test it on hourly EURUSD, it fails and you conclude EURUSD does not trend. If you test it on monthly EURUSD, it works and you conclude it does. Both conclusions are about the timeframe, not the market, and choosing the timeframe is therefore choosing what the market is. This article isolates the horizon dimension that the previous articles kept flagging. ["Why One Indicator Should Not Be Used on Every Market"](https://aligrithm.com/why-one-indicator-should-not-be-used-on-every-market/) listed the resolution horizon as one of three things that change across markets; this article makes the horizon the subject. The timeframe is not a presentation choice or a personal preference. It is the single decision that most determines whether your strategy sees trend or chop, because the noise level itself is a function of timeframe, and the article ["Efficiency Ratio Explained for Traders"](https://aligrithm.com/efficiency-ratio-explained-for-traders/) already showed the efficiency ratio is a curve over window length rather than a point. ## The same market has different personalities at different timeframes A market is not one thing. It is a stack of overlapping behaviors at different timescales, and which one you trade depends entirely on the timeframe you sample. The source framing is direct: long-term trends are driven by economic factors and show up on weekly and daily charts, while short-term moves are investor reactions to news and show up on high-frequency data. These are different processes generating different statistics, and a single instrument hosts all of them simultaneously. The efficiency ratio makes this measurable. At very short timeframes (minutes, hours) almost every market is high-noise, because microstructure (bid-ask bounce, inventory management, order-flow oscillation) dominates and net direction is tiny relative to the frantic path. As the timeframe lengthens, microstructure averages out and whatever structural drift the market has begins to dominate the path, so the efficiency ratio rises. A market with real long-horizon drift (an equity index) keeps rising and becomes strongly efficient at monthly scale. A market with no drift (a range-bound currency) rises modestly and plateaus, never becoming strongly efficient at any timeframe. The practical consequence is that the timeframe choice is upstream of the trend-versus-reversion choice from the prior articles. You do not first decide "I am a trend trader" and then pick a timeframe. The timeframe partly determines whether trend is even available. A trend strategy needs a timeframe where the efficiency ratio is high; a reversion strategy needs one where it is low; and for a given market those are different timeframes. ## The three constraints that set the timeframe The right timeframe is the one that satisfies three constraints simultaneously. Miss any one and the strategy fails for a reason that looks like a different problem. Constraint 1: the noise constraint. The timeframe must put the efficiency ratio on the right side of your strategy's family. For a trend system, pick the timeframe where the market's efficiency ratio is highest (longer, usually) and above your trend threshold. For a reversion system, pick the timeframe where it is lowest (shorter, usually) and the microstructure mean-reversion is strongest. This is the efficiency-ratio-curve reading applied as a selection rule. Constraint 2: the cost constraint. Shorter timeframes mean more trades, and more trades mean more round-trip costs. A signal that is profitable gross at the hourly scale can be unprofitable net because the cost-per-trade is paid hundreds of times, the exact arithmetic the article ["Why Transaction Costs Should Be Added Before You Fall in Love"](https://aligrithm.com/why-transaction-costs-should-be-added-before-you-fall-in-love/) insisted on. The cost constraint pushes the timeframe longer; the noise constraint for a reversion system pushes it shorter. The viable timeframe is where both are satisfied, and for many reversion ideas on retail-cost instruments that window is empty, which is itself the answer. Constraint 3: the sample constraint. Longer timeframes mean fewer independent observations in any fixed calendar history. A monthly trend system over 20 years has only 240 monthly bars and a handful of independent trades, far below the thresholds the articles ["Trade-Count Thresholds for Backtest Reliability"](https://aligrithm.com/trade-count-thresholds-for-backtest-reliability/) and ["Why 30 Trades Is Not a Strategy"](https://aligrithm.com/why-30-trades-is-not-a-strategy/) set for a reliable backtest. The sample constraint pushes the timeframe shorter to accumulate trades, directly opposing the noise constraint for a trend system. The resolution is usually to trade a longer timeframe across a basket of markets so the trade count accumulates cross-sectionally rather than over time. These three constraints pull in different directions, and the right timeframe is the compromise that satisfies all three at once for your specific strategy and instrument. There is no universally correct timeframe; there is the one that resolves this three-way tension for your case. ## A worked timeframe selection Suppose you want to build a trend system on EURUSD. Walk the constraints. The noise constraint says go long: EURUSD's efficiency ratio is 0.10 hourly, 0.18 daily, 0.24 weekly, 0.30 monthly. Trend needs the ratio above roughly 0.30, so the noise constraint points at the monthly timeframe, maybe weekly. The sample constraint pushes back hard: monthly EURUSD over 20 years is 240 bars and perhaps 15 to 25 trend trades, well under 100\. That is not enough to trust, so on a single instrument the monthly timeframe fails the sample constraint. The resolution: trade the weekly timeframe (efficiency ratio 0.24, weak but workable trend) across a basket of 8 to 12 G10 and EM currency pairs, so the trade count accumulates across markets to several hundred even though each market contributes few trades. The cost constraint is comfortable at the weekly scale because trades are infrequent and FX spreads are tight. All three constraints are satisfied: enough noise for trend (barely), enough trades cross-sectionally, low enough cost. Now contrast a reversion system on the same EURUSD. The noise constraint points short (hourly or daily, where the efficiency ratio is lowest and reversion strongest). The sample constraint is easily satisfied because short timeframes generate many trades. The cost constraint becomes the binding one: at the hourly scale you pay the spread hundreds of times, so unless the spread is very tight relative to the reversion amplitude, the net edge vanishes. The viable timeframe is the shortest one where the average reversion still clears the round-trip cost with margin, often the daily rather than the hourly scale for retail-cost access. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## The timeframe-mismatch failure that masquerades as something else The dangerous failure is testing a strategy at the wrong timeframe and misdiagnosing the result as a different problem. Three common misdiagnoses. Misdiagnosis 1: "this market does not trend". Often it means you tested the trend system at a timeframe where the efficiency ratio is low. The market may trend strongly at a longer timeframe you did not test. The fix is to read the efficiency-ratio curve across timeframes before concluding anything about trendability. Misdiagnosis 2: "my reversion edge is weak". Often it means you tested at a timeframe where costs ate the edge, not where the reversion was absent. The gross edge may be strong; the net edge at that trade frequency is weak. The fix is to separate the gross signal from the cost drag and find the timeframe where the gross edge survives costs. Misdiagnosis 3: "the strategy is unstable". Often it means you tested at a long timeframe with too few trades, so the result is sample noise the article ["The Hill, the Spike, and the Cliff: Reading Optimization Surfaces"](https://aligrithm.com/the-hill-the-spike-and-the-cliff-reading-optimization-surfaces/) would flag as an unreliable peak. The instability is in the estimate, not the strategy. The fix is to add markets at the same timeframe to grow the cross-sectional sample. In all three the error is the same shape: a property of the timeframe got attributed to the market or the strategy. The discipline is to hold the strategy and market fixed, sweep the timeframe, and read the efficiency ratio, the cost drag, and the trade count at each one before drawing any conclusion. ## Decision summary | Strategy family | Noise constraint | Cost constraint | Sample constraint | Typical resolution | | ----------------------- | -------------------------------- | ------------------------ | -------------------------- | ---------------------------------------------- | | Trend / breakout | Push longer (need high ER) | Comfortable (few trades) | Push shorter (need trades) | Longer timeframe across a basket of markets | | Mean reversion | Push shorter (need low ER) | Binding (many trades) | Comfortable (many trades) | Shortest timeframe where reversion clears cost | | Mixed / regime-switched | Read ER curve, gate by timeframe | Moderate | Moderate | Medium timeframe with regime gate | The right timeframe is the compromise that satisfies all three constraints for your specific strategy and instrument. Sweep the timeframe and measure before concluding anything about the market. ## Visualizing the timeframe-noise relationship ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_92-visual-0.png) ## KEY POINTS - The timeframe is not a presentation choice; it is the decision that most determines whether your strategy sees trend or chop, because the noise level itself is a function of timeframe. - The efficiency ratio rises with timeframe: almost every market is high-noise at minutes and hours (microstructure dominates) and becomes more efficient at longer timeframes (structural drift dominates). A market with real drift keeps rising; a no-drift market plateaus. - The timeframe choice is upstream of the trend-versus-reversion choice. The timeframe partly determines whether trend is even available, so you do not pick a family first and a timeframe second. - Three constraints set the right timeframe. The noise constraint puts the efficiency ratio on the right side of your family's threshold. The cost constraint penalizes short timeframes that pay the spread many times. The sample constraint penalizes long timeframes that produce too few trades. - The three constraints pull in different directions. For a trend system, noise pushes longer and sample pushes shorter; the usual resolution is a longer timeframe across a basket of markets to accumulate trades cross-sectionally. - For a reversion system, noise pushes shorter and cost pushes back; the usual resolution is the shortest timeframe where the average reversion still clears the round-trip cost with margin. - Three misdiagnoses come from testing at the wrong timeframe: "this market does not trend" (tested where the efficiency ratio is low), "my reversion edge is weak" (tested where costs ate it), "the strategy is unstable" (tested with too few trades). Each attributes a timeframe property to the market or strategy. - The discipline: hold the strategy and market fixed, sweep the timeframe, and read the efficiency ratio, the cost drag, and the trade count at each one before drawing any conclusion. - The next article, ["Price Density: A Visual Way to Measure Market Choppiness"](https://aligrithm.com/price-density-a-visual-way-to-measure-market-choppiness/), adds a second, more visual noise measure that cross-checks the efficiency ratio and is easier to read directly off a chart. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Trading Systems and Methods - Perry Kaufman (Amazon)](https://www.amazon.com/s?k=Trading+Systems+and+Methods+Perry+Kaufman&ref=aligrithm.com) - [Cybernetic Trading Strategies - Murray Ruggiero (Amazon)](https://www.amazon.com/s?k=Cybernetic+Trading+Strategies+Murray+Ruggiero&ref=aligrithm.com) - [The Art of Currency Trading - Brent Donnelly (Amazon)](https://www.amazon.com/s?k=The+Art+of+Currency+Trading+Brent+Donnelly&ref=aligrithm.com) ### 4.6 Why One Indicator Should Not Be Used on Every Market URL: https://aligrithm.com/why-one-indicator-should-not-be-used-on-every-market/ Last updated: 2026-07-27T13:24:54.000Z A trader builds a 14-period RSI mean-reversion rule, buy below 30 and sell above 70, and it works beautifully on EURUSD. Encouraged, the trader applies the same 14-period RSI with the same 30/70 thresholds to crude oil, to BTC, and to the SPX, and reports the blended result to show "the indicator is robust across markets". The blended result is mediocre, and the trader concludes the edge was weak. The edge was not weak. The 14-period RSI at 30/70 is a high-noise mean-reversion tool, and it was deployed on markets with completely different noise structures where the same overbought reading means a different thing. The indicator did not generalize because indicators do not generalize across noise regimes; they are calibrated to one. This article is the operational consequence of the trend-reversion split from the previous two articles. ["High Noise Markets Are Mean-Reversion Markets"](https://aligrithm.com/high-noise-markets-are-mean-reversion-markets/) and ["Low Noise Markets Are Trend-Following Markets"](https://aligrithm.com/low-noise-markets-are-trend-following-markets/) established that the noise level determines which family works. This article goes one level finer: even within a family, a single indicator with a single parameter set encodes assumptions about noise, volatility, and horizon that hold on one market and break on another. The article "Why Works on All Markets Is Usually a Red Flag" warned about cross-market robustness claims; this is the indicator-level mechanism behind why those claims fail. ## An indicator is a bet on a specific market structure Every indicator has hidden parameters that are really assumptions about the market it was built for. A 14-period RSI assumes that 14 bars is the horizon over which overbought and oversold conditions resolve. A 70 threshold assumes a return distribution where the 70 level is reached often enough to trade but rarely enough to mean something. A 20-day breakout assumes that 20 days is long enough to filter noise but short enough to catch trends. Change the market and you change the horizon at which conditions resolve, the return distribution, and the noise level, which means the indicator's hidden assumptions no longer match. Consider the overbought threshold concretely. On a high-noise market like EURUSD, an RSI of 70 is a genuine extreme that tends to revert, because the market mean-reverts at that horizon. On a low-noise trending market like a crude-oil run, an RSI of 70 is not an extreme at all; it is what a healthy uptrend looks like, and it can stay above 70 for weeks while the trend continues, the exact behavior the source material on the Commodity Channel Index noted when prices stay overbought during strong moves. Selling at 70 on the trending market is selling the middle of an uptrend. Same indicator, same threshold, opposite meaning, because the noise structure flipped. The article ["Why You Should Test Long and Short Thresholds Separately"](https://aligrithm.com/why-you-should-test-long-and-short-thresholds-separately/) made the related point that even within one market the long and short thresholds carry different information; across markets the asymmetry is larger. ## The three things that change across markets When you move an indicator from one market to another, three structural properties change, and each one invalidates a different part of the indicator's calibration. Property 1: the noise level. This determines whether the indicator should be a trend tool or a reversion tool at all, and it is the largest of the three. An RSI used as a reversion signal assumes the market reverts; on a low-noise market it does not, and the signal inverts in meaning. The article ["Efficiency Ratio Explained for Traders"](https://aligrithm.com/efficiency-ratio-explained-for-traders/) gives the measurement; the indicator's family has to match the market's efficiency ratio. Property 2: the volatility level and shape. This determines the thresholds. A fixed 30/70 RSI band or a fixed-point breakout distance assumes a particular volatility; on a market with twice the volatility, the same band is reached twice as often and means half as much. The article ["Why ATR Normalization Is More Than a Volatility Trick"](https://aligrithm.com/why-atr-normalization-is-more-than-a-volatility-trick/) gave the fix: normalize the indicator's distance to volatility so the threshold means the same thing across markets. An indicator that is not volatility-normalized is silently recalibrated every time you change instruments. Property 3: the horizon at which conditions resolve. This determines the lookback. A 14-period setting assumes conditions resolve over 14 bars; on a faster-resolving market they resolve in 7 and the indicator is too slow, on a slower-resolving one they resolve in 28 and the indicator is too fast. The article ["How to Choose the Right Timeframe for a Strategy"](https://aligrithm.com/how-to-choose-the-right-timeframe-for-a-strategy/) handles this; the indicator's period has to match the market's resolution horizon. A single indicator with a single parameter set fixes all three of these to the values that suited the market it was built on. Deploy it elsewhere and at least one of the three is wrong. ## The normalization that makes an indicator portable You cannot make an indicator truly market-agnostic, but you can make it less market-specific by replacing fixed parameters with adaptive ones tied to the market's own structure. Three substitutions. Substitution 1: replace fixed thresholds with volatility-normalized or percentile thresholds. Instead of "sell at RSI 70", use "sell when the indicator is in the top decile of its own recent distribution". The percentile adapts to each market's distribution, so the threshold means the same thing (a genuine extreme for this market) everywhere. This is the distributional thinking the article ["Why Indicator Histograms Matter"](https://aligrithm.com/why-indicator-histograms-matter/) pushed: read the indicator against its own histogram, not against a universal constant. Substitution 2: replace a fixed lookback with a noise-adaptive one. The adaptive moving average from the efficiency-ratio article does this: its effective speed varies with the market's noise, so it is slow in choppy markets and fast in efficient ones without you changing a parameter. The general pattern is to make the period a function of the measured efficiency ratio or measured cycle length rather than a hardcoded number. Substitution 3: gate the indicator's family on the regime. Even a well-normalized reversion indicator should be switched off when the efficiency ratio says the market is trending, and a trend indicator switched off when it says the market is chopping. The gate, from the previous two articles, is what stops a reversion indicator from firing sell signals into an uptrend. $$ \\text{signal} = \\begin{cases} \\text{reversion indicator} & \\text{if } \\text{ER}\_t < \\theta\_{\\text{low}} \\\\ \\text{trend indicator} & \\text{if } \\text{ER}\_t > \\theta\_{\\text{high}} \\\\ \\text{no position} & \\text{otherwise} \\end{cases} $$ The signal switches family based on the efficiency ratio at time t: below a low threshold the market is noisy and the reversion indicator is active, above a high threshold the market is efficient and the trend indicator is active, and in the mixed band between them neither fires because that band is where both families overfit, the danger zone from ["Efficiency Ratio Explained for Traders"](https://aligrithm.com/efficiency-ratio-explained-for-traders/). This is not one indicator on every market; it is a regime-aware switch that picks the right indicator for the measured structure. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Why "it works on everything" should worry you An indicator that posts a positive result on every market in a diverse universe is suspicious, not reassuring. Markets with opposite noise structures need opposite-family indicators, so a single fixed indicator that profits on a strong trend market and a strong reversion market at the same time is either capturing something so weak it survives both (and will not survive costs) or has been tuned on a sample where the structures happened to align. The article "Why Works on All Markets Is Usually a Red Flag" framed the skeptical reading; the noise framework gives the mechanism. The honest expectation is that a reversion indicator works on the high-noise markets and fails on the low-noise ones, and a trend indicator does the reverse. A clean split by noise level is the signature of a real edge; uniform success across noise levels is the signature of an artifact. This also reframes what robustness means. Robustness is not "the same indicator with the same parameters works everywhere", which is the property the article ["The Difference Between Robustness and Optimization"](https://aligrithm.com/the-difference-between-robustness-and-optimization/) warned is usually a sign of an over-flexible fit. Robustness is "the indicator works on the markets whose structure it matches, and I can predict which those are from the efficiency ratio before testing". A robust system knows where it should not be deployed. ## Decision summary | Symptom | Cause | Fix | | ------------------------------------------------ | -------------------------------------------------- | ---------------------------------------------------- | | Indicator works on market A, fails on market B | Different noise level; wrong family for B | Gate family on the efficiency ratio | | Thresholds hit too often on a new market | Higher volatility; fixed thresholds miscalibrated | Use volatility-normalized or percentile thresholds | | Indicator too slow or too fast on a new market | Different resolution horizon; fixed lookback wrong | Make the period adaptive to noise or cycle length | | "Works on everything" blended result is mediocre | Opposite structures averaged together | Stop blending; split by noise level and match family | The fix is never "find the one indicator that works everywhere". It is to make the indicator's family, thresholds, and horizon adaptive to each market's measured structure, and to accept that the indicator belongs only on the markets it matches. ## Visualizing the portability problem ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_91-visual-0.png) ## KEY POINTS - An indicator is a bet on a specific market structure. Its period, thresholds, and family encode hidden assumptions about the noise level, volatility, and resolution horizon of the market it was built for. - The same overbought threshold means opposite things across noise regimes. RSI 70 on a high-noise market is a genuine reversal extreme; on a low-noise trending market it is what a healthy uptrend looks like and can persist for weeks. - Three properties change when you move an indicator across markets: the noise level (determines whether it should be a trend or reversion tool), the volatility (determines whether fixed thresholds are calibrated), and the resolution horizon (determines whether the lookback fits). A single fixed parameter set gets at least one wrong. - Make indicators more portable with three substitutions: percentile or volatility-normalized thresholds instead of fixed levels, a noise-adaptive lookback instead of a hardcoded period, and a regime gate that switches the indicator's family on the efficiency ratio. - The regime switch fires a reversion indicator when the efficiency ratio is low, a trend indicator when it is high, and nothing in the mixed band where both families overfit. - An indicator that works on every market in a diverse universe is suspicious, not reassuring, because opposite noise structures need opposite-family indicators. A clean split by noise level is the signature of a real edge; uniform success is the signature of an artifact. - Robustness is not the same indicator working everywhere. It is the indicator working on the markets whose structure it matches, with the ability to predict which those are from the efficiency ratio before testing. A robust system knows where it should not be deployed. - The next article, ["How to Choose the Right Timeframe for a Strategy"](https://aligrithm.com/how-to-choose-the-right-timeframe-for-a-strategy/), isolates the horizon dimension: how the resolution timescale interacts with noise and why measuring at the wrong timeframe produces false conclusions about the right one. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Trading Systems and Methods - Perry Kaufman (Amazon)](https://www.amazon.com/s?k=Trading+Systems+and+Methods+Perry+Kaufman&ref=aligrithm.com) - [Cybernetic Trading Strategies - Murray Ruggiero (Amazon)](https://www.amazon.com/s?k=Cybernetic+Trading+Strategies+Murray+Ruggiero&ref=aligrithm.com) - [The Art of Currency Trading - Brent Donnelly (Amazon)](https://www.amazon.com/s?k=The+Art+of+Currency+Trading+Brent+Donnelly&ref=aligrithm.com) ### 4.5 Low Noise Markets Are Trend-Following Markets URL: https://aligrithm.com/low-noise-markets-are-trend-following-markets/ Last updated: 2026-07-27T13:24:54.000Z Run a 20-day breakout system, the kind that buys when price exceeds the highest high of the prior 20 days and exits at the lowest low of the prior 10, on a market with an efficiency ratio of 0.34\. The entries catch the start of sustained moves, the exits give back a little at the end, the winners are large and the losers are small and frequent. The equity curve is lumpy but it climbs. Now run the identical system on EURUSD at an efficiency ratio of 0.18\. Every breakout triggers near a local extreme that immediately reverses, the system buys the high and sells the low over and over, and the equity curve grinds down through a thousand small whipsaws. Same rules, opposite result, and the only thing that changed was that the second market converts almost none of its motion into net direction. This is the mirror of the previous article. ["High Noise Markets Are Mean-Reversion Markets"](https://aligrithm.com/high-noise-markets-are-mean-reversion-markets/) explained why choppy instruments feed fade systems; this article explains why efficient instruments, the top of the trend-quality ranking from ["How to Rank Markets by Trend Quality"](https://aligrithm.com/how-to-rank-markets-by-trend-quality/), feed trend systems, and why a trend system deployed in a high-noise market is not mistuned but misplaced. The two articles are deliberately symmetric, because the strategic divide between trend and reversion is the noise axis and nothing else. ## Why low noise mechanically favors trend A trend system makes one bet: a move in one direction is more likely to continue than to reverse. That bet pays exactly when the market's motion accumulates in one direction, which is the definition of low noise. A high efficiency ratio says net change is large relative to total path, which means the price keeps going rather than coming back, which means continuation, which is trend. The efficiency ratio and the trend edge are the same structure measured twice, the exact symmetry to the reversion case. The flow mechanism is the opposite of dealer inventory management. Low-noise persistence comes from slow, directional, repeating order flow: a supply shock that keeps a commodity bid for months, a central-bank divergence that keeps a currency pair on one side for a year, an earnings-and-growth regime that keeps an equity index drifting up. The article ["Noise Is Not Volatility"](https://aligrithm.com/noise-is-not-volatility/) named this: a market dominated by slow directional flow generates persistence because the underlying imbalance repeats. The same buyer keeps coming back, so the same direction keeps printing, so the efficiency ratio stays high and a trend system stays paid. The autocorrelation signature is positive at the trend horizon: an up move is more likely to be followed by another up move. Where reversion markets show negative first-lag autocorrelation, trend markets show positive, and that positive autocorrelation is what a moving-average or breakout system monetizes. The article "Market Personality" listed positive autocorrelation as the trend-favoring personality dimension; it is the time-series footprint of low noise. ## The trend payoff is the reversion payoff inverted Where mean reversion is many small winners and occasional large losers, trend following is many small losers and occasional large winners. Low win rate, high average win to average loss, profit concentrated in a few big moves. The article ["Why Profit Factor Can Lie"](https://aligrithm.com/why-profit-factor-can-lie/) applies here in the opposite direction: a trend system's profit factor can look weak because the win rate is low, even though the strategy is sound, because the few large winners carry everything. Write out the asymmetry. Suppose the breakout wins 35% of the time for an average of 2.5 volatility units and loses 65% of the time for an average of 0.7 volatility units (the small loss being the cost of a failed breakout that gets cut quickly). $$ E = p\_{\\text{win}} \\cdot g - (1 - p\_{\\text{win}}) \\cdot l = 0.35 \\times 2.5 - 0.65 \\times 0.7 = 0.875 - 0.455 = 0.42 $$ The expectancy E is 0.875 minus 0.455, equal to 0.42 volatility units per trade, far larger than the reversion system's thin 0.01, but it arrives with a 65% loss rate that the trader has to sit through. The psychological and statistical problem is the same: most trades lose, and the strategy depends on not cutting the rare large winners short. The article ["MAE/MFE Analysis: Seeing What Net Profit Hides"](https://aligrithm.com/mae-mfe-analysis-seeing-what-net-profit-hides/) matters here for the opposite reason it mattered in reversion: you need the maximum favorable excursion distribution to confirm the winners are being allowed to run to their full size rather than clipped by a tight target. The low win rate also means a trend system needs many trades and a long sample before its edge is statistically visible, the trade-count problem from ["Trade-Count Thresholds for Backtest Reliability"](https://aligrithm.com/trade-count-thresholds-for-backtest-reliability/) and ["Why 30 Trades Is Not a Strategy"](https://aligrithm.com/why-30-trades-is-not-a-strategy/). A handful of trend trades on one market tells you almost nothing, because the profit is concentrated in the tail and the tail is rare. You need either many years on one efficient market or a basket of efficient markets to accumulate the trade count that makes the edge measurable. ## The failure mode: trend that turns into chop A low-noise market is not permanently low-noise. The efficiency ratio drifts, and the worst environment for a trend system is an efficient market that goes choppy. Every trend system is implicitly long continuation, so a market that stops continuing and starts oscillating is its slow-bleed scenario: a string of breakouts that immediately fail, each one a small loss, with no large winner to pay for them. This is the trend-follower's drawdown, and it is structural, not a bug to be optimized away. The defense is the same gate, run in the opposite direction. Compute the rolling efficiency ratio at the trend horizon and reduce or disable new breakout entries when it falls below the trend threshold (drifting toward 0.20, into chop territory). The article ["Efficiency Ratio Explained for Traders"](https://aligrithm.com/efficiency-ratio-explained-for-traders/) described the adaptive moving average that does this automatically by slowing down when the efficiency ratio drops; a discretionary or rules-based trend book does it by sizing down or stepping aside when the regime turns choppy. A trend system without this gate will keep firing breakouts into a range and donate its prior gains back through whipsaw. The harder problem is that you cannot eliminate the chop drawdown entirely, only manage its size. Trend following accepts a structural cost: it gives back a portion of profits during chop regimes to be positioned when the next sustained move starts, because the system cannot know in advance which breakout is the real one. The article ["Why Systems Work Until They Don't"](https://aligrithm.com/why-systems-work-until-they-dont/) framed this acceptance; the chop drawdown is the premium a trend follower pays to always be present when the large move arrives. Trying to optimize it away usually means tightening the system until it misses the large moves too, which destroys the only thing that made it work. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Which markets, and at which horizons Trend quality is horizon-dependent, mirror to noise. Equity indices are trend markets at multi-week to multi-month horizons (efficiency ratio around 0.30) and reversion markets at the 1-to-5-day scale. Trending commodities are trend markets during supply-shock regimes and revert to choppy seasonality otherwise. Crypto is a trend market in active breakout regimes (efficiency ratio swinging to 0.50) and a choppy market during multi-month consolidations (efficiency ratio dropping toward 0.05). The article "Market Personality" catalogued these horizon-and-regime dependencies; the operational instruction is to measure the efficiency ratio at the horizon you intend to trend-follow and accept that the same instrument may be a trend market this quarter and a chop market next quarter. The instruments that are trend markets across most horizons are the ones with structural drift: equity indices (the equity risk premium produces a persistent up-bias) and commodities or currencies under sustained macro imbalances. The instruments that are trend markets only episodically are the no-drift ones (most FX, range-bound commodities) where trend appears during specific regimes and disappears otherwise. The discipline matches the reversion case exactly: read the efficiency ratio at your horizon, deploy the family the regime supports, and gate on the regime drifting against you. ## Decision summary | Efficiency ratio at trend horizon | Market type | Action | | --------------------------------- | ------------------------------------------------------------------ | ------------------------------------------------------------------------------ | | Above 0.30 | Strong trend (equity indices, trending commodities, active crypto) | Deploy breakout or moving-average, let winners run to MFE, accept low win rate | | 0.25 to 0.30 | Weak / emerging trend | Trend with smaller size, tighter regime gate | | Below 0.20 | Chop regime | Disable breakouts, the move will reverse | The trend system belongs in the top rows of the efficiency ratio and nowhere in the bottom rows. The chop drawdown is a structural cost to be sized, not a defect to be optimized away. ## Visualizing the trend edge and its failure ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_90-visual-0.png) ## KEY POINTS - Low noise means motion that accumulates in one direction, which is the raw material a trend system converts to profit. A high efficiency ratio and a trend edge are the same structure measured two ways. - The flow mechanism is slow, directional, repeating order flow: supply shocks, central-bank divergences, growth regimes. The same buyer keeps returning, so the same direction keeps printing and the efficiency ratio stays high. - The autocorrelation signature is positive first-lag return autocorrelation at the trend horizon: an up move tends to be followed by another up move. It is the time-series footprint of low noise. - The payoff is the reversion payoff inverted: many small losers, few large winners, low win rate, high average-win-to-average-loss. A 35% win rate at 2.5 units against 0.7-unit losers nets 0.42 units expectancy, far larger than reversion's thin edge but with a 65% loss rate to sit through. - A trend system needs many trades and a long sample because the profit is concentrated in a rare tail. A handful of trend trades on one market tells you almost nothing; accumulate trade count across years or across a basket of efficient markets. - The defining failure mode is a low-noise market turning choppy: breakouts fire into a range, each a small loss, with no large winner to pay for them. This is the structural trend-follower drawdown. - The defense is the efficiency-ratio gate run in reverse: reduce or disable breakouts when the rolling ratio falls toward chop territory. The chop drawdown can be sized but not eliminated; it is the premium paid to always be present for the next large move. - Trend quality is horizon-dependent. Equity indices trend at multi-week horizons and revert at the 1-to-5-day scale; crypto trends in active regimes and chops in consolidations. Measure the efficiency ratio at the horizon you intend to trend-follow. - The next article, ["Why One Indicator Should Not Be Used on Every Market"](https://aligrithm.com/why-one-indicator-should-not-be-used-on-every-market/), builds on this trend-reversion split to show why a single indicator with a single parameter set cannot serve markets with different noise structures. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Trading Systems and Methods - Perry Kaufman (Amazon)](https://www.amazon.com/s?k=Trading+Systems+and+Methods+Perry+Kaufman&ref=aligrithm.com) - [Cybernetic Trading Strategies - Murray Ruggiero (Amazon)](https://www.amazon.com/s?k=Cybernetic+Trading+Strategies+Murray+Ruggiero&ref=aligrithm.com) - [The Art of Currency Trading - Brent Donnelly (Amazon)](https://www.amazon.com/s?k=The+Art+of+Currency+Trading+Brent+Donnelly&ref=aligrithm.com) ### 4.4 High Noise Markets Are Mean-Reversion Markets URL: https://aligrithm.com/high-noise-markets-are-mean-reversion-markets/ Last updated: 2026-07-27T13:24:53.000Z Run a simple fade system on EURUSD: when the 2-day return pushes more than one standard deviation above its recent mean, sell; when it pushes more than one standard deviation below, buy; exit at the mean. On EURUSD intraday and short-horizon daily data, where the efficiency ratio sits around 0.18, this system produces a high hit rate, many small winners, and a smooth equity curve, because the price keeps returning to its slowly moving center. Run the identical system on a trending crude-oil run where the efficiency ratio is 0.34, and it bleeds: every fade sells into a one-way move that does not come back, and the few winners cannot pay for the line of losers. Same logic, opposite result, and the only thing that changed was the noise level of the market. This is the constructive half of the noise framework. The article ["Noise Is Not Volatility"](https://aligrithm.com/noise-is-not-volatility/) established that high noise means motion that cancels out; this article explains why that cancellation is precisely the raw material a mean-reversion system converts into profit, and why deploying reversion logic anywhere else is a structural mistake rather than a parameter problem. The instruments at the bottom of the trend-quality ranking from ["How to Rank Markets by Trend Quality"](https://aligrithm.com/how-to-rank-markets-by-trend-quality/) are not bad markets. They are mean-reversion markets, and they are good at it for reasons rooted in who trades them and how. ## Why high noise mechanically favors reversion A mean-reversion system makes one bet: a move away from a center is more likely to reverse than to continue. That bet pays exactly when the market's motion cancels out, which is the definition of high noise. The link is not a coincidence; it is the same property measured two ways. A low efficiency ratio says net change is small relative to total path. Small net change relative to path means the price keeps coming back. Keeps coming back means reversion. The efficiency ratio and the reversion edge are reading the same underlying structure. The microstructure mechanism is concrete. In a market dominated by two-sided market-making, every aggressive order is absorbed by a liquidity provider who then wants to offload the inventory, pushing price back toward where it was. The article ["Noise Is Not Volatility"](https://aligrithm.com/noise-is-not-volatility/) named this: dealer inventory management generates mean-reverting micro-structure, which shows up as high noise at short horizons. G10 FX is the cleanest example because interbank liquidity is deep and two-sided and there is no structural drift; the price oscillates around a fair value that moves slowly, and a fade system harvests the oscillation. The autocorrelation signature confirms it. Mean-reverting markets show negative first-lag return autocorrelation at the reversion horizon: an up move is more likely to be followed by a down move. Trending markets show positive autocorrelation: an up move is more likely to be followed by another up move. The article "Market Personality" listed autocorrelation as a personality dimension; for a reversion system you want it negative at your horizon, and high noise and negative autocorrelation tend to travel together. ## What the reversion edge actually looks like The payoff profile of a reversion system in a high-noise market is the mirror image of a trend system's. Many small winners, occasional larger losers, high win rate, profit factor that depends on keeping the losers from running. The article ["Why Profit Factor Can Lie"](https://aligrithm.com/why-profit-factor-can-lie/) warned that a high win rate can mask a fat left tail; in mean reversion that tail is the move that does not revert, and it is the entire risk of the strategy. Here is the asymmetry written out. Suppose the fade wins 70% of the time for an average of 0.4 volatility units and loses 30% of the time for an average of 0.9 volatility units when the reversion fails. $$ E = p\_{\\text{win}} \\cdot g - (1 - p\_{\\text{win}}) \\cdot l = 0.70 \\times 0.4 - 0.30 \\times 0.9 = 0.28 - 0.27 = 0.01 $$ The expectancy E is win probability times average gain minus loss probability times average loss. With these numbers it is 0.28 minus 0.27, equal to 0.01 volatility units per trade, barely positive. The high win rate hides how thin the edge is, because the losers are more than twice the size of the winners. Push the loss size from 0.9 to 1.1 (one bad regime where reversions fail more often and run further) and the expectancy goes negative. This is why mean-reversion in high-noise markets lives or dies on the loss tail, not the win rate, and why the article ["MAE/MFE Analysis: Seeing What Net Profit Hides"](https://aligrithm.com/mae-mfe-analysis-seeing-what-net-profit-hides/) matters more here than almost anywhere: you have to see the maximum adverse excursion distribution to know whether your stop sits inside or outside the reversion's normal breathing room. ## The failure mode: noise that turns into trend A high-noise market is not permanently high-noise. The efficiency ratio drifts, and the worst thing that can happen to a fade system is a market that was choppy becoming trendy. Every reversion strategy has a hidden short position against trend: it is implicitly betting the move reverses, so a sustained move is its maximum-loss scenario. The article ["When a Stop Loss Improves Risk but Destroys Edge"](https://aligrithm.com/when-a-stop-loss-improves-risk-but-destroys-edge/) framed the trade-off; in mean reversion the stop is not optional, because without it a single regime transition from chop to trend can erase months of small winners. The defense is a regime gate built on the efficiency ratio. Compute the rolling efficiency ratio at the reversion horizon and disable new fades when it rises above the chop threshold (say, above 0.25 to 0.30, climbing toward trend territory). This is the same gating logic the article ["Efficiency Ratio Explained for Traders"](https://aligrithm.com/efficiency-ratio-explained-for-traders/) described, applied defensively: the fade system runs while the market is noisy and steps aside when the market starts converting motion into direction. A reversion system without this gate is a reversion system that will eventually meet a trend and give back everything. A second defense is the loss-tail stop sized off the maximum adverse excursion, not off a round number. If the reversion normally breathes 0.6 volatility units against you before reverting, a stop at 0.5 cuts winners; a stop at 2.0 lets the failed-reversion tail run. The MAE distribution tells you where the reversion's normal range ends and the regime-change tail begins, and the stop belongs at that boundary. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Which markets, and at which horizons High noise is horizon-dependent, so a market can be a reversion market at one horizon and a trend market at another. EURUSD is strongly mean-reverting intraday and at the daily scale (efficiency ratio around 0.18), but at the multi-month horizon a carry or macro lean produces some persistence. Equity indices are the reverse: trend-friendly at multi-week horizons but mean-reverting at the 1-to-5-day scale, which is why short-horizon equity reversion (buy the dip in an uptrend) is a real strategy even though the index is a trend market at longer horizons. The article "Market Personality" listed short-horizon mean reversion as a fit for both FX and equity indices for exactly this reason. The instruments that are mean-reversion markets across most horizons are the structurally two-sided, no-drift ones: G10 FX crosses, and to a degree the front end of deeply liquid government-bond futures. The instruments that are reversion markets only at short horizons are the trending ones (equity indices, trending commodities) where the long-horizon drift dominates but the short-horizon flow still mean-reverts. The discipline: measure the efficiency ratio at the horizon you intend to fade, not at some other horizon, the same horizon-matching point the article ["How to Choose the Right Timeframe for a Strategy"](https://aligrithm.com/how-to-choose-the-right-timeframe-for-a-strategy/) develops. ## Decision summary | Efficiency ratio at fade horizon | Market type | Action | | -------------------------------- | --------------------------------------------- | ------------------------------------------------------------ | | Below 0.20 | Strong reversion (G10 FX, choppy commodities) | Deploy fade or grid, gate on rising ER, stop at MAE boundary | | 0.20 to 0.25 | Weak reversion / mixed | Fade with tighter gate, smaller size | | Above 0.30 | Trend regime | Disable fades, the move will not revert | The reversion system belongs in the bottom rows of noise and the top rows nowhere. The gate and the MAE-sized stop are not refinements; they are what keep a chop-to-trend transition from ending the strategy. ## Visualizing the reversion edge and its failure ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_89-visual-0.png) ## KEY POINTS - High noise means motion that cancels out, which is exactly the raw material a mean-reversion system converts to profit. A low efficiency ratio and a reversion edge are the same structural property measured two ways. - The microstructure mechanism: two-sided market-making absorbs aggressive orders and offloads inventory, pushing price back toward fair value. This generates mean-reverting micro-structure and high short-horizon noise. G10 FX is the cleanest example. - The autocorrelation signature: reversion markets show negative first-lag return autocorrelation at the reversion horizon. High noise and negative autocorrelation travel together. - The payoff profile is many small winners, occasional larger losers, high win rate. The edge can be thin: a 70% win rate at 0.4 units against 0.9-unit losers nets only 0.01 units expectancy, and a slightly worse loss size turns it negative. - Mean reversion lives or dies on the loss tail, not the win rate. The maximum adverse excursion distribution tells you whether your stop sits inside or outside the reversion's normal breathing room. - The defining failure mode is a high-noise market turning trendy. Every reversion system is implicitly short trend; a sustained move is its maximum-loss scenario. - Two defenses: a regime gate that disables new fades when the rolling efficiency ratio climbs toward trend territory, and a stop sized off the MAE boundary rather than a round number. - High noise is horizon-dependent. EURUSD reverts intraday and daily but leans at multi-month horizons; equity indices trend at multi-week horizons but revert at the 1-to-5-day scale. Measure the efficiency ratio at the horizon you intend to fade. - The next article, ["Low Noise Markets Are Trend-Following Markets"](https://aligrithm.com/low-noise-markets-are-trend-following-markets/), develops the mirror image: why the top of the trend-quality ranking is where trend systems earn their structural edge. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Trading Systems and Methods - Perry Kaufman (Amazon)](https://www.amazon.com/s?k=Trading+Systems+and+Methods+Perry+Kaufman&ref=aligrithm.com) - [Cybernetic Trading Strategies - Murray Ruggiero (Amazon)](https://www.amazon.com/s?k=Cybernetic+Trading+Strategies+Murray+Ruggiero&ref=aligrithm.com) - [The Art of Currency Trading - Brent Donnelly (Amazon)](https://www.amazon.com/s?k=The+Art+of+Currency+Trading+Brent+Donnelly&ref=aligrithm.com) ### 4.3 How to Rank Markets by Trend Quality URL: https://aligrithm.com/how-to-rank-markets-by-trend-quality/ Last updated: 2026-07-27T13:24:52.000Z You have a trend-following system that works. You have 40 instruments you could run it on and capital for maybe 8 of them. The naive approach is to backtest the system on all 40 and keep the 8 with the best Sharpe. That approach is a trap, because the 8 best backtest Sharpes on a single history are partly the 8 luckiest, the in-sample selection bias the article ["The Problem with One Sample of Market History"](https://aligrithm.com/the-problem-with-one-sample-of-market-history/) laid out. The better approach is to rank the 40 instruments by a structural property that predicts whether a trend system can work there at all, independent of the specific backtest, and then run the system on the top-ranked ones. That structural property is trend quality, and the efficiency ratio is the spine of measuring it. The previous article, ["Efficiency Ratio Explained for Traders"](https://aligrithm.com/efficiency-ratio-explained-for-traders/), gave the per-market reading. This article turns it into a ranking: a reproducible procedure that orders a universe of instruments from most trend-friendly to least, so you allocate trend systems to the markets that structurally support them and mean-reversion systems to the markets that structurally support those. The ranking is the allocation decision made before the backtest, which is the right order, because the backtest then confirms a structural prior rather than discovering a lucky one. ## Trend quality is more than the efficiency ratio The efficiency ratio is the primary input, but a single number on a single window hides three things you need: stability across time, stability across windows, and the cost-adjusted reachability of the trend. A market can post a high average efficiency ratio that comes entirely from one 6-month run inside a 5-year sample; that is not a tradeable trend property, it is a single regime. So the ranking score combines four components. Component 1: median efficiency ratio at the trading horizon. Use the median across rolling windows, not the mean, because the median discounts the one-off regime spikes that inflate the mean, the same reason the article ["Why the Median Often Beats the Mean in Trading Features"](https://aligrithm.com/why-the-median-often-beats-the-mean-in-trading-features/) preferred it for features. A market with a median 22-day efficiency ratio of 0.30 has trend structure present most of the time; a market with mean 0.30 but median 0.15 has chop most of the time and a few violent runs. Component 2: efficiency-ratio persistence. The fraction of time the rolling efficiency ratio stays above the trend threshold. A market that is above 0.30 in 70% of windows offers a trend system far more tradeable opportunity than one above 0.30 in 25% of windows, even if their medians match. Persistence is what lets a trend system stay deployed rather than waiting in cash for rare regimes. Component 3: window monotonicity. Whether the efficiency ratio rises with window length (good, the trend lives at longer horizons and is real) or is flat or falling (suspicious, the apparent efficiency is a short-window gap artifact). A genuine trend market reads more efficient as you lengthen the window; a fake one does not. Component 4: cost-adjusted trend reach. The net directional move available per trade, in volatility units, minus the round-trip cost in the same units. A market can have beautiful trend structure and spreads so wide that the trend system cannot extract it, the exact problem the article ["Why Transaction Costs Should Be Added Before You Fall in Love"](https://aligrithm.com/why-transaction-costs-should-be-added-before-you-fall-in-love/) insisted on pricing first. Trend quality you cannot reach after costs is not trend quality. ## The ranking score Combine the four into a single comparable score. Normalize each component to a 0-to-1 scale across the universe, then weight. $$ \\text{TQ} = w\_1 \\,\\widetilde{\\text{ER}} + w\_2 \\,\\rho\_{\\text{ER}} + w\_3 \\,m\_{\\text{ER}} + w\_4 \\,\\frac{R\_{\\text{net}}}{R\_{\\text{net}} + c} $$ The score TQ blends the median efficiency ratio (ER with a tilde, for median), the persistence fraction (rho, the share of windows above threshold), the monotonicity term (m, scored 1 if the efficiency ratio rises with window length and 0 if flat or falling), and the cost-adjusted reach (net move R divided by net move plus cost c, which approaches 1 when the move dwarfs the cost and falls toward 0 when cost eats the move). A reasonable starting weight set is 0.4, 0.3, 0.1, 0.2, putting most weight on how trend-friendly the market is and how often, with a meaningful haircut for unreachable-after-cost edge. The weights are arbitrary starting points, not laws; the ranking is more robust than the exact weights, which is the property you want, because a ranking that flips when you nudge a weight from 0.4 to 0.35 is a fragile ranking the article ["Parameter Stability Beats Best Parameter"](https://aligrithm.com/parameter-stability-beats-best-parameter/) would reject. ## A worked ranking Score a small universe at a 22-day horizon. The efficiency-ratio numbers tie to the cross-market readings from the personality article, and the persistence and cost figures are representative. | Instrument | Median ER | Persistence above 0.30 | ER rises with window | Cost reach | TQ score | Rank | | --------------------------- | --------- | ---------------------- | -------------------- | ---------- | -------- | ---- | | Crude oil (trending regime) | 0.34 | 0.55 | yes | 0.92 | 0.78 | 1 | | SPX index | 0.30 | 0.50 | yes | 0.98 | 0.74 | 2 | | BTC | 0.28 | 0.45 | yes | 0.80 | 0.64 | 3 | | Gold | 0.22 | 0.30 | yes | 0.95 | 0.52 | 4 | | EURUSD | 0.18 | 0.18 | weak | 0.97 | 0.40 | 5 | | Natural gas | 0.12 | 0.10 | no | 0.70 | 0.24 | 6 | The ranking sorts the universe into a trend-system allocation. The top three (crude in a trending regime, SPX, BTC) are where a trend system should run. The bottom two (EURUSD, natural gas) are where it should not, and where a mean-reversion system should run instead. Gold straddles, which matches its two-mode personality. Note that BTC's high efficiency ratio is partly discounted by its weaker cost reach (wide crypto spreads), which is the cost adjustment doing its job, dropping it below SPX despite a comparable raw efficiency ratio, the same cost-family point the article "Market Personality" made about crypto's 50-to-200-basis-point round trips. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Why ranking beats absolute thresholds A fixed threshold ("trade trend systems only where the efficiency ratio exceeds 0.30") fails in two directions. In a universe where everything is choppy (an all-FX book), it leaves you with nothing to trade even though some FX crosses are relatively more trend-friendly than others and a trend overlay on the best of them still beats cash. In a universe where everything is trendy (a commodity-heavy book during a macro shock), it tells you to trade everything and gives no basis for choosing the 8 you have capital for. A ranking adapts to the universe: it always identifies the relatively-best trend candidates and the relatively-best mean-reversion candidates, which is the decision you actually face. This is the same reason cross-sectional ranking beats absolute cutoffs in factor work; the relative ordering is more stable than the absolute level. The ranking also gives you the mean-reversion book for free. Invert it. The bottom of the trend-quality ranking is the top of the mean-reversion-quality ranking, because high noise is exactly what fade and grid systems need, the point the article ["Why Grid Systems Need Noise, Not Just Volatility"](https://aligrithm.com/why-grid-systems-need-noise-not-just-volatility/) develops. One ranking, two allocations: trend systems to the top, mean-reversion systems to the bottom. ## Rebuild the ranking on a schedule Trend quality is a structural property, but structures drift, slowly, the way the article ["Slow Wandering: The Most Dangerous Type of Market Change"](https://aligrithm.com/slow-wandering-the-most-dangerous-type-of-market-change/) described. A market that ranked first on trend quality in a 3-year supply-shock regime can fall to mid-pack when the shock resolves and the commodity goes back to range-bound seasonality. The discipline is to recompute the ranking on a rolling basis (quarterly is a reasonable cadence) and let allocations follow the ranking, with hysteresis so you are not churning instruments on small rank changes. An instrument has to move several ranks, not one, before you reallocate, because rank noise on a finite sample is real and the small-sample fragility from ["Trade-Count Thresholds for Backtest Reliability"](https://aligrithm.com/trade-count-thresholds-for-backtest-reliability/) applies to the ranking inputs too. The forward-looking caveat: the ranking predicts trend quality assuming the recent structural regime persists. It is a conditional bet, not a guarantee. A market that ranks high because of a regime that is about to end will disappoint, and there is no way to rank around a regime change you cannot see coming. The ranking reduces selection bias and aligns strategy to structure; it does not see the future, and pretending otherwise is the inductive overreach the article ["Induction in Trading: Why Past Patterns Are Always Uncertain"](https://aligrithm.com/induction-in-trading-why-past-patterns-are-always-uncertain/) warned about. ## Decision summary | Trend-quality rank | Allocation | Rationale | | ------------------ | ---------------------------------- | ------------------------------------------------------ | | Top tier | Trend, breakout, momentum systems | High median ER, high persistence, reachable after cost | | Middle tier | Regime-gated or dual systems | Mixed ER, switch family with the regime | | Bottom tier | Mean-reversion, fade, grid systems | High noise is the fuel for these families | The single ranking produces the whole allocation map: trend systems to the top, regime-gated systems to the middle, mean-reversion systems to the bottom, recomputed quarterly with hysteresis. ## Visualizing the trend-quality ranking ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_88-visual-0.png) ## KEY POINTS - Ranking markets by trend quality beats keeping the 8 best backtest Sharpes from a universe of 40, because the best backtest Sharpes are partly the luckiest (in-sample selection bias). Rank by structure first, confirm with backtest second. - Trend quality combines four components: median efficiency ratio at the trading horizon (use median, not mean, to discount one-off regime spikes), persistence (fraction of time above the trend threshold), window monotonicity (the efficiency ratio should rise with window length for a real trend), and cost-adjusted reach (net move per trade minus round-trip cost). - The composite score weights these roughly 0.4 / 0.3 / 0.1 / 0.2\. The weights are arbitrary starting points; the ranking is more robust than the exact weights, which is the property you want. - A worked universe ranks crude (trending) and SPX at the top, BTC third (its high efficiency ratio discounted by wide crypto spreads), gold straddling, EURUSD and natural gas at the bottom. - Ranking beats fixed thresholds because it adapts to the universe: in an all-choppy book it still finds the relatively-best trend candidate; in an all-trendy book it still discriminates among them for limited capital. - The ranking gives the mean-reversion book for free: invert it. The bottom of trend quality is the top of mean-reversion quality, because high noise is fade and grid fuel. - Recompute the ranking quarterly with hysteresis so small rank changes do not churn allocations. Rank noise on finite samples is real; require a move of several ranks before reallocating. - The ranking is a conditional bet that the recent structural regime persists. It reduces selection bias and aligns strategy to structure; it does not predict regime changes, and pretending otherwise is inductive overreach. - The next article, ["High Noise Markets Are Mean-Reversion Markets"](https://aligrithm.com/high-noise-markets-are-mean-reversion-markets/), develops the bottom of this ranking: why the choppy instruments are precisely where fade and reversion systems earn their edge. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Trading Systems and Methods - Perry Kaufman (Amazon)](https://www.amazon.com/s?k=Trading+Systems+and+Methods+Perry+Kaufman&ref=aligrithm.com) - [Cybernetic Trading Strategies - Murray Ruggiero (Amazon)](https://www.amazon.com/s?k=Cybernetic+Trading+Strategies+Murray+Ruggiero&ref=aligrithm.com) - [The Art of Currency Trading - Brent Donnelly (Amazon)](https://www.amazon.com/s?k=The+Art+of+Currency+Trading+Brent+Donnelly&ref=aligrithm.com) ### 4.2 Efficiency Ratio Explained for Traders URL: https://aligrithm.com/efficiency-ratio-explained-for-traders/ Last updated: 2026-07-27T13:24:52.000Z Take a 10-day window of closing prices: 100, 101, 100, 102, 101, 103, 102, 104, 103, 105\. The price ended at 105 and started at 100, so the net change is 5 points. Now add up every day's absolute move: from 100 to 101 is 1, 101 to 100 is 1, 100 to 102 is 2, 102 to 101 is 1, 101 to 103 is 2, 103 to 102 is 1, 102 to 104 is 2, 104 to 103 is 1, 103 to 105 is 2\. That sums to 13 points of total travel. Divide the net change by the total travel: 5 divided by 13 equals 0.385\. That number is the efficiency ratio for this window, and it says the market converted 38.5% of its motion into net direction. The rest, 61.5%, was noise that cancelled out. The efficiency ratio is the primary tool for measuring the noise the previous article, ["Noise Is Not Volatility"](https://aligrithm.com/noise-is-not-volatility/), isolated. It answers one question with one number between 0 and 1: of all the distance this market travelled, how much actually went somewhere. The article ["Market Personality: Why Gold, FX, Crypto, and Equities Need Different Systems"](https://aligrithm.com/market-personality-why-gold-fx-crypto-and-equities-need-different-systems/) used it to fingerprint instruments; this article is the full mechanics, the window-selection problem, the interpretation thresholds, and the failure modes, so you can compute it correctly and not fool yourself with it. ## The formula and what each part means Two ingredients: net change in the numerator, total path in the denominator. $$ \\text{ER}\_W = \\frac{|P\_t - P\_{t-W}|}{\\displaystyle\\sum\_{i=t-W+1}^{t} |P\_i - P\_{i-1}|} $$ The numerator is the absolute difference between the price now and the price W bars ago. It does not care about the route; it only cares about the two endpoints. The denominator is the sum of the absolute bar-to-bar moves across the whole window, the actual route length. Because the straight-line distance between two points is always less than or equal to the length of any wiggling path between them, the numerator can never exceed the denominator. The ratio is therefore trapped between 0 and 1\. A value of 1 means the path was a perfect straight line; every bar moved in the same direction with no backtracking. A value of 0 means the price returned exactly to where it started; all motion cancelled. Note that the denominator is closely related to the kind of motion an average true range captures, which is why the efficiency ratio is sometimes described as net change divided by summed range. The article ["Why ATR Normalization Is More Than a Volatility Trick"](https://aligrithm.com/why-atr-normalization-is-more-than-a-volatility-trick/) used range to scale volatility; the efficiency ratio reuses the same raw material but puts net change on top of it, turning a volatility quantity into a directionality quantity. ## The window length is the whole game The efficiency ratio is not one number; it is a curve indexed by the window W. The same market reads as high-efficiency on one window and low-efficiency on another, and this is not a bug, it is the signal. Compute the efficiency ratio on EURUSD at three windows. At a 5-day window it might read 0.12 (over a week the price chops and goes nowhere). At a 22-day window it reads around 0.18 (over a month a little net drift survives the chop). At a 60-day window it might read 0.25 (over a quarter a macro lean accumulates). The reading climbs with the window because short-horizon flow is mean-reverting and long-horizon flow carries a slow directional component. Now do the same on a trending commodity: 5-day 0.30, 22-day 0.45, 60-day 0.55\. Higher at every window, and rising faster, because the persistence is present even at short horizons. The practical rule: the efficiency ratio must be computed at the horizon your strategy actually trades. A system that holds positions for two to three weeks should read the efficiency ratio on a window matched to that holding period, not on a 5-day window that reflects microstructure the system never touches. Reading it on the wrong window is the same category of error the article ["How to Choose the Right Timeframe for a Strategy"](https://aligrithm.com/how-to-choose-the-right-timeframe-for-a-strategy/) addresses: you measured the wrong horizon's property and concluded something false about the horizon you trade. A second rule: report the efficiency ratio as a small profile, not a point. Three windows (short, your-horizon, long) tell you whether the market's directionality is concentrated at your horizon or borrowed from a longer one. A market that is efficient only at 60 days but choppy at your 10-day horizon is not a trend market for you, no matter how clean the quarterly chart looks. ## Interpretation thresholds, with the usual caveats The number is continuous, but trading decisions are discrete, so you need thresholds. These are approximate and instrument-dependent; treat them as starting calibrations, not constants. Below roughly 0.20: high noise. Mean-reversion and fade strategies have their best environment here. Trend systems struggle because the net direction is small relative to the chop they must endure. Most G10 FX sits here at short and medium horizons. Roughly 0.20 to 0.30: mixed. Neither family has a structural advantage; this is where parameter sensitivity is highest and where most overfitting happens because a system can be tuned to look good on either side. Treat results in this band with the skepticism the article ["The Hill, the Spike, and the Cliff: Reading Optimization Surfaces"](https://aligrithm.com/the-hill-the-spike-and-the-cliff-reading-optimization-surfaces/) recommends for fragile peaks. Above roughly 0.30: low noise, trend-friendly. Breakout and moving-average systems have their structural environment. Sustained equity-index uptrends, commodity supply-shock runs, and crypto breakout regimes live here. The article "Market Personality" put SPX around 0.30 and BTC around 0.28 at a 22-day window, which is why both support multi-week trend frameworks. These bands are why the efficiency ratio is a strategy-selection tool before it is a signal. You read the regime first and pick the family second, the sequencing the article "Market Personality" insisted on. ## A worked cross-market table Compute the 22-day efficiency ratio across a representative set, the same instruments the personality article fingerprinted, so the numbers tie out. | Instrument | 22-day ER (approx) | Read | | ----------- | ------------------ | ------------------------------------------------- | | SPX index | 0.30 | Trend-friendly, long-bias | | BTC | 0.28 | Trend-friendly in active regimes, regime-variable | | Gold | 0.22 | Mixed, two distinct modes | | EURUSD | 0.18 | High noise, mean-reversion territory | | Natural gas | 0.12 | Very high noise, seasonal | The pattern: equity indices and crypto sit in trend territory, gold straddles, and FX and natural gas sit in chop territory. A trend system deployed across this whole set without reading the column will earn its edge on SPX and BTC and donate it back on natural gas and EURUSD, producing a blended result that looks mediocre and tells you nothing, the exact "run it on everything" trap from "Why Works on All Markets Is Usually a Red Flag". ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Using the efficiency ratio as a live filter The efficiency ratio is not only a research-time classifier; it can gate live signals. The mechanism: compute the rolling efficiency ratio at your horizon and only allow trend entries when it is above your trend threshold, only allow fade entries when it is below your chop threshold. This is the original purpose the indicator was built for, adapting trend speed to noise, and it generalizes to a regime gate on any strategy. The adaptive moving average it underpins varies its own speed with the efficiency ratio: it scales a smoothing constant so the average moves fast when the efficiency ratio is high (a clean trend worth tracking closely) and slow when the efficiency ratio is low (a chop worth ignoring). $$ \\alpha\_t = \\big\[\\, \\text{ER}\_t \\cdot (\\alpha\_{\\text{fast}} - \\alpha\_{\\text{slow}}) + \\alpha\_{\\text{slow}} \\,\\big\]^2 $$ The smoothing constant alpha runs between a fast value (used when the efficiency ratio is near 1) and a slow value (used when the efficiency ratio is near 0). When the market is efficient, alpha is large and the average hugs price; when the market is noisy, alpha shrinks and the average flattens, refusing to chase whipsaws. The squaring sharpens the response so that mid-range efficiency readings still produce a slow average, which is the conservative choice because the mixed band is where false trends live. This is one concrete use; the broader point is that the efficiency ratio can be wired into entry permission, position sizing, or smoothing speed, all driven by the same noise measurement. ## Failure modes that will fool you Four ways the efficiency ratio misleads if you are careless. Failure 1: gap distortion. A single overnight gap inflates both numerator and denominator but can spike the ratio toward 1 on a short window, falsely signalling a clean trend when the move was one discontinuous jump. Cap or winsorize extreme single-bar moves before computing, or compute on a horizon long enough that one gap does not dominate. Failure 2: short-window instability. On a 5-day window the ratio is jumpy and the small-sample fragility the article ["Trade-Count Thresholds for Backtest Reliability"](https://aligrithm.com/trade-count-thresholds-for-backtest-reliability/) described applies in full; a single window of five bars is five data points. Do not make regime decisions off a single short-window reading; require persistence across several windows or use a longer horizon. Failure 3: trend-direction blindness. The numerator uses absolute value, so the efficiency ratio is direction-agnostic. A clean downtrend and a clean uptrend both read high. The efficiency ratio tells you a trend exists, not which way it points; you still need a separate direction input. Pairing it with a slope sign is the standard fix. Failure 4: regime non-stationarity. The efficiency ratio drifts with the market's personality, which the article ["Slow Wandering: The Most Dangerous Type of Market Change"](https://aligrithm.com/slow-wandering-the-most-dangerous-type-of-market-change/) warned moves slowly but inexorably. A threshold calibrated on 2015 data may be miscalibrated by 2025\. Recompute the instrument's efficiency-ratio distribution periodically and re-set the thresholds against the current distribution, not a frozen one. ## Decision summary | Efficiency ratio at your horizon | Regime | Strategy family | Sizing note | | -------------------------------- | ---------- | ---------------------------------- | --------------------------------------------- | | Below 0.20 | High noise | Mean-reversion, fade, grid | Scale to volatility, expect many small trades | | 0.20 to 0.30 | Mixed | Neither structurally favored | High overfit risk, demand robustness | | Above 0.30 | Low noise | Breakout, moving-average, momentum | Scale to volatility, expect few large trades | The table is the operational core: read the ratio at the horizon you trade, map it to the regime, pick the family the regime favors, and only then tune parameters. ## Visualizing the efficiency ratio ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_87-visual-0.png) ## KEY POINTS - The efficiency ratio is net directional change divided by total absolute path length over a window, trapped between 0 and 1\. It measures what fraction of a market's motion converted into net direction; the rest is noise. - Worked example: a 10-day window ending 5 points higher with 13 points of total travel reads 5 over 13, equal to 0.385\. The market converted 38.5% of its motion into direction. - The window length is not a detail; it is the signal. The same market reads choppy on a 5-day window and efficient on a 60-day window. Compute the ratio at the horizon your strategy actually trades. - Report a small profile (short, your-horizon, long) rather than a point. A market efficient only at long horizons but choppy at your horizon is not a trend market for you. - Threshold calibration: below 0.20 is high noise (mean-reversion territory), 0.20 to 0.30 is mixed (highest overfit risk), above 0.30 is low noise (trend territory). Approximate and instrument-dependent. - Cross-market 22-day readings tie out with the personality fingerprint: SPX 0.30, BTC 0.28, Gold 0.22, EURUSD 0.18, natural gas 0.12\. Equities and crypto sit in trend territory, FX and natural gas in chop. - The efficiency ratio is a strategy-selection tool before it is a signal. Read the regime first, pick the family second, tune parameters last. - It can gate live signals and drive an adaptive moving average whose smoothing constant scales with the ratio: fast average in efficient markets, flat average in noisy ones, with squaring to stay conservative in the mixed band. - Four failure modes: single-gap distortion (winsorize), short-window instability (require persistence), direction-blindness (the absolute value hides trend direction, pair with slope sign), and threshold non-stationarity (recompute the distribution periodically). - The next article, ["How to Rank Markets by Trend Quality"](https://aligrithm.com/how-to-rank-markets-by-trend-quality/), turns the efficiency ratio from a per-market reading into a cross-market ranking system for allocating strategies to instruments. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Trading Systems and Methods - Perry Kaufman (Amazon)](https://www.amazon.com/s?k=Trading+Systems+and+Methods+Perry+Kaufman&ref=aligrithm.com) - [Cybernetic Trading Strategies - Murray Ruggiero (Amazon)](https://www.amazon.com/s?k=Cybernetic+Trading+Strategies+Murray+Ruggiero&ref=aligrithm.com) - [The Art of Currency Trading - Brent Donnelly (Amazon)](https://www.amazon.com/s?k=The+Art+of+Currency+Trading+Brent+Donnelly&ref=aligrithm.com) ### 4.1 Noise Is Not Volatility URL: https://aligrithm.com/noise-is-not-volatility/ Last updated: 2026-07-27T13:24:51.000Z Take two markets over the same quarter. Market A is EURUSD: it closes the quarter 1.2% above where it opened, but the daily candles wander up and down so much that the price travelled a total absolute distance of roughly 18% to get there. Market B is a trending crude-oil run: it closes the quarter 14% higher and travelled a total absolute distance of 22% along the way. Measure each one with a standard 20-day annualized standard deviation and you get similar numbers, around 14% to 16% annualized. The volatility readings say these two markets are siblings. They are not. One delivered almost no net direction per unit of path travelled, and the other delivered direction efficiently. A trend system tuned to one will bleed on the other, and the volatility number told you nothing about which is which. This is the distinction the rest of Pillar 4 is built on. Volatility measures how far price moves. Noise measures how much of that movement cancels itself out before producing any net direction. They are different quantities, they are driven by different mechanisms, and conflating them is the single most common reason a strategy that backtests well on one instrument dies on another that looks statistically identical. The article ["Market Personality: Why Gold, FX, Crypto, and Equities Need Different Systems"](https://aligrithm.com/market-personality-why-gold-fx-crypto-and-equities-need-different-systems/) used the efficiency ratio to fingerprint markets; this article isolates the underlying property that the efficiency ratio is measuring, and explains why you cannot read it off a volatility chart. A drunk walking home covers a lot of ground. He staggers left, staggers right, lurches forward, drifts back. Measure his total foot-distance over an hour and it is large. Measure how far he ended up from where he started and it is small. A sober person walking the same hour covers less total foot-distance but ends up much further from the start. The drunk is high-noise, high-path, low-net. The sober walker is low-noise, lower-path, high-net. Volatility is the foot-distance. Noise is the staggering. Net direction is what you can actually trade. ## Two markets, identical volatility, opposite personality Here is the worked version, because the abstract claim does not land without numbers. Construct two synthetic 100-day price paths, both starting at 100, both with daily moves of fixed magnitude 1.0 (so the day-to-day "volatility" is identical by construction; every single day moves exactly 1 point). Path A alternates direction with high probability: up, down, up, down, with occasional runs. Over 100 days it ends at 104\. Path B keeps direction with high probability: long runs of up days broken by short pullbacks. Over 100 days it ends at 140. Both paths have the same per-day absolute move (1.0), so the same realized volatility. The total path length is the same too: 100 days times 1.0 equals 100 points of absolute travel for each. The difference is entirely in how that 100 points of travel converted to net change. Path A converted 4 points net. Path B converted 40 points net. Same volatility, same total motion, a tenfold difference in directional yield. The ratio of net change to total path length captures exactly this. For path A it is 4 divided by 100, equal to 0.04\. For path B it is 40 divided by 100, equal to 0.40. $$ \\text{Noise-adjusted yield} = \\frac{|P\_{\\text{end}} - P\_{\\text{start}}|}{\\sum\_{i=1}^{N} |P\_i - P\_{i-1}|} $$ The numerator is net directional change (where price ended versus where it started). The denominator is the sum of every individual day's absolute move, the total distance the price actually walked. A value near 1 means almost every step went in the same direction (a straight line, low noise). A value near 0 means the steps cancelled out (a drunk's walk, high noise). Volatility lives in the denominator alone; it has no idea what the numerator is doing. That is why two markets with the same denominator can have wildly different numerators, and why you cannot back out noise from a volatility reading. ## Why the distinction is not academic Three operational consequences follow directly, and each one costs money if ignored. Consequence 1: trend systems harvest the numerator, not the denominator. A breakout or moving-average system profits when net directional change is large relative to the chop it has to sit through. Feed it a high-volatility, high-noise market (large denominator, small numerator) and it generates frequent signals that whipsaw, because the volatility triggers entries while the noise reverses them before they pay. The article ["Why ATR Normalization Is More Than a Volatility Trick"](https://aligrithm.com/why-atr-normalization-is-more-than-a-volatility-trick/) handled the scaling problem; the noise problem is upstream of scaling and cannot be normalized away. Consequence 2: mean-reversion systems harvest the denominator against the numerator. A fade or grid system profits when price oscillates a lot (large denominator) without committing to a direction (small numerator). High noise is the fuel. Feed a mean-reversion system a low-noise trending market and it sells every rip and buys every dip into a one-way move, accumulating a portfolio of losers that never revert. The article ["Why Grid Systems Need Noise, Not Just Volatility"](https://aligrithm.com/why-grid-systems-need-noise-not-just-volatility/) develops this; the precondition is the noise measurement, not the volatility measurement. Consequence 3: volatility-targeting position sizing does not adjust for noise. If you size positions to a constant volatility target, you treat path A and path B as equally risky and equally tradeable. They are equally risky in the variance sense and completely different in the edge sense. Volatility targeting will hand the same capital to a market where your trend edge is zero as to one where it is strong, and the blended result will disappoint for reasons the risk report cannot explain. ## What drives noise, and why it is more stable than volatility Volatility is famously non-stationary; the article ["Why Volatility Is More Non-Stationary Than Trend"](https://aligrithm.com/why-volatility-is-more-non-stationary-than-trend/) documented how it clusters, spikes, and mean-reverts on its own clock. Noise behaves differently. The amount of directional persistence a market exhibits is tied to its participant structure and its information flow, and those change on a slower clock than volatility does. Trade-flow structure sets the floor on noise. A market dominated by two-sided market-making and short-horizon liquidity provision (G10 FX intraday, large-cap equities at the tightest spreads) mechanically generates mean-reverting micro-structure: every aggressive buy is met by an inventory-adjusting sell, and the price oscillates around a slowly moving fair value. That produces high noise at short horizons regardless of how volatile the day is. A market dominated by slow directional flow (a commodity in a supply shock, a currency under a sustained central-bank divergence) generates persistence: the same direction repeats because the underlying imbalance repeats. That produces low noise even on quiet days. Information arrival sets the spikes. News does not change a market's noise character; it changes its volatility. A surprise print can double the daily range without changing whether the subsequent moves persist or cancel. This is why you can have a high-volatility day inside a high-noise regime (a violent chop) and a low-volatility day inside a low-noise regime (a quiet grind higher). The two axes are close to independent. The practical upshot: you can characterize a market's noise level over a few years and trust it to drift slowly, the way the article ["Slow Wandering: The Most Dangerous Type of Market Change"](https://aligrithm.com/slow-wandering-the-most-dangerous-type-of-market-change/) framed slow regime drift. You cannot do the same with volatility, which can regime-shift in a week. Noise is a structural property; volatility is a state variable. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Measuring noise three ways The yield ratio above is the most common noise measure, but it is one of three, and they agree often enough to cross-check each other. Measure 1: the efficiency-style ratio (net change over total path length). Bounded between 0 and 1, cheap to compute, sensitive to the lookback window. This is the workhorse, developed in full in the next article, ["Efficiency Ratio Explained for Traders"](https://aligrithm.com/efficiency-ratio-explained-for-traders/). Measure 2: price density (how much of the high-low box the price actually fills). A choppy market paints the whole box; a trending market leaves most of it empty. Covered in ["Price Density: A Visual Way to Measure Market Choppiness"](https://aligrithm.com/price-density-a-visual-way-to-measure-market-choppiness/). Measure 3: fractal dimension (how space-filling the path is). A straight line has dimension near 1; a path that fills a plane approaches dimension 2\. High noise pushes the dimension toward 2\. It is the most expensive to estimate and the least stable on short windows, so it is a confirmation tool rather than a primary signal. All three answer the same question: per unit of motion, how much net direction did I get? They differ in computation, not in intent. The discipline is to pick one as primary (the efficiency ratio) and use a second as a sanity check, because any single window-dependent estimate can mislead on a short sample, exactly the small-sample fragility the article ["Trade-Count Thresholds for Backtest Reliability"](https://aligrithm.com/trade-count-thresholds-for-backtest-reliability/) warned about for performance metrics. ## The classification mistake this prevents The mistake is to read a volatility chart, see two markets at 15% annualized, and conclude they are interchangeable backtesting venues. That conclusion drives the "run it on everything" workflow the article "Why Works on All Markets Is Usually a Red Flag" criticized. The fix is to add the noise axis before deciding anything. A two-axis grid, volatility on one axis and noise on the other, separates four quadrants that demand different treatment. | | Low noise (efficient) | High noise (choppy) | | --------------- | --------------------------------------------- | ------------------------------------------- | | Low volatility | Quiet grind: trend systems, small size | Quiet chop: mean-reversion, scalping | | High volatility | Violent trend: trend systems, vol-scaled size | Violent chop: fade with caution, wide stops | The two columns are the strategic divide; the two rows are the sizing divide. Volatility tells you how big to bet. Noise tells you what game you are playing. Get the column wrong and no amount of getting the row right will save the strategy. ## Visualizing noise versus volatility ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/06/article_86-visual-0.png) ## KEY POINTS - Volatility measures how far price moves (the total path length). Noise measures how much of that movement cancels out before producing net direction. They are different quantities driven by different mechanisms. - Two markets can show identical realized volatility and identical total motion while delivering a tenfold difference in net directional yield. The volatility reading cannot distinguish them. - The core measure: net directional change divided by total absolute path length, bounded 0 to 1\. Near 1 is a straight line (low noise). Near 0 is a drunk walk (high noise). Volatility lives in the denominator alone and is blind to the numerator. - Trend systems harvest the numerator (net direction). High volatility with high noise produces whipsaw: the volatility triggers entries, the noise reverses them before they pay. - Mean-reversion and grid systems harvest the denominator against the numerator. High noise is their fuel; a low-noise trending market makes them fade a one-way move and accumulate losers. - Volatility-targeting position sizing does not adjust for noise. It hands equal capital to a zero-edge market and a strong-edge market because both can hit the same volatility target. - Noise is more stable than volatility because it is tied to participant structure and information flow, which drift slowly, while volatility is a fast-moving state variable that can regime-shift in a week. - Three noise measures agree often enough to cross-check: the efficiency ratio (primary), price density, and fractal dimension. Use one as primary and one as confirmation; any single window-dependent estimate can mislead on a short sample. - The classification fix: add a noise axis to the volatility axis before deciding anything. The column (noise) tells you what game to play; the row (volatility) tells you how big to bet. Getting the column wrong cannot be rescued by getting the row right. - The next article, ["Efficiency Ratio Explained for Traders"](https://aligrithm.com/efficiency-ratio-explained-for-traders/), develops the primary noise measure in full with worked calculations and window-selection guidance. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Trading Systems and Methods - Perry Kaufman (Amazon)](https://www.amazon.com/s?k=Trading+Systems+and+Methods+Perry+Kaufman&ref=aligrithm.com) - [Cybernetic Trading Strategies - Murray Ruggiero (Amazon)](https://www.amazon.com/s?k=Cybernetic+Trading+Strategies+Murray+Ruggiero&ref=aligrithm.com) - [The Art of Currency Trading - Brent Donnelly (Amazon)](https://www.amazon.com/s?k=The+Art+of+Currency+Trading+Brent+Donnelly&ref=aligrithm.com) ### 3.32 The Backtest Integrity Checklist URL: https://aligrithm.com/the-backtest-integrity-checklist/ Last updated: 2026-07-27T13:24:19.000Z A research team presents a long-short cross-sectional equity strategy to the investment committee. The IS Sharpe is 1.6 across 2005 to 2022, the maximum drawdown is 14%, the average trade count per year is 380, the post-cost expectancy is positive on every annual slice. The committee asks one question before approving deployment: "walk me through the integrity checklist." The lead researcher pulls up a 32-item checklist and reads through it line by line. Items 1 to 14 (data and pipeline) pass. Items 15 to 20 (cost modeling) pass with one note: "small-cap costs estimated at 20 bp round-trip, conservative versus historical 15 bp average." Items 21 to 27 (statistical-test discipline) pass with one note: "permutation test p-value 0.018, robust to block length between 5 and 20 days." Items 28 to 32 (deployment readiness) flag two issues: "execution latency assumption is 200 ms, the team's current infrastructure has 350 ms median; capacity estimate has not been stress-tested against the strategy's actual position sizing rules." The committee approves a probationary deployment at 30% target size pending resolution of the two flagged items. The checklist is the artifact that converts a backtest from an internal research exercise into a deployment-ready estimate. The articles in this pillar so far cover the individual disciplines: stationarity diagnostics, regime stratification, parameter stability, walk-forward and CSCV, Monte Carlo, MAE/MFE, profit-factor and Sharpe interpretation, transaction cost modeling. This article assembles the disciplines into a single procedural artifact that the team uses before any pitch deck, any committee review, and any deployment. The checklist is the operational form of having internalized the disciplines, and presupposes that understanding rather than replacing it. A strategy that fails one item on the checklist is not deployed; a strategy that passes all of them is at least not failing for a known reason. The checklist has 32 items organized into 7 categories. Each item has a specific test, a specific pass criterion, and a specific failure action. The team that fills out the checklist with each box backed by the underlying analysis is doing 80% of the integrity work; the team that skips the checklist or fills it out as a formality is doing none of it. ## The checklist's structural justification Three structural reasons. Reason 1: the research process is iterative and the iterations accumulate biases that the headline numbers hide. A team works on a strategy for 3 to 12 months. Decisions get made on partial information. Parameters get adjusted. Data sources get changed. Each decision is small and locally justified; the cumulative effect on the IS Sharpe is large and unobservable from the final numbers alone. The checklist forces the team to surface each decision as a checkbox. Reason 2: the deployment is irreversible at the cost of real capital. The team's incentive is to ship; the allocator's incentive is to fund only strategies that survive. The checklist is the shared artifact that both parties can audit. The ["How to Evaluate a Strategy Beyond Net Profit"](https://aligrithm.com/how-to-evaluate-a-strategy-beyond-net-profit/) article framed the 25-metric evaluation panel; the checklist is the procedural version of the panel that also covers data and methodology integrity. Reason 3: humans are bad at remembering all the failure modes at once. The number of ways a backtest can be wrong (data leakage, look-ahead bias, survivorship bias, post-selection bias, regime mismatch, cost underestimation, capacity overestimation, etc.) exceeds working memory. A written checklist is a memory aid that converts integrity from a personal-discipline problem to a procedural problem. ## Category 1: data integrity Six items. Item 1: source documentation. The team can name every data source, the vendor, the version, the date range, and the corrections-applied policy. A strategy built on data that the team cannot fully document is one regression-error or one corporate-action mistake away from being a different strategy. Item 2: point-in-time alignment. Every input to the strategy uses only data that would have been available at the trading decision time. Earnings data uses announcement date and reported date, not as-of-today restated values. Fundamental data uses the as-reported version, not the latest restated version. The article ["Why OOS Failure Is Often a Stationarity Failure"](https://aligrithm.com/why-oos-failure-is-often-a-stationarity-failure/) framed look-ahead as one of four causes of OOS failure; the data-integrity check is the first defense. Item 3: survivorship-free universe. The strategy's investable universe at time t includes the names that were investable at t, not the names that survive to today. Equity universes that exclude delisted names will inflate the strategy's apparent Sharpe by 1 to 3 absolute units; commodity universes that exclude failed contracts will do the same. Item 4: corporate-action handling. Splits, dividends, mergers, spin-offs are applied to the price series. The handling is documented and consistent across the universe. A common failure: forward-adjusted prices in the IS period and unadjusted prices in the OOS period, which produces nonsense. Item 5: missing-data treatment. The team has a documented rule for missing observations (drop, forward-fill, interpolate). The rule is applied consistently across the universe and the time period. The rule does not introduce look-ahead (interpolating across a future-known value would). Item 6: timezone and trading-calendar handling. The team can produce the strategy's behavior on each market's local calendar, including holidays, half-days, and timezone transitions. A strategy that fires entries on a US holiday because the calendar was incorrectly set to "weekdays only" is generating fake trades that will not exist in deployment. ## Category 2: pipeline integrity Five items. Item 7: causal data flow. Every transformation in the pipeline uses only past data. Normalizations are computed on rolling windows ending at t-1\. Regression coefficients are estimated on a fitting window that ends before the prediction window. The article ["Rolling Normalization: Useful Tool or Hidden Overfit?"](https://aligrithm.com/rolling-normalization-useful-tool-or-hidden-overfit/) framed the rolling-window discipline. Item 8: code review and reproduction. A second team member has reviewed the backtest code and can reproduce the headline numbers from the same data. A backtest that one person can reproduce and no one else can has a high probability of containing a bug that the original author has stopped seeing. Item 9: unit tests on critical functions. The position-sizing function, the cost-application function, the signal-generation function, and the risk-management function each have unit tests with known inputs and expected outputs. The tests are run in CI before any backtest result is reported. Item 10: random-seed control. Any stochastic component of the backtest (Monte Carlo resampling, randomized initialization) uses a controlled random seed. The same seed produces the same backtest output. A backtest whose numbers vary across runs is non-reproducible and the variation is an unmeasured source of result uncertainty. Item 11: data-pipeline regression test. The pipeline has been re-run from raw data after the strategy was finalized, and the result matches the version the team has been iterating on. This catches the failure mode where the team has been iterating on a stale intermediate cache. ## Category 3: statistical-test discipline Five items. Item 12: pre-specified hypothesis. The strategy is described as a hypothesis (e.g., "cross-sectional momentum at the 1- to 12-month horizon predicts future returns in large-cap US equities, post-cost Sharpe positive") before the backtest is run. The hypothesis is recorded with date and authors. Strategies that are constructed from a search through the data first and then hypothesized after are post-hoc and the apparent significance is inflated. Item 13: trade-count adequacy. The strategy has at least 200 effective independent trades in the IS sample. The article ["Trade-Count Thresholds for Backtest Reliability"](https://aligrithm.com/trade-count-thresholds-for-backtest-reliability/) gave the calibration; 100 is the lower bound, 200 is the comfortable minimum, 1000+ is the gold standard for tight CIs on derived metrics. Item 14: permutation-test significance. The strategy's IS Sharpe is tested against the null of no predictive power using a block permutation test. The article ["Permutation Tests for Indicator Significance"](https://aligrithm.com/permutation-tests-for-indicator-significance/) gave the mechanics. The p-value is below 0.05 with multiple-testing correction applied if more than one indicator was tested. Item 15: search-width bias quantification. The team reports k\_eff (the effective number of parameter combinations searched). The article ["Degrees of Freedom in Trading Systems"](https://aligrithm.com/degrees-of-freedom-in-trading-systems/) gave the counting framework. The bias-corrected IS Sharpe is reported alongside the raw IS Sharpe. $$ \\widehat{\\text{SR}}\_{\\text{corrected}} = \\widehat{\\text{SR}}\_{\\text{raw}} - \\sigma\_{\\text{IS}} \\sqrt{2 \\log k\_{\\text{eff}}}, \\qquad \\sigma\_{\\text{IS}} \\approx \\frac{1}{\\sqrt{N\_{\\text{trades}}}} $$ For a strategy with raw IS Sharpe 1.6, N = 500 trades, and k\_eff = 1000, the bias correction is approximately 0.045 × sqrt(2 × 6.9) = 0.17\. The corrected IS Sharpe is 1.43\. For k\_eff = 100000, the correction is 0.045 × sqrt(2 × 11.5) = 0.22, and the corrected Sharpe is 1.38\. The raw Sharpe overstates the strategy by an amount that the team has to acknowledge before deployment. Item 16: CSCV / PBO computation. The strategy has been evaluated under combinatorially symmetric cross-validation; the probability of backtest overfitting is below 0.30\. The article ["CSCV: A Direct Probability of Backtest Overfit"](https://aligrithm.com/cscv-a-direct-probability-of-backtest-overfit/) gave the procedure. ## Category 4: regime and robustness Five items. Item 17: regime stratification. The strategy's performance is reported by volatility regime, trend regime, and macro regime. Each regime has at least 30 trades. No regime has Sharpe below -0.5 unless the strategy explicitly excludes that regime via a gating rule. The article ["Regime Coverage: Why Your Backtest Needs Different Market States"](https://aligrithm.com/regime-coverage-why-your-backtest-needs-different-market-states/) gave the stratification framework. Item 18: cross-market sanity. The strategy has been run on at least 2 comparable markets in the same family; the cross-market Sharpes are positive. The article "Why Works on All Markets Is Usually a Red Flag" framed the skeptical view; the sanity check is positive Sharpes on related markets, not on every market across all families. Item 19: parameter perturbation. Each parameter has been perturbed by +/-25%; the Sharpe at each perturbed value is within 30% of the optimal-value Sharpe. The article ["Parameter Stability Beats Best Parameter"](https://aligrithm.com/parameter-stability-beats-best-parameter/) gave the procedure. Item 20: time-period robustness. The strategy has been evaluated on at least 2 non-overlapping sub-periods of the IS sample. The Sharpes are positive on each sub-period. A strategy whose IS Sharpe comes from a single 18-month window alone is regime-dependent and should be gated to that regime. Item 21: walk-forward validation. The strategy has been evaluated through walk-forward analysis. The per-step OOS Sharpes have no clear downward trend. The article ["Why Walk-Forward Testing Is Better Than One Big OOS Split"](https://aligrithm.com/why-walk-forward-testing-is-better-than-one-big-oos-split/) gave the procedure. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Category 5: cost modeling Four items. Item 22: realistic per-trade cost. The cost model includes commission, full bid-ask spread, slippage at the 75th percentile of historical observations, and market impact at the strategy's typical order size. The article ["Why Transaction Costs Should Be Added Before You Fall in Love"](https://aligrithm.com/why-transaction-costs-should-be-added-before-you-fall-in-love/) gave the cost categories and magnitudes. Item 23: cost-sensitivity table. The Sharpe is reported at 5 cost levels (optimistic, IS-assumed, mildly conservative, conservative, worst-case). A strategy whose Sharpe drops by more than 0.5 when the cost goes from IS-assumed to conservative is cost-fragile and the deployment depends on aggressive cost execution. Item 24: capacity estimate. The team has a per-AUM Sharpe estimate that includes a market-impact model. The capacity is reported at multiple AUM levels ($10M, $50M, $250M, $1B). The deployment AUM is below the level at which the Sharpe degrades by 25%. Item 25: live-vs-IS cost reconciliation plan. If the strategy is funded, the team commits to comparing live realized costs to IS-assumed costs at month 1, month 3, and month 6\. A divergence triggers a cost-model update and a capacity re-estimate. ## Category 6: risk and stops Three items. Item 26: bootstrap drawdown distribution. The strategy's expected maximum drawdown distribution has been generated via bootstrap of the historical trade sequence. The article ["Monte Carlo for Trading Systems"](https://aligrithm.com/monte-carlo-for-trading-systems/) gave the procedure. The hard kill-switch is set at the bootstrap 99th percentile, not the historical maximum. Item 27: stop-loss audit. If the strategy uses stop losses, the team has computed the strategy with and without the stop; the with-stop version dominates the without-stop version in the deployment-relevant metrics (lower max DD, comparable or higher Sharpe). The article ["When a Stop Loss Improves Risk but Destroys Edge"](https://aligrithm.com/when-a-stop-loss-improves-risk-but-destroys-edge/) gave the trade-off. Item 28: MAE/MFE diagnostic. The team has computed the MAE/MFE distribution per trade. The article ["MAE/MFE Analysis: Seeing What Net Profit Hides"](https://aligrithm.com/mae-mfe-analysis-seeing-what-net-profit-hides/) gave the procedure. The distribution is consistent with the strategy's hypothesized mechanism: a mean-reversion strategy has bounded MAE and slow-resolving MFE; a trend-following strategy has tighter MFE and wider MAE. ## Category 7: deployment readiness Four items. Item 29: execution-infrastructure match. The strategy's latency, fill-rate, and order-type assumptions are matched against the team's actual execution infrastructure. A strategy that assumes 200 ms execution and 95% fill rate does not deploy on infrastructure that delivers 350 ms and 88%. Item 30: written kill-switches. The strategy has 3 to 5 written kill-switch triggers (drawdown threshold, Sharpe threshold, profit-factor threshold, CUSUM trigger, execution-quality trigger). The thresholds are calibrated against the bootstrap and Monte Carlo distributions. The article ["How to Detect When a Trading System Is Dying"](https://aligrithm.com/how-to-detect-when-a-trading-system-is-dying/) gave the procedural manual. Item 31: written decommission policy. The decommission policy is in writing, signed by the team and the allocator, before any capital is deployed. The article ["Why Systems Work Until They Don't"](https://aligrithm.com/why-systems-work-until-they-dont/) framed the strategic case for written policies. Item 32: capacity and scaling plan. The team has a written plan for scaling capital from initial deployment to target AUM, including the trigger conditions for each scaling step (typically a passing-grade quarter or a passing-grade Monte Carlo-implied test). ## Using the checklist A procedural sequence. Step 1: the lead researcher fills out the checklist with one of three responses on each item: PASS, FAIL, or N/A WITH JUSTIFICATION. The justification for N/A is one sentence per item. Step 2: a second team member reviews each item independently. Disagreements between the two responses are flagged and resolved before the checklist is finalized. The independent review catches the cases where the original researcher has stopped seeing a failure mode. Step 3: the finalized checklist is attached to the strategy specification document. Any future change to the strategy that affects a checklist item triggers a re-evaluation of that item. Step 4: at the deployment review, the committee reads the checklist line by line. Any FAIL or any N/A WITH WEAK JUSTIFICATION blocks deployment until resolved. Step 5: post-deployment, the checklist is re-run at the 3-month, 6-month, and 12-month reviews. Items that pass at the IS stage may fail at the live stage (cost reconciliation diverging, execution latency higher than assumed, regime moving out of the validated distribution). The re-run is the operational form of the article ["How to Detect When a Trading System Is Dying"](https://aligrithm.com/how-to-detect-when-a-trading-system-is-dying/). ## Anti-patterns Five mistakes specific to the checklist artifact. Anti-pattern 1: filling the checklist out as a formality. The team marks every item PASS without doing the underlying analysis. The artifact looks like integrity work and is the opposite of it. The cure: independent review by a second team member, and a committee that asks specific follow-up questions on at least 5 random items per checklist. Anti-pattern 2: building the checklist into the research process at the end. The team does the research, falls in love with the strategy, then runs the checklist and rationalizes the failures. The checklist becomes a confirmation step rather than a gate. The cure: the checklist is filled out incrementally as the research progresses. Items 1 to 6 (data integrity) are passed before item 7 (causal pipeline) is started; item 14 (permutation test) is computed before item 21 (walk-forward). Anti-pattern 3: skipping items the team treats as "fine by default". The team marks item 3 (survivorship) PASS because "we are using a major vendor that handles delistings". The vendor's handling has not been audited. The strategy turns out to be sensitive to the small-cap survivorship treatment. The cure: every PASS has a one-sentence justification that names the specific check performed. Anti-pattern 4: not updating the checklist when the strategy changes. The team makes a small change to the parameter set or to the cost model; the previous checklist is left in place; the new strategy is now operating with a stale integrity statement. The cure: any strategy change triggers a re-run of the items affected by the change. Anti-pattern 5: treating the checklist as the integrity work. The checklist is the artifact of the integrity work, not the work itself. A team that has the checklist filled out but has not internalized the disciplines that the checklist enforces is one personnel change away from losing all the integrity it had. The cure: the checklist is used as a teaching artifact for new team members and as a memory aid for the experienced ones, not as a substitute for understanding why each item is on the list. ## Decision matrix | Checklist outcome | Deployment decision | Follow-up | | ---------------------------------------------------- | ---------------------------------------------------- | ---------------------------------------- | | All 32 items PASS | Deploy at target size | Re-run checklist at 3/6/12 months | | 1-2 items FAIL on low-criticality categories (5, 7) | Deploy at 30-50% size, fix flagged items in parallel | Re-run flagged items at 3 months | | 1+ items FAIL on data integrity (category 1) | Do not deploy | Fix data pipeline, re-run full checklist | | 1+ items FAIL on pipeline integrity (category 2) | Do not deploy | Fix pipeline, re-run categories 2 and 3 | | 1+ items FAIL on statistical discipline (category 3) | Do not deploy | Address overfit / sample / leakage cause | | 1+ items FAIL on robustness (category 4) | Do not deploy or deploy with gating | Add regime gate or shrink universe | | 1+ items FAIL on cost modeling (category 5) | Do not deploy | Rebuild cost model, re-evaluate | | 1+ items FAIL on risk/stops (category 6) | Do not deploy | Address stop or kill-switch design | | 1+ items FAIL on deployment readiness (category 7) | Probationary deployment at reduced size | Resolve infrastructure or policy gap | The matrix maps the type of failure to the type of action. Failures in categories 1 to 3 are blocking; failures in 5 and 7 may permit probationary deployment; failures in 4 and 6 require strategy modifications that may or may not preserve the original hypothesis. ## Visualizing the checklist ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/05/article_85-visual-0.png) ## KEY POINTS - The backtest integrity checklist is the procedural artifact that converts a research result into a deployment-ready estimate. It does not replace the underlying disciplines; it is the operational form of having internalized them. - The checklist has 32 items in 7 categories: data integrity (6 items), pipeline integrity (5), statistical-test discipline (5), regime and robustness (5), cost modeling (4), risk and stops (3), deployment readiness (4). - Category 1 (data) covers source documentation, point-in-time alignment, survivorship-free universe, corporate-action handling, missing-data treatment, timezone and calendar handling. - Category 2 (pipeline) covers causal data flow, independent reproduction, unit tests on critical functions, random-seed control, end-to-end pipeline regression. - Category 3 (statistics) covers pre-specified hypothesis, trade-count adequacy of 200+ effective independent trades, permutation-test significance, k\_eff and bias-corrected Sharpe reporting, CSCV / PBO below 0.30. - Category 4 (robustness) covers regime stratification with positive performance in every required cell, cross-market sanity on comparable instruments, parameter perturbation surviving +/-25%, time-period robustness on non-overlapping sub-periods, walk-forward with no downward trend. - Category 5 (costs) covers realistic per-trade cost modeling, cost-sensitivity table at 5 cost levels, capacity estimate at multiple AUM levels, live-vs-IS cost reconciliation plan at 3 and 6 months post-deployment. - Category 6 (risk) covers bootstrap drawdown distribution with kill-switch at the 99th percentile, stop-loss audit comparing with-stop and without-stop versions, MAE/MFE distribution consistent with the hypothesized mechanism. - Category 7 (deployment) covers execution-infrastructure match, written kill-switches with calibrated thresholds, written decommission policy signed before capital is deployed, written capacity scaling plan with trigger conditions. - The bias-corrected Sharpe formula: raw IS Sharpe minus sigma\_IS times sqrt(2 log k\_eff). For raw SR 1.6, N = 500 trades, k\_eff = 1000, the correction is approximately 0.17\. The corrected value is the deployment-relevant estimate. - The procedural sequence: lead researcher fills the checklist, second team member reviews each item independently, finalized checklist is attached to the strategy specification, committee reads line by line at deployment review, checklist is re-run at 3, 6, and 12 months post-deployment. - Anti-pattern: filling the checklist as a formality without the underlying analysis. Anti-pattern: running the checklist at the end rather than incrementally throughout research. Anti-pattern: skipping items that look "fine by default" without recording the specific check performed. Anti-pattern: failing to update the checklist when the strategy changes. Anti-pattern: treating the checklist itself as the integrity work. - The matrix: failures in data, pipeline, or statistical categories block deployment outright; failures in cost or deployment-readiness permit probationary deployment with parallel resolution; failures in robustness or risk require strategy modifications that may invalidate the original hypothesis. - The current article closes the Pillar 3 procedural arc. The combination of the diagnostic disciplines from the prior articles and the procedural checklist of this article gives the operational toolkit for converting an apparent IS edge into a deployment that is at least not failing for a known reason. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Testing and Tuning Market Trading Systems - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Testing+and+Tuning+Market+Trading+Systems+Timothy+Masters&ref=aligrithm.com) - [Data Mining Algorithms in C++ - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Data+Mining+Algorithms+in+C%2B%2B+Timothy+Masters&ref=aligrithm.com) ### 3.31 Why Transaction Costs Should Be Added Before You Fall in Love URL: https://aligrithm.com/why-transaction-costs-should-be-added-before-you-fall-in-love/ Last updated: 2026-07-27T13:24:18.000Z A junior researcher presents a backtest of a high-frequency mean-reversion strategy on US small-cap stocks. The IS Sharpe is 2.4 over a 3-year backtest with 18000 trades. The team is excited; the IS Sharpe is exceptional and the trade count is large enough for the result to be statistically meaningful. The team starts planning the deployment, the capital allocation, and the operational infrastructure. The senior researcher asks for the post-cost Sharpe. The junior researcher has not computed it. The team waits while the cost adjustment is applied: round-trip commission of 0.4 cents per share, average bid-ask spread of 5 cents on small-caps, slippage estimate of 1 cent on average. Total per-trade cost approximately 0.30% of the trade size. The strategy's pre-cost average win is 0.42% and average loss is -0.31%. The cost subtraction reduces both numbers symmetrically: post-cost average win 0.12%, post-cost average loss -0.61%. The post-cost expectancy is approximately -0.07% per trade. The post-cost Sharpe is approximately -0.4\. The strategy is unprofitable. The team's six weeks of excitement evaporated in the cost adjustment. The pre-cost Sharpe of 2.4 was an artifact of ignoring the cost structure of the small-cap universe. The strategy's gross signal was real (the mean-reversion pattern existed), but the gross signal was smaller than the cost overhead. The strategy was a classic "edge that exists in the data and disappears in deployment" pattern. The deployment would have produced cumulative losses approximately equal to the pre-cost cumulative profits. The team's emotional investment in the strategy had grown over the six weeks, and several team members resisted the cost-adjusted conclusion until the cost calculation was independently verified. The discipline that prevents this pattern is procedural: apply realistic costs to the backtest before any optimization, before any pitch deck, before any team conversation about deployment. The pre-cost backtest is a tool for confirming the gross signal exists; the post-cost backtest is the deployment-relevant estimate. The team that develops emotional attachment to the pre-cost numbers has a much harder time accepting the post-cost reality. The article ["How to Evaluate a Strategy Beyond Net Profit"](https://aligrithm.com/how-to-evaluate-a-strategy-beyond-net-profit/) framed the cost-related metrics in Category 7 of the evaluation panel. This article gives the procedural discipline for getting the costs right and the early-stage discipline for not falling in love with pre-cost numbers. ## Cost categories to include Five categories of trading cost, each with a typical magnitude. Cost 1: explicit commission. The brokerage fee per trade. For US equities at retail brokers: $0 to $5 per trade. For US equities at institutional brokers: $0.001-$0.005 per share. For futures: $0.50-$3.00 per contract. For FX: 0-1 basis point. The commission is the most-often-included cost. Cost 2: bid-ask spread. The difference between the best bid and best ask. For liquid US equities: 1-3 basis points. For mid-cap stocks: 5-15 basis points. For small-caps: 15-50 basis points. For thinly-traded stocks: 50+ basis points. For futures: 1-5 basis points on liquid contracts. For FX: 0.1-1 basis point on majors. The half-spread is typically charged on each side of a round-trip; the full spread is the round-trip cost. Cost 3: slippage. The difference between the expected fill price and the actual fill price. Slippage depends on order size relative to liquidity, time of day, and market conditions. For small orders in liquid markets: 0.1-0.5 basis points. For larger orders: 1-10 basis points. For market-on-close orders: 5-20 basis points. For stops triggered in fast markets: 50+ basis points. The slippage estimate should be conservative; the average slippage is typically larger than the median. Cost 4: market impact. The price movement caused by the trade itself. For small orders, market impact is approximately zero. For orders that exceed 1% of average daily volume, market impact starts to matter; for orders that exceed 5%, market impact dominates. Capacity-aware backtests include a model of market impact (often square-root or linear in trade size). Cost 5: opportunity cost. The cost of not being filled at the desired price. Stops that are triggered intraday and not filled until close, limit orders that do not fill, partial fills, and similar issues add to the realized cost. Opportunity cost is the hardest to model and is often missed in backtests. For a typical strategy on liquid US equities, the per-round-trip cost is approximately 5-15 basis points. For small-caps or futures, 15-30 bp. For thinly-traded instruments, 30-100 bp. The strategy's gross edge must exceed the round-trip cost times the trade frequency for the strategy to be profitable. ## The math of cost-versus-edge A simple framework. Let the gross edge per trade be E\_gross and the round-trip cost be C. The net edge is: $$ E\_{\\text{net}} = E\_{\\text{gross}} - C $$ For a strategy with gross edge 0.15% per trade (typical for a mean-reversion strategy) and cost 0.30% (typical for small-caps), the net edge is -0.15% per trade. The strategy is unprofitable, regardless of how many trades it generates or how high the Sharpe looks before costs. The annualized return relationship. If the strategy produces N trades per year with net edge E\_net per trade, the annualized return is approximately N times E\_net. The cost-related sensitivity is dominated by trade frequency: a strategy that trades 100 times per year with net edge 0.5% per trade returns 50%; a strategy that trades 10000 times per year needs net edge 0.005% (0.5 basis points) per trade to return 50%. High-frequency strategies are much more cost-sensitive than low-frequency strategies because the per-trade edge they target is smaller. $$ \\text{CAGR} \\approx N \\cdot E\_{\\text{net}} = N \\cdot (E\_{\\text{gross}} - C) $$ The discipline. Compute E\_gross and C separately. Verify that E\_gross > C with margin. The article ["Why Profit Factor Can Lie"](https://aligrithm.com/why-profit-factor-can-lie/) framed the importance of trade-level distribution; the cost adjustment applies at the trade level and propagates to all trade-level statistics. ## Applying costs in a backtest Five operational principles. Principle 1: apply costs at the trade level, not at the aggregate level. Each trade's P&L should be reduced by the round-trip cost before computing any aggregate. Computing aggregate P&L and then subtracting cost-times-trade-count is approximately equivalent for symmetric strategies but is wrong for strategies with regime-dependent trade frequency. Principle 2: use realistic, conservative cost estimates. The temptation is to use minimum-possible costs to flatter the strategy. The right discipline is to use realistic, slightly-conservative estimates so that surprises during deployment are upside, not downside. Conservative cost estimates: actual commission, full spread (not half), slippage estimate at the 75th percentile of historical observations, market impact for the typical order size. Principle 3: model the cost-vs-volume relationship. Costs are not constant; they scale with order size. A strategy that works at $10M AUM may not work at $100M AUM because market impact grows. Apply the cost-versus-AUM model from the start so that capacity is visible. Principle 4: differentiate fixed and variable costs. Per-trade commissions are fixed; spreads and slippage scale with trade size; market impact scales nonlinearly. The breakdown allows the team to understand which cost components are most damaging and how the strategy's design affects each. Principle 5: validate the cost model against actual fills. Once the strategy has live data, compare the assumed cost structure to the actual realized costs. The actual costs are often 20-50% higher than the IS-assumed costs because the IS assumed average market conditions and live includes worst-case fills. Update the cost model with the live data. ## The "fall in love" operational risk Three psychological mechanisms. Mechanism 1: sunk-cost commitment. The team has invested weeks on the strategy. Acknowledging the post-cost result invalidates the work. The team rationalizes alternate cost assumptions to preserve the investment. Mechanism 2: pitch-deck momentum. The pre-cost backtest has been included in pitch decks, internal reviews, and team discussions. The team's narrative around the strategy is built on the pre-cost numbers. Reverting to post-cost requires unwinding the narrative. Mechanism 3: comparison to competitors. The team has seen pre-cost numbers from competitors and other research and has implicitly calibrated its expectations to the pre-cost benchmark. The post-cost numbers seem disappointing relative to the pre-cost reference, even though the post-cost numbers are the operationally relevant ones. The fix. Apply costs in Stage 2 of the research process (the prototype stage from ["Optimization Comes After Testing, Not Before"](https://aligrithm.com/optimization-comes-after-testing-not-before/)). The first numbers the team sees are post-cost. The pitch deck, the internal reviews, and the team narrative are all anchored to post-cost. The pre-cost numbers are computed as a diagnostic (to verify the gross signal exists) but are not the headline. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Anti-patterns Five mistakes specific to cost handling. Anti-pattern 1: applying costs at the end of research. The team optimizes the strategy on pre-cost data, falls in love, then applies costs and sees a disappointing result. The right discipline is to apply costs from the start. Anti-pattern 2: using minimum-possible cost estimates. Costs are uncertain; the IS-assumed cost is the central tendency, not the worst case. Use slightly-conservative estimates. Anti-pattern 3: ignoring volume scaling. The strategy backtested at $1M position sizes will not have the same costs at $100M position sizes. Model market impact. Anti-pattern 4: assuming "low-cost broker" makes the strategy profitable. The bid-ask spread and slippage do not depend on the broker; only the commission does. A strategy that requires zero commission to be profitable is a strategy that has gross edge below the spread, which is not a real edge. Anti-pattern 5: failing to update costs with live data. Live realized costs are often 20-50% higher than IS-assumed. Update the cost model and the deployment expectation; do not assume the IS cost was accurate on the grounds that the strategy has been running. ## Decision matrix | Stage of research | Right cost discipline | Wrong cost discipline | | ---------------------------- | ------------------------------------------------- | -------------------------- | | Stage 2: prototype | Apply costs from the first backtest | Defer cost analysis | | Stage 3: parameter stability | Use cost-adjusted Sharpe in the surface map | Map pre-cost surface only | | Stage 4: validation | Walk-forward and CSCV use cost-adjusted P&L | Validate pre-cost results | | Pitch deck | Headline post-cost Sharpe; pre-cost as diagnostic | Headline pre-cost Sharpe | | Internal review | All discussions use post-cost numbers | Mix pre-cost and post-cost | | Capacity estimate | Model market impact at multiple AUM levels | Single-AUM cost assumption | | Live deployment | Update cost model with realized fills | Assume IS cost holds | | Cost validation | Compare live realized cost to IS-assumed | No cost validation | The matrix maps research stage to cost discipline. The pattern: cost discipline starts at Stage 2, not at the end. The early discipline prevents the emotional investment in pre-cost numbers. ## The cost sensitivity table A useful artifact. For the chosen strategy, compute the Sharpe at multiple cost levels: For a hypothetical mean-reversion strategy with gross Sharpe 1.5 and IS-assumed round-trip cost 0.10%: | Round-trip cost | Resulting Sharpe | Resulting expectancy | | --------------------------- | ---------------- | -------------------- | | 0.05% (optimistic) | 1.65 | +0.20% | | 0.10% (IS-assumed) | 1.50 | +0.15% | | 0.15% (mildly conservative) | 1.30 | +0.10% | | 0.20% (conservative) | 1.05 | +0.05% | | 0.25% (worst-case fills) | 0.75 | 0.00% | | 0.30% (small-cap-typical) | 0.45 | \-0.05% | The table reveals the strategy's sensitivity to cost assumptions. A strategy whose Sharpe drops by 0.5+ when the cost assumption goes from 0.10% to 0.20% has substantial cost sensitivity; the deployment depends on getting the cost estimate right. A strategy whose Sharpe is stable across the range has cost-robustness. ## Visualizing the cost adjustment ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/05/article_84-visual-0.png) ## KEY POINTS - Apply realistic transaction costs to the backtest before any optimization, pitch deck, or team conversation about deployment. The pre-cost backtest verifies the gross signal exists; the post-cost backtest is the deployment-relevant estimate. - Five cost categories: commission (fixed per trade or per share), bid-ask spread (varies with liquidity, half-spread per side or full per round-trip), slippage (varies with order size and market conditions), market impact (nonlinear in trade size relative to ADV), opportunity cost (missed fills, partial fills, intraday-trigger-overnight-fill). - Per-round-trip cost magnitudes: liquid US equities 5-15 bp, small-caps and futures 15-30 bp, thinly-traded instruments 30-100 bp. The strategy's gross edge per trade must exceed the round-trip cost with margin. - The annualized return scales with N times net edge. High-frequency strategies are more cost-sensitive than low-frequency strategies because the per-trade edge they target is smaller. A strategy that needs 0.5 bp net edge per trade is much more cost-fragile than one that needs 50 bp. - Five operational principles: apply costs at the trade level (not aggregate), use realistic-conservative estimates, model the cost-vs-volume relationship for capacity, differentiate fixed and variable cost components, validate the cost model against actual fills. - The "fall in love" risk is psychological: sunk-cost commitment to pre-cost work, pitch-deck momentum based on pre-cost numbers, implicit calibration to pre-cost competitor benchmarks. The fix is to apply costs in Stage 2 (prototype), so the first numbers the team sees are post-cost. - The cost sensitivity table shows the Sharpe at multiple cost levels; a strategy whose Sharpe drops by 0.5+ from a 10 bp change in cost assumption is cost-fragile. Cost-robust strategies have stable Sharpes across the realistic cost range. - Anti-pattern: applying costs at the end of research. The team has fallen in love with pre-cost numbers and resists the post-cost reality. - Anti-pattern: using minimum-possible cost estimates. Use slightly-conservative estimates so deployment surprises are upside, not downside. - Anti-pattern: ignoring volume scaling. Strategy at $1M AUM has different costs than at $100M AUM. Model market impact. - Anti-pattern: assuming a low-cost broker fixes the strategy. Spread and slippage do not depend on the broker; commission does. A strategy that needs zero commission has gross edge below the spread. - Anti-pattern: failing to update costs with live data. Live realized costs are often 20-50% higher than IS-assumed. Update. - The current article gives the cost discipline. The next article in the publication (["The Backtest Integrity Checklist"](https://aligrithm.com/the-backtest-integrity-checklist/)) gives the comprehensive procedural checklist that ensures cost handling is one of many integrity items the team verifies before deployment. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Testing and Tuning Market Trading Systems - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Testing+and+Tuning+Market+Trading+Systems+Timothy+Masters&ref=aligrithm.com) - [Data Mining Algorithms in C++ - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Data+Mining+Algorithms+in+C%2B%2B+Timothy+Masters&ref=aligrithm.com) ### 3.30 How to Evaluate a Strategy Beyond Net Profit URL: https://aligrithm.com/how-to-evaluate-a-strategy-beyond-net-profit/ Last updated: 2026-07-27T13:24:18.000Z A pitch deck for a long-short equity strategy reports three numbers prominently: net profit of +47.2% over 5 years, win rate of 58%, profit factor of 1.85\. The fund manager treats these as sufficient. The allocator who has been reading the prior articles in this pillar asks for the full evaluation panel. The pitch deck does not have one. The allocator declines to invest, not because the headline numbers are bad, but because the absence of a comprehensive evaluation suggests the manager has not done the work. A strategy is not its net profit. A strategy is the joint distribution of (return path, risk profile, regime sensitivity, trade-level metrics, capacity, costs, and decay diagnostics). The headline numbers are projections of this distribution onto one or two axes; the projections discard most of the information. The articles in this pillar so far have covered specific projections: Sharpe ratio (article on trade-count requirements), drawdown distribution (Monte Carlo), MAE/MFE (trade-level path), profit factor (the previous article), regime stratification (regime-coverage article), parameter stability (Hill-Spike-Cliff article). This article gives the comprehensive evaluation panel that combines them. The panel has approximately 25 metrics organized into 7 categories. A strategy is deployment-ready when the panel as a whole supports the deployment, not when any one number does. The article ["Why Profit Factor Can Lie"](https://aligrithm.com/why-profit-factor-can-lie/) framed one specific metric's failure mode. The present article gives the full panel. The article ["The Backtest Integrity Checklist"](https://aligrithm.com/the-backtest-integrity-checklist/) later in this pillar gives the procedural checklist that ensures the panel was computed correctly. ## The seven categories of strategy evaluation Each category answers a different operational question. ### Category 1: aggregate return metrics The first-order projections of the return distribution. Metric 1.1: net profit (CAGR or total return). The headline number. Useful as a sanity check; insufficient by itself. Metric 1.2: Sharpe ratio (annualized). Excess return per unit of volatility. The article ["Trade-Count Thresholds for Backtest Reliability"](https://aligrithm.com/trade-count-thresholds-for-backtest-reliability/) framed the standard error and required trade counts. Sharpe is the standard risk-adjusted return metric. Metric 1.3: Sortino ratio. Excess return per unit of downside volatility. Useful when the return distribution is asymmetric. A high Sortino vs low Sharpe ratio means the volatility is mostly upside. Metric 1.4: Calmar ratio. CAGR divided by maximum drawdown. Useful for comparing strategies' return-per-pain trade-off. ### Category 2: drawdown metrics The distribution of equity-curve declines. Metric 2.1: maximum drawdown. The largest peak-to-trough decline in the equity curve. Single realization; not the operational ceiling. Metric 2.2: bootstrap 95th and 99th percentile drawdown. From the bootstrap procedure in ["Monte Carlo for Trading Systems"](https://aligrithm.com/monte-carlo-for-trading-systems/). The realistic deployment ceiling. Metric 2.3: average drawdown duration. The time spent below the previous equity peak, averaged across drawdown episodes. Long durations are operationally costly. Metric 2.4: longest drawdown duration. The longest consecutive period below peak. The "psychological tolerance" requirement for deployment. Metric 2.5: ulcer index. The square root of the average squared drawdown across the entire equity curve. A single number capturing the typical drawdown intensity, beyond the single worst case. $$ \\text{UI} = \\sqrt{\\frac{1}{T} \\sum\_{t=1}^{T} \\left( 100 \\cdot \\frac{\\text{Equity}\_t - \\text{Peak}\_{\[1, t\]}}{\\text{Peak}\_{\[1, t\]}} \\right)^2} $$ ### Category 3: trade-level metrics The distribution of individual trade outcomes. Metric 3.1: total trade count and effective N. From ["Trade-Count Thresholds for Backtest Reliability"](https://aligrithm.com/trade-count-thresholds-for-backtest-reliability/). The denominator for any per-trade statistic. Metric 3.2: win rate. The fraction of trades that are profitable. Useful when paired with the average win and average loss. Metric 3.3: average win and average loss. The conditional means of winning and losing trades. Metric 3.4: expectancy per trade. The unconditional mean of trade P&L. The fundamental edge. Metric 3.5: profit factor and trimmed profit factor. From ["Why Profit Factor Can Lie"](https://aligrithm.com/why-profit-factor-can-lie/). The trimmed version is the operationally relevant. Metric 3.6: Gini coefficient of trade P&L. Concentration index. Above 0.75 is a red flag. Metric 3.7: distribution of R-multiples. The trade P&Ls expressed in units of average loss. The shape reveals the strategy's edge structure. ### Category 4: trade-level path metrics From ["MAE/MFE Analysis: Seeing What Net Profit Hides"](https://aligrithm.com/mae-mfe-analysis-seeing-what-net-profit-hides/). Metric 4.1: MAE distribution. The distribution of maximum adverse excursions across trades. Metric 4.2: MFE distribution. The distribution of maximum favorable excursions. Metric 4.3: bounceback rate at the typical adverse threshold. The fraction of trades with large MAE that recover to be winners. Metric 4.4: profit capture ratio. The ratio of average realized P&L to average MFE. ### Category 5: regime sensitivity From ["Regime Coverage: Why Your Backtest Needs Different Market States"](https://aligrithm.com/regime-coverage-why-your-backtest-needs-different-market-states/) and ["Volatility Regimes and Strategy Survival"](https://aligrithm.com/volatility-regimes-and-strategy-survival/). Metric 5.1: Sharpe stratified by volatility regime. Low-vol, normal-vol, high-vol, crisis-vol Sharpes. Metric 5.2: Sharpe stratified by trend regime. Strong-trend, weak-trend, range-bound Sharpes. Metric 5.3: Sharpe stratified by macro regime. Risk-on, risk-off, neutral Sharpes. Metric 5.4: per-regime trade counts. Whether each regime has enough trades to support the per-regime estimate (typically 50+ trades per regime is the threshold). ### Category 6: parameter and structural sensitivity From ["Parameter Stability Beats Best Parameter"](https://aligrithm.com/parameter-stability-beats-best-parameter/) and ["The Hill, the Spike, and the Cliff: Reading Optimization Surfaces"](https://aligrithm.com/the-hill-the-spike-and-the-cliff-reading-optimization-surfaces/). Metric 6.1: parameter-stability map. The IS Sharpe surface across parameters. Metric 6.2: neighborhood Sharpe and variance. The stability of the chosen parameter. Metric 6.3: perturbation table. IS Sharpe at +/-10%, +/-25%, +/-50% perturbations. Metric 6.4: PBO from CSCV. From ["CSCV: A Direct Probability of Backtest Overfit"](https://aligrithm.com/cscv-a-direct-probability-of-backtest-overfit/). The probability of overfit at the parameter-search level. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ### Category 7: capacity and cost The deployment-realism metrics. Metric 7.1: average daily trade volume. The total dollar volume traded per day. Metric 7.2: capacity estimate. The maximum AUM at which the strategy can be deployed without significant market impact, typically 1-5% of average daily volume of the traded instruments. Metric 7.3: per-trade cost (commission + slippage + spread). The realistic cost in basis points. Metric 7.4: pre-cost vs post-cost Sharpe. The strategy with and without realistic costs. The article ["Why Transaction Costs Should Be Added Before You Fall in Love"](https://aligrithm.com/why-transaction-costs-should-be-added-before-you-fall-in-love/) later in this pillar gives the discipline. ## Using the panel Three operational rules. Rule 1: report all 25 metrics. Pitch decks that report only 3-5 of these are hiding the rest. A complete evaluation panel is the first artifact of a competent research process. Rule 2: weight by deployment context. An allocator looking for a tail-hedge strategy will emphasize Categories 2 and 5 (drawdown and regime sensitivity); a long-biased equity allocator will emphasize Categories 1 and 7 (return and capacity); a high-frequency trader will emphasize Category 3 (trade-level) and Category 7 (cost). The same panel serves different decision criteria. Rule 3: identify the failure axis. A strategy with Sharpe 1.2, drawdown 9%, regime-stratified Sharpes 1.4 / 1.0 / 0.5 / -0.2 has a clear failure axis: high-vol regimes. The deployment decision is then either "do not deploy" or "deploy with explicit high-vol gating". The panel reveals the failure axis; the headline net profit hides it. ## A worked panel for a hypothetical strategy For the long-short equity strategy from the opening: Category 1: Net profit +47.2%, CAGR 8.0%, Sharpe 1.05, Sortino 1.32, Calmar 0.84. Category 2: Max DD -9.5%, Bootstrap 95th DD -13.2%, Bootstrap 99th DD -16.8%, average DD duration 47 days, longest DD duration 184 days, ulcer index 4.2. Category 3: 1240 trades over 5 years, effective N approximately 1240 (rebalance count for cross-sectional factor strategies is the relevant N), win rate 58%, average win +0.47%, average loss -0.32%, expectancy +0.14% per trade, PF 1.85, trimmed-top-3 PF 1.78, Gini 0.51, R-multiple distribution roughly bell-shaped with mild right skew. Category 4: MAE distribution mostly within 0 to -1.2 ATR, MFE distribution mostly within 0 to +1.5 ATR, bounceback rate at -1 ATR is 52%, profit capture ratio 0.73. Category 5: Sharpe in low vol 1.4, normal vol 1.2, high vol 0.6, crisis vol -0.3\. Sharpe in strong trend 1.4, weak trend 1.0, range-bound 0.7\. Per-regime trade counts adequate for low/normal/high (200+ each); thin for crisis (40 trades). Category 6: parameter-stability map shows broad hill at chosen parameter, neighborhood Sharpe 1.02 vs peak 1.05, perturbation +/-25% drops Sharpe by 9%, PBO 0.18. Category 7: average daily trade volume $50M, capacity estimate $50-200M (1-5% of ADV), per-trade cost 4 bps, pre-cost Sharpe 1.18, post-cost Sharpe 1.05. The reading. The strategy has a positive expectancy, robust across most regimes (Categories 1-4 are good), but the crisis-vol regime is a clear weakness and the high-vol Sharpe is below the deployment threshold. The deployment decision: deploy with explicit volatility gating (suspend trading when realized vol exceeds the high-vol threshold), tighter kill-switches, capacity-conservative sizing. Without the panel, the allocator might have deployed at full capital and been surprised by the crisis-vol underperformance. ## Anti-patterns Five mistakes specific to evaluation reporting. Anti-pattern 1: reporting only Categories 1-3 (return, drawdown, trade-level aggregates). The path-level (Category 4), regime (Category 5), parameter sensitivity (Category 6), and capacity (Category 7) information is operationally critical and is absent in most retail pitch decks. Anti-pattern 2: cherry-picking metrics. A team can report any subset of the 25 metrics. The right discipline is to report the full panel; the team's choice of which subset to highlight is a tell about what the team is hiding. Anti-pattern 3: confusing "high score on every metric" with "good strategy". A strategy that has Sharpe 2.0, drawdown 3%, profit factor 4.0, and zero crisis-regime exposure is suspicious; either the metrics were computed on a sample that did not include crisis events or the strategy is overfit. Real strategies have realistic numbers across the panel; impossibly good numbers indicate methodology issues. Anti-pattern 4: ignoring the per-regime trade counts in Category 5\. A strategy with Sharpe 0.5 in crisis-vol with 5 trades carries no information about crisis-vol behavior. The cell shows a sample-size problem, not a performance signal. Anti-pattern 5: not updating the panel as the strategy ages. Live trading produces additional data; the panel should be updated quarterly with the live data combined with the IS data. A strategy whose panel has not been updated in a year is undermonitored. ## Decision matrix | Reported metrics | Inference about the team | Allocator action | | ---------------------------------------------------------------- | ------------------------------------------------------- | --------------------------------------------- | | Only net profit, win rate, PF | Beginner-level pitch; team has not done full evaluation | Decline; ask for full panel | | Categories 1, 2, 3 fully reported | Standard institutional pitch; basic competence | Continue evaluation, ask about Categories 4-7 | | Categories 1-7 all reported with sources | Mature research team | Engage in deeper diligence | | Categories 1-7 reported with confidence intervals and bootstraps | Sophisticated research team | Strong candidate for allocation | | Per-regime breakdown showing failure axis | Team is honest about weaknesses | Trust more, allocate with appropriate gating | | All metrics impossibly good | Methodology issue or fraud | Investigate or decline | | Headline metrics good, missing trade-level path | Team has not done MAE/MFE | Ask for the path data | | Headline good, missing parameter stability | Team has not done robustness | Ask for the surface map | | Live updates after 6+ months | Team is monitoring | Trust the monitoring discipline | | No live updates | Team is not monitoring | Investigate operational discipline | The matrix is the allocator-side reading guide. The pattern: the panel completeness is itself the diagnostic of the team's competence and discipline. ## Visualizing the panel ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/05/article_83-visual-0.png) ## KEY POINTS - A strategy is the joint distribution of return path, risk profile, regime sensitivity, trade-level metrics, capacity, costs, and decay diagnostics. The headline net profit is one projection of this distribution. - The comprehensive evaluation panel has approximately 25 metrics organized into 7 categories: aggregate return (CAGR, Sharpe, Sortino, Calmar, net profit), drawdown (max, bootstrap percentiles, durations, ulcer index), trade-level (count, win rate, expectancy, PF and trimmed PF, Gini, R-multiples), trade-level path (MAE, MFE, bounceback, profit capture), regime sensitivity (Sharpe stratified by vol, trend, macro regimes), parameter and structural sensitivity (stability map, neighborhood Sharpe, perturbation table, PBO), capacity and cost (volume, capacity estimate, per-trade cost, pre-vs-post-cost Sharpe). - A strategy is deployment-ready when the panel as a whole supports the deployment, not when any one number does. The headline net profit and Sharpe are necessary but not sufficient. - Three operational rules: report all 25 metrics, weight by deployment context (different allocators emphasize different categories), identify the failure axis (the regime or parameter where the strategy underperforms). - The example panel for a long-short equity strategy: most metrics positive, but Sharpe in crisis-vol regime is -0.3\. The failure axis is crisis-vol. The deployment decision is "deploy with vol gating", not "do not deploy" and not "deploy at full capital". - The panel completeness is itself a diagnostic of the research team's competence. A pitch with only Categories 1-3 (return, drawdown, trade-level aggregates) is missing the operationally critical information. - Anti-pattern: cherry-picking metrics. The team's choice of which subset to report is a tell about what is hidden. - Anti-pattern: confusing "high score on every metric" with "good strategy". Real strategies have realistic numbers across the panel; impossibly good numbers indicate methodology issues. - Anti-pattern: ignoring per-regime trade counts. A regime with 5 trades is not informative; small-sample noise dominates. - Anti-pattern: not updating the panel as the strategy ages. Live trading data should be incorporated quarterly into the panel; the panel becomes the operational monitoring artifact. - Anti-pattern: applying the same threshold for "good" to every metric. Trend-following has different typical PF than mean reversion; equity has different typical drawdown durations than FX. The thresholds depend on the strategy family covered in "Market Personality: Why Gold, FX, Crypto, and Equities Need Different Systems". - The current article gives the comprehensive evaluation panel. The next article in the publication (["Why Transaction Costs Should Be Added Before You Fall in Love"](https://aligrithm.com/why-transaction-costs-should-be-added-before-you-fall-in-love/)) covers the cost discipline that determines whether the post-cost numbers in Category 7 are the right benchmarks. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Testing and Tuning Market Trading Systems - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Testing+and+Tuning+Market+Trading+Systems+Timothy+Masters&ref=aligrithm.com) - [Data Mining Algorithms in C++ - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Data+Mining+Algorithms+in+C%2B%2B+Timothy+Masters&ref=aligrithm.com) ### 3.29 Why Profit Factor Can Lie URL: https://aligrithm.com/why-profit-factor-can-lie/ Last updated: 2026-07-27T13:24:17.000Z Two strategies, both with 200 trades, both with profit factor of 2.0\. The pitch deck shows the profit factors and treats them as approximately equivalent in quality. They are not. Strategy A: 120 winners averaging +0.42% each, 80 losers averaging -0.32% each. Sum of wins = +50.4%. Sum of losses = -25.6%. Profit factor = 50.4 / 25.6 = 1.97\. Largest winner = +1.4%. Largest loser = -0.9%. The win and loss distributions are roughly bell-shaped with similar magnitudes. The strategy's profitability is broadly distributed across the trade history. Strategy B: 90 winners averaging +0.10% each, 110 losers averaging -0.20% each, plus three exceptional winners of +12.0%, +14.0%, +18.0%. Sum of "regular" wins = +9.0%. Sum of three exceptional wins = +44.0%. Total wins = +53.0%. Sum of losses = 110 \* 0.20 = +22.0% (in absolute value). Profit factor = 53.0 / 22.0 = 2.41\. Profit factor without the three exceptional winners = 9.0 / 22.0 = 0.41 (the strategy is unprofitable on its non-exceptional trades). Largest winner = +18.0%. Largest loser = -0.6%. The strategy's profitability is concentrated in three extreme trades. The other 197 trades, in aggregate, lose money. Strategy A's profit factor of approximately 2.0 reflects a stable, broadly-distributed edge. Strategy B's profit factor of 2.4 reflects a tail-dependent edge that depends on three specific historical trades. Strategy B's expected OOS profit factor is much lower than 2.4 because the OOS will not necessarily contain three +12% to +18% trades; if the OOS contains only "regular" trades from the underlying distribution, the OOS profit factor will be near 0.4, not 2.4\. The headline profit factor numbers cannot distinguish A from B. The article ["MAE/MFE Analysis: Seeing What Net Profit Hides"](https://aligrithm.com/mae-mfe-analysis-seeing-what-net-profit-hides/) framed the trade-level path information that aggregate statistics conceal. This article gives the same diagnostic for profit factor. Profit factor is one of the most-cited metrics in retail trading literature and one of the most easily manipulated. The article enumerates the failure modes, the diagnostic plots that reveal them, and the alternative metrics that complement profit factor without inheriting its weaknesses. ## Profit factor, defined The definition. Profit factor is the ratio of the sum of winning trades' P&L to the absolute value of the sum of losing trades' P&L. $$ \\text{PF} = \\frac{\\sum\_{i: \\text{P\\&L}\_i > 0} \\text{P\\&L}\_i}{\\sum\_{i: \\text{P\\&L}\_i < 0} |\\text{P\\&L}\_i|} $$ A profit factor above 1 means the strategy is profitable in aggregate. A profit factor of 2.0 is often presented as a "good" threshold, with 1.5 as "acceptable" and 3.0+ as "excellent" in retail literature. Profit factor's strengths. It is a single intuitive number that combines win rate, average win, and average loss into one ratio. It is computable from any trade record. It is invariant to the unit of position size as long as positions are sized consistently. Profit factor's weaknesses. It is dominated by extreme outliers. It does not penalize concentration of profits in a few trades. It does not differentiate between a stable edge and a tail-dependent edge. It shows high sensitivity to the IS sample's specific outliers. ## Three failure modes of profit factor Failure 1: outlier dominance. As in the Strategy B example, a few extreme winners can dominate the numerator. The profit factor reports the aggregate ratio without indicating that the aggregate is concentrated. Removing the top 1-3 trades and recomputing the profit factor reveals the dependence; for many strategies the "trimmed" profit factor is 30-70% lower than the headline. Failure 2: optimization gaming. A team optimizing on profit factor will produce strategies with high profit factors but potentially low Sharpe and high concentration. Profit factor rewards low-frequency strategies that catch occasional large moves; it does not reward strategies with consistent moderate edge. The optimization-target choice has structural consequences. Failure 3: instability across samples. Because profit factor is dominated by outliers, the profit factor on different IS sub-samples can vary across a wide range. A strategy with profit factor 2.0 on a 5-year sample may have profit factor 1.4 on years 1-2.5 and 2.6 on years 2.5-5\. Bootstrap or sub-sample analysis reveals the instability; the headline does not. ## Diagnostic plots and statistics Five diagnostics that complement the headline profit factor. Diagnostic 1: cumulative profit factor curve. Plot the running profit factor as trades accumulate from 1 to N. A stable strategy shows the curve converging to a steady value. A tail-dependent strategy shows the curve jumping at specific trades and otherwise drifting. The convergence pattern is informative. Diagnostic 2: profit factor minus top-K. Compute the profit factor with the top 1, top 3, top 5, top 10 winners removed. For a stable strategy, removing the top trades reduces profit factor by 5-15% per trade removed. For a concentrated strategy, removing the top 1 trade can drop profit factor by 30%+ and removing the top 3 can drop it by 50%+ or convert profitability to unprofitability. The "top-K-trimmed" profit factor is the operationally relevant number. Diagnostic 3: Gini coefficient of trade P&L. The Gini coefficient measures concentration. A Gini near 0 means the P&L is evenly distributed across trades; a Gini near 1 means concentrated in a few trades. Trading strategies typically have Gini between 0.4 and 0.8; values above 0.75 are red flags for tail-dependence. Diagnostic 4: distribution of wins and losses on log scale. The log-scale histogram reveals the shape of the win and loss distributions. A roughly normal-shaped distribution means most trades are in a typical range. A heavy-right-tailed win distribution means a few extreme trades dominate. Visualization reveals what the headline profit factor hides. Diagnostic 5: bootstrap of profit factor. The bootstrap distribution of profit factor (covered in ["Monte Carlo for Trading Systems"](https://aligrithm.com/monte-carlo-for-trading-systems/)) reveals the sampling-distribution variability. A stable strategy has a tight bootstrap distribution. A concentrated strategy has a wide distribution with a long left tail (when the bootstrap omits the extreme trades, the profit factor drops by 40-60% or more). ## Operational interpretation Three rules of thumb. Rule 1: report profit factor with the top-3-trimmed profit factor. The two numbers together reveal whether the headline depends on outliers. If trimming the top 3 trades drops the profit factor by less than 30%, the strategy has a broad-based edge. If trimming drops the profit factor by more than 50%, the strategy is concentrated. Rule 2: report profit factor alongside Sharpe and expectancy. A high profit factor with low Sharpe is suspicious (it suggests rare large winners with high variance). A high profit factor with high Sharpe is more credible. The combination is more informative than either metric alone. Rule 3: do not optimize on profit factor. Optimization on profit factor produces tail-dependent strategies. Optimize on Sharpe (which penalizes variance), expectancy after costs (which penalizes high frequency), or a combined objective that includes a concentration penalty. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Cases where profit factor is informative Three cases where profit factor is a useful diagnostic. Case 1: small-trade strategies where outliers are clipped. For strategies with explicit position caps and small per-trade size, the trade distribution has bounded outliers. Profit factor is approximately the ratio of average win to average loss times the win rate ratio, and it is informative. Case 2: trend-following with explicit catastrophic-loss protection. Trend followers expect a few large winners and many small losers. The profit factor of approximately 1.5-2.0 from such strategies is consistent with the structural mechanism. The catastrophic-loss protection (covered in ["When a Stop Loss Improves Risk but Destroys Edge"](https://aligrithm.com/when-a-stop-loss-improves-risk-but-destroys-edge/)) bounds the loss-side outliers. Case 3: high-frequency strategies with thousands of trades. With N greater than 5000, the law-of-large-numbers convergence makes profit factor a reasonably stable estimate (though still subject to outlier dominance if the underlying distribution is heavy-tailed). ## Anti-patterns Five mistakes specific to profit factor. Anti-pattern 1: comparing two strategies on profit factor alone. Two strategies with the same profit factor can have very different OOS expected performance. Always pair with Sharpe, expectancy after costs, and the trimmed profit factor. Anti-pattern 2: using profit factor as the optimization target. The optimization will favor tail-dependent strategies. Use Sharpe or expectancy after costs as the optimization target, with profit factor as a secondary diagnostic. Anti-pattern 3: reporting only the headline profit factor without the trimmed values. The dependence on top trades is invisible without the trimmed metric. The dependence is the operationally relevant information. Anti-pattern 4: applying the "PF > 2.0 is good" threshold uniformly across strategy families. Trend-following strategies typically have PF approximately 1.4-1.8 (many small losers, few large winners) and are deployable. Mean-reversion strategies typically have PF approximately 1.6-2.5 (many small winners, few medium losers) and are deployable. The threshold depends on the family. Anti-pattern 5: confusing PF stability with strategy stability. A strategy whose PF is stable across IS sub-samples may still have other instabilities (regime sensitivity, parameter fragility, decay over time). Combine PF stability with the diagnostics from "Walk-Forward Testing" and ["Parameter Stability Beats Best Parameter"](https://aligrithm.com/parameter-stability-beats-best-parameter/). ## Decision matrix | Profit factor pattern | Interpretation | Action | | -------------------------------------------- | -------------------------------- | ---------------------------------- | | Headline PF 2.0, trimmed-top-3 PF 1.7 | Stable broad-based edge | Treat as informative | | Headline PF 2.4, trimmed-top-3 PF 0.8 | Tail-dependent edge | Investigate; do not trust headline | | PF 2.0 with high Sharpe (>1.0) | Stable edge with low variance | Strong combination | | PF 2.0 with low Sharpe (<0.5) | Outlier-dependent | Investigate the outliers | | Cumulative PF curve smoothly converges | Stable | Trust | | Cumulative PF curve jumps at specific trades | Concentrated | Investigate the jumps | | Gini coefficient of trade P&L > 0.75 | High concentration | Red flag | | Gini coefficient < 0.55 | Low concentration | Stable | | Bootstrap PF distribution is wide | Sampling instability | Larger sample needed | | Bootstrap PF distribution is tight | Stable | Trust | | PF used as optimization target | Tail-dependent strategies result | Switch to Sharpe target | The matrix maps profit-factor pattern to action. The pattern: profit factor is one diagnostic; combine with concentration measures, Sharpe, and bootstrap analysis. ## Visualizing profit factor's lie ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/05/article_82-visual-0.png) ## KEY POINTS - Profit factor is the ratio of sum of winning trades to absolute sum of losing trades. PF > 1 means profitable in aggregate; "good" thresholds in retail literature are 1.5-3.0 depending on the strategy family. - Three failure modes: outlier dominance (a few extreme trades inflate the ratio), optimization gaming (optimizing on PF produces tail-dependent strategies), instability across samples (PF varies across a wide range over IS sub-samples for concentrated strategies). - Two strategies with the same headline PF can have very different OOS expectations. A stable broad-based PF 2.0 strategy is approximately as profitable OOS. A tail-dependent PF 2.4 strategy that depends on three large winners may have OOS PF near 0.4 if the OOS does not contain comparable outlier trades. - Five complementary diagnostics: cumulative PF curve (stable strategy converges smoothly), PF minus top-K (trimmed PF reveals concentration), Gini coefficient of trade P&L (concentration index), distribution of wins and losses on log scale (visualize the tail), bootstrap PF distribution (sampling stability). - Three operational rules: report headline PF with the top-3-trimmed PF (concentration diagnostic), report PF alongside Sharpe and expectancy (combined informativeness), do not optimize on PF (use Sharpe or expectancy after costs as optimization target). - Three cases when PF is informative: small-trade strategies with bounded outliers, trend-following with explicit catastrophic-loss protection, high-frequency strategies with thousands of trades. - The math: PF is dominated by the numerator's largest contributions and the denominator's largest contributions. A heavy-right-tailed P&L distribution produces an unstable PF. A bounded P&L distribution produces a stable PF. - Anti-pattern: comparing two strategies on PF alone. Pair with Sharpe, expectancy, trimmed PF. - Anti-pattern: using PF as the optimization target. Optimization will favor tail-dependent strategies. - Anti-pattern: reporting only headline PF without trimmed values. The concentration is invisible. - Anti-pattern: applying "PF > 2.0 is good" uniformly across families. Trend-following typically has PF 1.4-1.8 and is deployable; mean-reversion typically has PF 1.6-2.5 and is deployable. - Anti-pattern: confusing PF stability with strategy stability. Combine with parameter-stability and walk-forward diagnostics. - The current article gives the profit-factor-specific diagnostic. The next article in the publication (["How to Evaluate a Strategy Beyond Net Profit"](https://aligrithm.com/how-to-evaluate-a-strategy-beyond-net-profit/)) covers the comprehensive metrics framework that uses PF as one of many complementary diagnostics rather than the headline. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Testing and Tuning Market Trading Systems - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Testing+and+Tuning+Market+Trading+Systems+Timothy+Masters&ref=aligrithm.com) - [Data Mining Algorithms in C++ - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Data+Mining+Algorithms+in+C%2B%2B+Timothy+Masters&ref=aligrithm.com) ### 3.28 MAE/MFE Analysis: Seeing What Net Profit Hides URL: https://aligrithm.com/mae-mfe-analysis-seeing-what-net-profit-hides/ Last updated: 2026-07-27T13:24:16.000Z A breakout strategy on E-mini SPX has 320 trades over five years. The aggregate statistics: 58% win rate, average win +0.62%, average loss -0.41%, expectancy per trade +0.19%, annualized Sharpe 1.15, maximum drawdown 9.4%. The team is preparing to deploy. A junior researcher is asked to plot the Maximum Adverse Excursion (MAE) and Maximum Favorable Excursion (MFE) for each trade and look at the resulting scatter. The plots reveal patterns the aggregate statistics did not show. The MAE distribution: 31% of trades had an MAE between 0.0% and 0.5% (small adverse excursions, mostly winners). 42% had MAE between 0.5% and 1.5% (typical adverse range, mixed). 27% had MAE between 1.5% and 4.0% (large adverse excursions). Among the trades with MAE above 2%, 38% of them ended as winners (the position recovered) and 62% as losers (the position did not). The strategy regularly tolerates substantial adverse moves before recovering or being closed. The MFE distribution: 14% of trades had MFE between 0.0% and 0.3% (no meaningful favorable move; mostly losers or quickly-stopped winners). 49% had MFE between 0.3% and 1.5% (modest favorable move that the strategy mostly captured). 37% had MFE between 1.5% and 5.0% (large favorable moves). Among the 37% with MFE above 1.5%, only 31% closed at their MFE level; the others closed at less than half of their peak favorable move. The strategy leaves substantial money on the table on its largest winners. The aggregate statistics did not reveal either pattern. The 58% win rate told the team about the binary outcome but not about the path. The +0.62% average winner told the team about the realized profit but not about the unrealized profit that was given back. The -0.41% average loser told the team about the realized loss but not about the worst point during the trade. MAE/MFE analysis fills these gaps. The article ["When a Stop Loss Improves Risk but Destroys Edge"](https://aligrithm.com/when-a-stop-loss-improves-risk-but-destroys-edge/) framed the stop-loss trade-off; MAE/MFE is the diagnostic technique that quantifies which trades a stop-loss change would help or hurt before applying it. This article gives the framework, the standard plots, and the operational uses. ## MAE and MFE, defined Two trade-level statistics computed from the trade's price path. MAE (Maximum Adverse Excursion). The largest unrealized loss the trade reached at any point during its life, measured in price units, percentage units, or ATR units. MAE is always non-positive (or zero, if the trade never went underwater). For a trade that opens long at 100, drops to 96, recovers to 102, and closes at 102, the MAE is -4 (or -4% if expressed as a percentage of entry). MFE (Maximum Favorable Excursion). The largest unrealized gain the trade reached at any point during its life, measured in the same units as MAE. MFE is always non-negative. For the same example trade above, the MFE is +2 (or +2%). Both statistics depend on the bar interval. For daily-bar backtests, MAE/MFE are computed from daily highs and lows. For minute-bar backtests, the resolution is finer and the MAE/MFE values are typically larger in magnitude (the intra-bar excursions are visible). The choice of bar interval is a degree of freedom that should be specified explicitly. ## The standard MAE/MFE plots Four plots, each diagnostic. Plot 1: MAE histogram. Distribution of MAE values across all trades. The shape reveals how often the strategy goes underwater and by how much. A right-leaning distribution (most trades close to 0 MAE) suggests the strategy enters at favorable points; a left-leaning distribution (long tail of large negative MAE) suggests entries are often early and the strategy waits through adverse moves. Plot 2: MFE histogram. Distribution of MFE values. A right-leaning distribution means most trades reach modest favorable levels; a heavy right tail means some trades reach large favorable levels. The shape reveals the strategy's profit-potential profile. Plot 3: realized P&L vs MAE scatter. Each trade is a point with horizontal coordinate = MAE and vertical coordinate = realized P&L. The scatter shape reveals two things: (a) for trades with large MAE, what fraction recover to be winners (the "bounceback rate"), and (b) the typical P&L conditional on a given MAE level. For mean-reversion strategies, the scatter typically shows that trades with MAE in the -1 to -2 ATR range have higher win rates than trades with smaller MAE (counterintuitively, the trades that go underwater the most are the ones that recover). For trend-following strategies, the scatter typically shows the opposite: trades with large MAE are the ones that fail. Plot 4: realized P&L vs MFE scatter. Each trade is a point with horizontal coordinate = MFE and vertical coordinate = realized P&L. The scatter reveals how much profit the strategy captures from its favorable excursions. A diagonal scatter (P&L approximately equal to MFE) means the strategy exits near its peak. A scatter where realized P&L is consistently below MFE (e.g., realized P&L is approximately 50% of MFE) means the strategy gives back substantial unrealized profit before closing. ## Operational uses Five direct applications. Use 1: stop-loss design. The MAE distribution reveals where a stop loss would intervene. A stop at -2 ATR would close the 27% of trades with MAE > 2 ATR; if 38% of those would have recovered to winners (the bounceback rate from the MAE-vs-P&L scatter), the stop loss converts 38% \* 27% = 10% of total trades from winners to losers. The expected change in expectancy from the stop loss is computable directly from the MAE distribution and the bounceback rate. Use 2: profit-target design. The MFE distribution reveals where a profit target would intervene. A target at +1.5 ATR would close trades that reach +1.5 ATR before they continue to higher MFE. The realized-P&L-vs-MFE scatter shows what fraction of trades that reach +1.5 ATR continue to higher MFE versus reverse to lower realized P&L. For most trend-following strategies, the answer is "most continue", and the profit target costs more than it saves. For most mean-reversion strategies, the answer is "few continue", and the profit target captures profits that would otherwise be given back. Use 3: entry-timing diagnosis. The MAE distribution at trade entry can be cross-tabulated with the entry signal's strength. If the strongest entry signals correspond to the smallest MAE (the strongest signals enter at the most favorable moments), the entry timing is well-calibrated. If the relationship is inverted (strong signals enter at the worst moments and tolerate large MAE), the entry timing has a problem. Use 4: exit-timing diagnosis. The realized P&L vs MFE scatter quantifies the strategy's exit timing. A scatter where realized P&L is consistently below MFE by more than 50% indicates the strategy is exiting too late (giving back profits) or has stop losses that close trades on retracement before continuation. The article ["When a Stop Loss Improves Risk but Destroys Edge"](https://aligrithm.com/when-a-stop-loss-improves-risk-but-destroys-edge/) framed the stop-loss-related version of this; the MFE-based version applies more broadly. Use 5: outlier identification. Specific trades with very large MAE or MFE that fall outside the typical distribution are outliers. They may correspond to specific market events (news shocks, gaps, microstructure events) that the strategy was not designed to handle. Investigating these outliers often reveals strategy specification gaps. ## Limits of MAE/MFE Three honest limits. Limit 1: bar resolution. MAE/MFE on daily bars misses intraday excursions. A trade that opens long at 100, drops to 92 intraday, recovers to 99 by close, and closes at 99 next day shows MAE of -1 (close-to-close) on daily bars but MAE of -8 on intraday bars. The choice of bar resolution materially affects the MAE/MFE statistics. Limit 2: causal vs anti-causal use. MAE/MFE are anti-causal statistics (computed from the full trade path including future bars). Using them for entry filtering would be look-ahead bias. They are appropriate for design analysis (where the trade path is known after the fact) but not for real-time entry signals. Limit 3: small-sample noise. MAE/MFE distributions from a small number of trades (the article ["Trade-Count Thresholds for Backtest Reliability"](https://aligrithm.com/trade-count-thresholds-for-backtest-reliability/) framed this) are noisy. The 27% / 38% bounceback rate in the opening example assumes 320 trades is enough; for 50 trades, the same statistics would have wide confidence intervals. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Anti-patterns Five mistakes specific to MAE/MFE. Anti-pattern 1: reporting only the aggregate statistics without the MAE/MFE plots. The aggregate statistics hide the trade-level path information. The MAE/MFE plots reveal patterns that determine whether stop losses, profit targets, or trailing stops would improve or destroy the strategy. Anti-pattern 2: using MAE for real-time entry filtering. MAE is anti-causal; using it as if it were known at entry time is look-ahead bias. Anti-pattern 3: setting stops at the IS-data MAE distribution boundary. The 95th percentile of the IS MAE distribution is one realization; the actual 95th percentile (across the larger population the IS sample is drawn from) is wider. Use bootstrap or synthetic-path Monte Carlo (covered in ["Monte Carlo for Trading Systems"](https://aligrithm.com/monte-carlo-for-trading-systems/)) to estimate the operational MAE distribution. Anti-pattern 4: ignoring bar resolution. Daily-bar MAE/MFE understate the actual intra-trade excursions for any strategy with hold time longer than one bar. Decide the bar resolution based on the strategy's monitoring cadence, not on the data convenience. Anti-pattern 5: treating MAE/MFE as a strategy-improvement tool independent of structure. The patterns in the MAE/MFE plots reflect the strategy's mechanism. A stop loss that "fixes" the MAE distribution may break the underlying mechanism. Combine MAE/MFE diagnosis with structural understanding (covered in "Market Personality: Why Gold, FX, Crypto, and Equities Need Different Systems" and ["When a Stop Loss Improves Risk but Destroys Edge"](https://aligrithm.com/when-a-stop-loss-improves-risk-but-destroys-edge/)). ## Decision matrix | MAE/MFE pattern | Interpretation | Action | | ----------------------------------------------------------- | --------------------------- | ---------------------------------------- | | MAE distribution heavy in -1 to -2 ATR with high bounceback | Mean-reversion mechanism | No stop loss or wide stop | | MAE distribution heavy in -2+ ATR with low bounceback | Trend-following mechanism | Stop at 2-3 ATR | | MFE distribution diagonal scatter (realized P&L \~ MFE) | Exits near peaks | No profit target needed | | MFE distribution with realized P&L << MFE | Strategy gives back profits | Trailing stop or earlier profit target | | Strongest signals correlate with smallest MAE | Good entry timing | Continue | | Strongest signals correlate with largest MAE | Entry timing problem | Investigate signal definition | | Outlier trades with extreme MAE / MFE | Specification gap | Investigate per-event | | Small sample (less than 100 trades) | High statistical noise | Add more data before drawing conclusions | The matrix maps MAE/MFE pattern to action. The pattern: the trade-level distribution is the diagnostic tool that aggregate statistics cannot replace. ## The math of MAE-conditional expectancy A useful calculation. The bounceback rate at a given MAE threshold M is the fraction of trades with MAE < -M that end as winners. The conditional expectancy at MAE threshold M is the average realized P&L of trades with MAE < -M. $$ \\text{Bounceback}(M) = \\Pr(\\text{P\\&L} > 0 \\mid \\text{MAE} < -M), \\qquad \\text{Expectancy}(M) = \\mathbb{E}\[\\text{P\\&L} \\mid \\text{MAE} < -M\] $$ For the opening example, Bounceback(2 ATR) approximately 0.38 and Expectancy(2 ATR) approximately +0.05% (the trades that go beyond 2 ATR adverse on average still end up positive by a thin margin). Adding a stop at -2 ATR would convert all of these trades from their actual outcome to a -2 ATR loss; the change in expectancy would be approximately Expectancy(2 ATR) - (-2 ATR) = +0.05% - (-2%) = -2.05% per affected trade, multiplied by the 27% of trades affected = -0.55% per total trade in expectancy. The strategy expectancy would drop from +0.19% to approximately -0.36%; the strategy would become unprofitable. ## Visualizing MAE/MFE ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/05/article_81-visual-0.png) ## KEY POINTS - Maximum Adverse Excursion (MAE) is the largest unrealized loss a trade reached during its life. Maximum Favorable Excursion (MFE) is the largest unrealized gain. Both are computed from the trade's price path; both depend on the bar resolution. - Four standard plots: MAE histogram, MFE histogram, realized P&L vs MAE scatter, realized P&L vs MFE scatter. Each reveals patterns that aggregate statistics cannot. - MAE distribution shape: right-leaning means trades enter at favorable points; left-leaning with long tail means entries are often early and the strategy waits through adverse moves. MFE distribution similarly diagnoses profit-potential profile. - P&L vs MAE scatter shows the bounceback rate (fraction of trades with large MAE that recover to be winners). For mean reversion, bounceback at -1 to -2 ATR is often higher than for smaller MAE. For trend following, the relationship inverts. - P&L vs MFE scatter shows profit capture. Diagonal scatter = strategy exits near peaks. Below-diagonal scatter = strategy gives back profits. - Five operational uses: stop-loss design (where the stop would intervene and what fraction of affected trades would have recovered), profit-target design (where the target would close and what fraction would continue), entry-timing diagnosis (cross-tabulate signal strength with MAE), exit-timing diagnosis (realized P&L vs MFE), outlier identification (specific trades with extreme excursions reveal specification gaps). - The MAE-conditional expectancy quantifies the impact of adding a stop loss at a given level. For the opening example, a -2 ATR stop reduces strategy expectancy by approximately 0.55% per total trade, converting a +0.19% strategy into a -0.36% strategy. - Three honest limits of MAE/MFE: bar resolution materially affects the values, the statistics are anti-causal so they cannot be used for real-time entry filtering, small samples (less than 100 trades) produce noisy distributions. - Anti-pattern: reporting only aggregate statistics without MAE/MFE plots. The plots are the diagnostic tool for path-dependent decisions. - Anti-pattern: using MAE for real-time entry filtering. MAE is anti-causal; using it for entry signals is look-ahead bias. - Anti-pattern: setting stops at IS-data MAE boundary without acknowledging it is one realization. Use bootstrap or synthetic paths for the operational distribution. - Anti-pattern: ignoring bar resolution. Daily MAE understates the actual intra-trade excursions for hold times longer than one bar. - Anti-pattern: treating MAE/MFE as a strategy-improvement tool independent of structure. The patterns reflect the mechanism; "fixing" the MAE distribution can break the underlying signal. - The current article gives the trade-level path diagnostic. The next article in the publication (["Why Profit Factor Can Lie"](https://aligrithm.com/why-profit-factor-can-lie/)) covers a different aggregate statistic that, like the headline net profit, hides important distributional information. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Testing and Tuning Market Trading Systems - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Testing+and+Tuning+Market+Trading+Systems+Timothy+Masters&ref=aligrithm.com) - [Data Mining Algorithms in C++ - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Data+Mining+Algorithms+in+C%2B%2B+Timothy+Masters&ref=aligrithm.com) ### 3.27 When a Stop Loss Improves Risk but Destroys Edge URL: https://aligrithm.com/when-a-stop-loss-improves-risk-but-destroys-edge/ Last updated: 2026-07-27T13:24:16.000Z A mean-reversion strategy on E-mini SPX trades 240 times per year with average winning trade +0.42% and average losing trade -0.55%. The win rate is 64%. The expectancy per trade is approximately +0.07%. The annualized Sharpe before transaction costs is 1.05\. The maximum drawdown over the backtest is 11.3%. The team adds a stop loss at 1.5x the average true range from entry, hoping to reduce drawdown. The new statistics: 240 trades per year, average winning trade +0.40% (slightly lower because some winners are stopped out before reaching their natural exit), average losing trade -0.32% (much lower because losses are capped), win rate 51% (lower because many winners that would have recovered are now stopped out as losses). The expectancy per trade is approximately +0.04%. The annualized Sharpe is 0.78\. The maximum drawdown is 7.4%. The team has reduced the maximum drawdown from 11.3% to 7.4%, a 35% improvement. The team has also reduced the Sharpe from 1.05 to 0.78, a 26% reduction in risk-adjusted return. The team's risk committee approves the change because "drawdown is more visible to the allocator than Sharpe". The strategy is deployed with the stop loss. Three years later, the team's compounded return is approximately 8% lower than it would have been without the stop loss. The pattern is structural to mean-reversion strategies and many other trade-frequent strategies. A stop loss caps the worst outcomes, which lowers the variance and lowers the maximum drawdown. The same stop loss also cuts off winning trades that temporarily moved against the position before recovering, which lowers the win rate and lowers the expectancy. For most mean-reversion strategies the second effect is larger than the first because mean reversion explicitly relies on adverse moves preceding favorable moves. The article ["Why Volatility Is More Non-Stationary Than Trend"](https://aligrithm.com/why-volatility-is-more-non-stationary-than-trend/) framed the volatility-clustering structure that makes mean reversion work; the same structure makes stop losses costly. This article gives the operational framework for when a stop loss helps and when it destroys the edge it is supposed to protect. ## The mechanics of a stop loss Two effects on the trade distribution. Effect 1: truncation of the loss tail. Losses larger than the stop level are converted to losses at the stop level. The mean of the loss distribution is shifted up (less negative); the variance is reduced; the kurtosis is reduced. The drawdown distribution is shifted up (less severe). The risk side of the strategy is unambiguously improved. Effect 2: truncation of the winning trades that temporarily moved against the position. A trade that drops to -1.5 ATR before reversing to +0.4 ATR (a typical mean-reversion winner) is converted from a +0.4 ATR profit into a -1.5 ATR loss. The mean of the win distribution is shifted down. The win rate is reduced. The expectancy is reduced. The trade-off. The first effect helps risk-adjusted metrics. The second effect hurts them. For different strategy types, the balance is different. ## Strategy types and stop-loss responsiveness Five strategy categories with different responses to stop losses. Category 1: mean reversion. Mean reversion explicitly relies on adverse-then-favorable price paths. A stop loss truncates the path before the favorable phase. Mean reversion is the strategy class most damaged by stop losses. Tight stops (e.g., 1 ATR) typically drop the edge to zero. Wider stops (e.g., 3-4 ATR) preserve most of the edge but provide little risk reduction. The right stop loss for mean reversion is often "no stop, only a position-cap-and-time-stop combination". Category 2: trend following. Trend following is more compatible with stop losses because the strategy's expected path is monotonic in the entry direction. A trade that moves more than 2 ATR against the position has a higher loss probability than at entry (the trend has not materialized or has reversed). A stop loss at 2-3 ATR removes losers without truncating many winners. Trend following typically benefits from a stop loss in the form of an ATR-based stop or a trailing stop. Category 3: breakout strategies. Breakouts can be either trend-followers (the breakout continues) or fade-followers (the breakout fails). For breakout-as-trend, the stop-loss response is similar to trend following. For breakout-as-fade, the stop-loss response is similar to mean reversion. The strategy specification determines which response applies. Category 4: factor strategies (cross-sectional). Cross-sectional factor strategies (long top decile, short bottom decile, periodic rebalance) typically do not benefit from per-position stop losses because the strategy's risk profile is determined by the diversification across positions, not the per-position trajectory. A stop loss that exits one position before rebalance creates unnecessary sequencing dependence. Category 5: option strategies. Option strategies have nonlinear payoffs. A stop loss based on underlying price often interacts unexpectedly with option Greeks. Stop losses for option strategies should be based on option-position P&L, not underlying price. ## Cases where a stop loss earns its keep Three operational reasons. Reason 1: catastrophic-event protection. Some trades can lose multiples of their average loss in catastrophic-event scenarios (a single overnight gap, a regime break). A stop loss at 4-6 ATR provides protection against the very worst outcomes without truncating typical losers. The article ["Monte Carlo for Trading Systems"](https://aligrithm.com/monte-carlo-for-trading-systems/) framed the bootstrap-based estimate of the worst-case-loss distribution; a stop loss at the bootstrap 99th percentile of typical losses provides protection without much effect on typical performance. Reason 2: position-cap simulation. A stop loss can be used as a "position cap with a different mechanism": instead of capping the maximum dollar exposure to a single position, cap the maximum dollar loss. The two are equivalent in many cases, and the stop-loss form is more visible to the trader and the risk committee. Reason 3: hard-coded liquidation discipline. For human traders prone to "average down" or "let it run" emotional decisions, a hard stop loss enforces discipline. For systematic strategies the stop loss is a redundant hard-coded version of the strategy's exit rule, which is a different role from a discretionary stop. For systematic strategies on the mean-reversion or factor families, none of the three reasons typically applies, and the stop loss is most likely an unproductive constraint. ## Quantifying the trade-off The question to ask. For a given stop-loss level S (in ATR units), what is the change in expectancy and what is the change in standard deviation per trade? The math. Let the per-trade return distribution without stops have mean mu and standard deviation sigma. The stop loss at S truncates the distribution at -S sigma (if S is in standard-deviation units). The new mean and standard deviation depend on the shape of the distribution. For a normal distribution, the truncation moves the mean up by an amount that depends on the quantile and the tail-mass. For a fat-tailed distribution typical of trading P&L, the move is larger. The Sharpe change depends on the relative magnitudes of the mean shift (positive) and the implicit reduction from cutting winners that would have recovered (negative). $$ \\text{Sharpe}(S) = \\frac{\\mathbb{E}\[r \\mid r \\geq -S \\sigma\]}{\\sqrt{\\text{Var}\[r \\mid r \\geq -S \\sigma\]}} \\cdot \\sqrt{T} $$ For mean-reversion strategies, the per-trade conditional distribution given r >= -S sigma typically has lower mean and lower variance than the unconditional, with the mean reduction larger than the variance reduction. The Sharpe drops. For trend-following strategies, the conditional has higher mean and lower variance. The Sharpe rises. The discipline. Compute Sharpe(S) for a range of S values (e.g., S = 1, 2, 3, 4, 5, infinity) on the actual strategy's trade distribution. Plot. Choose S that maximizes Sharpe (with the structural-prior caveat that very tight stops are usually wrong even if the IS Sharpe says otherwise). For most mean-reversion strategies the maximum is at S = infinity (no stop) or at S = 4+ ATR (very wide stop). For trend-following the maximum is typically at S = 2-3 ATR. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Anti-patterns Five mistakes specific to stop-loss design. Anti-pattern 1: setting the stop loss based on personal pain tolerance, not on the trade distribution. A 1% stop loss on a strategy whose typical adverse excursion is 1.5% truncates most trades into losses. The stop loss should be sized to the strategy's natural distribution, not to the trader's emotional comfort. Anti-pattern 2: claiming the stop loss "improves risk" without measuring the Sharpe change. Drawdown reduction is one risk metric; Sharpe is another; expectancy is a third. A change that improves drawdown but reduces expectancy by more than the drawdown improvement is a net negative. Anti-pattern 3: applying a uniform stop-loss policy across strategy families. Mean reversion and trend following have opposite stop-loss responsiveness. A "all strategies must have a 2-ATR stop" policy is wrong for at least one of the families. Anti-pattern 4: optimizing the stop-loss level on the IS data without robustness checks. The IS-optimal stop loss is often at the boundary of the search range (e.g., the loosest tested) or at a sharp peak. Apply the parameter-stability discipline from ["Parameter Stability Beats Best Parameter"](https://aligrithm.com/parameter-stability-beats-best-parameter/) to the stop-loss parameter. Anti-pattern 5: using a stop loss to "rescue" a losing strategy. If a strategy has near-zero or negative expectancy, adding a stop loss does not create edge. The stop loss truncates the losing distribution but does not produce winners. The right response to a losing strategy is to fix the underlying signal, not to add risk-management constraints around it. ## Decision matrix | Strategy family | Stop-loss responsiveness | Recommended stop | | ------------------------------------------ | ----------------------------------- | ----------------------------------------------- | | Mean reversion (intraday or short-horizon) | Edge-destroying | None or very wide (4-6 ATR) | | Mean reversion (medium-horizon) | Edge-eroding | Wide (3-4 ATR) only for catastrophic protection | | Trend following | Edge-preserving and risk-improving | 2-3 ATR or trailing stop | | Breakout (as trend) | Like trend | 2-3 ATR | | Breakout (as fade) | Like mean reversion | None or very wide | | Cross-sectional factor | Mostly irrelevant | Position cap by exposure, not stop loss | | Option strategies | Nonlinear, requires option-P&L stop | Stop on option-position P&L, not underlying | | Strategy with no demonstrated edge | Stop loss does not help | Fix the signal, not the stop | The matrix maps strategy family to stop-loss recommendation. The pattern: stop losses are not universally beneficial. They help trend-followers and similar momentum strategies; they hurt mean reversion and many factor strategies. ## Visualizing the trade-off ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/05/article_80-visual-0.png) ## KEY POINTS - A stop loss has two effects on a strategy's trade distribution: it truncates the loss tail (improves drawdown, lowers variance) and it converts some winners that temporarily moved against the position into losses (lowers expectancy, lowers win rate). The net effect on Sharpe depends on which effect dominates. - For mean-reversion strategies, the second effect is typically larger than the first because mean reversion relies on adverse-then-favorable price paths. A tight stop (1-1.5 ATR) often destroys the edge; a wide stop (4+ ATR) preserves the edge but provides little risk reduction. - For trend-following strategies, the first effect typically dominates because trades that move 2 ATR or more against the position are unlikely to recover. A stop at 2-3 ATR or a trailing stop typically improves both Sharpe and drawdown. - Five strategy categories with different responses: mean reversion (edge-destroying), trend following (edge-preserving and risk-improving), breakout-as-trend (like trend), breakout-as-fade (like mean reversion), cross-sectional factor (mostly irrelevant; use position cap instead). - Three legitimate uses of a stop loss: catastrophic-event protection (S = 4-6 ATR, only the worst outcomes are truncated), position-cap simulation (limit dollar loss, equivalent to capping exposure), hard-coded liquidation discipline for human traders. - The Sharpe change from a stop loss should be computed on the actual strategy's trade distribution, not assumed. Plot Sharpe vs stop-loss level S; choose the S that maximizes Sharpe with structural-prior caveats. Most mean-reversion strategies maximize at S = infinity or S = 4+ ATR. - Anti-pattern: setting the stop loss based on personal pain tolerance, not on the trade distribution. The stop loss should be sized to the strategy's natural distribution. - Anti-pattern: claiming the stop loss "improves risk" without measuring the Sharpe change. Drawdown reduction without Sharpe verification can be a net negative. - Anti-pattern: applying a uniform stop-loss policy across strategy families. Mean reversion and trend following have opposite responsiveness. - Anti-pattern: optimizing the stop-loss level on IS without robustness checks. Apply parameter-stability discipline. - Anti-pattern: using a stop loss to "rescue" a losing strategy. If the underlying expectancy is near-zero or negative, the stop loss does not create edge. - The risk-management trade-off is concrete and measurable. The team that approved the stop loss in the opening example reduced drawdown by 35% but reduced compounded return by approximately 8% over three years; the trade-off was unfavorable for that strategy. - The current article gives the stop-loss-specific framework. The next article in the publication (["MAE/MFE Analysis: Seeing What Net Profit Hides"](https://aligrithm.com/mae-mfe-analysis-seeing-what-net-profit-hides/)) covers the diagnostic technique for understanding which trades the stop loss would help or hurt before applying it. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Testing and Tuning Market Trading Systems - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Testing+and+Tuning+Market+Trading+Systems+Timothy+Masters&ref=aligrithm.com) - [Data Mining Algorithms in C++ - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Data+Mining+Algorithms+in+C%2B%2B+Timothy+Masters&ref=aligrithm.com) ### 3.26 The Hill, the Spike, and the Cliff: Reading Optimization Surfaces URL: https://aligrithm.com/the-hill-the-spike-and-the-cliff-reading-optimization-surfaces/ Last updated: 2026-07-27T13:24:15.000Z Three strategies are presented to a research review committee. Each has been optimized over the same 2-parameter grid (entry threshold, exit threshold), and each has produced an IS-optimal Sharpe of 1.5\. The committee asks for the parameter-stability map of each, plotted as a heatmap of IS Sharpe over the two-parameter grid. The three heatmaps look very different. Strategy A's heatmap shows a single broad bright region centered at the IS-optimal point, with Sharpes above 1.3 across an area of approximately 30% of the grid and Sharpes above 1.0 across approximately 60% of the grid. The contour lines around the IS-optimum are widely spaced. The shape is a hill: a robust, gently-sloped maximum. Strategy B's heatmap shows a small bright pixel at the IS-optimal point with Sharpe 1.5, surrounded immediately by pixels with Sharpes around 0.7 to 0.9\. The contour lines around the IS-optimum are tightly packed. The bright region is one or two grid cells wide. The shape is a spike: a sharp narrow peak. Strategy C's heatmap shows a bright region with Sharpes above 1.3 occupying approximately 25% of the grid, but the bright region ends abruptly at a sharp boundary. On one side of the boundary the Sharpe is 1.4\. On the other side it drops to -0.3 within one or two grid cells. The IS-optimal point is near the boundary. The shape is a cliff: a flat plateau adjacent to a sharp regime transition. The article ["Parameter Stability Beats Best Parameter"](https://aligrithm.com/parameter-stability-beats-best-parameter/) gave the discipline of choosing the plateau center. This article gives the visual taxonomy of the three common surface shapes (and a few variants), what each implies about the strategy's mechanism, and the operational response to each. The taxonomy is the diagnostic toolkit; the strategy is deployable, fragile, or regime-conditional depending on which shape its surface displays. ## The hill Operational definition. A region where the Sharpe is high relative to the surrounding grid, with gentle gradients in all directions, and the local maximum is approximately at the center of the region. The hill shape, in operational terms. The strategy has a true Sharpe approximately equal to the hill's mean Sharpe (slightly below the peak Sharpe due to mild noise). The IS-optimum is the maximum of the hill, but parameters near the hill center produce similar Sharpes. The strategy is robust to parameter perturbation because the gentle gradient means small parameter shifts produce small Sharpe shifts. Operational response. Choose the parameter at the hill's center, not at the IS-optimum. The center parameter has slightly lower IS Sharpe and a much wider robust neighborhood. The deployment expectation is that OOS Sharpe will be approximately 80-95% of the chosen-parameter IS Sharpe (a small gap, mostly due to general OOS noise rather than parameter fragility). Diagnostic numbers. For a hill, the neighborhood Sharpe (across +/-25% perturbation in each parameter) is at least 80% of the peak Sharpe. The standard deviation of Sharpes across the neighborhood is small (less than 0.15 in absolute terms for a peak Sharpe of approximately 1.0). Perturbing the chosen parameter by +/-25% in any direction changes the Sharpe by less than 15%. Examples. The classic 12-1 momentum factor on equity indices produces a hill. The classic 22-day vol-targeting on G10 FX produces a hill. The 5-day mean reversion on equity intraday produces a hill within a defined parameter range. ## The spike Operational definition. A region where the Sharpe is high at one or two grid cells and drops by more than 30% within one grid step in all directions. The spike shape, in operational terms. The IS-optimum is a peak constructed from a small number of favorable trades that happened to cluster at this specific parameter set. The peak is a search-width artifact; the underlying strategy does not have a true Sharpe at this peak level. Small drifts in the underlying market dynamics will move the strategy off the spike, and the realized Sharpe will be far below the peak. Operational response. Reject the spike-shaped optimum. There is no plateau to fall back to; the strategy as configured is a search-width artifact. Either the strategy specification is noise-fitting by design (drop it) or the parameter grid is too coarse to reveal a real plateau (refine the grid). Re-running with a finer grid and re-mapping the surface is the right next step; only deploying if a hill emerges at finer resolution. Diagnostic numbers. For a spike, the neighborhood Sharpe is less than 50% of the peak Sharpe. The standard deviation of Sharpes across the neighborhood is large (greater than 0.30 for a peak Sharpe of approximately 1.0). Perturbing the chosen parameter by +/-25% drops the Sharpe by more than 40%. Examples. Most "discoveries" from random parameter searches with k\_eff in the thousands or higher produce spikes. The "amazing strategy" found via grid search on a small backtest is almost always a spike. ML-driven strategies without explicit regularization commonly produce spike-shaped surfaces. ## The cliff Operational definition. A region where the Sharpe is high across a substantial area but ends abruptly at a sharp boundary, with the IS-optimum at or near the boundary. The cliff shape, in operational terms. The strategy is effective in the high-Sharpe region and ineffective (or actively losing) on the other side of the boundary. The boundary represents a regime transition: a parameter range where the strategy switches from working to not working, often abruptly. Examples of regime transitions: Regime 1: trend vs mean-reversion. A strategy with very short lookback (e.g., 1-3 days) may be a mean-reversion strategy on a market and produce positive Sharpe; the same strategy at slightly longer lookback (e.g., 5-7 days) on the same market may flip to trend-following character and produce negative or zero Sharpe. The cliff is at the lookback that crosses the autocorrelation sign change. Regime 2: position-cap binding. A strategy with low position cap may be effective; the same strategy with higher position cap may concentrate risk during regime breaks and produce negative skew. The cliff is at the position-cap level where the concentrated-risk events start to dominate. Regime 3: cost crossing. A strategy that trades frequently may be profitable below a certain trade-frequency threshold (transaction costs are tolerable) and unprofitable above (costs swamp signal). The cliff is at the trade frequency where costs cross signal. Operational response. Choose a parameter set well away from the cliff (toward the center of the high-Sharpe region) rather than at the IS-optimum near the boundary. The IS-optimum near the cliff has the highest IS Sharpe but the highest risk of crossing the boundary under regime drift. The center-of-region parameter has slightly lower IS Sharpe but is far from the boundary and survives regime perturbations. Diagnostic numbers. For a cliff, the neighborhood Sharpe in the direction away from the cliff is at least 80% of the peak; the neighborhood Sharpe in the direction toward the cliff drops by more than 50%. The asymmetry is the cliff signature. The article ["Slow Wandering: The Most Dangerous Type of Market Change"](https://aligrithm.com/slow-wandering-the-most-dangerous-type-of-market-change/) framed the regime-drift problem; a cliff-positioned parameter is precisely the configuration most vulnerable to slow regime drift. Examples. Strategies whose lookback crosses the autocorrelation-sign transition. Strategies whose volatility threshold sits at the boundary between vol regimes. Strategies whose cost assumption crosses the break-even line. ## Three less common surface shapes Variant 1: the saddle. A surface with a saddle point: high in two directions and low in two others. The IS-optimum at a saddle is unstable in the directions where the Sharpe drops. Treat saddles like cliffs in the directions of decline. Variant 2: the maze. A surface with multiple disconnected high-Sharpe regions separated by valleys. Each region may correspond to a different regime where the strategy works for a different reason. A maze surface suggests the strategy is several strategies overlaid; identify which region matches the deployed regime and deploy only on that. Variant 3: the slope. A surface with no clear maximum, only a monotonic increase or decrease toward a boundary of the search range. The IS-optimum is at the boundary. The slope is a sign that the search range was insufficient; extend the range until the surface peaks within the searched region. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Diagnostic protocol Five-step procedure for reading any optimization surface. Step 1: visualize the surface. For 2 parameters, heatmap or contour. For 3+ parameters, show pairwise marginal heatmaps with other parameters fixed at the candidate plateau center. Step 2: identify the IS-optimum and mark it. The IS-optimum is one specific point on the surface. Mark it as a starting reference. Step 3: characterize the surrounding region. Compute neighborhood Sharpe (mean of Sharpes within +/-25% perturbation in each parameter). Compute neighborhood standard deviation. Compute gradient (rate of change of Sharpe near the IS-optimum). Step 4: classify by shape. Hill = high neighborhood mean, low neighborhood SD, gentle gradient. Spike = low neighborhood mean, high neighborhood SD, steep gradient. Cliff = high neighborhood mean in one direction, low in the opposite direction (asymmetric gradient). Step 5: choose action by shape. Hill = pick the center, deploy. Spike = reject or refine grid. Cliff = pick a parameter away from the cliff, toward the center of the high-Sharpe region. Deploy with regime monitoring. ## The math of why shapes predict OOS A simple model. Let SR\_true(theta) be the true Sharpe surface and IS\_Sharpe(theta) be the IS estimate. The IS estimate equals the true Sharpe plus mean-zero noise with magnitude depending on the per-cell sample size. The shape of the IS surface reflects the shape of the true surface plus the noise's contribution. $$ \\widehat{\\text{SR}}\_{\\text{IS}}(\\theta) = \\text{SR}\_{\\text{true}}(\\theta) + \\varepsilon(\\theta) $$ For a hill (broad smooth true surface), the noise contribution is small relative to the true surface, and the IS shape closely matches the true shape. The IS-optimum is a few percent above the hill's true Sharpe, and OOS realizations track. For a spike (no true Sharpe at the location, only noise), the spike is the maximum of the noise field over k\_eff comparisons, and the OOS realization (drawn from the true Sharpe at that location, which is approximately zero) is far below. For a cliff (true Sharpe surface has a regime transition), small drifts in regime move the cliff location, and a parameter at the cliff edge is most sensitive to the drift. ## Anti-patterns Five mistakes specific to surface reading. Anti-pattern 1: not plotting the surface. Skipping the visualization is the most common mistake. Without the surface map, hills, spikes, and cliffs are indistinguishable. The map is the minimum required diagnostic. Anti-pattern 2: trusting the IS-optimum on a spike-shaped surface. The spike's IS Sharpe is a search-width artifact. Deploying the spike's parameter is deploying noise. Anti-pattern 3: deploying at the cliff edge. The IS-optimum at the cliff has the highest IS Sharpe but the worst regime-drift sensitivity. Move toward the center of the high-Sharpe region. Anti-pattern 4: refining the grid only at the optimum. Refinement should cover the full neighborhood to confirm the shape. Refining only at the optimum cannot distinguish a spike (whose surrounding cells remain low) from a hill (whose surrounding cells remain high). Anti-pattern 5: not re-mapping the surface periodically. Surface shapes can change over time as the underlying dynamics drift. A strategy that was on a hill in 2015 may be on a cliff in 2024\. Re-map at the same cadence as walk-forward validation. ## Decision matrix | Surface shape | Neighborhood diagnostic | Deployment action | | ----------------------- | ------------------------------------------------------- | ------------------------------------------------------------- | | Hill | High mean, low variance, gentle gradient | Pick the center, deploy with standard monitoring | | Spike | Low mean, high variance, steep gradient | Reject or refine grid | | Cliff | High mean one side, low other side, asymmetric gradient | Pick parameter away from cliff, deploy with regime monitoring | | Saddle | Mixed: high in some directions, low in others | Treat as cliff in the decline directions | | Maze | Multiple disconnected high-Sharpe regions | Match region to deployed regime, ignore others | | Slope | Monotonic toward boundary of search | Extend search range, re-map | | No surface map produced | Unknown, treat as worst case | Reject pending mapping | The matrix maps surface shape to action. The pattern: produce the surface map, classify the shape, act accordingly. ## Visualizing the three shapes ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/05/article_79-visual-0.png) ## KEY POINTS - Optimization surfaces fall into three common shapes: hill (broad, gentle gradient, robust), spike (narrow peak, surrounded by drops, search-width artifact), cliff (wide region ending at a sharp boundary, regime transition). - Hill shape: pick the center, deploy with standard monitoring. The IS-optimum is approximately at the hill's center; the neighborhood Sharpe is at least 80% of peak; perturbations of +/-25% drop Sharpe by less than 15%. - Spike shape: reject or refine the grid. The neighborhood Sharpe is less than 50% of peak; perturbations drop Sharpe by more than 40%; the spike is a search-width artifact with no true Sharpe at the location. - Cliff shape: choose a parameter away from the boundary, toward the center of the high-Sharpe region. The neighborhood Sharpe is asymmetric: high in the direction away from the cliff, low toward it. The cliff is most vulnerable to slow regime drift. - Three less common shapes: saddle (high in some directions, low in others; treat as cliff in the decline directions), maze (multiple disconnected high-Sharpe regions, often regime-specific; match region to deployed regime), slope (monotonic toward boundary, signals insufficient search range; extend the range and re-map). - Five-step diagnostic protocol: visualize the surface, mark the IS-optimum, characterize the surrounding region (neighborhood Sharpe, SD, gradient), classify by shape, choose action by shape. - The IS Sharpe at the optimum tells you nothing about deployment without the surface shape. Three strategies with the same IS Sharpe of 1.5 may have OOS Sharpes of 1.10 (hill), 0.20 (spike), or 1.05 if positioned correctly off the cliff (cliff). - Anti-pattern: not plotting the surface. Without the visualization, all three shapes are indistinguishable. - Anti-pattern: trusting the IS-optimum on a spike. The spike's Sharpe is noise; deploying the spike's parameter deploys noise. - Anti-pattern: deploying at the cliff edge. Move toward the center of the high-Sharpe region. - Anti-pattern: refining the grid only at the optimum. Refinement should cover the full neighborhood to confirm shape. - Anti-pattern: not re-mapping the surface periodically. Shapes drift; a hill in 2015 may be a cliff in 2024. - The current article gives the visual taxonomy of optimization surfaces. The next article in the publication (["When a Stop Loss Improves Risk but Destroys Edge"](https://aligrithm.com/when-a-stop-loss-improves-risk-but-destroys-edge/)) begins the shift from validation discipline to specific risk-management trade-offs that show up in real strategy design. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Testing and Tuning Market Trading Systems - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Testing+and+Tuning+Market+Trading+Systems+Timothy+Masters&ref=aligrithm.com) - [Data Mining Algorithms in C++ - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Data+Mining+Algorithms+in+C%2B%2B+Timothy+Masters&ref=aligrithm.com) ### 3.25 Parameter Stability Beats Best Parameter URL: https://aligrithm.com/parameter-stability-beats-best-parameter/ Last updated: 2026-07-27T13:24:15.000Z A team optimizes a momentum strategy on a basket of futures. The optimization searches over entry-threshold values from 0.5 to 3.0 sigma in steps of 0.1, producing 26 candidate values. The IS-best entry threshold is 1.4 sigma, with IS Sharpe 1.62\. The team plots the IS Sharpe as a function of entry threshold and observes the curve: the Sharpe is approximately 0.8 at threshold 0.5, climbs across the range, peaks at 1.62 at threshold 1.4, drops to 1.20 at threshold 1.5 (a 26% Sharpe loss from a 0.1-sigma move), and falls to 0.7 at threshold 2.0\. The peak at 1.4 is narrow and isolated. The team deploys with threshold 1.4\. Three years later, the live Sharpe is 0.55\. The team's post-mortem traces the underperformance to the parameter being on a sharp peak; small drifts in the underlying volatility shifted the strategy's effective entry threshold (in standardized units) and the strategy operated on the steep edge of the peak rather than at its top. A different team optimizes the same strategy on the same data and arrives at a different conclusion. Their plot shows the IS Sharpe vs entry threshold, identifies the same peak at 1.4 with Sharpe 1.62, but also notices a wider plateau between thresholds 1.8 and 2.4 where the Sharpe is approximately 1.30 with very gentle gradient. They choose threshold 2.1 (the center of the plateau) and accept an IS Sharpe of 1.32 instead of the peak's 1.62\. The deployment three years later produces a live Sharpe of 1.10\. The plateau choice survived the underlying drift; the peak choice did not. The article ["The Difference Between Robustness and Optimization"](https://aligrithm.com/the-difference-between-robustness-and-optimization/) framed the conceptual difference between climbing the peak and choosing the plateau. This article gives the operational tools for parameter selection: how to map the optimization surface, how to identify plateaus versus peaks, and how to select the parameter from the plateau region rather than from the IS maximum. The technique is older than the academic literature on it; experienced systematic traders have been using parameter-stability mapping for decades, even if the formal frameworks are recent. The article ["The Hill, the Spike, and the Cliff: Reading Optimization Surfaces"](https://aligrithm.com/the-hill-the-spike-and-the-cliff-reading-optimization-surfaces/) later in this pillar gives the visual taxonomy of surface shapes; the present article gives the selection discipline. ## Stability, defined operationally Three measurable properties of a candidate parameter set. Property 1: neighborhood Sharpe. Define a neighborhood around the candidate parameter (e.g., +/-25% of the parameter value, or the nearest 5 grid points on each side). Compute the average IS Sharpe across all parameter sets in the neighborhood. The neighborhood Sharpe is the right number to compare across candidates; a candidate with high point Sharpe but low neighborhood Sharpe is on a peak. A candidate with similar point Sharpe and similar neighborhood Sharpe is on a plateau. $$ \\bar{\\text{SR}}\_{\\mathcal{N}}(\\theta\_0) = \\frac{1}{|\\mathcal{N}(\\theta\_0)|} \\sum\_{\\theta \\in \\mathcal{N}(\\theta\_0)} \\widehat{\\text{SR}}\_{\\text{IS}}(\\theta) $$ The neighborhood N(theta\_0) is the set of parameter values within the chosen perturbation radius. Property 2: Sharpe gradient. The slope of the IS Sharpe surface at the candidate parameter, in each parameter direction. A flat gradient means the parameter is at a plateau. A steep gradient means the parameter is on the side of a peak (which is fragile) or at the bottom of a valley (which is also fragile in the opposite direction). Property 3: Sharpe variance in the neighborhood. The variance of the Sharpes in the neighborhood. A small variance with a high mean is a plateau (good). A small variance with a low mean is a flat-bad region (avoid). A large variance is a peak-and-cliff region (avoid). The selection rule. Choose the parameter set that maximizes the neighborhood Sharpe, subject to a minimum variance threshold (the plateau is wide enough to be informative). Reject parameters with point Sharpe much higher than neighborhood Sharpe (peaks). ## Mapping the optimization surface Five practical steps. Step 1: choose the parameter axes to map. Most strategies have 2-5 parameters that matter for the optimization. Pick the parameters; ignore parameters with structurally fixed values (e.g., a position-cap from risk policy is not a parameter to optimize over). Step 2: define a grid for each parameter. The grid should span the structurally plausible range with resolution fine enough to distinguish plateaus from peaks. For a numerical parameter like entry threshold, 25-50 grid points across the plausible range is typical. Step 3: run the strategy at each grid point. For 2 parameters with 25 grid points each, the surface has 625 points. For 3 parameters, 15625 points. Computational cost grows quickly; for 4+ parameters, use a Latin hypercube or similar sparse sampling rather than full grid. Step 4: visualize the surface. For 1 parameter, plot Sharpe vs parameter value (a line plot). For 2 parameters, plot the surface as a heatmap or contour plot. For 3+ parameters, plot the marginal surfaces (Sharpe vs each parameter, with others held at their plateau-center) or use slice plots. Step 5: identify candidate plateau regions. Look for areas where the Sharpe is approximately constant across a wide range. Mark the center of each plateau. Compute the neighborhood Sharpe and variance at each candidate. Choose the plateau with highest neighborhood Sharpe. ## Cases where the plateau loses IS Sharpe The trade-off explicitly. The plateau-center parameter typically has IS Sharpe 10-30% below the peak parameter. The OOS Sharpe of the plateau choice is typically 5-15% below its IS Sharpe. The OOS Sharpe of the peak choice is typically 40-70% below its IS Sharpe. For typical search-width and noise levels, the plateau OOS exceeds the peak OOS, even though the peak IS exceeds the plateau IS. A simple model. Let SR\_peak = SR\_true\_peak + noise\_peak, SR\_plateau = SR\_true\_plateau + noise\_plateau. The peak's noise\_peak is large because the peak is the maximum over the local search; the plateau's noise\_plateau is small because the plateau is averaged over many comparable parameters. $$ \\mathbb{E}\[\\text{SR}\_{\\text{OOS}}^{\\text{plateau}}\] - \\mathbb{E}\[\\text{SR}\_{\\text{OOS}}^{\\text{peak}}\] = (\\text{SR}\_{\\text{true}}^{\\text{plateau}} - \\text{SR}\_{\\text{true}}^{\\text{peak}}) + \\text{search-width-bias}\_{\\text{peak}} $$ The first term is typically negative (the peak's true Sharpe is slightly higher than the plateau's). The second term is typically positive and large (the peak inherits the search-width bias). For most strategies, the second term dominates, and the plateau wins OOS. The discipline. Accept the lower IS Sharpe in exchange for the higher OOS Sharpe. The article ["Optimization Comes After Testing, Not Before"](https://aligrithm.com/optimization-comes-after-testing-not-before/) framed the procedural sequence; parameter-stability selection is the Stage 4 small-final-optimization that anchors to the plateau rather than the peak. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Reporting parameter selection Three artifacts that should accompany the chosen parameter. Artifact 1: the parameter-stability map. The plot of IS Sharpe across the parameter range, with the peak and the chosen plateau marked. The map allows the reader to verify the plateau exists and to assess how flat it is. Artifact 2: the neighborhood Sharpe and its variance. The chosen parameter's neighborhood Sharpe, the peak's neighborhood Sharpe, and the difference. Any reader should see that the chosen parameter has a higher neighborhood Sharpe than the peak (the peak's narrow neighborhood pulls the average down by 0.3 to 0.5 Sharpe units). Artifact 3: the perturbation table. A table showing the IS Sharpe at the chosen parameter and at +/-10%, +/-25%, +/-50% perturbations. A robust choice has gentle declines under perturbation. A fragile choice has steep declines. The right deployment threshold: the chosen parameter loses no more than 25% of its IS Sharpe under +/-25% perturbation. ## Anti-patterns Five mistakes specific to parameter selection. Anti-pattern 1: reporting only the IS-optimal parameter without the surface. The IS optimum is one number from a continuous surface. Without the surface, the reader cannot tell whether the optimum is on a plateau (robust) or a peak (fragile). The surface map is the minimum required artifact. Anti-pattern 2: confusing a plateau with a flat-bad region. A range of parameter values producing similar low Sharpes is a flat-bad region where the strategy does not work. A plateau requires high mean Sharpe. The plateau must have high mean Sharpe with low variance. Flat-bad regions have low mean Sharpe with low variance. Anti-pattern 3: choosing the plateau and then optimizing within it. The plateau center is the choice; further optimization within the plateau re-introduces search-width bias. Pick the center and stop. Anti-pattern 4: applying parameter-stability mapping only on one parameter while optimizing the others. The mapping is multi-dimensional. Mapping only the entry-threshold while optimizing the exit-threshold and the vol-target leaves the strategy on a peak in the joint surface. The mapping should cover all parameters that matter. Anti-pattern 5: ignoring the structural prior in plateau selection. Among multiple plateaus on a surface, choose the one whose center is closest to the structural prior (academic literature, mechanism, cross-asset analog). Choosing the highest-Sharpe plateau without regard to structural prior is a softer version of search-width-bias inflation. ## Decision matrix | Surface diagnostic | Interpretation | Action | | -------------------------------------------------------- | ----------------------------------------------- | ------------------------------------------------------------------------- | | Single wide plateau, high mean, low variance | Robust strategy, single clear choice | Pick plateau center | | Single narrow peak, high mean, with cliffs on both sides | Fragile strategy, peak is search-width artifact | Reject; search the surface for a plateau or accept exploratory status | | Multiple plateaus | Strategy has regime-dependent optima | Choose the plateau closest to structural prior, deploy with regime gating | | Wide flat-bad region (high variance, low mean) | Strategy does not work in this parameter range | Search elsewhere | | Steep gradient with no clear plateau | Strategy is on a slope, not at an optimum | Likely has fundamental design issue | | IS-optimum loses >25% Sharpe under +/-25% perturbation | Fragile choice | Reject | | IS-optimum loses <10% Sharpe under +/-25% perturbation | Robust choice | Deploy | | Neighborhood Sharpe > 80% of peak Sharpe | Plateau is broad enough | Deploy at plateau center | The matrix maps surface shape to action. The pattern: map the surface, find the plateau, choose the center. ## Visualizing parameter-stability mapping ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/05/article_78-visual-0.png) ## KEY POINTS - Parameter selection should choose the center of a plateau in the IS-Sharpe surface, not the IS maximum. The plateau choice typically has IS Sharpe 10-30% below the peak but OOS Sharpe 40-70% above the peak's OOS, because the peak inherits the search-width bias and the plateau does not. - Three measurable properties of a candidate parameter: neighborhood Sharpe (mean across +/-25% or nearest 5 grid points), Sharpe gradient (slope at the candidate, flat for plateau, steep for peak), Sharpe variance in the neighborhood (small variance + high mean = plateau). - The selection rule: maximize neighborhood Sharpe subject to a minimum-variance threshold. Reject candidates with point Sharpe much higher than neighborhood Sharpe (these are peaks). - Five steps to map the surface: choose parameter axes, define grids, run the strategy at each grid point, visualize as line / heatmap / marginal plots, identify plateau regions and pick the highest-mean plateau. - The OOS-Sharpe gap between the plateau choice and the peak choice is approximately the search-width bias of the peak minus the small true-Sharpe gap between plateau and peak. For typical bias magnitudes, the plateau wins OOS by a substantial margin. - Three required reporting artifacts: the parameter-stability map (the plot of IS Sharpe across parameters), the neighborhood Sharpe of the chosen parameter (versus the peak), the perturbation table (IS Sharpe at +/-10%, +/-25%, +/-50%). - Operational deployment threshold: the chosen parameter should lose no more than 25% of its IS Sharpe under +/-25% perturbation. A robust choice loses <10%; a fragile choice loses >40%. - Anti-pattern: reporting only the IS-optimum without the surface. The surface is the minimum required artifact for assessing robustness. - Anti-pattern: confusing a flat-bad region with a plateau. A plateau has high mean Sharpe and low variance; a flat-bad region has low mean and low variance. - Anti-pattern: optimizing within the plateau after choosing it. Pick the center and stop; further optimization re-introduces bias. - Anti-pattern: stability mapping on one parameter while optimizing others. The mapping is multi-dimensional. - Anti-pattern: ignoring structural prior in plateau selection. Among multiple plateaus, choose the one closest to the prior. - The current article gives the parameter-selection discipline. The next article in the publication (["The Hill, the Spike, and the Cliff: Reading Optimization Surfaces"](https://aligrithm.com/the-hill-the-spike-and-the-cliff-reading-optimization-surfaces/)) gives the visual taxonomy of common surface shapes and what each implies for the strategy. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Testing and Tuning Market Trading Systems - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Testing+and+Tuning+Market+Trading+Systems+Timothy+Masters&ref=aligrithm.com) - [Data Mining Algorithms in C++ - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Data+Mining+Algorithms+in+C%2B%2B+Timothy+Masters&ref=aligrithm.com) ### 3.24 Why Walk-Forward Testing Is Better Than One Big OOS Split URL: https://aligrithm.com/why-walk-forward-testing-is-better-than-one-big-oos-split/ Last updated: 2026-07-27T13:24:14.000Z A team validates a strategy using a 70-30 split: 14 years of IS (2004-2017), 6 years of OOS (2018-2023). The IS Sharpe is 0.95 with the optimized parameters. The OOS Sharpe is 0.62\. The team interprets the OOS as confirmation: "the strategy generalizes, OOS Sharpe is positive by a margin, deploy". The strategy ships in 2024 and produces a Sharpe of approximately 0.15 in the first year. The post-mortem reveals the problem with the single-split design. The 6-year OOS window happened to span 2018-2023, which included a benign equity environment with a brief 2020 disruption that recovered quickly and a 2022 drawdown that the strategy navigated reasonably. The OOS Sharpe of 0.62 was one realization of the strategy's performance; if the OOS had been 2008-2013 (financial-crisis aftermath), the realization would have been different. If the OOS had been 1999-2004 (dot-com bust), different again. The team had no way to know which OOS realization was representative because they had only one. The single-split design produces a single OOS Sharpe number, and the team treats it as the OOS estimate, when it is one draw from a wide distribution of possible OOS outcomes. Walk-forward testing produces many such draws. The procedure: divide the data into multiple consecutive IS-OOS pairs, each shorter than the single-split design. For each pair, optimize on the IS portion, evaluate on the OOS portion, then roll the window forward and repeat. The result is a sequence of OOS Sharpes, one per walk-forward step, that approximates the strategy's performance under the cadence at which it would be redeployed. The article ["CSCV: A Direct Probability of Backtest Overfit"](https://aligrithm.com/cscv-a-direct-probability-of-backtest-overfit/) gave a combinatorial alternative; this article covers the time-respecting alternative that simulates real-time deployment. Both reduce the variance of the OOS estimate compared to the single-split design; walk-forward additionally captures the temporal dynamics of strategy decay and adaptation. ## Walk-forward testing, mechanics Five steps with concrete numbers. Step 1: choose the in-sample window length and the out-of-sample window length. Common choices: IS = 3 years, OOS = 6 months. Or IS = 5 years, OOS = 1 year. The IS window must be long enough to support meaningful parameter optimization (typically 100+ effective trades; the article ["Trade-Count Thresholds for Backtest Reliability"](https://aligrithm.com/trade-count-thresholds-for-backtest-reliability/) framed the threshold). The OOS window must be long enough to produce a meaningful Sharpe estimate per step (typically 30+ trades, with the standard-error-of-Sharpe interpretation from the same article). Step 2: anchored or rolling. Anchored walk-forward keeps the IS window starting at the same date and grows it; the IS window at step k is from date 0 to date k \* OOS\_length. Rolling walk-forward keeps the IS window length fixed; the IS window slides forward at each step. Rolling is more responsive to drift but uses less data per step. Anchored uses more data but is slower to adapt. For most strategies, rolling with a 3-5 year IS is the operational standard. Step 3: at each step, optimize on IS and run on OOS. The optimization uses whatever discipline the article ["The Difference Between Robustness and Optimization"](https://aligrithm.com/the-difference-between-robustness-and-optimization/) framed (parameter-stability mapping, plateau selection, structural-prior anchoring). The OOS evaluation uses the chosen parameters with no further adjustment. The OOS Sharpe is recorded. Step 4: roll the window forward by OOS\_length. With OOS = 6 months, the window advances 6 months. The next IS window now includes the previous OOS period (or, for rolling, the IS shifts forward 6 months). Step 5: aggregate the results. The walk-forward Sharpe is computed from the concatenated OOS returns across all steps, not from the average of the per-step Sharpes. The walk-forward Sharpe is one number; the per-step Sharpes are the diagnostic distribution. For 20 years of data with IS = 5 years and OOS = 1 year (rolling), the walk-forward produces 15 IS-OOS pairs (the first IS uses years 0-5, the first OOS is year 5; the next IS uses years 1-6, the next OOS is year 6; etc.). The 15 OOS Sharpes are the per-step distribution; the concatenated 15-year OOS return series produces the walk-forward Sharpe. ## Walk-forward's additions over single-split Four diagnostic outputs. Output 1: per-step Sharpe distribution. The 15 per-step Sharpes are 15 realizations of the OOS Sharpe. The mean and standard deviation of the 15 give a confidence interval that is much tighter than the standard error of a single OOS Sharpe. The article ["Trade-Count Thresholds for Backtest Reliability"](https://aligrithm.com/trade-count-thresholds-for-backtest-reliability/) gave the single-OOS-Sharpe SE; the walk-forward SE is approximately the single-step SE divided by sqrt(15), assuming the per-step Sharpes are approximately independent. $$ \\text{SE}(\\overline{\\text{SR}}\_{\\text{WF}}) \\approx \\frac{\\text{SE}(\\text{SR}\_{\\text{single OOS}})}{\\sqrt{N\_{\\text{steps}}}} $$ Output 2: temporal trend in performance. If the per-step Sharpes decline monotonically across the walk-forward sequence, the strategy is decaying. The article ["Why Systems Work Until They Don't"](https://aligrithm.com/why-systems-work-until-they-dont/) framed the four mechanisms; the walk-forward trend visualization detects the decay before deployment. A flat per-step distribution is consistent with stable performance; a declining distribution is a deployment red flag. Output 3: regime-conditional behavior. The per-step Sharpes can be cross-tabulated with the regime classification of each OOS window (bull/bear, low-vol/high-vol, etc.). The article ["Regime Coverage: Why Your Backtest Needs Different Market States"](https://aligrithm.com/regime-coverage-why-your-backtest-needs-different-market-states/) framed the regime stratification; walk-forward composes with it because each step has a known calendar period and the regime of that period is known. Output 4: parameter-stability over time. The optimized parameters at each walk-forward step can be compared. If the parameters change across a wide range from step to step (e.g., the optimal lookback wanders from 14 to 42 to 18 to 36 across steps), the strategy is sensitive to the IS sample and the parameter "optimization" is fitting noise. The article "Parameter Stability Beats Best Parameter" framed the static version of this diagnostic; walk-forward gives the temporal version. ## Walk-forward's limits Three honest limits. Limit 1: walk-forward does not eliminate search-width bias. Each walk-forward step performs an optimization on its IS, which has the same search-width bias as a one-time optimization. The article ["Degrees of Freedom in Trading Systems"](https://aligrithm.com/degrees-of-freedom-in-trading-systems/) framed the bias; the per-step Sharpe at each walk-forward step is biased upward by the search-width contribution of that step's optimization. The OOS realization is unbiased relative to the IS-optimal parameter, but the IS-optimal parameter itself was chosen with bias. The mitigation. Use parameter-stability mapping or bias-corrected parameter selection at each walk-forward step. The article ["The Difference Between Robustness and Optimization"](https://aligrithm.com/the-difference-between-robustness-and-optimization/) framed the discipline; walk-forward applies it once per step. Limit 2: walk-forward depends on the IS and OOS window-length choices. Different IS / OOS combinations produce different walk-forward results. The choice is a degree of freedom that the team can implicitly optimize. The mitigation: report walk-forward results under multiple IS / OOS combinations and check sensitivity. A strategy whose walk-forward Sharpe varies by more than 30% across window choices is more fragile than one whose results stay within a tight band. Limit 3: walk-forward inherits the regime distribution of the historical data. If the historical data has no 2008-style crisis, the walk-forward has no 2008-style steps. The article ["Regime Coverage: Why Your Backtest Needs Different Market States"](https://aligrithm.com/regime-coverage-why-your-backtest-needs-different-market-states/) framed the synthesis-or-substitution remedies; walk-forward does not fix coverage gaps. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Walk-forward vs single OOS split, formally Three structural differences. Difference 1: number of OOS realizations. Single split = 1\. Walk-forward (20 years, IS=5, OOS=1) = 15. Difference 2: variance of the OOS Sharpe estimate. Single split SE depends on N\_OOS\_trades. Walk-forward SE is approximately the single-step SE divided by sqrt(N\_steps). For typical setups, walk-forward reduces the SE by a factor of 3-4. Difference 3: ability to detect time-varying performance. Single split = no. Walk-forward = yes, through the per-step trend. The point. Walk-forward is strictly more informative than a single OOS split, at the cost of more computation (each step requires its own optimization) and the additional choices of IS / OOS window lengths. For deployment-critical strategies, walk-forward is the standard; for exploratory research, the single split is acceptable as a first-pass screen. ## Anti-patterns Five mistakes specific to walk-forward. Anti-pattern 1: averaging the per-step Sharpes instead of computing the concatenated-OOS Sharpe. The average of 15 per-step Sharpes is not the same as the Sharpe of the 15-year OOS return series. The latter accounts for the variance contribution of each step weighted by trade count; the former does not. Report the concatenated Sharpe, with the per-step Sharpes as the diagnostic distribution. Anti-pattern 2: optimizing the IS and OOS window lengths to maximize the walk-forward Sharpe. Each window-length choice is a degree of freedom. Choosing the combination that produces the best walk-forward result is overfitting at the meta level. Pre-specify the windows from structural priors (typical: IS 3-5 years for daily strategies, OOS 6-12 months) and report the result. Anti-pattern 3: ignoring the per-step trend. A walk-forward Sharpe of 0.6 with a strong declining trend (year-1 step Sharpe 1.2, year-15 step Sharpe -0.1) is not the same as a walk-forward Sharpe of 0.6 with a flat distribution. The first is decaying; the second is stable. Report the trend. Anti-pattern 4: using walk-forward as a substitute for CSCV. Walk-forward simulates real-time cadence and detects time-varying behavior. CSCV measures parameter-search overfit. They are different diagnostics, both useful, neither replacing the other. Anti-pattern 5: re-running walk-forward with adjusted parameters when the result disappoints. A team runs walk-forward, sees a Sharpe of 0.4, adjusts the strategy specification, re-runs, sees 0.6, adjusts again, re-runs, sees 0.8\. The walk-forward is now part of the IS optimization. The right discipline: define the strategy specification before running walk-forward, run once, accept the result. ## Decision matrix | Walk-forward result | Interpretation | Action | | ---------------------------------------------------- | ---------------------------------------- | -------------------------------------------------------- | | Concatenated Sharpe high, flat per-step trend | Stable strategy, robust across regimes | Deploy | | Concatenated Sharpe high, declining per-step trend | Decaying strategy | Investigate decay mechanism, do not deploy on raw Sharpe | | Concatenated Sharpe moderate, high per-step variance | Regime-sensitive strategy | Add regime gating, redeploy | | Concatenated Sharpe low, declining trend | Strategy is in late lifecycle | Reject | | Per-step parameters change across a wide range | Optimization is fitting noise | Apply parameter-stability discipline, simplify | | Concatenated Sharpe matches single-OOS Sharpe | Single-split was representative | Both diagnostics consistent, deploy | | Concatenated Sharpe lower than single-OOS Sharpe | Single-split was favorable realization | Trust walk-forward, deploy with smaller capital | | Concatenated Sharpe higher than single-OOS Sharpe | Single-split was unfortunate realization | Trust walk-forward, deploy with confidence | The matrix maps walk-forward output to action. The pattern: the per-step distribution and trend are as important as the headline concatenated Sharpe. ## Visualizing walk-forward ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/05/article_77-visual-0.png) ## KEY POINTS - Walk-forward testing divides the data into multiple consecutive IS-OOS pairs, each shorter than a single-split design. For each pair, the strategy is optimized on IS and evaluated on OOS, then the window rolls forward and the procedure repeats. - The result is a sequence of per-step OOS Sharpes (typically 10-20 for a multi-year backtest) plus a concatenated walk-forward Sharpe computed from the joined OOS return series. The concatenated Sharpe has standard error approximately equal to the single-step SE divided by sqrt(N\_steps), which is 3-4x tighter than a single-OOS-split SE. - Two flavors: anchored walk-forward (IS window starts at date 0 and grows; uses more data, slower to adapt) and rolling walk-forward (IS window length is fixed and slides forward; less data per step, more responsive to drift). Rolling with 3-5 year IS is the operational standard for most strategies. - Four diagnostic outputs: per-step Sharpe distribution (gives confidence interval), temporal trend (detects decay), regime-conditional behavior (cross-tabulate with regime classification), parameter stability over time (compare optimized parameters across steps). - Walk-forward does not eliminate search-width bias at each step. Each step's optimization has the same bias as a one-time optimization. The mitigation is parameter-stability discipline at each step. - Walk-forward depends on the IS and OOS window-length choices. Pre-specify the windows from structural priors; do not optimize the windows to maximize the walk-forward Sharpe. - Walk-forward inherits the regime distribution of the historical data. It does not fix coverage gaps; combine with synthesis-or-substitution remedies for rare regimes. - Walk-forward is strictly more informative than a single OOS split: more OOS realizations, tighter SE, ability to detect time-varying behavior. The cost is computation and additional window-length choices. - Walk-forward and CSCV are different diagnostics. Walk-forward simulates real-time cadence and detects temporal patterns. CSCV measures parameter-search overfit by combinatorial enumeration. Both are useful; neither replaces the other. - Anti-pattern: averaging per-step Sharpes instead of computing the concatenated-OOS Sharpe. Report the concatenated Sharpe with the per-step distribution as diagnostic. - Anti-pattern: optimizing IS and OOS window lengths to maximize the walk-forward Sharpe. Pre-specify from structural priors. - Anti-pattern: ignoring the per-step trend. A flat distribution at Sharpe 0.6 is different from a declining distribution at the same headline Sharpe. - Anti-pattern: re-running walk-forward with adjusted parameters when the result disappoints. The walk-forward becomes part of the IS optimization. - The current article gives the time-respecting validation technique. The next article in the publication (["Parameter Stability Beats Best Parameter"](https://aligrithm.com/parameter-stability-beats-best-parameter/)) covers the parameter-selection discipline that should be applied at each walk-forward step rather than relying on the IS-optimum. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Testing and Tuning Market Trading Systems - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Testing+and+Tuning+Market+Trading+Systems+Timothy+Masters&ref=aligrithm.com) - [Data Mining Algorithms in C++ - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Data+Mining+Algorithms+in+C%2B%2B+Timothy+Masters&ref=aligrithm.com) ### 3.23 CSCV: A Direct Probability of Backtest Overfit URL: https://aligrithm.com/cscv-a-direct-probability-of-backtest-overfit/ Last updated: 2026-07-27T13:24:14.000Z A research team optimizes a strategy across 50 candidate parameter sets, picks the IS-best, reports the IS Sharpe of 1.7, and runs an OOS hold-out that produces a Sharpe of 0.4\. The team's question: was the OOS Sharpe of 0.4 a bad sample (the strategy is good but the OOS slice was unfortunate), or was the IS Sharpe of 1.7 the bad sample (the strategy is overfit and the OOS reveals the true capacity)? The standard methods covered earlier in this pillar give partial answers. The article ["Degrees of Freedom in Trading Systems"](https://aligrithm.com/degrees-of-freedom-in-trading-systems/) gave the search-width bias formula. The article ["Trade-Count Thresholds for Backtest Reliability"](https://aligrithm.com/trade-count-thresholds-for-backtest-reliability/) gave the standard error of the Sharpe estimate. Neither of them, by itself, tells the team how likely the strategy was to be overfit given the specific search the team performed. The Probability of Backtest Overfitting (PBO), computed via Combinatorially Symmetric Cross-Validation (CSCV), gives a direct numerical answer. PBO is defined as the probability that the IS-optimal parameter set, evaluated on a different slice of data, performs below the median of all candidate parameter sets evaluated on that slice. A PBO of 0.5 means the IS-optimal parameter is no better than a random pick on out-of-sample data; a PBO of 0.05 means the IS-optimal parameter outperforms its peers on out-of-sample data. The number is computed mechanically from the strategy's per-period return series for each candidate parameter set, without requiring assumptions about the return distribution or the specific candidate-set geometry. CSCV is a procedure that computes PBO from a matrix of per-period returns by parameter set. The procedure is mechanical, requires no calibration, and produces a single number that diagnoses the overfit risk specific to the search the team performed. This article gives the procedure step by step, the operational interpretation, and the legitimate uses and failure modes. ## CSCV in one paragraph The data is a T-by-K matrix M, where T is the number of time periods and K is the number of parameter sets that were tried during the optimization. The matrix entry M\[t, k\] is the return of parameter set k in period t. CSCV splits the T time periods into 2S equal-size blocks (S blocks for IS, S blocks for OOS), with all S-of-2S combinations enumerated. For each combination, the IS-optimal parameter k *is found by optimizing the Sharpe (or any other rank statistic) on the IS half. The OOS performance of k* is then ranked among the OOS performances of all K parameter sets. The OOS rank is mapped to a "logit" via log(rank / (K + 1 - rank)). The full distribution of logit values across all combinations is the relative-overfit distribution. The PBO is the fraction of combinations where the IS-optimal parameter has below-median OOS rank, equivalent to the fraction of negative logit values. The mechanical procedure removes the dependence on a single arbitrary IS-OOS split. The combinatorial enumeration produces many independent IS-OOS realizations from the same dataset, capturing the variability that the single-split design misses. ## The procedure step by step Six steps with concrete numbers from the example. Step 1: build the M matrix. The strategy's K = 50 candidate parameter sets are run on the same T = 1000 daily-return periods. The M matrix has 50 columns and 1000 rows. Each cell M\[t, k\] is the daily return of parameter set k on day t. Step 2: choose S. S is the number of blocks per half (so there are 2S blocks total). Common choices are S = 8 (16 blocks total, 12870 combinations of S-out-of-2S) or S = 6 (12 blocks total, 924 combinations). Larger S gives more combinations but smaller blocks; smaller S gives fewer combinations but larger blocks. S = 8 is the standard choice for daily data with several years of history. Step 3: enumerate combinations. For 2S = 16 blocks, the number of S-out-of-2S combinations is C(16, 8) = 12870\. Each combination J selects 8 blocks for IS and the remaining 8 for OOS. Step 4: for each combination J, compute the IS-best parameter. Compute the Sharpe of each of the 50 parameter sets on the 8 IS blocks. Find k\* = argmax\_k Sharpe\_IS(k). Step 5: rank the OOS performance of k*. Compute the Sharpe of each of the 50 parameter sets on the 8 OOS blocks. Rank them. Find the rank r* of k *among the 50 OOS Sharpes (r* between 1 and 50). Step 6: compute the logit. The OOS-rank logit is log(r */ (51 - r*)). A logit of 0 means k *has median OOS rank (rank 25 or 26 of 50). A negative logit means k* has below-median OOS rank (the IS-optimal parameter is worse than half of its peers OOS). A positive logit means above-median. Repeat steps 4-6 for all 12870 combinations. The empirical distribution of logits across combinations is the CSCV result. The PBO is the fraction of combinations with logit < 0. $$ \\text{PBO} = \\frac{1}{|\\mathcal{J}|} \\sum\_{J \\in \\mathcal{J}} \\mathbb{1}\\left\\{ \\text{rank}\_{\\text{OOS}}^{(J)}(k^\*\_{\\text{IS}}^{(J)}) < \\tfrac{K + 1}{2} \\right\\} $$ The set J is the set of all S-of-2S combinations. ## Reading the PBO number Operational interpretation calibrated against published practice and observed deployment outcomes. PBO < 0.10: the IS-optimal parameter robustly outperforms its peers on independent slices. The strategy is not overfit at the parameter-search level; the IS Sharpe is approximately the true Sharpe minus modest sampling noise. Deploy. PBO 0.10-0.30: moderate overfit risk. The IS-optimal parameter is better than median on most slices but not all. The IS Sharpe is moderately inflated. Deploy with smaller capital, tighter kill-switches, and the bias-corrected Sharpe from ["Degrees of Freedom in Trading Systems"](https://aligrithm.com/degrees-of-freedom-in-trading-systems/). PBO 0.30-0.50: high overfit risk. The IS-optimal parameter is approximately equivalent to a random pick on independent slices. The IS Sharpe is mostly noise. Do not deploy without further reduction in parameter count or extension of the data. PBO > 0.50: the IS-optimal parameter is below median on most slices. The IS Sharpe is purely from search-width inflation. The strategy has no detectable edge. Reject. The thresholds are heuristic. The published research on CSCV calibrates against historical strategies with known OOS outcomes; the rough mapping above is consistent with the published guidance. Different strategy families may have different operational thresholds (high-Sharpe-true strategies have lower PBO at any given search width; low-Sharpe-true strategies have higher PBO). ## CSCV vs a single OOS split Three structural advantages. Advantage 1: many IS-OOS realizations from the same data. The single-OOS-split design produces one realization of the OOS Sharpe. The CSCV design produces 12870 realizations (for S = 8). The variance across realizations reveals the strategy's true sensitivity to which slice is held out. A strategy with PBO 0.45 has high realization-to-realization variance: depending on which slice is held out, the IS-optimal parameter looks like the best or worst pick. A strategy with PBO 0.05 has low variance: the IS-optimal parameter is consistently good across slices. Advantage 2: no arbitrary split choice. The single-OOS design depends on the choice of IS-OOS boundary (e.g., 70-30 vs 80-20 vs the 2018 break point). Different boundaries can produce different OOS results, and the team's incentive is to choose the boundary that produces the best result. CSCV averages over all combinations, removing the boundary-selection degree of freedom. Advantage 3: measures overfit at the parameter-set level, beyond OOS performance on one slice. A single OOS split tells you the OOS Sharpe of one parameter set on one slice. CSCV tells you the relative rank of the IS-optimal parameter compared to its peers on independent slices. The relative-rank statistic is more robust to overall-strategy-quality variation across slices and isolates the overfit-specific risk. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Data requirements for CSCV Three requirements. Failures of any one make CSCV results unreliable. Requirement 1: per-parameter return time series for all K candidates. The CSCV procedure needs the full M matrix across all candidate parameter sets, beyond the IS-optimal parameter's track record alone. Many backtesting workflows save only the optimized result; CSCV requires re-running the strategy at all candidate parameter sets to populate M. Requirement 2: enough data for meaningful blocks. With 2S = 16 blocks, T should be at least 16 times the autocorrelation horizon of the returns. For daily data with weekly autocorrelation, T should be at least 80 days; in practice, T should be at least 500 to make blocks of 30+ days. Requirement 3: i.i.d.-like blocks. CSCV implicitly assumes the blocks are exchangeable. For strongly non-stationary data (the article ["Slow Wandering: The Most Dangerous Type of Market Change"](https://aligrithm.com/slow-wandering-the-most-dangerous-type-of-market-change/) framed the problem), the blocks from different eras may have different mean and variance, and the rank-based statistics are still valid but the interpretation requires regime-aware reading. Combining CSCV with the regime stratification from ["Regime Coverage: Why Your Backtest Needs Different Market States"](https://aligrithm.com/regime-coverage-why-your-backtest-needs-different-market-states/) gives a more robust diagnostic. ## Cases where PBO disagrees with the OOS Sharpe Three diagnostic patterns. Pattern 1: IS Sharpe high, OOS Sharpe low, PBO low (< 0.15). The strategy is sound but the chosen OOS slice is unrepresentative (e.g., a single regime that the strategy struggles in). The article ["Why OOS Failure Is Often a Stationarity Failure"](https://aligrithm.com/why-oos-failure-is-often-a-stationarity-failure/) framed the regime-mismatch diagnosis. The CSCV result is the more reliable estimate of the strategy's true performance because it averages across many slices. Pattern 2: IS Sharpe high, OOS Sharpe high, PBO high (> 0.40). The OOS slice happens to favor the IS-optimal parameter, but most other slices would not. The strategy has substantial overfit risk despite the favorable OOS realization. Do not deploy at full capital. Pattern 3: IS Sharpe high, OOS Sharpe low, PBO high (> 0.40). The strategy is overfit on both the headline numbers and the CSCV diagnostic. Reject. The PBO acts as a tiebreaker between the IS and the OOS. A low PBO supports the IS estimate; a high PBO supports skepticism even when the OOS happens to look good. ## Anti-patterns Five mistakes specific to CSCV and PBO. Anti-pattern 1: computing PBO on a single parameter set. PBO requires the full K-by-T matrix; computing it on K = 1 is meaningless. The strategy must have multiple candidate parameter sets that were considered during research; if only one was considered, the PBO concept does not apply (and the search-width bias from ["Degrees of Freedom in Trading Systems"](https://aligrithm.com/degrees-of-freedom-in-trading-systems/) is the right diagnostic). Anti-pattern 2: cherry-picking the parameter set after seeing PBO. A team computes PBO with K = 50 candidates, finds PBO = 0.42 (high), drops some candidates to lower the PBO, and re-reports. The cherry-picking is itself a form of overfitting and inflates the apparent significance. The K should be fixed before computing PBO. Anti-pattern 3: applying CSCV to optimization metrics other than Sharpe without checking sensitivity. The PBO depends on the choice of optimization metric. CSCV with Sharpe and CSCV with profit factor on the same strategy can give different PBO values. Report PBO under multiple metrics if the choice is not fixed by the strategy's deployment objective. Anti-pattern 4: treating PBO as a complete certification. PBO measures overfit at the parameter-search level. It does not measure structural-premise validity (the article ["Why Benchmarks Matter in Rule Evaluation"](https://aligrithm.com/why-benchmarks-matter-in-rule-evaluation/) framed this requirement), regime coverage (the article ["Regime Coverage: Why Your Backtest Needs Different Market States"](https://aligrithm.com/regime-coverage-why-your-backtest-needs-different-market-states/)), or transaction-cost realism. A strategy with PBO < 0.10 that ignores costs or has no structural premise is still bad. Anti-pattern 5: reporting only the PBO without the underlying logit distribution. The full distribution of logits across combinations is informative beyond the binary PBO. A distribution heavily concentrated near zero suggests the IS-optimal parameter is not strongly preferred. A distribution with a clear positive mode (positive logit) suggests genuine ranking benefit. Report the histogram alongside the PBO. ## Decision matrix | PBO | Interpretation | Action | | --------------------------------------- | --------------------------------------------------------- | -------------------------------------------------- | | < 0.05 | IS-optimal parameter strongly outperforms peers OOS | Deploy at full capital | | 0.05-0.15 | IS-optimal parameter robustly preferred | Deploy with standard monitoring | | 0.15-0.30 | Moderate overfit risk | Deploy with smaller capital, tighter kill-switches | | 0.30-0.50 | High overfit risk, IS-optimal is approximately random OOS | Do not deploy without parameter-count reduction | | 0.50+ | IS-optimal is worse than median OOS | Reject; pure search-width artifact | | Disagreement: low PBO, low OOS Sharpe | Trust CSCV; OOS slice unrepresentative | Investigate slice, redeploy | | Disagreement: high PBO, high OOS Sharpe | Trust CSCV; OOS slice happens to favor | Reduce capital, more validation | The matrix maps PBO range to action. The pattern: PBO is the headline overfit diagnostic; combine with the IS-OOS comparison and the structural-premise check. ## Visualizing CSCV ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/05/article_76-visual-0.png) ## KEY POINTS - The Probability of Backtest Overfitting (PBO) is the probability that the IS-optimal parameter set, evaluated on an independent slice, performs below the median of all candidate parameter sets on that slice. PBO of 0.5 means the IS-optimal parameter is no better than a random pick OOS; PBO of 0.05 means it outperforms peers OOS. - Combinatorially Symmetric Cross-Validation (CSCV) computes PBO mechanically from a T-by-K matrix M of per-period returns by parameter set. The procedure splits T into 2S blocks, enumerates all S-of-2S combinations, finds the IS-best parameter on each combination, ranks its OOS performance, and computes the fraction of below-median ranks. - The OOS-rank statistic is converted to a logit via log(r / (K+1-r)). The full distribution of logits across combinations is the CSCV result. PBO is the fraction of negative logits. - Operational thresholds: PBO < 0.10 = strong, deploy. 0.10-0.30 = moderate risk, deploy with smaller capital and kill-switches. 0.30-0.50 = high risk, do not deploy without parameter-count reduction. > 0.50 = pure search-width artifact, reject. - Three structural advantages over a single OOS split: many IS-OOS realizations (12870 for S=8), no arbitrary boundary choice, directly measures relative overfit rather than absolute OOS performance. - Three data requirements: per-parameter return series for all K candidates, enough data for meaningful blocks (T at least 16 times the autocorrelation horizon, in practice 500+ for daily), exchangeable-ish blocks (regime stratification helps for strongly non-stationary data). - Three diagnostic patterns when PBO disagrees with raw OOS: low PBO + low OOS = trust CSCV, OOS slice unrepresentative, redeploy; high PBO + high OOS = OOS slice happens to favor, reduce capital; high PBO + low OOS = overfit confirmed, reject. - Anti-pattern: computing PBO on a single parameter set. CSCV requires the full K-by-T matrix. - Anti-pattern: cherry-picking K after seeing PBO. K must be fixed before computing. - Anti-pattern: applying CSCV to one optimization metric without checking sensitivity. Report PBO under multiple metrics if the choice is not fixed by deployment objective. - Anti-pattern: treating PBO as complete certification. PBO measures overfit at the parameter-search level only. Combine with structural-premise check, regime coverage, transaction-cost realism. - Anti-pattern: reporting PBO without the logit-distribution histogram. The shape of the distribution is informative beyond the binary PBO. - The current article gives the most direct overfit diagnostic available. The next article in the publication (["Why Walk-Forward Testing Is Better Than One Big OOS Split"](https://aligrithm.com/why-walk-forward-testing-is-better-than-one-big-oos-split/)) covers a different validation technique that simulates real-time deployment cadence and complements CSCV's combinatorial approach. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Testing and Tuning Market Trading Systems - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Testing+and+Tuning+Market+Trading+Systems+Timothy+Masters&ref=aligrithm.com) - [Data Mining Algorithms in C++ - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Data+Mining+Algorithms+in+C%2B%2B+Timothy+Masters&ref=aligrithm.com) ### 3.22 Permutation Tests for Indicator Significance URL: https://aligrithm.com/permutation-tests-for-indicator-significance/ Last updated: 2026-07-27T13:24:13.000Z A researcher tests whether a 14-day RSI indicator predicts the next-day return on SPX over 2010-2024\. The protocol: compute RSI(14) on each trading day, classify each day as "RSI < 30" (oversold), "RSI in \[30, 70\]" (neutral), or "RSI > 70" (overbought). Compute the average next-day return in each bucket. The result: oversold days have average next-day return +0.18%, neutral +0.04%, overbought -0.07%. The differential between oversold and overbought is 0.25%. The researcher's question: is the 0.25% differential a real signal, or is it within the range of differentials that would arise by chance even if the indicator had no predictive power? The standard approach is a t-test, comparing the means of the two buckets. The t-test reports a t-statistic and a p-value. For this dataset (\~3500 trading days, \~250 oversold, \~250 overbought, sigma per-day approximately 1.1%), the t-statistic is approximately 2.5 and the p-value is approximately 0.012\. Conventional interpretation: significant at the 5% level, marginally significant at the 1% level. The t-test makes assumptions: i.i.d. observations, normal-or-near-normal return distribution, no serial correlation. SPX returns violate the second and third assumptions. Returns have fat tails (kurtosis approximately 5-7 vs the normal value of 3) and exhibit volatility clustering (next-day return autocorrelation is small but the squared-return autocorrelation is substantial). The t-test's p-value is biased; the true p-value (the actual rate at which the observed differential would arise by chance under the null) is different from the t-test estimate. The article ["Why Volatility Is More Non-Stationary Than Trend"](https://aligrithm.com/why-volatility-is-more-non-stationary-than-trend/) framed the autocorrelation structure of squared returns; the same structure invalidates the t-test's i.i.d. assumption. The permutation test does not make these assumptions. The procedure: under the null hypothesis ("the indicator has no predictive power"), the indicator-value-to-return assignment is arbitrary. Permute the indicator values across days while keeping the return sequence fixed. Compute the same statistic (the differential between oversold and overbought next-day returns) on the permuted dataset. Repeat 10000 times. The empirical distribution of the differential across permutations is the distribution under the null. Compare the original (unpermuted) statistic to this null distribution; the p-value is the fraction of permutation statistics that are as extreme as the observed one. For the SPX-RSI example, the permutation p-value might be 0.04 (less significant than the t-test's 0.012, by a factor of 3 or so), reflecting the fat-tail-and-clustering corrections. The conclusion shifts: the indicator's predictive power is at the boundary of the conventional significance threshold rather than well below it. The differential of 0.25% is real but smaller than the t-test claims. The article ["Monte Carlo for Trading Systems"](https://aligrithm.com/monte-carlo-for-trading-systems/) covered bootstrap and synthetic-path techniques for understanding strategy risk distributions. Permutation tests are a different resampling technique, designed for testing whether an indicator or signal has predictive power against the null hypothesis of no relationship. This article gives the framework, the implementation details, the legitimate uses, and the failure modes. ## The permutation test, formally The procedure formally. Let X = (x\_1, x\_2, ..., x\_T) be the indicator values and Y = (y\_1, y\_2, ..., y\_T) be the return values, both indexed by time. Let s = f(X, Y) be the test statistic of interest (mean of returns conditional on X above threshold, correlation of X and Y, Sharpe of a strategy long when X is above threshold and flat otherwise, etc.). The null hypothesis. Under H\_0, the indicator has no predictive power: knowing x\_t provides no information about y\_t (or y\_{t+1}, depending on the strategy's lag). Operationally, under H\_0, any permutation of X across days is equivalent to the original X. The permutation procedure. For b = 1 to B: 1\. Generate a random permutation pi\_b of {1, ..., T}. 2\. Construct the permuted indicator sequence X^{(b)} = (x\_{pi\_b(1)}, x\_{pi\_b(2)}, ..., x\_{pi\_b(T)}). 3\. Compute s^{(b)} = f(X^{(b)}, Y). The null distribution. The empirical distribution of {s^{(b)} : b = 1, ..., B} is the distribution of s under H\_0. The p-value. The permutation p-value is the fraction of s^{(b)} values that are as extreme as the observed s. $$ \\hat{p} = \\frac{1}{B + 1} \\left( 1 + \\sum\_{b=1}^{B} \\mathbb{1}\\{s^{(b)} \\geq s\_{\\text{obs}}\\} \\right) $$ The +1 in the numerator and denominator is the conservative correction that handles the case where no permutation exceeds the observed value. With B = 10000 and zero exceedances, the conservative p-value is 1/10001 = 0.0001. Permutation test, assumptions it skips. Independence of the y\_t sequence (the return autocorrelation is preserved because Y is fixed). Normality of the y\_t distribution (the test is non-parametric). Constant variance (the y\_t volatility clustering is preserved). The test is robust to all of these. Permutation test, assumptions it makes. Under H\_0, the joint distribution of (x\_t, y\_t) is exchangeable across t. This holds when there is no time-varying relationship between the indicator and the return. It can fail when the indicator and the return have a shared time-varying component (e.g., both are conditional on the same regime variable that is not being controlled for). ## Differences from the bootstrap Three key distinctions. Distinction 1: what is resampled. The bootstrap (covered in ["Monte Carlo for Trading Systems"](https://aligrithm.com/monte-carlo-for-trading-systems/)) resamples the data with replacement to estimate sampling distributions of statistics. The permutation test reorders one of the variables to test a null hypothesis of no relationship. Distinction 2: what is being estimated. The bootstrap estimates the variability of a statistic under repeated sampling from the same population. The permutation test estimates the distribution of a statistic under a null hypothesis. The two answer different questions. Distinction 3: what assumptions are required. The bootstrap requires that the data are representative of the population. The permutation test requires that under H\_0, the labels (the X values in the example) are exchangeable. The exchangeability assumption is sometimes weaker than the i.i.d.-from-a-population assumption, but it can also fail in ways the bootstrap does not. The choice. Use the bootstrap to estimate confidence intervals on a parameter (Sharpe, max drawdown, win rate). Use the permutation test to test the null hypothesis that an indicator has no predictive power. The two are complementary; neither replaces the other. ## Two implementations of the permutation test Implementation 1: simple permutation. Permute the indicator series X across days while keeping the return series Y fixed. The implementation in the SPX-RSI example. Strength: simple to implement, computationally cheap. Weakness: destroys all temporal structure in X. If X has its own autocorrelation (RSI is a smoothed moving average with strong day-to-day autocorrelation), the simple permutation produces null samples with very different X structure than the original. The null distribution corresponds to "no predictive power on a structureless permutation of X", which differs from "no predictive power on the original-structure X". The two and the test may be biased toward rejecting H\_0. Implementation 2: block permutation. Divide the time series into contiguous blocks of length L (e.g., 5 or 10 days). Permute the blocks rather than individual observations. The implementation preserves within-block autocorrelation while breaking between-block predictability. Strength: preserves the X autocorrelation structure within each block. The null distribution is closer to "no between-block predictive power". Weakness: requires choosing the block length L. A block too short loses the autocorrelation structure (similar to simple permutation). A block too long produces too few independent permutations (only T/L blocks to permute). The discipline. For indicators with substantial autocorrelation (most technical indicators), use block permutation. The block length should be approximately the autocorrelation half-life of the indicator, typically 5-20 days for most technical indicators on daily data. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Applications to indicator and strategy testing Five legitimate uses. Use 1: testing whether an indicator predicts returns. The opening example. Statistic: mean differential in next-day returns between oversold and overbought buckets. The p-value is the test result for whether the indicator carries predictive signal. Use 2: testing whether a strategy's Sharpe is non-zero. Statistic: the Sharpe of the strategy on the original data. Permute the indicator (the trigger sequence) across days. Compute the Sharpe of the permuted strategy. Repeat. The fraction of permuted Sharpes that are as high as the original is the p-value for "the strategy has no edge". Use 3: testing whether a strategy outperforms a benchmark. Statistic: the differential in Sharpe between the strategy and the benchmark. Permute the indicator. Compare the original differential to the null distribution. The fraction of permuted differentials as extreme as the original is the p-value for "the strategy is no better than the benchmark". Use 4: testing whether two indicators have predictive power beyond what each contributes individually. Statistic: the joint Sharpe of a strategy using both indicators minus the Sharpe of strategies using each indicator individually. Permute one indicator at a time, then both jointly. The pattern of p-values reveals whether the combined indicator has predictive power not in either component. Use 5: testing whether a strategy outperforms a "lookahead-shuffled" version of itself. Permute the indicator's relationship to its source data (e.g., shift the indicator by a random number of days). The permuted strategy is identical to the original except for the temporal alignment. If the original Sharpe is much higher than the permutation distribution, the indicator's temporal alignment carries information; if not, the apparent edge is alignment-independent (typically a sign of look-ahead bias or a benchmark-replication artifact). ## Failure modes Four ways the permutation test can mislead. Failure 1: shared regime structure. Both the indicator and the return are conditional on a regime variable (e.g., volatility regime) that is not being controlled for. The permutation breaks the indicator-return relationship but does not break the regime-driven covariation. The null distribution is mis-specified; the test may falsely reject or falsely fail to reject. Failure 2: ignoring search-width bias. Testing 100 indicators on the same return series and reporting the most significant one without multiple-comparison correction is a multiple-testing problem. The permutation test on the best-of-100 indicator is biased toward small p-values. Apply Bonferroni or Benjamini-Hochberg corrections. Failure 3: short time series. With T < 100 observations and an indicator with substantial autocorrelation, the block-permutation test has very few independent blocks (T / L < 10), and the null distribution has low resolution. The p-value is unreliable. Failure 4: misspecified statistic. The permutation test is conditional on the choice of statistic. Choosing the statistic post-hoc (after seeing the data, picking the comparison that produces the largest effect) inflates the apparent significance. Pre-specify the statistic before the test, and report all chosen statistics across the panel, including the non-significant ones. ## Anti-patterns Five mistakes specific to permutation tests. Anti-pattern 1: applying simple permutation to indicators with strong autocorrelation. The null distribution is mis-specified; the test rejects too easily. Use block permutation with a block length matched to the indicator's autocorrelation half-life. Anti-pattern 2: confusing the permutation test with the bootstrap. The bootstrap estimates variability under repeated sampling. The permutation test estimates the null distribution. They answer different questions. Anti-pattern 3: cherry-picking the test statistic. Running multiple statistics (mean, median, Sharpe, win rate, drawdown) and reporting the one with the smallest p-value inflates the significance. Pre-specify the statistic. Anti-pattern 4: ignoring multiple-testing corrections when testing many indicators. The probability of finding a "significant" indicator at p < 0.05 from 100 tests under the null is approximately 99%. Apply corrections. Anti-pattern 5: presenting the permutation p-value as proof the strategy works. The p-value tests whether the indicator has any predictive power; it does not test whether the predictive power is large enough to overcome transaction costs, whether the indicator's structural premise is sound, or whether the strategy survives OOS. The p-value is one piece of evidence; the survival diagnostics are the others. ## Decision matrix | Question | Permutation test | Other tools | | ----------------------------------------------- | ------------------------------------------------- | ----------------------------------------- | | Does an indicator predict returns? | Block permutation, signed test statistic | t-test (assumes i.i.d.), correlation test | | Is a strategy's Sharpe non-zero? | Permutation of indicator series, Sharpe statistic | Bootstrap-based confidence interval | | Does a strategy beat a benchmark? | Permutation of differential, paired statistic | Information-ratio significance test | | Are two indicators jointly predictive? | Permutation of each separately and jointly | Multivariate regression with significance | | Is a strategy's edge alignment-dependent? | Random-shift permutation of indicator | Comparison to lookahead-shuffled version | | What is the variability of the Sharpe estimate? | Bootstrap (NOT permutation) | Standard-error formula | | What is the maximum drawdown distribution? | Bootstrap (NOT permutation) | Synthetic-path Monte Carlo | | What is the OOS performance estimate? | Hold-out validation (NOT permutation) | Walk-forward analysis | The matrix maps question to right tool. The pattern: permutation tests are for null-hypothesis testing; bootstraps are for variability estimation; OOS hold-outs are for generalization. None replaces the others. ## Visualizing the permutation test ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/05/article_75-visual-0.png) ## KEY POINTS - The permutation test estimates the distribution of a test statistic under the null hypothesis that an indicator has no predictive power. Procedure: permute the indicator series across time while keeping the return series fixed, compute the statistic on each permutation, the fraction of permutations as extreme as the observed value is the p-value. - The permutation test does not assume i.i.d. observations, normal distribution, or constant variance. The return series Y is fixed across permutations, so its autocorrelation, fat tails, and volatility clustering are automatically respected. The test is non-parametric. - The permutation test assumes exchangeability of the (x\_t, y\_t) pairs under H\_0\. The assumption can fail when both the indicator and the return are conditional on an unobserved regime variable; in that case the null distribution is mis-specified. - Bootstrap and permutation are different: bootstrap resamples to estimate sampling-distribution variability of a statistic, permutation reorders to estimate the null distribution under "no relationship". The two answer different questions and complement each other. - Two implementations: simple permutation (reorder individual observations, fast, but destroys indicator autocorrelation), block permutation (reorder contiguous blocks of length L, preserves within-block autocorrelation, requires choice of L matching the indicator's autocorrelation half-life). - Five legitimate uses: testing whether an indicator predicts returns, testing whether a strategy's Sharpe is non-zero, testing whether a strategy beats a benchmark, testing whether two indicators are jointly predictive beyond their individual contributions, testing whether a strategy's edge is alignment-dependent (random-shift permutation). - Four failure modes: shared regime structure (indicator and return both conditional on an unobserved regime), ignoring search-width bias (testing many indicators inflates the significance of the best one), short time series (T < 100 with autocorrelated indicator has too few independent blocks), misspecified statistic (cherry-picking the statistic after seeing the data). - Anti-pattern: applying simple permutation to indicators with strong autocorrelation. Use block permutation. - Anti-pattern: confusing the permutation test with the bootstrap. Different tools, different questions. - Anti-pattern: cherry-picking the test statistic post-hoc. Pre-specify and report all chosen statistics. - Anti-pattern: ignoring multiple-testing corrections when testing many indicators. Apply Bonferroni or Benjamini-Hochberg. - Anti-pattern: treating the p-value as proof the strategy works. The p-value tests for any predictive power; it does not certify that the power is large enough to survive costs, that the structural premise is sound, or that the strategy generalizes OOS. - The current article completes the resampling-techniques arc that began with the bootstrap-and-synthetic-path coverage in ["Monte Carlo for Trading Systems"](https://aligrithm.com/monte-carlo-for-trading-systems/). The combination of permutation tests for indicator significance, bootstrap for risk-distribution estimation, and OOS hold-out for generalization gives the operational toolkit for converting an apparent IS edge into a deployment-ready estimate. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Testing and Tuning Market Trading Systems - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Testing+and+Tuning+Market+Trading+Systems+Timothy+Masters&ref=aligrithm.com) - [Data Mining Algorithms in C++ - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Data+Mining+Algorithms+in+C%2B%2B+Timothy+Masters&ref=aligrithm.com) ### 3.21 Monte Carlo for Trading Systems URL: https://aligrithm.com/monte-carlo-for-trading-systems/ Last updated: 2026-07-27T13:24:12.000Z A trading system has 480 trades over 6 years on E-mini SPX futures. The aggregate Sharpe is 1.05, the maximum drawdown is 14.2%, the longest losing streak is 7 trades, the longest losing streak in equity curve is 47 days, and the worst single month was -3.8%. The team prepares a deployment proposal that includes these numbers as the operational risk limits: "expected drawdown ceiling 14.2%, expected worst month -3.8%, expected longest losing streak 7 trades". The risk committee approves with these as the kill-switch thresholds. The strategy ships. Three months later, the live drawdown reaches 18.5%, exceeding the 14.2% IS-observed limit. The team is forced to either kill the strategy or violate the risk policy. The decision is made in panic. The strategy is killed. Six months after the kill, the same strategy template (re-deployed by another team that did not know the history) reaches Sharpe 1.0 over the next two years, with a max drawdown of 17%. The first team's kill was driven by an IS-observed maximum that was a single realization of a distribution; the actual distribution included drawdowns up to approximately 22% within a one-sigma confidence band. The IS maximum drawdown was an underestimate, not a ceiling. The article ["Trade-Count Thresholds for Backtest Reliability"](https://aligrithm.com/trade-count-thresholds-for-backtest-reliability/) framed the standard error of the Sharpe; the same logic applies to every other risk metric. The IS-observed maximum drawdown is a noisy realization of the true drawdown distribution. The IS-observed worst month is a noisy realization of the true monthly-return distribution. The IS-observed longest losing streak is a noisy realization of the true losing-streak distribution. Each of these is a single number from a wide distribution. The right operational practice is to estimate the distributions, not the point realizations, and set policy based on the distributional properties. Monte Carlo methods are the standard tool. This article gives the framework for two flavors (bootstrap of historical trades, synthetic-path generation) and the operational uses each enables. The article ["Permutation Tests for Indicator Significance"](https://aligrithm.com/permutation-tests-for-indicator-significance/) later in this pillar covers a different resampling technique focused on testing whether an indicator has predictive power; the present article covers the resampling techniques focused on understanding the strategy's risk distribution. ## Bootstrap Monte Carlo: resampling trades The simplest Monte Carlo for trading-system risk. The procedure: sample N trades with replacement from the historical N-trade record, in random order, to construct a synthetic equity curve. Repeat 10000 times. Compute the metric of interest (max drawdown, longest losing streak, worst month, terminal P&L) on each synthetic curve. The empirical distribution of the metric across the 10000 synthetic curves is the bootstrap estimate of the metric's sampling distribution. Bootstrap captures. The bootstrap captures the variation that comes from the order of trades and from the small-sample noise in the per-trade outcomes. A strategy with 480 historical trades has many possible orderings; the IS-observed equity curve is one of them. The bootstrap shows what other orderings would have produced. Bootstrap blind spots. The bootstrap does not generate trades that did not occur in the IS sample. If the IS sample has no trades during a 2008-style crisis, the bootstrap will not produce 2008-style trades. The bootstrap is conditional on the IS distribution and inherits its limitations. The article ["Regime Coverage: Why Your Backtest Needs Different Market States"](https://aligrithm.com/regime-coverage-why-your-backtest-needs-different-market-states/) framed the IS-coverage requirement; the bootstrap does not fix coverage gaps. The math. Let X\_1, X\_2, ..., X\_N be the per-trade P&L sequence in the IS sample. A bootstrap replication is constructed by drawing N indices uniformly with replacement from {1, ..., N} and forming the sequence X\_{i\_1}, X\_{i\_2}, ..., X\_{i\_N}. The metric of interest f (e.g., max drawdown) is computed on the bootstrapped equity curve. $$ \\widehat{F}\_f(x) = \\frac{1}{B} \\sum\_{b=1}^{B} \\mathbb{1}\\{f(\\mathbf{X}^{(b)}) \\leq x\\} $$ The empirical distribution F\_f(x) is the bootstrap estimate of the metric's CDF. From this distribution, percentiles can be computed: the 95th percentile of the max-drawdown distribution is the operationally relevant ceiling, not the IS-observed max-drawdown. ## Synthetic-path Monte Carlo: generating returns A more powerful and more dangerous technique. Instead of resampling historical trades, generate synthetic price paths from a fitted model of the underlying market dynamics. Run the strategy on the synthetic paths. Compute the metric distributions. Synthetic-path captures. Path-generation models can simulate market events that do not appear in the IS sample (vol regimes outside the IS range, extreme events, regime transitions). With a well-fitted model, the synthetic paths can extend the effective sample size beyond what the historical data alone provides. Synthetic-path blind spots. The path-generation model is itself a hypothesis about the market's dynamics. If the model is wrong, the synthetic paths are wrong, and the resulting metric distributions are wrong. The article ["Slow Wandering: The Most Dangerous Type of Market Change"](https://aligrithm.com/slow-wandering-the-most-dangerous-type-of-market-change/) framed the non-stationarity problem; many path-generation models implicitly assume stationarity, which is the opposite of what real markets exhibit. Common path-generation models, with their typical biases: Model 1: GBM (geometric Brownian motion). Simple, well-understood, generates lognormal returns with constant drift and vol. Biases: ignores fat tails, ignores vol clustering, ignores regime breaks. Useful for first-pass scenario analysis; not useful for tail-risk estimation. Model 2: GARCH-family models. Captures vol clustering. Biases: still assumes the basic return-generating process is stationary, ignores regime breaks, ignores macro-event jumps. Better than GBM for many purposes. Model 3: jump-diffusion (Merton, Kou). Adds discrete jumps to GBM. Biases: jump sizes and frequencies are calibrated from the IS sample, which underestimates true jump risk if the IS does not include enough jumps. Model 4: regime-switching models (Hamilton, Markov-switching). Allows regime changes. Biases: number and properties of regimes are calibrated from IS; out-of-sample regimes are not modeled. Model 5: nonparametric resampling of returns. Sample full days or weeks of returns from history. Captures empirical structure. Biases: same as bootstrap; cannot generate events outside the IS distribution. The discipline. Use synthetic-path Monte Carlo only when the model assumptions are explicit and the model is calibrated against multiple regimes. Report the model along with the metric distributions; do not present synthetic-path results as if they were data. ## Operational uses Five legitimate applications. Use 1: confidence intervals on risk metrics. The bootstrap distribution of the max drawdown gives the 95th-percentile drawdown, the 99th-percentile drawdown, and the median. The risk policy uses the 95th or 99th percentile (the realistic ceiling) rather than the IS-observed maximum (a single realization). For the SPX strategy in the opening, the bootstrap 95th-percentile max drawdown might be 18.5%, the 99th-percentile 22%, and the IS-observed 14.2% is the median. The risk policy should be set against the 95th or 99th percentile. Use 2: sequence-of-returns analysis. The bootstrap reveals how sensitive the equity curve is to the order of trades. A strategy with high terminal P&L but tight bootstrap variation in max drawdown is robust. A strategy with the same terminal P&L but wide bootstrap variation in max drawdown is fragile to bad luck in trade ordering. The fragility is a real operational concern that the IS curve hides. Use 3: position-sizing calibration. Combined with a Kelly or fractional-Kelly sizing scheme, the bootstrap distribution of equity-curve outcomes informs the right leverage. If the 5th percentile of bootstrap-equity-curve terminal P&L is severely negative under a given leverage, the leverage is too high. Use 4: kill-switch threshold setting. Use the bootstrap distribution of (max drawdown, time to recover, longest losing streak) to set kill-switch thresholds at the 95th or 99th percentile of the IS-distribution. A breach of the threshold is then strong evidence the strategy is operating outside the IS regime, beyond unlucky variance within it. Use 5: scenario stress testing. Use synthetic-path Monte Carlo with explicit stress regimes (high-vol, high-correlation, regime-break) to estimate strategy behavior in events not seen in the IS sample. The stress tests are conditional on the model assumptions; the results are not predictions but bounds. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Monte Carlo's limits Three honest limits. Limit 1: out-of-sample validation. Monte Carlo is conditional on the IS distribution. It does not provide an independent test of whether the strategy generalizes to OOS regimes. The article ["Why OOS Failure Is Often a Stationarity Failure"](https://aligrithm.com/why-oos-failure-is-often-a-stationarity-failure/) framed the OOS validation; Monte Carlo informs the IS-distribution diagnostics, not the OOS test. Limit 2: search-width-bias correction. Monte Carlo on the IS-optimal parameters inherits the IS-optimization bias. The Sharpe distribution from a bootstrap of an over-optimized strategy is centered on the over-optimized IS Sharpe, not on the true Sharpe. The article ["Degrees of Freedom in Trading Systems"](https://aligrithm.com/degrees-of-freedom-in-trading-systems/) framed the bias; bootstrap does not remove it. Limit 3: regime-coverage gaps. If the IS sample has no 2008-style trades, no Monte Carlo can produce them. The technique extends what is in the data, not what is missing. ## Anti-patterns Five mistakes specific to Monte Carlo applications. Anti-pattern 1: presenting Monte Carlo results as if they were OOS validation. The bootstrap of IS trades is not OOS. It is a refinement of the IS analysis. Reporting it alongside genuine OOS results without distinguishing the two misleads the reader. Anti-pattern 2: using GBM for tail-risk estimation. GBM has thin tails; real markets have fat tails. The 99th-percentile drawdown from a GBM-based Monte Carlo is severely underestimated. Use jump-diffusion or empirical resampling for tail-risk work. Anti-pattern 3: bootstrapping with replacement when the trades are serially correlated. The bootstrap assumes i.i.d. observations. Trades that exhibit autocorrelation (consecutive losses cluster, regime-conditional bursts) violate this. Use block-bootstrap (resample contiguous blocks of trades) instead of trade-by-trade bootstrap. Anti-pattern 4: choosing model assumptions to match the desired result. A team that wants to claim the strategy is robust may select a path-generation model that produces flattering metric distributions. The right discipline is to specify the model and its assumptions before running the simulation, not to choose the model that produces the desired output. Anti-pattern 5: treating 10000 Monte Carlo replications as "10000x more data". The replications are reordered or re-simulated versions of the same underlying information, with no addition of independent observations. The effective information content is approximately the IS sample, not 10000 times the IS sample. Bootstrap CIs are estimates of the IS-distribution properties, not estimates with 10000-trade-equivalent precision. ## Decision matrix | Question | Right MC technique | Wrong MC technique | | ------------------------------------- | ------------------------------------------------------ | ------------------------------ | | Confidence interval on max drawdown | Bootstrap of historical trades | GBM-based synthetic paths | | Tail-risk estimation (99th pct) | Jump-diffusion or empirical resampling | GBM (thin tails) | | Sensitivity to trade ordering | Bootstrap (i.i.d. or block) | Single equity curve | | Stress testing rare events | Regime-switching synthetic paths with explicit regimes | Bootstrap (cannot extrapolate) | | Position-sizing calibration | Bootstrap with leverage scaling | IS-observed metrics only | | Kill-switch threshold setting | Bootstrap 95th/99th percentile | IS-observed maximum | | Out-of-sample validation | OOS hold-out, not MC | Bootstrap presented as OOS | | Search-width-bias correction | Bias formula from DoF count, not MC | MC of optimized parameters | | Strategy comparison (which is better) | Bootstrap-paired comparison | Single-realization comparison | The matrix maps question to right technique. The pattern: bootstrap for IS-distribution diagnostics, synthetic paths only with explicit model and stress-regime context, never as a substitute for OOS validation. ## Visualizing the Monte Carlo result ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/05/article_74-visual-0.png) ## KEY POINTS - Monte Carlo for trading systems comes in two flavors: bootstrap of historical trades (resample with replacement to construct synthetic equity curves) and synthetic-path generation (simulate price paths from a fitted model and run the strategy on them). - The bootstrap captures variation from trade ordering and small-sample noise in per-trade outcomes. It does not capture trades that did not occur in the IS sample. It is conditional on the IS distribution and inherits its coverage limitations. - The synthetic-path approach can extend beyond the IS sample if the path-generation model is well-calibrated. Common models: GBM (thin tails, no clustering), GARCH (captures clustering, ignores regime breaks), jump-diffusion (adds jumps), regime-switching (allows regime changes), nonparametric resampling. Each has documented biases. - Five operational uses: confidence intervals on risk metrics (95th/99th percentile of max drawdown, longest losing streak, worst month), sequence-of-returns analysis (sensitivity of equity curve to trade order), position-sizing calibration (Kelly leverage informed by bootstrap distribution), kill-switch threshold setting (breach indicates regime change, not bad luck), scenario stress testing (synthetic paths with explicit stress regimes). - The IS-observed maximum drawdown is a single realization of the true drawdown distribution. The right operational ceiling is the bootstrap 95th or 99th percentile, not the IS-observed maximum. Setting policy on the IS maximum produces panic kills when the strategy operates within its true distribution. - For a strategy with 480 trades and IS max drawdown 14.2%, the bootstrap distribution might show median 13.5%, 95th percentile 18.5%, 99th percentile 22%. The IS-observed value is the median, not the ceiling. - Monte Carlo cannot replace three things: out-of-sample validation (MC is conditional on IS distribution), search-width-bias correction (MC of an over-optimized strategy inherits the bias), regime-coverage gaps (no MC can produce events outside the IS distribution). - Anti-pattern: presenting MC results as OOS validation. MC is a refinement of IS analysis, not a substitute for hold-out testing. - Anti-pattern: using GBM for tail-risk. GBM has thin tails; real markets have fat tails. Use jump-diffusion or empirical resampling. - Anti-pattern: bootstrapping i.i.d. when trades are serially correlated. Use block-bootstrap (contiguous blocks). - Anti-pattern: choosing model assumptions to match the desired result. Specify the model before running the simulation. - Anti-pattern: treating 10000 replications as 10000x more data. The replications are reorderings of the same underlying information; effective information content equals the IS sample. - The current article gives the resampling techniques for understanding strategy risk. The next article in the publication (["Permutation Tests for Indicator Significance"](https://aligrithm.com/permutation-tests-for-indicator-significance/)) covers a different resampling technique focused on testing whether an indicator has predictive power against the null hypothesis of no signal. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Testing and Tuning Market Trading Systems - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Testing+and+Tuning+Market+Trading+Systems+Timothy+Masters&ref=aligrithm.com) - [Data Mining Algorithms in C++ - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Data+Mining+Algorithms+in+C%2B%2B+Timothy+Masters&ref=aligrithm.com) ### 3.20 Why 30 Trades Is Not a Strategy URL: https://aligrithm.com/why-30-trades-is-not-a-strategy/ Last updated: 2026-07-27T13:24:12.000Z A trading-system seller posts a track record on a forum: "31 trades, 71% win rate, average win 2.3R, average loss 1.0R, expectancy 1.34R per trade, this strategy works". The post receives admiration. A reader counts the consecutive wins (the longest streak is six) and finds the result impressive. Some viewers accept the data as evidence the strategy is profitable and consider deploying capital. The objection that the seller anticipates and dismisses: "but you only have 30 trades". The seller's response is a paraphrase from intro statistics: "30 is the threshold for the central limit theorem. Anything above 30 is a valid sample." The forum accepts the response. Several readers deploy the strategy. Within six months, the realized win rate is 52%, the average win is 1.5R, the average loss is 1.4R, and the expectancy is approximately 0.1R per trade. The strategy was deployable on the 31-trade evidence in the same sense that a coin flipped 31 times that produced 22 heads is deployable as "biased toward heads" - which is to say, not at all. The "30 trades" threshold has a real origin in introductory statistics: the central limit theorem says the sample mean of independent observations becomes approximately normally distributed for n above approximately 30, regardless of the underlying distribution shape. The CLT is true. The CLT does not say what the trading-system seller above implied it said. The CLT applies to the distribution of the sample mean as an estimator, not to the precision of any specific estimate from a specific sample. The sample mean from 30 trades is normally distributed around the true mean with standard error sigma / sqrt(30); the standard error is large unless sigma is very small. For a typical trading-system R-multiple distribution with sigma approximately 1.5, the SE of the sample mean from 30 trades is approximately 0.27 R-multiples. The expectancy of 1.34 R-multiples reported above has a 95% CI of approximately \[0.80, 1.88\]. The true expectancy could be 0.5 R-multiples (a marginally profitable strategy that decays into noise) or 0.0 (no edge at all, the headline was sample variance), or 1.5 R-multiples (a real edge). The 31-trade record does not distinguish between these. The article ["Trade-Count Thresholds for Backtest Reliability"](https://aligrithm.com/trade-count-thresholds-for-backtest-reliability/) gave the general framework for sample-size requirements. This article isolates the specific and very common failure mode of treating 30 trades as adequate evidence. The 30-trade threshold is the most-cited piece of statistical folklore in retail trading, and it is the wrong threshold by a factor of 5-30 depending on which strategy property is being estimated. The article enumerates which properties the 30-trade threshold fails for and gives the right thresholds in each case. ## The CLT, narrowly stated The central limit theorem (Lindeberg-Levy version): for independent identically distributed observations X\_1, X\_2, ..., X\_n with finite mean mu and finite variance sigma^2, the sample mean (1/n) sum X\_i converges in distribution to N(mu, sigma^2 / n) as n grows. The threshold "n approximately 30" comes from the empirical observation that for many common distributions (uniform, exponential, mild-skew), the normal approximation is accurate at n approximately 30. The CLT's actual claim. The estimator (sample mean) has a known sampling distribution. The CLT's silences. The estimator from a specific sample is a precise estimate of mu. It is not. The standard error sigma / sqrt(n) for n = 30 and sigma = 1.5 is 0.27, which is large for almost any expectancy a trading system would claim. The CI half-width at 95% is approximately 2 SE = 0.54\. A reported expectancy of 1.0 R-multiple from 30 trades has a 95% CI of approximately \[0.46, 1.54\], which is consistent with edges ranging from "barely profitable" to "very strong". The CLT, five additional silences. Five things the CLT does not address, each material for trading-system validation. Silence 1: tail risk. The CLT applies to the sample mean. Tail-risk metrics (max drawdown, 95th-percentile loss, kurtosis) are functions of the entire empirical distribution, not the mean. Their estimation precision depends on the tail-region count, not the total n. With 30 trades, the largest observed loss is the 30th-quantile estimate; the actual 99th-quantile loss is unobserved. The strategy's tail risk is unmeasured. Silence 2: regime coverage. The CLT assumes i.i.d. observations. Trading-system observations are not i.i.d. across regimes. 30 trades from a single regime do not characterize the strategy's behavior across regimes. The article ["Regime Coverage: Why Your Backtest Needs Different Market States"](https://aligrithm.com/regime-coverage-why-your-backtest-needs-different-market-states/) framed the regime-stratification requirement; 30 trades is below the per-regime threshold for almost any meaningful regime stratification. Silence 3: serial correlation. Trading-system trades are often serially correlated (consecutive losses come in clusters during drawdowns; consecutive wins come in clusters during favorable regimes). The CLT's i.i.d. assumption is violated. The effective n is less than the headline n; for a strategy with autocorrelation rho approximately 0.3 across consecutive trades, the effective n is approximately n times (1 - rho) / (1 + rho) approximately 0.54 n. The effective sample size from 30 trades is approximately 16 trades. Silence 4: parameter-search bias. The 30 trades are typically the result of selecting from many candidate strategies and reporting the best. The article ["Degrees of Freedom in Trading Systems"](https://aligrithm.com/degrees-of-freedom-in-trading-systems/) framed the bias; for 30 trades and a search width k\_eff = 100, the bias on the IS expectancy is approximately sigma sqrt(2 log 100) / sqrt(30) approximately 0.78 R-multiples, which can swamp the apparent edge. Silence 5: cost-and-slippage uncertainty. 30 trades is a small sample for estimating the average cost impact, the slippage distribution, and the worst-case execution outcomes. The cost-uncertainty band on the expectancy is wider than the trade count suggests. ## Right thresholds for specific strategy properties Six strategy properties with their actual sample-size requirements. Property 1: expectancy / mean per-trade P&L. Required N for reliable estimate at typical sigma approximately 1.5: 200-500 trades. The 30-trade threshold underestimates by 7-17x. Property 2: Sharpe ratio. From ["Trade-Count Thresholds for Backtest Reliability"](https://aligrithm.com/trade-count-thresholds-for-backtest-reliability/), required N for reliable estimate at SR approximately 0.5-1.0: 250-500 trades. 30 trades is unreliable by 8-17x. Property 3: win rate (binary outcome). Standard error of win rate is sqrt(p (1-p) / N). For p = 0.6 and N = 30, SE = 0.089, CI half-width = 0.18\. The 71% win rate from 30 trades has 95% CI \[0.53, 0.89\]; the true win rate could be anywhere from "barely better than coin flip" to "strong edge". Required N for CI half-width of 0.05: approximately 384 trades for p = 0.5, fewer for very high or very low p. Property 4: maximum drawdown. The expected maximum drawdown from N trades scales with sqrt(N) and the per-trade volatility. A 30-trade backtest's worst-observed drawdown is approximately 25% of the long-run-expected maximum drawdown for a strategy with no negative skew. The strategy's true tail-risk profile is severely underestimated. Required N for stable max-drawdown estimation: 1000+ trades, with the regime-stratified version of the analysis. Property 5: per-regime behavior. With 4-6 distinct regimes and the requirement of approximately 50 trades per regime for any meaningful within-regime estimate, the total trade count for regime-stratified validation is 200-300 minimum, ideally 500-1000. Property 6: cost / slippage profile. The right-tail of the per-trade cost distribution (worst execution) is sampled poorly with 30 trades. Required N for stable cost-distribution estimation: 200+ trades, with explicit attention to the tail. The pattern. Every strategy property of interest has a sample-size requirement well above 30 trades. The CLT-derived 30-trade threshold is a starting point for asymptotic statistical theory, not a deployment threshold for trading-system validation. ## The 30-trade threshold's persistence Three structural reasons. Reason 1: it is taught as a rule of thumb in introductory statistics. The phrase "n above 30 for the CLT" is widespread in textbooks and is incorrectly transferred to "n above 30 is enough for reliable estimation". The transfer is wrong but easy to make. Reason 2: it justifies what retail traders want to do. A retail trader wants to deploy after running a strategy for a few weeks or a few months. 30 trades fits this timeline. The CLT-derived threshold provides a statistical-sounding justification for the desired conclusion. Reason 3: it is rarely challenged in retail forums. The seller cites the CLT; the audience accepts the citation; the few skeptics who object are dismissed as "too academic" or "moving the goalposts". The forum has no mechanism to enforce a higher threshold because there is no allocator-style deployment-failure feedback. The fix. Replace the CLT-threshold with the property-specific thresholds from the matrix above. Cite the actual sample-size requirement for the property being estimated, not the CLT. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## The math of 30-trade unreliability A simple calculation. For a strategy with true Sharpe SR = 0.7 (a strong but realistic structural edge), the observed IS Sharpe from 30 independent observations has SE approximately sqrt(1 + 0.5 (0.7)^2) / sqrt(30) approximately 0.21\. The 95% CI is approximately \[0.28, 1.12\]. $$ \\widehat{\\text{SR}}\_{30} \\in \[\\text{SR} - 2 \\cdot \\text{SE}, \\text{SR} + 2 \\cdot \\text{SE}\] = \[0.28, 1.12\] \\text{ when } \\text{SR} = 0.7, N = 30 $$ The CI of \[0.28, 1.12\] spans from "barely useful" to "exceptional". A 30-trade record cannot distinguish "this strategy has Sharpe 0.3" from "this strategy has Sharpe 1.1". The strategy's deployment decision should depend on which of these is true; the 30-trade record does not provide the information. For a strategy with true Sharpe SR = 0 (no edge at all), the observed IS Sharpe from 30 trades has SE approximately 0.18 and 95% CI approximately \[-0.37, 0.37\]. By chance alone, approximately 5% of zero-edge strategies will produce IS Sharpe above 0.37 from 30 trades. With multiple strategies tested (the article ["Degrees of Freedom in Trading Systems"](https://aligrithm.com/degrees-of-freedom-in-trading-systems/) framed the search-width bias), the probability of finding such a false positive grows quickly. $$ \\Pr(\\widehat{\\text{SR}}\_{30} > 0.5 \\mid \\text{SR} = 0) \\approx 0.07 $$ Given a zero-edge strategy and 30 trades, there is approximately a 7% probability of seeing an IS Sharpe above 0.5 by chance alone. With 10 strategies tested, the probability of at least one false positive at this threshold is approximately 1 - (1 - 0.07)^10 = 0.52\. The 30-trade record cannot distinguish a 7%-luck false positive from a real edge. ## Anti-patterns Five mistakes specific to the 30-trade misuse. Anti-pattern 1: citing the CLT as evidence the strategy is reliable. The CLT certifies the sampling distribution of the mean; it does not certify the precision of any specific estimate. The right citation is the standard error formula and the resulting CI, which for 30 trades is wide. Anti-pattern 2: declaring a strategy proven after 30 live trades. Live trades after deployment are valuable for OOS validation, but 30 OOS trades cannot reject the null hypothesis "the strategy has no edge" against a realistic alternative. The OOS sample needs to be large enough to detect the expected edge with adequate statistical power; for typical edges, 30 trades is insufficient by a factor of 5-15. Anti-pattern 3: averaging "30 trades good" with "more trades better" as if both perspectives have weight. The first perspective is wrong; averaging it with correct perspectives produces wrong conclusions. The right framing is that 30 trades is unreliable across all strategy properties of interest. Anti-pattern 4: applying the 30-trade threshold to per-regime trade counts. With 6 regimes and 30 trades total, each regime has 5 trades on average; per-regime estimates are useless. The per-regime requirement is approximately 50 trades minimum, which means the total trade-count requirement scales with the number of regimes. Anti-pattern 5: forum or social-media validation. A strategy with 30 trades that has been admired in a forum is not a validated strategy. The number of forum upvotes does not enter the standard error formula. The strategy needs the trade count, the parameter count, the regime coverage, the cost realism, and the OOS hold-out before deployment. None of these are in the forum metric. ## Decision matrix | Strategy property | Required N | 30 trades is short by | | ------------------------------- | --------------------------------- | --------------------- | | Expectancy (mean per-trade P&L) | 200-500 | 7-17x | | Sharpe ratio | 250-500 | 8-17x | | Win rate (CI half-width 0.05) | 384 | 13x | | Max drawdown (stable estimate) | 1000+ | 33x+ | | Per-regime behavior (6 regimes) | 300+ | 10x | | Cost / slippage tail | 200+ | 7x | | Search-width bias correction | depends on k\_eff, typically 500+ | 17x+ | | Out-of-sample validation | matches IS requirements | 7-30x | The matrix maps strategy property to actual sample-size requirement. The pattern: 30 trades is unreliable across every property of interest, by a factor of 5-30 depending on the property. ## Visualizing the 30-trade fallacy ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/05/article_73-visual-0.png) ## KEY POINTS - The "30 trades" threshold has a real origin in the central limit theorem: the sample mean of i.i.d. observations becomes approximately normally distributed for n above approximately 30\. The CLT says the sample mean has a normal sampling distribution; it does not say any specific 30-trade estimate is precise. - The standard error of a sample mean from 30 trades is sigma / sqrt(30) approximately 0.27 for typical sigma approximately 1.5 R-multiples. The 95% CI is approximately +/-0.54, which is wide for any expectancy a trading system would claim. - The CLT is silent on five trading-system requirements: tail risk (the largest observed loss in 30 trades is approximately the 30th-quantile, not the 99th-quantile), regime coverage (30 trades fits in one regime), serial correlation (effective n is approximately 0.5 n for typical autocorrelation), parameter-search bias (search-width inflation is large at small N), cost-and-slippage tail (the right-tail of cost distribution is undersampled). - Right thresholds for specific properties: expectancy 200-500 trades, Sharpe 250-500, win rate 384 for CI half-width 0.05, max drawdown 1000+, per-regime stratification 300+, cost tail 200+. 30 trades is short by 5-30x depending on the property. - For a true Sharpe of 0.7, the IS Sharpe from 30 trades has 95% CI approximately \[0.28, 1.12\]. The 30-trade record cannot distinguish a barely-profitable strategy from an exceptional one. - For a true Sharpe of 0, the probability of seeing IS Sharpe above 0.5 from 30 trades is approximately 7% by chance alone. With 10 strategies tested, the probability of at least one false positive is approximately 52%. - The 30-trade threshold persists because (1) it is taught as a CLT rule of thumb in introductory statistics, (2) it justifies what retail traders want to do (deploy after weeks/months not years), (3) it is rarely challenged in retail forums where there is no allocator-style deployment-failure feedback. - Anti-pattern: citing the CLT as evidence the strategy is reliable. The CLT certifies the sampling distribution of the mean, not the precision of any specific estimate. - Anti-pattern: declaring a strategy proven after 30 live trades. OOS sample needs adequate statistical power to detect the expected edge. - Anti-pattern: applying the 30-trade threshold to per-regime trade counts. With 6 regimes, the per-regime count from a 30-trade total is 5, which is useless. - Anti-pattern: forum or social-media validation. The number of upvotes does not enter the standard error formula. - The right framing: deployment requires the trade count appropriate to the property being estimated, not the CLT-folklore threshold. For most realistic strategies, the deployment-ready threshold is 250-1000 effective trades distributed across regimes. - The current article isolates the 30-trade misuse. The next article in the publication (["Monte Carlo for Trading Systems"](https://aligrithm.com/monte-carlo-for-trading-systems/)) covers the bootstrap and synthetic-path techniques that can extend the information from a given trade count to provide CI estimates and what-if scenarios. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Testing and Tuning Market Trading Systems - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Testing+and+Tuning+Market+Trading+Systems+Timothy+Masters&ref=aligrithm.com) - [Data Mining Algorithms in C++ - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Data+Mining+Algorithms+in+C%2B%2B+Timothy+Masters&ref=aligrithm.com) ### 3.19 Trade-Count Thresholds for Backtest Reliability URL: https://aligrithm.com/trade-count-thresholds-for-backtest-reliability/ Last updated: 2026-07-27T13:24:11.000Z A trader presents a backtest. The strategy traded 240 times over five years on a single instrument with average per-trade P&L of 0.4 R-multiples and standard deviation of 1.6 R-multiples. The annualized Sharpe is approximately 0.96\. The trader's pitch claims the Sharpe is "above 0.95, statistically significant, and reliable for deployment". The question that should follow but rarely does: how reliable is a Sharpe estimate from 240 trades? The answer is quantitative. The standard error of an annualized Sharpe estimate from N independent trades is approximately sqrt((1 + 0.5 SR^2) / N) for the unconditional case, or roughly 1/sqrt(N) for small Sharpes. For SR = 0.96 and N = 240, the standard error is approximately 0.07, and the 95% confidence interval is approximately \[0.82, 1.10\]. The true Sharpe could be anywhere in this band. If the true Sharpe is 0.5 (the realistic value for many "edge-y" strategies after mean reversion to the population), the OOS realization is plausibly within the band the IS estimate is consistent with, but the OOS would be 0.5 not 0.96\. The IS Sharpe of 0.96 carries a confidence band: "0.96 plus or minus 0.14, with a likely overestimate from search-width bias on top". The article ["Degrees of Freedom in Trading Systems"](https://aligrithm.com/degrees-of-freedom-in-trading-systems/) framed the search-width bias. The article ["The 10% Rule of Degrees of Freedom"](https://aligrithm.com/the-10-rule-of-degrees-of-freedom/) gave the relative ratio between parameters and trades. This article gives the absolute trade-count requirements: for a given Sharpe and a given confidence threshold, how many trades does the strategy need to produce a reliable estimate. The answer is calibrated by the standard error of the Sharpe and the desired statistical power. The article ["Why 30 Trades Is Not a Strategy"](https://aligrithm.com/why-30-trades-is-not-a-strategy/) later in this pillar covers the specific (and very common) misuse of the 30-trade central-limit-theorem threshold; the present article gives the general framework. ## The standard error of a Sharpe estimate For an unbiased Sharpe estimate from N independent observations of per-period returns with mean mu and standard deviation sigma, the standard error of the Sharpe (in the same time units as the observations) is approximately: $$ \\text{SE}(\\widehat{\\text{SR}}) \\approx \\sqrt{\\frac{1 + \\tfrac{1}{2} \\text{SR}^2}{N}} $$ For small Sharpes (SR < 1), the formula simplifies to approximately 1/sqrt(N). For SR = 0.5 and N = 100, SE approximately 0.10, 95% confidence interval approximately \[0.30, 0.70\]. For SR = 1.0 and N = 100, SE approximately 0.12, CI approximately \[0.76, 1.24\]. The CI is wide unless N is large. For the annualized Sharpe (where the per-period observations are typically daily or weekly), the formula adjusts for the annualization factor. If T is the number of trading periods per year (252 for daily, 52 for weekly, 12 for monthly), the annualized Sharpe is the per-period Sharpe times sqrt(T), and the standard error scales similarly. For a strategy with 250 trading days per year of returns: $$ \\text{SE}(\\widehat{\\text{SR}}\_{\\text{ann}}) \\approx \\sqrt{\\frac{1 + \\tfrac{1}{2} \\text{SR}\_{\\text{ann}}^2 / T}{N\_{\\text{years}} \\cdot T}} \\cdot \\sqrt{T} $$ The result. For an annualized Sharpe of 1.0 from one year of daily data (N approximately 250), SE approximately 0.45\. The 95% CI is approximately \[0.10, 1.90\]. One year of data is uninformative about the true Sharpe. For three years of data, SE approximately 0.27, CI approximately \[0.46, 1.54\]. For ten years of data, SE approximately 0.14, CI approximately \[0.72, 1.28\]. The trade-count requirement is large. ## Sample-size targets by Sharpe and confidence Operational targets calibrated against published practice and observed deployment outcomes. For a strategy with true Sharpe approximately 0.5 (a typical structural premium): | N trades | SE(SR) | 95% CI | OOS reliability | | -------- | ------ | -------------- | ------------------- | | 30 | 0.18 | \[0.14, 0.86\] | unreliable | | 100 | 0.10 | \[0.30, 0.70\] | weakly reliable | | 250 | 0.064 | \[0.37, 0.63\] | moderately reliable | | 500 | 0.045 | \[0.41, 0.59\] | reliable | | 1000 | 0.032 | \[0.44, 0.56\] | very reliable | For a strategy with true Sharpe approximately 1.0 (a strong edge): | N trades | SE(SR) | 95% CI | OOS reliability | | -------- | ------ | -------------- | ------------------- | | 30 | 0.22 | \[0.57, 1.43\] | unreliable | | 100 | 0.12 | \[0.76, 1.24\] | weakly reliable | | 250 | 0.077 | \[0.85, 1.15\] | moderately reliable | | 500 | 0.055 | \[0.89, 1.11\] | reliable | | 1000 | 0.039 | \[0.92, 1.08\] | very reliable | For a strategy with true Sharpe approximately 2.0 (an exceptional edge): | N trades | SE(SR) | 95% CI | OOS reliability | | -------- | ------ | -------------- | --------------- | | 30 | 0.32 | \[1.37, 2.63\] | unreliable | | 100 | 0.17 | \[1.66, 2.34\] | weakly reliable | | 250 | 0.11 | \[1.78, 2.22\] | reliable | | 500 | 0.077 | \[1.85, 2.15\] | very reliable | | 1000 | 0.055 | \[1.89, 2.11\] | very reliable | Reading the tables. "Reliable" means the 95% confidence interval excludes Sharpe = 0 by a wide margin and the half-width of the CI is small relative to the point estimate. "Weakly reliable" means the CI excludes Sharpe = 0 but the half-width is comparable to the point estimate, so the strategy has demonstrated nonzero edge but the magnitude is uncertain. "Unreliable" means the CI is wide enough that the strategy could have any value from near-zero to substantial edge. ## Effective trade count, defined The headline trade count from a backtest is rarely the effective N. Three corrections. Correction 1: hold-time overlap. A strategy that holds positions for 30 days and observes returns daily has approximately N\_observations / 30 effective independent trades. The 240 daily-observation strategy from the opening with a 5-day average hold has effective N approximately 50, not 240\. The standard error scales accordingly. Correction 2: cross-sectional dependence. A long-short equity strategy with 100 positions per side over 24 months has 4800 position-observations but the effective independent decision count is closer to 24 (the rebalance count) or 24 times the number of independent factor exposures (typically 3-7 in a well-constructed factor strategy). Effective N is the count of independent decisions, not positions. Correction 3: regime gating. A strategy that trades only when a regime indicator is active (covered in ["Volatility Regimes and Strategy Survival"](https://aligrithm.com/volatility-regimes-and-strategy-survival/)) has effective N equal to the trade count within the active regime, not the total observation count. A strategy gated to high-vol regimes that occur 20% of the time over 10 years of daily data has approximately 250 effective trades, not 2500. The honest count. Apply the corrections in sequence: divide by hold time, account for cross-sectional dependence, restrict to within-regime if gated. The result is the effective N to use in the Sharpe-confidence-interval calculation. ## Trust requirements beyond raw N Five additional requirements, each independently necessary. Requirement 1: sample regime coverage. The article ["Regime Coverage: Why Your Backtest Needs Different Market States"](https://aligrithm.com/regime-coverage-why-your-backtest-needs-different-market-states/) framed the requirement; the trade count must distribute across regimes the strategy will encounter live. A strategy with 1000 trades all in low-vol bull markets has zero effective N for high-vol bear-market deployment. The deployment regime determines which subset of trades is informative. Requirement 2: search-width correction. The article ["Degrees of Freedom in Trading Systems"](https://aligrithm.com/degrees-of-freedom-in-trading-systems/) framed the bias. The trade count provides the standard error; the parameter count provides the bias. Both matter. A high N with a high p / N ratio still produces an unreliable IS Sharpe because the bias from optimization dominates. Requirement 3: realistic costs. A backtest that ignores transaction costs over-states the Sharpe by 0.2-0.5 for active strategies. The trade-count threshold for "reliable" depends on the cost-adjusted Sharpe, not the gross Sharpe. Requirement 4: independent OOS slice. The IS Sharpe with confidence interval is one estimate. The OOS realization on a slice never seen during research is the validation. Both are required; neither alone is sufficient. The article ["Why OOS Failure Is Often a Stationarity Failure"](https://aligrithm.com/why-oos-failure-is-often-a-stationarity-failure/) framed the diagnostic flow when the OOS underperforms the IS. Requirement 5: structural premise. A statistically significant IS Sharpe on a strategy with no economic mechanism is suspicious in a way that a statistically significant Sharpe on a structural premium is not. The trade count is a noise-control diagnostic; it does not certify the underlying mechanism. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Extending N when needed Three approaches. Approach 1: lengthen the backtest. Use longer history. The trade-off: older data may be from a different personality (covered in ["Market Personality: Why Gold, FX, Crypto, and Equities Need Different Systems"](https://aligrithm.com/market-personality-why-gold-fx-crypto-and-equities-need-different-systems/)) and the resulting N is mixing trades from different regimes. The article ["Slow Wandering: The Most Dangerous Type of Market Change"](https://aligrithm.com/slow-wandering-the-most-dangerous-type-of-market-change/) framed the personality-drift problem. Approach 2: broaden the universe. Apply the strategy to multiple markets within a family (cross-market test from "Why Works on All Markets Is Usually a Red Flag", but within-family). The trade count multiplies. The trade-off: cross-market correlation reduces effective N by a factor that depends on the within-family correlation; for tightly-correlated indices like SPX and NDX, the multiplier is approximately 1.2-1.5x, not 2x. Approach 3: Monte Carlo augmentation. Bootstrap the existing trades to estimate the Sharpe distribution under resampling. Useful for confidence intervals but does not extend the underlying information; the bootstrap mean equals the IS mean and the bootstrap variance is determined by the IS variance. Covered in ["Monte Carlo for Trading Systems"](https://aligrithm.com/monte-carlo-for-trading-systems/) later in this pillar. The general rule. Lengthen and broaden first; bootstrap second; do not artificially extend N by counting overlapping observations. ## Anti-patterns Five mistakes specific to the trade-count question. Anti-pattern 1: declaring a backtest reliable based on the calendar length, not the trade count. A 5-year backtest of a strategy that trades twice per month has 120 trades. A 5-year backtest of a strategy that trades twice per day has 2500 trades. The reliability of the Sharpe estimate is very different despite the same calendar length. Anti-pattern 2: counting daily observations as trades for a hold-30-day strategy. The 240 effective trades become 8 effective trades after the hold-time correction. The Sharpe estimate's standard error is 5-6x larger than the naive calculation suggests. Anti-pattern 3: using the 30-trade threshold from intro statistics. The 30-trade threshold is for the central limit theorem to apply to the sample mean's distribution; it has nothing to do with the precision of the Sharpe estimate or the OOS reliability. Covered in detail in ["Why 30 Trades Is Not a Strategy"](https://aligrithm.com/why-30-trades-is-not-a-strategy/). Anti-pattern 4: ignoring search-width inflation when computing the confidence interval. The standard error formula assumes an unbiased estimator. The IS Sharpe from an optimization is biased. The right confidence interval is around the bias-corrected Sharpe, not the raw Sharpe. Anti-pattern 5: declaring a strategy "reliable" without the regime-coverage check. A strategy with 1000 trades all in one regime has 1000 noisy estimates of the within-regime Sharpe and zero estimates of the across-regime Sharpe. The reliability is regime-conditional. ## Decision matrix | True Sharpe | Required N for reliable estimate | Required N for very reliable | | ----------- | -------------------------------- | ---------------------------- | | 0.3 | 600+ | 2000+ | | 0.5 | 250 | 1000 | | 0.7 | 150 | 500 | | 1.0 | 100 | 400 | | 1.5 | 70 | 250 | | 2.0 | 50 | 200 | The matrix maps true Sharpe to required N. The pattern: stronger edges need fewer trades to validate. Most realistic strategies have true Sharpe between 0.3 and 0.8, requiring N between 250 and 1000 effective trades. ## Visualizing the trade-count requirement ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/05/article_72-visual-0.png) ## KEY POINTS - The standard error of a Sharpe estimate from N independent observations is approximately sqrt((1 + 0.5 SR^2) / N). For small Sharpes, SE approximately 1/sqrt(N). The 95% confidence interval is approximately \[SR - 2 SE, SR + 2 SE\]. - For an annualized Sharpe of 1.0 from one year of daily data, the SE is approximately 0.45 and the CI is approximately \[0.10, 1.90\]. One year of data is uninformative. - Operational trade-count thresholds: 100 trades = weakly reliable, 250 trades = moderately reliable, 500 trades = reliable, 1000+ trades = very reliable. The thresholds depend on the true Sharpe magnitude. - Effective N is not the headline trade count. Three corrections: divide by hold-time overlap, account for cross-sectional dependence (effective N approximately rebalance count for cross-sectional factors), restrict to within-regime trade count if gated. - A 240-day-observation strategy with 5-day hold time has effective N approximately 48, not 240\. A long-short factor with 4800 position-observations has effective N approximately 24-72, not 4800\. A regime-gated strategy active 20% of the time over 10 years has approximately 250 effective trades, not 2500. - The trade count provides the standard error of the Sharpe estimate. The parameter count (from ["Degrees of Freedom in Trading Systems"](https://aligrithm.com/degrees-of-freedom-in-trading-systems/)) provides the bias. Both matter; high N with high p / N still produces unreliable estimates. - Five additional requirements beyond raw N: sample regime coverage, search-width correction, realistic costs, independent OOS slice, structural premise. The trade count is a noise-control diagnostic, not a certification. - Three approaches to extending N: lengthen the backtest (with personality-drift caveat), broaden the universe within a market family (with correlation correction), Monte Carlo augmentation for CI estimation only (does not extend underlying information). - Anti-pattern: declaring a backtest reliable based on calendar length, not trade count. A 5-year backtest with 120 trades and one with 2500 trades are not equivalent. - Anti-pattern: counting daily observations as trades for hold-many-days strategies. Apply the hold-time correction. - Anti-pattern: using the 30-trade threshold from intro statistics. The 30-trade threshold is unrelated to Sharpe estimation precision. - Anti-pattern: ignoring search-width inflation when computing CIs. The CI should be around the bias-corrected Sharpe. - Anti-pattern: declaring "reliable" without regime-coverage check. Reliability is regime-conditional. - The current article gives the general framework for trade-count requirements. The next article in the publication (["Why 30 Trades Is Not a Strategy"](https://aligrithm.com/why-30-trades-is-not-a-strategy/)) covers the specific misuse of the 30-trade central-limit-theorem threshold in trading-system evaluation. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Testing and Tuning Market Trading Systems - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Testing+and+Tuning+Market+Trading+Systems+Timothy+Masters&ref=aligrithm.com) - [Data Mining Algorithms in C++ - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Data+Mining+Algorithms+in+C%2B%2B+Timothy+Masters&ref=aligrithm.com) ### 3.18 The 10% Rule of Degrees of Freedom URL: https://aligrithm.com/the-10-rule-of-degrees-of-freedom/ Last updated: 2026-07-27T13:24:11.000Z A breakout strategy on E-mini SPX futures runs from 2010 to 2024\. The backtest produces 87 trades. The strategy specification has 11 parameters by the conservative count from the article ["Degrees of Freedom in Trading Systems"](https://aligrithm.com/degrees-of-freedom-in-trading-systems/): 4 numeric parameters (entry lookback, exit lookback, ATR period, vol-target level), 2 structural choices (breakout type, exit-rule type), 2 filter choices (volatility-regime filter, time-of-day filter), 2 sizing choices (sizing scheme, position-cap policy), 1 cost-assumption choice. The IS Sharpe is 1.4\. The strategy ships in 2025. The 10% rule applied to this strategy: parameter count divided by trade count equals 11/87 = 12.6%. The rule says the ratio should be at most 10%, with discomfort starting at 5%. The strategy at 12.6% is in the "do not deploy" zone. The team deployed anyway. The OOS Sharpe through year 1 is approximately 0.1\. The post-mortem reveals that approximately 0.5 of the 1.3-Sharpe IS-OOS gap was directly attributable to the small number of trades relative to the parameter count. The other 0.8 was the structural decay of the strategy (the article ["Why Systems Work Until They Don't"](https://aligrithm.com/why-systems-work-until-they-dont/) framed the four mechanisms). The DoF-to-trade ratio was the single largest avoidable cause of the deployment failure. The 10% rule is a heuristic, not a theorem. The number is calibrated against thousands of backtest-to-live deployments observed across decades and across strategy families. The rule's structural justification: the trade count is the effective sample size for estimating the per-trade expectancy and Sharpe, while the parameter count is the search width over which the IS-optimal point was selected. When the search width approaches the sample size, the IS estimate is dominated by maximum-of-noise contributions and does not generalize. The 10% number is the empirical threshold below which the bias correction is small relative to the Sharpe being measured, and above which the bias correction dominates. This article gives the operational protocol for the rule. ## The rule, stated formally For a strategy with parameter count p and trade count N, the rule asserts: $$ \\frac{p}{N} \\leq 0.10 \\quad \\Rightarrow \\quad \\text{deploy candidate (subject to other checks)} $$$$ \\frac{p}{N} > 0.10 \\quad \\Rightarrow \\quad \\text{do not deploy. either reduce } p \\text{ or extend } N \\text{ or accept the strategy is exploratory} $$ The 10% threshold is an upper bound, not a target. A ratio of 1-3% is preferred. A ratio of 5-7% is workable for high-Sharpe structural strategies. A ratio of 8-10% requires the bias correction from ["Degrees of Freedom in Trading Systems"](https://aligrithm.com/degrees-of-freedom-in-trading-systems/) to be reported alongside the raw Sharpe. A ratio above 10% is in the "exploratory only" zone. The structural justification. The standard error of an IS Sharpe estimate with N trades is approximately 1/sqrt(N - 1). The bias from optimization across p parameters scales with sqrt(p) (each parameter is approximately one degree of freedom in the maximum-of-noise calculation). The bias-to-noise ratio is therefore approximately sqrt(p) / (1/sqrt(N)) = sqrt(p \* N). For the bias to be small relative to the Sharpe, we need p / N to be small. The 10% threshold corresponds to a bias-to-noise ratio of approximately 1, which is the largest tolerable contamination of the IS estimate. ## Counting p correctly The article ["Degrees of Freedom in Trading Systems"](https://aligrithm.com/degrees-of-freedom-in-trading-systems/) gave the eight categories. The 10% rule uses the same count. Five practical clarifications. Clarification 1: numeric parameters are counted as the cardinality of the grid considered, not 1\. An entry lookback chosen from 8 candidate values contributes 8 to the count, not 1\. Anyone who counts only the chosen value is undercounting by an order of magnitude. Clarification 2: structural alternatives count as the number of candidates considered. Choosing breakout-type from {Donchian, ATR-buffered, intraday, close-above-N-high} contributes 4 to the count. Clarification 3: filters that were tried and removed still count. If the team considered a 200-day MA filter and decided not to include it, the choice "filter or no filter" is still a 2-way decision, and the parameter range of the filter when it was tested is also a contribution. Hidden DoF is the largest underestimation source. Clarification 4: market and time-period selection count. The choice of "SPX rather than NDX" is a DoF if the team considered both. The choice of "2010-2024 rather than 2000-2014" is a DoF. Clarification 5: abandoned strategies count multiplicatively. If the team tried 8 strategies and reports the best, the implicit p is multiplied by 8 because the report is the maximum across the 8 attempts. The honest count. Apply the eight-category framework, multiply, get k\_eff. For the 10% rule, the effective p is approximately log2(k\_eff) (the number of bits of information the search width corresponds to). For k\_eff = 1024, p\_effective approximately 10\. For k\_eff = 1 million, p\_effective approximately 20\. Use p\_effective in the ratio. ## Counting N correctly The trade count is the number of independent realizations of the strategy's per-trade P&L. Three subtleties. Subtlety 1: hold time and overlap. If the strategy holds positions for 30 days and rebalances daily, the daily observations are not independent trades; the effective number of trades is approximately the total holding-period count divided by the hold time. A strategy with 5000 daily observations and 30-day holds has effective N approximately 5000 / 30 = 167 trades. Subtlety 2: cross-sectional positions. A long-short equity strategy with 100 longs and 100 shorts rebalanced monthly over 24 months has 200 positions per month times 24 months = 4800 position-observations, but the effective number of independent decisions is closer to 24 (the number of monthly rebalances) because the longs and shorts are mostly correlated through the market and sector factor structure. Effective N is the number of independent rebalance decisions, not the position count. Subtlety 3: regime-conditional trades. If the strategy is gated to a specific volatility regime that occurs in 30% of the sample, and the total observation count is 1000 days, the effective N is the trade count within the regime, which may be 30 or 50 trades. Strategies that are gated to rare regimes have small effective N regardless of the total observation count, and the 10% rule penalizes accordingly. The honest count. Use the effective number of independent trade outcomes, not the total observation count, not the total position count, not the total signal count. For most strategies, effective N is between 50 and 5000. ## Extending N vs reducing p Three scenarios. Scenario 1: the ratio is 12% with N = 100 trades and p = 12\. The strategy hits a small-trade-count limit. Extending N requires longer backtests (decades, not years) or more markets (cross-asset families per ["Market Personality: Why Gold, FX, Crypto, and Equities Need Different Systems"](https://aligrithm.com/market-personality-why-gold-fx-crypto-and-equities-need-different-systems/)). Reducing p requires removing parameters that do not improve survival per ["Why More Parameters Make a Strategy Easier to Sell and Easier to Break"](https://aligrithm.com/why-more-parameters-make-a-strategy-easier-to-sell-and-easier-to-break/). For most strategies in this scenario, reducing p is faster and easier than extending N. Scenario 2: the ratio is 12% with N = 5000 trades and p = 600\. The parameter count exceeds the threshold by a wide margin. Reduce p by removing parameters that do not have a structural justification. This is the most common scenario in machine-learning-heavy strategies and ensemble systems. Scenario 3: the ratio is 12% with N = 30 trades and p = 4\. Both numbers are too small. The strategy is exploratory; the trade count is too small for any reliable estimate even at the structural-prior parameters. Extend N (apply to more markets, longer history) or accept the exploratory status and do not deploy capital at scale. The general rule. Reducing p is usually faster, cheaper, and more effective than extending N. The article ["Why More Parameters Make a Strategy Easier to Sell and Easier to Break"](https://aligrithm.com/why-more-parameters-make-a-strategy-easier-to-sell-and-easier-to-break/) framed the parameter-removal discipline; the 10% rule provides the quantitative threshold for when to apply it. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Limits of the rule Three common misinterpretations. Misinterpretation 1: "if the ratio is below 10%, the strategy is good". The 10% rule is necessary, not sufficient. A strategy at 5% ratio with no structural premise, no economic mechanism, and no regime coverage is still bad. The rule eliminates one failure mode (parameter overfitting); it does not certify the others. Misinterpretation 2: "if the ratio is above 10%, the strategy is bad". A strategy at 15% ratio with strong economic mechanism and clear cross-market validation may still be deployable, but with a smaller capital commitment, longer probation period, and earlier kill-switches. The article ["How to Detect When a Trading System Is Dying"](https://aligrithm.com/how-to-detect-when-a-trading-system-is-dying/) framed the operational gating; the 10% rule informs the gating thresholds. Misinterpretation 3: "the rule is exact at 10%". The number is calibrated empirically. The right interpretation is: low ratios (under 5%) are robust to most parameter-counting errors, ratios in the 5-10% zone require careful counting and bias-corrected reporting, ratios above 10% are red flags that require explicit justification. ## Anti-patterns Five mistakes specific to the 10% rule. Anti-pattern 1: counting only the visible numeric parameters. A strategy reported as "4 parameters, 100 trades, ratio 4%" may be "60 effective parameters once structural choices and abandoned strategies are counted, ratio 60%". The rule is only useful with honest counting. Anti-pattern 2: counting position-observations as trades. A strategy with 5000 daily observations is reported with N = 5000\. The effective N (after hold-time correction and cross-section dependence) is closer to 100\. The reported ratio is misleading. Anti-pattern 3: applying the rule only to strategies that fail it. A strategy at 7% ratio is shipped without further checks. A strategy at 12% ratio is rejected. The rule is applied as a binary filter rather than as a quantitative input to the bias correction. The right application is: count ratio for every strategy, report bias-corrected Sharpe alongside raw, gate deployment based on combined diagnostics. Anti-pattern 4: ignoring the rule on "structural" strategies. The team claims "this is a long-only equity strategy with no real optimization, the ratio does not apply". The structural prior choice was a DoF (which paper to follow, which parameters from which paper). The market and time-period selection were DoFs. The rule applies, with a smaller numerator. Anti-pattern 5: treating the rule as a substitute for OOS testing. The rule informs the IS-OOS gap expectation; it does not replace the OOS validation. A strategy that passes the rule still needs the OOS check on a slice not seen during research. A strategy that fails the rule is unlikely to pass the OOS check, but the OOS check is the actual validation, not the rule. ## Decision matrix | p / N ratio | Deployment status | Required reporting | | -------------------------------------- | --------------------------------------------------------------- | -------------------------------------------------- | | 0-3% | Robust to parameter-counting errors, deploy | Raw IS Sharpe | | 3-5% | Solid, deploy with standard monitoring | Raw IS Sharpe + sensitivity | | 5-7% | Workable, deploy with bias-corrected reporting | Raw + bias-corrected Sharpe | | 7-10% | Marginal, deploy with smaller capital and tighter kill-switches | Bias-corrected Sharpe + OOS hold-out result | | 10-15% | Exploratory only, no large-capital deployment | Bias-corrected, OOS, regime-stratified | | 15-25% | Research stage, not deployment | Bias-corrected, OOS, parameter-count justification | | 25%+ | Pure overfit, do not deploy | Reduce p before any deployment consideration | | Any ratio with hidden DoF (Category 8) | Re-count with abandoned-strategy multiplier | Apply rule to corrected ratio | The matrix maps ratio to status. The pattern: lower is better, and full-category counting is more important than the threshold itself. ## Visualizing the rule ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/05/article_71-visual-0.png) ## KEY POINTS - The 10% rule of degrees of freedom: the parameter count divided by the effective trade count should be at most 10%. A ratio of 1-3% is preferred. A ratio of 5-10% requires bias-corrected reporting alongside raw IS Sharpe. A ratio above 10% is in the "exploratory only" zone. - The structural justification: the standard error of an IS Sharpe estimate with N trades is approximately 1/sqrt(N), the bias from optimization across p parameters scales with sqrt(p), and the bias-to-noise ratio scales with sqrt(p N). The 10% threshold corresponds to a bias-to-noise ratio of approximately 1. - The parameter count p is the eight-category total from ["Degrees of Freedom in Trading Systems"](https://aligrithm.com/degrees-of-freedom-in-trading-systems/): numeric grids, structural choices, filters and gating, sizing schemes, market and time-period selection, cost assumptions, data choices, abandoned-strategy selection. - The effective trade count N is the number of independent realizations of per-trade P&L, not the total observation count. Subtleties: hold-time overlap (effective N approximately observations divided by hold time), cross-sectional position dependence (effective N approximately rebalance count for long-short factor strategies), regime-conditional trades (effective N is the trade count within the gating regime). - When the ratio is too high, the typical fix is to reduce p, not to extend N. Reducing p is usually faster, cheaper, and more effective than extending N. - Three scenarios for ratio violations: small N strategies need extension or removal of parameters, large N strategies with high p need parameter pruning, very small N strategies are exploratory and should not be deployed at scale regardless of p. - The rule does not certify a strategy. A low ratio is necessary but not sufficient; a high ratio is a red flag but not an automatic rejection. The rule informs the deployment gating in combination with structural premise, regime coverage, and OOS validation. - Anti-pattern: counting only the visible numeric parameters. The full-category p is typically 5-10x the visible count. - Anti-pattern: counting position-observations as trades. Effective N corrects for hold-time overlap and cross-sectional dependence. - Anti-pattern: applying the rule as a binary filter rather than as a quantitative input. The right application: report ratio for every strategy, report bias-corrected Sharpe, gate deployment based on combined diagnostics. - Anti-pattern: ignoring the rule on "structural" strategies. The structural prior choices are still DoFs. - Anti-pattern: treating the rule as a substitute for OOS testing. The rule informs the IS-OOS gap expectation; the OOS check is the actual validation. - The current article gives the quantitative threshold for parameter-to-trade ratios. The next article in the publication (["Trade-Count Thresholds for Backtest Reliability"](https://aligrithm.com/trade-count-thresholds-for-backtest-reliability/)) gives the absolute trade-count requirements that complement the relative ratio. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Testing and Tuning Market Trading Systems - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Testing+and+Tuning+Market+Trading+Systems+Timothy+Masters&ref=aligrithm.com) - [Data Mining Algorithms in C++ - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Data+Mining+Algorithms+in+C%2B%2B+Timothy+Masters&ref=aligrithm.com) ### 3.17 Why More Parameters Make a Strategy Easier to Sell and Easier to Break URL: https://aligrithm.com/why-more-parameters-make-a-strategy-easier-to-sell-and-easier-to-break/ Last updated: 2026-07-27T13:24:10.000Z Two strategies are presented to the same allocator on the same morning. Strategy A is a long-short equity factor with three explicit parameters: 12-month formation period, equal-weighted decile portfolios, monthly rebalance. The pitch deck is twelve pages. The IS Sharpe is 0.62 over thirty years. The factor exposure decomposition is one paragraph. The allocator says: "looks too simple, what is the edge here that is not already arbitraged?". Strategy B is a multi-signal ensemble with seventeen parameters: regime classifier, volatility estimator, sector tilts, value-momentum-quality blend with adaptive weights, drawdown-conditional position sizing, machine-learning-driven entry timing, cross-asset confirmation. The pitch deck is fifty-eight pages with shaded charts, optimization heatmaps, and per-factor attribution. The IS Sharpe is 1.55 over the past decade. The allocator says: "this team has done serious work, the methodology is sophisticated". Three years later, Strategy A is still trading at approximately Sharpe 0.55\. Strategy B has decommissioned, returned 40% of capital, and the team is rebuilding. The allocator has fired Strategy B's manager and re-allocated to Strategy A's team. The pattern has repeated thousands of times in real allocator-manager relationships. The simple strategy looked unimpressive on the pitch but survived deployment. The complex strategy looked impressive on the pitch and broke under live conditions. The article ["Degrees of Freedom in Trading Systems"](https://aligrithm.com/degrees-of-freedom-in-trading-systems/) gave the formal counting framework. This article gives the asymmetry it produces in the marketing-vs-engineering tradeoff. Each additional parameter accomplishes two things: (1) it improves the IS Sharpe by raising the maximum-of-noise floor, which makes the pitch more compelling; (2) it raises k\_eff, which inflates the IS-OOS gap and makes the strategy more fragile in deployment. The two effects work in opposite directions for the strategy's lifecycle. Adding parameters helps win the mandate. Adding parameters loses the mandate when deployed. The asymmetry is the structural consequence of how the IS objective and the OOS performance respond differently to model complexity. Calling it a paradox hides the mechanism. ## The mechanics of "more sellable" Five mechanisms by which adding parameters improves the pitch. Mechanism 1: higher IS Sharpe. The expected maximum of k\_eff noisy realizations of the true Sharpe scales with sqrt(2 log k\_eff). Adding parameters raises k\_eff and the headline IS Sharpe with it. A pitch with IS Sharpe 1.5 sells more easily than a pitch with IS Sharpe 0.6, regardless of which one is real. Mechanism 2: more dimensions of "robustness narrative". Each parameter generates an optimization heatmap, a parameter-stability plot, a sensitivity analysis. The pitch deck has more content. Each chart suggests the team performed careful work. The total pages of analysis correlate (in the allocator's perception) with the rigor of the research, even though the actual rigor is the opposite of the parameter count. Mechanism 3: more flavors of explanation. With seventeen parameters, the team can explain why the strategy worked in any historical regime by pointing to the parameter that "captured" that regime. The strategy worked in 2008 because the volatility regulator activated. The strategy worked in 2020 because the drawdown filter dampened. Each explanation is post-hoc and possibly correct, but the pattern (each historical event has a corresponding parameter) is suspicious in the same way that an explanation that fits all data is suspicious in any field. Mechanism 4: defensibility under questioning. An allocator who asks "why did the strategy underperform in 2017?" gets a different answer from a 17-parameter strategy than from a 3-parameter strategy. The 17-parameter strategy can claim "the regime filter detected late, we have updated the threshold for next time". The 3-parameter strategy must answer "the strategy is meant to underperform in years like 2017 because the structural premise weakens; we did not change anything". The first answer sounds more responsive; the second is more honest. Mechanism 5: differentiation from competitors. Many strategies look similar at the structural-prior level (everyone has a momentum factor, a value factor, a quality factor). The differentiation in the pitch comes from the proprietary additions (the regime classifier, the adaptive weights, the cross-asset confirmation). Each addition is a parameter. The pitch becomes more distinct from competitors at the cost of becoming more fragile. ## The mechanics of "easier to break" Four mechanisms by which adding parameters reduces deployment survival. Mechanism 1: search-width inflation. Each additional parameter multiplies k\_eff. The article ["Degrees of Freedom in Trading Systems"](https://aligrithm.com/degrees-of-freedom-in-trading-systems/) framed the counting; the deployment-failure consequence is that the bias-corrected IS Sharpe drops faster than the raw IS Sharpe rises. Adding a parameter typically improves raw IS Sharpe by 0.1-0.3 units while inflating the bias by 0.05-0.2 units. The expected OOS gain is small or negative. Mechanism 2: regime-overfit per parameter. Each parameter is tuned to the IS regime distribution. When the OOS regime distribution drifts (covered in ["Slow Wandering: The Most Dangerous Type of Market Change"](https://aligrithm.com/slow-wandering-the-most-dangerous-type-of-market-change/)) each parameter contributes a regime-mismatch loss. The 17-parameter strategy has 17 sources of regime drift; the 3-parameter strategy has 3. Mechanism 3: implementation complexity. Each parameter requires data, monitoring, and operational discipline. The 17-parameter strategy has 17 things that can go wrong in production: a stale data feed, a missed reset, a vendor outage, a bug in one of the sub-models. The 3-parameter strategy has 3\. Operational reliability scales inversely with parameter count. Mechanism 4: communication failure on decommission. When the strategy underperforms, the allocator and the manager need to agree whether the underperformance is structural (decommission) or transient (extend runway). For a 3-parameter strategy, the diagnosis is straightforward: the structural premise either holds or not. For a 17-parameter strategy, the diagnosis is ambiguous: one parameter may be broken, two regimes may be mismatched, or the ensemble weights may be wrong. The ambiguity delays the decommission decision and increases the eventual losses. ## The marketing-engineering tradeoff curve A simple model. The expected pitch quality (probability of winning the mandate) scales monotonically with the raw IS Sharpe. The expected deployment survival scales inversely with k\_eff. The optimal parameter count from the manager's perspective (maximize pitch quality) and the optimal parameter count from the allocator's eventual P&L perspective (maximize survival) are different. $$ \\text{PitchQuality}(p) \\propto \\widehat{\\text{SR}}\_{\\text{IS}}(p) = \\text{SR}\_{\\text{true}}(p) + \\sigma\_{\\text{IS}} \\sqrt{2 \\log k\_{\\text{eff}}(p)} $$$$ \\text{Survival}(p) \\propto \\text{SR}\_{\\text{true}}(p) - \\alpha \\cdot \\log k\_{\\text{eff}}(p) $$ The pitch-quality curve increases roughly monotonically with parameter count p (more parameters, more search width, higher max-of-noise IS Sharpe). The survival curve has a maximum at a moderate parameter count and decreases beyond it (each parameter past the optimum costs more in regime overfit and operational complexity than it gains in true Sharpe). The asymmetry between the two curves is the structural cause of the misalignment between manager incentives and allocator outcomes. ## Legitimate reasons to add a parameter Three legitimate reasons to add a parameter. Reason 1: the parameter encodes a structural feature that demonstrably affects the strategy's risk profile. Vol-targeting is a parameter; it encodes the structural feature that volatility is non-stationary. A long-only equity strategy without vol-targeting has different drawdown behavior than one with vol-targeting; the parameter is operationally necessary, not search-width inflation. Reason 2: the parameter handles a known structural break. A regime indicator that switches between long-trend and short-trend modes captures a structural feature of certain markets (commodity carry decay covered in ["Slow Wandering: The Most Dangerous Type of Market Change"](https://aligrithm.com/slow-wandering-the-most-dangerous-type-of-market-change/), FX carry collapse during crisis). The parameter is justified by the structural behavior, not by IS optimization. Reason 3: the parameter is required by the operational constraints. A position-size cap is a parameter that prevents the strategy from concentrating risk in any single position. The cap is set by risk policy, not by IS optimization, and improves survival regardless of the IS gain. The discipline. For each parameter, the question is: "does this parameter improve survival, or does it improve the pitch?". If the answer is the latter, the parameter is search-width inflation and should be removed. If the answer is the former, the parameter is justified. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Resisting the parameter-creep pressure Five operational defenses. Defense 1: structural prior discipline. Every parameter starts at a structural prior (literature, mechanism, cross-asset analog). The parameter is set, never optimized. The article ["Optimization Comes After Testing, Not Before"](https://aligrithm.com/optimization-comes-after-testing-not-before/) framed the procedural sequencing. Defense 2: parameter budget. Set a maximum parameter count before research begins. A common heuristic is the 10% rule (covered in ["The 10% Rule of Degrees of Freedom"](https://aligrithm.com/the-10-rule-of-degrees-of-freedom/)): the parameter count should be at most 10% of the trade count. A strategy with 100 trades should have at most 10 parameters; with 1000 trades, at most 100. Defense 3: bias-corrected reporting. Always report k\_eff and the bias-corrected IS Sharpe alongside the raw. The bias-corrected number penalizes parameter creep automatically. The team's incentive to add parameters is reduced when each addition shows up as a higher k\_eff and a smaller bias-corrected Sharpe. Defense 4: independent OOS hold-out. Set aside 20-30% of the data as a true OOS slice that is never seen during research. Test the final strategy on the slice once, with no iteration. The slice imposes a hard penalty on parameter creep because each parameter raises the IS-OOS gap and the OOS slice realizes the penalty. Defense 5: post-decommission attribution. When a strategy is decommissioned, do an attribution: which parameter contributed most to the failure? If the attribution consistently points to the parameters added late in research, the team's research discipline is broken and the parameter budget should be tighter for the next strategy. ## Anti-patterns Five mistakes specific to the parameter-marketing problem. Anti-pattern 1: adding parameters in response to allocator feedback. The allocator says "what about regime changes?" and the team adds a regime classifier. The parameter exists to satisfy the allocator's question, not because the structural analysis demanded it. The strategy is now more sellable and less robust. The right response to allocator feedback is "the structural prior already handles this through X" or "we considered this and decided not to add a parameter because Y". Anti-pattern 2: adding parameters in response to OOS underperformance. The strategy underperforms; the team adds a filter to "fix" the underperformance; the new filter validates on the recent data; the next OOS underperforms again. The cycle continues. The article "Why OOS Failure Is Often a Stationarity Failure" framed the diagnostic flow; the parameter-creep response is the wrong action. Anti-pattern 3: ensemble dressing. The team builds three simple strategies and combines them with optimized weights into an ensemble. The ensemble has more parameters (the weights) and a higher IS Sharpe than any single component. The ensemble looks more sophisticated. The ensemble breaks first because the weights are tuned to the IS-period correlation structure, which does not survive OOS. Anti-pattern 4: machine-learning veneer. A traditional strategy is wrapped in an ML classifier that adds dozens of free parameters. The pitch claims the ML "learned the signal". The IS Sharpe is high. The OOS Sharpe is low. The ML did not learn the signal; the ML overfit the noise. ML is a legitimate tool when the parameter count is controlled with regularization and the signal-to-noise is high; ML is a parameter-creep accelerant when applied without that discipline. Anti-pattern 5: confusing complexity with sophistication. A 17-parameter strategy is more complex than a 3-parameter strategy. Complexity is not sophistication. Sophistication is the discipline of removing parameters that do not improve survival. A research team that produces a 3-parameter strategy with positive OOS is more sophisticated than a team that produces a 17-parameter strategy with negative OOS. ## Decision matrix | Parameter count | Typical pitch reception | Typical OOS survival | | ------------------------------ | ------------------------------- | -------------------------------------- | | 1-3 (structural prior only) | "Too simple, what is the edge?" | High survival, low IS Sharpe | | 4-7 (well-justified additions) | "Reasonable, want to see more" | Moderate survival, moderate IS Sharpe | | 8-15 (parameter creep zone) | "Sophisticated, impressive" | Low survival, high IS Sharpe | | 16+ (complexity for marketing) | "Highly sophisticated team" | Very low survival, very high IS Sharpe | | With bias-corrected reporting | Pitch reception drops by \~30% | Survival improves by \~50% | | With parameter budget enforced | Pitch reception drops by \~20% | Survival improves by \~30% | | With independent OOS hold-out | Pitch reception drops by \~10% | Survival improves by \~70% | The matrix is the misalignment quantified. Pitch reception and OOS survival move in opposite directions across parameter counts. The defensive measures (bias-corrected reporting, parameter budgets, independent OOS) shift survival without large pitch-reception costs. ## Visualizing the asymmetry ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/05/article_70-visual-0.png) ## KEY POINTS - More parameters make a strategy easier to sell because (1) the IS Sharpe inflates with sqrt(2 log k\_eff), (2) there are more dimensions of robustness narrative for the pitch deck, (3) post-hoc explanations of historical regimes become available, (4) the strategy is defensible under questioning, (5) it differentiates from competitors who all have the same structural priors. - More parameters make a strategy easier to break because (1) the bias-corrected IS Sharpe drops faster than the raw IS rises, (2) each parameter contributes a regime-mismatch loss when OOS regime drifts, (3) operational reliability scales inversely with parameter count, (4) decommission diagnosis is delayed by the ambiguity of which parameter failed. - The asymmetry is structural: pitch quality and survival respond differently to parameter count. Adding parameters helps the manager win mandates and hurts the allocator's eventual P&L. The misalignment is the consequence of how the IS objective and OOS performance respond differently to complexity. The "paradox" framing hides the mechanism. - Three legitimate reasons to add a parameter: encodes a structural feature (vol-targeting handles non-stationary vol), handles a known structural break (regime classifier for documented regime shifts), required by operational constraints (position-size cap from risk policy). For each parameter, ask: does this improve survival or improve the pitch? - Five operational defenses against parameter creep: structural prior discipline (parameters set, not tuned), parameter budget (10% rule), bias-corrected reporting (k\_eff and corrected Sharpe alongside raw), independent OOS hold-out (final test on never-seen slice), post-decommission attribution (which parameter caused the failure). - Anti-pattern: adding parameters in response to allocator questions. The allocator's question is satisfied by a parameter; the strategy becomes more sellable and less robust. - Anti-pattern: adding parameters in response to OOS underperformance. The new parameter validates on recent data; the next OOS fails again; the cycle continues. - Anti-pattern: ensemble dressing. Combining simple strategies with optimized weights raises IS Sharpe by tuning the IS-period correlation structure. The weights do not survive OOS. - Anti-pattern: machine-learning veneer. ML wrapped around a strategy adds dozens of free parameters. Without explicit regularization and high signal-to-noise, the ML overfits. - Anti-pattern: confusing complexity with sophistication. Sophistication is the discipline of removing parameters that do not improve survival. The 3-parameter strategy with positive OOS is more sophisticated than the 17-parameter strategy with negative OOS. - The pitch reception is not the right metric for parameter count. The right metric is bias-corrected IS Sharpe and OOS survival probability. The defensive measures shift the team's incentive toward the right metric without large pitch-reception costs. - The current article gives the marketing-vs-engineering trade-off. The next article in the publication (["The 10% Rule of Degrees of Freedom"](https://aligrithm.com/the-10-rule-of-degrees-of-freedom/)) gives the operational threshold for parameter counts relative to trade counts. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Testing and Tuning Market Trading Systems - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Testing+and+Tuning+Market+Trading+Systems+Timothy+Masters&ref=aligrithm.com) - [Data Mining Algorithms in C++ - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Data+Mining+Algorithms+in+C%2B%2B+Timothy+Masters&ref=aligrithm.com) ### 3.16 Degrees of Freedom in Trading Systems URL: https://aligrithm.com/degrees-of-freedom-in-trading-systems/ Last updated: 2026-07-27T13:24:10.000Z A retail trader presents a system at a conference. The pitch: "a simple breakout strategy on SPX futures. Buy when price closes above the 20-day high. Exit when price closes below the 10-day low. Position size based on 14-day ATR. The IS Sharpe is 1.8 over 2010-2020." The audience hears "simple" and counts what they consider the parameters: lookback for entry (20), lookback for exit (10), ATR period (14). Three parameters. Sounds reasonable. The system is purchased by a small fund and deployed in 2021\. The OOS Sharpe through 2024 is approximately 0.2. The post-mortem reveals the actual degrees of freedom. The breakout type was chosen (close-above-N-high vs intraday-above-N-high vs Donchian-channel vs ATR-buffered-breakout) - that is one binary or categorical choice across at least four candidates. The entry lookback (20) was selected from a small grid that the trader admits considering values of 10, 15, 20, 25, 30, and 40 - that is a 6-way choice. The exit lookback (10) was similarly chosen from {5, 7, 10, 14, 20} - a 5-way choice. The ATR period (14) is a 5-way choice from common values {7, 10, 14, 20, 28}. The position-sizing scheme was chosen (raw ATR vs ATR scaled by equity vs ATR scaled by 5-day average ATR) from at least three candidates. The market choice (SPX futures) was made from at least 5 liquid index futures (SPX, NDX, RTY, NIKKEI, EURO STOXX). The time period (2010-2020) was chosen rather than (2000-2010) or (1995-2010) or (2005-2015), each of which would have produced different IS Sharpes. The "no costs" framing was a methodological choice. The "no overnight gap risk" was unstated but implicit. The "no slippage above one tick" was an assumption. The actual degree-of-freedom count is approximately 4 (breakout type) x 6 (entry lookback) x 5 (exit lookback) x 5 (ATR period) x 3 (sizing scheme) x 5 (market choice) x 4 (time period) x 2 (cost assumption) = 144000 effective candidate configurations. The trader explored, consciously or unconsciously, a substantial fraction of this space and reported the maximum. The IS Sharpe of 1.8 was the maximum of approximately 144000 noise-plus-signal realizations; the OOS Sharpe of 0.2 is what remains when only the signal generalizes. The strategy was never "three parameters". It was thousands of effective parameters, dressed in a presentation that hid most of them. This article gives the framework for counting degrees of freedom without omission, so that the search-width bias from ["The Difference Between Robustness and Optimization"](https://aligrithm.com/the-difference-between-robustness-and-optimization/) can be made quantitative. ## Counting degrees of freedom Eight categories, often hidden. Category 1: explicit numeric parameters. The lookback windows, thresholds, ATR multipliers, vol-target levels, stop-loss levels, take-profit levels. These are the visible degrees of freedom. The discrete grid the researcher considered (e.g., entry lookbacks from 10 to 40 in steps of 5) is the relevant search width, not the single value reported. Category 2: structural choices. Indicator type (RSI vs MACD vs ROC vs custom), entry rule type (breakout vs mean reversion vs crossover vs filter), exit rule type (time stop vs price stop vs ATR stop vs signal reversal). Each structural choice is a categorical degree of freedom with cardinality equal to the number of candidates the researcher considered. Category 3: filter and gating choices. "Only trade when 200-day MA is rising". "Only trade in volatility regime X". "Skip the first and last 30 minutes of the session". "Skip the day before and after FOMC". Each filter is a degree of freedom whose presence/absence and parameterization were chosen from candidates. Category 4: position-sizing and risk-management choices. Fixed-size vs vol-targeted vs Kelly-fraction vs equity-curve-trailing. Each scheme has parameters (vol target, lookback, scaling factor). The choice of scheme and the values are degrees of freedom. Category 5: asset universe and market selection. Which markets are included, which excluded, which time period is used as IS, which as OOS. Each inclusion/exclusion decision is a degree of freedom. The article "Why Works on All Markets Is Usually a Red Flag" framed the cross-market overfitting; the present article frames the same problem as a degrees-of-freedom count. Category 6: cost and execution assumptions. Per-trade commission, slippage assumption (linear vs square-root vs ATR-scaled), bid-ask treatment (mid-price vs full spread vs realistic fills). The choice of assumption is a degree of freedom that often shifts the IS Sharpe by 20-50% in either direction. Category 7: data choices. Bar interval (5-minute vs 15-minute vs hourly vs daily), data source (free vs licensed, vendor A vs vendor B), corporate-action handling (split-adjusted vs total-return vs dividend-reinvested), survivorship handling (current-constituents vs historical-constituents). Each is a degree of freedom. Category 8: implicit constraints from researcher behavior. The strategy was not the first idea tried. Earlier ideas were abandoned and a new one tried. The researcher's selection across abandoned strategies is a degree of freedom that does not appear in the final report. This is the hardest category to count and is where most retail and small-shop research underestimates by an order of magnitude. ## Counting effective degrees of freedom The total effective DoF is approximately the product of the cardinalities across all eight categories (or the sum if you prefer log-space arithmetic). The relevant statistic is k\_eff, the effective number of independent configurations the researcher could have reported. $$ k\_{\\text{eff}} = \\prod\_{i=1}^{C} n\_i, \\qquad \\text{Bias}\_{\\text{IS}} \\approx \\sigma\_{\\text{IS}} \\cdot \\sqrt{2 \\log k\_{\\text{eff}}} $$ For the breakout-strategy example above: k\_eff approximately 144000, sqrt(2 log 144000) approximately 4.9\. The bias on the IS Sharpe estimate is approximately 4.9 standard errors. With N=200 trades, sigma\_IS approximately 1/sqrt(200) approximately 0.07, the bias is approximately 0.34\. The reported IS Sharpe of 1.8 implies a bias-corrected expected OOS Sharpe of approximately 1.5; the actual OOS of 0.2 suggests the bias correction is even larger because the configuration count was probably higher than 144000 once Category 8 (researcher selection across abandoned strategies) is included. The full counting principle. List every choice that was made, including the choice "not to" do something (e.g., "we did not include FOMC-day filtering" is a choice that could have gone the other way). For each, count the number of alternatives the researcher considered. Multiply. The product is k\_eff. The bias-corrected IS Sharpe is the reported IS Sharpe minus the search-width bias. Report both numbers, not the reported number alone. ## The DoF count vs the headline number Three structural reasons. Reason 1: the IS Sharpe inflates with k\_eff. The reported IS Sharpe is the maximum over k\_eff noisy realizations of the true Sharpe. The expected gap between the reported maximum and the true Sharpe scales with sqrt(2 log k\_eff). For k\_eff = 100, the gap is approximately 3 standard errors. For k\_eff = 10000, approximately 4 standard errors. For k\_eff = 1000000, approximately 5 standard errors. The IS Sharpe is meaningless without k\_eff in the denominator. Reason 2: the OOS test loses statistical power as k\_eff grows. The OOS test is meant to be an independent validation, but the test was implicitly part of the researcher's selection process if the researcher iterated until the OOS validated. The article ["Why OOS Failure Is Often a Stationarity Failure"](https://aligrithm.com/why-oos-failure-is-often-a-stationarity-failure/) framed one cause of OOS failure; another cause is the OOS being implicitly part of the IS through the researcher's iteration process. Reason 3: communication standards. A research team's discipline can be assessed by whether the team reports k\_eff alongside the IS Sharpe. A team that reports "Sharpe 1.8 from 3 parameters" is either deceptive or has not counted across all categories. A team that reports "Sharpe 1.8 from approximately 10000 effective configurations, bias-corrected estimate 1.4" is forthright. The latter is rare; the former is common. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Practical counting protocol Five steps to a defensible count. Step 1: list every numeric parameter and the grid considered. Even values "tried briefly and rejected" count. If the researcher's brain tried entry lookbacks 5, 10, 15, 20, 25, 30, 40, 60 before settling on 20, the count for that parameter is 8, not 1. Step 2: list every structural choice and the candidates. "We chose breakout over RSI; we considered Donchian vs ATR-buffered." Each enumerated alternative is a candidate. Step 3: list every filter and gating choice (Category 3). Each filter applied or not applied, with its parameter grid. Step 4: list every implicit choice from Categories 5-7 (markets, costs, data). Conservative count: assume at least 2-3 alternatives were considered for each implicit choice. Step 5: estimate Category 8 (selection across abandoned strategies). If the researcher tried 10 different strategies before settling on the final one, the effective k is multiplied by 10 because the final strategy is the maximum over the 10 abandoned attempts as well as the within-strategy parameter search. Multiply. Report k\_eff. Apply the bias correction. ## Anti-patterns Five mistakes specific to DoF counting. Anti-pattern 1: counting only the explicit numeric parameters. The breakout strategy has approximately 10000-100000 effective configurations once structural, filtering, sizing, market, and cost choices are included, far above the advertised "3 parameters". Anti-pattern 2: ignoring the abandoned-strategy selection. The researcher tried 10 strategies and reports the best one. The implicit search width is multiplied by 10\. The headline Sharpe of the best strategy is biased upward by the corresponding amount. Anti-pattern 3: claiming "we did not optimize" when the parameters came from a structural prior. If the parameters are from the academic literature and not adjusted by the researcher, the within-strategy DoF is zero. But the strategy choice itself (which paper to follow, which parameters from which paper, which market to apply to) is still a DoF that needs to be counted. Anti-pattern 4: treating k\_eff and bias correction as theoretical concerns. The bias is real, predictable, and quantifiable. The bias-corrected IS Sharpe is the right number to use for OOS expectation. Skipping the bias correction is the same as skipping the OOS test. Anti-pattern 5: reporting per-parameter sensitivity instead of total k\_eff. Showing that "the strategy is robust to entry lookback in the range 15-25" is necessary but not sufficient. The total k\_eff includes the choice of "entry-lookback as the parameter to vary" rather than other parameters. The robustness to one parameter does not imply robustness across all DoF. ## Decision matrix | DoF category | Typical count per strategy | Reporting requirement | | -------------------------------- | --------------------------------- | ------------------------------------------------- | | Numeric parameters | 3-15 each, total 50-1000 | Report grid considered, not single value | | Structural choices | 2-10 each, total 5-50 | Enumerate candidates considered | | Filters and gating | 0-10 filters, each 2-5 candidates | List all filters, including absences | | Sizing and risk-management | 3-10 candidates total | Enumerate alternatives considered | | Market and time-period selection | 2-20 candidates | Justify selection and the alternatives considered | | Cost and execution assumptions | 2-5 candidates | Report sensitivity to cost assumption | | Data choices | 2-10 candidates | Report sensitivity to data source | | Abandoned-strategy selection | 1 to 50+ | Report number of strategies tried before this one | | Total k\_eff | 10000 to 10^9 typical | Report bias-corrected IS Sharpe alongside raw | The matrix is operational. The pattern: count across all categories, report k\_eff, apply bias correction. ## Visualizing degrees of freedom ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/05/article_69-visual-0.png) ## KEY POINTS - Degrees of freedom in a trading system extend beyond the explicit numeric parameters. Eight categories: numeric parameters, structural choices, filters and gating, position sizing, market and time-period selection, cost and execution assumptions, data choices, abandoned-strategy selection. - The total effective DoF count k\_eff is approximately the product of the cardinalities across all categories. A strategy advertised as "three parameters" typically has k\_eff between 10000 and 10 million once all categories are counted in full. - The IS Sharpe is biased upward by approximately sigma\_IS times sqrt(2 log k\_eff). For k\_eff = 1.4 million and sigma\_IS = 0.07, the bias is approximately 0.37 Sharpe units. The bias-corrected IS Sharpe is the reported IS Sharpe minus this amount. - The bias-corrected IS Sharpe is the right number for OOS expectation. Skipping the bias correction is methodologically equivalent to skipping the OOS test. Reporting only the raw IS Sharpe is misleading. - A typical breakdown for a "simple" strategy: 50-1000 numeric configurations, 5-50 structural alternatives, 0-10 filters with their candidates, 3-10 sizing schemes, 2-20 market choices, 2-5 cost assumptions, 2-10 data choices, 1-50 abandoned prior strategies. Multiplied: 10000 to 10^9 effective configurations. - The hardest category to count is abandoned-strategy selection. The researcher tried 10 strategies before settling on the reported one. The reported strategy is the maximum over the 10 abandoned attempts as well as the within-strategy search; the implicit search width includes the 10x. - Anti-pattern: counting only the visible numeric parameters and reporting the strategy as "three parameters". The strategy is the visible parameters multiplied by the structural, filtering, sizing, market, cost, data, and abandoned-strategy choices. - Anti-pattern: ignoring the abandoned-strategy DoF. The researcher's brain searched across rejected strategies; the search width is multiplied accordingly. - Anti-pattern: claiming "we did not optimize" when the parameters came from a structural prior. The structural prior eliminates within-strategy DoF, but the choice of which structural prior to use is itself a DoF. - Anti-pattern: reporting per-parameter sensitivity as proof of low DoF. Robustness to one parameter is necessary but not sufficient; total k\_eff includes the choice of which parameter to test. - The reporting standard: list every choice with the number of alternatives considered, multiply to get k\_eff, apply the bias correction. Report both the raw IS Sharpe and the bias-corrected IS Sharpe. The latter is the OOS expectation. - The current article gives the counting framework. The next article in the publication (["Why More Parameters Make a Strategy Easier to Sell and Easier to Break"](https://aligrithm.com/why-more-parameters-make-a-strategy-easier-to-sell-and-easier-to-break/)) covers the marketing-vs-engineering trade-off that follows from the count. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Testing and Tuning Market Trading Systems - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Testing+and+Tuning+Market+Trading+Systems+Timothy+Masters&ref=aligrithm.com) - [Data Mining Algorithms in C++ - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Data+Mining+Algorithms+in+C%2B%2B+Timothy+Masters&ref=aligrithm.com) ### 3.15 Optimization Comes After Testing, Not Before URL: https://aligrithm.com/optimization-comes-after-testing-not-before/ Last updated: 2026-07-27T13:24:09.000Z A research team has an idea: cross-sectional momentum on US large-caps with vol-conditioned position sizing. The team has two choices for how to spend the next three months. Path A: implement the strategy with structural-prior parameters (12-month lookback, monthly rebalance, equal-vol sizing, 100 stocks long top decile, 100 short bottom decile), run the backtest from 1995 to 2024 with the same parameters, and look at the result. The implementation takes one week. The remaining eleven weeks go into diagnostics, regime stratification, robustness checks, and post-mortem analysis. Path B: implement the strategy with a parameter-search framework, optimize across lookbacks (3, 6, 9, 12, 18, 24 months), rebalance frequencies (weekly, monthly, quarterly), decile widths (top/bottom 5%, 10%, 20%), vol-target levels (8%, 10%, 12%, 15%), and shorting policies (long-only, market-neutral, dollar-neutral, beta-neutral). Path B takes three months and produces a parameter set with IS Sharpe 1.4\. Path A produces an IS Sharpe of 0.65 at structural-prior parameters. Most teams choose Path B. The path produces a more impressive number, looks more "rigorous" because it has a parameter-search framework, and fits the format expected by published research. The teams that choose Path A consistently outperform the teams that choose Path B in OOS deployment, by a margin that scales with the search width Path B used. The article ["The Difference Between Robustness and Optimization"](https://aligrithm.com/the-difference-between-robustness-and-optimization/) gave the conceptual reason: optimization climbs to the noisy peak. This article gives the procedural reason: optimization performed at the wrong stage of the research process compounds the problem with a worse one. Path A spends 90% of the time on diagnostics that reveal whether the strategy has any signal. Path B spends 90% of the time on tuning a strategy that may have no signal at all, and produces an IS number that does not let the team distinguish signal from search luck. The right sequencing is: prove the strategy has signal at structural-prior parameters first, then optimize. Most teams do the reverse: optimize first, then "validate" the optimized version. The reverse is overfitting dressed up as methodology. This article gives the procedural framework for getting the sequence right. ## The four-stage research process Each stage answers a different question. Each stage's output gates the next stage. Stage 1: hypothesis articulation. State the strategy thesis in plain language. What pattern is the strategy claiming exists? What mechanism explains why the pattern should persist? What market family does the pattern live on? What time horizon is relevant? The output is a written one-page hypothesis that is falsifiable. The article ["Why Benchmarks Matter in Rule Evaluation"](https://aligrithm.com/why-benchmarks-matter-in-rule-evaluation/) framed the testability requirement at the methodology level. Stage 2: minimal-parameter prototype. Implement the strategy at structural-prior parameters with no optimization. The parameters come from the academic literature, the underlying mechanism, or domain priors. Run the backtest with these parameters. The question this stage answers: "is there any signal at all?" If the prototype Sharpe is in the expected range (typically 0.3 to 0.8 for a well-known structural premium, higher for a novel insight), proceed. If the prototype Sharpe is near zero or negative, reconsider the hypothesis. No optimization yet. Stage 3: robustness exploration. Vary one parameter at a time, holding the others at the structural prior. Map the IS Sharpe as a function of each parameter individually. Look for plateaus where the Sharpe is approximately constant across a wide range. The output is a parameter-stability map (covered in detail in ["Parameter Stability Beats Best Parameter"](https://aligrithm.com/parameter-stability-beats-best-parameter/) later in this pillar). Choose the final parameter set from the center of the plateau region, not from the IS-maximum point. Stage 4: validation and optimization. With the parameters chosen from Stage 3, run the regime-stratified backtest, the cross-market check (within the family from ["Market Personality: Why Gold, FX, Crypto, and Equities Need Different Systems"](https://aligrithm.com/market-personality-why-gold-fx-crypto-and-equities-need-different-systems/)), the bootstrap robustness check, and the multi-period stability check. Only after all of these pass should any small final optimization be applied, and only within the plateau region identified in Stage 3. The wrong order. Most teams compress Stages 1 to 3 into a single combined search where the parameter optimization happens before the validation. The combined search produces an IS Sharpe that mixes the signal contribution with the search-width inflation, and the team cannot distinguish the two without doing the validation step on a separate slice. ## Structural prior, defined The parameters that come from outside the backtest. Three sources. Source 1: academic literature. The 12-month lookback for equity momentum is from Jegadeesh and Titman 1993 and replicated in dozens of papers since. The 5-day mean-reversion horizon is from numerous studies of short-horizon equity reversal. The 22-day window for vol estimation is roughly one trading month. These are not optimized in the current research; they come pre-vetted with a structural rationale. Use them and do not retune. Source 2: mechanism-based reasoning. If the strategy is a pairs trade, the lookback for cointegration estimation should match the half-life of the spread's mean reversion. If the spread half-life is 30 days, the cointegration lookback is 60 to 120 days (2-4x the half-life). The number comes from the mechanism, not from a search. Source 3: cross-asset analogs. If the strategy is novel and there is no academic literature, look at analog strategies on related markets. A strategy on BTC may borrow parameters from a similar strategy on equities (with appropriate scaling for volatility differences). The borrowed parameters are a prior, not a fit. The discipline. Once a structural prior is set, it is not subject to optimization until Stage 4\. The Stage 2 prototype runs at the structural prior. The Stage 3 robustness check varies parameters around the prior to find the plateau. The Stage 4 final selection picks from the plateau, not from the maximum. The optimization in Stage 4 is constrained, small, and only applied after the strategy has demonstrated signal at the prior. ## Reasons teams skip Stage 2 Three reasons, each diagnostic of a research-process problem. Reason 1: the prototype produces a low number. Sharpe 0.65 looks bad next to optimized Sharpe 1.4 in any presentation context (internal review, fund pitch, paper). The team optimizes to produce a better presentation number. The team is then evaluating an inflated version of the strategy, not the underlying signal. Reason 2: the team confuses optimization with rigor. Running a parameter-search framework looks more sophisticated than running one backtest with prior parameters. The team interprets sophistication as quality, when in research context the more rigorous approach is to test the prior first and only optimize if the prior shows signal. Reason 3: software defaults. Many backtesting platforms default to optimization workflows because they produce more visually appealing reports. The team accepts the default. The default is misaligned with research-process discipline. The fix. Make Stage 2 the explicit first deliverable. Before any team member opens an optimization framework, they must produce the Stage 2 backtest at structural priors, write a one-paragraph interpretation of the result, and decide whether to proceed. The friction is the discipline. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## The math of optimization-after-testing vs optimization-before-testing A simple comparison. Assume the strategy's true Sharpe at the structural prior is SR\_true. The Stage 2 prototype produces an unbiased estimate with standard error sigma\_IS / sqrt(N). The Stage 4 optimization within the plateau region searches over k\_local candidates, producing an estimate with bias sigma\_IS times sqrt(2 log k\_local). $$ \\widehat{\\text{SR}}\_{\\text{Stage 4}} = \\text{SR}\_{\\text{true}} + \\varepsilon\_{\\text{search}}, \\qquad \\mathbb{E}\[\\varepsilon\_{\\text{search}}\] \\approx \\sigma\_{\\text{IS}} \\cdot \\sqrt{2 \\log k\_{\\text{local}}} $$ For k\_local approximately 10 (a small final optimization within the plateau), the bias is approximately 2.1 sigma\_IS / sqrt(N). This is small compared to the true Sharpe. By contrast, optimization-before-testing searches over k\_global candidates spanning many regions of parameter space, with k\_global typically 100 to 10000. $$ \\mathbb{E}\[\\varepsilon\_{\\text{global}}\] \\approx \\sigma\_{\\text{IS}} \\cdot \\sqrt{2 \\log k\_{\\text{global}}}, \\quad k\_{\\text{global}} \\gg k\_{\\text{local}} $$ For k\_global = 1000, the bias is approximately 3.7 sigma\_IS / sqrt(N), which can swamp the true Sharpe. The Stage 4 optimization is statistically defensible because the plateau-region constraint limits k\_local; the optimization-before-testing approach is not defensible because k\_global is large and unconstrained. ## Anti-patterns Five mistakes specific to the optimization-sequence problem. Anti-pattern 1: presenting the optimized Sharpe as the strategy's performance. The optimized Sharpe is the upper end of the search-width distribution, not the unbiased estimate. The right number to present is the structural-prior Sharpe with the plateau-region confidence band; the optimized version is reported as a refinement, not as the headline. Anti-pattern 2: skipping Stage 2 because the prototype produces a low number. A low number at the structural prior is informative: the strategy may have a small or absent signal, and optimization will not create signal that is not there. The right response is to reconsider the hypothesis, not to optimize. Anti-pattern 3: combining the optimization and the validation into a single stage. The validation needs to happen on a slice of data not seen during the optimization. If the optimization touched all the available data, the validation is not independent. The article ["Why OOS Failure Is Often a Stationarity Failure"](https://aligrithm.com/why-oos-failure-is-often-a-stationarity-failure/) framed the IS-OOS-leakage problem; the present anti-pattern is one way the leakage is introduced. Anti-pattern 4: claiming the optimized parameters are "robust" because the team tried multiple values. Trying multiple values produces a search-width-inflated estimate; it does not produce a robust strategy. The robust strategy is the one chosen from the plateau in Stage 3, with the Stage 4 validation confirming the plateau survives OOS. Anti-pattern 5: re-running Stage 4 if the OOS underperforms. The OOS is the validation. If the OOS underperforms, the diagnosis is the diagnostic flow from ["Why OOS Failure Is Often a Stationarity Failure"](https://aligrithm.com/why-oos-failure-is-often-a-stationarity-failure/) and the action is the decommission policy from ["How to Detect When a Trading System Is Dying"](https://aligrithm.com/how-to-detect-when-a-trading-system-is-dying/). Re-running the optimization in response to OOS underperformance contaminates the validation slice and converts the optimization-after-testing approach into the optimization-before-testing failure mode. ## Decision matrix | Research stage | Right action | Wrong action | | ------------------- | --------------------------------------------------------- | -------------------------------------------- | | Stage 1: hypothesis | Write a one-page testable thesis with mechanism | Skip directly to backtest | | Stage 2: prototype | Backtest at structural-prior parameters, no optimization | Tune parameters to improve the number | | Stage 3: robustness | Map Sharpe vs each parameter individually, find plateau | Find peak via grid search | | Stage 4: validation | Regime-stratified, cross-market within family, bootstrap | Combined optimization-and-validation | | Stage 4 final tweak | Small optimization within Stage 3 plateau, k\_local <= 10 | Wide optimization across all parameter space | | OOS deployment | Trust the structural-prior + plateau result | Re-tune in response to live data | | OOS underperforms | Run diagnostic flow, decommission per policy | Re-optimize on extended IS | | Post-mortem | Update structural priors based on what failed | Skip post-mortem to ship next strategy | The matrix matches research stage to discipline. The pattern: parameter selection is a small final step, not the main research activity. Most of the research time goes to hypothesis articulation, prototype evaluation, and robustness checks. ## Visualizing the right sequence ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/05/article_68-visual-0.png) ## KEY POINTS - The right research process is four stages: hypothesis articulation, minimal-parameter prototype at structural priors, robustness exploration with parameter-stability mapping, validation with optional small final optimization within the plateau region. - Each stage answers a different question. Stage 2 answers "is there signal?". Stage 3 answers "is the signal robust to parameter perturbation?". Stage 4 answers "does the strategy survive cross-regime, cross-market, and bootstrap-robustness validation?". - The parameter optimization in Stage 4 is constrained to the plateau region from Stage 3, with k\_local on the order of 10 candidates. The optimization-induced bias is small and the resulting estimate is approximately unbiased. - The wrong order: combine Stages 2-4 into a single optimization that searches across thousands of parameter combinations and reports the maximum IS Sharpe. The bias from this approach scales with sqrt(2 log k\_global) and can swamp the true Sharpe. The IS estimate does not replicate OOS. - Structural priors come from three sources: academic literature, mechanism-based reasoning, cross-asset analogs. The priors are not subject to optimization until Stage 4. - Three reasons teams skip Stage 2: the prototype produces a low number that looks bad in presentation, the team confuses optimization complexity with research rigor, software defaults push the optimization-first workflow. - Anti-pattern: presenting the optimized Sharpe as the strategy's performance. The unbiased estimate is the structural-prior Sharpe with the plateau-region confidence band. The optimized version is a refinement, not a headline. - Anti-pattern: combining optimization and validation into one stage. The validation needs an independent slice; combined search uses all the data. - Anti-pattern: claiming "robustness" from a parameter search. Searching is not robustness. Robustness comes from the plateau-region selection and the Stage 4 cross-regime cross-market checks. - Anti-pattern: re-running optimization in response to OOS underperformance. Each re-optimization contaminates the validation. The right response is the diagnostic flow and the decommission policy. - Most retail and small-shop research compresses the four stages into one optimization run because optimization is the most software-supported part of the workflow. The discipline of separating the stages is the upstream defense against most of the OOS failures. - The current article gives the procedural sequence. The next article in the publication (["Degrees of Freedom in Trading Systems"](https://aligrithm.com/degrees-of-freedom-in-trading-systems/)) covers the parameter-counting framework that makes the search-width bias quantitative. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Testing and Tuning Market Trading Systems - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Testing+and+Tuning+Market+Trading+Systems+Timothy+Masters&ref=aligrithm.com) - [Data Mining Algorithms in C++ - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Data+Mining+Algorithms+in+C%2B%2B+Timothy+Masters&ref=aligrithm.com) ### 3.14 Market Personality: Why Gold, FX, Crypto, and Equities Need Different Systems URL: https://aligrithm.com/market-personality-why-gold-fx-crypto-and-equities-need-different-systems/ Last updated: 2026-07-27T13:24:08.000Z Compute the efficiency ratio (the ratio of net price change to total absolute path length over a fixed window) on five markets across the same calendar window of 2015-2024 with the same 22-day lookback. The values are diagnostic. SPX cash index: efficiency ratio averages approximately 0.30, with a long-run drift bias and persistent above-average ER during sustained bull regimes. EURUSD: efficiency ratio averages approximately 0.18, with no long-run drift and ER hovering near 0.20 across most regimes. Gold (XAU): efficiency ratio averages approximately 0.22, with two distinct regime modes (a low-ER chop mode below 0.15 and a high-ER trend mode above 0.30 during inflation hedging or risk-off episodes). Natural gas: efficiency ratio averages approximately 0.12, with high noise and seasonal-storage-driven shifts. BTC: efficiency ratio averages approximately 0.28, with extreme regime variation between 0.05 (sideways consolidation lasting months) and 0.50 (sustained breakout regimes lasting weeks). Five different markets, five different efficiency-ratio profiles. The same trend-following strategy that earns Sharpe 0.6 on SPX (where ER is reliably 0.30 and trend persistence is the dominant feature) will earn approximately Sharpe 0.0 to 0.2 on natural gas (where ER is 0.12 and noise dominates) regardless of how the parameters are tuned. The strategy is not failing; the strategy is being misapplied to a market whose personality does not provide trend signal at the relevant horizon. The article "Why Works on All Markets Is Usually a Red Flag" gave the skeptical view of cross-market claims. This article gives the constructive complement: each market class has a personality determined by its participants, microstructure, and macro driver, and the strategy framework needs to match the personality. The mistake that most retail and emerging-shop research makes is to treat markets as interchangeable backtesting venues. Run the same strategy on SPX, then on EURUSD, then on BTC, then on gold, and report the per-market results as if the strategy was tested across an unbiased sample of conditions. The conditions are very different. The strategy is doing different things on each market. The aggregated result is meaningless without the per-market context. The right discipline is to characterize each market's personality first, choose the strategy framework that matches the personality second, and only then begin parameter selection. ## Personality, defined operationally Five measurable dimensions, each diagnostic. Dimension 1: efficiency ratio and noise level. The efficiency ratio measures how much of the total path length translates into net directional change. ER near 1 is pure trend. ER near 0 is pure chop with no net direction. Most markets sit between 0.1 and 0.4\. Trend-following strategies need ER above approximately 0.25 to be profitable after costs. Mean-reversion strategies thrive at ER below 0.20. $$ \\text{ER}\_W = \\frac{|P\_t - P\_{t-W}|}{\\sum\_{i=1}^{W} |P\_{t-i+1} - P\_{t-i}|} $$ Dimension 2: autocorrelation structure. The first-lag autocorrelation of returns at the strategy's horizon. Positive autocorrelation favors trend strategies. Negative autocorrelation favors mean reversion. Near-zero autocorrelation argues for both being approximately useless. The article ["Why Volatility Is More Non-Stationary Than Trend"](https://aligrithm.com/why-volatility-is-more-non-stationary-than-trend/) framed the daily-horizon autocorrelation; the personality test extends to the strategy's actual horizon. Dimension 3: return distribution shape. Skewness (right or left) and kurtosis (fat tails or thin tails). SPX has slightly negative skew (tail in losses) and high kurtosis. EURUSD has approximately zero skew and moderate kurtosis. BTC has positive skew (long-tail upside) and very high kurtosis. Gold has near-zero skew and elevated kurtosis. Natural gas has positive skew and extreme kurtosis. The shape determines which risk metrics are meaningful and which strategy P&L profiles are sustainable. Dimension 4: volatility regime structure. The unconditional vol level, the vol-of-vol, the typical regime durations, the implied-vs-realized vol relationship. Covered in detail in ["Volatility Regimes and Strategy Survival"](https://aligrithm.com/volatility-regimes-and-strategy-survival/) and ["Why Volatility Is More Non-Stationary Than Trend"](https://aligrithm.com/why-volatility-is-more-non-stationary-than-trend/); for cross-market comparison the relevant numbers are the unconditional vol level and the vol-of-vol relative to that level. Dimension 5: macro driver and participant mix. SPX is dominated by institutional flow and ETF rebalancing, with a structural long-bias and a strong relationship to GDP growth and earnings. EURUSD is dominated by central-bank flows and trade balances. Gold is driven by real interest rates and macro hedge demand. BTC is driven by retail flows, exchange events, and a network effect. Natural gas is driven by weather, storage, and seasonal demand. Each driver implies a different signal hierarchy. ## Strategy frameworks by market personality Six market families with their structural strategy fits. Family 1: large-cap equity indices (SPX, NDX, RTY, NIKKEI, EURO STOXX). High ER, structural long-bias, deep liquidity, low transaction costs. Suited for: long-only buy-and-hold (the equity premium), trend-following on multi-week horizons, mean reversion on short (1-5 day) horizons, factor strategies (value, momentum, quality, low-vol). Hostile to: pure carry, pure microstructure (already arbitraged out), strategies that depend on heavy directional shorts. Family 2: G10 FX crosses (EURUSD, GBPUSD, USDJPY, AUDUSD). Low ER, near-zero unconditional drift, deep liquidity, very low costs (interbank). Suited for: carry trade (interest-rate differential), trend-following on multi-month horizons, mean reversion at intraday horizons, value-based strategies (PPP-deviation reversion). Hostile to: equity-style factor frameworks, microstructure strategies that assume retail-flow patterns. Family 3: precious metals (gold, silver, platinum). Mixed ER (regime-dependent), near-zero drift, moderate liquidity, moderate costs. Suited for: macro-hedge positioning (real-rates, inflation), trend-following on medium horizons (gold has multi-year trends), volatility-conditioned position sizing. Hostile to: short-horizon mean reversion (the regime breaks come too suddenly), high-frequency strategies. Family 4: industrial and energy commodities (copper, crude oil, natural gas). Low to mixed ER, no drift, seasonal patterns, wide spreads, storage and convenience-yield structure. Suited for: spread trades (calendar spreads, crack spreads), seasonal patterns, supply-demand-fundamental positioning. Hostile to: trend-following at short horizons (the noise dominates), naive long-bias (no positive expected return on the futures, only basis), strategies that ignore storage and roll structure. Family 5: government bonds (US Treasuries, Bunds, JGBs). Mostly low ER, dominant central-bank flows, very deep liquidity, very low costs. Suited for: yield-curve trades, macro-conditional positioning, term-premium-based strategies. Hostile to: technical strategies that ignore the central-bank reaction function, equity-style trend-following, high-frequency strategies that fight the market-maker hierarchy. Family 6: cryptocurrencies (BTC, ETH, major alts). High ER on regime-active periods, very high vol, high costs, fragmented liquidity, retail-dominated flows. Suited for: trend-following on multi-week horizons during active regimes, momentum on long horizons, vol-conditional position sizing. Hostile to: classical mean-reversion frameworks (the regime breaks are too violent), pair trades that assume stable cointegration, strategies that ignore exchange-specific risk and venue events. The point of the families: a strategy framework is not market-agnostic. Each family demands a different framework and the wrong framework on the wrong family produces near-zero returns regardless of parameter tuning. ## Diagnosing personality before strategy selection A four-step protocol. Step 1: compute the dimensions. ER, autocorrelation at strategy horizon, return distribution skewness and kurtosis, vol regime distribution, macro driver and participant mix. The first three are quantitative and produced by short scripts; the last two require domain knowledge. Step 2: classify the market into a family. The article ["Volatility Regimes and Strategy Survival"](https://aligrithm.com/volatility-regimes-and-strategy-survival/) gave the operational regime gating; the family classification is one level above and is determined by the structural characteristics rather than the current regime. Step 3: list the strategy frameworks compatible with the family. From the family-fit catalog above. Reject any framework that the family is hostile to before any backtesting. Step 4: prototype the strategy in the framework with structural-prior parameters. The first backtest is at the structural prior, no optimization. The result tells you whether the framework matches the personality. If the prototype is profitable at the structural prior, optimization (with the discipline from ["The Difference Between Robustness and Optimization"](https://aligrithm.com/the-difference-between-robustness-and-optimization/)) can improve it. If the prototype is near-zero or negative at the structural prior, optimization will not save it and the framework is wrong for the market. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Cases of personality drift Market personalities are not perfectly stationary. The article ["Slow Wandering: The Most Dangerous Type of Market Change"](https://aligrithm.com/slow-wandering-the-most-dangerous-type-of-market-change/) framed the slow-drift problem; the personality drift is one specific case. Three structural personality changes that have been visible in the data. Change 1: equity microstructure transition. Pre-decimalization SPX (before 2001) had wider spreads, higher mean-reversion at short horizons, more visible market-making structure. Post-2010 SPX has tight spreads, weaker short-horizon mean-reversion, dominant ETF arbitrage. A strategy tested on 1995-2000 SPX that does not test on 2010-2024 is testing on a different SPX personality. The article "Microstructure change" (mechanism 3 in ["Why Systems Work Until They Don't"](https://aligrithm.com/why-systems-work-until-they-dont/)) is the operational manifestation. Change 2: crypto market maturation. Pre-2017 BTC was a retail-heavy market with extreme regime breaks and low liquidity. Post-2020 BTC has institutional flows, futures and ETF infrastructure, and tighter spreads. The personality has shifted toward the equity-index family on multi-day horizons, while keeping crypto-specific traits at intraday horizons. Strategies tuned to 2014-2017 BTC will not generalize to 2024 BTC. Change 3: FX response to monetary policy regimes. Pre-2008, G10 FX carry trades were the dominant flow. Post-QE the carry-rate dispersion compressed and the trade decayed (covered in ["Slow Wandering: The Most Dangerous Type of Market Change"](https://aligrithm.com/slow-wandering-the-most-dangerous-type-of-market-change/)). Post-2022 with rate normalization, the carry trade has partly returned. The FX family's personality is regime-conditional in a way that requires monitoring rather than one-time characterization. The protocol response: re-characterize the personality every few years and verify the strategy framework still matches. A strategy that was correctly framed in 2015 may be misapplied in 2025 if the personality drifted. ## Anti-patterns Five mistakes specific to market-personality misapplication. Anti-pattern 1: applying the equity-index strategy framework to commodities. Commodities have no long-run drift and the equity-index trend strategy is implicitly long-biased. The strategy on commodities earns no premium and produces noise. Anti-pattern 2: applying the FX carry framework to crypto. Crypto has no carry in the FX sense (the rate-differential premise is absent) and the regime breaks are too violent for a carry-style framework that assumes slow drift. The framework is misapplied. Anti-pattern 3: applying the same strategy framework to a market whose personality changed structurally. Pre-2010 SPX is not the same market as post-2020 SPX, and a backtest spanning the structural change is testing two different personalities. Stratify by personality era as well as by regime. Anti-pattern 4: ignoring transaction costs that vary by family. SPX costs are 1-2 basis points round-trip. Crypto costs are 50-200 basis points. The same gross signal that produces Sharpe 1.0 on SPX produces Sharpe 0.2 on crypto after costs, even with identical price dynamics. Anti-pattern 5: assuming personality from headline stats only. SPX 14% annualized vol and EURUSD 8% annualized vol look like they differ only in scale. The personalities also differ in microstructure, drift, autocorrelation, and participant mix. Headline vol is one dimension; the personality is many. ## Decision matrix | Market family | Recommended frameworks | Hostile frameworks | | ------------------------------------------- | ------------------------------------------------------------------------------------------- | ---------------------------------------------------- | | Large-cap equity indices | Long-only equity premium, multi-week trend, short-horizon mean reversion, factor strategies | Pure carry, microstructure arbitrage | | G10 FX | Carry, multi-month trend, intraday mean reversion, PPP-value | Equity factors, retail-flow microstructure | | Precious metals | Macro-hedge, medium-horizon trend, vol-conditioned sizing | Short-horizon mean reversion, HFT | | Industrial / energy commodities | Calendar spreads, seasonal patterns, supply-demand fundamental | Naive long-bias, short-horizon trend ignoring noise | | Government bonds | Yield-curve trades, macro-conditional positioning, term premium | Technical strategies ignoring central-bank reaction | | Crypto majors | Multi-week trend in active regimes, momentum, vol-conditioned sizing | Classical mean reversion, stable-cointegration pairs | | Single-stock equities | Cross-sectional factors, event-driven, statistical arbitrage | Index-style trend without sector control | | Volatility products (VIX futures, vol ETFs) | Term-structure trades, vol carry with explicit gating | Naive long or short positioning ignoring structure | The matrix matches market family to compatible frameworks. The pattern: characterize the family first, select the framework second, optimize the parameters last. ## Visualizing market personality ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/05/article_67-visual-0.png) ## KEY POINTS - Each market class has a personality determined by efficiency ratio, autocorrelation structure, return distribution shape, volatility regime, and macro driver / participant mix. The personality determines which strategy frameworks can earn an edge on that market. - Five measurable dimensions: efficiency ratio (net change over total path length), autocorrelation at strategy horizon, return skewness and kurtosis, volatility regime distribution, macro driver and participant mix. - Six market families with structural strategy fits: large-cap equity indices, G10 FX, precious metals, industrial/energy commodities, government bonds, cryptocurrencies. Each family demands different frameworks and is hostile to others. - Equity indices suit long-bias trend, short-horizon mean reversion, and factor strategies. FX suits carry, multi-month trend, intraday mean reversion. Precious metals suit macro-hedge positioning. Commodities suit spreads and seasonal patterns. Bonds suit yield-curve and macro-conditional. Crypto suits multi-week trend and momentum with vol gating. - A four-step diagnostic protocol: compute the personality dimensions, classify into a family, list the compatible frameworks, prototype at structural-prior parameters with no optimization. The prototype tells you whether the framework matches. - Optimization at structural-prior parameters that produces near-zero or negative returns is a personality mismatch. Optimization will not save it; the framework is wrong for the market. - Personalities are not stationary. Three documented structural personality changes: equity microstructure transition (pre-2001 vs post-2010), crypto maturation (pre-2017 vs post-2020), FX carry decay across QE eras. Re-characterize every few years. - Anti-pattern: applying the equity-index strategy framework to commodities (no long-run drift). Anti-pattern: applying the FX carry framework to crypto (no carry in the FX sense, regime breaks too violent). Anti-pattern: same framework across pre- and post-personality-change eras. Anti-pattern: ignoring transaction-cost differences across families. - The right discipline: characterize first, select framework second, optimize parameters last. The parameters are the smallest decision; the framework choice is the larger one; the personality understanding is the largest. - The current article gives the constructive view. Combined with the prior article ("Why Works on All Markets Is Usually a Red Flag"), the operational rule: claim cross-market robustness only within a family, and check the personality before claiming any cross-family generalization. - The next article in the publication (["Optimization Comes After Testing, Not Before"](https://aligrithm.com/optimization-comes-after-testing-not-before/)) covers the procedural sequencing that follows from this framework: test at structural-prior parameters first, optimize last. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Testing and Tuning Market Trading Systems - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Testing+and+Tuning+Market+Trading+Systems+Timothy+Masters&ref=aligrithm.com) - [Data Mining Algorithms in C++ - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Data+Mining+Algorithms+in+C%2B%2B+Timothy+Masters&ref=aligrithm.com) ### 3.13 Why "Works on All Markets" Is Usually a Red Flag URL: https://aligrithm.com/why-works-on-all-markets-is-usually-a-red-flag/ Last updated: 2026-07-27T13:24:08.000Z A strategy is pitched in a published research paper. The headline claim: "the system delivers Sharpe ratios above 1.2 across 38 markets including SPX, NDX, RTY, NIKKEI, EUR-STOXX, EURUSD, GBPUSD, USDJPY, gold, copper, crude oil, natural gas, soybeans, corn, wheat, BTC, ETH, US 10-year Treasuries, German Bunds, and others." The reader's first instinct, which the article's tone is designed to encourage, is admiration: "wow, a strategy that works everywhere must capture something universal, like trend or mean reversion or risk premium." The reader's better instinct, which experienced researchers have learned the hard way, is skepticism: a strategy that delivers Sharpe 1.2 on natural gas (one of the most volatile, cost-heavy, regime-shifting markets) and on US 10-year Treasuries (one of the most heavily intermediated, central-bank-dominated markets) is doing something fishy in either the backtest, the optimization, or the framing. Markets are not interchangeable. SPX has a strong long-run drift, low overnight gaps, deep order books, and tight bid-ask spreads. Natural gas has no long-run drift, large overnight gaps, thin order books, and wide spreads with seasonal storage-driven structural shifts. EURUSD has a near-zero unconditional drift, deep liquidity, microstructure dominated by central-bank flows. BTC has high vol, fragmented liquidity, regime breaks driven by retail flows and exchange events. A strategy that "works" on all of these probably has been tuned per-market with different parameters, evaluated with different cost assumptions, or filtered to the regimes where each market happens to comply with the strategy's premise. The article ["The Difference Between Robustness and Optimization"](https://aligrithm.com/the-difference-between-robustness-and-optimization/) framed cross-market robustness as one tool among several. This article gives the contrarian view: cross-market claims, taken at face value, are more often a sign of overfitting or misframing than a sign of a universal edge. ## "Works on all markets", in practice Five common reframings of the universal claim, each less impressive than the headline. Reframing 1: works on all markets after per-market optimization. The strategy template (e.g., "buy when 14-day RSI is below 30, exit at 50") is applied to 38 markets, but the entry threshold, exit threshold, lookback period, vol target, and stop-loss are optimized separately for each market. The "universal" claim hides 38 separate optimizations, each with its own degrees of freedom and its own search-width bias. The article "Why More Parameters Make a Strategy Easier to Sell and Easier to Break" frames the per-strategy degrees-of-freedom problem; multiplied by 38 it becomes a much larger problem. Reframing 2: works on the markets in our cherry-picked sample. The 38 markets reported are the ones where the strategy worked. The other markets where the strategy failed (probably 60 to 100 of them) were dropped from the report. The selection bias inflates the apparent universality. Reframing 3: works after applying market-specific filters. The natural-gas instance excludes summer storage seasons. The Treasury instance excludes Fed announcement days. The SPX instance excludes the first and last 30 minutes. Each filter is reasonable in isolation; cumulatively they encode a substantial market-specific specification that contradicts the "universal" framing. Reframing 4: works at universal-direction-bias scale. The strategy is approximately long-biased and earns the market-wide long-run drift on most equity-class markets. The "universal" Sharpe is the universal long-bias premium plus residual noise. Strip out the long-bias contribution and the strategy has near-zero Sharpe on its own merits. This is the most common pattern in academic equity-strategy papers. Reframing 5: works after silencing the worst markets through "outlier filtering". Two or three markets in the universe produce extreme losses (e.g., natural gas during the 2021 Texas freeze, BTC during the 2022 LUNA collapse). The headline number is computed after Winsorizing or removing these "outlier" events. The Winsorization is presented as a methodology choice; in practice it removes precisely the events that would have killed the strategy's claim of universal applicability. ## Real edges are market-specific Three structural reasons. Reason 1: each market has its own microstructure. Tick size, bid-ask spread, exchange fee structure, liquidity profile, market-maker concentration, dark pool fraction. A microstructure-sensitive strategy (mean reversion at intraday horizons, market making, latency arbitrage) earns different gross returns on different microstructures and pays different transaction costs. A strategy that is parameter-tuned to SPX's microstructure does not naively transfer to BTC's microstructure. Reason 2: each market has its own participant mix. SPX is dominated by institutional flow, retail through ETFs, and HFT. Natural-gas futures are dominated by physical hedgers and producers. Gold has central-bank reserves, jewelry demand, and macro-hedge flows. EURUSD has trade flows, central-bank flows, and macro speculation. The participant mix determines what kinds of patterns are economically sustainable. Trend follows persist on FX because central banks intervene slowly. Mean reversion persists on equities because rebalancing flows. Carry persists on bonds because pension-funding constraints. The same strategy framework picks up different subsets of these patterns on different markets. Reason 3: each market has its own macro driver. Equities respond to growth and earnings. FX responds to interest-rate differentials and balance-of-payments. Bonds respond to inflation and central-bank policy. Commodities respond to supply, weather, and storage. The strategies that exploit one macro driver do not automatically exploit the others. A strategy "premised on supply-demand-imbalance trends" works on physical commodities and is misapplied on financial assets. The exception. A genuine universal pattern occasionally exists: long-only buy-and-hold of a broad equity index harvests the equity risk premium on every developed-market equity universe. The trend-following premium harvested by CTAs has been positive across most futures markets across decades. Both exceptions are documented, replicated, and have low Sharpes (0.4 to 0.7) that are consistent with their structural origin. A "universal" strategy that claims Sharpe 1.5+ across 38 markets makes a different and much stronger claim than a refinement of the equity-premium or trend-following premium, one that does not survive the standard skepticism. ## The cross-market test, done correctly The right cross-market protocol corrects for the failure modes above. Step 1: select a market family with a clear common structural premise. Equity indices with similar microstructure (SPX, NDX, RTY) are one family. G10 FX crosses are another. Liquid commodity futures are a third. Crypto majors are a fourth. The within-family test is informative because the structural premise is similar; the across-family test is mostly a red flag for over-claiming. Step 2: use the same parameter values across markets in the family. No per-market optimization. The parameter values come from a structural prior, the article ["The Difference Between Robustness and Optimization"](https://aligrithm.com/the-difference-between-robustness-and-optimization/) gave the selection rule. If the strategy needs different parameters for SPX and NDX, it is tuned to SPX-specific noise rather than to a universal SPX-NDX-RTY pattern. Step 3: apply realistic per-market transaction costs. SPX costs and BTC costs differ by an order of magnitude. The same strategy template that delivers Sharpe 1.0 on SPX after costs may deliver Sharpe 0.2 on BTC after costs, even with the same gross signal strength. Report after-cost results per market. Step 4: report the full matrix, not the headline aggregate. For each market in the family, report Sharpe, max drawdown, profit factor, and number of trades. The strategy is deployment-ready on a market when its row in the matrix is positive; it is not ready on a market where its row is near-zero or negative, regardless of the aggregate. Step 5: account for the cross-market correlation. The portfolio Sharpe of 38 strategies (one per market) is approximately sqrt(N\_effective) times the average per-market Sharpe, where N\_effective accounts for correlations between markets. The naive average overstates the result. SPX and NDX have rho approximately 0.95 so they count as approximately one independent strategy, not two. The naive aggregation overstates the diversification benefit. ## The math of the multiple-comparison problem A simple model. Suppose the researcher tests the strategy on K markets and reports the markets where the IS Sharpe exceeds threshold T. Under the null hypothesis (the strategy has zero true edge on any market), the probability that at least m of the K reported Sharpes exceed T by chance scales with K and with the per-market false-positive rate. $$ \\Pr(\\text{at least } m \\text{ false positives}) = 1 - \\sum\_{j=0}^{m-1} \\binom{K}{j} p^j (1 - p)^{K - j} $$ For K = 100 markets tested and per-market false-positive rate p = 0.05 (the conventional 5% level), the probability of finding at least 5 false positives is approximately 56%. The probability of finding at least 10 is approximately 1.5%. Reporting "the strategy works on 10 markets at p < 0.05" without correction for the 100-market search is misleading; the correct framing is "we cannot reject the null after multiple-comparison correction". The correction. Apply Bonferroni or Benjamini-Hochberg to the per-market p-values. Bonferroni is conservative (multiply each p-value by K). Benjamini-Hochberg is less conservative (controls the false-discovery rate). Either is correct; reporting raw per-market p-values without any correction is not. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Anti-patterns Five mistakes that show up in cross-market claims. Anti-pattern 1: "the strategy is robust because it works on 38 markets". Robust means insensitive to perturbation. Cross-market is one perturbation axis, but with per-market optimization the apparent robustness is an illusion. Run the constrained-parameters version of the cross-market test, not the per-market-optimized version. Anti-pattern 2: averaging Sharpes across uncorrelated and correlated markets equally. The portfolio Sharpe scales with sqrt(N\_effective), not sqrt(N). Equity indices in the same region carry pairwise correlations above 0.8 and contribute almost no diversification. The naive average inflates the apparent benefit. Anti-pattern 3: reporting only the markets where the strategy works. Cross-market validation requires reporting the full matrix including the markets where the strategy fails. A strategy that works on 10 of 38 markets is informative; a strategy reported on 10 markets without the other 28 is selection bias. Anti-pattern 4: applying the strategy to markets where the structural premise does not hold. A momentum strategy applied to mean-reverting commodity-spread markets is misapplied; the negative results on the misapplied markets are diagnostic rather than universal failure. The right cross-market test is within a market family with a shared structural premise. Anti-pattern 5: claiming universality as a virtue. The claim "this strategy works on all markets" carries less operational weight than "this strategy works on the markets where the underlying mechanism applies and fails elsewhere". The second form is falsifiable and more useful. The former is a marketing claim that real researchers should be suspicious of. ## Cases where the universal claim holds Three signs that distinguish an honest cross-market claim from an overfit one. Sign 1: the parameters are identical across markets and structurally motivated. The 12-1 momentum factor uses the same 12-1 horizon on every market because the horizon is set by the academic literature's structural prior, not by per-market tuning. The aggregate Sharpe of approximately 0.4 to 0.7 is consistent across decades. Sign 2: the strategy is reported with after-cost numbers per market and the worst markets are positive but small, not negative. A strategy that earns Sharpe 0.3 on the slowest market and Sharpe 0.7 on the fastest market is structurally consistent. A strategy that earns Sharpe 1.5 on every market is suspicious. Sign 3: the failure modes are documented per market. The honest paper says "the strategy struggles on natural gas during the storage-season transition months" or "the strategy underperforms on FX during high-vol risk-off regimes". The dishonest paper says "the strategy works everywhere". Honest cross-market reporting includes the per-market failure modes; dishonest cross-market reporting suppresses them. ## Decision matrix | Cross-market claim | Likely interpretation | Action | | --------------------------------------------------------------------------- | ------------------------------------------------------- | -------------------------------------- | | Same parameters, full matrix, after costs, Sharpes 0.3-0.7 | Honest, structurally consistent | Validate within market family | | Per-market optimized, Sharpes 1.0+ across 30+ markets | Per-market overfitting, multiple-comparison-uncorrected | Reject claim | | Headline aggregate without per-market matrix | Selection bias likely | Demand the full matrix | | Universal claim that crosses asset families (equity to bonds to crypto) | Different mechanisms | Reject the universality framing | | Sharpes high on backtest, low or negative on the largest hostile event days | Outlier suppression | Demand event-level transparency | | Failure modes documented per market | Honest reporting | Treat as informative robustness signal | | No documented failure modes | Suspicious | Investigate or reject | | Cross-market Sharpe matches sqrt(N\_effective) scaling, not sqrt(N) | Honest diversification accounting | Treat as informative | The matrix is operational. The pattern: ask for the full per-market matrix, the same parameters, after-cost numbers, and per-market failure-mode documentation. A claim that survives all four asks is informative. A claim that fails one or more is a red flag that the universality is at least partly methodology-driven. ## Visualizing the cross-market claim ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/05/article_66-visual-0.png) ## KEY POINTS - A strategy that claims to work on every market in a heterogeneous universe (equity indices, FX, commodities, bonds, crypto) is more often a sign of methodology issues than a sign of a universal edge. Markets have different microstructures, participant mixes, and macro drivers; a single strategy framework rarely captures all of them. - Five common reframings hide behind the universal claim: per-market parameter optimization, cherry-picked sample of markets where the strategy worked, market-specific filters that contradict the universality framing, exposure to a universal long-bias rather than a strategy-specific signal, outlier suppression that removes the worst events. - Three structural reasons genuine edges are usually market-specific: each market has its own microstructure (tick size, spreads, fee structure, liquidity), each market has its own participant mix (institutional, retail, HFT, hedgers), each market has its own macro driver (growth, rates, supply-demand, central-bank flows). - Two genuine universal patterns exist and have known low Sharpes: equity-premium long-only buy-and-hold (Sharpe approximately 0.4 to 0.5 across markets), trend-following premium across futures (Sharpe approximately 0.5 to 0.7 across decades). A claim of higher universal Sharpe is a different and much stronger claim that requires extraordinary evidence. - The right cross-market protocol: select a market family with a shared structural premise, use the same parameter values across the family, apply realistic per-market transaction costs, report the full matrix not the aggregate, account for cross-market correlation when computing portfolio Sharpe. - Multiple-comparison correction is mandatory. Testing 100 markets with per-market false-positive rate 0.05 produces an expected 5 spurious significant results under the null. Bonferroni or Benjamini-Hochberg correction is the operational fix. - Three signs of an honest cross-market claim: identical structurally-motivated parameters across markets, after-cost numbers per market with the worst markets positive but small not negative, documented failure modes per market. - Anti-pattern: treating "works on 38 markets" as proof of robustness without examining the parameter consistency, the multiple-comparison correction, or the per-market matrix. - Anti-pattern: averaging Sharpes across correlated markets equally. The portfolio Sharpe scales with sqrt(N\_effective), and N\_effective is much smaller than N when the markets are correlated. - Anti-pattern: reporting only the markets where the strategy works. The cross-market test requires the full matrix including failures. - The current article gives the skeptical view of cross-market claims. The next article in the publication (["Market Personality: Why Gold, FX, Crypto, and Equities Need Different Systems"](https://aligrithm.com/market-personality-why-gold-fx-crypto-and-equities-need-different-systems/)) covers the constructive version: what each market class requires from a strategy framework. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Testing and Tuning Market Trading Systems - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Testing+and+Tuning+Market+Trading+Systems+Timothy+Masters&ref=aligrithm.com) - [Data Mining Algorithms in C++ - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Data+Mining+Algorithms+in+C%2B%2B+Timothy+Masters&ref=aligrithm.com) ### 3.12 The Difference Between Robustness and Optimization URL: https://aligrithm.com/the-difference-between-robustness-and-optimization/ Last updated: 2026-07-27T13:24:07.000Z Two researchers build the same nominal strategy: a momentum filter on a basket of 30 futures contracts, with an entry threshold and an exit horizon. Researcher A spends six months optimizing the strategy. The final spec uses entry threshold 1.42 sigma, exit horizon 14 bars, vol-target 12%, instrument-specific position multipliers ranging from 0.7 to 1.3, and a session filter that excludes the first and last 30 minutes. The IS Sharpe is 1.85\. Researcher B spends six weeks. The final spec uses entry threshold 2.0 sigma, exit horizon 20 bars, vol-target 10%, equal weights across instruments, no session filter. The IS Sharpe is 0.95\. Both strategies ship in January 2022\. Live performance through 2024: Researcher A Sharpe 0.4, max drawdown 18%. Researcher B Sharpe 0.85, max drawdown 11%. Researcher A optimized. Researcher B chose robust. The IS Sharpe favored A by almost a factor of two. The OOS performance favored B by the same factor. The pattern is the structural consequence of the difference between optimization (best in-sample fit) and robustness (insensitivity to parameter and regime variation). Calling it luck or anomaly misses the mechanism. Most published research and most retail trading literature conflates the two. The articles in this pillar so far covered specific cases of the conflation: stationarity violations that killed a tuned strategy (["Stationarity: The Word Every Trader Ignores Until It Kills the Strategy"](https://aligrithm.com/stationarity-the-word-every-trader-ignores-until-it-kills-the-strategy-2/)), regime mismatch between IS and OOS (["Why OOS Failure Is Often a Stationarity Failure"](https://aligrithm.com/why-oos-failure-is-often-a-stationarity-failure/)), the rolling-window-length overfit (["Rolling Normalization: Useful Tool or Hidden Overfit?"](https://aligrithm.com/rolling-normalization-useful-tool-or-hidden-overfit/)). This article gives the upstream conceptual frame: optimization is the wrong objective, robustness is the right one, and the two are usually in direct opposition. ## Operational definitions Operational definitions, not philosophical ones. Optimization. The process of searching parameter space to maximize an in-sample objective (Sharpe, expectancy, profit factor, equity-curve smoothness). The output is a single parameter set that scored highest on the IS objective. The process is well-understood, well-supported by libraries, and produces visually compelling results. The process also produces parameter sets that are tuned to the specific noise of the IS sample. Robustness. The property that the strategy's performance does not depend sensitively on small changes to parameters, on the specific market or time period of the backtest, or on the inclusion or exclusion of specific trades. A robust strategy with parameter set X has approximately the same realized Sharpe at parameter set X', X+epsilon, and X-epsilon for small perturbations. A robust strategy on SPX has approximately the same Sharpe (within statistical noise) when applied to similar instruments. A robust strategy on 2010-2019 has approximately the same Sharpe on 2000-2009 (after accounting for known regime differences). The two are not synonyms. The optimal parameter set is typically not robust because the optimizer climbed to a sharp peak that does not survive perturbation. The most robust parameter set is typically not optimal in-sample because it sits at the center of a flat plateau rather than the top of a peak. The article ["The Hill, the Spike, and the Cliff: Reading Optimization Surfaces"](https://aligrithm.com/the-hill-the-spike-and-the-cliff-reading-optimization-surfaces/) later in this pillar covers the visual diagnosis of which surface you are climbing. ## Optimization loses to robustness OOS Three structural reasons. Reason 1: the IS objective contains noise. Any IS Sharpe is the true unobservable Sharpe plus sampling noise. The standard error of an IS Sharpe with N trades is approximately 1 / sqrt(N - 1). Optimizing to maximize IS Sharpe pulls the parameter set toward the peak of the noise plus the peak of the true edge. The OOS realization is drawn from the true distribution and the noise component does not replicate. The OOS Sharpe is lower by the magnitude of the noise contribution. Reason 2: the parameter surface has many local maxima. A strategy with five parameters over a reasonable grid has thousands or millions of candidate parameter sets. The "best" set is the maximum over a noisy realization of a function with thousands of local maxima. The maximum over many noisy values is biased upward by an amount that scales with the search width and the noise level. The article ["Why More Parameters Make a Strategy Easier to Sell and Easier to Break"](https://aligrithm.com/why-more-parameters-make-a-strategy-easier-to-sell-and-easier-to-break/) later in this pillar quantifies this concern; the present article frames the conceptual consequence. Reason 3: optimization rewards sharpness. A parameter set that produces a high but narrow peak in IS Sharpe is preferred by the optimizer over a parameter set that produces a slightly lower but wider plateau. The narrow peak is fragile to any change in the underlying distribution; the wide plateau is robust. Optimization, as a procedure, chooses the fragile option because the IS objective does not penalize fragility. ## Tools for choosing a robust parameter set Five operational tools. Tool 1: parameter-stability mapping. Plot the IS Sharpe as a function of each parameter, with all other parameters held at the optimum. A robust optimum sits on a wide plateau where the Sharpe is approximately constant across a range of values. A fragile optimum sits on a sharp peak. Choose the parameter from the plateau region rather than from the peak. The article ["Parameter Stability Beats Best Parameter"](https://aligrithm.com/parameter-stability-beats-best-parameter/) later in this pillar covers the technique in detail. Tool 2: bootstrap parameter selection. Bootstrap the IS sample many times and re-optimize on each bootstrap. The distribution of optimal parameters across bootstraps is the parameter uncertainty. A robust strategy has tight bootstrap parameter distributions. A fragile strategy has wide bootstrap distributions. The right parameter choice is the median or mode of the bootstrap distribution, not the global IS optimum. Tool 3: cross-market robustness. Run the same strategy on multiple comparable markets (SPX, NDX, RTY, EURO STOXX, NIKKEI). A robust parameter set produces positive Sharpe on most or all of them. A fragile parameter set is positive on the original market and near-zero or negative on the others. The article "Why Works on All Markets Is Usually a Red Flag" later in this pillar gives a contrarian view of the cross-market test; the present article uses the test as one diagnostic among several rather than as a sole criterion. Tool 4: time-period robustness. Run the same strategy on non-overlapping time periods (1990-1999, 2000-2009, 2010-2019). A robust parameter set is positive across most or all of the periods. A fragile set is positive on one and negative on others. Combined with regime stratification (covered in ["Regime Coverage: Why Your Backtest Needs Different Market States"](https://aligrithm.com/regime-coverage-why-your-backtest-needs-different-market-states/)), the time-period check is a useful but coarse test. Tool 5: parameter perturbation. Compute the IS Sharpe at the optimum and at small perturbations of each parameter (e.g., +/-10%, +/-25%, +/-50%). Plot the resulting Sharpes. A robust optimum has gentle declines under perturbation. A fragile optimum has steep declines. Reject parameter sets that lose more than 25% of the IS Sharpe under +/-25% perturbation; the OOS will be much worse than that. ## The math of the optimization-vs-robustness trade-off A simple model that captures the structural pattern. Assume the true Sharpe at parameter set theta is a smooth function with mean SR\_true(theta), and the realized IS Sharpe is the true Sharpe plus mean-zero noise with standard error sigma\_IS. $$ \\widehat{\\text{SR}}\_{\\text{IS}}(\\theta) = \\text{SR}\_{\\text{true}}(\\theta) + \\varepsilon(\\theta), \\qquad \\mathbb{E}\[\\varepsilon\] = 0, \\quad \\text{Var}\[\\varepsilon\] = \\sigma\_{\\text{IS}}^2 $$ The optimizer chooses theta\_hat = argmax\_theta SR\_IS(theta). The expected OOS Sharpe at theta\_hat is approximately the true Sharpe at the location where the noise was largest, not the location where the true Sharpe is largest. The expected loss from optimization (the gap between the IS optimum's IS Sharpe and the same parameter's OOS Sharpe) scales with sigma\_IS times the search width. $$ \\mathbb{E}\[\\widehat{\\text{SR}}\_{\\text{IS}}(\\hat{\\theta}) - \\text{SR}\_{\\text{true}}(\\hat{\\theta})\] \\approx \\sigma\_{\\text{IS}} \\cdot \\sqrt{2 \\log K} $$ K is the effective number of independent parameter combinations searched. For K = 100, the inflation is approximately 3 standard errors. For K = 10000, it is approximately 4 standard errors. The IS-optimal Sharpe is biased upward by this amount, and the OOS Sharpe falls short of the IS Sharpe by the same amount on average. The robust choice (the median of the parameter-perturbation distribution rather than the IS optimum) avoids the search-width bias and produces an unbiased estimate of OOS performance. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Cases where optimization fits Three conditions under which optimization is the right tool. Condition 1: the optimization target is robustness, not in-sample fit. An optimizer can be run on a robustness objective (e.g., minimize the variance of cell-stratified Sharpes from ["Regime Coverage: Why Your Backtest Needs Different Market States"](https://aligrithm.com/regime-coverage-why-your-backtest-needs-different-market-states/)). The optimizer is then choosing the parameter set that produces the most consistent performance across regimes, not the highest in-sample number. The result is robust by construction. Condition 2: the parameter space is small relative to the sample size. With three parameters and 5000 trades, the optimization-induced bias is small. With twenty parameters and 500 trades, the bias is overwhelming. Apply the "10% rule" from the article "The 10% Rule of Degrees of Freedom" as the operational threshold. Condition 3: the optimization is constrained to plateau regions. Some optimizers (Bayesian optimization with a smoothness prior, regularized optimization with a complexity penalty, multi-objective optimization that includes parameter stability) are designed to find robust optima rather than IS-optimal ones. These produce different parameter choices than the standard maximize-IS-Sharpe procedure. In most retail and small-shop research, none of the three conditions hold. The default optimizer maximizes IS Sharpe, the parameter space is wide, and the objective is not robustness. The output is by construction not robust. ## Anti-patterns Five mistakes specific to the optimization-vs-robustness conflation. Anti-pattern 1: presenting the IS-optimal parameter set as the strategy. The IS-optimal set is one realization of an estimator; it is not the strategy. The strategy is the parameter set that survives the robustness checks (parameter perturbation, cross-market, cross-period, bootstrap selection). Reporting only the IS-optimal set hides the strategy's actual fragility. Anti-pattern 2: treating "we optimized over many parameters" as a virtue. The optimization process inflates the IS Sharpe by an amount proportional to the search width. Wide searches produce more impressive IS results and worse OOS results. The right framing: the search width is a cost, not a feature. Anti-pattern 3: re-optimizing in response to OOS underperformance. The strategy underperforms; the researcher re-optimizes on the recent data; the new parameters validate well on the new IS and fail on the next OOS. The cycle continues. Each re-optimization is overfitting to the most recent regime. The right response to OOS underperformance is the diagnostic flow from ["Why OOS Failure Is Often a Stationarity Failure"](https://aligrithm.com/why-oos-failure-is-often-a-stationarity-failure/) and the decommission policy from ["How to Detect When a Trading System Is Dying"](https://aligrithm.com/how-to-detect-when-a-trading-system-is-dying/), not re-optimization. Anti-pattern 4: optimizing on a smoothed equity curve. The optimizer maximizes Sharpe, profit factor, or some similar aggregate. The aggregate hides the within-strategy fragility. A strategy with a smooth equity curve in IS that depends on perfectly-timed regime entries is more fragile than a strategy with a noisier equity curve that does not depend on regime timing. Anti-pattern 5: assuming optimization reveals the "best" strategy. The optimization reveals the best fit to the IS sample. The two are different. The best strategy is the one with the highest unconditional expected Sharpe, which is best estimated by the median of robust-selection methods, not by the maximum of an IS objective. ## Decision matrix | Research stage | Right tool | Wrong tool | | ------------------------------ | --------------------------------------- | -------------------------------------- | | Strategy hypothesis generation | Domain knowledge + structural priors | IS-Sharpe-driven parameter search | | Initial validation | Crude parameters, broad backtest | Optimized parameters before validation | | Parameter selection | Robustness checks, bootstrap median | Maximum IS Sharpe | | Cross-market test | Equal weights, broad sample | Per-market optimized parameters | | Pre-deployment | Parameter perturbation, regime coverage | "Final tweak" optimization | | Live monitoring | Diagnostic panel, kill switches | Re-optimization on live data | | Post-decommission | Post-mortem, robustness audit | Re-optimization to "save" the strategy | The matrix maps research stage to the right tool. The pattern: optimization belongs at the end of a robust-selection pipeline, not at the beginning, and not in response to live data. The article ["Optimization Comes After Testing, Not Before"](https://aligrithm.com/optimization-comes-after-testing-not-before/) later in this pillar covers the procedural version of this principle. ## Visualizing the trade-off ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/05/article_65-visual-0.png) ## KEY POINTS - Optimization maximizes the in-sample objective. Robustness is the property that performance does not depend sensitively on parameters, market, or time period. The two stand in direct opposition under most conditions, despite being conflated in practice. - The IS-optimal parameter set is typically not robust because the optimizer climbs to a sharp peak that does not survive perturbation. The most robust parameter set is typically not IS-optimal because it sits on a plateau. - Three structural reasons optimization loses OOS: the IS objective contains noise, the parameter surface has many local maxima with the maximum-of-noise bias, optimization rewards sharpness over plateau-width. - The optimization-induced bias scales with sigma\_IS times sqrt(2 log K) where K is the effective number of parameter combinations searched. For K=100, the bias is approximately 3 standard errors. For K=10000, approximately 4 standard errors. - Five tools for choosing robust parameters: parameter-stability mapping (plot Sharpe against each parameter), bootstrap parameter selection (re-optimize on bootstraps and use the median), cross-market robustness (positive on multiple comparable markets), time-period robustness (positive on multiple non-overlapping windows), parameter perturbation (Sharpe survives +/-25% changes). - Three conditions when optimization is appropriate: the objective is robustness rather than IS fit, the parameter space is small relative to sample size (the 10% rule), the optimizer is constrained to plateau regions through smoothness priors or complexity penalties. - Anti-pattern: presenting the IS-optimal parameter set as the strategy. The strategy is the parameter set that survives robustness checks, not the one that maximized IS. - Anti-pattern: treating "we optimized over many parameters" as a virtue. Search width is a cost, not a feature. - Anti-pattern: re-optimizing in response to OOS underperformance. Each re-optimization overfits to the most recent regime. The right response is the diagnostic flow and the decommission policy. - Anti-pattern: optimizing on a smoothed aggregate (Sharpe, profit factor) without checking the within-strategy regime sensitivity. - The current article gives the conceptual frame. The next article in the publication ("Why Works on All Markets Is Usually a Red Flag") covers a contrarian case: the cross-market robustness check, taken naively, can also mislead. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Testing and Tuning Market Trading Systems - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Testing+and+Tuning+Market+Trading+Systems+Timothy+Masters&ref=aligrithm.com) - [Data Mining Algorithms in C++ - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Data+Mining+Algorithms+in+C%2B%2B+Timothy+Masters&ref=aligrithm.com) ### 3.11 Regime Coverage: Why Your Backtest Needs Different Market States URL: https://aligrithm.com/regime-coverage-why-your-backtest-needs-different-market-states/ Last updated: 2026-07-27T13:24:07.000Z A research desk completes a backtest of a US equity beta-neutral pairs strategy on the period January 2010 to December 2019\. The backtest reports Sharpe 1.6, max drawdown 7%, 1850 round-trip trades, profit factor 1.7, win rate 56%. The desk ships the strategy in January 2020\. Live performance from 2020 to 2024: Sharpe 0.3, max drawdown 22%, profit factor 1.1\. The desk's first instinct is the same diagnostic flow covered in ["Why OOS Failure Is Often a Stationarity Failure"](https://aligrithm.com/why-oos-failure-is-often-a-stationarity-failure/): run permutation tests, audit the pipeline, check the sample size, run the regime overlap KS test on the IS-vs-OOS feature distributions. The KS test rejects equality on multiple features at p < 0.001\. The diagnosis is regime mismatch. The deeper question is upstream of the diagnosis. The backtest period (2010 to 2019) was a single macro regime: post-GFC recovery, persistent QE, low and falling interest rates, low and falling realized volatility, dominant equity beta as the cross-sectional risk factor. The regime was unusually stable for an unusually long time. The backtest was informative about that regime. The backtest was not informative about any other regime, because no other regime was in the sample. The strategy that the backtest validated is a strategy that works in 2010-to-2019 conditions; it is not a strategy that works in 2020-to-2024 conditions, and the backtest gave no information about the latter. The discipline that prevents this failure is regime coverage: the requirement that the backtest sample contains enough trading days in each of the regimes the strategy will face in deployment. The article ["Why OOS Failure Is Often a Stationarity Failure"](https://aligrithm.com/why-oos-failure-is-often-a-stationarity-failure/) framed regime coverage as the proper denominator for OOS validation. The articles ["Volatility Regimes and Strategy Survival"](https://aligrithm.com/volatility-regimes-and-strategy-survival/) and ["Why Volatility Is More Non-Stationary Than Trend"](https://aligrithm.com/why-volatility-is-more-non-stationary-than-trend/) framed the specific case of vol regime coverage. This article gives the operational machinery: how to enumerate the regimes that matter, how to verify coverage in the IS sample, how to report stratified performance, how to handle rare regimes through synthesis, and how to decide whether the strategy is deployment-ready or whether more data (or synthetic data) is needed. ## Enumerating the regime axes Five regime axes that matter for most strategies. The exact set depends on the strategy class. Axis 1: realized volatility. Three to four bins (low, normal, high, crisis), thresholded against the asset's own historical distribution. Covered in ["Volatility Regimes and Strategy Survival"](https://aligrithm.com/volatility-regimes-and-strategy-survival/). Axis 2: trend versus chop. Two to three bins by autocorrelation of returns at the relevant horizon, or by the absolute t-statistic of the rolling mean return. A strategy that bets on direction needs trending bins; a strategy that bets on mean reversion needs chop bins. Axis 3: cross-asset correlation regime. Two to three bins by rolling correlation of the strategy's primary asset to other reference assets (SPX, AGG, USD, gold). Strategies that consume correlation structure need each regime represented. Axis 4: macro regime. Two to four bins by macro state (recession vs expansion, inflation regime, monetary-policy stance). Macro-conditional strategies need each macro regime represented in IS. Axis 5: microstructure era. The structural state of the market: pre-decimalization, decimalization era, HFT era, post-Reg-NMS, post-2010 ETF dominance, post-2020 retail-flow regime. For high-frequency or microstructure-sensitive strategies, the era boundary is the regime boundary. A two-axis stratification (e.g., vol-by-trend) gives 6 to 9 regime cells. A three-axis stratification gives 12 to 27\. The minimum requirement: for each cell that the live deployment will face, the IS sample contains at least N trading days (typically 252 days as a minimum for crude estimates, 1000 for confident estimates). ## Computing the regime distribution of the IS sample Three steps. Step 1: classify each IS day into a regime cell. For the SPX example: each day is classified as (vol regime in {low, normal, high, crisis}) x (trend mode in {trend, chop}) using the thresholds from prior articles. Each day gets a tuple label. Step 2: count the IS days per cell. $$ N\_c^{(\\text{IS})} = \\#\\{t \\in \\text{IS sample} : \\text{regime}(t) = c\\}, \\qquad \\forall c \\in \\mathcal{C} $$ Step 3: compare the IS regime distribution to the historical or expected long-run regime distribution. $$ p\_c^{(\\text{IS})} = \\frac{N\_c^{(\\text{IS})}}{|\\text{IS sample}|}, \\qquad p\_c^{(\\text{long-run})} = \\frac{N\_c^{(\\text{long-run})}}{|\\text{long-run sample}|} $$ A material difference between the two distributions (e.g., IS p\_crisis = 0.005 versus long-run p\_crisis = 0.03) is the signature of an unrepresentative IS sample. The 2010 to 2019 SPX example: the IS distribution had p\_low\_vol\_trending = 0.78, p\_normal\_vol\_chop = 0.15, p\_high\_vol = 0.05, p\_crisis = 0.0\. The long-run distribution (1990 to 2024) had p\_low\_vol\_trending = 0.40, p\_normal\_vol\_chop = 0.30, p\_high\_vol = 0.25, p\_crisis = 0.05\. The IS sample concentrated 78% of trading days in one regime cell; that cell carries 78% of the IS Sharpe estimate, and the strategy was implicitly tuned for that one cell. ## Stratified Sharpe reporting The regime-stratified report. For each cell with N >= 252 days, compute the cell-specific Sharpe with its bootstrap confidence band. Report the matrix. $$ \\text{SR}\_c = \\frac{\\bar{r}\_c}{\\hat{\\sigma}\_c} \\cdot \\sqrt{252}, \\qquad \\text{CI}(\\text{SR}\_c) = \\pm \\frac{2}{\\sqrt{N\_c / 252}} $$ The aggregate IS Sharpe is the regime-weighted average of cell Sharpes. The right reporting: $$ \\begin{array}{l|c|c|c} \\text{Regime cell} & \\text{Days in IS} & \\text{Cell Sharpe} & \\text{Cell CI} \\\\ \\hline \\text{Low vol, trending} & 1973 & +1.85 & \[+1.40, +2.30\] \\\\ \\text{Normal vol, chop} & 380 & +0.10 & \[-0.40, +0.60\] \\\\ \\text{High vol} & 130 & -0.85 & \[-2.20, +0.50\] \\\\ \\text{Crisis} & 0 & \\text{N/A} & \\text{insufficient data} \\\\ \\end{array} $$ The aggregate IS Sharpe of 1.6 is dominated by the low-vol-trending cell (1973 days at Sharpe 1.85). The strategy is a low-vol-trending strategy in operational terms, with a near-zero contribution from the chop cell and a negative contribution from the high-vol cell. The crisis cell is uncovered. The aggregate number hides the fragility. The stratified report transforms the deployment decision. Without it, the desk sees Sharpe 1.6 and ships. With it, the desk sees that 78% of the IS sample is in one regime, the strategy is negative in another regime, and a key regime (crisis) has zero coverage. The conclusion: this strategy is not deployment-ready until either more data is collected (waiting for natural regime variation) or the missing regimes are covered through synthesis or alternative samples. ## Handling rare regimes Three approaches when the IS sample does not cover a key regime. Approach 1: extend the IS sample backward. The standard fix: pull historical data further back to capture more regime variation. The 2010-2019 SPX backtest extended to 1990-2019 covers the 2000-2002 dot-com regime, the 2008 GFC, and several intermediate regimes. The cost: the long history may include microstructure or macro eras that no longer apply. The 1990s pre-decimalization SPX is a different microstructure than the 2024 SPX. Decide which historical regimes are representative of the current era and which are not. Approach 2: synthesize regime-specific paths. Generate synthetic price paths that have the statistical properties of each missing regime (vol level, autocorrelation, return distribution, cross-asset correlation). Backtest the strategy on the synthetic paths. The synthetic Sharpe is informative about how the strategy would perform if the missing regime occurred. The article ["Monte Carlo for Trading Systems"](https://aligrithm.com/monte-carlo-for-trading-systems/) later in this pillar covers the synthesis machinery in detail. Approach 3: cross-asset / cross-instrument substitution. If SPX 2010-2019 covers only one regime, look at other equity indices, other countries, or other asset classes during the same period for analog regimes. The Nikkei 1990-2019 covers high-vol-deflationary regimes that SPX does not. Substituting a comparable regime from a different instrument provides a noisy estimate of how the strategy would have behaved in that regime on SPX. Each approach has costs. Backward extension may include irrelevant eras. Synthesis assumes a parametric model of the regime that may be wrong. Cross-asset substitution assumes the strategy generalizes across instruments. Use multiple approaches in parallel and report each separately; the agreement (or disagreement) between approaches is itself informative. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## The deployment decision rule A simple rule that makes regime coverage operational. $$ \\text{Deploy} \\iff \\forall c \\in \\mathcal{C}\_{\\text{required}}: \\; N\_c^{(\\text{IS or synthetic})} \\ge N\_{\\min} \\text{ AND } \\text{SR}\_c \\ge \\text{SR}\_{\\min} $$ The rule has two conditions. First, every required regime cell must have at least N\_min days of coverage (in IS or in synthesized data). Second, the cell-specific Sharpe must exceed SR\_min in every required cell. A strategy that earns positive returns in three out of four cells but is severely negative in the fourth is not deployment-ready: the negative cell will produce drawdowns proportional to its prevalence in the live deployment. Typical thresholds: N\_min = 252 days per cell (one year of representative trading days), SR\_min = 0.0 (the strategy must at least break even in each cell, ideally positive). Stricter thresholds for high-stakes deployment: N\_min = 1000, SR\_min = 0.3\. The thresholds are operational defaults, not laws of nature; calibrate to the strategy's risk budget and the team's tolerance for regime-conditional losses. ## Cases of impossible regime coverage Some regimes do not have enough historical instances to support a sample-size-adequate backtest. SPX crisis regimes (>30% annualized vol) constitute approximately 3% of days across 1990-2024, with the largest events clustered in 2008 and 2020\. A strategy needs more than three or four crisis events to claim it is robust to crises; the historical record does not provide that many independent events. The honest acknowledgment: certain strategies cannot be fully validated against historical regime coverage because the regime is too rare. The right response: explicitly document the uncovered regime in the strategy specification, size the strategy assuming the uncovered regime will occur, and have a written disable rule for when the live data enters the uncovered regime. The article ["Why Systems Work Until They Don't"](https://aligrithm.com/why-systems-work-until-they-dont/) framed the finite-life-budgeting approach; uncovered regimes are one specific case where the budgeting needs to be more conservative. ## Anti-patterns Five mistakes specific to regime coverage. Anti-pattern 1: the convenient backtest window. Picking 2010-2019 as the backtest period because the data is clean and the volatility is low produces a backtest that validates one regime. The convenience is also the bias. The article ["Why OOS Failure Is Often a Stationarity Failure"](https://aligrithm.com/why-oos-failure-is-often-a-stationarity-failure/) framed this in terms of OOS calendar selection; the same point applies to IS calendar selection. Anti-pattern 2: the aggregate-Sharpe-as-validation report. Reporting only the aggregate IS Sharpe without the stratified cell breakdown is not informative about the strategy's regime sensitivity. The aggregate hides the fragility. The right report is always stratified. Anti-pattern 3: assuming "the next decade will look like the last decade". The post-GFC SPX 2010-2019 was a single-regime decade. The post-2020 SPX has been a multi-regime period. Strategies validated on 2010-2019 implicitly assumed regime persistence and paid the cost when the regime changed. Plan for regime changes; do not assume continuity. Anti-pattern 4: synthesizing without acknowledgment. Some shops generate synthetic data to cover gaps but report the synthetic-augmented Sharpe as if it were a historical Sharpe. The synthesis is a parametric model assumption that may be wrong; the synthetic Sharpe should always be reported separately from the historical Sharpe. Anti-pattern 5: skipping the regime coverage check. The desk runs the backtest, sees a high Sharpe, and ships. The regime coverage check would have shown that the high Sharpe is from one regime cell that may not persist. The skipped check is the upstream cause of the OOS failure that the desk later misdiagnoses as overfitting. ## Decision matrix | IS regime coverage | Deploy decision | Action | | ----------------------------------------------------------- | ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ | | All required cells, all positive Sharpe, all N >= 252 | Deploy | Standard production protocol | | All cells covered but one negative Sharpe | Deploy with cell gating | Disable in negative-Sharpe regime per ["Volatility Regimes and Strategy Survival"](https://aligrithm.com/volatility-regimes-and-strategy-survival/) | | Some cells under-covered (N < 100) but positive | Deploy at reduced size | Half allocation, monitor regime entry | | Some cells uncovered (N = 0) | Do not deploy | Extend IS sample, synthesize, or substitute | | Aggregate Sharpe high but driven by one cell (>70% of days) | Do not deploy | Strategy is single-regime; reframe accordingly | | Synthetic-augmented coverage shows positive Sharpe per cell | Conditional deploy | Document synthesis assumptions in spec, monitor real regime entry | | Crisis cell uncovered, no synthesis | Deploy with hard disable rule | Decommission on crisis entry per ["How to Detect When a Trading System Is Dying"](https://aligrithm.com/how-to-detect-when-a-trading-system-is-dying/) | The matrix is operational. The key principle: a backtest is informative about the regimes it covers and silent about the rest. The deployment decision is the question of whether the live regime distribution will match the IS regime distribution well enough for the IS Sharpe to be predictive. ## Visualizing regime coverage ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/05/article_64-visual-0.png) ## KEY POINTS - A backtest is informative about the regimes its sample covers and silent about regimes it does not. The 2010-2019 SPX backtest covers one regime well and most other regimes barely or not at all. Strategies validated on this sample are validated for that regime, not for all regimes. - Five regime axes matter for most strategies: realized volatility, trend versus chop, cross-asset correlation, macro regime, microstructure era. A two-axis stratification gives 6 to 9 cells; a three-axis stratification gives 12 to 27. - Regime coverage check: classify each IS day into a regime cell, count days per cell, compare to the long-run or expected distribution. Material under-coverage of any cell is the signature of a regime-biased IS sample. - Stratified Sharpe report: cell-specific Sharpe with bootstrap confidence bands, plus the cell's count of IS days. The aggregate IS Sharpe is the regime-weighted average and is dominated by the most-represented cell. - The 2010-2019 SPX example: aggregate IS Sharpe 1.6, dominated by low-vol-trending (1973 days at Sharpe 1.85). Other cells are near-zero or negative. Crisis cells are uncovered. The aggregate hides the fragility. - Three approaches to handle rare or uncovered regimes: extend the IS sample backward (cost: includes irrelevant historical eras), synthesize regime-specific paths (cost: parametric model assumption), cross-asset substitution (cost: assumes generalization across instruments). Use multiple approaches and report each separately. - Deployment decision rule: every required regime cell must have N >= N\_min days of coverage (in IS or synthesized) AND cell-specific Sharpe must be at least SR\_min. Typical defaults: N\_min = 252, SR\_min = 0.0\. Stricter for high-stakes: N\_min = 1000, SR\_min = 0.3. - When a regime cannot be covered (e.g., crisis events too rare for meaningful backtest), explicitly document the uncovered regime, size the strategy assuming it will occur, write a disable rule for live entry into the uncovered regime. - Anti-pattern: the convenient backtest window (2010-2019 SPX). Convenience is also bias. Anti-pattern: aggregate-Sharpe-as-validation. The aggregate hides fragility. Anti-pattern: assuming the next decade resembles the last. Anti-pattern: synthesizing without acknowledgment. Anti-pattern: skipping the regime coverage check. - The current article gives the regime-coverage discipline that the rest of Pillar 3 (walk-forward, Monte Carlo, CSCV, parameter stability, stop-loss analysis, profit-factor critique) all depend on. A strategy that passes those tests on a single-regime sample is not validated. The next article in the publication (["The Difference Between Robustness and Optimization"](https://aligrithm.com/the-difference-between-robustness-and-optimization/)) frames the upstream principle that regime coverage is a property of the testing process, not of the optimization process. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Testing and Tuning Market Trading Systems - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Testing+and+Tuning+Market+Trading+Systems+Timothy+Masters&ref=aligrithm.com) - [Data Mining Algorithms in C++ - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Data+Mining+Algorithms+in+C%2B%2B+Timothy+Masters&ref=aligrithm.com) ### 3.10 Rolling Normalization: Useful Tool or Hidden Overfit? URL: https://aligrithm.com/rolling-normalization-useful-tool-or-hidden-overfit/ Last updated: 2026-07-27T13:24:06.000Z A research team builds a cross-asset trend-following strategy with a single momentum signal: 90-day return divided by an exponentially weighted moving standard deviation of returns. The team tries three values of the EWMA window: 60, 120, and 240 days. The 60-day version produces an IS Sharpe of 1.2\. The 120-day version produces 1.0\. The 240-day version produces 0.8\. The team selects 60 days based on the IS Sharpe and ships the strategy. Live performance: Sharpe 0.4 across the next 24 months, well below all three IS estimates and well below even the worst of the three. The diagnosis is not that 60 days was the wrong window in isolation. The diagnosis is that the team picked a window by optimizing against the IS Sharpe. The window is a hyperparameter, the IS comparison was an optimization, and the realized OOS performance reflects the post-selection bias. Had the team picked the 240-day window from a structural prior (matched to the regime half-life) rather than from the IS comparison, the OOS Sharpe would have been around 0.7 to 0.8, not 0.4\. The optimization that looked like prudent calibration was an overfit on the rolling-window length itself. This article covers the trap and the operational defenses against it. The article ["How to Make Indicators More Stationary"](https://aligrithm.com/how-to-make-indicators-more-stationary/) gave rolling normalization as one of several tools. The article ["When Forcing Stationarity Destroys Information"](https://aligrithm.com/when-forcing-stationarity-destroys-information/) gave the case where the tool destroys signal. This article covers the more subtle failure mode where the tool is appropriate for the strategy and the parameter (the window length) silently becomes the overfitting hyperparameter. Most production rolling-normalization implementations have this problem to some degree; the question is how much, and what discipline reduces the cost. ## The mechanics of rolling normalization The standard form. At each time t, compute the rolling mean and rolling standard deviation of a feature X over the past W observations, then standardize the current value. $$ z\_t = \\frac{X\_t - \\hat{\\mu}\_t^{(W)}}{\\hat{\\sigma}\_t^{(W)}}, \\qquad \\hat{\\mu}\_t^{(W)} = \\frac{1}{W} \\sum\_{i=1}^{W} X\_{t-i}, \\quad \\hat{\\sigma}\_t^{(W)} = \\sqrt{\\frac{1}{W-1} \\sum\_{i=1}^{W} (X\_{t-i} - \\hat{\\mu}\_t^{(W)})^2} $$ The rolling rank is the percentile-based variant. Rolling z-score is the moment-based variant. Both transform a feature whose unconditional distribution is non-stationary into a feature whose within-window distribution is approximately uniform (rank) or approximately N(0,1) (z-score). The transformation matters when the strategy threshold rule needs the input distribution to be stable. The hidden parameter is W. Different choices of W produce different transformed features. The transformed feature is a different feature for each W, with different predictive properties, different stationarity profiles, and different OOS behavior. The phrase "the same feature with a different window" misdescribes the operation. ## The window-length trade-off Three structural properties move with W in opposite directions. Property 1: regime-tracking speed. A short W (e.g., 30 to 60 days) tracks regime changes quickly. The transformed feature responds to a vol regime shift within weeks. A long W (e.g., 1000+ days) tracks regime changes slowly; the transformed feature lags the regime shift by months. Property 2: signal preservation. A short W eats slow drift in the underlying signal. If the strategy depends on a multi-year cross-sectional momentum, a 60-day rolling z-score within each asset removes precisely the signal (the example in ["When Forcing Stationarity Destroys Information"](https://aligrithm.com/when-forcing-stationarity-destroys-information/)). A long W preserves slow drift. Property 3: estimation precision. A short W produces noisy estimates of mean and std. A long W produces precise estimates. The standard error of the rolling z-score scales with 1/sqrt(W) in the noise component. $$ \\text{SE}(z\_t) \\approx \\sqrt{\\frac{1}{W} \\cdot \\left(1 + \\frac{z\_t^2}{2}\\right)} $$ The operational consequence: there is no globally correct W. The right choice depends on whether the strategy needs fast regime tracking (short W is right), or signal preservation (long W is right), or estimation precision (longer is generally better). Different strategies in the same portfolio may need different W values for the same underlying feature. A unified W "for all features in the strategy" is a reasonable default but a structural compromise. ## The optimization-against-the-backtest trap The empirical pattern from the opening example. The researcher tries multiple W values and picks the one with the highest IS Sharpe. The trap has three failure modes that compound. Failure 1: in-sample noise. The IS Sharpe at each W has its own standard error. A Sharpe of 1.2 for W=60 versus 1.0 for W=120 may not be a meaningful difference; it may be sampling noise. The researcher who picks the higher one is selecting on noise rather than on edge. Failure 2: lookback selection bias. The set of W values tried is itself an information leakage. If the researcher tried W = 30, 60, 90, 120, 180, 240, 365, 500, 1000 and picked the best, the implicit search width is much larger than three values. The right inference accounts for the search width across all considered Ws, not the chosen one alone. Failure 3: the overfit grows with the granularity. Trying W = 60, 65, 70, 75, ..., 240 gives 37 candidates. The IS Sharpe of the best of 37 is higher than the IS Sharpe of any one structural choice, with the gap reflecting search luck rather than edge. The article ["Why More Parameters Make a Strategy Easier to Sell and Easier to Break"](https://aligrithm.com/why-more-parameters-make-a-strategy-easier-to-sell-and-easier-to-break/) framed the general degrees-of-freedom concern; rolling normalization windows fall within that concern. ## Causality and look-ahead Two specific look-ahead patterns to audit. Pattern 1: full-sample normalization. Some implementations compute the mean and std on the entire sample, then apply the standardization point-by-point. The implementation is fast and produces a clean-looking transformed series. The implementation is also look-ahead biased: the mean and std at time t use observations from t+1, t+2, ..., T. The OOS estimate that depends on full-sample normalization will not generalize because the live trading version cannot use future observations to compute the historical mean. Pattern 2: warm-up period filled with the first valid value. The rolling estimate is undefined for the first W observations. Some implementations fill these with the first valid rolling estimate (backfilling the warm-up with the W+1-th value). The backfill is a soft form of look-ahead because the W+1-th value uses observations 1 through W+1, but the strategy is then allowed to trade on observations 1 through W using the backfilled estimate. The standard fix: do not trade during the warm-up. Live trading does not generate the warm-up issue because the strategy starts after sufficient history; backtests need to drop the warm-up window. Pattern 3: shifted rolling estimates that include the current observation. The rolling estimate at t should be computed from observations t-1, t-2, ..., t-W. Some implementations include t in the rolling window (using observations t, t-1, ..., t-W+1). Including t means the standardization at t uses information from t itself, which is an instantaneous-information leakage. The standardized value at t is a function of X\_t which includes the current observation; the test that the standardized z\_t predicts the next return is contaminated by the contemporaneous coupling. Each of these patterns can be present without the researcher noticing, with code paths that use vectorized libraries the common offender. The article ["Why OOS Failure Is Often a Stationarity Failure"](https://aligrithm.com/why-oos-failure-is-often-a-stationarity-failure/) framed the leakage detection step. Run the audit on every rolling-normalization implementation before trusting the results. ## Choosing W without optimization Five operational rules. Rule 1: pick W from a structural prior, not from the IS Sharpe. The structural prior is set by the strategy's intended timescale. A strategy that holds positions for 5 days needs a W that is much longer than 5 days (so the rolling window does not see only the holding period). A strategy that targets a specific regime timescale (e.g., the macro cycle, approximately 5 to 7 years) needs W matched to that timescale. The article ["How to Make Indicators More Stationary"](https://aligrithm.com/how-to-make-indicators-more-stationary/) gave the timescale-matching guideline. Rule 2: pick W once and document the choice. The choice is part of the strategy specification. Subsequent retraining or recalibration uses the same W. Changing W in production is a methodology change that requires the same rigor as the original deployment. Rule 3: report the strategy's IS performance across multiple W values, beyond the chosen one alone. The robustness check: if the strategy's Sharpe varies across a wide range over W = 100, 252, 500, 1000, the result is fragile and the strategy is not robust to the rolling-window choice. If the Sharpe is approximately constant across the range, the choice is non-critical and the strategy is robust. Rule 4: avoid the "we will retune W if the strategy underperforms" trap. Retuning W in response to underperformance is a discretionary override of the original calibration and is a common form of look-ahead bias in live trading. The strategy specification should set W at deployment and the operational protocol should not allow W to be changed in response to OOS performance. Rule 5: when the IS Sharpe is very sensitive to W, treat the sensitivity as a signal that the strategy is overfit to a specific regime. A robust strategy is approximately insensitive to small changes in W. A strategy that drops from Sharpe 1.2 at W=60 to Sharpe 0.5 at W=120 is overfit to the W=60 regime structure, regardless of which W is "correct". ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Rolling normalization done right Three checks that confirm the tool is appropriate. Check 1: the underlying feature has a non-stationary distribution that is uncorrelated with the strategy's signal axis. The feature has slow drift in mean and variance that the strategy does not depend on for its edge. Rolling normalization removes the drift without removing the signal. Check 2: the strategy uses the feature as an input to a threshold rule whose threshold is fixed in standardized units. The standardized feature has a stable distribution by construction, so the threshold has consistent operational meaning across regimes. Check 3: the strategy's signal lives in the high-frequency component of the feature, not in the slow drift. Rolling normalization is a high-pass filter on the feature; the strategy that consumes high-frequency variation benefits from the filtering. If any of the three checks fails, the tool is wrong for the strategy. If check 1 fails, rolling normalization eats the drift that was the signal; use cross-sectional or regime-conditional methods instead. If check 2 fails, the threshold rule is not fixed in standardized units and the standardization adds noise; consider rank-based or quantile-based thresholds. If check 3 fails, rolling normalization removes the slow drift that the strategy needs; do not apply the transformation, or apply it with a much longer W. ## Anti-patterns Five mistakes specific to rolling normalization. Anti-pattern 1: the W-grid search. Trying W = 30, 60, 90, ..., 1000 and picking the best IS Sharpe is overfitting. The right discipline is structural prior + insensitivity check. Anti-pattern 2: the implicit W-search through "robustness analysis". Reporting "the strategy works for W from 60 to 240 with Sharpe between 0.9 and 1.2" is itself a search over six values; the post-search Sharpe distribution is biased. Report the IS Sharpe at the structurally chosen W; do robustness checks on a separate validation slice. Anti-pattern 3: full-sample normalization for "convenience". The convenience is that the implementation is faster and the rolling-estimate noise is lower. The cost is look-ahead bias. The convenience is not worth the bias. Anti-pattern 4: changing W in response to live underperformance. The retune is a soft form of overfitting because it incorporates live performance into the training. The strategy specification should set W at deployment; live underperformance triggers the decommission protocol from ["How to Detect When a Trading System Is Dying"](https://aligrithm.com/how-to-detect-when-a-trading-system-is-dying/), not a recalibration of W. Anti-pattern 5: assuming the strategy is robust to W on the grounds that the IS Sharpe is similar across two values. Two W values, both picked from a search, can be near-equivalent in IS and very different in OOS. The robustness check needs to use values that span the structural range and were not part of the original search. ## Decision matrix | Strategy timescale | Strategy goal | Recommended W | Verification | | -------------------------- | -------------------------------- | ------------------------------------------- | ---------------------------------------- | | Intraday | Signal lives in minutes-to-hours | 30 to 60 minutes (within-day rolling) | Stable distribution within typical day | | Multi-day swing | Signal lives in days-to-weeks | 60 to 250 trading days | Histograms by month equal | | Multi-month trend | Signal lives in months | 500 to 1000 trading days | Histograms by year equal | | Multi-year factor | Signal lives in years | 2500+ trading days or full sample (passive) | Histograms by 5-year window equal | | Vol regime detection | Regime classification | 60 to 250 trading days with hysteresis | Rolling stat stable, transitions visible | | Cross-sectional comparison | Universe-relative | Use cross-sectional norm, not time-series | Cross-sectional rank stable per date | | Spread / pairs | Within-pair mean reversion | Match to half-life of mean reversion | Half-life test from Pillar 2 | | Macro cycle | Cycle-conditional positioning | 1000 to 2500 trading days | Within-cycle distribution stable | The matrix matches W to strategy timescale rather than to backtest performance. The discipline: pick from the matrix, document the choice, run the insensitivity check, do not optimize. ## Visualizing the W trap ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/05/article_63-visual-0.png) ## KEY POINTS - Rolling normalization (z-score or rank within a rolling W-bar window) is a useful tool for stationarizing a feature whose unconditional distribution drifts. The window length W is a hyperparameter and the most common source of overfitting in this part of the pipeline. - Three structural properties move with W in opposite directions: regime-tracking speed (short W is faster), signal preservation (long W preserves slow drift), estimation precision (long W is more precise). There is no globally correct W; the right choice depends on the strategy. - The optimization-against-backtest trap. Picking W by maximizing IS Sharpe inflates the in-sample estimate by an amount that scales with the search width. The resulting OOS Sharpe is materially lower than the IS Sharpe. - Three look-ahead patterns to audit: full-sample normalization (uses the whole sample's mean and std), warm-up period backfilled with first valid value, shifted rolling estimates that include the current observation. Each contaminates the backtest. - Five operational rules for choosing W: pick from a structural prior matched to the strategy's intended timescale, document the choice in the strategy spec, report performance across multiple W values for robustness, do not retune W in response to live underperformance, treat W-sensitivity as a sign of overfitting to a specific regime. - Three checks that confirm rolling normalization is the right tool: feature non-stationarity is uncorrelated with the strategy's signal axis, threshold rule is in standardized units, signal lives in the high-frequency component. - Anti-pattern: the W-grid search across 30 to 1000 with selection on best IS Sharpe. Anti-pattern: the implicit W-search through "robustness analysis". Anti-pattern: full-sample normalization for convenience. Anti-pattern: changing W in response to live underperformance. Anti-pattern: assuming W-robustness from a small IS comparison. - The right discipline matches W to strategy timescale: intraday signal gets minutes to hours, multi-day swing gets 60 to 250 trading days, multi-month trend gets 500 to 1000, multi-year factor gets 2500+ or full-sample-passive, vol regime detection gets 60 to 250 with hysteresis. - Cross-sectional strategies use cross-sectional normalization, not time-series rolling normalization. The article ["When Forcing Stationarity Destroys Information"](https://aligrithm.com/when-forcing-stationarity-destroys-information/) framed this distinction; the present article gives the operational rule. - The current article gives the rolling-normalization-specific discipline. The next article in the publication (["Regime Coverage: Why Your Backtest Needs Different Market States"](https://aligrithm.com/regime-coverage-why-your-backtest-needs-different-market-states/)) covers the upstream question that often drives the W choice: how to ensure the backtest covers enough regimes that the W choice is validated against the right distribution of conditions. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Testing and Tuning Market Trading Systems - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Testing+and+Tuning+Market+Trading+Systems+Timothy+Masters&ref=aligrithm.com) - [Data Mining Algorithms in C++ - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Data+Mining+Algorithms+in+C%2B%2B+Timothy+Masters&ref=aligrithm.com) ### 3.9 When Forcing Stationarity Destroys Information URL: https://aligrithm.com/when-forcing-stationarity-destroys-information/ Last updated: 2026-07-27T13:24:06.000Z A researcher builds a cross-sectional momentum strategy on US large-cap equities. The signal is the 12-month total return minus the 1-month total return, ranked across the universe. The historical Sharpe of this signal in its raw form (1965 to 2024) is approximately 0.55\. The researcher reads the prior article on stationarity transformations, applies a rolling 60-day z-score to each stock's 12-1 momentum series before ranking, and re-runs the backtest. The new Sharpe is 0.18\. The researcher tries a 252-day z-score window. New Sharpe 0.31\. The researcher tries 1000 days. Sharpe 0.46\. The original raw form, with no z-scoring, beats every transformed version. The transformation pipeline that the methodology demanded was a destruction-of-signal operation, not a stationarization-of-feature operation. The diagnosis: the 12-1 momentum signal lives in the non-stationary component of each stock's return distribution. Stocks that are in a sustained uptrend have shifted their return distribution upward; the ranking captures this shift. The rolling z-score within each stock's own history removes exactly the shift that the strategy was supposed to consume. After z-scoring, every stock looks "normal" relative to its own recent history, and the cross-sectional ranking has no information left. The transformation worked perfectly as a stationarizer (the within-stock distribution is stable across windows). The transformation also destroyed the signal. This is the case the prior article (["How to Make Indicators More Stationary"](https://aligrithm.com/how-to-make-indicators-more-stationary/)) flagged at the end. Stationarization is a precondition for reliable consumption when the strategy depends on a stable feature distribution. Stationarization is a destructive operation when the strategy depends on the non-stationary component of the feature itself. The two cases look identical until you ask the right question: what part of the feature is the strategy trading? The answer determines whether the transformation is the right tool or the wrong one. This article gives the framework for distinguishing the two cases, the diagnostic that catches over-aggressive stationarization, and the operational rule for when to leave a feature alone. ## The signal-vs-stationarity dichotomy Three categories, each with a different prescription. Category 1: stationarity-violating noise. The non-stationary component is irrelevant to the signal the strategy consumes. Example: the absolute price level of SPX is non-stationary; a strategy designed around price-relative-to-MA consumes the deviation, with the level discarded. Stationarize aggressively (log-relative, divide by reference). The signal is preserved by the transformation; the noise (level drift) is removed. Category 2: stationarity-violating signal. The non-stationary component is the signal. Example: the 12-1 momentum factor; the cross-sectional dispersion of multi-month returns; the long-horizon trend in interest-rate spreads. Stationarize cautiously or not at all. The transformation removes the very thing the strategy targets. Category 3: mixed. The feature has both stationary signal and non-stationary noise components, and a stationary signal component, and a non-stationary signal component. Example: RSI on a single asset; the indicator has stable structure (the bounded oscillator behavior) plus a slow drift in the within-bounds distribution that may or may not be signal. Apply a partial transformation that removes only the noise component, or accept some non-stationarity in the signal component. The classification is not always obvious from the indicator definition. The right question to ask: is the strategy trading the absolute level of this feature, or the relative position of this feature against its own history, or the cross-sectional rank of this feature against other instruments? Each answer points to a different transformation strategy. ## The information-loss test A diagnostic that catches over-aggressive transformation. Compute the mutual information between the raw feature and the target return, and between the transformed feature and the target return. The ratio gives the fraction of predictive information preserved by the transformation. $$ \\text{Information retention} = \\frac{I(\\tilde{X}; Y)}{I(X; Y)} $$ A retention near 1.0 means the transformation kept the signal. A retention well below 1.0 (say, under 0.6) means the transformation removed signal. A retention near 0 means the transformation removed the signal. The article ["Relative Entropy as an Indicator Quality Score"](https://aligrithm.com/relative-entropy-as-an-indicator-quality-score/) framed the information-theoretic measurement at the indicator-quality level; the same machinery applies here at the transformation-evaluation level. In practice, mutual information estimation needs care: it is sensitive to binning, sample size, and the joint distribution's tail behavior. A simpler proxy: compute the IS Sharpe of the strategy with raw feature and with transformed feature on the same window. If the transformed-feature Sharpe is materially lower than the raw-feature Sharpe, the transformation destroyed signal. The 12-1 momentum example at the top of this article is a clean case: raw Sharpe 0.55, z-scored Sharpe 0.18 to 0.46 depending on window. The transformation cost most of the signal. ## Specific cases where stationarization destroys signal Five common patterns. Pattern 1: differencing a trending price for a trend-following strategy. The strategy targets the trend, which is the non-stationary component of the price. Differencing converts price to returns, which are approximately stationary. The strategy now consumes a feature with no trend information. The trend strategy on differenced returns has near-zero Sharpe by construction. Pattern 2: rolling z-scoring a long-horizon momentum factor. The example at the top of this article. The strategy targets the cross-sectional difference in long-horizon momentum, which lives in the non-stationary mean-shift of each stock's distribution. Rolling z-score within each stock removes the shift. The transformation works as stationarizer and as signal destroyer simultaneously. Pattern 3: dividing a regime indicator by its own moving average. A vol-regime classifier is built to detect when current vol is high relative to long-run vol. Dividing current vol by its own rolling average creates a ratio that is approximately stationary by construction (the long-run mean of the ratio is approximately 1.0). The ratio has no information about the absolute regime; a 30%-vol regime and a 12%-vol regime can both have ratio 1.0\. The vol-regime gate that consumes the ratio cannot distinguish high-vol from low-vol in absolute terms. Pattern 4: ranking a long-horizon trend within a too-short window. A multi-year trend signal is ranked within a 60-day rolling window. The 60-day rolling rank has no memory of the multi-year trend; every observation is in the same rolling distribution as the recent observations only. The trend strategy that consumed the rank has lost the long-horizon information. Pattern 5: applying GARCH-residualization to remove all heteroskedasticity. A strategy that consumes the volatility regime as its signal will be devastated by GARCH-residualization, which removes precisely the heteroskedasticity that the strategy wanted. The residual feature is approximately Gaussian and approximately stationary; it is also approximately useless for the original strategy. The pattern across the five: the transformation removes the same component of the feature that the strategy depends on. The transformation is technically correct as stationarization but operationally wrong because it does not preserve the signal. ## The right framework Three principles for deciding when to stationarize. Principle 1: identify the signal axis before applying any transformation. The signal axis is the variable that the strategy depends on for its edge: the cross-sectional rank, the within-asset deviation, the absolute level, the rate of change, the regime category. The transformation must preserve the signal axis. If the transformation eats the signal axis, do not apply it. Principle 2: stationarize in the orthogonal direction. If the signal is in the cross-sectional dimension, stationarize each instrument's time series so the cross-section is comparable but the within-instrument signal is preserved. If the signal is in the time-series dimension, stationarize the cross-section so the within-asset signal is preserved. The two transformations are orthogonal in their stationarization-vs-signal trade-off. Principle 3: accept some non-stationarity in the signal direction. The strategy is robust to feature non-stationarity in the direction the signal lives, because the strategy was built to consume that non-stationarity. Trying to "fix" the non-stationarity in that direction destroys the strategy. The right discipline is the regime-coverage discipline (covered in "Regime Coverage: Why Your Backtest Needs Different Market States" later in this pillar): make sure the backtest covers enough regimes that the signal is validated across the non-stationary component the strategy needs. A worked example. The cross-sectional 12-1 momentum strategy: signal axis is the cross-sectional rank of long-horizon returns. Stationarize across stocks (so a 5% return on Apple and a 5% return on a microcap are comparable in vol-adjusted terms) but do not stationarize within each stock's time series (because the within-stock drift is the signal). The right transformation: divide each stock's 12-1 return by the cross-sectional dispersion of 12-1 returns at that date, so the cross-sectional comparison is on a consistent scale, but each stock's own time series retains its trend information. $$ \\tilde{m}\_t^{(i)} = \\frac{m\_t^{(i)} - \\overline{m}\_t^{(\\text{cross})}}{\\sigma\_t^{(\\text{cross})}}, \\qquad m\_t^{(i)} = \\text{12-1 momentum of stock } i $$ The transformation is a cross-sectional z-score (each stock's momentum standardized against the cross-section at the same time), not a time-series z-score (each stock's momentum standardized against its own history). The cross-sectional version preserves the within-stock trend signal and stationarizes the cross-section. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Cases for leaving a feature raw Three signals that argue against any transformation. Signal 1: the raw feature already passes the stationarity diagnostics on the strategy's intended deployment regime. The rolling mean and std are stable, the histograms by window overlap, the threshold percentile is consistent. No transformation is needed. Signal 2: the strategy's edge IS the non-stationary component, and the non-stationarity is structural rather than spurious. Trends in interest rates that reflect Fed policy are signal. Cross-sectional dispersion in earnings growth that reflects economic structure is signal. The non-stationarity in these features is the data the strategy is trying to read. Removing it removes the signal. Signal 3: the transformation candidates fail the information-loss test. The transformed-feature Sharpe is materially lower than the raw-feature Sharpe across multiple windows and parameter choices. The transformation is destroying signal regardless of which window or parameter is chosen. In all three cases, leave the feature alone. Apply other defenses against non-stationarity (regime-coverage backtesting, walk-forward retraining, regime-conditional gating from the prior articles in this pillar) rather than feature-level transformation. ## Anti-patterns Five mistakes specific to the over-stationarization failure mode. Anti-pattern 1: applying every transformation in the prior article's recipe matrix without testing the information-loss. The recipe matrix gives starting points; each strategy needs verification that the transformation preserved signal. The information-loss test (or the IS-Sharpe-with-raw-vs-transformed comparison) catches the cases where the recipe was wrong for the strategy's signal axis. Anti-pattern 2: assuming that "more stationary" is always "better". A perfectly stationary feature has no time-series structure and is uninformative for any time-dependent signal. The right amount of stationarity is the minimum needed for the strategy to consume the feature reliably, not the maximum the toolkit can produce. Anti-pattern 3: confusing the strategy class. A strategy that trades cross-sectional rank does not need within-instrument time-series stationarity. A strategy that trades within-instrument level deviations does not need cross-sectional comparability. Applying both transformations because "we use both kinds of strategies" produces a feature that is not appropriate for either. Anti-pattern 4: stationarizing before the diagnostic. The right order is: diagnose the violation, identify whether the violation is signal or noise, then apply the matching transformation. The wrong order is: apply the standard transformation pipeline, then diagnose the result. The wrong order makes the diagnostic conditional on the transformation, which masks signal destruction. Anti-pattern 5: rejecting a strategy because the raw feature failed standard stationarity tests. ADF and KPSS tests on the raw 12-1 momentum factor fail because the factor is correctly non-stationary by construction (the cross-sectional dispersion of multi-month returns drifts). Rejecting the strategy on the test result alone discards a real signal. The right discipline: pass the test on the cross-sectional comparability axis, accept the non-stationarity on the signal axis, validate by regime coverage instead of by feature-level test. ## Decision matrix | Strategy class | Signal axis | Right stationarization | Wrong stationarization | | ----------------------------- | --------------------------------- | -------------------------------------------------- | -------------------------------------------------- | | Cross-sectional momentum | Cross-sectional rank | Cross-sectional z-score | Time-series z-score per asset | | Time-series momentum | Within-asset trend | None or vol-normalize | Differencing the price | | Mean reversion (within-asset) | Within-asset deviation | Rolling rank with W matched to regime | Rank with W matched to holding period | | Vol-regime gating | Absolute vol level | Long-window scaling against multi-decade reference | Short-window ratio against own MA | | Spread / yield-curve | Persistent drift | None or KPSS-passing differencing | Aggressive z-scoring | | Volatility prediction | Vol regime + clustering | Log-vol with persistent component | GARCH-residualization to remove heteroskedasticity | | Earnings surprise | Cross-sectional shock magnitude | Cross-sectional z-score | Time-series z-score per stock | | Pairs trading | Within-pair spread mean reversion | Difference + rolling mean subtract on the spread | Aggressive ranking that loses the level | The matrix is operational, illustrative. Each strategy needs verification of the signal axis before any transformation choice. ## Visualizing the trade-off ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/05/article_62-visual-0.png) ## KEY POINTS - Stationarization is a destructive operation when the strategy depends on the non-stationary component of the feature. The cross-sectional 12-1 momentum strategy's raw Sharpe is 0.55\. The same strategy after rolling-z-scoring each stock's series gives Sharpe 0.18 to 0.46 depending on window. The transformation worked as stationarizer and destroyed signal. - Three categories of feature non-stationarity: stationarity-violating noise (transform aggressively), stationarity-violating signal (do not transform), mixed (partial transformation in the noise direction only). - The classification depends on the signal axis: the variable the strategy depends on for its edge. Cross-sectional rank, within-asset deviation, absolute level, rate of change, regime category. The transformation must preserve the signal axis. - The information-loss test catches over-aggressive transformation. Compare the IS Sharpe of the strategy with the raw feature versus with the transformed feature; a material drop is evidence the transformation removed signal. Mutual information ratio between transformed and raw with the target return is the formal version. - Five common patterns where stationarization destroys signal: differencing a trending price for trend-following, rolling z-scoring a long-horizon momentum factor, dividing a regime indicator by its own MA, ranking a long-horizon trend within a too-short window, applying GARCH-residualization to a vol-trading signal. - Three principles for the right framework: identify the signal axis before transforming, stationarize in the orthogonal direction only, accept non-stationarity in the signal direction and use regime-coverage backtesting instead of feature-level fixes. - Worked example: cross-sectional 12-1 momentum gets cross-sectional z-score (standardize across stocks at each date), not time-series z-score per stock. The cross-sectional version preserves the within-stock trend signal that the strategy needs. - Three signals to leave a feature raw: rolling stats and histograms already pass the stationarity diagnostics on the deployment regime, the strategy's edge is the non-stationary component and the non-stationarity is structural, the information-loss test rejects every transformation candidate. - Anti-pattern: applying every recipe from the standard transformation pipeline without testing for information loss. The pipeline gives starting points, not guarantees. - Anti-pattern: assuming "more stationary is always better". A perfectly stationary feature has no time-series structure and is uninformative for time-dependent signals. The right amount is the minimum needed for reliable consumption. - Anti-pattern: rejecting a strategy because the raw feature fails ADF or KPSS. Some signals are correctly non-stationary by construction; rejection on test alone discards real edge. - The current article gives the trade-off framework. The next article in the publication (["Rolling Normalization: Useful Tool or Hidden Overfit?"](https://aligrithm.com/rolling-normalization-useful-tool-or-hidden-overfit/)) covers one specific transformation in operational depth, including how the window-length choice itself can become an overfitting hyperparameter. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Testing and Tuning Market Trading Systems - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Testing+and+Tuning+Market+Trading+Systems+Timothy+Masters&ref=aligrithm.com) - [Data Mining Algorithms in C++ - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Data+Mining+Algorithms+in+C%2B%2B+Timothy+Masters&ref=aligrithm.com) ### 3.8 How to Make Indicators More Stationary URL: https://aligrithm.com/how-to-make-indicators-more-stationary/ Last updated: 2026-07-27T13:24:05.000Z Take the standard 14-period RSI on EURUSD daily across 1995 to 2024\. The unconditional histogram of the indicator looks reasonable: a roughly symmetric distribution centered near 50, with most mass between 30 and 70 and tails reaching toward 20 and 80\. Slice the same indicator into 5-year windows and the unconditional appearance breaks down. The 1995 to 1999 RSI distribution has median 49.8 and 90th percentile 73\. The 2010 to 2014 distribution has median 50.3 and 90th percentile 76\. The 2020 to 2024 distribution has median 51.9 and 90th percentile 71\. The shifts are small but operationally lethal: a threshold rule "go long when RSI < 30" fires at the 12th percentile of the 1995 to 1999 distribution, the 9th percentile of 2010 to 2014, and the 14th percentile of 2020 to 2024\. The same rule selects different fractions of trading days across windows, with different post-trigger return distributions, and the strategy backtest mixes those windows into a single composite estimate that is correct for none of them. The diagnosis is non-stationarity in the indicator distribution. The fix is to transform the raw indicator into a more stationary form before any threshold rule consumes it. The article ["Stationarity: The Word Every Trader Ignores Until It Kills the Strategy"](https://aligrithm.com/stationarity-the-word-every-trader-ignores-until-it-kills-the-strategy-2/) gave five tools at high level. This article gives the practical recipes by indicator class: which transformation matches which indicator, why, and what diagnostic confirms the transformation worked. The next article in the publication (["When Forcing Stationarity Destroys Information"](https://aligrithm.com/when-forcing-stationarity-destroys-information/)) covers the counter-case where over-aggressive transformation destroys the very signal the strategy was supposed to consume. ## The transformation pipeline A four-step workflow that applies to any raw indicator. Step 1: diagnose the non-stationarity. Plot the indicator's rolling 252-day mean and rolling 252-day standard deviation across the full sample. Plot the indicator's distribution histograms for non-overlapping 4 to 5 year windows. Apply Kolmogorov-Smirnov or Anderson-Darling on consecutive windows. The article ["Why OOS Failure Is Often a Stationarity Failure"](https://aligrithm.com/why-oos-failure-is-often-a-stationarity-failure/) framed the regime-overlap test machinery; the same machinery applies here at the feature level. Step 2: identify the violation type. The diagnosis produces one of four patterns. Mean drift (rolling mean moves persistently). Variance drift (rolling std varies by more than 50% across windows). Distributional shape change (the histogram shape changes even if mean and variance are stable). Bounded-but-shifting (the indicator is bounded by construction but the within-bounds distribution shifts). Step 3: apply the matching transformation. The match between violation type and transformation determines the right tool. Step 4: verify the transformation. Re-run step 1 diagnostics on the transformed feature. The transformed feature should have approximately stable rolling mean, approximately stable rolling std, and approximately stable histogram shape across windows. If it does not, the transformation was the wrong choice or the underlying indicator has multiple non-stationarity modes. ## Recipes by indicator class Eight common indicator classes, with the matching recipe for each. Class 1: raw price. Indicators that consume raw price (the price itself, simple moving averages, MA crossovers stated in absolute price units) inherit the unbounded non-stationary level. Recipe: convert to log returns or to log-relative measures (log of price divided by the moving average). The transformation produces a feature that is approximately stationary in mean. $$ \\tilde{X}\_t = \\log\\!\\left(\\frac{P\_t}{\\text{MA}\_t}\\right) \\quad \\text{instead of} \\quad X\_t = P\_t - \\text{MA}\_t $$ The log-relative form is invariant to the price level and the percent-deviation interpretation is constant across decades. The article ["The Case Against Raw Price Indicators"](https://aligrithm.com/the-case-against-raw-price-indicators/) (covered in Pillar 2) framed this point at the indicator-design level. Class 2: moving-average crossover differences. Indicators like MACD that compute the difference between two MAs are stationary in mean but the variance scales with the price level (a 50-point MACD signal in a 4000-level SPX is the same percent move as a 1-point MACD in a 80-level SPX). Recipe: divide the MACD-style signal by an ATR or rolling std reference. $$ \\tilde{X}\_t = \\frac{\\text{EMA}\_{12}(P) - \\text{EMA}\_{26}(P)}{\\text{ATR}\_{14}(P)} $$ The ATR-scaled MACD is approximately stationary in variance across regimes. The article ["Why ATR Normalization Is More Than a Volatility Trick"](https://aligrithm.com/why-atr-normalization-is-more-than-a-volatility-trick/) covered this technique in detail. Class 3: range and volatility indicators. ATR, true range, Parkinson estimator. The raw values scale with the price level (a 50-point true range on SPX 4000 is the same as a 0.5-point true range on SPX 40). Recipe: divide by a longer-window reference (multi-year ATR or by the price level itself). $$ \\tilde{\\text{ATR}}\_t = \\frac{\\text{ATR}\_{14}(t)}{\\text{ATR}\_{252}(t)} \\quad \\text{or} \\quad \\frac{\\text{ATR}\_{14}(t)}{P\_t} $$ The first form (short ATR / long ATR) is the relative-vol form and is stationary in level by construction (the ratio is dimensionless and its long-run mean is approximately 1.0). The second form (ATR / price) is the percent-vol form and is stationary on multi-decade windows. Class 4: bounded oscillators. RSI, Stochastics, Williams %R. The raw indicator is bounded but the within-bounds distribution shifts (the example at the top of this article). Recipe: rolling rank or rolling z-score within a moderate window. $$ \\text{rank}\_W(X\_t) = \\frac{\\#\\{X\_{t-i} : i \\in \[0, W-1\], X\_{t-i} \\le X\_t\\}}{W} $$ The rolling rank maps the indicator to \[0, 1\] uniformly within the rolling window. The transformed feature is uniform on its rolling distribution by construction, which makes threshold rules invariant to the within-window distribution shape. The article ["Why Indicator Histograms Matter"](https://aligrithm.com/why-indicator-histograms-matter/) framed this for the design phase. Class 5: unbounded momentum / ROC indicators. ROC, MOM, signed cumulative returns over a window. The raw indicator is unbounded and its variance scales with the price-return distribution which is itself non-stationary. Recipe: scale by a forecast or rolling estimate of the return std. $$ \\tilde{X}\_t = \\frac{\\text{ROC}\_{20}(P\_t)}{\\hat{\\sigma}\_{\\text{return}, 20}(t)} $$ The result is a vol-normalized momentum feature whose distribution has approximately stable variance across regimes. The point is that a 5% return in a 8%-vol regime is a different signal than a 5% return in a 18%-vol regime; vol-normalization makes the two comparable. Class 6: volume indicators. Raw volume is non-stationary in mean (it grows with the long-run growth of trading activity), in variance (volume is heteroskedastic), and often has unit-root properties. Recipe: log-transform first, then subtract a rolling mean. $$ \\tilde{V}\_t = \\log(V\_t + 1) - \\overline{\\log(V)}\_{252} $$ The log handles the multiplicative growth and the heteroskedasticity. The rolling-mean subtraction handles the slow drift in baseline activity. The +1 in the log is a numerical safeguard for zero-volume bars. Class 7: spread and rate indicators. Yield-curve slopes, credit spreads, FX-cross spreads. The raw indicator is approximately stationary in some regimes and has slow drift in others. Recipe: differencing for the persistent component and rolling z-scoring for the regime-conditional level. $$ \\tilde{X}\_t = \\Delta X\_t \\quad \\text{(if drift is the issue)} \\quad \\text{or} \\quad z\_t = \\frac{X\_t - \\hat{\\mu}\_t}{\\hat{\\sigma}\_t} \\quad \\text{(if regime level is the issue)} $$ The choice depends on the diagnostic. If the indicator has a unit root (KPSS rejects, ADF fails to reject), use differencing. If the indicator has slow drift but is otherwise level-stationary, use rolling z-score with a long window (1000+ days) so the z-score does not eat slow drift that the strategy was supposed to trade. Class 8: distribution-shape indicators. Skewness, kurtosis, percentile-based features. The raw indicator is itself a higher moment of a non-stationary distribution and is doubly non-stationary. Recipe: estimate over rolling windows, then apply rolling z-score on the rolling estimate. The general principle: for any indicator that is itself a moment estimator of a non-stationary process, the transformation needs to handle both the level non-stationarity of the moment and the structural non-stationarity of the underlying distribution. Two-stage transformations (log-then-z, or rank-then-z) are common for this class. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Verification diagnostics Three checks that confirm the transformation worked. Check 1: rolling mean and std on the transformed feature. Plot the rolling 252-day mean and std across the full sample. The transformed feature should have rolling mean and std that vary within tight bands. The article ["Stationarity: The Word Every Trader Ignores Until It Kills the Strategy"](https://aligrithm.com/stationarity-the-word-every-trader-ignores-until-it-kills-the-strategy-2/) gave the rolling-statistic check as the cheapest diagnostic; reuse it here. Check 2: histograms by window. Slice the transformed feature into 4 to 5 year windows. Plot the histograms overlaid. The transformed feature's histograms should look approximately the same shape across windows. KS or Anderson-Darling on consecutive windows should fail to reject equality at p < 0.05. Check 3: threshold-position stability. Compute the percentile of the threshold value in each window (the same calculation as in the EURUSD RSI example at the top of this article). The transformed feature's threshold percentile should be stable across windows. If the rule is "go long when transformed feature < q", then q should sit at the same rolling percentile across windows by construction (true for rolling-rank transforms) or by approximation (true for well-calibrated rolling z-score transforms). If any of the three checks fails, the transformation is wrong for the underlying violation type, the underlying indicator has multiple non-stationarity modes that one transformation cannot handle, or the chosen window length is wrong (too short and the transform tracks short-term noise; too long and the transform does not adapt to slow drift). ## Choosing window lengths The most common parameter in stationarity transforms is the rolling window length W. The wrong choice is a frequent failure mode. Three guidelines. Guideline 1: match W to the regime timescale, not to the strategy's holding period. If the strategy holds positions for 5 days but the regime drift has a half-life of 250 days, the rolling window should be on the order of the regime half-life, not the holding period. A 5-day rolling z-score eats the regime signal that the strategy was supposed to trade. Guideline 2: avoid optimizing W against the backtest. A W chosen to maximize the strategy's IS Sharpe is overfit and the transformation will not work in OOS. Pick W from a structural prior (the regime half-life, the macro cycle length, the autocorrelation half-life of the underlying indicator) and stick with it. Guideline 3: longer is safer for slow drift, shorter is safer for fast regime breaks. The right window depends on which non-stationarity you are most exposed to. Most strategies face slow drift (covered in ["Slow Wandering: The Most Dangerous Type of Market Change"](https://aligrithm.com/slow-wandering-the-most-dangerous-type-of-market-change/)) more than abrupt regime breaks, so 252 to 1000 day windows are reasonable defaults. Shorter windows (60 to 100 days) are appropriate only when fast regime detection is the priority and the cost of eating slow signal is acceptable. The article ["Rolling Normalization: Useful Tool or Hidden Overfit?"](https://aligrithm.com/rolling-normalization-useful-tool-or-hidden-overfit/) later in this pillar covers the window-length question in operational detail. ## Anti-patterns Five mistakes specific to indicator stationarization. Anti-pattern 1: applying every transformation to every indicator "to be safe". A pipeline that log-transforms then z-scores then ranks then differences a feature destroys most of the predictive information. Each transformation has a cost. Apply the minimum needed and verify with the diagnostic checks. Anti-pattern 2: choosing the transformation by what looks "smoother". A smoother transformed feature is not necessarily a more stationary one. Smoothness reduces high-frequency noise; stationarity stabilizes the unconditional distribution. The two are different properties. Use the verification diagnostics to decide, not visual smoothness. Anti-pattern 3: applying the transformation in-sample only. The transformation must be causal: at every time t, the transformation parameters (rolling means, rolling stds, rolling ranks) must be computed using only data up to t-1\. A transformation that uses the full-sample mean for normalization is contaminated with look-ahead bias and the backtest is unreliable. The article ["Why OOS Failure Is Often a Stationarity Failure"](https://aligrithm.com/why-oos-failure-is-often-a-stationarity-failure/) covered the leakage-detection step. Anti-pattern 4: forgetting that the transformation parameter (W) is itself a hyperparameter. A z-score with W=60 is a different feature from a z-score with W=252\. Treating them as equivalent and "trying both" without proper degrees-of-freedom accounting inflates the IS-OOS gap. The article "Degrees of Freedom in Trading Systems" frames this concern. Anti-pattern 5: assuming the transformed feature has the predictive power of the raw feature. Each transformation removes some non-stationary structure but also some signal. The next article in the publication (["When Forcing Stationarity Destroys Information"](https://aligrithm.com/when-forcing-stationarity-destroys-information/)) covers this trade-off in detail. The right framing: stationarization is a precondition for reliable consumption by a strategy, not a free lunch on signal quality. ## Decision matrix | Indicator class | Violation type | Recipe | Verification | | ------------------------------- | --------------------------------- | ------------------------------------------- | ------------------------------------------ | | Raw price | Unbounded level drift | Log-relative to MA | Rolling mean of log-relative, KS by window | | MA crossover (MACD-like) | Variance scales with price | Divide by ATR | Rolling std stable, histogram by window | | ATR / range | Level scales with price | Divide by long-window ATR | Ratio centered near 1, stable | | Bounded oscillator (RSI etc.) | Within-bounds distribution shifts | Rolling rank within W=252+ | Threshold percentile stable across windows | | Unbounded momentum (ROC, MOM) | Variance non-stationary | Divide by rolling return std | Rolling std stable | | Volume | Multiplicative drift | Log + rolling-mean subtract | Rolling mean stable, histogram by window | | Spreads / rates | Mixed regime / drift | Difference or rolling z (depending on test) | KPSS pass, ADF reject | | Higher moments (skew, kurtosis) | Doubly non-stationary | Rolling estimate + rolling z | Rolling mean and std stable | The matrix is operational, illustrative. Each strategy needs verification on its specific feature set; the matrix gives the starting recipe. ## Visualizing the transformation effect ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/05/article_61-visual-0.png) ## KEY POINTS - Raw indicator distributions drift over time even when the indicator looks bounded or well-defined. EURUSD 14-day RSI median moves from 49.8 in 1995-1999 to 51.9 in 2020-2024; the same threshold rule selects different fractions of trading days across windows. - The transformation pipeline has four steps: diagnose with rolling stats and KS by window, identify the violation type (mean drift, variance drift, distributional shape change, bounded-but-shifting), apply the matching transformation, verify with the same diagnostics on the transformed feature. - Eight indicator classes, eight recipes. Raw price gets log-relative transform. MA crossovers (MACD) get ATR scaling. ATR and range get long-window ratio scaling. Bounded oscillators (RSI, Stochastics) get rolling rank. Unbounded momentum (ROC) gets vol normalization. Volume gets log + rolling-mean subtract. Spreads get differencing or rolling z depending on the test. Higher moments get two-stage rolling estimate + rolling z. - The match between violation type and transformation determines the result. The wrong transformation does not stationarize and may destroy signal. Apply the minimum needed and verify. - Three verification checks: rolling mean and std of transformed feature stable across the sample, histograms by 4-5 year window approximately equal (KS p > 0.05), threshold percentile stable across windows. - Window length W is the most common parameter and the most common failure mode. Match W to the regime timescale, not the strategy's holding period. Avoid optimizing W against the backtest. Default to 252-1000 days for slow drift, 60-100 days only when fast regime detection is the priority. - Causal pipeline is mandatory. At every time t, the transformation parameters must use only data up to t-1\. Full-sample-mean normalization is look-ahead bias and contaminates the backtest. The leakage-detection step in ["Why OOS Failure Is Often a Stationarity Failure"](https://aligrithm.com/why-oos-failure-is-often-a-stationarity-failure/) applies. - Anti-pattern: applying every transformation to every indicator "to be safe". Each transformation has a cost. Verify with the diagnostic checks before stacking transformations. - Anti-pattern: choosing transformation by visual smoothness. Smoothness is not stationarity. Use the rolling-stat and histogram diagnostics, not the eye. - Anti-pattern: forgetting that the transformation window W is a hyperparameter. Treating "W=60 z-score" and "W=252 z-score" as equivalent inflates the degrees-of-freedom count and the IS-OOS gap. - Anti-pattern: assuming the transformed feature retains the raw feature's predictive power. Each transformation trades signal for stationarity. The article ["When Forcing Stationarity Destroys Information"](https://aligrithm.com/when-forcing-stationarity-destroys-information/) covers this trade-off; over-aggressive transformation can destroy the very signal the strategy needed. - The current article gives the recipes. The next article in the publication (["When Forcing Stationarity Destroys Information"](https://aligrithm.com/when-forcing-stationarity-destroys-information/)) gives the limit case where stationarity-forcing eats the signal and the strategy that stationarized too aggressively performs worse than the one that left the raw feature alone. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Testing and Tuning Market Trading Systems - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Testing+and+Tuning+Market+Trading+Systems+Timothy+Masters&ref=aligrithm.com) - [Data Mining Algorithms in C++ - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Data+Mining+Algorithms+in+C%2B%2B+Timothy+Masters&ref=aligrithm.com) ### 3.7 Why Volatility Is More Non-Stationary Than Trend URL: https://aligrithm.com/why-volatility-is-more-non-stationary-than-trend/ Last updated: 2026-07-27T13:24:04.000Z Compute the first-lag autocorrelation of daily log returns on SPX from 1990 to 2024\. The value is approximately -0.04\. Compute the same statistic on the absolute value of daily log returns (a proxy for daily volatility). The value is approximately +0.27\. The two numbers say something structural: returns have almost no day-to-day memory, while volatility has strong day-to-day memory. Compute the variance ratio of the realized vol time series at horizons of 5, 22, 60, and 252 days. All four exceed 1.0, with the longest horizon at approximately 1.8\. The same variance-ratio test on returns hovers near 1.0 at all horizons. Volatility moves are persistent. Return moves are not. This is the core empirical fact that should govern most decisions in a trader's research workflow. Direction is hard to predict because direction is approximately a random walk on most timescales. Volatility is comparatively easy to predict because volatility has long memory. Yet most retail trading literature, technical analysis, and chart-pattern research is dedicated to predicting direction, while the structurally easier prediction (volatility) is treated as a sizing detail rather than the main signal. The article ["Volatility Regimes and Strategy Survival"](https://aligrithm.com/volatility-regimes-and-strategy-survival/) gave the operational gating framework. This article covers the underlying reason that the vol-regime gating decision is the highest-leverage decision in the pillar: vol drifts on a timescale that is short enough to matter for almost any strategy, while trend drifts on a timescale that is too long to easily exploit. ## Measuring the persistence asymmetry Three diagnostics, each pointing to the same conclusion. Diagnostic 1: autocorrelation function. The first-lag autocorrelation of daily SPX returns from 1990 to 2024 is approximately -0.04\. The autocorrelation decays to zero within 1 to 2 lags. The autocorrelation of absolute returns starts at +0.27, remains above +0.15 at lag 10, above +0.08 at lag 22, and only crosses zero around lag 100 (4 to 5 months). The integrated autocorrelation of vol exceeds the integrated autocorrelation of returns by a factor of 50 to 100. $$ \\text{ACF}\_r(k) = \\frac{\\text{Cov}(r\_t, r\_{t-k})}{\\text{Var}(r\_t)}, \\qquad \\text{ACF}\_{|r|}(k) = \\frac{\\text{Cov}(|r\_t|, |r\_{t-k}|)}{\\text{Var}(|r\_t|)} $$ The interpretation is direct: today's return tells you almost nothing about tomorrow's return; today's volatility tells you a lot about tomorrow's volatility. Diagnostic 2: variance ratio test. For a stationary process with no autocorrelation, the variance of the k-period sum is k times the 1-period variance. The variance ratio is the empirical k-period variance divided by k times the 1-period variance. $$ \\text{VR}(k) = \\frac{\\text{Var}(X\_t + X\_{t-1} + \\dots + X\_{t-k+1})}{k \\cdot \\text{Var}(X\_t)} $$ For SPX daily returns: VR(5) ≈ 1.02, VR(22) ≈ 1.08, VR(60) ≈ 1.04, VR(252) ≈ 0.95\. All near 1.0, consistent with returns being approximately a random walk on these horizons. For SPX daily realized vol (or absolute returns): VR(5) ≈ 1.4, VR(22) ≈ 1.6, VR(60) ≈ 1.7, VR(252) ≈ 1.8\. The variance ratios climb monotonically with the horizon, consistent with strong long memory. Diagnostic 3: half-life of mean reversion. Fit an AR(1) model to each series. The half-life is the number of bars for the AR(1) shock to decay by half. $$ \\text{half-life} = -\\frac{\\ln 2}{\\ln \\rho}, \\qquad \\rho = \\text{AR(1) coefficient} $$ For SPX daily returns: ρ ≈ -0.04, half-life is undefined (no mean reversion or anti-persistence at one lag). For SPX log realized vol: ρ ≈ 0.95, half-life ≈ 14 days. For SPX log realized vol on monthly data: ρ ≈ 0.85, half-life ≈ 4 months. Volatility shocks decay measurably within weeks to months. Return shocks decay within hours to days (and the small negative autocorrelation is more about microstructure than predictability). ## Volatility has long memory Three structural reasons. Reason 1: information arrival is itself clustered. Volatility scales with the rate of arrival of new information that traders disagree about. Information arrival is bursty (earnings releases, macroeconomic announcements, geopolitical shocks come in clusters with quiet periods between). The vol process inherits the clustering of the information process. Reason 2: volatility creates more volatility. A large move triggers margin calls, position liquidations, hedging flows, and forced rebalancing by passive funds. Each of these creates additional flow that is itself volatile. The mechanism is amplifying rather than damping. A 5-sigma return on day t increases the conditional probability of another large return on day t+1 because the day-t move has set off cascades that take days to unwind. Reason 3: volatility-of-volatility is also non-stationary. The conditional volatility of the vol process itself drifts across regimes. In low-vol regimes, vol clusters tightly around its mean. In high-vol regimes, vol can swing widely from day to day. The two-level non-stationarity (drift in vol mean, drift in vol-of-vol) compounds the persistence. ## Direction has no long memory Three structural reasons that mirror the above. Reason 1: arbitrage damps directional persistence. If returns had a +0.3 first-lag autocorrelation on a daily horizon (vol-like persistence), participants would buy after up days and sell after down days, eroding the autocorrelation until the trade is no longer profitable. The market's no-free-lunch boundary prevents directional persistence from accumulating beyond the boundary set by transaction costs. Reason 2: directional flow is two-sided. A large up move creates buyers (trend followers) and sellers (rebalancing, profit-taking, mean-reverters). The two flows roughly cancel on the next day. A large vol move creates only one type of follow-on flow (more hedging, more margin calls), which is structurally one-sided. Reason 3: directional opinions are diverse. At any moment, the market contains long-horizon investors, short-horizon traders, hedgers, arbitrageurs, and noise traders. Their directional views are heterogeneous. The aggregate direction at the daily horizon is a weighted average of many competing views and the residual is approximately mean-zero noise. Volatility, by contrast, is one-dimensional: everyone agrees that "the market is volatile" or "the market is calm". The agreement amplifies the persistence. The asymmetry is not absolute. Long-horizon momentum and trend following do work, with annualized Sharpes of 0.4 to 0.7 on most asset classes across decades. The article "Equity 12-1 momentum factor decay" (covered in ["Slow Wandering: The Most Dangerous Type of Market Change"](https://aligrithm.com/slow-wandering-the-most-dangerous-type-of-market-change/)) frames the long-term decay of one specific directional anomaly. The point of the asymmetry is operational: directional edges exist, but they are smaller per unit of variance and harder to extract than vol-regime edges. ## Operational implications Five consequences for strategy design and portfolio operation. Implication 1: vol-targeting works. The classical mean-variance position-sizing rule (set position size inversely to forecast volatility) is the simplest application of vol's predictability. Because the forecast vol is reasonably accurate at short horizons, the position-sized portfolio has more stable realized vol than the unscaled portfolio. The article ["Why ATR Normalization Is More Than a Volatility Trick"](https://aligrithm.com/why-atr-normalization-is-more-than-a-volatility-trick/) covered the indicator-level version of this technique. Implication 2: vol-of-vol-targeting is the next-order refinement. A portfolio that targets a fixed vol level but does not adjust for vol-of-vol underestimates risk in unstable-vol regimes. The refined rule scales position size by the inverse of forecast vol times an adjustment for forecast vol-of-vol. The refinement matters most for short-vol strategies where vol-of-vol regime detection is the difference between profitability and ruin. Implication 3: trend-following needs more samples than vol-trading. To detect a directional edge with statistical confidence, the trader needs many independent samples because each sample carries low signal-to-noise. To detect a vol regime, the trader needs few samples because the regime is persistent. A 50-sample directional backtest is approximately worthless; a 50-sample vol-regime classification is approximately reliable. Implication 4: leverage is asymmetric across strategy types. A vol-trading strategy with predictable vol can run at higher leverage than a directional strategy with unpredictable direction, at equal Sharpe, because the vol-trading drawdown distribution is tighter. The asymmetry is masked in standard reporting because both strategies are presented at constant vol target. The Sharpe-equivalent leverage is different. Implication 5: backtest sample-size requirements differ. The article ["Trade-Count Thresholds for Backtest Reliability"](https://aligrithm.com/trade-count-thresholds-for-backtest-reliability/) later in this pillar gives the general framework. The specific lesson here: vol-prediction backtests can validate on shorter samples than direction-prediction backtests because the underlying signal is stronger. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Indicator design implications The article block on indicator engineering (Pillar 2) covered how to construct features. The volatility persistence has direct implications for which features carry the most information. Volatility-derived features (ATR, GARCH-fitted vol, realized vol, range/Parkinson estimators) carry strong predictive information about future volatility. The signal is large enough to overcome estimation noise on samples of 100 to 500 observations. Direction-derived features (momentum, trend strength, signed returns) carry weak predictive information about future direction. The signal is small enough that it requires 1000 to 10000 observations and very careful denoising. The article ["Why Predictive Power Often Lives in the Tails"](https://aligrithm.com/why-predictive-power-often-lives-in-the-tails/) framed one specific case where the directional signal becomes detectable (in the extremes of the indicator distribution). The asymmetry argues for a feature-engineering hierarchy: the strongest features are vol-derived, the moderately strong features are vol-conditional or vol-normalized direction features (e.g., the signed return divided by the contemporaneous ATR), and the weakest features are raw direction features. ## Anti-patterns Five mistakes that follow from ignoring the persistence asymmetry. Anti-pattern 1: building a directional strategy without a vol filter. A momentum strategy without a vol-regime gating rule is on during low-vol regimes (where momentum is too small to overcome costs) and during high-vol regimes (where momentum reverses in fast and abrupt swings). Add the vol filter; the gated strategy has a higher Sharpe than the ungated version on most asset classes. Anti-pattern 2: using a single-horizon vol estimator across all use cases. The right horizon depends on the use case. Position sizing calls for a fast estimator (5 to 10 days) to track current vol. Strategy gating calls for a slower estimator (22 to 60 days) to identify regime. Tail-risk hedging calls for the longest estimator that still tracks the regime. A single 22-day vol used everywhere underestimates the regime in transitions and overestimates short-term sizing precision. Anti-pattern 3: assuming vol forecasts have the same error distribution as direction forecasts. Vol forecast errors are right-skewed (vol can spike but cannot go below zero) and the forecast error variance scales with the level. Direction forecast errors are approximately symmetric but with much higher variance per unit time. Position-sizing rules need to be calibrated to the right error distribution; using a Gaussian approximation for vol forecast errors underestimates the tail risk. Anti-pattern 4: cross-asset vol comparisons without scale correction. SPX 14% annualized vol, EURUSD 8% annualized vol, and crude oil 28% annualized vol are not directly comparable because the underlying processes have different return distributions. Vol-regime classification needs to be done in each asset's own scale (rolling percentiles, z-scores against the asset's own history) rather than in absolute units. Anti-pattern 5: assuming GARCH parameters are stationary across regimes. The GARCH persistence parameter (the AR coefficient on conditional variance) is itself non-stationary across decades. The 1990s SPX GARCH-fitted persistence is different from the 2010s. Re-fit the model on rolling windows; do not assume the model parameters are constants. ## Decision matrix | Use case | Vol estimator | Direction estimator | Notes | | --------------------- | ---------------------------- | ------------------------------------ | ------------------------------------------------------------- | | Position sizing | 10-day realized | None | Use inverse vol scaling; refine with vol-of-vol if available | | Strategy gating | 22-60 day realized | None | Use thresholds with hysteresis from prior article | | Tail-risk hedging | 60-252 day realized | None | Match hedge horizon to risk horizon | | Trend following | 252-day realized for filter | 12-1 momentum on returns | Disable trend when vol below floor | | Mean reversion | 22-day realized for gating | 5-day reversal on returns | Disable reversion when vol above ceiling | | Vol-carry | All horizons (regime detect) | None | Implied-realized spread is the signal; vol-of-vol is the gate | | Volatility prediction | GARCH(1,1) or HAR | None | Vol persistence makes this tractable; refit annually | | Direction prediction | Vol-conditional features | Many candidates, low signal-to-noise | Combine with vol filter, accept low IR | The matrix maps the use case to the dominant signal type. The pattern: vol estimators dominate, direction estimators are supplementary, and strategies that can be expressed as vol-prediction or vol-conditional are typically more reliable than pure direction strategies. ## Visualizing the persistence asymmetry ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/05/article_60-visual-0.png) ## KEY POINTS - Direction is approximately a random walk on daily-to-monthly horizons. First-lag autocorrelation of SPX daily returns is approximately -0.04\. Variance ratios at 5, 22, 60, 252 days hover near 1.0. - Volatility has strong long memory. First-lag autocorrelation of SPX absolute daily returns is approximately +0.27\. Variance ratios climb from 1.4 at 5 days to 1.8 at 252 days. The half-life of vol shocks is 2-4 weeks at daily resolution and 4 months at monthly resolution. - Three structural reasons for vol persistence: information arrival is bursty, volatility creates volatility through forced flows (margin calls, hedging, rebalancing), vol-of-vol is itself non-stationary so the persistence compounds. - Three structural reasons direction lacks persistence: arbitrage damps directional autocorrelation to the boundary set by transaction costs, directional flow is two-sided (buyers and sellers respond to the same move), directional opinions are heterogeneous and the aggregate residual is mean-zero. - The asymmetry is operational. Vol-targeting works because vol forecasts are accurate enough at short horizons. Trend-following and momentum work because directional edges exist but they are smaller per unit variance than vol-regime edges and require many more samples to validate. - Operational consequence: a 50-sample directional backtest is approximately worthless. A 50-sample vol-regime classification is approximately reliable. Sample-size requirements differ by an order of magnitude between vol-prediction and direction-prediction strategies. - Feature engineering hierarchy: vol-derived features carry strong predictive information, vol-conditional or vol-normalized direction features (e.g., signed return / ATR) carry moderate information, raw direction features carry weak information. - Anti-pattern: building a directional strategy without a vol filter. The same momentum signal is profitable in some vol regimes and loss-making in others; the gated version has a higher Sharpe. - Anti-pattern: a single-horizon vol estimator across all use cases. Position sizing calls for 10 days, gating for 22-60 days, tail-risk hedging for 60-252 days. One window does not serve all. - Anti-pattern: cross-asset vol comparisons without scale correction. SPX, EURUSD, and crude have different return distributions; vol-regime classification must be done in each asset's own scale. - Anti-pattern: assuming GARCH or HAR persistence parameters are stationary. The persistence itself drifts across decades. Refit on rolling windows. - The current article gives the empirical fact and structural reasons. The next article in the publication (["How to Make Indicators More Stationary"](https://aligrithm.com/how-to-make-indicators-more-stationary/)) covers the toolset for transforming raw features into stationary inputs that strategies can consume reliably. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Testing and Tuning Market Trading Systems - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Testing+and+Tuning+Market+Trading+Systems+Timothy+Masters&ref=aligrithm.com) - [Data Mining Algorithms in C++ - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Data+Mining+Algorithms+in+C%2B%2B+Timothy+Masters&ref=aligrithm.com) ### 3.6 Volatility Regimes and Strategy Survival URL: https://aligrithm.com/volatility-regimes-and-strategy-survival/ Last updated: 2026-07-27T13:24:04.000Z A diversified multi-strategy book runs four strategies side by side: equity mean reversion (5-day reversal on the SPX 500 universe), equity trend following (12-1 momentum on the same universe), short-volatility carry (selling SPX 30-delta strangles weekly), and FX carry (long G10 highest-yielders, short lowest-yielders). The book is sized at 10% annualized volatility target. Across calendar 2017 to 2024 the realized P&L per strategy looks very different by year, with the volatility regime each year was in explaining almost all of the difference. In 2017 (full year, SPX realized vol approximately 7%), mean reversion delivered Sharpe 1.4, trend following Sharpe 0.2, short vol Sharpe 2.6, FX carry Sharpe 1.1\. In 2018 (SPX realized vol approximately 17%, with the February vol spike and Q4 selloff), mean reversion Sharpe -0.4, trend following Sharpe 0.6, short vol Sharpe -2.8 (the year of the XIV blow-up), FX carry Sharpe 0.0\. In 2019 (vol approximately 12%), mean reversion 0.8, trend following 0.7, short vol 1.6, FX carry 0.4\. In 2020 (vol approximately 35%, Covid spike), mean reversion 1.2 (the rebound was a giant mean-reverting move), trend following 1.5 (sustained directional moves in many assets), short vol -3.5 (the March vol spike was a 5-sigma event for the strategy), FX carry -1.2\. In 2021 (vol approximately 14%), mean reversion 0.6, trend following 0.4, short vol 1.9, FX carry 0.3\. In 2022 (vol approximately 22% with persistent uptrend then drawdown), mean reversion -0.1, trend following 1.8, short vol 0.0, FX carry 1.4 (the dollar trend was strong). In 2023 (vol approximately 13%), mean reversion 0.7, trend following 0.5, short vol 1.7, FX carry 0.6\. In 2024 (vol approximately 12%), mean reversion 0.6, trend following 0.4, short vol 1.5, FX carry 0.5. The lifetime Sharpe of each strategy is set first and foremost by how the realized volatility regime distribution stacks up against the strategy's regime preference. Short vol is the most extreme example: 2017 alone delivered enough P&L to cover 2018 and 2020 combined, and the lifetime Sharpe across 2017 to 2024 is approximately 0.5, not the 2.0 that the median year suggests. The article ["Why Systems Work Until They Don't"](https://aligrithm.com/why-systems-work-until-they-dont/) framed the four mechanisms of strategy decay; this article zooms in on the most operationally important regime axis (volatility) and shows how to gate strategies on that axis. The next article in the publication (["Why Volatility Is More Non-Stationary Than Trend"](https://aligrithm.com/why-volatility-is-more-non-stationary-than-trend/)) covers the empirical fact that makes volatility-regime gating the highest-leverage decision in this pillar. ## Defining the volatility regime Three regime axes, each measurable in real time. Axis 1: realized volatility level. The 22-day or 60-day annualized standard deviation of daily returns. Three rough levels for SPX (each market has its own scale): low vol below 10% annualized, normal vol 10 to 18%, high vol above 18%. Crisis regimes (>30%) are in the long tail of the high-vol distribution. $$ \\sigma\_t^{(\\text{realized})} = \\sqrt{252} \\cdot \\sqrt{\\frac{1}{N-1} \\sum\_{i=0}^{N-1} (r\_{t-i} - \\bar{r}\_t)^2}, \\qquad N = 22 \\text{ or } 60 $$ Axis 2: volatility-of-volatility. The standard deviation of the rolling realized vol estimate itself, on a longer window. High vol-of-vol regimes are unstable (vol can spike or collapse); low vol-of-vol regimes are stable. Vol-of-vol matters because it determines whether short-vol strategies will be hit by a sudden vol expansion. Axis 3: implied versus realized vol spread. The 30-day SPX implied volatility (VIX) minus the 22-day realized vol. The spread is positive on average (insurance premium) and widens before crises and tightens during low-stress regimes. The spread is the headline P&L driver for short-vol strategies and a useful regime signal for the broader market. A fourth axis (the volatility surface curvature, the term structure of VIX, the volatility skew) is useful at the high-frequency strategy level but optional for first-order regime classification. ## Mapping strategies to regime preference Each strategy class has a preferred vol regime where it earns its full edge and a hostile regime where it loses money structurally. Equity mean reversion (5-day reversal, pairs trading, statistical arbitrage). Lives in low-to-normal vol regimes with weak trends. Earns from short-term overreaction and noise. In high-vol or trending regimes, the "overreaction" is real information and the strategy fades real moves until it gets stopped out. Hostile regime: high vol with directional persistence. Equity trend following (12-1 momentum, breakout systems). Lives in normal-to-high vol regimes with directional moves. Earns from sustained directional persistence. In low-vol regimes the directional moves are too small to overcome transaction costs and stops. In choppy high-vol with no direction the strategy whipsaws. Hostile regime: low vol with no trend, or high vol with chop. Short volatility (selling SPX strangles, VIX futures shorts, vol-carry overlays). Lives in low-to-normal vol regimes that are stable. Earns the implied-realized vol premium. In vol-expansion regimes the strategy can lose 30 to 50% in a single week or month. The 2018 February spike, the 2020 March spike, and the multiple smaller spikes in 2022 to 2024 are the canonical hostile events. The article ["Why Profit Factor Can Lie"](https://aligrithm.com/why-profit-factor-can-lie/) later in this pillar covers the issue that short-vol headline statistics often hide. FX carry (long high-yielders, short low-yielders). Lives in low-to-normal vol regimes with stable risk appetite. Earns interest-rate differential. In high-vol regimes the carry currencies (high-yielders are typically EM or commodity-linked) sell off violently. The carry P&L tends to "go up the stairs and down the elevator" because vol regimes flip from low to high quickly. Cross-sectional equity factors (value, quality, low-vol, momentum). Each factor has its own vol-regime preference. Quality and low-vol perform best in high-vol regimes (defensive bid). Value performs best in transitions. Momentum performs worst at vol-regime turning points (the "momentum crashes" of 2009 and 2020 are the canonical examples). The mapping is empirical, calibrated from history rather than derived from first principles. The point of having an explicit mapping is operational: when the live volatility regime moves out of the strategy's preferred regime, the strategy's allocation should be reduced or zeroed. ## Real-time regime classification Three classifiers, with increasing sophistication. Classifier 1: deterministic threshold. Compute the 22-day realized vol. Map to a regime label by fixed thresholds (e.g., low if below the 33rd percentile of the long-run distribution, normal between 33rd and 67th, high above 67th). Cheap, transparent, reasonable for first-pass gating. Weakness: sharp transitions between regimes can produce frequent flips at the threshold. Classifier 2: smoothed threshold with hysteresis. Same as classifier 1 but with two thresholds per boundary (a higher threshold to enter "high vol" and a lower threshold to exit). The hysteresis prevents flip-flopping at the boundary. The thresholds are calibrated against the historical regime distribution. Classifier 3: regime-switching model. Fit a Markov-switching model to the realized vol time series. The model produces a posterior probability of being in each regime at each time. Use the posterior probability as a continuous gating signal rather than a binary classifier. The article ["How to Make Indicators More Stationary"](https://aligrithm.com/how-to-make-indicators-more-stationary/) later in this pillar covers the broader use of regime-switching machinery. The choice depends on the strategy. Short-vol strategies benefit from the fastest classifier with the lowest false-positive rate (a missed regime shift is a 30 to 50% loss). Mean-reversion and trend-following strategies tolerate a slower classifier with hysteresis (a false positive costs the strategy a few weeks of edge; a missed regime shift costs roughly 20 to 30 basis points of underperformance per month). ## Operational gating rules Five rules that the diversified-book example above suggests. Rule 1: short-vol strategies disable when the implied-realized vol spread compresses below a threshold. The compression is the warning that the insurance premium is no longer adequate compensation for the tail risk. A compressed spread plus a vol-of-vol uptick is the signature of an imminent vol expansion. Rule 2: trend-following strategies disable in low-vol regimes below a threshold. The directional moves are too small to overcome transaction costs and stops. Re-enable when the realized vol crosses back above the threshold sustainedly (with hysteresis). Rule 3: mean-reversion strategies disable in high-vol regimes above a threshold. The "overreaction" in high vol is real information and the reversion thesis fails. Re-enable when realized vol drops below the threshold sustainedly. Rule 4: FX carry strategies disable when global cross-asset vol exceeds a threshold (VIX above 25 or equivalent). The carry-currency drawdown in high-vol regimes is structural and large. Re-enable when global vol normalizes. Rule 5: at the portfolio level, do not let any one strategy carry more than 30% of the active risk budget when the regime favors it. The regime-favorable sizing is tempting (the in-favor strategy earns the most P&L per unit risk in its regime) but creates portfolio-level fragility when the regime turns. Maintain regime-diversified weights even when one regime is the current state. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## The vol regime distribution and its asymmetry The realized-vol distribution is right-skewed. Most days are low or normal vol; the tail is in the high-vol regime. Across SPX 1990 to 2024, approximately 40% of days are low vol (below 10% annualized), 45% are normal (10 to 18%), 12% are high (18 to 30%), 3% are crisis (above 30%). The fraction of total return variance contributed by the high-vol and crisis days is much larger than the fraction of days, because variance scales with squared returns. $$ \\frac{\\sum\_{t \\in \\text{high vol}} r\_t^2}{\\sum\_{t \\in \\text{all days}} r\_t^2} \\approx 50\\% \\text{ from } 15\\% \\text{ of days} $$ The asymmetry has two operational implications. First, a strategy that is on during high-vol days and off during low-vol days is exposed to most of the year's risk in a small number of days. Second, the in-sample vs out-of-sample regime mismatch (covered in ["Why OOS Failure Is Often a Stationarity Failure"](https://aligrithm.com/why-oos-failure-is-often-a-stationarity-failure/)) matters most when the IS and OOS differ in their high-vol-day count, because most of the variance is concentrated in those days. ## Anti-patterns Five mistakes specific to volatility-regime management. Anti-pattern 1: assuming the realized vol distribution is stationary. The unconditional vol distribution drifts across decades (1990s averaged 14% SPX vol, 2000 to 2008 averaged 18%, 2010 to 2019 averaged 14%, post-2020 has averaged 18%). Threshold rules calibrated on one decade may be wrong on another. The article ["Why Volatility Is More Non-Stationary Than Trend"](https://aligrithm.com/why-volatility-is-more-non-stationary-than-trend/) covers this drift in detail. Anti-pattern 2: using a single vol threshold across all strategies. Each strategy has a different regime preference and a different threshold. A unified "high vol = risk off" rule reduces the diversified-book Sharpe by averaging across regime-mismatched strategies. Each strategy needs its own gating rule. Anti-pattern 3: short vol presented as a high-Sharpe strategy without the tail. The headline Sharpe of short vol across normal-vol years is 1.5 to 2.5\. The lifetime Sharpe across the full vol-distribution including the tails is 0.3 to 0.7\. The headline number is correct conditional on the regime; the lifetime number is correct unconditionally. Strategy evaluation needs the unconditional number. Anti-pattern 4: vol gating as a substitute for sizing. A strategy that is on in its favorable regime and off in its hostile regime still needs to be sized at the portfolio level. The favorable regime can produce 5-sigma-of-the-strategy moves that swamp the portfolio if the position is too large. Gating reduces the worst-regime exposure but does not eliminate it. Anti-pattern 5: backtesting a strategy that is implicitly gated by survivor bias. The strategy "happened to be off" during the 2018 vol spike or the 2020 March spike because the parameters were tuned on a window that excluded those events. The backtest reports a strong Sharpe; the live deployment will be on during the next equivalent event. Verify the gating rule is causal (uses only past data) and that the backtest applies the same gating across the entire history including hostile regimes. ## Decision matrix | Strategy class | Preferred regime | Hostile regime | Gating rule | | ---------------------------------- | --------------------------------- | ---------------------------------- | --------------------------------------------------------------- | | Equity 5-day reversal | Low-to-normal vol, weak trend | High vol with persistent direction | Disable when realized vol > 22% | | Equity 12-1 momentum | Normal-to-high vol with direction | Very low vol or whipsaw high vol | Disable when realized vol < 8% | | Short SPX strangles | Low-to-normal vol, low vol-of-vol | Vol expansion | Disable when (implied - realized) < 1% or vol-of-vol > 80th pct | | FX G10 carry | Low global vol, stable risk | High global vol, risk-off | Disable when VIX > 25 | | Cross-sectional value | Transition regimes | Persistent low-vol bull | Continuous, soft gating | | Cross-sectional momentum | Trending regimes | Vol-regime turning points | Reduce at vol-regime transitions | | Quality / low-vol factor | High vol (defensive bid) | Low vol (chasing trash) | Continuous, soft gating | | Trend-following CTA on multi-asset | Multi-asset regime divergence | Sustained low-vol risk-on | Continuous, with vol filter on each asset | The matrix is operational, illustrative thresholds. Each portfolio needs its own calibration against its own historical sample and risk budget. ## Visualizing strategy survival across vol regimes ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/05/article_59-visual-0.png) ## KEY POINTS - Strategy lifetime P&L is set first and foremost by the volatility regime distribution the strategy sees during deployment. The favorable-regime Sharpe is the brochure number; the lifetime Sharpe is the regime-distribution-weighted average and is typically much lower. - Three regime axes are measurable in real time: realized volatility level (22- or 60-day annualized std), vol-of-vol (rolling std of the realized vol estimate), implied-minus-realized spread (VIX minus 22-day realized). - Mapping strategies to regime preference: equity mean reversion lives in low-to-normal vol with weak trend; equity trend following lives in normal-to-high vol with direction; short volatility lives in low-to-normal vol with low vol-of-vol; FX carry lives in low global vol with stable risk appetite. - Each strategy has a hostile regime where it loses money structurally. Mean reversion in high vol with persistent direction. Trend following in low vol or whipsaw high vol. Short vol in vol-expansion events. FX carry in cross-asset vol spikes. - Real-time regime classification: deterministic thresholds with hysteresis are sufficient for slow-moving strategies; Markov-switching posterior probabilities are appropriate for fast-moving short-vol strategies where missed regime shifts produce 30 to 50% losses. - Five operational gating rules: disable short vol when the implied-realized spread compresses, disable trend-following in very low vol, disable mean reversion in high vol with persistent direction, disable FX carry when global VIX exceeds 25, cap any single strategy at 30% of active risk budget regardless of current regime favorability. - The realized-vol distribution is right-skewed. Across SPX 1990 to 2024, low-vol days are 40% of the sample, normal 45%, high 12%, crisis 3%. The high-vol and crisis days contain approximately 50% of the squared-return variance. Strategy exposure during these days drives most of the lifetime P&L variance. - Anti-pattern: assuming the unconditional vol distribution is stationary. Vol-distribution drift across decades shifts threshold-rule calibration. The next article in the publication (["Why Volatility Is More Non-Stationary Than Trend"](https://aligrithm.com/why-volatility-is-more-non-stationary-than-trend/)) covers this in detail. - Anti-pattern: short-vol headline Sharpe (1.5 to 2.5 in favorable years) presented as the unconditional Sharpe (0.3 to 0.7 across full distribution including tails). Strategy evaluation needs the unconditional number. - Anti-pattern: vol gating as a substitute for sizing. Gating reduces hostile-regime exposure but does not eliminate it. Position sizing must still cap the portfolio impact of a single strategy's worst-regime loss. - The current article gives the operational regime-gating framework. The article ["Why Volatility Is More Non-Stationary Than Trend"](https://aligrithm.com/why-volatility-is-more-non-stationary-than-trend/) covers the empirical fact that makes vol gating the highest-leverage decision in this pillar. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Testing and Tuning Market Trading Systems - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Testing+and+Tuning+Market+Trading+Systems+Timothy+Masters&ref=aligrithm.com) - [Data Mining Algorithms in C++ - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Data+Mining+Algorithms+in+C%2B%2B+Timothy+Masters&ref=aligrithm.com) ### 3.5 Why OOS Failure Is Often a Stationarity Failure URL: https://aligrithm.com/why-oos-failure-is-often-a-stationarity-failure/ Last updated: 2026-07-27T13:24:03.000Z A researcher builds an SPX overnight-gap reversion strategy. Hold-out scheme: train on 1995 to 2014, validate on 2015 to 2019, deploy in 2020\. In-sample Sharpe 1.4\. Validation Sharpe 1.1\. Live 2020 to 2025 Sharpe 0.2\. The researcher's first instinct is the textbook diagnosis: the strategy was overfit to the 1995 to 2014 window, the validation period was too short to detect it, the strategy now reveals its true near-zero edge. The researcher decommissions the strategy, posts a "I should have used CSCV and walk-forward" lesson on social media, and moves on. The diagnosis is wrong. Run the same strategy with the same parameters on a different out-of-sample slice (1990 to 1994 plus 2007 to 2008): Sharpe 1.3\. Run it on 2010 to 2014 OOS only (with training on 1990 to 2009): Sharpe 1.2\. Run it on 2020 to 2025 with training on any window that does not overlap 2020 to 2025: Sharpe near 0.2\. The strategy is not overfit. The strategy worked on every window before 2020 and stopped working on 2020 to 2025\. The cause is not the researcher's process; the cause is the regime change between 2019 and 2020 (microstructure shift in overnight session, the increase of options-driven gamma flows, the persistent retail-driven gap fading). The 2015 to 2019 validation worked because that window is in the same regime as 1995 to 2014\. The 2020 to 2025 deployment failed because that window is a different regime. This is the most common form of OOS failure in practical research, and the most often misdiagnosed. The researcher attributes the failure to a fixable methodology problem (overfitting) and rebuilds the methodology in ways that do not address the actual cause (non-stationarity between IS and OOS). The next iteration of the strategy looks more rigorous and fails for the same reason. The article ["Stationarity: The Word Every Trader Ignores Until It Kills the Strategy"](https://aligrithm.com/stationarity-the-word-every-trader-ignores-until-it-kills-the-strategy-2/) framed the broad case for stationarity discipline. This article applies the framework to the specific question of OOS failure: when a strategy fails OOS, how often is it a stationarity problem versus an overfitting problem, and what diagnostic distinguishes the two. ## The four causes of OOS failure A strategy that performs worse OOS than IS has four candidate explanations. Each has a different fix. Cause 1: parameter overfit. The strategy has too many degrees of freedom relative to the IS sample, the optimizer found IS-specific noise, the OOS performance reflects the underlying near-zero true edge. Fix: reduce parameters, regularize, use CSCV, run permutation tests on the IS Sharpe. The article ["Degrees of Freedom in Trading Systems"](https://aligrithm.com/degrees-of-freedom-in-trading-systems/) later in this pillar covers the parameter-overfit framework in detail. Cause 2: data leakage / look-ahead. The strategy used information at training time that it would not have at trading time (future data, in-sample-fitted normalizations applied to OOS, post-hoc-adjusted prices). Fix: rebuild the data pipeline so every transformation is causally clean, with rolling estimators that use only past data. Cause 3: insufficient sample. The IS sample produced a Sharpe estimate with high standard error; the realized OOS performance is within the noise band of the IS estimate. The strategy was never as good as it looked, and the OOS produced a lower draw from the same distribution. Fix: more data, longer track record, lower expected effect-size threshold for accepting a strategy. The article ["Trade-Count Thresholds for Backtest Reliability"](https://aligrithm.com/trade-count-thresholds-for-backtest-reliability/) covers the sample-size question. Cause 4: regime change between IS and OOS. The strategy was correct under the IS regime and is incorrect under the OOS regime. The data-generating process changed in some material way (microstructure, macro regime, participant mix, volatility level). The strategy is well-fit to a regime that the OOS is no longer in. The label "overfit" miscategorizes the failure mode. Fix: diagnose the regime change, decide whether to retrain on recent data, decide whether the regime change invalidates the strategy class outright. The four causes look identical when you only see the IS Sharpe and the OOS Sharpe. They are distinguishable when you look at the right diagnostics. Most published OOS failure cases are diagnosed with cause 1 because parameter overfit is the canonical methodology failure mode and the easiest to test for. In practical research, cause 4 is at least as common, and the false attribution to cause 1 produces no useful corrective action. ## The diagnostic flow A simple decision tree distinguishes the four causes. $$ \\begin{array}{l|l|l} \\text{Test} & \\text{Outcome} & \\text{Implies} \\\\ \\hline \\text{Permutation test on IS Sharpe} & \\text{not significant} & \\text{cause 1: overfit} \\\\ \\text{Permutation test on IS Sharpe} & \\text{significant} & \\text{not cause 1} \\\\ \\text{IS pipeline audit} & \\text{look-ahead found} & \\text{cause 2: leakage} \\\\ \\text{IS pipeline audit} & \\text{clean} & \\text{not cause 2} \\\\ \\text{IS sample sufficiency} & N < 100 \\text{ trades} & \\text{cause 3: noise} \\\\ \\text{IS sample sufficiency} & N > 200 \\text{ trades} & \\text{not cause 3} \\\\ \\text{Regime overlap test (IS, OOS)} & \\text{overlap low} & \\text{cause 4: regime} \\\\ \\text{Regime overlap test (IS, OOS)} & \\text{overlap high} & \\text{re-examine} \\\\ \\end{array} $$ The flow is sequential. If the permutation test rejects the null (the IS Sharpe is significant under the no-edge null), cause 1 is unlikely. If the pipeline audit finds no look-ahead, cause 2 is unlikely. If the IS sample is large enough, cause 3 is unlikely. If the regime overlap between IS and OOS is low, cause 4 is the leading explanation. The order matters because each test is cheap and each rules out a candidate cause cleanly. The permutation test mechanics: shuffle the timing of the strategy's returns thousands of times to break any predictive structure, recompute the IS Sharpe on each shuffle, place the realized IS Sharpe in the resulting null distribution. A realized Sharpe in the top 5% rejects the no-edge null. The article ["Permutation Tests for Indicator Significance"](https://aligrithm.com/permutation-tests-for-indicator-significance/) covers the technique in detail. The regime overlap test mechanics: characterize each window (IS and OOS) by a vector of macro and microstructure features (rolling realized vol, rolling autocorrelation, rolling correlation to other assets, rolling effective spread). Compute the distance between the IS feature distribution and the OOS feature distribution. Standard tools: Kolmogorov-Smirnov on each feature individually, energy distance for the joint distribution, or a simple sliced-Wasserstein. Low overlap (KS p-value below 0.05 on multiple features, or large joint distance) is the regime-mismatch signature. ## The IS / OOS regime overlap test A practical implementation. Define the regime feature vector at time t. $$ F\_t = \\bigl(\\,\\text{vol}\_t,\\; \\text{ac}\_t,\\; \\rho\_t,\\; \\text{spread}\_t,\\; \\text{trend}\_t,\\; \\dots\\,\\bigr) $$ Each component is a rolling-window estimator of a stable structural feature. Realized volatility on 252-day window. First-lag autocorrelation of daily returns on 252-day window. Rolling correlation to a broad reference (SPX or AGG) on 252-day window. Effective spread per round-trip. Trend strength (the absolute t-statistic of the daily-return mean over a 252-day window). Add or subtract features depending on what the strategy depends on. For each feature, compute the empirical distribution across the IS window and across the OOS window separately. Run KS or Anderson-Darling on the two distributions. Aggregate the test statistics or, more practically, plot the two distributions overlaid and look at the visual overlap. A clean IS/OOS regime overlap shows distributions with strong overlap on every feature. A regime-mismatch shows visible separation on at least one feature. For the SPX overnight-gap example above: the relevant features are overnight realized volatility, the autocorrelation of overnight returns, and the share of total variance contributed by the overnight session. All three shift visibly between 2015 to 2019 and 2020 to 2025 (overnight-session variance share rose, overnight autocorrelation flipped sign in the gap-reversion direction). The KS test rejects equality of distributions on each of these features at p < 0.001\. The strategy is operating in a different regime from the one it was tuned on. ## Regime overlap, in practice Three operational interpretations. Interpretation 1: the OOS performance is informative about the strategy's robustness only when the IS and OOS regimes overlap. A strategy that performs well on a high-overlap OOS slice has been validated against one realization of its target regime. A strategy that performs well on a low-overlap OOS slice has been validated against a different regime than the one it was built for, which is a different and weaker claim. Interpretation 2: live-deployment risk depends on the overlap between the historical regime distribution and the live-period regime. A strategy that was trained on a wide regime distribution and validated on a wide overlapping distribution has lower regime risk in deployment. A strategy trained narrowly faces high regime risk if the live period is outside the training distribution. Interpretation 3: the proper denominator for "OOS validation" is the regime distribution, not the calendar. "Five years OOS" is a calendar measure that may cover one regime or six. The right OOS specification is "OOS coverage spans low-vol, high-vol, trending, and mean-reverting regimes with at least N samples in each", which is the topic of the article "Regime Coverage: Why Your Backtest Needs Different Market States" later in this pillar. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Regime mismatch as the diagnosis Three actions when the diagnosis lands on cause 4. Action 1: do not deploy the strategy. The IS performance was real for the IS regime, the OOS performance is real for the OOS regime, and the live regime is the OOS regime. The strategy works for a regime that is not the one currently operating. Deploying it is funding a strategy that is mismatched to the live environment. Action 2: retrain on the OOS regime data. If the strategy class is robust enough that retraining on the new regime produces meaningful parameters, the right action is to use the OOS window as the new IS and re-validate on a fresh OOS that is also in the new regime (if available). The retrained strategy is the same strategy class with new parameters appropriate to the live regime. The article ["Why Walk-Forward Testing Is Better Than One Big OOS Split"](https://aligrithm.com/why-walk-forward-testing-is-better-than-one-big-oos-split/) covers the operational machinery. Action 3: abandon the strategy class. If the regime change is structural (microstructure overhaul, persistent macro regime change, participant-mix change that is unlikely to revert), the strategy class itself may not survive. The retrain produces parameters that fit the new regime but the underlying signal is no longer there. The post-mortem is the lesson; the research bandwidth moves to a different strategy class. Distinguishing actions 2 and 3 requires a structural-mechanism understanding beyond pure statistics. Did the regime change because of a permanent structural shift (decimalization, electronic exchange transition, ETF arbitrage growth, persistent low-rate environment) or because of a transient cycle (a particular macro shock, a single-quarter unusual event)? The structural answer drives the action. ## Anti-patterns Five mistakes specific to OOS-failure diagnosis. Anti-pattern 1: every OOS failure attributed to overfitting. The default narrative in published quant research is "the strategy looked good IS, failed OOS, must have been overfit". The narrative is convenient because it implies a fixable methodology problem. The diagnosis is correct in some fraction of cases and wrong in a comparable fraction. Run the diagnostic flow before committing to the diagnosis. Anti-pattern 2: rebuilding the methodology to "be more rigorous" without diagnosing the failure. After an OOS failure, the researcher adds CSCV, more cross-validation folds, more permutation tests, and ships the next strategy with more methodology and the same regime mismatch. The next strategy fails the same way. Rigor on the wrong cause does not fix the right cause. Anti-pattern 3: assuming a longer IS window automatically helps. Doubling the IS window doubles the number of regimes the strategy is fit across, which dilutes the parameter fit across all of them. A strategy that performs well in a single regime may perform mediocrely after being fit on a wide multi-regime window. The right framing is "fit narrowly to a specific regime, validate on the same regime, deploy when the live data is in that regime". The article ["Why More Parameters Make a Strategy Easier to Sell and Easier to Break"](https://aligrithm.com/why-more-parameters-make-a-strategy-easier-to-sell-and-easier-to-break/) covers the related concern about within-strategy degrees of freedom. Anti-pattern 4: choosing the OOS window for convenience rather than regime coverage. The default OOS window in many research frameworks is "the most recent 20% of the data". The window is chosen for cleanness of the time-series split, not for regime coverage. The 20% may all be in one regime, in which case the OOS validates one regime. The proper OOS specification is regime-stratified, not calendar-stratified. Anti-pattern 5: declaring a strategy validated when the IS and OOS regimes are demonstrably different. The IS and OOS Sharpes both look acceptable, the researcher ships. The KS test on the regime feature distributions rejects equality at p < 0.01\. The validation says "the strategy works in regime A and in regime B"; it does not say "the strategy works in regime C", which is what live deployment will face. Acceptable IS and OOS Sharpes across different regimes is a robustness claim, not a same-regime validation. ## Decision matrix | Test result | Most likely cause | Action | | ----------------------------------------- | ------------------------- | ---------------------------------------------------------------- | | IS permutation test fails to reject | Cause 1: overfit | Reduce parameters, regularize, retest | | IS pipeline audit finds look-ahead | Cause 2: leakage | Fix pipeline, retest from scratch | | IS sample N < 100 trades | Cause 3: noise | Collect more data or lower expected effect size | | Regime overlap KS rejects on key features | Cause 4: regime change | Retrain on new regime or abandon strategy class | | Multiple causes simultaneously | Combined failure | Address each cause in the order overfit, leakage, sample, regime | | All tests clean and OOS still failed | Re-examine the test setup | Check for non-overlapping subtle effects | The matrix is operational. Most real OOS failures fall into causes 1, 3, and 4 with cause 4 being the most often misdiagnosed. ## Visualizing the IS/OOS regime mismatch ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/05/article_58-visual-0.png) ## KEY POINTS - Most OOS failures get the textbook diagnosis "the strategy was overfit". The default narrative is convenient because it implies a fixable methodology problem. In practical research, regime change between IS and OOS is at least as common a cause and is the most often misdiagnosed. - Four candidate causes of OOS failure: parameter overfit, data leakage, insufficient sample, regime change. Each has a different fix; the wrong diagnosis produces no useful corrective action. - The diagnostic flow is sequential and cheap. Permutation test on IS Sharpe rules out overfit. Pipeline audit rules out leakage. Sample-size check rules out noise. Regime overlap test (KS or Wasserstein on feature distributions) identifies regime mismatch. - The regime overlap test compares the IS distribution and OOS distribution of structural features (realized vol, autocorrelation, cross-asset correlation, effective spread, trend strength). Significant separation on any feature is a regime-mismatch signature. - The proper denominator for "OOS validation" is the regime distribution, not the calendar. Five years OOS may cover one regime or six. The right specification is regime-stratified, not date-stratified. - When the diagnosis is regime change, three actions follow: do not deploy if the live regime differs from the historical training distribution, retrain on the new regime if the strategy class survives, abandon the strategy class if the regime change is structural. - Distinguishing "retrain on new regime" from "abandon strategy class" requires a mechanism-level understanding, not a statistical one. Structural shifts (microstructure, persistent macro regime, participant mix) point to abandonment; transient cycles point to retraining. - Anti-pattern: rebuilding the methodology with more rigor without diagnosing the failure. Adding CSCV and more cross-validation folds does not fix a regime mismatch. - Anti-pattern: assuming a longer IS window automatically helps. Doubling the IS window dilutes the parameter fit across more regimes. Fit narrowly to a specific regime, deploy when the live data is in that regime. - Anti-pattern: choosing the OOS window for calendar cleanness rather than regime coverage. The default 80/20 split or the most-recent-20% rule may not span the regimes the live deployment will face. - The current article gives the diagnostic flow. The next article in the publication (["Volatility Regimes and Strategy Survival"](https://aligrithm.com/volatility-regimes-and-strategy-survival/)) covers one specific regime axis (volatility) in operational detail, including how to gate strategies on volatility regime in real time. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Testing and Tuning Market Trading Systems - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Testing+and+Tuning+Market+Trading+Systems+Timothy+Masters&ref=aligrithm.com) - [Data Mining Algorithms in C++ - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Data+Mining+Algorithms+in+C%2B%2B+Timothy+Masters&ref=aligrithm.com) ### 3.4 How to Detect When a Trading System Is Dying URL: https://aligrithm.com/how-to-detect-when-a-trading-system-is-dying/ Last updated: 2026-07-27T13:24:02.000Z A futures trend system has been live for 38 months on a portfolio of 12 liquid contracts. The original walk-forward backtest (built on 2008 to 2019 data) projected an annualized Sharpe of 0.85, expectancy of $312 per round-trip lot, win rate of 41%, average win of $1,420, average loss of $620, profit factor of 1.75, and time-in-market of 64%. Live results across the 38 months: Sharpe 0.71 in months 1 to 18, Sharpe 0.43 in months 19 to 30, Sharpe 0.18 in the most recent 8 months. Profit factor has fallen from 1.81 to 1.62 to 1.21\. Expectancy has dropped from $295 to $190 to $58\. Win rate is unchanged at 40 to 42% across the three sub-periods. Average win has shrunk from $1,440 to $1,280 to $980\. Average loss is unchanged at $610 to $640. The system is not breaking randomly. It is dying in a specific, diagnostic way. The win rate is intact, which rules out signal failure. The average loss is intact, which rules out a stop-loss problem or a slippage explosion. The average win is shrinking, which is the signature of trend exhaustion: the trades still trigger correctly and run in the right direction, but the moves are smaller than they used to be. The profit factor and expectancy compression are the downstream consequences of the smaller wins. The article ["Why Systems Work Until They Don't"](https://aligrithm.com/why-systems-work-until-they-dont/) framed this as one of four decay mechanisms (regime drift in the substrate). This article gives the operational diagnostic that distinguishes the four mechanisms in real time, so the trader can act on the right defense rather than the wrong one. A trading system never dies suddenly without warning. It produces several months of measurable signal degradation before the realized P&L collapses. The trader who is monitoring the right metrics catches the decay early and either retrains, halves allocation, or decommissions cleanly. The trader who is watching only the equity curve catches the decay only after the equity curve has already paid for it. The difference is operationally large: most decommission decisions made on equity-curve signals alone are 6 to 18 months too late. ## The diagnostic panel Eight metrics, monitored on rolling windows, with the deployment-era baseline as the reference. Each metric isolates a different aspect of system health. Metric 1: rolling Sharpe ratio. The headline metric and the noisiest. A 12-month rolling Sharpe varies by approximately 0.3 to 0.5 standard deviations even for a stationary strategy with true Sharpe of 0.8\. Use it as a coarse alarm but do not act on it alone. Metric 2: rolling profit factor. Total gross profit divided by total gross loss across the rolling window. More stable than Sharpe because it does not depend on the timing of the gains and losses. A profit factor falling from 1.8 to 1.3 over 12 months is a stronger decay signal than the same change in Sharpe. Metric 3: rolling expectancy per trade. The average P&L per round-trip in dollars or in R-multiples (the ratio of trade P&L to the trade's initial risk). Expectancy decay is the cleanest indicator that the underlying edge is shrinking, because it strips out trade frequency and position sizing. Metric 4: win rate. The fraction of trades that close profitable. A stable win rate during expectancy decay is the trend-exhaustion signature. A falling win rate during expectancy decay points to signal failure. Metric 5: average win and average loss separately. Tracking the two together is cheap and informative. Average loss is roughly bounded by the system's stop-loss rule and should be stable; if it grows, the strategy is missing stops or paying execution slippage. Average win has no upper bound and decays first when the substrate compresses (lower volatility, smaller trends, faster mean reversion). Metric 6: maximum adverse excursion (MAE) per trade. The deepest drawdown the trade experienced before it closed. A growing MAE distribution means trades are sitting at deeper losses before resolving, which signals lower conviction in the entry signal or unstable entry timing. The article ["MAE/MFE Analysis: Seeing What Net Profit Hides"](https://aligrithm.com/mae-mfe-analysis-seeing-what-net-profit-hides/) later in this pillar covers the tool in detail. Metric 7: maximum favorable excursion (MFE) per trade. The largest open profit the trade reached before it closed. A shrinking MFE distribution at constant win rate is the canonical trend-exhaustion signature: the strategy still catches the right direction but the move is shorter. Metric 8: time-in-market and trade frequency. A strategy that suddenly trades less (or more) than its historical baseline is interacting with the market differently. Often this signals that the entry filters are firing under different conditions, which is itself a regime-change diagnostic. ## Calibrating the alarm thresholds A drop in any single metric can be statistical noise. The key question: how large a drop, sustained over how many months, exceeds what stationary noise should produce? For a strategy with true Sharpe of 0.8 and 12 trades per month, the 12-month rolling Sharpe has a standard deviation of approximately 0.4 under stationarity. A one-month dip below 0.4 is unremarkable. A 12-month sustained Sharpe below 0.4 is a 1-sigma event under the null and a strong signal under the regime-change alternative. $$ \\sigma\_{\\hat{\\text{SR}}} \\approx \\sqrt{\\frac{1 + \\text{SR}^2 / 2}{n}}, \\qquad n = \\text{number of independent observations in the window} $$ The formula gives the asymptotic standard error of an estimated Sharpe ratio under the assumption of independent returns. For a daily-return strategy with n = 252 (one year), SR = 0.8, the standard error is approximately 0.07 in annualized units. For a monthly-return aggregation with n = 12 (one year), the standard error is approximately 0.31\. The right threshold for "this is decay, not noise" depends on the window and the underlying volatility of the metric. The complementary tool is the Monte Carlo bootstrap: resample the strategy's historical trade-by-trade P&L distribution, build a 12-month rolling-Sharpe distribution under the null of stationarity, and place the realized rolling Sharpe in that distribution. Persistent residence in the bottom 10% of the bootstrap distribution is decay; a single dip is noise. The article ["Monte Carlo for Trading Systems"](https://aligrithm.com/monte-carlo-for-trading-systems/) later in this pillar gives the bootstrap mechanics. ## Distinguishing the four decay mechanisms The mechanisms (alpha crowding, regime drift, microstructure change, capacity saturation) leave different fingerprints on the diagnostic panel. Alpha crowding fingerprint: average win shrinks (less profit per trade because faster arbitrage), transaction costs rise (slippage on entry/exit grows with crowd size), win rate roughly stable. The strategy still finds the same setups but the post-cost edge is compressing. Regime drift fingerprint: average win shrinks (smaller moves in the new regime), MFE distribution shifts down, transaction costs stable, win rate may be stable or falling depending on whether the entry filter still selects valid setups in the new regime. The substrate is doing different things, the strategy is doing the same thing. Microstructure change fingerprint: a step-change in execution metrics. Effective spread doubles, slippage per trade jumps, fill rates drop, average loss grows because the strategy is paying more on each round-trip. The change is dated; the diagnostic is the date alignment with the venue change. Capacity saturation fingerprint: the headline strategy P&L diverges from a small-size shadow book of the same strategy. The shadow shows expected performance, the live large book shows compressed performance. The compression scales with the live position size relative to available liquidity. The four fingerprints do not always present cleanly. A real failure is often a combination. The decision matrix in the prior article gives the rough mapping; the diagnostic panel here gives the inputs to the decision. ## The kill switches Four decommission triggers, each tied to a specific metric and threshold. The triggers are written into the strategy specification at deployment and are not negotiable in the moment. $$ \\begin{array}{l|l} \\text{Trigger} & \\text{Threshold} \\\\ \\hline \\text{Drawdown trigger} & \\text{Realized DD} > 1.5 \\times \\text{historical max DD} \\\\ \\text{Sharpe trigger} & \\text{12-month rolling SR} < 0.3 \\text{ for 6 consecutive months} \\\\ \\text{Profit-factor trigger} & \\text{12-month rolling PF} < 1.1 \\text{ for 6 consecutive months} \\\\ \\text{CUSUM trigger} & \\text{CUSUM on standardized expectancy crosses } h = 5 \\\\ \\end{array} $$ The thresholds are illustrative. Each strategy needs calibration against its own deployment-era baseline and Monte Carlo distribution. The principle: the trigger is automatic, written down before the strategy is funded, and triggered without discretionary override. The article ["Why Systems Work Until They Don't"](https://aligrithm.com/why-systems-work-until-they-dont/) framed the discretionary-override pitfall; the present article gives the procedural defense. A complementary set of soft triggers reduces allocation rather than decommissioning outright. Halve allocation if the rolling Sharpe falls below 0.5 for 3 months. Halve again if it stays there for 6 months. Decommission at the hard threshold. Tapering by allocation is operationally useful because it lets the strategy continue to provide diagnostic data while reducing the cost of waiting for the hard trigger. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Distinguishing decay from drawdown A normal drawdown on a healthy strategy looks identical to early decay if you only look at the equity curve. The discrimination requires the diagnostic panel. Healthy drawdown signatures: win rate within historical band, average win within historical band, average loss within historical band, profit factor temporarily depressed by a streak of losers, recovery to baseline within the historical recovery time. The Monte Carlo bootstrap places the drawdown well within the stationary distribution. Decay signatures: at least one of (win rate falling, average win shrinking, profit factor compressing) sustained across multiple months, the drawdown exceeds the bootstrap distribution's 90th percentile, the recovery time is longer than the historical maximum, the diagnostic CUSUM has crossed its threshold. The discrimination is not always clean in real time. The first 3 months of a true decay look identical to a normal drawdown. The diagnostic value comes from waiting for the multi-month signature to confirm before acting, while not waiting so long that the cost of acting is high. The trade-off was discussed in detail in ["Slow Wandering: The Most Dangerous Type of Market Change"](https://aligrithm.com/slow-wandering-the-most-dangerous-type-of-market-change/). ## Operational cadence A monthly review process makes the diagnostic panel actionable. The minimum cadence: Weekly: review the live P&L, check for any single-day or single-week anomaly that exceeds the 99th percentile of the historical distribution. Verify execution quality (effective spread, fill rate, markout) is within the historical band. Monthly: refresh the rolling-window metrics, compare each to the bootstrap-implied confidence band, log any metric that has crossed below its band. Check the CUSUM detectors. Update the soft-trigger allocation if any threshold has been breached. Quarterly: full diagnostic panel review. Stratify the realized performance by regime (volatility regime, market mode). Compare to the regime-stratified bootstrap distribution. Decide on retraining if the diagnostics indicate slow drift, on decommissioning if the diagnostics indicate structural break. Annually: re-derive the bootstrap distribution from the longest available trade history. Re-calibrate the alarm thresholds against the updated bootstrap. Audit the post-mortem log of any decommissioned strategies for prior-update lessons. ## Decision matrix | Diagnostic finding | Most likely state | Action | | ------------------------------------- | ------------------------------------ | -------------------------------------------- | | All metrics within bootstrap band | Healthy, normal drawdown if any | Continue, monitor | | Sharpe below band, others within band | Statistical noise on Sharpe | Continue, increase monitoring frequency | | Avg win shrinking, win rate stable | Trend-exhaustion or regime drift | Quarterly retrain, monitor regime indicators | | Avg win shrinking, costs rising | Alpha crowding | Reduce allocation, monitor capacity | | Avg loss growing, win rate stable | Execution problem or microstructure | Audit execution, check venue changelog | | Win rate falling, avg win stable | Signal failure | Decommission, post-mortem signal definition | | Live P&L diverges from shadow book | Capacity saturation | Reduce allocation to shadow level | | CUSUM trigger fires on expectancy | Confirmed decay | Decommission per policy, post-mortem | | Drawdown exceeds Monte Carlo 99th pct | Real failure regardless of mechanism | Decommission per policy | The matrix is operational, not exhaustive. It maps the diagnostic outputs to the action plan, with the article ["Why Systems Work Until They Don't"](https://aligrithm.com/why-systems-work-until-they-dont/) providing the strategic frame and the present article providing the procedural manual. ## Anti-patterns Five mistakes that show up around system-death detection. Anti-pattern 1: equity curve as the only monitor. The equity curve aggregates every metric into one signal that is too noisy and too lagged. By the time the equity curve makes the decay obvious, the diagnostic panel has been flashing for months. Read the metrics, not the curve. Anti-pattern 2: monthly P&L as the test. A single month is too short. A single year is borderline. The monthly P&L of a strategy with true Sharpe 0.8 has a standard deviation of approximately 6% on a 1-vol-target portfolio. A 6% loss month is one standard deviation, which is unremarkable. Decisions should be made on the rolling windows and the bootstrap-calibrated thresholds, not on individual months. Anti-pattern 3: discretionary override of the hard trigger. The trigger fires. The trader looks at the realized P&L, decides "this strategy will recover", and continues funding it. Six months later the strategy has lost more capital than the prior year of P&L. The cure: the trigger is automatic. If the trader wants to override, the override requires written justification and external sign-off. The friction of the process is the defense. Anti-pattern 4: re-tuning the alarm threshold to suppress an alarm. The strategy's CUSUM crosses h=5\. The trader raises the threshold to h=8 and observes that the alarm no longer fires. The alarm has been muted but the underlying decay is the same. The cure: thresholds are set at deployment time and changed only with the same rigor as the original calibration. Anti-pattern 5: monitoring everything but acting on nothing. The trader builds an elaborate monitoring dashboard, watches it daily, and never decommissions a strategy. The dashboard is a comfort blanket, not a decision tool. The cure: every metric on the dashboard has a written threshold and a written action. If the threshold has been crossed and the action has not been taken, the dashboard has failed. ## Visualizing the diagnostic panel ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/05/article_57-visual-0.png) ## KEY POINTS - A trading system never dies suddenly without warning. It produces several months of measurable signal degradation before the realized P&L collapses. The trader who monitors the right metrics catches the decay early; the trader who watches only the equity curve catches it 6 to 18 months late. - The diagnostic panel is eight metrics: rolling Sharpe, rolling profit factor, rolling expectancy, win rate, average win and average loss separately, MAE distribution, MFE distribution, time-in-market and trade frequency. - Each metric isolates a different aspect of strategy health. Win rate isolates signal validity. Average loss isolates execution and stops. Average win and MFE isolate the substrate's trend-exhaustion or compression. Time-in-market isolates entry-filter regime sensitivity. - The four decay mechanisms (alpha crowding, regime drift, microstructure change, capacity saturation) leave distinct fingerprints on the panel. Crowding compresses average win and raises costs. Regime drift compresses average win at stable costs. Microstructure change steps the average loss up and execution metrics down. Capacity saturation diverges the live book from the shadow book. - Alarm thresholds are calibrated against the bootstrap distribution of the diagnostic metrics under the null of stationarity. Persistent residence below the bootstrap 10th percentile is decay; a single dip is noise. - Hard kill switches are written at deployment, not at decommission: drawdown trigger at 1.5x historical max, Sharpe trigger at 0.3 for 6 consecutive months, profit-factor trigger at 1.1 for 6 consecutive months, CUSUM trigger at h=5 on standardized expectancy. - Soft triggers taper allocation rather than decommission outright. Halve allocation at the soft Sharpe threshold; halve again on persistence; decommission at the hard threshold. - Distinguishing decay from drawdown requires the diagnostic panel plus the bootstrap distribution. A normal drawdown looks identical to early decay on the equity curve alone. - Operational cadence: weekly P&L and execution audit, monthly diagnostic panel review, quarterly regime-stratified analysis with retraining decision, annual bootstrap recalibration. - Anti-pattern: the equity curve as the only monitor. Anti-pattern: discretionary override of hard triggers. Anti-pattern: re-tuning the alarm threshold to suppress alarms. Anti-pattern: monitoring everything and acting on nothing. - The current article is the procedural manual that the prior strategic framework (["Why Systems Work Until They Don't"](https://aligrithm.com/why-systems-work-until-they-dont/)) implies. The next article in the publication (["Why OOS Failure Is Often a Stationarity Failure"](https://aligrithm.com/why-oos-failure-is-often-a-stationarity-failure/)) covers the upstream question: why does the OOS performance differ from in-sample, and how often is the answer not overfitting but a regime change between IS and OOS windows. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Testing and Tuning Market Trading Systems - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Testing+and+Tuning+Market+Trading+Systems+Timothy+Masters&ref=aligrithm.com) - [Data Mining Algorithms in C++ - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Data+Mining+Algorithms+in+C%2B%2B+Timothy+Masters&ref=aligrithm.com) ### 3.3 Why Systems Work Until They Don't URL: https://aligrithm.com/why-systems-work-until-they-dont/ Last updated: 2026-07-27T13:24:02.000Z A trading shop runs four strategies in production. Strategy A is an equity-pairs mean-reverter built in 2014 on a basket of 200 large-cap US names. Strategy B is a CTA-style trend follower on the G10 FX block built in 2017\. Strategy C is a vol-carry overlay on SPX index options built in 2019\. Strategy D is an intraday market-making book on a single Tier-1 crypto venue built in 2022\. As of 2025, the four strategies have produced very different lifetime trajectories. Strategy A peaked in 2016 to 2018 (Sharpe 1.4) and decayed through 2019 to 2024 (Sharpe 0.6 by 2023, near-zero in 2024). The shop pulled it in late 2024 after CUSUM on the realized predictive R-squared crossed the decommission threshold. Strategy B has been stable: the realized Sharpe across 2017 to 2025 is 0.7 with no clear trend. The trend-following premium on G10 has not crowded out at the multi-week horizon at which the strategy operates. Strategy C ran beautifully through 2019 to early 2020 (Sharpe 1.8), produced a 22% drawdown in March 2020, recovered, and has run a deteriorating Sharpe (1.0, 0.5, 0.2) through 2022 to 2024 as the SPX implied-vs-realized spread compressed. The shop is keeping it on at a halved allocation while research finishes a vol-skew variant. Strategy D ran for fourteen months at Sharpe 2.4 and then died abruptly when the venue's order-book microstructure changed (a fee-tier overhaul that doubled the effective spread on the strategy's order sizes). The shop pulled it within two weeks. The replacement is in research and may or may not work. Four strategies, four lifecycles. Each was right when it was built, each delivered some edge, each will eventually be replaced. The shop's lifetime P&L depends less on which strategies it found and more on how it managed the inevitable decay of each one. The prior articles in the publication (["Stationarity: The Word Every Trader Ignores Until It Kills the Strategy"](https://aligrithm.com/stationarity-the-word-every-trader-ignores-until-it-kills-the-strategy-2/) and ["Slow Wandering: The Most Dangerous Type of Market Change"](https://aligrithm.com/slow-wandering-the-most-dangerous-type-of-market-change/)) gave the diagnostic machinery for detecting when the substrate is changing. This article generalizes the diagnostic problem to the steady-state portfolio question: how to operate a portfolio of strategies that all eventually die, how to size each one against its expected remaining life, and how to budget research bandwidth around the replacement problem rather than the discovery problem. ## The lifecycle of a trading strategy Four stages, distinguishable by the relationship between realized and expected performance. Stage 1: discovery and validation. The strategy is built, backtested, walk-forward validated, and stress-tested. The realized OOS performance roughly matches the in-sample backtest after honest cost assumptions. The strategy passes the Pillar 3 testing checklist (covered across the rest of this pillar). Duration: typically 3 to 12 months of research. Stage 2: live deployment. The strategy is funded and runs at full target allocation. Realized performance tracks the backtested expectation within the expected statistical band. The trader monitors stationarity diagnostics on the input features, drift detectors on the prediction quality, and standard P&L metrics. Duration: typically 1 to 5 years for a healthy strategy. Stage 3: decay. Realized performance starts deteriorating relative to the deployment-era baseline. The deterioration may be a slow drift (covered in detail in ["Slow Wandering: The Most Dangerous Type of Market Change"](https://aligrithm.com/slow-wandering-the-most-dangerous-type-of-market-change/)) or a sequence of small abrupt shifts (regime breaks, microstructure changes). Drift detectors trigger one or more alarms. The trader takes one of three actions: continue at reduced allocation, retrain on recent data, or begin decommission. Duration: typically 6 months to 2 years. Stage 4: death. The strategy is decommissioned. The decommission may be triggered by a hard alarm (drawdown exceeding policy, OOS Sharpe below threshold for N months, structural change in the substrate that invalidates the model). It may also be a research-driven retirement (a better strategy supersedes the old one). The capital is reallocated to other strategies or held in cash pending the next deployment. Duration: instantaneous (the decommission decision) followed by post-mortem. ## Four mechanisms of decay Each mechanism has a different timescale and a different defense. Diagnosing which mechanism is killing a specific strategy is the hard part. Mechanism 1: alpha crowding. Other participants discover the same edge and trade it, eroding the realized return. Crowding shows up as a slow Sharpe decline, often accompanied by widening transaction costs (slippage on entry and exit grows as more capital chases the same trades). The 12-1 equity momentum factor decay covered in ["Slow Wandering: The Most Dangerous Type of Market Change"](https://aligrithm.com/slow-wandering-the-most-dangerous-type-of-market-change/) is the canonical example. Defense: the strategy capacity is finite and known approximately. Size below capacity. Reduce allocation as crowding increases. Mechanism 2: regime drift in the substrate. The market's structural parameters change. The prior article block covered this in detail. Examples: G10 yield convergence killing FX carry, SPX implied-vs-realized vol compression killing vol-selling overlays, equity-bond correlation flips invalidating risk-parity priors. Defense: monitor the substrate-level features that the strategy depends on. Disable when the features move outside the deployment regime. Mechanism 3: microstructure change. The exchange, market-maker, or fee structure changes. The change may be an obvious one (a fee tier overhaul, a tick-size change, the introduction of a new auction format) or a subtle one (a large new participant changes the order-flow composition). Microstructure-dependent strategies (market-making, latency arbitrage, order-flow prediction) are most exposed. Defense: maintain a venue-level changelog, monitor execution quality metrics (effective spread, fill rate, markout) at high frequency, decommission immediately when execution quality breaks. Mechanism 4: capacity saturation from the strategy's own growth. The shop's own capital deployed against the strategy outgrows the available liquidity. Slippage rises, fill rates drop, the realized P&L falls below the small-size backtest. This is the failure mode that scaling shops misdiagnose as alpha decay when the cause is internal. Defense: track scaled vs unscaled performance, run a small "shadow book" at a fixed reference size, compare the live large-book performance to the shadow. A divergence between the two is capacity saturation, not alpha decay. The four mechanisms are not mutually exclusive. A strategy can be hit by crowding and capacity at the same time, or by regime drift while crowding is also accumulating. Diagnosis matters because the defense for each is different. Allocation cuts handle capacity saturation but not alpha decay. Substrate monitoring handles regime drift but not crowding. The article ["How to Detect When a Trading System Is Dying"](https://aligrithm.com/how-to-detect-when-a-trading-system-is-dying/) gives the operational checklist that distinguishes the mechanisms. ## Modeling expected life A simple but useful model. Treat the realized Sharpe of the strategy as a slowly decaying function of time since deployment, with stochastic noise on top. $$ \\text{SR}(t) = \\text{SR}\_0 \\cdot e^{-t / \\tau} + \\eta\_t, \\qquad \\eta\_t \\sim \\mathcal{N}(0, \\sigma\_{\\text{SR}}^2) $$ The half-life of the edge is τ · log 2\. A strategy with τ = 5 years and SR\_0 = 1.0 has an expected Sharpe of 0.74 after 2 years, 0.55 after 4 years, 0.37 after 6 years. The half-life parameter τ is a Bayesian prior, set by judgment from the population of similar strategies. A single strategy's history does not support estimating it. Reasonable priors by strategy class: $$ \\begin{array}{l|c|c} \\text{Strategy class} & \\tau \\text{ (years)} & \\text{Notes} \\\\ \\hline \\text{Equity statistical arbitrage} & 2-3 & \\text{High crowding pressure} \\\\ \\text{Cross-sectional equity factors} & 5-10 & \\text{Slow decay, well-documented} \\\\ \\text{CTA / trend following} & 10-20 & \\text{Robust but capacity-limited} \\\\ \\text{Vol-selling / carry} & 3-5 & \\text{Episodic blowups} \\\\ \\text{Market-making / microstructure} & 0.5-2 & \\text{Venue-dependent} \\\\ \\text{Macro / discretionary-systematic hybrid} & 5-10 & \\text{Wide variance} \\\\ \\end{array} $$ The values are stylized priors, not measured constants. Each strategy class has a wide distribution of realized lives and the τ above is a rough mode. The point of the prior is not precision; the point is to size positions and budget research bandwidth around a finite-life assumption rather than the implicit assumption that the strategy will work forever. ## Capital allocation under finite life A strategy's expected future P&L depends on three things: the current Sharpe, the half-life, and the funded capital. The classical Kelly fraction maximizes log-wealth assuming a stationary edge. Under finite life, the right allocation is smaller because the future expected edge is a decaying function of time. $$ f^\* = \\frac{\\text{SR}}{\\sigma\_{\\text{ret}}} \\cdot \\frac{1}{1 + (T\_{\\text{horizon}} / 2\\tau)} $$ The first factor is the standard mean-variance allocation. The second factor is the finite-life discount: a strategy with shorter half-life or longer planning horizon gets a smaller allocation because the expected forward edge is smaller than the current edge suggests. A strategy with τ = 2 years and a 1-year planning horizon takes a 0.8x discount. A strategy with τ = 1 year and a 5-year horizon takes a 0.29x discount; the strategy is mostly going to be dead by the time the horizon arrives. This is the operational reason that micro-timescale strategies (market-making, latency arb) get smaller portfolio allocations than macro-timescale strategies (trend, value) at equal current Sharpe. The micro-timescale strategy is expected to die faster and the portfolio should not over-commit to a high-Sharpe strategy with short remaining life. The article ["How to Evaluate a Strategy Beyond Net Profit"](https://aligrithm.com/how-to-evaluate-a-strategy-beyond-net-profit/) later in this pillar covers the broader framework for strategy evaluation under uncertainty. The exact functional form above is a stylized model, not a derivation from first principles; the operational lesson is the direction of the adjustment, not the formula. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Portfolio-level discipline Five rules that fall out of the finite-life assumption. Rule 1: no single strategy carries more than 30% of the portfolio risk budget. A strategy can die in a quarter (microstructure change, regime break, sudden crowding event). The portfolio must survive any single death. The article ["Why More Parameters Make a Strategy Easier to Sell and Easier to Break"](https://aligrithm.com/why-more-parameters-make-a-strategy-easier-to-sell-and-easier-to-break/) gives the related point about within-strategy risk concentration. Rule 2: the portfolio runs a research pipeline of at least 2x the production strategies. If 4 strategies are funded, at least 8 are in some stage of research, validation, or paper-trading. The pipeline is the supply curve for replacement. A shop with no pipeline has nothing to redeploy capital into when a production strategy dies. Rule 3: every strategy has a written decommission policy at deployment time, not at decommission time. The policy specifies the metrics, thresholds, and decision rules. Examples: "decommission if 18-month rolling Sharpe falls below 0.3", "decommission if effective spread on average trade exceeds 1.5x the deployment baseline", "decommission if CUSUM on predicted-vs-realized return crosses h=5". The policy is automatic. Discretionary holds on dying strategies are the failure mode the policy exists to prevent. Rule 4: the post-mortem is mandatory. Every decommissioned strategy gets a written analysis of which decay mechanism killed it, when the first detection signal triggered, why the trader did or did not act on it, and what the research pipeline learned. The post-mortem is the input to better priors on τ for the next strategy in the same class. Rule 5: budget research bandwidth around replacement, not discovery. The new shop spends 100% of research time on discovering new edges. The mature shop spends 30% on discovery and 70% on monitoring, debugging, retraining, and decommissioning the existing portfolio. The mature ratio assumes the portfolio is generating most of the lifetime P&L from already-discovered strategies that are being kept alive longer through better operations. ## Behavioral pitfalls Three failure modes that show up around strategy death. Pitfall 1: hold-and-hope on a familiar strategy. The trader has been running the strategy for years. The strategy made the trader's career. The recent underperformance "must be" a temporary drawdown. The CUSUM alarm "must be" a false positive. Personal attachment overrides the decommission policy. The cure is a written, hard-coded decommission policy that the trader cannot override without external sign-off. Pitfall 2: premature decommission of a healthy strategy. The opposite mistake. A normal drawdown gets misclassified as the start of decay. The strategy is pulled. The drawdown was statistical noise. The trader has destroyed an asset by acting on weak evidence. The cure is discipline on the false-alarm rate of the detector. CUSUM with h=5 has a calibrated false-alarm rate (approximately one false alarm per 200 days under stationarity). Tighter thresholds than the calibration require corresponding research evidence that the trade-off is worth it. Pitfall 3: chasing the dead strategy with retrains. The strategy underperforms. The trader retrains on the most recent year. The retrained strategy validates beautifully on 2010 to 2024 and performs poorly on 2025\. The trader retrains again. The retrained strategy validates on 2010 to 2025 and performs poorly on 2026\. Each retrain is fitting the recent regime that has already moved on. The cure is to recognize that retraining is a fix for slow drift and not for structural break. When the substrate has structurally changed (new microstructure regime, new macro regime, new participant mix), the retrain produces a strategy that is well-fit to a regime that no longer exists. Decommission is the right action. ## Decision matrix | Decay signal | Most likely mechanism | Action | | --------------------------------------------------------------- | --------------------- | ---------------------------------------------- | | Sharpe slow decline, transaction costs rising | Alpha crowding | Reduce allocation, monitor capacity | | Sharpe slow decline, transaction costs flat | Slow regime drift | Continuous walk-forward retrain | | Sharpe step-down at venue change | Microstructure change | Decommission immediately, research replacement | | Sharpe step-down at macro break | Regime break | Disable until regime stabilizes | | Sharpe decline correlated with allocation increase | Capacity saturation | Reduce allocation, run shadow book | | OOS Sharpe diverges from IS Sharpe at deployment | Backtest overfit | Decommission, post-mortem the research process | | Drawdown larger than historical max but within Monte Carlo band | Statistical noise | Hold, increase monitoring | | Drawdown larger than Monte Carlo 99th percentile | Real failure | Decommission per policy | The matrix is operational, not exhaustive. The article ["How to Detect When a Trading System Is Dying"](https://aligrithm.com/how-to-detect-when-a-trading-system-is-dying/) gives the diagnostic checklist that distinguishes the rows. ## Anti-patterns Five mistakes that show up around strategy lifecycle management. Anti-pattern 1: assuming the discovery moment is the most important moment. The discovery is one moment in a multi-year strategy life. The harder problem is keeping the strategy operational through its decay phase and decommissioning it cleanly when the time comes. Shops that frame their job as "finding alpha" lose to shops that frame their job as "operating a portfolio of dying alphas plus a research pipeline of new ones". Anti-pattern 2: backtesting forever to find the strategy that "doesn't decay". No strategy escapes the four decay mechanisms. The search for an immortal strategy is a search for a unicorn and the time spent searching is time not spent on operating mortal strategies well. The right framing is the actuarial one: every strategy has a finite life, the portfolio is a population of strategies in different lifecycle stages, the operator's job is the steady-state management problem. Anti-pattern 3: no shadow book. Without a small-size shadow book running the same strategy as the funded large-size book, the operator cannot distinguish capacity saturation from alpha decay. The two failure modes look identical at the funded book's P&L level. The shadow book is cheap (a fraction of a percent of funded capital) and decisive. Anti-pattern 4: research bandwidth dedicated to one moonshot. A research team that spends a year on one ambitious strategy has a 1-in-N hit rate where N is large. The portfolio benefits more from a research process that produces multiple modest strategies (each with τ of 3 to 5 years and Sharpe of 0.5 to 1.0) than from one ambitious strategy that may or may not survive validation. Diversification at the strategy-discovery level matters as much as diversification at the strategy-portfolio level. Anti-pattern 5: ignoring the post-mortem. The decommissioned strategy is the most informative dataset the shop has about the decay process for that strategy class. Skipping the post-mortem skips the chance to update τ priors, refine the decommission policy, and identify research-process failures. The post-mortem is the feedback loop that makes the next strategy live longer. ## Visualizing strategy lifecycles ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/05/article_56-visual-0.png) ## KEY POINTS - Every trading strategy has a finite life. The question is not whether it dies but when, by which mechanism, and how the portfolio is structured to survive each death without catastrophic drawdown. - Four decay mechanisms: alpha crowding (slow Sharpe decline with rising transaction costs), regime drift in the substrate (slow Sharpe decline with stable costs), microstructure change (sharp Sharpe step-down at a venue or fee event), capacity saturation from the shop's own growth (Sharpe declines correlated with allocation increases). - Each mechanism has a different defense. Allocation cuts handle capacity. Substrate monitoring handles regime drift. Continuous walk-forward retrain handles slow drift. Immediate decommission and venue-changelog monitoring handle microstructure. - The lifecycle has four stages: discovery and validation (3 to 12 months), live deployment (1 to 5 years for healthy strategies), decay (6 months to 2 years), death (instantaneous decommission plus mandatory post-mortem). - A simple exponential-decay prior on Sharpe with strategy-class-specific half-life tau gives the right framing for capital allocation. Stylized priors: equity stat-arb tau approximately 2 to 3 years, cross-sectional equity factors tau approximately 5 to 10 years, CTA trend following tau approximately 10 to 20 years, market-making tau approximately 0.5 to 2 years. - Capital allocation under finite life is the standard mean-variance allocation discounted by a factor that depends on the ratio of planning horizon to half-life. Short-half-life strategies get smaller allocations at equal current Sharpe. - Five portfolio rules: no single strategy more than 30% of risk budget, research pipeline at least 2x the production strategies, written decommission policy at deployment time not at decommission time, mandatory post-mortem on every decommission, research bandwidth budget weighted approximately 30% discovery and 70% operations once the portfolio is mature. - Three behavioral pitfalls: hold-and-hope on a familiar strategy past its decommission criteria, premature decommission of a healthy strategy on weak evidence, chasing a dead strategy with successive retrains that fit a regime that has already moved on. - The shadow book is the cheapest tool for distinguishing capacity saturation from alpha decay. Run a small fixed-size shadow alongside the funded book and compare. A divergence is capacity. A common decline is alpha or regime. - Discovery is one moment in a multi-year strategy life. The mature shop's lifetime P&L is mostly produced by operating mortal strategies well, not by discovering immortal ones. Frame the operator's job as steady-state portfolio management, not as ongoing discovery. - The next article in the publication (["How to Detect When a Trading System Is Dying"](https://aligrithm.com/how-to-detect-when-a-trading-system-is-dying/)) gives the operational diagnostic checklist that distinguishes the four decay mechanisms in real time. The current article is the strategic frame; the next article is the procedural manual. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Testing and Tuning Market Trading Systems - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Testing+and+Tuning+Market+Trading+Systems+Timothy+Masters&ref=aligrithm.com) - [Data Mining Algorithms in C++ - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Data+Mining+Algorithms+in+C%2B%2B+Timothy+Masters&ref=aligrithm.com) ### 3.2 Slow Wandering: The Most Dangerous Type of Market Change URL: https://aligrithm.com/slow-wandering-the-most-dangerous-type-of-market-change/ Last updated: 2026-07-27T13:24:01.000Z A trader builds a long-only momentum system on US large-cap equities. The signal is the 12-month total return minus the 1-month total return, ranked across the S&P 500 universe, top decile held equal-weight, rebalanced monthly. Backtest 1995 to 2010: Sharpe 0.71, annualized return 11.4%, max drawdown 32% (in 2009). Backtest 1995 to 2015 with the same rules: Sharpe 0.62\. Backtest 1995 to 2020: Sharpe 0.48\. Backtest 1995 to 2025: Sharpe 0.31\. The strategy never had a single year of catastrophic failure. It had a slow, monotone deterioration of the edge. The trader who deployed it in 2010 captured most of the original edge. The trader who deployed it in 2018 captured a third of it. The trader who deployed it in 2024 captured roughly nothing. Nothing broke. The trade-execution stack worked. The signal definition was unchanged. The universe was unchanged. The rebalance frequency was unchanged. The only thing that changed was the underlying joint distribution of (12-1 momentum, forward 1-month return) across the S&P universe. The slope of the conditional expectation flattened by approximately 0.04 standard deviations per year. Over fifteen years that is 0.6 standard deviations of edge silently erased. This is slow wandering. The prior article in the publication (["Stationarity: The Word Every Trader Ignores Until It Kills the Strategy"](https://aligrithm.com/stationarity-the-word-every-trader-ignores-until-it-kills-the-strategy-2/)) covered the broad case for stationarity discipline and named slow drift as the failure mode that standard tests miss. This article gives slow wandering its own treatment because the diagnostic problem is different from the abrupt-regime problem and most testing machinery in the trader's toolkit is calibrated against the wrong threat. Abrupt breaks are noisy and obvious. Slow drift is silent and accumulates. The trader who only watches for abrupt breaks loses to slow drift on the strategies that survive long enough for the drift to matter. ## What slow wandering is Slow wandering is non-stationarity in which the parameters of the data-generating process drift slowly relative to the within-window noise. Formally, decompose the feature into a slowly-varying mean and a fast-varying residual. $$ X\_t = \\mu(t) + \\varepsilon\_t, \\qquad \\mu(t) - \\mu(t - W) = \\delta \\cdot W, \\qquad \\text{Var}\[\\varepsilon\_t\] = \\sigma\_W^2 $$ The drift rate δ is the change in the mean per unit time. The within-window noise σ\_W is the standard deviation of the residual on the rolling window of length W. Slow wandering is the regime where δ is small relative to σ\_W per W, but the cumulative drift δ · T over the deployment horizon T is comparable to σ\_W or larger. The strategy backtest sees the small per-window drift and treats it as noise. The strategy live performance pays the cumulative drift as a slowly accumulating loss. The signal-to-noise ratio that determines whether a stationarity test detects the drift on a window of length W is the drift over the window divided by the noise on the window. $$ \\text{SNR}(W) = \\frac{\\delta \\cdot W}{\\sigma\_W / \\sqrt{W}} = \\frac{\\delta \\cdot W^{3/2}}{\\sigma\_W} $$ The 3/2 power on W is the operational fact that determines everything that follows. To detect slow drift you need either a slower drift to be larger relative to short-window noise (impossible to control), or you need to use a longer window. Doubling the window length increases the SNR by a factor of approximately 2.83\. Quadrupling it increases the SNR by a factor of 8\. Standard ADF and KPSS implementations on 252-day or 1000-day windows have low power against drifts that take 5 to 10 years to fully express. ## Why standard tests miss it Three structural reasons. Reason 1: window length. ADF, KPSS, and the rolling-mean check from the prior article are all run on a fixed window. The window is short enough to keep the test responsive to abrupt changes (typically 252 days, sometimes 1000) and long enough to give the test reasonable statistical power against fast drift. Slow drifts whose half-life is several years are below the detection floor of these tests at standard windows. Running the same tests on 5-year or 10-year windows raises detection power but trades responsiveness for it. Reason 2: in-sample fitting absorbs the drift. When a strategy is fit (parameters tuned, thresholds calibrated) on a long historical window that contains the drift, the optimizer treats the drift as part of the noise structure. The fitted parameters are a compromise across the drift trajectory. The compromise looks reasonable on the in-sample data because the optimizer minimized over it, but the out-of-sample data is one more step along the drift path, not a fresh draw from the same distribution. The strategy underperforms its in-sample backtest by an amount proportional to the cumulative drift across the OOS window. Reason 3: standard rolling-mean checks compare the rolling mean to itself, not to a long-run reference. A 252-day rolling mean of a slowly drifting feature traces the drift trajectory smoothly. The rolling mean does not look unusual at any single point. The drift only becomes visible when the rolling mean is compared to a much longer reference (a 5-year or 10-year mean, or a fixed pre-deployment baseline). The diagnostic that catches slow drift is not "is the rolling mean changing" but "is the rolling mean different from the mean of the regime we deployed against". ## Where slow wandering shows up in markets Four examples where the slow drift was the actual story and most operators noticed only after the strategy was already underperforming. Example 1: equity momentum decay. The 12-1 momentum factor on US large-caps had a Sharpe of approximately 0.7 from the 1980s through the early 2000s, decayed to approximately 0.4 by the mid-2010s, and runs near 0.2 in the most recent decade. The factor was not killed by a single event. It was eroded by widening participation (more crowded trade), declining transaction costs (faster arbitrage), and a structural shift in the equity universe toward larger and lower-vol mega-caps that exhibit weaker momentum. None of these is a regime break in the abrupt sense. Each is a slow trend that compounded over decades. Example 2: equity-bond correlation drift. Rolling 250-day correlation of SPX daily returns to 10-year Treasury daily returns ran negative through most of 2000 to 2020 (mean approximately -0.35), shifted to slightly positive around 2022 (mean approximately +0.10), and has stabilized around -0.15 in 2024 to 2025\. The 60/40 portfolio's diversification benefit is a function of this correlation. The 2022 shift was partially abrupt and partially the culmination of slow drift in the macro regime (inflation expectations rising structurally). Strategies that assumed a stationary -0.35 correlation experienced surprising drawdowns proportional to the cumulative correlation drift. Example 3: FX carry-trade decay. The G10 carry trade (long highest-yielders, short lowest-yielders) earned a robust premium through the 1980s and 1990s, decayed through the 2000s, and has produced near-zero risk-adjusted returns post-2008\. The 2008 crisis is the convenient narrative, but the underlying driver was the slow convergence of central-bank policy rates across the developed world (the cross-sectional yield dispersion shrank from over 800 basis points in the 1990s to under 200 basis points by the late 2010s). The carry premium scales with the dispersion. A strategy that assumed historical carry returns were the relevant prior was making a stationarity assumption against a slow drift in the macro substrate. Example 4: realized volatility long-run drift. SPX realized volatility on the 252-day rolling window has averaged approximately 16% across the post-1990 sample, but the conditional mean depends on the macro regime. The 1990s averaged around 14%, the 2000 to 2008 window averaged around 18%, the 2010 to 2019 window averaged around 14%, and the post-2020 window has averaged around 18%. Volatility-targeting strategies that normalize against a long historical mean carry the drift in their position-sizing rule. Strategies that use rolling estimates with too-short a window adapt fast but pay an estimation-noise tax. The trade-off between drift adaptation and estimation noise is operational and non-trivial. ## Detecting slow wandering Three detection tools. Each addresses a different failure mode of the standard tests. Tool 1: long-window mean vs short-window mean divergence. Compute the rolling mean on two windows: a short window (252 bars) and a long window (1260 bars, approximately 5 years). Compute the divergence as the short-minus-long. $$ D\_t = \\hat{\\mu}\_t^{(\\text{short})} - \\hat{\\mu}\_t^{(\\text{long})}, \\qquad D\_t \\text{ should be near zero under stationarity} $$ Under stationarity, D\_t is mean-zero with a known standard deviation that depends on the window lengths. Under slow drift, D\_t accumulates a non-zero expected value proportional to the drift rate. A persistent divergence of D\_t in one direction across multiple short windows is the signature of slow drift. Standardize D\_t by its theoretical stationary standard deviation and trigger an alarm when the standardized value exceeds 2 or 3 sigma persistently. Tool 2: CUSUM chart. The cumulative sum of standardized deviations from a reference mean detects slow drift faster than the rolling-mean check. $$ S\_t = \\max\\!\\left(0,\\; S\_{t-1} + \\frac{X\_t - \\mu\_0}{\\sigma\_0} - k\\right), \\qquad \\text{alarm when } S\_t > h $$ The reference mean μ\_0 and reference std σ\_0 are estimated from a clean baseline period (the strategy development window). The slack parameter k controls sensitivity to small drifts (typical k = 0.5 detects drifts of size 1·σ\_0 efficiently). The threshold h controls the false-alarm rate (h = 4 to 5 is conservative). CUSUM was designed for industrial process control where slow drift is the dominant failure mode, and it transfers directly to the trading-feature monitoring problem. The detection delay for a drift of size d is approximately h / d in standardized units, which is order-of-magnitude faster than waiting for the rolling mean to visibly move. Tool 3: Bayesian online change-point detection. A formal model of the drift hypothesis. Maintain a posterior distribution over the time of the last regime change. The posterior is updated with each new observation. When the posterior mass concentrates on a recent date, the algorithm has detected a change. This is more sensitive than CUSUM but requires more modeling commitment (priors on regime length, parametric form of the change). Useful when you need a probabilistic alarm rather than a binary one and have a clear prior on regime length. The article ["Volatility Regimes and Strategy Survival"](https://aligrithm.com/volatility-regimes-and-strategy-survival/) later in this pillar covers the application of these tools to volatility specifically. The mechanics generalize to any feature whose mean or variance is at risk of slow drift. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Defending against slow wandering Five operational defenses. Each addresses a different aspect of the problem. Defense 1: continuous walk-forward. The article ["Why Walk-Forward Testing Is Better Than One Big OOS Split"](https://aligrithm.com/why-walk-forward-testing-is-better-than-one-big-oos-split/) later in this pillar argues this in detail. The mechanism: retrain the strategy at quarterly or annual cadence so the in-sample window slides forward and absorbs the drift. The retrain is not a backtest; it is a recalibration of parameters against the most recent regime. The cost is parameter instability, which itself can be a sign of overfitting. Monitor the parameter trajectory and reject retrains that produce wild swings. Defense 2: parameter aging. Apply an explicit half-life to the weight given to old observations. A strategy that uses an EWMA estimator with half-life of 2 years gives 50% weight to data more than 2 years old, 25% to data more than 4 years old, and so on. The half-life is a hyperparameter that itself overfits if chosen by backtest, but a fixed economically-motivated half-life (e.g., one business cycle) is a defensible prior. Parameter aging is the time-domain analog of rolling normalization (covered in ["Rolling Normalization: Useful Tool or Hidden Overfit?"](https://aligrithm.com/rolling-normalization-useful-tool-or-hidden-overfit/)). Defense 3: multi-timescale regime detection. Run regime detection (CUSUM, Bayesian change-point, regime-switching model) at multiple timescales simultaneously. Short timescales (weeks to months) catch abrupt breaks. Long timescales (years) catch slow drift. The strategy gating layer combines both: disable on abrupt break, taper on slow drift. The tapering rule is operationally important: cut position size as the drift signal accumulates, rather than waiting for a binary disable trigger that may arrive years too late. Defense 4: explicit decommission criteria. Every strategy has a written specification of the conditions under which it is shut down. The criteria include: cumulative drift on key features exceeds X sigma against the deployment baseline, OOS Sharpe falls below Y for Z consecutive months, drawdown exceeds the historical maximum by W%. The decommission decision is automatic against the criteria, not discretionary. The article ["How to Detect When a Trading System Is Dying"](https://aligrithm.com/how-to-detect-when-a-trading-system-is-dying/) later in this pillar gives the operational checklist. Defense 5: portfolio-level finite-life budgeting. Assume each strategy has a finite life. Size each strategy's allocation so the portfolio survives the loss of any single strategy without catastrophic drawdown. Maintain a research pipeline of new strategies to replace dying ones. The article ["Why Systems Work Until They Don't"](https://aligrithm.com/why-systems-work-until-they-dont/) frames this as the steady-state portfolio management problem rather than the one-time strategy-selection problem. ## The trade-off: detection lag vs false alarms Slow drift detection is a hypothesis-testing problem. Type I error is a false alarm (declaring drift when none exists, leading to unnecessary strategy decommissioning or retraining). Type II error is a missed detection (failing to declare drift, leading to silent edge erosion). The two errors trade off against each other through the detection threshold. $$ \\Pr(\\text{false alarm} \\mid \\text{stationary}) = \\alpha, \\qquad \\Pr(\\text{detect} \\mid \\text{drift rate } \\delta) = 1 - \\beta(\\delta) $$ For a CUSUM detector with threshold h, lowering h reduces detection lag (catch drift sooner) but raises false-alarm rate (more unnecessary decommissions). Raising h reduces false alarms but increases the time to detection (more cumulative loss before action). The right operating point depends on the cost ratio: how much edge do you lose per unit of detection delay versus how much do you lose by retraining or decommissioning a still-good strategy. For most retail and small-shop quant operations, the cost of unnecessary decommission is high (rebuilding edge is expensive) and the cost of detection delay is moderate (the drift accumulates slowly by definition). This argues for conservative thresholds (h = 5 to 6, k = 0.5) and pairing CUSUM with multiple corroborating signals before taking action. For institutional operations with diversified strategy pools, the cost of unnecessary decommission is lower (other strategies are running) and the cost of detection delay is higher (concentrated risk). This argues for tighter thresholds and faster action. ## Decision matrix | Drift type | Detection tool | Defense | Speed of response | | ----------------------------------- | -------------------------------------- | ----------------------------------- | ----------------- | | Abrupt regime break (1-day shock) | Z-score on daily change | Disable strategy on alarm | Same-day | | Fast drift (weeks to months) | Rolling-mean check at 252-day window | Walk-forward retrain | Quarterly | | Slow drift (1 to 5 years) | CUSUM with k=0.5, h=5 | Parameter aging + position taper | Annual | | Multi-decade structural shift | Long-window vs short-window divergence | Decommission strategy | Multi-year | | Mean drift | CUSUM on standardized feature | Rolling normalization | Per detector | | Variance drift | CUSUM on log realized vol | ATR scaling on inputs | Per detector | | Correlation drift | Rolling correlation vs long-run | Disable correlation-dependent rules | Quarterly | | Distributional drift (shape change) | KS test on rolling distribution | Re-fit rank or sigmoid transforms | Annual | The matrix is operational, not exhaustive. Each row pairs a class of non-stationarity with a detector and a defense. The detector and defense are matched to the speed of the drift, which is the parameter that determines what tools have power. ## Anti-patterns Five common mistakes specific to slow drift. Anti-pattern 1: "ADF rejected, the feature is stationary." ADF on a 1000-day window has low power against drifts whose half-life exceeds 3 years. The test passes even when the cumulative drift across the deployment horizon will erase the strategy edge. Run the long-window vs short-window divergence and CUSUM to cover the slow-drift case explicitly. Anti-pattern 2: "I'll backtest on a longer history to capture more regimes." A longer history captures more abrupt regimes, which is good. It also bakes in more cumulative drift, which is bad: the optimizer averages across a moving target and produces parameters that are a compromise across all the historical regimes, rather than tuned to the current regime. The right framing is to backtest on a long history for diagnostic purposes (does the strategy survive every individual regime?) and to fit parameters to a recent window (is the strategy tuned to the current regime?). The two questions need different windows. Anti-pattern 3: "I'll use a 60-day rolling normalization to handle the drift." Sixty days is long enough to capture abrupt vol regime changes and short enough to track slow drift, which sounds ideal. The problem: a 60-day rolling normalization on a feature that has a multi-year cycle will track the cycle and remove the very signal that the strategy was supposed to trade. The cure becomes the disease. Match the rolling-window length to the drift timescale, not to the abrupt-shock timescale. Anti-pattern 4: "The strategy underperformed for one year, but it always recovers." A single underperforming year on a slow-drift-decayed strategy looks identical to a normal drawdown on a still-healthy strategy. The discrimination requires the drift detection. Without the detection, the trader will hold a dying strategy through several underperforming years before noticing, and the held losses will be large. Run the detector. Trust the detector when it triggers. The detection logic, not the P&L, is the disable trigger. Anti-pattern 5: "Strategy decay is the cost of doing business." It is the cost, but it is not an excuse to avoid measuring it. The trader who measures decay can decommission early and reallocate capital to fresh strategies. The trader who treats decay as an unmeasurable fact-of-life loses to the trader who measures it. The article ["How to Detect When a Trading System Is Dying"](https://aligrithm.com/how-to-detect-when-a-trading-system-is-dying/) frames the operational version of this discipline. ## Visualizing slow wandering ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/05/article_55-visual-0.png) ## KEY POINTS - Slow wandering is non-stationarity in which the parameters of the data-generating process drift slowly relative to within-window noise. The cumulative drift across the deployment horizon is the loss; the per-window drift looks like noise. - Detection power scales with W^(3/2) where W is the test window. Standard 252-day or 1000-day rolling tests have low power against drifts whose half-life exceeds 3 years. - Three structural reasons standard tests miss slow drift: window length too short, in-sample fitting absorbs the drift into the parameter set, rolling-mean checks compare the rolling mean to itself rather than to a deployment-era baseline. - Four real market examples: equity 12-1 momentum factor decay over 30 years, equity-bond correlation drift across 2020 to 2024, FX carry-trade decay across 1990 to 2010, SPX realized vol regime drift across decades. None had an abrupt break large enough for standard tests to catch as a single event. - Three detection tools: long-window vs short-window divergence (cheap, intuitive), CUSUM (sensitive, designed for slow drift, fast detection), Bayesian online change-point (most sensitive, most modeling commitment). - Five defenses: continuous walk-forward retraining, explicit parameter aging via EWMA half-life, multi-timescale regime detection running short and long detectors in parallel, written decommission criteria, portfolio-level finite-life budgeting that assumes every strategy dies eventually. - The detection-lag vs false-alarm trade-off is set by the threshold h on the CUSUM (or analogous parameter on other detectors). Conservative thresholds favor avoiding unnecessary decommission. Tight thresholds favor catching drift early. The right operating point depends on the cost ratio of edge erosion vs decommission cost. - Anti-pattern: ADF rejection treated as proof of stationarity. ADF rules out a unit root on the test window and is silent on slow drift below the test's power threshold. - Anti-pattern: longer backtest window assumed to be safer. A longer window dilutes parameter fits across a moving target. Diagnostic backtest on long history, parameter fit on recent regime; two questions, two windows. - Anti-pattern: a single underperforming year treated as a routine drawdown. A drawdown on a slow-decayed strategy looks identical to a normal drawdown. Run the drift detector and let the detector, not the P&L, drive the disable decision. - The current article gives slow drift its own treatment because the detection problem is structurally different from abrupt-break detection. The next article in the publication (["Why Systems Work Until They Don't"](https://aligrithm.com/why-systems-work-until-they-dont/)) generalizes the slow-drift framework to the steady-state question of strategy lifecycle: how to operate a portfolio of strategies that all eventually decay, and how to budget capital and research bandwidth around that finite-life assumption. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Testing and Tuning Market Trading Systems - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Testing+and+Tuning+Market+Trading+Systems+Timothy+Masters&ref=aligrithm.com) - [Data Mining Algorithms in C++ - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Data+Mining+Algorithms+in+C%2B%2B+Timothy+Masters&ref=aligrithm.com) ### 3.1 Stationarity: The Word Every Trader Ignores Until It Kills the Strategy URL: https://aligrithm.com/stationarity-the-word-every-trader-ignores-until-it-kills-the-strategy-2/ Last updated: 2026-07-27T13:24:00.000Z A trader builds a mean-reversion system on EURUSD daily. The rule is one line: enter long when 14-day RSI prints below 30, exit at 50, mirror short on RSI > 70 to 50\. Backtest 2010 to 2014: Sharpe 1.6, max drawdown 6%, 312 round trips, profit factor 1.9\. The trader ships it in January 2015\. By December 2015 the system is down 14% and has not produced a single winning month since the SNB unpegged the franc. The trader assumes a parameter problem. The trader retunes RSI to 21 days and the thresholds to 25/75\. The retune validates beautifully on 2010 to 2015\. By mid-2016 the new version is also bleeding. The trader gives up. The system never broke. The market it was designed for stopped existing. EURUSD daily realized volatility ran 8.4% annualized in 2010 to 2014, then climbed to 11.2% in 2015 and 9.8% in 2016, while the autocorrelation of daily returns flipped sign at the 1 to 5 day lag (mean-reverting in the first window, mildly trend-following in the second). The 14-day RSI distribution shifted: the median moved from 49.6 to 51.8, the lower 10th percentile from 24 to 27, the upper 90th from 76 to 73\. None of these shifts looks dramatic on its own. Together they relocate the entire signal distribution out of the regime where the threshold rule lived. The strategy was statistically dead before the trader noticed. This is the stationarity problem. Every trading rule, indicator, threshold, and statistical test assumes the data-generating process is stable enough across the deployment window for the in-sample statistics to be informative about the out-of-sample distribution. Markets violate this assumption almost continuously. Most violations are small enough to ignore for a year or two and then catastrophic on the third. The prior article in the publication (["Why Market Cycles Are Evanescent"](https://aligrithm.com/why-market-cycles-are-evanescent/)) established the case for one specific component (cycle period). This article generalizes the argument to every strategy component and opens Pillar 3 (Robust Systems Lab), which is the testing discipline you need once you accept that stationarity is the assumption you cannot safely take for granted. ## What stationarity means operationally A stochastic process is strictly stationary when the joint distribution of any finite collection of samples is invariant under time shifts. That definition is useless for trading because it is unverifiable from a single price path. The operational definition that matters is weak (or covariance) stationarity: the first and second moments are time-invariant. $$ \\text{Weak stationarity: } \\quad \\mathbb{E}\[X\_t\] = \\mu, \\quad \\text{Var}\[X\_t\] = \\sigma^2, \\quad \\text{Cov}\[X\_t, X\_{t+k}\] = \\gamma(k) $$ The mean does not drift, the variance does not drift, and the autocovariance depends only on the lag k, not on the absolute time t. If those three hold, the rolling estimate of any statistic of X converges as the window grows, and a backtest computed on one slice is informative about another slice. The stronger version (joint distribution invariance) is what predictive models silently require. A logistic regression that maps RSI to next-day return needs the joint distribution of (RSI, next-day return) to be stable. A threshold rule needs the marginal of RSI to be stable so the threshold sits at the same percentile in production as in the backtest. A volatility-targeted position-sizing rule needs the variance estimator to be unbiased. Each of these is a stationarity assumption with a different functional form. ## What is non-stationary in markets Three components, each violating stationarity in a different way. Raw price is non-stationary in level. The mean of SPX in 2010 was 1140; in 2025 it is around 6000\. The variance scales with the level (multiplicative noise). No useful statistical model takes raw price as input without first transforming it. Returns (log differences) are closer to stationary in mean (zero or near-zero unconditional mean) but the variance is still time-varying. Volatility is non-stationary in mean and variance. Realized volatility on SPX daily moves between 8% (2017) and 35% (2008, 2020). It is mean-reverting on the 30 to 60 day horizon but the long-run mean itself drifts with the macro regime. Volatility is the most reliably non-stationary feature in markets, which is why it is the easiest one to model with explicit time-varying machinery (GARCH, HAR, regime switching) and the hardest one to ignore. Cross-sectional structure is non-stationary in correlation. The correlation between bonds and equities was negative from 2000 to 2020, positive in 2022 and 2023, negative again in 2024\. Strategies built on a fixed correlation assumption (60/40, risk parity with constant priors) carry the regime risk implicitly. Correlation is the second most non-stationary structural feature, after volatility. Indicator distributions inherit non-stationarity from their inputs. RSI is bounded in \[0, 100\] but its conditional distribution given the regime is not bounded. The 90th percentile of RSI on EURUSD daily was 76 in 2010 to 2014 and 73 in 2015 to 2016\. Threshold rules calibrated on one window misfire in the other. The article ["Why Most Indicators Should Be Transformed Before Modeling"](https://aligrithm.com/why-most-indicators-should-be-transformed-before-modeling/) framed this for individual features; the present article frames it as a strategy-level concern. Cycle structure is non-stationary in period, amplitude, and phase. The prior article (["Why Market Cycles Are Evanescent"](https://aligrithm.com/why-market-cycles-are-evanescent/)) covered this in detail. ## How to test for stationarity Three operational tests, each with different sensitivity. Test 1: rolling mean and rolling standard deviation. Compute the mean and std on rolling 252-day windows of the feature you care about. If the rolling mean drifts by more than one rolling std across the sample, the feature is non-stationary in mean. If the rolling std varies by more than 50% across windows, the feature is non-stationary in variance. This is the cheapest, most robust diagnostic and the one most traders skip. $$ \\hat{\\mu}\_t = \\frac{1}{W} \\sum\_{i=0}^{W-1} X\_{t-i}, \\qquad \\hat{\\sigma}\_t = \\sqrt{\\frac{1}{W-1} \\sum\_{i=0}^{W-1} (X\_{t-i} - \\hat{\\mu}\_t)^2} $$ Test 2: Augmented Dickey-Fuller (ADF). Tests the null hypothesis that the series has a unit root (non-stationary in level). Rejecting the null at p < 0.05 is evidence that the series is stationary or trend-stationary. The test is sensitive to lag selection and weak against slow drift; passing ADF is necessary but not sufficient. $$ \\Delta X\_t = \\alpha + \\beta t + \\gamma X\_{t-1} + \\sum\_{i=1}^{p} \\delta\_i \\Delta X\_{t-i} + \\varepsilon\_t $$ The null is γ = 0\. Reject the null when the t-statistic on γ is more negative than the critical value (approximately −2.86 at 5% for the constant-only specification on long samples). Daily SPX log prices fail to reject (non-stationary, as expected). Daily SPX log returns reject (stationary in level). RSI on SPX daily rejects on most subsamples but the rejection is noisier than the rolling-statistic check would suggest, because ADF tests level stationarity, not distributional stationarity. Test 3: KPSS (Kwiatkowski-Phillips-Schmidt-Shin). The complementary test: null is stationarity, alternative is non-stationarity. Use ADF and KPSS together. If ADF rejects (data look stationary) and KPSS does not reject (data look stationary), the verdict is consistent. If they disagree, the data are in the ambiguous middle, often because of slow drift. Slow drift is the topic of the next article (["Slow Wandering: The Most Dangerous Type of Market Change"](https://aligrithm.com/slow-wandering-the-most-dangerous-type-of-market-change/)) and is precisely the regime where ADF/KPSS disagree. For trading specifically, the rolling-statistic check is the most useful because the question is not "is this series mathematically stationary" (almost no market series is) but "is the violation small enough that the strategy survives." That is a magnitude question, not a hypothesis-test question. ## How to make a feature more stationary Five operational tools, each with a cost. Tool 1: differencing. Replace X\_t with ΔX\_t = X\_t - X\_{t-1}. This removes a unit root and is the standard fix for non-stationary level. Cost: differencing destroys the level information. Differenced log price is daily log return, which is stationary in mean but the variance is still time-varying. Differencing is necessary but not sufficient. Tool 2: log differencing (returns). Replace X\_t with log(X\_t) - log(X\_{t-1}). For positive multiplicative processes (price, volume, volatility) this is the right transformation because the noise is multiplicative in the original scale. Most of Pillar 2's machinery operates on log returns rather than raw price for this reason. Tool 3: ATR scaling (volatility normalization). Divide the feature by a rolling estimate of its scale. $$ \\tilde{X}\_t = \\frac{X\_t}{\\hat{\\sigma}\_t}, \\qquad \\hat{\\sigma}\_t = \\text{ATR}(t, W) \\;\\text{or}\\; \\text{rolling std}(X, W) $$ ATR scaling makes the feature invariant to the volatility regime. A 1-ATR move in 2017 (low vol) and a 1-ATR move in 2020 (high vol) are comparable in the scaled feature. The article ["Why ATR Normalization Is More Than a Volatility Trick"](https://aligrithm.com/why-atr-normalization-is-more-than-a-volatility-trick/) gave the case for this transformation in detail. Cost: scaling shrinks the dynamic range and destroys the absolute-level signal. Tool 4: rolling normalization (z-scoring). Subtract the rolling mean and divide by the rolling std. $$ z\_t = \\frac{X\_t - \\hat{\\mu}\_t}{\\hat{\\sigma}\_t} $$ This is the strongest stationarity-forcing transformation. The rolling z-score has approximately zero mean and unit variance by construction, and the marginal distribution is approximately stable across regimes. Cost: the transformation destroys the long-run drift information that is sometimes the actual edge, and the rolling-window length W is a hyperparameter that itself overfits if chosen by backtest. The article ["Rolling Normalization: Useful Tool or Hidden Overfit?"](https://aligrithm.com/rolling-normalization-useful-tool-or-hidden-overfit/) later in the pillar covers this trap. Tool 5: rank transform (percentile normalization). Replace X\_t with its rank within a rolling window, mapped to \[0, 1\] or \[-1, 1\]. The transformed feature is uniform on its rolling-window distribution by construction. Cost: ranks are insensitive to magnitude. A 1-sigma move and a 5-sigma move both end up at rank 1.0 if no other observation in the window is more extreme. The choice between these tools is a strategy-design question, not a statistical question. The article ["When Forcing Stationarity Destroys Information"](https://aligrithm.com/when-forcing-stationarity-destroys-information/) later in this pillar argues the case that aggressive normalization can destroy the actual signal you wanted to trade. The right principle: apply the minimum transformation needed to keep the feature distribution stable across the regimes you expect to operate in, and verify with the rolling-statistic check rather than just trusting ADF rejection. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## What this changes for backtesting Six operational shifts. These are the framework that the rest of Pillar 3 builds on. Shift 1: report stationarity diagnostics for every feature in the strategy. Rolling mean and std plotted across the full sample. ADF and KPSS p-values. Distribution histograms by regime. The diagnostic is the audit trail. A strategy without stationarity diagnostics is a strategy that has not been examined. Shift 2: split backtests by regime, not by year. The convention of "2018 to 2020 in-sample, 2021 to 2023 out-of-sample" is a calendar split. The right split is by regime: low-vol vs high-vol, trend mode vs cycle mode (using the classification from ["Why Market Cycles Are Evanescent"](https://aligrithm.com/why-market-cycles-are-evanescent/)), bond-equity correlation regime, macro regime. A strategy that survives across all regimes is more robust than one that survives 2018 to 2023 with the regime-mix luck of that window. Shift 3: walk-forward with regime-aware retraining. The article ["Why Walk-Forward Testing Is Better Than One Big OOS Split"](https://aligrithm.com/why-walk-forward-testing-is-better-than-one-big-oos-split/) later in the pillar argues this in detail. The mechanism: retrain at quarterly cadence so the in-sample window slides across regime boundaries and the strategy adapts. Shift 4: stress-test the strategy on synthetic non-stationary data. Generate paths with deliberate regime breaks (vol spike, mean shift, correlation flip). Measure how the strategy behaves. A strategy that drawdowns 30% on synthetic regime breaks will drawdown more on real ones. The article ["Monte Carlo for Trading Systems"](https://aligrithm.com/monte-carlo-for-trading-systems/) frames the synthetic-path machinery. Shift 5: declare the assumed regime explicitly in the strategy spec. "This strategy assumes EURUSD daily realized vol between 6% and 10% annualized and bond-equity correlation between -0.5 and 0.0." When the assumption is violated, disable the strategy. The disable rule is part of the strategy, not a discretionary override. Shift 6: budget for system death. The article ["Why Systems Work Until They Don't"](https://aligrithm.com/why-systems-work-until-they-dont/) covers this. Even with stationarity diagnostics and regime-aware testing, every strategy eventually fails because the regime moves outside the training distribution. The portfolio should be sized assuming each strategy has finite life. ## The slow-wandering preview The next article in the publication (["Slow Wandering: The Most Dangerous Type of Market Change"](https://aligrithm.com/slow-wandering-the-most-dangerous-type-of-market-change/)) covers the specific non-stationarity that ADF and most standard tests miss. The setup: imagine a feature whose mean drifts at 0.001 sigma per day. After 1000 days the mean has moved by one full sigma. ADF on the full sample rejects the unit root because the within-window variability dominates the drift. Rolling-mean check at 252-day window detects the drift but only after several windows have passed. The strategy backtest, computed on 5 years of data with a constant threshold, looks fine because the in-sample drift is small. The strategy fails in production once the drift accumulates past the threshold's robust range. Slow wandering is the most dangerous form of non-stationarity precisely because it does not look broken until it has already broken the strategy. The current article frames the problem at the conceptual level. The next article frames it at the diagnostic level and gives the operational test for slow wandering. Together they are the foundation for everything else in Pillar 3. ## Decision matrix | Feature | Stationarity status | Recommended transformation | Diagnostic | | ------------------------ | -------------------------------------------------- | ----------------------------- | ----------------------------- | | Raw price (level) | Non-stationary in level | Log difference (return) | Rolling mean, ADF | | Log returns | Stationary in mean, non-stat in variance | ATR scaling | Rolling std | | RSI / oscillator outputs | Approximately stationary in mean, regime-dependent | Rolling normalization or rank | Rolling histogram | | Realized volatility | Non-stationary in mean and variance | Log + rolling mean subtract | Rolling mean of log vol | | Cross-asset correlation | Non-stationary structurally | Regime classifier as input | Rolling correlation by regime | | Cycle period (T) | Non-stationary (covered in prior article) | Live periodogram estimate | Rolling σ\_T | | Indicator threshold | Inherits from input | Rolling percentile rank | Threshold drift over time | The diagnostic column is the rolling-statistic check that should be on every feature before the strategy is deployed. ## Anti-patterns Five mistakes that show up when stationarity is treated as a checkbox rather than an audit. Anti-pattern 1: "I ran ADF on the feature, it passed, the feature is stationary." ADF tests level stationarity against a unit-root null. Passing ADF rules out one specific failure mode. It does not rule out variance non-stationarity, distributional drift, or slow wandering. Combine ADF with KPSS and rolling-statistic checks before you trust the verdict. Anti-pattern 2: "I'll fix non-stationarity with rolling normalization on a 60-bar window." Rolling normalization on a short window destroys the slow signal you might have wanted to trade. On a long window it does not adapt fast enough to regime breaks. The window length is a hyperparameter that overfits if chosen by backtest. The article ["Rolling Normalization: Useful Tool or Hidden Overfit?"](https://aligrithm.com/rolling-normalization-useful-tool-or-hidden-overfit/) covers this in detail. Anti-pattern 3: "The backtest was 5 years, that is enough." Five years of EURUSD daily covers approximately one to two macro regimes, depending on the segmentation. A strategy validated across one regime is not validated. Aim for backtests that span at least three macro regimes and report performance per regime. Anti-pattern 4: "I differenced the price, so the input is stationary." Log returns are stationary in mean and weakly stationary in variance for short windows but exhibit strong volatility clustering and fat tails. The naive normality assumption that often follows differencing is violated. Account for the heteroskedasticity (GARCH-like models, ATR scaling) and the tail behavior (the article ["Why Predictive Power Often Lives in the Tails"](https://aligrithm.com/why-predictive-power-often-lives-in-the-tails/) frames this). Anti-pattern 5: "Stationarity does not matter, the model will figure it out." A flexible model (random forest, gradient boosting, neural network) trained on non-stationary features will capture the in-sample regime structure as if it were the true relationship. In production, when the regime shifts, the model produces confidently wrong predictions because the test distribution is now different from the training distribution. Flexibility magnifies the cost of feature non-stationarity rather than absorbing it. ## Visualizing stationarity ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/05/article_54-visual-0-1.png) ## KEY POINTS - Every trading rule, indicator, threshold, and statistical test silently assumes the data-generating process is stationary across the deployment window. Markets violate this assumption. - Operational stationarity is weak (covariance) stationarity: time-invariant mean, time-invariant variance, autocovariance that depends only on lag. Threshold rules, regression models, and volatility-scaled positions each invoke stationarity in a different functional form. - Five components are the most reliably non-stationary in markets: raw price level, return variance, cross-asset correlation, indicator distributions, and cycle structure. The first four matter for every strategy; the fifth was covered in ["Why Market Cycles Are Evanescent"](https://aligrithm.com/why-market-cycles-are-evanescent/). - The rolling-statistic check (rolling mean and std on 252-day windows) is the cheapest and most robust stationarity diagnostic. Run it before any other test. - ADF and KPSS are complementary hypothesis tests. Use them together. Disagreement is the signature of slow drift, the topic of the next article (["Slow Wandering: The Most Dangerous Type of Market Change"](https://aligrithm.com/slow-wandering-the-most-dangerous-type-of-market-change/)). - Five tools force a feature to look more stationary: differencing, log differencing, ATR scaling, rolling z-scoring, rank transform. Each destroys some information. The right one depends on what edge the strategy is consuming. - Five backtesting shifts follow from accepting the stationarity problem: report diagnostics per feature, split by regime not by year, walk-forward with regime-aware retraining, stress-test on synthetic non-stationary data, declare the assumed regime explicitly, and budget for system death. - Anti-pattern: ADF passed therefore the feature is stationary. ADF rules out one failure mode. Distributional drift, variance non-stationarity, and slow wandering need other diagnostics. - Anti-pattern: a flexible model will absorb non-stationarity. Flexible models magnify the cost of non-stationary features rather than absorbing it because they fit the in-sample regime structure as if it were the true relationship. - The current article opens Pillar 3 (Robust Systems Lab) by establishing the structural reason that backtesting needs explicit anti-overfitting machinery: not because researchers are dishonest, but because the world is non-stationary and the assumption of stable distributions is the load-bearing assumption every trading rule depends on. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Testing and Tuning Market Trading Systems - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Testing+and+Tuning+Market+Trading+Systems+Timothy+Masters&ref=aligrithm.com) - [Data Mining Algorithms in C++ - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Data+Mining+Algorithms+in+C%2B%2B+Timothy+Masters&ref=aligrithm.com) ### 2.31 Why Market Cycles Are Evanescent URL: https://aligrithm.com/why-market-cycles-are-evanescent/ Last updated: 2026-07-27T13:23:55.000Z A trader runs an autocorrelation periodogram on SPX daily through December 2019\. The peak sits cleanly at 22 bars with a sharp confidence band of ±2 bars. The trader builds a swing-trading strategy keyed to that 22-bar cycle: BPF at T=22, threshold rules on the AGC output, entry on zero crossings, target at the next peak. Backtest over the prior three years shows Sharpe 1.4 with reasonable drawdowns. The trader ships the strategy in mid-March 2020. By April 2020 the SPX dominant cycle has compressed to 8 bars. The trader's 22-bar BPF is the wrong filter for the new regime. The strategy generates one false signal per week and accumulates losses. By Q3 2020 the trader has retuned to a 12-bar cycle. The retune works for two months. In late 2020 cycles dissolve entirely (the strong V-recovery is a trend-mode regime with no measurable dominant period). The strategy goes idle for six months. In 2022 cycles return, this time at approximately 14 bars. The trader retunes again. In 2024 the period drifts back toward 20 bars. Across six years of operating, the trader has retuned the strategy four times. Each retune was correct in the moment. Each one was wrong by the time the next regime arrived. The strategy's lifetime Sharpe across all retunings is approximately 0.5, much lower than any individual three-year backtest suggested. The mistake was not in the filter theory, the AGC normalization, or the periodogram measurement. All of those worked as the prior articles in this series described. The mistake was assuming the cycle period would persist long enough for the strategy to harvest a stable edge. Market cycles are evanescent. They exist, they are measurable, and they are useful while they last. They do not have the constancy of physical oscillators. Their period drifts, their amplitude decays, their phase loses coherence. Strategies that consume cycle structure must gate themselves on cycle quality and accept idle periods when the cycle dissolves. This article closes the filter-theory block (the prior articles starting with ["No Filter Is Predictive: What Traders Misunderstand About Smoothing"](https://aligrithm.com/no-filter-is-predictive-what-traders-misunderstand-about-smoothing/)) by establishing the structural humility that all the filter machinery depends on: the substrate it filters is itself non-stationary. The next article in the publication (["Stationarity: The Word Every Trader Ignores Until It Kills the Strategy"](https://aligrithm.com/stationarity-the-word-every-trader-ignores-until-it-kills-the-strategy-2/)) opens Pillar 3 by generalizing the non-stationarity argument from cycle structure to all strategy components. ## What "evanescent" means precisely A cycle is persistent (in the physics sense) when its period, amplitude, and phase are constant or near-constant across many cycle lengths. A simple pendulum oscillates at the same period for thousands of cycles. A radio carrier wave is coherent for billions of cycles. Persistent cycles support prediction: knowing the current phase tells you the next peak's timing with arbitrary precision. A cycle is evanescent when its parameters are time-varying within the observation window. Three measurable non-stationary properties define evanescence: $$ \\text{Evanescent cycle: } \\quad \\dot{T}(t) \\ne 0, \\quad \\dot{A}(t) \\ne 0, \\quad \\Delta\\phi(t+T(t)) \\ne 0 $$ Period drift: T(t) changes over time. The cycle that was 22 bars in 2019 becomes 8 bars in 2020 becomes 14 bars in 2022\. The drift is not gradual; it can be abrupt at regime boundaries. Amplitude decay: A(t) changes over time. The cycle that was clearly above the noise floor in 2017 fades into noise in late 2020\. The strategy that condition on cycle amplitude must detect the decay and disable itself. Phase incoherence: the phase difference Δφ between successive cycles is not zero. In a coherent cycle, each peak occurs T bars after the prior peak. In an evanescent cycle, the inter-peak spacing varies by 20-40% even within a stable-period regime. The next peak's exact timing is not predictable from the prior peak's location. All three properties are measurable. The dominant cycle period from the autocorrelation periodogram is the period T(t). The amplitude of the BPF output centered at T(t) is A(t). The deviation between observed and predicted zero crossings is the phase incoherence Δφ. The article ["Dominant Cycle Estimation Without Astrology"](https://aligrithm.com/dominant-cycle-estimation-without-astrology/) gave the period measurement. This article uses the same methods to characterize evanescence. ## Why cycles are evanescent Four structural reasons that market cycles cannot persist like physical oscillators. Reason 1: arbitrage decay. A persistent, predictable cycle is a free option that other participants will exploit until the cycle is no longer predictable. If SPX reliably bottoms every 22 days, traders front-run the bottom by buying on day 20, the bottom moves to day 20, then day 18, then disappears. Markets enforce the no-free-lunch boundary on persistent cycles by the same arbitrage mechanism that enforces it on any other reliable pattern. Reason 2: regime shifts. Macroeconomic and structural changes (Fed policy shifts, sector rotation, geopolitical events, microstructure changes like decimalization or the introduction of HFT) reset the market's effective participant mix. Each participant class has its own preferred trading horizon, and the dominant cycle period is the emergent property of the current mix. When the mix changes, the dominant period changes. Reason 3: time-horizon mixing. At any moment, the market contains traders operating on multiple horizons (HFT, day, swing, position, macro). Each horizon contributes cycle energy at its preferred frequency. The dominant cycle is the largest peak in the composite, but the composition shifts as different participant classes engage and disengage. The "20-bar cycle" is not a property of the market; it is a property of the current participant composition. Reason 4: volatility-driven amplitude variation. Cycle amplitude scales with volatility. During low-volatility regimes, cycle amplitudes shrink toward the noise floor and the cycle becomes operationally invisible even if it persists structurally. During high-volatility regimes, cycle amplitudes expand and previously masked sub-cycles become visible. The visibility of the cycle is itself non-stationary because volatility is non-stationary. The combination: no single mechanism keeps cycles stable. Each can change independently. The dominant cycle measurement at any moment is correct, but the assumption of stationarity is wrong by construction. ## Quantifying evanescence Three measurable properties give the operational characterization. Property 1: period stability. Measure the standard deviation of the dominant cycle period over rolling 30-day windows. $$ \\sigma\_T(t) \\;=\\; \\text{std}\_{30}\\bigl(T(t-i)\\bigr), \\quad i = 0, \\dots, 29 $$ For SPX 2018-2024, the rolling 30-day std of T is typically 3-5 bars during cycle-mode regimes (the cycle period jitters by 15-25% even when "stable"). During regime transitions, it spikes above 8 bars. A high σ\_T value is the operational signal that the cycle structure is shifting. Property 2: amplitude persistence. Measure the duration in bars over which the BPF amplitude exceeds a threshold (e.g., the rolling 30-day median amplitude). $$ L\_A(t) \\;=\\; \\text{count}\_{i \\ge 0}\\bigl(\\,A(t-i) > A\_{\\text{floor}}, \\; \\text{contiguous}\\,\\bigr) $$ For SPX, cycle-mode regimes (sustained periods where A > A\_floor) typically last 20-60 bars before the amplitude drops below the floor. Persistence longer than 60 bars is uncommon. Persistence of 1-5 bars indicates the regime is in noise mode and the cycle is not operationally tradeable. Property 3: phase coherence. Measure the standard deviation of observed inter-zero-crossing spacing relative to T(t)/2. $$ \\sigma\_\\phi(t) \\;=\\; \\text{std}\\bigl(2 \\cdot \\Delta\_{\\text{zero-crossing}} - T(t)\\bigr) $$ For SPX cycle-mode regimes, σ\_φ is typically 2-4 bars (the next zero crossing arrives within ±2-4 bars of T/2). For random walk inputs with no cycle structure, σ\_φ would be much larger (variance proportional to the inter-crossing time itself). The measured σ\_φ confirms that markets have *some* cycle structure but not enough to enable precise phase prediction. ## The empirical picture: SPX 2018-2024 Apply the autocorrelation periodogram to SPX daily across the 2018-2024 window. The dominant cycle period series tells the regime story. $$ \\begin{array}{l|c|c|c|c} \\text{Period} & \\text{Avg } T(t) \\text{ (bars)} & \\sigma\_T \\text{ (bars)} & \\text{Avg } A(t) \\text{ (rel)} & \\text{Regime} \\\\ \\hline \\text{Jan 2018 - Dec 2019} & 22.4 & 3.1 & 1.00 & \\text{Cycle mode (stable)} \\\\ \\text{Feb 2020 - Apr 2020} & 9.2 & 2.8 & 1.45 & \\text{Cycle mode (fast cycle)} \\\\ \\text{May 2020 - Dec 2020} & 14.6 & 5.7 & 0.72 & \\text{Mixed / unstable} \\\\ \\text{Jan 2021 - Dec 2021} & 28.5 & 8.4 & 0.41 & \\text{Trend mode (low cycle amplitude)} \\\\ \\text{Jan 2022 - Dec 2022} & 14.2 & 4.2 & 1.18 & \\text{Cycle mode} \\\\ \\text{Jan 2023 - Dec 2024} & 19.8 & 3.6 & 0.85 & \\text{Cycle mode (drifting period)} \\\\ \\end{array} $$ Six regimes in seven years. The 22-bar cycle that held through 2019 was not a permanent feature; it was a property of the 2018-2019 participant mix that happened to be stable for two years. The cycle disappeared in 2021 (the V-recovery's strong directional move suppressed cycle amplitude below the noise floor) and reappeared at a different period in 2022. A strategy keyed to "the 22-bar SPX cycle" had two windows where it could work (2018-2019, partially in 2023-2024) and four windows where it should have been disabled. The trader who left the strategy active across all six regimes paid the lifetime drawdown for that decision. ## Cycle mode vs trend mode The operational classification of every bar into one of two regimes (or a transition): Cycle mode: the autocorrelation periodogram has a clear peak above the noise floor, the period estimate is stable across recent bars (σ\_T < threshold), and the cycle amplitude exceeds the noise floor (A > A\_floor). Cycle-mode strategies (BPF triggers, Hilbert phase, sine-wave indicator) are operational. Trend-mode strategies (MA crossovers, momentum) are likely to generate false signals because the cycle's mean-reversion structure dominates. Trend mode: the autocorrelation periodogram is approximately flat (no dominant peak), or the peak is very broad, and the BPF amplitude is below the noise floor. Cycle-mode strategies generate false signals because there is no cycle to read. Trend-mode strategies (MA-based trend following, decycler trend extraction) are operational. Transition: the periodogram has a peak that is shifting in period, the σ\_T is elevated, or the amplitude is fluctuating around the noise floor. Neither regime is clearly active. The operational stance is to disable both cycle-mode and trend-mode strategies until the regime stabilizes. The gating thresholds: $$ \\begin{array}{l|c} \\text{Regime} & \\text{Gating criteria} \\\\ \\hline \\text{Cycle mode} & \\sigma\_T < 5 \\text{ bars}, \\; A(t) > A\_{\\text{floor}}, \\; L\_A > 10 \\text{ bars} \\\\ \\text{Trend mode} & \\text{periodogram peak-to-floor ratio} < 1.5, \\; A(t) < A\_{\\text{floor}} \\\\ \\text{Transition} & \\sigma\_T > 5 \\text{ bars} \\text{ or amplitude crossing } A\_{\\text{floor}} \\\\ \\end{array} $$ The thresholds are illustrative. Each market and timeframe needs calibration of the floors against historical regime classification, and the calibration itself is a non-trivial estimation problem (the noise floor is non-stationary). ## What this changes in practice Six operational shifts. Operate cycle-mode strategies on a gated basis. The strategy is on when the periodogram peak quality, amplitude, and period stability are above their thresholds. It is off otherwise. The strategy's lifetime P&L attribution should separate "active P&L" (during cycle mode) from "passive P&L" (zero, during trend mode and transitions). Retune cycle-mode parameters bar-by-bar from the live periodogram, not at quarterly review meetings. The article ["Dominant Cycle Estimation Without Astrology"](https://aligrithm.com/dominant-cycle-estimation-without-astrology/) gave the live estimation. The retuning is automatic if the BPF center period, decycler cutoff, and AGC window length all consume the live estimate as input. This is the adaptive indicator framework: the filter parameters are state variables, not hyperparameters. Backtest with explicit regime stratification. Aggregate Sharpe across years is meaningless when 60% of the years are trend mode (strategy idle) and 40% are cycle mode (strategy active). Report Sharpe per regime, with the number of bars in each regime, so the strategy's true edge is visible. The article ["How to Think About Indicator Lag Before Backtesting"](https://aligrithm.com/how-to-think-about-indicator-lag-before-backtesting/) framed this for lag accounting; the same principle applies to regime accounting. Walk-forward validation is mandatory for cycle-based strategies. The cycle structure changes regime within years, not across decades. A strategy validated on 2018-2019 is not validated for 2020-2024 because the regimes are structurally different. Walk-forward with quarterly retraining catches more of the regime variability. Treat trend-mode and cycle-mode as separate strategies in the portfolio. Each has its own activation criteria, its own backtest history, its own drawdown profile. Combining them into a single "regime-adaptive" strategy is operationally valid but the components should be tracked separately for diagnostics. Reject claims of "permanent cycle structure" in market data. Any framework that asserts cycles persist for years or decades (Elliott Wave's long cycles, Gann's annual cycles, K-wave economic cycles) without measurable per-bar verification is making the constancy assumption that the empirical data falsifies. The article ["Dominant Cycle Estimation Without Astrology"](https://aligrithm.com/dominant-cycle-estimation-without-astrology/) framed the falsifiability test. ## Implications for backtesting This article is the natural bridge from Pillar 2 (Indicator Engineering) to Pillar 3 (Robust Systems Lab). The non-stationarity of cycle structure is one specific case of a general principle: market structure itself is non-stationary, and backtesting must account for the non-stationarity at every level. Three direct implications for the next pillar. Implication 1: stationarity matters. The next article in the publication (["Stationarity: The Word Every Trader Ignores Until It Kills the Strategy"](https://aligrithm.com/stationarity-the-word-every-trader-ignores-until-it-kills-the-strategy-2/)) generalizes the cycle-evanescence argument: stationarity is the property all strategies implicitly assume and most markets violate. Indicator engineering produces features; stationarity testing verifies that the feature distribution is stable enough for the strategy to consume. Implication 2: walk-forward is necessary. The article ["Why OOS Failure Is Often a Stationarity Failure"](https://aligrithm.com/why-oos-failure-is-often-a-stationarity-failure/) covers this. Out-of-sample backtests fail when the OOS data is from a regime not represented in the training data. Cycle evanescence is one specific case; volatility regime shifts and macro regime shifts are others. Implication 3: regime detection is part of the strategy. The article ["Volatility Regimes and Strategy Survival"](https://aligrithm.com/volatility-regimes-and-strategy-survival/) covers this. A strategy that does not detect its operating regime cannot disable itself during inappropriate regimes and pays the cost of the misclassified bars. Regime detection is not optional; it is a structural component of any strategy that consumes non-stationary features. The closing thesis of this pillar: indicator engineering produces useful features when the substrate is stationary enough. The substrate is not always stationary. The pillar's machinery is necessary but not sufficient. The next pillar provides the missing piece: the discipline of testing strategies against non-stationarity systematically rather than hoping for the best. ## Decision matrix | Regime | Active strategies | Gating | | -------------------------------- | --------------------------------------------------- | ----------------------------------- | | Cycle mode (stable T, A > floor) | BPF triggers, decycler oscillator, Hilbert phase | σ\_T < 5, L\_A > 10 | | Trend mode (no cycle peak) | MA-based trend following, decycler-based trend | Periodogram flat, A < floor | | Transition (shifting T or A) | None active (idle) | σ\_T > 5 or A near floor | | Multi-cycle (multiple peaks) | Multi-band BPF bank, separate per-period strategies | Multiple sharp peaks in periodogram | | Drift (slow T shift over months) | Adaptive cycle-mode with live retuning | Cycle structure stable per quarter | | Crash (amplitude spike) | Disable all strategies until amplitude settles | A > 3x recent median, σ\_T spike | ## Anti-patterns Five mistakes that show up when evanescence is ignored. Anti-pattern 1: hardcoding a cycle period from one backtest window. The 22-bar SPX cycle from 2018-2019 is not the universal SPX cycle. Hardcoding the period commits the strategy to one regime and breaks during all others. Anti-pattern 2: long-window aggregate Sharpe as the strategy quality metric. A strategy active for 30% of the time (cycle mode) and idle for 70% (other regimes) has its aggregate Sharpe dragged toward zero by the idle bars. Report active-bar Sharpe separately to see the true edge. Anti-pattern 3: treating trend-mode and cycle-mode strategies as substitutes. They are complements. Each operates in regimes where the other should be idle. The portfolio needs both, with regime-aware activation. Anti-pattern 4: not gating on cycle quality. A strategy that runs the same BPF threshold rule in cycle mode and trend mode generates false signals during trend mode (the BPF is producing noise) and misses signals during cycle mode (the threshold is wrong for the current amplitude). Apply AGC and gate on amplitude. Anti-pattern 5: assuming cycle evanescence implies cycles are not real. The cycles are measurable. They have statistical structure. They are operationally tradeable during cycle-mode regimes. The mistake is in assuming permanence; the correction is to add the gating layer, not to abandon cycle-based strategies entirely. ## Visualizing evanescence ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/05/article_53-visual-0.png) ## KEY POINTS - Market cycles exist and are measurable, but they are evanescent: period drifts, amplitude decays, phase loses coherence. Persistent oscillators (like physical pendulums) are not the right model. - Three measurable non-stationary properties define evanescence: period stability σ\_T, amplitude persistence L\_A, phase coherence σ\_φ. - Four structural reasons cycles cannot persist: arbitrage decay (free options get traded away), regime shifts (macro and structural), time-horizon mixing (participant composition changes), volatility-driven amplitude variation. - On SPX 2018-2024, the dominant cycle period passes through six regimes: stable 22-bar (2018-2019), covid 8-bar (early 2020), mixed (mid-2020), trend mode no cycle (2021), 14-bar (2022), drifting 20-bar (2023-2024). - A strategy keyed to "the 22-bar SPX cycle" worked in two of the six regimes. Hardcoding the period commits the strategy to one regime and pays drawdowns in all others. - Cycle-mode strategies (BPF, Hilbert, decycler oscillator) are operational only when σ\_T is low, amplitude is above the noise floor, and the periodogram peak is sharp. Otherwise they generate false signals. - Trend-mode strategies (MA-based trend following, decycler trend extraction) are operational when no cycle dominates and the periodogram is approximately flat. They generate false signals in cycle mode. - Transition regimes (shifting period or amplitude crossing the floor) are idle periods for both strategy families. The right operational stance is "do nothing" until the regime stabilizes. - The cycle period estimate must update bar-by-bar from the live autocorrelation periodogram (covered in ["Dominant Cycle Estimation Without Astrology"](https://aligrithm.com/dominant-cycle-estimation-without-astrology/)). All cycle-mode indicators (BPF, decycler, AGC) consume this estimate as their parameter. - Backtest with regime stratification. Aggregate Sharpe across years averages cycle-mode performance with idle bars. Report Sharpe per regime, bars per regime, drawdown per regime to see the true edge. - Walk-forward validation is mandatory for cycle-based strategies. The cycle structure changes within years, not across decades. A strategy validated on 2018-2019 is not validated for 2020-2024 because the underlying regimes differ. - This article closes the filter-theory block of Pillar 2\. The structural humility: filter machinery only matters when the substrate has structure. The substrate (market structure) is itself non-stationary, which motivates Pillar 3 (Robust Systems Lab) and the next article (["Stationarity: The Word Every Trader Ignores Until It Kills the Strategy"](https://aligrithm.com/stationarity-the-word-every-trader-ignores-until-it-kills-the-strategy-2/)) that generalizes the non-stationarity argument from cycles to all strategy components. ## Don't forget to Sign up Subscribe Email sent! Check your inbox to complete your signup. for the independent research publication on market structure, dynamics, quantitative and systematic trading. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Statistically Sound Indicators for Financial Market Prediction - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Statistically+Sound+Indicators+for+Financial+Market+Prediction+Timothy+Masters&ref=aligrithm.com) - [Cycle Analytics for Traders - John Ehlers (Amazon)](https://www.amazon.com/s?k=Cycle+Analytics+for+Traders+John+Ehlers&ref=aligrithm.com) ### 2.30 Dominant Cycle Estimation Without Astrology URL: https://aligrithm.com/dominant-cycle-estimation-without-astrology/ Last updated: 2026-07-27T13:23:54.000Z A trader reads an analyst's note predicting "a major SPX turn on October 13, 2024" based on a Bradley turn-date model derived from planetary aspects. The note presents the date with confidence and includes a chart of historical Bradley dates "marking" past turns. The trader buys puts ahead of the date. October 13 arrives. SPX closes up 0.4% on no news. The next day SPX is up another 0.6%. The trader's puts expire worthless. The analyst's next note explains that the turn "manifested as a sideways consolidation" and references a new date in November. Two structural problems with this. First, the prediction was not falsifiable: any market behavior on October 13 can be reframed as "the turn manifesting." Second, there exists a precise, falsifiable, signal-processing-based method for estimating the current dominant cycle period of the SPX, and applying it to data through October 10 would have produced a numerical estimate ("dominant cycle currently approximately 27 bars, mid-cycle phase, no zero crossing expected in the next 5-7 bars"). The numerical estimate disagrees with the planetary forecast. The numerical estimate is testable. The planetary forecast is not. This article gives the empirical alternative to cycle astrology. The dominant cycle period of any time series is a measurable quantity with known statistical properties. Four established methods produce numerical estimates with known precision and known failure modes. The estimates are imperfect because real market cycles are weak, time-varying, and often multiple, but the imperfection is itself a measurable property rather than a rationalization. Empirical cycle estimation is not a substitute for cycle prediction; it is a substitute for cycle hallucination. The article ["Band-Pass Filters: The Most Underused Tool in Technical Analysis"](https://aligrithm.com/band-pass-filters-the-most-underused-tool-in-technical-analysis/) introduced the BPF zero-crossing method as a basic cycle measurement. The article ["Decyclers: Extracting Trend by Removing Cycle Energy"](https://aligrithm.com/decyclers-extracting-trend-by-removing-cycle-energy/) used dominant cycle period as a parameter for the decycler. The article ["Automatic Gain Control for Trading Indicators"](https://aligrithm.com/automatic-gain-control-for-trading-indicators/) needed a cycle-period estimate for window-length tuning. All three articles assumed access to a dominant cycle estimate. This article gives the four canonical methods, compares them on real data, and frames the structural limitations the next article in this series (["Why Market Cycles Are Evanescent"](https://aligrithm.com/why-market-cycles-are-evanescent/)) expands on. ## What "dominant cycle" means precisely A dominant cycle is the period T at which the autocorrelation function (or equivalent spectral measure) of a detrended time series has its largest non-trivial peak. Formally: $$ T\_{\\text{dom}}(t) \\;=\\; \\underset{T \\in \[T\_{\\min}, T\_{\\max}\]}{\\arg\\max} \\; \\rho(T; t) $$ where ρ(T; t) is the autocorrelation at lag T computed over a rolling window ending at time t, and the search range \[T\_min, T\_max\] is restricted to operationally meaningful periods (typically 5 to 50 bars for daily-bar trading, excluding trivial peaks at very short lags). The definition is testable. Given a time series, the autocorrelation function is computable in closed form. The peak is well-defined when the function is smooth (after appropriate smoothing). The numerical answer is a single positive real number that can be compared across methods, across time, and across instruments. The definition is not predictive. A dominant cycle estimate at time t reflects the cycle structure of past data ending at t. It does not extrapolate to t+1 or beyond. The article ["No Filter Is Predictive: What Traders Misunderstand About Smoothing"](https://aligrithm.com/no-filter-is-predictive-what-traders-misunderstand-about-smoothing/) framed the general result: any filter on past data produces past summaries, not future forecasts. The cycle estimate is the same kind of object. What the estimate enables: tuning cycle-mode indicators to the current period (the "adaptive indicator" framework), characterizing the current regime (trend mode if no cycle dominates, cycle mode if a clear peak exists), and validating cycle-based strategies on testable measurements rather than on visual chart pattern claims. ## Method 1: BPF zero-crossing The simplest cycle estimator. Apply a wide-band band-pass filter to the detrended price, count the spacing between successive zero crossings of the BPF output, multiply by 2 to get the cycle period. $$ T\_{\\text{measured}} \\;=\\; 2 \\cdot \\bigl(\\text{spacing between successive zero crossings of BPF}(x\_t)\\bigr) $$ The article ["Band-Pass Filters: The Most Underused Tool in Technical Analysis"](https://aligrithm.com/band-pass-filters-the-most-underused-tool-in-technical-analysis/) detailed the BPF construction. For cycle measurement, use a wide bandwidth (δ ≥ 0.5, Q ≤ 2) to avoid the filter's own resonance biasing the measurement toward the BPF's center period. Properties: - Lag: T/4 to T/3 bars (the BPF lag at the cycle frequency) - Precision: ±1 bar typically, ±2-3 bars during regime transitions - Robustness: medium (single noisy bar can shift a zero crossing) - Implementation cost: O(1) per bar after the BPF state is maintained Best for: real-time estimation when computational simplicity matters, secondary cycle confirmation alongside another method. Limitations: the BPF must be tuned to the expected cycle range. If the actual dominant cycle is far outside the BPF's passband, the measurement is biased toward the BPF's center. The wide-band fix sacrifices some selectivity to avoid this bias. ## Method 2: DFT periodogram The classical method. Compute the Discrete Fourier Transform over a rolling window of N bars, take the magnitude squared at each frequency, find the peak. $$ P(f; t) \\;=\\; \\left| \\sum\_{k=0}^{N-1} x\_{t-k} \\cdot e^{-2\\pi j f k} \\right|^2, \\qquad T\_{\\text{dom}} \\;=\\; \\frac{1}{\\arg\\max\_f P(f; t)} $$ For daily SPX with N = 128, the DFT samples 64 frequencies between 0 and Nyquist (0.5 cycles per bar). The frequency resolution is 1/128 = 0.0078 cycles per bar. The period resolution at the 20-bar cycle is approximately 3 bars (period 18 to period 22 are not distinguishable). Coarser than the BPF zero-crossing for typical trading cycles. Properties: - Lag: N/2 (the windowing inherently centers on the window midpoint) - Precision: coarse (period resolution = T²/N at period T) - Robustness: medium (DFT is a linear operation; outliers contaminate the spectrum) - Implementation cost: O(N log N) per bar with FFT, O(N²) naive - Failure mode: spectral dilation (the longer cycles dominate the periodogram even when they are not the structurally relevant cycle) The spectral dilation problem: in financial data the longer cycles (annual, semi-annual) carry vastly more energy than the trading-timescale cycles (15-30 bar). A raw DFT periodogram on SPX returns the annual cycle as "dominant," which is not useful for swing trading. Detrending before the DFT (HPF the input at the longest cycle of interest) is necessary but adds lag and removes information. Best for: long-history analysis where the cycle structure is approximately stationary; spectrum visualization for diagnostics. Limitations: poor frequency resolution at the trading-relevant scales, spectral dilation issues, requires careful detrending preprocessing. ## Method 3: autocorrelation periodogram The preferred method for trading-timescale cycle estimation: the autocorrelation periodogram. The measurement has less latency than a windowed DFT, a wider range of amplitude swings, does not require historical averaging, and does not require spectral calculation compensation. The construction: compute the autocorrelation at lags 1 through L (typically L = 48 for daily-bar trading), then transform to the period domain via a Goertzel-like recursive computation. For each candidate period T, compute the correlation between the input and a sine wave at that period: $$ R(T; t) \\;=\\; \\sum\_{i=0}^{L-1} \\rho\_i(t) \\cdot \\cos\\!\\left(\\frac{2\\pi i}{T}\\right) $$ where ρ\_i(t) is the autocorrelation at lag i. The dominant cycle period is the T that maximizes R(T; t) over the operationally relevant range. Properties: - Lag: approximately L/4 bars (faster than DFT despite using comparable lookback) - Precision: fine (sub-bar period resolution within the chosen range) - Robustness: high (autocorrelation is naturally smoothing; outliers contribute at most 1/L of the autocorrelation magnitude) - Implementation cost: O(L²) per bar with naive computation, O(L) per bar with recursive updates - Spectral dilation: handled by the periodogram's per-period normalization, no preprocessing required The combination of less latency, finer resolution, and structural robustness makes the autocorrelation periodogram the operational default for dominant cycle estimation on trading data. Best for: production cycle estimation feeding adaptive indicators. Limitations: requires sufficient history (L ≥ 32 bars typically) before the first valid estimate. The first L bars after deployment are warmup. ## Method 4: Hilbert-derived instantaneous frequency The most responsive method. Apply the Hilbert transform to the detrended price to produce an analytic signal, compute the instantaneous phase, take the derivative to get the instantaneous frequency, invert to get the period. $$ z(t) \\;=\\; x(t) + j \\cdot \\mathcal{H}\\{x(t)\\}, \\qquad \\phi(t) \\;=\\; \\arg(z(t)), \\qquad T\_{\\text{inst}}(t) \\;=\\; \\frac{2\\pi}{\\phi(t) - \\phi(t-1)} $$ The Hilbert transform produces a 90-degree phase-shifted version of the input. Combined with the original, it forms a complex-valued analytic signal whose phase carries cycle position information. The phase derivative is the instantaneous frequency. Inverting gives the instantaneous period. The article "The Hilbert Transform: Extracting Instantaneous Phase and Amplitude" in this pillar covers the construction in detail. For dominant cycle estimation, the Hilbert method's responsiveness (lag of 1-2 bars when cycles are present) is its primary advantage. Properties: - Lag: 1-2 bars (the fastest of the four methods) - Precision: high during pure cycle mode, low when trend dominates - Robustness: low (the phase derivative is sensitive to noise; smoothing is required) - Implementation cost: O(N) for the FIR Hilbert approximation - Failure mode: phase unwrapping ambiguity (the instantaneous frequency can flip sign when amplitude is small) The modified Hilbert transform used in trading uses an FIR approximation with finite lag (the ideal Hilbert transformer is infinite-length and not realizable). The modified version sacrifices some accuracy for usable lag. Best for: real-time adaptive systems where lag is critical and cycle quality is known to be high. Limitations: requires the input to be in cycle mode; produces garbage during trend mode or low-amplitude periods. Needs to be gated by amplitude (only trust the period estimate when amplitude is above a threshold). ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Comparison on SPX daily Apply all four methods to SPX daily, 2020-2024\. Compare period estimates, lag, and stability. $$ \\begin{array}{l|c|c|c|c} \\text{Method} & \\text{Avg period estimate} & \\text{Estimate std dev} & \\text{Lag (bars)} & \\text{Failure modes} \\\\ \\hline \\text{BPF zero-crossing } (T\_{\\text{center}}{=}20,\\, \\delta{=}0.5) & 23.4 & 4.8 & 5.0 & \\text{BPF resonance bias toward 20} \\\\ \\text{DFT periodogram (N=128, detrended)} & 26.1 & 6.2 & 32.0 & \\text{Spectral dilation if undetrended} \\\\ \\text{Autocorrelation periodogram (L=48)} & 24.7 & 3.1 & 12.0 & \\text{Warmup 48 bars} \\\\ \\text{Hilbert instantaneous (FIR)} & 22.8 & 8.6 & 2.0 & \\text{Phase flip during trend mode} \\\\ \\end{array} $$ Four readings. The BPF zero-crossing is the simplest but has visible resonance bias: the average estimate of 23.4 bars sits close to the BPF center period of 20, suggesting the BPF is pulling the measurement toward its passband. The DFT periodogram with proper detrending produces a comparable mean estimate (26.1 bars) but with poorer resolution and substantially higher lag (32 bars vs the others). The lag makes it unsuitable for adaptive indicator tuning. The autocorrelation periodogram is the best balance: 24.7-bar mean estimate, lowest standard deviation (3.1 bars), and acceptable lag (12 bars). The standard deviation is approximately one-third of the DFT's, indicating much more stable bar-to-bar estimates. The Hilbert method has the lowest lag (2 bars) but the highest standard deviation (8.6 bars). The high variance reflects the phase-derivative noise. Used alone, the Hilbert estimate is too jittery for practical adaptive indicators. Combined with a smoother method (e.g., autocorrelation periodogram for the steady-state estimate and Hilbert for transition detection), it adds responsiveness without sacrificing stability. ## The astrology contrast Three cycle theories are widely circulated but produce no testable measurement. Elliott Wave: claims the market moves in a five-wave impulse and three-wave correction structure with specific Fibonacci ratios between wave lengths. The "wave count" is determined by the analyst's visual interpretation of the chart and is not produced by any algorithm. Multiple wave counts can be drawn on the same data. When a prediction fails, the analyst revises the count post hoc. There is no falsifiable claim about the dominant cycle period that can be compared to the autocorrelation periodogram. Gann theory: uses geometric angles (45 degrees, 1×1, 2×1, etc.) and time cycles derived from astronomical periods to predict turning points. The "law of vibration" is asserted as a master cycle. Cycle periods are claimed but not derived from the data; they are imposed from astronomical or numerological relationships. When applied to actual price data, Gann angles produce visually plausible levels that do not correspond to any spectral peak in the autocorrelation analysis. Bradley turn dates: derived from planetary aspect calculations. The dates are precise (October 13, 2024, for example). The claim that turns occur on these dates is publicly testable. Studies that have tested it find no statistical significance beyond chance, but the dates continue to circulate in trading commentary because individual hits are memorable and individual misses are excused. The structural failure mode of all three: they make claims about cycle structure that cannot be validated against the autocorrelation periodogram or any other measurable spectral quantity. They are not theories of market cycles; they are pattern recognition in the absence of measurement. This article is not a debunking of all "cycle" claims. Markets do have measurable cycles (the autocorrelation periodogram on SPX has clear peaks in the 15-35-bar range). The claim is that the empirical cycle structure is the thing worth measuring, and the empirical methods give answers that disagree with the astrology methods. ## What this changes in practice Five operational shifts. Default to the autocorrelation periodogram for dominant cycle estimation. The construction has the right balance of latency, precision, and robustness for production use. The Hilbert method serves as a low-latency complement when transition detection matters. Feed the cycle estimate to all cycle-mode indicators. The article ["Band-Pass Filters: The Most Underused Tool in Technical Analysis"](https://aligrithm.com/band-pass-filters-the-most-underused-tool-in-technical-analysis/) used the cycle period as the BPF's center parameter. The article ["Decyclers: Extracting Trend by Removing Cycle Energy"](https://aligrithm.com/decyclers-extracting-trend-by-removing-cycle-energy/) used it as the decycler's cutoff. The article ["Automatic Gain Control for Trading Indicators"](https://aligrithm.com/automatic-gain-control-for-trading-indicators/) used it for the AGC window length. Each indicator becomes adaptive when fed the live estimate. Report cycle estimates with confidence intervals. The autocorrelation periodogram's peak has a width that indicates confidence: a sharp peak means high confidence in the period estimate, a broad peak means low confidence. Strategies that consume the estimate condition their behavior on the confidence: only act when confidence is high, default to trend-mode logic when confidence is low. Gate cycle-mode strategies by the periodogram peak quality. When no clear peak exists (the periodogram is approximately flat across the period range), the market is in trend mode and cycle-mode strategies should be disabled. The peak-vs-floor ratio of the periodogram is the gating metric. Reject all "cycle" claims that do not produce a numerical period estimate. Elliott Wave, Gann, Bradley, lunar cycles, and similar frameworks claim cycle structure without specifying which spectral peak they reference. The rejection is operational, not philosophical: if a cycle framework cannot produce a number that the autocorrelation periodogram can confirm or refute, it cannot inform adaptive indicator tuning and cannot be backtested. ## Decision matrix | Use case | Recommended method | Reason | | ---------------------------------------- | ----------------------------------------- | ---------------------------------------------- | | Production adaptive indicator tuning | Autocorrelation periodogram | Best balance of latency, precision, robustness | | Real-time transition detection | Hilbert instantaneous + autocorr fallback | Hilbert speed, autocorr stability | | Diagnostic spectrum visualization | DFT periodogram (with detrending) | Full spectrum view | | Quick check / lightweight implementation | BPF zero-crossing (wide-band) | O(1) per bar | | Multi-cycle decomposition | DFT or comb filter bank | Resolve multiple peaks | | Cycle mode vs trend mode gate | Autocorrelation periodogram peak quality | Peak-to-floor ratio | | Cross-instrument cycle comparison | Autocorrelation periodogram | Consistent definition across instruments | | Backtesting cycle-period regime regimes | Autocorr periodogram at every bar | Reproducible numerical history | ## Anti-patterns Five mistakes that show up in cycle-based trading. Anti-pattern 1: trading off Elliott Wave counts, Gann angles, Bradley turn dates, or any other cycle framework that does not produce a numerical period that can be validated against the autocorrelation periodogram. The frameworks are unfalsifiable, and the trading edge (if any) is indistinguishable from random pattern recognition. Anti-pattern 2: assuming "the dominant cycle is 20 bars" as a constant. Real cycle periods drift over weeks and months. A static cycle assumption breaks adaptive indicators precisely when they are most needed (regime transitions). Anti-pattern 3: using DFT without detrending on raw price data. The annual and semi-annual cycles dominate the spectrum and obscure the trading-relevant 15-35-bar peaks. The fix: HPF the input at the longest cycle of interest before the DFT. Anti-pattern 4: trusting the Hilbert period estimate when the cycle amplitude is low. The phase derivative becomes noise-dominated during trend mode. Gate the Hilbert estimate by an amplitude threshold; use the autocorrelation periodogram as fallback when amplitude is low. Anti-pattern 5: treating the cycle period estimate as a prediction. The estimate describes the current cycle structure based on past data. It does not predict when the next peak or trough will occur. Strategies that condition on "the next peak is N bars away" are extrapolating from a backward-looking estimate. ## Visualizing the empirical cycle ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/05/article_52-visual-0.png) ## KEY POINTS - The dominant cycle period is a measurable quantity: the period T at which the autocorrelation function of the detrended series has its largest non-trivial peak. Computable, falsifiable, comparable across methods. - Four established methods produce numerical estimates with known tradeoffs: - BPF zero-crossing: simplest, O(1), 5-bar lag, sensitive to BPF resonance bias - DFT periodogram: classical, 32-bar lag, coarse resolution, spectral dilation issues - Autocorrelation periodogram: preferred default, 12-bar lag, fine resolution, robust - Hilbert instantaneous: lowest 2-bar lag, high variance, requires amplitude gating - The autocorrelation periodogram is the operational default. Best balance of latency, precision, and robustness. - The Hilbert method is the complement for low-latency transition detection. Use both: autocorr for steady-state, Hilbert for change detection. - On SPX 2020-2024: dominant cycle period estimates cluster around 24-26 bars with std deviations of 3-9 bars depending on method. - Cycle estimates are not predictive. They describe current cycle structure based on past data. They do not extrapolate to when the next peak or trough will arrive. - Cycle estimates feed adaptive indicators: BPF center period, decycler cutoff, AGC window length all update bar-by-bar from the live estimate. - Gate cycle-mode strategies by the periodogram peak quality. When no clear peak exists, the market is in trend mode and cycle-mode logic should be disabled. - Elliott Wave, Gann angles, Bradley turn dates, and similar frameworks make cycle claims that do not produce numerical period estimates testable against the autocorrelation periodogram. They are pattern recognition in the absence of measurement. - Reject cycle frameworks that cannot specify which spectral peak they reference. The test is operational: if the framework cannot produce a number, it cannot inform adaptive indicators or backtests. - Real market cycles are weak, time-varying, and often multiple. The empirical estimate is imperfect but the imperfection is itself a measurable property. The next article in this series (["Why Market Cycles Are Evanescent"](https://aligrithm.com/why-market-cycles-are-evanescent/)) expands on the persistence and stability properties. - The astrology contrast is operational, not philosophical: if a cycle theory cannot disagree with the autocorrelation periodogram, it does not inform any trading decision the periodogram cannot inform more honestly. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Statistically Sound Indicators for Financial Market Prediction - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Statistically+Sound+Indicators+for+Financial+Market+Prediction+Timothy+Masters&ref=aligrithm.com) - [Cycle Analytics for Traders - John Ehlers (Amazon)](https://www.amazon.com/s?k=Cycle+Analytics+for+Traders+John+Ehlers&ref=aligrithm.com) ### 2.29 Automatic Gain Control for Trading Indicators URL: https://aligrithm.com/automatic-gain-control-for-trading-indicators/ Last updated: 2026-07-27T13:23:53.000Z A trader tunes an RSI(14) mean-reversion strategy on SPY 2017 data. The rule: short when RSI crosses above 70, cover at 50\. The backtest shows 22 winning trades, 4 losing trades, Sharpe 1.6 over the calendar year. The trader ships the strategy and runs it through March 2020\. RSI hits 70 on March 2, 75 on March 4, 80 on March 6, and stays above 70 for eleven consecutive sessions while SPY drops from 311 to 219\. By the time RSI finally drops back below 70 on March 23, the strategy has lost three months of accumulated profit on a single position. The trader concludes the strategy "needs different thresholds for high-vol regimes" and adds a VIX filter that disables the strategy when VIX > 30\. The patch works but the underlying problem is not fixed. The same diagnosis applies to BPF outputs, MACD readings, close-minus-MA distances, Stochastic levels, and every other oscillator-based threshold rule. The indicator's output amplitude scales with the input's volatility. A threshold tuned for one volatility regime is wrong for any other regime by construction. Automatic gain control fixes the problem at the structural level. Instead of applying fixed thresholds to an indicator whose amplitude varies with volatility, normalize the indicator's amplitude continuously so the output stays in a fixed range regardless of the input regime. Threshold rules applied to the AGC output are regime-independent. The same RSI > 70 rule that worked in 2017 works in March 2020 because the AGC has rescaled both periods to a comparable amplitude range. The article ["Why ATR Normalization Is More Than a Volatility Trick"](https://aligrithm.com/why-atr-normalization-is-more-than-a-volatility-trick/) gave the structurally optimal scaling for price-unit features (divide by ATR with the √k factor). The article ["Taming Indicator Tails with Sigmoid Transforms"](https://aligrithm.com/taming-indicator-tails-with-sigmoid-transforms/) gave the bounded-but-monotone scaling for heavy-tailed features (sigmoid into the linear region). AGC is the third member of this family: bounded, regime-adaptive, designed for indicators that need consistent visual amplitude across all market conditions. This article gives the constructions, the structural caveats, and the operational discipline. The next article in this series (["Dominant Cycle Estimation Without Astrology"](https://aligrithm.com/dominant-cycle-estimation-without-astrology/)) covers how AGC enables stable cycle measurements across regimes. ## The structural problem A linear filter applied to price preserves the price's amplitude in its output. For an SPX BPF centered at T = 20: $$ y\_{\\text{BPF}}(t) \\;\\approx\\; A\_{\\text{cycle}}(t) \\cdot \\cos(\\omega t + \\phi) $$ The output amplitude A\_cycle(t) is proportional to the underlying cycle amplitude of the input. When SPX cycles 1% peak-to-trough, the BPF output ranges in ±0.5%. When SPX cycles 6% peak-to-trough, the BPF output ranges in ±3%. A trader who set a "buy when BPF < −0.5" threshold based on 2017 data triggers continuously in March 2020 and never triggers in summer 2023. The same property holds for every linear indicator covered in the prior articles in this series. The article ["The Frequency Response of Trading Indicators"](https://aligrithm.com/the-frequency-response-of-trading-indicators/) gave the universal framework: a linear filter has unit gain (or some constant gain) at its passband frequency. The output amplitude at the passband is the input amplitude multiplied by the gain. Threshold rules that ignore the regime-dependent input amplitude are reading the indicator wrong by construction. Three options to fix this. Option 1: change the threshold for each regime (regime-detection layer + threshold lookup). Operationally fragile. Option 2: replace the threshold rule with a relative comparison (current value vs rolling mean of the same indicator). Better, but still depends on the rolling window length. Option 3: rescale the indicator's amplitude continuously to a fixed range. This is the AGC approach. ## The basic AGC construction The simplest AGC normalizes by the rolling maximum absolute value: $$ y\_{\\text{AGC}}(t) \\;=\\; \\frac{x(t)}{\\max\_{0 \\le i < N} |x(t-i)|} $$ The output is bounded in \[-1, 1\] by construction. The denominator is the largest absolute value the indicator has reached in the rolling window. The current value, divided by this maximum, places the current reading on a \[-1, 1\] scale relative to the window's extremes. For an SPX BPF output: in March 2020 the window's max is approximately 3% (the largest BPF swing). The current BPF value of −2% becomes (−2/3) = −0.667 after AGC. In summer 2023 the window's max is approximately 0.4%. The same proportional cycle position produces the same AGC value of approximately −0.667 even though the raw BPF value is now −0.27%. The AGC output is the cycle's *relative position* within the current regime, not its absolute amplitude. Threshold rules on the AGC output become regime-independent. "Buy when AGC(BPF) < −0.7" fires when the BPF reaches 70% of its recent peak excursion, whether the recent peak excursion is 3% or 0.4%. ## The RMS-based AGC Max-abs normalization has one structural flaw: it is dominated by the single largest outlier in the window. One news-day spike contaminates the denominator for N bars. The "even better sine wave" construction uses an RMS-based normalization that distributes the influence across all bars in the window: $$ y\_{\\text{AGC-RMS}}(t) \\;=\\; \\frac{x(t)}{\\sqrt{\\frac{1}{N} \\sum\_{i=0}^{N-1} x(t-i)^2}} $$ The denominator is the rolling RMS (root mean square) of the indicator's absolute amplitude. The output is centered at zero with approximately unit-variance scaling. Unlike max-abs, the RMS averages the squared amplitudes across the window, so a single outlier contributes 1/N of its squared value rather than fully determining the denominator. The RMS AGC output is not strictly bounded by \[-1, 1\] (a value at 3σ amplitude registers as +3 after RMS normalization), but it is centered on zero with consistent scale across regimes. Threshold rules use multiples of unit amplitude: "buy when AGC-RMS < −1.5" maps to "buy when current indicator is 1.5x its recent RMS amplitude on the negative side." The choice between max-abs and RMS depends on the indicator. For BPF outputs and other oscillators with bounded natural amplitude, max-abs gives a cleaner \[-1, 1\] bounded output. For indicators that occasionally produce large spikes (close-minus-MA at major turning points, momentum during news days), RMS is more stable. ## Comparison to alternative scaling methods Five scaling approaches cover the indicator-normalization space. Each has structural trade-offs. $$ \\begin{array}{l|c|c|c} \\text{Method} & \\text{Output range} & \\text{Outlier sensitivity} & \\text{Best for} \\\\ \\hline \\text{Max-abs AGC} & \\text{\[-1, 1\] strict} & \\text{High (single outlier dominates)} & \\text{Bounded oscillators, BPF} \\\\ \\text{RMS AGC} & \\text{Centered, scale 1} & \\text{Moderate (quadratic)} & \\text{Indicators with occasional spikes} \\\\ \\text{Median-abs AGC} & \\text{\[-k, k\] typical} & \\text{Low (robust)} & \\text{Heavy-tailed indicators} \\\\ \\text{Z-score} & \\text{Centered, scale 1} & \\text{High (mean and std)} & \\text{Approximately Gaussian indicators} \\\\ \\text{Robust z-score (median, MAD)} & \\text{Centered, scale \~1} & \\text{Low} & \\text{Heavy-tailed indicators} \\\\ \\text{Percentile rank} & \\text{\[0, 1\] strict} & \\text{Low} & \\text{Indicators where rank matters more than magnitude} \\\\ \\text{Sigmoid (tanh)} & \\text{(-1, 1) asymptotic} & \\text{Low} & \\text{Heavy tails, see ["Taming Indicator Tails with Sigmoid Transforms"](https://aligrithm.com/taming-indicator-tails-with-sigmoid-transforms/)} \\\\ \\text{ATR normalization} & \\text{Scaled by ATR} & \\text{Moderate} & \\text{Price-unit features, see ["Why ATR Normalization Is More Than a Volatility Trick"](https://aligrithm.com/why-atr-normalization-is-more-than-a-volatility-trick/)} \\\\ \\end{array} $$ Eight methods. The choice depends on the indicator's structural properties. For cycle-mode indicators (BPF outputs, Hilbert sine, sine-wave indicator), max-abs AGC or RMS AGC is the standard choice. The output is naturally bounded by the cycle's amplitude, so max-abs preserves the \[-1, 1\] interpretation. For indicators with rare outliers, RMS or median-abs AGC gives a more stable denominator. For statistical features that approximate Gaussian behavior, z-score is appropriate. For heavy-tailed features (most real trading features), robust z-score using median and MAD is the better choice. The article ["Why the Median Often Beats the Mean in Trading Features"](https://aligrithm.com/why-the-median-often-beats-the-mean-in-trading-features/) detailed the robust-statistics argument. For price-unit features (price minus MA, momentum), ATR normalization is structurally optimal because it accounts for the random-walk √k factor in the variance scaling. The article ["Why ATR Normalization Is More Than a Volatility Trick"](https://aligrithm.com/why-atr-normalization-is-more-than-a-volatility-trick/) detailed the construction. ## Structural issues with AGC Three issues that show up when AGC is applied without care. Issue 1: window length tradeoff. Too short (N < 10) makes the denominator noisy and the AGC output unstable. Too long (N > 100) makes the AGC slow to adapt to regime changes. The article ["How to Think About Indicator Lag Before Backtesting"](https://aligrithm.com/how-to-think-about-indicator-lag-before-backtesting/) frames this as a lag-budgeting question: the AGC denominator carries the lag of an N-bar smoother on the absolute-value channel. For SPX daily, N = 40 to N = 100 is the typical operational range. Shorter windows produce visible jitter in the AGC output. Longer windows lag the regime change so the AGC output continues to use 2017-style scaling for 2020-style price action for several weeks after the regime shift. Issue 2: max-abs is dominated by outliers. A single news-day spike sets the denominator for N bars. After the spike, the AGC output is compressed (everything looks smaller than it is) until the spike rolls out of the window. Solutions: switch to RMS or median-abs, or apply a Hampel pre-filter (covered in ["Why Median Filters Are Useful for Volume and Outliers"](https://aligrithm.com/why-median-filters-are-useful-for-volume-and-outliers/)) to remove spikes from the absolute-value series. Issue 3: memory effects. The AGC denominator is a function of past values. After a true regime shift (sustained volatility expansion), the denominator takes N bars to fully reflect the new regime. During the transition, the AGC output gives mixed-regime readings that are not directly comparable to either the old or new regime's typical readings. The mitigation: use a faster AGC window during anticipated regime transitions (e.g., during VIX expansion phases), and a slower window during stable regimes. The dual-window approach is operationally simple but adds complexity to the indicator pipeline. ## Robust AGC hybrids Three hybrid constructions improve on the basic AGC for specific failure modes. Hybrid 1: median-abs AGC. Replace the max-abs denominator with the rolling median of |x|. The median is robust to single-spike contamination (covered in ["Why Median Filters Are Useful for Volume and Outliers"](https://aligrithm.com/why-median-filters-are-useful-for-volume-and-outliers/)), so the AGC denominator is stable even when the indicator has occasional outlier readings. $$ y\_{\\text{AGC-med}}(t) \\;=\\; \\frac{x(t)}{k \\cdot \\text{median}\_N(|x(t-i)|)}, \\qquad k \\approx 1.4826 \\text{ (for Gaussian scale equivalence)} $$ The factor k makes the median-abs denominator approximately equal to one standard deviation for Gaussian inputs. The construction is the most robust of the AGC family. Hybrid 2: EMA-of-abs AGC. Replace the rolling max-abs with an EMA of |x|. The denominator becomes a smoothly updating estimate of the typical absolute amplitude rather than a hard maximum: $$ \\hat{a}(t) \\;=\\; \\alpha \\cdot |x(t)| + (1 - \\alpha) \\cdot \\hat{a}(t-1), \\qquad y\_{\\text{AGC-EMA}}(t) \\;=\\; \\frac{x(t)}{\\hat{a}(t)} $$ The output is unbounded (extreme values produce extreme AGC readings) but the denominator transitions smoothly across regimes. The EMA's lag is the standard EMA lag at the equivalent lookback (covered in ["The Hidden Cost of Every Moving Average: Lag"](https://aligrithm.com/the-hidden-cost-of-every-moving-average-lag/)). Hybrid 3: Hampel-pre-filtered AGC. Apply a Hampel filter to the indicator output first to remove outlier spikes, then apply standard AGC to the cleaned series. The AGC denominator is not contaminated by spikes because the spikes are removed before scaling. The choice among hybrids depends on the data class. For volume-weighted indicators or true-range-based indicators, Hampel-pre-filtered AGC is the default (the underlying data has known outlier structure). For BPF cycle outputs on clean price data, max-abs or RMS AGC is sufficient. For long-tail indicators (close-minus-MA at turning points), median-abs AGC is the most robust. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Worked example: SPX BPF across regimes SPX daily. Apply a BPF(T=20, δ=0.3) and compare four AGC variants across three volatility regimes. $$ \\begin{array}{l|c|c|c|c} \\text{Period} & \\text{VIX avg} & \\text{BPF raw amplitude} & \\text{No AGC} & \\text{AGC output range} \\\\ \\hline \\text{Calendar 2017} & 11.1 & \\pm 0.4\\% & \\text{\[-0.4\\%, 0.4\\%\]} & \\text{\[-1, 1\]} \\\\ \\text{Mar 2020 (covid)} & 57.7 & \\pm 3.1\\% & \\text{\[-3.1\\%, 3.1\\%\]} & \\text{\[-1, 1\]} \\\\ \\text{Calendar 2023} & 17.0 & \\pm 0.9\\% & \\text{\[-0.9\\%, 0.9\\%\]} & \\text{\[-1, 1\]} \\\\ \\end{array} $$ The raw BPF amplitude varies by a factor of approximately 8 across the three regimes. A "buy when BPF < −0.5%" rule fires 100% of the time in March 2020 (every bar) and 0% of the time in 2017 (the BPF never reaches −0.5% in that period). After AGC normalization, the same proportional cycle position produces the same AGC value in all three regimes. A "buy when AGC < −0.7" rule fires at structurally comparable points across all three periods. Now compare the AGC variants on the same data. $$ \\begin{array}{l|c|c|c|c} \\text{AGC variant} & \\text{Bounded?} & \\text{Outlier impact} & \\text{Lag to regime shift} & \\text{Best use} \\\\ \\hline \\text{Max-abs(N=60)} & \\text{\[-1, 1\]} & \\text{High (single spike sets denom 60 bars)} & \\text{Slow (denom fixed at max)} & \\text{Clean oscillators} \\\\ \\text{RMS(N=60)} & \\text{Soft (typically \[-3, 3\])} & \\text{Moderate (quadratic)} & \\text{Medium} & \\text{Occasional spikes} \\\\ \\text{Median-abs(N=60)} & \\text{Soft (typically \[-3, 3\])} & \\text{Low} & \\text{Medium} & \\text{Heavy-tailed indicators} \\\\ \\text{EMA-of-abs(} \\alpha{=}0.04) & \\text{Unbounded} & \\text{Moderate (decays)} & \\text{Fast} & \\text{Real-time regime adaptation} \\\\ \\end{array} $$ The trade-off: max-abs gives the cleanest bounded output but is the slowest to adapt to regime shifts. EMA-of-abs adapts fastest but produces unbounded output. RMS and median-abs sit in the middle, with median-abs being the most robust choice for indicators with heavy-tailed behavior. For most cycle-mode trading applications, RMS(N=60) or median-abs(N=60) is the default. Max-abs is reserved for indicators that need strict \[-1, 1\] bounds for downstream consumption (visual display, fixed-bound classifiers). ## Where AGC fails Four scenarios where AGC produces misleading or unhelpful output. Scenario 1: indicators with no natural zero. AGC assumes the indicator is centered at zero (positive and negative excursions roughly balanced). For indicators with DC offset (raw price, cumulative sums), AGC produces garbage. Apply a centering operation (close-minus-MA, HPF) first to remove the DC, then AGC. Scenario 2: when absolute magnitude matters. AGC throws away the absolute amplitude information by construction. If the strategy needs to know "is the current cycle large or small in absolute terms," AGC removes the answer. Use a parallel raw-amplitude channel alongside the AGC channel. Scenario 3: regime-shift outliers. A sudden volatility expansion that lasts longer than N bars causes the AGC to gradually rescale to the new regime. During the transition (first N bars of the new regime), the AGC output mixes old and new regime scales. Threshold rules during this transition are ambiguous. Scenario 4: cross-instrument comparisons. AGC normalizes within an instrument's own volatility regime. Comparing AGC outputs across SPX and crude oil produces relative readings within each instrument's history, not directly comparable readings. For cross-instrument comparison, use a structurally cross-comparable normalization like ATR (covered in ["Why ATR Normalization Is More Than a Volatility Trick"](https://aligrithm.com/why-atr-normalization-is-more-than-a-volatility-trick/)) or rank-based normalization. ## What this changes in practice Five operational shifts. Apply AGC to every threshold-based oscillator rule. The fix is one extra pipeline stage and removes the regime-dependence of fixed thresholds. RSI thresholds (70/30), BPF triggers, MACD signals all benefit from AGC normalization before the threshold is applied. Default to RMS or median-abs AGC for production. Max-abs AGC is the textbook construction but is fragile to single outliers. RMS or median-abs gives comparable bounding with much better robustness. Match the AGC window length to the regime persistence. For markets with rapid regime shifts (crypto, leveraged products), use shorter windows (N = 20-40). For markets with slower regime evolution (large-cap equity indices), use longer windows (N = 60-100). Report indicator readings paired with the AGC method explicit. "RSI(14) AGC-RMS(60)" is unambiguous. "RSI(14)" without disclosure of the scaling is regime-ambiguous. Audit existing strategies for "fixed thresholds on raw oscillator" anti-patterns. Any strategy with hardcoded threshold values on RSI, MACD, Stochastic, BPF, or any other amplitude-sensitive indicator is at risk of regime-shift breakdown. The fix is to insert AGC between the indicator and the threshold. ## Decision matrix | Use case | Recommended AGC | Reason | | ---------------------------------------------------- | ------------------------------ | ------------------------------- | | BPF cycle oscillator on equity index | Max-abs or RMS, N = 60 | Bounded output for clean visual | | Cycle-mode features with occasional spikes | RMS, N = 60 | Quadratic robustness | | Heavy-tailed indicators (close-minus-MA at extremes) | Median-abs, N = 60 | Robust to outliers | | Volume-weighted indicators | Hampel pre-filter + RMS | Volume has heavy right tail | | True-range smoothing | Hampel pre-filter + median-abs | TR spikes on news days | | Real-time regime adaptation needed | EMA-of-abs, α = 0.05 | Fast adaptation, smooth update | | Cross-instrument comparison | Not AGC; use ATR or rank | AGC is within-instrument only | | Raw oscillator with DC offset | Center first (HPF), then AGC | AGC assumes zero-centered input | | Need absolute amplitude information | AGC + parallel raw channel | AGC discards amplitude | ## Anti-patterns Five mistakes when applying AGC. Anti-pattern 1: fixed thresholds on raw oscillator output. RSI > 70 / RSI < 30 rules are regime-dependent. Apply AGC first, then threshold. Anti-pattern 2: using max-abs AGC on indicators with heavy-tailed behavior. A single outlier sets the denominator for N bars and compresses the AGC output. Use RMS or median-abs instead. Anti-pattern 3: too short an AGC window. N < 10 produces noisy denominators that make the AGC output jitter independently of the underlying signal. Use N ≥ 20. Anti-pattern 4: AGC on an uncentered indicator. AGC assumes the input is zero-centered. For DC-biased inputs (raw price, cumulative volume), AGC produces garbage. Center first with an HPF or a close-minus-MA construction. Anti-pattern 5: cross-instrument comparison via AGC. AGC normalizes within each instrument's own history. The same AGC value on SPX and crude oil means "70% of the recent extreme in this instrument," not "comparable cycle states." Use ATR-based normalization or cross-sectional ranking instead. ## Visualizing AGC ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/05/article_51-visual-0.png) ## KEY POINTS - Linear indicators preserve their input's amplitude. A BPF on SPX in 2017 (low vol) produces outputs in ±0.4%. The same BPF in March 2020 (high vol) produces outputs in ±3.1%. The 8x amplitude difference breaks any fixed-threshold rule. - AGC continuously rescales the indicator output so the amplitude stays in a fixed range regardless of input volatility. Threshold rules on the AGC output are regime-independent. - Basic AGC: divide the current indicator value by the rolling max of the absolute value over N bars. Output is bounded in \[-1, 1\]. - RMS AGC: divide by the rolling RMS instead of max-abs. Output is centered with approximately unit-variance scaling. More robust to single-spike contamination. - Median-abs AGC: divide by the rolling median of |x|. Most robust of the AGC family, suited for heavy-tailed indicators. - EMA-of-abs AGC: smooth recursive update of the denominator. Fastest adaptation to regime shifts, but unbounded output. - AGC is one of three structural normalization families: AGC for oscillators, ATR for price-unit features, sigmoid for heavy-tailed indicators with bounded targets. - Three structural issues with AGC: window length tradeoff (short = noisy, long = lagging), max-abs outlier sensitivity, memory effects during regime transitions. - Anti-pattern: applying AGC to uncentered indicators (raw price, cumulative sums). AGC assumes zero-centered input. Apply HPF or close-minus-MA first. - Anti-pattern: cross-instrument comparison via AGC. AGC is within-instrument only. For cross-instrument comparison use ATR or cross-sectional ranking. - Default AGC choice for production: RMS(N=60) or median-abs(N=60) on cycle-mode indicators. Max-abs is reserved for strict \[-1, 1\] bounding when required. - Existing strategies with hardcoded thresholds on raw RSI, MACD, Stochastic, BPF outputs are at structural risk of regime-shift breakdown. The fix is one pipeline stage: insert AGC before the threshold. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Statistically Sound Indicators for Financial Market Prediction - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Statistically+Sound+Indicators+for+Financial+Market+Prediction+Timothy+Masters&ref=aligrithm.com) - [Cycle Analytics for Traders - John Ehlers (Amazon)](https://www.amazon.com/s?k=Cycle+Analytics+for+Traders+John+Ehlers&ref=aligrithm.com) ### 2.28 Why Median Filters Are Useful for Volume and Outliers URL: https://aligrithm.com/why-median-filters-are-useful-for-volume-and-outliers/ Last updated: 2026-07-27T13:23:51.000Z A trader builds a volume-weighted moving average to confirm trend signals. The construction: weight each close by its volume, smooth with a 20-bar mean. On a typical day, volume sits in a 1x to 1.5x range relative to its 20-day median. On August 5, 2024 (the SPX volatility-spike day), volume prints at 4.2x the prior week's typical level. The VWMA reading for the next 20 trading days carries the August 5 print as a heavyweight observation. Strategy signals during August 6 through September 5 are dominated by what happened on one day three weeks earlier. The trader investigates and concludes the strategy "needs better volume filtering." The instinct is right; the diagnosis is wrong. The strategy does not need better filtering. It needs a different filter class. Every linear filter (SMA, EMA, WMA, super-smoother, Hann window, decycler) integrates every input bar including outliers. The August 5 spike is integrated into the VWMA for the full 20-bar window because that is what a linear filter does. The fix is not a longer window or a different linear smoother. The fix is to replace the mean operation with a median operation, which by construction rejects the spike. Volume data, true range data, range-based volatility estimates, and tick-by-tick price prints all share a structural property: they have heavy right tails punctuated by isolated outliers (news days, halt events, fat-finger prints, exchange glitches). Linear filters are the wrong tool for these inputs. The article ["Why the Median Often Beats the Mean in Trading Features"](https://aligrithm.com/why-the-median-often-beats-the-mean-in-trading-features/) established the robust-statistics case for the median as a point estimator. This article extends the argument to *filters*: rolling-window operations that produce time-series outputs. The same breakdown-point math applies, with an additional structural property that linear filters do not have. The article ["The Frequency Response of Trading Indicators"](https://aligrithm.com/the-frequency-response-of-trading-indicators/) gave the universal framework for analyzing linear indicators. Median filters do not have a transfer function (the operation is nonlinear), so the framework does not directly apply. This article gives the complementary framework for nonlinear filters: characterize by impulse response, edge response, and effective lag, not by frequency response. The next article in this series (["Automatic Gain Control for Trading Indicators"](https://aligrithm.com/automatic-gain-control-for-trading-indicators/)) covers another nonlinear construction designed to handle range expansion. ## The structural problem Linear filters compute weighted sums of input bars: $$ y\_t \\;=\\; \\sum\_{i=0}^{N-1} w\_i \\cdot x\_{t-i}, \\qquad \\sum w\_i = 1 \\text{ (for unit DC gain)} $$ For an SMA the weights are all 1/N. For an EMA the weights decay exponentially. For a Hann window the weights are sinusoidal. In every case, every input bar contributes to the output through its weight, and an outlier at any bar contributes a weight × outlier\_value term to the output. The contamination scales with the outlier magnitude divided by the effective N. For volume that spikes 10x during a news day, the SMA(20) of volume carries an extra 0.5x of the baseline in its output (10x weighted by 1/20 = 0.5x). The output is biased upward by 50% for 20 bars, then drops back. The "smoothed volume" reading misrepresents the typical volume for the entire post-spike window. Median filters do not integrate. They select. $$ y\_t \\;=\\; \\text{median}(x\_t, x\_{t-1}, \\dots, x\_{t-N+1}) \\;=\\; x\_{(\\lceil N/2 \\rceil)} \\text{ of the sorted window} $$ For an odd N, the output is the single middle value of the sorted window. For an even N, the output is the average of the two middle values. The construction has no weights and no sum. A single outlier in the window does not contribute to the output unless its rank places it at the median position. For a window of 20, an outlier remains a single sorted-list entry until 10 other extreme values join it, which requires the outlier to no longer be an outlier. ## The breakdown point The robust-statistics framing from ["Why the Median Often Beats the Mean in Trading Features"](https://aligrithm.com/why-the-median-often-beats-the-mean-in-trading-features/) applies directly. The breakdown point of an estimator is the maximum fraction of contaminated data the estimator can absorb while remaining bounded. $$ \\text{Breakdown point}\_{\\text{mean}} \\;=\\; 0\\%, \\qquad \\text{Breakdown point}\_{\\text{median}} \\;=\\; 50\\% $$ Mean: a single outlier of arbitrary magnitude can drag the mean to an arbitrary value. The breakdown point is zero. Median: the median changes only when more than half the window is contaminated in the same direction. The breakdown point is 50%. For a rolling-window filter the same property holds bar-by-bar. The SMA(20) is contaminated by every outlier it sees, with linear scaling in magnitude. The median(20) is contaminated only when more than 10 outliers enter the window together. For real volume and true-range data, isolated spikes (1 to 3 outliers per month) do not cluster enough to break the median. The median filter's output is structurally immune to typical news-day contamination. ## What characterizes a median filter Median filters are nonlinear. The framework from ["The Frequency Response of Trading Indicators"](https://aligrithm.com/the-frequency-response-of-trading-indicators/) does not apply because the operation is not a linear combination of input bars. The transfer function H(z) is undefined. The substitution z⁻¹ = exp(−2πjf) produces no meaningful magnitude curve. Three properties characterize median filters in place of the linear framework. Impulse response: how does the output react to a single spike at time t? For a linear filter, the impulse response is the filter's coefficient vector (the spike propagates for N bars with weights w\_i). For a median filter, an isolated spike causes zero output change if the window is at least 3 bars (the spike is filtered out by ranking). For multiple adjacent spikes, the output change is proportional to how many spikes fit in the window. Edge response: how does the output transition at a step change in the input? For a linear filter, the output transitions smoothly over N bars (the step is convolved with the filter weights). For a median filter, the output transitions sharply when the post-step bars outnumber the pre-step bars in the window. The transition is N/2 bars, but it happens cleanly without smearing. Effective lag: how late is the output relative to a current input change? For a median filter of window N, the effective lag is approximately N/2 bars, comparable to an SMA(N). The lag is the same; the contamination behavior is different. The combination: median filters have similar lag to linear filters of the same window, but they reject impulses and preserve edges. The trade is signal characteristics: linear filters smooth gradients but smear spikes and edges; median filters preserve edges but ignore smooth low-amplitude variation. ## Why volume data benefits most Volume data has three structural properties that make it the worst-case input for linear filters and the best-case input for median filters. Property 1: heavy right tail. Volume is bounded below by 0 and unbounded above. The distribution is right-skewed with rare extreme outliers (news days, earnings releases, halt events, end-of-quarter rebalancing). Property 2: impulsive structure. The outliers are typically isolated single-bar spikes, not sustained level changes. After a news-day volume spike, volume returns to baseline within 1-3 bars in most cases. Property 3: indicator dependence. Many widely-used indicators (OBV, VWMA, VWAP, A/D line, volume-weighted RSI, money flow index) take volume as a primary input. Any volume contamination propagates into all of these. For SPX daily volume during the 2020-2024 period, the ratio of mean volume to median volume across rolling 20-day windows is consistently above 1.3, with spikes above 2.0 during March 2020 and August 2024\. The mean is biased upward by 30-100% relative to the typical day's volume across most months. Indicators built on rolling mean volume are reading a contaminated estimate of baseline volume by construction. Replacing the mean with the median fixes the contamination at no operational cost. The median(20) of volume is bias-free for typical days and changes only when sustained volume regime shifts occur (which is the signal the indicator is trying to capture in the first place). ## Worked example: SPX August 5, 2024 SPX daily, August 1-30, 2024\. August 5 prints at 4.2x typical recent volume. Compare smoothed volume estimates from four constructions. $$ \\begin{array}{l|c|c|c} \\text{Construction} & \\text{Reading on Aug 5} & \\text{Reading on Aug 19 (10 bars later)} & \\text{Spike contamination} \\\\ \\hline \\text{Mean(20) of volume} & 1.45 \\times \\text{baseline} & 1.21 \\times \\text{baseline} & \\text{High, decays over 20 bars} \\\\ \\text{Median(20) of volume} & 1.00 \\times \\text{baseline} & 1.00 \\times \\text{baseline} & \\text{None} \\\\ \\text{EMA(20) of volume} & 1.21 \\times \\text{baseline} & 1.08 \\times \\text{baseline} & \\text{High at spike, decays} \\\\ \\text{Hampel(20) of volume} & 1.00 \\times \\text{baseline} & 1.00 \\times \\text{baseline} & \\text{None} \\\\ \\end{array} $$ Four constructions. The Mean(20) reports a 45% elevation on the spike day and is still 21% elevated 10 bars later. The EMA(20) is biased by 21% on the spike day and decays to 8% by bar 10\. The Median(20) and Hampel(20) constructions are immune to the spike and report the baseline level on both dates. For the same period, true-range smoothing shows the analogous pattern. SPX true range expands 3.5x on August 5\. Mean-smoothed ATR(20) is biased upward by 15% for the next 20 trading days. Median-smoothed ATR(20) returns to baseline by the second bar after the spike. Strategies that consume smoothed volume or smoothed true range for position sizing carry the contamination into their sizing logic. The August 5 spike causes mean-based sizing to under-allocate (or over-allocate, depending on the sizing rule's direction) for 20 bars after the event. Median-based sizing is unaffected. ## Hybrid constructions Three useful hybrids that combine median and linear filtering. Hybrid 1: median pre-filter then linear smoother. Pass the raw input through a short median filter (3-5 bars) to remove isolated spikes, then through a linear filter (EMA, super-smoother, BPF) for the cycle-isolation work the linear filter is good at. $$ y\_t \\;=\\; \\text{linear filter}\\bigl(\\text{median}\_{N\_1}(x\_{t-i})\\bigr)\_{N\_2} $$ The construction cleans outliers before any frequency-band processing. The linear filter sees clean data and the frequency-response framework applies. Best practice for volume-based indicators: median(5) of volume before any VWMA or volume-weighted construction. Hybrid 2: Hampel filter. A median filter combined with an outlier-detection step. For each bar, compute the median and the median absolute deviation (MAD) over a rolling window. If the current bar deviates from the median by more than k × MAD (typically k = 3), replace it with the median; otherwise, keep the original. Output the cleaned value. $$ y\_t \\;=\\; \\begin{cases} x\_t & \\text{if } |x\_t - \\text{median}\_N(x)| \\le k \\cdot \\text{MAD}\_N(x) \\\\ \\text{median}\_N(x) & \\text{otherwise} \\end{cases} $$ The Hampel filter preserves the input on normal bars and replaces outliers with the median. It is the best general-purpose pre-cleaning filter for trading data because it does not smear normal-day signal while still rejecting spikes. Hybrid 3: trimmed mean. Sort the window, discard the top k% and bottom k%, average the remaining values. $$ y\_t \\;=\\; \\frac{1}{N(1-2\\alpha)} \\sum\_{i = \\lfloor \\alpha N \\rfloor + 1}^{\\lceil (1-\\alpha) N \\rceil} x\_{(i)}, \\qquad \\alpha \\in \[0, 0.5\] $$ For α = 0 the trimmed mean is the regular mean. For α = 0.5 it is the median. In between (typically α = 0.1 or 0.2) the construction interpolates between the two, removing the worst outliers while retaining more information than the strict median. The choice between hybrids depends on the data. Median pre-filter + EMA is the cleanest pipeline for cycle-isolation work on noisy data. Hampel is best for tick-data cleaning where most bars are clean and only occasional bars need replacement. Trimmed mean is best when the data has both outliers and informative variation, and a strict median would discard too much of the variation. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Where median filters fail Three scenarios where the median is the wrong choice. Scenario 1: smooth data with no outliers. On clean data, the median is a less statistically efficient estimator than the mean. For Gaussian data, the median has efficiency π/2 ≈ 1.57 worse variance than the mean. If the input has no outlier behavior, the linear filter wins on smoothness. Scenario 2: cycle isolation. Median filters do not have transfer functions and cannot be designed in frequency space. For isolating a specific cycle band (the BPF use case from ["Band-Pass Filters: The Most Underused Tool in Technical Analysis"](https://aligrithm.com/band-pass-filters-the-most-underused-tool-in-technical-analysis/)), the median filter is structurally the wrong tool. Use the BPF after pre-cleaning with a Hampel or median pre-filter. Scenario 3: phase-aware operations. The Hilbert transform, sine-wave indicator, and any construction that depends on the phase of a smooth oscillation require a smooth input. Median filters produce step-shaped outputs that have ill-defined phase. Use a smooth linear filter for these constructions. The rule: use the median for the cleaning step, use linear filters for the band-isolation step. The two operations live in series, not in competition. ## What this changes in practice Five operational shifts. Default to median for volume, true range, range-based volatility, and tick-data inputs. Any data class with heavy right tails and impulsive outliers should be pre-cleaned with a median or Hampel filter before any linear processing. The cost is one extra pipeline stage; the benefit is contamination-free downstream output. Replace the mean in VWMA, VWAP, OBV, MFI, and similar volume-weighted constructions with a Hampel-cleaned mean. The construction stays compatible with the standard formula but no longer carries spike contamination. For ATR-based position sizing, use median-of-true-range or Hampel-of-true-range as the baseline volatility estimate. The mean-based ATR introduces sizing errors after volatility spikes that take 20 bars to decay. Audit existing strategies for "linear filter on impulsive input" anti-patterns. Any indicator that takes volume, true range, range-based volatility, or unfiltered tick data through a linear smoother is likely contaminated. The fix is the median pre-filter or Hampel cleaning, applied as a one-time pipeline change. Report indicator readings paired with the cleaning method. "VWMA(20) (Hampel-cleaned volume)" is honest. "VWMA(20)" without disclosure of the volume input's cleaning state is ambiguous about whether the reading is contaminated. ## Decision matrix | Data class | Default filter | Reason | | ------------------------------------------------ | --------------------------------- | ------------------------------------------------ | | Volume | Median(20) or Hampel | Heavy right tail, impulsive spikes | | True range | Median(14) or Hampel | Spike-prone on news days, halts | | Range-based volatility (Parkinson, Garman-Klass) | Median or Hampel | Inherits range's spike behavior | | Tick-level price | Hampel(5) | Defends against fat-finger and exchange glitches | | Bar-level close | Linear (EMA, super-smoother) | Typically clean; outliers are rare | | Daily returns | Linear | Outliers exist but are signal, not noise | | Realized volatility | Hampel | Spike-prone, signal mixed with noise | | Funding rate (crypto) | Median | Discrete jumps at funding settlement | | Order flow imbalance | Hampel | Spike-prone at news events | | Cycle isolation | Linear BPF after median pre-clean | Median can't isolate frequencies | | Phase extraction | Linear LPF after median pre-clean | Smooth output required for phase | ## Anti-patterns Five mistakes that show up when median filters are misused or unused. Anti-pattern 1: linear-smoothing volume directly. The mean(N) or EMA(N) of volume is contaminated by every news day for N bars after the event. Replace with median or Hampel. Anti-pattern 2: median-filtering smooth price data unnecessarily. On clean daily SPX close, the median(20) is a less efficient estimator than the EMA(20) and adds no outlier protection because there are no outliers. Reserve the median for inputs with documented outlier behavior. Anti-pattern 3: trying to compute frequency response of a median filter. The operation is nonlinear; H(z) is undefined. Characterize by impulse response, edge response, and effective lag, not by frequency response. Anti-pattern 4: using a very long median window (N > 50) for cleaning. The median's lag grows with N. For outlier rejection, short windows (3-5 bars) catch isolated spikes with minimal lag. Long windows are appropriate only when the outlier structure has multi-bar clusters. Anti-pattern 5: applying a median filter after the linear smoother. The order matters: median pre-filter (cleaning) then linear filter (cycle isolation). The reverse order lets the linear filter integrate the outliers first, after which the median cannot recover the clean baseline. ## Visualizing the filter difference ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/05/article_50-visual-0.png) ## KEY POINTS - Linear filters integrate every input bar including outliers. A single spike contaminates the output for N bars with magnitude proportional to spike/N. - Median filters select the typical value rather than integrating. An isolated outlier in a window of N=20 changes the output by zero until enough outliers join it. - Breakdown point: 0% for mean (one outlier can corrupt arbitrarily), 50% for median (half the window must be contaminated). The robust-statistics property applies to filters bar-by-bar. - Median filters are nonlinear. They have no transfer function and no frequency response. Characterize by impulse response (zero for isolated spikes), edge response (sharp transitions preserved), and effective lag (similar to linear filter of same window). - Volume data has heavy right tail with impulsive spikes (news days, halts, end-of-quarter). Mean-smoothed volume is biased 30-100% relative to typical-day volume across most months. Median-smoothed volume is bias-free. - For SPX August 5, 2024 (volume spike day): Mean(20) of volume shows 45% bias on day 1, 21% bias on day 10\. Median(20) and Hampel(20) show 0% bias throughout. - The hybrid pipeline: median or Hampel pre-filter (cleaning) followed by linear filter (cycle isolation). The two operations live in series, not in competition. - Hampel filter: replace outliers (|deviation| > k × MAD) with the median, keep normal bars unchanged. Best general-purpose pre-cleaner for trading data. - Trimmed mean: discard the top and bottom k% of the sorted window, average the rest. Interpolates between mean (k=0) and median (k=50%). - Median filters are the wrong tool for cycle isolation, phase extraction, or any frequency-band-design problem. Use them as the cleaning stage before the linear filter, not as a replacement for the linear filter. - Replace the mean in VWMA, VWAP, OBV, MFI, and similar constructions with Hampel-cleaned mean. ATR-based position sizing should use median-of-TR or Hampel-of-TR baseline. - Default to median for volume, true range, range-based volatility, tick price, realized volatility, funding rates, and order flow imbalance. Use linear filters for daily close, daily returns, and clean cycle-isolation work. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Statistically Sound Indicators for Financial Market Prediction - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Statistically+Sound+Indicators+for+Financial+Market+Prediction+Timothy+Masters&ref=aligrithm.com) - [Cycle Analytics for Traders - John Ehlers (Amazon)](https://www.amazon.com/s?k=Cycle+Analytics+for+Traders+John+Ehlers&ref=aligrithm.com) ### 2.27 How to Think About Indicator Lag Before Backtesting URL: https://aligrithm.com/how-to-think-about-indicator-lag-before-backtesting/ Last updated: 2026-07-27T13:23:50.000Z A trader designs a mean-reversion strategy on SPX daily. The construction: smooth the close with EMA(20), compute close-minus-EMA, feed into RSI(14), trigger long entries when RSI dips below 30\. The trader runs the backtest, sees Sharpe 1.8 over 15 years, and ships to paper trading. Two months later the live results show Sharpe 0.4 with negative skew. The trader blames "regime change" and shelves the strategy. The strategy did not fail because the regime changed. It failed because the backtest ran without a lag audit. The EMA(20) introduces 9.5 bars of group delay. The RSI(14)'s internal EMA introduces another 6.5 bars of cascade lag at the cycle frequencies the strategy targets. The combined critical-path lag is approximately 16 bars. The strategy's target holding period is 5 bars (mean-reversion to the EMA, typically completing in less than a week). The lag is more than three times the holding period. Every signal arrives so late that the mean reversion has already happened by the time the entry triggers. The backtest showed Sharpe 1.8 because the backtest engine, like every backtest engine, runs the signal computation on time-indexed data without distinguishing between "what the trader sees now" and "what the data eventually showed at this timestamp." The 16-bar lag was absorbed into the backtest as look-ahead masquerading as smoothing. Live trading exposes the lag because the trader cannot wait for the smoothed value to stabilize before placing the trade. The article ["The Hidden Cost of Every Moving Average: Lag"](https://aligrithm.com/the-hidden-cost-of-every-moving-average-lag/) established the structural lag of common smoothers. The article ["Why Moving Averages Can Lie at Turning Points"](https://aligrithm.com/why-moving-averages-can-lie-at-turning-points/) showed the lag's operational cost at structural inflection points. The article ["The Frequency Response of Trading Indicators"](https://aligrithm.com/the-frequency-response-of-trading-indicators/) gave the universal framework for reading any linear indicator's response. This article gives the pre-backtest planning discipline: how to compute the lag budget of a proposed strategy, compare it to the holding period and the expected move time, and make the go-or-no-go decision before the first backtest run. The next article in this series (["Why Median Filters Are Useful for Volume and Outliers"](https://aligrithm.com/why-median-filters-are-useful-for-volume-and-outliers/)) covers a class of nonlinear filters whose lag properties differ from linear filters in operationally useful ways. ## What lag actually means Two formal definitions of lag, both derived from the filter's transfer function H(z). Either can be the right answer depending on what part of the signal the strategy uses. Group delay: the delay applied to the envelope of a wavetrain at frequency ω. Derived from the negative derivative of the phase response: $$ \\tau\_{\\text{group}}(\\omega) \\;=\\; -\\frac{d \\arg(H(e^{j\\omega}))}{d \\omega} $$ Group delay matters when the strategy reads the indicator's level or trend (slope). The level at time t reflects the data from t minus τ\_group bars ago. Phase delay: the delay applied to a single sinusoid at frequency ω. The ratio of the phase to the frequency: $$ \\tau\_{\\text{phase}}(\\omega) \\;=\\; -\\frac{\\arg(H(e^{j\\omega}))}{\\omega} $$ Phase delay matters when the strategy reads the indicator's cycle position (zero crossing, peak, trough). The cycle at time t crosses zero τ\_phase bars later than the underlying data does. For a symmetric FIR filter (SMA, Hann window, Blackman window) the two delays are equal and constant across all frequencies. For an IIR filter (EMA, recursive HPF/LPF, BPF) the two delays are frequency-dependent and not equal to each other. The operational shortcut: for any linear indicator, compute the lag at the dominant frequency of the cycle being read. That is the lag the strategy pays. ## Lag formulas for the standard toolkit Lag in bars at the cycle frequency for which each filter is designed. $$ \\begin{array}{l|c|c} \\text{Filter} & \\text{Lag formula} & \\text{Lag (bars) at default lookback} \\\\ \\hline \\text{SMA}(N) & (N-1)/2 & 9.5 \\text{ for } N{=}20, \\; 24.5 \\text{ for } N{=}50 \\\\ \\text{EMA}(\\alpha) & (1-\\alpha)/(2\\alpha) \\approx (N-1)/2 & 9.5 \\text{ for } N{=}20, \\; 24.5 \\text{ for } N{=}50 \\\\ \\text{WMA}(N) & (N-1)/3 & 6.3 \\text{ for } N{=}20 \\\\ \\text{1-pole LPF (cutoff T)} & \\approx T/(2\\pi) & 3.2 \\text{ for } T{=}20 \\\\ \\text{2-pole LPF (cutoff T)} & \\approx T/\\pi & 6.4 \\text{ for } T{=}20 \\\\ \\text{Super-smoother (T)} & \\approx T/2 & 10 \\text{ for } T{=}20 \\\\ \\text{1-pole HPF (cutoff T)} & \\approx T/(2\\pi) & 3.2 \\text{ for } T{=}20 \\\\ \\text{2-pole HPF (cutoff T)} & \\approx T/\\pi & 6.4 \\text{ for } T{=}20 \\\\ \\text{BPF (center T, } \\delta{=}0.3) & \\approx T/3 \\text{ to } T/4 & 5.0 \\text{ to } 6.7 \\text{ for } T{=}20 \\\\ \\text{Decycler (HPF cutoff T)} & \\approx T/(2\\pi) & 14.5 \\text{ for } T{=}30 \\\\ \\text{Differencer } x\_t - x\_{t-1} & 0.5 & 0.5 \\\\ \\text{N-bar momentum } x\_t - x\_{t-N} & N/2 & 10 \\text{ for } N{=}20 \\\\ \\end{array} $$ Twelve formulas. The lag of every linear operation in the strategy is computable from this table in seconds. The numbers reveal the structural tradeoffs covered in prior articles. Same-cutoff comparison: a 1-pole LPF has roughly half the lag of a super-smoother but lets through about ten times more cycle leakage. The BPF has comparable lag to a 2-pole LPF but operates on a much narrower frequency band. ## The lag budget The lag budget of a strategy is the sum of the lags of the linear operations along the critical signal path, plus any sampling and execution delays. Critical signal path: the longest sequence of linear operations from input data to trade decision. Branching computations (e.g., two indicators evaluated in parallel and combined) take the maximum of the two branch lags, not the sum. Cascade rule: when one filter's output feeds another's input (cascade), the lags add. Two EMA(20)s in series produce a 2-pole-like response with combined lag 9.5 + 9.5 = 19 bars at the dominant frequency. Parallel rule: when two filters operate on the same input and their outputs are combined linearly (sum, difference), the resulting filter's lag is the lag of the slower branch in the band where it dominates. For MACD = EMA(12) − EMA(26), the lag is approximately the EMA(26) lag (12.5 bars) at the cycle band where MACD peaks. Nonlinearity rule: nonlinear operations (gain/loss separation in RSI, range normalization in Stochastic, max/min over rolling window) do not change the underlying linear filter's lag. They shape the output distribution but the cycle-band reading still happens at the linear filter's lag. Sampling delay: a strategy that decides on the close of bar t and executes on the open of bar t+1 carries a 0.5 to 1 bar sampling delay. This is structural for any close-of-bar strategy. Filter warmup: the first N bars of any filter's output are not at the filter's steady-state response. The warmup adds lag during the first N bars of operation. After warmup, the lag is at the table values. The full budget: $$ \\tau\_{\\text{budget}} \\;=\\; \\sum\_{\\text{operations on critical path}} \\tau\_{\\text{filter}} \\;+\\; \\tau\_{\\text{sampling}} \\;+\\; \\tau\_{\\text{warmup correction (if applicable)}} $$ For the opening example (EMA(20) → close-minus-EMA → RSI(14) → trigger): - EMA(20): 9.5 bars - Close minus EMA: 0 bars (subtraction has no lag) - RSI(14)'s internal EMA on gains and losses: 6.5 bars - Sampling delay: 1 bar - Total: 17 bars Critical-path lag for the strategy: 17 bars. Holding period for the strategy: 5 bars. The lag is 3.4 times the holding period. The strategy has no operational room for its own decision latency. ## The four costs of lag Lag is not a single number on a P&L line. It produces four distinct costs that each show up at different stages of the strategy lifecycle. Cost 1: missed move. By the time the signal fires, part of the move has already happened. For a 5% move that takes 10 bars to develop, a 5-bar lag captures about 50% of the move. A 15-bar lag captures roughly zero (the move is over). Cost 2: stale state. The indicator level read at time t reflects the data from t minus τ bars ago. Strategies that condition on level (close above EMA, RSI below 30) are conditioning on a state that no longer exists. The decision is made against past truth. Cost 3: false signals. Lag makes filters slow to respond to true regime changes but fast to respond to phantom regime changes (transient noise that the slow filter integrates as a real shift). The article ["Why the SMA Is Often a Terrible Smoother"](https://aligrithm.com/why-the-sma-is-often-a-terrible-smoother/) detailed the sidelobe-induced false signal problem. Cost 4: P&L attribution corruption. The backtest's reported edge mixes (a) true forecasting alpha and (b) lag-absorbed look-ahead. The two cannot be separated without explicit lag accounting. Strategies that look strong in backtest and fail in live are typically those where the lag-absorbed component was larger than the true alpha. ## Pre-backtest checklist Four questions, answered in order, before any backtest runs. Q1: What is the lag budget? Sum the lags of every linear operation on the critical path. Add sampling delay and any warmup correction. Express in bars. Q2: What is the holding period? For a strategy with explicit holding period (exit after N bars), this is N. For a strategy with dynamic exit (target or stop), use the median historical hold across similar strategies as the estimate. If no estimate exists, the strategy specification is incomplete. Q3: What is the time-to-half-move? For the typical winning trade, how many bars does it take for half the move to develop? This is the bar count after which a late signal stops being useful. For mean-reversion to an MA, time-to-half-move is roughly the band's typical reversion time, often 2-5 bars. For trend-following moves, time-to-half-move is roughly half the trend's typical duration, often 20-50 bars. Q4: Is the lag budget less than half the time-to-half-move? If yes, the strategy has room. The lag is small enough that the signal arrives while the move is still unfolding. If no, the strategy is structurally late and will lose the bulk of every move to the lag. A failed Q4 means the strategy needs redesign before backtesting. Either reduce the lag (replace cascaded filters with single-stage filters, switch from MA-based detection to cycle-mode detection) or accept a longer holding period that matches the lag budget. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Worked example: mean-reversion strategy redesign The opening example, audited and redesigned. Original construction: $$ \\begin{array}{l|c} \\text{Operation} & \\text{Lag (bars)} \\\\ \\hline \\text{EMA(20) of close} & 9.5 \\\\ \\text{Close minus EMA} & 0 \\\\ \\text{RSI(14) on the deviation} & 6.5 \\\\ \\text{Sampling delay (close-to-next-open)} & 1.0 \\\\ \\hline \\text{Critical-path lag} & 17.0 \\\\ \\end{array} $$ Holding period: 5 bars. Time-to-half-move (for mean reversion to the EMA): 2.5 bars. Lag budget: 17 bars. The lag is 6.8 times the time-to-half-move. Q4 fails catastrophically. The strategy is structurally broken. Redesigned construction using a single BPF for the same cycle band: $$ \\begin{array}{l|c} \\text{Operation} & \\text{Lag (bars)} \\\\ \\hline \\text{BPF(T=20, } \\delta{=}0.3\\text{) of close} & 5.0 \\\\ \\text{Threshold trigger on BPF output} & 0 \\\\ \\text{Sampling delay (close-to-next-open)} & 1.0 \\\\ \\hline \\text{Critical-path lag} & 6.0 \\\\ \\end{array} $$ Critical-path lag: 6 bars. Holding period: 5 bars. Time-to-half-move: 2.5 bars. The lag is 2.4 times the time-to-half-move. Q4 still fails, but less severely. Final redesign: extend the holding period to match the lag, or use a faster cycle-mode detector. Option A: extend the holding period to 10 bars (target reversion to 20-bar BPF's full half-cycle). Time-to-half-move becomes 5 bars. Lag budget (6) is now 1.2 times the time-to-half-move. Q4 still marginal. Option B: switch to a faster cycle detector, the autocorr reversal indicator covered in the article "Why Markets Are Not Random: The Autocorrelation Story", with lag of 2-4 bars. Lag budget drops to 3-5 bars. Q4 passes for the original 5-bar holding period. The pre-backtest audit takes 5 minutes and saves weeks of backtesting work on a strategy that was structurally broken before the first run. ## Hidden lag sources Three sources of lag that do not appear in the transfer function but contribute to the operational lag budget. Hidden source 1: lookback bias. Many backtest engines compute the signal at bar t using close\[t\], then assume the trade fills at close\[t\] or open\[t+1\]. If the indicator computation requires close\[t\] to produce the bar-t signal, the trade cannot fill until open\[t+1\] at earliest. The strategy specification must distinguish between "signal computed at close of t" and "trade filled at open of t+1." The 1-bar sampling delay is the minimum; any close-of-bar strategy has at least this lag. Hidden source 2: filter initialization. The first N bars of any IIR filter's output are not at the steady-state response. For an EMA(50), the first 50-100 bars are warming up. For a BPF, the first 4T bars are warming up. Backtests that start at bar 1 carry warmup artifacts that look like signal. The fix: discard the warmup bars from the backtest results and start measurement only after the filter has stabilized. Hidden source 3: data revision. For intraday data, the close of the bar is sometimes revised after the bar closes (exchange corrections, late prints). Strategies that depend on the close of the most recent bar for the current bar's signal cannot trade until after the revision window closes. This adds operational lag that is invisible in backtest data because the backtest sees only the final values. Each hidden source adds 1 to 5 bars of operational lag that the transfer-function calculation misses. Pre-backtest budgeting should include explicit estimates of each. ## What this changes in practice Five operational shifts. Compute the lag budget before any backtest. The budget is a 5-minute calculation. It catches structurally broken strategies before any computation time is spent. Report strategy performance with the lag explicit. The line "Sharpe 1.8 (16-bar lag budget, 5-bar holding period)" is honest. The line "Sharpe 1.8" is misleading because the reader cannot tell whether the edge survives the lag. Choose holding period and lag budget together. A 5-bar holding period mandates a sub-2-bar lag budget. A 50-bar holding period tolerates 10-15 bars of lag. The two parameters are jointly chosen, not separately optimized. Cap indicator cascades. If the lag budget is 5 bars, the strategy cannot use any cascade longer than 5 bars / typical\_filter\_lag operations. Cascading three EMAs (28-bar combined lag) on a 5-bar holding period is mechanically broken. Reject strategies that fail Q4 at design time. A strategy with 17-bar lag on a 5-bar holding period is not "interesting to backtest." It is structurally broken. Spend the backtest time on strategies that pass the lag audit. ## Decision matrix | Holding period | Max lag budget | Allowed constructions | | --------------------------- | -------------- | ----------------------------------------- | | 1-3 bars (intraday, scalp) | 1-2 bars | Differencer, 1-bar momentum, no smoothing | | 5-10 bars (short swing) | 3-5 bars | BPF at cycle, fast EMA, decycler | | 20-50 bars (medium swing) | 10-15 bars | EMA(20-30), super-smoother, decycler | | 50-200 bars (position) | 25-50 bars | EMA(50-100), slow super-smoother | | 200+ bars (regime) | 50-100 bars | Long-term decycler, multi-pole LPF | | Cycle-mode (BPF zero cross) | T/4 to T/3 | BPF at cycle period | | Turning point (cycle-mode) | 2-5 bars | Autocorr reversal, Hilbert phase | | Mean reversion (1-3 bar) | 1-2 bars | Differencer-based trigger | ## Anti-patterns Five mistakes that show up in strategy backtests. Anti-pattern 1: optimizing lookbacks against backtest P&L without lag accounting. The optimization will find lookbacks where the lag-absorbed look-ahead is largest. The resulting strategy fails in live. Anti-pattern 2: cascading three or four indicators without summing the lags. A common construction: "smooth with EMA, normalize with rolling std, smooth again, threshold." Each step adds 5-10 bars. The composite lag is 20-30 bars and the strategy targets 5-bar moves. Mechanically impossible. Anti-pattern 3: treating "EMA(50) is still up" as a present-tense reading. The EMA's value at time t reflects the trend 25 bars ago. A strategy that conditions on this for a 5-bar trade is using stale state. Anti-pattern 4: assuming "we'll fix it in optimization" works for lag. Lag is structural, not a hyperparameter. Optimization can only move along the lag-vs-noise Pareto frontier; it cannot escape the frontier. Anti-pattern 5: backtesting filter-based strategies with no warmup discard. The first N bars of any IIR filter are not at steady state. Including those bars in the backtest's Sharpe calculation inflates the result by the warmup artifact. ## Visualizing the lag budget ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/05/article_49-visual-0.png) ## KEY POINTS - Every linear indicator has a known lag derived from its transfer function. The lag is computable from a small table for every filter in the standard toolkit. - Group delay is the lag of the indicator's level and slope. Phase delay is the lag of the indicator's cycle position (zero crossings). Use whichever matches the strategy's signal source. - The lag budget is the sum of the lags of every linear operation on the critical signal path, plus sampling delay and any warmup correction. - Cascade rule: lags add. Two EMA(20)s in series carry 19 bars of lag. Parallel rule: lags take the maximum of the dominant branch. Nonlinearity rule: nonlinear shaping does not change the underlying linear filter's lag. - Pre-backtest checklist: lag budget, holding period, time-to-half-move, Q4 (lag less than half the time-to-half-move?). A failed Q4 means the strategy is structurally broken and must be redesigned before any backtest runs. - Common construction failure: EMA + RSI cascade for mean-reversion. Critical-path lag 17 bars, holding period 5 bars, lag-to-holding ratio 3.4\. The lag absorbs the move before the signal fires. - Redesign with a single BPF at the same cycle band cuts lag from 17 to 6 bars. Further redesign with autocorr reversal or Hilbert phase brings it under 4 bars and passes Q4. - Hidden lag sources: lookback bias (close-of-bar strategies have at least 1-bar sampling delay), filter initialization (IIR warmup), data revision (intraday close revisions). - Backtests that report Sharpe without disclosing lag are misleading. Sharpe and lag are jointly meaningful. Report both. - Holding period and lag budget are jointly chosen. A 5-bar holding period mandates a sub-2-bar lag budget. A 50-bar holding period tolerates 10-15 bars. The two parameters are not separately optimizable. - Lag is not a hyperparameter. Optimization moves along the lag-vs-noise Pareto frontier but cannot escape it. The escape comes from changing construction class, not from tuning. - Anti-pattern: "we'll fix it in backtest optimization." Lag failures show up in backtest as look-ahead-absorbed Sharpe that disappears in live. The fix is structural redesign, not tuning. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Statistically Sound Indicators for Financial Market Prediction - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Statistically+Sound+Indicators+for+Financial+Market+Prediction+Timothy+Masters&ref=aligrithm.com) - [Cycle Analytics for Traders - John Ehlers (Amazon)](https://www.amazon.com/s?k=Cycle+Analytics+for+Traders+John+Ehlers&ref=aligrithm.com) ### 2.26 The Frequency Response of Trading Indicators URL: https://aligrithm.com/the-frequency-response-of-trading-indicators/ Last updated: 2026-07-27T13:23:49.000Z A trader stacks RSI(14), MACD(12, 26, 9), and Bollinger Bands(20, 2) on the same chart. The three indicators "look different," so the trader assumes they carry independent information and uses all three for confluence signals. The system trips because the indicators agree most of the time, disagree at the noisy edges, and the disagreement is treated as a high-conviction signal that turns out to be uncorrelated noise. The structural problem: nobody computed what frequency content each indicator passes. RSI(14) and Stochastic(14) are nearly identical band-pass-ish operations on the same data. MACD(12, 26) and close-minus-EMA(20) target almost the same cycle band. Three "different" indicators are reading the same cycle component three different ways and calling the redundancy "confirmation." Frequency response is the ground-truth description of every linear indicator. The magnitude of the transfer function evaluated on the unit circle (substitute z⁻¹ = exp(−2πjf) and take the modulus) gives a curve from frequency zero to the Nyquist frequency that tells you exactly which cycles the indicator passes, attenuates, or destroys. Two indicators with overlapping response curves carry duplicate information. Two indicators with disjoint curves carry independent information. The visual appearance on a chart is irrelevant to this question; the response curve is. This article is the framework article for the filter-theory block. The prior articles in the series (["No Filter Is Predictive: What Traders Misunderstand About Smoothing"](https://aligrithm.com/no-filter-is-predictive-what-traders-misunderstand-about-smoothing/), ["Why the SMA Is Often a Terrible Smoother"](https://aligrithm.com/why-the-sma-is-often-a-terrible-smoother/), ["EMA vs SMA: Why Simplicity Still Matters"](https://aligrithm.com/ema-vs-sma-why-simplicity-still-matters/), ["The Trader's Guide to Low-Pass Filters"](https://aligrithm.com/the-traders-guide-to-low-pass-filters/), ["High-Pass Filters for Traders"](https://aligrithm.com/high-pass-filters-for-traders/), ["Band-Pass Filters: The Most Underused Tool in Technical Analysis"](https://aligrithm.com/band-pass-filters-the-most-underused-tool-in-technical-analysis/), ["Decyclers: Extracting Trend by Removing Cycle Energy"](https://aligrithm.com/decyclers-extracting-trend-by-removing-cycle-energy/)) each gave the response curve for a specific filter family. This article gives the universal recipe: how to compute the response of any linear indicator, what each region of the curve means, and how to use the framework to diagnose the standard TA toolkit. The next article (["How to Think About Indicator Lag Before Backtesting"](https://aligrithm.com/how-to-think-about-indicator-lag-before-backtesting/)) turns the framework into a backtesting-cost discipline. ## The recipe Every linear indicator is a filter. Every filter has a transfer function expressed as a ratio of polynomials in z⁻¹: $$ H(z) \\;=\\; \\frac{b\_0 + b\_1 z^{-1} + b\_2 z^{-2} + \\dots + b\_M z^{-M}}{a\_0 + a\_1 z^{-1} + a\_2 z^{-2} + \\dots + a\_N z^{-N}} $$ The numerator coefficients (b\_i) describe how the input contributes to the output. The denominator coefficients (a\_j) describe how past output values contribute (the recursive part). For an FIR filter (no recursion), the denominator is just a\_0\. For an IIR filter (EMA, recursive HPF/LPF), both numerator and denominator are non-trivial. To get the frequency response, substitute z⁻¹ = exp(−2πjf) where f is normalized frequency (cycles per bar, ranging from 0 at DC to 0.5 at Nyquist): $$ H(f) \\;=\\; H(z)\\,\\Big|\_{z^{-1} = e^{-2\\pi j f}}, \\qquad |H(f)| \\;=\\; \\text{magnitude response} $$ The magnitude |H(f)| is the gain at each frequency. A magnitude of 1 means the cycle at frequency f passes through unchanged. A magnitude of 0 means the cycle is completely destroyed (a "zero" of the filter). A magnitude greater than 1 means the cycle is amplified. The full curve from f = 0 to f = 0.5 is the indicator's frequency response. The reciprocal relationship between frequency and period gives the practical reading: $$ T \\;=\\; \\frac{1}{f}, \\qquad f \\;=\\; \\frac{1}{T} $$ A frequency of 0.05 corresponds to a 20-bar cycle. A frequency of 0.1 corresponds to a 10-bar cycle. A frequency of 0.5 corresponds to a 2-bar cycle (the fastest possible oscillation in discrete-time data, the Nyquist limit). ## What each region of the curve means Five structural readings. DC gain (f = 0): the response to a constant (no-cycle) input. For a low-pass filter, DC gain is 1 (trend passes through). For a high-pass filter, DC gain is 0 (trend is cancelled). For a properly normalized band-pass filter, DC gain is 0 (trend cancelled, only the band passes). Rolloff slope: how aggressively the filter attenuates frequencies outside the desired band. For a 1-pole LPF, the rolloff is 20 dB/decade above the cutoff. For a 2-pole LPF, it is 40 dB/decade. Steeper rolloff means better cycle isolation but more lag. Zeros: frequencies where the response is exactly 0\. The cycle at that frequency is completely destroyed. The SMA(N) has zeros at every multiple of 1/N (covered in the article ["Why the SMA Is Often a Terrible Smoother"](https://aligrithm.com/why-the-sma-is-often-a-terrible-smoother/)). The differencer (HPF with N=1) has a zero at DC. Zeros are structural and unavoidable for the filter that produces them. Peaks: frequencies where the response is maximum. For an LPF, the peak is at DC. For an HPF, the peak is at Nyquist. For a BPF, the peak is at the center frequency. A peak above 1 means the filter amplifies the corresponding cycle, which is a design choice (resonant filters) or an artifact (sidelobes). Sidelobes: secondary peaks outside the main passband. The SMA's sinc function has sidelobes that leak about 22% of nearby frequencies into the filter output. Hann and Blackman windows have much smaller sidelobes. The article ["Why the SMA Is Often a Terrible Smoother"](https://aligrithm.com/why-the-sma-is-often-a-terrible-smoother/) detailed the sidelobe critique. ## Catalog of canonical responses The filter-theory block of this pillar produced a small set of distinct response shapes. Most of the TA indicator catalog falls into one of these shapes or a combination. $$ \\begin{array}{l|c|c|c} \\text{Filter} & \\text{DC gain} & \\text{Nyquist gain} & \\text{Shape} \\\\ \\hline \\text{SMA(N)} & 1 & 0 \\text{ (if N even)} & \\text{sinc with sidelobes, zeros at } 1/N, 2/N, \\dots \\\\ \\text{EMA(N)} & 1 & \\approx \\alpha/(2-\\alpha) & \\text{monotonic decay, no zeros, no sidelobes} \\\\ \\text{Hann LPF(N)} & 1 & 0 & \\text{tapered, sidelobes 32 dB down} \\\\ \\text{Super-smoother(T)} & 1 & 0 & \\text{2-pole LPF, critically damped, 40 dB/decade} \\\\ \\text{1-pole HPF(T)} & 0 & 1 & \\text{monotonic rise, complement of EMA} \\\\ \\text{2-pole HPF(T)} & 0 & 1 & \\text{steeper rise, 40 dB/decade} \\\\ \\text{BPF(T, } \\delta\\text{)} & 0 & 0 & \\text{peaked at frequency } 1/T \\\\ \\text{Decycler(T)} & 1 & 0 & \\text{equivalent to 1-pole LPF with HPF parameterization} \\\\ \\text{Differencer (} x\_t - x\_{t-1}\\text{)} & 0 & 2 & \\text{HPF-like, amplifies highest frequencies} \\\\ \\text{N-bar momentum (} x\_t - x\_{t-N}\\text{)} & 0 & \\le 2 & \\text{HPF with notches at } k/N \\\\ \\end{array} $$ Ten shapes. Every indicator in the standard TA literature reduces to one of these or to a combination produced by cascading (multiplying transfer functions) and parallel composition (adding transfer functions). ## Diagnosing standard TA indicators Apply the framework to the standard catalog. Each indicator's structural identity becomes clear once the underlying frequency response is computed. ### RSI The standard 14-period RSI is built from two operations: compute the bar-to-bar change (a 1-bar differencer, which is an HPF), separate gains and losses, then smooth each with an exponential average using α = 1/14\. The final ratio is a non-linear transform of the smoothed gain and loss series, but the underlying linear structure is HPF cascaded with LPF. The effective frequency response (treating the local linearization around moderate values) is a band-pass-like shape with peak around the 28-bar cycle and rolloff in both directions. The 1-bar differencer cancels DC; the EMA(14) smooths the very fast noise. The 14 in "RSI(14)" sets the EMA smoothing, not the differencer length. The implication: RSI is approximately a band-pass filter on the cycle band centered around 28 bars. Two traders running "RSI(14)" and "BPF(T=28)" are reading the same cycle band through two different lenses. ### MACD MACD is the difference of two EMAs (EMA(12) minus EMA(26)). The transfer function is the algebraic difference: $$ H\_{\\text{MACD}}(z) \\;=\\; H\_{\\text{EMA}(12)}(z) - H\_{\\text{EMA}(26)}(z) $$ At DC both EMAs have gain 1, so the MACD has DC gain 0 (trend is cancelled). At Nyquist both EMAs have gain ≈ α/(2−α), so the MACD has small Nyquist gain. The peak of the MACD response sits in the cycle band where the two EMAs disagree most, roughly at the 18-to-30-bar period. The MACD is a band-pass filter built by EMA difference. The direct BPF construction does the same job with a single recursive section, lower lag, and explicit parameter control. The article ["Band-Pass Filters: The Most Underused Tool in Technical Analysis"](https://aligrithm.com/band-pass-filters-the-most-underused-tool-in-technical-analysis/) detailed the comparison. ### Stochastic oscillator Stochastic(N) computes (close − min\_N) / (max\_N − min\_N) where min\_N and max\_N are the rolling minimum and maximum over N bars. The construction is nonlinear (the min and max are not linear operations), but the local linearization treats min\_N + max\_N as approximately a trend baseline. The Stochastic is then approximately close minus baseline, divided by range, which is a normalized HPF. The Stochastic's effective frequency response is roughly an HPF with cutoff at frequency 1/N. The %K and %D smoothing (typically 3-period SMA on top) adds a mild LPF that turns the composite into a wide band-pass. The implication: Stochastic(14, 3, 3) is approximately a band-pass on cycles around the 14-bar period, normalized to the \[0, 1\] range. RSI(14) and Stochastic(14) read nearly identical cycle bands and produce highly correlated outputs. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ### Bollinger Bands The Bollinger Band center line is an SMA(20). The frequency response is the sinc function with zeros at every multiple of 1/20 = 0.05 (cycles of period 20, 10, 6.67, 5, ...). The 20-bar cycle is annihilated. The bands themselves are SMA ± k × rolling\_std, which is nonlinear (rolling std is a quadratic operation). The implication: Bollinger Bands inherit the SMA's sinc-and-sidelobe response in the center line. Any cycle near a 20-bar period or its harmonics is destroyed by the center line. The bands' width is a separate signal driven by the rolling std, which is not a linear filter response. ### ADX ADX is built on the True Range and Directional Movement components. True Range is a 1-bar nonlinear function of OHLC. Directional Movement is a 1-bar difference of high and low values. Both are then smoothed by an EMA with α = 1/14 for standard ADX. The underlying linear structure is a 1-bar differencer cascaded with an EMA(14). The frequency response is similar to RSI: band-pass around the 28-bar period. ADX adds further nonlinear processing (max, abs, division) that shapes the output into a \[0, 100\] range but does not move the underlying cycle band. The implication: ADX and RSI read approximately the same cycle band through different nonlinear lenses. The "directional" framing of ADX vs the "overbought-oversold" framing of RSI is interpretive, not structural; both indicators sit on the same cycle band. ## Worked example: response curves overlaid SPX daily. Plot the magnitude response of five indicators on a single chart, frequency axis from 0 to 0.25 (cycles per bar, equivalent to periods from infinity down to 4 bars). $$ \\begin{array}{l|c|c|c|c} \\text{Indicator} & \\text{Peak frequency} & \\text{Peak period} & \\text{DC gain} & \\text{Band coverage} \\\\ \\hline \\text{RSI(14)} & 0.036 & 28\\text{ bars} & 0 & \\text{15-50 bars} \\\\ \\text{MACD (12, 26)} & 0.042 & 24\\text{ bars} & 0 & \\text{16-40 bars} \\\\ \\text{Stochastic(14, 3, 3)} & 0.05 & 20\\text{ bars} & 0 & \\text{12-30 bars} \\\\ \\text{EMA(20) (as trend)} & 0 & \\infty & 1 & \\text{30+ bars} \\\\ \\text{Super-smoother T=20 (as trend)} & 0 & \\infty & 1 & \\text{30+ bars} \\\\ \\text{BPF(T=20, } \\delta\\text{=0.3)} & 0.05 & 20\\text{ bars} & 0 & \\text{17-23 bars} \\\\ \\text{Close minus EMA(20)} & 0.05 & 20\\text{ bars} & 0 & \\text{8-30 bars} \\\\ \\text{1-day momentum} & 0.5 & 2\\text{ bars} & 0 & \\text{2-6 bars} \\\\ \\end{array} $$ Eight indicators. Five of them (RSI, MACD, Stochastic, BPF, close-minus-EMA) target overlapping cycle bands centered around 20-30 bar periods. Two (EMA, super-smoother) target the trend (DC and very low frequencies). One (1-day momentum) targets the fastest noise band. The redundancy is now visible. RSI(14), MACD(12,26), and Stochastic(14) all sit in the 15-to-40-bar cycle band. A trader using all three is reading the same band through three nonlinear lenses. The "confluence" signal where all three agree is the signal that the 20-30-bar cycle is in a coherent phase; the disagreement signal is the noise from each indicator's nonlinear shaping. The BPF(T=20, δ=0.3) is the narrowest and cleanest of the band-pass family. The close-minus-EMA(20) is the widest, with the most leakage. If the goal is to isolate the 20-bar cycle, the BPF is the structurally best choice. The RSI/MACD/Stochastic combo achieves no better band isolation than the single BPF and adds nonlinearity that obscures the underlying signal. ## What this changes in practice Five operational shifts. Choose indicators by their frequency response, not by their lookback or their popular name. The frequency-response curve specifies what cycle bands the indicator reads. Two indicators with overlapping responses are redundant; two with disjoint responses are complementary. Plot the response curve for every indicator entering the strategy. The plot is a 30-second computation (sample H(z) at 256 points along the unit circle, take the modulus). The plot reveals overlap, gaps, and miscalibration that no amount of chart inspection can show. Compose indicators by composing transfer functions. Cascading two filters (output of one feeds the input of the next) multiplies their transfer functions. Parallel composition (sum or difference of two filter outputs) adds or subtracts their transfer functions. The compositions are designable in frequency space, not by trial and error. Replace nonlinear-but-structurally-redundant indicators with their underlying linear equivalents. RSI(14), MACD(12,26), Stochastic(14) are all approximately the same band-pass operation. Pick one if cycle-band reading is the goal; pick the direct BPF if cleaner isolation matters. Detect band gaps. If the indicator suite covers the 15-to-40-bar band heavily but ignores the 5-to-15-bar band entirely, the strategy is blind to faster cycles. The response-curve overlay reveals coverage gaps that ad-hoc indicator stacking cannot expose. ## Decision matrix | Goal | Recommended response shape | Recommended construction | | ------------------------------------------ | --------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Smooth trend, minimal lag | LPF with steep rolloff, DC gain 1 | Super-smoother or Hann window | | Detrend, keep deviations | HPF, DC gain 0 | 1-pole HPF or 2-pole HPF | | Isolate specific cycle band | BPF, peak at target frequency | Direct BPF | | Cancel specific cycle | LPF with zero at target frequency | Decycler at cycle period | | Trend-cycle decomposition | LPF + HPF pair, sum to 1 | Decycler + matched HPF | | Multi-band cycle decomposition | Bank of BPFs at multiple T | Bank of direct BPFs | | Predictor: not possible with linear filter | N/A | See ["No Filter Is Predictive: What Traders Misunderstand About Smoothing"](https://aligrithm.com/no-filter-is-predictive-what-traders-misunderstand-about-smoothing/) | ## Anti-patterns Five mistakes when working without the frequency-response framework. Anti-pattern 1: stacking indicators with overlapping response bands and calling the agreement "confirmation." The agreement is structural redundancy. Reduce to one band-pass per cycle band of interest. Anti-pattern 2: picking lookbacks by trial and error on backtest performance. The lookback maps to a cutoff frequency that should be chosen by what cycle band the strategy targets, not by what produces the highest historical Sharpe. The article ["How to Think About Indicator Lag Before Backtesting"](https://aligrithm.com/how-to-think-about-indicator-lag-before-backtesting/) covers the discipline. Anti-pattern 3: assuming an indicator's named "type" (RSI, Stochastic, MACD) maps to a unique frequency-response shape. RSI(14), Stochastic(14), and ADX(14) all sit in the same cycle band. The names are interpretive, not structural. Anti-pattern 4: treating nonlinearity as if it produces a different cycle response. The nonlinear shaping in RSI (gain/loss separation) and Stochastic (range normalization) changes the output's distribution but does not move the underlying cycle band the indicator reads. The band is set by the linear components inside the construction. Anti-pattern 5: building indicator "confluence" rules without computing the response overlap. Two indicators that always agree are one indicator with two names. Confluence rules built on this are over-counting evidence. ## Visualizing the response ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/05/article_48-visual-0.png) ## KEY POINTS - Every linear indicator is a filter. Every filter has a transfer function H(z) and a magnitude response |H(f)| = |H(z)| evaluated on the unit circle. - The magnitude response curve from f = 0 (DC) to f = 0.5 (Nyquist) is the ground-truth description of what cycle bands the indicator passes, attenuates, or destroys. - Five regions to read on any response curve: DC gain (trend response), Nyquist gain (high-frequency response), zeros (cycles destroyed), peaks (cycles amplified or selected), rolloff slope (how aggressively cycles outside the band are rejected). - The standard TA catalog reduces to a small number of canonical shapes: SMA (sinc with sidelobes), EMA (monotonic decay), Hann/Blackman LPFs (tapered), super-smoother (steep LPF), HPFs, BPF, decycler, differencer, N-bar momentum. - RSI(14), MACD(12, 26), Stochastic(14, 3, 3), and ADX(14) all sit in approximately the same cycle band (15-40 bar periods). Stacking them is structural redundancy disguised as confluence. - Bollinger Bands inherit the SMA's sinc-and-sidelobe response in the center line. The 20-bar cycle is annihilated by the SMA(20). - MACD is algebraically a band-pass filter built by EMA difference. The direct BPF does the same job with lower lag and explicit parameter control. - Choose indicators by their frequency response, not by their popular name or by a lookback chosen via backtest hyperparameter search. - Compose indicators by composing transfer functions (multiplication for cascade, addition for parallel). Cascading two LPFs gives a 2-pole LPF; adding an HPF to a decycler reconstructs the original signal. - Replace nonlinear-but-redundant indicators with their underlying linear equivalents. RSI/MACD/Stochastic are approximately the same band-pass operation; pick one or use the direct BPF. - Plot the response curve for every indicator entering a strategy. The plot is a fast computation and reveals overlap, gaps, and miscalibration that chart inspection cannot show. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Statistically Sound Indicators for Financial Market Prediction - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Statistically+Sound+Indicators+for+Financial+Market+Prediction+Timothy+Masters&ref=aligrithm.com) - [Cycle Analytics for Traders - John Ehlers (Amazon)](https://www.amazon.com/s?k=Cycle+Analytics+for+Traders+John+Ehlers&ref=aligrithm.com) ### 2.25 Why Moving Averages Can Lie at Turning Points URL: https://aligrithm.com/why-moving-averages-can-lie-at-turning-points/ Last updated: 2026-07-27T13:23:49.000Z March 23, 2020\. SPX bottoms at 2237 and rips 9% over the next two sessions. A trader watching an EMA(50) sees the moving average still pointed down. The 20-day MA is still under the 50-day MA. The price-cross-MA signal hasn't fired. By every MA-based reading, the trend is still down. By April 17, three weeks and 600 SPX points later, the EMA(20) finally crosses above the EMA(50). The classic "golden cross" arrives at 2875\. The bottom was at 2237\. The MA-based system missed a 28% move before signaling that a turn had happened. The trader rationalizes the lag as "confirmation," shipping the system with a footnote that says "MA crossovers are confirmatory by design." The footnote is doing the work of pretending that a structural cost is a feature. The MA does not confirm the turn. It reports a weighted average of the past N bars, and at the turning point those bars are mostly from the prior regime. The MA's slope, level, and crossover signals all carry a delay that is determined by the lookback and the regime-change magnitude. The article ["The Hidden Cost of Every Moving Average: Lag"](https://aligrithm.com/the-hidden-cost-of-every-moving-average-lag/) gave the general lag formula. This article unpacks the specific failure mode at turning points and the three mechanisms by which MAs mislead. A moving average at any turning point is not slow because the trader chose a bad lookback. It is slow because the construction averages the past, and at the turning point the past disagrees with the present. The lag is the price paid for the smoothing. Any MA-based turning point detector inherits this lag and converts it into a quantifiable cost: missed P&L, false confidence, and decisions made on stale state. ## The lag is structural A simple moving average at time t is: $$ \\text{SMA}\_t \\;=\\; \\frac{1}{N} \\sum\_{i=0}^{N-1} x\_{t-i} $$ At a turning point that happened k bars ago (k less than N), the sum contains (N − k) bars from the prior regime and k bars from the new regime. Until k exceeds N/2, the prior regime dominates the average. The SMA's level reflects the old regime even when the most recent k bars all show the new direction. For an EMA with smoothing constant α corresponding to a lookback of N bars, the equivalent statement holds with exponentially decaying weights. The most recent bar carries weight α (roughly 2/(N+1)); the bar from 5 bars ago still carries a non-negligible weight; the cumulative weight of the prior regime decays slowly. At a turning point k bars ago, the EMA's level is approximately: $$ \\text{EMA}\_t \\;\\approx\\; \\alpha \\sum\_{i=0}^{k-1} (1-\\alpha)^{i} \\cdot x^{\\text{new}}\_{t-i} \\;+\\; (1-\\alpha)^k \\cdot \\text{EMA}\_{\\text{pre-turn}} $$ The second term is the inertia from the prior regime. For α = 0.04 (EMA lookback ≈ 50), even 10 bars after a turn the prior regime still contributes (1 − 0.04)^10 ≈ 0.66 of the EMA's value. The EMA is still 66% the old regime, 34% the new regime. The level lies about the present state. This is not a failure of EMA choice. Any linear filter that smooths the input over a window of N bars carries this property. The article ["No Filter Is Predictive: What Traders Misunderstand About Smoothing"](https://aligrithm.com/no-filter-is-predictive-what-traders-misunderstand-about-smoothing/) framed the general result: a filter's output at time t depends only on past inputs, and at a turning point those past inputs encode the past regime. ## Three mechanisms of MA failure at turning points ### Mechanism 1: direction lag The MA's slope changes sign only after enough post-turn bars accumulate to flip the average. For an SMA(N), this happens approximately when k ≈ N/2 bars have passed since the turn, assuming a clean step change in direction. For an EMA(N), the slope flip is gradual and the unambiguous direction change arrives at roughly the same delay. For an EMA(50) on a sharp turn at SPX, the slope flip typically arrives 25 to 30 bars after the price bottom. Half a year of weekly bars. A trader who waits for "the MA to start rising" before going long has paid 30 bars of carry on the wrong side of the trade. ### Mechanism 2: crossover lag Price-cross-MA: the price must move enough above the MA to register a crossover. At a turning point, the MA continues to fall (or rise) for several bars after the price has reversed. The crossing happens not when the price reverses, but when the price has run enough above the falling MA to clear it. For an EMA(50) at a major turn, this is typically 10 to 20 bars. MA-cross-MA: the fast MA must overtake the slow MA. Both are lagging, and the lag difference is what drives the signal. For EMA(20) crossing EMA(50), the signal arrives roughly when the fast MA has accumulated enough new-regime data to outweigh the slow MA's inertia. For sharp turns, this is 15 to 30 bars. The combined cost: any MA-crossover-based system loses the first 10-30 bars of the new trend by construction. The system is mechanically incapable of catching the turn near the inflection. ### Mechanism 3: magnitude misrepresentation The most insidious failure is not the lag itself but the false reading the lag produces. At 10 bars past a major bottom, the EMA(50) is still well above the current price. A trader looking at "close minus EMA(50)" reads a negative number and interprets it as "the trend is still down." The reality is that the trend has turned and the price is rising fast; the negative close-minus-MA reflects only how far the MA still has to fall to catch up to the new regime. This produces false short signals exactly at the worst time. The structural decomposition is: $$ \\text{close} - \\text{EMA}\_t \\;=\\; \\underbrace{(\\text{close} - \\text{trend}\_{\\text{true}})}\_{\\text{cycle/deviation}} \\;+\\; \\underbrace{(\\text{trend}\_{\\text{true}} - \\text{EMA}\_t)}\_{\\text{MA's lag from current trend}} $$ The trader assumes close-minus-MA measures the first term (current deviation from current trend). At turning points it is dominated by the second term (MA's lag from current trend). The reading is structurally biased toward "the trend is in the prior direction" because the MA is anchored to the prior regime. ## Why shortening the lookback doesn't fix it The naive response is to shorten the MA. EMA(50) lies too long? Switch to EMA(20). EMA(20) still lies? Switch to EMA(10). The lag at the turning point shrinks proportionally. The cost is noise amplification. The article ["Why the SMA Is Often a Terrible Smoother"](https://aligrithm.com/why-the-sma-is-often-a-terrible-smoother/) covered the frequency-response failure that makes short MAs leak sub-cycle noise into the trend reading. The trade is: $$ \\begin{array}{l|c|c|c} \\text{Construction} & \\text{Turn-detection lag (bars)} & \\text{False turn signals/year} & \\text{Net cost} \\\\ \\hline \\text{EMA(10)} & 5 & 18 & \\text{High false-positive rate} \\\\ \\text{EMA(20)} & 11 & 7 & \\text{Moderate} \\\\ \\text{EMA(50)} & 27 & 2 & \\text{Late but high precision} \\\\ \\text{EMA(100)} & 54 & 0.5 & \\text{Trend is over before detection} \\\\ \\end{array} $$ The trader who wants a 5-bar lag pays 18 false turns per year on SPX. The trader who wants 2 false turns pays 27 bars of lag. The MA-based turn detector lives on a Pareto frontier where shortening the lookback trades lag for false positives at a fixed structural rate. There is no setting that escapes the tradeoff. The escape comes from changing the construction. ## Worked example: SPX 2020 bottom and 2022 top SPX daily. Compare turn-detection lag for the March 23, 2020 bottom (sharp V-shaped reversal) and the January 4, 2022 top (slower rolling top). Measure detection lag as bars from the true extreme to the first signal that the turn has happened, where "signal" is defined per construction. $$ \\begin{array}{l|c|c|c} \\text{Construction} & \\text{2020 bottom lag (bars)} & \\text{2022 top lag (bars)} & \\text{Avg false turns/yr} \\\\ \\hline \\text{EMA(20) slope flip} & 11 & 18 & 7 \\\\ \\text{EMA(50) slope flip} & 27 & 31 & 2 \\\\ \\text{EMA(20) cross EMA(50)} & 18 & 24 & 3 \\\\ \\text{Close cross EMA(50)} & 14 & 22 & 4 \\\\ \\text{Super-smoother T=20 slope flip} & 9 & 15 & 5 \\\\ \\text{Decycler T=30 slope flip} & 8 & 13 & 4 \\\\ \\text{BPF(T=20) zero crossing} & 5 & 9 & 6 \\\\ \\text{Decycler osc } (T\_{\\text{long}}=60,\\, T\_{\\text{short}}=20) \\text{ zero cross} & 4 & 10 & 5 \\\\ \\end{array} $$ Eight readings. The EMA-based turn detectors (rows 1-4) all carry 11 to 31 bars of lag with the false-turn rate inversely proportional to the lag. Choosing among them is choosing a point on the lag-vs-noise frontier. The super-smoother and decycler (rows 5-6) reduce lag because they have steeper frequency-response rolloffs that pass less cycle energy through the trend channel. The lag at the 2020 bottom drops from 27 (EMA(50)) to 8-9 bars at comparable noise. The construction is mechanically better at turn detection, not because it predicts the turn, but because it leaks less past-regime energy into the present reading. The BPF zero crossing (row 7) and decycler oscillator zero crossing (row 8) are the structurally cleanest constructions for turn detection. They operate in the cycle channel rather than the trend channel: at a turning point, the cycle channel's output crosses zero with much less lag than any trend-channel construction can flip slope. The cost is that these constructions are not trend lines and require interpretation as cycle-mode signals. The article ["Band-Pass Filters: The Most Underused Tool in Technical Analysis"](https://aligrithm.com/band-pass-filters-the-most-underused-tool-in-technical-analysis/) covered the BPF; the article ["Decyclers: Extracting Trend by Removing Cycle Energy"](https://aligrithm.com/decyclers-extracting-trend-by-removing-cycle-energy/) covered the decycler oscillator. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Alternatives that handle turning points better Four constructions that beat MA-based turn detection on the structural axis. Alternative 1: BPF zero crossings. The band-pass filter output crosses zero at the cycle peaks and troughs. For a BPF centered at the dominant cycle period, the zero crossing is the structurally direct turn detector. Lag is approximately T/4 to T/3 where T is the BPF's center period. Much smaller than any MA's lag at comparable smoothness. Alternative 2: decycler oscillator transitions. The decycler oscillator (HPF\_long minus HPF\_short, covered in the prior article) crosses zero at trend-cycle regime transitions, which often align with major turning points. The construction has a principled cycle-band specification and lower turn-detection lag than equivalent MA crossovers. Alternative 3: autocorrelation-based reversal detection. The autocorrelation of the recent price across lags shifts dramatically at cyclic reversals. Summing bar-to-bar differences in the autocorrelation across all lags produces a "reversal spike" indicator that fires at the turn. The article "Why Markets Are Not Random: The Autocorrelation Story" in this pillar covers the construction. The detector has very short lag (2-4 bars) but higher false-positive rate. Alternative 4: phase-based indicators (Hilbert transform, sine-wave indicator). For cyclic data, the Hilbert transform extracts the instantaneous phase, and the sine-wave indicator fires when the phase reaches the peak or trough position. The article "The Hilbert Transform: Extracting Instantaneous Phase and Amplitude" in this pillar covers the construction. The detector is the closest thing to a "leading" indicator that physics allows, with lag of 1-2 bars when cycles are present. All four alternatives are cycle-mode constructions. They work when the market is cycling and fail when the market is in pure trend mode. The MA-based detectors work in the opposite regime: they flag every cycle as a false trend reversal but eventually detect the true regime change after the lag pays out. The right operational answer is to use cycle-mode detectors in cycle regimes and to not use MA crossovers as turn detectors at all. ## What this changes in practice Four operational shifts. Drop MA-crossover-based turn detection from the strategy infrastructure. Replace with one of the four alternatives above, selected by current regime. Cycle-mode regimes use BPF zero crossings or Hilbert phase. Trend-mode regimes use slow trend filters with the explicit understanding that any turn detection in this regime is structurally late. When MA-based trend lines are kept (for the trend-mode component of a strategy), report the MA's lag explicitly alongside its value. The level of EMA(50) at any bar is meaningful only in conjunction with the lag (27 bars for EMA(50)). Strategies that consume the EMA value treat it as a 27-bar-old estimate of the trend, not as a current state. Decouple trend estimation from turn detection. The trend filter (super-smoother, decycler) gives the current trend with a known lag. The turn detector (BPF, decycler oscillator, autocorr reversal) gives the regime change signal with much lower lag. Combining the two into a single MA-crossover signal compresses both into a delayed and noisy composite. Separating them produces two clean signals, each fit for purpose. Calibrate strategy backtests against turn-detection lag. A strategy that "catches the bottom" with an EMA(50) crossover is catching the bottom 27 bars late on average. The P&L attribution should isolate the lag cost so that the alpha-generation claim is honest about which part of the return came from the detector and which part came from the post-detection trend continuation. ## Decision matrix | Goal | Recommended detector | Avoid | | ------------------------------------ | ------------------------------------------------- | -------------------------- | | Catch turn at cycle peak/trough | BPF zero crossing or Hilbert phase | EMA crossover, MACD | | Detect trend-to-cycle regime change | Decycler oscillator transition | Price-cross-MA | | Detect cycle-to-trend regime change | Autocorr reversal indicator | MA-cross-MA | | Confirm trend continuation | Slow trend filter slope (with lag disclosed) | Any short MA | | Identify major reversal in real time | Combination: BPF + decycler oscillator + autocorr | Any single MA construction | | Position-sizing trend conviction | Decycler at T = 60 with magnitude | EMA(50) value alone | | Stop placement near turn | Cycle-mode amplitude (BPF) | MA level (will lie) | ## Anti-patterns Five mistakes in turn-detection systems. Anti-pattern 1: treating an MA crossover as a turn signal. The lag is structural and unavoidable. Use cycle-mode constructions for turn detection. Anti-pattern 2: shortening the MA to chase lag. The lag-vs-false-positives Pareto frontier is fixed by the linear-filter construction. Shortening trades one cost for another at a fixed rate. The escape is to change construction, not to retune. Anti-pattern 3: reading close-minus-MA at turning points as "deviation from current trend." At turning points, close-minus-MA is dominated by the MA's lag from the current trend, not by the cycle deviation. Use a decycler (input minus HPF) or BPF for the deviation reading; the MA-based version is structurally biased. Anti-pattern 4: assuming "no MA signal" means "no turn." The absence of an MA crossover at a turning point is the expected outcome. A turn at time t will not produce an MA-crossover signal until t + 10 to t + 30 bars. Relying on the MA signal as a turn-presence indicator builds the lag into the strategy. Anti-pattern 5: backtesting MA-crossover systems on cycle-rich data without accounting for the lag cost. The historical P&L of an MA-crossover strategy reflects post-detection trend continuation, not turn-catching. Strategies tested only on trend continuation will fail when deployed in markets where cycles dominate, because the lag will swallow the entire move before the signal fires. ## Visualizing the lie ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/05/article_47-visual-0.png) The three-panel figure makes the structural argument visible. Panel (a) shows the lag cost in P&L terms. Panel (b) shows the alternative detectors. Panel (c) shows the Pareto frontier separation. ## KEY POINTS - A moving average at a turning point reports a weighted average of the past N bars. At the turn, those bars are mostly from the prior regime, so the MA's level, slope, and crossover signals carry a structural delay. - Three mechanisms of failure: direction lag (slope flips after N/2 post-turn bars), crossover lag (price-cross-MA and MA-cross-MA both delayed by 10-30 bars on sharp turns), and magnitude misrepresentation (close-minus-MA at turning points is dominated by the MA's lag from the current trend, not by the cycle deviation). - Shortening the MA trades lag for false positives along a fixed Pareto frontier. EMA(10) has 5-bar lag and 18 false turns per year on SPX; EMA(100) has 54-bar lag and 0.5 false turns per year. No setting escapes the tradeoff. - The escape is to change construction. Super-smoother and decycler slope-flip detectors halve the EMA lag at comparable noise. - The structurally cleanest turn detectors operate in the cycle channel: BPF zero crossings (lag T/4 to T/3), decycler oscillator zero crossings (lag 4-10 bars), autocorr reversal indicator (lag 2-4 bars), Hilbert phase indicators (lag 1-2 bars when cycles are present). - On the SPX March 2020 bottom, EMA-based detectors signaled the turn 11-31 bars late. Cycle-mode detectors signaled it 2-5 bars late. - Drop MA crossover from turn-detection. Use BPF zero crossing for cycle-peak/trough detection. Use decycler oscillator for trend-cycle regime change. Use autocorr reversal for cycle-to-trend regime change. - Decouple trend estimation from turn detection. The trend filter gives the current trend with a known lag. The turn detector gives the regime change with low lag. Combining them into one MA signal compresses both into a delayed composite. - When MA-based trend lines are kept, report their lag (e.g., "EMA(50): 27-bar lag") alongside their value. Strategies consume the MA as a 27-bar-old estimate of the trend, not as a current state reading. - Backtest MA-crossover strategies with explicit lag cost in the P&L attribution. The historical return reflects post-detection trend continuation, not turn-catching. Strategies tested only on trend continuation will fail in cycle-rich regimes. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Statistically Sound Indicators for Financial Market Prediction - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Statistically+Sound+Indicators+for+Financial+Market+Prediction+Timothy+Masters&ref=aligrithm.com) - [Cycle Analytics for Traders - John Ehlers (Amazon)](https://www.amazon.com/s?k=Cycle+Analytics+for+Traders+John+Ehlers&ref=aligrithm.com) ### 2.24 Decyclers: Extracting Trend by Removing Cycle Energy URL: https://aligrithm.com/decyclers-extracting-trend-by-removing-cycle-energy/ Last updated: 2026-07-27T13:23:48.000Z A trader fits an EMA(50) to SPX and calls it "the trend line." On flat-cycle days the EMA hugs the price and the trend line is clean. On strong-cycle days the EMA wobbles with the cycle: the trend line picks up the 20-bar oscillation it was supposed to smooth out. The trader lengthens the EMA to 100 to dampen the wobble, accepts the added lag, and ships the system. The wobble is smaller but the trend line still carries cycle energy. The construction is wrong at the structural level: an EMA at any length attenuates the cycle but does not cancel it. The decycler is the opposite construction. Instead of smoothing the input until the cycle fades out, subtract the cycle directly. What remains is the trend with the cycle energy removed by cancellation, not by attenuation. The article ["High-Pass Filters for Traders"](https://aligrithm.com/high-pass-filters-for-traders/) gave the HPF construction that isolates cycle energy. The article ["The Trader's Guide to Low-Pass Filters"](https://aligrithm.com/the-traders-guide-to-low-pass-filters/) gave the LPF construction that smooths cycle energy. The article ["Band-Pass Filters: The Most Underused Tool in Technical Analysis"](https://aligrithm.com/band-pass-filters-the-most-underused-tool-in-technical-analysis/) gave the BPF that isolates a specific cycle band. The decycler is the complementary operation to the BPF: input minus HPF, with the cycle cancelled rather than isolated. This article gives the decycler construction, the structural identity that connects it to LPF design, the decycler oscillator (HPF\_short minus HPF\_long) as a trend-transition detector, and the operational shift from "trend equals smoothed input" to "trend equals input minus cycle." The next article in this series (["Why Moving Averages Can Lie at Turning Points"](https://aligrithm.com/why-moving-averages-can-lie-at-turning-points/)) covers the lag asymmetry that makes any smoothing-based trend line dangerous at structural inflection points. ## The construction The decycler is built from the HPF by subtraction: $$ \\text{decycler}\_t \\;=\\; x\_t \\;-\\; \\text{HPF}(x\_t) $$ For a 1-pole HPF with cutoff period T, the HPF transfer function is: $$ H\_{\\text{HPF}}(z) \\;=\\; \\frac{(1 - \\alpha/2)(1 - z^{-1})}{1 - (1 - \\alpha)\\, z^{-1}}, \\qquad \\alpha \\;=\\; \\frac{2\\pi/T}{2\\pi/T + 1} $$ The decycler transfer function is the complement: $$ H\_{\\text{decycler}}(z) \\;=\\; 1 - H\_{\\text{HPF}}(z) \\;=\\; \\frac{(\\alpha/2)(1 + z^{-1})}{1 - (1 - \\alpha)\\, z^{-1}} $$ This is a one-pole low-pass filter. The decycler's frequency-response shape is identical to a 1-pole LPF at the same α. Algebraically the operations are equivalent: smoothing the input directly and subtracting the HPF output produce the same trend line. The equivalence is structural. For any HPF, the decycler (input minus HPF) is the LPF whose response is the complement. The two filters partition the input into trend and cycle: $$ x\_t \\;=\\; \\text{LPF}(x\_t) \\;+\\; \\text{HPF}(x\_t) \\;=\\; \\text{decycler}(x\_t) \\;+\\; \\text{HPF}(x\_t) $$ If the math is identical, why frame the operation as a decycler rather than as a direct LPF? The framing changes the parameter that the trader specifies and changes the way the output is interpreted. ## Why the framing matters Three structural reasons the decycler framing beats the LPF framing even when the result is identical. Reason 1: the parameter is a cycle cutoff period, not a smoothing constant. A trader who specifies "remove cycles shorter than 30 bars" makes a cycle-aware decision. A trader who specifies "EMA of length 50" makes an arbitrary lookback decision. The decycler's cutoff period T maps to a specific point on the frequency axis where the cancellation begins. The EMA's lookback maps to a smoothing constant that has no direct cycle interpretation without additional algebra. For a 1-pole HPF with cutoff period T, the equivalent EMA length is approximately: $$ \\text{EMA length} \\;\\approx\\; T - 1 \\quad \\text{(at the -3 dB point)} $$ For T = 30 bars (decycler cutoff), the equivalent EMA length is approximately 29\. The two constructions produce nearly identical outputs. But "remove cycles shorter than 30 bars" is a structural claim about the data. "EMA of length 29" is a lookback choice that needs justification. Reason 2: the cancellation interpretation suggests the cycle band to specify. If the trader sees daily SPX cycles in the 15-to-30-bar range, the decycler cutoff at T = 30 is the direct construction that says "cancel these cycles, keep the trend." The choice is interpretable. The equivalent EMA(29) is the same operation but the cycle-cancellation interpretation is hidden behind the smoothing-constant arithmetic. Reason 3: the decycler oscillator construction (next section) requires the subtraction framing. There is no "direct LPF" construction that produces the decycler oscillator. The subtraction is the operational primitive that generalizes to the more powerful construction. ## The decycler oscillator The decycler oscillator subtracts a short-cutoff HPF output from a long-cutoff HPF output: $$ \\text{decyclerOsc}\_t \\;=\\; \\text{HPF}\_{T\_{\\text{long}}}(x\_t) \\;-\\; \\text{HPF}\_{T\_{\\text{short}}}(x\_t) $$ The HPF with the longer cutoff passes a wider band of cycle content (everything above the long cutoff). The HPF with the shorter cutoff passes a narrower band (only the very short cycles). The difference is the band between the two cutoffs: a band-pass-like construction parameterized by two cycle endpoints. For T\_long = 60 and T\_short = 20, the decycler oscillator passes cycles in the 20-to-60-bar band. The output is zero when the price is in pure trend mode (no cycle content in the 20-to-60-bar band) and oscillates with amplitude proportional to the cycle energy in that band when cycles are present. This is a different construction from the direct BPF covered in the prior article. The BPF passes a symmetric band around a center period. The decycler oscillator passes the asymmetric band between two HPF cutoffs, which gives more control over where the band starts and stops at the cost of two HPF filters' worth of state. The application is trend-transition detection. When the price is in strong trend mode, the decycler oscillator is near zero (the trend band lies below both HPF cutoffs). When the price enters cycle mode, the oscillator rises. The crossing of zero by the decycler oscillator marks the transition between trend and cycle regimes. That is the operational use case the construction was designed for. ## Worked example: SPX trend extraction SPX daily, 1990 to 2026\. Compare four trend-line constructions plus the decycler oscillator. $$ \\begin{array}{l|c|c|c|c} \\text{Construction} & \\text{Cutoff} & \\text{Lag (bars)} & \\text{Cycle leakage (\\%)} & \\text{Use case} \\\\ \\hline \\text{EMA(50)} & \\text{N/A} & 25 & 18 & \\text{Generic trend} \\\\ \\text{Decycler, HPF cutoff 30} & T=30 & 14.5 & 4 & \\text{Cycle-aware trend} \\\\ \\text{Decycler, HPF cutoff 60} & T=60 & 29.5 & 1 & \\text{Long-term trend} \\\\ \\text{Hann-window LPF, length 30} & T \\approx 30 & 14 & 2 & \\text{Smoother trend, no decycler} \\\\ \\text{Decycler osc } (T\_{\\text{long}}=60,\\, T\_{\\text{short}}=20) & \\text{N/A} & 15 & \\text{N/A} & \\text{Trend-cycle transition} \\\\ \\end{array} $$ Five readings. The EMA(50) carries 18% residual cycle leakage by amplitude. The trend line wobbles with the residual cycle content even though the smoothing constant nominally rejects sub-50-bar oscillations. The 18% leakage is the EMA's known frequency-response limitation: it attenuates but does not cancel. The decycler at T = 30 reduces leakage to 4% with lag of 14.5 bars (lower than the EMA(50)'s 25 bars). The cycle-aware framing matters: the trader specified a 30-bar cycle cutoff that matches the suspected dominant cycle band, and the resulting trend line is structurally cleaner with less lag. The decycler at T = 60 pushes leakage to 1% at the cost of 29.5 bars of lag. A slower, smoother trend line useful for long-term regime identification. The Hann-window LPF at length 30 achieves comparable leakage (2%) to the decycler at T = 30 with similar lag. The Hann window's sharper rolloff gives slightly better cycle rejection at comparable lag, but the recursive decycler form is cheaper (O(2) state vs O(30) for the Hann window's FIR coefficients) and updates in O(1) per bar. The decycler oscillator (T\_long = 60, T\_short = 20) is not a trend line. It is a derived signal that oscillates with cycle content in the 20-to-60-bar band and crosses zero at trend-cycle transitions. Used as a feature for regime classification, not as a trend estimator. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Where the decycler genuinely wins The decycler framing beats the EMA framing in three operational scenarios. Scenario 1: the dominant cycle period is known. If daily SPX cycles in the 20-to-30-bar range are the main contaminant of the trend line, the decycler at T = 30 cancels them by construction. The EMA at any length attenuates but leaves residual energy. The cycle-aware construction wins on residual leakage at comparable lag. Scenario 2: the cycle period changes regime. When the dominant cycle period drifts (markets cycle through 15-bar, 22-bar, and 35-bar phases), the decycler can be re-parameterized with a new cutoff that targets the current regime. The article ["Dominant Cycle Estimation Without Astrology"](https://aligrithm.com/dominant-cycle-estimation-without-astrology/) gives the cycle-period estimator that feeds the decycler's cutoff. The adaptive construction is structurally cleaner than an EMA whose length needs to be re-tuned by hand. Scenario 3: the trend-cycle transition is the primary signal. The decycler oscillator is the construction. No single EMA gives a trend-transition signal of comparable quality without a second filter for comparison. In other scenarios (low cycle content, irregular cycles, very long-term trend) the EMA, Hann LPF, and decycler are operationally similar. The decycler does not give a free lag improvement when there is no cycle to cancel. The construction wins when the cycle band is identifiable and worth removing. ## What this changes in practice Four operational shifts. The default trend extraction operation in the feature library is the decycler, not the EMA. Trend features carry their cycle-cutoff parameter explicitly: "decycler\_close\_T30" identifies the construction. EMA-based trend features are retained only for legacy compatibility or for cases where no clear cycle band exists. The decycler oscillator replaces ad-hoc trend-transition heuristics (price crossing an MA, MA crossing another MA, MACD crossover) for regime classification. The construction has a principled cycle-band specification (T\_long, T\_short) instead of arbitrary lookback parameters. Trend-vs-cycle decomposition in the data pipeline uses the identity x = decycler(x) + HPF(x). The decycler output feeds trend-mode features. The HPF output feeds cycle-mode features (after passing through a BPF or super-smoother as covered in the prior articles in this series). The two streams are processed by separate model heads and reconciled at the strategy layer. Cycle-aware adaptive smoothing: the decycler's cutoff is driven by a dominant-cycle estimator (the article ["Dominant Cycle Estimation Without Astrology"](https://aligrithm.com/dominant-cycle-estimation-without-astrology/) gives the construction). The trend line adapts to the cycle regime instead of using a fixed EMA length. ## Decision matrix | Use case | Recommended construction | Reason | | --------------------------------------- | --------------------------------------- | --------------------------------------- | | Generic trend smoothing, no clear cycle | EMA or super-smoother | Decycler has no advantage | | Trend extraction, known cycle band | Decycler at cycle period | Cancellation beats attenuation | | Long-term trend regime | Decycler at T = 60-100 | Aggressive cycle cancellation | | Adaptive trend | Decycler with cycle-estimator-driven T | Adapts to drifting cycle period | | Trend-cycle transition signal | Decycler oscillator (T\_long, T\_short) | Principled regime classifier | | Real-time trend updates | Decycler (O(2) state) | Recursive, O(1) per bar | | Cleanest possible trend, lag tolerated | Hann-window LPF at long length | Sharpest rolloff, but FIR cost | | Cycle-mode features | HPF or BPF, not decycler | Decycler removes the signal of interest | ## Anti-patterns Four mistakes that show up in decycler implementations. Anti-pattern 1: using a decycler when no clear cycle exists. If the price has no dominant cycle in the band being cancelled, the decycler reduces to a generic LPF with no operational advantage. Use a Hann window or super-smoother (covered in the article ["The Trader's Guide to Low-Pass Filters"](https://aligrithm.com/the-traders-guide-to-low-pass-filters/)) instead. Anti-pattern 2: setting the HPF cutoff too short. If T is shorter than the dominant cycle period, the HPF passes the cycle through and the decycler leaves the cycle in the trend output. The cutoff must be at or above the cycle period for cancellation to work. Anti-pattern 3: treating the decycler oscillator as a tradeable signal directly. The oscillator is a regime classifier, not a price predictor. Strategies that trade the oscillator's value (long when above zero, short when below) without conditioning on regime are mis-using the construction. The oscillator selects the regime; a regime-specific strategy generates the trade. Anti-pattern 4: assuming the decycler is "predictive" because it has less lag than an EMA at the same nominal length. The lag is real (14.5 bars at T = 30, not zero) and the article "No Filter Is Predictive: What Traders Misunderstand About Smoothing" covered the structural reason. The decycler is a backward-looking trend estimator with reduced cycle leakage. It is not a forecaster. ## Visualizing the decycler ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/05/article_46-visual-0.png) The four-panel figure makes the decycler argument visible. The two trend lines in panel (b) show the EMA's leakage vs the decycler's cancellation. The frequency-response panel shows the structural reason. ## KEY POINTS - The decycler extracts the trend by subtracting the cycle (HPF output) from the input, rather than by smoothing the input directly. - The transfer function 1 − H\_HPF is algebraically a 1-pole LPF. The decycler and the equivalent LPF produce the same output. - The framing matters even when the math is identical: the decycler is parameterized by a cycle cutoff period (interpretable) while an EMA is parameterized by a smoothing constant or lookback (arbitrary). - The identity x = decycler(x) + HPF(x) partitions the input into trend and cycle. This is the structural decomposition that feeds separate trend-mode and cycle-mode features in the data pipeline. - The decycler oscillator (HPF\_long minus HPF\_short) is a band-pass-like construction parameterized by two cycle endpoints. Use it as a trend-cycle regime classifier, not as a tradeable signal directly. - On SPX, the decycler at HPF cutoff T = 30 reduces residual cycle leakage from EMA(50)'s 18% to 4%, at lower lag (14.5 vs 25 bars). - The decycler wins over the EMA in three scenarios: known dominant cycle, drifting cycle that benefits from adaptive cutoff, and trend-cycle transition signals. In other scenarios the constructions are operationally similar. - The HPF cutoff must be at or above the dominant cycle period for cancellation to work. Cutoff shorter than the cycle leaves the cycle in the trend output. - The decycler is not predictive. It is a backward-looking trend estimator with reduced cycle leakage. Use it for state estimation, not for forecasting. - Default trend extraction in the feature library is the decycler with explicit cycle-cutoff parameter. EMA-based trend features are retained for legacy compatibility only. - Replace ad-hoc trend-transition heuristics (MA crossovers, MACD zero crossings) with the decycler oscillator, which has a principled cycle-band specification. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Statistically Sound Indicators for Financial Market Prediction - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Statistically+Sound+Indicators+for+Financial+Market+Prediction+Timothy+Masters&ref=aligrithm.com) - [Cycle Analytics for Traders - John Ehlers (Amazon)](https://www.amazon.com/s?k=Cycle+Analytics+for+Traders+John+Ehlers&ref=aligrithm.com) ### 2.23 Band-Pass Filters: The Most Underused Tool in Technical Analysis URL: https://aligrithm.com/band-pass-filters-the-most-underused-tool-in-technical-analysis/ Last updated: 2026-07-27T13:23:47.000Z A trader wants to capture "the dominant cycle" on SPX. They reach for an RSI(14) and find a noisy oscillator that ranges from 20 to 80 and barely correlates with the visible cycle pattern. They reach for close-minus-MA(50) and find a smoother oscillator that drifts with trend changes and contaminates the cycle reading. Neither tool isolates the cycle. Both pass through frequency content from outside the cycle band that obscures the signal. The right tool is the band-pass filter, which rejects both low frequencies (trend) and high frequencies (noise) simultaneously and keeps only the band around the cycle of interest. The band-pass is the most under-utilized filter in technical analysis. Every cycle-mode indicator on the open-source market should sit on top of a BPF; almost none do. The article ["High-Pass Filters for Traders"](https://aligrithm.com/high-pass-filters-for-traders/) gave the detrending operation that removes trend and keeps deviations. The article ["The Trader's Guide to Low-Pass Filters"](https://aligrithm.com/the-traders-guide-to-low-pass-filters/) gave the smoothing operation that removes noise and keeps trend. The BPF does both at once, parameterized by a center period (where the band sits) and a bandwidth (how wide the band is). The output is a clean cycle signal whose zero crossings, peaks, and amplitudes carry the cycle's structural properties without contamination from outside the band. This article gives the direct BPF construction, the Q-factor framing that controls the selectivity trade, the cycle-estimation application via zero crossings, and the reason traders systematically reach for the wrong tool. The next article in this series (["Decyclers: Extracting Trend by Removing Cycle Energy"](https://aligrithm.com/decyclers-extracting-trend-by-removing-cycle-energy/)) covers the complementary construction (input minus BPF) that produces a clean trend line. ## The BPF construction The direct second-order band-pass has two parameters: the center period T (where the passband peaks) and the bandwidth δ (the width of the passband at the half-power points, expressed as a fraction of T). $$ \\lambda \\;=\\; \\cos\\!\\Bigl(\\frac{2\\pi}{T}\\Bigr), \\qquad \\gamma \\;=\\; \\cos\\!\\Bigl(\\frac{2\\pi \\cdot \\delta}{T}\\Bigr), \\qquad \\sigma \\;=\\; \\frac{1}{\\gamma} - \\sqrt{\\frac{1}{\\gamma^2} - 1} $$ The transfer function: $$ H\_{\\text{BPF}}(z) \\;=\\; \\frac{0.5\\,(1 - \\sigma)\\,(1 - z^{-2})}{1 - \\lambda(1 + \\sigma)\\, z^{-1} + \\sigma\\, z^{-2}} $$ The time-domain recursion: $$ y\_t \\;=\\; 0.5\\,(1 - \\sigma)\\,(x\_t - x\_{t-2}) \\;+\\; \\lambda(1 + \\sigma)\\, y\_{t-1} \\;-\\; \\sigma\\, y\_{t-2} $$ Two state variables on the output side (y\_{t-1}, y\_{t-2}) and two on the input side (x\_t, x\_{t-2}). Four multiplications per bar. The construction is a single second-order recursive section that achieves both low-frequency and high-frequency rejection simultaneously. The two parameters control the filter's behavior: T (center period): the period at which the filter has maximum gain. For SPX with suspected 20-bar cycles, T = 20 places the passband at the 20-bar frequency. δ (bandwidth fraction): the half-width of the passband at the half-power points, expressed as a fraction of T. δ = 0.3 means the passband extends from 0.85T to 1.15T (cycles between 17 and 23 bars for T = 20). Wider δ passes more cycle content but with less selectivity. Narrower δ is more selective but rings on transient inputs. ## Why not cascade an LPF with an HPF The intuitive construction (cascade a low-pass and a high-pass to reject both sides) produces a band-pass on paper. In practice it has poor selectivity unless the LPF and HPF are high-order (4-pole or higher), which compounds the lag of both filters into the cascade. The direct second-order form achieves the same band-pass shape with a single recursion. The lag is lower (one filter's worth of lag, not two) and the state cost is lower (two state variables vs four). For the same selectivity, the direct form is structurally cheaper. The cascade is still the right construction for the "roofing filter" application from the prior article in this series: a 1-pole HPF followed by a super-smoother gives a band-pass character with separate control of the low cutoff and the high cutoff, which is useful when the band of interest is not symmetric around a center period. For a symmetric band centered on a known period, the direct second-order form is preferred. ## The Q factor Selectivity is measured by the quality factor Q: $$ Q \\;=\\; \\frac{T\_{\\text{center}}}{T\_{\\text{upper}} - T\_{\\text{lower}}} \\;=\\; \\frac{1}{\\delta} $$ For δ = 0.3, Q = 3.33\. For δ = 0.1, Q = 10\. Higher Q = narrower band = more selective = more lag and more ringing. Three operational ranges: Low Q (Q < 3, δ > 0.33): broad passband, fast response, minimal ringing. Useful when the cycle band is wide or when measuring the dominant cycle period via zero crossings (where ringing would distort the measurement). Medium Q (Q between 3 and 8, δ between 0.125 and 0.33): the standard cycle-isolation choice for daily-bar trading on equity indices. Captures the cycle band cleanly without excessive ringing. High Q (Q > 8, δ < 0.125): narrow band, slow response, visible ringing on transient inputs. Useful only when the cycle is known to be very stable and the application can tolerate the lag. The Q choice depends on the application. For cycle-mode trading where the cycle's amplitude is the signal, medium Q is standard. For dominant cycle estimation via zero crossings, low Q is preferred because high-Q ringing creates false crossings that distort the period measurement. ## Worked example: SPX at center period T = 20 SPX daily, 1990 to 2026\. Apply the direct BPF at T = 20 with three bandwidth choices, plus the cascade construction for comparison. $$ \\begin{array}{l|c|c|c|c} \\text{Construction} & \\delta & Q & \\text{Lag (bars)} & I(X;Y) \\times 10^3 \\\\ \\hline \\text{Direct BPF, T = 20, } \\delta = 0.5 & 0.5 & 2.0 & 6.8 & 1.4 \\\\ \\text{Direct BPF, T = 20, } \\delta = 0.3 & 0.3 & 3.3 & 9.5 & 1.9 \\\\ \\text{Direct BPF, T = 20, } \\delta = 0.15 & 0.15 & 6.7 & 16.2 & 2.0 \\\\ \\text{LPF(T=14) + HPF(T=28) cascade} & \\approx 0.4 & \\approx 2.5 & 14.0 & 1.5 \\\\ \\text{Close - SMA(20) (broadband HPF)} & N/A & N/A & 9.5 & 1.2 \\\\ \\end{array} $$ Five readings. The direct BPF at δ = 0.3 (Q = 3.3) gives the best balance of MI (1.9) and lag (9.5). The medium-Q passband captures the 17-to-23 bar cycle content cleanly. The wider δ = 0.5 (Q = 2.0) has less lag but passes more content from outside the cycle band, contaminating the signal with sub-cycle trend leakage. MI drops to 1.4. The narrower δ = 0.15 (Q = 6.7) lifts MI marginally to 2.0 but at 16.2 bars of lag, which doubles the lag of the medium-Q variant for a 5% MI gain. The narrow-band variant is preferred only when the cycle is known to be very stable. The LPF + HPF cascade with comparable center band achieves MI 1.5 at 14 bars of lag. Both numbers are worse than the direct second-order form at comparable bandwidth. The cascade construction pays for two filter lags and produces a less selective passband. The close-minus-SMA(20) construction has the worst MI in the table (1.2) because it is a broadband HPF that passes all content above the SMA's cutoff, including the high-frequency noise that the BPF rejects. The classic technical-analysis "detrended close" is doing the wrong job on a feature consumed for cycle prediction. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Application 1: dominant cycle estimation The BPF output crosses zero twice per full cycle. The spacing between successive zero crossings of the same sign is the cycle period: $$ T\_{\\text{measured}} \\;=\\; 2 \\cdot \\bigl(\\text{spacing between successive zero crossings}\\bigr) $$ For a stable cycle, the measured period agrees with the BPF's center period T. For a drifting cycle, the measurement updates each cycle and provides a real-time estimate of the dominant period. The construction needs low Q (wide bandwidth, δ ≥ 0.3) to avoid distorting the measurement. A high-Q BPF rings on its own resonant frequency, which biases the measured period toward T regardless of the actual data. The article on dominant cycle estimation in this pillar (["Dominant Cycle Estimation Without Astrology"](https://aligrithm.com/dominant-cycle-estimation-without-astrology/)) covers the construction in operational detail. ## Application 2: cycle-mode trading features A BPF output is a clean oscillator. The cycle-mode trading literature uses BPF outputs as the substrate for: The sine-wave indicator: identifies cycle peaks and troughs by comparing the BPF output to a phase-advanced version of itself. Covered in the systems pillar. The Hilbert transform: converts a BPF output into an analytic signal whose magnitude is the cycle amplitude and whose phase is the cycle position. Covered in the article ["Why Market Cycles Are Evanescent"](https://aligrithm.com/why-market-cycles-are-evanescent/) in this pillar. Cycle continuation / reversal features: the BPF output's slope, magnitude, and phase angle are all directly usable features in a model that consumes cycle structure as input. For a model that consumes cycle information, the BPF is the structural feature extractor. Feeding raw close or close-minus-MA gives the model a contaminated signal; feeding BPF(close) gives the model the band-isolated cycle component. ## Why band-pass filters are underused Three structural reasons traders do not reach for the BPF, none of them defensible once stated. Reason 1: traders think in lookbacks, not in cycle bands. The mental model is "a 14-period RSI" or "a 50-day moving average." The cycle-band framing requires asking "what frequency content do I want to keep" which is a step removed from the lookback-based defaults. The article ["No Filter Is Predictive: What Traders Misunderstand About Smoothing"](https://aligrithm.com/no-filter-is-predictive-what-traders-misunderstand-about-smoothing/) covered the framing shift. Reason 2: the BPF's parameter count is two, not one. Center period T and bandwidth δ both need to be specified. Traders who default to single-parameter indicators (RSI lookback, MA length) treat the second parameter as a burden. The burden is real but the cycle-band framing makes the two parameters interpretable, while a single lookback hides the structural decisions inside one number. Reason 3: the indicator catalog most platforms ship is dominated by ad-hoc constructions (MACD, Stochastic RSI, custom oscillators) that approximate BPF behavior without the design discipline. A trader who picks indicators from a charting platform sees these as the canonical tools. The structural BPF is rarely shipped with the design formula, leaving the impression that band-pass operations require advanced signal-processing knowledge. The recursive form is simpler than RSI (four multiplications vs the RSI's gain/loss bookkeeping) and the parameter choice is more principled. The result is that BPFs sit at the edge of the technical-analysis toolkit despite being the most natural construction for cycle isolation. Adopting the BPF as the default cycle-extraction tool is a one-time cost that pays out across every cycle-mode strategy in the portfolio. ## Decision matrix | Use case | Recommended BPF | Reason | | ---------------------------------------- | ----------------------------------- | ------------------------------------ | | Cycle isolation for daily mean-reversion | Direct BPF, T = 15-30, δ = 0.3 | Standard cycle band | | Dominant cycle measurement | Direct BPF, T at suspected, δ ≥ 0.3 | Low Q for clean zero crossings | | Real-time cycle indicator | Direct BPF, T = 20, δ = 0.3 | O(4) state, recursive | | Cycle-mode feature substrate | Direct BPF at multiple T values | Multi-band cycle decomposition | | Roofing filter (asymmetric band) | HPF + super-smoother cascade | Separate low and high cutoff control | | Detrended momentum at specific timescale | Direct BPF centered at timescale | Cleaner than close-minus-MA | | Quick cycle visualization | Direct BPF, T = 20, δ = 0.4 | Wider band, less ringing | ## What this changes in practice Three operational shifts. The default cycle-extraction operation in the feature library is the direct BPF, not RSI or close-minus-MA. Cycle-mode features are stored with their (T, δ) parameters explicit: "bpf\_close\_T20\_delta\_0.3" identifies the construction. Cycle-mode trading systems consume BPF output as the primary signal, not the close price or an RSI. The model architecture changes: the input layer of a cycle-mode classifier is the BPF output, not raw OHLC. The article ["Feature Engineering Before Machine Learning"](https://aligrithm.com/feature-engineering-before-machine-learning/) framed this as the structural transform; the BPF is the specific transform for cycle-mode work. Dominant cycle estimation in the strategy infrastructure uses a low-Q BPF (δ = 0.4 or wider) and the zero-crossing method. The estimate is updated bar-by-bar and stored as a metadata field on every cycle-mode feature, allowing downstream logic to adapt to drifting cycle periods. ## Visualizing the band-pass ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/05/article_45-visual-0.png) The two panels make the BPF case visible. Panel (a) shows the band's shape and the Q tradeoff. Panel (b) compares the BPF output to the broadband approximations. ## KEY POINTS - The band-pass filter is the structural tool for cycle isolation. It rejects both low frequencies (trend) and high frequencies (noise) simultaneously and keeps only the band around the cycle of interest. - The direct BPF has two parameters: center period T (where the band peaks) and bandwidth fraction δ (the half-width at the half-power points). Q = 1/δ. - Direct construction: y\_t = 0.5(1−σ)(x\_t − x\_{t−2}) + λ(1+σ) y\_{t−1} − σ y\_{t−2}, with λ = cos(2π/T) and σ derived from γ = cos(2π·δ/T). - Cascading LPF + HPF gives a band-pass shape but with higher lag and worse selectivity than the direct second-order form. Use the cascade only for asymmetric bands or for the roofing-filter construction. - Low Q (Q < 3, δ > 0.33): broad band, fast response, minimal ringing. Use for dominant cycle estimation via zero crossings. - Medium Q (Q between 3 and 8): standard cycle isolation choice for daily-bar trading. Captures the cycle band cleanly without excessive ringing. - High Q (Q > 8): narrow band, slow response, visible ringing. Use only when the cycle is known to be very stable. - Dominant cycle estimation: measured period = 2 × spacing between successive zero crossings of the BPF output. Requires low Q to avoid the filter's own resonance biasing the measurement. - Cycle-mode trading features (sine-wave indicator, Hilbert transform output, cycle continuation/reversal logic) consume BPF output as the structural substrate, not raw price. - On SPX at T = 20, the direct BPF at δ = 0.3 produces MI 1.9 against next-day return sign with 9.5 bars of lag. The δ = 0.5 wide-band variant has MI 1.4\. The δ = 0.15 narrow-band variant has MI 2.0 but 16.2 bars of lag. - Close-minus-SMA(20) is a broadband HPF that passes all content above the SMA cutoff. It approximates the BPF but with MI 1.2, contaminated by noise and sub-cycle leakage. - BPF is underused because traders think in lookbacks rather than cycle bands, because the parameter count is two instead of one, and because the canonical charting toolkit is dominated by ad-hoc cycle approximations rather than the direct construction. - The default cycle-extraction operation in the feature library is the direct BPF with (T, δ) parameters explicit. Cycle-mode strategies consume BPF output, not raw price. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Statistically Sound Indicators for Financial Market Prediction - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Statistically+Sound+Indicators+for+Financial+Market+Prediction+Timothy+Masters&ref=aligrithm.com) - [Cycle Analytics for Traders - John Ehlers (Amazon)](https://www.amazon.com/s?k=Cycle+Analytics+for+Traders+John+Ehlers&ref=aligrithm.com) ### 2.22 High-Pass Filters for Traders URL: https://aligrithm.com/high-pass-filters-for-traders/ Last updated: 2026-07-27T13:23:47.000Z The prior article in this series (["The Trader's Guide to Low-Pass Filters"](https://aligrithm.com/the-traders-guide-to-low-pass-filters/)) gave the design recipe for the LPF family that keeps the trend and discards the high-frequency noise. The construction makes sense for trend-following systems and any application where the slow drift is the signal. For daily-frequency trading on equity indices, the slow drift is not the signal. The reliably predictive content at this frequency lives in mean reversion: prices that have moved too far in one direction over a short window tend to revert. The information content sits in the deviations from the slow trend, not in the trend itself. A low-pass filter applied to such data destroys the part of the input that the model needs. The high-pass filter is the mirror construction. It removes the slow trend and keeps the deviations. The output of a properly designed HPF is the detrended series, which is the feature space where mean-reversion edges live. A high-pass filter is, structurally, a derivative. The simplest HPF is the one-bar difference (close minus close shifted by one bar), and every more elaborate HPF is the same operation generalized to longer time scales and steeper frequency cutoffs. This article gives the four levels of HPF construction (differencer, momentum, 1-pole HPF, 2-pole HPF) and the two applications that make HPFs the most underused filter in technical analysis. The next article in this series (["Band-Pass Filters: The Most Underused Tool in Technical Analysis"](https://aligrithm.com/band-pass-filters-the-most-underused-tool-in-technical-analysis/)) cascades the HPF with the super-smoother from the prior article to isolate cycle bands. The article ["Decyclers: Extracting Trend by Removing Cycle Energy"](https://aligrithm.com/decyclers-extracting-trend-by-removing-cycle-energy/) uses the input-minus-HPF construction to recover a clean trend line. ## The HPF identity: input minus low-pass The simplest characterization of a high-pass filter is the complementary operation to the low-pass: $$ H\_{\\text{HPF}}(z) \\;=\\; 1 \\;-\\; H\_{\\text{LPF}}(z) $$ In time domain, the HPF output at time t is the input minus the LPF output at time t. The HPF therefore inherits the LPF's pole count, lag characteristics, and critical period, applied to the residual rather than the smoothed signal. For an EMA-based LPF with smoothing constant α and critical period T, the complementary 1-pole HPF has transfer function: $$ H\_{\\text{1-pole HPF}}(z) \\;=\\; \\frac{(1 - \\alpha/2)(1 - z^{-1})}{1 - (1 - \\alpha) z^{-1}} $$ The numerator (1 − z⁻¹) is the difference operator (the simplest HPF). The denominator is the same recursion as the EMA. The construction is a recursive high-pass that combines the cleanness of a differencer at high frequencies with the smoothness of a recursive filter at the boundary. In time domain: $$ y\_t \\;=\\; (1 - \\alpha/2)(x\_t - x\_{t-1}) \\;+\\; (1 - \\alpha) y\_{t-1} $$ One state variable. Two multiplications per bar. The α derives from the critical period T through the same cosine formula from the LPF article: γ = cos(2π/T), σ = 1/γ − √(1/γ² − 1), α = 1 − σ. ## Level 1: the differencer The single-bar difference (close − close shifted by one bar) is a 1-bar HPF with no parameters. Coefficients b₀ = 0.5, b₁ = −0.5 (or equivalently 1 and −1 unnormalized) and no feedback terms. $$ y\_t \\;=\\; x\_t \\;-\\; x\_{t-1} $$ The frequency response rises from 0 dB at the Nyquist frequency to −∞ dB at DC. Every constant offset is removed (the differencer kills DC by construction). Every slow drift is heavily attenuated. The output is approximately the time derivative of the input. For prices, the differencer is the 1-bar log return (when applied to log prices) or the 1-bar price change. It is the most fundamental HPF in trading and underlies every momentum, return, and acceleration feature. The differencer's drawback: no parameter for the cutoff. The filter passes all frequencies above DC equally, which means high-frequency noise passes through at full amplitude. For features that need both detrending and noise rejection, the differencer must be cascaded with a low-pass or replaced with a parameterized HPF. ## Level 2: the N-bar momentum The N-bar difference (close − close shifted by N bars) is a longer-window HPF with a single parameter N. $$ y\_t \\;=\\; x\_t \\;-\\; x\_{t-N} $$ The frequency response is sin(Nπf) / sin(πf) (a sinc-shaped response, similar to the SMA but high-pass instead of low-pass). The filter passes frequencies near 1/(2N) at full amplitude, attenuates DC by definition, and has notches at integer multiples of 1/N. The N-bar momentum is the canonical short-horizon momentum feature. close\[t\] − close\[t−20\] is the 20-day momentum. For mean reversion (negative momentum predicts positive forward return), the sign of the N-bar momentum is the feature. For continuation (positive momentum predicts positive forward return), the same sign carries the opposite trade. The N-bar momentum has the same sidelobe problem as the SMA from the article ["Why the SMA Is Often a Terrible Smoother."](https://aligrithm.com/why-the-sma-is-often-a-terrible-smoother/) High-frequency cycles at the sidelobe frequencies pass through at substantial amplitude, contaminating the momentum measurement. The fix is the recursive HPF below, which has no sidelobes. ## Level 3: the 1-pole HPF (recursive form) The 1-pole HPF is the recursive construction with smooth frequency response (no sidelobes) and a single tunable parameter (the critical period T): $$ y\_t \\;=\\; (1 - \\alpha/2)(x\_t - x\_{t-1}) \\;+\\; (1 - \\alpha)\\, y\_{t-1} $$ The frequency response rolls off at 6 dB/octave below the critical period. Above the critical period, the output is approximately the original input minus a slow drift. Below the critical period, content is attenuated proportionally to how far below T the frequency sits. For T = 30: cycles longer than 60 bars are attenuated by more than 6 dB (less than half amplitude). Cycles between 15 and 60 bars pass at modest attenuation. Cycles shorter than 15 bars pass at full amplitude. The 1-pole HPF is the structural detrender. The output is the input with frequencies below T removed. The lag of the 1-pole HPF at critical period T is approximately T/π bars (same as the 1-pole LPF), with the sign reversed in some interpretations: the HPF responds quickly to fast changes and slowly to slow changes, so the effective lag is small on the high-frequency content that the filter is meant to pass. ## Level 4: the 2-pole HPF Each additional pole adds 6 dB/octave of low-frequency rolloff and one bar of lag. The 2-pole HPF is the mirror of the 2-pole LPF from the prior article: $$ H\_{\\text{2-pole HPF}}(z) \\;=\\; \\frac{(1 - \\alpha/2)^2 (1 - z^{-1})^2}{1 - 2(1-\\alpha) z^{-1} + (1-\\alpha)^2 z^{-2}} $$ In time domain: $$ y\_t \\;=\\; (1 - \\alpha/2)^2 (x\_t - 2 x\_{t-1} + x\_{t-2}) \\;+\\; 2(1-\\alpha)\\, y\_{t-1} \\;-\\; (1-\\alpha)^2 \\, y\_{t-2} $$ Two state variables, four multiplications per bar. The rolloff is 12 dB/octave below T. The 2-pole HPF is the canonical detrender for daily-bar trading features because the steep low-frequency cutoff removes the slow drift cleanly while leaving the high-frequency content (where the signal lives) at full amplitude. ## Worked example: SPX detrended four ways SPX daily, 1990 to 2026\. Compute the detrended close using four HPF constructions, all matched approximately to critical period T = 30 (target: remove cycles longer than 60 bars). $$ \\begin{array}{l|c|c|c|c} \\text{HPF construction} & \\text{Poles} & \\text{Lag (bars)} & \\text{Residual low-freq content} & I(X;Y) \\times 10^3 \\\\ \\hline \\text{Differencer (1-bar diff)} & 1 (no T) & 0.5 & 0.05 & 0.8 \\\\ \\text{30-bar momentum (} P\_t - P\_{t-30}\\text{)} & 1 (no T) & 14.5 & 0.12 & 1.7 \\\\ \\text{1-pole HPF, T = 30} & 1 & 9.5 & 0.18 & 1.8 \\\\ \\text{2-pole HPF, T = 30} & 2 & 19.0 & 0.04 & 2.1 \\\\ \\end{array} $$ Four readings. The differencer has near-zero lag but no parameter to tune the cutoff. The output is dominated by single-bar noise. The MI (0.8) is the lowest because the model cannot distinguish noise from short-cycle signal. The 30-bar momentum is the standard technical-analysis HPF. The MI is 1.7, comparable to the 1-pole HPF, because the sinc-shape sidelobes do not affect the daily-frequency signal much. The lag is the (N−1)/2 = 14.5 bars of an SMA-like FIR. The 1-pole HPF at T = 30 has slightly higher MI (1.8) and less lag (9.5) than the momentum. The smooth frequency response (no sidelobes) protects against contamination at specific cycle frequencies. The 2-pole HPF at T = 30 has the highest MI (2.1) because the steep 12 dB/octave rolloff removes more of the slow trend that was acting as noise to the mean-reversion signal. The lag (19 bars) is the cost. The choice between the four depends on the application. For short-horizon mean reversion, the 2-pole HPF is the cleanest detrender. For audit-required or simple deployments, the 30-bar momentum is the operational default. For real-time low-latency use, the differencer or the 1-pole HPF is the right choice. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Application 1: detrended features The standard pipeline for a mean-reversion feature uses an HPF as the first stage: 1. Apply 2-pole HPF at critical period T (T = 20 to 60 bars for daily data). 2. Use the HPF output directly as a feature, or as input to a further transform. The feature library entries follow the convention "hpf\_close\_2pole\_T30" for the 2-pole HPF at T = 30 applied to log close. The feature is stationary by construction (HPFs attenuate DC and slow drift) and lives in the same units as the original return-space (when applied to log prices). Compared to the close-minus-MA construction commonly used in technical analysis, the HPF has three advantages: The frequency cutoff is explicit (T), not implicit in the MA length. The trader knows which frequency content is being preserved and which is being discarded. The construction is recursive, with O(1) state for the 1-pole HPF and O(2) state for the 2-pole HPF. The MA-based detrender requires O(N) state. The HPF rolls off smoothly. The MA-based detrender inherits the SMA's sidelobe leakage at specific frequencies, contaminating the detrended output with cycle content the trader did not intend to pass. ## Application 2: the roofing filter The roofing filter is the canonical preprocessing block for cycle-mode indicators. The construction: 1. Apply a 1-pole HPF with critical period T\_high (typically 40 to 80 bars) to remove the slow trend. 2. Apply a super-smoother (2-pole LPF, from the article ["The Trader's Guide to Low-Pass Filters"](https://aligrithm.com/the-traders-guide-to-low-pass-filters/)) with critical period T\_low (typically 10 to 20 bars) to remove the high-frequency noise. The composition is a band-pass filter that retains content with periods between T\_low and T\_high. The output is a clean cycle signal that downstream cycle-detection logic (dominant cycle estimation, Hilbert transform, sine-wave indicator) can operate on without being confused by trend or noise. The article ["Band-Pass Filters: The Most Underused Tool in Technical Analysis"](https://aligrithm.com/band-pass-filters-the-most-underused-tool-in-technical-analysis/) gives the direct band-pass construction that achieves the same effect in a single recursion with fewer state variables. ## Application 3: detrend-to-prediction substrate A model that consumes raw close as input wastes capacity learning the slow trend. The trend on equity indices is structurally upward (\~7% annualized real), positive almost everywhere, and easy to estimate; the model learns it in a few epochs and then spends the rest of its capacity on the residual. The "wasted capacity" is not free: the model parameters used to learn the trend are not available for learning the mean-reversion structure that actually predicts forward returns. A model that consumes HPF(close) as input starts with the trend removed. Every parameter is available for learning the residual structure. The retained MI lift on properly detrended features is typically 0.2 to 0.5 bits × 10⁻³, which compounds across the feature portfolio. The HPF is to features what the log return is to prices: a structural transform that puts the data on the axis where the model can find the signal. ## Decision matrix | Use case | Recommended HPF | Reason | | ---------------------------------- | -------------------------------- | -------------------------------- | | Standard mean-reversion feature | 2-pole HPF | Clean detrend, steep cutoff | | Real-time streaming | 1-pole HPF | O(1) state, smooth response | | Tick-frequency derivative | Differencer | Simplest, near-zero lag | | Audit-required computation | N-bar momentum | Hand-verifiable, system-portable | | Roofing-filter input | 1-pole HPF + super-smoother | Composes into band-pass | | Decycler construction | 1-pole HPF (subtract from input) | Yields clean trend line | | Cycle-mode indicator preprocessing | Roofing filter | Band-passes the cycle band | ## What this changes in practice Three operational shifts. The default detrending operation in the feature library is the 2-pole HPF at a documented critical period T, not the close-minus-MA construction. The feature name carries T explicitly: "hpf\_2pole\_close\_T30" identifies the construction. Every price-derived feature is offered with both a non-detrended and a detrended variant in the library. The detrended variant uses the 2-pole HPF as the structural transform. The non-detrended variant is kept only for features where the absolute level carries the signal (rare on equity indices). The "what frequency band does this feature live in" question is answered at feature definition. A momentum feature has a band-pass character; a trend feature has a low-pass character; a mean-reversion feature has a high-pass character. The feature library tags each feature with its dominant frequency band, which is the answer to the article "No Filter Is Predictive" question about which frequency content carries the signal. ## Visualizing the HPF response ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/05/article_44-visual-0.png) The two panels make the HPF case visually. Panel (a) shows the frequency-response trade. Panel (b) shows the time-domain output traders should actually feed into a model. ## KEY POINTS - The HPF is the structural detrender. Output = input minus low-pass. For daily-frequency mean reversion (the only reliably predictive signal on equity indices), the HPF isolates the part of the input that carries the signal. - The differencer (1-bar diff) is the simplest HPF: no parameter, full cutoff at DC, passes everything above. Underlies all return and momentum features. - The N-bar momentum (close minus close shifted by N) is a sinc-shaped HPF with notches at integer multiples of 1/N. Inherits the SMA's sidelobe leakage; the recursive 1-pole HPF avoids it. - The 1-pole HPF: y\_t = (1 − α/2)(x\_t − x\_{t−1}) + (1 − α) y\_{t−1}, with α derived from the critical period T via the cosine formula. 6 dB/octave rolloff below T, no sidelobes, O(1) state. - The 2-pole HPF: 12 dB/octave rolloff, two state variables, four multiplications per bar. The default 2-pole detrender for daily-bar trading features. - Each additional pole adds 6 dB/octave of low-frequency rolloff and one bar of lag. Same trade as the LPF family. - The HPF lag at critical period T is approximately T/π for the 1-pole and 2T/π for the 2-pole, applied to the signal that the filter is meant to pass. - The HPF can be used as a feature on its own (detrended series) or as the first stage of a roofing filter (HPF then super-smoother, equivalent to a band-pass). - On SPX at T = 30, the differencer has MI 0.8, the 30-bar momentum has 1.7, the 1-pole HPF has 1.8, and the 2-pole HPF has 2.1\. The structural detrending lifts the MI by 0.3 to 1.3 bits × 10⁻³ depending on the filter. - A model that consumes raw close wastes capacity learning the trend. A model that consumes HPF(close) starts with the trend removed and uses all capacity on the residual. - The default detrending operation in the feature library is the 2-pole HPF at a documented T. Close-minus-MA constructions are migrated to the HPF equivalent at the same critical period. - Every price-derived feature carries a frequency-band tag in the library: low-pass (trend), high-pass (deviation), band-pass (cycle). The tag is the answer to the "which frequency content carries the signal" question. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Statistically Sound Indicators for Financial Market Prediction - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Statistically+Sound+Indicators+for+Financial+Market+Prediction+Timothy+Masters&ref=aligrithm.com) - [Cycle Analytics for Traders - John Ehlers (Amazon)](https://www.amazon.com/s?k=Cycle+Analytics+for+Traders+John+Ehlers&ref=aligrithm.com) ### 2.21 The Trader's Guide to Low-Pass Filters URL: https://aligrithm.com/the-traders-guide-to-low-pass-filters/ Last updated: 2026-07-27T13:23:46.000Z The EMA from the prior article in this series (["EMA vs SMA: Why Simplicity Still Matters"](https://aligrithm.com/ema-vs-sma-why-simplicity-still-matters/)) is a single-pole low-pass filter. The transfer function has one pole, which translates to a 6 dB/octave asymptotic rolloff. A high-frequency component sitting one octave above the EMA's critical period is attenuated by 6 dB (50% amplitude). Two octaves above, by 12 dB (25%). At realistic frequencies for noise rejection on daily bars, the EMA passes through more high-frequency content than most traders assume. The fix is not to "lengthen the EMA." A longer EMA shifts the critical period without changing the rolloff rate. The asymptotic falloff is still 6 dB/octave; the noise floor at any frequency above the critical period is still half-power one octave up. The trader who wants a low-pass with steeper rolloff needs more poles. This article gives the design recipe for the 2-pole low-pass family that does most of the smoothing work in this pillar. Two pole counts (1-pole EMA, 2-pole low-pass), one critically-damped variant (the super-smoother), one design formula (a cosine-based coefficient generator), and one set of operational rules for choosing between them. The next article in this series (["High-Pass Filters for Traders"](https://aligrithm.com/high-pass-filters-for-traders/)) builds the mirror construction. Together, the LPF and HPF families compose into the band-pass filter from ["Band-Pass Filters: The Most Underused Tool in Technical Analysis"](https://aligrithm.com/band-pass-filters-the-most-underused-tool-in-technical-analysis/) and the decycler from ["Decyclers: Extracting Trend by Removing Cycle Energy."](https://aligrithm.com/decyclers-extracting-trend-by-removing-cycle-energy/) ## Pole count and rolloff Each pole in a low-pass filter contributes 6 dB/octave to the asymptotic rolloff at frequencies well above the critical period. $$ \\text{rolloff at high frequencies} \\;=\\; -6 \\cdot P \\text{ dB/octave}, \\quad P = \\text{number of poles} $$ A 1-pole filter (the EMA) attenuates at 6 dB/octave. A 2-pole filter at 12 dB/octave. A 4-pole filter at 24 dB/octave. The trade is one paid per pole. The cost of adding poles is twofold. Each pole adds one state variable (one bar of historical output that must be carried forward), and each pole adds approximately the same lag as the underlying EMA at the same critical period. A 2-pole low-pass with critical period T has approximately double the lag of a 1-pole EMA at the same T. The choice of pole count is governed by the operational question: how steep does the cutoff need to be, and how much lag is the strategy willing to pay for it? For daily-frequency trading where the critical period sits between 10 and 100 bars, 2-pole filters are the standard choice. Higher pole counts (4-pole, 6-pole) exist for cycle-detection pipelines where the cleanliness of the cutoff matters more than the lag cost. ## The cosine-based design formula The 1-pole EMA's α was parameterized two ways in the prior article (by length, by lag). For the 2-pole filter, neither convention is enough; the user-facing parameter is the critical period T (the period at which the filter attenuates by half power, equivalent to −3 dB). The cosine-based design formula derives the filter coefficients from T: $$ \\gamma \\;=\\; \\cos\\!\\Bigl(\\frac{2\\pi}{T}\\Bigr), \\qquad \\sigma \\;=\\; \\frac{1}{\\gamma} - \\sqrt{\\frac{1}{\\gamma^2} - 1} $$ The α parameter for a 1-pole EMA matched to critical period T: $$ \\alpha\_{\\text{EMA}} \\;=\\; 1 - \\sigma $$ The coefficients for the 2-pole low-pass at the same T: $$ H\_{\\text{2-pole LPF}}(z) \\;=\\; \\frac{\\alpha^2}{1 - 2(1-\\alpha) z^{-1} + (1-\\alpha)^2 z^{-2}}, \\qquad \\alpha = 1 - \\sigma $$ The recursion in time domain: $$ y\_t \\;=\\; \\alpha^2 \\, x\_t \\;+\\; 2(1-\\alpha)\\, y\_{t-1} \\;-\\; (1-\\alpha)^2 \\, y\_{t-2} $$ Two state variables (y\_{t-1} and y\_{t-2}). One parameter T (which derives α through the formula). Three multiplications per bar. The construction is the cascade of two identical 1-pole EMAs with the same α, which is why the rolloff is 12 dB/octave (two single-pole rolloffs summed in log scale). ## The super-smoother The 2-pole low-pass above is a cascade of two identical EMAs. The cascade has a particular property: at the critical period T, the filter overshoots slightly because the two poles sit at the same frequency. The overshoot shows up as a small "ringing" in the time-domain response to a step input. The super-smoother modifies the coefficients so the two poles sit at conjugate locations in the z-plane, producing a critically damped response (no overshoot, monotonic recovery from a step). The super-smoother is the 2-pole Butterworth-style filter customized for trading data: $$ a\_1 \\;=\\; e^{-\\sqrt{2}\\pi / T}, \\qquad b\_1 \\;=\\; 2 a\_1 \\cos\\!\\Bigl(\\frac{\\sqrt{2}\\pi}{T}\\Bigr), \\qquad b\_2 \\;=\\; a\_1^2 $$ The recursion: $$ y\_t \\;=\\; c\_1 (x\_t + x\_{t-1}) / 2 \\;+\\; c\_2 \\, y\_{t-1} \\;+\\; c\_3 \\, y\_{t-2}, \\qquad c\_1 = 1 - c\_2 - c\_3, \\;\\; c\_2 = b\_1, \\;\\; c\_3 = -b\_2 $$ Two state variables for the output (y\_{t-1}, y\_{t-2}) plus one for the input (x\_{t-1}). Three multiplications per bar. The frequency response has the same 12 dB/octave rolloff as the cascade-of-two-EMAs but with no overshoot in the time-domain step response. The super-smoother is the default low-pass for any application where the time-domain response matters as much as the frequency-domain response. Strategies that depend on the filter's response to a sudden price move (breakout detection, regime-change identification) benefit from the critically-damped behavior because the filter does not ring at the critical period after a step. ## Lag comparison The 1-pole EMA at critical period T has lag approximately T/π bars (this is a slightly different formulation than the by-length and by-lag conventions; T-based lag matches the critical-period parameter). The 2-pole low-pass at the same T has lag approximately 2T/π bars (two single-pole lags in series). The super-smoother at the same T has lag approximately 2T/π bars (same as the cascade, because the lag is governed by pole count and critical period, not by overshoot behavior). For T = 20 bars: EMA lag ≈ 6.4 bars, 2-pole LPF lag ≈ 12.7 bars, super-smoother lag ≈ 12.7 bars. The 2× lag cost of moving from 1-pole to 2-pole is the price of the 12 dB/octave rolloff. The choice between cascade-of-two-EMAs and super-smoother does not affect lag; it affects only the step-response shape. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Worked example: SPX low-pass smoothing at T = 20 SPX daily, 1990 to 2026\. Compute three low-pass-smoothed versions of close, all at critical period T = 20. $$ \\begin{array}{l|c|c|c|c} \\text{Filter} & \\text{Poles} & \\text{Lag (bars)} & \\text{Residual HF variance} & I(X;Y)\\times 10^3 \\\\ \\hline \\text{1-pole EMA} & 1 & 6.4 & 0.28 & 1.9 \\\\ \\text{2-pole LPF (cascade EMA)} & 2 & 12.7 & 0.11 & 2.1 \\\\ \\text{Super-smoother (2-pole Butterworth)} & 2 & 12.7 & 0.10 & 2.2 \\\\ \\end{array} $$ Three readings. The 1-pole EMA has the least lag and the most residual high-frequency content. The 6 dB/octave rolloff lets through twice as much noise as the 2-pole filters across the spectrum. The MI on the close-minus-EMA feature is the lowest of the three. The 2-pole cascade has half the residual HF variance of the 1-pole EMA. The lag doubled, the noise rejection improved by 6 dB across the spectrum. The MI lifts to 2.1. The super-smoother matches the 2-pole cascade on lag and residual variance, with a slight MI advantage from cleaner step response. For a strategy that consumes the filter output through a step-detection logic, the super-smoother is the better choice; the difference is small but consistent across the sample. ## The decision matrix | Use case | Recommended LPF | Reason | | --------------------------------- | ---------------------------- | ------------------------------------- | | General mean-reversion feature | 2-pole LPF or super-smoother | Steep rolloff, acceptable lag | | Real-time tick processing | 1-pole EMA | O(1) state, lowest lag | | Cycle detection / dominant period | Super-smoother | Critically damped, no ringing | | Breakout / step-response systems | Super-smoother | Clean step response | | Audit-required computation | 1-pole EMA | Simplest reproducible construction | | Feature library building block | 1-pole EMA | Composes into HPF, BPF, decycler | | High-precision noise rejection | 4-pole or 6-pole | Lag cost is acceptable for cycle work | The matrix is read by use case. For most feature-engineering work on daily bars, the 2-pole low-pass or super-smoother is the right choice. The EMA is reserved for real-time and composable contexts. ## Anti-patterns Three patterns that produce a worse filter than expected. Lowering T below the data's natural lookback creates a filter that the data cannot drive. A super-smoother with T = 3 on daily SPX is trying to attenuate cycles shorter than 3 days, which is below the data's resolvable frequency content. The output is dominated by the filter's own ringing and the initial transient. T should be at least 5 bars on daily data. Stacking a super-smoother on top of a 1-pole EMA buys nothing. The cascade is a 3-pole filter with three single-pole lags. The rolloff is 18 dB/octave but at three times the lag of the 1-pole. If 18 dB/octave is what the application needs, design a single 3-pole filter, not a cascade. Re-tuning T inside the backtest loop introduces look-ahead. T is a structural design parameter and is fixed at the feature definition stage. Optimizing T against the backtest is the same overfitting problem the article ["How to Test Indicator Thresholds Without Fooling Yourself"](https://aligrithm.com/how-to-test-indicator-thresholds-without-fooling-yourself/) covered for indicator thresholds. Permutation-test any T choice that emerged from a search. ## What this changes in practice Three operational shifts. The default low-pass smoother in the feature library is the 2-pole filter at a documented critical period T, not the EMA. The EMA is used as a building block or in real-time contexts. Feature names carry T explicitly: "lpf\_close\_T\_20" identifies the construction. Critical period T is the user-facing parameter, not α or length. The cosine-based design formula derives α from T. Feature library entries store T as the canonical parameter and α as the derived quantity. Two features with the same T and different filter types (1-pole, 2-pole, super-smoother) are different features and are logged separately. The super-smoother is the default 2-pole choice for any feature whose output is consumed by step-detection or regime-change logic. The 2-pole cascade is the default for general smoothing where the time-domain step response is not critical. The distinction is recorded in the feature definition. ## Visualizing the rolloff ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/05/article_43-visual-0.png) The figure shows that all three filters meet at the critical period and diverge above it. The choice of filter is the choice of how aggressively to attenuate the content above T. ## KEY POINTS - Each pole in a low-pass filter contributes 6 dB/octave of asymptotic rolloff. 1-pole EMA: 6 dB/oct. 2-pole LPF: 12 dB/oct. 4-pole: 24 dB/oct. - Each pole adds approximately one bar of lag at the critical period. 2-pole LPF lag is roughly double the 1-pole EMA's lag at the same T. - The user-facing parameter is the critical period T (the period at which the filter attenuates by −3 dB). The α and the higher-order coefficients are derived from T via the cosine-based design formula. - 2-pole low-pass: y\_t = α² x\_t + 2(1−α) y\_{t−1} − (1−α)² y\_{t−2}, with α = 1 − σ derived from T. Two state variables, three multiplications per bar. - The super-smoother is a 2-pole filter with critically damped coefficients. Same 12 dB/octave rolloff as the cascade-of-two-EMAs but no overshoot in the time-domain step response. - For daily-frequency trading where T sits between 10 and 100 bars, 2-pole filters are the default. The 1-pole EMA is for real-time, composable, or audit-required contexts. - The super-smoother is the default 2-pole choice for features consumed by step-detection or regime-change logic. The 2-pole cascade is the default for general smoothing. - On SPX at T = 20, the 1-pole EMA has residual HF variance 0.28 and lag 6.4\. The 2-pole LPF has 0.11 and 12.7\. The super-smoother has 0.10 and 12.7 with slightly higher retained MI. - Lowering T below 5 bars on daily data produces a filter dominated by its own ringing. T must be larger than the data's resolvable timescale. - Stacking super-smoother on EMA produces a 3-pole filter at 3× the EMA lag. Design a 3-pole filter directly if 18 dB/octave is required. - T is a structural parameter fixed at feature definition. Re-tuning T inside the backtest loop is look-ahead and requires a permutation test to validate. - The 1-pole EMA composes into HPF (subtract from input), BPF (cascade with HPF), decycler (input minus HPF). The 2-pole filters do not compose as cleanly and are used as endpoints, not building blocks. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Statistically Sound Indicators for Financial Market Prediction - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Statistically+Sound+Indicators+for+Financial+Market+Prediction+Timothy+Masters&ref=aligrithm.com) - [Cycle Analytics for Traders - John Ehlers (Amazon)](https://www.amazon.com/s?k=Cycle+Analytics+for+Traders+John+Ehlers&ref=aligrithm.com) ### 2.20 EMA vs SMA: Why Simplicity Still Matters URL: https://aligrithm.com/ema-vs-sma-why-simplicity-still-matters/ Last updated: 2026-07-27T13:23:45.000Z The prior article in this series (["Why the SMA Is Often a Terrible Smoother"](https://aligrithm.com/why-the-sma-is-often-a-terrible-smoother/)) tore down the SMA on frequency-response grounds. First sidelobe at −13.3 dB. Critical period 2N instead of N. Notch at the exact period the trader picked. The article ended by pointing at the Hann and Blackman windows as the right replacements: same lag, dramatically better stopband. A reader who took only that argument out of the article would conclude that no recursive smoother (EMA, the family this article covers) should ever be used, since both SMA and EMA are inferior to window functions on the frequency-response axis. The conclusion is wrong. The EMA survives in production despite worse residual high-frequency content than Hann or Blackman because three properties (O(1) state, recursive form, monotonic frequency response) make it the right default for streaming applications, real-time systems, intraday data, and any context where the EMA is also a building block for the higher-order recursive filters covered in the next four articles in this series. The complex window-based smoothers (Kaiser, Dolph-Chebyshev) buy frequency-response improvement at the cost of construction complexity, parameter count, and audit-trail difficulty. The EMA's combination of single-parameter tuning, single-state memory, and recursive composability makes it the simplest filter that is also the workhorse of every higher-order recursive construction this pillar uses. The simplicity is not nostalgia; it is the constraint that makes the EMA the only filter that works in every deployment context. The next article in this series (["The Trader's Guide to Low-Pass Filters"](https://aligrithm.com/the-traders-guide-to-low-pass-filters/)) covers the proper low-pass family (2-pole, super-smoother) where the EMA is the single-pole reference case. This article is the case for the single-pole filter on its own merits. ## The construction The EMA's defining recursion: $$ y\_t \\;=\\; \\alpha \\, x\_t \\;+\\; (1 - \\alpha)\\, y\_{t-1}, \\qquad 0 < \\alpha \\le 1 $$ One state variable y\_{t-1}. One parameter α. One multiplication and one addition per bar. The recursion is the simplest possible recursive low-pass filter. The transfer function follows from the recursion. Taking the z-transform: $$ H\_{\\text{EMA}}(z) \\;=\\; \\frac{Y(z)}{X(z)} \\;=\\; \\frac{\\alpha}{1 - (1 - \\alpha) z^{-1}} $$ The denominator has a single pole at z = 1 − α. The numerator has no zeros. The EMA is a single-pole all-pole filter, which is the defining structure of the simplest IIR (infinite impulse response) low-pass filter. Two tuning conventions are in common use: By length: α = 2 / (N + 1). N is the "equivalent SMA length" that matches the EMA's center of mass. EMA with N = 20 has α ≈ 0.0952 and behaves comparably to an SMA(20) in terms of smoothing time scale. By lag: α = 1 / (L + 1). L is the target lag in bars. EMA with L = 10 has α ≈ 0.0909 and produces an output that lags the input by approximately 10 bars on a slowly varying signal. The two conventions disagree by a factor of \~2\. The "by length" convention is the one most platforms use because it makes the EMA's name match the equivalent SMA's name. The "by lag" convention is the one a filter designer would pick because it directly specifies the property that matters. ## Three properties EMA has that window-function smoothers do not The case for EMA over Hann or Blackman rests on three properties that have nothing to do with frequency response. Property 1: O(1) state. The EMA at bar t depends only on x\_t and y\_{t-1}. The full history is compressed into one number, regardless of N. An SMA(50) requires storing the last 50 values; a Hann(50)-windowed MA requires the same. The EMA requires one. For streaming applications (real-time market data feeds), intraday systems (where every tick must be processed in microseconds), embedded systems (FPGA-based execution venues), and any context with strict memory or latency budgets, the O(1) state is decisive. The window-function smoothers cannot be deployed in these contexts without an explicit buffer that the EMA does not need. Property 2: recursive form composes naturally. The EMA recursion can be inverted, combined, and stacked to produce filters that the window-function family cannot easily express. Subtracting an EMA from the input produces a high-pass filter (the article ["High-Pass Filters for Traders"](https://aligrithm.com/high-pass-filters-for-traders/), forthcoming). Cascading two EMAs produces a 2-pole low-pass with steeper rolloff (the article ["The Trader's Guide to Low-Pass Filters"](https://aligrithm.com/the-traders-guide-to-low-pass-filters/), forthcoming). Cascading an EMA-based low-pass with an EMA-based high-pass produces a band-pass (the article ["Band-Pass Filters: The Most Underused Tool in Technical Analysis"](https://aligrithm.com/band-pass-filters-the-most-underused-tool-in-technical-analysis/), forthcoming). The decycler (the article ["Decyclers: Extracting Trend by Removing Cycle Energy"](https://aligrithm.com/decyclers-extracting-trend-by-removing-cycle-energy/), forthcoming) is constructed from an EMA-based HPF subtracted from the input. All of these constructions inherit the EMA's O(1) state and tunable-α structure. The window-function family does not generalize this way. A Hann-window-based HPF requires a separate window design (and is rarely done in practice); a Hann-windowed BPF requires a more complex cascade with substantial state. The EMA is the building block of a recursive filter library that scales to band-pass, AGC (the article ["Automatic Gain Control for Trading Indicators"](https://aligrithm.com/automatic-gain-control-for-trading-indicators/), forthcoming), and adaptive filters; the window functions are point solutions. Property 3: monotonically decreasing weights → no frequency-response sidelobes. The EMA's coefficients on the lagged input values are α, α(1−α), α(1−α)², α(1−α)³, .... Every coefficient is non-negative and the sequence decays monotonically to zero. The Fourier transform of a monotonically decreasing positive sequence is a monotonically decreasing function of frequency (no sidelobes). The SMA's frequency response has sidelobes that pass noise at specific frequencies at 22% amplitude (the −13.3 dB first sidelobe). The EMA's frequency response has no sidelobes. The asymptotic falloff is slower (6 dB/octave for the EMA vs 18 dB/octave for Hann), so the EMA leaks more total high-frequency noise than the Hann across the full spectrum. But the leakage is spread evenly across all high frequencies, with no surprise peaks at specific cycles. For a model that may pick up sidelobe-frequency noise as spurious signal, the EMA's monotonic leakage is operationally safer than the SMA's sidelobe leakage. ## Three properties EMA shares with SMA (and Hann does not) The EMA inherits some of the simplicity that made the SMA survive in production for forty years. Single parameter. α is one number. Library implementations across Python, R, MATLAB, Excel, Bloomberg, and TradingView agree on the EMA's definition. Window-function smoothers require specifying the window family (Hann vs Hamming vs Blackman) and the coefficient convention (some libraries normalize the weight sum, some do not). The EMA has no convention disputes. Audit-trail clarity. Reproducing an EMA output requires only α, the initial state y\_0 (usually set to x\_0), and the input sequence. A regulator or post-trade analyst can verify an EMA by hand for any reasonable N. Hann-windowed outputs require knowing the exact window coefficients, which adds a step that the audit infrastructure must support. Cross-system reproducibility. EMA outputs computed in different systems agree to the last decimal as long as the initialization convention agrees (which is documented in any reasonable feature library). Window-function outputs agree only after the implementations agree on the coefficient convention, and quiet disagreements at the 1% to 3% level appear between common libraries. ## The honest cost: worse stopband than window functions The EMA pays for its simplicity in frequency-response quality. At any given lag, the Hann or Blackman window has dramatically better high-frequency rejection. $$ \\begin{array}{l|c|c|c|c} \\text{Filter (N or equiv.)} & \\text{Lag} & \\text{Sidelobe pattern} & \\text{Asymptotic falloff} & \\text{Residual HF variance} \\\\ \\hline \\text{SMA(50)} & 24.5 & -13.3\\text{ dB peaks} & 6 \\text{ dB/oct} & 0.41 \\\\ \\text{EMA (eq. N = 50)} & 24.5 & \\text{none (monotonic)} & 6 \\text{ dB/oct} & 0.32 \\\\ \\text{Hann(50)} & 24.5 & -31.5\\text{ dB peaks} & 18 \\text{ dB/oct} & 0.18 \\\\ \\text{Blackman(50)} & 24.5 & -58.1\\text{ dB peaks} & 18 \\text{ dB/oct} & 0.09 \\\\ \\end{array} $$ Four readings. The EMA at equivalent N = 50 has residual high-frequency variance 0.32, smaller than the SMA's 0.41 because the EMA's monotonic frequency response avoids the SMA's sidelobe leakage. The EMA is a better smoother than the SMA at the same effective length, on the residual-noise axis as well as on the state-cost axis. The EMA's 0.32 is larger than the Hann's 0.18 and the Blackman's 0.09\. The window functions have steeper rolloff (18 dB/octave vs 6 dB/octave) and dramatically lower sidelobes. For a researcher running offline backtests on daily data with no state constraints, the Hann or Blackman is the better choice on the noise-rejection axis. For a researcher running offline backtests where O(1) state matters or where the EMA is the building block for other filters in the strategy, the EMA's structural advantages outweigh the noise-rejection gap. The 0.14-variance difference between EMA and Hann is real but small relative to the daily-bar noise floor. For real-time, intraday, embedded, or composable applications, the EMA is the only choice in the comparison. The window functions are not deployable in those contexts at the required state and computation budgets. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## The decision matrix | Context | Best smoother | Reason | | --------------------------------------- | ---------------- | ------------------------------------------------- | | Offline daily backtest, no state limit | Hann or Blackman | Best frequency response at same lag | | Real-time streaming / intraday | EMA | O(1) state, O(1) computation per bar | | Building block for HPF / BPF / AGC | EMA | Recursive form composes; window functions do not | | Audit-required, regulatory | SMA | Hand-verifiable, system-portable | | Tick-frequency feature pipeline | EMA | Latency and memory budget | | Long-history feature with sidelobe risk | Hann or EMA | Monotonic response, no specific-frequency leakage | | Strategy needing adaptive smoothing | EMA-based | Recursive form allows α to be made time-varying | The matrix is read by context, not by frequency-response preference. The right smoother depends on what the smoother has to do downstream, not on which one wins on a single signal-processing axis. ## Worked example: SPX EMA at multiple α values SPX daily, 1990 to 2026\. Compute the close-minus-EMA feature at four α values (parameterized by equivalent length N). $$ \\begin{array}{l|c|c|c|c} \\text{Construction} & \\alpha & \\text{Lag} & I(X;Y) \\times 10^3 & \\text{Notes} \\\\ \\hline \\text{close} - \\text{EMA(eq. N = 10)} & 0.182 & 4.5 & 2.2 & \\text{captures short-term mean reversion} \\\\ \\text{close} - \\text{EMA(eq. N = 20)} & 0.095 & 9.5 & 1.9 & \\text{standard mean-reversion feature} \\\\ \\text{close} - \\text{EMA(eq. N = 50)} & 0.039 & 24.5 & 1.6 & \\text{trend deviation, slower} \\\\ \\text{close} - \\text{EMA(eq. N = 200)} & 0.010 & 99.5 & 0.7 & \\text{long-trend deviation, low signal} \\\\ \\end{array} $$ Four readings. The shorter EMAs produce features with higher MI. The 10-bar EMA's close-minus-EMA captures the mean-reversion signal that lives at short timescales, which carries the most predictive content on SPX. The 200-bar EMA captures a long-trend deviation feature whose MI is at the noise floor. The feature is the difference (close − EMA), not the EMA value itself. The article ["No Filter Is Predictive: What Traders Misunderstand About Smoothing"](https://aligrithm.com/no-filter-is-predictive-what-traders-misunderstand-about-smoothing/) covered why the raw EMA value is not a feature: it is a delayed estimate of the input. The (close − EMA) construction is a usable feature because it measures the gap between the current state and the lagging smoother, and the gap is a real-time quantity. The two-way tuning (by length vs by lag) does not change the MI ranking but changes the parameter intuition. For a researcher who thinks in lag bars, α = 1/(L+1) is the natural specification. For a researcher who thinks in equivalent SMA periods, α = 2/(N+1) is natural. The feature library stores both conventions to avoid the confusion. ## What this changes in practice Three operational shifts. The default smoother in the feature library depends on the deployment context, not on a single preference. For offline daily backtests, Hann or Blackman is the default (per the article ["Why the SMA Is Often a Terrible Smoother"](https://aligrithm.com/why-the-sma-is-often-a-terrible-smoother/)). For streaming systems and any feature that feeds into a higher-order recursive filter (HPF, BPF, AGC), EMA is the default. For audit-required or regulator-facing computations, SMA is the default. EMA features carry α as metadata, with both length-equivalent and lag-equivalent values stored. "ema\_close\_alpha\_0.0952\_eqN\_20\_eqL\_19" is the verbose form; the short form is "ema\_close\_N\_20". The convention disagreement between platforms is resolved by storing α directly. The EMA is treated as a building block, not a standalone smoother. Features that use the EMA's output directly (e.g., the close-minus-EMA mean-reversion feature) are documented as derived. Features that use the EMA recursion as part of a larger filter (HPF, BPF, decycler, AGC) point back to the EMA's α as the underlying parameter. The feature library tracks the composition so a change to the underlying α propagates through every dependent feature. ## Visualizing the trade ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/05/article_42-visual-0.png) The two panels make the trade-off visible. Panel (a) shows the Hann is the best filter on the frequency axis. Panel (b) shows the EMA is the best filter on every other axis. The right choice is not panel (a) alone. ## KEY POINTS - EMA is the single-pole IIR recursion y\_t = α x\_t + (1−α) y\_{t−1}. One state variable, one parameter, one multiplication per bar. - Two tuning conventions: by length α = 2/(N+1) and by lag α = 1/(L+1). The two disagree by a factor of \~2; store both in the feature library. - The EMA has O(1) state regardless of N. The SMA and any window-function smoother require O(N) state. For streaming, intraday, embedded, and real-time applications, the O(1) state is decisive. - The EMA's recursive form composes naturally into HPF (subtract EMA from input), 2-pole LPF (cascade two EMAs), BPF (cascade LPF and HPF), decycler (input minus HPF), and AGC (the article ["Automatic Gain Control for Trading Indicators"](https://aligrithm.com/automatic-gain-control-for-trading-indicators/), forthcoming). Window functions do not generalize this way. - The EMA's frequency response has no sidelobes (monotonically decreasing coefficients → monotonically decreasing transfer function). The SMA has −13.3 dB sidelobes; the EMA is a structurally cleaner default at the same N. - The honest cost: 6 dB/octave asymptotic falloff vs 18 dB/octave for Hann. Across the full spectrum, the EMA leaks more high-frequency noise than the Hann. At the first-sidelobe frequency, the EMA leaks less than the SMA. - For offline daily backtests with no state limit, Hann or Blackman is the right choice. For streaming, real-time, or composable applications, EMA is the right choice. The decision is by context, not by single-axis preference. - The EMA shares the SMA's three production advantages (single parameter, audit clarity, cross-system reproducibility) and adds two more (O(1) state, recursive composability). - On SPX, the close-minus-EMA feature MI ranks by lag: EMA(eq. N=10) has MI 2.2, EMA(eq. N=20) has 1.9, EMA(eq. N=50) has 1.6, EMA(eq. N=200) has 0.7\. The shorter EMAs capture more predictive content because mean reversion is a short-timescale signal. - The raw EMA value is not a feature; it is a delayed estimate of the input. The (close − EMA) construction is the usable feature, with effective lag of −group\_delay (a current-bar measurement of the gap to the lagging smoother). - EMA features carry α directly as metadata, plus the length-equivalent N and the lag-equivalent L. The convention disagreement between platforms is resolved by storing α as the canonical parameter. - The EMA is the building block for the entire recursive filter family in this pillar. The next four articles (low-pass, high-pass, band-pass, decycler) use the EMA recursion as the underlying primitive. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Statistically Sound Indicators for Financial Market Prediction - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Statistically+Sound+Indicators+for+Financial+Market+Prediction+Timothy+Masters&ref=aligrithm.com) - [Cycle Analytics for Traders - John Ehlers (Amazon)](https://www.amazon.com/s?k=Cycle+Analytics+for+Traders+John+Ehlers&ref=aligrithm.com) ### 2.19 Why the SMA Is Often a Terrible Smoother URL: https://aligrithm.com/why-the-sma-is-often-a-terrible-smoother/ Last updated: 2026-07-27T13:23:44.000Z Generate a clean signal: pure 60-day cosine wave plus 5% Gaussian noise. Smooth with SMA(50). The intuition says SMA(50) "averages out" the noise and leaves the 60-day trend; the visual appearance of the output is a smoother curve that traces the original wave. The intuition is wrong on two counts. First, the SMA(50) does not attenuate the 60-day cycle by much. The frequency response of SMA(50) at the period-60 frequency is roughly −2.4 dB (a 24% amplitude reduction). The wave that emerges is a slightly smaller version of the input wave, with no structural change to its shape. Second, the noise is not "averaged out" the way the intuition suggests. The SMA's frequency response has sidelobes that pass through high-frequency content at amplitudes that range from −13.3 dB (the first sidelobe) to silently smaller. The output retains visible noise at frequencies the trader assumed were eliminated. Worse, the noise that does survive the SMA arrives shifted in phase, so it looks correlated with the underlying wave even when the original noise was white. The visual smoothness of the SMA line is a partial fact. The SMA averages noise less effectively than almost any other window of the same length. The point is structural: the SMA is the worst common smoother in signal-processing terms, and traders use it most because it is the easiest to explain. This article catalogs four structural failures of the SMA, compares its frequency response to better alternatives (Hann, Hamming, Blackman windows and the recursive low-pass family covered in the article ["The Trader's Guide to Low-Pass Filters"](https://aligrithm.com/the-traders-guide-to-low-pass-filters/)), and gives the operational rule: do not smooth with SMA. The narrow exception is the case the next article in this series (["EMA vs SMA: Why Simplicity Still Matters"](https://aligrithm.com/ema-vs-sma-why-simplicity-still-matters/)) makes for the SMA's pedagogical and audit-trail value despite its filtering deficiencies. ## The four structural failures The SMA's transfer function in continuous frequency is: $$ H\_{\\text{SMA}}(2\\pi f) \\;=\\; \\frac{\\sin(N \\pi f)}{N \\sin(\\pi f)} $$ This is the discrete-time sinc function (the Dirichlet kernel). The shape is a tall main lobe centered at f = 0 (DC, where H = 1) followed by alternating-sign sidelobes that decay slowly. Four failure modes follow from the shape. Failure 1: rectangular window in time → sinc in frequency. The SMA puts equal weight (1/N) on every bar in the window and zero weight outside. The Fourier transform of a rectangular window is a sinc function. Sinc functions have sidelobes by mathematical necessity; no choice of N removes them. Smooth tapered windows (Hann, Hamming, Blackman) trade a slightly wider main lobe for dramatically smaller sidelobes. The SMA is the only common window that does not trade. Failure 2: first sidelobe at −13.3 dB. The first sidelobe of the SMA's frequency response sits at −13.3 dB relative to the DC pass, regardless of N. A high-frequency component at the first-sidelobe frequency passes through the SMA with 22% of its original amplitude. The "noise reduction" the trader expects is at most an order of magnitude less than the trader assumes. $$ \\begin{array}{l|c|c} \\text{Window} & \\text{First sidelobe (dB)} & \\text{Asymptotic falloff (dB/octave)} \\\\ \\hline \\text{Rectangular (SMA)} & -13.3 & 6 \\\\ \\text{Hann} & -31.5 & 18 \\\\ \\text{Hamming} & -42.7 & 6 \\\\ \\text{Blackman} & -58.1 & 18 \\\\ \\end{array} $$ The Hann window has the same length as the SMA and rejects the first sidelobe 18 dB better. The Blackman is 45 dB better. For the same N (the same lag tax from the article ["The Hidden Cost of Every Moving Average: Lag"](https://aligrithm.com/the-hidden-cost-of-every-moving-average-lag/)), the SMA leaks 22% of the first-sidelobe noise where Hann leaks 2.6% and Blackman leaks 0.12%. Failure 3: "critical period" (half-power cutoff) is approximately 2N, not N. The identity: the SMA(N) attenuates a sinusoid of period 2N by 3 dB (half power). The trader who reads "SMA(50)" as "filters out cycles shorter than 50 days" is off by a factor of two. The actual half-power cutoff for SMA(50) is 100-day cycles; everything shorter passes through at more than half power. $$ T\_{\\text{critical}}^{\\text{SMA}}(N) \\;\\approx\\; 2N \\text{ bars} $$ For SMA(50): cycles longer than 100 days pass with > 70% amplitude. Cycles of 50 to 100 days are attenuated by 3 to 12 dB. Cycles of 25 to 50 days are attenuated by 10 to 20 dB but contaminated by sidelobe content. Cycles shorter than 25 days are heavily attenuated except at the first-sidelobe-equivalent frequencies, which pass at −13.3 dB regardless of N. Failure 4: the first zero is at period N exactly, not at N/2\. The SMA(N) has a true zero (infinite dB attenuation) at the frequency 1/N, which corresponds to a period of exactly N bars. SMA(50) annihilates the 50-day cycle perfectly. The construction is a notch filter at one specific period. Traders who pick N because they "want to see cycles around N days" have selected the one cycle the SMA will erase entirely. The cycles at N−5 and N+5 pass at −1 to −2 dB. The four failures compound. An SMA(50) used to "smooth out short-term noise" passes most cycles in the 25-to-200-day range with substantial amplitude, annihilates the exact 50-day cycle, and rejects truly high-frequency noise at only the rectangular window's modest −13 dB level. The output looks smoother because some noise is removed, but the noise that survives is the worst case (correlated, phase-shifted, near the sidelobe frequencies) and the legitimate cycle content is partially erased. ## What window functions buy you Window functions trade main-lobe width for sidelobe height. The SMA has the narrowest main lobe and the worst sidelobes. The Hann, Hamming, and Blackman windows have wider main lobes (so they cut off slightly less steeply at the critical period) and dramatically better sidelobes. The Hann window of length N: $$ w\_{\\text{Hann}}(k) \\;=\\; 0.5 \\cdot \\Bigl(1 - \\cos\\!\\Bigl(\\frac{2\\pi k}{N - 1}\\Bigr)\\Bigr), \\quad k = 0, 1, \\ldots, N-1 $$ The coefficients taper from zero at the edges of the window to 1 at the center. The center bar carries the most weight; the edge bars carry near-zero weight. The Fourier transform of this tapered shape has its first sidelobe at −31.5 dB and decays at 18 dB per octave, both dramatically better than the SMA. The Hamming window is similar to Hann with a small offset in the coefficient formula (the cosine multiplier is 0.46 instead of 0.5). The first sidelobe is at −42.7 dB but the asymptotic falloff is 6 dB per octave (same as SMA). The Hamming is the right choice when the first sidelobe matters most; the Hann is the right choice when the asymptotic noise floor matters. The Blackman window has a three-cosine taper: $$ w\_{\\text{Blackman}}(k) \\;=\\; 0.42 - 0.5 \\cos\\!\\Bigl(\\frac{2\\pi k}{N-1}\\Bigr) + 0.08 \\cos\\!\\Bigl(\\frac{4\\pi k}{N-1}\\Bigr) $$ The first sidelobe sits at −58.1 dB and the asymptotic falloff is 18 dB per octave. Blackman is the right choice when noise rejection across the whole stopband matters more than steep cutoff at the critical period. All three windows have the same lag as the SMA of the same length: (N−1)/2 bars. The lag tax from the article ["The Hidden Cost of Every Moving Average: Lag"](https://aligrithm.com/the-hidden-cost-of-every-moving-average-lag/) is unchanged. The frequency-response quality improves at no lag cost. The only price is the computation, which on modern hardware is irrelevant. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Worked example: SPX smoothed by SMA(50), Hann(50), and a 2-pole low-pass SPX daily, 1990 to 2026\. Compute three smoothed versions of close: SMA(50), Hann-windowed-MA(50), and a 2-pole recursive low-pass with critical period 100 days (matched approximately to the SMA(50) half-power cutoff). Report the residual high-frequency content (variance of close minus smoothed value, restricted to periods shorter than 30 days) and the mutual information of the smoothed-value-minus-close feature against next-day return sign. $$ \\begin{array}{l|c|c|c|c} \\text{Smoother} & \\text{Lag (bars)} & \\text{Residual HF variance (norm.)} & \\text{First sidelobe (dB)} & I(X;Y) \\times 10^3 \\\\ \\hline \\text{SMA(50)} & 24.5 & 0.41 & -13.3 & 1.4 \\\\ \\text{Hann-windowed MA(50)} & 24.5 & 0.18 & -31.5 & 1.9 \\\\ \\text{Blackman-windowed MA(50)} & 24.5 & 0.09 & -58.1 & 2.0 \\\\ \\text{2-pole recursive LPF (T = 100)} & 22.0 & 0.07 & -45.0 & 2.1 \\\\ \\end{array} $$ Four readings. The SMA(50) leaves the most residual high-frequency content (variance 0.41 in normalized units). The output looks smooth on a chart but the spectral content beyond the critical period leaks through at the first-sidelobe amplitude of 22%. Models that consume the (close − SMA(50)) feature pick up the residual noise as if it were signal. The Hann-windowed MA(50) cuts the residual variance by 56% (from 0.41 to 0.18). The lag is identical (24.5 bars). The construction is one multiplication per bar more expensive than the SMA. The MI lift from 1.4 to 1.9 is half a unit of bits × 10⁻³, which compounds across a research portfolio of features. The Blackman window cuts the residual variance by 78% relative to SMA. The MI improves only marginally relative to Hann because the dominant signal-blocking is already done by the Hann. The Blackman is overkill for daily SPX; it would be the right choice for intraday data where the noise floor below −30 dB still carries enough content to interfere. The 2-pole recursive low-pass (covered constructively in the article ["The Trader's Guide to Low-Pass Filters"](https://aligrithm.com/the-traders-guide-to-low-pass-filters/)) matches or exceeds the Blackman on residual variance while running with slightly less lag. The construction is recursive (an IIR filter) instead of FIR; the trade-off and the proper parameterization belong to that article. ## Why the SMA is still in the textbooks Three reasons the SMA survives despite the structural failures, each one defensible only in narrow contexts. Pedagogical simplicity. The SMA is the easiest filter to explain. The output at bar t is the average of the last N closes. The formula fits in one line and computes in O(1) per bar. Any introduction to technical analysis starts with the SMA because the first step is to teach that smoothing exists and how it works. The cost is that most students never learn that other windows exist. Audit-trail clarity. The SMA's output is a deterministic function of N raw closes. Reproducing the calculation requires no parameter beyond N and no convention about the windowing function. For a regulator, a fund administrator, or a post-trade analyst, the SMA is the indicator that can be verified by hand. Window-function smoothers require knowing the window family and parameters, which adds reconciliation overhead. For audit-critical applications, the SMA's transparency outweighs its filtering deficiencies. Cross-system compatibility. SMA outputs from Bloomberg, Reuters, Excel, Python, and TradingView agree to the last decimal. Hann and Blackman outputs depend on the exact coefficient convention (some libraries normalize the sum of weights to 1, some to N), and 1% to 3% differences appear between implementations. For features that must be reproducible across systems, the SMA is the only safe choice. Outside these three contexts, the SMA should be replaced with a Hann, Blackman, or 2-pole recursive low-pass at the same N. The lag tax is identical, the computation cost is negligible, and the frequency response is dramatically better. ## What this changes in practice Three operational shifts. The default smoother in the feature library is the Hann-windowed MA, not the SMA. The replacement is a one-line change at the feature definition. The retained-MI lift typically runs 0.3 to 0.5 bits × 10⁻³ per feature, which compounds across the portfolio. SMA features are kept in the library only where audit, pedagogy, or cross-system reconciliation forces them. The decision is recorded with each SMA feature: "audit-required" or "deprecated, use Hann equivalent." Features without one of the documented justifications are migrated. The "critical period" of every smoothing filter is logged as metadata, computed from the filter's transfer function. For SMA(N), critical period is \~2N. For Hann(N), critical period is \~1.4N. For the 2-pole recursive LPF, critical period is the design parameter T. Strategy code that depends on "the 50-day smoother filters out 50-day cycles" is wrong for every windowing choice and is corrected by reference to the actual critical period. ## Visualizing the sidelobe problem ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/05/article_41-visual-0.png) The figure makes the sidelobe-amplitude problem visible. Same N, same lag, four windows, four completely different stopband behaviors. ## KEY POINTS - The SMA's frequency response is the discrete sinc function H(2πf) = sin(Nπf) / (N sin(πf)). The shape has alternating-sign sidelobes that pass high-frequency content at substantial amplitudes. - The first sidelobe of the SMA sits at −13.3 dB below DC regardless of N. A high-frequency noise component at the first-sidelobe frequency passes through the SMA with 22% of its original amplitude. - The Hann window has first sidelobe at −31.5 dB (2.6% leakage). The Hamming at −42.7 dB (0.7%). The Blackman at −58.1 dB (0.12%). All three have the same lag (N−1)/2 as the SMA of equal length. - The "critical period" (half-power cutoff) of an SMA(N) is approximately 2N bars, not N. SMA(50) attenuates 50-day cycles by only 3 dB; it does not "filter out" cycles shorter than 50 days as commonly assumed. - The first true zero in the SMA's response is at period exactly N. SMA(50) annihilates the 50-day cycle precisely. Cycles at N−5 and N+5 pass at −1 to −2 dB. The notch is narrower than the filter's name suggests. - The rectangular weighting (equal weight on all N bars) is the worst possible window in signal-processing terms. Tapered weights (Hann, Hamming, Blackman) trade a slightly wider main lobe for an order-of-magnitude better stopband. - Window-function smoothers are computationally negligible compared to SMA on modern hardware (one multiplication per bar more). The frequency-response improvement is free at the lag and CPU level. - On SPX 1990 to 2026, SMA(50) leaves residual high-frequency variance 0.41 (normalized). Hann(50) leaves 0.18, Blackman(50) leaves 0.09\. The retained MI on the (close − smoother) feature lifts from 1.4 (SMA) to 1.9 (Hann) to 2.0 (Blackman) bits × 10⁻³. - The SMA survives in three narrow contexts: pedagogical simplicity (easiest to explain), audit-trail clarity (deterministic, system-portable), and cross-system reconciliation (no convention disputes). Outside these, the Hann or Blackman is the right replacement at the same lag. - The default smoother in the feature library is the Hann-windowed MA. SMA features are documented as "audit-required" or migrated to the Hann equivalent. - The "critical period" of every smoothing filter is logged with the feature definition. SMA(N) critical period ≈ 2N; Hann(N) critical period ≈ 1.4N; 2-pole recursive LPF critical period is the design parameter T. Strategy code that assumes "filter N → blocks cycles shorter than N" is wrong for every windowing choice. - The next article in this series argues for the EMA as a recursive alternative with different trade-offs from FIR window functions. The EMA is not better than Hann or Blackman in frequency response, but its O(1) state and explicit lag parameter make it the right choice in specific deployment contexts. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Statistically Sound Indicators for Financial Market Prediction - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Statistically+Sound+Indicators+for+Financial+Market+Prediction+Timothy+Masters&ref=aligrithm.com) - [Cycle Analytics for Traders - John Ehlers (Amazon)](https://www.amazon.com/s?k=Cycle+Analytics+for+Traders+John+Ehlers&ref=aligrithm.com) ### 2.18 The Hidden Cost of Every Moving Average: Lag URL: https://aligrithm.com/the-hidden-cost-of-every-moving-average-lag/ Last updated: 2026-07-27T13:23:44.000Z SPX bottoms on March 23, 2020 at 2,237\. The trend has changed. By April 7, the index is back above 2,650, a 19% rally in eleven trading days. The 50-day SMA does not cross above the 200-day SMA until June 1, when SPX trades at 3,055\. The "golden cross" buy signal fires 50 trading days late and 37% of the rally above the low. The lateness is not a coincidence of one regime change. It is the structural lag of a 50-day filter: group delay (50−1)/2 = 24.5 bars. The 200-day filter contributes another 99.5 bars of lag. The crossover detection is dominated by the slower filter and arrives roughly 100 bars after the underlying trend changed. The trader does not see the trend change; they see a delayed report of a trend change that already happened. The article ["No Filter Is Predictive: What Traders Misunderstand About Smoothing"](https://aligrithm.com/no-filter-is-predictive-what-traders-misunderstand-about-smoothing/) stated the structural identity: filter output at time t is a function of past inputs only. This article quantifies the cost of that identity. Every moving average pays a lag tax. The tax is paid in three currencies (money, information, confidence), the receipt is in the filter's transfer function, and there is no clever construction that escapes it within the standard linear-filter family. The next article in this series (["Why the SMA Is Often a Terrible Smoother"](https://aligrithm.com/why-the-sma-is-often-a-terrible-smoother/)) goes deeper on why the SMA in particular has poor stopband behavior on top of its lag. The narrow point of this article is the lag identity that all moving averages share. ## The lag identity For a symmetric non-recursive filter of length N (the SMA family, the Hann window, the Blackman window, and any other FIR with coefficients symmetric about the center), the group delay is constant across all frequencies: $$ \\text{lag}\_{\\text{symmetric FIR}}(N) \\;=\\; \\frac{N - 1}{2} \\text{ bars} $$ For an exponential moving average with smoothing constant α (or equivalently length N where α = 2/(N+1)), the lag is approximately: $$ \\text{lag}\_{\\text{EMA}}(\\alpha) \\;=\\; \\frac{1 - \\alpha}{\\alpha} \\;\\approx\\; \\frac{N - 1}{2} \\text{ bars} $$ For an EMA tuned by lag instead of length, the relationship inverts: a desired lag L corresponds to smoothing constant α = 1 / (L + 1). An EMA with α = 0.1 has lag 9 bars; an EMA with α = 0.05 has lag 19 bars. For the WMA (weighted moving average with linearly decreasing weights from N at the most recent bar to 1 at the oldest bar), the lag is: $$ \\text{lag}\_{\\text{WMA}}(N) \\;=\\; \\frac{N - 1}{3} \\text{ bars} $$ The WMA has less lag than the SMA at the same length. The unflattering reality: the WMA was a trader's construction built without the filter-theory math to back it. The asymmetric weighting that buys the reduced lag also produces a non-flat group delay (different lags at different frequencies, which distorts the input's shape) and worse stopband attenuation (high-frequency noise leaks through more than it does in an SMA of equal length). The WMA's lag advantage is paid for in two structural costs the trader rarely notices. The Hull moving average, the DEMA, the TEMA, and the various "low-lag" constructions in the technical analysis literature use combinations of EMAs to reduce lag below the EMA equivalent. The cost is always one of three: amplified high-frequency noise, non-flat group delay, or both. The article ["Why Moving Averages Can Lie at Turning Points"](https://aligrithm.com/why-moving-averages-can-lie-at-turning-points/) (forthcoming in this series) covers the structural failure mode of these constructions. ## Three flavors of lag cost Lag is paid in three currencies. Each one matters for a different downstream consumer. Cost 1: money. A trend-following strategy that enters on a moving-average crossover misses the first half of the average's length in P&L. A 50-day SMA breakout misses 25 bars of the move on average; the move's first 25 bars are typically the steepest because trends accelerate at the regime change. The realized strategy P&L is what survives after the lag tax. Concrete numbers on SPX 1990 to 2026, trends defined as moves of 10% or more from a swing low or high: $$ \\begin{array}{l|c|c|c|c} \\text{Filter} & \\text{Mean lag (bars)} & \\text{\\% of move captured} & \\text{Conditional Sharpe} & \\text{Lag tax (\\% of move)} \\\\ \\hline \\text{SMA(20)} & 9.5 & 79\\% & 0.42 & 21\\% \\\\ \\text{SMA(50)} & 24.5 & 56\\% & 0.31 & 44\\% \\\\ \\text{SMA(200)} & 99.5 & 18\\% & 0.09 & 82\\% \\\\ \\text{EMA(20)} & 9.5 & 81\\% & 0.45 & 19\\% \\\\ \\text{EMA(50)} & 24.5 & 58\\% & 0.33 & 42\\% \\\\ \\text{WMA(20)} & 6.3 & 84\\% & 0.38 & 16\\% \\\\ \\end{array} $$ Three readings. A 200-day SMA catches 18% of the average 10% move on SPX. The remaining 82% of the move happens before the filter detects it. The strategy that triggers on the 200-day SMA breakout is structurally late on every regime change and harvests only the tail of the move. A 50-day SMA catches 56% of the move. Better than 200-day, worse than half of the available P&L. The 50-day SMA has been the default trend filter for four decades because the trade-off between false signals (which a shorter filter generates) and lag (which a longer filter exacerbates) lands near 50 days for daily SPX. The "default" is a compromise, not an optimum. The WMA(20) catches 84% of the move, more than the EMA(20) at 81% and the SMA(20) at 79%. The WMA's lag advantage shows up in this metric. The cost (worse high-frequency rejection) shows up in the conditional Sharpe, which is lower than the EMA(20) despite catching more of the move on average. The WMA gets in earlier and gets stopped out more. Cost 2: information. A model that consumes the SMA(50) value at time t is consuming an estimate of the price approximately 25 bars in the past. The feature's effective timestamp is not t; it is t − 25\. A pipeline that uses SMA(50) as a "current state" feature is feeding the model a delayed measurement and treating it as if it were synchronized. The fix is to acknowledge the lag in the feature definition. The article ["How to Build Stationary Indicators from Non-Stationary Prices"](https://aligrithm.com/how-to-build-stationary-indicators-from-non-stationary-prices/) used differences of two MAs as a stationarity construction; the difference partially cancels the common lag (both filters lag, the difference lags by the lag of the longer minus the lag of the shorter). The right way to use a slow filter as a current-state feature is to use the difference (close − SMA), which has the lag of close (zero) minus the lag of SMA (group\_delay), giving an effective lag of −group\_delay. The negative lag here means the feature responds to the current bar plus carries the gap to the stale SMA, which is a usable signal because the gap is computed at the current bar. Cost 3: false confidence. A smoother line looks like cleaner information. The 200-day SMA on a stock chart appears more stable, more trustworthy, more "true" than the noisy raw close. The visual stability is the lag, made into a chart. The smoother the line, the more bars of the past it integrates into the present, and the more the present reading reflects the past instead of the current state. Traders who interpret "smoothness equals quality" make decisions based on filter outputs that are structurally weeks or months behind. A model that down-weights noisy features in favor of smooth ones is committing the same error in algorithmic form. The right feature audit asks "what does this feature represent at time t" and answers honestly: an estimate of the past, with the staleness given by the group delay. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## The lag-smoothness trade In the standard linear-filter family (SMA, EMA, WMA, Hann, Hamming, Blackman), increasing the smoothness costs lag at approximately a 1:1 ratio. Doubling the filter length doubles the lag and roughly halves the residual high-frequency content. $$ \\text{lag} \\cdot \\text{noise attenuation factor} \\;\\approx\\; \\text{constant within a filter family} $$ The constant depends on the filter family. The SMA family has a less favorable constant than the Blackman family, which has a less favorable constant than the Kaiser family at the same lag. The recursive low-pass and high-pass designs (covered in the articles ["The Trader's Guide to Low-Pass Filters"](https://aligrithm.com/the-traders-guide-to-low-pass-filters/) and ["High-Pass Filters for Traders"](https://aligrithm.com/high-pass-filters-for-traders/)) push the constant lower than any window-function family, at the cost of more complex coefficient calculation. What the trade does not allow: arbitrarily low lag and arbitrarily high smoothness simultaneously. The Heisenberg-like constraint is structural. Any filter that claims both is either (a) acausal (uses future data), (b) non-linear (uses median or other rank-based smoothers, which have different trade-offs), or (c) lying about one of the two properties. The Hull MA, DEMA, and TEMA constructions buy reduced lag by subtracting a slow filter from a fast filter and scaling the residual. The reduced lag is real. The cost is that the residual contains the high-frequency content the fast filter passed plus a sign-flipped version of the slow filter's content, which amplifies the noise content of the input. The Hull MA on a noisy series produces a less-laggy but visibly noisier output than the EMA of equal length. The trade is real; only the marketing is misleading. ## Lag-aware backtesting A backtest that uses a filter for entry timing and a different filter for exit timing must account for both lags. Two common failure modes appear: Failure mode 1: entry on a fast filter, exit on a slow filter. The strategy enters near the start of the move and exits long after the move ended. The realized PnL captures less of the move than the entry would suggest because the exit lag eats both sides of the move (late to enter the reversal, late to exit the trend). The right convention is to use filters with comparable lag on entry and exit, or to make the asymmetry explicit and justified. Failure mode 2: trailing stops based on a slow filter. A 50-day trailing-MA stop catches the bottom 25 bars late on average. The stop's lag is paid as drawdown beyond what the entry-side analysis would predict. The lag-aware backtesting protocol records the effective lag of every filter in the strategy stack and reports the cumulative lag-tax line item. A strategy whose nominal Sharpe is 0.9 with a 25-bar effective lag stack and a 0.4 lag-tax penalty has a true Sharpe of 0.5 once the lag is paid in P&L. The article ["How to Think About Indicator Lag Before Backtesting"](https://aligrithm.com/how-to-think-about-indicator-lag-before-backtesting/) (forthcoming in this series) covers the protocol mechanics. ## What this changes in practice Three operational shifts. Every moving average in the feature library carries its group delay as metadata. "ema\_50\_close" stores not only the construction but also the explicit "lag = 24.5 bars." Feature consumption code that aligns features in time accounts for the lag at the consumption point, not at the model. Crossover strategies are budgeted by lag, not by historical Sharpe. A 200-day crossover has 100-bar lag and harvests at most the tail of long trends. The strategy is sized to the post-lag P&L expectation, not to the in-sample Sharpe that ignored the lag. WMA, Hull MA, DEMA, and TEMA are not in the feature library by default. The "low-lag" constructions trade lag for noise amplification or phase distortion, both of which break downstream usage. The article on low-lag attempts (forthcoming) covers when one of them might survive a careful audit; none of them pass by default. ## Visualizing the lag tax ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/05/article_40-visual-0.png) The figure makes the lag tax visible in calendar time. The shorter filters get the signal sooner; none of them get it in time. ## KEY POINTS - Lag is the structural cost of every smoothing filter. The lag identity for a symmetric FIR filter of length N is (N−1)/2 bars, constant across frequencies. The EMA with smoothing α has lag ≈ (1−α)/α. - The WMA has lag (N−1)/3, less than the SMA of equal length. The cost is non-flat group delay (phase distortion) and worse high-frequency rejection. The construction predates the filter-theory math that would have flagged the trade-offs. - The lag-smoothness trade is structural. Lag × noise attenuation factor is approximately constant within a filter family. Doubling smoothness doubles lag. - Hull MA, DEMA, and TEMA buy reduced lag by amplifying high-frequency content or distorting phase. The reduced lag is real; the cost is in the noise or shape of the output. - Lag is paid in three currencies: money (late entries miss the first half of a move), information (feature at time t represents input at time t − lag), false confidence (a smoother line looks more stable when it is staler). - On SPX 1990 to 2026, the lag tax on 10%+ moves is 21% of the move for SMA(20), 44% for SMA(50), 82% for SMA(200). The "default 50-day" is a compromise between false signals and lag, not an optimum. - A model that uses SMA(50) as a "current state" feature is consuming an estimate of the price 25 bars ago. The right construction is the difference (close − SMA(50)), which has effective lag of -25 (a feature about the current gap to the lagging average), not the SMA value alone. - Backtests must account for the lag of every filter in the strategy stack. A strategy with 25-bar effective lag has a lag tax that compounds on every entry and exit. - Crossover strategies are budgeted by lag, not by historical Sharpe. A 200-day crossover harvests only the tail of long trends because by the time it fires, 80% of the move is gone. - The lag tax cannot be escaped within the linear-filter family. Any filter that claims arbitrarily low lag and arbitrarily high smoothness is using future data, using non-linear (rank-based) operations, or lying. - The next article in this series (["Why the SMA Is Often a Terrible Smoother"](https://aligrithm.com/why-the-sma-is-often-a-terrible-smoother/)) covers the SMA's specific stopband failures, which compound the lag tax on top of the basic lag identity. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Statistically Sound Indicators for Financial Market Prediction - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Statistically+Sound+Indicators+for+Financial+Market+Prediction+Timothy+Masters&ref=aligrithm.com) - [Cycle Analytics for Traders - John Ehlers (Amazon)](https://www.amazon.com/s?k=Cycle+Analytics+for+Traders+John+Ehlers&ref=aligrithm.com) ### 2.17 No Filter Is Predictive: What Traders Misunderstand About Smoothing URL: https://aligrithm.com/no-filter-is-predictive-what-traders-misunderstand-about-smoothing/ Last updated: 2026-07-27T13:23:43.000Z The screen shows SPX with two lines drawn through it: a 20-day SMA and a 50-day SMA. The fast line just crossed above the slow line. The trader's framework names the event "a buy signal" and reads it as a forecast of the next move. The event is not a forecast. It is an accounting identity. The 20-day average of the last 20 closes is now above the 50-day average of the last 50 closes. The statement is entirely about the past 50 bars. The next bar does not appear in either calculation. The crossover at time t is fully determined by data through t−1, with no t+1 dependence anywhere. The core thesis of filter theory: no filter is predictive. Every linear filter, every recursive filter, every multi-stage cascade has an output that is a function of past inputs and nothing else. The output is deterministic given the inputs. The future cannot enter the formula because the future has not happened. This article opens the filter-theory track in this pillar. The statistical track that closed with the article ["Feature Engineering Before Machine Learning"](https://aligrithm.com/feature-engineering-before-machine-learning/) treated indicators as statistical objects: distributions, transforms, entropy. The filter-theory track treats them as filter outputs: transfer functions, frequency responses, lag, gain. The two tracks ask different questions about the same primitives. This article makes the first question concrete: what does a filter actually do, and what does it not do? ## The transfer function framing A linear filter takes an input series x(t) and produces an output series y(t) through a relationship of the form: $$ y(t) \\;=\\; \\sum\_{k=0}^{N} b\_k\\, x(t-k) \\;-\\; \\sum\_{k=1}^{M} a\_k\\, y(t-k) $$ The first sum is the feedforward part (the b coefficients applied to past inputs). The second sum is the feedback part (the a coefficients applied to past outputs). A non-recursive filter has all a\_k = 0\. A recursive filter has at least one non-zero a\_k. The simple moving average SMA(N) is the non-recursive case with b\_k = 1/N for k = 0 to N−1. The transfer function H(z) is the ratio of the output to the input in the z-transform domain: $$ H(z) \\;=\\; \\frac{Y(z)}{X(z)} \\;=\\; \\frac{b\_0 + b\_1 z^{-1} + b\_2 z^{-2} + \\ldots + b\_N z^{-N}}{1 + a\_1 z^{-1} + a\_2 z^{-2} + \\ldots + a\_M z^{-M}} $$ Every term in H(z) is a negative power of z. The z⁻¹ operator represents a one-bar delay. There is no z⁺¹ term anywhere. A linear filter cannot reference the future because the algebra has no symbol for it. The transfer function tells you four things about the filter, none of which is "what will happen at t+1": The gain |H(e^{j 2π f})| at frequency f tells you how much the filter amplifies or attenuates a sine wave at that frequency. The phase ∠ H(e^{j 2π f}) at frequency f tells you how much the filter shifts a sine wave at that frequency. A linear filter introduces phase lag at all frequencies. The group delay tells you how much the filter delays the bulk of the signal energy. For a symmetric SMA(N), the group delay is (N−1)/2 bars at all frequencies. The pole-zero structure tells you which frequencies the filter passes and which it attenuates. None of these properties is forward-looking. They are descriptions of how the filter maps a past input to a present output. The output is, by construction, a delayed and frequency-reshaped version of the past. ## Where the "predictive" confusion comes from Four common mistakes that conflate filter output with prediction. Each one has a specific source and a specific consequence. Mistake 1: smoothed indicators look stable, so they appear to capture "the trend" which the trader then treats as the future. The stability comes from averaging past observations, not from estimating future ones. A 50-day SMA is a stable line because it integrates 50 bars of noise into one number; the integration is over the past 50 bars and contains no information about bar 51\. The line's stability is a property of the filter, not of the data's future. Mistake 2: faster moving averages crossing slower moving averages are read as "the fast one is leading the slow one." Both filters lag the input. The fast one lags less. A 10-day SMA lags the current price by 4.5 bars (group delay (N−1)/2 = 4.5). A 50-day SMA lags by 24.5 bars. The crossover is the moment when the 10-day catch-up has overtaken the 50-day catch-up, which is a property of how delayed each filter is, not a property of where the price is going next. Mistake 3: crossover signals are read as "the trend has changed." The trend changed earlier, by an amount equal to the difference in the two lags. By the time SMA(10) has crossed SMA(50), the actual trend change (whatever "trend" means) happened roughly 10 to 25 bars in the past. The crossover is a delayed detection event, not a forward prediction. Mistake 4: stacking filters ("smooth the smoothed series to remove residual noise") is read as a precision-improvement step. Cascading two filters multiplies their transfer functions. Cascading two SMA(10) filters produces a single equivalent filter with longer group delay and stronger attenuation, but no predictive content has been added because none was there to begin with. Doubling the smoothing doubles the lag. The four mistakes share a structural error: treating the filter's output at time t as if it were an estimate of the input at time t+1\. The filter's output at time t is an estimate of the input at time (t − group\_delay) under a frequency-content reshaping rule. The two estimates are not the same, and they are not even close on data that has any drift or trend. ## What filters actually do: frequency decomposition A filter does one thing structurally: it separates the input series into frequency components and reweights them. A low-pass filter keeps the slow oscillations and attenuates the fast ones. A high-pass filter does the reverse. A band-pass filter keeps a band of intermediate frequencies and attenuates everything outside the band. A notch filter attenuates one specific frequency. $$ \\text{output power at frequency } f \\;=\\; |H(e^{j 2 \\pi f})|^2 \\cdot \\text{input power at frequency } f $$ The framing converts every "what should I smooth with" question into "which frequencies do I want to keep, and which do I want to remove?" Three observations follow. Smoothing keeps the trend (low-frequency content) and removes the noise (high-frequency content) only if the trend you want is actually in the low-frequency band of the input. If the trend you want lives at a frequency that the filter attenuates, smoothing has destroyed the signal. Mean reversion lives in high-to-medium frequencies. A 50-day low-pass filter on a mean-reverting series removes the mean-reversion signal along with the noise. The right filter for mean reversion is a high-pass or band-pass, not a low-pass. The article ["High-Pass Filters for Traders"](https://aligrithm.com/high-pass-filters-for-traders/) (forthcoming in this series) details the construction. Cycle features live in the band-pass region. A 30-day band-pass filter retains oscillations with periods in the 25-to-35-day range and attenuates everything else. The output is a feature whose information content is the cycle's phase and amplitude, not a prediction. The article ["Band-Pass Filters: The Most Underused Tool in Technical Analysis"](https://aligrithm.com/band-pass-filters-the-most-underused-tool-in-technical-analysis/) covers the construction. The filter is a feature extractor. It exposes the part of the input that lives in a specific frequency band. The predictive content is what a model does with that exposed feature, not the filter output itself. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Worked example: SMA(50) on SPX SPX daily, 1990 to 2026\. Compute SMA(50) of close. Test two claims about its output. Claim 1: SMA(50) is "predictive" of next-day return. Operationalize: regress next-day log return on the current SMA(50) value. Claim 2: SMA(50) state is a feature whose value conditions next-day return. Operationalize: bin observations by the sign of (close − SMA(50)) and compute conditional expected next-day return in each bin. $$ \\begin{array}{l|c|c|c} \\text{Claim} & \\text{Statistic} & \\text{Value} & \\text{Verdict} \\\\ \\hline \\text{SMA(50) value predicts next-day return} & R^2 \\text{ of regression} & 0.0002 & \\text{no predictive content} \\\\ \\text{Sign of (close} - \\text{SMA(50)) conditions next-day return} & \\Delta\\bar{r}\_{t+1}\\text{ between bins} & 4.1 \\text{ bps} & \\text{feature, small edge} \\\\ \\text{SMA(50) slope sign conditions next-day return} & \\Delta\\bar{r}\_{t+1}\\text{ between bins} & 6.7 \\text{ bps} & \\text{feature, slightly larger edge} \\\\ \\text{SMA(10) crossover SMA(50) timing predicts next 20-day return} & \\text{conditional Sharpe} & 0.08 & \\text{near-zero edge} \\\\ \\end{array} $$ Four readings. The raw SMA(50) value has no predictive content as a regressor of next-day return. The R² is at the sampling-noise floor. The number that traders read as "the trend line" carries no information about tomorrow's return when regressed directly. The sign of (close − SMA(50)) does condition next-day return by about 4 basis points. The feature is real but small. The 4-bp edge is a structural feature of equity indices (mean reversion above the moving average is weaker than mean reversion below), not a property of the filter doing anything predictive. The filter exposed a state; the data carried the conditional structure. The slope sign of SMA(50) (positive or negative one-bar change in the SMA) conditions next-day return by about 6.7 basis points. The slope is a high-pass-ish feature derived from the low-pass filter, and the small edge is the residual momentum content the filter did not erase. Again, the filter is a feature extractor; the edge belongs to the data. The classic "buy when SMA(10) crosses above SMA(50)" rule has a conditional Sharpe of 0.08 over a 20-day forward window on SPX 1990 to 2026\. Below transaction costs on most realistic deployments. The crossover is a delayed detector of past trend changes, and by the time it fires, most of the move it would have caught has already happened. The rule produces a near-zero edge because the lag eats whatever signal the crossover represents. The pattern is consistent: filters expose state, state can be a feature, and a feature can have a small edge if the state correlates with the conditional distribution of forward returns. Nothing in this pattern is the filter "predicting" anything. ## The lag tax Every filter that smooths charges a tax in lag. The identity: $$ \\text{lag of a symmetric filter} \\;=\\; \\frac{N - 1}{2} \\text{ bars, where } N \\text{ is the filter length} $$ For an SMA(50), lag is 24.5 bars. For an SMA(200), lag is 99.5 bars. For an EMA with smoothing constant α, lag is approximately (1 − α) / α bars, which for α = 0.1 (roughly the 19-day EMA) is 9 bars. The lag tax is paid in two currencies. Money: late detection of trend changes means trades open after the bulk of the move. Information: the filter's output reflects the state of the market N/2 bars ago, not now. A model that consumes a 50-day SMA value at time t is consuming an estimate of the price at time t − 25, not the current price. The next article in this series (["The Hidden Cost of Every Moving Average: Lag"](https://aligrithm.com/the-hidden-cost-of-every-moving-average-lag/)) makes the lag-cost argument structurally and gives the operational implications. The point here is that the lag tax is not optional. Every smoothing filter charges it, and the receipt is in the group delay of the filter's transfer function. ## What this changes in practice Three operational shifts. Filter outputs are stored in the feature library as features, not as predictions. The naming convention reflects the role: "sma\_50\_close" is a feature column, not a signal. A model is fit on the feature; the model's output is the prediction. The two roles are kept separate in the code and in the audit trail. The crossover rule and its variants are not in the strategy library by default. A strategy that uses a crossover is using a delayed detector. If the delayed detection still produces an edge after transaction costs, the strategy ships; if not, it does not. The decision is made by walk-forward backtest with permutation testing (the article ["How to Test Indicator Thresholds Without Fooling Yourself"](https://aligrithm.com/how-to-test-indicator-thresholds-without-fooling-yourself/) covers the protocol), not by the visual appeal of the crossover. Filter choice is made by frequency-content reasoning, not by trial-and-error. The question "which moving average should I use" is replaced by "which frequency band carries the signal." Mean reversion is high-pass; trend following is low-pass; cycles are band-pass. The frequency reasoning determines the filter family before the parameters are tuned, which removes most of the parameter search the trial-and-error approach generates. ## Visualizing the past-only structure ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/05/article_39-visual-0.png) The figure makes the structural point visually. The filter's domain is the past. The future is not a term in the algebra. Smoothing summarizes; it does not extrapolate. ## KEY POINTS - Every linear filter has the form y(t) = Σ b\_k x(t−k) − Σ a\_k y(t−k). The output is a function of past inputs and past outputs only. There is no z⁺¹ term in the transfer function H(z). - "No filter is predictive" is a structural statement, not a soft warning. The algebra has no symbol for the future. A filter cannot predict what its construction has no representation for. - The transfer function H(z) tells you four things: gain at each frequency, phase shift at each frequency, group delay, and pole-zero structure. None of the four is forward-looking. - The "fast MA leads the slow MA" framing is wrong. Both filters lag. The fast one lags less. The crossover is the moment the smaller lag overtook the larger lag, not the moment the future tipped. - Crossover signals are delayed detectors of past changes. By the time SMA(10) crosses SMA(50), the underlying trend change happened 10 to 25 bars earlier. - Stacking filters cascades their transfer functions. Two SMA(10) filters in series produce a single filter with longer group delay than either alone. Smoothing the smoothing doubles the lag without adding signal. - A filter is a feature extractor, not a predictor. It exposes a state (the low-frequency content for an SMA, the high-frequency residual for a difference operator, the band-passed cycle for a band-pass). The predictive content lives in the conditional distribution of forward returns given the state. - Smoothing keeps low-frequency content and removes high-frequency content. If the signal you want is in the high frequencies (mean reversion, short-horizon momentum), smoothing destroys it. - Mean reversion is a high-pass signal. Trend following is a low-pass signal. Cycle phenomena are band-pass signals. Choose the filter family by which frequency band the signal lives in. - Lag of a symmetric N-bar filter is (N−1)/2 bars. SMA(50) lags by 24.5 bars; SMA(200) lags by 99.5 bars. The lag is a structural property of the transfer function and is paid in late entries and stale information. - On SPX 1990 to 2026, regressing next-day return on SMA(50) value has R² ≈ 0.0002\. Binning by sign of (close − SMA(50)) shows a 4-bp conditional difference. The SMA(10)-cross-SMA(50) timing rule has conditional Sharpe 0.08 over a 20-day window. The filter exposes states; the states carry small structural edges. - Filter outputs are stored as features in the feature library, not as signals. The model is what produces the prediction; the filter is what shapes the model's input. - The right opening question for any filter choice is "which frequency band carries the signal." The wrong opening question is "which moving average should I tune." The frequency question fixes the filter family before any parameter is searched. - The next article in this series quantifies the lag cost of moving averages and gives the operational implications for backtesting and execution. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Statistically Sound Indicators for Financial Market Prediction - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Statistically+Sound+Indicators+for+Financial+Market+Prediction+Timothy+Masters&ref=aligrithm.com) - [Cycle Analytics for Traders - John Ehlers (Amazon)](https://www.amazon.com/s?k=Cycle+Analytics+for+Traders+John+Ehlers&ref=aligrithm.com) ### 2.16 Feature Engineering Before Machine Learning URL: https://aligrithm.com/feature-engineering-before-machine-learning/ Last updated: 2026-07-27T13:23:43.000Z Run the same XGBoost with the same hyperparameters on four feature sets, same target (next-day SPX return sign), same 1990 to 2010 training window, same 2010 to 2026 test window. Feature set A: raw OHLCV columns plus a 50-day moving average of close. Test AUC: 0.502\. Indistinguishable from a coin flip. Feature set B: log returns, percent changes, raw RSI. The basic transforms most pipelines stop at. Test AUC: 0.509. Feature set C: the full statistical feature pipeline from the prior fifteen articles. Stationary by construction, ATR-normalized, TCR-aware sigmoid where appropriate, median-centered, walk-forward statistics. Test AUC: 0.524. Feature set D: same as C plus a small set of filter-theory features (the subject of the next article in this series). Test AUC: 0.531. Same model. Same data. Same target. Same training window. The AUC swings from "no edge" to "publishable result" by changing nothing except the columns the model consumes. This article is the closing argument of the statistical-feature block. The thesis: on market data, feature engineering contributes 90 to 95% of the achievable edge, and model selection contributes the remaining 5 to 10%. The right order is to spend most of the research budget on the feature side and only then to pick a model. The reverse order, which is what most retail and institutional ML pipelines do, is why most retail and institutional ML pipelines produce features set B numbers. The next article in this series (["No Filter Is Predictive: What Traders Misunderstand About Smoothing"](https://aligrithm.com/no-filter-is-predictive-what-traders-misunderstand-about-smoothing/)) starts the parallel argument from filter theory. The pipeline this article synthesizes is the statistical track; the filter-theory track adds another layer of pre-ML structure on top. ## Why feature engineering dominates on market data Three structural reasons. None of them have to do with the relative sophistication of the modeling toolkit. The signal-to-noise ratio is brutal. A daily-frequency feature with a real edge has an information content of 1 to 5 × 10⁻³ bits against a binary target. The target itself has variance close to 1\. The model is being asked to detect a 0.1 to 0.5% nudge in conditional probability against background noise that is 200 to 1000 times the signal. Model capacity that cannot find that needle in the haystack is irrelevant; model capacity that can find it does not need much capacity. What matters is whether the feature exposed the needle or hid it. Sample size is fixed and small. SPX has 9,000 daily bars since 1990\. AAPL has 11,000 bars since IPO. Every instrument has a finite history that cannot be expanded. A deep-learning model with 10⁶ parameters sees, at most, 9,000 samples per instrument. The model is data-starved by orders of magnitude relative to the parameter count, and the only fix is to compress the input space at the feature stage. Hand-engineered features encode prior knowledge about market structure (mean reversion in oscillators, drift in indices, volatility clustering) that the model would need 100 to 10,000 times more data to learn from scratch. The downstream loss function is unforgiving. A trading strategy turns model output into capital allocation. A model that produces correct ranks but miscalibrated magnitudes still loses money on a stop-and-reverse strategy. The features that survive in production are the ones whose distributional properties (boundedness, stationarity, regime invariance) make the model's output stable under deployment. The features that fail in production are the ones whose in-sample distribution differs from their live distribution, regardless of how clever the model wrapped around them was. The asymmetry is structural. Improving the model on poor features adds capacity to a search that cannot find the signal. Improving the features changes what signal the model can find at all. ## The pipeline, in order The fifteen prior articles in this pillar each addressed one stage. The full sequence: Stage 1: stationarize. The article ["The Case Against Raw Price Indicators"](https://aligrithm.com/the-case-against-raw-price-indicators/) rejected raw price; the article ["How to Build Stationary Indicators from Non-Stationary Prices"](https://aligrithm.com/how-to-build-stationary-indicators-from-non-stationary-prices/) gave the constructive ladder. The output of stage 1 is a feature with stationary mean and approximately stationary variance, verified by ADF p-value, train-test feature-range coverage, and rolling-mean drift. Stage 2: diagnose the histogram. The article ["Why Indicator Histograms Matter"](https://aligrithm.com/why-indicator-histograms-matter/) catalogued seven shapes and the action for each. The histogram is read before any scalar diagnostic. The output of stage 2 is a shape label (light-tailed bell, heavy-tailed bell, U-shape, bimodal, one-sided exponential, clump-with-outliers, uniform) and the corresponding action. Stage 3: diagnose the tail content. The article ["Range/IQR: A Simple Test for Indicator Tail Problems"](https://aligrithm.com/range-iqr-a-simple-test-for-indicator-tail-problems/) gave the R/IQR diagnostic. The article ["Why Predictive Power Often Lives in the Tails"](https://aligrithm.com/why-predictive-power-often-lives-in-the-tails/) gave the TCR (Tail Concentration Ratio) per-decile decomposition. The output of stage 3 is a (R/IQR, TCR) pair that determines whether the tails carry signal or noise. Stage 4: apply the structural transform. The article ["Taming Indicator Tails with Sigmoid Transforms"](https://aligrithm.com/taming-indicator-tails-with-sigmoid-transforms/) gave the recipe for low-TCR heavy-tailed bells. Roots and logs handle one-sided exponentials. Inverse logistic handles U-shapes. Regime splits handle bimodal features. The output of stage 4 is a transformed feature with a histogram in the "ship" range and a retained MI that has not dropped. Stage 5: pick the right central-tendency and spread statistics. The article ["Why the Median Often Beats the Mean in Trading Features"](https://aligrithm.com/why-the-median-often-beats-the-mean-in-trading-features/) set the L1 default for centering, smoothing, correlation, and rolling regression on heavy-tailed series. The output of stage 5 is a feature whose internal statistics (medians, MADs, Spearman correlations) are robust to outliers by construction. Stage 6: verify with entropy and mutual information. The article ["Relative Entropy as an Indicator Quality Score"](https://aligrithm.com/relative-entropy-as-an-indicator-quality-score/) gave the indicator-side scalar. The retained MI against the target is the second scalar. A feature whose RE is below 0.7 or whose MI is at the sampling-noise floor is rejected at stage 6 regardless of how clean its earlier diagnostics looked. The output of stage 6 is a binary pass/fail on indicator quality. Stage 7: validate the threshold and the direction. The article ["How to Test Indicator Thresholds Without Fooling Yourself"](https://aligrithm.com/how-to-test-indicator-thresholds-without-fooling-yourself/) gave the permutation test for the threshold search. The article ["Why You Should Test Long and Short Thresholds Separately"](https://aligrithm.com/why-you-should-test-long-and-short-thresholds-separately/) gave the asymmetric two-sided null. The output of stage 7 is a permutation-corrected p-value per direction. Stage 8: walk-forward audit. All statistics used in stages 1 through 7 are computed from training-window data only. The walk-forward updates them forward in time without re-fitting on the held-out window. The output of stage 8 is a feature definition that lives in the feature library with all statistics, all transforms, and all parameters stored as metadata. After stage 8, the feature ships. The model is fit on the output. The model's contribution to the edge is bounded above by what the stage-8 feature exposed; choosing a different model after stage 8 moves the AUC by 1 to 2 points at most. Choosing different features before stage 8 moves the AUC by 10 to 20 points. ## The "deep learning fixes this" objection A common objection: end-to-end deep-learning models do their own feature engineering by training on raw inputs. The objection is true on image data (ImageNet has 14 million labeled images), true on language (a large corpus has 10¹¹ tokens), and false on market data. The deep-learning feature-discovery argument depends on dataset size. A convolutional network learns edge detectors in its first layer because there are enough labeled images for the supervised signal to drive the gradient through to those features. The supervised signal per parameter is high. On market data, the supervised signal per parameter is the inverse: 9,000 daily samples per instrument against 10⁶ parameters in a small transformer. The gradient through to first-layer features is noisy and the model defaults to learning the in-sample noise structure of the training window. The empirical evidence backs the math. Published deep-learning results on daily-frequency trading either (a) use multi-instrument pools to artificially expand the effective sample size, (b) use intraday data where N is in the millions per instrument, or (c) report in-sample AUC numbers that do not survive walk-forward evaluation. The category-(c) papers are the loud ones; the category-(a) and (b) papers are doing feature engineering implicitly through their data construction. The honest position: deep learning is a feature-learning architecture, and on market data with low SNR and small N, the features it learns are dominated by the noise. The same hand-engineered features that a careful researcher builds are the features a deep-learning model would learn from with 100,000× more data. Since the data is not available, the hand engineering is the substitute. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## What feature engineering cannot do Three structural limitations. Naming them prevents the pipeline from being oversold. Feature engineering cannot create alpha where none exists. If the underlying market is a martingale at the frequency of interest, every feature-engineered signal has expected value zero after transaction costs. The pipeline finds the signal that is structurally present; it does not manufacture signal. The article ["Why Trading Is a Probability Business, Not a Certainty Business"](https://aligrithm.com/why-trading-is-a-probability-business-not-a-certainty-business/) in the scientific-trader pillar covered the martingale baseline. Feature engineering cannot predict regime shifts. The pipeline assumes stationarity over a forecast horizon. Regime shifts that change the conditional structure of the target invalidate the features that were stationary before the shift. The diagnostic for an upcoming regime shift is a separate research program (covered in the systems pillar). Inside a regime, the features work; across a regime boundary, the features need to be re-validated. Feature engineering cannot survive structural alpha decay. A feature that was predictive in 1990 to 2010 because of pre-electronification market microstructure may carry zero signal in 2010 to 2026 because the microstructure changed. The walk-forward audit catches the decay in retrospect; nothing in the feature library catches it prospectively. The right response is feature retirement and replacement, not feature reconfiguration. These limitations do not weaken the case for feature engineering. They sharpen what feature engineering is for. The pipeline finds the signal that is there. It does not make signal where there is none. ## Worked example: four feature sets, same model XGBoost with depth 4, 200 boosting rounds, learning rate 0.05\. Target y\_t = sign(P\_{t+1}/P\_t − 1) on SPX daily. Train 1990 to 2010, test 2010 to 2026. $$ \\begin{array}{l|c|c|c|c} \\text{Feature set} & \\text{N features} & \\text{Train AUC} & \\text{Test AUC} & \\text{Avg coverage} \\\\ \\hline \\text{A: raw OHLCV + MA(50)} & 6 & 0.612 & 0.502 & 41\\% \\\\ \\text{B: basic (log ret, \\% chg, raw RSI)} & 8 & 0.547 & 0.509 & 91\\% \\\\ \\text{C: full statistical pipeline} & 14 & 0.539 & 0.524 & 99\\% \\\\ \\text{D: C + filter-theory features (preview)} & 18 & 0.541 & 0.531 & 99\\% \\\\ \\end{array} $$ Four readings. Feature set A has the highest in-sample AUC (0.612) and the worst test AUC (0.502). The 11-point gap is overfitting on the in-sample noise of the raw price feature. Average train-test coverage is 41%, meaning more than half the test bars sit outside the training range of at least one feature. The model is extrapolating constantly. Feature set B closes the train-test gap from 11 points to 4 points by switching to stationary transforms (log return, percent change). The test AUC lifts from 0.502 to 0.509\. The 7-point lift comes from removing the extrapolation problem, not from a smarter model. Same model, different inputs. Feature set C closes the train-test gap to 1.5 points and lifts test AUC to 0.524\. The 15-point delta over set B (1.5%-AUC) comes from stage 3 (tail-aware transforms), stage 5 (median centering), stage 6 (entropy / MI filtering), and stage 7 (walk-forward statistics). The model has not changed; the features have absorbed the structure that the model was previously asked to extract from noise. Feature set D adds four filter-theory features (a low-pass-filtered momentum, a high-pass-filtered residual, a band-pass-isolated oscillator, and a decycler trend). Test AUC lifts another 0.7 points to 0.531\. The filter-theory track is a parallel discipline to the statistical track; the article ["No Filter Is Predictive: What Traders Misunderstand About Smoothing"](https://aligrithm.com/no-filter-is-predictive-what-traders-misunderstand-about-smoothing/) starts the explanation. The delta from A to D is +2.9 AUC points. Of those, +2.2 points come from statistical feature engineering. The model contributes zero. The data contributes zero (same dataset, same dates). The features did the work. ## Visualizing the AUC ladder ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/05/article_38-visual-0.png) The two panels make the asymmetry concrete. Panel (a) is the AUC ladder; panel (b) is the structural reason for it (coverage of the feature distribution between train and test). Feature set A's failure is not "the model is too simple" — it is that the model is asked to extrapolate on 59% of test bars. The fix is upstream of the model. ## What this changes in practice Three operational shifts. The research budget allocation inverts the usual ratio. 80% of researcher time goes to feature design, validation, and audit. 20% goes to model selection and hyperparameter tuning. Most institutional ML projects spend the budget the other way around. The feature-first budget allocation is the corrective. The feature library is the artifact, not the model. Models are commodities (XGBoost, LightGBM, MLP, transformer). Features are intellectual property. A new model with the same features produces a similar AUC. A new feature with the same model produces a different AUC. The feature library is what gets versioned, audited, and protected. The model code is what gets swapped when the next library improves. Pre-modeling diagnostics are gates, not advisories. A feature that fails the ADF test does not enter the model. A feature whose TCR is above 0.6 and that has been sigmoid-compressed is rejected at the audit. A feature whose Spearman and Pearson correlations differ by more than 0.10 is sent back for outlier inspection. The gates are enforced before the model runs, not after the backtest disappoints. ## KEY POINTS - On market data, feature engineering contributes 90 to 95% of the achievable edge. Model selection contributes 5 to 10%. The asymmetry comes from low SNR, fixed-and-small sample size, and unforgiving downstream loss functions. - The statistical feature pipeline runs in eight stages: stationarize, diagnose histogram, diagnose tail content, apply structural transform, pick L1 statistics, verify with RE and MI, validate threshold and direction, walk-forward audit. Each stage gates the next. - Stage 1 rejects raw price and produces stationary features. Stage 2 reads the histogram before any scalar. Stage 3 measures (R/IQR, TCR) to decide whether tails carry signal. Stage 4 applies the transform the diagnostic prescribed. - Stage 5 defaults centering to median, spread to MAD or IQR, correlation to Spearman, rolling regression to LAD or Huber. Stage 6 confirms with RE > 0.7 and MI above the sampling-noise floor. Stage 7 permutation-tests thresholds and asymmetric directions. Stage 8 holds all statistics to training-window-only computation. - The "deep learning fixes feature engineering" objection is correct on image and language data and wrong on market data. The supervised signal per parameter is the inverse: \~9,000 daily samples against 10⁶ parameters. Hand engineering substitutes for the data the model does not have. - Feature engineering cannot create alpha where none exists, predict regime shifts, or survive structural alpha decay. The limitations are not a weakness of the pipeline. They define what the pipeline is for. - On SPX daily with XGBoost, raw OHLC features give test AUC 0.502 (extrapolation). Basic transforms give 0.509 (stationary). Full statistical pipeline gives 0.524\. Adding filter-theory features lifts to 0.531\. The model never changed. - The research budget should be 80% features, 20% model. Most institutional pipelines invert this and produce the corresponding AUC numbers. - The feature library is the IP. The model is the commodity. The library is versioned, audited, and protected. Models are swapped between releases. - Pre-modeling diagnostics are gates, not advisories. ADF, R/IQR, TCR, KS-drift, Spearman/Pearson divergence each have rejection thresholds. Features that fail the gates do not enter the model. - The next article in this series starts the filter-theory parallel discipline. Filter theory adds a second layer of pre-ML structure on top of the statistical pipeline. The two disciplines are complementary; the AUC numbers from feature set C to feature set D quantify the magnitude. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Statistically Sound Indicators for Financial Market Prediction - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Statistically+Sound+Indicators+for+Financial+Market+Prediction+Timothy+Masters&ref=aligrithm.com) - [Cycle Analytics for Traders - John Ehlers (Amazon)](https://www.amazon.com/s?k=Cycle+Analytics+for+Traders+John+Ehlers&ref=aligrithm.com) ### 2.15 Why the Median Often Beats the Mean in Trading Features URL: https://aligrithm.com/why-the-median-often-beats-the-mean-in-trading-features/ Last updated: 2026-07-27T13:23:42.000Z Compute the rolling 100-day mean of SPX daily log return on February 28, 2020\. The value is +0.0004 (roughly 10% annualized). Compute the same statistic on March 20, 2020, three weeks later. The value is −0.0019 (roughly −48% annualized). The estimator dropped by 5 standard deviations in 14 trading days. The "estimate of the central tendency" of SPX daily returns moved by half a year's worth of nominal return because seven days in March produced double-digit moves and each one entered the average at weight 1/100. Compute the rolling 100-day median of the same series. On February 28: +0.0006\. On March 20: +0.0004\. The median moved by 0.0002, or roughly 0.4 of a daily standard deviation. The crash days entered the sorted list but did not change which day sat at the 50th percentile of the window. The mean is the wrong estimator of the center for any trading feature whose distribution has tails. The median is the right estimator. The same logic applies up the L1 / L2 ladder: MAD and IQR beat standard deviation for spread measurement, Spearman beats Pearson for correlation on heavy-tailed pairs, LAD beats OLS for rolling regression on noisy market series, and rolling median filters beat moving averages for smoothing volume, range, and other outlier-prone primitives. This article catalogs the five places L1 statistics beat L2 statistics in trading feature design, gives the structural reason (breakdown point and heavy-tail efficiency), and names the counter-cases where the mean is still correct. The operational rule: if the indicator does not have outliers use mean; if it has outliers use median. Most trading features have outliers. The mean is the default that should have been the median. ## The breakdown point The breakdown point of an estimator is the smallest fraction of observations that, when corrupted arbitrarily, can drive the estimator to an arbitrary value. $$ \\text{breakdown}(\\text{mean}) \\;=\\; \\frac{1}{n}, \\qquad \\text{breakdown}(\\text{median}) \\;=\\; 0.5 $$ A single observation can push the mean to any value. Half the observations must be corrupted to push the median by more than a fixed amount. On a 100-bar rolling window, one large outlier shifts the mean by 1% of its full value, the same outlier shifts the median by 0 (it does not change which observation sits at the sorted middle). Three related estimators share the median's breakdown property: The median absolute deviation (MAD = median of |x\_i − median(x)|) has breakdown 0.5\. The standard deviation has breakdown 0 (a single observation squared dominates the sum). The IQR (75th percentile minus 25th percentile) has breakdown 0.25\. The range has breakdown 1/n. Spearman's rank correlation has breakdown 0.5 for the rank computation (a single pair-swap moves the rank correlation by at most 2/n). Pearson's product-moment correlation has breakdown 0 because the products in the numerator are unbounded in the raw observations. Breakdown is not a niche concern on trading data. SPX daily returns have a kurtosis around 25\. A 5-sigma daily move appears 5 to 10 times per decade. On any 100-bar window that contains one such day, the mean and the std are 5 to 30% off their "no-outlier" values; the median and MAD are within 1 to 2% of theirs. ## Five places L1 beats L2 Application 1: centering a feature for downstream transformation. The article ["Taming Indicator Tails with Sigmoid Transforms"](https://aligrithm.com/taming-indicator-tails-with-sigmoid-transforms/) used the training-sample median as the centering statistic. The reason is exactly the breakdown argument: the in-sample mean is contaminated by the same tails the sigmoid is meant to tame. Centering by the mean shifts the bulk of the distribution by an amount proportional to the contamination, which the sigmoid then maps off-center. Centering by the median places the bulk's symmetric center at zero regardless of the tails. Application 2: spread measurement for normalization. The article ["Range/IQR: A Simple Test for Indicator Tail Problems"](https://aligrithm.com/range-iqr-a-simple-test-for-indicator-tail-problems/) and the article ["How to Build Stationary Indicators from Non-Stationary Prices"](https://aligrithm.com/how-to-build-stationary-indicators-from-non-stationary-prices/) both used IQR over standard deviation for variance-scaling normalizations. The MAD is the closer L1 analogue of the standard deviation: $$ \\text{MAD}(X) \\;=\\; \\text{median}\\bigl(|X\_i - \\text{median}(X)|\\bigr) $$ For Gaussian data, the standard deviation equals 1.4826 × MAD. For heavy-tailed data, the standard deviation is dragged up by the tails and overestimates the bulk's spread; the MAD measures the bulk's spread directly. A spread-normalization divides by MAD (or by IQR, which is 1.35 × MAD for Gaussian data and a similar-magnitude robust statistic in general). Dividing by std on a heavy-tailed feature under-normalizes the body and over-normalizes the tails. Application 3: smoothing outlier-prone primitives. Volume, true range, absolute return, and intraday range are heavy-tailed by construction. A moving average of these primitives carries the outlier days at weight 1/W for W bars after the day, dragging the smoothed value upward for a 1/W-decaying duration. A rolling median of the same primitives kills the spike immediately: as soon as the outlier exits the sorted middle of the window, the median returns to baseline. $$ \\text{rolling\\\_median}(X, W)\_t \\;=\\; \\text{median}\\bigl(X\_{t-W+1},\\,\\ldots,\\,X\_t\\bigr) $$ Rolling median filters have a second property worth naming: they preserve edges. A step change in the underlying signal (say, a regime shift in volume after an index inclusion) appears as a step in the rolling median. A moving average of the same step rounds the corners over W bars. For smoothing primitives that can have genuine step changes mixed with spikes, the median is structurally the right filter. Application 4: correlation on heavy-tailed pairs. Pearson correlation between two return series, two volume series, or any pair where both legs are heavy-tailed is dominated by a small number of joint-tail observations. One co-movement on a crash day can push Pearson correlation from 0.3 to 0.7 and back again over a 100-bar window. Spearman rank correlation operates on the ranks of the observations, not the values. The crash-day pair becomes "rank 100 paired with rank 100" instead of "−10% paired with −8%". The contribution of the joint-tail observation is bounded above by the largest possible rank-product. Spearman's correlation moves slowly and stably, which is what is wanted for any feature built on rolling correlation. Kendall's τ has the same property and is preferred for very small samples. Application 5: rolling regression for prediction-deviation features. Indicators of the form "predicted − actual" built from rolling regressions (Detrended RSI, cross-market deviation indicators) suffer when the regression is fit with OLS on heavy-tailed data. One outlier pair pushes the OLS slope by a multiple of the noise scale, and the resulting "deviation" is dominated by the regression's own fitting error rather than the structural deviation it was meant to detect. LAD (least absolute deviations) regression and Huber regression replace the squared-error loss with an L1 or robust loss. The fitted slope is no longer dominated by single-pair outliers; the deviation indicator measures the structural mispricing it was designed for. The cost is computational (LAD has no closed form and Huber needs iterative reweighting), and on modern hardware the cost is negligible for the rolling-window sizes used in feature engineering. ## When the mean is correct Three counter-cases where the mean is the right choice and the median is wasted robustness. Light-tailed bell features. RSI(14) on SPX, Stochastic K(14), and other bounded-by-construction oscillators have R/IQR around 2 to 4 (per the article ["Range/IQR: A Simple Test for Indicator Tail Problems"](https://aligrithm.com/range-iqr-a-simple-test-for-indicator-tail-problems/)). The tails are not contaminated, the breakdown concern does not bite, and the mean is more efficient than the median on near-Gaussian data. Use the mean. Additive aggregations. Computing the total realized variance over a window is a sum, and the sum's expected value is N × E\[X²\]. Replacing the sum with the median × N would be mathematically wrong. When the downstream computation requires the additive interpretation (portfolio P&L, total volume, integrated variance), the mean is the correct estimator regardless of tail behavior. MSE-target downstream models. A model whose loss function is mean squared error implicitly treats the mean as the correct point estimate of the target. Using the median to center a feature consumed by an MSE-loss model creates a mild inconsistency between the feature's center and the model's loss-minimizing prediction. For most practical purposes the inconsistency is irrelevant (the model adjusts in the bias term), but for some loss-function variants (quantile regression, asymmetric losses) the centering should match the loss. The decision rule: use the median when the distribution has tails and the downstream usage does not require the additive or MSE-aligned interpretation. Use the mean when the distribution is light-tailed or the additive interpretation is structurally required. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## The efficiency cost is not what you think A common objection: the median has only 64% efficiency of the mean on Gaussian data, so the median is a worse estimator. The objection is correct on Gaussian data. The objection misses the point on market data. Statistical efficiency is measured relative to the variance of the estimator. For Gaussian data, the median's variance is (π / 2) × σ² / n ≈ 1.57 σ² / n, while the mean's variance is σ² / n. The median needs more observations to achieve the same precision. For heavy-tailed data, the calculation reverses. The variance of the sample mean depends on the underlying variance, which is large on heavy-tailed distributions. The variance of the sample median depends on the local density at the median, which is the same for Gaussian and heavy-tailed bells (both have a smooth bulk around the center). On a Student's t-distribution with 3 degrees of freedom (kurtosis around 9, similar to many market features), the median's efficiency is roughly 1.5× the mean's. On a distribution with kurtosis 25 (SPX daily returns), the median's efficiency is 3× to 5× the mean's. The relevant efficiency comparison for market features is on the actual distribution, not on a hypothetical Gaussian. On real distributions, the median is more efficient than the mean for most trading features whose downstream use only needs a center estimate. The Gaussian-efficiency penalty is a textbook artifact, not a real cost. ## Worked example: SPX rolling center estimate through March 2020 SPX daily log returns, 100-bar rolling window. Three center estimators: mean, median, and 20% trimmed mean (a hybrid that drops the top and bottom 10% before averaging). $$ \\begin{array}{l|c|c|c|c} \\text{Date} & \\text{rolling mean} & \\text{rolling median} & \\text{20\\% trimmed mean} & \\text{notes} \\\\ \\hline \\text{2020-02-28 (pre-crash)} & +0.0004 & +0.0006 & +0.0005 & \\text{light tails in window} \\\\ \\text{2020-03-13 (mid-crash)} & -0.0011 & +0.0005 & -0.0001 & \\text{6 large negative days in window} \\\\ \\text{2020-03-20 (peak stress)} & -0.0019 & +0.0004 & -0.0003 & \\text{worst readings} \\\\ \\text{2020-04-30 (recovery)} & -0.0008 & +0.0007 & +0.0001 & \\text{positive days re-entering} \\\\ \\text{2020-06-30 (post-crisis)} & +0.0006 & +0.0008 & +0.0007 & \\text{crash days still in window} \\\\ \\end{array} $$ Three readings. The rolling mean spikes downward by 5 daily-σ from February 28 to March 20, even though the underlying "central tendency" of SPX daily returns (whatever the true value is) did not change by anything close to that. The estimator absorbed seven extreme days at weight 1/100 each and reported a "center" that lived nowhere in the data's bulk. The rolling median moved by 0.0002 over the same period. The seven extreme days entered the sorted window but did not change which day sat at the 50th percentile. The median's estimate of the center is robust to the crash and is the operationally useful number. The 20% trimmed mean is the intermediate option. It throws away the top and bottom 10% of the window before averaging the rest. The trimmed mean is more efficient than the median on heavy-tailed but not pathological distributions and moves slightly with the crash (the bulk shifts down by a fraction of a daily-σ as positive days exit the window and are replaced by less-extreme but still-negative days). The 20% trimmed mean is a reasonable default when the median is too aggressive and the mean is too fragile. For the centering application (subtract this from the feature before downstream processing), the median is the right choice. For an estimate of the realized central tendency over the window (the additive interpretation), the mean is the right choice. Different applications, different estimators, both names of the same number on Gaussian data and completely different numbers on market data. ## Visualizing the crash-window divergence ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/05/article_37-visual-0.png) The two-panel diagnostic makes the breakdown-point argument concrete. The raw input has seven crash-week spikes; the rolling mean inhales them at weight 1/100 and drifts five daily-sigma off the bulk's actual center, while the rolling median ignores them by construction. The 20% trimmed mean is the intermediate option for cases where the median is too aggressive. ## What this changes in practice Three operational shifts. Centering statistics in the feature library default to median, not mean. The feature definition stores the median and the IQR (or MAD) of the training window. The mean is computed and reported alongside for diagnostic purposes but is not used in the transform. Smoothing primitives default to rolling median for outlier-prone series (volume, true range, absolute return) and rolling mean for bounded-by-construction series (RSI, Stochastic K). The default is recorded in the feature definition, not chosen at modeling time. Switching from rolling mean to rolling median on a known-outlier-prone primitive is a feature-design correction, not a modeling decision. Correlation features in the feature library default to Spearman rank correlation, not Pearson. The Pearson correlation is computed and stored for diagnostic comparison. A feature whose Spearman and Pearson correlations differ by more than 0.10 over the same window is flagged for outlier inspection; the difference usually traces to one or two joint-tail observations that Pearson absorbed and Spearman did not. ## KEY POINTS - The mean has breakdown point 1/n. A single outlier can drive the mean to any value. The median has breakdown 0.5\. Half the observations must be corrupted to move the median by more than a fixed amount. - On heavy-tailed distributions (kurtosis 9 or higher), the median is more statistically efficient than the mean, not less. The "64% efficiency" Gaussian penalty does not apply to market data. - Centering for sigmoid prep and forced centering: use the training-sample median. The training-sample mean is contaminated by the same tails the transform is being applied to remove. - Spread measurement for variance scaling: use MAD or IQR. Standard deviation is dragged up by the tails and overestimates the bulk's spread. IQR ≈ 1.35 × MAD for Gaussian; both are L1-style robust alternatives. - Smoothing for outlier-prone primitives (volume, true range, absolute return): use a rolling median filter. The moving average carries spikes at weight 1/W for W bars; the rolling median kills the spike as soon as it exits the sorted middle. - Rolling median filters preserve edges (step changes). Moving averages round the corners of step changes over W bars. For primitives with mixed spike-and-step behavior, the median is the right filter. - Correlation on heavy-tailed pairs: use Spearman or Kendall, not Pearson. Pearson can swing by 0.4 from a single joint-tail observation. Spearman moves at most 2/n per pair swap. - Rolling-regression-based indicators (detrended RSI, deviation-from-prediction features): fit with LAD or Huber regression instead of OLS. OLS slopes are pushed by single outlier pairs; LAD and Huber slopes are stable. - Use the mean when the distribution is light-tailed (RSI, Stochastic K), when the downstream computation requires an additive interpretation (portfolio P&L, total variance), or when the loss function is MSE. - On SPX 100-bar rolling center, the mean spikes by 5 daily-σ through March 2020 and the median moves by 0.4 daily-σ. The mean is reporting the contamination; the median is reporting the bulk's center. - The 20% trimmed mean is a hybrid that is more efficient than the median on moderately heavy-tailed data and less fragile than the mean. It is a reasonable default when the median is too aggressive and the mean is too fragile. - The feature library defaults: centering = median, smoothing of outlier-prone primitives = rolling median, correlation = Spearman, robust regression = LAD or Huber. The mean is computed alongside for diagnostic comparison but is not the operational statistic. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Statistically Sound Indicators for Financial Market Prediction - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Statistically+Sound+Indicators+for+Financial+Market+Prediction+Timothy+Masters&ref=aligrithm.com) - [Cycle Analytics for Traders - John Ehlers (Amazon)](https://www.amazon.com/s?k=Cycle+Analytics+for+Traders+John+Ehlers&ref=aligrithm.com) ### 2.14 Taming Indicator Tails with Sigmoid Transforms URL: https://aligrithm.com/taming-indicator-tails-with-sigmoid-transforms/ Last updated: 2026-07-27T13:23:42.000Z A common reflex: indicator has a heavy-tailed bell shape (Shape 2 from the article ["Why Indicator Histograms Matter"](https://aligrithm.com/why-indicator-histograms-matter/)), so apply tanh to the raw values. The histogram comes back tight and bounded in (−1, +1). R/IQR drops from 12 to 1.4\. The diagnostic says clean. Refit the model. AUC drops from 0.516 to 0.504\. The naive tanh did not just compress the tails. It also collapsed the central body of the distribution into the steep part of the sigmoid, where small differences in the raw indicator produce small differences in the transformed value. The signal in the body was lost to the sigmoid's slope, and the signal in the tails was lost to its saturation. The transform took the whole feature with it. The cure is not a different sigmoid. The cure is the same sigmoid, applied correctly. The rule: scale the raw values so the bulk of the distribution lives in the linear region of the sigmoid (the region where the sigmoid behaves like a straight line) and only the genuine tail outliers reach the saturating knees. A correctly scaled sigmoid preserves the body's signal at near-linear resolution and clips the tails. An incorrectly scaled sigmoid does both jobs wrong. This article is the recipe. Four construction steps, three family choices (tanh, normal CDF, logistic), one tuning parameter (the linear-region scale), and a worked example on a heavy-tailed feature where the correct scale recovers the AUC the naive scale destroyed. The decision of *when* to use a sigmoid (low-TCR features with heavy tails) versus when to skip it (high-TCR features where the tails carry signal) is the subject of the article ["Why Predictive Power Often Lives in the Tails"](https://aligrithm.com/why-predictive-power-often-lives-in-the-tails/). This article assumes you already know the feature is low-TCR and the sigmoid is the right transform. The question is how to scale it. ## The four-step construction Step 1: Center the distribution. The sigmoid family operates around zero. A feature with a non-zero center pushes its bulk off-center in the sigmoid's argument and the linear region no longer covers the bulk. Subtract the in-sample median: $$ X^c\_t \\;=\\; X\_t \\;-\\; \\text{median}\_{\\text{train}}(X) $$ The median is preferred over the mean because the in-sample mean is contaminated by the same tails the sigmoid is meant to tame. Use the training-sample median, not the rolling median, because the rolling median changes the centering as a function of t and breaks the sigmoid's interpretation. Step 2: Measure the bulk's spread. The bulk is the central 50% to 80% of the distribution. The IQR is the standard measurement and is robust to the tails: $$ s \\;=\\; \\text{IQR}\_{\\text{train}}(X^c) \\;=\\; X^c\_{\[0.75\]} - X^c\_{\[0.25\]} $$ The standard deviation can be used instead of IQR when the indicator has light tails. For heavy-tailed indicators (the case this article addresses), IQR is the correct choice because a single outlier inflates std and silently rescales every observation by the wrong factor. The article ["Range/IQR: A Simple Test for Indicator Tail Problems"](https://aligrithm.com/range-iqr-a-simple-test-for-indicator-tail-problems/) covered the IQR's robustness rationale. Step 3: Scale the centered values so the bulk lands in the linear region. The HTAN function (hyperbolic tangent) is linear within roughly ±1.5 in its argument. The normal CDF is linear within roughly ±1.0\. The logistic function is linear within roughly ±1.0\. To map the central 80% of the distribution into the linear region, multiply the centered values by a scale factor: $$ Y\_t \\;=\\; \\tanh\\!\\Bigl(\\,\\alpha \\cdot \\frac{X^c\_t}{s}\\,\\Bigr), \\qquad \\alpha \\approx 1.0 \\text{ to } 1.5 $$ For tanh, α = 1.2 puts the IQR boundaries at ±1.2 in the sigmoid's argument, which sits inside the linear region (±1.5). For the normal CDF, α = 0.8 puts the IQR boundaries at ±0.8, which sits inside its linear region (±1.0). The α parameter is the single tuning knob that controls how aggressively the sigmoid compresses the tails. Step 4: Optionally rescale the output. Tanh outputs to (−1, +1). Normal CDF outputs to (0, 1). Logistic outputs to (0, 1). For downstream consumers that want a (−1, +1) output, the normal CDF and logistic need a shift and rescale (output × 2 − 1). For consumers that want a (0, 1) output, tanh needs (output + 1) / 2\. The output rescaling does not change the feature's information content; it only changes the axis label. The full construction in one line: $$ \\tilde{X}\_t \\;=\\; \\tanh\\!\\Bigl(\\,\\alpha \\cdot \\frac{X\_t - m}{s}\\,\\Bigr), \\quad m = \\text{median}\_{\\text{train}}(X), \\quad s = \\text{IQR}\_{\\text{train}}(X) $$ The training-sample statistics m and s are stored with the feature and held fixed in production. Re-computing them on test data is the look-ahead anti-pattern the article ["How to Build Stationary Indicators from Non-Stationary Prices"](https://aligrithm.com/how-to-build-stationary-indicators-from-non-stationary-prices/) catalogged. ## The linear region rule The sigmoid is approximately linear in a window around zero and saturating outside that window. The scaling rule operationalizes the "stay in the linear region" guidance with a specific quantile target. Three operational targets, in order of aggressiveness: Aggressive compression: α scaled so the in-sample 95th percentile of |X^c / s| lands at the edge of the linear region (±1.5 for tanh). The central 90% of the distribution is in the linear region and the top/bottom 5% are pulled toward the saturating knees. Equivalent to α ≈ 0.6 to 0.8. Moderate compression (default): α scaled so the in-sample 90th percentile lands at the edge of the linear region. The central 80% is linear; the top/bottom 10% are in the knees. Equivalent to α ≈ 1.0 to 1.2. Mild compression: α scaled so the in-sample 75th percentile (the IQR boundary) lands at the edge of the linear region. The central 50% is linear; the rest is in the knees. Equivalent to α ≈ 1.5 to 2.0\. Useful when the TCR is borderline and partial tail preservation matters. For a typical low-TCR heavy-tailed-bell indicator on equity indices, moderate compression (α ≈ 1.0 to 1.2) is the default. The aggressive variant is reserved for indicators known to have a few data errors mixed into the tail. The mild variant is for indicators that almost qualified as high-TCR but still need some tail protection. The choice of α is a structural parameter of the feature, not a hyperparameter of the model. It is set once at the feature design stage, stored with the feature definition, and not re-optimized on the backtest. ## Sigmoid family options Three sigmoid functions appear in the indicator-design literature. They differ in details that matter for specific downstream consumers. Hyperbolic tangent (tanh). Output range (−1, +1). Linear region ±1.5 in argument. Symmetric around zero. Derivative at zero is 1 (so the linear region has slope 1). The default for indicators that already live on a signed axis (oscillators, mean-reversion features, normalized differences). $$ \\tanh(x) \\;=\\; \\frac{e^x - e^{-x}}{e^x + e^{-x}} $$ Normal CDF (Φ). Output range (0, 1). Linear region ±1.0 in argument. Symmetric around the midpoint (Φ(0) = 0.5). Derivative at zero is 1/√(2π) ≈ 0.4 (linear region has slope ≈ 0.4). Often preferred when the downstream model is a logistic regression or a probability-output model, because the CDF interpretation aligns with probability outputs. Logistic. Output range (0, 1). Linear region ±1.0 in argument. Symmetric around the midpoint. Derivative at zero is 0.25\. Computationally cheaper than the normal CDF (no error function required). Behaves almost identically to tanh after the linear shift Y = 2 × logistic − 1. For practical purposes, tanh and the rescaled logistic are interchangeable. The normal CDF is the right choice only when there is a structural reason to want a CDF interpretation (Bayesian models, probabilistic ensembles). Pick tanh by default; pick the normal CDF when the downstream model is built around probabilities. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Worked example: SPX Range/Close, three sigmoid scalings SPX daily, 1990 to 2026\. Feature: same-bar Range divided by Close (a Shape 2 heavy-tailed bell, low TCR ≈ 0.30 because the signal lives in the body, not the extreme range days). Train 1990 to 2010, test 2010 to 2026. Four candidate features. The raw feature, then three tanh variants with different α scaling. $$ \\begin{array}{l|c|c|c|c} \\text{Feature} & \\text{R/IQR} & \\alpha & \\text{Test AUC} & I(X;Y) \\times 10^3 \\\\ \\hline \\text{Raw Range/Close} & 12.4 & - & 0.503 & 1.1 \\\\ \\tanh(\\alpha \\cdot (X-m)/s),\\; \\alpha = 4.0 \\text{ (over-tight)} & 1.05 & 4.0 & 0.504 & 1.0 \\\\ \\tanh(\\alpha \\cdot (X-m)/s),\\; \\alpha = 1.2 \\text{ (moderate)} & 1.42 & 1.2 & 0.519 & 2.0 \\\\ \\tanh(\\alpha \\cdot (X-m)/s),\\; \\alpha = 0.3 \\text{ (over-loose)} & 8.9 & 0.3 & 0.508 & 1.4 \\\\ \\end{array} $$ Four readings. The raw feature has R/IQR = 12.4 and test AUC barely above the coin flip. The model trained on the 1990-to-2010 distribution sees test-period tail values the training set did not contain, and the loss function gets dominated by a few large observations. The MI is real (1.1) but the model cannot consume it. The over-tight sigmoid (α = 4.0) puts the IQR boundary at ±4.0 in the tanh argument, where tanh has already saturated. The bulk of the distribution is mapped into the saturating region. Differences in the raw feature in the body produce almost no difference in the transformed feature. The histogram looks clean (R/IQR = 1.05, the tightest bound the diagnostic recognizes), but the AUC and MI confirm the transform destroyed the signal it was meant to preserve. The moderate sigmoid (α = 1.2) puts the IQR boundary at ±1.2 in the tanh argument, which is inside tanh's linear region (±1.5). The body of the distribution is approximately linear after the transform; the tails are pulled into the knees. The R/IQR comes back at 1.42, the AUC lifts to 0.519, and the MI lifts to 2.0 (higher than the raw because the model can now consume what the transform exposes). This is the correct scale. The over-loose sigmoid (α = 0.3) puts the IQR boundary at ±0.3 in the tanh argument, which is so close to zero that the whole sigmoid behaves like a linear function on this data. The R/IQR drops only to 8.9; the tails are barely touched. The AUC and MI improve over raw but not enough to justify the transform. The over-loose variant is the same as not transforming. The middle option is the only one that produces a useful feature. The two failure modes (over-tight and over-loose) bracket it on both sides. ## When not to sigmoid The article ["Why Predictive Power Often Lives in the Tails"](https://aligrithm.com/why-predictive-power-often-lives-in-the-tails/) covered the case structurally. The decision rule, restated for the sigmoid construction: | TCR | R/IQR | Action | | ---------- | ------- | --------------------------------------------------------------- | | Below 0.3 | Above 5 | Sigmoid with moderate α — body holds signal, tails are noise | | 0.3 to 0.5 | Above 5 | Sigmoid with mild α (≥ 1.5) — preserve partial tail gradient | | 0.5 to 0.6 | Above 5 | Wide-scale sigmoid or fourth-root — tails carry partial signal | | Above 0.6 | Above 5 | Winsorize at 99.5%, do not sigmoid — tails carry primary signal | | any | Below 3 | No transform — feature is already in shape | A sigmoid applied to a high-TCR feature is the worst case. It removes the tail differentiation that carried the signal and leaves the body that was carrying noise. The article's opening example (sigmoid on TCR = 0.78 realized vol, AUC drops from 0.518 to 0.506) shows the magnitude. ## Anti-patterns Four patterns that produce a feature that looks sigmoid-compressed but is functionally broken. Sigmoid on an uncentered distribution. A one-sided exponential (Shape 3 from the article ["Why Indicator Histograms Matter"](https://aligrithm.com/why-indicator-histograms-matter/)) has its mode at zero and no mass on the negative side. Subtracting the median moves the mode to the negative side but the distribution is still asymmetric. The sigmoid applied to such a feature compresses the upper tail and leaves the lower side functionally untouched. The right transform for one-sided distributions is a root or a log, not a sigmoid. Sigmoid on a bimodal distribution. Bimodal (Shape 5) features have two modes with a valley between them. The sigmoid maps the valley to the linear region and the two modes to opposite knees. The output looks unimodal at the histogram level but the conditional structure (the two regimes that produced the bimodality) is now hidden. The article ["Why Indicator Histograms Matter"](https://aligrithm.com/why-indicator-histograms-matter/) covered the regime-split fix; the sigmoid is not it. Sigmoid stacked on a log transform. The log is a transform from the root/log family that already compresses the tail. Applying tanh on top of log values produces double compression: the log handles the multiplicative tail and the sigmoid handles the residual additive tail. The combined transform's behavior is almost always over-tight on the body and adds no benefit over either transform alone. Pick one family, not both. Sigmoid with full-sample statistics. Computing m and s using the full training+test sample is the look-ahead anti-pattern the article ["How to Build Stationary Indicators from Non-Stationary Prices"](https://aligrithm.com/how-to-build-stationary-indicators-from-non-stationary-prices/) catalogged. The bound that the sigmoid imposes is a function of m and s, and m and s computed on the test set are not available in production. Use training-sample-only statistics, and update them forward in the walk-forward. ## Visualizing the scale parameter ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/05/article_36-visual-0.png) The three panels are the scale-parameter diagnostic. Same data, same sigmoid family, three α values, three feature qualities. Only the middle one survives. ## What this changes in practice Three operational shifts. Sigmoid transforms in the feature library carry α as part of the feature identifier. "Range\_over\_close\_tanh\_alpha\_1.2" and "Range\_over\_close\_tanh\_alpha\_0.8" are different features. The α is stored as metadata, not as a model hyperparameter. The α default for moderate compression is set to 1.2 for tanh, 0.8 for normal CDF, and 1.0 for logistic. These defaults map the in-sample IQR boundary to the edge of each sigmoid's linear region. Departures from the default require justification (recorded in the feature audit) tied to the TCR of the feature. The sigmoid m and s are stored with the feature definition, computed from training-window data only, and updated forward in walk-forward without re-fitting on the held-out window. The audit checks that the test-period feature distribution lies within the training-period support after the transform; a > 5% test-period saturation rate on the bounded output triggers a feature-drift review. ## KEY POINTS - The naive sigmoid (tanh applied to raw values, no scaling) compresses the body of the distribution into the saturating knees and destroys the signal that the body carried. The AUC penalty is typically 0.005 to 0.015 per feature. - The four-step construction: subtract the training-sample median (center), measure the IQR (spread), scale by α so the IQR boundary lands inside the sigmoid's linear region, apply the sigmoid. - The linear region of tanh is ±1.5 in argument. The linear region of the normal CDF is ±1.0\. The linear region of the logistic is ±1.0\. Scale the input so the bulk of the distribution lives inside that region. - α = 1.2 is the moderate-compression default for tanh. It maps the in-sample IQR boundary to ±1.2 in the tanh argument, leaving 80% of the distribution in the linear region and the top/bottom 10% in the knees. - α below 0.5 leaves the tails untouched (over-loose). α above 2.5 collapses the body into the knees (over-tight). Both failure modes are diagnosable from the histogram of the transformed feature. - tanh, the normal CDF, and the logistic are interchangeable for most purposes. The normal CDF is preferred when the downstream model is probabilistic. tanh is the default everywhere else. - The IQR is preferred to the standard deviation as the spread measure because a single tail outlier inflates the std and silently rescales the entire feature by the wrong factor. - The median is preferred to the mean as the center because the in-sample mean is contaminated by the same tails the sigmoid is being applied to remove. - The m and s used in the transform are computed from training-window data only. Full-sample statistics leak the test distribution into the feature definition. - A sigmoid on a high-TCR feature (TCR above 0.6) destroys the tail differentiation that carried the signal. Use winsorization instead. The decision matrix maps TCR and R/IQR to the correct transform. - A sigmoid on a one-sided exponential (Shape 3) is the wrong family entirely. Use a root or a log. A sigmoid on a bimodal feature (Shape 5) hides the regime structure. Split into regime-conditional features instead. - Stacking a sigmoid on top of a log is double compression with no benefit. Pick one family per feature, not both. - On SPX Range/Close (low TCR), the raw feature has test AUC 0.503 and the over-tight α = 4.0 sigmoid has test AUC 0.504\. The correctly scaled α = 1.2 sigmoid has test AUC 0.519 and MI 2.0, both higher than the raw because the model can now consume the body without the tail-leverage problem. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Statistically Sound Indicators for Financial Market Prediction - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Statistically+Sound+Indicators+for+Financial+Market+Prediction+Timothy+Masters&ref=aligrithm.com) - [Cycle Analytics for Traders - John Ehlers (Amazon)](https://www.amazon.com/s?k=Cycle+Analytics+for+Traders+John+Ehlers&ref=aligrithm.com) ### 2.13 Why Indicator Histograms Matter URL: https://aligrithm.com/why-indicator-histograms-matter/ Last updated: 2026-07-27T13:23:41.000Z You log three indicators in your feature library. All three have R/IQR = 4.2\. All three have relative entropy = 0.79\. All three have mutual information against next-day return sign of 1.8 × 10⁻³ bits. Three identical readings on the three scalar diagnostics. The histograms tell three different stories. The first histogram is a clean bell shape with light tails. Ship it. The second histogram is a tight spike at zero with two thin wings reaching out to the diagnostic-allowed edges. Most of the mass lives in 5% of the range. The scalars say it is fine. The model trained on it will spend all its capacity on the central 5%, see a flat surface across the rest, and miss the predictive content that the thin wings carry. The third histogram is bimodal: two equal humps at opposite ends with a valley in the middle. The scalars say it is fine. The indicator is mixing two market regimes whose conditional distributions of forward return are opposite, and the bimodality is the signal that the indicator should be split into two regime-conditional features. This article is about the histogram as the primary visual diagnostic for indicator design. The summary scalars (R/IQR from the article ["Range/IQR: A Simple Test for Indicator Tail Problems"](https://aligrithm.com/range-iqr-a-simple-test-for-indicator-tail-problems/), relative entropy from ["Relative Entropy as an Indicator Quality Score"](https://aligrithm.com/relative-entropy-as-an-indicator-quality-score/), TCR from ["Why Predictive Power Often Lives in the Tails"](https://aligrithm.com/why-predictive-power-often-lives-in-the-tails/)) are derived from the histogram. They compress it into one number each, and the compression loses information about shape that the scalars cannot represent. You read the histogram first. The scalars confirm what the histogram already said. The constructive answer for one specific histogram shape (heavy-tailed bell) is the subject of the next article in this series (["Taming Indicator Tails with Sigmoid Transforms"](https://aligrithm.com/taming-indicator-tails-with-sigmoid-transforms/)). This article is the catalog of shapes and the decision matrix that follows from each one. ## The seven shapes you will see Most real indicators land in one of seven distributional shapes. Each shape has a structural cause, a structural consequence for the model, and a specific corrective action. Shape 1: light-tailed bell. Roughly symmetric, single mode, tails that decay smoothly. The healthy default. RSI(14), 14-day stochastic K, Williams %R, and most ATR-normalized momenta on liquid instruments produce this shape. R/IQR is 2 to 4, relative entropy is 0.7 to 0.9, no transform required. Ship as-is. Shape 2: heavy-tailed bell. Symmetric around a central mode, but the tails decay slowly and a small number of extreme observations sit far from the bulk. R/IQR is 6 to 15, relative entropy drops to 0.5 to 0.7 because most observations clump in the center. Same-bar OHLC differences before normalization, volume-based indicators, and most price-change indicators on stressed instruments produce this shape. The decision depends on tail-concentration: high TCR features keep their tails, low TCR features get a sigmoid. The article ["Taming Indicator Tails with Sigmoid Transforms"](https://aligrithm.com/taming-indicator-tails-with-sigmoid-transforms/) gives the construction. Shape 3: one-sided exponential decay. Mode at one edge of the range (usually zero), monotonic decay toward the other edge, no mass on the negative side. Volume, traded notional, absolute return, realized variance, and unsigned momentum produce this shape. R/IQR is high but irrelevant because the distribution is one-sided; relative entropy is low because most observations live near zero. The fix is a root (square root for mild, fourth root for moderate) or a log transform. Sigmoid does not help because the distribution is not centered. Shape 4: U-shape. Modes at both edges, valley in the middle. RSI(2), short-lookback stochastic, and most divergence indicators on choppy markets produce this shape. R/IQR is low (the distribution is bounded), relative entropy is low (mass concentrated at edges). The shape is informative on its own (extreme readings are more common than central readings) but breaks linear regression and any model that assumes monotonic input-output relationships. The fix: inverse logistic transform, which maps the U-shape back to a bell. The article ["How to Build Stationary Indicators from Non-Stationary Prices"](https://aligrithm.com/how-to-build-stationary-indicators-from-non-stationary-prices/) mentioned the inverse logistic in passing; the operational rule is "U-shape → inverse logistic, no exceptions." Shape 5: bimodal with separation. Two distinct humps, deep valley between them, similar mass in each hump. The indicator is mixing two market regimes whose distributions are different enough that the indicator cannot represent them as one feature. Cross-sectional features computed across mixed universes (equities + crypto) produce this shape. Half-life-based mean-reversion indicators on instruments that transition between trending and ranging produce this shape. The fix is not a transform; it is to split the indicator into two regime-conditional features and let the model condition on the regime as a separate input. The article on regime conditioning (forthcoming in the systems pillar) covers the split mechanics. Shape 6: tight central clump with empty tails. Most observations in a narrow band (often 5 to 10% of the apparent range), a few isolated extreme values far out in the tails. R/IQR is high (the extremes inflate the range), relative entropy is low (the clump dominates the bin probabilities). The structural cause is usually a few outliers — data errors, splits not adjusted, a single Black Swan day — sitting in a feature otherwise well-behaved. The fix is winsorization at 99.5% or 99.9% before any other transform. The article ["Range/IQR: A Simple Test for Indicator Tail Problems"](https://aligrithm.com/range-iqr-a-simple-test-for-indicator-tail-problems/) covered the diagnostic; the histogram shows whether the offending observations are a continuous tail (real signal) or a few isolated points (data errors). Shape 7: roughly uniform. Equal probability across the range. R/IQR is 1.5 to 2, relative entropy is above 0.95\. The shape models prefer in principle. Almost never observed in raw indicators; usually appears only after an aggressive transform (rank, percentile, or histogram equalization). Uniform shape suggests the transform may have discarded magnitude information; check the retained MI against the un-transformed version to confirm the transform did not destroy signal. ## What the scalars miss Three diagnostics commonly used (R/IQR, relative entropy, MI) reduce the histogram to a number. The histogram contains shape information the number does not. R/IQR confuses one isolated outlier with a heavy continuous tail. R/IQR = 12 from a feature with a clean bell shape and one Black Monday observation is a different problem from R/IQR = 12 from a feature with a continuously fattened tail. The first is fixed by winsorization at 99.9%. The second needs a sigmoid or fourth-root. The scalar does not distinguish them; the histogram does at a glance. Relative entropy confuses a bimodal distribution with a uniform distribution at the same bin count. Two equal humps with a valley in between and a uniform distribution can produce identical relative entropy values when binned coarsely. The histogram shows the difference immediately; the entropy number reports them as identical-quality indicators. Mutual information confuses a feature with signal in the tails and a feature with signal in the body. Same MI = 1.8 × 10⁻³ from a TCR = 0.78 feature (signal lives in the tail deciles) and from a TCR = 0.22 feature (signal lives in the body) are two different operational problems. The first needs the tails preserved; the second tolerates a sigmoid. The histogram, plotted alongside the per-decile MI bar chart from the article ["Why Predictive Power Often Lives in the Tails"](https://aligrithm.com/why-predictive-power-often-lives-in-the-tails/), shows the difference. Read the histogram first. Run the scalars second. When the scalars disagree with the histogram, the histogram wins. ## Bin count and binning rules The number of bins matters because shape diagnostics depend on resolution. Too few bins (B = 5 to 10) smooth out the modes. A U-shape with B = 6 looks like an inverted bell. A bimodal distribution with B = 8 looks like a wide unimodal distribution. The cure is more bins. Too many bins (B > N / 10 for N observations) introduce empty bins and sampling noise. A U-shape with B = 100 on N = 500 observations shows a jagged histogram where every other bin is empty, and the eye reads the noise rather than the shape. The cure is fewer bins. The Freedman-Diaconis rule sets bin width to 2 × IQR / N^{1/3}. For SPX daily over 36 years (N = 9100), this gives B around 30 to 50 depending on the indicator. The Sturges rule gives B around log₂(N) + 1 ≈ 14, which is too few for the diagnostic purpose. The Freedman-Diaconis rule is the default; Sturges is for presentation only. For the diagnostic histogram, use 30 to 50 bins as a default and re-bin if the shape is ambiguous. The histogram is a tool, not an artwork. The bin count that resolves the shape is the right bin count. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Worked example: same scalars, three histograms Three SPX-derived features. Each one has R/IQR ≈ 4.2, relative entropy ≈ 0.79, and MI against next-day return sign ≈ 1.8 × 10⁻³ bits. $$ \\begin{array}{l|c|c|c|c} \\text{Feature} & \\text{R/IQR} & \\text{RE} & \\text{MI} \\times 10^3 & \\text{Histogram shape} \\\\ \\hline \\text{20d ATR-norm momentum (Shape 1)} & 4.1 & 0.81 & 1.8 & \\text{light-tailed bell} \\\\ \\text{Range/Close at bar (Shape 2)} & 4.3 & 0.78 & 1.8 & \\text{heavy-tailed bell} \\\\ \\text{Realized-skewness 60d (Shape 5)} & 4.2 & 0.79 & 1.8 & \\text{bimodal} \\\\ \\end{array} $$ Three readings. The 20-day ATR-normalized momentum has a light-tailed bell shape. The scalars say healthy. The histogram confirms. Ship as-is. Range/Close at bar has a heavy-tailed bell with the same R/IQR as the ATR-normalized momentum. The scalars say the same thing. The histogram says the tail is continuous (a real heavy tail, not isolated outliers). The decision is TCR-dependent: at TCR 0.55 (which Range/Close typically scores on equity indices), a wide-scale sigmoid is the right transform. The article ["Taming Indicator Tails with Sigmoid Transforms"](https://aligrithm.com/taming-indicator-tails-with-sigmoid-transforms/) details the construction. Realized 60-day skewness is bimodal: one hump around −0.4 (regimes with negative-skew returns) and one hump around +0.1 (regimes with mild positive skew). The two humps correspond to two different volatility regimes. The scalars report the bimodal feature as identical in quality to the unimodal features. The histogram says the feature is mixing two regimes and should be split into a "skew-magnitude" feature and a "skew-sign" indicator, or the regime should be conditioned on separately. Keeping the bimodal feature as a single column forces the model to learn the non-monotonic mapping between feature value and forward return, which trees can do with deep splits and linear models cannot do at all. Three identical scalar readings, three completely different operational decisions. ## The decision matrix Each histogram shape maps to one action. | Shape | Diagnostic signature | Action | | ------------------------------- | --------------------------------------------------------- | -------------------------------------------------------- | | Light-tailed bell | R/IQR 2-4, RE > 0.7, single mode | Ship as-is | | Heavy-tailed bell, low TCR | R/IQR > 5, RE 0.5-0.7, continuous tail, body holds signal | Sigmoid or fourth-root | | Heavy-tailed bell, high TCR | R/IQR > 5, RE 0.5-0.7, continuous tail, tails hold signal | Winsorize at 99.5%, otherwise keep | | One-sided exponential | R/IQR > 5, mode at edge, no mass on opposite side | Square root, fourth root, or log | | U-shape | R/IQR low, modes at edges, valley in middle | Inverse logistic | | Bimodal | Two humps with valley between, similar mass | Split into regime-conditional features; do not transform | | Tight clump + isolated outliers | R/IQR > 8, body in < 10% of range, < 1% of mass in tail | Winsorize at 99.9% first, then re-diagnose | | Roughly uniform | RE > 0.95, R/IQR \~ 1.5-2 | Verify retained MI vs pre-transform | The matrix is read in order: identify the shape from the histogram, look up the diagnostic signature to confirm, take the action. Shapes 2 and 3 are the only ones that need the TCR diagnostic; the other shapes are resolved by the histogram alone. ## Visualizing the seven shapes ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/05/article_35-visual-0.png) The seven panels are the central reference. The eye reads the shape in one second; the scalars take three queries to a database. ## Operational discipline Three rules that follow from the histogram-first principle. The histogram is logged with every feature. The feature library entry stores a 30-bin to 50-bin histogram of the training-set values, the bin edges, and the bin counts. New features cannot ship without a histogram artifact. The histogram is regenerated at every walk-forward step and compared to the previous step's histogram with a Kolmogorov-Smirnov test; a KS statistic above 0.05 between consecutive windows triggers a feature-drift review. Shape classification is part of the feature audit. The seven shapes are a small enumeration. The audit pipeline labels each feature with one of the seven shapes (via a simple shape-classifier or a manual review for borderline cases) and the corresponding action. Features whose shape changes across walk-forward windows are flagged as regime-dependent and re-examined. The scalar diagnostics confirm the histogram, they do not replace it. R/IQR, relative entropy, TCR, and MI are reported alongside the histogram for every feature. When the scalars are clean and the histogram is not, the verdict is "investigate the histogram." When the scalars are dirty and the histogram is fine, the verdict is "investigate the scalars' computation." The two never disagree when both are computed correctly. ## KEY POINTS - The histogram is the primary visual diagnostic for indicator design. Scalar summaries (R/IQR, relative entropy, MI, TCR) are compressions of the histogram and lose shape information. - Seven recurring shapes: light-tailed bell, heavy-tailed bell, one-sided exponential, U-shape, bimodal, tight clump with outliers, roughly uniform. Each shape has a specific corrective action. - Light-tailed bell ships as-is. ATR-normalized momenta, RSI(14), Stochastic K(14) on liquid instruments produce this shape on most samples. - Heavy-tailed bell needs a sigmoid or fourth-root if TCR is low (body holds signal), or winsorization at 99.5% if TCR is high (tails hold signal). The decision depends on tail-concentration, not on R/IQR alone. - One-sided exponential distributions (volume, realized variance, absolute return) need a root or log transform. Sigmoid does not help because the distribution is not centered. - U-shape is informative on its own but breaks linear models and monotonic-assumption models. Inverse logistic transforms it back to a bell. - Bimodal distributions are not fixed by transforms. The indicator is mixing two regimes whose conditional distributions differ enough that one feature cannot represent both. Split into regime-conditional features. - Tight clump with isolated outliers usually indicates data errors or a few Black Swan days. Winsorize at 99.9% before any other transform. - Roughly uniform shape is what models prefer but is rarely observed in raw indicators. When it appears after a transform, verify that the transform did not destroy magnitude information by checking retained MI. - Bin count matters: use Freedman-Diaconis (2 × IQR / N^{1/3}) for the diagnostic histogram. Sturges gives too few bins for shape detection. 30 to 50 bins is the operational default for daily-frequency data over 5+ years. - The decision matrix maps each shape to one action: ship, sigmoid, root, inverse logistic, regime split, winsorize, verify. The matrix is read in order: shape first, signature confirmed, action taken. - The feature library logs a histogram artifact with every feature. New features ship only with a histogram. Walk-forward steps regenerate the histogram and trigger a drift review when KS distance to the previous window exceeds 0.05. - Three indicators with identical R/IQR, identical relative entropy, and identical MI can have three completely different histograms and three completely different correct actions. The scalars never tell you which one is which. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Statistically Sound Indicators for Financial Market Prediction - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Statistically+Sound+Indicators+for+Financial+Market+Prediction+Timothy+Masters&ref=aligrithm.com) - [Cycle Analytics for Traders - John Ehlers (Amazon)](https://www.amazon.com/s?k=Cycle+Analytics+for+Traders+John+Ehlers&ref=aligrithm.com) ### 2.12 CMMA: A Better Momentum Primitive Than Price-minus-MA Alone URL: https://aligrithm.com/cmma-a-better-momentum-primitive-than-price-minus-ma-alone/ Last updated: 2026-07-27T13:23:41.000Z The plain Close minus Moving Average indicator (C\_t − MA\_k(C)\_t) is the most common mean-reversion primitive in technical trading. On SPX with k = 50, it produces a tidy oscillator that crosses zero around 90 times a year and matches the visual intuition of "is the price above or below its recent average." The same primitive evaluated as a feature for a predictive model fails on three structural problems. A 30-point deviation on SPX in 1995 (when SPX was at 600) is a 5% move. A 30-point deviation on SPX in 2025 (when SPX is at 5500) is a 0.5% move. The same feature value means two different events. The model trained across the 1995-to-2025 sample is forced to learn a mapping from a feature whose meaning is time-varying. A 30-point deviation in calm 2017 (realized vol 7%) is a 4-σ event. A 30-point deviation in stressed 2020 (realized vol 34%) is a 0.5-σ event. The same feature value means two different events in two different regimes within the same year. A 30-point deviation against a 50-day MA and a 30-point deviation against a 200-day MA are two different statements about how far the price has wandered. The plain construction gives them the same value. CMMA (Close Minus Moving Average) is the construction that fixes all three problems with a small number of additions: log prices instead of raw prices, MA over previous bars only, ATR in the denominator with current bar included, and a √(k+1) divisor that makes the result lookback-independent. The article ["How to Build Stationary Indicators from Non-Stationary Prices"](https://aligrithm.com/how-to-build-stationary-indicators-from-non-stationary-prices/) placed CMMA on the transform ladder; the article ["Why ATR Normalization Is More Than a Volatility Trick"](https://aligrithm.com/why-atr-normalization-is-more-than-a-volatility-trick/) argued for the ATR denominator. This article assembles the full construction, justifies each ingredient, and compares the result to alternatives. ## The full construction $$ \\text{CMMA}(k)\_t \\;=\\; \\frac{\\ln(C\_t) \\;-\\; \\frac{1}{k}\\sum\_{i=1}^{k}\\ln(C\_{t-i})}{\\text{ATR}\_W(\\ln C)\_t \\cdot \\sqrt{k+1}} $$ Four ingredients. Each one fixes a specific failure of the plain Close − MA construction. ## Ingredient 1: Log prices, not raw prices The numerator uses log(C\_t) − mean of log(C\_{t−i}) for i = 1..k, rather than C\_t − mean of C\_{t−i}. Two structural reasons. A log-price difference is a proportional return. ln(C\_t) − ln(C\_{t−i}) equals ln(C\_t / C\_{t−i}), which is the i-bar log return. The numerator of CMMA is therefore the difference between the current log price and the average log price over the last k bars, which is the same shape as the average i-step log return for i in 1..k. The construction lives in return-space, not in price-space, and return-space is where the stationary signal sits. The article ["The Case Against Raw Price Indicators"](https://aligrithm.com/the-case-against-raw-price-indicators/) covered the price-space-vs-return-space argument; CMMA inherits the result by construction. Cross-instrument comparability is automatic. A CMMA value of −0.4 on SPX, AAPL, and BTC is the same conditional event in each instrument's own scale: the current log-price is about 0.4 ATR-units below the recent log-price average, scaled for the lookback. The article ["Why ATR Normalization Is More Than a Volatility Trick"](https://aligrithm.com/why-atr-normalization-is-more-than-a-volatility-trick/) showed why ATR gives the cross-instrument coherence. The log transform gives the cross-time coherence within a single instrument. Both are needed; either one alone is not enough. What breaks if you skip it: replace log with raw and the numerator inherits the price-level multiplier. A 30-point deviation at SPX = 600 and a 30-point deviation at SPX = 5500 produce identical numerator values, but the conditional distributions of forward returns at those two regimes are not the same. The model trained on a pooled sample is forced to absorb the time-varying meaning of the feature in the rest of its capacity. ## Ingredient 2: MA over previous bars only The MA in the numerator is computed over bars t−1 through t−k. Bar t is the bar being evaluated; it does not enter its own moving average. The exclusion rule serves two purposes. It prevents the numerator from canceling part of its own signal. If bar t is included in the MA over k+1 bars, then the numerator becomes (C\_t − (k × MA\_prev + C\_t) / (k+1)) = (k / (k+1)) × (C\_t − MA\_prev). The current bar's contribution to the MA shrinks the deviation by a factor of k/(k+1). For k = 50 that is a 2% attenuation, small but present. For k = 5 it is a 17% attenuation, which dominates short-lookback variants. It enforces the timing the model needs. The MA must be computable from data strictly before bar t. The feature library convention is that any statistic used to build a feature at time t lives in the closed half-line (−∞, t−1\]. Including bar t in the MA violates the convention even when it is mathematically harmless on the value side, because the convention is what protects the rest of the pipeline from forgetting which window each computation lived in. What breaks if you skip it: an MA that includes the current bar attenuates the signal by k/(k+1) and contaminates the timing discipline. On a short-lookback CMMA the attenuation is enough to drop the retained MI by 15 to 20%. On a long-lookback CMMA the attenuation is small but the timing-discipline failure is the same. ## Ingredient 3: ATR with the current bar included The denominator's ATR is computed over W bars and includes bar t. The asymmetry between ingredient 2 (MA excludes current bar) and ingredient 3 (ATR includes current bar) is intentional. The MA is in the same units as the numerator and would attenuate the signal if it absorbed the current bar. The ATR is in the denominator and serves a different purpose: it captures the volatility regime that bar t lives in. Excluding bar t from the ATR would compute a regime-of-yesterday denominator for a regime-of-today numerator, which under-normalizes on the days when the regime is shifting. Including bar t puts the numerator and denominator in the same regime. The inclusion is not a look-ahead because the denominator is a scaling factor, not a predictive component. The feature's predictive content lives in the numerator and is computed entirely from data through t−1\. The denominator's job is to rescale the predictive content to a comparable axis, and the current bar's true range is part of the bar's own observed data at time t, not future data. W in the range 50 to 250 days, per the structural argument in the article ["Why ATR Normalization Is More Than a Volatility Trick"](https://aligrithm.com/why-atr-normalization-is-more-than-a-volatility-trick/). W = 100 is the default for k up to 30; W = 250 for k up to 100. What breaks if you skip it: ATR excluding bar t under-normalizes on regime-shift days and produces a feature with non-stationary variance during the most informative regime changes. The non-stationarity is small in absolute terms (a few percent per regime change) but compounds across the backtest as the precise days the model needs to handle correctly. ## Ingredient 4: The √(k+1) divisor The numerator has standard deviation that scales as √(k+1) when the log prices are an i.i.d. random walk. Dividing by √(k+1) removes the lookback dependence from the feature. The math is the same √k identity from the article ["Why ATR Normalization Is More Than a Volatility Trick"](https://aligrithm.com/why-atr-normalization-is-more-than-a-volatility-trick/), applied to a slightly different numerator. The deviation of bar t from the average of the previous k bars has variance: $$ \\text{Var}\\Bigl(\\ln C\_t - \\frac{1}{k}\\sum\_{i=1}^{k}\\ln C\_{t-i}\\Bigr) \\;=\\; \\sigma^2 \\cdot \\Bigl(1 + \\frac{1}{k}\\Bigr) \\;\\approx\\; \\frac{k+1}{k}\\,\\sigma^2 $$ Up to a factor of k that approaches 1 for large lookbacks, the standard deviation of the numerator scales as σ × √(k+1). Dividing by ATR × √(k+1) cancels both the volatility regime (the σ) and the lookback dependence (the √(k+1)). The practical consequence: CMMA(10), CMMA(50), and CMMA(200) all live on the same axis. A value of −0.5 means the same thing across lookbacks: the current log price is about half a "unit" below where the previous-k-bar average sits, where the unit is the per-bar log-price ATR. The model that consumes CMMA(10), CMMA(50), and CMMA(200) as three features sees three coherent measurements at three timescales, not three differently scaled measurements that need separate handling. What breaks if you skip it: an ATR-only normalization (no √(k+1)) leaves a feature with stationary variance per fixed lookback and lookback-dependent variance across lookbacks. The model that consumes ATR-only-normalized CMMA(10) and CMMA(200) sees two features whose distributions are shaped differently, and it learns separate mappings for each. Adding the √(k+1) collapses the two mappings into one and frees model capacity. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## The optional fifth ingredient: nonlinear compression CMMA values are unbounded in principle. In practice, a 6-σ day produces a CMMA reading around ±3 in lookback-normalized units, with extremes around ±5 on the most stressed days in the sample. The bulk of values lie in (−1.5, +1.5) for daily data on liquid instruments. A model with bounded activations (sigmoid, tanh) handles the bulk without issue but saturates on the tails. A model without bounded activations (trees, linear regression) handles the bulk and the tails without issue. The compression decision depends on the downstream model. When compression is applied, the standard form is a sigmoid scaled so the linear region (±1.5 in the sigmoid's argument) covers the bulk of the CMMA distribution. With CMMA's bulk in (−1.5, +1.5), a sigmoid with scale c that maps c × CMMA to (−1.5, +1.5) at the bulk's edges leaves the linear region for the bulk and the saturating region for the extremes. For c = 1 the linear region matches the bulk exactly; for smaller c, more of the distribution lives in the linear region at the cost of less differentiation in the tails. The article ["Why Predictive Power Often Lives in the Tails"](https://aligrithm.com/why-predictive-power-often-lives-in-the-tails/) covered when to leave tails alone (high TCR features) and when to compress them (low TCR features). CMMA on equity indices has TCR in the 0.45 to 0.60 range, which puts it in the "preserve partial tail gradient" bucket. A wide-scale sigmoid (c = 0.5 to 0.8) is the right compression; a narrow-scale sigmoid (c ≥ 1.5) flattens too much of the tail and removes the mean-reversion signal that lives there. For tree-based models, the compression is unnecessary. CMMA is fed raw into the tree and the splits handle the tails by leaf weights. ## Worked example: CMMA versus alternatives on SPX SPX daily, 1990 to 2026\. Same numerator base (current price relative to its recent average), four denominator constructions. Train 1990 to 2010, test 2010 to 2026. $$ \\begin{array}{l|c|c|c|c} \\text{Feature} & \\text{Train std} & \\text{Test std} & \\text{ratio test/train} & I(X;Y)\\;\\text{(bits} \\times 10^3\\text{)} \\\\ \\hline C\_t - \\text{MA}\_{50}(C) & 18.4 & 96.7 & 5.25 & 1.4 \\\\ (C\_t - \\text{MA}\_{50}(C)) / \\text{std}\_{100}(C) & 0.81 & 1.83 & 2.26 & 1.7 \\\\ (\\ln C\_t - \\text{MA}\_{50}(\\ln C)) / \\text{ATR}\_{100}(\\ln C) & 0.94 & 0.98 & 1.04 & 2.0 \\\\ \\text{Full CMMA}(50) \\text{ with } \\sqrt{51}\\text{ divisor} & 0.131 & 0.135 & 1.03 & 2.0 \\\\ \\text{CMMA}(50) \\text{ with sigmoid } c = 0.7 & 0.118 & 0.123 & 1.04 & 2.0 \\\\ \\end{array} $$ Five readings. Raw Close − MA(50) has a 5.25× ratio between test and train standard deviation. The model trained on 1990-to-2010 sees feature values in 2010-to-2026 that are five times the scale it ever saw in training. The retained MI (1.4) is the lowest in the table because the model spends capacity learning the time-varying meaning. Dividing the same numerator by the rolling 100-day standard deviation of close (the naive normalization) cuts the ratio to 2.26 and lifts the MI to 1.7\. The denominator now responds to volatility but lags and misses gap variance. The improvement is real but partial. Replacing the numerator with log-price deviation and the denominator with ATR(log close, 100 bars) cuts the ratio to 1.04 and lifts the MI to 2.0\. The feature is now regime-invariant. This is the result from the article ["Why ATR Normalization Is More Than a Volatility Trick"](https://aligrithm.com/why-atr-normalization-is-more-than-a-volatility-trick/) applied to the close-minus-MA primitive. Adding the √51 divisor (the full CMMA construction) does not change the ratio or the MI. The √(k+1) divisor is a rescaling that does not affect the per-feature statistics on its own. The MI is unchanged because the model sees the same information. The reason to keep the √(k+1) divisor is the cross-lookback comparability: when CMMA(10), CMMA(50), and CMMA(200) are all in the feature set, their distributions need to be on the same axis, and the √(k+1) is what does it. Adding the sigmoid compression with c = 0.7 slightly reduces the standard deviation (the tails are pulled in toward the linear region) and preserves the MI. The compression is operationally helpful for sigmoid-activation networks and operationally neutral for trees. The MI does not move because c = 0.7 is in the "preserve partial tail gradient" range. ## Cross-instrument pooling The same CMMA(50) computed on SPX, AAPL, and BTC produces three features on the same axis. The conditional distribution of next-day log return at CMMA = −1 looks similar on all three instruments (mean reversion edge of approximately 0.05 to 0.08 standard deviations in the direction opposite to the deviation, with substantial noise). The conditional distribution at CMMA = +1 is the mirror, with the equity drift adding a small positive bias on SPX and AAPL that is not present on BTC. The pooling property is the operational payoff. A panel regression on CMMA(50) across 500 US equities + 30 ETFs + 10 crypto perps can be fit with a single coefficient on CMMA, not 540 instrument-specific coefficients. The cross-sectional variance comes from the differences in the conditional distributions, not from the differences in the feature's scale. The article on cross-sectional rank features in the systems pillar covers the alternative approach: rank the CMMA value within the universe at each timestamp. Cross-sectional ranks discard absolute magnitude in exchange for distributional invariance across days. CMMA already gives the distributional invariance, so the rank is a free option that can be added when the downstream model benefits from a uniform marginal. ## What this changes in practice Three operational shifts. The feature library replaces Close − MA(k) with CMMA(k, W) as the default mean-reversion primitive. The two parameters (lookback k for the MA, ATR window W) are explicit in the feature name. Sigmoid compression is a third optional parameter with a default of "none" for tree models and c = 0.7 for activation-bounded networks. The MA is computed over (t−k, t−1) and the ATR is computed over (t−W+1, t). The asymmetric inclusion rule is documented in the feature library and is part of the construction's correctness, not a style choice. Cross-lookback feature sets (CMMA(10), CMMA(20), CMMA(50), CMMA(200)) are stored as four columns with the √(k+1) divisor applied to each. The √(k+1) is what makes them pool-compatible across lookbacks. A model that consumes CMMA at four lookbacks without the √(k+1) divisor learns four different scalings and uses model capacity to do work the construction should have done. ## Visualizing the CMMA stack ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/05/article_34-visual-0.png) The four panels are the stepwise construction. Each panel adds one ingredient and the diagnostic stamp shows what that ingredient bought. ## KEY POINTS - CMMA fixes three structural failures of plain Close − MA: price-level multiplier (1995 vs 2025 SPX), volatility regime (2017 vs 2020 SPX), and lookback dependence (MA(50) vs MA(200)). All three are killed in the construction, not in the model. - The four ingredients: log close in the numerator, MA over previous bars only, ATR with current bar included, √(k+1) divisor for lookback invariance. - Log close removes the price-level multiplier and gives cross-instrument scale comparability. Raw close gives a feature whose meaning is time-varying. - MA over previous bars only avoids attenuating the signal by k/(k+1) and enforces the timing-discipline convention that statistics used at time t live in (−∞, t−1\]. - ATR includes the current bar deliberately. The MA exclusion and the ATR inclusion are asymmetric because the MA is in the numerator (signal) and the ATR is in the denominator (scale). - The √(k+1) divisor cancels the lookback dependence under the random walk model. CMMA(10), CMMA(50), and CMMA(200) all live on the same axis after the divisor. - Optional sigmoid compression with c ≈ 0.7 keeps the bulk of CMMA in the linear region of the sigmoid and preserves the partial-tail-gradient signal. Trees skip the compression; bounded-activation networks use it. - On SPX, plain Close − MA(50) has a test/train standard deviation ratio of 5.25 and retained MI of 1.4\. Full CMMA(50) has a ratio of 1.03 and retained MI of 2.0\. The √(k+1) and the sigmoid do not change the per-feature MI but provide cross-lookback comparability and optional compression respectively. - CMMA is cross-instrument pool-compatible by construction. The same CMMA(50) on SPX, AAPL, and BTC produces conditional distributions of forward return that look similar in shape at matched feature values. A single coefficient suffices in a pooled regression. - The asymmetric inclusion rule (MA excludes current bar, ATR includes current bar) is documented in the feature library. Reversing the convention silently attenuates the signal and under-normalizes on regime-shift days. - The default mean-reversion primitive in the feature library is CMMA(k, W), not Close − MA(k). Plain Close − MA(k) is deprecated. The deprecation is a feature-library decision, not a modeling decision. - For tree models, raw CMMA is fed in. For sigmoid-activation networks, CMMA with c ≈ 0.7 compression. For linear models, CMMA without compression because linear models are not bounded. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Statistically Sound Indicators for Financial Market Prediction - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Statistically+Sound+Indicators+for+Financial+Market+Prediction+Timothy+Masters&ref=aligrithm.com) - [Cycle Analytics for Traders - John Ehlers (Amazon)](https://www.amazon.com/s?k=Cycle+Analytics+for+Traders+John+Ehlers&ref=aligrithm.com) ### 2.11 Why ATR Normalization Is More Than a Volatility Trick URL: https://aligrithm.com/why-atr-normalization-is-more-than-a-volatility-trick/ Last updated: 2026-07-27T13:23:39.000Z Compute 20-day SPX momentum as P\_t − P\_{t−20}. Plot its distribution in two slices: calendar 2017 (low vol, realized 7%) and calendar 2020 (high vol, realized 34%). The two histograms barely overlap. The 2020 standard deviation of the momentum feature is roughly five times the 2017 standard deviation. A model trained on 2017 sees 2020 as a continuous out-of-distribution event. Now divide the same momentum by 100-day ATR. The 2017 and 2020 histograms align almost on top of each other. The means sit within 0.05 of each other; the standard deviations differ by less than 15%. The model that failed on raw 20-day momentum sees 2017 and 2020 as the same distribution after the ATR division. The prior article in this series (["How to Build Stationary Indicators from Non-Stationary Prices"](https://aligrithm.com/how-to-build-stationary-indicators-from-non-stationary-prices/)) ranked ATR normalization as transform 5 of 6 and deferred the structural argument. This article makes the argument. ATR is not a generic volatility number that happens to work as a denominator. It is the structurally correct denominator for any price-unit numerator on a price series whose noise process scales with its own absolute volatility. Four properties, one piece of random-walk math, and one comparison table separate it from alternatives. The applied primitive that follows from this argument (Close-minus-Moving-Average divided by ATR) is the subject of the next article in this series (["CMMA: A Better Momentum Primitive Than Price-minus-MA Alone"](https://aligrithm.com/cmma-a-better-momentum-primitive-than-price-minus-ma-alone/)). The structural justification lives here. ## Four properties ATR has that plain volatility does not ATR\_W(P)\_t is the mean of true range over a lookback W: $$ \\text{TR}\_t \\;=\\; \\max\\bigl(H\_t - L\_t,\\;|H\_t - C\_{t-1}|,\\;|L\_t - C\_{t-1}|\\bigr), \\qquad \\text{ATR}\_W(P)\_t \\;=\\; \\frac{1}{W}\\sum\_{i=1}^{W}\\text{TR}\_{t-i} $$ Property 1: ATR captures within-bar and between-bar movement. The three branches of the true range definition cover three distinct events. (H − L) is the intra-bar range and dominates on continuous-trading days. |H − C\_{prev}| and |L − C\_{prev}| cover overnight gaps and weekend gaps. A close-to-close standard deviation misses the gap component, which on US equities is 30% of total daily variance and on futures rolls is structurally larger. The closing-price std treats Friday-to-Monday as a single price change; ATR records the gap and the intraday range as separate contributions. Property 2: ATR is in the price units of the instrument. SPX ATR is in index points. AAPL ATR is in dollars. BTC ATR is in dollars per coin. Any numerator built from the same instrument's prices is in the same units. The ratio is unitless by construction, which is what makes ATR-normalized features cross-instrument comparable. A 2-ATR momentum on SPX and a 2-ATR momentum on AAPL are the same number relative to each instrument's own typical move. A "2 standard deviations of close" on SPX and a "2 standard deviations of close" on AAPL are not. Property 3: ATR is tail-resistant without tail-killing. The mean of true range over W bars (W = 50 to 250 for daily) gets contributions from every bar in the window. A single 10-σ day adds its true range to the mean once, weighted at 1/W. A rolling std with the same window squares the deviation before averaging, which gives the single day an order-of-magnitude larger contribution. The article ["Why Predictive Power Often Lives in the Tails"](https://aligrithm.com/why-predictive-power-often-lives-in-the-tails/) covered the asymmetric cost of squashing tails. ATR's L1-style averaging keeps the tail event in the denominator at a proportional weight, which is the right weight for normalizing a feature that has its own tail content to preserve. Property 4: ATR scales with the same generative noise that produces the numerator. The next section makes this property precise. The short statement: if the numerator is a price difference whose typical magnitude scales with the local volatility σ, then dividing by ATR (a proxy for σ in price units) produces a ratio whose typical magnitude is constant in σ. The variance regime cancels structurally, not by tuning. ## The random walk math Treat the log-price as a random walk with time-varying per-bar volatility σ\_t. The k-bar price difference over a window where σ is approximately constant has standard deviation that scales as σ × √k: $$ \\text{Var}(P\_t - P\_{t-k}) \\;\\approx\\; k\\,\\sigma\_t^2 \\;\\;\\Longrightarrow\\;\\; \\text{std}(P\_t - P\_{t-k}) \\;\\approx\\; \\sigma\_t \\sqrt{k} $$ ATR over a long window is approximately a constant multiple of σ\_t in the price units of the instrument (the constant depends on the distribution of intraday paths but is stable across regimes for any given instrument). So dividing the k-bar price difference by ATR alone leaves a residual √k scaling: $$ \\frac{P\_t - P\_{t-k}}{\\text{ATR}\_W(P)\_t} \\;\\sim\\; \\frac{\\sigma\_t \\sqrt{k}}{c\\,\\sigma\_t} \\;=\\; \\frac{\\sqrt{k}}{c} $$ The σ\_t cancels (the volatility regime is removed) but the √k remains. The standard deviation of the normalized feature still depends on the lookback k. Two ATR-normalized momentum features computed with k = 5 and k = 20 are not on the same scale, even though both are regime-invariant. The full correction (the CMMA form) divides by ATR × √k: $$ M^{\\text{atr},\\sqrt{k}}\_t \\;=\\; \\frac{P\_t - P\_{t-k}}{\\text{ATR}\_W(P)\_t \\cdot \\sqrt{k}} $$ After the √k correction, the standard deviation of the feature is approximately constant across both the volatility regime and the lookback choice. The same lookback-invariance result is what makes the Sharpe ratio annualizable: annualized Sharpe = (daily Sharpe) × √252 is the same √k identity in a different application. The CMMA construction inherits this property and is the subject of the next article. ## Worked comparison: SPX 20-day momentum across normalizations SPX daily, 1990 to 2026\. Compute the same numerator (P\_t − P\_{t−20}) and divide by five different denominators. Report the standard deviation of the resulting feature in two volatility regimes (2017 calm, 2020 stressed) and the retained mutual information against y\_t = sign(P\_{t+1}/P\_t − 1). $$ \\begin{array}{l|c|c|c|c} \\text{Denominator} & \\text{std (2017)} & \\text{std (2020)} & \\text{ratio 2020/2017} & I(X;Y)\\;\\text{(bits} \\times 10^3\\text{)} \\\\ \\hline \\text{none (raw } P - P\_{-20}\\text{)} & 35 & 175 & 5.0 & 1.6 \\\\ \\text{percent change } (P/P\_{-20} - 1) & 0.014 & 0.063 & 4.5 & 1.7 \\\\ \\text{rolling std}(C)\\;\\text{over 100d} & 0.91 & 2.20 & 2.4 & 1.9 \\\\ \\text{ATR}\_{100}\\;\\text{(no }\\sqrt{k}\\text{)} & 1.12 & 1.18 & 1.05 & 2.1 \\\\ \\text{ATR}\_{100} \\cdot \\sqrt{20} & 0.25 & 0.26 & 1.04 & 2.1 \\\\ \\end{array} $$ Five readings. Raw momentum is calibrated to one regime and broken in the other. The 5× ratio between 2017 and 2020 standard deviations is the volatility regime contaminating the feature. Percent change reduces the ratio from 5.0 to 4.5\. The proportional rescaling assumption (a 1% move in 2017 is the same kind of event as a 1% move in 2020) is only half-correct on indices, because realized volatility is not a constant multiple of price level. Percent change works on individual stocks across decades better than it works on indices across regimes, but it does not remove regime contamination. Rolling std of close cuts the ratio to 2.4 and lifts the MI to 1.9\. The denominator now responds to volatility but lags. Close-to-close std misses overnight gaps, and the 100-day window is slower than the regime change. The 2020 ratio is still wrong by a factor of 2.4. ATR over 100 days (no √k correction) cuts the ratio to 1.05 and lifts the MI to 2.1\. The regime is structurally removed. The remaining 5% difference between 2017 and 2020 is the higher-moment mismatch that no scale normalization fixes. ATR × √20 produces the same ratio (1.04) and the same MI (2.1) but rescales the feature so it is on the same axis as a 5-day or a 60-day ATR-normalized momentum. The √k correction does not affect the regime-invariance result; it only standardizes the feature across lookbacks. The MI numbers in the table tell the second story. The raw and percent-change versions lose MI relative to the ATR versions because the model is wasting capacity learning a piecewise-linear relationship between feature value and regime, when the ATR-normalized version puts both regimes on the same axis and the model gets to learn one mapping instead of two. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## What ATR replaces Three specific failure modes that other denominators have and ATR does not. Close-to-close standard deviation misses overnight gaps. On a futures roll, on an earnings overnight, on a weekend Fed announcement, the gap is a structural part of the bar's information content and a structural part of the noise process. ATR captures it through max(|H − C\_{prev}|, |L − C\_{prev}|). Std of close ignores it. The article ["Why Predictive Power Often Lives in the Tails"](https://aligrithm.com/why-predictive-power-often-lives-in-the-tails/) is relevant here: gaps are the right-tail events that carry the signal, and a denominator that misses them rescales them too aggressively. Percent change assumes proportional dynamics. The model that says a 1% move on SPX in 2025 is the same kind of event as a 1% move on SPX in 1995 is wrong on indices because the equity premium is not a constant multiplier of price level. Implied vol is mean-reverting around its own level, not around a constant times the price level. ATR captures the dynamics in absolute price units, which is what realized vol actually does. Rolling IQR of close is outlier-robust but slow. The 100-day IQR of close is dominated by the central 50% of the window. On a volatility regime change, the IQR takes the full window to absorb the shift. ATR with the same window weights every bar at 1/W, so a 20-day cluster of high-true-range bars moves the ATR by 20/100 = 20% of the cluster's average TR. The IQR moves only when enough of the new regime occupies the central 50% of the sorted window, which can take 50+ bars. ATR is the right speed for normalizing features whose own lookback is 5 to 60 days. ## The window choice Two windows control the construction: the numerator's lookback k and the ATR's lookback W. The constraint is W much larger than k. Specific guidance. W in the range 50 to 250 days for daily data. W = 100 is the default for k ≤ 30\. W = 250 for k ≤ 60\. The reason is regime persistence: volatility regimes on equity indices last 60 to 250 bars on average. The ATR window should be long enough to average across a regime, not to react to it. An ATR\_20 normalization absorbs the regime change the numerator is trying to measure. W must not include the current bar. ATR\_W(P)\_t should be computed from bars t−1, t−2, ..., t−W. Including bar t couples the numerator's volatility to its own denominator and creates a small but real look-ahead. The bias is tiny per bar but compounds over 250-day backtests into a Sharpe inflation of 0.05 to 0.10 on momentum strategies. For futures with overnight sessions, ATR should be computed on session bars (not 24-hour bars). The H − C\_{prev} term should reflect the gap from the previous regular session close, not from a thin overnight close. The article on session boundaries (covered in the systems pillar) details the construction. For instruments with structural daily gaps (single-listed stocks, ETFs not trading overnight), ATR is the right denominator. For 24-hour instruments (BTC, FX, perpetual futures), the H − C\_{prev} terms collapse to (H − L) on a continuous-time bar, and ATR effectively becomes the average bar range. The construction still works; the gap contribution is just smaller. The cross-instrument comparability survives. ## The cross-instrument property A consequence of ATR being in the instrument's own price units: an ATR-normalized feature is a single column in a cross-sectional model. A 2-ATR move on SPX (≈ 110 points at current levels) is the same conditional event as a 2-ATR move on AAPL (≈ 9 dollars) and a 2-ATR move on BTC (≈ 4000 dollars). Each is a "two-typical-bar magnitude" event in the instrument's own scale. Pooling them in a panel regression or a cross-sectional ranking is statistically coherent because the feature is comparable by construction. Compare to the alternatives. Pooling raw close prices is incoherent (different instruments live on different scales). Pooling percent changes is coherent for stable-multiplier instruments but biased on instruments whose volatility decoupled from price level (commodities, vol-of-vol products). Pooling rolling-std-normalized closes is coherent but introduces the close-to-close gap-blindness for all instruments simultaneously. ATR-normalization is the only denominator that produces cross-instrument coherence without missing structural information on any of the instruments in the pool. ## Visualizing the regime collapse ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/05/article_33-visual-0.png) The two panels are the central diagnostic. The same numerator, two denominators, two completely different stories about whether the feature is regime-invariant. ## What this changes in practice Three operational shifts. Every price-unit feature carries an ATR-normalized variant in the feature library, with the ATR window stored as part of the feature identifier. "mom\_20d\_atr100" and "mom\_20d\_atr250" are different features. The √k correction is stored as a separate transform tag. The feature audit pipeline runs the regime-invariance check by default. Split the sample by realized-volatility quartiles (not by calendar date) and compute the feature standard deviation in each quartile. A feature whose Q4 std exceeds the Q1 std by more than a factor of 1.5 is flagged as regime-dependent. ATR-normalized features pass this check by construction. Std-of-close-normalized features pass it weakly. Raw and percent-change features fail. The cross-instrument pool is built from ATR-normalized features and from log returns. Anything else requires per-instrument rescaling at the model layer, which is the layer that the feature library is supposed to remove the burden from. ## KEY POINTS - ATR is not a generic volatility number. It is the structurally correct denominator for price-unit numerators because it (a) captures within-bar and between-bar movement, (b) is in the instrument's own price units, (c) is tail-resistant without tail-killing, and (d) scales with the same generative noise that produces the numerator. - The random walk variance identity: std(P\_t − P\_{t−k}) ≈ σ\_t × √k. Dividing by ATR removes σ\_t. Dividing by ATR × √k removes both σ\_t and the lookback dependence. - Close-to-close standard deviation misses overnight gaps. On US equities, gaps are 30% of total daily variance. A std-of-close denominator under-normalizes gap days and over-normalizes the rest. - Percent change assumes the volatility is a constant multiple of price level. Equity indices violate this assumption because implied vol mean-reverts around its own level, not around a constant times price. Percent change does not remove regime contamination on indices. - Rolling IQR of close is outlier-robust but reacts too slowly to regime changes. The IQR moves only when enough of the new regime occupies the central 50% of the sorted window, which can take 50+ bars. ATR moves at 1/W per new bar. - The ATR window W must be much larger than the numerator's lookback k. W in the range 50 to 250 days, k typically 5 to 60\. W = 100 is the default for k ≤ 30. - ATR must be computed from bars strictly before the numerator's timestamp. Including the current bar in the ATR window introduces a small look-ahead that compounds into a Sharpe inflation of 0.05 to 0.10 on momentum backtests. - On SPX 20-day momentum, the ratio of 2020 standard deviation to 2017 standard deviation is 5.0 for raw, 4.5 for percent change, 2.4 for std-of-close, and 1.05 for ATR. The retained mutual information against next-day return sign is highest for the ATR-normalized version. - ATR-normalized features are cross-instrument comparable by construction. A 2-ATR move on SPX, AAPL, and BTC is the same conditional event in each instrument's own scale. Pooling raw or percent-change features across instruments is biased; pooling ATR-normalized features is coherent. - The √k correction does not affect regime-invariance. It only standardizes the feature across lookbacks, which is the property that makes ATR-normalized momenta with different k values pool-compatible. - The applied primitive built on this construction (Close minus Moving Average divided by ATR × √k) is the canonical CMMA indicator. The next article in this series unpacks why it outperforms plain Price minus MA. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Statistically Sound Indicators for Financial Market Prediction - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Statistically+Sound+Indicators+for+Financial+Market+Prediction+Timothy+Masters&ref=aligrithm.com) - [Cycle Analytics for Traders - John Ehlers (Amazon)](https://www.amazon.com/s?k=Cycle+Analytics+for+Traders+John+Ehlers&ref=aligrithm.com) ### 2.10 How to Build Stationary Indicators from Non-Stationary Prices URL: https://aligrithm.com/how-to-build-stationary-indicators-from-non-stationary-prices/ Last updated: 2026-07-27T13:23:38.000Z The prior article in this series (["The Case Against Raw Price Indicators"](https://aligrithm.com/the-case-against-raw-price-indicators/)) rejected raw close, raw moving averages, and raw band envelopes at the feature-design stage. The argument was structural: non-stationary mean, non-stationary variance, no cross-instrument comparability, and a guaranteed extrapolation failure once live prices leave the training support. That article ended at the diagnostic. This one is the recipe. Six transforms, in increasing order of how aggressively they kill non-stationarity, with the explicit trade that each transform discards information about the raw price level along with the non-stationarity it removes. Pick the lightest transform that passes the diagnostics, not the heaviest one. The ATR-normalization case (transform 5 below) is covered in depth in the next article in this series (["Why ATR Normalization Is More Than a Volatility Trick"](https://aligrithm.com/why-atr-normalization-is-more-than-a-volatility-trick/)). The summary appears here; the structural argument lives there. ## The stationarity-information tradeoff Every transform that removes non-stationarity from a price series discards some part of the raw level information that the original series carried. The tradeoff is structural and unavoidable. The ranking by aggressiveness, from lightest to heaviest: $$ \\begin{array}{l|c|c|l} \\text{Transform} & \\text{Removes mean drift} & \\text{Removes variance drift} & \\text{Information discarded} \\\\ \\hline \\text{Log return } r\_t = \\ln(P\_t / P\_{t-1}) & \\text{yes} & \\text{partial} & \\text{absolute level} \\\\ \\text{Oscillator } X\_t - X\_{t-k} & \\text{yes (if } k \\text{ moderate)} & \\text{partial} & \\text{slow-trend level} \\\\ \\text{Same-bar OHLC diff} & \\text{yes (by construction)} & \\text{no} & \\text{intra-bar absolute scale} \\\\ \\text{Direct variance scaling: } X\_t / \\text{IQR}\_t & \\text{no} & \\text{yes} & \\text{absolute magnitude} \\\\ \\text{ATR-normalized diff: } (X\_t - X\_{t-k}) / \\text{ATR}\_t & \\text{yes} & \\text{yes} & \\text{level and absolute magnitude} \\\\ \\text{Forced centering: } X\_t - \\text{median}\_{t,W} & \\text{yes (aggressive)} & \\text{no} & \\text{the full level, including useful drift} \\\\ \\end{array} $$ The right transform for a given indicator is the lightest one that produces a series passing the three diagnostics from the prior article: ADF p ≤ 0.05, train-test feature-range coverage ≥ 95%, and rolling 252-bar mean drift below one in-sample standard deviation. ## Transform 1: Log returns and first differencing The default starting point. r\_t = ln(P\_t / P\_{t-1}) converts a non-stationary price series into a stationary return series for all practical equity, FX, and most futures markets. Two structural properties matter. The log return is additive across time: ln(P\_t / P\_{t-k}) = sum of the k single-step log returns. The k-step return and the 1-step return live in the same units and can be modeled jointly. The simple percentage return P\_t / P\_{t-1} − 1 does not have this additivity and breaks for any multi-step compounded analysis. Log returns absorb the proportional structure of price moves. A $5 move on a $50 stock and a $50 move on a $500 stock produce the same log return. A pooled cross-sectional model on log returns has comparable features across instruments without further normalization. The article ["The Case Against Raw Price Indicators"](https://aligrithm.com/the-case-against-raw-price-indicators/) covered the cross-instrument incomparability of raw prices; log returns remove it at the construction step. The residual non-stationarity in log returns is variance, not mean. Equity index log returns have stationary mean (close to zero) but non-stationary variance (volatility clustering, regime shifts). The ADF test on SPX daily log returns passes by a wide margin. The variance ratio test detects the variance non-stationarity, which is the cue to apply a variance-scaling transform on top. ## Transform 2: Oscillators built from two prices The construction subtracts two quantities that share the same drift, so the difference is stationary in mean by construction. $$ O\_t \\;=\\; \\text{MA}\_n(P\_t) \\;-\\; \\text{MA}\_m(P\_t), \\qquad n < m $$ The fast moving average MA\_n tracks the recent price level. The slow moving average MA\_m tracks the slower-moving level. The drift in P\_t enters both quantities at almost the same rate, so the subtraction nets it out. If the mean drifts up or down, the act of subtraction cancels out most of the mean drift. Two construction details determine whether the oscillator is stationary. The lag separation should be moderate. n = 10 and m = 50 produces a small enough gap that the two MAs share most of their drift. n = 2 and m = 200 produces a gap large enough that the long MA is structurally lagging the short MA by 100 bars on average, and the difference accumulates the trend over that span. The "large difference of lags" case fails the ADF test on trending instruments. The same-indicator-with-lag variant works similarly. RSI(14)*t − RSI(14)*{t−14} is stationary because RSI is bounded and the subtraction removes any slow drift in the rolling RSI mean. This construction discards the absolute RSI level (whether the market is in an overbought regime overall) but keeps the change in regime, which is usually the predictive part. The residual non-stationarity is variance. The oscillator's variance scales with the variance of the underlying price moves, so volatility regimes leak through. Apply transform 4 or 5 on top. ## Transform 3: Same-bar OHLC differences The construction takes two prices recorded on the same bar (same minute, same day), so the drift between them is structurally zero. $$ \\text{Range}\_t = H\_t - L\_t, \\quad \\text{UpperShadow}\_t = H\_t - \\max(O\_t, C\_t), \\quad \\text{Body}\_t = |C\_t - O\_t| $$ The same-bar drift between H and L is zero by construction. There is no calendar offset between the two quantities. The difference has no mean drift regardless of how the underlying price level evolves. The variance is still non-stationary because the absolute range scales with the price level. A $5 daily range on a $50 stock is large; a $5 daily range on a $500 stock is small. The fix is to normalize by either the bar's own midpoint (Range\_t / ((H\_t + L\_t) / 2)) or by an external scale (ATR over a long window). The normalized same-bar OHLC differences are some of the most stationary primitives available: Body / Range, UpperShadow / Range, Range / ATR. All three are bounded, drift-free, and comparable across instruments. The Stochastic K indicator is a same-bar OHLC primitive at heart: (Close − Low\_n) / (High\_n − Low\_n) over a lookback n. The reason Stochastic K is stationary is the same reason any same-bar OHLC ratio is stationary. ## Transform 4: Direct variance scaling The construction divides the indicator by a rolling measure of its own variation. The mean stays where it was; the variance is forced toward unity. $$ \\tilde{X}\_t \\;=\\; \\frac{X\_t}{\\sqrt{\\text{IQR}\_{W}(X)\_t}} \\quad \\text{or} \\quad \\frac{X\_t}{\\text{std}\_W(X)\_t} $$ The lookback W must be much longer than any indicator lookback used to construct X. W = 252 days when X is a 14-day RSI or a 20-day momentum. Shorter W means the scaling absorbs the variance change you are trying to detect; longer W means the scaling lags the regime. IQR is preferred to standard deviation when the indicator has tails. The article ["Range/IQR: A Simple Test for Indicator Tail Problems"](https://aligrithm.com/range-iqr-a-simple-test-for-indicator-tail-problems/) covered the rationale. A single outlier in the lookback window inflates the std and silently compresses every other observation by the wrong factor. The IQR is immune to single-bar outliers and produces a scale that reflects the typical, not the extreme, indicator behavior. Direct variance scaling does not remove mean drift. It only stabilizes the variance. A non-stationary-mean indicator divided by its rolling IQR is still non-stationary in mean. Stack transform 4 on top of transform 1, 2, or 3 (which handled the mean), never as a standalone. ## Transform 5: ATR-normalized differences The construction divides a price difference by the ATR over a longer window. ATR captures absolute volatility in price units, which is the natural denominator for any price-unit numerator. $$ M^{\\text{atr}}\_t \\;=\\; \\frac{P\_t - P\_{t-k}}{\\text{ATR}\_W(P)\_t}, \\qquad W \\gg k $$ The numerator is a same-instrument price difference, which already removes part of the drift over a short k. The denominator is the typical price-unit volatility over a long window, which scales the difference into a unitless quantity comparable across instruments and across volatility regimes. The choice of W (the ATR window) and k (the difference window) controls the stationarity-information tradeoff. Long W (252 days), moderate k (20 days) gives high stationarity and retains a 20-day momentum signal. Short W (20 days) is too reactive: the denominator moves with the numerator and the ratio drifts toward zero in trending regimes. The deeper structural reason ATR normalization works on price-unit numerators is the subject of the next article, ["Why ATR Normalization Is More Than a Volatility Trick"](https://aligrithm.com/why-atr-normalization-is-more-than-a-volatility-trick/). The condensed argument: ATR captures the same generative noise process whose variance is non-stationary, so dividing by it gives a ratio whose variance is approximately stationary by construction, not by luck. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Transform 6: Forced centering The heaviest transform. Subtract a rolling median or rolling mean from the indicator, leaving only the deviation from the local center. $$ \\hat{X}\_t \\;=\\; X\_t - \\text{median}\_{W}(X)\_t $$ The result is stationary in mean by construction: the rolling median is, by definition, the local center, and subtracting it produces a series with zero rolling mean. The variance is not necessarily stabilized. The information cost is the largest of any transform in this list. The indicator's meaning changes from "what is the value of X right now" to "how far is X from where it has been recently." On indicators where the absolute value carries the signal (RSI levels, distance from cost basis, drawdown depth), forced centering discards the part you wanted to keep. Use forced centering only when transforms 1 through 5 fail to produce a stationary result and the absolute level is known to be uninformative on structural grounds. If an indicator is so non-stationary that you feel compelled to force it into stationarity, the underlying construction is almost certainly the problem. Forced centering is a last resort, not a default. ## The two-step protocol Stationarity comes in two flavors. Treat them in order. Step 1: induce mean stationarity. Apply transform 1, 2, or 6 depending on the indicator. Re-run the ADF test. If ADF p ≤ 0.05, the mean is stationary. If not, the transform is too light for this series; escalate. Step 2: induce variance stationarity. Apply transform 4 or 5 to the mean-stationary output of step 1\. Re-run a rolling-variance test. If the rolling 252-bar variance ranges within a factor of 2 across the sample, the variance is stationary enough. If not, the variance scaling window may be too long; consider a regime-aware filter (covered in the systems pillar). Skipping step 1 and going straight to variance scaling produces a series with stationary variance and a still-drifting mean, which the ADF test detects. Skipping step 2 and going straight to mean-only transforms leaves a series with stable mean and volatility-regime-dependent variance, which a rolling variance test detects. Both failures are common and both are caught by the diagnostics, not by inspection. ## Worked example: SPX close through each transform SPX daily, 1990 to 2026\. Compute six candidate features from the raw close. Test each on ADF, on train-test feature-range coverage (1990-2010 train, 2010-2026 test), and on retained mutual information against y\_t = sign(P\_{t+1}/P\_t − 1). $$ \\begin{array}{l|c|c|c|c} \\text{Feature} & \\text{ADF } p & \\text{Coverage} & I(X;Y)\\;\\text{(bits} \\times 10^3\\text{)} & \\text{Verdict} \\\\ \\hline \\text{Raw } P\_t & 0.99 & 29\\% & - & \\text{reject (mean)} \\\\ \\text{Log return } r\_t & 0.00 & 100\\% & 0.8 & \\text{accept (light)} \\\\ \\text{20d momentum } (P\_t - P\_{t-20}) & 0.00 & 78\\% & 1.6 & \\text{accept (light)} \\\\ \\text{ATR-normalized 20d momentum} & 0.00 & 99\\% & 2.1 & \\text{accept (medium)} \\\\ \\text{20d MA diff } (\\text{MA}\_{10} - \\text{MA}\_{50}) & 0.00 & 81\\% & 1.4 & \\text{accept (light)} \\\\ \\text{ATR-normalized MA diff} & 0.00 & 99\\% & 1.8 & \\text{accept (medium)} \\\\ \\text{Forced-centered: } P\_t - \\text{median}\_{252}(P) & 0.00 & 96\\% & 0.9 & \\text{accept (heavy)} \\\\ \\end{array} $$ Five readings. Raw price fails ADF and fails coverage. The single column that the prior article rejected. No further work justifies keeping it. Log return passes everything but carries the smallest mutual information of the surviving features. It is the lightest possible transform and the cheapest baseline. Most pipelines should include it. The plain 20-day price difference has a coverage of 78%. The mean drift is removed but the variance non-stationarity leaks through: a 100-point SPX move in 1995 was a 6% move; a 100-point SPX move in 2025 is a 1.8% move. The difference's distribution shifts even though the mean is stationary. ATR normalization of the same 20-day momentum lifts coverage to 99% and lifts the retained MI to the highest in the table (2.1). The transform removed the variance non-stationarity that the plain difference left behind, and the retained MI confirms it did so without throwing out the signal. The next article unpacks why this construction is structural rather than a heuristic. Forced centering by the 252-day median gives high coverage and stationarity but the lowest MI of any accepted feature (0.9). The transform discarded the drift component that the model would have used. The fact that the diagnostic passes does not mean the transform was a good choice; it means it produced a stationary series, at the cost of the predictive content that the lighter transforms kept. ## The look-ahead anti-pattern Three patterns that produce a series that looks stationary on the training set and breaks live, because the transform itself used information from outside the training window. Full-sample z-scoring. Computing (X − μ) / σ with μ and σ taken from the entire history (train + test) leaks the test-set statistics into the training feature. The training-set distribution looks bounded; the live distribution has no equivalent bound because the live deployment cannot recompute μ and σ using future data. Use only the training window's μ and σ, and update them forward in the walk-forward. Forward-looking rolling statistics. Computing the rolling median or rolling ATR with a centered window (using bars after t to compute the statistic at t) leaks future information into the feature. The rolling window must end at t and look backward, not span t. The same applies to any normalization based on rolling statistics. In-sample re-fitting of the hedge ratio in cointegrated spreads. β in P\_1 − β P\_2 is a parameter, and re-fitting it on test data is the same kind of leakage. β is estimated once on the training window and held fixed through the test window. The cointegration article in the systems pillar covers the failure modes when this rule is broken. All three anti-patterns are caught by the same forward-window discipline: any statistic used to define a feature must be computable from data strictly before the feature's timestamp. ## Visualizing the transform stack ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/05/article_32-visual-0.png) The five panels are the visual diagnostic. The same SPX data, five different feature constructions, five different stationarity verdicts. The diagnostic chooses the lightest acceptable transform, which is usually a log return alone for simple momentum and an ATR-normalized difference for price-unit features. ## What this changes in practice Three operational shifts. Every feature carries a transform tag in the feature library. "raw\_close", "log\_return", "atr\_norm\_mom\_20d", "ma\_diff\_10\_50\_atr\_normalized". The tag is part of the feature's identity. Two features with the same numerator and different transforms are two different features and get logged separately. The feature audit pipeline runs in order: construct, ADF test, coverage check, rolling-variance check, retained-MI check against a baseline log return. A feature that passes the first three but loses MI relative to the baseline log return is a heavier transform than the data required, and the lighter transform replaces it. The cross-instrument pool is built from features whose construction makes them comparable across instruments. Log returns and ATR-normalized differences are pool-eligible by construction. Same-instrument oscillators with absolute units (MA\_10 − MA\_50 in price units) are not pool-eligible until they are normalized. The pool eligibility tag lives in the feature library next to the transform tag. ## KEY POINTS - Stationarizing a price-derived indicator costs information about the absolute level. The right transform is the lightest one that passes the diagnostics, not the heaviest. - Six transforms ranked by aggressiveness: log returns, oscillators of two prices, same-bar OHLC differences, direct variance scaling, ATR-normalized differences, forced centering by rolling median. - Log returns are the default baseline. ADF passes, coverage is near 100%, the construction absorbs proportional structure across instruments. Residual non-stationarity is variance only. - Oscillators built from two related prices (MA\_n − MA\_m, or X\_t − X\_{t-k}) are stationary in mean by construction when the lag separation is moderate. Large lag separations fail on trending instruments. - Same-bar OHLC differences (range, body, shadows) have structurally zero drift because both prices are recorded on the same bar. Variance still scales with price level and must be normalized separately. - Direct variance scaling divides by a long-window rolling IQR or standard deviation. IQR is preferred when the indicator has tails. The window must be much longer than the indicator's own lookback. - ATR normalization divides a price-unit difference by ATR over a long window. The result is unitless, comparable across instruments, and stationary in both mean and variance. The structural reason is the subject of the next article in this series. - Forced centering by a rolling median is the heaviest transform. Use only when lighter transforms fail and the absolute level is structurally uninformative. Most "needs forced centering" indicators are poorly designed. - Stationarize in two steps: mean first, variance second. Skipping the mean step leaves a drifting mean under stable variance. Skipping the variance step leaves volatility-regime contamination under a stable mean. - Three look-ahead anti-patterns: full-sample z-scoring (leaks test-set statistics), centered rolling windows (leaks future bars), in-sample re-fitting of cointegration hedge ratios. Every statistic used to define a feature must be computable from data strictly before the feature's timestamp. - On SPX 1990 to 2026, ATR-normalized 20-day momentum produces the highest retained MI (2.1 bits × 10^-3) of any tested transform, with 99% train-test coverage and ADF p = 0.00\. Plain 20-day momentum has stationary mean but 78% coverage from variance drift, and forced centering passes diagnostics but discards more than half the signal of the ATR-normalized version. - Every feature carries a transform tag and a pool-eligibility tag in the feature library. Two features with the same numerator and different transforms are different features and are logged separately. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Statistically Sound Indicators for Financial Market Prediction - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Statistically+Sound+Indicators+for+Financial+Market+Prediction+Timothy+Masters&ref=aligrithm.com) - [Cycle Analytics for Traders - John Ehlers (Amazon)](https://www.amazon.com/s?k=Cycle+Analytics+for+Traders+John+Ehlers&ref=aligrithm.com) ### 2.9 The Case Against Raw Price Indicators URL: https://aligrithm.com/the-case-against-raw-price-indicators/ Last updated: 2026-07-27T13:23:34.000Z # You train an XGBoost on SPX from 1990 to 2015 with a feature set that includes raw close price, raw 50-day moving average, and raw 200-day moving average. The model produces clean in-sample numbers and a respectable 2010 to 2015 walk-forward. Cross-validated AUC: 0.531\. You ship it. The model goes live in 2016 with SPX around 2000\. Through 2020 the index trades up to 3300\. Through 2024 it trades up to 5800\. By 2026 it sits above 6000\. The feature "raw close" has been outside the maximum value the model ever saw in training for 84% of the live period. The model is extrapolating on its primary input every day. The 2010 to 2015 walk-forward looked fine because SPX traded between 1100 and 2100 in that window, which sat inside the 1990 to 2015 training range. The walk-forward never tested the failure mode that the live deployment was guaranteed to hit. The model was not tested against the only thing that mattered: a price level outside the training support. This article catalogs why raw price as a feature is a structural mistake. The constructive transforms that replace it (log returns, ratios, oscillators, ATR-normalized differences) belong to the next article, ["How to Build Stationary Indicators from Non-Stationary Prices"](https://aligrithm.com/how-to-build-stationary-indicators-from-non-stationary-prices/). The job here is to name the failure modes so a researcher rejects raw price at the feature-design stage rather than after the model breaks in production. ## Three structural failures of raw price Raw price is non-stationary in mean. SPX in 1990 averaged around 330\. SPX in 2025 averages around 5500\. The unconditional mean of the feature is a function of the year. A model that learned "close = 1200 is a sell signal" learned a calendar fact, not a market fact. The same calendar fact says nothing about close = 5800 in 2026 because 2026 was not in the training set. Raw price is non-stationary in variance. The absolute dollar move per day scales with the price level. A one-percent SPX move in 1995 is 5 points. A one-percent SPX move in 2025 is 55 points. A feature like "close − close shifted 5" has a variance that grew by an order of magnitude over the sample. The model's split thresholds, distance metrics, and gradient magnitudes are all sized to the wrong variance for the live regime. Raw price is incomparable across instruments. SPX trades at 5500\. AAPL trades at 230\. BTC trades at 90000\. A feature defined as "close price" is not a single feature across a multi-instrument universe. It is one feature per instrument, plus an arbitrary scale factor between them. Cross-sectional models that treat raw price as a column cannot be trained at all on a pooled panel without an instrument-specific normalization, which is the same thing as not using raw price. Raw price encodes the calendar, not the market. A model whose top feature is the price level is, by construction, a model of when the data was collected. The article ["Stationarity: The Word Every Trader Ignores Until It Kills the Strategy"](https://aligrithm.com/stationarity-the-word-every-trader-ignores-until-it-kills-the-strategy-2/) (forthcoming in the systems pillar) covers the broader stationarity argument. The narrow point here is that raw price fails the stationarity test by construction, not by accident. ## The extrapolation trap Three model families, three failure modes, same root cause. Decision trees and gradient-boosted ensembles split on feature values. The split at "close ≤ 2100" partitions the training data into two non-empty leaves. The same split applied to a test set where every observation has close > 2100 sends 100% of the test data to one leaf. The model's behavior on the held-out side is the marginal prediction of that leaf, fitted on the training observations near the boundary. Outside the training support, every tree degenerates to the prediction of its nearest training leaf, regardless of whether that nearest leaf is statistically appropriate. Neural networks with bounded activations saturate. A close-price feature normalized by the training-set mean and standard deviation produces input values around 0 to 2 for the training years and input values around 5 to 10 for the live years. Tanh and sigmoid activations saturate above 3\. The network's first layer has identical activations for close = 4000 and close = 5500, because both inputs saturate at +1 after the activation. The model cannot distinguish them. Distance-based methods (kNN, kernel regression, RBF networks) compute distances in the raw-price space. The distance from a live observation at close = 5500 to the nearest training observation at close = 2100 is 3400 in feature units. Every nearest-neighbor query in the live deployment returns the same training point (the one with the highest close), and the prediction is the local average around that training point. The model has degenerated to a constant. Linear models extrapolate linearly. The fitted slope on "close" is determined by the in-sample relationship between price and the target. Extrapolating that slope by a factor of 3 in the live period multiplies the contribution of the feature by 3 in the model's output. If the slope was small and noisy in-sample, it dominates the prediction out-of-sample for the wrong reason. Linear models are the only family where extrapolation produces a non-constant prediction, and the non-constant prediction is the worst kind: it looks confident. ## Worked example: SPX raw close versus log return Single-feature XGBoost, SPX daily, train 1990 to 2010, test 2010 to 2026\. Target y\_t = sign(P\_{t+1}/P\_t − 1). Two models. Model A uses raw close as the only feature. Model B uses 1-day log return ln(P\_t / P\_{t−1}) as the only feature. Identical hyperparameters. $$ \\begin{array}{l|c|c|c|c} \\text{Feature} & \\text{Train AUC} & \\text{Test AUC} & \\% \\text{ test rows outside train range} & \\text{Verdict} \\\\ \\hline \\text{Raw close } P\_t & 0.582 & 0.498 & 71\\% & \\text{extrapolation} \\\\ \\text{Log return } \\ln(P\_t/P\_{t-1}) & 0.508 & 0.507 & 0.4\\% & \\text{stationary} \\\\ \\end{array} $$ Three readings. Raw close shows a strong in-sample AUC (0.582) and a test AUC indistinguishable from a coin flip (0.498). The 8.4-point gap is the extrapolation penalty. 71% of the test rows have a raw close value larger than the maximum close seen in training. The model's splits on the raw close feature are all calibrated to a price range the test set has left behind. Log return shows a weak in-sample AUC (0.508) and a test AUC almost identical to it (0.507). The feature has 0.4% of test rows outside the training range, which is the right-tail extreme moves from the COVID crash and the 2022 inflation regime. The model generalizes because the feature distribution is stable. The "raw close" model is not modeling SPX. It is modeling the year. The "log return" model is weakly modeling SPX. The weak model is the honest one. The strong model is the one that broke in production. Repeating the experiment with raw 50-day MA and raw 200-day MA reproduces the same result: large in-sample AUC, test AUC at the coin-flip line, and feature range coverage below 30%. Any feature that inherits the price level inherits the extrapolation failure. ## Three diagnostics that flag a raw-price contaminated feature Run these before the feature ships. Each one takes seconds on a daily-frequency series and rejects a feature that the backtest cannot. ADF test on the feature itself. The augmented Dickey-Fuller test rejects the null of a unit root for stationary series and fails to reject for non-stationary series. A feature with ADF p-value above 0.05 is non-stationary at conventional levels. Raw close on SPX has ADF p ≈ 0.99 (failing to reject the unit root by construction). A 14-day RSI has ADF p ≈ 0.00\. The ADF test does not distinguish between drift-stationary and trend-stationary processes for our purposes; a feature that fails ADF is rejected regardless of which non-stationarity is the cause. Train-test feature-range coverage. Compute the fraction of test-set rows whose feature value lies within \[min(feature in train), max(feature in train)\]. Coverage below 95% is a structural extrapolation problem the model has no defense against. Raw close on SPX over a 20-year split has coverage around 30%. Log return over the same split has coverage near 100%. This diagnostic is independent of the model and applies to every feature regardless of its construction. Rolling-window mean and variance drift. Compute the rolling 252-bar mean and variance of the feature across the sample. Plot both. A feature whose rolling mean drifts by more than 1 in-sample standard deviation across the sample, or whose rolling variance changes by more than a factor of 3, is non-stationary. Raw close fails both conditions on every long equity sample. Log returns fail only the variance condition during volatility regime shifts, which is the cue to apply the ATR normalization covered in the next article. A feature passes if all three diagnostics pass. A feature fails if any one of them fails, regardless of how good its in-sample AUC looked. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## When raw price appears to work and does not Four ways raw price sneaks into the model with apparent success, all of them bias dressed as signal. In-sample-only backtests. The train and test sets are drawn from the same overall period without a temporal split. The price level appears in both. The model learns the level. The "test" performance is a function of the calendar, not the feature. Walk-forward windows shorter than the regime length. A 5-year training window followed by a 1-year test window keeps the price range close enough between the two that the extrapolation penalty is small per window. The cumulative walk-forward across 20 years looks fine because each individual window had near-full coverage. The model fails the moment it is deployed live and asked to extrapolate beyond the most recent training window, which by definition is the first thing live trading does. Label leakage through normalization. Z-scoring the raw close using the full sample's mean and standard deviation, then splitting, hides the level in a feature that now looks bounded. The bound is a function of the full-sample statistics, which the live deployment does not have access to. The diagnostic: re-fit the normalization using only the training window's statistics, then re-evaluate. The performance collapse from the re-fit equals the magnitude of the leak. Spurious common-factor effects on cointegrated pairs. A pair trade with two raw price legs can look stable because the two non-stationary series are individually non-stationary but their linear combination is stationary. The model that takes both raw prices as features cannot tell that the stationarity lives in their difference, not in either one. Replacing the two raw prices with the spread (P\_1 − β P\_2) is one feature instead of two and the only stationary one. The article ["Stationarity: The Word Every Trader Ignores Until It Kills the Strategy"](https://aligrithm.com/stationarity-the-word-every-trader-ignores-until-it-kills-the-strategy-2/) covers the cointegration case in detail. ## Visualizing the extrapolation cliff ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/05/ChatGPT-Image-May-22--2026--05_09_37-PM.png) The two panels are the central diagnostic. Same underlying data, two feature constructions, two different stories about what the model is doing in production. ## What can stay raw Two narrow cases survive the case against raw price, both because the construction nets out the drift before it reaches the feature. Same-bar OHLC differences. (high − low), (high − close), (close − low) on the same bar are not raw prices. They are differences of two prices recorded within the same one-minute or one-day window. The drift between them is structurally zero. The variance scales with the price level (a $5 range on a $20 stock is different from a $5 range on a $500 stock), so the difference must still be normalized by a volatility scale (ATR, IQR, or the bar's own midpoint). The construction is covered in the next article. The relevant point here: same-bar OHLC differences are not "raw price" in the sense this article rejects. Cointegrated-pair spreads. P\_1 − β P\_2 for two cointegrated legs is stationary by construction, regardless of how non-stationary either leg is alone. The hedge ratio β must be estimated on training data and held fixed; re-fitting β on test data is the same kind of leakage that the normalization case above describes. Cointegrated spreads are an exception to the rule, not a relaxation of it. Every other raw price feature fails for the reasons in the diagnostic section. The constructive recipe lives in the article ["How to Build Stationary Indicators from Non-Stationary Prices"](https://aligrithm.com/how-to-build-stationary-indicators-from-non-stationary-prices/). ## What this changes in practice Three operational shifts. Raw close, raw OHLC, raw moving averages, and raw band envelopes (Bollinger middles, Donchian channels in price units) do not enter the feature set. The pre-modeling feature audit rejects any column whose ADF p-value is above 0.05, whose train-test coverage is below 95%, or whose rolling 252-bar mean drifts by more than one in-sample standard deviation. The audit runs before any model is trained. Every backtest reports the train-test feature-range coverage alongside the AUC. A model whose top-importance feature has coverage below 95% is flagged in the model registry as "extrapolation-bound" and does not deploy until the feature is replaced. The walk-forward window length is sized to the regime length, not to a default of 252 bars. Cross-instrument pooled models normalize at the feature definition, not at the model. A feature defined as "close" is not pooled across instruments. A feature defined as "1-day log return" is. The decision lives in the feature library, not in the modeling code, because the feature library is the contract that the modeling code consumes. --- ## KEY POINTS - Raw price is non-stationary in mean. SPX averaged 330 in 1990 and 5500 in 2025\. A model trained on the 1990 distribution has no support at the 2025 distribution. - Raw price is non-stationary in variance. The absolute dollar move per day scales with the level. Model split thresholds and gradient magnitudes calibrated to the 1990 variance are wrong for the 2025 variance by an order of magnitude. - Raw price is incomparable across instruments. SPX at 5500 and AAPL at 230 are not on the same scale. A cross-sectional model cannot pool raw close as a column without an instrument-specific normalization, which is the same as not using raw close. - Trees and gradient-boosted models extrapolate by sending all out-of-support observations to a single leaf. The prediction degenerates to the constant value of that leaf. - Neural networks with bounded activations saturate beyond a few in-sample standard deviations. Distinct large values become indistinguishable post-activation. - Distance-based methods compute distances in raw-price units. Out-of-support observations have the same nearest training neighbor (the largest training value), and the model collapses to a constant. - Linear models extrapolate linearly. A small noisy slope multiplied by a 3× feature value in the live period dominates the prediction for the wrong reason. - Three pre-modeling diagnostics flag raw-price contamination: ADF p-value above 0.05, train-test feature-range coverage below 95%, rolling 252-bar mean drift exceeding one in-sample standard deviation. A feature failing any one is rejected. - A walk-forward window shorter than the regime length masks the extrapolation problem in-sample. The cumulative walk-forward looks fine because each window has near-full coverage. The live deployment fails because the first live observation extrapolates beyond the last training window. - Z-scoring the raw close using the full sample's mean and standard deviation before splitting is label leakage. The bound is a function of statistics the live deployment cannot recompute. Re-fit normalization on training-window statistics only. - On SPX 1990 to 2010 train and 2010 to 2026 test, a single-feature XGBoost on raw close shows in-sample AUC 0.582 and test AUC 0.498\. The same model on 1-day log returns shows in-sample AUC 0.508 and test AUC 0.507\. The strong in-sample model is the one that broke; the weak in-sample model is the honest one. - Two narrow constructions can stay close to raw: same-bar OHLC differences (drift nets to zero by construction, must still be volatility-scaled) and cointegrated-pair spreads (stationary by construction, hedge ratio fixed from training). Both are exceptions, not relaxations. - The constructive transforms that replace raw price (log returns, log differences, ATR-normalized differences, oscillators of two prices) are covered in the article ["How to Build Stationary Indicators from Non-Stationary Prices"](https://aligrithm.com/how-to-build-stationary-indicators-from-non-stationary-prices/) --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Statistically Sound Indicators for Financial Market Prediction - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Statistically+Sound+Indicators+for+Financial+Market+Prediction+Timothy+Masters&ref=aligrithm.com) - [Cycle Analytics for Traders - John Ehlers (Amazon)](https://www.amazon.com/s?k=Cycle+Analytics+for+Traders+John+Ehlers&ref=aligrithm.com) ### 2.8 Why You Should Test Long and Short Thresholds Separately URL: https://aligrithm.com/why-you-should-test-long-and-short-thresholds-separately/ Last updated: 2026-07-27T13:23:34.000Z # You take 14-day RSI on SPX and run the permutation-corrected threshold search from the article ["How to Test Indicator Thresholds Without Fooling Yourself"](https://aligrithm.com/how-to-test-indicator-thresholds-without-fooling-yourself/). The long side ("buy when RSI below θ\_L") returns θ\_L = 24 with permutation p = 0.07\. The short side ("sell when RSI above θ\_S") returns θ\_S = 78 with permutation p = 0.49\. You combine them into a single oscillator strategy with a 24/78 channel and report a portfolio Sharpe of 0.71. Live, the long leg roughly tracks its backtest. The short leg loses money in a straight line for 14 months. The "Sharpe 0.71" you reported was a long-side edge contaminated by a short-side that was permutation-rejected before you bolted it on. The mistake is structural. You treated the indicator as one object with one threshold scan, when the data is telling you it is two indicators with two scans and two null hypotheses. The long-side signal and the short-side signal live on different parts of the distribution, on different subpopulations, against different drifts. They have to be tested separately, reported separately, and accepted or rejected separately. ## Why long and short are not mirror images Four structural reasons, none of them controversial once stated. Equities drift up. SPX has a roughly 7% annualized real drift and a positive daily mean. Any long-side rule inherits part of that drift as free baseline return. Any short-side rule has to swim against it. The same statistical edge produces different long-PF and short-PF numbers because the underlying unconditional means are different. Return distributions are skewed. Equity daily returns have negative skew and fat left tails. A short-side rule that triggers in volatile regimes is selecting from a wider, fatter-tailed conditional distribution than the long-side mirror. The variance of the short-PF estimator is larger than the variance of the long-PF estimator at equal sample size. Indicator distributions are asymmetric in their predictive content. The article ["Why Predictive Power Often Lives in the Tails"](https://aligrithm.com/why-predictive-power-often-lives-in-the-tails/) showed that signal concentrates in extreme deciles. The signal in the upper decile and the signal in the lower decile are independent samples. RSI above 80 is overbought-regime conditional. RSI below 20 is panic-regime conditional. Two different market states with two different forward dynamics. Symmetry of the indicator construction does not imply symmetry of the predictive surface. Selection bias doubles when you scan both sides. The permutation test in the prior article corrected for K candidate thresholds in one direction. If you scan K\_L long thresholds and K\_S short thresholds and report the best of either, your effective family-wise error rate is governed by K\_L + K\_S, not by K\_L alone. The naive permutation that fixes a direction first does not correct for the cross-side selection. ## The combined-test trap A common framing: "the indicator carries information if either the long or the short threshold passes." This is true, and it is also the entry point for the bias. Two protocols that look similar produce different results: Protocol A. Test the long side at level α. Test the short side at level α. Report both p-values. Use long if its p passes α, use short if its p passes α, use both if both pass, use neither if neither pass. Protocol B. Test both sides at level α. Report the better one. Apply no correction. Protocol B is the one that produces the "Sharpe 0.71" headline. The family-wise error rate of Protocol B is approximately 2α minus the dependence-corrected overlap, which for nearly-independent long and short signals on a single indicator runs around 1.7α to 1.9α. A nominal 5% test reports false positives at 9% under Protocol B. The remedy is Bonferroni across the two sides, which works because the directional partition is exactly two and the conservatism cost is small. Run each side at α/2 = 0.025, or report the worse-side-adjusted p. The harder failure is Protocol C, the silent one. The researcher computes long-side p and short-side p, looks at both, and writes the paper around whichever side passed. The reader sees one test. The selection across two has already happened. ## Worked example: SPX 14-day RSI, two scans, two stories SPX daily, 1990 to 2026, 9100 bars. Target y\_t = sign(P\_{t+1}/P\_t - 1). Indicator: 14-day RSI. Long-side scan: 41 candidate thresholds θ\_L from 5 to 45 in steps of 1\. Statistic: mean forward log return conditional on RSI ≤ θ\_L versus unconditional mean. Short-side scan: 41 candidate thresholds θ\_S from 55 to 95 in steps of 1\. Statistic: mean forward log return conditional on RSI ≥ θ\_S versus unconditional mean, sign-flipped (short profit is positive when forward return is negative). $$ \\begin{array}{l|c|c|c|c} \\text{Side} & \\text{Best } \\theta & t \\text{ at best} & p\_{\\text{naive}} & p\_{\\text{block-perm, L=10}} \\\\ \\hline \\text{Long (RSI} \\le \\theta\_L\\text{)} & 24 & 2.18 & 0.029 & 0.07 \\\\ \\text{Short (RSI} \\ge \\theta\_S\\text{)} & 78 & 0.86 & 0.39 & 0.49 \\\\ \\text{Symmetric combined (} \\theta = 50 \\pm \\delta\\text{, scan }\\delta\\text{)} & \\delta = 26 & 1.74 & 0.082 & 0.21 \\\\ \\text{Two-sided Bonferroni at } \\alpha = 0.05 & - & - & - & \\max(2 \\cdot 0.07,\\, 2 \\cdot 0.49) \\\\ \\end{array} $$ Five readings. The long side has a borderline edge. Permutation-corrected p = 0.07 is above α = 0.05 but in the zone where a forward-window walk-forward (covered in the prior article) can confirm or reject. The structural story is consistent: RSI ≤ 24 on SPX selects panic-flush days that mean-revert. The sample size at this threshold is around 410 bars out of 9100. The short side has no edge. Permutation-corrected p = 0.49 is consistent with pure chance. The structural story is also consistent: RSI ≥ 78 on SPX selects strong-trend days, and on an instrument with a 7% drift, those days continue more often than they reverse. The indicator's "overbought" reading is not a reversal signal on a trending instrument. It is a continuation signal disguised as a reversal by analogy to the long side. The symmetric combined scan is the worst of three worlds. It assumes θ\_L = 50 − δ and θ\_S = 50 + δ for a single δ, which is the mirror-image constraint. The permutation p of the constrained scan is 0.21, which is above α. The constraint forced a fit to a symmetric model the data does not support. A single δ ≈ 26 splits the difference between a real long edge at δ\_L = 26 and a non-edge short side at δ\_S = 22, which is not where the short side's permutation maximum landed. The constraint averages a signal and a non-signal into a permutation-rejected blob. Bonferroni at α = 0.05 across two sides demands each side's permutation p be below 0.025\. Long fails at 0.07, short fails at 0.49\. The honest verdict on the combined two-sided test is reject. The single-side verdict on the long side at α = 0.05 is borderline-reject. The single-side verdict on the short side at α = 0.05 is reject. The bundled "Sharpe 0.71" oscillator is built on a borderline long edge plus a confirmed non-edge plus a symmetry assumption the data rejects. ## The mirror-image fallacy in indicator design A common construction: an oscillator centered at zero or fifty, with the rule "long when below −c, short when above +c, flat in between." The construction implies the predictive surface is anti-symmetric: whatever the indicator says about the long side at −c, it says the opposite for the short side at +c. Three reasons the construction breaks on real data. The base rate is not zero. Equities have positive drift. Currencies have positive interest carry on one leg. Commodities have term-structure roll. The neutral state of the rule is not "no position," it is "exposed to the baseline drift." A symmetric oscillator that closes positions in the middle subtracts the baseline drift it could otherwise harvest. The conditional distributions on the two sides are not mirror images. RSI ≤ 24 selects high-volatility, large-negative-recent-return days. RSI ≥ 76 selects moderate-volatility, large-positive-recent-return days. The two subsamples have different volatilities, different higher moments, and different forward-window dynamics. The mean and the variance of the forward return inside each subsample are not related by a sign flip. The thresholds that maximize each side's edge are not symmetric around the indicator's center. Long-side optima on equity indices typically sit deep in the lower tail (RSI 20 to 30). Short-side optima, when they exist, typically sit shallower (RSI 65 to 75) or at much deeper levels (RSI above 90) corresponding to blow-off tops rather than ordinary overbought regimes. Forcing θ\_S = 100 − θ\_L throws away both the optimal short threshold and the information that the optimal short threshold may not exist. The right construction is two indicators glued together: I\_long(t) = 1{X\_t ≤ θ\_L} with one threshold, and I\_short(t) = 1{X\_t ≥ θ\_S} with a separately optimized and separately tested threshold. The two indicators share a common signal X but have independent thresholds, independent permutation tests, and independent walk-forward windows. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## The asymmetric null The block-bootstrap permutation from the prior article has to be run twice, once per side, with the same shuffled targets. Procedure: 1. Compute the long-side maximum t-statistic across K\_L candidate thresholds on the real data. Record t\_obs,L and θ\_obs,L. 2. Compute the short-side maximum t-statistic across K\_S candidate thresholds on the real data. Record t\_obs,S and θ\_obs,S. 3. Block-bootstrap the target series y with block length L (10 days for daily data). 4. On the shuffled (X, y\_b), re-run both the long-side scan and the short-side scan. Record t\_max,L,b and t\_max,S,b. 5. Repeat for B = 5000 replications. 6. Compute p\_L = (1/B) Σ\_b 1{t\_max,L,b ≥ t\_obs,L} and p\_S = (1/B) Σ\_b 1{t\_max,S,b ≥ t\_obs,S}. 7. Two-sided Bonferroni decision: accept side d if p\_d ≤ α/2\. Accept the indicator if either side passes. $$ p\_L \\;=\\; \\frac{1}{B}\\sum\_{b=1}^{B}\\mathbb{1}\\bigl(t\_{\\max,L,b}^{(\\text{shuffled})}\\geq t\_{\\text{obs},L}\\bigr), \\quad p\_S \\;=\\; \\frac{1}{B}\\sum\_{b=1}^{B}\\mathbb{1}\\bigl(t\_{\\max,S,b}^{(\\text{shuffled})}\\geq t\_{\\text{obs},S}\\bigr) $$ Three properties of this protocol matter. The same shuffled targets are reused across the two sides per replication. The dependence between p\_L and p\_S under H\_0 is preserved, so the joint distribution is correct rather than reconstructed from two independent runs. Each side carries its own t\_obs. The two p-values are not comparable in magnitude. A p\_L of 0.07 and a p\_S of 0.49 are two separate verdicts, not a single "p = 0.28 on average." Averaging would erase the directional information. Bonferroni across two sides is small loss. The α/2 cut at 0.025 is 64% of the power of the α cut at 0.05 for a t-distribution. The cost of the correction is one-third of the power. The cost of skipping the correction is a doubled false-discovery rate on every two-sided indicator in your research portfolio. ## The Long PF and Short PF table Masters' framing computes the profit factor on each side at every candidate threshold, with the position held while the indicator satisfies the threshold and the win/loss tallied per bar (not per trade, which would artificially inflate the PF on consecutive same-direction bars). The Long PF at threshold θ\_L is sum of positive forward log returns conditional on X ≤ θ\_L divided by sum of absolute negative forward log returns under the same condition. The Short PF at θ\_S is the same calculation with the sign of the forward return flipped and the condition X ≥ θ\_S. Plot both as functions of the cumulative fraction of bars selected. The interpretive read of the plot is the one that survives outside the backtest. A long-PF curve that rises smoothly as the threshold tightens (smaller selected fraction) is what the long-side edge looks like when it exists. The PF growth as the selection gets stricter is the tail-concentration result from ["Why Predictive Power Often Lives in the Tails"](https://aligrithm.com/why-predictive-power-often-lives-in-the-tails/) expressed in PF units. A short-PF curve that hovers near 1.0 across the threshold range, or that rises in the moderate threshold range and falls back at extreme thresholds, is what a non-edge looks like. The "rises in the middle" shape is the optimizer locating noise; the lack of consistency outside the middle is the data telling you the short-side signal is not structural. A short-PF curve that rises only at the deepest tail (RSI above 90, for instance) is a separate signal from any moderate-overbought rule. The deep tail picks up blow-off tops. Treat it as its own indicator with its own threshold and its own permutation test. ## Visualizing the asymmetry ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/05/ChatGPT-Image-May-22--2026--05_03_05-PM.png) The two panels are the central diagnostic. One indicator construction, two scans, two completely different verdicts. ## What this changes in practice Three operational shifts. Every directional indicator is tested as two indicators. Long-side scan and short-side scan are run independently against the same shuffled targets in a single permutation harness. Each side gets its own p-value, its own threshold, its own plateau check, and its own walk-forward. The combined "Sharpe of the bundled strategy" is reported only after both sides have independently passed their tests. If only one side passes, only that side ships, and the other side is documented as rejected, not silently dropped. Symmetric thresholds are not assumed. The constraint θ\_S = 100 − θ\_L is a hypothesis to be tested, not a default. The two-sided permutation harness emits θ\_L and θ\_S as independent quantities. If the structural distance between them disagrees with the symmetric prior by more than a few percent of the indicator's range, the symmetric construction is rejected and the two thresholds are stored as two distinct numbers in the indicator-threshold registry. Bonferroni at α/2 is applied across the two sides by default. The 33% power loss is the price for the doubled selection. Reporting only the better side without correction is the silent version of the bias and is treated as a research-process failure regardless of the underlying signal quality. --- ## KEY POINTS - The long-side threshold scan and the short-side threshold scan are two separate hypotheses on the same indicator. Equity drift, return skew, and conditional-distribution asymmetry make them statistically and structurally non-mirror. - Equities have positive drift. Long-side rules harvest a fraction of that drift as baseline. Short-side rules fight it. Equal underlying edge produces different long-PF and short-PF numbers. - Return distributions are negatively skewed on equity indices. Short-side conditional distributions are wider and fatter-tailed. The variance of the short-PF estimator exceeds the variance of the long-PF estimator at equal sample size. - Scanning K\_L long thresholds and K\_S short thresholds and reporting the better is a selection across K\_L + K\_S, not across K\_L. The naive single-direction permutation correction undercounts the family-wise error. - The two-sided permutation harness runs both scans against the same block-bootstrap shuffled targets in each replication. The shared targets preserve the dependence between long and short null distributions. - Bonferroni at α/2 across the two sides is the default correction. The 33% power loss is small relative to the doubled false-discovery rate of the uncorrected protocol. - Symmetric thresholds (θ\_S = C − θ\_L for some center C) are a hypothesis, not a default. The two-sided scan emits independent θ\_L and θ\_S. Mirror-image is tested against the data, not assumed. - Long-side optima on equity indices typically sit in the lower tail of RSI-style oscillators (20 to 30). Short-side optima, when they exist, either sit shallower or much deeper (above 90). Forcing symmetry erases both possibilities. - A short-PF curve that hovers near 1.0 across the threshold range with a small bump in the middle is a non-edge. The bump is the optimizer locating the noise peak. The structural reading is no signal, regardless of the local maximum's nominal p. - The bundled strategy Sharpe is reported only after both sides have independently passed their permutation tests. If only one side passes, only that side ships. The rejected side is documented, not silently bundled. - On SPX 14-day RSI from 1990 to 2026, the long-side permutation p is 0.07 and the short-side permutation p is 0.49\. The honest two-sided verdict at α = 0.05 with Bonferroni is reject. Some long-only deployments may still be defended at α = 0.10 with a walk-forward confirmation. No two-sided RSI oscillator on SPX survives the protocol. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Statistically Sound Indicators for Financial Market Prediction - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Statistically+Sound+Indicators+for+Financial+Market+Prediction+Timothy+Masters&ref=aligrithm.com) - [Cycle Analytics for Traders - John Ehlers (Amazon)](https://www.amazon.com/s?k=Cycle+Analytics+for+Traders+John+Ehlers&ref=aligrithm.com) ### Start Here URL: https://aligrithm.com/start-here/ Last updated: 2026-10-16T09:34:56.000Z A thread quotes a 71% win rate for fading bitcoin funding. A bank note says buy AUDJPY for the yield. On $100,000, a 4% gap between Australian and yen rates is about $4,000 a year, and a normal day moves the pair by roughly fifty times that paycheck. Into 2008 the gap had been near 7% a year, and AUDJPY then lost nearly half its value in about three months. The paycheck was the premium for warehousing that crash. The person forwarding either pitch usually cannot name the payer, or the date the payer stops. This page is the web version of The Quant Map: the map's own stations, numbered and named as on the plate, each with the articles that answer it. Pick the question you cannot answer about a trade you already have, and read that row. A blank link means the piece is written and queued. The printable PDF carries the plate in vector, five worked descents (example research from observation to live rule), and the full 835-entry strategy library. It waits at the end of the page. Keep four words apart. A theory explains why someone pays. A feature is a number you could have computed that morning. A strategy turns the number into a repeated decision, with a size and a wrapper. A factor asks the same question across a universe. Between the feature and the strategy sit a forecast, a threshold, and a cost check. Any one of those can zero the trade. --- ## The grand map ![The Quant Map grand plate: seventeen stations, two rings, fourteen markets](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/10/quant-map-plate.png) Every box on the plate is a section of this page. [Open the plate full size](https://aligrithm.com/content/files/2026/10/quant-map-plate-full.png) to read it closely. The PDF at the end of the page carries the same plate as a vector A2 sheet, printable for the wall. --- ## The spine, stage by stage Seventeen stations, numbered and named as on the plate. Find the station where your trade broke, or walk them in order and stop at the first one you cannot pass. --- ### I · Foundations *the toolkit you think with, pedagogy, not alpha* A strategy that wins 55% of trades still hits eight losers in a row about 0.17% of the time, once or twice per thousand trades. That run is the strategy working. Risk 2% of a $5,000 account and the streak costs about $800\. Risk 15% because the setup looked clean and the same streak is the account. A second trap sits in the average: +60% then −40% advertises +10% a year, and $100 becomes $96\. Compounding pays the variance. The average does not. | Article | Why it is on this row | | -------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------- | | [Why Trading Is a Probability Business, Not a Certainty Business](https://aligrithm.com/why-trading-is-a-probability-business-not-a-certainty-business/) | One trade is one draw. The standard deviation of trade returns runs several times the edge, so the signal shows up through sample size. | | [Fat Tails: Why Gaussian Thinking Breaks Trading Systems](https://aligrithm.com/fat-tails-why-gaussian-thinking-breaks-trading-systems/) | The streak table gets worse once the left tail is fatter than a normal curve. Gaussian size is how the account dies on an "ordinary" day. | | [Get-Even-Itis: The Most Expensive Disease in Trading](https://aligrithm.com/get-even-itis-the-most-expensive-disease-in-trading/) | The streak is when you edit the size. The calm version of the rule has to be the one that is already written. | | [Average Drawdown vs Extreme Drawdown](https://aligrithm.com/average-drawdown-vs-extreme-drawdown/) | One worst loss is a single point. The average drawdown is the distribution you size to. | The full station, every article in reading order: [I · Foundations](https://aligrithm.com/spine-i-foundations/). --- ### II · Market architecture *where edges live: underlying markets × instruments, two different axes* The market is the thing whose price you care about. The wrapper is the contract you hold: spot, forward, future, perpetual, ETF, CFD. Four ways to be long bitcoin are one view and four P&Ls. Spot has custody. A quarterly future makes you pay or earn the basis every roll. A spot ETF sleeps while bitcoin does not. A perpetual never expires, and the crowded side pays funding, about 11% a year when the 8-hour print is 0.01%, and much more in a mania. In April 2020, spot oil recovered from the negative print. Holders of the rolled-futures wrapper did not get that recovery back. Being right about the market and wrong about the wrapper still loses the money. | Article | Why it is on this row | | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------- | | [How FX PnL Actually Works](https://aligrithm.com/how-fx-pnl-actually-works/) | The pair, the account currency, and the roll are three different numbers. The backtest on a clean mid misses two of them. | | [Market Orders vs Limit Orders in FX](https://aligrithm.com/market-orders-vs-limit-orders-in-fx/) | The order type chooses who pays the spread. That choice belongs in the strategy. | | [Why Bid/Ask Bounce Matters for Intraday FX Systems](https://aligrithm.com/why-bid-ask-bounce-matters-for-intraday-fx-systems/) | On a short horizon the bounce is a large fraction of the move you think you forecast. | | [Currency Quoting Conventions: Numerator, Denominator, and the Priority Order](https://aligrithm.com/currency-quoting-conventions-numerator-denominator-and-the-priority-order/) | Which currency is the price, and which is the thing priced, changes the sign of every feature you build. | The full station, every article in reading order: [II · Market architecture](https://aligrithm.com/spine-ii-market-architecture/). --- ### III · Market theory *how markets can work in principle, efficiency is the null hypothesis* Events arrive in bursts. Volatility clusters. A liquidity shock can rewire correlations inside an afternoon. Clock time treats those as evenly spaced samples, and a lot of the structure lives in the waiting time, the path between prints, and the shape of the book of correlations. The articles here are frontier reading: they assume you already know what a null is, what a feature is, and what a cost line is. Read them cold and you collect vocabulary. Vocabulary does not size a position. | Article | Why it is on this row | | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------- | | [Market Data Is Social Data: From Fractals to Topological Data Analysis](https://aligrithm.com/market-data-is-social-data-from-fractals-to-topological-data-analysis/) | The cross-section has a shape. Persistent homology is one way to measure when that shape tears. | | [The Omori Law: Why Aftershocks Follow Market Crashes](https://aligrithm.com/the-omori-law-why-aftershocks-follow-market-crashes/) | After a crash the intensity of large moves decays in a way you can fit. The clock for that decay is event time. | | [Long-Range Dependence: Real Memory or Just Short-Range Echo?](https://aligrithm.com/long-range-dependence-real-memory-or-just-short-range-echo/) | A slow autocorrelation can be real memory or a short-memory process with a break. The distinction changes the forecast. | | [Markets Are Getting More Random: Non-Stationarity of Randomness Itself](https://aligrithm.com/markets-are-getting-more-random-non-stationarity-of-randomness-itself/) | The amount of structure is a time series. A rule fit to a structured decade meets a noisier one. | The full station, every article in reading order: [III · Market theory](https://aligrithm.com/spine-iii-market-theory/). --- ### IV · Edge mechanisms *why should an edge exist?, the root question of the map* Name the payer before you name the indicator. Reachable money is usually a risk someone better-resourced refuses to warehouse, a capacity limit that makes the trade a rounding error for them, or a flow that has to deal. A $5 million niche earning 40% a year is a living at your size and 0.01% of a $20 billion fund, which is why the fund leaves it. If the answer to "why does this pay" is "the backtest went up," you do not have a mechanism yet. | Article | Why it is on this row | | ------------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------- | | [The Theory of Edge: Why the Market Pays You](https://aligrithm.com/the-theory-of-edge-why-the-market-pays-you/) | Two checks: why you make money, and why you keep it. The barrier is measured against the participant across from you. | | [Who Is the Marginal Buyer?](https://aligrithm.com/who-is-the-marginal-buyer/) | Price is the last clip. If you cannot name who bids here, nothing is under the print. | | [Large Trades Are Insider Trades by Definition](https://aligrithm.com/large-trades-are-insider-trades-by-definition/) | Impact scales near the square root of size. The part that does not revert is the information. | | [Predictive Power vs Long Bias: The Hidden Trap in Backtests](https://aligrithm.com/predictive-power-vs-long-bias-the-hidden-trap-in-backtests/) | A QQQ trend rule's 22% split into about 16% equity drift, 5% tech tilt, and 1% timing. | The full station, every article in reading order: [IV · Edge mechanisms](https://aligrithm.com/spine-iv-edge-mechanisms/). --- ### V · Observable world *what can we measure?, point-in-time, or it does not exist* An observable is whatever was knowable then: price, a rate, funding, open interest, a commitment-of-traders print, the book. A revised series, a delisting filled in later, a hedge ratio fit on the whole sample, none of those existed that morning. Point-in-time is the filter. If the series fails it, the feature you build on top is a look at the answer. | Article | Why it is on this row | | ----------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------- | | [Noise Is Not Volatility](https://aligrithm.com/noise-is-not-volatility/) | Two paths with the same range can be a trend or a chop. The efficiency ratio separates them: net move over path length. | | [Efficiency Ratio Explained for Traders](https://aligrithm.com/efficiency-ratio-explained-for-traders/) | The number, 0 to 1, that tells you whether the market you are about to trade is the kind your rule wants. | | [How to Rank Markets by Trend Quality](https://aligrithm.com/how-to-rank-markets-by-trend-quality/) | Rank the universe by the observable before you drop one system on all of them. | | [Reading the COT Report: Three Trader Groups, One Edge](https://aligrithm.com/reading-the-cot-report-three-trader-groups-one-edge/) | Positioning is public, weekly, and late. The lag is why a slow account can still use it. | The full station, every article in reading order: [V · Observable world](https://aligrithm.com/spine-v-observable-world/). --- ### VI · Feature engineering *transforms of observable state, one raw variable, many research questions* One raw series supports many questions, and they are different hypotheses. Carry level asks how big the paycheck is. Carry over volatility asks how big it is per unit of pain. Carry rank asks whether this pair is the best paid in the set. Carry change asks whether the gap is widening. Carry conditioned on trend asks whether price already agrees. Five features, two rates and a spot. RSI, MACD, and a stochastic on the same close are often one band agreeing with itself. Stacking them feels like confirmation. The frequency response says it is one feature in costume. The filter shelf past this table is the same question asked of one operator: which band it keeps, and how late. Butterworth, sinc, Hilbert, roofing, and the named oscillators are later visits. Start with the primer and the frequency response. A filter output is a feature. The forecast is a later station. | Article | Why it is on this row | | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- | | [The Indicator Is More Important Than the Model](https://aligrithm.com/the-indicator-is-more-important-than-the-model/) | No transform of a weak input carries more information about the target than the input did. The ceiling is the feature. | | [Why Most Indicators Should Be Transformed Before Modeling](https://aligrithm.com/why-most-indicators-should-be-transformed-before-modeling/) | Scale, tails, stationarity, and cross-market units. The transform is part of the feature's name, and its window is not a backtest knob. | | [Feature Engineering Before Machine Learning](https://aligrithm.com/feature-engineering-before-machine-learning/) | On daily bars the sample is small and the signal is thin. The pipeline that stationarizes and checks the histogram does more than the model swap. | | [No Filter Is Predictive: What Traders Misunderstand About Smoothing](https://aligrithm.com/no-filter-is-predictive-what-traders-misunderstand-about-smoothing/) | A causal filter is a function of the past. A crossover is the moment the less-lagged average caught up, after the turn. | The full station, every article in reading order: [VI · Feature engineering](https://aligrithm.com/spine-vi-feature-engineering/). --- ### VII · Hypothesis *“if mechanism X is real, feature F should predict Y over horizon H”* Write the death condition before the run. "Ranked by carry-to-volatility, the top third of G10 beats the bottom third over the next one to three months, after costs" can die: the buckets fail to rise in order, or the whole result lives in 2002 to 2007\. "This chart looks like a head and shoulders" cannot die. Explanation is cheap after the print. A prediction is a dated, specific claim you cannot edit. The scientific bar is the pre-commit: hypothesis, null, statistic, threshold, held-out window, trial count, in a file whose timestamp is earlier than the result. | Article | Why it is on this row | | --------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------- | | [Technical Analysis as a Scientific Hypothesis](https://aligrithm.com/technical-analysis-as-a-scientific-hypothesis/) | The rule, the statistic, the null, the threshold, written first. A popular moving-average cross on SPX often fails to clear 0.05. | | [How to Make Technical Analysis Falsifiable](https://aligrithm.com/how-to-make-technical-analysis-falsifiable/) | Turn the pattern into a function, a horizon, a success count, and a baseline. Then publish the rejection. | | [The Null Hypothesis for Trading Systems](https://aligrithm.com/the-null-hypothesis-for-trading-systems/) | Start from "no predictive power on detrended data." A rule that has not beaten that is a candidate. | | [The Difference Between Explanation and Prediction in Markets](https://aligrithm.com/the-difference-between-explanation-and-prediction-in-markets/) | A story fitted after the close had the answer. A forecast did not. Keep a dated log if you want the second. | The full station, every article in reading order: [VII · Hypothesis](https://aligrithm.com/spine-vii-hypothesis/). --- ### VIII · Validation *is it real, and does it survive costs?, most ideas die here, and that is the point* Most ideas die here, and the archive is the output. A single Sharpe is a sample statistic with a wide interval: a Sharpe near 1 on a few years of daily data is often consistent with a true Sharpe near 0\. The best of forty-one thresholds is a maximum. At a 5% bar the family-wise false-positive rate on that scan is about 88% before you correct it. Add the spread, the fees, and an honest slippage before you like the curve. A printed machine-learning paper can fail on its own numbers: a do-nothing baseline, a base rate, a cost line that was never subtracted. | Article | Why it is on this row | | ---------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------ | | [Why One Backtest Tells You Almost Nothing](https://aligrithm.com/why-one-backtest-tells-you-almost-nothing/) | Report the interval. A 12% annualized mean on a noisy daily sample can have a 95% band that includes a loss. | | [Backtesting Is an Experiment, Not a Screenshot](https://aligrithm.com/backtesting-is-an-experiment-not-a-screenshot/) | Pre-commit, detrend, permute, correct for the number of tries. One equity curve is one draw. | | [CSCV: A Direct Probability of Backtest Overfit](https://aligrithm.com/cscv-a-direct-probability-of-backtest-overfit/) | The probability that the in-sample winner is a loser out of sample, computed from the search you ran. | | [Permutation Tests for Indicator Significance](https://aligrithm.com/permutation-tests-for-indicator-significance/) | Shuffle the target, rerun the whole scan, and compare your best statistic to the best of each shuffle. | The full station, every article in reading order: [VIII · Validation](https://aligrithm.com/spine-viii-validation/). --- ### IX · Forecast / signal *a statement about the future, not a trade yet* AUDJPY at +1.3 and EURCHF at −0.8 are comparable units, and neither is a position. The station is the map from feature to expected value; in a cross-section, the stronger version is often an order, not a number. | Article | Why it is on this row | | --------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------- | | [Ranking Beats Forecasting for Many Trading Problems](https://aligrithm.com/ranking-beats-forecasting-for-many-trading-problems/) | The cross-section often needs an order. Rank, then hold the wings. | | [Ranking Beats Forecasting — But Ranking What?](https://aligrithm.com/p/a9fbe29e-35a0-4e84-8723-b22f061d03ef/) | The target you rank on decides the trade. A rank of the wrong residual is a clean sort of noise. | | [From Indicator Value to Expected Value](https://aligrithm.com/from-indicator-value-to-expected-value/) | Map the feature to a conditional expectation before you turn it into a position. A z-score still has to become an edge in currency units. | The full station, every article in reading order: [IX · Forecast / signal](https://aligrithm.com/spine-ix-forecast-signal/). --- ### X · Decision policy *forecast → position: trade only if E\[r\] clears costs + threshold* The decision is the arithmetic after the score: expected edge next month, about a 4.1% gap divided by 12, near 34 basis points, against a couple of basis points to get in and out at institutional size. Clear the toll with room, then size by risk so one wild pair cannot own the book. Refuse a rule whose expected gross edge per trade sits under about three times the all-in round trip. That filter deletes most of what gets marketed as a five-minute system. | Article | Why it is on this row | | -------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------- | | [Expectancy: The Most Important Formula in Trading](https://aligrithm.com/expectancy-the-most-important-formula-in-trading/) | Win rate times average win, minus loss rate times average loss. The tail term is the one the win rate hides. | | [Cost-Aware Ranking: The Missing Step in Cross-Sectional Strategies](https://aligrithm.com/cost-aware-ranking-the-missing-step-in-cross-sectional-strategies/) | A name that ranks first and costs 40 basis points to enter is a different trade from a name that ranks third and costs 2. | | [Why Transaction Costs Should Be Added Before You Fall in Love](https://aligrithm.com/why-transaction-costs-should-be-added-before-you-fall-in-love/) | Put the toll in the backtest on the first run. A curve you have already bonded with will argue for a kinder cost. | | [The Trader's Real Job: Control Losses, Not Predict Everything](https://aligrithm.com/the-traders-real-job-control-losses-not-predict-everything/) | Direction is the variable you do not set. Size, exposure, the drawdown stop, and the kill criterion are the ones you do. | The full station, every article in reading order: [X · Decision policy](https://aligrithm.com/spine-x-decision-policy/). --- ### XI · Strategy *a decision rule repeated through time, and not everything that trades is alpha* A strategy is the decision frozen: monthly, rank by carry-to-vol, long the top three, short the bottom three, equal risk, rebalance. Morning mood is no longer an input. A clever way to buy is plumbing, a regime model is weather, and a risk limit is a seat belt. Useful, and none of them is a reason someone is paying you. High-noise markets want reversion. Low-noise markets want trend. One indicator on every chart is a refusal to read the observable. | Article | Why it is on this row | | ------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- | | [The Difference Between a Trading Rule, a Strategy, and a Portfolio](https://aligrithm.com/the-difference-between-a-trading-rule-a-strategy-and-a-portfolio/) | A rule emits a score. A strategy adds size, a vol target, and a kill. A portfolio is the set of strategies that can survive one of them dying. | | [Trading Systems Are Recipes, Not Predictions](https://aligrithm.com/trading-systems-are-recipes-not-predictions/) | Precise enough that a stranger gets your trades on your data. The recipe includes the kill, because the regime the recipe assumes will end. | | [Matching Strategy Families to Market Conditions](https://aligrithm.com/matching-strategy-families-to-market-conditions/) | Family first: trend, reversion, carry, breakout. The market's noise level picks the family. | | [High Noise Markets Are Mean-Reversion Markets](https://aligrithm.com/high-noise-markets-are-mean-reversion-markets/) | Path length long relative to net move. Fading is the family that matches the observable. | The full station, every article in reading order: [XI · Strategy](https://aligrithm.com/spine-xi-strategy/). --- ### XII · Factors *the same mechanism, systematically, across an entire universe* Factors come after one asset works end to end. A carry factor is the same question asked of every pair: high carry against low carry, and you no longer care whether AUDJPY rises. | Article | Why it is on this row | | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------- | | [What Is a Factor, Really](https://aligrithm.com/what-is-a-factor-really/) | The same mechanism, applied to a universe, long the high slice and short the low slice. | | [What a Factor Actually Is: α + βλ, and Why the Market Is Factor #1](https://aligrithm.com/what-a-factor-actually-is-a-bl-and-why-the-market-is-factor-1/) | Split the return into the market, the premia you are paid to hold, and a residual. Sell the residual last. | | [Predict Residual Returns, Not Gross](https://aligrithm.com/predict-residual-returns-not-gross/) | Gross return smuggles the beta. The thing you meant to forecast is what is left after the factor exposures. | | [Portfolio Sorts From Scratch: Deciles, Monotonicity, and the Long-Short Spread](https://aligrithm.com/portfolio-sorts-from-scratch-deciles-monotonicity-and-the-long-short-spread/) | Buckets have to rise in order. The long-short spread is the claim. A pretty top decile with a messy middle is a story. | The full station, every article in reading order: [XII · Factors](https://aligrithm.com/spine-xii-factors/). --- ### XIII · Multi-factor models *composites: carry + momentum + value, regime-conditioned* Composition reopens the search. A leftover alpha in a regression is often a factor you left out. A zoo of published factors is a counted search; the t-stat you remember was the maximum of many tries. Timing the factors, turning them on and off with a second model, mostly fails once you count that second search. Cross-sectional machine learning, in the study that measured it carefully, lifts about 20%. | Article | Why it is on this row | | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------- | | [The GRS Test: Your Alpha Is Probably a Missing Factor](https://aligrithm.com/the-grs-test-your-alpha-is-probably-a-missing-factor/) | A joint test on whether the leftovers are all zero. One shiny intercept is a single name in that test. | | [Fama-MacBeth Two-Pass and the Shanken Correction Nobody Applies](https://aligrithm.com/fama-macbeth-two-pass-and-the-shanken-correction-nobody-applies/) | Cross-sectional slopes, then the error in those slopes. Skipping the correction makes the t-stats look finished. | | [The Factor Zoo and the Multiple-Testing Reckoning](https://aligrithm.com/the-factor-zoo-and-the-multiple-testing-reckoning/) | Hundreds of published factors, one history. The bar moves once you count the tries. | | [Factor Timing Mostly Fails — the Honest Version](https://aligrithm.com/factor-timing-mostly-fails-the-honest-version-2/) | A second model that switches factors reopens the search. Most of the switching does not survive it. | The full station, every article in reading order: [XIII · Multi-factor models](https://aligrithm.com/spine-xiii-multi-factor-models/). --- ### XIV · Alpha ensembles *combining edges into one forecast engine* Two edges with a Sharpe near 0.5, roughly uncorrelated, combine to about 0.5 times the square root of 2, near 0.7\. Five copies of the same idea, correlation 0.85, give you almost nothing and one decay date. | Article | Why it is on this row | | ------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------- | | [The Death of the Single-System Trader](https://aligrithm.com/the-death-of-the-single-system-trader/) | One rule is one decay away from a multi-year sideways grind. A handful of structurally different systems is the comparison set. | | [Signal Averaging: Killing Noise with Redundant Alphas](https://aligrithm.com/signal-averaging-killing-noise-with-redundant-alphas/) | Average signals that share a mechanism and differ in noise. Averaging five paraphrases of one price series does not. | | [Don't Trust Your Strategy-Weighting Scheme](https://aligrithm.com/dont-trust-your-strategy-weighting-scheme/) | The scheme that maximized in-sample Sharpe is another search. Flat risk weights are the baseline it has to beat. | The full station, every article in reading order: [XIV · Alpha ensembles](https://aligrithm.com/spine-xiv-alpha-ensembles/). --- ### XV · Portfolio construction *from forecasts to risk: budget it, do not just add it* Budget a portfolio volatility, cut every position when realized vol doubles, cap one name's share of risk, and write the drawdown rule before the drawdown. Signal quality and portfolio quality come apart: a real signal with the wrong weights is a bad portfolio. Overbetting is the failure mode the Kelly fraction exists to refuse. | Article | Why it is on this row | | ------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------- | | [Why Volatility-Adjusted Position Sizing Matters](https://aligrithm.com/why-volatility-adjusted-position-sizing-matters/) | Size to risk. Equal vol contribution stops one pair owning the book. | | [Why Portfolio Construction Is Part of the Signal](https://aligrithm.com/why-portfolio-construction-is-part-of-the-signal/) | The weights change the equity curve as much as the forecast does. Treat them as part of the recipe. | | [The Difference Between Signal Quality and Portfolio Quality](https://aligrithm.com/the-difference-between-signal-quality-and-portfolio-quality/) | A good score and a good book fail for different reasons. Diagnose the layer you broke. | | [Drawdown Is Not Just a Number: It Is a Diagnosis](https://aligrithm.com/drawdown-is-not-just-a-number-it-is-a-diagnosis/) | Depth, length, and whether the other systems are fine. Those three say different things. | The full station, every article in reading order: [XV · Portfolio construction](https://aligrithm.com/spine-xv-portfolio-construction/). --- ### XVI · Implementation *where paper alpha meets the market* On Hyperliquid BTC in a studied window, a refused post-only buy returned about 0.31 basis points over ten seconds. The taker fee sat around 1.44 to 2.7 basis points. Stand aside as a taker; the edge does not clear the toll. Markouts are the truth serum: where the mid is a short time after you filled. Your own size walks the book at something near the square root of the fraction of volume you took. A simulation that fills you at mid, with no queue and no clock drift, is a different market from the one that sends the reject. | Article | Why it is on this row | | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------- | | [Five Definitions of Fair Value, and the Trade Each One Implies](https://aligrithm.com/p/b545996b-0800-4e48-ade2-a484d5288611/) | Name the subtraction before the trade. If the forecast is inside the fee, there is no trade. | | [Markouts: The Truth Serum of Market Making](https://aligrithm.com/markouts-the-truth-serum-of-market-making/) | Mark the fill against the mid a fixed time later. The backtest mark is a story you told yourself. | | [Adverse Selection Explained for Traders](https://aligrithm.com/adverse-selection-explained-for-traders/) | The flow that hits your quote knew something. The spread is the rent you charge for that. | | [Adverse Selection Is Adverse Selection: Porting Fast-Fills-Are-Bad-Fills to FX and Futures](https://aligrithm.com/adverse-selection-is-adverse-selection-porting-fast-fills-are-bad-fills-to-fx-and-futures/) | The fast fill is the toxic one in FX and futures too. | The full station, every article in reading order: [XVI · Implementation](https://aligrithm.com/spine-xvi-implementation/). --- ### XVII · Live research *strategies are mortal, monitor, adapt, retire* The live rule is an experiment that does not end. Watch the paycheck (the gap can close because policy changed), the crowding (positioning stretched means a narrow exit), and the weather. Two exits are both legitimate. Adapt: go back to the mechanism, the observable, and the feature with what the break taught you. Retire: archive the rule with a post-mortem so the next version of you does not pay the tuition twice. A crowded edge grinding lower as the trade gets popular does not heal by waiting. A barriered edge inside a historical-looking drawdown is often noise. The difference is the diagnosis, and it has to be written before you are in the drawdown. | Article | Why it is on this row | | ------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------- | | [How to Detect When a Trading System Is Dying](https://aligrithm.com/how-to-detect-when-a-trading-system-is-dying/) | Decay is usually a grind. You need a measure that fires before the equity curve feels wrong. | | [When to Switch Off a Trading System](https://aligrithm.com/when-to-switch-off-a-trading-system/) | The kill criterion is a number you set on a calm day: drawdown, hit rate over a counted window, or a structural breach. | | [Why Systems Work Until They Don't](https://aligrithm.com/why-systems-work-until-they-dont/) | The rule was conditional on a regime. The post-mortem names which condition ended. | | [Why OOS Failure Is Often a Stationarity Failure](https://aligrithm.com/why-oos-failure-is-often-a-stationarity-failure/) | The held-out window failed because the distribution moved. | The full station, every article in reading order: [XVII · Live research](https://aligrithm.com/spine-xvii-live-research/). --- ## The two rings (and where ML sits) Two conditions wrap every station of the spine. --- ### The regime / state ring *which market are you in today?* The same carry rule is an income stream in calm markets and a crash in a yen squeeze. Into August 2024 the Bank of Japan lifted rates to about a quarter of a percent, USDJPY fell from about 162 to about 142 in weeks, and the Nikkei dropped about 12% in a day. The mechanism was intact. The weather had changed. A backtest that lived inside one regime is a sample of that weather. Regime conditions the feature, the forecast, and the size at once. | Article | Why it is on this row | | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------- | | [Stationarity: The Word Every Trader Ignores Until It Kills the Strategy](https://aligrithm.com/stationarity-the-word-every-trader-ignores-until-it-kills-the-strategy-2/) | Yesterday's conditional distribution is the thing the live rule is betting still holds. | | [Slow Wandering: The Most Dangerous Type of Market Change](https://aligrithm.com/slow-wandering-the-most-dangerous-type-of-market-change/) | The quiet multi-year drift kills more rules than the crash you can see on a chart. | | [Volatility Regimes and Strategy Survival](https://aligrithm.com/volatility-regimes-and-strategy-survival/) | Vol is more regime than trend is. A rule sized to one vol state meets the next one oversized. | | [Regime Coverage: Why Your Backtest Needs Different Market States](https://aligrithm.com/regime-coverage-why-your-backtest-needs-different-market-states/) | One friendly decade is one state. Coverage means the hostile states are in the sample on purpose. | | [Why Market Cycles Are Evanescent](https://aligrithm.com/why-market-cycles-are-evanescent/) | A measured cycle drifts, fades, and loses phase. A hardcoded period is a bet that this one stays. | | [Correlation Regimes: Positive Feedback, Breakdowns, and "USDCAD Is the Truth"](https://aligrithm.com/correlation-regimes-positive-feedback-breakdowns-and-usdcad-is-the-truth/) | The hedge you wrote in one correlation regime is a different position after the break. | | [Mutual Information as a Regime / Noise Filter](https://aligrithm.com/mutual-information-as-a-regime-noise-filter/) | Use dependence as a gate on when the rule is allowed to trade. | | [Entropy as a Market Choppiness Gauge](https://aligrithm.com/entropy-as-a-market-choppiness-gauge/) | Choppiness you can compute is a regime input. A flat periodogram is the rule standing down. | --- ### The research governance ring *is the research telling the truth?* Governance polices every arrow between the stations. Before the run: the hypothesis, the null, the benchmark, the number of trials, the point-in-time rule, and the result that would kill it. A parameter that works only at 13 and fails from 11 to 17 memorized the sample. A loose region, Sharpe stable across a band, is the one you can defend. Simplicity is the degrees-of-freedom cap: effective sample size in markets is a few hundred independent bets, and each free parameter spends some of them. | Article | Why it is on this row | | -------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | [Why "Quantitative" Does Not Automatically Mean Scientific](https://aligrithm.com/why-quantitative-does-not-automatically-mean-scientific/) | A falsifiable claim, a null, a benchmark, and a rerun. Those four are the method. | | [The 10% Rule of Degrees of Freedom](https://aligrithm.com/the-10-rule-of-degrees-of-freedom/) | Cap free parameters near a tenth of the sample. The search space grows as the product of the knobs. | | [Degrees of Freedom in Trading Systems](https://aligrithm.com/degrees-of-freedom-in-trading-systems/) | Every threshold, lookback, and market you kept after looking is a trial. Count them. | | [Loose Pants Fit Everyone: Why General Trading Ideas Survive Longer](https://aligrithm.com/loose-pants-fit-everyone-why-general-trading-ideas-survive-longer/) | Robustness ratio: scatter of Sharpe across the sweep, divided by the mean. A spike at one setting is a tailored fit. | | [Why Simplicity Is a Statistical Weapon](https://aligrithm.com/why-simplicity-is-a-statistical-weapon/) | Added parameters fit noise in sample. Out of sample they spend the edge they appeared to add. | | [Why Benchmarks Matter in Rule Evaluation](https://aligrithm.com/why-benchmarks-matter-in-rule-evaluation/) | 15% a year against zero and 15% against a bias-matched random rule are different claims. | | [The Problem with One Sample of Market History](https://aligrithm.com/the-problem-with-one-sample-of-market-history/) | Resampling shrinks the inner error. It does not give you a second history. Size for that. | | The Red-Flag Field Guide: How to Spot a Fake ML Trading Paper | Printed accuracy, a missing cost, a base rate, a copied table. The failure is usually on the page. | | Read the Paper for Its One Load-Bearing Number | One figure has to survive contact with the abstract. Find it before you discuss the architecture. | | Replication Notebook Template: Claim, Replicate, Stress, Verdict | The shape of a check: the claim, the rerun, the stress, the verdict. Archive the dead ones. | | [Building a Statistically Valid Backtest: The Checklist Nobody Runs](https://aligrithm.com/building-a-statistically-valid-backtest-the-checklist-nobody-runs/) | Published predictors run about 26% weaker out of sample and 58% weaker after print. The list is a frozen holdout, a cost, a size cap, and a test of the whole set. | | [The Vice of Complexity: Four Times the Simple Model Won](https://aligrithm.com/p/90af2d81-a24e-4d7d-9ada-4bfcc4be1232/) | 12,000 features on 12 months. The re-estimated complex book earns a Sharpe of 0.49, and the small fit is the one that clears the next window. | --- ### Where machine learning sits A fitted model weights and conditions signals that already have a payer. "I trained XGBoost" is a method. The mechanism is still the question in the first section. On this site the fit shows up late: after the feature is stationary, after the band is named, after the null exists. Trees will spend their splits on a raw price level and call it skill. Regularization is the complexity penalty you can write down. Below a few minutes the function class can matter; above an hour the feature usually matters more. A model that only works as an interaction still has to clear costs as a trade. | Article | Why it is on this row | | -------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------- | | [The Limits of Linear Models](https://aligrithm.com/the-limits-of-linear-models/) | Linear is the baseline the nonlinear fit has to beat on a held-out window, after costs. | | [Linear Models' Hidden Symmetry Advantage](https://aligrithm.com/linear-models-hidden-symmetry-advantage/) | A linear fit has a constraint the tree does not. Sometimes the constraint is the robustness. | | [From One Tree to Forests to Boosting](https://aligrithm.com/from-one-tree-to-forests-to-boosting/) | The ensemble averages trees. It does not average away a leaked label or a non-stationary input. | | [How a Decision Tree Engineers a New Alpha](https://aligrithm.com/how-a-decision-tree-engineers-a-new-alpha/) | The splits are a feature constructor. Read the leaves as a rule, and count the trials. | | [Ridge Above 1h, XGBoost Below 5min](https://aligrithm.com/ridge-above-1h-xgboost-below-5min/) | Horizon picks the function class. A five-minute booster on a daily fundamental is a costume. | | [Regularization from First Principles](https://aligrithm.com/regularization-from-first-principles/) | The penalty is a degrees-of-freedom cap with a dial. Set the dial outside the backtest you are grading. | | [Does ML Actually Help Asset Pricing? Kelly's 20%, Not 2–3×](https://aligrithm.com/does-ml-actually-help-asset-pricing-kellys-20-not-2-3x/) | About 20% lift in the cross-section, in the study that tried to measure the help. That is the scale. | | [Predictive but Uncorrelated: Alphas That Only Work as Interactions](https://aligrithm.com/predictive-but-uncorrelated-alphas-that-only-work-as-interactions/) | A feature with no standalone correlation can still matter inside a product. Test the product, and pay its turnover. | --- ## The markets The map walks fourteen markets and the instrument layer on top of them. One family gets its own shelf here: prediction markets, where a fair book is arithmetic you can check. ### Alternative / event markets On a prediction market the cleanest money is a book that cannot all happen at once: two contracts that each look fair and together cost more or less than the certain payoff. That is a Dutch book, and the trade is a projection back onto the set of prices that add up. Forecasting the event is further down the list, and it is the edge that crowds fastest. The same spine still applies. Who pays, what you could have measured, what would kill it, whether the fill clears the fee. A copied arbitrage that crosses the spread is a different trade from the one the detector saw. | Article | Why it is on this row | | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------- | | [The Nine Sources of Edge, Ranked Cleanest to Dirtiest](https://aligrithm.com/the-nine-sources-of-edge-ranked-cleanest-to-dirtiest/) | Structural inconsistency at the clean end. A forecast of the outcome at the dirty end. Start at the clean end. | | [The Dutch Book Theorem, Plainly, and Why Polymarket Will Always Have Arbitrage](https://aligrithm.com/the-dutch-book-theorem-plainly-and-why-polymarket-will-always-have-arbitrage/) | Prices that violate addition are a locked payoff. The venue's matching does not close every one of them. | | [Outcome Geometry: Why Two Fair Markets Can Be Jointly Unfair](https://aligrithm.com/outcome-geometry-why-two-fair-markets-can-be-jointly-unfair/) | Each margin can look sane. The joint book is the object that has to be fair. | | [The Marginal Polytope: One Shape That Contains Every Fair Price](https://aligrithm.com/the-marginal-polytope-one-shape-that-contains-every-fair-price/) | The allowed prices are a shape. A quote outside it is the trade. | | [Arbitrage Is Just Projection: One Calculation Gives Trade, Profit, and Target Prices](https://aligrithm.com/arbitrage-is-just-projection-one-calculation-gives-trade-profit-and-target-prices/) | One projection gives the basket, the profit, and the prices you are pushing toward. | | [The Six Ways to Lose Money on Polymarket](https://aligrithm.com/the-six-ways-to-lose-money-on-polymarket/) | Resolution, fees, missed fills, and sizing take the money even when the inconsistency was real. | | [A Good Trade Is Not a Correct Prediction — The Five Invariants](https://aligrithm.com/a-good-trade-is-not-a-correct-prediction-the-five-invariants-2/) | The trade can be right about the book and wrong about the world, and still be the trade you meant. | | [Who Profits From Prediction? Execution, Not Information](https://aligrithm.com/p/51c16ede-a12c-4d06-91c9-92fcd12ba3b7/) | On the published flow, the profit sits with the account that got filled. | --- ## Playing a different game: the retail edge Game selection beats skill. A small account has real advantages, and none of them is speed. ### Which table can this account win? One taker round trip a day at 10 basis points costs about 25% a year (252 times 10 basis points). Ten a day costs 252%, before the rule has earned a dollar. Latency games, market making on major venues, and reacting to a scheduled print put you across from a machine that saw it first and pays a fraction of your fee. The tables a small account can win are the ones where patience, a tiny capacity, and a public calendar matter, and speed does not: days to months, neglected names, funding you can collect, pre-announced rebalances. Boredom is the cheap toll. A prop-firm fee is a product. Read the daily-loss rule as a sizing constraint before you pay it. | Article | Why it is on this row | | ----------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------- | | [Why Traders Lose Even When Their Ideas Are Good](https://aligrithm.com/why-traders-lose-even-when-their-ideas-are-good/) | Seven layers between a signal and live P&L. Costs, a 30-trade sample, and long-bias drift each take a cut. | | [Prop Firm Math: Edge, Barriers, and Cash After the Fee](https://aligrithm.com/prop-firm-math-edge-barriers-and-cash-after-the-fee/) | The fee, the daily loss cap, and the deadline are the game. Pass the sizing arithmetic before the challenge. | | [Maker vs Taker Edge: Same Signal, Different Economics](https://aligrithm.com/maker-vs-taker-edge-same-signal-different-economics/) | The same signal can be worthless when you pay the spread and useful when you earn it. | | [Why Small Alphas Matter More for Makers Than Takers](https://aligrithm.com/why-small-alphas-matter-more-for-makers-than-takers/) | A small edge that fails as a take can pay as a quote improvement. | | [FX Is Not One Market: Retail vs Wholesale Structure](https://aligrithm.com/fx-is-not-one-market-retail-vs-wholesale-structure/) | Retail and interbank are different products. The broker's routing is part of the signal's economics. | | [Why Retail FX Execution Is Not the Same as Interbank FX](https://aligrithm.com/why-retail-fx-execution-is-not-the-same-as-interbank-fx/) | The overnight financing on a retail roll can eat the carry you came for. Measure your swap against the rate gap. | | [MFT Execution Is Built on HFT Market-Making](https://aligrithm.com/mft-execution-is-built-on-hft-market-making/) | Your "fast" on a major book is someone else's leftover. The horizon decides whether that matters. | | [Match Your Edge to Your Resources](https://aligrithm.com/match-your-edge-to-your-resources/) | Rank the edge by what you have: data, speed, capital, patience. A mismatch is a donated table. | | [Why Good Trading Feels Boring](https://aligrithm.com/why-good-trading-feels-boring/) | A 10% vol target on $100,000 is about $630 of daily noise. If the P&L is entertaining, the size is the entertainment. | --- ## Take the map with you The spine, the two rings, fourteen markets, five worked descents, and a library of about 835 strategy entries, each with a named payer and a named failure mode, in one 65-page PDF. The grand-map plate inside is vector at A2\. Print that page for the wall and read the rest as a booklet. The tables above are rows of that library written out in full; the PDF tells you which rows to want next. [Download The Quant Map (PDF, free, no email)](https://aligrithm.com/content/files/2026/10/quant-map.pdf) ### 2.7 How to Test Indicator Thresholds Without Fooling Yourself URL: https://aligrithm.com/how-to-test-indicator-thresholds-without-fooling-yourself/ Last updated: 2026-07-27T13:23:33.000Z # You compute a 14-day RSI on SPX, target sign(P\_{t+1}/P\_t - 1), and scan 41 candidate "buy below this RSI" thresholds from 5 to 45 in steps of 1\. Threshold 27 gives mean forward return +0.082% per day at a t-statistic of 2.41\. The naive p-value at α = 0.05 looks clean. You ship it. Live performance over 18 months: indistinguishable from the unconditional mean. The threshold did not break. The p-value broke. This article is about the gap between the t-statistic the optimizer printed and the one a third party would have to defend in front of new data. The gap is large enough that most published "RSI below X works" results are noise dressed in 2.0 to 2.5 t-statistic clothing. ## Why "best threshold" is the wrong frame When you scan K thresholds and report the best, the statistic you printed is not the t-statistic of one test. It is the maximum of K t-statistics computed on the same data. Under the null hypothesis that the indicator carries no information, the maximum of K test statistics has a known distribution. It is not the marginal distribution of a single test. The simple version of the inflation: $$ P\\bigl(\\max\_{k=1}^{K} t\_k > c \\,\\big|\\, H\_0\\bigr) \\;=\\; 1 - \\prod\_{k=1}^{K} \\bigl(1 - P(t\_k > c)\\bigr) $$ If the K tests are independent and each has marginal p-value α, the family-wise error rate is 1 - (1-α)^K. For K = 41 and α = 0.05, the chance of at least one t-statistic clearing 1.96 under H\_0 is 1 - 0.95^41 ≈ 0.88\. Almost certain. The candidate thresholds are not independent. They are smooth shifts of the same underlying ranking, so the effective number of tests is smaller than 41\. The naive Bonferroni correction (multiply your reported p by K) is too conservative. The right correction is empirical: estimate the distribution of the maximum-statistic under the null by permutation, then compare your observed maximum to that distribution. ## The permutation test of the entire search The procedure has five steps. The cost is one shuffle and one full re-scan per replication. Sub-second on any modern laptop for a 9000-bar series. 1. Compute the statistic of interest at every candidate threshold on the real data. Record the maximum and the threshold at which it occurred. Call these t\_obs and θ\_obs. 2. Shuffle the target series y relative to the indicator X. The shuffle breaks the temporal link between indicator state and forward return while preserving the marginal distribution of both. 3. Re-run the full threshold scan on (X, y\_shuffled). Find the new maximum t\_max,b across the same K candidate thresholds. 4. Repeat steps 2 and 3 for B replications. B = 5000 is the floor. 5. The permutation p-value of the search is the fraction of replications where t\_max,b ≥ t\_obs. $$ p\_{\\text{search}} \\;=\\; \\frac{1}{B} \\sum\_{b=1}^{B} \\mathbb{1}\\bigl(\\, t\_{\\max,b}^{(\\text{shuffled})} \\;\\geq\\; t\_{\\text{obs}} \\,\\bigr) $$ The permutation p-value is the honest one. It accounts for the search process. If 14% of the 5000 shuffled datasets produced a "best threshold" t-statistic at least as large as 2.41, your real result is consistent with what the search process produces from noise alone. For block-time-series data, replace the iid shuffle with a block bootstrap (5 to 20 day blocks). The block bootstrap preserves autocorrelation and volatility clustering. The article ["The Null Hypothesis for Trading Systems"](https://aligrithm.com/the-null-hypothesis-for-trading-systems/) covers null model selection. ## Worked example: RSI thresholds on SPX SPX daily, 1990 to 2026, 9100 bars. Target y\_t = sign(P\_{t+1}/P\_t - 1). Indicator: 14-day RSI. Scan 41 "buy below θ" thresholds from 5 to 45 in steps of 1\. Statistic: t-statistic of the conditional mean forward return inside the threshold versus the unconditional mean. $$ \\begin{array}{l|c|c|c} \\text{Procedure} & \\text{Best } \\theta & t \\text{ at best} & p \\\\ \\hline \\text{Naive: best of 41 thresholds} & 27 & 2.41 & 0.016 \\\\ \\text{Bonferroni correction (K = 41)} & 27 & 2.41 & 0.66 \\\\ \\text{IID search-permutation, B = 5000} & 27 & 2.41 & 0.18 \\\\ \\text{Block bootstrap permutation, L = 10} & 27 & 2.41 & 0.31 \\\\ \\end{array} $$ Four readings. The naive p of 0.016 looks publishable. It is the p-value of one specific test of one specific threshold. You did not test one threshold. You tested 41. Bonferroni at K = 41 gives p = 0.66\. Far too pessimistic, since the 41 thresholds are correlated draws on the same underlying ranking. Bonferroni assumes independence the data does not have. The IID permutation of the full search gives p = 0.18\. Above the 0.05 bar. The search-adjusted result is consistent with chance under a null that preserves the marginal distributions but breaks the autocorrelation. The block bootstrap permutation, which preserves the autocorrelation and volatility clustering structure of the data, gives p = 0.31\. The IID null underestimates how often "winning" threshold statistics emerge from clustered noise. The realistic null says 31%. The "RSI < 27 works on SPX" result is the search process picking the maximum of 41 correlated noisy samples on clustered data. The signal is not distinguishable from the noise once the search is accounted for. ## The plateau requirement A second test runs alongside the permutation: the threshold must sit in a smooth region of the threshold-statistic curve. The same loose-pants argument from the article ["Loose Pants Fit Everyone: Why General Trading Ideas Survive Longer"](https://aligrithm.com/loose-pants-fit-everyone-why-general-trading-ideas-survive-longer/) applies here. Plot the conditional statistic (mean forward return, profit factor, Sharpe) as a function of the threshold across the candidate range. Two readings. A smooth concave region with the chosen threshold near its peak. Moving the threshold ±20% of its range changes the statistic by less than 10%. Real signal. The threshold is a property of the region, not of one number. A single needle spike at the optimizer's chosen value, with neighboring thresholds at half the statistic. Curve-fit. The threshold is the maximum of a noisy surface and lives at a fragile point. Operationally, after the permutation test passes, fit a smoothing spline to the threshold-statistic curve. The spline's local maximum is closer to the structural threshold than the raw optimizer's pick. Deploy the spline maximum if the underlying curve has a plateau. Reject the indicator if the curve is a single spike, regardless of how good the permutation p-value looked. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Walk-forward as the second filter Permutation handles the search-bias problem on a single dataset. It does not handle the regime-shift problem on a non-stationary series. Walk-forward threshold selection: 1. Split history into rolling training and forward windows. Typical: 5-year training, 1-year forward. 2. Run the full threshold scan plus permutation test on the training window. If permutation p > α, drop the indicator. If p ≤ α, record the threshold from the training window. 3. Evaluate the indicator at the recorded threshold on the forward window, with no re-optimization. 4. Roll the window forward and repeat. The walk-forward concatenation of forward-window evaluations is the out-of-sample performance. If the indicator passes the in-sample permutation test in window 1 but the threshold drifts by 30% across windows, the threshold is fitting noise per window. The indicator is not structurally informative; the search is relocating the noise peak. The walk-forward also generates a confidence interval on the live behavior. The 5th and 95th percentiles of the forward-window statistics across rolling windows are a non-parametric CI that reflects both the sampling uncertainty and the regime drift. Report the CI alongside the point estimate. The article ["The Problem with One Sample of Market History"](https://aligrithm.com/the-problem-with-one-sample-of-market-history/) covers why this CI matters more than the in-sample one. ## The single-threshold alternative The cheapest defense against threshold-search bias is to skip the search entirely. Pre-commit to a single threshold from a structural property of the data. Three candidates, in order of preference: - The indicator's in-sample median. A two-sample test "above median vs below median" is one test, not K. No selection bias. - A theoretical threshold from the indicator's construction. RSI has 30 and 70 as boundary values designed by construction. They are not optimized to your data. - The quantile that splits the indicator population at the prior probability of the target event. For a target with a 53% positive rate, the 53rd percentile of the indicator. A single pre-committed threshold gives one t-statistic with one p-value and no search inflation. The cost is power: the pre-committed threshold may not be where the signal lives. The benefit is honesty. The right deployment is to use the single threshold for the indicator's "does this carry information" test, and to use the search-with-permutation for the threshold's "where does the information concentrate" test. Two different questions, two different protocols. Mixing them produces a search inflation dressed as a confirmation test. ## Visualizing the search bias ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/05/ChatGPT-Image-May-22--2026--01_02_21-AM.png) The two panels are the central diagnostic. The optimizer always finds a maximum. The honest question is how often noise finds a maximum at least as big. ## What this changes in practice Three operational shifts. The reported p-value is the search-adjusted permutation p-value, not the naive p-value of the winning threshold. If the indicator's permutation p is above α, the indicator does not ship, regardless of how good the winning t-statistic looked. Every threshold deployed to live trading has a recorded plateau check. The threshold is replaced by the local maximum of a smoothing spline through the threshold-statistic curve when the underlying curve has a plateau. The threshold is rejected when the underlying curve is a single spike, regardless of permutation p-value. The permutation handles the false discovery rate. The plateau handles the fragility. Walk-forward thresholds are recorded across rolling windows. If the threshold drifts by more than 30% across windows, the indicator is fitting noise per window. The forward-window concatenation provides the out-of-sample CI that the in-sample t-statistic cannot. --- ## KEY POINTS - Reporting the best of K threshold tests treats a maximum-of-K statistic as if it were a single test. Under H₀, the family-wise error rate is 1 - (1-α)^K. For K = 41 candidate thresholds at α = 0.05, the probability of at least one false positive is around 88%. - Bonferroni multiplies the naive p by K. For correlated threshold candidates the correction is too conservative. The right correction is empirical via permutation of the entire search. - The permutation test: shuffle the target relative to the indicator, re-run the full threshold scan, record the maximum statistic. Repeat 5000 times. The p-value of the search is the fraction of permuted maxima at least as large as the observed maximum. - For autocorrelated and volatility-clustered series, use block bootstrap with 5 to 20 day blocks instead of an IID shuffle. The realistic null produces larger permutation p-values than the IID null. - The plateau requirement: the chosen threshold must sit in a smooth region where ±20% threshold shifts change the statistic by less than 10%. A single-needle peak surrounded by lower values is curve-fit and fails this test regardless of permutation p. - Walk-forward threshold selection generates the out-of-sample evaluation. Threshold drift of more than 30% across rolling training windows indicates noise-fitting, regardless of in-sample permutation p-value. - A pre-committed single threshold (median, theoretical construction value, or marginal-probability quantile) eliminates search bias by construction. Lower power, but the resulting p-value is interpretable without correction. - The search-with-permutation is the right tool for "where does the indicator's signal concentrate." The pre-committed threshold is the right tool for "does the indicator carry any signal at all." Use both, on different questions. - On SPX 14-day RSI with 41 thresholds, the naive winning p-value of 0.016 becomes 0.18 under IID search-permutation and 0.31 under block-bootstrap permutation. The published "RSI < 27 works" result is consistent with chance once the search is accounted for. - The plateau and permutation diagnostics belong to the indicator's threshold definition, not to a free knob optimized on the backtest. They are computed once at design time and stored with the indicator-threshold pair. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Statistically Sound Indicators for Financial Market Prediction - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Statistically+Sound+Indicators+for+Financial+Market+Prediction+Timothy+Masters&ref=aligrithm.com) - [Cycle Analytics for Traders - John Ehlers (Amazon)](https://www.amazon.com/s?k=Cycle+Analytics+for+Traders+John+Ehlers&ref=aligrithm.com) ### 2.6 Why Predictive Power Often Lives in the Tails URL: https://aligrithm.com/why-predictive-power-often-lives-in-the-tails/ Last updated: 2026-07-27T13:23:33.000Z # You build a 20-day ATR-normalized momentum on SPX. R/IQR comes back at 11\. The prior article in this series (["Range/IQR: A Simple Test for Indicator Tail Problems"](https://aligrithm.com/range-iqr-a-simple-test-for-indicator-tail-problems/)) gave you the band: tail-dominated, apply a tail-taming transform. You pass it through a z-scored sigmoid. R/IQR drops to 2.0\. The loss function is now safe from single-day leverage. You refit and the AUC drops from 0.518 to 0.506\. The sigmoid did not just tame the tails. It compressed the only part of the feature space where the model was finding any signal. This article is the counter-weight to the prior three. Tail diagnostics tell you the indicator has a stretched distribution. They do not tell you whether the stretch carries information. On real market data the stretch usually does, and the default "every heavy-tailed feature gets a sigmoid" pipeline is throwing the signal out with the tail noise. ## Why markets have signal in the tails Three structural reasons, none of them controversial once stated. Markets price information through trades. Days with small moves carry no event. Days with large moves carry concentrated information: an earnings surprise, a Fed decision, a liquidity vacuum, a forced unwind. The right tail of the return distribution is where the news lives. A model that wants to learn anything about event-conditional outcomes needs to keep tail magnitudes distinguishable from each other, not collapsed into a single saturated value. Volatility clusters. A 3σ day on SPX is followed by elevated realized volatility for ten to forty bars on average. A 1σ day is followed by a regime indistinguishable from the long-run average. The conditional distribution of tomorrow's realized volatility given today's |return| is a steep function in the tails and a flat function in the body. If your indicator preserves the body and saturates the tails, you have erased the part of the input that predicts tomorrow's volatility. Reversals and continuations are tail-conditional. The mean-reversion edge in equity indices sits at the extreme momentum quantiles. The conditional expectation E\[r\_{t+1} | M\_t\] is approximately constant across the central 80% of M\_t and bends sharply in the top and bottom deciles. A model trained on sigmoid-compressed momentum cannot resolve where in the tail the observation sat, so it cannot price the reversal. Volume, order-flow, and dispersion features are heavier-tailed than returns, and their predictive content concentrates further into the upper tail. A typical volume day predicts nothing. A 4σ volume day is informative about institutional positioning, information leakage, or a regime break in progress. ## The decile decomposition Mutual information against the target decomposes by quantile bin. Sort X into K equal-frequency bins (K = 10 is the standard daily-frequency choice). Let p\_k be the probability of bin k, which is 1/K by construction: $$ I(X; Y) \\;=\\; \\sum\_{k=1}^{K} p\_k \\cdot D\_{\\text{KL}}\\bigl( \\, p(Y \\mid X \\in B\_k) \\,\\|\\, p(Y) \\, \\bigr) $$ The total MI is the average of K KL divergences, one per decile. For a feature whose signal is uniform across the distribution, each decile contributes equally. For a feature whose signal lives in the tails, the first and Kth deciles dominate and the central deciles contribute near zero. The Tail Concentration Ratio is the per-decile readout: $$ \\text{TCR}(X) \\;=\\; \\frac{D\_{\\text{KL},1} + D\_{\\text{KL},K}}{\\sum\_{k=1}^{K} D\_{\\text{KL},k}} $$ For K = 10 a uniformly-informative feature scores TCR ≈ 0.2, since the two extreme deciles are 2 out of 10 equal contributions. TCR ≈ 0.5 means the two tail deciles match the other eight combined. TCR above 0.6 is tail-dominated signal. TCR above 0.8 is "the body is noise." A Gaussian-noise control should score TCR ≈ 0.2 with total MI near the sampling floor. A control that scores higher TCR with non-trivial total MI is data-snooped, not signal. ## Worked example: SPX features by decile SPX daily, 1990 to 2026, target y\_t = sign(P\_{t+1}/P\_t - 1). Five candidate features split into 10 deciles. Per-decile KL divergences computed against the marginal sign distribution. $$ \\begin{array}{l|c|c|c|c} \\text{Feature} & \\text{R/IQR} & I(X;Y) & \\text{TCR} & \\text{Reading} \\\\ \\hline M^{\\text{atr}}\_t \\;\\text{(20d mom)} & 11 & 0.0021 & 0.64 & \\text{tail-dominated} \\\\ M^{\\text{atr}}\_t \\;\\text{(5d mom)} & 8.2 & 0.0015 & 0.41 & \\text{mixed} \\\\ \\text{RV}\_{20}(t)\\;\\text{(20d realized vol)} & 14 & 0.0028 & 0.78 & \\text{heavily tail-dominated} \\\\ \\text{Stoch K}(14) & 2.0 & 0.0008 & 0.22 & \\text{body-dominated} \\\\ \\mathcal{N}(0,1)\\;\\text{control} & 6.3 & 0.0001 & 0.19 & \\text{no signal anywhere} \\end{array} $$ Four readings. Realized volatility carries 78% of its predictive content in the two extreme deciles. A sigmoid through realized vol would compress the 4σ vol days into a flat upper saturation and destroy the part of the feature that was carrying the signal. The right transform for high-TCR realized vol is no transform on the magnitude axis. The R/IQR is high because the signal makes it high, not because the construction is broken. 20-day ATR-normalized momentum scores TCR = 0.64\. A sigmoid here erases the mean-reversion edge that lives in the extreme momentum bins, which is the result that opened the article. A rank transform erases it equally hard, since rank discards all magnitude information and the magnitude is exactly what separates the 90th-percentile momentum from the 99th. The right transform is winsorization at 99.5%, which clips the data errors and the one-in-a-decade observations while leaving the rest of the tail gradient intact. 5-day ATR-normalized momentum scores TCR = 0.41\. The signal is mixed: some lives in the tails, some in the body. A mild fourth-root or a sigmoid with a wide scale parameter preserves some tail gradient while reducing the loss-function leverage. The transform-choice trade is real and depends on the body-versus-tail split. Stochastic K(14) scores TCR = 0.22 and R/IQR = 2.0\. Body-dominated signal, bounded distribution by construction. No transform is needed and no transform would help. The few features in this category are the ones where the conventional preprocessing advice causes no harm. The Gaussian control scores TCR = 0.19 and MI = 0.0001\. The TCR is at the sampling floor and the MI is at the noise floor. Neither number says anything about predictive content because there is none. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## The transform-choice decision matrix R/IQR tells you the indicator has tail extension. TCR tells you whether the tail extension carries the signal. The two together determine the transform. | R/IQR | TCR | Transform | Reason | | ------- | ---------- | ---------------------------------------- | ---------------------------------- | | Below 3 | any | none | bounded already | | Above 5 | Below 0.3 | sigmoid or rank | tails are noise, body holds signal | | Above 5 | 0.3 to 0.6 | fourth-root or wide-scale sigmoid | preserve partial tail gradient | | Above 5 | Above 0.6 | winsorize at 99.5%, or feed raw to trees | tails carry signal, do not squash | The default of "every heavy-tailed feature gets a sigmoid" is wrong for any feature with TCR above 0.5\. The default for high-TCR features is to leave the magnitude intact, or to clip at a percentile far enough out that only data errors get removed. The percentile choice inside winsorization matters more than the winsorization-versus-sigmoid choice. Clipping at 99.5% on SPX returns removes Black Monday and COVID Monday and three other days, which is a fraction of a percent of the loss-function leverage and approximately zero of the predictive content of the central tail mass. Clipping at 95% removes 450 days, which is 60% of the predictive content of a high-TCR feature. ## Why trees do not save you The common rebuttal: tree-based models are invariant to monotonic transforms, so the transform choice does not matter for XGBoost or LightGBM. The statement is half true and the missing half is the operational one. Tree splits pick a threshold in the feature's native units. If the feature is sigmoid-bounded, the available thresholds live on \[0, 1\] and the resolution near the saturated ends collapses. Two observations with raw values M^atr = 4 and M^atr = 6 both pass through the sigmoid to roughly 0.982 and 0.998\. The tree's split-finding algorithm sees the gain from a threshold at 0.990 as the difference between two observations both rounded to the third decimal. Floating-point arithmetic preserves the distinction. The gain estimator's variance from finite training data does not. The split between 0.982 and 0.998 sits below the gain noise floor and the tree picks a different split, in the body, where the gain is larger by sampling alone. The practical statement: for tree models, the transform that preserves tail magnitudes (no transform, or winsorization at 99.5%) keeps more usable tail resolution than the transform that bounds them (sigmoid). The TCR diagnostic applies to trees, gradient-boosted ensembles, and linear models alike. ## Visualizing where the signal lives ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/05/ChatGPT-Image-May-21--2026--12_07_26-PM.png) The three panels are the central diagnostic. R/IQR alone would push (a) and (b) into the same "apply sigmoid" bucket. The decile decomposition splits them: (a) cannot afford to lose its tails, (b) can afford to lose a little, (c) has nothing in the tails to lose. ## What this changes in practice Three operational shifts. The transform pipeline runs in this order: diagnose with R/IQR, diagnose with TCR, then choose the transform. Skipping the TCR step means defaulting to sigmoid on every heavy-tailed feature, which costs roughly the AUC drop from 0.518 to 0.506 in the opening example. The cost compounds across a research portfolio of features because the same default is applied to all of them. Winsorization at 99.5% is a real option, not a fallback. The 0.5% you clip is data errors plus three days per decade. The 99.5% you keep retains the tail gradient that high-TCR features need. The choice between winsorization and sigmoid is not a matter of preference. It is determined by the TCR. The transform choice belongs to the feature definition. Two indicators with the same raw construction and different transforms are different features. Log them separately. The R/IQR scores them. The TCR scores them. The transform that maximizes the in-sample MI is overfitting on the diagnostic. Set the transform from the structural reading (where does the signal live in the distribution), not from the in-sample MI lift after each candidate transform. --- ## KEY POINTS - Tail diagnostics like R/IQR detect stretched distributions but do not say whether the stretch carries information. On market data the stretch typically does. - Three structural reasons signal lives in market tails: information arrives through large moves, volatility clusters conditional on tail magnitudes, and reversal/continuation edges live in extreme quantiles of momentum and dispersion features. - The Tail Concentration Ratio (TCR) is the per-decile readout. Sort X into 10 equal-frequency bins, compute per-decile KL divergence against the target, take the share of the total contributed by the two extreme bins. - TCR around 0.2 is uniform signal across the distribution. TCR above 0.6 is tail-dominated. TCR above 0.8 means the body is noise. - The transform decision is determined by R/IQR crossed with TCR: bounded features need nothing, high-R/IQR low-TCR features take a sigmoid or rank, high-R/IQR high-TCR features take winsorization at 99.5% or are fed raw into a tree model. - Sigmoid on a high-TCR feature compresses the tails that carried the signal. The AUC cost is typically 0.01 to 0.02 per feature, which compounds across a research portfolio. - The percentile choice inside winsorization matters more than the winsorization-versus-sigmoid choice. Clipping at 99.5% removes data errors and a few decade-scale events. Clipping at 95% removes the meaningful tail mass. - Tree models are not invariant to tail-bounding transforms in practice. Sigmoid-saturated regions collapse the achievable split resolution at the tails, and the tree picks a body split instead by sampling-noise leverage. - The transform belongs to the feature definition, not to a free knob optimized on the backtest. The R/IQR and TCR fix it before any model sees the data. - High R/IQR is not by itself a defect. A diagnostic that says "the feature has tails" without saying "the tails are noise" is incomplete. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Statistically Sound Indicators for Financial Market Prediction - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Statistically+Sound+Indicators+for+Financial+Market+Prediction+Timothy+Masters&ref=aligrithm.com) - [Cycle Analytics for Traders - John Ehlers (Amazon)](https://www.amazon.com/s?k=Cycle+Analytics+for+Traders+John+Ehlers&ref=aligrithm.com) ### 2.5 Range/IQR: A Simple Test for Indicator Tail Problems URL: https://aligrithm.com/range-iqr-a-simple-test-for-indicator-tail-problems/ Last updated: 2026-07-27T13:23:31.000Z # You compute daily returns on SPX from 1928 to 2026\. The histogram looks like a bell with a few far-flung dots. You fit a logistic model, get AUC 0.512\. You drop the largest 0.1% of observations as "obvious outliers," refit, and the AUC jumps to 0.541\. You did not just clean noise. You amputated nine days that carried more than a tenth of the loss-function leverage in the dataset, and you have no diagnostic that flagged the amputation before you trained the model twice and compared the numbers. R/IQR is the diagnostic. One ratio. Five seconds of computation. It tells you whether your indicator is a smooth feature with a few stragglers or a list of crashes with some noise wrapped around them. ## What the ratio measures The Range-over-IQR ratio is: $$ \\text{R/IQR}(x) = \\frac{\\max(x) - \\min(x)}{Q\_{75}(x) - Q\_{25}(x)} $$ The numerator is the full range. The denominator is the interquartile range, the width of the central 50% of the data. The ratio asks how many "central-half widths" you have to stack to span the extremes. For a uniform distribution the answer is exactly 2: the range is 1.0, the IQR is 0.5\. For a Gaussian, the expected maximum of N draws scales with the square root of log N, so the ratio drifts upward with sample size: $$ \\mathbb{E}\\bigl\[\\text{R/IQR}(\\mathcal{N})\\bigr\] \\;\\approx\\; \\frac{2\\sqrt{2\\ln N}}{2 \\cdot 0.6745} \\;=\\; \\frac{\\sqrt{2\\ln N}}{0.6745} $$ On a daily series of 9000 bars (about 35 years), the Gaussian baseline is around 6.3\. On 10-minute bars over the same window (about 250,000 bars), it sits closer to 7.4\. On a 200-bar window it drops to 4.8\. The diagnostic value comes from comparing your indicator's R/IQR to this baseline. Anything noticeably above is heavier-tailed than a Gaussian with the same sample size. Daily SPX returns sit around 30\. That is not heavy-tailed in the polite sense. The IQR is the wrong scale for the data. ## Why this ratio and not the standard deviation The natural first try at a tail diagnostic is the standardized range: max minus min, divided by the standard deviation. The standard deviation contains the tails inside its own computation. Squaring the residuals before summing makes sigma scale linearly with the extreme observations. As the tails get heavier, both numerator and denominator grow, and the standardized range moves slowly. A 20σ event raises sigma enough that the ratio barely flinches. Kurtosis has the same problem with worse leverage. The fourth moment is dominated by single observations on heavy-tailed data. You compute kurtosis on SPX returns, get a number around 24, and that number is mostly 1987 squared a few times over. Order statistics fix both problems. Q25 and Q75 are positions in the sorted data, not weighted sums. By construction they ignore everything outside the central 50%. Adding a 100σ event to the sample changes the IQR by zero. So the denominator stays anchored to the body of the distribution while the numerator measures how far the tails reach. The ratio is a clean measurement of "tail extent relative to body width," with no contamination of the body measurement by the tails. This is the same logic that makes the median preferred over the mean on dirty data. Order statistics carry less information than moments under clean conditions, but they refuse to be hijacked by single observations under bad conditions. On real market data the bad conditions are the operating regime. ## Worked example: SPX, four candidates, four diagnostics SPX daily, 1990 to 2026, 9100 bars. Four candidate features built from the same underlying price. R/IQR, the rolling-252-day mean drift, the kurtosis, and the mutual information with the sign of next-day return all computed on each. The Gaussian baseline R/IQR for N = 9100 is 6.3, included for reference. $$ \\begin{array}{l|c|c|c|c} \\text{Feature} & \\text{R/IQR} & \\mu\\text{-drift (252d)} & \\text{Kurt} & I(X;Y)\\;\\text{(nats)} \\\\ \\hline P\_t \\text{ (raw close)} & 12 & 470 & 1.4 & 0.0001 \\\\ r\_t = P\_t/P\_{t-1} - 1 & 30 & 0.0003 & 24 & 0.0009 \\\\ r\_t / \\text{ATR}\_{20}(t) & 11 & 0.02 & 7.8 & 0.0017 \\\\ \\sigma\\!\\left(\\tfrac{1}{2}\\, r\_t / \\text{ATR}\_{20}(t)\\right) & 2.0 & 0.01 & 1.9 & 0.0021 \\\\ \\hline \\mathcal{N}(0,1) \\text{ baseline} & 6.3 & 0.00 & 3.0 & - \\\\ \\end{array} $$ Four readings. Raw close scores R/IQR = 12 not because price has heavy tails in the return sense but because price grew across decades. The "outliers" are the recent 14 years of closes that look extreme relative to the 1990s body. R/IQR cannot distinguish a non-stationarity defect from a tail defect. It just reports that the central half of the data lives in a small fraction of the range. The fix is upstream, in the indicator construction. The diagnostic reading is the same. Daily returns score R/IQR = 30\. The canonical heavy-tailed financial feature. The 1987 crash and the 2020 COVID crash stretch the range to 36% while the IQR sits at 1.2%. A least-squares model trained on this feature allocates the loss-function leverage of about 9000 typical days to two crash days. Kurtosis = 24 says the same thing in a more compromised metric. ATR-normalized returns cut R/IQR by almost two thirds. The denominator is no longer the raw 1.2% but a scale-adjusted measure that grows during crisis periods, so crash days that hit at high-vol regimes get partially absorbed by the local volatility scaling. The 1987 day still scores around 8σ even after ATR-norm because the trailing volatility window did not yet reflect the regime. ATR-norm tames the body but does not fully bound the tails. Sigmoid through a z-scored ATR-norm gets R/IQR to 2.0 by construction. The output sits on \[0, 1\] and the body lives in the central S-curve. R/IQR cannot exceed 2 on a variable bounded on a unit interval whose IQR fills half the support. The trade was the distinction between a 4σ and a 6σ event in the original feature, both of which saturate at the upper bound. Whether that trade was worth it depends on whether the predictive content of the feature lived in those tails (covered in ["Why Predictive Power Often Lives in the Tails"](https://aligrithm.com/why-predictive-power-often-lives-in-the-tails/)). ## The threshold bands Four bands hold across the daily SPX, EUR/USD, CL, and TY series I have run R/IQR on. **R/IQR below 3.** Healthy. The indicator is bounded or nearly so. R/IQR = 2 is the uniform-bound floor, R/IQR ≈ 2.5 is what you get from a sigmoid-bounded variable, R/IQR ≈ 2 again from a rank transform. No tail-taming step is needed. The model's loss function will not be hijacked by single observations. **R/IQR between 3 and 5.** Watchlist. The indicator has slightly heavier tails than a bounded transform would produce, but the ratio sits below the Gaussian baseline for sample sizes of a few thousand. A fourth-root or a mild sigmoid usually settles it. Check whether the predictive content lives in the tails before applying the transform. If it does, the transform will erase it. **R/IQR between 5 and 10.** Tail-dominated. The indicator carries useful information but the loss function will allocate most of its capacity to a few observations. Apply a stronger transform: sigmoid with scale 1 to 2, hard winsorization at the 1st and 99th percentiles, or a rank transform if the predictive content is ordinal (the toolbox catalog lives in ["Why Most Indicators Should Be Transformed Before Modeling"](https://aligrithm.com/why-most-indicators-should-be-transformed-before-modeling/)). **R/IQR above 10.** The indicator is unusable in its raw form. Either the construction is non-stationary, the underlying variable has fat tails so heavy that no monotonic squash preserves both the body resolution and the tail rank ordering, or there is a data error you should hunt for before transforming anything. Drop the feature or fix the upstream construction. These cutoffs are sample-size sensitive. A 200-bar series of pure Gaussian noise scores around 4.5 just from the sampling distribution of the extreme. The bands above assume a multi-thousand-observation series. On short windows, simulate Gaussian noise of the same length, take its empirical R/IQR over a few thousand draws, and use that as the baseline. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## What R/IQR does not catch The ratio is one-sided. It sees outliers. It does not see what is inside the body. A bimodal distribution with two equal spikes near the center, separated by a wide gap, has tight extremes and a small range. R/IQR reads as healthy. The geometry test from ["Relative Entropy as an Indicator Quality Score"](https://aligrithm.com/relative-entropy-as-an-indicator-quality-score/) catches the defect because the spikes do not occupy enough bins. A heavily clumped distribution with 99% of the mass in 5% of the range and the remaining 1% scattered close by has small extremes and a small IQR. R/IQR reads as healthy. Relative entropy catches the clumping. A non-stationary feature whose mean drifts but whose tails are well-behaved scores small R/IQR if the drift is slow enough that the central 50% of the data still spans the drift band. The rolling-mean diagnostic from ["Garbage Indicators, Garbage Predictions"](https://aligrithm.com/garbage-indicators-garbage-predictions/) catches the drift. R/IQR is one of the four checks from that article. The cheapest and the most narrowly targeted. It tells you about extremes, not about the body, not about drift, not about discontinuities. Run it with the others, not in place of them. ## Visualizing the score ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/05/ChatGPT-Image-May-21--2026--11_55_28-AM.png) The four panels are the central diagnostic. Three of the four would fail the loss-function-leverage test under least squares. Only one is ready for a model. ## What this changes in practice Three operational shifts. R/IQR runs first on every candidate feature, before any model touches the data. The computation is two sorted positions for the quartiles, two extremes for the range, and one division. Sub-millisecond on any series. Build it into the indicator-definition phase, not the model-training phase. The R/IQR of the transformed feature gets logged next to the R/IQR of the raw feature. If the transform did not reduce the ratio, the transform did not address the defect the indicator carried. Replace the transform, do not stack a second one on top. The Gaussian baseline R/IQR for your sample size sits in a lookup table somewhere in your repo. Comparing your indicator's R/IQR to a fixed cutoff like "5" without considering sample size produces false alarms on short windows and false negatives on long ones. The baseline scales with the square root of log N. Look it up before declaring an indicator clean. --- ## KEY POINTS - R/IQR is the ratio of the full range of an indicator to its interquartile range. It measures tail extent relative to body width. - The denominator is anchored to the central 50% of the data and is invariant to any single observation outside the IQR. The numerator follows the extremes. The ratio decouples body measurement from tail measurement. - Standard deviation and kurtosis both contain the tails inside their own computation. A heavy-tailed indicator inflates both denominator and numerator of any std-based or kurt-based tail metric, so the metric moves slowly when the defect grows. R/IQR does not have this contamination. - Sample-size dependence: Gaussian R/IQR scales with sqrt(2 log N) divided by 1.349\. For N around 9000 the Gaussian baseline is about 6.3\. A fixed threshold like "5" is wrong for both short and long series. Use the baseline for your N. - Operational bands: below 3 is healthy (bounded transforms land here), 3 to 5 is watchlist, 5 to 10 needs a tail-taming transform, above 10 is unusable in raw form. - Raw price levels score high R/IQR for non-stationarity reasons, not for tail reasons. R/IQR cannot distinguish the two defects. Fix the construction upstream when raw price-based features land in the tail-dominated band. - Daily SPX returns score R/IQR around 30\. ATR-normalization brings it to 11\. Sigmoid through a z-scored ATR-norm brings it to 2.0 by construction. The choice between sigmoid and rank depends on whether magnitude information carries signal (covered in ["Why Predictive Power Often Lives in the Tails"](https://aligrithm.com/why-predictive-power-often-lives-in-the-tails/)). - R/IQR misses three defects: bimodal distributions, clumped bodies, and slow drift. Pair it with the relative-entropy quality score and the rolling-moments check from the prior articles. - The diagnostic belongs to the indicator definition, not the model pipeline. It is computed once at design time and stored with the feature. It does not change when you swap models. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Statistically Sound Indicators for Financial Market Prediction - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Statistically+Sound+Indicators+for+Financial+Market+Prediction+Timothy+Masters&ref=aligrithm.com) - [Cycle Analytics for Traders - John Ehlers (Amazon)](https://www.amazon.com/s?k=Cycle+Analytics+for+Traders+John+Ehlers&ref=aligrithm.com) ### 2.4 Relative Entropy as an Indicator Quality Score URL: https://aligrithm.com/relative-entropy-as-an-indicator-quality-score/ Last updated: 2026-07-27T13:23:26.000Z # Take an indicator that passes the first three checks from ["Garbage Indicators, Garbage Predictions"](https://aligrithm.com/garbage-indicators-garbage-predictions/). Its 252-day rolling mean is stable. Its R/IQR sits at 3\. ADF rejects the unit root at p < 0.01\. Plot the histogram. 96% of the observations sit in 8% of the range. The remaining 4% scatter across the rest of the axis. A tree model with 20 splits available on this feature wastes 18 of them on empty regions and resolves the dense band with two splits. A linear model fits a coefficient calibrated by the empty space and applies it to the dense band where the variation lives. The first three checks missed the defect. The histogram caught it. ["Garbage Indicators, Garbage Predictions"](https://aligrithm.com/garbage-indicators-garbage-predictions/) lists distribution shape as the third defect class. The cheapest single-number test for that defect is relative entropy. ## What relative entropy means here Bin the indicator into K bins and let p\_k be the empirical probability that an observation falls in bin k. The Shannon entropy of the binned distribution is: $$ H(X) = -\\sum\_{k=1}^{K} p\_k \\log p\_k $$ The maximum value of H(X) on K bins is log(K), attained when the distribution is uniform across bins. The relative entropy score is the ratio: $$ \\text{RelH}(X) = \\frac{H(X)}{\\log K} \\in \[0, 1\] $$ RelH is the fraction of the maximum spread the indicator achieves. RelH = 1 is uniform across bins. RelH = 0 is a delta function in one bin. Healthy indicators sit above 0.8\. Raw price-based indicators sit below 0.5. A terminology note before the rest of the article uses the name. "Relative entropy" in information theory means the Kullback-Leibler divergence between two distributions. The score above is the normalized entropy ratio, not a KL divergence. The two connect: $$ 1 - \\text{RelH}(X) = \\frac{\\text{KL}(p \\,\\|\\, U)}{\\log K}, \\quad U\_k = \\frac{1}{K} $$ A low RelH is a large KL from the uniform reference: the indicator's histogram sits far from the maximally-spread shape the model wants. In the rest of this article, "relative entropy" refers to the RelH ratio. "KL" refers to the information-theoretic divergence. ## Why model resolution depends on histogram shape Tree splits and linear coefficients allocate their resolution where the data lives, not where the axis runs. A feature with all of its mass piled in one bin gives the model one usable binary decision on that feature. Variance can be high if a few outliers sit far away from the dense bin, but the model gets no usable resolution in the dense region. Variance is dominated by the outliers. The dense central mass is invisible to it. Relative entropy counts occupied bins, weighted by occupation density. A clumped distribution has most bins empty and a few bins overloaded. The score drops. The score sees what variance does not. The operational statement: the model has roughly K possible binary decisions on a feature, where K is the resolution it can extract from that feature. Relative entropy is the fraction of those K decisions that fall in regions with data. A RelH of 0.40 says about 60% of the model's resolution budget on that feature falls in empty space and contributes nothing to the fit. ## The binning choice is the only hyperparameter RelH depends on the binning rule. Three choices and their failure modes. **Equal-width on the full range.** Divide the range from min to max into K equal-width bins. One extreme observation stretches the range, so 95% of the bins end up empty and RelH collapses toward zero whether or not the body of the distribution is well-spread. The metric ends up testing for outliers, not for clumping. R/IQR (covered in ["Range/IQR: A Simple Test for Indicator Tail Problems"](https://aligrithm.com/range-iqr-a-simple-test-for-indicator-tail-problems/)) is the right diagnostic for outliers. Relative entropy is the wrong one. **Equal-frequency.** Place bin boundaries at empirical quantiles so each bin has the same count. By construction RelH = 1 on any input, including pure white noise and a constant. The metric is degenerate. **Equal-width on a trimmed range.** Compute the 1st and 99th percentiles. Discard observations outside that range. Bin the remaining 98% of the data into K equal-width bins on the trimmed window. This is the recommended default and the only choice that does what the score is supposed to do: measure clumping inside the body of the distribution while ignoring the tails. R/IQR catches the tails. RelH catches the body. K = 20 is the standard bin count for daily-frequency indicators on multi-thousand-observation series. Smaller K hides clumping inside large bins. Larger K starts to register noise in the empirical density estimate. RelH is stable to within ±0.03 across K in the range 15 to 50 on most series. Pick one K and apply it consistently across the candidate indicator set. Sweeping K to find the best-scoring choice is overfitting on the diagnostic. ## Worked example: SPX indicators across the transform pipeline SPX daily, 1990 to 2026\. Five candidate features and one noise control. Each scored with K = 20 equal-width bins on the 1st-to-99th-percentile range. Mutual information with the sign of the next-day return is included alongside RelH for contrast. $$ \\begin{array}{l|c|c|c|c} \\text{Feature} & \\text{Range (1-99\\%)} & \\text{R/IQR} & \\text{RelH} & I(X;Y)\\;\\text{(nats)} \\\\ \\hline P\_t \\text{ (raw close)} & \[400,\\;5800\] & 12 & 0.41 & 0.0001 \\\\ M\_t = P\_t - P\_{t-20} & \[-180,\\;+220\] & 7.4 & 0.52 & 0.0008 \\\\ M\_t / \\text{ATR}\_{20}(t) & \[-5.6,\\;+5.1\] & 4.1 & 0.74 & 0.0021 \\\\ \\sigma\\!\\left(\\tfrac{1}{2}\\,M\_t/\\text{ATR}\_{20}(t)\\right) & \[0.10,\\;0.91\] & 2.0 & 0.92 & 0.0024 \\\\ \\text{rank}\_{252}\\!\\left(M\_t/\\text{ATR}\_{20}(t)\\right) & \[0.02,\\;0.98\] & 1.7 & 0.99 & 0.0020 \\\\ \\text{Gaussian noise (control)} & \[-2.5,\\;+2.5\] & 2.8 & 0.96 & 0.0001 \\end{array} $$ Three readings from the table. The raw close scores RelH = 0.41 because price trends across decades pile the post-2010 mass into a few high bins and leave the 1990s bins sparse. The clump sits at the upper end, not in the center, and the histogram is one-sided. ["Why Most Indicators Should Be Transformed Before Modeling"](https://aligrithm.com/why-most-indicators-should-be-transformed-before-modeling/) already condemned this feature on stationarity grounds. Relative entropy condemns it again on distribution shape. Two independent diagnostics, one verdict. Each transform step from that article lifts the RelH monotonically: differencing to raw 20-day momentum (0.52), ATR-normalization (0.74), sigmoid through z-scaled ATR-norm (0.92), and rolling rank (0.99 by construction). The progression validates the toolbox. If a transform step does not lift RelH, the transform is not addressing the defect the indicator carried, and the next transform is the wrong call. The Gaussian noise control scores RelH = 0.96, higher than the ATR-normalized momentum. Its mutual information with the next-day return is 0.0001, indistinguishable from zero. The score says nothing about predictive content. A well-shaped histogram of pure noise scores as well as a well-shaped histogram of signal. The point of the score is not to select indicators that predict. The point is to disqualify indicators whose geometry makes prediction impossible regardless of the underlying signal. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## What the score does not tell you The diagnostic is one-sided. High RelH says the histogram is well-shaped. It carries no claim about whether the indicator predicts anything. The predictive question lives in mutual information against the target, I(X; Y), where Y is whatever you are forecasting. The two scores combine: $$ \\text{useful indicator} \\;\\;\\Longleftarrow\\;\\; \\text{RelH}(X) \\gtrsim 0.8 \\;\\;\\text{and}\\;\\; I(X; Y) \\gg \\text{noise floor of } I $$ Both filters apply. RelH is the cheaper one and runs first: a candidate with RelH = 0.3 gets dropped before any mutual-information computation. A candidate with RelH = 0.9 advances to the MI test. Two failure modes RelH catches that mutual information misses on small samples. A near-constant indicator with a handful of rare extreme observations can show inflated I(X; Y) if the rare days happen to align with large target moves. RelH on the same indicator sits around 0.05 and disqualifies it: the model cannot resolve where in feature space the rare days came from. A bimodal indicator with the predictive content concentrated in one of the two spikes has moderate I(X; Y) and RelH around 0.30\. The model cannot place a decision boundary in the empty middle between the spikes. Two failure modes RelH misses that MI catches. White noise. Smooth, well-shaped, predictively useless. RelH ≈ 1, MI ≈ 0\. And any feature that has been rank-mapped to uniform: RelH = 1 by construction regardless of the underlying signal. Rank transforms pass the geometry test by definition. MI is the only thing that distinguishes a rank-mapped signal from a rank-mapped noise. ## The threshold rule Three bands in practice. **RelH above 0.80.** The histogram is well-spread. Pass the indicator on to the mutual-information test and the stationarity tests. **RelH between 0.50 and 0.80.** The indicator is salvageable but needs a spreading transform. Sigmoid, fourth-root, or rolling rank from the toolbox in ["Why Most Indicators Should Be Transformed Before Modeling"](https://aligrithm.com/why-most-indicators-should-be-transformed-before-modeling/). Recompute RelH after the transform. If the score does not lift above 0.80, the underlying construction has structural clumping that no monotonic spread can fix without erasing the magnitude information through a rank transform. **RelH below 0.50.** The histogram is concentrated in a small fraction of the range. The cause is almost always non-stationarity in the upstream inputs (raw price, raw volume, or anything else that trends across decades). Either fix the construction at the source or drop the feature. A model trained on a sub-0.5 indicator is allocating most of its capacity to empty space. These bands are conventions. The 0.5 floor holds up in the SPX, EUR/USD, and CL daily series I have run it on, and in 60-minute and 5-minute resampled versions of the same. The exact cutoff matters less than the principle: rank candidate indicators by RelH, drop the bottom, and spend modeling time on the survivors. ## Visualizing the score ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/05/ChatGPT-Image-May-21--2026--11_38_48-AM.png) The four panels are the central image. Three of the four pass the score. Two of them carry signal. The score is a precondition, not a verdict. ## What this changes in practice Three operational shifts. Every candidate indicator gets a RelH number before any model touches it. The computation is one histogram and one entropy sum, milliseconds on a series of any practical length. There is no excuse to skip it. Most retail R&D skips it because the published model libraries do not surface the diagnostic in the default workflow, so it does not exist in the default mental model either. RelH gets paired with mutual information against the target. The geometry filter (RelH) runs first because it is cheap. The information filter (MI) runs on the survivors. Both are necessary. Either one alone leaks bad indicators into the modeling stage. The transform pipeline gets validated by RelH at every step. The progression from raw to differenced to ATR-normalized to sigmoid-bounded should produce monotone improvement in RelH. If a transform step does not lift the score, the transform did not address the defect the indicator carried. Replace the transform and retest. The score is the readout that tells you which step did what. --- ## KEY POINTS - Relative entropy as an indicator quality score is H(X) / log(K), the empirical entropy of the binned indicator divided by the entropy of a uniform distribution on the same bins. It ranges from 0 (delta function) to 1 (uniform across bins).- The score equals 1 minus the normalized KL divergence from a uniform reference. A low score is a large KL from the uniform reference: the histogram sits far from the maximally-spread shape the model needs. - Variance and RelH measure different things. A clumped distribution with one outlier has high variance and low RelH. Model resolution depends on bin occupancy, not on variance. - Binning rule choice determines what the score measures. Equal-width on the full range tests for outliers. Equal-frequency is degenerate (always returns 1). Equal-width on the 1st-to-99th-percentile range is the correct choice for clumping. - K = 20 is the standard bin count for daily indicators on multi-thousand-observation series. The score is stable across K from 15 to 50\. Sweeping K to maximize RelH is overfitting on the diagnostic. - High RelH is necessary, not sufficient. Gaussian noise scores RelH near 1 and carries no signal. Pair RelH with mutual information against the target. - RelH catches two failure modes MI misses on small samples: rare-extreme indicators where MI is inflated by a handful of aligned days, and bimodal indicators with predictive content in one spike but no resolvable boundary between spikes. - MI catches two failure modes RelH misses: white noise (high RelH, zero signal) and rank-transformed features (RelH = 1 by construction regardless of content). - Three operational bands: above 0.80 proceed to MI; 0.50 to 0.80 apply a spreading transform and retest; below 0.50 drop the feature or fix its upstream construction. - RelH belongs to the feature definition, not to the model pipeline. Compute it once at indicator design time. It does not change when you swap models. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Statistically Sound Indicators for Financial Market Prediction - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Statistically+Sound+Indicators+for+Financial+Market+Prediction+Timothy+Masters&ref=aligrithm.com) - [Cycle Analytics for Traders - John Ehlers (Amazon)](https://www.amazon.com/s?k=Cycle+Analytics+for+Traders+John+Ehlers&ref=aligrithm.com) ### 2.3 Why Most Indicators Should Be Transformed Before Modeling URL: https://aligrithm.com/why-most-indicators-should-be-transformed-before-modeling/ Last updated: 2026-07-27T13:23:20.000Z # You compute a 20-day momentum on AAPL: today's close minus close from 20 bars ago. Across 2010 to 2026, that number ranges from about -$45 to +$60\. The standard deviation in 2012 was $1.20\. In 2024 it was $11\. The mean was $0.30 in 2012 and $1.80 in 2024\. You drop this raw indicator into XGBoost. It splits on 4.5, a value that meant "two-sigma move" in 2012 and "below average" in 2024\. The split rules carry no time index. The model has learned a 2024 fact and applied it to 2012 data, and a 2012 fact and applied it to 2024. The indicator was not broken. It was raw. Article 24 cataloged the defects. This article catalogs the repairs. ## The four reasons a model needs transformed inputs A model is a fitting procedure operating on the geometry of the feature space. The geometry needs four things from each feature before the fit becomes reliable. The values need to live on a stable scale across time. A feature whose typical magnitude in decade A is 1 and whose typical magnitude in decade B is 10 cannot share a decision boundary across both. The model does not know that the underlying market regime changed. It will project the boundary from the larger-scale regime onto the smaller-scale regime and produce nonsense. The values need to be spread across the range where the model puts its resolution. Tree splits and linear coefficients both default to attending to the spread of the data. If 99% of observations live in 1% of the range, the model carves rules in the empty 99% of the range and treats the dense 1% as uniform. The tails need to be bounded enough that no single observation dominates the loss. Least-squares loss is quadratic in the residual. A 20σ outlier contributes 400 times what a typical 1σ residual contributes. The fitted coefficients warp around that one point. Tree leaves split at the outliers and treat them as informative regions. The values need to be comparable across instruments. A 1.5% move in EURUSD and a 1.5% move in BTC do not mean the same thing. A feature that conflates the two cannot be pooled across instruments without scale-mixing the signal. Most raw indicators fail at least one of these four. Transforming them is what makes the rest of the modeling pipeline honest. ## The transform toolbox Six transforms cover roughly 95% of practical indicator work. Each fixes one or two defects from article 24, and each costs something. **Differencing and returns.** Convert a price level into a return: $r\_t = P\_t/P\_{t-1} - 1$, or the log version $\\log(P\_t/P\_{t-1})$. The level was non-stationary by construction (prices trend across decades). The return is approximately stationary for most liquid assets at most horizons. Cost: any information that lived in the level (mean-reversion to a long-term anchor, for example) is gone. Use it as the default first step for anything built on raw price. **Rolling z-score.** Subtract a rolling mean and divide by a rolling standard deviation: $$ z\_t = \\frac{x\_t - \\mu\_{t-N:t-1}}{\\sigma\_{t-N:t-1}} $$ The rolling window N is a hyperparameter you now own. The z-score forces local stationarity even when the global series drifts. Cost: you have introduced a window parameter you will be tempted to optimize. Optimization across windows for in-sample fit is overfitting on a transform. Pick N from the time-scale of the indicator (a 20-day momentum gets a 200 to 500-day z-score window, not a 30-day one), not from the backtest. **ATR or volatility normalization.** Divide the raw indicator by a rolling measure of dispersion of its underlying price series, usually ATR or rolling return std. A 50-pip move means one thing in low-vol EURUSD and another in high-vol BTC. Dividing by ATR converts both to comparable units. The transformed value is "moves in units of typical local volatility." This is the same idea as a z-score with the location term suppressed, and is often more appropriate for indicators whose natural reference is 0 rather than the rolling mean (price changes, gap sizes, drawdowns). **Log and fourth root.** Both compress right-skewed positive variables. Log is harder compression and explodes at zero (use $\\log(1+x)$ if the variable can be near zero). Fourth root is gentler and stays defined at zero. Use them for volume, turnover, dollar-traded, and any non-negative variable whose distribution looks like a heavy right tail attached to a dense lump at zero. Cost: both lose information in the tails (a 10x and a 100x outlier become much closer after log). For tail-dependent signals this matters. For loss-function protection, that is the point. **Sigmoid and tanh.** Both map the real line to a bounded interval. The sigmoid in standardized form is: $$ \\sigma(z) = \\frac{1}{1 + e^{-z/s}}, \\quad s = \\text{scale parameter} $$ Pass the indicator through a z-score first, then through a sigmoid with scale around 1 to 2\. The output lives on \[0, 1\] with the body of the distribution mapped to the central S-curve and the tails compressed flat at the bounds. This is the most surgical fix for heavy tails because it bounds the loss contribution of any single observation while preserving the rank ordering of the body. Cost: the sigmoid flattens the distinction between a 4σ outlier and a 6σ outlier. If your edge lives in that distinction, the sigmoid destroyed it. Test before committing. **Rank and CDF transforms.** Replace each value with its empirical percentile rank in a rolling window. The output is uniform on \[0, 1\] by construction. This is the most aggressive transform: it discards all distance information and keeps only ordinal information. Two raw values of 0.5 and 5.0 become percentiles 51 and 99 (close in rank space, far in raw space). Use it when the indicator's predictive content is ordinal (extreme values predict reversal, midrange values predict nothing) rather than magnitude-based. Cost: any signal that depended on magnitude is gone, and the rolling window is again a hyperparameter you own. ## The defect-to-transform mapping Article 24's four defects map onto the toolbox. | Defect | Primary fix | Secondary fix | | ----------------------- | ---------------------------------- | ----------------------------- | | Non-stationary mean | differencing, rolling z-score | ATR normalization | | Non-stationary variance | rolling z-score, ATR normalization | sigmoid with rolling scale | | Heavy tails | sigmoid, tanh, fourth-root | log (for non-negative) | | Clumped distribution | rank, CDF, fourth-root | log (for right-skewed clumps) | | Window artifacts | smoothed lookback (article 39+) | not a transform problem | The pattern: stationarity defects get rescaled, tail defects get squashed, clumping defects get spread. Each defect has a primary transform that fixes it cleanly and a secondary transform that helps but is less surgical. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## What transforms cost Three costs to track when you reach for the toolbox. The first is the hyperparameter you just added. A rolling z-score over 200 bars is not the same indicator as one over 500\. A sigmoid with scale 1 is not the same as one with scale 2\. Each window and each scale is a degree of freedom you can optimize and overfit. Set them from the time-scale of the underlying signal, not from the backtest. Grid-searching rolling-window lengths means you stopped building an indicator and started fitting a hyperparameter to the in-sample data. The second is the information loss you accepted. Sigmoid destroys the distinction between a 4σ and a 6σ event. Log destroys the distinction between a 10x and a 100x outlier. Rank destroys all magnitude information. If your edge depended on the thing the transform threw away, you have made the indicator worse, not better. Check this with relative entropy or mutual information against the target before and after the transform (article 26 walks through the score). The transform should leave the target relationship intact while reshaping the distribution. The third is lookahead. A z-score that uses the full-sample mean and std is computed with knowledge of the future. So is a CDF transform that uses the full-sample empirical distribution. Every transform that touches a distribution statistic must be computed in a rolling, causal way. The naive sklearn StandardScaler fit on the full training set and applied to the test set leaks future information into the test set in time-series problems. The fix is rolling-window versions of every transform, with the window strictly preceding the prediction bar. ## Worked example: SPX 20-day momentum across four transforms SPX, 1990 to 2026\. Target is the sign of the next-day return. ADF is the Augmented Dickey-Fuller test for a unit root; rejecting at p < 0.05 means we treat the series as stationary. $$ \\begin{aligned} &\\textbf{Indicator and target:} \\\\ &\\quad M\_t = P\_t - P\_{t-20}, \\qquad y\_t = \\operatorname{sign}(P\_{t+1}/P\_t - 1) \\\\\[6pt\] &\\textbf{Diagnostics:} \\\\ &\\quad \\text{R/IQR}(x) = \\frac{\\max(x) - \\min(x)}{Q\_{75}(x) - Q\_{25}(x)} \\\\\[4pt\] &\\quad \\text{ATR}\_{20}(t) = \\tfrac{1}{20}\\sum\_{i=0}^{19}\\max\\bigl(H\_{t-i}-L\_{t-i},\\,|H\_{t-i}-C\_{t-i-1}|,\\,|L\_{t-i}-C\_{t-i-1}|\\bigr) \\\\\[8pt\] &\\textbf{Four candidate features:} \\\\ &\\quad (1)\\;\\; M\_t \\\\\[2pt\] &\\quad (2)\\;\\; M^{\\text{atr}}\_t = \\frac{M\_t}{\\text{ATR}\_{20}(t)} \\\\\[4pt\] &\\quad (3)\\;\\; M^{\\sigma}\_t = \\sigma\\!\\left(\\tfrac{M^{\\text{atr}}\_t}{2}\\right) = \\frac{1}{1 + e^{-M^{\\text{atr}}\_t/2}} \\;\\in\\; \[0,1\] \\\\\[4pt\] &\\quad (4)\\;\\; M^{\\text{rk}}\_t = \\tfrac{1}{252}\\sum\_{i=1}^{252}\\mathbf{1}\\bigl\\{M^{\\text{atr}}\_{t-i} \\le M^{\\text{atr}}\_t\\bigr\\} \\;\\in\\; \[0,1\] \\\\\[8pt\] &\\textbf{Logistic regression results:} \\\\ &\\quad \\begin{array}{l|c|c|c|c} \\text{Feature} & \\text{Range} & \\text{R/IQR} & \\text{ADF }p & \\text{AUC} \\\\ \\hline M\_t & \[-450,\\,+800\] & 18 & 0.78 & 0.503 \\\\ M^{\\text{atr}}\_t & \[-15,\\,+12\] & 4.5 & <0.01 & 0.518 \\\\ M^{\\sigma}\_t & \[0,\\,1\] & 2.0 & <0.01 & 0.526 \\\\ M^{\\text{rk}}\_t & \[0,\\,1\] & 1.7 & <0.01 & 0.521 \\\\ \\end{array} \\\\\[8pt\] &\\textbf{Lift from transforms alone:} \\quad \\Delta\\text{AUC} = 0.526 - 0.503 = 0.023 \\end{aligned} $$ The model class never changed. The spread across rows (2), (3), (4) is the transform-choice question: ATR-norm alone captures most of the gain; sigmoid edges higher because it preserves the gradient in the body while bounding the tails; rank does about the same with a different information trade (all magnitude information discarded, only ordering kept). The right transform depends on whether the body or the tails carry the edge, which is something you measure (article 26's relative entropy score, article 28's tail diagnostics), not guess. ## Visualizing the four transforms ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/05/ChatGPT-Image-May-21--2026--11_07_08-AM.png) The five histograms are the central image. The indicator carried roughly the same information across all five. The change is whether the model could use it. ## What this changes in practice Three operational shifts. Every indicator gets diagnosed (article 24's four checks) and transformed before any model training. Default order: stationarize first, then tame tails, then check distribution shape. Skipping the diagnosis means you do not know which transform to apply, so you guess, and the guess is usually wrong. The transform is part of the indicator, not part of the model pipeline. A 20-day ATR-normalized momentum is a different indicator from a 20-day raw momentum. Treat them as separate features in your research log. The transform's hyperparameters (rolling-window length, sigmoid scale) belong to the feature definition, not to free knobs you sweep at backtest time. Causal computation is non-negotiable. Every rolling statistic, every rolling rank, every rolling z-score uses only past data. Lookahead bias from a non-causal transform is the single most common reason a research backtest looks great and the live equity curve does not. Fix it at the transform layer, before any model sees the data. --- ## KEY POINTS - A model needs four things from each feature: stable scale, spread distribution, bounded tails, and cross-instrument comparability. Most raw indicators fail at least one of the four. - Six transforms cover most practical work: differencing, rolling z-score, ATR normalization, log and fourth-root, sigmoid and tanh, and rank/CDF. - The defect-to-transform mapping is mechanical. Non-stationarity gets rescaled. Heavy tails get squashed. Clumping gets spread. Each defect has a primary fix and a secondary fix. - Every transform has a cost: a hyperparameter to set, information loss in the part of the distribution the transform compresses, and a risk of lookahead if the statistics are computed non-causally. - Sigmoid preserves the gradient in the body while bounding the tails. Use it when the edge lives in the central distribution and the tails are noise. - Rank discards all magnitude information and keeps only ordinal content. Use it when the edge is ordinal (extremes predict reversal) rather than magnitude-based. - The hyperparameters of a transform (window length, sigmoid scale) belong to the feature definition. Optimizing them on the backtest is overfitting on the transform. - Every rolling statistic must be computed causally. Full-sample StandardScaler on a time series leaks future information and inflates backtest performance. - Use mutual information or relative entropy (article 26) against the target to check that the transform reshaped the distribution without destroying the signal. - The improvement from a transform is usually larger than the improvement from a model upgrade. The model class is the constant. The indicator transform is the variable. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Statistically Sound Indicators for Financial Market Prediction - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Statistically+Sound+Indicators+for+Financial+Market+Prediction+Timothy+Masters&ref=aligrithm.com) - [Cycle Analytics for Traders - John Ehlers (Amazon)](https://www.amazon.com/s?k=Cycle+Analytics+for+Traders+John+Ehlers&ref=aligrithm.com) ### 2.2 Garbage Indicators, Garbage Predictions URL: https://aligrithm.com/garbage-indicators-garbage-predictions/ Last updated: 2026-07-27T13:23:16.000Z # Pick a popular volume indicator and plot it across twenty years of SPY. Mean in 2003: roughly 80 million shares. Mean in 2024: roughly 70 million shares, but with two regime shifts and a five-fold spike in 2020\. The standard deviation tripled, halved, and tripled again. You train a model on the 2003–2018 distribution. You deploy on the 2024 distribution. The "indicator" you fed the model in training did not exist in production. The model is now mapping new inputs through the wrong function. It will lose money in a structurally consistent way until you turn it off. This is what garbage looks like. The indicator did not have a noisy correlation. It had a defect. Articles in this pillar split into two halves: how to diagnose the defects, and how to repair them. This article is the catalog. The repairs come in articles 25 through 38. ## What "garbage" means operationally A garbage indicator is one of four things, sometimes more than one at once. 1. The distribution drifts over time. The mean in decade A is different from the mean in decade B. The variance shifts. A model trained on A predicts using statistics that no longer apply in B. 2. The distribution has outliers heavy enough to dominate the model's loss function. The model spends its parameter budget fitting three days from 1987 and 2008 instead of the central 99% of observations. 3. The distribution has values clumped into narrow bands separated by wide empty regions. The model carves rules in the empty regions where there is no data, and ignores the dense clumps where the predictive content sits. 4. The indicator construction injects artifacts. The lookback window flips, sign-inverts, or wildly rescales the indicator at points unrelated to anything the market did. None of these defects can be fixed by switching from logistic regression to a transformer. The model class does not see the defect. The model sees what you fed it and treats it as the truth. The defect propagates into the forecast. ## Failure mode 1: non-stationarity You compute an indicator on SPX from 1990 to 2026\. You plot its rolling 252-day mean. If that mean wanders from 12 in 2000 to 480 in 2024, the indicator's distribution in 2000 has nothing to do with its distribution in 2024\. The same numerical value carries opposite information in the two regimes. A 50 in 2000 was an extreme. A 50 in 2024 is below the floor. Any model trained across both regimes is averaging contradictions. Stationarity is the requirement that the statistical properties of an indicator stay roughly constant across time. It is the floor on indicator usability, not a polish step. You cannot train on past samples and deploy on future ones if the past samples are not drawn from the same distribution as the future ones. The bridge between training and live trading is the assumption of stationarity. Break the assumption and you break the bridge. Three diagnostics catch most non-stationarity in practice. Plot a rolling mean over a window long enough to average out short-term noise (200 to 500 bars). If the rolling mean wanders monotonically or has visible regime breaks, the indicator is non-stationary in the first moment. Plot a rolling standard deviation or IQR over the same window. If the spread expands or contracts across decades, the indicator is non-stationary in the second moment. This is the more common defect and the more dangerous one because models tend to under-react to it. Run an Augmented Dickey-Fuller test on the indicator series. If the test fails to reject the null of a unit root at p < 0.05, treat the indicator as suspect and look for a construction that produces a stationary version. The repair is upstream. Indicators built directly on raw price levels, raw volume, or anything else that trends across decades inherit the non-stationarity of their inputs. Indicators built on returns, ratios, percentile ranks, ATR-normalized differences, and rolling z-scores tend to be stationary by construction. Article 32 walks through the conversions. ## Failure mode 2: heavy tails and outliers Take SPX one-day returns from 1928 to 2026\. The max is around +16% (1933). The min is around -20% (1987). The interquartile range across the full sample is around 0.012\. The range over IQR ratio: $$ \\text{R/IQR} = \\frac{\\max(x) - \\min(x)}{Q\_{75}(x) - Q\_{25}(x)} = \\frac{0.16 - (-0.20)}{0.012} = \\frac{0.36}{0.012} = 30 $$ R/IQR of 30 means the extreme values stretch the range to thirty times the width of the central half of the data. A reasonable indicator sits at R/IQR between 2 and 3\. Anything above 5 is dominated by tails. At 30, the indicator is not a feature, it is a list of crashes with some noise around them. Heavy tails corrupt models in three measurable ways. The least-squares loss is quadratic in the residual, so a single outlier of magnitude 20σ contributes 400 times what a typical 1σ residual contributes. The fitted coefficients warp to absorb that one observation. Tree-based models split the feature space at the outliers, producing leaf nodes with one or two samples that get treated as informative regions. Standardization (subtracting mean, dividing by standard deviation) is also corrupted because the standard deviation is pulled up by the outliers, so the rescaled values for the central 99% of observations get compressed into a narrow band where no model can resolve them. The diagnostic is the R/IQR ratio itself. Compute it on every candidate indicator before you compute anything else. If R/IQR is above 5, the indicator needs a tail-taming transform: a fourth-root for mild compression, log for moderate compression, or a sigmoid for hard bounds. Articles 33 and 36 walk through the choices and their distortions. ## Failure mode 3: clumped distributions You build an indicator whose histogram has one tall spike near zero and a long thin tail to the right. Most days the indicator reads 0.01 to 0.05\. A few days a year it reads 2 to 5\. The model sees a feature that is approximately constant 99% of the time. It carves its decision boundary in the tail where it has fifteen training samples. It treats the dense central mass as a flat region with no information. The predictive content of the indicator may actually live in the clump, not in the tail. Models default to attending to the spread because the spread is where the variance is. Variance is not the same as information. A clumped distribution can hide rich structure inside the clump that the model never resolves because the resolution of its decision rules is set by the global scale of the feature. The diagnostic is visual and entropic. Plot the histogram with enough bins to see the shape (50 to 200 bins, depending on sample size). Look for one tall narrow spike, two spikes separated by emptiness, or a long thin tail attached to a dense ball. Then compute the differential entropy of the distribution after binning, normalized by the entropy of a uniform distribution with the same range. A value below 0.5 indicates the data is concentrating in narrow regions. Article 26 turns this into a quality score. The repair is a monotonic transform that spreads the clump out: rank transforms, fourth-root for mild spread, or a CDF transform that maps the empirical distribution to uniform. The transformed indicator carries the same information by the data-processing inequality, but the model can now resolve it. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Failure mode 4: construction artifacts The raw stochastic oscillator computes (close - low) / (high - low) over a lookback window. As the window rolls forward by one bar, the oldest bar drops out and the newest one enters. If the dropping bar was the high or low of the window, the denominator changes discontinuously. The indicator value jumps even if the close did not move. Across thousands of bars, the indicator carries as much window-artifact noise as it does market signal. Most indicators built on lookback ranges have a version of this defect. Bollinger bands have it on the standard deviation. Donchian channels have it on the high and low. Any rule of the form "compare today to the extreme of the last N bars" inherits the discontinuity at the edges of the window. The diagnostic is to compute the indicator twice: once with lookback N, once with lookback N+1\. If the two series diverge by more than the underlying price moved, the indicator is reading the window, not the market. Article 23's smoothing discussion and articles 39 onward on filter design give the alternative: indicators built on smoothed weighted averages of the lookback range, not on the raw extreme values, are less brittle at the window edges. ## Why the defects compound A model trained on a non-stationary, heavy-tailed, clumped indicator with window artifacts is not failing on four small problems. It is failing on a single composite problem where each defect masks the others. The non-stationarity makes the heavy tails look like signal because the tails align with regime breaks. The clumping hides the non-stationarity because the rolling mean stays inside the same dense band even as the underlying distribution shifts. The window artifacts get absorbed into the model's noise floor and inflate the standard error of every estimated coefficient. The result is a strategy whose in-sample Sharpe looks fine, whose out-of-sample Sharpe is half of in-sample, and whose live Sharpe is half of out-of-sample. Each of the three drops is a defect compounding through the deployment chain. ## Visualizing what bad distributions look like ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/05/ChatGPT-Image-May-21--2026--01_59_36-AM.png) The three histograms are the central diagnostic. Look at the shape before you look at the backtest. ## What this changes in practice Compute four numbers on every candidate indicator before you train a model. The rolling mean and rolling standard deviation across the full sample, plotted as two time series. Eyeball them for drift and regime breaks. Run an ADF test if the visual is ambiguous. The R/IQR ratio across the full sample. If it is above 5, you have a tail problem. If it is above 10, the indicator is unusable in its raw form. The histogram with 50 to 200 bins. Look for clumping, gaps, and asymmetries that no monotonic transform will fix without losing information. A consistency check by perturbing the lookback by ±1 bar. If the indicator jumps, you have a window artifact. These four checks take five minutes of code each and rule out 80% of bad indicators before any model touches them. Most retail R&D skips them because the tools point in the other direction (model libraries are convenient, indicator diagnostics are not packaged). Most performance is left on the table at this step. ## KEY POINTS - A garbage indicator has one of four structural defects: non-stationary distribution, heavy tails, clumped values, or construction artifacts from lookback windows. - Switching models does not fix any of these defects. The model treats the input as the truth. The defect propagates into the forecast unchanged. - Stationarity is the precondition for any prediction. If the indicator's mean or variance drifts across decades, the model trained on one decade is predicting the wrong thing in the next. - R/IQR is the cheapest tail diagnostic. Below 3 is healthy, above 5 needs a tail-taming transform, above 10 is unusable in raw form. - Clumped distributions hide predictive content inside dense regions where models default to ignoring it. Monotonic spreading transforms (rank, root, CDF) recover the information. - Lookback-based indicators (stochastic, Bollinger, Donchian) carry window artifacts whose magnitude can exceed the market signal. Perturb the lookback by one bar to detect this. - Defects compound. Non-stationarity, heavy tails, and clumping together produce a strategy where each defect masks the others, and the live Sharpe is a fraction of the in-sample one. - Four diagnostic checks (rolling moments, R/IQR, histogram shape, lookback perturbation) take twenty minutes and rule out most bad indicators before a model is involved. - The repair lives upstream of the model. Stationary construction, tail transforms, spreading transforms, and smoothed lookbacks. Articles 25 through 38 work through each. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Statistically Sound Indicators for Financial Market Prediction - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Statistically+Sound+Indicators+for+Financial+Market+Prediction+Timothy+Masters&ref=aligrithm.com) - [Cycle Analytics for Traders - John Ehlers (Amazon)](https://www.amazon.com/s?k=Cycle+Analytics+for+Traders+John+Ehlers&ref=aligrithm.com) ### 2.1 The Indicator Is More Important Than the Model URL: https://aligrithm.com/the-indicator-is-more-important-than-the-model/ Last updated: 2026-07-27T13:23:10.000Z # Most trading research effort flows toward models. The trader downloads a new ML library, tunes hyperparameters, runs grid searches, swaps in transformers, ensembles XGBoost with random forests, and reads papers about new architectures for time series. Months disappear. The strategy still underperforms. The reason is structural. The model is the smaller part of the predictive stack. The indicator is the bigger part. Most retail research has the proportions inverted, spending the bulk of the time on the smallest contributor to performance. The cleanest formulation: a linear regression on a high-quality indicator will outperform a deep neural network on a low-quality one in most empirical settings. This is not opinion. It is a consequence of how information flows through models, and it is the foundation of the entire indicator engineering pillar of this publication. ## The data-processing inequality Information theory gives the formal statement. For any function f applied to an input X to produce a prediction about target Y: $$ I(f(X); Y) \\leq I(X; Y) $$ Where I(·; ·) is the mutual information between two random variables. The inequality says that no transformation of X can carry more information about Y than X itself does. A model is a transformation. Its output cannot be more informative about the target than the input was. The practical translation: the indicator sets the ceiling on what any model can extract. If the mutual information between the indicator and tomorrow's return is low, no model on that indicator can predict tomorrow's return well. The model can fail to extract what is there. It cannot create signal from nothing. This is the formal version of "garbage in, garbage out." It is also the reason that switching from linear regression to a neural network rarely helps when the indicators are weak. The ceiling is the same. ## The hierarchy of trading R&D Real trading research has four layers, in descending order of impact. 1. **Data quality.** Point-in-time accuracy, survivorship-free universes, dividend handling, corporate actions, trade-time fills not close-time fills. A model trained on bad data is wrong before it starts. 2. **Indicator quality.** Mutual information with the target, stationary distribution, well-spread values, controlled tails, theoretical motivation. The indicators are the language in which the model thinks. Good indicators put the predictive content in front of the model. Bad indicators bury it. 3. **Model quality.** Regularization, validation methodology, choice of objective, hyperparameter selection. The model decides how to combine indicators into a forecast. A good model on good indicators improves marginally over a bad model on the same indicators. A good model on bad indicators usually performs the same as a bad model on bad indicators. 4. **Execution quality.** Slippage, latency, fills, transaction costs. The trade is the realization of the forecast. Good execution converts forecasts into P&L. Bad execution destroys forecasts before they realize. The retail and academic R&D worlds invert this. Most published work focuses on layer 3\. Most retail effort focuses on layer 3\. The marginal returns on layer 3 work are small once layers 1 and 2 are clean. The marginal returns on layer 2 work are large at every stage. ## Why the field gets this wrong Three forces push effort toward the model layer. Models are flashy. A new neural architecture has a paper, a brand, a leaderboard. An ATR-normalized momentum indicator has none of that. Effort follows visible reward, and the visible reward is in the model layer. ML libraries are convenient. Scikit-learn, PyTorch, and XGBoost make it trivial to drop in a new model with five lines of code. Indicator engineering requires domain knowledge, careful thought, and code written from scratch. Convenience beats correctness in default behavior. Marketing aligns with model effort, not indicator effort. The vendors of ML tools, ML courses, and ML books have an incentive to convey that the model is the critical part. The vendors of "indicator quality assessment tools" do not exist as an industry, so there is no marketing budget for the correct answer. The result is a research culture that polishes the smallest contributor to performance. Capital follows attention. Attention follows the loudest marketing. The loudest marketing is in the wrong place. ## Worked example: same target, different inputs Target: predict the sign of SPX's next-day return. Input 1: raw close price level. Train a logistic regression. Out-of-sample AUC: approximately 0.50\. The raw price level has almost no mutual information with the next-day return direction. Input 2: yesterday's one-day return. Same model. Out-of-sample AUC: approximately 0.51\. Slightly informative because of weak autocorrelation, but not by much. Input 3: yesterday's one-day return, ATR-normalized (divided by 20-day ATR), then passed through a sigmoid transform that compresses tails. Same logistic regression. Out-of-sample AUC: approximately 0.53. Now try XGBoost on Input 1\. AUC: still 0.50\. The model upgrade did nothing because the input had no information. Try a deep transformer on Input 2\. AUC: 0.51 ± noise. The model upgrade did nothing because the input did not have much more to give. The linear regression on Input 3 beats the transformer on Input 1 and the transformer on Input 2\. The difference between the worst result and the best result was entirely in the indicator design. The model layer was a constant. This pattern repeats across most empirical studies of indicator quality versus model complexity. Investing the marginal hour in better indicators usually pays more than investing it in better models. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## What "indicator quality" means Pillar 2 of this publication is a tour of the components of indicator quality. The components, in brief: **Information content.** Mutual information, correlation, or other measures of dependence between the indicator and the target. This is the upper bound on what any model can extract. **Stationarity.** The indicator's distribution should be stable over time. An indicator whose mean and variance drift across decades is not predicting the same thing in different decades. **Distribution shape.** Values should be spread across a meaningful range, not clumped into one or two narrow bands. Clumped distributions force models to focus on the spread and ignore the clumps, where the predictive content often hides. **Tail behavior.** Heavy tails with outliers degrade most models. Tails should be controlled by transformation (sigmoid, log, rank) so that extreme values do not dominate the model's loss function. **Theoretical motivation.** The indicator should have a reason to be predictive based on market structure, behavioral patterns, or microstructure. An indicator that has no theoretical reason to predict the target is most often capturing noise. The remaining articles in this pillar work through these components individually: relative entropy as a quality score, range/IQR as a tail diagnostic, sigmoid transformations, ATR normalization, stationarity tests, smoothing filters, frequency-response analysis, and the trade-offs each technique involves. ## Visualizing the ceiling ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/05/ChatGPT-Image-May-20--2026--06_44_43-PM.png) The picture is the central claim. The model axis does not lift the curve. The indicator axis does. ## Why a complex model on bad inputs is worse than useless The asymmetry is not symmetric. A complex model on a bad indicator does not just fail to extract more signal. It actively overfits the noise that is there. A linear regression on a low-information indicator produces a near-zero coefficient and a near-flat forecast. The model is honest about the input's lack of content. A complex model on the same input has enough capacity to find spurious patterns in the noise. The in-sample fit looks better. The out-of-sample performance is worse because the complex model has memorized noise the simple model ignored. This is the bias-variance trade-off from article 21 applied to indicator engineering. When the indicator carries no real information, every additional parameter is a parameter consumed by fitting noise. Simple models fail gracefully on bad inputs. Complex models fail catastrophically. The corollary: a complex model that outperforms a simple one is mild evidence that the inputs contain real structure. A complex model that underperforms a simple one is strong evidence that the inputs are mostly noise. The model class is a diagnostic for the indicator's content, not just a tool for prediction. ## What this changes operationally Three shifts in how research time gets allocated. Spend most of the R&D time on indicator design, not model selection. The default time budget for a new project is roughly 70% on data preparation and indicator engineering, 20% on modeling, 10% on validation and stress testing. Most retail teams allocate the inverse and underperform for that reason. Use linear regression as the default model. It is fast, transparent, and reveals indicator problems immediately. If linear regression cannot extract anything from an indicator set, the inputs are weak. Escalating to a more complex model is unlikely to help. The complex model will overfit and look better in sample without extracting more real information. When a model is failing, fix the inputs before fixing the model. The instinct is to swap models. The right move is to look at the indicators: are they stationary, well-distributed, transformed appropriately, and theoretically motivated? Almost always at least one of these is broken. Fixing it produces more performance than any model change. --- ## KEY POINTS - The model is the smaller part of the predictive stack. The indicator is the bigger part. Most retail R&D inverts this allocation. - The data-processing inequality says I(f(X); Y) ≤ I(X; Y). No model can extract more information about the target than the indicator carries. The indicator sets the ceiling. - The R&D hierarchy in order of impact is: data quality, indicator quality, model quality, execution quality. Effort should flow in that order. Most published work focuses on the third layer. - A linear regression on a high-quality indicator will usually outperform a deep neural network on a low-quality one. The model upgrade does not break through the ceiling that the indicator sets. - Indicator quality has five components: information content, stationarity, distribution shape, tail behavior, and theoretical motivation. Each is the subject of subsequent articles in this pillar. - Complex models on bad indicators do not just fail to extract more signal. They actively overfit the noise that is there and underperform simple models out of sample. - The model class is a diagnostic for indicator quality. A complex model that does not beat a simple one is strong evidence that the inputs are mostly noise. - When a model is failing, the default suspect is the inputs, not the model class. Diagnose the indicators first. Escalate to a more complex model only after the inputs have been cleaned. - Default time allocation for a new project: 70% on data and indicator engineering, 20% on modeling, 10% on validation. Most teams use the inverse. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Statistically Sound Indicators for Financial Market Prediction - Timothy Masters (Amazon)](https://www.amazon.com/s?k=Statistically+Sound+Indicators+for+Financial+Market+Prediction+Timothy+Masters&ref=aligrithm.com) - [Cycle Analytics for Traders - John Ehlers (Amazon)](https://www.amazon.com/s?k=Cycle+Analytics+for+Traders+John+Ehlers&ref=aligrithm.com) ### 1.22 The Scientific Method for Building Trading Systems URL: https://aligrithm.com/the-scientific-method-for-building-trading-systems/ Last updated: 2026-07-27T13:22:59.000Z # Most trading strategies fail in live markets because the development process skipped steps. The skip is not deliberate. It is the default. The default process is to find a pattern, run a backtest, see a number, and ship. The result is a strategy that performs well in the sample it was built on and badly in the sample it will be deployed against. The scientific method is the alternative. It is not a philosophy. It is an engineering protocol with explicit stages, pass/fail gates, and decision criteria. The protocol is demanding by design. Most candidate strategies die in the middle. The few that survive all stages have earned the only kind of confidence that survives contact with live markets: confidence based on rigorous pre-committed falsification attempts that the strategy passed. This article closes the Scientific Trader pillar by integrating the tools from the previous articles into a single end-to-end workflow. Each stage references the article that handled the underlying machinery. ## The eleven stages The protocol has eleven stages. Each stage has a pass criterion and a stop criterion. A strategy that fails the stop criterion at any stage is discarded or reformulated. A strategy that passes all eleven stages is ready for live deployment. ### Stage 1\. Observation A candidate phenomenon is identified. The source can be real-world experience (a market behavior the trader has noticed), published research, a structural feature of the market (settlement effects, expiration cycles, central bank announcements), or anomaly detection on data. *Stop criterion:* the phenomenon cannot be described in precise words. If the candidate is "stocks tend to go up after a news event but only when the mood is right," the observation has not yet been articulated. ### Stage 2\. Hypothesis formulation The observation is transformed into a falsifiable hypothesis (the seven-step transformation in article 19). The hypothesis names a specific variable, a specific condition, a specific horizon, and a specific direction. It can be coded into a deterministic function that two independent researchers running on the same data compute identically. *Stop criterion:* the hypothesis cannot be coded. It is still subjective, still time-unbounded, or still result-dependent. Return to Stage 1 or discard. ### Stage 3\. Null hypothesis specification The null hypothesis is defined (article 16). For a trading rule, the canonical null is "the rule has no predictive power; its expected return on detrended data is zero or negative." The null model that will be simulated must be specified: bootstrap, block bootstrap, signal permutation, or factor regression. *Stop criterion:* the null cannot be made realistic. The trader cannot construct a generator of "this rule under no edge" that respects the volatility clustering, autocorrelation, and fat tails of the underlying market data. ### Stage 4\. Test statistic and decision rule A test statistic is chosen (mean return per signal, Sharpe ratio, hit rate, or information ratio against the primary benchmark). An α threshold is committed to. The standard is 0.05; for live-capital decisions, stricter values (0.01 or 0.005) are appropriate (article 16). *Stop criterion:* either choice is post-committed. If the test statistic or threshold is left flexible until after the result is seen, the test has no scientific value. ### Stage 5\. Backtest construction The backtest is built with the constraints that distinguish an experiment from a screenshot (article 12). Point-in-time data, no lookahead, realistic transaction costs and slippage, detrending of market returns, and explicit benchmarks (article 17). *Stop criterion:* any of these constraints is violated. A backtest with lookahead, no costs, or no benchmark is decoration, not data. ### Stage 6\. Statistical inference The test statistic is computed on the actual backtest. The same statistic is computed on thousands of simulations from the null model. The p-value is the fraction of null simulations that produced a statistic at least as extreme as the observed one. The 95% confidence interval on the effect size is computed from the bootstrap (article 13). *Pass criterion:* p < α AND the lower bound of the CI on the effect size is meaningfully positive (not just barely above zero). *Stop criterion:* p > α OR the CI contains zero. The strategy has not falsified the null. Discard. ### Stage 7\. Bias decomposition The six diagnostic tests from article 18 are run: detrended re-run, signal permutation, bias-matched random rule, regime split, factor regression, and instrument swap. Each test isolates a different form of hidden bias. *Pass criterion:* residual alpha remains after every exposure is stripped out. The strategy's edge is not driven by sector tilt, factor exposure, regime selection, or any other implicit bias. *Stop criterion:* any diagnostic reveals that the apparent edge is actually exposure. Discard. ### Stage 8\. Robustness validation Three robustness checks (article 7). Cross-instrument: the rule works on related markets with reasonable parameter values, not only the original instrument with hand-picked parameters. Cross-period: the rule works on different sub-windows of history, including ones that contain different volatility and trend regimes. Cross-parameter: small perturbations of the parameters do not destroy the edge. The performance surface in parameter space is smooth, not spiky. *Pass criterion:* the rule produces positive results across instruments, periods, and parameter perturbations. The edge is loose pants, not skin-tight. *Stop criterion:* the rule's performance is sharply localized to specific parameter values, specific instruments, or specific time windows. Discard. ### Stage 9\. Walk-forward and paper trading The rule is implemented as production code. The code is walk-forward-tested with rolling re-fits (parameters re-estimated periodically using only data available before the trade date). The code runs in a paper-trading mode on live data for a calibration period (typically several weeks to several months depending on signal frequency). *Pass criterion:* walk-forward performance is consistent with the backtest distribution. Paper-trading execution matches the backtest in fill rates, slippage, and timing. *Stop criterion:* execution behavior differs from backtest assumptions, signals fire at different times than expected, or walk-forward performance is well below backtest performance. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ### Stage 10\. Live deployment with monitoring Capital is allocated. The size respects the wider true uncertainty captured in articles 13 and 14: the backtest's confidence interval is the lower bound on uncertainty, and the live confidence interval is wider because of regime drift and non-stationarity. Every signal, every trade, every fill, every P&L attribution is logged. Live performance is compared to the backtest's confidence interval continuously. Kill criteria are defined in advance. *Pass criterion:* live performance falls within the backtest's 95% CI over the monitoring window. *Stop criterion:* live performance falls outside the CI for a sustained period (typically defined as a number of consecutive months or a cumulative drawdown threshold). The strategy is paused. ### Stage 11\. Iteration and falsification in production The hypothesis remains provisionally accepted only as long as live data continues to support it. Markets are non-stationary (article 14). An edge that was real in the backtest period may dissipate, reverse, or change shape in live trading. The protocol must be re-run periodically as more data accumulates. When live performance falls outside the expected distribution for a sustained period, the hypothesis is rejected. The honest response is to discard or reformulate the strategy. The dishonest response is to "tweak parameters until live works again," which is in-sample fitting on the live data and produces the same overfitting problem the protocol was designed to avoid. ## The integrated decision rule A strategy ships only if every stage passes. Formally: $$ \\text{ship} \\iff \\bigwedge\_{i=1}^{11} \\text{stage}\_i = \\text{pass} $$ The conjunction is the entire pipeline. A single stage failure stops the strategy. If the tests are approximately independent and the null hypothesis is true (the strategy has no edge), the probability of false-shipping a worthless strategy through the full pipeline is roughly the product of the individual stage failure rates: $$ P(\\text{ship} \\mid H\_0) \\approx \\prod\_{i=1}^{N} \\alpha\_i $$ With even a modest N at α = 0.05 per stage, the joint false-positive rate is tiny. The tests are not fully independent in practice, so the real rate is higher, but the principle holds: stacking pre-committed tests produces multi-layer protection against false positives that no single test can deliver. The cost of the protocol is that many genuine edges are also rejected. This is the Type II error from article 16\. The trade-off is accepted because Type I errors (worthless strategies in live trading) consume capital, attention, and confidence, while Type II errors (real edges that were missed) cost only opportunity. ## Visualizing the pipeline ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/05/ChatGPT-Image-May-20--2026--05_04_02-PM.png) The picture is the entire article. Hundreds of candidate strategies enter the funnel. A handful exits to live capital. Each stage is a gate that filters the candidates that fail its criterion. ## Why most strategies fail this protocol The protocol is demanding by design. The empirical observation across decades is consistent: most apparent trading edges are noise, bias, selection effects, or factor exposures that look like edge. Each stage is calibrated to detect one or more of these failure modes. Stage 6 (statistical inference) eliminates strategies whose backtest performance is consistent with chance. Stage 7 (bias decomposition) eliminates strategies whose returns are explained by hidden exposures. Stage 8 (robustness validation) eliminates strategies that are narrowly fit to one instrument, one period, or one parameter set. Stage 9 (walk-forward) eliminates strategies that work in idealized backtests but not in realistic execution. Stage 10 (monitoring) eliminates strategies whose live behavior diverges from the backtest, even if they passed every earlier stage. The strategies that survive all eleven stages are rare. They are also unglamorous. They tend to have few parameters (article 21), modest backtested Sharpe ratios, and broad generalization across instruments and periods. They look unimpressive next to the unfiltered candidates with backtested Sharpe ratios of 3.0 that died at Stage 7\. They are the only strategies that pay in live trading. ## The role of iteration Science is not a single test. It is a continuous process of falsification and refinement. Even strategies that pass the full protocol will eventually be falsified by changing markets. An edge that worked in the backtest sample because of a specific market structure (low rates, tight spreads, retail dominance, market-maker behavior, regulatory regime) may stop working when that structure changes. The protocol must be re-run periodically on live data. New observations are appended to the historical record. The null hypothesis test is recomputed with the larger sample. The bias decomposition is rerun in case new exposures have emerged. The robustness checks are repeated against the new regimes. A strategy that fails the re-run is paused. The hypothesis has been falsified by the new data. The honest response is to acknowledge the falsification and either reformulate or retire the strategy. The dishonest response is to retroactively change the protocol to keep the strategy alive. This is the same overfitting move the protocol was designed to prevent, applied to live data instead of backtest data, and it produces the same result. A living strategy is one that continuously survives re-falsification. Most strategies do not last for decades. A few do. The ones that do are the ones whose underlying mechanism is general enough that it transcends specific market regimes (article 7's loose-pants principle), and they survive precisely because their generality makes them robust to regime change. ## What this changes for the practitioner Three operational shifts. Codify the protocol as a checklist in the development environment. Every candidate strategy passes through every gate, in order, with explicit pass/fail outcomes logged for each stage. No shortcuts. No skipping. No moving the gate after a strategy stumbles. The checklist is the contract between the trader and the trader's future self, who will be tempted to ship a beautiful backtest without doing the work. Accept that the survival rate will be low. For every published anomaly that becomes a real strategy, dozens die in the middle of the pipeline. The development process is not "find a strategy." It is "filter candidates until one survives." The candidates are cheap. The filter is the value. Re-run the protocol on live strategies periodically. The strategies that earn capital next year are the ones that pass the protocol next year, regardless of how they performed last year. A strategy that returned 18% per year for a decade and falls outside its confidence interval in the eleventh year is a strategy that has been falsified by the new data. The honest response is to pause and re-evaluate, not to assume the eleventh year is noise. --- ## KEY POINTS - The scientific method for trading is an eleven-stage protocol with explicit pass/fail criteria at each stage. The protocol is demanding by design and filters most candidate strategies. - The stages are: observation, hypothesis formulation, null hypothesis specification, test statistic and decision rule, backtest construction, statistical inference, bias decomposition, robustness validation, walk-forward and paper trading, live deployment with monitoring, and iteration and falsification in production. - Each stage is supported by machinery developed earlier in this pillar. Stage 2 uses article 19, Stage 3 uses article 16, Stage 5 uses articles 12 and 17, Stage 6 uses article 13, Stage 7 uses article 18, Stage 8 uses article 7, Stages 10 and 11 use articles 13 and 14. - A strategy ships only if every stage passes. The joint false-positive rate of stacked pre-committed tests is far lower than any single test, providing multi-layer protection against shipping worthless strategies. - The protocol accepts a higher Type II error rate (real edges missed) in exchange for a much lower Type I error rate (worthless strategies shipped). The asymmetry favors strictness because Type I errors cost capital and Type II errors cost only opportunity. - The strategies that survive all eleven stages are rare and unglamorous: few parameters, modest Sharpe ratios, broad generalization. They are the only strategies worth running. - The protocol is not a one-time gate. Markets are non-stationary, so strategies must be re-evaluated periodically. The honest response to live failure is to discard or reformulate, not to tweak parameters until live performance returns. - A living strategy is one that continuously survives re-falsification. The strategies that last for decades have underlying mechanisms general enough that they transcend specific market regimes. - Operational fix: codify the protocol as a development checklist, run every candidate through every gate, accept low survival rates, and re-run the protocol on live strategies as new data accumulates. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Evidence-Based Technical Analysis - David Aronson (Amazon)](https://www.amazon.com/s?k=Evidence-Based+Technical+Analysis+David+Aronson&ref=aligrithm.com) - [Systematic Trading - Robert Carver (Amazon)](https://www.amazon.com/s?k=Systematic+Trading+Robert+Carver&ref=aligrithm.com) ### 1.21 Why Simplicity Is a Statistical Weapon URL: https://aligrithm.com/why-simplicity-is-a-statistical-weapon/ Last updated: 2026-07-27T13:22:58.000Z # Most traders treat simplicity as a stylistic preference. The clean code, the elegant rule, the chart with one line on it. The aesthetic part of simplicity is real. The aesthetic part is also the least interesting reason to be simple. The reason to be simple is statistical. Simpler models generalize better. They overfit less. They produce out-of-sample performance closer to their in-sample performance. The mechanism is mathematical and unavoidable: every additional parameter consumes data, inflates the variance of estimates, and provides another lever for noise to be mistaken for signal. Adding complexity is taking on statistical debt that has to be paid back later, usually in live trading. Simplicity is not a defensive posture. It is an offensive weapon against overfitting, against data mining bias, and against the human tendency to mistake fitted noise for discovered structure. The trader who deploys simplicity as a deliberate tool destroys more bad strategies before they ship than any single backtest filter. ## Why complex models look better in-sample Every model has parameters. A moving-average crossover rule has two (the fast period and the slow period). A multi-factor stat-arb model has dozens. A neural network has millions. Each parameter is a number the researcher chose by fitting the model to data. Each parameter is also a degree of freedom for the model to bend toward the training sample. With one parameter, the model can fit one shape. With two parameters, the model can fit two shapes simultaneously. With enough parameters, the model can fit any pattern in any sample, including all the noise. This is not metaphor. It is provable. A polynomial of degree N can fit any N+1 data points exactly. A neural network with enough hidden units can approximate any function on the training set. The fit on the training data does not measure whether the model captured anything real. It measures the model's capacity to fit, which complex models have in abundance. The in-sample fit of a complex model is therefore not evidence that the model has signal. It is evidence that the model has enough parameters to bend toward whatever was in the sample, including the noise. Out-of-sample, the noise is different, and the bent model fits the new noise badly. ## The bias-variance trade-off The formal version of this story is the bias-variance trade-off. Every model's expected prediction error decomposes into three parts: $$ \\mathbb{E}\[(\\hat{Y} - Y)^2\] = \\text{Bias}^2(\\hat{Y}) + \\text{Var}(\\hat{Y}) + \\sigma^2\_{\\text{irreducible}} $$ Where the bias is the systematic error of the model (how wrong it is on average), the variance is how much the model's predictions change across different training samples, and the irreducible variance is the noise in the outcome that no model could remove. Simple models tend to have high bias and low variance. They are wrong in consistent ways but stable across samples. Complex models have the opposite: low bias and high variance. They can fit any sample exactly, but their fits differ wildly between samples. The total error is minimized at some intermediate complexity. For data with high signal-to-noise ratio (physics experiments, controlled trials), the optimum is on the complex side. For data with low signal-to-noise ratio (financial markets), the optimum is much closer to the simple end. Most traders set their complexity at the wrong end of the curve for the data they are working with. ## The data-per-parameter problem The simplest intuition for why complexity is dangerous is the data-per-parameter ratio. $$ \\text{observations per parameter} = \\frac{N}{K} $$ Where N is the number of independent observations in the dataset and K is the number of parameters in the model. Common rules of thumb in econometrics require at least 10 to 30 observations per parameter for reliable estimation. Fewer than 10 means parameter estimates are too noisy to trust. A daily-bar strategy backtested on five years of data has roughly 1260 observations. The effective N is smaller because of autocorrelation and regime changes, often closer to 200–400\. With K = 5 parameters, observations per parameter is 40 to 80\. Borderline. With K = 20 parameters, the ratio drops to 10 to 20\. Marginal. With K = 100 parameters, the ratio is 2 to 4\. Guaranteed overfit. The number of independent observations in trading data is fundamentally limited by calendar time and market non-stationarity. The number of parameters in any model is chosen by the researcher. The trader who chooses many parameters has chosen to overfit and is then surprised when live results disappoint. ## The exploding search space The data-per-parameter ratio understates the problem because researchers typically search over many possible parameter values, not just one. Take a rule with five parameters, each tested at ten possible values. The number of distinct rule variations is: 10 × 10 × 10 × 10 × 10 = 100,000 variations The researcher runs all 100,000 backtests and picks the best one. The "best" variation has a backtested Sharpe that is, by sheer chance, well above the average of the 100,000\. The reported Sharpe is the maximum of a distribution of 100,000 noisy samples, not the true expected value of the chosen rule. The selection process is the source of most of the apparent performance. This is data mining bias and it scales with the search space. A rule with two parameters, each at five values, has 25 variations. The best of 25 is moderately above average. A rule with five parameters at ten values each has 100,000 variations. The best of 100,000 is dramatically above average, even with no real signal in the data. Simpler rules have smaller search spaces and therefore smaller data mining bias. This is one of the strongest reasons to keep parameter counts low. Every parameter is multiplicative in the search space and the data mining penalty. ## Information criteria: the formal penalty for complexity Statisticians have built explicit penalties for complexity into model selection. The two most common are the Akaike Information Criterion (AIC) and the Bayesian Information Criterion (BIC): $$ \\text{AIC} = 2K - 2 \\ln(\\hat{L}) $$ $$ \\text{BIC} = K \\ln(N) - 2 \\ln(\\hat{L}) $$ Where K is the number of parameters, N is the sample size, and L̂ is the maximized likelihood of the model. Both criteria reward in-sample fit (the −2 ln(L̂) term) and penalize complexity (the term involving K). Lower is better. BIC penalizes complexity more aggressively than AIC, especially for large samples. The practical use is to compare two models with different numbers of parameters. The simpler model can have a slightly worse in-sample fit and still be preferred if the gap in fit is smaller than the penalty for the extra parameters. The criteria formalize the intuition that complexity should pay its way: a parameter that improves fit by less than its complexity cost is destroying out-of-sample performance even when it improves in-sample performance. For trading model selection, AIC and BIC are crude but useful. They are not the right criterion to optimize directly (live performance is the right criterion) but they are a sanity check on whether a more complex model is justified by its in-sample gains. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Visualizing the overfitting curve ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/05/ChatGPT-Image-May-20--2026--04_56_21-PM.png) The picture is the central fact of model selection. The in-sample curve is what backtests show. The out-of-sample curve is what live trading delivers. The gap between them is the cost of every additional parameter. ## Empirical evidence in trading The case for simplicity in trading is not just theoretical. The empirical record across decades is consistent. Trend-following with a single moving-average parameter has produced edge across markets, asset classes, and decades. The same family of strategies with elaborate multi-parameter optimization has rarely outperformed the simpler version in live trading. Long-only equity index strategies (zero parameters) have outperformed the majority of actively managed funds (many parameters) over multi-decade horizons. Risk parity strategies that allocate by volatility (one parameter per asset, the target volatility) have outperformed mean-variance optimization (a full covariance matrix worth of parameters) on most realistic out-of-sample windows. Carry strategies in FX and commodities (one or two parameters: the carry signal and the holding period) have produced more consistent live results than parametric models attempting to forecast the underlying carry returns. The pattern is not that simple always wins. The pattern is that simple wins more often than expected, and the gap between in-sample and out-of-sample performance is smaller for simple strategies. ## When is complexity justified Complexity is sometimes justified. Three conditions, all of which must hold: The researcher has substantially more data than parameters. The effective N/K ratio is high enough that each parameter is estimated with low standard error. For trading data, this usually requires either intraday data on very liquid instruments or cross-sectional data across many assets. Each additional parameter captures a distinct, mechanistically motivated source of variation. The parameters are not redundant or competing for the same signal. Each one has a story for why it should improve out-of-sample performance. The complex model demonstrates out-of-sample improvement over the simple model under realistic conditions, including transaction costs, sampling variability, and regime changes. The improvement is robust across different out-of-sample windows, not specific to one chosen window. When all three conditions hold, complexity pays its way. When any of them fails, complexity destroys value. The default assumption is that the conditions do not hold and that the simpler model is the better choice. ## What this changes operationally Three operational shifts. Track the parameter count of every strategy and treat it as a risk metric on par with leverage or concentration. A strategy with 20 parameters and a Sharpe of 1.5 carries more model risk than a strategy with 3 parameters and a Sharpe of 1.0, even though the first looks better in the backtest. Add complexity only when each addition pays for itself out of sample. Start with the simplest rule that could plausibly capture the target effect. Add a parameter only when its addition demonstrably improves out-of-sample performance on a held-out window that was not used in any earlier modeling decision. Treat published strategies with skepticism that scales with their parameter count. A published rule with two parameters has had less opportunity to overfit and is more likely to be real. A published rule with 30 parameters has had more opportunity to overfit and is more likely to be noise that fit the publication's specific sample. --- ## KEY POINTS - Simplicity is not a stylistic preference. It is a statistical advantage. Simpler models generalize better, overfit less, and produce out-of-sample performance closer to in-sample performance. - The bias-variance trade-off decomposes prediction error into bias² + variance + irreducible noise. Simple models have high bias and low variance. Complex models have the opposite. Total error is minimized at intermediate complexity; for noisy market data the optimum is much closer to the simple end than most practitioners assume. - The data-per-parameter ratio (N/K) is the simplest sanity check. Common rules of thumb require at least 10–30 observations per parameter. Trading data with effective N of 200–400 supports only a small number of parameters before overfitting becomes inevitable. - The search space explodes multiplicatively with parameters. A rule with 5 parameters at 10 values each has 100,000 variations. The "best" of 100,000 is dramatically above average by chance alone, even with no real signal. - AIC and BIC formalize the penalty for complexity: AIC = 2K − 2 ln(L̂), BIC = K ln(N) − 2 ln(L̂). Lower is better. Both reward in-sample fit and penalize parameter count. - Empirical evidence across decades favors simplicity: single-parameter trend-following, zero-parameter index strategies, single-parameter risk parity, low-parameter carry. The pattern is that simple wins more often than expected and survives the in-sample to out-of-sample transition better. - Complexity is justified only when three conditions hold: large N/K ratio, each parameter captures a distinct mechanism, and the complex model demonstrably improves out-of-sample performance. Default assumption: the conditions do not hold. - Operational fix: treat parameter count as a risk metric, add complexity only when each addition pays for itself out of sample, and apply skepticism to published strategies that scales with their parameter count. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Evidence-Based Technical Analysis - David Aronson (Amazon)](https://www.amazon.com/s?k=Evidence-Based+Technical+Analysis+David+Aronson&ref=aligrithm.com) - [Systematic Trading - Robert Carver (Amazon)](https://www.amazon.com/s?k=Systematic+Trading+Robert+Carver&ref=aligrithm.com) ### 1.20 The Difference Between Explanation and Prediction in Markets URL: https://aligrithm.com/the-difference-between-explanation-and-prediction-in-markets/ Last updated: 2026-07-27T13:22:57.000Z # A market prediction is a statement made before an outcome about what the outcome will be. A market explanation is a statement made after an outcome about why the outcome occurred. The two look similar on the page and behave differently when capital is at stake. Prediction is economically useful and statistically demanding. Explanation is psychologically comforting and statistically cheap. Mistaking one for the other is the most common cognitive failure in market commentary and one of the most expensive errors in trading. The fix is to be precise about which side of the outcome the claim was constructed on. ## The temporal definition The cleanest version of the distinction is temporal. $$ \\text{Prediction: } \\hat{Y} = f(X), \\quad \\text{computed at time } t < T\_{\\text{outcome}} $$ $$ \\text{Explanation: } Y = g(X), \\quad \\text{constructed at time } t > T\_{\\text{outcome}} $$ Where Y is the outcome variable (next-week return, year-end index level, recession indicator), X is the available information, and f and g are functions that map information to claims about Y. The function f is computed before knowing the outcome. The function g is constructed after knowing the outcome. The mathematical difference is which side of T\_outcome the inference happens on. The practical difference is which functions are available. After T\_outcome, g has access to Y itself when picking which variables in X matter. Before T\_outcome, f does not. The information asymmetry is the entire distinction. Prediction commits without knowing the answer. Explanation chooses with the answer in hand. ## Why explanation is cheap Any market event has many possible explanations. Markets rose because inflation cooled. Markets rose because earnings surprised. Markets rose because liquidity expanded. Markets rose because positioning was short. Markets rose because seasonality was favorable. Markets rose because the calendar flipped. Each of these claims can be supported by selectively cited data. None of them is uniquely correct in any rigorous sense. After the outcome is known, the explainer picks the explanation that fits. The fit is engineered, not discovered. The explanation feels correct because the explainer used the outcome to select among a menu of post-hoc stories. The cost of generating an explanation is approximately zero. The market dropped. Six explanations are available within an hour. Most newsroom commentary is the production of these explanations at industrial scale. The activity is sometimes useful for entertainment. It is not useful for trading. ## Why prediction is hard A prediction commits in advance. Once committed, the prediction is right or wrong when the outcome arrives. There is no menu to choose from after the fact. A real prediction has three properties. 1. **Dated.** The prediction was made at a known time before the outcome was known. 2. **Specific.** The prediction names a variable, a direction, a magnitude, and a horizon. 3. **Preserved.** The prediction is logged in a way that cannot be edited after the outcome arrives. Almost no published market commentary meets these three criteria. The few participants who do (registered investment advisors making numerical forecasts, hedge funds with publicly tracked performance, prediction market participants) are the only ones whose claims can be evaluated as predictions. Everyone else is making explanations dressed in the grammar of predictions. ## The four scenarios Models in markets fall into four categories based on whether they predict and whether they explain. **Predicts and explains.** The ideal scientific model. It produces accurate forward forecasts and provides a mechanism by which the forecasts work. This category is rare in markets and common in physics. **Predicts but does not explain.** Pure statistical edge. A black-box model that produces accurate forecasts without anyone understanding why. Plenty of profitable strategies fall into this category. The trader does not know the mechanism but can verify the prediction. **Explains but does not predict.** Economic theory that produces compelling causal stories but no measurable forward edge. Much of behavioral finance falls into this category. The story is satisfying. The trading P&L is flat or negative. **Does neither.** Most market commentary. The commentator produces a satisfying narrative after the event with no demonstrable forecasting record. The narrative is consumed for the comfort of feeling that something is understood, not for any operational use. The trader who confuses categories 2 and 3 is dangerous to themselves. A predictive model with no explanation is profitable and uncomfortable. An explanatory model with no prediction is unprofitable and comfortable. Choosing the latter for psychological reasons is one of the more common ways to lose money in markets. ## The retail trap The retail trap is reading post-event explanations as if they were pre-event predictions. The cognitive process looks like this: The market dropped 4% yesterday. A commentator explains that the drop was caused by hawkish Fed remarks. The commentator's tone is confident. The story is internally coherent. The reader concludes, implicitly, that the same commentator could have predicted the drop in advance. Therefore the reader believes the commentator has predictive insight. Each step in this chain is wrong. The commentator did not commit to this explanation in advance. The commentator had hundreds of possible explanations available and picked the one that fit. The internal coherence of the story is a feature of human storytelling, not evidence of predictive accuracy. The implicit transfer from "explains well" to "predicts well" is unjustified. The fix is to track. Anyone who claims predictive insight should be willing to log forecasts before outcomes. The forecast log is the only evidence that the claimed insight exists. In the absence of a log, every confident explanation should be read as an explanation, not a prediction. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## The hindsight engine Hindsight bias is the cognitive tendency to view past events as more predictable than they were before they occurred. The mechanism is selective recall: the explainer remembers the cues that pointed toward the outcome and forgets the cues that pointed the other way. The result is a sincere belief that the outcome was foreseeable, even when no one foresaw it. Markets are an unusually fertile environment for hindsight bias because the available information at any moment supports many possible narratives, and the outcome eliminates all but one. After the dust settles, the surviving narrative looks inevitable. The trader who internalizes the surviving narrative as evidence of foresight is feeding their hindsight bias and learning nothing about the future. The corrective is to write forecasts down before the outcome. The written forecast is immune to selective recall. Six months later, the reader can compare what was actually predicted to what actually happened, with no opportunity to retroactively edit the prediction to match the outcome. ## Visualizing the temporal asymmetry ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/05/ChatGPT-Image-May-20--2026--04_52_05-PM.png) The picture is the entire distinction in one image. Everything on the left side commits before knowing. Everything on the right side selects after knowing. The geometry of the time axis is the structure of the difference. ## When each is useful Prediction and explanation each have legitimate uses. The error is using one in place of the other. Prediction is the basis of alpha. Every trading strategy is a prediction generator. The strategy says: at this point in time, given this information, the expected return of this position is X. The position is taken. The outcome arrives. The prediction is right or wrong. Aggregated over many predictions, the strategy has a measurable edge or it does not. Explanation is the basis of risk decomposition and post-mortem learning. When a strategy loses money, explanation answers the question of which exposure failed. When a portfolio's drawdown is larger than expected, explanation answers the question of which tail risk realized. This kind of explanation is bounded and useful because it operates on data that already exists and helps the next prediction be better calibrated. The error is using explanation as a substitute for prediction. "The strategy lost because of X" is a partial answer if the next version of the strategy will use X as a forward predictor and avoid the same exposure going forward. It is not a complete answer if no such forward use is planned and the explanation is consumed for its narrative coherence alone. ## What this changes operationally Three concrete consequences. Treat every piece of market commentary you read as explanation by default. Until proven otherwise with a dated forecast log, the commentator is producing post-hoc stories. Read for curiosity, not for operational decisions. Build prediction systems that log forecasts before outcomes arrive. The log is the asset. Without it, the system cannot be evaluated as a predictor, only as an explainer. A trader who runs a strategy without logging forecasts is doing prediction and explanation simultaneously and cannot tell which one is responsible for the results. Separate post-mortems from forecast generation. After a loss, write down the explanation. Use the explanation to refine the prediction system for the next period. Do not promote an explanation to the status of prediction without an explicit forward test against new data. --- ## KEY POINTS - A prediction is a statement about an outcome made before the outcome is known. An explanation is a statement about an outcome made after the outcome is known. The difference is which side of T\_outcome the inference happens on. - Explanation is cheap because the explainer has access to the outcome and can select from many possible stories. Prediction is hard because it commits to one claim before any selection is possible. - A real prediction has three properties: dated, specific, and preserved in a way that cannot be edited after the outcome arrives. - Models fall into four categories: predicts and explains (ideal), predicts but does not explain (statistical edge, often profitable), explains but does not predict (most behavioral finance, comfortable but unprofitable), does neither (most market commentary). - The retail trap is reading post-event explanations as evidence of predictive insight. The explainer's confident narrative is a feature of storytelling, not evidence of forecasting accuracy. - Hindsight bias makes outcomes look more predictable than they were because the surviving narrative is remembered and the discarded ones are not. The corrective is written forecasts that cannot be edited after the fact. - Prediction is the basis of alpha. Explanation is the basis of risk decomposition and post-mortem learning. Each has legitimate uses; substituting one for the other is the standard error. - The only evidence that a commentator has predictive insight is a dated forecast log. In the absence of a log, every confident explanation should be read as explanation, not prediction. - Operational fix: treat market commentary as explanation by default, log forecasts before outcomes arrive, separate post-mortem explanation from prediction generation, and refuse to promote an explanation to a prediction without an explicit forward test. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Evidence-Based Technical Analysis - David Aronson (Amazon)](https://www.amazon.com/s?k=Evidence-Based+Technical+Analysis+David+Aronson&ref=aligrithm.com) - [Systematic Trading - Robert Carver (Amazon)](https://www.amazon.com/s?k=Systematic+Trading+Robert+Carver&ref=aligrithm.com) ### 1.19 How to Make Technical Analysis Falsifiable URL: https://aligrithm.com/how-to-make-technical-analysis-falsifiable/ Last updated: 2026-07-27T13:22:56.000Z # Most technical analysis is not wrong. It is unfalsifiable. The distinction matters because unfalsifiable claims sit outside the domain of evidence. They cannot be tested, proven, or disproven. They can only be agreed with or disagreed with on aesthetic grounds. A claim that cannot be falsified by any possible observation is a claim about taste, not about markets. Every TA claim can be made falsifiable. The transformation is operational and mechanical. It requires turning fuzzy verbal descriptions into deterministic functions, bounded predictions, and pre-committed thresholds. The transformation is what most published TA refuses to do, because the transformation often kills the claim. The trader who learns to perform the transformation systematically will discard most of the TA they previously trusted. The remaining patterns, if any, will be the few worth running. ## Seven structural defects that block falsifiability A TA claim fails the falsifiability test when it has one or more of these defects. **Subjective pattern recognition.** "A head and shoulders pattern appears here." Different analysts looking at the same chart disagree on whether the pattern is present. The criterion for "the pattern exists" lives in human judgment, not in the data. Two researchers running the same test on the same data get different inputs and different outputs. **Open-ended time horizons.** "Eventually the price will revert." Eventually is not a time horizon. Any prediction can be true eventually if the horizon is unbounded. A claim that does not specify when the prediction must occur cannot be tested before then, which means it cannot be tested at all. **Missing parameters.** "Buy when the moving average crosses." Which moving average? Crosses what? Buy how much? Hold for how long? Without parameters, the claim is not a rule. It is a sketch. **Cherry-picked instances.** "Look at this chart, the pattern worked here, and here, and here." The published examples are the ones that worked. The implicit population of attempts (including the failures that were never published) is unknown. The headline win rate is the win rate of the surviving examples, which is unrelated to the true population rate. **Missing baseline.** "After this pattern, the market went up 60% of the time." Without knowing how often the market went up unconditionally during the same period, the conditional rate carries no information. If the market went up 60% of the time anyway, the pattern adds nothing. **Result-dependent definition.** "This was a true breakout because the price kept rising. A breakout that fails is a false breakout, not a true one." The pattern's definition includes the outcome. The pattern, defined this way, can never fail because failure is excluded by definition. This is the classic move that turns a falsifiable claim into an unfalsifiable one. **Goalpost-shifting on failure.** "The pattern works, but only in trending markets, and only when volume confirms, and only when the broader trend agrees." Each new condition is added after a failure. The original claim recedes behind a growing list of exceptions, each of which was discovered after the fact. The growing claim is unfalsifiable by construction because any failure becomes evidence for a missing condition rather than evidence against the claim. A TA claim that exhibits any of these defects is not falsifiable as stated. Fixing the defect is the first step of the transformation. ## The seven-step transformation Take any TA claim. Apply these seven steps in order. The output is a falsifiable specification. **Step 1\. Operationalize the trigger.** Replace subjective pattern recognition with a deterministic function. The function takes market data as input and returns a binary signal as output. Two researchers running the function on the same data produce identical signal series. If two researchers cannot do this, the claim has not yet been operationalized. **Step 2\. Operationalize the prediction.** Replace vague directional statements ("the market will go up") with measurable outcomes. The prediction names a specific variable (close price, return, drawdown), a specific direction (up, down), and a specific magnitude (greater than X%, less than X%, between A% and B%). **Step 3\. Bound the prediction in time.** The prediction must specify when it is evaluated. "Within 10 trading days after the trigger fires" is a bound. "Eventually" is not. Without a bound, the claim cannot be falsified by any observation. **Step 4\. Quantify the success criterion.** The claim must specify what fraction of historical instances meeting the prediction qualifies as "the pattern works." If 60% of instances meet the prediction, is that success? Is 55%? Is 51%? The threshold is committed in advance, not chosen after seeing the result. **Step 5\. Quantify the failure criterion.** The claim must specify what fraction qualifies as "the pattern does not work." This is not always the complement of the success criterion. The failure criterion includes both the case where too few instances meet the prediction and the case where the conditional rate is statistically indistinguishable from the baseline rate. **Step 6\. Specify the benchmark.** The claim must specify what the pattern's outcome is compared against. The unconditional base rate of the same outcome during the same period is the minimum benchmark. A permutation null (signals randomized in time) is a stronger benchmark. A bias-matched random rule is a stronger benchmark still. **Step 7\. Pre-commit all parameters.** Every choice from steps 1 through 6 is written to a file with a timestamp before any historical data is queried. Modifying any choice after seeing the result invalidates the test and removes its falsification value. A claim that has been through all seven steps is falsifiable. A claim that has been through fewer is still vibes. ## Worked example: head and shoulders Start with the published claim: "A head and shoulders pattern is a bearish reversal signal." The claim is not falsifiable as stated. Apply the seven steps. **Trigger (Step 1).** "A head and shoulders pattern is present at time t if all of the following hold: - A local maximum P1 exists between t−60 and t−40 (the left shoulder) - A local maximum P2 exists between t−40 and t−20 (the head), with P2 > P1 by at least 3% - A local maximum P3 exists between t−20 and t−5 (the right shoulder), with |P3 − P1| < 1.5% - A 'neckline' is the line connecting the troughs T1 (between P1 and P2) and T2 (between P2 and P3) - The closing price at time t is below the neckline extrapolated to t." The trigger is now a function. Any researcher running it on the same data produces the same signal series. **Prediction (Step 2).** "After the trigger fires, the price will close below the neckline by at least 5% in subsequent observations." **Time bound (Step 3).** "The prediction is evaluated 20 trading days after the trigger fires. If the price has closed at least 5% below the neckline at any point in those 20 days, the prediction is met. If not, the prediction fails." **Success criterion (Step 4).** "The pattern is considered to have signal if at least 55% of historical trigger instances meet the prediction, AND the conditional win rate exceeds the baseline win rate by a margin that is statistically significant at the 5% level." **Failure criterion (Step 5).** "The pattern is rejected if either the conditional win rate is below 55%, OR the conditional win rate is not statistically distinguishable from the baseline at the 5% level." **Benchmark (Step 6).** "The baseline is the unconditional rate at which the price closes at least 5% below an arbitrary reference line within any 20-day window during the same historical period. A permutation null is also computed by randomizing the timestamps of all trigger instances 5000 times and recomputing the conditional rate." **Pre-commitment (Step 7).** "All parameters (40-day shoulder windows, 3% head magnitude, 1.5% shoulder symmetry tolerance, 5% follow-through magnitude, 20-day evaluation horizon, 55% threshold, 5% significance level) are committed to a file with a timestamp before any backtest is executed." The claim has been transformed. It is now a falsifiable hypothesis. It can be tested against historical data. The result will be either rejection or non-rejection of the null at the 5% level. Most rigorous tests of patterns like head and shoulders find conditional win rates close to the unconditional baseline, with differences that are not statistically significant. The pattern, once operationalized, fails the test. The claim that the pattern is a reversal signal is rejected by the data. The trader who has done the work to falsify a popular pattern is ahead of the trader who has not. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## The statistical machinery The success criterion requires comparing two conditional probabilities: the rate at which the prediction is met given the pattern fires, and the rate at which the prediction is met given a random or baseline condition. The difference is tested for statistical significance. $$ \\Delta p = p\_{\\text{pattern}} - p\_{\\text{baseline}} $$ $$ \\text{SE}(\\Delta p) = \\sqrt{\\frac{p\_{\\text{pattern}}(1 - p\_{\\text{pattern}})}{N\_{\\text{pattern}}} + \\frac{p\_{\\text{baseline}}(1 - p\_{\\text{baseline}})}{N\_{\\text{baseline}}}} $$ Where p\_pattern is the observed conditional rate, p\_baseline is the unconditional rate, and N\_pattern and N\_baseline are the corresponding sample sizes. The z-score for the comparison is z = Δp / SE(Δp). If |z| > 1.96, the difference is significant at the 5% level and the pattern has evidence of signal. If |z| < 1.96, the difference is consistent with sampling noise and the pattern has not been distinguished from the baseline. Concrete example. A head and shoulders trigger fires 200 times in a 20-year sample. Of those, 110 lead to a 5% drop within 20 days. The conditional rate is 0.55\. The unconditional rate of 5% drops within any 20-day window during the same period is 0.51\. The difference is 0.04\. The standard error is approximately √(0.25/200 + 0.25/4000) ≈ 0.036\. The z-score is 0.04 / 0.036 ≈ 1.11\. The difference is not significant at the 5% level. The pattern has not been distinguished from the baseline. The claim is rejected. A 4-percentage-point conditional advantage that looks impressive in a marketing slide collapses under the standard error of the comparison. This is the experience most published TA patterns deliver when tested rigorously. ## Visualizing the transformation ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/05/ChatGPT-Image-May-20--2026--04_44_04-PM.png) The picture is the entire article in one image. The left side is most published TA. The right side is the only form of TA that can be evaluated. ## Claims that cannot be made falsifiable Some TA claims are structurally unfalsifiable and cannot be saved by any transformation. These should be rejected before any testing. Circular claims. "The trend is your friend" is true by definition if "trend" is defined as the price direction that is going to continue. The claim contains no testable content because the definition includes the outcome. Claims with infinite escape hatches. "This works when the conditions are right" is unfalsifiable because any failure becomes evidence of wrong conditions, not evidence against the claim. The list of conditions can be extended without limit to absorb any counter-example. Result-dependent definitions. "A real breakout is one where the price keeps going. A failed breakout was never a real breakout." The pattern is defined by the outcome, so the pattern can never fail. The transformation cannot fix this without redefining the pattern, which changes the claim. Claims that reject all tests. Some TA practitioners reject any quantitative test as "missing the art of the chart." This is the explicit declaration that the claim is not in the domain of evidence. It is honest in a way most TA is not. It also makes the claim irrelevant to anyone running real capital. ## What this changes for the practitioner Three practical consequences. Refuse to evaluate any TA claim that has not been put through the seven-step transformation. If the claim cannot be coded into a deterministic function, it is not a rule. If it has no time horizon, it is not a prediction. If it has no benchmark, it is not testable. Treat the failure of a transformed claim as the more common outcome. When real TA patterns are tested with operational definitions and proper benchmarks, the majority of them produce no significant signal. This is the empirical pattern. Expecting it in advance prevents disappointment and saves capital. Be willing to publish the rejections. The TA practitioner who tests 30 patterns and reports the 2 that survived has done useful work for the community. The practitioner who reports only the 2 successes has done the opposite. --- ## KEY POINTS - Most published TA is unfalsifiable because the claims are subjective, time-unbounded, parameter-free, cherry-picked, baseline-free, result-dependent, or built to absorb failures with new conditions. - Every TA claim that is not structurally unfalsifiable can be transformed into a falsifiable specification through a seven-step procedure: operationalize the trigger, operationalize the prediction, bound the horizon, quantify the success criterion, quantify the failure criterion, specify the benchmark, and pre-commit all parameters. - The trigger must be a deterministic function. Two researchers running the same function on the same data must produce identical signals. Subjective pattern recognition does not survive this step. - The benchmark is the unconditional rate of the same outcome during the same period. A permutation null is a stronger benchmark. A bias-matched random rule is stronger still. - The statistical test compares the conditional rate to the baseline rate. Δp = p\_pattern − p\_baseline, with SE computed from both sample sizes. |z| > 1.96 is significant at the 5% level. - Worked head-and-shoulders example: 200 triggers, 110 successes (55%), baseline 51%, z ≈ 1.11, p > 0.05\. The pattern is not distinguished from the baseline. The claim is rejected. This is the typical empirical result for popular patterns. - Some TA claims are structurally unfalsifiable and cannot be saved: circular claims, claims with infinite escape hatches, result-dependent definitions, and claims that reject all quantitative testing on principle. These should be rejected before any testing. - Practitioner consequences: refuse to evaluate untransformed claims, expect most transformed claims to fail the test, publish the rejections alongside the survivors. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Evidence-Based Technical Analysis - David Aronson (Amazon)](https://www.amazon.com/s?k=Evidence-Based+Technical+Analysis+David+Aronson&ref=aligrithm.com) - [Systematic Trading - Robert Carver (Amazon)](https://www.amazon.com/s?k=Systematic+Trading+Robert+Carver&ref=aligrithm.com) ### 1.18 Predictive Power vs Long Bias: The Hidden Trap in Backtests URL: https://aligrithm.com/predictive-power-vs-long-bias-the-hidden-trap-in-backtests/ Last updated: 2026-07-27T13:22:56.000Z # A backtest that shows a positive return proves nothing about predictive power. The returns might be drift collected by long bias, sector exposure collected by an implicit tilt, factor exposure collected by structural design, or any combination. The "edge" the trader sees on the equity curve might be zero once every exposure is accounted for. The previous articles in this series covered the simplest version of this problem: a long-biased rule in an upward-drifting market collects free return from the drift. That is the visible trap. The hidden trap is more dangerous because it operates through exposures the trader did not explicitly choose and does not see in the rule's parameters. Sector tilt, factor tilt, regime tilt, and time-of-day tilt all produce returns that look identical to predictive edge on the surface and behave completely differently in live trading. Most retail strategies that "work" in backtests and fail in live trading fail for exactly this reason. The backtest measured exposure. Live trading exposed the exposure. ## The general exposure decomposition Every backtest return can be decomposed into a sum of exposures times factor returns, plus a residual predictive contribution, plus noise. $$ R\_{\\text{backtest}} = \\sum\_{i} \\beta\_i \\cdot \\mu\_i + \\alpha\_{\\text{predictive}} + \\epsilon $$ Where β\_i is the rule's average exposure to factor i during the test window, μ\_i is the realized average return of factor i during the same window, α\_predictive is the part of the return not explained by any factor exposure, and ε is residual noise. The factors in the sum can be anything: market direction, momentum, value, size, quality, low volatility, specific sectors, specific countries, specific currencies, time of day, day of week, volatility regime, interest-rate regime. Any source of return that the rule is exposed to belongs in the sum. A rule with edge has a non-zero α\_predictive after every relevant β has been accounted for. A rule without edge has α\_predictive ≈ 0; whatever the backtest showed came from the exposures. The job of evaluation is to estimate every β\_i × μ\_i correctly so that what is left in α is actually edge and not just an exposure the analyst forgot to subtract. ## The forms of hidden bias The visible bias (long/short percentage on a single market) is the easiest to detect and the easiest to fix. The hidden biases require explicit diagnostic work. **Sector or factor tilt.** A rule that signals "long" on the QQQ from 2010 to 2021 is largely a long-tech bet. The rule's returns track the tech sector's outperformance over the period. The same rule on equal-weighted broad indices over the same period looks nothing like the QQQ version. The sector tilt was the source of the return, not the signal logic. **Time-of-day tilt.** Most equity index returns historically accrue overnight. A rule that systematically holds positions across the close picks up the overnight risk premium regardless of any signal logic. Two rules that look identical except for whether they hold overnight or only intraday can have backtest Sharpe ratios that differ by a factor of two purely from this exposure. **Volatility regime tilt.** A rule that is leveraged or fully invested only during low-volatility periods has selected the calmest portion of the historical sample. Low-vol periods historically correlate with positive drift. The rule's apparent risk-adjusted return is partially the regime selection, not the timing of any specific signal. **Country or currency tilt.** A rule trading on multiple instruments that happens to be long EM during a strong EM decade is collecting the EM beta. The rule's returns generalize to nothing if EM weakens. Same logic applies to currencies, commodities, and any cross-sectional concentration. **Period selection tilt.** Choosing the test window itself can hide bias. The 2009 to 2021 window has different statistical properties than any window containing 2000 to 2002 or 2008\. A rule "tested over the past 12 years" on US equities is being evaluated in one specific drift regime. The choice of window is implicit bias and is often unreported. Every one of these tilts shows up as positive backtest return without any signal-level edge. They are invisible in the rule's parameters and visible only after explicit factor accounting. ## Worked example: the QQQ "trend follower" A simple 50-day / 200-day moving average crossover rule on the QQQ ETF from 2010 to 2021\. Long when the 50-day is above the 200-day, flat otherwise. Headline backtest results: - Annualized return: 22% - Sharpe ratio: 1.4 - Long 88% of the time over the period - Max drawdown: 17% These numbers look like a discovery. They are not. The decomposition: - Buy-and-hold QQQ over 2010-2021: roughly 18% annualized. The rule was long 88% of the time, so its baseline drift collection is 0.88 × 18% ≈ 16%. - The remaining 6% might be edge, or it might be additional exposure. - The QQQ was a concentrated tech bet during this period. Equal-weighted S&P 500 returned roughly 14% annualized. The "tech sector tilt" beyond broad equity exposure contributed roughly 4 percentage points per year to QQQ. The rule's long QQQ exposure inherited this. - The signal timing (the part that says "be long when 50 crosses above 200") contributed roughly 1 percentage point per year after both the drift and the tech tilt are removed. The full decomposition: - 16% from broad equity drift × rule's long exposure - 5% from tech sector tilt × rule's long QQQ exposure - 1% from actual signal timing (the edge) - 22% headline The rule is 95% equity beta and tech sector tilt. The signal contribution is small enough that transaction costs would consume most of it. In live trading from 2022 onward, the rule lost money. The tech tilt that contributed 5% per year on the way up subtracted on the way down. The "edge" was beta in disguise. ## Diagnostic tests for hidden bias Six tests, in increasing order of sophistication, that separate bias from edge. **1\. Detrended re-run.** Subtract the market's mean daily return from each day's return. Re-run the rule on the detrended series. A bias-driven rule's return collapses toward zero. An edge-driven rule keeps most of its performance. **2\. Signal permutation.** Keep the actual market returns intact. Randomly permute the rule's signal timestamps 5000 times. Compute the rule's return on each permutation. The distribution of permuted returns is what the rule earns from its exposures alone, with no timing edge. If the actual rule return sits inside this distribution, the rule has no timing edge. If it sits outside the tail, there is signal. **3\. Bias-matched random rule.** Generate 5000 random rules with the same long/short proportions as the actual rule. Each random rule has no signal logic. Compute the mean return of each. The actual rule's return minus the mean of the matched-random distribution is the predictive contribution. If this gap is small relative to the noise in the matched-random distribution, the rule is bias. **4\. Regime split.** Partition the test window into bull, bear, and sideways sub-periods using a regime classifier (e.g., 200-day moving average sign, trailing 12-month return sign, or volatility regime). Compute the rule's return in each sub-period. A pure-bias rule shows extreme regime dependence (great in bulls, terrible in bears). An edge-driven rule shows moderate regime dependence and positive results across regimes. **5\. Factor regression.** Regress the strategy's daily returns on a multifactor model (market, size, value, momentum, quality, plus relevant sector or currency factors). The intercept of the regression is the alpha after factor exposures. The R² is how much of the strategy's return the factors explain. A strategy with R² of 0.9 against a standard factor model has 10% of its return left over as alpha. That 10% is the candidate edge; the rest is factor exposure that could be obtained more cheaply via ETFs. **6\. Instrument swap.** Apply the same rule with the same parameters to a different but related instrument (e.g., QQQ rule applied to SPY, or SPX rule applied to FTSE, or US trend rule applied to JP trend). Edge tends to generalize across closely related instruments. Pure bias does not, because the bias was specific to the drift of the original instrument during the original window. A strategy that passes all six tests has earned the benefit of the doubt. A strategy that fails any of them is bias dressed as edge. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Visualizing the iceberg ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/05/ChatGPT-Image-May-20--2026--04_39_05-PM.png) The picture is the trap. The trader sees the tip. The actual return generation is below the waterline. When the regime changes, the layers below the waterline change sign and the tip melts. ## The non-stationarity twist Hidden bias becomes more dangerous when the regime shifts. A 90% long bias on US equities from 2009 to 2021 collected approximately 10% per year of free drift. The same 90% long bias from 2000 to 2002 lost that drift back, and then some. A rule that was implicitly betting on tech sector outperformance from 2010 to 2021 was implicitly betting on tech sector underperformance whenever that tilt reversed. Bias is a directional bet whose value depends on the regime. In a backtest restricted to one regime, the bet looks like a free return. In live trading across regime shifts, the same bet is a leveraged exposure to whichever direction the regime takes. This is the mechanism by which "great backtests" become "live losses." The hidden bias was net positive in the historical sample and turns net negative in the next sample. The signal logic, which was never the source of the return, has nothing to contribute when the bias contribution goes negative. ## Why the trap stays hidden The trap stays hidden because every default reporting practice favors the headline number. The annualized return is reported. The Sharpe ratio against zero is reported. The equity curve goes up. The drawdown looks manageable. None of these surface the underlying decomposition. The factor regression, the regime split, the bias-matched comparison, and the signal permutation are all extra work. Most strategy reports omit them. The trader who runs them sees fewer publishable strategies, makes fewer overconfident claims, and has fewer live failures. The trader who does not run them ships more strategies, sounds more confident, and discovers the decomposition in production at considerable cost. The path of least cognitive resistance is to accept the headline. The path of least financial damage is to refuse to. ## What this changes operationally Three changes. Every strategy report should include the bias decomposition. Headline return, exposure contributions, residual alpha. Without this breakdown, the headline number is incomplete information. Every strategy should pass at least three of the six diagnostic tests before being approved for live trading. Different rules require different diagnostics: a long-biased equity timing rule needs the detrended re-run and the regime split; a multi-asset rule needs the factor regression and the instrument swap; a timing rule on a single asset needs the signal permutation and the matched-random comparison. Treat positive backtest returns as a starting point for investigation, not as proof of anything. The default assumption is that the return came from bias. The burden of proof is on the trader to show that residual alpha exists after every exposure is accounted for. --- ## KEY POINTS - A backtest with positive return proves nothing about predictive power. The return decomposes into a sum of exposure × factor contributions plus a residual α\_predictive plus noise. - Hidden biases beyond visible long/short percentage include sector tilt, time-of-day tilt, volatility regime tilt, country/currency tilt, and test-window selection tilt. Each generates positive return without any signal-level edge. - The QQQ 50/200 trend rule from 2010 to 2021 returned 22% headline. Decomposition: 16% from broad equity drift × long exposure, 5% from tech sector tilt, 1% from actual signal timing. The signal contributed almost nothing once factor exposures were stripped out. - Six diagnostic tests separate bias from edge: detrended re-run, signal permutation, bias-matched random rule, regime split, factor regression, instrument swap. A strategy that passes all six has earned the benefit of the doubt. - Bias is a directional bet whose value depends on the regime. The same long bias that collected 10% per year of drift in a bull regime can lose more than that in a bear regime. Backtests over single regimes hide this risk. - The trap stays hidden because the default reporting (headline return, Sharpe vs zero, equity curve) does not surface the decomposition. The factor regression, regime split, and permutation test are extra work that most strategy reports skip. - Treat positive backtest returns as a starting point for investigation, not as proof. The default assumption is that the return came from bias. The burden of proof is on the trader to demonstrate residual alpha after every exposure is accounted for. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Evidence-Based Technical Analysis - David Aronson (Amazon)](https://www.amazon.com/s?k=Evidence-Based+Technical+Analysis+David+Aronson&ref=aligrithm.com) - [Systematic Trading - Robert Carver (Amazon)](https://www.amazon.com/s?k=Systematic+Trading+Robert+Carver&ref=aligrithm.com) ### 1.17 Why Benchmarks Matter in Rule Evaluation URL: https://aligrithm.com/why-benchmarks-matter-in-rule-evaluation/ Last updated: 2026-07-27T13:22:55.000Z # A trading rule's performance number, on its own, is meaningless. A strategy that returned 15% per year is excellent if the market returned 0%, mediocre if the market returned 14%, and embarrassing if the market returned 25%. The number 15% in isolation carries no information about whether the rule did anything useful. Information appears only when the rule is compared to a benchmark. The choice of benchmark is the choice of what claim the rule is allowed to make. A bad benchmark turns a worthless rule into a publishable result. A good benchmark exposes the parts of the return that came from the rule and separates them from the parts that came from anything else. Most retail backtests pick the worst possible benchmark (zero) and call the result evidence of edge. The result is the same long-biased momentum-tilted strategies dressed up as discoveries, year after year. ## Everyone is a genius in a bull market The cleanest statement of the benchmark problem: any rule that is long-biased during a period when the market drifted up will produce positive returns regardless of whether the rule itself has any predictive content. A simple thought experiment. Take a roulette wheel and use it to generate buy and sell signals on the SPX over the past 25 years. The roulette wheel produces buys 60% of the time and sells 40% of the time. The roulette wheel has no predictive power. It is a roulette wheel. The SPX drifted up at roughly 0.04% per day over that period. The roulette wheel, by virtue of being long more often than short, captures part of that drift. Its expected return under no predictive power is: (0.60 × 0.04%) − (0.40 × 0.04%) = 0.008% per day ≈ 2% per year Two percent per year of pure free return for a system that is randomness. A backtest of this roulette-wheel strategy, reported against a zero benchmark, would show "the strategy generates 2% per year over a 25-year sample." That sentence is true. The implication that the strategy is doing something is false. The fix is to compare the rule against a benchmark that captures the same structural exposure as the rule but without any predictive content. The remainder, if positive, is what the rule actually contributed. ## The bias-matched random rule The cleanest benchmark for testing predictive timing is a random rule with the same long/short bias as the actual rule. The expected return of a random rule that is long with probability P\_L and short with probability P\_S = 1 − P\_L, on a market with average return μ\_market per period, is: $$ \\mathbb{E}\[R\_{\\text{random}}\] = (P\_L - P\_S) \\cdot \\mu\_{\\text{market}} $$ This is the return the rule would produce purely from its directional bias interacting with the market's drift. It contains zero predictive content. Anything the actual rule generates above this number is the rule's predictive contribution. The adjusted return of the rule, the part attributable to actual predictive power, is: $$ R\_{\\text{adjusted}} = R\_{\\text{observed}} - \\mathbb{E}\[R\_{\\text{random}}\] $$ Worked example. A rule on the SPX from 2000 to 2024 spends 90% of the time long and 10% of the time short. The observed average daily return is 0.10%. The SPX drifted up at 0.05% per day over the same window. Random rule expected return = (0.90 − 0.10) × 0.05% = 0.04% per day Annualized random contribution = 0.04% × 252 ≈ 10% Annualized observed return = 0.10% × 252 ≈ 25% Adjusted return = 25% − 10% = 15% The rule actually contributed 15 percentage points per year. The other 10 percentage points came from being long-biased in a market that went up. A backtest reported against a zero benchmark would claim 25%. A backtest reported against the bias-matched benchmark claims 15%. The two numbers describe different things, and the second one is the one that captures the rule's actual skill. ## The hierarchy of benchmarks Different benchmarks answer different questions. A complete evaluation uses several. From least to most informative: **Zero return.** The naive benchmark. Useful only for market-neutral strategies that have no directional exposure. For any long-biased or short-biased rule, the zero benchmark is wrong because it credits the rule with returns that came from drift. **Risk-free rate.** The floor any risky capital allocation must clear. A strategy that returns less than cash is a strategy that destroyed value. Useful as a hard minimum but rarely informative beyond that. **Buy-and-hold of the underlying.** The relevant benchmark for any long-only strategy on a single asset. If a fund is long-only on the SPX and underperforms buy-and-hold SPX, the fund is destroying value relative to a no-effort alternative. Many published equity strategies fail this benchmark. **Bias-matched random rule.** The benchmark for testing pure predictive timing. Strips out the contribution of directional bias × market drift. The cleanest test of whether the rule's signals carry information about future returns. **Peer strategies in the same class.** The benchmark for competitive ranking. A trend-following strategy should be compared to other trend-following strategies (or to indices that track the trend-following industry). Outperforming the peer benchmark is the claim that the strategy is differentiated, not just that it works. **Factor model.** The benchmark for risk-adjusted return. A regression of the strategy's returns on a set of factors (market, size, value, momentum, quality) decomposes the return into the part explained by known factors and the part that is unexplained. The unexplained part is the alpha. Any strategy whose returns can be fully explained by known factors is not adding value; it is delivering factor exposure that could be obtained more cheaply. A rigorous evaluation reports the rule's performance against several of these benchmarks. The pattern of results tells the trader what the rule is actually doing. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Why the benchmark choice changes the conclusion The same backtest can look excellent against one benchmark and worthless against another. This is not a flaw in the test. It is the point of the test. Consider a long-biased trend-following rule on SPX over the past 15 years. Against zero: 12% per year. Looks great. Against buy-and-hold SPX: roughly flat. The rule captured the same drift the market captured. Against a bias-matched random rule: marginal positive. The rule's specific timing added maybe 1% per year. Against a momentum factor: roughly zero alpha. The rule's returns are mostly explained by exposure to the momentum factor, which is freely available through ETFs at low cost. Same rule. Four benchmarks. Four conclusions. The honest report is all four numbers, not the one that flatters the rule. The dishonest report is the 12% against zero. ## Visualizing the benchmark hierarchy ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/05/ChatGPT-Image-May-20--2026--04_35_09-PM.png) The visual is the essence of the benchmark problem. The same rule shrinks dramatically as the benchmark gets more honest. The marketer picks the leftmost bar. The honest evaluator reports all five. ## The information ratio: a unified comparison The information ratio extends the Sharpe ratio idea to performance relative to any benchmark. It is the Sharpe ratio computed on the difference between the rule's returns and the benchmark's returns. $$ \\text{IR} = \\frac{\\mathbb{E}\[R\_{\\text{rule}} - R\_{\\text{benchmark}}\]}{\\sigma(R\_{\\text{rule}} - R\_{\\text{benchmark}})} $$ The numerator is the mean excess return over the benchmark. The denominator is the standard deviation of that excess return. The IR is high when the rule consistently beats the benchmark by a stable margin. The IR is low when the rule occasionally beats the benchmark by a lot but is unreliable, or when the rule's average outperformance is small. A strategy with a Sharpe ratio of 1.2 against zero and an information ratio of 0.1 against buy-and-hold is a strategy that captures drift, not edge. A strategy with a Sharpe ratio of 1.2 against zero and an information ratio of 1.0 against buy-and-hold is a strategy that genuinely beats the alternative on a risk-adjusted basis. The IR against the right benchmark is the single number that summarizes whether a rule adds value. The Sharpe ratio against zero, in isolation, does not. ## The retail mistake Almost every retail backtest reports a single annualized return and a single Sharpe ratio, both computed against zero. This is the worst possible evaluation. A few patterns recur in retail backtest claims: A long-only SPX trend-following rule that beats zero by 12% but matches buy-and-hold gets sold as a 12% strategy. It is a 0% strategy. A short-vol rule that pays steady 8% returns most of the time and blows up every few years gets sold as a high-Sharpe strategy against zero. Against a properly bias-matched benchmark that captures the same volatility-selling exposure, the alpha is negative. A momentum strategy that returns 18% per year gets sold as a quant edge. Against the momentum factor, the alpha is zero. The strategy is delivering known factor exposure at a markup. The benchmark fix in each case is simple and the conclusion changes completely. The strategy seller does not want the conclusion to change. The honest researcher does. ## What this changes operationally Three changes to standard practice. Always report multiple benchmarks. At minimum: zero, the relevant buy-and-hold, and a bias-matched random rule. For institutional reporting, add a factor model. The pattern across benchmarks tells the reader what the rule is. Pre-commit to which benchmark is the primary one. The primary benchmark is the one that matches the claim the rule is making. A market-timing rule's primary benchmark is the bias-matched random rule. A long-only equity rule's primary benchmark is buy-and-hold. A market-neutral rule's primary benchmark is the risk-free rate. Committing in advance prevents post-hoc benchmark shopping. Report the information ratio against the primary benchmark, not just the Sharpe ratio against zero. The IR is the single number that compresses the evaluation into one digit. The Sharpe against zero is the number that flatters the rule. ## KEY POINTS --- - A trading rule's return in isolation is meaningless. Information appears only when the return is compared to a benchmark. - Any long-biased rule in an upward-drifting market produces positive returns even with zero predictive content. The zero benchmark credits the rule for returns it did not generate. - The bias-matched random rule benchmark strips out the directional-bias × market-drift contribution: E\[R\_random\] = (P\_L − P\_S) × μ\_market. Anything the actual rule generates above this is the predictive contribution. - A worked SPX example with 90% long bias on a 0.05% drift yields 10% per year of free return from bias alone. A rule reporting 25% annualized has actually contributed 15%, not 25%. - The hierarchy of benchmarks, from least to most informative: zero, risk-free rate, buy-and-hold of the underlying, bias-matched random rule, peer strategies in the same class, factor model. - Different benchmarks answer different questions. A rigorous evaluation reports the rule's performance against several of them. The pattern across benchmarks shows what the rule is actually doing. - The information ratio (Sharpe ratio of excess return over the benchmark) is the unified comparison metric. IR against the right benchmark is the single most informative number for the rule. - The retail mistake is reporting one number against zero. Long-only trend strategies sold as 12% strategies are often 0% strategies against buy-and-hold. Momentum strategies sold as quant edge are often zero-alpha factor exposure. - Operational fix: report multiple benchmarks, pre-commit to the primary benchmark before testing, report the IR against the primary benchmark alongside the Sharpe against zero. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Evidence-Based Technical Analysis - David Aronson (Amazon)](https://www.amazon.com/s?k=Evidence-Based+Technical+Analysis+David+Aronson&ref=aligrithm.com) - [Systematic Trading - Robert Carver (Amazon)](https://www.amazon.com/s?k=Systematic+Trading+Robert+Carver&ref=aligrithm.com) ### 1.16 The Null Hypothesis for Trading Systems URL: https://aligrithm.com/the-null-hypothesis-for-trading-systems/ Last updated: 2026-07-27T13:22:54.000Z # Every trading system test starts with a single sentence. The sentence is: "this rule has no predictive power." Until evidence forces a change, that sentence is the working belief. The technical name for it is the null hypothesis. Building a trading system is the process of trying to falsify the null. The reason this matters operationally is that the null hypothesis governs every claim the trader is allowed to make. A rule that has not falsified the null is not a strategy. It is a candidate. A rule that has falsified the null is a strategy worth running. The line between the two is not the equity curve. The line is whether the backtest forced the null hypothesis into rejection. Most retail trading research skips this step entirely. The trader runs a backtest, sees a profit, and ships. No null was stated. No falsification was attempted. The output is a hope dressed as evidence. ## What the null hypothesis is For a trading system, the null hypothesis is the formal version of "this rule is worthless." Worthless is defined precisely: the rule's expected return on detrended data is zero or negative. Detrended data removes the market's free drift, so what is left is whatever predictive content the rule actually carries. The formal statement of the null and alternative hypotheses: $$ H\_0: \\mu^{\\text{detrended}}\_{\\text{rule}} \\leq 0 \\quad \\text{vs.} \\quad H\_A: \\mu^{\\text{detrended}}\_{\\text{rule}} > 0 $$ Where μ\_rule^detrended is the rule's true mean return per period on detrended market data. H₀ says the rule has zero or negative edge once free drift is stripped away. H\_A says the rule has positive edge. The two propositions are mutually exclusive (only one can be true) and exhaustive (one of them must be true). Falsifying H₀ logically forces H\_A. The detrending is not a cosmetic detail. A rule that holds long 90% of the time on a market that drifts up at 0.05% per day collects free return that has nothing to do with the rule. The null hypothesis applies to the part of the return the rule actually generated, which is the detrended component. Article 12 covered this decomposition. ## Why H₀ is the falsification target, not H\_A This is the single most important asymmetry in hypothesis testing. The reasoning is logical, not philosophical. H₀ is a single specific claim. It picks one value of the parameter (μ = 0) and says "the true value is this or worse." To falsify H₀, the test only has to show that the observed evidence is improbable under one specific assumption. H\_A is an infinite set of claims. The rule's true edge could be 0.001% per day, or 0.01%, or 0.1%, or anything in between or above. To falsify H\_A directly, the test would have to rule out every possible positive value of the parameter. That requires an infinite number of tests. Not feasible. So the scientific procedure is: assume H₀, see if the data forces its rejection, and accept H\_A by default when H₀ falls. The asymmetry is not arbitrary. It is the only direction in which finite evidence can deliver a decision. ## Why the null is assumed true at the start Two reasons. Both are domain-independent. The first is skepticism. The default scientific position toward any new claim is disbelief. Most claims, including most claims about trading edges, turn out to be wrong. Starting from "this rule has edge" and looking for confirming evidence is what humans do naturally and what science refuses to do. The presumption of innocence in a criminal trial works the same way. The accused is innocent until evidence proves otherwise. A trading rule has no edge until evidence proves otherwise. The second is simplicity. Occam's razor says the simpler explanation for an observation is more likely to be correct than a more elaborate one. Two explanations exist for a profitable backtest: the rule has predictive power, or the rule got lucky in the specific sample. Luck is the simpler hypothesis. It requires no additional structure in the world. Predictive power requires an actual mechanism connecting the rule's signal to future returns. The simpler hypothesis is the default. The data has to do work to dislodge it. Both reasons converge on the same operational rule: assume the rule is worthless until the evidence is strong enough to force a different conclusion. ## The three ingredients of a hypothesis test A complete hypothesis test for a trading rule has exactly three components. None of them are optional. 1. **The null hypothesis itself.** The precise statement of what the rule would do if it had no predictive power. For trading systems, the canonical form is μ\_rule^detrended ≤ 0. 2. **A test statistic.** A single number computed on the backtest that summarizes the rule's observed performance. Mean return per signal, Sharpe ratio, hit rate, profit factor. The choice of test statistic must be made before the test runs. 3. **A sampling distribution under H₀.** The distribution the test statistic would take if H₀ were true. This is the hardest piece. It requires a model of what "the rule has no edge" actually looks like in terms of generated data. The model must be realistic enough to reflect the noise structure of real markets. A test missing any of these three components is not a hypothesis test. It is a backtest with confidence theater attached. ## Constructing the null model The null hypothesis is a verbal statement. The null model is the machinery that turns it into something testable. The choice of null model is the choice that determines whether the test is rigorous or decorative. Three levels of null model exist, in increasing order of realism. The naive null assumes returns are independent, identically distributed normal draws with mean zero. This is what introductory textbooks teach. It is wrong for markets because real returns have fat tails, volatility clustering, and serial autocorrelation. A p-value computed against the naive null overstates the rule's significance, because the naive null underestimates how extreme outcomes can be under chance alone. The bootstrap null resamples returns from the actual market series (after detrending) with replacement. The new series has the same marginal distribution as the real one (same mean of zero by construction, same standard deviation, same skew and kurtosis) but the time order is destroyed. Any rule that depended on the time structure of returns will produce results consistent with no edge on the bootstrap null. Any rule that produced its edge from the marginal distribution alone will look identical. The block bootstrap null is the most realistic. It resamples blocks of consecutive returns (typically 5 to 20 days long) instead of individual days. Blocks preserve short-term autocorrelation and volatility clustering. The new series has the same marginal distribution and approximately the same dependence structure as the real one. This is the null that most trading rule tests should use. A fourth option is the permutation null. Keep the actual market returns intact, but randomly permute the rule's signals so they fire on different days than they originally fired. This breaks the rule's specific timing claim while leaving everything else about the data unchanged. The permutation null is the cleanest test of "does this rule's signal predict anything?" because it isolates the predictive claim from everything else. The choice of null model is not a technical detail. A bad null produces fake significance. A realistic null produces honest results, including the result "this rule has no edge that the data supports." ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## The decision rule The hypothesis test rejects H₀ when the observed test statistic is improbable under H₀. The precise statement: $$ \\text{Reject } H\_0 \\quad \\text{if} \\quad p = P(\\bar{R}^{\*} \\geq \\bar{R}\_{\\text{obs}} \\mid H\_0) < \\alpha $$ Where R\_obs is the observed test statistic from the actual backtest, R\* is the test statistic computed on a single draw from the null model, and α is the significance threshold committed to before the test ran. The standard choice is α = 0.05. If p < α, the observed result is rare enough under H₀ that the trader rejects H₀ and accepts H\_A. The rule has evidence of edge. If p ≥ α, the observed result is consistent with what H₀ could produce by chance. The trader fails to reject H₀. The rule does not have evidence of edge. This is not the same as proving the rule has no edge. The rule might have edge that this test was not powerful enough to detect. The honest report is "we cannot reject the null at the chosen threshold," not "the rule has no edge." ## Visualizing the null hypothesis test ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/05/ChatGPT-Image-May-20--2026--04_23_31-PM.png) The picture is the entire null hypothesis framework in one image. The grey histogram is what the rule's performance would look like under H₀. The dashed red line is the threshold the trader committed to before running the test. The green line is what the rule produced. The shaded tail is the p-value. Crossing the dashed line is the moment of falsification. ## Type I and Type II errors Every hypothesis test can fail in two ways. Type I error: rejecting H₀ when H₀ is actually true. The trader concludes the rule has edge when it does not. This is a false positive. The cost in trading is severe. The trader allocates capital, time, and operational complexity to a strategy that will eventually reveal it has no edge. The α threshold directly controls the Type I error rate. At α = 0.05, the trader is willing to accept a 5% probability of a false positive on any single test. Type II error: failing to reject H₀ when H₀ is actually false. The trader concludes the rule has no edge when it does. This is a false negative. The cost is opportunity: a real edge is discarded. The Type II error rate is controlled by the test's power, which depends on the size of the true edge, the sample size, and the noise in the data. For trading systems, Type I errors are usually more expensive than Type II errors. A worthless strategy in live trading consumes capital, attention, and confidence. A real edge that was missed costs only the opportunity. The asymmetric cost is one reason the conventional α = 0.05 threshold tends to be conservative in trading research. Some practitioners use α = 0.01 or even α = 0.005 for important live-capital decisions. ## What this changes in practice Three operational consequences follow. Before any backtest, write down the null hypothesis, the test statistic, the null model, and the α threshold. Save the file with a timestamp. Run the backtest. Compute the p-value. Apply the decision rule. Nothing about any of these four choices can be modified after seeing the result. Treat "fail to reject" as the default outcome. Most rules will not falsify the null. That is correct behavior, not a bug. The rules that survive a properly constructed null test are rare. Those are the rules worth trading. Report the null test alongside the backtest performance. "Strategy returned 14% with Sharpe 1.2, p-value against block bootstrap null = 0.03" is honest. "Strategy returned 14% with Sharpe 1.2" is incomplete. --- ## KEY POINTS - The null hypothesis for any trading system is the formal version of "this rule has no predictive power." Operationally, μ\_rule^detrended ≤ 0. - The null and alternative hypotheses are mutually exclusive and exhaustive. Falsifying H₀ forces acceptance of H\_A. - H₀ is the falsification target because it is a single specific claim. H\_A is an infinite set of claims and cannot be falsified directly. - Two reasons justify assuming H₀ true at the start: scientific skepticism (default disbelief in new claims) and Occam's razor (luck is a simpler explanation than predictive power). - A complete hypothesis test has three ingredients: a precise null, a defined test statistic, and a realistic sampling distribution under the null. - The null model determines the rigor of the test. Choices, in order of realism: naive iid normal (worst), bootstrap (preserves marginal distribution), block bootstrap (preserves dependence too), permutation of signals (cleanest test of timing claim). - Reject H₀ if p < α (typically 0.05). Otherwise fail to reject. "Fail to reject" is not the same as "accept the null." The rule might have edge the test could not detect. - Type I errors (false positives) are usually more expensive than Type II errors (false negatives) in trading. A worthless strategy consumes capital; a missed edge only costs opportunity. Conservative α thresholds reflect this asymmetry. - Before any backtest, commit in writing to the null, the test statistic, the null model, and the α threshold. Modifying any of them after seeing the result invalidates the test. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Evidence-Based Technical Analysis - David Aronson (Amazon)](https://www.amazon.com/s?k=Evidence-Based+Technical+Analysis+David+Aronson&ref=aligrithm.com) - [Systematic Trading - Robert Carver (Amazon)](https://www.amazon.com/s?k=Systematic+Trading+Robert+Carver&ref=aligrithm.com) ### 1.15 Induction in Trading: Why Past Patterns Are Always Uncertain URL: https://aligrithm.com/induction-in-trading-why-past-patterns-are-always-uncertain/ Last updated: 2026-07-27T13:22:54.000Z # Every claim a trader makes about the future is inductive. The trader observes past data, infers a pattern, and projects the pattern forward. That structure is the definition of induction, and induction is the form of inference whose conclusions are never certain. This is not a technical limitation of any specific method. It is a property of the type of reasoning involved. Even a perfect backtest, on a complete sample, with infinite computational power, produces a conclusion that holds only with some degree of probability and only if the future resembles the past in the relevant ways. The trader who claims certainty has either misunderstood the logical structure of the claim or is selling something. ## Deduction versus induction Two forms of inference run through every analytical argument. They behave differently. Deduction starts with a general premise and derives a specific conclusion. If the premises are true, the conclusion is necessarily true. The classic example: all men are mortal, Socrates is a man, therefore Socrates is mortal. The conclusion adds no new information beyond what the premises already contain. Deduction is certain but does not produce new knowledge about the world. Induction starts with specific observations and infers a general claim. The conclusion extends beyond what the premises strictly justify. The classic example: every healthy dog observed so far had four legs, therefore all healthy dogs have four legs. The conclusion is plausible but not certain. The next dog might have three legs. Induction adds new information about the world but pays for it in uncertainty. Trading runs almost entirely on induction. The chart pattern, the seasonality, the moving average crossover, the Sharpe-1.4 strategy backtested over 20 years: each is a generalization from a finite set of past observations to a claim about unobserved future cases. The premises are facts about history. The conclusions are bets about a future no one has seen yet. ## The standard inductive argument in trading The structure of a typical trading claim is induction by enumeration. The argument has two parts. Premise: over the past 20 years, a specific rule fired 1000 times, and in 700 of those instances the market closed higher 10 days later. Conclusion: in the future, when the rule fires, there is a 70% chance the market will close higher 10 days later. The premise is true. The historical signals fired the number of times stated, and the historical hit rate is the number observed. The conclusion is uncertain. It extends a claim about 1000 past cases to an unknown number of future cases that have not yet been drawn. The classical statistical machinery makes the uncertainty quantitative. The 95% confidence interval on the true hit rate, assuming the future is drawn from the same distribution as the past, is: $$ \\hat{p} \\pm 1.96 \\cdot \\sqrt{\\frac{\\hat{p}(1 - \\hat{p})}{N}} $$ Where p̂ is the observed hit rate and N is the number of signals. For p̂ = 0.7 and N = 1000: 0.7 ± 1.96 × √(0.21/1000) = 0.7 ± 0.028 = \[0.672, 0.728\] The interval says "with 95% confidence, the true hit rate is between 67.2% and 72.8% if the future is drawn from the same distribution as the past." That last clause is the catch. The interval captures sampling uncertainty. It does not capture the uncertainty about whether the same distribution still applies. ## Hume's problem The deeper uncertainty in induction is not statistical. It is logical. To justify projecting past patterns into the future, the inductive reasoner has to assume that the future will resemble the past in the relevant ways. The standard name for this assumption is the uniformity of nature. Without it, no inductive conclusion follows. The problem: there is no way to prove the uniformity of nature without using induction itself. Any argument that "the future has resembled the past in many cases, therefore the future will continue to resemble the past" is itself an inductive argument that depends on the very assumption being defended. The justification is circular. This problem is not new and does not go away with more data or better methods. Every scientific field deals with it. Every scientific field accepts induction as the only way to make new claims about the world while accepting that the resulting claims are not certain. What changes from field to field is how badly induction fails when the assumption breaks down. In physics, the uniformity of nature holds extremely well. In markets, it does not. ## Two dimensions of inductive strength Some inductive arguments are stronger than others. The strength depends on two factors. Quantity. The more independent observations supporting the generalization, the tighter the confidence interval on the sample statistic. The binomial CI shrinks proportional to 1/√N. Going from 100 to 10,000 observations narrows the interval by a factor of 10\. There is no exception to this in any well-behaved estimator. Quality. The degree to which the observed cases are representative of the future cases the trader wants to predict. A rule tested only in low-volatility bull markets has bad quality evidence for high-volatility bear markets. A signal tested only on the SPX has bad quality evidence for emerging-market equities. The CI calculated from the sample does not capture this dimension. The CI assumes the sample is representative. When it is not, the CI is wrong in a direction that cannot be detected from the sample alone. Quantity is easy to measure. Quality requires judgment about what the future will look like and which features of the past are relevant. Most retail induction is high-quantity and low-quality. The number of observations is large. The diversity of regimes those observations cover is small. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Visualizing the induction problem ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/05/ChatGPT-Image-May-20--2026--04_14_40-PM.png) The diagram shows the two layers of uncertainty. The narrow bar is sampling variability, which standard statistics handles. The wider bar is the inductive leap from past to future, which standard statistics cannot reach. ## Why induction fails harder in trading than in physics Three properties of markets weaken inductive inference relative to fields like physics. Non-stationarity. The underlying data-generating process changes over time. The volatility regime that produced the 1990s data is not the regime that produced the 2008–2009 data, which is not the regime that produced the 2020 covid data. Each chunk of history is a sample from a different population. Pooling them increases nominal N but breaks the assumption that all observations come from the same distribution. Active participants. When a pattern is published, traders act on it. The act of trading on the pattern changes the data-generating process. An edge that existed before publication may not exist after publication. This feedback is unique to social and financial systems and absent from physics. One history. There is no second universe to replay. Even if the same data-generating process were running, the trader has only one realization to study. Article 14 covered this problem. It interacts with induction by making the quality dimension of inductive evidence harder to assess from the data alone. The combined effect is that historical regularities in markets carry less inductive weight than historical regularities in repeatable, stationary, participant-independent domains. Trading is the hard mode of induction. ## The hypothetico-deductive method Science deals with the problem of induction by combining inductive observation with deductive testing. The five-stage process: 1. **Observation.** Notice a pattern in the data. This step is inductive. 2. **Hypothesis.** State the pattern as a precise, falsifiable claim about the world. This step is creative. 3. **Deductive prediction.** Derive a specific testable consequence of the hypothesis. If the hypothesis is true, this consequence must hold. 4. **Test.** Compare the prediction to new evidence. The new evidence either supports or contradicts the prediction. 5. **Conclusion.** If the prediction holds, the hypothesis survives this round. If the prediction fails, the hypothesis is rejected. The method does not make induction certain. It makes inductive hypotheses falsifiable. A hypothesis that survives many rigorous tests is more credible than one that has survived none. A hypothesis that fails any rigorous test is rejected. The process accumulates evidence without ever achieving certainty. For trading, the hypothetico-deductive method is the same machinery covered in earlier articles: state the rule, define a null hypothesis, run a test, get a p-value or confidence interval, decide. The framework treats every trading claim as a falsifiable hypothesis whose acceptance is provisional and revisable. ## Practical implications The point of recognizing induction's structure is not to abandon induction. There is no alternative. The point is to handle inductive conclusions with the appropriate level of confidence. Treat point estimates as midpoints of intervals. A backtested hit rate of 70% with 1000 observations is not "the strategy has a 70% hit rate." It is "the strategy had a 70% hit rate on this sample, and the true rate, conditional on the future resembling the past, sits somewhere between 67% and 73%." Build inductive arguments with multiple kinds of evidence. Cross-section across instruments, cross-period across decades, synthetic data via simulation. Each is an independent inductive argument. Conclusions supported by all of them are stronger than conclusions supported by one. Watch for regime shifts in live trading. The strongest inductive argument can be invalidated by a structural change. Position sizing and drawdown controls protect against the case where the underlying distribution has moved and the historical data no longer applies. Be ready to abandon hypotheses on evidence. The defining feature of a scientific (versus dogmatic) inductive belief is willingness to revise it when the data demands. A trader who refuses to update beliefs in light of contradicting evidence is no longer doing induction. They are doing decoration. --- ## KEY POINTS - Every claim a trader makes about the future is inductive. The trader observes past data, infers a generalization, and projects it forward. - Induction's conclusions are inherently uncertain because they extend beyond what the premises strictly justify. This is true even with perfect data and infinite computation. - The standard trading argument is induction by enumeration: "rule fired 1000 times, 700 went up, therefore 70% hit rate going forward." The premise is true. The conclusion is uncertain. - The 95% confidence interval on a 70% sample hit rate from 1000 observations is roughly \[67.2%, 72.8%\]. This captures sampling uncertainty under the assumption that the future is drawn from the same distribution as the past. - The deeper uncertainty (the assumption itself) is Hume's problem. The uniformity of nature cannot be proven without circular reasoning. Every scientific field accepts this. - Two factors govern inductive strength: quantity (more observations tighten the sampling CI) and quality (how representative the observations are of future cases). The CI captures quantity. Quality requires separate judgment about regime coverage. - Markets weaken induction more than physics does. Non-stationarity shifts the underlying distribution. Active participants change the process when patterns are published. One realized history limits quality evidence. - The hypothetico-deductive method handles induction by combining inductive observation with deductive, falsifiable tests. It does not make conclusions certain. It makes them revisable in light of evidence. - Practical response: treat point estimates as interval midpoints, use multiple kinds of evidence, watch for regime shifts, abandon hypotheses when data demands. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Evidence-Based Technical Analysis - David Aronson (Amazon)](https://www.amazon.com/s?k=Evidence-Based+Technical+Analysis+David+Aronson&ref=aligrithm.com) - [Systematic Trading - Robert Carver (Amazon)](https://www.amazon.com/s?k=Systematic+Trading+Robert+Carver&ref=aligrithm.com) ### 1.14 The Problem with One Sample of Market History URL: https://aligrithm.com/the-problem-with-one-sample-of-market-history/ Last updated: 2026-07-27T13:22:53.000Z # The previous article showed that one backtest is one sample drawn from a distribution of possible backtests. This article goes one level deeper. Even if a researcher could run an infinite number of backtests over all of recorded market history, the conclusions would still rest on one sample, because market history itself is one trajectory through possibility space. The market plays out once. There is no second universe in which the same starting conditions produced a different sequence of price moves for comparison. This is not a methodological issue that can be fixed with better technique. It is an epistemological limit on what any participant in a non-repeatable system can learn from observation alone. ## The two-layer sampling problem A backtest result sits inside two layers of uncertainty. The inner layer is sampling variability within history. Given the realized price sequence of SPX from 1990 to 2024, a strategy run on that sequence produces one point estimate of its true performance. Different windows of the same sequence produce different point estimates. The block bootstrap and the standard error formulas address this layer. The previous article was about this. The outer layer is the realized history itself. The SPX 1990 to 2024 is one possible path through the joint distribution of every variable that drives prices: monetary policy, technology adoption, demographic shifts, geopolitics, and the behavioral patterns of every trader who showed up. Under slightly different initial conditions or slightly different shocks, the realized sequence would look different. The realized history is one draw from a meta-distribution of possible histories. A simple decomposition makes the two layers visible: $$ \\hat{\\mu}\_{\\text{backtest}} = \\mu\_{\\text{regime}} + \\epsilon\_{\\text{sampling}} + \\delta\_{\\text{regime drift}} $$ Where μ̂\_backtest is the estimated edge from the backtest, μ\_regime is the true edge under the current data-generating regime, ε\_sampling is the inner-layer noise from finite N within history, and δ\_regime drift is the outer-layer noise from the fact that the realized history is one trajectory and the regime parameters themselves shift over time. Statistics can shrink ε\_sampling toward zero with more data and better methods. Statistics cannot shrink δ\_regime drift, because there is no second history to compare against. ## Why physics gets away with what trading cannot Physics solves the one-sample problem by repetition. The same experiment runs millions of times in the lab or in nature: every dropped object, every electron, every photon. The repeated trials are draws from the same underlying distribution. Sample sizes are practically infinite. Climate science sits in a harder position. There is one Earth. Climate scientists work around this by collecting many proxies (ice cores, tree rings, ocean sediment cores), looking at other planets, and building physical models that have to match the one trajectory the planet took. Their conclusions are weaker than particle physics conclusions for exactly this reason. Trading is even worse. There is one financial history, and that history is the output of active participants who see the data, respond to it, and change the system. Markets adapt. The data-generating process of 2024 is not the data-generating process of 1990\. A strategy that worked on the older data was operating in a different statistical world. This is the non-stationarity problem and it interacts with the one-sample problem in a vicious way. ## Non-stationarity makes it worse Standard statistical inference assumes the population stays the same while different samples are drawn from it. In markets, the population drifts. Volatility regimes change. Correlations shift. Liquidity providers come and go. Interest rate regimes flip. Crowding makes formerly profitable patterns disappear. When the population is non-stationary, a sample from one decade is not interchangeable with a sample from another decade. Pooling them increases nominal N but does not increase information about the current population. A model fit on 1980 to 2020 is fitting to multiple populations stitched together. The fit minimizes joint error across populations that no longer all exist. The result is that effective N is much smaller than nominal N. A 30-year backtest of an equities strategy might span only two or three relevant regimes. The other 27 years are information about markets that no longer exist. That is not 30 years of evidence about today's market. That is roughly three samples of three different markets, with no guarantee any of them still applies. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Visualizing the one-sample problem ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/05/ChatGPT-Image-May-20--2026--04_05_17-PM.png) The thick navy line is the only thing any trader has ever had access to. Every formula, every backtest, every published edge was estimated on that one line. The grey fan is the world the researcher would need to see to be confident about any inference. The fan is not available. ## What can be done despite the one-sample limit Five techniques partially compensate for the missing fan. None of them solve the problem. 1. **Cross-section as a sample multiplier.** Instead of one market over 30 years, study 30 markets over the same window. If a rule works on equities, fixed income, FX, and commodities simultaneously, the evidence is stronger than if it works on equities alone. The instruments are not independent samples, but their idiosyncratic noise is partly independent, so the joint test is more demanding than any single one. 2. **Cross-period replication.** Split history into non-overlapping windows large enough to span at least one economic cycle. Estimate the rule on each window independently. Stable results across windows are evidence that the rule does not depend on the one global trajectory. Unstable results are evidence that it does. 3. **Synthetic histories via simulation.** Build a stochastic model of returns that respects the documented properties of markets (fat tails, volatility clustering, autocorrelation, regime switching). Generate thousands of synthetic price paths from the model. Test the rule on each. The synthetic distribution of rule outcomes substitutes for the missing meta-distribution. The substitution is imperfect because the model is wrong (always wrong, in different ways). The exercise still produces a sharper estimate than the single historical path. 4. **Robust phenomena as priors.** Effects that show up across centuries, across asset classes, and across countries (the value premium, the momentum effect, the volatility risk premium, the term premium) have survived many implicit out-of-sample tests. They are not guaranteed to survive the next one, but they are stronger starting points than effects that have only been observed in one market over one decade. 5. **Position sizing for the larger uncertainty.** The math says the standard error on any backtested edge is wider than the standard formulas suggest, because nominal N overstates effective N when history is non-stationary. The honest response is smaller positions, broader diversification, and lower expected returns than the headline backtest suggests. ## The active-participant problem There is one more wrinkle that makes financial history different from any other one-sample problem. In physics, the experimenter does not influence the electron. In climate science, the climatologist does not influence the ice age. In markets, the participants see the published research, the backtests, and the strategies, and trade on them. Edges get crowded out. Patterns get arbitraged away. The act of discovering and publishing a pattern changes the data-generating process going forward. This means the one realized history is not a passive sample of a fixed underlying process. It is the output of a feedback loop where participants are constantly updating their behavior based on what they think the process is. The realized path of the system is correlated with the beliefs of the participants about the path of the system. The practical consequence is that historical evidence depreciates faster than statistical formulas suggest. The longer an edge has been known, the more it has been traded against, and the less reliable historical data on it becomes. The half-life of a published anomaly is finite and often short. ## The honest statement Every backtested edge is the edge that worked on the one history that happened. The single-sample problem does not make backtesting useless. It makes confident conclusions from any backtest, no matter how clean the methodology, dishonest. A trader who reports a backtested Sharpe of 1.5 with the implicit claim "this is the edge of the strategy" has overstated what was learned. The accurate version is: "this is the edge of the strategy on the one history available, with methodological uncertainty captured in the confidence interval and additional uncertainty from the one-history problem captured in the priors used to interpret the confidence interval." That sentence does not fit on a marketing slide. It is the correct sentence anyway. --- ## KEY POINTS - A backtest result sits inside two layers of uncertainty. Inner layer: sampling variability within history. Outer layer: history itself is one trajectory from a meta-distribution of possible histories. - The decomposition μ̂\_backtest = μ\_regime + ε\_sampling + δ\_regime drift makes the layers explicit. Statistics can shrink ε\_sampling. Statistics cannot shrink δ\_regime drift. - Physics handles the one-sample problem with repetition. Climate science handles it with proxies and physical models. Trading has neither, plus an active-participant feedback loop that changes the data-generating process while it runs. - Non-stationarity compounds the one-sample limit. A 30-year backtest typically spans only two or three relevant regimes. Effective N is much smaller than nominal N. - Five partial workarounds: cross-section across instruments, cross-period replication, synthetic histories via simulation, priors from effects that survive across centuries and asset classes, and position sizing that respects the larger true uncertainty. - Edges depreciate after publication. The history that contained the edge before it was known is not the same as the future history that will contain the edge after it is traded against. - None of these workarounds solve the problem. They tighten the inference relative to what one backtest alone provides. - The honest statement about any backtested edge is that it is the edge that worked on the one history available. Reporting it as the edge of the strategy is overstating what was learned. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Evidence-Based Technical Analysis - David Aronson (Amazon)](https://www.amazon.com/s?k=Evidence-Based+Technical+Analysis+David+Aronson&ref=aligrithm.com) - [Systematic Trading - Robert Carver (Amazon)](https://www.amazon.com/s?k=Systematic+Trading+Robert+Carver&ref=aligrithm.com) ### 1.13 Why One Backtest Tells You Almost Nothing URL: https://aligrithm.com/why-one-backtest-tells-you-almost-nothing/ Last updated: 2026-07-27T13:22:53.000Z # A backtest is a single draw from a distribution of possible outcomes. The number that comes out of it (annualized return, Sharpe ratio, win rate) is a point estimate with a confidence interval attached. The interval is wide enough that the point estimate, on its own, is almost uninformative. This is not an opinion about backtesting. It is a property of finite samples in noisy data. The math is the same math that governs polling, drug trials, and weather forecasting. The size of the standard error in trading data is the part that surprises most retail researchers when they see it for the first time. ## The single-number illusion A backtest produces one Sharpe, one annualized return, one max drawdown. Each is a sample statistic, not a population parameter. The trader who writes "this strategy has a Sharpe of 1.4" has stated the sample value as if it were the truth. The truth is "this strategy produced a Sharpe of 1.4 in this specific backtest sample, which is one of many possible samples." The distinction is not pedantic. The distance between the sample value and the true value depends on the standard error of the estimator, and the standard error in trading data is large enough that the sample value can sit far from the truth without anyone noticing. ## The standard error of the mean Every sample statistic has a standard error. For the mean return of a strategy, the standard error is: $$ \\text{SE}(\\bar{R}) = \\frac{\\sigma\_R}{\\sqrt{N}} $$ Where σ\_R is the standard deviation of the per-period returns and N is the number of independent return observations. The 95% confidence interval on the true mean is the sample mean plus or minus 1.96 standard errors. Worked example. A strategy generates daily returns over four years (about 1000 trading days). The sample mean daily return is 0.05% (about 12.6% annualized at 252 days per year). The sample standard deviation of daily returns is 1.0%. The standard error of the mean is: SE = 1.0% / √1000 ≈ 0.0316% The 95% confidence interval on the true daily mean is: 0.05% ± 1.96 × 0.0316% = \[−0.012%, +0.112%\] Annualized, the 95% interval on the true mean return is approximately: \[−3.0%, +28.2%\] The point estimate is 12.6% per year. The interval says the truth could be a 3% per year loss or a 28% per year gain. Both numbers are consistent with what the backtest produced. This is what "almost nothing" means quantitatively. The headline number is real. The uncertainty band attached to it is wider than most decisions can tolerate. ## Sharpe ratio: same problem, slightly tighter The Sharpe ratio is more stable than the raw mean, but its standard error is still large at typical sample sizes. A reasonable approximation for the standard error of the annualized Sharpe with N independent daily observations is: $$ \\text{SE}(\\text{Sharpe}\_{\\text{ann}}) \\approx \\sqrt{\\frac{1 + \\frac{\\text{Sharpe}\_{\\text{daily}}^{2}}{2}}{N}} \\cdot \\sqrt{252} $$ With a sample annualized Sharpe of 1.0 (daily Sharpe ≈ 0.063) and N = 1000 daily observations: SE(Sharpe\_ann) ≈ √((1 + 0.002)/1000) × √252 ≈ 0.0316 × 15.87 ≈ 0.50 The 95% CI on the true annualized Sharpe is approximately: 1.0 ± 1.96 × 0.50 = \[0.02, 1.98\] A backtest Sharpe of 1.0 is consistent with a true Sharpe close to zero or close to two. The point estimate by itself does not distinguish these worlds. One of them is a strategy worth running. The other is a strategy that will refuse to deliver in live trading. ## The window-shift experiment Take any published rule. Run it on 2000 to 2010\. Note the result. Now run it on 2000.5 to 2010.5 (shift the window by six months). Note the new result. The two results almost never match. The Sharpe might move from 0.9 to 1.4 or from 1.2 to 0.6\. The annualized return might shift by 30 percentage points in either direction. The maximum drawdown might double or halve. The rule did not change. The data source did not change. The parameter set did not change. The only thing that changed was the slice of history that happened to be sampled. This sensitivity is not a flaw in the rule. It is a property of the data. Market returns have high noise relative to mean, and any sample of finite length lands somewhere on the sampling distribution of possible outcomes. Different windows land in different places. The window-shift experiment is the simplest demonstration that one backtest result is unstable. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Visualizing the sampling distribution ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/05/ChatGPT-Image-May-20--2026--03_56_00-PM.png) The bold curve is what gets posted on social media. The fan is what would happen if the same rule could be run on alternative draws of the same underlying process. The fan is the truth about what one backtest tells the researcher. ## The block bootstrap solution The window-shift experiment is informative but limited. It only gives a few samples. The block bootstrap gives thousands. The procedure. Take the daily returns of the backtest. Cut them into blocks of length L (typically 5 to 20 days, long enough to preserve autocorrelation and volatility clustering). Resample the blocks with replacement until the new return series has the same length as the original. Run the strategy metric (mean, Sharpe, drawdown) on the resampled series. Repeat 5000 times. The resulting 5000 metric values are an empirical estimate of the sampling distribution. Take the 2.5th and 97.5th percentiles to get a 95% confidence interval. Compare this interval to the headline backtest number. The gap between the point estimate and the interval edges is the size of "almost nothing." For most retail rules the gap is enormous. For a few rules it is small enough to support a decision. Those are the rules worth running. ## Why the interval is so wide Three factors drive the width. 1. **Return-to-noise ratio.** The daily Sharpe of a good strategy is around 0.05 to 0.10\. The daily volatility dwarfs the daily mean. Distinguishing signal from noise requires many observations. 2. **Sample size.** A four-year daily backtest has roughly 1000 observations. That is small by statistical standards. Polling firms reject sample sizes that small for far less noisy questions. 3. **Autocorrelation and regime structure.** Market returns are not independent draws. Volatility clusters, regimes persist, and effective sample size is smaller than nominal sample size. Five years of daily data has nominal N = 1260 but effective N closer to 250 to 500 once dependence is accounted for. The standard error formula assumes independence, so the naive interval is already optimistic. The combination of these three factors makes single-backtest evidence weak in a way that is mathematical, not behavioral. ## What this changes operationally Stop reporting backtest results as single numbers. Always report a confidence interval. "Sharpe 1.2 with 95% CI \[0.4, 2.0\]" is honest. "Sharpe 1.2" is misleading. Stop comparing strategies on point estimates. Compare them on intervals. Strategy A with Sharpe 1.0 and CI \[0.6, 1.4\] is more credible than Strategy B with Sharpe 1.4 and CI \[−0.1, 2.9\], even though B's point estimate is higher. Stop concluding from a single backtest. The unit of inference is the sampling distribution, not the sample. Use longer histories where the rule is plausibly stationary, or use higher-frequency data where more observations exist per unit calendar time. Both shrink the standard error. Walk-forward testing across multiple non-overlapping windows generates multiple samples and exposes the spread between them, which is the same information from a different angle. --- ## KEY POINTS - A backtest is one realization from a distribution of possible outcomes. The output statistics (mean return, Sharpe, drawdown) are sample estimates with standard errors, not population truths. - The standard error of the mean is SE = σ / √N. For typical strategies with daily data and a few years of history, the 95% confidence interval on annualized return spans tens of percent. - A backtest showing 12.6% annualized return on 1000 trading days with σ\_daily = 1% has a 95% CI of approximately \[−3%, +28%\] on the true annualized mean. The point estimate carries little information by itself. - The Sharpe ratio is more stable than the raw mean but still has a large standard error. A sample annualized Sharpe of 1.0 on N = 1000 daily observations is consistent with a true Sharpe close to zero or close to two. - Shifting the test window by six months on the same rule and the same data source produces a substantially different result. This sensitivity is a property of noisy data, not a flaw in the rule. - The block bootstrap reconstructs the sampling distribution of any backtest metric. Resample blocks of the original returns with replacement, recompute the metric thousands of times, and take percentiles for a 95% interval. - Three factors widen the interval: low return-to-noise ratio in markets, small effective sample sizes in typical backtests, and autocorrelation that further reduces effective N. - Always report confidence intervals next to point estimates. Compare strategies on intervals, not on headline numbers. The unit of inference is the sampling distribution, not the sample. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Evidence-Based Technical Analysis - David Aronson (Amazon)](https://www.amazon.com/s?k=Evidence-Based+Technical+Analysis+David+Aronson&ref=aligrithm.com) - [Systematic Trading - Robert Carver (Amazon)](https://www.amazon.com/s?k=Systematic+Trading+Robert+Carver&ref=aligrithm.com) ### 1.12 Backtesting Is an Experiment, Not a Screenshot URL: https://aligrithm.com/backtesting-is-an-experiment-not-a-screenshot/ Last updated: 2026-07-27T13:22:52.000Z # A backtest is data, not a picture. The equity curve that comes out of a strategy run on historical prices is one observation drawn from a distribution of possible outcomes. The retail mindset stares at the curve, finds it beautiful, and ships the strategy. The experimental mindset treats the curve as a single sample and asks what would have happened under a thousand variations of the same world. The two approaches look identical up to the moment money is risked. After that they diverge violently. The reason most live strategies disappoint is not that backtesting is broken. It is that most people do not run backtests as experiments. They run them as demonstrations. ## What an experiment requires A real experiment has six properties. Drop any of them and the result is decoration. 1. **A pre-committed hypothesis.** The rule, the parameter set, the universe of instruments, and the test window are written down before the test runs. No retrofitting after seeing the result. 2. **A null hypothesis.** What would the result look like if the rule had no predictive power? Without a null, "the rule made 14% per year" is a number with no reference frame. 3. **Controlled inputs.** No lookahead, no survivorship bias, no in-sample peeking, no parameters tuned on the same data used to evaluate. The test environment is as close to live conditions as the data allows. 4. **A test statistic.** One number that summarizes the outcome. Mean return per trade, Sharpe ratio, mean forward return after a signal. The same number is computed on the actual backtest and on every simulated null replication. 5. **A sampling distribution.** Some procedure that generates the distribution of the test statistic under the null. The procedure can be analytical (formulas) or computational (permutation, bootstrap, Monte Carlo). The sampling distribution is the reference frame that turns one number into evidence. 6. **A decision rule.** A p-value threshold or a confidence-interval criterion committed to before the test. Crossing the threshold is the rule for keeping the strategy. Failing to cross it is the rule for discarding it. A backtest that fails any of these six checks is a screenshot. The equity curve might be true, but it is not evidence. ## The two-component decomposition Every backtest return decomposes into two parts. $$ R\_{\\text{backtest}} = R\_{\\text{predictive}} + ( \\beta\_{\\text{bias}} \\cdot \\mu\_{\\text{trend}} ) $$ Where R\_backtest is the mean return per day observed in the test, R\_predictive is the rule's true predictive contribution (positive, zero, or negative), β\_bias is the rule's long/short bias (+1 = always long, −1 = always short, 0 = perfectly balanced), and μ\_trend is the average daily return of the market during the test window. The second term is free return from drift. It has nothing to do with the rule. A strategy that holds long 90 percent of the time during a period when the market drifted up at 0.05 percent per day collects 0.8 × 0.05% = 0.04% per day from drift alone. Over 252 trading days that compounds to roughly 10 percent per year of completely unearned return. Worked example. Take a rule with zero predictive power applied to SPX from 2009 to 2021. - The SPX had a positive net trend over the period. Average daily return ≈ 0.05%. - The rule, by construction, is long 90% of the time and short 10%. β\_bias = 0.8. - Unwanted contribution = 0.8 × 0.05% = 0.04% per day. - Annualized unwanted contribution ≈ 10%. - The backtest equity curve looks great. Up 10% per year. Smooth. - The rule has zero predictive power. Every cent of that 10% is drift × long bias. In live trading this rule earns whatever the market drifts, minus slippage and costs. In a flat or down regime it loses money. The backtest never told the trader anything about the rule. It told the trader about the market. The screenshot view sees +10% per year and ships the strategy. The experimental view detrends the data, recomputes the result, finds 0%, and discards the rule. ## Detrending: removing the unwanted component The fix is mechanical. Subtract the market's mean return from each day's return before running the rule on the data. The detrended series has zero net trend. Any positive return the rule produces on the detrended series is predictive power, not drift exposure. $$ R^{\\text{detrended}}\_t = R\_t - \\bar{R}\_{\\text{market}} $$ Where R\_t is the day's return and the bar denotes the sample mean over the full backtest window. Run the rule on R\_detrended instead of R. Whatever performance comes out is the part that depends on the rule's signal timing, not on the wind being at its back. This single adjustment kills more rules than any other validation step. It also keeps the rules that survive, because surviving on detrended data is a much harder bar. ## One backtest is one sample Even after detrending, a single equity curve is one observation drawn from a distribution. The same rule applied to a slightly different sample of the same market (different start date, different end date, different universe, different time of day) produces a different curve. The variation across those possible curves is the sampling variability, and it is what hypothesis testing measures. The question "did the rule work?" needs a precise form. The precise form is: how often would chance alone produce a backtest at least this good? To answer that, generate many fake versions of the world in which the rule has no predictive power. Permute the returns. Bootstrap them. Simulate paths with the same volatility and autocorrelation but no exploitable structure. Apply the rule to each fake version. Count what fraction of the fakes produce a result as extreme as the real one. $$ p = \\frac{ \\#\\{R^{\*}\_i \\geq R\_{\\text{observed}}\\} }{N} $$ Where R\_observed is the rule's actual detrended mean return, R\*\_i is the detrended mean return under the i-th permuted dataset, and N is the number of permutations. If p is below the threshold (0.05 is the convention), the actual result is unlikely under the null and the rule earns the benefit of the doubt. If p is above the threshold, the result is consistent with luck. A backtest without this number is incomplete. A backtest with this number is an experiment. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Visualizing the screenshot vs the experiment ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/05/ChatGPT-Image-May-20--2026--03_44_22-PM.png) The screenshot view sees one curve and reads it as evidence. The experimental view sees one curve in the context of what chance alone would produce and reads the comparison as evidence. ## The experimental backtest checklist A backtest qualifies as an experiment when every item below is true. 1. The rule, parameters, universe, and test window are written down before the test runs. 2. The data is point-in-time accurate. No future information leaks into past decisions. 3. The market series is detrended, or returns are computed relative to a benchmark. 4. A test statistic is defined in advance (mean return, Sharpe, hit rate, profit factor). 5. A null model is defined in advance and is realistic (preserves volatility clustering, autocorrelation, and fat tails of the original series). 6. The null model is simulated enough times to estimate a p-value or confidence interval with low Monte Carlo noise. Five thousand replications is the floor. 7. The p-value or confidence interval is reported next to the headline performance metric. 8. The rejection threshold was committed to before the test ran. 9. If multiple rules or parameter combinations were tested, a multiple-testing correction is applied (Bonferroni, FDR, or a Reality Check style bootstrap). 10. The full pipeline is reproducible from raw data to final p-value given the code, the data, and the random seed. A backtest that ticks all ten boxes can fail in live trading. A backtest that ticks fewer than ten almost certainly will. ## What changes when this mindset is adopted The number of rules that look good goes up. The number of rules that survive testing goes down. Most of the rules that get killed are the ones a trader would have most enjoyed running, because they were the ones with the highest screenshot equity curves and the lowest underlying predictive power. The strategies that survive are unglamorous. Their backtested returns are smaller than the strategies that get discarded. Their confidence intervals are tight. Their p-values sit below the threshold. They make money in regimes where the rejected strategies would have collapsed. The discipline is uncomfortable. The reward is that the strategies that ship behave in live trading the way they behaved in the backtest. --- ## KEY POINTS - A backtest is data, not a picture. The equity curve is one sample drawn from a distribution of possible outcomes under the rule. - A real experiment requires six properties: pre-committed hypothesis, null hypothesis, controlled inputs, defined test statistic, sampling distribution, and decision rule. - Every backtest return decomposes into a predictive component and an unwanted component (long/short bias × market net trend). The unwanted component can produce double-digit annual returns from drift alone with zero predictive power. - Detrending the market series before running the rule removes the unwanted component. Performance on detrended data is the part attributable to the rule's signal timing. - One equity curve cannot prove anything. Validity requires comparing the result to the distribution of outcomes under a realistic null model. - The p-value is the fraction of null simulations that produce a result at least as good as the observed backtest. A backtest without a p-value or confidence interval is incomplete. - The ten-item experimental backtest checklist (pre-committed parameters, PIT data, detrending, defined test statistic, realistic null, sufficient simulations, reported uncertainty, pre-committed threshold, multiple-testing correction, reproducibility) is the minimum viable protocol. - Adopting the experimental mindset kills many strategies that look good. The ones that survive are the only ones worth running. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Evidence-Based Technical Analysis - David Aronson (Amazon)](https://www.amazon.com/s?k=Evidence-Based+Technical+Analysis+David+Aronson&ref=aligrithm.com) - [Systematic Trading - Robert Carver (Amazon)](https://www.amazon.com/s?k=Systematic+Trading+Robert+Carver&ref=aligrithm.com) ### 1.11 Technical Analysis as a Scientific Hypothesis URL: https://aligrithm.com/technical-analysis-as-a-scientific-hypothesis/ Last updated: 2026-07-27T13:22:52.000Z # A trading rule is a hypothesis. The claim "buying when the 200-day moving average is crossed from below produces positive 7-day forward returns" is a falsifiable statement about market behavior. The market either supports it or refutes it. The trader's job is to set up the test correctly and let the data speak. Most retail technical analysis fails to be scientific because the practitioners do not test the patterns as hypotheses. A chart with an arrow drawn on it is a claim. Without a defined test statistic, a null model, and a rejection threshold, the claim is unfalsifiable. Unfalsifiable claims sit outside the domain of evidence. They are vibes. ## Three properties of a hypothesis A scientific hypothesis has three properties. It is specific. "Markets trend" is not a hypothesis. "The mean 5-day forward return after a positive 50-day return on the SPX from 1990 to 2024 is greater than zero" is a hypothesis. The first cannot be tested. The second can. It is falsifiable. There must exist an observable result that would make the hypothesis false. For a trading rule, the observable result is the backtest performance. If the backtest performance is statistically indistinguishable from what the same rule would produce on data with no underlying structure, the hypothesis is rejected. It has a quantitative prediction. The rule predicts something measurable: mean return per signal, Sharpe ratio, or hit rate above some threshold. The prediction is what you test. A rule that lacks any one of these three properties is a description, not a hypothesis. Descriptions sit outside the domain where evidence applies. ## There is only one hypothesis test Most introductory statistics courses teach hypothesis testing as a zoo of named tests. T-test for means, chi-squared for proportions, F-test for variances, ANOVA for groups, Kolmogorov-Smirnov for distributions. Each test comes with its own table, its own assumptions, its own degrees of freedom rules. The framework underneath is the same. The single framework, written out: $$ p = P( \\delta\_{\\text{simulated}} \\geq \\delta\_{\\text{observed}} \\mid H\_0 ) $$ In plain words. δ\_observed is a number computed from your actual data, the test statistic. It might be the rule's mean return, its Sharpe ratio, or any other property you can compute on a backtest. δ\_simulated is the same number computed on a synthetic dataset generated under the null hypothesis. H₀ is the model of the world where the rule has no edge. The p-value is the probability that random data alone produces a test statistic at least as extreme as the one you saw. Five steps make this concrete. 1. Compute the test statistic on the real data. Get one number, δ\_observed. 2. Define a null model. The model has to be capable of generating fake datasets that look similar in structure to the real one but contain no signal. For a TA rule, the standard null model is the rule applied to a randomly shuffled or bootstrap-resampled version of the original return series. 3. Generate many fake datasets from the null model. A few thousand is enough. 4. Compute the same test statistic on each fake dataset. You now have a distribution of test statistics under H₀. 5. Count the fraction of fake test statistics that are at least as extreme as δ\_observed. That fraction is the p-value. If the p-value is below your threshold (the standard threshold is 0.05), the data is hard to explain under the null model, and you reject the null. If the p-value is above the threshold, you cannot reject the null, which means you cannot claim the rule has edge. Every named statistical test is a special case of this procedure with a particular choice of test statistic, a particular choice of null model, and an analytical shortcut for computing the p-value. The shortcuts were necessary when computers were slow. They are not necessary now. Direct simulation of the null is more flexible, more transparent, and forces you to specify what you believe about the world instead of inheriting it from a textbook. ## Visualizing the test ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/05/ChatGPT-Image-May-20--2026--02_00_43-PM.png) The visual shows the entire framework in one image. The bell shape is the distribution of what the test statistic looks like when the rule has no edge. The vertical line is what your rule produced. The shaded tail is the p-value. The dashed line is the threshold you committed to before you ran the test. ## Applying it to a real TA rule Take a concrete rule. "Long the SPX for 7 days after the 50-day moving average crosses above the 200-day moving average." Run it from 1990 to 2024. Step 1\. Compute the test statistic. The mean 7-day forward return after the cross signal is +0.42% across all signals in the historical sample. That is δ\_observed. Step 2\. Define the null model. The null hypothesis is that the cross signal carries no predictive information. To generate fake datasets, take the historical SPX returns and shuffle them randomly to break the time structure. The shuffled return series has the same distribution as the real one (same mean, same volatility, same fat tails) but no temporal pattern. Apply the same MA cross rule to the shuffled series. Step 3\. Generate 5,000 shuffled return series and apply the rule to each. For each, compute the mean 7-day forward return after a signal. Step 4\. Plot the 5,000 mean returns. You have an empirical distribution of what the rule produces under the null. The distribution is centered near zero, with some spread due to the random shuffling. Step 5\. Count the fraction of those 5,000 means that are ≥ +0.42%. Suppose 380 of them are. The p-value is 380 / 5000 = 0.076. That p-value is above 0.05\. The rule's observed performance sits inside what the null model can produce by chance. You cannot reject the null. The "edge" you saw might be real, but the data does not support that claim at the standard threshold. The rule is not falsified, but it is not validated either. This is the result that most retail-shared TA rules produce when tested with the right protocol. The pattern looks good on a chart. The performance looks good on a backtest. The hypothesis test says "this might be luck." ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Operational consequences Three operational consequences follow from treating a rule as a hypothesis. The hypothesis is stated before the test. You write down the rule, the test statistic, the null model, and the rejection threshold in a file. Then you run the test. The file is timestamped. You do not get to retrofit any of those choices after seeing the result. The null model is explicit. The shape of the null model affects the result. A naive null where returns are independent identically distributed normals will reject more rules than a realistic null that preserves the autocorrelation and volatility clustering of the original series. Choosing the more realistic null is harder and produces fewer "wins." It also produces fewer false positives in live trading. The result has a confidence interval, not a point estimate. The same simulation procedure that generates the p-value also generates confidence intervals around the test statistic. A backtest mean return of +0.42% with a 95% confidence interval of \[−0.15%, +1.0%\] means something different from the same mean with an interval of \[+0.30%, +0.55%\]. The point estimate is the same in both, but the first is uncertain and the second is decisive. Report both. ## The catch: data mining bias The procedure above tests one specific, pre-committed rule. It does not protect against testing many rules and picking the best. If you test 100 rules, each at a 5% significance level, around 5 will appear "significant" by chance alone, even when none of them have edge. The p-value of the best rule in a 100-rule competition is meaningless without a multiple-testing correction. The proper procedure is to either lower the threshold (Bonferroni correction divides α by the number of tests) or simulate the entire multiple-testing process (the bootstrap-reality-check family of methods). Most retail "I tested this rule and it works" claims come from undisclosed parameter sweeps that test thousands of combinations. The reported p-value is the p-value of the best rule, not the p-value of the procedure that produced the best rule. The two differ by orders of magnitude. --- ## KEY POINTS - A trading rule is a hypothesis when it is specific, falsifiable, and produces a quantitative prediction. Anything less is a description. - All hypothesis tests reduce to one framework: compute a test statistic on real data, simulate the null model many times, count what fraction of simulations produce a test statistic at least as extreme as observed. - The framework is p = P(δ\_simulated ≥ δ\_observed | H₀). Every named test (t-test, chi-squared, ANOVA) is a special case with a particular test statistic and a particular null model. - For trading rules, the standard null model is the rule applied to a randomly shuffled or bootstrap-resampled version of the return series. Shuffling breaks the time structure while preserving the marginal distribution. - The rule, test statistic, null model, and rejection threshold must be written down before the test runs. Retrofitting any of them invalidates the result. - Report confidence intervals alongside p-values. A point estimate without an interval is a non-answer. - The single-test framework protects against luck in one pre-committed test. It does not protect against testing many rules and picking the best. Multiple-testing correction is mandatory when more than one rule is tested. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Evidence-Based Technical Analysis - David Aronson (Amazon)](https://www.amazon.com/s?k=Evidence-Based+Technical+Analysis+David+Aronson&ref=aligrithm.com) - [Systematic Trading - Robert Carver (Amazon)](https://www.amazon.com/s?k=Systematic+Trading+Robert+Carver&ref=aligrithm.com) ### 1.10 Why Good Trading Feels Boring URL: https://aligrithm.com/why-good-trading-feels-boring/ Last updated: 2026-07-27T13:22:51.000Z # A trader who is excited about their day is doing something wrong. The dopamine is coming from somewhere that is not the edge. The edge is small, the variance per trade is small relative to the account, and the daily change in equity is small enough to forget about. If your trading day produces emotional spikes, the position sizes are too big, the strategy frequency is too high, the discretionary overrides are too frequent, or all three at once. Good trading is boring by design. The boredom is the signature of a right-sized, automated, low-discretion system. Excitement is the warning sign that the trader is paying for entertainment with risk-adjusted returns. ## The three features that make trading addictive Three properties make any activity behave like a slot machine: the illusion of control, frequent near misses, and rapid continuous stimulation. Each one shows up in trading, and each one produces the wrong incentives. The illusion of control. Picking your own setup, drawing your own trend line, calling your own entry feels like skill. The feeling is a manufactured byproduct of being the one who clicked the button. The real skill is in the rule, the sizing, and the kill criterion. The click is the smallest part of the chain. Frequent near misses. A trade that almost worked is more emotionally engaging than a trade that worked outright or that lost outright. The "would have been a big winner if I had held two more bars" feeling is what keeps gamblers at the table. In trading, it is what keeps traders second-guessing their stops and adjusting their parameters in the wrong direction. Rapid continuous stimulation. Day trading with one-minute charts produces a steady stream of micro-events. Each event is a small dopamine hit. The brain learns to seek the stream. The stream does not produce edge. It produces a behavioral addiction that looks like work. Systematic trading removes all three. Pre-computed signals remove the illusion of control over each click. Volatility-targeted sizing flattens near misses into normal noise. Daily or longer evaluation removes the rapid stimulation. The trader keeps the edge and loses the entertainment. ## Boredom in practice A correctly run systematic strategy on a retail account, sized for a 10% annual volatility target on $100,000 of capital, produces daily P&L with a standard deviation of: $$ \\sigma\_{\\text{daily}} = \\frac{ \\sigma\_{\\text{annual}} \\cdot \\text{Capital} }{ \\sqrt{252} } $$ In plain words: annual volatility (10% in this case, set by the trader) times account capital (the dollar amount at risk) gives the dollar standard deviation per year. Divide by √252 (the number of trading days in a year) to get the standard deviation per day, because variance scales with time and standard deviation scales with the square root of time. For the numbers above: σ\_daily = (0.10 × $100,000) / √252 ≈ $630. That is the expected daily move. Most days will see equity change by less than $630\. Big days might see $1,200 or $1,500\. A bad day might be −$1,200\. The numbers are small enough that the daily change does not justify watching the screen. A trader running the same $100,000 with discretionary overrides, no volatility targeting, and one-minute scalping might see daily P&L with standard deviation of $4,000 to $6,000\. The dopamine difference is the source of the appeal. The risk-adjusted return difference, after costs and behavioral errors, is in favor of the boring version. ## Boring equity curves vs exciting equity curves ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/05/ChatGPT-Image-May-20--2026--01_47_15-PM.png) The visual makes the asymmetry concrete. The boring curve and the exciting curve can end at similar levels. They are not similar products. The boring curve is reproducible across multiple traders running the same rules. The exciting curve is one path out of many possible paths, most of which end below $100,000. ## The dopamine substitution problem Removing excitement from trading creates a vacuum. The trader who used to get a hit from each click now has nothing to click. The brain looks for the missing stimulation. If the trader does not plan for this, the dopamine seeking returns through the back door. The standard failure mode is "research" that is entertainment in disguise. The trader who removed discretionary trading from their day spends six hours a week reading market commentary, watching trader podcasts, and tweaking parameters that should not be tweaked. The activity feels productive. It produces no edge. It is the same dopamine search the trader was running before, with a more sophisticated label. Two ways to handle the substitution. The first is to channel the energy into research that has a measurable output. Build a new system, test a new instrument, run a permutation test on an existing rule. The research is bounded by an objective (find a signal, validate a hypothesis, retire a dead system) and produces an artifact (a notebook, a test result, a kill decision). The activity scratches the curiosity itch and adds to the trader's pipeline. The second is to accept that trading does not have to fill the day. The boring trader spends 15 minutes a morning checking that systems ran correctly, batches the day's orders, and walks away. The other 7 hours are for something else. Not every job benefits from being a full-time obsession. Trading is one of the jobs that does not. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## The career arc difference The exciting trader's career arc looks dramatic in screenshots. Quick rise and steeper fall, posts during the rise and silence during the fall, then a return three years later with a new system and the same psychology. The screenshots circulate. The longitudinal P&L does not. The boring trader's career arc looks tedious in screenshots and good in account statements. Mid-teens annual returns, single-digit drawdowns, same routine for ten years. The compound effect produces real wealth at a pace that does not photograph well. The difference is psychological architecture, not skill at picking trades. The exciting trader optimized for the experience of trading. The boring trader optimized for the output of trading. The output is what compounds. ## Designing for boredom Boredom is the output of deliberate engineering. The operational moves. Pre-compute signals before the market opens. The trader executes what the system picked, not what the trader felt like trading. Batch orders to the broker once per day or once per session. Continuous order flow invites discretionary intervention; batched flow blocks it. Size every position to a fixed fraction of the account, scaled by the instrument's volatility. Position size is a function of the rule's output and the volatility model, not a per-trade decision. Remove the live equity feed during the trading session. The dollar value of unrealized P&L is irrelevant to any decision the trader needs to make. Watching it produces emotional engagement without informational gain. Set a fixed weekly review window. Equity, drawdown, and decay metrics get reviewed once a week, with the trader away from the screen the rest of the week. The review is structured, the rest is not. The set of moves above looks like restraint. It is also what produces the kind of trading that lasts thirty years instead of three. --- ## KEY POINTS - Excitement in trading is a warning sign. The dopamine is coming from oversized positions, high frequency, or discretionary overrides, not from the edge. - Three addictive features (illusion of control, frequent near misses, rapid continuous stimulation) drive the experience of bad trading. Systematic processes remove all three. - A correctly sized 10% annual volatility target on $100,000 produces daily P&L noise of around $630\. The number is too small to be entertaining. - Boring equity curves and exciting equity curves can end at similar levels. The boring curve is reproducible. The exciting curve is one survivorship-biased path out of many. - Removing the dopamine from execution creates a vacuum. Channel the freed energy into bounded research that produces artifacts, or into something other than trading. - The exciting trader optimized for the experience. The boring trader optimized for the output. Output compounds. Experience does not. - Engineer boredom directly: pre-compute signals, batch orders, fix position size to a volatility model, hide the live equity feed, fix the review cadence. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Evidence-Based Technical Analysis - David Aronson (Amazon)](https://www.amazon.com/s?k=Evidence-Based+Technical+Analysis+David+Aronson&ref=aligrithm.com) - [Systematic Trading - Robert Carver (Amazon)](https://www.amazon.com/s?k=Systematic+Trading+Robert+Carver&ref=aligrithm.com) ### 1.9 Why Trading Is a Probability Business, Not a Certainty Business URL: https://aligrithm.com/why-trading-is-a-probability-business-not-a-certainty-business/ Last updated: 2026-07-27T13:23:05.000Z # Most retail trading pain comes from treating each trade as a verdict on the system. The trade closes at a loss, the trader concludes "the system is wrong," and starts adjusting. The trade closes at a profit, the trader concludes "I am right," and increases size. Both conclusions are statistical errors. A single trade is one sample from a distribution. The distribution decides whether the system works. The single sample carries little of the information needed to update on the distribution. A professional treats every trade as a draw from a known distribution with known properties and a known tolerance for variance. The trade outcome adds one data point to the running estimate of the distribution. The next decision depends on the distribution, not on the last draw. This is the mental switch that separates a trader from a gambler. The math is the easy part. The discipline is the slow part. ## The certainty mindset The amateur mindset treats each trade as a binary event with a binary outcome. "This signal is strong, I expect this to be a winner." "That trade lost, the system is broken." "Three losers in a row, I'm sitting out the next signal." "Four winners in a row, I am in good form, time to scale up." Each of these sentences is a category error. The signal is not "strong" or "weak" except in the sense that it shifts the conditional probability of an outcome by a measurable amount. The system is not "broken" because one trade closed in the red. Three losers in a row is what a 55% win rate strategy produces about 9% of the time. Four winners in a row is what the same strategy produces about 9% of the time. Neither carries information about the next trade. The certainty mindset produces a predictable behavior: the trader adjusts position size, parameters, or rules based on the most recent few trades. The adjustments are responses to noise. The result is that the trader manufactures their own underperformance by interpreting normal variance as signal. ## The probability mindset The probability mindset starts with a different model of what a trade is. A trade is a random draw from a distribution. The distribution has a mean (expected return per trade) and a spread (the standard deviation of trade returns). Both are properties of the rule plus the market, estimated from a large sample of past trades. The single trade outcome is one observation from this distribution. It carries no more information about the next trade than one coin flip carries about the next coin flip, given that you already know the coin is biased 55/45. The relevant decisions for the trader are not "did this trade work" or "should I take the next signal." The relevant decisions are: 1. Is the distribution still the same one I estimated? (Has the rule decayed?) 2. Is my position sizing correct for the variance of that distribution? 3. Is my account survival probability acceptable across the next 1,000 draws? None of these are about any single trade. All of them are about properties of the underlying distribution and the trader's exposure to it. ## The math of why one trade tells you nothing The expected value of a single trade is the average across the distribution. $$ E\[X\] = p \\cdot W - (1-p) \\cdot L $$ In plain words: the win probability times the average win, minus the loss probability times the average loss. For a system with a 55% win rate, an average win of $120, and an average loss of $100: $$ E\[X\] = 0.55 \\times 120 - 0.45 \\times 100 $$ $$ = 66 - 45 $$ $$ = \\$21 \\text{ per trade} $$ Now the noise. The standard deviation of trade returns σ for this system works out to around $110 per trade, given the spread of wins and losses around the mean. The single trade is drawn from a distribution centered at +$21 with a width of $110\. The next trade can come in anywhere from −$200 to +$240 and still be a normal draw. The single-trade outcome is dominated by noise, not by the edge. The edge emerges as you average across many trades. The standard error of the mean shrinks with the square root of the sample size. $$ SE(\\bar{X}) = \\frac{\\sigma}{\\sqrt{N}} $$ In plain words: the uncertainty in your estimate of the true mean shrinks as you collect more samples, but only as the square root of the sample count. To halve the uncertainty, you need four times the samples. To get the uncertainty down to a useful level, you need a lot of samples. Three sample sizes for the same system. After 1 trade: standard error is $110\. The mean of $21 is buried in noise of $110\. You know nothing about the system from one trade. After 25 trades: standard error is $110 / √25 = $22\. The mean of $21 sits at the boundary of one standard error from zero. The system might be profitable, might not. You still know almost nothing. After 200 trades: standard error is $110 / √200 = $7.8\. The mean of $21 sits at about 2.7 standard errors above zero. Now you can say with reasonable confidence that the edge is real and positive. This is the gap between certainty and probability. After 1 trade, the trader who fires the "system doesn't work" decision is acting on noise that is 5x the signal. After 200 trades, the trader has earned the right to update their view. ## Subscribe to Aligrithm Subscribe Email sent! Check your inbox to complete your signup. Evidence-based quant trading research. New essays straight to your inbox. ## Visualizing the noise ![](https://storage.ghost.io/c/27/cb/27cb0fc8-2c77-4434-af9e-d5d32a916994/content/images/2026/05/ChatGPT-Image-May-20--2026--01_28_47-PM.png) The visual makes the noise concrete. The single-trade outcome can land anywhere across that distribution. The mean of 25 trades is also noisy, often by 50% or more in either direction. Only at 200+ trades does the running mean stabilize close to the true edge. ## The behavioral implications Once you accept that a trade is a draw, several common retail behaviors look wrong. Adjusting parameters after a losing trade. The losing trade was a draw from the distribution. The distribution did not change because the draw was negative. Adjusting parameters fits the rule to the most recent few draws, which is fitting to noise. Sitting out a signal after a losing streak. The next signal is a fresh draw. The fact that the last three draws were below the mean does not change the distribution of the next draw. Skipping the next signal removes one positive-expectancy event from your sample. Done enough times, this is the dominant source of underperformance versus the backtest. Scaling up size after a winning streak. The next signal is still drawn from the same distribution. The fact that the last four draws were above the mean does not change the distribution. Scaling up size at the top of a normal positive run sets you up to take an oversized loss on the next normal negative run. Judging a month or a quarter by its result. Twenty trading days at 1 trade per day is 20 draws. Twenty draws has a standard error of σ/√20 ≈ 0.22σ. For a system with an edge of 0.1σ per trade, the monthly result will be negative around 30% of the time even though the edge is real. Quarterly results are tighter but still dominated by noise. The shorter the evaluation window, the more the result is noise. The longer the window, the more the result is signal. Most retail traders evaluate on windows that are too short to be informative, then react to the noise as if it were signal, then wonder why the live results diverge from the backtest. ## Operational implications The probability mindset translates into specific rules of thumb. Do not change anything in the system based on fewer than 50 trades of new data. Below 50, the standard error of your estimate is too large for any change to be justified. Do not increase position size after winning streaks or decrease after losing streaks. Position size should be a function of the distribution's properties, not the recent draw history. Do not interpret a drawdown as evidence the system is broken until the drawdown is outside the 95th percentile of historical drawdowns for the system. Below that threshold, the drawdown is a normal sample. Above it, you have evidence to investigate. Do report performance with confidence intervals, not point estimates. A backtest Sharpe of 1.2 with a 95% confidence interval of \[0.6, 1.8\] means something different from a backtest Sharpe of 1.2 with an interval of \[1.1, 1.3\]. The first is "the system might work." The second is "the system works." Do think about every trade decision as expected value plus variance, not as a guess about direction. The signal carries predictive content. The single outcome carries close to none. --- ## KEY POINTS - A trade is a draw from a distribution. The distribution decides whether the system works. The single draw carries almost none of the information about the distribution. - The expected value of a trade is E\[X\] = p·W − (1−p)·L. The standard deviation σ of trade returns runs 5x to 10x the absolute value of E\[X\] for most retail systems. - The standard error of the mean shrinks as σ/√N. To halve uncertainty, you need four times the trades. - One trade contains no information. 25 trades contain little. 200 trades contain enough to update on the system's edge. - Adjusting parameters after a few losing trades, sitting out signals after streaks, and scaling up after wins are all responses to noise treated as signal. - Monthly and quarterly returns for retail systems are dominated by noise. Do not interpret a negative month as a broken system unless the drawdown exceeds the 95th percentile of the system's historical drawdown distribution. - Report performance with confidence intervals, not point estimates. A point Sharpe with no interval is a non-answer dressed up as one. - Treat every trade as expected value plus variance. The signal carries predictive content. The single outcome carries close to none. --- **A note on AI.* The ideas, research, analysis, and conclusions in this article are my own. I use AI tools to help with editing and wordsmithing, because English is not my first language, and I am not shy about that. AI-generated ideas and AI-assisted writing are not the same thing: the first is empty slop from a generic prompt, the second is a tool for communicating years of real research more clearly. Judge the work by its substance, not by whether software helped polish the prose.* ## References - [Evidence-Based Technical Analysis - David Aronson (Amazon)](https://www.amazon.com/s?k=Evidence-Based+Technical+Analysis+David+Aronson&ref=aligrithm.com) - [Systematic Trading - Robert Carver (Amazon)](https://www.amazon.com/s?k=Systematic+Trading+Robert+Carver&ref=aligrithm.com) - [Comparing Discretionary and Systematic Hedge Fund Performance](https://papers.ssrn.com/sol3/papers.cfm?abstract%5Fid=2880641&ref=aligrithm.com) - [Systematic Funds Outperform Discretionary Funds](https://papers.ssrn.com/sol3/Delivery.cfm/SSRN%5FID1516020%5Fcode1210072.pdf?abstractid=1516020&ref=aligrithm.com) - [Systematic Hedge Fund Risk Management: An Overview](https://papers.ssrn.com/sol3/Delivery.cfm/SSRN%5FID3438382%5Fcode1054906.pdf?abstractid=3431610&mirid=1&type=2&ref=aligrithm.com) - [Performance, Risk and Persistence of the CTA Industry: Systematic vs Discretionary Trading](https://papers.ssrn.com/sol3/Delivery.cfm/SSRN%5FID2085714%5Fcode1853882.pdf?abstractid=2081937&mirid=1&ref=aligrithm.com) - [A Hybrid Systematic-Discretionary Channel Breakout Framework for Nasdaq E-mini Futures](https://papers.ssrn.com/sol3/Delivery.cfm/6469419.pdf?abstractid=6469419&mirid=1&ref=aligrithm.com) - [Dissecting Retail Investor Trading Tendencies](https://papers.ssrn.com/sol3/Delivery.cfm/5168527.pdf?abstractid=5168527&mirid=1&type=2&ref=aligrithm.com) - [A Comparative Analysis of Quantitative vs. Discretionary Strategies in Equity Markets](https://papers.ssrn.com/sol3/Delivery.cfm/5356167.pdf?abstractid=5356167&mirid=1&ref=aligrithm.com) - [Can Retail Investors Beat the Market by Using Technical Trading Rules?](https://aaltodoc.aalto.fi/bitstreams/5403f3a4-616d-4518-a788-09f2391bf4d5/download?ref=aligrithm.com) ## References - [Evidence-Based Technical Analysis - David Aronson (Amazon)](https://www.amazon.com/s?k=Evidence-Based+Technical+Analysis+David+Aronson&ref=aligrithm.com) - [Systematic Trading - Robert Carver (Amazon)](https://www.amazon.com/s?k=Systematic+Trading+Robert+Carver&ref=aligrithm.com) _Truncated after 5 MiB. Use `/sitemap.xml` for the complete archive of public content._