Predictability of asset returns

Predictability is arguably the most fundamental question in empirical finance. If asset returns were completely unpredictable, factor investing would simply have no interest and would in fact not exist. Conversely, if returns were too easy to forecast, arbitrage forces would erase any such patterns. In practice, the truth lies somewhere between these two extremes, and the literature has spent decades (with no convergence in sight by construction, thankfully!) debating the extent, sources, and stability of return predictability.

The topic splits along two dimensions: the horizon at which predictions are made (short- versus long-run) and the scope (time-series versus cross-section). In time-series predictability, the question is whether (aggregate) market returns (e.g., the equity premium) can be forecasted. In cross-sectional predictability, the question is whether the relative performance of individual stocks or portfolios can be anticipated using characteristics. This short chapter focuses primarily on the cross-sectional dimension (due to its link to factors), which is where ML tools have had the greatest practical impact, while drawing connections to the time-series literature wherever this might be illuminating.

A recurring theme throughout this chapter is the distinction between in-sample (IS) and out-of-sample (OOS) performance. A predictor that fits past data well but fails on new observations offers little value for investors (in ML parlance, we would say that it does not generalize well). As Goyal & Welch (2008) famously showed, many variables that appear predictive in-sample lose their forecasting power when evaluated out of sample. This pattern is sometimes attributed to data-mining, structural breaks, or overfit models. Subsequent work by Campbell & Thompson (2008) offered a more nuanced view, arguing that constrained forecasting models can deliver modest but reliable OOS gains.

Beyond average performance, there is growing evidence that predictability is time-varying. Farmer et al. (2023) introduce the notion of pockets of predictability: subperiods during which out-of-sample $R^2$ is significantly positive, interspersed with stretches of near-zero or even negative predictability. This chapter develops and illustrates this idea in detail, using rolling OOS $R^2$ as a key diagnostic. The replication study Cakici et al. (2025) offers a critical take and underscores the methodological care required when assessing such pockets - but the original argument still holds: predictability is time-varying in nature.

The chapter is organised as follows. Section Section 10.1 introduces the core metrics used to quantify predictability — OOS $R^2$, information coefficient, and hit ratio. Section Section 10.2 reviews the time-series literature on equity premium forecasting. Section Section 10.3 covers cross-sectional predictability, focusing on how firm characteristics relate to expected returns. Section Section 10.4 is the methodological heart of the chapter: it implements the rolling OOS $R^2$ framework and illustrates the dynamic nature of predictability on our dataset. Sections Section 10.5 and Section 10.6 cover two specialised but increasingly prominent sub-topics: predicting anomaly returns and forecasting company fundamentals.

10.1Metrics for predictability¶

10.1.1Out-of-sample $R^2$¶

The out-of-sample $R^2$ (OOS $R^2$) is the most natural way to assess whether a model’s predictions beat a simple benchmark — typically the historical average return $\bar{r}$. Formally, for a set of $T$ OOS predictions $\hat{r}_t$ compared to realizations $r_t$:

where $\bar{r}_t$ is the historical mean computed recursively using only information available up to $t-1$. A positive OOS $R^2$ means the model reduces mean squared prediction error relative to the mean benchmark; a negative OOS $R^2$ means it performs worse. In the cross-sectional context, the benchmark is typically the cross-sectional mean return at each date: stocks are ranked and the naive predictor assigns the same expected return to all stocks. Sometimes, it is possible to take $\bar{r}=0$ in the denominator so that we simply compare the model fit to a zero prediction (the most agnostic one).

One practical subtlety is that OOS $R^2$ must be evaluated on a forecast made before the outcome is known, using a model estimated without any look-ahead. This requires an expanding or rolling training window. This point may seem obvious but that Cakici et al. (2025) show is sometimes violated even in prominent academic work.

10.1.2Information coefficient¶

In the cross-sectional context, a popular alternative to OOS $R^2$ is the information coefficient (IC). At each date $t$, the IC is the rank correlation between predicted and realized returns across the $N_t$ stocks in the universe:

where $\hat{r}_{t,n}$ and $r_{t,n}$ are the predicted and realized return of stock $n$ at date $t$. The use of ranks (rather than levels) makes the IC robust to outliers in return distributions, which are ubiquitous in equity data. A mean IC of around 3–5% is generally considered economically meaningful in practice, and a t-statistic on IC is a standard test for cross-sectional predictive skill (see Qu et al. (2023) for formal tests comparing forecasting performance in cross-sections).

A related concept is the information ratio (IR), defined as the ratio of mean IC to the standard deviation of IC across time: $IR = \bar{IC}/\sigma_{IC}$. Whereas $\bar{IC}$ measures average predictive accuracy, $IR$ penalises for inconsistency. A model may have a moderate average IC but a very high IR if that IC is reliably positive month after month, a property which more valuable to most investors.

10.1.3Hit ratio¶

The hit ratio is the simplest of all metrics: the fraction of periods (or stocks) for which the sign of the prediction agrees with the sign of the realization. A hit ratio above 50% indicates better-than-random directional accuracy. Its appeal is transparency; its limitation is that it ignores the magnitude of errors. A model that correctly identifies every large positive return as positive but also mislabels every large negative return as positive may have a good hit ratio while being catastrophically wrong on the most important observations. Note however, that due to transaction costs, hit ratios below 52% usually do not translate to profitable strategies (more on this in Chapter \ref{chap_backtest}).

10.2Time-series predictability of the equity premium¶

10.2.1An open question¶

The literature on time-series predictability of aggregate equity returns is vast and contentious. Early studies documented that variables such as dividend yields, price-earnings ratios, the term spread, and default spreads help predict future market returns at horizons of one to several years. But Goyal & Welch (2008) conducted an exhaustive audit of these predictors and reached a pessimistic conclusion: despite strong in-sample evidence, virtually none of the proposed variables reliably predict the equity premium out of sample. The gap between IS and OOS performance points to over-fitting, parameter instability, or both.

Campbell & Thompson (2008) offered some kind of a partial rehabilitation (almost synchronously). They show that imposing an economically motivated constraint — that the equity premium must be non-negative — significantly improves OOS performance. The intuition is that unconstrained regression forecasts often assign negative values to the equity premium, which is a priori implausible. Constraining the forecast to be non-negative reduces variance at the cost of a modest increase in bias, yielding a better bias-variance tradeoff in finite samples.

Rapach et al. (2010) demonstrate that forecast combination improves OOS equity premium forecasts substantially. Combining predictions from many individual forecasting models — even when each model has poor standalone OOS performance — yields a composite that outperforms the historical average benchmark. This finding echoes the ensemble methods discussed in Chapter Ensemble models, and highlights that the question is not only which model to use, but how to aggregate information from multiple sources.

Lin et al. (2025) extend this theme by decomposing market returns into components (dividends, earnings growth, and repricing) and showing that different predictors contribute most effectively to forecasting different components. Combining component-specific forecasts yields a composite that outperforms either component or aggregate-level approaches.

10.2.2Non-stationarity and regime dependence¶

A recurring message in the time-series literature is that predictability is not a fixed property of markets but varies across economic regimes. Farmer et al. (2023) formalise this by showing that OOS $R^2$ is heterogeneous across time: the equity premium appears forecastable in some subperiods and not in others — precisely the pockets of predictability that we will illustrate in Section Section 10.4. This finding connects naturally to the broader theme of Chapter Two key concepts: causality and non-stationarity: structural breaks and non-stationarity are not merely statistical nuisances but carry economic information about changing risk premia and investor beliefs.

10.3Cross-sectional predictability¶

10.3.1The landscape¶

Cross-sectional return predictability — the ability to sort stocks by expected return using observed characteristics — is the empirical foundation of Chapter Factor investing and asset pricing anomalies. Here we focus less on which characteristics predict returns and more on the statistical properties and limits of such predictions.

Gu et al. (2020) provide the most comprehensive ML-based assessment of cross-sectional predictability. Using a universe of US stocks from 1957 to 2016, they find that neural networks and gradient-boosted trees substantially outperform linear models in terms of both OOS $R^2$ and portfolio performance. Their reported monthly OOS $R^2$ is modest in absolute terms (around 0.4%) but corresponds to economically large risk-adjusted portfolio returns. The small magnitude of $R^2$ is not surprising: monthly stock returns are dominated by idiosyncratic noise, and any predictable component represents a small fraction of total variance.

Kozak et al. (2020) show that cross-sectional predictability can be understood through the lens of a stochastic discount factor spanned by a small number of factors. Their shrinkage approach regularises the pricing of many anomalies simultaneously, capturing the bulk of cross-sectional variation with a parsimonious model.

Cong et al. (2026) introduce the metaphor of mosaics of predictability — the idea that no single model, feature, or subperiod dominates, but that assembling the pieces yields a coherent picture. Their analysis emphasises the heterogeneity of predictability across the cross-section (some stocks are more predictable than others) and over time (some periods offer richer signals than others). This view complements Farmer et al. (2023) on the time-series dimension.

10.3.2The tails of the return distribution¶

Most predictability research focuses on the mean return. Uribe et al. (2025) argue that important information is contained in the tails of the cross-sectional return distribution. Extreme outcomes — the best and worst performing stocks — may be driven by different mechanisms than the average stock, and their distributional features (variance, skewness, kurtosis) can themselves be predicted by firm characteristics. This links to a broader literature on distribution regression and quantile prediction in financial settings.

10.3.3Stability and the decline of anomalies¶

Even where predictability exists, it tends to erode after publication. The classic result is that anomaly returns shrink substantially once the strategy is publicly known (see McLean & Pontiff (2016) discussed in Chapter Factor investing and asset pricing anomalies). Bowles et al. (2024) take a second-order approach: rather than predicting raw returns, they ask whether the return of a known anomaly strategy can itself be predicted. Their evidence suggests that, with the right set of instruments, one can anticipate which anomalies will do well in the near future — adding a meta-layer of predictability on top of the first-order signals.

10.4Pockets of predictability¶

10.4.1The core idea¶

The concept of pockets of predictability captures a simple but important observation: average OOS $R^2$ can conceal rich time variation. Imagine a model that achieves an overall OOS $R^2$ of zero. This could mean the model is uniformly uninformative. It could also mean there are periods of strong positive predictability — pockets — offset by equally strong negative predictability. An investor who could identify the pockets in advance would find the model invaluable; an investor who cannot would be no better off than using the historical mean.

The formal approach of Farmer et al. (2023) is based on a kernel smoother applied to the sequence of rolling squared prediction errors. By varying the bandwidth of the kernel, one can uncover predictability at different time scales. Importantly, their approach requires a strict out-of-sample setup: at each forecast date, only past information may be used. Cakici et al. (2025) point out that the original code by Farmer et al. (2023) inadvertently used a two-sided kernel, allowing future data to inform the forecast — a look-ahead bias that, once corrected, substantially weakens the pocket evidence.

We adopt a simpler but equally transparent approach. We train a ridge regression on an expanding window of observations (all past data up to date $t-1$), forecast cross-sectional returns at date $t$, and compute a local OOS $R^2$ over a rolling window of $W$ months. Periods when this rolling OOS $R^2$ is positive constitute pockets of predictability.

10.4.2Implementation¶

We begin by importing the required libraries and loading the data.

The expanding-window loop below trains a ridge regression at each date using all prior observations and records the cross-sectional OOS $R^2$ and IC for that month. We use only features_short to keep the computation tractable, and we require a minimum training window of 24 months.

The summary statistics already reveal an important pattern: the mean monthly OOS $R^2$ across all periods is close to zero (sometimes slightly positive, sometimes slightly negative depending on the regularisation strength), while the standard deviation is substantial. This spread is precisely what motivates the pocket analysis: individual months can be far more (or far less) predictable than the aggregate picture suggests.

We now compute the rolling OOS $R^2$ over a window of $W$ months. This smoothed series highlights clusters of predictable and unpredictable regimes.

Figure 10.1 reveals that predictability is far from uniform over time. There are clear pockets where the rolling OOS $R^2$ is sustainedly positive, and stretches where the model underperforms the naive cross-sectional mean. Several observations are worth noting:

• The pockets do not coincide trivially with known market regimes (bull/bear markets), suggesting that predictability has a complex relationship with economic conditions.

• The transitions between predictable and unpredictable periods can be sharp, consistent with the structural-break interpretation in Farmer et al. (2023).

• The width of the window $W$ matters: a short window ($W=12$) reveals more erratic patterns; a long window ($W=48$) smooths out the signal but may miss shorter pockets.

This figure is the central empirical contribution of the chapter. It encapsulates the message that predictability should be thought of as a dynamic property of financial markets rather than a fixed parameter to be estimated.

10.4.3The information coefficient through time¶

The IC provides a complementary view by focusing on the rank ordering of returns rather than their magnitudes. A positive IC means the model successfully identifies which stocks will outperform; a negative IC means it systematically ranks them in the wrong order.

Figure 10.2 is best read alongside Figure 10.1. The IC fluctuates substantially from month to month, with persistent clusters of positive and negative values. The t-statistic on mean IC quantifies whether cross-sectional predictability is statistically significant on average. Periods with mean IC well above 0.05 correspond closely (though imperfectly) to the pockets identified through OOS $R^2$.

The fraction of months with positive IC is itself a useful diagnostic. Even a mean IC close to zero can deliver positive portfolio performance if the IC is positive in high-volatility months (when sorting errors are most costly) and negative in low-volatility months (when errors matter less). This observation underpins the IC-Volatility adjusted measures used in the systematic investing industry.

10.4.4Feature-level predictability¶

Predictability is not just heterogeneous over time: it also varies across features. Some characteristics (e.g., momentum) may be reliably predictive in trending markets but fare poorly in mean-reverting regimes. Others (e.g., value signals) may exhibit the opposite pattern. Decomposing the aggregate IC into feature-level contributions illuminates which parts of the signal are driving the pockets.

Below we compute, for each feature and each year, the average monthly IC of a univariate ridge regression using that feature alone.

Figure 10.3 makes the heterogeneity of predictability vivid. Several observations emerge:

• No feature dominates uniformly. Even the most predictive feature in aggregate has years of strongly negative IC. This corroborates the mosaic metaphor of Cong et al. (2026): assembling pieces from multiple sources yields a more stable signal than relying on any single characteristic.

• Momentum features (e.g., Mom_LT) tend to cluster: periods of positive IC for momentum features coincide, consistent with the idea that momentum payoffs are driven by common macroeconomic or sentiment factors.

• Value and quality signals (e.g., PB, Ocf) sometimes exhibit IC patterns that are out of phase with momentum, offering diversification potential at the feature level.

• The pockets visible in Figure 10.1 can often be traced to synchronised positive IC across multiple features. Conversely, the unpredictable stretches correspond to a mixture of negative and positive IC values that cancel when aggregated.

This analysis points to an important lesson for practitioners: evaluating features on the basis of their average IC alone is insufficient. One should examine the time-profile of IC and, ideally, condition on macro regimes. Pesaran et al. (2024) formalise this within a panel forecasting framework, distinguishing between parameter heterogeneity (coefficients differ across units/periods) and estimation uncertainty (limited data inflates the variance of estimated coefficients), and show that the relative importance of these two sources varies substantially with horizon and sample size.

10.4.5Cumulative OOS $R^2$ and investor value¶

A complementary visualisation is the cumulative OOS $R^2$, which shows whether the model accumulates forecasting advantage over the full sample or whether periods of positive predictability are offset by negative ones.

The cumulative chart is instructive in a different way from the rolling chart. If the cumulative advantage is monotonically increasing, the model is consistently better than the naive benchmark. If it oscillates around zero, gains and losses cancel. If it trends downward towards the end, the model has recently deteriorated. This picture is especially relevant for risk management: an investor who deployed the model would have accumulated gains in some periods, but also experienced drawdowns in others. The model is not free — its value depends critically on the investor’s ability to stay the course during unpredictable stretches.

10.5Predicting anomaly returns¶

Most of the predictability literature takes individual stock returns as the quantity to be forecast. An alternative and increasingly influential approach is to forecast the performance of strategies — i.e., the returns of long-short portfolios constructed on the basis of known anomalies. This “meta-prediction” framework asks: given that we know a particular anomaly (value, momentum, profitability...) has worked historically, can we predict when it will work in the future?

Bowles et al. (2024) tackle this question systematically. They use a rich set of macro and market variables — including uncertainty indicators, sentiment measures, and technical signals — to forecast the returns of 93 equity anomalies. They find that anomaly returns are predictable to a meaningful degree: a strategy that overweights currently-predictable anomalies achieves a substantial Sharpe ratio improvement over naive equal-weighting.

This work connects naturally to the concept of dynamic factor allocation: rather than holding all factors at constant weights, tilting towards those most likely to deliver excess returns in the near future. The challenge, of course, is that predicting predictability is itself subject to overfitting — and requires a rigorous OOS evaluation protocol of the kind advocated throughout this chapter.

Singha (2025) push the idea further by investigating regime-conditional cross-sectional predictability. They document that the cross-sectional factor structure itself evolves through time via drift regimes, and that recognising these regimes unlocks predictability that is invisible to unconditional models. Their findings align with the drift-regime framework Farmer et al. (2023) use for time-series pockets, suggesting that regime-awareness is a unifying theme in modern predictability research.

The Wang et al. (2025) paper contributes a useful prior step: measuring the inherent forecastability of a time series before any modelling is attempted. Using spectral entropy and Lyapunov exponents, they provide model-free measures of how predictable a given series is in principle. Applied to financial returns, such metrics can help triage which assets or anomalies are worth the modelling effort.

10.6Forecasting company fundamentals¶

A complementary approach to predicting returns directly is to forecast the fundamental quantities that drive valuations — earnings, cash flows, profit margins — and then derive return predictions from those forecasts through a valuation model. This two-stage approach has several potential advantages:

1. Interpretability: A model that first forecasts earnings and then derives a price target is more transparent than one that maps characteristics directly to returns.

2. Stability: Fundamental quantities may be more stationary than returns themselves, making them easier to model and reducing the risk of spurious patterns.

3. Linkage to the real economy: Macro forecasts, analyst estimates, and industry dynamics can be integrated more naturally into a fundamentals model.

Divo et al. (2025) investigate the use of modern ML tools — including gradient-boosted trees and attention-based architectures — to forecast earnings per share, return on equity, and revenue growth. They find substantial improvements over linear benchmarks, particularly for larger and more data-rich firms, and show that the forecastability of fundamentals is itself heterogeneous across industries and time.

Cao & You (2024) develop a comprehensive ML-based fundamental analysis framework. By training models on a broad set of accounting ratios and firm characteristics, they show that ML-based fundamental signals significantly outperform traditional analyst scores and accounting-based valuation ratios for predicting future returns. Their study is also careful to use an OOS evaluation protocol and to test robustness to transaction costs.

The forecasting literature on fundamentals has its own version of the pockets problem. Pesaran et al. (2024) formalise this in a panel setting: when individual firms’ parameters are heterogeneous (different firms respond differently to the same accounting signal), pooling all firms together in a single model introduces misspecification bias. They derive optimal forecasts that balance estimation efficiency (from pooling) against parameter heterogeneity (from keeping firm-specific models), providing a principled framework for practitioners deciding between pooled and firm-specific approaches.

A different decomposition is offered by Lin et al. (2025), who work at the aggregate level. By separating market returns into dividend yield, earnings growth, and repricing components, they show that the predictable portion of the equity premium is concentrated in the repricing component — consistent with the variance-decomposition insight of Campbell & Thompson (2008) and the canonical results of Goyal & Welch (2008) on predictability concentrating in valuation-ratio driven components. Combining models tailored to each component yields OOS gains that are larger and more stable than those from a single aggregate predictor.

The cross-sectional analogue of return decomposition is quantile prediction. Rather than forecasting the mean return for each stock, one can target the distribution of returns, asking which stocks are more likely to appear in the top or bottom decile. Uribe et al. (2025) pursue exactly this approach, using distributional regression to explain the tails of the cross-sectional return distribution with firm characteristics. Their results suggest that the tails are more predictable than the center, which has direct implications for tail-risk strategies and skewness tilts in portfolio construction.

10.7Comparing predictability across models¶

A practical challenge is how to statistically compare the OOS performance of two competing forecasters. Classical tests of equal predictive accuracy — such as the Diebold-Mariano test — assume stationarity and may be poorly sized when predictions are generated from models estimated on overlapping samples. Qu et al. (2023) develop tests specifically designed for cross-sectional forecasting competitions, accounting for the fact that at each date the two models generate $N_t$ forecast-error pairs rather than a single pair. Their framework allows for heterogeneous and correlated errors across stocks, making it robust to the clustering of errors within industries or size groups.

This inferential framework is important because it guards against over-interpreting OOS differences. With a large cross-section, even tiny improvements in MSE are highly significant under conventional tests. Qu et al. (2023) provide corrections that yield better calibrated size while maintaining power, enabling more honest assessments of whether model improvements are genuinely exploitable.

The question of what makes a predictor consistently better is also explored in Cong et al. (2026) through the lens of mosaic assembly: combining a large number of individually weak predictors — each associated with a specific data type, horizon, or stock segment — into a mosaic that captures the full complexity of the cross-sectional return distribution. Their empirical results show that the mosaic approach dominates any single predictor and that the gains are not confined to a single pocket but persist across time, suggesting genuine diversification of the predictability sources.

10.8Conclusion¶

Predictability in financial markets is neither a binary phenomenon (present or absent) nor a fixed parameter (constant over time). The evidence, both theoretical and empirical, points to a richer picture:

• Pockets of predictability are real: OOS $R^2$ and IC exhibit significant time variation, with stretches of genuine predictive accuracy interspersed with flat or negative performance. The rolling analyses in this chapter make this concrete on real equity data.

• Feature heterogeneity matters: some signals work in some regimes and fail in others. A robust investment process monitors feature-level IC over time and reweights signals accordingly, rather than relying on average performance computed over the entire backtest.

• Strict OOS discipline is non-negotiable. The cautionary tale of Cakici et al. (2025) — where a two-sided kernel allowed future information to contaminate forecasts — underscores that even small procedural lapses can create an illusion of predictability. All the results in this chapter are produced with a strict expanding-window protocol.

• Fundamentals and higher moments extend the predictability frontier. Predicting company fundamentals (Divo et al. (2025), Cao & You (2024)) and the tails of the return distribution (Uribe et al. (2025)) opens additional dimensions of predictability that the mean-return framework misses.

These conclusions have a direct bearing on portfolio construction (Chapter Portfolio backtesting): a strategy whose signal has time-varying predictability should dynamically adjust its confidence and hence its position sizing. Periods of low or negative rolling OOS $R^2$ warrant risk reduction; periods of strong pockets warrant full deployment. This adaptive approach is, in a sense, the portfolio-management implementation of the statistical insight that predictability comes in pockets.

10.9Exercises¶

1. Window sensitivity: Recompute Figure 10.1 using rolling windows of 12, 36, and 48 months. How do the identified pockets change? Which window seems most useful for a practitioner who rebalances monthly?

2. Model comparison: Replace the ridge regression in the expanding-window loop with (a) a simple OLS regression and (b) an XGBoost model. Plot all three rolling OOS $R^2$ series on the same chart. Are the pockets model-specific or do they coincide across methods?

3. Single-feature pockets: Using the feature-level IC heatmap, identify the feature with the most persistent positive IC. Now compute its standalone rolling OOS $R^2$ and compare to the full-feature model. Does adding more features help or hurt during unpredictable periods?

4. Hit ratio through time: Compute the monthly hit ratio alongside the monthly OOS $R^2$ and IC. Do the three metrics agree on when the pockets occur? Compute their pairwise Pearson and Spearman correlations across months.

5. Forecast combination: At each forecast date, generate predictions from three univariate models (one per feature: EPS, Mom_LT, PB). Compute a simple equal-weight combination of the three predictions and compare its rolling OOS $R^2$ to each individual model. Does combination reduce the depth of the negative (red) pockets?