Design uncertainty and nonstandard errors

Every empirical result is the endpoint of a long chain of decisions. Which firms enter the sample? How are missing values handled? Where is the sorting threshold placed? How are portfolio positions weighted? None of these questions has a single defensible answer. Each one is a fork, and each fork multiplies the number of studies that could have been written from the same data.

The previous chapter treated uncertainty that comes from the data. This chapter treats uncertainty that comes from the analyst. The two are distinct. Sampling noise shrinks as observations accumulate, while the dispersion caused by methodological choices does not. A larger dataset does not tell you where to place a quantile cutoff.

We first formalize a research protocol as a sequence of choices, following Coqueret & Pérignon (2026). We then define the nonstandard error, introduced by Menkveld et al. (2024), and show how it can be separated from the standard error within a single variance decomposition. We close with a worked example on the value premium, where three simple choices generate eighteen defensible estimates of the same quantity.

17.1The garden of forking paths¶

Gelman & Loken (2014) describe the garden of forking paths: a researcher who analyzes a dataset makes many decisions that are contingent on the data, and the resulting result is one draw from a space of results that were never reported. The problem is not dishonesty. An analyst who runs a single specification and reports it faithfully still hides the variability that the unreported specifications would have revealed.

Two experimental designs have been used to map this space. In a multi analyst study, independent teams receive the same data and the same question. Silberzahn et al. (2018) gave 29 teams one dataset and asked whether soccer referees give more red cards to dark skinned players. The estimated effects ranged from no effect to a large one. Botvinik-Nezer et al. (2020) repeated the exercise on a neuroimaging dataset with 70 teams and found that no two teams used the same analysis pipeline. Huntington-Klein et al. (2021) document the same dispersion in applied microeconomics.

In a multi path study, a single team enumerates the choices explicitly and runs all of the combinations. Simonsohn et al. (2020) call the resulting object a specification curve. In finance, Mitton (2022) spans 1,024 paths in corporate finance, Soebhag et al. (2024) and Walter et al. (2024) do so for portfolio sorts, Fieberg et al. (2024) for cryptocurrencies, and Battistella et al. (2026) and Cakici et al. (2026) across asset classes. The example in this chapter is a small multi path study.

17.1.1A protocol as a sequence of choices¶

Following Coqueret & Pérignon (2026), we model an empirical study as a composition of operations applied to a dataset $\mathbb{D}$. If the protocol has $J$ steps, the reported result is

where $\hat{b}$ is the object of interest. It may be an average return, a $t$ statistic, or an alpha. Each step $f_j$ offers $r_j \ge 2$ defensible options, so the number of distinct paths is

The multiplicative form is what makes this a practical concern. Seven binary or ternary choices already generate several hundred paths. Coqueret & Pérignon (2026) use seven forks and obtain $P = 576$; Walter et al. (2024) use fourteen and obtain 69,120. Since each path yields its own estimate, we index outcomes by the path and write $\hat{b}_p$ for $p = 1, \dots, P$.

Two properties of this notation matter. First, $\hat{b}_p$ is random because $\mathbb{D}$ is random, so each path has a sampling distribution. Second, two paths that share many steps will produce similar answers on the same data. Both properties reappear below.

17.2Standard and nonstandard errors¶

Menkveld et al. (2024) coined the term nonstandard error for the dispersion in results that is attributable to the choices of the analyst rather than to the data. Their multi analyst study of market microstructure found these errors to be sizable, and comparable in magnitude to the standard errors that researchers habitually report.

The literature measures the nonstandard error either as the interquartile range of outcomes across paths (Menkveld et al. (2024), Walter et al. (2024)) or as their cross sectional standard deviation (Soebhag et al. (2024), Fieberg et al. (2024)). The standard error is measured separately, usually by bootstrapping the returns of each path after the paths have been run. The two quantities are therefore built on different reference objects and do not combine into anything.

Coqueret & Pérignon (2026) close this gap. Suppose the analyst runs every path on several samples, so that each outcome $\hat{b}_p(\mathbb{D})$ is indexed by both a path $p$ and a sample $\mathbb{D}$. The law of total variance (see chapter 9 of Blitzstein & Hwang (2019)) can then be applied twice to the same quantity $\sigma_b^2 = \mathbb{V}[\hat{b}_p(\mathbb{D})]$. Conditioning on the sample gives

where the second term is the variance across paths within a sample, averaged over samples. That is the nonstandard error of the literature, with the refinement that it is averaged rather than computed on a single dataset. Conditioning on the path instead gives

where the second term is the variance across samples within a path, averaged over paths. That is the standard error of the literature.

Both identities are valid and neither is preferable, so Coqueret & Pérignon (2026) take one half of each pair:

Adding 17.3 and 17.4 and dividing by two shows that these definitions satisfy

This is the appeal of the construction. The two errors are now shares of one budget, so the statement that methodological uncertainty dominates sampling uncertainty becomes a statement about a decomposition rather than a comparison of two unrelated numbers. We verify 17.7 numerically below.

17.3Why correlation across paths matters¶

Having many estimates is only useful if they carry independent information. Let $\hat{\mu}_b = P^{-1}\sum_p \hat{b}_p$ be the average outcome across paths. Its variance is not $\sigma_b^2 / P$, because the outcomes are not independent:

Combining 17.8 with 17.7 gives the canonical expression

Three quantities therefore drive the width of any confidence interval around the mean effect: the standard error, the nonstandard error, and the average correlation across paths. If the correlations were all zero, the last term would collapse to $1/P$ and adding paths would sharpen inference. If the correlations are close to one, the term stays close to one and the $P$ paths deliver no more information than a single one.

It may seem paradoxical that agreement across paths is undesirable. The resolution is that each path is a separate model learning from the data. Ideally all of them reach the same conclusion, but for different reasons. Agreement that emerges from different perspectives is evidence of robustness. Agreement that emerges because the paths are looking at the same rows of the same table is not. The same intuition underlies bagging and random forests, where the benefit of an ensemble comes from decorrelating its members (Breiman (2001)), a point developed in Chapter Ensemble models.

This suggests an operational fix. Analysts normally run every path on one fixed dataset, which we call common resampling when the exercise is repeated on perturbed versions of that dataset. Coqueret & Pérignon (2026) instead recommend path specific resampling: draw a fresh sample immediately before running each path. The paths then no longer share their sampling noise, the correlations collapse toward zero, and the sum in 17.9 falls to roughly $1/P$. In their study of 107 anomalies, the average correlation falls from about 0.17 to essentially zero, which narrows the confidence interval for the mean by a factor of about five.

17.4The nonstandard Sharpe ratio¶

If the nonstandard error measures how much an estimate moves when the protocol moves, then a return should be judged against it. Coqueret & Pérignon (2026) define the nonstandard Sharpe ratio as the mean outcome scaled by the nonstandard error:

where $r_p$ is the average return of the portfolio built along path $p$ and the $\omega_p$ are weights that sum to one. Uniform weights $\omega_p = 1/P$ treat all paths as equally defensible. Unequal weights let the analyst favor the conventional specification and downweight exotic ones.

When each path is run once and correlations can be assumed away, the nonstandard error is simply the dispersion of outcomes across paths and the ratio becomes

The interpretation parallels the Sharpe ratio, with cross path dispersion replacing time series volatility. A factor with a large average return can still have a low nonstandard Sharpe ratio if its performance depends on how it is built. Up to a scaling constant, the NSR is the $t$ statistic for testing whether the mean across paths differs from zero, so it doubles as a decision rule. Coqueret & Pérignon (2026) call an anomaly persistent when it survives both sampling noise and methodological variation, and they find that persistence is determined almost entirely by the NSR, with a threshold near one. Of the 107 anomalies they examine, 26 are persistent under every configuration they test.

17.5A worked example: the value premium¶

Asness & Frazzini (2013) is the natural illustration. Their title, the devil in HML’s details, refers to the fact that the value factor of Fama & French (1993) is defined with annually updated book values matched to lagged prices, and that seemingly innocuous departures from that convention change the measured premium substantially. Value is a good test case precisely because the underlying idea, that cheap stocks outperform expensive ones, is simple and uncontested. Whatever dispersion we find comes from implementation, not from disagreement about the hypothesis.

We keep three forks, each with an obvious menu of options:

• Sorting quantile: the tails used for the two legs, at 10%, 20% or 30%. Chen & Zimmermann (2022) document that published studies mostly use quintiles or deciles, with no consensus.

• Size filter: whether the smallest firms are removed, and how many. We try no filter, dropping the smallest 5%, and dropping the smallest 10%. Small firms are illiquid and expensive to trade, and they dominate equally weighted portfolios.

• Weighting: equal weights or capitalization weights. Plyakha et al. (2021) show that this choice alone changes the conclusion of asset pricing tests.

That gives $3 \times 3 \times 2 = 18$ designs. This is a deliberately small design space, so it runs in under a minute; the studies cited above span hundreds or thousands of paths.

Because the ranks are uniform within each month, the two design steps are easy to express. A size floor of 0.05 removes exactly the smallest 5% of firms in every cross section. A sorting quantile of 0.10 puts the cheapest tenth in the long leg and the most expensive tenth in the short leg. The function below implements one design; the second function runs all eighteen and reuses the ranking, which depends on the size filter but not on the other two choices.

We now run the eighteen designs on the full sample. Returns are expressed in percent per month.

Every design produces a positive premium, so the sign of the effect is not in question. The magnitude is. The largest estimate is roughly 2.8 times the smallest, and the weakest design by a wide margin is the one that sorts on the extreme deciles, applies no size filter, and weights equally. This is the configuration in which the long leg loads most heavily on tiny illiquid firms, and it is exactly the sensitivity that Asness & Frazzini (2013) emphasize. An analyst who reported only that path, and an analyst who reported only the strongest path, would both be reporting a defensible number.

17.5.1Resampling and the correlation across paths¶

We now repeat the exercise on perturbed versions of the data. Each sample is a random 40% of the stock month observations, drawn without replacement. Under common resampling one sample is drawn and all eighteen designs are run on it. Under path specific resampling a fresh sample is drawn immediately before each design. The two schemes differ only in the order of two lines, but that order determines whether the paths share their sampling noise.

The contrast is stark. Sharing a sample across designs induces an average correlation near 0.69, while drawing a separate sample for each design brings it to zero. Recall from 17.9 that the relevant quantity is $\sum_{p,q}\rho_{p,q}/P^2$, which equals $1/P + (1 - 1/P)\bar{\rho}$ when the correlations are homogeneous. With $P = 18$ this is about 0.71 under common resampling and about 0.05 under path specific resampling, a difference of more than a factor of ten in the variance of the mean.

We now apply the decomposition of 17.5 and 17.6. Because the outcomes form a balanced grid of samples by paths, the population variances (with ddof=0) satisfy the additivity of 17.7 exactly, which the code checks.

The additivity of 17.7 holds to the last digit, as it must. The economically relevant line is the last one: the standard deviation of the mean effect falls from about 0.058 to about 0.017 when resampling is done path by path. The estimate of the average premium is unchanged, but the precision with which we know it improves by a factor of more than three.

Note also that the standard error and the nonstandard error are of comparable size here. That is a consequence of the small design space. With only three forks the paths are close relatives of one another, so their dispersion is limited, while a 40% subsample injects a substantial amount of sampling noise. Coqueret & Pérignon (2026) span 576 paths and find the nonstandard error to be an order of magnitude larger than the standard error for most anomalies. The nonstandard error grows with the number of forks, so a study that enumerates more choices will attribute more of its total variance to methodology.

The top panel shows two well separated distributions. Under common resampling almost every pair of designs has a correlation above 0.4, and many exceed 0.8. Under path specific resampling the correlations are symmetric around zero and concentrated in a narrow band, which is what one would obtain from independent series estimated on 50 observations. The bottom panel translates this into the interval that a reader would use. Both intervals exclude zero, so the value premium survives, but only the narrow one is informative about magnitude.

17.5.2Which choice matters most?¶

A multi path study should also report how each individual fork moves the result. The standard diagnostic is a conditional average: fix one option, average over everything else, and compare. We add the nonstandard Sharpe ratio of 17.11, computed on the full sample with uniform weights.

The nonstandard Sharpe ratio is far above the threshold of one that Coqueret & Pérignon (2026) associate with persistence. On this sample and against these three choices, the value premium is robust: no defensible combination of the three forks makes it disappear.

The conditional averages identify the sorting quantile as the most consequential fork. Moving from the extreme deciles to the extreme quintiles raises the average premium by roughly a fifth. This is the opposite of the usual intuition, which holds that sharper sorts isolate stronger signals. The explanation is that the extreme deciles are thin, so their returns are noisy and dominated by small firms. The size filter and the weighting scheme matter less on average, but recall from the table that they interact: the size filter is what rescues the equally weighted decile sort. Conditional averages summarize a fork in isolation and can hide interactions of this kind, which is a reason to report the full distribution alongside them.

17.6Practical guidance¶

Multi path analysis is now cheap enough that it should be the default for any empirical claim about an anomaly. Four recommendations follow from the material above.

• Enumerate the forks before looking at the results. Writing the design space down first turns implicit degrees of freedom into an explicit object. It also removes the temptation described by Chen (2021), since a path that is chosen after the fact is no longer one draw from a declared space.

• Resample before running each path, not after. This is the single change with the largest effect on inference. It costs one extra line of code and it is what makes the number of paths work in your favor.

• Report the distribution, not only its center. A boxplot or a specification curve of the outcomes conveys the magnitude of design uncertainty. The mean alone conceals it, which is what the exercise above is meant to show.

• Scale the effect by the nonstandard error. The nonstandard Sharpe ratio answers the question a reader actually has, namely whether the result would survive in someone else’s hands.

The broader literature on replication in finance reaches compatible conclusions. Hou et al. (2020) find that a majority of published anomalies weaken considerably under a uniform protocol, while Jensen et al. (2023) argue that most survive once the protocol is chosen carefully, and Pérignon et al. (2024) report that only about half of a sample of finance papers can be reproduced from their own code and data. These studies disagree about how much of the factor zoo is real. Multi design analysis reframes the disagreement: rather than asking which single protocol is correct, it measures how much the answer depends on the question.

17.7Exercises¶

1. Add a fourth fork to the example: the holding period, with monthly and quarterly rebalancing. Recompute the 36 designs and report the new nonstandard Sharpe ratio. Does adding a fork raise the nonstandard error, and by how much?

2. Replace the value characteristic by momentum (mom_252) and run the same eighteen designs. Compare the nonstandard Sharpe ratio of momentum with that of value. Which anomaly is more sensitive to design choices?

3. Vary the resampling fraction from 0.2 to 0.9 while keeping the path specific scheme. Plot the estimated SE and NSE against the fraction. Explain why one of the two curves is far more stable than the other.

4. Implement the non uniform weighting of 17.10. Take the design with a 20% quantile, a 5% size floor and capitalization weights as the baseline, and give every other path a weight proportional to $0.75^{d}$, where $d$ is the number of choices that differ from the baseline. How does the weighted mean compare with the uniform one?

5. Using the outcome matrix from common resampling, compute the correlation between each pair of designs and regress it on the number of steps the two designs share. Verify that commonality between paths predicts correlation between outcomes.

6. Conceptually, the paths in this chapter are all sorted portfolios. Suppose instead that the object of interest is the out of sample $R^2$ of a machine learning forecast, as in Chapter Validating and tuning. List five forks that a researcher would face, and explain which of them you would expect to generate the largest nonstandard error.