A backtest looks impressive when you pick the best of many tries and report it as if it were your only try. Walk-forward analysis checks whether a strategy keeps working on data it was not tuned on, one later period at a time. The Deflated Sharpe Ratio (DSR) asks whether the Sharpe ratio you selected is still convincing once you account for how many variants you tried and for non-normal returns. They answer different questions, so use both. Neither one guarantees future performance, and this article is education, not investment advice.
Below you will reproduce the problem on pure noise, build a chronological out-of-sample test with scikit-learn’s TimeSeriesSplit, and compute a DSR with NumPy and SciPy. You will also see how to count the trials that feed it, which is where most DSR calculations go wrong.
As an Amazon Associate I earn from qualifying purchases.
Why is my backtest lying to me?
A backtest is a historical simulation, and a search through it is an experiment. Try 40 lookback windows, five thresholds and three stop rules, and you have run hundreds of experiments. Report only the winner and you have a winner’s-curse problem. The maximum Sharpe ratio among many candidates tends to grow with the number of candidates, even if none has a real edge. David H. Bailey and Marcos López de Prado make this point in “The Deflated Sharpe Ratio: Correcting for Selection Bias, Backtest Overfitting and Non-Normality” (Journal of Portfolio Management, vol. 40, no. 5, pp. 94–107, 2014). Their argument is that not accounting for the number of trials produces overly optimistic expectations.
Two separate defences address this:
| Tool | Question it answers | What it needs | What it does not do |
|---|---|---|---|
| Single in-sample backtest | How well did the best setting fit this history? | One data set | Say anything about data it was not fitted to |
| Walk-forward analysis | Does a procedure that only uses past data keep performing in later periods? | Ordered, ideally equally spaced observations; fixed train/test/gap rules | Correct for the many protocols or ideas you tried before settling on this one |
| Deflated Sharpe Ratio | Is the selected Sharpe statistically compelling after multiple testing and non-normal returns? | Sample length, skewness, kurtosis, spread of trial Sharpes, a defensible trial count | Fix lookahead bugs, unrealistic costs, or regime change |
Reproduce the problem on data with no edge
The fastest way to stop trusting an in-sample result is to watch it succeed on random numbers. The script below builds a return series with zero expected return, then “discovers” the best momentum lookback.
#1 Best Overall
import numpy as np
import pandas as pd
rng = np.random.default_rng(7)
n = 2500
idx = pd.bdate_range("2015-01-01", periods=n)
ret = pd.Series(rng.normal(0.0, 0.01, n), index=idx) # pure noise, no edge
def strategy_returns(ret, lookback):
# Signal uses data up to t; the position is applied to the t+1 return.
signal = np.sign(ret.rolling(lookback).mean())
return (signal.shift(1) * ret).fillna(0.0)
def sharpe(r):
# Per-period (not annualised) Sharpe ratio
return r.mean() / r.std(ddof=1)
lookbacks = list(range(5, 205, 5)) # 40 candidate settings
cands = pd.DataFrame({L: strategy_returns(ret, L) for L in lookbacks})
in_sample = cands.apply(sharpe)
best = in_sample.idxmax()
print("best lookback:", best)
print("best in-sample annualised Sharpe:", in_sample.max() * np.sqrt(252))
This is illustrative code that has not been run for this article, and no outputs are quoted. Run it with several seeds. The best in-sample Sharpe will usually be positive and sometimes attractive, even though the data-generating process has no edge by construction. That gap between “best of 40” and “true value of zero” is the selection bias that DSR is designed to deflate.
How do I do walk-forward analysis in Python?
The rule is simple: for each step, choose parameters using only observations before the test window, score the test window, then move forward. Stitch the test windows into one chronological out-of-sample return series. Scikit-learn’s TimeSeriesSplit generates the ordered index sets. Its documentation (scikit-learn 1.9.1) describes expanding training sets, with each test fold later in time than its training fold. It is an index generator, not a trading backtester, so you supply the strategy logic, execution, fees and slippage.
The controls TimeSeriesSplit gives you
| Parameter | Effect | How to choose it |
|---|---|---|
n_splits |
Number of train/test pairs | Enough folds to see dispersion, but each test window must be long enough to be meaningful |
test_size |
Samples per test window | Match how often you would re-fit or re-evaluate in live use |
max_train_size |
Caps training length, giving a rolling rather than expanding window | Use it if you believe old data is stale; expanding is the default |
gap |
Drops samples at the end of the training set, before the test set | Set it from your label or holding horizon and any signal or execution latency |
No single window size, gap or split count suits every strategy, and the API does not offer finance-specific defaults. Fix them from the strategy’s decision cadence and horizon before looking at results, and report how the outcome changes when you vary them. If you tweak the protocol until the stitched result looks good, you have simply moved the overfitting one level up.
A complete walk-forward loop
from sklearn.model_selection import TimeSeriesSplit
splitter = TimeSeriesSplit(
n_splits=8,
test_size=250, # about one trading year per test window
gap=5, # decide this from your horizon and latency, not from results
)
oos_parts, chosen = [], []
for train_idx, test_idx in splitter.split(cands):
train_sr = cands.iloc[train_idx].apply(sharpe) # selection uses train only
best_L = train_sr.idxmax()
chosen.append(best_L)
oos_parts.append(cands.iloc[test_idx][best_L]) # untouched test window
oos = pd.concat(oos_parts)
print("chosen lookbacks per fold:", chosen)
print("stitched OOS annualised Sharpe:", sharpe(oos) * np.sqrt(252))
print("per-fold annualised Sharpe:",
[round(sharpe(p) * np.sqrt(252), 2) for p in oos_parts])
On the noise series, expect the stitched out-of-sample Sharpe to sit much closer to zero than the in-sample winner, with its sign depending on the seed. Look also at the chosen lookbacks. If they jump around from fold to fold, the parameter is not a stable feature of the data.
This loop is safe because every signal here is causal. Each position uses only data up to the previous bar, so computing strategy returns once over the full series and slicing by index does not leak information. With a fitted model, that shortcut stops being safe, and everything learned from data must be refit inside each fold.
What gap does and does not do
gap removes samples between the end of training and the start of testing. It does not automatically purge every form of overlapping label or portfolio exposure. If your label looks 10 days ahead, training rows near the boundary contain outcomes that overlap the test window, so the gap must cover that horizon. Open positions carried across a boundary need handling too.
Data spacing
The documentation assumes equally spaced samples if fold metrics are to be comparable. With irregular observations, such as event bars or missing sessions, 250 rows may span very different calendar periods. Either use a defensible resampling scheme or write a date-aware splitter that cuts on timestamps.
What is the Deflated Sharpe Ratio?
The authors describe the DSR as a Probabilistic Sharpe Ratio (PSR) whose rejection threshold is raised to reflect how many trials were run. In their words: “The Deflated Sharpe Ratio (DSR) corrects for two leading sources of performance inflation: Selection bias under multiple testing and non-Normally distributed returns.” It is not a raw Sharpe ratio with a flat haircut. It combines these inputs:
- the estimated Sharpe ratio and the sample length
T; - the skewness and kurtosis of returns;
- the dispersion (variance) of the Sharpe estimates across the trials;
- the effective number of independent trials,
N.
In the paper’s notation, with Sharpe ratios in per-period units, skewness g3 and non-excess kurtosis g4 (a normal distribution has 3):
PSR(SR*) = Phi( (SR − SR*) × sqrt(T − 1) / sqrt(1 − g3×SR + (g4 − 1)/4 × SR²) )SR0 = sqrt(V) × [ (1 − γ) × Phi⁻¹(1 − 1/N) + γ × Phi⁻¹(1 − 1/(N×e)) ]DSR = PSR(SR0)
Here V is the variance of the Sharpe estimates across trials, and γ is the Euler–Mascheroni constant, 0.5772156649, which appears in the paper’s approximation of the expected maximum Sharpe across independent trials. SR0 is the Sharpe you would expect the best of N skill-less trials to reach. The DSR is then the probability that your observed Sharpe genuinely exceeds that bar.
from scipy.stats import norm, skew, kurtosis
EULER_GAMMA = 0.5772156649
def expected_max_sharpe(n_trials, sr_variance):
if n_trials <= 1:
return 0.0
return np.sqrt(sr_variance) * (
(1 - EULER_GAMMA) * norm.ppf(1 - 1 / n_trials)
+ EULER_GAMMA * norm.ppf(1 - 1 / (n_trials * np.e))
)
def probabilistic_sharpe(returns, sr_benchmark=0.0):
r = np.asarray(returns, dtype=float)
T = len(r)
sr = r.mean() / r.std(ddof=1) # per-period
g3 = skew(r)
g4 = kurtosis(r, fisher=False) # non-excess: normal = 3
denom = np.sqrt(1 - g3 * sr + (g4 - 1) / 4 * sr ** 2)
return norm.cdf((sr - sr_benchmark) * np.sqrt(T - 1) / denom)
def deflated_sharpe(returns, n_trials, sr_variance):
return probabilistic_sharpe(
returns, expected_max_sharpe(n_trials, sr_variance)
)
Keep every Sharpe in the same units. If you feed an annualised Sharpe into sr but a per-period variance into V, the result is meaningless. Annualise only for display.
Do these 3 things before closing this tab:
1Repair Windows errors before they cause bigger problems2Fix the driver behind crashes, sound loss and screen glitches3Clear out junk files and repair common Windows errorsApplying it to the in-sample winner
trial_var = in_sample.var(ddof=1) # spread of per-period Sharpes across trials
best_returns = cands[best]
for n_eff in (1, 5, 10, 40):
print(n_eff, deflated_sharpe(best_returns, n_eff, trial_var))
The loop prints the DSR for several assumed trial counts rather than one. The paper treats effective trial count as something to estimate, so showing how the verdict moves with N is more honest than committing to a single number you cannot defend.
A hand-worked illustration with assumed inputs
These inputs are assumptions for illustration, not measured data. Suppose a strategy has an annualised Sharpe of 1.5 over 1,260 daily returns (about five years). That is about 0.0945 per day. Assume roughly normal returns (skew 0, kurtosis 3), N = 100 independent trials, and a trial-Sharpe standard deviation of 0.02 per day.
SR0comes to about 0.0506 per day, roughly 0.80 annualised. That is the Sharpe the best of 100 skill-less trials would be expected to reach under these assumptions.- The test statistic is about (0.0945 − 0.0506) × √1259 ≈ 1.55, so the DSR is about 0.94.
- With
N= 1 the benchmark is zero, the statistic is about 3.35, and the probability is about 0.9996.
Same returns, same length, but selecting from 100 trials cuts the evidence markedly. Raise the trial-Sharpe spread to 0.04 per day and SR0 rises to about 0.10, above the observed 0.0945, so the DSR falls below 0.5. Your conclusion depends heavily on the trial count and spread, which is why you should record them.
How many backtests did I run?
This is the question that determines whether your DSR means anything. The paper’s N is the number of independent trials, and two mistakes are common:
- Undercounting. Only the final grid is counted, not the earlier signals, universes, data cleaning variants, and abandoned ideas that shaped the one you kept.
- Overcounting by treating correlated runs as independent. A 20-day and a 25-day lookback on the same data produce highly similar return streams. They are not 2 independent experiments. The authors discuss estimating effective independent trials when tests are correlated.
The paper’s stated corrections are for selection bias under multiple testing and for non-normality. A defensible workflow:
- Log every strategy variant you evaluate from the first day: parameters, universe, date range, and per-period return series. A simple results table or a folder of saved series is enough.
- Compute the per-period Sharpe of each logged trial. That gives you the variance
Vdirectly. - Estimate effective independence, for example by examining the correlation structure of the logged return series, and state the method you used.
- Report DSR across a plausible range of
N, as in the loop above, instead of one favourable value.
If the conclusion flips between a reasonable low and high N, treat the evidence as inconclusive.
Walk-forward results need the same accounting. The stitched out-of-sample series is itself a candidate, and if you tried several window lengths, gaps, or model families before settling on one, those are trials too. A DSR on that series should count the walk-forward configurations and ideas you tried, not only the parameter grid inside each fold.
What neither tool can tell you
- There is no universal pass mark. The paper reports no industry-wide backtest failure rate and no DSR cutoff that applies to all strategies. Treat any quoted threshold as a convention you choose and disclose, not a law.
- DSR does not fix a flawed simulation. Lookahead bias, survivorship bias, unrealistic fills, and ignored costs are outside its stated scope.
- Walk-forward does not remove regime risk. A strategy that survived the sample’s periods can still fail when conditions change.
- A high DSR is a statistical statement about a historical record. It is not a forecast, and a profitable backtest is not a guarantee of future returns.
Implementation mistakes to check for
- Shuffling observations or using ordinary random K-fold on autocorrelated time series, which lets future-like observations into training.
- Choosing settings on a test fold, then reporting that same fold as out-of-sample.
- Fitting scalers, feature selection, or other transforms across all dates before splitting. Fit each on the training data only, ideally inside a pipeline refit per fold.
- Ignoring overlapping forward labels, signal latency, transaction costs, slippage, or a
gapthat is too short for the strategy’s horizon. - Reporting only the best fold or best parameter set instead of the full chronological out-of-sample record and the dispersion across folds.
- Feeding an unexamined trial count into the DSR, or implying it corrects for every possible bias.
Pre-flight checklist
- Fix train, test and gap rules from the deployment cadence and label horizon before viewing results.
- Confirm all features are causal and that every fitted transform is refit per fold.
- Apply costs and slippage inside the test windows.
- Save the stitched out-of-sample series and the per-fold metrics, and inspect fold-to-fold parameter stability.
- Keep a trial log from the start and compute the Sharpe variance across trials.
- Report the DSR for a range of effective trial counts, with the method behind each.
- Disclose how sensitive the result is to window, gap and split choices.
For deeper treatment of these ideas, Marcos López de Prado’s book Advances in Financial Machine Learning is an optional follow-up, since the author of the DSR paper wrote it. Check the current edition before buying.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
