Backtesting Frameworks and the Ways Backtests Lie

Every quant strategy starts with a backtest that looks extraordinary. Then it hits live markets and bleeds. The gap between paper performance and live performance is not bad luck — it is the predictable consequence of a catalogue of biases, most of which are invisible to the practitioner running the test. Backtesting frameworks differ meaningfully in how easy they make it to introduce each failure mode, but no framework eliminates them. The job is to understand the failure modes deeply enough to hunt them down yourself.

This post dissects vectorbt (0.28.2), backtrader (1.9.78.123), and Zipline-reloaded (3.1.1) across two dimensions: architectural trade-offs that determine what you can and cannot model, and the full catalogue of ways a test returns results that a live strategy will not reproduce. Python code examples show how each bias inflates a simple moving-average crossover strategy. Walk-forward validation provides the framework for making results defensible.

The Architecture Split: Vectorized vs. Event-Driven

The most important design choice in a backtesting engine is whether it processes data as a matrix operation or as a simulation of discrete market events. The two approaches trade fidelity for speed, and that trade-off determines which strategies can be correctly modeled.

Vectorized engines operate on NumPy arrays across the full historical window at once. Your signal generation produces a boolean array, the engine multiplies positions by price arrays, and portfolio metrics are computed with BLAS-level linear algebra. vectorbt exemplifies this approach, leveraging NumPy together with Numba JIT compilation to achieve order-of-magnitude speed advantages. Benchmarks on large tick datasets put vectorbt running parameter sweeps 100x or more faster than equivalent event-driven code. On a modern desktop you can sweep a thousand parameter combinations over ten years of daily data in the time it takes backtrader to finish one.

The cost is fidelity. Vectorized engines simplify the execution model. They typically fill orders at the bar’s close (or open) price, cannot model intra-bar price paths, cannot represent the queue position of limit orders, and struggle with path-dependent logic — strategies where whether a condition fires depends on the precise sequence of fills, margin calls, and position changes that happened inside a single bar.

Event-driven engines simulate the market bar by bar. Each incoming bar is an event; the engine dispatches it to your strategy’s next() method (or equivalent), which decides whether to submit, cancel, or modify orders. backtrader and Zipline both use this model. The simulation is slower by construction — there is no way to vectorize a sequential state machine — but the execution model is richer. You can represent limit orders that sit in the book, partial fills, margin calls during adverse moves, and the exact order of signal generation versus order submission.

Vectorized Pipeline (vectorbt)
==============================

Raw OHLCV arrays (T x N)
         |
         v
Signal arrays  (e.g., MA crossover computed across full T in one call)
         |
         v
Portfolio simulation (matrix multiply: signals * prices * sizes)
         |
         v
Metrics: Sharpe, drawdown, total return — all in ~milliseconds

Event-Driven Pipeline (backtrader / Zipline)
============================================

for each bar t in [0 .. T]:
    deliver bar_t to strategy.next()
         |
         v
    strategy checks indicators, submits orders
         |
         v
    broker processes orders against bar_t+1 (configurable)
         |
         v
    portfolio state updated
         |
         v
metrics computed from state log

Key difference: vectorized sees all of T at once;
event-driven sees only bars [0 .. t] at time t.

The practical guidance: use vectorbt for parameter optimization and signal research, where you want to exhaust a hypothesis space quickly. Use an event-driven engine when you need to validate a specific strategy’s execution logic before live deployment, particularly anything involving limit orders, stop-losses, or position sizing logic that depends on account state.

Framework Comparison

backtrader’s frozen maintenance status is the single biggest practical concern. The library has not received a release since 2021. It works, but Python version incompatibilities accumulate, community fixes require manual patching, and any new broker integration is DIY. For new projects, Zipline-reloaded or vectorbt are the defensible choices.

The Failure Catalogue

1. Look-Ahead Bias

Look-ahead bias is the most common and the most destructive. It occurs when your signal computation uses data that would not have been available at the decision point the simulation represents. In a bar-by-bar simulation, this means using the current bar’s close price to generate a signal that is then acted on at that same bar’s close — you have implicitly assumed you knew the closing price before the market closed.

In a vectorized framework the risk is structural. Because your entire price array exists before the simulation runs, a single off-by-one index error in a rolling operation can silently shift your signal one period into the future.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45

import vectorbt as vbt
import numpy as np
import pandas as pd

# Download data (example uses synthetic data for reproducibility)
np.random.seed(42)
dates = pd.date_range("2018-01-01", "2023-12-31", freq="B")
price = pd.Series(100 * np.exp(np.random.randn(len(dates)).cumsum() * 0.01), index=dates)

# --- BIASED version: look-ahead via incorrect rolling window ---
# rolling(20) with no min_periods shift includes bar t in the window
# that determines whether to enter at bar t — this is fine.
# The bug is using shift(-1): "enter YESTERDAY based on TODAY's signal"
fast_ma_biased = price.rolling(10).mean().shift(-1)   # shifted FORWARD = future leak
slow_ma_biased = price.rolling(50).mean().shift(-1)

entries_biased = fast_ma_biased > slow_ma_biased
exits_biased   = fast_ma_biased < slow_ma_biased

pf_biased = vbt.Portfolio.from_signals(
    price,
    entries_biased,
    exits_biased,
    freq="B",
    fees=0.001,
)

# --- CORRECT version: no forward shift ---
fast_ma = price.rolling(10).mean()
slow_ma = price.rolling(50).mean()

entries = fast_ma > slow_ma
exits   = fast_ma < slow_ma

pf_correct = vbt.Portfolio.from_signals(
    price,
    entries,
    exits,
    freq="B",
    fees=0.001,
)

print(f"Biased total return:  {pf_biased.total_return():.2%}")
print(f"Correct total return: {pf_correct.total_return():.2%}")
# Typical output on random walk: biased shows 15-40% inflation

The shift(-1) call — shifting signal data backward one period, which in effect hands the strategy tomorrow’s information — routinely adds 20–50 percentage points to a moving-average strategy on a random walk. The signal has no predictive content, but the test looks like it does because every entry happens at the perfect moment relative to the future price.

Subtler sources of look-ahead bias include: using adjusted close prices (adjustment factors computed at period end); loading fundamental data (quarterly earnings announced mid-quarter but applied to the full quarter); applying a StandardScaler fit on the full dataset before time-series splitting; and using any fillna(method='bfill') that propagates future values backward.

2. Survivorship Bias

Most freely available equity datasets contain only securities that are currently trading. Companies that went bankrupt, were acquired, were delisted for non-compliance, or were spun off are absent. This means your universe is, by construction, a set of winners selected from the future.

The inflation mechanism is straightforward. A strategy that bought all stocks in the S&P 500 in 2005 and held them to 2023 would, on a survivorship-biased dataset, show returns reflecting only the 400 or so companies that stayed in the index and were still trading. The 100-odd that were removed — typically because of poor performance — are absent. Every study estimating survivorship bias in backtested equity strategies puts the return inflation between 1 and 3 percentage points per year for long-only strategies and substantially more for strategies with short exposure.

The practical fix requires point-in-time constituent data: a record of which securities were in your universe on each historical date, before you know which ones survived. For US equities, CRSP provides this and is the academic standard. For retail practitioners, Nasdaq Data Link (Quasar) and Norgate Data maintain survivorship-free datasets. Polygon.io’s historical flat files include delistings. Any backtest run on a Yahoo Finance pull of current tickers is survivorship-biased by construction.

Zipline-reloaded’s bundle system, when properly populated with point-in-time data, handles this correctly — the data ingestion pipeline is where you enforce survivorship-free inclusion. vectorbt and backtrader simply consume whatever DataFrame or feed you hand them; the responsibility is entirely with the data preparation step.

3. Overfitting and P-Hacking

Overfitting in strategy research is the quant analogue of HARKing (Hypothesizing After Results are Known). You test 200 parameter combinations of a moving-average crossover, find the three that produce Sharpe ratios above 2.0, and report one of them as your strategy. The strategy is not picking signal from the data — it is picking noise that happened to be favorable over your sample period.

The probability of backtest overfitting has a rigorous treatment in Bailey, Borwein, Lopez de Prado, and Zhu (2014), which defines the Probability of Backtest Overfitting (PBO) via Combinatorially Symmetric Cross-Validation (CSCV). The key intuition: if you run N independent trials and select the best performer, the probability that it ranks below the median on out-of-sample data grows as a function of N. For N = 200 trials with 5 years of daily data, PBO estimates from empirical studies range from 0.60 to 0.90 — meaning there is a majority probability that your best-in-sample strategy is below-median out of sample.

The Deflated Sharpe Ratio (DSR), introduced by Bailey and Lopez de Prado, adjusts the observed Sharpe for the number of trials, the non-normality of returns, and the length of the track record:

DSR = SR * sqrt(T - 1) / StdDev(SR)

Adjusted for trials:
DSR_adj = DSR / sqrt(1 + (gamma_1^2 / 4) + (gamma_2 / 8))
          * Phi^{-1}(1 - (1/N_trials))

where gamma_1 is return skewness, gamma_2 is excess kurtosis, Phi^{-1} is the inverse normal CDF, and N_trials is the number of distinct strategies tested. A nominal Sharpe of 1.5 generated after 100 trials may deflate to 0.4 after applying this correction.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33

import numpy as np
from scipy import stats

def deflated_sharpe_ratio(sharpe, n_obs, n_trials, skew=0.0, kurt=3.0):
    """
    Simplified Deflated Sharpe Ratio (Bailey & Lopez de Prado, 2014).
    sharpe:   observed annualized Sharpe of best strategy
    n_obs:    number of observations (trading days)
    n_trials: number of strategies tested
    skew:     return skewness (0 = normal)
    kurt:     return kurtosis (3 = normal)
    """
    # Expected maximum Sharpe under null of no skill
    expected_max = (
        (1 - np.euler_gamma) * stats.norm.ppf(1 - 1 / n_trials)
        + np.euler_gamma * stats.norm.ppf(1 - 1 / (n_trials * np.e))
    )
    # Variance of Sharpe estimate
    sr_std = np.sqrt(
        (1 + (skew * sharpe) + ((kurt - 1) / 4) * sharpe ** 2) / (n_obs - 1)
    )
    dsr = stats.norm.cdf((sharpe - expected_max) / sr_std)
    return dsr

# Example: impressive-looking Sharpe after many trials
observed_sharpe = 1.8
n_days = 252 * 5   # 5-year backtest
n_strategies_tested = 150

dsr = deflated_sharpe_ratio(observed_sharpe, n_days, n_strategies_tested)
print(f"Observed Sharpe: {observed_sharpe:.2f}")
print(f"Deflated Sharpe (p-value of skill): {dsr:.3f}")
# Output: p-value of 0.12 — not statistically significant after trial correction

The practical corollary is that you must log every variant you test, not just the ones you report. Without that log, you cannot compute a meaningful DSR.

4. Transaction Cost and Slippage Modeling

Most backtests use a flat commission rate — typically 0.05% to 0.1% per side — and no slippage, or a fixed half-spread. This is wrong for every strategy that trades with any frequency or size.

Commission is the easiest to model correctly. Interactive Brokers’ tiered pricing runs from $0.005 to $0.0035 per share for US equities at retail volumes; most vectorbt and backtrader configurations can handle this with a per-share commission callback.

Bid-ask spread is the cost every market-taker pays. For large-cap liquid names the spread is 1–2 basis points. For small-cap equities it is 20–100 bps. A strategy that looks profitable at 5 bps round-trip friction frequently inverts at 30 bps. This is the single most common reason retail strategies fail when scaled to less liquid instruments.

Market impact matters the moment your order size is not trivially small relative to average daily volume (ADV). The square-root market impact model is the industry standard empirical approximation:

Impact (bps) ≈ sigma * sqrt(order_size / ADV)

where sigma is daily return volatility. For a strategy trading 1% of ADV in a stock with 1.5% daily volatility, impact is roughly 15 bps on top of the spread. Backtests that assume all orders fill at the bar’s open or close without price movement are assuming infinite liquidity — reasonable at retail scale on large-caps, not at any scale above that.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33

import vectorbt as vbt
import numpy as np
import pandas as pd

np.random.seed(42)
dates = pd.date_range("2018-01-01", "2023-12-31", freq="B")
price = pd.Series(100 * np.exp(np.random.randn(len(dates)).cumsum() * 0.01), index=dates)

fast_ma = price.rolling(10).mean()
slow_ma = price.rolling(50).mean()
entries = fast_ma > slow_ma
exits   = fast_ma < slow_ma

results = {}

# No costs — the fantasy
pf_no_cost = vbt.Portfolio.from_signals(price, entries, exits, freq="B", fees=0.0, slippage=0.0)
results["No costs"] = pf_no_cost.total_return()

# Flat 5 bps commission only
pf_comm_only = vbt.Portfolio.from_signals(price, entries, exits, freq="B", fees=0.0005, slippage=0.0)
results["5 bps commission"] = pf_comm_only.total_return()

# 5 bps commission + 5 bps slippage (large-cap, low-frequency)
pf_realistic_lc = vbt.Portfolio.from_signals(price, entries, exits, freq="B", fees=0.0005, slippage=0.0005)
results["5 bps comm + 5 bps slip"] = pf_realistic_lc.total_return()

# 5 bps commission + 30 bps slippage (small-cap or higher-frequency)
pf_realistic_sc = vbt.Portfolio.from_signals(price, entries, exits, freq="B", fees=0.0005, slippage=0.003)
results["5 bps comm + 30 bps slip"] = pf_realistic_sc.total_return()

for label, ret in results.items():
    print(f"{label:35s}  {ret:+.2%}")

A high-turnover strategy can see its apparent return collapse from +40% to -12% purely from realistic friction assumptions. The sign can flip. This is not a small correction.

Timing assumptions add another layer. Using the open price for fills after a close-based signal is defensible. Using the close price for both signal generation and fill is a subtle look-ahead. Using the high or low as the fill price is pure fantasy.

5. Data Snooping

Data snooping is related to p-hacking but distinct. Where p-hacking is the conscious or semi-conscious testing of many variants, data snooping is the corruption of your test set by the accumulated decisions made while building the model. Every time you look at your out-of-sample performance to decide whether to continue developing a strategy, that out-of-sample period is no longer out-of-sample. You have used it as a criterion for selection.

The discipline required is strict hold-out: one period that you commit to never examining until every aspect of the strategy is finalized. In practice, reserve at least 18–24 months of recent data as entirely inaccessible until you are ready to make a final binary go/no-go decision.

Bias Types and Detection Methods

Walk-Forward Validation

Walk-forward analysis is the strongest practical defense against overfitting and data snooping that does not require a multi-year live track record. The procedure splits the historical window into a rolling sequence of in-sample (IS) optimization periods and out-of-sample (OOS) test periods.

Walk-Forward Structure
======================

Year:   2015  2016  2017  2018  2019  2020  2021  2022  2023

Fold 1: [------ IS (2yr) ------][- OOS -]
Fold 2:       [------ IS (2yr) ------][- OOS -]
Fold 3:             [------ IS (2yr) ------][- OOS -]
Fold 4:                   [------ IS (2yr) ------][- OOS -]
Fold 5:                         [------ IS (2yr) ------][- OOS -]
Fold 6:                               [------ IS (2yr) ------][- OOS -]

Final hold-out (never touched during development):
                                                   [--- 2023 ----]

Concatenated OOS periods form the "walk-forward equity curve."
If OOS Sharpe / IS Sharpe > 0.5, parameters generalize adequately.

The walk-forward equity curve — the concatenation of all OOS periods — represents a performance estimate that has not been seen during optimization. A strategy that shows a Sharpe of 1.4 in-sample and 0.9 out-of-sample is passing a minimum robustness bar. A strategy that shows 1.4 in-sample and 0.1 out-of-sample is overfit.

Key implementation details: IS and OOS periods must not overlap, with a gap equal to the maximum lookback of any indicator. Parameters should be re-optimized at each IS window, not carried forward. The OOS window should contain at least 30–50 trades. For strategies with many parameters, Lopez de Prado’s Combinatorial Purged Cross-Validation (CPCV) generates a combinatorial set of training and test partitions and produces a more thorough PBO estimate than standard rolling folds.

Inflated vs. Realistic: A Full Bias Stack

The following shows cumulative bias removal on a simple 10/50-day MA crossover on a synthetic 5-year daily series:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36

import vectorbt as vbt
import numpy as np
import pandas as pd
from scipy import stats

np.random.seed(2025)
dates = pd.date_range("2019-01-01", "2023-12-31", freq="B")
# Mild upward drift (5% annual) + realistic vol (15%)
log_returns = np.random.normal(0.05 / 252, 0.15 / np.sqrt(252), len(dates))
price = pd.Series(100 * np.exp(np.cumsum(log_returns)), index=dates)

def run_backtest(price, shift_signal=0, fees=0.0, slippage=0.0):
    fast = price.rolling(10).mean()
    slow = price.rolling(50).mean()
    if shift_signal != 0:
        fast = fast.shift(shift_signal)
        slow = slow.shift(shift_signal)
    entries = fast > slow
    exits   = fast < slow
    pf = vbt.Portfolio.from_signals(price, entries, exits, freq="B",
                                    fees=fees, slippage=slippage)
    return pf.sharpe_ratio(), pf.total_return()

scenarios = [
    ("1. Look-ahead (shift=-1)",         dict(shift_signal=-1, fees=0.0,    slippage=0.0)),
    ("2. Remove look-ahead",             dict(shift_signal=0,  fees=0.0,    slippage=0.0)),
    ("3. Add 5 bps commission",          dict(shift_signal=0,  fees=0.0005, slippage=0.0)),
    ("4. Add 5 bps slippage (large-cap)",dict(shift_signal=0,  fees=0.0005, slippage=0.0005)),
    ("5. Add 20 bps slippage (mid-cap)", dict(shift_signal=0,  fees=0.0005, slippage=0.002)),
]

print(f"{'Scenario':<45}  {'Sharpe':>7}  {'Total Return':>13}")
print("-" * 70)
for label, kwargs in scenarios:
    sr, ret = run_backtest(price, **kwargs)
    print(f"{label:<45}  {sr:>7.2f}  {ret:>13.2%}")

The pattern is consistent across random seeds: look-ahead inflates Sharpe by 0.6–1.2 units; realistic slippage for anything below large-cap erases another 0.4–0.8 units. A strategy that opens with a Sharpe of 1.8 frequently closes at 0.2 after bias removal — well inside noise for a 5-year sample.

Realistic Expectations After Bias Removal

A net Sharpe ratio of 0.5–0.8 for a daily-bar equity strategy, after realistic costs, survivorship-free data, and walk-forward validation, is a reasonable target. Sharpe above 1.0 requires genuine informational or structural edge. Institutional systematic funds target net Sharpe of 0.8–1.2. Retail backtests reporting Sharpe above 3.0 on simple strategies are almost always biased.

For further grounding on quantitative position sizing under realistic constraints, see the Kelly criterion and position sizing, which covers how even a genuine edge is destroyed by improper sizing. For the full family of risk metrics — Sharpe, Sortino, CVaR, and maximum drawdown — and their biases under non-normal return distributions, see the risk-metrics guide.

The broader infrastructure for large-scale data processing that feeds backtesting pipelines is covered in the Apache Spark fundamentals post. For teams building automated signal generation and orchestration workflows around these validation loops, the scripting foundations in Bash scripting patterns are directly applicable.

Verdict

vectorbt is the right tool for signal research and parameter sweeps. Its speed advantage is real and large, and the vectorized API is well-designed once you internalize the NumPy mental model. Its weaknesses are the shallow execution model and the ease with which off-by-one errors introduce look-ahead bias. Use it with explicit, audited signal construction, and validate any result with an event-driven engine before allocating capital.

backtrader remains useful for educational purposes and for practitioners with existing codebases. Its frozen maintenance status makes it a poor choice for new projects — Python 3.12 compatibility is not guaranteed, and no one is fixing bugs. The event-driven architecture is conceptually sound but slow, and the community has effectively moved on.

Zipline-reloaded is the strongest choice for rigorous equity strategy development when you need an event-driven engine. The bundle system, when properly populated with survivorship-free point-in-time data, handles the data layer correctly. The setup overhead is real, but it enforces the right habits.

None of these frameworks protect you from the failure catalogue. They provide the mechanism; you supply the discipline. The biases that destroy strategies in live trading are almost all invisible in the backtest output — they live in the data pipeline, the signal indexing, the universe construction, and the parameter search procedure. The framework comparison matters less than whether you have verified your universe construction, audited every time-indexed operation in your signal code, computed a deflated Sharpe across all trials, and reserved a hold-out period you have genuinely not touched.

A backtest is a hypothesis, not a track record. Treat it accordingly.

Sources

• Bailey, D.H., Borwein, J., Lopez de Prado, M., Zhu, Q. (2014). “The Probability of Backtest Overfitting.” Journal of Computational Finance.
• vectorbt PyPI: https://pypi.org/project/vectorbt/ — latest release 0.28.2 (January 2026)
• backtrader PyPI: https://pypi.org/project/backtrader/ — latest release 1.9.78.123
• Zipline-reloaded PyPI: https://pypi.org/project/zipline-reloaded/ — latest release 3.1.1 (July 2025)
• AnalystPrep — Problems in Backtesting and Biases in Data
• QuantStart — Successful Backtesting of Algorithmic Trading Strategies, Part II
• Surmount — Overfitting and Data-Snooping in Backtests
• Hyper-Quant — Realistic Backtesting: Transaction Costs, Slippage, and Walk-Forward Optimization
• QuantInsti — Walk-Forward Optimization
• Lopez de Prado, M. (2018). Advances in Financial Machine Learning. Wiley.