Holdout Labs

Holdout Labs · strategy audits

Is your backtest real?

Scientific audits for trading strategies. We measure how much of your backtest survives the tests that catch overfitting, and tell you what to fix.

Order a $149 Quick Audit See a sample report
Illustration: a backtest before and after the holdout split A schematic line rises steadily through the in-sample period, then flattens and wanders after the split into the held-out period. IN-SAMPLE: WHAT YOU OPTIMIZED ON HELD OUT NEVER SEEN
Schematic, not data. The question an audit answers: does the curve you optimized keep its shape on data the search never touched?

How it works

Three steps, fixed price, written report

  1. Send your strategy

    Your rules or code, the data or a data recipe, and an honest count of how many variants you tried before this one. That count matters more than most people expect.

  2. We test it for overfitting

    Deflated Sharpe, probability of backtest overfitting, multiple-testing haircuts, a reality check against data snooping, holdout degradation, and stress tests on costs, parameters and regimes. Every method is published and cited.

  3. You get a grade and a fix list

    An A–F fragility grade computed by a rubric printed in the report, the numbers behind it, and a ranked list of what to change before you trust the backtest.

Services

Three levels of scrutiny

Quick Audit

$149

3–5 business days

  • Holdout split and walk-forward check
  • Deflated Sharpe for the variants you tried
  • Parameter and cost sensitivity
  • A–F fragility grade and fix list
Order Quick Audit

Deep Audit

$490

5–10 business days

  • Everything in the Quick Audit
  • PBO via CSCV, Harvey–Liu haircut, White / Hansen SPA
  • Regime splits, placebo strategies, bootstrap drawdowns
  • Report hash timestamped under RFC 3161
Order Deep Audit

Sealed Forward Trial

$1,500 + $99/mo

60–90 day live registry

  • Preregistered protocol, frozen before the trial
  • Daily hypothetical signals sealed with two independent timestamp authorities
  • A certificate anyone can verify with openssl
Order Sealed Trial

Why trust the grade

We hold ourselves to the same tests

  • Built by a practitionerFounded by Raghuram Vadapally, former Managing Director and Head of Software Engineering at Carlson Capital, a multi-strategy hedge fund, with earlier trading and risk technology roles at Eton Park, Sculptor and J.P. Morgan. About the founder.
  • Published methodsEvery statistic in a report comes from the peer-reviewed literature, with the citation beside it. No proprietary score, no black box. Read the method.
  • Open-source toolkitThe audit engine is open source. You can rerun any statistic in a report yourself. What is open.
  • Preregistered case studyWe tested a widely circulated lunar-cycle claim about Bitcoin volatility on ourselves: two preregistrations, timestamped seals, ten rounds of red-team review. It came back null, and we published it. Read the case study.
  • No signals, no promisesWe never tell you what to buy or sell and never promise performance. The grade describes the evidence, not the future.

Find out before the market does the audit for you.

A Quick Audit is $149, fixed price, delivered in 3–5 business days. If it turns out your backtest is overfit, you get a list of what to fix.

Order a $149 Quick Audit Ask a question first