[Quant Trading System 1/5] From no_trade Discipline to Reproducible Databento Ingestion

A Research System, Not A Trading Bot

When I started this quant trading system, the first idea I wanted to make explicit was simpler than it looked. Building a trading system does not mean the system should keep finding trades. If anything, a serious research system should be very good at deciding not to trade most of the time.

Algorithmic trading is a domain where it is easy to fool oneself. A small positive equity curve, one parameter that looks good on validation, or a strategy name seen on Reddit or GitHub can make it feel as if an edge has been found. In practice, most signals collapse under cost, slippage, regime shift, leakage, or day concentration. So the starting question for this project was not “which model should I train?” It was closer to “which conditions must fail before I am allowed to not train a model?”

The long-term goal can still be live trading. But live-ready does not simply mean that a broker API can submit orders. The version I wanted was slower and more uncomfortable. Preserve raw market data. Separate bronze and silver artifacts. Freeze train, validation, and test splits. Keep the test split locked until the end. Make deterministic baselines fail or survive first. Only after that should ML become a legitimate next step.

That is why the most important default in this system is not buy or sell. It is no_trade.

Treating no_trade As A Real Result

When writing backtests, it is easy to make only active strategies look like outcomes. A rule triggers, the code calculates performance, the best-looking rule is selected, and then the weaknesses are explained afterward. I wanted the opposite order. Doing nothing should always remain inside the candidate set, and if an active candidate cannot beat doing nothing, it should not be selected.

This idea kept repeating later in the candidate policy, strategy zoo, parameter sweep, and research gate dashboard. The names changed, but the structure remained similar. Generate candidates, split train and validation, subtract costs, then check whether the candidate is better than no-trade. If it does not pass, it may still be an interesting observation, but it is not a strategy.

The cost assumption was separated into its own object for the same reason. A small number can change the whole conclusion when the instrument is MES and the horizon is short.

@dataclass(frozen=True)
class CostAssumptions:
    version: str = "ibkr_mes_retail_v1"
    mes_fee_per_side_usd: float = 0.62
    mes_tick_size_points: float = 0.25
    mes_contract_multiplier_usd_per_point: float = 5.0
    spread_ticks_round_turn: float = 1.0
    slippage_ticks_round_turn: float = 2.0

    @property
    def round_turn_cost_points(self) -> float:
        return self.execution_cost_points + self.fee_cost_points

Under the current research assumption, round-turn cost is 0.998 MES points, or about $4.99 per contract. That is not a small number. If the average edge at a five-minute horizon is only a few ticks, a slightly more realistic execution assumption can erase the strategy. So this system makes me look at cost arithmetic before I look at attractive performance charts.

A Structure For Delaying ML

From the outside, this can look like an ML project. The repository now has a training matrix and many feature families. But the actual conclusion so far is closer to “do not train a model yet.” I do not consider that a failure. A good pipeline should block bad training before it becomes expensive and convincing.

The first question I wanted the system to answer was:

At this minute, is MES one of the best long opportunities available today?

That question is stricter than a simple directional classifier. It does not ask the model to predict every minute. It asks the system to find only the moments worth trading inside a day. That is why trade count was capped. At most five trades per day, and zero trades is allowed. This small condition changes the character of the whole system. A model that can reject trades matters more than a model that always finds activity.

The initial strategy decisions in the README carried the same shape:

Tradable instrument: MES
Reference data: ES plus MES context
Main horizon: 5-minute opportunities first
Trade count: 0 to 5 trades per day
Direction: Long-only first, both long and short later
Pace: Slow and steady: baselines before complex ML

The important part is “slow and steady.” In trading research, the moment something looks quickly discovered is often the dangerous moment. When validation PnL turns positive inside a notebook, the human already wants to believe the conclusion. This quant trading system is deliberately shaped to delay that belief.

The First Success Is Not Trading

The results so far have kept that principle intact. The first deterministic candidate policy selected no_trade. The strategy zoo selected no_trade. A parameter sweep produced a validation-positive row, but it failed on train. Exit-label sensitivity produced a better-looking validation row, but it failed train and cost stress. The market advantage scorecard and multi-horizon diagnostics eventually returned to the same conclusion.

It can feel strange to write this in a blog post. Usually, the more attractive story is “the model improved,” “the signal worked,” or “the backtest found profit.” But in real research, the most valuable result is often the opposite: discovering that an idea is not yet strong enough to trade before touching the test split, before training ML, and before starting paper trading.

The current state of the quant trading system can be summarized as:

{
  "selected_candidate_id": "no_trade",
  "candidate_ready_for_test_or_ml": false
}

I like this result. Not because it made money, but because the system can say it is not convinced. The first job of a research pipeline is not to find a good strategy. It is to prevent a bad strategy from looking good.

The Attitude A Quant Trading System Needs

As I worked on this project, I kept thinking about where responsibility belongs. Trading code requires more caution than a normal toy project. The numbers connect to money, and a small leakage bug or wrong cost assumption can flip the conclusion completely. So I see this repository less as “code that makes profit” and more as “code that makes judgment conservative.”

A good research system should calm the researcher down. If validation is positive but train is broken, stop. If train and validation look good but cost stress breaks, stop. If cost survives but day concentration is too high, stop. Until all of those gates pass, the test split should stay closed.

There is no successful model-training story yet. The more important record is why I decided not to train, why the test split stayed locked, and how no_trade became a legitimate result of research discipline.

Declaring that default was not enough. The first implementation work had to freeze data provenance and integrity before any strategy code could become persuasive.

Data Fails Before Strategies Do

The most tempting thing to do when starting trading research is to write a strategy immediately. SMA crossover, mean reversion, order-book imbalance, anything with a name feels more interesting. But the first serious work in this quant trading system was not signal design. It was moving, preserving, validating, and documenting the data well enough that later results could be trusted.

The order matters. Once market data is mixed incorrectly, everything downstream becomes contaminated. If I forget where the raw files came from, which symbology was requested, whether a batch job was split by symbol, or how conversion was performed, then even a good-looking backtest becomes hard to interpret. I cannot tell whether a strategy failed or the data was wrong.

So the quant trading system separated the Git repository from the market-data root from the start.

/path/to/quant-trading-system
/path/to/quant-trading-market-data

The repository holds code, documentation, and small manifests. Large DBN, DBN.zst, and Parquet artifacts stay outside Git. This was not only a storage decision. It was a boundary between lightweight research history and reproducible local data.

Preserving Databento Batches By Job

The first Databento GLBX downloads were parent-symbol requests. Because they used stype_in=parent, stype_out=instrument_id, and split_symbols=false, splitting the downloaded files into ES/MES directories too early would have hidden the original request shape. The raw layout therefore preserved the Databento job id:

raw/databento/GLBX.MDP3/jobs/<job_id>/<schema>/

That decision appears directly in the code. JobInfo reads dataset, schema, symbols, symbology, time range, encoding, compression, and split settings from the download metadata, then calculates the canonical raw location.

@dataclass(frozen=True)
class JobInfo:
    source_dir: Path
    job_id: str
    dataset: str
    schema: str
    symbols: tuple[str, ...]
    stype_in: str
    stype_out: str
    start_ns: int
    end_ns: int
    encoding: str
    compression: str
    split_symbols: bool

    @property
    def job_schema_dir(self) -> Path:
        return Path("raw") / "databento" / self.dataset / "jobs" / self.job_id / self.schema

I like this structure because it reduces later ambiguity. Raw data should not look as if it has already been cleaned or interpreted. It should continue to show which batch job it came from, which schema it used, and how the vendor request was shaped.

Even Copying Needs Integrity Checks

Copying data looks trivial, but in this project it is the first integrity gate. databento_pipeline.py does not simply copy files. It fails if file sizes differ. If a manifest hash exists, it verifies SHA-256. Existing destination files are accepted only when they match the expected size and hash.

def copy_file(src: Path, dst: Path, expected_hash: str | None, verify_hashes: bool) -> str:
    dst.parent.mkdir(parents=True, exist_ok=True)
    if dst.exists():
        if dst.stat().st_size != src.stat().st_size:
            raise RuntimeError(f"Existing destination has different size: {dst}")
        if verify_hashes and expected_hash and sha256_file(dst) != expected_hash:
            raise RuntimeError(f"Existing destination hash mismatch: {dst}")
        return "skipped-existing"

    tmp = dst.with_suffix(dst.suffix + ".tmp")
    shutil.copy2(src, tmp)
    if verify_hashes and expected_hash and sha256_file(tmp) != expected_hash:
        tmp.unlink(missing_ok=True)
        raise RuntimeError(f"Copied hash mismatch: {src} -> {dst}")
    tmp.replace(dst)
    return "copied"

This may feel excessive at first. But market data is not something I want to treat as casually replaceable. It costs money, the request conditions matter, and vendor condition metadata can change. So the pipeline records organize, convert, and evaluate steps separately.

The first workflow was:

python common/data/databento_pipeline.py organize ... --verify-hashes
python common/data/databento_pipeline.py convert
python common/data/databento_pipeline.py evaluate

organize preserves the raw job folders. convert turns DBN/DBN.zst files into bronze Parquet. evaluate records row counts, anomalies, degraded dates, and storage footprint. Separating those steps makes later “the data looks strange” conversations much more precise.

The First Inventory

The first evaluation result gave me more reasons to be careful than to be excited.

Schema	Raw Files	Parquet Files	Rows
`mbp-1`	26	26	387,252,384
`ohlcv-1m`	1	1	5,484,713
`definition`	1,565	1,565	83,657

The total market-data root was about 18G: roughly 7.4G raw and 11G processed bronze Parquet. This is not a tiny toy dataset, but it is still small enough to iterate locally.

The OHLCV data passed the basic sanity checks:

invalid OHLC rows: 0
negative-volume rows: 0

The MBP data immediately showed why a cleaning policy would be needed:

crossed top-of-book rows: 859
negative-price rows: 78

Those numbers can look small compared with 387M rows. But when building execution realism or microstructure features, crossed books and negative prices can distort tail metrics. For that reason, MBP did not become a feature source immediately. It later became a separate cleaning and execution-calibration track.

Keeping Degraded Dates Visible

The Databento condition metadata also produced a degraded-date list:

This list became important later in the silver dataset and training matrix. Degraded dates can be dropped or retained with flags, but the choice must be explicit. In this system, the silver dataset excludes degraded rows and records that decision in the report.

This kind of record is what determines research quality. Model performance can come later. But if provenance collapses, even good performance cannot be explained. On the other hand, if the data boundary is boringly clear from the beginning, later strategy failures become easier to diagnose.

The Result Of This Phase

The Databento ingestion phase did not produce alpha. There was no signal, no model, and no trading decision. What it produced was more foundational: raw-to-bronze paths, manifests, conversion reports, evaluation reports, degraded-date lists, and the first MBP anomaly surface.

That became the first real base of the quant trading system. Before writing a strategy, I needed to know the shape of the data. In futures data, parent symbols, instrument ids, contract rolls, and schema choices all change the meaning of a result. “I have ES/MES data” is not enough. I need to know which schema, which symbology, which period, and which quality state.

At the end of this phase, I still did not trust any trade candidate. That was a good start. A trustworthy strategy can only be built on top of trustworthy data boundaries.