[Quant Trading System 2/5] From an MES/ES Silver Dataset to a Strict Training Matrix

Raw Futures Data Is Not A Feature Matrix

After converting Databento GLBX data into bronze Parquet, the data was still not ready for baselines or models. Futures data requires more decisions than an equity daily-bar dataset. Even when the symbols are called ES and MES, the files contain many expiring contracts, and a parent-symbol request does not automatically create the continuous series I want.

So the first real dataset task in the quant trading system was to build a silver OHLCV dataset. The goal was simple: treat MES as the tradable instrument, join same-minute ES as the reference instrument, and choose which MES/ES contracts to use on each trade date with an explicit rule.

The important part was not handcrafting the most attractive continuous contract. The rule had to be past-only, auditable, and frozen before looking at validation or test results. The first rule I chose was the previous available regular-session volume leader.

Trading instrument: MES
Reference instrument: ES
Source schema: Databento ohlcv-1m
Contract selection: previous available regular-session volume leader
Session filter: 10:00 to 15:30 US/Eastern
Label: net forward 5-minute MES close-to-close return
Splits: chronological by trade date, 70% train, 15% validation, 15% test

This rule is not perfect. But it satisfies the conditions for a good first rule. It does not use future volume, it can be reproduced from definition metadata and OHLCV data, and it leaves an artifact showing exactly which contract was selected on each date.

Making The Dataset Contract Explicit In DuckDB

The silver builder stacks several DuckDB views. It first keeps outright ES/MES futures from the definition data, then selects the latest definition row for each instrument_id. After that, it joins definition metadata into OHLCV rows and computes trade dates and times in America/New_York.

con.execute(
    """
    create or replace temp view rth_bars as
    select
      *,
      trade_date in (select trade_date from degraded_dates) as is_degraded_date
    from ohlcv_enriched
    where time_et >= TIME '10:00:00'
      and time_et <= TIME '15:30:00'
      and high >= open
      and high >= close
      and high >= low
      and low <= open
      and low <= close
      and low <= high
      and volume >= 0
    """
)

This code is not flashy, but it is one of the most important parts of the research pipeline. Before model features, there must be a dataset contract. The code has to say which time window is used, which rows are removed, and which quality checks are applied. Otherwise, when results improve later, I cannot separate a better signal from a changed data selection.

Why I Started With A Narrow Regular Session

The first trading window is 10:00 to 15:30 US/Eastern. It avoids the first 30 minutes after the U.S. equity open and the last 30 minutes before the close.

That choice is conservative. The open and close often have stronger volatility and more fragile execution assumptions. Of course, some strategies may find edge precisely there. But for the first dataset, stable measurement mattered more than signal hunting. I would rather validate the pipeline inside a narrower window than make the first result depend on the most unstable parts of the session.

I also built a separate session calendar. The silver report distinguishes full regular sessions, exchange-holiday partial sessions, early closes, and vendor anomalies. The later strict training matrix keeps only complete regular sessions. That makes holiday and early-close handling an explicit filter instead of a hidden loss of data.

First Silver Dataset Results

The first silver OHLCV dataset writes three main artifacts:

ohlcv_1m_volume_leader_v1.parquet
contract_selection_v1.parquet
dataset_manifest_v1.json

The summary looked like this:

Metric	Value
Rows	418,644
Trade dates	1,284
UTC range	2021-05-18 to 2026-05-15
Degraded rows	0
Missing ES reference rows	2
Incomplete session rows	7,873
Incomplete session dates	43
Labeled rows	412,214
Average net forward 5m return	-0.00019931

Contract coverage was another important check:

Root	Selected trade dates	Selected contracts
ES	1,284	21
MES	1,284	21

This result did not mean “the model can run now” as much as “I can now explain what the model would see.” I could identify which ES/MES contracts were selected, where the roll dates were, and why rows without forward labels were still retained.

Labels Are Experimental Conditions, Not Truth

The silver label is net forward five-minute MES close-to-close return. It records cost-adjusted net return alongside gross return, and the cost assumption is written into the report.

{
  "mes_contract_multiplier_usd_per_point": 5.0,
  "mes_fee_per_side_usd": 0.62,
  "mes_tick_size_points": 0.25,
  "slippage_ticks_round_turn": 2.0,
  "spread_ticks_round_turn": 1.0,
  "version": "ibkr_mes_retail_v1"
}

This label is not truth. It is not an actual fill, broker state, or queue-position simulation. But at the research stage, it is a clear experimental condition. It lets deterministic rules and future models be compared under the same data and cost assumption.

The silver dataset use notes therefore carry these principles:

Features use only current and earlier completed one-minute bars.
Labels use future five-minute MES closes and must not be used as features.
Holiday and early-close sessions are retained with explicit flags.
MBP fields are intentionally absent until the anomaly report is reviewed.

The last line mattered. MBP data already existed, but I did not force it into the silver OHLCV dataset. MBP needed anomaly review and a cleaning policy, and its coverage was not yet suitable for model features. Having data does not mean every field should become a feature.

The Model Boundary Left By Silver Data

Building the silver dataset made one thing very clear: a large part of quant research happens before modeling. Which contract should be selected? Which session should be used? Which dates should be excluded? Which labels should be generated? Where should costs be subtracted? How long should the test split stay locked? These decisions define the meaning of model performance before a model exists.

There was still no good strategy, but I could now explain what MES and ES meant inside the dataset: a past-volume-selected contract pair from parent-symbol data, one-minute bars inside the 10:00-15:30 ET window, degraded dates excluded, and incomplete sessions retained as explicit flags.

That kind of boring definition does not guarantee a good result. It provides the tools to distrust one that looks too good. But an explainable silver row is not automatically a row a model should see. The next boundary was not running a model; it was locking the columns and rows the model must never be allowed to consume.

The Work Before Model Training

Once the silver OHLCV dataset existed, it was tempting to train a model immediately. MES and ES were aligned, forward-return labels existed, and train/validation/test splits were already assigned. It looked ready for XGBoost or LightGBM. But I deliberately stopped there.

Before training a model, I needed to freeze the feature-matrix contract. Which columns are features? Which columns are labels? Which columns are metadata that must never enter the model? Where is split information stored, and how is the test split kept locked? Without those boundaries, a better model score could mean real signal, or it could mean that labels or future metadata accidentally leaked into the feature set.

So the next step in the quant trading system was the strict training matrix. The name means that only model-ready rows survive, but the more important meaning is that columns not meant for the model are physically separated.

Physically Separating Features, Labels, Metadata, And Splits

The training matrix builder writes one wide matrix, but it also writes four separated artifacts.

training_matrix_v1.parquet
features_v1.parquet
labels_v1.parquet
metadata_v1.parquet
splits_v1.parquet
training_matrix_manifest_v1.json

row_id is the stable key used to join those files back together. This is a little inconvenient, but it reduces mistakes. If model-training code reads only features_v1.parquet, it is much harder to accidentally include labels or metadata as inputs.

The boundary is explicit in code:

FEATURE_COLUMNS = (
    "minute_of_day_et",
    "session_progress",
    "mes_return_1m",
    "mes_return_5m",
    "mes_return_15m",
    "mes_return_30m",
    "mes_rsi_14m",
    "mes_atr_14m_points",
    "mes_vwap_session_diff_points",
    "es_return_5m",
    "es_rsi_14m",
    "mes_es_return_5m_spread",
)

LABEL_COLUMNS = (
    "gross_forward_5m_points",
    "net_forward_5m_points",
    "daily_net_forward_5m_rank",
    "is_top_5_daily_opportunity",
)

The real feature list is longer. It includes Bollinger-style z-scores, RSI, ATR, stochastic oscillators, VWAP distance, realized volatility, rolling volume normalization, SMA distance, price-action shape, and ES-reference divergence. But the main point is not the number of features. It is that the feature/label boundary is closed in code.

The Strict Filter

Not every silver row enters the matrix. Rows are removed if they lack a forward label, lack an ES reference, fall on degraded dates, come from incomplete sessions, or contain null feature/label values.

create or replace temp view strict_rows as
select
  row_number() over (order by ts_event_utc) as row_id,
  ...
from read_parquet(...)
where has_forward_5m_label
  and has_es_reference
  and not is_degraded_date
  and is_complete_regular_session
  and all feature columns are non-null
  and all label columns are non-null

This filter is conservative. It reduces coverage and removes holidays and early closes. But in the first training matrix, clarity mattered more than coverage. I can study incomplete sessions separately later. I did not want the first model or baseline to silently mix those edge cases into the main experiment.

The strict matrix ended up with:

Metric	Value
Rows	331,341
Trade dates	1,241
MES contracts	21
ES contracts	21
Train rows	232,017
Validation rows	49,128
Test rows	50,196

The important thing is not just that test rows exist. It is that they are not used yet. The test split is present in the matrix, but baseline and diagnostic tooling exclude it by default. I wanted a structure where the test set exists but remains closed.

Opportunity Ranking As The Target Shape

I did not want the first model target to be a simple directional classifier. The more important question was whether a minute was one of the few moments worth trading that day.

That is why the labels include daily ranking and top-k opportunity flags, not only forward return.

gross_forward_5m_points
gross_forward_5m_return
net_forward_5m_points
net_forward_5m_return
clears_min_net_return
daily_net_forward_5m_rank
is_top_5_daily_opportunity

This target design shaped the later candidate policy and walk-forward baseline. Even if a model scores every minute, the trading decision should happen only a few times per day. That makes “is this row one of today’s best opportunities?” more important than “is this row slightly positive?”

This also matches real execution better. A dataset may contain many rows with slightly positive five-minute returns, but after costs and slippage most are not worth trading. A good strategy should not find many positive rows. It should reject almost everything and keep only a few sufficiently good rows.

Feature Engineering Versus Feature Audit

It is true that this stage added many features. But the goal was not to build an impressive alpha library. It was closer to creating a minimum feature surface that made baselines and audits possible.

For example, the matrix includes MES short-term returns, RSI, ATR, Bollinger z-score, and ES-reference divergence. MES is the traded instrument, while ES is the larger reference market. To test whether the five-minute return spread or RSI spread matters, the first requirement is a stable same-minute join.

But more features also mean more leakage risk. So the important work at this stage was not only generating features, but making past-only behavior auditable. A later lookahead audit recomputed selected feature families from raw OHLCV prefixes and compared them against stored full-run values. The sampled mismatch count was 0.

The training matrix became the foundation for that audit.

Why I Still Did Not Train A Model

At this point, I could have trained a model. The matrix existed, features existed, labels existed, and splits existed. But “can” is not the same as “should.” Deterministic baselines were not good yet, cost and execution realism had not been fully examined, and no validation-looking signal had passed train and robustness gates.

So the first conclusion of the strict training matrix was not model training. It was deferring model training.

I think that deferral matters. In ML projects, once a training pipeline exists, running the model feels natural. In trading, the model gives convincing answers too quickly. If features are numerous and labels are noisy, some combination will look good on validation. Before trusting that combination, the data contract and baseline gates need to exist.

The strict training matrix was therefore both “data for the model” and “data that prevents premature modeling.” It made model training possible, while making the decision not to train yet more explicit.

The Standard This Phase Left Behind

After this phase, every new feature or candidate in the quant trading system had a standard to satisfy. Features must be listed in FEATURE_COLUMNS. Labels must not mix into features. Metadata is for explanation and audit, not model input. Splits are chronological by trade date. The test split exists but must not be used for selection or tuning.

This adds code. It is more annoying than writing one wide matrix and selecting columns casually. But that annoyance makes the research faster later, because when a result looks strange, the search space for possible mistakes is smaller.

Good trading ML does not begin with model architecture. At least in this project, it began by deciding which rows and columns the model must never be allowed to see.