The first time the pipeline looked stable, I did not trust it. That instinct was useful. Stable output is not the same thing as correct output, especially when the data is large enough to hide small mistakes.

The dangerous failures were not the loud crashes. Loud crashes stop the run and force attention. The failures that worried me were quieter: nullable fields interpreted too casually, terminal events attached to the wrong temporal direction, schema changes that still produced parquet files, and labels that looked plausible while poisoning the training set.

This phase was about treating data repair as part of the system, not as cleanup after the real work. If the project was going to make decisions from model output, label integrity had to become an explicit engineering concern.

Schema drift as a real error

The orchestrator side eventually had to ingest grouped traces, synthetic traces, and live Kubernetes snapshots. That made schema validation more important than I expected. I wrote validators that treated drift as a named problem instead of letting Python fail somewhere later with a vague KeyError or TypeError.

def _parse_int(value: Any, *, field: str, row_index: int) -> int:
    try:
        return int(value)
    except (TypeError, ValueError) as exc:
        raise ValueError(
            f"Schema drift: row[{row_index}] field {field!r} must be integer-like, got {value!r}."
        ) from exc

def _validate_grouped_row(row: dict[str, Any], row_index: int) -> None:
    if "timestamp" not in row:
        raise ValueError(f"Schema drift: grouped row[{row_index}] missing timestamp key.")
    _parse_int(row["timestamp"], field="timestamp", row_index=row_index)

That style made the code a little less pretty, but it made failures readable. When I was running long local experiments, readable failure messages mattered more than elegance.

The label bug I kept checking for

The most important label check was temporal. A row should not become positive because of a terminal event that had already happened before the usage window ended. That sounds obvious, but when the dataset is built from separate usage and event sources, it is easy to accidentally make future knowledge leak into the feature row.

The repair was to keep the time-to-terminal calculation visible and to preserve a separate flag for terminal events that were already before the window end.

(pl.col("is_failure_terminal_event")
 & pl.col("time_to_terminal_event_us").is_not_null()
 & (pl.col("time_to_terminal_event_us") >= 0)
 & (pl.col("time_to_terminal_event_us") <= horizon_us)
).alias("failure_within_horizon")

(pl.col("final_event_type").is_not_null()
 & (pl.col("last_event_time") < pl.col("end_time"))
).alias("terminal_event_before_window_end")

I did not want to delete that second flag just because it was not the training target. It was useful during debugging because it gave me a way to see when terminal state was attached to a row in a suspicious way.

Repair reports, not just repaired code

Another thing I changed in this phase was how I recorded progress. If I only fixed scripts and moved on, I would forget why a repair existed. The reports in the repository became a lightweight lab notebook: what was broken, what was repaired, what remained risky, and which generated artifacts were expected after the next run.

That habit helped later when the project split into several tracks: baseline forecaster, advanced XGBoost, orchestrator stack, Optuna/Ray tuning, local dual-cluster comparison, and dashboard work. Without progress reports, those tracks would have blurred together.

The repair loop was slow but necessary

The least fun part was re-running work after a repair. A schema fix could force regenerated parquet. A label fix could force retraining. A feature change could make older metrics no longer comparable. It felt slow, but it was better than building a dashboard on top of a rotten label definition.

I had a simple rule during this part: if I could not explain what a row meant, I did not want to train on it. That rule sounds a bit dramatic, but it kept the project grounded. Later, when I watched dashboards show risk, queue length, decisions, and reward traces, I knew those values came from contracts I had already fought with.

The repair changed the trust boundary

After the repair pass, I was more comfortable moving to advanced modeling. The target label had a clearer temporal meaning, grouped trace ingestion had stricter validation, and artifact layout was less ambiguous. I still expected problems, but the project was no longer held together by hope and print statements.

The project now felt less like a one-off classifier and more like a systems experiment. The data contract, not the model, had become the center of gravity. A successful run had to produce not only data, but evidence that the data deserved to be used.

Only then did advanced modeling become worth attempting. The next risk was no longer silent corruption; it was overtrusting a model whose validation metric looked good but whose output had not yet acquired an operational meaning.

Multi-horizon thinking

The first baseline target was basically: failure within the configured horizon. The advanced track expanded that idea. In a controller setting, the horizon matters. A risk that appears very near-term should not be interpreted the same way as a weaker, longer-horizon risk. I wanted the model artifacts and reports to keep that distinction visible.

python scripts/build_advanced_xgboost_dataset.py --clusters b,c,d,e,f,g
python scripts/train_advanced_xgboost.py --clusters b,c,d,e,f,g

The important part was not the command itself. The important part was isolating the advanced workspace so the generated features, tuned models, and reports could evolve without overwriting the baseline path.

The model code stayed intentionally plain

For the orchestrator-side XGBoost brains, I kept the training function compact. Risk was binary. Demand was regression. Both used histogram tree building and saved model files that the live loop could load later.

def train_safety_model(x: np.ndarray, y: np.ndarray, out_path: str | Path) -> Path:
    xgb = _require_xgboost()
    dtrain = xgb.DMatrix(x, label=y)
    params = {
        "max_depth": 6,
        "eta": 0.06,
        "subsample": 0.9,
        "colsample_bytree": 0.9,
        "objective": "binary:logistic",
        "eval_metric": "aucpr",
        "tree_method": "hist",
    }
    booster = xgb.train(params=params, dtrain=dtrain, num_boost_round=300)
    out = Path(out_path)
    out.parent.mkdir(parents=True, exist_ok=True)
    booster.save_model(str(out))
    return out

This was not the most exotic part of the project, but it was one of the most important. I needed a model path that was easy to reproduce and easy to inspect. The orchestration system would already be complex enough. I did not want the model-loading layer to become another mystery.

Feature importance was useful, but not enough

Feature importance reports helped me check whether the model was paying attention to plausible signals: utilization, requests, rolling deltas, scheduling and priority fields, and machine-level context. But I was careful not to treat feature importance as proof. It is a debugging lens, not a guarantee that the model will behave well inside a controller.

The more useful question became: can the model produce risk and demand values that are stable enough for an agent to reason about? A model can have an acceptable validation metric and still behave poorly inside a control loop if its output jumps too much or if the thresholds create constant action flipping.

Thresholds became part of the architecture

The agent thresholds were simple at first, but they gave the risk model a behavioral meaning. A high risk score could become replicate. A medium risk score could become migrate or throttle. Low risk became no-op. That translation from probability to action was where the ML experiment started becoming an orchestration experiment.

node_id, score = max(obs.p_fail_scores.items(), key=lambda kv: kv[1])
if score >= 0.83:
    return AgentAction("AgentA", ActionKind.REPLICATE, target=node_id, score=float(score))
if score >= 0.7:
    return AgentAction("AgentA", ActionKind.MIGRATE, target=node_id, score=float(score))
if score >= 0.5:
    return AgentAction("AgentA", ActionKind.THROTTLE, target=node_id, score=float(score))
return AgentAction("AgentA", ActionKind.NOOP, score=float(score))

I changed my mind several times about these thresholds. If they were too low, the controller looked dramatic and noisy. If they were too high, the risk model became decorative. I eventually treated them as controller parameters rather than sacred ML outputs.

The awkward part of interpreting results

The advanced model reports were useful, but they did not answer the bigger systems question by themselves. AUCPR, precision at top-k, calibration bins, and feature importance all helped me decide whether the model was sane. They did not tell me whether a cluster would be better off when an agent used those scores.

That realization pushed the project toward the six-layer orchestrator. I needed a place where model outputs became observations, observations became proposals, proposals conflicted, and the dashboard showed the conflict. Without that, I would only have a pile of model files and a weak story.

From model scores to control signals

The advanced XGBoost path produced enough model infrastructure to feed a controller. The risk model and demand model were no longer isolated experiments; they were ready to become the brains layer in a larger system.

That larger system was where the project could become what I originally wanted while staring at Kubernetes: not just prediction, but visible control behavior.

The important move was from model training toward model interpretation. A metric can be read in isolation; an engineer still has to explain what it should and should not be allowed to decide.