Most infrastructure projects that are worth finishing begin as a vague discomfort before they become architecture. This one started with a very specific kind of frustration: I could watch Kubernetes react, but I could not always explain the timing before the damage became visible.

At the company where I was working, I spent a lot of time around EKS, Kubernetes traces, GitHub changes, kubectl output, HPA behavior, and Karpenter behavior. From far away the cluster could look calm, but the operational story underneath was rarely calm. Workloads pushed CPU, HPA reacted later than I wanted, Karpenter added capacity on its own cadence, pods sat Pending, and the evidence lived across dashboards, terminal output, and small fragments of event history.

The first version of this project was an attempt to turn that discomfort into a measurable system. Traces would go in, failure or demand risk would come out, agents would propose actions, a referee would select one, Kubernetes would react, and a dashboard would show the whole chain without hiding the uncomfortable parts.

The first shape of the idea

The earliest version of the project was much smaller than what it became. I wanted a failure forecaster. Given resource usage and terminal event information, could I score which tasks were likely to fail within a future window? That question was attractive because it was technical enough to be concrete, but close enough to Kubernetes operations to matter.

  • Borg traces would provide task and machine behavior over time.
  • Feature windows would summarize CPU, memory, request, priority, scheduling class, and recent deltas.
  • A target label would mark terminal failure within a fixed horizon.
  • A baseline model would expose the first obvious risk score.
  • Later, that risk score could become an input to an orchestrator instead of staying as a notebook metric.

The part I underestimated was the word later. A model alone is very easy to overvalue. If I only trained a classifier and printed AUCPR, the project would have stopped at a static ML experiment. The more I worked on it, the more I cared about the path from model output to control decision. That is why the project eventually became dashboard-heavy. The dashboard was not decoration. It was the only way I could keep myself honest about what the system was actually doing.

Directory layout before modeling

I kept the large data outside the repository from the beginning. The Borg data was too large and too mechanical to belong in git, and I did not want generated parquet files mixed into the source tree. The repo became the code and documentation layer. The external directories became the raw, processed, model, and report layer.

mkdir -p ~/Documents/borg_data
mkdir -p ~/Documents/borg_processed
mkdir -p ~/Documents/borg_xgboost_workspace/{raw,processed,models,reports,runtime,config}

export BORG_DATA_DIR="$HOME/Documents/borg_data"
export BORG_PROCESSED_DIR="$HOME/Documents/borg_processed"
export BORG_XGBOOST_WORKSPACE="$HOME/Documents/borg_xgboost_workspace"

That split was a small decision, but it saved me later. Once I had baseline forecaster data, advanced XGBoost features, orchestrator runtime traces, Optuna reports, and dashboard screenshots, I needed to know which artifact belonged to which track. Otherwise every failed run would have left behind files with unclear meaning.

Why Borg traces instead of only Kubernetes metrics

I could have started from live Kubernetes metrics only. That would have been more immediately familiar: pod CPU, memory, HPA desired replicas, Pending pods, node readiness. But I wanted more than a reactive demo. Borg traces gave me a dataset where machine/task behavior could be processed offline, repaired, and replayed. They also forced me to think in terms of terminal events and prediction horizons, which is exactly the thing I felt was missing when watching Kubernetes react after the fact.

The first baseline command was intentionally boring. I wanted reproducible CLI entrypoints before I wanted clever architecture.

python scripts/make_dataset.py
python scripts/make_forecaster_dataset.py
python scripts/train_forecaster_baseline.py

The first target label

The first important label was simple on paper: mark a row positive if the task has a failure terminal event within the prediction horizon. The actual implementation was where the first real care was needed. It had to avoid terminal events that had already happened before the usage window ended, and it had to keep the time arithmetic explicit.

dataset = pl.scan_parquet(dataset_file(cluster_id))

frame = (
    dataset
    .sort(["collection_id", "instance_index", "end_time"])
    .with_columns([
        (pl.col("last_event_time") - pl.col("end_time")).alias("time_to_terminal_event_us"),
        pl.col("final_event_type").is_in(failure_event_types).alias("is_failure_terminal_event"),
    ])
    .with_columns([
        (
            pl.col("is_failure_terminal_event")
            & pl.col("time_to_terminal_event_us").is_not_null()
            & (pl.col("time_to_terminal_event_us") >= 0)
            & (pl.col("time_to_terminal_event_us") <= horizon_us)
        ).alias("failure_within_horizon")
    ])
)

This is the kind of code that looks plain, but it decides whether the entire project is meaningful. If the label leaks future state incorrectly, every later model and dashboard can look impressive while being nonsense. I had to keep reminding myself that the project was only as good as the boring label plumbing.

From prediction to a visible control question

At this point I was not thinking in terms of a final dashboard yet. I was thinking in terms of an experiment that could eventually answer questions like: if risk rises before a visible failure, what should the controller do? If demand is low, can efficiency actions happen without stepping on safety? If queue health degrades, should admission control override power savings? And if all of that happens, can I see the actual reasoning and not just the final number?

The Kubernetes operating friction came first. The model was my first attempt to make that frustration measurable. The dashboard came later because I eventually stopped trusting invisible experiments.

That shift turned an operational feeling into an engineering question. The project was no longer about whether I could train a model on a public trace dataset. It was about whether prediction could become a visible, inspectable control signal.

The forecaster idea became useful only after that trust boundary was implemented in the data path. With raw and generated artifacts separated, the next job was to join usage, event, and machine evidence without letting a successful Parquet write masquerade as correctness.

Joining usage, events, and machines

The first real dataset was a joined table. Usage rows carried the time-window behavior. Events carried terminal state and task lifecycle evidence. Machines added context. I used Polars because I wanted streaming scans over parquet instead of loading everything into memory and hoping for the best.

usage = pl.scan_parquet(str(usage_path))
events = pl.scan_parquet(str(events_path))
machines = pl.scan_parquet(str(machines_path))

dataset = (
    usage
    .join(events, on=["collection_id", "instance_index"], how="left", suffix="_event")
    .join(machines, on="machine_id", how="left", suffix="_machine")
    .collect(engine="streaming")
)

dataset.write_parquet(output_path)

This join was one of those points where I had to stay suspicious. A successful parquet write did not mean the dataset was correct. I checked row counts, positive label rates, missing machine ids, and whether terminal event timing still made sense after the join.

Why I did not jump straight to advanced features

I was tempted to build the complicated feature set immediately. Rolling windows, deltas, request ratios, priority encodings, cluster-level pressure, all of that was more interesting than verifying joins. But doing that too early made every later error harder to isolate. So I kept the first track intentionally plain: produce the joined dataset, produce a forecaster frame, train a baseline, inspect the predictions.

The baseline was not meant to be the final model. It was a sanity instrument. If the baseline could not produce a plausible risk ranking, then the advanced model would only hide the failure behind more parameters.

The first baseline training loop

The baseline training script standardized the forecaster frames, split validation data, trained a model, and wrote the outputs I knew I would want later: validation predictions and top risk alerts. I cared about those files because they let me inspect actual rows instead of only reading aggregate metrics.

python scripts/train_forecaster_baseline.py   --clusters b,c,d,e,f,g   --feature-profile baseline

The useful outputs were not just the model file. The validation predictions parquet gave me a ranked surface to inspect. The top-risk alerts parquet gave me the first version of what later became the control-plane idea: risk is more useful when it can be attached to an action candidate.

  • validation_predictions.parquet: scored validation rows, sorted later by risk_score.
  • top_risk_alerts.parquet: high-risk rows kept for inspection.
  • cluster forecaster parquet files: per-cluster feature/label frames.
  • metrics text and reports: enough context to compare later runs.

The first friction-heavy trial-and-error loop

The difficult part was that every small pipeline change could invalidate something downstream. I would fix an optional column, then a cluster would produce different feature coverage. I would change the label horizon, then the positive rate would move. I would repair a schema issue, then model results would look better, but I still had to check whether the improvement was real or just caused by a different filtered dataset.

This is where I started building the habit that carried through the rest of the project: every artifact needed to be inspectable. A model score alone was not enough. A dashboard alone was not enough. A parquet file without a local explanation was not enough. I wanted the project to be explicit enough that I could keep moving without fooling myself.

The first reliable measuring tool

The completed baseline path could read the joined Borg-derived datasets, build forecaster frames with a 15-minute-style failure target, train a baseline forecaster, and write predictions I could inspect. It was still not the orchestrator. It was the first reliable measuring tool.

That measuring tool also exposed the next problem: schema drift and broken labels. At that point the pipeline stopped being a clean sequence and became a repair project.

The result was not a clever model. It was a workspace that made later experiments accountable—the kind of unglamorous foundation that quietly determines whether every later claim is defensible.