The six-layer orchestrator was the point where the project stopped being a model pipeline and became a control-plane experiment. That change was important because scores alone do not operate systems. Decisions do.

A risk score is useful only if something knows how to consume it. A demand estimate matters only if it changes how capacity, admission, or safety tradeoffs are evaluated. Once I accepted that, the project needed boundaries: where raw observations enter, where models speak, where agents propose, where conflict is resolved, and where the UI explains the decision.

This phase was my attempt to make that control flow explicit enough to criticize. If the system made a bad decision, I wanted to know which layer produced the mistake instead of blaming a vague black box.

Reading this dashboard

This screenshot is close to the mental picture I wanted from the beginning. At the top, the dashboard shows the active controller state instead of making me dig through logs. In this run the max risk is 0.950 and the selected decision is AgentA:replicate against borg-experimental-worker2. That means the safety agent won the referee decision because the risk path crossed the high threshold.

The live orchestration flow is also important. It shows that the system is not one model call. The path is: Kubernetes cluster snapshot, workload exerciser stimulus, XGBoost risk/demand inference, Agent A/B/C proposals, referee selection, reward scoreboard, and event emission. I wanted this displayed because otherwise it is too easy to talk about the orchestrator as if it were a single black box.

Agent responsibilities

The agent split was intentionally blunt. Agent A is safety. Agent B is efficiency. Agent C is admission and queue pressure. I did not want one giant policy object that quietly mixed all objectives. Separate agents made conflict visible, and visible conflict made the dashboard more honest.

@dataclass(slots=True)
class AgentARiskMitigator:
    priority: int = 1

    def act(self, obs: Observation) -> AgentAction:
        if not obs.p_fail_scores:
            return AgentAction("AgentA", ActionKind.NOOP, score=0.0, priority=self.priority)
        node_id, score = max(obs.p_fail_scores.items(), key=lambda kv: kv[1])
        if score >= 0.83:
            return AgentAction("AgentA", ActionKind.REPLICATE, target=node_id, score=float(score), priority=self.priority)
        if score >= 0.7:
            return AgentAction("AgentA", ActionKind.MIGRATE, target=node_id, score=float(score), priority=self.priority)
        if score >= 0.5:
            return AgentAction("AgentA", ActionKind.THROTTLE, target=node_id, score=float(score), priority=self.priority)
        return AgentAction("AgentA", ActionKind.NOOP, score=float(score), priority=self.priority)

Agent B and Agent C were equally direct. Agent B looked for low demand and proposed power-state, DVFS, or memory balloon actions. Agent C looked at queue length, SLA pressure, and overloaded nodes, then proposed admission or resource caps. I liked this split because it matched the operational tension I had seen around Kubernetes: safety, efficiency, and admission often pull in different directions.

The referee was the uncomfortable part

Once the agents existed, I needed to decide what happened when they disagreed. I did not want silent winner-takes-all behavior. The referee had to produce a selected action and a rationale, plus an overridden map so the dashboard could show what lost and why.

SAFETY_ACTIONS = {ActionKind.MIGRATE, ActionKind.REPLICATE, ActionKind.THROTTLE}
EFFICIENCY_ACTIONS = {ActionKind.POWER_STATE, ActionKind.DVFS, ActionKind.MEMORY_BALLOON}
PROTECTIVE_ADMISSION_DECISIONS = {"queue", "reject", "deprioritize"}

if agent_a_safety is not None:
    return RefereeDecision(
        action=agent_a_safety,
        rationale=f"agent-a {safety_label} preempts lower-priority actions",
        overridden=overridden,
    )

if restrictive_admission is not None:
    return RefereeDecision(
        action=restrictive_admission,
        rationale="agent-c admission protection preempts efficiency actions",
        overridden=overridden,
    )

The referee policy is conservative on purpose. Safety can preempt efficiency. Admission protection can preempt non-safety actions. Efficiency gets a chance when the system is not screaming. This made the controller less flashy, but much easier to reason about.

Why the dashboard became central

At this point I realized the dashboard was not a final reporting layer. It was part of the debugging process. If Agent A was selected, I needed to see the risk score and target. If Agent B lost, I needed to see the demand estimate. If Agent C proposed admission control, I needed to see queue length. If the active stage was complete but the cluster was still unhealthy, I needed the dashboard to show that too.

That is why I kept adding state fields: current decision, proposal list, reward summary, stage progress, artifact list, Optuna history, Ray status, cluster snapshot, event stream. It made the UI dense, but the project needed density. A polished dashboard that hides the control path would not have served the experiment.

An inspectable architecture still needed reality

The basic six-layer stack could train or load XGBoost brains, run agents, resolve conflicts, score rewards, and emit dashboard state. It finally had the shape I wanted: a cloud-systems experiment where the model had to live inside a visible controller.

The stack made responsibility visible by separating prediction, proposal, arbitration, execution, and explanation into inspectable parts. That turned the project from a model demo into an engineering system, but it was still a controlled design. The next test was whether the same path could survive messy timing and incomplete signals from a live Kubernetes cluster.

The live loop

The live loop is where the control-plane pieces came together. It collects a Kubernetes snapshot, creates an Observation, asks each agent for a proposal, resolves the proposals through the referee, emits the decision, steps the backend, updates reward state, and then writes the dashboard state files.

proposals = [agent.act(obs) for agent in agents]
action = resolve(proposals)
reason = _decision_reason(snapshot, action.agent_name, action.kind.value)
action_label = _action_label(action)

decision_payload = {
    "agent": action.agent_name,
    "kind": action.kind.value,
    "target": action.target,
    "payload": dict(action.payload),
    "action_label": action_label,
    "score": float(action.score),
    "proposal_count": len(proposals),
    "proposals": [
        {"agent": p.agent_name, "kind": p.kind.value, "target": p.target, "score": float(p.score)}
        for p in proposals
    ],
}

state.decision(decision_payload)

The detail I cared about here was the proposal list. A dashboard that only shows the final action hides the conflict. When the final action is AgentA:replicate, I still want to know what AgentB and AgentC wanted. Otherwise I cannot tell whether the system was calmly aligned or whether safety just overrode everything.

Running the live Kubernetes path

The local run command became long because I wanted the loop to be explicit: which config, which event directory, which kubeconfig, how often to sample, whether to tune or skip tuning, and whether to apply exercise stimuli.

PYTHONPATH=orchestrator_stack .venv/bin/python orchestrator_stack/run.py live-kubernetes-run   --config orchestrator_stack/config/orchestrator.example.json   --event-dir orchestrator_stack/runtime/visualization-experimental   --kubeconfig "$HOME/Documents/borg_orchestrator_clusters/kubeconfig-experimental"   --interval-seconds 3   --max-iterations 5   --namespace-prefixes borg-orchestrator-exercise,borg-comparison-workload,default,test-   --trace-out orchestrator_stack/runtime/visualization-experimental/live_kubernetes_trace.json   --trials 3   --prometheus-base-url http://127.0.0.1:19090   --no-policy --no-tune   --exercise-cluster   --exercise-namespace borg-orchestrator-exercise   --exercise-interval-iterations 1   --exercise-randomize --exercise-seed 31

I often ran in fast mode with tuning disabled because I wanted to debug live behavior without waiting for policy training. That is why some dashboard captures show Ray and Optuna as disabled. That was not a bug in the dashboard. It was the run mode I selected to get lively Kubernetes state quickly.

04_experimental_full_page.png

What the full dashboard showed me

The full dashboard helped me catch mismatches between the story I wanted to tell and the state the system was actually producing. If the active stage was complete but the event log had no cluster samples, something was wrong. If reward changed but decisions did not, I needed to inspect the backend. If the exerciser was active but queue pressure stayed flat, the Kubernetes stimulus was probably not doing what I thought.

In the captured run, the event sequence is visibly not a single static recommendation. It moves through AgentA replicate, AgentB memory balloon proposals, AgentC admission/deprioritize behavior, and then another AgentA replicate as risk and SLA state change. That was the kind of liveness I wanted: not animation for its own sake, but traceable state changes.

The awkward live-data problems

Live Kubernetes data introduced boring but real problems. Metrics Server can lag. Prometheus port-forwards can fail. A Kind cluster can behave differently from EKS. Pending pods can be caused by deliberate unschedulable node selectors, resource pressure, or controller choices. I had to make the dashboard expose enough detail to interpret those cases instead of flattening everything into a single score.

That is why the dashboard kept both raw-ish cluster state and interpreted decision state. The raw state tells me what Kubernetes is showing. The decision state tells me what the orchestrator thinks it should do. The interesting debugging happens when those two disagree.

What live validation established

The live experimental dashboard could keep up with the local Kubernetes loop. It was not yet a fair baseline comparison, but it was the first time the project felt like a control system instead of a batch experiment.

The live path made the project harder to explain but easier to trust. It introduced messy timing, missing signals, and awkward state transitions, which is exactly why it mattered. Real systems rarely validate themselves under clean laboratory conditions.