Once the live loop existed, the problem shifted from model performance to control behavior. A controller can be accurate in one narrow sense and still behave badly if the reward function teaches it the wrong priority.

That is why Optuna and Ray/RLlib entered the project. I did not want tuning to remain invisible. If reward weights changed, the dashboard needed to show the trial. If RL was too expensive to keep in the fast loop, that decision also needed to be visible. The experiment had to preserve the reasoning trail, not only the final chosen parameters.

This phase was about making optimization accountable. The question was not simply which configuration wins. It was whether the search process itself could be inspected well enough to trust the behavior it produced.

The reward problem

The orchestrator had three competing instincts: keep tasks alive, reduce waste, and protect the queue. I represented those as Agent A, Agent B, and Agent C rewards. The combined score used weights alpha, beta, and gamma. At first that felt too simple, but it was useful because I could see how changing the weights changed controller personality.

@dataclass(slots=True)
class Score:
    raw_rewards: dict[str, float]
    alpha: float = 1.0
    beta: float = 1.0
    gamma: float = 1.0

    @property
    def total(self) -> float:
        return (
            self.alpha * self.raw_rewards.get("AgentA", 0.0)
            + self.beta * self.raw_rewards.get("AgentB", 0.0)
            + self.gamma * self.raw_rewards.get("AgentC", 0.0)
        )

I did not treat this as a perfect reward design. It was a practical controller surface. If Agent A dominated everything, the system became safety-heavy. If Agent B was too attractive, the controller could chase efficiency while queue health was poor. If Agent C was too strong, admission behavior could become overly defensive.

Optuna as a visible search process

Optuna gave me a way to tune reward weights and policy parameters without pretending I had discovered perfect constants by hand. The important part was exporting trial history and reflecting it in the dashboard. I wanted to see completed trials, best values, and selected parameters as part of the runtime, not as a separate notebook I would forget to update.

study = optuna.create_study(
    direction="maximize",
    storage=storage,
    study_name=study_name,
    load_if_exists=True,
)

state.optuna_history(
    study_name,
    history,
    best_value=study.best_value if study.best_trial else None,
    best_params=dict(study.best_params) if study.best_trial else {},
    status="running",
)

The trial values were less important than the workflow. A controller experiment should make its tuning state visible. If I publish a dashboard screenshot where Optuna is disabled, the reader should know it was a fast run. If I publish one where Optuna completed trials, the dashboard should show the best parameters and history.

Ray/RLlib was useful but expensive to keep in the loop

Ray/RLlib gave me a more formal multi-agent policy path. The environment exposed AgentA, AgentB, and AgentC as separate agents with shared observation vectors and separate action spaces. That matched the architecture well, but local development made me careful. PPO bootstrap can slow down iteration, so I often kept it disabled while debugging live Kubernetes behavior.

self.possible_agents = ["AgentA", "AgentB", "AgentC"]

self.observation_spaces = {
    "AgentA": Box(low=0.0, high=1.0, shape=(6,), dtype=float),
    "AgentB": Box(low=0.0, high=1.0, shape=(6,), dtype=float),
    "AgentC": Box(low=0.0, high=1.0, shape=(6,), dtype=float),
}

self.action_spaces = {
    "AgentA": Discrete(POLICY_SPACES["AgentA"].action_count),
    "AgentB": Discrete(POLICY_SPACES["AgentB"].action_count),
    "AgentC": Discrete(POLICY_SPACES["AgentC"].action_count),
}

Why the dashboard mattered here

Tuning without dashboard state felt like changing knobs in a dark room. The learning/reward view let me see whether reward history, current action, agent proposals, Optuna state, and Ray status agreed with the run mode. When something looked off, I could tell whether I had a tuning problem, a disabled-feature problem, or a live data problem.

This phase also made me less interested in claiming one magic policy. The project was more interesting as a visible control experiment. Sometimes I wanted the deterministic agents and referee because they were readable. Sometimes I wanted Optuna search. Sometimes I wanted RLlib policy bootstrapping. The dashboard needed to show which one was active.

From reward tuning to comparative evidence

The system could now expose reward trajectories, optimization state, and policy state alongside live controller decisions. Reward tuning forced every objective to compete with another objective and made the tradeoffs visible instead of hiding them inside a single score.

That visibility made the controller easier to inspect, but the experimental side was still being evaluated in isolation. To turn optimization into evidence, I needed the same external pressure applied to a recognizable HPA plus Karpenter-like baseline, with a clear boundary around what the local comparison could claim.

How I launched the comparison

I used two Kind clusters: borg-experimental and borg-baseline. The experimental side received the orchestrator loop and bounded Agent A/B/C remediation. The baseline side received HPA and a local warm-node activation controller that approximated Karpenter-like behavior inside Kind. I kept the interpretation boundary visible because local Kind warm-node activation is not AWS Karpenter.

OPEN_BROWSER=0 ./orchestrator_stack/scripts/start_local_dual_cluster_stack.sh

PYTHONPATH=orchestrator_stack .venv/bin/python -m orchestrator.dashboard_server   --port 8765   --event-dir orchestrator_stack/runtime/visualization-experimental

PYTHONPATH=orchestrator_stack .venv/bin/python -m orchestrator.comparison_dashboard_server   --port 8876   --experimental-kubeconfig ~/Documents/borg_orchestrator_clusters/kubeconfig-experimental   --baseline-kubeconfig ~/Documents/borg_orchestrator_clusters/kubeconfig-baseline   --experimental-event-dir orchestrator_stack/runtime/visualization-experimental

The comparison became much more meaningful after I applied the same style of pressure to both clusters. The captured dashboard came from a severe admission-cap style stimulus: 130 replicas, explicit CPU and memory requests, and an intentionally constrained scheduling situation. I wanted the dashboard to show backlog, not just happy-path autoscaling.

EXPERIMENTAL_KUBECONFIG=$HOME/Documents/borg_orchestrator_clusters/kubeconfig-experimental BASELINE_KUBECONFIG=$HOME/Documents/borg_orchestrator_clusters/kubeconfig-baseline PHASE_INDEX=3 EXERCISE_RANDOMIZE=1 EXERCISE_SEED=27 ./orchestrator_stack/scripts/apply_comparison_stimulus.sh

Interpreting the top dashboard

The top comparison view is dense, but that is why I like it. It shows the experimental latest decision, the baseline HPA state, local Karpenter active/warm nodes, and the direct pressure comparison. In the captured run the baseline HPA is maxed at 1/1 replicas with CPU far above the target, while the baseline still has a large Pending backlog. The experimental side is not magically free of pressure, but it is visibly handling the shared stimulus differently.

The most important number in the screenshot is not a single reward score. It is the backlog gap: experimental pending 3 vs baseline pending 130. The dashboard labels this as experimental better by 127, which is much more readable than showing a raw delta and forcing me to remember which direction is good.

06_comparison_objectives_timeline.png

Why I added semantic comparison labels

At first the comparison dashboard was basically a metric pile. That was not enough. Lower pending pods is better. Lower restarts is better. Lower dynamic power is usually better under the same controlled stimulus. Higher ready workers can be better when schedulable capacity matters. A raw negative delta does not tell the reader any of that.

So I made the dashboard speak in terms like experimental better, baseline ahead, matched behavior, and objective evidence. It still exposes the numbers, but the UI tells me the direction of the claim. That matters because the dashboard is part of the argument.

specs = [
    ("Ready workers", ("ready_workers",), "Higher means more schedulable worker capacity is available."),
    ("Pending pods", ("pending_pods",), "Higher pending count means queue/backlog pressure."),
    ("Restarts", ("pod_summary", "restarts"), "Container restarts indicate instability or churn."),
    (
        "Controlled dynamic power W",
        ("controlled_resource_totals", "estimated_power_watts"),
        "Controller-relevant utilization-derived dynamic power, excluding node idle/control-plane noise.",
    ),
]

07_comparison_agent_goal_matrix.png

The Agent Goal Matrix was my answer to a problem I kept running into: the experimental controller and the baseline are not optimizing the same internal objective. HPA is reactive scaling logic. Karpenter is capacity provisioning. Agent A/B/C is an explicit multi-objective controller. The dashboard needed to compare observable outcomes while still saying which objective each agent was responsible for.

08_comparison_pressure_charts.png

Controlled resource totals

One repair I made here was narrowing the energy/resource comparison to controlled namespaces. If I included every bit of cluster noise, the dashboard could accidentally compare observability overhead or control-plane background work. That would make the energy story weak. The comparison dashboard therefore tracks controlled CPU, memory, requests, and utilization-derived dynamic watts for the shared comparison and exercise namespaces.

10_comparison_full_page.png

Full comparison dashboard capture. I kept the full page because the top cards, objective evidence, timelines, agent matrix, controller narrative, and interpretation boundary are meant to be read together.

What the comparison proved—and the gap it exposed

This local comparison did not prove that my controller beats production EKS HPA plus real AWS Karpenter. That would be a much bigger claim and would require real cloud runs. What it did prove for me was narrower and still useful: I could run two local clusters under shared pressure, collect live metrics from both, apply bounded experimental remediation on one side, and show the behavioral difference in a dashboard.

That was the point where the project finally matched the frustration that started it. Pressure, controller decisions, and baseline behavior were visible at the same time instead of being reconstructed from HPA and Pending pods after the fact.

Side-by-side behavior, semantic labels, and controlled resource totals replaced vague confidence with evidence. They also exposed the final trust gap: a recommendation in the event stream was not proof that Kubernetes had been changed. Closing the loop required a deliberately narrow executor and a dashboard that distinguished proposal, attempted mutation, and observed outcome.

Bounded action execution

The executor records the command, return code, stdout, and stderr for every operation. I wanted this because silent mutation is dangerous. If the controller claims it scaled or capped something, I need the action trail in the decision payload.

def _record_operation(
    kubeconfig: str | Path,
    operations: list[dict[str, Any]],
    description: str,
    args: list[str],
) -> None:
    completed = _run_kubectl(kubeconfig, args)
    operations.append({
        "description": description,
        "command": "kubectl " + " ".join(args),
        "returncode": completed.returncode,
        "stdout": completed.stdout.strip(),
        "stderr": completed.stderr.strip(),
    })

Scaling and resource changes are similarly bounded. The executor discovers deployments only by the orchestrator exerciser label, then applies a small set of allowed changes. It can scale exercise deployments, cap a comparison load generator, or restart controlled work. It is not a general-purpose cluster automation tool, and I prefer it that way.

def execute_live_kubernetes_action(
    action: AgentAction,
    kubeconfig: str | Path,
    *,
    namespace: str = DEFAULT_EXERCISE_NAMESPACE,
    workload_namespace: str = DEFAULT_WORKLOAD_NAMESPACE,
) -> dict[str, Any]:
    names, discovery_error = _deployment_names(kubeconfig, namespace)
    operations: list[dict[str, Any]] = []
    result: dict[str, Any] = {
        "status": "observed",
        "namespace": namespace,
        "agent": action.agent_name,
        "kind": action.kind.value,
        "target": action.target,
        "payload": dict(action.payload),
        "matched_deployments": names,
        "workload_namespace": workload_namespace,
        "operations": operations,
    }
    if discovery_error:
        result["status"] = "error"
        result["error"] = discovery_error
        return result
    if not names or action.kind == ActionKind.NOOP:
        result["status"] = "no_targets" if not names else "noop"
        return result

Attaching execution to the dashboard state

The live loop attaches the execution result to the decision payload. This changed the meaning of the dashboard. A decision could now show not just AgentA:replicate, but also whether the bounded Kubernetes action was attempted and what kubectl returned.

if exercise_cluster:
    decision_payload["kubernetes_execution"] = execute_live_kubernetes_action(
        action,
        kubeconfig_path,
        namespace=exercise_namespace,
    )

state.decision(decision_payload)

This made the dashboard less clean, but more honest. Failed operations, no target matches, and no-op states are part of the experiment. I would rather show that mess than publish a controller story that hides whether anything happened.

Energy limits

Energy was the easiest metric to overclaim, so I kept the boundary explicit. The project uses a utilization-derived dynamic power estimate, not a physical wattmeter. It is useful for comparing controlled workload pressure under the same local model. It is not a claim about exact machine power consumption.

@dataclass(frozen=True, slots=True)
class PowerCalibration:
    idle_watts: float = 80.0
    cpu_full_scale_watts: float = 120.0
    mem_full_scale_watts: float = 60.0
    source: str = "default_utilization_model"

def estimate_node_power_watts(
    cpu_util: float,
    mem_util: float,
    calibration: PowerCalibration | None = None,
) -> float:
    calibrated = calibration or DEFAULT_POWER_CALIBRATION
    return (
        calibrated.idle_watts
        + (calibrated.cpu_full_scale_watts * _bounded_ratio(cpu_util))
        + (calibrated.mem_full_scale_watts * _bounded_ratio(mem_util))
    )

The comparison dashboard also separates controlled dynamic power from whole-cluster background noise. That was important because otherwise I could accidentally make the experimental side look worse or better because of unrelated observability or control-plane activity.

What Finally Remained

The final local system has Borg-derived features, XGBoost risk and demand models, a six-layer orchestrator, Agent A/B/C proposals, a deterministic referee, optional Optuna and Ray/RLlib adaptation, live Kubernetes exercise loops, a dual-cluster comparison setup, and dashboards that show the moving parts. It is not a production autoscaler. It is a personal cloud-systems research rig that lets me ask more precise questions than I could ask by staring at HPA events alone.

The most useful thing I got from the project was not a single metric. It was a workflow: create pressure, collect live state, let the controller propose, show the conflict, compare against a baseline, and keep the interpretation boundary visible. That workflow is why the dashboards became the main part of the project.

If I continue this later, the next version should run longer repeated experiments, preserve more dashboard snapshots per run, and eventually move the comparison from Kind to EKS with real Karpenter. For now, the local version is complete enough for me to explain the whole path from Kubernetes frustration to a live orchestration experiment.

This final phase made the project complete enough to explain without overclaiming. It showed where prediction ended, where decision began, where Kubernetes was actually touched, and where measurement had to stay humble. That boundary is the difference between a polished demo and an engineering artifact I can stand behind.