Theo Kim - Backend & Cloud Systems Engineer

Scaling Healthcare Chat Beyond an Echo WebSocket

ystc1247@gmail.com — Sun, 26 Jul 2026 06:00:00 GMT

An echo server proves that a WebSocket frame can travel through one process. It does not answer the questions that make chat reliable: who authenticated the connection, whether an Origin is allowed, what happens when the sender and receiver reach different instances, when a message is acknowledged, how duplicates are handled, and what happens when a client cannot drain its socket.

I built a synthetic healthcare chat experiment with two application instances behind Nginx, Redis Streams for durable message records, Redis Pub/Sub for live cross-instance fan-out, message-ID deduplication, bounded payloads and rates, slow-consumer protection, and Prometheus metrics. The payloads are synthetic and contain no protected health information.

The integration check connects a sender directly to instance A and a receiver directly to instance B. It proves the message crosses the shared Redis path and that resending the same message ID does not append or broadcast it twice.

WebSocket changes the session, not all of HTTP

WebSocket begins as an HTTP Upgrade request and then becomes a persistent full-duplex connection. The server can send frames without waiting for a new request.

That does not mean ordinary HTTP or gRPC creates a new TCP connection for every request. HTTP keep-alive reuses connections, and HTTP/2 multiplexes streams; gRPC commonly runs on long-lived HTTP/2 connections. The distinctive WebSocket concern is the application session attached to a bidirectional socket.

Long-lived sessions change the operating model:

deploys and load-balancer timeouts disconnect clients;
authentication can expire while the connection remains open;
each instance holds connection state;
slow readers accumulate buffered data;
reconnects and duplicate sends are normal;
cross-instance delivery needs shared infrastructure.

The measured architecture

The local topology is:

browser / test clients
          |
      Nginx :18060
       /          \
  app-a :8080   app-b :8080
       \          /
        Redis :6379
       /          \
 Streams       Pub/Sub
 durable log   live fan-out

Nginx uses WebSocket Upgrade forwarding and balances new connections. Existing WebSocket connections stay on the instance that accepted them, so sticky sessions do not solve cross-instance messaging. The shared channel does.

The browser UI shows which instance accepted the connection and the Redis stream ID assigned before acknowledgement:

Authenticate before accepting application messages

The lab requires the first frame within five seconds:

{
  "type": "authenticate",
  "token": "local-lab-token",
  "consultationId": "demo-consultation"
}

An invalid or missing authentication frame closes the socket with policy code 1008. The consultation ID is syntax-validated and bounded before it becomes part of a Redis key.

The first-frame mechanism keeps the experiment small. In production, authenticate during the HTTP handshake when possible, use a secure cookie or supported subprotocol rather than a token in the URL, validate consultation membership server-side, and bind the session to a short-lived identity. Query-string tokens leak into logs and histories.

Authorization is not complete at connection time. Membership can be revoked, a token can expire, and each message type can require a different permission. Long-lived sessions need revalidation or bounded lifetime.

Validate Origin separately from authentication

Browsers send an Origin header during the handshake. The server allows exactly the expected site origin and rejects other upgrades with HTTP 403:

server.on('upgrade', (request, socket, head) => {
  if (
    request.url !== '/ws' ||
    request.headers.origin !== allowedOrigin
  ) {
    socket.write(
      'HTTP/1.1 403 Forbidden\r\n' +
      'Connection: close\r\n\r\n'
    );
    socket.destroy();
    return;
  }

  websocketServer.handleUpgrade(
    request, socket, head,
    ws => websocketServer.emit('connection', ws)
  );
});

Origin validation reduces cross-site WebSocket hijacking risk. It is not identity; non-browser clients can choose their headers. Keep authentication and authorization independent.

Avoid setAllowedOrigins("*") on an authenticated WebSocket endpoint. Use the exact production origins and test rejection.

Persist before acknowledging

The message path is:

client frame
  -> parse, size, schema, rate, and authorization checks
  -> claim message ID with SET NX
  -> XADD to consultation stream
  -> PUBLISH live event
  -> send acknowledgement to sender
  -> every subscribed instance forwards to local sockets

The critical code is:

const duplicateKey =
  `consultation:${consultationId}:message:${message.messageId}`;

const firstSeen = await redis.set(
  duplicateKey, '1', { NX: true, EX: 300 }
);

if (!firstSeen) {
  socket.send(JSON.stringify({
    type: 'ack',
    messageId: message.messageId,
    duplicate: true
  }));
  return;
}

const streamId = await redis.xAdd(
  `consultation:${consultationId}:messages`,
  '*',
  {
    messageId: message.messageId,
    text: message.text,
    instanceId
  }
);

await redis.publish(channel, liveEvent);
socket.send(JSON.stringify({
  type: 'ack', messageId, streamId, duplicate: false
}));

The client receives success only after the durable stream append. Pub/Sub is for low-latency fan-out, not durability. A subscriber disconnected at publish time misses that notification and must recover from the stream.

There is still a failure boundary between XADD and PUBLISH: a process can persist the message and die before live fan-out. A production system needs a stream-driven dispatcher or recovery scan so durable-but-unpublished messages are eventually delivered. The compact lab demonstrates the boundary rather than claiming to eliminate it.

Make client retries idempotent

Mobile networks disappear. A client can send a message, lose the acknowledgement, reconnect, and send again. Every client message therefore carries a stable messageId.

The Redis SET NX marker prevents a second stream append and returns an acknowledgement marked duplicate. The integration test sends the same ID twice and verifies one delivery.

The five-minute marker TTL is sufficient only for this lab. Production deduplication retention must cover the longest retry and offline window, or uniqueness must be durable in the message store. A database unique key or stream-side index can provide a longer boundary.

Backpressure must have a policy

socket.send() does not mean the peer consumed the data. If the client is slow, the server-side buffered amount grows.

The experiment closes a socket with 1013 when its buffered amount exceeds 64 KiB:

if (session.socket.bufferedAmount > 65_536) {
  rejected.inc({
    instance: instanceId,
    reason: 'slow_consumer'
  });
  session.socket.close(1013, 'consumer is too slow');
  continue;
}

The client can reconnect and resume from a durable cursor. Keeping an unlimited buffer would turn one slow or abandoned connection into a memory leak.

Other bounded inputs are equally important:

Boundary	Lab policy
Frame size	Maximum 4 KiB; close `1009` when exceeded
Text length	Maximum 1,000 characters
Send rate	Maximum 10 messages/s per connection
Authentication delay	First frame within 5 s
Buffered outbound data	Close at 64 KiB with `1013`
Invalid schema or policy	Close with `1008`

Per-connection rate limiting is not enough for abuse prevention. Production also needs account, consultation, tenant, and IP-aware limits at an appropriate shared boundary.

Scale-out needs a reconnect protocol

Redis Pub/Sub forwards live events to both app instances. Each instance sends only to its local authenticated sessions for that consultation.

When a client reconnects, it should provide its last durable stream ID. The server can read messages after that cursor, deliver the gap, and then resume live subscription. The browser in this lab does not implement cursor recovery; the stream exists so the missing production step is explicit.

A robust protocol defines:

message ID and server stream ID;
acknowledgement semantics;
reconnect cursor and replay limit;
duplicate behavior;
ordering per consultation;
authorization during replay;
tombstone or edit semantics;
retention and archival behavior.

Without a cursor, reconnect means “start receiving now,” which is not enough for chat.

Protect healthcare data by reducing it

Realtime healthcare messaging can contain sensitive data. The safest observability payload is usually metadata, not message text.

I would record:

active authenticated connections by instance;
accepted and rejected message counts by bounded reason;
authentication and authorization failures;
Redis append and publish latency;
send-buffer closures;
reconnect and replay counts;
delivery acknowledgement latency.

I would not put consultation IDs, patient IDs, message IDs, or message bodies into metric labels. Logs should use access-controlled correlation identifiers and a retention policy appropriate to the data classification. Traces must redact payloads and credentials.

TLS is mandatory outside the loopback lab: use wss://, validate proxy trust boundaries, and set idle timeouts and heartbeat behavior consistently across client, load balancer, ingress, and server.

Failure behavior is the design

Failure	Required behavior
App instance terminates	Socket reconnects to another instance and resumes from cursor
Redis Pub/Sub notification missed	Recover durable message from Stream
Duplicate client send	Acknowledge duplicate; do not append or broadcast twice
Redis unavailable before append	Do not acknowledge success
Persist succeeds, publish fails	Dispatcher later republishes from durable record
Client stops reading	Bound buffer, close, and require cursor resume
Authorization revoked	Close session and reject replay
Deploy drains instance	Stop accepting new sockets, notify/close existing sessions, then terminate

The integration test currently proves cross-instance live delivery, message-ID deduplication, instance health, and Prometheus availability. The next production-grade test would kill the sender instance after persistence, reconnect the receiver with a cursor, and prove recovery without duplication.

The WebSocket protocol specification, Spring Framework WebSocket documentation, Redis Pub/Sub documentation, and Redis Streams documentation define the protocol and storage primitives. Reliability comes from the application contract built around them.

A Kubernetes Rollout That Failed Safely: Readiness, Deadlines, and Rollback

ystc1247@gmail.com — Sun, 26 Jul 2026 06:00:00 GMT

A deployment strategy is only credible when a failed release has been exercised. I built a local Kubernetes experiment that rolls four replicas from v1 to v2, continuously probes the service, deploys an image that never becomes ready, waits for the Deployment progress deadline, and rolls back.

The observed result was:

Stage	Outcome
Initial v1 deployment	Four available replicas
v1 → v2 rolling update	Completed
Continuous availability	264 probes, zero non-200 responses
Broken image rollout	Exceeded its progress deadline
`kubectl rollout undo`	Restored four available v2 replicas
Versions served	v1 and v2; broken never entered Service endpoints

This does not prove every Kubernetes rollout is zero-downtime. It proves that this manifest, application, and failure mode behaved as designed in Kind.

Make availability a Deployment constraint

The core manifest is:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
  namespace: rollout-lab
spec:
  replicas: 4
  revisionHistoryLimit: 5
  minReadySeconds: 3
  progressDeadlineSeconds: 15
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 0
      maxSurge: 1
  template:
    spec:
      terminationGracePeriodSeconds: 10
      containers:
        - name: api
          image: rollout-lab:v1
          imagePullPolicy: Never
          ports:
            - containerPort: 8080
          readinessProbe:
            httpGet:
              path: /ready
              port: 8080
            periodSeconds: 1
            failureThreshold: 2
          livenessProbe:
            httpGet:
              path: /health
              port: 8080
            periodSeconds: 5

maxUnavailable: 0 tells the Deployment controller not to reduce available replicas during the rollout. maxSurge: 1 allows one extra pod while replacement proceeds. With four replicas, the controller can run up to five during transition.

minReadySeconds: 3 requires a pod to remain ready briefly before it counts as available. progressDeadlineSeconds: 15 makes a stalled rollout observable as ProgressDeadlineExceeded; it does not roll the Deployment back automatically.

imagePullPolicy: Never is deliberate only for Kind because the experiment loads local images into the node. Production should use immutable registry digests and a pull policy consistent with that release process.

Readiness and liveness answer different questions

The test application exposes:

mux.HandleFunc("/health", func(w http.ResponseWriter, _ *http.Request) {
    w.WriteHeader(http.StatusOK)
})

mux.HandleFunc("/ready", func(w http.ResponseWriter, _ *http.Request) {
    if broken == "true" {
        http.Error(w, "not ready", http.StatusServiceUnavailable)
        return
    }
    w.WriteHeader(http.StatusOK)
})

Liveness asks whether Kubernetes should restart the container. Readiness asks whether the pod should receive Service traffic. A dependency outage often should fail readiness without failing liveness; restarting a healthy process does not repair a remote database.

The broken image remains live but never becomes ready. Kubernetes starts it, probes it, and keeps it out of Service endpoints. Existing v2 pods continue serving while the rollout stalls.

Readiness must represent the ability to serve the endpoint safely. A hard-coded 200 response can produce a “successful” rollout that sends traffic to an unusable application.

Probe availability during the rollout

The experiment starts a background request loop after v1 is available:

while true; do
  timestamp=$(date -u +%Y-%m-%dT%H:%M:%SZ)
  response=$(curl -sS --max-time 1 \
    -w '|%{http_code}' \
    http://localhost:18059/ \
    | tr -d '\n' || true)
  printf '%s|%s\n' "$timestamp" "$response" \
    >> artifacts/availability.log
  sleep 0.1
done

The JSON response includes the application version, so the artifact proves both v1 and v2 served traffic. It also records the status of every probe.

My first parser counted the JSON line and HTTP status line separately, reporting 244 false failures even though every request returned 200. Removing the response newline corrected the evidence. Measurement code is production code: validate its record format before trusting a deployment conclusion.

Exercise a healthy rollout

The release command changes the image and waits for the controller:

kubectl --context kind-rollout-lab \
  -n rollout-lab \
  set image deployment/api api=rollout-lab:v2

kubectl --context kind-rollout-lab \
  -n rollout-lab \
  rollout status deployment/api --timeout=90s

With maxUnavailable: 0, the controller creates a v2 pod, waits until it is available, then removes an old pod. The rollout repeats until four v2 pods are available.

The external loop recorded no non-200 response. That says more than rollout status: it observes the Service path while the controller changes pods.

Inject a readiness failure and wait for the deadline

The broken image is compiled with readiness forced false:

kubectl --context kind-rollout-lab \
  -n rollout-lab \
  set image deployment/api api=rollout-lab:broken

kubectl --context kind-rollout-lab \
  -n rollout-lab \
  rollout status deployment/api --timeout=25s

The new pod cannot become available, so the Deployment cannot continue without violating maxUnavailable: 0. After the configured progress deadline, Kubernetes reports the stalled rollout. The old ready replicas keep serving.

The response should begin with conditions and events:

kubectl -n rollout-lab describe deployment api
kubectl -n rollout-lab get pods -o wide
kubectl -n rollout-lab get events --sort-by=.lastTimestamp
kubectl -n rollout-lab logs <broken-pod>

Do not immediately restart everything. Preserve the evidence that distinguishes image pull, scheduling, startup, readiness, crash, and application failures.

Roll back deliberately

Kubernetes stores previous ReplicaSet revisions up to revisionHistoryLimit. The experiment runs:

kubectl --context kind-rollout-lab \
  -n rollout-lab \
  rollout undo deployment/api

kubectl --context kind-rollout-lab \
  -n rollout-lab \
  rollout status deployment/api --timeout=90s

The Deployment returns to v2 and reaches four available replicas. The availability loop remains active until rollback completes.

Rollback safety depends on more than the image. Database migrations, queue schemas, cache formats, and external side effects must remain backward compatible. An application rollback cannot reverse a destructive migration or undo messages already published.

Understand what the PDB does not do

The lab includes:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: api
spec:
  minAvailable: 3
  selector:
    matchLabels:
      app: rollout-api

The PDB limits voluntary disruptions such as a node drain. It is not the control for Deployment rolling-update availability; maxUnavailable and maxSurge govern that. A PDB also cannot protect against involuntary node failure.

This is why copying a PDB into a manifest does not guarantee uptime. It protects one class of disruption when the eviction path honors it.

Rolling, blue-green, and canary answer different risks

The experiment measures rolling update behavior. Other strategies change the exposure model:

Strategy	Exposure	Rollback path	Main cost
Rolling	Replace pods gradually	Reverse Deployment revision	Old and new versions coexist
Blue-green	Switch traffic between full environments	Switch traffic back	Duplicate environment capacity and state compatibility
Canary	Send a controlled traffic fraction to new version	Remove canary traffic	Requires trustworthy segmentation and automated analysis

Canary is useful only if the team can measure the canary separately and has enough representative traffic. Blue-green is safe only if data contracts work in both directions and the traffic switch is tested. Rolling is simple only when old and new versions can coexist.

Production gates I would add

Use immutable image digests and record the release revision.
Set startup, readiness, and liveness probes from measured application behavior.
Define CPU and memory requests so surge pods can actually schedule.
Verify termination handling and connection draining.
Require backward-compatible database and event-schema changes.
Probe the real ingress path during rollout, not only pod IPs.
Gate on service-level error and latency signals, not pod readiness alone.
Set a progress deadline and make stalled rollout alerts actionable.
Test rollback in a staging environment with realistic state.
Preserve rollout logs, conditions, and external probe evidence.

The Kubernetes Deployment documentation defines rolling-update and rollback behavior, while the probe documentation and PodDisruptionBudget documentation define the health and disruption boundaries used here.

Kafka Delivery Semantics in Practice: Ordering, Duplicates, Rebalances, Retries, and DLT

ystc1247@gmail.com — Sun, 26 Jul 2026 06:00:00 GMT

Kafka is a replicated log, not a promise that every business effect happens exactly once. It gives strong building blocks—ordered partition logs, offsets, consumer groups, replication, and an idempotent producer—but application correctness still depends on keys, transaction boundaries, duplicate handling, retry policy, and dead-letter operations.

I tested those boundaries with a three-partition orders.v1 topic, a Spring Boot producer, two consumer instances, PostgreSQL projection tables, bounded retries, and orders.v1-dlt. The experiment sent ordered events, duplicated one event, injected one permanent failure, and added a second consumer while traffic continued.

The final evidence was:

Scenario	Observed result
Five events for `order-a`	`last_sequence=5`, `applied_events=5`
Same `event_id` for `order-b` sent twice	One database effect
Six events for `order-c` while consumer group changed	`last_sequence=6`, `applied_events=6`
Permanent poison event	One record in `orders.v1-dlt` after bounded retries
Consumer group after settling	Zero lag on partitions containing records

That result demonstrates a design under controlled failure. It does not turn Kafka into an exactly-once database.

Ordering exists inside one partition

A Kafka topic is divided into partition logs. Records in one partition have monotonically increasing offsets. Kafka does not define a single total order across all partitions.

For an order workflow, the producer key is the aggregate identifier:

kafkaTemplate.send(
        "orders.v1",
        event.orderId(),
        serializedEvent
);

Records with the same key are assigned to the same partition under a stable partitioning scheme, so order-a events can be consumed in partition order. Different orders can proceed in parallel.

This leads to several operational consequences:

A hot key cannot be parallelized across consumers without changing the domain model.
Increasing the topic’s partition count can change key-to-partition mapping for new records.
Ordering is lost if a retry design republishes failed events to an unrelated topic and later merges them without sequence checks.
The consumer should persist an aggregate sequence when missing or stale transitions would be dangerous.

“Kafka preserves order” is incomplete. The useful statement is “Kafka preserves record order within a partition, and this producer deliberately maps one aggregate to one partition.”

Producer idempotence is necessary but not end-to-end idempotence

The producer configuration is:

spring:
  kafka:
    producer:
      acks: all
      retries: 2147483647
      properties:
        enable.idempotence: true
        max.in.flight.requests.per.connection: 5
      key-serializer: org.apache.kafka.common.serialization.StringSerializer
      value-serializer: org.apache.kafka.common.serialization.StringSerializer

acks=all waits for the configured in-sync replica acknowledgement. Idempotence lets the broker deduplicate producer retries within the producer session and preserves ordering under the supported in-flight limit.

It does not deduplicate:

an HTTP caller submitting the same business command twice;
an application publishing the same event ID in two separate calls;
a consumer replay after its process crashes;
an external side effect such as a payment or email.

The experiment proves this by publishing the same order-b event twice. Both records may exist in Kafka. The database projection applies the event once because event_id is unique.

Put consumer idempotency in the same database transaction

The consumer inserts a processed-event marker and updates its read model in one PostgreSQL transaction:

@Transactional
public boolean apply(
        OrderEvent event,
        int partition,
        long offset
) {
    int inserted = jdbc.sql("""
        INSERT INTO processed_events(
            event_id, order_id, sequence,
            partition_id, partition_offset
        )
        VALUES (?, ?, ?, ?, ?)
        ON CONFLICT DO NOTHING
        """)
        .params(
            event.eventId(), event.orderId(), event.sequence(),
            partition, offset
        )
        .update();

    if (inserted == 0) return false;

    jdbc.sql("""
        INSERT INTO order_projection(
            order_id, last_sequence, applied_events
        )
        VALUES (?, ?, 1)
        ON CONFLICT (order_id) DO UPDATE
        SET last_sequence = EXCLUDED.last_sequence,
            applied_events = order_projection.applied_events + 1,
            updated_at = now()
        """)
        .params(event.orderId(), event.sequence())
        .update();

    return true;
}

The listener uses record acknowledgement. If the database transaction commits but the process dies before Kafka commits the offset, Kafka delivers the record again. The duplicate insert does nothing, so the read model does not receive a second effect.

This pattern requires marker retention to match replay expectations. Deleting processed_events after seven days while retaining Kafka data for thirty days makes an older replay unsafe.

For a strict state machine, I would also reject a sequence that is not exactly the expected successor. Kafka partition order reduces out-of-order delivery; it does not validate the domain transition.

Rebalance changes ownership, not record identity

Consumer-group members divide partitions among themselves. When a member joins, leaves, stalls beyond its liveness interval, or subscriptions change, Kafka reassigns partitions.

The lab starts one consumer container, publishes records, then scales to two containers and publishes the order-c sequence. Each application instance configures listener concurrency three, so six consumer threads compete for three partitions. Only three can own work; extra consumers are idle.

That is a useful capacity rule:

maximum active consumers in one group for one topic
  <= number of assigned partitions

More consumer processes do not create more parallelism after every partition has an owner. They may improve failover, but they also increase group coordination and resource cost.

During reassignment, a record can be processed and then redelivered if its effect completes before its offset is committed. The database idempotency key makes that replay safe. Long processing should also use realistic poll, heartbeat, batch, and timeout settings; otherwise the consumer can trigger avoidable rebalances.

Bound retries and separate permanent failures

The listener error handler uses two 500 ms retries, then publishes the record to a matching partition in orders.v1-dlt:

@Bean
CommonErrorHandler orderErrorHandler(
        KafkaTemplate<String, String> template
) {
    DeadLetterPublishingRecoverer recoverer =
        new DeadLetterPublishingRecoverer(
            template,
            (record, error) -> new TopicPartition(
                "orders.v1-dlt", record.partition())
        );

    return new DefaultErrorHandler(
        recoverer,
        new FixedBackOff(500L, 2L)
    );
}

Infinite retry on a poison record blocks its partition and every later record behind it. Immediate dead-lettering treats transient network failure as permanent. Bounded retry makes the decision explicit.

The DLT is not a trash can. It needs:

the original topic, partition, offset, key, timestamp, and failure class;
an alert on arrival and growth;
an owner who can distinguish bad data from a code or dependency failure;
a replay tool that preserves idempotency and ordering decisions;
retention long enough for the response process;
access control because failed payloads can still contain sensitive data.

If later events depend on the failed one, continuing after dead-lettering can create a domain gap. The correct choice may be to stop that key, quarantine the aggregate, or make the projection sequence-aware.

Inspect both the log and the effect

The Kafka UI showed 14 input records and one dead-letter record across three partitions:

The 14 records break down as five order-a events, two copies of one order-b event, one poison event, and six order-c events. The PostgreSQL artifact is equally important:

{
  "processed": [
    {"order_id":"order-a","last_sequence":5,"applied_events":5},
    {"order_id":"order-b","last_sequence":1,"applied_events":1},
    {"order_id":"order-c","last_sequence":6,"applied_events":6}
  ],
  "dltRecords": 1
}

Broker message count alone cannot prove a correct business effect. Database state alone cannot show backlog or poison records. The operating view needs both.

Useful metrics include:

consumer lag and oldest unprocessed record age;
rebalance count and duration;
processing latency and failure rate by bounded exception category;
retry and DLT rates;
duplicate-event count;
producer send latency and error rate;
under-replicated or offline partitions.

Serialization is a contract

The lab serializes the event to JSON explicitly and sends a string. That avoided an accidental dependency on an optional Jackson serializer and made the wire payload visible in tests.

A production event needs more than a Java record:

public record OrderEvent(
    UUID eventId,
    String orderId,
    int sequence,
    boolean failAlways
) {}

Define schema ownership, compatibility rules, required fields, default behavior, and how consumers handle unknown versions. Avoid publishing internal entity shapes. Whether the registry uses Avro, Protobuf, or JSON Schema, compatibility should be checked before deployment.

What the guarantees really are

Layer	Guarantee used here	What it does not guarantee
Partition	Ordered offsets	Global topic order
Key	Same aggregate routes together under stable partitioning	Ordering after careless repartitioning
Idempotent producer	Deduplicates producer retries in its session	Business-command or consumer deduplication
Consumer group	One partition owner per group assignment	Exactly one external side effect
Database event key	One projection effect per event ID	Correct domain order by itself
Bounded retry + DLT	Poison record stops blocking forever	Safe continuation or automatic repair

My practical Kafka checklist is now:

Choose a key from the ordering boundary.
Size partitions from required parallelism and future cost, not fashion.
Enable appropriate producer durability and idempotence.
Make consumer effects transactional and idempotent.
Test a crash after the effect but before offset commit.
Add a consumer during traffic and observe the rebalance.
Bound retries and operate the DLT as a queue with an owner.
Version the event schema independently of application entities.
Alert on lag age, failures, duplicates, and DLT growth.
State the guarantee precisely; do not write “exactly once” without naming the boundary.

The Kafka design documentation describes the log, replication, and delivery model, while the Spring for Apache Kafka reference documents listener containers, error handling, transactions, and dead-letter recovery.

The HTTP Client Boundary Behind a Feign PATCH Failure

ystc1247@gmail.com — Sun, 26 Jul 2026 05:55:00 GMT

This failure looked like an annotation problem:

java.net.ProtocolException: Invalid HTTP method: PATCH
  at java.net.HttpURLConnection.setRequestMethod(...)
  at feign.Client$Default.execute(...)

@PatchMapping was correct. The abstraction below it was not the transport I thought the application was using. The stack trace showed Feign.Client$Default and HttpURLConnection, which made the failure a client-boundary problem rather than a controller problem.

A robust fix needs more than adding an OkHttp dependency. It must make the concrete transport explicit, verify the bytes that leave the process, and define timeouts, mutation retry policy, idempotency, error mapping, and log redaction together.

I tested that boundary with Spring Boot 4.1, Spring Cloud 2025.1.2, OpenFeign, OkHttp, and MockWebServer.

Trace the abstraction downward

When a declarative HTTP call fails, inspect these layers in order:

Layer	Question
Java method	Are path, verb, headers, and body declared correctly?
Feign contract	Did Spring MVC annotations become the intended request template?
Feign client bean	Which `feign.Client` implementation was injected?
Transport	Does the concrete client support the method, TLS, proxy, and timeout behavior?
Network/upstream	What request arrived, and what response or delay occurred?
Application boundary	How are conflicts, timeouts, and retries represented to callers?

The stack trace is evidence. If it names Client$Default, changing an unrelated encoder or controller is unlikely to solve a transport method failure.

Also verify the dependency graph. Spring Boot and Spring Cloud versions must come from compatible release trains; mixing arbitrary Feign artifacts can produce a configuration that compiles but does not activate the intended client.

Make the declarative contract narrow

The client describes an inventory mutation:

@FeignClient(
    name = "inventory",
    url = "${inventory.base-url}",
    configuration = InventoryClientConfiguration.class
)
interface InventoryClient {
    @PatchMapping(
        path = "/inventory/{sku}",
        consumes = "application/json"
    )
    InventoryResponse adjust(
        @PathVariable String sku,
        @RequestHeader("Idempotency-Key") String idempotencyKey,
        @RequestHeader(HttpHeaders.AUTHORIZATION) String authorization,
        @RequestBody InventoryAdjustment request
    );
}

record InventoryAdjustment(int delta) {}
record InventoryResponse(String sku, int available) {}

The interface contains the wire contract. Application code calls a gateway instead of depending on Feign response types or exceptions directly.

Register the transport explicitly

Relying on a property to discover an optional transport is easy to misconfigure. The experiment creates the OkHttp transport and exposes the exact Feign client bean:

@Bean
feign.Client inventoryTransport() {
    okhttp3.OkHttpClient transport =
        new okhttp3.OkHttpClient.Builder()
            .connectTimeout(Duration.ofMillis(300))
            .readTimeout(Duration.ofMillis(500))
            .callTimeout(Duration.ofSeconds(1))
            .build();

    return new feign.okhttp.OkHttpClient(transport);
}

This is an application boundary, so explicit wiring is useful. It removes ambiguity about which HTTP stack will execute PATCH.

The timeouts have different meanings:

Timeout	Bounds
Connect	Establishing the connection
Read	Waiting for response bytes
Call	The entire exchange, including connection and body

A timeout is not a latency SLO. It is a resource and failure boundary. Set it from downstream behavior and the caller’s total deadline, not from a copied default.

Do not retry a mutation blindly

After a timeout, the caller may not know whether the upstream applied the PATCH. An automatic retry can decrement inventory twice.

The experiment disables Feign retries:

@Bean
Retryer inventoryRetryer() {
    return Retryer.NEVER_RETRY;
}

The gateway requires a stable command ID and sends it as an idempotency key:

public InventoryResult adjust(
        String sku,
        int delta,
        String commandId,
        String serviceToken
) {
    Objects.requireNonNull(
        commandId,
        "a stable commandId is required for mutation idempotency"
    );

    InventoryResponse response = client.adjust(
        sku,
        commandId,
        "Bearer " + serviceToken,
        new InventoryAdjustment(delta)
    );

    return new InventoryResult(response.sku(), response.available());
}

An idempotency header helps only if the upstream stores the key and the resulting outcome atomically. The client cannot declare a mutation safe by itself.

For reads, bounded retries may be acceptable when the remaining deadline, backoff, and load amplification are understood. For writes, start with no automatic retry and add one only after end-to-end idempotency is proven.

Map upstream semantics at the gateway

A 409 response has business meaning in this API. The Feign error decoder translates it into an application exception:

@Bean
ErrorDecoder inventoryErrors() {
    return (methodKey, response) -> response.status() == 409
        ? new InventoryConflictException(
              "inventory version conflict")
        : new ErrorDecoder.Default()
              .decode(methodKey, response);
}

The rest of the application should not need to parse Feign exception messages. The gateway can map:

400 to a request-contract defect or caller error;
401/403 to service authentication or authorization failure;
404 to domain absence when that is part of the upstream contract;
409 to a version/idempotency conflict;
429/503 to controlled availability failure with retry metadata;
timeout to an ambiguous outcome for a mutation.

Preserve the upstream status and correlation metadata in structured diagnostics, but do not return internal response bodies blindly to clients.

Verify the actual request

A context test with MockWebServer records the request at the transport boundary:

upstream.enqueue(new MockResponse()
    .setResponseCode(200)
    .addHeader("Content-Type", "application/json")
    .setBody("{\"sku\":\"MED-42\",\"available\":11}"));

InventoryResult result = gateway.adjust(
    "MED-42", -1, "cmd-018", "secret-token");

RecordedRequest request = upstream.takeRequest(1, TimeUnit.SECONDS);

assertThat(request.getMethod()).isEqualTo("PATCH");
assertThat(request.getHeader("Idempotency-Key")).isEqualTo("cmd-018");
assertThat(request.getHeader("Authorization"))
    .isEqualTo("Bearer secret-token");
assertThat(request.getBody().readUtf8())
    .isEqualTo("{\"delta\":-1}");
assertThat(result.available()).isEqualTo(11);

This proves the annotation, encoder, headers, transport, and decoder as one path. A unit test of the interface cannot do that.

Two additional tests protect failure semantics:

A 409 response becomes InventoryConflictException.
An upstream delayed by 800 ms exceeds the 500 ms read timeout and produces exactly one recorded request.

The request-count assertion is critical. It proves the timed-out mutation was not retried invisibly.

Log the boundary without leaking it

Feign logging can expose Authorization headers and bodies. The experiment uses BASIC logging, which records method, URL, status, and timing without full payloads.

In production I would emit structured metrics and traces for:

client name, normalized route, method, status class, and outcome;
latency and timeout phase;
retry attempt count;
circuit-breaker state when present;
a correlation or trace ID.

Do not use SKU, user ID, raw URL, or exception text as unbounded metric labels. Redact credentials and sensitive bodies in logs and traces.

A practical debugging sequence

When an HTTP client method fails:

Read the deepest relevant stack frame and identify the concrete transport.
Inspect the resolved dependency graph and compatible Spring release train.
Make the feign.Client bean explicit.
Reproduce against a recording server and assert method, path, headers, and body.
Test timeout and response mapping, not only the 200 path.
Prove mutation idempotency before enabling retries.
Keep credentials and bodies out of client logs.
Monitor the client boundary with bounded labels and traces.

The PATCH exception was useful because it exposed a hidden dependency boundary. The durable solution is not “use OkHttp whenever PATCH fails.” It is to treat outbound HTTP as a production interface with an explicit transport and tested failure semantics.

The primary references are the Spring Cloud OpenFeign documentation, OpenFeign project documentation, OkHttp documentation, and the HTTP PATCH specification, RFC 5789.

Protecting a JPA Aggregate: Invariants, DTOs, and Optimistic Locking

ystc1247@gmail.com — Sun, 26 Jul 2026 05:55:00 GMT

The problem with entity setters is not that setName is aesthetically unpleasant. The problem is that a public mutation can bypass the rules that make several fields and child rows valid together.

I built a small purchase-order aggregate around three invariants:

A line quantity must be between 1 and 100.
An order cannot be confirmed without at least one line.
A confirmed order cannot be edited.

The design uses no public setters. It exposes commands such as addLine, changeQuantity, and confirm, returns immutable read views, and verifies stale concurrent edits with real Hibernate optimistic locking.

Start from an invalid state that setters allow

This shape is convenient but cannot protect the aggregate:

@Entity
@Getter
@Setter
class PurchaseOrder {
    private OrderStatus status;

    @OneToMany(mappedBy = "order")
    private List<OrderLine> lines;
}

Any caller can now:

order.setStatus(CONFIRMED); // even when lines is empty
order.setLines(List.of());  // even after confirmation

A service may remember to validate first, but another service, mapper, test fixture, or deserializer may not. The entity has no single boundary where its rules are guaranteed.

This is not an argument against Lombok or every getter. A generated accessor does not create a Swagger endpoint; API documentation comes from controller mappings and schemas. A getter also does not cause LazyInitializationException by itself. That exception occurs when code accesses an uninitialized lazy association after its persistence context is unavailable. The real questions are which state is public, who may mutate it, and where loading occurs.

Put commands on the aggregate root

The root owns its lines and exposes business operations:

@Entity
@Table(name = "purchase_orders")
public class PurchaseOrder {
    @Id @GeneratedValue
    private Long id;

    @Version
    private long version;

    @Enumerated(EnumType.STRING)
    private OrderStatus status = OrderStatus.DRAFT;

    private long changeSequence;

    @OneToMany(
        mappedBy = "order",
        cascade = CascadeType.ALL,
        orphanRemoval = true
    )
    private final List<OrderLine> lines = new ArrayList<>();

    protected PurchaseOrder() {}

    public static PurchaseOrder draft() {
        return new PurchaseOrder();
    }
}

The protected no-argument constructor exists for JPA. Application code uses the named factory, which starts the aggregate in a valid state.

Each method says what the caller intends and enforces the relevant rules:

public void addLine(String sku, int quantity) {
    requireDraft();
    Objects.requireNonNull(sku, "sku");
    lines.add(new OrderLine(this, sku, quantity));
    changeSequence++;
}

public void changeQuantity(String sku, int quantity) {
    requireDraft();
    OrderLine line = lines.stream()
            .filter(candidate -> candidate.sku().equals(sku))
            .findFirst()
            .orElseThrow(() -> new IllegalArgumentException(
                    "unknown sku: " + sku));
    line.changeQuantity(quantity);
    changeSequence++;
}

public void confirm() {
    requireDraft();
    if (lines.isEmpty()) {
        throw new IllegalStateException(
                "an order needs at least one line");
    }
    status = OrderStatus.CONFIRMED;
    changeSequence++;
}

private void requireDraft() {
    if (status != OrderStatus.DRAFT) {
        throw new IllegalStateException(
                "confirmed orders cannot be edited");
    }
}

changeQuantity is more useful than setQuantity. It identifies the line, verifies aggregate status, delegates the line-level range rule, and participates in aggregate versioning.

The child still protects its local invariant:

void changeQuantity(int quantity) {
    if (quantity < 1 || quantity > 100) {
        throw new IllegalArgumentException(
                "quantity must be between 1 and 100");
    }
    this.quantity = quantity;
}

The root protects cross-object rules; the child protects a rule about its own value.

Do not serialize the entity as the API

Returning JPA entities from controllers couples the HTTP schema to persistence and makes lazy loading, bidirectional relationships, and accidental field exposure part of request handling.

The read boundary is an immutable DTO:

public record OrderView(
        Long id,
        long version,
        String status,
        List<PurchaseOrder.OrderLineView> lines
) {
    public static OrderView from(PurchaseOrder order) {
        return new OrderView(
                order.id(),
                order.version(),
                order.status().name(),
                order.lineViews()
        );
    }
}

Mapping should occur inside a transaction or after a query that deliberately fetches the required graph. The controller returns OrderView, not a live Hibernate proxy.

This also makes the API contract explicit. Adding an internal persistence field does not automatically publish it, and removing a getter from the entity does not unexpectedly break JSON.

Optimistic locking must cover the whole aggregate

@Version prevents a stale root-row update from silently overwriting a newer one. The usual SQL shape is conceptually:

UPDATE purchase_orders
SET status = ?, version = version + 1
WHERE id = ? AND version = ?;

If no row matches, Hibernate reports an optimistic-lock failure.

There is a subtle boundary: changing only an OrderLine does not necessarily dirty the purchase_orders row. A version field on the root therefore may not protect concurrent child-only edits.

The experiment makes every aggregate mutation increment changeSequence on the root. That deliberately dirties the root row, so its version advances when a line changes. Other valid strategies include putting versions on children, forcing an optimistic lock on the root, or remodeling the transaction boundary. The important part is testing the actual SQL behavior rather than assuming @Version automatically covers a graph.

The integration test loads two detached copies, commits one editor, then attempts to merge the stale editor:

PurchaseOrder firstEditor = loadDetached(id);
PurchaseOrder staleEditor = loadDetached(id);

firstEditor.changeQuantity("SKU-1", 2);
mergeAndCommit(firstEditor);

staleEditor.changeQuantity("SKU-1", 3);
assertThatThrownBy(() -> mergeAndCommit(staleEditor))
        .isInstanceOf(OptimisticLockException.class);

That test originally exposed the child-only version gap. Adding a root mutation made the test pass for the right reason.

Test the domain without Hibernate, then test Hibernate

Most invariants are plain unit tests:

@Test
void confirmedOrderCannotBeMutated() {
    PurchaseOrder order = PurchaseOrder.draft();
    order.addLine("SKU-1", 2);
    order.confirm();

    assertThatThrownBy(() ->
            order.changeQuantity("SKU-1", 3))
        .isInstanceOf(IllegalStateException.class);
}

These are fast and show that the object protects itself. The separate @DataJpaTest proves mapping, cascade behavior, and optimistic locking against Hibernate and H2.

Both levels are necessary. A unit test cannot prove SQL version checks. A persistence test should not be the only place a business rule is readable.

Where accessors remain reasonable

Accessors are not forbidden. I use them deliberately:

Type	Typical choice
Request/response DTO	Immutable record or explicit constructor
Configuration binding	Framework-compatible immutable properties where possible
JPA identifier/version	Narrow read access when application code needs it
Aggregate collection	Immutable view or mapped DTO, never the mutable list
Business mutation	Named method that enforces the rule
Persistence-only constructor	`protected` no-argument constructor

The test is whether the public method preserves meaning. order.confirm() describes a transition. order.setStatus(CONFIRMED) exposes storage.

Transaction and loading boundaries still matter

A well-encapsulated entity does not solve every JPA problem. Services still need to:

load the aggregate with the relationships required for the command;
keep one transaction around the command and commit;
avoid N+1 query patterns when mapping lists of aggregates;
decide how optimistic-lock conflicts become API responses;
avoid leaking entities beyond the persistence boundary;
publish external events only after durable state is committed.

The concrete lesson is not “never use @Getter.” It is: make the aggregate the only place where its invariants can be changed, make the API a separate model, and test the concurrency boundary with the real ORM.

The Jakarta Persistence specification defines entity and version semantics, while the Hibernate user guide documents Hibernate’s mapping and optimistic-lock behavior.

When a Redis Leaderboard Lies: Outbox, Idempotency, and Reconciliation

ystc1247@gmail.com — Sun, 26 Jul 2026 05:55:00 GMT

A Redis sorted set makes a leaderboard read cheap: ZREVRANGE returns the highest scores without asking the relational database to aggregate every request. The difficult part is not the ZSET operation. It is keeping that projection trustworthy when the database commit, Redis write, worker process, or network fails.

The unsafe implementation is also the most tempting:

scoreRepository.increment(userId, delta);     // database write
redisTemplate.opsForZSet()
        .incrementScore("leaderboard", userId, delta); // Redis write

Those writes do not share a transaction. If the database commits and Redis fails, the leaderboard is stale. If Redis succeeds and the database rolls back, the leaderboard invents a score. A retry can apply the delta twice. Reversing the order changes which inconsistency is possible; it does not remove the inconsistency.

I built a failure experiment around a different contract:

PostgreSQL is the source of truth.
A score command has a stable event ID.
The score change and an outbox row commit in one database transaction.
A worker projects absolute score and version into Redis.
Projection is idempotent and rejects stale versions.
Reconciliation detects and repairs missing, stale, ahead, or corrupt cache state.

This is an eventually consistent read model, not a distributed transaction disguised as a service method.

Define the consistency contract

Before choosing code, decide what the product may observe:

Question	Contract in this experiment
Authoritative score	PostgreSQL `user_scores.score`
Leaderboard freshness	Eventual; visible lag while the worker is down
Duplicate client command	One database effect per `event_id`
Duplicate worker delivery	Safe; absolute score and version are idempotent
Redis loss	Rebuild from PostgreSQL
Redis corruption	Detect and repair through reconciliation
Read during lag	Return the projection with explicit freshness, or fall back when the product requires it

Redis is allowed to be behind. It is never allowed to silently become the authority for a business score.

Commit the score and outbox atomically

The relational schema separates command identity, current state, and pending projection work:

CREATE TABLE score_events (
  event_id uuid PRIMARY KEY,
  user_id text NOT NULL,
  delta integer NOT NULL CHECK (delta BETWEEN -1000 AND 1000),
  created_at timestamptz NOT NULL DEFAULT now()
);

CREATE TABLE user_scores (
  user_id text PRIMARY KEY,
  score bigint NOT NULL DEFAULT 0,
  version bigint NOT NULL DEFAULT 0,
  updated_at timestamptz NOT NULL DEFAULT now()
);

CREATE TABLE leaderboard_outbox (
  id bigserial PRIMARY KEY,
  event_id uuid NOT NULL UNIQUE REFERENCES score_events(event_id),
  user_id text NOT NULL,
  absolute_score bigint NOT NULL,
  score_version bigint NOT NULL,
  created_at timestamptz NOT NULL DEFAULT now(),
  published_at timestamptz
);

CREATE INDEX leaderboard_outbox_pending_idx
  ON leaderboard_outbox (id)
  WHERE published_at IS NULL;

The command handler performs three changes inside one PostgreSQL transaction:

await client.query('BEGIN');

const inserted = await client.query(
  `INSERT INTO score_events(event_id, user_id, delta)
   VALUES ($1, $2, $3)
   ON CONFLICT DO NOTHING
   RETURNING event_id`,
  [eventId, userId, delta]
);

if (inserted.rowCount === 0) {
  await client.query('COMMIT');
  return { duplicate: true };
}

const score = await client.query(
  `INSERT INTO user_scores(user_id, score, version)
   VALUES ($1, $2, 1)
   ON CONFLICT (user_id) DO UPDATE
   SET score = user_scores.score + EXCLUDED.score,
       version = user_scores.version + 1,
       updated_at = now()
   RETURNING user_id, score, version`,
  [userId, delta]
);

await client.query(
  `INSERT INTO leaderboard_outbox
     (event_id, user_id, absolute_score, score_version)
   VALUES ($1, $2, $3, $4)`,
  [eventId, userId, score.rows[0].score, score.rows[0].version]
);

await client.query('COMMIT');

The primary key on score_events.event_id is the command idempotency boundary. If an HTTP retry sends the same command again, the score is not incremented twice.

The outbox stores the absolute score and monotonic version, not another delta. That decision makes worker replay safe. Applying “score is 17 at version 2” twice is idempotent; applying “add 7” twice is not.

Project with at-least-once delivery in mind

The worker claims pending rows with FOR UPDATE SKIP LOCKED, which allows several workers to claim different batches:

SELECT id, event_id, user_id, absolute_score, score_version
FROM leaderboard_outbox
WHERE published_at IS NULL
ORDER BY id
FOR UPDATE SKIP LOCKED
LIMIT 100;

For every row it updates three Redis structures atomically through Lua:

local currentVersion = tonumber(
  redis.call('HGET', KEYS[3], ARGV[1]) or '-1'
)
local incomingVersion = tonumber(ARGV[3])

if redis.call('EXISTS', KEYS[1]) == 1 then
  return 'duplicate'
end

if incomingVersion > currentVersion then
  redis.call('ZADD', KEYS[2], ARGV[2], ARGV[1])
  redis.call('HSET', KEYS[3], ARGV[1], incomingVersion)
  redis.call('SET', KEYS[1], '1', 'EX', ARGV[4])
  return 'applied'
end

redis.call('SET', KEYS[1], '1', 'EX', ARGV[4])
return 'stale'

The keys represent an event marker, the leaderboard ZSET, and a user-version hash. Lua makes the comparison and writes atomic inside Redis.

The version check remains important after the event marker expires. If an old outbox row is replayed later, it cannot overwrite a newer absolute score. The marker is an optimization for duplicate work; the monotonic version is the correctness boundary.

Design for the crash between Redis and PostgreSQL

The worker performs this sequence:

Claim an outbox row in PostgreSQL.
Apply the Redis projection.
Mark the outbox row published.
Commit.

It can crash after step 2. PostgreSQL rolls back the claim, so another worker reads the same row. The Redis operation must therefore tolerate duplicate delivery. That is why the projection uses an event marker, an absolute score, and a version check.

Holding the database transaction open across a Redis call is a conscious tradeoff in this compact experiment. SKIP LOCKED prevents one slow batch from stopping all workers, but a production design should bound Redis timeouts, keep batches small, monitor lock age, and consider a leased-claim state when external latency is high.

Exactly-once delivery is not the goal. One durable database effect plus an idempotent at-least-once projection is easier to reason about and test.

Reconcile because idempotency is not repair

Idempotency handles duplicates. It does not repair deletion, manual edits, expired Redis data, or a software defect that wrote the wrong score.

The reconciliation job reads source rows and classifies each projection:

export function compareProjection(source, projection) {
  if (!projection) return 'missing';
  if (Number(projection.version) < Number(source.version)) return 'stale';
  if (Number(projection.version) > Number(source.version)) return 'ahead';
  if (Number(projection.score) !== Number(source.score)) return 'corrupt';
  return 'current';
}

ahead is not automatically safe. It can mean the source read is stale, the wrong database is being queried, or Redis contains invented state. I would alert and investigate before forcing a repair in a system where cross-region or replica lag is possible.

For missing, stale, or corrupt rows, the repair path writes the source’s absolute score and version. A full rebuild can delete the projection and replay every source row into a new key, then atomically switch readers to that key.

What the failure experiment observed

The experiment intentionally duplicated a command, stopped the worker, and corrupted one Redis score:

Stage	PostgreSQL source	Redis projection	Result
First command for `user-a`	score 10, version 1	score 10	Applied
Same event ID again	score 10, version 1	score 10	Duplicate detected; no second effect
Worker stopped	`user-a` 17, `user-b` 25	only `user-a` 10	Expected visible projection lag
Worker restarted	`user-a` 17, `user-b` 25	both projected	Outbox drained
Redis `user-a` changed to 999	source still 17	corrupt score 999	Reconciliation classified corruption
Repair complete	`user-b` 25, `user-a` 17	identical ranking	One of two rows repaired

The final artifact reported two rows checked: one corrupt and one current. It repaired the corrupt row, and the final source ordering exactly matched the ZSET ordering.

That result demonstrates the failure paths; it does not establish throughput. A production run also needs outbox age, pending count, worker error rate, reconciliation drift count, Redis latency, and leaderboard freshness metrics.

Expose freshness to the product

Eventual consistency becomes a product problem when nobody defines “eventual.” Useful options include:

return projectedThrough or projection age with the leaderboard;
fall back to a database query when the outbox age exceeds a limit;
hide a user’s just-submitted rank until their version is projected;
offer read-your-write behavior from the command response while global ranking catches up;
fail closed for rewards or payouts that require authoritative ordering.

The Redis score is a double. Integer values are exact only within the IEEE-754 safe integer range, so very large counters or composite ranking scores need a deliberate representation and tie-break rule.

The operational checklist

Keep the business score authoritative in one durable system.
Require a stable command/event ID at the API boundary.
Commit source state and outbox work in one database transaction.
Project absolute state with a monotonic version.
Assume the worker will deliver at least once.
Make Redis comparison and writes atomic.
Monitor outbox age, not only row count.
Reconcile on a schedule and make full rebuilds routine.
Define freshness and fallback behavior for readers.
Never award money or irreversible benefits from an unverified cache projection.

Redis documents ZSET behavior in its sorted-set documentation, while Spring’s Redis reference covers the client-side integration. Neither replaces the application-level consistency contract.

Reducing Spring Boot Log Cost Without Hiding Failures

ystc1247@gmail.com — Sun, 26 Jul 2026 05:45:00 GMT

Log volume is a reliability and cost concern, but content-substring filtering is a dangerous fix. Dropping every message that contains /health, kube-probe, or ELB-HealthChecker can hide an exception whose text happens to include the same string. It also breaks when a proxy changes wording.

The safer boundary is logger identity: classify the request from structured HTTP state, send routine successful probes to a dedicated logger, keep business access events separate, and force every 5xx event onto an unsuppressed ERROR path.

I measured that design with two otherwise identical Spring Boot 4.1 containers. The baseline enabled health access logs; the filtered variant disabled only the dedicated health logger. Each received 2,000 health checks, 100 successful business requests, and 10 injected 5xx requests.

Decide which signal is redundant

Before suppressing a log, write down what would be lost and what equivalent evidence remains:

Signal	Keep in logs?	Replacement or reason
Successful health access event	Usually no	Counter, status metric, and probe alert
Failed health request	Yes	Incident evidence; log at ERROR
Successful business access event	Yes or sampled	Request audit and latency context
Business 5xx	Yes	Never suppress through the health logger
Request body or credentials	No by default	High privacy and security risk

“The log platform is expensive” is not enough. Measure the source volume, prove another signal exists, and define how to restore samples during an incident.

Classify from the parsed request, not formatted text

The classifier uses an exact path comparison and gives errors priority:

final class AccessLogClassifier {
    static final String HEALTH_LOGGER = "lab.access.health";
    static final String APPLICATION_LOGGER = "lab.access.application";
    static final String ERROR_LOGGER = "lab.access.error";

    static String loggerFor(String path, int status) {
        if (status >= 500) return ERROR_LOGGER;
        return path.equals("/actuator/health")
                ? HEALTH_LOGGER
                : APPLICATION_LOGGER;
    }
}

/actuator/health is probe traffic. /api/errors/actuator/health is an application path and must not be suppressed. A 503 from /actuator/health goes to lab.access.error, not lab.access.health.

The filter emits structured fields after the response completes:

String path = request.getRequestURI();
if (path.equals("/actuator/health")) probeRequests.increment();

var logger = LoggerFactory.getLogger(
        AccessLogClassifier.loggerFor(path, response.getStatus()));

LoggingEventBuilder event = response.getStatus() >= 500
        ? logger.atError()
        : logger.atInfo();

event.addKeyValue("http.request.method", request.getMethod())
     .addKeyValue("url.path", path)
     .addKeyValue("http.response.status_code", response.getStatus())
     .addKeyValue("event.duration_ms", durationMs)
     .log("request completed");

The ERROR call is not cosmetic. During the experiment I found that routing a 5xx to lab.access.error while still calling atInfo() allowed an ERROR logger threshold to discard it. The corrected implementation selects both the error logger and the error level, and the measurement includes explicit 5xx requests so the regression cannot hide.

Disable one logger through configuration

Spring Boot structured logging produces machine-readable ECS JSON. The only difference between variants is the health logger level:

logging:
  structured:
    format:
      console: ecs
  level:
    lab.access.health: ${HEALTH_ACCESS_LOG_LEVEL:OFF}
    lab.access.application: INFO
    lab.access.error: ERROR

This is more stable than a Logback filter that inspects formattedMessage. The application has already decided what kind of event it is, and configuration decides whether that class is retained.

Structured fields also avoid fragile parsing. http.response.status_code=500 is safer to query than a regular expression over a custom sentence. Request IDs are validated and bounded before entering MDC; secrets and bodies are not logged.

Preserve the metric before removing the log

The filter increments a Micrometer counter independently of logging:

probeRequests = Counter.builder("http.probe.requests")
        .description("Health probe requests independent of access-log sampling")
        .register(registry);

Prometheus can alert on failed probes or unexpected probe volume without ingesting one access event per successful check. The experiment verifies that the counter remains queryable after lab.access.health is disabled.

Metrics and logs are not interchangeable. A metric preserves count and trend; an error log preserves diagnostic context. The design intentionally keeps both where each is strongest.

Measure before and after

Both variants received the same bounded workload. The captured container logs produced:

Signal	Baseline	Health logger off
Captured bytes	933,744	51,220
Successful health access events	2,001	0
Successful business access events	100	100
Error events	10	10
Byte reduction	—	94.51%

The extra baseline health event is the readiness request used before the measured loop. It is included rather than silently adjusted away.

The result demonstrates selectivity: every business access event and every injected error remained, while routine health access events disappeared. It does not prove the same percentage for another service because JSON shape, stack traces, sidecars, probe intervals, and traffic mix differ.

Make the cost projection explicit

The projection uses the measured byte difference per health request:

avoided bytes/request
  = (baseline bytes - filtered bytes) / 2,000

monthly probe requests
  = 30 days × 24 hours × 60 minutes × 12 probes/minute

For one replica, one probe every five seconds, and an illustrative ingestion rate of $0.50/GiB, the model yields:

Projection input or output	Value
Avoided ingestion	0.213 GiB/month
Illustrative avoided ingestion charge	$0.1065/month

That tiny number is an important result. For one small service, this optimization barely affects the bill. The value can grow across many replicas, several probes, verbose access formats, retention tiers, and indexing charges, but it must be calculated from the actual platform. Do not present an illustrative ingestion rate as a cloud invoice.

Guardrails for production

I would ship this only with the following conditions:

Match a normalized route or dedicated filter-chain attribute, not message text.
Check status first so 5xx events can never enter the suppressible logger.
Emit 5xx events at ERROR and test the configured threshold.
Preserve probe counts and failures as metrics with alerts.
Keep business access logging and application exceptions independently configurable.
Validate request IDs and never log tokens, credentials, or bodies by default.
Re-enable or sample the health logger temporarily when incident analysis requires it.
Compare actual ingest, index, retention, and query costs before claiming savings.

If log pressure comes from business traffic rather than probes, use deliberate sampling, aggregation, shorter retention, or narrower structured fields. Do not hide all successful requests without deciding what audit and latency evidence the service still needs.

The implementation choices follow the Spring Boot logging reference, Spring Boot Actuator metrics reference, and Logback filter documentation.

A Reproducible JMeter Load Test: Throughput, p95, Errors, and Evidence

ystc1247@gmail.com — Sun, 26 Jul 2026 05:45:00 GMT

JMeter is a load and performance test runner, not a unit-testing framework. A useful JMeter result is also not “the server handled 500 TPS.” It is a reproducible statement about one build, one workload shape, one environment, and several signals: offered load, achieved throughput, latency distribution, errors, and saturation.

I built a bounded experiment around a deterministic HTTP API and JMeter 5.6.3. The plan lives in a saved .jmx file, runs in non-GUI mode, writes raw JTL samples, generates the official HTML report, and passes or fails through an independently calculated gate.

Define the question before the thread group

The experiment asks a deliberately narrow question:

Can this local target sustain approximately 20 requests per second for 30 seconds while keeping the error ratio below 5% and p95 latency below 250 ms?

The target is synthetic. It injects known behavior so the analysis has something to find:

export function classifyRequest(sequence) {
  if (!Number.isInteger(sequence) || sequence < 1) {
    return { status: 400, delayMs: 0 };
  }
  if (sequence % 25 === 0) return { status: 503, delayMs: 30 };
  if (sequence % 10 === 0) return { status: 200, delayMs: 180 };
  return { status: 200, delayMs: 20 };
}

The API is not pretending to represent a real capacity limit. It makes latency and failure signals deterministic enough to verify the test machinery.

Concurrency is not throughput

A JMeter thread is a virtual user. Twenty threads do not imply 20 requests per second: response time, timers, ramp-up, connection reuse, and client-side scheduling all affect the achieved rate.

The saved plan controls the important variables:

Variable	Value	Reason
Threads	20	Enough concurrency to sustain the target rate
Ramp-up	10 s	Avoid an accidental synchronized connection burst
Duration	30 s	Bounds the experiment
Throughput timer	1,200/min	Targets approximately 20 requests/s
Connect timeout	1 s	Separates connection failure from slow response
Response timeout	2 s	Bounds stalled samples
Keep-alive	Enabled	Exercises normal connection reuse

The core plan is version-controlled, not reconstructed through screenshots:

<ThreadGroup testname="Steady state" enabled="true">
  <stringProp name="ThreadGroup.num_threads">${__P(threads,20)}</stringProp>
  <stringProp name="ThreadGroup.ramp_time">10</stringProp>
  <boolProp name="ThreadGroup.scheduler">true</boolProp>
  <stringProp name="ThreadGroup.duration">${__P(duration,30)}</stringProp>
</ThreadGroup>

<ConstantThroughputTimer testname="Bounded throughput" enabled="true">
  <doubleProp>
    <name>throughput</name>
    <value>1200.0</value>
  </doubleProp>
  <intProp name="calcMode">1</intProp>
</ConstantThroughputTimer>

The workload sends a unique request ID, uses per-thread query data, and calls one GET endpoint. A production scenario would also define authentication, realistic data distribution, cache state, think time, and the ratio between different operations.

Run JMeter without the GUI

The GUI is useful for authoring and debugging. It is a poor load generator because listeners, result trees, and rendering consume memory and CPU that should be generating traffic.

The repeatable execution path is:

jmeter \
  -n \
  -t /tests/api-load-plan.jmx \
  -Jthreads=20 \
  -Jduration=30 \
  -l /artifacts/results.jtl \
  -e \
  -o /artifacts/report \
  -f

-n selects non-GUI mode, -t selects the saved plan, -l preserves raw samples, and -e -o generates the HTML report. Keeping the JTL matters: it lets another program verify the report instead of treating a screenshot as the source of truth.

Read the distribution, not the average

The measured run produced:

Signal	Observed value
Samples	617
Failed samples	16
Error ratio	2.59%
Achieved throughput	20.58 requests/s
p50	24 ms
p95	184 ms
p99	188 ms
Maximum	193 ms

The average was 37.72 ms, which hides the deliberately slow tenth-request path. p50 stays near the 20 ms baseline, while p95 and p99 move into the 180 ms class. That is why an average alone is unsafe for an SLO or capacity decision.

The 16 failures also require context. The injected sequence is per virtual user. Because threads ramp up over ten seconds, only 16 threads reached their 25th request before the 30-second scheduler ended. The observed 2.59% is therefore consistent with the actual workload, but it is not a universal failure probability.

The achieved 20.58 requests/s is close to the requested 20. It is not exactly 20 because the scheduler starts and stops samples around the duration boundary, and throughput is calculated from first-sample to last-sample timestamps.

Verify the result independently

The gate reads the JTL-derived JSON rather than scraping the HTML:

const failures = [];

if (summary.errorRate > 0.05) {
  failures.push(`error rate ${summary.errorRate} exceeds 0.05`);
}
if (summary.latencyMs.p95 > 250) {
  failures.push(`p95 ${summary.latencyMs.p95}ms exceeds 250ms`);
}
if (summary.samples < 300) {
  failures.push(`only ${summary.samples} samples were recorded`);
}

if (failures.length) {
  console.error(failures.join('\n'));
  process.exit(1);
}

Three checks are important here:

The error ratio is part of the result, not discarded before latency analysis.
The percentile is calculated from individual elapsed samples.
A minimum sample count prevents an accidentally short run from passing.

A stronger production gate would distinguish warm-up from steady state, define acceptable response codes per request type, verify the achieved load band, and compare several repetitions rather than one run.

Correlate the bottleneck

JMeter describes what the client observed. It does not identify the bottleneck. During a real test, align the JTL timestamps with:

application request rate, error ratio, and latency histograms;
CPU throttling, allocation rate, garbage collection, and thread pools;
database connection-pool wait time, slow queries, locks, and I/O;
downstream latency, retries, circuit breakers, and rate limits;
load-generator CPU, memory, sockets, and network throughput.

If the generator is saturated, a flat throughput curve may describe the client machine rather than the service. Run the generator separately from the system under test and monitor both.

What this run does not prove

This local result proves that the saved scenario and gates behave as designed. It does not prove production capacity, a safe autoscaling target, or the maximum sustainable throughput.

Before making one of those claims, I would add:

representative production data volume and cache state;
a warm-up phase and a stable measurement phase;
repeated runs of the same commit with variance reported;
step, stress, and soak profiles in addition to steady load;
server-side saturation evidence and a clearly defined stop condition;
a separate test for coordinated omission if request timing requires it.

The practical rule is simple: save the workload, run it without the GUI, preserve raw samples, report percentiles and errors together, and refuse to call one uncontrolled run a capacity number.

The JMeter getting-started guide, dashboard report documentation, and best-practices guide are the primary references for the execution choices above.

Operating a Spring Boot SLO with Prometheus, Alertmanager, and Grafana

ystc1247@gmail.com — Sun, 26 Jul 2026 05:45:00 GMT

Installing Prometheus and Grafana is not an observability strategy. The useful unit is an operating loop: define reliability, collect a bounded signal, turn expensive queries into recording rules, page on sustained budget consumption, and make the whole path testable.

I built that loop around a small Spring Boot 4.1 API. The experiment generated 240 known requests—216 normal, 12 deliberately slow, and 12 failed—then verified the application, scrape target, Prometheus rules, Alertmanager, and provisioned dashboard. The point was not the dashboard screenshot. It was proving that every panel and alert represented a decision I could act on.

Start with the service-level objective

An alert such as “CPU above 80%” says that a resource is busy. It does not say whether users are receiving a reliable service. I started with two user-facing objectives instead:

Indicator	Objective	Error budget
Non-5xx HTTP requests	99.5% over 30 days	0.5% may fail
Request latency	95% below 250 ms	5% may be slower

The availability SLI for one window is:

success ratio = 1 - (5xx request rate / total request rate)

The 0.5% error budget is not permission to ignore failures. It is a common scale for release risk and incident urgency. Consuming it slowly may justify a ticket; consuming it 14.4 times faster than planned is a paging condition.

This lab uses five-minute and one-hour windows so the behavior is visible locally. A production alert policy should use windows derived from the real SLO period, traffic shape, and on-call response time.

Keep the metrics endpoint off the public edge

Spring Boot exposes the Prometheus registry through Actuator:

implementation 'org.springframework.boot:spring-boot-starter-actuator'
implementation 'io.micrometer:micrometer-registry-prometheus'

management:
  server:
    port: 9091
  endpoints:
    web:
      exposure:
        include: health,prometheus
  metrics:
    distribution:
      percentiles-histogram:
        http.server.requests: true
      slo:
        http.server.requests: 50ms,100ms,250ms,500ms,1s

The application listens on 8080; the management server listens on 9091. In the Compose experiment only 8080 is published. Prometheus reaches app:9091 on the private network.

That distinction matters. Do not solve production monitoring by opening Grafana on 3000 and Prometheus on 9090 to the internet. Operator surfaces expose topology, labels, queries, and sometimes control endpoints. Put them behind private networking and authenticated TLS termination. Disable default credentials, restrict administration, and treat Prometheus lifecycle or write endpoints as privileged operations.

The local ports in this experiment are loopback-bound so they cannot accept remote traffic:

Component	Host binding	Purpose
API	`127.0.0.1:18049`	Generate known application traffic
Grafana	`127.0.0.1:13049`	Read the provisioned dashboard
Prometheus	`127.0.0.1:19090`	Inspect targets and PromQL
Alertmanager	`127.0.0.1:19093`	Inspect alert routing
Actuator	Not published	Scraped only inside the Compose network

Cardinality is part of the design

A metric name is only half of a Prometheus time series. Every unique label set creates another series. A label such as user_id, request UUID, raw exception message, or unnormalized URL can turn one useful metric into millions of expensive series.

For HTTP metrics, use bounded labels such as method, status class, service, and route template. /orders/{orderId} is bounded; /orders/6fdb... is not. Put request-specific evidence in logs or traces, where it can be sampled and retained under a different cost model.

Before adding a label, I ask three questions:

Is its value set bounded and understood?
Will I aggregate or alert on this dimension?
Could a trace or structured log answer the same high-cardinality question?

Dashboards and alerts should not each embed a slightly different copy of a long query. Prometheus recording rules evaluate once and store a reusable time series:

groups:
  - name: api-slo-recording
    interval: 5s
    rules:
      - record: job:http_requests:rate5m
        expr: >-
          sum by (job) (
            rate(http_server_requests_seconds_count{uri!="/actuator/prometheus"}[5m])
          )

      - record: job:http_errors:ratio_rate5m
        expr: >-
          sum by (job) (
            rate(http_server_requests_seconds_count{status=~"5.."}[5m])
          ) / clamp_min(job:http_requests:rate5m, 0.001)

      - record: job:http_request_duration_seconds:p95_5m
        expr: >-
          histogram_quantile(
            0.95,
            sum by (job, le) (
              rate(http_server_requests_seconds_bucket[5m])
            )
          )

clamp_min keeps a near-idle denominator from producing an undefined ratio. It does not make a low-traffic SLO statistically meaningful; that still requires enough events or a different indicator.

The latency query aggregates histogram buckets by le before calling histogram_quantile. Client-side percentiles cannot be averaged across instances. Histograms keep the aggregation possible, but bucket boundaries must still match the latency decisions the service cares about.

Page on budget burn, not on a decorative graph

Prometheus evaluates alert rules. Alertmanager then groups, inhibits, silences, and routes alert instances. Grafana visualizes the same state; it is not a replacement for the alert-delivery path.

The local fast-burn rule requires both windows to exceed the same threshold:

- alert: ApiErrorBudgetFastBurn
  expr: >-
    job:http_errors:ratio_rate5m > (14.4 * 0.005)
    and
    job:http_errors:ratio_rate1h > (14.4 * 0.005)
  for: 1m
  labels:
    severity: page
  annotations:
    summary: API is rapidly consuming its 99.5% success SLO error budget
    response: Check recent releases and dependencies; roll back or mitigate first.

At a 99.5% objective, 14.4 × 0.005 is a 7.2% error ratio. The short window detects a sharp event; the longer window reduces sensitivity to a brief spike. Production policies usually combine multiple burn rates and windows so fast incidents page while slower erosion creates a ticket.

Every alert needs an owner and a first action. “API errors high” is weak. “Check the latest release and dependencies; mitigate or roll back before investigating lower-priority causes” gives the responder a useful starting point.

Provision the dashboard as code

The data source and dashboard are committed and mounted into Grafana. A recreated container produces the same panels and queries, so dashboard drift becomes reviewable.

The burn chart is more useful than a wall of JVM gauges because it connects failures to the reliability objective. JVM memory, connection pools, and CPU remain valuable diagnostic panels, but they answer “why” after the service-level signal answers “whether users are affected.”

Verify the entire path

The experiment runs these checks rather than trusting that containers started:

docker compose exec -T prometheus \
  promtool check config /etc/prometheus/prometheus.yml

docker compose exec -T prometheus \
  promtool check rules /etc/prometheus/rules.yml

curl -fsS http://localhost:19090/-/ready
curl -fsS http://localhost:19093/-/ready
curl -fsS http://localhost:13049/api/health

It also queries the Prometheus API for the scrape target and a recording-rule result. A green Grafana container with an empty data source is not a successful monitoring deployment.

The observed traffic mix was deliberately unhealthy enough to exercise the queries:

Request class	Count	Intended signal
Normal	216	Baseline request rate and latency
Slow	12	Upper histogram buckets and p95 movement
Failed	12	5xx ratio and error-budget burn
Total	240	Known denominator for the experiment

This is functional observability validation, not a capacity benchmark. It proves signal wiring and rule behavior.

Put cost into the review

Prometheus cost is driven by active series, scrape frequency, retention, rule evaluation, query load, and replication. Grafana cost is driven by its database, rendering/query activity, plugins, and high availability. Long-term storage adds its own object-store and compaction model.

For every new metric family, estimate:

series ≈ product of label cardinalities
samples/day ≈ series × 86,400 / scrape_interval_seconds

Then measure actual active-series and ingestion changes after deployment. Do not infer a cloud bill from one formula; compression, retention, remote write, replicas, and vendor pricing change the result.

My production checklist is now short and strict:

Define the SLI and SLO before building the dashboard.
Keep labels bounded and route-normalized.
Put shared PromQL in tested recording rules.
Route actionable alerts through Alertmanager with an owner and response.
Keep management and operator surfaces private, authenticated, and encrypted.
Provision dashboards and data sources from version-controlled files.
Measure series growth and ingestion cost after every instrumentation change.

The relevant primary references are the Spring Boot Actuator metrics documentation, Prometheus recording-rule documentation, Prometheus alerting-rule documentation, and Grafana provisioning documentation.

Building PostgreSQL Consistency Lab: From Invisible Race Conditions to an Evidence-Backed Workbench

ystc1247@gmail.com — Sun, 26 Jul 2026 01:00:00 GMT

The failures that do not look like failures

Some of the most expensive database bugs do not begin with an error. Both requests return successfully. Both transactions commit. The logs contain no stack trace. The API emits two 200 responses, but one business operation quietly disappears.

That class of failure bothered me more than an ordinary syntax or constraint error. A broken query that raises immediately is unpleasant, but it is visible. A lost update, write skew, double-claimed job, missing outbox event, or badly chosen index can survive code review and remain invisible until concurrency or traffic gives it the exact conditions it needs. By the time the symptom reaches an operator, the transaction boundary that caused it is gone.

I had encountered the ingredients many times in backend work: transaction annotations that looked correct from the application side, lock waits that only made sense after inspecting PostgreSQL, retries that accidentally repeated a side effect, indexes that improved one read while making every write more expensive, and monitoring graphs whose numbers could be read three different ways. The recurring problem was not a lack of definitions. I knew what READ COMMITTED, SERIALIZABLE, SELECT FOR UPDATE, SKIP LOCKED, partial indexes, and the transactional outbox were. The problem was that definitions are much easier to remember than behavior is to prove.

PostgreSQL Consistency Lab began as an attempt to close that gap.

The first repository description was deliberately narrow: a transaction, lock, and index failure simulator. Its original definition of done for each scenario already contained the important constraints:

start from a production-shaped story;
reproduce the failure deterministically;
implement and run a correct alternative;
measure the result instead of relying on intuition;
preserve lock, SQLSTATE, or query-plan evidence where it matters;
explain when the fix is appropriate and what it costs.

The earliest scaffold framed this partly as backend engineering practice. That was useful for choosing the first scenarios, but it became too small a description of the finished system. I did not want another repository that listed concurrency anomalies in Markdown or showed a polished UI over fixture data. I wanted one local environment where I could create a real PostgreSQL failure, stop at the interesting moment, inspect the database, compare strategies, replay the evidence, and then prove that the lab cleaned up only what it owned.

That requirement changed the project from a scenario runner into a database engineering workbench.

The first architecture was intentionally disposable

The repository was created on July 9, 2026 as a docs-first Kotlin and Gradle scaffold. The initial architecture was small:

CLI
 └── ScenarioRunner
      ├── ScenarioCatalog
      ├── DatabaseFixture
      ├── ConcurrentActors
      ├── ProbeCollectors
      └── ReportWriter

It was a reasonable starting point. PostgreSQL would run in Docker Compose. Each scenario would own a fixture. Concurrent actors would meet at explicit barriers. Probe collectors would retain outcomes, final state, latency, locks, plans, and SQLSTATEs. A Markdown-first report writer would make the evidence portable.

The first ADR selected Kotlin because it was familiar for backend work and expressive enough for the initial harness. Before scenario implementation began, however, the target changed. I wanted a custom dashboard and a local MCP server in addition to the CLI. Long-lived transactions needed to remain attached to real database connections. Workloads needed visible worker and pool state. The MCP adapter needed structured objects rather than console parsing. A single runtime would have to serve several transports without letting each transport invent its own database behavior.

At that point I replaced the Kotlin anchor with Go 1.25 and pgx.

The language change was not about claiming that Go was universally better. It was about making the concurrency model of this particular tool obvious. Goroutines and channels were a compact fit for two database actors meeting at a barrier. pgx exposed PostgreSQL transactions, SQLSTATEs, pools, and extended-protocol execution directly. The official Go MCP SDK could be added later without creating another runtime. Rust would have offered stronger compile-time ownership, but it also would have put more language machinery between the reader and the transaction schedule I wanted the project to explain.

The second ADR therefore established a more durable dependency direction:

CLI / HTTP / MCP
       │
       ▼
shared Go application services
       │
       ▼
pgx / PostgreSQL 16

MCP would be an adapter. HTTP would be an adapter. The CLI would be an adapter. None of them would shell out to another interface or parse human-readable output. This decision looks ordinary in the final repository, but it prevented a great deal of later drift. When emergency stop, sandbox validation, actor ownership, or scenario evidence changed, there was one implementation to update.

The smallest experiment: losing one of two committed updates

The first executable scenario was tx-lost-update. I chose it because it captures the reason the lab exists: a correctness failure can occur even though both transactions commit.

The fixture starts with one account:

create table consistency_lab.accounts (
    id bigint primary key,
    balance bigint not null,
    version bigint not null default 0
);

insert into consistency_lab.accounts (id, balance)
values (1, 100);

Two actors each add 10. The broken implementation reads the balance in application code and later writes the calculated value. Under READ COMMITTED, both actors can read 100, both calculate 110, and both commit an update to 110. The database did exactly what it was asked to do. The business invariant did not survive.

Coordinator        Actor A                 Actor B
    |                  |                       |
    |              BEGIN RC                BEGIN RC
    |              SELECT 100              SELECT 100
    |<------------- both actors ready ----------------->|
    |------------- release update barrier -------------->|
    |              UPDATE 110              UPDATE 110 waits
    |              COMMIT                  lock acquired
    |                                      UPDATE 110
    |                                      COMMIT
    |---------------- final balance: 110 ---------------->|

The important part is the barrier. A test based on sleep(100 * time.Millisecond) might fail on one machine and pass on another. It would demonstrate scheduler luck, not a transaction anomaly. The coordinator releases the updates only after both transactions have recorded the stale read. The final balance of 110 is therefore the expected broken outcome, not a flaky surprise.

The first fix used an atomic update:

update consistency_lab.accounts
set balance = balance + $1
where id = 1;

PostgreSQL serializes the row updates, and the expression is evaluated against the current row version. Both commits are represented, so the final balance becomes 120.

The completed scenario later grew to five strategies:

Strategy	Mechanism	Result and cost to expose
`broken`	application read-modify-write	both commits can produce `110/120`
`atomic`	one in-database update expression	preserves the increment with row serialization
`optimistic_version`	version predicate and conflict detection	preserves correctness by surfacing a conflict
`select_for_update`	pessimistic row lock before calculation	preserves correctness while making waiting explicit
`serializable_retry`	serializable transaction plus bounded retry	preserves the invariant and may expose SQLSTATE `40001`

This set became the pattern for the rest of the catalog. The lab does not label a strategy “fixed” merely because the final value is correct. It also records the mechanism that made it correct: wait, conflict, retry, constraint, plan change, additional index size, or extra ledger write. A repair without its cost is only half an explanation.

Moving from one fixture to marked sandboxes

The original lost-update runner used a shared schema and an advisory lock to prevent two executions from resetting the fixture simultaneously. That was adequate for one bounded CLI scenario. It was not enough for a dashboard that could create schemas, open long transactions, run workloads, build indexes, and accept commands from MCP.

The next vertical slice introduced an explicit PostgreSQL ownership boundary.

Every writable database must contain a lab marker. If the marker is missing or its environment is not lab, mutation is refused. The marker is intentionally simple, because its job is not to provide authentication. Its job is to make an accidental connection to an ordinary database fail closed.

create table if not exists public.pg_consistency_lab_marker (
    marker_id boolean primary key default true check (marker_id),
    instance_id uuid not null,
    created_at timestamptz not null default now(),
    environment text not null check (environment = 'lab')
);

The database then separates durable control records from disposable workload objects:

PostgreSQL 16
├── public.pg_consistency_lab_marker
├── lab_control
│   ├── sandboxes
│   ├── experiments and ordered events
│   ├── actors and actor operations
│   ├── workload runs and samples
│   ├── artifacts
│   └── audit log
└── lab_sbx_<opaque id>
    ├── seeded application tables
    ├── scenario-specific temporary objects
    ├── indexes, functions, and triggers
    └── generated non-login runtime role

User-provided names remain display labels. SQL identifiers are generated. A sandbox schema has the lab_sbx_ prefix and a random opaque suffix. Each sandbox is paired with a generated non-login role that receives privileges only for that schema. Query Lab and composition assertions execute under that role with a sandbox search path, transaction mode, timeouts, and result bounds. The dashboard never receives PostgreSQL credentials.

Creation is transactional. The manager verifies the marker, resolves a named profile, validates the sandbox kind and display label, creates the control record and schema, installs and seeds the template, runs ANALYZE, and finally marks the sandbox ready. If seeding fails, the schema and registration roll back together.

Deletion follows the ownership record in the opposite direction. It loads a registered sandbox, refuses a schema outside the generated prefix, checks for active lab owners, drops the exact schema, and removes the registry entry. It does not accept an arbitrary schema name from the UI. DROP SCHEMA ... CASCADE is powerful, but the dangerous part is not the SQL keyword; it is whether the caller can widen the target.

A production-shaped seed without pretending to be production-sized

I wanted the default dataset to support realistic access patterns without turning a laptop launch into a scale test. The built-in commerce and operations model therefore contains 11 connected tables:

Table	Diagnostic purpose
`tenants`	tenancy, plan, and regional skew
`users`	tenant-scoped identities, status, recency, and JSON profile data
`products`	catalog state, price, lifecycle, and JSON attributes
`inventory`	reservations, sold counts, hot rows, and optimistic versions
`orders`	idempotency, lifecycle, tenant feeds, totals, and versions
`order_items`	multi-row order/product fan-out
`payments`	provider uniqueness, attempts, and payment state
`idempotency_keys`	request and result deduplication
`job_queue`	priority, availability, lease, retry, and worker ownership
`outbox_events`	transactional event publication state
`api_requests`	route, latency, database time, and query-count history

The template is intentionally index-rich. It contains primary and unique constraints, composite B-tree indexes, partial indexes, covering indexes, GIN indexes for JSON data, and BRIN indexes for time-shaped history. Three functions support timestamp maintenance, guarded inventory reservation, and FOR UPDATE SKIP LOCKED job claiming. Triggers update order and payment timestamps.

The smoke profile plans 195,005 rows: 5 tenants, 1,000 users, 2,000 products, 10,000 orders, and 100,000 request-history rows plus their connected data. In one observed run, PostgreSQL reported 11 tables, 36 index relations, 3 functions, and roughly 46.9 MB of total table relation storage after seeding and ANALYZE.

The local profile is much larger—25 tenants, 100,000 users, 100,000 products, 1,000,000 orders, and 5,000,000 history rows, with more than 10 million planned rows overall. It is defined so the same model can grow, but seeding it requires an explicit confirmation. Routine validation does not silently turn into a disk and memory experiment.

This distinction became one of the project’s recurring principles: production-shaped is not the same as production-sized. Skew, access paths, queues, outbox records, JSON attributes, history, and competing reads and writes can reveal the behavior I care about at bounded scale. A larger row count is useful only when the question actually depends on it.

Why a real actor must own a real connection

An HTTP request is stateless. A PostgreSQL transaction is not. Its locks, backend PID, isolation level, temporary settings, and failure state belong to one connection. If the API returned after BEGIN and later chose any pool connection for the next command, the lab would be simulating a transaction rather than holding one.

The actor manager therefore owns a dedicated PostgreSQL connection for each actor handle. The handle is opaque to clients, but its control record contains the sandbox, experiment, backend PID, owner, purpose, transaction state, TTL, cancellation state, and cleanup result.

An actor can begin, run a reviewed operation, commit, roll back, cancel, or close. A blocking database operation runs asynchronously and returns an operation handle, allowing the API request to finish while the PostgreSQL connection remains blocked. The client can inspect the operation and the lock graph separately.

Observation joins the owned identities with real PostgreSQL state from pg_stat_activity, pg_locks, and pg_blocking_pids. The wait-for edge is not reconstructed from a UI label. The lab can say that actor B is waiting on actor A because PostgreSQL reports the exact backend relationship and both PIDs map to registered actors.

Actor safety is deliberately stricter than an ordinary connection pool:

the smoke profile permits at most eight actors;
each actor has a bounded TTL, with an absolute maximum of 300 seconds;
one actor can have at most one pending operation;
operation duration, returned rows, and returned bytes are bounded;
statement, lock, and idle-in-transaction timeouts are set on the session;
cancellation and process shutdown roll back incomplete transactions;
a backend can be terminated only when it is registered as lab-owned;
a failed session reset removes the connection from the pool rather than returning contaminated state.

This work made intentional lock experiments possible. More importantly, it made their cleanup a first-class result rather than a hopeful defer at the edge of an HTTP handler.

A small transaction language instead of arbitrary orchestration

Predefined scenarios are useful, but database incidents often depend on a schedule that is slightly different from the catalog. I wanted to compose those schedules without exposing a general SQL workflow engine.

The transaction composer is a bounded JSON state machine. A composition declares one to eight actors, a TTL, and at most 64 steps. Supported steps are intentionally limited:

begin
execute_operation
wait_at_barrier
release_barrier
commit
rollback
assert_sqlstate
assert_invariant

Operations come from the actor allowlist. An invariant is one bounded read-only SELECT executed under the sandbox role. expectBlocked does not sleep for an arbitrary interval and assume a lock exists. It polls PostgreSQL until the exact actor is blocked, then stores a lock snapshot filtered to the composition’s actor IDs.

The schedule can therefore express a row-blocking experiment like this:

Actor A: BEGIN
Actor A: update inventory row
Actor A: arrive at "lock-held"

Actor B: BEGIN
Actor B: attempt update on the same row asynchronously
Coordinator: prove B is blocked by A in PostgreSQL

Actor A: COMMIT
Actor B: operation completes
Actor B: COMMIT
Coordinator: assert the final inventory version

Every terminal path—success, validation failure, timeout, explicit cancellation, emergency stop, or process shutdown—rolls back and closes every actor opened by the composition. A completed report stores step outcomes and captured evidence as a durable artifact.

This was a better fit than arbitrary SQL scripts for two reasons. First, the limited step vocabulary makes ownership and cleanup inspectable. Second, the JSON schedule is transport-neutral: the dashboard and MCP server can submit the same composition to the same runtime.

Turning the catalog into 16 executable comparisons

The original scenario catalog was a backlog. By the interactive-platform milestone, all 16 entries were executable against real PostgreSQL. Each one has a production story, invariant, broken strategy, fixed strategy, expected outcome, typed evidence, and exact cleanup.

Transaction anomalies

tx-lost-update is the foundational case. It compares stale application read-modify-write with atomic update, optimistic versioning, row locking, and serializable retry.

tx-write-skew models a constraint that spans multiple rows. Two transactions each observe that another row satisfies the invariant, then update different rows. Row-level repeatable reads can be individually consistent while the cross-row business rule fails. The fixed path uses SERIALIZABLE with bounded retry and preserves SQLSTATE 40001 when PostgreSQL aborts a dangerous serialization graph.

tx-capacity-phantom models two requests that both see remaining capacity and both insert. The repair moves the invariant onto a capacity counter protected by a lock instead of trusting two independent check-then-insert decisions.

tx-idempotency-race shows why checking for an idempotency key before doing work is not itself idempotency. Two requests can both observe absence. A unique ledger makes the database arbitrate the race and lets one request own the side effect.

Locks and queues

lock-deadlock-order makes two transactions update the same rows in opposite order. The broken path produces a real PostgreSQL deadlock victim with SQLSTATE 40P01. The fixed path acquires the rows in stable order, so both transactions commit. The comparison retains commit count, victim count, duration, SQLSTATE, and cleanup state rather than showing only a red or green badge.

lock-hot-row-contention compares an application read-modify-write counter with an atomic update while recording correctness and latency. It is a reminder that the same row lock can be both the mechanism that preserves correctness and the source of tail-latency pressure.

queue-double-claim lets two workers select a ready job before either records ownership. The fixed claim uses FOR UPDATE SKIP LOCKED so one lease maps to one worker.

queue-skip-locked isolates a different question: should one worker holding the first job prevent another worker from making progress on the next ready job? The comparison preserves blocking evidence and SQLSTATE 55P03 where relevant.

queue-stuck-lease models the worker that dies after claiming a job. A ready-only query can strand the work forever. The fixed strategy makes expired leases eligible again without stealing a lease that is still live.

Index and plan behavior

idx-bad-composite-order compares a composite index whose leading column does not match the tenant-and-time access path with the correct tenant/time ordering.

idx-partial-comparison compares a full status index with a predicate-matched index over the active subset. The lab records index size and plan evidence instead of assuming that a smaller partial index is always selected.

idx-covering-comparison contrasts a heap-dependent read with an INCLUDE covering index. The report retains heap fetch and buffer evidence where PostgreSQL exposes it.

idx-write-amplification compares minimal indexing with five secondary indexes under writes. This scenario exists because an index review that considers reads only is incomplete. Extra access paths have to be maintained, produce WAL, occupy storage, and alter write latency.

Index scenarios preserve PostgreSQL FORMAT JSON plan nodes rather than parsed console text. Cost, actual timing when requested, rows, loops, buffers, index names, sizes, heap fetches, latency, and WAL evidence remain structured so both the report and dashboard can use them.

Event consistency

event-missing-after-commit shows the gap between a successful business commit and a later process-local publish. If the process fails in between, downstream state never learns about a change that the database already accepted. The transactional-outbox path writes business state and publication intent together.

event-transactional-outbox makes the boundary even more explicit by comparing split transactions with one transaction that owns both the business row and the outbox row.

event-idempotent-consumer replays delivery after partial success. The broken consumer repeats its side effect. The fixed consumer records a durable ledger entry so at-least-once delivery does not become at-least-once business mutation.

The important acceptance rule across all 16 scenarios is slightly counterintuitive: a broken strategy that unexpectedly passes is a failed experiment. The lab is not rewarding green output. It is proving that the intended failure was actually reproduced, that the repair actually held, and that both paths removed their exact fixtures and released their sessions.

Workloads: pressure has a shape

Individual scenarios isolate one invariant. They do not show how several ordinary operations interact under sustained pressure. The workload engine adds that layer without turning the tool into an unrestricted load generator.

A run has separate warm-up, measurement, and cool-down phases. Workers and connection-pool slots are configured independently so connection acquisition pressure remains visible. Traffic can be rate-limited or run as quickly as the bounded worker and pool envelope permits. A deterministic seed controls each worker’s selection path.

The engine includes 12 named presets:

Preset	Pressure shape
`steady-traffic`	read-heavy product, order, session, queue, and outbox traffic
`peak-hour`	recent-order and checkout pressure with tenant skew
`flash-sale`	write-heavy contention on a deliberately small inventory set
`hot-tenant`	one tenant dominates feeds and writes
`hot-product`	one product dominates point reads and version updates
`queue-burst`	concurrent claims, completion, and outbox publication
`reporting-vs-oltp`	aggregation and history scans compete with point work
`index-build-under-load`	normal traffic continues during an index comparison
`function-regression`	stored inventory functions compete with direct updates
`connection-saturation`	short work exposes acquisition and pool pressure
`wal-heavy-writes`	sustained writes produce visible WAL movement
`large-sort-hash`	analytical queries expose temporary-file behavior

Custom mixes combine 14 reviewed operations, from product reads and recent orders to inventory reservations, order creation, payment callbacks, idempotency acquisition, queue claim/complete, outbox publication, aggregation, historical scans, large sorts, and one sandbox-scoped custom read query.

The configuration surface is deliberately detailed because “four workers for ten seconds” says little about the resulting system. It includes workers, pool size, target operations per second, statements per transaction, isolation, seed, uniform or Zipf-like distribution, hot-key and hot-tenant ratios, payload size, think time, retry count and backoff, statement and lock timeouts, and operation, row, WAL, temporary-byte, and latency budgets.

Workers record actual state transitions:

rate_wait
pool_wait
connection_acquire
executing
retry_backoff
think_time
finished

Every completed operation records its selected target, latency, transaction outcome, SQLSTATE, and affected rows. One-second samples contain interval operation/error/transaction/row deltas, fixed latency buckets, worker states, connection-pool pressure, and bounded hot-target aggregates. The final report retains cumulative counts and database baseline, experiment, and recovery deltas.

The distinction between interval and cumulative data matters. PostgreSQL transaction counters and WAL values do not become rates merely because they are drawn as a line. The monitoring service labels current gauges, cumulative counters, derived interval rates, experiment deltas, and recovery deltas separately.

WAL, temporary bytes, and some database counters are cluster-wide signals. The lab uses them as conservative safety budgets and correlation evidence. It does not claim that every observed byte belongs exclusively to the selected workload. That limitation is more useful on the screen than a false level of attribution.

Query Lab, Index Studio, and bounded SQL

The project needed custom queries, but “local only” was not enough reason to expose arbitrary database control. Query Lab therefore accepts one PostgreSQL statement and executes it through the extended protocol. It does not split a string on semicolons and hope for the best.

The request specifies read or write mode, statement and lock timeouts, row and byte limits, and whether to capture a plan. Execution assumes the generated non-login sandbox role. Read mode is transaction-read-only. SQL text is bounded, output is capped at 1,000 rows and 1 MiB, and duration is capped at 30 seconds. Results include normalized query information, SQL digest and preview, rows, bytes, SQLSTATE, timing, WAL delta, and a structured JSON plan artifact when requested.

There is a bounded lexical classifier, but it is only defense in depth. The actual safety boundary is the combination of one extended-protocol statement, transaction mode, sandbox role, PostgreSQL privileges, qualified ownership, timeouts, cancellation, and result limits. Calling a lexical check a full SQL parser would create exactly the kind of false confidence the lab is meant to avoid.

Index Studio performs typed mutations. Creation receives a table, columns, method, optional INCLUDE columns, and a reviewed predicate. Identifiers are built from inspected catalog metadata. Drop refuses constraint-owned indexes. PostgreSQL progress remains visible, although index create/drop is still synchronous at the adapter boundary and does not yet have its own cancellable long-operation handle.

Function Lab is intentionally more observational. It exposes inspected functions, volatility, parallel safety, definitions, and triggers. Function execution can use Query Lab, but arbitrary function creation was not opened merely to make the page look complete.

One runtime, four clients

By the main platform milestone, internal/platform.Runtime composed the sandbox, actor, composer, scenario, workload, query, monitoring, control, and safety services once. Four clients used it in different ways.

The CLI remained the smallest route. list shows the implemented catalog. A run creates an exact temporary custom smoke sandbox, starts the scenario through the shared service, polls the durable experiment, prints text or JSON, and deletes the sandbox. It does not own a second scenario implementation.

The HTTP API exposes loopback-only REST commands and snapshots under /api/v1. Request bodies are bounded, unknown JSON fields are rejected, and domain failures map to stable safe error codes. High-value route families cover sandboxes, scenarios, compositions, actors, locks, workloads, queries, indexes, monitoring, experiments, replay, artifacts, and emergency stop.

Server-Sent Events carry ordered lifecycle notifications. An event is persisted before it is fanned out. Reconnecting clients can replay at most 500 events using Last-Event-ID, and a heartbeat keeps the one-way stream observable. SSE is enough because commands already travel through REST; adding WebSocket would have created a second bidirectional command path without solving a present problem.

The MCP server exposes 32 typed stdio tools over the same runtime. It can discover profiles and capabilities, create and inspect sandboxes, run scenarios and compositions, manage actors, inspect locks, start and cancel workloads, run bounded queries, perform typed index changes, inspect experiments and artifacts, and invoke emergency stop. It does not shell out to the CLI, parse console text, accept credentials, or accept an arbitrary backend PID.

The React and TypeScript dashboard is the complete operator surface. It calls the HTTP API, subscribes to SSE, and never connects to PostgreSQL. The first usable version already had ten operational pages. The visual-observability milestone added an eleventh: Experiment Workspace.

The dashboard had to explain time, not only display numbers

The early dashboard could operate the lab, but diagnosis still required mentally joining metric cards, JSON blocks, tables, and several pages. That was not enough. Database behavior is temporal and relational. I needed to see which phase was active, when tail latency moved, which actor waited on which backend, whether a plan changed, and whether the evidence on screen was live, durable, cumulative, sampled, or unavailable.

The final dashboard contains these workspaces:

Workspace	What it owns
Overview	service health, active sandbox, safety limits, and recent experiment state
Experiment Workspace	synchronized lifecycle, workload, wait, actor, plan, and raw evidence replay
Sandboxes	create, seed, select, inspect, and delete isolated schemas
Scenarios	browse and run all 16 comparisons and inspect artifacts
Transaction & Lock Lab	manage actors, inspect blockers, and run bounded compositions
Workload Studio	configure pressure and inspect live and durable telemetry
Query Lab	execute bounded SQL and inspect plan tree, flame, diff, and JSON views
Index Studio	inspect access paths and run validated create/drop comparisons
Function Lab	inspect functions, volatility, safety, definitions, and triggers
Database Monitoring	sessions, locks, queries, tables, indexes, functions, progress, and rates
Experiment History	durable status, cancellation, artifacts, and replayable evidence

Standard time, area, bar, and line charts use Recharts. PostgreSQL-specific relationships use bounded native SVG and HTML: wait-for graphs, actor lanes, schema topology, plan trees, plan flame views, histogram heatmaps, and worker-state matrices. I did not add a general graph library for data whose maximum size and semantics were already known.

The backend gained a read-only visualization service that composes one experiment with its chronological events, artifacts, plans, workload report and samples, phase ranges, captured lock snapshots, and optional live monitoring. Completed experiments never receive a current monitoring snapshot disguised as historical data. Sandbox topology is a separate bounded catalog DTO containing tables, columns, foreign keys, indexes, functions, triggers, and storage ownership.

The Experiment Workspace uses one inspection cursor across the timeline, charts, workload samples, phases, and the nearest real captured lock snapshot. This sounds like a presentation detail, but it is an evidence contract. If the operator selects one moment, every chronological panel must describe the same interval.

A null array exposed a deeper time-contract problem

One of the most useful bugs appeared between workload creation and the first one-second sample. The Go manager initialized Samples as an empty slice. A clone operation copied the zero-length slice onto nil, so JSON encoded a valid live run as:

{
  "samples": null
}

The TypeScript contract declared an array, and the workload visualization immediately called array operations. The whole page fell into its error boundary during a legitimate empty state.

The direct fix was small: clone into an allocated zero-length slice so the API emits "samples":[]. A regression test checks both non-nil identity and the exact JSON contract. The frontend also normalizes older or transitional null collections at the API boundary, and evidence views remain null-safe.

I could have stopped there. Instead, the failure made another inconsistency obvious. Overview, Monitoring, and Experiment Workspace had one notion of a selected window; Workload Studio and History had none. A line chart could show one minute while its histogram, worker matrix, phase band, or event list still represented the full run.

The fix introduced one shared TimeRange model:

relative: recent 1m, 5m, or 15m
all:      every retained timestamp
custom:   validated absolute start and end

Relative ranges anchor to the latest retained evidence, not the wall clock. Otherwise opening yesterday’s replay and choosing “5m” would produce an empty chart. Each page merges the bounds of its available evidence, resolves one inclusive interval, filters before downsampling, clips overlapping phase bands, and sends the same interval to charts, events, locks, histograms, worker states, and hot-target views.

The selected cursor clears when the range changes. Invalid or out-of-retention custom ranges remain unapplied and show an inline error without destroying the operator’s draft. Switching sandboxes clears browser-only monitoring and event history so evidence cannot cross an ownership boundary. Monitoring history is capped at 360 snapshots—about 18 minutes at the nominal three-second interval—while rendered charts are downsampled to bounded point counts.

The rule underneath all of this is simple: a measured zero and missing evidence are different facts.

If no transaction is waiting in a captured snapshot, zero is meaningful. If the old experiment never stored a lock snapshot, the wait graph is unavailable. If a workload predates latency buckets, the heatmap is unavailable. If the browser was paused, resume does not invent the missed samples. The dashboard says so directly.

Emergency stop is an ownership test

A lab that intentionally creates locks, long transactions, DDL, connection pressure, and expensive queries needs a credible stop path. A global red button that calls pg_terminate_backend on whatever looks busy would be worse than no button at all.

Every HTTP and MCP mutation takes a read side of a shared runtime lease. Emergency stop takes the write side, marks the runtime as stopping, waits for in-flight mutations to leave their critical section, rejects new mutations, cancels active scenarios, compositions, workloads, and queries, rolls back registered actors, and then verifies cleanup.

No API accepts a caller-provided PID for termination. The runtime can act only on backend identities already registered to lab-owned actors or operations. Sandbox deletion is refused while a managed owner remains active. Scenario cleanup drops only the exact objects it tracked. Composer cleanup owns only its actor IDs. The emergency path returns evidence of what it attempted and what remained.

The safety envelope is explicit rather than implied:

Boundary	Limit
shared Go pool	80 connections, 2 minimum, 5-minute idle lifetime
smoke profile	4 workers, 8 actors, 60 seconds
local profile	32 workers, 64 actors, 900 seconds; seeded data requires confirmation
actor	300-second absolute TTL and one pending operation
actor operation	60 seconds, 100 rows, 64 KiB result
composition	8 actors, 64 steps, 120 seconds plus profile cap
workload	10,000 ops/s, batch 100, 10 retries, 5,000,000 operations plus budgets
workload evidence	600 samples and 50,000 retained latency observations
Query Lab	30 seconds, 1,000 rows, 1 MiB result, 32 KiB SQL
events and artifacts	500-event replay, 64 KiB event, 2 MiB artifact

The numbers are not presented as ideal limits for every machine. They are reviewable defaults that prevent an innocent dashboard input from becoming an unbounded operation.

Validation had to use PostgreSQL, not a mock of PostgreSQL

Unit tests cover validation, catalogs, adapters, serialization, transforms, and frontend components. They cannot prove a deadlock, a serialization failure, a blocker edge, SKIP LOCKED ownership, a query plan, or session cleanup.

The integration suite therefore runs against the real Docker PostgreSQL 16 instance. It covers actor blocking and TTL expiry, composer barriers, every scenario report, workload phases and safety budgets, Query and Index boundaries, audit and ordered events, REST and MCP parity, cancellation, emergency stop, CLI temporary-sandbox cleanup, and profile seeding. Selected concurrency packages also run under Go’s race detector using a cgo-capable container.

The dashboard has separate type checking, unit tests, and a production build. The final time-range delivery passed five frontend test files with 15 tests. The production build kept chart code in lazy chunks so Recharts did not enter the initial application bundle.

The most important tests assert cleanup as part of success. The all-catalog integration run executed all 16 entries against one exact custom smoke sandbox, confirmed every broken and fixed expectation, matched the number of comparison artifacts to the catalog, and then verified that no tracked scenario tables or experiment sessions remained.

The main platform acceptance run observed, among other results:

lost update produced 110/120 for the broken path and 120/120 for all four repairs;
serializable retry exposed a real 40001 conflict;
the deadlock comparison produced one 40P01 victim under opposite ordering and two commits under stable ordering;
queue blocking exposed 55P03 evidence;
an event-consumer comparison produced two side effects for two broken deliveries, versus one side effect and one ledger row for the fixed path;
a row-lock composition completed ten of ten steps, preserved its invariant, and left zero actors;
one query/index run changed a sequential scan plus sort at 1.461 ms and 144 shared hits to an index scan at 0.155 ms, 8 hits, and 2 reads in that local state;
emergency stop ended a custom serializable workload in about 517 ms and left zero pcl/% backends.

The visual-observability acceptance run later recorded a flash-sale workload with 8,646 attempts, 8,643 successes, 3 timeouts, 1 retry, p95 latency of 6.237 ms, and 1,886,284 observed WAL bytes. Its query-plan comparison changed an estimated root cost from 4,907 to 8.34. The responsive dashboard fit a 430-pixel viewport without document-level horizontal overflow.

These are useful results because they prove the evidence path in one controlled environment. They are not PostgreSQL capacity claims. Warm local caches, Docker Desktop, one PostgreSQL version, a smoke dataset, and short measurement windows cannot predict production throughput or latency. The article would be less accurate if it presented the most impressive number without its boundary.

The project failed in useful ways while it was being built

The clean final architecture hides several failures that influenced it.

The first was architectural. The Kotlin scaffold was replaced before it accumulated real implementation. Keeping it only because it was the first choice would have made the later dashboard and MCP work harder to compose. The lesson was not “always use Go”; it was that a scaffold should be cheap enough to discard when the control-plane requirements become clearer.

The second appeared in concurrent integration tests. A workload test used a 64 MiB cluster-WAL ceiling while index scenarios were running in parallel. The workload could stop because another valid lab experiment moved the same cluster-wide counter. That was not merely test flakiness. It exposed the difference between a conservative safety signal and exact per-run attribution. The tests were split so one path uses a deliberately high shared-counter ceiling and another dedicated run proves the one-byte WAL stop.

The same concurrent suite found composer artifacts containing unrelated global actor edges. The fix filtered lock snapshots to the exact composition actor IDs. Again, the failure improved the product contract: evidence that is correct globally can still be wrong for a scoped experiment.

The dashboard proxy intermittently returned 502 through a Docker Desktop host-hairpin route even though direct API and container-to-container health checks passed. The application services were placed on an idempotent named Docker network, and Vite now proxies directly to the API container by DNS while published ports remain loopback-only.

The samples: null crash demonstrated that a typed frontend cannot repair a dishonest transport contract by declaration alone. Fixing both producer and consumer was necessary because durable or older payloads can outlive the backend bug.

The time controls exposed a subtler interaction problem: controlled native date inputs could lose an in-progress edit during a periodic parent render. The final control lets the browser own the draft until Apply, then validates and commits it to shared range state.

Each of these failures points to the same engineering rule. Observability is not the act of drawing data. It is preserving ownership, time, scope, and absence all the way from the database to the screen.

What I would not claim after building it

PostgreSQL Consistency Lab is intentionally broad, but it is not a production database administrator.

It does not connect to a real production database. It has no remote authentication model, no multi-user tenancy, and no unrestricted database, role, extension, or server control. MCP is local stdio. The API and dashboard bind to loopback. Open transactions do not survive control-plane process restart.

Monitoring history outside an experiment remains browser-memory-only. If the browser was closed, a 15-minute selection cannot recover samples that were never collected. Durable experiment replay is the historical source of truth.

Lock history exists only where a composition or capture path persisted a snapshot. A completed scenario report cannot reconstruct every intermediate lock edge after the fact. Actor lanes also cannot recreate metadata that was never stored in older events.

Index creation and deletion remain synchronous. A large index build can be observed through PostgreSQL progress, but it does not yet have a dedicated cancellable operation handle. Plan-node matching intentionally leaves a replaced scan strategy unmatched rather than inventing a false correspondence.

WAL, temporary-file, and database counters are useful cluster-wide evidence, not perfect workload attribution. Query Lab’s lexical classifier is not a complete SQL parser. Local timings are comparative evidence, not capacity promises. The guarded local profile is defined, but the default validation story remains the smoke profile.

Those limits are not an appendix added to make the project sound responsible. They are part of its design. The lab became useful precisely because it stopped pretending that every visible number had the same provenance.

What the project became

The repository began with a list of database failures that I wanted to reproduce. It ended with one shared Go runtime, a marked PostgreSQL instance, generated sandbox schemas and roles, durable control records, managed connection-bound actors, a bounded transaction composer, 16 deterministic scenario comparisons, 12 workload presets, 14 reviewed operations, Query and Index Labs, PostgreSQL monitoring, replayable events, 32 MCP tools, and an eleven-workspace React dashboard.

The size of that list is not the part I value most.

The important part is that the system connects a business invariant to the database event that violated it. It can show that two successful commits produced the wrong balance. It can stop while one actor is genuinely blocked by another. It can retain the SQLSTATE that made a retry necessary. It can compare a query plan without erasing the write cost of the index. It can display zero when zero was measured and unavailable when nothing was captured. It can generate dangerous conditions and still refuse to touch an unregistered backend.

That is why the dashboard did not become decoration around the scenario runner. It became the place where all of those evidence boundaries meet.

What I originally wanted was a better way to study transaction anomalies. What I actually built was a local environment for asking a harder question: when a database system says that something succeeded, what evidence is required before I believe that the business invariant survived?

Repository: PostgreSQL Consistency Lab

[Quant Trading System 5/5] Confirming no_trade with Research Gates and Multi-Horizon Diagnostics

ystc1247@gmail.com — Sat, 25 Jul 2026 02:40:00 GMT

The More Candidates Exist, The More Rejection Rules Matter

After the early baselines and sensitivity passes, the candidate universe grew. There were time-of-day reversals, session-extreme reversals, MACD crosses, ES/MES spreads, volatility compression rules, and roll-week relative-value ideas. Some were validation-positive. Some beat random baselines in null tests. Some retained validation net even after execution stress.

At this point, the system needed clearer rejection rules more than more candidates. As the candidate universe grows, the human naturally wants to look at the most attractive row: highest validation net, best profit factor, or most plausible name. That is why the quant trading system needed a research gate dashboard.

The goal was simple:

put every scored candidate in one table
record the first gate that kills each candidate
keep the test split locked
include pathwise execution stress
keep no_trade until every gate passes

The dashboard was less a visualization and more a decision artifact. Its job was not to say “this candidate looks good.” Its job was to say “this candidate died here.”

Fresh Hypotheses And Null Tests

Before the research gate dashboard, I added fresh OHLCV hypothesis families. Instead of endlessly tuning old parameter-sweep rows, I created more economically motivated families: time-of-day open reversal, volatility compression breakout, ES lead-lag, liquidity regime trend, MES/ES relative value, and roll-week transition.

The best validation-looking row was time_of_day_open_reversal_long.

best fresh validation hypothesis: time_of_day_open_reversal_long
validation net: 114.620 points across 60 trades
validation average net: 1.910333 points
validation top-5 day concentration: 0.385194
same hypothesis train net: -418.882 points
inner train-fold positive share: 0.0

On validation alone, this looks decent. Its top-five day concentration is much lower than the previous exit-label winner. Some validation rows also beat random baselines in null tests. But train net and inner train-fold stability remained the blockers.

The important decision was not “it beats a null, therefore it passes.” Null tests are necessary, not sufficient. If train is negative and inner train folds are unstable, beating random timestamps on validation does not make a candidate.

Market Advantage Scorecard

The market advantage scorecard then collected validation net, random-baseline percentile, execution feasibility, train failure, and concentration into one ranking surface.

The top rows looked like this:

Candidate	Score	Validation net	Main failure
`time_of_day_open_reversal_long`	11.0	114.620	train net, inner folds
`session_extreme_reversal_short...score_ge_50p0`	11.0	67.576	train net, inner folds
`time_of_day_open_reversal_short`	11.0	43.142	train net, inner folds
`horizon_30m_mes_vs_es_spread_gt_0p0001`	10.0	175.124	train net, folds, concentration

The interesting thing is that a high score is still not a pass. The advantage score is useful for ranking candidates worth inspecting. It does not decide whether a candidate is tradeable. Candidate promotion must depend on gate survival, not score.

The final scorecard selected policy was again no_trade.

Failure Attribution

The top candidates were then decomposed with failure attribution.

Candidate	Validation-positive evidence	Failure evidence
`time_of_day_open_reversal_long`	validation net `114.620`	train net `-418.882`, worst train era `train_middle` at `-185.044`, fold share `0.0`
`session_extreme_reversal_short...`	validation net `67.576`	train net `-151.662`, worst train era `train_late` at `-85.386`, fold share `0.333333`
`horizon_30m_mes_vs_es_spread_gt_0p0001`	validation net `175.124`	train net `-2,894.096`, worst train era `train_middle` at `-1,300.972`, top-five validation day concentration `0.555726`, fold share `0.0`

This attribution was not meant to make candidates look better. It was meant to make their failure understandable. “Failed train net positive” is useful, but knowing the train era, session segment, and concentration pattern gives the next research question a sharper shape.

The Regime Map Was Not An Entry Generator

I also built an OHLCV regime map. It split market conditions by volatility, volume, session phase, roll phase, trend/chop state, ES trend direction, and ES/MES divergence.

The rule was that the regime map was not an entry generator. If I inspect validation and then pick favorable cells, I have created another overfit surface. So I treated the regime map only as an avoid-regime design surface.

The report conclusion followed that rule:

The regime map did not create entries.
The strongest validation single-factor rows still had negative average directional net after fixed OHLCV costs.
Any future regime filter must be frozen before validation and retested as a new deterministic candidate.

I think sentences like this create project quality. Some analyses exist not to create a result, but to constrain the next analysis so it does not fool itself.

Pathwise Execution Stress

The MBP execution diagnostics were then attached to OHLCV candidate trade paths. Active-symbol five-second p90 adverse penalties were about 0.625 points per side, while 30-second p90 adverse penalties were about 1.375 to 1.625 points. All-MES calibration was harsher.

Once this stress is applied, validation-positive candidates weaken quickly.

fixed OHLCV cost assumptions are not enough for candidate promotion
active-symbol 1000ms stress is now a gate
all-MES stress remains a stricter optional gate

At this point, the candidate dashboard was no longer just a train/validation PnL table. It became a control panel combining cost, nulls, concentration, fold stability, and execution stress.

Gate Dashboard Results

The final research gate dashboard produced 19 candidate rows and 171 gate rows.

Metric	Value
Dashboard rows	19
Gate matrix rows	171
Passing candidates	0

The main failed-gate counts were:

Gate	Failed candidates
`inner_train_folds_stable`	19
`train_net_positive`	18
`pathwise_active_1000ms_positive`	15
`beats_one_random_p95`	14
`cost_stress_net_positive`	14
`validation_net_positive`	14
`random_baseline_percentile_gte_0p50`	12
`day_concentration_lte_0p50`	6

Even the first row of the dashboard did not pass.

time_of_day_open_reversal_long
advantage_score: 11
train_total_net_points: -418.882
validation_total_net_points: 114.620
pathwise_active_1000ms_net_points: 77.120
primary_kill_reason: train_net_positive
passes_advantage_gate: false

This row captures the message of the whole phase. Validation is positive, and validation net remains after pathwise stress. But train fails. Therefore it is rejected.

What The Dashboard Is For

Building this dashboard made the research loop calmer. When a candidate looks good, I do not have to manually restart suspicion from zero. The gate matrix tells me where it dies. And if it does not pass, it stays behind no_trade no matter how interesting it looks.

In trading research, a dashboard should be decision discipline, not decoration. It matters less which candidates survived, and more why candidates were rejected. In this phase, none survived. But the remaining standard became much stronger.

The current policy remained no_trade. That result is not a failure. It is evidence that the gates did their job.

The dashboard established a consistent rejection discipline, but one final question remained. I needed to know whether the conclusion depended too heavily on the five-minute horizon, and whether the same frozen entries and gates would survive a change in the time axis.

Why I Revisited The Horizon

By this point, the system had become fairly strict. Raw tick data had been turned into a silver dataset. One-minute OHLCV and MBP-based execution diagnostics were in place. Several hypothesis families were evaluated on the same chronological split. The result was no longer judged only by whether validation net was positive. A candidate had to survive the no-trade baseline, sign flip, random timestamp tests, same time-of-day random day tests, day shuffling, and execution stress before it could move forward.

A candidate could be weak at five minutes but structurally meaningful at thirty or sixty minutes. The opposite could also be true. A sixty-minute candidate could look good simply because holding longer allowed a lucky validation window to dominate the result. So in the last stage, I did not create new candidates. I kept the existing candidate entry timestamps fixed and changed only the label horizon.

The important part was not treating this as a new search surface. If I looked across horizons and picked whichever validation result looked best, the system would become another overfitting machine. The question I wanted to ask was not “which horizon made money?” It was “does the same candidate survive when the time axis is moved slightly?”

Horizon Was Not A Search Surface

The diagnostic scope was fixed on purpose. The horizon grid was 1m, 3m, 5m, 10m, 15m, 30m, 60m, and the candidate universe was every row in market_advantage_scorecard_v1. The entry policy was also fixed. The same candidate entry timestamps were relabeled. Entries were not reselected, and thresholds were not retuned by horizon.

Another important rule was train-first evaluation. If I found a good validation horizon first and checked train afterward, validation would already have been used as a search surface. So the order was reversed. Train net, inner train folds, and train adjacent-horizon stability were checked before validation gates. The test split remained locked and excluded.

The code makes that intention explicit.

HORIZON_GRID_MINUTES = (1, 3, 5, 10, 15, 30, 60)

GATE_ORDER = (
    "train_net_positive",
    "inner_train_folds_stable",
    "adjacent_train_horizon_positive",
    "validation_net_positive",
    "adjacent_validation_horizon_positive",
    "validation_day_concentration_lte_0p50",
    "validation_beats_no_trade",
    "validation_beats_all_random_medians",
    "validation_beats_one_random_p95",
    "validation_beats_sign_flip_and_no_trade",
    "mbp_active_1000ms_cost_stress_positive",
    "execution_feasibility_gte_0p50",
)

This gate order is more than a code constant. It is a research posture. It says that the prettiest validation number does not get first priority. A candidate that loses money on train, fails inner folds, or cannot survive nearby horizons does not move into model training just because it looks attractive in a validation table.

First, The Five-Minute Label Was Locked Again

Before running the multi-horizon diagnostic, I recomputed the existing five-minute label. If that check had shown a mismatch, every later horizon comparison would have been unstable. The five-minute label in the existing training matrix had to match the newly computed five-minute label exactly so that the rest of the horizon grid could be interpreted on the same basis.

Metric	Value
Matrix rows	281,145
Compared rows	281,145
Missing 5m label rows	0
Mismatches	0
Max gross abs diff	0
Max net abs diff	0

I liked this table not because the numbers were impressive, but because nothing happened. All 281,145 rows were compared, and the five-minute gross/net label difference was zero. That meant the multi-horizon result was not an artifact of a silently changed label path.

I keep adding checks like this for a simple reason. In quant research, the most dangerous failure is not a model failing openly. It is a data generation path changing quietly while the researcher mistakes the movement for performance. The label recompute check was a small guard against that mistake.

The Best-Looking Validation Rows Still Failed

The most tempting part of the report was the set of top validation-looking rows. A few candidates were clearly positive on validation. At the sixty-minute horizon especially, some rows looked good enough to make training feel plausible.

candidate_id	horizon	train net	validation net	active 1000ms stressed net	primary kill
`horizon_30m_mes_vs_es_spread_gt_0p0001`	60m	-1980.15	402.11	312.735	`train_net_positive`
`vol_compression_breakout_long`	60m	-426.906	330.782	101.657	`train_net_positive`
`vol_compression_breakout_short`	60m	-134.448	180.368	84.493	`train_net_positive`
`time_of_day_open_reversal_long`	30m	-206.882	155.37	57.87	`train_net_positive`

If this table were read only from the validation side, it would be attractive. The first candidate had 402.11 validation net points, and still had 312.735 points after active MES 1000ms stress. vol_compression_breakout_long also showed 330.782 validation net points.

But all of these candidates died for the same reason. Their train net was negative.

This was the point where I felt the most temptation. When validation is good, some null tests are beaten, and execution stress still leaves a positive number, it is very easy to want to train a model. But changing the rule at that moment would mean letting the result rewrite the research process. What I wanted from this project was not a way to force one candidate through. I wanted a process where the pass conditions were defined in advance, and only candidates that survived those conditions could move forward.

Beating A Null Test Does Not Override The Train Gate

One interesting detail was that some candidates did beat validation null tests quite well. For example, the sixty-minute version of horizon_30m_mes_vs_es_spread_gt_0p0001 produced 402.11 validation net points and beat no-trade, sign-flipped direction, same trade count random timestamps, circular shifts, same time-of-day random days, and shuffled day labels. Against random timestamps, its actual percentile was 0.995.

{
  "candidate_id": "horizon_30m_mes_vs_es_spread_gt_0p0001",
  "horizon_minutes": 60,
  "actual_total_net_points": 402.11,
  "same_trade_count_random_timestamps_p95": 215.547,
  "actual_percentile_vs_random_timestamps": 0.995
}

Even so, the candidate did not pass. The reason is simple. A null test checks whether the validation result looks accidental under that validation setting. It does not give a candidate permission to ignore a negative train result.

vol_compression_breakout_long showed a similar pattern. Its sixty-minute validation actual net was 330.782 points, but the same trade count random timestamps p95 was 345.307 points, and the actual percentile was 0.945. It also failed the p95 threshold under same time-of-day random days. Even a good-looking candidate did not beat every null test.

This made the role of multi-horizon diagnostics clearer. Extending the horizon should not be a way to rescue a candidate. It should be a pressure test that reveals whether the candidate shines only on one convenient time scale.

Execution Realism Reduced The Edge Further

The last layer was MBP-based execution stress. This is not a perfect simulation of fills, but adding active MES symbol assumptions, 1000ms latency, and adverse movement penalties shows how much validation net survives once execution becomes less idealized.

candidate_id	horizon	trades	base net	stressed net	avg adverse penalty	feasibility
`horizon_30m_mes_vs_es_spread_gt_0p0001`	60m	55	402.11	312.735	1.625	0.782653
`vol_compression_breakout_long`	60m	141	330.782	101.657	1.625	0.782653
`time_of_day_open_reversal_long`	5m	60	114.62	77.12	0.625	0.912182

Some candidates remained positive even after this stress. But positivity alone was not the point. vol_compression_breakout_long fell from 330.782 base net to 101.657 stressed net. It was still positive, but a large part of the apparent edge depended on execution assumptions.

Numbers like this should lead to more questions, not immediate model training. Did the candidate capture a real market structure, or did a validation window, horizon choice, latency assumption, and specific date mix leave behind a fragile positive residue? The kind of result I kept trying to avoid was exactly this ambiguous positive number. A negative result is simple. The difficult result is a positive number that becomes convincing if the explanation is adjusted slightly.

The Gate Matrix Made The Decision

The full gate failure count made the conclusion clearer.

gate_name	failed rows
`inner_train_folds_stable`	132
`train_net_positive`	120
`adjacent_train_horizon_positive`	118
`mbp_active_1000ms_cost_stress_positive`	112
`validation_beats_one_random_p95`	106
`validation_net_positive`	98
`adjacent_validation_horizon_positive`	88
`validation_beats_all_random_medians`	78
`validation_day_concentration_lte_0p50`	38
`execution_feasibility_gte_0p50`	7

The most frequent failures were inner_train_folds_stable, train_net_positive, and adjacent_train_horizon_positive. That matters. The candidates were not failing only on validation. They were already breaking at earlier stability gates.

So the final decision was recorded like this.

{
  "candidate_ready_for_test_or_ml": false,
  "horizon_minutes": null,
  "reason": "no candidate horizon passed train, fold, adjacent-horizon, validation, null, cost-stress, and MBP feasibility gates",
  "selected_candidate_id": "no_trade",
  "source_benchmark": "none"
}

This JSON is disappointing, but I like it. The research system did not say, “one candidate looks good, so train a model anyway.” It passed every candidate through the same procedure and recorded that no candidate passed.

Not Training A Model Is Also A Result

After this stage, I still did not open the test split. I did not train an ML model. I did not start paper trading. From the outside, that can look like the project stopped. To me, it is the opposite. The system finally gained the ability to decide not to act.

The dangerous flow in quant research is usually similar. Generate many candidates, find one that looks good on validation, attach a plausible explanation, train a model, and keep running more backtests. That flow is fast and enjoyable, but almost every step can hide overfitting.

The multi-horizon diagnostic was built to interrupt that flow. Keep the same entry policy across horizons. Look at train first. Check adjacent horizons. Add null tests. Add execution stress. Only then allow a candidate to move forward. If the result is no_trade, that is still a legitimate system output.

In fact, the conclusion made the next step clearer. What the system needs now is not model training. It needs better candidate generation. Better features, clearer market microstructure hypotheses, stricter regime segmentation, and a more realistic execution model should come first. A model is a tool for compressing an edge when an edge exists. It is not a machine for creating edge where none has been demonstrated.

So the current conclusion of this series is simple. The Quant Trading System is not ready to trade yet. But it is ready to refuse weak trades. I think that difference matters more than it first appears.

[Quant Trading System 4/5] Decomposing the Train-Validation Gap with MBP Execution Reality

ystc1247@gmail.com — Sat, 25 Jul 2026 02:30:00 GMT

Stopping At “Validation Good, Train Bad” Is Not Enough

When the parameter sweep produced its first good-looking row, the easy reaction would have been one of two extremes. Either trust validation and push forward, or reject it immediately because train is negative. Neither was enough.

The row was a session-extreme reversal short rule:

session_extreme_reversal_short__skip_open_1030_1500__max1__score_ge_50p0

The visible result was split sharply:

Split	Trades	Net points	Avg net points
Train	169	-151.662	-0.897
Validation	38	67.576	1.778

A gap this large should not be filed away as just “train/validation gap.” I needed to know whether it was regime change, entry-timing artifact, cost-model fragility, exit-path calculation error, or a contract-period effect. Otherwise, the next research step would be blind.

The Full Trigger Family Was Bad In Both Splits

The first step was to inspect what the selected sweep actually chose. The important finding was that the whole signal family was not good. If I looked at all qualifying triggers, both train and validation were negative.

train all triggers: 4,317 rows, -4,786.866 net points
validation all triggers: 880 rows, -416.490 net points

The selected sweep only takes the first qualifying trigger per day.

train first trigger per day: 169 trades, -151.662 net points
validation first trigger per day: 38 trades, 67.576 net points

So the validation result was not evidence that the whole session-extreme short family worked. It was evidence that a first-trigger timing rule worked in one validation period. That distinction matters. If all triggers had been positive, I could trust the family more. Here, entry timing was carrying the conclusion.

There was another tempting detail. If I selected max-score or last triggers, both train and validation looked strong:

train max-score trigger per day: 687.088 net points
validation max-score trigger per day: 296.326 net points
train last trigger per day: 606.838 net points
validation last trigger per day: 222.576 net points

But that is diagnostic, not a candidate. Max-score and last-trigger selection can easily require knowing the rest of the day. If the entry rule cannot know it at entry time, it is not tradeable. Good-looking numbers did not justify promoting it.

Time Of Day And Contract Periods

Next I split the result by time of day and contract period. Train losses were especially concentrated in the 14:00 ET hour.

train 14:00 ET: 45 trades, -98.910 net points
validation 14:00 ET: 5 trades, -18.240 net points

Validation gains came mostly from 11:00 ET through 13:00 ET.

validation 11:00 ET: 7 trades, 23.264 net points
validation 12:00 ET: 14 trades, 46.028 net points
validation 13:00 ET: 11 trades, 19.272 net points

Contract periods also gave clues. Train losses clustered in specific MES contracts:

MESU2: 24 trades, -65.952 net points
MESM4: 8 trades, -41.484 net points
MESH2: 24 trades, -29.702 net points
MESH3: 15 trades, -27.970 net points

Validation was mostly MESH5 and MESM5:

MESH5: 14 trades, 25.778 net points
MESM5: 21 trades, 46.292 net points

This does not prove the answer by itself. But it is enough to suspect that the validation period matched a narrow market condition. A good research system should turn that suspicion into a gate before turning it into a signal.

Hidden Issue: Short Exit Paths Were Calculated Like Longs

The deep dive also found a real bug. The source five-minute parameter-sweep trades already handled short direction correctly, so this bug did not explain the original -151.662 train result.

But the alternate exit-path diagnostics in OHLCV robustness were calculating short time exits, MFE, MAE, and stop/target first-touch outcomes as if every lead were long. That could distort later exit-label sensitivity.

After the fix, exit paths became direction-aware:

short time exits: direction-aware
short MFE/MAE: direction-aware
first-touch stop/target: direction-aware
unit coverage: added for short-direction exits

This kind of bug can inflate returns or kill a valid candidate. The point was not to find a bug to excuse the validation winner. It was to make the diagnostic surface trustworthy before candidate promotion.

Configured-Positive Still Failed Cost Stress

After the direction-aware rebuild, a session-extreme short 15-minute exit variant briefly looked promising:

variant:
parameter_session_extreme_reversal_short...time_exit_15m...session_early_or_middle

configured train: 27.230 net points over 115 trades
configured validation: 34.558 net points over 29 trades

At first glance, this seems to pass both train and validation. But the average train edge was too small.

configured average train edge: 0.236783 points per trade
required dominant-active p99 extra stress: 0.250000 points per trade
extra stress cost: 115 * 0.25 = 28.750 points
stressed train net: 27.230 - 28.750 = -1.520 points

The result is mathematically clear. The configured edge was smaller than one extra tick of spread stress. A candidate like that should not proceed to model training. Saying “train and validation are slightly positive” becomes much less meaningful when one extra tick turns train negative.

Filter Hardening: 13,650 Rows Checked, 0 Passed

I did not stop immediately. Without buying more data, I tried adding one train-derived filter at a time to see whether the candidate could be hardened. Thresholds were computed from train only, and the candidate had to pass early, middle, and late train folds.

The result was again no_trade.

Metric	Value
Candidate filter rows checked	13,650
Prefold pass rows	66
All-gate pass rows	0
Selected filter	`no_trade`

Even the best-looking prefold filter broke in late train:

Split/Fold	Dominant p99 net
Train total	25.726
Validation total	72.308
Train early	63.074
Train middle	8.826
Train late	-46.174

This was fairly decisive. Early train and validation looked good, but late train was negative. The validation strength had not repeated across the whole train period.

Conclusion: Temporal Instability And Cost Fragility

The conclusion of this deep dive was more specific than “this candidate does not work.”

The signal family has a conditional reversal edge that appears only in some market periods.

The original selected sweep was negative on train and positive only on validation. The full trigger family was negative in both splits. The direction-aware bug fix did not explain the original train failure. The configured-positive variant failed one tick of p99 cost stress. Filter hardening checked 13,650 rows and found no row that passed all inner train folds.

So the decision stayed the same:

selected policy: no_trade
train/test split: test untouched
model training: not started
paper trading: not started

What I liked about this phase was that the failure became clearer. It was no longer just “validation good, train bad.” It became “a fragile entry-timing reversal family that fails late train and cannot survive one tick of extra cost stress.”

Once failure is explained that concretely, not training a model becomes much easier.

The analysis made the candidate’s temporal instability concrete. But OHLCV bars could not say whether the extra tick used in p99 stress was conservative or optimistic in the actual market. The next step therefore used MBP-1 not as a feature source, but as an execution-calibration surface.

OHLCV Backtests Do Not Prove Fills

OHLCV backtests are fast and convenient. I can build signals from one-minute bars, use the close five minutes later as the label, subtract costs in points, and run research quickly. But that does not prove the trade could be executed. For MES, where tick size is small and the strategy horizon is short, top-of-book spread and latency can change the entire conclusion.

The next stage of the quant trading system therefore used MBP-1 data as an execution-realism calibration surface. I deliberately say calibration surface, not feature source. The MBP coverage was mostly close to the canonical test period, and it was not a timestamp-matched train/validation fill dataset for older OHLCV trades. So I did not mix MBP into labels or model features.

Instead, I changed the question:

How realistic is the current top-of-book spread assumption?
Does a marketable order still satisfy that assumption after 0ms, 100ms, 500ms, or 1000ms latency?
How large is adverse mid movement over short horizons?
How different are dominant active MES/ES contracts from the all-symbol universe?

This does not find a strategy. But it is very useful for killing candidates.

First, The MBP Cleaning Policy

I did not use MBP data directly because the raw anomaly report had already found crossed top-of-book rows and negative prices. The first MBP cleaning policy therefore defined a contract for clean outright ES/MES top-of-book rows.

exclude spread symbols
exclude negative prices
drop missing or nonpositive top-of-book rows
drop locked books
drop crossed books
keep outright ES/MES rows only

The result was:

Metric	Value
Raw MBP rows	387,252,384
Outright ES/MES target rows	386,559,005
Clean outright top-of-book rows	386,556,485
Spread-symbol rows excluded	693,379
Negative-price rows flagged	78
Nonpositive/missing top rows dropped	1,884
Locked top rows dropped	69
Crossed top rows dropped	859
Event dates	29
Symbols	25

The important thing is not only that there are many clean rows. It is that each rejected row has a reason. If a later p99 spread report looks strange, I need to know whether the cause is a crossed book, stale contract, or real tail spread.

Static Spread Realism

The first execution-realism report measured spread and top-size distributions across clean top-of-book rows.

clean top-of-book rows: 386,556,485
event dates: 28
symbols: 10
median spread: 1.0 tick
p90 spread: 2.0 ticks
p99 spread: 7.0 ticks
rows within assumed 1 spread tick: 0.875708
rows inside full 0.998 point round-turn cost: 0.943512
dominant MESM6 rows within assumed spread tick: 0.931609
dominant ESM6 rows within assumed spread tick: 0.931816

The result cuts both ways. A median spread of one tick means the basic spread assumption is not absurd. But p90 is two ticks and p99 is seven ticks. Across the all-symbol universe, only about 87.6% of rows are inside the one-tick spread assumption.

So fixed OHLCV costs may be fine as a baseline assumption, but they are not enough for candidate promotion. If a validation edge is small or exposed to inactive contracts, tail spread can erase it.

Latency And Adverse Movement

The MBP execution diagnostics went one step beyond static spread. It created 30-second anchors, then used as-of joins to inspect bid/ask snapshots after 0ms, 100ms, 500ms, and 1000ms latency. It then measured mid-price drift and adverse markout over 1s, 5s, 30s, and 60s horizons.

The key is to inspect the quote when an order may actually arrive, not only the quote at the signal timestamp.

LATENCY_OFFSETS_MS = (0, 100, 500, 1000)
HORIZON_OFFSETS_MS = (1000, 5000, 30000, 60000)
MAX_EVENT_STALENESS_MS = 2000

diagnostic["market_buy_entry_px"] = diagnostic["latency_ask_px"]
diagnostic["market_sell_entry_px"] = diagnostic["latency_bid_px"]
diagnostic["market_buy_adverse_after_fill_points"] = (
    -diagnostic["market_buy_entry_markout_points"]
).clip(lower=0.0)

The result was stricter than static spread:

Metric	Value
Anchor rows	60,035
Diagnostic rows	897,944
Symbols	10
Median latency spread	2.0 ticks
p90 latency spread	4.0 ticks
p99 latency spread	27.0 ticks
Latency spread within one-tick assumption	0.484892
Avg absolute horizon mid drift	0.638999 points
Avg market-buy adverse mid movement	0.313413 points
Avg market-sell adverse mid movement	0.325586 points

Across the all-symbol universe, the one-tick assumption breaks often. But dominant active contracts look different. Contracts such as MESM6 and ESM6 have median and p90 spreads close to one tick, with roughly 96-97% of diagnostic rows inside the one-tick assumption.

That distinction matters. Execution realism does not say “MES is always bad” or “MES is always fine.” It shows that cost realism depends heavily on the contract universe.

MBP Microstructure Report

I also built an MBP microstructure report. It measures update rates, quote/trade activity, spread distribution, top-size collapse, imbalance, and microprice drift.

clean seconds: 1,690,818
event dates: 22
symbols: 10
average event updates per second: 127.695576
average quote updates per second: 121.759004
average trades per second: 5.936572
all-symbol median spread: 3.0 ticks
all-symbol p90 spread: 4.0 ticks
top-size collapse share: 0.006857
dominant active MESM6 and ESM6 median/p90 spread: 1.0 tick

This report was not meant to create model features immediately. It was closer to a lab notebook for future microstructure features. The all-symbol aggregate is much rougher because stale and inactive contracts remain in the universe. Active symbols are much cleaner.

Preparing Pathwise Stress For OHLCV Candidates

This execution layer later feeds into pathwise execution stress. Even if an OHLCV candidate is validation-positive, it must survive MBP-calibrated p90 spread and adverse-movement penalties.

The important part was not overusing MBP. Because current MBP coverage is not full historical train/validation fill coverage, I did not attach timestamp-matched fills to every old OHLCV trade. Instead, I built scenarios such as active-symbol p90 stress and all-MES stress, then used them as candidate promotion gates.

That choice is conservative and honest. Use the data as much as it can support, but do not pretend it proves more than it does.

Fixed Costs Are Only The Beginning

After this phase, I trusted the fixed OHLCV cost assumption much less. A 0.998 point round-turn cost is useful for research baselines, but not sufficient for promotion. A candidate must survive not only configured costs, but also spread tails, latency, adverse movement, and contract-activity differences.

Later gates became stricter because of this. A validation-positive row can still be rejected if it fails pathwise active 1000ms stress. All-symbol or all-MES stress kills even more candidates. This is annoying, but it is the right kind of annoying for a trading system.

OHLCV backtests are a good start. They do not prove execution. After adding MBP execution realism, I could see the distance between a good-looking number and a number that might actually be tradable.

[Quant Trading System 3/5] Why no_trade Baselines Made Positive Validation More Suspicious

ystc1247@gmail.com — Sat, 25 Jul 2026 02:20:00 GMT

The Purpose Of The First Baselines

Once the training matrix exists, the next temptation is a model. In this quant trading system, I ran deterministic baselines first. The reason was simple: if human-readable rules all fail after costs, putting ML on top of the same surface can become more dangerous, not less.

Baselines can search for profit, but their more important role is to provide a stop signal. If common rules fail, feature IC sends strange warnings, and candidate selection keeps choosing no_trade, the conclusion should not be “train a more complex model.” It should be closer to “do not train yet.”

The first baseline set was deliberately conservative.

test split excluded
chronological train/validation only
no overlapping positions
max 5 trades per day
cost-adjusted net return
no_trade always available as a candidate

The no-overlap and max-trades-per-day rules matter. A research signal may look active every minute, but real trading decisions are sequential. If a position is already open, the next trigger cannot be traded. Trading unlimited signals in a day is also unrealistic. So the baselines included some execution-like constraints from the beginning.

Walk-Forward Baseline

The first walk-forward baseline evaluated simple deterministic scores sequentially inside each day. The hold period was five minutes, each strategy could trade at most five times per day, and the test split was excluded.

The result was cold:

selected trade rows: 46,879
validation folds: 10
test trades: 0
maximum trades in one day: 5
top validation baseline: no_trade

I liked this result because the harness did not force an active rule to be selected. Many backtesting scripts present the least bad active rule as if it were the best strategy. In a trading system, doing nothing may be the best choice. The baseline harness has to allow that choice naturally.

Candidate Policy: The First Feature-Only Failure

The next step was a feature-only candidate policy. It used training-matrix features to create mean-reversion, momentum, and trend-pullback rules, then checked whether any policy was positive on both train and validation.

The rules were simple, but the boundaries were strict. Policies must not reference label columns, must not load the test split, and must respect no-overlap plus the daily trade cap.

POLICY_FEATURES = (
    "mes_return_5m",
    "mes_bb_20m_zscore",
    "mes_rsi_14m",
    "mes_vwap_session_diff_pct",
    "es_return_5m",
    "mes_es_return_5m_spread",
)

def validate_policy_features() -> None:
    label_overlap = sorted(set(POLICY_FEATURES) & set(LABEL_COLUMNS))
    if label_overlap:
        raise RuntimeError(f"Candidate policies reference label columns: {label_overlap}")

The first candidate policy also selected no_trade.

Metric	Value
Active policy rows scanned	95,856
Test trades	0
Max trades per day	5
Overlap count	0
Best active validation policy	`trend_pullback_score_2p5`
Best active validation net points	`-360.97`

A result this poor is at least easy to interpret. The feature-only heuristic did not beat costs. If I trained a model at this point, it would probably find some noisy feature combination. But that would be closer to the start of overfitting than the start of a strategy.

The Warning From Feature IC

I then built a feature IC report. In the spirit of Alphalens or Qlib, it looks at rank correlation, quantile spreads, turnover, and decay between features and forward returns. The test split stayed excluded.

The numbers were interesting:

input rows: 281,145 train/validation rows
features evaluated: 47
daily IC rows: 49,491
strongest validation mean absolute IC: mes_sma_5m, 0.214682

But this should not be read as an alpha claim. The strongest features were mostly raw price, SMA, and VWAP-level features. That can be a normalization warning. Futures rolls, price levels, and session behavior can make rank correlations look useful without implying a tradable signal.

So the feature IC conclusion was not “I found a good feature.” It was closer to “feature normalization and benchmark design need more suspicion before modeling.” The better a feature looks, the more I need to explain why.

Strategy Zoo: Rebuilding External Ideas Locally

After that, I built a strategy zoo. The goal was to reimplement common GitHub strategy families on top of the local data contract: SMA/EMA crossovers, MACD, RSI reversals, Bollinger mean reversion and breakout, Donchian breakout, and opening-range breakout/failure.

The important thing was not that these ideas came from outside. It was that every outside idea had to pass through the same local gate. GitHub stars, README backtests, and attractive charts are not evidence in this system. Everything must be rebuilt on local MES/ES data, costs, no-overlap rules, and train/validation splits.

The strategy zoo remained conservative:

Metric	Value
Input rows	281,145
Strategies evaluated	46
Strategy trades scanned	141,528
Validation strategy trades	24,409
Test trades	0
Selected strategy	`no_trade`
Best active validation strategy	`vol_expansion_breakout_short`
Best active validation net points	`-21.166`

The fact that the best active validation strategy was still negative is a useful stop signal. One could say, “Maybe XGBoost can still find it.” But when deterministic families all fail to beat costs, any model-found pattern needs much stronger evidence.

The Lookahead Audit Passed, But The Strategies Failed

An interesting detail is that the data leakage audit looked clean at this point. The lookahead audit recomputed selected feature families from raw OHLCV prefixes and compared them with stored feature values.

audited features: 26
sampled rows: 216
feature observations: 5,616
mismatches: 0
max absolute difference: 4.96548580031e-10
forbidden future/label/rank/net feature-name tokens: 0

This matters. When baselines fail, there are two possible explanations. Either the code/data is broken, or the signal is weak. The audit is not final proof, but it does show that the features were not accidentally looking into the future to produce attractive results. And the results were not attractive anyway.

That combination leads to an honest conclusion: the data contract was getting cleaner, but a tradable edge was not visible yet.

no_trade Is Not A Lazy Answer

The results at this point all pointed in the same direction.

walk-forward baseline: no_trade
candidate policy: no_trade
strategy zoo: no_trade
test split used: 0 trades
model training: not started

no_trade is not a lazy answer. It is closer to evidence that the baseline system is working correctly. The system should not be forced to pick something. If active rules cannot beat costs and validation, doing nothing should win.

What I gained from this stage was not a strategy. I gained a standard. Any future candidate must at least beat this baseline set. And even if it looks good on validation, it still has to survive train, cost stress, concentration, and execution realism.

The model still had to wait. When baselines return a negative conclusion, ignoring that conclusion and escaping into a more complex model is not research. It is wishful thinking.

That clear stop signal did not end the research. It changed the job of the next experiments. Sensitivity analysis became less a search for something to promote and more a pressure test of whether the first positive validation rows could still be rejected under the rules already in place.

When The First Good-Looking Numbers Appeared

This was the first point where attractive validation numbers appeared. A 30-minute horizon row produced 175.124 validation net points, and exit-label sensitivity reached 256.048 validation net points. A session-extreme short row from the parameter sweep also showed a good-looking profit factor.

This is the dangerous moment. When a research loop that had been finding nothing finally prints a positive result, the human wants to explain it. Maybe the signal fits 30 minutes better than 5 minutes. Maybe the entry is fine and only the exit is wrong. Maybe a session filter can preserve the edge.

But the point of this phase was not to find positive validation rows. It was to test whether the system could keep rejecting them through train, concentration, cost-stress, no-overlap, and test-locking rules.

Horizon Sensitivity: 30 Minutes Looked Better, But Train Collapsed

The first horizon sensitivity pass kept the current roll rule and recomputed labels at 5m, 10m, 15m, and 30m. It also checked that the recomputed five-minute labels matched the strict matrix labels exactly.

horizon label rows: 1,082,461
strategy trade rows scanned: 160,845
test labels: 0
test trades: 0
5-minute label recomputation mismatches: 0

The most visible result was a 30-minute ES/MES spread row.

Metric	Value
Best active validation row	`30m mes_vs_es_spread_gt_0p0001`
Validation net points	`175.124`
Same row train net points	`-2,894.096`

On validation alone, this is interesting. But with train failing that badly, it is not a candidate. A train loss of -2,894.096 points is not a small weakness. It is much closer to a signal that happened to survive in validation.

The horizon sensitivity conclusion was therefore simple: the 30-minute horizon may be a research clue, but it is not ready for training or test evaluation.

Roll-Rule Sensitivity: Contract Selection Did Not Rescue The Structure

The next pass was roll-rule sensitivity. It compared the previous-volume leader, front contract, and same-day-volume leader. Same-day-volume leader was marked as a lookahead diagnostic only, because it uses full-day volume.

The result was even more conservative:

roll-rule input rows: 1,014,039
strategy trade rows scanned: 121,315
test rows used: 0
front-contract differs from previous-volume leader: 53 trade dates
same-day-volume differs from previous-volume leader: 17 trade dates
best non-lookahead active validation net: -140.334

Changing the roll rule did not make the five-minute baseline family work. Contract selection matters, but it is not a magic knob that turns a bad signal into a good one.

This also blocked a tempting mistake. Same-day volume can look useful, but if the full day’s volume is used for that same day’s entry decision, it is lookahead. A tradeable rule must choose contracts using only previous information.

Parameter Sweep: The Good-Looking Row Broke On Train

The parameter sweep reused strategy-zoo signals and varied trading windows plus score floors. It made the rule stricter by allowing at most one trade per sweep variant per day, and the test split remained excluded.

This produced the first plausible validation-positive row.

selected sweep: no_trade
sweep variants: 192
sweep trades scanned: 114,967
validation sweep trades: 20,117
best active validation sweep:
  session_extreme_reversal_short__skip_open_1030_1500__max1__score_ge_50p0
best active validation net points: 67.576
best active validation profit factor: 2.408655
same sweep train net points: -151.662

The validation result is tempting. Net points are positive and profit factor looks strong. But train is -151.662 points. This combination requires caution. A rule that fails on train and works only on validation may reflect regime change, but it may also be a selection artifact.

I did not promote it at this stage. I sent it into a deeper diagnostic pass. The important thing was that the validation winner did not immediately become a candidate.

Exit-Label Sensitivity: Exit Changes Almost Looked Like A Rescue

The most dangerous result came from exit-label sensitivity. For the existing OHLCV robustness leads, it tested fixed time exits, stop/target first-touch exits, and train-derived regime filters.

The code deliberately forced train-derived thresholds:

REGIME_FILTERS = (
    RegimeFilterSpec("no_filter", "All source trades."),
    RegimeFilterSpec("session_early_or_middle", "Session progress before late-third cutoff."),
    RegimeFilterSpec(
        "train_high_mes_volume_20m_zscore",
        "MES volume z-score is at or above the source-lead train 67th percentile.",
        "mes_volume_20m_zscore",
        0.67,
    ),
)

The result looked strong:

Metric	Value
Source exit rows	22,925
Variant trade rows	98,903
Active variants	147
Best validation variant	`horizon_30m...time_exit_30m...train_high_mes_volume_20m_zscore`
Best validation net points	`256.048`
Best validation profit factor	`2.652178`
Best validation top-5 day share	`0.750012`
Same variant train net points	`-1,242.548`

If I looked only at validation PnL, it would feel like a discovery. But a top-five day concentration of 0.750012 means too much of the result came from a few dates. And with train at -1,242.548, this is diagnostic evidence, not a candidate.

Another variant was subtler. After a direction-aware rebuild, a session-extreme short 15-minute exit looked positive on both train and validation under configured costs.

configured train net: 27.230
configured validation net: 34.558
dominant-active p99 stress train net: -1.520
all-clean p99 stress train net: -145.270

The important number is not 27.230. It is -1.520. The average train edge was so small that a p99 spread-stress adjustment pushed it negative. If a result cannot survive one tick of extra cost, it is not live-ready, and it is not even model-training-ready.

Validation Is A Question, Not An Answer

This phase changed how I read validation. Validation is not an answer. It is a question generator. Why did this work only here? Why did train fail? Did a few dates create the whole result? Does cost stress erase it? Is entry the problem or exit? Was this filter chosen after seeing validation?

A good research system should ask those questions before promoting a validation-positive row. Most rows will not survive them.

The final selection in this phase remained no_trade.

horizon sensitivity: no_trade
roll-rule sensitivity: no_trade
parameter sweep: no_trade
exit-label sensitivity: no_trade
test split touched: no
model training started: no

Why This Failure Was Useful

These results may look like failures, but they made the next research direction clearer. The problem was no longer just “the baselines are bad.” I could now describe how they were bad.

The 30-minute horizon works in validation but collapses on train. Changing the roll rule does not rescue the structure. Session-extreme short looks good on validation but is negative on train. Exit changes briefly improve results, but concentration and cost stress reject them.

After decomposing the failures this far, not training a model becomes easier. ML is not a magic rescue tool for weak deterministic signals. If I train without understanding why a weak signal is weak, I risk wrapping a validation artifact in a more convincing function.

So the conclusion stayed the same. Do not trade yet. Do not open the test split yet. Do not train yet. Build better questions.

[Quant Trading System 2/5] From an MES/ES Silver Dataset to a Strict Training Matrix

ystc1247@gmail.com — Sat, 25 Jul 2026 02:10:00 GMT

Raw Futures Data Is Not A Feature Matrix

After converting Databento GLBX data into bronze Parquet, the data was still not ready for baselines or models. Futures data requires more decisions than an equity daily-bar dataset. Even when the symbols are called ES and MES, the files contain many expiring contracts, and a parent-symbol request does not automatically create the continuous series I want.

So the first real dataset task in the quant trading system was to build a silver OHLCV dataset. The goal was simple: treat MES as the tradable instrument, join same-minute ES as the reference instrument, and choose which MES/ES contracts to use on each trade date with an explicit rule.

The important part was not handcrafting the most attractive continuous contract. The rule had to be past-only, auditable, and frozen before looking at validation or test results. The first rule I chose was the previous available regular-session volume leader.

Trading instrument: MES
Reference instrument: ES
Source schema: Databento ohlcv-1m
Contract selection: previous available regular-session volume leader
Session filter: 10:00 to 15:30 US/Eastern
Label: net forward 5-minute MES close-to-close return
Splits: chronological by trade date, 70% train, 15% validation, 15% test

This rule is not perfect. But it satisfies the conditions for a good first rule. It does not use future volume, it can be reproduced from definition metadata and OHLCV data, and it leaves an artifact showing exactly which contract was selected on each date.

Making The Dataset Contract Explicit In DuckDB

The silver builder stacks several DuckDB views. It first keeps outright ES/MES futures from the definition data, then selects the latest definition row for each instrument_id. After that, it joins definition metadata into OHLCV rows and computes trade dates and times in America/New_York.

con.execute(
    """
    create or replace temp view rth_bars as
    select
      *,
      trade_date in (select trade_date from degraded_dates) as is_degraded_date
    from ohlcv_enriched
    where time_et >= TIME '10:00:00'
      and time_et <= TIME '15:30:00'
      and high >= open
      and high >= close
      and high >= low
      and low <= open
      and low <= close
      and low <= high
      and volume >= 0
    """
)

This code is not flashy, but it is one of the most important parts of the research pipeline. Before model features, there must be a dataset contract. The code has to say which time window is used, which rows are removed, and which quality checks are applied. Otherwise, when results improve later, I cannot separate a better signal from a changed data selection.

Why I Started With A Narrow Regular Session

The first trading window is 10:00 to 15:30 US/Eastern. It avoids the first 30 minutes after the U.S. equity open and the last 30 minutes before the close.

That choice is conservative. The open and close often have stronger volatility and more fragile execution assumptions. Of course, some strategies may find edge precisely there. But for the first dataset, stable measurement mattered more than signal hunting. I would rather validate the pipeline inside a narrower window than make the first result depend on the most unstable parts of the session.

I also built a separate session calendar. The silver report distinguishes full regular sessions, exchange-holiday partial sessions, early closes, and vendor anomalies. The later strict training matrix keeps only complete regular sessions. That makes holiday and early-close handling an explicit filter instead of a hidden loss of data.

First Silver Dataset Results

The first silver OHLCV dataset writes three main artifacts:

ohlcv_1m_volume_leader_v1.parquet
contract_selection_v1.parquet
dataset_manifest_v1.json

The summary looked like this:

Metric	Value
Rows	418,644
Trade dates	1,284
UTC range	2021-05-18 to 2026-05-15
Degraded rows	0
Missing ES reference rows	2
Incomplete session rows	7,873
Incomplete session dates	43
Labeled rows	412,214
Average net forward 5m return	-0.00019931

Contract coverage was another important check:

Root	Selected trade dates	Selected contracts
ES	1,284	21
MES	1,284	21

This result did not mean “the model can run now” as much as “I can now explain what the model would see.” I could identify which ES/MES contracts were selected, where the roll dates were, and why rows without forward labels were still retained.

Labels Are Experimental Conditions, Not Truth

The silver label is net forward five-minute MES close-to-close return. It records cost-adjusted net return alongside gross return, and the cost assumption is written into the report.

{
  "mes_contract_multiplier_usd_per_point": 5.0,
  "mes_fee_per_side_usd": 0.62,
  "mes_tick_size_points": 0.25,
  "slippage_ticks_round_turn": 2.0,
  "spread_ticks_round_turn": 1.0,
  "version": "ibkr_mes_retail_v1"
}

This label is not truth. It is not an actual fill, broker state, or queue-position simulation. But at the research stage, it is a clear experimental condition. It lets deterministic rules and future models be compared under the same data and cost assumption.

The silver dataset use notes therefore carry these principles:

Features use only current and earlier completed one-minute bars.
Labels use future five-minute MES closes and must not be used as features.
Holiday and early-close sessions are retained with explicit flags.
MBP fields are intentionally absent until the anomaly report is reviewed.

The last line mattered. MBP data already existed, but I did not force it into the silver OHLCV dataset. MBP needed anomaly review and a cleaning policy, and its coverage was not yet suitable for model features. Having data does not mean every field should become a feature.

The Model Boundary Left By Silver Data

Building the silver dataset made one thing very clear: a large part of quant research happens before modeling. Which contract should be selected? Which session should be used? Which dates should be excluded? Which labels should be generated? Where should costs be subtracted? How long should the test split stay locked? These decisions define the meaning of model performance before a model exists.

There was still no good strategy, but I could now explain what MES and ES meant inside the dataset: a past-volume-selected contract pair from parent-symbol data, one-minute bars inside the 10:00-15:30 ET window, degraded dates excluded, and incomplete sessions retained as explicit flags.

That kind of boring definition does not guarantee a good result. It provides the tools to distrust one that looks too good. But an explainable silver row is not automatically a row a model should see. The next boundary was not running a model; it was locking the columns and rows the model must never be allowed to consume.

The Work Before Model Training

Once the silver OHLCV dataset existed, it was tempting to train a model immediately. MES and ES were aligned, forward-return labels existed, and train/validation/test splits were already assigned. It looked ready for XGBoost or LightGBM. But I deliberately stopped there.

Before training a model, I needed to freeze the feature-matrix contract. Which columns are features? Which columns are labels? Which columns are metadata that must never enter the model? Where is split information stored, and how is the test split kept locked? Without those boundaries, a better model score could mean real signal, or it could mean that labels or future metadata accidentally leaked into the feature set.

So the next step in the quant trading system was the strict training matrix. The name means that only model-ready rows survive, but the more important meaning is that columns not meant for the model are physically separated.

Physically Separating Features, Labels, Metadata, And Splits

The training matrix builder writes one wide matrix, but it also writes four separated artifacts.

training_matrix_v1.parquet
features_v1.parquet
labels_v1.parquet
metadata_v1.parquet
splits_v1.parquet
training_matrix_manifest_v1.json

row_id is the stable key used to join those files back together. This is a little inconvenient, but it reduces mistakes. If model-training code reads only features_v1.parquet, it is much harder to accidentally include labels or metadata as inputs.

The boundary is explicit in code:

FEATURE_COLUMNS = (
    "minute_of_day_et",
    "session_progress",
    "mes_return_1m",
    "mes_return_5m",
    "mes_return_15m",
    "mes_return_30m",
    "mes_rsi_14m",
    "mes_atr_14m_points",
    "mes_vwap_session_diff_points",
    "es_return_5m",
    "es_rsi_14m",
    "mes_es_return_5m_spread",
)

LABEL_COLUMNS = (
    "gross_forward_5m_points",
    "net_forward_5m_points",
    "daily_net_forward_5m_rank",
    "is_top_5_daily_opportunity",
)

The real feature list is longer. It includes Bollinger-style z-scores, RSI, ATR, stochastic oscillators, VWAP distance, realized volatility, rolling volume normalization, SMA distance, price-action shape, and ES-reference divergence. But the main point is not the number of features. It is that the feature/label boundary is closed in code.

The Strict Filter

Not every silver row enters the matrix. Rows are removed if they lack a forward label, lack an ES reference, fall on degraded dates, come from incomplete sessions, or contain null feature/label values.

create or replace temp view strict_rows as
select
  row_number() over (order by ts_event_utc) as row_id,
  ...
from read_parquet(...)
where has_forward_5m_label
  and has_es_reference
  and not is_degraded_date
  and is_complete_regular_session
  and all feature columns are non-null
  and all label columns are non-null

This filter is conservative. It reduces coverage and removes holidays and early closes. But in the first training matrix, clarity mattered more than coverage. I can study incomplete sessions separately later. I did not want the first model or baseline to silently mix those edge cases into the main experiment.

The strict matrix ended up with:

Metric	Value
Rows	331,341
Trade dates	1,241
MES contracts	21
ES contracts	21
Train rows	232,017
Validation rows	49,128
Test rows	50,196

The important thing is not just that test rows exist. It is that they are not used yet. The test split is present in the matrix, but baseline and diagnostic tooling exclude it by default. I wanted a structure where the test set exists but remains closed.

Opportunity Ranking As The Target Shape

I did not want the first model target to be a simple directional classifier. The more important question was whether a minute was one of the few moments worth trading that day.

That is why the labels include daily ranking and top-k opportunity flags, not only forward return.

gross_forward_5m_points
gross_forward_5m_return
net_forward_5m_points
net_forward_5m_return
clears_min_net_return
daily_net_forward_5m_rank
is_top_5_daily_opportunity

This target design shaped the later candidate policy and walk-forward baseline. Even if a model scores every minute, the trading decision should happen only a few times per day. That makes “is this row one of today’s best opportunities?” more important than “is this row slightly positive?”

This also matches real execution better. A dataset may contain many rows with slightly positive five-minute returns, but after costs and slippage most are not worth trading. A good strategy should not find many positive rows. It should reject almost everything and keep only a few sufficiently good rows.

Feature Engineering Versus Feature Audit

It is true that this stage added many features. But the goal was not to build an impressive alpha library. It was closer to creating a minimum feature surface that made baselines and audits possible.

For example, the matrix includes MES short-term returns, RSI, ATR, Bollinger z-score, and ES-reference divergence. MES is the traded instrument, while ES is the larger reference market. To test whether the five-minute return spread or RSI spread matters, the first requirement is a stable same-minute join.

But more features also mean more leakage risk. So the important work at this stage was not only generating features, but making past-only behavior auditable. A later lookahead audit recomputed selected feature families from raw OHLCV prefixes and compared them against stored full-run values. The sampled mismatch count was 0.

The training matrix became the foundation for that audit.

Why I Still Did Not Train A Model

At this point, I could have trained a model. The matrix existed, features existed, labels existed, and splits existed. But “can” is not the same as “should.” Deterministic baselines were not good yet, cost and execution realism had not been fully examined, and no validation-looking signal had passed train and robustness gates.

So the first conclusion of the strict training matrix was not model training. It was deferring model training.

I think that deferral matters. In ML projects, once a training pipeline exists, running the model feels natural. In trading, the model gives convincing answers too quickly. If features are numerous and labels are noisy, some combination will look good on validation. Before trusting that combination, the data contract and baseline gates need to exist.

The strict training matrix was therefore both “data for the model” and “data that prevents premature modeling.” It made model training possible, while making the decision not to train yet more explicit.

The Standard This Phase Left Behind

After this phase, every new feature or candidate in the quant trading system had a standard to satisfy. Features must be listed in FEATURE_COLUMNS. Labels must not mix into features. Metadata is for explanation and audit, not model input. Splits are chronological by trade date. The test split exists but must not be used for selection or tuning.

This adds code. It is more annoying than writing one wide matrix and selecting columns casually. But that annoyance makes the research faster later, because when a result looks strange, the search space for possible mistakes is smaller.

Good trading ML does not begin with model architecture. At least in this project, it began by deciding which rows and columns the model must never be allowed to see.

[Quant Trading System 1/5] From no_trade Discipline to Reproducible Databento Ingestion

ystc1247@gmail.com — Sat, 25 Jul 2026 02:00:00 GMT

A Research System, Not A Trading Bot

When I started this quant trading system, the first idea I wanted to make explicit was simpler than it looked. Building a trading system does not mean the system should keep finding trades. If anything, a serious research system should be very good at deciding not to trade most of the time.

Algorithmic trading is a domain where it is easy to fool oneself. A small positive equity curve, one parameter that looks good on validation, or a strategy name seen on Reddit or GitHub can make it feel as if an edge has been found. In practice, most signals collapse under cost, slippage, regime shift, leakage, or day concentration. So the starting question for this project was not “which model should I train?” It was closer to “which conditions must fail before I am allowed to not train a model?”

The long-term goal can still be live trading. But live-ready does not simply mean that a broker API can submit orders. The version I wanted was slower and more uncomfortable. Preserve raw market data. Separate bronze and silver artifacts. Freeze train, validation, and test splits. Keep the test split locked until the end. Make deterministic baselines fail or survive first. Only after that should ML become a legitimate next step.

That is why the most important default in this system is not buy or sell. It is no_trade.

Treating no_trade As A Real Result

When writing backtests, it is easy to make only active strategies look like outcomes. A rule triggers, the code calculates performance, the best-looking rule is selected, and then the weaknesses are explained afterward. I wanted the opposite order. Doing nothing should always remain inside the candidate set, and if an active candidate cannot beat doing nothing, it should not be selected.

This idea kept repeating later in the candidate policy, strategy zoo, parameter sweep, and research gate dashboard. The names changed, but the structure remained similar. Generate candidates, split train and validation, subtract costs, then check whether the candidate is better than no-trade. If it does not pass, it may still be an interesting observation, but it is not a strategy.

The cost assumption was separated into its own object for the same reason. A small number can change the whole conclusion when the instrument is MES and the horizon is short.

@dataclass(frozen=True)
class CostAssumptions:
    version: str = "ibkr_mes_retail_v1"
    mes_fee_per_side_usd: float = 0.62
    mes_tick_size_points: float = 0.25
    mes_contract_multiplier_usd_per_point: float = 5.0
    spread_ticks_round_turn: float = 1.0
    slippage_ticks_round_turn: float = 2.0

    @property
    def round_turn_cost_points(self) -> float:
        return self.execution_cost_points + self.fee_cost_points

Under the current research assumption, round-turn cost is 0.998 MES points, or about $4.99 per contract. That is not a small number. If the average edge at a five-minute horizon is only a few ticks, a slightly more realistic execution assumption can erase the strategy. So this system makes me look at cost arithmetic before I look at attractive performance charts.

A Structure For Delaying ML

From the outside, this can look like an ML project. The repository now has a training matrix and many feature families. But the actual conclusion so far is closer to “do not train a model yet.” I do not consider that a failure. A good pipeline should block bad training before it becomes expensive and convincing.

The first question I wanted the system to answer was:

At this minute, is MES one of the best long opportunities available today?

That question is stricter than a simple directional classifier. It does not ask the model to predict every minute. It asks the system to find only the moments worth trading inside a day. That is why trade count was capped. At most five trades per day, and zero trades is allowed. This small condition changes the character of the whole system. A model that can reject trades matters more than a model that always finds activity.

The initial strategy decisions in the README carried the same shape:

Tradable instrument: MES
Reference data: ES plus MES context
Main horizon: 5-minute opportunities first
Trade count: 0 to 5 trades per day
Direction: Long-only first, both long and short later
Pace: Slow and steady: baselines before complex ML

The important part is “slow and steady.” In trading research, the moment something looks quickly discovered is often the dangerous moment. When validation PnL turns positive inside a notebook, the human already wants to believe the conclusion. This quant trading system is deliberately shaped to delay that belief.

The First Success Is Not Trading

The results so far have kept that principle intact. The first deterministic candidate policy selected no_trade. The strategy zoo selected no_trade. A parameter sweep produced a validation-positive row, but it failed on train. Exit-label sensitivity produced a better-looking validation row, but it failed train and cost stress. The market advantage scorecard and multi-horizon diagnostics eventually returned to the same conclusion.

It can feel strange to write this in a blog post. Usually, the more attractive story is “the model improved,” “the signal worked,” or “the backtest found profit.” But in real research, the most valuable result is often the opposite: discovering that an idea is not yet strong enough to trade before touching the test split, before training ML, and before starting paper trading.

The current state of the quant trading system can be summarized as:

{
  "selected_candidate_id": "no_trade",
  "candidate_ready_for_test_or_ml": false
}

I like this result. Not because it made money, but because the system can say it is not convinced. The first job of a research pipeline is not to find a good strategy. It is to prevent a bad strategy from looking good.

The Attitude A Quant Trading System Needs

As I worked on this project, I kept thinking about where responsibility belongs. Trading code requires more caution than a normal toy project. The numbers connect to money, and a small leakage bug or wrong cost assumption can flip the conclusion completely. So I see this repository less as “code that makes profit” and more as “code that makes judgment conservative.”

A good research system should calm the researcher down. If validation is positive but train is broken, stop. If train and validation look good but cost stress breaks, stop. If cost survives but day concentration is too high, stop. Until all of those gates pass, the test split should stay closed.

There is no successful model-training story yet. The more important record is why I decided not to train, why the test split stayed locked, and how no_trade became a legitimate result of research discipline.

Declaring that default was not enough. The first implementation work had to freeze data provenance and integrity before any strategy code could become persuasive.

Data Fails Before Strategies Do

The most tempting thing to do when starting trading research is to write a strategy immediately. SMA crossover, mean reversion, order-book imbalance, anything with a name feels more interesting. But the first serious work in this quant trading system was not signal design. It was moving, preserving, validating, and documenting the data well enough that later results could be trusted.

The order matters. Once market data is mixed incorrectly, everything downstream becomes contaminated. If I forget where the raw files came from, which symbology was requested, whether a batch job was split by symbol, or how conversion was performed, then even a good-looking backtest becomes hard to interpret. I cannot tell whether a strategy failed or the data was wrong.

So the quant trading system separated the Git repository from the market-data root from the start.

/path/to/quant-trading-system
/path/to/quant-trading-market-data

The repository holds code, documentation, and small manifests. Large DBN, DBN.zst, and Parquet artifacts stay outside Git. This was not only a storage decision. It was a boundary between lightweight research history and reproducible local data.

Preserving Databento Batches By Job

The first Databento GLBX downloads were parent-symbol requests. Because they used stype_in=parent, stype_out=instrument_id, and split_symbols=false, splitting the downloaded files into ES/MES directories too early would have hidden the original request shape. The raw layout therefore preserved the Databento job id:

raw/databento/GLBX.MDP3/jobs/<job_id>/<schema>/

That decision appears directly in the code. JobInfo reads dataset, schema, symbols, symbology, time range, encoding, compression, and split settings from the download metadata, then calculates the canonical raw location.

@dataclass(frozen=True)
class JobInfo:
    source_dir: Path
    job_id: str
    dataset: str
    schema: str
    symbols: tuple[str, ...]
    stype_in: str
    stype_out: str
    start_ns: int
    end_ns: int
    encoding: str
    compression: str
    split_symbols: bool

    @property
    def job_schema_dir(self) -> Path:
        return Path("raw") / "databento" / self.dataset / "jobs" / self.job_id / self.schema

I like this structure because it reduces later ambiguity. Raw data should not look as if it has already been cleaned or interpreted. It should continue to show which batch job it came from, which schema it used, and how the vendor request was shaped.

Even Copying Needs Integrity Checks

Copying data looks trivial, but in this project it is the first integrity gate. databento_pipeline.py does not simply copy files. It fails if file sizes differ. If a manifest hash exists, it verifies SHA-256. Existing destination files are accepted only when they match the expected size and hash.

def copy_file(src: Path, dst: Path, expected_hash: str | None, verify_hashes: bool) -> str:
    dst.parent.mkdir(parents=True, exist_ok=True)
    if dst.exists():
        if dst.stat().st_size != src.stat().st_size:
            raise RuntimeError(f"Existing destination has different size: {dst}")
        if verify_hashes and expected_hash and sha256_file(dst) != expected_hash:
            raise RuntimeError(f"Existing destination hash mismatch: {dst}")
        return "skipped-existing"

    tmp = dst.with_suffix(dst.suffix + ".tmp")
    shutil.copy2(src, tmp)
    if verify_hashes and expected_hash and sha256_file(tmp) != expected_hash:
        tmp.unlink(missing_ok=True)
        raise RuntimeError(f"Copied hash mismatch: {src} -> {dst}")
    tmp.replace(dst)
    return "copied"

This may feel excessive at first. But market data is not something I want to treat as casually replaceable. It costs money, the request conditions matter, and vendor condition metadata can change. So the pipeline records organize, convert, and evaluate steps separately.

The first workflow was:

python common/data/databento_pipeline.py organize ... --verify-hashes
python common/data/databento_pipeline.py convert
python common/data/databento_pipeline.py evaluate

organize preserves the raw job folders. convert turns DBN/DBN.zst files into bronze Parquet. evaluate records row counts, anomalies, degraded dates, and storage footprint. Separating those steps makes later “the data looks strange” conversations much more precise.

The First Inventory

The first evaluation result gave me more reasons to be careful than to be excited.

Schema	Raw Files	Parquet Files	Rows
`mbp-1`	26	26	387,252,384
`ohlcv-1m`	1	1	5,484,713
`definition`	1,565	1,565	83,657

The total market-data root was about 18G: roughly 7.4G raw and 11G processed bronze Parquet. This is not a tiny toy dataset, but it is still small enough to iterate locally.

The OHLCV data passed the basic sanity checks:

invalid OHLC rows: 0
negative-volume rows: 0

The MBP data immediately showed why a cleaning policy would be needed:

crossed top-of-book rows: 859
negative-price rows: 78

Those numbers can look small compared with 387M rows. But when building execution realism or microstructure features, crossed books and negative prices can distort tail metrics. For that reason, MBP did not become a feature source immediately. It later became a separate cleaning and execution-calibration track.

Keeping Degraded Dates Visible

The Databento condition metadata also produced a degraded-date list:

This list became important later in the silver dataset and training matrix. Degraded dates can be dropped or retained with flags, but the choice must be explicit. In this system, the silver dataset excludes degraded rows and records that decision in the report.

This kind of record is what determines research quality. Model performance can come later. But if provenance collapses, even good performance cannot be explained. On the other hand, if the data boundary is boringly clear from the beginning, later strategy failures become easier to diagnose.

The Result Of This Phase

The Databento ingestion phase did not produce alpha. There was no signal, no model, and no trading decision. What it produced was more foundational: raw-to-bronze paths, manifests, conversion reports, evaluation reports, degraded-date lists, and the first MBP anomaly surface.

That became the first real base of the quant trading system. Before writing a strategy, I needed to know the shape of the data. In futures data, parent symbols, instrument ids, contract rolls, and schema choices all change the meaning of a result. “I have ES/MES data” is not enough. I need to know which schema, which symbology, which period, and which quality state.

At the end of this phase, I still did not trust any trade candidate. That was a good start. A trustworthy strategy can only be built on top of trustworthy data boundaries.

[Borg-Orchestrator 4/4] Tuning, Comparing, and Bounding a Live Kubernetes Orchestrator

ystc1247@gmail.com — Sat, 25 Jul 2026 01:30:00 GMT

Once the live loop existed, the problem shifted from model performance to control behavior. A controller can be accurate in one narrow sense and still behave badly if the reward function teaches it the wrong priority.

That is why Optuna and Ray/RLlib entered the project. I did not want tuning to remain invisible. If reward weights changed, the dashboard needed to show the trial. If RL was too expensive to keep in the fast loop, that decision also needed to be visible. The experiment had to preserve the reasoning trail, not only the final chosen parameters.

This phase was about making optimization accountable. The question was not simply which configuration wins. It was whether the search process itself could be inspected well enough to trust the behavior it produced.

The reward problem

The orchestrator had three competing instincts: keep tasks alive, reduce waste, and protect the queue. I represented those as Agent A, Agent B, and Agent C rewards. The combined score used weights alpha, beta, and gamma. At first that felt too simple, but it was useful because I could see how changing the weights changed controller personality.

@dataclass(slots=True)
class Score:
    raw_rewards: dict[str, float]
    alpha: float = 1.0
    beta: float = 1.0
    gamma: float = 1.0

    @property
    def total(self) -> float:
        return (
            self.alpha * self.raw_rewards.get("AgentA", 0.0)
            + self.beta * self.raw_rewards.get("AgentB", 0.0)
            + self.gamma * self.raw_rewards.get("AgentC", 0.0)
        )

I did not treat this as a perfect reward design. It was a practical controller surface. If Agent A dominated everything, the system became safety-heavy. If Agent B was too attractive, the controller could chase efficiency while queue health was poor. If Agent C was too strong, admission behavior could become overly defensive.

Optuna as a visible search process

Optuna gave me a way to tune reward weights and policy parameters without pretending I had discovered perfect constants by hand. The important part was exporting trial history and reflecting it in the dashboard. I wanted to see completed trials, best values, and selected parameters as part of the runtime, not as a separate notebook I would forget to update.

study = optuna.create_study(
    direction="maximize",
    storage=storage,
    study_name=study_name,
    load_if_exists=True,
)

state.optuna_history(
    study_name,
    history,
    best_value=study.best_value if study.best_trial else None,
    best_params=dict(study.best_params) if study.best_trial else {},
    status="running",
)

The trial values were less important than the workflow. A controller experiment should make its tuning state visible. If I publish a dashboard screenshot where Optuna is disabled, the reader should know it was a fast run. If I publish one where Optuna completed trials, the dashboard should show the best parameters and history.

Ray/RLlib was useful but expensive to keep in the loop

Ray/RLlib gave me a more formal multi-agent policy path. The environment exposed AgentA, AgentB, and AgentC as separate agents with shared observation vectors and separate action spaces. That matched the architecture well, but local development made me careful. PPO bootstrap can slow down iteration, so I often kept it disabled while debugging live Kubernetes behavior.

self.possible_agents = ["AgentA", "AgentB", "AgentC"]

self.observation_spaces = {
    "AgentA": Box(low=0.0, high=1.0, shape=(6,), dtype=float),
    "AgentB": Box(low=0.0, high=1.0, shape=(6,), dtype=float),
    "AgentC": Box(low=0.0, high=1.0, shape=(6,), dtype=float),
}

self.action_spaces = {
    "AgentA": Discrete(POLICY_SPACES["AgentA"].action_count),
    "AgentB": Discrete(POLICY_SPACES["AgentB"].action_count),
    "AgentC": Discrete(POLICY_SPACES["AgentC"].action_count),
}

Why the dashboard mattered here

Tuning without dashboard state felt like changing knobs in a dark room. The learning/reward view let me see whether reward history, current action, agent proposals, Optuna state, and Ray status agreed with the run mode. When something looked off, I could tell whether I had a tuning problem, a disabled-feature problem, or a live data problem.

This phase also made me less interested in claiming one magic policy. The project was more interesting as a visible control experiment. Sometimes I wanted the deterministic agents and referee because they were readable. Sometimes I wanted Optuna search. Sometimes I wanted RLlib policy bootstrapping. The dashboard needed to show which one was active.

From reward tuning to comparative evidence

The system could now expose reward trajectories, optimization state, and policy state alongside live controller decisions. Reward tuning forced every objective to compete with another objective and made the tradeoffs visible instead of hiding them inside a single score.

That visibility made the controller easier to inspect, but the experimental side was still being evaluated in isolation. To turn optimization into evidence, I needed the same external pressure applied to a recognizable HPA plus Karpenter-like baseline, with a clear boundary around what the local comparison could claim.

How I launched the comparison

I used two Kind clusters: borg-experimental and borg-baseline. The experimental side received the orchestrator loop and bounded Agent A/B/C remediation. The baseline side received HPA and a local warm-node activation controller that approximated Karpenter-like behavior inside Kind. I kept the interpretation boundary visible because local Kind warm-node activation is not AWS Karpenter.

OPEN_BROWSER=0 ./orchestrator_stack/scripts/start_local_dual_cluster_stack.sh

PYTHONPATH=orchestrator_stack .venv/bin/python -m orchestrator.dashboard_server   --port 8765   --event-dir orchestrator_stack/runtime/visualization-experimental

PYTHONPATH=orchestrator_stack .venv/bin/python -m orchestrator.comparison_dashboard_server   --port 8876   --experimental-kubeconfig ~/Documents/borg_orchestrator_clusters/kubeconfig-experimental   --baseline-kubeconfig ~/Documents/borg_orchestrator_clusters/kubeconfig-baseline   --experimental-event-dir orchestrator_stack/runtime/visualization-experimental

The comparison became much more meaningful after I applied the same style of pressure to both clusters. The captured dashboard came from a severe admission-cap style stimulus: 130 replicas, explicit CPU and memory requests, and an intentionally constrained scheduling situation. I wanted the dashboard to show backlog, not just happy-path autoscaling.

EXPERIMENTAL_KUBECONFIG=$HOME/Documents/borg_orchestrator_clusters/kubeconfig-experimental BASELINE_KUBECONFIG=$HOME/Documents/borg_orchestrator_clusters/kubeconfig-baseline PHASE_INDEX=3 EXERCISE_RANDOMIZE=1 EXERCISE_SEED=27 ./orchestrator_stack/scripts/apply_comparison_stimulus.sh

Interpreting the top dashboard

The top comparison view is dense, but that is why I like it. It shows the experimental latest decision, the baseline HPA state, local Karpenter active/warm nodes, and the direct pressure comparison. In the captured run the baseline HPA is maxed at 1/1 replicas with CPU far above the target, while the baseline still has a large Pending backlog. The experimental side is not magically free of pressure, but it is visibly handling the shared stimulus differently.

The most important number in the screenshot is not a single reward score. It is the backlog gap: experimental pending 3 vs baseline pending 130. The dashboard labels this as experimental better by 127, which is much more readable than showing a raw delta and forcing me to remember which direction is good.

Why I added semantic comparison labels

At first the comparison dashboard was basically a metric pile. That was not enough. Lower pending pods is better. Lower restarts is better. Lower dynamic power is usually better under the same controlled stimulus. Higher ready workers can be better when schedulable capacity matters. A raw negative delta does not tell the reader any of that.

So I made the dashboard speak in terms like experimental better, baseline ahead, matched behavior, and objective evidence. It still exposes the numbers, but the UI tells me the direction of the claim. That matters because the dashboard is part of the argument.

specs = [
    ("Ready workers", ("ready_workers",), "Higher means more schedulable worker capacity is available."),
    ("Pending pods", ("pending_pods",), "Higher pending count means queue/backlog pressure."),
    ("Restarts", ("pod_summary", "restarts"), "Container restarts indicate instability or churn."),
    (
        "Controlled dynamic power W",
        ("controlled_resource_totals", "estimated_power_watts"),
        "Controller-relevant utilization-derived dynamic power, excluding node idle/control-plane noise.",
    ),
]

The Agent Goal Matrix was my answer to a problem I kept running into: the experimental controller and the baseline are not optimizing the same internal objective. HPA is reactive scaling logic. Karpenter is capacity provisioning. Agent A/B/C is an explicit multi-objective controller. The dashboard needed to compare observable outcomes while still saying which objective each agent was responsible for.

Controlled resource totals

One repair I made here was narrowing the energy/resource comparison to controlled namespaces. If I included every bit of cluster noise, the dashboard could accidentally compare observability overhead or control-plane background work. That would make the energy story weak. The comparison dashboard therefore tracks controlled CPU, memory, requests, and utilization-derived dynamic watts for the shared comparison and exercise namespaces.

Full comparison dashboard capture. I kept the full page because the top cards, objective evidence, timelines, agent matrix, controller narrative, and interpretation boundary are meant to be read together.

What the comparison proved—and the gap it exposed

This local comparison did not prove that my controller beats production EKS HPA plus real AWS Karpenter. That would be a much bigger claim and would require real cloud runs. What it did prove for me was narrower and still useful: I could run two local clusters under shared pressure, collect live metrics from both, apply bounded experimental remediation on one side, and show the behavioral difference in a dashboard.

That was the point where the project finally matched the frustration that started it. Pressure, controller decisions, and baseline behavior were visible at the same time instead of being reconstructed from HPA and Pending pods after the fact.

Side-by-side behavior, semantic labels, and controlled resource totals replaced vague confidence with evidence. They also exposed the final trust gap: a recommendation in the event stream was not proof that Kubernetes had been changed. Closing the loop required a deliberately narrow executor and a dashboard that distinguished proposal, attempted mutation, and observed outcome.

Bounded action execution

The executor records the command, return code, stdout, and stderr for every operation. I wanted this because silent mutation is dangerous. If the controller claims it scaled or capped something, I need the action trail in the decision payload.

def _record_operation(
    kubeconfig: str | Path,
    operations: list[dict[str, Any]],
    description: str,
    args: list[str],
) -> None:
    completed = _run_kubectl(kubeconfig, args)
    operations.append({
        "description": description,
        "command": "kubectl " + " ".join(args),
        "returncode": completed.returncode,
        "stdout": completed.stdout.strip(),
        "stderr": completed.stderr.strip(),
    })

Scaling and resource changes are similarly bounded. The executor discovers deployments only by the orchestrator exerciser label, then applies a small set of allowed changes. It can scale exercise deployments, cap a comparison load generator, or restart controlled work. It is not a general-purpose cluster automation tool, and I prefer it that way.

def execute_live_kubernetes_action(
    action: AgentAction,
    kubeconfig: str | Path,
    *,
    namespace: str = DEFAULT_EXERCISE_NAMESPACE,
    workload_namespace: str = DEFAULT_WORKLOAD_NAMESPACE,
) -> dict[str, Any]:
    names, discovery_error = _deployment_names(kubeconfig, namespace)
    operations: list[dict[str, Any]] = []
    result: dict[str, Any] = {
        "status": "observed",
        "namespace": namespace,
        "agent": action.agent_name,
        "kind": action.kind.value,
        "target": action.target,
        "payload": dict(action.payload),
        "matched_deployments": names,
        "workload_namespace": workload_namespace,
        "operations": operations,
    }
    if discovery_error:
        result["status"] = "error"
        result["error"] = discovery_error
        return result
    if not names or action.kind == ActionKind.NOOP:
        result["status"] = "no_targets" if not names else "noop"
        return result

Attaching execution to the dashboard state

The live loop attaches the execution result to the decision payload. This changed the meaning of the dashboard. A decision could now show not just AgentA:replicate, but also whether the bounded Kubernetes action was attempted and what kubectl returned.

if exercise_cluster:
    decision_payload["kubernetes_execution"] = execute_live_kubernetes_action(
        action,
        kubeconfig_path,
        namespace=exercise_namespace,
    )

state.decision(decision_payload)

This made the dashboard less clean, but more honest. Failed operations, no target matches, and no-op states are part of the experiment. I would rather show that mess than publish a controller story that hides whether anything happened.

Energy limits

Energy was the easiest metric to overclaim, so I kept the boundary explicit. The project uses a utilization-derived dynamic power estimate, not a physical wattmeter. It is useful for comparing controlled workload pressure under the same local model. It is not a claim about exact machine power consumption.

@dataclass(frozen=True, slots=True)
class PowerCalibration:
    idle_watts: float = 80.0
    cpu_full_scale_watts: float = 120.0
    mem_full_scale_watts: float = 60.0
    source: str = "default_utilization_model"

def estimate_node_power_watts(
    cpu_util: float,
    mem_util: float,
    calibration: PowerCalibration | None = None,
) -> float:
    calibrated = calibration or DEFAULT_POWER_CALIBRATION
    return (
        calibrated.idle_watts
        + (calibrated.cpu_full_scale_watts * _bounded_ratio(cpu_util))
        + (calibrated.mem_full_scale_watts * _bounded_ratio(mem_util))
    )

The comparison dashboard also separates controlled dynamic power from whole-cluster background noise. That was important because otherwise I could accidentally make the experimental side look worse or better because of unrelated observability or control-plane activity.

What Finally Remained

The final local system has Borg-derived features, XGBoost risk and demand models, a six-layer orchestrator, Agent A/B/C proposals, a deterministic referee, optional Optuna and Ray/RLlib adaptation, live Kubernetes exercise loops, a dual-cluster comparison setup, and dashboards that show the moving parts. It is not a production autoscaler. It is a personal cloud-systems research rig that lets me ask more precise questions than I could ask by staring at HPA events alone.

The most useful thing I got from the project was not a single metric. It was a workflow: create pressure, collect live state, let the controller propose, show the conflict, compare against a baseline, and keep the interpretation boundary visible. That workflow is why the dashboards became the main part of the project.

If I continue this later, the next version should run longer repeated experiments, preserve more dashboard snapshots per run, and eventually move the comparison from Kind to EKS with real Karpenter. For now, the local version is complete enough for me to explain the whole path from Kubernetes frustration to a live orchestration experiment.

This final phase made the project complete enough to explain without overclaiming. It showed where prediction ended, where decision began, where Kubernetes was actually touched, and where measurement had to stay humble. That boundary is the difference between a polished demo and an engineering artifact I can stand behind.

[Borg-Orchestrator 3/4] Building the Six-Layer Orchestrator and Validating It on Live Kubernetes

ystc1247@gmail.com — Sat, 25 Jul 2026 01:20:00 GMT

The six-layer orchestrator was the point where the project stopped being a model pipeline and became a control-plane experiment. That change was important because scores alone do not operate systems. Decisions do.

A risk score is useful only if something knows how to consume it. A demand estimate matters only if it changes how capacity, admission, or safety tradeoffs are evaluated. Once I accepted that, the project needed boundaries: where raw observations enter, where models speak, where agents propose, where conflict is resolved, and where the UI explains the decision.

This phase was my attempt to make that control flow explicit enough to criticize. If the system made a bad decision, I wanted to know which layer produced the mistake instead of blaming a vague black box.

Reading this dashboard

This screenshot is close to the mental picture I wanted from the beginning. At the top, the dashboard shows the active controller state instead of making me dig through logs. In this run the max risk is 0.950 and the selected decision is AgentA:replicate against borg-experimental-worker2. That means the safety agent won the referee decision because the risk path crossed the high threshold.

The live orchestration flow is also important. It shows that the system is not one model call. The path is: Kubernetes cluster snapshot, workload exerciser stimulus, XGBoost risk/demand inference, Agent A/B/C proposals, referee selection, reward scoreboard, and event emission. I wanted this displayed because otherwise it is too easy to talk about the orchestrator as if it were a single black box.

Agent responsibilities

The agent split was intentionally blunt. Agent A is safety. Agent B is efficiency. Agent C is admission and queue pressure. I did not want one giant policy object that quietly mixed all objectives. Separate agents made conflict visible, and visible conflict made the dashboard more honest.

@dataclass(slots=True)
class AgentARiskMitigator:
    priority: int = 1

    def act(self, obs: Observation) -> AgentAction:
        if not obs.p_fail_scores:
            return AgentAction("AgentA", ActionKind.NOOP, score=0.0, priority=self.priority)
        node_id, score = max(obs.p_fail_scores.items(), key=lambda kv: kv[1])
        if score >= 0.83:
            return AgentAction("AgentA", ActionKind.REPLICATE, target=node_id, score=float(score), priority=self.priority)
        if score >= 0.7:
            return AgentAction("AgentA", ActionKind.MIGRATE, target=node_id, score=float(score), priority=self.priority)
        if score >= 0.5:
            return AgentAction("AgentA", ActionKind.THROTTLE, target=node_id, score=float(score), priority=self.priority)
        return AgentAction("AgentA", ActionKind.NOOP, score=float(score), priority=self.priority)

Agent B and Agent C were equally direct. Agent B looked for low demand and proposed power-state, DVFS, or memory balloon actions. Agent C looked at queue length, SLA pressure, and overloaded nodes, then proposed admission or resource caps. I liked this split because it matched the operational tension I had seen around Kubernetes: safety, efficiency, and admission often pull in different directions.

The referee was the uncomfortable part

Once the agents existed, I needed to decide what happened when they disagreed. I did not want silent winner-takes-all behavior. The referee had to produce a selected action and a rationale, plus an overridden map so the dashboard could show what lost and why.

SAFETY_ACTIONS = {ActionKind.MIGRATE, ActionKind.REPLICATE, ActionKind.THROTTLE}
EFFICIENCY_ACTIONS = {ActionKind.POWER_STATE, ActionKind.DVFS, ActionKind.MEMORY_BALLOON}
PROTECTIVE_ADMISSION_DECISIONS = {"queue", "reject", "deprioritize"}

if agent_a_safety is not None:
    return RefereeDecision(
        action=agent_a_safety,
        rationale=f"agent-a {safety_label} preempts lower-priority actions",
        overridden=overridden,
    )

if restrictive_admission is not None:
    return RefereeDecision(
        action=restrictive_admission,
        rationale="agent-c admission protection preempts efficiency actions",
        overridden=overridden,
    )

The referee policy is conservative on purpose. Safety can preempt efficiency. Admission protection can preempt non-safety actions. Efficiency gets a chance when the system is not screaming. This made the controller less flashy, but much easier to reason about.

Why the dashboard became central

At this point I realized the dashboard was not a final reporting layer. It was part of the debugging process. If Agent A was selected, I needed to see the risk score and target. If Agent B lost, I needed to see the demand estimate. If Agent C proposed admission control, I needed to see queue length. If the active stage was complete but the cluster was still unhealthy, I needed the dashboard to show that too.

That is why I kept adding state fields: current decision, proposal list, reward summary, stage progress, artifact list, Optuna history, Ray status, cluster snapshot, event stream. It made the UI dense, but the project needed density. A polished dashboard that hides the control path would not have served the experiment.

An inspectable architecture still needed reality

The basic six-layer stack could train or load XGBoost brains, run agents, resolve conflicts, score rewards, and emit dashboard state. It finally had the shape I wanted: a cloud-systems experiment where the model had to live inside a visible controller.

The stack made responsibility visible by separating prediction, proposal, arbitration, execution, and explanation into inspectable parts. That turned the project from a model demo into an engineering system, but it was still a controlled design. The next test was whether the same path could survive messy timing and incomplete signals from a live Kubernetes cluster.

The live loop

The live loop is where the control-plane pieces came together. It collects a Kubernetes snapshot, creates an Observation, asks each agent for a proposal, resolves the proposals through the referee, emits the decision, steps the backend, updates reward state, and then writes the dashboard state files.

proposals = [agent.act(obs) for agent in agents]
action = resolve(proposals)
reason = _decision_reason(snapshot, action.agent_name, action.kind.value)
action_label = _action_label(action)

decision_payload = {
    "agent": action.agent_name,
    "kind": action.kind.value,
    "target": action.target,
    "payload": dict(action.payload),
    "action_label": action_label,
    "score": float(action.score),
    "proposal_count": len(proposals),
    "proposals": [
        {"agent": p.agent_name, "kind": p.kind.value, "target": p.target, "score": float(p.score)}
        for p in proposals
    ],
}

state.decision(decision_payload)

The detail I cared about here was the proposal list. A dashboard that only shows the final action hides the conflict. When the final action is AgentA:replicate, I still want to know what AgentB and AgentC wanted. Otherwise I cannot tell whether the system was calmly aligned or whether safety just overrode everything.

Running the live Kubernetes path

The local run command became long because I wanted the loop to be explicit: which config, which event directory, which kubeconfig, how often to sample, whether to tune or skip tuning, and whether to apply exercise stimuli.

PYTHONPATH=orchestrator_stack .venv/bin/python orchestrator_stack/run.py live-kubernetes-run   --config orchestrator_stack/config/orchestrator.example.json   --event-dir orchestrator_stack/runtime/visualization-experimental   --kubeconfig "$HOME/Documents/borg_orchestrator_clusters/kubeconfig-experimental"   --interval-seconds 3   --max-iterations 5   --namespace-prefixes borg-orchestrator-exercise,borg-comparison-workload,default,test-   --trace-out orchestrator_stack/runtime/visualization-experimental/live_kubernetes_trace.json   --trials 3   --prometheus-base-url http://127.0.0.1:19090   --no-policy --no-tune   --exercise-cluster   --exercise-namespace borg-orchestrator-exercise   --exercise-interval-iterations 1   --exercise-randomize --exercise-seed 31

I often ran in fast mode with tuning disabled because I wanted to debug live behavior without waiting for policy training. That is why some dashboard captures show Ray and Optuna as disabled. That was not a bug in the dashboard. It was the run mode I selected to get lively Kubernetes state quickly.

What the full dashboard showed me

The full dashboard helped me catch mismatches between the story I wanted to tell and the state the system was actually producing. If the active stage was complete but the event log had no cluster samples, something was wrong. If reward changed but decisions did not, I needed to inspect the backend. If the exerciser was active but queue pressure stayed flat, the Kubernetes stimulus was probably not doing what I thought.

In the captured run, the event sequence is visibly not a single static recommendation. It moves through AgentA replicate, AgentB memory balloon proposals, AgentC admission/deprioritize behavior, and then another AgentA replicate as risk and SLA state change. That was the kind of liveness I wanted: not animation for its own sake, but traceable state changes.

The awkward live-data problems

Live Kubernetes data introduced boring but real problems. Metrics Server can lag. Prometheus port-forwards can fail. A Kind cluster can behave differently from EKS. Pending pods can be caused by deliberate unschedulable node selectors, resource pressure, or controller choices. I had to make the dashboard expose enough detail to interpret those cases instead of flattening everything into a single score.

That is why the dashboard kept both raw-ish cluster state and interpreted decision state. The raw state tells me what Kubernetes is showing. The decision state tells me what the orchestrator thinks it should do. The interesting debugging happens when those two disagree.

What live validation established

The live experimental dashboard could keep up with the local Kubernetes loop. It was not yet a fair baseline comparison, but it was the first time the project felt like a control system instead of a batch experiment.

The live path made the project harder to explain but easier to trust. It introduced messy timing, missing signals, and awkward state transitions, which is exactly why it mattered. Real systems rarely validate themselves under clean laboratory conditions.

[Borg-Orchestrator 2/4] Repairing Borg Data and Interpreting Multi-Horizon XGBoost Signals

ystc1247@gmail.com — Sat, 25 Jul 2026 01:10:00 GMT

The first time the pipeline looked stable, I did not trust it. That instinct was useful. Stable output is not the same thing as correct output, especially when the data is large enough to hide small mistakes.

The dangerous failures were not the loud crashes. Loud crashes stop the run and force attention. The failures that worried me were quieter: nullable fields interpreted too casually, terminal events attached to the wrong temporal direction, schema changes that still produced parquet files, and labels that looked plausible while poisoning the training set.

This phase was about treating data repair as part of the system, not as cleanup after the real work. If the project was going to make decisions from model output, label integrity had to become an explicit engineering concern.

Schema drift as a real error

The orchestrator side eventually had to ingest grouped traces, synthetic traces, and live Kubernetes snapshots. That made schema validation more important than I expected. I wrote validators that treated drift as a named problem instead of letting Python fail somewhere later with a vague KeyError or TypeError.

def _parse_int(value: Any, *, field: str, row_index: int) -> int:
    try:
        return int(value)
    except (TypeError, ValueError) as exc:
        raise ValueError(
            f"Schema drift: row[{row_index}] field {field!r} must be integer-like, got {value!r}."
        ) from exc

def _validate_grouped_row(row: dict[str, Any], row_index: int) -> None:
    if "timestamp" not in row:
        raise ValueError(f"Schema drift: grouped row[{row_index}] missing timestamp key.")
    _parse_int(row["timestamp"], field="timestamp", row_index=row_index)

That style made the code a little less pretty, but it made failures readable. When I was running long local experiments, readable failure messages mattered more than elegance.

The label bug I kept checking for

The most important label check was temporal. A row should not become positive because of a terminal event that had already happened before the usage window ended. That sounds obvious, but when the dataset is built from separate usage and event sources, it is easy to accidentally make future knowledge leak into the feature row.

The repair was to keep the time-to-terminal calculation visible and to preserve a separate flag for terminal events that were already before the window end.

(pl.col("is_failure_terminal_event")
 & pl.col("time_to_terminal_event_us").is_not_null()
 & (pl.col("time_to_terminal_event_us") >= 0)
 & (pl.col("time_to_terminal_event_us") <= horizon_us)
).alias("failure_within_horizon")

(pl.col("final_event_type").is_not_null()
 & (pl.col("last_event_time") < pl.col("end_time"))
).alias("terminal_event_before_window_end")

I did not want to delete that second flag just because it was not the training target. It was useful during debugging because it gave me a way to see when terminal state was attached to a row in a suspicious way.

Repair reports, not just repaired code

Another thing I changed in this phase was how I recorded progress. If I only fixed scripts and moved on, I would forget why a repair existed. The reports in the repository became a lightweight lab notebook: what was broken, what was repaired, what remained risky, and which generated artifacts were expected after the next run.

That habit helped later when the project split into several tracks: baseline forecaster, advanced XGBoost, orchestrator stack, Optuna/Ray tuning, local dual-cluster comparison, and dashboard work. Without progress reports, those tracks would have blurred together.

The repair loop was slow but necessary

The least fun part was re-running work after a repair. A schema fix could force regenerated parquet. A label fix could force retraining. A feature change could make older metrics no longer comparable. It felt slow, but it was better than building a dashboard on top of a rotten label definition.

I had a simple rule during this part: if I could not explain what a row meant, I did not want to train on it. That rule sounds a bit dramatic, but it kept the project grounded. Later, when I watched dashboards show risk, queue length, decisions, and reward traces, I knew those values came from contracts I had already fought with.

The repair changed the trust boundary

After the repair pass, I was more comfortable moving to advanced modeling. The target label had a clearer temporal meaning, grouped trace ingestion had stricter validation, and artifact layout was less ambiguous. I still expected problems, but the project was no longer held together by hope and print statements.

The project now felt less like a one-off classifier and more like a systems experiment. The data contract, not the model, had become the center of gravity. A successful run had to produce not only data, but evidence that the data deserved to be used.

Only then did advanced modeling become worth attempting. The next risk was no longer silent corruption; it was overtrusting a model whose validation metric looked good but whose output had not yet acquired an operational meaning.

Multi-horizon thinking

The first baseline target was basically: failure within the configured horizon. The advanced track expanded that idea. In a controller setting, the horizon matters. A risk that appears very near-term should not be interpreted the same way as a weaker, longer-horizon risk. I wanted the model artifacts and reports to keep that distinction visible.

python scripts/build_advanced_xgboost_dataset.py --clusters b,c,d,e,f,g
python scripts/train_advanced_xgboost.py --clusters b,c,d,e,f,g

The important part was not the command itself. The important part was isolating the advanced workspace so the generated features, tuned models, and reports could evolve without overwriting the baseline path.

The model code stayed intentionally plain

For the orchestrator-side XGBoost brains, I kept the training function compact. Risk was binary. Demand was regression. Both used histogram tree building and saved model files that the live loop could load later.

def train_safety_model(x: np.ndarray, y: np.ndarray, out_path: str | Path) -> Path:
    xgb = _require_xgboost()
    dtrain = xgb.DMatrix(x, label=y)
    params = {
        "max_depth": 6,
        "eta": 0.06,
        "subsample": 0.9,
        "colsample_bytree": 0.9,
        "objective": "binary:logistic",
        "eval_metric": "aucpr",
        "tree_method": "hist",
    }
    booster = xgb.train(params=params, dtrain=dtrain, num_boost_round=300)
    out = Path(out_path)
    out.parent.mkdir(parents=True, exist_ok=True)
    booster.save_model(str(out))
    return out

This was not the most exotic part of the project, but it was one of the most important. I needed a model path that was easy to reproduce and easy to inspect. The orchestration system would already be complex enough. I did not want the model-loading layer to become another mystery.

Feature importance was useful, but not enough

Feature importance reports helped me check whether the model was paying attention to plausible signals: utilization, requests, rolling deltas, scheduling and priority fields, and machine-level context. But I was careful not to treat feature importance as proof. It is a debugging lens, not a guarantee that the model will behave well inside a controller.

The more useful question became: can the model produce risk and demand values that are stable enough for an agent to reason about? A model can have an acceptable validation metric and still behave poorly inside a control loop if its output jumps too much or if the thresholds create constant action flipping.

Thresholds became part of the architecture

The agent thresholds were simple at first, but they gave the risk model a behavioral meaning. A high risk score could become replicate. A medium risk score could become migrate or throttle. Low risk became no-op. That translation from probability to action was where the ML experiment started becoming an orchestration experiment.

node_id, score = max(obs.p_fail_scores.items(), key=lambda kv: kv[1])
if score >= 0.83:
    return AgentAction("AgentA", ActionKind.REPLICATE, target=node_id, score=float(score))
if score >= 0.7:
    return AgentAction("AgentA", ActionKind.MIGRATE, target=node_id, score=float(score))
if score >= 0.5:
    return AgentAction("AgentA", ActionKind.THROTTLE, target=node_id, score=float(score))
return AgentAction("AgentA", ActionKind.NOOP, score=float(score))

I changed my mind several times about these thresholds. If they were too low, the controller looked dramatic and noisy. If they were too high, the risk model became decorative. I eventually treated them as controller parameters rather than sacred ML outputs.

The awkward part of interpreting results

The advanced model reports were useful, but they did not answer the bigger systems question by themselves. AUCPR, precision at top-k, calibration bins, and feature importance all helped me decide whether the model was sane. They did not tell me whether a cluster would be better off when an agent used those scores.

That realization pushed the project toward the six-layer orchestrator. I needed a place where model outputs became observations, observations became proposals, proposals conflicted, and the dashboard showed the conflict. Without that, I would only have a pile of model files and a weak story.

From model scores to control signals

The advanced XGBoost path produced enough model infrastructure to feed a controller. The risk model and demand model were no longer isolated experiments; they were ready to become the brains layer in a larger system.

That larger system was where the project could become what I originally wanted while staring at Kubernetes: not just prediction, but visible control behavior.

The important move was from model training toward model interpretation. A metric can be read in isolation; an engineer still has to explain what it should and should not be allowed to decide.

[Borg-Orchestrator 1/4] From Kubernetes Frustration to a Reproducible Failure Forecaster

ystc1247@gmail.com — Sat, 25 Jul 2026 01:00:00 GMT

Most infrastructure projects that are worth finishing begin as a vague discomfort before they become architecture. This one started with a very specific kind of frustration: I could watch Kubernetes react, but I could not always explain the timing before the damage became visible.

At the company where I was working, I spent a lot of time around EKS, Kubernetes traces, GitHub changes, kubectl output, HPA behavior, and Karpenter behavior. From far away the cluster could look calm, but the operational story underneath was rarely calm. Workloads pushed CPU, HPA reacted later than I wanted, Karpenter added capacity on its own cadence, pods sat Pending, and the evidence lived across dashboards, terminal output, and small fragments of event history.

The first version of this project was an attempt to turn that discomfort into a measurable system. Traces would go in, failure or demand risk would come out, agents would propose actions, a referee would select one, Kubernetes would react, and a dashboard would show the whole chain without hiding the uncomfortable parts.

The first shape of the idea

The earliest version of the project was much smaller than what it became. I wanted a failure forecaster. Given resource usage and terminal event information, could I score which tasks were likely to fail within a future window? That question was attractive because it was technical enough to be concrete, but close enough to Kubernetes operations to matter.

Borg traces would provide task and machine behavior over time.
Feature windows would summarize CPU, memory, request, priority, scheduling class, and recent deltas.
A target label would mark terminal failure within a fixed horizon.
A baseline model would expose the first obvious risk score.
Later, that risk score could become an input to an orchestrator instead of staying as a notebook metric.

The part I underestimated was the word later. A model alone is very easy to overvalue. If I only trained a classifier and printed AUCPR, the project would have stopped at a static ML experiment. The more I worked on it, the more I cared about the path from model output to control decision. That is why the project eventually became dashboard-heavy. The dashboard was not decoration. It was the only way I could keep myself honest about what the system was actually doing.

Directory layout before modeling

I kept the large data outside the repository from the beginning. The Borg data was too large and too mechanical to belong in git, and I did not want generated parquet files mixed into the source tree. The repo became the code and documentation layer. The external directories became the raw, processed, model, and report layer.

mkdir -p ~/Documents/borg_data
mkdir -p ~/Documents/borg_processed
mkdir -p ~/Documents/borg_xgboost_workspace/{raw,processed,models,reports,runtime,config}

export BORG_DATA_DIR="$HOME/Documents/borg_data"
export BORG_PROCESSED_DIR="$HOME/Documents/borg_processed"
export BORG_XGBOOST_WORKSPACE="$HOME/Documents/borg_xgboost_workspace"

That split was a small decision, but it saved me later. Once I had baseline forecaster data, advanced XGBoost features, orchestrator runtime traces, Optuna reports, and dashboard screenshots, I needed to know which artifact belonged to which track. Otherwise every failed run would have left behind files with unclear meaning.

Why Borg traces instead of only Kubernetes metrics

I could have started from live Kubernetes metrics only. That would have been more immediately familiar: pod CPU, memory, HPA desired replicas, Pending pods, node readiness. But I wanted more than a reactive demo. Borg traces gave me a dataset where machine/task behavior could be processed offline, repaired, and replayed. They also forced me to think in terms of terminal events and prediction horizons, which is exactly the thing I felt was missing when watching Kubernetes react after the fact.

The first baseline command was intentionally boring. I wanted reproducible CLI entrypoints before I wanted clever architecture.

python scripts/make_dataset.py
python scripts/make_forecaster_dataset.py
python scripts/train_forecaster_baseline.py

The first target label

The first important label was simple on paper: mark a row positive if the task has a failure terminal event within the prediction horizon. The actual implementation was where the first real care was needed. It had to avoid terminal events that had already happened before the usage window ended, and it had to keep the time arithmetic explicit.

dataset = pl.scan_parquet(dataset_file(cluster_id))

frame = (
    dataset
    .sort(["collection_id", "instance_index", "end_time"])
    .with_columns([
        (pl.col("last_event_time") - pl.col("end_time")).alias("time_to_terminal_event_us"),
        pl.col("final_event_type").is_in(failure_event_types).alias("is_failure_terminal_event"),
    ])
    .with_columns([
        (
            pl.col("is_failure_terminal_event")
            & pl.col("time_to_terminal_event_us").is_not_null()
            & (pl.col("time_to_terminal_event_us") >= 0)
            & (pl.col("time_to_terminal_event_us") <= horizon_us)
        ).alias("failure_within_horizon")
    ])
)

This is the kind of code that looks plain, but it decides whether the entire project is meaningful. If the label leaks future state incorrectly, every later model and dashboard can look impressive while being nonsense. I had to keep reminding myself that the project was only as good as the boring label plumbing.

From prediction to a visible control question

At this point I was not thinking in terms of a final dashboard yet. I was thinking in terms of an experiment that could eventually answer questions like: if risk rises before a visible failure, what should the controller do? If demand is low, can efficiency actions happen without stepping on safety? If queue health degrades, should admission control override power savings? And if all of that happens, can I see the actual reasoning and not just the final number?

The Kubernetes operating friction came first. The model was my first attempt to make that frustration measurable. The dashboard came later because I eventually stopped trusting invisible experiments.

That shift turned an operational feeling into an engineering question. The project was no longer about whether I could train a model on a public trace dataset. It was about whether prediction could become a visible, inspectable control signal.

The forecaster idea became useful only after that trust boundary was implemented in the data path. With raw and generated artifacts separated, the next job was to join usage, event, and machine evidence without letting a successful Parquet write masquerade as correctness.

Joining usage, events, and machines

The first real dataset was a joined table. Usage rows carried the time-window behavior. Events carried terminal state and task lifecycle evidence. Machines added context. I used Polars because I wanted streaming scans over parquet instead of loading everything into memory and hoping for the best.

usage = pl.scan_parquet(str(usage_path))
events = pl.scan_parquet(str(events_path))
machines = pl.scan_parquet(str(machines_path))

dataset = (
    usage
    .join(events, on=["collection_id", "instance_index"], how="left", suffix="_event")
    .join(machines, on="machine_id", how="left", suffix="_machine")
    .collect(engine="streaming")
)

dataset.write_parquet(output_path)

This join was one of those points where I had to stay suspicious. A successful parquet write did not mean the dataset was correct. I checked row counts, positive label rates, missing machine ids, and whether terminal event timing still made sense after the join.

Why I did not jump straight to advanced features

I was tempted to build the complicated feature set immediately. Rolling windows, deltas, request ratios, priority encodings, cluster-level pressure, all of that was more interesting than verifying joins. But doing that too early made every later error harder to isolate. So I kept the first track intentionally plain: produce the joined dataset, produce a forecaster frame, train a baseline, inspect the predictions.

The baseline was not meant to be the final model. It was a sanity instrument. If the baseline could not produce a plausible risk ranking, then the advanced model would only hide the failure behind more parameters.

The first baseline training loop

The baseline training script standardized the forecaster frames, split validation data, trained a model, and wrote the outputs I knew I would want later: validation predictions and top risk alerts. I cared about those files because they let me inspect actual rows instead of only reading aggregate metrics.

python scripts/train_forecaster_baseline.py   --clusters b,c,d,e,f,g   --feature-profile baseline

The useful outputs were not just the model file. The validation predictions parquet gave me a ranked surface to inspect. The top-risk alerts parquet gave me the first version of what later became the control-plane idea: risk is more useful when it can be attached to an action candidate.

validation_predictions.parquet: scored validation rows, sorted later by risk_score.
top_risk_alerts.parquet: high-risk rows kept for inspection.
cluster forecaster parquet files: per-cluster feature/label frames.
metrics text and reports: enough context to compare later runs.

The first friction-heavy trial-and-error loop

The difficult part was that every small pipeline change could invalidate something downstream. I would fix an optional column, then a cluster would produce different feature coverage. I would change the label horizon, then the positive rate would move. I would repair a schema issue, then model results would look better, but I still had to check whether the improvement was real or just caused by a different filtered dataset.

This is where I started building the habit that carried through the rest of the project: every artifact needed to be inspectable. A model score alone was not enough. A dashboard alone was not enough. A parquet file without a local explanation was not enough. I wanted the project to be explicit enough that I could keep moving without fooling myself.

The first reliable measuring tool

The completed baseline path could read the joined Borg-derived datasets, build forecaster frames with a 15-minute-style failure target, train a baseline forecaster, and write predictions I could inspect. It was still not the orchestrator. It was the first reliable measuring tool.

That measuring tool also exposed the next problem: schema drift and broken labels. At that point the pipeline stopped being a clean sequence and became a repair project.

The result was not a clever model. It was a workspace that made later experiments accountable—the kind of unglamorous foundation that quietly determines whether every later claim is defensible.

[DevBerth 2/2] I Closed the Window. DevBerth Kept Using 69% CPU.

ystc1247@gmail.com — Thu, 23 Jul 2026 06:10:00 GMT

I closed the window, but the work kept going

DevBerth is supposed to stay alive in the menu bar when its main window closes. What it was not supposed to do was keep a large part of one CPU core busy.

I noticed it in Activity Monitor. With the main window and menu popover closed, the Release build accumulated 623.86 CPU-seconds over 901 seconds: an average of 69.2% CPU. Nothing was visible and I was not interacting with the app.

My first guess was lsof. DevBerth does a lot of process discovery, so an expensive parser or classification pass seemed likely. Instruments told a more interesting story. The individual scans were not the whole problem; DevBerth was taking “nothing changed” and turning it into fresh UI and persistence work every two seconds.

Before changing anything, I fixed the comparison so I would not accidentally optimize one scenario and publish numbers from another.

Item	Measurement condition
Machine	MacBook Pro `Mac17,2`, 10-core Apple M5, 24 GB
OS / toolchain	macOS 26.4, Xcode 26.4
Runtime scale	Roughly 70 live host/Docker runtime rows, observed read-only
Build	Application and helper installed from the same Release build
Closed-window sample	Main window and menu popover closed for 901 seconds
Profiler	Approximately 30 seconds of the same idle scenario in Time Profiler
Safety	No unrelated process, service, or container was signaled

The baseline commit was 8cc670e, and I recorded the optimized measurements on 2026-07-22. These are before-and-after numbers from one development Mac, not a benchmark for every Mac.

Activity Monitor was great for spotting the problem. To find it, I used Apple Instruments, os_signpost intervals, and small internal counters around discovery, enrichment, Docker, diffing, persistence, and SwiftUI publication.

The baseline Time Profiler trace made the hidden cost visible. The main thread spent 14.657 sampled seconds flushing SwiftUI run-loop observers and 13.566 seconds in AttributeGraph updates. The window was gone, but a retained graph was still being told that everything had changed.

The fix was to stop treating every refresh as news

The first bug was almost embarrassingly small. ObservedListener stored lastDetectedAt, and the process fingerprint stored detectedAt. Those timestamps changed on every scan, so ordinary value equality said the whole snapshot was new.

fixed two-second poll
  → fresh timestamps
  → whole snapshot unequal
  → SwiftUI publication
  → AttributeGraph update
  → lifecycle/history persistence
  → pruning and derived recomputation

I still needed the timestamps because evidence freshness matters. The fix was to separate fresh evidence from a meaningful runtime transition. A changed protocol, address, port, process identity, project, owner, Docker relationship, or managed-service relationship can be real news. A newer timestamp by itself is not.

That one distinction took deterministic unchanged-snapshot tests from 30 transactions and 3,460 persistent-history changes per minute to zero writes.

The next problem was cadence. Two seconds feels fine while I am looking at the Runtime screen. It makes no sense after the app has been hidden and stable for several minutes. The monitor now has one actor, one cancellable loop, and at most one scan in flight. Extra refresh requests coalesce instead of starting parallel work.

Mode	Cadence	When it applies
Transition	0.75 s	For 15 seconds after startup, wake, a real runtime change, or a known mutation
Active	configured 2 s	Main window or menu popover is actually visible in the foreground
Background	at least 10 s	No monitoring surface is visible
Idle	at least 30 s	Hidden with no semantic change for 180 seconds

Getting “actually visible” right took more work than expected. SwiftUI’s onDisappear was not enough because AppKit backing windows could remain around after the surface closed. DevBerth now looks at real NSWindow visibility, occlusion, minimize and key-window state, plus whether the app is active or hidden. A menu-bar backing window only counts when the popover is truly active.

Then I reduced all the little child processes that had been multiplying each other. Process enrichment is cached only when a full native identity matches: PID, UID, start time, parent PID, executable path and file identity, argument digest, and working directory. The cache can save discovery work, but it can never authorize Stop or Restart; control still does a fresh identity and ownership check.

Resource usage moved to one batched ps with slower 1/5/30/60-second schedules as the surface moves from transition to idle. Docker uses one list and one batched inspect, a short success cache, and backoff when unavailable. Health checks have bounded concurrency. Logs drain continuously but only update the visible model in small batches when their revision changes.

In a 141-second idle observation, the monitor created nine direct children, or 3.8 per minute. The deterministic pre-metadata baseline was at least 126 per minute, so the rate fell by about 97%. Very short ps calls can slip between 200 ms samples, which is why I call that result a lower bound rather than zero work.

One last surprise came from UDP. High-numbered, interface-bound UDP endpoints appeared and disappeared often enough to keep resetting the fast transition cadence. DevBerth still observes them, but while hidden it no longer treats that narrow kind of churn as a reason to poll at 0.75 seconds. TCP, lower UDP ports, and wildcard or loopback UDP changes still speed the monitor up immediately.

Finally, SwiftUI publication became surface-aware. AppModel always keeps the newest snapshot, but while hidden it does not wake SwiftUI for timestamp-only or resource-only refreshes. In the foreground it publishes meaningful listener changes, CPU moves of at least one percentage point, or memory moves above the larger of 1 MiB and 5%. When the first monitoring surface comes back, the newest hidden snapshot is published once.

The result, and the parts I did not hide

The matched Release comparison looked like this:

Metric	Baseline Release	Optimized Release	Change
Closed-window CPU	623.86 CPU-s / 901 s = 69.2%	1.06 CPU-s / 901 s = 0.118%	99.83% less
Time Profiler CPU	24.4 CPU-s / ~30 s	0.027 CPU-s / 30.742 s	99.89% less
Main-thread sampled CPU	19.558 CPU-s	0.001 CPU-s	99.995% less
Direct monitoring children	at least 126/min	9 / 141 s = 3.8/min	about 97.0% fewer
Main-window foreground CPU	34.17 / 69 s = 49.5%	1.57 / 60 s = 2.62%	94.7% less

The final idle Time Profiler trace contained 27 samples and 0.027 CPU-seconds over 30.742 seconds. The main thread had one 0.001-second sample, with no potential hang above 250 ms. The idle SwiftUI trace also reported no hitch or hang.

The diagnostics screenshot is a live snapshot, not the source of the benchmark. It shows cadence, scan duration, cache hit rate, and enrichment totals without exposing commands, paths, environments, logs, or secrets. It only polls while the sheet is open.

The Activity Monitor image is also just a spot check. Several devberth-mcp rows can belong to different active Codex clients, so I did not add them together and compare that total with the single-app closed-window sample.

I also kept a repeatable five-minute soak test. run_performance_soak.sh launches an isolated Release app with in-memory persistence, fixed listener and resource fixtures, unavailable test Docker, and no production socket. It samples and terminates only the PID it launched.

In the final 300-second run, CPU time rose by 0.11 seconds over the 293 seconds after startup, or about 0.038%. RSS settled between 132,992 and 165,568 KiB and ended at 133,040 KiB. Threads settled from 11 to five or six. Every sample saw zero direct children, and the application log contained no error, fatal, or crash lines. The warnings-as-errors run passed all 210 tests at the time, and the broader soak gate passed twice.

Not every loose end disappeared. The old timestamp bug had already left roughly 118 MB of persistent-history backlog in the production store. The optimized sample stopped that false growth, but I did not quietly vacuum or destructively compact existing user data as part of a CPU fix.

I also did not invent measurements I could not get. Command-line Instruments did not expose Power Profiler on macOS 26.4. System Trace export never produced a usable artifact. There was no untouched-baseline kernel wakeup count, so there is no wakeup-reduction percentage here. leaks -quiet found three 80-byte CGRegion allocations, 240 bytes total, but no growing app-owned signature; that is not the same as claiming zero leaks.

Polling still exists because the macOS 14 deployment baseline has no public event API that fully reports TCP and UDP listener ownership. Hidden evidence can be up to 30 seconds old, and a brief high-numbered interface UDP endpoint can disappear between scans. Foreground navigation also still produced two potential hangs at 279.19 and 319.81 ms. Those hitches are another profiling job, not something I wanted to bury under a good idle number.

The biggest improvement was not a clever parser or a faster poll. It was teaching the app that a newer observation is not always a new event. DevBerth is built to notice runtime changes, and the best performance work was finally giving it a good way to say, “nothing happened.”

Previous: I Wanted a Better Way to See What Was Running on My Mac

[DevBerth 1/2] I Wanted a Better Way to See What Was Running on My Mac

ystc1247@gmail.com — Thu, 23 Jul 2026 06:10:00 GMT

The initiative: understand the whole local workspace

DevBerth began with a familiar local-development problem. I would start a frontend or API, discover that 5173 or 8080 was already occupied, and open a terminal to work out what was there. The routine was always similar: run lsof, copy a PID, check ps, look at Docker just in case, and try to remember whether I had started the process from a terminal, an IDE, or another tool that would bring it straight back after I killed it.

Finding a PID was easy. Understanding what the PID represented was not.

The first initiative was deliberately small: build a native macOS port viewer that connected TCP and useful UDP listeners to their processes. I wanted one place where I could answer “what is using this port?” without assembling the answer from several commands. A sortable table, a process inspector, and a Stop button seemed like enough.

As soon as I tried to make Stop reliable, the project became much larger.

A modern development workspace is rarely one process. It may contain a frontend dev server, an API, a database, a mobile bundler, a reverse proxy, a tunnel, a worker, and several containers. One process may own several listeners. A host port may really be published by Docker. A Compose service may have a container, a host process, a restart policy, and a project directory that all matter. Homebrew or launchd may recreate a process after it exits. An IDE task and a system process may share the same executable name. A PID can even be reused between inspection and execution.

Traditional port tools are good at isolated facts. Process managers are good at processes they launched themselves. I wanted DevBerth to connect those two views without pretending they had the same level of trust.

That led to the central product rule: seeing a process is not the same as having authority to control it.

The rule becomes easier to reason about when it is split into three questions:

What is running right now?
What did I deliberately configure to run?
What is DevBerth actually allowed to control right now?

The first answer comes from operating-system evidence. ObservedListener and ObservedProcess are built from bounded lsof and ps collection plus structured Docker output. They describe the machine during one refresh. The information may be incomplete, and it begins to age as soon as it is collected.

The second answer is durable user intent. A ManagedServiceConfiguration records an executable or launch mechanism, separate arguments, a working directory, expected ports, dependencies, readiness and health checks, shutdown and restart policies, bounded log settings, and opaque Keychain references. A RuntimeInstance represents one actual execution of that reviewed definition.

ObservedListener / ObservedProcess
  what the machine looks like now

ManagedServiceConfiguration
  what I reviewed and intended to run

RuntimeInstance
  one actual run of that configuration

The distinction matters because a process snapshot is not a launch recipe. It may omit shell expansion, environment variables, credentials, supervisor behavior, or the real working directory. A visible Docker process may only be an implementation detail of a Compose service. A shell command may be the middle of a longer chain.

DevBerth can discover service candidates from project files, but discovery stops at suggestion. It never executes a candidate while scanning and never silently promotes an observed process into something restartable. I have to review the definition before the app can claim it knows how to launch it again.

The third answer—current control authority—must be resolved at action time. A familiar command, matching port, inferred project, old PID, or similar container is useful context, but none of those alone authorizes a mutation.

This separation became the foundation of the entire product:

Runtime shows what the machine currently exposes.
Projects describe how reviewed services belong together.
Sessions capture the workspace state I expect.
Managed Services define what DevBerth may reliably launch.
History records what actually changed.
Docker adds container and Compose ownership when it can be proved.
Settings exposes safe preferences, permissions, diagnostics, and integrations.
MCP lets Codex use the same application-owned control plane instead of creating another one.

The app is native SwiftUI and local-only. It does not require an account, administrator privileges, or a cloud service. There is no analytics, telemetry, advertising SDK, cloud sync, crash-upload service, or observed-runtime upload endpoint. Process and resource values stay on the Mac. Secret-like environment values live in Keychain rather than SwiftData.

On first launch, the welcome guide explains these boundaries before offering four useful starting points: review the current Runtime, import or create a Project, create and validate a Managed Service, or capture a Workspace Session. It also explains that macOS visibility may be limited, that discovery is not management, and that reliable restart requires a reviewed definition.

DevBerth works without Full Disk Access, but macOS may hide some same-user process metadata. Settings can open the correct Privacy & Security page when broader visibility is useful; the app cannot and does not grant that permission to itself.

The project is currently distributed from source, and the complete repository is public at ysbc1247/portpilot-macos on GitHub. The stable build script creates the Release app, verifies the bundle, replaces /Applications/DevBerth.app atomically, and installs the matching MCP helper under Application Support. There is not yet a Developer ID-notarized download or automatic updater, so I keep that limitation explicit rather than presenting the source build as a finished consumer distribution.

That is the initiative in one sentence: make the whole local workspace understandable, then allow control only where the current evidence and reviewed intent actually support it.

A complete tour from Runtime to Settings

The product uses one stable navigation order: Runtime, Projects, Sessions, Managed Services, History, Docker, and Settings. Each destination has a main collection view, while a contextual inspector or preview explains the selected item. I chose that structure because the app contains a lot of evidence, but I did not want a separate dashboard inventing a second version of the same state.

Runtime is the live view of the machine. It discovers TCP listeners and meaningful bound UDP endpoints across IPv4 and IPv6, then connects them to the process metadata macOS makes available. The same snapshot can be shown as a customizable table or grouped by inferred project.

Search covers ports, protocols, addresses, process names, commands, and projects. Sorting covers port, process, project, runtime, and uptime. Saved views narrow the list to All Runtime, Managed, Unexpected, Unhealthy, Docker, or Externally Reachable. The table keeps the fields I check most often together: status and protocol, port, process and PID, project, ownership, restart trust, health, runtime, uptime, CPU, and resident memory.

Native selection supports keyboard traversal and multiple rows. Multi-selection summarizes listener and process counts and resource totals, but it deliberately does not offer an ambiguous “stop everything selected” button. A group of rows may contain several owners and several safety levels, so bulk destruction would hide exactly the distinction the app is meant to preserve.

Selecting one runtime opens the inspector. It explains the network listeners, process identity, observed command, working directory, parent evidence, project inference, managed-service relationship, restart trust, health, recent lifecycle events, Docker association, logs, and currently safe actions. “Why is this running?” is answered with bounded lineage and confidence-labeled ownership evidence. Missing or permission-limited data is shown as unavailable rather than guessed.

Transient CPU and resident-memory usage come from one bounded batched reader. Docker metadata is attached when an exact container or Compose relationship is available. A listener can show active managed state, unexpected state, unhealthy state, or externally reachable scope without turning those classifications into control permission.

Inspect and Stop remain close to each process because they are common tasks, but Stop is routed through the resolved owner. Root-owned and recognized system processes stay protected. An observed host process requires explicit confirmation and fresh fingerprint, listener-edge, user, protected-process, and owner-context checks.

Projects move the unit of work from a port to a workspace. A project groups reviewed services, their expected ports, and dependency edges. The project view shows current observed or DevBerth-controlled activity, health and lifecycle state, Docker relationships, startup layers, incomplete dependencies, cycles, and recent failures.

Start All launches only stopped definitions that are currently verified. It preserves the complete dependency graph, so a database can become ready before the API that needs it and independent services can start in parallel where the graph allows. Stop All walks the graph in reverse, attempts every relevant target even if one fails, and keeps visible per-service progress and final results. Start, Stop, and Inspect are also available on each service row.

Projects can be created manually or populated through Discover Services. Discovery reads supported files only inside a root I explicitly select. It recognizes JavaScript package managers, Gradle, Maven, Python, Go, Cargo, Docker Compose, Procfile, and Process Compose definitions. The scan is bounded and non-recursive, rejects unsafe paths, and never evaluates a project command.

Discovered services remain unreviewed candidates. I can edit arguments, directories, expected ports, checks, and dependencies before importing them. Projects can also export a versioned devberth-runtime.json manifest. The export deliberately excludes secret values and even Keychain reference identifiers, so it can describe runtime intent without becoming a credential bundle.

Sessions capture a development workspace without pretending to capture a shell session. A Workspace Session stores selected projects and the expected running or stopped state of their managed services. It does not persist arbitrary process objects or assume that an old PID still matters.

Opening a session compares that expected state with the current machine and shows drift. Restore always begins with a fresh preview, including when it is launched from the command palette. The preview checks that definitions still exist, Keychain references resolve, restart trust remains current, expected ports are available, dependencies are complete, directories and executables still exist, and no conflict has appeared.

The preview supports dry-run review, explicit confirmation of issues, expected-stopped handling, and optional rollback. Starts execute in dependency layers. If one layer fails, rollback can stop only the services started by that restore. It never stops an unmanaged process or a service that was already running as compensation. Restore history remains available so the result is inspectable after the workspace has changed again.

This is useful for work that spans more than one repository or mixes host and Docker services. Instead of remembering which five commands I ran yesterday, I can describe the state I want, compare it with today’s runtime, and review the difference before DevBerth changes anything.

Managed Services are the durable definitions behind reliable Start and Restart. A service can use a generic executable or command, npm, pnpm, Yarn, or Bun scripts, Gradle, Maven, an explicitly reviewed custom shell, a Docker container, or a verified Compose service. The definition records command boundaries rather than flattening everything into one shell string.

It also records the working directory, expected ports, readiness and health checks, shutdown behavior, restart policy, dependencies, log policy, and secret references. Supported checks include TCP, HTTP, reviewed command, file, Docker, and dependency conditions. DevBerth treats process-running, listener-open, service-ready, and service-healthy as separate facts; one does not automatically imply the next.

New and edited services are not automatically restartable. Validation launches the exact definition in isolation, observes expected ports and reviewed readiness checks, and performs a controlled stop. If the sequence succeeds, the exact configuration digest becomes Verified restartable.

Changing a safety-relevant field invalidates that trust. Editing the executable, arguments, working directory, expected ports, dependencies, checks, shutdown behavior, or secret references sends the service back to review. This can feel stricter than a typical process manager, but it keeps Start and Restart tied to the definition that was actually tested.

Each service row keeps its own operation state. Stopping one service does not make every other button look busy. Progress, success, refusal, and failure remain visible until a newer operation replaces them. The same row can open logs and the working directory, copy the reviewed command, inspect evidence, stop a current runtime, or start and restart when the exact trust state permits it.

Managed stdout and stderr are bounded in memory and on disk. Output passes through a streaming redactor before it reaches either place, including when a secret crosses chunk boundaries. Restart policies are attached to actual runtime instances and bounded by crash-loop limits rather than implemented as a UI toggle.

History answers “what changed while I was away?” It indexes bounded lifecycle evidence for filtering by time, severity, source, event type, service, summary, and result. The data includes launches, stops, exits, readiness transitions, health degradation and recovery, automatic-restart attempts, crash-loop limits, listener changes, Docker transitions, and session restore results.

History keeps events factual. Incident summaries are deterministic and built from ordered evidence rather than generated prose. A partial project stop stays partial. A service that is running but not listening is not rewritten as healthy. Related runtime, service, project, and session identifiers make it possible to move from the summary back to the underlying evidence.

Retention is bounded and configurable. Logs, history exports, diagnostics, and audit details are already redacted and size-limited before they leave their original service. The goal is enough history to explain behavior without turning a local runtime manager into an unbounded data collector.

Docker is optional. Runtime monitoring still works when Docker is not installed or its daemon is stopped. When it is available, DevBerth reads structured container data, published ports, state, health, restart policy, recent logs, and canonical Compose labels.

Inspection can be broad, but mutation is intentionally narrow. A standalone container is addressed by its full current identity. A Compose mutation is available only after DevBerth verifies the exact project name, project directory, configuration and environment files, configuration hash, and current container membership. One-off containers never gain service-level authority, and DevBerth never falls back to signaling a container’s host PID.

If Docker becomes unavailable, the app reports that state and backs off its checks without taking host listener monitoring down with it. That separation matters on laptops where Docker Desktop is not always running.

Settings collects the safe configuration around the product. It exposes monitoring preferences, bounded history retention, appearance, permission guidance, the welcome guide, safe diagnostics, and the Codex/MCP integration. Full Disk Access opens the correct System Settings location rather than pretending the app can grant it.

Performance Diagnostics shows bounded aggregates such as current monitoring mode and interval, scan duration, coalescing counts, cache size and hit rate, Docker duration, and recent warnings. It excludes commands, paths, process details, environments, logs, and secrets. The sheet reads its snapshot only while it is open.

The Codex & MCP settings can install or repair the stable helper, preview a global or project Codex configuration change, test the connection, and run MCP validation. The editor preserves unrelated TOML, rejects duplicate DevBerth tables and symlinks, writes atomically, verifies the result, and keeps a timestamped backup.

The main window is not the only way to use these features. The menu bar shows counts for active managed services, unexpected listeners, and unhealthy services. It offers listener search, favorite service control, recent project Start and Stop, session capture, monitoring controls, and a route back to the main window.

Pressing ⌘K opens the command palette. It searches ports, PIDs, processes, commands, projects, managed services, and sessions. It can open every destination, refresh or toggle monitoring, start or stop projects, capture a session, open a restore preview, open or copy managed-service data, and start, stop, or restart a service. Restart appears only when the exact current digest is Verified restartable.

The menu bar and palette are shortcuts, not privileged surfaces. They call the same application actions as the main window and cannot bypass ownership, validation, confirmation, or restart-trust rules. Native labels, text alongside critical color states, keyboard navigation, accessibility descriptions, light and dark appearance, and localization-ready user strings are part of the same product surface.

One control plane, including Codex

All those features would be much less useful if each one invented a different definition of safety. DevBerth therefore has one monitor, one current runtime snapshot, and application-owned domain services for ownership, lifecycle, projects, sessions, persistence, and secrets. SwiftUI depends on service protocols; views do not call Process, lsof, ps, kill, Docker, or a shell directly.

The same boundary applies to Codex.

The easiest MCP implementation would have let a helper run its own discovery and shell commands. It also would have created a second runtime authority with its own cache, ownership logic, persistence, and failure modes. Instead, devberth-mcp is a thin STDIO adapter. It validates typed MCP input, communicates with the already-running app over a current-user Unix socket, and returns structured results.

Production currently exposes 82 tools plus resources and prompts for runtime inspection, ports, projects, services, dependencies, sessions, Docker, logs, history, settings, diagnostics, destructive previews, and coordinated change sets. The helper does not run its own lsof, ps, Docker, shell, database, or Keychain queries. The GUI and Codex are looking at the same snapshot and asking the same app services to act.

Production has no TCP control port. The socket lives under DevBerth’s Application Support directory with a private parent, restrictive file mode, matching-UID peer checks, bounded frames, deadlines, and protocol negotiation. Production and development use separate sockets and handshakes.

The identity of DevBerth itself is also checked. During development, a Debug control host and the installed Release app can share a display name and bundle identifier. Production activation therefore validates the exact /Applications/DevBerth.app, executable, Release marker, production socket, and persistence mode. Development activation requires an explicit app path and Debug marker. A familiar label is not strong identity, whether the target is a user process or DevBerth itself.

For host processes, a strong ProcessFingerprint combines PID, UID, executable path and file identity when available, start time, command digest, parent PID, and observation time. DevBerth also keeps the exact listener-to-process edge and resolves the lifecycle owner.

Immediately before a signal, it resolves the target again. A graceful stop waits for the configured timeout. Force escalation performs another fresh resolution instead of reusing the earlier decision. A changed fingerprint, listener edge, owner route, user, protected status, or controller context is a refusal.

observe
  → identify the process and listener
  → resolve the lifecycle owner
  → show the exact action preview
  → obtain confirmation
  → resolve and validate again
  → signal the exact scope
  → validate once more before force escalation

The rules differ by runtime state:

Runtime state	Inspect	Stop	Start or Restart
Observed listener or process	Yes	Explicit confirmation plus fresh exact-owner revalidation	Observation grants no launch authority
Live DevBerth-managed runtime	Yes	Through its registered policy and revalidated process group	Only from the exact verified definition
Verified Docker or Compose owner	Yes	Through the verified controller context	Only while that exact context remains valid
Protected or unverifiable process	Limited evidence	Refused	Refused

MCP destructive actions add a two-step authorization contract. operation_preview captures stable targets, revisions, process fingerprints, listener edges, owner routes, expected effects, risks, and compensation. The token expires after five minutes and can be used once. operation_execute consumes it only after resolving the current target again.

Five minutes is not a promise that the host will stay unchanged. It is only the maximum lifetime of the authorization. Changed revisions, replay, stale identity, different ownership, protected targets, or missing controller context still fail. Configuration batches use an equivalent change-set preview and execution flow, with compensation for already-applied configuration steps when a later step fails.

Secrets never cross this control boundary. SwiftData stores opaque Keychain references, not values. MCP can say whether a secret name is configured and currently resolvable, but it cannot retrieve the value or reference UUID. Profile edits stage Keychain changes and roll them back if validation or persistence fails. Duplicated services receive independent references rather than sharing secret lifecycle.

The local-only privacy model is equally important. DevBerth does not upload observed runtimes. Diagnostics omit commands, paths, environments, logs, and secrets. Manifests omit secret values and Keychain identifiers. Resource data and live process objects are transient. Tests use in-memory stores, no production socket, and static or application-owned fixtures; they do not enumerate or signal unrelated user processes or containers.

The current implementation covers the complete sidebar, menu bar, command palette, onboarding, service discovery, orchestration, sessions, lifecycle and health evidence, Docker and Compose routing, privacy controls, and MCP. The latest warnings-as-errors run passed all 216 tests across unit, persistence, harmless owned-process integration, UI, MCP, and application-identity coverage.

There are still limits. DevBerth cannot reconstruct an arbitrary process’s original shell session or complete environment. Root-owned and recognized system processes remain protected because there is no privilege-escalation helper. UDP does not have a universal TCP-style listening state. Project inference walks only bounded parents of a verified working directory instead of scanning the disk. The editor currently exposes one dependency selector even though the domain model supports larger graphs. Distribution remains source-first.

Those limits are intentional parts of the explanation, not details to hide after the feature list. DevBerth is useful because it connects a lot of local runtime information, but it is trustworthy only when it also says what it cannot prove.

What began as “what is using port 5173?” became a native runtime explorer, guarded service controller, workspace restorer, lifecycle record, Docker inspector, menu-bar tool, and MCP control plane. Every feature returns to the same initiative: understand the local workspace without guessing, and make control follow the evidence rather than the other way around.

The next article covers the performance problem that appeared after all of this was working. The monitor was collecting the right evidence, but it treated every refresh as meaningful change and used 69.2% CPU with the window closed.

Next: I Closed the Window. DevBerth Kept Using 69% CPU.

Springfox Swagger Parse Error in Spring Boot: Reproducing the Failure and Migrating to springdoc

ystc1247@gmail.com — Fri, 10 Jul 2026 02:35:00 GMT

This Was Not Just A Swagger UI Problem

Springfox can fail before Swagger UI has a chance to render. The useful debugging boundary is the request-mapping metadata that Springfox scans while the application starts, not the browser page that never loads.

The runnable lab in labs/swagger-springfox-migration-lab/ exercises the same tiny /orders/{orderId} API through three cases:

Spring Boot 2.7.18 + Springfox 3.0.0 + PathPatternParser.
Spring Boot 2.7.18 + Springfox 3.0.0 + AntPathMatcher plus a handler-mapping filter.
Spring Boot 3.4.7 + springdoc-openapi-starter-webmvc-ui 2.8.17, grouped docs, and bearer-token OpenAPI metadata.

The goal is not to argue that one documentation library is always better. The goal is to make a version-boundary failure observable, then choose the smallest responsible fix for each application generation.

The Failure

The failure looked like a Swagger parse error, but the important line was lower in the stack trace:

Failed to start bean 'documentationPluginsBootstrapper';
nested exception is java.lang.NullPointerException:
Cannot invoke "org.springframework.web.servlet.mvc.condition.PatternsRequestCondition.getPatterns()"
because "this.condition" is null

at springfox.documentation.spring.web.WebMvcPatternsRequestConditionWrapper.getPatterns
at springfox.documentation.RequestHandler.sortedPaths
at springfox.documentation.spring.web.plugins.WebMvcRequestHandlerProvider.requestHandlers

That tells us Springfox is not merely failing to render UI resources. It is failing while scanning Spring MVC request mappings and sorting discovered paths before the docs endpoint can be served.

The reproduced evidence is saved in /resources/blog/post-50-swagger/springfox-default-path-pattern-failure.txt.

Why Path Matching Matters

Spring MVC has two path-matching models that matter here:

Model	Rough shape	Why it matters for Springfox
`AntPathMatcher`	String-pattern based matching	Older libraries often expect `PatternsRequestCondition` to be populated
`PathPatternParser`	Parsed path-pattern matching	Request mapping metadata can be represented without the old condition object Springfox expects

Spring Framework’s path matching documentation describes PathPatternParser as the parsed-pattern alternative to AntPathMatcher. Spring Boot’s 2.7 web documentation explains the matching-strategy property. A Spring Boot 2.6 documentation issue also records that the MVC auto-configuration default was actually path-pattern-parser in that line.

Springfox 3.0.0 was built around older Spring MVC assumptions. Its own Spring Boot application notes mainly describe dependency and annotation changes for Springfox 3.x. They do not turn Springfox into a modern Spring Boot 3 documentation stack.

That is why “Swagger broke” is too vague. The actual bug is:

Springfox’s request-handler scanner expects mapping metadata that is not available in the same shape when Spring MVC uses parsed path patterns.

Lab Design

The lab uses Maven inside a JDK 17 Docker container:

services:
  maven:
    image: maven:3.9.9-eclipse-temurin-17
    working_dir: /workspace
    volumes:
      - .:/workspace
      - maven-cache:/root/.m2

There are two modules:

springfox-boot27
springdoc-boot3

Both expose the same API:

@RestController
class OrderController {
    @GetMapping("/orders/{orderId}")
    OrderResponse getOrder(@PathVariable String orderId) {
        return new OrderResponse(orderId, "PAID", 42000);
    }
}

Run it:

cd labs/swagger-springfox-migration-lab
./scripts/run-matrix.sh

The successful rerun used ID 20260710T024421Z.

Observed Matrix

The clean run produced this result:

Scenario	Endpoint	Observed outcome
Boot 2.7.18 + Springfox 3.0.0 + `PathPatternParser`	`/v2/api-docs`	application startup failed before the endpoint could be served
Boot 2.7.18 + Springfox 3.0.0 + `AntPathMatcher` + handler-mapping filter	`/v2/api-docs`	returned a Swagger 2 document
Boot 3.4.7 + springdoc-openapi 2.8.17	`/v3/api-docs` and `/v3/api-docs/orders`	returned OpenAPI 3.1 documents with bearer auth metadata

The generated summary is saved at /resources/blog/post-50-swagger/matrix-summary.md.

The positive Springfox case returned:

Scenario: Spring Boot 2.7.18 + Springfox 3.0.0 + AntPathMatcher + handler-mapping filter
Outcome: /v2/api-docs returned a Swagger 2 document
Contains swagger 2 marker: true
Contains /orders/{orderId}: true
Contains OrderResponse schema: true

The springdoc migration case returned:

Scenario: Spring Boot 3.4.7 + springdoc-openapi-starter-webmvc-ui 2.8.17
Outcome: /v3/api-docs returned an OpenAPI document
HTTP status: 200 OK
Contains openapi marker: true
Contains /orders/{orderId}: true
Contains OrderResponse schema: true
Contains bearerAuth security scheme: true
Grouped endpoint /v3/api-docs/orders status: 200 OK
Unauthenticated GET /orders/demo-1 status: 403 FORBIDDEN

Those two lines are more important than a Swagger UI screenshot. They prove that the generated machine-readable contract exists and includes the expected operation and schema.

Boot 2.7: If You Cannot Migrate Immediately

For an existing Spring Boot 2.7 service that still uses Springfox, the compatibility path in the lab is explicit:

spring.mvc.pathmatch.matching-strategy=ant-path-matcher
lab.springfox.compatibility-filter=true

The handler-mapping filter is intentionally isolated behind a property:

@Configuration
@ConditionalOnProperty(name = "lab.springfox.compatibility-filter", havingValue = "true")
class SpringfoxCompatibilityConfig {
    @Bean
    static BeanPostProcessor springfoxHandlerMappingFilter() {
        return new BeanPostProcessor() {
            @Override
            public Object postProcessAfterInitialization(Object bean, String beanName) {
                if ("springfox.documentation.spring.web.plugins.WebMvcRequestHandlerProvider"
                    .equals(bean.getClass().getName())) {
                    filterParsedPatternMappings(bean);
                }
                return bean;
            }
        };
    }
}

The real implementation in the lab reflects into Springfox’s handlerMappings list and removes mappings whose getPatternParser() is not null. This is a compatibility patch, not a clean architecture pattern. It depends on Springfox internals and should be covered by an integration test that actually calls /v2/api-docs.

My rule after reproducing this is:

If the application is staying on Boot 2 for a while, keep the workaround narrow and tested.
If the application is moving to Boot 3, do not carry Springfox forward.

Boot 3: Migrate The Documentation Stack

Spring Boot 3 moved to Jakarta namespaces and a newer Spring Framework baseline. Treating Springfox as something to patch through that migration is the wrong default.

The springdoc path in the lab uses:

<dependency>
    <groupId>org.springdoc</groupId>
    <artifactId>springdoc-openapi-starter-webmvc-ui</artifactId>
    <version>2.8.17</version>
</dependency>

The migrated module also adds spring-boot-starter-security, a SecurityFilterChain, a grouped OpenAPI bean, and a bearer-token security scheme:

@Configuration
@SecurityScheme(
    name = "bearerAuth",
    type = SecuritySchemeType.HTTP,
    scheme = "bearer",
    bearerFormat = "JWT"
)
class OpenApiConfig {
    @Bean
    GroupedOpenApi ordersApi() {
        return GroupedOpenApi.builder()
            .group("orders")
            .pathsToMatch("/orders/**")
            .build();
    }
}

The security filter keeps generated docs reachable while protecting the API endpoint:

@Bean
SecurityFilterChain securityFilterChain(HttpSecurity http) throws Exception {
    return http
        .csrf(AbstractHttpConfigurer::disable)
        .authorizeHttpRequests(auth -> auth
            .requestMatchers("/v3/api-docs/**", "/swagger-ui/**", "/swagger-ui.html").permitAll()
            .anyRequest().authenticated()
        )
        .build();
}

The springdoc documentation says this starter exposes Swagger UI and the OpenAPI JSON endpoint for Spring Boot applications. In the lab, /v3/api-docs returned:

{
  "openapi": "3.1.0",
  "paths": {
    "/orders/{orderId}": {
      "get": {
        "summary": "Find one order"
      }
    }
  }
}

The important migration difference is not just endpoint naming:

Springfox path	springdoc path
`/v2/api-docs`	`/v3/api-docs`
Swagger 2 model by default	OpenAPI 3.x model
`Docket` configuration	starter auto-configuration plus `@Operation`, `@Tag`, `GroupedOpenApi`
Boot 2 compatibility workaround may be needed	Boot 3-compatible library line
Ad hoc security snippets	explicit docs allowlist plus OpenAPI `bearerAuth` scheme

If the frontend, QA tooling, or API gateway still expects /v2/api-docs, the migration must include that consumer contract. Do not silently change the docs endpoint and call the backend migration done.

Security Configuration Is A Separate Test

Swagger endpoints also need an explicit Spring Security decision. A Boot 2-era allowlist often looks like this:

.antMatchers("/swagger-ui/**", "/v3/api-docs", "/v2/api-docs").permitAll()

That line is version-sensitive:

Boot 2 / Spring Security 5 commonly used antMatchers.
Boot 3 / Spring Security 6 uses the newer authorizeHttpRequests style and requestMatchers.

More importantly, a docs endpoint should not automatically be public in every environment. For internal admin APIs, it may be better to expose docs only behind VPN, SSO, basic auth, or a non-production profile.

The lab now turns that into a regression test:

GET /v3/api-docs as an unauthenticated user
expected: 200 OK

GET /orders/demo-1 as an unauthenticated user
expected: 403 FORBIDDEN

That test tells you more than a copied allowlist snippet.

Debugging Checklist

When Swagger/OpenAPI generation breaks after a Spring Boot upgrade, check it in this order:

Identify the docs library and version.

Springfox, springdoc, and raw Swagger UI are not interchangeable. The failure mode depends on which library is scanning Spring MVC mappings.
Find the endpoint that should exist.

For Springfox it is usually /v2/api-docs. For springdoc it is usually /v3/api-docs. Test the JSON endpoint before debugging the UI.
Read the stack trace at the scanner layer.

If the stack contains documentationPluginsBootstrapper, WebMvcRequestHandlerProvider, or WebMvcPatternsRequestConditionWrapper, suspect Springfox and MVC path matching.
Check Spring MVC path matching.

If you are on Boot 2.6 or 2.7 and still using Springfox, compare path-pattern-parser and ant-path-matcher with an integration test.
Separate compatibility from migration.

A Boot 2 compatibility patch may be acceptable as a temporary hold. It should not become the Boot 3 migration strategy.
Verify security and deployment behavior.

The JSON docs endpoint, the Swagger UI static assets, reverse-proxy path rewriting, and auth filters can fail independently.

What This Proves In An Interview

This lab is small, but it demonstrates a better debugging habit than “I followed a Swagger setup tutorial.”

It proves that I can:

Reproduce a framework compatibility failure with a minimal app.
Turn a stack trace into a hypothesis about request-mapping internals.
Separate UI rendering from generated API contract availability.
Keep a risky workaround contained and tested.
Define a migration path that accounts for consumers of /v2/api-docs and /v3/api-docs.

The practical conclusion is direct:

For existing Boot 2.7 services, patch Springfox only as a tested bridge. For Boot 3 services, migrate to springdoc and verify /v3/api-docs as part of the upgrade.

That is the engineering decision hidden behind what first looked like a Swagger parse error.

Reducing Kubernetes Load Balancer Cost With Istio Gateway: A Local Routing Lab

ystc1247@gmail.com — Fri, 10 Jul 2026 02:16:37 GMT

The Cost Problem Is Real, But The Fix Is Not Free

A familiar EKS cost problem appears when every module gets its own ALB-backed Ingress and the number of idle load balancers grows with the number of exposed services. AWS Elastic Load Balancing charges for each hour or partial hour that an Application Load Balancer or Network Load Balancer is running, then adds capacity-unit and related charges on top. If four low-traffic services each get their own ALB, the base hourly floor alone is already four times larger than a single shared edge.

But “use Istio Gateway” is not a complete answer. It moves complexity from cloud load balancer count into gateway operations, certificate ownership, routing correctness, blast radius, and quota management.

The local lab in labs/istio-gateway-cost-lab/ does not create AWS resources or claim to reproduce an AWS bill. It proves the routing shape locally with kind and Istio:

Four public hostnames.
Four backend Kubernetes Services.
One Istio ingress gateway Service.
One Istio Gateway.
Four VirtualService objects attached to the shared gateway.

Then it calculates the base load-balancer hourly floor separately from AWS’s current pricing examples.

The Baseline: One ALB-Style Ingress Per Module

The baseline is easy to understand and easy to operate at small scale:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: api-alb
  namespace: apps
  annotations:
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/target-type: ip
    alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:region:account:certificate/api-placeholder
    alb.ingress.kubernetes.io/ssl-redirect: "443"
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS":443}]'
spec:
  ingressClassName: alb
  rules:
    - host: api.local.test
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: api
                port:
                  number: 8080

Each service owns its own edge resource. That is attractive because the blast radius is small: if the api ingress is wrong, admin does not necessarily break. DNS, certificate, WAF, and access log ownership are also easier to reason about per module.

The cost downside is the number of load balancers. In the lab’s baseline manifest, four hostnames produce four ALB-style Ingress objects:

api.local.test
admin.local.test
billing.local.test
backoffice.local.test

That is a reasonable choice for high-value production services. It becomes wasteful for low-traffic development, release, admin, or internal-facing modules that still need stable HTTPS exposure.

The Shared Gateway Model

The shared model puts the external edge behind one gateway Service. In EKS, that Service is commonly backed by one NLB. Istio’s ingress gateway then handles host-based routing through Gateway and VirtualService resources.

The lab’s Gateway is intentionally small:

apiVersion: networking.istio.io/v1
kind: Gateway
metadata:
  name: shared-edge-gateway
  namespace: istio-system
spec:
  selector:
    istio: ingressgateway
  servers:
    - port:
        number: 80
        name: http
        protocol: HTTP
      hosts:
        - api.local.test
        - admin.local.test
        - billing.local.test
        - backoffice.local.test

Each application gets a route:

apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: billing-route
  namespace: apps
spec:
  hosts:
    - billing.local.test
  gateways:
    - istio-system/shared-edge-gateway
  http:
    - route:
        - destination:
            host: billing.apps.svc.cluster.local
            port:
              number: 8080

The key difference is ownership. The cloud load balancer is no longer the only routing object. It becomes the entry point to an Envoy gateway, and the actual host routing lives in Istio configuration.

Local Lab

Run the lab:

cd labs/istio-gateway-cost-lab
./scripts/run-lab.sh

The successful rerun used ID 20260710T023921Z.

Tooling:

Tool	Version from the run
Docker	29.3.0
kind	v0.31.0
kubectl client	v1.34.1
Helm	v4.1.4

The script creates a kind cluster, installs Istio with Helm, deploys four hashicorp/http-echo services, creates a local self-signed TLS secret, applies the shared Gateway, VirtualServices, DestinationRule, PDB, and HPA manifests, scales the ingress gateway to two replicas, port-forwards ports 80 and 443, and then runs HTTP, HTTPS, load-loop, failure, and rollback checks.

The gateway Service looked like this in the local cluster:

NAME                   TYPE           CLUSTER-IP     EXTERNAL-IP   PORT(S)                                      AGE   SELECTOR
istio-ingressgateway   LoadBalancer   10.96.173.72   <pending>     15021:32402/TCP,80:30705/TCP,443:30744/TCP   20s   app=istio-ingressgateway,istio=ingressgateway

The <pending> external IP is expected in a plain kind cluster. The lab uses kubectl port-forward to exercise the gateway locally.

Istio resources:

NAMESPACE      NAME                  AGE
istio-system   shared-edge-gateway   1s

NAMESPACE   NAME               GATEWAYS                               HOSTS
apps        admin-route        ["istio-system/shared-edge-gateway"]   ["admin.local.test"]
apps        api-route          ["istio-system/shared-edge-gateway"]   ["api.local.test"]
apps        backoffice-route   ["istio-system/shared-edge-gateway"]   ["backoffice.local.test"]
apps        billing-route      ["istio-system/shared-edge-gateway"]   ["billing.local.test"]

NAMESPACE   NAME            HOST
apps        echo-services   *.apps.svc.cluster.local

Route checks:

### api.local.test
api service via shared istio gateway

### admin.local.test
admin service via shared istio gateway

### billing.local.test
billing service via shared istio gateway

### backoffice.local.test
backoffice service via shared istio gateway

The full evidence is saved under /resources/blog/post-72-istio-gateway/.

The rerun also verified HTTPS routing through the Istio gateway. The script generated a one-day self-signed certificate for the four *.local.test hostnames and stored it as a Kubernetes TLS secret named shared-local-tls. Then it port-forwarded 18443:443 and called each hostname with curl --resolve:

### api.local.test
api service via shared istio gateway

### admin.local.test
admin service via shared istio gateway

### billing.local.test
billing service via shared istio gateway

### backoffice.local.test
backoffice service via shared istio gateway

That is not a production certificate workflow, but it proves the Istio-side TLS termination path used by the manifest. In production, I would replace this with cert-manager, ACM-backed termination at the NLB, or a centrally managed secret rotation process.

The load loop was intentionally small, but it keeps the route checks from being a single lucky request:

api.local.test ok_requests=20
admin.local.test ok_requests=20
billing.local.test ok_requests=20
backoffice.local.test ok_requests=20

For failure and rollback, the lab patched the billing-route VirtualService to a missing backend host, observed a failed route, then reapplied the known-good manifest:

### before failure
billing service via shared istio gateway

### after bad route patch
http_code=503
body=

### after rollback
billing service via shared istio gateway

That is the operational check I care about. A shared gateway design must have a boring rollback path, not only a happy-path route.

The gateway scale and availability objects were also captured:

NAME                   READY   UP-TO-DATE   AVAILABLE
istio-ingressgateway   2/2     2            2

poddisruptionbudget.policy/istio-ingressgateway   MIN AVAILABLE 1
horizontalpodautoscaler.autoscaling/istio-ingressgateway   MINPODS 2   MAXPODS 4

The HPA shows unknown CPU in kind because the lab does not install metrics-server. That is expected. The manifest still documents the autoscaling contract, while the actual clean run proves that two gateway replicas can be rolled out in the local cluster.

Cost-Floor Model

The lab’s cost script compares the controlled manifests:

Model	Public hostnames	Load-balancer-shaped resources	730-hour monthly floor
Per-module ALB Ingress	4	4	$65.70
Shared Istio Gateway/NLB	4	1	$16.43
Difference	0	-3	-$49.28

The formula is deliberately limited:

load_balancer_count * 0.0225 USD/hour * 730 hours

The 0.0225 USD/hour value comes from the AWS ELB pricing examples for US-East-1, checked on 2026-07-10. AWS also charges for LCU/NLCU usage, data transfer, public IPv4 addresses, and other services that may sit in front of or behind the load balancer. NAT Gateway cost is not part of this public ingress path unless the architecture sends egress or private-subnet flows through NAT; the model lists it separately as excluded rather than pretending it is zero everywhere.

The script also prints a simplified traffic-dependent estimate. It uses processed bytes only, so it is not a bill and it does not include Envoy worker-node cost:

Profile	Processed bytes	Capacity units used for estimate	4 ALB model	1 NLB + Istio model	Difference
low	0.3 MB/s	1.08	$72.01	$21.16	-$50.85
medium	5.0 MB/s	18.00	$170.82	$95.27	-$75.56
high	25.0 MB/s	90.00	$591.30	$410.63	-$180.68

So the claim is not “Istio saves exactly $49.28.” The claim is:

Consolidating four low-traffic public edges into one shared gateway removes three running load-balancer hourly floors before traffic-dependent charges.

That is a narrower and more defensible statement. At higher traffic, the answer depends on bytes, new connections, active connections, rule evaluations, cross-zone behavior, gateway pod size, and whether ALB L7 features such as WAF integration are replacing logic that would otherwise move into Envoy.

TLS And Certificates

A shared NLB path can terminate TLS with attached ACM certificate ARNs, but that is not the only design.

There are two common options:

TLS model	Where TLS terminates	What to check
NLB TLS listener	AWS NLB terminates TLS and forwards HTTP/TCP to the gateway	ACM certificate list, listener config, AWS Load Balancer Controller annotations, certificate quota
Istio gateway TLS	NLB passes traffic to Envoy and Istio terminates TLS with Kubernetes secrets	Secret ownership, SDS behavior, gateway TLS config, namespace boundaries, cert-manager workflow

The local lab exercises the second option with a self-signed secret:

apiVersion: networking.istio.io/v1
kind: Gateway
spec:
  servers:
    - port:
        number: 443
        name: https
        protocol: HTTPS
      tls:
        mode: SIMPLE
        credentialName: shared-local-tls

The sample patch in the lab shows the NLB-annotation style:

{
  "metadata": {
    "annotations": {
      "service.beta.kubernetes.io/aws-load-balancer-type": "external",
      "service.beta.kubernetes.io/aws-load-balancer-nlb-target-type": "ip",
      "service.beta.kubernetes.io/aws-load-balancer-scheme": "internet-facing",
      "service.beta.kubernetes.io/aws-load-balancer-ssl-cert": "arn:aws:acm:region:account:certificate/api-placeholder,arn:aws:acm:region:account:certificate/admin-placeholder",
      "service.beta.kubernetes.io/aws-load-balancer-backend-protocol": "http",
      "service.beta.kubernetes.io/aws-load-balancer-ssl-ports": "443"
    }
  }
}

In a real cluster, I would not apply that blindly. I would first decide whether TLS belongs at the AWS edge or in Istio, then verify certificate count, rotation ownership, access logs, and whether the application needs client certificate or SNI-specific behavior.

Quotas Are Part Of The Design

AWS quota pages matter more as soon as one gateway represents many applications.

The NLB quota page lists limits such as certificates per NLB, listeners per NLB, NLBs per Region, and targets per Availability Zone per NLB. The ALB quota page lists its own certificate, rule, target, and load-balancer quotas.

The engineering check is simple:

number of hosts
number of certificates
number of listeners
number of backend Services
number of gateway pods
number of target registrations per AZ
expected connections and bytes

If one shared gateway approaches those limits, the cost-saving design becomes a scaling risk. At that point, split by environment, traffic class, domain ownership, or service criticality.

Operational Trade-Offs

The shared gateway pattern is useful when the services are low traffic, operationally related, and owned by a team that can operate the ingress plane well.

It is risky when unrelated systems are put behind the same edge only because the bill looks cleaner.

The main trade-offs:

Concern	Per-module ALB	Shared Istio Gateway
Load-balancer hourly floor	Higher as modules increase	Lower
Failure blast radius	Smaller	Larger
Routing flexibility	AWS ALB rules and controller behavior	Istio routing model
Certificate ownership	Per ingress or group	Centralized or gateway-managed
Debug path	AWS LB -> target group -> Service	AWS LB -> Envoy gateway -> VirtualService -> Service
Required platform skill	Kubernetes Ingress and AWS LB controller	Istio, Envoy, gateway operations, mesh observability

For production, I would add at least:

Two or more ingress gateway replicas.
PodDisruptionBudget for the gateway.
HPA or fixed capacity sized from connection and CPU data.
Readiness checks and synthetic route checks per hostname.
Access logs and Envoy metrics.
Alerting on 5xx, route misses, and gateway saturation.
A documented certificate rotation path.
A clear split rule for when a service must get its own edge.

The lab now includes the first three as manifests or execution steps: gateway-ha-overlays.yaml defines a PDB, HPA, and DestinationRule; run-lab.sh scales the gateway deployment to two replicas and records the result. It does not prove multi-AZ behavior because kind is a single local cluster, so cross-zone target registration and zonal failure handling remain cloud-environment validation items.

Layer 4 versus Layer 7 is the core architecture decision. An NLB in front of Istio keeps the cloud edge simple and can preserve source IP depending on target type and configuration, but HTTP routing, retries, mTLS policy, and per-host behavior move into Envoy. An ALB Ingress keeps AWS-native L7 routing, health checks, and WAF attachment closer to the edge, but multiplying ALBs increases fixed hourly cost and rule ownership. HTTP/2 and gRPC make this choice sharper: verify protocol negotiation, idle timeouts, and whether TLS terminates at the NLB, ALB, or Istio gateway.

When I Would Use It

I would use a shared Istio Gateway for:

Development and release environments.
Low-traffic admin tools.
Internal dashboards that still need stable DNS and HTTPS.
Services owned by the same platform or product team.
Hostnames whose certificate and DNS ownership is already centralized.

I would avoid it, or split the gateway, for:

High-traffic customer-facing paths.
Services with separate compliance boundaries.
Services with incompatible TLS requirements.
Teams that cannot safely operate Istio routing.
Any route where a shared gateway outage would be unacceptable.

Sources And Reproduction

Lab code: labs/istio-gateway-cost-lab/
Cost summary: /resources/blog/post-72-istio-gateway/cost-summary.md
Route checks: /resources/blog/post-72-istio-gateway/route-checks.txt
TLS route checks: /resources/blog/post-72-istio-gateway/tls-route-checks.txt
Load loop: /resources/blog/post-72-istio-gateway/load-loop.txt
Failure and rollback output: /resources/blog/post-72-istio-gateway/failure-rollback.txt
Gateway HA objects: /resources/blog/post-72-istio-gateway/gateway-ha-objects.txt
Gateway and VirtualService output: /resources/blog/post-72-istio-gateway/gateway-virtualservices.txt
AWS ELB pricing: Elastic Load Balancing pricing
AWS NLB quotas: Quotas for your Network Load Balancers
AWS ALB quotas: Quotas for your Application Load Balancers
Istio Gateway reference: Gateway
Istio VirtualService reference: VirtualService
Istio Helm install: Install with Helm

Hibernate Slow Query Logging in Spring Boot: YAML, .properties, and Provider Settings

ystc1247@gmail.com — Fri, 10 Jul 2026 02:03:44 GMT

The Real Bug Was Not YAML

“Hibernate slow query logging works in hibernate.properties, but not in YAML” is an imprecise diagnosis. The useful question is narrower:

Did the Hibernate slow-query threshold enter Hibernate’s provider settings, or did it only exist somewhere in Spring’s external configuration?

Those are different states. A value can appear in application.yml and still not be forwarded to the JPA provider. A value can also be overridden by another config file before Hibernate ever sees it.

The runnable lab in labs/hibernate-slow-query-lab/ compares Spring Boot 2.7.18 with Hibernate 5.6.15.Final and Spring Boot 3.3.6 with Hibernate 6.5.3.Final, using PostgreSQL 16 in Docker and a deterministic slow SQL probe:

select 1;
select 1 from pg_sleep(0.02);

The threshold is 1 ms in the positive cases. The slow query sleeps for about 20 ms, so a correctly delivered threshold should emit an org.hibernate.SQL_SLOW log.

Short Answer

For Spring Boot JPA applications, put Hibernate-native settings under spring.jpa.properties.

For Hibernate 5.6:

spring:
  jpa:
    properties:
      hibernate:
        session:
          events:
            log:
              LOG_QUERIES_SLOWER_THAN_MS: 1

Or:

spring.jpa.properties.hibernate.session.events.log.LOG_QUERIES_SLOWER_THAN_MS=1

For Hibernate 6.5, this shorter provider key also worked in the lab:

spring.jpa.properties.hibernate.log_slow_query=1

Do not put the provider key at the top level and expect Spring Boot JPA auto-configuration to pass it to Hibernate:

hibernate:
  session:
    events:
      log:
        LOG_QUERIES_SLOWER_THAN_MS: 1

In this lab, that top-level form produced no slow-query log in both Boot 2/Hibernate 5 and Boot 3/Hibernate 6.

Why This Is Easy To Misconfigure

There are three independent layers that are easy to collapse into one mental model:

Layer	What it owns	Failure mode
Spring external configuration	Loads `application.yml`, `application.properties`, environment variables, command-line args, and profile files	The value is overridden or loaded in a different order than expected
Spring Boot JPA auto-configuration	Builds the JPA `EntityManagerFactory` and provider property map	The value exists in Spring config but is not forwarded to Hibernate
Hibernate	Interprets provider-native settings and emits SQL logs	The key name is wrong for the Hibernate version or logger level is too low

Spring Boot’s externalized configuration documentation describes the property-source order and explicitly recommends using one file format for the application. It also says that when .properties and YAML config files are in the same location, .properties wins.

Spring Boot’s application properties appendix lists spring.jpa.properties.* as the namespace for additional native JPA provider properties. That is the bridge this post is about. Hibernate’s own key is not the full Spring Boot key; Spring Boot’s spring.jpa.properties. prefix is how the native Hibernate key gets placed into the provider map.

Lab Design

The lab uses Docker Compose rather than an in-memory database:

services:
  postgres:
    image: postgres:16-alpine
    ports:
      - "55433:5432"
    environment:
      POSTGRES_DB: hibernate_slow_query
      POSTGRES_USER: postgres
      POSTGRES_PASSWORD: postgres

  gradle:
    image: gradle:8.10.2-jdk17
    depends_on:
      postgres:
        condition: service_healthy

I used PostgreSQL because pg_sleep(0.02) creates an actual slow database statement. The test is not measuring controller latency, Java sleeps, or result processing time. It is testing whether Hibernate’s JDBC statement logger sees a query execution that crosses the configured threshold.

The base Spring Boot test configuration intentionally keeps Hibernate statistics disabled:

spring:
  jpa:
    properties:
      hibernate:
        format_sql: false
        generate_statistics: false

logging:
  level:
    org.hibernate.SQL: off
    org.hibernate.SQL_SLOW: info

That matters because slow-query logging should not require turning on every Hibernate statistic. The lab listens to org.hibernate.SQL_SLOW, not to the broad org.hibernate logger.

Run the lab:

cd labs/hibernate-slow-query-lab
./scripts/run-matrix.sh

The successful rerun used ID 20260710T024828Z. Its generated matrix is saved at /resources/blog/post-66-hibernate-slow-query-logging-yaml-properties/matrix-summary.md, and a short log excerpt is saved at /resources/blog/post-66-hibernate-slow-query-logging-yaml-properties/slow-query-lines.txt.

How The Test Avoids False Positives

My first version used Spring Boot’s output-capture test extension and searched the entire captured console output for SlowQuery. That was too loose. A multi-context Gradle test process can include earlier log lines, and Hibernate 5 and Hibernate 6 do not use exactly the same slow-query message text.

The final test helper attaches a short-lived Logback appender directly around the probe queries:

static boolean observesSlowQuery(EntityManager entityManager) {
    Logger slowQueryLogger = (Logger) LoggerFactory.getLogger("org.hibernate.SQL_SLOW");
    ListAppender<ILoggingEvent> appender = new ListAppender<>();
    appender.start();
    slowQueryLogger.addAppender(appender);

    try {
        entityManager.createNativeQuery("select 1").getSingleResult();
        entityManager.createNativeQuery("select 1 from pg_sleep(0.02)").getSingleResult();
    } finally {
        slowQueryLogger.detachAppender(appender);
        appender.stop();
    }

    return appender.list.stream()
        .map(ILoggingEvent::getFormattedMessage)
        .anyMatch(message -> message.contains("SlowQuery") || message.contains("Slow query"));
}

That does two useful things:

It checks only log events produced during the probe queries.
It accepts both Hibernate 5’s SlowQuery: format and Hibernate 6’s Slow query took ... format.

Observed Results

The clean run produced this matrix:

Boot	Hibernate	Config case	Format	Property path	Observed log
2.7.18	5.6.15.Final	spring jpa properties	properties	`spring.jpa.properties.hibernate.session.events.log.LOG_QUERIES_SLOWER_THAN_MS`	true
2.7.18	5.6.15.Final	spring jpa properties	yaml	`spring.jpa.properties.hibernate.session.events.log.LOG_QUERIES_SLOWER_THAN_MS`	true
2.7.18	5.6.15.Final	properties overrides yaml	yaml+properties	`spring.jpa.properties.hibernate.session.events.log.LOG_QUERIES_SLOWER_THAN_MS`	false
2.7.18	5.6.15.Final	top-level hibernate property	yaml	`hibernate.session.events.log.LOG_QUERIES_SLOWER_THAN_MS`	false
3.3.6	6.5.3.Final	spring jpa properties	properties	`spring.jpa.properties.hibernate.session.events.log.LOG_QUERIES_SLOWER_THAN_MS`	true
3.3.6	6.5.3.Final	spring jpa properties	yaml	`spring.jpa.properties.hibernate.session.events.log.LOG_QUERIES_SLOWER_THAN_MS`	true
3.3.6	6.5.3.Final	hibernate 6 short alias	properties	`spring.jpa.properties.hibernate.log_slow_query`	true
3.3.6	6.5.3.Final	properties overrides yaml	yaml+properties	`spring.jpa.properties.hibernate.session.events.log.LOG_QUERIES_SLOWER_THAN_MS`	false
3.3.6	6.5.3.Final	top-level hibernate property	yaml	`hibernate.session.events.log.LOG_QUERIES_SLOWER_THAN_MS`	false

The positive cases produced the expected logger output:

Hibernate 5.6.15.Final:
SlowQuery: 21 milliseconds. SQL: '... select 1 from pg_sleep(0.02)'

Hibernate 6.5.3.Final:
Slow query took 23 milliseconds [select 1 from pg_sleep(0.02)]

This is the important conclusion: YAML was not the problem. YAML worked when the property was nested under spring.jpa.properties. The top-level hibernate.* key was the problem because Spring Boot did not forward that value as a JPA provider property in this setup.

Why `.properties` Changed The Result

The precedence case deliberately puts two contradictory thresholds in the same profile:

spring:
  jpa:
    properties:
      hibernate:
        session:
          events:
            log:
              LOG_QUERIES_SLOWER_THAN_MS: 1

spring.jpa.properties.hibernate.session.events.log.LOG_QUERIES_SLOWER_THAN_MS=100000

The 20 ms pg_sleep() query should log if the threshold is 1 ms. It should not log if the threshold is 100,000 ms. The matrix observed false, which means the .properties value won in that same-location mixed-format scenario.

That matches Spring Boot’s documented guidance: do not mix YAML and .properties for the same configuration surface unless you are intentionally relying on precedence. For operational config, “the value exists somewhere” is not enough. The winning value is what matters.

Hibernate 5 vs Hibernate 6 Key Names

Hibernate 5.6 exposes the slow-query setting as:

hibernate.session.events.log.LOG_QUERIES_SLOWER_THAN_MS

The Hibernate 5.6 constant values page lists that string under LOG_SLOW_QUERY.

Hibernate 6.5’s JdbcSettings.LOG_SLOW_QUERY documents the shorter key:

hibernate.log_slow_query

The lab also observed that the longer key still worked with Spring Boot 3.3.6 and Hibernate 6.5.3.Final when it was passed through spring.jpa.properties. I would not build future production configuration around that observation alone. For Hibernate 6, prefer the documented Hibernate 6 key unless you have a compatibility reason to keep the older-looking one.

The configuration path is therefore explicit: Spring loads the winning external value, Spring Boot copies spring.jpa.properties.* into the provider map, and Hibernate interprets the native key and emits the log.

Production Checklist

When slow-query logging does not appear, debug it in this order:

Confirm the provider path.

The key should be under spring.jpa.properties. If the native Hibernate key starts at the top level, Spring Boot may load it as environment config but still not put it into Hibernate’s provider settings.
Confirm the active profile and winning file.

Check whether application-prod.properties overrides application-prod.yml, whether command-line args override both, and whether an environment variable is setting a different threshold.
Confirm the Hibernate version.

Hibernate 5.6 and Hibernate 6.5 have different documented key names and different log message shapes.
Confirm the logger.

For this signal, enable org.hibernate.SQL_SLOW. Turning on broad SQL logging is noisier and does not prove the threshold is working.
Confirm the query is actually slow at the JDBC statement boundary.

The lab uses pg_sleep(0.02) because it forces database-side execution time. A slow API response can be caused by connection acquisition, transaction waits, serialization, remote calls, or application code that Hibernate’s slow-query logger will not see.

What I Would Put In A Real Service

For a Spring Boot 3 and Hibernate 6 service, I would start with:

spring:
  jpa:
    properties:
      hibernate:
        log_slow_query: 200

logging:
  level:
    org.hibernate.SQL_SLOW: info

The threshold is workload-dependent. 200 ms might be too noisy for an OLAP-heavy internal tool and too relaxed for a low-latency transactional endpoint. I would set it after looking at request SLOs, database latency percentiles, and log volume.

For Spring Boot 2 and Hibernate 5:

spring:
  jpa:
    properties:
      hibernate:
        session:
          events:
            log:
              LOG_QUERIES_SLOWER_THAN_MS: 200

logging:
  level:
    org.hibernate.SQL_SLOW: info

I would also keep one format per environment file. If the project uses YAML, keep the JPA/Hibernate config in YAML. If it uses .properties, keep it there. Mixing both can be useful for a test matrix, but it is a bad default for production readability.

Sources And Reproduction

Lab code: labs/hibernate-slow-query-lab/
Generated matrix: /resources/blog/post-66-hibernate-slow-query-logging-yaml-properties/matrix-summary.md
Slow-query log excerpt: /resources/blog/post-66-hibernate-slow-query-logging-yaml-properties/slow-query-lines.txt
Spring Boot external configuration: Externalized Configuration
Spring Boot provider properties: Common Application Properties
Hibernate 5.6 constant values: LOG_SLOW_QUERY
Hibernate 6.5 JDBC settings: JdbcSettings.LOG_SLOW_QUERY

OLTP vs. OLAP Benchmarking With HammerDB, PostgreSQL, and Query Plans

ystc1247@gmail.com — Fri, 10 Jul 2026 01:40:01 GMT

The Misleading Simplicity Of OLTP vs. OLAP

OLTP handles many small transactions, while OLAP handles analytical queries. That distinction is correct, but it is not enough for engineering work. A benchmark becomes useful only when the workload, dataset, measurement window, and interpretation boundary are explicit.

The local benchmark lab in labs/oltp-olap-benchmark/ uses PostgreSQL in Docker, HammerDB 5.0 for TPROC-C and TPROC-H workloads, and a PostgreSQL EXPLAIN (ANALYZE, BUFFERS) probe for a targeted analytical query.

The most useful result was not that one side was “faster.” The useful result was this:

A date index changed the analytical query from a parallel sequential scan to a parallel index-only scan, reduced read buffers, and cut execution time from 213.717 ms to 134.184 ms in this local run.

That is the benchmark lesson I want to keep. OLTP and OLAP are not labels to memorize. They are different ways of creating pressure on the database.

Test environment
Local architecture
Building the benchmark database
OLTP workload with TPROC-C
OLAP workload with TPROC-H
Analytical query-plan probe
What the index result proves
What the numbers mean
Production rules

Test Environment

This was a local Docker benchmark, not an official TPC result and not a production capacity claim.

Item	Value
Run date	2026-07-10
Host	macOS/Darwin on Apple Silicon
Docker	29.3.0
Docker Compose	v5.1.0
PostgreSQL image	`postgres:16-alpine`
PostgreSQL version observed	PostgreSQL 16.14
HammerDB image	`tpcorg/hammerdb:latest`
HammerDB CLI version observed	5.0
HammerDB platform	`linux/amd64` under Docker Desktop
Lab path	`labs/oltp-olap-benchmark/`

The HammerDB image ran as linux/amd64 on an ARM64 laptop. That matters. Docker Desktop virtualization and architecture emulation can distort absolute numbers, especially for CPU-bound and I/O-heavy work. The results are still useful for comparing behavior inside the same local environment, but they should not be treated as universal PostgreSQL performance values.

The clean rerun used ID 20260710T025057Z. Its generated summary is committed at /resources/blog/post-54-benchmarking-types-oltp-olap/benchmark-summary.md, and the raw HammerDB and EXPLAIN excerpts are linked below.

The full run command is intentionally boring:

cd labs/oltp-olap-benchmark
./scripts/run-all.sh

Cleanup is also explicit:

./scripts/cleanup.sh

Local Architecture

Stage	Role
`run-all.sh`	Starts from a clean state, runs both workloads, and collects logs and plans
Docker Compose	Keeps PostgreSQL and HammerDB in isolated containers
PostgreSQL 16	Hosts the TPROC-C and TPROC-H databases
HammerDB 5.0	Runs the transactional and analytical workloads
SQL probes	Captures dataset checks and `EXPLAIN (ANALYZE, BUFFERS)` evidence

The Compose file keeps the database and benchmark runner separate:

services:
  postgres:
    image: postgres:16-alpine
    ports:
      - "55432:5432"
    environment:
      POSTGRES_USER: postgres
      POSTGRES_PASSWORD: postgres
      POSTGRES_DB: postgres

  hammerdb:
    image: tpcorg/hammerdb:latest
    platform: linux/amd64
    working_dir: /home/HammerDB-5.0
    volumes:
      - .:/workspace

The important design choice is that all code shown in this post lives in the lab. The article does not use isolated snippets that cannot be run.

Building The Benchmark Database

The lab builds two separate databases:

Database	Workload	Main dataset evidence
`tpcc`	HammerDB TPROC-C	2 warehouses, 60,000 customers, 306,070 orders, 3,058,117 order lines
`tpch`	HammerDB TPROC-H	scale factor 1, 150,000 customers, 1,500,000 orders, 5,999,168 line items

The saved dataset summary is available at /resources/blog/post-54-benchmarking-types-oltp-olap/dataset-summary.txt.

The local database sizes after the run were:

Database	Size
`tpcc`	654 MB
`tpch`	2,064 MB

HammerDB calls its built-in workloads TPROC-C and TPROC-H. HammerDB documents these as workloads derived from TPC-C and TPC-H, not official audited TPC results. That distinction matters when writing or publishing benchmark claims.

OLTP Workload With TPROC-C

The TPROC-C script uses a small local configuration:

dbset db pg
dbset bm TPC-C

diset connection pg_host postgres
diset connection pg_port 5432

diset tpcc pg_count_ware 2
diset tpcc pg_num_vu 2
diset tpcc pg_rampup 1
diset tpcc pg_duration 1
diset tpcc pg_storedprocs true
diset tpcc pg_allwarehouse true

The benchmark used 2 active virtual users, a 1-minute ramp-up, and a 1-minute measured duration. That is deliberately short. I wanted a reproducible article lab, not a long capacity test.

HammerDB reported:

Metric	Value
Active virtual users	2
Warehouses	2
Ramp-up	1 minute
Measured duration	1 minute
PostgreSQL TPM	219,223
NOPM	93,918

The transaction timing breakdown is more useful than the headline number:

Transaction	Calls	p50	p95	p99	Average
New Order	492,140	0.336 ms	0.453 ms	0.717 ms	0.368 ms
Payment	489,294	0.222 ms	0.311 ms	0.479 ms	0.247 ms
Stock Level	48,766	0.378 ms	1.013 ms	56.618 ms	2.430 ms
Delivery	49,168	0.673 ms	0.990 ms	1.505 ms	0.729 ms
Order Status	49,064	0.137 ms	0.201 ms	0.268 ms	0.141 ms

The stock-level transaction had a much fatter p99 than payment or order-status, even though its p95 was close to 1 ms. That is the kind of signal OLTP benchmarking should surface. A single TPM value does not tell me which operation carries the tail-latency risk.

The Docker stats snapshot after TPROC-C showed:

Metric	Snapshot
PostgreSQL CPU	16.45%
PostgreSQL memory	203.2 MiB / 7.653 GiB
Network I/O	313 MB / 506 MB
Block I/O	45.1 kB / 11.5 GB

Again, this is a snapshot, not a full resource profile. It is still enough to remind me that OLTP pressure is not only query latency. It also creates write volume, lock pressure, WAL activity, and checkpoint/vacuum work.

The raw HammerDB result is saved at /resources/blog/post-54-benchmarking-types-oltp-olap/hammerdb-tprocc-result.txt.

OLAP Workload With TPROC-H

For the analytical side, the lab builds HammerDB’s PostgreSQL TPROC-H schema at scale factor 1:

dbset db pg
dbset bm TPC-H

diset connection pg_host postgres
diset connection pg_port 5432

diset tpch pg_scale_fact 1
diset tpch pg_num_tpch_threads 2
diset tpch pg_total_querysets 1
diset tpch pg_degree_of_parallel 2

The TPROC-H run executed one query set: 22 analytical queries.

Metric	Value
Scale factor	1
Query sets	1
Queries	22
Total query-set time	9 seconds
Geometric mean of returning query times	0.26616 seconds

Some query times from the run:

Query	Time
Query 18	2.209 s
Query 17	0.959 s
Query 1	0.890 s
Query 15	0.866 s
Query 9	0.527 s
Query 13	0.413 s

This workload feels different from TPROC-C even before looking at results. TPROC-C repeatedly mutates small transactional state. TPROC-H loads a larger analytical schema and spends its time scanning, joining, grouping, and aggregating.

The raw HammerDB result is saved at /resources/blog/post-54-benchmarking-types-oltp-olap/hammerdb-tproch-result.txt.

Analytical Query-Plan Probe

HammerDB gives workload-level evidence, but I also wanted one query-plan example I could reason about directly. The lab runs this analytical query against lineitem:

explain (analyze, buffers, timing)
select
  l_shipdate,
  count(*) as line_count,
  sum(l_extendedprice * (1 - l_discount)) as discounted_revenue
from lineitem
where l_shipdate >= date '1995-01-01'
  and l_shipdate < date '1995-04-01'
group by l_shipdate
order by l_shipdate;

Then it creates a plausible covering index:

create index idx_lineitem_shipdate_revenue_probe
  on lineitem (l_shipdate)
  include (l_extendedprice, l_discount);

The expectation was simple: filtering by l_shipdate and reading l_extendedprice plus l_discount should benefit from the index. The actual result was more interesting.

Probe	Plan shape	Execution time	Buffer evidence
Before index	parallel sequential scan + partial hash aggregate	213.717 ms	shared hit 3,179 / read 126,193
After index	parallel index-only scan + group aggregate	134.184 ms	shared hit 23,469 / read 11,303 / heap fetches 33,813

The index reduced read buffers and improved this query in the clean run. That result is useful, but it is still not a rule that “uses an index” means “better.” It is one measured outcome for this dataset, predicate, visibility state, and Docker environment.

The full plan output is saved at /resources/blog/post-54-benchmarking-types-oltp-olap/olap-index-probe.txt.

What The Index Result Proves

The indexed plan was not magic. PostgreSQL used a parallel index-only scan:

Parallel Index Only Scan using idx_lineitem_shipdate_revenue_probe on lineitem
  Index Cond: ((l_shipdate >= '1995-01-01'::date)
    AND (l_shipdate < '1995-04-01'::date))
  Heap Fetches: 33813

The baseline plan used a parallel sequential scan:

Parallel Seq Scan on lineitem
  Rows Removed by Filter: 1924843

The query selected about 224,637 rows after filtering. That is not a tiny lookup, but it was selective enough for the targeted index to reduce table reads substantially in this run. The optimized plan still performed 33,813 heap fetches, and creating the index took 2.163 seconds in the lab. The benefit is therefore conditional: it helped this read query, but it also adds write overhead, storage, vacuum interaction, and maintenance cost.

This is why EXPLAIN (ANALYZE, BUFFERS) matters. PostgreSQL’s documentation is explicit that ANALYZE executes the statement and adds actual timing and row statistics, while BUFFERS shows shared/local/temp block activity. Without those actuals, I would only have a plausible explanation.

What The Numbers Actually Mean

The benchmark supports a narrow set of claims:

HammerDB 5.0 ran PostgreSQL TPROC-C and TPROC-H workloads locally.
The TPROC-C run produced 219,223 PostgreSQL TPM and 93,918 NOPM under a 2-warehouse, 2-VU, 1-minute measured configuration.
The TPROC-H scale-factor-1 run completed one 22-query set in 9 seconds with a 0.26616-second geometric mean for returning query times.
The custom analytical query improved with the targeted index in this local run: 213.717 ms before index, 134.184 ms after index.
The indexed plan reduced read buffers and changed the query to a parallel index-only scan, but still required 33,813 heap fetches.

The benchmark does not support these claims:

PostgreSQL can handle 219,223 TPM in production.
A laptop Docker result predicts managed database performance.
This index will always improve this query in every environment.
Indexes are always good for OLAP queries.
HammerDB local output is an official TPC result.

Those unsupported claims would be overreach.

Production Implications

My final rule is:

Name the workload before choosing the metric, and inspect the plan before trusting the optimization.

For OLTP, I care about:

transaction mix;
p95/p99 latency by transaction type;
lock waits and blocking;
write amplification;
error rate;
checkpoint, vacuum, and WAL pressure;
throughput after warm-up, not just peak samples.

For OLAP, I care about:

row counts and dataset shape;
scan volume;
join and aggregate strategy;
memory and temp files;
buffer reads and hits;
parallelism;
whether an index helps the real query, not the imagined query.

The index probe is the lesson I would carry into production code review. A developer might propose lineitem(l_shipdate) include (...) and the idea would sound reasonable. The review should not stop there. The next questions are:

How selective is the date range?
Does the query still need many heap fetches?
Does the index disable a better parallel plan?
How much write overhead does the extra index add?
Is this query frequent enough to justify the storage and maintenance cost?

Benchmarking is useful only when it makes those questions answerable.

References

HammerDB documentation, checked on 2026-07-10.
HammerDB TPROC-C and TPROC-H naming note, checked on 2026-07-10.
PostgreSQL EXPLAIN documentation, checked on 2026-07-10.
PostgreSQL guide to using EXPLAIN, checked on 2026-07-10.

Thoughts on What Developers Should Do in an Age Where AI Writes Code

ystc1247@gmail.com — Thu, 02 Jul 2026 22:19:52 GMT

From Code Writing To Work Design

I think back to when I first joined the company in early 2024. If I remember correctly, it was right before GPT-4o came out. Even then, LLMs were quite helpful for writing code. But the nature of that help was very different from what it is now.

For example, suppose I had to build an API in Spring Boot. I would first complete one example API myself, flowing from controller to facade, service, and repository. Then I would copy the entire code and ask the model to create another API with the same convention and the same logic. In other words, the LLM was closer to a replication tool than a designer. If a human had already laid the path, it could follow that path to some extent.

Of course, I sometimes asked questions when I got stuck. But hallucinations were severe, and once the complexity of the task increased even slightly, syntax errors and logical flaws appeared throughout the code. At the time, the natural procedure in development was still to search through official documentation, inspect library code, read error logs, and debug directly. The LLM was an auxiliary tool, not something one could entrust work to.

After that, model performance continued to rise. At some point, the amount of time I spent relying on answers slowly began to exceed the time I spent looking things up myself. Then coding agents appeared. The first one I properly used was from the Gemini line, and to be honest, its performance was terrible. The fact that it could directly modify local files was interesting, but it was much slower than when I worked by hand, and the results were not stable. From that point on, I tried my own kind of harness engineering. I split tasks into smaller units, specified rules, forced tests, and limited the scope of modification. Even so, it could not match the quality of code written directly by a human.

The same problems repeated even in repositories where the conventions had already been established. It modified files without understanding the structure, forced together pieces of code that merely looked similar, and almost always caused problems during compilation. In the end, a developer still had to read and fix the code again. The agent may have written the code, but the responsibility still belonged entirely to the developer.

Then, around the time I left the company in early 2026, I started using newer Codex-line models, and the difference became completely tangible. The code became more precise. It performed compilation checks after the work. It began to handle not only simple implementation, but also revision and verification as a single flow. Now I could leave one task to an agent and work on something else myself. With some exaggeration, I could produce two or three times my previous productivity.

After returning to school and working on several projects, this change accelerated even further. New agents and models were released roughly once every month or two, and their performance and speed visibly improved. The parts that developers had to fix after a task was completed decreased, and the code became increasingly refined. It reached the point where, on a monthly basis, or even weekly, I felt that I would fall behind if I did not keep up. Now writing code without a coding agent feels strange.

The way I work has also changed completely. I specify branch conventions and Git flow, then run several tasks in parallel. Even PR reviews are no longer something I read entirely by hand. I built a custom dashboard to help review other people’s code, or more precisely, code written by other people’s agents. With the rise of MCP, I can connect Notion, Figma, browsers, and various other connectors, compressing into a single flow tasks that previously required moving back and forth between windows.

Where Developer Judgment Still Matters

Server incident response is the same. In the past, when a problem occurred, I directly wrote Prometheus queries, checked Grafana dashboards, dug through CloudWatch metrics, and followed JVM flame graphs or API traces in Datadog APM to find the cause. When a database issue occurred, I would spend the whole day attached to the query console, checking processes, finding expensive queries, reading execution plans, and fixing indexes and query structures. Now, a significant portion of this process can be delegated to an agent. Through an MCP server, I can have it read metrics, open a tunnel into an infrastructure server, perform Java profile capture, and assist with analyzing the result. A large portion of the repetitive exploration and text analysis humans had to perform can now be completed in an overwhelmingly shorter amount of time.

This progress was astonishing. But at the same time, it felt hollow.

It seemed as though the tasks I had been good at could now be done by anyone. The time I spent reading code, digging into libraries, spending entire days debugging, and accumulating knowledge through official documentation felt almost meaningless. In the past, that was competitiveness. The ability to understand problems faster than others, fix them more accurately, and implement them in a more stable structure. But AI began to imitate that ability rapidly. It felt as though the scarcity of the skills I had built was disappearing.

But while collaborating with several developers, my thinking slowly changed. AI was not reducing the gap between developers. It was revealing that gap even more clearly.

Many people talk about the importance of prompts. I feel it painfully as well. But the prompt I am talking about here is not the ability to write pretty sentences. It is the ability to understand the essence of a task, know the points that must be decided, understand the context of the system in which the code will live, and set conditions so that the agent cannot make careless assumptions. In the end, a good prompt comes from good development knowledge.

Suppose we are making a simple login page. If you ask for “a login page view and backend logic,” including both frontend and backend, today’s AI can produce a working login feature in under five minutes. The problem is precisely that it works. It works so convincingly that, unless you know what is missing, you simply move on.

But login is not merely a feature where a user enters an email and password, sends a POST request to the server, and is redirected to the main page on success. One must decide how sessions will be managed, how the lifecycle of access tokens and refresh tokens will be handled if JWT is used, how HttpOnly, Secure, and SameSite policies will be configured if cookies are used, how passwords will be hashed and stored, where server-side authentication and authorization checks will be performed, how to respond to credential stuffing and brute force attacks, how login failure counts and rate limits will be handled, how access privileges will be separated by user role, how logout and token invalidation will be guaranteed, and how much error information will be exposed between security and usability.

Working With Agents Without Losing Thought

These are not things that easily come to mind if one has never built a login page directly. AI can produce an answer. But the developer must know what to ask. A person who does not know what must be decided will accept the assumptions filled in by AI as they are. At that moment, the code has not been “completed.” It has entered the repository carrying invisible debt.

That is why I find it strange to treat “AI contribution” as an independent value. If we assume the contribution to code quality is 100, some people say that in the past, developer ability accounted for 90 and LLM assistance added 10, while now AI accounts for 90 and the developer handles only 10. But I do not think so. The quality of the 90 that AI produces does not fall from the sky as a separate value. It still stands on the developer’s prior knowledge, understanding of the system, ability to decompose requirements, and ability to use AI well. Without that foundation, the 90 produced by AI is not quality. It is a convincing bundle of assumptions.

In fact, the gap may grow even wider. In the past, the difference between an excellent developer and an average developer appeared in coding speed and debugging ability. Now that difference appears in the way they handle agents. One person tells AI, “Make a login.” Another person organizes authentication policy, security requirements, error handling, test scope, and integration with the existing architecture before assigning the task. Both use AI. But the results are completely different.

The hiring market is similar. These days, people often say that the productivity of existing developers has increased, so new developers are no longer needed, and that is why the market has become difficult. Of course, there is some truth to that. If a skilled developer uses agents well, one person can handle far more work than before. The need for people who only perform simple implementation is bound to decrease.

But from what I hear from people in the field, the issue does not seem to be simply that companies are hiring fewer juniors. It also feels strongly as though there are fewer people worth hiring. Development looks too easy now. Output appears even without digging deeply into code. One can complete a project without having implemented things directly. But if engineering thinking is not properly trained in that process, one does not understand why the code was written that way. When an error occurs, one cannot narrow down the cause. When structures collide, one does not know where to begin unraveling them. The problem is not that one cannot write code without AI. The problem is being unable to properly read even the code AI has written.

Of course, studying code itself has also become much harder. One can say that the ideal approach is to write code without AI and understand logic from the bottom up, as people did before. I also believe that is important. But when there is a tool right in front of me that can perform a task I cannot do at ten times the speed, how could I ignore it? Saying we should not use AI is not realistic. What matters now is not whether we use AI, but how we use AI without losing thought.

Before AI, learning development was mostly a bottom-up process. You learned syntax, then data structures, networks, and databases. You built small features by hand, then slowly climbed toward more complex systems. That was not necessarily because bottom-up learning was always the best methodology. It was because there were few alternatives. Without the fundamentals, you could not write high-level code, especially code that carried complex responsibilities and abstractions.

With AI, a top-down approach has become practical. You can first have an agent produce higher-level code, then analyze why the code is shaped that way, and dig backward into the parts you do not yet understand. From a learner’s point of view, this can compress study time dramatically. Andrej Karpathy, whose recent LLM observations helped popularize the practice of writing files such as CLAUDE.md to improve Claude Code’s performance, tweeted a similar idea in 2020. His advice was to take on concrete projects, learn deeply on demand, and summarize what you learned in your own words. At the time, that sounded ideal but not always practical. In the AI era, it has become a much more realistic learning strategy.

There is one method I at least try to practice. Before assigning a task, I write the requirements in as much detail as possible. Then, at the end, I always add this question.

What do I need to decide in advance in order to perform this task? Do not assume any workflow or logic. Have me review every process in detail.

This question matters because AI does not leave blanks empty. Any instruction a developer does not include in the prompt becomes something the model has to infer. It will proceed with the implementation according to what looks most appropriate from its point of view. But every project has its own development direction. Once data modeling, authentication style, deployment structure, incident response, and future extensibility are involved, the “right” direction cannot be decided by general best practice alone. That judgment ultimately belongs to the developer responsible for the system.

A good prompt is not an attempt to use AI less. It is closer to separating the execution area where AI should be powerful from the design area the developer must decide in advance. With a weak prompt, the human decides 10% and the remaining 90% becomes AI assumptions. With a precise prompt, the developer provides 90% of the context, constraints, forbidden paths, and review criteria, while AI fills only the remaining 10% with tactical judgment. In both cases, AI writes code. But the nature of the output is completely different.

Responsibility Moves Upstream

Just adding this one question changes the result significantly. No matter how detailed I think the prompt is, something I had forgotten always comes out. Sometimes, an entire procedure I had never considered returns as a question. I must then study those questions again and make decisions. Only then do I avoid making the agent redo the same work twice, three times, or ten times.

Understanding the written code is also important. Code written by an agent is not code one can skip reading. Rather, it is code one must read even more carefully. The first task always starts off looking plausible. The first feature in a new repository is clean. But if one continues assigning follow-up tasks without understanding that structure, spaghetti code eventually accumulates. Logic collides, responsibilities blur, exception handling is duplicated, and the same concept spreads under multiple names. At first, it seems fast. Later, it collapses beyond control.

In the age of AI, developers may write less code. But that does not mean they can think less. If anything, they must think more. The time spent typing directly decreases, but one must judge what to build, what structure is appropriate, which decisions must not be deferred, and which assumptions must be prevented. Implementation labor decreases, but design, review, and responsibility become more distinct.

That is why the difference between a developer and a coder now seems even clearer.

A coder receives code written by AI. A developer decides what kind of code AI should write. A coder thinks it is over when it works. A developer looks at why it works, where it can break, and what kind of structure can survive when the next feature is added. A coder writes prompts as commands. A developer writes prompts like design documents.

The age in which AI writes code has arrived. This does not mean developers are no longer needed. It means, rather, that what a developer knows has become more important. In the past, we learned what we did not know by struggling through it ourselves. Now AI covers up what we do not know too quickly. That is why we must dig into it more consciously. We should not merely look at the answer AI filled in. We must check what AI assumed.

In the end, competitiveness has not disappeared. Its position has merely changed. It has moved from how much code one can type to how accurately one can define what kind of code should be written. And this change is colder than it first appears. AI is open to everyone, but not everyone obtains the same result.

The False Sense of Safety Around PriorityClass and PodDisruptionBudget

ystc1247@gmail.com — Sun, 21 Sep 2025 05:08:10 GMT

Where the Problem Started

A PodDisruptionBudget can sound like a hard guarantee. But Kubernetes can still evict lower-priority pods to make room for more important ones, and drain behavior can produce results that surprise operators. This post examines the false sense of safety that appears when PriorityClass and PDB are used together.

When using Kubernetes, it is common to configure PriorityClass and PodDisruptionBudget(PDB) together. PriorityClass is a mechanism that says, “this pod is important,” by assigning priority, and PDB is a constraint that says, “at least this many must stay alive.”

At first, it is easy to think this way.

“Since I configured a PDB, the minimum count should be guaranteed, right?”

But in real operations, pods can disappear even when a PDB exists. Especially in scheduler preemption situations or node drain operations, PDB is not absolutely guaranteed. In this post, I will summarize why this happens, how it is handled in the scheduler architecture and at the code level, and how to verify it with actual examples.

The Roles of PriorityClass and PDB

PriorityClass

Pods with a high PriorityClass value are treated as more important by the scheduler. If cluster resources are insufficient, the scheduler forcibly evicts lower PriorityClass pods in order to run higher PriorityClass pods. This process is called Preemption.

PDB (PodDisruptionBudget)

It prevents too many pods from going down at the same time during voluntary disruptions such as drain or rolling updates. For example, if minAvailable: 2 is set, at least two pods must stay alive. However, situations like node failure or preemption are not guaranteed.

How PDB Handling Looks Inside the Scheduler Code

The scheduler does not actually ignore PDB completely. If you look at pkg/scheduler/framework/plugins/defaultpreemption/default_preemption.go, there is code that checks PDBs when selecting victims:

// SelectVictimsOnNode finds minimum set of pods on the given node that should be preempted in order to make enough room
// for "pod" to be scheduled.
func (pl *DefaultPreemption) SelectVictimsOnNode(
	ctx context.Context,
	state fwk.CycleState,
	pod *v1.Pod,
	nodeInfo fwk.NodeInfo,
	pdbs []*policy.PodDisruptionBudget) ([]*v1.Pod, int, *fwk.Status) {
	logger := klog.FromContext(ctx)
	var potentialVictims []fwk.PodInfo
	removePod := func(rpi fwk.PodInfo) error {
		if err := nodeInfo.RemovePod(logger, rpi.GetPod()); err != nil {
			return err
		}
		status := pl.fh.RunPreFilterExtensionRemovePod(ctx, state, pod, rpi, nodeInfo)
		if !status.IsSuccess() {
			return status.AsError()
		}
		return nil
	}
	addPod := func(api fwk.PodInfo) error {
		nodeInfo.AddPodInfo(api)
		status := pl.fh.RunPreFilterExtensionAddPod(ctx, state, pod, api, nodeInfo)
		if !status.IsSuccess() {
			return status.AsError()
		}
		return nil
	}
	// As the first step, remove all pods eligible for preemption from the node and
	// check if the given pod can be scheduled without them present.
	for _, pi := range nodeInfo.GetPods() {
		if pl.isPreemptionAllowed(nodeInfo, pi, pod) {
			potentialVictims = append(potentialVictims, pi)
			if err := removePod(pi); err != nil {
				return nil, 0, fwk.AsStatus(err)
			}
		}
	}

In other words, the scheduler respects PDBs when it can. But if every viable path requires breaking a PDB, Kubernetes can still choose preemption and evict the lower-priority pod.

The official documentation also says this:

PodDisruptionBudget is supported, but not guaranteed. The scheduler tries not to violate PDB when possible, but if it cannot find an alternative victim, it removes lower-priority pods even if that breaks PDB.

Kubernetes Docs

Example Where PDB Is Broken During Preemption

Create PriorityClass

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: low-priority
value: 1000
globalDefault: false
description: "낮은 우선순위 파드"

---
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: high-priority
value: 100000
globalDefault: false
description: "높은 우선순위 파드"

Create Deployment + PDB

apiVersion: apps/v1
kind: Deployment
metadata:
  name: demo-deploy
spec:
  replicas: 3
  selector:
    matchLabels:
      app: demo
  template:
    metadata:
      labels:
        app: demo
    spec:
      priorityClassName: low-priority
      containers:
      - name: nginx
        image: nginx

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: demo-pdb
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: demo

Add a high PriorityClass pod

apiVersion: v1
kind: Pod
metadata:
  name: important-pod
spec:
  priorityClassName: high-priority
  containers:
  - name: busy
    image: busybox
    command: ["sh", "-c", "sleep 3600"]
    resources:
      requests:
        cpu: "500m"
        memory: "512Mi"

If cluster resources are insufficient, the scheduler sacrifices one of the demo-deploy pods to run important-pod. During this process, the PDB(minAvailable: 2) condition can be broken.

If you run kubectl describe pdb demo-pdb, CurrentHealthy may have dropped to 1 instead of 2.

Example Where PDB Is Broken During Drain

Now test the same assumption during drain.

drain-demo Deployment and PDB

apiVersion: apps/v1
kind: Deployment
metadata:
  name: drain-demo
spec:
  replicas: 3
  selector:
    matchLabels:
      app: drain-demo
  template:
    metadata:
      labels:
        app: drain-demo
    spec:
      priorityClassName: low-priority
      containers:
      - name: nginx
        image: nginx

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: drain-demo-pdb
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: drain-demo

Run node drain

kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data

Check the result

If you run kubectl get pods, you may see that only one pod is left.

Result of kubectl describe pdb drain-demo-pdb:

Status:
  Current Healthy:   1
  Desired Healthy:   2
  Disruptions Allowed: 0

In other words, during the drain process, the PDB condition is broken and pods disappear.

Why Does This Happen?

PDB is a drain/update protection mechanism, but not an absolute guarantee

Even though drain itself is a voluntary disruption, there are cases where the PDB condition cannot be fully honored while emptying a node.

Asynchronous controller structure

Because the PDB controller and scheduler run separately, if the timing of status updates is off, the scheduler can make decisions based on stale information.

A philosophical choice

Kubernetes follows the philosophy of “while preserving service availability as much as possible, running the more important pod comes first.” That is why PDB violations are allowed.

Things To Watch In Operations

PDB is not an absolute shield. For truly important workloads, you should raise PriorityClass and use PDB as a supporting mechanism.
A drain strategy is necessary. In production environments, you need to design deployment/update strategy on the assumption that kubectl drain may ignore PDB.
Monitoring is mandatory. You should periodically check disruptionsAllowed and currentHealthy with kubectl get pdb.
Practice beforehand. Intentionally trigger preemption and drain in a controlled environment before relying on the policy in production.

Summary

PriorityClass determines pod priority and removes lower PriorityClass pods when needed.
PDB limits voluntary disruptions, but it is not guaranteed in situations such as preemption or drain.
Kubernetes’ basic philosophy is “run important pods first,” and breaking PDB is also allowed in that process.
For pods you really want to protect, design them with a PriorityClass + PDB combination and think through the drain strategy as well.

Availability Takeaway

PDB is an important safety mechanism, but not an absolute shield. Scheduling, preemption, drain, and controller update timing all affect the real availability story. For important workloads, PDB needs to be designed together with priority, replica placement, rollout strategy, and monitoring.

Why an ORM Query Missed a VARCHAR Index in SQL Server

ystc1247@gmail.com — Sat, 25 Jan 2025 16:25:18 GMT

Where the Problem Started

When an index exists but is not used, intuition usually fails until the execution plan is inspected. With an ORM, a condition that looks harmless in application code can become a different plan once JDBC parameter types and database type conversion are involved. This post follows a SQL Server case where a VARCHAR index was skipped.

I observed that when an ORM uses a string value in a query, it sends the query using the NVARCHAR type. So when an index contains columns declared as VARCHAR, and the query tries to use that index, a type mismatch causes the index not to be used.

For example, consider PK__CUST_INFO__7AA72534 (CARE_ORG_ID char(8), CUST_NO varchar(20)), a simple two-column index.

I also suspected that statistics might have had an error and prevented the index from being used smoothly, so I ran a query that ignores statistics through option(recompile).

Implementation Path

select * from cust_info where CARE_ORG_ID = '15887361' and CUST_NO = '#1030197' option(recompile)
select * from cust_info where CARE_ORG_ID = N'15887361' and CUST_NO = N'#1030197' option(recompile)

How I Verified It

This is Query Analysis in DataGrip. The first query used the intended index, but the second query, which uses nvarchar parameters, unexpectedly performed an INDEX_CUST_INFO_CELL_PHONE index seek before using the PK index.

Because the CELL_PHONE index is non-clustered, it does not contain every column. SQL Server appears to fetch candidate rows from that index and then join/look up the remaining row data through the PK.

This is the result of dropping the CELL_PHONE index from a copied CUST_INFO table and running the query with nvarchar params.

The inner join in a single-table query looks odd at first. My inference is that because the parameters were bound as nvarchar, SQL Server had to convert CARE_ORG_ID and CUST_NO values to nvarchar during plan execution, which broke the expected direct index usage.

I then changed both CARE_ORG_ID and CUST_NO to nvarchar and reran the same two queries.

Performance Takeaway

An ORM hides SQL generation, but it does not hide database behavior. Parameter type, implicit conversion, collation, and index column type can make a good index useless. Performance debugging has to read application code and the actual database execution plan together.

When the column types are also nvarchar, the index is used regardless of whether the parameter is sent as nvarchar or varchar.

Why did the previous table column type conversion happen? This can be checked at https://learn.microsoft.com/en-us/sql/t-sql/data-types/data-type-precedence-transact-sql?view=sql-server-ver16.

In short, when querying a varchar type column with nvarchar, nvarchar has higher precedence, so type conversion occurs on every row of the entire table. But when querying an nvarchar column with varchar, conversion occurs only on the query parameter before the query runs, so the same query plan occurs.

The fact that even the simplest form of index failed to be used made it clear that all queries executed by the ORM, where there are often joins across as many as 10 or more tables, were not using the intended indexes at all. Three possible solutions come to mind.

Convert every column type used by indexes to nvarchar.

Because nvarchar supports Unicode, unlike varchar/char, which uses 1 byte per character, it consumes 2 bytes per character. This increases not only the overall database size but also the index size.

Connected to the above, because the data value becomes larger, I/O work takes more time and memory usage also increases.

Changing a column’s type requires dropping the index, changing the type, and recreating the index if that column is used in an index.

We confirmed above that even when using nvarchar columns, a varchar parameter can still use the index, but there may be unknown side effects, especially in programs that use native query.

Keep string parameters in the ORM as nvarchar by default, but exceptionally send queries as varchar/char for columns that need to use indexes.

This is the most ideal solution, but applying it in a Spring Boot project is strangely painful. I will write a separate document for it.

Conversely, make all string parameters varchar, and handle some columns declared as nvarchar.

Tracing SQL Server JDBC Unicode Parameter Behavior Inside the Library

ystc1247@gmail.com — Sat, 25 Jan 2025 16:17:26 GMT

Where the Problem Started

The existence of a configuration option does not guarantee that it is applied at the point one expects. When parameter binding is decided inside a database driver, there can be a gap between documented behavior and the actual call path. This post traces why setSendStringParametersAsUnicode=false did not behave as expected inside the SQL Server JDBC driver.

As a follow-up to the SQL Server VARCHAR index issue, I needed a way to leave parameters as the existing nvarchar values while querying parameters that use an index as varchar/char.

One option is to send all string parameters as varchar by adding setSendStringParametersAsUnicode=false to the JDBC URL.

When that is set, parameters that used to be sent as nvarchar are all sent as varchar. But a few columns in some DBs are exceptionally declared as nvarchar. To handle those, I used the @Nationalized annotation. With that in place, even if the JDBC URL declares that parameters should be sent as varchar, the attribute’s isNationalized value is set to true, so an nvarchar type query is generated.

The problem here is that when saving, the query is sent as nvarchar, so all unicode special characters are saved properly. But when reading this column value from some DBs, the driver raises Could not extract column x from JDBC ResultSet - The conversion from varchar to NCHAR is unsupported.

Implementation Path

public void contributeTypes(TypeContributions typeContributions, ServiceRegistry serviceRegistry) {
    // by default, not much to do...
    registerColumnTypes( typeContributions, serviceRegistry );
    final NationalizationSupport nationalizationSupport = getNationalizationSupport();
    final JdbcTypeRegistry jdbcTypeRegistry = typeContributions.getTypeConfiguration().getJdbcTypeRegistry();
    if ( nationalizationSupport == NationalizationSupport.EXPLICIT ) {
       jdbcTypeRegistry.addDescriptor( NCharJdbcType.INSTANCE );
       jdbcTypeRegistry.addDescriptor( NVarcharJdbcType.INSTANCE );
       jdbcTypeRegistry.addDescriptor( LongNVarcharJdbcType.INSTANCE );
       jdbcTypeRegistry.addDescriptor( NClobJdbcType.DEFAULT );
    }

When @Nationalized is declared, NVarcharJdbcType is added to the jdbcTypeRegistry in the contributeTypes method of SQLServerDialect.java. Then, when Hibernate tries to map db values → Java objects, the extract function in BasicExtractor.java, which is used for that mapping,

@Override
public J extract(ResultSet rs, int paramIndex, WrapperOptions options) throws SQLException {
    final J value = doExtract( rs, paramIndex, options );
    if ( value == null || rs.wasNull() ) {
       if ( JdbcExtractingLogging.LOGGER.isTraceEnabled() ) {
          JdbcExtractingLogging.logNullExtracted(
                paramIndex,
                getJdbcType().getDefaultSqlTypeCode()
          );
       }
       return null;
    }

takes the abstract function doExtract from NVarcharJdbcType.

@Override
public <X> ValueExtractor<X> getExtractor(final JavaType<X> javaType) {
    return new BasicExtractor<>( javaType, this ) {
       @Override
       protected X doExtract(ResultSet rs, int paramIndex, WrapperOptions options) throws SQLException {
          return javaType.wrap( rs.getNString( paramIndex ), options );
       }
       @Override
       protected X doExtract(CallableStatement statement, int index, WrapperOptions options) throws SQLException {
          return javaType.wrap( statement.getNString( index ), options );
       }
       @Override
       protected X doExtract(CallableStatement statement, String name, WrapperOptions options) throws SQLException {
          return javaType.wrap( statement.getNString( name ), options );
       }
    };
}

Looking at the getNString function,

@Override
public String getNString(int columnIndex) throws SQLException {
    loggerExternal.entering(getClassNameLogging(), "getNString", columnIndex);
    checkClosed();
    String value = (String) getValue(columnIndex, JDBCType.NCHAR);
    loggerExternal.exiting(getClassNameLogging(), "getNString", value);
    return value;
}

the driver tries to retrieve the stored value as NCHAR.

I added the @Nationalized annotation to handle columns that were declared as NVARCHAR at the request of some clients. But when most database columns that are actually stored as VARCHAR are converted this way, The conversion from varchar to NCHAR is unsupported. gets raised.

If we dig into where the error is ultimately thrown,

dtv.java

Object getValue(DTV dtv, JDBCType jdbcType, int scale, InputStreamGetterArgs streamGetterArgs, Calendar cal,
        TypeInfo typeInfo, CryptoMetadata cryptoMetadata, TDSReader tdsReader, SQLServerStatement statement) throws SQLServerException {
    SQLServerConnection con = tdsReader.getConnection();
    Object convertedValue = null;
    byte[] decryptedValue;
    boolean encrypted = false;
    SSType baseSSType = typeInfo.getSSType();
    // If column encryption is not enabled on connection or on statement, cryptoMeta will be null.
    if (null != cryptoMetadata) {
        assert (SSType.VARBINARY == typeInfo.getSSType()) || (SSType.VARBINARYMAX == typeInfo.getSSType());
        baseSSType = cryptoMetadata.baseTypeInfo.getSSType();
        encrypted = true;
        if (aeLogger.isLoggable(java.util.logging.Level.FINE)) {
            aeLogger.fine("Data is encrypted, SQL Server Data Type: " + baseSSType + ", Encryption Type: "
                    + cryptoMetadata.getEncryptionType());
        }
    }
    // Note that the value should be prepped
    // only for columns whose values can be read of the wire.
    // If valueMark == null and isNull, it implies that
    // the column is null according to NBCROW and that
    // there is nothing to be read from the wire.
    if (null == valueMark && (!isNull))
        getValuePrep(typeInfo, tdsReader);
    // either there should be a valueMark
    // or valueMark should be null and isNull should be set to true(NBCROW case)
    assert ((valueMark != null) || (valueMark == null && isNull));
    if (null != streamGetterArgs) {
        if (!streamGetterArgs.streamType.convertsFrom(typeInfo))
            DataTypes.throwConversionError(typeInfo.getSSType().toString(), streamGetterArgs.streamType.toString());
    } else {
        if (!baseSSType.convertsTo(jdbcType) && !isNull) {
            // if the baseSSType is Character or NCharacter and jdbcType is Longvarbinary,
            // does not throw type conversion error, which allows getObject() on Long Character types.
            if (encrypted) {
                if (!Util.isBinaryType(jdbcType.getIntValue())) {
                    DataTypes.throwConversionError(baseSSType.toString(), jdbcType.toString());
                }
            } else {
                DataTypes.throwConversionError(baseSSType.toString(), jdbcType.toString());
            }
        }

boolean convertsTo(JDBCType jdbcType) {
    return GetterConversion.converts(this, jdbcType);
}

static final boolean converts(SSType fromSSType, JDBCType toJDBCType) {
    return conversionMap.get(fromSSType.category).contains(toJDBCType.category);
}

The current situation is that fromSSType(DB declared type) is varchar and toJDBCType(Application declared type) is NCHAR. In this case, SSTYPE becomes CHARACTER, and because NCHAR is not in the map, the error occurs.

How I Verified It

The possible solutions I considered were all unsatisfying:

Remove @Nationalized, accept that some special characters cannot be saved/read correctly, and treat everything as varchar
At runtime, when executing a select query, inspect the column type and forcibly change the descriptor in jdbcTypeRegistry
Keep the columns declared as nvarchar in a separate table and do step 2
Remove the @Nationalized annotation and use native query in CUD query to force type casting to nvarchar (of course, since reads are done as varchar, there will be defects for some special characters in columns declared as nvarchar)
Clone the mssql-jdbc library and create v2 If I were to modify the driver, adding SSType.Category.NCHAR to CHARACTER in http://m.microsoft.sqlserver.jdbc.SSType.GetterConversion looked like the relevant point. Looking at http://m.microsoft.sqlserver.jdbc.JDBCType.UpdaterConversion, CHARACTER already contains SSType.Category.NCHAR, so the write path appears to avoid the same failure. That makes the read-side gap feel especially unfortunate.

Below is source code that forcibly adds NCHARACTER to the CHARACTER enum map of GetterConversion. Since it directly mutates library internals, it absolutely needs strict review before use.

Reflection, Unsafe, immutable enum internals, and error suppression all make this unpleasant. If there is a cleaner approach, I would much rather use that.

import jakarta.annotation.PostConstruct
import org.springframework.context.annotation.Configuration
import sun.misc.Unsafe
import java.lang.reflect.Field
import java.lang.reflect.Modifier
import java.util.Collections
import java.util.EnumSet

@Configuration
class SSTypeConfiguration {

    @PostConstruct
    fun modifyCharacterEnumSet() {
        try {
            val getterConversionClass = Class.forName("com.microsoft.sqlserver.jdbc.SSType\$GetterConversion")
            val characterField: Field = getterConversionClass.getDeclaredField("CHARACTER")
            characterField.isAccessible = true

            val characterEnum = characterField.get(null)

            val categoriesField: Field = characterEnum.javaClass.getDeclaredField("to")
            categoriesField.isAccessible = true

            val unsafeField = Unsafe::class.java.getDeclaredField("theUnsafe")
            unsafeField.isAccessible = true
            val unsafe = unsafeField.get(null) as Unsafe

            val fieldOffset = unsafe.objectFieldOffset(categoriesField)

            @Suppress("UNCHECKED_CAST")
            val enumSet = categoriesField.get(characterEnum) as EnumSet<*>

            val nCharacterCategory = Class.forName("com.microsoft.sqlserver.jdbc.JDBCType\$Category")
                .enumConstants.first { (it as Enum<*>).name == "NCHARACTER" }

            @Suppress("UNCHECKED_CAST")
            (enumSet as MutableSet<Any>).add(nCharacterCategory)

            val immutableEnumSet = Collections.unmodifiableSet(enumSet)

            unsafe.putObject(characterEnum, fieldOffset, immutableEnumSet)

            val conversionMapField = getterConversionClass.getDeclaredField("conversionMap")
            conversionMapField.isAccessible = true
            @Suppress("UNCHECKED_CAST")
            val conversionMap = conversionMapField.get(null) as MutableMap<Any, EnumSet<*>>

            val sstypeCategoryClass = Class.forName("com.microsoft.sqlserver.jdbc.SSType\$Category")
            val characterCategory = sstypeCategoryClass.enumConstants.first { (it as Enum<*>).name == "CHARACTER" }

            @Suppress("UNCHECKED_CAST")
            (conversionMap[characterCategory] as MutableSet<Any>?)?.add(nCharacterCategory)
           } catch (e: Exception) {
            // runtime에 터뜨리고 롤백과정을 밟는 것이 이상적일듯함
        }
    }
}

Driver Boundary Takeaway

Problems like this do not end at application configuration. The driver’s parameter-binding behavior has to be understood, and sometimes that means reading library source. Production performance issues often become most complex at the boundary between framework, driver, and database engine.

What to Know When Scaling Pods Down to Zero with KEDA

ystc1247@gmail.com — Wed, 09 Oct 2024 09:32:03 GMT

Where the Problem Started

Scale-to-zero looks attractive because it reduces cost, but the behavior is not fully explained by the default Kubernetes HPA alone. Once replicas reach zero, metric collection and scaling triggers become a different responsibility. This post follows KEDA and HPA through the source code to understand that division.

Kubernetes has HPA(Horizontal Pod Autoscaler) as its basic scaler. It exists for scale out and scale in, and adjusts the pod replica count.

The basis for this adjustment is whether the resource metric(CPU, memory, etc.) specified through yaml or api is above or below the target utilization.

Start with a simple HPA YAML.

Implementation Path

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: example-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: example-deployment
  minReplicas: 2
  maxReplicas: 10
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 80

With this HPA, Kubernetes increases replicas when average CPU utilization goes above 80% and scales in when it falls below the target.

The interesting part is how that CPU calculation works when multiple pods are involved.

HPA uses a formula like this.

desiredReplicas = ceil[currentReplicas * (currentMetricValue / targetMetricValue)]

For example, assume there are currently 4 replicas, averageUtilization is declared as 80%, and the current average cpu of the pods is 120%.

desiredReplicas = ceil[4 * (120 / 80)] = ceil[6] = 6 replicas

Applying the formula above shows that it should scale out from 4 replicas to 6 replicas.

While using HPA like this, I suddenly thought that it would be economical to reduce the replica count to 0 on development/release servers at dawn, when there is no resource usage at all. So I set minReplica to 0 and applied it, but the replica count never dropped below 1.

From a developer’s perspective, the biggest advantage of an open source project is that you can open up the source code, so I looked through the k8s HPA code.

func startHPAControllerWithMetricsClient(ctx context.Context, controllerContext ControllerContext, metricsClient metrics.MetricsClient) (controller.Interface, bool, error) {

	hpaClient := controllerContext.ClientBuilder.ClientOrDie("horizontal-pod-autoscaler")
	hpaClientConfig := controllerContext.ClientBuilder.ConfigOrDie("horizontal-pod-autoscaler")

	// we don't use cached discovery because DiscoveryScaleKindResolver does its own caching,
	// so we want to re-fetch every time when we actually ask for it
	scaleKindResolver := scale.NewDiscoveryScaleKindResolver(hpaClient.Discovery())
	scaleClient, err := scale.NewForConfig(hpaClientConfig, controllerContext.RESTMapper, dynamic.LegacyAPIPathResolverFunc, scaleKindResolver)
	if err != nil {
		return nil, false, err
	}

	go podautoscaler.NewHorizontalController(
		ctx,
		hpaClient.CoreV1(),
		scaleClient,
		hpaClient.AutoscalingV2(),
		controllerContext.RESTMapper,
		metricsClient,
		controllerContext.InformerFactory.Autoscaling().V2().HorizontalPodAutoscalers(),
		controllerContext.InformerFactory.Core().V1().Pods(),
		controllerContext.ComponentConfig.HPAController.HorizontalPodAutoscalerSyncPeriod.Duration,
		controllerContext.ComponentConfig.HPAController.HorizontalPodAutoscalerDownscaleStabilizationWindow.Duration,
		controllerContext.ComponentConfig.HPAController.HorizontalPodAutoscalerTolerance,
		controllerContext.ComponentConfig.HPAController.HorizontalPodAutoscalerCPUInitializationPeriod.Duration,
		controllerContext.ComponentConfig.HPAController.HorizontalPodAutoscalerInitialReadinessDelay.Duration,
	).Run(ctx, int(controllerContext.ComponentConfig.HPAController.ConcurrentHorizontalPodAutoscalerSyncs))
	return nil, true, nil
}

The controller setup creates the HPA controller. Following that path into NewHorizontalController,

replicaCalc := NewReplicaCalculator(
    metricsClient,
    hpaController.podLister,
    tolerance,
    cpuInitializationPeriod,
    delayOfInitialReadinessStatus,
)

the replica calculator stands out. The fields are roughly:

metricsClient : Client for fetching and aggregating CPU or memory util

podLister : Client for fetching actual pod metrics such as current readiness and health

tolerance : Scaling only happens when the replica count calculation exceeds this number

cpuInitializationPeriod : Since cpu is unstable when a pod is first created, it is excluded from calculation during this period

delayOfInitialReadinessStatus : This does not exclude the pod; it is just the period during which HPA will not perform replica scaling

Inside GetResourceReplicas in replica_calculator.go, the final replica count calculation becomes visible.

// re-run the utilization calculation with our new numbers
newUsageRatio, _, _, err := metricsclient.GetResourceUtilizationRatio(metrics, requests, targetUtilization)
if err != nil {
    return 0, utilization, rawUtilization, time.Time{}, err
}

if math.Abs(1.0-newUsageRatio) <= c.tolerance || (usageRatio < 1.0 && newUsageRatio > 1.0) || (usageRatio > 1.0 && newUsageRatio < 1.0) {
    // return the current replicas if the change would be too small,
    // or if the new usage ratio would cause a change in scale direction
    return currentReplicas, utilization, rawUtilization, timestamp, nil
}

newReplicas := int32(math.Ceil(newUsageRatio * float64(len(metrics))))
if (newUsageRatio < 1.0 && newReplicas > currentReplicas) || (newUsageRatio > 1.0 && newReplicas < currentReplicas) {
    // return the current replicas if the change of metrics length would cause a change in scale direction
    return currentReplicas, utilization, rawUtilization, timestamp, nil
}

// return the result, where the number of replicas considered is
// however many replicas factored into our calculation
return newReplicas, utilization, rawUtilization, timestamp, nil

But here, I cannot find any validation saying newReplicas must not be 0. It was the same when I looked through the private functions that appear in between.

So I went back to horizontal.go.

The relevant snippet looks like this:

var minReplicas int32

if hpa.Spec.MinReplicas != nil {
    minReplicas = *hpa.Spec.MinReplicas
} else {
    // Default value
    minReplicas = 1
}

rescale := true
logger := klog.FromContext(ctx)

if currentReplicas == 0 && minReplicas != 0 {
    // Autoscaling is disabled for this resource
    desiredReplicas = 0
    rescale = false
    setCondition(hpa, autoscalingv2.ScalingActive, v1.ConditionFalse, "ScalingDisabled", "scaling is disabled since the replica count of the target is zero")
} else if currentReplicas > hpa.Spec.MaxReplicas {
    rescaleReason = "Current number of replicas above Spec.MaxReplicas"
    desiredReplicas = hpa.Spec.MaxReplicas
} else if currentReplicas < minReplicas {
    rescaleReason = "Current number of replicas below Spec.MinReplicas"
    desiredReplicas = minReplicas
}

minReplicas is a user-declared value, and in the YAML above it is set to 2. No matter how low the resource metric becomes, HPA must keep at least that many replicas. If it is not declared, the default becomes 1, and Spec.MinReplicas cannot normally be set to 0.

// minReplicas is the lower limit for the number of replicas to which the autoscaler
// can scale down.  It defaults to 1 pod.  minReplicas is allowed to be 0 if the
// alpha feature gate HPAScaleToZero is enabled and at least one Object or External
// metric is configured.  Scaling is active as long as at least one metric value is
// available.
// +optional
MinReplicas *int32 `json:"minReplicas,omitempty" protobuf:"varint,2,opt,name=minReplicas"`

The comment explains the rule: unless the alpha feature gate is enabled and Object/External metrics are configured, minReplicas cannot be 0. validate.go confirms the same constraint in code.

func validateMetrics(metrics []autoscaling.MetricSpec, fldPath *field.Path, minReplicas *int32) field.ErrorList {
    allErrs := field.ErrorList{}
    hasObjectMetrics := false
    hasExternalMetrics := false

    for i, metricSpec := range metrics {
       idxPath := fldPath.Index(i)
       if targetErrs := validateMetricSpec(metricSpec, idxPath); len(targetErrs) > 0 {
          allErrs = append(allErrs, targetErrs...)
       }
       if metricSpec.Type == autoscaling.ObjectMetricSourceType {
          hasObjectMetrics = true
       }
       if metricSpec.Type == autoscaling.ExternalMetricSourceType {
          hasExternalMetrics = true
       }
    }

    if minReplicas != nil && *minReplicas == 0 {
       if !hasObjectMetrics && !hasExternalMetrics {
          allErrs = append(allErrs, field.Forbidden(fldPath, "must specify at least one Object or External metric to support scaling to zero replicas"))
       }
    }

    return allErrs
}

However, alpha feature gates are settings for unstable experimental features that are not yet ready for general use. In the end, this means you cannot reduce minReplicas to 0 when using only k8s in production.

This is where Keda comes in. Like k8s, it is a CNCF graduate project, and it is a scaler widely used as an alternative to HPA.

As you can tell from “alternative”, you cannot declare HPA and Keda at the same time for the same target.

How I Verified It

This is the key flow image for understanding Keda. The most important part is that Keda is responsible for adjusting the replica count from 0->1 and 1->0, while HPA handles the rest. You can find the details in the Keda docs.

The following is a yaml file that can make the pod replica count 0.

{{- if or (eq .Values.env "develop") (eq .Values.env "release") }}
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: some-application-{{ .Values.env }}-scaledobject
  namespace: some-application
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: some-application-{{ .Values.env }}
  cooldownPeriod: 300
  minReplicaCount: 0
  maxReplicaCount: 10
  triggers:
    - type: cron
      metadata:
        timezone: Asia/Seoul
        start: "30 08 * * *"
        end: "30 19 * * *"
        desiredReplicas: "1"
    - type: memory
      metricType: Utilization
      metadata:
        value: "90"
{{- end }}

Operational Takeaway

It is not enough to say that KEDA enables replicas to reach zero. To operate it safely, one needs to know which scaler detects events, when HPA takes over again, and where minReplicas is enforced. Cost optimization has to be understood together with runtime semantics.

Because only development/release pods should scale to zero, it is convenient to declare the environment in templated values and branch from there.

The familiar HPA values appear here with slightly different names: minReplicaCount and maxReplicaCount. cooldownPeriod defines how long KEDA waits before scaling down after activity drops, giving the workload time to stabilize after sudden traffic or metric changes.

The trigger is the core of KEDA. KEDA supports many trigger types: CloudWatch log or metric signals, Prometheus custom metrics, cron schedules, queue depth, and more.

Since the initial goal was to scale the pod count to 0 at a specific time, I created a cron type trigger, set the pod count to 1 from 08:30 to 19:30, and did not declare it for other times.

I also added a memory type trigger. When multiple triggers exist like this, replicas are created according to the maximum value from each trigger calculation. (https://github.com/kedacore/keda/issues/5078)

Running Kafka and Debezium on Kubernetes with Strimzi

ystc1247@gmail.com — Wed, 24 Jul 2024 09:57:27 GMT

Where the Problem Started

The hard part of introducing CDC is not only starting a connector. It is deciding how database changes become an event pipeline, how Kafka and Debezium should run inside the application cluster, and how to recover when offsets break. This post records the Strimzi setup and operational points I ran into while running Kafka and Debezium on Kubernetes.

Debezium is a common CDC(Change Data Capture) tool.

The reason for using CDC is to quickly detect database changes, emit events, and run work from them.

To briefly explain the use case that led to introducing it,

A POST API request enters the application -> information is stored in database A -> an external API is called -> if the external call succeeds, information is stored in database B -> follow-up service methods run

In that flow, database A could be updated even when the external API failed, so the whole process was not protected by one transaction boundary.

Adding @Transactional and rolling back on failure sounds like the obvious answer, but the external API call was being sent asynchronously, so that did not solve the consistency problem.

The target design was to call the external API first, then persist a detectable row only after success. Debezium could then capture that change, emit an event, and trigger the remaining workflow.

Several pub/sub systems or message queues can handle that event flow. This post focuses on Kafka, and more specifically on running Kafka and Debezium as pods inside the Kubernetes cluster instead of operating them as separate instances.

Strimzi is one practical way to deploy Kafka on Kubernetes. A Kafka cluster, Kafka Connect, and connector resources can be managed with a small set of YAML files, so I will organize the setup around that path.

Define the namespace first.

Implementation Path

kubectl create ns strimzi-debezium

Additionally, if you are using helm:

helm install strimzi-operator strimzi/strimzi-kafka-operator -n strimzi-debezium

Create the Kafka cluster resource next.

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: my-cluster
  namespace: strimzi-debezium
spec:
  kafka:
    version: 3.7.0
    replicas: 3
    listeners:
      - name: plain
        port: 9092
        type: internal
        tls: false
      - name: tls
        port: 9093
        type: internal
        tls: true
    storage:
      type: jbod
      volumes:
        - id: 0
          type: persistent-claim
          size: 10Gi
          deleteClaim: false
    config:
      offsets.topic.replication.factor: 3
      transaction.state.log.replication.factor: 3
      transaction.state.log.min.isr: 2
      default.replication.factor: 3
      min.insync.replicas: 2
      ssl.endpoint.identification.algorithm: ""
      log4j.logger.org.apache.kafka: DEBUG
      log4j.logger.kafka: DEBUG
    resources:
      requests:
        memory: 1Gi
        cpu: "0.5"
      limits:
        memory: 2Gi
        cpu: "1"
    template:
      kafkaContainer:
        env:
          - name: KAFKA_HEAP_OPTS
            value: "-Xms1G -Xmx1G"
  zookeeper:
    replicas: 3
    storage:
      type: persistent-claim
      size: 10Gi
      deleteClaim: false
    resources:
      requests:
        memory: 512Mi
        cpu: "0.5"
      limits:
        memory: 1Gi
        cpu: "1"
    config:
      autopurge.snapRetainCount: 3
      autopurge.purgeInterval: 1
      clientCnxnSocket: org.apache.zookeeper.ClientCnxnSocketNetty
      serverCnxnFactory: org.apache.zookeeper.server.NettyServerCnxnFactory
      quorum.ssl.enabled: true
      ssl.quorum.protocol: TLSv1.2
      tickTime: 2000
      initLimit: 15
      syncLimit: 10
    logging:
      type: inline
      loggers:
        zookeeper.root.logger: "INFO,CONSOLE"
    template:
      pod:
        securityContext:
          runAsUser: 1001
          fsGroup: 1001
  entityOperator:
    topicOperator: {}
    userOperator: {}
  clusterCa:
    generateCertificateAuthority: true
  clientsCa:
    generateCertificateAuthority: true

Here, Kafka and ZooKeeper are each created with three replicas, and CPU/memory are allocated fairly generously. The replica count should follow the use case, but these resource values were close to the minimum I felt comfortable with for reliable database-change delivery.

For apiVersion, you can use versions like v1alpha in addition to v1beta2, but some conf properties are missing depending on the version, so do not mix them casually.

Kafka is set to 3.7.0, which was the latest version at the time. When I pinned the Strimzi version and brought the stack up together, Kafka versions around 3.2 or lower were not compatible.

I exposed two Kafka listeners: one plaintext port and one TLS port. While experimenting with the settings, I saw connection failures when a client tried to connect without the expected TLS certificate, so this part is worth setting carefully from the beginning.

ssl.endpoint.identification.algorithm: ""

This part disables hostname verification for SSL connection, which is not recommended in a prod environment.

template:
  kafkaContainer:
    env:
      - name: KAFKA_HEAP_OPTS
        value: "-Xms1G -Xmx1G"

This config sets the Kafka heap allocation. Without it, the pod can run into memory pressure during startup.

clusterCa:
  generateCertificateAuthority: true
clientsCa:
  generateCertificateAuthority: true

This part is optional. You can also create and use the CA and pem yourself.

openssl req -x509 -newkey rsa:4096 -keyout key.pem -out cert.pem -days 365 -nodes -subj "/CN=kafka-connect"
kubectl create secret generic kafka-connect-tls --from-file=tls.key=key.pem --from-file=tls.crt=cert.pem -n strimzi-debezium

This is the manual certificate creation path. The remaining details follow the standard Kubernetes secret and Strimzi certificate flow.

Next is kafka-connect.yaml.

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnect
metadata:
  name: my-connect-cluster
  namespace: strimzi-debezium
  annotations:
    strimzi.io/use-connector-resources: "true"
spec:
  image: ystc1247/kafka-strimzi:0.0.3
  replicas: 1
  bootstrapServers: my-cluster-kafka-bootstrap:9093
  tls:
    trustedCertificates:
      - secretName: my-cluster-cluster-ca-cert
        certificate: ca.crt
  config:
    offset.storage.topic: connect-cluster-offsets
    config.storage.topic: connect-cluster-configs
    status.storage.topic: connect-cluster-status
    config.storage.replication.factor: 1
    offset.storage.replication.factor: 1
    status.storage.replication.factor: 1
    config.providers: file
    config.providers.file.class: org.apache.kafka.common.config.provider.FileConfigProvider

annotations:
  strimzi.io/use-connector-resources: "true"

This annotation is important because it allows connector resources to be managed properly. For the image, I used one I built and pushed to DockerHub with Strimzi and the MySQL connector included. A different image is fine as long as it contains the required connector.

I am attaching the Dockerfile.

FROM strimzi/kafka:0.20.1-kafka-2.5.0
USER root:root
RUN mkdir -p /opt/kafka/plugins/debezium
COPY ./debezium-connector-mysql/ /opt/kafka/plugins/debezium/
USER 1001

The MySQL connector I used:

curl -L https://repo1.maven.org/maven2/io/debezium/debezium-connector-mysql/2.7.0.Final/debezium-connector-mysql-2.7.0.Final-plugin.tar.gz | tar -xz

bootstrapServers: my-cluster-kafka-bootstrap:9093

You can use 9092, which was configured without tls, for this part, but authentication issues may occur.

Next is kafka-connector.yaml.

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnector
metadata:
  name: inventory-connector
  namespace: strimzi-debezium
  labels:
    strimzi.io/cluster: my-connect-cluster
spec:
  class: io.debezium.connector.mysql.MySqlConnector
  tasksMax: 1
  config:
    topic.prefix: ${원하는 topic prefix}
    database.hostname: dbhost
    database.port: dbport
    database.user: 유저
    database.password: 비번
    database.dbname: db이름
    database.server.id: 185239 (고유한 id를 사용)
    database.server.name: db server
    table.include.list: 중요! 변화를 감지하고자 하는 table
    schema.history.internal.kafka.bootstrap.servers: "my-cluster-kafka-bootstrap:9092"
    schema.history.internal.kafka.topic: "schema-changes.inventory"
    database.history.kafka.bootstrap.servers: "my-cluster-kafka-bootstrap:9092"
    database.history.kafka.topic: "database-changes.inventory"
    include.schema.changes: "true"
    database.history.kafka.reset.offset: "true"
    snapshot.mode: "when_needed"

snapshot.mode is important because Debezium provides many different snapshot behaviors. Choose the mode that matches the recovery behavior you need. The detailed options are in the Debezium MySQL connector documentation.

If pod creation and deletion repeat and Debezium misses an offset, a mismatch can occur between the current snapshot and the available snapshot.

│ INFO: Connected to {} at mysql-bin-changelog.400734/891 (sid:185054, cid:15583019)                                                                          │
│ 2024-07-24 06:42:19,297 INFO [inventory-connector|task-0] Connected to binlog at dev.data.yeogiya.io:3306, starting at BinlogOffsetContext{sourceInfoSchema=Schema{io.debezium.co │
│ 2024-07-24 06:42:19,297 INFO [inventory-connector|task-0] Waiting for keepalive thread to start (io.debezium.connector.binlog.BinlogStreamingChangeEventSource) [debezium-mysqlco │
│ 2024-07-24 06:42:19,297 INFO [inventory-connector|task-0] Creating thread debezium-mysqlconnector-{}-binlog-client (io.debezium.util.Threads) [blc-{} │
│ 2024-07-24 06:42:19,297 ERROR [inventory-connector|task-0] Error during binlog processing. Last offset stored = null, binlog reader near position = mysql-bin-changelog.400734/89 │
│ 2024-07-24 06:42:19,298 ERROR [inventory-connector|task-0] Producer failure (io.debezium.pipeline.ErrorHandler) [blc-{}:3306]                                    │
│ io.debezium.DebeziumException: Could not find first log file name in binary log index file Error code: 1236; SQLSTATE: HY000.

This error usually means Debezium is trying to read a binary log file that no longer exists, such as mysql-bin-changelog.400734. A MySQL version change can also create related offset/snapshot mismatches.

For the missing-binlog case, start here.

SHOW VARIABLES LIKE 'log_bin';

Check that binary logging is enabled.

mysql> SHOW BINARY LOGS;
+----------------------------+-----------+-----------+
| Log_name                   | File_size | Encrypted |
+----------------------------+-----------+-----------+
| mysql-bin-changelog.400803 |      2436 | No        |
| mysql-bin-changelog.400804 |      2436 | No        |
| mysql-bin-changelog.400805 |       891 | No        |
+----------------------------+-----------+-----------+

Compare the version Debezium is currently reading with the current latest version.

Additionally, there could be problems with global variables like expire_logs_days and binlog_expire_logs_seconds, but if you have not changed them and they are still at the default, they probably are not the issue.

Then force Kafka/Debezium to move the consumer offset to a valid position.

apiVersion: v1
kind: Pod
metadata:
  name: kafka-client
  namespace: default
spec:
  containers:
    - name: kafka
      image: confluentinc/cp-kafka:latest
      command:
        - sleep
        - "3600"

Bring up this helper pod first. If the Kafka shell tools are already available elsewhere, this step can be skipped.

After that:

kubectl exec -it kafka-client -n default -- /bin/sh
cd /usr/bin
./kafka-consumer-groups --bootstrap-server my-cluster-kafka-bootstrap.strimzi-debezium.svc.cluster.local:9092 --group inventory-connector --reset-offsets --to-latest --all-topics --execute

This resets the connector group’s offsets for all topics to the latest position. Any database changes between the old offset and the new one will not be captured, so this is a recovery operation that must be used carefully.

Here are the results.

How I Verified It

kubectl exec -it my-cluster-kafka-0 -n strimzi-debezium -- bin/kafka-console-consumer.sh --bootstrap-server my-cluster-kafka-bootstrap:9092 --topic {topic-name} --from-beginning

Operational Takeaway

Strimzi lifts Kafka operation into Kubernetes objects, but it does not remove operational responsibility. TLS, heap sizing, connector images, topic replication, binlog retention, and offset reset all connect directly to real failure modes. Once a CDC pipeline is attached, it becomes part of the data flow, so failure and recovery need to be designed from the beginning.

Running this shows the messages Kafka is producing.

Calling APIs from Database Triggers: Usage and Risk

ystc1247@gmail.com — Sat, 01 Jun 2024 22:59:15 GMT

Where the Problem Started

Database triggers are powerful, but dangerous. They run outside the application layer, which makes flow harder to see, and calling an external API from that point complicates failure propagation and transaction boundaries. This post records how the approach works and why it should be handled carefully.

In databases, triggers are often used for business needs such as maintaining data integrity and validation. When an operation such as INSERT or DELETE happens on a specific table, row, or column, the trigger is bound to the same transaction and performs additional work on top of the original operation.

But you can also attach an API call, something normally handled in the server application layer, to that flow.

The first thing to emphasize upfront is that this is absolutely not recommended. As mentioned earlier, the natural procedure is to call APIs from the controller and service layers alongside database operations executed by the application. There is no reason to create unpredictable database load by calling APIs from sensitive database work that needs to preserve ACID properties.

With that warning in mind, this post looks at how an API call can be attached to a database trigger, and what kind of load and coupling that creates. This belongs in the category of last-resort engineering: useful only when business constraints leave no cleaner option.

For the database, I used plain SQL Server 2019, and for analysis I used Datagrip.

Start with a table that has only a simple name column.

Implementation Path

CREATE TABLE BasicTable (
    id INT IDENTITY(1,1) PRIMARY KEY,
    name NVARCHAR(100) NOT NULL
);
GO

For the HTTP call, I used a public API that predicts age from a name parameter. When I passed in my own name, it returned a predicted age of 66.

How I Verified It

SQL Server requires an additional option before a trigger can call an external API directly:

EXEC sp_configure 'show advanced options', 1;
RECONFIGURE;
EXEC sp_configure 'Ole Automation Procedures', 1;
RECONFIGURE;

This option is for using OLE Automation Object inside Microsoft’s COM(Component Object Model), which is used only in MSSQL. Specifically, because we will use MSXML2.ServerXMLHTTP among the COM components for the current HTTP request, enabling this option is required. If you are wondering what OLE is, read https://en.wikipedia.org/wiki/Object_Linking_and_Embedding.

The rough procedures used to make the HTTP call and receive the response are as follows.

sp_OACreate: Creates an instance of an OLE Automation object.
sp_OADestroy: Destroys an instance of an OLE Automation object.
sp_OAGetProperty: Gets a property value of an OLE Automation object.
sp_OASetProperty: Sets a property value of an OLE Automation object.
sp_OAMethod: Calls a method of an OLE Automation object.

Create a separate table to log what the trigger does.

CREATE TABLE ApiResponseLog (
    id INT IDENTITY(1,1) PRIMARY KEY,
    operation_type NVARCHAR(10),
    name NVARCHAR(100),
    response NVARCHAR(1000),
    timestamp DATETIME DEFAULT CURRENT_TIMESTAMP
);
GO

I will insert the operation_type (UPDATE, SELECT, INSERT, DELETE), the name to be passed to the API, and the corresponding response.

But for performance testing, I also wanted to record the API call time, DB insert time, and so on.

ALTER TABLE ApiResponseLog
ADD api_call_start_time DATETIME2,
    api_call_end_time DATETIME2,
    api_call_duration_ms INT,
    insert_start_time DATETIME2,
    insert_end_time DATETIME2,
    insert_duration_ms INT;
GO

The trigger itself looks like this.

CREATE TRIGGER trgAfterAllOperations ON BasicTable
AFTER INSERT, UPDATE, DELETE
AS
BEGIN
    DECLARE @name NVARCHAR(100);
    DECLARE @url NVARCHAR(200);
    DECLARE @response NVARCHAR(1000);
    DECLARE @object INT;
    DECLARE @status INT;
    DECLARE @operation_type NVARCHAR(10);
    DECLARE @api_call_start_time DATETIME2;
    DECLARE @api_call_end_time DATETIME2;
    DECLARE @insert_start_time DATETIME2;
    DECLARE @insert_end_time DATETIME2;

    -- operation type 결정
    -- update 의 경우 delete/insert pseudotable 모두에 행이 생성됨
    IF EXISTS (SELECT * FROM inserted) AND EXISTS (SELECT * FROM deleted)
        SET @operation_type = 'UPDATE';
    ELSE IF EXISTS (SELECT * FROM inserted)
        SET @operation_type = 'INSERT';
    ELSE IF EXISTS (SELECT * FROM deleted)
        SET @operation_type = 'DELETE';

    IF @operation_type = 'INSERT' OR @operation_type = 'UPDATE'
    BEGIN
        SELECT TOP 1 @name = name FROM inserted;
    END
    ELSE IF @operation_type = 'DELETE'
    BEGIN
        SELECT TOP 1 @name = name FROM deleted;
    END

    -- API URL 지정
    SET @url = 'https://api.agify.io?name=' + @name;

    -- API call 시작시간 기록
    SET @api_call_start_time = SYSDATETIME();

    -- API call!!
    EXEC @status = sp_OACreate 'MSXML2.ServerXMLHTTP', @object OUT;
    IF @status = 0 EXEC @status = sp_OAMethod @object, 'open', NULL, 'GET', @url, 'false';
    IF @status = 0 EXEC @status = sp_OAMethod @object, 'send';
    IF @status = 0 EXEC @status = sp_OAGetProperty @object, 'responseText', @response OUT;
    IF @status = 0 EXEC sp_OADestroy @object;

    -- API call 끝난시간 기록
    SET @api_call_end_time = SYSDATETIME();

    -- insert 시작시간 기록
    SET @insert_start_time = SYSDATETIME();

    -- 트리거 로깅
    INSERT INTO ApiResponseLog (
        operation_type,
        name,
        response,
        api_call_start_time,
        api_call_end_time,
        api_call_duration_ms,
        insert_start_time,
        insert_end_time,
        insert_duration_ms
    )
    VALUES (
        @operation_type,
        @name,
        @response,
        @api_call_start_time,
        @api_call_end_time,
        DATEDIFF(MILLISECOND, @api_call_start_time, @api_call_end_time),
        @insert_start_time,
        SYSDATETIME(), -- insert 끝난시간 기록
        DATEDIFF(MILLISECOND, @insert_start_time, SYSDATETIME())
    );
END;
GO

The snippet is long, but the important part is that the HTTP request is executed inside the trigger’s transaction path.

After creating the trigger, modify BasicTable.

Execute the change.

The result confirms that the API call ran and that the response was saved.

The key column is api_call_duration_ms. It records 222ms, which is ordinary for a server-to-server API call. The problem is that the time is now inside a database transaction. In contrast, insert_duration_ms is recorded at a sub-millisecond scale.

A single operation that should remain integral now has meaningful time consumption because the trigger makes an API call.

Then I checked the database load.

-- Insert 20 records
INSERT INTO BasicTable (name) VALUES ('Name1');
INSERT INTO BasicTable (name) VALUES ('Name2');
INSERT INTO BasicTable (name) VALUES ('Name3');
INSERT INTO BasicTable (name) VALUES ('Name4');
INSERT INTO BasicTable (name) VALUES ('Name5');
INSERT INTO BasicTable (name) VALUES ('Name6');
INSERT INTO BasicTable (name) VALUES ('Name7');
INSERT INTO BasicTable (name) VALUES ('Name8');
INSERT INTO BasicTable (name) VALUES ('Name9');
INSERT INTO BasicTable (name) VALUES ('Name10');
INSERT INTO BasicTable (name) VALUES ('Name11');
INSERT INTO BasicTable (name) VALUES ('Name12');
INSERT INTO BasicTable (name) VALUES ('Name13');
INSERT INTO BasicTable (name) VALUES ('Name14');
INSERT INTO BasicTable (name) VALUES ('Name15');
INSERT INTO BasicTable (name) VALUES ('Name16');
INSERT INTO BasicTable (name) VALUES ('Name17');
INSERT INTO BasicTable (name) VALUES ('Name18');
INSERT INTO BasicTable (name) VALUES ('Name19');
INSERT INTO BasicTable (name) VALUES ('Name20');

-- Update 15 records
UPDATE BasicTable SET name = 'UpdatedName1' WHERE id = 1;
UPDATE BasicTable SET name = 'UpdatedName2' WHERE id = 2;
UPDATE BasicTable SET name = 'UpdatedName3' WHERE id = 3;
UPDATE BasicTable SET name = 'UpdatedName4' WHERE id = 4;
UPDATE BasicTable SET name = 'UpdatedName5' WHERE id = 5;
UPDATE BasicTable SET name = 'UpdatedName6' WHERE id = 6;
UPDATE BasicTable SET name = 'UpdatedName7' WHERE id = 7;
UPDATE BasicTable SET name = 'UpdatedName8' WHERE id = 8;
UPDATE BasicTable SET name = 'UpdatedName9' WHERE id = 9;
UPDATE BasicTable SET name = 'UpdatedName10' WHERE id = 10;
UPDATE BasicTable SET name = 'UpdatedName11' WHERE id = 11;
UPDATE BasicTable SET name = 'UpdatedName12' WHERE id = 12;
UPDATE BasicTable SET name = 'UpdatedName13' WHERE id = 13;
UPDATE BasicTable SET name = 'UpdatedName14' WHERE id = 14;
UPDATE BasicTable SET name = 'UpdatedName15' WHERE id = 15;

-- Delete 15 records
DELETE FROM BasicTable WHERE id = 1;
DELETE FROM BasicTable WHERE id = 2;
DELETE FROM BasicTable WHERE id = 3;
DELETE FROM BasicTable WHERE id = 4;
DELETE FROM BasicTable WHERE id = 5;
DELETE FROM BasicTable WHERE id = 6;
DELETE FROM BasicTable WHERE id = 7;
DELETE FROM BasicTable WHERE id = 8;
DELETE FROM BasicTable WHERE id = 9;
DELETE FROM BasicTable WHERE id = 10;
DELETE FROM BasicTable WHERE id = 11;
DELETE FROM BasicTable WHERE id = 12;
DELETE FROM BasicTable WHERE id = 13;
DELETE FROM BasicTable WHERE id = 14;
DELETE FROM BasicTable WHERE id = 15;

First, remove the trigger and run the test without it.

[2024-06-02 07:23:53] 50 rows affected in 26 ms

The run completed 20 inserts, 15 updates, and 15 deletes in about 0.02 seconds. Looking more closely:

For insert, the plan shows the cost and actual execution time of the clustered index insert.

Then enable the trigger and run the benchmark again.

        INSERT INTO BasicTable (name) VALUES ('Name28');
        INSERT INTO BasicTable (name) VALUES ('Name29');
        INSERT INTO BasicTable (name) VALUES ('Name30');
        INSERT INTO BasicTable (name) VALUES ('Name31');
        INSERT INTO BasicTable (name) VALUES ('Name32');
        INSERT INTO BasicTable (name) VALUES ('Name33');
        INSERT INTO BasicTable (name) VALUES ('Name34');
        INSERT INTO BasicTable (name) VALUES ('Name35');
        INSERT INTO BasicTable (name) VALUES ('Name36');
        INSERT INTO BasicTable (name) VALUES ('Name37');
        INSERT INTO BasicTable (name) VALUES ('Name38');
        INSERT INTO BasicTable (name) VALUES ('Name39');
        INSERT INTO BasicTable (name) VALUES ('Name40');
        INSERT INTO BasicTable (name) VALUES ('Name41');
        INSERT INTO BasicTable (name) VALUES ('Name42');
        INSERT INTO BasicTable (name) VALUES ('Name43');
        INSERT INTO BasicTable (name) VALUES ('Name44');
        INSERT INTO BasicTable (name) VALUES ('Name45');
        INSERT INTO BasicTable (name) VALUES ('Name46');
        INSERT INTO BasicTable (name) VALUES ('Name47');
[2024-06-02 07:33:20] 20 rows affected in 4 s 894 ms
master> -- Update 15 records
        UPDATE BasicTable SET name = 'UpdatedName28' WHERE id = 28;
        UPDATE BasicTable SET name = 'UpdatedName29' WHERE id = 29;
        UPDATE BasicTable SET name = 'UpdatedName30' WHERE id = 30;
        UPDATE BasicTable SET name = 'UpdatedName31' WHERE id = 31;
        UPDATE BasicTable SET name = 'UpdatedName32' WHERE id = 32;
        UPDATE BasicTable SET name = 'UpdatedName33' WHERE id = 33;
        UPDATE BasicTable SET name = 'UpdatedName34' WHERE id = 34;
        UPDATE BasicTable SET name = 'UpdatedName35' WHERE id = 35;
        UPDATE BasicTable SET name = 'UpdatedName36' WHERE id = 36;
        UPDATE BasicTable SET name = 'UpdatedName37' WHERE id = 37;
        UPDATE BasicTable SET name = 'UpdatedName38' WHERE id = 38;
        UPDATE BasicTable SET name = 'UpdatedName39' WHERE id = 39;
        UPDATE BasicTable SET name = 'UpdatedName40' WHERE id = 40;
        UPDATE BasicTable SET name = 'UpdatedName41' WHERE id = 41;
        UPDATE BasicTable SET name = 'UpdatedName42' WHERE id = 42;
[2024-06-02 07:33:23] 15 rows affected in 3 s 642 ms
master> -- Delete 15 records
        DELETE FROM BasicTable WHERE id = 28;
        DELETE FROM BasicTable WHERE id = 29;
        DELETE FROM BasicTable WHERE id = 30;
        DELETE FROM BasicTable WHERE id = 31;
        DELETE FROM BasicTable WHERE id = 32;
        DELETE FROM BasicTable WHERE id = 33;
        DELETE FROM BasicTable WHERE id = 34;
        DELETE FROM BasicTable WHERE id = 35;
        DELETE FROM BasicTable WHERE id = 36;
        DELETE FROM BasicTable WHERE id = 37;
        DELETE FROM BasicTable WHERE id = 38;
        DELETE FROM BasicTable WHERE id = 39;
        DELETE FROM BasicTable WHERE id = 40;
        DELETE FROM BasicTable WHERE id = 41;
        DELETE FROM BasicTable WHERE id = 42;
[2024-06-02 07:33:26] 15 rows affected in 3 s 91 ms

The trigger-backed run took 4.892s for 20 INSERTs, 3.642s for 15 UPDATEs, and 3.091s for 15 DELETEs, for a total of around 11 seconds. That result is exactly what should be expected once API calls are tied to trigger execution.

Then were the API calls and inserts into the ApiResponseLog table performed correctly?

Unfortunately, no.

After going into SSMS, I observed that the SQL trigger was consuming CPU time. As this accumulates, concurrency seems to spike, and some objects apparently fail to be delivered properly.

Not only that, but if the server does not deliver the API response in a normal amount of time, the transaction can also blow up after reaching the Trigger execution limit. This can create bottlenecks or locks, and even if the transaction passes through with a silent throw, the desired API response was never received, so the logic still failed to achieve its goal.

In conclusion, do not ruin the operation that should be the lightest by attaching API calls to triggers in a database where you have worked so hard to reduce latency. Even if the API call is light enough to ignore, the server making the API call and the server running the database are different, so a dependency is inevitable.

API calls inside triggers also damage the atomicity of the transaction. External calls cannot become part of database transaction management.

Also, if an error occurs, debugging hell will open up. You might not be able to debug it at all.

Unless it is really, really, really necessary for the business logic, do not use it. Just do not use it.

Design Conclusion

Trigger-based API calls can look like a fast workaround. But once ownership moves into the database, observability, retry behavior, rollback semantics, and testability all become harder. It can be used, but only when there is a clear reason not to use application events or an outbox-style design instead.

Understanding Salting and Hashing as a Security Boundary

ystc1247@gmail.com — Fri, 03 May 2024 15:25:19 GMT

Where the Problem Started

Password storage is not simply encrypting a string. Even if two users choose the same password, the stored values should differ, and a database leak should not make the original password recoverable. This post organizes hashing and salting as a security boundary rather than a vocabulary exercise.

When building a user information system, the most important element is protecting user passwords.

The technique used here is called salting, and its etymology is supposedly https://stackoverflow.com/questions/244903/why-is-a-password-salt-called-a-salt.

There is no especially satisfying answer, but the origins of common placeholder terms like foo and bar are often more arbitrary than they look.

Start with hashing. In short, hashing is a one-way process that transforms a string into another string. One-way means the hashed value cannot be restored to the original input.

By contrast, in two-way encryption, there is a method shared by both encryption and decryption, such as a key, that can infer the original string and the changed string.

Various algorithms are used for hashing. There are Phpass, libsodium, sha256, sha512, and so on, but look up which algorithms are safe and officially recognized.

MD5, SHA1, SHA2, and similar algorithms are based on the Merkle-Damgard construction and can be exposed to length extension attacks, so they should not be used for password storage.

The password authentication flow is easier to reason about step by step.

The user creates an account.
The password is stored in the DB after hashing.
When the user tries to log in, the password entered by the user is encrypted using the hashing algorithm and compared with the password in the DB.
If the hashes match, authentication passes.

In most login systems, when step 4 fails, the service does not reveal whether the ID or password was wrong. That is not just a frustrating UX choice; it is a small security measure that makes username-aware brute force attacks harder.

Is security that has gone through hashing safe? Of course not.

If the hashing algorithm used by a service is identified, then after the user database is breached, inferring the password is not necessarily difficult.

Even if the hashing algorithm is not known at first, an attacker can still brute force likely algorithms and candidate passwords. With enough preparation, they may also use lookup tables or rainbow tables.

https://en.wikipedia.org/wiki/Rainbow_table#:~:text=A%20rainbow%20table%20is%20a,form%2C%20but%20as%20hash%20values.

How I Verified It

In short, it is a key-value style table that maps every string hashed by a hashing algorithm back to its original string, so naturally the table size is huge.

This is because complex hashing algorithms do not replace each character with some other string. They perform a complex form of encryption intertwined across the characters, so the encryption of “abcdef” and “abcdee” is completely different.

That is where salt appears.

The operating principle of lookup tables and rainbow tables is that one string is replaced with exactly one other string.

For example, suppose there is a password called “password.”

When one service encrypts “password,” no matter how complex the hashing algorithm is, every encrypted “password” corresponds to “password.” That is why a special string is appended to “password” before hashing, and that string is called salt.

There are several caveats when using salt.

First is reusing salt. The moment you harm salt uniqueness, there is little difference from the security state before adding salt to the string. After the attacker figures out what the salt is, they can simply append the salt to the password and use a reverse lookup table.

Salt must be different for each user. You could use a language-provided random function to generate each character, but use a safer CSPRNG instead.

https://en.wikipedia.org/wiki/Cryptographically_secure_pseudorandom_number_generator I do not really know the principle, but it is a cryptographically safer random function. CSPRNG libraries also exist in most languages. For example, Java’s java.security.SecureRandom and Python’s secrets.

Next is a small salt string size. For example, if a salt uses only 4 ASCII characters, 95^4 = 81450625 salts can be generated. It may look like a lot, but given the speed and storage capacity of modern computers, it is nowhere near enough.

Experts say the salt should be at least as large as the hashed string after salt is added. For example, a string hashed with SHA256 has a size of 256bits == 32bytes == 64 characters. If you use SHA512, the minimum salt size is 128 char.

Then hash the salted string.

There are fast and safe algorithms, or algorithms once considered safe, such as SHA256, SHA512, and WHIRLPOOL. There are also algorithms such as PBKDF2, bcrypt, and scrypt, which are a bit slower but aligned with modern security.

It may be overengineering, so choose appropriately based on the scale and capability of the service.

https://cs.opensource.google/go/x/crypto/+/refs/tags/v0.22.0:scrypt/scrypt.go This is the Go library source code for scrypt, and seeing how extremely complex it is, it certainly looks safe.

The following is a short Python function from creating salt to password hashing. As expected, short functions? Python.

Implementation Path

class EncryptPassword:

    @staticmethod

    def salt_password(password):

        salt = os.urandom(64) # salt 생성

        salt_hex = salt.hex() # 저장을 위한 salt > hex 로 변환

		# PBKDF2 (SHA-512 의 100,000 반복) 으로 해싱
        dk = hashlib.pbkdf2_hmac('sha512', password.encode(), salt, 100000, dklen=64)

        hashed_password_hex = dk.hex() # 저장을 위한 비밀번호 > hex 로 변환

        return hashed_password_hex, salt_hex

Security Takeaway

Knowing the words hash and salt is different from designing safe password storage. Algorithm choice, salt generation, work factor, verification flow, and migration strategy all matter. Security features should be judged not just by whether they work, but by what information an attacker can still obtain.

Propagating TraceId Context Across Async Work

ystc1247@gmail.com — Sat, 09 Mar 2024 11:07:44 GMT

Where the Problem Started

The frustrating part of tracing is not having too few logs; it is having many logs that cannot be tied back to one request. Async work changes threads, and context can disappear at that boundary. When traceId is lost, incident analysis becomes much harder. This post records how I restored traceId propagation in a Spring Boot async flow.

In a Spring Boot service, when server logic fails, traceId is often the fastest way to find the related logs. For example, you can search for "${traceId}" in an AWS CloudWatch log group.

That makes consistent traceId logging important. In single-threaded synchronous work, it is usually straightforward.

But when you want asynchronous work, the logic may continue on another thread while it runs. In those cases, the traceId from the original thread (parent thread) can fail to propagate to the worker thread (child thread). This needs to propagate so that, if a problem occurs in the worker thread, you can follow the traceId and investigate the error efficiently.

In Spring Boot 2.x, Spring Sleuth handled traceId propagation smoothly. After moving to Spring Boot 3.x, Sleuth was deprecated and the tracing path moved toward Micrometer. Micrometer is commonly used for metrics and tracing integration, but the async traceId propagation I relied on before no longer happened automatically. This became one of several migration issues that appeared while moving existing project logic to Spring Boot 3.x.

Implementation Path

implementation("io.micrometer", "micrometer-observation")
implementation("io.micrometer", "micrometer-tracing")
implementation("io.micrometer", "micrometer-tracing-bridge-brave")
implementation("io.micrometer", "context-propagation")

Add the tracing and context propagation dependencies. The context propagation project is available here: https://github.com/micrometer-metrics/context-propagation.

@Configuration
class AsynchronousConfiguration: AsyncConfigurer {
    @Bean
    fun threadPoolTaskExecutor(): ThreadPoolTaskExecutor = ThreadPoolTaskExecutor().apply {
        corePoolSize = CORE_POOL_SIZE
        maxPoolSize = MAX_POOL_SIZE
        queueCapacity = QUEUE_CAPACITY
        setThreadNamePrefix("async-")
    }

    @Bean(ASYNC_TASK_EXECUTOR_BEAN_NAME)
    fun asyncTaskExecutor(delegate: ThreadPoolTaskExecutor): DelegatingSecurityContextAsyncTaskExecutor {
        return DelegatingSecurityContextAsyncTaskExecutor(delegate)
    }

    companion object {
        const val ASYNC_TASK_EXECUTOR_BEAN_NAME = "asyncTaskExecutor"
        const val CORE_POOL_SIZE = 10
        const val MAX_POOL_SIZE = 50
        const val QUEUE_CAPACITY = 50
    }
}

Assume a standard asynchronous configuration class. Declaring threadPoolTaskExecutor as a bean lets us configure thread pool settings such as core size, max size, and queue capacity.

After that, asynchronous methods can refer to this executor by name.

I used a small service method to observe how the async thread behaves.

One async method calls another, creating a parent/child-style async flow that makes trace propagation easier to inspect.

@Async(AsynchronousConfiguration.ASYNC_TASK_EXECUTOR_BEAN_NAME)
    fun startParentTask(traceId: String) {

        val parentThreadName = Thread.currentThread().name
        log.info("Parent task started with traceId: $traceId on thread: $parentThreadName")

        childService.startChildTask(traceId)

        Thread.sleep(1000)

        log.info("Parent task completed with traceId: $traceId on thread: $parentThreadName")
    }

First is the parent method. For thread tracing in the logs, it receives a randomly generated traceId as an argument along with the current thread name, then logs with slf4j.

I add a sleep to the Thread to artificially force the logic to take some time while it runs.

@Async(AsynchronousConfiguration.ASYNC_TASK_EXECUTOR_BEAN_NAME)
    fun startChildTask(traceId: String) {
        val childThreadName = Thread.currentThread().name
        val traceId = Thread.currentThread().stackTrace
        log.info("Child task started with traceId: $traceId on thread: $childThreadName")

        Thread.sleep(1000)

        log.info("Child task completed with traceId: $traceId on thread: $childThreadName")
    }

The child method follows the same shape.

@GetMapping("/test-trace")
    fun testTraceId(): String {
        val traceId = java.util.UUID.randomUUID().toString()
        log.info("Starting asynchronous job with traceId: $traceId")
        parentService.startParentTask(traceId)
        return "Asynchronous job started with traceId: $traceId"
    }

This controller is only for testing. It creates a random traceId, logs the request entry point, and starts the async flow so multiple requests can be sent in quick succession.

logging:
  pattern:
    dateformat: "yyyy-MM-dd HH:mm:ss.SSS"
    level: "%5p [${spring.application.name:},%X{traceId:-},%X{spanId:-}]"

Configure the log pattern so traceId and spanId are visible in every line.

Create a propagation bean through TaskDecorator.

 @Bean
    fun taskDecorator(): TaskDecorator {
        return TaskDecorator { runnable: Runnable? ->
            ContextSnapshot.captureAll(*arrayOfNulls(0)).wrap(
                runnable!!,
            )
        }
    }

https://blog.gangnamunni.com/post/mdc-context-task-decorator/ This is a somewhat old post from Gangnam Unni’s tech blog, but it solves an issue in a similar situation.

https://docs.spring.io/spring-framework/docs/current/javadoc-api/org/springframework/core/task/TaskDecorator.html This is the official TaskDecorator documentation. As the name suggests, it wraps a task before execution, which makes it a useful hook for monitoring, metrics, and context propagation.

After creating the decorator, attach it to the threadPoolTaskExecutor.

@Bean
    fun threadPoolTaskExecutor(): ThreadPoolTaskExecutor = ThreadPoolTaskExecutor().apply {
        corePoolSize = CORE_POOL_SIZE
        maxPoolSize = MAX_POOL_SIZE
        queueCapacity = QUEUE_CAPACITY
        setThreadNamePrefix("async-")
        setTaskDecorator(taskDecorator()) << 이거 추가
    }

Send several requests and check whether the logs keep the same trace context across async execution.

2024-03-09 19:51:03.182  INFO [,65ec3f17e6d38686445bf9e99014b824,445bf9e99014b824] 25486 --- [nio-8080-exec-1] kr.projects.trace.TraceController        : Starting asynchronous job with traceId: 3f0f0f1e-c0b3-4cef-99e3-7cbfd809e164
2024-03-09 19:51:03.183  INFO [,65ec3f17e6d38686445bf9e99014b824,445bf9e99014b824] 25486 --- [nio-8080-exec-1] kr.projects.trace.ParentService          : Parent task started with traceId: 3f0f0f1e-c0b3-4cef-99e3-7cbfd809e164 on thread: http-nio-8080-exec-1
2024-03-09 19:51:03.183  INFO [,65ec3f17e6d38686445bf9e99014b824,445bf9e99014b824] 25486 --- [nio-8080-exec-1] kr.projects.trace.ChildService           : Child task started with traceId: [Ljava.lang.StackTraceElement;@6223d89e on thread: http-nio-8080-exec-1
2024-03-09 19:51:04.189  INFO [,65ec3f17e6d38686445bf9e99014b824,445bf9e99014b824] 25486 --- [nio-8080-exec-1] kr.projects.trace.ChildService           : Child task completed with traceId: [Ljava.lang.StackTraceElement;@6223d89e on thread: http-nio-8080-exec-1
2024-03-09 19:51:05.191  INFO [,65ec3f17e6d38686445bf9e99014b824,445bf9e99014b824] 25486 --- [nio-8080-exec-1] kr.projects.trace.ParentService          : Parent task completed with traceId: 3f0f0f1e-c0b3-4cef-99e3-7cbfd809e164 on thread: http-nio-8080-exec-1
2024-03-09 19:51:44.421  INFO [,65ec3f40987e7c543582282a039a134d,3582282a039a134d] 25486 --- [nio-8080-exec-2] kr.projects.trace.TraceController        : Starting asynchronous job with traceId: dfebee98-f8e6-450e-be36-bd9703a46923
2024-03-09 19:51:44.434  INFO [,65ec3f40987e7c543582282a039a134d,3582282a039a134d] 25486 --- [nio-8080-exec-2] kr.projects.trace.ParentService          : Parent task started with traceId: dfebee98-f8e6-450e-be36-bd9703a46923 on thread: http-nio-8080-exec-2
2024-03-09 19:51:44.436  INFO [,65ec3f40987e7c543582282a039a134d,3582282a039a134d] 25486 --- [nio-8080-exec-2] kr.projects.trace.ChildService           : Child task started with traceId: [Ljava.lang.StackTraceElement;@7cf9c5a5 on thread: http-nio-8080-exec-2
2024-03-09 19:51:44.620  INFO [,65ec3f405ae8bf117fd3ec3d607efd2d,7fd3ec3d607efd2d] 25486 --- [nio-8080-exec-3] kr.projects.trace.TraceController        : Starting asynchronous job with traceId: 3e7b5aaf-0ee9-477b-9db9-0a8d98746945
2024-03-09 19:51:44.637  INFO [,65ec3f405ae8bf117fd3ec3d607efd2d,7fd3ec3d607efd2d] 25486 --- [nio-8080-exec-3] kr.projects.trace.ParentService          : Parent task started with traceId: 3e7b5aaf-0ee9-477b-9db9-0a8d98746945 on thread: http-nio-8080-exec-3
2024-03-09 19:51:44.637  INFO [,65ec3f405ae8bf117fd3ec3d607efd2d,7fd3ec3d607efd2d] 25486 --- [nio-8080-exec-3] kr.projects.trace.ChildService           : Child task started with traceId: [Ljava.lang.StackTraceElement;@7ebc7ff2 on thread: http-nio-8080-exec-3
2024-03-09 19:51:44.824  INFO [,65ec3f4064b2ba57b991da1e9a58f904,b991da1e9a58f904] 25486 --- [nio-8080-exec-4] kr.projects.trace.TraceController        : Starting asynchronous job with traceId: db6dee41-5500-4f49-a902-d4ed52dfdf1a
2024-03-09 19:51:44.825  INFO [,65ec3f4064b2ba57b991da1e9a58f904,b991da1e9a58f904] 25486 --- [nio-8080-exec-4] kr.projects.trace.ParentService          : Parent task started with traceId: db6dee41-5500-4f49-a902-d4ed52dfdf1a on thread: http-nio-8080-exec-4
2024-03-09 19:51:44.825  INFO [,65ec3f4064b2ba57b991da1e9a58f904,b991da1e9a58f904] 25486 --- [nio-8080-exec-4] kr.projects.trace.ChildService           : Child task started with traceId: [Ljava.lang.StackTraceElement;@2c146429 on thread: http-nio-8080-exec-4
2024-03-09 19:51:45.023  INFO [,65ec3f4105c42d6e1380816172851f52,1380816172851f52] 25486 --- [nio-8080-exec-5] kr.projects.trace.TraceController        : Starting asynchronous job with traceId: 08d2f0c0-b588-4a84-a65c-8d6c1a06993f
2024-03-09 19:51:45.024  INFO [,65ec3f4105c42d6e1380816172851f52,1380816172851f52] 25486 --- [nio-8080-exec-5] kr.projects.trace.ParentService          : Parent task started with traceId: 08d2f0c0-b588-4a84-a65c-8d6c1a06993f on thread: http-nio-8080-exec-5
2024-03-09 19:51:45.024  INFO [,65ec3f4105c42d6e1380816172851f52,1380816172851f52] 25486 --- [nio-8080-exec-5] kr.projects.trace.ChildService           : Child task started with traceId: [Ljava.lang.StackTraceElement;@27562cf1 on thread: http-nio-8080-exec-5
2024-03-09 19:51:45.220  INFO [,65ec3f41b88d12392f0aed8c54e60a0a,2f0aed8c54e60a0a] 25486 --- [nio-8080-exec-6] kr.projects.trace.TraceController        : Starting asynchronous job with traceId: 26adfe5e-2730-4a31-97eb-49c0eacb9a28
2024-03-09 19:51:45.220  INFO [,65ec3f41b88d12392f0aed8c54e60a0a,2f0aed8c54e60a0a] 25486 --- [nio-8080-exec-6] kr.projects.trace.ParentService          : Parent task started with traceId: 26adfe5e-2730-4a31-97eb-49c0eacb9a28 on thread: http-nio-8080-exec-6
2024-03-09 19:51:45.220  INFO [,65ec3f41b88d12392f0aed8c54e60a0a,2f0aed8c54e60a0a] 25486 --- [nio-8080-exec-6] kr.projects.trace.ChildService           : Child task started with traceId: [Ljava.lang.StackTraceElement;@6b7c68eb on thread: http-nio-8080-exec-6
2024-03-09 19:51:45.438  INFO [,65ec3f40987e7c543582282a039a134d,3582282a039a134d] 25486 --- [nio-8080-exec-2] kr.projects.trace.ChildService           : Child task completed with traceId: [Ljava.lang.StackTraceElement;@7cf9c5a5 on thread: http-nio-8080-exec-2
2024-03-09 19:51:45.638  INFO [,65ec3f405ae8bf117fd3ec3d607efd2d,7fd3ec3d607efd2d] 25486 --- [nio-8080-exec-3] kr.projects.trace.ChildService           : Child task completed with traceId: [Ljava.lang.StackTraceElement;@7ebc7ff2 on thread: http-nio-8080-exec-3
2024-03-09 19:51:45.829  INFO [,65ec3f4064b2ba57b991da1e9a58f904,b991da1e9a58f904] 25486 --- [nio-8080-exec-4] kr.projects.trace.ChildService           : Child task completed with traceId: [Ljava.lang.StackTraceElement;@2c146429 on thread: http-nio-8080-exec-4
2024-03-09 19:51:46.026  INFO [,65ec3f4105c42d6e1380816172851f52,1380816172851f52] 25486 --- [nio-8080-exec-5] kr.projects.trace.ChildService           : Child task completed with traceId: [Ljava.lang.StackTraceElement;@27562cf1 on thread: http-nio-8080-exec-5
2024-03-09 19:51:46.224  INFO [,65ec3f41b88d12392f0aed8c54e60a0a,2f0aed8c54e60a0a] 25486 --- [nio-8080-exec-6] kr.projects.trace.ChildService           : Child task completed with traceId: [Ljava.lang.StackTraceElement;@6b7c68eb on thread: http-nio-8080-exec-6
2024-03-09 19:51:46.442  INFO [,65ec3f40987e7c543582282a039a134d,3582282a039a134d] 25486 --- [nio-8080-exec-2] kr.projects.trace.ParentService          : Parent task completed with traceId: dfebee98-f8e6-450e-be36-bd9703a46923 on thread: http-nio-8080-exec-2
2024-03-09 19:51:46.639  INFO [,65ec3f405ae8bf117fd3ec3d607efd2d,7fd3ec3d607efd2d] 25486 --- [nio-8080-exec-3] kr.projects.trace.ParentService          : Parent task completed with traceId: 3e7b5aaf-0ee9-477b-9db9-0a8d98746945 on thread: http-nio-8080-exec-3
2024-03-09 19:51:46.830  INFO [,65ec3f4064b2ba57b991da1e9a58f904,b991da1e9a58f904] 25486 --- [nio-8080-exec-4] kr.projects.trace.ParentService          : Parent task completed with traceId: db6dee41-5500-4f49-a902-d4ed52dfdf1a on thread: http-nio-8080-exec-4
2024-03-09 19:51:47.032  INFO [,65ec3f4105c42d6e1380816172851f52,1380816172851f52] 25486 --- [nio-8080-exec-5] kr.projects.trace.ParentService          : Parent task completed with traceId: 08d2f0c0-b588-4a84-a65c-8d6c1a06993f on thread: http-nio-8080-exec-5
2024-03-09 19:51:47.226  INFO [,65ec3f41b88d12392f0aed8c54e60a0a,2f0aed8c54e60a0a] 25486 --- [nio-8080-exec-6] kr.projects.trace.ParentService          : Parent task completed with traceId: 26adfe5e-2730-4a31-97eb-49c0eacb9a28 on thread: http-nio-8080-exec-6

Traceability Takeaway

TraceId propagation is not about making logs look neat. It is about preserving the ability to reconstruct a request in production. Whenever a thread boundary appears, context propagation has to be handled explicitly, and migrations must check whether automatic behavior from older tracing libraries has disappeared.

In each line, the second and third values inside the brackets after INFO are the traceId and spanId. Even when execution crosses an async boundary, the original trace context is preserved.

Database Benchmarking Criteria and Using HammerDB

ystc1247@gmail.com — Tue, 24 Oct 2023 17:56:27 GMT

Where the Problem Started

Database benchmarks are easy to misread when they become only a comparison of numbers. A high TPS value matters less than the schema and workload that produced it. This post records how I used HammerDB while thinking through which benchmark conditions make the result meaningful.

Choosing a database system has long-term consequences.

If you need to replace the database during operation after making the choice, migration is not easy, so you need to anticipate and detect problems early in development.

Things to think about during the selection process include the following.

Schema and record size
Number of clients
Query format and access patterns
Ratio of read and write queries
Range of variation in the variables above

After that, there is a process of building a test cluster and simulating workloads, which can be done with tools such as YCSB(serving benchmark) and hammerDB.

The following is the hammerDB configuration.

Create a hammerDB Dockerfile in the directory you want.

Implementation Path

cd <directory>
touch Dockerfile
vi Dockerfile

FROM  docker.io/tpcorg/hammerdb:oracle as oracle

FROM  docker.io/tpcorg/hammerdb:mssqls

# Install and configure IBM Db2 client libraries,
# You will need to pre-download IDB Db2 client libraries and place in the local folder
# RUN mkdir -p db2_cli_odbc_driver/odbc_cli
# ADD odbc_cli db2_cli_odbc_driver/odbc_cli/
# RUN apt update && \
#    echo 'debconf debconf/frontend select Noninteractive' | debconf-set-selections && \
#    apt -y install libxml2 && \
#    echo 'export DB2_CLI_DRIVER_INSTALL_PATH="/home/db2_cli_odbc_driver/odbc_cli/clidriver"' >> ~/.bashrc && \
#    echo 'export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/home/db2_cli_odbc_driver/odbc_cli/clidriver/lib"' >> ~/.bashrc && \
#    echo 'export LIBPATH="/home/db2_cli_odbc_driver/odbc_cli/clidriver/lib"' >> ~/.bashrc && \
#    echo 'export PATH="$PATH:/home/db2_cli_odbc_driver/odbc_cli/clidriver/bin"' >> ~/.bashrc && \
#    echo 'export PATH="$PATH:/home/db2_cli_odbc_driver/odbc_cli/clidriver/adm"' >>  ~/.bashrc

COPY --from=oracle /home/instantclient_21_5 /home/instantclient_21_5
ENV ORACLE_LIBRARY=/home/instantclient_21_5/libclntsh.so
RUN echo 'export LD_LIBRARY_PATH=/home/instantclient_21_5/:$LD_LIBRARY_PATH'  >> ~/.bashrc

RUN apt-get update && \
    DEBIAN_FRONTEND=noninteractive apt-get install -y libmariadb3 libpq-dev libmysqlclient21 && \
    rm -rf /var/lib/apt/lists/*

Build the image.

docker build -t hammerdb .

If you want to create a container,

docker run -it --name <container name> hammerdb bash

Database benchmarking starts from the HammerDB CLI. First, locate the CLI path.

docker exec -it hammerdb /bin/bash

Go in and find the path, then run

docker exec -it <container name> <path name>

and you will enter the cli.

Choose the database type for the benchmark.

ora mssqls db2 mysql pg maria

The options are oracle, MsSQL, IBM Db2, MySQL, PostgreSQL, and MariaDB.

For this run, I selected MySQL.

Choose the benchmark type according to the workload you want to model. HammerDB documents the available benchmark families here: https://www.hammerdb.com/benchmarks.html

For now, I will test with the TCP-C type.

hammerdb> dbset bm TPC-C

You need to check the dict(dictionary). This can differ depending on the hammerDB version, database type, and benchmark type, so the settings can vary. For example, in some situations you can set the host and other values with just the set keyword, while in others you may need to use diset to change the dictionary itself.

hammerdb>print dict
Dictionary Settings for MySQL
connection {
 mysql_host               = 127.0.0.1
 mysql_port               = 3306
 mysql_socket             = /tmp/mysql.sock
 mysql_ssl                = false
 mysql_ssl_two_way        = false
 mysql_ssl_linux_capath   = /etc/mysql/certs
 mysql_ssl_windows_capath = C:\mysql\certs
 mysql_ssl_ca             = ca-cert.pem
 mysql_ssl_cert           = client-cert.pem
 mysql_ssl_key            = client-key.pem
 mysql_ssl_cipher         = server
}
tpcc       {
 mysql_count_ware       = 1
 mysql_num_vu           = 1
 mysql_user             = root
 mysql_pass             = mysql
 mysql_dbase            = tpcc
 mysql_storage_engine   = innodb
 mysql_partition        = false
 mysql_prepared         = false
 mysql_total_iterations = 10000000
 mysql_raiseerror       = false
 mysql_keyandthink      = false
 mysql_driver           = timed
 mysql_rampup           = 2
 mysql_duration         = 5
 mysql_allwarehouse     = false
 mysql_timeprofile      = false
 mysql_async_scale      = false
 mysql_async_client     = 10
 mysql_async_verbose    = false
 mysql_async_delay      = 1000
 mysql_connect_pool     = false
}

This is the current dict structure. If you want to change the host, port, user, pass, and so on,

diset connection mysql_port <your port>
diset connection mysql_host <your host>
diset tpcc mysql_user <your user>
diset tpcc mysql_pass <your pass>

run these commands.

If you are trying to load an AWS RDS database, you need to open the IP in the Security Group.

Load the benchmark schema next.

vudestroy

You can think of this as a kind of clean step.

buildschema

This is the command for loading the schema. If the settings were completed correctly, the host and port will appear, and there will be some waiting time.

Vuser 1 created - WAIT IDLE
Vuser 1:RUNNING
Vuser 1:CREATING TPCC SCHEMA
Vuser 1:Ssl_cipher
Vuser 1:CREATING DATABASE tpcc
Vuser 1:CREATING TPCC TABLES
Vuser 1:Loading Item

Refer to https://stackoverflow.com/questions/73624954/how-long-it-takes-to-build-schema-or-my-hammerdb-not-working

Vuser 1:Orders Done
Vuser 1:End:Tue Oct 24 17:43:06 UTC 2023
Vuser 1:CREATING TPCC STORED PROCEDURES
Vuser 1:GATHERING SCHEMA STATISTICS
Vuser 1:TPCC SCHEMA COMPLETE
Vuser 1:FINISHED SUCCESS
ALL VIRTUAL USERS COMPLETE
Schema Build jobid=6537FC1D608703E253535343

After the load finishes, HammerDB shows the completed schema state.

From there, run the benchmark and inspect the performance metrics. On RDS, CloudWatch can provide additional database-level monitoring. Before executing the run, configure the virtual user count to match the scenario.

vurun

Benchmarking Takeaway

A benchmark never fully reproduces production. Still, when schema, client count, query mix, and read/write ratio are explicit, it becomes a comparison that can be reasoned about. The important part is not merely obtaining a number, but being able to explain the conditions behind it.

SWMaestro 14th Cohort Acceptance Retrospective

ystc1247@gmail.com — Thu, 13 Apr 2023 17:56:52 GMT

SWMaestro, along with BOB, was a talent development program I had heard about from seniors since my freshman year. During the previous cohort’s application period, both my algorithm skills and project quality were vague, so I remember stopping midway through the application and deciding to aim for the 14th cohort instead. This time, I felt I could explain my projects properly and bring my coding-test performance up to a stable minimum.

Overall Preparation

Acceptance retrospectives can easily become only a record of the result. What mattered more in this process was not how many keywords I memorized, but how clearly I could explain the projects I had built. This post captures how I prepared separately for the coding tests and interviews, and what kinds of questions I expected to defend.

I organized my preparation into three tracks.

For the coding tests, I focused less on solving every problem and more on securing the problems I could implement without mistakes.
For the interview, I prepared to explain my projects from structure, technology choices, failure modes, and possible improvements.
For CS and backend fundamentals, the goal was not keyword memorization. I wanted to be able to keep answering when follow-up questions came in.

Looking back, the most important thing was not flashy knowledge but organized explanation. Especially in the project interview, “I used this technology” mattered less than “why I used it, and where I would look if something broke.”

Coding Tests

The tests were held on Programmers for two hours. The format was 4 algorithm problems and 1 SQL problem. The first round was invalidated once because of a server issue, and then it was held again in the same format.

What I Felt From Round 1

Round 1 had a strong implementation and simulation flavor. There were simple implementation, coordinate-plane traversal, combination search, DFS-like search, and a SQL problem with string-processing characteristics. The ideas themselves were not extremely novel. The hard part was implementing them within the time limit without missing conditions.

These are the types I felt I needed to secure.

Type	Preparation point
Implementation / simulation	Split conditions into explicit states and write edge cases first
Brute force / combinations	Quickly judge whether the input size allows brute force
DFS / backtracking	Define state and visited conditions clearly
SQL	Review string functions in addition to SELECT, JOIN, and GROUP BY

For implementation problems in particular, even if you know the direction, mistakes appear once the code gets long. So if I were preparing for SWMaestro again, I would spend more time solving implementation-heavy problems with a timer than digging into new advanced algorithms.

What I Felt From Round 2

Round 2 also had 4 algorithm problems and 1 SQL problem. The set included implementation, DP, queue-based simulation, and a graph problem mixing Dijkstra and DP. Personally, handling edge cases in the DP problem was harder than the graph problem.

Both rounds felt tight on time. Roughly two solved problems felt like the stable line, and I heard from a mentor that many trainees in Round 2 only solved Problem 1 and SQL clearly. If you have steadily studied the essential types for job coding tests, the test itself is not unfamiliar. But if your implementation stamina is weak, it can shake you more than expected.

If I prepared again, I would prioritize this order.

Solve implementation and simulation problems regularly, even if only briefly
For DP, focus on state definition and edge cases more than only the recurrence
Keep BFS, DFS, and Dijkstra templates comfortable enough to write quickly
Solve at least one SQL problem each using JOIN, GROUP BY, UNION, and string functions

Interview Preparation

At first, I spread the interview topics too widely, which made the notes messy. Later, I grouped them like this. It was much more useful.

1. Project Explanation

The biggest preparation area was my own projects. I expected the interviewers to dig deeply into the projects I presented.

For each project, I prepared answers to these questions.

What problem was this project trying to solve?
Which part did I own?
Why did I choose that technology stack?
What did the DB schema, API, and deployment structure look like?
If an incident happened, what would I check first?
If I built it again now, what would I change?

For example, for a Redis-based project, I prepared the caching target, DB fallback when Redis fails, Base62 short URL generation, MariaDB character set configuration, Docker Compose depends_on, and the CSV parsing flow as one connected explanation. It was not enough to say “I used Redis.” I needed to explain why Redis was necessary and what would happen if Redis went down.

For a GPT-based project, I prepared not only implementation details but also business model, cost structure, and user acquisition. The SWMaestro interview did not feel like a pure technical interview. It also checked whether a person could carry a team project to the end.

2. Backend Fundamentals

For backend fundamentals, I did not try to go infinitely deep. I focused on the range connected to the technologies I had actually used.

Area	What I reviewed
Spring / Spring Boot	Dependency management, embedded server, initial setup, Spring Security and CORS
REST API	URI design, CRUD, JSON response, common frontend/backend integration issues
Testing	When to use JUnit5, Spock, and mocking
Database	Normalization, RDB vs NoSQL, MySQL/MariaDB/PostgreSQL/Redis characteristics
Cache	Redis caching targets, session/API response caching, fallback strategy
Docker / deployment	Container communication, Docker Compose, EC2 deployment, environment configuration
Git / collaboration	Branch rules, commit units, pull/review habits

I used to try to memorize this kind of material as a list of keywords, but in interviews that becomes obvious quickly. Questions do not always stop at “what is normalization?” They can continue into “why was this schema acceptable for your project?” Connecting fundamentals back to projects was much better.

3. Collaboration and Leadership

Collaboration questions also felt important. The expected questions were roughly like these.

What would you do if a teammate disagreed with you?
What would you do if the initial plan could no longer be executed?
How would you communicate with a teammate using a different technology stack?
If you were the leader, how would you manage schedule and backlog?

I prepared answers around making the initial plan detailed, while reflecting changed requirements into the backlog and timeline during implementation. When teammates used different stacks, I planned to document contact points such as API contract, data format, CORS, and security, and check likely frontend/backend merge issues early.

In-depth Interview

I heard that the atmosphere and question direction differed depending on the division and interviewers. Division 1, where I interviewed, had a balanced mix of technical and personality questions. Personality questions were often common questions given to all five candidates.

The most memorable question came near the end. They asked which candidate in the room I would want to team up with, and who seemed suitable as a leader. It was difficult to answer unless you had listened to the other candidates carefully. Some people answered vaguely, while others chose a specific candidate and gave a reason.

The technical interview went deep around the project I had presented. If a project looked weak, they seemed to ask more persistently about planning intent and technology choices. For the coding test, printed copies of the Round 1 and Round 2 problems were handed out, but only a few coding-test questions came up across all candidates. Still, the interviewers knew the Programmers submission history, submitted code, and partial scores, so it was not something you could hand-wave away.

In conclusion, if you thoroughly understand the projects you built and prepare for follow-up questions on the technologies you used, the interview is not especially hard to answer. After passing both coding tests, the remaining question felt closer to how clearly I could explain my own experience.

Interview Takeaway

In the end, preparing for SWMaestro was not about collecting impressive technology names. It was about reaching the point where I could explain what I had actually built, including its structure, failure modes, and what I would redesign today. Once that was clear, the interview became much less intimidating.

Theo Kim - Backend & Cloud Systems Engineer

Scaling Healthcare Chat Beyond an Echo WebSocket

WebSocket changes the session, not all of HTTP

The measured architecture

Authenticate before accepting application messages

Validate Origin separately from authentication

Persist before acknowledging

Make client retries idempotent

Backpressure must have a policy

Scale-out needs a reconnect protocol

Protect healthcare data by reducing it

Failure behavior is the design

A Kubernetes Rollout That Failed Safely: Readiness, Deadlines, and Rollback

Make availability a Deployment constraint

Readiness and liveness answer different questions

Probe availability during the rollout

Exercise a healthy rollout

Inject a readiness failure and wait for the deadline

Roll back deliberately

Understand what the PDB does not do

Rolling, blue-green, and canary answer different risks

Production gates I would add

Kafka Delivery Semantics in Practice: Ordering, Duplicates, Rebalances, Retries, and DLT

Ordering exists inside one partition

Producer idempotence is necessary but not end-to-end idempotence

Put consumer idempotency in the same database transaction

Rebalance changes ownership, not record identity

Bound retries and separate permanent failures

Inspect both the log and the effect

Serialization is a contract

What the guarantees really are

The HTTP Client Boundary Behind a Feign PATCH Failure

Trace the abstraction downward

Make the declarative contract narrow

Register the transport explicitly

Do not retry a mutation blindly

Map upstream semantics at the gateway

Verify the actual request

Log the boundary without leaking it

A practical debugging sequence

Protecting a JPA Aggregate: Invariants, DTOs, and Optimistic Locking

Start from an invalid state that setters allow

Put commands on the aggregate root

Do not serialize the entity as the API

Optimistic locking must cover the whole aggregate

Test the domain without Hibernate, then test Hibernate

Where accessors remain reasonable

Transaction and loading boundaries still matter

When a Redis Leaderboard Lies: Outbox, Idempotency, and Reconciliation

Define the consistency contract

Commit the score and outbox atomically

Project with at-least-once delivery in mind

Design for the crash between Redis and PostgreSQL

Reconcile because idempotency is not repair

What the failure experiment observed

Expose freshness to the product

The operational checklist

Reducing Spring Boot Log Cost Without Hiding Failures

Decide which signal is redundant

Classify from the parsed request, not formatted text

Disable one logger through configuration

Preserve the metric before removing the log

Measure before and after

Make the cost projection explicit

Guardrails for production

A Reproducible JMeter Load Test: Throughput, p95, Errors, and Evidence

Define the question before the thread group

Concurrency is not throughput

Run JMeter without the GUI

Read the distribution, not the average

Verify the result independently

Correlate the bottleneck

What this run does not prove

Operating a Spring Boot SLO with Prometheus, Alertmanager, and Grafana

Start with the service-level objective

Keep the metrics endpoint off the public edge

Cardinality is part of the design

Record the queries the dashboard and alerts share

Page on budget burn, not on a decorative graph

Provision the dashboard as code