What I found

Vector had grown to 3.6 GB on a 4 GB instance with no swap and taken the primary AWS VPN down with it. Failover then did exactly what it was built to do: the VIP moved to the backup, traffic kept flowing, and nothing degraded enough to alert. The outage lasted a week because the recovery worked.

What changed

Memory bounded on the gateway, and the gap went straight into the status page catalog. A successful failover is indistinguishable from health unless something alerts on running degraded.

What I found

A managed OpenSearch domain was failing most requests. The load balancer target group was still pinned to ENI addresses that no longer existed, left behind by a volume-shrink blue/green that had otherwise completed cleanly.

What changed

Target group repaired, then the pressure underneath it: deleting 1,556 stale indices took the shard count from the 4,000 cap down to 918, and heap from 79-85% down to 43-50%.

What I found

A wildcard rotation went out with the leaf but no intermediate. Browsers papered over it by fetching the missing certificate through AIA, so every manual check looked fine. Go and Node clients do not do that, and they failed. The verification step had used curl -k, which skips the exact check that would have caught it.

What changed

Rotation reverted, and a strict-verification test written that fails the way a real client fails rather than the way a browser does.

What I found

Recurring MirrorMaker2 lag pages, arriving at idle traffic levels, which ruled out load. Not the NIC, not Aerospike XDR, not WireGuard. It was packet loss on the public transatlantic transit underneath, on a path that was asymmetric: one carrier outbound, a different one on the way back.

What changed

This is what the cross-DC mtr exporter was built for. Fixed-IP ping probes could not survive a reroute or say which hop was losing; per-hop probing with carrier attribution can.

What I found

A compaction concurrency change I had made filled a 600 Gi volume. Expanding it live to 900 Gi fixed the immediate problem and exposed a second one: replica pressure on three control-plane root disks, traced to the filesystem trim cron running against Longhorn.

What changed

Scratch sizing documented per compaction group, so the next person raising concurrency knows what it costs in disk before they raise it.

What I found

Leaseweb’s NTP served bad time in three episodes. Clocks stepped forward by anywhere from 4 seconds to 8 minutes, a VPN address flapped twice because its health check read wall-clock time, and an alert reported reboots across the fleet. It was measuring boot time, which moves when the clock steps.

What changed

Monotonic uptime proved no host had rebooted. Then multi-source chrony replaced single-source timesyncd on 105 hosts across three regions, one host at a time, with no downtime.