← Back home

Postmortems

A service went down and stayed down for 37.5 hours before anyone noticed — despite Uptime Kuma monitoring it and, on paper, having already alerted. The outage looked like a monitoring blind spot at first: had Kuma actually seen it go down, or had the check itself failed silently?

Uptime KumaMonitoringAlerting·2026-08-08
Jellyfin Freeze — Longhorn Instance-Manager Memory Leak42 days to surface; recurred 55 days later

Jellyfin streams froze because longhorn-instance-manager leaked ~170MB/day for 42 days, traced to a 430-snapshot pileup left over from the [June 19th incident](/postmortems/2026-06-19-k3s-longhorn-disk-incident) with no recurring job to prune them. Fixed by pruning the chain, tuning replica timeouts, and adding monitoring.

LonghornMemory LeakJellyfinMonitoringKnown Upstream Bug·2026-07-31
x — The Silent Database Bootstrap Bugcaught same-day, no production impact

While building four new features in one push (Lists, Explore, Bookmarks, Notifications), a database bootstrap function got called in the wrong order relative to model imports — and it failed with no error at all. Tables that should have existed silently didn't.

xSQLAlchemyBug·2026-07-28
Prometheus and Alertmanager Silently Stopped Restartingunknown — caught by a watchdog, not a report

Prometheus and Alertmanager, both part of the monitoring stack on tspi, stopped coming back after a host reboot — not crash-looping, not erroring, just absent from `docker ps`. Nothing else in the same compose stack was affected, and there was no alert for it, since the thing that was supposed to alert was one of the two containers missing.

DockerMonitoringPrometheusAlertmanagerlibnetworktspi·2026-06-27

Same Longhorn webhook deadlock as [June 2nd](/postmortems/2026-06-02-k3s-outage), this time triggered by disk at 94-95% full stalling SQLite writes. Cluster self-recovered in ~30 minutes; root disk-pressure alert had been firing unnoticed since 70%.

k3sLonghornDisk PressureAlerting·2026-06-19

Restoring a missing Longhorn CSI driver cascaded into a wrong-version install, missing CRDs, and the Longhorn uninstaller deleting the live MySQL volume. Recovered by migrating MySQL to local-path storage and manually rebuilding schemas — with data loss.

MySQLLonghornData LossRecovery·2026-06-02

k3s's SQLite state DB grew to 1.6GB from crash-loop spam, stalling the API server. Restarting k3s then hit a Longhorn admission webhook stuck registered with no backing pods, which blocked flannel from initializing and took every sacenpapier.org endpoint offline until the webhook was force-deleted.

k3sSQLiteLonghornNetworking·2026-06-02