2026-06-02·~1h

k3s Outage — SQLite Bloat + Longhorn Webhook Deadlock

k3sSQLiteLonghornNetworking

What happened

k3s's SQLite state DB grew to 1.6GB from crash-loop spam, stalling the API server. Restarting k3s then hit a Longhorn admission webhook stuck registered with no backing pods, which blocked flannel from initializing and took every sacenpapier.org endpoint offline until the webhook was force-deleted.

Root cause

Two compounding causes. First, the k3s SQLite state DB (kine backend) grew to 1.6GB / 558k rows from an NFS provisioner crash loop and Longhorn lease spam, causing write timeouts. Second — the real outage trigger — Longhorn's admission webhooks stayed registered in Kubernetes with no backing pods running. Every time k3s tried to set the control-plane role label on the node, it called the dead webhook, failed, and retried every second indefinitely, which prevented the flannel.1 VXLAN interface from ever being created — so there were no routes to the OCI worker nodes.

Fix

Stopped k3s, vacuumed the SQLite DB (deleted stale kine rows, ran VACUUM — 558k → 264k rows), then deleted the stuck ValidatingWebhookConfiguration and MutatingWebhookConfiguration for Longhorn. Restarted k3s on the control-plane and k3s-agent on both OCI nodes to rebuild flannel routes. Also rolled back a Traefik deployment that had been stuck pending for 9 days on a pod anti-affinity conflict.

Prevention

Added a weekly SQLite compact cron job (Sundays 3am). Documented the recovery order for next time: delete Longhorn webhooks first if Longhorn is broken, restart k3s on the control-plane, then restart k3s-agent on the workers, and wait for flannel.1 routes to reappear.

More postmortems