sacenpapier.org

A routine homelab shutdown (elitedesk → TrueNAS → OPNsense) turned into a multi-hour recovery after several independent issues compounded: `docker compose down` removed all 19 containers on elitedesk instead of just stopping them; elitedesk was moved from its onboard gig NIC to a new USB 2.5GbE adapter mid-session, changing its LAN IP from `.60` to `.99`; TrueNAS and OPNsense both power-cycled unexpectedly on their own; and the ISP router's port-forward rules (pointing at elitedesk's old `.60` IP) went stale, taking `jellyfin.sacenpapier.org`, `seerr.sacenpapier.org`, `ssh.sacenpapier.org`, and `mc.sacenpapier.org` down externally even after all services were healthy internally.

Set out to move Bob (the homelab's Hermes Agent instance) from Ollama/gpt-oss-20b to vLLM running Gemma 4 12B on an RTX 5080, chasing better tool-calling reliability. Got vLLM fully working after root-causing six separate platform bugs — then hit a service-killing SIGTERM every 20-40 seconds under real load that never got explained, despite ruling out every plausible cause. Abandoned vLLM/WSL2 entirely and shipped the same model (Gemma 4 12B) through Ollama running natively on Windows instead — which turned out to be the better architecture anyway, and along the way surfaced two pre-existing regressions in the gaming-safety system that had been silently broken for the entire session.

Auditing what was actually under ArgoCD turned up an unreachable but still-running MySQL instance (`mysql-longhorn`) holding real data. Its creation date and PVC name matched the Longhorn-backed instance from the June 2nd incident — migrated off in that fix, but never actually decommissioned.

Rolling out ArgoCD across six apps, two repos (`Apercu`, `x`) turned out to declare an identically-named shared MySQL resource. Pointing separate ArgoCD Applications at that collision caused the shared database volume to be recreated from scratch, silently wiping live data for three apps — one of them (BotWhy) kept running normally the whole time, never throwing an error.

Jellyfin streams froze because longhorn-instance-manager leaked ~170MB/day for 42 days, traced to a 430-snapshot pileup left over from the June 19th incident with no recurring job to prune them. Fixed by pruning the chain, tuning replica timeouts, and adding monitoring.

While building four new features in one push (Lists, Explore, Bookmarks, Notifications), a database bootstrap function got called in the wrong order relative to model imports — and it failed with no error at all. Tables that should have existed silently didn't.

Same Longhorn webhook deadlock as June 2nd, this time triggered by disk at 94-95% full stalling SQLite writes. Cluster self-recovered in ~30 minutes; root disk-pressure alert had been firing unnoticed since 70%.

Restoring a missing Longhorn CSI driver cascaded into a wrong-version install, missing CRDs, and the Longhorn uninstaller deleting the live MySQL volume. Recovered by migrating MySQL to local-path storage and manually rebuilding schemas — with data loss.

k3s's SQLite state DB grew to 1.6GB from crash-loop spam, stalling the API server. Restarting k3s then hit a Longhorn admission webhook stuck registered with no backing pods, which blocked flannel from initializing and took every sacenpapier.org endpoint offline until the webhook was force-deleted.

The first attempt at a working GitHub Actions deploy pipeline for BotWhy took a long, visible trail of small failed commits over about a week — SSH connection failures, Docker build errors, `git pull` conflicts, React build issues — each fixed one at a time until the pipeline finally ran clean.