A cron job on `elitedesk`, running every 5 minutes, that watches for a specific recurring failure signature in Longhorn's manager logs — `webhook ... context deadline exceeded` — and, once it's seen more than 3 times in a 5-minute window, deletes Longhorn's own stale validating and mutating webhook configurations so it re-registers itself cleanly.
11 NetworkPolicies across 5 apps plus the default namespace: frontends locked down to Traefik-only ingress, and the shared MySQL instance locked to only its 4 real client apps.
Every container across all 13 Deployments in the cluster — 10 in `default`, 3 in `homelab-dashboard` — now has CPU/memory requests and limits set, sized from real `kubectl top pods` observations rather than guessed.
The same observability shape rolled out to all five FastAPI backends — homelab-sacenpapier, PopRoom, BotWhy, Apercu, and x: a `/metrics` endpoint via `prometheus-fastapi-instrumentator`, an `/api/health` check, and one app-specific business gauge per app that a pure infra-level check can't see. Each app gets its own NodePort metrics Service and a Prometheus scrape job on `tspi`, tagged with a `project` label.
Two ServiceAccounts running with full `cluster-admin` got scoped down to exactly the permissions they actually use: `homelab-dashboard`'s to read-only `get`/`list` on nodes and pods, and `kube-system`'s Prometheus SA to `get` on `/metrics` and `nodes/metrics` only.
An explicit `kustomization.yaml` per repo's `k3s/` directory, replacing ArgoCD's previous behavior of applying every `.yaml` file it found there. `homelab-sacenpapier` is done; `sacenpapier`, `Apercu`, `BotWhy`, `PopRoom`, and `x` are in progress.
Every app that actually holds secrets — Apercu, BotWhy, PopRoom, x, and homelab-sacenpapier — moved off hand-applied `k3s/secrets/*.yaml` files and onto the **Doppler Kubernetes Operator**. A `DopplerSecret` CRD per repo (safe to commit — no values, just a project/config reference) now syncs each app's real secrets from Doppler straight into the same Kubernetes Secret its Deployment already expects, with auto-reload on change.
Moved all six apps in the fleet (PopRoom, BotWhy, x, Apercu, sacenpapier, homelab-sacenpapier) from manual `kubectl rollout restart` deploys to ArgoCD-managed GitOps. Every app's `k3s/` manifests are now the single source of truth — the cluster continuously reconciles against git and self-heals if anything drifts.
Added `podAntiAffinity` to the homelab dashboard's frontend deployment, so its three replicas spread across different nodes instead of potentially landing on the same one.
The homelab dashboard's k3s section now shows which pod and node actually served the current request — visible directly on the page, with a rotate button to re-route and watch the highlight change — plus inline skeleton loading states across all live values instead of blank space while data loads, and light/dark mode.