Two ServiceAccounts running with full `cluster-admin` got scoped down to exactly the permissions they actually use: `homelab-dashboard`'s to read-only `get`/`list` on nodes and pods, and `kube-system`'s Prometheus SA to `get` on `/metrics` and `nodes/metrics` only.
An explicit `kustomization.yaml` per repo's `k3s/` directory, replacing ArgoCD's previous behavior of applying every `.yaml` file it found there. `homelab-sacenpapier` is done; `sacenpapier`, `Apercu`, `BotWhy`, `PopRoom`, and `x` are in progress.
Every app that actually holds secrets — Apercu, BotWhy, PopRoom, x, and homelab-sacenpapier — moved off hand-applied `k3s/secrets/*.yaml` files and onto the **Doppler Kubernetes Operator**. A `DopplerSecret` CRD per repo (safe to commit — no values, just a project/config reference) now syncs each app's real secrets from Doppler straight into the same Kubernetes Secret its Deployment already expects, with auto-reload on change.
All 5 FastAPI backends now push OpenTelemetry traces to a Grafana Tempo instance on `tspi`, completing the observability trifecta alongside the metrics (Prometheus) and logs (Loki) that already existed. Every traced request also generates RED metrics and service-graph edges in Prometheus automatically, with zero extra instrumentation.
All 6 apps switched from SSH-triggered deploys to ArgoCD Image Updater. CI now pushes images tagged by the commit SHA instead of reusing `:latest`; Image Updater detects the new tag, writes it into each app's `kustomization.yaml`, and ArgoCD syncs it like any other git change. The `appleboy/ssh-action` step — and the `SSH_HOST`/`SSH_USER`/`SSH_PRIVATE_KEY` secrets behind it — is gone from all 6 workflows.
One `blackbox-exporter` instance on the monitoring Pi (`tspi`) now probes all six public URLs — sacenpapier.org, Apercu, BotWhy, PopRoom, x, and the homelab dashboard — instead of any one-off, per-app uptime check. A single Prometheus scrape job hits each site through the exporter's `/probe` endpoint, each tagged with a `project` label, backed by two new alert rules.
Both integrations that talk to TrueNAS — the homelab dashboard's backend and a separate Prometheus alert-forwarding script — moved off TrueNAS's REST API and onto the official `truenas_api_client` library over `wss://` (JSON-RPC 2.0/WebSocket).
Two monitoring gaps closed the same day: firewall-level visibility into OPNsense (previously host-level only), and a ZFS pool health check for TrueNAS — the single most important alert missing for a mirrored pool.
Swapped the local model behind Bob (the homelab's always-on Hermes Agent) from `gpt-oss-20b` to `gemma4:12b`, served through Ollama with a 65,536-token context and verified real tool-calling — actual terminal commands executed and reported accurately, not just described. Same `gpu-proxy` layer as before (model-alias mapping, gaming-safety integration, Wake-on-LAN), so nothing downstream had to change.
`PopRoom`, `BotWhy`, `Apercu`, and `x` each connect to the shared MySQL instance (`shared-mysql`), but all four were using the exact same login — one username and password with broad `ALL PRIVILEGES` grants across every app's database. Replaced that with four dedicated MySQL users, each scoped to only its own database: `apercu_app` → `apercu`, `botwhy_app` → `chatbox_db`, `x_app` → `twitter_db`, `poproom_app` → `poproom`. The old shared login's grants on all four databases were revoked once every app confirmed healthy on its new credentials.