https://sacenpapier.org/updates Shipped features and changes across sacenpapier.org's projects. https://sacenpapier.org/updates/2026-10-05-gaming-detection-simplified-ssh-lock-state-instead-of-a-windows-agent https://sacenpapier.org/updates/2026-10-05-gaming-detection-simplified-ssh-lock-state-instead-of-a-windows-agent Mon, 05 Oct 2026 00:00:00 GMT `gpu-proxy`'s gaming guard — the thing that stops a local-LLM wake-up (or unloads an already-loaded model) from competing with a game for the one GPU in the house — used to run as a Windows-side agent polling `nvidia-smi` and maintaining an exclusion list of system/vendor processes (`dwm.exe`, `explorer.exe`, `NVIDIA Overlay.exe`, ...) to tell "a game" apart from normal desktop chrome. That agent is gone. In its place: `gpu-proxy` on tselitedesk polls Windows's lock state every 5 seconds over the SSH connection it already holds — `[bool](Get-Process -Name LogonUI -ErrorAction SilentlyContinue)` — and nothing runs on the gaming PC for this anymore. https://sacenpapier.org/updates/2026-09-18-two-tier-automated-container-updates-with-watchtower https://sacenpapier.org/updates/2026-09-18-two-tier-automated-container-updates-with-watchtower Fri, 18 Sep 2026 00:00:00 GMT Watchtower runs on both non-cluster hosts, but not every container gets the same treatment: stateless infra pieces auto-update on sight, while anything holding real user data or state — Sonarr, Radarr, Jellyfin, qBittorrent, Prowlarr, byparr, Seerr, Gluetun — is kept on monitor-only. A separate script reads Watchtower's own logs daily, and if any of those protected containers has a new image waiting, it fires a real alert into Alertmanager instead of updating silently or getting forgotten. https://sacenpapier.org/updates/2026-09-18-nightly-config-backup-for-the-homelab-fleet https://sacenpapier.org/updates/2026-09-18-nightly-config-backup-for-the-homelab-fleet Fri, 18 Sep 2026 00:00:00 GMT Both `elitedesk` and `tspi` — the two hosts running things outside the k3s cluster itself — cron a nightly script that pulls their own live configuration back into the `homelab` git repo and pushes it: Docker Compose files, Prometheus/Alertmanager configs, every Grafana dashboard (exported fresh via the Grafana API, not hand-copied), and each host's own crontab. https://sacenpapier.org/updates/2026-09-18-longhorn-webhook-watchdog https://sacenpapier.org/updates/2026-09-18-longhorn-webhook-watchdog Fri, 18 Sep 2026 00:00:00 GMT A cron job on `elitedesk`, running every 5 minutes, that watches for a specific recurring failure signature in Longhorn's manager logs — `webhook ... context deadline exceeded` — and, once it's seen more than 3 times in a 5-minute window, deletes Longhorn's own stale validating and mutating webhook configurations so it re-registers itself cleanly. https://sacenpapier.org/updates/2026-09-15-auto-login-demo-user-on-x https://sacenpapier.org/updates/2026-09-15-auto-login-demo-user-on-x Tue, 15 Sep 2026 00:00:00 GMT An auto-login demo user on `x.sacenpapier.org`, confirmed working end-to-end. https://sacenpapier.org/updates/2026-09-14-trivy-cve-gate-in-ci-catches-real-vulnerabilities-fleet-wide https://sacenpapier.org/updates/2026-09-14-trivy-cve-gate-in-ci-catches-real-vulnerabilities-fleet-wide Mon, 14 Sep 2026 00:00:00 GMT Every one of the 6 GitHub Actions pipelines now builds a local, unpushed image first, scans it with Trivy for `CRITICAL` severity CVEs, and only proceeds to the real multi-platform build-and-push if it's clean. An image with a critical, fixable vulnerability now never reaches Docker Hub at all — a true gate, not a post-push alert Image Updater could still pick up. https://sacenpapier.org/updates/2026-09-14-python-jose-cve-fix https://sacenpapier.org/updates/2026-09-14-python-jose-cve-fix Mon, 14 Sep 2026 00:00:00 GMT A confirmed fix for a known CVE in the `python-jose` dependency, closing out a flagged item from the security-hardening backlog. https://sacenpapier.org/updates/2026-09-14-networkpolicies-shipped-across-the-fleet https://sacenpapier.org/updates/2026-09-14-networkpolicies-shipped-across-the-fleet Mon, 14 Sep 2026 00:00:00 GMT 11 NetworkPolicies across 5 apps plus the default namespace: frontends locked down to Traefik-only ingress, and the shared MySQL instance locked to only its 4 real client apps. https://sacenpapier.org/updates/2026-09-14-kubernetes-resource-requests-and-limits-audit https://sacenpapier.org/updates/2026-09-14-kubernetes-resource-requests-and-limits-audit Mon, 14 Sep 2026 00:00:00 GMT Every container across all 13 Deployments in the cluster — 10 in `default`, 3 in `homelab-dashboard` — now has CPU/memory requests and limits set, sized from real `kubectl top pods` observations rather than guessed. https://sacenpapier.org/updates/2026-09-13-shared-observability-pattern-across-the-fleet https://sacenpapier.org/updates/2026-09-13-shared-observability-pattern-across-the-fleet Sun, 13 Sep 2026 00:00:00 GMT The same observability shape rolled out to all five FastAPI backends — homelab-sacenpapier, PopRoom, BotWhy, Apercu, and x: a `/metrics` endpoint via `prometheus-fastapi-instrumentator`, an `/api/health` check, and one app-specific business gauge per app that a pure infra-level check can't see. Each app gets its own NodePort metrics Service and a Prometheus scrape job on `tspi`, tagged with a `project` label. https://sacenpapier.org/updates/2026-09-13-scoping-serviceaccounts-off-cluster-admin https://sacenpapier.org/updates/2026-09-13-scoping-serviceaccounts-off-cluster-admin Sun, 13 Sep 2026 00:00:00 GMT Two ServiceAccounts running with full `cluster-admin` got scoped down to exactly the permissions they actually use: `homelab-dashboard`'s to read-only `get`/`list` on nodes and pods, and `kube-system`'s Prometheus SA to `get` on `/metrics` and `nodes/metrics` only. https://sacenpapier.org/updates/2026-09-13-kustomize-migration-for-k3s-and-argocd https://sacenpapier.org/updates/2026-09-13-kustomize-migration-for-k3s-and-argocd Sun, 13 Sep 2026 00:00:00 GMT An explicit `kustomization.yaml` per repo's `k3s/` directory, replacing ArgoCD's previous behavior of applying every `.yaml` file it found there. `homelab-sacenpapier` is done; `sacenpapier`, `Apercu`, `BotWhy`, `PopRoom`, and `x` are in progress. https://sacenpapier.org/updates/2026-09-13-doppler-secrets-management-across-the-fleet https://sacenpapier.org/updates/2026-09-13-doppler-secrets-management-across-the-fleet Sun, 13 Sep 2026 00:00:00 GMT Every app that actually holds secrets — Apercu, BotWhy, PopRoom, x, and homelab-sacenpapier — moved off hand-applied `k3s/secrets/*.yaml` files and onto the **Doppler Kubernetes Operator**. A `DopplerSecret` CRD per repo (safe to commit — no values, just a project/config reference) now syncs each app's real secrets from Doppler straight into the same Kubernetes Secret its Deployment already expects, with auto-reload on change. https://sacenpapier.org/updates/2026-09-13-distributed-tracing-with-opentelemetry-and-tempo https://sacenpapier.org/updates/2026-09-13-distributed-tracing-with-opentelemetry-and-tempo Sun, 13 Sep 2026 00:00:00 GMT All 5 FastAPI backends now push OpenTelemetry traces to a Grafana Tempo instance on `tspi`, completing the observability trifecta alongside the metrics (Prometheus) and logs (Loki) that already existed. Every traced request also generates RED metrics and service-graph edges in Prometheus automatically, with zero extra instrumentation. https://sacenpapier.org/updates/2026-09-13-closing-the-ci-to-gitops-loop-with-argocd-image-updater https://sacenpapier.org/updates/2026-09-13-closing-the-ci-to-gitops-loop-with-argocd-image-updater Sun, 13 Sep 2026 00:00:00 GMT All 6 apps switched from SSH-triggered deploys to ArgoCD Image Updater. CI now pushes images tagged by the commit SHA instead of reusing `:latest`; Image Updater detects the new tag, writes it into each app's `kustomization.yaml`, and ArgoCD syncs it like any other git change. The `appleboy/ssh-action` step — and the `SSH_HOST`/`SSH_USER`/`SSH_PRIVATE_KEY` secrets behind it — is gone from all 6 workflows. https://sacenpapier.org/updates/2026-09-13-blackbox-exporter-for-all-public-urls https://sacenpapier.org/updates/2026-09-13-blackbox-exporter-for-all-public-urls Sun, 13 Sep 2026 00:00:00 GMT One `blackbox-exporter` instance on the monitoring Pi (`tspi`) now probes all six public URLs — sacenpapier.org, Apercu, BotWhy, PopRoom, x, and the homelab dashboard — instead of any one-off, per-app uptime check. A single Prometheus scrape job hits each site through the exporter's `/probe` endpoint, each tagged with a `project` label, backed by two new alert rules. https://sacenpapier.org/updates/2026-09-12-truenas-integration-migrates-from-rest-to-json-rpc https://sacenpapier.org/updates/2026-09-12-truenas-integration-migrates-from-rest-to-json-rpc Sat, 12 Sep 2026 00:00:00 GMT Both integrations that talk to TrueNAS — the homelab dashboard's backend and a separate Prometheus alert-forwarding script — moved off TrueNAS's REST API and onto the official `truenas_api_client` library over `wss://` (JSON-RPC 2.0/WebSocket). https://sacenpapier.org/updates/2026-09-11-opnsense-firewall-monitoring-and-zfs-pool-health-checks https://sacenpapier.org/updates/2026-09-11-opnsense-firewall-monitoring-and-zfs-pool-health-checks Fri, 11 Sep 2026 00:00:00 GMT Two monitoring gaps closed the same day: firewall-level visibility into OPNsense (previously host-level only), and a ZFS pool health check for TrueNAS — the single most important alert missing for a mirrored pool. https://sacenpapier.org/updates/2026-08-18-gemma-4-12b-replaces-gpt-oss-20b-as-bob-s-local-model https://sacenpapier.org/updates/2026-08-18-gemma-4-12b-replaces-gpt-oss-20b-as-bob-s-local-model Tue, 18 Aug 2026 00:00:00 GMT Swapped the local model behind Bob (the homelab's always-on Hermes Agent) from `gpt-oss-20b` to `gemma4:12b`, served through Ollama with a 65,536-token context and verified real tool-calling — actual terminal commands executed and reported accurately, not just described. Same `gpu-proxy` layer as before (model-alias mapping, gaming-safety integration, Wake-on-LAN), so nothing downstream had to change. https://sacenpapier.org/updates/2026-08-14-per-app-mysql-users-ending-shared-database-credentials https://sacenpapier.org/updates/2026-08-14-per-app-mysql-users-ending-shared-database-credentials Fri, 14 Aug 2026 00:00:00 GMT `PopRoom`, `BotWhy`, `Apercu`, and `x` each connect to the shared MySQL instance (`shared-mysql`), but all four were using the exact same login — one username and password with broad `ALL PRIVILEGES` grants across every app's database. Replaced that with four dedicated MySQL users, each scoped to only its own database: `apercu_app` → `apercu`, `botwhy_app` → `chatbox_db`, `x_app` → `twitter_db`, `poproom_app` → `poproom`. The old shared login's grants on all four databases were revoked once every app confirmed healthy on its new credentials. https://sacenpapier.org/updates/2026-08-14-argocd-gitops-rollout https://sacenpapier.org/updates/2026-08-14-argocd-gitops-rollout Fri, 14 Aug 2026 00:00:00 GMT Moved all six apps in the fleet (PopRoom, BotWhy, x, Apercu, sacenpapier, homelab-sacenpapier) from manual `kubectl rollout restart` deploys to ArgoCD-managed GitOps. Every app's `k3s/` manifests are now the single source of truth — the cluster continuously reconciles against git and self-heals if anything drifts. https://sacenpapier.org/updates/2026-07-28-x-four-features-one-day https://sacenpapier.org/updates/2026-07-28-x-four-features-one-day Tue, 28 Jul 2026 00:00:00 GMT Four features landed in a single push: Lists (create/delete, member viewing, with its own backend models and router), an Explore page backed by a new trending-tweets endpoint, Bookmarks (per-tweet toggle plus a dedicated page), and Notifications (triggered by likes, retweets, comments, and follows) — replacing what had been placeholder "coming soon" pages. https://sacenpapier.org/updates/2026-07-11-poproom-park-generation https://sacenpapier.org/updates/2026-07-11-poproom-park-generation Sat, 11 Jul 2026 00:00:00 GMT Procedural park generation for PopRoom's canvas — decor rendering, boulders, and wildlife (rabbits and birds) scattered across the play space, plus improved WebSocket error handling and a proper launch configuration for running frontend and backend together. https://sacenpapier.org/updates/2026-07-09-ssh-hardening-and-alert-tuning https://sacenpapier.org/updates/2026-07-09-ssh-hardening-and-alert-tuning Thu, 09 Jul 2026 00:00:00 GMT A security pass across the homelab boxes' SSH configuration, plus a tuning fix on a false-alarm memory alert triggered by Watchtower's nightly auto-updates squeezing `glances`' memory limit. https://sacenpapier.org/updates/2026-07-09-botwhy-trending-moderation https://sacenpapier.org/updates/2026-07-09-botwhy-trending-moderation Thu, 09 Jul 2026 00:00:00 GMT Trending post reporting with author-only deletion, all trending posts surfaced in the admin moderation view, and a manual OpenRouter spend reconciliation tool in the admin panel. https://sacenpapier.org/updates/2026-07-06-botwhy-voice-mode https://sacenpapier.org/updates/2026-07-06-botwhy-voice-mode Mon, 06 Jul 2026 00:00:00 GMT Voice input and voice replies for BotWhy's chat: a `useSpeechRecognition` hook for talking to the bot, a `useSpeechSynthesis` hook for hearing its replies read aloud, a tri-state `VoiceModeButton`, and a per-message read-aloud button (replacing an earlier auto-speak-everything approach). https://sacenpapier.org/updates/2026-07-05-botwhy-admin-ui https://sacenpapier.org/updates/2026-07-05-botwhy-admin-ui Sun, 05 Jul 2026 00:00:00 GMT A full admin panel: user list and detail views, role management with a self-demotion guard, soft-delete and reactivate for accounts, a credit adjustment endpoint with an audit trail, and a transaction ledger view — plus the admin frontend routes and pages to use all of it. https://sacenpapier.org/updates/2026-07-03-botwhy-rate-limiting-the-ai-endpoint https://sacenpapier.org/updates/2026-07-03-botwhy-rate-limiting-the-ai-endpoint Fri, 03 Jul 2026 00:00:00 GMT A per-IP rate limit (20 requests/minute via `slowapi`) and a 500-character message length cap on BotWhy's `/openai/answer` endpoint — the one route that actually calls out to a paid AI provider on every request. https://sacenpapier.org/updates/2026-07-03-botwhy-mobile-viewport https://sacenpapier.org/updates/2026-07-03-botwhy-mobile-viewport Fri, 03 Jul 2026 00:00:00 GMT Eventually landed: a stable mobile layout where the chat input stays anchored at the bottom and the header stays sticky even when the on-screen keyboard opens. https://sacenpapier.org/updates/2026-07-01-botwhy-credit-payment-system https://sacenpapier.org/updates/2026-07-01-botwhy-credit-payment-system Wed, 01 Jul 2026 00:00:00 GMT A full pay-as-you-go credit system for BotWhy: Stripe checkout and webhooks, a credit ledger with transaction history, free credits on signup, a free-tier message allowance with a countdown, model restrictions for free-tier users, and a dedicated Credits page with pack selection and balance display. https://sacenpapier.org/updates/2026-05-23-homelab-pod-antiaffinity https://sacenpapier.org/updates/2026-05-23-homelab-pod-antiaffinity Sat, 23 May 2026 00:00:00 GMT Added `podAntiAffinity` to the homelab dashboard's frontend deployment, so its three replicas spread across different nodes instead of potentially landing on the same one. https://sacenpapier.org/updates/2026-05-02-homelab-dashboard-opnsense-page-and-windows-node-via-glances https://sacenpapier.org/updates/2026-05-02-homelab-dashboard-opnsense-page-and-windows-node-via-glances Sat, 02 May 2026 00:00:00 GMT Two small but real additions to the homelab dashboard: a dedicated `/opnsense` page describing the Protectli V1210 firewall's role and access, added with real client-side routing via a `Router` component; and a fix to how the Windows gaming node reports its stats, switching to the Glances HTTP API on its correct port. https://sacenpapier.org/updates/2026-04-20-homelab-pod-node-highlighting https://sacenpapier.org/updates/2026-04-20-homelab-pod-node-highlighting Mon, 20 Apr 2026 00:00:00 GMT The homelab dashboard's k3s section now shows which pod and node actually served the current request — visible directly on the page, with a rotate button to re-route and watch the highlight change — plus inline skeleton loading states across all live values instead of blank space while data loads, and light/dark mode. https://sacenpapier.org/updates/2026-03-24-botwhy-cookie-consent-and-trending-engagement https://sacenpapier.org/updates/2026-03-24-botwhy-cookie-consent-and-trending-engagement Tue, 24 Mar 2026 00:00:00 GMT A `CookieConsent` component for the frontend, and a real rewrite of how trending conversations track likes and comments: per-user like tracking (instead of a bare counter), unlike, and comment add/delete endpoints. https://sacenpapier.org/updates/2026-03-21-x-retweets-and-comment-overlays https://sacenpapier.org/updates/2026-03-21-x-retweets-and-comment-overlays Sat, 21 Mar 2026 00:00:00 GMT Retweets and a comment overlay landed on `TweetListUser`: a retweet label/action on each tweet, an overlay showing full tweet detail with comment submission and liking inside it, plus a new `Status` page for viewing an individual tweet and its comments directly, a `Coming Soon` page for unbuilt sections, empty-state handling in the Messages list, and a profile background picture. https://sacenpapier.org/updates/2026-03-21-x-rename-and-cookie-consent https://sacenpapier.org/updates/2026-03-21-x-rename-and-cookie-consent Sat, 21 Mar 2026 00:00:00 GMT Renamed the project from "twitterclone" to "x" across the README, HTML files, and deployment configuration, added a CookieBanner component, and wired up auto-logout on any 401/403 response from the backend. https://sacenpapier.org/updates/2026-03-14-apercu-event-platform https://sacenpapier.org/updates/2026-03-14-apercu-event-platform Sat, 14 Mar 2026 00:00:00 GMT The core event-organizing surface: an Organizer page for event management, a HostPage covering the hosting process and FAQs, a redesigned Speakers page with search and a hero section, and comments/likes on blog posts — plus the database columns needed to support all of it. https://sacenpapier.org/updates/2026-03-11-poproom-core-game https://sacenpapier.org/updates/2026-03-11-poproom-core-game Wed, 11 Mar 2026 00:00:00 GMT The foundational gameplay loop for PopRoom: camera follow and viewport adjustments, zoom and touch controls, stickman idle/movement animation, a WebSocket ping-pong mechanism to keep connections alive, and — a few days later — a janitor NPC that periodically cleans up the popped-balloon pile. https://sacenpapier.org/updates/2025-03-05-early-infrastructure-terraform-and-ansible-on-oracle-cloud https://sacenpapier.org/updates/2025-03-05-early-infrastructure-terraform-and-ansible-on-oracle-cloud Wed, 05 Mar 2025 00:00:00 GMT Before any of this became a k3s homelab, the fleet ran on Oracle Cloud free-tier instances provisioned by two Terraform repos — `oci-terraform-network` (VCN, subnet, security rules, an internet gateway, block storage) and `oci-terraform-cluster` (compute instances, including a later ARM instance) — with `oci-product-service`'s Ansible playbook handling the actual deployment: installing dependencies, running the Docker-based services, and updating Cloudflare DNS records on every deploy. https://sacenpapier.org/updates/2025-02-12-x-first-gitops-attempt-flux https://sacenpapier.org/updates/2025-02-12-x-first-gitops-attempt-flux Wed, 12 Feb 2025 00:00:00 GMT Added a `kustomization.yaml` for FluxCD to the `x` (then still called `twitterclone`) repo — the first attempt at GitOps on this infrastructure. https://sacenpapier.org/updates/2024-11-02-botwhy-first-cicd https://sacenpapier.org/updates/2024-11-02-botwhy-first-cicd Sat, 02 Nov 2024 00:00:00 GMT A working GitHub Actions deploy pipeline for BotWhy — reached via a long, visible trail of small failed commits over about a week: SSH connection failures, Docker build errors, `git pull` conflicts, React build issues, each fixed one at a time until the pipeline finally ran clean. https://sacenpapier.org/updates/2024-10-17-apercu-password-reset https://sacenpapier.org/updates/2024-10-17-apercu-password-reset Thu, 17 Oct 2024 00:00:00 GMT Forgot-password flow for Apercu's admin accounts, plus the email delivery path that sends the reset link. https://sacenpapier.org/updates/2024-09-16-apercu-bilingual-marketing-site-with-pricing-tiers https://sacenpapier.org/updates/2024-09-16-apercu-bilingual-marketing-site-with-pricing-tiers Mon, 16 Sep 2024 00:00:00 GMT A separate marketing front, `first-look`, sitting alongside the main app: a Service section, a Pricing section, and three service tiers (Basic, Intermediate, Advanced) each with their own page — all fully bilingual via `react-i18next`, with complete English and French translation files. https://sacenpapier.org/updates/2024-09-04-apercu-vite-migration https://sacenpapier.org/updates/2024-09-04-apercu-vite-migration Wed, 04 Sep 2024 00:00:00 GMT Moved Apercu's frontend off Create React App onto Vite, and renamed the project from a generic working title to Aperçu. https://sacenpapier.org/updates/2024-08-16-botwhy-building-the-core-chat-app https://sacenpapier.org/updates/2024-08-16-botwhy-building-the-core-chat-app Fri, 16 Aug 2024 00:00:00 GMT The founding build of BotWhy: a conversation/messages API, the first chatbox pages, request validation, a mobile-responsive layout, and the first Docker-based deployment — landing as one continuous week of work right after Google auth (and its [FedCM migration](/updates/2024-08-06-botwhy-fedcm-migration)) gave the app somewhere to attach a conversation to. https://sacenpapier.org/updates/2024-08-06-botwhy-fedcm-migration https://sacenpapier.org/updates/2024-08-06-botwhy-fedcm-migration Tue, 06 Aug 2024 00:00:00 GMT Migrated BotWhy's Google sign-in from the old Google Identity Services flow to FedCM (Federated Credential Management), the new browser-native standard for federated login. https://sacenpapier.org/updates/2024-06-04-apercu-building-the-core-app https://sacenpapier.org/updates/2024-06-04-apercu-building-the-core-app Tue, 04 Jun 2024 00:00:00 GMT The actual foundation of Apercu: a full rebuild in April 2024 from an earlier Create React App prototype onto Docker + React + FastAPI + MySQL, with a contact form that sends real email, an admin "agent" interface for managing the underlying data, password hashing, and a first (later superseded) forget-password flow. https://sacenpapier.org/updates/2024-05-28-x-building-the-core-app https://sacenpapier.org/updates/2024-05-28-x-building-the-core-app Tue, 28 May 2024 00:00:00 GMT The founding build of what would become x: a three-section layout, tweets (post/view), follow/following with a suggestions feed, per-tweet likes, profile editing, a seeded database for local dev, and the first Docker-based deployment — all landing as one continuous build over the project's first three weeks, alongside direct messages ([written up separately](/updates/2024-05-19-x-direct-messages)). https://sacenpapier.org/updates/2024-05-19-x-direct-messages https://sacenpapier.org/updates/2024-05-19-x-direct-messages Sun, 19 May 2024 00:00:00 GMT Direct messages between users: a `Message` model and FastAPI router (send, list by user, delete), a Messages page with a threaded `MessageList`, and a compose flow — `ComposeMessageForm` wired to the existing user search so a conversation starts by searching for someone rather than needing an existing thread first.