docs: refresh stale-grav-version doc — cross-link config-without-code sibling

ce-compound-refresh (scope: stale-grav-version-blocks-api-plugin-install). Update (not consolidate — distinct triggers, moderate overlap): add a Related back-link to the new grav-plugin-config-without-code-wont-enable sibling and broaden the "verify ls user/plugins/<name>" prevention bullet to note the config-without-code desync as a second non-functional-plugin case. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Np4cMQLF77i664CAQXySzU
docs: add local→test→prod deploy-cycle runbook
2026-07-05 10:44:13 +02:00 · 2026-07-05 10:40:04 +02:00 · 2026-07-05 10:34:39 +02:00 · 2026-07-05 10:19:54 +02:00 · 2026-07-05 00:16:33 +02:00 · 2026-07-05 00:05:11 +02:00
15 changed files with 1028 additions and 35 deletions
@@ -134,6 +134,8 @@ After updating, also create the new trip's page tree under `user/pages/01.trips/
 Always use `make` commands for anything on the production server (`make remote-install-plugins`, `make remote-clean`, etc.) — never SSH directly since credentials live in `.env`. If a remote operation isn't covered by an existing `make` command, either ask the user to run it manually or suggest adding a new `make` command if it seems reusable.
 For a full upgrade/deploy through local → test → prod (ordered steps, smoke checklist, rollback), follow the runbook at [`docs/guides/deploy-cycle.md`](docs/guides/deploy-cycle.md).
 ### Content sync
 - `make content-push` — commit and push `user/` to Gitea (triggers production pull via webhook)
@@ -171,15 +173,55 @@ Active settings in `user/config/system.yaml`:
 With these settings, Grav rebuilds templates on every request. This is intentionally slower but means you never need to flush cache after editing a `.html.twig` file.
-### Production mode (not yet configured)
+### Production mode (per-environment override)
-Before going live, change in `user/config/system.yaml`:
+Production needs different Twig settings than dev, but **never change the
 committed `user/config/system.yaml`** — `twig.cache: false` (and `debug`/
 `auto_reload: true`) are the *intended dev values*, and committing prod values
 there breaks local development for everyone.
-| Setting | Prod value | Why |
+Instead, prod values are a **per-environment override** deployed to the server
-|---|---|---|
+only, via Grav's per-environment config (`environment://config`, keyed on the
-| `twig.cache` | `true` | Templates compiled once and reused; safe because theme files don't change at runtime |
+request hostname):
-**Pre-launch smoke test required:** with `twig.cache: true`, submit one post via `/post` and confirm the entry appears in the trip page feed at `/trips/italy-2026-demo` immediately. This verifies the cache-on-save plugin (BUG-001 fix) works correctly with caching enabled.
+| Setting | Dev (committed) | Prod (override) | Why prod differs |
 |---|---|---|---|
 | `twig.cache` | `false` | `true` | Compile templates once and reuse |
 | `twig.debug` | `true` | `false` | No debug functions in prod |
 | `twig.auto_reload` | `true` | `false` | Don't stat templates every request |
 - **Source of truth:** `deploy/env/prod/system.yaml` (version-controlled).
 - **Deploy:** `make remote-apply-env-prod` — writes it to
  `<webroot>/user/env/<hostname>/config/system.yaml` and clears cache. It
  deep-merges over the committed `system.yaml`.
 - **Not synced by content:** `user/env/` is outside the content repo's tracked
  folders, so `content-push` / git-sync / `remote-fetch-content` do **not**
  restore it. **Re-run `make remote-apply-env-prod` after any fresh install.**
 - The hostname segment defaults to `REMOTE_HOST`; override with `WEB_HOST` in
  `.env.<env>` if Grav sees a different host than the SSH host.
 > **⚠️ Once `user/env/<hostname>/` exists, Grav's Admin saves ALL config there.**
 > Creating the env override dir has a site-wide side effect: Grav's Admin panel
 > writes **every** config change (system *and* plugin) into the active
 > environment's config tree — e.g. editing a plugin on prod saves to
 > `user/env/intotheeast.com/config/plugins/<name>.yaml`, **not**
 > `user/config/plugins/<name>.yaml`. Consequences you must remember:
 > - Config edited via **Admin on the server is server-only**: `user/env/` is
 >   outside the content repo's tracked folders, so it is **not committed** and
 >   **not synced by git-sync** (which syncs only `pages`/`config`/`themes`).
 >   Good for secrets — `git-sync.yaml` (token) safely lives at the env path —
 >   but it means prod Admin config edits silently do **not** reach Gitea/local.
 > - When reading/writing server config, check **both** `user/config/...` and
 >   `user/env/<host>/config/...` (env wins). Server tooling must search the env
 >   path first — see `scripts/git-sync-toggle.sh` and `make remote-diag`.
 > - Repo-authored config (`user/config/...` via `make content-push`) still
 >   applies everywhere; the env tree only holds per-host overrides + Admin-on-
 >   server edits. Full details: `docs/working/git-sync-notes.md`.
 **Pre-launch smoke test required:** with the prod override applied, submit one
 post via `/post` and confirm the entry appears in the trip page feed
 immediately. This verifies the cache-on-save plugin (BUG-001 fix) works
 correctly with caching enabled.
 ### What the cache-on-save plugin handles
@@ -14,6 +14,10 @@ REMOTE_PORT ?= 22
 SSH := ssh -p $(REMOTE_PORT) $(REMOTE_USER)@$(REMOTE_HOST)
 WEBROOT ?= $(REMOTE_HOME)/public_html
 SITE_CONFIG_DIR ?= $(REMOTE_HOME)/site-config
 # Hostname Grav uses to pick its per-environment config (user/env/<host>/).
 # Defaults to the SSH host; override in .env.<ENV> only if the web hostname
 # Grav sees differs from the SSH host (e.g. an addon domain on a shared box).
 WEB_HOST ?= $(REMOTE_HOST)
 # ── Environment guard + generated per-env remote targets ──────────────────────
 # Every remote-* target below gains `-test` / `-prod` variants, e.g.
@@ -22,7 +26,8 @@ SITE_CONFIG_DIR ?= $(REMOTE_HOME)/site-config
 REMOTE_TARGETS := remote-env-setup remote-env-remove remote-wipe remote-install \
 	remote-fetch remote-fetch-content remote-install-plugins remote-update-plugins \
 	remote-upgrade-grav remote-git-sync-disable remote-git-sync-enable \
-	remote-content-status remote-clean remote-maintenance-on remote-maintenance-off
+	remote-content-status remote-clean remote-diag remote-apply-env \
 	remote-gpm-install remote-maintenance-on remote-maintenance-off
 ENVS := test prod
 guard-env:
@@ -153,7 +158,7 @@ remote-fetch-content: guard-env
 	$(SSH) "git -C $(WEBROOT)/user fetch origin main && git -C $(WEBROOT)/user sparse-checkout disable && git -C $(WEBROOT)/user reset --hard origin/main"
 remote-install-plugins: guard-env
-	$(SSH) "cd $(WEBROOT) && php bin/gpm install $(shell cat plugins.txt | tr '\n' ' ') -y"
+	$(SSH) "cd $(WEBROOT) && php bin/gpm index -f && php bin/gpm install $(shell cat plugins.txt | tr '\n' ' ') -y"
 remote-update-plugins: guard-env
 	$(SSH) "cd $(WEBROOT) && php bin/gpm update -y && php bin/grav cache"
@@ -162,10 +167,10 @@ remote-upgrade-grav: guard-env
 	$(SSH) "cd $(WEBROOT) && php bin/gpm self-upgrade -y && php bin/grav cache"
 remote-git-sync-disable: guard-env
-	$(SSH) "bash -s -- '$(WEBROOT)/user/config/plugins/git-sync.yaml' false" < scripts/git-sync-toggle.sh
+	$(SSH) "bash -s -- '$(WEBROOT)' false" < scripts/git-sync-toggle.sh
 remote-git-sync-enable: guard-env
-	$(SSH) "bash -s -- '$(WEBROOT)/user/config/plugins/git-sync.yaml' true" < scripts/git-sync-toggle.sh
+	$(SSH) "bash -s -- '$(WEBROOT)' true" < scripts/git-sync-toggle.sh
 remote-content-status: guard-env
 	$(SSH) "cd $(WEBROOT)/user && git status --short && echo '--- config diff ---' && git diff -- config/"
@@ -173,6 +178,37 @@ remote-content-status: guard-env
 remote-clean: guard-env
 	$(SSH) "cd $(WEBROOT) && php bin/grav clearcache"
 # Install a single GPM package on the server (e.g. git-sync, which is
 # intentionally NOT in plugins.txt — it is remote-only).
 # Usage: make remote-gpm-install-prod PKG=git-sync
 remote-gpm-install: guard-env
 	@test -n "$(PKG)" || { echo "ERROR: set PKG=<plugin-slug>"; exit 1; }
 	$(SSH) "cd $(WEBROOT) && php bin/gpm index -f && php bin/gpm install $(PKG) -y && php bin/grav clearcache"
 # Deploy per-environment Grav config overrides to the server's
 # user/env/<WEB_HOST>/config/ tree (deep-merged over the committed config).
 # Source of truth: deploy/env/$(ENV)/system.yaml (version-controlled). This
 # tree is outside the content repo, so it is NOT restored by content sync —
 # re-run after any fresh install.
 remote-apply-env: guard-env
 	@test -f deploy/env/$(ENV)/system.yaml || { echo "ERROR: missing deploy/env/$(ENV)/system.yaml"; exit 1; }
 	@host="$${WEB_HOST:-$(REMOTE_HOST)}"; \
 	  test -n "$$host" || { echo "ERROR: WEB_HOST/REMOTE_HOST unresolved"; exit 1; }; \
 	  $(SSH) "mkdir -p $(WEBROOT)/user/env/$$host/config && cat > $(WEBROOT)/user/env/$$host/config/system.yaml && cd $(WEBROOT) && php bin/grav clearcache" < deploy/env/$(ENV)/system.yaml; \
 	  echo "Applied deploy/env/$(ENV)/system.yaml -> $(WEBROOT)/user/env/$$host/config/system.yaml"
 # Read-only health check: plugin install state, versions, key config, log tail.
 remote-diag: guard-env
 	$(SSH) "cd $(WEBROOT) && \
 		echo '=== Grav version ==='; php bin/grav --version 2>/dev/null; \
 		echo '=== installed plugin versions ==='; for p in login admin2 flex-objects form api; do printf '%s: ' \"\$$p\"; grep -m1 '^version:' user/plugins/\$$p/blueprints.yaml 2>/dev/null || echo '(NOT installed)'; done; \
 		echo '=== what does GPM say about api? ==='; php bin/gpm info api 2>&1 | head -12; \
 		echo '=== api override (enabled/route/session) ==='; grep -nE '^enabled:|^route:|session_enabled:' user/config/plugins/api.yaml 2>&1; \
 		echo '=== per-env override present? ==='; for f in user/env/*/config/system.yaml; do echo \"\$$f:\"; cat \"\$$f\" 2>/dev/null | grep -E 'cache:|debug:|auto_reload:'; done; \
 		echo '=== twig cache populating? (non-empty => cache on) ==='; ls cache/twig/ 2>/dev/null | head -1 || echo '(empty)'; \
 		echo '=== git-sync config (secrets redacted) ==='; grep -vaiE 'password|token|secret' user/config/plugins/git-sync.yaml user/env/*/config/plugins/git-sync.yaml 2>/dev/null; \
 		echo '=== grav.log tail ==='; tail -8 logs/grav.log 2>/dev/null"
 remote-maintenance-on: guard-env
 	$(SSH) "bash -s on $(WEBROOT)" < scripts/server-maintenance.sh
@@ -0,0 +1,38 @@
 # Production-only Grav config overrides.
 #
 # Deep-merged OVER the committed user/config/system.yaml via Grav's
 # per-environment config mechanism: on the server this file is deployed to
 #   <webroot>/user/env/<hostname>/config/system.yaml
 # and Grav's `environment://config` stream (keyed on the request hostname)
 # layers it on top of `user://config`.
 #
 # These values are deliberately NOT in the committed system.yaml because they
 # would break local development (see CLAUDE.md §1 — dev keeps twig.cache:false
 # so theme edits take effect immediately). Prod is the only place they apply.
 #
 # Deploy with:  make remote-apply-env-prod
 # The user/env/ tree is outside the content repo's tracked folders, so it is
 # NOT restored by content-push / git-sync / remote-fetch-content — re-run the
 # target above after any fresh install.
 twig:
  cache: true
  debug: false
  auto_reload: false
 # Compression / connection handling.
 #
 # This host is not FastCGI (no fastcgi_finish_request()), so Grav's shutdown
 # "early connection close" falls back to emitting `Content-Encoding: identity`
 # to ask the webserver not to compress. But Apache's mod_deflate compresses
 # anyway and adds `Content-Encoding: gzip`, giving TWO conflicting headers —
 # the browser can't decode the body and renders raw gzip bytes (a garbage
 # page). Note: allow_webserver_gzip:true takes the SAME identity branch, so it
 # does not help. The real fix is to disable the early-close path, so Grav never
 # emits the bogus header and mod_deflate compresses cleanly (single header).
 debugger:
  shutdown:
    close_connection: false
 # Let the webserver own gzip; Grav does not compress or double-label.
 cache:
  gzip: false
  allow_webserver_gzip: false
@@ -0,0 +1,133 @@
 # Upgrade & Deploy Cycle: local → test → prod
 This runbook is the repeatable procedure for shipping a Grav upgrade or any
 server-affecting change (core version, plugins, config, theme) through the three
 environments. It was distilled from the 2026-07 Grav 2.0.4→2.0.7 cutover, where
 every production surprise traced back to one of the desyncs this procedure now
 forces you to check.
 **Governing principle:** `test` is a **full dress rehearsal of `prod`** — same
 config, same `-test`/`-prod` make targets, same order. A gotcha only gets caught
 on test if test is a faithful mirror of prod. Do not shortcut test.
 All server operations go through `make remote-*` targets (never raw SSH — the
 targets build the SSH connection from `.env.<env>`, which must never be read
 directly). Every `remote-*` target has `-test` and `-prod` variants; a bare
 target fails via `guard-env`.
 ---
 ## The mental model: three places state lives
 Every failure in the reference cutover was a desync between these three layers.
 Before and after each deploy step, ask: *are they in sync?*
 | Layer | Location | Synced by | Failure mode |
 |---|---|---|---|
 | Plugin **code** | `user/plugins/<name>/` | GPM only (gitignored `/plugins/*`) | can vanish while config remains → plugin won't enable |
 | **Repo config** | `user/config/…` | `content-push` / git-sync | holds GPM channel + is where the version floor bites |
 | **Host config** | `user/env/<host>/config/…` | nothing — server-only | not restored on fresh install; must be re-applied; **must be gitignored** |
 Referenced gotcha docs:
 - `docs/solutions/integration-issues/grav-plugin-config-without-code-wont-enable.md` — code-vs-config desync.
 - `docs/solutions/integration-issues/stale-grav-version-blocks-api-plugin-install.md` — stale `GRAV_VERSION` / version floor.
 - `docs/solutions/architecture-patterns/git-sync-secret-exposure-and-tracked-file-boomerang.md` — gitignore is the sync boundary; env-tree leak.
 - `docs/solutions/conventions/grav-plugin-config-must-be-tracked-override.md` — plugin config must live in the tracked override.
 ---
 ## Phase 0 — Local (author + prove the change)
 1. Make the change in the repo:
   - GPM channel: `gpm.releases: stable` in `user/config/system.yaml` (authoritative; reaches servers via content pull, so it must be right **before** any server GPM op).
   - `plugins.txt` — the GPM-managed set only. **Never** add `git-sync` (it is remote-only).
   - Prod-only overrides (Twig cache/debug, `debugger.shutdown.close_connection: false`) in `deploy/env/prod/system.yaml` — **never** commit prod values into `user/config/system.yaml`.
   - **Bump `GRAV_VERSION` in `.env.test` and `.env.prod`** to the target version. A stale value here installs the wrong core (an rc), which then blocks the `api` plugin and 404s admin.
 2. `make build-assets` if you touched `js/src/*` (never hand-edit the bundled `js/*.js`).
 3. Run the dev server (`docker compose … up`) and the Playwright suite.
 4. Pre-flight assertions:
   - `gpm.releases` is `stable`.
   - `plugins.txt` is correct and does **not** contain `git-sync`.
   - No prod Twig values leaked into the committed `system.yaml`.
 5. Commit. `make content-push`.
 ---
 ## Phase 1 — Test (the rehearsal — catch things here)
 ### Pre-flight
 - `make remote-git-sync-disable-test` **before any content reset.** This is the safety catch for the whole window: it stops a half-migrated state (e.g. a fresh install-time `versions.yaml`) from auto-committing and pushing on the first sync.
 ### Apply — in this fixed order
 ```
 make remote-fetch-content-test              # 1. clean-reset synced folders to repo state
 make remote-upgrade-grav-test               # 2. gpm self-upgrade (rewrites schema — expect drift)
 make remote-update-plugins-test             # 3. gpm update the plugins.txt set
 make remote-gpm-install-test PKG=git-sync   # 4. EXPLICITLY (re)install each remote-only plugin
 make remote-apply-env-test                  # 5. re-deploy the env override (not synced; gone after install)
 ```
 Why each matters:
 - **Step 4** is non-optional even if git-sync "was already there" — remote-only plugins are not in `plugins.txt`, so nothing in steps 1–3 restores them. If the code is missing, the plugin is inert despite valid config.
 - **Step 5** re-writes `user/env/<host>/config/…` from `deploy/env/<env>/`. The env tree is not synced by anything, so a fresh install loses it until you re-apply.
 ### Verify (smoke checklist — this is the payoff)
 - **Code present, not just config:** `ls user/plugins/<name>/` for every expected plugin (especially `git-sync`). An empty/absent dir = reinstall (step 4). *(Do this via an ssh one-liner you run, or `make remote-diag-test`.)*
 - **HTTP:** `/` → 200, `/admin` → 200, `/api/v1/pages` → 401, `/gpx-manager` → 200. Watch for the double-`Content-Encoding` garbage page (fix: `debugger.shutdown.close_connection: false` in the env override — already in `deploy/env/prod/system.yaml`).
 - **Post smoke test:** submit one entry via `/post` and confirm it appears in the trip feed immediately. This proves the `cache-on-save` plugin works with prod caching on.
 - **Config drift:** `make remote-diag-test` — diff server config against the repo. Fold any *intended* schema migration (e.g. the Twig-3 `strict_mode` flags a `self-upgrade` writes) back into `user/config/system.yaml`, or the next `fetch-content` reverts it.
 ### Re-enable + prove sync
 - `make remote-git-sync-enable-test`.
 - Confirm a content push round-trips to the server, **and** that no secret/boomerang commit lands on Gitea. Verify `/env/` is gitignored so the env tree (which holds the token, JWT, CSRF salt) can never enter the sync add-set.
 ---
 ## Phase 2 — Prod (repeat identically — should be mechanical)
 Run the **exact same sequence** with `-prod` targets. Because test rehearsed it,
 prod holds no surprises. Differences to layer on:
 - Optional: `make remote-maintenance-on-prod` at the start, `remote-maintenance-off-prod` at the end, for a clean window.
 - Confirm secrets are valid/rotated and `/env/` is gitignored **before** `remote-git-sync-enable-prod`. Re-enable git-sync **last**.
 - After a clean cutover, bump the outer-repo submodule pin to the finished `user/` commit — and **push `user/` before the outer repo** (the superproject references a child SHA that must already exist upstream).
 ```
 make remote-git-sync-disable-prod
 make remote-fetch-content-prod
 make remote-upgrade-grav-prod
 make remote-update-plugins-prod
 make remote-gpm-install-prod PKG=git-sync
 make remote-apply-env-prod
 # ── smoke checklist (same as test) ──
 make remote-git-sync-enable-prod
 ```
 For a first-time / from-scratch prod bring-up, `make remote-install-prod` does the
 full install; then still run `remote-apply-env-prod` and the smoke checklist, and
 reinstall remote-only plugins explicitly.
 ---
 ## Rollback & safety
 - **git-sync stays disabled through the whole apply window** on each host — it is the catch that prevents a half-migrated state from auto-pushing.
 - **Content** is a git repo: a bad content deploy is recoverable with `make remote-fetch-content-<env>` back to a known commit.
 - **Core + plugins** are GPM-reinstallable (`remote-upgrade-grav`, `remote-update-plugins`, `remote-gpm-install PKG=…`).
 - The one thing tooling cannot regenerate is the un-synced `user/env/<host>/` tree — its source of truth is `deploy/env/<env>/`, so keep that current and re-apply with `remote-apply-env-<env>`.
 ---
 ## One-line invariants (the through-line)
 1. `test` is config-identical to `prod`, run with the same targets in the same order.
 2. Verify the **code layer** (`ls user/plugins/<name>/`), not just config, on every deploy.
 3. Reinstall **remote-only** plugins (git-sync) explicitly — nothing else restores them.
 4. `GRAV_VERSION` in `.env.<env>` and `gpm.releases: stable` are correct **before** any server GPM op.
 5. Re-apply the **env override** after every install; keep `/env/` **gitignored**.
 6. git-sync **off** during the window, **on** last; confirm the round-trip carries no secrets.
 7. Diagnose actual state before changing config — an `ls` or `remote-diag` beats a guess.
@@ -0,0 +1,243 @@
 ---
 title: "Secret exposure under bidirectional Grav git-sync: gitignore is the only boundary (and the tracked-file boomerang trap)"
 date: 2026-07-05
 module: git-sync
 problem_type: architecture_pattern
 component: tooling
 severity: high
 category: architecture-patterns
 applies_when:
  - "Enabling bidirectional Grav git-sync (direction: both, on_save: true) so prod can push content back to Gitea"
  - "Auditing whether the server can leak secrets (tokens, password hashes, signing salts) off the production host"
  - "A per-install runtime-generated value is being persisted into a tracked config file that also carries functional config"
  - "Deciding where a per-install secret must live so it never round-trips"
  - "A config value keeps ping-ponging or re-committing itself across environments after each sync"
 tags:
  - git-sync
  - secret-exposure
  - gitignore
  - grav
  - popularity-salt
  - config-boundary
  - bidirectional-sync
  - per-install-secret
  - env-tree-leak
 ---
 # Secret exposure under bidirectional Grav git-sync: gitignore is the only boundary (and the tracked-file boomerang trap)
 > **Correction (2026-07-05):** An earlier version of this doc claimed the sync
 > add-set was *scoped to the configured `folders`*, and concluded that
 > `accounts/` and `user/env/` were "safe by construction" because they sit
 > outside `pages/config/themes`. **That model is wrong and caused a live secret
 > leak.** git-sync's auto-commit stages files **outside** the configured folders;
 > the only reliable exclusion is `.gitignore`. The corrected model is below.
 ## Context
 The intotheeast.com Grav site runs the `git-sync` plugin on production in
 **bidirectional** mode: `direction: both`, `on_save: true`, a webhook at
 `/_git-sync`, and configured `folders: pages, config, themes`. Bidirectional
 means the server both *pulls* content authored elsewhere **and** *pushes*
 content authored on the server (Admin edits, `/post` submissions, uploads) back
 to the shared Gitea repo.
 The operator's question was: **"Can I safely enable bidirectional prod → Gitea
 sync without leaking secrets?"** Production holds things that must never reach a
 shared repo — API tokens, JWT signing secrets, CSRF salts, the git-sync token
 itself.
 The dangerous, tempting answer is "the plugin only syncs `pages/config/themes`,
 so anything outside those folders is safe." **This is false, and acting on it
 leaked secrets.** See the incident below.
 ## The incident (what actually happened)
 Production's git-sync auto-commit `9337003` ("(Grav GitSync) Automatic Commit
 ...") pushed the **entire `user/env/intotheeast.com/config/` tree** to Gitea —
 including `api-private.php` (JWT secret), `security-private.php` (CSRF salt), and
 `git-sync.yaml` (the sync **token + webhook secret**) — plus `accounts/mischa.yaml`
 (password hash), `system.yaml`, and `post-form.md`.
 **None of those paths is under the configured `folders: pages, config, themes`.**
 `user/env/` is a sibling of `user/config/`, not a subfolder of it. Yet git-sync
 staged and pushed them anyway. That single fact refutes the "folders scope the
 add-set" model empirically: **the `folders` setting does not scope what the
 auto-commit stages.** Whatever git-sync's exact `git add` invocation, the
 operational truth is that its commit sweeps the whole `user/` working tree.
 Remediation: disable sync → `.gitignore` `/env/` and `git rm --cached` it →
 `reset --hard` prod to the gitignored state → regenerate the leaked JWT/CSRF
 salts (delete the `*-private.php` files; Grav regenerates them) → **rotate the
 Gitea token and webhook secret** (they were exposed in `git-sync.yaml`). Rotation
 is what actually neutralizes the leak; the history rewrite is optional for a
 private repo.
 ## Guidance
 The safety question reduces to one predicate — but **not** the one the original
 doc used:
 > **A file round-trips to the shared repo if and only if it is NOT gitignored.**
 > The configured `folders` setting does **not** narrow this. Treat the
 > round-trippable set as *everything under `user/` that git will track* — i.e.
 > everything not matched by `user/.gitignore`.
 Consequences, corrected:
 1. **`.gitignore` is the only reliable boundary.** Do not rely on a file being
   "outside the synced folders." If it is under `user/` and not gitignored, a
   bidirectional sync can push it. Design exclusions with `.gitignore`, and
   verify with `git -C user status` / `git -C user check-ignore <path>`.
 2. **`accounts/` is NOT structurally excluded.** `user/accounts/*.yaml` (bcrypt
   password hashes) is a *tracked* content folder and is not gitignored, so it
   **does** round-trip — `accounts/mischa.yaml` was in the leak commit. If you
   need an account file to stay server-local, it must be gitignored explicitly
   (as `accounts/testrunner.yaml` already is). The earlier "password hashes never
   leave the server" claim was wrong.
 3. **The per-environment tree `user/env/<host>/` MUST be gitignored** — it is
   **not** inherently safe. It holds the live git-sync token, JWT secret, and
   CSRF salt (Grav writes all server-side Admin config there once the env dir
   exists). Because it is not under `user/config/` people assumed it was outside
   the sync scope; the incident proved it is not. It is now gitignored
   (`/env/` in `user/.gitignore`, commit `6e8eadb`). Keep it that way.
   > Side effect worth remembering: once `user/env/<host>/` exists, Grav's Admin
   > writes **all** config changes there (system and plugin), not into
   > `user/config/`. So server-side Admin edits are server-only — but "server-only"
   > now depends entirely on `/env/` being gitignored, not on folder scope. When
   > auditing, check **both** `user/config/...` and `user/env/<host>/config/...`
   > (env wins at runtime). See `docs/working/git-sync-notes.md`.
 4. **Per-install secrets go in gitignored companion files.** Grav's convention
   splits a per-install secret out of the functional YAML into a sibling that is
   gitignored: the JWT secret in `api-private.php`, the CSRF/nonce + rate-limit
   salt in `security-private.php`, plus `security.yaml` and `versions.yaml`.
   These are safe **because they are gitignored**, not because of where they sit.
 Run every secret through predicate #1 (is it gitignored?) and the answer falls
 out — but you must actually enumerate what is *not* gitignored, not what is
 "outside the folders."
 ### The tracked-file boomerang (a separate trap)
 Independently of the folder-scope error above, there is a second trap that the
 original doc got right and that still holds: **a per-install value that a plugin
 regenerates at runtime and persists into a tracked, functional config file.**
 The concrete case: the `api` plugin's *popularity* feature generates
 `popularity.salt` and writes it **into `user/config/plugins/api.yaml`** — a file
 that also carries must-be-shared functional config. That file is tracked, so on
 prod the popularity feature regenerates the salt, git-sync stages the change,
 commits, and **pushes prod's salt back to the shared repo**. Another environment
 pulls it, regenerates *its own* salt, pushes again. The value **ping-pongs across
 installs**, producing endless noise commits.
 The critical realization: **you cannot gitignore a single key inside a file that
 also carries functional config.** `.gitignore` operates on whole files. `api.yaml`
 must be tracked because the rest of it must be shared; therefore the salt inside
 it is tracked too; therefore it boomerangs.
 ### The rules
 For any per-install runtime-generated value, pick one of exactly three
 resolutions — and do **not** reach for the fourth (stripping the line), which
 cannot work under sync:
 - **Isolate the value into a gitignored companion `<name>-private.php`** — the
  pattern Grav uses for the JWT secret via `api-private.php`.
 - **Disable the feature that generates it** (e.g. `popularity.enabled: false`).
 - **Consciously accept the churn** when the value is genuinely low-stakes
  (`popularity.salt` is an IP-hashing salt, not a credential).
 Do **not** keep stripping the value from the tracked file — bidirectional sync
 brings it right back on the next save.
 ### Before / after: what sticks and what doesn't
 **A standalone file gitignored + untracked sticks.** For `security-private.php`
 (a file that contains *only* the secret) or the whole `user/env/` tree:
 ```bash
 git -C user rm --cached -r config/security-private.php   # or: rm --cached -r env
 printf '/config/security-private.php\n/env/\n' >> user/.gitignore
 ```
 This works permanently. The path is no longer tracked, so the sync's add-set
 skips it forever. The fix sticks because the secret owns its own gitignored path.
 **A key inside a tracked functional file — stripping the line does NOT stick.**
 For `popularity.salt` inside `api.yaml`, deleting just the `salt:` line and
 committing looks clean locally, but Grav re-appends it at runtime and the next
 sync re-commits and re-pushes it. The only durable fixes are the
 companion-private-file pattern or disabling the feature.
 ## Why This Matters
 Two quiet, cross-environmental failure modes:
 1. **The folder-scope illusion.** Assuming "only `pages/config/themes` sync" is a
   security control leads you to leave secrets in `env/` or `accounts/` unignored
   — and a single Admin save on prod pushes them to a shared repo. This actually
   happened here. The only defensible mental model is *gitignore is the boundary*;
   enumerate the un-ignored set, not the "un-foldered" set.
 2. **The boomerang.** A "cleanup" commit that strips a secret from a *tracked*
   file looks done locally but silently reappears upstream on the next content
   save, because the plugin regenerates it and the sync re-commits it.
 Getting both right is what lets you answer "is bidirectional sync safe?" honestly.
 The answer is **yes, once `user/.gitignore` actually excludes every sensitive
 path** — `env/`, the per-install `*-private.php` files, `security.yaml`,
 `versions.yaml`, and any account file that must stay server-local — and once every
 runtime-regenerated value either lives in its own gitignored file or is a
 consciously-accepted low-stakes churn. It is emphatically **not** safe on the
 strength of folder scoping alone.
 ## When to Apply
 - **Enabling or auditing bidirectional git-sync** on a server that authors
  content. Enumerate the round-trippable set as *everything under `user/` not
  matched by `.gitignore`* — then confirm no secret is in it.
 - **Deciding where a new secret or per-install generated value should live.**
  Standalone gitignored file for anything sensitive; never a key inside a shared
  functional YAML; never "outside the folders" as the sole justification.
 - **Reviewing a "stop tracking this secret" cleanup** for whether it will stick:
  is the secret in its own gitignored path (holds) or a line inside a tracked
  functional file that something regenerates (boomerangs)?
 ## Examples
 **The leak (what "folder scope is safe" cost).** With `folders: pages, config,
 themes` and `/env/` **not** gitignored, prod's auto-commit `9337003` pushed
 `user/env/intotheeast.com/config/**` (JWT, CSRF salt, git-sync token + webhook
 secret), `accounts/mischa.yaml`, `system.yaml`, and `post-form.md` to Gitea —
 all outside the configured folders. Fix: gitignore + untrack `/env/`, regenerate
 the JWT/CSRF salts, **rotate the token and webhook secret**.
 **Safe after remediation.** Same bidirectional config, but now `user/.gitignore`
 excludes `/env/`, `config/plugins/git-sync.yaml`, `config/plugins/api-private.php`,
 `config/security.yaml`, `config/security-private.php`, `config/versions.yaml`.
 Running each secret through *is-it-gitignored*: all sensitive paths are excluded →
 none is in the round-trippable set. Verified: prod's `git status` shows only the
 intended tracked content, and no boomerang/secret commit lands on the remote.
 **Boomerang example.** `popularity.salt` in the tracked `api.yaml` regenerates
 per-install and re-commits under sync. The fix that *sticks* is the
 companion-private-file pattern or `popularity.enabled: false` — **not** stripping
 the `salt:` line.
 ## Related
 - `docs/solutions/conventions/grav-plugin-config-must-be-tracked-override.md` — the
  origin of the "functional plugin config in tracked `user/config/plugins/`,
  secrets/per-install values in gitignored `*-private.php`" rule. That doc covers
  *where config must live to deploy*; this doc covers *why gitignore — not folder
  scope — is the sync boundary, and why a runtime-written tracked value boomerangs*.
 - `docs/working/git-sync-notes.md` — operational notes on git-sync's synced
  folders and the per-environment tree. Corrected in the same 2026-07-05 pass to
  drop the "env/ is outside the sync scope so it's safe" claim.
 - `docs/solutions/integration-issues/stale-grav-version-blocks-api-plugin-install.md`
  — adjacent context from the same Grav production cutover, different failure mode.
@@ -0,0 +1,86 @@
 ---
 title: "Grav plugin config must live in the tracked user/config/plugins/ override, not the plugin folder"
 date: 2026-07-04
 category: docs/solutions/conventions
 module: "grav / plugin configuration"
 problem_type: convention
 component: tooling
 severity: high
 applies_when:
  - "Editing functional config for any GPM-managed Grav plugin"
  - "user/plugins/ is gitignored and only pages/config/accounts/themes are tracked"
  - "Preparing a fresh install or production cutover"
  - "A plugin behaves correctly locally but ships with only default config on deploy"
 related_components:
  - "grav"
  - "gpm"
  - "api plugin"
  - "content repo"
  - "deployment"
 tags:
  - grav
  - plugin-config
  - gpm
  - config-override
  - gitignore
  - deployment
  - api-plugin
 ---
 # Grav plugin config must live in the tracked user/config/plugins/ override, not the plugin folder
 ## Context
 Grav resolves a plugin's config by deep-merging two layers: the plugin's own shipped file `user/plugins/<name>/<name>.yaml` (installed by GPM, part of the package) and the tracked override `user/config/plugins/<name>.yaml` (which wins). In this project the content repo tracks only `pages/`, `config/`, `accounts/`, `themes/`; `user/plugins/` and `user/data/` are gitignored (GPM manages plugin *code*). So any functional config a developer edits into a plugin's own `user/plugins/<name>/<name>.yaml` is invisible to version control.
 It was this gap that left the `api` plugin unconfigured on the fresh prod install. Its `enabled`/`route`/`session_enabled`/cors/rate_limit config existed only in the untracked plugin folder locally, while the committed `user/config/plugins/api.yaml` held only a runtime `popularity.salt`. The local machine worked because the plugin folder had been hand-edited; every fresh environment got only the plugin's shipped defaults.
 ## Guidance
 Put **functional** plugin configuration in the TRACKED override `user/config/plugins/<name>.yaml`. Grav deep-merges it over the plugin's shipped defaults, so it need only carry the keys that must differ (or the full config, for clarity). Keep **secrets and per-install generated values** OUT of the tracked file — JWT secrets, salts, encrypted tokens belong in gitignored `*-private.php` companion files (e.g. `api-private.php`, `security-private.php`) or should be regenerated per-install.
 Never rely on edits to the plugin's own `user/plugins/<name>/<name>.yaml`: it is gitignored (won't deploy) and is overwritten on the next `php bin/gpm update`.
 Concrete before/after, using the `api` plugin:
 ```yaml
 # WRONG: user/plugins/api/api.yaml  (gitignored, GPM-managed, wiped on update)
 enabled: true
 route: /api
 auth:
  session_enabled: true
 ```
 ```yaml
 # RIGHT: user/config/plugins/api.yaml  (tracked, deploys, survives gpm update)
 enabled: true
 route: /api
 version_prefix: v1
 auth:
  session_enabled: true
  # JWT secret intentionally NOT here — it lives in the gitignored api-private.php
 ```
 ## Why This Matters
 Reproducible deploys: a fresh clone or `make remote-install-<env>` must produce a working site from the repo alone. Config stranded in the gitignored plugin folder silently yields a plugin with only its shipped defaults on every new environment — which, for a plugin whose behavior depends on non-default config, means it's misconfigured or effectively off. On prod the `api` plugin's route/auth simply didn't work.
 The failure is silent and per-environment: it works on the developer's machine (where the plugin folder was hand-edited) and breaks everywhere else. `gpm update` compounds it by wiping the folder edit even locally, so the "working" state is not just unshared — it is also unstable on the one machine that had it.
 ## When to Apply
 - Any time you configure a Grav plugin whose non-default settings must work on a server (prod/test) or survive a plugin update.
 - Especially for plugins whose function depends on config: `api` (route/auth/cors), `admin2`, `flex-objects`, form/media settings, etc.
 - When auditing a fresh-install failure: check whether the "working" local config actually lives in a tracked path (`git ls-files user/config/plugins/<name>.yaml`) or was stranded in `user/plugins/<name>/`.
 ## Examples
 - **api plugin (this project):** the functional config was moved from the untracked `user/plugins/api/api.yaml` into the tracked `user/config/plugins/api.yaml`, then `make content-push` + `make remote-fetch-content-<env>` deployed it. The JWT secret stayed in the gitignored `api-private.php`.
 - **Quick audit command:** `git -C user ls-files config/plugins/` shows exactly which plugin configs are tracked/deployable; anything you rely on that isn't listed is a latent fresh-install failure.
 ## Related
 - `docs/solutions/integration-issues/stale-grav-version-blocks-api-plugin-install.md` — the config gap documented here was the *other* latent problem surfaced in that same investigation: the `api` plugin also had to be *installed* first before any config could take effect. The install gap (GPM version floor) and this config-tracking gap compounded each other on the fresh prod environment.
 - `docs/working/git-sync-notes.md` — the related third config location: on prod, Grav Admin saves config into the per-environment tree `user/env/<host>/config/`, which is *also* untracked. Same "config that doesn't reach the repo" family.
 - `docs/solutions/architecture-patterns/git-sync-secret-exposure-and-tracked-file-boomerang.md` — the sync-boomerang consequence of this rule: a per-install value that a plugin regenerates into a *tracked* functional config file (e.g. `popularity.salt` in `api.yaml`) re-commits itself and ping-pongs across environments under bidirectional git-sync. The `*-private.php` companion pattern this doc establishes is exactly the durable fix.
 - **CLAUDE.md §0 (plugin-management model):** only `pages/`, `config/`, `accounts/`, `themes/` are tracked in the `user/` repo; `plugins/` and `data/` are gitignored and GPM-managed. That tracking boundary is exactly why functional config must live under `config/plugins/`, not in the plugin's own folder.
@@ -0,0 +1,123 @@
 ---
 title: "Grav garbage page from two conflicting Content-Encoding headers (identity + gzip)"
 date: 2026-07-04
 category: docs/solutions/integration-issues
 module: "grav / production deploy"
 problem_type: integration_issue
 component: tooling
 severity: high
 symptoms:
  - "Homepage and every dynamic Grav page render as full-screen binary/mojibake garbage in the browser"
  - "curl without Accept-Encoding returns clean HTML, so the page looks fine from a naive curl"
  - "curl -H \"Accept-Encoding: gzip\" (a browser-style request) returns raw gzip bytes"
  - "Response carries two conflicting headers: content-encoding: identity AND content-encoding: gzip"
  - "Only appeared after switching prod to production Twig mode (twig.debug: false)"
 root_cause: config_error
 resolution_type: config_change
 related_components:
  - "Apache mod_deflate"
  - "DirectAdmin shared host"
  - "Grav shutdown handler (system/src/Grav/Common/Grav.php)"
  - "deploy/env/prod/system.yaml"
  - "make remote-apply-env-prod"
 tags:
  - grav
  - content-encoding
  - gzip
  - mod-deflate
  - fastcgi-finish-request
  - apache
  - production-deploy
  - twig-debug
 ---
 # Grav garbage page from two conflicting Content-Encoding headers (identity + gzip)
 ## Problem
 Grav renders every dynamic page as binary garbage in the browser because it emits two conflicting `Content-Encoding` headers. The response body is valid gzip, but because the server advertises both `content-encoding: identity` and `content-encoding: gzip`, the browser cannot decide how (or whether) to inflate it, and paints the raw compressed bytes to screen. Static assets are unaffected — only Grav's own dynamically generated pages are broken. The problem surfaced only after switching the site to production Twig mode (`twig.debug: false`).
 ## Symptoms
 - The homepage and all dynamic Grav pages show a full screen of binary/mojibake characters in the browser (completely unreadable). The surrounding HTML shell and static assets are fine.
 - A naive `curl https://site/` (with **no** `Accept-Encoding` header) returns clean, correct HTML — so a quick curl sanity check looks perfectly healthy and completely hides the bug.
 - A browser-style request exposes it. `curl -H "Accept-Encoding: gzip" -D - -o /dev/null https://site/` shows **two** `Content-Encoding` response headers:
  ```
  content-encoding: identity
  content-encoding: gzip
  ```
  The body is valid gzip and `gunzip`s to the correct HTML.
 - A static asset served by the webserver alone (e.g. a CSS/JS file) shows a **single** clean `content-encoding: gzip` under the same request — confirming the webserver's gzip is fine and the duplication is Grav-originated.
 - The bug only appeared after switching the site to production Twig mode (`twig.debug: false`), which activates Grav's full shutdown/output path.
 ## What Didn't Work
 - **First fix attempt: `cache.gzip: false` + `allow_webserver_gzip: true`.** This was the key dead end. It had **no effect** — the duplicated headers were unchanged. Reading Grav's source explained why: the branch that emits the bogus header fires on `if ($config->get('system.cache.gzip') || $config->get('system.cache.allow_webserver_gzip'))`. Setting `allow_webserver_gzip: true` satisfies the **same** `||` condition, so Grav still takes the identical `header('Content-Encoding: identity')` code path. The two knobs that *look* like they control this are both on the wrong side of the problem.
 - **Verifying with a naive `curl` (no `Accept-Encoding: gzip`).** This hid the problem entirely, because the server only compresses when the client advertises gzip support. Any healthcheck that omits `Accept-Encoding: gzip` reports a false "all clear." Reproduce it the way a browser does — send `Accept-Encoding: gzip`, or take a real headless-browser (Playwright) screenshot.
 ## Solution
 Root the fix in Grav's shutdown handler, `system/src/Grav/Common/Grav.php` (~lines 615–631):
 ```php
 if ($config->get('system.debugger.shutdown.close_connection', true)) {
    $success = function_exists('fastcgi_finish_request') ? @fastcgi_finish_request() : false;
    if (!$success) {
        if (!ini_get('zlib.output_compression')) {
            if ($config->get('system.cache.gzip') || $config->get('system.cache.allow_webserver_gzip')) {
                header('Content-Encoding: identity');   // <-- the bogus header
            } elseif (function_exists('apache_setenv')) {
                @apache_setenv('no-gzip', '1');
            } else {
                header('Content-Encoding: none');
            }
            header('Content-Length: ' . ob_get_length());
        }
        header('Connection: close');
        ob_end_flush();
    }
 }
 ```
 The entire problematic block is gated by `system.debugger.shutdown.close_connection` (default `true`). Disable it so the whole branch is skipped and Grav never touches `Content-Encoding` at all.
 In this project it is applied as a **per-environment (prod-only) override** so local dev is untouched — `deploy/env/prod/system.yaml`, deployed via `make remote-apply-env-prod`:
 ```yaml
 debugger:
  shutdown:
    close_connection: false
 ```
 Verify — the response must show **exactly one** `content-encoding`:
 ```bash
 curl -s -D - -o /dev/null -H "Accept-Encoding: gzip" https://site/ | grep -i content-encoding
 # content-encoding: gzip
 ```
 Then take a screenshot of the rendered page to confirm it displays correctly. A `curl`-without-gzip check is **not** sufficient proof — it would have passed even while the bug was live.
 ## Why This Works
 `debugger.shutdown.close_connection` (default `true`) makes Grav flush the full response and close the connection to the browser **early**, so slow shutdown tasks (logging, debugger teardown) don't keep the visitor waiting. On a FastCGI/PHP-FPM host, Grav does this cleanly via `fastcgi_finish_request()` and never manipulates headers — which is why the bug is invisible on most stacks.
 On a **non-FastCGI** host (LiteSpeed, suPHP, plain CGI), `fastcgi_finish_request()` does not exist, so `$success` is `false` and Grav falls back to closing the connection *manually*. To do that it must set an explicit `Content-Length`, and to keep that length honest it tries to tell the webserver "do not compress this body" — which, on the `cache.gzip`/`allow_webserver_gzip` branch, it expresses as `header('Content-Encoding: identity')`.
 But `identity` is not a real content transformation and is **not** a recognized "suppress compression" signal to Apache `mod_deflate`. `mod_deflate` ignores it, compresses the body anyway, and appends its **own** `Content-Encoding: gzip`. The response now carries two contradictory `Content-Encoding` headers (`identity` and `gzip`). Browsers cannot reconcile the contradiction, fail to inflate the gzip stream, and render the raw compressed bytes — the "binary garbage" screen.
 Setting `close_connection: false` means Grav never enters the manual connection-close path, never emits `Content-Encoding: identity`, and leaves the webserver as the **sole** authority on compression. The webserver then sends a single, correct `Content-Encoding: gzip`, and the browser inflates and renders normally.
 ## Prevention
 - On **non-FastCGI PHP hosts with server-side gzip** (Apache `mod_deflate`, LiteSpeed), set `debugger.shutdown.close_connection: false` for that environment. Deliver it as a **per-environment override**, never by editing the committed `system.yaml` (which would silently change dev behavior too).
 - **Reproduce compression bugs the way a browser sees them.** Always test with `curl -H "Accept-Encoding: gzip" -D -` and/or a headless-browser screenshot. A plain `curl` negotiates no compression and silently masks encoding bugs.
 - **Health check:** dynamic pages must return **exactly one** `Content-Encoding` header. Two of them (`identity` + `gzip`) is the unambiguous signature of this bug. Add this assertion to any smoke test.
 - **Know the trigger.** This can stay completely hidden in development mode and only appear once a site is switched to production mode (`twig.debug: false`), which activates Grav's full shutdown/output path. Re-run the browser-style compression check as part of any production cutover.
 ## Related
 - **Grav per-environment override mechanism** — `deploy/env/prod/system.yaml` applied via `make remote-apply-env-prod`, described in `CLAUDE.md` §1 ("Production mode — per-environment override"). The pattern that lets a prod-only setting like `debugger.shutdown.close_connection: false` ship without mutating the committed dev `system.yaml`.
 - **`docs/working/git-sync-notes.md`** — documents the `user/env/<hostname>/config/` override tree (where this fix physically lives on the server) and the caveat that config saved via Admin on the server stays server-only.
 - **`docs/solutions/integration-issues/stale-grav-version-blocks-api-plugin-install.md`** — sibling from the same cutover: Admin2 login failed because a stale `GRAV_VERSION` installed an rc core and GPM wouldn't serve the `api` plugin. Different root cause, same deploy.
 - **`docs/solutions/test-failures/new-user-grants-api-not-admin-on-admin2.md`** — a sibling gotcha from the same 2026-07-04 Grav 2.0.4 production cutover (account permission provisioning); different root cause, same deploy.
@@ -0,0 +1,100 @@
 ---
 title: "Grav plugin won't enable because its code is missing while its config persists in the env tree"
 date: 2026-07-05
 category: integration-issues
 module: git-sync
 problem_type: integration_issue
 component: tooling
 severity: high
 symptoms:
  - "git-sync plugin will not enable on prod despite enabled: true in its config"
  - "Plugin does not appear / cannot be toggled on in the Grav Admin UI"
  - "Config-level fix attempts (editing plugin YAML) have no effect"
  - "make remote-gpm-install-prod PKG=git-sync reports a FRESH install, not 'already installed'"
 root_cause: incomplete_setup
 resolution_type: dependency_update
 related_components:
  - documentation
  - development_workflow
 tags:
  - grav
  - git-sync
  - gpm
  - plugin-management
  - env-config
  - config-without-code
  - troubleshooting-order
  - remote-only-plugin
 ---
 # Grav plugin won't enable because its code is missing while its config persists in the env tree
 ## Problem
 After remediating an unrelated git-sync secret leak on prod (intotheeast.com, Grav 2.0.7 on a DirectAdmin/Apache shared host), the operator went to re-enable the `git-sync` plugin. Setting `enabled: true` in its config had no effect: the plugin would not appear as enabled in the Grav Admin plugins UI, and toggling it on manually in the browser did not take either. It looked fully "configured" — the config file was right there — but the plugin was inert and sync never ran.
 The trap is that the plugin's config file existed (in the per-host env tree at `user/env/intotheeast.com/config/plugins/git-sync.yaml`), so any inspection that reads only config concluded the plugin was present and just needed enabling. The actual problem was one layer down: the plugin's **code** was missing from `user/plugins/git-sync/`. A Grav plugin cannot load or enable without its code on disk, no matter what its config says.
 ## Symptoms
 - The git-sync plugin does not appear as enabled, and cannot be enabled, in the Grav Admin UI — despite `enabled: true` being present in its config.
 - Toggling `enabled` in config, or flipping the toggle in the Admin UI, produces no working plugin. Sync does not run.
 - The plugin's config file DOES exist (in the per-environment tree `user/env/intotheeast.com/config/plugins/git-sync.yaml`), so the plugin appears "present" whenever only the config is inspected — masking the real state.
 ## What Didn't Work
 1. **Setting / ensuring `enabled: true` in the git-sync config.** No effect. Config was never the problem.
 2. **Enabling the plugin manually in the Admin UI.** The toggle wouldn't take.
 Both failed attempts operate on the **config** layer. But the plugin's **code** was absent from `user/plugins/git-sync/`, and Grav can't load a plugin without its code. Grav (and any tooling that reads the config tree) reports a plugin as "configured" purely from the presence of its config file, which masks the absence of code. Diagnosing and poking at the config layer could never fix a missing-code problem — and guessing at config changes before running a simple `ls` on the plugin directory cost real time here.
 ## Solution
 First, run the decisive diagnostic on the server — confirm whether the plugin code actually exists *before* touching config:
 ```
 ls -la $WEBROOT/user/plugins/git-sync/    # empty/absent  => missing code, reinstall
 ```
 (In this project, do that via a make target or an ssh one-liner the user runs — never raw SSH by the assistant. All server ops go through `make remote-*`.)
 With the directory confirmed empty/absent, reinstall the plugin's code via GPM:
 ```
 make remote-gpm-install-prod PKG=git-sync   # GPM fresh-installs Git Sync v3.4.4
 ```
 That make target runs, on the server:
 ```
 php bin/gpm index -f && php bin/gpm install git-sync -y && php bin/grav clearcache
 ```
 The install output read **"Preparing to install Git Sync [v3.4.4] ... Success!"** — a **fresh** install, not "already installed." That fresh-install line is exactly what confirmed the code had been absent all along. After the reinstall plus cache clear, the plugin enabled and sync worked.
 ## Why This Works
 Grav resolves a plugin from **two independent locations**:
 - **Code** at `user/plugins/<name>/` — installed by GPM. Note `user/plugins/` is gitignored (`/plugins/*`) and is NOT tracked by the content repo.
 - **Config** — the tracked `user/config/plugins/<name>.yaml` and/or the per-host `user/env/<host>/config/plugins/<name>.yaml`.
 These two can **desync**: config can exist with no code behind it. Config alone makes the plugin look present to any tool that only reads config, but the plugin stays inert until its code is on disk. GPM reinstall restores the code; `clearcache` makes Grav re-scan and pick it up.
 A project-specific amplifier made this worse: `git-sync` is a **remote-only, GPM-managed** plugin. It is deliberately NOT in `plugins.txt`, so `make install-plugins` and the normal `make remote-install` flow do **not** restore it. Only an explicit `php bin/gpm install git-sync` (via `make remote-gpm-install-prod PKG=git-sync`) does. So when its code goes missing, it does not self-heal through the standard install path — you must reinstall it explicitly.
 How the code went missing here is **unconfirmed**. It happened around the git-sync secret-leak remediation, but the exact step that wiped `user/plugins/git-sync/` was not established — don't assume a specific cause.
 ## Prevention
 - **Check the code layer before the config layer.** When a Grav plugin "won't enable" and config toggles do nothing, FIRST verify the code exists: `ls user/plugins/<name>/` on the server. Config-without-code is the failure class; the empty directory is the tell.
 - **Enumerate both layers in all locations when diagnosing.** Plugins have a code layer (`user/plugins/<name>/`) and a config layer, and on prod the config can live in the env tree (`user/env/<host>/config/plugins/<name>.yaml`) and persist completely independently of the code. Remember: once `user/env/<host>/` exists, Grav Admin writes ALL config there, so always check both `user/config/...` and the env path (env wins).
 - **Know which plugins are remote-only.** The 3-category model: GPM-via-`plugins.txt` (admin2 / api / flex-objects), custom-in-repo (cache-on-save / story-blocks), and remote-only (git-sync — never in `plugins.txt`). Remote-only plugins are NOT restored by the standard install/content flows, so reinstall them explicitly via GPM after any operation that could have wiped `user/plugins/`.
 - **Diagnose actual state before proposing config fixes.** An `ls` is cheaper than a guess. Establishing that the code was missing would have pointed straight at the reinstall instead of a round of config poking.
 ## Related
 - `docs/solutions/integration-issues/stale-grav-version-blocks-api-plugin-install.md` — **closest sibling.** Same family (a plugin non-functional on prod, resolved by a GPM install + cache clear), same 2026-07-04/05 cutover context, same `plugins.txt` / `make remote-*` / GPM machinery. **Distinct trigger:** there, GPM refuses to *offer* the plugin because the installed core is below the version floor; here, the plugin's *code folder is simply missing* while its config persists (config-without-code desync). Two different ways a plugin ends up absent/inert on prod.
 - `docs/solutions/architecture-patterns/git-sync-secret-exposure-and-tracked-file-boomerang.md` — same module (git-sync) and explains **why the config survived without code**: Grav Admin writes `git-sync.yaml` into the per-environment tree `user/env/<host>/config/plugins/`, which is untracked/gitignored and not part of the plugin package. The orphaned config here is the flip side of that env-tree behavior.
 - `docs/solutions/conventions/grav-plugin-config-must-be-tracked-override.md` — establishes the plugin **code (GPM/gitignored) vs config (tracked override / env tree)** split that this bug exploits. This doc is a concrete failure of that split going the other way: config present (in the env tree), code absent.
 - `docs/working/git-sync-notes.md` — operational notes on git-sync's per-environment tree, where `git-sync.yaml` lives server-only. Context for where the orphaned config resided.
@@ -0,0 +1,98 @@
 ---
 title: "Admin2 login fails silently because a stale GRAV_VERSION installs an rc core below the api plugin's version floor"
 date: 2026-07-04
 category: docs/solutions/integration-issues
 module: "grav / production deploy / plugin install"
 problem_type: integration_issue
 component: authentication
 severity: high
 symptoms:
  - "Admin2 login at /admin silently fails: button disables then re-enables, no visible error, nothing written to grav.log"
  - "admin2 SPA background login POST to /api/... returns 404"
  - "/api/v1/pages returns 404 on prod but 401 locally (api plugin route not registered)"
  - "user/plugins/api directory does not exist on prod (plugin never installed)"
  - "gpm install api reports 'These packages were not found on Grav: api' even after gpm index -f"
 root_cause: config_error
 resolution_type: environment_setup
 related_components:
  - "gpm"
  - "admin2 plugin"
  - "api plugin"
  - "scripts/server-install.sh"
  - "Makefile remote targets"
  - ".env.prod"
 tags:
  - grav
  - gpm
  - admin2
  - api-plugin
  - plugin-dependency
  - version-compatibility
  - production-deploy
  - env-config
 ---
 # Admin2 login fails silently because a stale GRAV_VERSION installs an rc core below the api plugin's version floor
 ## Problem
 On a fresh Grav production install, Admin2 login fails silently because the `api` plugin — which Admin2 authenticates through — never installed. GPM refused to serve it: a stale `GRAV_VERSION` in `.env.prod` had installed Grav `2.0.0-rc.10`, and the `api` plugin requires Grav core `>=2.0.4`. GPM filters offered packages by the installed core version, so on an rc.10 core the `api` plugin was excluded from results entirely and reported as "not found." Admin2 was present (and depends on `api`), but its login POST hit an `/api/...` route that was never registered, so authentication silently 404'd before it ever reached Grav's auth layer.
 ## Symptoms
 - Admin login at `/admin` silently fails: the login button disables briefly, re-enables, and shows no error. **Nothing appears in `logs/grav.log`** — a wrong password *would* log a failed-attempt warning, so its absence means auth was never reached.
 - The Admin2 SPA's background login request (to an `/api/...` endpoint) returns **HTTP 404** with `content-type: application/json`.
 - `GET /api/v1/pages` returns **404** on prod, but **401 Unauthorized** on the working local install — i.e. the api route isn't registered on prod at all.
 - `ls user/plugins/api` on the server: **No such file or directory** — the plugin was never installed, even though `admin2` (which depends on it) was.
 - `php bin/gpm install ... api -y` → `"These packages were not found on Grav: api"`, even after `php bin/gpm index -f`.
 ## What Didn't Work
 - **Committing/deploying the api plugin config** (`enabled` / `route` / `session_enabled`, moved from the untracked `user/plugins/api/api.yaml` into the tracked `user/config/plugins/api.yaml`). This was a real, necessary fix for a *different* latent problem, but it did not fix login: you cannot configure a plugin that isn't installed. Still 404.
 - **Forcing a GPM index refresh** (`php bin/gpm index -f`). No effect. "Package not found" here is not a stale-index problem — GPM filters the packages it offers by the installed Grav **core** version, and rc.10 is below the api plugin's `>=2.0.4` requirement, so `api` is excluded from results entirely.
 - **Assuming "same channel = same availability."** Local (Grav 2.0.4, `stable` channel) found `api` via `gpm info api`; prod (also `stable`) reported it "not found." The channel was identical — the difference was the Grav **core** version, which silently filtered the plugin out.
 ## Solution
 The real cause is that prod was running the wrong Grav core. `scripts/server-install.sh` downloads `grav-admin-v${GRAV_VERSION}.zip`, and `.env.prod` still carried the stale pre-upgrade `GRAV_VERSION=2.0.0-rc.10`.
 1. Upgrade the Grav core in place to stable (rc.10 → 2.0.7):
   ```bash
   make remote-upgrade-grav-prod        # php bin/gpm self-upgrade -y && php bin/grav cache
   ```
 2. Install the plugins now that a compatible core is present (the api plugin resolves):
   ```bash
   make remote-install-plugins-prod     # php bin/gpm index -f && php bin/gpm install <plugins.txt> -y
   # => "Preparing to install API [v1.0.8] ... Success!"
   ```
 3. Clear cache, then verify the api route is live and login works:
   ```bash
   make remote-clean-prod
   curl -s -o /dev/null -w '%{http_code}\n' https://site/api/v1/pages
   # 401  (was 404) => plugin installed + routed
   ```
 4. **Prevent recurrence:** update `.env.prod` to `GRAV_VERSION=2.0.4` so a future *fresh* install doesn't reinstall rc.10 (self-upgrade fixed the running server, not the env file). Keep the api plugin's functional config in the tracked `user/config/plugins/api.yaml` so it deploys on a clean clone.
 ## Why This Works
 GPM (Grav Package Manager) only offers a plugin version whose declared Grav requirement is satisfied by the **installed core**. The `api` plugin requires Grav `>=2.0.4`; on a `2.0.0-rc.10` core there is no compatible version, so GPM reports the package as "not found" rather than a version conflict. Admin2 declares `api` as a hard dependency and performs all authentication over the api plugin's `/api/v1` JWT endpoints, so with `api` absent the login POST hits a route that doesn't exist (404) and never reaches Grav's auth layer — hence the silent failure with no `grav.log` entry. Upgrading the core to a stable `>=2.0.4` build makes GPM offer `api` again; installing it registers `/api/v1`, and Admin2's login flow succeeds.
 ## Prevention
 - **Keep `.env.<env>` `GRAV_VERSION` current.** It is the version a *fresh* `make remote-install-<env>` bakes in; a stale value silently installs an old core. After any core upgrade, bump the env file too — self-upgrade only moves the running server. Note this is a *third* version-authority surface alongside `user/config/system.yaml` `gpm.releases` (channel) and `plugins.txt` — they must stay in sync.
 - **When GPM says "package not found" for a package you know is on your channel, check the target's Grav core version first** (`php bin/grav --version` on the server, or `make remote-diag-<env>`). GPM filters by core compatibility; "not found" often means "no version compatible with your core," not "missing from the index." `gpm index -f` will not help.
 - **Don't trust a top-level install "Success" to mean dependencies installed.** A fresh install can leave a plugin's declared dependency unsatisfied (here `admin2` installed but its `api` dependency didn't). Verify with `ls user/plugins/<dependency>`. The same `ls` guards a *second*, distinct way a plugin ends up non-functional: its **code folder can be missing while its config persists** (e.g. in the per-host env tree), so it looks configured but never loads. Checking `ls user/plugins/<name>` catches both the missing-dependency and the config-without-code cases — see `grav-plugin-config-without-code-wont-enable.md`.
 - **Know the Admin2 ⇄ api coupling.** Admin2 authenticates via the api plugin's `/api/v1` endpoints; a missing or unrouted api plugin makes admin login fail *silently* (login POST 404s, nothing logged). A quick `curl /api/v1/pages` expecting `401` (not `404`) is a good post-deploy smoke check.
 ## Related
 This is one of three independent gotchas from the same **2026-07-04 Grav 2.0.4 production cutover**:
 - `docs/solutions/integration-issues/grav-double-content-encoding-garbage-page.md` — sibling: garbage-rendered pages from a double `Content-Encoding` header on a non-FastCGI host. Different root cause (HTTP compression), same deploy.
 - `docs/solutions/test-failures/new-user-grants-api-not-admin-on-admin2.md` — sibling: an authenticated account is denied an admin-gated page because `login new-user` auto-detect granted `api.*` but not `admin.*`. Different root cause (permission provisioning), same admin2/api area.
 - `docs/working/plans/2026-07-04-grav-2.0.4-upgrade.md` — the upgrade plan whose Global Constraints spell out the GPM version floors (`grav >=2.0.4`, `api >=1.0.6`) that cause the "package not found" on an rc core.
 - `docs/solutions/conventions/grav-plugin-config-must-be-tracked-override.md` — the *other* latent problem from this same investigation: the api plugin's functional config (`enabled` / `route` / `session_enabled`) must live in the tracked `user/config/plugins/api.yaml` to deploy at all. Necessary but not sufficient here (the plugin must be installed first), but a durable convention in its own right.
 A closely-related **sibling in the "plugin absent/non-functional on prod" family** (from the 2026-07-05 follow-up, not one of the three cutover gotchas above):
 - `docs/solutions/integration-issues/grav-plugin-config-without-code-wont-enable.md` — same outcome (a plugin inert on prod, fixed by a GPM install + cache clear), **different trigger**: there, GPM won't *offer* the plugin because the core is below the version floor; there, the plugin's *code folder is simply missing* while its config persists in the env tree (config-without-code desync). Same `ls user/plugins/<name>` smoke check flushes both out.
@@ -78,5 +78,6 @@ This site runs **Admin2 only** (the classic `admin` plugin is disabled), so auto
 ## Related Issues
 - `docs/working/plans/2026-07-04-grav-2.0.4-upgrade.md` — the self-contained test-account infrastructure shipped alongside the Grav 2.0.4 upgrade.
 - Sibling gotchas from the same 2026-07-04 Grav 2.0.4 production cutover (all surface around admin2/api but with distinct root causes): `docs/solutions/integration-issues/stale-grav-version-blocks-api-plugin-install.md` (stale `GRAV_VERSION` → rc core → GPM won't serve the `api` plugin → login 404s) and `docs/solutions/integration-issues/grav-double-content-encoding-garbage-page.md` (double `Content-Encoding` header → garbage page).
 - `docs/solutions/architecture-patterns/dual-repo-submodule-workflow.md` — accounts live in the `user/` repo; the `testrunner` account is gitignored so it never reaches production.
 - GPX manager auth model (`access.admin.login: true` frontmatter + Login plugin) — see the project's GPX manager notes.
@@ -6,11 +6,12 @@ Ideas and improvements not yet planned or scheduled.
 ## Production — remaining items
- [ ] Set `twig.cache: true` in `user/config/system.yaml` on the server (do not commit — breaks local dev)
+- [x] Prod Twig prod-mode (`cache: true`, `debug/auto_reload: false`) — applied as a per-environment override via `make remote-apply-env-prod` (source: `deploy/env/prod/system.yaml`); committed `system.yaml` stays dev
 - [ ] Smoke test: submit one post via `/post`, confirm entry appears in dailies immediately (verifies cache-on-save with twig cache on)
- [ ] Confirm `/post` requires login — unauthenticated visitors must not be able to post
+- [x] Confirm `/post` requires login — verified on prod (returns the login gate to unauthenticated visitors)
 - [ ] Register at carto.com and review terms for production traffic
- [ ] Japan & Korea 2026 trip page: set `date_start`, add `cover_image`, upload GPX route file(s)
+- [ ] Update `GRAV_VERSION` in `.env.prod` to `2.0.4` (was stale `2.0.0-rc.10`; fixed on the running server via self-upgrade, but a future fresh install would repeat the RC)
 - [ ] git-sync on prod: install, add encrypted token, apply `folders:` fix, enable after first content round-trip
 ---
@@ -1,17 +1,46 @@
 # Git Sync Plugin — Setup Notes
-## Folders YAML bug
+## ⚠️ Config lives in the ENVIRONMENT tree, not `user/config/` (IMPORTANT)
-The plugin UI always saves the folders field as a single comma-string:
+Prod has a per-environment override directory `user/env/<hostname>/config/`
 (created for Twig prod-mode — see CLAUDE.md §1). **A crucial Grav side effect:
 once that env dir exists, the Admin panel saves ALL config changes — system and
 plugin — into the active environment's config tree**, not `user/config/`.
-```yaml
+So on prod, `git-sync.yaml` (configured via Admin) lives at:
-folders:
+
-  - 'pages,config,themes'
+```
 user/env/intotheeast.com/config/plugins/git-sync.yaml   ← here (env tree)
 user/config/plugins/git-sync.yaml                       ← NOT here
 ```
-But the plugin code iterates the array expecting separate items. This causes `git status pages,config,themes` to be passed as a single path, so git sees nothing to commit and sync silently does nothing.
+Why this matters:
-**Fix:** Edit `user/config/plugins/git-sync.yaml` directly:
+- **⚠️ `user/env/` is NOT safe unless gitignored — it is NOT scoped out by the
  `folders` setting.** An earlier version of this note claimed `user/env/`
  "never reaches Gitea" because it is outside git-sync's synced folders. **That
  is wrong and caused a live secret leak (2026-07-05).** git-sync's auto-commit
  stages files *outside* the configured `folders`; on prod it pushed the whole
  `user/env/intotheeast.com/config/` tree — JWT secret, CSRF salt, **and the
  git-sync token + webhook secret** — to Gitea. The fix was to **gitignore
  `/env/`** (commit `6e8eadb`). So: prod Admin config edits stay server-only
  *only because `/env/` is now gitignored*, not because of folder scope. Author
  durable config in the repo, not prod Admin. Full analysis:
  `docs/solutions/architecture-patterns/git-sync-secret-exposure-and-tracked-file-boomerang.md`.
 - **Look in both places.** When inspecting/toggling server config, check
  `user/config/plugins/<name>.yaml` **and**
  `user/env/<host>/config/plugins/<name>.yaml` (env wins).
 - **Tooling is env-path-aware.** `scripts/git-sync-toggle.sh` takes a `WEBROOT`
  and searches `user/env/*/config/plugins/git-sync.yaml` first, then
  `user/config/plugins/git-sync.yaml`. `make remote-git-sync-disable/enable-<env>`
  and `make remote-diag-<env>` use it.
 ## Folders format
 Older plugin versions' UI saved the `folders` field as a single comma-string
 (`- 'pages,config,themes'`), which the plugin iterated as one path, so sync
 silently did nothing. **git-sync v3.4.4 (installed on prod 2026-07-04) saves it
 correctly** as separate list items:
 ```yaml
 folders:
@@ -20,8 +49,39 @@ folders:
  - themes
 ```
-Never use the Admin UI to change folders — it will rewrite the broken format.
+If you see the comma-string form on an older version, fix it by editing
 `git-sync.yaml` directly (at whichever path it lives — see above); do not
 re-save folders via the Admin UI on the buggy version.
-## Files to gitignore
+## Per-install / secret files — must be gitignored (gitignore is the boundary)
-`user/config/plugins/git-sync.yaml` contains an encrypted token and is server-specific. `user/config/security.yaml` contains Grav nonces/salts, also server-specific. Both are in `.gitignore` and must never be committed.
+git-sync's auto-commit stages **everything under `user/` that is not
 gitignored** — the `folders` setting does *not* scope the commit add-set (a
 2026-07-05 leak proved this by pushing `user/env/**`, outside the configured
 folders). So `.gitignore` — not folder scope — is the only thing keeping a
 per-install or secret file off Gitea. Keep all of these gitignored in
 `user/.gitignore`:
 | Path | Why |
 |---|---|
 | `env/` | **whole per-host env tree** — holds the live git-sync token, JWT secret, CSRF salt + all server-side Admin config. Gitignored + untracked 2026-07-05 (commit `6e8eadb`) after it leaked to Gitea. NOT safe on folder scope alone. |
 | `config/plugins/git-sync.yaml` | encrypted token; server-specific (also lives at env path on prod) |
 | `config/plugins/api-private.php` | API JWT secret |
 | `config/security.yaml` | Grav nonces/salts (legacy location) |
 | `config/versions.yaml` | per-install Grav schema-migration state — differs per env (dev 2.0.4, prod 2.0.7); Grav regenerates it. Untracked 2026-07-04. |
 | `config/security-private.php` | CSRF/nonce + admin rate-limit signing salt; gitignored + untracked 2026-07-05 (commit 2840018). Each env keeps its own; untracking regenerates prod's salt (one-time admin re-login). |
 > **Why a key inside a *tracked* config file (e.g. `popularity.salt` in `api.yaml`) can't just be stripped** — it regenerates at runtime and boomerangs back via git-sync's `git add -A`. See `docs/solutions/architecture-patterns/git-sync-secret-exposure-and-tracked-file-boomerang.md` for the full round-trippable-set model.
 ## git-sync config summary (prod, 2026-07-04)
 - `repository: https://git.gorinskat.nl/m038/intotheeast-com-content.git`,
  `branch: main`, HTTPS + token auth (SSH is Tailscale-only).
 - `sync.direction: both`, `on_save/on_delete/on_media: true` → prod Admin edits
  and `/post` push to Gitea; content-repo pushes pull to prod **via webhook**
  (`/_git-sync`). The webhook is configured in Gitea repo settings (same secret
  as the test instance).
 - **Before enabling on a fresh server**, reset the synced folders clean
  (`make remote-fetch-content-<env>`) so no install-time drift (e.g. a stale
  `versions.yaml`) gets pushed on the first sync. Toggle with
  `make remote-git-sync-disable/enable-<env>`.
@@ -2,7 +2,7 @@
 > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
-**Status:** ✅ Complete (2026-07-04) — local (Phase 1) validated and shipped; Task 5 (remote test-env, Phase 2) is user-gated and parked; Phase 3 (prod) is documentation-only per design. See "Known issue" below re: Form 9.1.10 filepond.
+**Status:** ✅ Complete — Phase 1 (local), Phase 2 (test env), and **Phase 3 (production) all executed and shipped**. Phase 3 was run for real on 2026-07-05 (see the Phase 3 section for the execution outcome and the three `docs/solutions/` gotchas it produced). git-sync re-enabled on test and set up on prod. See "Known issue" below re: Form 9.1.10 filepond.
 **Goal:** Upgrade Grav core `2.0.0-rc.10` → `2.0.4` stable and promote `admin2`/`api`/`flex-objects` to GPM management, validated on local then the remote test env (prod is documented-only).
@@ -533,15 +533,38 @@ Content, config, and accounts are in git, so no data restore is required — but
 **Tasks 3 & 4 (remote Makefile targets + server-install cleanup): shipped** (committed on this branch).
-**Task 5 (Phase 2, remote test-env): parked, user-gated.** Not executed. Requires the user to confirm which branch the test server tracks and that `.env.test` is ready. All `-test` make targets exist and dry-run clean.
+**Task 5 (Phase 2, remote test-env): executed and validated (2026-07-04).** Sequence run: `remote-git-sync-disable-test` → `content-push` → `remote-fetch-content-test` → `remote-upgrade-grav-test` (core rc.10 → **2.0.7**; stable served a newer patch than the 2.0.4 floor) → `remote-update-plugins-test` (all plugins to stable) → `remote-content-status-test`. Smoke test on `https://test.intotheeast.com`: `/` (renders), `/admin` (admin2 panel; note the server routes admin at `/admin`, not `/admin2`), and `/gpx-manager` all return 200 with no Twig/PHP errors. git-sync was **re-enabled** afterward at the user's request.
 **Config-drift gotcha (reconciled).** The server's `bin/gpm self-upgrade` ran Grav's schema migration, which rewrote `system.yaml` `strict_mode` from the 1.7-era `twig_compat: false` to `twig2_compat: false` + `twig3_compat: true`. The **local** upgrade never triggered this because it was a fresh Docker-image build, not a `self-upgrade` — so the repo's `system.yaml` was stale and a future `remote-fetch-content` (`reset --hard`) would have reverted the server. Fix: folded the Twig 3 flags into the repo's `user/config/system.yaml` (committed `2e32a85`, content-pushed), verified the local Grav 2.0.x container renders 200 under them, then reset the server to the new `origin/main` before re-enabling git-sync so its working tree was clean. `versions.yaml` and `accounts/.htaccess` drift is install-local and left to Grav to manage.
 ---
-## Phase 3 — Production (fresh install, NOT executed)
+## Phase 3 — Production (fresh install) — EXECUTED 2026-07-05
-Production is empty, so this is a **fresh install**, not an upgrade — and it is **documentation only**. Do not run it as part of this plan.
+> **Execution outcome (2026-07-05):** the fresh prod install was run for real
 > (`make remote-install-prod`) and the site is live at `https://intotheeast.com`.
 > The runbook below was followed, but three non-obvious gotchas surfaced — each
 > now has its own learning in `docs/solutions/`:
 > - **Stale `.env.prod GRAV_VERSION`** installed Grav rc.10, so GPM wouldn't
 >   serve the `api` plugin (needs ≥2.0.4) → Admin2 login 404'd silently. Fixed
 >   via `make remote-upgrade-grav-prod` (→ 2.0.7) + reinstall. See
 >   `docs/solutions/integration-issues/stale-grav-version-blocks-api-plugin-install.md`.
 >   **TODO: bump `.env.prod GRAV_VERSION` to `2.0.4`** so a future fresh install
 >   doesn't repeat the RC.
 > - **Double `Content-Encoding` header** (non-FastCGI host + mod_deflate)
 >   rendered a garbage page once prod switched to `twig.debug: false`. Fixed via
 >   `debugger.shutdown.close_connection: false` in the prod env override. See
 >   `docs/solutions/integration-issues/grav-double-content-encoding-garbage-page.md`.
 > - **Plugin config stranded in the untracked `user/plugins/`** doesn't deploy.
 >   See `docs/solutions/conventions/grav-plugin-config-must-be-tracked-override.md`.
 >
 > Twig prod-mode is applied as a per-environment override (`deploy/env/prod/system.yaml`
 > via `make remote-apply-env-prod`); git-sync is installed, configured, and enabled
 > (see `docs/working/git-sync-notes.md`). Remaining minor follow-ups: gitignore
 > `config/security-private.php` (committed salt); optional `popularity.salt` strip.
-When prod is provisioned:
+The original runbook (production was empty, so this was a **fresh install**, not
 an upgrade):
 1. **Provision creds:** copy the REMOTE section of `.env.example` into `.env.prod` with production values (never commit it). Run `make remote-env-setup-prod`.
 2. **Fresh install at 2.0.4:** `make remote-install-prod` with `GRAV_VERSION=2.0.4` in `.env.prod`. `scripts/server-install.sh` installs core, then all of `plugins.txt` — `admin2`/`api`/`flex-objects` now install purely via `php bin/gpm install` (no zip-stash; that special-casing was removed in Task 4). The `gpm.releases: stable` channel arrives with the `user/` content clone.
@@ -1,13 +1,22 @@
 #!/bin/bash
 set -e
-FILE="$1"
+# Enable/disable the git-sync plugin by flipping `enabled:` in its config.
 #
 # git-sync.yaml may live in the per-environment config tree
 # (user/env/<host>/config/plugins/) when an env override dir exists — Grav's
 # Admin saves config there when an environment is active — otherwise in the
 # standard user/config/plugins/. Search both, env path first.
 WEBROOT="$1"
 STATE="$2"
-: "${FILE:?usage: git-sync-toggle.sh <git-sync.yaml path> <true|false>}"
+: "${WEBROOT:?usage: git-sync-toggle.sh <webroot> <true|false>}"
-: "${STATE:?usage: git-sync-toggle.sh <git-sync.yaml path> <true|false>}"
+: "${STATE:?usage: git-sync-toggle.sh <webroot> <true|false>}"
-if [ ! -f "$FILE" ]; then
+FILE=$(ls "$WEBROOT"/user/env/*/config/plugins/git-sync.yaml \
-    echo "ERROR: $FILE not found — is git-sync installed on this server?" >&2
+          "$WEBROOT"/user/config/plugins/git-sync.yaml 2>/dev/null | head -1)
 if [ -z "$FILE" ] || [ ! -f "$FILE" ]; then
    echo "ERROR: git-sync.yaml not found under $WEBROOT — is git-sync installed/configured?" >&2
    exit 1
 fi
@@ -17,4 +26,4 @@ else
    printf 'enabled: %s\n' "$STATE" | cat - "$FILE" > "$FILE.tmp" && mv "$FILE.tmp" "$FILE"
 fi
-echo "git-sync now: $(grep -E '^enabled:' "$FILE")"
+echo "git-sync now: $(grep -E '^enabled:' "$FILE")  ($FILE)"
Author	SHA1	Message	Date
m038andClaude Opus 4.8	0a7997c92d	docs: refresh stale-grav-version doc — cross-link config-without-code sibling ce-compound-refresh (scope: stale-grav-version-blocks-api-plugin-install). Update (not consolidate — distinct triggers, moderate overlap): add a Related back-link to the new grav-plugin-config-without-code-wont-enable sibling and broaden the "verify ls user/plugins/<name>" prevention bullet to note the config-without-code desync as a second non-functional-plugin case. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Np4cMQLF77i664CAQXySzU	2026-07-05 10:44:13 +02:00
m038andClaude Opus 4.8	cf21e199bc	docs: add local→test→prod deploy-cycle runbook Distills the 2026-07 Grav 2.0.4→2.0.7 cutover into a repeatable procedure: the three-layer state model (plugin code / repo config / host env tree), ordered per-phase make-target sequences, a smoke checklist that catches the code-vs-config, stale-version, garbage-page, and git-sync-boomerang gotchas, plus rollback and one-line invariants. Linked from CLAUDE.md Remote operations. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Np4cMQLF77i664CAQXySzU	2026-07-05 10:40:04 +02:00
m038andClaude Opus 4.8	b5fc43d208	docs: add integration-issue — Grav plugin won't enable when code missing but config persists in env tree Documents the git-sync prod incident where the plugin was inert despite `enabled: true`, because its CODE was missing from user/plugins/git-sync/ while its CONFIG survived in the per-host env tree. Fix: GPM reinstall (make remote-gpm-install-prod PKG=git-sync) + clearcache. Captures the diagnostic (ls the plugin dir before touching config) and the remote-only plugin caveat (not in plugins.txt, not restored by the standard flows). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Np4cMQLF77i664CAQXySzU	2026-07-05 10:34:39 +02:00
m038andClaude Opus 4.8	6dc6af6359	docs: correct git-sync secret-exposure model (gitignore is the boundary, not folder scope) The prior model claimed git-sync's add-set is scoped to the configured `folders` (pages/config/themes), so accounts/ and user/env/ were "safe by construction." That was wrong: prod auto-commit 9337003 pushed the whole user/env/<host>/config tree (JWT secret, CSRF salt, git-sync token + webhook secret), accounts/mischa.yaml, and system.yaml to Gitea — all outside the configured folders. git-sync stages everything under user/ not gitignored; .gitignore is the only reliable exclusion. - Rewrite the architecture-patterns doc around the corrected predicate and document the incident + remediation (gitignore /env/, rotate token+webhook). - Correct git-sync-notes.md: env/ is NOT safe on folder scope; drop the "never reaches Gitea" claim; fix the secret-files table intro + add /env/. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Np4cMQLF77i664CAQXySzU	2026-07-05 10:19:54 +02:00
m038andClaude Opus 4.8	e0e2e1e7b5	docs: mark Grav 2.0.4 plan Phase 3 (prod) executed + outcome Prod was actually deployed (LIVE at intotheeast.com, Grav 2.0.7); the plan previously said Phase 3 was documentation-only/not-executed. Record the three deploy gotchas (docs/solutions/) and remaining follow-ups. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Np4cMQLF77i664CAQXySzU	2026-07-05 00:16:33 +02:00
m038andClaude Opus 4.8	39d42119b2	docs: capture Grav plugin-config-must-be-tracked convention (ce-compound) Knowledge-track: functional Grav plugin config must live in the tracked user/config/plugins/<name>.yaml override, not the gitignored (and gpm-update-wiped) user/plugins/<name>/<name>.yaml. Secrets stay in gitignored *-private.php. Root of the api plugin being unconfigured on the fresh prod install. Cross-linked with the api/rc.10 install-failure doc. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Np4cMQLF77i664CAQXySzU	2026-07-05 00:05:11 +02:00
m038andClaude Opus 4.8	66438836de	docs: capture stale-GRAV_VERSION api-plugin-install failure (ce-compound) Fresh prod install ran Grav rc.10 (stale .env.prod GRAV_VERSION), so GPM would not serve the api plugin (needs >=2.0.4); admin2 (auth via /api/v1) then 404'd login silently. Documents the dead ends (deploying api config, gpm index -f, same-channel assumption) and the fix (self-upgrade core + reinstall + bump .env.prod). Adds reciprocal 'same 2026-07-04 cutover' cross-links across the three sibling deploy gotchas. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Np4cMQLF77i664CAQXySzU	2026-07-04 23:59:44 +02:00
m038andClaude Opus 4.8	3ad055d4a8	docs: capture Grav double Content-Encoding garbage-page fix (ce-compound) Non-FastCGI host + Apache mod_deflate: Grav's shutdown early-close emits Content-Encoding: identity while mod_deflate adds gzip -> two conflicting headers -> browsers render raw gzip bytes. Fix: debugger.shutdown.close_ connection:false in the prod env override. Documents the dead-end (cache.gzip/allow_webserver_gzip take the same code path) and the browser-style curl + screenshot verification. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Np4cMQLF77i664CAQXySzU	2026-07-04 23:53:11 +02:00
m038andClaude Opus 4.8	dcf9c13455	docs: document Grav env-tree config save behavior + git-sync env path Once user/env/<host>/ exists, Grav Admin saves ALL config (system + plugin) into that env tree, not user/config/ — so prod's git-sync.yaml lives at the env path, and Admin-on-server config edits are server-only (not committed, not synced). Documented the footgun in CLAUDE.md §1 (session-loaded) and rewrote git-sync-notes.md: env-path location, v3.4.4 folders format, gitignore table (incl. versions.yaml + security-private.php TODO), and fresh-server enable procedure. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Np4cMQLF77i664CAQXySzU	2026-07-04 23:40:11 +02:00
m038andClaude Opus 4.8	425c7b8e20	chore: bump user/ pin to versions.yaml untrack (4aa9ae9) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Np4cMQLF77i664CAQXySzU	2026-07-04 23:35:11 +02:00
m038andClaude Opus 4.8	d61de6f3f7	build: make git-sync toggle + diag env-path aware Grav Admin saves plugin config into the active environment's config tree (user/env/<host>/config/plugins/) when an env override dir exists — so git-sync.yaml landed there, not in user/config/plugins/. Update the toggle script to take a WEBROOT and search both locations (env path first), and update remote-git-sync-disable/enable to pass WEBROOT. remote-diag now surfaces git-sync config (secrets redacted) from either location. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Np4cMQLF77i664CAQXySzU	2026-07-04 23:35:11 +02:00
m038andClaude Opus 4.8	cc40c23ea8	build: add remote-gpm-install (single out-of-plugins.txt package) For installing remote-only plugins like git-sync that are deliberately excluded from plugins.txt. Usage: make remote-gpm-install-prod PKG=git-sync Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Np4cMQLF77i664CAQXySzU	2026-07-04 23:18:51 +02:00
m038andClaude Opus 4.8	4aeff39756	fix(prod): stop garbage page from double Content-Encoding header On this non-FastCGI host Grav's shutdown early-connection-close can't fastcgi_finish_request(), so it emits 'Content-Encoding: identity' to ask the webserver not to compress — but Apache mod_deflate compresses anyway and adds 'Content-Encoding: gzip'. Two conflicting headers => browsers render raw gzip bytes (garbage page). Surfaced when prod flipped to debug:false today (activated Grav's full output path). Fix in the prod env override: debugger.shutdown.close_connection:false skips the identity-header path so mod_deflate compresses cleanly (single header). Verified: one 'content-encoding: gzip', page renders correctly. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Np4cMQLF77i664CAQXySzU	2026-07-04 23:12:27 +02:00
m038andClaude Opus 4.8	0e597c5329	chore: bump user/ pin to API plugin config fix (b8a7bed) Pins the superproject to the tracked API plugin config (deployed to prod to restore Admin2 login). Already published to content repo main. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Np4cMQLF77i664CAQXySzU	2026-07-04 23:01:36 +02:00
m038andClaude Opus 4.8	41e61fc148	build: per-environment Grav config override (prod Twig prod-mode) Prod needs twig.cache:true / debug:false / auto_reload:false, but those values break local dev, so they must not live in the committed system.yaml. Add a per-environment override deployed only to the server via Grav's environment://config (user/env/<host>/config/system.yaml): - deploy/env/prod/system.yaml — version-controlled source of truth. - make remote-apply-env-prod — writes it to the server + clears cache; resolves the host in-recipe (WEB_HOST \|\| REMOTE_HOST) to avoid the recursive-make empty-export trap. - remote-diag now shows the deployed override + whether twig cache is populating, so prod-mode can be verified not assumed. - CLAUDE.md §1 rewritten: never flip committed system.yaml; use the override. Backlog updated (twig prod-mode + /post login-gate done; note stale .env.prod GRAV_VERSION and pending git-sync). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Np4cMQLF77i664CAQXySzU	2026-07-04 22:54:59 +02:00
m038andClaude Opus 4.8	553d9e4759	build: force GPM index refresh before install; add remote-diag - remote-install-plugins now runs 'gpm index -f' before install so a fresh server doesn't rely on the stale package index bundled in the grav-admin zip (which can miss recently-published plugins). - Add read-only remote-diag target: Grav version, installed plugin versions, GPM view of the api plugin, api override keys, and log tail. Surfaced the prod rc.10 vs 2.0.4 mismatch that blocked the api plugin. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Np4cMQLF77i664CAQXySzU	2026-07-04 22:49:06 +02:00
m038andClaude Opus 4.8	a35eb4f288	chore: bump user/ pin to Twig 3 strict_mode config (2e32a85) End-of-feature pointer bump for the Grav 2.0.4 upgrade. Pins user/ to the reconciled Twig 3 strict_mode config, already published to Gitea main. Aligns the superproject with the finished content-repo state. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Np4cMQLF77i664CAQXySzU	2026-07-04 20:36:25 +02:00
m038andClaude Opus 4.8	db50b84bfd	docs: record Phase 2 test-env upgrade executed + twig3 config reconciliation Test env upgraded to Grav 2.0.7, plugins to stable, smoke-tested, git-sync re-enabled. Documents the strict_mode Twig 3 migration gotcha (fresh-image local build skips the schema migration that self-upgrade runs). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Np4cMQLF77i664CAQXySzU	2026-07-04 20:15:57 +02:00