Merge remote-tracking branch 'origin/main' into feat/journal-post-form

2026-07-05 11:32:40 +02:00
parent c56265824b 58a8504a47
commit 7329852497
9 changed files with 516 additions and 16 deletions
@@ -132,6 +132,8 @@ After updating, also create the new trip's page tree under `user/pages/01.trips/
 Always use `make` commands for anything on the production server (`make remote-install-plugins`, `make remote-clean`, etc.) — never SSH directly since credentials live in `.env`. If a remote operation isn't covered by an existing `make` command, either ask the user to run it manually or suggest adding a new `make` command if it seems reusable.
 For a full upgrade/deploy through local → test → prod (ordered steps, smoke checklist, rollback), follow the runbook at [`docs/guides/deploy-cycle.md`](docs/guides/deploy-cycle.md).
 ### Content sync
 - `make content-push` — commit and push `user/` to Gitea (triggers production pull via webhook)
@@ -1,4 +1,8 @@
-# Production-only Grav config overrides.
+# Deployed-environment Grav config overrides (test AND prod).
 #
 # Both server environments share this one file so test stays a faithful dress
 # rehearsal of prod: deploy/env/test/system.yaml is a symlink to this file.
 # Edit here and both environments move together — never let them drift.
 #
 # Deep-merged OVER the committed user/config/system.yaml via Grav's
 # per-environment config mechanism: on the server this file is deployed to
@@ -8,9 +12,10 @@
 #
 # These values are deliberately NOT in the committed system.yaml because they
 # would break local development (see CLAUDE.md §1 — dev keeps twig.cache:false
-# so theme edits take effect immediately). Prod is the only place they apply.
+# so theme edits take effect immediately). They apply only on the deployed
 # hosts, never on a local dev checkout.
 #
-# Deploy with:  make remote-apply-env-prod
+# Deploy with:  make remote-apply-env-test  /  make remote-apply-env-prod
 # The user/env/ tree is outside the content repo's tracked folders, so it is
 # NOT restored by content-push / git-sync / remote-fetch-content — re-run the
 # target above after any fresh install.
@@ -0,0 +1 @@
 ../prod/system.yaml
@@ -0,0 +1,133 @@
 # Upgrade & Deploy Cycle: local → test → prod
 This runbook is the repeatable procedure for shipping a Grav upgrade or any
 server-affecting change (core version, plugins, config, theme) through the three
 environments. It was distilled from the 2026-07 Grav 2.0.4→2.0.7 cutover, where
 every production surprise traced back to one of the desyncs this procedure now
 forces you to check.
 **Governing principle:** `test` is a **full dress rehearsal of `prod`** — same
 config, same `-test`/`-prod` make targets, same order. A gotcha only gets caught
 on test if test is a faithful mirror of prod. Do not shortcut test.
 All server operations go through `make remote-*` targets (never raw SSH — the
 targets build the SSH connection from `.env.<env>`, which must never be read
 directly). Every `remote-*` target has `-test` and `-prod` variants; a bare
 target fails via `guard-env`.
 ---
 ## The mental model: three places state lives
 Every failure in the reference cutover was a desync between these three layers.
 Before and after each deploy step, ask: *are they in sync?*
 | Layer | Location | Synced by | Failure mode |
 |---|---|---|---|
 | Plugin **code** | `user/plugins/<name>/` | GPM only (gitignored `/plugins/*`) | can vanish while config remains → plugin won't enable |
 | **Repo config** | `user/config/…` | `content-push` / git-sync | holds GPM channel + is where the version floor bites |
 | **Host config** | `user/env/<host>/config/…` | nothing — server-only | not restored on fresh install; must be re-applied; **must be gitignored** |
 Referenced gotcha docs:
 - `docs/solutions/integration-issues/grav-plugin-config-without-code-wont-enable.md` — code-vs-config desync.
 - `docs/solutions/integration-issues/stale-grav-version-blocks-api-plugin-install.md` — stale `GRAV_VERSION` / version floor.
 - `docs/solutions/architecture-patterns/git-sync-secret-exposure-and-tracked-file-boomerang.md` — gitignore is the sync boundary; env-tree leak.
 - `docs/solutions/conventions/grav-plugin-config-must-be-tracked-override.md` — plugin config must live in the tracked override.
 ---
 ## Phase 0 — Local (author + prove the change)
 1. Make the change in the repo:
   - GPM channel: `gpm.releases: stable` in `user/config/system.yaml` (authoritative; reaches servers via content pull, so it must be right **before** any server GPM op).
   - `plugins.txt` — the GPM-managed set only. **Never** add `git-sync` (it is remote-only).
   - Prod-only overrides (Twig cache/debug, `debugger.shutdown.close_connection: false`) in `deploy/env/prod/system.yaml` — **never** commit prod values into `user/config/system.yaml`.
   - **Bump `GRAV_VERSION` in `.env.test` and `.env.prod`** to the target version. A stale value here installs the wrong core (an rc), which then blocks the `api` plugin and 404s admin.
 2. `make build-assets` if you touched `js/src/*` (never hand-edit the bundled `js/*.js`).
 3. Run the dev server (`docker compose … up`) and the Playwright suite.
 4. Pre-flight assertions:
   - `gpm.releases` is `stable`.
   - `plugins.txt` is correct and does **not** contain `git-sync`.
   - No prod Twig values leaked into the committed `system.yaml`.
 5. Commit. `make content-push`.
 ---
 ## Phase 1 — Test (the rehearsal — catch things here)
 ### Pre-flight
 - `make remote-git-sync-disable-test` **before any content reset.** This is the safety catch for the whole window: it stops a half-migrated state (e.g. a fresh install-time `versions.yaml`) from auto-committing and pushing on the first sync.
 ### Apply — in this fixed order
 ```
 make remote-fetch-content-test              # 1. clean-reset synced folders to repo state
 make remote-upgrade-grav-test               # 2. gpm self-upgrade (rewrites schema — expect drift)
 make remote-update-plugins-test             # 3. gpm update the plugins.txt set
 make remote-gpm-install-test PKG=git-sync   # 4. EXPLICITLY (re)install each remote-only plugin
 make remote-apply-env-test                  # 5. re-deploy the env override (not synced; gone after install)
 ```
 Why each matters:
 - **Step 4** is non-optional even if git-sync "was already there" — remote-only plugins are not in `plugins.txt`, so nothing in steps 1–3 restores them. If the code is missing, the plugin is inert despite valid config.
 - **Step 5** re-writes `user/env/<host>/config/…` from `deploy/env/<env>/`. The env tree is not synced by anything, so a fresh install loses it until you re-apply.
 ### Verify (smoke checklist — this is the payoff)
 - **Code present, not just config:** `ls user/plugins/<name>/` for every expected plugin (especially `git-sync`). An empty/absent dir = reinstall (step 4). *(Do this via an ssh one-liner you run, or `make remote-diag-test`.)*
 - **HTTP:** `/` → 200, `/admin` → 200, `/api/v1/pages` → 401, `/gpx-manager` → 200. Watch for the double-`Content-Encoding` garbage page (fix: `debugger.shutdown.close_connection: false` in the env override — already in `deploy/env/prod/system.yaml`).
 - **Post smoke test:** submit one entry via `/post` and confirm it appears in the trip feed immediately. This proves the `cache-on-save` plugin works with prod caching on.
 - **Config drift:** `make remote-diag-test` — diff server config against the repo. Fold any *intended* schema migration (e.g. the Twig-3 `strict_mode` flags a `self-upgrade` writes) back into `user/config/system.yaml`, or the next `fetch-content` reverts it.
 ### Re-enable + prove sync
 - `make remote-git-sync-enable-test`.
 - Confirm a content push round-trips to the server, **and** that no secret/boomerang commit lands on Gitea. Verify `/env/` is gitignored so the env tree (which holds the token, JWT, CSRF salt) can never enter the sync add-set.
 ---
 ## Phase 2 — Prod (repeat identically — should be mechanical)
 Run the **exact same sequence** with `-prod` targets. Because test rehearsed it,
 prod holds no surprises. Differences to layer on:
 - Optional: `make remote-maintenance-on-prod` at the start, `remote-maintenance-off-prod` at the end, for a clean window.
 - Confirm secrets are valid/rotated and `/env/` is gitignored **before** `remote-git-sync-enable-prod`. Re-enable git-sync **last**.
 - After a clean cutover, bump the outer-repo submodule pin to the finished `user/` commit — and **push `user/` before the outer repo** (the superproject references a child SHA that must already exist upstream).
 ```
 make remote-git-sync-disable-prod
 make remote-fetch-content-prod
 make remote-upgrade-grav-prod
 make remote-update-plugins-prod
 make remote-gpm-install-prod PKG=git-sync
 make remote-apply-env-prod
 # ── smoke checklist (same as test) ──
 make remote-git-sync-enable-prod
 ```
 For a first-time / from-scratch prod bring-up, `make remote-install-prod` does the
 full install; then still run `remote-apply-env-prod` and the smoke checklist, and
 reinstall remote-only plugins explicitly.
 ---
 ## Rollback & safety
 - **git-sync stays disabled through the whole apply window** on each host — it is the catch that prevents a half-migrated state from auto-pushing.
 - **Content** is a git repo: a bad content deploy is recoverable with `make remote-fetch-content-<env>` back to a known commit.
 - **Core + plugins** are GPM-reinstallable (`remote-upgrade-grav`, `remote-update-plugins`, `remote-gpm-install PKG=…`).
 - The one thing tooling cannot regenerate is the un-synced `user/env/<host>/` tree — its source of truth is `deploy/env/<env>/`, so keep that current and re-apply with `remote-apply-env-<env>`.
 ---
 ## One-line invariants (the through-line)
 1. `test` is config-identical to `prod`, run with the same targets in the same order.
 2. Verify the **code layer** (`ls user/plugins/<name>/`), not just config, on every deploy.
 3. Reinstall **remote-only** plugins (git-sync) explicitly — nothing else restores them.
 4. `GRAV_VERSION` in `.env.<env>` and `gpm.releases: stable` are correct **before** any server GPM op.
 5. Re-apply the **env override** after every install; keep `/env/` **gitignored**.
 6. git-sync **off** during the window, **on** last; confirm the round-trip carries no secrets.
 7. Diagnose actual state before changing config — an `ls` or `remote-diag` beats a guess.
@@ -0,0 +1,243 @@
 ---
 title: "Secret exposure under bidirectional Grav git-sync: gitignore is the only boundary (and the tracked-file boomerang trap)"
 date: 2026-07-05
 module: git-sync
 problem_type: architecture_pattern
 component: tooling
 severity: high
 category: architecture-patterns
 applies_when:
  - "Enabling bidirectional Grav git-sync (direction: both, on_save: true) so prod can push content back to Gitea"
  - "Auditing whether the server can leak secrets (tokens, password hashes, signing salts) off the production host"
  - "A per-install runtime-generated value is being persisted into a tracked config file that also carries functional config"
  - "Deciding where a per-install secret must live so it never round-trips"
  - "A config value keeps ping-ponging or re-committing itself across environments after each sync"
 tags:
  - git-sync
  - secret-exposure
  - gitignore
  - grav
  - popularity-salt
  - config-boundary
  - bidirectional-sync
  - per-install-secret
  - env-tree-leak
 ---
 # Secret exposure under bidirectional Grav git-sync: gitignore is the only boundary (and the tracked-file boomerang trap)
 > **Correction (2026-07-05):** An earlier version of this doc claimed the sync
 > add-set was *scoped to the configured `folders`*, and concluded that
 > `accounts/` and `user/env/` were "safe by construction" because they sit
 > outside `pages/config/themes`. **That model is wrong and caused a live secret
 > leak.** git-sync's auto-commit stages files **outside** the configured folders;
 > the only reliable exclusion is `.gitignore`. The corrected model is below.
 ## Context
 The intotheeast.com Grav site runs the `git-sync` plugin on production in
 **bidirectional** mode: `direction: both`, `on_save: true`, a webhook at
 `/_git-sync`, and configured `folders: pages, config, themes`. Bidirectional
 means the server both *pulls* content authored elsewhere **and** *pushes*
 content authored on the server (Admin edits, `/post` submissions, uploads) back
 to the shared Gitea repo.
 The operator's question was: **"Can I safely enable bidirectional prod → Gitea
 sync without leaking secrets?"** Production holds things that must never reach a
 shared repo — API tokens, JWT signing secrets, CSRF salts, the git-sync token
 itself.
 The dangerous, tempting answer is "the plugin only syncs `pages/config/themes`,
 so anything outside those folders is safe." **This is false, and acting on it
 leaked secrets.** See the incident below.
 ## The incident (what actually happened)
 Production's git-sync auto-commit `9337003` ("(Grav GitSync) Automatic Commit
 ...") pushed the **entire `user/env/intotheeast.com/config/` tree** to Gitea —
 including `api-private.php` (JWT secret), `security-private.php` (CSRF salt), and
 `git-sync.yaml` (the sync **token + webhook secret**) — plus `accounts/mischa.yaml`
 (password hash), `system.yaml`, and `post-form.md`.
 **None of those paths is under the configured `folders: pages, config, themes`.**
 `user/env/` is a sibling of `user/config/`, not a subfolder of it. Yet git-sync
 staged and pushed them anyway. That single fact refutes the "folders scope the
 add-set" model empirically: **the `folders` setting does not scope what the
 auto-commit stages.** Whatever git-sync's exact `git add` invocation, the
 operational truth is that its commit sweeps the whole `user/` working tree.
 Remediation: disable sync → `.gitignore` `/env/` and `git rm --cached` it →
 `reset --hard` prod to the gitignored state → regenerate the leaked JWT/CSRF
 salts (delete the `*-private.php` files; Grav regenerates them) → **rotate the
 Gitea token and webhook secret** (they were exposed in `git-sync.yaml`). Rotation
 is what actually neutralizes the leak; the history rewrite is optional for a
 private repo.
 ## Guidance
 The safety question reduces to one predicate — but **not** the one the original
 doc used:
 > **A file round-trips to the shared repo if and only if it is NOT gitignored.**
 > The configured `folders` setting does **not** narrow this. Treat the
 > round-trippable set as *everything under `user/` that git will track* — i.e.
 > everything not matched by `user/.gitignore`.
 Consequences, corrected:
 1. **`.gitignore` is the only reliable boundary.** Do not rely on a file being
   "outside the synced folders." If it is under `user/` and not gitignored, a
   bidirectional sync can push it. Design exclusions with `.gitignore`, and
   verify with `git -C user status` / `git -C user check-ignore <path>`.
 2. **`accounts/` is NOT structurally excluded.** `user/accounts/*.yaml` (bcrypt
   password hashes) is a *tracked* content folder and is not gitignored, so it
   **does** round-trip — `accounts/mischa.yaml` was in the leak commit. If you
   need an account file to stay server-local, it must be gitignored explicitly
   (as `accounts/testrunner.yaml` already is). The earlier "password hashes never
   leave the server" claim was wrong.
 3. **The per-environment tree `user/env/<host>/` MUST be gitignored** — it is
   **not** inherently safe. It holds the live git-sync token, JWT secret, and
   CSRF salt (Grav writes all server-side Admin config there once the env dir
   exists). Because it is not under `user/config/` people assumed it was outside
   the sync scope; the incident proved it is not. It is now gitignored
   (`/env/` in `user/.gitignore`, commit `6e8eadb`). Keep it that way.
   > Side effect worth remembering: once `user/env/<host>/` exists, Grav's Admin
   > writes **all** config changes there (system and plugin), not into
   > `user/config/`. So server-side Admin edits are server-only — but "server-only"
   > now depends entirely on `/env/` being gitignored, not on folder scope. When
   > auditing, check **both** `user/config/...` and `user/env/<host>/config/...`
   > (env wins at runtime). See `docs/working/git-sync-notes.md`.
 4. **Per-install secrets go in gitignored companion files.** Grav's convention
   splits a per-install secret out of the functional YAML into a sibling that is
   gitignored: the JWT secret in `api-private.php`, the CSRF/nonce + rate-limit
   salt in `security-private.php`, plus `security.yaml` and `versions.yaml`.
   These are safe **because they are gitignored**, not because of where they sit.
 Run every secret through predicate #1 (is it gitignored?) and the answer falls
 out — but you must actually enumerate what is *not* gitignored, not what is
 "outside the folders."
 ### The tracked-file boomerang (a separate trap)
 Independently of the folder-scope error above, there is a second trap that the
 original doc got right and that still holds: **a per-install value that a plugin
 regenerates at runtime and persists into a tracked, functional config file.**
 The concrete case: the `api` plugin's *popularity* feature generates
 `popularity.salt` and writes it **into `user/config/plugins/api.yaml`** — a file
 that also carries must-be-shared functional config. That file is tracked, so on
 prod the popularity feature regenerates the salt, git-sync stages the change,
 commits, and **pushes prod's salt back to the shared repo**. Another environment
 pulls it, regenerates *its own* salt, pushes again. The value **ping-pongs across
 installs**, producing endless noise commits.
 The critical realization: **you cannot gitignore a single key inside a file that
 also carries functional config.** `.gitignore` operates on whole files. `api.yaml`
 must be tracked because the rest of it must be shared; therefore the salt inside
 it is tracked too; therefore it boomerangs.
 ### The rules
 For any per-install runtime-generated value, pick one of exactly three
 resolutions — and do **not** reach for the fourth (stripping the line), which
 cannot work under sync:
 - **Isolate the value into a gitignored companion `<name>-private.php`** — the
  pattern Grav uses for the JWT secret via `api-private.php`.
 - **Disable the feature that generates it** (e.g. `popularity.enabled: false`).
 - **Consciously accept the churn** when the value is genuinely low-stakes
  (`popularity.salt` is an IP-hashing salt, not a credential).
 Do **not** keep stripping the value from the tracked file — bidirectional sync
 brings it right back on the next save.
 ### Before / after: what sticks and what doesn't
 **A standalone file gitignored + untracked sticks.** For `security-private.php`
 (a file that contains *only* the secret) or the whole `user/env/` tree:
 ```bash
 git -C user rm --cached -r config/security-private.php   # or: rm --cached -r env
 printf '/config/security-private.php\n/env/\n' >> user/.gitignore
 ```
 This works permanently. The path is no longer tracked, so the sync's add-set
 skips it forever. The fix sticks because the secret owns its own gitignored path.
 **A key inside a tracked functional file — stripping the line does NOT stick.**
 For `popularity.salt` inside `api.yaml`, deleting just the `salt:` line and
 committing looks clean locally, but Grav re-appends it at runtime and the next
 sync re-commits and re-pushes it. The only durable fixes are the
 companion-private-file pattern or disabling the feature.
 ## Why This Matters
 Two quiet, cross-environmental failure modes:
 1. **The folder-scope illusion.** Assuming "only `pages/config/themes` sync" is a
   security control leads you to leave secrets in `env/` or `accounts/` unignored
   — and a single Admin save on prod pushes them to a shared repo. This actually
   happened here. The only defensible mental model is *gitignore is the boundary*;
   enumerate the un-ignored set, not the "un-foldered" set.
 2. **The boomerang.** A "cleanup" commit that strips a secret from a *tracked*
   file looks done locally but silently reappears upstream on the next content
   save, because the plugin regenerates it and the sync re-commits it.
 Getting both right is what lets you answer "is bidirectional sync safe?" honestly.
 The answer is **yes, once `user/.gitignore` actually excludes every sensitive
 path** — `env/`, the per-install `*-private.php` files, `security.yaml`,
 `versions.yaml`, and any account file that must stay server-local — and once every
 runtime-regenerated value either lives in its own gitignored file or is a
 consciously-accepted low-stakes churn. It is emphatically **not** safe on the
 strength of folder scoping alone.
 ## When to Apply
 - **Enabling or auditing bidirectional git-sync** on a server that authors
  content. Enumerate the round-trippable set as *everything under `user/` not
  matched by `.gitignore`* — then confirm no secret is in it.
 - **Deciding where a new secret or per-install generated value should live.**
  Standalone gitignored file for anything sensitive; never a key inside a shared
  functional YAML; never "outside the folders" as the sole justification.
 - **Reviewing a "stop tracking this secret" cleanup** for whether it will stick:
  is the secret in its own gitignored path (holds) or a line inside a tracked
  functional file that something regenerates (boomerangs)?
 ## Examples
 **The leak (what "folder scope is safe" cost).** With `folders: pages, config,
 themes` and `/env/` **not** gitignored, prod's auto-commit `9337003` pushed
 `user/env/intotheeast.com/config/**` (JWT, CSRF salt, git-sync token + webhook
 secret), `accounts/mischa.yaml`, `system.yaml`, and `post-form.md` to Gitea —
 all outside the configured folders. Fix: gitignore + untrack `/env/`, regenerate
 the JWT/CSRF salts, **rotate the token and webhook secret**.
 **Safe after remediation.** Same bidirectional config, but now `user/.gitignore`
 excludes `/env/`, `config/plugins/git-sync.yaml`, `config/plugins/api-private.php`,
 `config/security.yaml`, `config/security-private.php`, `config/versions.yaml`.
 Running each secret through *is-it-gitignored*: all sensitive paths are excluded →
 none is in the round-trippable set. Verified: prod's `git status` shows only the
 intended tracked content, and no boomerang/secret commit lands on the remote.
 **Boomerang example.** `popularity.salt` in the tracked `api.yaml` regenerates
 per-install and re-commits under sync. The fix that *sticks* is the
 companion-private-file pattern or `popularity.enabled: false` — **not** stripping
 the `salt:` line.
 ## Related
 - `docs/solutions/conventions/grav-plugin-config-must-be-tracked-override.md` — the
  origin of the "functional plugin config in tracked `user/config/plugins/`,
  secrets/per-install values in gitignored `*-private.php`" rule. That doc covers
  *where config must live to deploy*; this doc covers *why gitignore — not folder
  scope — is the sync boundary, and why a runtime-written tracked value boomerangs*.
 - `docs/working/git-sync-notes.md` — operational notes on git-sync's synced
  folders and the per-environment tree. Corrected in the same 2026-07-05 pass to
  drop the "env/ is outside the sync scope so it's safe" claim.
 - `docs/solutions/integration-issues/stale-grav-version-blocks-api-plugin-install.md`
  — adjacent context from the same Grav production cutover, different failure mode.
@@ -82,4 +82,5 @@ The failure is silent and per-environment: it works on the developer's machine (
 - `docs/solutions/integration-issues/stale-grav-version-blocks-api-plugin-install.md` — the config gap documented here was the *other* latent problem surfaced in that same investigation: the `api` plugin also had to be *installed* first before any config could take effect. The install gap (GPM version floor) and this config-tracking gap compounded each other on the fresh prod environment.
 - `docs/working/git-sync-notes.md` — the related third config location: on prod, Grav Admin saves config into the per-environment tree `user/env/<host>/config/`, which is *also* untracked. Same "config that doesn't reach the repo" family.
 - `docs/solutions/architecture-patterns/git-sync-secret-exposure-and-tracked-file-boomerang.md` — the sync-boomerang consequence of this rule: a per-install value that a plugin regenerates into a *tracked* functional config file (e.g. `popularity.salt` in `api.yaml`) re-commits itself and ping-pongs across environments under bidirectional git-sync. The `*-private.php` companion pattern this doc establishes is exactly the durable fix.
 - **CLAUDE.md §0 (plugin-management model):** only `pages/`, `config/`, `accounts/`, `themes/` are tracked in the `user/` repo; `plugins/` and `data/` are gitignored and GPM-managed. That tracking boundary is exactly why functional config must live under `config/plugins/`, not in the plugin's own folder.
@@ -0,0 +1,100 @@
 ---
 title: "Grav plugin won't enable because its code is missing while its config persists in the env tree"
 date: 2026-07-05
 category: integration-issues
 module: git-sync
 problem_type: integration_issue
 component: tooling
 severity: high
 symptoms:
  - "git-sync plugin will not enable on prod despite enabled: true in its config"
  - "Plugin does not appear / cannot be toggled on in the Grav Admin UI"
  - "Config-level fix attempts (editing plugin YAML) have no effect"
  - "make remote-gpm-install-prod PKG=git-sync reports a FRESH install, not 'already installed'"
 root_cause: incomplete_setup
 resolution_type: dependency_update
 related_components:
  - documentation
  - development_workflow
 tags:
  - grav
  - git-sync
  - gpm
  - plugin-management
  - env-config
  - config-without-code
  - troubleshooting-order
  - remote-only-plugin
 ---
 # Grav plugin won't enable because its code is missing while its config persists in the env tree
 ## Problem
 After remediating an unrelated git-sync secret leak on prod (intotheeast.com, Grav 2.0.7 on a DirectAdmin/Apache shared host), the operator went to re-enable the `git-sync` plugin. Setting `enabled: true` in its config had no effect: the plugin would not appear as enabled in the Grav Admin plugins UI, and toggling it on manually in the browser did not take either. It looked fully "configured" — the config file was right there — but the plugin was inert and sync never ran.
 The trap is that the plugin's config file existed (in the per-host env tree at `user/env/intotheeast.com/config/plugins/git-sync.yaml`), so any inspection that reads only config concluded the plugin was present and just needed enabling. The actual problem was one layer down: the plugin's **code** was missing from `user/plugins/git-sync/`. A Grav plugin cannot load or enable without its code on disk, no matter what its config says.
 ## Symptoms
 - The git-sync plugin does not appear as enabled, and cannot be enabled, in the Grav Admin UI — despite `enabled: true` being present in its config.
 - Toggling `enabled` in config, or flipping the toggle in the Admin UI, produces no working plugin. Sync does not run.
 - The plugin's config file DOES exist (in the per-environment tree `user/env/intotheeast.com/config/plugins/git-sync.yaml`), so the plugin appears "present" whenever only the config is inspected — masking the real state.
 ## What Didn't Work
 1. **Setting / ensuring `enabled: true` in the git-sync config.** No effect. Config was never the problem.
 2. **Enabling the plugin manually in the Admin UI.** The toggle wouldn't take.
 Both failed attempts operate on the **config** layer. But the plugin's **code** was absent from `user/plugins/git-sync/`, and Grav can't load a plugin without its code. Grav (and any tooling that reads the config tree) reports a plugin as "configured" purely from the presence of its config file, which masks the absence of code. Diagnosing and poking at the config layer could never fix a missing-code problem — and guessing at config changes before running a simple `ls` on the plugin directory cost real time here.
 ## Solution
 First, run the decisive diagnostic on the server — confirm whether the plugin code actually exists *before* touching config:
 ```
 ls -la $WEBROOT/user/plugins/git-sync/    # empty/absent  => missing code, reinstall
 ```
 (In this project, do that via a make target or an ssh one-liner the user runs — never raw SSH by the assistant. All server ops go through `make remote-*`.)
 With the directory confirmed empty/absent, reinstall the plugin's code via GPM:
 ```
 make remote-gpm-install-prod PKG=git-sync   # GPM fresh-installs Git Sync v3.4.4
 ```
 That make target runs, on the server:
 ```
 php bin/gpm index -f && php bin/gpm install git-sync -y && php bin/grav clearcache
 ```
 The install output read **"Preparing to install Git Sync [v3.4.4] ... Success!"** — a **fresh** install, not "already installed." That fresh-install line is exactly what confirmed the code had been absent all along. After the reinstall plus cache clear, the plugin enabled and sync worked.
 ## Why This Works
 Grav resolves a plugin from **two independent locations**:
 - **Code** at `user/plugins/<name>/` — installed by GPM. Note `user/plugins/` is gitignored (`/plugins/*`) and is NOT tracked by the content repo.
 - **Config** — the tracked `user/config/plugins/<name>.yaml` and/or the per-host `user/env/<host>/config/plugins/<name>.yaml`.
 These two can **desync**: config can exist with no code behind it. Config alone makes the plugin look present to any tool that only reads config, but the plugin stays inert until its code is on disk. GPM reinstall restores the code; `clearcache` makes Grav re-scan and pick it up.
 A project-specific amplifier made this worse: `git-sync` is a **remote-only, GPM-managed** plugin. It is deliberately NOT in `plugins.txt`, so `make install-plugins` and the normal `make remote-install` flow do **not** restore it. Only an explicit `php bin/gpm install git-sync` (via `make remote-gpm-install-prod PKG=git-sync`) does. So when its code goes missing, it does not self-heal through the standard install path — you must reinstall it explicitly.
 How the code went missing here is **unconfirmed**. It happened around the git-sync secret-leak remediation, but the exact step that wiped `user/plugins/git-sync/` was not established — don't assume a specific cause.
 ## Prevention
 - **Check the code layer before the config layer.** When a Grav plugin "won't enable" and config toggles do nothing, FIRST verify the code exists: `ls user/plugins/<name>/` on the server. Config-without-code is the failure class; the empty directory is the tell.
 - **Enumerate both layers in all locations when diagnosing.** Plugins have a code layer (`user/plugins/<name>/`) and a config layer, and on prod the config can live in the env tree (`user/env/<host>/config/plugins/<name>.yaml`) and persist completely independently of the code. Remember: once `user/env/<host>/` exists, Grav Admin writes ALL config there, so always check both `user/config/...` and the env path (env wins).
 - **Know which plugins are remote-only.** The 3-category model: GPM-via-`plugins.txt` (admin2 / api / flex-objects), custom-in-repo (cache-on-save / story-blocks), and remote-only (git-sync — never in `plugins.txt`). Remote-only plugins are NOT restored by the standard install/content flows, so reinstall them explicitly via GPM after any operation that could have wiped `user/plugins/`.
 - **Diagnose actual state before proposing config fixes.** An `ls` is cheaper than a guess. Establishing that the code was missing would have pointed straight at the reinstall instead of a round of config poking.
 ## Related
 - `docs/solutions/integration-issues/stale-grav-version-blocks-api-plugin-install.md` — **closest sibling.** Same family (a plugin non-functional on prod, resolved by a GPM install + cache clear), same 2026-07-04/05 cutover context, same `plugins.txt` / `make remote-*` / GPM machinery. **Distinct trigger:** there, GPM refuses to *offer* the plugin because the installed core is below the version floor; here, the plugin's *code folder is simply missing* while its config persists (config-without-code desync). Two different ways a plugin ends up absent/inert on prod.
 - `docs/solutions/architecture-patterns/git-sync-secret-exposure-and-tracked-file-boomerang.md` — same module (git-sync) and explains **why the config survived without code**: Grav Admin writes `git-sync.yaml` into the per-environment tree `user/env/<host>/config/plugins/`, which is untracked/gitignored and not part of the plugin package. The orphaned config here is the flip side of that env-tree behavior.
 - `docs/solutions/conventions/grav-plugin-config-must-be-tracked-override.md` — establishes the plugin **code (GPM/gitignored) vs config (tracked override / env tree)** split that this bug exploits. This doc is a concrete failure of that split going the other way: config present (in the env tree), code absent.
 - `docs/working/git-sync-notes.md` — operational notes on git-sync's per-environment tree, where `git-sync.yaml` lives server-only. Context for where the orphaned config resided.
@@ -81,7 +81,7 @@ GPM (Grav Package Manager) only offers a plugin version whose declared Grav requ
 - **Keep `.env.<env>` `GRAV_VERSION` current.** It is the version a *fresh* `make remote-install-<env>` bakes in; a stale value silently installs an old core. After any core upgrade, bump the env file too — self-upgrade only moves the running server. Note this is a *third* version-authority surface alongside `user/config/system.yaml` `gpm.releases` (channel) and `plugins.txt` — they must stay in sync.
 - **When GPM says "package not found" for a package you know is on your channel, check the target's Grav core version first** (`php bin/grav --version` on the server, or `make remote-diag-<env>`). GPM filters by core compatibility; "not found" often means "no version compatible with your core," not "missing from the index." `gpm index -f` will not help.
- **Don't trust a top-level install "Success" to mean dependencies installed.** A fresh install can leave a plugin's declared dependency unsatisfied (here `admin2` installed but its `api` dependency didn't). Verify with `ls user/plugins/<dependency>`.
+- **Don't trust a top-level install "Success" to mean dependencies installed.** A fresh install can leave a plugin's declared dependency unsatisfied (here `admin2` installed but its `api` dependency didn't). Verify with `ls user/plugins/<dependency>`. The same `ls` guards a *second*, distinct way a plugin ends up non-functional: its **code folder can be missing while its config persists** (e.g. in the per-host env tree), so it looks configured but never loads. Checking `ls user/plugins/<name>` catches both the missing-dependency and the config-without-code cases — see `grav-plugin-config-without-code-wont-enable.md`.
 - **Know the Admin2 ⇄ api coupling.** Admin2 authenticates via the api plugin's `/api/v1` endpoints; a missing or unrouted api plugin makes admin login fail *silently* (login POST 404s, nothing logged). A quick `curl /api/v1/pages` expecting `401` (not `404`) is a good post-deploy smoke check.
 ## Related
@@ -92,3 +92,7 @@ This is one of three independent gotchas from the same **2026-07-04 Grav 2.0.4 p
 - `docs/solutions/test-failures/new-user-grants-api-not-admin-on-admin2.md` — sibling: an authenticated account is denied an admin-gated page because `login new-user` auto-detect granted `api.*` but not `admin.*`. Different root cause (permission provisioning), same admin2/api area.
 - `docs/working/plans/2026-07-04-grav-2.0.4-upgrade.md` — the upgrade plan whose Global Constraints spell out the GPM version floors (`grav >=2.0.4`, `api >=1.0.6`) that cause the "package not found" on an rc core.
 - `docs/solutions/conventions/grav-plugin-config-must-be-tracked-override.md` — the *other* latent problem from this same investigation: the api plugin's functional config (`enabled` / `route` / `session_enabled`) must live in the tracked `user/config/plugins/api.yaml` to deploy at all. Necessary but not sufficient here (the plugin must be installed first), but a durable convention in its own right.
 A closely-related **sibling in the "plugin absent/non-functional on prod" family** (from the 2026-07-05 follow-up, not one of the three cutover gotchas above):
 - `docs/solutions/integration-issues/grav-plugin-config-without-code-wont-enable.md` — same outcome (a plugin inert on prod, fixed by a GPM install + cache clear), **different trigger**: there, GPM won't *offer* the plugin because the core is below the version floor; there, the plugin's *code folder is simply missing* while its config persists in the env tree (config-without-code desync). Same `ls user/plugins/<name>` smoke check flushes both out.
@@ -16,12 +16,17 @@ user/config/plugins/git-sync.yaml                       ← NOT here
 Why this matters:
- **Server-only, not synced, not committed.** `user/env/` is outside the
+- **⚠️ `user/env/` is NOT safe unless gitignored — it is NOT scoped out by the
-  content repo's tracked folders (`pages`/`config`/`accounts`/`themes`) and is
+  `folders` setting.** An earlier version of this note claimed `user/env/`
-  not one of git-sync's synced folders (`pages`/`config`/`themes`). So config
+  "never reaches Gitea" because it is outside git-sync's synced folders. **That
-  saved via Admin *on the server* never reaches Gitea or local. This is ideal
+  is wrong and caused a live secret leak (2026-07-05).** git-sync's auto-commit
-  for the git-sync token (it stays server-only) but means **prod Admin config
+  stages files *outside* the configured `folders`; on prod it pushed the whole
-  edits silently diverge** — author durable config in the repo, not prod Admin.
+  `user/env/intotheeast.com/config/` tree — JWT secret, CSRF salt, **and the
  git-sync token + webhook secret** — to Gitea. The fix was to **gitignore
  `/env/`** (commit `6e8eadb`). So: prod Admin config edits stay server-only
  *only because `/env/` is now gitignored*, not because of folder scope. Author
  durable config in the repo, not prod Admin. Full analysis:
  `docs/solutions/architecture-patterns/git-sync-secret-exposure-and-tracked-file-boomerang.md`.
 - **Look in both places.** When inspecting/toggling server config, check
  `user/config/plugins/<name>.yaml` **and**
  `user/env/<host>/config/plugins/<name>.yaml` (env wins).
@@ -48,19 +53,25 @@ If you see the comma-string form on an older version, fix it by editing
 `git-sync.yaml` directly (at whichever path it lives — see above); do not
 re-save folders via the Admin UI on the buggy version.
-## Per-install / secret files — must be gitignored (never synced)
+## Per-install / secret files — must be gitignored (gitignore is the boundary)
-git-sync syncs the `config/` folder, so any per-install or secret file tracked
+git-sync's auto-commit stages **everything under `user/` that is not
-there would get pushed to Gitea and pollute every environment. Keep these out
+gitignored** — the `folders` setting does *not* scope the commit add-set (a
-of the content repo (all in `user/.gitignore`):
+2026-07-05 leak proved this by pushing `user/env/**`, outside the configured
 folders). So `.gitignore` — not folder scope — is the only thing keeping a
 per-install or secret file off Gitea. Keep all of these gitignored in
 `user/.gitignore`:
-| File | Why |
+| Path | Why |
 |---|---|
 | `env/` | **whole per-host env tree** — holds the live git-sync token, JWT secret, CSRF salt + all server-side Admin config. Gitignored + untracked 2026-07-05 (commit `6e8eadb`) after it leaked to Gitea. NOT safe on folder scope alone. |
 | `config/plugins/git-sync.yaml` | encrypted token; server-specific (also lives at env path on prod) |
 | `config/plugins/api-private.php` | API JWT secret |
 | `config/security.yaml` | Grav nonces/salts (legacy location) |
 | `config/versions.yaml` | per-install Grav schema-migration state — differs per env (dev 2.0.4, prod 2.0.7); Grav regenerates it. Untracked 2026-07-04. |
-| `config/security-private.php` | **TODO:** committed salt secret; should be gitignored like `api-private.php` (deferred — untracking resets server sessions) |
+| `config/security-private.php` | CSRF/nonce + admin rate-limit signing salt; gitignored + untracked 2026-07-05 (commit 2840018). Each env keeps its own; untracking regenerates prod's salt (one-time admin re-login). |
 > **Why a key inside a *tracked* config file (e.g. `popularity.salt` in `api.yaml`) can't just be stripped** — it regenerates at runtime and boomerangs back via git-sync's `git add -A`. See `docs/solutions/architecture-patterns/git-sync-secret-exposure-and-tracked-file-boomerang.md` for the full round-trippable-set model.
 ## git-sync config summary (prod, 2026-07-04)