# Upgrade & Deploy Cycle: local → test → prod This runbook is the repeatable procedure for shipping a Grav upgrade or any server-affecting change (core version, plugins, config, theme) through the three environments. It was distilled from the 2026-07 Grav 2.0.4→2.0.7 cutover, where every production surprise traced back to one of the desyncs this procedure now forces you to check. **Governing principle:** `test` is a **full dress rehearsal of `prod`** — same config, same `-test`/`-prod` make targets, same order. A gotcha only gets caught on test if test is a faithful mirror of prod. Do not shortcut test. All server operations go through `make remote-*` targets (never raw SSH — the targets build the SSH connection from `.env.`, which must never be read directly). Every `remote-*` target has `-test` and `-prod` variants; a bare target fails via `guard-env`. --- ## The mental model: three places state lives Every failure in the reference cutover was a desync between these three layers. Before and after each deploy step, ask: *are they in sync?* | Layer | Location | Synced by | Failure mode | |---|---|---|---| | Plugin **code** | `user/plugins//` | GPM only (gitignored `/plugins/*`) | can vanish while config remains → plugin won't enable | | **Repo config** | `user/config/…` | `content-push` / git-sync | holds GPM channel + is where the version floor bites | | **Host config** | `user/env//config/…` | nothing — server-only | not restored on fresh install; must be re-applied; **must be gitignored** | Referenced gotcha docs: - `docs/solutions/integration-issues/grav-plugin-config-without-code-wont-enable.md` — code-vs-config desync. - `docs/solutions/integration-issues/stale-grav-version-blocks-api-plugin-install.md` — stale `GRAV_VERSION` / version floor. - `docs/solutions/architecture-patterns/git-sync-secret-exposure-and-tracked-file-boomerang.md` — gitignore is the sync boundary; env-tree leak. - `docs/solutions/conventions/grav-plugin-config-must-be-tracked-override.md` — plugin config must live in the tracked override. --- ## Phase 0 — Local (author + prove the change) 1. Make the change in the repo: - GPM channel: `gpm.releases: stable` in `user/config/system.yaml` (authoritative; reaches servers via content pull, so it must be right **before** any server GPM op). - `plugins.txt` — the GPM-managed set only. **Never** add `git-sync` (it is remote-only). - Prod-only overrides (Twig cache/debug, `debugger.shutdown.close_connection: false`) in `deploy/env/prod/system.yaml` — **never** commit prod values into `user/config/system.yaml`. - **Bump `GRAV_VERSION` in `.env.test` and `.env.prod`** to the target version. A stale value here installs the wrong core (an rc), which then blocks the `api` plugin and 404s admin. 2. `make build-assets` if you touched `js/src/*` (never hand-edit the bundled `js/*.js`). 3. Run the dev server (`docker compose … up`) and the Playwright suite. 4. Pre-flight assertions: - `gpm.releases` is `stable`. - `plugins.txt` is correct and does **not** contain `git-sync`. - No prod Twig values leaked into the committed `system.yaml`. 5. Commit. `make content-push`. --- ## Phase 1 — Test (the rehearsal — catch things here) ### Pre-flight - `make remote-git-sync-disable-test` **before any content reset.** This is the safety catch for the whole window: it stops a half-migrated state (e.g. a fresh install-time `versions.yaml`) from auto-committing and pushing on the first sync. ### Apply — in this fixed order ``` make remote-fetch-content-test # 1. clean-reset synced folders to repo state make remote-upgrade-grav-test # 2. gpm self-upgrade (rewrites schema — expect drift) make remote-update-plugins-test # 3. gpm update the plugins.txt set (auto-applies deploy/patches/) make remote-gpm-install-test PKG=git-sync # 4. EXPLICITLY (re)install each remote-only plugin make remote-apply-env-test # 5. re-deploy the env override (not synced; gone after install) ``` Why each matters: - **Step 3** re-applies `deploy/patches/*.patch` automatically (it chains `remote-apply-plugin-patches`). GPM install/update lays down **pristine** third-party plugins, wiping local fixes to git-ignored `user/plugins/` — the patch step restores them. Content pulls (step 1) do **not** touch `plugins/`, so the patch only needs re-applying after a GPM op, not after every sync. Run `make remote-apply-plugin-patches-test` standalone if you ever GPM-install outside this sequence. Requires the `patch` tool on the server. See `deploy/patches/README.md`. - **Step 4** is non-optional even if git-sync "was already there" — remote-only plugins are not in `plugins.txt`, so nothing in steps 1–3 restores them. If the code is missing, the plugin is inert despite valid config. - **Step 5** re-writes `user/env//config/…` from `deploy/env//`. The env tree is not synced by anything, so a fresh install loses it until you re-apply. ### Verify (smoke checklist — this is the payoff) - **Code present, not just config:** `ls user/plugins//` for every expected plugin (especially `git-sync`). An empty/absent dir = reinstall (step 4). *(Do this via an ssh one-liner you run, or `make remote-diag-test`.)* - **Plugin patches applied:** confirm the add-page-by-form fix survived the GPM op — `grep -c toArray user/plugins/add-page-by-form/add-page-by-form.php` should be ≥1 (0 = pristine, re-run `make remote-apply-plugin-patches-test`). Functional check: edit a journal entry and add a photo — a pristine plugin 500s on save. - **HTTP:** `/` → 200, `/admin` → 200, `/api/v1/pages` → 401, `/gpx-manager` → 200. Watch for the double-`Content-Encoding` garbage page (fix: `debugger.shutdown.close_connection: false` in the env override — already in `deploy/env/prod/system.yaml`). - **Post smoke test:** submit one entry via `/post` and confirm it appears in the trip feed immediately. This proves the `cache-on-save` plugin works with prod caching on. - **Config drift:** `make remote-diag-test` — diff server config against the repo. Fold any *intended* schema migration (e.g. the Twig-3 `strict_mode` flags a `self-upgrade` writes) back into `user/config/system.yaml`, or the next `fetch-content` reverts it. ### Re-enable + prove sync - `make remote-git-sync-enable-test`. - Confirm a content push round-trips to the server, **and** that no secret/boomerang commit lands on Gitea. Verify `/env/` is gitignored so the env tree (which holds the token, JWT, CSRF salt) can never enter the sync add-set. --- ## Phase 2 — Prod (repeat identically — should be mechanical) Run the **exact same sequence** with `-prod` targets. Because test rehearsed it, prod holds no surprises. Differences to layer on: - Optional: `make remote-maintenance-on-prod` at the start, `remote-maintenance-off-prod` at the end, for a clean window. - Confirm secrets are valid/rotated and `/env/` is gitignored **before** `remote-git-sync-enable-prod`. Re-enable git-sync **last**. - After a clean cutover, bump the outer-repo submodule pin to the finished `user/` commit — and **push `user/` before the outer repo** (the superproject references a child SHA that must already exist upstream). ``` make remote-git-sync-disable-prod make remote-fetch-content-prod make remote-upgrade-grav-prod make remote-update-plugins-prod make remote-gpm-install-prod PKG=git-sync make remote-apply-env-prod # ── smoke checklist (same as test) ── make remote-git-sync-enable-prod ``` For a first-time / from-scratch prod bring-up, `make remote-install-prod` does the full install; then still run `remote-apply-env-prod` and the smoke checklist, and reinstall remote-only plugins explicitly. --- ## Rollback & safety - **git-sync stays disabled through the whole apply window** on each host — it is the catch that prevents a half-migrated state from auto-pushing. - **Content** is a git repo: a bad content deploy is recoverable with `make remote-fetch-content-` back to a known commit. - **Core + plugins** are GPM-reinstallable (`remote-upgrade-grav`, `remote-update-plugins`, `remote-gpm-install PKG=…`). - The one thing tooling cannot regenerate is the un-synced `user/env//` tree — its source of truth is `deploy/env//`, so keep that current and re-apply with `remote-apply-env-`. --- ## One-line invariants (the through-line) 1. `test` is config-identical to `prod`, run with the same targets in the same order. 2. Verify the **code layer** (`ls user/plugins//`), not just config, on every deploy. 3. Reinstall **remote-only** plugins (git-sync) explicitly — nothing else restores them. 4. `GRAV_VERSION` in `.env.` and `gpm.releases: stable` are correct **before** any server GPM op. 5. Re-apply the **env override** after every install; keep `/env/` **gitignored**. 6. git-sync **off** during the window, **on** last; confirm the round-trip carries no secrets. 7. Diagnose actual state before changing config — an `ls` or `remote-diag` beats a guess.