m038andClaude Opus 4.8
28e57f62c2
A `reset --hard` content deploy leaves Grav's compiled-Twig/page cache stale, so the first visitor pays the recompile. `remote-warmup` clears the cache then crawls the public site (homepage + trips listing + every trip page linked from it) to pre-render pages. Grav has no native warmup command, so it's an HTTP crawl — which also doubles as a smoke test (non-2xx pages flagged). Wired into REMOTE_TARGETS (-test/-prod variants) and added as the final step in both deploy-cycle.md phases. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RDS6t8wcpbwKvvrxykVQ5K
152 lines
10 KiB
Markdown
152 lines
10 KiB
Markdown
# Upgrade & Deploy Cycle: local → test → prod
|
||
|
||
This runbook is the repeatable procedure for shipping a Grav upgrade or any
|
||
server-affecting change (core version, plugins, config, theme) through the three
|
||
environments. It was distilled from the 2026-07 Grav 2.0.4→2.0.7 cutover, where
|
||
every production surprise traced back to one of the desyncs this procedure now
|
||
forces you to check.
|
||
|
||
**Governing principle:** `test` is a **full dress rehearsal of `prod`** — same
|
||
config, same `-test`/`-prod` make targets, same order. A gotcha only gets caught
|
||
on test if test is a faithful mirror of prod. Do not shortcut test.
|
||
|
||
All server operations go through `make remote-*` targets (never raw SSH — the
|
||
targets build the SSH connection from `.env.<env>`, which must never be read
|
||
directly). Every `remote-*` target has `-test` and `-prod` variants; a bare
|
||
target fails via `guard-env`.
|
||
|
||
---
|
||
|
||
## The mental model: three places state lives
|
||
|
||
Every failure in the reference cutover was a desync between these three layers.
|
||
Before and after each deploy step, ask: *are they in sync?*
|
||
|
||
| Layer | Location | Synced by | Failure mode |
|
||
|---|---|---|---|
|
||
| Plugin **code** | `user/plugins/<name>/` | GPM only (gitignored `/plugins/*`) | can vanish while config remains → plugin won't enable |
|
||
| **Repo config** | `user/config/…` | `content-push` / git-sync | holds GPM channel + is where the version floor bites |
|
||
| **Host config** | `user/env/<host>/config/…` | nothing — server-only | not restored on fresh install; must be re-applied; **must be gitignored** |
|
||
|
||
Referenced gotcha docs:
|
||
- `docs/solutions/integration-issues/grav-plugin-config-without-code-wont-enable.md` — code-vs-config desync.
|
||
- `docs/solutions/integration-issues/stale-grav-version-blocks-api-plugin-install.md` — stale `GRAV_VERSION` / version floor.
|
||
- `docs/solutions/architecture-patterns/git-sync-secret-exposure-and-tracked-file-boomerang.md` — gitignore is the sync boundary; env-tree leak.
|
||
- `docs/solutions/conventions/grav-plugin-config-must-be-tracked-override.md` — plugin config must live in the tracked override.
|
||
|
||
These three layers describe the **servers**. Locally there is a fourth: the Grav
|
||
**core** is baked into the Docker **image** (`Dockerfile`), not in any layer above —
|
||
so the local core upgrades by an image rebuild, never by the `gpm self-upgrade` the
|
||
servers use. See `docs/solutions/tooling-decisions/upgrade-local-grav-core-rebuild-docker-image.md`.
|
||
|
||
---
|
||
|
||
## Phase 0 — Local (author + prove the change)
|
||
|
||
1. Make the change in the repo:
|
||
- GPM channel: `gpm.releases: stable` in `user/config/system.yaml` (authoritative; reaches servers via content pull, so it must be right **before** any server GPM op).
|
||
- `plugins.txt` — the GPM-managed set only. **Never** add `git-sync` (it is remote-only).
|
||
- Prod-only overrides (Twig cache/debug, `debugger.shutdown.close_connection: false`) in `deploy/env/prod/system.yaml` — **never** commit prod values into `user/config/system.yaml`.
|
||
- **Bump `GRAV_VERSION` in `.env.test` and `.env.prod`** to the target version. A stale value here installs the wrong core (an rc), which then blocks the `api` plugin and 404s admin. This governs fresh **remote** installs only.
|
||
- **If the core version is changing, upgrade the local dev core too** so you prove the change against the target version — bump the hardcoded `grav-admin-v<ver>.zip` URL in `Dockerfile`, `docker compose build grav`, then `docker rm -f intotheeast_grav && docker compose up -d grav`. The local core is baked into the image, so `.env GRAV_VERSION` does *not* touch it and an in-container `gpm self-upgrade` is non-durable. See `docs/solutions/tooling-decisions/upgrade-local-grav-core-rebuild-docker-image.md`.
|
||
2. `make build-assets` if you touched `js/src/*` (never hand-edit the bundled `js/*.js`).
|
||
3. Run the dev server (`docker compose … up`) and the Playwright suite.
|
||
4. Pre-flight assertions:
|
||
- `gpm.releases` is `stable`.
|
||
- `plugins.txt` is correct and does **not** contain `git-sync`.
|
||
- No prod Twig values leaked into the committed `system.yaml`.
|
||
5. Commit. `make content-push`.
|
||
|
||
---
|
||
|
||
## Phase 1 — Test (the rehearsal — catch things here)
|
||
|
||
### Pre-flight
|
||
|
||
- `make remote-git-sync-disable-test` **before any content reset.** This is the safety catch for the whole window: it stops a half-migrated state (e.g. a fresh install-time `versions.yaml`) from auto-committing and pushing on the first sync.
|
||
|
||
### Apply — in this fixed order
|
||
|
||
```
|
||
make remote-fetch-content-test # 1. clean-reset synced folders to repo state
|
||
make remote-upgrade-grav-test # 2. gpm self-upgrade (rewrites schema — expect drift)
|
||
make remote-update-plugins-test # 3. gpm update the plugins.txt set (auto-applies deploy/patches/)
|
||
make remote-gpm-install-test PKG=git-sync # 4. EXPLICITLY (re)install each remote-only plugin
|
||
make remote-apply-env-test # 5. re-deploy the env override (not synced; gone after install)
|
||
make remote-warmup-test # 6. clear + warm cache — a reset deploy leaves it stale
|
||
```
|
||
|
||
> **Always finish a deploy with `remote-warmup-<env>`** — even a content-only
|
||
> deploy. A `reset --hard` (step 1) changes files under Grav without going
|
||
> through it, so the compiled-Twig/page cache is stale and the first visitor
|
||
> eats the recompile. `remote-warmup` clears the cache, then crawls the public
|
||
> pages (homepage + trips listing + every trip page linked from it) to
|
||
> pre-render them. Grav has no native warmup command — this is an HTTP crawl, so
|
||
> it also doubles as a smoke test (a non-2xx on any page is flagged loudly).
|
||
|
||
Why each matters:
|
||
- **Step 3** re-applies `deploy/patches/*.patch` automatically (it chains `remote-apply-plugin-patches`). GPM install/update lays down **pristine** third-party plugins, wiping local fixes to git-ignored `user/plugins/` — the patch step restores them. Content pulls (step 1) do **not** touch `plugins/`, so the patch only needs re-applying after a GPM op, not after every sync. Run `make remote-apply-plugin-patches-test` standalone if you ever GPM-install outside this sequence. Requires the `patch` tool on the server. See `deploy/patches/README.md`.
|
||
- **Step 4** is non-optional even if git-sync "was already there" — remote-only plugins are not in `plugins.txt`, so nothing in steps 1–3 restores them. If the code is missing, the plugin is inert despite valid config.
|
||
- **Step 5** re-writes `user/env/<host>/config/…` from `deploy/env/<env>/`. The env tree is not synced by anything, so a fresh install loses it until you re-apply.
|
||
|
||
### Verify (smoke checklist — this is the payoff)
|
||
|
||
- **Code present, not just config:** `ls user/plugins/<name>/` for every expected plugin (especially `git-sync`). An empty/absent dir = reinstall (step 4). *(Do this via an ssh one-liner you run, or `make remote-diag-test`.)*
|
||
- **Plugin patches applied:** confirm the add-page-by-form fix survived the GPM op — `grep -c toArray user/plugins/add-page-by-form/add-page-by-form.php` should be ≥1 (0 = pristine, re-run `make remote-apply-plugin-patches-test`). Functional check: edit a journal entry and add a photo — a pristine plugin 500s on save.
|
||
- **HTTP:** `/` → 200, `/admin` → 200, `/api/v1/pages` → 401, `/gpx-manager` → 200. Watch for the double-`Content-Encoding` garbage page (fix: `debugger.shutdown.close_connection: false` in the env override — already in `deploy/env/prod/system.yaml`).
|
||
- **Post smoke test:** submit one entry via `/post` and confirm it appears in the trip feed immediately. This proves the `cache-on-save` plugin works with prod caching on.
|
||
- **Config drift:** `make remote-diag-test` — diff server config against the repo. Fold any *intended* schema migration (e.g. the Twig-3 `strict_mode` flags a `self-upgrade` writes) back into `user/config/system.yaml`, or the next `fetch-content` reverts it.
|
||
|
||
### Re-enable + prove sync
|
||
|
||
- `make remote-git-sync-enable-test`.
|
||
- Confirm a content push round-trips to the server, **and** that no secret/boomerang commit lands on Gitea. Verify `/env/` is gitignored so the env tree (which holds the token, JWT, CSRF salt) can never enter the sync add-set.
|
||
|
||
---
|
||
|
||
## Phase 2 — Prod (repeat identically — should be mechanical)
|
||
|
||
Run the **exact same sequence** with `-prod` targets. Because test rehearsed it,
|
||
prod holds no surprises. Differences to layer on:
|
||
|
||
- Optional: `make remote-maintenance-on-prod` at the start, `remote-maintenance-off-prod` at the end, for a clean window.
|
||
- Confirm secrets are valid/rotated and `/env/` is gitignored **before** `remote-git-sync-enable-prod`. Re-enable git-sync **last**.
|
||
- After a clean cutover, bump the outer-repo submodule pin to the finished `user/` commit — and **push `user/` before the outer repo** (the superproject references a child SHA that must already exist upstream).
|
||
|
||
```
|
||
make remote-git-sync-disable-prod
|
||
make remote-fetch-content-prod
|
||
make remote-upgrade-grav-prod
|
||
make remote-update-plugins-prod
|
||
make remote-gpm-install-prod PKG=git-sync
|
||
make remote-apply-env-prod
|
||
make remote-warmup-prod # clear + warm cache; also HTTP-smokes public pages
|
||
# ── smoke checklist (same as test) ──
|
||
make remote-git-sync-enable-prod
|
||
```
|
||
|
||
For a first-time / from-scratch prod bring-up, `make remote-install-prod` does the
|
||
full install; then still run `remote-apply-env-prod` and the smoke checklist, and
|
||
reinstall remote-only plugins explicitly.
|
||
|
||
---
|
||
|
||
## Rollback & safety
|
||
|
||
- **git-sync stays disabled through the whole apply window** on each host — it is the catch that prevents a half-migrated state from auto-pushing.
|
||
- **Content** is a git repo: a bad content deploy is recoverable with `make remote-fetch-content-<env>` back to a known commit.
|
||
- **Core + plugins** are GPM-reinstallable (`remote-upgrade-grav`, `remote-update-plugins`, `remote-gpm-install PKG=…`).
|
||
- The one thing tooling cannot regenerate is the un-synced `user/env/<host>/` tree — its source of truth is `deploy/env/<env>/`, so keep that current and re-apply with `remote-apply-env-<env>`.
|
||
|
||
---
|
||
|
||
## One-line invariants (the through-line)
|
||
|
||
1. `test` is config-identical to `prod`, run with the same targets in the same order.
|
||
2. Verify the **code layer** (`ls user/plugins/<name>/`), not just config, on every deploy.
|
||
3. Reinstall **remote-only** plugins (git-sync) explicitly — nothing else restores them.
|
||
4. `GRAV_VERSION` in `.env.<env>` and `gpm.releases: stable` are correct **before** any server GPM op.
|
||
5. Re-apply the **env override** after every install; keep `/env/` **gitignored**.
|
||
6. git-sync **off** during the window, **on** last; confirm the round-trip carries no secrets.
|
||
7. Diagnose actual state before changing config — an `ls` or `remote-diag` beats a guess.
|