m038andClaude Opus 5
ed6e43ae51
CLAUDE.md is loaded into context on every request, so every line has a recurring cost. Applies one rule to decide what earns its place: keep what changes behaviour (rules and gotchas Claude cannot discover before it acts); extract what merely describes code (Claude reads the code anyway, and prose about code silently drifts). The four stale facts fixed in the previous commit were all in the "describes code" class -- active_trip, the Admin2 version, demo-load's scope, the gitignore list. None were rules. That is the argument for moving this material next to what it documents. Extracted (kept as pointers): - entry-map + trip-feed-col parameter contracts (56 lines) -> reference/ architecture.md "Shared partial contracts". CLAUDE.md keeps only the invariants: single map path, must assign window.tripMap/homeMap, keep trip-feed-col single-purpose, initTripStats depends on MapUtils. - Prod override runbook (49 -> 9 lines) -> guides/deploy-cycle.md "The env override tree", incl. the Twig dev/prod table and WEB_HOST. CLAUDE.md keeps the two behavioural rules: never commit prod values, and Admin on the server writes to the env tree (so check both config paths, env wins). - GPX API routes, session auth and the Blob/FormData upload gotcha -> guides/gpx-manager.md "How the manager is wired". - Trip-switch procedure -> guides/trip-switching.md. CLAUDE.md keeps the one rule that matters: never re-add pageconfig.parent to post-form.md. - Also trimmed the dev-command table and custom-plugin table added in the previous commit; both largely restated the Makefile and blueprints. Fixed the guides being pointed into, so the pointers lead to truth: - trip-switching.md instructed editing a pageconfig.parent that no longer exists -- its whole "two files must be updated together" premise was obsolete and would have reintroduced the desync it warned about. - architecture.md: Grav 2.0.4->2.0.7, Admin2 2.0.10->2.0.12, corrected the posting pipeline to show cache-on-save injecting parent before the write, added entry-actions to the custom-plugin list. - japan-korea-2026 -> denmark-2026 across guides/reference (docs/solutions keeps its historical references intact -- those are incident records). Verified: every markdown link resolves, every referenced section heading exists, and each extracted item was confirmed present in its new home. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
198 lines
13 KiB
Markdown
198 lines
13 KiB
Markdown
# Upgrade & Deploy Cycle: local → test → prod
|
||
|
||
This runbook is the repeatable procedure for shipping a Grav upgrade or any
|
||
server-affecting change (core version, plugins, config, theme) through the three
|
||
environments. It was distilled from the 2026-07 Grav 2.0.4→2.0.7 cutover, where
|
||
every production surprise traced back to one of the desyncs this procedure now
|
||
forces you to check.
|
||
|
||
**Governing principle:** `test` is a **full dress rehearsal of `prod`** — same
|
||
config, same `-test`/`-prod` make targets, same order. A gotcha only gets caught
|
||
on test if test is a faithful mirror of prod. Do not shortcut test.
|
||
|
||
All server operations go through `make remote-*` targets (never raw SSH — the
|
||
targets build the SSH connection from `.env.<env>`, which must never be read
|
||
directly). Every `remote-*` target has `-test` and `-prod` variants; a bare
|
||
target fails via `guard-env`.
|
||
|
||
---
|
||
|
||
## The mental model: three places state lives
|
||
|
||
Every failure in the reference cutover was a desync between these three layers.
|
||
Before and after each deploy step, ask: *are they in sync?*
|
||
|
||
| Layer | Location | Synced by | Failure mode |
|
||
|---|---|---|---|
|
||
| Plugin **code** | `user/plugins/<name>/` | GPM only (gitignored `/plugins/*`) | can vanish while config remains → plugin won't enable |
|
||
| **Repo config** | `user/config/…` | `content-push` / git-sync | holds GPM channel + is where the version floor bites |
|
||
| **Host config** | `user/env/<host>/config/…` | nothing — server-only | not restored on fresh install; must be re-applied; **must be gitignored** |
|
||
|
||
Referenced gotcha docs:
|
||
- `docs/solutions/integration-issues/grav-plugin-config-without-code-wont-enable.md` — code-vs-config desync.
|
||
- `docs/solutions/integration-issues/stale-grav-version-blocks-api-plugin-install.md` — stale `GRAV_VERSION` / version floor.
|
||
- `docs/solutions/architecture-patterns/git-sync-secret-exposure-and-tracked-file-boomerang.md` — gitignore is the sync boundary; env-tree leak.
|
||
- `docs/solutions/conventions/grav-plugin-config-must-be-tracked-override.md` — plugin config must live in the tracked override.
|
||
|
||
These three layers describe the **servers**. Locally there is a fourth: the Grav
|
||
**core** is baked into the Docker **image** (`Dockerfile`), not in any layer above —
|
||
so the local core upgrades by an image rebuild, never by the `gpm self-upgrade` the
|
||
servers use. See `docs/solutions/tooling-decisions/upgrade-local-grav-core-rebuild-docker-image.md`.
|
||
|
||
---
|
||
|
||
## The env override tree (`user/env/<host>/`)
|
||
|
||
Prod needs different Twig settings than dev. These are **never** committed to
|
||
`user/config/system.yaml` — `twig.cache: false` and `debug`/`auto_reload: true`
|
||
are the *intended dev values*, and committing prod values there breaks local
|
||
development for everyone. Instead they ship as a per-environment override via
|
||
Grav's `environment://config`, keyed on the request hostname.
|
||
|
||
| Setting | Dev (committed) | Prod (override) | Why prod differs |
|
||
|---|---|---|---|
|
||
| `twig.cache` | `false` | `true` | Compile templates once and reuse |
|
||
| `twig.debug` | `true` | `false` | No debug functions in prod |
|
||
| `twig.auto_reload` | `true` | `false` | Don't stat templates every request |
|
||
|
||
- **Source of truth:** `deploy/env/prod/system.yaml` (version-controlled).
|
||
- **Deploy:** `make remote-apply-env-prod` — writes it to
|
||
`<webroot>/user/env/<hostname>/config/system.yaml` and clears cache. It
|
||
deep-merges over the committed `system.yaml`.
|
||
- **Hostname segment** defaults to `REMOTE_HOST`; override with `WEB_HOST` in
|
||
`.env.<env>` if Grav sees a different host than the SSH host.
|
||
- **Not restored by anything.** `user/env/` is outside the content repo's tracked
|
||
folders, so `content-push` / git-sync / `remote-fetch-content` do **not** bring
|
||
it back. **Re-run `make remote-apply-env-<env>` after any fresh install.**
|
||
|
||
### Side effect: Admin writes ALL config into the env tree
|
||
|
||
Once `user/env/<hostname>/` exists, Grav's Admin saves **every** config change
|
||
(system *and* plugin) there — e.g. editing a plugin on prod writes
|
||
`user/env/intotheeast.com/config/plugins/<name>.yaml`, **not**
|
||
`user/config/plugins/<name>.yaml`. Consequences:
|
||
|
||
- Config edited via **Admin on the server is server-only**: the env tree is not
|
||
committed and not synced by git-sync (which syncs only `pages`/`config`/
|
||
`themes`), so prod Admin edits silently never reach Gitea or local. This is
|
||
*good* for secrets — `git-sync.yaml` (token), the JWT and CSRF salt safely
|
||
live there — but it means config drift is invisible to the repo.
|
||
- When reading or writing server config, check **both** `user/config/…` and
|
||
`user/env/<host>/config/…` (env wins). Server tooling must search the env path
|
||
first — see `scripts/git-sync-toggle.sh` and `make remote-diag`.
|
||
- Repo-authored config (`user/config/…` via `make content-push`) still applies
|
||
everywhere; the env tree holds only per-host overrides + Admin-on-server edits.
|
||
|
||
Full details: `docs/working/git-sync-notes.md`.
|
||
|
||
---
|
||
|
||
## Phase 0 — Local (author + prove the change)
|
||
|
||
1. Make the change in the repo:
|
||
- GPM channel: `gpm.releases: stable` in `user/config/system.yaml` (authoritative; reaches servers via content pull, so it must be right **before** any server GPM op).
|
||
- `plugins.txt` — the GPM-managed set only. **Never** add `git-sync` (it is remote-only).
|
||
- Prod-only overrides (Twig cache/debug, `debugger.shutdown.close_connection: false`) in `deploy/env/prod/system.yaml` — **never** commit prod values into `user/config/system.yaml`.
|
||
- **Bump `GRAV_VERSION` in `.env.test` and `.env.prod`** to the target version. A stale value here installs the wrong core (an rc), which then blocks the `api` plugin and 404s admin. This governs fresh **remote** installs only.
|
||
- **If the core version is changing, upgrade the local dev core too** so you prove the change against the target version — bump the hardcoded `grav-admin-v<ver>.zip` URL in `Dockerfile`, `docker compose build grav`, then `docker rm -f intotheeast_grav && docker compose up -d grav`. The local core is baked into the image, so `.env GRAV_VERSION` does *not* touch it and an in-container `gpm self-upgrade` is non-durable. See `docs/solutions/tooling-decisions/upgrade-local-grav-core-rebuild-docker-image.md`.
|
||
2. `make build-assets` if you touched `js/src/*` (never hand-edit the bundled `js/*.js`).
|
||
3. Run the dev server (`docker compose … up`) and the Playwright suite.
|
||
4. Pre-flight assertions:
|
||
- `gpm.releases` is `stable`.
|
||
- `plugins.txt` is correct and does **not** contain `git-sync`.
|
||
- No prod Twig values leaked into the committed `system.yaml`.
|
||
5. Commit. `make content-push`.
|
||
|
||
---
|
||
|
||
## Phase 1 — Test (the rehearsal — catch things here)
|
||
|
||
### Pre-flight
|
||
|
||
- `make remote-git-sync-disable-test` **before any content reset.** This is the safety catch for the whole window: it stops a half-migrated state (e.g. a fresh install-time `versions.yaml`) from auto-committing and pushing on the first sync.
|
||
|
||
### Apply — in this fixed order
|
||
|
||
```
|
||
make remote-fetch-content-test # 1. clean-reset synced folders to repo state
|
||
make remote-upgrade-grav-test # 2. gpm self-upgrade (rewrites schema — expect drift)
|
||
make remote-update-plugins-test # 3. gpm update the plugins.txt set (auto-applies deploy/patches/)
|
||
make remote-gpm-install-test PKG=git-sync # 4. EXPLICITLY (re)install each remote-only plugin
|
||
make remote-apply-env-test # 5. re-deploy the env override (not synced; gone after install)
|
||
make remote-warmup-test # 6. clear + warm cache — a reset deploy leaves it stale
|
||
```
|
||
|
||
> **Always finish a deploy with `remote-warmup-<env>`** — even a content-only
|
||
> deploy. A `reset --hard` (step 1) changes files under Grav without going
|
||
> through it, so the compiled-Twig/page cache is stale and the first visitor
|
||
> eats the recompile. `remote-warmup` clears the cache, then crawls the public
|
||
> pages (homepage + trips listing + every trip page linked from it) to
|
||
> pre-render them. Grav has no native warmup command — this is an HTTP crawl, so
|
||
> it also doubles as a smoke test (a non-2xx on any page is flagged loudly).
|
||
|
||
Why each matters:
|
||
- **Step 3** re-applies `deploy/patches/*.patch` automatically (it chains `remote-apply-plugin-patches`). GPM install/update lays down **pristine** third-party plugins, wiping local fixes to git-ignored `user/plugins/` — the patch step restores them. Content pulls (step 1) do **not** touch `plugins/`, so the patch only needs re-applying after a GPM op, not after every sync. Run `make remote-apply-plugin-patches-test` standalone if you ever GPM-install outside this sequence. Requires the `patch` tool on the server. See `deploy/patches/README.md`.
|
||
- **Step 4** is non-optional even if git-sync "was already there" — remote-only plugins are not in `plugins.txt`, so nothing in steps 1–3 restores them. If the code is missing, the plugin is inert despite valid config.
|
||
- **Step 5** re-writes `user/env/<host>/config/…` from `deploy/env/<env>/`. The env tree is not synced by anything, so a fresh install loses it until you re-apply.
|
||
|
||
### Verify (smoke checklist — this is the payoff)
|
||
|
||
- **Code present, not just config:** `ls user/plugins/<name>/` for every expected plugin (especially `git-sync`). An empty/absent dir = reinstall (step 4). *(Do this via an ssh one-liner you run, or `make remote-diag-test`.)*
|
||
- **Plugin patches applied:** confirm the add-page-by-form fix survived the GPM op — `grep -c toArray user/plugins/add-page-by-form/add-page-by-form.php` should be ≥1 (0 = pristine, re-run `make remote-apply-plugin-patches-test`). Functional check: edit a journal entry and add a photo — a pristine plugin 500s on save.
|
||
- **HTTP:** `/` → 200, `/admin` → 200, `/api/v1/pages` → 401, `/gpx-manager` → 200. Watch for the double-`Content-Encoding` garbage page (fix: `debugger.shutdown.close_connection: false` in the env override — already in `deploy/env/prod/system.yaml`).
|
||
- **Post smoke test:** submit one entry via `/post` and confirm it appears in the trip feed immediately. This proves the `cache-on-save` plugin works with prod caching on.
|
||
- **Config drift:** `make remote-diag-test` — diff server config against the repo. Fold any *intended* schema migration (e.g. the Twig-3 `strict_mode` flags a `self-upgrade` writes) back into `user/config/system.yaml`, or the next `fetch-content` reverts it.
|
||
|
||
### Re-enable + prove sync
|
||
|
||
- `make remote-git-sync-enable-test`.
|
||
- Confirm a content push round-trips to the server, **and** that no secret/boomerang commit lands on Gitea. Verify `/env/` is gitignored so the env tree (which holds the token, JWT, CSRF salt) can never enter the sync add-set.
|
||
|
||
---
|
||
|
||
## Phase 2 — Prod (repeat identically — should be mechanical)
|
||
|
||
Run the **exact same sequence** with `-prod` targets. Because test rehearsed it,
|
||
prod holds no surprises. Differences to layer on:
|
||
|
||
- Optional: `make remote-maintenance-on-prod` at the start, `remote-maintenance-off-prod` at the end, for a clean window.
|
||
- Confirm secrets are valid/rotated and `/env/` is gitignored **before** `remote-git-sync-enable-prod`. Re-enable git-sync **last**.
|
||
- After a clean cutover, bump the outer-repo submodule pin to the finished `user/` commit — and **push `user/` before the outer repo** (the superproject references a child SHA that must already exist upstream).
|
||
|
||
```
|
||
make remote-git-sync-disable-prod
|
||
make remote-fetch-content-prod
|
||
make remote-upgrade-grav-prod
|
||
make remote-update-plugins-prod
|
||
make remote-gpm-install-prod PKG=git-sync
|
||
make remote-apply-env-prod
|
||
make remote-warmup-prod # clear + warm cache; also HTTP-smokes public pages
|
||
# ── smoke checklist (same as test) ──
|
||
make remote-git-sync-enable-prod
|
||
```
|
||
|
||
For a first-time / from-scratch prod bring-up, `make remote-install-prod` does the
|
||
full install; then still run `remote-apply-env-prod` and the smoke checklist, and
|
||
reinstall remote-only plugins explicitly.
|
||
|
||
---
|
||
|
||
## Rollback & safety
|
||
|
||
- **git-sync stays disabled through the whole apply window** on each host — it is the catch that prevents a half-migrated state from auto-pushing.
|
||
- **Content** is a git repo: a bad content deploy is recoverable with `make remote-fetch-content-<env>` back to a known commit.
|
||
- **Core + plugins** are GPM-reinstallable (`remote-upgrade-grav`, `remote-update-plugins`, `remote-gpm-install PKG=…`).
|
||
- The one thing tooling cannot regenerate is the un-synced `user/env/<host>/` tree — its source of truth is `deploy/env/<env>/`, so keep that current and re-apply with `remote-apply-env-<env>`.
|
||
|
||
---
|
||
|
||
## One-line invariants (the through-line)
|
||
|
||
1. `test` is config-identical to `prod`, run with the same targets in the same order.
|
||
2. Verify the **code layer** (`ls user/plugins/<name>/`), not just config, on every deploy.
|
||
3. Reinstall **remote-only** plugins (git-sync) explicitly — nothing else restores them.
|
||
4. `GRAV_VERSION` in `.env.<env>` and `gpm.releases: stable` are correct **before** any server GPM op.
|
||
5. Re-apply the **env override** after every install; keep `/env/` **gitignored**.
|
||
6. git-sync **off** during the window, **on** last; confirm the round-trip carries no secrets.
|
||
7. Diagnose actual state before changing config — an `ls` or `remote-diag` beats a guess.
|