intotheeast-com/docs/guides/deploy-cycle.md

Upgrade & Deploy Cycle: local → test → prod

This runbook is the repeatable procedure for shipping a Grav upgrade or any server-affecting change (core version, plugins, config, theme) through the three environments. It was distilled from the 2026-07 Grav 2.0.4→2.0.7 cutover, where every production surprise traced back to one of the desyncs this procedure now forces you to check.

Governing principle: test is a full dress rehearsal of prod — same config, same -test/-prod make targets, same order. A gotcha only gets caught on test if test is a faithful mirror of prod. Do not shortcut test.

All server operations go through make remote-* targets (never raw SSH — the targets build the SSH connection from .env.<env>, which must never be read directly). Every remote-* target has -test and -prod variants; a bare target fails via guard-env.

---

The mental model: three places state lives

Every failure in the reference cutover was a desync between these three layers. Before and after each deploy step, ask: are they in sync?

Layer	Location	Synced by	Failure mode
Plugin code	user/plugins/<name>/	GPM only (gitignored /plugins/*)	can vanish while config remains → plugin won't enable
Repo config	user/config/…	content-push / git-sync	holds GPM channel + is where the version floor bites
Host config	user/env/<host>/config/…	nothing — server-only	not restored on fresh install; must be re-applied; must be gitignored

Referenced gotcha docs:

• docs/solutions/integration-issues/grav-plugin-config-without-code-wont-enable.md — code-vs-config desync.
• docs/solutions/integration-issues/stale-grav-version-blocks-api-plugin-install.md — stale GRAV_VERSION / version floor.
• docs/solutions/architecture-patterns/git-sync-secret-exposure-and-tracked-file-boomerang.md — gitignore is the sync boundary; env-tree leak.
• docs/solutions/conventions/grav-plugin-config-must-be-tracked-override.md — plugin config must live in the tracked override.

These three layers describe the servers. Locally there is a fourth: the Grav core is baked into the Docker image (Dockerfile), not in any layer above — so the local core upgrades by an image rebuild, never by the gpm self-upgrade the servers use. See docs/solutions/tooling-decisions/upgrade-local-grav-core-rebuild-docker-image.md.

---

The env override tree (user/env/<host>/)

Prod needs different Twig settings than dev. These are never committed to user/config/system.yaml — twig.cache: false and debug/auto_reload: true are the intended dev values, and committing prod values there breaks local development for everyone. Instead they ship as a per-environment override via Grav's environment://config, keyed on the request hostname.

Setting	Dev (committed)	Prod (override)	Why prod differs
twig.cache	false	true	Compile templates once and reuse
twig.debug	true	false	No debug functions in prod
twig.auto_reload	true	false	Don't stat templates every request

• Source of truth: deploy/env/prod/system.yaml (version-controlled).
• Deploy: make remote-apply-env-prod — writes it to <webroot>/user/env/<hostname>/config/system.yaml and clears cache. It deep-merges over the committed system.yaml.
• Hostname segment defaults to REMOTE_HOST; override with WEB_HOST in .env.<env> if Grav sees a different host than the SSH host.
• Not restored by anything. user/env/ is outside the content repo's tracked folders, so content-push / git-sync / remote-fetch-content do not bring it back. Re-run make remote-apply-env-<env> after any fresh install.

Side effect: Admin writes ALL config into the env tree

Once user/env/<hostname>/ exists, Grav's Admin saves every config change (system and plugin) there — e.g. editing a plugin on prod writes user/env/intotheeast.com/config/plugins/<name>.yaml, not user/config/plugins/<name>.yaml. Consequences:

• Config edited via Admin on the server is server-only: the env tree is not committed and not synced by git-sync (which syncs only pages/config/ themes), so prod Admin edits silently never reach Gitea or local. This is good for secrets — git-sync.yaml (token), the JWT and CSRF salt safely live there — but it means config drift is invisible to the repo.
• When reading or writing server config, check both user/config/… and user/env/<host>/config/… (env wins). Server tooling must search the env path first — see scripts/git-sync-toggle.sh and make remote-diag.
• Repo-authored config (user/config/… via make content-push) still applies everywhere; the env tree holds only per-host overrides + Admin-on-server edits.

Full details: docs/working/git-sync-notes.md.

---

Phase 0 — Local (author + prove the change)

1. Make the change in the repo:
   • GPM channel: gpm.releases: stable in user/config/system.yaml (authoritative; reaches servers via content pull, so it must be right before any server GPM op).
   • plugins.txt — the GPM-managed set only. Never add git-sync (it is remote-only).
   • Prod-only overrides (Twig cache/debug, debugger.shutdown.close_connection: false) in deploy/env/prod/system.yaml — never commit prod values into user/config/system.yaml.
   • Bump GRAV_VERSION in .env.test and .env.prod to the target version. A stale value here installs the wrong core (an rc), which then blocks the api plugin and 404s admin. This governs fresh remote installs only.
   • If the core version is changing, upgrade the local dev core too so you prove the change against the target version — bump the hardcoded grav-admin-v<ver>.zip URL in Dockerfile, docker compose build grav, then docker rm -f intotheeast_grav && docker compose up -d grav. The local core is baked into the image, so .env GRAV_VERSION does not touch it and an in-container gpm self-upgrade is non-durable. See docs/solutions/tooling-decisions/upgrade-local-grav-core-rebuild-docker-image.md.
2. make build-assets if you touched js/src/* (never hand-edit the bundled js/*.js).
3. Run the dev server (docker compose … up) and the Playwright suite.
4. Pre-flight assertions:
   • gpm.releases is stable.
   • plugins.txt is correct and does not contain git-sync.
   • No prod Twig values leaked into the committed system.yaml.
5. Commit. make content-push.

---

Phase 1 — Test (the rehearsal — catch things here)

Pre-flight

• make remote-git-sync-disable-test before any content reset. This is the safety catch for the whole window: it stops a half-migrated state (e.g. a fresh install-time versions.yaml) from auto-committing and pushing on the first sync.

Apply — in this fixed order

make remote-fetch-content-test              # 1. clean-reset synced folders to repo state
make remote-upgrade-grav-test               # 2. gpm self-upgrade (rewrites schema — expect drift)
make remote-update-plugins-test             # 3. gpm update the plugins.txt set (auto-applies deploy/patches/)
make remote-gpm-install-test PKG=git-sync   # 4. EXPLICITLY (re)install each remote-only plugin
make remote-apply-env-test                  # 5. re-deploy the env override (not synced; gone after install)
make remote-warmup-test                     # 6. clear + warm cache — a reset deploy leaves it stale

Always finish a deploy with remote-warmup-<env> — even a content-only deploy. A reset --hard (step 1) changes files under Grav without going through it, so the compiled-Twig/page cache is stale and the first visitor eats the recompile. remote-warmup clears the cache, then crawls the public pages (homepage + trips listing + every trip page linked from it) to pre-render them. Grav has no native warmup command — this is an HTTP crawl, so it also doubles as a smoke test (a non-2xx on any page is flagged loudly).

Why each matters:

• Step 3 re-applies deploy/patches/*.patch automatically (it chains remote-apply-plugin-patches). GPM install/update lays down pristine third-party plugins, wiping local fixes to git-ignored user/plugins/ — the patch step restores them. Content pulls (step 1) do not touch plugins/, so the patch only needs re-applying after a GPM op, not after every sync. Run make remote-apply-plugin-patches-test standalone if you ever GPM-install outside this sequence. Requires the patch tool on the server. See deploy/patches/README.md.
• Step 4 is non-optional even if git-sync "was already there" — remote-only plugins are not in plugins.txt, so nothing in steps 1–3 restores them. If the code is missing, the plugin is inert despite valid config.
• Step 5 re-writes user/env/<host>/config/… from deploy/env/<env>/. The env tree is not synced by anything, so a fresh install loses it until you re-apply.

Verify (smoke checklist — this is the payoff)

• Code present, not just config: ls user/plugins/<name>/ for every expected plugin (especially git-sync). An empty/absent dir = reinstall (step 4). (Do this via an ssh one-liner you run, or make remote-diag-test.)
• Plugin patches applied: confirm the add-page-by-form fix survived the GPM op — grep -c toArray user/plugins/add-page-by-form/add-page-by-form.php should be ≥1 (0 = pristine, re-run make remote-apply-plugin-patches-test). Functional check: edit a journal entry and add a photo — a pristine plugin 500s on save.
• HTTP: / → 200, /admin → 200, /api/v1/pages → 401, /gpx-manager → 200. Watch for the double-Content-Encoding garbage page (fix: debugger.shutdown.close_connection: false in the env override — already in deploy/env/prod/system.yaml).
• Post smoke test: submit one entry via /post and confirm it appears in the trip feed immediately. This proves the cache-on-save plugin works with prod caching on.
• Config drift: make remote-diag-test — diff server config against the repo. Fold any intended schema migration (e.g. the Twig-3 strict_mode flags a self-upgrade writes) back into user/config/system.yaml, or the next fetch-content reverts it.

Re-enable + prove sync

• make remote-git-sync-enable-test.
• Confirm a content push round-trips to the server, and that no secret/boomerang commit lands on Gitea. Verify /env/ is gitignored so the env tree (which holds the token, JWT, CSRF salt) can never enter the sync add-set.

---

Phase 2 — Prod (repeat identically — should be mechanical)

Run the exact same sequence with -prod targets. Because test rehearsed it, prod holds no surprises. Differences to layer on:

• Optional: make remote-maintenance-on-prod at the start, remote-maintenance-off-prod at the end, for a clean window.
• Confirm secrets are valid/rotated and /env/ is gitignored before remote-git-sync-enable-prod. Re-enable git-sync last.
• After a clean cutover, bump the outer-repo submodule pin to the finished user/ commit — and push user/ before the outer repo (the superproject references a child SHA that must already exist upstream).

make remote-git-sync-disable-prod
make remote-fetch-content-prod
make remote-upgrade-grav-prod
make remote-update-plugins-prod
make remote-gpm-install-prod PKG=git-sync
make remote-apply-env-prod
make remote-warmup-prod                     # clear + warm cache; also HTTP-smokes public pages
# ── smoke checklist (same as test) ──
make remote-git-sync-enable-prod

For a first-time / from-scratch prod bring-up, make remote-install-prod does the full install; then still run remote-apply-env-prod and the smoke checklist, and reinstall remote-only plugins explicitly.

---

Rollback & safety

• git-sync stays disabled through the whole apply window on each host — it is the catch that prevents a half-migrated state from auto-pushing.
• Content is a git repo: a bad content deploy is recoverable with make remote-fetch-content-<env> back to a known commit.
• Core + plugins are GPM-reinstallable (remote-upgrade-grav, remote-update-plugins, remote-gpm-install PKG=…).
• The one thing tooling cannot regenerate is the un-synced user/env/<host>/ tree — its source of truth is deploy/env/<env>/, so keep that current and re-apply with remote-apply-env-<env>.

---

One-line invariants (the through-line)

1. test is config-identical to prod, run with the same targets in the same order.
2. Verify the code layer (ls user/plugins/<name>/), not just config, on every deploy.
3. Reinstall remote-only plugins (git-sync) explicitly — nothing else restores them.
4. GRAV_VERSION in .env.<env> and gpm.releases: stable are correct before any server GPM op.
5. Re-apply the env override after every install; keep /env/ gitignored.
6. git-sync off during the window, on last; confirm the round-trip carries no secrets.
7. Diagnose actual state before changing config — an ls or remote-diag beats a guess.