intotheeast-com/docs/guides/deploy-cycle.md

Upgrade & Deploy Cycle: local → test → prod

This runbook is the repeatable procedure for shipping a Grav upgrade or any server-affecting change (core version, plugins, config, theme) through the three environments. It was distilled from the 2026-07 Grav 2.0.4→2.0.7 cutover, where every production surprise traced back to one of the desyncs this procedure now forces you to check.

Governing principle: test is a full dress rehearsal of prod — same config, same -test/-prod make targets, same order. A gotcha only gets caught on test if test is a faithful mirror of prod. Do not shortcut test.

All server operations go through make remote-* targets (never raw SSH — the targets build the SSH connection from .env.<env>, which must never be read directly). Every remote-* target has -test and -prod variants; a bare target fails via guard-env.

---

The mental model: three places state lives

Every failure in the reference cutover was a desync between these three layers. Before and after each deploy step, ask: are they in sync?

Layer	Location	Synced by	Failure mode
Plugin code	user/plugins/<name>/	GPM only (gitignored /plugins/*)	can vanish while config remains → plugin won't enable
Repo config	user/config/…	content-push / git-sync	holds GPM channel + is where the version floor bites
Host config	user/env/<host>/config/…	nothing — server-only	not restored on fresh install; must be re-applied; must be gitignored

Referenced gotcha docs:

• docs/solutions/integration-issues/grav-plugin-config-without-code-wont-enable.md — code-vs-config desync.
• docs/solutions/integration-issues/stale-grav-version-blocks-api-plugin-install.md — stale GRAV_VERSION / version floor.
• docs/solutions/architecture-patterns/git-sync-secret-exposure-and-tracked-file-boomerang.md — gitignore is the sync boundary; env-tree leak.
• docs/solutions/conventions/grav-plugin-config-must-be-tracked-override.md — plugin config must live in the tracked override.

These three layers describe the servers. Locally there is a fourth: the Grav core is baked into the Docker image (Dockerfile), not in any layer above — so the local core upgrades by an image rebuild, never by the gpm self-upgrade the servers use. See docs/solutions/tooling-decisions/upgrade-local-grav-core-rebuild-docker-image.md.

---

Phase 0 — Local (author + prove the change)

1. Make the change in the repo:
   • GPM channel: gpm.releases: stable in user/config/system.yaml (authoritative; reaches servers via content pull, so it must be right before any server GPM op).
   • plugins.txt — the GPM-managed set only. Never add git-sync (it is remote-only).
   • Prod-only overrides (Twig cache/debug, debugger.shutdown.close_connection: false) in deploy/env/prod/system.yaml — never commit prod values into user/config/system.yaml.
   • Bump GRAV_VERSION in .env.test and .env.prod to the target version. A stale value here installs the wrong core (an rc), which then blocks the api plugin and 404s admin. This governs fresh remote installs only.
   • If the core version is changing, upgrade the local dev core too so you prove the change against the target version — bump the hardcoded grav-admin-v<ver>.zip URL in Dockerfile, docker compose build grav, then docker rm -f intotheeast_grav && docker compose up -d grav. The local core is baked into the image, so .env GRAV_VERSION does not touch it and an in-container gpm self-upgrade is non-durable. See docs/solutions/tooling-decisions/upgrade-local-grav-core-rebuild-docker-image.md.
2. make build-assets if you touched js/src/* (never hand-edit the bundled js/*.js).
3. Run the dev server (docker compose … up) and the Playwright suite.
4. Pre-flight assertions:
   • gpm.releases is stable.
   • plugins.txt is correct and does not contain git-sync.
   • No prod Twig values leaked into the committed system.yaml.
5. Commit. make content-push.

---

Phase 1 — Test (the rehearsal — catch things here)

Pre-flight

• make remote-git-sync-disable-test before any content reset. This is the safety catch for the whole window: it stops a half-migrated state (e.g. a fresh install-time versions.yaml) from auto-committing and pushing on the first sync.

Apply — in this fixed order

make remote-fetch-content-test              # 1. clean-reset synced folders to repo state
make remote-upgrade-grav-test               # 2. gpm self-upgrade (rewrites schema — expect drift)
make remote-update-plugins-test             # 3. gpm update the plugins.txt set
make remote-gpm-install-test PKG=git-sync   # 4. EXPLICITLY (re)install each remote-only plugin
make remote-apply-env-test                  # 5. re-deploy the env override (not synced; gone after install)

Why each matters:

• Step 4 is non-optional even if git-sync "was already there" — remote-only plugins are not in plugins.txt, so nothing in steps 1–3 restores them. If the code is missing, the plugin is inert despite valid config.
• Step 5 re-writes user/env/<host>/config/… from deploy/env/<env>/. The env tree is not synced by anything, so a fresh install loses it until you re-apply.

Verify (smoke checklist — this is the payoff)

• Code present, not just config: ls user/plugins/<name>/ for every expected plugin (especially git-sync). An empty/absent dir = reinstall (step 4). (Do this via an ssh one-liner you run, or make remote-diag-test.)
• HTTP: / → 200, /admin → 200, /api/v1/pages → 401, /gpx-manager → 200. Watch for the double-Content-Encoding garbage page (fix: debugger.shutdown.close_connection: false in the env override — already in deploy/env/prod/system.yaml).
• Post smoke test: submit one entry via /post and confirm it appears in the trip feed immediately. This proves the cache-on-save plugin works with prod caching on.
• Config drift: make remote-diag-test — diff server config against the repo. Fold any intended schema migration (e.g. the Twig-3 strict_mode flags a self-upgrade writes) back into user/config/system.yaml, or the next fetch-content reverts it.

Re-enable + prove sync

• make remote-git-sync-enable-test.
• Confirm a content push round-trips to the server, and that no secret/boomerang commit lands on Gitea. Verify /env/ is gitignored so the env tree (which holds the token, JWT, CSRF salt) can never enter the sync add-set.

---

Phase 2 — Prod (repeat identically — should be mechanical)

Run the exact same sequence with -prod targets. Because test rehearsed it, prod holds no surprises. Differences to layer on:

• Optional: make remote-maintenance-on-prod at the start, remote-maintenance-off-prod at the end, for a clean window.
• Confirm secrets are valid/rotated and /env/ is gitignored before remote-git-sync-enable-prod. Re-enable git-sync last.
• After a clean cutover, bump the outer-repo submodule pin to the finished user/ commit — and push user/ before the outer repo (the superproject references a child SHA that must already exist upstream).

make remote-git-sync-disable-prod
make remote-fetch-content-prod
make remote-upgrade-grav-prod
make remote-update-plugins-prod
make remote-gpm-install-prod PKG=git-sync
make remote-apply-env-prod
# ── smoke checklist (same as test) ──
make remote-git-sync-enable-prod

For a first-time / from-scratch prod bring-up, make remote-install-prod does the full install; then still run remote-apply-env-prod and the smoke checklist, and reinstall remote-only plugins explicitly.

---

Rollback & safety

• git-sync stays disabled through the whole apply window on each host — it is the catch that prevents a half-migrated state from auto-pushing.
• Content is a git repo: a bad content deploy is recoverable with make remote-fetch-content-<env> back to a known commit.
• Core + plugins are GPM-reinstallable (remote-upgrade-grav, remote-update-plugins, remote-gpm-install PKG=…).
• The one thing tooling cannot regenerate is the un-synced user/env/<host>/ tree — its source of truth is deploy/env/<env>/, so keep that current and re-apply with remote-apply-env-<env>.

---

One-line invariants (the through-line)

1. test is config-identical to prod, run with the same targets in the same order.
2. Verify the code layer (ls user/plugins/<name>/), not just config, on every deploy.
3. Reinstall remote-only plugins (git-sync) explicitly — nothing else restores them.
4. GRAV_VERSION in .env.<env> and gpm.releases: stable are correct before any server GPM op.
5. Re-apply the env override after every install; keep /env/ gitignored.
6. git-sync off during the window, on last; confirm the round-trip carries no secrets.
7. Diagnose actual state before changing config — an ls or remote-diag beats a guess.