Files
intotheeast-com/docs/solutions/architecture-patterns/git-sync-secret-exposure-and-tracked-file-boomerang.md
m038andClaude Opus 4.8 9295914238 docs: capture history-rewrite-under-live-git-sync + secrets-audit tooling
Update the git-sync secret-exposure solution doc with today's operational
lesson: untracking an already-committed secret under a live bidirectional
sync. Covers the direction:both force-push-revert trap, the freeze-every-
server-first sequence, audit-before-reset (authoritative secret in env/),
the stale origin/main ref + sparse-checkout gotchas, and the ignore:-field
mechanism.

Add Makefile targets that supported the fix:
- remote-secrets-audit: secret-safe (existence + size + git ls-files, never
  contents) audit of config/ vs env/<host>/config secret locations
- remote-content-status: also show the .gitignore diff git-sync regenerates

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-05 19:25:32 +02:00

17 KiB
Raw Permalink Blame History

title, date, last_updated, module, problem_type, component, severity, category, applies_when, tags
title date last_updated module problem_type component severity category applies_when tags
Secret exposure under bidirectional Grav git-sync: gitignore is the only boundary (and the tracked-file boomerang trap) 2026-07-05 2026-07-05 git-sync architecture_pattern tooling high architecture-patterns
Enabling bidirectional Grav git-sync (direction: both, on_save: true) so prod can push content back to Gitea
Auditing whether the server can leak secrets (tokens, password hashes, signing salts) off the production host
A per-install runtime-generated value is being persisted into a tracked config file that also carries functional config
Deciding where a per-install secret must live so it never round-trips
A config value keeps ping-ponging or re-committing itself across environments after each sync
git-sync
secret-exposure
gitignore
grav
popularity-salt
config-boundary
bidirectional-sync
per-install-secret
env-tree-leak

Secret exposure under bidirectional Grav git-sync: gitignore is the only boundary (and the tracked-file boomerang trap)

Correction (2026-07-05): An earlier version of this doc claimed the sync add-set was scoped to the configured folders, and concluded that accounts/ and user/env/ were "safe by construction" because they sit outside pages/config/themes. That model is wrong and caused a live secret leak. git-sync's auto-commit stages files outside the configured folders; the only reliable exclusion is .gitignore. The corrected model is below.

Context

The intotheeast.com Grav site runs the git-sync plugin on production in bidirectional mode: direction: both, on_save: true, a webhook at /_git-sync, and configured folders: pages, config, themes. Bidirectional means the server both pulls content authored elsewhere and pushes content authored on the server (Admin edits, /post submissions, uploads) back to the shared Gitea repo.

The operator's question was: "Can I safely enable bidirectional prod → Gitea sync without leaking secrets?" Production holds things that must never reach a shared repo — API tokens, JWT signing secrets, CSRF salts, the git-sync token itself.

The dangerous, tempting answer is "the plugin only syncs pages/config/themes, so anything outside those folders is safe." This is false, and acting on it leaked secrets. See the incident below.

The incident (what actually happened)

Production's git-sync auto-commit 9337003 ("(Grav GitSync) Automatic Commit ...") pushed the entire user/env/intotheeast.com/config/ tree to Gitea — including api-private.php (JWT secret), security-private.php (CSRF salt), and git-sync.yaml (the sync token + webhook secret) — plus accounts/mischa.yaml (password hash), system.yaml, and post-form.md.

None of those paths is under the configured folders: pages, config, themes. user/env/ is a sibling of user/config/, not a subfolder of it. Yet git-sync staged and pushed them anyway. That single fact refutes the "folders scope the add-set" model empirically: the folders setting does not scope what the auto-commit stages. Whatever git-sync's exact git add invocation, the operational truth is that its commit sweeps the whole user/ working tree.

Remediation: disable sync → .gitignore /env/ and git rm --cached it → reset --hard prod to the gitignored state → regenerate the leaked JWT/CSRF salts (delete the *-private.php files; Grav regenerates them) → rotate the Gitea token and webhook secret (they were exposed in git-sync.yaml). Rotation is what actually neutralizes the leak; the history rewrite is optional for a private repo.

Guidance

The safety question reduces to one predicate — but not the one the original doc used:

A file round-trips to the shared repo if and only if it is NOT gitignored. The configured folders setting does not narrow this. Treat the round-trippable set as everything under user/ that git will track — i.e. everything not matched by user/.gitignore.

Consequences, corrected:

  1. .gitignore is the only reliable boundary. Do not rely on a file being "outside the synced folders." If it is under user/ and not gitignored, a bidirectional sync can push it. Design exclusions with .gitignore, and verify with git -C user status / git -C user check-ignore <path>.

  2. accounts/ is NOT structurally excluded. user/accounts/*.yaml (bcrypt password hashes) is a tracked content folder and is not gitignored, so it does round-trip — accounts/mischa.yaml was in the leak commit. If you need an account file to stay server-local, it must be gitignored explicitly (as accounts/testrunner.yaml already is). The earlier "password hashes never leave the server" claim was wrong.

  3. The per-environment tree user/env/<host>/ MUST be gitignored — it is not inherently safe. It holds the live git-sync token, JWT secret, and CSRF salt (Grav writes all server-side Admin config there once the env dir exists). Because it is not under user/config/ people assumed it was outside the sync scope; the incident proved it is not. It is now gitignored (/env/ in user/.gitignore, commit 6e8eadb). Keep it that way.

    Side effect worth remembering: once user/env/<host>/ exists, Grav's Admin writes all config changes there (system and plugin), not into user/config/. So server-side Admin edits are server-only — but "server-only" now depends entirely on /env/ being gitignored, not on folder scope. When auditing, check both user/config/... and user/env/<host>/config/... (env wins at runtime). See docs/working/git-sync-notes.md.

  4. Per-install secrets go in gitignored companion files. Grav's convention splits a per-install secret out of the functional YAML into a sibling that is gitignored: the JWT secret in api-private.php, the CSRF/nonce + rate-limit salt in security-private.php, plus security.yaml and versions.yaml. These are safe because they are gitignored, not because of where they sit.

Run every secret through predicate #1 (is it gitignored?) and the answer falls out — but you must actually enumerate what is not gitignored, not what is "outside the folders."

The tracked-file boomerang (a separate trap)

Independently of the folder-scope error above, there is a second trap that the original doc got right and that still holds: a per-install value that a plugin regenerates at runtime and persists into a tracked, functional config file.

The concrete case: the api plugin's popularity feature generates popularity.salt and writes it into user/config/plugins/api.yaml — a file that also carries must-be-shared functional config. That file is tracked, so on prod the popularity feature regenerates the salt, git-sync stages the change, commits, and pushes prod's salt back to the shared repo. Another environment pulls it, regenerates its own salt, pushes again. The value ping-pongs across installs, producing endless noise commits.

The critical realization: you cannot gitignore a single key inside a file that also carries functional config. .gitignore operates on whole files. api.yaml must be tracked because the rest of it must be shared; therefore the salt inside it is tracked too; therefore it boomerangs.

The rules

For any per-install runtime-generated value, pick one of exactly three resolutions — and do not reach for the fourth (stripping the line), which cannot work under sync:

  • Isolate the value into a gitignored companion <name>-private.php — the pattern Grav uses for the JWT secret via api-private.php.
  • Disable the feature that generates it (e.g. popularity.enabled: false).
  • Consciously accept the churn when the value is genuinely low-stakes (popularity.salt is an IP-hashing salt, not a credential).

Do not keep stripping the value from the tracked file — bidirectional sync brings it right back on the next save.

Before / after: what sticks and what doesn't

A standalone file gitignored + untracked sticks. For security-private.php (a file that contains only the secret) or the whole user/env/ tree:

git -C user rm --cached -r config/security-private.php   # or: rm --cached -r env
printf '/config/security-private.php\n/env/\n' >> user/.gitignore

This works permanently. The path is no longer tracked, so the sync's add-set skips it forever. The fix sticks because the secret owns its own gitignored path.

A key inside a tracked functional file — stripping the line does NOT stick. For popularity.salt inside api.yaml, deleting just the salt: line and committing looks clean locally, but Grav re-appends it at runtime and the next sync re-commits and re-pushes it. The only durable fixes are the companion-private-file pattern or disabling the feature.

Untracking an already-committed secret under a live sync — freeze every server first

.gitignore (and git-sync's ignore field) only affects untracked files. Once a secret has actually been committed to the shared repo, ignoring it does nothing — removing it means rewriting history. Doing that while a bidirectional sync is live has its own trap.

A direction: both server silently reverts your force-push. Concrete incident (2026-07-05, a second occurrence of this doc's trap): config/security-private.php and config/versions.yaml had been committed to Gitea main (auto-commit d1643a7, merged at f8c45fc). Force-pushing main back to the clean commit 32c3d8c looked successful — and within seconds Gitea was back at f8c45fc. Cause: prod's git-sync is direction: both and its local HEAD was still f8c45fc; on its next sync it re-pushed the stale, secret-bearing commit and undid the rewrite. The pull-only test server never fought back — only push-enabled servers do.

The rule: freeze git-sync on every server (push and pull) before rewriting shared history. A pull server mid-rewrite can also resurrect a half-removed state. The safe sequence that worked:

  1. Freeze all sync. make remote-git-sync-disable-{test,prod} (flips enabled: false in the env-path git-sync.yaml).
  2. Audit where the live secret actually lives — before any reset --hard. A destructive reset deletes working-tree files tracked now but absent in the target commit. config/security-private.php was such a file — but it was a stray duplicate; the authoritative 290-byte copy lives at env/intotheeast.com/config/security-private.php (mode 600), which git-sync had copied into config/. Because env/ is gitignored and outside every tracked folder, it survives the reset and wins Grav's config merge — so dropping the config/ copy is safe. Verify this first with a secret-safe audit that lists existence + size + git ls-files tracking and never prints contents (added as make remote-secrets-audit; it ls / git ls-files, never cat).
  3. Force-push main to the clean commit. It sticks now — no server is pushing.
  4. Reset each server with make remote-fetch-content-{test,prod} (fetchsparse-checkout disablereset --hard origin/main). This deletes the stray tracked config/ copies; the env/ originals remain.
  5. Verify git ls-files shows no secret tracked and the env/ copy is intact on every host.
  6. Re-enable sync (make remote-git-sync-enable-*), preserving each server's direction. Local HEAD now equals Gitea main, so there is nothing bad to push.

Two gotchas inside step 4:

  • Stale remote-tracking ref. reset --hard origin/main resets to the server's cached refs/remotes/origin/main, not to Gitea directly. If that ref is stale the reset lands on the wrong commit — confirm the fetch force-updated it (+ f8c45fc...32c3d8c main -> origin/main (forced update)) before trusting the reset.
  • sparse-checkout disable before reset --hard — otherwise the reset only touches paths inside the sparse pattern and can skip/wipe directories outside it.

Durable exclusion goes in git-sync's ignore: config field, never a hand-edited .gitignore. git-sync owns .gitignore: on load it regenerates it from folders (/*, !/pages, !/config, !/themes) and appends the ignore: entries. Hand edits are clobbered on the next sync; ignore: entries persist because git-sync writes them back every time. So the secret paths belong in ignore: — but that only prevents future tracking. The history rewrite (steps 15) is still required in addition to the ignore entries to remove a secret that is already committed, not instead of them.

Why This Matters

Two quiet, cross-environmental failure modes:

  1. The folder-scope illusion. Assuming "only pages/config/themes sync" is a security control leads you to leave secrets in env/ or accounts/ unignored — and a single Admin save on prod pushes them to a shared repo. This actually happened here. The only defensible mental model is gitignore is the boundary; enumerate the un-ignored set, not the "un-foldered" set.

  2. The boomerang. A "cleanup" commit that strips a secret from a tracked file looks done locally but silently reappears upstream on the next content save, because the plugin regenerates it and the sync re-commits it.

Getting both right is what lets you answer "is bidirectional sync safe?" honestly. The answer is yes, once user/.gitignore actually excludes every sensitive pathenv/, the per-install *-private.php files, security.yaml, versions.yaml, and any account file that must stay server-local — and once every runtime-regenerated value either lives in its own gitignored file or is a consciously-accepted low-stakes churn. It is emphatically not safe on the strength of folder scoping alone.

When to Apply

  • Enabling or auditing bidirectional git-sync on a server that authors content. Enumerate the round-trippable set as everything under user/ not matched by .gitignore — then confirm no secret is in it.
  • Deciding where a new secret or per-install generated value should live. Standalone gitignored file for anything sensitive; never a key inside a shared functional YAML; never "outside the folders" as the sole justification.
  • Reviewing a "stop tracking this secret" cleanup for whether it will stick: is the secret in its own gitignored path (holds) or a line inside a tracked functional file that something regenerates (boomerangs)?
  • Rewriting shared history (force-push, filter-repo, reset --hard) on a git-sync-managed repo — freeze sync on every server first, audit where the live secret authoritatively lives before any destructive reset, then re-enable.

Examples

The leak (what "folder scope is safe" cost). With folders: pages, config, themes and /env/ not gitignored, prod's auto-commit 9337003 pushed user/env/intotheeast.com/config/** (JWT, CSRF salt, git-sync token + webhook secret), accounts/mischa.yaml, system.yaml, and post-form.md to Gitea — all outside the configured folders. Fix: gitignore + untrack /env/, regenerate the JWT/CSRF salts, rotate the token and webhook secret.

Safe after remediation. Same bidirectional config, but now user/.gitignore excludes /env/, config/plugins/git-sync.yaml, config/plugins/api-private.php, config/security.yaml, config/security-private.php, config/versions.yaml. Running each secret through is-it-gitignored: all sensitive paths are excluded → none is in the round-trippable set. Verified: prod's git status shows only the intended tracked content, and no boomerang/secret commit lands on the remote.

Boomerang example. popularity.salt in the tracked api.yaml regenerates per-install and re-commits under sync. The fix that sticks is the companion-private-file pattern or popularity.enabled: falsenot stripping the salt: line.

Force-push revert example. With config/security-private.php + config/versions.yaml already committed to Gitea main (f8c45fc), a git push --force origin main back to the clean 32c3d8c was undone within seconds — prod's direction: both git-sync re-pushed its stale f8c45fc HEAD. The rewrite only held after make remote-git-sync-disable-{test,prod} froze both servers first; then force-push → make remote-fetch-content-{test,prod} → re-enable. Verified afterward: git ls-files on every host lists no secret, and each host's env/…/security-private.php is intact.

  • docs/solutions/conventions/grav-plugin-config-must-be-tracked-override.md — the origin of the "functional plugin config in tracked user/config/plugins/, secrets/per-install values in gitignored *-private.php" rule. That doc covers where config must live to deploy; this doc covers why gitignore — not folder scope — is the sync boundary, and why a runtime-written tracked value boomerangs.
  • docs/working/git-sync-notes.md — operational notes on git-sync's synced folders and the per-environment tree. Corrected in the same 2026-07-05 pass to drop the "env/ is outside the sync scope so it's safe" claim.
  • docs/solutions/integration-issues/stale-grav-version-blocks-api-plugin-install.md — adjacent context from the same Grav production cutover, different failure mode.