docs: refine spec — _pipeline tag namespace, processed marker, visual-similarity deferral, header formatting

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

docs/superpowers/specs/2026-06-27-immich-photo-flow-design.md

@@ -1,8 +1,8 @@
 # immich-photo-flow — Design (M1: Foundation + Trip-Cluster)
 
-**Date:** 2026-06-27
+- **Date:** 2026-06-27
-**Status:** Approved design, pending implementation plan
+- **Status:** Approved design, pending implementation plan
-**Scope of this spec:** Milestone M1 only (shared foundation + the `trip-cluster` app). Later milestones are documented as a roadmap, each to get its own spec→plan→build cycle.
+- **Scope of this spec:** Milestone M1 only (shared foundation + the `trip-cluster` app). Later milestones are documented as a roadmap, each to get its own spec→plan→build cycle.
 
 ## Purpose
 
@@ -98,7 +98,7 @@ The single Immich REST client (consolidates the duplicated `immich.py` from both
 - `resolve_tag_id(name)`, list tags / tag inventory.
 - Search assets (paginated `POST /api/search/metadata`) by date range, by tag, and broadly — returning at least: `id`, `originalFileName`, `localDateTime`, GPS lat/lon (when present), city/country (when present), `type`, existing tags, rating.
 - `download_thumbnail(asset_id)` (`…/thumbnail?size=preview`).
-- `upsert_tag(name)`, `tag_assets(tag_id, asset_ids)` (write-back of trip / non-trip tags).
+- `upsert_tag(name)`, `tag_assets(tag_id, asset_ids)` (write-back of trip / `_pipeline/*` tags), plus a helper for the shared `_pipeline/` meta-tag namespace (see Tag conventions).
 
 ### `shared/core`
 SQLite store + domain dataclasses + persistence (connection, schema/migrations, data-access helpers). Owns the on-disk schema. Designed as a clean data-access layer so other apps can adopt it later.
@@ -141,6 +141,8 @@ categorize apply                                                      # 4. write
 ### 1. Ingest (automatic, scopeable)
 Pull assets from Immich via `shared/immich`, upsert metadata into SQLite, download thumbnails. **Incremental & idempotent** (fetch only changed-since-last via Immich `updatedAt`; upsert by `immich_id`).
 
+**Read-back of pipeline state:** ingest also reads each asset's existing tags, including `_pipeline/*`. Assets carrying `_pipeline/processed` are marked already-adjudicated in SQLite, so the working DB can be **rebuilt from Immich** after a loss and review resumes without redoing finished work.
+
 **Scopeable from day one** — the mechanism for the vet-first plan. Instead of forcing a full 40k pull, `ingest` accepts a bounded scope:
 - `--from / --to` (date window) → e.g. the last trip plus surrounding everyday photos;
 - `--tag NAME` → a single already-tagged trip;
@@ -155,7 +157,7 @@ Pure, algorithmic, **no API calls** (keeps the POC nearly free). Produces candid
 2. **Timestamp gap clustering** → primary structure: sort by `taken_at`, split where the inter-photo gap exceeds a tunable threshold.
 3. **Location anchors** (existing location tags like "Kiev" + GPS when present) → refine boundaries, propose names.
 4. **Coverage detection** → assets *inside* a confirmed trip's time window but *missing* its tag are flagged "likely belongs here" (the completeness gap); assets carrying a trip tag but *outside* their cluster are flagged as outliers.
-5. **Visual similarity** (Immich CLIP / local pHash) → *optional* tie-breaker for ambiguous boundaries; marked optional so it can never block the slice.
+5. **Visual similarity** → **deferred from M1; leading post-M1 enhancement.** Trips are defined by time + place, not visual likeness, so the signals above resolve the large majority; visual similarity only helps a narrow case (an ambiguous time gap where two bursts may be one trip). pHash is the wrong tool (it finds near-duplicates, not trip-level similarity). CLIP is the right tool, but Immich's REST API does not cleanly expose raw embedding vectors. **Viable path: read-only access to Immich's Postgres pgvector embeddings** (do nearest-neighbor/clustering ourselves), pending a feasibility check against the live Immich version. Especially valuable here because the old library is GPS-poor, so visual continuity may be one of the few secondary signals for those photos.
 
 **Confidence & kind_guess** (echoing image-rater's confidence/floor approach): tight time window + existing trip tag + consistent location → high confidence; sparse, untagged, no GPS → low ("needs your eye"). Low-volume scattered clusters → `kind_guess = everyday` (suggested non-trip).
 
@@ -171,14 +173,24 @@ To keep it fast: high-confidence clusters arrive **pre-filled**, and an **"appro
 
 ### 4. Write-back (to Immich, idempotent)
 On approval (per-cluster or batch `apply`):
-- **Approve** → write the trip tag (via `upsert_tag` + `tag_assets`) to all member assets, respecting the existing trip-tag convention.
+- **Approve** → write the trip tag (via `upsert_tag` + `tag_assets`) to all member assets, respecting the existing **content** trip-tag convention (trip names are user-facing, not namespaced).
-- **Mark non-trip** → apply a `non-trip` tag (named to fit the user's existing scheme) so those assets are filtered out and never resurface as unreviewed.
+- **Mark non-trip** → apply `_pipeline/non-trip` so those assets are filtered out and never resurface as unreviewed.
+- **Mark processed** → every adjudicated asset (trip-assigned, non-trip, **or** reviewed-and-skipped) gets `_pipeline/processed`. This is the durable "done" flag in the source of truth: it captures the reviewed-but-untagged case, and lets a fresh `ingest` re-derive what's already handled even if the SQLite working DB is lost (see Ingest read-back).
 - **Idempotent**: `writeback_log` records applied changes; re-runs skip what's already done. **Explicit confirmation before any write** (mirrors image-rater's export safety).
 
-All durable state lands in **Immich**; SQLite remains the working/review layer.
+All durable state lands in **Immich**; SQLite remains the working/review layer that can be rebuilt from Immich tags.
 
-### Tag-scheme reconciliation (important)
+### Tag conventions (shared across all apps)
-Before introducing any tag convention (trip names, the `non-trip` mark, or any future `ai-*` scheme), **inspect the live Immich instance and reconcile with what already exists** rather than inventing a parallel scheme. The user's trips already follow a convention — trip-cluster must read and respect it.
+Two clearly separated kinds of tags:
+
+- **Content / organizational tags** — trip names ("Italy 2019"), locations ("Kiev"), people. User-facing, part of the existing convention, **never namespaced**. trip-cluster must **read and respect** the existing trip-tag convention rather than invent a parallel one.
+- **Pipeline meta-tags** — everything the tooling generates as machinery, nested under a single parent **`_pipeline/`** so the whole set can be removed by deleting the parent and never clutters the tag list. Defined as a **shared convention in `shared/immich`** and used by every app in the monorepo:
+  - `_pipeline/processed` — categorizer: asset adjudicated (any outcome)
+  - `_pipeline/non-trip` — categorizer: everyday/noise
+  - `_pipeline/ai-rating/<0-5>` — image-rater's `ai-rating/<n>` moves under this root when migrated (M4)
+  - room for future meta-tags (review-state, etc.)
+
+**Reconciliation caveat:** before writing, **inspect the live Immich instance** and reconcile with anything already present (e.g. image-rater's existing un-namespaced `ai-rating/<n>`) rather than blindly creating duplicates.
 
 ## App shape & conventions
 - `categorize` CLI (argparse subcommands: `ingest` · `cluster` · `serve` · `apply`) + Flask factory `create_app()`.