From 33ac3ae69c1481f63ee41512bb4d72faf63a7b30 Mon Sep 17 00:00:00 2001 From: mischa Date: Sat, 27 Jun 2026 20:19:09 +0200 Subject: [PATCH] docs(spec): harden pgvector spike from ce-doc-review (11 findings) Multi-persona review (coherence, feasibility, product-lens, security-lens, adversarial) surfaced that the spike proved DB access but not that DB access was the right path or that the signal was useful, and pinned a contract against a private, in-flight-changing schema. Applied 11 fixes: - Requirement 0: test the "REST can't expose embeddings" premise instead of asserting it; record which endpoints were checked and why insufficient. - Reframe Req 1 "real go/no-go" to access-only; signal-usefulness is M1.5's first task, not this spike's. - Mark the schema unsupported/internal, version-pinned; require an M1.5 re-probe/version-guard per Immich upgrade; tie shape to recorded model+ver. - Fix probe correctness: pgvector adapter / server-side vector_dims (psycopg3 returns vector as string); Postgres-internal join with optional SQLite cross-check; coverage over the image/embeddable population (both ratios). - DB-enforced read-only session; standalone .env loading; psycopg+pgvector added to Deliverables; roadmap correction now fixes both false claims. Co-Authored-By: Claude Opus 4.8 --- ...6-06-27-pgvector-embedding-spike-design.md | 114 +++++++++++++----- 1 file changed, 81 insertions(+), 33 deletions(-) diff --git a/docs/superpowers/specs/2026-06-27-pgvector-embedding-spike-design.md b/docs/superpowers/specs/2026-06-27-pgvector-embedding-spike-design.md index 69a3965..87f1353 100644 --- a/docs/superpowers/specs/2026-06-27-pgvector-embedding-spike-design.md +++ b/docs/superpowers/specs/2026-06-27-pgvector-embedding-spike-design.md @@ -14,9 +14,10 @@ on a prerequisite that never ran. This spec defines that prerequisite. M1.5 wants to cluster photos by visual similarity using the CLIP embeddings Immich already computes — the rescue signal for the GPS-poor old library where timestamp/GPS are weakest. -Immich's REST API does not cleanly expose raw embedding vectors, so the viable path is -**read-only access to Immich's Postgres pgvector embeddings**. Before M1.5 can build -clustering on that path, this spike must prove the path exists and pin its shape. +The working assumption is that Immich's REST API does not cleanly expose raw embedding +vectors, so the viable path is **read-only access to Immich's Postgres pgvector embeddings**. +That assumption is load-bearing — it must be *tested*, not asserted (see Requirement 0). Before +M1.5 can build clustering on that path, this spike must prove the path exists and pin its shape. ## Scope @@ -28,21 +29,43 @@ clustering on that path, this spike must prove the path exists and pin its shape ## Requirements (pass/fail) -The spike succeeds when it answers all four against the **live** database and records the -answers in the findings doc. Phrased as plain questions: +The spike succeeds when it answers all of the following against the **live** database and +records the answers in the findings doc. Phrased as plain questions: +0. **Does M1.5 actually need raw vectors (vs. a REST similarity query)?** Before relying on + DB access, state what M1.5's clustering needs — raw embedding vectors or a pairwise + similarity-neighbor query — and record which Immich REST endpoints were checked (e.g. + `/api/search/smart`, any asset-similarity/duplicate endpoint) and *why each is + insufficient*. This turns the load-bearing "REST can't expose embeddings" premise into a + documented finding; if a REST path suffices, the DB-access path below is unnecessary. 1. **Can we read the CLIP embeddings at all?** Connect read-only over the LAN and read the - embedding vectors Immich stores. *This is the real go/no-go: if the embeddings are not - readable, M1.5 needs a different plan.* -2. **Can we join each embedding back to a photo we already track?** Our SQLite store keys - assets by Immich asset ID. Confirm each embedding row carries that same ID so an - embedding can be matched to a known asset. *Without the join the vectors are unusable.* + embedding vectors Immich stores. *This de-risks **access** only: if the embeddings are not + readable, M1.5 needs a different plan. It does **not** prove the signal is useful (see the + signal-usefulness note below).* +2. **Can we join each embedding back to a photo we already track?** Confirm **inside + Postgres** that each embedding row's key references `assets.id` — the join that must hold + for the vectors to be usable. That asset ID is the same key our SQLite store uses, so a + cross-check against SQLite (`shared/photoflow/core`) is *optional* confirmation and must + degrade gracefully when the store is unpopulated (a fresh checkout has no `ingest` run), + rather than failing this requirement for the wrong reason. 3. **What are the exact shapes?** Record the table name, embedding column, **vector - dimension**, and the correct pgvector **distance operator** for similarity. These have - drifted across Immich versions, so they must be read from *this* DB, not assumed. -4. **What is the coverage?** What fraction of library assets currently have an embedding - (the user is re-running CLIP with a stronger model — has it finished?). *Tells M1.5 how - much it can lean on the signal.* + dimension**, and the correct pgvector **distance operator** for similarity. These are + **model- and version-dependent** (and have drifted across Immich versions), so they must be + read from *this* DB, not assumed — and the recorded shape is valid only for the model + + Immich version observed (see Requirement 4 / Risks). +4. **What is the coverage?** What fraction of the **embeddable** library currently has an + embedding. Compute against the image/embeddable population, not all assets — + `count(distinct embedding.assetId) / count(assets WHERE type = 'IMAGE')` (or Immich's + equivalent asset-type filter) — since videos and other non-image rows CLIP never embeds + would otherwise deflate the ratio. Record both the raw and image-only ratios. *Tells M1.5 + how much it can lean on the signal — but the user is re-running CLIP with a stronger model, + so coverage (and the dimension in Requirement 3) is a snapshot of whichever model is live + when the probe runs.* + +**Signal-usefulness is M1.5's first task, not this spike's.** Passing Requirements 1–4 proves +the embeddings are reachable and well-shaped; it does **not** prove visual similarity actually +rescues trip detection in the GPS-poor library. Before building clustering, M1.5 must validate +the signal (e.g. eyeball nearest-neighbour quality on a sample of the GPS-poor set). ## What we already know (to verify, not assume) @@ -53,6 +76,11 @@ vectors. Names and dimension have changed across versions and the user's re-run stronger model, so the probe **discovers** these rather than trusting them — the list above is only the set of candidates to probe first. +This is Immich's **undocumented, internal** schema with no deprecation contract: it can be +renamed or restructured on any Immich upgrade. The probe therefore records the exact Immich +version observed alongside the shape, and M1.5 must budget a re-probe / version-guard on every +Immich upgrade — the "contract" the findings doc hands M1.5 is version-pinned, not durable. + ## Approach Disposable read-only probe + two durable artifacts (chosen over a throwaway-only or a @@ -63,11 +91,23 @@ a written contract is the cheap part that saves M1.5 from guessing). - New optional env var **`IMMICH_DB_URL`** (a Postgres DSN), added to `.env.example` and loaded by `config.py` as an **optional** field (REST creds stay required; the DSN is only - needed for the spike / M1.5). -- Simplest credentials that work: Immich's existing Postgres user, kept read-only by the - probe issuing **only `SELECT`**. A dedicated least-privilege read-only role is documented as - recommended hardening for M1.5 but is **not required** to prove feasibility. -- Driver: `psycopg` (psycopg3). Add to the spike's dependencies; M1.5 will formalize it. + needed for the spike / M1.5). `config.py` reads only `os.environ`, and the repo loads `.env` + solely via docker-compose's `env_file`, so a standalone host-run probe must populate the + environment itself — either run it via `docker compose run` (so `env_file` applies) or load + `.env` explicitly first (e.g. `set -a; source .env; set +a`). If run in-container, confirm + container-to-Immich-Postgres network reachability. +- Read-only must be **enforced at the DB/session layer**, not just by which statements the + script issues: open the session read-only (`SET default_transaction_read_only = on` / + `SET TRANSACTION READ ONLY`) or connect via a role granted only `SELECT`. The simplest + credentials that work are Immich's existing Postgres user, but that user is write/DDL-capable + against the **source-of-truth** DB, so the session-level guard is required to remove write + capability while the spike runs. A dedicated least-privilege read-only role remains + recommended hardening for M1.5. +- Driver: `psycopg` (psycopg3), plus the **`pgvector`** Python package. psycopg3 returns a + `vector` column as a *string* unless the adapter is registered, so call + `pgvector.psycopg.register_vector(conn)` after connect — or read the dimension server-side + (`SELECT vector_dims(embedding) …` / catalog `atttypmod`) and run the `<=>` check in SQL, + which needs no adapter. Add these to the spike's dependencies; M1.5 will formalize them. ### The probe @@ -78,13 +118,17 @@ read-only. It: 2. discovers candidate embedding tables/columns from the catalog (probe `smart_search` / `embedding` first, then fall back to scanning `information_schema` for `vector`-typed columns) — so it survives version drift; -3. reads one embedding and reports its **dimension** and a sample of the **distance operator** - working (e.g. a `... ORDER BY embedding <=> embedding LIMIT 5` self-similarity sanity - check); -4. verifies the **asset-ID join**: the embedding row's key matches `assets.id`, and that same - ID is one we store in SQLite (`shared/photoflow/core`); -5. computes **coverage**: `count(embeddings) / count(assets)` (and, if cheap, broken down by - whether the asset is an image vs video); +3. reads one embedding and reports its **dimension** — via the registered `pgvector` adapter + or server-side `vector_dims(embedding)`, **not** the length of a raw string — and a sample + of the **distance operator** working (e.g. a `... ORDER BY embedding <=> embedding LIMIT 5` + self-similarity sanity check, casting literals to `::vector` as needed); +4. verifies the **asset-ID join inside Postgres**: the embedding row's key references + `assets.id`. As *optional* confirmation it also checks whether that ID is one we store in + SQLite (`shared/photoflow/core`), degrading gracefully (not failing the check) when the + store is unpopulated; +5. computes **coverage** against the embeddable population: + `count(distinct embedding.assetId) / count(assets WHERE type = 'IMAGE')` (or Immich's + equivalent asset-type filter), recording **both** the raw and image-only ratios; 6. prints a human-readable report **and** writes/refreshes the findings doc. It hardcodes nothing destructive, takes no write path, and is safe to re-run. @@ -93,13 +137,17 @@ It hardcodes nothing destructive, takes no write path, and is safe to re-run. 1. `scripts/pgvector_spike.py` — disposable read-only probe (removed or left as a documented one-off after M1.5 internalizes its findings). -2. `IMMICH_DB_URL` in `.env.example`; optional `immich_db_url` field in `config.py`. -3. **`docs/superpowers/specs/2026-06-27-pgvector-embedding-findings.md`** — the one-page pinned +2. `IMMICH_DB_URL` in `.env.example`; optional `immich_db_url` field in `config.py`. Spike + dependencies added: `psycopg` (psycopg3) and the `pgvector` Python package. +3. **`docs/superpowers/specs/2026-06-27-pgvector-embedding-findings.md`** — the one-page schema that becomes M1.5's contract: table, embedding column, vector dimension, distance - operator, the embedding → `assetId` → SQLite `asset` join, coverage %, Immich version, and - the (optional) read-only-role recipe. -4. Roadmap correction: M1.5's status note updated to "dependency spike pending/this spec" - instead of "verified in M1." + operator, the embedding → `assetId` → SQLite `asset` join, coverage % (raw and image-only), + the model **and** Immich version observed, and the (optional) read-only-role recipe. The doc + must state that this is Immich's **unsupported internal schema**, valid only for the recorded + model + version, and that M1.5 re-probes / version-guards on every Immich upgrade. +4. Roadmap correction: fix **both** false claims named in "Why this exists" — the narrative + that M1 "runs a read-only feasibility spike" (→ "spike defined/pending per this spec") **and** + M1.5's status note "verified in M1" (→ "dependency spike pending/this spec"). ## Out of scope (explicitly M1.5)