> **Plain-language companion:** [v0.47.0.md](v0.47.0.md)

## v0.47.0 — Complete Embedding Programme: Sparse Vectors, Hybrid Search & Embedding API

**Status: Planned.** Derived from [plans/ecosystem/PLAN_PGVECTOR.md](../plans/ecosystem/PLAN_PGVECTOR.md) §6.3–6.4, merged from former v0.48.0 and v0.49.0.

> **Release Theme**
> Complete the AI/RAG feature set in one release: sparse/half-precision vector
> aggregates for tiered storage, reactive distance-predicate subscriptions for
> real-time anomaly detection, hybrid-search (BM25 + vector + metadata)
> cookbook, ergonomic `embedding_stream_table()` API, per-tenant ANN patterns,
> outbox-emitted embedding events, and a comprehensive vector benchmark suite.

> **Hard prerequisite:** v0.41.0–v0.46.0 must complete their hardening and
> embedding-infrastructure exit criteria before v0.47.0 begins feature work.

---

### Features

| ID | Title | Effort | Priority | Source |
|----|-------|--------|----------|--------|
| VH-1 | `sparsevec_avg`, `sparsevec_sum`, `halfvec_avg`, `halfvec_sum` aggregates | M | P1 | former v0.48 |
| VH-2 | Reactive subscriptions over distance predicates (`embedding <=> q < ε`) | M | P1 | former v0.48 |
| VH-3 | Hybrid-search corpus cookbook: BM25 + vector + metadata on one ST | S | P2 | former v0.48 |
| VH-4 | Benchmark suite: differential refresh + HNSW throughput, aggregate cost | M | P2 | former v0.48 |
| VA-1 | `embedding_stream_table()` ergonomic API: one-call RAG corpus setup | L | P1 | former v0.49 |
| VA-2 | Materialised k-NN graph research spike for fixed-pivot retrieval | L | P3 | former v0.49 |
| VA-3 | Per-tenant ANN indexing patterns: RLS-scoped embedding corpora | M | P2 | former v0.49 |
| VA-4 | Outbox-emitted embedding events for downstream consumption | M | P2 | former v0.49 |
| VA-5 | Starter repo and ecosystem positioning *(best-effort, not a release gate)* | S | P2 | former v0.49 |

**VH-1 — Sparse and half-precision aggregates.**
Extend the algebraic-aggregate framework to support pgvector type variants:
`sparsevec_avg`, `sparsevec_sum`, `halfvec_avg`, and `halfvec_sum`. The running
state for `halfvec_avg` is upcast to `vector(d)` for numerical stability; sparse
variants aggregate over the union of dimensions while treating absent entries as
zero. A stream table can maintain tiered storage: raw `vector(1536)` →
`halfvec(1536)` → `sparsevec(1536)`, each layer incrementally maintained and
independently indexed.

**VH-2 — Reactive distance subscriptions.**
Extend reactive subscriptions to support distance-predicate queries where the
query vector is bound at subscription-creation time. The subscription fires only
when rows newly satisfy or cease to satisfy the predicate, avoiding spurious
re-emits on every refresh. Uses the same NOTIFY-based delivery as other reactive
subscriptions.

**VH-3 — Hybrid-search cookbook.**
Publish `docs/tutorials/HYBRID_SEARCH_PATTERNS.md` with flat denormalised
corpora, RLS-scoped corpora, tiered search patterns, and performance-tuning
notes. The goal is a copy-paste path to BM25 + vector + metadata retrieval on
incrementally maintained tables.

**VH-4 — Vector benchmark suite.**
Add `benches/pgvector_bench.rs` to measure differential refresh plus ANN insert
throughput, vector aggregate reducer cost, drift-detection overhead, and storage
savings across `vector`, `halfvec`, and `sparsevec` representations.

**VA-1 — `embedding_stream_table()` ergonomic API.**
Add a high-level function that auto-generates the denormalization query, creates
the stream table, provisions indexes, configures post-refresh actions, and
returns a dry-run preview when `dry_run => true`. The API covers common
single-source-plus-joins RAG patterns without removing SQL control from power
users. Required: explicit index-inference documentation for `vector` and
`halfvec` cases.

**VA-2 — Materialised k-NN graph research spike.**
Explore whether pre-computing neighbour relationships for fixed pivot vectors is
worth the storage and maintenance overhead relative to ANN index scans. This is
research, not a release gate: the deliverable is a real trade-off analysis and,
optionally, a proof of concept if the numbers look promising.

**VA-3 — Per-tenant ANN indexing patterns.**
Document and example the production pattern for multi-tenant RAG using RLS and
tenant-scoped vector corpora. The emphasis is on safe defaults and explicit
security review, not just on query examples.

**VA-4 — Outbox-emitted embedding events.**
Extend the outbox surface so embedding changes can be emitted as downstream
application events. This lets external agents or search services react to
embedding churn without polling.

**VA-5 — Starter repo and ecosystem positioning.**
Provide a public starter repository and best-effort ecosystem material showing
how pg_trickle fits alongside pgvector after the hardening arc. Must never block
the release.

### Test Coverage

| ID | Title | Effort | Priority |
|----|-------|--------|----------|
| T-VH1 | Integration test: tiered storage (`vector` → `halfvec` → `sparsevec`) | M | P1 |
| T-VH2 | Reactive distance subscriptions: new anomaly alerts | M | P1 |
| T-VH3 | Hybrid-search corpus under write load | M | P2 |
| T-VA1 | Integration test: `embedding_stream_table()` generates correct stream tables | M | P1 |
| T-VA2 | Research benchmark: k-NN graph trade-off measurement | L | P2 |
| T-VA3 | Multi-tenant security test for ANN stream tables | M | P1 |
| T-VA4 | Outbox event emission for embedding changes | M | P2 |

**T-VH1.**
Build a multi-layer stream-table stack and verify all layers refresh correctly
when the source changes. Check both aggregate correctness and ANN query
consistency up to expected quantisation loss.

**T-VH2.**
Create a reactive distance subscription, insert and update rows around a chosen
similarity threshold, and assert that notifications fire only when predicate
truth changes.

**T-VH3.**
Run a hybrid-search corpus under sustained write load, record refresh lag and
query latency, and only write hard latency assertions once the CI baseline is
measured on the actual runner hardware.

**T-VA1.**
Call `embedding_stream_table()` with a realistic joins-and-aggregates spec,
verify the generated stream table and indexes, and compare results with the
manually written equivalent definition.

**T-VA2.**
If the k-NN research spike includes code, benchmark pivot-neighbour queries
against direct ANN index scans so the trade-off is evidence-based.

**T-VA3.**
Verify that tenant-scoped ANN stream tables respect RLS policies and do not leak
cross-tenant embeddings.

**T-VA4.**
Create an embedding-aware outbox stream and verify downstream subscribers see
correctly structured events when embeddings change.

### Conflicts & Risks

- **VH-1** needs careful handling of dimension mismatches and all-zero sparse
  vectors. The result definition must be explicit and heavily tested.
- **VH-2** depends on reactive subscription correctness remaining solid after
  the hardening arc; do not layer distance semantics on top of a shaky base.
- **VH-4** should publish measured baselines, not aspirational numbers.
- **VA-1** hides real query complexity. `dry_run => true` and explicit index
  heuristics are essential so expert users can audit the generated surface.
- **VA-2** is intentionally non-blocking research. Keep it from turning into an
  unbounded scope sink for a release that already includes a large ergonomic API.
- **VA-3** must be reviewed with the security model documented in the hardening
  arc; RLS examples are only useful if their trust boundaries are explicit.

### Exit Criteria

- [ ] VH-1: sparse and half-precision aggregates implemented and pass equivalence tests against non-incremental computation
- [ ] VH-1: `halfvec_avg` precision characterised and kept within an explicit error contract
- [ ] VH-2: reactive distance subscriptions fire only on predicate-truth changes
- [ ] VH-3: hybrid-search cookbook published with worked examples
- [ ] VH-4: benchmark suite runs in CI and publishes results alongside other performance data
- [ ] VA-1: `embedding_stream_table()` generates correct SQL, indexes, and monitoring configuration
- [ ] VA-1: `dry_run => true` returns the generated SQL without side effects
- [ ] VA-1: documented index-inference heuristics exist for vector and halfvec cases
- [ ] VA-2: research findings for materialised k-NN graphs are published; implementation remains optional
- [ ] VA-3: multi-tenant ANN documentation and security tests are complete
- [ ] VA-4: embedding outbox events emit and validate correctly
- [ ] VA-5: starter repo or equivalent public examples are available
- [ ] Extension upgrade path tested (`0.46.0 → 0.47.0`)
- [ ] `just lint` passes with zero warnings
- [ ] `just test-all` passes
