> **Plain-language companion:** [v0.38.0.md](v0.38.0.md)

## v0.38.0 — EC-01 Correctness Sprint (Hard Gate)

**Status: Planned.** This is a focused release with a single objective: close EC-01 (join phantom rows) with a hard release gate. Derived from [plans/PLAN_OVERALL_ASSESSMENT_8.md](plans/PLAN_OVERALL_ASSESSMENT_8.md) §4.1 (Finding C1).

> **Release Theme**
> v0.38.0 will NOT be tagged until EC-01 is proven closed. This release breaks
> the deferral pattern that has carried EC-01 through 6+ releases (v0.20–v0.37).
> The only feature in v0.38.0 is unconditional PH-D1 cleanup plus join row-id
> convergence, verified by a comprehensive DIFF-vs-FULL property test. All
> operational truthfulness, docs, and ecosystem work moves to v0.39.0.
>
> **Release Gate:** EC-01 property convergence test must pass with 100% success
> before v0.38.0 is tagged. If EC-01 is not closed when the v0.38.0 window
> closes, the release is postponed—not the feature.

---

### Features (EC-01 Only)

| ID | Title | Effort | Priority |
|----|-------|--------|----------|
| C38-1 | Unconditional PH-D1 cleanup plus join row-id convergence | L | P0 |
| C38-2 | Plan-time `RowIdSchema` verification in DVM planner | M | P0 |

**C38-1 — Unconditional PH-D1 cleanup plus join row-id convergence.**

Join deltas currently return `is_deduplicated: false` for all joins, allowing
phantom rows to accumulate silently across refresh cycles. This fix has two
parts:

1. **Make PH-D1 cleanup unconditional.** For all non-deduplicated join outputs,
   apply phantom-row cleanup after each refresh without condition checks on
   join shape, prior residuals, or refresh count. This stops new phantoms from
   accumulating while the root fix lands.

2. **Rework the join row-id formula.** The current formula diverges between:
   - **Part 1a** (left INSERT new, join with current right): uses right row-id
     from R₁
   - **Part 1b** (left DELETE old, join with pre-change right): uses right row-id
     from R₀
   - **Part 2** (pre-change left, join with RIGHT delta): mixed semantics

   When the right side has deleted rows, Part 1a and Part 1b produce different
   row_ids for the same logical join result, breaking weight aggregation.
   v0.38.0 redesigns the formula so all three parts converge on the same logical
   identity, making join outputs truly deduplicated or ensuring PH-D1 cleanup
   catches all phantoms.

3. **Wire `RowIdSchema` verification.** The schema exists in v0.36.0 but is
   mostly dormant. Carry it through `DiffResult`, check compatibility at plan
   time, and either prove join is safe or explicitly force FULL refresh. This
   turns row-id design into a guardrail.

**Acceptance criteria:**
- Join deltas either return `is_deduplicated: true` or are unconditionally cleaned
  by PH-D1
- Q07 and Q15 are no longer in `IMMEDIATE_SKIP_ALLOWLIST`
- No phantom accumulation across 100 random-mutation cycles

**C38-2 — Plan-time `RowIdSchema` verification in DVM planner.**

`RowIdSchema` exists as a type but the verifier is mostly dormant. This task
enforces schema compatibility at plan time so incompatible joins either fail
creation with a clear error or explicitly route to FULL refresh. The check must
be inexpensive and clear so users understand why a join requires FULL mode if it
does.

### Test Coverage (EC-01 Only)

| ID | Title | Effort | Priority |
|----|-------|--------|----------|
| T-C38-1 | EC-01 DIFF-vs-FULL property generator for joins | L | P0 |

**T-C38-1 — EC-01 DIFF-vs-FULL property generator (Release Gate).**

Create a dedicated test harness that:

1. **Generates random DML sequences**: insert/update/delete in random order and
   timing over tables participating in multi-table joins.

2. **Covers EC-01 failure modes:**
   - Left/right co-deletes (the case where R₀ ≠ R₁)
   - Scalar-subquery joins (where the subquery result can shift)
   - Q07-style 5-table joins
   - Q15-style derived-table + subquery-in-WHERE shapes

3. **After every refresh cycle:** Compare DIFF result set byte-for-byte against
   FULL result set. If any divergence is found, the test FAILS and the release
   is blocked.

4. **Scale to confidence:** Run at least 1000–10,000 cycles per join shape,
   randomizing left/right table order, join key types (int/uuid/composite),
   and DML timing.

5. **Report clearly:** If divergence is found, output the exact row_id values,
   action counts (INSERT/DELETE), aggregated weights, and the FULL-vs-DIFF
   row_id/value pairs that differ.

**This test must pass 100% before v0.38.0 is tagged.**

### What's NOT in v0.38.0

- Operational truthfulness (backpressure, wake, docs) — moves to v0.39.0
- SQLSTATE-first retry classification rollout — moves to v0.39.0
- Generated `CONFIGURATION.md` — moves to v0.39.0
- `UPGRADING.md` through v0.37 — moves to v0.39.0
- OpenTelemetry operator guide — moves to v0.39.0
- pg_partman / TimescaleDB ecosystem coverage — moves to v0.40.0 or later
- `history_prune_interval_seconds` wiring — moves to v0.40.0

### Release Confidence & Deferral Protection

> **Why no other features?** Correctness is the foundation. Adding operational
> polish or new features without first closing EC-01 repeats the last 6 releases
> of deferral.
>
> **Deferral Protection:** If EC-01 is still not closed when the v0.38.0 window
> approaches completion, the release is POSTPONED. No more "EC-01 will ship next
> time." The release date is flexible; EC-01 closure is not.
Verify that a refresh span is exported when enabled, that timeouts do not wedge
the refresh path, and that collector failures degrade predictably.

**T-H38-4.**
Add CI checks that diff source-discovered GUC metadata against
`docs/CONFIGURATION.md` and ensure `docs/UPGRADING.md`'s latest documented
target matches the extension version. This is the cheapest way to prevent the
same docs drift from recurring.

**T-H38-5.**
If the deferred v0.37 ecosystem coverage is still open, add explicit daily or
manual workflows proving pg_partman and TimescaleDB compatibility rather than
leaving the support story implied.

### Conflicts & Risks

- **H38-1/H38-2** touch the DVM core and therefore carry high regression risk.
  They require property tests and DIFF-vs-FULL comparison before the release is
  considered complete.
- **H38-3** must not promise more safety than the implementation provides. If
  durable hold semantics are not ready, deprecate or narrow the surface instead
  of shipping ambiguous pause behavior.
- **H38-6/H38-7** must be source-driven wherever possible. Hand-edited docs are
  exactly what created the current drift.

### Exit Criteria

- [ ] H38-1: Non-deduplicated join outputs no longer accumulate phantom rows in Q07/Q15-style and random property workloads
- [ ] H38-2: Incompatible `RowIdSchema` pipelines fail planning or explicitly force FULL refresh with a clear reason
- [ ] H38-3: `pg_trickle.enforce_backpressure` is either fully wired with hysteresis and tests or clearly marked experimental/off with truthful docs
- [ ] H38-4: SQLSTATE drives retry classification on scheduler, refresh, and catalog hot paths
- [ ] H38-5: Wake docs, warnings, and tests no longer imply background-worker `LISTEN` support
- [ ] H38-6: `docs/CONFIGURATION.md`, `docs/SCALING.md`, and `docs/TROUBLESHOOTING.md` are source-aligned and include v0.37 trace/vector GUCs
- [ ] H38-7: `docs/UPGRADING.md` is updated through v0.37 and TRUNCATE/CDC semantics are explicitly documented
- [ ] H38-8: OTEL operator guide is published and collector integration tests pass
- [ ] H38-9: `history_prune_interval_seconds` is wired into scheduler cadence or removed from the public surface
- [ ] H38-10: Deferred pg_partman / TimescaleDB coverage is either complete or explicitly carried with a tracked issue
- [ ] Extension upgrade path tested (`0.37.0 → 0.38.0`)
- [ ] `just check-version-sync` passes

---
