# Testing And Release

The project has several validation layers. Use the narrowest test that covers
your change while preserving the release gates for cross-boundary behavior.

## Fast Local Checks

```bash
cd graph
cargo fmt --check
cargo test --features pg17
```

Run pgrx integration tests:

```bash
cd graph
cargo pgrx test pg17
```

Build docs:

```bash
cd graph
cargo doc --features pg17 --no-deps
```

Check docs against local references, the SQL-facing Rust API, registered GUCs,
and the Rust source map:

```bash
scripts/check_docs_drift.sh
```

The aggregate script runs `scripts/check_doc_references.py`,
`scripts/check_sql_api_drift.py`, and `scripts/check_rust_doc_map_drift.py`.
Use `python3 scripts/check_sql_api_drift.py --list-implemented` or
`python3 scripts/check_rust_doc_map_drift.py --list` when you need to inspect
the implementation-side inventory.

Check dependency freshness without updating manifests:

```bash
python3 scripts/check_dependency_updates.py
```

The dependency checker enforces the release policy that recommended dependency
updates must be at least 6 hours old. It flags newer releases for review and
requires explicit confirmation, for example
`--update cargo:serde --yes`, before it rewrites supported manifests. Any
package-manager fetch or install needed after a manifest change must go through
`sfw`.

## Rust Unit Tests

Module-local Rust tests cover:

| Module | Examples |
|---|---|
| `node_store.rs` | active bits, tombstones, mmap invariants |
| `edge_store.rs` | CSR construction, dedupe, reverse store, weights |
| `bfs.rs` | depth limits, edge filters, overlay traversal, paths |
| `filter_index.rs` | dense/sparse storage and typed filter checks |
| `persistence.rs` | roundtrip and corrupt artifact rejection |
| `types.rs` | parser helpers and filter operation boundaries |
| `config.rs` | GUC parser fallbacks |
| `safety.rs` | SQLSTATE mapping |

## pgrx SQL Tests

`src/pg_tests/` covers installed extension behavior inside PostgreSQL:

| File | Scope |
|---|---|
| `discovery.rs` | schema and table-set discovery |
| `gql.rs` | GQL SQL facade, mapped `CREATE`, ACL, tenant, and internal overlay proof tests |
| `cypher.rs` | openCypher compatibility facade and shared-plan SQL behavior |
| `traversal_api.rs` | traversal helpers, labels, weights, directions, strategies |
| `traversal_paths.rs` | paths, aggregation, strict specs |
| `filters.rs` | typed filter behavior |
| `registration_search.rs` | registration and source search |
| `sync_config_build.rs` | sync mode, GUC, OOM, build scanner, and schema drift behavior |
| `components_jobs.rs` | component functions and job APIs |
| `maintenance_admin.rs` | sync apply, vacuum, maintenance |
| `named_graphs.rs` | default-graph compatibility and named-graph policy guardrails |
| `workflow_search_api.rs` | `find`, `expand`, `find_related` |
| `workflow_relationship_api.rs` | `path`, `connection`, `neighborhood` |
| `workflow_validation.rs` | argument validation and unsupported modes |
| `synthetic_release.rs` | synthetic release-gate SQL fixture through pgrx |

See [SQL Tests](./sql-tests) for the full SQL test inventory and the boundary
each layer is meant to cover.

### Provision pgrx Test PostgreSQL

The pgrx SQL tests need a PostgreSQL installation for the target major version
and a pgrx test cluster. They do not need preloaded application tables or
external datasets. Test fixtures in `src/pg_tests/common.rs` create and drop
their own source tables, call `graph.reset()`, register graph metadata, and
build the graph inside the test database.

Install PostgreSQL and ensure the matching `pg_config` is available. Examples:

```bash
# macOS/Homebrew PostgreSQL 17
export PG_CONFIG=/opt/homebrew/opt/postgresql@17/bin/pg_config

# Debian/Ubuntu PostgreSQL 17
export PG_CONFIG=/usr/lib/postgresql/17/bin/pg_config
```

Install Nix on machines that own release dependency freshness remediation. The
dependency checker can report flake input updates, but `flake.lock` should be
regenerated and validated with `nix flake lock` rather than edited by hand. On
macOS, upstream Nix requires the daemon installer because single-user installs
are no longer supported.

Install and initialize pgrx once for that PostgreSQL version:

```bash
cargo install cargo-pgrx --version 0.19.1 --locked
cargo pgrx init --pg17 "$PG_CONFIG"
```

Run the SQL tests from the crate directory:

```bash
cd graph
cargo pgrx test pg17
```

Run a focused pgrx test by name when iterating:

```bash
cd graph
cargo pgrx test pg17 synthetic_fixture_exercises_release_gate_sql_shape
```

If pgrx reports an extension or SQL-definition mismatch after code changes,
reinstall the extension against the same PostgreSQL version and rerun:

```bash
cd graph
cargo pgrx install --pg-config "$PG_CONFIG" --features pg17 --no-default-features
cargo pgrx test pg17
```

## Heavy Tests

Some behavior must be tested outside `cargo pgrx test` because pgrx SQL tests
run inside generated SQL functions and do not reflect normal client/server
error boundaries.

```bash
cd graph
PG_VERSION_FEATURE=pg17 DBNAME=pggraph_boundary ./tests/heavy/run_sqlstate_acl_boundary.sh
```

Heavy suites:

| Script | Purpose |
|---|---|
| `run_sqlstate_acl_boundary.sh` | Client-visible SQLSTATEs and restricted-role ACL behavior |
| `tx_delta_lifecycle.sh` | Internal mutable-overlay transaction lifecycle proof |
| `gql_create_tx_lifecycle.sh` | Public mapped GQL `CREATE` rollback and commit lifecycle proof |
| `gql_set_tx_lifecycle.sh` | Public mapped GQL property `SET` rollback and filter-delta lifecycle proof |
| `gql_delete_tx_lifecycle.sh` | Public mapped GQL edge-row `DELETE` rollback and bidirectional tombstone lifecycle proof |
| `gql_write_recheck_race.sh` | Public mapped GQL write predicate recheck race proof |
| `gql_merge_race.sh` | Public mapped GQL `MERGE` same-key race proof |
| `tx_delta_crash_recovery.sh` | Internal mutable-overlay crash/reload proof |
| `run_pgbench_sync.sh` | Concurrent DML and trigger sync stress |
| `crash_recovery.sh` | Persisted graph recovery and corruption rejection after postmaster kill |
| `run_memory_sanitizers.sh` | pgrx suite plus optional ASan/Valgrind |
| `package_validate.sh` | Packaging checks |
| `fresh_install_smoke.sh` | Install and quick use smoke |
| `function_metadata_audit.sh` | SQL signature and metadata audit |
| `backup_restore_validate.sh` | Dump/restore state validation |
| `docker_smoke.sh` | Container smoke test |
| `pg_upgrade_validate.sh` | Upgrade validation |
| `background_job_lock_regression.sh` | Background job lock/failure behavior |
| `build_lock_regression.sh` | Build/vacuum lock conflict behavior |
| `concurrency_stress.sh` | Mixed build/sync/vacuum/traversal stress |
| `synthetic_release_smoke.sh` | Deterministic synthetic release-gate graph |
| `run_synthetic_release_evidence.sh` | Synthetic release smoke with metadata and artifact report |
| `playground_release_gate.sh` | Docker-backed playground query catalog and result stability |
| `measure_build_rss.sh` | Peak RSS measurement |
| `measure_mmap_pss.sh` | Linux PSS measurement for mmap sharing |

See [SQL Tests](./sql-tests) for the full heavy-script inventory.

### SQLSTATE And ACL Boundary

`run_sqlstate_acl_boundary.sh` validates errors from client-side `psql` output
instead of wrapping calls in nested PL/pgSQL exception handlers. That is closer
to real application behavior and avoids confusing pgrx panic/error plumbing
with PL/pgSQL's nested exception machinery.

It covers representative public errors:

| SQLSTATE | Boundary |
|---|---|
| `PG003` | Traversal before the graph has been built. |
| `PG010` | Missing seed node. |
| `PG005` | Invalid Unicode-heavy filter input. |
| `55000` | Graph queries disabled by configuration. |
| `PG002` | Restricted-role access to global analytics. |

### Concurrent Sync Stress

`run_pgbench_sync.sh` creates a disposable schema, enables trigger-buffered
sync, runs concurrent insert/update/delete traffic through `pgbench`, applies
the sync buffer, and checks post-sync query behavior.

Useful knobs:

```bash
cd graph
PG_VERSION_FEATURE=pg17 DBNAME=pggraph_pgbench CLIENTS=16 JOBS=4 TIME=120 \
  RATE=100 MAX_APPLY_MS=5000 MAX_QUERY_MS=250 MIN_ROWS_APPLIED=0 \
  ./tests/heavy/run_pgbench_sync.sh
```

`RATE` keeps the workload bounded so the release gate measures sync behavior
instead of turning `graph.apply_sync()` into an accidental soak test.

### Read Latency Under Sync Backlog

`read_latency_under_sync.sh` captures baseline read latency before changing
query-time sync semantics. It builds a line graph, enables trigger sync, then
records p50/p95/p99 latency for traversal, shortest path, and weighted shortest
path reads across four scenarios: no pending sync, a small pending backlog, a
large pending backlog, and concurrent writers appending to `graph._sync_log`.
For fixed-backlog scenarios, each query kind is measured from a rebuilt graph
with a fresh backlog, and backlog counts come from `graph.sync_health()`.

Useful knobs:

```bash
cd graph
DBNAME=pggraph_read_latency NODE_COUNT=10000 SAMPLES=25 \
  SMALL_BACKLOG=100 LARGE_BACKLOG=5000 CLIENTS=4 JOBS=2 TIME=30 RATE=100 \
  ./tests/heavy/read_latency_under_sync.sh
```

Set `QUERY_FRESHNESS=off` to compare the compatibility/manual catch-up path
against the default query-time catch-up behavior.

The script writes `read-latency-samples.csv` and
`read-latency-summary.csv` in the current directory. Keep those outputs with
milestone evidence when validating topology-read auto-sync behavior.

### Crash Recovery

`crash_recovery.sh` must run only against a disposable PostgreSQL instance. It
reads `PGDATA/postmaster.pid`, sends `kill -9` to that postmaster, restarts the
instance with `pg_ctl`, verifies graph auto-load, corrupts the persisted graph
file, and verifies the loader rejects the corruption cleanly.

```bash
cd graph
PG_VERSION_FEATURE=pg17 DBNAME=pggraph_crash PGDATA=/path/to/disposable/pgdata \
  ./tests/heavy/crash_recovery.sh
```

When the instance needs custom server options, pass the same restart options
with `POSTGRES_OPTS`, for example `POSTGRES_OPTS="-p 55433 -k /tmp"`.

`tx_delta_crash_recovery.sh` uses the same disposable-cluster pattern but
installs the extension with the internal `pg_test` surface. It opens an
uncommitted transaction-local edge overlay, kills the postmaster, restarts the
cluster, and verifies the persisted base graph reloads without the uncommitted
overlay.

```bash
cd graph
PG_VERSION_FEATURE=pg17 DBNAME=pggraph_tx_delta_crash \
  PGDATA=/path/to/disposable/pgdata ./tests/heavy/tx_delta_crash_recovery.sh
```

### Memory And FFI Boundary

`run_memory_sanitizers.sh` runs the pgrx SQL suite and a long-lived pgbench sync
smoke by default. ASan and Valgrind are opt-in because they need compatible
toolchains and PostgreSQL builds.

```bash
cd graph
DBNAME=pggraph_pgbench ./tests/heavy/run_memory_sanitizers.sh
RUN_ASAN=1 ./tests/heavy/run_memory_sanitizers.sh
RUN_VALGRIND=1 PGDATA=/path/to/pgdata POSTGRES_BIN=/path/to/postgres \
  ./tests/heavy/run_memory_sanitizers.sh
```

### Packaging, Docker, Upgrade, And Workers

```bash
cd graph
PG_VERSION_FEATURE=pg17 ./tests/heavy/package_validate.sh
PG_VERSION_FEATURE=pg17 DBNAME=pggraph_install ./tests/heavy/fresh_install_smoke.sh
PG_VERSION_FEATURE=pg17 DBNAME=pggraph_metadata ./tests/heavy/function_metadata_audit.sh
PG_VERSION_FEATURE=pg17 SOURCE_DB=pggraph_backup_src RESTORE_DB=pggraph_backup_dst ./tests/heavy/backup_restore_validate.sh
./tests/heavy/docker_smoke.sh
OLD_BINDIR=/path/to/old/bin NEW_BINDIR=/path/to/new/bin \
  OLD_DATADIR=/tmp/pg-old NEW_DATADIR=/tmp/pg-new \
  ./tests/heavy/pg_upgrade_validate.sh
```

Worker and memory sizing checks:

```bash
cd graph
PG_VERSION_FEATURE=pg17 DBNAME=pggraph_background_lock ./tests/heavy/background_job_lock_regression.sh
PG_VERSION_FEATURE=pg17 DBNAME=pggraph_build_lock ./tests/heavy/build_lock_regression.sh
PG_VERSION_FEATURE=pg17 DBNAME=pggraph_concurrency ./tests/heavy/concurrency_stress.sh
PG_VERSION_FEATURE=pg17 DBNAME=pggraph_rss NODE_COUNT=200000 MAX_RSS_MB=0 ./tests/heavy/measure_build_rss.sh
DBNAME=graph_test BACKENDS=50 ./tests/heavy/measure_mmap_pss.sh
```

`measure_mmap_pss.sh` is Linux-only. It keeps graph-query backends alive and
reads `/proc/<pid>/smaps_rollup` so release evidence can use proportional set
size for mmap sharing instead of summed RSS.

### Synthetic Release Dataset

`synthetic_release_smoke.sh` creates a deterministic synthetic relational graph
for release gates. This is not a benchmark claim; it is a reproducible test
fixture that validates build, persistence, auto-load, traversal, search,
filters, shortest paths, weighted paths, and component summaries without
requiring a real dataset.

```bash
cd graph
DBNAME=pggraph_synthetic NODE_COUNT=50000 HUB_FANOUT=1000 \
  MAX_BUILD_MS=60000 MAX_QUERY_MS=1000 \
  ./tests/heavy/synthetic_release_smoke.sh
```

The release gate runs it by default. Tune with:

```bash
RUN_SYNTHETIC=1 SYNTHETIC_NODE_COUNT=50000 SYNTHETIC_HUB_FANOUT=1000 \
  SYNTHETIC_MAX_BUILD_MS=60000 SYNTHETIC_MAX_QUERY_MS=1000 \
  ./tests/heavy/run_release_gate.sh
```

Set `RUN_SYNTHETIC=0` only when the local environment cannot run disposable
PostgreSQL databases.

To capture metadata and inspect the persisted `.pggraph` artifact:

```bash
cd graph
NODE_COUNT=50000 HUB_FANOUT=1000 ./tests/heavy/run_synthetic_release_evidence.sh
```

The evidence wrapper writes JSON under `target/release-evidence/` and
uses `scripts/inspect_pggraph_artifact.py` when the artifact is present.

### Playground Release Gate

`playground_release_gate.sh` validates the Streamlit playground as a heavy
release gate. It prepares the Docker-backed Panama dataset, imports the same
shared query catalog used by the playground UI, and checks every sidebar example
against fixed result summaries. Stable examples compare row counts and result
hashes; operational examples with volatile job IDs or timings compare row
counts.

```bash
cd graph
./tests/heavy/playground_release_gate.sh
PGGRAPH_PLAYGROUND_MODE=mutable ./tests/heavy/playground_release_gate.sh
```

By default the script uses the sandbox Docker container and PostgreSQL port
`55432` and validates the `csr_readonly` playground catalog. Set
`PGGRAPH_PLAYGROUND_MODE=mutable` to prepare `mutable_overlay` and validate the
mutable GQL examples. Set `PREPARE_PLAYGROUND=0` and provide `PGGRAPH_DSN` when
validating an already-prepared playground database. Set
`PGGRAPH_PLAYGROUND_YES=1` only when release automation is allowed to approve
the first Panama dataset download.

## Release Gate

```bash
cd graph
PG_VERSION_FEATURE=pg17 ./tests/heavy/run_release_gate.sh
```

`run_release_gate.sh` is the release-evidence aggregator. It runs formatting,
clippy, docs, Rust tests, pgrx SQL tests, cargo-deny, fuzz binary compile,
projection manifest/segment fuzz seed corpora, package validation, install
smoke, metadata audit, SQLSTATE/ACL boundaries, backup/restore, lock
regressions, concurrency stress, synthetic release smoke, GQL write lifecycle
gates, pgbench sync stress, playground validation, optional crash gates, and
optional Docker smoke depending on its `RUN_*` environment variables.

The default command is not the full production release gate: Docker and crash
gates are off unless explicitly enabled, and crash gates also require a
disposable `PGDATA`. For staged release evidence, keep skipped pieces explicit
in the command and record the reason:

```bash
cd graph
PG_VERSION_FEATURE=pg17 DB_PREFIX=pggraph_release_m16 \
  RUN_PLAYGROUND=0 RUN_PGBENCH=0 RUN_DOCKER=0 \
  RUN_CRASH=0 RUN_TX_DELTA_CRASH=0 \
  ./tests/heavy/run_release_gate.sh
```

Enable the heavy production-only pieces explicitly when the environment is
ready:

```bash
cd graph
PG_VERSION_FEATURE=pg17 DB_PREFIX=pggraph_release \
  RUN_DOCKER=1 RUN_CRASH=1 RUN_TX_DELTA_CRASH=1 \
  PGDATA=/path/to/disposable/pgdata \
  ./tests/heavy/run_release_gate.sh
```

Run crash gates only against a disposable PostgreSQL cluster because they kill
and restart the postmaster:

```bash
cd graph
PGHOST=/tmp/pggraph-crash PGPORT=55437 PGDATA=/tmp/pggraph-crash/data \
  POSTGRES_CTL=/path/to/pg_ctl POSTGRES_OPTS="-p 55437 -k /tmp/pggraph-crash" \
  DBNAME=pggraph_release_crash ./tests/heavy/crash_recovery.sh

PGHOST=/tmp/pggraph-crash PGPORT=55437 PGDATA=/tmp/pggraph-crash/data \
  POSTGRES_CTL=/path/to/pg_ctl POSTGRES_OPTS="-p 55437 -k /tmp/pggraph-crash" \
  PG_CONFIG=/path/to/pg_config PG_VERSION_FEATURE=pg17 \
  DBNAME=pggraph_release_tx_delta_crash \
  ./tests/heavy/tx_delta_crash_recovery.sh
```

`crash_recovery.sh` proves committed trigger rows survive postmaster death,
explicit sync replay works after restart, maintenance can rebuild/persist, and
corrupt persisted artifacts are rejected. `tx_delta_crash_recovery.sh` proves
transaction-local overlays are visible before crash but do not survive restart.

Use Docker for PostgreSQL major matrix validation when local headers are not
available:

```bash
cd graph
./tests/heavy/run_pg_matrix_docker.sh
```

Set `RUN_FULL_MATRIX=1` on `run_release_gate.sh` to run every configured
PostgreSQL major through `run_pg_matrix.sh`. Set `RUN_RSS=1` to include a
large-build peak RSS measurement; use `MAX_RSS_MB` when there is a recorded
release threshold.

External gates can also be run independently:

```bash
cd graph
DBNAME=pggraph_release_pgbench CLIENTS=4 JOBS=2 TIME=30 \
  ./tests/heavy/run_pgbench_sync.sh

IMAGE=pggraph:smoke CONTAINER=pggraph-smoke PG_MAJOR=17 \
  ./tests/heavy/docker_smoke.sh

PGGRAPH_PLAYGROUND_YES=1 ./tests/heavy/playground_release_gate.sh
```

Do not mark release verification complete when a gate is skipped or
interrupted. Either rerun the gate successfully or record an explicit
release-owner waiver with the measured evidence that justifies it.

## Package Release Workflow

The package workflow is manually dispatched in two phases so the expensive
multi-architecture Docker builds can run before the public release window.

After release metadata is updated and merged, create and push the intended
release tag:

```bash
git tag vX.Y.Z
git push origin vX.Y.Z
```

Run the `Prepare Or Publish Packages` GitHub Actions workflow with:

```text
mode: prepare
tag: vX.Y.Z
```

The prepare phase validates the tag and metadata, builds PostgreSQL 14 through
18 Docker images, pushes prepared GHCR tags, verifies those prepared images, and
uploads the PGXN source archive as a workflow artifact. Prepared Docker tags use
the release tag plus a `-prepared` suffix, for example:

```text
ghcr.io/<owner>/pggraph:pg17-vX.Y.Z-prepared
```

After prepare passes, create the GitHub Release for the same tag. Then dispatch
the workflow again with:

```text
mode: publish
tag: vX.Y.Z
```

The publish phase verifies the GitHub Release exists, checks that the prepared
tag matches the prepared commit-sha tag, promotes prepared Docker manifests to
the public release tags, publishes PGXN, attaches the PGXN archive to the GitHub
Release, and verifies the final GHCR tags. It must not rebuild the Docker images.

## Fuzzing

Fuzz targets cover:

| Target | Boundary |
|---|---|
| `load_graph_file` | Artifact loader |
| `node_ref_string` | Node-ref parser |
| `traverse_options` | Traversal option parser |
| `sync_properties` | Sync JSON property parser |
| `structured_filter` | Structured filter shape validation |
| `gql_parser` | GQL parser frontend |
| `cypher_parser` | openCypher compatibility parser frontend |
| `load_projection_manifest` | Durable projection manifest loader |
| `load_projection_segment` | Durable projection segment loader |

Compile smoke:

```bash
cd graph/fuzz
cargo fuzz build
```

Projection release seed corpus:

```bash
cd graph
./fuzz/run_projection_seed_corpora.sh
```

## Benchmarking

See [Benchmarking](./benchmarking) for Criterion benchmark usage, synthetic
release-gate timing, and SQL-facing benchmark requirements.

## Documentation Checks

Rust public API docs are part of the contract. Before calling documentation
complete:

```bash
cd graph
cargo doc --features pg17 --no-deps
cargo test --doc --features pg17
```

If doctests require PostgreSQL, mark them `no_run` or keep examples in MDX user
docs instead of rustdoc.
