vault-link/frontend/deterministic-tests/README.md

# Sync test harnesses (API v4)

Run the complete harness check from **any working directory**:

```sh
E2E_WORKERS=2 /path/to/vault-link/scripts/e2e.sh
```

The `E2E_WORKERS` variable is the number of fuzz workers / simultaneous scripted tests.
The runner builds the server and all frontend packages, then runs core/adapter
unit tests, harness self-tests, recovery regressions, protocol/real-filesystem tests,
scripted scenarios, concrete regression traces, and seeded fuzz workers.
Install frontend dependencies with `npm ci` first. Node's
built-in WebSocket, Rust, and a C compiler (`cc`, for test-only exclusive rename
and OS-lock primitives on macOS/Linux) are required.

CI installs those dependencies in `frontend` before invoking the runner. It sets
`E2E_ARTIFACTS=logs/e2e` and uploads that same directory even when a suite fails.
The runner owns all test servers; no separate server or legacy cleanup script is
needed in the workflow.

Each server gets a fresh temporary database, working directory, and port. Only
owned processes are terminated. There is no `pkill`, fixed database deletion,
or shared server PID file. Failing tests and teardown errors give nonzero exits;
the remaining suites still run. Logs, concrete fuzz traces, results, and failure
snapshots are retained in the printed artifact directory. Snapshots are captured
before teardown so cleanup failures retain the original evidence.

Environment options:

- `E2E_WORKERS=1`: number of fuzz workers / simultaneous scripted tests (max 32).
- `E2E_SEED`: optional first uint32 seed; otherwise a fresh random seed is recorded.
  Workers use successive seeds, with no uint32 overflow.
- `E2E_ITERATIONS=25`: generated rounds per worker; scheduled CI uses 100.
- `E2E_TIMEOUT_SECONDS=900`: timeout for each build/suite/worker command.
- `E2E_ARTIFACTS=/absolute/path`: optional output directory; defaults to `mkdtemp`.

## Running individual layers

From `frontend`:

```sh
# Rebuild protocol peers and run scripted tests (name filter optional).
npm run test --workspace deterministic-tests -- --filter v4- --concurrency 2

# Adapter/oracle/lifecycle self-tests and strict recovery regressions.
npm run test:harness

# After rebuilding: real worker death, server SIGKILL/restart, CAS/WS ordering,
# and host filesystem contract checks.
npm run test:protocol

npm run build --workspace sync-client --workspace deterministic-tests --workspace test-client
node test-client/dist/cli.js --seed 42 --iterations 25
node test-client/dist/cli.js --replay /path/to/trace-42.json
```

`test:harness` includes strict regressions for **engine** bugs.
A failing default run is not an expected-failure test or a harness skip. See
[`../test-support/README.md`](../test-support/README.md) for their scope.

## What the scripted runner checks

Real `SyncClient` instances use the shared v4 adapter in `../test-support`.
External editor operations and engine filesystem operations are separate: engine
writes never generate self-notifications. Editor identity notifications continue
while transport is disabled. Delayed notifications can be explicitly queued and
flushed. Snapshots and persisted state are deep copies, never shared references.

Scripted and seeded clients explicitly set `syncIntervalMs: 0`, with assertions
that it stays disabled. Barriers observe convergence; they do not wake the engine.
Network retry timers remain enabled. Separate fake-clock tests in
`test-support/polling.test.ts` exercise optional periodic reconciliation.

A barrier checks byte-for-byte replica equality **and** the server's canonical
UUID/path map, file bytes, independently computed base/observed hashes, content
and manifest CAS parents, event watermark,
contiguous event log, and absence of pending requests.
Internal recovery files are excluded from the user-visible file list, but are
present in failure disk images. Scenario assertions separately check intended
content, document identity, or preservation markers; equality alone is not an
adequate data-loss oracle. Custom `assert-consistent` verifiers must check exact
content for the entire file set; partial substring assertions alone fail. Harness
self-tests verify that shared corruption, duplicate files and identity reuse are
rejected by these oracles.

Clients start disabled. `enable-sync` waits for the initial sync attempt/retries,
except while the server is intentionally paused: that attempt is tracked so a
later script step can resume the server. Thus sequential enables really establish
a winner. To test _independent creates_, first bootstrap both clients empty, then
disable them and create files. Initial adoption of pre-existing files during
bootstrap is a different policy from two already-initialized clients creating
distinct UUIDs at one path.

## Controls and assertions

- `create`, `update`, `create-bytes`, `rename`, `delete`: external editor actions.
  Editor rename may explicitly replace its target; engine rename never may.
- `sync`, `barrier`, `enable-sync`, `disable-sync`.
- `reset`: restart transports; **not** a crash and not a database reset.
- `pause-websocket`, `resume-websocket`: buffer only WS delivery. HTTP catchup
  remains active; this is deliberately not called "disconnect".
- `pause-observation`, `wait-for-observation`, `resume-observation`: explicitly
  gate HTTP catchup as well as WS, including replies already in flight. The
  checkpoint must be exercised and released, and held reads respect cancellation.
- `delay-notifications`, `flush-notifications`: delay editor notifications without
  disabling preservation assertions.
- `hold-request`, `wait-for-request`, `release-request`: gate create/content/manifest
  CAS, content downloads or event reads before sending or after receiving a reply.
  After-response gates can require a response type. `enable-sync` with `wait: false`
  allows the script to reach and release a gate. Missing or unreleased gates fail.
- `assert-response`: prove a required CAS response (for example `StaleBase`) occurred.
- `drop-response`: `{client, kind: "create" | "content" | "manifest", point:
"before" | "after"}`. After faults fire only on an `Accepted` CAS response.
- `wait-for-response-drop`: explicit checkpoint; every armed fault must fire.
  Every interrupted request must be retried with an identical request ID, target
  and body before a different CAS is sent. Fault kinds match exactly.
- `rename-next-write`: inject an editor rename after the engine's content write,
  before its metadata save.
- `pause-server`, `resume-server`: SIGSTOP/SIGCONT, not a crash.
- `crash-server`, `restart-server`: SIGKILL and restart with the same database/port.
  Server-control scenarios have dedicated processes.
- `crash-client`, `restart-client`: terminate a worker at an accepted response or
  filesystem/persistence boundary, then start a fresh runtime from saved state.
  Prepare the client's work while disabled. Post-crash disk edits are supported;
  interrupted CAS requests must retain their request ID, target and payload.
- `fail-save`, `wait-for-save-failure`: reject before or after durable metadata
  replacement and require that the injection actually fired.
- `remember-identity`, `assert-identity`, `assert-files`, `assert-markers`.
- `remember-local-identity`: bind an offline create's UUID from its durable journal
  before network reconciliation. `{{key}}` in a path resolves a bound identity.
- `assert-documents`: a complete expected set of `{key, path, content, conflict?}`
  records. Content is text or a byte array. Keys bind UUIDs on the first successful
  check and must retain them on later checks; new keys cannot reuse deleted UUIDs.
  `conflict: true` resolves the UUID-derived conflict name for `path`.
- `assert-consistent` can provide `verify(state)`: exact text/bytes, UUID lookups,
  `assertIdentity`, and `conflictPath(original)` for UUID-derived conflict names.

Scripted race scenarios use observable checkpoints instead of sleep steps.
Only known coherent-scan races and actually injected
transient network errors may retry; they still must converge before the deadline.
Other background errors, unhandled rejections, and cleanup failures fail the run.

## Reproducibility limits

Seeds and JSON traces reproduce harness choices and logical actions. They do not
control the OS scheduler, server thread scheduling, or production-generated UUIDs.
Results retain the exercised checkpoint schedule; replay requires the selected
boundaries and responses to occur again. The seeded client automatically reduces
failing programs while retaining complete independent expectations. See
[`../test-client/README.md`](../test-client/README.md) for corpus and coverage details.
The recovery sweep enumerates adapter/save boundaries; worker termination proves
abrupt runtime loss. Neither is a proof covering every real filesystem or hardware
power-loss behavior. Run multiple seeds, repeat timing-sensitive cases, and keep
the scenario's independent oracle when reducing a failing trace.