mor

frontend/deterministic-tests/README.md

@@ -7,10 +7,10 @@ E2E_WORKERS=2 /path/to/vault-link/scripts/e2e.sh
 ```
 
 The `E2E_WORKERS` variable is the number of fuzz workers / simultaneous scripted tests.
-The runner builds the server and the three in-scope frontend packages, then runs
-the harness self-tests, recovery regressions, protocol/real-filesystem tests,
-scripted scenarios, and seeded fuzz workers. It does not build the Obsidian plugin
-or CLI application. Install frontend dependencies with `npm ci` first. Node's
+The runner builds the server and all frontend packages, then runs core/adapter
+unit tests, harness self-tests, recovery regressions, protocol/real-filesystem tests,
+scripted scenarios, concrete regression traces, and seeded fuzz workers.
+Install frontend dependencies with `npm ci` first. Node's
 built-in WebSocket, Rust, and a C compiler (`cc`, for test-only exclusive rename
 and OS-lock primitives on macOS/Linux) are required.
 
@@ -29,8 +29,9 @@ before teardown so cleanup failures retain the original evidence.
 Environment options:
 
 - `E2E_WORKERS=1`: number of fuzz workers / simultaneous scripted tests (max 32).
-- `E2E_SEED=1`: first uint32 seed; workers use successive seeds.
-- `E2E_ITERATIONS=25`: generated conflict scenarios per worker.
+- `E2E_SEED`: optional first uint32 seed; otherwise a fresh random seed is recorded.
+  Workers use successive seeds, with no uint32 overflow.
+- `E2E_ITERATIONS=25`: generated rounds per worker; scheduled CI uses 100.
 - `E2E_TIMEOUT_SECONDS=900`: timeout for each build/suite/worker command.
 - `E2E_ARTIFACTS=/absolute/path`: optional output directory; defaults to `mkdtemp`.
 
@@ -49,7 +50,7 @@ npm run test:harness
 # and host filesystem contract checks.
 npm run test:protocol
 
-npm run build --workspace sync-client --workspace test-client
+npm run build --workspace sync-client --workspace deterministic-tests --workspace test-client
 node test-client/dist/cli.js --seed 42 --iterations 25
 node test-client/dist/cli.js --replay /path/to/trace-42.json
 ```
@@ -78,7 +79,10 @@ contiguous event log, and absence of pending requests.
 Internal recovery files are excluded from the user-visible file list, but are
 present in failure disk images. Scenario assertions separately check intended
 content, document identity, or preservation markers; equality alone is not an
-adequate data-loss oracle.
+adequate data-loss oracle. Custom `assert-consistent` verifiers must check exact
+content for the entire file set; partial substring assertions alone fail. Harness
+self-tests verify that shared corruption, duplicate files and identity reuse are
+rejected by these oracles.
 
 Clients start disabled. `enable-sync` waits for the initial sync attempt/retries,
 except while the server is intentionally paused: that attempt is tracked so a
@@ -101,6 +105,11 @@ distinct UUIDs at one path.
   checkpoint must be exercised and released, and held reads respect cancellation.
 - `delay-notifications`, `flush-notifications`: delay editor notifications without
   disabling preservation assertions.
+- `hold-request`, `wait-for-request`, `release-request`: gate create/content/manifest
+  CAS, content downloads or event reads before sending or after receiving a reply.
+  After-response gates can require a response type. `enable-sync` with `wait: false`
+  allows the script to reach and release a gate. Missing or unreleased gates fail.
+- `assert-response`: prove a required CAS response (for example `StaleBase`) occurred.
 - `drop-response`: `{client, kind: "create" | "content" | "manifest", point:
 "before" | "after"}`. After faults fire only on an `Accepted` CAS response.
 - `wait-for-response-drop`: explicit checkpoint; every armed fault must fire.
@@ -111,7 +120,15 @@ distinct UUIDs at one path.
 - `pause-server`, `resume-server`: SIGSTOP/SIGCONT, not a crash.
 - `crash-server`, `restart-server`: SIGKILL and restart with the same database/port.
   Server-control scenarios have dedicated processes.
+- `crash-client`, `restart-client`: terminate a worker at an accepted response or
+  filesystem/persistence boundary, then start a fresh runtime from saved state.
+  Prepare the client's work while disabled. Post-crash disk edits are supported;
+  interrupted CAS requests must retain their request ID, target and payload.
+- `fail-save`, `wait-for-save-failure`: reject before or after durable metadata
+  replacement and require that the injection actually fired.
 - `remember-identity`, `assert-identity`, `assert-files`, `assert-markers`.
+- `remember-local-identity`: bind an offline create's UUID from its durable journal
+  before network reconciliation. `{{key}}` in a path resolves a bound identity.
 - `assert-documents`: a complete expected set of `{key, path, content, conflict?}`
   records. Content is text or a byte array. Keys bind UUIDs on the first successful
   check and must retain them on later checks; new keys cannot reuse deleted UUIDs.
@@ -119,8 +136,8 @@ distinct UUIDs at one path.
 - `assert-consistent` can provide `verify(state)`: exact text/bytes, UUID lookups,
   `assertIdentity`, and `conflictPath(original)` for UUID-derived conflict names.
 
-Legacy timing scenarios are retained. Prefer explicit fault checkpoints and
-barriers for new tests. Only known coherent-scan races and actually injected
+Scripted race scenarios use observable checkpoints instead of sleep steps.
+Only known coherent-scan races and actually injected
 transient network errors may retry; they still must converge before the deadline.
 Other background errors, unhandled rejections, and cleanup failures fail the run.
 
@@ -128,6 +145,10 @@ Other background errors, unhandled rejections, and cleanup failures fail the run
 
 Seeds and JSON traces reproduce harness choices and logical actions. They do not
 control the OS scheduler, server thread scheduling, or production-generated UUIDs.
+Results retain the exercised checkpoint schedule; replay requires the selected
+boundaries and responses to occur again. The seeded client automatically reduces
+failing programs while retaining complete independent expectations. See
+[`../test-client/README.md`](../test-client/README.md) for corpus and coverage details.
 The recovery sweep enumerates adapter/save boundaries; worker termination proves
 abrupt runtime loss. Neither is a proof covering every real filesystem or hardware
 power-loss behavior. Run multiple seeds, repeat timing-sensitive cases, and keep