power loss

frontend/deterministic-tests/README.md

@@ -1,118 +1,134 @@
-# Deterministic Tests
+# Sync test harnesses (API v4)
 
-Scripted multi-client (with an in-memory filesystem) sync tests that run against a real server. Each test defines a sequence of file operations, sync/server controls, and assertions to exercise a specific conflict or edge case.
-
-Complements the fuzz-based E2E tests (`test-client`): fuzz tests discover bugs through random operations; deterministic tests pin down exact reproduction sequences for known scenarios.
-
-## How it works
-
-Each test is a `TestDefinition`: a client count and an ordered list of steps. The test name is derived from the registry key (which matches the file name). The `TestRunner` spins up N `DeterministicAgent` instances (each wrapping a real `SyncClient` with an `InMemoryFileSystem`) pointed at a shared vault on the server, then executes steps one by one.
-
-Tests that don't pause the server share a single server process (vault-name isolation). Tests that use `pause-server`/`resume-server` (SIGSTOP/SIGCONT) each get a dedicated server, since SIGSTOP freezes the entire process.
-
-The runner executes two sequential phases: regular tests on the shared server, then pause-server tests on dedicated servers. Within each phase tests run in parallel up to a concurrency limit.
-
-## Step types
-
-Clients always start with syncing disabled.
-
-**File operations** (per-client, fire-and-forget — sync is enqueued but not awaited):
-
-- `create`, `update`, `rename`, `delete`
-- `rename-next-write` — arm a deferred rename that fires the next time the given path is written. Lets a test race a user-rename against an in-flight remote create that's about to land at the same path.
-
-**Sync control:**
-
-- `sync` — wait for a specific client or all clients to finish pending operations
-- `barrier` — retry until all clients converge to identical file state (60s timeout)
-- `enable-sync` / `disable-sync` — simulate going online/offline
-- `reset` — reset a client's tracked sync state (keeps disk files); equivalent to a forced re-handshake on next enable
-- `sleep` — wall-clock pause; use sparingly, prefer `barrier` / `sync`
-
-**WebSocket control** (per-client):
-
-- `pause-websocket` / `resume-websocket` — buffer/release WebSocket messages for a specific client
-
-**Server control:**
-
-- `pause-server` / `resume-server` — SIGSTOP/SIGCONT the server process
-- `resume-server-until-history-then-pause` — resume the server, wait until a specific client observes a matching history entry (`CREATE`/`UPDATE`/`DELETE` for a path), then re-pause. Used to land exactly one operation across the wire.
-
-**Fault injection** (per-client):
-
-- `drop-next-create-response` — arm a one-shot interceptor that lets the next `POST /documents` reach the server (commit happens) but throws `SyncResetError` before the client sees the response, simulating connection loss after server commit.
-- `wait-for-dropped-create-response` — wait until the armed drop has fired.
-
-**Assertions:**
-
-- `assert-consistent` — all clients have identical files; optionally takes a custom `verify(state: AssertableState)` callback
-
-## Running
+Run the complete harness check from **any working directory**:
 
 ```sh
-# Build server first
-cd sync-server && cargo build --release && cd -
-
-# Run all tests
-cd frontend && npm run build -w sync-client && npm run test -w deterministic-tests
-
-# Filter by name
-npm run test -w deterministic-tests -- --filter=rename
-
-# Control parallelism (default: number of CPU cores)
-npm run test -w deterministic-tests -- -j 4
+E2E_WORKERS=2 /path/to/vault-link/scripts/e2e.sh
 ```
 
-## Adding a test
+The `E2E_WORKERS` variable is the number of fuzz workers / simultaneous scripted tests.
+The runner builds the server and the three in-scope frontend packages, then runs
+the harness self-tests, recovery regressions, protocol/real-filesystem tests,
+scripted scenarios, and seeded fuzz workers. It does not build the Obsidian plugin
+or CLI application. Install frontend dependencies with `npm ci` first. Node's
+built-in WebSocket, Rust, and a C compiler (`cc`, for test-only exclusive rename
+and OS-lock primitives on macOS/Linux) are required.
 
-1. Create `src/tests/my-scenario.test.ts`:
+CI installs those dependencies in `frontend` before invoking the runner. It sets
+`E2E_ARTIFACTS=logs/e2e` and uploads that same directory even when a suite fails.
+The runner owns all test servers; no separate server or legacy cleanup script is
+needed in the workflow.
 
-```typescript
-import type { TestDefinition } from "../test-definition";
+Each server gets a fresh temporary database, working directory, and port. Only
+owned processes are terminated. There is no `pkill`, fixed database deletion,
+or shared server PID file. Failing tests and teardown errors give nonzero exits;
+the remaining suites still run. Logs, concrete fuzz traces, results, and failure
+snapshots are retained in the printed artifact directory. Snapshots are captured
+before teardown so cleanup failures retain the original evidence.
 
-export const myScenarioTest: TestDefinition = {
-  description:
-    "Client 0 creates A.md offline. After syncing, both clients should have the file.",
-  clients: 2,
-  steps: [
-    { type: "create", client: 0, path: "A.md", content: "hello" },
-    { type: "enable-sync", client: 0 },
-    { type: "enable-sync", client: 1 },
-    { type: "barrier" },
-    {
-      type: "assert-consistent",
-      verify: (s) => {
-        s.assertFileCount(1).assertContent("A.md", "hello");
-      }
-    }
-  ]
-};
+Environment options:
+
+- `E2E_WORKERS=1`: number of fuzz workers / simultaneous scripted tests (max 32).
+- `E2E_SEED=1`: first uint32 seed; workers use successive seeds.
+- `E2E_ITERATIONS=25`: generated conflict scenarios per worker.
+- `E2E_TIMEOUT_SECONDS=900`: timeout for each build/suite/worker command.
+- `E2E_ARTIFACTS=/absolute/path`: optional output directory; defaults to `mkdtemp`.
+
+## Running individual layers
+
+From `frontend`:
+
+```sh
+# Rebuild protocol peers and run scripted tests (name filter optional).
+npm run test --workspace deterministic-tests -- --filter v4- --concurrency 2
+
+# Adapter/oracle/lifecycle self-tests and strict recovery regressions.
+npm run test:harness
+
+# After rebuilding: real worker death, server SIGKILL/restart, CAS/WS ordering,
+# and host filesystem contract checks.
+npm run test:protocol
+
+npm run build --workspace sync-client --workspace test-client
+node test-client/dist/cli.js --seed 42 --iterations 25
+node test-client/dist/cli.js --replay /path/to/trace-42.json
 ```
 
-The `verify` callback receives an `AssertableState` object with chainable assertion methods:
+`test:harness` includes strict regressions for **engine** bugs.
+A failing default run is not an expected-failure test or a harness skip. See
+[`../test-support/README.md`](../test-support/README.md) for their scope.
 
-```typescript
-s.assertFileCount(n);                       // exact file count
-s.assertFileExists("path");                 // file must exist
-s.assertFileNotExists("path");              // file must not exist
-s.assertContent("path", "expected");        // exact content match
-s.assertContains("path", "a", "b");         // all substrings present in file
-s.assertContainsAny("path", "a", "b");      // at least one substring present
-s.assertAnyFileContains("text");            // substring present in some file
-s.assertNoFileContains("text");             // substring absent from every file
-s.assertSubstringCount("path", "x", 3);     // substring appears exactly N times
-s.assertContentInAtMostOneFile("text");     // no duplicate content
-s.ifFileExists("path", (s) => { /* … */ }); // conditional block
-s.getContent("path");                       // raw content (or "" if missing)
-```
+## What the scripted runner checks
 
-2. Register it in `src/test-registry.ts`:
+Real `SyncClient` instances use the shared v4 adapter in `../test-support`.
+External editor operations and engine filesystem operations are separate: engine
+writes never generate self-notifications. Editor identity notifications continue
+while transport is disabled. Delayed notifications can be explicitly queued and
+flushed. Snapshots and persisted state are deep copies, never shared references.
 
-```typescript
-import { myScenarioTest } from "./tests/my-scenario.test";
+Scripted and seeded clients explicitly set `syncIntervalMs: 0`, with assertions
+that it stays disabled. Barriers observe convergence; they do not wake the engine.
+Network retry timers remain enabled. Separate fake-clock tests in
+`test-support/polling.test.ts` exercise optional periodic reconciliation.
 
-const TESTS = {
-  // ...
-  "my-scenario": myScenarioTest
-};
-```
+A barrier checks byte-for-byte replica equality **and** the server's canonical
+UUID/path map, file bytes, independently computed base/observed hashes, content
+and manifest CAS parents, event watermark,
+contiguous event log, and absence of pending requests/application journals.
+Internal recovery files are excluded from the user-visible file list, but are
+present in failure disk images. Scenario assertions separately check intended
+content, document identity, or preservation markers; equality alone is not an
+adequate data-loss oracle.
+
+Clients start disabled. `enable-sync` waits for the initial sync attempt/retries,
+except while the server is intentionally paused: that attempt is tracked so a
+later script step can resume the server. Thus sequential enables really establish
+a winner. To test _independent creates_, first bootstrap both clients empty, then
+disable them and create files. Initial adoption of pre-existing files during
+bootstrap is a different policy from two already-initialized clients creating
+distinct UUIDs at one path.
+
+## Controls and assertions
+
+- `create`, `update`, `create-bytes`, `rename`, `delete`: external editor actions.
+  Editor rename may explicitly replace its target; engine rename never may.
+- `sync`, `barrier`, `enable-sync`, `disable-sync`.
+- `reset`: restart transports; **not** a crash and not a database reset.
+- `pause-websocket`, `resume-websocket`: buffer only WS delivery. HTTP catchup
+  remains active; this is deliberately not called "disconnect".
+- `pause-observation`, `wait-for-observation`, `resume-observation`: explicitly
+  gate HTTP catchup as well as WS, including replies already in flight. The
+  checkpoint must be exercised and released, and held reads respect cancellation.
+- `delay-notifications`, `flush-notifications`: delay editor notifications without
+  disabling preservation assertions.
+- `drop-response`: `{client, kind: "create" | "content" | "manifest", point:
+"before" | "after"}`. After faults fire only on an `Accepted` CAS response.
+- `wait-for-response-drop`: explicit checkpoint; every armed fault must fire.
+  Every interrupted request must be retried with an identical request ID, target
+  and body before a different CAS is sent. Fault kinds match exactly.
+- `rename-next-write`: inject an editor rename after the engine's durable install
+  rename. The historical name is retained for fixtures; v4 never writes in place.
+- `pause-server`, `resume-server`: SIGSTOP/SIGCONT, not a crash.
+- `crash-server`, `restart-server`: SIGKILL and restart with the same database/port.
+  Server-control scenarios have dedicated processes.
+- `remember-identity`, `assert-identity`, `assert-files`, `assert-markers`.
+- `assert-documents`: a complete expected set of `{key, path, content, conflict?}`
+  records. Content is text or a byte array. Keys bind UUIDs on the first successful
+  check and must retain them on later checks; new keys cannot reuse deleted UUIDs.
+  `conflict: true` resolves the UUID-derived conflict name for `path`.
+- `assert-consistent` can provide `verify(state)`: exact text/bytes, UUID lookups,
+  `assertIdentity`, and `conflictPath(original)` for UUID-derived conflict names.
+
+Legacy timing scenarios are retained. Prefer explicit fault checkpoints and
+barriers for new tests. Only known coherent-scan races and actually injected
+transient network errors may retry; they still must converge before the deadline.
+Other background errors, unhandled rejections, and cleanup failures fail the run.
+
+## Reproducibility limits
+
+Seeds and JSON traces reproduce harness choices and logical actions. They do not
+control the OS scheduler, server thread scheduling, or production-generated UUIDs.
+The recovery sweep enumerates adapter/save boundaries; worker termination proves
+abrupt runtime loss. Neither is a proof covering every real filesystem or hardware
+power-loss behavior. Run multiple seeds, repeat timing-sensitive cases, and keep
+the scenario's independent oracle when reducing a failing trace.

frontend/deterministic-tests/package.json

@@ -8,7 +8,7 @@
     "scripts": {
         "dev": "webpack watch --mode development",
         "build": "webpack --mode production",
-        "test": "npm run build && node dist/cli.js"
+        "test": "cargo build --release --manifest-path ../../sync-server/Cargo.toml && npm run build --workspace sync-client && npm run build && node dist/cli.js"
     },
     "devDependencies": {
         "commander": "^14.0.2",

frontend/deterministic-tests/src/cli.ts

@@ -9,10 +9,11 @@ import { runWithConcurrency } from "./run-with-concurrency";
 import { TOKEN, SERVER_BINARY_PATH, CONFIG_PATH } from "./consts";
 import * as path from "node:path";
 import * as fs from "node:fs";
-import { debugging, Logger } from "sync-client";
+import * as os from "node:os";
+import { Logger } from "sync-client";
 
 const logger = new Logger();
-debugging.logToConsole(logger, { useColors: true });
+logger.onLogEmitted.add((line) => console.log(line.message));
 
 process.on("unhandledRejection", (reason) => {
     logger.error(`Unhandled Rejection: ${reason}`);
@@ -32,6 +33,8 @@ function testUsesPauseServer(test: TestDefinition): boolean {
         (step) =>
             step.type === "pause-server" ||
             step.type === "resume-server" ||
+            step.type === "crash-server" ||
+            step.type === "restart-server" ||
             step.type === "resume-server-until-history-then-pause"
     );
 }
@@ -42,7 +45,7 @@ function testUsesPauseServer(test: TestDefinition): boolean {
  */
 function findProjectRoot(): string {
     let dir = path.dirname(__filename);
-    const {root} = path.parse(dir);
+    const { root } = path.parse(dir);
     while (dir !== root) {
         if (
             fs.existsSync(path.join(dir, "sync-server")) &&
@@ -114,11 +117,7 @@ async function runDedicatedServerTest(
         }
         return { name, result };
     } finally {
-        try {
-            await server.stop();
-        } catch {
-            // best-effort cleanup
-        }
+        await server.stop();
         serverManager.untrack(server);
     }
 }
@@ -138,6 +137,11 @@ async function main(): Promise<void> {
     }
 
     const { filter, concurrency } = parseArgs(process.argv);
+    const artifacts =
+        process.env.E2E_ARTIFACTS ??
+        fs.mkdtempSync(path.join(os.tmpdir(), "vault-link-deterministic-"));
+    fs.mkdirSync(artifacts, { recursive: true });
+    logger.info(`Artifacts: ${artifacts}`);
 
     const testsToRun: [string, TestDefinition][] = [];
     for (const [key, test] of Object.entries(TESTS)) {
@@ -193,13 +197,7 @@ async function main(): Promise<void> {
 
             allResults.push(...results);
         } finally {
-            try {
-                await sharedServer.stop();
-            } catch (error) {
-                logger.warn(
-                    `Error stopping shared server: ${error instanceof Error ? error.message : String(error)}`
-                );
-            }
+            await sharedServer.stop();
             serverManager.untrack(sharedServer);
         }
     }
@@ -221,6 +219,10 @@ async function main(): Promise<void> {
 
     const passed = allResults.filter((r) => r.result.success);
     const failed = allResults.filter((r) => !r.result.success);
+    fs.writeFileSync(
+        path.join(artifacts, "deterministic-results.json"),
+        JSON.stringify(allResults, null, 2)
+    );
 
     logger.info(
         `\n--- Results: ${passed.length}/${allResults.length} passed ---`

frontend/deterministic-tests/src/crash-worker.ts

@@ -0,0 +1,93 @@
+import { parentPort, workerData } from "node:worker_threads";
+import assert from "node:assert/strict";
+import { SyncClient } from "sync-client";
+import {
+    MemoryDisk,
+    MemoryPersistence,
+    type DiskImage,
+    type StoredClient
+} from "../../test-support/storage";
+import {
+    classifyRequest,
+    type RequestRecord,
+    type RequestKind
+} from "../../test-support/network";
+
+export interface CrashWorkerInput {
+    disk: DiskImage;
+    stored: StoredClient;
+    stopAt?: RequestKind;
+    editAfterSync?: { path: string; content: string };
+}
+const input = workerData as CrashWorkerInput;
+const disk = new MemoryDisk();
+disk.restore(input.disk);
+const persistence = new MemoryPersistence(input.stored);
+const requests: RequestRecord[] = [];
+let editedBase: { documentId: string; vaultUpdateId: number } | undefined;
+const snapshot = () => ({
+    disk: disk.image(),
+    stored: persistence.snapshot(),
+    requests,
+    editedBase
+});
+
+async function main() {
+    const client = await SyncClient.create({
+        fs: disk.session(),
+        persistence,
+        fetch: async (url, init) => {
+            const record = classifyRequest(url, init);
+            if (record) requests.push(record);
+            const response = await fetch(url, init);
+            if (record && response.ok && input.stopAt === record.kind) {
+                const receipt = (await response.clone().json()) as {
+                    type: string;
+                };
+                if (receipt.type === "Accepted") {
+                    parentPort!.postMessage({
+                        type: "checkpoint",
+                        ...snapshot()
+                    });
+                    // Parent terminates this worker. No destroy(), no state flush,
+                    // no finally block, and no old runtime survives the restart.
+                    await new Promise<never>(() => {});
+                }
+            }
+            return response;
+        }
+    });
+    assert.equal(
+        client.getSettings().syncIntervalMs,
+        0,
+        "Crash recovery must preserve the disabled polling setting"
+    );
+    await client.start();
+    await client.waitUntilFinished();
+    if (input.editAfterSync) {
+        const state = persistence.snapshot().database!;
+        const documentId = Object.keys(state.local!).find(
+            (id) => state.local![id] === input.editAfterSync!.path
+        );
+        assert(documentId, "Edit must target an established document");
+        editedBase = {
+            documentId,
+            vaultUpdateId: state.documents![documentId].base!.vaultUpdateId
+        };
+        await disk.userWrite(
+            input.editAfterSync.path,
+            new TextEncoder().encode(input.editAfterSync.content)
+        );
+        await client.syncLocallyUpdatedFile({
+            relativePath: input.editAfterSync.path
+        });
+        await client.waitUntilFinished();
+    }
+    await client.destroy();
+    parentPort!.postMessage({ type: "finished", ...snapshot() });
+    parentPort!.close();
+}
+main().catch((error: unknown) => {
+    parentPort!.postMessage({ type: "error", error: String(error) });
+    process.exit(1);
+});

frontend/deterministic-tests/src/deterministic-agent.ts

@@ -1,483 +1,340 @@
-import type {
-    HistoryEntry,
-    StoredDatabase,
-    SyncSettings,
-    RelativePath,
-    TextWithCursors
-} from "sync-client";
 import {
     SyncClient,
-    SyncResetError,
-    debugging,
     LogLevel,
-    utils
+    type HistoryEntry,
+    type SyncSettings
 } from "sync-client";
+import { MemoryDisk, MemoryPersistence } from "../../test-support/storage";
+import { NetworkFaults, type RequestKind } from "../../test-support/network";
 import { assert } from "./utils/assert";
 import { sleep } from "./utils/sleep";
 import { withTimeout } from "./utils/with-timeout";
-import {
-    IS_SYNC_ENABLED_BY_DEFAULT,
-    WAIT_TIMEOUT_MS,
-    WEBSOCKET_CONNECT_TIMEOUT_MS,
-    WEBSOCKET_POLL_INTERVAL_MS
-} from "./consts";
+import { WAIT_TIMEOUT_MS, WEBSOCKET_CONNECT_TIMEOUT_MS } from "./consts";
 import { ManagedWebSocketFactory } from "./managed-websocket";
 
-export class DeterministicAgent extends debugging.InMemoryFileSystem {
-    public readonly clientId: number;
-    private readonly logger: (msg: string) => void;
-    private client!: SyncClient;
-    private data: Partial<{
-        settings: Partial<SyncSettings>;
-        database: Partial<StoredDatabase>;
-    }> = {};
-    private isSyncEnabled = IS_SYNC_ENABLED_BY_DEFAULT;
-    private readonly syncErrors: Error[] = [];
-    private readonly pendingSyncOperations = new Set<Promise<void>>();
+/** User actions live here; the engine only receives disk.session(). */
+export class DeterministicAgent {
+    public readonly disk = new MemoryDisk();
+    public readonly persistence: MemoryPersistence;
+    public readonly network = new NetworkFaults();
+    private client?: SyncClient;
+    private readonly errors: Error[] = [];
+    private readonly pending = new Set<Promise<void>>();
     private readonly wsFactory = new ManagedWebSocketFactory();
-    private nextWriteRename:
-        | {
-            oldPath: RelativePath;
-            newPath: RelativePath;
-        }
-        | undefined;
-    private nextCreateResponseDrop:
-        | {
-            dropped: Promise<void>;
-            resolveDropped: () => void;
-        }
-        | undefined;
+    private nextWriteRename?: { oldPath: string; newPath: string };
+    private delayedNotifications = false;
+    private disposed = false;
+    private readonly notifications: (() => Promise<void>)[] = [];
 
     public constructor(
-        clientId: number,
-        initialSettings: Partial<SyncSettings>,
-        logger: (msg: string) => void
+        public readonly clientId: number,
+        settings: Partial<SyncSettings>,
+        private readonly logger: (msg: string) => void
     ) {
-        super();
-        this.clientId = clientId;
-        this.logger = logger;
-        this.data.settings = { ...initialSettings };
+        this.persistence = new MemoryPersistence({
+            settings: {
+                networkRetryIntervalMs: 25,
+                webSocketRetryIntervalMs: 50,
+                requestTimeoutMs: 5_000,
+                ...settings,
+                // Scripted and seeded scenarios must exercise notification delivery.
+                syncIntervalMs: 0
+            }
+        });
+        this.disk.boundary = async (label) => {
+            const rename = this.nextWriteRename;
+            // v4 installs through an exclusive rename, never write-in-place.
+            if (
+                rename &&
+                label.startsWith("durable:rename:") &&
+                label.endsWith(`->${rename.oldPath}`)
+            ) {
+                this.nextWriteRename = undefined;
+                await this.rename(rename.oldPath, rename.newPath);
+            }
+        };
     }
 
     public async init(
-        fetchImplementation: typeof globalThis.fetch
+        fetchImplementation: typeof globalThis.fetch = fetch
     ): Promise<void> {
         this.client = await SyncClient.create({
-            fs: this,
-            persistence: {
-                load: async () => this.data,
-                save: async (data) => void (this.data = data)
-            },
-            fetch: this.wrapFetch(fetchImplementation),
+            fs: this.disk.session(),
+            persistence: this.persistence,
+            fetch: this.network.wrap(fetchImplementation),
             webSocket: this.wsFactory.constructorFn
         });
-
-        this.client.logger.onLogEmitted.add((line) => {
-            const prefix = `[Client ${this.clientId}]`;
-            switch (line.level) {
-                case LogLevel.ERROR:
-                    this.logger(`${prefix} ERROR: ${line.message}`);
-                    break;
-                case LogLevel.WARNING:
-                    this.logger(`${prefix} WARN: ${line.message}`);
-                    break;
-                case LogLevel.INFO:
-                    this.logger(`${prefix} INFO: ${line.message}`);
-                    break;
-                case LogLevel.DEBUG:
-                    this.logger(`${prefix} DEBUG: ${line.message}`);
-                    break;
-            }
-        });
-
-        await this.client.start();
-
-        const connectionCheck = await this.client.checkConnection();
         assert(
-            connectionCheck.isSuccessful,
+            this.client.getSettings().syncIntervalMs === 0,
+            "Correctness scenarios must run with periodic polling disabled"
+        );
+        this.client.logger.onLogEmitted.add((line) => {
+            this.logger(
+                `[Client ${this.clientId}] ${line.level}: ${line.message}`
+            );
+            if (this.disposed && line.level === LogLevel.ERROR)
+                throw new Error(
+                    `Client ${this.clientId} error after cleanup: ${line.message}`
+                );
+            if (
+                line.level === LogLevel.ERROR &&
+                !this.isRetryableFailure(line.message)
+            )
+                this.errors.push(new Error(line.message));
+        });
+        await this.client.start();
+        assert(
+            (await this.client.checkConnection()).isSuccessful,
             `Client ${this.clientId} connection check failed`
         );
+    }
 
-        if (this.isSyncEnabled) {
-            await this.waitForWebSocket();
+    public database() {
+        return this.persistence.snapshot().database;
+    }
+    public files(): Map<string, Uint8Array> {
+        return this.disk.userFiles();
+    }
+    public async listFilesRecursively(): Promise<string[]> {
+        return [...this.files().keys()].sort();
+    }
+    public async getFileContent(path: string): Promise<string> {
+        const bytes = this.files().get(path);
+        assert(bytes !== undefined, `Missing file: ${path}`);
+        return new TextDecoder("utf-8", { ignoreBOM: true }).decode(bytes);
+    }
+    public async read(path: string): Promise<Uint8Array> {
+        const snapshot = await this.disk.readSnapshot(path);
+        assert(snapshot !== undefined, `Missing file: ${path}`);
+        return snapshot.content;
+    }
+    public async write(path: string, content: Uint8Array): Promise<void> {
+        const created = !(await this.disk.exists(path));
+        await this.disk.userWrite(path, content);
+        this.notify(() =>
+            created
+                ? this.client!.syncLocallyCreatedFile(path)
+                : this.client!.syncLocallyUpdatedFile({ relativePath: path })
+        );
+    }
+    public async rename(oldPath: string, relativePath: string): Promise<void> {
+        await this.disk.userRename(oldPath, relativePath);
+        this.notify(() =>
+            this.client!.syncLocallyUpdatedFile({ oldPath, relativePath })
+        );
+    }
+    public async delete(path: string): Promise<void> {
+        await this.disk.userDelete(path);
+        this.notify(() => this.client!.syncLocallyDeletedFile(path));
+    }
+    public delayNotifications(): void {
+        this.delayedNotifications = true;
+    }
+    public flushNotifications(): void {
+        this.delayedNotifications = false;
+        for (const notify of this.notifications.splice(0)) this.enqueue(notify);
+    }
+    private notify(operation: () => Promise<void>): void {
+        // Offline means transport disabled, not that editor identity events vanish.
+        if (this.delayedNotifications) this.notifications.push(operation);
+        else this.enqueue(operation);
+    }
+    private enqueue(operation: () => Promise<void>): void {
+        const promise = operation().catch((error: unknown) => {
+            if (!this.isRetryableFailure(String(error)))
+                this.errors.push(
+                    error instanceof Error ? error : new Error(String(error))
+                );
+        });
+        this.pending.add(promise);
+        void promise.then(() => this.pending.delete(promise));
+    }
+    private async drain(): Promise<void> {
+        while (this.pending.size) await Promise.all([...this.pending]);
+    }
+    private isRetryableFailure(message: string): boolean {
+        // A coherent scan rejecting an editor race is a required adapter
+        // behavior. It must subsequently converge; never blanket-ignore errors.
+        return (
+            this.network.isExpectedFailure(message) ||
+            /^(Sync paused for retry: )?Error: File changed during scan: .+$/.test(
+                message
+            )
+        );
+    }
+    private assertHealthy(): void {
+        if (this.errors.length) {
+            const errors = this.errors.splice(0);
+            throw new AggregateError(
+                errors,
+                `Client ${this.clientId}: ${errors.map(String).join("; ")}`
+            );
         }
     }
-
-    public pauseWebSocket(): void {
-        this.log("Pausing WebSocket message delivery");
-        this.wsFactory.pause();
-    }
-
-    public resumeWebSocket(): void {
-        this.log("Resuming WebSocket message delivery");
-        this.wsFactory.resume();
-    }
-
-    public dropNextCreateResponse(): void {
+    public async waitForSync(): Promise<void> {
         assert(
-            this.nextCreateResponseDrop === undefined,
-            `Client ${this.clientId} already has a create response drop armed`
-        );
-        let resolveDropped!: () => void;
-        const dropped = new Promise<void>((resolve) => {
-            resolveDropped = resolve;
-        });
-        this.nextCreateResponseDrop = {
-            dropped,
-            resolveDropped
-        };
-        this.log("Armed next create response drop");
-    }
-
-    public async waitForDroppedCreateResponse(): Promise<void> {
-        assert(
-            this.nextCreateResponseDrop !== undefined,
-            `Client ${this.clientId} has no create response drop armed`
+            this.client!.getSettings().syncIntervalMs === 0,
+            "Periodic polling was enabled during a correctness scenario"
         );
         await withTimeout(
-            this.nextCreateResponseDrop.dropped,
+            (async () => {
+                await this.drain();
+                const deadline = Date.now() + WAIT_TIMEOUT_MS;
+                for (;;) {
+                    try {
+                        await this.client!.waitUntilFinished();
+                        break;
+                    } catch (error) {
+                        if (
+                            !this.isRetryableFailure(String(error)) ||
+                            Date.now() >= deadline
+                        )
+                            throw error;
+                        await sleep(30);
+                    }
+                }
+                this.assertHealthy();
+            })(),
             WAIT_TIMEOUT_MS,
-            `Client ${this.clientId} timed out waiting for create response drop`
+            `Client ${this.clientId} sync timeout`
         );
-        this.log("Create response was dropped after server commit");
     }
-
+    public async disableSync(): Promise<void> {
+        await this.drain();
+        await this.client!.setSetting("isSyncEnabled", false);
+        this.assertHealthy();
+    }
+    public async enableSync(wait = true): Promise<void> {
+        // setSetting waits for a sync attempt in v4. Track it, but don't block
+        // the script from resuming a paused server or releasing a checkpoint.
+        this.enqueue(() => this.client!.setSetting("isSyncEnabled", true));
+        const deadline = Date.now() + WEBSOCKET_CONNECT_TIMEOUT_MS;
+        while (
+            !this.persistence.snapshot().settings?.isSyncEnabled &&
+            Date.now() < deadline
+        )
+            await sleep(1);
+        assert(
+            !!this.persistence.snapshot().settings?.isSyncEnabled,
+            "Enable did not persist"
+        );
+        if (wait) await this.waitForSync();
+    }
+    public async reset(): Promise<void> {
+        await this.drain();
+        await this.client!.reset();
+    }
+    public pauseWebSocket(): void {
+        this.wsFactory.pause();
+    }
+    public pauseObservation(): void {
+        this.pauseWebSocket();
+        this.network.pauseObservation();
+    }
+    public resumeObservation(): void {
+        this.network.resumeObservation();
+        this.resumeWebSocket();
+    }
+    public async waitForObservation(): Promise<void> {
+        await withTimeout(
+            this.network.waitForObservation(),
+            WAIT_TIMEOUT_MS,
+            "HTTP observation checkpoint never reached"
+        );
+    }
+    public resumeWebSocket(): void {
+        this.wsFactory.resume();
+    }
+    public dropNextCreateResponse(): void {
+        this.network.arm("create");
+    }
+    public dropNextResponse(
+        kind: RequestKind,
+        point: "before" | "after" = "after"
+    ): void {
+        this.network.arm(kind, point);
+    }
+    public async waitForDroppedCreateResponse(): Promise<void> {
+        await withTimeout(
+            this.network.wait(),
+            WAIT_TIMEOUT_MS,
+            "Armed response drop never fired"
+        );
+    }
+    public renameNextWrite(oldPath: string, newPath: string): void {
+        assert(
+            !this.nextWriteRename,
+            "Previous install/rename hook never fired"
+        );
+        this.nextWriteRename = { oldPath, newPath };
+    }
     public async waitForHistoryEntry(
         matches: (entry: HistoryEntry) => boolean,
         onMatch?: (entry: HistoryEntry) => void
     ): Promise<void> {
-        const existing = this.client.getHistoryEntries().find(matches);
-        if (existing !== undefined) {
-            onMatch?.(existing);
-            return;
-        }
-
-        await withTimeout(
-            new Promise<void>((resolve) => {
-                const unsubscribe = this.client.onSyncHistoryUpdated.add(() => {
-                    const entry = this.client
-                        .getHistoryEntries()
-                        .find(matches);
-                    if (entry === undefined) {
-                        return;
-                    }
-
-                    unsubscribe();
-                    onMatch?.(entry);
-                    resolve();
-                });
-            }),
-            WAIT_TIMEOUT_MS,
-            `Client ${this.clientId} timed out waiting for history entry`
-        );
-    }
-
-    public async waitForSync(): Promise<void> {
-        this.log("Waiting for sync to complete...");
-        // Drain agent-level sync operations first. These are the fire-and-forget
-        // promises from enqueueSync() that call into the SyncClient's methods.
-        // Without this, waitUntilFinished() might return before the SyncClient
-        // has even been told about the operation.
-        await this.drainPendingSyncOperations();
-        await withTimeout(
-            this.client.waitUntilFinished(),
-            WAIT_TIMEOUT_MS,
-            `Client ${this.clientId} waitForSync timed out after ${WAIT_TIMEOUT_MS}ms`
-        );
-        if (this.syncErrors.length > 0) {
-            const errors = this.syncErrors.splice(0);
-            throw new Error(
-                `Client ${this.clientId} had ${errors.length} sync error(s):\n${errors.map((e) => e.message).join("\n")}`
-            );
-        }
-        this.log("Sync complete");
-    }
-
-    public async reset(): Promise<void> {
-        this.log("Resetting client (clears tracked state, keeps disk files)");
-        await this.drainPendingSyncOperations();
-        await this.client.reset();
-        if (this.isSyncEnabled) {
-            await this.waitForWebSocket();
-        }
-    }
-
-    public async disableSync(): Promise<void> {
-        this.log("Disabling sync");
-        // Drain pending enqueued operations before disabling so the SyncClient
-        // knows about all operations that were enqueued while sync was enabled.
-        await this.drainPendingSyncOperations();
-        await this.client.setSetting("isSyncEnabled", false);
-        this.isSyncEnabled = false;
-        // Wait for in-flight operations to drain. Disabling sync triggers
-        // a reset, which aborts in-flight fetches with SyncResetError.
+        let unsubscribe = () => {};
         try {
             await withTimeout(
-                this.client.waitUntilFinished(),
+                new Promise<void>((resolve) => {
+                    const check = () => {
+                        const entry =
+                            this.client!.getHistoryEntries().find(matches);
+                        if (entry) {
+                            onMatch?.(entry);
+                            resolve();
+                        }
+                    };
+                    unsubscribe = this.client!.onSyncHistoryUpdated.add(check);
+                    check();
+                }),
                 WAIT_TIMEOUT_MS,
-                `Client ${this.clientId} disableSync drain timed out`
+                "History checkpoint not reached"
             );
-        } catch (error) {
-            if (error instanceof Error && error.name === "SyncResetError") {
-                this.log("Disable sync drain interrupted by reset (expected)");
-            } else {
-                throw error;
-            }
+        } finally {
+            unsubscribe();
         }
     }
-
-    public async enableSync(): Promise<void> {
-        this.log("Enabling sync");
-        await this.client.setSetting("isSyncEnabled", true);
-        this.isSyncEnabled = true;
-        await this.waitForWebSocket();
-    }
-
-    public async getFileContent(path: string): Promise<string> {
-        const bytes = await this.read(path);
-        return new TextDecoder().decode(bytes);
-    }
-
-    public renameNextWrite(oldPath: RelativePath, newPath: RelativePath): void {
-        assert(
-            this.nextWriteRename === undefined,
-            `Client ${this.clientId} already has a next-write rename armed`
-        );
-        this.nextWriteRename = { oldPath, newPath };
-        this.log(`Armed next write rename: ${oldPath} -> ${newPath}`);
-    }
-
     public async cleanup(): Promise<void> {
-        this.log("Cleaning up...");
-        // Guard against uninitialized client (init() failed partway).
-        // The class field uses `!:` so TS thinks this is always defined,
-        // but at runtime it can be undefined when init() throws partway.
-        const maybeClient = this.client as SyncClient | undefined;
-        if (maybeClient === undefined) {
-            this.log("Client not initialized, nothing to clean up");
-            return;
+        if (!this.client) return;
+        const errors: unknown[] = [];
+        this.resumeObservation();
+        this.flushNotifications();
+        try {
+            await this.waitForSync();
+        } catch (error) {
+            errors.push(error);
+        }
+        try {
+            this.network.assertConsumed();
+            assert(!this.nextWriteRename, "Install/rename hook never fired");
+        } catch (error) {
+            errors.push(error);
         }
         try {
-            await this.drainPendingSyncOperations();
             await withTimeout(
-                this.client.waitUntilFinished(),
+                this.client.destroy(),
                 WAIT_TIMEOUT_MS,
-                `Client ${this.clientId} cleanup waitUntilFinished timed out`
+                "Destroy timed out"
+            );
+            await withTimeout(
+                this.wsFactory.finish(),
+                WAIT_TIMEOUT_MS,
+                "WebSocket callbacks did not finish during cleanup"
             );
         } catch (error) {
-            if (error instanceof Error && error.name === "SyncResetError") {
-                this.log(`Cleanup interrupted by reset (expected): ${error}`);
-            } else {
-                this.log(`Cleanup waitUntilFinished failed: ${error}`);
-            }
+            errors.push(error);
         }
-        // Surface any background sync errors that arrived after the last
-        // waitForSync (e.g. between the final assert-consistent and here).
-        // Without this, regressions that fault the engine during the very
-        // last step of a test would be silently swallowed.
-        const pendingErrors = this.syncErrors.splice(0);
-        await this.client.destroy();
-        this.log("Cleanup complete");
-        if (pendingErrors.length > 0) {
-            throw new Error(
-                `Client ${this.clientId} had ${pendingErrors.length} background sync error(s) during cleanup:\n${pendingErrors.map((e) => e.message).join("\n")}`
-            );
-        }
-    }
-
-    public override async read(path: RelativePath): Promise<Uint8Array> {
-        await Promise.resolve();
-        return super.read(path);
-    }
-
-    public override async write(
-        path: RelativePath,
-        content: Uint8Array
-    ): Promise<void> {
-        await Promise.resolve();
-        const isNew = !this.files.has(path);
-        await super.write(path, content);
-
-        if (this.isSyncEnabled && isNew) {
-            this.enqueueSync(async () => {
-                this.client.syncLocallyCreatedFile(path);
-            });
-        }
-
-        const { nextWriteRename } = this;
-        if (
-            nextWriteRename !== undefined &&
-            nextWriteRename.oldPath === path
-        ) {
-            this.nextWriteRename = undefined;
-            await super.rename(
-                nextWriteRename.oldPath,
-                nextWriteRename.newPath
-            );
-            if (this.isSyncEnabled) {
-                this.enqueueSync(async () => {
-                    this.client.syncLocallyUpdatedFile({
-                        oldPath: nextWriteRename.oldPath,
-                        relativePath: nextWriteRename.newPath
-                    });
-                });
-            }
-            // The rename consumed `path`. Skip the post-update enqueue below
-            // — it would send a syncLocallyUpdatedFile for a path that no
-            // longer exists.
-            return;
-        }
-
-        if (!this.isSyncEnabled) {
-            return;
-        }
-
-        if (!isNew) {
-            this.enqueueSync(async () => {
-                this.client.syncLocallyUpdatedFile({ relativePath: path });
-            });
-        }
-    }
-
-    public override async atomicUpdateText(
-        path: RelativePath,
-        updater: (current: TextWithCursors) => TextWithCursors
-    ): Promise<string> {
-        const result = await super.atomicUpdateText(path, updater);
-        if (this.isSyncEnabled) {
-            this.enqueueSync(async () => {
-                this.client.syncLocallyUpdatedFile({ relativePath: path });
-            });
-        }
-        return result;
-    }
-
-    public override async delete(path: RelativePath): Promise<void> {
-        await super.delete(path);
-        if (this.isSyncEnabled) {
-            this.enqueueSync(async () => {
-                this.client.syncLocallyDeletedFile(path);
-            });
-        }
-    }
-
-    public override async rename(
-        oldPath: RelativePath,
-        newPath: RelativePath
-    ): Promise<void> {
-        await super.rename(oldPath, newPath);
-        if (this.isSyncEnabled) {
-            this.enqueueSync(async () => {
-                this.client.syncLocallyUpdatedFile({
-                    oldPath,
-                    relativePath: newPath
-                });
-            });
-        }
-    }
-
-    private async waitForWebSocket(): Promise<void> {
-        const deadline = Date.now() + WEBSOCKET_CONNECT_TIMEOUT_MS;
-        while (!this.client.isWebSocketConnected && Date.now() < deadline) {
-            await sleep(WEBSOCKET_POLL_INTERVAL_MS);
-        }
-        assert(
-            this.client.isWebSocketConnected,
-            `Client ${this.clientId} WebSocket failed to connect within ${WEBSOCKET_CONNECT_TIMEOUT_MS}ms`
-        );
-    }
-
-    /**
-     * Wait until all agent-level enqueued sync operations have completed.
-     * Uses a loop because completing one operation can trigger new enqueues.
-     */
-    private async drainPendingSyncOperations(): Promise<void> {
-        while (this.pendingSyncOperations.size > 0) {
-            await utils.awaitAll([...this.pendingSyncOperations]);
-        }
-    }
-
-    private enqueueSync(operation: () => Promise<void>): void {
-        const promise = this.executeSyncOperation(operation).catch(
-            (error: unknown) => {
-                const err =
-                    error instanceof Error ? error : new Error(String(error));
-                this.log(`Background sync failed: ${err.message}`);
-                this.syncErrors.push(err);
-            }
-        );
-        this.pendingSyncOperations.add(promise);
-        void promise.finally(() => {
-            this.pendingSyncOperations.delete(promise);
-        });
-    }
-
-    private async executeSyncOperation(
-        operation: () => Promise<void>
-    ): Promise<void> {
+        this.client = undefined;
+        this.disposed = true;
         try {
-            await operation();
+            this.assertHealthy();
         } catch (error) {
-            if (error instanceof Error && error.name === "SyncResetError") {
-                this.log(`Sync operation interrupted by reset: ${error}`);
-                return;
-            }
-            if (
-                error instanceof Error &&
-                error.message.includes("has been destroyed")
-            ) {
-                this.log(`Sync operation interrupted by destroy: ${error}`);
-                return;
-            }
-
-            throw error;
+            errors.push(error);
         }
-    }
-
-    private log(message: string): void {
-        this.logger(`[Client ${this.clientId}] ${message}`);
-    }
-
-    private wrapFetch(
-        fetchImplementation: typeof globalThis.fetch
-    ): typeof globalThis.fetch {
-        return async (input, init) => {
-            const response = await fetchImplementation(input, init);
-            const drop = this.nextCreateResponseDrop;
-            if (
-                drop !== undefined &&
-                DeterministicAgent.isCreateDocumentRequest(input, init)
-            ) {
-                this.nextCreateResponseDrop = undefined;
-                try {
-                    await response.body?.cancel();
-                } catch {
-                    // Best-effort — body may already be consumed/closed.
-                }
-                drop.resolveDropped();
-                throw new SyncResetError();
-            }
-            return response;
-        };
-    }
-
-    private static isCreateDocumentRequest(
-        input: RequestInfo | URL,
-        init: RequestInit | undefined
-    ): boolean {
-        const method =
-            init?.method ??
-            (typeof Request !== "undefined" && input instanceof Request
-                ? input.method
-                : "GET");
-        if (method.toUpperCase() !== "POST") {
-            return false;
-        }
-
-        const url =
-            input instanceof URL
-                ? input
-                : new URL(typeof input === "string" ? input : input.url);
-        return /\/documents\/?$/.test(url.pathname);
+        if (errors.length)
+            throw new AggregateError(
+                errors,
+                `Client ${this.clientId} cleanup failed: ${errors.map(String).join("; ")}`
+            );
     }
 }

frontend/deterministic-tests/src/managed-websocket.ts

@@ -209,8 +209,13 @@ export class ManagedWebSocketFactory {
     // WS before the agent reconnects would silently see the new socket
     // start un-paused and miss the messages it meant to buffer.
     private currentlyPaused = false;
+    private disposed = false;
 
     public get constructorFn(): typeof globalThis.WebSocket {
+        const ensureActive = () => {
+            if (this.disposed)
+                throw new Error("WebSocket created after harness cleanup");
+        };
         const trackInstance = (instance: ManagedWebSocket): void => {
             this.instances.push(instance);
             if (this.currentlyPaused) {
@@ -222,6 +227,7 @@ export class ManagedWebSocketFactory {
                 url: string | URL,
                 protocols?: string | string[]
             ) {
+                ensureActive();
                 super(url, protocols);
                 trackInstance(this);
             }
@@ -229,6 +235,21 @@ export class ManagedWebSocketFactory {
         return TrackedManagedWebSocket;
     }
 
+    public async finish(): Promise<void> {
+        this.disposed = true;
+        await Promise.all(
+            this.instances.map((socket) =>
+                socket.readyState === WebSocket.CLOSED
+                    ? Promise.resolve()
+                    : new Promise<void>((resolve) =>
+                          socket.addEventListener("close", () => resolve(), {
+                              once: true
+                          })
+                      )
+            )
+        );
+    }
+
     public pause(): void {
         this.currentlyPaused = true;
         for (const ws of this.instances) {

frontend/deterministic-tests/src/parse-args.ts

@@ -7,8 +7,8 @@ export interface CliArgs {
 }
 
 function parsePositiveInt(value: string): number {
-    const n = parseInt(value, 10);
-    if (isNaN(n) || n <= 0) {
+    const n = Number(value);
+    if (!/^\d+$/.test(value) || !Number.isSafeInteger(n) || n <= 0) {
         throw new InvalidArgumentError("must be a positive integer");
     }
     return n;

frontend/deterministic-tests/src/server-control.ts

@@ -20,10 +20,13 @@ export class ServerControl {
     private _port: number | undefined;
     private tempDir: string | undefined;
     private _isPaused = false;
+    private preserveOnStop = false;
+    private expectedExit = false;
+    private unexpectedExit?: Error;
 
     public constructor(serverPath: string, configPath: string, logger: Logger) {
-        this.serverPath = serverPath;
-        this.baseConfigPath = configPath;
+        this.serverPath = path.resolve(serverPath);
+        this.baseConfigPath = path.resolve(configPath);
         this.logger = logger;
     }
 
@@ -38,6 +41,15 @@ export class ServerControl {
         return `http://localhost:${this.port}`;
     }
 
+    public get databaseDirectory(): string {
+        if (!this.tempDir) throw new Error("Server has no database directory");
+        return path.join(this.tempDir, "databases");
+    }
+
+    public get isPaused(): boolean {
+        return this._isPaused;
+    }
+
     public async start(): Promise<void> {
         if (this.process !== null) {
             throw new Error("Server is already running");
@@ -66,14 +78,25 @@ export class ServerControl {
     }
 
     private async startOnce(): Promise<void> {
-        const reservation = await findFreePort();
-        this._port = reservation.port;
+        const reservation =
+            this._port === undefined ? await findFreePort() : undefined;
+        this._port ??= reservation!.port;
         const tmpBase = os.tmpdir();
-        this.tempDir = fs.mkdtempSync(path.join(tmpBase, "vault-link-test-"));
-        const tempConfigPath = path.join(this.tempDir, "config.yml");
-        const dbDir = path.join(this.tempDir, "databases");
-
-        this.writeConfigFile(tempConfigPath, dbDir);
+        let tempConfigPath: string;
+        try {
+            this.tempDir ??= fs.mkdtempSync(
+                path.join(tmpBase, "vault-link-test-")
+            );
+            tempConfigPath = path.join(this.tempDir, "config.yml");
+            this.writeConfigFile(
+                tempConfigPath,
+                path.join(this.tempDir, "databases")
+            );
+        } catch (error) {
+            reservation?.release();
+            this.cleanupTempDir();
+            throw error;
+        }
 
         this.logger.info(
             `Starting server: ${this.serverPath} (port ${this._port})`
@@ -81,12 +104,15 @@ export class ServerControl {
 
         // Release the port reservation right before spawning to minimize
         // the TOCTOU window between port discovery and server binding.
-        reservation.release();
+        reservation?.release();
 
         this.process = spawn(this.serverPath, [tempConfigPath], {
+            cwd: this.tempDir,
             stdio: ["ignore", "pipe", "pipe"],
             detached: false
         });
+        this.expectedExit = false;
+        this.unexpectedExit = undefined;
 
         this.process.stdout?.on("data", (data: Buffer) => {
             this.logger.info(`[SERVER] ${data.toString().trim()}`);
@@ -102,6 +128,10 @@ export class ServerControl {
 
         const currentProcess = this.process;
         currentProcess.on("exit", (code, signal) => {
+            if (!this.expectedExit)
+                this.unexpectedExit = new Error(
+                    `Owned server exited unexpectedly (code ${code}, signal ${signal})`
+                );
             this.logger.info(
                 `Server exited with code ${code}, signal ${signal}`
             );
@@ -139,7 +169,9 @@ export class ServerControl {
                 );
             }
             try {
-                const response = await fetch(pingUrl);
+                const response = await fetch(pingUrl, {
+                    signal: AbortSignal.timeout(1_000)
+                });
                 if (response.ok) {
                     this.logger.info("[SERVER] Ready");
                     return;
@@ -194,11 +226,13 @@ export class ServerControl {
     public async stop(): Promise<void> {
         const proc = this.process;
         if (proc?.pid === undefined) {
-            this.cleanupTempDir();
+            if (!this.preserveOnStop) this.cleanupTempDir();
+            if (this.unexpectedExit) throw this.unexpectedExit;
             return;
         }
+        this.expectedExit = true;
 
-        // Resume if paused — a SIGSTOP'd process ignores SIGKILL
+        // SIGKILL terminates even a stopped process; SIGCONT is harmless here.
         if (this._isPaused) {
             try {
                 process.kill(proc.pid, "SIGCONT");
@@ -212,7 +246,7 @@ export class ServerControl {
 
         // Set up a promise that resolves when the process actually exits.
         const exitPromise = new Promise<void>((resolve) => {
-            if (proc.exitCode !== null) {
+            if (proc.exitCode !== null || proc.signalCode !== null) {
                 resolve();
                 return;
             }
@@ -229,11 +263,49 @@ export class ServerControl {
 
         // Wait for the process to actually exit before cleaning up,
         // with a 5s safety timeout to avoid hanging forever.
-        await Promise.race([exitPromise, sleep(STOP_TIMEOUT_MS)]);
+        let timer: ReturnType<typeof setTimeout> | undefined;
+        try {
+            await Promise.race([
+                exitPromise,
+                new Promise<never>((_, reject) => {
+                    timer = setTimeout(
+                        () =>
+                            reject(
+                                new Error(
+                                    "Owned server did not exit after SIGKILL"
+                                )
+                            ),
+                        STOP_TIMEOUT_MS
+                    );
+                })
+            ]);
+        } finally {
+            clearTimeout(timer);
+        }
 
         this.process = null;
         this._isPaused = false;
-        this.cleanupTempDir();
+        if (!this.preserveOnStop) this.cleanupTempDir();
+        if (this.unexpectedExit) throw this.unexpectedExit;
+    }
+
+    /** A real process death: retain the exact database and port for restart. */
+    public async crash(): Promise<void> {
+        if (!this.isRunning()) throw new Error("Cannot crash a stopped server");
+        this.preserveOnStop = true;
+        try {
+            await this.stop();
+        } finally {
+            this.preserveOnStop = false;
+        }
+    }
+
+    public async restart(): Promise<void> {
+        if (!this.tempDir || this.isRunning())
+            throw new Error(
+                "Restart requires a crashed server with retained storage"
+            );
+        await this.startOnce();
     }
 
     public isRunning(): boolean {
@@ -273,23 +345,27 @@ export class ServerControl {
         // one `databases_directory_path:` (under `server:` and `database:`
         // respectively)
         const baseConfig = fs.readFileSync(this.baseConfigPath, "utf-8");
+        for (const key of ["port", "databases_directory_path"]) {
+            if (
+                (baseConfig.match(new RegExp(`^  ${key}:`, "gm")) ?? [])
+                    .length !== 1
+            )
+                throw new Error(`Expected exactly one config key: ${key}`);
+        }
         const config = baseConfig
             .replace(/^\s*port:\s*\d+/m, `  port: ${this._port}`)
             .replace(
                 /^\s*databases_directory_path:\s*.+/m,
-                `  databases_directory_path: ${dbDir}`
+                `  databases_directory_path: ${JSON.stringify(dbDir)}`
             );
         fs.writeFileSync(destPath, config);
     }
 
     private cleanupTempDir(): void {
         if (this.tempDir !== undefined) {
-            try {
-                fs.rmSync(this.tempDir, { recursive: true, force: true });
-            } catch {
-                // Best-effort cleanup
-            }
+            fs.rmSync(this.tempDir, { recursive: true, force: true });
             this.tempDir = undefined;
+            this._port = undefined;
         }
     }
 }

frontend/deterministic-tests/src/server-manager.ts

@@ -59,9 +59,8 @@ export class ServerManager {
         // Last-resort synchronous cleanup. Runs even when the process is
         // exiting via process.exit() from unhandledRejection /
         // uncaughtException — paths where async stopAll() cannot complete.
-        // SIGSTOP'd servers MUST receive SIGCONT before SIGKILL or the
-        // kernel keeps them as zombies holding the test's tmpdir, and the
-        // next CI run can't reuse the port.
+        // Kill only the server processes owned by this runner. SIGKILL also
+        // terminates a SIGSTOP'd process; forceKillSync's SIGCONT is harmless.
         process.on("exit", () => {
             for (const server of this.activeServers) {
                 server.forceKillSync();

frontend/deterministic-tests/src/test-definition.ts

@@ -3,6 +3,18 @@ import type { AssertableState } from "./utils/assertable-state";
 export interface ClientState {
     files: Map<string, string>;
     clientFiles: Map<string, string>[];
+    manifests: Record<string, string>[];
+    canonical: Record<string, string>;
+    bytes: Map<string, Uint8Array>;
+}
+
+/** A scenario's independent, complete expected state. Keys survive moves;
+ * a new key must receive a previously unseen UUID, including after deletion. */
+export interface ExpectedDocument {
+    key: string;
+    path: string;
+    conflict?: boolean;
+    content: string | number[];
 }
 
 export type TestStep =
@@ -31,10 +43,41 @@ export type TestStep =
     | { type: "assert-consistent"; verify?: (state: AssertableState) => void }
     | { type: "pause-websocket"; client: number }
     | { type: "resume-websocket"; client: number }
+    | {
+          type:
+              | "pause-observation"
+              | "resume-observation"
+              | "wait-for-observation";
+          client: number;
+      }
     | { type: "drop-next-create-response"; client: number }
     | { type: "wait-for-dropped-create-response"; client: number }
     | { type: "sleep"; ms: number }
-    | { type: "reset"; client: number };
+    | { type: "reset"; client: number }
+    | { type: "create-bytes"; client: number; path: string; bytes: number[] }
+    | {
+          type: "drop-response";
+          client: number;
+          kind: "create" | "content" | "manifest";
+          point?: "before" | "after";
+      }
+    | { type: "wait-for-response-drop"; client: number }
+    | { type: "delay-notifications" | "flush-notifications"; client: number }
+    | {
+          type: "remember-identity" | "assert-identity";
+          path: string;
+          key: string;
+      }
+    | {
+          type: "assert-files";
+          expected: Record<string, string>;
+          absent?: string[];
+          count?: number;
+      }
+    | { type: "assert-markers"; markers: string[]; removed?: string[] }
+    | { type: "assert-documents"; expected: ExpectedDocument[] }
+    | { type: "crash-server" }
+    | { type: "restart-server" };
 
 export interface TestDefinition {
     description?: string;
@@ -46,4 +89,5 @@ export interface TestResult {
     success: boolean;
     error?: string;
     duration?: number;
+    diagnostics?: unknown;
 }

frontend/deterministic-tests/src/test-registry.ts

@@ -1,4 +1,5 @@
 import type { TestDefinition } from "./test-definition";
+import { V4_TESTS } from "./v4-regressions";
 import { renameCreateConflictTest } from "./tests/rename-create-conflict.test";
 import { renameChainTest } from "./tests/rename-chain.test";
 import { renameUpdateConflictTest } from "./tests/rename-update-conflict.test";
@@ -106,6 +107,7 @@ import { remoteQuickWriteRenameBeforeRecordTest } from "./tests/remote-quick-wri
 import { selfMergePendingRenameAliasesSecondCreateTest } from "./tests/self-merge-pending-rename-aliases-second-create.test";
 
 export const TESTS: Partial<Record<string, TestDefinition>> = {
+    ...V4_TESTS,
     "rename-create-conflict": renameCreateConflictTest,
     "rename-chain": renameChainTest,
     "rename-update-conflict": renameUpdateConflictTest,

frontend/deterministic-tests/src/test-runner.ts

@@ -13,6 +13,8 @@ import {
     IS_SYNC_ENABLED_BY_DEFAULT
 } from "./consts";
 import { randomUUID } from "node:crypto";
+import { assertCanonical } from "../../test-support/canonical";
+import { ContentLedger } from "../../test-support/oracles";
 
 export class TestRunner {
     private agents: DeterministicAgent[] = [];
@@ -20,6 +22,8 @@ export class TestRunner {
     private readonly token: string;
     private readonly remoteUri: string;
     private readonly logger: Logger;
+    private vaultName = "";
+    private readonly identities = new Map<string, string>();
 
     public constructor(
         serverControl: ServerControl,
@@ -44,6 +48,8 @@ export class TestRunner {
         }
         this.logger.info(`Clients: ${test.clients}`);
         this.logger.info(`Steps: ${test.steps.length}`);
+        this.identities.clear();
+        let diagnostics: TestResult["diagnostics"];
 
         try {
             assert(
@@ -61,6 +67,11 @@ export class TestRunner {
                 await this.executeStep(step);
             }
 
+            assert(
+                this.serverControl.isRunning(),
+                "Server died before test completion"
+            );
+            diagnostics = this.captureDiagnostics();
             await this.cleanup();
 
             const duration = Date.now() - startTime;
@@ -71,25 +82,46 @@ export class TestRunner {
                 duration
             };
         } catch (error) {
+            diagnostics ??= this.captureDiagnostics();
             const duration = Date.now() - startTime;
             const errorMessage =
                 error instanceof Error ? error.message : String(error);
             this.logger.info(`\n✗ Test failed: ${name}`);
             this.logger.info(`Error: ${errorMessage}`);
 
-            await this.cleanup();
+            try {
+                await this.cleanup();
+            } catch (cleanupError) {
+                return {
+                    success: false,
+                    error: `${errorMessage}\nCleanup: ${String(cleanupError)}`,
+                    duration,
+                    diagnostics
+                };
+            }
 
             return {
                 success: false,
                 error: errorMessage,
+                diagnostics,
                 duration
             };
         }
     }
 
+    private captureDiagnostics(): unknown {
+        return this.agents.map((agent) => ({
+            client: agent.clientId,
+            database: agent.database(),
+            disk: agent.disk.image(),
+            requests: structuredClone(agent.network.requests)
+        }));
+    }
+
     private async initializeAgents(count: number): Promise<void> {
         assert(count > 0, `Client count must be positive, got ${count}`);
         const vaultName = `test-${randomUUID()}`;
+        this.vaultName = vaultName;
         this.logger.info(
             `Initializing ${count} agents with vault: ${vaultName}`
         );
@@ -129,6 +161,87 @@ export class TestRunner {
 
     private async executeStep(step: TestStep): Promise<void> {
         switch (step.type) {
+            case "pause-observation":
+                this.getAgent(step.client).pauseObservation();
+                break;
+            case "resume-observation":
+                this.getAgent(step.client).resumeObservation();
+                break;
+            case "wait-for-observation":
+                await this.getAgent(step.client).waitForObservation();
+                break;
+            case "create-bytes":
+                await this.getAgent(step.client).write(
+                    step.path,
+                    new Uint8Array(step.bytes)
+                );
+                break;
+            case "drop-response":
+                this.getAgent(step.client).dropNextResponse(
+                    step.kind,
+                    step.point
+                );
+                break;
+            case "wait-for-response-drop":
+                await this.getAgent(step.client).waitForDroppedCreateResponse();
+                break;
+            case "delay-notifications":
+                this.getAgent(step.client).delayNotifications();
+                break;
+            case "flush-notifications":
+                this.getAgent(step.client).flushNotifications();
+                break;
+            case "remember-identity":
+            case "assert-identity":
+                await this.assertConsistent((state) => {
+                    if (step.type === "remember-identity")
+                        this.identities.set(
+                            step.key,
+                            state.documentId(step.path)
+                        );
+                    else {
+                        const id = this.identities.get(step.key);
+                        assert(
+                            id !== undefined,
+                            `Identity ${step.key} was never recorded`
+                        );
+                        state.assertIdentity(step.path, id);
+                    }
+                });
+                break;
+            case "assert-files":
+                await this.assertConsistent((state) => {
+                    for (const [path, value] of Object.entries(step.expected))
+                        state.assertContent(path, value);
+                    for (const path of step.absent ?? [])
+                        state.assertFileNotExists(path);
+                    if (step.count !== undefined)
+                        state.assertFileCount(step.count);
+                });
+                break;
+            case "assert-documents":
+                await this.assertConsistent((state) =>
+                    state.assertDocuments(step.expected, this.identities)
+                );
+                break;
+            case "assert-markers": {
+                const ledger = new ContentLedger();
+                for (const marker of step.markers) ledger.add(marker);
+                for (const marker of step.removed ?? [])
+                    ledger.removedBy(
+                        marker,
+                        "trace records explicit deletion/overwrite"
+                    );
+                for (const agent of this.agents)
+                    ledger.assertPreserved(agent.files());
+                break;
+            }
+            case "crash-server":
+                await this.serverControl.crash();
+                break;
+            case "restart-server":
+                await this.serverControl.restart();
+                break;
             case "create":
             case "update":
                 await this.getAgent(step.client).write(
@@ -170,7 +283,9 @@ export class TestRunner {
                 break;
 
             case "enable-sync":
-                await this.getAgent(step.client).enableSync();
+                await this.getAgent(step.client).enableSync(
+                    !this.serverControl.isPaused
+                );
                 break;
 
             case "pause-server":
@@ -190,7 +305,9 @@ export class TestRunner {
                     (entry) =>
                         entry.details.type === step.syncType &&
                         entry.details.relativePath === step.path,
-                    () => { this.serverControl.pause(); }
+                    () => {
+                        this.serverControl.pause();
+                    }
                 );
                 this.serverControl.resume();
                 await historySeen;
@@ -305,6 +422,11 @@ export class TestRunner {
 
         // Snapshot all agents' file states upfront to minimize the window
         // where background sync could mutate state between reads.
+        const canonical = await assertCanonical(
+            this.agents,
+            `${this.remoteUri}/vaults/${encodeURIComponent(this.vaultName)}`,
+            this.token
+        );
         const clientFiles: Map<string, string>[] = [];
         for (const agent of this.agents) {
             const sortedFiles = (await agent.listFilesRecursively()).sort();
@@ -360,7 +482,12 @@ export class TestRunner {
                 verify(
                     new AssertableState({
                         files: clientFiles[0],
-                        clientFiles
+                        clientFiles,
+                        manifests: this.agents.map(
+                            (agent) => agent.database()!.local!
+                        ),
+                        canonical: canonical.fileManifest.entries,
+                        bytes: this.agents[0].files()
                     })
                 );
             } catch (error) {
@@ -384,10 +511,12 @@ export class TestRunner {
         }
 
         this.logger.info("\nCleaning up agents...");
-        for (const agent of this.agents) {
+        const errors: unknown[] = [];
+        for (const agent of this.agents.splice(0)) {
             try {
                 await agent.cleanup();
             } catch (error) {
+                errors.push(error);
                 this.logger.warn(
                     `Agent cleanup error: ${error instanceof Error ? error.message : String(error)}`
                 );
@@ -395,5 +524,10 @@ export class TestRunner {
         }
         this.agents = [];
         this.logger.info("Cleanup complete");
+        if (errors.length)
+            throw new AggregateError(
+                errors,
+                `Cleanup failed: ${errors.map(String).join("; ")}`
+            );
     }
 }

frontend/deterministic-tests/src/utils/assertable-state.ts

@@ -1,12 +1,103 @@
-import type { ClientState } from "../test-definition";
+import assert from "node:assert/strict";
+import type { ClientState, ExpectedDocument } from "../test-definition";
 
 export class AssertableState {
     public readonly files: Map<string, string>;
     public readonly clientFiles: Map<string, string>[];
+    public readonly manifests: Record<string, string>[];
+    public readonly canonical: Record<string, string>;
+    public readonly bytes: Map<string, Uint8Array>;
 
     public constructor(state: ClientState) {
         this.files = state.files;
         this.clientFiles = state.clientFiles;
+        this.manifests = state.manifests;
+        this.canonical = state.canonical;
+        this.bytes = state.bytes;
+    }
+
+    public documentId(path: string): string {
+        const id = Object.keys(this.canonical).find(
+            (id) => this.canonical[id] === path
+        );
+        if (!id) throw new Error(`No document identity for ${path}`);
+        return id;
+    }
+
+    public assertDocuments(
+        expected: readonly ExpectedDocument[],
+        identities: Map<string, string>
+    ): this {
+        this.assertFileCount(expected.length);
+        assert.equal(
+            new Set(expected.map((doc) => doc.key)).size,
+            expected.length,
+            "Duplicate expected identity key"
+        );
+        const next = new Map(identities);
+        const paths = new Set<string>();
+        for (const doc of expected) {
+            const path = doc.conflict ? this.conflictPath(doc.path) : doc.path;
+            assert(
+                !paths.has(path),
+                `Two expected documents resolved to ${path}`
+            );
+            paths.add(path);
+            this.assertBytes(
+                path,
+                typeof doc.content === "string"
+                    ? new TextEncoder().encode(doc.content)
+                    : new Uint8Array(doc.content)
+            );
+            const id = this.documentId(path);
+            const recorded = next.get(doc.key);
+            if (recorded !== undefined) this.assertIdentity(path, recorded);
+            else {
+                assert(
+                    ![...next.values()].includes(id),
+                    `New document ${doc.key} reused an existing or deleted UUID`
+                );
+                next.set(doc.key, id);
+            }
+        }
+        // A failed check must not establish a partial identity oracle.
+        for (const [key, id] of next) identities.set(key, id);
+        return this;
+    }
+
+    public conflictPath(original: string): string {
+        const dot = original.lastIndexOf(".");
+        const split =
+            dot > original.lastIndexOf("/") + 1 ? dot : original.length;
+        const matches = Object.entries(this.canonical).filter(
+            ([id, path]) =>
+                path ===
+                `${original.slice(0, split)} (conflict ${id})${original.slice(split)}`
+        );
+        if (matches.length !== 1)
+            throw new Error(
+                `Expected one UUID-derived conflict path for ${original}, found ${matches.length}`
+            );
+        return matches[0][1];
+    }
+
+    public assertIdentity(path: string, id: string): this {
+        if (this.documentId(path) !== id)
+            throw new Error(
+                `Document identity changed at ${path}: expected ${id}, got ${this.documentId(path)}`
+            );
+        return this;
+    }
+
+    public assertBytes(path: string, expected: Uint8Array): this {
+        const actual = this.bytes.get(path);
+        if (
+            !actual ||
+            actual.length !== expected.length ||
+            actual.some((byte, i) => byte !== expected[i])
+        )
+            throw new Error(`Byte mismatch at ${path}`);
+        return this;
     }
 
     public assertFileCount(expected: number): this {

frontend/deterministic-tests/src/v4-regressions.ts

@@ -0,0 +1,253 @@
+import type { TestDefinition, TestStep } from "./test-definition";
+
+const online: TestStep[] = [
+    { type: "enable-sync", client: 0 },
+    { type: "enable-sync", client: 1 },
+    { type: "barrier" }
+];
+const offline: TestStep[] = [
+    { type: "disable-sync", client: 0 },
+    { type: "disable-sync", client: 1 }
+];
+export const V4_TESTS: Record<string, TestDefinition> = {
+    "v4-text-byte-and-component-boundaries": {
+        clients: 2,
+        steps: [
+            ...online,
+            { type: "create", client: 0, path: "empty.md", content: "" },
+            {
+                type: "create",
+                client: 0,
+                path: "bom.md",
+                content: "\ufefffirst\r\nlast"
+            },
+            {
+                type: "create",
+                client: 0,
+                path: `${"a".repeat(252)}.md`,
+                content: "255 ASCII bytes"
+            },
+            {
+                type: "create",
+                client: 0,
+                path: `${"é".repeat(126)}.md`,
+                content: "255 UTF-8 bytes"
+            },
+            { type: "barrier" },
+            {
+                type: "assert-files",
+                count: 4,
+                expected: {
+                    "empty.md": "",
+                    "bom.md": "\ufefffirst\r\nlast",
+                    [`${"a".repeat(252)}.md`]: "255 ASCII bytes",
+                    [`${"é".repeat(126)}.md`]: "255 UTF-8 bytes"
+                }
+            }
+        ]
+    },
+    "v4-binary-bytes-and-independent-identities": {
+        clients: 2,
+        steps: [
+            ...online,
+            ...offline,
+            {
+                type: "create-bytes",
+                client: 0,
+                path: "nested/blob.bin",
+                bytes: [0, 255, 254, 128, 13, 10]
+            },
+            {
+                type: "create-bytes",
+                client: 1,
+                path: "nested/blob.bin",
+                bytes: [0, 254, 255, 129, 10, 13]
+            },
+            ...online,
+            {
+                type: "assert-consistent",
+                verify: (state) => {
+                    state
+                        .assertFileCount(2)
+                        .assertBytes(
+                            "nested/blob.bin",
+                            new Uint8Array([0, 255, 254, 128, 13, 10])
+                        );
+                    state.assertBytes(
+                        state.conflictPath("nested/blob.bin"),
+                        new Uint8Array([0, 254, 255, 129, 10, 13])
+                    );
+                }
+            }
+        ]
+    },
+    "v4-three-file-cycle-preserves-identities": {
+        clients: 2,
+        steps: [
+            ...online,
+            { type: "create", client: 0, path: "a.md", content: "A" },
+            { type: "create", client: 0, path: "b.md", content: "B" },
+            { type: "create", client: 0, path: "c.md", content: "C" },
+            { type: "barrier" },
+            { type: "remember-identity", key: "a", path: "a.md" },
+            { type: "remember-identity", key: "b", path: "b.md" },
+            { type: "remember-identity", key: "c", path: "c.md" },
+            { type: "disable-sync", client: 0 },
+            { type: "delay-notifications", client: 0 },
+            { type: "rename", client: 0, oldPath: "a.md", newPath: "temp.md" },
+            { type: "rename", client: 0, oldPath: "b.md", newPath: "a.md" },
+            { type: "rename", client: 0, oldPath: "c.md", newPath: "b.md" },
+            { type: "rename", client: 0, oldPath: "temp.md", newPath: "c.md" },
+            {
+                type: "update",
+                client: 0,
+                path: "c.md",
+                content: "A edited offline"
+            },
+            { type: "flush-notifications", client: 0 },
+            { type: "enable-sync", client: 0 },
+            { type: "barrier" },
+            { type: "assert-identity", key: "a", path: "c.md" },
+            { type: "assert-identity", key: "b", path: "a.md" },
+            { type: "assert-identity", key: "c", path: "b.md" },
+            {
+                type: "assert-files",
+                count: 3,
+                expected: {
+                    "a.md": "B",
+                    "b.md": "C",
+                    "c.md": "A edited offline"
+                },
+                absent: ["temp.md"]
+            }
+        ]
+    },
+    "v4-case-alias-and-file-directory-conflicts": {
+        clients: 2,
+        steps: [
+            ...online,
+            ...offline,
+            {
+                type: "create",
+                client: 0,
+                path: "folder/note.md",
+                content: "nested file"
+            },
+            {
+                type: "create",
+                client: 0,
+                path: "README.md",
+                content: "uppercase file"
+            },
+            {
+                type: "create",
+                client: 1,
+                path: "folder",
+                content: "blocking file"
+            },
+            {
+                type: "create",
+                client: 1,
+                path: "readme.md",
+                content: "lowercase file"
+            },
+            ...online,
+            {
+                type: "assert-consistent",
+                verify: (state) => {
+                    state
+                        .assertFileCount(4)
+                        .assertContent("folder/note.md", "nested file")
+                        .assertContent("README.md", "uppercase file");
+                    state.assertAnyFileContains(
+                        "blocking file",
+                        "lowercase file"
+                    );
+                }
+            },
+            {
+                type: "assert-markers",
+                markers: [
+                    "nested file",
+                    "blocking file",
+                    "uppercase file",
+                    "lowercase file"
+                ]
+            }
+        ]
+    },
+    "v4-server-sigkill-retains-acknowledged-state": {
+        clients: 2,
+        steps: [
+            ...online,
+            {
+                type: "create",
+                client: 0,
+                path: "a.md",
+                content: "durable bytes"
+            },
+            { type: "barrier" },
+            { type: "remember-identity", key: "a", path: "a.md" },
+            ...offline,
+            { type: "crash-server" },
+            { type: "restart-server" },
+            ...online,
+            { type: "assert-identity", key: "a", path: "a.md" },
+            {
+                type: "assert-files",
+                count: 1,
+                expected: { "a.md": "durable bytes" }
+            }
+        ]
+    }
+};
+
+for (const kind of ["create", "content", "manifest"] as const) {
+    for (const point of ["before", "after"] as const) {
+        V4_TESTS[`v4-${kind}-${point}-commit-network-loss`] = {
+            clients: 2,
+            steps: [
+                ...online,
+                ...(kind === "content"
+                    ? ([
+                          {
+                              type: "create",
+                              client: 0,
+                              path: "file.md",
+                              content: "old bytes"
+                          },
+                          { type: "barrier" },
+                          {
+                              type: "remember-identity",
+                              key: "updated-file",
+                              path: "file.md"
+                          }
+                      ] as TestStep[])
+                    : []),
+                { type: "drop-response", client: 0, kind, point },
+                {
+                    type: kind === "content" ? "update" : "create",
+                    client: 0,
+                    path: "file.md",
+                    content: "must survive retry"
+                },
+                { type: "wait-for-response-drop", client: 0 },
+                { type: "barrier" },
+                ...(kind === "content"
+                    ? [
+                          {
+                              type: "assert-identity",
+                              key: "updated-file",
+                              path: "file.md"
+                          } as TestStep
+                      ]
+                    : []),
+                {
+                    type: "assert-files",
+                    count: 1,
+                    expected: { "file.md": "must survive retry" }
+                }
+            ]
+        };
+    }
+}

frontend/deterministic-tests/tsconfig.json

@@ -2,6 +2,7 @@
     "compilerOptions": {
         "baseUrl": ".",
         "strict": true,
+        "skipLibCheck": true,
         "target": "ES2022",
         "module": "CommonJS",
         "esModuleInterop": true,

frontend/deterministic-tests/webpack.config.js

@@ -2,8 +2,9 @@ const path = require("path");
 const webpack = require("webpack");
 
 module.exports = {
-    entry: "./src/cli.ts",
+    entry: { cli: "./src/cli.ts", "crash-worker": "./src/crash-worker.ts" },
     target: "node",
+    externals: { "sync-client": "commonjs sync-client" },
     mode: "production",
     optimization: {
         minimize: false
@@ -21,7 +22,7 @@ module.exports = {
     },
     output: {
         globalObject: "this",
-        filename: "cli.js",
+        filename: "[name].js",
         path: path.resolve(__dirname, "dist")
     },
     plugins: [