diff --git a/.forgejo/workflows/e2e.yml b/.forgejo/workflows/e2e.yml index 74a17bf0..43cee8e8 100644 --- a/.forgejo/workflows/e2e.yml +++ b/.forgejo/workflows/e2e.yml @@ -17,6 +17,7 @@ env: RUSTFLAGS: "-Dwarnings" E2E_ARTIFACTS: logs/e2e E2E_WORKERS: 8 + # Each campaign records a fresh random seed; concrete corpus traces always run. jobs: build: @@ -42,6 +43,10 @@ jobs: toolchain: "1.92.0" components: clippy, rustfmt + - name: Expand scheduled fuzz campaigns + if: github.event_name == 'schedule' + run: echo "E2E_ITERATIONS=100" >> "$GITHUB_ENV" + - name: E2E tests run: scripts/e2e.sh diff --git a/frontend/deterministic-tests/README.md b/frontend/deterministic-tests/README.md index c137bc72..915f4bd1 100644 --- a/frontend/deterministic-tests/README.md +++ b/frontend/deterministic-tests/README.md @@ -7,10 +7,10 @@ E2E_WORKERS=2 /path/to/vault-link/scripts/e2e.sh ``` The `E2E_WORKERS` variable is the number of fuzz workers / simultaneous scripted tests. -The runner builds the server and the three in-scope frontend packages, then runs -the harness self-tests, recovery regressions, protocol/real-filesystem tests, -scripted scenarios, and seeded fuzz workers. It does not build the Obsidian plugin -or CLI application. Install frontend dependencies with `npm ci` first. Node's +The runner builds the server and all frontend packages, then runs core/adapter +unit tests, harness self-tests, recovery regressions, protocol/real-filesystem tests, +scripted scenarios, concrete regression traces, and seeded fuzz workers. +Install frontend dependencies with `npm ci` first. Node's built-in WebSocket, Rust, and a C compiler (`cc`, for test-only exclusive rename and OS-lock primitives on macOS/Linux) are required. @@ -29,8 +29,9 @@ before teardown so cleanup failures retain the original evidence. Environment options: - `E2E_WORKERS=1`: number of fuzz workers / simultaneous scripted tests (max 32). -- `E2E_SEED=1`: first uint32 seed; workers use successive seeds. -- `E2E_ITERATIONS=25`: generated conflict scenarios per worker. +- `E2E_SEED`: optional first uint32 seed; otherwise a fresh random seed is recorded. + Workers use successive seeds, with no uint32 overflow. +- `E2E_ITERATIONS=25`: generated rounds per worker; scheduled CI uses 100. - `E2E_TIMEOUT_SECONDS=900`: timeout for each build/suite/worker command. - `E2E_ARTIFACTS=/absolute/path`: optional output directory; defaults to `mkdtemp`. @@ -49,7 +50,7 @@ npm run test:harness # and host filesystem contract checks. npm run test:protocol -npm run build --workspace sync-client --workspace test-client +npm run build --workspace sync-client --workspace deterministic-tests --workspace test-client node test-client/dist/cli.js --seed 42 --iterations 25 node test-client/dist/cli.js --replay /path/to/trace-42.json ``` @@ -78,7 +79,10 @@ contiguous event log, and absence of pending requests. Internal recovery files are excluded from the user-visible file list, but are present in failure disk images. Scenario assertions separately check intended content, document identity, or preservation markers; equality alone is not an -adequate data-loss oracle. +adequate data-loss oracle. Custom `assert-consistent` verifiers must check exact +content for the entire file set; partial substring assertions alone fail. Harness +self-tests verify that shared corruption, duplicate files and identity reuse are +rejected by these oracles. Clients start disabled. `enable-sync` waits for the initial sync attempt/retries, except while the server is intentionally paused: that attempt is tracked so a @@ -101,6 +105,11 @@ distinct UUIDs at one path. checkpoint must be exercised and released, and held reads respect cancellation. - `delay-notifications`, `flush-notifications`: delay editor notifications without disabling preservation assertions. +- `hold-request`, `wait-for-request`, `release-request`: gate create/content/manifest + CAS, content downloads or event reads before sending or after receiving a reply. + After-response gates can require a response type. `enable-sync` with `wait: false` + allows the script to reach and release a gate. Missing or unreleased gates fail. +- `assert-response`: prove a required CAS response (for example `StaleBase`) occurred. - `drop-response`: `{client, kind: "create" | "content" | "manifest", point: "before" | "after"}`. After faults fire only on an `Accepted` CAS response. - `wait-for-response-drop`: explicit checkpoint; every armed fault must fire. @@ -111,7 +120,15 @@ distinct UUIDs at one path. - `pause-server`, `resume-server`: SIGSTOP/SIGCONT, not a crash. - `crash-server`, `restart-server`: SIGKILL and restart with the same database/port. Server-control scenarios have dedicated processes. +- `crash-client`, `restart-client`: terminate a worker at an accepted response or + filesystem/persistence boundary, then start a fresh runtime from saved state. + Prepare the client's work while disabled. Post-crash disk edits are supported; + interrupted CAS requests must retain their request ID, target and payload. +- `fail-save`, `wait-for-save-failure`: reject before or after durable metadata + replacement and require that the injection actually fired. - `remember-identity`, `assert-identity`, `assert-files`, `assert-markers`. +- `remember-local-identity`: bind an offline create's UUID from its durable journal + before network reconciliation. `{{key}}` in a path resolves a bound identity. - `assert-documents`: a complete expected set of `{key, path, content, conflict?}` records. Content is text or a byte array. Keys bind UUIDs on the first successful check and must retain them on later checks; new keys cannot reuse deleted UUIDs. @@ -119,8 +136,8 @@ distinct UUIDs at one path. - `assert-consistent` can provide `verify(state)`: exact text/bytes, UUID lookups, `assertIdentity`, and `conflictPath(original)` for UUID-derived conflict names. -Legacy timing scenarios are retained. Prefer explicit fault checkpoints and -barriers for new tests. Only known coherent-scan races and actually injected +Scripted race scenarios use observable checkpoints instead of sleep steps. +Only known coherent-scan races and actually injected transient network errors may retry; they still must converge before the deadline. Other background errors, unhandled rejections, and cleanup failures fail the run. @@ -128,6 +145,10 @@ Other background errors, unhandled rejections, and cleanup failures fail the run Seeds and JSON traces reproduce harness choices and logical actions. They do not control the OS scheduler, server thread scheduling, or production-generated UUIDs. +Results retain the exercised checkpoint schedule; replay requires the selected +boundaries and responses to occur again. The seeded client automatically reduces +failing programs while retaining complete independent expectations. See +[`../test-client/README.md`](../test-client/README.md) for corpus and coverage details. The recovery sweep enumerates adapter/save boundaries; worker termination proves abrupt runtime loss. Neither is a proof covering every real filesystem or hardware power-loss behavior. Run multiple seeds, repeat timing-sensitive cases, and keep diff --git a/frontend/deterministic-tests/src/crash-worker.ts b/frontend/deterministic-tests/src/crash-worker.ts index af3945dc..210ed36e 100644 --- a/frontend/deterministic-tests/src/crash-worker.ts +++ b/frontend/deterministic-tests/src/crash-worker.ts @@ -18,6 +18,11 @@ export interface CrashWorkerInput { stored: StoredClient; stopAt?: RequestKind; editAfterSync?: { path: string; content: string }; + stopAtBoundary?: { + type: "persistence" | "filesystem"; + label: string; + afterAccepted?: RequestKind; + }; } const input = workerData as CrashWorkerInput; const disk = new MemoryDisk(); @@ -25,13 +30,35 @@ disk.restore(input.disk); const persistence = new MemoryPersistence(input.stored); const requests: RequestRecord[] = []; let editedBase: { documentId: string; vaultUpdateId: number } | undefined; +let acceptedKind: RequestKind | undefined; +let boundary: string | undefined; const snapshot = () => ({ disk: disk.image(), stored: persistence.snapshot(), requests, - editedBase + editedBase, + boundary }); +for (const [type, target] of [ + ["filesystem", disk], + ["persistence", persistence] +] as const) { + target.boundary = async (label) => { + const stop = input.stopAtBoundary; + if ( + stop?.type === type && + stop.label === label && + (stop.afterAccepted === undefined || + acceptedKind === stop.afterAccepted) + ) { + boundary = label; + parentPort!.postMessage({ type: "checkpoint", ...snapshot() }); + await new Promise(() => {}); + } + }; +} + async function main() { const client = await SyncClient.create({ fs: disk.session(), @@ -40,11 +67,16 @@ async function main() { const record = classifyRequest(url, init); if (record) requests.push(record); const response = await fetch(url, init); - if (record && response.ok && input.stopAt === record.kind) { + if (record && response.ok) { const receipt = (await response.clone().json()) as { type: string; }; - if (receipt.type === "Accepted") { + record.responseType = receipt.type; + if (receipt.type === "Accepted") acceptedKind = record.kind; + if ( + receipt.type === "Accepted" && + input.stopAt === record.kind + ) { parentPort!.postMessage({ type: "checkpoint", ...snapshot() diff --git a/frontend/deterministic-tests/src/deterministic-agent.ts b/frontend/deterministic-tests/src/deterministic-agent.ts index 295663ce..b603e706 100644 --- a/frontend/deterministic-tests/src/deterministic-agent.ts +++ b/frontend/deterministic-tests/src/deterministic-agent.ts @@ -11,6 +11,8 @@ import { sleep } from "./utils/sleep"; import { withTimeout } from "./utils/with-timeout"; import { WAIT_TIMEOUT_MS, WEBSOCKET_CONNECT_TIMEOUT_MS } from "./consts"; import { ManagedWebSocketFactory } from "./managed-websocket"; +import { terminateAtCheckpoint } from "./worker-control"; +import type { CrashWorkerInput } from "./crash-worker"; /** User actions live here; the engine only receives disk.session(). */ export class DeterministicAgent { @@ -20,7 +22,15 @@ export class DeterministicAgent { private client?: SyncClient; private readonly errors: Error[] = []; private readonly pending = new Set>(); - private readonly wsFactory = new ManagedWebSocketFactory(); + private wsFactory = new ManagedWebSocketFactory(); + private crashed = false; + private interruptedRequest?: import("../../test-support/network").RequestRecord; + private saveFailure?: { + fired: boolean; + reached: Promise; + resolve: () => void; + message: string; + }; private nextWriteRename?: { oldPath: string; newPath: string }; private delayedNotifications = false; private disposed = false; @@ -131,6 +141,7 @@ export class DeterministicAgent { for (const notify of this.notifications.splice(0)) this.enqueue(notify); } private notify(operation: () => Promise): void { + if (this.crashed) return; // Offline means transport disabled, not that editor identity events vanish. if (this.delayedNotifications) this.notifications.push(operation); else this.enqueue(operation); @@ -145,13 +156,15 @@ export class DeterministicAgent { this.pending.add(promise); void promise.then(() => this.pending.delete(promise)); } - private async drain(): Promise { + public async drain(): Promise { while (this.pending.size) await Promise.all([...this.pending]); } private isRetryableFailure(message: string): boolean { // A coherent scan rejecting an editor race is a required adapter // behavior. It must subsequently converge; never blanket-ignore errors. return ( + (this.saveFailure?.fired === true && + message.includes(this.saveFailure.message)) || this.network.isExpectedFailure(message) || /^(Sync paused for retry: )?Error: File changed during scan: .+$/.test( message @@ -176,7 +189,7 @@ export class DeterministicAgent { (async () => { await this.drain(); const deadline = Date.now() + WAIT_TIMEOUT_MS; - for (; ;) { + for (;;) { try { await this.client!.waitUntilFinished(); break; @@ -220,6 +233,99 @@ export class DeterministicAgent { await this.drain(); await this.client!.reset(); } + public async setSettings(settings: Partial): Promise { + await this.client!.setSettings(settings); + } + public failSave(point: "before" | "durable"): void { + assert( + !this.saveFailure || this.saveFailure.fired, + "Save fault never fired" + ); + const reached = Promise.withResolvers(); + const fault = { + fired: false, + reached: reached.promise, + resolve: () => reached.resolve(), + message: `Injected ${point} persistence failure` + }; + this.saveFailure = fault; + this.persistence.boundary = (label) => { + if (label !== `${point}:save` || fault.fired) return; + fault.fired = true; + this.persistence.boundary = () => {}; + this.network.schedule.push({ type: "save-failure", detail: point }); + fault.resolve(); + throw new Error(fault.message); + }; + } + public async waitForSaveFailure(): Promise { + assert(this.saveFailure !== undefined, "Save fault not armed"); + await withTimeout( + this.saveFailure!.reached, + WAIT_TIMEOUT_MS, + "Save fault never fired" + ); + await this.drain(); + } + public async crashDuringSync( + at: Pick, + powerLoss = false + ): Promise { + assert( + !this.crashed && this.client !== undefined, + "Client already crashed" + ); + await this.drain(); + assert( + !this.client.getSettings().isSyncEnabled, + "Prepare crash work with transport disabled" + ); + this.network.assertConsumed(); + await this.client.destroy(); + await this.wsFactory.finish(); + const stored = this.persistence.snapshot(); + stored.settings = { ...stored.settings, isSyncEnabled: true }; + const snapshot = await terminateAtCheckpoint({ + disk: this.disk.image(), + stored, + ...at + }); + this.disk.restore(snapshot.disk); + this.disk.crash(powerLoss); + await this.persistence.save(snapshot.stored); + this.network.requests.push(...snapshot.requests); + this.network.schedule.push({ + type: "crash", + detail: { at, powerLoss, boundary: snapshot.boundary } + }); + this.interruptedRequest = snapshot.stored.database?.pending + ? snapshot.requests.at(-1) + : undefined; + this.client = undefined; + this.crashed = true; + } + public async restartAfterCrash(): Promise { + assert(this.crashed, "Restart requires a crashed client"); + this.wsFactory = new ManagedWebSocketFactory(); + this.crashed = false; + await this.init(); + await this.waitForSync(); + if (this.interruptedRequest) { + const request = this.interruptedRequest; + const retries = this.network.requests.filter( + (r) => r.body.requestId === request.body.requestId + ); + assert(retries.length >= 2, "Crashed CAS was never retried"); + for (const retry of retries) + assert( + JSON.stringify(retry.body) === + JSON.stringify(request.body) && + retry.url === request.url, + "Crash retry changed request identity, payload or target" + ); + } + this.interruptedRequest = undefined; + } public pauseWebSocket(): void { this.wsFactory.pause(); } @@ -268,7 +374,7 @@ export class DeterministicAgent { matches: (entry: HistoryEntry) => boolean, onMatch?: (entry: HistoryEntry) => void ): Promise { - let unsubscribe = () => { }; + let unsubscribe = () => {}; try { await withTimeout( new Promise((resolve) => { @@ -291,9 +397,20 @@ export class DeterministicAgent { } } public async cleanup(): Promise { + assert(!this.crashed, "Crashed client was never restarted"); + assert( + !this.saveFailure || this.saveFailure.fired, + "Save fault never fired" + ); if (!this.client) return; const errors: unknown[] = []; + try { + this.network.assertCheckpointsConsumed(); + } catch (error) { + errors.push(error); + } this.resumeObservation(); + this.network.releaseRequests(); this.flushNotifications(); try { await this.waitForSync(); diff --git a/frontend/deterministic-tests/src/test-definition.ts b/frontend/deterministic-tests/src/test-definition.ts index 3e4c5fe0..be26bc20 100644 --- a/frontend/deterministic-tests/src/test-definition.ts +++ b/frontend/deterministic-tests/src/test-definition.ts @@ -1,4 +1,5 @@ import type { AssertableState } from "./utils/assertable-state"; +import type { RequestCheckpoint } from "../../test-support/network"; export interface ClientState { files: Map; @@ -15,6 +16,7 @@ export interface ExpectedDocument { path: string; conflict?: boolean; content: string | number[]; + alternatives?: (string | number[])[]; } export type TestStep = @@ -30,7 +32,40 @@ export type TestStep = | { type: "delete"; client: number; path: string } | { type: "sync"; client?: number } | { type: "disable-sync"; client: number } - | { type: "enable-sync"; client: number } + | { type: "enable-sync"; client: number; wait?: boolean } + | ({ type: "hold-request"; client: number } & RequestCheckpoint) + | { type: "wait-for-request" | "release-request"; client: number } + | { + type: "remember-local-identity"; + client: number; + path: string; + key: string; + } + | { + type: "set-settings"; + client: number; + settings: Partial; + } + | { + type: "crash-client"; + client: number; + at: Pick< + import("./crash-worker").CrashWorkerInput, + "stopAt" | "stopAtBoundary" + >; + powerLoss?: boolean; + } + | { type: "restart-client"; client: number } + | { type: "fail-save"; client: number; point: "before" | "durable" } + | { type: "wait-for-save-failure"; client: number } + | { + type: "assert-response"; + client: number; + kind: "create" | "content" | "manifest"; + responseType: string; + count?: number; + sinceCheckpoint?: boolean; + } | { type: "pause-server" } | { type: "resume-server" } | { @@ -90,4 +125,9 @@ export interface TestResult { error?: string; duration?: number; diagnostics?: unknown; + coverage?: Record; + schedule?: { + client: number; + events: { type: string; detail: unknown }[]; + }[]; } diff --git a/frontend/deterministic-tests/src/test-runner.ts b/frontend/deterministic-tests/src/test-runner.ts index 2ea7db3a..22e3a6dc 100644 --- a/frontend/deterministic-tests/src/test-runner.ts +++ b/frontend/deterministic-tests/src/test-runner.ts @@ -24,6 +24,7 @@ export class TestRunner { private readonly logger: Logger; private vaultName = ""; private readonly identities = new Map(); + private readonly coverage: Record = {}; public constructor( serverControl: ServerControl, @@ -49,7 +50,9 @@ export class TestRunner { this.logger.info(`Clients: ${test.clients}`); this.logger.info(`Steps: ${test.steps.length}`); this.identities.clear(); + for (const key of Object.keys(this.coverage)) delete this.coverage[key]; let diagnostics: TestResult["diagnostics"]; + let evidence: Pick | undefined; try { assert( @@ -64,7 +67,9 @@ export class TestRunner { this.logger.info( `Step ${i + 1}/${test.steps.length}: ${JSON.stringify(step)}` ); - await this.executeStep(step); + this.coverage[`step:${step.type}`] = + (this.coverage[`step:${step.type}`] ?? 0) + 1; + await this.executeStep(this.resolveStep(step)); } assert( @@ -72,6 +77,7 @@ export class TestRunner { "Server died before test completion" ); diagnostics = this.captureDiagnostics(); + evidence = this.captureEvidence(); await this.cleanup(); const duration = Date.now() - startTime; @@ -79,10 +85,12 @@ export class TestRunner { return { success: true, - duration + duration, + ...evidence }; } catch (error) { diagnostics ??= this.captureDiagnostics(); + evidence ??= this.captureEvidence(); const duration = Date.now() - startTime; const errorMessage = error instanceof Error ? error.message : String(error); @@ -96,7 +104,8 @@ export class TestRunner { success: false, error: `${errorMessage}\nCleanup: ${String(cleanupError)}`, duration, - diagnostics + diagnostics, + ...evidence }; } @@ -104,17 +113,86 @@ export class TestRunner { success: false, error: errorMessage, diagnostics, - duration + duration, + ...evidence }; } } + private resolvePath(path: string): string { + return path.replace(/\{\{([^}]+)\}\}/g, (_, key: string) => { + const id = this.identities.get(key); + assert(id !== undefined, `Unknown path identity ${key}`); + return id; + }); + } + + private captureEvidence(): Pick { + const coverage = { ...this.coverage }; + const count = (key: string) => { + coverage[key] = (coverage[key] ?? 0) + 1; + }; + for (const agent of this.agents) { + for (const request of agent.network.requests) { + count( + `response:${request.kind}:${request.responseType ?? request.status ?? "interrupted"}` + ); + const payload = request.body.content as + | { type?: string } + | undefined; + if (payload?.type) count(`payload:${payload.type}`); + } + for (const event of agent.network.schedule) { + count(`checkpoint:${event.type}`); + if (event.type === "reached") { + const { spec } = event.detail as { + spec: { kind: string; point: string }; + }; + count(`held:${spec.kind}:${spec.point}`); + } else if (event.type === "crash") { + const { at, boundary } = event.detail as { + at: { stopAt?: string }; + boundary?: string; + }; + count( + `crash:${at.stopAt ?? boundary?.split(":").slice(0, 2).join(":")}` + ); + } else if (event.type === "save-failure") + count(`save-failure:${event.detail}`); + } + } + return { + coverage, + schedule: this.agents.map((agent) => ({ + client: agent.clientId, + events: structuredClone(agent.network.schedule) + })) + }; + } + + private resolveStep(step: TestStep): TestStep { + const next = { ...step }; + for (const field of ["path", "oldPath", "newPath"] as const) + if (field in next) + (next as unknown as Record)[field] = + this.resolvePath( + (next as unknown as Record)[field] + ); + if (next.type === "assert-documents") + next.expected = next.expected.map((doc) => ({ + ...doc, + path: this.resolvePath(doc.path) + })); + return next; + } + private captureDiagnostics(): unknown { return this.agents.map((agent) => ({ client: agent.clientId, database: agent.database(), disk: agent.disk.image(), - requests: structuredClone(agent.network.requests) + requests: structuredClone(agent.network.requests), + schedule: structuredClone(agent.network.schedule) })); } @@ -161,6 +239,88 @@ export class TestRunner { private async executeStep(step: TestStep): Promise { switch (step.type) { + case "fail-save": + this.getAgent(step.client).failSave(step.point); + break; + case "wait-for-save-failure": + await this.getAgent(step.client).waitForSaveFailure(); + break; + case "crash-client": + await this.getAgent(step.client).crashDuringSync( + step.at, + step.powerLoss + ); + break; + case "restart-client": + await this.getAgent(step.client).restartAfterCrash(); + break; + case "set-settings": + await this.getAgent(step.client).setSettings(step.settings); + break; + case "remember-local-identity": { + const agent = this.getAgent(step.client); + await agent.drain(); + const stored = agent.persistence.snapshot(); + const created = stored.localChanges + ?.filter( + (change) => + change.type === "create" && + change.path === step.path + ) + .at(-1); + const id = + (created?.type === "create" + ? created.documentId + : undefined) ?? + Object.entries(agent.database()?.local ?? {}).find( + ([, path]) => path === step.path + )?.[0]; + assert(id !== undefined, `No local identity at ${step.path}`); + const previous = this.identities.get(step.key); + assert( + previous === undefined || previous === id, + `Identity ${step.key} changed before publication` + ); + assert( + previous !== undefined || + ![...this.identities.values()].includes(id), + `New document ${step.key} reused an identity` + ); + this.identities.set(step.key, id); + break; + } + case "hold-request": + this.getAgent(step.client).network.hold(step); + break; + case "wait-for-request": + await withTimeout( + this.getAgent(step.client).network.waitForHeldRequest(), + CONVERGENCE_TIMEOUT_MS, + "Request checkpoint never reached" + ); + break; + case "release-request": + this.getAgent(step.client).network.resumeRequest(); + break; + case "assert-response": { + const network = this.getAgent(step.client).network; + const matching = ( + step.sinceCheckpoint + ? network.requestsSinceCheckpoint() + : network.requests + ).filter( + (r) => + r.kind === step.kind && + r.responseType === step.responseType + ); + assert( + step.count === undefined + ? matching.length > 0 + : matching.length === step.count, + `Expected ${step.count ?? "at least one"} ${step.kind} ${step.responseType} response, got ${matching.length}` + ); + break; + } case "pause-observation": this.getAgent(step.client).pauseObservation(); break; @@ -284,7 +444,7 @@ export class TestRunner { case "enable-sync": await this.getAgent(step.client).enableSync( - !this.serverControl.isPaused + step.wait ?? !this.serverControl.isPaused ); break; @@ -319,7 +479,14 @@ export class TestRunner { break; case "assert-consistent": - await this.assertConsistent(step.verify); + await this.assertConsistent( + step.verify + ? (state) => { + step.verify!(state); + state.assertComplete(); + } + : undefined + ); break; case "pause-websocket": diff --git a/frontend/deterministic-tests/src/tests/binary-pending-create-not-displaced.test.ts b/frontend/deterministic-tests/src/tests/binary-pending-create-not-displaced.test.ts index f2582964..c5fed404 100644 --- a/frontend/deterministic-tests/src/tests/binary-pending-create-not-displaced.test.ts +++ b/frontend/deterministic-tests/src/tests/binary-pending-create-not-displaced.test.ts @@ -34,10 +34,9 @@ export const binaryPendingCreateNotDisplacedTest: TestDefinition = { type: "assert-consistent", verify: (s: AssertableState): void => { s.assertFileCount(2) - .assertFileExists("data.bin") - .assertFileExists(s.conflictPath("data.bin")) - .assertAnyFileContains( - "binary data from client 0", + .assertContent("data.bin", "binary data from client 0") + .assertContent( + s.conflictPath("data.bin"), "binary data from client 1" ); } diff --git a/frontend/deterministic-tests/src/tests/binary-to-text-transition.test.ts b/frontend/deterministic-tests/src/tests/binary-to-text-transition.test.ts index 8b934c1b..fc0cf3f9 100644 --- a/frontend/deterministic-tests/src/tests/binary-to-text-transition.test.ts +++ b/frontend/deterministic-tests/src/tests/binary-to-text-transition.test.ts @@ -38,11 +38,7 @@ export const binaryToTextTransitionTest: TestDefinition = { { type: "assert-consistent", verify: (s: AssertableState): void => { - s.assertFileCount(1).assertContainsAny( - "data.bin", - "version A", - "version B" - ); + s.assertFileCount(1).assertContent("data.bin", "version A"); } }, @@ -90,7 +86,10 @@ export const binaryToTextTransitionTest: TestDefinition = { { type: "assert-consistent", verify: (s: AssertableState): void => { - s.assertFileCount(1).assertContains("data.md", "alpha", "beta"); + s.assertFileCount(1).assertContent( + "data.md", + "alpha\nmiddle line\nbeta" + ); } } ] diff --git a/frontend/deterministic-tests/src/tests/coalesce-update-remote-update-data-loss.test.ts b/frontend/deterministic-tests/src/tests/coalesce-update-remote-update-data-loss.test.ts index 1972526a..5cc7e4df 100644 --- a/frontend/deterministic-tests/src/tests/coalesce-update-remote-update-data-loss.test.ts +++ b/frontend/deterministic-tests/src/tests/coalesce-update-remote-update-data-loss.test.ts @@ -45,14 +45,12 @@ export const coalesceUpdateRemoteUpdateDataLossTest: TestDefinition = { { type: "assert-consistent", - verify: (state: AssertableState): void => { - state - .assertFileCount(1) - .assertContains( - "doc.md", - "client 0 addition", - "client 1 addition" - ); + verify: (s: AssertableState): void => { + s.assertFileCount(1); + s.assertContent( + "doc.md", + "client 1 addition\nline 1\nline 2\nline 3\nclient 0 addition" + ); } } ] diff --git a/frontend/deterministic-tests/src/tests/concurrent-edit-exact-same-position.test.ts b/frontend/deterministic-tests/src/tests/concurrent-edit-exact-same-position.test.ts index 5c141a0e..39ca805f 100644 --- a/frontend/deterministic-tests/src/tests/concurrent-edit-exact-same-position.test.ts +++ b/frontend/deterministic-tests/src/tests/concurrent-edit-exact-same-position.test.ts @@ -42,7 +42,11 @@ export const concurrentEditExactSamePositionTest: TestDefinition = { verify: (state: AssertableState): void => { state .assertFileCount(1) - .assertContains("doc.md", "slow", "fast", "brown fox"); + .assertContentOneOf( + "doc.md", + "the fast slow brown fox", + "the slow fast brown fox" + ); } } ] diff --git a/frontend/deterministic-tests/src/tests/concurrent-rename-and-create-at-target-create-first.test.ts b/frontend/deterministic-tests/src/tests/concurrent-rename-and-create-at-target-create-first.test.ts index d76d4053..cc8d1163 100644 --- a/frontend/deterministic-tests/src/tests/concurrent-rename-and-create-at-target-create-first.test.ts +++ b/frontend/deterministic-tests/src/tests/concurrent-rename-and-create-at-target-create-first.test.ts @@ -1,49 +1,52 @@ import type { AssertableState } from "../utils/assertable-state"; import type { TestDefinition } from "../test-definition"; -export const concurrentRenameAndCreateAtTargetCreateFirstTest: TestDefinition = { - description: - "One client renames X to Y while another creates a new file at Y, " + - "both offline. After syncing, Y should contain merged content from " + - "both the renamed file and the newly created file.", - clients: 2, - steps: [ - { - type: "create", - client: 0, - path: "X.md", - content: "original file X" - }, - { type: "enable-sync", client: 0 }, - { type: "enable-sync", client: 1 }, - { type: "barrier" }, +export const concurrentRenameAndCreateAtTargetCreateFirstTest: TestDefinition = + { + description: + "One client renames X to Y while another creates a new file at Y, " + + "both offline. After syncing, Y should contain merged content from " + + "both the renamed file and the newly created file.", + clients: 2, + steps: [ + { + type: "create", + client: 0, + path: "X.md", + content: "original file X" + }, + { type: "enable-sync", client: 0 }, + { type: "enable-sync", client: 1 }, + { type: "barrier" }, - { type: "disable-sync", client: 0 }, - { type: "disable-sync", client: 1 }, + { type: "disable-sync", client: 0 }, + { type: "disable-sync", client: 1 }, - { type: "rename", client: 0, oldPath: "X.md", newPath: "Y.md" }, + { type: "rename", client: 0, oldPath: "X.md", newPath: "Y.md" }, - { - type: "create", - client: 1, - path: "Y.md", - content: "brand new Y content" - }, + { + type: "create", + client: 1, + path: "Y.md", + content: "brand new Y content" + }, - { type: "enable-sync", client: 1 }, - { type: "sync", client: 1 }, + { type: "enable-sync", client: 1 }, + { type: "sync", client: 1 }, - { type: "enable-sync", client: 0 }, - { type: "barrier" }, + { type: "enable-sync", client: 0 }, + { type: "barrier" }, - { - type: "assert-consistent", - verify: (state: AssertableState): void => { - state - .assertFileCount(2) - .assertContains(state.conflictPath("Y.md"), "original file X") - .assertContains("Y.md", "brand new Y content"); + { + type: "assert-consistent", + verify: (s: AssertableState): void => { + s.assertFileCount(2) + .assertContent("Y.md", "brand new Y content") + .assertContent( + s.conflictPath("Y.md"), + "original file X" + ); + } } - } - ] -}; + ] + }; diff --git a/frontend/deterministic-tests/src/tests/concurrent-rename-and-create-at-target-rename-first.test.ts b/frontend/deterministic-tests/src/tests/concurrent-rename-and-create-at-target-rename-first.test.ts index 3ae30504..92180cfc 100644 --- a/frontend/deterministic-tests/src/tests/concurrent-rename-and-create-at-target-rename-first.test.ts +++ b/frontend/deterministic-tests/src/tests/concurrent-rename-and-create-at-target-rename-first.test.ts @@ -1,52 +1,51 @@ import type { AssertableState } from "../utils/assertable-state"; import type { TestDefinition } from "../test-definition"; -export const concurrentRenameAndCreateAtTargetRenameFirstTest: TestDefinition = { - description: - "One client renames X to Y while another creates a new file at Y, " + - "both offline. We can't merge the create because it would result in a cycle", - clients: 2, - steps: [ - { - type: "create", - client: 0, - path: "X.md", - content: "original file X" - }, - { type: "enable-sync", client: 0 }, - { type: "enable-sync", client: 1 }, - { type: "barrier" }, +export const concurrentRenameAndCreateAtTargetRenameFirstTest: TestDefinition = + { + description: + "One client renames X to Y while another creates a new file at Y, " + + "both offline. We can't merge the create because it would result in a cycle", + clients: 2, + steps: [ + { + type: "create", + client: 0, + path: "X.md", + content: "original file X" + }, + { type: "enable-sync", client: 0 }, + { type: "enable-sync", client: 1 }, + { type: "barrier" }, - { type: "disable-sync", client: 0 }, - { type: "disable-sync", client: 1 }, + { type: "disable-sync", client: 0 }, + { type: "disable-sync", client: 1 }, - { type: "rename", client: 0, oldPath: "X.md", newPath: "Y.md" }, + { type: "rename", client: 0, oldPath: "X.md", newPath: "Y.md" }, - { - type: "create", - client: 1, - path: "Y.md", - content: "brand new Y content" - }, + { + type: "create", + client: 1, + path: "Y.md", + content: "brand new Y content" + }, - { type: "enable-sync", client: 0 }, - { type: "sync", client: 0 }, + { type: "enable-sync", client: 0 }, + { type: "sync", client: 0 }, - { type: "enable-sync", client: 1 }, - { type: "barrier" }, + { type: "enable-sync", client: 1 }, + { type: "barrier" }, - { - type: "assert-consistent", - verify: (state: AssertableState): void => { - state - .assertFileNotExists("X.md") - .assertFileExists("Y.md") - .assertFileExists(state.conflictPath("Y.md")) - .assertAnyFileContains( - "original file X", - "brand new Y content" - ); + { + type: "assert-consistent", + verify: (s: AssertableState): void => { + s.assertFileCount(2) + .assertContent("Y.md", "original file X") + .assertContent( + s.conflictPath("Y.md"), + "brand new Y content" + ); + } } - } - ] -}; + ] + }; diff --git a/frontend/deterministic-tests/src/tests/concurrent-rename-same-target.test.ts b/frontend/deterministic-tests/src/tests/concurrent-rename-same-target.test.ts index dcd8200d..7b68dfd2 100644 --- a/frontend/deterministic-tests/src/tests/concurrent-rename-same-target.test.ts +++ b/frontend/deterministic-tests/src/tests/concurrent-rename-same-target.test.ts @@ -25,14 +25,10 @@ export const concurrentRenameSameTargetTest: TestDefinition = { { type: "assert-consistent", - verify: (state: AssertableState): void => { - state - .assertFileCount(2) - .assertFileNotExists("A.md") - .assertFileNotExists("B.md") - .assertFileExists("C.md") - .assertFileExists(state.conflictPath("C.md")) - .assertAnyFileContains("content-a", "content-b"); + verify: (s: AssertableState): void => { + s.assertFileCount(2) + .assertContent("C.md", "content-a") + .assertContent(s.conflictPath("C.md"), "content-b"); } } ] diff --git a/frontend/deterministic-tests/src/tests/create-delete-noop.test.ts b/frontend/deterministic-tests/src/tests/create-delete-noop.test.ts index 6c766001..a16969f9 100644 --- a/frontend/deterministic-tests/src/tests/create-delete-noop.test.ts +++ b/frontend/deterministic-tests/src/tests/create-delete-noop.test.ts @@ -20,7 +20,7 @@ export const createDeleteNoopTest: TestDefinition = { { type: "assert-consistent", verify: (s: AssertableState): void => { - s.assertFileNotExists("temp.md"); + s.assertFileCount(0).assertFileNotExists("temp.md"); } } ] diff --git a/frontend/deterministic-tests/src/tests/create-merge-delete.test.ts b/frontend/deterministic-tests/src/tests/create-merge-delete.test.ts index ef7ea5c3..4553b7d6 100644 --- a/frontend/deterministic-tests/src/tests/create-merge-delete.test.ts +++ b/frontend/deterministic-tests/src/tests/create-merge-delete.test.ts @@ -17,10 +17,14 @@ export const createMergeDeleteTest: TestDefinition = { { type: "assert-consistent", - verify: (state: AssertableState): void => { - state - .assertFileCount(1) - .assertContains("A.md", "from-zero", "from-one"); + verify: (s: AssertableState): void => { + s.assertFileCount(1).assertContentOneOf( + "A.md", + "from-zerofrom-one", + "from-onefrom-zero", + "from-zero from-one", + "from-one from-zero" + ); } }, diff --git a/frontend/deterministic-tests/src/tests/create-merge-preserves-renamed-update.test.ts b/frontend/deterministic-tests/src/tests/create-merge-preserves-renamed-update.test.ts index a9bc37d4..3c54c6af 100644 --- a/frontend/deterministic-tests/src/tests/create-merge-preserves-renamed-update.test.ts +++ b/frontend/deterministic-tests/src/tests/create-merge-preserves-renamed-update.test.ts @@ -17,8 +17,14 @@ export const createMergePreservesRenamedUpdateTest: TestDefinition = { { type: "assert-consistent", - verify: (state: AssertableState): void => { - state.assertContains("doc.md", "alpha", "beta"); + verify: (s: AssertableState): void => { + s.assertFileCount(1).assertContentOneOf( + "doc.md", + "alphabeta", + "betaalpha", + "alpha beta", + "beta alpha" + ); } }, diff --git a/frontend/deterministic-tests/src/tests/create-rename-response-skips-file.test.ts b/frontend/deterministic-tests/src/tests/create-rename-response-skips-file.test.ts index aa24b110..ec90dd12 100644 --- a/frontend/deterministic-tests/src/tests/create-rename-response-skips-file.test.ts +++ b/frontend/deterministic-tests/src/tests/create-rename-response-skips-file.test.ts @@ -29,7 +29,8 @@ export const createRenameResponseSkipsFileTest: TestDefinition = { { type: "assert-consistent", verify: (s: AssertableState): void => { - s.assertFileCount(1).assertAnyFileContains("the-content"); + s.assertFileCount(1); + s.assertContent("renamed.md", "the-content"); } } ] diff --git a/frontend/deterministic-tests/src/tests/delete-by-other-client-then-recreate.test.ts b/frontend/deterministic-tests/src/tests/delete-by-other-client-then-recreate.test.ts index dfef9961..0d4fa9ba 100644 --- a/frontend/deterministic-tests/src/tests/delete-by-other-client-then-recreate.test.ts +++ b/frontend/deterministic-tests/src/tests/delete-by-other-client-then-recreate.test.ts @@ -18,7 +18,7 @@ export const deleteByOtherClientThenRecreateTest: TestDefinition = { { type: "assert-consistent", verify: (s: AssertableState): void => { - s.assertFileNotExists("A.md"); + s.assertFileCount(0).assertFileNotExists("A.md"); } }, diff --git a/frontend/deterministic-tests/src/tests/delete-during-pending-create.test.ts b/frontend/deterministic-tests/src/tests/delete-during-pending-create.test.ts index 3ba393b8..8768826c 100644 --- a/frontend/deterministic-tests/src/tests/delete-during-pending-create.test.ts +++ b/frontend/deterministic-tests/src/tests/delete-during-pending-create.test.ts @@ -3,15 +3,15 @@ import type { TestDefinition } from "../test-definition"; export const deleteDuringPendingCreateTest: TestDefinition = { description: - "Client 0 creates a file while the server is paused, then deletes it before the server resumes. " + - "After resume, the file should end up deleted on both clients.", + "Client 0 creates a file while its create request is held, then deletes it before releasing the request. " + + "After release, the file should end up deleted on both clients.", clients: 2, steps: [ { type: "enable-sync", client: 0 }, { type: "enable-sync", client: 1 }, { type: "barrier" }, - { type: "pause-server" }, + { type: "hold-request", client: 0, kind: "create", point: "before" }, { type: "create", @@ -19,10 +19,11 @@ export const deleteDuringPendingCreateTest: TestDefinition = { path: "ephemeral.md", content: "this will be deleted" }, + { type: "wait-for-request", client: 0 }, { type: "delete", client: 0, path: "ephemeral.md" }, - { type: "resume-server" }, + { type: "release-request", client: 0 }, { type: "barrier" }, { diff --git a/frontend/deterministic-tests/src/tests/delete-recreate-concurrent-update.test.ts b/frontend/deterministic-tests/src/tests/delete-recreate-concurrent-update.test.ts index 6cb4cb98..57169798 100644 --- a/frontend/deterministic-tests/src/tests/delete-recreate-concurrent-update.test.ts +++ b/frontend/deterministic-tests/src/tests/delete-recreate-concurrent-update.test.ts @@ -35,7 +35,8 @@ export const deleteRecreateConcurrentUpdateTest: TestDefinition = { { type: "assert-consistent", verify: (s: AssertableState): void => { - s.assertFileExists("A.md").assertContains("A.md", "recreated"); + s.assertFileCount(1); + s.assertContent("A.md", "recreated by client 0"); } } ] diff --git a/frontend/deterministic-tests/src/tests/delete-recreated-pending-create-with-stale-deleting-record.test.ts b/frontend/deterministic-tests/src/tests/delete-recreated-pending-create-with-stale-deleting-record.test.ts index 80e95f48..1a19edb2 100644 --- a/frontend/deterministic-tests/src/tests/delete-recreated-pending-create-with-stale-deleting-record.test.ts +++ b/frontend/deterministic-tests/src/tests/delete-recreated-pending-create-with-stale-deleting-record.test.ts @@ -4,10 +4,9 @@ import type { TestDefinition } from "../test-definition"; export const deleteRecreatedPendingCreateWithStaleDeletingRecordTest: TestDefinition = { description: - "A local delete for a recreated pending create must target the " + - "new pending create, not an older same-path record whose server " + - "delete has been acked but whose WebSocket delete receipt is " + - "still paused.", + "Delete two successive generations while each create request is " + + "held before send. Retrying and acknowledging the old generation " + + "must not resurrect either file, even with delayed WebSocket hints.", clients: 2, steps: [ { type: "enable-sync", client: 0 }, @@ -15,28 +14,38 @@ export const deleteRecreatedPendingCreateWithStaleDeletingRecordTest: TestDefini { type: "barrier" }, { type: "pause-websocket", client: 0 }, - { type: "pause-server" }, + { + type: "hold-request", + client: 0, + kind: "create", + point: "before" + }, { type: "create", client: 0, path: "binary-14.bin", content: "BINARY:first" }, - { type: "sleep", ms: 100 }, + { type: "wait-for-request", client: 0 }, { type: "delete", client: 0, path: "binary-14.bin" }, - { type: "resume-server" }, + { type: "release-request", client: 0 }, { type: "sync", client: 0 }, - { type: "pause-server" }, + { + type: "hold-request", + client: 0, + kind: "create", + point: "before" + }, { type: "create", client: 0, path: "binary-14.bin", content: "BINARY:second" }, - { type: "sleep", ms: 100 }, + { type: "wait-for-request", client: 0 }, { type: "delete", client: 0, path: "binary-14.bin" }, - { type: "resume-server" }, + { type: "release-request", client: 0 }, { type: "sync", client: 0 }, { type: "resume-websocket", client: 0 }, diff --git a/frontend/deterministic-tests/src/tests/delete-rename-conflict.test.ts b/frontend/deterministic-tests/src/tests/delete-rename-conflict.test.ts index 91e6289b..2b37989a 100644 --- a/frontend/deterministic-tests/src/tests/delete-rename-conflict.test.ts +++ b/frontend/deterministic-tests/src/tests/delete-rename-conflict.test.ts @@ -15,7 +15,8 @@ export const deleteRenameConflictTest: TestDefinition = { { type: "assert-consistent", verify: (s: AssertableState): void => { - s.assertFileExists("A.md").assertFileExists("B.md"); + s.assertContent("A.md", "content-a"); + s.assertContent("B.md", "content-b"); } }, @@ -32,11 +33,8 @@ export const deleteRenameConflictTest: TestDefinition = { { type: "assert-consistent", verify: (s: AssertableState): void => { + s.assertFileCount(1); s.assertContent("B.md", "content-b"); - s.assertFileNotExists("A.md"); - s.ifFileExists("C.md", (inner) => - inner.assertContent("C.md", "content-a") - ); } } ] diff --git a/frontend/deterministic-tests/src/tests/idempotency-after-server-pause.test.ts b/frontend/deterministic-tests/src/tests/idempotency-after-server-pause.test.ts index 551c702d..325ad962 100644 --- a/frontend/deterministic-tests/src/tests/idempotency-after-server-pause.test.ts +++ b/frontend/deterministic-tests/src/tests/idempotency-after-server-pause.test.ts @@ -3,13 +3,13 @@ import type { TestDefinition } from "../test-definition"; export const idempotencyAfterServerPauseTest: TestDefinition = { description: - "Client 0 creates a file, then the server is paused mid-response. " + - "After the server resumes, both clients must converge to a single copy of the file with no duplicates.", + "Lose an accepted create response and require an identical retry, preserving one exact copy.", clients: 2, steps: [ { type: "enable-sync", client: 0 }, { type: "enable-sync", client: 1 }, { type: "barrier" }, + { type: "drop-response", client: 0, kind: "create", point: "after" }, { type: "create", @@ -17,9 +17,7 @@ export const idempotencyAfterServerPauseTest: TestDefinition = { path: "doc.md", content: "important data" }, - { type: "pause-server" }, - - { type: "resume-server" }, + { type: "wait-for-response-drop", client: 0 }, { type: "barrier" }, diff --git a/frontend/deterministic-tests/src/tests/interrupted-delete-retry.test.ts b/frontend/deterministic-tests/src/tests/interrupted-delete-retry.test.ts index 3ae7eda5..4378c24b 100644 --- a/frontend/deterministic-tests/src/tests/interrupted-delete-retry.test.ts +++ b/frontend/deterministic-tests/src/tests/interrupted-delete-retry.test.ts @@ -3,20 +3,19 @@ import type { TestDefinition } from "../test-definition"; export const interruptedDeleteRetryTest: TestDefinition = { description: - "Client 0 deletes a file, then the server is paused. " + - "After the server resumes, both clients should have zero files.", + "Lose an accepted deletion manifest response and require an identical retry, with both clients empty.", clients: 2, steps: [ { type: "create", client: 0, path: "doc.md", content: "to be deleted" }, { type: "enable-sync", client: 0 }, { type: "enable-sync", client: 1 }, { type: "barrier" }, + { type: "drop-response", client: 0, kind: "manifest", point: "after" }, { type: "delete", client: 0, path: "doc.md" }, - { type: "pause-server" }, + { type: "wait-for-response-drop", client: 0 }, - { type: "resume-server" }, { type: "barrier" }, { diff --git a/frontend/deterministic-tests/src/tests/local-edit-lost-during-create-merge.test.ts b/frontend/deterministic-tests/src/tests/local-edit-lost-during-create-merge.test.ts index 20925889..5d5e7588 100644 --- a/frontend/deterministic-tests/src/tests/local-edit-lost-during-create-merge.test.ts +++ b/frontend/deterministic-tests/src/tests/local-edit-lost-during-create-merge.test.ts @@ -30,10 +30,12 @@ export const localEditLostDuringCreateMergeTest: TestDefinition = { { type: "assert-consistent", verify: (s: AssertableState): void => { - s.assertFileCount(1).assertContains( + s.assertFileCount(1).assertContentOneOf( "doc.md", - "from-client-1", - "local-edit-during-create" + "from-client-1local-edit-during-create", + "local-edit-during-createfrom-client-1", + "from-client-1 local-edit-during-create", + "local-edit-during-create from-client-1" ); } } diff --git a/frontend/deterministic-tests/src/tests/local-rename-survives-remote-rename.test.ts b/frontend/deterministic-tests/src/tests/local-rename-survives-remote-rename.test.ts index c2b80af3..7d1ca40b 100644 --- a/frontend/deterministic-tests/src/tests/local-rename-survives-remote-rename.test.ts +++ b/frontend/deterministic-tests/src/tests/local-rename-survives-remote-rename.test.ts @@ -1,21 +1,8 @@ -import type { AssertableState } from "../utils/assertable-state"; import type { TestDefinition } from "../test-definition"; export const localRenameSurvivesRemoteRenameTest: TestDefinition = { description: - "Drain processes a RemoteChange (remote rename for doc D) while a " + - "LocalUpdate (user rename of D) is also queued behind it. " + - "`processRemoteUpdate` moves the disk file and, because there is a " + - "pending LocalUpdate, takes the else branch — but its setDocument " + - "uses the stale `record.path` (= the user-rename target) instead of " + - "the actualPath the file just moved to. The queued LocalUpdate then " + - "reads from `record.path`, throws FileNotFoundError, and is " + - "silently dropped. Setup pins the queue order: a sentinel " + - "LocalUpdate keeps drain busy on a SIGSTOPped HTTP roundtrip while " + - "we resume client 0's WebSocket (enqueues RemoteChange) and then " + - "user-rename D (enqueues LocalUpdate after the RemoteChange). On " + - "server resume the drain pops the sentinel, then RemoteChange, then " + - "LocalUpdate — exactly the order that triggers the bug.", + "A held remote manifest races with a local rename and a separate edit; the first committed rename wins and both identities survive.", clients: 2, steps: [ { type: "enable-sync", client: 0 }, @@ -23,58 +10,31 @@ export const localRenameSurvivesRemoteRenameTest: TestDefinition = { { type: "create", client: 0, path: "doc.md", content: "v1\n" }, { type: "create", client: 0, path: "sentinel.md", content: "s\n" }, { type: "barrier" }, - - // Pause client 0's WebSocket so the upcoming remote rename buffers. - { type: "pause-websocket", client: 0 }, - - // Server applies remote rename of doc.md -> remote.md. Broadcast - // is buffered on client 0's WebSocket. + { + type: "assert-documents", + expected: [ + { key: "note", path: "doc.md", content: "v1\n" }, + { key: "sentinel", path: "sentinel.md", content: "s\n" } + ] + }, + { type: "hold-request", client: 0, kind: "events", point: "after" }, { type: "rename", client: 1, oldPath: "doc.md", newPath: "remote.md" }, - { type: "sync", client: 1 }, - - // Pause the server BEFORE arming the sentinel, so the sentinel's - // HTTP request will buffer at the kernel and keep drain occupied. - { type: "pause-server" }, - - // Sentinel: a LocalUpdate on a *different* doc that drain pops - // first. Its HTTP roundtrip stalls on SIGSTOP, freezing drain - // until we resume the server. While drain is frozen we can grow - // the queue with additional events whose order we control. + { type: "wait-for-request", client: 0 }, { type: "update", client: 0, path: "sentinel.md", content: "s\nedit\n" }, - - // Resume the WebSocket — buffered remote rename enqueues as a - // RemoteChange. Drain is still stuck on the sentinel HTTP. - { type: "resume-websocket", client: 0 }, - - // User renames doc.md -> local.md on client 0. queue.enqueue - // mutates the doc's record.path to "local.md" and pushes a - // LocalUpdate(rename) onto the tail of the queue. Queue is now - // [sentinel-update (in-flight), RemoteChange, LocalUpdate-rename]. { type: "rename", client: 0, oldPath: "doc.md", newPath: "local.md" }, - - // Resume the server. Drain pops sentinel-update (succeeds), then - // RemoteChange. Pre-fix: processRemoteUpdate moves disk - // local.md -> remote.md, takes the else branch, and - // setDocument(record.path = "local.md", …) leaves record.path - // stale. Drain pops the LocalUpdate-rename and reads from the - // stale record.path, hits FileNotFoundError, silent skip. - // Post-fix: when a local event is pending, we re-queue the - // remote update without touching disk or record, so the local - // rename drains first and both ends converge. - { type: "resume-server" }, - + { type: "release-request", client: 0 }, { type: "barrier" }, - { - type: "assert-consistent", - verify: (state: AssertableState): void => { - state.assertFileCount(2); - } + type: "assert-documents", + expected: [ + { key: "note", path: "remote.md", content: "v1\n" }, + { key: "sentinel", path: "sentinel.md", content: "s\nedit\n" } + ] } ] }; diff --git a/frontend/deterministic-tests/src/tests/local-update-survives-remote-rename.test.ts b/frontend/deterministic-tests/src/tests/local-update-survives-remote-rename.test.ts index 0d8348c0..56afc341 100644 --- a/frontend/deterministic-tests/src/tests/local-update-survives-remote-rename.test.ts +++ b/frontend/deterministic-tests/src/tests/local-update-survives-remote-rename.test.ts @@ -3,17 +3,7 @@ import type { TestDefinition } from "../test-definition"; export const localUpdateSurvivesRemoteRenameTest: TestDefinition = { description: - "Client 0 has a local content edit pending while a remote rename for " + - "the same doc arrives over the WebSocket. The remote rename's internal " + - "move relocates the disk file from the old path (where the user wrote) " + - "to the new server path. Previously, the queued LocalUpdate's " + - "`event.path` was left pointing at the now-vacated old path, so " + - "`skipIfOversized`'s `getFileSize(event.path)` threw " + - "`FileNotFoundError`, which `processEvent`'s catch silently swallowed " + - "as 'Skipping sync event 'local-update' because the file no longer " + - "exists' — and the user's edit was lost. The fix routes the size " + - "check through `tracked.path` (the doc's current disk path), " + - "matching the path `processLocalUpdate` itself reads from.", + "Hold the event response carrying a remote rename, edit the old local path, then release catchup. Preserve the exact user edit at the renamed path.", clients: 2, steps: [ { type: "create", client: 0, path: "doc.md", content: "v1\n" }, @@ -21,11 +11,7 @@ export const localUpdateSurvivesRemoteRenameTest: TestDefinition = { { type: "enable-sync", client: 1 }, { type: "barrier" }, - // Pause client 0's WebSocket so the upcoming remote rename buffers - // there until we've already enqueued client 0's local content - // edit. This guarantees the LocalUpdate sits in client 0's queue - // when the rename's RemoteChange drains. - { type: "pause-websocket", client: 0 }, + { type: "hold-request", client: 0, kind: "events", point: "after" }, { type: "rename", @@ -34,11 +20,8 @@ export const localUpdateSurvivesRemoteRenameTest: TestDefinition = { newPath: "renamed.md" }, { type: "sync", client: 1 }, + { type: "wait-for-request", client: 0 }, - // Client 0 still believes the file is at `doc.md` (its WebSocket is - // paused, so the rename hasn't reached it). The user edits content - // at `doc.md`. This pushes a LocalUpdate(D, path=doc.md, - // originalPath=doc.md, isUserRename=false) into client 0's queue. { type: "update", client: 0, @@ -46,14 +29,7 @@ export const localUpdateSurvivesRemoteRenameTest: TestDefinition = { content: "v1\nclient 0 edit\n" }, - // Resume the WebSocket. The buffered remote rename (server-broadcast) - // drains. `processRemoteUpdate` does an internal `move(doc.md, - // renamed.md)` and, because there's a pending LocalUpdate for D, - // takes the else branch (re-enqueue v_K, setDocument(renamed.md, …)). - // Then drain reaches the LocalUpdate. Pre-fix: skipped silently. - // Post-fix: PUTs the user's content to the doc (at its new path, - // since this is a content-only edit, not a user rename). - { type: "resume-websocket", client: 0 }, + { type: "release-request", client: 0 }, { type: "barrier" }, diff --git a/frontend/deterministic-tests/src/tests/mc-cross-create-rename-same-target.test.ts b/frontend/deterministic-tests/src/tests/mc-cross-create-rename-same-target.test.ts index d986a733..c6b56338 100644 --- a/frontend/deterministic-tests/src/tests/mc-cross-create-rename-same-target.test.ts +++ b/frontend/deterministic-tests/src/tests/mc-cross-create-rename-same-target.test.ts @@ -18,7 +18,8 @@ export const mcCrossCreateRenameSameTargetTest: TestDefinition = { { type: "assert-consistent", verify: (s: AssertableState): void => { - s.assertFileExists("X.md").assertFileExists("Y.md"); + s.assertContent("X.md", "content-x"); + s.assertContent("Y.md", "content-y"); } }, @@ -36,10 +37,8 @@ export const mcCrossCreateRenameSameTargetTest: TestDefinition = { type: "assert-consistent", verify: (s: AssertableState): void => { s.assertFileCount(2) - .assertFileNotExists("X.md") - .assertFileNotExists("Y.md") - .assertFileExists("Z.md") - .assertAnyFileContains("content-x", "content-y"); + .assertContent("Z.md", "content-x") + .assertContent(s.conflictPath("Z.md"), "content-y"); } } ] diff --git a/frontend/deterministic-tests/src/tests/mc-delete-then-offline-rename.test.ts b/frontend/deterministic-tests/src/tests/mc-delete-then-offline-rename.test.ts index 6727e99d..e16b349d 100644 --- a/frontend/deterministic-tests/src/tests/mc-delete-then-offline-rename.test.ts +++ b/frontend/deterministic-tests/src/tests/mc-delete-then-offline-rename.test.ts @@ -27,12 +27,8 @@ export const mcDeleteThenOfflineRenameTest: TestDefinition = { { type: "assert-consistent", verify: (s: AssertableState): void => { - s.assertContent("C.md", "unrelated").assertFileNotExists( - "A.md" - ); - s.ifFileExists("B.md", (inner) => - inner.assertContent("B.md", "original") - ); + s.assertFileCount(1); + s.assertContent("C.md", "unrelated"); } } ] diff --git a/frontend/deterministic-tests/src/tests/mc-multi-delete-offline-rename.test.ts b/frontend/deterministic-tests/src/tests/mc-multi-delete-offline-rename.test.ts index 8db90aab..1f33b91d 100644 --- a/frontend/deterministic-tests/src/tests/mc-multi-delete-offline-rename.test.ts +++ b/frontend/deterministic-tests/src/tests/mc-multi-delete-offline-rename.test.ts @@ -35,14 +35,10 @@ export const mcMultiDeleteOfflineRenameTest: TestDefinition = { { type: "assert-consistent", verify: (s: AssertableState): void => { - s.assertFileExists("file-1.md") - .assertFileExists("file-3.md") - .assertFileExists("file-5.md") - .assertFileNotExists("file-2.md") - .assertFileNotExists("file-4.md"); - s.ifFileExists("renamed.md", (inner) => - inner.assertContent("renamed.md", "content-2") - ); + s.assertFileCount(3); + s.assertContent("file-1.md", "content-1"); + s.assertContent("file-3.md", "content-3"); + s.assertContent("file-5.md", "content-5"); } } ] diff --git a/frontend/deterministic-tests/src/tests/mc-three-client-rename-offline-update.test.ts b/frontend/deterministic-tests/src/tests/mc-three-client-rename-offline-update.test.ts index 4167b925..af1c85ba 100644 --- a/frontend/deterministic-tests/src/tests/mc-three-client-rename-offline-update.test.ts +++ b/frontend/deterministic-tests/src/tests/mc-three-client-rename-offline-update.test.ts @@ -32,9 +32,8 @@ export const mcThreeClientRenameOfflineUpdateTest: TestDefinition = { { type: "assert-consistent", verify: (s: AssertableState): void => { - s.assertFileCount(1) - .assertFileNotExists("A.md") - .assertContains("B.md", "updated-by-client-2"); + s.assertFileCount(1); + s.assertContent("B.md", "updated-by-client-2"); } } ] diff --git a/frontend/deterministic-tests/src/tests/merging-update-response-survives-user-rename.test.ts b/frontend/deterministic-tests/src/tests/merging-update-response-survives-user-rename.test.ts index e93240f9..edfc2eea 100644 --- a/frontend/deterministic-tests/src/tests/merging-update-response-survives-user-rename.test.ts +++ b/frontend/deterministic-tests/src/tests/merging-update-response-survives-user-rename.test.ts @@ -1,77 +1,38 @@ -import type { AssertableState } from "../utils/assertable-state"; import type { TestDefinition } from "../test-definition"; export const mergingUpdateResponseSurvivesUserRenameTest: TestDefinition = { description: - "Client 1 sends a content update with a stale `parent_version_id` " + - "(its WebSocket is paused, so it hasn't seen Client 0's intervening " + - "edit). The server merges and replies with `MergingUpdate` carrying " + - "the merged text. Before the response lands, the user renames the " + - "doc on Client 1, vacating the disk path the in-flight " + - "`processLocalUpdate` captured. Pre-fix: " + - "`handleMaybeMergingResponse`'s `operations.write(diskPath, …)` " + - "hits the `we wont recreate it` early-return inside `write`, " + - "silently dropping the server-merged content — Client 0's edit is " + - "lost on Client 1's disk, and Client 1's next local-update PUT " + - "(rebased on the now-untracked merged version) deletes Client 0's " + - "edit on the server too. Post-fix: the response is written to the " + - "doc's current tracked disk path, preserving both edits.", + "A proven stale content CAS is reconciled after the user renames its file.", clients: 2, steps: [ + { type: "create", client: 0, path: "doc.md", content: "0\n" }, { type: "enable-sync", client: 0 }, { type: "enable-sync", client: 1 }, - { type: "create", client: 0, path: "doc.md", content: "0\n" }, { type: "barrier" }, - - // Stop Client 1 from seeing Client 0's next edit, so its next - // outbound PUT carries a stale `parent_version_id` and the server - // is forced to merge. - { type: "pause-websocket", client: 1 }, - - // Server now holds v_b = "0\nA\n". Client 1's tracked parent - // version stays at v_a = "0\n". + { + type: "assert-documents", + expected: [{ key: "note", path: "doc.md", content: "0\n" }] + }, + { type: "hold-request", client: 1, kind: "content", point: "before" }, + { type: "update", client: 1, path: "doc.md", content: "0\nB\n" }, + { type: "wait-for-request", client: 1 }, { type: "update", client: 0, path: "doc.md", content: "0\nA\n" }, { type: "sync", client: 0 }, - - // Pause the server. Subsequent HTTP PUTs from Client 1 buffer at - // the OS layer until resume. This guarantees the merge response - // for Client 1's update is still in flight when the rename below - // mutates `queue.documents`. - { type: "pause-server" }, - - // Client 1 edits doc.md with "B". The drain pops the LocalUpdate, - // captures `diskPath = "doc.md"`, reads the file, and sends the - // HTTP PUT — which buffers because the server is SIGSTOPped. - { type: "update", client: 1, path: "doc.md", content: "0\nB\n" }, - - // User renames the file while the previous PUT is still in flight. - // `queue.enqueue`'s rename branch updates `documents` to point at - // `renamed.md` synchronously, but `processLocalUpdate`'s captured - // `diskPath` ("doc.md") is a local — it can't be retargeted. { type: "rename", client: 1, oldPath: "doc.md", newPath: "renamed.md" }, - - // Resume the server. It reconciles parent=v_a, latest=v_b, - // new="0\nB\n" → v_c with both edits, replies `MergingUpdate`. - // Pre-fix: write("doc.md", …) sees no file at that path - // (renamed.md now holds the data) and bails out without ever - // writing the merged bytes. Post-fix: the merged bytes land at - // the tracked path (renamed.md). - { type: "resume-server" }, - { type: "resume-websocket", client: 1 }, - + { type: "release-request", client: 1 }, { type: "barrier" }, - { - type: "assert-consistent", - verify: (state: AssertableState): void => { - state.assertFileCount(1); - state.assertFileExists("renamed.md"); - state.assertFileNotExists("doc.md"); - // Both edits survive: Client 0's "A" and Client 1's "B". - // The reconcile may interleave them either way; assert - // both tokens are present in the converged content. - state.assertContains("renamed.md", "A", "B"); - } + type: "assert-response", + client: 1, + kind: "content", + responseType: "StaleBase", + count: 1 + }, + { + type: "assert-documents", + expected: [ + { key: "note", path: "renamed.md", content: "0\nA\nB\n" } + ] } ] }; diff --git a/frontend/deterministic-tests/src/tests/move-and-concurrent-remote-update.test.ts b/frontend/deterministic-tests/src/tests/move-and-concurrent-remote-update.test.ts index 86657f0f..026f08f0 100644 --- a/frontend/deterministic-tests/src/tests/move-and-concurrent-remote-update.test.ts +++ b/frontend/deterministic-tests/src/tests/move-and-concurrent-remote-update.test.ts @@ -34,9 +34,8 @@ export const moveAndConcurrentRemoteUpdateTest: TestDefinition = { { type: "assert-consistent", verify: (s: AssertableState): void => { - s.assertFileCount(1) - .assertFileNotExists("A.md") - .assertContains("B.md", "updated by client 1"); + s.assertFileCount(1); + s.assertContent("B.md", "updated by client 1"); } } ] diff --git a/frontend/deterministic-tests/src/tests/move-preserves-remote-update.test.ts b/frontend/deterministic-tests/src/tests/move-preserves-remote-update.test.ts index 13e27349..33df098b 100644 --- a/frontend/deterministic-tests/src/tests/move-preserves-remote-update.test.ts +++ b/frontend/deterministic-tests/src/tests/move-preserves-remote-update.test.ts @@ -35,13 +35,10 @@ export const movePreservesRemoteUpdateTest: TestDefinition = { { type: "assert-consistent", verify: (s: AssertableState): void => { - s.assertFileCount(1); - const [content] = Array.from(s.files.values()); - if (!content.includes("client 1 edit")) { - throw new Error( - `Expected merged content to include "client 1 edit", got: "${content}"` - ); - } + s.assertFileCount(1).assertContent( + "renamed.md", + "line 1\nclient 1 edit\nline 2" + ); } } ] diff --git a/frontend/deterministic-tests/src/tests/multi-file-operations.test.ts b/frontend/deterministic-tests/src/tests/multi-file-operations.test.ts index a47f5a2a..c26d65d9 100644 --- a/frontend/deterministic-tests/src/tests/multi-file-operations.test.ts +++ b/frontend/deterministic-tests/src/tests/multi-file-operations.test.ts @@ -33,12 +33,9 @@ export const multiFileOperationsTest: TestDefinition = { { type: "assert-consistent", verify: (s: AssertableState): void => { - s.assertContains("B.md", "updated") - .assertFileExists("C.md") - .assertFileNotExists("A.md"); - s.ifFileExists("D.md", (inner) => - inner.assertContent("D.md", "content-a") - ); + s.assertFileCount(2); + s.assertContent("B.md", "updated by client 1"); + s.assertContent("C.md", "content-c"); } } ] diff --git a/frontend/deterministic-tests/src/tests/offline-concurrent-renames.test.ts b/frontend/deterministic-tests/src/tests/offline-concurrent-renames.test.ts index 6c946b9c..1b2b18b4 100644 --- a/frontend/deterministic-tests/src/tests/offline-concurrent-renames.test.ts +++ b/frontend/deterministic-tests/src/tests/offline-concurrent-renames.test.ts @@ -44,15 +44,8 @@ export const offlineConcurrentRenamesTest: TestDefinition = { { type: "assert-consistent", verify: (s: AssertableState): void => { - s.assertFileNotExists("A.md") - .assertFileCount(1) - .assertAnyFileContains("shared-content"); - s.ifFileExists("B.md", (inner) => - inner.assertContent("B.md", "shared-content") - ); - s.ifFileExists("C.md", (inner) => - inner.assertContent("C.md", "shared-content") - ); + s.assertFileCount(1); + s.assertContent("B.md", "shared-content"); } } ] diff --git a/frontend/deterministic-tests/src/tests/offline-create-same-path-mergeable.test.ts b/frontend/deterministic-tests/src/tests/offline-create-same-path-mergeable.test.ts index cbd59a4a..c8d28d02 100644 --- a/frontend/deterministic-tests/src/tests/offline-create-same-path-mergeable.test.ts +++ b/frontend/deterministic-tests/src/tests/offline-create-same-path-mergeable.test.ts @@ -28,13 +28,13 @@ export const offlineCreateSamePathMergeableTest: TestDefinition = { { type: "assert-consistent", verify: (s: AssertableState): void => { - s.assertFileCount(1) - .assertFileExists("notes.md") - .assertContains( - "notes.md", - "alpha wrote this line", - "beta wrote this different line" - ); + s.assertFileCount(1).assertContentOneOf( + "notes.md", + "alpha wrote this linebeta wrote this different line", + "beta wrote this different linealpha wrote this line", + "alpha wrote this line beta wrote this different line", + "beta wrote this different line alpha wrote this line" + ); } } ] diff --git a/frontend/deterministic-tests/src/tests/offline-edit-remote-rename.test.ts b/frontend/deterministic-tests/src/tests/offline-edit-remote-rename.test.ts index ffc41b89..29d0fed3 100644 --- a/frontend/deterministic-tests/src/tests/offline-edit-remote-rename.test.ts +++ b/frontend/deterministic-tests/src/tests/offline-edit-remote-rename.test.ts @@ -40,9 +40,8 @@ export const offlineEditRemoteRenameTest: TestDefinition = { { type: "assert-consistent", verify: (s: AssertableState): void => { - s.assertFileNotExists("A.md") - .assertFileCount(1) - .assertContains("B.md", "edited by client 0"); + s.assertFileCount(1); + s.assertContent("B.md", "edited by client 0"); } } ] diff --git a/frontend/deterministic-tests/src/tests/offline-rename-remote-create-old-path.test.ts b/frontend/deterministic-tests/src/tests/offline-rename-remote-create-old-path.test.ts index c1b2913a..f2273df5 100644 --- a/frontend/deterministic-tests/src/tests/offline-rename-remote-create-old-path.test.ts +++ b/frontend/deterministic-tests/src/tests/offline-rename-remote-create-old-path.test.ts @@ -41,10 +41,8 @@ export const offlineRenameRemoteCreateOldPathTest: TestDefinition = { { type: "assert-consistent", verify: (s: AssertableState): void => { - s.assertFileCount(1).assertContains( - "Y.md", - "updated-by-client-1" - ); + s.assertFileCount(1); + s.assertContent("Y.md", "updated-by-client-1"); } } ] diff --git a/frontend/deterministic-tests/src/tests/online-both-create-same-path-deconflict.test.ts b/frontend/deterministic-tests/src/tests/online-both-create-same-path-deconflict.test.ts index 3e2ebf51..9b29b778 100644 --- a/frontend/deterministic-tests/src/tests/online-both-create-same-path-deconflict.test.ts +++ b/frontend/deterministic-tests/src/tests/online-both-create-same-path-deconflict.test.ts @@ -12,22 +12,24 @@ export const onlineBothCreateSamePathDeconflictTest: TestDefinition = { { type: "enable-sync", client: 1 }, { type: "barrier" }, - { type: "pause-websocket", client: 1 }, + { type: "pause-observation", client: 1 }, { type: "create", client: 0, path: "A.md", content: " from-client-0 " }, { type: "update", client: 0, path: "A.md", content: " updated-by-0 " }, { type: "sync" }, { type: "create", client: 1, path: "A.md", content: " from-client-1 " }, - { type: "resume-websocket", client: 1 }, + { type: "wait-for-observation", client: 1 }, + { type: "sync", client: 0 }, + { type: "resume-observation", client: 1 }, { type: "barrier" }, { type: "assert-consistent", - verify: (state: AssertableState): void => { - state - .assertFileCount(2) - .assertAnyFileContains("updated-by-0", "from-client-1 "); + verify: (s: AssertableState): void => { + s.assertFileCount(2) + .assertContent("A.md", " updated-by-0 ") + .assertContent(s.conflictPath("A.md"), " from-client-1 "); } } ] diff --git a/frontend/deterministic-tests/src/tests/online-create-update-while-other-creates-same-path.test.ts b/frontend/deterministic-tests/src/tests/online-create-update-while-other-creates-same-path.test.ts index e0ddc21a..d315ba22 100644 --- a/frontend/deterministic-tests/src/tests/online-create-update-while-other-creates-same-path.test.ts +++ b/frontend/deterministic-tests/src/tests/online-create-update-while-other-creates-same-path.test.ts @@ -11,7 +11,7 @@ export const onlineCreateUpdateWhileOtherCreatesSamePathTest: TestDefinition = { { type: "enable-sync", client: 0 }, { type: "enable-sync", client: 1 }, - { type: "pause-websocket", client: 1 }, + { type: "pause-observation", client: 1 }, { type: "create", client: 0, @@ -30,18 +30,21 @@ export const onlineCreateUpdateWhileOtherCreatesSamePathTest: TestDefinition = { path: "data.bin", content: "BINARY:other-content" }, - { type: "resume-websocket", client: 1 }, + { type: "wait-for-observation", client: 1 }, + { type: "sync", client: 0 }, + { type: "resume-observation", client: 1 }, { type: "barrier" }, { type: "assert-consistent", - verify: (state: AssertableState): void => { - state - .assertFileCount(2) - .assertNoFileContains("content-v1") - .assertAnyFileContains("content-v2") - .assertAnyFileContains("other-content"); + verify: (s: AssertableState): void => { + s.assertFileCount(2) + .assertContent("data.bin", "BINARY:content-v2") + .assertContent( + s.conflictPath("data.bin"), + "BINARY:other-content" + ); } } ] diff --git a/frontend/deterministic-tests/src/tests/overlapping-edits-same-section.test.ts b/frontend/deterministic-tests/src/tests/overlapping-edits-same-section.test.ts index a93a6f69..ed37240b 100644 --- a/frontend/deterministic-tests/src/tests/overlapping-edits-same-section.test.ts +++ b/frontend/deterministic-tests/src/tests/overlapping-edits-same-section.test.ts @@ -41,12 +41,10 @@ export const overlappingEditsSameSectionTest: TestDefinition = { { type: "assert-consistent", verify: (s: AssertableState): void => { - s.assertFileCount(1).assertContains( + s.assertFileCount(1); + s.assertContent( "doc.md", - "# Title", - "alpha addition", - "beta addition", - "footer" + "# Title\nalpha addition\n\nbeta addition\nfooter" ); } } diff --git a/frontend/deterministic-tests/src/tests/queue-reset-loses-coalesced-local-edit.test.ts b/frontend/deterministic-tests/src/tests/queue-reset-loses-coalesced-local-edit.test.ts index 6d89acf4..b8608f30 100644 --- a/frontend/deterministic-tests/src/tests/queue-reset-loses-coalesced-local-edit.test.ts +++ b/frontend/deterministic-tests/src/tests/queue-reset-loses-coalesced-local-edit.test.ts @@ -25,10 +25,12 @@ export const queueResetLosesCoalescedLocalEditTest: TestDefinition = { { type: "assert-consistent", verify: (s: AssertableState): void => { - s.assertFileCount(1).assertContains( + s.assertFileCount(1).assertContentOneOf( "doc.md", - "alpha", - "charlie" + "alpha bravocharlie delta", + "charlie deltaalpha bravo", + "alpha bravo charlie delta", + "charlie delta alpha bravo" ); } } diff --git a/frontend/deterministic-tests/src/tests/queued-create-delete-does-not-hijack-reused-path.test.ts b/frontend/deterministic-tests/src/tests/queued-create-delete-does-not-hijack-reused-path.test.ts index a29f8314..db0dcf80 100644 --- a/frontend/deterministic-tests/src/tests/queued-create-delete-does-not-hijack-reused-path.test.ts +++ b/frontend/deterministic-tests/src/tests/queued-create-delete-does-not-hijack-reused-path.test.ts @@ -12,14 +12,14 @@ export const queuedCreateDeleteDoesNotHijackReusedPathTest: TestDefinition = { { type: "enable-sync", client: 1 }, { type: "barrier" }, - { type: "pause-server" }, + { type: "hold-request", client: 1, kind: "create", point: "before" }, { type: "create", client: 1, path: "blocker.bin", content: "BINARY:blocker" }, - { type: "sleep", ms: 100 }, + { type: "wait-for-request", client: 1 }, { type: "create", client: 1, @@ -39,7 +39,7 @@ export const queuedCreateDeleteDoesNotHijackReusedPathTest: TestDefinition = { oldPath: "source.bin", newPath: "target.bin" }, - { type: "resume-server" }, + { type: "release-request", client: 1 }, { type: "barrier" }, { diff --git a/frontend/deterministic-tests/src/tests/rapid-create-update-delete-cycle.test.ts b/frontend/deterministic-tests/src/tests/rapid-create-update-delete-cycle.test.ts index f9c58753..814edb01 100644 --- a/frontend/deterministic-tests/src/tests/rapid-create-update-delete-cycle.test.ts +++ b/frontend/deterministic-tests/src/tests/rapid-create-update-delete-cycle.test.ts @@ -3,15 +3,15 @@ import type { TestDefinition } from "../test-definition"; export const rapidCreateUpdateDeleteCycleTest: TestDefinition = { description: - "Client 0 rapidly creates, updates, deletes, then re-creates a file while the server is paused. " + - "After the server resumes, client 1 must see only the final file.", + "Client 0 rapidly creates, updates, deletes, then re-creates a file while its create request is held. " + + "After releasing the request, client 1 must see only the final file.", clients: 2, steps: [ { type: "enable-sync", client: 0 }, { type: "enable-sync", client: 1 }, { type: "barrier" }, - { type: "pause-server" }, + { type: "hold-request", client: 0, kind: "create", point: "before" }, { type: "create", @@ -19,6 +19,7 @@ export const rapidCreateUpdateDeleteCycleTest: TestDefinition = { path: "cycle.md", content: "version 1" }, + { type: "wait-for-request", client: 0 }, { type: "update", client: 0, @@ -27,7 +28,7 @@ export const rapidCreateUpdateDeleteCycleTest: TestDefinition = { }, { type: "delete", client: 0, path: "cycle.md" }, - { type: "resume-server" }, + { type: "release-request", client: 0 }, { type: "sync" }, { diff --git a/frontend/deterministic-tests/src/tests/rapid-edit-delete-online-convergence.test.ts b/frontend/deterministic-tests/src/tests/rapid-edit-delete-online-convergence.test.ts index 48c062e0..f2915cf0 100644 --- a/frontend/deterministic-tests/src/tests/rapid-edit-delete-online-convergence.test.ts +++ b/frontend/deterministic-tests/src/tests/rapid-edit-delete-online-convergence.test.ts @@ -16,7 +16,9 @@ export const rapidEditDeleteOnlineConvergenceTest: TestDefinition = { { type: "enable-sync", client: 1 }, { type: "barrier" }, + { type: "hold-request", client: 0, kind: "content", point: "before" }, { type: "update", client: 0, path: "A.md", content: "A edit 1" }, + { type: "wait-for-request", client: 0 }, { type: "update", client: 0, path: "B.md", content: "B edit 1" }, { type: "update", client: 0, path: "C.md", content: "C edit 1" }, { type: "delete", client: 1, path: "A.md" }, @@ -26,22 +28,15 @@ export const rapidEditDeleteOnlineConvergenceTest: TestDefinition = { { type: "update", client: 0, path: "B.md", content: "B edit 2" }, { type: "update", client: 0, path: "C.md", content: "C edit 2" }, + { type: "sync", client: 1 }, + { type: "release-request", client: 0 }, { type: "barrier" }, { type: "assert-consistent", verify: (s: AssertableState): void => { - for (const [path, content] of s.files) { - for (const clientFiles of s.clientFiles) { - if ( - clientFiles.has(path) && - clientFiles.get(path) !== content - ) { - throw new Error( - `Content mismatch for ${path}: "${clientFiles.get(path)}" vs "${content}"` - ); - } - } - } + s.assertFileCount(2) + .assertContent("B.md", "B edit 2") + .assertContent("D.md", "content D"); } } ] diff --git a/frontend/deterministic-tests/src/tests/rapid-updates-after-merge.test.ts b/frontend/deterministic-tests/src/tests/rapid-updates-after-merge.test.ts index 6f97ff05..624438ab 100644 --- a/frontend/deterministic-tests/src/tests/rapid-updates-after-merge.test.ts +++ b/frontend/deterministic-tests/src/tests/rapid-updates-after-merge.test.ts @@ -42,7 +42,8 @@ export const rapidUpdatesAfterMergeTest: TestDefinition = { { type: "assert-consistent", verify: (s: AssertableState): void => { - s.assertFileCount(1).assertContains("doc.md", "update 3"); + s.assertFileCount(1); + s.assertContent("doc.md", "update 3"); } } ] diff --git a/frontend/deterministic-tests/src/tests/remote-rename-collides-with-pending-local-create.test.ts b/frontend/deterministic-tests/src/tests/remote-rename-collides-with-pending-local-create.test.ts index d30fdc67..b11290a9 100644 --- a/frontend/deterministic-tests/src/tests/remote-rename-collides-with-pending-local-create.test.ts +++ b/frontend/deterministic-tests/src/tests/remote-rename-collides-with-pending-local-create.test.ts @@ -30,7 +30,7 @@ export const remoteRenameCollidesWithPendingLocalCreateTest: TestDefinition = { // Pause client 0's WS so the upcoming remote rename buffers and // we can stage a colliding local create before the rename // drains on client 0. - { type: "pause-websocket", client: 0 }, + { type: "pause-observation", client: 0 }, // Client 1 renames the doc. Server commits, broadcasts to // client 0 (buffered). @@ -51,7 +51,8 @@ export const remoteRenameCollidesWithPendingLocalCreateTest: TestDefinition = { // Resume client 0's WS. The buffered RemoteChange drains. // The reconciler must converge without ever leaving a // conflict-uuid stash on disk. - { type: "resume-websocket", client: 0 }, + { type: "wait-for-observation", client: 0 }, + { type: "resume-observation", client: 0 }, { type: "barrier" }, @@ -68,6 +69,7 @@ export const remoteRenameCollidesWithPendingLocalCreateTest: TestDefinition = { } state.assertFileExists("target.md"); state.assertContent("target.md", "v1\n"); + state.assertContent(state.conflictPath("target.md"), "extra\n"); // The local create gets server-deconflicted to a // sibling path (e.g. `target (1).md`). } diff --git a/frontend/deterministic-tests/src/tests/remote-update-resurrects-deleted-doc.test.ts b/frontend/deterministic-tests/src/tests/remote-update-resurrects-deleted-doc.test.ts index eb2ed86d..386e3644 100644 --- a/frontend/deterministic-tests/src/tests/remote-update-resurrects-deleted-doc.test.ts +++ b/frontend/deterministic-tests/src/tests/remote-update-resurrects-deleted-doc.test.ts @@ -1,14 +1,8 @@ -import type { AssertableState } from "../utils/assertable-state"; import type { TestDefinition } from "../test-definition"; export const remoteUpdateResurrectsDeletedDocTest: TestDefinition = { description: - "Client 1 updates, deletes, and recreates P (with a new docId D2). " + - "While the buffered remote events are being processed by client 0, " + - "client 0 also makes a local edit to P. The local edit lands in the " + - "queue while v17 is mid-process, sending v17 down processRemoteUpdate's " + - "re-enqueue branch. The deferred v17 must NOT later resurrect D1 as a " + - "conflict-… file at P after the delete and the D2 create have drained.", + "Hold a remote content download while the peer deletes and recreates its path. A local edit to the old generation during this download must not resurrect it or overwrite the new identity.", clients: 2, steps: [ { type: "enable-sync", client: 0 }, @@ -17,7 +11,13 @@ export const remoteUpdateResurrectsDeletedDocTest: TestDefinition = { { type: "create", client: 1, path: "P.md", content: "v8 content\n" }, { type: "barrier" }, - { type: "pause-websocket", client: 0 }, + { type: "remember-identity", path: "P.md", key: "original" }, + { + type: "hold-request", + client: 0, + kind: "read-content", + point: "after" + }, { type: "update", @@ -26,6 +26,7 @@ export const remoteUpdateResurrectsDeletedDocTest: TestDefinition = { content: "v17 content from client 1\n" }, { type: "sync", client: 1 }, + { type: "wait-for-request", client: 0 }, { type: "delete", client: 1, path: "P.md" }, { type: "sync", client: 1 }, { @@ -36,7 +37,12 @@ export const remoteUpdateResurrectsDeletedDocTest: TestDefinition = { }, { type: "sync", client: 1 }, - { type: "resume-websocket", client: 0 }, + { + type: "remember-local-identity", + client: 1, + path: "P.md", + key: "replacement" + }, { type: "update", @@ -45,15 +51,18 @@ export const remoteUpdateResurrectsDeletedDocTest: TestDefinition = { content: "local edit by client 0\n" }, + { type: "release-request", client: 0 }, { type: "barrier" }, { - type: "assert-consistent", - verify: (state: AssertableState): void => { - state - .assertFileCount(1) - .assertContent("P.md", "v21 content (D2)\n"); - } + type: "assert-documents", + expected: [ + { + key: "replacement", + path: "P.md", + content: "v21 content (D2)\n" + } + ] } ] }; diff --git a/frontend/deterministic-tests/src/tests/remote-update-survives-user-rename.test.ts b/frontend/deterministic-tests/src/tests/remote-update-survives-user-rename.test.ts index b78ad143..86340f80 100644 --- a/frontend/deterministic-tests/src/tests/remote-update-survives-user-rename.test.ts +++ b/frontend/deterministic-tests/src/tests/remote-update-survives-user-rename.test.ts @@ -1,84 +1,32 @@ -import type { AssertableState } from "../utils/assertable-state"; import type { TestDefinition } from "../test-definition"; export const remoteUpdateSurvivesUserRenameTest: TestDefinition = { description: - "Client 0 updates a tracked doc; while Client 1 is processing the " + - "broadcast and parked on the GET for the new version's content, the " + - "user renames the doc on Client 1. Pre-fix: `processRemoteUpdate` " + - "captures `actualPath` before the await and, after the GET returns, " + - "calls `write(actualPath, …)` (no-op — file was renamed away), " + - "`updateCache(actualPath, …)`, and `setDocument(actualPath, …)`. " + - "`setDocument` mutates the same record in place so its `path` is " + - "yanked from the user's renamed slot back to the pre-rename path, " + - "wiping the rename out of the queue's documents map. The queued " + - "`LocalUpdate` then reads from the now-stale `record.path`, hits " + - "`FileNotFoundError`, and is silently dropped — the user's rename " + - "never reaches the server. Post-fix: the handler defers when a " + - "local event landed mid-await, so the rename drains first and " + - "the deferred remote update is folded into the broadcast that " + - "follows the rename round-trip.", + "An observed remote content download is held while the user renames its destination.", clients: 2, steps: [ { type: "create", client: 0, path: "doc.md", content: "v1\n" }, { type: "enable-sync", client: 0 }, { type: "enable-sync", client: 1 }, { type: "barrier" }, - - // Buffer Client 1's incoming broadcasts so it doesn't see - // Client 0's update until we've paused the server. - { type: "pause-websocket", client: 1 }, - - // Server now holds v=2 of doc.md. - { type: "update", client: 0, path: "doc.md", content: "v2\n" }, - { type: "sync", client: 0 }, - - // Pause the server. Client 1's upcoming GET for the new version - // content blocks at the OS layer until resume. - { type: "pause-server" }, - - // Release the buffered broadcast. Client 1's drain enters - // `processRemoteUpdate`, captures `actualPath`, fires the GET, - // and parks awaiting the response. - { type: "resume-websocket", client: 1 }, - - // Yield long enough for the drain to traverse all microtask - // hops between the WS handler and the GET, so the HTTP request - // is queued at the (paused) server before the rename runs. - // Without this yield the rename would be enqueued before - // `processRemoteUpdate`'s entry-time `hasPendingLocalEvents` - // check and the early-defer branch would mask the bug. - { type: "sleep", ms: 50 }, - - // While the GET is in flight the user renames the doc. The queue - // mutates `record.path` to "renamed.md" in place and pushes a - // LocalUpdate carrying the rename target. { - type: "rename", - client: 1, - oldPath: "doc.md", - newPath: "renamed.md" + type: "assert-documents", + expected: [{ key: "note", path: "doc.md", content: "v1\n" }] }, - - // Resume the server. The GET response unblocks - // `processRemoteUpdate`. With the fix in place it sees the - // queued LocalUpdate and defers; without the fix it walks past - // the rename and clobbers the documents map, dropping the - // pending LocalUpdate's read on the way back through. - { type: "resume-server" }, - - { type: "barrier" }, - { - type: "assert-consistent", - verify: (s: AssertableState): void => { - s.assertFileCount(1); - s.assertFileExists("renamed.md"); - s.assertFileNotExists("doc.md"); - // Both edits survive: the user's rename and Client 0's - // content update at v=2. - s.assertContent("renamed.md", "v2\n"); - } + type: "hold-request", + client: 1, + kind: "read-content", + point: "after" + }, + { type: "update", client: 0, path: "doc.md", content: "v2\n" }, + { type: "wait-for-request", client: 1 }, + { type: "rename", client: 1, oldPath: "doc.md", newPath: "renamed.md" }, + { type: "release-request", client: 1 }, + { type: "barrier" }, + { + type: "assert-documents", + expected: [{ key: "note", path: "renamed.md", content: "v2\n" }] } ] }; diff --git a/frontend/deterministic-tests/src/tests/rename-chain-during-pending-create.test.ts b/frontend/deterministic-tests/src/tests/rename-chain-during-pending-create.test.ts index 822e83df..ffbbbed9 100644 --- a/frontend/deterministic-tests/src/tests/rename-chain-during-pending-create.test.ts +++ b/frontend/deterministic-tests/src/tests/rename-chain-during-pending-create.test.ts @@ -3,33 +3,17 @@ import type { TestDefinition } from "../test-definition"; export const renameChainDuringPendingCreateTest: TestDefinition = { description: - "User creates a doc, then renames it twice while the LocalCreate's " + - "HTTP roundtrip is still in flight (server paused). Each rename " + - "pushes a LocalUpdate whose `documentId` is the create's Promise " + - "(see `pendingDocumentId` in `SyncEventQueue.enqueue`). After the " + - "create resolves, the first rename drains successfully and " + - "`setDocument` walks `events[]` to retarget queued LocalUpdates' " + - "`event.path` to the new disk location — but the comparison " + - "`e.documentId === record.documentId` mismatches the still-Promise " + - "references, so the second rename's `event.path` stays at the " + - "vacated previous slot. On the next drain step `skipIfOversized`'s " + - "`getFileSize(event.path)` throws FileNotFoundError, which " + - "`processEvent` swallows as 'Skipping sync event ... because the " + - "file no longer exists' — losing the user's final rename. " + - "Post-fix: `resolveCreate` (and the displacement-merge branch in " + - "`processCreate`) swap the Promise references for the resolved id " + - "before `setDocument` runs, so retarget works.", + "Hold a create before sending it, then rename its document twice. Both clients must retain the original bytes at the final path.", clients: 2, steps: [ { type: "enable-sync", client: 0 }, { type: "enable-sync", client: 1 }, { type: "barrier" }, - // Pause the server so client 0's create stalls on the HTTP PUT - // while we queue rename events behind it. - { type: "pause-server" }, + { type: "hold-request", client: 0, kind: "create", point: "before" }, { type: "create", client: 0, path: "first.md", content: "v1\n" }, + { type: "wait-for-request", client: 0 }, { type: "rename", client: 0, @@ -43,12 +27,7 @@ export const renameChainDuringPendingCreateTest: TestDefinition = { newPath: "third.md" }, - // Resume — drain pops LocalCreate (now resolves), then the two - // queued LocalUpdates. Pre-fix: only the first rename's - // file-system effect lands; the second is silently dropped. - // The server ends up with the doc at second.md, leaving - // client 0's local third.md untracked / out-of-sync. - { type: "resume-server" }, + { type: "release-request", client: 0 }, { type: "barrier" }, diff --git a/frontend/deterministic-tests/src/tests/rename-overwrites-pending-create-then-delete.test.ts b/frontend/deterministic-tests/src/tests/rename-overwrites-pending-create-then-delete.test.ts index 0b47c781..3d7c7bda 100644 --- a/frontend/deterministic-tests/src/tests/rename-overwrites-pending-create-then-delete.test.ts +++ b/frontend/deterministic-tests/src/tests/rename-overwrites-pending-create-then-delete.test.ts @@ -16,7 +16,7 @@ export const renameOverwritesPendingCreateThenDeleteTest: TestDefinition = { }, { type: "barrier" }, - { type: "pause-server" }, + { type: "hold-request", client: 0, kind: "create", point: "before" }, { type: "create", @@ -24,6 +24,7 @@ export const renameOverwritesPendingCreateThenDeleteTest: TestDefinition = { path: "pending.bin", content: "BINARY:pending" }, + { type: "wait-for-request", client: 0 }, { type: "rename", client: 0, @@ -38,7 +39,7 @@ export const renameOverwritesPendingCreateThenDeleteTest: TestDefinition = { }, { type: "delete", client: 0, path: "final.bin" }, - { type: "resume-server" }, + { type: "release-request", client: 0 }, { type: "barrier" }, { diff --git a/frontend/deterministic-tests/src/tests/rename-pending-create-before-response.test.ts b/frontend/deterministic-tests/src/tests/rename-pending-create-before-response.test.ts index 26623c43..fa532ac2 100644 --- a/frontend/deterministic-tests/src/tests/rename-pending-create-before-response.test.ts +++ b/frontend/deterministic-tests/src/tests/rename-pending-create-before-response.test.ts @@ -3,13 +3,13 @@ import type { TestDefinition } from "../test-definition"; export const renamePendingCreateBeforeResponseTest: TestDefinition = { description: - "Client 0 creates a file while the server is paused, then renames it before the create completes. After the server resumes, both clients should converge with the file at the renamed path.", + "Client 0 creates a file while its create request is held, then renames it before the create completes. After releasing the request, both clients should converge with the file at the renamed path.", clients: 2, steps: [ { type: "enable-sync", client: 0 }, { type: "enable-sync", client: 1 }, - { type: "pause-server" }, + { type: "hold-request", client: 0, kind: "create", point: "before" }, { type: "create", @@ -17,6 +17,7 @@ export const renamePendingCreateBeforeResponseTest: TestDefinition = { path: "doc.md", content: "original-content" }, + { type: "wait-for-request", client: 0 }, { type: "rename", @@ -25,7 +26,7 @@ export const renamePendingCreateBeforeResponseTest: TestDefinition = { newPath: "renamed.md" }, - { type: "resume-server" }, + { type: "release-request", client: 0 }, { type: "barrier" }, diff --git a/frontend/deterministic-tests/src/tests/rename-pending-create-onto-pending-delete-path.test.ts b/frontend/deterministic-tests/src/tests/rename-pending-create-onto-pending-delete-path.test.ts index 0906f209..251776fa 100644 --- a/frontend/deterministic-tests/src/tests/rename-pending-create-onto-pending-delete-path.test.ts +++ b/frontend/deterministic-tests/src/tests/rename-pending-create-onto-pending-delete-path.test.ts @@ -21,14 +21,14 @@ export const renamePendingCreateOntoPendingDeletePathTest: TestDefinition = { }, { type: "barrier" }, - { type: "pause-server" }, + { type: "hold-request", client: 1, kind: "create", point: "before" }, { type: "create", client: 1, path: "blocker.md", content: "blocker\n" }, - { type: "sleep", ms: 100 }, + { type: "wait-for-request", client: 1 }, { type: "create", client: 1, @@ -42,7 +42,7 @@ export const renamePendingCreateOntoPendingDeletePathTest: TestDefinition = { oldPath: "file-23.md", newPath: "file-17.md" }, - { type: "resume-server" }, + { type: "release-request", client: 1 }, { type: "barrier" }, { diff --git a/frontend/deterministic-tests/src/tests/rename-to-path-of-unconfirmed-delete.test.ts b/frontend/deterministic-tests/src/tests/rename-to-path-of-unconfirmed-delete.test.ts index 34a3867c..c79af51b 100644 --- a/frontend/deterministic-tests/src/tests/rename-to-path-of-unconfirmed-delete.test.ts +++ b/frontend/deterministic-tests/src/tests/rename-to-path-of-unconfirmed-delete.test.ts @@ -34,10 +34,8 @@ export const renameToPathOfUnconfirmedDeleteTest: TestDefinition = { { type: "assert-consistent", verify: (s: AssertableState): void => { - s.assertFileNotExists("B.md").assertContains( - "A.md", - "content B" - ); + s.assertFileCount(1); + s.assertContent("A.md", "content B"); } } ] diff --git a/frontend/deterministic-tests/src/tests/rename-to-pending-path-fallback.test.ts b/frontend/deterministic-tests/src/tests/rename-to-pending-path-fallback.test.ts index 8747218a..7e90a07e 100644 --- a/frontend/deterministic-tests/src/tests/rename-to-pending-path-fallback.test.ts +++ b/frontend/deterministic-tests/src/tests/rename-to-pending-path-fallback.test.ts @@ -33,10 +33,8 @@ export const renameToPendingPathFallbackTest: TestDefinition = { { type: "assert-consistent", verify: (s: AssertableState): void => { - s.assertFileNotExists("B.md").assertContains( - "A.md", - "tracked B content" - ); + s.assertFileCount(1); + s.assertContent("A.md", "tracked B content"); } } ] diff --git a/frontend/deterministic-tests/src/tests/rename-update-conflict.test.ts b/frontend/deterministic-tests/src/tests/rename-update-conflict.test.ts index 18d4c101..db3f1c39 100644 --- a/frontend/deterministic-tests/src/tests/rename-update-conflict.test.ts +++ b/frontend/deterministic-tests/src/tests/rename-update-conflict.test.ts @@ -35,7 +35,8 @@ export const renameUpdateConflictTest: TestDefinition = { { type: "assert-consistent", verify: (s: AssertableState): void => { - s.assertFileNotExists("A.md").assertContains("B.md", "updated"); + s.assertFileCount(1); + s.assertContent("B.md", "updated by client 1"); } } ] diff --git a/frontend/deterministic-tests/src/tests/renamed-pending-create-reused-path-then-delete.test.ts b/frontend/deterministic-tests/src/tests/renamed-pending-create-reused-path-then-delete.test.ts index 3ffb376e..f92d5277 100644 --- a/frontend/deterministic-tests/src/tests/renamed-pending-create-reused-path-then-delete.test.ts +++ b/frontend/deterministic-tests/src/tests/renamed-pending-create-reused-path-then-delete.test.ts @@ -13,14 +13,14 @@ export const renamedPendingCreateReusedPathThenDeleteTest: TestDefinition = { { type: "enable-sync", client: 1 }, { type: "barrier" }, - { type: "pause-server" }, + { type: "hold-request", client: 1, kind: "create", point: "before" }, { type: "create", client: 1, path: "blocker.md", content: "blocker\n" }, - { type: "sleep", ms: 100 }, + { type: "wait-for-request", client: 1 }, { type: "create", @@ -41,14 +41,8 @@ export const renamedPendingCreateReusedPathThenDeleteTest: TestDefinition = { content: "new\n" }, - { - type: "resume-server-until-history-then-pause", - client: 1, - syncType: "CREATE", - path: "file-33.md" - }, { type: "delete", client: 1, path: "file-33.md" }, - { type: "resume-server" }, + { type: "release-request", client: 1 }, { type: "barrier" }, { diff --git a/frontend/deterministic-tests/src/tests/reset-clears-recently-deleted-resurrection.test.ts b/frontend/deterministic-tests/src/tests/reset-clears-recently-deleted-resurrection.test.ts index e0a1565c..b4da66f7 100644 --- a/frontend/deterministic-tests/src/tests/reset-clears-recently-deleted-resurrection.test.ts +++ b/frontend/deterministic-tests/src/tests/reset-clears-recently-deleted-resurrection.test.ts @@ -24,7 +24,7 @@ export const resetClearsRecentlyDeletedResurrectionTest: TestDefinition = { { type: "assert-consistent", verify: (s: AssertableState): void => { - s.assertFileNotExists("ghost.md"); + s.assertFileCount(0).assertFileNotExists("ghost.md"); } }, diff --git a/frontend/deterministic-tests/src/tests/same-doc-id-collapse-after-remote-quick-write-and-pending-rename.test.ts b/frontend/deterministic-tests/src/tests/same-doc-id-collapse-after-remote-quick-write-and-pending-rename.test.ts index e78c2310..930129ae 100644 --- a/frontend/deterministic-tests/src/tests/same-doc-id-collapse-after-remote-quick-write-and-pending-rename.test.ts +++ b/frontend/deterministic-tests/src/tests/same-doc-id-collapse-after-remote-quick-write-and-pending-rename.test.ts @@ -13,8 +13,6 @@ export const sameDocIdCollapseAfterRemoteQuickWriteAndPendingRenameTest: TestDef steps: [ { type: "enable-sync", client: 0 }, - // Seed a deleted document before client 1 bootstraps. Later creates - // at the reused path must still receive separate identities. { type: "create", client: 0, @@ -35,7 +33,12 @@ export const sameDocIdCollapseAfterRemoteQuickWriteAndPendingRenameTest: TestDef { type: "enable-sync", client: 1 }, { type: "sync", client: 1 }, - { type: "pause-websocket", client: 1 }, + { + type: "hold-request", + client: 1, + kind: "read-content", + point: "after" + }, { type: "create", @@ -45,12 +48,7 @@ export const sameDocIdCollapseAfterRemoteQuickWriteAndPendingRenameTest: TestDef }, { type: "sync", client: 0 }, - // Let client 1's buffered RemoteCreate enter the quick-write - // path, but hold the content fetch until the local create has - // appeared and moved away from doc.md. - { type: "pause-server" }, - { type: "resume-websocket", client: 1 }, - { type: "sleep", ms: 100 }, + { type: "wait-for-request", client: 1 }, { type: "create", @@ -65,7 +63,7 @@ export const sameDocIdCollapseAfterRemoteQuickWriteAndPendingRenameTest: TestDef newPath: "renamed.md" }, - { type: "resume-server" }, + { type: "release-request", client: 1 }, { type: "barrier" }, { diff --git a/frontend/deterministic-tests/src/tests/self-merge-pending-rename-aliases-second-create.test.ts b/frontend/deterministic-tests/src/tests/self-merge-pending-rename-aliases-second-create.test.ts index e3504098..3dd9040e 100644 --- a/frontend/deterministic-tests/src/tests/self-merge-pending-rename-aliases-second-create.test.ts +++ b/frontend/deterministic-tests/src/tests/self-merge-pending-rename-aliases-second-create.test.ts @@ -3,44 +3,18 @@ import type { TestDefinition } from "../test-definition"; export const selfMergePendingRenameAliasesSecondCreateTest: TestDefinition = { description: - "Single client makes two distinct creates that briefly share a path. " + - "Client 0 POSTs the first create at primary.md while the server is " + - "paused. While that POST is in flight: a second create is queued at " + - "staging.md, primary.md is renamed to moved.md (rewriting the in- " + - "flight create's event.path to moved.md and pushing a rename " + - "LocalUpdate at the queue tail), and staging.md is renamed onto the " + - "now-vacated primary.md slot (rewriting the second create's " + - "event.path to primary.md and pushing another rename LocalUpdate). " + - "Client 0's WS is paused throughout, so its watermark stays at 0. " + - "On resume the first POST commits Doc-X at primary.md (creation_vuid " + - "= N). The drain then processes the second LocalCreate (POST " + - "relativePath=primary.md, last_seen=0); the server's path-based " + - "dedup sees N > 0 and merges the second create into Doc-X " + - "(MergingUpdate). The buggy behaviour: processCreate's resolveCreate " + - "calls upsertRecord with localPath=primary.md, but the existing " + - "record (from the first create) already holds localPath=moved.md, " + - "and upsertRecord's `existing.localPath !== undefined` guard " + - "silently drops the new claim. The file at primary.md is left " + - "orphaned: tracked by no record, never broadcast, never deleted. " + - "After the user's renames the expected user-visible state is two " + - "distinct files at moved.md and primary.md — both clients must " + - "converge to that.", + "Hold the first create before sending it, then create a second file " + + "and move both generations through primary.md. Releasing the request " + + "must preserve distinct documents at moved.md and primary.md, with " + + "each generation's exact content and the unrelated remote file intact.", clients: 2, steps: [ - // Both clients online so the WS connection is established before - // the test starts pausing things. { type: "enable-sync", client: 0 }, { type: "enable-sync", client: 1 }, { type: "barrier" }, - // Delay socket notifications while client 0 continues local work. - // HTTP catchup must preserve the two generations at the reused path. { type: "pause-websocket", client: 0 }, - // Client 1 commits a doc to push the server's vuid above 0. - // Without this filler, Doc-X's create vuid could be 1 and - // client 0's last_seen.add(1) would advance min to 1, killing - // the watermark gap that triggers the merge. { type: "create", client: 1, @@ -49,14 +23,8 @@ export const selfMergePendingRenameAliasesSecondCreateTest: TestDefinition = { }, { type: "sync", client: 1 }, - // Pause the server so client 0's first create POST hangs in - // flight, giving us a deterministic window in which to enqueue - // the second create and the renames. - { type: "pause-server" }, + { type: "hold-request", client: 0, kind: "create", point: "before" }, - // First create — Doc-X. The wire-loop drains it, captures - // requestPath = event.path = "primary.md", reads the bytes, - // sends the POST, and stalls on the response. { type: "create", client: 0, @@ -64,16 +32,8 @@ export const selfMergePendingRenameAliasesSecondCreateTest: TestDefinition = { content: "primary content " }, - // Make sure the POST is actually on the wire with - // relativePath="primary.md" before we rewrite event.path. - // Without this delay the rename can win the race, the POST - // goes out with relativePath="moved.md", and the server-side - // path-collision merge never fires. - { type: "sleep", ms: 100 }, + { type: "wait-for-request", client: 0 }, - // Second create at a staging path. The wire-loop is still - // blocked on Doc-X's POST, so this LocalCreate just queues at - // index 1. { type: "create", client: 0, @@ -81,15 +41,6 @@ export const selfMergePendingRenameAliasesSecondCreateTest: TestDefinition = { content: "secondary content " }, - // Rename Doc-X's path. enqueue's pending-create branch - // rewrites Doc-X's event.path in place (moved.md) and pushes - // a LocalUpdate(rename, originalPath=moved.md) at the END of - // the queue. Note the ordering: this LocalUpdate is enqueued - // AFTER the staging LocalCreate above. That ordering is - // load-bearing — it is what makes the second create's POST - // drain (and trigger the server-side merge) before Doc-X's - // rename PUT moves the doc away from primary.md on the - // server. { type: "rename", client: 0, @@ -97,12 +48,6 @@ export const selfMergePendingRenameAliasesSecondCreateTest: TestDefinition = { newPath: "moved.md" }, - // Rename the staging file onto Doc-X's now-vacated primary.md - // slot. enqueue rewrites the staging LocalCreate's event.path - // to primary.md and pushes a LocalUpdate(rename, - // originalPath=primary.md) at the queue tail. After this the - // disk has: moved.md = Doc-X's bytes, primary.md = Doc-Y's - // bytes. { type: "rename", client: 0, @@ -110,9 +55,7 @@ export const selfMergePendingRenameAliasesSecondCreateTest: TestDefinition = { newPath: "primary.md" }, - // Let everything fly: server processes the queued POSTs; - // client 0 catches up on broadcasts. - { type: "resume-server" }, + { type: "release-request", client: 0 }, { type: "resume-websocket", client: 0 }, { type: "barrier" }, @@ -120,27 +63,14 @@ export const selfMergePendingRenameAliasesSecondCreateTest: TestDefinition = { { type: "assert-consistent", verify: (state: AssertableState): void => { - // The user did two distinct creates (Doc-X and Doc-Y); - // both contents must survive on both clients. state.assertFileCount(3); - state.assertFileExists("filler.md"); + state.assertContent("filler.md", "filler-content "); state.assertFileExists("moved.md"); state.assertFileExists("primary.md"); - // After the renames the user expects: - // - moved.md = the file that was originally created - // at primary.md (Doc-X's content). - // - primary.md = the file that was originally created - // at staging.md (Doc-Y's content). - state.assertContains("moved.md", "primary content"); - state.assertContains("primary.md", "secondary content"); + state.assertContent("moved.md", "primary content "); + state.assertContent("primary.md", "secondary content "); - // No content cross-contamination: each contribution - // should land in exactly one of the user-visible - // files. Under the bug, the orphan at primary.md - // carries Doc-X's content (because Doc-Y's PUT was - // aliased onto Doc-X's record and read Doc-X's bytes - // from moved.md), so this catches the leak too. state.assertContentInAtMostOneFile("primary content"); state.assertContentInAtMostOneFile("secondary content"); } diff --git a/frontend/deterministic-tests/src/tests/server-pause-both-clients-create.test.ts b/frontend/deterministic-tests/src/tests/server-pause-both-clients-create.test.ts index f99cf92d..c048e194 100644 --- a/frontend/deterministic-tests/src/tests/server-pause-both-clients-create.test.ts +++ b/frontend/deterministic-tests/src/tests/server-pause-both-clients-create.test.ts @@ -32,10 +32,9 @@ export const serverPauseBothClientsCreateTest: TestDefinition = { { type: "assert-consistent", verify: (s: AssertableState): void => { - s.assertContains("alpha.md", "from client 0").assertContains( - "beta.md", - "from client 1" - ); + s.assertFileCount(2); + s.assertContent("alpha.md", "from client 0"); + s.assertContent("beta.md", "from client 1"); } } ] diff --git a/frontend/deterministic-tests/src/tests/server-pause-both-edit-same-file.test.ts b/frontend/deterministic-tests/src/tests/server-pause-both-edit-same-file.test.ts index ff8cf194..475c6592 100644 --- a/frontend/deterministic-tests/src/tests/server-pause-both-edit-same-file.test.ts +++ b/frontend/deterministic-tests/src/tests/server-pause-both-edit-same-file.test.ts @@ -39,10 +39,9 @@ export const serverPauseBothEditSameFileTest: TestDefinition = { { type: "assert-consistent", verify: (s: AssertableState): void => { - s.assertFileCount(1).assertContains( + s.assertFileCount(1).assertContent( "shared.md", - "edited by client 0", - "edited by client 1" + "line 1: edited by client 0\nline 2: original\nline 3: edited by client 1" ); } }, @@ -58,7 +57,7 @@ export const serverPauseBothEditSameFileTest: TestDefinition = { { type: "assert-consistent", verify: (s: AssertableState): void => { - s.assertFileCount(1).assertContains( + s.assertFileCount(1).assertContent( "shared.md", "post-merge edit from client 0" ); diff --git a/frontend/deterministic-tests/src/tests/text-pending-create-not-displaced.test.ts b/frontend/deterministic-tests/src/tests/text-pending-create-not-displaced.test.ts index 28243525..4b636ecf 100644 --- a/frontend/deterministic-tests/src/tests/text-pending-create-not-displaced.test.ts +++ b/frontend/deterministic-tests/src/tests/text-pending-create-not-displaced.test.ts @@ -27,9 +27,11 @@ export const textPendingCreateNotDisplacedTest: TestDefinition = { { type: "assert-consistent", verify: (s: AssertableState): void => { - s.assertFileCount(1) - .assertFileExists("data.txt") - .assertAnyFileContains("client-0", "client-1"); + s.assertFileCount(1).assertContentOneOf( + "data.txt", + "text data from client-0 client-1", + "text data from client-1 client-0" + ); } } ] diff --git a/frontend/deterministic-tests/src/tests/three-client-rename-create-delete.test.ts b/frontend/deterministic-tests/src/tests/three-client-rename-create-delete.test.ts index 80478adc..2a94de06 100644 --- a/frontend/deterministic-tests/src/tests/three-client-rename-create-delete.test.ts +++ b/frontend/deterministic-tests/src/tests/three-client-rename-create-delete.test.ts @@ -46,9 +46,9 @@ export const threeClientRenameCreateDeleteTest: TestDefinition = { { type: "assert-consistent", verify: (s: AssertableState): void => { - s.assertFileNotExists("X.md").assertAnyFileContains( - "new from C" - ); + s.assertFileCount(2) + .assertContent("Y.md", "original from A") + .assertContent(s.conflictPath("Y.md"), "new from C"); } } ] diff --git a/frontend/deterministic-tests/src/tests/update-during-create-processing.test.ts b/frontend/deterministic-tests/src/tests/update-during-create-processing.test.ts index ca53244e..2dc8897f 100644 --- a/frontend/deterministic-tests/src/tests/update-during-create-processing.test.ts +++ b/frontend/deterministic-tests/src/tests/update-during-create-processing.test.ts @@ -3,14 +3,14 @@ import type { TestDefinition } from "../test-definition"; export const updateDuringCreateProcessingTest: TestDefinition = { description: - "Client 0 creates a file while the server is paused, then immediately updates it. After the server resumes, both clients should converge with the updated content.", + "Client 0 creates a file while its create request is held, then immediately updates it. After releasing the request, both clients should converge with the updated content.", clients: 2, steps: [ { type: "enable-sync", client: 0 }, { type: "enable-sync", client: 1 }, { type: "barrier" }, - { type: "pause-server" }, + { type: "hold-request", client: 0, kind: "create", point: "before" }, { type: "create", @@ -18,6 +18,7 @@ export const updateDuringCreateProcessingTest: TestDefinition = { path: "file.md", content: "initial" }, + { type: "wait-for-request", client: 0 }, { type: "update", @@ -26,7 +27,7 @@ export const updateDuringCreateProcessingTest: TestDefinition = { content: "updated during create" }, - { type: "resume-server" }, + { type: "release-request", client: 0 }, { type: "barrier" }, { diff --git a/frontend/deterministic-tests/src/tests/user-parenthesized-file-not-deleted.test.ts b/frontend/deterministic-tests/src/tests/user-parenthesized-file-not-deleted.test.ts index 58bebb44..16d473c9 100644 --- a/frontend/deterministic-tests/src/tests/user-parenthesized-file-not-deleted.test.ts +++ b/frontend/deterministic-tests/src/tests/user-parenthesized-file-not-deleted.test.ts @@ -41,10 +41,13 @@ export const userParenthesizedFileNotDeletedTest: TestDefinition = { verify: (state: AssertableState): void => { state .assertFileCount(3) - .assertFileExists("Chapter.bin") + .assertContent("Chapter.bin", "chapter one") .assertFileExists("Chapter (1).bin") .assertContent("Chapter (1).bin", "chapter one notes") - .assertContent(state.conflictPath("Chapter.bin"), "chapter one notes"); + .assertContent( + state.conflictPath("Chapter.bin"), + "chapter one notes" + ); } } ] diff --git a/frontend/deterministic-tests/src/tests/watermark-advances-on-skip.test.ts b/frontend/deterministic-tests/src/tests/watermark-advances-on-skip.test.ts index 9615b5aa..43ba6a05 100644 --- a/frontend/deterministic-tests/src/tests/watermark-advances-on-skip.test.ts +++ b/frontend/deterministic-tests/src/tests/watermark-advances-on-skip.test.ts @@ -28,7 +28,9 @@ export const watermarkAdvancesOnSkipTest: TestDefinition = { { type: "assert-consistent", verify: (s: AssertableState): void => { - s.assertFileCount(2).assertFileExists("doc.md").assertAnyFileContains("from client 0", "from client 1"); + s.assertFileCount(2) + .assertContent("doc.md", "from client 0") + .assertContent(s.conflictPath("doc.md"), "from client 1"); } } ] diff --git a/frontend/deterministic-tests/src/utils/assertable-state.ts b/frontend/deterministic-tests/src/utils/assertable-state.ts index c208fa13..81534c0d 100644 --- a/frontend/deterministic-tests/src/utils/assertable-state.ts +++ b/frontend/deterministic-tests/src/utils/assertable-state.ts @@ -2,6 +2,8 @@ import assert from "node:assert/strict"; import type { ClientState, ExpectedDocument } from "../test-definition"; export class AssertableState { + private readonly exactPaths = new Set(); + private assertedEmpty = false; public readonly files: Map; public readonly clientFiles: Map[]; public readonly manifests: Record[]; @@ -43,12 +45,23 @@ export class AssertableState { `Two expected documents resolved to ${path}` ); paths.add(path); - this.assertBytes( - path, - typeof doc.content === "string" - ? new TextEncoder().encode(doc.content) - : new Uint8Array(doc.content) + const variants = [doc.content, ...(doc.alternatives ?? [])].map( + (value) => + typeof value === "string" + ? new TextEncoder().encode(value) + : new Uint8Array(value) ); + const actual = this.bytes.get(path); + assert( + actual && + variants.some( + (value) => + actual.length === value.length && + actual.every((byte, i) => byte === value[i]) + ), + `Byte mismatch at ${path}` + ); + this.exactPaths.add(path); const id = this.documentId(path); const recorded = next.get(doc.key); if (recorded !== undefined) this.assertIdentity(path, recorded); @@ -97,6 +110,7 @@ export class AssertableState { actual.some((byte, i) => byte !== expected[i]) ) throw new Error(`Byte mismatch at ${path}`); + this.exactPaths.add(path); return this; } @@ -107,6 +121,7 @@ export class AssertableState { `Expected ${expected} file(s), got ${this.files.size}: [${keys}]` ); } + if (expected === 0) this.assertedEmpty = true; return this; } @@ -136,7 +151,31 @@ export class AssertableState { `Expected "${path}" to have content "${expected}", got: "${actual}"` ); } - return this; + return this.assertBytes(path, new TextEncoder().encode(expected)); + } + + public assertContentOneOf(path: string, ...expected: string[]): this { + this.assertFileExists(path); + assert( + expected.includes(this.files.get(path)!), + `Unexpected complete content at ${path}: ${JSON.stringify(this.files.get(path))}` + ); + return this.assertBytes( + path, + new TextEncoder().encode(this.files.get(path)!) + ); + } + + public assertComplete(): void { + assert( + this.exactPaths.size > 0 || this.assertedEmpty, + "Scenario has no exact content oracle" + ); + assert.deepEqual( + [...this.exactPaths].sort(), + [...this.files.keys()].sort(), + "Scenario must assert exact bytes for the complete file set" + ); } public assertContains(path: string, ...substrings: string[]): this { diff --git a/frontend/deterministic-tests/src/v4-regressions.ts b/frontend/deterministic-tests/src/v4-regressions.ts index 75841fc4..a79315e3 100644 --- a/frontend/deterministic-tests/src/v4-regressions.ts +++ b/frontend/deterministic-tests/src/v4-regressions.ts @@ -159,8 +159,12 @@ export const V4_TESTS: Record = { .assertFileCount(4) .assertContent("folder/note.md", "nested file") .assertContent("README.md", "uppercase file"); - state.assertAnyFileContains( - "blocking file", + state.assertContent( + state.conflictPath("folder"), + "blocking file" + ); + state.assertContent( + state.conflictPath("readme.md"), "lowercase file" ); } diff --git a/frontend/deterministic-tests/src/worker-control.ts b/frontend/deterministic-tests/src/worker-control.ts new file mode 100644 index 00000000..b7485c64 --- /dev/null +++ b/frontend/deterministic-tests/src/worker-control.ts @@ -0,0 +1,58 @@ +import { Worker } from "node:worker_threads"; +import * as path from "node:path"; +import type { CrashWorkerInput } from "./crash-worker"; +import type { DiskImage, StoredClient } from "../../test-support/storage"; +import type { RequestRecord } from "../../test-support/network"; +import { withTimeout } from "./utils/with-timeout"; + +export interface WorkerSnapshot { + type: "checkpoint" | "finished"; + disk: DiskImage; + stored: StoredClient; + requests: RequestRecord[]; + boundary?: string; +} + +/** Termination discards the entire runtime, including pending callbacks. */ +export async function terminateAtCheckpoint( + input: CrashWorkerInput +): Promise { + const worker = new Worker( + path.resolve( + __dirname, + "../../../frontend/deterministic-tests/dist/crash-worker.js" + ), + { workerData: input } + ); + try { + return await withTimeout( + new Promise((resolve, reject) => { + worker.once("error", reject); + worker.once("exit", (code) => + reject( + new Error( + `Worker exited before crash checkpoint: ${code}` + ) + ) + ); + worker.once( + "message", + (message: WorkerSnapshot & { error?: string }) => { + if (message.type !== "checkpoint") + reject( + new Error( + message.error ?? + "Crash checkpoint never reached" + ) + ); + else resolve(message); + } + ); + }), + 15_000, + "Crash checkpoint never reached" + ); + } finally { + await worker.terminate(); + } +} diff --git a/frontend/package.json b/frontend/package.json index 17a03fb3..abb0b127 100644 --- a/frontend/package.json +++ b/frontend/package.json @@ -19,7 +19,7 @@ "dev": "concurrently --kill-others \"npm run dev -w sync-client\" \"npm run dev -w obsidian-plugin\"", "test": "npm run test --workspaces", "check:harness": "tsc --noEmit -p test-support/tsconfig.json && tsc --noEmit -p deterministic-tests/tsconfig.json && tsc --noEmit -p test-client/tsconfig.json", - "test:harness": "npm run check:harness && tsx --test 'test-support/harness.test.ts' 'test-support/oracle-regressions.test.ts' 'test-support/recovery.test.ts' 'test-support/file-application.test.ts' 'test-support/observer.test.ts' 'test-support/event-delivery.test.ts' 'test-support/polling.test.ts' 'test-support/engine-regressions.test.ts' 'test-support/sync-audit.test.ts' 'test-client/src/**/*.test.ts' '../scripts/e2e-runner.test.mjs'", + "test:harness": "npm run check:harness && tsx --test 'test-support/harness.test.ts' 'test-support/hardening.test.ts' 'test-support/oracle-regressions.test.ts' 'test-support/recovery.test.ts' 'test-support/file-application.test.ts' 'test-support/observer.test.ts' 'test-support/event-delivery.test.ts' 'test-support/polling.test.ts' 'test-support/engine-regressions.test.ts' 'test-support/sync-audit.test.ts' 'test-client/src/**/*.test.ts' '../scripts/e2e-runner.test.mjs'", "test:protocol": "tsx --test 'test-support/protocol.test.ts' 'test-support/restart-reconciliation.test.ts' 'test-support/content-limits.test.ts' 'test-support/server-restore.test.ts' 'test-support/protocol-audit.test.ts' 'test-support/real-disk.test.ts' 'test-support/production-adapters.test.ts'", "lint": "eslint --fix sync-client obsidian-plugin test-client deterministic-tests local-client-cli && prettier --write \"**/*.ts\"", "update": "ncu -u -ws" diff --git a/frontend/test-client/README.md b/frontend/test-client/README.md index 230f42d7..d58b0e85 100644 --- a/frontend/test-client/README.md +++ b/frontend/test-client/README.md @@ -1,36 +1,71 @@ # Seeded sync workloads -The old random mock filesystem/agent have been replaced by the same v4 adapter, -client wrapper, and canonical-state assertions as the scripted suite. -Periodic polling is explicitly disabled so it cannot repair missed notifications -and hide failures in these workloads. Polling has separate fake-clock tests. +The fuzzer runs three real clients against an isolated real server. It shares the +scripted suite's filesystem adapter and canonical-state checks. Polling is disabled +so it cannot repair missing notifications; separate fake-clock tests cover polling. -From `frontend`: +From `frontend`, after building the server: ```sh -npm run build --workspace sync-client --workspace test-client +npm run build --workspace sync-client --workspace deterministic-tests --workspace test-client node test-client/dist/cli.js --seed 42 --iterations 25 node test-client/dist/cli.js --replay /path/to/trace-42.json ``` -The server binary must already be built. Each invocation starts and cleans up its -own server/database, and prints the replay command before executing any actions. -`--artifacts` chooses the trace/result directory. Failures include database, -visible/durable disk images, and CAS request records. The default 25 iterations -are scenarios selected in shuffled cycles of nine operations, using two or three -clients. Existing documents are repeatedly edited, moved, swapped, deleted and -recreated. Concurrent edits share an established base; delayed editor events stay -queued through reconnection and a subsequent remote edit. Create, content-update -and manifest requests receive before/after network faults. +Each invocation owns its server and database. `--artifacts` selects the output +directory. The trace is written before execution; results retain exercised request +and crash checkpoints, response types and coverage counts. Failures also retain +metadata, visible/durable disk images and CAS requests before cleanup. -Each checkpoint serializes a complete expected file set with exact bytes, paths -and persistent identity keys. Independent creates must remain distinct, and a -recreated file must obtain a new UUID. Later checkpoints reassert earlier edits -and moves, rather than retaining only their markers. Old traces without a client -count still replay with two clients. +## Workload and oracle -All preservation assertions run even with delayed editor events. Explicit deletes -and overwrites exempt only their specific markers; unrelated content still must -appear exactly once. UUID continuity and canonical server state are checked too. -Seeds reproduce choices, not OS scheduling. See the +Version 2 traces contain an editor-intent program and its compiled steps. +`model.ts` independently interprets that intent into complete expected file sets: +exact bytes, paths and persistent identity keys. It imports no production merge +implementation and never adopts observed output as the expected state. Deleted +identity keys remain reserved. Concurrent text cases specify exact allowed results; +binary conflicts specify the documented first-committed winner. + +Each epoch composes arbitrary creates, replacements, range deletions, renames, +swaps and immediate delete/recreate operations before synchronization. Conflict +copies stay in the document pool and are subsequently edited, renamed and deleted. +Sixteen fault/concurrency families run in seeded shuffled cycles, including: + +- Three writers editing separated fields and two writers making overlapping edits. +- Forced stale-base races: hold a content CAS while another writer commits, rename + the local file, then require a new `StaleBase` response and the exact merged bytes. +- Online primitive histories and offline batches, delayed editor notifications, + and failures before or after durable metadata saves. +- Requests held before send or after receipt while further writes/renames occur; + dropped create, content and manifest requests with exact retry checks. +- Abrupt worker termination after accepted CAS responses, at metadata-save + boundaries, and during remote file installation, followed by user edits and + recovery. Some histories also kill and restart the server. + +Inputs include empty text, invalid UTF-8 bytes, Unicode, combining characters, +emoji, BOMs, CRLFs, size/cache boundaries, long filenames, case/normalization +aliases and file/directory conflicts. Initial inventories include 70 and 130 +documents to exercise paginated catchup; caches range from zero to 64 KiB. +Every checkpoint checks the full expected state on all replicas and the server. +Full-state checks also repeat after delayed events and crash recovery. + +## Campaigns, replay and reduction + +`scripts/e2e.sh` replays every checked-in `corpus/*.json` trace before fresh random +seeds. `E2E_SEED` pins a campaign; without it the runner prints and saves a fresh +uint32 seed range. Scheduled CI runs use 100 rounds per worker. The aggregate +`coverage.json` reports actions, CAS responses/payloads and exercised fault +boundaries. These are behavioral counters, not source branch-coverage percentages. + +On failure, the CLI automatically tries to remove epochs and individual actions, +recompiling every expected state from the remaining intent. Dependency-invalid +histories are skipped. A reduction must reproduce the same failure twice before +acceptance. `--shrink-attempts` controls the budget (default 32); `--no-shrink` +retains just the original reproducer. Replay `minimized-.json`, fix the bug, +and commit that concrete program to `corpus/`. Version 1 action traces still replay. + +Request gates and crash targets are replayed and must actually fire. UUID-derived +paths use symbolic identity keys. Seeds do not control OS or server scheduling, +and replay does not reuse production UUIDs. This is broader bug discovery, not a +proof of all possible interleavings or correctness. See the [full harness documentation](../deterministic-tests/README.md). diff --git a/frontend/test-client/corpus/conflict-copy.json b/frontend/test-client/corpus/conflict-copy.json new file mode 100644 index 00000000..0c55d1d6 --- /dev/null +++ b/frontend/test-client/corpus/conflict-copy.json @@ -0,0 +1,59 @@ +{ + "version": 2, + "seed": 1, + "iterations": 3, + "clients": 3, + "steps": [], + "program": { + "clients": 3, + "cacheBytes": 0, + "initial": [ + { + "type": "create", + "key": "sentinel", + "path": "untouched.bin", + "content": [0, 255, 254, 128, 13, 10] + } + ], + "epochs": [ + { + "writer": 0, + "actions": [ + { + "type": "pair", + "key": "first", + "other": "conflict", + "peer": 1, + "path": "café/note.md", + "content": "\ufefffirst\r\n", + "peerContent": "日本語 🦊\n" + } + ] + }, + { + "writer": 2, + "actions": [ + { + "type": "write", + "key": "conflict", + "content": [0, 255, 1] + }, + { "type": "rename", "key": "conflict", "path": "moved.bin" } + ] + }, + { + "writer": 1, + "actions": [ + { "type": "delete", "key": "conflict" }, + { + "type": "create", + "key": "new", + "path": "moved.bin", + "content": "new text\n" + }, + { "type": "write", "key": "first", "content": "" } + ] + } + ] + } +} diff --git a/frontend/test-client/corpus/stale-rename.json b/frontend/test-client/corpus/stale-rename.json new file mode 100644 index 00000000..0cf172f9 --- /dev/null +++ b/frontend/test-client/corpus/stale-rename.json @@ -0,0 +1,53 @@ +{ + "version": 2, + "seed": 2, + "iterations": 1, + "clients": 3, + "program": { + "clients": 3, + "cacheBytes": 1024, + "initial": [ + { + "type": "create", + "key": "note", + "path": "note.md", + "content": "header\n\nfirst\n\nsecond\n\nfooter\n" + } + ], + "epochs": [ + { + "writer": 0, + "actions": [ + { + "type": "concurrent", + "key": "note", + "writes": [ + { + "client": 0, + "content": "header\n\nLOCAL\n\nsecond\n\nfooter\n" + }, + { + "client": 1, + "content": "header\n\nfirst\n\nREMOTE\n\nfooter\n" + } + ], + "expected": ["header\n\nLOCAL\n\nREMOTE\n\nfooter\n"] + } + ], + "hold": { + "kind": "content", + "point": "before" + }, + "contender": 1, + "tail": [ + { + "type": "rename", + "key": "note", + "path": "renamed.md" + } + ] + } + ] + }, + "steps": [] +} diff --git a/frontend/test-client/corpus/uncertain-edits.json b/frontend/test-client/corpus/uncertain-edits.json new file mode 100644 index 00000000..4aefd0bc --- /dev/null +++ b/frontend/test-client/corpus/uncertain-edits.json @@ -0,0 +1,73 @@ +{ + "version": 2, + "seed": 0, + "iterations": 3, + "clients": 3, + "steps": [], + "program": { + "clients": 3, + "cacheBytes": 1024, + "initial": [ + { + "type": "create", + "key": "note", + "path": "note.md", + "content": "BASE\n" + } + ], + "epochs": [ + { + "writer": 0, + "actions": [ + { + "type": "write", + "key": "note", + "content": "BASE\nEDIT\n" + } + ], + "hold": { "kind": "content", "point": "before" }, + "drop": { "kind": "content", "point": "after" }, + "tail": [ + { "type": "rename", "key": "note", "path": "renamed.md" }, + { + "type": "write", + "key": "note", + "content": "BASE\nEDIT\nLATER\n" + } + ] + }, + { + "writer": 1, + "actions": [ + { "type": "delete", "key": "note" }, + { + "type": "create", + "key": "replacement", + "path": "renamed.md", + "content": "NEW GENERATION\n" + } + ] + }, + { + "writer": 2, + "actions": [ + { + "type": "create", + "key": "crash", + "path": "crash.md", + "content": "ACCEPTED\n" + } + ], + "crash": { "stopAt": "create" }, + "serverCrash": true, + "tail": [ + { + "type": "write", + "key": "crash", + "content": "ACCEPTED\nEDIT AFTER KILL\n" + } + ] + } + ] + } +} diff --git a/frontend/test-client/src/cli.ts b/frontend/test-client/src/cli.ts index eb407615..1b8dbced 100644 --- a/frontend/test-client/src/cli.ts +++ b/frontend/test-client/src/cli.ts @@ -7,6 +7,8 @@ import { ServerControl } from "../../deterministic-tests/src/server-control"; import { ServerManager } from "../../deterministic-tests/src/server-manager"; import { TestRunner } from "../../deterministic-tests/src/test-runner"; import { generateTrace, traceTest, type Trace } from "./workload"; +import { compileProgram } from "./model"; +import { minimizeProgram } from "./minimize"; function integer(value: string, zero = false): number { const number = Number(value); @@ -25,6 +27,8 @@ async function main(): Promise { seed: { type: "string", default: "1" }, iterations: { type: "string", default: "25" }, replay: { type: "string" }, + "no-shrink": { type: "boolean", default: false }, + "shrink-attempts": { type: "string", default: "32" }, artifacts: { type: "string" }, "server-binary": { type: "string" }, config: { type: "string" } @@ -40,6 +44,7 @@ async function main(): Promise { : fs.mkdtempSync(path.join(os.tmpdir(), "vault-link-fuzz-")); fs.mkdirSync(artifacts, { recursive: true }); const tracePath = path.join(artifacts, `trace-${trace.seed}.json`); + if (trace.version === 2) trace.steps = compileProgram(trace.program!); fs.writeFileSync(tracePath, JSON.stringify(trace, null, 2)); console.log(`Replay: node ${__filename} --replay ${tracePath}`); const logger = new Logger(); @@ -66,7 +71,65 @@ async function main(): Promise { path.join(artifacts, `result-${trace.seed}.json`), JSON.stringify(result, null, 2) ); - if (!result.success) throw new Error(result.error); + if (!result.success) { + if (trace.program && !values["no-shrink"]) { + try { + const reduced = await minimizeProgram( + trace.program, + result.error ?? "", + async (program, attempt) => { + await server.stop(); + await server.start(); + const candidate = { + ...trace, + program, + steps: compileProgram(program) + }; + const outcome = await new TestRunner( + server, + logger, + "test-token-change-me", + server.remoteUri + ).runTest( + `shrink-${attempt}`, + traceTest(candidate) + ); + fs.writeFileSync( + path.join(artifacts, `shrink-${attempt}.json`), + JSON.stringify(outcome, null, 2) + ); + return outcome; + }, + integer(values["shrink-attempts"]!) + ); + const minimized = { + ...trace, + program: reduced.program, + steps: compileProgram(reduced.program) + }; + const minimizedPath = path.join( + artifacts, + `minimized-${trace.seed}.json` + ); + fs.writeFileSync( + minimizedPath, + JSON.stringify(minimized, null, 2) + ); + console.log( + `Reduced in ${reduced.attempts} attempts: node ${__filename} --replay ${minimizedPath}` + ); + } catch (error) { + fs.writeFileSync( + path.join(artifacts, "shrink-error.txt"), + String(error) + ); + console.error( + `Reduction failed; original trace and failure retained: ${String(error)}` + ); + } + } + throw new Error(result.error); + } } finally { await server.stop(); manager.untrack(server); diff --git a/frontend/test-client/src/minimize.ts b/frontend/test-client/src/minimize.ts new file mode 100644 index 00000000..bec0c58e --- /dev/null +++ b/frontend/test-client/src/minimize.ts @@ -0,0 +1,80 @@ +import { compileProgram, type Program } from "./model"; + +export interface Failure { + success: boolean; + error?: string; +} +export function failureSignature(error: string): string { + return error + .replace( + /[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}/gi, + "" + ) + .replace(/localhost:\d+/g, "localhost:"); +} + +/** Dependency-invalid candidates never run. Every candidate regenerates its + * oracle from surviving editor intents; expected states are never weakened. */ +export async function minimizeProgram( + original: Program, + failure: string, + run: (program: Program, attempt: number) => Promise, + budget = 32 +): Promise<{ program: Program; attempts: number }> { + let current = structuredClone(original), + attempts = 0; + const signature = failureSignature(failure); + const accept = async (candidate: Program): Promise => { + if (attempts + 2 > budget) return false; + try { + compileProgram(candidate); + } catch { + return false; + } + // Require a reproducible failure, including its oracle and target. + for (let repeat = 0; repeat < 2; repeat++) { + const result = await run(candidate, ++attempts); + if ( + result.success || + failureSignature(result.error ?? "") !== signature + ) + return false; + } + current = candidate; + return true; + }; + for ( + let chunk = Math.max(1, Math.floor(current.epochs.length / 2)); + chunk >= 1; + chunk = Math.floor(chunk / 2) + ) { + for ( + let index = 0; + index < current.epochs.length && attempts + 2 <= budget; + + ) { + const candidate = structuredClone(current); + candidate.epochs.splice(index, chunk); + if (!(await accept(candidate))) index += chunk; + } + } + for ( + let epoch = 0; + epoch < current.epochs.length && attempts + 2 <= budget; + epoch++ + ) { + for (const field of ["actions", "tail"] as const) { + for ( + let index = 0; + index < (current.epochs[epoch][field]?.length ?? 0) && + attempts + 2 <= budget; + + ) { + const candidate = structuredClone(current); + candidate.epochs[epoch][field]!.splice(index, 1); + if (!(await accept(candidate))) index++; + } + } + } + return { program: current, attempts }; +} diff --git a/frontend/test-client/src/model.ts b/frontend/test-client/src/model.ts new file mode 100644 index 00000000..80701a86 --- /dev/null +++ b/frontend/test-client/src/model.ts @@ -0,0 +1,438 @@ +import assert from "node:assert/strict"; +import type { + ExpectedDocument, + TestStep +} from "../../deterministic-tests/src/test-definition"; +import type { CrashWorkerInput } from "../../deterministic-tests/src/crash-worker"; +import type { RequestKind } from "../../test-support/network"; + +export type Content = string | number[]; +export type Mutation = + | { type: "create"; key: string; path: string; content: Content } + | { type: "write"; key: string; content: Content } + | { type: "rename"; key: string; path: string } + | { type: "delete"; key: string } + | { type: "swap"; key: string; other: string; temporary: string } + | { + type: "pair"; + key: string; + other: string; + peer: number; + path: string; + peerPath?: string; + content: Content; + peerContent: Content; + } + | { + type: "concurrent"; + key: string; + writes: { client: number; content: Content }[]; + expected: Content[]; + }; +export interface Epoch { + writer: number; + online?: boolean; + actions: Mutation[]; + tail?: Mutation[]; + hold?: { kind: RequestKind; point: "before" | "after" }; + contender?: number; + drop?: { kind: RequestKind; point: "before" | "after" }; + crash?: Pick; + serverCrash?: boolean; + delayedTail?: boolean; + saveFailure?: "before" | "durable"; + download?: { + peer: number; + actions: Mutation[]; + crashPath: string; + point: "before" | "visible" | "durable"; + }; +} +export interface Program { + clients: number; + initial: Extract[]; + epochs: Epoch[]; + cacheBytes: number; +} + +/** The oracle only evaluates editor intent. It never imports sync/merge code or + * reads an implementation result. Compilation is also used after shrinking: + * removing an action recomputes every subsequent expected state. */ +export class ReferenceModel { + public readonly documents = new Map(); + private readonly used = new Set(); + public get(key: string): ExpectedDocument { + const doc = this.documents.get(key); + assert(doc, `Invalid program: missing document ${key}`); + return doc; + } + public add(key: string, path: string, content: Content): void { + path = path.normalize("NFC"); + assert(!this.used.has(key), `Invalid program: reused key ${key}`); + assert( + ![...this.documents.values()].some((d) => d.path === path), + `Invalid program: occupied path ${path}` + ); + this.used.add(key); + this.documents.set(key, { + key, + path, + content: structuredClone(content) + }); + } + public apply(action: Mutation): void { + switch (action.type) { + case "create": + this.add(action.key, action.path, action.content); + break; + case "pair": { + this.add( + action.key, + action.path.normalize("NFC"), + action.content + ); + const path = action.peerPath ?? action.path; + const dot = path.lastIndexOf("."); + const split = + dot > path.lastIndexOf("/") + 1 ? dot : path.length; + this.add( + action.other, + `${path.slice(0, split)} (conflict {{${action.other}}})${path.slice(split)}`.normalize( + "NFC" + ), + action.peerContent + ); + break; + } + case "write": { + const doc = this.get(action.key); + doc.content = structuredClone(action.content); + delete doc.alternatives; + break; + } + case "rename": { + assert( + ![...this.documents.values()].some( + (d) => + d.key !== action.key && + d.path === action.path.normalize("NFC") + ), + "Invalid program: occupied rename" + ); + this.get(action.key).path = action.path.normalize("NFC"); + break; + } + case "delete": + this.get(action.key); + this.documents.delete(action.key); + break; + case "swap": { + const a = this.get(action.key), + b = this.get(action.other); + assert(a !== b, "Invalid program: self swap"); + assert( + ![...this.documents.values()].some( + (d) => d.path === action.temporary.normalize("NFC") + ), + "Invalid program: occupied temporary path" + ); + [a.path, b.path] = [b.path, a.path]; + break; + } + case "concurrent": { + assert( + action.expected.length > 0 && action.writes.length >= 2, + "Invalid program: empty concurrency oracle" + ); + const doc = this.get(action.key); + doc.content = structuredClone(action.expected[0]); + doc.alternatives = structuredClone(action.expected.slice(1)); + break; + } + } + } + public expected(): ExpectedDocument[] { + return structuredClone([...this.documents.values()]); + } +} + +export function compileProgram(program: Program): TestStep[] { + assert( + Number.isInteger(program.clients) && + program.clients >= 2 && + program.clients <= 8, + "Invalid program: clients" + ); + assert( + Number.isFinite(program.cacheBytes) && program.cacheBytes >= 0, + "Invalid program: cache size" + ); + const steps: TestStep[] = []; + const model = new ReferenceModel(); + const all = (type: "enable-sync" | "disable-sync", first = 0) => { + for (let i = 0; i < program.clients; i++) + steps.push({ type, client: (first + i) % program.clients }); + }; + const checkpoint = () => + steps.push( + { type: "barrier" }, + { type: "assert-documents", expected: model.expected() } + ); + const write = (client: number, path: string, content: Content) => + steps.push( + typeof content === "string" + ? { type: "update", client, path, content } + : { type: "create-bytes", client, path, bytes: content } + ); + const mutate = (action: Mutation, client: number, remember: boolean) => { + switch (action.type) { + case "create": + write(client, action.path, action.content); + if (remember) + steps.push({ + type: "remember-local-identity", + client, + path: action.path, + key: action.key + }); + break; + case "pair": + assert( + remember && + action.peer !== client && + Number.isInteger(action.peer) && + action.peer >= 0 && + action.peer < program.clients, + "Invalid program: independent creates must be offline on distinct clients" + ); + for (const [owner, key, content] of [ + [client, action.key, action.content], + [action.peer, action.other, action.peerContent] + ] as const) { + const path = + owner === client + ? action.path + : (action.peerPath ?? action.path); + write(owner, path, content); + steps.push({ + type: "remember-local-identity", + client: owner, + path, + key + }); + } + break; + case "write": + write(client, model.get(action.key).path, action.content); + break; + case "delete": + steps.push({ + type: "delete", + client, + path: model.get(action.key).path + }); + break; + case "rename": + steps.push({ + type: "rename", + client, + oldPath: model.get(action.key).path, + newPath: action.path + }); + break; + case "swap": { + const a = model.get(action.key).path, + b = model.get(action.other).path; + steps.push( + { + type: "rename", + client, + oldPath: a, + newPath: action.temporary + }, + { type: "rename", client, oldPath: b, newPath: a }, + { + type: "rename", + client, + oldPath: action.temporary, + newPath: b + } + ); + break; + } + case "concurrent": + assert( + remember, + "Invalid program: concurrent edits need an established offline base" + ); + for (const edit of action.writes) { + assert( + edit.client >= 0 && edit.client < program.clients, + "Invalid program: writer" + ); + write( + edit.client, + model.get(action.key).path, + edit.content + ); + } + break; + } + model.apply(action); + }; + all("enable-sync"); + steps.push({ type: "barrier" }); + all("disable-sync"); + for (let client = 0; client < program.clients; client++) + steps.push({ + type: "set-settings", + client, + settings: { diffCacheSizeMB: program.cacheBytes / (1024 * 1024) } + }); + for (const doc of program.initial) mutate(doc, 0, true); + all("enable-sync"); + checkpoint(); + for (const epoch of program.epochs) { + const writer = epoch.writer; + assert( + Number.isInteger(writer) && + writer >= 0 && + writer < program.clients && + epoch.actions.length, + "Invalid program: empty epoch or writer" + ); + assert( + !epoch.online || + !(epoch.hold || epoch.drop || epoch.crash || epoch.serverCrash), + "Invalid program: online fault setup" + ); + assert( + !epoch.tail?.length || epoch.hold || epoch.crash || epoch.download, + "Invalid program: tail has no scheduling boundary" + ); + assert( + !epoch.serverCrash || epoch.crash, + "Invalid program: server crash requires a client crash" + ); + assert( + !epoch.delayedTail || epoch.hold, + "Invalid program: delayed tail requires a request gate" + ); + if (epoch.contender !== undefined) + assert( + epoch.hold?.kind === "content" && + epoch.hold.point === "before" && + Number.isInteger(epoch.contender) && + epoch.contender >= 0 && + epoch.contender < program.clients && + epoch.contender !== writer, + "Invalid program: stale-content race needs a distinct contender and before-content gate" + ); + if (!epoch.online) all("disable-sync"); + if (epoch.saveFailure) + steps.push({ + type: "fail-save", + client: writer, + point: epoch.saveFailure + }); + for (const [index, action] of epoch.actions.entries()) { + mutate(action, writer, !epoch.online); + if (epoch.saveFailure && index === 0) + steps.push({ type: "wait-for-save-failure", client: writer }); + } + if (epoch.drop) + steps.push({ + type: "drop-response", + client: writer, + ...epoch.drop + }); + if (epoch.download) { + assert( + !epoch.online && !epoch.hold && !epoch.drop && !epoch.crash, + "Invalid program: download crash controls" + ); + assert( + epoch.download.peer !== writer, + "Invalid program: self download" + ); + for (const action of epoch.download.actions) + mutate(action, epoch.download.peer, true); + steps.push({ type: "enable-sync", client: epoch.download.peer }); + steps.push({ type: "disable-sync", client: epoch.download.peer }); + steps.push({ + type: "crash-client", + client: writer, + at: { + stopAtBoundary: { + type: "filesystem", + label: `${epoch.download.point}:write:${epoch.download.crashPath}` + } + } + }); + for (const action of epoch.tail ?? []) { + assert( + action.type === "write", + "Invalid program: post-crash namespace notification" + ); + mutate(action, writer, false); + } + steps.push({ type: "restart-client", client: writer }); + } else if (epoch.crash) { + assert( + !epoch.hold && !epoch.drop, + "Invalid program: mixed crash gates" + ); + steps.push({ + type: "crash-client", + client: writer, + at: epoch.crash + }); + if (epoch.serverCrash) + steps.push( + { type: "crash-server" }, + { type: "restart-server" } + ); + for (const action of epoch.tail ?? []) { + assert( + action.type === "write", + "Invalid program: post-crash namespace notification" + ); + mutate(action, writer, false); + } + steps.push({ type: "restart-client", client: writer }); + } else if (epoch.hold) { + steps.push({ type: "hold-request", client: writer, ...epoch.hold }); + steps.push( + { type: "enable-sync", client: writer, wait: false }, + { type: "wait-for-request", client: writer } + ); + if (epoch.delayedTail) + steps.push({ type: "delay-notifications", client: writer }); + if (epoch.contender !== undefined) + steps.push({ type: "enable-sync", client: epoch.contender }); + for (const action of epoch.tail ?? []) + mutate(action, writer, false); + steps.push({ type: "release-request", client: writer }); + } + if (epoch.drop) + steps.push( + { type: "enable-sync", client: writer, wait: false }, + { type: "wait-for-response-drop", client: writer } + ); + if (!epoch.online) all("enable-sync", writer); + checkpoint(); + if (epoch.contender !== undefined) + steps.push({ + type: "assert-response", + client: writer, + kind: "content", + responseType: "StaleBase", + sinceCheckpoint: true + }); + if (epoch.delayedTail) { + steps.push({ type: "flush-notifications", client: writer }); + checkpoint(); + } + } + return steps; +} diff --git a/frontend/test-client/src/stateful-workload.test.ts b/frontend/test-client/src/stateful-workload.test.ts new file mode 100644 index 00000000..38e0630f --- /dev/null +++ b/frontend/test-client/src/stateful-workload.test.ts @@ -0,0 +1,149 @@ +import assert from "node:assert/strict"; +import { test } from "node:test"; +import { generateTrace, traceTest } from "./workload"; +import { compileProgram, type Program } from "./model"; +import { minimizeProgram } from "./minimize"; + +test("stateful programs replay primitive batches, crash boundaries, byte domains and conflict histories", () => { + const seen = new Set(); + let large = false, + empty = false, + binary = false, + unicode = false, + alias = false, + immediateRecreate = false; + for (let seed = 0; seed < 12; seed++) { + const trace = generateTrace(seed, 25); + assert.deepEqual(trace, generateTrace(seed, 25)); + assert.deepEqual( + traceTest(trace), + traceTest(JSON.parse(JSON.stringify(trace))) + ); + assert(trace.program); + large ||= trace.program.initial.length >= 70; + for (const step of trace.steps) seen.add(step.type); + for (const epoch of trace.program.epochs) { + assert(epoch.actions.length); + for (let i = 0; i < epoch.actions.length; i++) { + const action = epoch.actions[i]; + if (action.type === "pair") + alias ||= action.peerPath !== action.path; + immediateRecreate ||= + action.type === "delete" && + epoch.actions[i + 1]?.type === "create"; + if ("content" in action) { + empty ||= action.content === ""; + binary ||= Array.isArray(action.content); + unicode ||= + typeof action.content === "string" && + /[^\x00-\x7f]/.test(action.content); + } + } + } + } + for (const type of [ + "hold-request", + "wait-for-request", + "crash-client", + "restart-client", + "crash-server", + "restart-server", + "fail-save", + "wait-for-save-failure", + "delay-notifications", + "assert-response", + "assert-documents" + ]) + assert(seen.has(type), `Missing ${type}`); + assert(large && empty && binary && unicode && alias && immediateRecreate); +}); + +test("shrinking retains the failing editor intent and regenerates a complete independent oracle", async () => { + const program: Program = { + clients: 3, + cacheBytes: 1024, + initial: [ + { type: "create", key: "a", path: "a.md", content: "A" }, + { type: "create", key: "b", path: "b.md", content: "B" } + ], + epochs: [ + { + writer: 0, + actions: [{ type: "write", key: "a", content: "irrelevant" }] + }, + { + writer: 1, + actions: [ + { type: "write", key: "a", content: "also irrelevant" }, + { type: "write", key: "b", content: "TRIGGER" } + ] + }, + { + writer: 2, + actions: [{ type: "rename", key: "a", path: "moved.md" }] + } + ] + }; + const failure = "Custom verification failed: Byte mismatch at b.md"; + const reduced = await minimizeProgram( + program, + failure, + async (candidate) => { + const expected = compileProgram(candidate) + .filter((s) => s.type === "assert-documents") + .at(-1)!; + assert.equal( + expected.expected.length, + 2, + "Reducer removed unrelated preservation obligations" + ); + const fails = candidate.epochs.some((epoch) => + epoch.actions.some( + (action) => + action.type === "write" && action.content === "TRIGGER" + ) + ); + return { success: !fails, error: fails ? failure : undefined }; + } + ); + assert.deepEqual(reduced.program.epochs, [ + { + writer: 1, + actions: [{ type: "write", key: "b", content: "TRIGGER" }] + } + ]); + const expected = compileProgram(reduced.program) + .filter((s) => s.type === "assert-documents") + .at(-1)!.expected; + assert.deepEqual(expected, [ + { key: "a", path: "a.md", content: "A" }, + { key: "b", path: "b.md", content: "TRIGGER" } + ]); +}); + +test("shrinking rejects dependency-invalid histories and different failure mechanisms", async () => { + const trace = generateTrace(0, 1); + const original = trace.program!; + const reduced = await minimizeProgram( + original, + "Byte mismatch at a.md", + async () => ({ success: false, error: "Checkpoint never reached" }), + 4 + ); + assert.deepEqual(reduced.program, original); + assert.throws( + () => + compileProgram({ + ...original, + epochs: [ + { + writer: 0, + actions: [ + { type: "write", key: "missing", content: "x" } + ] + } + ] + }), + /Invalid program/ + ); +}); diff --git a/frontend/test-client/src/stateful-workload.ts b/frontend/test-client/src/stateful-workload.ts new file mode 100644 index 00000000..ee562a27 --- /dev/null +++ b/frontend/test-client/src/stateful-workload.ts @@ -0,0 +1,481 @@ +import { Random } from "../../test-support/random"; +import { + compileProgram, + ReferenceModel, + type Content, + type Epoch, + type Mutation, + type Program +} from "./model"; +import type { Trace } from "./workload"; + +const words = [ + "alpha", + "日本語", + "café", + "e\u0301", + "🦊", + "🌍", + "مرحبا", + "שלום", + "नमस्ते", + "\ufeff", + "\r\n", + "\n", + " ", + "\t" +]; +export function randomContent(random: Random): Content { + const kind = random.int(8); + if (kind === 0) return ""; + if (kind === 1) + return Array.from( + { length: random.pick([1, 2, 255, 1023, 1024, 1025, 4096]) }, + () => random.int(256) + ); + if (kind === 2) return [0, 255, 254, 128, 13, 10]; + const text = Array.from({ length: 1 + random.int(30) }, () => + random.pick(words) + ).join(""); + return kind === 3 ? text.repeat(128) : text; +} + +/** Random primitive histories share a persistent model, including conflict + * copies. Only checkpoints settle the system; each epoch composes 2–12 edits. */ +export function generateStatefulTrace(seed: number, iterations: number): Trace { + const random = new Random(seed); + const clients = 3; + const model = new ReferenceModel(); + let serial = 0; + const key = () => `d${serial++}`; + const path = () => + `${random.pick(["shared", "日本語", "café", "space folder"])}/${key()}.${random.pick(["md", "txt", "bin"])}`; + const initialCount = random.pick([4, 12, 70, 130]); + const program: Program = { + clients, + cacheBytes: random.pick([0, 1024, 4096, 65536]), + initial: [], + epochs: [] + }; + for (let i = 0; i < initialCount; i++) { + const doc: Extract = { + type: "create", + key: key(), + path: i === 0 ? `${"é".repeat(126)}.md` : path(), + content: randomContent(random) + }; + program.initial.push(doc); + model.apply(doc); + } + const chooseDoc = () => random.pick([...model.documents.values()]); + const push = (epoch: Epoch, action: Mutation) => { + epoch.actions.push(action); + model.apply(action); + }; + const families: number[] = []; + for (let round = 0; round < iterations; round++) { + const writer = random.int(clients); + const epoch: Epoch = { writer, actions: [] }; + // Cover each fault family in shuffled cycles, composing it with the + // arbitrary primitive history and identities left by earlier rounds. + if (!families.length) { + families.push(...Array.from({ length: 16 }, (_, index) => index)); + for (let i = families.length - 1; i > 0; i--) { + const j = random.int(i + 1); + [families[i], families[j]] = [families[j], families[i]]; + } + } + const family = families.pop()!; + if (family === 0) { + const first = key(), + second = key(); + const names = random.pick([ + [`alias-${round}/note.md`, `alias-${round}/note.md`], + [`alias-${round}/README.md`, `alias-${round}/readme.md`], + [`alias-${round}/café.md`, `alias-${round}/cafe\u0301.md`], + [`alias-${round}/folder/note.md`, `alias-${round}/folder`] + ]); + push(epoch, { + type: "pair", + key: first, + other: second, + peer: (writer + 1) % clients, + path: names[0], + peerPath: names[1], + content: randomContent(random), + peerContent: randomContent(random) + }); + epoch.drop = { + kind: random.pick(["create", "manifest"]), + point: random.pick(["before", "after"]) + }; + } else if (family === 1) { + // Three writers replace three separated fields of one established base. + const doc = chooseDoc(); + const base = `header ${round}\n\nfirst field\n\nsecond field\n\nthird field\n\nfooter\n`; + // Establish a known text base; the following epoch races all three writes. + const prepare: Epoch = { + writer, + actions: [ + { type: "rename", key: doc.key, path: `merge-${round}.md` }, + { type: "write", key: doc.key, content: base } + ] + }; + for (const action of prepare.actions) model.apply(action); + program.epochs.push(prepare); + const fields = ["first field", "second field", "third field"]; + const replacements = fields.map( + (_, i) => + `changed ${i} ${random.pick(["日本語", "🦊", "café"])}` + ); + push(epoch, { + type: "concurrent", + key: doc.key, + writes: fields.map((field, client) => ({ + client, + content: base.replace(field, replacements[client]) + })), + expected: [ + fields.reduce( + (text, field, i) => + text.replace(field, replacements[i]), + base + ) + ] + }); + } else if (family === 8 || family === 9) { + const doc = chooseDoc(); + const text = family === 8; + const base: Content = text ? "the quick brown fox\n" : [0, 255, 12]; + const prepare: Epoch = { + writer, + actions: [ + { + type: "rename", + key: doc.key, + path: `overlap-${round}.${text ? "md" : "bin"}` + }, + { type: "write", key: doc.key, content: base } + ] + }; + for (const action of prepare.actions) model.apply(action); + program.epochs.push(prepare); + const values: Content[] = text + ? ["the slow brown fox\n", "the fast brown fox\n"] + : [ + [0, 254, random.int(256)], + [0, 253, random.int(256)] + ]; + push(epoch, { + type: "concurrent", + key: doc.key, + writes: values.map((content, i) => ({ + client: (writer + i) % clients, + content + })), + expected: text + ? ["the fast slow brown fox\n", "the slow fast brown fox\n"] + : [values[0]] + }); + } else if (family === 15) { + const doc = chooseDoc(); + const base = `header ${round}\n\nfirst field\n\nsecond field\n\nfooter\n`; + const prepare: Epoch = { + writer, + actions: [ + { type: "rename", key: doc.key, path: `stale-${round}.md` }, + { type: "write", key: doc.key, content: base } + ] + }; + for (const action of prepare.actions) model.apply(action); + program.epochs.push(prepare); + epoch.contender = (writer + 1) % clients; + push(epoch, { + type: "concurrent", + key: doc.key, + writes: [ + { + client: writer, + content: base.replace("first field", "local 🦊") + }, + { + client: epoch.contender, + content: base.replace("second field", "remote 日本語") + } + ], + expected: [ + base + .replace("first field", "local 🦊") + .replace("second field", "remote 日本語") + ] + }); + epoch.hold = { kind: "content", point: "before" }; + epoch.tail = [ + { + type: "rename", + key: doc.key, + path: `stale-renamed-${round}.md` + } + ]; + for (const action of epoch.tail) model.apply(action); + } else if (family >= 12) { + const doc = chooseDoc(); + const content = `content ${round} 日本語\r\n`; + push(epoch, { type: "write", key: doc.key, content }); + epoch.tail = [ + { + type: "write", + key: doc.key, + content: content + "EDIT DURING UNCERTAIN OUTCOME\r\n" + } + ]; + if (family === 12) { + epoch.hold = { + kind: "content", + point: random.pick(["before", "after"]) + }; + epoch.drop = { + kind: "content", + point: random.pick(["before", "after"]) + }; + epoch.tail.push({ type: "rename", key: doc.key, path: path() }); + } else { + epoch.crash = + family === 13 + ? { stopAt: "content" } + : { + stopAtBoundary: { + type: "persistence", + label: random.pick([ + "before:save", + "durable:save" + ]), + afterAccepted: "content" + } + }; + epoch.serverCrash = random.int(2) === 0; + } + for (const action of epoch.tail) model.apply(action); + } else if (family === 10) { + const doc = chooseDoc(); + const crashPath = `download-${round}.md`; + const base = `BASE ${round}\n`; + const prepare: Epoch = { + writer, + actions: [ + { type: "rename", key: doc.key, path: crashPath }, + { type: "write", key: doc.key, content: base } + ] + }; + for (const action of prepare.actions) model.apply(action); + program.epochs.push(prepare); + // Unrelated local work survives the interrupted remote installation too. + push(epoch, { + type: "create", + key: key(), + path: `unrelated-${round}.bin`, + content: [0, 255, 13, 10] + }); + const remote: Mutation = { + type: "write", + key: doc.key, + content: base + "REMOTE\n" + }; + epoch.download = { + peer: (writer + 1) % clients, + actions: [remote], + crashPath, + point: random.pick(["before", "visible", "durable"]) + }; + model.apply(remote); + epoch.tail = [ + { + type: "write", + key: doc.key, + content: base + "REMOTE\nAFTER INTERRUPTION\n" + } + ]; + for (const action of epoch.tail) model.apply(action); + } else { + epoch.online = family === 2; + if (family === 2 && random.int(2) === 0) { + epoch.online = false; + epoch.saveFailure = random.pick(["before", "durable"]); + push(epoch, { + type: "rename", + key: chooseDoc().key, + path: path() + }); + } + const count = 2 + random.int(11); + for (let i = 0; i < count; i++) { + const doc = chooseDoc(); + switch (random.int(6)) { + case 0: + push(epoch, { + type: "create", + key: key(), + path: path(), + content: randomContent(random) + }); + break; + case 1: + push(epoch, { + type: "write", + key: doc.key, + content: randomContent(random) + }); + break; + case 2: + push(epoch, { + type: "rename", + key: doc.key, + path: path() + }); + break; + case 3: { + const oldPath = doc.path; + push(epoch, { type: "delete", key: doc.key }); + // Recreate immediately, with no synchronization or checkpoint. + push(epoch, { + type: "create", + key: key(), + path: oldPath, + content: randomContent(random) + }); + break; + } + case 4: { + const other = random.pick( + [...model.documents.values()].filter( + (d) => d.key !== doc.key + ) + ); + push(epoch, { + type: "swap", + key: doc.key, + other: other.key, + temporary: `temporary-${round}-${i}.md` + }); + break; + } + default: + push(epoch, { + type: "write", + key: doc.key, + content: + typeof doc.content === "string" + ? Array.from(doc.content) + .slice( + random.int( + Array.from(doc.content) + .length + 1 + ) + ) + .join("") + + `\nreplacement ${round}-${i} 🦊\n` + : [ + ...doc.content.slice( + random.int(doc.content.length + 1) + ), + 0, + random.int(256) + ] + }); + } + } + if (family >= 3 && family <= 7) { + // A fresh surviving upload guarantees the chosen fault is exercised, + // regardless of coalescing in the preceding primitive history. + const target = key(); + push(epoch, { + type: "create", + key: target, + path: `fault-${round}.md`, + content: `fault base ${round}\n` + }); + if (family === 3 || family === 4) { + epoch.hold = { + kind: "create", + point: family === 3 ? "before" : "after" + }; + epoch.drop = { + kind: "manifest", + point: random.pick(["before", "after"]) + }; + epoch.tail = [ + { + type: "write", + key: target, + content: `fault base ${round}\nTAIL ${round}\n` + } + ]; + if (family === 3) + epoch.tail.push({ + type: "rename", + key: target, + path: `tail-${round}.md` + }); + epoch.delayedTail = family === 4; + for (const action of epoch.tail) model.apply(action); + } else { + const kind = family === 5 ? "create" : "manifest"; + epoch.crash = + family === 7 + ? { + stopAtBoundary: { + type: "persistence", + label: random.pick([ + "before:save", + "durable:save" + ]), + afterAccepted: kind + } + } + : { stopAt: kind }; + epoch.serverCrash = family === 6; + epoch.tail = [ + { + type: "write", + key: target, + content: `fault base ${round}\nAFTER CRASH ${round}\n` + } + ]; + for (const action of epoch.tail) model.apply(action); + } + } + } + program.epochs.push(epoch); + if (family === 0) { + const pair = epoch.actions[0]; + if (pair.type !== "pair") + throw new Error("Missing independent creates"); + const follow: Epoch = { writer: random.int(clients), actions: [] }; + push(follow, { + type: "write", + key: pair.other, + content: randomContent(random) + }); + push(follow, { type: "rename", key: pair.other, path: path() }); + if (random.int(2)) { + const previous = model.get(pair.other).path; + push(follow, { type: "delete", key: pair.other }); + push(follow, { + type: "create", + key: key(), + path: previous, + content: randomContent(random) + }); + } + program.epochs.push(follow); + } + } + return { + version: 2, + seed, + iterations, + clients, + program, + steps: compileProgram(program) + }; +} diff --git a/frontend/test-client/src/workload.test.ts b/frontend/test-client/src/workload.test.ts index 8d9814f7..4663756f 100644 --- a/frontend/test-client/src/workload.test.ts +++ b/frontend/test-client/src/workload.test.ts @@ -1,6 +1,6 @@ import { test } from "node:test"; import assert from "node:assert/strict"; -import { generateTrace, traceTest } from "./workload"; +import { generateLegacyTrace as generateTrace, traceTest } from "./workload"; test("concrete traces replay all choices and retain preservation assertions with delayed events", () => { const trace = generateTrace(42, 12); diff --git a/frontend/test-client/src/workload.ts b/frontend/test-client/src/workload.ts index 4f199f46..7aecf9d8 100644 --- a/frontend/test-client/src/workload.ts +++ b/frontend/test-client/src/workload.ts @@ -1,3 +1,5 @@ +import { generateStatefulTrace } from "./stateful-workload"; +import { compileProgram, type Program } from "./model"; import type { ExpectedDocument, TestDefinition, @@ -6,7 +8,8 @@ import type { import { Random } from "../../test-support/random"; export interface Trace { - version: 1; + version: 1 | 2; + program?: Program; seed: number; iterations: number; /** Old traces omit this and retain their original two-client behavior. */ @@ -20,7 +23,7 @@ type Document = ExpectedDocument & { content: string }; * Every round checks the whole history, including earlier bytes and UUIDs. * A shuffled cycle covers nine actions; most revisit existing documents. */ -export function generateTrace(seed: number, iterations: number): Trace { +export function generateLegacyTrace(seed: number, iterations: number): Trace { const random = new Random(seed); const clients = 2 + random.int(2); const steps: TestStep[] = []; @@ -295,16 +298,21 @@ export function generateTrace(seed: number, iterations: number): Trace { export function traceTest(trace: Trace): TestDefinition { if ( - trace.version !== 1 || - !Array.isArray(trace.steps) || - !trace.steps.length || + ![1, 2].includes(trace.version) || + (trace.version === 1 && + (!Array.isArray(trace.steps) || !trace.steps.length)) || + (trace.version === 2 && + (!trace.program || trace.clients !== trace.program.clients)) || (trace.clients !== undefined && (!Number.isInteger(trace.clients) || trace.clients < 2)) ) throw new Error("Invalid/empty replay trace"); return { clients: trace.clients ?? 2, - description: `Seed ${trace.seed}; ${trace.iterations} generated conflict scenarios`, - steps: trace.steps + description: `Seed ${trace.seed}; ${trace.iterations} generated rounds`, + steps: + trace.version === 2 ? compileProgram(trace.program!) : trace.steps }; } + +export const generateTrace = generateStatefulTrace; diff --git a/frontend/test-support/README.md b/frontend/test-support/README.md index 41ff9a30..15d20343 100644 --- a/frontend/test-support/README.md +++ b/frontend/test-support/README.md @@ -2,25 +2,29 @@ Test-only code shared by the deterministic and seeded clients. -| Suite | Coverage | -| --- | --- | -| `harness.test.ts`, `oracle-regressions.test.ts` | Independent content/identity oracles, exercised faults, exclusive operations, persistence ownership, cleanup and seeds | -| `file-application.test.ts` | Swaps, file/directory transitions, visible conflict files, live edits, exclusions, and one metadata save per application | -| `recovery.test.ts` | Interrupt every application filesystem/save boundary, reopen metadata, modify disk, and verify stable ordinary scans; manifest policy and notification identity | -| `engine-regressions.test.ts`, `sync-audit.test.ts` | Offline identities, unknown network outcomes, rejection/retry, exclusions, interrupted in-memory pagination, namespace races and lifecycle changes | -| `event-delivery.test.ts`, `polling.test.ts` | WebSocket hints with HTTP-only event reads, missed notifications, opt-in polling, shutdown and retry | -| `protocol.test.ts` | Abrupt worker termination after accepted requests, exact retries, real server SIGKILL, CAS and event ordering | -| `restart-reconciliation.test.ts` | Interrupted file application followed by user edits, two-client/server convergence and a second restart | -| `server-restore.test.ts` | History checkpoints, server backup restores and interrupted metadata resets | -| `content-limits.test.ts`, `protocol-audit.test.ts` | Content validation and API invariants | -| `real-disk.test.ts` | Production Node adapter direct writes, no-replace moves, unsafe links, and rename cycles | -| `production-adapters.test.ts` | Actual CLI processes, raw watchers, Obsidian adapter, complete metadata and restart after SIGKILL | +| Suite | Coverage | +| -------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `harness.test.ts`, `hardening.test.ts`, `oracle-regressions.test.ts` | Independent content/identity oracles, corruption/duplication mutation probes, exercised faults, exclusive operations, persistence ownership, cleanup and seeds | +| `file-application.test.ts` | Swaps, file/directory transitions, visible conflict files, live edits, exclusions, and one metadata save per application | +| `recovery.test.ts` | Interrupt every application filesystem/save boundary, reopen metadata, modify disk, and verify stable ordinary scans; manifest policy and notification identity | +| `engine-regressions.test.ts`, `sync-audit.test.ts` | Offline identities, unknown network outcomes, rejection/retry, exclusions, interrupted in-memory pagination, namespace races and lifecycle changes | +| `event-delivery.test.ts`, `polling.test.ts` | WebSocket hints with HTTP-only event reads, missed notifications, opt-in polling, shutdown and retry | +| `protocol.test.ts` | Abrupt worker termination after accepted requests, exact retries, real server SIGKILL, CAS and event ordering | +| `restart-reconciliation.test.ts` | Interrupted file application followed by user edits, two-client/server convergence and a second restart | +| `server-restore.test.ts` | History checkpoints, server backup restores and interrupted metadata resets | +| `content-limits.test.ts`, `protocol-audit.test.ts` | Content validation and API invariants | +| `real-disk.test.ts` | Production Node adapter direct writes, no-replace moves, unsafe links, and rename cycles | +| `production-adapters.test.ts` | Actual CLI processes, raw watchers, Obsidian adapter, complete metadata and restart after SIGKILL | -The filesystem interruption tests assert the supported contract: complete metadata -remains usable, scans observe the current files, and a second scan is stable. -They do not demand the pre-interruption file contents, identities or a replay of -unfinished operations. Normal-operation tests still check exact bytes and identities. -There is no filesystem journal or hidden recovery archive. +The filesystem interruption tests require complete metadata to remain usable, +ordinary scans to observe current files, and a second scan to remain stable. They +also carry independent preservation obligations across every interruption: +unreplaced/moved bytes must survive exactly once, unrelated durable identities must +survive, and edits made after interruption must remain intact. The two-client +restart regressions check the explicit `USER EDIT` and `AFTER INTERRUPTION` markers +exactly once on both replicas through a second restart. An interrupted authorized +replacement need not retain the replaced bytes or replay unfinished operations; +there is no filesystem journal or hidden recovery archive. `MemoryDisk` exposes `before:`, `visible:` and `durable:` boundaries. A process interruption retains visible changes; a simulated power loss rolls unflushed @@ -30,7 +34,7 @@ is not an APFS/ext4 writeback model or a production durability requirement. after replacement. Abandoned filesystem sessions are fenced against later writes. `crash-worker.ts` runs a complete client in a worker, which the parent terminates -at accepted-response checkpoints without graceful shutdown. A new runtime receives +at accepted-response or filesystem/persistence checkpoints without graceful shutdown. A new runtime receives the saved disk and metadata. Real-disk tests exercise the production Node adapter. The production integration suite launches built CLI processes and an Obsidian filesystem adapter against a real server. Obsidian unit tests exercise unsaved editor diff --git a/frontend/test-support/checkpoint.ts b/frontend/test-support/checkpoint.ts new file mode 100644 index 00000000..8f79b74a --- /dev/null +++ b/frontend/test-support/checkpoint.ts @@ -0,0 +1,45 @@ +import assert from "node:assert/strict"; + +/** A one-shot, observable scheduling boundary. An unexercised gate is a failure. */ +export class Checkpoint { + private readonly reached = Promise.withResolvers(); + private readonly released = Promise.withResolvers(); + public fired = false; + public resumed = false; + public value?: T; + + public async enter(value: T, signal?: AbortSignal | null): Promise { + assert(!this.fired, "Checkpoint entered twice"); + this.fired = true; + this.value = structuredClone(value); + this.reached.resolve(this.value); + signal?.throwIfAborted(); + let abort = () => {}; + try { + await Promise.race([ + this.released.promise, + new Promise((_, reject) => { + abort = () => + reject( + signal?.reason ?? new Error("Checkpoint aborted") + ); + signal?.addEventListener("abort", abort, { once: true }); + }) + ]); + } finally { + signal?.removeEventListener("abort", abort); + } + } + + public wait(): Promise { + return this.reached.promise; + } + public resume(): void { + this.resumed = true; + this.released.resolve(); + } + public assertConsumed(): void { + assert(this.fired, "Checkpoint never reached"); + assert(this.resumed, "Checkpoint never released"); + } +} diff --git a/frontend/test-support/hardening.test.ts b/frontend/test-support/hardening.test.ts new file mode 100644 index 00000000..54283b5c --- /dev/null +++ b/frontend/test-support/hardening.test.ts @@ -0,0 +1,140 @@ +import assert from "node:assert/strict"; +import { test } from "node:test"; +import { NetworkFaults } from "./network"; +import { AssertableState } from "../deterministic-tests/src/utils/assertable-state"; +import { TESTS } from "../deterministic-tests/src/test-registry"; +import type { ExpectedDocument } from "../deterministic-tests/src/test-definition"; + +const id = "00000000-0000-4000-8000-000000000001"; +function state( + files: Record, + identities = Object.fromEntries( + Object.keys(files).map((path, i) => [ + i ? id.replace(/1$/, String(i + 1)) : id, + path + ]) + ) +) { + const map = new Map(Object.entries(files)); + return new AssertableState({ + files: map, + clientFiles: [map, new Map(map)], + canonical: identities, + manifests: [identities, identities], + bytes: new Map( + [...map].map(([path, text]) => [ + path, + new TextEncoder().encode(text) + ]) + ) + }); +} + +test("scripted verifiers reject the concrete shared-loss and duplication counterexamples", () => { + const cases = [ + [ + "multi-file-operations", + { "B.md": "updated", "C.md": "CORRUPTED", "unexpected.md": "EXTRA" } + ], + [ + "concurrent-edit-exact-same-position", + { "doc.md": "slow slow fast brown fox CORRUPTED" } + ] + ] as const; + for (const [name, files] of cases) { + const step = TESTS[name]!.steps.filter( + (s) => s.type === "assert-consistent" + ).at(-1)!; + assert.throws(() => step.verify!(state(files))); + } + const partial = state({ "a.md": "A", "b.md": "B" }); + partial.assertContent("a.md", "A"); + assert.throws(() => partial.assertComplete(), /complete file set/); +}); + +test("complete document oracles kill lost-byte, duplicate-file and identity-reuse mutations", () => { + const expected: ExpectedDocument[] = [ + { key: "a", path: "a.md", content: "exact bytes 🦊\r\n" } + ]; + const identities = new Map(); + state({ "a.md": "exact bytes 🦊\r\n" }).assertDocuments( + expected, + identities + ); + const corruptions: Record[] = [ + { "a.md": "exact bytes 🦊\n" }, + { "a.md": "exact bytes 🦊\r\nexact bytes 🦊\r\n" }, + { "a.md": "exact bytes 🦊\r\n", "extra.md": "EXTRA" } + ]; + for (const corrupted of corruptions) + assert.throws(() => + state(corrupted).assertDocuments(expected, identities) + ); + assert.throws( + () => + state( + { "a.md": "exact bytes 🦊\r\n" }, + { [id.replace(/1$/, "2")]: "a.md" } + ).assertDocuments(expected, identities), + /identity changed/ + ); + assert.throws( + () => + state({ "a.md": "exact bytes 🦊\r\n" }).assertDocuments( + [{ ...expected[0], key: "recreated" }], + identities + ), + /reused/ + ); +}); + +test("request checkpoints hold the exact boundary and reject unexercised response predicates", async () => { + const network = new NetworkFaults(); + let sent = 0, + completed = false; + const input = "http://localhost/vaults/test/documents/document"; + const init = { + method: "PUT", + body: JSON.stringify({ requestId: "one", parentVersionId: 1 }) + }; + const fetcher = network.wrap(async () => { + sent++; + return Response.json({ type: "StaleBase" }); + }); + network.hold({ kind: "content", point: "before" }); + assert.throws(() => network.resumeRequest(), /before it is reached/); + const request = fetcher(input, init).then(() => { + completed = true; + }); + await network.waitForHeldRequest(); + assert.equal(sent, 0); + assert.equal(completed, false); + assert.throws(() => network.assertConsumed(), /never released/); + network.resumeRequest(); + await request; + network.assertConsumed(); + assert.equal(network.requests[0].responseType, "StaleBase"); + network.hold({ kind: "content", point: "after", responseType: "Accepted" }); + assert.equal( + network.requestsSinceCheckpoint().length, + 0, + "An earlier response must not satisfy a new race assertion" + ); + await fetcher(input, init); + assert.throws(() => network.assertConsumed(), /never reached/); + network.releaseRequests(); +}); + +test("scripted races have no sleep steps and the stale-content regression asserts its response", () => { + for (const [name, scenario] of Object.entries(TESTS)) + assert( + !scenario!.steps.some((s) => s.type === "sleep"), + `${name} still relies on timing` + ); + assert( + TESTS["merging-update-response-survives-user-rename"]!.steps.some( + (s) => + s.type === "assert-response" && s.responseType === "StaleBase" + ) + ); +}); diff --git a/frontend/test-support/harness.test.ts b/frontend/test-support/harness.test.ts index 515604a3..15b77d28 100644 --- a/frontend/test-support/harness.test.ts +++ b/frontend/test-support/harness.test.ts @@ -84,7 +84,7 @@ for (const phase of ["before", "visible", "durable"] as const) { }; await assert.rejects(session.rename("a", "b"), /crash/); assert(fired); - disk.boundary = () => { }; + disk.boundary = () => {}; const moved = phase === "durable" || (phase === "visible" && !powerLoss); assert.deepEqual([...disk.userFiles().keys()], [moved ? "b" : "a"]); @@ -168,10 +168,13 @@ test("unconsumed and before-send faults cannot silently pass", async () => { ); assert(!called); assert.throws(() => net.assertConsumed(), /never retried/); - await net.wrap(async () => new Response())("http://test/file-manifest", { - method: "PUT", - body: '{"requestId":"r"}' - }); + await net.wrap(async () => Response.json({ type: "Accepted" }))( + "http://test/file-manifest", + { + method: "PUT", + body: '{"requestId":"r"}' + } + ); net.assertConsumed(); }); @@ -231,7 +234,7 @@ test("seeded actions are repeatable, including seed zero", () => { test("runner cleanup errors fail an otherwise successful test", async () => { const server = { isRunning: () => true, - resume: () => { } + resume: () => {} } as unknown as ServerControl; const runner = new TestRunner( server, @@ -241,13 +244,16 @@ test("runner cleanup errors fail an otherwise successful test", async () => { ); let cleaned = 0; Object.assign(runner, { - initializeAgents: async () => { }, + initializeAgents: async () => {}, agents: [ { clientId: 0, database: () => ({ evidence: "before cleanup" }), disk: { image: () => ({ bytes: [0, 255] }) }, - network: { requests: [{ body: { requestId: "original" } }] }, + network: { + requests: [{ body: { requestId: "original" } }], + schedule: [] + }, cleanup: async () => { cleaned++; throw new Error("late background error"); @@ -267,7 +273,8 @@ test("runner cleanup errors fail an otherwise successful test", async () => { client: 0, database: { evidence: "before cleanup" }, disk: { bytes: [0, 255] }, - requests: [{ body: { requestId: "original" } }] + requests: [{ body: { requestId: "original" } }], + schedule: [] } ]); }); @@ -282,7 +289,7 @@ test("disposed harness rejects late websocket reconnects", async () => { test("runner preserves primary failure and cleanup diagnostics", async () => { const server = { isRunning: () => true, - resume: () => { } + resume: () => {} } as unknown as ServerControl; const runner = new TestRunner( server, @@ -291,7 +298,7 @@ test("runner preserves primary failure and cleanup diagnostics", async () => { "http://unused" ); Object.assign(runner, { - initializeAgents: async () => { }, + initializeAgents: async () => {}, executeStep: async () => { throw new Error("primary assertion failed"); }, @@ -300,7 +307,7 @@ test("runner preserves primary failure and cleanup diagnostics", async () => { clientId: 0, database: () => ({}), disk: { image: () => ({}) }, - network: { requests: [] }, + network: { requests: [], schedule: [] }, cleanup: async () => { throw new Error("cleanup failed too"); } diff --git a/frontend/test-support/network.ts b/frontend/test-support/network.ts index 7df61083..9d9cd6b5 100644 --- a/frontend/test-support/network.ts +++ b/frontend/test-support/network.ts @@ -1,4 +1,5 @@ import assert from "node:assert/strict"; +import { Checkpoint } from "./checkpoint"; export type RequestKind = "create" | "content" | "manifest"; export interface RequestRecord { @@ -7,6 +8,13 @@ export interface RequestRecord { url: string; body: Record; status?: number; + responseType?: string; +} + +export interface RequestCheckpoint { + kind: RequestKind | "read-content" | "events"; + point: "before" | "after"; + responseType?: string; } interface InjectedFault { @@ -52,6 +60,79 @@ export function classifyRequest( export class NetworkFaults { public readonly requests: RequestRecord[] = []; + public readonly schedule: { type: string; detail: unknown }[] = []; + private readonly checkpoints: { + spec: RequestCheckpoint; + gate: Checkpoint; + firstRequest: number; + }[] = []; + public hold(spec: RequestCheckpoint): void { + assert( + spec.point === "after" || spec.responseType === undefined, + "Response predicates require an after-response checkpoint" + ); + this.checkpoints.at(-1)?.gate.assertConsumed(); + this.checkpoints.push({ + spec, + gate: new Checkpoint(), + firstRequest: this.requests.length + }); + this.schedule.push({ type: "armed", detail: spec }); + } + public waitForHeldRequest(): Promise { + const current = this.checkpoints.at(-1); + assert(current, "No request checkpoint armed"); + return current.gate.wait(); + } + public requestsSinceCheckpoint(): RequestRecord[] { + const current = this.checkpoints.at(-1); + assert(current, "No request checkpoint armed"); + return this.requests.slice(current.firstRequest); + } + public resumeRequest(): void { + const current = this.checkpoints.at(-1); + assert(current, "No request checkpoint armed"); + assert( + current.gate.fired, + "Cannot release a request checkpoint before it is reached" + ); + current.gate.resume(); + this.schedule.push({ type: "released", detail: current.spec }); + } + public releaseRequests(): void { + for (const { gate } of this.checkpoints) gate.resume(); + } + public assertCheckpointsConsumed(): void { + for (const { gate } of this.checkpoints) gate.assertConsumed(); + assert( + !this.observation || + (this.observation.fired && this.observation.released), + "Observation checkpoint was not exercised and released" + ); + } + private async requestCheckpoint( + kind: RequestCheckpoint["kind"] | undefined, + point: RequestCheckpoint["point"], + detail: unknown, + signal?: AbortSignal | null, + responseType?: string + ): Promise { + const current = this.checkpoints.at(-1); + if ( + !current || + current.gate.fired || + current.spec.kind !== kind || + current.spec.point !== point || + (current.spec.responseType !== undefined && + current.spec.responseType !== responseType) + ) + return; + this.schedule.push({ + type: "reached", + detail: structuredClone({ spec: current.spec, request: detail }) + }); + await current.gate.enter(detail, signal); + } private observation?: { released: boolean; fired: boolean; @@ -150,11 +231,7 @@ export class NetworkFaults { await this.armed.reached; } public assertConsumed(): void { - assert( - !this.observation || - (this.observation.fired && this.observation.released), - "Observation checkpoint was not exercised and released" - ); + this.assertCheckpointsConsumed(); for (const fault of this.faults) { assert(fault.fired, "Armed network fault never fired"); assert(fault.retried, "Interrupted CAS request was never retried"); @@ -175,6 +252,13 @@ export class NetworkFaults { (input instanceof Request ? input.signal : undefined); if (observing) await this.observationCheckpoint(signal); const record = classifyRequest(input, init); + const kind = + record?.kind ?? + (observing && /\/documents\/.*\/content$/.test(url.pathname) + ? "read-content" + : observing && url.pathname.endsWith("/events-since") + ? "events" + : undefined); if (record) { // A client serializes its persisted CAS requests. Its next CAS // after a lost response must replay the interrupted operation, @@ -192,6 +276,10 @@ export class NetworkFaults { "Retry changed payload/request identity" ); this.armed!.retried = true; + this.schedule.push({ + type: `retry:${record.kind}`, + detail: structuredClone(record) + }); } const previous = this.requests.find( (r) => r.body.requestId === record.body.requestId @@ -216,23 +304,45 @@ export class NetworkFaults { const fail = () => { fault!.interrupted = structuredClone(record!); fault!.fired = true; + this.schedule.push({ + type: `drop:${fault!.kind}:${fault!.point}`, + detail: structuredClone(record) + }); fault!.resolve(); throw new TypeError( `Injected ${fault!.point}-commit ${fault!.kind} network failure` ); }; if (matches && fault.point === "before") fail(); + await this.requestCheckpoint( + kind, + "before", + record ?? { url: url.href }, + signal + ); const response = await fetch(input, init); if (observing) await this.observationCheckpoint(signal); - if (record) record.status = response.status; + if (record) { + record.status = response.status; + if (response.ok) + record.responseType = ( + (await response.clone().json()) as { type?: string } + ).type; + } + await this.requestCheckpoint( + kind, + "after", + record ?? { url: url.href }, + signal, + record?.responseType + ); // Only drop accepted replies, never confuse a rejected CAS with commit. if ( matches && !fault.fired && fault.point === "after" && response.ok && - ((await response.clone().json()) as { type?: string }).type === - "Accepted" + record?.responseType === "Accepted" ) { await response.body?.cancel(); fail(); diff --git a/frontend/test-support/recovery.test.ts b/frontend/test-support/recovery.test.ts index d948603f..79d4967d 100644 --- a/frontend/test-support/recovery.test.ts +++ b/frontend/test-support/recovery.test.ts @@ -35,10 +35,7 @@ async function fixture( } for (const [id, path] of Object.entries(initial.local)) initial.documents[id] = { - - observedHash: ( - await hashSnapshot((await disk.read(path))!) - ).hash + observedHash: (await hashSnapshot((await disk.read(path))!)).hash }; const persistence = new MemoryPersistence({ database: initial }); const restart = (changes: LocalChange[] = []) => { @@ -55,7 +52,7 @@ async function fixture( database, config, undefined, - { entries: () => changes, flush: async () => { } } + { entries: () => changes, flush: async () => {} } ) }; }; @@ -64,7 +61,10 @@ async function fixture( persistence, restart, initial, - permanentFault: (_label: string): void => { }, + allowedInterruptedContent: undefined as + | { path: string; variants: Uint8Array[] } + | undefined, + permanentFault: (_label: string): void => {}, expectedLocal: undefined as Record | undefined }; } @@ -126,6 +126,13 @@ async function mergeFixture() { const current = await fixture({ a: { path: "note.md", text: local } }); return { ...current, + allowedInterruptedContent: { + path: "note.md", + variants: [ + bytes(local), + bytes("local first paragraph\n\nremote last paragraph\n") + ] + }, next: structuredClone(current.initial), writes: { a: { @@ -201,7 +208,27 @@ for (const [name, create] of [ ["permanent destination fallback", permanentDestinationFixture] ] as const) { test(`${name}: interrupted operations leave metadata usable by ordinary scans`, async (t) => { - const baseline = await create(); + const protectedFixture = async () => { + const current = await create(); + const content = bytes("UNTOUCHED DURABLE BYTES\0\xff"); + await current.disk.userWrite("untouched.bin", content); + const stored = current.persistence.snapshot(); + stored.database!.local!.protected = "untouched.bin"; + stored.database!.documents!.protected = { + observedHash: (await hashSnapshot({ content })).hash + }; + await current.persistence.save(stored); + current.next.local.protected = "untouched.bin"; + current.next.documents.protected = { + observedHash: (await hashSnapshot({ content })).hash + }; + if (current.expectedLocal) + (current.expectedLocal as Record).protected = + "untouched.bin"; + current.expected.set("untouched.bin", content); + return current; + }; + const baseline = await protectedFixture(); const labels: string[] = []; const trace = (label: string) => { if (!label.startsWith("read:")) labels.push(label); @@ -211,7 +238,9 @@ for (const [name, create] of [ trace(label); }; baseline.persistence.boundary = trace; - await baseline.restart().files.applyChanges(baseline.next, baseline.writes); + await baseline + .restart() + .files.applyChanges(baseline.next, baseline.writes); assert.deepEqual(baseline.disk.userFiles(), baseline.expected); assert.deepEqual( baseline.persistence.snapshot().database!.local, @@ -223,7 +252,55 @@ for (const [name, create] of [ await t.test( `${powerLoss ? "power" : "process"} interruption at ${labels[cut]}`, async () => { - const current = await create(); + const current = await protectedFixture(); + // Moving a file does not authorize losing its bytes. Writes + // and explicit deletions have separate replacement policies. + const required = Object.entries( + current.persistence.snapshot().database!.local! + ) + .filter( + ([id]) => + id in current.next.local && + !(id in current.writes) + ) + .map( + ([, path]) => + current.disk.userFiles().get(path)! + ); + const preserve = () => { + const allowed = current.allowedInterruptedContent; + if (allowed) { + const content = current.disk + .userFiles() + .get(allowed.path); + assert( + content && + allowed.variants.some((variant) => + Buffer.from(variant).equals( + Buffer.from(content) + ) + ), + "Interrupted merge lost or corrupted the user's pre-existing edit" + ); + } + const files = [ + ...current.disk.userFiles().values() + ].map((value) => Buffer.from(value)); + for (const content of required) + assert.equal( + files.filter((value) => + value.equals(Buffer.from(content)) + ).length, + 1, + `Interrupted ${name} lost or duplicated existing bytes at ${labels[cut]}` + ); + assert.deepEqual( + current.disk + .userFiles() + .get("after-restart.md"), + bytes("user edit") + ); + }; let seen = 0; let fired = false; const interrupt = (label: string) => { @@ -242,18 +319,22 @@ for (const [name, create] of [ await assert.rejects( current .restart() - .files.applyChanges(current.next, current.writes), + .files.applyChanges( + current.next, + current.writes + ), /interrupted/ ); assert(fired); - current.disk.boundary = () => { }; - current.persistence.boundary = () => { }; + current.disk.boundary = () => {}; + current.persistence.boundary = () => {}; // The user may edit after the interruption, including a partial // write or a file whose metadata was never saved. await current.disk.userWrite( "after-restart.md", bytes("user edit") ); + preserve(); const runtime = current.restart(); const observed = current.disk.userFiles(); await scanChanges(runtime, []); @@ -263,6 +344,12 @@ for (const [name, create] of [ "A scan must not replay old filesystem intentions" ); assertManifest(runtime.database.state.local); + preserve(); + assert.equal( + runtime.database.state.local.protected, + "untouched.bin", + "Unrelated durable identity changed" + ); assert.deepEqual( new Set( Object.values(runtime.database.state.local) @@ -294,6 +381,7 @@ for (const [name, create] of [ state, "Second startup must be stable" ); + preserve(); } ); } @@ -399,7 +487,7 @@ test("scan failures cannot masquerade as deletions", async () => { database: runtime.database, changes: [], files: runtime.files, - guard: () => { }, + guard: () => {}, ignored: () => false, oversized: () => false }), @@ -426,7 +514,7 @@ for (const phase of ["before", "durable"]) { database: runtime.database, changes, files: runtime.files, - guard: () => { }, + guard: () => {}, ignored: () => false, oversized: () => false }); @@ -436,7 +524,7 @@ for (const phase of ["before", "durable"]) { if (label === `${phase}:save`) throw new Error("save failure"); }; await assert.rejects(scan(), /save failure/); - current.persistence.boundary = () => { }; + current.persistence.boundary = () => {}; const state = await scan(); assert.equal( state.local.a, @@ -459,7 +547,7 @@ async function scanChanges( database: runtime.database, changes, files: runtime.files, - guard: () => { }, + guard: () => {}, ignored: () => false, oversized: () => false }); @@ -540,7 +628,7 @@ for (const equalContent of [false, true]) { phase === "durable" ? expected : current.initial.local, "The fault must occur on the intended side of the durable save" ); - current.persistence.boundary = () => { }; + current.persistence.boundary = () => {}; // Model an uncertain save in a live client: durable state reloads, // but the original editor notifications remain in memory. Do not // clear, reconstruct or deduplicate that queue in the harness. @@ -574,7 +662,7 @@ test("normalization-sensitive disk does not lose a distinct decomposed filename" database: runtime.database, changes: [], files: runtime.files, - guard: () => { }, + guard: () => {}, ignored: () => false, oversized: () => false }); diff --git a/frontend/test-support/restart-reconciliation.test.ts b/frontend/test-support/restart-reconciliation.test.ts index 897b3627..fc40200a 100644 --- a/frontend/test-support/restart-reconciliation.test.ts +++ b/frontend/test-support/restart-reconciliation.test.ts @@ -4,6 +4,7 @@ import { fixture, bytes, randomUUID } from "./sync-server-fixture"; import { assertCanonical } from "./canonical"; import { SyncClient } from "../sync-client/src/sync-client"; import { MemoryDisk, MemoryPersistence } from "./storage"; +import { ContentLedger } from "./oracles"; for (const powerLoss of [false, true]) for (const operation of ["create", "replace", "move", "metadata"] as const) @@ -98,6 +99,9 @@ for (const powerLoss of [false, true]) files: () => f.disk.userFiles(), database: () => f.store.snapshot().database }; + const edits = new ContentLedger(); + edits.add("USER EDIT"); + edits.add("AFTER INTERRUPTION"); const peerDisk = new MemoryDisk(); const peerStore = new MemoryPersistence({ settings: f.store.snapshot().settings @@ -119,6 +123,8 @@ for (const powerLoss of [false, true]) f.url, "test-token-change-me" ); + for (const client of clients) + edits.assertPreserved(client.files()); assert.deepEqual( f.disk.userFiles().get("after.md"), bytes("AFTER INTERRUPTION\n") @@ -141,6 +147,8 @@ for (const powerLoss of [false, true]) files, "A second restart must not replay old writes" ); + for (const client of clients) + edits.assertPreserved(client.files()); } finally { f.disk.boundary = () => {}; f.store.boundary = () => {}; diff --git a/scripts/e2e-runner.mjs b/scripts/e2e-runner.mjs index b626a5c9..7436b379 100644 --- a/scripts/e2e-runner.mjs +++ b/scripts/e2e-runner.mjs @@ -1,9 +1,12 @@ +import { randomInt } from "node:crypto"; import { spawn } from "node:child_process"; import { createWriteStream, mkdirSync, mkdtempSync, writeFileSync, + readdirSync, + readFileSync, } from "node:fs"; import { tmpdir } from "node:os"; import { dirname, join, resolve } from "node:path"; @@ -12,10 +15,13 @@ import { fileURLToPath } from "node:url"; export function parseOptions(env = process.env) { const workers = positive(env.E2E_WORKERS ?? "1", "E2E_WORKERS"); - const seed = Number(env.E2E_SEED ?? "1"); + if (workers > 32) throw new Error("E2E_WORKERS must be at most 32"); + const seedValue = + env.E2E_SEED ?? String(randomInt(0, 0x100000000 - workers + 1)); + const seed = Number(seedValue); if ( - !/^\d+$/.test(env.E2E_SEED ?? "1") || + !/^\d+$/.test(seedValue) || !Number.isInteger(seed) || seed < 0 || seed + workers - 1 > 0xffffffff @@ -125,7 +131,7 @@ function positive(value, label) { ) throw new Error(`${label} must be a positive integer`); return result; -}; +} async function main() { const options = parseOptions(); @@ -187,17 +193,7 @@ async function main() { throw new Error("Server build failed"); } - if ( - !(await run( - "build-clients", - "npm", - [ - "run", - "build", - ], - frontend, - )) - ) { + if (!(await run("build-clients", "npm", ["run", "build"], frontend))) { throw new Error( "Client build failed (run npm ci in frontend if dependencies are missing)", ); @@ -215,7 +211,21 @@ async function main() { ["run", "test", "--workspace", "sync-client"], frontend, ); - await run("adapter-unit-tests", "npm", ["run", "test"], frontend); + await run( + "adapter-unit-tests", + "npm", + [ + "run", + "test", + "--workspace", + "vault-link-obsidian-plugin", + "--workspace", + "local-client-cli", + "--workspace", + "test-client", + ], + frontend, + ); await run("harness-self-tests", "npm", ["run", "test:harness"], frontend); await run("crash-and-protocol", "npm", ["run", "test:protocol"], frontend); @@ -225,6 +235,21 @@ async function main() { String(options.workers), ]); + // Concrete regression traces are stable even when the generator evolves. + const corpus = join(frontend, "test-client/corpus"); + for (const file of readdirSync(corpus) + .filter((file) => file.endsWith(".json")) + .sort()) { + const name = `corpus-${file.replace(/\.json$/, "")}`; + await run(name, process.execPath, [ + join(frontend, "test-client/dist/cli.js"), + "--replay", + join(corpus, file), + "--artifacts", + join(artifacts, name), + ]); + } + // Execute all suites even if one reports a regression; failure is never // inferred from log text and successful workers cannot mask failed ones. await Promise.all( @@ -236,10 +261,29 @@ async function main() { "--iterations", String(options.iterations), "--artifacts", - artifacts, + join(artifacts, `fuzz-${options.seed + i}`), ]), ), ); + const coverage = {}; + for (const entry of readdirSync(artifacts, { withFileTypes: true }).filter( + (entry) => entry.isDirectory(), + )) { + const directory = join(artifacts, entry.name); + for (const file of readdirSync(directory).filter((file) => + /^result-.*\.json$/.test(file), + )) { + const result = JSON.parse( + readFileSync(join(directory, file), "utf8"), + ); + for (const [key, count] of Object.entries(result.coverage ?? {})) + coverage[key] = (coverage[key] ?? 0) + count; + } + } + writeFileSync( + join(artifacts, "coverage.json"), + JSON.stringify(coverage, null, 2), + ); if (interrupted || results.some((result) => !result.success)) { throw new Error( `Tests failed; inspect ${join(artifacts, "results.json")}`, diff --git a/scripts/e2e-runner.test.mjs b/scripts/e2e-runner.test.mjs index 84ac0f04..b35cbcce 100644 --- a/scripts/e2e-runner.test.mjs +++ b/scripts/e2e-runner.test.mjs @@ -149,3 +149,30 @@ test("E2E includes both peers' unit suites and the permanent audit regressions", ), ); }); + +test("campaigns support replayable explicit seeds and fresh default seeds", () => { + const explicit = parseOptions({ E2E_SEED: "42", E2E_WORKERS: "8" }); + assert.equal(explicit.seed, 42); + const fresh = parseOptions({ E2E_WORKERS: "8" }); + assert( + Number.isInteger(fresh.seed) && + fresh.seed >= 0 && + fresh.seed + 7 <= 0xffffffff, + ); + assert.throws(() => parseOptions({ E2E_WORKERS: "33" }), /at most 32/); + assert.throws( + () => parseOptions({ E2E_WORKERS: "2", E2E_SEED: "4294967295" }), + /uint32/, + ); + const runner = readFileSync( + new URL("./e2e-runner.mjs", import.meta.url), + "utf8", + ); + assert.match(runner, /test-client\/corpus/); + assert.match(runner, /coverage\.json/); + const workflow = readFileSync( + new URL("../.forgejo/workflows/e2e.yml", import.meta.url), + "utf8", + ); + assert.match(workflow, /E2E_ITERATIONS=100/); +});