andras/vault-link
andras/vault-link
Watch
1
0
Fork
You've already forked vault-link
 0
Code Issues Pull requests 18 Projects Releases 55 Packages Wiki Activity Actions 33

power loss

Browse source
This commit is contained in:
Andras Schmelczer 2026-09-21 21:29:52 +01:00
commit d56084e4bc
82 changed files with 18667 additions and 3902 deletions
Download patch file Download diff file Expand all files Collapse all files

224
frontend/deterministic-tests/README.md
View file

@ -1,118 +1,134 @@
# Deterministic Tests
# Sync test harnesses (API v4)
Scripted multi-client (with an in-memory filesystem) sync tests that run against a real server. Each test defines a sequence of file operations, sync/server controls, and assertions to exercise a specific conflict or edge case.
Complements the fuzz-based E2E tests (`test-client`): fuzz tests discover bugs through random operations; deterministic tests pin down exact reproduction sequences for known scenarios.
## How it works
Each test is a `TestDefinition`: a client count and an ordered list of steps. The test name is derived from the registry key (which matches the file name). The `TestRunner` spins up N `DeterministicAgent` instances (each wrapping a real `SyncClient` with an `InMemoryFileSystem`) pointed at a shared vault on the server, then executes steps one by one.
Tests that don't pause the server share a single server process (vault-name isolation). Tests that use `pause-server`/`resume-server` (SIGSTOP/SIGCONT) each get a dedicated server, since SIGSTOP freezes the entire process.
The runner executes two sequential phases: regular tests on the shared server, then pause-server tests on dedicated servers. Within each phase tests run in parallel up to a concurrency limit.
## Step types
Clients always start with syncing disabled.
**File operations** (per-client, fire-and-forget — sync is enqueued but not awaited):
- `create`, `update`, `rename`, `delete`
- `rename-next-write` — arm a deferred rename that fires the next time the given path is written. Lets a test race a user-rename against an in-flight remote create that's about to land at the same path.
**Sync control:**
- `sync` — wait for a specific client or all clients to finish pending operations
- `barrier` — retry until all clients converge to identical file state (60s timeout)
- `enable-sync` / `disable-sync` — simulate going online/offline
- `reset` — reset a client's tracked sync state (keeps disk files); equivalent to a forced re-handshake on next enable
- `sleep` — wall-clock pause; use sparingly, prefer `barrier` / `sync`
**WebSocket control** (per-client):
- `pause-websocket` / `resume-websocket` — buffer/release WebSocket messages for a specific client
**Server control:**
- `pause-server` / `resume-server` — SIGSTOP/SIGCONT the server process
- `resume-server-until-history-then-pause` — resume the server, wait until a specific client observes a matching history entry (`CREATE`/`UPDATE`/`DELETE` for a path), then re-pause. Used to land exactly one operation across the wire.
**Fault injection** (per-client):
- `drop-next-create-response` — arm a one-shot interceptor that lets the next `POST /documents` reach the server (commit happens) but throws `SyncResetError` before the client sees the response, simulating connection loss after server commit.
- `wait-for-dropped-create-response` — wait until the armed drop has fired.
**Assertions:**
- `assert-consistent` — all clients have identical files; optionally takes a custom `verify(state: AssertableState)` callback
## Running
Run the complete harness check from **any working directory**:
```sh
# Build server first
cd sync-server && cargo build --release && cd -
# Run all tests
cd frontend && npm run build -w sync-client && npm run test -w deterministic-tests
# Filter by name
npm run test -w deterministic-tests -- --filter=rename
# Control parallelism (default: number of CPU cores)
npm run test -w deterministic-tests -- -j 4
E2E_WORKERS=2 /path/to/vault-link/scripts/e2e.sh
```
## Adding a test
The `E2E_WORKERS` variable is the number of fuzz workers / simultaneous scripted tests.
The runner builds the server and the three in-scope frontend packages, then runs
the harness self-tests, recovery regressions, protocol/real-filesystem tests,
scripted scenarios, and seeded fuzz workers. It does not build the Obsidian plugin
or CLI application. Install frontend dependencies with `npm ci` first. Node's
built-in WebSocket, Rust, and a C compiler (`cc`, for test-only exclusive rename
and OS-lock primitives on macOS/Linux) are required.
1. Create `src/tests/my-scenario.test.ts`:
CI installs those dependencies in `frontend` before invoking the runner. It sets
`E2E_ARTIFACTS=logs/e2e` and uploads that same directory even when a suite fails.
The runner owns all test servers; no separate server or legacy cleanup script is
needed in the workflow.
```typescript
import type { TestDefinition } from "../test-definition";
Each server gets a fresh temporary database, working directory, and port. Only
owned processes are terminated. There is no `pkill`, fixed database deletion,
or shared server PID file. Failing tests and teardown errors give nonzero exits;
the remaining suites still run. Logs, concrete fuzz traces, results, and failure
snapshots are retained in the printed artifact directory. Snapshots are captured
before teardown so cleanup failures retain the original evidence.
export const myScenarioTest: TestDefinition = {
description:
"Client 0 creates A.md offline. After syncing, both clients should have the file.",
clients: 2,
steps: [
{ type: "create", client: 0, path: "A.md", content: "hello" },
{ type: "enable-sync", client: 0 },
{ type: "enable-sync", client: 1 },
{ type: "barrier" },
{
type: "assert-consistent",
verify: (s) => {
s.assertFileCount(1).assertContent("A.md", "hello");
}
}
]
};
Environment options:
- `E2E_WORKERS=1`: number of fuzz workers / simultaneous scripted tests (max 32).
- `E2E_SEED=1`: first uint32 seed; workers use successive seeds.
- `E2E_ITERATIONS=25`: generated conflict scenarios per worker.
- `E2E_TIMEOUT_SECONDS=900`: timeout for each build/suite/worker command.
- `E2E_ARTIFACTS=/absolute/path`: optional output directory; defaults to `mkdtemp`.
## Running individual layers
From `frontend`:
```sh
# Rebuild protocol peers and run scripted tests (name filter optional).
npm run test --workspace deterministic-tests -- --filter v4- --concurrency 2
# Adapter/oracle/lifecycle self-tests and strict recovery regressions.
npm run test:harness
# After rebuilding: real worker death, server SIGKILL/restart, CAS/WS ordering,
# and host filesystem contract checks.
npm run test:protocol
npm run build --workspace sync-client --workspace test-client
node test-client/dist/cli.js --seed 42 --iterations 25
node test-client/dist/cli.js --replay /path/to/trace-42.json
```
The `verify` callback receives an `AssertableState` object with chainable assertion methods:
`test:harness` includes strict regressions for **engine** bugs.
A failing default run is not an expected-failure test or a harness skip. See
[`../test-support/README.md`](../test-support/README.md) for their scope.
```typescript
s.assertFileCount(n); // exact file count
s.assertFileExists("path"); // file must exist
s.assertFileNotExists("path"); // file must not exist
s.assertContent("path", "expected"); // exact content match
s.assertContains("path", "a", "b"); // all substrings present in file
s.assertContainsAny("path", "a", "b"); // at least one substring present
s.assertAnyFileContains("text"); // substring present in some file
s.assertNoFileContains("text"); // substring absent from every file
s.assertSubstringCount("path", "x", 3); // substring appears exactly N times
s.assertContentInAtMostOneFile("text"); // no duplicate content
s.ifFileExists("path", (s) => { /* … */ }); // conditional block
s.getContent("path"); // raw content (or "" if missing)
```
## What the scripted runner checks
2. Register it in `src/test-registry.ts`:
Real `SyncClient` instances use the shared v4 adapter in `../test-support`.
External editor operations and engine filesystem operations are separate: engine
writes never generate self-notifications. Editor identity notifications continue
while transport is disabled. Delayed notifications can be explicitly queued and
flushed. Snapshots and persisted state are deep copies, never shared references.
```typescript
import { myScenarioTest } from "./tests/my-scenario.test";
Scripted and seeded clients explicitly set `syncIntervalMs: 0`, with assertions
that it stays disabled. Barriers observe convergence; they do not wake the engine.
Network retry timers remain enabled. Separate fake-clock tests in
`test-support/polling.test.ts` exercise optional periodic reconciliation.
const TESTS = {
// ...
"my-scenario": myScenarioTest
};
```
A barrier checks byte-for-byte replica equality **and** the server's canonical
UUID/path map, file bytes, independently computed base/observed hashes, content
and manifest CAS parents, event watermark,
contiguous event log, and absence of pending requests/application journals.
Internal recovery files are excluded from the user-visible file list, but are
present in failure disk images. Scenario assertions separately check intended
content, document identity, or preservation markers; equality alone is not an
adequate data-loss oracle.
Clients start disabled. `enable-sync` waits for the initial sync attempt/retries,
except while the server is intentionally paused: that attempt is tracked so a
later script step can resume the server. Thus sequential enables really establish
a winner. To test _independent creates_, first bootstrap both clients empty, then
disable them and create files. Initial adoption of pre-existing files during
bootstrap is a different policy from two already-initialized clients creating
distinct UUIDs at one path.
## Controls and assertions
- `create`, `update`, `create-bytes`, `rename`, `delete`: external editor actions.
Editor rename may explicitly replace its target; engine rename never may.
- `sync`, `barrier`, `enable-sync`, `disable-sync`.
- `reset`: restart transports; **not** a crash and not a database reset.
- `pause-websocket`, `resume-websocket`: buffer only WS delivery. HTTP catchup
remains active; this is deliberately not called "disconnect".
- `pause-observation`, `wait-for-observation`, `resume-observation`: explicitly
gate HTTP catchup as well as WS, including replies already in flight. The
checkpoint must be exercised and released, and held reads respect cancellation.
- `delay-notifications`, `flush-notifications`: delay editor notifications without
disabling preservation assertions.
- `drop-response`: `{client, kind: "create" | "content" | "manifest", point:
"before" | "after"}`. After faults fire only on an `Accepted` CAS response.
- `wait-for-response-drop`: explicit checkpoint; every armed fault must fire.
Every interrupted request must be retried with an identical request ID, target
and body before a different CAS is sent. Fault kinds match exactly.
- `rename-next-write`: inject an editor rename after the engine's durable install
rename. The historical name is retained for fixtures; v4 never writes in place.
- `pause-server`, `resume-server`: SIGSTOP/SIGCONT, not a crash.
- `crash-server`, `restart-server`: SIGKILL and restart with the same database/port.
Server-control scenarios have dedicated processes.
- `remember-identity`, `assert-identity`, `assert-files`, `assert-markers`.
- `assert-documents`: a complete expected set of `{key, path, content, conflict?}`
records. Content is text or a byte array. Keys bind UUIDs on the first successful
check and must retain them on later checks; new keys cannot reuse deleted UUIDs.
`conflict: true` resolves the UUID-derived conflict name for `path`.
- `assert-consistent` can provide `verify(state)`: exact text/bytes, UUID lookups,
`assertIdentity`, and `conflictPath(original)` for UUID-derived conflict names.
Legacy timing scenarios are retained. Prefer explicit fault checkpoints and
barriers for new tests. Only known coherent-scan races and actually injected
transient network errors may retry; they still must converge before the deadline.
Other background errors, unhandled rejections, and cleanup failures fail the run.
## Reproducibility limits
Seeds and JSON traces reproduce harness choices and logical actions. They do not
control the OS scheduler, server thread scheduling, or production-generated UUIDs.
The recovery sweep enumerates adapter/save boundaries; worker termination proves
abrupt runtime loss. Neither is a proof covering every real filesystem or hardware
power-loss behavior. Run multiple seeds, repeat timing-sensitive cases, and keep
the scenario's independent oracle when reducing a failing trace.

2
frontend/deterministic-tests/package.json
View file

@ -8,7 +8,7 @@
"scripts": {
"dev": "webpack watch --mode development",
"build": "webpack --mode production",
"test": "npm run build && node dist/cli.js"
"test": "cargo build --release --manifest-path ../../sync-server/Cargo.toml && npm run build --workspace sync-client && npm run build && node dist/cli.js"
},
"devDependencies": {
"commander": "^14.0.2",

32
frontend/deterministic-tests/src/cli.ts
View file

@ -9,10 +9,11 @@ import { runWithConcurrency } from "./run-with-concurrency";
import { TOKEN, SERVER_BINARY_PATH, CONFIG_PATH } from "./consts";
import * as path from "node:path";
import * as fs from "node:fs";
import { debugging, Logger } from "sync-client";
import * as os from "node:os";
import { Logger } from "sync-client";
const logger = new Logger();
debugging.logToConsole(logger, { useColors: true });
logger.onLogEmitted.add((line) => console.log(line.message));
process.on("unhandledRejection", (reason) => {
logger.error(`Unhandled Rejection: ${reason}`);
@ -32,6 +33,8 @@ function testUsesPauseServer(test: TestDefinition): boolean {
(step) =>
step.type === "pause-server" ||
step.type === "resume-server" ||
step.type === "crash-server" ||
step.type === "restart-server" ||
step.type === "resume-server-until-history-then-pause"
);
}
@ -42,7 +45,7 @@ function testUsesPauseServer(test: TestDefinition): boolean {
*/
function findProjectRoot(): string {
let dir = path.dirname(__filename);
const {root} = path.parse(dir);
const { root } = path.parse(dir);
while (dir !== root) {
if (
fs.existsSync(path.join(dir, "sync-server")) &&
@ -114,11 +117,7 @@ async function runDedicatedServerTest(
}
return { name, result };
} finally {
try {
await server.stop();
} catch {
// best-effort cleanup
}
await server.stop();
serverManager.untrack(server);
}
}
@ -138,6 +137,11 @@ async function main(): Promise<void> {
}
const { filter, concurrency } = parseArgs(process.argv);
const artifacts =
process.env.E2E_ARTIFACTS ??
fs.mkdtempSync(path.join(os.tmpdir(), "vault-link-deterministic-"));
fs.mkdirSync(artifacts, { recursive: true });
logger.info(`Artifacts: ${artifacts}`);
const testsToRun: [string, TestDefinition][] = [];
for (const [key, test] of Object.entries(TESTS)) {
@ -193,13 +197,7 @@ async function main(): Promise<void> {
allResults.push(...results);
} finally {
try {
await sharedServer.stop();
} catch (error) {
logger.warn(
`Error stopping shared server: ${error instanceof Error ? error.message : String(error)}`
);
}
await sharedServer.stop();
serverManager.untrack(sharedServer);
}
}
@ -221,6 +219,10 @@ async function main(): Promise<void> {
const passed = allResults.filter((r) => r.result.success);
const failed = allResults.filter((r) => !r.result.success);
fs.writeFileSync(
path.join(artifacts, "deterministic-results.json"),
JSON.stringify(allResults, null, 2)
);
logger.info(
`\n--- Results: ${passed.length}/${allResults.length} passed ---`

93
frontend/deterministic-tests/src/crash-worker.ts Normal file
View file

@ -0,0 +1,93 @@
import { parentPort, workerData } from "node:worker_threads";
import assert from "node:assert/strict";
import { SyncClient } from "sync-client";
import {
MemoryDisk,
MemoryPersistence,
type DiskImage,
type StoredClient
} from "../../test-support/storage";
import {
classifyRequest,
type RequestRecord,
type RequestKind
} from "../../test-support/network";
export interface CrashWorkerInput {
disk: DiskImage;
stored: StoredClient;
stopAt?: RequestKind;
editAfterSync?: { path: string; content: string };
}
const input = workerData as CrashWorkerInput;
const disk = new MemoryDisk();
disk.restore(input.disk);
const persistence = new MemoryPersistence(input.stored);
const requests: RequestRecord[] = [];
let editedBase: { documentId: string; vaultUpdateId: number } | undefined;
const snapshot = () => ({
disk: disk.image(),
stored: persistence.snapshot(),
requests,
editedBase
});
async function main() {
const client = await SyncClient.create({
fs: disk.session(),
persistence,
fetch: async (url, init) => {
const record = classifyRequest(url, init);
if (record) requests.push(record);
const response = await fetch(url, init);
if (record && response.ok && input.stopAt === record.kind) {
const receipt = (await response.clone().json()) as {
type: string;
};
if (receipt.type === "Accepted") {
parentPort!.postMessage({
type: "checkpoint",
...snapshot()
});
// Parent terminates this worker. No destroy(), no state flush,
// no finally block, and no old runtime survives the restart.
await new Promise<never>(() => {});
}
}
return response;
}
});
assert.equal(
client.getSettings().syncIntervalMs,
0,
"Crash recovery must preserve the disabled polling setting"
);
await client.start();
await client.waitUntilFinished();
if (input.editAfterSync) {
const state = persistence.snapshot().database!;
const documentId = Object.keys(state.local!).find(
(id) => state.local![id] === input.editAfterSync!.path
);
assert(documentId, "Edit must target an established document");
editedBase = {
documentId,
vaultUpdateId: state.documents![documentId].base!.vaultUpdateId
};
await disk.userWrite(
input.editAfterSync.path,
new TextEncoder().encode(input.editAfterSync.content)
);
await client.syncLocallyUpdatedFile({
relativePath: input.editAfterSync.path
});
await client.waitUntilFinished();
}
await client.destroy();
parentPort!.postMessage({ type: "finished", ...snapshot() });
parentPort!.close();
}
main().catch((error: unknown) => {
parentPort!.postMessage({ type: "error", error: String(error) });
process.exit(1);
});

713
frontend/deterministic-tests/src/deterministic-agent.ts
View file

@ -1,483 +1,340 @@
import type {
HistoryEntry,
StoredDatabase,
SyncSettings,
RelativePath,
TextWithCursors
} from "sync-client";
import {
SyncClient,
SyncResetError,
debugging,
LogLevel,
utils
type HistoryEntry,
type SyncSettings
} from "sync-client";
import { MemoryDisk, MemoryPersistence } from "../../test-support/storage";
import { NetworkFaults, type RequestKind } from "../../test-support/network";
import { assert } from "./utils/assert";
import { sleep } from "./utils/sleep";
import { withTimeout } from "./utils/with-timeout";
import {
IS_SYNC_ENABLED_BY_DEFAULT,
WAIT_TIMEOUT_MS,
WEBSOCKET_CONNECT_TIMEOUT_MS,
WEBSOCKET_POLL_INTERVAL_MS
} from "./consts";
import { WAIT_TIMEOUT_MS, WEBSOCKET_CONNECT_TIMEOUT_MS } from "./consts";
import { ManagedWebSocketFactory } from "./managed-websocket";
export class DeterministicAgent extends debugging.InMemoryFileSystem {
public readonly clientId: number;
private readonly logger: (msg: string) => void;
private client!: SyncClient;
private data: Partial<{
settings: Partial<SyncSettings>;
database: Partial<StoredDatabase>;
}> = {};
private isSyncEnabled = IS_SYNC_ENABLED_BY_DEFAULT;
private readonly syncErrors: Error[] = [];
private readonly pendingSyncOperations = new Set<Promise<void>>();
/** User actions live here; the engine only receives disk.session(). */
export class DeterministicAgent {
public readonly disk = new MemoryDisk();
public readonly persistence: MemoryPersistence;
public readonly network = new NetworkFaults();
private client?: SyncClient;
private readonly errors: Error[] = [];
private readonly pending = new Set<Promise<void>>();
private readonly wsFactory = new ManagedWebSocketFactory();
private nextWriteRename:
| {
oldPath: RelativePath;
newPath: RelativePath;
}
| undefined;
private nextCreateResponseDrop:
| {
dropped: Promise<void>;
resolveDropped: () => void;
}
| undefined;
private nextWriteRename?: { oldPath: string; newPath: string };
private delayedNotifications = false;
private disposed = false;
private readonly notifications: (() => Promise<void>)[] = [];
public constructor(
clientId: number,
initialSettings: Partial<SyncSettings>,
logger: (msg: string) => void
public readonly clientId: number,
settings: Partial<SyncSettings>,
private readonly logger: (msg: string) => void
) {
super();
this.clientId = clientId;
this.logger = logger;
this.data.settings = { ...initialSettings };
this.persistence = new MemoryPersistence({
settings: {
networkRetryIntervalMs: 25,
webSocketRetryIntervalMs: 50,
requestTimeoutMs: 5_000,
...settings,
// Scripted and seeded scenarios must exercise notification delivery.
syncIntervalMs: 0
}
});
this.disk.boundary = async (label) => {
const rename = this.nextWriteRename;
// v4 installs through an exclusive rename, never write-in-place.
if (
rename &&
label.startsWith("durable:rename:") &&
label.endsWith(`->${rename.oldPath}`)
) {
this.nextWriteRename = undefined;
await this.rename(rename.oldPath, rename.newPath);
}
};
}
public async init(
fetchImplementation: typeof globalThis.fetch
fetchImplementation: typeof globalThis.fetch = fetch
): Promise<void> {
this.client = await SyncClient.create({
fs: this,
persistence: {
load: async () => this.data,
save: async (data) => void (this.data = data)
},
fetch: this.wrapFetch(fetchImplementation),
fs: this.disk.session(),
persistence: this.persistence,
fetch: this.network.wrap(fetchImplementation),
webSocket: this.wsFactory.constructorFn
});
this.client.logger.onLogEmitted.add((line) => {
const prefix = `[Client ${this.clientId}]`;
switch (line.level) {
case LogLevel.ERROR:
this.logger(`${prefix} ERROR: ${line.message}`);
break;
case LogLevel.WARNING:
this.logger(`${prefix} WARN: ${line.message}`);
break;
case LogLevel.INFO:
this.logger(`${prefix} INFO: ${line.message}`);
break;
case LogLevel.DEBUG:
this.logger(`${prefix} DEBUG: ${line.message}`);
break;
}
});
await this.client.start();
const connectionCheck = await this.client.checkConnection();
assert(
connectionCheck.isSuccessful,
this.client.getSettings().syncIntervalMs === 0,
"Correctness scenarios must run with periodic polling disabled"
);
this.client.logger.onLogEmitted.add((line) => {
this.logger(
`[Client ${this.clientId}] ${line.level}: ${line.message}`
);
if (this.disposed && line.level === LogLevel.ERROR)
throw new Error(
`Client ${this.clientId} error after cleanup: ${line.message}`
);
if (
line.level === LogLevel.ERROR &&
!this.isRetryableFailure(line.message)
)
this.errors.push(new Error(line.message));
});
await this.client.start();
assert(
(await this.client.checkConnection()).isSuccessful,
`Client ${this.clientId} connection check failed`
);
}
if (this.isSyncEnabled) {
await this.waitForWebSocket();
public database() {
return this.persistence.snapshot().database;
}
public files(): Map<string, Uint8Array> {
return this.disk.userFiles();
}
public async listFilesRecursively(): Promise<string[]> {
return [...this.files().keys()].sort();
}
public async getFileContent(path: string): Promise<string> {
const bytes = this.files().get(path);
assert(bytes !== undefined, `Missing file: ${path}`);
return new TextDecoder("utf-8", { ignoreBOM: true }).decode(bytes);
}
public async read(path: string): Promise<Uint8Array> {
const snapshot = await this.disk.readSnapshot(path);
assert(snapshot !== undefined, `Missing file: ${path}`);
return snapshot.content;
}
public async write(path: string, content: Uint8Array): Promise<void> {
const created = !(await this.disk.exists(path));
await this.disk.userWrite(path, content);
this.notify(() =>
created
? this.client!.syncLocallyCreatedFile(path)
: this.client!.syncLocallyUpdatedFile({ relativePath: path })
);
}
public async rename(oldPath: string, relativePath: string): Promise<void> {
await this.disk.userRename(oldPath, relativePath);
this.notify(() =>
this.client!.syncLocallyUpdatedFile({ oldPath, relativePath })
);
}
public async delete(path: string): Promise<void> {
await this.disk.userDelete(path);
this.notify(() => this.client!.syncLocallyDeletedFile(path));
}
public delayNotifications(): void {
this.delayedNotifications = true;
}
public flushNotifications(): void {
this.delayedNotifications = false;
for (const notify of this.notifications.splice(0)) this.enqueue(notify);
}
private notify(operation: () => Promise<void>): void {
// Offline means transport disabled, not that editor identity events vanish.
if (this.delayedNotifications) this.notifications.push(operation);
else this.enqueue(operation);
}
private enqueue(operation: () => Promise<void>): void {
const promise = operation().catch((error: unknown) => {
if (!this.isRetryableFailure(String(error)))
this.errors.push(
error instanceof Error ? error : new Error(String(error))
);
});
this.pending.add(promise);
void promise.then(() => this.pending.delete(promise));
}
private async drain(): Promise<void> {
while (this.pending.size) await Promise.all([...this.pending]);
}
private isRetryableFailure(message: string): boolean {
// A coherent scan rejecting an editor race is a required adapter
// behavior. It must subsequently converge; never blanket-ignore errors.
return (
this.network.isExpectedFailure(message) ||
/^(Sync paused for retry: )?Error: File changed during scan: .+$/.test(
message
)
);
}
private assertHealthy(): void {
if (this.errors.length) {
const errors = this.errors.splice(0);
throw new AggregateError(
errors,
`Client ${this.clientId}: ${errors.map(String).join("; ")}`
);
}
}
public pauseWebSocket(): void {
this.log("Pausing WebSocket message delivery");
this.wsFactory.pause();
}
public resumeWebSocket(): void {
this.log("Resuming WebSocket message delivery");
this.wsFactory.resume();
}
public dropNextCreateResponse(): void {
public async waitForSync(): Promise<void> {
assert(
this.nextCreateResponseDrop === undefined,
`Client ${this.clientId} already has a create response drop armed`
);
let resolveDropped!: () => void;
const dropped = new Promise<void>((resolve) => {
resolveDropped = resolve;
});
this.nextCreateResponseDrop = {
dropped,
resolveDropped
};
this.log("Armed next create response drop");
}
public async waitForDroppedCreateResponse(): Promise<void> {
assert(
this.nextCreateResponseDrop !== undefined,
`Client ${this.clientId} has no create response drop armed`
this.client!.getSettings().syncIntervalMs === 0,
"Periodic polling was enabled during a correctness scenario"
);
await withTimeout(
this.nextCreateResponseDrop.dropped,
(async () => {
await this.drain();
const deadline = Date.now() + WAIT_TIMEOUT_MS;
for (;;) {
try {
await this.client!.waitUntilFinished();
break;
} catch (error) {
if (
!this.isRetryableFailure(String(error)) ||
Date.now() >= deadline
)
throw error;
await sleep(30);
}
}
this.assertHealthy();
})(),
WAIT_TIMEOUT_MS,
`Client ${this.clientId} timed out waiting for create response drop`
`Client ${this.clientId} sync timeout`
);
this.log("Create response was dropped after server commit");
}
public async disableSync(): Promise<void> {
await this.drain();
await this.client!.setSetting("isSyncEnabled", false);
this.assertHealthy();
}
public async enableSync(wait = true): Promise<void> {
// setSetting waits for a sync attempt in v4. Track it, but don't block
// the script from resuming a paused server or releasing a checkpoint.
this.enqueue(() => this.client!.setSetting("isSyncEnabled", true));
const deadline = Date.now() + WEBSOCKET_CONNECT_TIMEOUT_MS;
while (
!this.persistence.snapshot().settings?.isSyncEnabled &&
Date.now() < deadline
)
await sleep(1);
assert(
!!this.persistence.snapshot().settings?.isSyncEnabled,
"Enable did not persist"
);
if (wait) await this.waitForSync();
}
public async reset(): Promise<void> {
await this.drain();
await this.client!.reset();
}
public pauseWebSocket(): void {
this.wsFactory.pause();
}
public pauseObservation(): void {
this.pauseWebSocket();
this.network.pauseObservation();
}
public resumeObservation(): void {
this.network.resumeObservation();
this.resumeWebSocket();
}
public async waitForObservation(): Promise<void> {
await withTimeout(
this.network.waitForObservation(),
WAIT_TIMEOUT_MS,
"HTTP observation checkpoint never reached"
);
}
public resumeWebSocket(): void {
this.wsFactory.resume();
}
public dropNextCreateResponse(): void {
this.network.arm("create");
}
public dropNextResponse(
kind: RequestKind,
point: "before" | "after" = "after"
): void {
this.network.arm(kind, point);
}
public async waitForDroppedCreateResponse(): Promise<void> {
await withTimeout(
this.network.wait(),
WAIT_TIMEOUT_MS,
"Armed response drop never fired"
);
}
public renameNextWrite(oldPath: string, newPath: string): void {
assert(
!this.nextWriteRename,
"Previous install/rename hook never fired"
);
this.nextWriteRename = { oldPath, newPath };
}
public async waitForHistoryEntry(
matches: (entry: HistoryEntry) => boolean,
onMatch?: (entry: HistoryEntry) => void
): Promise<void> {
const existing = this.client.getHistoryEntries().find(matches);
if (existing !== undefined) {
onMatch?.(existing);
return;
}
await withTimeout(
new Promise<void>((resolve) => {
const unsubscribe = this.client.onSyncHistoryUpdated.add(() => {
const entry = this.client
.getHistoryEntries()
.find(matches);
if (entry === undefined) {
return;
}
unsubscribe();
onMatch?.(entry);
resolve();
});
}),
WAIT_TIMEOUT_MS,
`Client ${this.clientId} timed out waiting for history entry`
);
}
public async waitForSync(): Promise<void> {
this.log("Waiting for sync to complete...");
// Drain agent-level sync operations first. These are the fire-and-forget
// promises from enqueueSync() that call into the SyncClient's methods.
// Without this, waitUntilFinished() might return before the SyncClient
// has even been told about the operation.
await this.drainPendingSyncOperations();
await withTimeout(
this.client.waitUntilFinished(),
WAIT_TIMEOUT_MS,
`Client ${this.clientId} waitForSync timed out after ${WAIT_TIMEOUT_MS}ms`
);
if (this.syncErrors.length > 0) {
const errors = this.syncErrors.splice(0);
throw new Error(
`Client ${this.clientId} had ${errors.length} sync error(s):\n${errors.map((e) => e.message).join("\n")}`
);
}
this.log("Sync complete");
}
public async reset(): Promise<void> {
this.log("Resetting client (clears tracked state, keeps disk files)");
await this.drainPendingSyncOperations();
await this.client.reset();
if (this.isSyncEnabled) {
await this.waitForWebSocket();
}
}
public async disableSync(): Promise<void> {
this.log("Disabling sync");
// Drain pending enqueued operations before disabling so the SyncClient
// knows about all operations that were enqueued while sync was enabled.
await this.drainPendingSyncOperations();
await this.client.setSetting("isSyncEnabled", false);
this.isSyncEnabled = false;
// Wait for in-flight operations to drain. Disabling sync triggers
// a reset, which aborts in-flight fetches with SyncResetError.
let unsubscribe = () => {};
try {
await withTimeout(
this.client.waitUntilFinished(),
new Promise<void>((resolve) => {
const check = () => {
const entry =
this.client!.getHistoryEntries().find(matches);
if (entry) {
onMatch?.(entry);
resolve();
}
};
unsubscribe = this.client!.onSyncHistoryUpdated.add(check);
check();
}),
WAIT_TIMEOUT_MS,
`Client ${this.clientId} disableSync drain timed out`
"History checkpoint not reached"
);
} catch (error) {
if (error instanceof Error && error.name === "SyncResetError") {
this.log("Disable sync drain interrupted by reset (expected)");
} else {
throw error;
}
} finally {
unsubscribe();
}
}
public async enableSync(): Promise<void> {
this.log("Enabling sync");
await this.client.setSetting("isSyncEnabled", true);
this.isSyncEnabled = true;
await this.waitForWebSocket();
}
public async getFileContent(path: string): Promise<string> {
const bytes = await this.read(path);
return new TextDecoder().decode(bytes);
}
public renameNextWrite(oldPath: RelativePath, newPath: RelativePath): void {
assert(
this.nextWriteRename === undefined,
`Client ${this.clientId} already has a next-write rename armed`
);
this.nextWriteRename = { oldPath, newPath };
this.log(`Armed next write rename: ${oldPath} -> ${newPath}`);
}
public async cleanup(): Promise<void> {
this.log("Cleaning up...");
// Guard against uninitialized client (init() failed partway).
// The class field uses `!:` so TS thinks this is always defined,
// but at runtime it can be undefined when init() throws partway.
const maybeClient = this.client as SyncClient | undefined;
if (maybeClient === undefined) {
this.log("Client not initialized, nothing to clean up");
return;
if (!this.client) return;
const errors: unknown[] = [];
this.resumeObservation();
this.flushNotifications();
try {
await this.waitForSync();
} catch (error) {
errors.push(error);
}
try {
this.network.assertConsumed();
assert(!this.nextWriteRename, "Install/rename hook never fired");
} catch (error) {
errors.push(error);
}
try {
await this.drainPendingSyncOperations();
await withTimeout(
this.client.waitUntilFinished(),
this.client.destroy(),
WAIT_TIMEOUT_MS,
`Client ${this.clientId} cleanup waitUntilFinished timed out`
"Destroy timed out"
);
await withTimeout(
this.wsFactory.finish(),
WAIT_TIMEOUT_MS,
"WebSocket callbacks did not finish during cleanup"
);
} catch (error) {
if (error instanceof Error && error.name === "SyncResetError") {
this.log(`Cleanup interrupted by reset (expected): ${error}`);
} else {
this.log(`Cleanup waitUntilFinished failed: ${error}`);
}
errors.push(error);
}
// Surface any background sync errors that arrived after the last
// waitForSync (e.g. between the final assert-consistent and here).
// Without this, regressions that fault the engine during the very
// last step of a test would be silently swallowed.
const pendingErrors = this.syncErrors.splice(0);
await this.client.destroy();
this.log("Cleanup complete");
if (pendingErrors.length > 0) {
throw new Error(
`Client ${this.clientId} had ${pendingErrors.length} background sync error(s) during cleanup:\n${pendingErrors.map((e) => e.message).join("\n")}`
);
}
}
public override async read(path: RelativePath): Promise<Uint8Array> {
await Promise.resolve();
return super.read(path);
}
public override async write(
path: RelativePath,
content: Uint8Array
): Promise<void> {
await Promise.resolve();
const isNew = !this.files.has(path);
await super.write(path, content);
if (this.isSyncEnabled && isNew) {
this.enqueueSync(async () => {
this.client.syncLocallyCreatedFile(path);
});
}
const { nextWriteRename } = this;
if (
nextWriteRename !== undefined &&
nextWriteRename.oldPath === path
) {
this.nextWriteRename = undefined;
await super.rename(
nextWriteRename.oldPath,
nextWriteRename.newPath
);
if (this.isSyncEnabled) {
this.enqueueSync(async () => {
this.client.syncLocallyUpdatedFile({
oldPath: nextWriteRename.oldPath,
relativePath: nextWriteRename.newPath
});
});
}
// The rename consumed `path`. Skip the post-update enqueue below
// — it would send a syncLocallyUpdatedFile for a path that no
// longer exists.
return;
}
if (!this.isSyncEnabled) {
return;
}
if (!isNew) {
this.enqueueSync(async () => {
this.client.syncLocallyUpdatedFile({ relativePath: path });
});
}
}
public override async atomicUpdateText(
path: RelativePath,
updater: (current: TextWithCursors) => TextWithCursors
): Promise<string> {
const result = await super.atomicUpdateText(path, updater);
if (this.isSyncEnabled) {
this.enqueueSync(async () => {
this.client.syncLocallyUpdatedFile({ relativePath: path });
});
}
return result;
}
public override async delete(path: RelativePath): Promise<void> {
await super.delete(path);
if (this.isSyncEnabled) {
this.enqueueSync(async () => {
this.client.syncLocallyDeletedFile(path);
});
}
}
public override async rename(
oldPath: RelativePath,
newPath: RelativePath
): Promise<void> {
await super.rename(oldPath, newPath);
if (this.isSyncEnabled) {
this.enqueueSync(async () => {
this.client.syncLocallyUpdatedFile({
oldPath,
relativePath: newPath
});
});
}
}
private async waitForWebSocket(): Promise<void> {
const deadline = Date.now() + WEBSOCKET_CONNECT_TIMEOUT_MS;
while (!this.client.isWebSocketConnected && Date.now() < deadline) {
await sleep(WEBSOCKET_POLL_INTERVAL_MS);
}
assert(
this.client.isWebSocketConnected,
`Client ${this.clientId} WebSocket failed to connect within ${WEBSOCKET_CONNECT_TIMEOUT_MS}ms`
);
}
/**
* Wait until all agent-level enqueued sync operations have completed.
* Uses a loop because completing one operation can trigger new enqueues.
*/
private async drainPendingSyncOperations(): Promise<void> {
while (this.pendingSyncOperations.size > 0) {
await utils.awaitAll([...this.pendingSyncOperations]);
}
}
private enqueueSync(operation: () => Promise<void>): void {
const promise = this.executeSyncOperation(operation).catch(
(error: unknown) => {
const err =
error instanceof Error ? error : new Error(String(error));
this.log(`Background sync failed: ${err.message}`);
this.syncErrors.push(err);
}
);
this.pendingSyncOperations.add(promise);
void promise.finally(() => {
this.pendingSyncOperations.delete(promise);
});
}
private async executeSyncOperation(
operation: () => Promise<void>
): Promise<void> {
this.client = undefined;
this.disposed = true;
try {
await operation();
this.assertHealthy();
} catch (error) {
if (error instanceof Error && error.name === "SyncResetError") {
this.log(`Sync operation interrupted by reset: ${error}`);
return;
}
if (
error instanceof Error &&
error.message.includes("has been destroyed")
) {
this.log(`Sync operation interrupted by destroy: ${error}`);
return;
}
throw error;
errors.push(error);
}
}
private log(message: string): void {
this.logger(`[Client ${this.clientId}] ${message}`);
}
private wrapFetch(
fetchImplementation: typeof globalThis.fetch
): typeof globalThis.fetch {
return async (input, init) => {
const response = await fetchImplementation(input, init);
const drop = this.nextCreateResponseDrop;
if (
drop !== undefined &&
DeterministicAgent.isCreateDocumentRequest(input, init)
) {
this.nextCreateResponseDrop = undefined;
try {
await response.body?.cancel();
} catch {
// Best-effort — body may already be consumed/closed.
}
drop.resolveDropped();
throw new SyncResetError();
}
return response;
};
}
private static isCreateDocumentRequest(
input: RequestInfo | URL,
init: RequestInit | undefined
): boolean {
const method =
init?.method ??
(typeof Request !== "undefined" && input instanceof Request
? input.method
: "GET");
if (method.toUpperCase() !== "POST") {
return false;
}
const url =
input instanceof URL
? input
: new URL(typeof input === "string" ? input : input.url);
return /\/documents\/?$/.test(url.pathname);
if (errors.length)
throw new AggregateError(
errors,
`Client ${this.clientId} cleanup failed: ${errors.map(String).join("; ")}`
);
}
}

21
frontend/deterministic-tests/src/managed-websocket.ts
View file

@ -209,8 +209,13 @@ export class ManagedWebSocketFactory {
// WS before the agent reconnects would silently see the new socket
// start un-paused and miss the messages it meant to buffer.
private currentlyPaused = false;
private disposed = false;
public get constructorFn(): typeof globalThis.WebSocket {
const ensureActive = () => {
if (this.disposed)
throw new Error("WebSocket created after harness cleanup");
};
const trackInstance = (instance: ManagedWebSocket): void => {
this.instances.push(instance);
if (this.currentlyPaused) {
@ -222,6 +227,7 @@ export class ManagedWebSocketFactory {
url: string | URL,
protocols?: string | string[]
) {
ensureActive();
super(url, protocols);
trackInstance(this);
}
@ -229,6 +235,21 @@ export class ManagedWebSocketFactory {
return TrackedManagedWebSocket;
}
public async finish(): Promise<void> {
this.disposed = true;
await Promise.all(
this.instances.map((socket) =>
socket.readyState === WebSocket.CLOSED
? Promise.resolve()
: new Promise<void>((resolve) =>
socket.addEventListener("close", () => resolve(), {
once: true
})
)
)
);
}
public pause(): void {
this.currentlyPaused = true;
for (const ws of this.instances) {

4
frontend/deterministic-tests/src/parse-args.ts
View file

@ -7,8 +7,8 @@ export interface CliArgs {
}
function parsePositiveInt(value: string): number {
const n = parseInt(value, 10);
if (isNaN(n) || n <= 0) {
const n = Number(value);
if (!/^\d+$/.test(value) || !Number.isSafeInteger(n) || n <= 0) {
throw new InvalidArgumentError("must be a positive integer");
}
return n;

120
frontend/deterministic-tests/src/server-control.ts
View file

@ -20,10 +20,13 @@ export class ServerControl {
private _port: number | undefined;
private tempDir: string | undefined;
private _isPaused = false;
private preserveOnStop = false;
private expectedExit = false;
private unexpectedExit?: Error;
public constructor(serverPath: string, configPath: string, logger: Logger) {
this.serverPath = serverPath;
this.baseConfigPath = configPath;
this.serverPath = path.resolve(serverPath);
this.baseConfigPath = path.resolve(configPath);
this.logger = logger;
}
@ -38,6 +41,15 @@ export class ServerControl {
return `http://localhost:${this.port}`;
}
public get databaseDirectory(): string {
if (!this.tempDir) throw new Error("Server has no database directory");
return path.join(this.tempDir, "databases");
}
public get isPaused(): boolean {
return this._isPaused;
}
public async start(): Promise<void> {
if (this.process !== null) {
throw new Error("Server is already running");
@ -66,14 +78,25 @@ export class ServerControl {
}
private async startOnce(): Promise<void> {
const reservation = await findFreePort();
this._port = reservation.port;
const reservation =
this._port === undefined ? await findFreePort() : undefined;
this._port ??= reservation!.port;
const tmpBase = os.tmpdir();
this.tempDir = fs.mkdtempSync(path.join(tmpBase, "vault-link-test-"));
const tempConfigPath = path.join(this.tempDir, "config.yml");
const dbDir = path.join(this.tempDir, "databases");
this.writeConfigFile(tempConfigPath, dbDir);
let tempConfigPath: string;
try {
this.tempDir ??= fs.mkdtempSync(
path.join(tmpBase, "vault-link-test-")
);
tempConfigPath = path.join(this.tempDir, "config.yml");
this.writeConfigFile(
tempConfigPath,
path.join(this.tempDir, "databases")
);
} catch (error) {
reservation?.release();
this.cleanupTempDir();
throw error;
}
this.logger.info(
`Starting server: ${this.serverPath} (port ${this._port})`
@ -81,12 +104,15 @@ export class ServerControl {
// Release the port reservation right before spawning to minimize
// the TOCTOU window between port discovery and server binding.
reservation.release();
reservation?.release();
this.process = spawn(this.serverPath, [tempConfigPath], {
cwd: this.tempDir,
stdio: ["ignore", "pipe", "pipe"],
detached: false
});
this.expectedExit = false;
this.unexpectedExit = undefined;
this.process.stdout?.on("data", (data: Buffer) => {
this.logger.info(`[SERVER] ${data.toString().trim()}`);
@ -102,6 +128,10 @@ export class ServerControl {
const currentProcess = this.process;
currentProcess.on("exit", (code, signal) => {
if (!this.expectedExit)
this.unexpectedExit = new Error(
`Owned server exited unexpectedly (code ${code}, signal ${signal})`
);
this.logger.info(
`Server exited with code ${code}, signal ${signal}`
);
@ -139,7 +169,9 @@ export class ServerControl {
);
}
try {
const response = await fetch(pingUrl);
const response = await fetch(pingUrl, {
signal: AbortSignal.timeout(1_000)
});
if (response.ok) {
this.logger.info("[SERVER] Ready");
return;
@ -194,11 +226,13 @@ export class ServerControl {
public async stop(): Promise<void> {
const proc = this.process;
if (proc?.pid === undefined) {
this.cleanupTempDir();
if (!this.preserveOnStop) this.cleanupTempDir();
if (this.unexpectedExit) throw this.unexpectedExit;
return;
}
this.expectedExit = true;
// Resume if paused — a SIGSTOP'd process ignores SIGKILL
// SIGKILL terminates even a stopped process; SIGCONT is harmless here.
if (this._isPaused) {
try {
process.kill(proc.pid, "SIGCONT");
@ -212,7 +246,7 @@ export class ServerControl {
// Set up a promise that resolves when the process actually exits.
const exitPromise = new Promise<void>((resolve) => {
if (proc.exitCode !== null) {
if (proc.exitCode !== null || proc.signalCode !== null) {
resolve();
return;
}
@ -229,11 +263,49 @@ export class ServerControl {
// Wait for the process to actually exit before cleaning up,
// with a 5s safety timeout to avoid hanging forever.
await Promise.race([exitPromise, sleep(STOP_TIMEOUT_MS)]);
let timer: ReturnType<typeof setTimeout> | undefined;
try {
await Promise.race([
exitPromise,
new Promise<never>((_, reject) => {
timer = setTimeout(
() =>
reject(
new Error(
"Owned server did not exit after SIGKILL"
)
),
STOP_TIMEOUT_MS
);
})
]);
} finally {
clearTimeout(timer);
}
this.process = null;
this._isPaused = false;
this.cleanupTempDir();
if (!this.preserveOnStop) this.cleanupTempDir();
if (this.unexpectedExit) throw this.unexpectedExit;
}
/** A real process death: retain the exact database and port for restart. */
public async crash(): Promise<void> {
if (!this.isRunning()) throw new Error("Cannot crash a stopped server");
this.preserveOnStop = true;
try {
await this.stop();
} finally {
this.preserveOnStop = false;
}
}
public async restart(): Promise<void> {
if (!this.tempDir || this.isRunning())
throw new Error(
"Restart requires a crashed server with retained storage"
);
await this.startOnce();
}
public isRunning(): boolean {
@ -273,23 +345,27 @@ export class ServerControl {
// one `databases_directory_path:` (under `server:` and `database:`
// respectively)
const baseConfig = fs.readFileSync(this.baseConfigPath, "utf-8");
for (const key of ["port", "databases_directory_path"]) {
if (
(baseConfig.match(new RegExp(`^ ${key}:`, "gm")) ?? [])
.length !== 1
)
throw new Error(`Expected exactly one config key: ${key}`);
}
const config = baseConfig
.replace(/^\s*port:\s*\d+/m, ` port: ${this._port}`)
.replace(
/^\s*databases_directory_path:\s*.+/m,
` databases_directory_path: ${dbDir}`
` databases_directory_path: ${JSON.stringify(dbDir)}`
);
fs.writeFileSync(destPath, config);
}
private cleanupTempDir(): void {
if (this.tempDir !== undefined) {
try {
fs.rmSync(this.tempDir, { recursive: true, force: true });
} catch {
// Best-effort cleanup
}
fs.rmSync(this.tempDir, { recursive: true, force: true });
this.tempDir = undefined;
this._port = undefined;
}
}
}

5
frontend/deterministic-tests/src/server-manager.ts
View file

@ -59,9 +59,8 @@ export class ServerManager {
// Last-resort synchronous cleanup. Runs even when the process is
// exiting via process.exit() from unhandledRejection /
// uncaughtException — paths where async stopAll() cannot complete.
// SIGSTOP'd servers MUST receive SIGCONT before SIGKILL or the
// kernel keeps them as zombies holding the test's tmpdir, and the
// next CI run can't reuse the port.
// Kill only the server processes owned by this runner. SIGKILL also
// terminates a SIGSTOP'd process; forceKillSync's SIGCONT is harmless.
process.on("exit", () => {
for (const server of this.activeServers) {
server.forceKillSync();

46
frontend/deterministic-tests/src/test-definition.ts
View file

@ -3,6 +3,18 @@ import type { AssertableState } from "./utils/assertable-state";
export interface ClientState {
files: Map<string, string>;
clientFiles: Map<string, string>[];
manifests: Record<string, string>[];
canonical: Record<string, string>;
bytes: Map<string, Uint8Array>;
}
/** A scenario's independent, complete expected state. Keys survive moves;
* a new key must receive a previously unseen UUID, including after deletion. */
export interface ExpectedDocument {
key: string;
path: string;
conflict?: boolean;
content: string | number[];
}
export type TestStep =
@ -31,10 +43,41 @@ export type TestStep =
| { type: "assert-consistent"; verify?: (state: AssertableState) => void }
| { type: "pause-websocket"; client: number }
| { type: "resume-websocket"; client: number }
| {
type:
| "pause-observation"
| "resume-observation"
| "wait-for-observation";
client: number;
}
| { type: "drop-next-create-response"; client: number }
| { type: "wait-for-dropped-create-response"; client: number }
| { type: "sleep"; ms: number }
| { type: "reset"; client: number };
| { type: "reset"; client: number }
| { type: "create-bytes"; client: number; path: string; bytes: number[] }
| {
type: "drop-response";
client: number;
kind: "create" | "content" | "manifest";
point?: "before" | "after";
}
| { type: "wait-for-response-drop"; client: number }
| { type: "delay-notifications" | "flush-notifications"; client: number }
| {
type: "remember-identity" | "assert-identity";
path: string;
key: string;
}
| {
type: "assert-files";
expected: Record<string, string>;
absent?: string[];
count?: number;
}
| { type: "assert-markers"; markers: string[]; removed?: string[] }
| { type: "assert-documents"; expected: ExpectedDocument[] }
| { type: "crash-server" }
| { type: "restart-server" };
export interface TestDefinition {
description?: string;
@ -46,4 +89,5 @@ export interface TestResult {
success: boolean;
error?: string;
duration?: number;
diagnostics?: unknown;
}

2
frontend/deterministic-tests/src/test-registry.ts
View file

@ -1,4 +1,5 @@
import type { TestDefinition } from "./test-definition";
import { V4_TESTS } from "./v4-regressions";
import { renameCreateConflictTest } from "./tests/rename-create-conflict.test";
import { renameChainTest } from "./tests/rename-chain.test";
import { renameUpdateConflictTest } from "./tests/rename-update-conflict.test";
@ -106,6 +107,7 @@ import { remoteQuickWriteRenameBeforeRecordTest } from "./tests/remote-quick-wri
import { selfMergePendingRenameAliasesSecondCreateTest } from "./tests/self-merge-pending-rename-aliases-second-create.test";
export const TESTS: Partial<Record<string, TestDefinition>> = {
...V4_TESTS,
"rename-create-conflict": renameCreateConflictTest,
"rename-chain": renameChainTest,
"rename-update-conflict": renameUpdateConflictTest,

144
frontend/deterministic-tests/src/test-runner.ts
View file

@ -13,6 +13,8 @@ import {
IS_SYNC_ENABLED_BY_DEFAULT
} from "./consts";
import { randomUUID } from "node:crypto";
import { assertCanonical } from "../../test-support/canonical";
import { ContentLedger } from "../../test-support/oracles";
export class TestRunner {
private agents: DeterministicAgent[] = [];
@ -20,6 +22,8 @@ export class TestRunner {
private readonly token: string;
private readonly remoteUri: string;
private readonly logger: Logger;
private vaultName = "";
private readonly identities = new Map<string, string>();
public constructor(
serverControl: ServerControl,
@ -44,6 +48,8 @@ export class TestRunner {
}
this.logger.info(`Clients: ${test.clients}`);
this.logger.info(`Steps: ${test.steps.length}`);
this.identities.clear();
let diagnostics: TestResult["diagnostics"];
try {
assert(
@ -61,6 +67,11 @@ export class TestRunner {
await this.executeStep(step);
}
assert(
this.serverControl.isRunning(),
"Server died before test completion"
);
diagnostics = this.captureDiagnostics();
await this.cleanup();
const duration = Date.now() - startTime;
@ -71,25 +82,46 @@ export class TestRunner {
duration
};
} catch (error) {
diagnostics ??= this.captureDiagnostics();
const duration = Date.now() - startTime;
const errorMessage =
error instanceof Error ? error.message : String(error);
this.logger.info(`\n✗ Test failed: ${name}`);
this.logger.info(`Error: ${errorMessage}`);
await this.cleanup();
try {
await this.cleanup();
} catch (cleanupError) {
return {
success: false,
error: `${errorMessage}\nCleanup: ${String(cleanupError)}`,
duration,
diagnostics
};
}
return {
success: false,
error: errorMessage,
diagnostics,
duration
};
}
}
private captureDiagnostics(): unknown {
return this.agents.map((agent) => ({
client: agent.clientId,
database: agent.database(),
disk: agent.disk.image(),
requests: structuredClone(agent.network.requests)
}));
}
private async initializeAgents(count: number): Promise<void> {
assert(count > 0, `Client count must be positive, got ${count}`);
const vaultName = `test-${randomUUID()}`;
this.vaultName = vaultName;
this.logger.info(
`Initializing ${count} agents with vault: ${vaultName}`
);
@ -129,6 +161,87 @@ export class TestRunner {
private async executeStep(step: TestStep): Promise<void> {
switch (step.type) {
case "pause-observation":
this.getAgent(step.client).pauseObservation();
break;
case "resume-observation":
this.getAgent(step.client).resumeObservation();
break;
case "wait-for-observation":
await this.getAgent(step.client).waitForObservation();
break;
case "create-bytes":
await this.getAgent(step.client).write(
step.path,
new Uint8Array(step.bytes)
);
break;
case "drop-response":
this.getAgent(step.client).dropNextResponse(
step.kind,
step.point
);
break;
case "wait-for-response-drop":
await this.getAgent(step.client).waitForDroppedCreateResponse();
break;
case "delay-notifications":
this.getAgent(step.client).delayNotifications();
break;
case "flush-notifications":
this.getAgent(step.client).flushNotifications();
break;
case "remember-identity":
case "assert-identity":
await this.assertConsistent((state) => {
if (step.type === "remember-identity")
this.identities.set(
step.key,
state.documentId(step.path)
);
else {
const id = this.identities.get(step.key);
assert(
id !== undefined,
`Identity ${step.key} was never recorded`
);
state.assertIdentity(step.path, id);
}
});
break;
case "assert-files":
await this.assertConsistent((state) => {
for (const [path, value] of Object.entries(step.expected))
state.assertContent(path, value);
for (const path of step.absent ?? [])
state.assertFileNotExists(path);
if (step.count !== undefined)
state.assertFileCount(step.count);
});
break;
case "assert-documents":
await this.assertConsistent((state) =>
state.assertDocuments(step.expected, this.identities)
);
break;
case "assert-markers": {
const ledger = new ContentLedger();
for (const marker of step.markers) ledger.add(marker);
for (const marker of step.removed ?? [])
ledger.removedBy(
marker,
"trace records explicit deletion/overwrite"
);
for (const agent of this.agents)
ledger.assertPreserved(agent.files());
break;
}
case "crash-server":
await this.serverControl.crash();
break;
case "restart-server":
await this.serverControl.restart();
break;
case "create":
case "update":
await this.getAgent(step.client).write(
@ -170,7 +283,9 @@ export class TestRunner {
break;
case "enable-sync":
await this.getAgent(step.client).enableSync();
await this.getAgent(step.client).enableSync(
!this.serverControl.isPaused
);
break;
case "pause-server":
@ -190,7 +305,9 @@ export class TestRunner {
(entry) =>
entry.details.type === step.syncType &&
entry.details.relativePath === step.path,
() => { this.serverControl.pause(); }
() => {
this.serverControl.pause();
}
);
this.serverControl.resume();
await historySeen;
@ -305,6 +422,11 @@ export class TestRunner {
// Snapshot all agents' file states upfront to minimize the window
// where background sync could mutate state between reads.
const canonical = await assertCanonical(
this.agents,
`${this.remoteUri}/vaults/${encodeURIComponent(this.vaultName)}`,
this.token
);
const clientFiles: Map<string, string>[] = [];
for (const agent of this.agents) {
const sortedFiles = (await agent.listFilesRecursively()).sort();
@ -360,7 +482,12 @@ export class TestRunner {
verify(
new AssertableState({
files: clientFiles[0],
clientFiles
clientFiles,
manifests: this.agents.map(
(agent) => agent.database()!.local!
),
canonical: canonical.fileManifest.entries,
bytes: this.agents[0].files()
})
);
} catch (error) {
@ -384,10 +511,12 @@ export class TestRunner {
}
this.logger.info("\nCleaning up agents...");
for (const agent of this.agents) {
const errors: unknown[] = [];
for (const agent of this.agents.splice(0)) {
try {
await agent.cleanup();
} catch (error) {
errors.push(error);
this.logger.warn(
`Agent cleanup error: ${error instanceof Error ? error.message : String(error)}`
);
@ -395,5 +524,10 @@ export class TestRunner {
}
this.agents = [];
this.logger.info("Cleanup complete");
if (errors.length)
throw new AggregateError(
errors,
`Cleanup failed: ${errors.map(String).join("; ")}`
);
}
}

93
frontend/deterministic-tests/src/utils/assertable-state.ts
View file

@ -1,12 +1,103 @@
import type { ClientState } from "../test-definition";
import assert from "node:assert/strict";
import type { ClientState, ExpectedDocument } from "../test-definition";
export class AssertableState {
public readonly files: Map<string, string>;
public readonly clientFiles: Map<string, string>[];
public readonly manifests: Record<string, string>[];
public readonly canonical: Record<string, string>;
public readonly bytes: Map<string, Uint8Array>;
public constructor(state: ClientState) {
this.files = state.files;
this.clientFiles = state.clientFiles;
this.manifests = state.manifests;
this.canonical = state.canonical;
this.bytes = state.bytes;
}
public documentId(path: string): string {
const id = Object.keys(this.canonical).find(
(id) => this.canonical[id] === path
);
if (!id) throw new Error(`No document identity for ${path}`);
return id;
}
public assertDocuments(
expected: readonly ExpectedDocument[],
identities: Map<string, string>
): this {
this.assertFileCount(expected.length);
assert.equal(
new Set(expected.map((doc) => doc.key)).size,
expected.length,
"Duplicate expected identity key"
);
const next = new Map(identities);
const paths = new Set<string>();
for (const doc of expected) {
const path = doc.conflict ? this.conflictPath(doc.path) : doc.path;
assert(
!paths.has(path),
`Two expected documents resolved to ${path}`
);
paths.add(path);
this.assertBytes(
path,
typeof doc.content === "string"
? new TextEncoder().encode(doc.content)
: new Uint8Array(doc.content)
);
const id = this.documentId(path);
const recorded = next.get(doc.key);
if (recorded !== undefined) this.assertIdentity(path, recorded);
else {
assert(
![...next.values()].includes(id),
`New document ${doc.key} reused an existing or deleted UUID`
);
next.set(doc.key, id);
}
}
// A failed check must not establish a partial identity oracle.
for (const [key, id] of next) identities.set(key, id);
return this;
}
public conflictPath(original: string): string {
const dot = original.lastIndexOf(".");
const split =
dot > original.lastIndexOf("/") + 1 ? dot : original.length;
const matches = Object.entries(this.canonical).filter(
([id, path]) =>
path ===
`${original.slice(0, split)} (conflict ${id})${original.slice(split)}`
);
if (matches.length !== 1)
throw new Error(
`Expected one UUID-derived conflict path for ${original}, found ${matches.length}`
);
return matches[0][1];
}
public assertIdentity(path: string, id: string): this {
if (this.documentId(path) !== id)
throw new Error(
`Document identity changed at ${path}: expected ${id}, got ${this.documentId(path)}`
);
return this;
}
public assertBytes(path: string, expected: Uint8Array): this {
const actual = this.bytes.get(path);
if (
!actual ||
actual.length !== expected.length ||
actual.some((byte, i) => byte !== expected[i])
)
throw new Error(`Byte mismatch at ${path}`);
return this;
}
public assertFileCount(expected: number): this {

253
frontend/deterministic-tests/src/v4-regressions.ts Normal file
View file

@ -0,0 +1,253 @@
import type { TestDefinition, TestStep } from "./test-definition";
const online: TestStep[] = [
{ type: "enable-sync", client: 0 },
{ type: "enable-sync", client: 1 },
{ type: "barrier" }
];
const offline: TestStep[] = [
{ type: "disable-sync", client: 0 },
{ type: "disable-sync", client: 1 }
];
export const V4_TESTS: Record<string, TestDefinition> = {
"v4-text-byte-and-component-boundaries": {
clients: 2,
steps: [
...online,
{ type: "create", client: 0, path: "empty.md", content: "" },
{
type: "create",
client: 0,
path: "bom.md",
content: "\ufefffirst\r\nlast"
},
{
type: "create",
client: 0,
path: `${"a".repeat(252)}.md`,
content: "255 ASCII bytes"
},
{
type: "create",
client: 0,
path: `${"é".repeat(126)}.md`,
content: "255 UTF-8 bytes"
},
{ type: "barrier" },
{
type: "assert-files",
count: 4,
expected: {
"empty.md": "",
"bom.md": "\ufefffirst\r\nlast",
[`${"a".repeat(252)}.md`]: "255 ASCII bytes",
[`${"é".repeat(126)}.md`]: "255 UTF-8 bytes"
}
}
]
},
"v4-binary-bytes-and-independent-identities": {
clients: 2,
steps: [
...online,
...offline,
{
type: "create-bytes",
client: 0,
path: "nested/blob.bin",
bytes: [0, 255, 254, 128, 13, 10]
},
{
type: "create-bytes",
client: 1,
path: "nested/blob.bin",
bytes: [0, 254, 255, 129, 10, 13]
},
...online,
{
type: "assert-consistent",
verify: (state) => {
state
.assertFileCount(2)
.assertBytes(
"nested/blob.bin",
new Uint8Array([0, 255, 254, 128, 13, 10])
);
state.assertBytes(
state.conflictPath("nested/blob.bin"),
new Uint8Array([0, 254, 255, 129, 10, 13])
);
}
}
]
},
"v4-three-file-cycle-preserves-identities": {
clients: 2,
steps: [
...online,
{ type: "create", client: 0, path: "a.md", content: "A" },
{ type: "create", client: 0, path: "b.md", content: "B" },
{ type: "create", client: 0, path: "c.md", content: "C" },
{ type: "barrier" },
{ type: "remember-identity", key: "a", path: "a.md" },
{ type: "remember-identity", key: "b", path: "b.md" },
{ type: "remember-identity", key: "c", path: "c.md" },
{ type: "disable-sync", client: 0 },
{ type: "delay-notifications", client: 0 },
{ type: "rename", client: 0, oldPath: "a.md", newPath: "temp.md" },
{ type: "rename", client: 0, oldPath: "b.md", newPath: "a.md" },
{ type: "rename", client: 0, oldPath: "c.md", newPath: "b.md" },
{ type: "rename", client: 0, oldPath: "temp.md", newPath: "c.md" },
{
type: "update",
client: 0,
path: "c.md",
content: "A edited offline"
},
{ type: "flush-notifications", client: 0 },
{ type: "enable-sync", client: 0 },
{ type: "barrier" },
{ type: "assert-identity", key: "a", path: "c.md" },
{ type: "assert-identity", key: "b", path: "a.md" },
{ type: "assert-identity", key: "c", path: "b.md" },
{
type: "assert-files",
count: 3,
expected: {
"a.md": "B",
"b.md": "C",
"c.md": "A edited offline"
},
absent: ["temp.md"]
}
]
},
"v4-case-alias-and-file-directory-conflicts": {
clients: 2,
steps: [
...online,
...offline,
{
type: "create",
client: 0,
path: "folder/note.md",
content: "nested file"
},
{
type: "create",
client: 0,
path: "README.md",
content: "uppercase file"
},
{
type: "create",
client: 1,
path: "folder",
content: "blocking file"
},
{
type: "create",
client: 1,
path: "readme.md",
content: "lowercase file"
},
...online,
{
type: "assert-consistent",
verify: (state) => {
state
.assertFileCount(4)
.assertContent("folder/note.md", "nested file")
.assertContent("README.md", "uppercase file");
state.assertAnyFileContains(
"blocking file",
"lowercase file"
);
}
},
{
type: "assert-markers",
markers: [
"nested file",
"blocking file",
"uppercase file",
"lowercase file"
]
}
]
},
"v4-server-sigkill-retains-acknowledged-state": {
clients: 2,
steps: [
...online,
{
type: "create",
client: 0,
path: "a.md",
content: "durable bytes"
},
{ type: "barrier" },
{ type: "remember-identity", key: "a", path: "a.md" },
...offline,
{ type: "crash-server" },
{ type: "restart-server" },
...online,
{ type: "assert-identity", key: "a", path: "a.md" },
{
type: "assert-files",
count: 1,
expected: { "a.md": "durable bytes" }
}
]
}
};
for (const kind of ["create", "content", "manifest"] as const) {
for (const point of ["before", "after"] as const) {
V4_TESTS[`v4-${kind}-${point}-commit-network-loss`] = {
clients: 2,
steps: [
...online,
...(kind === "content"
? ([
{
type: "create",
client: 0,
path: "file.md",
content: "old bytes"
},
{ type: "barrier" },
{
type: "remember-identity",
key: "updated-file",
path: "file.md"
}
] as TestStep[])
: []),
{ type: "drop-response", client: 0, kind, point },
{
type: kind === "content" ? "update" : "create",
client: 0,
path: "file.md",
content: "must survive retry"
},
{ type: "wait-for-response-drop", client: 0 },
{ type: "barrier" },
...(kind === "content"
? [
{
type: "assert-identity",
key: "updated-file",
path: "file.md"
} as TestStep
]
: []),
{
type: "assert-files",
count: 1,
expected: { "file.md": "must survive retry" }
}
]
};
}
}

1
frontend/deterministic-tests/tsconfig.json
View file

@ -2,6 +2,7 @@
"compilerOptions": {
"baseUrl": ".",
"strict": true,
"skipLibCheck": true,
"target": "ES2022",
"module": "CommonJS",
"esModuleInterop": true,

5
frontend/deterministic-tests/webpack.config.js
View file

@ -2,8 +2,9 @@ const path = require("path");
const webpack = require("webpack");
module.exports = {
entry: "./src/cli.ts",
entry: { cli: "./src/cli.ts", "crash-worker": "./src/crash-worker.ts" },
target: "node",
externals: { "sync-client": "commonjs sync-client" },
mode: "production",
optimization: {
minimize: false
@ -21,7 +22,7 @@ module.exports = {
},
output: {
globalObject: "this",
filename: "cli.js",
filename: "[name].js",
path: path.resolve(__dirname, "dist")
},
plugins: [

45
frontend/local-client-cli/src/node-filesystem.test.ts
View file

@ -51,7 +51,7 @@ test("NodeFileSystemOperations - exists with forward slashes", async () => {
}
});
test("NodeFileSystemOperations - delete with forward slashes", async () => {
test("NodeFileSystemOperations - deleteFile with forward slashes", async () => {
const tempDir = await fs.mkdtemp(path.join(os.tmpdir(), "vaultlink-test-"));
const fsOps = new NodeFileSystemOperations(tempDir);
@ -59,8 +59,10 @@ test("NodeFileSystemOperations - delete with forward slashes", async () => {
await fsOps.write("test.txt", new TextEncoder().encode("test"));
assert.equal(await fsOps.exists("test.txt"), true);
await fsOps.delete("test.txt");
await fsOps.deleteFile("test.txt");
assert.equal(await fsOps.exists("test.txt"), false);
await fsOps.deleteFile("test.txt");
} finally {
await fs.rm(tempDir, { recursive: true, force: true });
}
@ -154,9 +156,46 @@ test("NodeFileSystemOperations - handles paths with forward slashes on all platf
const readContent = await fsOps.read(testPath);
assert.equal(new TextDecoder().decode(readContent), "test");
await fsOps.delete(testPath);
await fsOps.deleteFile(testPath);
assert.equal(await fsOps.exists(testPath), false);
} finally {
await fs.rm(tempDir, { recursive: true, force: true });
}
});
test("NodeFileSystemOperations - delete prunes empty directory trees", async () => {
const tempDir = await fs.mkdtemp(path.join(os.tmpdir(), "vaultlink-test-"));
const fsOps = new NodeFileSystemOperations(tempDir);
try {
await fs.mkdir(path.join(tempDir, "parent", "child"), {
recursive: true
});
await fsOps.delete("parent");
assert.equal(await fsOps.exists("parent"), false);
await fsOps.delete("parent");
} finally {
await fs.rm(tempDir, { recursive: true, force: true });
}
});
test("NodeFileSystemOperations - delete does not remove files", async () => {
const tempDir = await fs.mkdtemp(path.join(os.tmpdir(), "vaultlink-test-"));
const fsOps = new NodeFileSystemOperations(tempDir);
try {
await fsOps.write(
"parent/file.txt",
new TextEncoder().encode("preserve me")
);
await assert.rejects(async () => fsOps.delete("parent"));
assert.equal(await fsOps.exists("parent/file.txt"), true);
await assert.rejects(async () => fsOps.delete("parent/file.txt"));
assert.equal(await fsOps.exists("parent/file.txt"), true);
} finally {
await fs.rm(tempDir, { recursive: true, force: true });
}
});

65
frontend/local-client-cli/src/node-filesystem.ts
View file

@ -15,7 +15,7 @@ import { toUnixPath } from "./path-utils";
export const VAULTLINK_DIR = ".vaultlink";
export class NodeFileSystemOperations implements FileSystemOperations {
public constructor(private readonly basePath: string) { }
public constructor(private readonly basePath: string) {}
public async listFilesRecursively(
directory: RelativePath | undefined
@ -105,9 +105,36 @@ export class NodeFileSystemOperations implements FileSystemOperations {
}
public async delete(relativePath: RelativePath): Promise<void> {
if (!relativePath) {
throw new Error("Cannot delete the vault root");
}
const fullPath = path.join(this.basePath, relativePath);
try {
await fs.unlink(fullPath);
await this.deleteDirectoryTree(fullPath);
} catch (error) {
throw new Error(
`Failed to delete directory ${fullPath}: ${error instanceof Error ? error.message : String(error)}`
);
}
}
public async deleteFile(relativePath: RelativePath): Promise<void> {
const fullPath = path.join(this.basePath, relativePath);
try {
const entry = await fs.lstat(fullPath).catch((error: unknown) => {
if (this.isMissing(error)) return undefined;
throw error;
});
if (!entry) return;
if (!entry.isFile() || entry.nlink !== 1) {
throw new Error("Cannot unlink a non-regular or linked file");
}
await fs.unlink(fullPath).catch((error: unknown) => {
if (!this.isMissing(error)) throw error;
});
await this.syncDirectory(path.dirname(fullPath));
} catch (error) {
throw new Error(
`Failed to delete file ${fullPath}: ${error instanceof Error ? error.message : String(error)}`
@ -171,6 +198,40 @@ export class NodeFileSystemOperations implements FileSystemOperations {
}
}
private async deleteDirectoryTree(fullPath: string): Promise<void> {
const entry = await fs.lstat(fullPath).catch((error: unknown) => {
if (this.isMissing(error)) return undefined;
throw error;
});
if (!entry) return;
if (!entry.isDirectory()) {
throw new Error("Cannot delete a regular file as a directory");
}
for (const child of await fs.readdir(fullPath, {
withFileTypes: true
})) {
if (!child.isDirectory()) {
throw new Error(
`Directory contains a non-directory entry: ${path.join(fullPath, child.name)}`
);
}
await this.deleteDirectoryTree(path.join(fullPath, child.name));
}
await fs.rmdir(fullPath);
await this.syncDirectory(path.dirname(fullPath));
}
private isMissing(error: unknown): boolean {
return (
typeof error === "object" &&
error !== null &&
"code" in error &&
error.code === "ENOENT"
);
}
private async walkDirectory(
relativePath: string,
files: RelativePath[]

32
frontend/obsidian-plugin/src/obsidian-file-system.ts
View file

@ -152,9 +152,20 @@ export class ObsidianFileSystemOperations implements FileSystemOperations {
}
public async delete(path: RelativePath): Promise<void> {
if (!(await this.vault.adapter.trashSystem(normalizePath(path)))) {
return this.vault.adapter.remove(normalizePath(path));
if (!path) {
throw new Error("Cannot delete the vault root");
}
await this.deleteDirectoryTree(normalizePath(path));
}
public async deleteFile(path: RelativePath): Promise<void> {
path = normalizePath(path);
const entry = await this.vault.adapter.stat(path);
if (!entry) return;
if (entry.type !== "file") {
throw new Error(`Cannot delete non-file: ${path}`);
}
await this.vault.adapter.remove(path);
}
public async rename(
@ -173,4 +184,21 @@ export class ObsidianFileSystemOperations implements FileSystemOperations {
return file;
}
private async deleteDirectoryTree(path: string): Promise<void> {
const entry = await this.vault.adapter.stat(path);
if (!entry) return;
if (entry.type !== "folder") {
throw new Error(`Cannot delete non-directory: ${path}`);
}
const children = await this.vault.adapter.list(path);
if (children.files.length > 0) {
throw new Error(`Cannot delete non-empty directory: ${path}`);
}
for (const folder of children.folders) {
await this.deleteDirectoryTree(folder);
}
await this.vault.adapter.rmdir(path, false);
}
}

1128
frontend/package-lock.json generated
View file

File diff suppressed because it is too large Load diff

Some files were not shown because too many files have changed in this diff Show more