mor
This commit is contained in:
parent
ff1f474324
commit
e151208e15
11 changed files with 2933 additions and 784 deletions
|
|
@ -1,36 +1,71 @@
|
|||
# Seeded sync workloads
|
||||
|
||||
The old random mock filesystem/agent have been replaced by the same v4 adapter,
|
||||
client wrapper, and canonical-state assertions as the scripted suite.
|
||||
Periodic polling is explicitly disabled so it cannot repair missed notifications
|
||||
and hide failures in these workloads. Polling has separate fake-clock tests.
|
||||
The fuzzer runs three real clients against an isolated real server. It shares the
|
||||
scripted suite's filesystem adapter and canonical-state checks. Polling is disabled
|
||||
so it cannot repair missing notifications; separate fake-clock tests cover polling.
|
||||
|
||||
From `frontend`:
|
||||
From `frontend`, after building the server:
|
||||
|
||||
```sh
|
||||
npm run build --workspace sync-client --workspace test-client
|
||||
npm run build --workspace sync-client --workspace deterministic-tests --workspace test-client
|
||||
node test-client/dist/cli.js --seed 42 --iterations 25
|
||||
node test-client/dist/cli.js --replay /path/to/trace-42.json
|
||||
```
|
||||
|
||||
The server binary must already be built. Each invocation starts and cleans up its
|
||||
own server/database, and prints the replay command before executing any actions.
|
||||
`--artifacts` chooses the trace/result directory. Failures include database,
|
||||
visible/durable disk images, and CAS request records. The default 25 iterations
|
||||
are scenarios selected in shuffled cycles of nine operations, using two or three
|
||||
clients. Existing documents are repeatedly edited, moved, swapped, deleted and
|
||||
recreated. Concurrent edits share an established base; delayed editor events stay
|
||||
queued through reconnection and a subsequent remote edit. Create, content-update
|
||||
and manifest requests receive before/after network faults.
|
||||
Each invocation owns its server and database. `--artifacts` selects the output
|
||||
directory. The trace is written before execution; results retain exercised request
|
||||
and crash checkpoints, response types and coverage counts. Failures also retain
|
||||
metadata, visible/durable disk images and CAS requests before cleanup.
|
||||
|
||||
Each checkpoint serializes a complete expected file set with exact bytes, paths
|
||||
and persistent identity keys. Independent creates must remain distinct, and a
|
||||
recreated file must obtain a new UUID. Later checkpoints reassert earlier edits
|
||||
and moves, rather than retaining only their markers. Old traces without a client
|
||||
count still replay with two clients.
|
||||
## Workload and oracle
|
||||
|
||||
All preservation assertions run even with delayed editor events. Explicit deletes
|
||||
and overwrites exempt only their specific markers; unrelated content still must
|
||||
appear exactly once. UUID continuity and canonical server state are checked too.
|
||||
Seeds reproduce choices, not OS scheduling. See the
|
||||
Version 2 traces contain an editor-intent program and its compiled steps.
|
||||
`model.ts` independently interprets that intent into complete expected file sets:
|
||||
exact bytes, paths and persistent identity keys. It imports no production merge
|
||||
implementation and never adopts observed output as the expected state. Deleted
|
||||
identity keys remain reserved. Concurrent text cases specify exact allowed results;
|
||||
binary conflicts specify the documented first-committed winner.
|
||||
|
||||
Each epoch composes arbitrary creates, replacements, range deletions, renames,
|
||||
swaps and immediate delete/recreate operations before synchronization. Conflict
|
||||
copies stay in the document pool and are subsequently edited, renamed and deleted.
|
||||
Sixteen fault/concurrency families run in seeded shuffled cycles, including:
|
||||
|
||||
- Three writers editing separated fields and two writers making overlapping edits.
|
||||
- Forced stale-base races: hold a content CAS while another writer commits, rename
|
||||
the local file, then require a new `StaleBase` response and the exact merged bytes.
|
||||
- Online primitive histories and offline batches, delayed editor notifications,
|
||||
and failures before or after durable metadata saves.
|
||||
- Requests held before send or after receipt while further writes/renames occur;
|
||||
dropped create, content and manifest requests with exact retry checks.
|
||||
- Abrupt worker termination after accepted CAS responses, at metadata-save
|
||||
boundaries, and during remote file installation, followed by user edits and
|
||||
recovery. Some histories also kill and restart the server.
|
||||
|
||||
Inputs include empty text, invalid UTF-8 bytes, Unicode, combining characters,
|
||||
emoji, BOMs, CRLFs, size/cache boundaries, long filenames, case/normalization
|
||||
aliases and file/directory conflicts. Initial inventories include 70 and 130
|
||||
documents to exercise paginated catchup; caches range from zero to 64 KiB.
|
||||
Every checkpoint checks the full expected state on all replicas and the server.
|
||||
Full-state checks also repeat after delayed events and crash recovery.
|
||||
|
||||
## Campaigns, replay and reduction
|
||||
|
||||
`scripts/e2e.sh` replays every checked-in `corpus/*.json` trace before fresh random
|
||||
seeds. `E2E_SEED` pins a campaign; without it the runner prints and saves a fresh
|
||||
uint32 seed range. Scheduled CI runs use 100 rounds per worker. The aggregate
|
||||
`coverage.json` reports actions, CAS responses/payloads and exercised fault
|
||||
boundaries. These are behavioral counters, not source branch-coverage percentages.
|
||||
|
||||
On failure, the CLI automatically tries to remove epochs and individual actions,
|
||||
recompiling every expected state from the remaining intent. Dependency-invalid
|
||||
histories are skipped. A reduction must reproduce the same failure twice before
|
||||
acceptance. `--shrink-attempts` controls the budget (default 32); `--no-shrink`
|
||||
retains just the original reproducer. Replay `minimized-<seed>.json`, fix the bug,
|
||||
and commit that concrete program to `corpus/`. Version 1 action traces still replay.
|
||||
|
||||
Request gates and crash targets are replayed and must actually fire. UUID-derived
|
||||
paths use symbolic identity keys. Seeds do not control OS or server scheduling,
|
||||
and replay does not reuse production UUIDs. This is broader bug discovery, not a
|
||||
proof of all possible interleavings or correctness. See the
|
||||
[full harness documentation](../deterministic-tests/README.md).
|
||||
|
|
|
|||
59
frontend/test-client/corpus/conflict-copy.json
Normal file
59
frontend/test-client/corpus/conflict-copy.json
Normal file
|
|
@ -0,0 +1,59 @@
|
|||
{
|
||||
"version": 2,
|
||||
"seed": 1,
|
||||
"iterations": 3,
|
||||
"clients": 3,
|
||||
"steps": [],
|
||||
"program": {
|
||||
"clients": 3,
|
||||
"cacheBytes": 0,
|
||||
"initial": [
|
||||
{
|
||||
"type": "create",
|
||||
"key": "sentinel",
|
||||
"path": "untouched.bin",
|
||||
"content": [0, 255, 254, 128, 13, 10]
|
||||
}
|
||||
],
|
||||
"epochs": [
|
||||
{
|
||||
"writer": 0,
|
||||
"actions": [
|
||||
{
|
||||
"type": "pair",
|
||||
"key": "first",
|
||||
"other": "conflict",
|
||||
"peer": 1,
|
||||
"path": "café/note.md",
|
||||
"content": "\ufefffirst\r\n",
|
||||
"peerContent": "日本語 🦊\n"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"writer": 2,
|
||||
"actions": [
|
||||
{
|
||||
"type": "write",
|
||||
"key": "conflict",
|
||||
"content": [0, 255, 1]
|
||||
},
|
||||
{ "type": "rename", "key": "conflict", "path": "moved.bin" }
|
||||
]
|
||||
},
|
||||
{
|
||||
"writer": 1,
|
||||
"actions": [
|
||||
{ "type": "delete", "key": "conflict" },
|
||||
{
|
||||
"type": "create",
|
||||
"key": "new",
|
||||
"path": "moved.bin",
|
||||
"content": "new text\n"
|
||||
},
|
||||
{ "type": "write", "key": "first", "content": "" }
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
53
frontend/test-client/corpus/stale-rename.json
Normal file
53
frontend/test-client/corpus/stale-rename.json
Normal file
|
|
@ -0,0 +1,53 @@
|
|||
{
|
||||
"version": 2,
|
||||
"seed": 2,
|
||||
"iterations": 1,
|
||||
"clients": 3,
|
||||
"program": {
|
||||
"clients": 3,
|
||||
"cacheBytes": 1024,
|
||||
"initial": [
|
||||
{
|
||||
"type": "create",
|
||||
"key": "note",
|
||||
"path": "note.md",
|
||||
"content": "header\n\nfirst\n\nsecond\n\nfooter\n"
|
||||
}
|
||||
],
|
||||
"epochs": [
|
||||
{
|
||||
"writer": 0,
|
||||
"actions": [
|
||||
{
|
||||
"type": "concurrent",
|
||||
"key": "note",
|
||||
"writes": [
|
||||
{
|
||||
"client": 0,
|
||||
"content": "header\n\nLOCAL\n\nsecond\n\nfooter\n"
|
||||
},
|
||||
{
|
||||
"client": 1,
|
||||
"content": "header\n\nfirst\n\nREMOTE\n\nfooter\n"
|
||||
}
|
||||
],
|
||||
"expected": ["header\n\nLOCAL\n\nREMOTE\n\nfooter\n"]
|
||||
}
|
||||
],
|
||||
"hold": {
|
||||
"kind": "content",
|
||||
"point": "before"
|
||||
},
|
||||
"contender": 1,
|
||||
"tail": [
|
||||
{
|
||||
"type": "rename",
|
||||
"key": "note",
|
||||
"path": "renamed.md"
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
},
|
||||
"steps": []
|
||||
}
|
||||
73
frontend/test-client/corpus/uncertain-edits.json
Normal file
73
frontend/test-client/corpus/uncertain-edits.json
Normal file
|
|
@ -0,0 +1,73 @@
|
|||
{
|
||||
"version": 2,
|
||||
"seed": 0,
|
||||
"iterations": 3,
|
||||
"clients": 3,
|
||||
"steps": [],
|
||||
"program": {
|
||||
"clients": 3,
|
||||
"cacheBytes": 1024,
|
||||
"initial": [
|
||||
{
|
||||
"type": "create",
|
||||
"key": "note",
|
||||
"path": "note.md",
|
||||
"content": "BASE\n"
|
||||
}
|
||||
],
|
||||
"epochs": [
|
||||
{
|
||||
"writer": 0,
|
||||
"actions": [
|
||||
{
|
||||
"type": "write",
|
||||
"key": "note",
|
||||
"content": "BASE\nEDIT\n"
|
||||
}
|
||||
],
|
||||
"hold": { "kind": "content", "point": "before" },
|
||||
"drop": { "kind": "content", "point": "after" },
|
||||
"tail": [
|
||||
{ "type": "rename", "key": "note", "path": "renamed.md" },
|
||||
{
|
||||
"type": "write",
|
||||
"key": "note",
|
||||
"content": "BASE\nEDIT\nLATER\n"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"writer": 1,
|
||||
"actions": [
|
||||
{ "type": "delete", "key": "note" },
|
||||
{
|
||||
"type": "create",
|
||||
"key": "replacement",
|
||||
"path": "renamed.md",
|
||||
"content": "NEW GENERATION\n"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"writer": 2,
|
||||
"actions": [
|
||||
{
|
||||
"type": "create",
|
||||
"key": "crash",
|
||||
"path": "crash.md",
|
||||
"content": "ACCEPTED\n"
|
||||
}
|
||||
],
|
||||
"crash": { "stopAt": "create" },
|
||||
"serverCrash": true,
|
||||
"tail": [
|
||||
{
|
||||
"type": "write",
|
||||
"key": "crash",
|
||||
"content": "ACCEPTED\nEDIT AFTER KILL\n"
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
|
|
@ -7,6 +7,8 @@ import { ServerControl } from "../../deterministic-tests/src/server-control";
|
|||
import { ServerManager } from "../../deterministic-tests/src/server-manager";
|
||||
import { TestRunner } from "../../deterministic-tests/src/test-runner";
|
||||
import { generateTrace, traceTest, type Trace } from "./workload";
|
||||
import { compileProgram } from "./model";
|
||||
import { minimizeProgram } from "./minimize";
|
||||
|
||||
function integer(value: string, zero = false): number {
|
||||
const number = Number(value);
|
||||
|
|
@ -25,6 +27,8 @@ async function main(): Promise<void> {
|
|||
seed: { type: "string", default: "1" },
|
||||
iterations: { type: "string", default: "25" },
|
||||
replay: { type: "string" },
|
||||
"no-shrink": { type: "boolean", default: false },
|
||||
"shrink-attempts": { type: "string", default: "32" },
|
||||
artifacts: { type: "string" },
|
||||
"server-binary": { type: "string" },
|
||||
config: { type: "string" }
|
||||
|
|
@ -40,6 +44,7 @@ async function main(): Promise<void> {
|
|||
: fs.mkdtempSync(path.join(os.tmpdir(), "vault-link-fuzz-"));
|
||||
fs.mkdirSync(artifacts, { recursive: true });
|
||||
const tracePath = path.join(artifacts, `trace-${trace.seed}.json`);
|
||||
if (trace.version === 2) trace.steps = compileProgram(trace.program!);
|
||||
fs.writeFileSync(tracePath, JSON.stringify(trace, null, 2));
|
||||
console.log(`Replay: node ${__filename} --replay ${tracePath}`);
|
||||
const logger = new Logger();
|
||||
|
|
@ -66,7 +71,65 @@ async function main(): Promise<void> {
|
|||
path.join(artifacts, `result-${trace.seed}.json`),
|
||||
JSON.stringify(result, null, 2)
|
||||
);
|
||||
if (!result.success) throw new Error(result.error);
|
||||
if (!result.success) {
|
||||
if (trace.program && !values["no-shrink"]) {
|
||||
try {
|
||||
const reduced = await minimizeProgram(
|
||||
trace.program,
|
||||
result.error ?? "",
|
||||
async (program, attempt) => {
|
||||
await server.stop();
|
||||
await server.start();
|
||||
const candidate = {
|
||||
...trace,
|
||||
program,
|
||||
steps: compileProgram(program)
|
||||
};
|
||||
const outcome = await new TestRunner(
|
||||
server,
|
||||
logger,
|
||||
"test-token-change-me",
|
||||
server.remoteUri
|
||||
).runTest(
|
||||
`shrink-${attempt}`,
|
||||
traceTest(candidate)
|
||||
);
|
||||
fs.writeFileSync(
|
||||
path.join(artifacts, `shrink-${attempt}.json`),
|
||||
JSON.stringify(outcome, null, 2)
|
||||
);
|
||||
return outcome;
|
||||
},
|
||||
integer(values["shrink-attempts"]!)
|
||||
);
|
||||
const minimized = {
|
||||
...trace,
|
||||
program: reduced.program,
|
||||
steps: compileProgram(reduced.program)
|
||||
};
|
||||
const minimizedPath = path.join(
|
||||
artifacts,
|
||||
`minimized-${trace.seed}.json`
|
||||
);
|
||||
fs.writeFileSync(
|
||||
minimizedPath,
|
||||
JSON.stringify(minimized, null, 2)
|
||||
);
|
||||
console.log(
|
||||
`Reduced in ${reduced.attempts} attempts: node ${__filename} --replay ${minimizedPath}`
|
||||
);
|
||||
} catch (error) {
|
||||
fs.writeFileSync(
|
||||
path.join(artifacts, "shrink-error.txt"),
|
||||
String(error)
|
||||
);
|
||||
console.error(
|
||||
`Reduction failed; original trace and failure retained: ${String(error)}`
|
||||
);
|
||||
}
|
||||
}
|
||||
throw new Error(result.error);
|
||||
}
|
||||
} finally {
|
||||
await server.stop();
|
||||
manager.untrack(server);
|
||||
|
|
|
|||
80
frontend/test-client/src/minimize.ts
Normal file
80
frontend/test-client/src/minimize.ts
Normal file
|
|
@ -0,0 +1,80 @@
|
|||
import { compileProgram, type Program } from "./model";
|
||||
|
||||
export interface Failure {
|
||||
success: boolean;
|
||||
error?: string;
|
||||
}
|
||||
export function failureSignature(error: string): string {
|
||||
return error
|
||||
.replace(
|
||||
/[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}/gi,
|
||||
"<uuid>"
|
||||
)
|
||||
.replace(/localhost:\d+/g, "localhost:<port>");
|
||||
}
|
||||
|
||||
/** Dependency-invalid candidates never run. Every candidate regenerates its
|
||||
* oracle from surviving editor intents; expected states are never weakened. */
|
||||
export async function minimizeProgram(
|
||||
original: Program,
|
||||
failure: string,
|
||||
run: (program: Program, attempt: number) => Promise<Failure>,
|
||||
budget = 32
|
||||
): Promise<{ program: Program; attempts: number }> {
|
||||
let current = structuredClone(original),
|
||||
attempts = 0;
|
||||
const signature = failureSignature(failure);
|
||||
const accept = async (candidate: Program): Promise<boolean> => {
|
||||
if (attempts + 2 > budget) return false;
|
||||
try {
|
||||
compileProgram(candidate);
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
// Require a reproducible failure, including its oracle and target.
|
||||
for (let repeat = 0; repeat < 2; repeat++) {
|
||||
const result = await run(candidate, ++attempts);
|
||||
if (
|
||||
result.success ||
|
||||
failureSignature(result.error ?? "") !== signature
|
||||
)
|
||||
return false;
|
||||
}
|
||||
current = candidate;
|
||||
return true;
|
||||
};
|
||||
for (
|
||||
let chunk = Math.max(1, Math.floor(current.epochs.length / 2));
|
||||
chunk >= 1;
|
||||
chunk = Math.floor(chunk / 2)
|
||||
) {
|
||||
for (
|
||||
let index = 0;
|
||||
index < current.epochs.length && attempts + 2 <= budget;
|
||||
|
||||
) {
|
||||
const candidate = structuredClone(current);
|
||||
candidate.epochs.splice(index, chunk);
|
||||
if (!(await accept(candidate))) index += chunk;
|
||||
}
|
||||
}
|
||||
for (
|
||||
let epoch = 0;
|
||||
epoch < current.epochs.length && attempts + 2 <= budget;
|
||||
epoch++
|
||||
) {
|
||||
for (const field of ["actions", "tail"] as const) {
|
||||
for (
|
||||
let index = 0;
|
||||
index < (current.epochs[epoch][field]?.length ?? 0) &&
|
||||
attempts + 2 <= budget;
|
||||
|
||||
) {
|
||||
const candidate = structuredClone(current);
|
||||
candidate.epochs[epoch][field]!.splice(index, 1);
|
||||
if (!(await accept(candidate))) index++;
|
||||
}
|
||||
}
|
||||
}
|
||||
return { program: current, attempts };
|
||||
}
|
||||
438
frontend/test-client/src/model.ts
Normal file
438
frontend/test-client/src/model.ts
Normal file
|
|
@ -0,0 +1,438 @@
|
|||
import assert from "node:assert/strict";
|
||||
import type {
|
||||
ExpectedDocument,
|
||||
TestStep
|
||||
} from "../../deterministic-tests/src/test-definition";
|
||||
import type { CrashWorkerInput } from "../../deterministic-tests/src/crash-worker";
|
||||
import type { RequestKind } from "../../test-support/network";
|
||||
|
||||
export type Content = string | number[];
|
||||
export type Mutation =
|
||||
| { type: "create"; key: string; path: string; content: Content }
|
||||
| { type: "write"; key: string; content: Content }
|
||||
| { type: "rename"; key: string; path: string }
|
||||
| { type: "delete"; key: string }
|
||||
| { type: "swap"; key: string; other: string; temporary: string }
|
||||
| {
|
||||
type: "pair";
|
||||
key: string;
|
||||
other: string;
|
||||
peer: number;
|
||||
path: string;
|
||||
peerPath?: string;
|
||||
content: Content;
|
||||
peerContent: Content;
|
||||
}
|
||||
| {
|
||||
type: "concurrent";
|
||||
key: string;
|
||||
writes: { client: number; content: Content }[];
|
||||
expected: Content[];
|
||||
};
|
||||
export interface Epoch {
|
||||
writer: number;
|
||||
online?: boolean;
|
||||
actions: Mutation[];
|
||||
tail?: Mutation[];
|
||||
hold?: { kind: RequestKind; point: "before" | "after" };
|
||||
contender?: number;
|
||||
drop?: { kind: RequestKind; point: "before" | "after" };
|
||||
crash?: Pick<CrashWorkerInput, "stopAt" | "stopAtBoundary">;
|
||||
serverCrash?: boolean;
|
||||
delayedTail?: boolean;
|
||||
saveFailure?: "before" | "durable";
|
||||
download?: {
|
||||
peer: number;
|
||||
actions: Mutation[];
|
||||
crashPath: string;
|
||||
point: "before" | "visible" | "durable";
|
||||
};
|
||||
}
|
||||
export interface Program {
|
||||
clients: number;
|
||||
initial: Extract<Mutation, { type: "create" }>[];
|
||||
epochs: Epoch[];
|
||||
cacheBytes: number;
|
||||
}
|
||||
|
||||
/** The oracle only evaluates editor intent. It never imports sync/merge code or
|
||||
* reads an implementation result. Compilation is also used after shrinking:
|
||||
* removing an action recomputes every subsequent expected state. */
|
||||
export class ReferenceModel {
|
||||
public readonly documents = new Map<string, ExpectedDocument>();
|
||||
private readonly used = new Set<string>();
|
||||
public get(key: string): ExpectedDocument {
|
||||
const doc = this.documents.get(key);
|
||||
assert(doc, `Invalid program: missing document ${key}`);
|
||||
return doc;
|
||||
}
|
||||
public add(key: string, path: string, content: Content): void {
|
||||
path = path.normalize("NFC");
|
||||
assert(!this.used.has(key), `Invalid program: reused key ${key}`);
|
||||
assert(
|
||||
![...this.documents.values()].some((d) => d.path === path),
|
||||
`Invalid program: occupied path ${path}`
|
||||
);
|
||||
this.used.add(key);
|
||||
this.documents.set(key, {
|
||||
key,
|
||||
path,
|
||||
content: structuredClone(content)
|
||||
});
|
||||
}
|
||||
public apply(action: Mutation): void {
|
||||
switch (action.type) {
|
||||
case "create":
|
||||
this.add(action.key, action.path, action.content);
|
||||
break;
|
||||
case "pair": {
|
||||
this.add(
|
||||
action.key,
|
||||
action.path.normalize("NFC"),
|
||||
action.content
|
||||
);
|
||||
const path = action.peerPath ?? action.path;
|
||||
const dot = path.lastIndexOf(".");
|
||||
const split =
|
||||
dot > path.lastIndexOf("/") + 1 ? dot : path.length;
|
||||
this.add(
|
||||
action.other,
|
||||
`${path.slice(0, split)} (conflict {{${action.other}}})${path.slice(split)}`.normalize(
|
||||
"NFC"
|
||||
),
|
||||
action.peerContent
|
||||
);
|
||||
break;
|
||||
}
|
||||
case "write": {
|
||||
const doc = this.get(action.key);
|
||||
doc.content = structuredClone(action.content);
|
||||
delete doc.alternatives;
|
||||
break;
|
||||
}
|
||||
case "rename": {
|
||||
assert(
|
||||
![...this.documents.values()].some(
|
||||
(d) =>
|
||||
d.key !== action.key &&
|
||||
d.path === action.path.normalize("NFC")
|
||||
),
|
||||
"Invalid program: occupied rename"
|
||||
);
|
||||
this.get(action.key).path = action.path.normalize("NFC");
|
||||
break;
|
||||
}
|
||||
case "delete":
|
||||
this.get(action.key);
|
||||
this.documents.delete(action.key);
|
||||
break;
|
||||
case "swap": {
|
||||
const a = this.get(action.key),
|
||||
b = this.get(action.other);
|
||||
assert(a !== b, "Invalid program: self swap");
|
||||
assert(
|
||||
![...this.documents.values()].some(
|
||||
(d) => d.path === action.temporary.normalize("NFC")
|
||||
),
|
||||
"Invalid program: occupied temporary path"
|
||||
);
|
||||
[a.path, b.path] = [b.path, a.path];
|
||||
break;
|
||||
}
|
||||
case "concurrent": {
|
||||
assert(
|
||||
action.expected.length > 0 && action.writes.length >= 2,
|
||||
"Invalid program: empty concurrency oracle"
|
||||
);
|
||||
const doc = this.get(action.key);
|
||||
doc.content = structuredClone(action.expected[0]);
|
||||
doc.alternatives = structuredClone(action.expected.slice(1));
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
public expected(): ExpectedDocument[] {
|
||||
return structuredClone([...this.documents.values()]);
|
||||
}
|
||||
}
|
||||
|
||||
export function compileProgram(program: Program): TestStep[] {
|
||||
assert(
|
||||
Number.isInteger(program.clients) &&
|
||||
program.clients >= 2 &&
|
||||
program.clients <= 8,
|
||||
"Invalid program: clients"
|
||||
);
|
||||
assert(
|
||||
Number.isFinite(program.cacheBytes) && program.cacheBytes >= 0,
|
||||
"Invalid program: cache size"
|
||||
);
|
||||
const steps: TestStep[] = [];
|
||||
const model = new ReferenceModel();
|
||||
const all = (type: "enable-sync" | "disable-sync", first = 0) => {
|
||||
for (let i = 0; i < program.clients; i++)
|
||||
steps.push({ type, client: (first + i) % program.clients });
|
||||
};
|
||||
const checkpoint = () =>
|
||||
steps.push(
|
||||
{ type: "barrier" },
|
||||
{ type: "assert-documents", expected: model.expected() }
|
||||
);
|
||||
const write = (client: number, path: string, content: Content) =>
|
||||
steps.push(
|
||||
typeof content === "string"
|
||||
? { type: "update", client, path, content }
|
||||
: { type: "create-bytes", client, path, bytes: content }
|
||||
);
|
||||
const mutate = (action: Mutation, client: number, remember: boolean) => {
|
||||
switch (action.type) {
|
||||
case "create":
|
||||
write(client, action.path, action.content);
|
||||
if (remember)
|
||||
steps.push({
|
||||
type: "remember-local-identity",
|
||||
client,
|
||||
path: action.path,
|
||||
key: action.key
|
||||
});
|
||||
break;
|
||||
case "pair":
|
||||
assert(
|
||||
remember &&
|
||||
action.peer !== client &&
|
||||
Number.isInteger(action.peer) &&
|
||||
action.peer >= 0 &&
|
||||
action.peer < program.clients,
|
||||
"Invalid program: independent creates must be offline on distinct clients"
|
||||
);
|
||||
for (const [owner, key, content] of [
|
||||
[client, action.key, action.content],
|
||||
[action.peer, action.other, action.peerContent]
|
||||
] as const) {
|
||||
const path =
|
||||
owner === client
|
||||
? action.path
|
||||
: (action.peerPath ?? action.path);
|
||||
write(owner, path, content);
|
||||
steps.push({
|
||||
type: "remember-local-identity",
|
||||
client: owner,
|
||||
path,
|
||||
key
|
||||
});
|
||||
}
|
||||
break;
|
||||
case "write":
|
||||
write(client, model.get(action.key).path, action.content);
|
||||
break;
|
||||
case "delete":
|
||||
steps.push({
|
||||
type: "delete",
|
||||
client,
|
||||
path: model.get(action.key).path
|
||||
});
|
||||
break;
|
||||
case "rename":
|
||||
steps.push({
|
||||
type: "rename",
|
||||
client,
|
||||
oldPath: model.get(action.key).path,
|
||||
newPath: action.path
|
||||
});
|
||||
break;
|
||||
case "swap": {
|
||||
const a = model.get(action.key).path,
|
||||
b = model.get(action.other).path;
|
||||
steps.push(
|
||||
{
|
||||
type: "rename",
|
||||
client,
|
||||
oldPath: a,
|
||||
newPath: action.temporary
|
||||
},
|
||||
{ type: "rename", client, oldPath: b, newPath: a },
|
||||
{
|
||||
type: "rename",
|
||||
client,
|
||||
oldPath: action.temporary,
|
||||
newPath: b
|
||||
}
|
||||
);
|
||||
break;
|
||||
}
|
||||
case "concurrent":
|
||||
assert(
|
||||
remember,
|
||||
"Invalid program: concurrent edits need an established offline base"
|
||||
);
|
||||
for (const edit of action.writes) {
|
||||
assert(
|
||||
edit.client >= 0 && edit.client < program.clients,
|
||||
"Invalid program: writer"
|
||||
);
|
||||
write(
|
||||
edit.client,
|
||||
model.get(action.key).path,
|
||||
edit.content
|
||||
);
|
||||
}
|
||||
break;
|
||||
}
|
||||
model.apply(action);
|
||||
};
|
||||
all("enable-sync");
|
||||
steps.push({ type: "barrier" });
|
||||
all("disable-sync");
|
||||
for (let client = 0; client < program.clients; client++)
|
||||
steps.push({
|
||||
type: "set-settings",
|
||||
client,
|
||||
settings: { diffCacheSizeMB: program.cacheBytes / (1024 * 1024) }
|
||||
});
|
||||
for (const doc of program.initial) mutate(doc, 0, true);
|
||||
all("enable-sync");
|
||||
checkpoint();
|
||||
for (const epoch of program.epochs) {
|
||||
const writer = epoch.writer;
|
||||
assert(
|
||||
Number.isInteger(writer) &&
|
||||
writer >= 0 &&
|
||||
writer < program.clients &&
|
||||
epoch.actions.length,
|
||||
"Invalid program: empty epoch or writer"
|
||||
);
|
||||
assert(
|
||||
!epoch.online ||
|
||||
!(epoch.hold || epoch.drop || epoch.crash || epoch.serverCrash),
|
||||
"Invalid program: online fault setup"
|
||||
);
|
||||
assert(
|
||||
!epoch.tail?.length || epoch.hold || epoch.crash || epoch.download,
|
||||
"Invalid program: tail has no scheduling boundary"
|
||||
);
|
||||
assert(
|
||||
!epoch.serverCrash || epoch.crash,
|
||||
"Invalid program: server crash requires a client crash"
|
||||
);
|
||||
assert(
|
||||
!epoch.delayedTail || epoch.hold,
|
||||
"Invalid program: delayed tail requires a request gate"
|
||||
);
|
||||
if (epoch.contender !== undefined)
|
||||
assert(
|
||||
epoch.hold?.kind === "content" &&
|
||||
epoch.hold.point === "before" &&
|
||||
Number.isInteger(epoch.contender) &&
|
||||
epoch.contender >= 0 &&
|
||||
epoch.contender < program.clients &&
|
||||
epoch.contender !== writer,
|
||||
"Invalid program: stale-content race needs a distinct contender and before-content gate"
|
||||
);
|
||||
if (!epoch.online) all("disable-sync");
|
||||
if (epoch.saveFailure)
|
||||
steps.push({
|
||||
type: "fail-save",
|
||||
client: writer,
|
||||
point: epoch.saveFailure
|
||||
});
|
||||
for (const [index, action] of epoch.actions.entries()) {
|
||||
mutate(action, writer, !epoch.online);
|
||||
if (epoch.saveFailure && index === 0)
|
||||
steps.push({ type: "wait-for-save-failure", client: writer });
|
||||
}
|
||||
if (epoch.drop)
|
||||
steps.push({
|
||||
type: "drop-response",
|
||||
client: writer,
|
||||
...epoch.drop
|
||||
});
|
||||
if (epoch.download) {
|
||||
assert(
|
||||
!epoch.online && !epoch.hold && !epoch.drop && !epoch.crash,
|
||||
"Invalid program: download crash controls"
|
||||
);
|
||||
assert(
|
||||
epoch.download.peer !== writer,
|
||||
"Invalid program: self download"
|
||||
);
|
||||
for (const action of epoch.download.actions)
|
||||
mutate(action, epoch.download.peer, true);
|
||||
steps.push({ type: "enable-sync", client: epoch.download.peer });
|
||||
steps.push({ type: "disable-sync", client: epoch.download.peer });
|
||||
steps.push({
|
||||
type: "crash-client",
|
||||
client: writer,
|
||||
at: {
|
||||
stopAtBoundary: {
|
||||
type: "filesystem",
|
||||
label: `${epoch.download.point}:write:${epoch.download.crashPath}`
|
||||
}
|
||||
}
|
||||
});
|
||||
for (const action of epoch.tail ?? []) {
|
||||
assert(
|
||||
action.type === "write",
|
||||
"Invalid program: post-crash namespace notification"
|
||||
);
|
||||
mutate(action, writer, false);
|
||||
}
|
||||
steps.push({ type: "restart-client", client: writer });
|
||||
} else if (epoch.crash) {
|
||||
assert(
|
||||
!epoch.hold && !epoch.drop,
|
||||
"Invalid program: mixed crash gates"
|
||||
);
|
||||
steps.push({
|
||||
type: "crash-client",
|
||||
client: writer,
|
||||
at: epoch.crash
|
||||
});
|
||||
if (epoch.serverCrash)
|
||||
steps.push(
|
||||
{ type: "crash-server" },
|
||||
{ type: "restart-server" }
|
||||
);
|
||||
for (const action of epoch.tail ?? []) {
|
||||
assert(
|
||||
action.type === "write",
|
||||
"Invalid program: post-crash namespace notification"
|
||||
);
|
||||
mutate(action, writer, false);
|
||||
}
|
||||
steps.push({ type: "restart-client", client: writer });
|
||||
} else if (epoch.hold) {
|
||||
steps.push({ type: "hold-request", client: writer, ...epoch.hold });
|
||||
steps.push(
|
||||
{ type: "enable-sync", client: writer, wait: false },
|
||||
{ type: "wait-for-request", client: writer }
|
||||
);
|
||||
if (epoch.delayedTail)
|
||||
steps.push({ type: "delay-notifications", client: writer });
|
||||
if (epoch.contender !== undefined)
|
||||
steps.push({ type: "enable-sync", client: epoch.contender });
|
||||
for (const action of epoch.tail ?? [])
|
||||
mutate(action, writer, false);
|
||||
steps.push({ type: "release-request", client: writer });
|
||||
}
|
||||
if (epoch.drop)
|
||||
steps.push(
|
||||
{ type: "enable-sync", client: writer, wait: false },
|
||||
{ type: "wait-for-response-drop", client: writer }
|
||||
);
|
||||
if (!epoch.online) all("enable-sync", writer);
|
||||
checkpoint();
|
||||
if (epoch.contender !== undefined)
|
||||
steps.push({
|
||||
type: "assert-response",
|
||||
client: writer,
|
||||
kind: "content",
|
||||
responseType: "StaleBase",
|
||||
sinceCheckpoint: true
|
||||
});
|
||||
if (epoch.delayedTail) {
|
||||
steps.push({ type: "flush-notifications", client: writer });
|
||||
checkpoint();
|
||||
}
|
||||
}
|
||||
return steps;
|
||||
}
|
||||
149
frontend/test-client/src/stateful-workload.test.ts
Normal file
149
frontend/test-client/src/stateful-workload.test.ts
Normal file
|
|
@ -0,0 +1,149 @@
|
|||
import assert from "node:assert/strict";
|
||||
import { test } from "node:test";
|
||||
import { generateTrace, traceTest } from "./workload";
|
||||
import { compileProgram, type Program } from "./model";
|
||||
import { minimizeProgram } from "./minimize";
|
||||
|
||||
test("stateful programs replay primitive batches, crash boundaries, byte domains and conflict histories", () => {
|
||||
const seen = new Set<string>();
|
||||
let large = false,
|
||||
empty = false,
|
||||
binary = false,
|
||||
unicode = false,
|
||||
alias = false,
|
||||
immediateRecreate = false;
|
||||
for (let seed = 0; seed < 12; seed++) {
|
||||
const trace = generateTrace(seed, 25);
|
||||
assert.deepEqual(trace, generateTrace(seed, 25));
|
||||
assert.deepEqual(
|
||||
traceTest(trace),
|
||||
traceTest(JSON.parse(JSON.stringify(trace)))
|
||||
);
|
||||
assert(trace.program);
|
||||
large ||= trace.program.initial.length >= 70;
|
||||
for (const step of trace.steps) seen.add(step.type);
|
||||
for (const epoch of trace.program.epochs) {
|
||||
assert(epoch.actions.length);
|
||||
for (let i = 0; i < epoch.actions.length; i++) {
|
||||
const action = epoch.actions[i];
|
||||
if (action.type === "pair")
|
||||
alias ||= action.peerPath !== action.path;
|
||||
immediateRecreate ||=
|
||||
action.type === "delete" &&
|
||||
epoch.actions[i + 1]?.type === "create";
|
||||
if ("content" in action) {
|
||||
empty ||= action.content === "";
|
||||
binary ||= Array.isArray(action.content);
|
||||
unicode ||=
|
||||
typeof action.content === "string" &&
|
||||
/[^\x00-\x7f]/.test(action.content);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
for (const type of [
|
||||
"hold-request",
|
||||
"wait-for-request",
|
||||
"crash-client",
|
||||
"restart-client",
|
||||
"crash-server",
|
||||
"restart-server",
|
||||
"fail-save",
|
||||
"wait-for-save-failure",
|
||||
"delay-notifications",
|
||||
"assert-response",
|
||||
"assert-documents"
|
||||
])
|
||||
assert(seen.has(type), `Missing ${type}`);
|
||||
assert(large && empty && binary && unicode && alias && immediateRecreate);
|
||||
});
|
||||
|
||||
test("shrinking retains the failing editor intent and regenerates a complete independent oracle", async () => {
|
||||
const program: Program = {
|
||||
clients: 3,
|
||||
cacheBytes: 1024,
|
||||
initial: [
|
||||
{ type: "create", key: "a", path: "a.md", content: "A" },
|
||||
{ type: "create", key: "b", path: "b.md", content: "B" }
|
||||
],
|
||||
epochs: [
|
||||
{
|
||||
writer: 0,
|
||||
actions: [{ type: "write", key: "a", content: "irrelevant" }]
|
||||
},
|
||||
{
|
||||
writer: 1,
|
||||
actions: [
|
||||
{ type: "write", key: "a", content: "also irrelevant" },
|
||||
{ type: "write", key: "b", content: "TRIGGER" }
|
||||
]
|
||||
},
|
||||
{
|
||||
writer: 2,
|
||||
actions: [{ type: "rename", key: "a", path: "moved.md" }]
|
||||
}
|
||||
]
|
||||
};
|
||||
const failure = "Custom verification failed: Byte mismatch at b.md";
|
||||
const reduced = await minimizeProgram(
|
||||
program,
|
||||
failure,
|
||||
async (candidate) => {
|
||||
const expected = compileProgram(candidate)
|
||||
.filter((s) => s.type === "assert-documents")
|
||||
.at(-1)!;
|
||||
assert.equal(
|
||||
expected.expected.length,
|
||||
2,
|
||||
"Reducer removed unrelated preservation obligations"
|
||||
);
|
||||
const fails = candidate.epochs.some((epoch) =>
|
||||
epoch.actions.some(
|
||||
(action) =>
|
||||
action.type === "write" && action.content === "TRIGGER"
|
||||
)
|
||||
);
|
||||
return { success: !fails, error: fails ? failure : undefined };
|
||||
}
|
||||
);
|
||||
assert.deepEqual(reduced.program.epochs, [
|
||||
{
|
||||
writer: 1,
|
||||
actions: [{ type: "write", key: "b", content: "TRIGGER" }]
|
||||
}
|
||||
]);
|
||||
const expected = compileProgram(reduced.program)
|
||||
.filter((s) => s.type === "assert-documents")
|
||||
.at(-1)!.expected;
|
||||
assert.deepEqual(expected, [
|
||||
{ key: "a", path: "a.md", content: "A" },
|
||||
{ key: "b", path: "b.md", content: "TRIGGER" }
|
||||
]);
|
||||
});
|
||||
|
||||
test("shrinking rejects dependency-invalid histories and different failure mechanisms", async () => {
|
||||
const trace = generateTrace(0, 1);
|
||||
const original = trace.program!;
|
||||
const reduced = await minimizeProgram(
|
||||
original,
|
||||
"Byte mismatch at a.md",
|
||||
async () => ({ success: false, error: "Checkpoint never reached" }),
|
||||
4
|
||||
);
|
||||
assert.deepEqual(reduced.program, original);
|
||||
assert.throws(
|
||||
() =>
|
||||
compileProgram({
|
||||
...original,
|
||||
epochs: [
|
||||
{
|
||||
writer: 0,
|
||||
actions: [
|
||||
{ type: "write", key: "missing", content: "x" }
|
||||
]
|
||||
}
|
||||
]
|
||||
}),
|
||||
/Invalid program/
|
||||
);
|
||||
});
|
||||
481
frontend/test-client/src/stateful-workload.ts
Normal file
481
frontend/test-client/src/stateful-workload.ts
Normal file
|
|
@ -0,0 +1,481 @@
|
|||
import { Random } from "../../test-support/random";
|
||||
import {
|
||||
compileProgram,
|
||||
ReferenceModel,
|
||||
type Content,
|
||||
type Epoch,
|
||||
type Mutation,
|
||||
type Program
|
||||
} from "./model";
|
||||
import type { Trace } from "./workload";
|
||||
|
||||
const words = [
|
||||
"alpha",
|
||||
"日本語",
|
||||
"café",
|
||||
"e\u0301",
|
||||
"🦊",
|
||||
"🌍",
|
||||
"مرحبا",
|
||||
"שלום",
|
||||
"नमस्ते",
|
||||
"\ufeff",
|
||||
"\r\n",
|
||||
"\n",
|
||||
" ",
|
||||
"\t"
|
||||
];
|
||||
export function randomContent(random: Random): Content {
|
||||
const kind = random.int(8);
|
||||
if (kind === 0) return "";
|
||||
if (kind === 1)
|
||||
return Array.from(
|
||||
{ length: random.pick([1, 2, 255, 1023, 1024, 1025, 4096]) },
|
||||
() => random.int(256)
|
||||
);
|
||||
if (kind === 2) return [0, 255, 254, 128, 13, 10];
|
||||
const text = Array.from({ length: 1 + random.int(30) }, () =>
|
||||
random.pick(words)
|
||||
).join("");
|
||||
return kind === 3 ? text.repeat(128) : text;
|
||||
}
|
||||
|
||||
/** Random primitive histories share a persistent model, including conflict
|
||||
* copies. Only checkpoints settle the system; each epoch composes 2–12 edits. */
|
||||
export function generateStatefulTrace(seed: number, iterations: number): Trace {
|
||||
const random = new Random(seed);
|
||||
const clients = 3;
|
||||
const model = new ReferenceModel();
|
||||
let serial = 0;
|
||||
const key = () => `d${serial++}`;
|
||||
const path = () =>
|
||||
`${random.pick(["shared", "日本語", "café", "space folder"])}/${key()}.${random.pick(["md", "txt", "bin"])}`;
|
||||
const initialCount = random.pick([4, 12, 70, 130]);
|
||||
const program: Program = {
|
||||
clients,
|
||||
cacheBytes: random.pick([0, 1024, 4096, 65536]),
|
||||
initial: [],
|
||||
epochs: []
|
||||
};
|
||||
for (let i = 0; i < initialCount; i++) {
|
||||
const doc: Extract<Mutation, { type: "create" }> = {
|
||||
type: "create",
|
||||
key: key(),
|
||||
path: i === 0 ? `${"é".repeat(126)}.md` : path(),
|
||||
content: randomContent(random)
|
||||
};
|
||||
program.initial.push(doc);
|
||||
model.apply(doc);
|
||||
}
|
||||
const chooseDoc = () => random.pick([...model.documents.values()]);
|
||||
const push = (epoch: Epoch, action: Mutation) => {
|
||||
epoch.actions.push(action);
|
||||
model.apply(action);
|
||||
};
|
||||
const families: number[] = [];
|
||||
for (let round = 0; round < iterations; round++) {
|
||||
const writer = random.int(clients);
|
||||
const epoch: Epoch = { writer, actions: [] };
|
||||
// Cover each fault family in shuffled cycles, composing it with the
|
||||
// arbitrary primitive history and identities left by earlier rounds.
|
||||
if (!families.length) {
|
||||
families.push(...Array.from({ length: 16 }, (_, index) => index));
|
||||
for (let i = families.length - 1; i > 0; i--) {
|
||||
const j = random.int(i + 1);
|
||||
[families[i], families[j]] = [families[j], families[i]];
|
||||
}
|
||||
}
|
||||
const family = families.pop()!;
|
||||
if (family === 0) {
|
||||
const first = key(),
|
||||
second = key();
|
||||
const names = random.pick([
|
||||
[`alias-${round}/note.md`, `alias-${round}/note.md`],
|
||||
[`alias-${round}/README.md`, `alias-${round}/readme.md`],
|
||||
[`alias-${round}/café.md`, `alias-${round}/cafe\u0301.md`],
|
||||
[`alias-${round}/folder/note.md`, `alias-${round}/folder`]
|
||||
]);
|
||||
push(epoch, {
|
||||
type: "pair",
|
||||
key: first,
|
||||
other: second,
|
||||
peer: (writer + 1) % clients,
|
||||
path: names[0],
|
||||
peerPath: names[1],
|
||||
content: randomContent(random),
|
||||
peerContent: randomContent(random)
|
||||
});
|
||||
epoch.drop = {
|
||||
kind: random.pick(["create", "manifest"]),
|
||||
point: random.pick(["before", "after"])
|
||||
};
|
||||
} else if (family === 1) {
|
||||
// Three writers replace three separated fields of one established base.
|
||||
const doc = chooseDoc();
|
||||
const base = `header ${round}\n\nfirst field\n\nsecond field\n\nthird field\n\nfooter\n`;
|
||||
// Establish a known text base; the following epoch races all three writes.
|
||||
const prepare: Epoch = {
|
||||
writer,
|
||||
actions: [
|
||||
{ type: "rename", key: doc.key, path: `merge-${round}.md` },
|
||||
{ type: "write", key: doc.key, content: base }
|
||||
]
|
||||
};
|
||||
for (const action of prepare.actions) model.apply(action);
|
||||
program.epochs.push(prepare);
|
||||
const fields = ["first field", "second field", "third field"];
|
||||
const replacements = fields.map(
|
||||
(_, i) =>
|
||||
`changed ${i} ${random.pick(["日本語", "🦊", "café"])}`
|
||||
);
|
||||
push(epoch, {
|
||||
type: "concurrent",
|
||||
key: doc.key,
|
||||
writes: fields.map((field, client) => ({
|
||||
client,
|
||||
content: base.replace(field, replacements[client])
|
||||
})),
|
||||
expected: [
|
||||
fields.reduce(
|
||||
(text, field, i) =>
|
||||
text.replace(field, replacements[i]),
|
||||
base
|
||||
)
|
||||
]
|
||||
});
|
||||
} else if (family === 8 || family === 9) {
|
||||
const doc = chooseDoc();
|
||||
const text = family === 8;
|
||||
const base: Content = text ? "the quick brown fox\n" : [0, 255, 12];
|
||||
const prepare: Epoch = {
|
||||
writer,
|
||||
actions: [
|
||||
{
|
||||
type: "rename",
|
||||
key: doc.key,
|
||||
path: `overlap-${round}.${text ? "md" : "bin"}`
|
||||
},
|
||||
{ type: "write", key: doc.key, content: base }
|
||||
]
|
||||
};
|
||||
for (const action of prepare.actions) model.apply(action);
|
||||
program.epochs.push(prepare);
|
||||
const values: Content[] = text
|
||||
? ["the slow brown fox\n", "the fast brown fox\n"]
|
||||
: [
|
||||
[0, 254, random.int(256)],
|
||||
[0, 253, random.int(256)]
|
||||
];
|
||||
push(epoch, {
|
||||
type: "concurrent",
|
||||
key: doc.key,
|
||||
writes: values.map((content, i) => ({
|
||||
client: (writer + i) % clients,
|
||||
content
|
||||
})),
|
||||
expected: text
|
||||
? ["the fast slow brown fox\n", "the slow fast brown fox\n"]
|
||||
: [values[0]]
|
||||
});
|
||||
} else if (family === 15) {
|
||||
const doc = chooseDoc();
|
||||
const base = `header ${round}\n\nfirst field\n\nsecond field\n\nfooter\n`;
|
||||
const prepare: Epoch = {
|
||||
writer,
|
||||
actions: [
|
||||
{ type: "rename", key: doc.key, path: `stale-${round}.md` },
|
||||
{ type: "write", key: doc.key, content: base }
|
||||
]
|
||||
};
|
||||
for (const action of prepare.actions) model.apply(action);
|
||||
program.epochs.push(prepare);
|
||||
epoch.contender = (writer + 1) % clients;
|
||||
push(epoch, {
|
||||
type: "concurrent",
|
||||
key: doc.key,
|
||||
writes: [
|
||||
{
|
||||
client: writer,
|
||||
content: base.replace("first field", "local 🦊")
|
||||
},
|
||||
{
|
||||
client: epoch.contender,
|
||||
content: base.replace("second field", "remote 日本語")
|
||||
}
|
||||
],
|
||||
expected: [
|
||||
base
|
||||
.replace("first field", "local 🦊")
|
||||
.replace("second field", "remote 日本語")
|
||||
]
|
||||
});
|
||||
epoch.hold = { kind: "content", point: "before" };
|
||||
epoch.tail = [
|
||||
{
|
||||
type: "rename",
|
||||
key: doc.key,
|
||||
path: `stale-renamed-${round}.md`
|
||||
}
|
||||
];
|
||||
for (const action of epoch.tail) model.apply(action);
|
||||
} else if (family >= 12) {
|
||||
const doc = chooseDoc();
|
||||
const content = `content ${round} 日本語\r\n`;
|
||||
push(epoch, { type: "write", key: doc.key, content });
|
||||
epoch.tail = [
|
||||
{
|
||||
type: "write",
|
||||
key: doc.key,
|
||||
content: content + "EDIT DURING UNCERTAIN OUTCOME\r\n"
|
||||
}
|
||||
];
|
||||
if (family === 12) {
|
||||
epoch.hold = {
|
||||
kind: "content",
|
||||
point: random.pick(["before", "after"])
|
||||
};
|
||||
epoch.drop = {
|
||||
kind: "content",
|
||||
point: random.pick(["before", "after"])
|
||||
};
|
||||
epoch.tail.push({ type: "rename", key: doc.key, path: path() });
|
||||
} else {
|
||||
epoch.crash =
|
||||
family === 13
|
||||
? { stopAt: "content" }
|
||||
: {
|
||||
stopAtBoundary: {
|
||||
type: "persistence",
|
||||
label: random.pick([
|
||||
"before:save",
|
||||
"durable:save"
|
||||
]),
|
||||
afterAccepted: "content"
|
||||
}
|
||||
};
|
||||
epoch.serverCrash = random.int(2) === 0;
|
||||
}
|
||||
for (const action of epoch.tail) model.apply(action);
|
||||
} else if (family === 10) {
|
||||
const doc = chooseDoc();
|
||||
const crashPath = `download-${round}.md`;
|
||||
const base = `BASE ${round}\n`;
|
||||
const prepare: Epoch = {
|
||||
writer,
|
||||
actions: [
|
||||
{ type: "rename", key: doc.key, path: crashPath },
|
||||
{ type: "write", key: doc.key, content: base }
|
||||
]
|
||||
};
|
||||
for (const action of prepare.actions) model.apply(action);
|
||||
program.epochs.push(prepare);
|
||||
// Unrelated local work survives the interrupted remote installation too.
|
||||
push(epoch, {
|
||||
type: "create",
|
||||
key: key(),
|
||||
path: `unrelated-${round}.bin`,
|
||||
content: [0, 255, 13, 10]
|
||||
});
|
||||
const remote: Mutation = {
|
||||
type: "write",
|
||||
key: doc.key,
|
||||
content: base + "REMOTE\n"
|
||||
};
|
||||
epoch.download = {
|
||||
peer: (writer + 1) % clients,
|
||||
actions: [remote],
|
||||
crashPath,
|
||||
point: random.pick(["before", "visible", "durable"])
|
||||
};
|
||||
model.apply(remote);
|
||||
epoch.tail = [
|
||||
{
|
||||
type: "write",
|
||||
key: doc.key,
|
||||
content: base + "REMOTE\nAFTER INTERRUPTION\n"
|
||||
}
|
||||
];
|
||||
for (const action of epoch.tail) model.apply(action);
|
||||
} else {
|
||||
epoch.online = family === 2;
|
||||
if (family === 2 && random.int(2) === 0) {
|
||||
epoch.online = false;
|
||||
epoch.saveFailure = random.pick(["before", "durable"]);
|
||||
push(epoch, {
|
||||
type: "rename",
|
||||
key: chooseDoc().key,
|
||||
path: path()
|
||||
});
|
||||
}
|
||||
const count = 2 + random.int(11);
|
||||
for (let i = 0; i < count; i++) {
|
||||
const doc = chooseDoc();
|
||||
switch (random.int(6)) {
|
||||
case 0:
|
||||
push(epoch, {
|
||||
type: "create",
|
||||
key: key(),
|
||||
path: path(),
|
||||
content: randomContent(random)
|
||||
});
|
||||
break;
|
||||
case 1:
|
||||
push(epoch, {
|
||||
type: "write",
|
||||
key: doc.key,
|
||||
content: randomContent(random)
|
||||
});
|
||||
break;
|
||||
case 2:
|
||||
push(epoch, {
|
||||
type: "rename",
|
||||
key: doc.key,
|
||||
path: path()
|
||||
});
|
||||
break;
|
||||
case 3: {
|
||||
const oldPath = doc.path;
|
||||
push(epoch, { type: "delete", key: doc.key });
|
||||
// Recreate immediately, with no synchronization or checkpoint.
|
||||
push(epoch, {
|
||||
type: "create",
|
||||
key: key(),
|
||||
path: oldPath,
|
||||
content: randomContent(random)
|
||||
});
|
||||
break;
|
||||
}
|
||||
case 4: {
|
||||
const other = random.pick(
|
||||
[...model.documents.values()].filter(
|
||||
(d) => d.key !== doc.key
|
||||
)
|
||||
);
|
||||
push(epoch, {
|
||||
type: "swap",
|
||||
key: doc.key,
|
||||
other: other.key,
|
||||
temporary: `temporary-${round}-${i}.md`
|
||||
});
|
||||
break;
|
||||
}
|
||||
default:
|
||||
push(epoch, {
|
||||
type: "write",
|
||||
key: doc.key,
|
||||
content:
|
||||
typeof doc.content === "string"
|
||||
? Array.from(doc.content)
|
||||
.slice(
|
||||
random.int(
|
||||
Array.from(doc.content)
|
||||
.length + 1
|
||||
)
|
||||
)
|
||||
.join("") +
|
||||
`\nreplacement ${round}-${i} 🦊\n`
|
||||
: [
|
||||
...doc.content.slice(
|
||||
random.int(doc.content.length + 1)
|
||||
),
|
||||
0,
|
||||
random.int(256)
|
||||
]
|
||||
});
|
||||
}
|
||||
}
|
||||
if (family >= 3 && family <= 7) {
|
||||
// A fresh surviving upload guarantees the chosen fault is exercised,
|
||||
// regardless of coalescing in the preceding primitive history.
|
||||
const target = key();
|
||||
push(epoch, {
|
||||
type: "create",
|
||||
key: target,
|
||||
path: `fault-${round}.md`,
|
||||
content: `fault base ${round}\n`
|
||||
});
|
||||
if (family === 3 || family === 4) {
|
||||
epoch.hold = {
|
||||
kind: "create",
|
||||
point: family === 3 ? "before" : "after"
|
||||
};
|
||||
epoch.drop = {
|
||||
kind: "manifest",
|
||||
point: random.pick(["before", "after"])
|
||||
};
|
||||
epoch.tail = [
|
||||
{
|
||||
type: "write",
|
||||
key: target,
|
||||
content: `fault base ${round}\nTAIL ${round}\n`
|
||||
}
|
||||
];
|
||||
if (family === 3)
|
||||
epoch.tail.push({
|
||||
type: "rename",
|
||||
key: target,
|
||||
path: `tail-${round}.md`
|
||||
});
|
||||
epoch.delayedTail = family === 4;
|
||||
for (const action of epoch.tail) model.apply(action);
|
||||
} else {
|
||||
const kind = family === 5 ? "create" : "manifest";
|
||||
epoch.crash =
|
||||
family === 7
|
||||
? {
|
||||
stopAtBoundary: {
|
||||
type: "persistence",
|
||||
label: random.pick([
|
||||
"before:save",
|
||||
"durable:save"
|
||||
]),
|
||||
afterAccepted: kind
|
||||
}
|
||||
}
|
||||
: { stopAt: kind };
|
||||
epoch.serverCrash = family === 6;
|
||||
epoch.tail = [
|
||||
{
|
||||
type: "write",
|
||||
key: target,
|
||||
content: `fault base ${round}\nAFTER CRASH ${round}\n`
|
||||
}
|
||||
];
|
||||
for (const action of epoch.tail) model.apply(action);
|
||||
}
|
||||
}
|
||||
}
|
||||
program.epochs.push(epoch);
|
||||
if (family === 0) {
|
||||
const pair = epoch.actions[0];
|
||||
if (pair.type !== "pair")
|
||||
throw new Error("Missing independent creates");
|
||||
const follow: Epoch = { writer: random.int(clients), actions: [] };
|
||||
push(follow, {
|
||||
type: "write",
|
||||
key: pair.other,
|
||||
content: randomContent(random)
|
||||
});
|
||||
push(follow, { type: "rename", key: pair.other, path: path() });
|
||||
if (random.int(2)) {
|
||||
const previous = model.get(pair.other).path;
|
||||
push(follow, { type: "delete", key: pair.other });
|
||||
push(follow, {
|
||||
type: "create",
|
||||
key: key(),
|
||||
path: previous,
|
||||
content: randomContent(random)
|
||||
});
|
||||
}
|
||||
program.epochs.push(follow);
|
||||
}
|
||||
}
|
||||
return {
|
||||
version: 2,
|
||||
seed,
|
||||
iterations,
|
||||
clients,
|
||||
program,
|
||||
steps: compileProgram(program)
|
||||
};
|
||||
}
|
||||
|
|
@ -1,6 +1,6 @@
|
|||
import { test } from "node:test";
|
||||
import assert from "node:assert/strict";
|
||||
import { generateTrace, traceTest } from "./workload";
|
||||
import { generateLegacyTrace as generateTrace, traceTest } from "./workload";
|
||||
|
||||
test("concrete traces replay all choices and retain preservation assertions with delayed events", () => {
|
||||
const trace = generateTrace(42, 12);
|
||||
|
|
|
|||
|
|
@ -1,3 +1,5 @@
|
|||
import { generateStatefulTrace } from "./stateful-workload";
|
||||
import { compileProgram, type Program } from "./model";
|
||||
import type {
|
||||
ExpectedDocument,
|
||||
TestDefinition,
|
||||
|
|
@ -6,7 +8,8 @@ import type {
|
|||
import { Random } from "../../test-support/random";
|
||||
|
||||
export interface Trace {
|
||||
version: 1;
|
||||
version: 1 | 2;
|
||||
program?: Program;
|
||||
seed: number;
|
||||
iterations: number;
|
||||
/** Old traces omit this and retain their original two-client behavior. */
|
||||
|
|
@ -20,7 +23,7 @@ type Document = ExpectedDocument & { content: string };
|
|||
* Every round checks the whole history, including earlier bytes and UUIDs.
|
||||
* A shuffled cycle covers nine actions; most revisit existing documents.
|
||||
*/
|
||||
export function generateTrace(seed: number, iterations: number): Trace {
|
||||
export function generateLegacyTrace(seed: number, iterations: number): Trace {
|
||||
const random = new Random(seed);
|
||||
const clients = 2 + random.int(2);
|
||||
const steps: TestStep[] = [];
|
||||
|
|
@ -295,16 +298,21 @@ export function generateTrace(seed: number, iterations: number): Trace {
|
|||
|
||||
export function traceTest(trace: Trace): TestDefinition {
|
||||
if (
|
||||
trace.version !== 1 ||
|
||||
!Array.isArray(trace.steps) ||
|
||||
!trace.steps.length ||
|
||||
![1, 2].includes(trace.version) ||
|
||||
(trace.version === 1 &&
|
||||
(!Array.isArray(trace.steps) || !trace.steps.length)) ||
|
||||
(trace.version === 2 &&
|
||||
(!trace.program || trace.clients !== trace.program.clients)) ||
|
||||
(trace.clients !== undefined &&
|
||||
(!Number.isInteger(trace.clients) || trace.clients < 2))
|
||||
)
|
||||
throw new Error("Invalid/empty replay trace");
|
||||
return {
|
||||
clients: trace.clients ?? 2,
|
||||
description: `Seed ${trace.seed}; ${trace.iterations} generated conflict scenarios`,
|
||||
steps: trace.steps
|
||||
description: `Seed ${trace.seed}; ${trace.iterations} generated rounds`,
|
||||
steps:
|
||||
trace.version === 2 ? compileProgram(trace.program!) : trace.steps
|
||||
};
|
||||
}
|
||||
|
||||
export const generateTrace = generateStatefulTrace;
|
||||
|
|
|
|||
Loading…
Reference in a new issue