power loss
This commit is contained in:
parent
0f6b618c0b
commit
d56084e4bc
1 changed files with 18667 additions and 3902 deletions
|
|
@ -1,118 +1,134 @@
|
|||
# Deterministic Tests
|
||||
# Sync test harnesses (API v4)
|
||||
|
||||
Scripted multi-client (with an in-memory filesystem) sync tests that run against a real server. Each test defines a sequence of file operations, sync/server controls, and assertions to exercise a specific conflict or edge case.
|
||||
|
||||
Complements the fuzz-based E2E tests (`test-client`): fuzz tests discover bugs through random operations; deterministic tests pin down exact reproduction sequences for known scenarios.
|
||||
|
||||
## How it works
|
||||
|
||||
Each test is a `TestDefinition`: a client count and an ordered list of steps. The test name is derived from the registry key (which matches the file name). The `TestRunner` spins up N `DeterministicAgent` instances (each wrapping a real `SyncClient` with an `InMemoryFileSystem`) pointed at a shared vault on the server, then executes steps one by one.
|
||||
|
||||
Tests that don't pause the server share a single server process (vault-name isolation). Tests that use `pause-server`/`resume-server` (SIGSTOP/SIGCONT) each get a dedicated server, since SIGSTOP freezes the entire process.
|
||||
|
||||
The runner executes two sequential phases: regular tests on the shared server, then pause-server tests on dedicated servers. Within each phase tests run in parallel up to a concurrency limit.
|
||||
|
||||
## Step types
|
||||
|
||||
Clients always start with syncing disabled.
|
||||
|
||||
**File operations** (per-client, fire-and-forget — sync is enqueued but not awaited):
|
||||
|
||||
- `create`, `update`, `rename`, `delete`
|
||||
- `rename-next-write` — arm a deferred rename that fires the next time the given path is written. Lets a test race a user-rename against an in-flight remote create that's about to land at the same path.
|
||||
|
||||
**Sync control:**
|
||||
|
||||
- `sync` — wait for a specific client or all clients to finish pending operations
|
||||
- `barrier` — retry until all clients converge to identical file state (60s timeout)
|
||||
- `enable-sync` / `disable-sync` — simulate going online/offline
|
||||
- `reset` — reset a client's tracked sync state (keeps disk files); equivalent to a forced re-handshake on next enable
|
||||
- `sleep` — wall-clock pause; use sparingly, prefer `barrier` / `sync`
|
||||
|
||||
**WebSocket control** (per-client):
|
||||
|
||||
- `pause-websocket` / `resume-websocket` — buffer/release WebSocket messages for a specific client
|
||||
|
||||
**Server control:**
|
||||
|
||||
- `pause-server` / `resume-server` — SIGSTOP/SIGCONT the server process
|
||||
- `resume-server-until-history-then-pause` — resume the server, wait until a specific client observes a matching history entry (`CREATE`/`UPDATE`/`DELETE` for a path), then re-pause. Used to land exactly one operation across the wire.
|
||||
|
||||
**Fault injection** (per-client):
|
||||
|
||||
- `drop-next-create-response` — arm a one-shot interceptor that lets the next `POST /documents` reach the server (commit happens) but throws `SyncResetError` before the client sees the response, simulating connection loss after server commit.
|
||||
- `wait-for-dropped-create-response` — wait until the armed drop has fired.
|
||||
|
||||
**Assertions:**
|
||||
|
||||
- `assert-consistent` — all clients have identical files; optionally takes a custom `verify(state: AssertableState)` callback
|
||||
|
||||
## Running
|
||||
Run the complete harness check from **any working directory**:
|
||||
|
||||
```sh
|
||||
# Build server first
|
||||
cd sync-server && cargo build --release && cd -
|
||||
|
||||
# Run all tests
|
||||
cd frontend && npm run build -w sync-client && npm run test -w deterministic-tests
|
||||
|
||||
# Filter by name
|
||||
npm run test -w deterministic-tests -- --filter=rename
|
||||
|
||||
# Control parallelism (default: number of CPU cores)
|
||||
npm run test -w deterministic-tests -- -j 4
|
||||
E2E_WORKERS=2 /path/to/vault-link/scripts/e2e.sh
|
||||
```
|
||||
|
||||
## Adding a test
|
||||
The `E2E_WORKERS` variable is the number of fuzz workers / simultaneous scripted tests.
|
||||
The runner builds the server and the three in-scope frontend packages, then runs
|
||||
the harness self-tests, recovery regressions, protocol/real-filesystem tests,
|
||||
scripted scenarios, and seeded fuzz workers. It does not build the Obsidian plugin
|
||||
or CLI application. Install frontend dependencies with `npm ci` first. Node's
|
||||
built-in WebSocket, Rust, and a C compiler (`cc`, for test-only exclusive rename
|
||||
and OS-lock primitives on macOS/Linux) are required.
|
||||
|
||||
1. Create `src/tests/my-scenario.test.ts`:
|
||||
CI installs those dependencies in `frontend` before invoking the runner. It sets
|
||||
`E2E_ARTIFACTS=logs/e2e` and uploads that same directory even when a suite fails.
|
||||
The runner owns all test servers; no separate server or legacy cleanup script is
|
||||
needed in the workflow.
|
||||
|
||||
```typescript
|
||||
import type { TestDefinition } from "../test-definition";
|
||||
Each server gets a fresh temporary database, working directory, and port. Only
|
||||
owned processes are terminated. There is no `pkill`, fixed database deletion,
|
||||
or shared server PID file. Failing tests and teardown errors give nonzero exits;
|
||||
the remaining suites still run. Logs, concrete fuzz traces, results, and failure
|
||||
snapshots are retained in the printed artifact directory. Snapshots are captured
|
||||
before teardown so cleanup failures retain the original evidence.
|
||||
|
||||
export const myScenarioTest: TestDefinition = {
|
||||
description:
|
||||
"Client 0 creates A.md offline. After syncing, both clients should have the file.",
|
||||
clients: 2,
|
||||
steps: [
|
||||
{ type: "create", client: 0, path: "A.md", content: "hello" },
|
||||
{ type: "enable-sync", client: 0 },
|
||||
{ type: "enable-sync", client: 1 },
|
||||
{ type: "barrier" },
|
||||
{
|
||||
type: "assert-consistent",
|
||||
verify: (s) => {
|
||||
s.assertFileCount(1).assertContent("A.md", "hello");
|
||||
}
|
||||
}
|
||||
]
|
||||
};
|
||||
Environment options:
|
||||
|
||||
- `E2E_WORKERS=1`: number of fuzz workers / simultaneous scripted tests (max 32).
|
||||
- `E2E_SEED=1`: first uint32 seed; workers use successive seeds.
|
||||
- `E2E_ITERATIONS=25`: generated conflict scenarios per worker.
|
||||
- `E2E_TIMEOUT_SECONDS=900`: timeout for each build/suite/worker command.
|
||||
- `E2E_ARTIFACTS=/absolute/path`: optional output directory; defaults to `mkdtemp`.
|
||||
|
||||
## Running individual layers
|
||||
|
||||
From `frontend`:
|
||||
|
||||
```sh
|
||||
# Rebuild protocol peers and run scripted tests (name filter optional).
|
||||
npm run test --workspace deterministic-tests -- --filter v4- --concurrency 2
|
||||
|
||||
# Adapter/oracle/lifecycle self-tests and strict recovery regressions.
|
||||
npm run test:harness
|
||||
|
||||
# After rebuilding: real worker death, server SIGKILL/restart, CAS/WS ordering,
|
||||
# and host filesystem contract checks.
|
||||
npm run test:protocol
|
||||
|
||||
npm run build --workspace sync-client --workspace test-client
|
||||
node test-client/dist/cli.js --seed 42 --iterations 25
|
||||
node test-client/dist/cli.js --replay /path/to/trace-42.json
|
||||
```
|
||||
|
||||
The `verify` callback receives an `AssertableState` object with chainable assertion methods:
|
||||
`test:harness` includes strict regressions for **engine** bugs.
|
||||
A failing default run is not an expected-failure test or a harness skip. See
|
||||
[`../test-support/README.md`](../test-support/README.md) for their scope.
|
||||
|
||||
```typescript
|
||||
s.assertFileCount(n); // exact file count
|
||||
s.assertFileExists("path"); // file must exist
|
||||
s.assertFileNotExists("path"); // file must not exist
|
||||
s.assertContent("path", "expected"); // exact content match
|
||||
s.assertContains("path", "a", "b"); // all substrings present in file
|
||||
s.assertContainsAny("path", "a", "b"); // at least one substring present
|
||||
s.assertAnyFileContains("text"); // substring present in some file
|
||||
s.assertNoFileContains("text"); // substring absent from every file
|
||||
s.assertSubstringCount("path", "x", 3); // substring appears exactly N times
|
||||
s.assertContentInAtMostOneFile("text"); // no duplicate content
|
||||
s.ifFileExists("path", (s) => { /* … */ }); // conditional block
|
||||
s.getContent("path"); // raw content (or "" if missing)
|
||||
```
|
||||
## What the scripted runner checks
|
||||
|
||||
2. Register it in `src/test-registry.ts`:
|
||||
Real `SyncClient` instances use the shared v4 adapter in `../test-support`.
|
||||
External editor operations and engine filesystem operations are separate: engine
|
||||
writes never generate self-notifications. Editor identity notifications continue
|
||||
while transport is disabled. Delayed notifications can be explicitly queued and
|
||||
flushed. Snapshots and persisted state are deep copies, never shared references.
|
||||
|
||||
```typescript
|
||||
import { myScenarioTest } from "./tests/my-scenario.test";
|
||||
Scripted and seeded clients explicitly set `syncIntervalMs: 0`, with assertions
|
||||
that it stays disabled. Barriers observe convergence; they do not wake the engine.
|
||||
Network retry timers remain enabled. Separate fake-clock tests in
|
||||
`test-support/polling.test.ts` exercise optional periodic reconciliation.
|
||||
|
||||
const TESTS = {
|
||||
// ...
|
||||
"my-scenario": myScenarioTest
|
||||
};
|
||||
```
|
||||
A barrier checks byte-for-byte replica equality **and** the server's canonical
|
||||
UUID/path map, file bytes, independently computed base/observed hashes, content
|
||||
and manifest CAS parents, event watermark,
|
||||
contiguous event log, and absence of pending requests/application journals.
|
||||
Internal recovery files are excluded from the user-visible file list, but are
|
||||
present in failure disk images. Scenario assertions separately check intended
|
||||
content, document identity, or preservation markers; equality alone is not an
|
||||
adequate data-loss oracle.
|
||||
|
||||
Clients start disabled. `enable-sync` waits for the initial sync attempt/retries,
|
||||
except while the server is intentionally paused: that attempt is tracked so a
|
||||
later script step can resume the server. Thus sequential enables really establish
|
||||
a winner. To test _independent creates_, first bootstrap both clients empty, then
|
||||
disable them and create files. Initial adoption of pre-existing files during
|
||||
bootstrap is a different policy from two already-initialized clients creating
|
||||
distinct UUIDs at one path.
|
||||
|
||||
## Controls and assertions
|
||||
|
||||
- `create`, `update`, `create-bytes`, `rename`, `delete`: external editor actions.
|
||||
Editor rename may explicitly replace its target; engine rename never may.
|
||||
- `sync`, `barrier`, `enable-sync`, `disable-sync`.
|
||||
- `reset`: restart transports; **not** a crash and not a database reset.
|
||||
- `pause-websocket`, `resume-websocket`: buffer only WS delivery. HTTP catchup
|
||||
remains active; this is deliberately not called "disconnect".
|
||||
- `pause-observation`, `wait-for-observation`, `resume-observation`: explicitly
|
||||
gate HTTP catchup as well as WS, including replies already in flight. The
|
||||
checkpoint must be exercised and released, and held reads respect cancellation.
|
||||
- `delay-notifications`, `flush-notifications`: delay editor notifications without
|
||||
disabling preservation assertions.
|
||||
- `drop-response`: `{client, kind: "create" | "content" | "manifest", point:
|
||||
"before" | "after"}`. After faults fire only on an `Accepted` CAS response.
|
||||
- `wait-for-response-drop`: explicit checkpoint; every armed fault must fire.
|
||||
Every interrupted request must be retried with an identical request ID, target
|
||||
and body before a different CAS is sent. Fault kinds match exactly.
|
||||
- `rename-next-write`: inject an editor rename after the engine's durable install
|
||||
rename. The historical name is retained for fixtures; v4 never writes in place.
|
||||
- `pause-server`, `resume-server`: SIGSTOP/SIGCONT, not a crash.
|
||||
- `crash-server`, `restart-server`: SIGKILL and restart with the same database/port.
|
||||
Server-control scenarios have dedicated processes.
|
||||
- `remember-identity`, `assert-identity`, `assert-files`, `assert-markers`.
|
||||
- `assert-documents`: a complete expected set of `{key, path, content, conflict?}`
|
||||
records. Content is text or a byte array. Keys bind UUIDs on the first successful
|
||||
check and must retain them on later checks; new keys cannot reuse deleted UUIDs.
|
||||
`conflict: true` resolves the UUID-derived conflict name for `path`.
|
||||
- `assert-consistent` can provide `verify(state)`: exact text/bytes, UUID lookups,
|
||||
`assertIdentity`, and `conflictPath(original)` for UUID-derived conflict names.
|
||||
|
||||
Legacy timing scenarios are retained. Prefer explicit fault checkpoints and
|
||||
barriers for new tests. Only known coherent-scan races and actually injected
|
||||
transient network errors may retry; they still must converge before the deadline.
|
||||
Other background errors, unhandled rejections, and cleanup failures fail the run.
|
||||
|
||||
## Reproducibility limits
|
||||
|
||||
Seeds and JSON traces reproduce harness choices and logical actions. They do not
|
||||
control the OS scheduler, server thread scheduling, or production-generated UUIDs.
|
||||
The recovery sweep enumerates adapter/save boundaries; worker termination proves
|
||||
abrupt runtime loss. Neither is a proof covering every real filesystem or hardware
|
||||
power-loss behavior. Run multiple seeds, repeat timing-sensitive cases, and keep
|
||||
the scenario's independent oracle when reducing a failing trace.
|
||||
|
|
|
|||
Loading…
Reference in a new issue