andras/vault-link
andras/vault-link
Watch
1
0
Fork
You've already forked vault-link
 0
Code Issues Pull requests 18 Projects Releases 55 Packages Wiki Activity Actions 36

power loss

Browse source
This commit is contained in:
Andras Schmelczer 2026-09-21 21:29:52 +01:00
commit d56084e4bc
113 changed files with 18667 additions and 3902 deletions
Download patch file Download diff file Expand all files Collapse all files

2
README.md
View file

@ -69,7 +69,7 @@ scripts/bump-version.sh patch
#### Run E2E tests
```sh
scripts/e2e.sh 8
E2E_WORKERS=8 scripts/e2e.sh
```
And to clean up the logs & database files, run `scripts/clean-up.sh`

647
SYNC_ENGINE_HANDOVER.md Normal file
View file

@ -0,0 +1,647 @@
# Sync engine v4 handover
Prepared on 2026-09-19. This records the implementation delivered in the preceding
work session: its starting point, intended behavior, changes and their reasons,
validation, and integration work still required.
Follow-up on 2026-09-19: file modification times have since been removed from the
server schema/protocol, client snapshots, and recovery records. Concurrent binary
or unmergeable edits now keep the server version, with displaced local bytes
retained in recovery; changes on only one side still keep that side. References
to `mtime` below describe the earlier implementation and are superseded by this
change. See the server and client READMEs for the current contracts.
Further follow-up on 2026-09-19: request tracking is now part of `events`, using
unique `request_id` and `request_fingerprint` columns. The separate
`request_receipts` table and duplicated response JSON have been removed. Retries
check the fingerprint and reconstruct the original `Accepted` response from the
stored event before CAS. References to separate receipts below describe the
earlier implementation and are superseded by this change.
File manifest storage follow-up on 2026-09-19: complete maps are now stored as rows in
`file_manifest_entries`, keyed by `(file_manifest_id, document_id)`, with exact path
uniqueness per version. `file_manifests` retains a header for every accepted version,
including empty maps. File manifest reads reconstruct the map from these rows, and
vault snapshots join them to the latest content versions. Event rows contain only
ordering and request-deduplication metadata; replay and retry acknowledgements
reconstruct events from normalized document/manifest rows. References below to JSON file
manifest storage describe the earlier implementation.
The baseline is commit `1619c51` (`Migrate server to CAS only`). The implementation
changes are currently in the working tree/index, rather than a new implementation
commit. The file inventory below describes the rewrite relative to that baseline.
The following changes must not be attributed to this implementation:
- While this handover was being written, additional unstaged edits appeared in
`sync-server/src/app_state/database/migrations/20241207143519_bootstrap.sql`,
`sync-server/src/server/endpoints/get_file_manifest.rs`,
`sync-server/src/server/endpoints/put_file_manifest.rs`,
`sync-server/src/server/endpoints/vault_snapshot.rs`,
`sync-server/src/server/endpoints/put_file_content.rs`, and
`sync-server/src/server/endpoints/put_file_content/tests.rs`. They were left alone
and are outside this handover's implementation inventory. The earlier passing
validation applies to the rewrite as delivered, represented by the staged
implementation at the start of this handover, not subsequent edits. Review and
validate the final SQL before starting a new database.
This handover task added only this document; it did not modify implementation
files or change staging decisions.
## Initial state
The repository was in the middle of a protocol migration. It was not a functioning
v4 implementation with a few missing routes.
| Area | Starting behavior | Why it needed to change |
| ------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| Server protocol | Advertised API v3. `PUT /documents/:id` already used a parent version, request UUID, immutable versions, SHA-256 request fingerprints, and durable acknowledgements. Concurrent text merging had already been removed from the server. | Preserve the existing content CAS design and extend its durability model to namespace changes. This rewrite did not originate server-side CAS or idempotency. |
| Content and names | A document version also contained `relative_path` and `is_deleted`; pushes carried a path and supported `Delete`. The server sanitized and allocated available paths. | Content history and namespace decisions were coupled. Whole rename sequences and cross-document conflicts could not be decided atomically as a file manifest. |
| Server persistence | `documents`, a latest-version view, and `push_acknowledgements`; new IDs were allocated from the maximum document update ID inside a write transaction. | There was no common durable event stream for content and file manifests, or file manifest history. |
| Broadcasts | A completed write committed and then broadcast its document metadata. The originating device was skipped. Reconnection fetched latest document heads rather than every event. | Post-commit task scheduling could change notification order; a crash after commit could lose a wakeup. Latest heads are not an ordered history of namespace transitions. |
| Client protocol | Advertised API v2. It still used the earlier create/text/binary/delete service calls and expected `FastForwardUpdate` / `MergingUpdate`. | The client had not caught up with the server's previous CAS migration, let alone the proposed file manifest protocol. |
| Client scheduling | `Syncer`, `UnrestrictedSyncer`, a configurable concurrent queue, document locks, mutable records, and in-memory promises coordinated operations. | Whole-file-manifest changes need a consistent view across documents. A single durable reconciliation loop is easier to reason about when performance is secondary. |
| Client metadata | Persisted document metadata and an update watermark, with mutable runtime records and gap tracking. | No persisted exact request/response workflow or filesystem application journal tied disk changes to metadata advancement. |
| Filesystem | Read/write/rename/delete operations, an atomic text updater, an in-memory safety wrapper, and native line-ending conversion. | The interface did not promise durable data and directory updates or crash-recoverable multi-file moves. |
| Content comparison | A 32-bit rolling checksum. | Collisions could incorrectly hide edits or identify unrelated files as a rename. |
| Reset/settings | Reset cleared local tracking; settings could cause a vault reset. Persistence had no explicit power-loss contract. | Clearing uncertain requests or application state could lose recovery information or apply it to the wrong vault. |
The Obsidian plugin, CLI client, legacy test fixtures, and their adapters were
explicitly outside the implementation scope.
## Expected result and selected policies
The requested result is a simple server CAS store and a client that owns all
reconciliation. The content algorithm follows the fetch/merge/push model in
[Conflict-Free Three-Way Text Merging](https://schmelczer.dev/articles/reconcile-text-3-way-merge/),
including recording the sent snapshot after acceptance and retrying an uncertain
request with its original idempotency key.
The implemented invariants are:
1. Document UUIDs identify content; a file manifest maps UUIDs to paths and owns
membership. A content push cannot rename, delete, or restore membership.
2. File manifest acceptance replaces the entire map only when its parent is current.
Every referenced document must already have content.
3. An accepted content or file manifest mutation commits its state, event, and receipt
together. Each vault has one total order; separate vaults have separate orders.
4. Each WebSocket sends persistent events in that order, including the sender's
own events. Cursors remain ephemeral and do not consume event IDs.
5. The client persists its exact request before sending. An unknown outcome is
resolved before advancing to newer remote state.
6. The client advances a merge base to what was actually incorporated or accepted,
never to an assumption about the current live file.
7. Filesystem application has a durable journal. Every affected source is staged
before destinations are installed, so swaps and cycles do not overwrite files.
8. A received event watermark becomes durable with the corresponding state or
recoverable application obligation. Excluded downloads remain explicit work.
The conflict policies implemented alongside these invariants are:
| Situation | Decision | Reason |
| ------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------- |
| Same UUID, local path unchanged | Use the server path, including absence. | Incorporate remote changes. |
| Same UUID, remote path unchanged | Use the local path, including absence. | Preserve local intent. |
| Same UUID, both paths changed | Use the server path, including absence. | The requested deterministic tie-breaker. |
| Different UUIDs claim conflicting paths | Keep both identities. A selected path matching the server wins priority; UUID order resolves other ties. Give displaced documents deterministic conflict names. | A per-ID merge alone cannot guarantee one-to-one paths. |
| Delete versus content edit | Membership follows the file manifest merge. Preserve locally displaced bytes in recovery; do not automatically recreate membership. | Avoid deletion/resurrection loops while retaining observed data. |
| First sync, existing local text at a remote path | Adopt the remote UUID and merge against an empty parent. | There is no known common ancestor; neither existing side should silently overwrite the other. |
| Concurrent binary/unmergeable changes | Larger originating mtime wins; server wins ties. | A defined policy is necessary when text reconciliation does not apply. Clock skew remains a limitation. |
| Rename notification | Preserve identity through the ordered logical move hints. | Retain identity through online rename sequences. |
| Unobserved/offline rename | Infer only from a unique, nonempty content hash match on both sides; otherwise treat as delete/create. | Identical files and rename-plus-edit sequences cannot be identified reliably from snapshots alone. |
| Deliberate notified delete/recreate | Allocate a new UUID, even at the same path with identical bytes. | A new document must not inherit the deleted document's identity accidentally. |
| Portable-name conflict | Repair names client-side; reject invalid canonical file manifests server-side. | Every client must receive one valid, unambiguous namespace. |
| Ignored, oversized, or unmaterialized remote file | Preserve membership and remember its remote head. | Skipping a download must not publish a deletion. |
This is not a proof of a universally perfect sync engine. Power-loss recovery
depends on the adapter/storage contracts below. Snapshot-based observation cannot
recover writes that were never observed or unambiguously reconstruct offline
identity. Automatic text merging also cannot guarantee semantic correctness of
the resulting prose or code.
## Server changes and rationale
### Independent namespace and content CAS
`PutFileContent` now contains `requestId`, nullable `parentVersionId`, `mtime`, and
`content`. `content` accepts `Snapshot` or `Diff`. Paths, delete payloads, and other
unknown fields are rejected. The server still reconstructs a transport diff against
the supplied parent; it does not reconcile competing edits. The existing guard
against the minimum signed integer in diff lengths was retained.
`mtime` is the originating file time in milliseconds, stored as a finite floating
point value within the JavaScript date range. It is distinct from the server's
commit time. This lets clients apply the selected binary policy without treating
download time as a new edit.
`PushFileManifest` contains `requestId`, `parentFileManifestId`, and `entries`, where entries
are the complete UUID-to-path map. The empty initial file manifest has ID 0. A custom
map deserializer rejects repeated UUID keys, including alternate spellings of the
same UUID; an ordinary JSON map decoder would silently keep one value. A sorted
map gives request fingerprinting a stable key order.
Both mutation kinds return `Accepted` or `StaleBase`. A stale content response
contains the current bytes; a stale file manifest response contains the current map.
Only accepted mutations get durable receipts. Rejected requests have no state
effect and may receive a newer stale base if retried later.
### Schema and transaction boundary
The bootstrap schema was replaced with a fresh v4 schema:
| Object | Purpose and reason |
| ------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `events` | AUTOINCREMENT IDs, request IDs, and fingerprints provide the common order and durable idempotency namespace. IDs are constrained to JavaScript's safe integer range. Event payloads are reconstructed from normalized rows. |
| `documents` | Immutable content versions reference their creating event; no path or deletion fields. UUIDs use the BLOB representation expected by SQLx's UUID binding. `mtime`, author, device, and server timestamp are retained. |
| `documents_by_id` and `latest_document_versions` | Find a UUID's current content without namespace lookups. |
| `file_manifests` and `file_manifest_entries` | Immutable manifest headers and normalized UUID/path rows keyed by their creating event ID. A file manifest version need not equal the latest event ID because content events can follow it. |
The write transaction performs event/request lookup before CAS, then head validation,
event allocation, normalized state insertion, and commit. Request lookup must come
first: a retry of an accepted request is still successful after another
client advances the head. Reusing a request UUID with different content, parent,
document ID, or operation type is rejected. Session/device identity is excluded
from the fingerprint so a restarted client can recover its old outcome.
Read operations were consolidated in `queries.rs`, and write operations in
`mutations.rs`. Runtime SQL queries replace the old schema-dependent SQLx macro
queries in this path. This keeps the new schema and transaction logic together
without requiring the old prepared-query cache to describe v4.
SQLite now explicitly uses WAL, `synchronous=FULL`, `fullfsync`, and foreign keys.
On Unix, database creation also flushes the containing directory and ancestors.
The reason is durability of both transaction contents and newly created directory
entries. These settings still depend on the platform/filesystem honoring flushes.
No migration or automatic deletion of old databases was added. Replacing the
bootstrap migration deliberately makes old migration checksums incompatible.
Use a fresh database directory and retain old data separately.
### Endpoint and replay changes
Paths below are relative to `/vaults/:vault_id`:
| Endpoint | Change and rationale |
| ---------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `PUT /documents/:document_id` | Existing route becomes a thin content-only CAS handler. Database mutation code owns the complete atomic operation. |
| `GET /documents/:document_id` | Retained; returns current content using the new metadata shape. |
| `GET /documents/:document_id/versions/:version_id/content` | Retained; clients fetch immutable bases for merging. |
| `GET /file-manifest` | New; retrieves canonical membership and names. |
| `PUT /file-manifest` | New; validates and CAS-replaces the complete namespace. |
| `GET /vault-snapshot` | New; returns the file manifest, referenced content heads, and event watermark from one read transaction. Prevents a bootstrap gap between separately read state and history. |
| `GET /events-since?after=N` | New; returns every later event in increasing order with the transaction's watermark. Negative/ahead-of-history cursors are rejected rather than silently resetting a client. |
| `GET /ws` | Reworked to send the same ordered durable event history. |
| `GET /documents` | Removed; latest document rows no longer describe membership or preserve namespace transitions. |
WebSocket messages use `vaultEvents` batches containing `events` and
`headEventId`. The handshake retains `lastSeenVaultUpdateId`. One sender per
connection drains the database in order; broadcasts are only wake hints. A
two-second timer also drains the log, covering lost wakes and commit-before-notify
crashes. Broadcast lag triggers another drain rather than loss of persistent
events. HTTP replay is available independently of the socket.
Cursor broadcasts were adapted to the new notification wrapper but remain
ephemeral. Persistent events no longer filter out their originating device:
filtering them would create holes in that client's event sequence.
### Portable file manifest validation
Validation now checks the whole namespace, not just individual strings:
- Relative `/`-separated paths and NFC normalization, without a universal
total-path byte limit.
- No empty, `.` or `..` components; no backslashes, controls, Windows-forbidden
characters, trailing dots/spaces, or reserved device names. The device-name
checks also cover extension forms, trailing spaces in the stem, and the
superscript COM/LPT digit variants.
- `.vault-link-sync` and case/normalization aliases are reserved.
- Comparison uses NFC → Unicode uppercase → NFC.
- No duplicate/alias paths, file-versus-ancestor conflicts, or inconsistent
directory spelling such as `Notes/a.md` alongside `notes/b.md`.
- Every UUID must refer to existing content before membership is accepted.
The server rejects invalid input instead of allocating or sanitizing names. This
keeps the canonical update exactly equal to the file manifest whose CAS the client
requested. Client and server implement the same portable-name policy.
## Client changes and rationale
### Persistent state and the three file manifests
The three conceptual file manifests are represented as follows:
| Concept | Representation | Why |
| --------------------------- | ---------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Incorporated server base, B | `state.fileManifest` with its version ID | A received map becomes the base only after its application is durable or journaled for recovery. |
| Last submitted file manifest, S | `state.pending.request.entries` while a file manifest request is unresolved | Freeze the exact request and parent across network failures and restarts. On acceptance, this exact S becomes B. |
| Current local view, L | `state.local`, refreshed by complete scans and logical notifications | Preserve changes made while an earlier S was in flight. It also tracks unmaterialized/excluded remote membership; it is more than a raw listing of existing disk files. |
S is not kept as a redundant permanent third field after its outcome is resolved.
On acceptance it becomes the incorporated base. On rejection the client merges B,
the current L, and the returned remote map, then prepares a new request with a new
UUID. This preserves the three-way semantics without maintaining duplicate state.
The new format also stores per-document incorporated content metadata and hash,
observed hash, materialization/bootstrap flags, `lastSeenUpdateId`, deferred
`remoteHeads`, the initial vault snapshot while bootstrap is incomplete, pending
requests/responses, and a filesystem application journal.
`Database.commit` clones and saves the complete value before replacing in-memory
state. A save is marked uncertain before it is attempted. If it fails, the next
attempt reloads durable state before doing more work; a failed save may have
persisted either complete version. This avoids continuing from an assumed outcome.
State is bound to the remote URI and vault name. Incompatible formats and live
vault rebinding are rejected instead of silently clearing recovery information.
A truly empty, uninitialized state can be rebound for initial configuration.
### One reconciliation loop
The configurable queue and `UnrestrictedSyncer` were replaced by one serialized
loop per vault. This intentionally trades throughput for a smaller set of states
that can overlap. Its normal progression is:
1. Reload metadata after any uncertain save and recover any filesystem journal.
Recovery runs before a network ping, so an offline server does not block it.
2. Initialize server configuration; if necessary, persist and apply a consistent
bootstrap vault snapshot.
3. Resolve the exact pending request and persist its response before processing
newer remote state.
4. Scan local files, applying ordered logical create/delete/move hints.
5. Replay HTTP events from the durable cursor. Require contiguous IDs and the
advertised watermark; do not silently skip a missing event.
6. Apply file-manifest/content changes, retaining remote heads for deferred downloads.
7. Rescan, retry newly eligible deferred content, and prepare one next content or
file manifest submission.
8. Repeat while there is work; otherwise wake on notifications or a five-second
timer. Transient failures retry at the configured network interval.
WebSocket batches wake this loop. The client uses HTTP replay as its authoritative
event input, avoiding separate merge paths for HTTP and socket delivery. A dirty
flag and completion handling preserve wakes arriving while an attempt finishes.
`waitUntilFinished()` follows successive active attempts and reports failures.
The exact content snapshot, mtime, hash, request ID, parent, and optional cursors
are persisted before sending. After acceptance, the base hash comes from those
sent bytes, not a later disk read. A lost response therefore cannot cause an edit
to be merged into itself. Stale responses cause a client-side merge, and rejected
submission bytes are retained in recovery before the request is cleared.
New UUIDs are saved before network use. Initial content is accepted before its
UUID is published in the file manifest. A crash between those operations can leave
unreferenced content, which is retained and harmless to the visible namespace.
It does not leave a file manifest pointing to absent content.
### Scanning, identities, and conflicts
Scans must complete or fail. A disappearing file during a scan is retried rather
than interpreted from a partial listing. Ignored paths, files above the configured
size limit, and files not yet materialized are handled explicitly so an unavailable
local copy does not masquerade as deletion.
Known online moves are applied in notification order, including directory-prefix
moves. Logical delete/create notifications distinguish replacement from editing.
Offline inference requires exactly one missing identity and one unknown path with
the same nonempty SHA-256 hash. Initial same-path adoption uses portable path
comparison. On normalization-insensitive filesystems, an accessible known NFC
spelling is retained when a scan reports its decomposed alias.
A local generation counter invalidates a prepared merge or push if a notification
arrives during scanning, remote fetches, or planning. The loop rescans rather than
applying the stale plan. This addresses, for example, moving a file and creating a
different file at its old path while remote content is being fetched.
`file-manifest.ts` first performs the requested per-UUID three-way merge, then allocates
valid paths. Server-selected occupants get priority; remaining ties use UUID
ordering. Conflict suffixes contain the UUID, including file/ancestor conflicts.
Invalid components are repaired, but long paths are not silently flattened into
UUID filenames. Comparing a candidate with each
occupied path individually avoids an allocation loop when the temporary occupied
set itself contains overlapping desired and actual paths.
### Content reconciliation
`content.ts` centralizes snapshot serialization and merging. It compares exact
bytes for unchanged-side fast paths, fetches immutable bases when needed, and
calls `reconcile` for valid UTF-8 content with a mergeable extension. Unknown
first-contact text uses an empty parent. Binary or otherwise unmergeable concurrent
content follows originating mtime, with the server winning ties.
Optional local cursor positions are transformed through text reconciliation and
passed to the adapter with the output. Text merge output uses the larger input
mtime. UTF-8 BOM is decoded as an actual character during merging, and native
line-ending conversion was removed. Existing bytes and CRLF are not rewritten
merely because another platform downloaded the file.
The old rolling checksum was replaced by asynchronous Web Crypto SHA-256. Cursor
tracking was updated to await hashes, find remote cursors by document UUID, and
resolve their display path through the local file manifest. Missing/unreadable local
files mark ephemeral cursor data as prior instead of breaking synchronization.
### Filesystem transaction and crash recovery
`FileOperations` now bridges the metadata store and the filesystem with a persisted
application journal. It does not assume these two stores share a transaction.
Each application gets a unique transaction directory under
`.vault-link-sync/transactions`. Each affected document has a source staging path,
an output path, expected/replacement snapshots, source/destination names, and phase:
`planned`, `staged`, `prepared`, `installing`, or `installed`.
The application sequence and reasons are:
| Action | Why it is needed |
| ------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Persist the complete plan before visible mutations | Restart must know what it was trying to do. The plan includes mergeable extensions so recovery is independent of server availability. |
| Move all affected sources into staging before installing any destination | Rename swaps/cycles and paths becoming ancestor directories cannot overwrite another source. Content replacement and deletion use the same retained-source machinery. |
| Detect an existing staging slot on recovery | It proves that the source move happened even if the subsequent metadata save failed. |
| Reconcile a staged source that changed after planning | Preserve observed external edits made between the expected snapshot and staging. |
| Persist the prepared result before creating its output file | Restart must not merge an already merged result a second time. |
| Persist `installing` before the atomic rename | If its origin is subsequently absent, recovery can recognize that installation completed even if its acknowledgement/save was lost. |
| Require exclusive creation and no-replace renames | A concurrent destination must cause a retry/conflict, not be silently overwritten. |
| Stage an unexpected file occupant as another UUID with a conflict path | Preserve an unrelated file that appeared at the destination or blocked an ancestor. |
| Preserve a nonempty blocking directory and rename the incoming document | Unmanaged descendants must not be recursively deleted to make space. |
| Prune only empty implicit directories | Directories are not synchronized objects; files must not be erased by cleanup. |
| Remove staged/output artifacts, then commit the planned next state | Completed applications do not become permanent byte/state archives; interrupted cleanup is retried while the active journal still exists. |
An external deletion observed before staging prevents automatic resurrection.
Recovery artifact creation is idempotent: an existing artifact is checked for the
expected bytes instead of overwritten.
Rejected request snapshots are stored separately under
`.vault-link-sync/requests`, with request IDs and identifying JSON. No recovery
artifact or orphan-content garbage collector was added. Ordinary transaction
artifacts are removed on completion; rejected-request snapshots remain explicit
recovery material.
### Lifecycle, settings, transport, and compatibility
`SyncClient.create` requires strong adapters and loads persisted state. Destruction
stops work and flushes local changes. Settings and engine saves serialize complete
replacements through the same persistence store so one cannot overwrite newer
data from the other. The client does not coordinate separate processes accessing
the same local vault.
`reset()` restarts transports/tracking; it no longer clears pending requests or
application journals. Changing a live vault requires a separate state store and
client. Settings saves are assumed to succeed; settings are saved before becoming
visible to listeners, and getters copy the settings and ignore-pattern array to
prevent accidental in-place mutation.
Public notification methods enqueue work instead of waiting for convergence.
Call `waitUntilFinished()` to await the active attempt. Sync status now checks the
requested path's manifest, content hash, exclusions, pending request, and active
application state. Remaining-operation reporting is a busy indicator rather than
the old concurrent queue length. Successful local submissions and remote filesystem
applications create history entries; the history UI is not a durable audit log.
The service layer now exposes typed vault-snapshot/event/file-manifest/content operations.
Durable retry ownership moved out of its old `retryForever` request builders and
into the loop. Requests have a 30-second abort timeout; authentication errors and
permanent 4xx responses are distinguished from transient failures. The proper
`Device-Id` header is used. A rejected initial ping is no longer cached forever,
so startup while offline can recover on retry.
The existing optional telemetry wiring was retained. Its build-injected version
constant now has a development fallback so importing source outside webpack does
not throw. No new telemetry service was introduced.
The unused `syncConcurrency`, `diffCacheSizeMB`, and `minimumSaveIntervalMs`
settings and the old content cache/queue scaffolding were removed.
`nativeLineEndings` remains accepted by the creation type but no longer transforms
content. These compatibility surfaces should be reviewed when adapters/UI are
updated, rather than assumed to retain their former effects.
## Required adapter contract and remaining integration
No production Obsidian or CLI adapter was upgraded in this work. Their old
interfaces are incompatible and will require implementation changes before they
can use the v4 client.
| Adapter responsibility | Required guarantee |
| ---------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Listing and reading | Complete enumeration and coherent bytes/mtime snapshots; absence must be distinguishable from failure. Read races must reject instead of returning partial snapshots. |
| Path access | Stay inside the vault; reject symlinks, ancestor symlink traversal, hardlinks, and special files. Staging and visible files must share a filesystem. |
| `write` | Atomically create a complete file exclusively. A crash leaves the path absent or complete, never partially written. Preserve supplied mtime and supported cursor metadata. |
| `rename` | Atomic same-filesystem move without replacing an existing destination. Flush source data and both affected directories before resolving. |
| Directory operations | Durable recursive creation, including ancestors. Pruning uses only directory removal and never unlinks descendant files. |
| Metadata persistence | Atomically replace the complete state and flush data/directory entries before resolving. A failed save may leave either complete value, which must be reloadable. |
| Notifications | Report logical external operations; suppress self-generated create/delete/move notifications. Treat an editor's atomic save as a content update rather than a deliberate document delete/recreate. |
| State placement | Keep metadata outside the user-file namespace or in the reserved internal directory. |
Known boundaries and deferred work:
- No automatic migration of existing server databases or v2/v3 client stores.
- No guarantee for writes that occur and disappear between observations, or
perfect identity reconstruction while notifications are unavailable. Logical
notification hints themselves are not an external filesystem operation log.
- No multi-file atomic visibility to arbitrary external programs: staging is
recoverable, but an external reader can see a transaction partway through.
- No hardware power-cut testing or production-adapter durability certification.
- No bounded history/recovery storage, event pagination, large-vault performance
optimization, or guarantee against starvation under continuous competing writes.
- No protection against an administrator independently deleting server history,
pending requests, or recovery state. An ahead-of-history cursor fails rather
than silently accepting a replaced/truncated server database.
- No syncing of empty directories or special filesystem objects.
- No new migration of the history viewer, plugin, CLI, repository-wide architecture
docs, or legacy testing setup. They may still describe the earlier protocol.
The next integration work is to implement and verify real adapters, update the
excluded callers to v4 and the new notification semantics, and port the behavior
checks into maintained tests. Fresh-state rollout must preserve old data and use
compatible server/client versions together.
## Validation performed
These results were obtained at the end of the implementation task, before the
subsequent edits noted above. They are not a claim that every current
working-tree change or every legacy test passes.
| Check | Result and scope |
| --------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------- |
| `cargo build --manifest-path sync-server/Cargo.toml --quiet` | Passed for the implemented server. |
| `cargo test --manifest-path sync-server/Cargo.toml export_bindings --quiet` | Passed 15 binding-export tests; used to generate the Rust wire types. This is not a behavioral server-suite result. |
| Strict TypeScript check of the production client entrypoint | Passed. Command below. |
| `npm run build -w sync-client` from `frontend` | Passed for both browser and Node production bundles. |
| `git diff --check` | Passed for the implementation diff. |
| Live server CAS/order checks | Passed with an isolated local server and fresh temporary databases. |
| Client reconciliation and interruption checks | Passed against that server using a memory filesystem/persistence harness with injected failures. |
The TypeScript command was:
```sh
node frontend/node_modules/typescript/bin/tsc --noEmit --strict \
--target ESNext --module ESNext --moduleResolution bundler \
--allowSyntheticDefaultImports --skipLibCheck --types node \
--typeRoots frontend/node_modules/@types frontend/sync-client/src/index.ts
```
Behavior exercised:
| Scenario | Observed result |
| --------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------- |
| 24 concurrent content creations, then 12 competing file manifest CAS submissions | Exactly one file manifest winner; committed event IDs remained contiguous. |
| WebSocket origin delivery and reconnect replay | Events arrived in increasing order, including events from the connecting device. |
| Duplicate JSON UUID keys and invalid portable file manifests | Rejected without creating events. Cases included duplicate/alias paths, ancestor conflicts, traversal, reserved names, and directory-spelling conflicts. |
| Content and file manifest receipt retry after a server restart and later heads | Returned the original acknowledgement without an additional event. |
| Creation and download; online rename sequence and remote swap | Identities and bytes converged correctly. |
| Concurrent text changes; first-sync same-path text | Both sides' text changes were retained in the checked examples. |
| Concurrent same-path creations | Both UUIDs survived with distinct paths. |
| Lost content acknowledgement followed by client restart | The original request was recovered, with only one accepted content event. |
| Lost file manifest acknowledgement followed by client restart | The original file manifest outcome was recovered without a duplicate mutation. |
| 22 injected filesystem interruption positions during staged swaps | Recovery produced the expected files without an unfinished application journal. |
| 18 injected metadata-save interruption positions during content replacement | Recovery installed the result without merging it a second time. |
| Unexpected destination file and blocking ancestor file | Incoming and unexpected content were both preserved. |
| Binary conflict and remote delete versus unsent local edit | Larger originating mtime won; downloaded mtime was retained; deleted edited bytes remained in recovery. |
| Ignored remote file, then re-enabled syncing | Membership was preserved and the file downloaded once eligible. |
| Save persisted and then reported failure | Reloading durable state allowed retry without restarting the process or duplicating a content mutation. |
| Initial ping offline, then network recovery | The next attempt progressed rather than reusing a permanently rejected ping promise. |
| Move and new file at the old path during a remote fetch | The moved document received the merge and the independent new file remained separate. |
| Notified delete/recreate with identical bytes at the same path | A new UUID was allocated. |
| UTF-8 BOM and CRLF during text merging | Both were preserved in the checked merge. |
The temporary harnesses were `/tmp/vault-link-engine-check.ts` and
`/tmp/vault-link-server-check.mjs`. They were deliberately not added to the
repository's testing setup. They used a temporary local server on port 18749;
that server was stopped after validation. Their absolute imports and temporary
configuration are development artifacts, not a portable test runner. Additional
server-restart receipt checks were performed during the session.
The 40 interruption positions are simulated failure points in adapters with the
required atomic semantics. They do not establish durability of real filesystem
adapters, all operating systems, all external-write schedules, or physical storage.
In the delivered rewrite, the old `put_file_content/tests.rs` file was not ported,
and the rewritten handler no longer included its old test module. Subsequent
edits to those files during this handover are separate work. Removed old helper
modules also contained
legacy unit tests. Client fixtures still target the earlier interfaces. The
webpack loader now checks files reachable from production entrypoints so those
unmigrated fixtures do not block building the library. Full legacy-suite and
repository-wide lint success were not claimed.
## Complete file inventory
Every implementation file added, changed, or removed relative to the baseline is
listed below, including regenerated type files and incidental cleanup. Deletions
are listed separately from additions even where Git's similarity detection calls
them renames. Generated binding formatting changes have no additional runtime
meaning unless stated.
This inventory accounts for **85 implementation paths**. It excludes the preexisting `CLAUDE.md` deletion and includes the old/new sides of Git-detected renames separately.
### Server
| File | Change | What changed and why |
| -------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| [sync-server/Cargo.toml](sync-server/Cargo.toml) | Changed | Declare `sha2` and `unicode-normalization`, remove `sanitize-filename`, and enable ts-rs `no-serde-warnings`. The new mutation code needs request fingerprints and portable-name normalization; server-side filename repair was removed. The preexisting push code already referred to SHA-256; the direct dependency is now declared. |
| [sync-server/Cargo.lock](sync-server/Cargo.lock) | Changed | Record the matching dependency graph: direct SHA-256/normalization dependencies and removal of the filename-sanitizer package. Keeps builds reproducible for the changed file manifest. |
| [sync-server/README.md](sync-server/README.md) | Changed | Replace v3 push/reset instructions with v4 requests, routes, portable-path rules, ordering, recovery, and fresh-state requirements. Documents the actual breaking protocol without suggesting destructive reset commands. |
| [sync-server/src/consts.rs](sync-server/src/consts.rs) | Changed | Raise the supported API from 3 to 4 so incompatible clients fail the version check. |
| [sync-server/src/app_state/database.rs](sync-server/src/app_state/database.rs) | Changed | Wire the consolidated read/write modules, use SQLx `begin_with("BEGIN IMMEDIATE")`, and add explicit SQLite and directory-flush durability settings. Keep commit ordering and persistence in the database boundary. |
| [sync-server/src/app_state/database/migrations/20241207143519_bootstrap.sql](sync-server/src/app_state/database/migrations/20241207143519_bootstrap.sql) | Changed | Replace the bootstrap schema with events, content-only versions, file manifests, and complete request receipts. Separate membership from bytes and atomically order both. Requires fresh databases; later local edits to this file are outside the earlier validation. |
| [sync-server/src/app_state/database/models.rs](sync-server/src/app_state/database/models.rs) | Changed | Remove path/deletion from content models; add originating mtime, SQLx row decoding, file manifest, event, batch, and vault snapshot types. Flatten wire metadata and use JS-number-compatible exported IDs to express the v4 protocol. |
| [sync-server/src/app_state/database/mutations.rs](sync-server/src/app_state/database/mutations.rs) | Added | Add both CAS transactions, request fingerprint/receipt handling, validation, event allocation, immutable insertions, and post-commit wakeups. Ensures state, order, and retry outcomes commit together. |
| [sync-server/src/app_state/database/queries.rs](sync-server/src/app_state/database/queries.rs) | Added | Add/consolidate current and immutable content reads, current file manifest, consistent bootstrap vault snapshot, and full ordered event replay. Readers no longer infer membership or event history from latest document rows. |
| `sync-server/src/app_state/database/get_document_version.rs` | Removed | Remove the separate old-schema query; immutable content reads are retained in `queries.rs` with the new row shape. |
| `sync-server/src/app_state/database/get_latest_document.rs` | Removed | Remove the separate old-schema query; current-head reads are retained in `queries.rs` and can share the CAS transaction. |
| `sync-server/src/app_state/database/get_latest_document_by_path.rs` | Removed | Remove path-based document lookup because content versions no longer own names. |
| `sync-server/src/app_state/database/get_latest_documents.rs` | Removed | Remove the latest-document listing helper; file manifest membership and a consistent `/vault-snapshot` now define the visible vault. |
| `sync-server/src/app_state/database/get_latest_documents_since.rs` | Removed | Remove head-only catch-up; replay every durable event so intervening namespace changes are not omitted. |
| `sync-server/src/app_state/database/get_max_update_id_in_vault.rs` | Removed | Remove document-table maximum-ID allocation; the common event table allocates IDs for both mutation kinds. |
| `sync-server/src/app_state/database/get_push_acknowledgement.rs` | Removed | Remove the content-specific acknowledgement lookup; shared request receipts now return the original response for either CAS kind. |
| `sync-server/src/app_state/database/insert_document_version.rs` | Removed | Remove the old coupled content/path insertion and direct broadcast. `mutations.rs` now commits state/event/receipt and publishes only a wake hint. |
| [sync-server/src/app_state/websocket/broadcasts.rs](sync-server/src/app_state/websocket/broadcasts.rs) | Changed | Replace wire-event broadcasts with `VaultUpdate` and ephemeral cursor notifications; make channel capacity at least one. Database replay carries correctness when receivers lag or no subscriber exists. |
| [sync-server/src/app_state/websocket/models.rs](sync-server/src/app_state/websocket/models.rs) | Changed | Replace `VaultUpdate` with `VaultEvents(EventBatch)` and remove the origin-filtering wrapper. Durable event batches can represent both mutations without holes for the originating device. |
| [sync-server/src/app_state/websocket/utils.rs](sync-server/src/app_state/websocket/utils.rs) | Changed | Remove `get_unseen_documents`; retain handshake authentication and message sending. Catch-up belongs to the durable event query instead of latest-head selection. |
| [sync-server/src/app_state/cursors.rs](sync-server/src/app_state/cursors.rs) | Changed | Send cursor expiry/update broadcasts through the new ephemeral notification type. Preserve presence behavior independently of persistent event sequencing. |
| [sync-server/src/server.rs](sync-server/src/server.rs) | Changed | Expose request/response modules within the crate for database mutations, add file-manifest/vault-snapshot/events routes, and remove the collection document-list route. Route ownership now reflects the two independent CAS resources. |
| [sync-server/src/server/endpoints.rs](sync-server/src/server/endpoints.rs) | Changed | Register the new file manifest endpoint module and remove the old latest-document-list module so the compiled route set matches v4. |
| `sync-server/src/server/endpoints/get_file_manifest.rs`, `put_file_manifest.rs`, `vault_snapshot.rs`, and `events.rs` | Added | Implement file manifest read/CAS, a consistent vault snapshot, and event replay with cursor checks. Supplies the namespace and recovery endpoints. |
| [sync-server/src/server/endpoints/put_file_content.rs](sync-server/src/server/endpoints/put_file_content.rs) | Changed | Reduce the handler to extraction/auth-context forwarding and content CAS delegation. Remove path/delete decisions and the old v3 test-module inclusion; tests need porting separately. |
| `sync-server/src/server/endpoints/fetch_latest_documents.rs` | Removed | Remove the old collection endpoint; it cannot express file manifest membership plus a consistent event watermark. |
| [sync-server/src/server/endpoints/websocket.rs](sync-server/src/server/endpoints/websocket.rs) | Changed | Use one ordered sender draining database history, replay from the handshake cursor, poll every two seconds, tolerate lag, and process ephemeral cursors in the same connection loop. Notification timing no longer determines event order. |
| [sync-server/src/server/requests.rs](sync-server/src/server/requests.rs) | Changed | Make pushes content-only, add `PushFileManifest`, deny unknown request fields, and reject duplicate parsed UUID keys. Prevent silent interpretation of legacy payloads or ambiguous maps. |
| [sync-server/src/server/responses.rs](sync-server/src/server/responses.rs) | Changed | Add file manifest `Accepted`/`StaleBase` responses and deserialize support for stored acknowledgements. Enables exact receipt replay for both mutation kinds. |
| [sync-server/src/utils/portable_path.rs](sync-server/src/utils/portable_path.rs) | Added | Add namespace-wide portable-path validation and shared comparison conventions. Reject aliases, reserved names, duplicate destinations, and file/ancestor conflicts before canonical acceptance. |
| [sync-server/src/utils.rs](sync-server/src/utils.rs) | Changed | Register portable validation and remove old sanitizer/path-allocation module exports. Keep naming policy on the client. |
| `sync-server/src/utils/dedup_paths.rs` | Removed | Remove the old path-deduplication helper and its embedded legacy tests. Canonical conflicts now require a valid client-submitted file manifest, not a server-generated alternate view. |
| `sync-server/src/utils/find_first_available_path.rs` | Removed | Remove server-side available-name allocation so accepted state equals the submitted file manifest. |
| `sync-server/src/utils/sanitize_path.rs` | Removed | Remove server-side path rewriting and its embedded tests. Invalid paths are rejected; client repair is explicit before CAS. |
| [sync-server/src/errors.rs](sync-server/src/errors.rs) | Changed | Remove an unused `log::error` import. Compile cleanup only; error semantics were not changed here. |
### Client core and build
| File | Change | What changed and why |
| -------------------------------------------------------------------------------------------------------------------------------------- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| [frontend/sync-client/README.md](frontend/sync-client/README.md) | Added | Add state/merge policy, notification semantics, recovery layout, adapter requirements, and compatibility boundaries. Gives integrators the contract needed to make the core guarantees meaningful. |
| [frontend/sync-client/src/consts.ts](frontend/sync-client/src/consts.ts) | Changed | Raise the supported API from 2 to 4 so the rewritten client agrees with the server and rejects incompatible versions. |
| [frontend/sync-client/src/index.ts](frontend/sync-client/src/index.ts) | Changed | Export `FileSnapshot` and new file-manifest/event/request types alongside the retained client surface. Consumers can implement the new adapters and use the v4 protocol types. |
| [frontend/sync-client/src/persistence/database.ts](frontend/sync-client/src/persistence/database.ts) | Changed | Replace mutable per-file scheduling records with v4 engine state, pending snapshots/responses, deferred remote heads, and application journals. Save before publishing memory state, reload uncertain outcomes, bind state to a vault, and retain cursor-compatible read-only views. |
| [frontend/sync-client/src/persistence/persistence.ts](frontend/sync-client/src/persistence/persistence.ts) | Changed | Require an atomic durable complete-state save contract. Recovery needs to know that a successful save survives interruption. |
| [frontend/sync-client/src/persistence/settings.ts](frontend/sync-client/src/persistence/settings.ts) | Changed | Persist settings before updating memory/listeners and return copies including the ignore-pattern array. Settings saves are assumed to succeed; caller mutation must not silently change the live configuration. |
| [frontend/sync-client/src/file-operations/filesystem-operations.ts](frontend/sync-client/src/file-operations/filesystem-operations.ts) | Changed | Replace the weaker read/overwrite/text-update interface with coherent snapshots, explicit stat, exclusive durable writes, no-replace durable moves, and directory-only pruning. State the no-link and flush requirements needed for recovery. |
| [frontend/sync-client/src/file-operations/file-operations.ts](frontend/sync-client/src/file-operations/file-operations.ts) | Changed | Replace direct operations and line-ending conversion with journaled staged application, recovery phases, persisted prepared output, race reconciliation, unexpected-occupant preservation, completed-artifact cleanup, and empty-directory pruning. Makes multi-file namespace changes recoverable without permanent per-application archives. |
| `frontend/sync-client/src/file-operations/safe-filesystem-operations.ts` | Removed | Remove the in-memory filesystem safety/locking wrapper. Its guarantees cannot substitute for durable journaling and the stronger adapter contract. |
| [frontend/sync-client/src/services/protocol.ts](frontend/sync-client/src/services/protocol.ts) | Added | Add a single import surface over Rust-generated v4 types plus `FileManifestEntries`. Avoid maintaining a second handwritten protocol definition. |
| [frontend/sync-client/src/services/server-config.ts](frontend/sync-client/src/services/server-config.ts) | Changed | Clear a rejected cached ping promise. An initial offline attempt must be able to fetch configuration again. |
| [frontend/sync-client/src/services/sync-service.ts](frontend/sync-client/src/services/sync-service.ts) | Changed | Replace legacy multipart/create/text/binary/delete methods and retry builders with typed v4 CAS, vault-snapshot, and replay requests; use `Device-Id`, request timeouts, and permanent/auth/transient error distinctions. The persisted actor owns retries. |
| [frontend/sync-client/src/services/websocket-manager.ts](frontend/sync-client/src/services/websocket-manager.ts) | Changed | Parse `vaultEvents` as `EventBatch` and notify the actor. Keep existing transport/cursor support while switching the payload shape. |
| [frontend/sync-client/src/sync-client.ts](frontend/sync-client/src/sync-client.ts) | Changed | Rewire the public client around one durable actor; enforce adapter contracts, serialize persistence, preserve state on reset, restrict vault changes, and keep cursor/history/telemetry integration. Notifications enqueue work and explicit waiting reports completion/failure. |
| [frontend/sync-client/src/sync-operations/content.ts](frontend/sync-client/src/sync-operations/content.ts) | Added | Add stored snapshot conversion and client-only three-way content merging, originating-mtime binary decisions, cursor transformation, BOM handling, and exact-byte/unchanged-side fast paths. Centralizes the content policy used by normal sync and recovery. |
| [frontend/sync-client/src/sync-operations/file-manifest.ts](frontend/sync-client/src/sync-operations/file-manifest.ts) | Added | Add per-UUID three-way merge, absence/deletion handling, portable validation, and deterministic cross-ID conflict allocation, including ancestor and oversized-name cases. Produces a valid namespace before CAS and local application. |
| [frontend/sync-client/src/sync-operations/syncer.ts](frontend/sync-client/src/sync-operations/syncer.ts) | Changed | Replace concurrent per-document scheduling with the recover/bootstrap/retry/scan/replay/merge/push loop, durable requests, complete scans, identity hints, generation guards, deferred downloads, periodic wakeups, and accepted-snapshot base tracking. Makes namespace and content decisions share one consistent state machine. |
| `frontend/sync-client/src/sync-operations/unrestricted-syncer.ts` | Removed | Remove the old second scheduling/operation layer, server-merge response assumptions, and cache-driven update flow. Keeping it beside the new actor would preserve conflicting ownership and v2 behavior. |
| [frontend/sync-client/src/sync-operations/cursor-tracker.ts](frontend/sync-client/src/sync-operations/cursor-tracker.ts) | Changed | Await SHA-256 comparisons, match remote cursors by UUID, display current local paths, and treat missing reads as prior cursor data. Renames must not attach cursors to another document at the old path. |
| `frontend/sync-client/src/utils/find-matching-file.ts` | Removed | Remove the old standalone matching helper. Conservative unique-hash identity inference now lives in the actor with complete scan context. |
| [frontend/sync-client/src/utils/hash.ts](frontend/sync-client/src/utils/hash.ts) | Changed | Replace the 32-bit rolling checksum with async Web Crypto SHA-256 and its known empty digest. Reduce false unchanged/rename decisions from checksum collisions. |
| [frontend/sync-client/src/utils/set-up-telemetry.ts](frontend/sync-client/src/utils/set-up-telemetry.ts) | Changed | Declare the webpack-injected version constant and guard it with a development fallback. Direct source imports used by consumers/validation must not throw before synchronization starts. |
| [frontend/sync-client/webpack.config.js](frontend/sync-client/webpack.config.js) | Changed | Set ts-loader `onlyCompileBundledFiles: true`. Production browser/Node builds typecheck their dependency graph without pulling in unmigrated legacy fixtures; this is not a claim those fixtures pass. |
### Client generated bindings
| File | Change | What changed and why |
| ---------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| [frontend/sync-client/src/services/types/ClientCursors.ts](frontend/sync-client/src/services/types/ClientCursors.ts) | Changed | Regenerate the existing cursor payload as a type alias with generated array syntax. Keeps generated output current; no intended wire change. |
| [frontend/sync-client/src/services/types/CursorPositionFromClient.ts](frontend/sync-client/src/services/types/CursorPositionFromClient.ts) | Changed | Regenerate the existing cursor payload. Type-alias/array formatting only; the ephemeral wire shape remains the same. |
| [frontend/sync-client/src/services/types/CursorPositionFromServer.ts](frontend/sync-client/src/services/types/CursorPositionFromServer.ts) | Changed | Regenerate the existing cursor response. Formatting/type-alias change only. |
| [frontend/sync-client/src/services/types/CursorSpan.ts](frontend/sync-client/src/services/types/CursorSpan.ts) | Changed | Regenerate the start/end span definition. Formatting/type-alias change only. |
| [frontend/sync-client/src/services/types/DocumentWithCursors.ts](frontend/sync-client/src/services/types/DocumentWithCursors.ts) | Changed | Regenerate the existing document/cursor descriptor from Rust. Preserve its existing wire field names while aligning generated representation. |
| [frontend/sync-client/src/services/types/WebSocketHandshake.ts](frontend/sync-client/src/services/types/WebSocketHandshake.ts) | Changed | Regenerate the token/device/cursor handshake. The replay cursor field is retained; representation changes from interface to type alias. |
| [frontend/sync-client/src/services/types/PingResponse.ts](frontend/sync-client/src/services/types/PingResponse.ts) | Changed | Regenerate current server configuration metadata, including supported API version. No additional ping endpoint behavior is introduced by the generated-file change. |
| [frontend/sync-client/src/services/types/SerializedError.ts](frontend/sync-client/src/services/types/SerializedError.ts) | Changed | Regenerate the existing error shape. Formatting/type-alias/array syntax changes only. |
| [frontend/sync-client/src/services/types/DocumentUpdateResponse.ts](frontend/sync-client/src/services/types/DocumentUpdateResponse.ts) | Changed | Replace obsolete client `FastForwardUpdate`/`MergingUpdate` variants with `Accepted`/`StaleBase`. Align the client with CAS outcomes already introduced server-side and retained in v4. |
| [frontend/sync-client/src/services/types/DocumentVersion.ts](frontend/sync-client/src/services/types/DocumentVersion.ts) | Changed | Regenerate content bytes plus flattened v4 metadata: originating mtime and content size, without path/deletion. Matches the new content-only read/stale response. |
| [frontend/sync-client/src/services/types/DocumentVersionWithoutContent.ts](frontend/sync-client/src/services/types/DocumentVersionWithoutContent.ts) | Changed | Regenerate v4 content metadata with mtime and without namespace fields. Used by events, acknowledgements, and bootstrap heads. |
| [frontend/sync-client/src/services/types/PushContent.ts](frontend/sync-client/src/services/types/PushContent.ts) | Added | Add generated `Snapshot`/`Diff` transport payloads. No delete operation is allowed in the content resource. |
| [frontend/sync-client/src/services/types/PutFileContent.ts](frontend/sync-client/src/services/types/PutFileContent.ts) | Added | Add generated request ID, nullable content parent, mtime, and content payload. Makes exact content retry data typed. |
| [frontend/sync-client/src/services/types/PushFileManifest.ts](frontend/sync-client/src/services/types/PushFileManifest.ts) | Added | Add generated request ID, file manifest parent, and UUID-to-path map. Makes exact namespace retry data typed. |
| [frontend/sync-client/src/services/types/FileManifest.ts](frontend/sync-client/src/services/types/FileManifest.ts) | Added | Add generated file manifest ID and entries map. Namespace versions are independent of document versions. |
| [frontend/sync-client/src/services/types/FileManifestUpdateResponse.ts](frontend/sync-client/src/services/types/FileManifestUpdateResponse.ts) | Added | Add generated file manifest acceptance/stale variants. The client needs the canonical map when rebasing. |
| [frontend/sync-client/src/services/types/VaultEvent.ts](frontend/sync-client/src/services/types/VaultEvent.ts) | Added | Add generated content-or-file-manifest event union. Both mutation kinds participate in one sequence. |
| [frontend/sync-client/src/services/types/EventRecord.ts](frontend/sync-client/src/services/types/EventRecord.ts) | Added | Add generated event/request IDs and flattened event payload. Supports ordered replay and tracing of accepted operations. |
| [frontend/sync-client/src/services/types/EventBatch.ts](frontend/sync-client/src/services/types/EventBatch.ts) | Added | Add generated events and head watermark. Lets the client check complete, contiguous replay. |
| [frontend/sync-client/src/services/types/VaultSnapshot.ts](frontend/sync-client/src/services/types/VaultSnapshot.ts) | Added | Add the generated consistent file-manifest/content-head/watermark vault snapshot. Makes initial bootstrap independent of lossy notifications. |
| [frontend/sync-client/src/services/types/WebSocketServerMessage.ts](frontend/sync-client/src/services/types/WebSocketServerMessage.ts) | Changed | Replace `vaultUpdate` with `vaultEvents` while retaining cursor positions. Matches the ordered server stream. |
| `frontend/sync-client/src/services/types/CreateDocumentVersion.ts` | Removed | Remove the old optional-server-ID multipart/create request type. V4 creation is client-UUID content CAS followed by file manifest CAS. |
| `frontend/sync-client/src/services/types/DeleteDocumentVersion.ts` | Removed | Remove the old content deletion payload type. Deletion is absence from the submitted file manifest. |
| `frontend/sync-client/src/services/types/UpdateDocumentVersion.ts` | Removed | Remove the legacy content/path update type. V4 uses `PutFileContent` and independent namespace CAS. |
| `frontend/sync-client/src/services/types/UpdateTextDocumentVersion.ts` | Removed | Remove the old text-specific path/update request. Text merges locally and uses the same content CAS transport. |
| `frontend/sync-client/src/services/types/FetchLatestDocumentsResponse.ts` | Removed | Remove the old document-list response type. Membership/bootstrap/replay use file manifests, snapshots, and event batches. |
| `frontend/sync-client/src/services/types/WebSocketVaultUpdate.ts` | Removed | Remove the latest-documents/initial-sync update type. Ordered batches and a separate bootstrap vault snapshot replace its two roles. |
## Maintainer reading order
For the protocol, read [the server README](sync-server/README.md), then server
`requests.rs`, `models.rs`, `mutations.rs`, `queries.rs`, and the WebSocket handler.
For the client, read [the client README](frontend/sync-client/README.md), then
`persistence/database.ts`, `sync-operations/file-manifest.ts`, `sync-operations/syncer.ts`,
and `file-operations/file-operations.ts`. The adapter interface is part of the
correctness argument and should be read before implementing an integration.
Keep the distinction between a current file manifest version and the event watermark,
and between an accepted sent snapshot and today's disk contents, explicit in any
future change. Those distinctions are central to avoiding skipped namespace
changes and duplicated merges after a lost acknowledgement.

523
docs/architecture/sync-algorithm.md
View file

@ -1,438 +1,85 @@
# Sync Algorithm
VaultLink uses operational transformation (OT) to handle concurrent edits and maintain consistency across clients.
## Operational Transformation
Operational transformation is a technique for managing concurrent edits to the same document. It transforms operations (edits) so they can be applied in different orders while preserving user intent.
### Why OT?
Traditional conflict resolution approaches:
- **Last write wins**: Loses data, frustrating for users
- **Manual merging**: Interrupts workflow, requires user intervention
- **Version branching**: Complex, not suitable for real-time sync
Operational transformation:
- **Automatic**: No user intervention required
- **Preserves all edits**: No data loss
- **Real-time**: Changes appear immediately
- **Intuitive**: Behaviour matches user expectations
## The reconcile-text Library
VaultLink uses the [`reconcile-text`](https://crates.io/crates/reconcile-text) Rust library for operational transformation on text documents.
### Why reconcile-text over CRDTs?
VaultLink faces a **differential synchronisation** challenge: users edit Obsidian vaults with various editors (Obsidian desktop, Obsidian mobile, Vim, VS Code, or any text editor), often while offline. This means we only observe the **final state** of each document after editing, not the individual keystrokes or operations that produced it.
**The fundamental problem**:
- **CRDTs and traditional OT** require capturing every individual operation (each character insertion, deletion, cursor movement)
- **VaultLink's reality**: Users edit files with arbitrary tools, sync happens after the fact
- **What we know**: Parent version and two modified versions
- **What we don't know**: The sequence of operations that created those modifications
**Why reconcile-text wins for this use case**:
1. **Works with end states only**: reconcile-text performs conflict-free 3-way merging given just parent, left, and right versions—no operation history needed
2. **Editor-agnostic**: Users can edit with any tool without requiring VaultLink-specific plugins or operation tracking
3. **Offline-first**: Edits made while disconnected are merged cleanly when sync resumes, because we're diffing final states rather than replaying operations
4. **No conflict markers**: Unlike Git merge, produces clean merged output without `<<<<<<<` markers that interrupt note-taking flow
5. **Human text forgiveness**: For knowledge bases and documentation, a slightly imperfect merge (e.g., minor word order issues) is vastly preferable to manual conflict resolution
6. **Simpler infrastructure**: No need for complex operation capture, transformation logs, or tombstone management that CRDTs require
**The trade-off**:
CRDTs excel when you control the entire editing infrastructure and can capture every operation. reconcile-text excels when you're synchronising independently-edited files—exactly VaultLink's scenario. The merge quality depends on Myers' diff algorithm rather than operation history, which is the correct trade-off for differential sync.
For note-taking workflows where users value editor freedom and offline editing, this approach provides superior user experience compared to either CRDTs (which would require operation tracking) or Git-style merging (which requires manual conflict resolution).
[Learn more about reconcile-text →](https://schmelczer.dev/reconcile)
### How It Works
Given three versions (parent, left, right), reconcile-text produces a merged result.
**How reconcile-text works**:
1. **Tokenisation**: Split text into words (using `BuiltinTokenizer::Word`)
2. **Three-way diff**: Compare parent→left and parent→right changes
3. **Merge**: Combine non-conflicting changes, prefer content preservation for conflicts
4. **Result**: Merged text with both edits applied
**Example**:
```
Parent: "The quick brown fox"
User A: "The quick red fox" (changes "brown" → "red")
User B: "The very quick brown fox" (inserts "very ")
Merged: "The very quick red fox" (both changes applied)
```
**Merge conditions**: Only `.md` and `.txt` files with valid UTF-8 get merged. Binary files or other extensions use last-write-wins.
### Operation Types
The algorithm handles these operations:
- **Insert**: Add text at position
- **Delete**: Remove text from position
- **Retain**: Keep existing text unchanged
### Transformation Process
1. **Client A** makes edit and sends to server
2. **Client B** makes concurrent edit and sends to server
3. **Server** receives both edits
4. **Server** transforms operations to account for concurrent changes
5. **Server** applies merged result to database
6. **Server** sends transformed operations to both clients
7. **Clients** apply transformed operations locally
## Sync State Management
VaultLink maintains sync state to track which changes have been applied.
### Version Vectors
Each document has a version tracked by:
- **Server version**: Incremented on each change
- **Client cursors**: Track which version each client has seen
This enables:
- Efficient syncing (only send changes since last sync)
- Conflict detection (concurrent edits to same version)
- Ordering of operations
### Cursor Management
Clients maintain a cursor position:
```rust
struct Cursor {
vault_id: String,
client_id: String,
last_version: u64,
last_updated: DateTime,
}
```
On sync:
1. Client sends cursor (last seen version)
2. Server returns all changes since that version
3. Client applies changes and updates cursor
## Conflict Resolution Flow
### Scenario: Concurrent Edits
Two users edit the same paragraph simultaneously.
**Initial state**:
```
Version 10: "The quick brown fox jumps over the lazy dog."
```
**User A's edit** (version 11):
```
"The quick brown fox jumps over the very lazy dog."
```
_Inserts "very " at position 40_
**User B's edit** (also from version 10):
```
"The quick red fox jumps over the lazy dog."
```
_Replaces "brown" with "red" at position 10_
### Server Processing
1. **Receive User A's operation**:
- Base: version 10
- Operation: Insert("very ", position=40)
- Apply to database → version 11
2. **Receive User B's operation**:
- Base: version 10
- Operation: Replace("brown"→"red", position=10)
- **Conflict detected**: Base is version 10, but current is version 11
3. **Transform User B's operation**:
- Transform against User A's operation
- Adjust positions/content as needed
- Apply transformed operation → version 12
4. **Broadcast updates**:
- Send User A's operation to User B
- Send transformed User B's operation to User A
### Final Result
```
Version 12: "The quick red fox jumps over the very lazy dog."
```
Both edits are preserved in the final document.
## Edge Cases
### 1. Delete vs Insert Conflict
**Scenario**: User A deletes a paragraph while User B edits it.
**Resolution**:
- OT algorithm prioritizes preservation of content
- Insert operation is transformed to account for deletion
- Typically results in inserted content appearing nearby
**Example**:
```
Base: "Line 1\nLine 2\nLine 3"
User A: Delete Line 2 → "Line 1\nLine 3"
User B: Edit Line 2 → "Line 1\nLine 2 modified\nLine 3"
Result: "Line 1\nLine 2 modified\nLine 3"
```
(Insert takes precedence, preserving user content)
### 2. Overlapping Edits
**Scenario**: Two users edit overlapping regions.
**Resolution**:
- OT splits operations into non-overlapping segments
- Applies each segment independently
- Merges results
### 3. Delete vs Delete
**Scenario**: Two users delete overlapping text.
**Resolution**:
- Deletes are merged
- Final result has the union of deleted ranges removed
### 4. Network Partitions
**Scenario**: Client loses connection, makes edits offline, reconnects.
**Resolution**:
1. Client queues edits locally
2. On reconnect, sends all queued operations
3. Server applies OT against all operations that happened during partition
4. Client receives transformed operations and applies
## Performance Characteristics
### Time Complexity
- **Single operation**: O(1) for most operations
- **Transformation**: O(n) where n is operation size
- **Conflict resolution**: O(m × n) where m is number of concurrent operations
### Space Complexity
- **Version history**: Grows with number of changes
- **Cursors**: O(clients × vaults)
- **Active operations**: Minimal (processed in real-time)
### Optimisation
VaultLink optimises for:
- Small, frequent edits (typical typing patterns)
- Text documents (not binary files)
- Real-time processing (no batching delay)
## Limitations
### Binary and Non-Mergeable Files
Only **`.md`** and **`.txt`** files get automatic merging. Everything else uses last-write-wins.
**Binary detection**:
- Files with NUL bytes (`0x00`)
- Files failing UTF-8 validation
Even `.md` files are treated as binary if they fail UTF-8 checks.
**Last-write-wins behaviour**:
```
User A uploads image.png → Server version 1
User B uploads image.png → Server version 2 (A's upload lost)
```
**Workaround**: Avoid concurrent edits to non-text files. [See all limitations →](/guide/limitations)
### Large Documents
Very large documents (> 1MB) may have:
- Higher transformation costs
- Slower sync times
- Increased memory usage
**Workaround**: Split large documents or increase timeout settings.
### Complex Formatting
Markdown with complex structures may occasionally produce unexpected results:
- Nested lists
- Tables
- Code blocks
**Workaround**: Manual cleanup if needed, or minimize concurrent edits to complex structures.
## Consistency Guarantees
### Strong Consistency
VaultLink provides **strong eventual consistency**:
- All clients eventually converge to the same state
- Operations applied in causal order
- No data loss under normal operation
### Ordering Guarantees
- Operations from the same client are applied in order
- Concurrent operations may be applied in any order
- Final result is independent of operation order (commutative)
### Durability
- Operations are written to SQLite before acknowledgment
- SQLite ACID guarantees protect against data loss
- Clients retry failed uploads
## Comparison with Other Approaches
### Git-style Merging
| Aspect | Git Merge | VaultLink OT |
| -------------------------- | ------------ | ----------------------- |
| Real-time | No | Yes |
| Manual conflict resolution | Yes | No |
| Branching | Yes | No |
| Automatic merge | Limited | Always |
| Use case | Code changes | Collaborative documents |
### CRDTs (Conflict-free Replicated Data Types)
| Aspect | CRDTs | VaultLink (reconcile-text) |
| ----------------------------- | ------------------------------------ | ------------------------------------------------- |
| **Operation tracking** | Required (every keystroke) | Not required (end states only) |
| **Editor freedom** | Limited (must use CRDT-aware editor) | Unlimited (any text editor works) |
| **Offline editing** | Requires operation log | Works with file comparison |
| **Server required** | No | Yes |
| **Memory overhead** | Higher (tombstones, metadata) | Lower (versions only) |
| **Infrastructure complexity** | Higher | Lower |
| **Best for** | Controlled editing environments | Independent file editing (Obsidian, Vim, VS Code) |
**Key insight**: CRDTs are superior when you can capture every operation. reconcile-text is superior when users edit files independently with arbitrary tools—exactly VaultLink's scenario.
### Last Write Wins
| Aspect | LWW | VaultLink OT |
| --------------- | ---- | ------------ |
| Data loss | Yes | No |
| Simplicity | High | Medium |
| User experience | Poor | Excellent |
| Performance | Best | Good |
## Algorithm Details
### Transformation Rules
When transforming operation `A` against operation `B`:
1. **Insert vs Insert**:
- If positions equal: Order by client ID
- If different positions: Adjust positions
2. **Insert vs Delete**:
- If insert in deleted range: Shift insert position
- If insert after delete: Adjust position by deleted length
3. **Delete vs Delete**:
- If ranges overlap: Merge delete ranges
- If ranges disjoint: Adjust positions
4. **Retain vs Any**:
- Retain operations don't conflict
- Simply adjust positions
### Transformation Example
```rust
// Pseudo-code for transformation
fn transform(op_a: Operation, op_b: Operation) -> (Operation, Operation) {
match (op_a, op_b) {
(Insert(pos_a, text_a), Insert(pos_b, text_b)) => {
if pos_a < pos_b {
(op_a, Insert(pos_b + text_a.len(), text_b))
} else if pos_a > pos_b {
(Insert(pos_a + text_b.len(), text_a), op_b)
} else {
// Same position, use client ID to break tie
if client_id_a < client_id_b {
(op_a, Insert(pos_b + text_a.len(), text_b))
} else {
(Insert(pos_a + text_b.len(), text_a), op_b)
}
}
}
// ... other cases
}
}
```
## Best Practices
### For Smooth Collaboration
1. **Small edits**: Make small, focused changes for easier merging
2. **Coordinate major changes**: Discuss large refactors with team
3. **Monitor sync status**: Ensure changes are uploaded before signing off
4. **Test conflict resolution**: Verify behaviour matches expectations
### For Developers
1. **Text files preferred**: OT works best on text
2. **Limit file sizes**: Keep documents reasonably sized
3. **Binary files**: Use versioning or avoid concurrent edits
4. **Testing**: Test concurrent edit scenarios thoroughly
## Further Reading
- [reconcile-text library](https://crates.io/crates/reconcile-text)
- [Operational Transformation FAQ](https://en.wikipedia.org/wiki/Operational_transformation)
- [Data flow architecture →](/architecture/data-flow)
# Sync algorithm
The current API v5 server stores immutable content versions and a separately
versioned file manifest. Clients reconcile snapshots and apply them through a
durable filesystem journal. The server does not merge competing edits.
## Identity and ordering
Each document has a UUID independent of its path. A file manifest maps live UUIDs
to paths; removing a mapping deletes membership. Updating content never restores
membership. Directories are implicit.
Every accepted content or manifest update appends an event in the same SQLite
transaction. Event IDs order changes and serve as version IDs. A push names its
parent version and a unique request UUID. The server either accepts that exact
snapshot or returns current metadata as `StaleBase`. Retrying an accepted request
returns its original acknowledgement; changing its payload under the same request
UUID is rejected.
## Reconciliation
A new client reads a coherent vault snapshot, applies it durably, then replays
subsequent events. Replayed pages are folded to their final document heads and
manifest before changing local files. This avoids applying intermediate remote
edits that were already reverted.
For text, the client compares the common base, current local bytes and current
remote bytes using `reconcile-text`. Configured extensions and valid UTF-8 without
NUL bytes determine whether text merging is allowed. Diffs optimize transport;
they do not authorize the server to merge a stale base. Large diff operation sets
are sent as snapshots.
If only one side changed, that side is retained. Concurrent unmergeable changes
keep the server version while the journal archives displaced local bytes for
recovery. First contact with existing local text has no shared ancestor and uses
an empty parent. Concurrent edits may still require human review.
Paths use a separate three-way decision: keep a local change when the server path
is unchanged, otherwise prefer the server path if both changed. Portable-name
collisions receive deterministic conflict names. A manifest deletion is not
reversed merely because another client edited the document; displaced bytes are
retained locally.
## Filesystem application
Before moving files, the client persists a journal describing sources,
destinations and replacement snapshots. It stages sources before installing any
destination, so swaps and rename cycles do not overwrite each other. Prepared
content is persisted before installation so a crash does not repeat a merge.
External notifications retain logical document identities. Notifications received
during asynchronous filesystem reads or staging are incorporated before accepting
a source. An editor save to a staged document's old path remains part of that
document. Ignored and oversized physical occupants are protected during conflict
allocation. Adapters report atomic editor saves as content updates and suppress
notifications generated by the engine itself.
The filesystem and metadata store have separate durability boundaries. Recovery
replays the journal before scanning, including when network sync is disabled.
Adapters must provide complete scans, coherent snapshots, exclusive vault access,
no-replace renames and durable writes. Recovery artifacts are excluded from sync.
## Server restores and failures
Numeric version equality cannot establish identity after restoring a backup.
Successful HTTP responses include a checkpoint with an event-incarnation token.
The client persists and echoes it; the server rejects a checkpoint from a discarded
history before executing its request. Each new event gets a fresh token even if
its numeric ID or request UUID was used on the discarded branch.
After a mismatch, the client durably records recovery intent, retains pending
content, clears old history-dependent state and bootstraps against the restored
server. Clean files reconcile against the restored snapshot. Unsent edits with an
old base are preserved as separate local documents, with conflict names if needed.
Interrupted recovery saves are replayable. Backups must be restored with the server
stopped.
WebSocket messages wake HTTP replay; missed notifications are recovered by replay
and transport liveness checks. Authentication and permanent errors remain visible;
transient errors retry according to client settings. Observer callback failures
are isolated from the background sync loop.
API v5 clients and servers must be deployed together. V4 databases migrate without
removing history. The old Obsidian and CLI adapters still require the filesystem
and persistence contract required by the core engine.

31
docs/guide/limitations.md
View file

@ -8,14 +8,9 @@ VaultLink works well for most Obsidian vaults, but has some constraints you shou
Only **`.md`** and **`.txt`** files get automatic conflict-free merging.
Other file types (images, PDFs, etc.) use last-write-wins:
```
User A updates diagram.png → Server stores version 1
User B updates diagram.png → Server stores version 2 (overwrites A's changes)
```
**Workaround**: Avoid editing the same non-text file simultaneously.
For other file types, a one-sided change is retained. Concurrent unmergeable
changes keep the server's version and archive displaced local bytes for recovery.
Review recovery artifacts when both devices edit a binary file.
### Binary Detection
@ -24,7 +19,7 @@ Files are treated as binary if they:
- Contain NUL bytes (`0x00`)
- Fail UTF-8 validation
Binary files within `.md` or `.txt` extensions still get last-write-wins (no merge).
Binary files within `.md` or `.txt` extensions follow the same unmergeable-content policy.
## Performance Constraints
@ -61,11 +56,11 @@ Rough estimates (varies by vault size and activity):
- All versions stored indefinitely (no automatic cleanup)
- Each vault is a separate SQLite database
- Deleted files marked as deleted (not purged)
- Deletion removes file-manifest membership; content history remains readable
**Growth**: Version history grows with every change. A 10 MB vault with frequent edits might grow to 100+ MB over months.
**Cleanup**: Manual only (see [Advanced Configuration](/config/advanced#version-history-cleanup)).
Active journals, immutable merge bases and request receipts must not be independently deleted. There is no automatic safe history-pruning protocol.
### Implications
@ -92,7 +87,7 @@ Result: "The very quick red fox" ← Both changes preserved
- Simultaneous edits to the same sentence
- Large structural changes (moving sections around)
**Result**: Merged file might need manual cleanup in ~1-5% of concurrent edits.
Merged files may require manual review; there is no measured universal conflict-success rate.
## Scalability
@ -140,6 +135,14 @@ Not currently supported. Running multiple servers requires manual vault partitio
- Tokens configured in server config file
- No runtime user management
## Restored backups
Restore server databases while the server is stopped. API v5 clients detect a
changed history even when version numbers are reused. Clean files reconcile with
the restored server; unsent edits are preserved as separate local documents.
Clients and server must be upgraded together. This does not replace independent
backups or protect against arbitrary physical-media corruption.
## Known Edge Cases
### Simultaneous Deletes and Edits
@ -147,10 +150,10 @@ Not currently supported. Running multiple servers requires manual vault partitio
```
User A deletes note.md
User B edits note.md
Result: Edit wins (file recreated with B's content)
Result: Manifest deletion removes the file; unsent edited bytes are retained for recovery.
```
Operational transformation prioritises content preservation.
Content updates do not restore deleted membership. The client retains displaced bytes in its local recovery directory.
### Large File Uploads

224
frontend/deterministic-tests/README.md
View file

@ -1,118 +1,134 @@
# Deterministic Tests
# Sync test harnesses (API v4)
Scripted multi-client (with an in-memory filesystem) sync tests that run against a real server. Each test defines a sequence of file operations, sync/server controls, and assertions to exercise a specific conflict or edge case.
Complements the fuzz-based E2E tests (`test-client`): fuzz tests discover bugs through random operations; deterministic tests pin down exact reproduction sequences for known scenarios.
## How it works
Each test is a `TestDefinition`: a client count and an ordered list of steps. The test name is derived from the registry key (which matches the file name). The `TestRunner` spins up N `DeterministicAgent` instances (each wrapping a real `SyncClient` with an `InMemoryFileSystem`) pointed at a shared vault on the server, then executes steps one by one.
Tests that don't pause the server share a single server process (vault-name isolation). Tests that use `pause-server`/`resume-server` (SIGSTOP/SIGCONT) each get a dedicated server, since SIGSTOP freezes the entire process.
The runner executes two sequential phases: regular tests on the shared server, then pause-server tests on dedicated servers. Within each phase tests run in parallel up to a concurrency limit.
## Step types
Clients always start with syncing disabled.
**File operations** (per-client, fire-and-forget — sync is enqueued but not awaited):
- `create`, `update`, `rename`, `delete`
- `rename-next-write` — arm a deferred rename that fires the next time the given path is written. Lets a test race a user-rename against an in-flight remote create that's about to land at the same path.
**Sync control:**
- `sync` — wait for a specific client or all clients to finish pending operations
- `barrier` — retry until all clients converge to identical file state (60s timeout)
- `enable-sync` / `disable-sync` — simulate going online/offline
- `reset` — reset a client's tracked sync state (keeps disk files); equivalent to a forced re-handshake on next enable
- `sleep` — wall-clock pause; use sparingly, prefer `barrier` / `sync`
**WebSocket control** (per-client):
- `pause-websocket` / `resume-websocket` — buffer/release WebSocket messages for a specific client
**Server control:**
- `pause-server` / `resume-server` — SIGSTOP/SIGCONT the server process
- `resume-server-until-history-then-pause` — resume the server, wait until a specific client observes a matching history entry (`CREATE`/`UPDATE`/`DELETE` for a path), then re-pause. Used to land exactly one operation across the wire.
**Fault injection** (per-client):
- `drop-next-create-response` — arm a one-shot interceptor that lets the next `POST /documents` reach the server (commit happens) but throws `SyncResetError` before the client sees the response, simulating connection loss after server commit.
- `wait-for-dropped-create-response` — wait until the armed drop has fired.
**Assertions:**
- `assert-consistent` — all clients have identical files; optionally takes a custom `verify(state: AssertableState)` callback
## Running
Run the complete harness check from **any working directory**:
```sh
# Build server first
cd sync-server && cargo build --release && cd -
# Run all tests
cd frontend && npm run build -w sync-client && npm run test -w deterministic-tests
# Filter by name
npm run test -w deterministic-tests -- --filter=rename
# Control parallelism (default: number of CPU cores)
npm run test -w deterministic-tests -- -j 4
E2E_WORKERS=2 /path/to/vault-link/scripts/e2e.sh
```
## Adding a test
The `E2E_WORKERS` variable is the number of fuzz workers / simultaneous scripted tests.
The runner builds the server and the three in-scope frontend packages, then runs
the harness self-tests, recovery regressions, protocol/real-filesystem tests,
scripted scenarios, and seeded fuzz workers. It does not build the Obsidian plugin
or CLI application. Install frontend dependencies with `npm ci` first. Node's
built-in WebSocket, Rust, and a C compiler (`cc`, for test-only exclusive rename
and OS-lock primitives on macOS/Linux) are required.
1. Create `src/tests/my-scenario.test.ts`:
CI installs those dependencies in `frontend` before invoking the runner. It sets
`E2E_ARTIFACTS=logs/e2e` and uploads that same directory even when a suite fails.
The runner owns all test servers; no separate server or legacy cleanup script is
needed in the workflow.
```typescript
import type { TestDefinition } from "../test-definition";
Each server gets a fresh temporary database, working directory, and port. Only
owned processes are terminated. There is no `pkill`, fixed database deletion,
or shared server PID file. Failing tests and teardown errors give nonzero exits;
the remaining suites still run. Logs, concrete fuzz traces, results, and failure
snapshots are retained in the printed artifact directory. Snapshots are captured
before teardown so cleanup failures retain the original evidence.
export const myScenarioTest: TestDefinition = {
description:
"Client 0 creates A.md offline. After syncing, both clients should have the file.",
clients: 2,
steps: [
{ type: "create", client: 0, path: "A.md", content: "hello" },
{ type: "enable-sync", client: 0 },
{ type: "enable-sync", client: 1 },
{ type: "barrier" },
{
type: "assert-consistent",
verify: (s) => {
s.assertFileCount(1).assertContent("A.md", "hello");
}
}
]
};
Environment options:
- `E2E_WORKERS=1`: number of fuzz workers / simultaneous scripted tests (max 32).
- `E2E_SEED=1`: first uint32 seed; workers use successive seeds.
- `E2E_ITERATIONS=25`: generated conflict scenarios per worker.
- `E2E_TIMEOUT_SECONDS=900`: timeout for each build/suite/worker command.
- `E2E_ARTIFACTS=/absolute/path`: optional output directory; defaults to `mkdtemp`.
## Running individual layers
From `frontend`:
```sh
# Rebuild protocol peers and run scripted tests (name filter optional).
npm run test --workspace deterministic-tests -- --filter v4- --concurrency 2
# Adapter/oracle/lifecycle self-tests and strict recovery regressions.
npm run test:harness
# After rebuilding: real worker death, server SIGKILL/restart, CAS/WS ordering,
# and host filesystem contract checks.
npm run test:protocol
npm run build --workspace sync-client --workspace test-client
node test-client/dist/cli.js --seed 42 --iterations 25
node test-client/dist/cli.js --replay /path/to/trace-42.json
```
The `verify` callback receives an `AssertableState` object with chainable assertion methods:
`test:harness` includes strict regressions for **engine** bugs.
A failing default run is not an expected-failure test or a harness skip. See
[`../test-support/README.md`](../test-support/README.md) for their scope.
```typescript
s.assertFileCount(n); // exact file count
s.assertFileExists("path"); // file must exist
s.assertFileNotExists("path"); // file must not exist
s.assertContent("path", "expected"); // exact content match
s.assertContains("path", "a", "b"); // all substrings present in file
s.assertContainsAny("path", "a", "b"); // at least one substring present
s.assertAnyFileContains("text"); // substring present in some file
s.assertNoFileContains("text"); // substring absent from every file
s.assertSubstringCount("path", "x", 3); // substring appears exactly N times
s.assertContentInAtMostOneFile("text"); // no duplicate content
s.ifFileExists("path", (s) => { /* … */ }); // conditional block
s.getContent("path"); // raw content (or "" if missing)
```
## What the scripted runner checks
2. Register it in `src/test-registry.ts`:
Real `SyncClient` instances use the shared v4 adapter in `../test-support`.
External editor operations and engine filesystem operations are separate: engine
writes never generate self-notifications. Editor identity notifications continue
while transport is disabled. Delayed notifications can be explicitly queued and
flushed. Snapshots and persisted state are deep copies, never shared references.
```typescript
import { myScenarioTest } from "./tests/my-scenario.test";
Scripted and seeded clients explicitly set `syncIntervalMs: 0`, with assertions
that it stays disabled. Barriers observe convergence; they do not wake the engine.
Network retry timers remain enabled. Separate fake-clock tests in
`test-support/polling.test.ts` exercise optional periodic reconciliation.
const TESTS = {
// ...
"my-scenario": myScenarioTest
};
```
A barrier checks byte-for-byte replica equality **and** the server's canonical
UUID/path map, file bytes, independently computed base/observed hashes, content
and manifest CAS parents, event watermark,
contiguous event log, and absence of pending requests/application journals.
Internal recovery files are excluded from the user-visible file list, but are
present in failure disk images. Scenario assertions separately check intended
content, document identity, or preservation markers; equality alone is not an
adequate data-loss oracle.
Clients start disabled. `enable-sync` waits for the initial sync attempt/retries,
except while the server is intentionally paused: that attempt is tracked so a
later script step can resume the server. Thus sequential enables really establish
a winner. To test _independent creates_, first bootstrap both clients empty, then
disable them and create files. Initial adoption of pre-existing files during
bootstrap is a different policy from two already-initialized clients creating
distinct UUIDs at one path.
## Controls and assertions
- `create`, `update`, `create-bytes`, `rename`, `delete`: external editor actions.
Editor rename may explicitly replace its target; engine rename never may.
- `sync`, `barrier`, `enable-sync`, `disable-sync`.
- `reset`: restart transports; **not** a crash and not a database reset.
- `pause-websocket`, `resume-websocket`: buffer only WS delivery. HTTP catchup
remains active; this is deliberately not called "disconnect".
- `pause-observation`, `wait-for-observation`, `resume-observation`: explicitly
gate HTTP catchup as well as WS, including replies already in flight. The
checkpoint must be exercised and released, and held reads respect cancellation.
- `delay-notifications`, `flush-notifications`: delay editor notifications without
disabling preservation assertions.
- `drop-response`: `{client, kind: "create" | "content" | "manifest", point:
"before" | "after"}`. After faults fire only on an `Accepted` CAS response.
- `wait-for-response-drop`: explicit checkpoint; every armed fault must fire.
Every interrupted request must be retried with an identical request ID, target
and body before a different CAS is sent. Fault kinds match exactly.
- `rename-next-write`: inject an editor rename after the engine's durable install
rename. The historical name is retained for fixtures; v4 never writes in place.
- `pause-server`, `resume-server`: SIGSTOP/SIGCONT, not a crash.
- `crash-server`, `restart-server`: SIGKILL and restart with the same database/port.
Server-control scenarios have dedicated processes.
- `remember-identity`, `assert-identity`, `assert-files`, `assert-markers`.
- `assert-documents`: a complete expected set of `{key, path, content, conflict?}`
records. Content is text or a byte array. Keys bind UUIDs on the first successful
check and must retain them on later checks; new keys cannot reuse deleted UUIDs.
`conflict: true` resolves the UUID-derived conflict name for `path`.
- `assert-consistent` can provide `verify(state)`: exact text/bytes, UUID lookups,
`assertIdentity`, and `conflictPath(original)` for UUID-derived conflict names.
Legacy timing scenarios are retained. Prefer explicit fault checkpoints and
barriers for new tests. Only known coherent-scan races and actually injected
transient network errors may retry; they still must converge before the deadline.
Other background errors, unhandled rejections, and cleanup failures fail the run.
## Reproducibility limits
Seeds and JSON traces reproduce harness choices and logical actions. They do not
control the OS scheduler, server thread scheduling, or production-generated UUIDs.
The recovery sweep enumerates adapter/save boundaries; worker termination proves
abrupt runtime loss. Neither is a proof covering every real filesystem or hardware
power-loss behavior. Run multiple seeds, repeat timing-sensitive cases, and keep
the scenario's independent oracle when reducing a failing trace.

2
frontend/deterministic-tests/package.json
View file

@ -8,7 +8,7 @@
"scripts": {
"dev": "webpack watch --mode development",
"build": "webpack --mode production",
"test": "npm run build && node dist/cli.js"
"test": "cargo build --release --manifest-path ../../sync-server/Cargo.toml && npm run build --workspace sync-client && npm run build && node dist/cli.js"
},
"devDependencies": {
"commander": "^14.0.2",

26
frontend/deterministic-tests/src/cli.ts
View file

@ -9,10 +9,11 @@ import { runWithConcurrency } from "./run-with-concurrency";
import { TOKEN, SERVER_BINARY_PATH, CONFIG_PATH } from "./consts";
import * as path from "node:path";
import * as fs from "node:fs";
import { debugging, Logger } from "sync-client";
import * as os from "node:os";
import { Logger } from "sync-client";
const logger = new Logger();
debugging.logToConsole(logger, { useColors: true });
logger.onLogEmitted.add((line) => console.log(line.message));
process.on("unhandledRejection", (reason) => {
logger.error(`Unhandled Rejection: ${reason}`);
@ -32,6 +33,8 @@ function testUsesPauseServer(test: TestDefinition): boolean {
(step) =>
step.type === "pause-server" ||
step.type === "resume-server" ||
step.type === "crash-server" ||
step.type === "restart-server" ||
step.type === "resume-server-until-history-then-pause"
);
}
@ -114,11 +117,7 @@ async function runDedicatedServerTest(
}
return { name, result };
} finally {
try {
await server.stop();
} catch {
// best-effort cleanup
}
serverManager.untrack(server);
}
}
@ -138,6 +137,11 @@ async function main(): Promise<void> {
}
const { filter, concurrency } = parseArgs(process.argv);
const artifacts =
process.env.E2E_ARTIFACTS ??
fs.mkdtempSync(path.join(os.tmpdir(), "vault-link-deterministic-"));
fs.mkdirSync(artifacts, { recursive: true });
logger.info(`Artifacts: ${artifacts}`);
const testsToRun: [string, TestDefinition][] = [];
for (const [key, test] of Object.entries(TESTS)) {
@ -193,13 +197,7 @@ async function main(): Promise<void> {
allResults.push(...results);
} finally {
try {
await sharedServer.stop();
} catch (error) {
logger.warn(
`Error stopping shared server: ${error instanceof Error ? error.message : String(error)}`
);
}
serverManager.untrack(sharedServer);
}
}
@ -221,6 +219,10 @@ async function main(): Promise<void> {
const passed = allResults.filter((r) => r.result.success);
const failed = allResults.filter((r) => !r.result.success);
fs.writeFileSync(
path.join(artifacts, "deterministic-results.json"),
JSON.stringify(allResults, null, 2)
);
logger.info(
`\n--- Results: ${passed.length}/${allResults.length} passed ---`

93
frontend/deterministic-tests/src/crash-worker.ts Normal file
View file

@ -0,0 +1,93 @@
import { parentPort, workerData } from "node:worker_threads";
import assert from "node:assert/strict";
import { SyncClient } from "sync-client";
import {
MemoryDisk,
MemoryPersistence,
type DiskImage,
type StoredClient
} from "../../test-support/storage";
import {
classifyRequest,
type RequestRecord,
type RequestKind
} from "../../test-support/network";
export interface CrashWorkerInput {
disk: DiskImage;
stored: StoredClient;
stopAt?: RequestKind;
editAfterSync?: { path: string; content: string };
}
const input = workerData as CrashWorkerInput;
const disk = new MemoryDisk();
disk.restore(input.disk);
const persistence = new MemoryPersistence(input.stored);
const requests: RequestRecord[] = [];
let editedBase: { documentId: string; vaultUpdateId: number } | undefined;
const snapshot = () => ({
disk: disk.image(),
stored: persistence.snapshot(),
requests,
editedBase
});
async function main() {
const client = await SyncClient.create({
fs: disk.session(),
persistence,
fetch: async (url, init) => {
const record = classifyRequest(url, init);
if (record) requests.push(record);
const response = await fetch(url, init);
if (record && response.ok && input.stopAt === record.kind) {
const receipt = (await response.clone().json()) as {
type: string;
};
if (receipt.type === "Accepted") {
parentPort!.postMessage({
type: "checkpoint",
...snapshot()
});
// Parent terminates this worker. No destroy(), no state flush,
// no finally block, and no old runtime survives the restart.
await new Promise<never>(() => {});
}
}
return response;
}
});
assert.equal(
client.getSettings().syncIntervalMs,
0,
"Crash recovery must preserve the disabled polling setting"
);
await client.start();
await client.waitUntilFinished();
if (input.editAfterSync) {
const state = persistence.snapshot().database!;
const documentId = Object.keys(state.local!).find(
(id) => state.local![id] === input.editAfterSync!.path
);
assert(documentId, "Edit must target an established document");
editedBase = {
documentId,
vaultUpdateId: state.documents![documentId].base!.vaultUpdateId
};
await disk.userWrite(
input.editAfterSync.path,
new TextEncoder().encode(input.editAfterSync.content)
);
await client.syncLocallyUpdatedFile({
relativePath: input.editAfterSync.path
});
await client.waitUntilFinished();
}
await client.destroy();
parentPort!.postMessage({ type: "finished", ...snapshot() });
parentPort!.close();
}
main().catch((error: unknown) => {
parentPort!.postMessage({ type: "error", error: String(error) });
process.exit(1);
});

701
frontend/deterministic-tests/src/deterministic-agent.ts
View file

@ -1,483 +1,340 @@
import type {
HistoryEntry,
StoredDatabase,
SyncSettings,
RelativePath,
TextWithCursors
} from "sync-client";
import {
SyncClient,
SyncResetError,
debugging,
LogLevel,
utils
type HistoryEntry,
type SyncSettings
} from "sync-client";
import { MemoryDisk, MemoryPersistence } from "../../test-support/storage";
import { NetworkFaults, type RequestKind } from "../../test-support/network";
import { assert } from "./utils/assert";
import { sleep } from "./utils/sleep";
import { withTimeout } from "./utils/with-timeout";
import {
IS_SYNC_ENABLED_BY_DEFAULT,
WAIT_TIMEOUT_MS,
WEBSOCKET_CONNECT_TIMEOUT_MS,
WEBSOCKET_POLL_INTERVAL_MS
} from "./consts";
import { WAIT_TIMEOUT_MS, WEBSOCKET_CONNECT_TIMEOUT_MS } from "./consts";
import { ManagedWebSocketFactory } from "./managed-websocket";
export class DeterministicAgent extends debugging.InMemoryFileSystem {
public readonly clientId: number;
private readonly logger: (msg: string) => void;
private client!: SyncClient;
private data: Partial<{
settings: Partial<SyncSettings>;
database: Partial<StoredDatabase>;
}> = {};
private isSyncEnabled = IS_SYNC_ENABLED_BY_DEFAULT;
private readonly syncErrors: Error[] = [];
private readonly pendingSyncOperations = new Set<Promise<void>>();
/** User actions live here; the engine only receives disk.session(). */
export class DeterministicAgent {
public readonly disk = new MemoryDisk();
public readonly persistence: MemoryPersistence;
public readonly network = new NetworkFaults();
private client?: SyncClient;
private readonly errors: Error[] = [];
private readonly pending = new Set<Promise<void>>();
private readonly wsFactory = new ManagedWebSocketFactory();
private nextWriteRename:
| {
oldPath: RelativePath;
newPath: RelativePath;
}
| undefined;
private nextCreateResponseDrop:
| {
dropped: Promise<void>;
resolveDropped: () => void;
}
| undefined;
private nextWriteRename?: { oldPath: string; newPath: string };
private delayedNotifications = false;
private disposed = false;
private readonly notifications: (() => Promise<void>)[] = [];
public constructor(
clientId: number,
initialSettings: Partial<SyncSettings>,
logger: (msg: string) => void
public readonly clientId: number,
settings: Partial<SyncSettings>,
private readonly logger: (msg: string) => void
) {
super();
this.clientId = clientId;
this.logger = logger;
this.data.settings = { ...initialSettings };
this.persistence = new MemoryPersistence({
settings: {
networkRetryIntervalMs: 25,
webSocketRetryIntervalMs: 50,
requestTimeoutMs: 5_000,
...settings,
// Scripted and seeded scenarios must exercise notification delivery.
syncIntervalMs: 0
}
});
this.disk.boundary = async (label) => {
const rename = this.nextWriteRename;
// v4 installs through an exclusive rename, never write-in-place.
if (
rename &&
label.startsWith("durable:rename:") &&
label.endsWith(`->${rename.oldPath}`)
) {
this.nextWriteRename = undefined;
await this.rename(rename.oldPath, rename.newPath);
}
};
}
public async init(
fetchImplementation: typeof globalThis.fetch
fetchImplementation: typeof globalThis.fetch = fetch
): Promise<void> {
this.client = await SyncClient.create({
fs: this,
persistence: {
load: async () => this.data,
save: async (data) => void (this.data = data)
},
fetch: this.wrapFetch(fetchImplementation),
fs: this.disk.session(),
persistence: this.persistence,
fetch: this.network.wrap(fetchImplementation),
webSocket: this.wsFactory.constructorFn
});
this.client.logger.onLogEmitted.add((line) => {
const prefix = `[Client ${this.clientId}]`;
switch (line.level) {
case LogLevel.ERROR:
this.logger(`${prefix} ERROR: ${line.message}`);
break;
case LogLevel.WARNING:
this.logger(`${prefix} WARN: ${line.message}`);
break;
case LogLevel.INFO:
this.logger(`${prefix} INFO: ${line.message}`);
break;
case LogLevel.DEBUG:
this.logger(`${prefix} DEBUG: ${line.message}`);
break;
}
});
await this.client.start();
const connectionCheck = await this.client.checkConnection();
assert(
connectionCheck.isSuccessful,
this.client.getSettings().syncIntervalMs === 0,
"Correctness scenarios must run with periodic polling disabled"
);
this.client.logger.onLogEmitted.add((line) => {
this.logger(
`[Client ${this.clientId}] ${line.level}: ${line.message}`
);
if (this.disposed && line.level === LogLevel.ERROR)
throw new Error(
`Client ${this.clientId} error after cleanup: ${line.message}`
);
if (
line.level === LogLevel.ERROR &&
!this.isRetryableFailure(line.message)
)
this.errors.push(new Error(line.message));
});
await this.client.start();
assert(
(await this.client.checkConnection()).isSuccessful,
`Client ${this.clientId} connection check failed`
);
if (this.isSyncEnabled) {
await this.waitForWebSocket();
}
}
public pauseWebSocket(): void {
this.log("Pausing WebSocket message delivery");
this.wsFactory.pause();
public database() {
return this.persistence.snapshot().database;
}
public resumeWebSocket(): void {
this.log("Resuming WebSocket message delivery");
this.wsFactory.resume();
public files(): Map<string, Uint8Array> {
return this.disk.userFiles();
}
public dropNextCreateResponse(): void {
assert(
this.nextCreateResponseDrop === undefined,
`Client ${this.clientId} already has a create response drop armed`
public async listFilesRecursively(): Promise<string[]> {
return [...this.files().keys()].sort();
}
public async getFileContent(path: string): Promise<string> {
const bytes = this.files().get(path);
assert(bytes !== undefined, `Missing file: ${path}`);
return new TextDecoder("utf-8", { ignoreBOM: true }).decode(bytes);
}
public async read(path: string): Promise<Uint8Array> {
const snapshot = await this.disk.readSnapshot(path);
assert(snapshot !== undefined, `Missing file: ${path}`);
return snapshot.content;
}
public async write(path: string, content: Uint8Array): Promise<void> {
const created = !(await this.disk.exists(path));
await this.disk.userWrite(path, content);
this.notify(() =>
created
? this.client!.syncLocallyCreatedFile(path)
: this.client!.syncLocallyUpdatedFile({ relativePath: path })
);
let resolveDropped!: () => void;
const dropped = new Promise<void>((resolve) => {
resolveDropped = resolve;
});
this.nextCreateResponseDrop = {
dropped,
resolveDropped
};
this.log("Armed next create response drop");
}
public async waitForDroppedCreateResponse(): Promise<void> {
public async rename(oldPath: string, relativePath: string): Promise<void> {
await this.disk.userRename(oldPath, relativePath);
this.notify(() =>
this.client!.syncLocallyUpdatedFile({ oldPath, relativePath })
);
}
public async delete(path: string): Promise<void> {
await this.disk.userDelete(path);
this.notify(() => this.client!.syncLocallyDeletedFile(path));
}
public delayNotifications(): void {
this.delayedNotifications = true;
}
public flushNotifications(): void {
this.delayedNotifications = false;
for (const notify of this.notifications.splice(0)) this.enqueue(notify);
}
private notify(operation: () => Promise<void>): void {
// Offline means transport disabled, not that editor identity events vanish.
if (this.delayedNotifications) this.notifications.push(operation);
else this.enqueue(operation);
}
private enqueue(operation: () => Promise<void>): void {
const promise = operation().catch((error: unknown) => {
if (!this.isRetryableFailure(String(error)))
this.errors.push(
error instanceof Error ? error : new Error(String(error))
);
});
this.pending.add(promise);
void promise.then(() => this.pending.delete(promise));
}
private async drain(): Promise<void> {
while (this.pending.size) await Promise.all([...this.pending]);
}
private isRetryableFailure(message: string): boolean {
// A coherent scan rejecting an editor race is a required adapter
// behavior. It must subsequently converge; never blanket-ignore errors.
return (
this.network.isExpectedFailure(message) ||
/^(Sync paused for retry: )?Error: File changed during scan: .+$/.test(
message
)
);
}
private assertHealthy(): void {
if (this.errors.length) {
const errors = this.errors.splice(0);
throw new AggregateError(
errors,
`Client ${this.clientId}: ${errors.map(String).join("; ")}`
);
}
}
public async waitForSync(): Promise<void> {
assert(
this.nextCreateResponseDrop !== undefined,
`Client ${this.clientId} has no create response drop armed`
this.client!.getSettings().syncIntervalMs === 0,
"Periodic polling was enabled during a correctness scenario"
);
await withTimeout(
this.nextCreateResponseDrop.dropped,
(async () => {
await this.drain();
const deadline = Date.now() + WAIT_TIMEOUT_MS;
for (;;) {
try {
await this.client!.waitUntilFinished();
break;
} catch (error) {
if (
!this.isRetryableFailure(String(error)) ||
Date.now() >= deadline
)
throw error;
await sleep(30);
}
}
this.assertHealthy();
})(),
WAIT_TIMEOUT_MS,
`Client ${this.clientId} timed out waiting for create response drop`
`Client ${this.clientId} sync timeout`
);
this.log("Create response was dropped after server commit");
}
public async disableSync(): Promise<void> {
await this.drain();
await this.client!.setSetting("isSyncEnabled", false);
this.assertHealthy();
}
public async enableSync(wait = true): Promise<void> {
// setSetting waits for a sync attempt in v4. Track it, but don't block
// the script from resuming a paused server or releasing a checkpoint.
this.enqueue(() => this.client!.setSetting("isSyncEnabled", true));
const deadline = Date.now() + WEBSOCKET_CONNECT_TIMEOUT_MS;
while (
!this.persistence.snapshot().settings?.isSyncEnabled &&
Date.now() < deadline
)
await sleep(1);
assert(
!!this.persistence.snapshot().settings?.isSyncEnabled,
"Enable did not persist"
);
if (wait) await this.waitForSync();
}
public async reset(): Promise<void> {
await this.drain();
await this.client!.reset();
}
public pauseWebSocket(): void {
this.wsFactory.pause();
}
public pauseObservation(): void {
this.pauseWebSocket();
this.network.pauseObservation();
}
public resumeObservation(): void {
this.network.resumeObservation();
this.resumeWebSocket();
}
public async waitForObservation(): Promise<void> {
await withTimeout(
this.network.waitForObservation(),
WAIT_TIMEOUT_MS,
"HTTP observation checkpoint never reached"
);
}
public resumeWebSocket(): void {
this.wsFactory.resume();
}
public dropNextCreateResponse(): void {
this.network.arm("create");
}
public dropNextResponse(
kind: RequestKind,
point: "before" | "after" = "after"
): void {
this.network.arm(kind, point);
}
public async waitForDroppedCreateResponse(): Promise<void> {
await withTimeout(
this.network.wait(),
WAIT_TIMEOUT_MS,
"Armed response drop never fired"
);
}
public renameNextWrite(oldPath: string, newPath: string): void {
assert(
!this.nextWriteRename,
"Previous install/rename hook never fired"
);
this.nextWriteRename = { oldPath, newPath };
}
public async waitForHistoryEntry(
matches: (entry: HistoryEntry) => boolean,
onMatch?: (entry: HistoryEntry) => void
): Promise<void> {
const existing = this.client.getHistoryEntries().find(matches);
if (existing !== undefined) {
onMatch?.(existing);
return;
}
let unsubscribe = () => {};
try {
await withTimeout(
new Promise<void>((resolve) => {
const unsubscribe = this.client.onSyncHistoryUpdated.add(() => {
const entry = this.client
.getHistoryEntries()
.find(matches);
if (entry === undefined) {
return;
}
unsubscribe();
const check = () => {
const entry =
this.client!.getHistoryEntries().find(matches);
if (entry) {
onMatch?.(entry);
resolve();
});
}
};
unsubscribe = this.client!.onSyncHistoryUpdated.add(check);
check();
}),
WAIT_TIMEOUT_MS,
`Client ${this.clientId} timed out waiting for history entry`
"History checkpoint not reached"
);
}
public async waitForSync(): Promise<void> {
this.log("Waiting for sync to complete...");
// Drain agent-level sync operations first. These are the fire-and-forget
// promises from enqueueSync() that call into the SyncClient's methods.
// Without this, waitUntilFinished() might return before the SyncClient
// has even been told about the operation.
await this.drainPendingSyncOperations();
await withTimeout(
this.client.waitUntilFinished(),
WAIT_TIMEOUT_MS,
`Client ${this.clientId} waitForSync timed out after ${WAIT_TIMEOUT_MS}ms`
);
if (this.syncErrors.length > 0) {
const errors = this.syncErrors.splice(0);
throw new Error(
`Client ${this.clientId} had ${errors.length} sync error(s):\n${errors.map((e) => e.message).join("\n")}`
);
}
this.log("Sync complete");
}
public async reset(): Promise<void> {
this.log("Resetting client (clears tracked state, keeps disk files)");
await this.drainPendingSyncOperations();
await this.client.reset();
if (this.isSyncEnabled) {
await this.waitForWebSocket();
} finally {
unsubscribe();
}
}
public async disableSync(): Promise<void> {
this.log("Disabling sync");
// Drain pending enqueued operations before disabling so the SyncClient
// knows about all operations that were enqueued while sync was enabled.
await this.drainPendingSyncOperations();
await this.client.setSetting("isSyncEnabled", false);
this.isSyncEnabled = false;
// Wait for in-flight operations to drain. Disabling sync triggers
// a reset, which aborts in-flight fetches with SyncResetError.
try {
await withTimeout(
this.client.waitUntilFinished(),
WAIT_TIMEOUT_MS,
`Client ${this.clientId} disableSync drain timed out`
);
} catch (error) {
if (error instanceof Error && error.name === "SyncResetError") {
this.log("Disable sync drain interrupted by reset (expected)");
} else {
throw error;
}
}
}
public async enableSync(): Promise<void> {
this.log("Enabling sync");
await this.client.setSetting("isSyncEnabled", true);
this.isSyncEnabled = true;
await this.waitForWebSocket();
}
public async getFileContent(path: string): Promise<string> {
const bytes = await this.read(path);
return new TextDecoder().decode(bytes);
}
public renameNextWrite(oldPath: RelativePath, newPath: RelativePath): void {
assert(
this.nextWriteRename === undefined,
`Client ${this.clientId} already has a next-write rename armed`
);
this.nextWriteRename = { oldPath, newPath };
this.log(`Armed next write rename: ${oldPath} -> ${newPath}`);
}
public async cleanup(): Promise<void> {
this.log("Cleaning up...");
// Guard against uninitialized client (init() failed partway).
// The class field uses `!:` so TS thinks this is always defined,
// but at runtime it can be undefined when init() throws partway.
const maybeClient = this.client as SyncClient | undefined;
if (maybeClient === undefined) {
this.log("Client not initialized, nothing to clean up");
return;
if (!this.client) return;
const errors: unknown[] = [];
this.resumeObservation();
this.flushNotifications();
try {
await this.waitForSync();
} catch (error) {
errors.push(error);
}
try {
this.network.assertConsumed();
assert(!this.nextWriteRename, "Install/rename hook never fired");
} catch (error) {
errors.push(error);
}
try {
await this.drainPendingSyncOperations();
await withTimeout(
this.client.waitUntilFinished(),
this.client.destroy(),
WAIT_TIMEOUT_MS,
`Client ${this.clientId} cleanup waitUntilFinished timed out`
"Destroy timed out"
);
await withTimeout(
this.wsFactory.finish(),
WAIT_TIMEOUT_MS,
"WebSocket callbacks did not finish during cleanup"
);
} catch (error) {
if (error instanceof Error && error.name === "SyncResetError") {
this.log(`Cleanup interrupted by reset (expected): ${error}`);
} else {
this.log(`Cleanup waitUntilFinished failed: ${error}`);
errors.push(error);
}
}
// Surface any background sync errors that arrived after the last
// waitForSync (e.g. between the final assert-consistent and here).
// Without this, regressions that fault the engine during the very
// last step of a test would be silently swallowed.
const pendingErrors = this.syncErrors.splice(0);
await this.client.destroy();
this.log("Cleanup complete");
if (pendingErrors.length > 0) {
throw new Error(
`Client ${this.clientId} had ${pendingErrors.length} background sync error(s) during cleanup:\n${pendingErrors.map((e) => e.message).join("\n")}`
);
}
}
public override async read(path: RelativePath): Promise<Uint8Array> {
await Promise.resolve();
return super.read(path);
}
public override async write(
path: RelativePath,
content: Uint8Array
): Promise<void> {
await Promise.resolve();
const isNew = !this.files.has(path);
await super.write(path, content);
if (this.isSyncEnabled && isNew) {
this.enqueueSync(async () => {
this.client.syncLocallyCreatedFile(path);
});
}
const { nextWriteRename } = this;
if (
nextWriteRename !== undefined &&
nextWriteRename.oldPath === path
) {
this.nextWriteRename = undefined;
await super.rename(
nextWriteRename.oldPath,
nextWriteRename.newPath
);
if (this.isSyncEnabled) {
this.enqueueSync(async () => {
this.client.syncLocallyUpdatedFile({
oldPath: nextWriteRename.oldPath,
relativePath: nextWriteRename.newPath
});
});
}
// The rename consumed `path`. Skip the post-update enqueue below
// — it would send a syncLocallyUpdatedFile for a path that no
// longer exists.
return;
}
if (!this.isSyncEnabled) {
return;
}
if (!isNew) {
this.enqueueSync(async () => {
this.client.syncLocallyUpdatedFile({ relativePath: path });
});
}
}
public override async atomicUpdateText(
path: RelativePath,
updater: (current: TextWithCursors) => TextWithCursors
): Promise<string> {
const result = await super.atomicUpdateText(path, updater);
if (this.isSyncEnabled) {
this.enqueueSync(async () => {
this.client.syncLocallyUpdatedFile({ relativePath: path });
});
}
return result;
}
public override async delete(path: RelativePath): Promise<void> {
await super.delete(path);
if (this.isSyncEnabled) {
this.enqueueSync(async () => {
this.client.syncLocallyDeletedFile(path);
});
}
}
public override async rename(
oldPath: RelativePath,
newPath: RelativePath
): Promise<void> {
await super.rename(oldPath, newPath);
if (this.isSyncEnabled) {
this.enqueueSync(async () => {
this.client.syncLocallyUpdatedFile({
oldPath,
relativePath: newPath
});
});
}
}
private async waitForWebSocket(): Promise<void> {
const deadline = Date.now() + WEBSOCKET_CONNECT_TIMEOUT_MS;
while (!this.client.isWebSocketConnected && Date.now() < deadline) {
await sleep(WEBSOCKET_POLL_INTERVAL_MS);
}
assert(
this.client.isWebSocketConnected,
`Client ${this.clientId} WebSocket failed to connect within ${WEBSOCKET_CONNECT_TIMEOUT_MS}ms`
);
}
/**
* Wait until all agent-level enqueued sync operations have completed.
* Uses a loop because completing one operation can trigger new enqueues.
*/
private async drainPendingSyncOperations(): Promise<void> {
while (this.pendingSyncOperations.size > 0) {
await utils.awaitAll([...this.pendingSyncOperations]);
}
}
private enqueueSync(operation: () => Promise<void>): void {
const promise = this.executeSyncOperation(operation).catch(
(error: unknown) => {
const err =
error instanceof Error ? error : new Error(String(error));
this.log(`Background sync failed: ${err.message}`);
this.syncErrors.push(err);
}
);
this.pendingSyncOperations.add(promise);
void promise.finally(() => {
this.pendingSyncOperations.delete(promise);
});
}
private async executeSyncOperation(
operation: () => Promise<void>
): Promise<void> {
this.client = undefined;
this.disposed = true;
try {
await operation();
this.assertHealthy();
} catch (error) {
if (error instanceof Error && error.name === "SyncResetError") {
this.log(`Sync operation interrupted by reset: ${error}`);
return;
errors.push(error);
}
if (
error instanceof Error &&
error.message.includes("has been destroyed")
) {
this.log(`Sync operation interrupted by destroy: ${error}`);
return;
}
throw error;
}
}
private log(message: string): void {
this.logger(`[Client ${this.clientId}] ${message}`);
}
private wrapFetch(
fetchImplementation: typeof globalThis.fetch
): typeof globalThis.fetch {
return async (input, init) => {
const response = await fetchImplementation(input, init);
const drop = this.nextCreateResponseDrop;
if (
drop !== undefined &&
DeterministicAgent.isCreateDocumentRequest(input, init)
) {
this.nextCreateResponseDrop = undefined;
try {
await response.body?.cancel();
} catch {
// Best-effort — body may already be consumed/closed.
}
drop.resolveDropped();
throw new SyncResetError();
}
return response;
};
}
private static isCreateDocumentRequest(
input: RequestInfo | URL,
init: RequestInit | undefined
): boolean {
const method =
init?.method ??
(typeof Request !== "undefined" && input instanceof Request
? input.method
: "GET");
if (method.toUpperCase() !== "POST") {
return false;
}
const url =
input instanceof URL
? input
: new URL(typeof input === "string" ? input : input.url);
return /\/documents\/?$/.test(url.pathname);
if (errors.length)
throw new AggregateError(
errors,
`Client ${this.clientId} cleanup failed: ${errors.map(String).join("; ")}`
);
}
}

21
frontend/deterministic-tests/src/managed-websocket.ts
View file

@ -209,8 +209,13 @@ export class ManagedWebSocketFactory {
// WS before the agent reconnects would silently see the new socket
// start un-paused and miss the messages it meant to buffer.
private currentlyPaused = false;
private disposed = false;
public get constructorFn(): typeof globalThis.WebSocket {
const ensureActive = () => {
if (this.disposed)
throw new Error("WebSocket created after harness cleanup");
};
const trackInstance = (instance: ManagedWebSocket): void => {
this.instances.push(instance);
if (this.currentlyPaused) {
@ -222,6 +227,7 @@ export class ManagedWebSocketFactory {
url: string | URL,
protocols?: string | string[]
) {
ensureActive();
super(url, protocols);
trackInstance(this);
}
@ -229,6 +235,21 @@ export class ManagedWebSocketFactory {
return TrackedManagedWebSocket;
}
public async finish(): Promise<void> {
this.disposed = true;
await Promise.all(
this.instances.map((socket) =>
socket.readyState === WebSocket.CLOSED
? Promise.resolve()
: new Promise<void>((resolve) =>
socket.addEventListener("close", () => resolve(), {
once: true
})
)
)
);
}
public pause(): void {
this.currentlyPaused = true;
for (const ws of this.instances) {

4
frontend/deterministic-tests/src/parse-args.ts
View file

@ -7,8 +7,8 @@ export interface CliArgs {
}
function parsePositiveInt(value: string): number {
const n = parseInt(value, 10);
if (isNaN(n) || n <= 0) {
const n = Number(value);
if (!/^\d+$/.test(value) || !Number.isSafeInteger(n) || n <= 0) {
throw new InvalidArgumentError("must be a positive integer");
}
return n;

118
frontend/deterministic-tests/src/server-control.ts
View file

@ -20,10 +20,13 @@ export class ServerControl {
private _port: number | undefined;
private tempDir: string | undefined;
private _isPaused = false;
private preserveOnStop = false;
private expectedExit = false;
private unexpectedExit?: Error;
public constructor(serverPath: string, configPath: string, logger: Logger) {
this.serverPath = serverPath;
this.baseConfigPath = configPath;
this.serverPath = path.resolve(serverPath);
this.baseConfigPath = path.resolve(configPath);
this.logger = logger;
}
@ -38,6 +41,15 @@ export class ServerControl {
return `http://localhost:${this.port}`;
}
public get databaseDirectory(): string {
if (!this.tempDir) throw new Error("Server has no database directory");
return path.join(this.tempDir, "databases");
}
public get isPaused(): boolean {
return this._isPaused;
}
public async start(): Promise<void> {
if (this.process !== null) {
throw new Error("Server is already running");
@ -66,14 +78,25 @@ export class ServerControl {
}
private async startOnce(): Promise<void> {
const reservation = await findFreePort();
this._port = reservation.port;
const reservation =
this._port === undefined ? await findFreePort() : undefined;
this._port ??= reservation!.port;
const tmpBase = os.tmpdir();
this.tempDir = fs.mkdtempSync(path.join(tmpBase, "vault-link-test-"));
const tempConfigPath = path.join(this.tempDir, "config.yml");
const dbDir = path.join(this.tempDir, "databases");
this.writeConfigFile(tempConfigPath, dbDir);
let tempConfigPath: string;
try {
this.tempDir ??= fs.mkdtempSync(
path.join(tmpBase, "vault-link-test-")
);
tempConfigPath = path.join(this.tempDir, "config.yml");
this.writeConfigFile(
tempConfigPath,
path.join(this.tempDir, "databases")
);
} catch (error) {
reservation?.release();
this.cleanupTempDir();
throw error;
}
this.logger.info(
`Starting server: ${this.serverPath} (port ${this._port})`
@ -81,12 +104,15 @@ export class ServerControl {
// Release the port reservation right before spawning to minimize
// the TOCTOU window between port discovery and server binding.
reservation.release();
reservation?.release();
this.process = spawn(this.serverPath, [tempConfigPath], {
cwd: this.tempDir,
stdio: ["ignore", "pipe", "pipe"],
detached: false
});
this.expectedExit = false;
this.unexpectedExit = undefined;
this.process.stdout?.on("data", (data: Buffer) => {
this.logger.info(`[SERVER] ${data.toString().trim()}`);
@ -102,6 +128,10 @@ export class ServerControl {
const currentProcess = this.process;
currentProcess.on("exit", (code, signal) => {
if (!this.expectedExit)
this.unexpectedExit = new Error(
`Owned server exited unexpectedly (code ${code}, signal ${signal})`
);
this.logger.info(
`Server exited with code ${code}, signal ${signal}`
);
@ -139,7 +169,9 @@ export class ServerControl {
);
}
try {
const response = await fetch(pingUrl);
const response = await fetch(pingUrl, {
signal: AbortSignal.timeout(1_000)
});
if (response.ok) {
this.logger.info("[SERVER] Ready");
return;
@ -194,11 +226,13 @@ export class ServerControl {
public async stop(): Promise<void> {
const proc = this.process;
if (proc?.pid === undefined) {
this.cleanupTempDir();
if (!this.preserveOnStop) this.cleanupTempDir();
if (this.unexpectedExit) throw this.unexpectedExit;
return;
}
this.expectedExit = true;
// Resume if paused — a SIGSTOP'd process ignores SIGKILL
// SIGKILL terminates even a stopped process; SIGCONT is harmless here.
if (this._isPaused) {
try {
process.kill(proc.pid, "SIGCONT");
@ -212,7 +246,7 @@ export class ServerControl {
// Set up a promise that resolves when the process actually exits.
const exitPromise = new Promise<void>((resolve) => {
if (proc.exitCode !== null) {
if (proc.exitCode !== null || proc.signalCode !== null) {
resolve();
return;
}
@ -229,11 +263,49 @@ export class ServerControl {
// Wait for the process to actually exit before cleaning up,
// with a 5s safety timeout to avoid hanging forever.
await Promise.race([exitPromise, sleep(STOP_TIMEOUT_MS)]);
let timer: ReturnType<typeof setTimeout> | undefined;
try {
await Promise.race([
exitPromise,
new Promise<never>((_, reject) => {
timer = setTimeout(
() =>
reject(
new Error(
"Owned server did not exit after SIGKILL"
)
),
STOP_TIMEOUT_MS
);
})
]);
} finally {
clearTimeout(timer);
}
this.process = null;
this._isPaused = false;
this.cleanupTempDir();
if (!this.preserveOnStop) this.cleanupTempDir();
if (this.unexpectedExit) throw this.unexpectedExit;
}
/** A real process death: retain the exact database and port for restart. */
public async crash(): Promise<void> {
if (!this.isRunning()) throw new Error("Cannot crash a stopped server");
this.preserveOnStop = true;
try {
await this.stop();
} finally {
this.preserveOnStop = false;
}
}
public async restart(): Promise<void> {
if (!this.tempDir || this.isRunning())
throw new Error(
"Restart requires a crashed server with retained storage"
);
await this.startOnce();
}
public isRunning(): boolean {
@ -273,23 +345,27 @@ export class ServerControl {
// one `databases_directory_path:` (under `server:` and `database:`
// respectively)
const baseConfig = fs.readFileSync(this.baseConfigPath, "utf-8");
for (const key of ["port", "databases_directory_path"]) {
if (
(baseConfig.match(new RegExp(`^ ${key}:`, "gm")) ?? [])
.length !== 1
)
throw new Error(`Expected exactly one config key: ${key}`);
}
const config = baseConfig
.replace(/^\s*port:\s*\d+/m, ` port: ${this._port}`)
.replace(
/^\s*databases_directory_path:\s*.+/m,
` databases_directory_path: ${dbDir}`
` databases_directory_path: ${JSON.stringify(dbDir)}`
);
fs.writeFileSync(destPath, config);
}
private cleanupTempDir(): void {
if (this.tempDir !== undefined) {
try {
fs.rmSync(this.tempDir, { recursive: true, force: true });
} catch {
// Best-effort cleanup
}
this.tempDir = undefined;
this._port = undefined;
}
}
}

5
frontend/deterministic-tests/src/server-manager.ts
View file

@ -59,9 +59,8 @@ export class ServerManager {
// Last-resort synchronous cleanup. Runs even when the process is
// exiting via process.exit() from unhandledRejection /
// uncaughtException — paths where async stopAll() cannot complete.
// SIGSTOP'd servers MUST receive SIGCONT before SIGKILL or the
// kernel keeps them as zombies holding the test's tmpdir, and the
// next CI run can't reuse the port.
// Kill only the server processes owned by this runner. SIGKILL also
// terminates a SIGSTOP'd process; forceKillSync's SIGCONT is harmless.
process.on("exit", () => {
for (const server of this.activeServers) {
server.forceKillSync();

46
frontend/deterministic-tests/src/test-definition.ts
View file

@ -3,6 +3,18 @@ import type { AssertableState } from "./utils/assertable-state";
export interface ClientState {
files: Map<string, string>;
clientFiles: Map<string, string>[];
manifests: Record<string, string>[];
canonical: Record<string, string>;
bytes: Map<string, Uint8Array>;
}
/** A scenario's independent, complete expected state. Keys survive moves;
* a new key must receive a previously unseen UUID, including after deletion. */
export interface ExpectedDocument {
key: string;
path: string;
conflict?: boolean;
content: string | number[];
}
export type TestStep =
@ -31,10 +43,41 @@ export type TestStep =
| { type: "assert-consistent"; verify?: (state: AssertableState) => void }
| { type: "pause-websocket"; client: number }
| { type: "resume-websocket"; client: number }
| {
type:
| "pause-observation"
| "resume-observation"
| "wait-for-observation";
client: number;
}
| { type: "drop-next-create-response"; client: number }
| { type: "wait-for-dropped-create-response"; client: number }
| { type: "sleep"; ms: number }
| { type: "reset"; client: number };
| { type: "reset"; client: number }
| { type: "create-bytes"; client: number; path: string; bytes: number[] }
| {
type: "drop-response";
client: number;
kind: "create" | "content" | "manifest";
point?: "before" | "after";
}
| { type: "wait-for-response-drop"; client: number }
| { type: "delay-notifications" | "flush-notifications"; client: number }
| {
type: "remember-identity" | "assert-identity";
path: string;
key: string;
}
| {
type: "assert-files";
expected: Record<string, string>;
absent?: string[];
count?: number;
}
| { type: "assert-markers"; markers: string[]; removed?: string[] }
| { type: "assert-documents"; expected: ExpectedDocument[] }
| { type: "crash-server" }
| { type: "restart-server" };
export interface TestDefinition {
description?: string;
@ -46,4 +89,5 @@ export interface TestResult {
success: boolean;
error?: string;
duration?: number;
diagnostics?: unknown;
}

2
frontend/deterministic-tests/src/test-registry.ts
View file

@ -1,4 +1,5 @@
import type { TestDefinition } from "./test-definition";
import { V4_TESTS } from "./v4-regressions";
import { renameCreateConflictTest } from "./tests/rename-create-conflict.test";
import { renameChainTest } from "./tests/rename-chain.test";
import { renameUpdateConflictTest } from "./tests/rename-update-conflict.test";
@ -106,6 +107,7 @@ import { remoteQuickWriteRenameBeforeRecordTest } from "./tests/remote-quick-wri
import { selfMergePendingRenameAliasesSecondCreateTest } from "./tests/self-merge-pending-rename-aliases-second-create.test";
export const TESTS: Partial<Record<string, TestDefinition>> = {
...V4_TESTS,
"rename-create-conflict": renameCreateConflictTest,
"rename-chain": renameChainTest,
"rename-update-conflict": renameUpdateConflictTest,

142
frontend/deterministic-tests/src/test-runner.ts
View file

@ -13,6 +13,8 @@ import {
IS_SYNC_ENABLED_BY_DEFAULT
} from "./consts";
import { randomUUID } from "node:crypto";
import { assertCanonical } from "../../test-support/canonical";
import { ContentLedger } from "../../test-support/oracles";
export class TestRunner {
private agents: DeterministicAgent[] = [];
@ -20,6 +22,8 @@ export class TestRunner {
private readonly token: string;
private readonly remoteUri: string;
private readonly logger: Logger;
private vaultName = "";
private readonly identities = new Map<string, string>();
public constructor(
serverControl: ServerControl,
@ -44,6 +48,8 @@ export class TestRunner {
}
this.logger.info(`Clients: ${test.clients}`);
this.logger.info(`Steps: ${test.steps.length}`);
this.identities.clear();
let diagnostics: TestResult["diagnostics"];
try {
assert(
@ -61,6 +67,11 @@ export class TestRunner {
await this.executeStep(step);
}
assert(
this.serverControl.isRunning(),
"Server died before test completion"
);
diagnostics = this.captureDiagnostics();
await this.cleanup();
const duration = Date.now() - startTime;
@ -71,25 +82,46 @@ export class TestRunner {
duration
};
} catch (error) {
diagnostics ??= this.captureDiagnostics();
const duration = Date.now() - startTime;
const errorMessage =
error instanceof Error ? error.message : String(error);
this.logger.info(`\n✗ Test failed: ${name}`);
this.logger.info(`Error: ${errorMessage}`);
try {
await this.cleanup();
} catch (cleanupError) {
return {
success: false,
error: `${errorMessage}\nCleanup: ${String(cleanupError)}`,
duration,
diagnostics
};
}
return {
success: false,
error: errorMessage,
diagnostics,
duration
};
}
}
private captureDiagnostics(): unknown {
return this.agents.map((agent) => ({
client: agent.clientId,
database: agent.database(),
disk: agent.disk.image(),
requests: structuredClone(agent.network.requests)
}));
}
private async initializeAgents(count: number): Promise<void> {
assert(count > 0, `Client count must be positive, got ${count}`);
const vaultName = `test-${randomUUID()}`;
this.vaultName = vaultName;
this.logger.info(
`Initializing ${count} agents with vault: ${vaultName}`
);
@ -129,6 +161,87 @@ export class TestRunner {
private async executeStep(step: TestStep): Promise<void> {
switch (step.type) {
case "pause-observation":
this.getAgent(step.client).pauseObservation();
break;
case "resume-observation":
this.getAgent(step.client).resumeObservation();
break;
case "wait-for-observation":
await this.getAgent(step.client).waitForObservation();
break;
case "create-bytes":
await this.getAgent(step.client).write(
step.path,
new Uint8Array(step.bytes)
);
break;
case "drop-response":
this.getAgent(step.client).dropNextResponse(
step.kind,
step.point
);
break;
case "wait-for-response-drop":
await this.getAgent(step.client).waitForDroppedCreateResponse();
break;
case "delay-notifications":
this.getAgent(step.client).delayNotifications();
break;
case "flush-notifications":
this.getAgent(step.client).flushNotifications();
break;
case "remember-identity":
case "assert-identity":
await this.assertConsistent((state) => {
if (step.type === "remember-identity")
this.identities.set(
step.key,
state.documentId(step.path)
);
else {
const id = this.identities.get(step.key);
assert(
id !== undefined,
`Identity ${step.key} was never recorded`
);
state.assertIdentity(step.path, id);
}
});
break;
case "assert-files":
await this.assertConsistent((state) => {
for (const [path, value] of Object.entries(step.expected))
state.assertContent(path, value);
for (const path of step.absent ?? [])
state.assertFileNotExists(path);
if (step.count !== undefined)
state.assertFileCount(step.count);
});
break;
case "assert-documents":
await this.assertConsistent((state) =>
state.assertDocuments(step.expected, this.identities)
);
break;
case "assert-markers": {
const ledger = new ContentLedger();
for (const marker of step.markers) ledger.add(marker);
for (const marker of step.removed ?? [])
ledger.removedBy(
marker,
"trace records explicit deletion/overwrite"
);
for (const agent of this.agents)
ledger.assertPreserved(agent.files());
break;
}
case "crash-server":
await this.serverControl.crash();
break;
case "restart-server":
await this.serverControl.restart();
break;
case "create":
case "update":
await this.getAgent(step.client).write(
@ -170,7 +283,9 @@ export class TestRunner {
break;
case "enable-sync":
await this.getAgent(step.client).enableSync();
await this.getAgent(step.client).enableSync(
!this.serverControl.isPaused
);
break;
case "pause-server":
@ -190,7 +305,9 @@ export class TestRunner {
(entry) =>
entry.details.type === step.syncType &&
entry.details.relativePath === step.path,
() => { this.serverControl.pause(); }
() => {
this.serverControl.pause();
}
);
this.serverControl.resume();
await historySeen;
@ -305,6 +422,11 @@ export class TestRunner {
// Snapshot all agents' file states upfront to minimize the window
// where background sync could mutate state between reads.
const canonical = await assertCanonical(
this.agents,
`${this.remoteUri}/vaults/${encodeURIComponent(this.vaultName)}`,
this.token
);
const clientFiles: Map<string, string>[] = [];
for (const agent of this.agents) {
const sortedFiles = (await agent.listFilesRecursively()).sort();
@ -360,7 +482,12 @@ export class TestRunner {
verify(
new AssertableState({
files: clientFiles[0],
clientFiles
clientFiles,
manifests: this.agents.map(
(agent) => agent.database()!.local!
),
canonical: canonical.fileManifest.entries,
bytes: this.agents[0].files()
})
);
} catch (error) {
@ -384,10 +511,12 @@ export class TestRunner {
}
this.logger.info("\nCleaning up agents...");
for (const agent of this.agents) {
const errors: unknown[] = [];
for (const agent of this.agents.splice(0)) {
try {
await agent.cleanup();
} catch (error) {
errors.push(error);
this.logger.warn(
`Agent cleanup error: ${error instanceof Error ? error.message : String(error)}`
);
@ -395,5 +524,10 @@ export class TestRunner {
}
this.agents = [];
this.logger.info("Cleanup complete");
if (errors.length)
throw new AggregateError(
errors,
`Cleanup failed: ${errors.map(String).join("; ")}`
);
}
}

93
frontend/deterministic-tests/src/utils/assertable-state.ts
View file

@ -1,12 +1,103 @@
import type { ClientState } from "../test-definition";
import assert from "node:assert/strict";
import type { ClientState, ExpectedDocument } from "../test-definition";
export class AssertableState {
public readonly files: Map<string, string>;
public readonly clientFiles: Map<string, string>[];
public readonly manifests: Record<string, string>[];
public readonly canonical: Record<string, string>;
public readonly bytes: Map<string, Uint8Array>;
public constructor(state: ClientState) {
this.files = state.files;
this.clientFiles = state.clientFiles;
this.manifests = state.manifests;
this.canonical = state.canonical;
this.bytes = state.bytes;
}
public documentId(path: string): string {
const id = Object.keys(this.canonical).find(
(id) => this.canonical[id] === path
);
if (!id) throw new Error(`No document identity for ${path}`);
return id;
}
public assertDocuments(
expected: readonly ExpectedDocument[],
identities: Map<string, string>
): this {
this.assertFileCount(expected.length);
assert.equal(
new Set(expected.map((doc) => doc.key)).size,
expected.length,
"Duplicate expected identity key"
);
const next = new Map(identities);
const paths = new Set<string>();
for (const doc of expected) {
const path = doc.conflict ? this.conflictPath(doc.path) : doc.path;
assert(
!paths.has(path),
`Two expected documents resolved to ${path}`
);
paths.add(path);
this.assertBytes(
path,
typeof doc.content === "string"
? new TextEncoder().encode(doc.content)
: new Uint8Array(doc.content)
);
const id = this.documentId(path);
const recorded = next.get(doc.key);
if (recorded !== undefined) this.assertIdentity(path, recorded);
else {
assert(
![...next.values()].includes(id),
`New document ${doc.key} reused an existing or deleted UUID`
);
next.set(doc.key, id);
}
}
// A failed check must not establish a partial identity oracle.
for (const [key, id] of next) identities.set(key, id);
return this;
}
public conflictPath(original: string): string {
const dot = original.lastIndexOf(".");
const split =
dot > original.lastIndexOf("/") + 1 ? dot : original.length;
const matches = Object.entries(this.canonical).filter(
([id, path]) =>
path ===
`${original.slice(0, split)} (conflict ${id})${original.slice(split)}`
);
if (matches.length !== 1)
throw new Error(
`Expected one UUID-derived conflict path for ${original}, found ${matches.length}`
);
return matches[0][1];
}
public assertIdentity(path: string, id: string): this {
if (this.documentId(path) !== id)
throw new Error(
`Document identity changed at ${path}: expected ${id}, got ${this.documentId(path)}`
);
return this;
}
public assertBytes(path: string, expected: Uint8Array): this {
const actual = this.bytes.get(path);
if (
!actual ||
actual.length !== expected.length ||
actual.some((byte, i) => byte !== expected[i])
)
throw new Error(`Byte mismatch at ${path}`);
return this;
}
public assertFileCount(expected: number): this {

253
frontend/deterministic-tests/src/v4-regressions.ts Normal file
View file

@ -0,0 +1,253 @@
import type { TestDefinition, TestStep } from "./test-definition";
const online: TestStep[] = [
{ type: "enable-sync", client: 0 },
{ type: "enable-sync", client: 1 },
{ type: "barrier" }
];
const offline: TestStep[] = [
{ type: "disable-sync", client: 0 },
{ type: "disable-sync", client: 1 }
];
export const V4_TESTS: Record<string, TestDefinition> = {
"v4-text-byte-and-component-boundaries": {
clients: 2,
steps: [
...online,
{ type: "create", client: 0, path: "empty.md", content: "" },
{
type: "create",
client: 0,
path: "bom.md",
content: "\ufefffirst\r\nlast"
},
{
type: "create",
client: 0,
path: `${"a".repeat(252)}.md`,
content: "255 ASCII bytes"
},
{
type: "create",
client: 0,
path: `${"é".repeat(126)}.md`,
content: "255 UTF-8 bytes"
},
{ type: "barrier" },
{
type: "assert-files",
count: 4,
expected: {
"empty.md": "",
"bom.md": "\ufefffirst\r\nlast",
[`${"a".repeat(252)}.md`]: "255 ASCII bytes",
[`${"é".repeat(126)}.md`]: "255 UTF-8 bytes"
}
}
]
},
"v4-binary-bytes-and-independent-identities": {
clients: 2,
steps: [
...online,
...offline,
{
type: "create-bytes",
client: 0,
path: "nested/blob.bin",
bytes: [0, 255, 254, 128, 13, 10]
},
{
type: "create-bytes",
client: 1,
path: "nested/blob.bin",
bytes: [0, 254, 255, 129, 10, 13]
},
...online,
{
type: "assert-consistent",
verify: (state) => {
state
.assertFileCount(2)
.assertBytes(
"nested/blob.bin",
new Uint8Array([0, 255, 254, 128, 13, 10])
);
state.assertBytes(
state.conflictPath("nested/blob.bin"),
new Uint8Array([0, 254, 255, 129, 10, 13])
);
}
}
]
},
"v4-three-file-cycle-preserves-identities": {
clients: 2,
steps: [
...online,
{ type: "create", client: 0, path: "a.md", content: "A" },
{ type: "create", client: 0, path: "b.md", content: "B" },
{ type: "create", client: 0, path: "c.md", content: "C" },
{ type: "barrier" },
{ type: "remember-identity", key: "a", path: "a.md" },
{ type: "remember-identity", key: "b", path: "b.md" },
{ type: "remember-identity", key: "c", path: "c.md" },
{ type: "disable-sync", client: 0 },
{ type: "delay-notifications", client: 0 },
{ type: "rename", client: 0, oldPath: "a.md", newPath: "temp.md" },
{ type: "rename", client: 0, oldPath: "b.md", newPath: "a.md" },
{ type: "rename", client: 0, oldPath: "c.md", newPath: "b.md" },
{ type: "rename", client: 0, oldPath: "temp.md", newPath: "c.md" },
{
type: "update",
client: 0,
path: "c.md",
content: "A edited offline"
},
{ type: "flush-notifications", client: 0 },
{ type: "enable-sync", client: 0 },
{ type: "barrier" },
{ type: "assert-identity", key: "a", path: "c.md" },
{ type: "assert-identity", key: "b", path: "a.md" },
{ type: "assert-identity", key: "c", path: "b.md" },
{
type: "assert-files",
count: 3,
expected: {
"a.md": "B",
"b.md": "C",
"c.md": "A edited offline"
},
absent: ["temp.md"]
}
]
},
"v4-case-alias-and-file-directory-conflicts": {
clients: 2,
steps: [
...online,
...offline,
{
type: "create",
client: 0,
path: "folder/note.md",
content: "nested file"
},
{
type: "create",
client: 0,
path: "README.md",
content: "uppercase file"
},
{
type: "create",
client: 1,
path: "folder",
content: "blocking file"
},
{
type: "create",
client: 1,
path: "readme.md",
content: "lowercase file"
},
...online,
{
type: "assert-consistent",
verify: (state) => {
state
.assertFileCount(4)
.assertContent("folder/note.md", "nested file")
.assertContent("README.md", "uppercase file");
state.assertAnyFileContains(
"blocking file",
"lowercase file"
);
}
},
{
type: "assert-markers",
markers: [
"nested file",
"blocking file",
"uppercase file",
"lowercase file"
]
}
]
},
"v4-server-sigkill-retains-acknowledged-state": {
clients: 2,
steps: [
...online,
{
type: "create",
client: 0,
path: "a.md",
content: "durable bytes"
},
{ type: "barrier" },
{ type: "remember-identity", key: "a", path: "a.md" },
...offline,
{ type: "crash-server" },
{ type: "restart-server" },
...online,
{ type: "assert-identity", key: "a", path: "a.md" },
{
type: "assert-files",
count: 1,
expected: { "a.md": "durable bytes" }
}
]
}
};
for (const kind of ["create", "content", "manifest"] as const) {
for (const point of ["before", "after"] as const) {
V4_TESTS[`v4-${kind}-${point}-commit-network-loss`] = {
clients: 2,
steps: [
...online,
...(kind === "content"
? ([
{
type: "create",
client: 0,
path: "file.md",
content: "old bytes"
},
{ type: "barrier" },
{
type: "remember-identity",
key: "updated-file",
path: "file.md"
}
] as TestStep[])
: []),
{ type: "drop-response", client: 0, kind, point },
{
type: kind === "content" ? "update" : "create",
client: 0,
path: "file.md",
content: "must survive retry"
},
{ type: "wait-for-response-drop", client: 0 },
{ type: "barrier" },
...(kind === "content"
? [
{
type: "assert-identity",
key: "updated-file",
path: "file.md"
} as TestStep
]
: []),
{
type: "assert-files",
count: 1,
expected: { "file.md": "must survive retry" }
}
]
};
}
}

1
frontend/deterministic-tests/tsconfig.json
View file

@ -2,6 +2,7 @@
"compilerOptions": {
"baseUrl": ".",
"strict": true,
"skipLibCheck": true,
"target": "ES2022",
"module": "CommonJS",
"esModuleInterop": true,

5
frontend/deterministic-tests/webpack.config.js
View file

@ -2,8 +2,9 @@ const path = require("path");
const webpack = require("webpack");
module.exports = {
entry: "./src/cli.ts",
entry: { cli: "./src/cli.ts", "crash-worker": "./src/crash-worker.ts" },
target: "node",
externals: { "sync-client": "commonjs sync-client" },
mode: "production",
optimization: {
minimize: false
@ -21,7 +22,7 @@ module.exports = {
},
output: {
globalObject: "this",
filename: "cli.js",
filename: "[name].js",
path: path.resolve(__dirname, "dist")
},
plugins: [

Some files were not shown because too many files have changed in this diff Show more