no power loss
This commit is contained in:
parent
d56084e4bc
commit
2fcdbfd27d
17 changed files with 1374 additions and 3676 deletions
|
|
@ -1,163 +1,52 @@
|
|||
# Shared sync harness support
|
||||
|
||||
This is test-only code, shared by `deterministic-tests` and `test-client` rather
|
||||
than exported as debugging machinery by the production sync client.
|
||||
Test-only code shared by the deterministic and seeded clients.
|
||||
|
||||
The journal suites also cover repeated saves replacing earlier unsynced text,
|
||||
tracked file/directory moves into ignored paths, and saves crossing the size
|
||||
limit. They check exact bytes and identities through recovery, including real
|
||||
server reads. Client and Rust unit suites bound deep-path validation resources.
|
||||
| Suite | Coverage |
|
||||
| --- | --- |
|
||||
| `harness.test.ts`, `oracle-regressions.test.ts` | Independent content/identity oracles, exercised faults, exclusive operations, persistence ownership, cleanup and seeds |
|
||||
| `file-application.test.ts` | Swaps, file/directory transitions, visible conflict files, live edits, exclusions, and one metadata save per application |
|
||||
| `recovery.test.ts` | Interrupt every application filesystem/save boundary, reopen metadata, modify disk, and verify stable ordinary scans; manifest policy and notification identity |
|
||||
| `engine-regressions.test.ts`, `sync-audit.test.ts` | Offline identities, unknown network outcomes, rejection/retry, exclusions, interrupted in-memory pagination, namespace races and lifecycle changes |
|
||||
| `event-delivery.test.ts`, `polling.test.ts` | WebSocket hints with HTTP-only event reads, missed notifications, opt-in polling, shutdown and retry |
|
||||
| `protocol.test.ts` | Abrupt worker termination after accepted requests, exact retries, real server SIGKILL, CAS and event ordering |
|
||||
| `restart-reconciliation.test.ts` | Interrupted file application followed by user edits, two-client/server convergence and a second restart |
|
||||
| `server-restore.test.ts` | History checkpoints, server backup restores and interrupted metadata resets |
|
||||
| `content-limits.test.ts`, `protocol-audit.test.ts` | Content validation and API invariants |
|
||||
| `real-disk.test.ts` | Host filesystem exclusive create/rename, unsafe links, and rename cycles |
|
||||
|
||||
| Layer | What it establishes |
|
||||
| ---------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `harness.test.ts` | Divergence/self-comparison detection, binary-safe comparisons, marker loss/duplication, cleanup failure propagation, deep persistence ownership, exclusive operations, exercised faults, seed reproducibility |
|
||||
| `oracle-regressions.test.ts` | Negative checks for regenerated retries, wrong fault kinds, duplicate clients, corrupt hashes, BOMs, path byte limits, idempotent unlink, directory durability, and complete identity/content expectations |
|
||||
| `recovery.test.ts` | Real engine journal recovery at every enumerated apply and replay mutation/save boundary; two interruptions followed by another power loss; identity, retained bytes and merge-policy regressions |
|
||||
| `engine-regressions.test.ts` | Recovery archives, offline notifications, rejection/retry, exclusions, reverted remote updates, lifecycle races and editor changes during staging/install |
|
||||
| `event-delivery.test.ts` | Real sync loop with checkpoint-controlled WS bursts before an iteration and during event application; no polling or extra wakeups to hide queued events |
|
||||
| `polling.test.ts` | Real sync loop with fake clocks: missed local/remote notifications, disabled controls, repeated polls, shutdown and network retries independent of polling |
|
||||
| `protocol.test.ts` | Terminated worker runtimes after accepted creates, existing-document Snapshot/Diff updates, and manifest requests, exact durable retries after newer remote heads and real server SIGKILL; concurrent CAS and ordered WS catchup |
|
||||
| `real-disk.test.ts` | Host filesystem exclusive create/rename, symlink/hardlink rejection and a real engine rename-cycle application |
|
||||
| Scripted suite | Existing multi-client regression scenarios plus v4 binary, identity, path-conflict, retry, and server-restart cases |
|
||||
| Seeded workloads | Concrete replay traces, offline and rapid online moves/edits, concurrent creates, exact deletion/overwrite exemptions, nested Unicode paths, delayed notifications and before/after network failures |
|
||||
The filesystem interruption tests assert the supported contract: complete metadata
|
||||
remains usable, scans observe the current files, and a second scan is stable.
|
||||
They do not demand the pre-interruption file contents, identities or a replay of
|
||||
unfinished operations. Normal-operation tests still check exact bytes and identities.
|
||||
There is no filesystem journal or hidden recovery archive.
|
||||
|
||||
`scripts/e2e-runner.test.mjs` also guards clean-install workspace/lockfile
|
||||
consistency, CI dependency installation and artifact upload wiring, subprocess
|
||||
exit codes, log failures, spawn failures, and timeouts.
|
||||
`MemoryDisk` exposes `before:`, `visible:` and `durable:` boundaries. A process
|
||||
interruption retains visible changes; a simulated power loss rolls unflushed
|
||||
changes back. This provides different partial-application states for testing; it
|
||||
is not an APFS/ext4 writeback model or a production durability requirement.
|
||||
`MemoryPersistence` atomically replaces complete objects and can reject before or
|
||||
after replacement. Abandoned filesystem sessions are fenced against later writes.
|
||||
|
||||
`MemoryDisk` distinguishes visible namespace/data from durable entries. Each
|
||||
mutation has `before:`, `visible:`, and `durable:` boundaries. `crash(false)`
|
||||
models process death with dirty kernel state retained; `crash(true)` discards
|
||||
unflushed state. Operations that return successfully have flushed their affected
|
||||
entries. Idempotent unlink flushes absent entries too, and durable directory
|
||||
removal makes all former descendants unreachable. This is a deliberately small atomic-operation model, not an emulation
|
||||
of APFS/ext4 writeback. A crashed session is fenced against subsequent writes.
|
||||
`crash-worker.ts` runs a complete client in a worker, which the parent terminates
|
||||
at accepted-response checkpoints without graceful shutdown. A new runtime receives
|
||||
the saved disk and metadata. The separate real-disk fixture uses a test-only native
|
||||
exclusive-rename helper and ordinary writes without flushes. It is not a production
|
||||
adapter or a sandbox against hostile concurrent changes to ancestor directories.
|
||||
|
||||
`MemoryPersistence` atomically saves complete, cloned objects. Failure before a
|
||||
save leaves the old object; failure after durable replacement leaves the new one.
|
||||
The journal sweep recreates the production database/file-operations objects after
|
||||
each fault. From each interrupted state it traces recovery and injects a second
|
||||
process death or power loss at every replay mutation/save boundary. Final recovery
|
||||
is replayed again and followed by another power loss. Cases cover swaps with new
|
||||
output, cross-directory moves with a surviving sibling, concurrent text merges,
|
||||
binary replacement, deletion and recovery archive cleanup. Exact visible bytes,
|
||||
UUID/path mappings, retained bytes and absence of transaction files are checked. An intent is resubmitted only if nothing was durably saved;
|
||||
resubmitting a partially recovered operation would hide journal defects.
|
||||
Scripted and seeded suites run multiple clients against real servers. They check
|
||||
visible bytes, UUID mappings, canonical server state, content heads and event cursors.
|
||||
Polling is disabled except in tests specifically exercising it. Faults must fire;
|
||||
missed injections fail cleanup rather than silently passing. Workload traces,
|
||||
per-worker results and logs are saved by `scripts/e2e.sh` for reproduction.
|
||||
|
||||
`crash-worker.ts` runs a complete SyncClient in a separate worker. The parent
|
||||
terminates it at an explicit accepted-response checkpoint without `destroy()`;
|
||||
no old timers, websocket handlers, or promises survive. Update cases bootstrap an existing UUID and assert the non-null parent and exact
|
||||
Snapshot/Diff request type before termination. Its saved memory-disk image and persisted database seed a fresh runtime. The separate real-disk fixture
|
||||
uses a test-only native exclusive-rename primitive. It is **not** a
|
||||
production adapter or an adversarial `openat` sandbox: concurrent hostile changes
|
||||
to ancestor directories and actual machine power cuts are outside its coverage.
|
||||
|
||||
## Regression coverage
|
||||
|
||||
The previously exposed WS queue, notified-rename UUID, uncertain-save replay,
|
||||
and cross-directory durability failures are fixed and remain asserted by the
|
||||
default harness. Scripted, seeded, event-delivery and engine regression clients
|
||||
explicitly disable polling so it cannot hide a missing wakeup. Crash workers also
|
||||
assert that this setting survives restart. Polling is enabled only in separate
|
||||
tests that intentionally omit notifications and advance a fake clock. Identity
|
||||
assertions distinguish documents even when bytes
|
||||
are equal. The engine regressions additionally exercise retained recovery bytes,
|
||||
durable offline notifications, permanently rejected requests, ignored namespace
|
||||
conflicts, reverted remote updates, and concurrent lifecycle changes.
|
||||
|
||||
Tests assert intended invariants, including bytes and UUID mappings, rather than
|
||||
accepting the observed output as an oracle. The memory fault model and real-disk
|
||||
checks complement one another; neither proves behavior under every real power cut.
|
||||
|
||||
## September 2026 audit regressions
|
||||
|
||||
`scripts/e2e.sh` now runs the Rust and client unit suites as well as the harness,
|
||||
real-server protocol checks, scripted scenarios, and seeded workloads. A failing
|
||||
unit suite makes the complete run fail. The audit tests are separate, individually
|
||||
quoted arguments in the default scripts, with a runner regression guarding that wiring.
|
||||
|
||||
The audit cases were first run with assertions for the required behavior and
|
||||
observed failing before the corresponding fixes. Coverage is organized by the
|
||||
original finding numbers:
|
||||
|
||||
| Findings | Permanent coverage and resulting guarantee |
|
||||
| -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| 1 | Engine and real-server lost-reply/rejected-retry tests preserve later content and rename undos; interrupted receipt saves and paged replay are tested after process replacement. |
|
||||
| 2, 6 | Ignored/oversized unmanaged occupants keep their paths, including arrivals after scanning; submitted manifests reserve excluded server paths and remain valid. |
|
||||
| 3, 12 | Oversized remote heads defer uploads; oversized local files are rejected before snapshot reads; scans retain hashes rather than encoded vault contents; content notifications cannot starve namespace scans. |
|
||||
| 4 | A subprocess deadline detects synchronous conflict-allocation hangs caused by long extensions. |
|
||||
| 5 | Persistent destination errors select a durable root fallback. The recovery sweep keeps that error active through every first and second crash boundary, checking exact bytes, identities and retained originals. |
|
||||
| 7–9 | Bootstrap notifications adopt remote identities; settings updates resume enabled engines, disabling sync persists across reset, and initial vault bindings survive restart; disabled startup recovers local journals. |
|
||||
| 10–11 | Fake-clock silent-socket recovery and real-server connection admission, slot release and handshake expiry. |
|
||||
| 13 | Rust tests bound event count and aggregate bytes per page; client tests fold remote reversions before application and preserve the replay position across restart. |
|
||||
| 14–15 | Metadata-only manifest reference validation and active-pool cleanup tests ensure unrelated payload decoding and checked-out transactions cannot block these operations. |
|
||||
| 16 | Dirty cursor updates invalidate precise positions; disconnect/reset clear presented cursors. |
|
||||
| 17 | Read-only configuration preservation and concurrent initialization verify that existing credentials are never rewritten or replaced. |
|
||||
|
||||
The follow-up audit added these independently reproduced failures. Each behavior
|
||||
was asserted before its fix; the default test commands include every regression.
|
||||
|
||||
| Additional failure | Regression coverage |
|
||||
| ------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
|
||||
| Delayed content receipts crash after local deletion | `sync-audit.test.ts` verifies receipt consumption, preserved deletion, and cursor advancement without a local path. |
|
||||
| Cursor reads restore presence after disconnect/reset | Cursor unit tests hold an active read and queue another update, retire the connection, then release both. |
|
||||
| Failed WebSocket upgrades strand clients with polling disabled | `polling.test.ts` requires exact remote content and rename catchup over HTTP after a disconnect signal. |
|
||||
| WebSocket construction errors escape startup/retry callbacks | Transport unit tests fail the first or second construction and require a successful subsequent connection. |
|
||||
| Opening a locked vault blocks unrelated vaults | Rust tests hold a real SQLite writer lock during lazy opening; another vault must remain readable. Concurrent openings must share the configured pool, including after cancellation. |
|
||||
| Failure clearing rejected requests leaves an enabled client stopped | Settings tests fail the second save, before and after commit, and require later notifications to wake synchronization. |
|
||||
| Caller mutations bypass the checked settings snapshot | A settings test changes the caller's vault name and exclusion array while transports pause; persisted settings must retain the original input. |
|
||||
| Cursor freshness uses metadata captured before an asynchronous read | A cursor test advances the document while the read is suspended and requires the old cursor to be marked outdated. |
|
||||
| Throwing/silent socket closes leave live handlers after shutdown | Transport tests require disconnected status and no message delivery after a failed or timed-out close. |
|
||||
| Handshake send errors escape the socket-open callback | A transport test drops the send during opening and requires safe retirement and retry. |
|
||||
| Old connection cleanup deletes its replacement's cursors | A real-server overlapping reconnect test preserves the replacement; Rust tests also reject delayed updates from the old session. |
|
||||
|
||||
These cases exercise transitions the earlier tests did not combine: a deletion
|
||||
between submission and acknowledgement, disconnects during reads, failures after
|
||||
the settings save, and two connections sharing a device identity. Happy-path
|
||||
reconnect and single-save fault tests do not establish those guarantees.
|
||||
|
||||
Replay pages contain at most 64 events and approximately 1 MiB of aggregate event
|
||||
JSON. One atomic manifest can exceed that budget; it is sent alone so replay still
|
||||
advances. `endEventId` identifies the page boundary while `headEventId` identifies
|
||||
the vault head. Deploy the updated client and server together: older clients
|
||||
require each batch to reach the vault head. Local partial replay folds retain only
|
||||
latest document heads, the latest manifest, and matching outstanding receipts;
|
||||
local files are reconciled only when catchup reaches the head.
|
||||
|
||||
Permanent destination failures use `Recovered <document ID><extension>` in the
|
||||
vault root, resolving any occupied name. The journal retains the original bytes
|
||||
when content is replaced. Permission or storage failures that also prevent that
|
||||
fallback remain explicit recovery failures with the journal intact.
|
||||
|
||||
A rejected retry retains an unconfirmed receipt until its request appears in the
|
||||
event log. Absence from one catchup is not proof that an earlier in-flight attempt
|
||||
cannot still commit. Receipt/artifact pruning remains an explicit operational concern; these tests do not claim
|
||||
formal completeness or simulate arbitrary physical-media corruption.
|
||||
|
||||
Reproduce the complete validation run from the repository root:
|
||||
Run from the repository root:
|
||||
|
||||
```sh
|
||||
E2E_ITERATIONS=100 E2E_SEED=8201 E2E_WORKERS=2 scripts/e2e.sh
|
||||
E2E_WORKERS=32 E2E_SEED=12 E2E_TIMEOUT_SECONDS=9000 E2E_ITERATIONS=100 scripts/e2e.sh
|
||||
```
|
||||
|
||||
## Sync regression coverage added after the September audit
|
||||
|
||||
The normal test commands include these regressions:
|
||||
|
||||
| Failure | Maintained coverage |
|
||||
| ------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| Ignored occupant renamed into scope; incoming download discarded by unrelated notifications | `journal-protocol.test.ts` checks visible files, original IDs and actual server content. |
|
||||
| Source path reused or moved during staging; editor save becomes a duplicate | `journal-interleavings.test.ts` injects moves, delete/recreate, move/recreate and content saves at every enumerated read, mutation and state-save boundary. Assertions specify bytes and document identities independently. |
|
||||
| Journal/editor interaction followed by a crash | `journal-protocol.test.ts` crosses three editor operations with failures before/after state persistence, process replacement and simulated power loss, then checks another reset. |
|
||||
| Restored server history falsely appears current | `server-restore.test.ts` restores actual SQLite backups and restarts Rust, covering equal/lower counters, dirty files, lost acknowledgements, interrupted recovery saves and old client state without checkpoints. |
|
||||
| Oversized remote fallback and stale responses | `content-limits.test.ts` measures payload downloads, checks metadata-only responses and verifies highly fragmented local edits fall back to snapshots. |
|
||||
| Adversarial diff work | Rust endpoint tests reject excessive operations without appending events and exercise Unicode reconstruction. |
|
||||
| Observer exception kills sync | `observer.test.ts` runs throwing/rejecting subscribers in a child process and verifies a second sync; listener unit tests verify continued delivery. |
|
||||
| Outgoing cursor version race | `outgoing-cursor.test.ts` changes version, identity, path and existence during a read, deliberately including identical bytes. |
|
||||
| Unpersisted credentials | Rust config tests reject both missing users and missing user lists while preserving existing file bytes; initialization tests verify stable saved defaults. |
|
||||
| Extensionless filenames enable merging | Extension unit tests include root/nested `md` and `txt` and dots in ancestor directories. |
|
||||
| Corrupt vault blocks startup | Rust database regression opens a healthy vault alongside an invalid SQLite file and verifies the invalid file remains untouched. |
|
||||
|
||||
The filesystem sweep reports the number of boundaries and injected interleavings.
|
||||
A passing sweep covers this adapter model and these operations; arbitrary physical
|
||||
media corruption and every possible editor/OS behavior remain outside its scope.
|
||||
The command rebuilds peers, runs unit/harness/protocol/scripted suites, and then
|
||||
runs seeds 12 through 43. `E2E_ARTIFACTS` optionally selects the artifact directory.
|
||||
The runner tests also verify subprocess exit codes, log errors, spawn failures,
|
||||
timeouts, workspace dependency consistency and CI artifact wiring.
|
||||
|
|
|
|||
|
|
@ -26,7 +26,7 @@ export async function getJson<T>(url: string, token: string): Promise<T> {
|
|||
}
|
||||
|
||||
/** Equality alone can pass when every replica loses the same bytes. Check the
|
||||
* canonical UUIDs, content heads, receipts, journals and event cursor as well.
|
||||
* canonical UUIDs, content heads, receipts and event cursor as well.
|
||||
* Scenario-specific content/identity oracles still run separately.
|
||||
*/
|
||||
export async function assertCanonical(
|
||||
|
|
|
|||
|
|
@ -1,170 +1,144 @@
|
|||
import assert from "node:assert/strict";
|
||||
import { PermanentSyncError } from "../sync-client/src/errors/errors";
|
||||
import { test } from "node:test";
|
||||
import {
|
||||
Database,
|
||||
emptyState,
|
||||
type StoredDatabase
|
||||
} from "../sync-client/src/persistence/database";
|
||||
import { Settings } from "../sync-client/src/persistence/settings";
|
||||
import { FileOperations } from "../sync-client/src/file-operations/file-operations";
|
||||
import { Database, emptyState } from "../sync-client/src/persistence/database";
|
||||
import { Syncer } from "../sync-client/src/sync-operations/syncer";
|
||||
import { SyncClient } from "../sync-client/src/sync-client";
|
||||
import { toStoredSnapshot } from "../sync-client/src/sync-operations/content";
|
||||
import { Logger } from "../sync-client/src/tracing/logger";
|
||||
import { SyncHistory } from "../sync-client/src/tracing/sync-history";
|
||||
import { FixedSizeDocumentCache } from "../sync-client/src/utils/data-structures/fix-sized-cache";
|
||||
import { EventListeners } from "../sync-client/src/utils/data-structures/event-listeners";
|
||||
import { allocatePortablePath } from "../sync-client/src/utils/portable-path";
|
||||
import type { ServerConfig } from "../sync-client/src/services/server-config";
|
||||
import type { SyncService } from "../sync-client/src/services/sync-service";
|
||||
import type { WebSocketManager } from "../sync-client/src/services/websocket-manager";
|
||||
import { MemoryDisk, MemoryPersistence } from "./storage";
|
||||
import type { LocalChange } from "../sync-client/src/sync-operations/local-changes";
|
||||
|
||||
const bytes = (text: string) => new TextEncoder().encode(text);
|
||||
const head = (id: string, version: number, content: string) => ({
|
||||
documentId: id,
|
||||
vaultUpdateId: version,
|
||||
contentSize: bytes(content).length,
|
||||
updatedDate: "2026-09-19T00:00:00Z",
|
||||
userId: "user",
|
||||
deviceId: "device"
|
||||
import { bytes, head, fixture } from "./sync-fixture";
|
||||
|
||||
test("offline content notifications need no saves and restart reads the current bytes", async () => {
|
||||
const disk = new MemoryDisk();
|
||||
await disk.userWrite("a.md", bytes("original"));
|
||||
const initial = emptyState(JSON.stringify(["http://offline.test", "test"]));
|
||||
initial.initialized = true;
|
||||
initial.local = { a: "a.md" };
|
||||
initial.documents.a = {
|
||||
materialized: true,
|
||||
observedHash: (await toStoredSnapshot({ content: bytes("original") }))
|
||||
.hash
|
||||
};
|
||||
const store = new MemoryPersistence({
|
||||
database: initial,
|
||||
settings: { remoteUri: "http://offline.test", vaultName: "test" }
|
||||
});
|
||||
let client = await SyncClient.create({ fs: disk, persistence: store });
|
||||
try {
|
||||
const saved = store.snapshot();
|
||||
store.boundary = () => {
|
||||
throw new Error("content notifications must not save");
|
||||
};
|
||||
for (let edit = 0; edit < 20; edit++) {
|
||||
await disk.userWrite("a.md", bytes(`edit ${edit}`));
|
||||
await client.syncLocallyUpdatedFile({ relativePath: "a.md" });
|
||||
}
|
||||
await client.destroy();
|
||||
assert.deepEqual(store.snapshot(), saved);
|
||||
store.boundary = () => {};
|
||||
// A final edit has no notification at all.
|
||||
await disk.userWrite("a.md", bytes("latest offline bytes"));
|
||||
client = await SyncClient.create({ fs: disk, persistence: store });
|
||||
const internals = client as unknown as {
|
||||
database: Database;
|
||||
syncer: {
|
||||
stopped: boolean;
|
||||
scan(): Promise<void>;
|
||||
preparePush(): Promise<boolean>;
|
||||
};
|
||||
};
|
||||
await internals.syncer.scan();
|
||||
internals.syncer.stopped = false;
|
||||
assert.equal(await internals.syncer.preparePush(), true);
|
||||
const pending = internals.database.state.pending;
|
||||
assert(pending?.type === "content");
|
||||
assert.equal(pending.documentId, "a");
|
||||
assert.equal(
|
||||
Buffer.from(pending.snapshot.contentBase64, "base64").toString(),
|
||||
"latest offline bytes"
|
||||
);
|
||||
} finally {
|
||||
store.boundary = () => {};
|
||||
await client.destroy();
|
||||
}
|
||||
});
|
||||
|
||||
async function fixture(
|
||||
contents: Record<string, string> = { "a.md": "local" },
|
||||
service: Partial<SyncService> = {}
|
||||
) {
|
||||
test("legacy content notifications preserve the acknowledged namespace prefix on restart", async () => {
|
||||
const disk = new MemoryDisk();
|
||||
const initial = emptyState("test");
|
||||
await disk.userWrite("d.md", bytes("A"));
|
||||
await disk.userWrite("b.md", bytes("B"));
|
||||
const initial = emptyState(JSON.stringify(["http://offline.test", "test"]));
|
||||
initial.initialized = true;
|
||||
for (const [path, content] of Object.entries(contents)) {
|
||||
await disk.userWrite(path, bytes(content));
|
||||
const id = path.split(".")[0];
|
||||
initial.local[id] = path;
|
||||
initial.local = { a: "c.md", b: "b.md" };
|
||||
initial.fileManifest.entries = { ...initial.local };
|
||||
for (const [id, content] of [
|
||||
["a", "A"],
|
||||
["b", "B"]
|
||||
])
|
||||
initial.documents[id] = {
|
||||
materialized: true,
|
||||
observedHash: (await toStoredSnapshot({ content: bytes(content) }))
|
||||
.hash
|
||||
};
|
||||
initial.lastAppliedLocalChangeId = "consumed-update";
|
||||
const saved = {
|
||||
database: initial,
|
||||
settings: { remoteUri: "http://offline.test", vaultName: "test" },
|
||||
localChangesVaultKey: initial.vaultKey,
|
||||
localChanges: [
|
||||
{
|
||||
type: "move",
|
||||
oldPath: "a.md",
|
||||
relativePath: "b.md",
|
||||
identities: { a: "a.md" },
|
||||
changeId: "consumed-move"
|
||||
},
|
||||
{ type: "update", path: "c.md", changeId: "consumed-update" },
|
||||
{ type: "update", path: "b.md", changeId: "new-update" },
|
||||
{
|
||||
type: "move",
|
||||
oldPath: "c.md",
|
||||
relativePath: "d.md",
|
||||
identities: { a: "c.md" },
|
||||
changeId: "new-move"
|
||||
}
|
||||
]
|
||||
};
|
||||
const store = new MemoryPersistence(saved);
|
||||
const client = await SyncClient.create({ fs: disk, persistence: store });
|
||||
try {
|
||||
const internals = client as unknown as {
|
||||
database: Database;
|
||||
syncer: { scan(): Promise<void> };
|
||||
};
|
||||
await internals.syncer.scan();
|
||||
assert.deepEqual(internals.database.state.local, {
|
||||
a: "d.md",
|
||||
b: "b.md"
|
||||
});
|
||||
assert.equal(
|
||||
internals.database.state.lastAppliedLocalChangeId,
|
||||
"new-move"
|
||||
);
|
||||
} finally {
|
||||
await client.destroy();
|
||||
}
|
||||
initial.fileManifest = { fileManifestId: 1, entries: { ...initial.local } };
|
||||
const persistence = new MemoryPersistence({ database: initial });
|
||||
const logger = new Logger();
|
||||
const database = new Database(
|
||||
logger,
|
||||
initial,
|
||||
async (next) => persistence.save({ database: next }),
|
||||
"test",
|
||||
async () => (await persistence.load()).database as StoredDatabase
|
||||
);
|
||||
const config = {
|
||||
initialize: async () => {},
|
||||
getConfig: async () => ({ mergeableFileExtensions: ["md"] }),
|
||||
reset: () => {}
|
||||
} as unknown as ServerConfig;
|
||||
const settings = new Settings(
|
||||
logger,
|
||||
{ syncIntervalMs: 0 },
|
||||
async () => {}
|
||||
);
|
||||
const changes: LocalChange[] = [];
|
||||
const files = new FileOperations(
|
||||
disk.session(),
|
||||
database,
|
||||
config,
|
||||
undefined,
|
||||
{ entries: () => changes, flush: async () => {} }
|
||||
);
|
||||
const websocket = {
|
||||
onWebSocketStatusChanged: new EventListeners(),
|
||||
onRemoteVaultUpdateReceived: new EventListeners()
|
||||
} as unknown as WebSocketManager;
|
||||
const syncer = new Syncer(
|
||||
"device",
|
||||
logger,
|
||||
database,
|
||||
settings,
|
||||
service as SyncService,
|
||||
websocket,
|
||||
files,
|
||||
config,
|
||||
new SyncHistory(logger),
|
||||
new FixedSizeDocumentCache(1024),
|
||||
{ entries: changes, save: async () => {} }
|
||||
);
|
||||
const internals = syncer as unknown as {
|
||||
finishPending(): Promise<void>;
|
||||
preparePush(): Promise<boolean>;
|
||||
scan(): Promise<void>;
|
||||
incorporateEventBatch(
|
||||
batch: import("../sync-client/src/services/types/EventBatch").EventBatch
|
||||
): Promise<void>;
|
||||
stopped: boolean;
|
||||
incorporateFileManifest(remote: {
|
||||
fileManifestId: number;
|
||||
entries: Record<string, string>;
|
||||
}): Promise<void>;
|
||||
incorporateContent(remote: ReturnType<typeof head>): Promise<void>;
|
||||
};
|
||||
return {
|
||||
disk,
|
||||
database,
|
||||
files,
|
||||
settings,
|
||||
syncer,
|
||||
internals,
|
||||
persistence,
|
||||
changes
|
||||
};
|
||||
}
|
||||
|
||||
test("remote binary replacement retains displaced bytes outside the visible namespace", async () => {
|
||||
const f = await fixture({ "a.bin": "private edit" });
|
||||
const next = structuredClone(f.database.state);
|
||||
await f.files.apply(next, {
|
||||
a: {
|
||||
expected: await f.files.snapshot("a.bin"),
|
||||
replacement: await toStoredSnapshot({ content: bytes("remote") })
|
||||
}
|
||||
});
|
||||
assert.deepEqual(f.disk.userFiles(), new Map([["a.bin", bytes("remote")]]));
|
||||
const retained = (await f.disk.listFilesRecursively()).filter((path) =>
|
||||
path.startsWith(".vault-link-sync/recovery/")
|
||||
);
|
||||
assert(
|
||||
(
|
||||
await Promise.all(retained.map((path) => f.disk.readSnapshot(path)))
|
||||
).some(
|
||||
(value) =>
|
||||
value &&
|
||||
Buffer.from(value.content).toString() === "private edit"
|
||||
)
|
||||
);
|
||||
await f.files.recover();
|
||||
assert.deepEqual(
|
||||
await f.disk.listFilesRecursively(),
|
||||
["a.bin", ...retained].sort()
|
||||
);
|
||||
});
|
||||
|
||||
test("remote deletion retains unsent bytes for recovery", async () => {
|
||||
const f = await fixture();
|
||||
test("remote binary replacement and deletion do not create backup artifacts", async () => {
|
||||
const f = await fixture({ "a.bin": "private edit" });
|
||||
await f.files.apply(structuredClone(f.database.state), {
|
||||
a: { replacement: await toStoredSnapshot({ content: bytes("remote") }) }
|
||||
});
|
||||
assert.deepEqual(await f.disk.listFilesRecursively(), ["a.bin"]);
|
||||
assert.deepEqual(f.disk.userFiles(), new Map([["a.bin", bytes("remote")]]));
|
||||
await f.internals.incorporateFileManifest({
|
||||
fileManifestId: 2,
|
||||
entries: {}
|
||||
});
|
||||
assert.equal(f.disk.userFiles().size, 0);
|
||||
const retained = await f.disk.listFilesRecursively();
|
||||
assert(
|
||||
(
|
||||
await Promise.all(retained.map((path) => f.disk.readSnapshot(path)))
|
||||
).some(
|
||||
(value) =>
|
||||
value && Buffer.from(value.content).toString() === "local"
|
||||
)
|
||||
);
|
||||
assert.deepEqual(await f.disk.listFilesRecursively(), []);
|
||||
});
|
||||
|
||||
test("rejected content recovery remains idempotent across a rename and network failure", async () => {
|
||||
|
|
@ -281,45 +255,6 @@ test("unfinished bootstrap cannot be rebound to another vault", async () => {
|
|||
}
|
||||
});
|
||||
|
||||
test("local swap arriving after a remote journal is saved preserves both identities and contents", async () => {
|
||||
const f = await fixture({ "a.md": "A", "b.md": "B" });
|
||||
let fired = false;
|
||||
f.persistence.boundary = async (label) => {
|
||||
if (
|
||||
fired ||
|
||||
label !== "durable:save" ||
|
||||
!f.persistence.snapshot().database?.application
|
||||
)
|
||||
return;
|
||||
fired = true;
|
||||
for (const [oldPath, relativePath] of [
|
||||
["a.md", "tmp.md"],
|
||||
["b.md", "a.md"],
|
||||
["tmp.md", "b.md"]
|
||||
]) {
|
||||
await f.disk.userRename(oldPath, relativePath);
|
||||
await f.syncer.syncLocallyUpdatedFile({ oldPath, relativePath });
|
||||
}
|
||||
};
|
||||
const next = structuredClone(f.database.state);
|
||||
next.local = { a: "remote-a.md", b: "remote-b.md" };
|
||||
await f.files.apply(next);
|
||||
assert(fired);
|
||||
assert.equal(
|
||||
Buffer.from(
|
||||
f.disk.userFiles().get(f.database.state.local.a)!
|
||||
).toString(),
|
||||
"A"
|
||||
);
|
||||
assert.equal(
|
||||
Buffer.from(
|
||||
f.disk.userFiles().get(f.database.state.local.b)!
|
||||
).toString(),
|
||||
"B"
|
||||
);
|
||||
assert.deepEqual(f.database.state.local, { a: "b.md", b: "a.md" });
|
||||
});
|
||||
|
||||
test("a rejected upload is retained locally and does not block another file or a corrected snapshot", async () => {
|
||||
let reject = true;
|
||||
const sent: string[] = [];
|
||||
|
|
@ -375,7 +310,7 @@ test("a transient upload failure retries the exact request and recovers its ackn
|
|||
assert.equal(f.database.state.pending, undefined);
|
||||
});
|
||||
|
||||
test("a persisted upload rejection finishes locally after a recovery write fails", async () => {
|
||||
test("a persisted upload rejection finishes without filesystem writes", async () => {
|
||||
let uploads = 0;
|
||||
const f = await fixture(undefined, {
|
||||
putFileContent: async () => {
|
||||
|
|
@ -385,29 +320,14 @@ test("a persisted upload rejection finishes locally after a recovery write fails
|
|||
});
|
||||
f.internals.stopped = false;
|
||||
await f.internals.preparePush();
|
||||
const { pending } = f.database.state;
|
||||
assert(pending);
|
||||
const { requestId } = pending.request;
|
||||
const retained = `.vault-link-sync/requests/${requestId}.content`;
|
||||
const { boundary } = f.disk;
|
||||
f.disk.boundary = (label): void => {
|
||||
if (label === `before:write:${retained}`)
|
||||
throw new Error("recovery write failed");
|
||||
f.disk.boundary = (label) => {
|
||||
if (!label.startsWith("read:"))
|
||||
throw new Error("Unexpected filesystem mutation");
|
||||
};
|
||||
await assert.rejects(f.internals.finishPending(), /recovery write failed/);
|
||||
assert.equal(
|
||||
f.persistence.snapshot().database?.pending?.rejection,
|
||||
"HTTP 413"
|
||||
);
|
||||
f.disk.boundary = boundary;
|
||||
await f.internals.finishPending();
|
||||
assert.equal(uploads, 1);
|
||||
assert.equal(f.database.state.pending, undefined);
|
||||
assert.equal(f.database.state.documents.a.rejected?.message, "HTTP 413");
|
||||
assert.deepEqual(
|
||||
(await f.disk.readSnapshot(retained))?.content,
|
||||
bytes("local")
|
||||
);
|
||||
assert.deepEqual(f.disk.userFiles(), new Map([["a.md", bytes("local")]]));
|
||||
});
|
||||
|
||||
|
|
@ -749,79 +669,6 @@ test("an incoming ignored document cannot force an existing ignored file to move
|
|||
);
|
||||
});
|
||||
|
||||
for (const boundary of ["visible", "durable"] as const) {
|
||||
for (const action of [
|
||||
"move-installed",
|
||||
"recreate-staged-source"
|
||||
] as const) {
|
||||
test(`${action} at a ${boundary} filesystem boundary keeps original identities and all bytes`, async () => {
|
||||
const f = await fixture({ "a.md": "A", "b.md": "B" });
|
||||
let changed = false;
|
||||
f.disk.boundary = async (label) => {
|
||||
if (changed || !label.startsWith(`${boundary}:rename:`)) return;
|
||||
if (action === "move-installed" && label.endsWith("->b.md")) {
|
||||
changed = true;
|
||||
await f.disk.userRename("b.md", "later.md");
|
||||
await f.syncer.syncLocallyUpdatedFile({
|
||||
oldPath: "b.md",
|
||||
relativePath: "later.md"
|
||||
});
|
||||
} else if (
|
||||
action === "recreate-staged-source" &&
|
||||
label.includes("rename:a.md->")
|
||||
) {
|
||||
changed = true;
|
||||
await f.disk.userWrite("a.md", bytes("new"));
|
||||
await f.syncer.syncLocallyCreatedFile("a.md");
|
||||
}
|
||||
};
|
||||
const next = structuredClone(f.database.state);
|
||||
next.local = { a: "b.md", b: "a.md" };
|
||||
next.fileManifest = {
|
||||
fileManifestId: 2,
|
||||
entries: { ...next.local }
|
||||
};
|
||||
await f.files.apply(next);
|
||||
assert(changed);
|
||||
assert.equal(
|
||||
Buffer.from(
|
||||
(await f.disk.readSnapshot(f.database.state.local.a))!
|
||||
.content
|
||||
).toString(),
|
||||
"A"
|
||||
);
|
||||
assert.equal(
|
||||
Buffer.from(
|
||||
(await f.disk.readSnapshot(f.database.state.local.b))!
|
||||
.content
|
||||
).toString(),
|
||||
"B"
|
||||
);
|
||||
assert.deepEqual(
|
||||
[...f.disk.userFiles().values()]
|
||||
.map((value) => Buffer.from(value).toString())
|
||||
.sort(),
|
||||
action === "move-installed" ? ["A", "B"] : ["A", "B", "new"]
|
||||
);
|
||||
f.disk.crash(true);
|
||||
assert.equal(
|
||||
Buffer.from(
|
||||
(await f.disk.readSnapshot(f.database.state.local.a))!
|
||||
.content
|
||||
).toString(),
|
||||
"A"
|
||||
);
|
||||
assert.equal(
|
||||
Buffer.from(
|
||||
(await f.disk.readSnapshot(f.database.state.local.b))!
|
||||
.content
|
||||
).toString(),
|
||||
"B"
|
||||
);
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
test("rejected content is reported to callers and an explicit retry clears the rejection", async () => {
|
||||
const f = await fixture(undefined, {
|
||||
putFileContent: async () => {
|
||||
|
|
|
|||
|
|
@ -21,11 +21,11 @@ import { MemoryDisk, MemoryPersistence } from "./storage";
|
|||
|
||||
for (const [count, delivery] of [
|
||||
[1, "before the loop"],
|
||||
[3, "before the loop"],
|
||||
[9, "before the loop"],
|
||||
[3, "during the first batch"]
|
||||
] as const) {
|
||||
test(
|
||||
`${count} WS batches delivered ${delivery} drain without polling or another notification`,
|
||||
`${count} WS hints delivered ${delivery} catch up through HTTP without polling`,
|
||||
{ timeout: 5_000 },
|
||||
async () => {
|
||||
const logger = new Logger();
|
||||
|
|
@ -33,7 +33,6 @@ for (const [count, delivery] of [
|
|||
initial.initialized = true;
|
||||
const persistence = new MemoryPersistence({ database: initial });
|
||||
const database = new Database(
|
||||
logger,
|
||||
initial,
|
||||
async (next) => persistence.save({ database: next }),
|
||||
initial.vaultKey,
|
||||
|
|
@ -85,13 +84,20 @@ for (const [count, delivery] of [
|
|||
fileManifest: { fileManifestId: index + 1, entries: {} }
|
||||
})
|
||||
);
|
||||
let available: number = delivery === "before the loop" ? count : 1;
|
||||
const fetchedAfter: number[] = [];
|
||||
const service = {
|
||||
// A correct client may drain the queue or catch up over HTTP. Either
|
||||
// must reach the same head without the test manually waking it.
|
||||
events: async (after: number): Promise<EventBatch> => ({
|
||||
headEventId: count,
|
||||
events: events.filter((event) => event.eventId > after)
|
||||
})
|
||||
events: async (after: number): Promise<EventBatch> => {
|
||||
fetchedAfter.push(after);
|
||||
return {
|
||||
headEventId: available,
|
||||
events: events.filter(
|
||||
(event) =>
|
||||
event.eventId > after &&
|
||||
event.eventId <= available
|
||||
)
|
||||
};
|
||||
}
|
||||
} as unknown as SyncService;
|
||||
const websocket = {
|
||||
onWebSocketStatusChanged: new EventListeners<
|
||||
|
|
@ -116,7 +122,15 @@ for (const [count, delivery] of [
|
|||
const deliver = async (event: EventRecord) => {
|
||||
await websocket.onRemoteVaultUpdateReceived.triggerAsync({
|
||||
headEventId: event.eventId,
|
||||
events: [event]
|
||||
// Payloads are deliberately inconsistent with HTTP. The socket
|
||||
// only wakes the loop; it cannot supply reconciliation state.
|
||||
events: [
|
||||
{
|
||||
...event,
|
||||
type: "fileManifest",
|
||||
fileManifest: { fileManifestId: 999, entries: {} }
|
||||
}
|
||||
]
|
||||
});
|
||||
};
|
||||
try {
|
||||
|
|
@ -129,6 +143,7 @@ for (const [count, delivery] of [
|
|||
await deliver(events[0]);
|
||||
initialized.resolve();
|
||||
await firstBatch.promise;
|
||||
available = count;
|
||||
for (const event of events.slice(1)) await deliver(event);
|
||||
continueBatch.resolve();
|
||||
}
|
||||
|
|
@ -140,11 +155,14 @@ for (const [count, delivery] of [
|
|||
);
|
||||
assert.deepEqual(
|
||||
applied,
|
||||
events.map((event) => event.eventId),
|
||||
"Every event must be committed in order"
|
||||
delivery === "before the loop" ? [count] : [1, count],
|
||||
"Each HTTP catchup commits its final cursor"
|
||||
);
|
||||
assert.deepEqual(
|
||||
fetchedAfter,
|
||||
delivery === "before the loop" ? [0] : [0, 1]
|
||||
);
|
||||
assert.equal(database.state.fileManifest.fileManifestId, count);
|
||||
assert.equal(database.state.application, undefined);
|
||||
assert.equal(database.state.pending, undefined);
|
||||
assert.equal(syncer.isBusy, false);
|
||||
} finally {
|
||||
|
|
|
|||
219
frontend/test-support/file-application.test.ts
Normal file
219
frontend/test-support/file-application.test.ts
Normal file
|
|
@ -0,0 +1,219 @@
|
|||
import assert from "node:assert/strict";
|
||||
import { test } from "node:test";
|
||||
import { fixture, bytes } from "./sync-fixture";
|
||||
import { toStoredSnapshot } from "../sync-client/src/sync-operations/content";
|
||||
import { LocalChangesDuringReconciliation } from "../sync-client/src/errors/errors";
|
||||
|
||||
for (const equal of [false, true])
|
||||
test(`swaps preserve identities with ${equal ? "equal" : "distinct"} content and save once`, async () => {
|
||||
const f = await fixture({ "a.md": "A", "b.md": equal ? "A" : "B" });
|
||||
let saves = 0;
|
||||
f.persistence.boundary = (label) => {
|
||||
if (label === "before:save") saves++;
|
||||
};
|
||||
const next = structuredClone(f.database.state);
|
||||
next.local = { a: "b.md", b: "a.md" };
|
||||
await f.files.apply(next);
|
||||
assert.deepEqual(f.database.state.local, next.local);
|
||||
assert.deepEqual(
|
||||
f.disk.userFiles(),
|
||||
new Map([
|
||||
["a.md", bytes(equal ? "A" : "B")],
|
||||
["b.md", bytes("A")]
|
||||
])
|
||||
);
|
||||
assert.equal(saves, 1);
|
||||
assert.deepEqual(await f.disk.listFilesRecursively(), ["a.md", "b.md"]);
|
||||
});
|
||||
|
||||
for (const direction of ["file-to-directory", "directory-to-file"])
|
||||
test(direction, async () => {
|
||||
const f = await fixture(
|
||||
direction === "file-to-directory"
|
||||
? { folder: "A", "b.md": "B" }
|
||||
: { "a.md": "A", "folder/b.md": "B" }
|
||||
);
|
||||
const ids = Object.keys(f.database.state.local);
|
||||
const next = structuredClone(f.database.state);
|
||||
next.local =
|
||||
direction === "file-to-directory"
|
||||
? { [ids[0]]: "folder/a.md", [ids[1]]: "b.md" }
|
||||
: { [ids[0]]: "folder", [ids[1]]: "other.md" };
|
||||
const expected = structuredClone(next.local);
|
||||
await f.files.apply(next);
|
||||
assert.deepEqual(f.database.state.local, expected);
|
||||
assert.deepEqual(
|
||||
new Set(f.disk.userFiles().keys()),
|
||||
new Set(Object.values(f.database.state.local))
|
||||
);
|
||||
assert.deepEqual(
|
||||
[...f.disk.userFiles().values()]
|
||||
.map((value) => Buffer.from(value).toString())
|
||||
.sort(),
|
||||
["A", "B"]
|
||||
);
|
||||
});
|
||||
|
||||
test("an unexpected occupant is preserved under a visible conflict name", async () => {
|
||||
const f = await fixture();
|
||||
await f.disk.userWrite("remote.md", bytes("unexpected"));
|
||||
const next = structuredClone(f.database.state);
|
||||
next.local.a = "remote.md";
|
||||
await f.files.apply(next);
|
||||
assert.equal(f.database.state.local.a, "remote.md");
|
||||
assert.equal(f.disk.userFiles().size, 2);
|
||||
assert.deepEqual(
|
||||
[...f.disk.userFiles().values()]
|
||||
.map((value) => Buffer.from(value).toString())
|
||||
.sort(),
|
||||
["local", "unexpected"]
|
||||
);
|
||||
assert.equal(Object.keys(f.database.state.local).length, 2);
|
||||
});
|
||||
|
||||
test("content edited after planning is merged with the incoming snapshot", async () => {
|
||||
const f = await fixture({ "a.md": "base\n" });
|
||||
const expected = (await f.files.snapshot("a.md"))!;
|
||||
await f.disk.userWrite("a.md", bytes("base\nlocal\n"));
|
||||
await f.files.apply(structuredClone(f.database.state), {
|
||||
a: {
|
||||
expected,
|
||||
replacement: await toStoredSnapshot({
|
||||
content: bytes("remote\nbase\n")
|
||||
})
|
||||
}
|
||||
});
|
||||
assert.deepEqual(
|
||||
f.disk.userFiles().get("a.md"),
|
||||
bytes("remote\nbase\nlocal\n")
|
||||
);
|
||||
});
|
||||
|
||||
test("deleting the source after planning does not resurrect it", async () => {
|
||||
const f = await fixture();
|
||||
const next = structuredClone(f.database.state);
|
||||
next.local.a = "remote.md";
|
||||
await f.disk.userDelete("a.md");
|
||||
await f.files.apply(next, {
|
||||
a: { replacement: await toStoredSnapshot({ content: bytes("remote") }) }
|
||||
});
|
||||
assert.deepEqual(f.disk.userFiles(), new Map());
|
||||
assert.deepEqual(f.database.state.local, {});
|
||||
});
|
||||
|
||||
for (const protect of ["ignore", "size"] as const)
|
||||
test(`protect ${protect} at application time`, async () => {
|
||||
const f = await fixture();
|
||||
const next = structuredClone(f.database.state);
|
||||
next.local.a = "remote.md";
|
||||
if (protect === "ignore")
|
||||
await f.settings.setSetting("ignorePatterns", ["a.md"]);
|
||||
else await f.settings.setSetting("maxFileSizeMB", 0);
|
||||
await f.files.apply(next, {
|
||||
a: {
|
||||
replacement: await toStoredSnapshot({
|
||||
content: bytes("remote")
|
||||
})
|
||||
}
|
||||
});
|
||||
assert.deepEqual(
|
||||
f.disk.userFiles(),
|
||||
new Map([["a.md", bytes("local")]])
|
||||
);
|
||||
assert.equal(f.database.state.local.a, "a.md");
|
||||
assert.equal(f.database.state.excluded?.a, "a.md");
|
||||
});
|
||||
|
||||
for (const when of ["planning", "installed"])
|
||||
test(`notified move during ${when} is attributed to the right identity on rescan`, async () => {
|
||||
const f = await fixture({ "a.md": "A", "b.md": "B" });
|
||||
let changed = false;
|
||||
f.disk.boundary = async (label) => {
|
||||
if (
|
||||
changed ||
|
||||
label !==
|
||||
(when === "planning"
|
||||
? "read:stat:a.md"
|
||||
: "visible:rename:a.md->b.md")
|
||||
)
|
||||
return;
|
||||
changed = true;
|
||||
const oldPath = when === "planning" ? "a.md" : "b.md";
|
||||
await f.disk.userRename(oldPath, "editor.md");
|
||||
await f.syncer.syncLocallyUpdatedFile({
|
||||
oldPath,
|
||||
relativePath: "editor.md"
|
||||
});
|
||||
};
|
||||
const next = structuredClone(f.database.state);
|
||||
next.local = { a: "b.md", b: "a.md" };
|
||||
if (when === "planning")
|
||||
await assert.rejects(
|
||||
f.files.apply(next),
|
||||
LocalChangesDuringReconciliation
|
||||
);
|
||||
else await f.files.apply(next);
|
||||
assert(changed);
|
||||
await f.internals.scan();
|
||||
assert.equal(f.database.state.local.a, "editor.md");
|
||||
assert.deepEqual(
|
||||
f.disk.userFiles().get(f.database.state.local.a),
|
||||
bytes("A")
|
||||
);
|
||||
assert.deepEqual(
|
||||
f.disk.userFiles().get(f.database.state.local.b),
|
||||
bytes("B")
|
||||
);
|
||||
});
|
||||
|
||||
for (const action of ["move", "delete"] as const)
|
||||
test(`notified ${action}/recreate between file operations does not repurpose the replacement file`, async () => {
|
||||
const f = await fixture({ "a.md": "A", "b.md": "B" });
|
||||
await f.settings.setSetting("ignorePatterns", ["private.md"]);
|
||||
let changed = false;
|
||||
f.disk.boundary = async (label) => {
|
||||
if (changed || label !== "visible:rename:a.md->remote-a.md") return;
|
||||
changed = true;
|
||||
if (action === "move") {
|
||||
await f.disk.userRename("b.md", "private.md");
|
||||
await f.syncer.syncLocallyUpdatedFile({
|
||||
oldPath: "b.md",
|
||||
relativePath: "private.md"
|
||||
});
|
||||
} else {
|
||||
await f.disk.userDelete("b.md");
|
||||
await f.syncer.syncLocallyDeletedFile("b.md");
|
||||
}
|
||||
await f.disk.userWrite("b.md", bytes("replacement"));
|
||||
await f.syncer.syncLocallyCreatedFile("b.md");
|
||||
};
|
||||
const next = structuredClone(f.database.state);
|
||||
next.local = { a: "remote-a.md", b: "remote-b.md" };
|
||||
await f.files.apply(next);
|
||||
assert(changed);
|
||||
await f.internals.scan();
|
||||
assert.deepEqual(f.disk.userFiles().get("b.md"), bytes("replacement"));
|
||||
assert.equal(f.disk.userFiles().has("remote-b.md"), false);
|
||||
assert.equal(
|
||||
f.database.state.local.b,
|
||||
action === "move" ? "private.md" : undefined
|
||||
);
|
||||
if (action === "move")
|
||||
assert.deepEqual(f.disk.userFiles().get("private.md"), bytes("B"));
|
||||
});
|
||||
|
||||
test("a file and nested directory exchange their exact paths", async () => {
|
||||
const f = await fixture({ a: "A", "b/deep/child": "B" });
|
||||
const [a, b] = Object.keys(f.database.state.local);
|
||||
const next = structuredClone(f.database.state);
|
||||
next.local = { [a]: "b", [b]: "a/deep/child" };
|
||||
await f.files.apply(next);
|
||||
assert.deepEqual(f.database.state.local, { [a]: "b", [b]: "a/deep/child" });
|
||||
assert.deepEqual(
|
||||
f.disk.userFiles(),
|
||||
new Map([
|
||||
["b", bytes("A")],
|
||||
["a/deep/child", bytes("B")]
|
||||
])
|
||||
);
|
||||
});
|
||||
|
|
@ -1,407 +0,0 @@
|
|||
import assert from "node:assert/strict";
|
||||
import { test } from "node:test";
|
||||
import { fixture, bytes, head } from "./sync-fixture";
|
||||
import { toStoredSnapshot } from "../sync-client/src/sync-operations/content";
|
||||
|
||||
for (const origin of ["bootstrap", "history recovery"])
|
||||
test(`cancelling a replacement retains the ${origin} merge context`, async () => {
|
||||
const f = await fixture(
|
||||
{ "a.md": "BASE\n" },
|
||||
{ getDocumentVersionContent: async () => bytes("REMOTE\nBASE\n") }
|
||||
);
|
||||
const recoveryBase = await toStoredSnapshot({
|
||||
content: bytes("BASE\n")
|
||||
});
|
||||
if (origin === "bootstrap")
|
||||
f.database.state.documents.a.bootstrap = true;
|
||||
else f.database.state.documents.a.recoveryBase = recoveryBase;
|
||||
await f.database.save();
|
||||
await f.settings.setSetting("ignorePatterns", ["private.md"]);
|
||||
let injected = false;
|
||||
f.disk.boundary = async (label) => {
|
||||
if (
|
||||
injected ||
|
||||
!label.startsWith("before:mkdir:.vault-link-sync/transactions/")
|
||||
)
|
||||
return;
|
||||
injected = true;
|
||||
await f.disk.userRename("a.md", "private.md");
|
||||
await f.disk.userWrite("private.md", bytes("BASE\nLOCAL\n"));
|
||||
await f.syncer.syncLocallyUpdatedFile({
|
||||
oldPath: "a.md",
|
||||
relativePath: "private.md"
|
||||
});
|
||||
};
|
||||
await f.internals.incorporateContent(head("a", 2, "REMOTE\nBASE\n"));
|
||||
assert(injected);
|
||||
const deferred = f.database.state.documents.a;
|
||||
assert.equal(deferred.base, undefined);
|
||||
if (origin === "bootstrap") assert.equal(deferred.bootstrap, true);
|
||||
else assert.deepEqual(deferred.recoveryBase, recoveryBase);
|
||||
f.disk.boundary = () => {};
|
||||
await f.settings.setSetting("ignorePatterns", []);
|
||||
await f.internals.incorporateFileManifest(
|
||||
f.database.state.fileManifest
|
||||
);
|
||||
const content = Buffer.from(
|
||||
f.disk.userFiles().get("private.md")!
|
||||
).toString();
|
||||
assert.equal(content.match(/REMOTE/g)?.length, 1);
|
||||
assert.equal(content.match(/LOCAL/g)?.length, 1);
|
||||
assert.equal(f.database.state.documents.a.base?.vaultUpdateId, 2);
|
||||
assert.equal(f.database.state.documents.a.recoveryBase, undefined);
|
||||
});
|
||||
|
||||
for (const timing of ["before", "durable"])
|
||||
test(`a save that exceeds the size limit ${timing} staging keeps its exact bytes`, async () => {
|
||||
const f = await fixture({ "a.bin": "PUBLIC" });
|
||||
await f.settings.setSetting("maxFileSizeMB", 100 / (1024 * 1024));
|
||||
const large = "PRIVATE".repeat(100);
|
||||
let injected = false;
|
||||
f.disk.boundary = async (label) => {
|
||||
if (
|
||||
injected ||
|
||||
!label.startsWith(
|
||||
`${timing}:rename:a.bin->.vault-link-sync/transactions/`
|
||||
)
|
||||
)
|
||||
return;
|
||||
injected = true;
|
||||
await f.disk.userWrite("a.bin", bytes(large));
|
||||
await f.syncer.syncLocallyUpdatedFile({ relativePath: "a.bin" });
|
||||
};
|
||||
await f.files.apply(structuredClone(f.database.state), {
|
||||
a: {
|
||||
expected: await toStoredSnapshot({ content: bytes("PUBLIC") }),
|
||||
replacement: await toStoredSnapshot({
|
||||
content: bytes("REMOTE")
|
||||
})
|
||||
}
|
||||
});
|
||||
assert(injected);
|
||||
assert.equal(
|
||||
Buffer.from(f.disk.userFiles().get("a.bin")!).toString(),
|
||||
large
|
||||
);
|
||||
assert.equal(f.database.state.excluded?.a, "a.bin");
|
||||
assert.deepEqual(f.database.state.local, { a: "a.bin" });
|
||||
});
|
||||
|
||||
test("an excluded move after installation survives an uncertain journal acknowledgement", async () => {
|
||||
const f = await fixture({ "a.md": "PUBLIC" });
|
||||
await f.settings.setSetting("ignorePatterns", ["private.md"]);
|
||||
let moved = false,
|
||||
crashed = false;
|
||||
f.disk.boundary = async (label) => {
|
||||
if (
|
||||
moved ||
|
||||
!label.startsWith("durable:rename:") ||
|
||||
!label.endsWith(".output->a.md")
|
||||
)
|
||||
return;
|
||||
moved = true;
|
||||
await f.disk.userRename("a.md", "private.md");
|
||||
await f.disk.userWrite("private.md", bytes("SECRET"));
|
||||
await f.syncer.syncLocallyUpdatedFile({
|
||||
oldPath: "a.md",
|
||||
relativePath: "private.md"
|
||||
});
|
||||
};
|
||||
f.persistence.boundary = (label) => {
|
||||
if (!moved || crashed || label !== "durable:save") return;
|
||||
crashed = true;
|
||||
throw new Error("lost journal acknowledgement");
|
||||
};
|
||||
await assert.rejects(
|
||||
f.files.apply(structuredClone(f.database.state), {
|
||||
a: {
|
||||
expected: await toStoredSnapshot({ content: bytes("PUBLIC") }),
|
||||
replacement: await toStoredSnapshot({
|
||||
content: bytes("PUBLIC REMOTE")
|
||||
})
|
||||
}
|
||||
}),
|
||||
/lost journal acknowledgement/
|
||||
);
|
||||
assert(moved && crashed);
|
||||
f.disk.boundary = () => {};
|
||||
f.persistence.boundary = () => {};
|
||||
await f.database.recoverPersistence();
|
||||
await f.files.recover();
|
||||
assert.deepEqual(f.database.state.local, { a: "private.md" });
|
||||
assert.equal(
|
||||
Buffer.from(f.disk.userFiles().get("private.md")!).toString(),
|
||||
"SECRET"
|
||||
);
|
||||
});
|
||||
|
||||
test("moving a tracked file into an ignored path during a journal leaves it protected", async () => {
|
||||
const f = await fixture({ "a.md": "PUBLIC", "b.md": "OTHER" });
|
||||
await f.settings.setSetting("ignorePatterns", ["private.md"]);
|
||||
const next = structuredClone(f.database.state);
|
||||
next.local.b = "remote-b.md";
|
||||
let injected = false;
|
||||
f.disk.boundary = async (label) => {
|
||||
if (
|
||||
injected ||
|
||||
!label.startsWith("before:mkdir:.vault-link-sync/transactions/")
|
||||
)
|
||||
return;
|
||||
injected = true;
|
||||
await f.disk.userRename("a.md", "private.md");
|
||||
await f.disk.userWrite("private.md", bytes("SECRET"));
|
||||
await f.syncer.syncLocallyUpdatedFile({
|
||||
oldPath: "a.md",
|
||||
relativePath: "private.md"
|
||||
});
|
||||
};
|
||||
await f.files.apply(next);
|
||||
assert(injected);
|
||||
assert.equal(
|
||||
Buffer.from(f.disk.userFiles().get("private.md") ?? []).toString(),
|
||||
"SECRET"
|
||||
);
|
||||
assert.equal(f.database.state.local.a, "private.md");
|
||||
assert.equal(f.database.state.excluded?.a, "a.md");
|
||||
});
|
||||
|
||||
test("a directory move protects each ignored descendant during a journal", async () => {
|
||||
const f = await fixture({ "folder/a.md": "PUBLIC", "b.md": "OTHER" });
|
||||
await f.settings.setSetting("ignorePatterns", ["private/**"]);
|
||||
const next = structuredClone(f.database.state);
|
||||
next.local.b = "remote-b.md";
|
||||
let injected = false;
|
||||
f.disk.boundary = async (label) => {
|
||||
if (
|
||||
injected ||
|
||||
!label.startsWith("before:mkdir:.vault-link-sync/transactions/")
|
||||
)
|
||||
return;
|
||||
injected = true;
|
||||
await f.disk.userRename("folder/a.md", "private/a.md");
|
||||
await f.disk.userWrite("private/a.md", bytes("SECRET"));
|
||||
await f.syncer.syncLocallyUpdatedFile({
|
||||
oldPath: "folder",
|
||||
relativePath: "private"
|
||||
});
|
||||
};
|
||||
await f.files.apply(next);
|
||||
assert(injected);
|
||||
assert.equal(
|
||||
Buffer.from(f.disk.userFiles().get("private/a.md") ?? []).toString(),
|
||||
"SECRET"
|
||||
);
|
||||
assert.equal(f.database.state.local["folder/a"], "private/a.md");
|
||||
assert.equal(f.database.state.excluded?.["folder/a"], "folder/a.md");
|
||||
});
|
||||
|
||||
test("a later editor save replaces earlier unsynced text after a merge was prepared", async () => {
|
||||
const f = await fixture({ "a.md": "base" });
|
||||
const next = structuredClone(f.database.state);
|
||||
let saves = 0;
|
||||
f.disk.boundary = async (label) => {
|
||||
if (saves === 0 && label.startsWith("before:rename:a.md->")) {
|
||||
saves++;
|
||||
await f.disk.userWrite("a.md", bytes("base ONE"));
|
||||
await f.syncer.syncLocallyUpdatedFile({ relativePath: "a.md" });
|
||||
} else if (
|
||||
saves === 1 &&
|
||||
label.startsWith("before:write:") &&
|
||||
label.endsWith(".output")
|
||||
) {
|
||||
saves++;
|
||||
await f.disk.userWrite("a.md", bytes("base TWO"));
|
||||
await f.syncer.syncLocallyUpdatedFile({ relativePath: "a.md" });
|
||||
}
|
||||
};
|
||||
await f.files.apply(next, {
|
||||
a: {
|
||||
expected: await toStoredSnapshot({ content: bytes("base") }),
|
||||
replacement: await toStoredSnapshot({
|
||||
content: bytes("base REMOTE")
|
||||
})
|
||||
}
|
||||
});
|
||||
assert.equal(saves, 2);
|
||||
assert.equal(
|
||||
Buffer.from(f.disk.userFiles().get("a.md")!).toString(),
|
||||
"base TWO REMOTE"
|
||||
);
|
||||
assert.deepEqual(f.database.state.local, { a: "a.md" });
|
||||
});
|
||||
|
||||
test("a content save during remote application remains on the same document", async () => {
|
||||
const f = await fixture({ "a.md": "base" });
|
||||
f.database.state.documents.a.base = {
|
||||
...head("a", 1, "base"),
|
||||
hash: f.database.state.documents.a.observedHash!
|
||||
};
|
||||
const next = structuredClone(f.database.state);
|
||||
next.documents.a.base = {
|
||||
...head("a", 2, "base REMOTE"),
|
||||
hash: (await toStoredSnapshot({ content: bytes("base REMOTE") })).hash
|
||||
};
|
||||
let injected = false;
|
||||
f.disk.boundary = async (label) => {
|
||||
if (
|
||||
injected ||
|
||||
!label.startsWith(
|
||||
"durable:rename:a.md->.vault-link-sync/transactions/"
|
||||
)
|
||||
)
|
||||
return;
|
||||
injected = true;
|
||||
await f.disk.userWrite("a.md", bytes("base LOCAL"));
|
||||
await f.syncer.syncLocallyUpdatedFile({ relativePath: "a.md" });
|
||||
};
|
||||
await f.files.apply(next, {
|
||||
a: {
|
||||
expected: await toStoredSnapshot({ content: bytes("base") }),
|
||||
replacement: await toStoredSnapshot({
|
||||
content: bytes("base REMOTE")
|
||||
})
|
||||
}
|
||||
});
|
||||
|
||||
assert.equal(f.disk.userFiles().size, 1);
|
||||
const result = Buffer.from(
|
||||
(await f.disk.readSnapshot("a.md"))!.content
|
||||
).toString();
|
||||
assert(result.includes("LOCAL") && result.includes("REMOTE"));
|
||||
});
|
||||
|
||||
async function setup() {
|
||||
const f = await fixture({ "a.md": "ORIGINAL", "b.md": "OTHER" });
|
||||
f.database.state.documents.a.base = {
|
||||
...head("a", 1, "ORIGINAL"),
|
||||
hash: f.database.state.documents.a.observedHash!
|
||||
};
|
||||
const next = structuredClone(f.database.state);
|
||||
next.local = { a: "remote.md", b: "remote-b.md" };
|
||||
next.fileManifest = { fileManifestId: 3, entries: { ...next.local } };
|
||||
const expected = await toStoredSnapshot({ content: bytes("ORIGINAL") });
|
||||
const replacement = await toStoredSnapshot({
|
||||
content: bytes("ORIGINAL REMOTE")
|
||||
});
|
||||
next.documents.a.base = {
|
||||
...head("a", 2, "ORIGINAL REMOTE"),
|
||||
hash: replacement.hash
|
||||
};
|
||||
return { f, next, writes: { a: { expected, replacement } } };
|
||||
}
|
||||
test("editor changes preserve exact bytes and identities at every filesystem read, mutation and persistence boundary", async (t) => {
|
||||
const { f, next, writes } = await setup();
|
||||
const labels: string[] = [];
|
||||
const trace = (l: string) => {
|
||||
labels.push(l);
|
||||
};
|
||||
f.disk.boundary = trace;
|
||||
f.persistence.boundary = trace;
|
||||
await f.files.apply(next, writes);
|
||||
const failures = [];
|
||||
let exercised = 0;
|
||||
for (const kind of ["move", "move-recreate", "delete-recreate", "save"])
|
||||
for (let target = 0; target < labels.length; target++) {
|
||||
const { f, next, writes } = await setup();
|
||||
let count = 0,
|
||||
injected = false,
|
||||
from: string | undefined;
|
||||
const boundary = async (l: string) => {
|
||||
if (injected) return;
|
||||
if (count++ !== target) return;
|
||||
const files = f.disk.userFiles();
|
||||
from = files.has("a.md")
|
||||
? "a.md"
|
||||
: files.has("remote.md")
|
||||
? "remote.md"
|
||||
: undefined;
|
||||
if (kind === "save" && !from) from = "a.md";
|
||||
if (!from) return;
|
||||
injected = true;
|
||||
exercised++;
|
||||
if (kind === "save") {
|
||||
await f.disk.userWrite(
|
||||
from,
|
||||
bytes(
|
||||
Buffer.from(
|
||||
files.get(from) ?? bytes("ORIGINAL")
|
||||
).toString() + " LOCAL"
|
||||
)
|
||||
);
|
||||
await f.syncer.syncLocallyUpdatedFile({
|
||||
relativePath: from
|
||||
});
|
||||
} else if (kind === "delete-recreate") {
|
||||
await f.disk.userDelete(from);
|
||||
await f.syncer.syncLocallyDeletedFile(from);
|
||||
} else {
|
||||
await f.disk.userRename(from, "mine.md");
|
||||
await f.syncer.syncLocallyUpdatedFile({
|
||||
oldPath: from,
|
||||
relativePath: "mine.md"
|
||||
});
|
||||
}
|
||||
if (kind !== "move" && kind !== "save") {
|
||||
await f.disk.userWrite(from, bytes("REPLACEMENT"));
|
||||
await f.syncer.syncLocallyCreatedFile(from);
|
||||
}
|
||||
};
|
||||
f.disk.boundary = boundary;
|
||||
f.persistence.boundary = boundary;
|
||||
let error;
|
||||
try {
|
||||
await f.files.apply(next, writes);
|
||||
} catch (e) {
|
||||
error = String(e);
|
||||
}
|
||||
f.disk.boundary = () => {};
|
||||
f.persistence.boundary = () => {};
|
||||
try {
|
||||
await f.database.recoverPersistence();
|
||||
await f.files.recover();
|
||||
await f.internals.scan();
|
||||
} catch (e) {
|
||||
error += ";retry=" + String(e);
|
||||
}
|
||||
if (!injected) continue;
|
||||
const fs = Object.fromEntries(
|
||||
[...f.disk.userFiles()].map(([p, b]) => [
|
||||
p,
|
||||
Buffer.from(b).toString()
|
||||
])
|
||||
);
|
||||
const bad =
|
||||
kind === "save"
|
||||
? ![
|
||||
"ORIGINAL LOCAL REMOTE",
|
||||
"ORIGINAL REMOTE LOCAL"
|
||||
].includes(fs[f.database.state.local.a]) ||
|
||||
Object.keys(f.database.state.local).length !== 2
|
||||
: kind === "delete-recreate"
|
||||
? f.database.state.local.a !== undefined
|
||||
: f.database.state.local.a !== "mine.md" ||
|
||||
fs["mine.md"] !== "ORIGINAL REMOTE";
|
||||
if (
|
||||
bad ||
|
||||
(kind !== "move" &&
|
||||
kind !== "save" &&
|
||||
fs[from!] !== "REPLACEMENT")
|
||||
)
|
||||
failures.push({
|
||||
kind,
|
||||
target,
|
||||
label: labels[target],
|
||||
error,
|
||||
ids: f.database.state.local,
|
||||
files: fs
|
||||
});
|
||||
}
|
||||
assert(
|
||||
exercised > 50,
|
||||
"The sweep must actually exercise the injected actions"
|
||||
);
|
||||
t.diagnostic(
|
||||
`${labels.length} boundaries; ${exercised} editor interleavings`
|
||||
);
|
||||
assert.deepEqual(failures, []);
|
||||
});
|
||||
|
|
@ -1,612 +0,0 @@
|
|||
import assert from "node:assert/strict";
|
||||
import { test } from "node:test";
|
||||
import { fixture, bytes, randomUUID } from "./sync-server-fixture";
|
||||
|
||||
for (const exclusion of ["size", "ignore"] as const)
|
||||
for (const phase of [
|
||||
"planned",
|
||||
...(exclusion === "size" ? ["staged", "prepared"] : []),
|
||||
"installed"
|
||||
])
|
||||
for (const recovery of [
|
||||
"none",
|
||||
"restart",
|
||||
"before:save",
|
||||
"durable:save"
|
||||
] as const)
|
||||
test(`reenabling ${exclusion} exclusion at ${phase} preserves the content base through ${recovery}`, async () => {
|
||||
const f = await fixture();
|
||||
try {
|
||||
const installed = phase === "installed";
|
||||
const id = randomUUID();
|
||||
const base = await f.put(
|
||||
"/documents/" + id,
|
||||
f.content("BASE\n")
|
||||
);
|
||||
await f.put("/file-manifest", {
|
||||
requestId: randomUUID(),
|
||||
parentFileManifestId: 0,
|
||||
entries: { [id]: "a.md" }
|
||||
});
|
||||
await f.client.start();
|
||||
await f.client.setSetting("isSyncEnabled", false);
|
||||
await f.client.setSettings({
|
||||
maxFileSizeMB:
|
||||
exclusion === "size" ? 100 / (1024 * 1024) : 1,
|
||||
ignorePatterns: ["private.md"]
|
||||
});
|
||||
const remote = await f.put(
|
||||
"/documents/" + id,
|
||||
f.content("REMOTE\nBASE\n", base.vaultUpdateId)
|
||||
);
|
||||
const local = "BASE\n" + "LOCAL\n".repeat(30);
|
||||
const path = exclusion === "size" ? "a.md" : "private.md";
|
||||
let injected = false,
|
||||
armed = false,
|
||||
crashed = false;
|
||||
f.disk.boundary = async (label) => {
|
||||
const atBoundary =
|
||||
phase === "installed"
|
||||
? label.startsWith("durable:rename:") &&
|
||||
label.endsWith(".output->a.md")
|
||||
: phase === "staged"
|
||||
? label.startsWith("durable:rename:a.md->")
|
||||
: phase === "prepared"
|
||||
? label.startsWith("before:write:") &&
|
||||
label.endsWith(".output")
|
||||
: label.startsWith(
|
||||
"before:mkdir:.vault-link-sync/transactions/"
|
||||
);
|
||||
if (injected || !atBoundary) return;
|
||||
injected = true;
|
||||
if (exclusion === "ignore")
|
||||
await f.disk.userRename("a.md", path);
|
||||
// When installation already completed, deleting REMOTE
|
||||
// is a subsequent local edit, which must not be undone.
|
||||
await f.disk.userWrite(path, bytes(local));
|
||||
await f.client.syncLocallyUpdatedFile({
|
||||
relativePath: path,
|
||||
...(exclusion === "ignore"
|
||||
? { oldPath: "a.md" }
|
||||
: {})
|
||||
});
|
||||
armed = true;
|
||||
};
|
||||
f.store.boundary = (label) => {
|
||||
if (!armed || crashed || label !== recovery) return;
|
||||
crashed = true;
|
||||
f.disk.crash(true);
|
||||
throw new Error("interrupted exclusion save");
|
||||
};
|
||||
if (recovery.endsWith(":save")) {
|
||||
await assert.rejects(
|
||||
f.client.setSetting("isSyncEnabled", true),
|
||||
/interrupted exclusion save/
|
||||
);
|
||||
assert(crashed);
|
||||
} else await f.client.setSetting("isSyncEnabled", true);
|
||||
assert(injected);
|
||||
f.disk.boundary = f.store.boundary = () => {};
|
||||
if (recovery !== "none") {
|
||||
await f.reopenClient();
|
||||
await f.client.start();
|
||||
}
|
||||
|
||||
assert.deepEqual(
|
||||
(await f.disk.readSnapshot(path))?.content,
|
||||
bytes(local)
|
||||
);
|
||||
const stored = f.store.snapshot().database!;
|
||||
assert.equal(
|
||||
stored.documents![id].base?.vaultUpdateId,
|
||||
installed ? remote.vaultUpdateId : base.vaultUpdateId
|
||||
);
|
||||
assert.equal(
|
||||
stored.remoteHeads![id].vaultUpdateId,
|
||||
remote.vaultUpdateId
|
||||
);
|
||||
assert.equal(
|
||||
(await f.get("/documents/" + id)).contentBase64,
|
||||
Buffer.from("REMOTE\nBASE\n").toString("base64")
|
||||
);
|
||||
await f.client.setSettings({
|
||||
maxFileSizeMB: 1,
|
||||
ignorePatterns: []
|
||||
});
|
||||
const expected = installed ? local : "REMOTE\n" + local;
|
||||
const verify = async () => {
|
||||
assert.deepEqual(
|
||||
(await f.get("/file-manifest")).entries,
|
||||
{ [id]: path }
|
||||
);
|
||||
assert.deepEqual(
|
||||
(await f.disk.readSnapshot(path))?.content,
|
||||
bytes(expected)
|
||||
);
|
||||
assert.equal(
|
||||
(await f.get("/documents/" + id)).contentBase64,
|
||||
Buffer.from(expected).toString("base64")
|
||||
);
|
||||
};
|
||||
await verify();
|
||||
await f.client.reset();
|
||||
await verify();
|
||||
} finally {
|
||||
await f.dispose();
|
||||
}
|
||||
});
|
||||
|
||||
for (const at of [undefined, "before:save", "durable:save"])
|
||||
for (const powerLoss of at ? [false, true] : [false])
|
||||
test(`repeated saves retain only the latest local edit through ${at ?? "completion"} and powerLoss=${powerLoss}`, async () => {
|
||||
const f = await fixture();
|
||||
try {
|
||||
const id = randomUUID();
|
||||
const base = await f.put("/documents/" + id, f.content("base"));
|
||||
await f.put("/file-manifest", {
|
||||
requestId: randomUUID(),
|
||||
parentFileManifestId: 0,
|
||||
entries: { [id]: "a.md" }
|
||||
});
|
||||
await f.client.start();
|
||||
await f.client.setSetting("isSyncEnabled", false);
|
||||
await f.put(
|
||||
"/documents/" + id,
|
||||
f.content("base REMOTE", base.vaultUpdateId)
|
||||
);
|
||||
let saves = 0,
|
||||
armed = false,
|
||||
crashed = false;
|
||||
f.disk.boundary = async (label) => {
|
||||
if (
|
||||
saves === 0 &&
|
||||
label.startsWith("before:rename:a.md->")
|
||||
) {
|
||||
saves++;
|
||||
await f.disk.userWrite("a.md", bytes("base ONE"));
|
||||
await f.client.syncLocallyUpdatedFile({
|
||||
relativePath: "a.md"
|
||||
});
|
||||
} else if (
|
||||
saves === 1 &&
|
||||
label.startsWith("before:write:") &&
|
||||
label.endsWith(".output")
|
||||
) {
|
||||
saves++;
|
||||
await f.disk.userWrite("a.md", bytes("base TWO"));
|
||||
await f.client.syncLocallyUpdatedFile({
|
||||
relativePath: "a.md"
|
||||
});
|
||||
armed = true;
|
||||
}
|
||||
};
|
||||
f.store.boundary = (label) => {
|
||||
if (!armed || crashed || label !== at) return;
|
||||
crashed = true;
|
||||
f.disk.crash(powerLoss);
|
||||
throw new Error("injected repeated-save crash");
|
||||
};
|
||||
if (at) {
|
||||
await assert.rejects(
|
||||
f.client.setSetting("isSyncEnabled", true),
|
||||
/injected repeated-save crash/
|
||||
);
|
||||
assert(crashed);
|
||||
f.disk.boundary = () => {};
|
||||
f.store.boundary = () => {};
|
||||
await f.reopenClient();
|
||||
await f.client.start();
|
||||
} else await f.client.setSetting("isSyncEnabled", true);
|
||||
assert.equal(saves, 2);
|
||||
const verify = async () => {
|
||||
assert.deepEqual((await f.get("/file-manifest")).entries, {
|
||||
[id]: "a.md"
|
||||
});
|
||||
assert.deepEqual(
|
||||
(await f.disk.readSnapshot("a.md"))?.content,
|
||||
bytes("base TWO REMOTE")
|
||||
);
|
||||
assert.equal(
|
||||
Buffer.from(
|
||||
(await f.get("/documents/" + id)).contentBase64,
|
||||
"base64"
|
||||
).toString(),
|
||||
"base TWO REMOTE"
|
||||
);
|
||||
};
|
||||
await verify();
|
||||
await f.client.reset();
|
||||
await verify();
|
||||
} finally {
|
||||
await f.dispose();
|
||||
}
|
||||
});
|
||||
|
||||
for (const timing of ["before", "durable"])
|
||||
for (const crash of [false, true])
|
||||
test(`oversized save ${timing} staging stays local across crash=${crash}`, async () => {
|
||||
const f = await fixture();
|
||||
try {
|
||||
const id = randomUUID(),
|
||||
large = "PRIVATE".repeat(100);
|
||||
const base = await f.put(
|
||||
"/documents/" + id,
|
||||
f.content("PUBLIC")
|
||||
);
|
||||
await f.put("/file-manifest", {
|
||||
requestId: randomUUID(),
|
||||
parentFileManifestId: 0,
|
||||
entries: { [id]: "a.bin" }
|
||||
});
|
||||
await f.client.start();
|
||||
await f.client.setSetting("isSyncEnabled", false);
|
||||
await f.client.setSetting("maxFileSizeMB", 100 / (1024 * 1024));
|
||||
await f.put(
|
||||
"/documents/" + id,
|
||||
f.content("REMOTE", base.vaultUpdateId)
|
||||
);
|
||||
let injected = false,
|
||||
armed = false,
|
||||
crashed = false;
|
||||
f.disk.boundary = async (label) => {
|
||||
if (
|
||||
injected ||
|
||||
!label.startsWith(
|
||||
`${timing}:rename:a.bin->.vault-link-sync/transactions/`
|
||||
)
|
||||
)
|
||||
return;
|
||||
injected = true;
|
||||
await f.disk.userWrite("a.bin", bytes(large));
|
||||
await f.client.syncLocallyUpdatedFile({
|
||||
relativePath: "a.bin"
|
||||
});
|
||||
armed = true;
|
||||
};
|
||||
f.store.boundary = (label) => {
|
||||
if (!crash || !armed || crashed || label !== "durable:save")
|
||||
return;
|
||||
crashed = true;
|
||||
f.disk.crash(true);
|
||||
throw new Error("injected oversized-save crash");
|
||||
};
|
||||
if (crash) {
|
||||
await assert.rejects(
|
||||
f.client.setSetting("isSyncEnabled", true),
|
||||
/injected oversized-save crash/
|
||||
);
|
||||
assert(crashed);
|
||||
f.disk.boundary = () => {};
|
||||
f.store.boundary = () => {};
|
||||
await f.reopenClient();
|
||||
await f.client.start();
|
||||
} else await f.client.setSetting("isSyncEnabled", true);
|
||||
assert(injected);
|
||||
const verify = async () => {
|
||||
assert.deepEqual(
|
||||
(await f.disk.readSnapshot("a.bin"))?.content,
|
||||
bytes(large)
|
||||
);
|
||||
assert.deepEqual((await f.get("/file-manifest")).entries, {
|
||||
[id]: "a.bin"
|
||||
});
|
||||
assert.equal(
|
||||
Buffer.from(
|
||||
(await f.get("/documents/" + id)).contentBase64,
|
||||
"base64"
|
||||
).toString(),
|
||||
"REMOTE"
|
||||
);
|
||||
};
|
||||
await verify();
|
||||
await f.client.reset();
|
||||
await verify();
|
||||
} finally {
|
||||
await f.dispose();
|
||||
}
|
||||
});
|
||||
|
||||
test("a tracked file moved into an ignored path during remote application never uploads its new bytes", async () => {
|
||||
const f = await fixture();
|
||||
try {
|
||||
const a = randomUUID(),
|
||||
b = randomUUID();
|
||||
await f.put("/documents/" + a, f.content("PUBLIC"));
|
||||
await f.put("/documents/" + b, f.content("OTHER"));
|
||||
const m = await f.put("/file-manifest", {
|
||||
requestId: randomUUID(),
|
||||
parentFileManifestId: 0,
|
||||
entries: { [a]: "a.md", [b]: "b.md" }
|
||||
});
|
||||
await f.client.setSetting("ignorePatterns", ["private.md"]);
|
||||
await f.client.start();
|
||||
await f.client.setSetting("isSyncEnabled", false);
|
||||
await f.put("/file-manifest", {
|
||||
requestId: randomUUID(),
|
||||
parentFileManifestId: m.fileManifestId,
|
||||
entries: { [a]: "a.md", [b]: "remote-b.md" }
|
||||
});
|
||||
let injected = false;
|
||||
f.disk.boundary = async (label) => {
|
||||
if (
|
||||
injected ||
|
||||
!label.startsWith("before:mkdir:.vault-link-sync/transactions/")
|
||||
)
|
||||
return;
|
||||
injected = true;
|
||||
await f.disk.userRename("a.md", "private.md");
|
||||
await f.disk.userWrite("private.md", bytes("SECRET"));
|
||||
await f.client.syncLocallyUpdatedFile({
|
||||
oldPath: "a.md",
|
||||
relativePath: "private.md"
|
||||
});
|
||||
};
|
||||
await f.client.setSetting("isSyncEnabled", true);
|
||||
assert(injected);
|
||||
const manifest = await f.get("/file-manifest");
|
||||
const contents = await Promise.all(
|
||||
Object.keys(manifest.entries).map((id) => f.get("/documents/" + id))
|
||||
);
|
||||
assert(
|
||||
!contents.some(
|
||||
(c) =>
|
||||
Buffer.from(c.contentBase64, "base64").toString() ===
|
||||
"SECRET"
|
||||
)
|
||||
);
|
||||
assert.deepEqual(
|
||||
(await f.disk.readSnapshot("private.md"))?.content,
|
||||
bytes("SECRET")
|
||||
);
|
||||
assert.equal(manifest.entries[a], "a.md");
|
||||
} finally {
|
||||
await f.dispose();
|
||||
}
|
||||
});
|
||||
test("remote creation plus unrelated notification must not delete remote membership on retry", async () => {
|
||||
const f = await fixture();
|
||||
try {
|
||||
await f.client.start();
|
||||
await f.client.setSetting("isSyncEnabled", false);
|
||||
const id = randomUUID();
|
||||
await f.put("/documents/" + id, f.content("REMOTE"));
|
||||
await f.put("/file-manifest", {
|
||||
requestId: randomUUID(),
|
||||
parentFileManifestId: 0,
|
||||
entries: { [id]: "remote.md" }
|
||||
});
|
||||
let injected = false;
|
||||
f.disk.boundary = async (label) => {
|
||||
if (
|
||||
injected ||
|
||||
!label.startsWith("before:mkdir:.vault-link-sync/transactions/")
|
||||
)
|
||||
return;
|
||||
injected = true;
|
||||
await f.disk.userWrite("local.md", bytes("LOCAL"));
|
||||
await f.client.syncLocallyCreatedFile("local.md");
|
||||
};
|
||||
await f.client.setSetting("isSyncEnabled", true);
|
||||
assert(injected);
|
||||
f.disk.boundary = () => {};
|
||||
await f.client.reset();
|
||||
const manifest = await f.get("/file-manifest");
|
||||
|
||||
assert.equal(manifest.entries[id], "remote.md");
|
||||
} finally {
|
||||
await f.dispose();
|
||||
}
|
||||
});
|
||||
test("notified ignored occupant during remote rename must never upload its bytes", async () => {
|
||||
const f = await fixture();
|
||||
try {
|
||||
const id = randomUUID();
|
||||
await f.put("/documents/" + id, f.content("REMOTE"));
|
||||
const m = await f.put("/file-manifest", {
|
||||
requestId: randomUUID(),
|
||||
parentFileManifestId: 0,
|
||||
entries: { [id]: "a.md" }
|
||||
});
|
||||
await f.client.setSetting("ignorePatterns", ["private"]);
|
||||
await f.client.start();
|
||||
await f.client.setSetting("isSyncEnabled", false);
|
||||
await f.put("/file-manifest", {
|
||||
requestId: randomUUID(),
|
||||
parentFileManifestId: m.fileManifestId,
|
||||
entries: { [id]: "private/note.md" }
|
||||
});
|
||||
let injected = false;
|
||||
f.disk.boundary = async (label) => {
|
||||
if (
|
||||
injected ||
|
||||
!label.startsWith("before:mkdir:.vault-link-sync/transactions/")
|
||||
)
|
||||
return;
|
||||
injected = true;
|
||||
await f.disk.userWrite("private", bytes("SECRET"));
|
||||
await f.client.syncLocallyCreatedFile("private");
|
||||
};
|
||||
await f.client.setSetting("isSyncEnabled", true);
|
||||
f.disk.boundary = () => {};
|
||||
const manifest = await f.get("/file-manifest");
|
||||
const contents = await Promise.all(
|
||||
Object.keys(manifest.entries).map((id) => f.get("/documents/" + id))
|
||||
);
|
||||
|
||||
assert(injected);
|
||||
assert.deepEqual(
|
||||
(await f.disk.readSnapshot("private"))?.content,
|
||||
bytes("SECRET")
|
||||
);
|
||||
assert(
|
||||
!contents.some(
|
||||
(c) =>
|
||||
Buffer.from(c.contentBase64, "base64").toString() ===
|
||||
"SECRET"
|
||||
)
|
||||
);
|
||||
} finally {
|
||||
await f.dispose();
|
||||
}
|
||||
});
|
||||
test("notified delete/recreate before staging retains replacement file", async () => {
|
||||
const f = await fixture();
|
||||
try {
|
||||
const id = randomUUID();
|
||||
const h = await f.put("/documents/" + id, f.content("ORIGINAL"));
|
||||
const m = await f.put("/file-manifest", {
|
||||
requestId: randomUUID(),
|
||||
parentFileManifestId: 0,
|
||||
entries: { [id]: "a.md" }
|
||||
});
|
||||
await f.client.start();
|
||||
await f.client.setSetting("isSyncEnabled", false);
|
||||
await f.put(
|
||||
"/documents/" + id,
|
||||
f.content("ORIGINAL REMOTE", h.vaultUpdateId)
|
||||
);
|
||||
await f.put("/file-manifest", {
|
||||
requestId: randomUUID(),
|
||||
parentFileManifestId: m.fileManifestId,
|
||||
entries: { [id]: "remote.md" }
|
||||
});
|
||||
let injected = false;
|
||||
f.disk.boundary = async (label) => {
|
||||
if (
|
||||
injected ||
|
||||
!label.startsWith(
|
||||
"before:rename:a.md->.vault-link-sync/transactions/"
|
||||
)
|
||||
)
|
||||
return;
|
||||
injected = true;
|
||||
await f.disk.userDelete("a.md");
|
||||
await f.client.syncLocallyDeletedFile("a.md");
|
||||
await f.disk.userWrite("a.md", bytes("REPLACEMENT"));
|
||||
await f.client.syncLocallyCreatedFile("a.md");
|
||||
};
|
||||
await f.client.setSetting("isSyncEnabled", true);
|
||||
f.disk.boundary = () => {};
|
||||
const manifest = await f.get("/file-manifest");
|
||||
|
||||
assert.deepEqual(
|
||||
(await f.disk.readSnapshot("a.md"))?.content,
|
||||
bytes("REPLACEMENT")
|
||||
);
|
||||
} finally {
|
||||
await f.dispose();
|
||||
}
|
||||
});
|
||||
|
||||
for (const mode of ["delete", "move", "save"] as const)
|
||||
for (const powerLoss of [false, true])
|
||||
for (const at of ["before:save", "durable:save"])
|
||||
test(`${mode} during staging survives ${at} and ${powerLoss ? "power loss" : "process replacement"}`, async () => {
|
||||
const f = await fixture();
|
||||
try {
|
||||
const id = randomUUID();
|
||||
const h = await f.put(
|
||||
"/documents/" + id,
|
||||
f.content("ORIGINAL")
|
||||
);
|
||||
await f.put("/file-manifest", {
|
||||
requestId: randomUUID(),
|
||||
parentFileManifestId: 0,
|
||||
entries: { [id]: "a.md" }
|
||||
});
|
||||
await f.client.start();
|
||||
await f.client.setSetting("isSyncEnabled", false);
|
||||
await f.put(
|
||||
"/documents/" + id,
|
||||
f.content("ORIGINAL REMOTE", h.vaultUpdateId)
|
||||
);
|
||||
let injected = false,
|
||||
armed = false,
|
||||
crashed = false;
|
||||
f.disk.boundary = async (label) => {
|
||||
const point = mode === "save" ? "durable" : "before";
|
||||
if (
|
||||
injected ||
|
||||
!label.startsWith(
|
||||
`${point}:rename:a.md->.vault-link-sync/transactions/`
|
||||
)
|
||||
)
|
||||
return;
|
||||
injected = true;
|
||||
if (mode === "save") {
|
||||
await f.disk.userWrite(
|
||||
"a.md",
|
||||
bytes("ORIGINAL LOCAL")
|
||||
);
|
||||
await f.client.syncLocallyUpdatedFile({
|
||||
relativePath: "a.md"
|
||||
});
|
||||
} else {
|
||||
if (mode === "delete") {
|
||||
await f.disk.userDelete("a.md");
|
||||
await f.client.syncLocallyDeletedFile("a.md");
|
||||
} else {
|
||||
await f.disk.userRename("a.md", "mine.md");
|
||||
await f.client.syncLocallyUpdatedFile({
|
||||
oldPath: "a.md",
|
||||
relativePath: "mine.md"
|
||||
});
|
||||
}
|
||||
await f.disk.userWrite(
|
||||
"a.md",
|
||||
bytes("REPLACEMENT")
|
||||
);
|
||||
await f.client.syncLocallyCreatedFile("a.md");
|
||||
}
|
||||
armed = true;
|
||||
};
|
||||
f.store.boundary = (label) => {
|
||||
if (!armed || crashed || label !== at) return;
|
||||
crashed = true;
|
||||
f.disk.crash(powerLoss);
|
||||
throw new Error("injected journal crash");
|
||||
};
|
||||
await assert.rejects(
|
||||
f.client.setSetting("isSyncEnabled", true),
|
||||
/injected journal crash/
|
||||
);
|
||||
assert(injected && crashed);
|
||||
f.disk.boundary = () => {};
|
||||
f.store.boundary = () => {};
|
||||
await f.reopenClient();
|
||||
await f.client.start();
|
||||
const manifest = await f.get("/file-manifest");
|
||||
if (mode === "save") {
|
||||
assert.deepEqual(manifest.entries, { [id]: "a.md" });
|
||||
const actual = Buffer.from(
|
||||
(await f.disk.readSnapshot("a.md"))!.content
|
||||
).toString();
|
||||
assert.equal(actual, "ORIGINAL LOCAL REMOTE");
|
||||
} else {
|
||||
assert.deepEqual(
|
||||
(await f.disk.readSnapshot("a.md"))?.content,
|
||||
bytes("REPLACEMENT")
|
||||
);
|
||||
assert.notEqual(
|
||||
Object.keys(manifest.entries).find(
|
||||
(key) => manifest.entries[key] === "a.md"
|
||||
),
|
||||
id
|
||||
);
|
||||
if (mode === "move") {
|
||||
assert.equal(manifest.entries[id], "mine.md");
|
||||
assert.deepEqual(
|
||||
(await f.disk.readSnapshot("mine.md"))?.content,
|
||||
bytes("ORIGINAL REMOTE")
|
||||
);
|
||||
} else assert.equal(manifest.entries[id], undefined);
|
||||
}
|
||||
await f.client.reset();
|
||||
assert.deepEqual(
|
||||
(await f.get("/file-manifest")).entries,
|
||||
manifest.entries
|
||||
);
|
||||
} finally {
|
||||
await f.dispose();
|
||||
}
|
||||
});
|
||||
|
|
@ -111,16 +111,6 @@ export function assertQuiescent(
|
|||
undefined,
|
||||
"Pending CAS request at convergence"
|
||||
);
|
||||
assert.equal(
|
||||
state.eventReplay,
|
||||
undefined,
|
||||
"Unfinished event replay at convergence"
|
||||
);
|
||||
assert.equal(
|
||||
state.application,
|
||||
undefined,
|
||||
"Unfinished filesystem journal at convergence"
|
||||
);
|
||||
assert.deepEqual(
|
||||
state.local,
|
||||
entries,
|
||||
|
|
|
|||
|
|
@ -101,7 +101,6 @@ async function fixture(syncIntervalMs: number | undefined) {
|
|||
const persistence = new MemoryPersistence({ database: initial });
|
||||
const logger = new Logger();
|
||||
const database = new Database(
|
||||
logger,
|
||||
initial,
|
||||
async (next) => persistence.save({ database: next }),
|
||||
initial.vaultKey,
|
||||
|
|
|
|||
|
|
@ -248,7 +248,6 @@ for (const [kind, format, originalPath] of [
|
|||
"Restart changed the persisted request"
|
||||
);
|
||||
assert.equal(restarted.stored.database!.pending, undefined);
|
||||
assert.equal(restarted.stored.database!.application, undefined);
|
||||
const recovered = new MemoryDisk();
|
||||
recovered.restore(restarted.disk);
|
||||
assert.deepEqual(
|
||||
|
|
|
|||
|
|
@ -8,12 +8,11 @@ import { promisify } from "node:util";
|
|||
import { RealDisk } from "./real-disk";
|
||||
import { Database, emptyState } from "../sync-client/src/persistence/database";
|
||||
import { FileOperations } from "../sync-client/src/file-operations/file-operations";
|
||||
import { Logger } from "../sync-client/src/tracing/logger";
|
||||
import type { ServerConfig } from "../sync-client/src/services/server-config";
|
||||
import { toStoredSnapshot } from "../sync-client/src/sync-operations/content";
|
||||
|
||||
test(
|
||||
"real filesystem: exclusive operations, unsafe entries, and journal swaps",
|
||||
"real filesystem: exclusive operations, unsafe entries, and file swaps",
|
||||
{ timeout: 20_000 },
|
||||
async () => {
|
||||
const directory = await fs.mkdtemp(
|
||||
|
|
@ -53,11 +52,9 @@ test(
|
|||
);
|
||||
await disk.write("a.md", snapshot("A"));
|
||||
await disk.write("b.md", snapshot("B"));
|
||||
// Recovery must flush the old source's ancestors even if another
|
||||
// writer has replaced one of those directories with a file.
|
||||
// A file in an ancestor position makes descendants absent.
|
||||
assert.equal(await disk.stat("a.md/absent.md"), undefined);
|
||||
assert.equal(await disk.readSnapshot("a.md/absent.md"), undefined);
|
||||
await disk.flushPaths(["a.md/absent.md"]);
|
||||
await assert.rejects(disk.rename("a.md", "b.md"));
|
||||
await assert.rejects(disk.delete("a.md"), /regular file/);
|
||||
await fs.symlink("a.md", path.join(root, "symlink"));
|
||||
|
|
@ -105,7 +102,6 @@ test(
|
|||
).hash
|
||||
};
|
||||
const database = new Database(
|
||||
new Logger(),
|
||||
initial,
|
||||
async () => {},
|
||||
"real",
|
||||
|
|
@ -129,7 +125,6 @@ test(
|
|||
).toString(),
|
||||
"A"
|
||||
);
|
||||
assert.equal(database.state.application, undefined);
|
||||
} finally {
|
||||
await fs.rm(directory, { recursive: true });
|
||||
}
|
||||
|
|
|
|||
|
|
@ -4,7 +4,6 @@ import * as path from "node:path";
|
|||
import { constants } from "node:fs";
|
||||
import { execFile } from "node:child_process";
|
||||
import { promisify } from "node:util";
|
||||
import { randomUUID } from "node:crypto";
|
||||
import type { FileSystemOperations, FileSnapshot } from "sync-client";
|
||||
|
||||
const exec = promisify(execFile);
|
||||
|
|
@ -52,17 +51,6 @@ export class RealDisk implements FileSystemOperations {
|
|||
throw error;
|
||||
}
|
||||
}
|
||||
private async flush(absolute: string): Promise<void> {
|
||||
const handle = await fs.open(
|
||||
absolute,
|
||||
constants.O_RDONLY | constants.O_NOFOLLOW
|
||||
);
|
||||
try {
|
||||
await handle.sync();
|
||||
} finally {
|
||||
await handle.close();
|
||||
}
|
||||
}
|
||||
public async stat(relative: string) {
|
||||
const info = await this.info(await this.resolve(relative));
|
||||
return info
|
||||
|
|
@ -140,8 +128,6 @@ export class RealDisk implements FileSystemOperations {
|
|||
throw error;
|
||||
}
|
||||
assert((await fs.lstat(next)).isDirectory());
|
||||
await this.flush(next);
|
||||
await this.flush(current);
|
||||
current = next;
|
||||
}
|
||||
}
|
||||
|
|
@ -154,44 +140,16 @@ export class RealDisk implements FileSystemOperations {
|
|||
"This disk-only fixture has no editor selections"
|
||||
);
|
||||
const destination = await this.resolve(relative);
|
||||
const staging = ".vault-link-sync/adapter-staging";
|
||||
await this.createDirectory(staging);
|
||||
const source = path.join(this.root, staging, randomUUID());
|
||||
const handle = await fs.open(source, "wx", 0o600);
|
||||
try {
|
||||
await handle.writeFile(snapshot.content);
|
||||
await handle.sync();
|
||||
} finally {
|
||||
await handle.close();
|
||||
}
|
||||
try {
|
||||
await exec(this.helper, ["rename", source, destination]);
|
||||
await this.flush(path.dirname(source));
|
||||
await this.flush(path.dirname(destination));
|
||||
} finally {
|
||||
await fs.unlink(source).catch((error: NodeJS.ErrnoException) => {
|
||||
if (error.code !== "ENOENT") throw error;
|
||||
});
|
||||
}
|
||||
await fs.writeFile(destination, snapshot.content, {
|
||||
flag: "wx",
|
||||
mode: 0o600
|
||||
});
|
||||
}
|
||||
public async rename(from: string, to: string): Promise<void> {
|
||||
const source = await this.resolve(from),
|
||||
destination = await this.resolve(to);
|
||||
assert((await this.info(source))?.isFile());
|
||||
await this.flush(source);
|
||||
await exec(this.helper, ["rename", source, destination]);
|
||||
await this.flush(path.dirname(source));
|
||||
await this.flush(path.dirname(destination));
|
||||
}
|
||||
public async flushPaths(paths: readonly string[]): Promise<void> {
|
||||
for (const relative of paths) {
|
||||
let absolute = await this.resolve(relative);
|
||||
for (;;) {
|
||||
if (await this.info(absolute)) await this.flush(absolute);
|
||||
if (absolute === this.root) break;
|
||||
absolute = path.dirname(absolute);
|
||||
}
|
||||
}
|
||||
}
|
||||
public async delete(relative: string): Promise<void> {
|
||||
assert(relative, "Cannot prune root");
|
||||
|
|
@ -202,7 +160,6 @@ export class RealDisk implements FileSystemOperations {
|
|||
for (const child of await fs.readdir(absolute))
|
||||
await this.delete(`${relative}/${child}`);
|
||||
await fs.rmdir(absolute);
|
||||
await this.flush(path.dirname(absolute));
|
||||
}
|
||||
public async deleteFile(relative: string): Promise<void> {
|
||||
const absolute = await this.resolve(relative);
|
||||
|
|
@ -210,6 +167,5 @@ export class RealDisk implements FileSystemOperations {
|
|||
if (!entry) return;
|
||||
assert(entry.isFile() && entry.nlink === 1, "Cannot unlink non-file");
|
||||
await fs.unlink(absolute);
|
||||
await this.flush(path.dirname(absolute));
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1,6 +1,5 @@
|
|||
import assert from "node:assert/strict";
|
||||
import { test } from "node:test";
|
||||
import { isDeepStrictEqual } from "node:util";
|
||||
import { MemoryDisk, MemoryPersistence } from "./storage";
|
||||
import {
|
||||
Database,
|
||||
|
|
@ -9,7 +8,6 @@ import {
|
|||
} from "../sync-client/src/persistence/database";
|
||||
import { FileOperations } from "../sync-client/src/file-operations/file-operations";
|
||||
import { toStoredSnapshot } from "../sync-client/src/sync-operations/content";
|
||||
import { Logger } from "../sync-client/src/tracing/logger";
|
||||
import type { ServerConfig } from "../sync-client/src/services/server-config";
|
||||
import { mergeFileManifests } from "../sync-client/src/sync-operations/file-manifest";
|
||||
import { assertManifest, pathDecision } from "./oracles";
|
||||
|
|
@ -45,7 +43,6 @@ async function fixture(
|
|||
const persistence = new MemoryPersistence({ database: initial });
|
||||
const restart = (changes: LocalChange[] = []) => {
|
||||
const database = new Database(
|
||||
new Logger(),
|
||||
persistence.snapshot().database,
|
||||
async (next) => persistence.save({ database: next }),
|
||||
"recovery-test",
|
||||
|
|
@ -85,7 +82,25 @@ async function swapFixture() {
|
|||
["b.md", bytes("A")],
|
||||
["nested/c.md", bytes("C")]
|
||||
]);
|
||||
return { ...current, next, writes, expected, retained: [] as string[] };
|
||||
return { ...current, next, writes, expected };
|
||||
}
|
||||
|
||||
async function directoryCycleFixture() {
|
||||
const current = await fixture({
|
||||
a: { path: "a", text: "A" },
|
||||
b: { path: "b/deep/child", text: "B" }
|
||||
});
|
||||
const next = structuredClone(current.initial);
|
||||
next.local = { a: "b", b: "a/deep/child" };
|
||||
return {
|
||||
...current,
|
||||
next,
|
||||
writes: {},
|
||||
expected: new Map([
|
||||
["b", bytes("A")],
|
||||
["a/deep/child", bytes("B")]
|
||||
])
|
||||
};
|
||||
}
|
||||
|
||||
async function crossDirectoryFixture() {
|
||||
|
|
@ -101,7 +116,7 @@ async function crossDirectoryFixture() {
|
|||
["dst/a.md", bytes("A")],
|
||||
["src/keep.md", bytes("untouched sibling")]
|
||||
]);
|
||||
return { ...current, next, writes: {}, expected, retained: [] as string[] };
|
||||
return { ...current, next, writes: {}, expected };
|
||||
}
|
||||
|
||||
async function mergeFixture() {
|
||||
|
|
@ -123,8 +138,7 @@ async function mergeFixture() {
|
|||
"note.md",
|
||||
bytes("local first paragraph\n\nremote last paragraph\n")
|
||||
]
|
||||
]),
|
||||
retained: [local]
|
||||
])
|
||||
};
|
||||
}
|
||||
|
||||
|
|
@ -145,8 +159,7 @@ async function deletionFixture() {
|
|||
})
|
||||
}
|
||||
},
|
||||
expected: new Map([["a.bin", bytes("remote binary\0")]]),
|
||||
retained: ["private binary edit\0", "unsent deleted bytes"]
|
||||
expected: new Map([["a.bin", bytes("remote binary\0")]])
|
||||
};
|
||||
}
|
||||
|
||||
|
|
@ -175,19 +188,19 @@ async function permanentDestinationFixture() {
|
|||
expected: new Map([
|
||||
["Recovered a.md", bytes("REMOTE")],
|
||||
["good.md", bytes("B")]
|
||||
]),
|
||||
retained: ["A"]
|
||||
])
|
||||
};
|
||||
}
|
||||
|
||||
for (const [name, create] of [
|
||||
["swap and create", swapFixture],
|
||||
["cross-directory move with surviving sibling", crossDirectoryFixture],
|
||||
["merge during recovery", mergeFixture],
|
||||
["binary replacement, deletion and retained bytes", deletionFixture],
|
||||
["permanent destination replan", permanentDestinationFixture]
|
||||
["file/directory cycle", directoryCycleFixture],
|
||||
["cross-directory move", crossDirectoryFixture],
|
||||
["content merge", mergeFixture],
|
||||
["binary replacement and deletion", deletionFixture],
|
||||
["permanent destination fallback", permanentDestinationFixture]
|
||||
] as const) {
|
||||
test(`${name}: interrupt every apply and replay boundary, then power-cycle again`, async (t) => {
|
||||
test(`${name}: interrupted operations leave metadata usable by ordinary scans`, async (t) => {
|
||||
const baseline = await create();
|
||||
const labels: string[] = [];
|
||||
const trace = (label: string) => {
|
||||
|
|
@ -200,172 +213,90 @@ for (const [name, create] of [
|
|||
baseline.persistence.boundary = trace;
|
||||
await baseline.restart().files.apply(baseline.next, baseline.writes);
|
||||
assert.deepEqual(baseline.disk.userFiles(), baseline.expected);
|
||||
assert(labels.some((label) => label.startsWith("visible:rename:")));
|
||||
assert(labels.includes("durable:save"));
|
||||
let replayCrashes = 0;
|
||||
for (const powerLoss of [false, true]) {
|
||||
for (let index = 0; index < labels.length; index++) {
|
||||
assert.deepEqual(
|
||||
baseline.persistence.snapshot().database!.local,
|
||||
baseline.expectedLocal ?? baseline.next.local
|
||||
);
|
||||
assert(labels.length > 0);
|
||||
for (const powerLoss of [false, true])
|
||||
for (let cut = 0; cut < labels.length; cut++) {
|
||||
await t.test(
|
||||
`${powerLoss ? "power" : "process"} crash #${index}: ${labels[index]}`,
|
||||
`${powerLoss ? "power" : "process"} interruption at ${labels[cut]}`,
|
||||
async () => {
|
||||
const current = await create();
|
||||
const clearFaults = () => {
|
||||
current.disk.boundary = current.permanentFault;
|
||||
current.persistence.boundary = () => {};
|
||||
};
|
||||
const interrupt = async (
|
||||
action: () => Promise<void>,
|
||||
at: number,
|
||||
power: boolean
|
||||
) => {
|
||||
let position = 0,
|
||||
fired = false;
|
||||
const fault = (label: string) => {
|
||||
if (label.startsWith("read:")) return;
|
||||
if (position++ === at) {
|
||||
fired = true;
|
||||
current.disk.crash(power);
|
||||
throw new Error("Injected interruption");
|
||||
}
|
||||
};
|
||||
current.disk.boundary = (label) => {
|
||||
current.permanentFault(label);
|
||||
fault(label);
|
||||
};
|
||||
current.persistence.boundary = fault;
|
||||
try {
|
||||
await assert.rejects(
|
||||
action(),
|
||||
/Injected interruption/
|
||||
);
|
||||
assert(fired, "Fault index did not occur");
|
||||
} finally {
|
||||
clearFaults();
|
||||
let seen = 0;
|
||||
let fired = false;
|
||||
const interrupt = (label: string) => {
|
||||
if (label.startsWith("read:")) return;
|
||||
if (seen++ === cut) {
|
||||
fired = true;
|
||||
current.disk.crash(powerLoss);
|
||||
throw new Error("interrupted");
|
||||
}
|
||||
};
|
||||
const verify = (
|
||||
runtime: ReturnType<typeof current.restart>
|
||||
) => {
|
||||
assert.equal(
|
||||
runtime.database.state.application,
|
||||
undefined
|
||||
);
|
||||
assert.deepEqual(
|
||||
runtime.database.state.local,
|
||||
current.expectedLocal ?? current.next.local
|
||||
);
|
||||
assert.deepEqual(
|
||||
current.disk.userFiles(),
|
||||
current.expected,
|
||||
"Exact files must survive interrupted replay"
|
||||
);
|
||||
const retained = current.disk
|
||||
.image()
|
||||
.visible.filter(
|
||||
([path, entry]) =>
|
||||
entry &&
|
||||
path.startsWith(
|
||||
".vault-link-sync/recovery/"
|
||||
) &&
|
||||
path.endsWith(".content")
|
||||
)
|
||||
.map(([, entry]) =>
|
||||
Buffer.from(entry!.content).toString()
|
||||
)
|
||||
.sort();
|
||||
assert.deepEqual(
|
||||
retained,
|
||||
[...current.retained].sort(),
|
||||
"Recovery archives lost or duplicated displaced bytes"
|
||||
);
|
||||
assert(
|
||||
!current.disk
|
||||
.image()
|
||||
.visible.some(
|
||||
([path, entry]) =>
|
||||
entry &&
|
||||
path.startsWith(
|
||||
".vault-link-sync/transactions/"
|
||||
)
|
||||
),
|
||||
"Completed recovery left transaction artifacts"
|
||||
);
|
||||
};
|
||||
const finish = async (resubmit: boolean) => {
|
||||
let runtime = current.restart();
|
||||
await runtime.files.recover();
|
||||
if (resubmit)
|
||||
await runtime.files.apply(
|
||||
current.next,
|
||||
current.writes
|
||||
);
|
||||
await runtime.files.recover();
|
||||
verify(runtime);
|
||||
current.disk.crash(true);
|
||||
runtime = current.restart();
|
||||
await runtime.files.recover();
|
||||
verify(runtime);
|
||||
};
|
||||
await interrupt(
|
||||
() =>
|
||||
current
|
||||
.restart()
|
||||
.files.apply(current.next, current.writes),
|
||||
index,
|
||||
powerLoss
|
||||
);
|
||||
const saved = current.persistence.snapshot();
|
||||
const diskImage = current.disk.image();
|
||||
const intentWasNotSaved = isDeepStrictEqual(
|
||||
saved.database,
|
||||
current.initial
|
||||
);
|
||||
// Trace replay from the actual interrupted state, including
|
||||
// recovery-only branches such as already-visible installs.
|
||||
const replayLabels: string[] = [];
|
||||
const traceReplay = (label: string) => {
|
||||
if (!label.startsWith("read:"))
|
||||
replayLabels.push(label);
|
||||
};
|
||||
current.disk.boundary = (label) => {
|
||||
current.permanentFault(label);
|
||||
traceReplay(label);
|
||||
interrupt(label);
|
||||
};
|
||||
current.persistence.boundary = traceReplay;
|
||||
await current.restart().files.recover();
|
||||
clearFaults();
|
||||
await finish(intentWasNotSaved);
|
||||
for (const secondPowerLoss of [false, true]) {
|
||||
for (
|
||||
let replay = 0;
|
||||
replay < replayLabels.length;
|
||||
replay++
|
||||
) {
|
||||
current.disk.crash();
|
||||
current.disk.restore(diskImage);
|
||||
await current.persistence.save(saved);
|
||||
try {
|
||||
await interrupt(
|
||||
() => current.restart().files.recover(),
|
||||
replay,
|
||||
secondPowerLoss
|
||||
);
|
||||
await finish(false);
|
||||
replayCrashes++;
|
||||
} catch (error) {
|
||||
throw new Error(
|
||||
`Second ${secondPowerLoss ? "power" : "process"} crash #${replay}: ${replayLabels[replay]}`,
|
||||
{ cause: error }
|
||||
);
|
||||
}
|
||||
}
|
||||
current.persistence.boundary = interrupt;
|
||||
await assert.rejects(
|
||||
current
|
||||
.restart()
|
||||
.files.apply(current.next, current.writes),
|
||||
/interrupted/
|
||||
);
|
||||
assert(fired);
|
||||
current.disk.boundary = () => {};
|
||||
current.persistence.boundary = () => {};
|
||||
// The user may edit after the interruption, including a partial
|
||||
// write or a file whose metadata was never saved.
|
||||
await current.disk.userWrite(
|
||||
"after-restart.md",
|
||||
bytes("user edit")
|
||||
);
|
||||
const runtime = current.restart();
|
||||
const observed = current.disk.userFiles();
|
||||
await scanChanges(runtime, []);
|
||||
assert.deepEqual(
|
||||
current.disk.userFiles(),
|
||||
observed,
|
||||
"A scan must not replay old filesystem intentions"
|
||||
);
|
||||
assertManifest(runtime.database.state.local);
|
||||
assert.deepEqual(
|
||||
new Set(
|
||||
Object.values(runtime.database.state.local)
|
||||
),
|
||||
new Set(observed.keys())
|
||||
);
|
||||
for (const [id, path] of Object.entries(
|
||||
runtime.database.state.local
|
||||
)) {
|
||||
assert(
|
||||
runtime.database.state.documents[id]
|
||||
.materialized
|
||||
);
|
||||
assert.equal(
|
||||
runtime.database.state.documents[id]
|
||||
.observedHash,
|
||||
(await runtime.files.snapshot(path))!.hash
|
||||
);
|
||||
}
|
||||
assert(
|
||||
(await runtime.files.listFilesRecursively()).every(
|
||||
(path) => !path.startsWith(".vault-link-sync/")
|
||||
)
|
||||
);
|
||||
const state = structuredClone(runtime.database.state);
|
||||
await scanChanges(current.restart(), []);
|
||||
assert.deepEqual(
|
||||
current.persistence.snapshot().database,
|
||||
state,
|
||||
"Second startup must be stable"
|
||||
);
|
||||
}
|
||||
);
|
||||
}
|
||||
}
|
||||
assert(replayCrashes > 0, "No interrupted recovery was exercised");
|
||||
t.diagnostic(`${replayCrashes} second crashes during journal replay`);
|
||||
});
|
||||
}
|
||||
|
||||
|
|
@ -647,49 +578,3 @@ test("normalization-sensitive disk does not lose a distinct decomposed filename"
|
|||
"Distinct Unicode file vanished from the scan"
|
||||
);
|
||||
});
|
||||
|
||||
test("notified editor move during a remote swap keeps the moved document's UUID", async () => {
|
||||
const current = await fixture();
|
||||
const changes: LocalChange[] = [];
|
||||
const runtime = current.restart(changes);
|
||||
let fired = false;
|
||||
current.persistence.boundary = async (label) => {
|
||||
if (
|
||||
!fired &&
|
||||
label === "durable:save" &&
|
||||
current.persistence.snapshot().database?.application
|
||||
) {
|
||||
fired = true;
|
||||
await current.disk.userRename("b.md", "editor.md");
|
||||
changes.push({
|
||||
type: "move",
|
||||
oldPath: "b.md",
|
||||
relativePath: "editor.md"
|
||||
});
|
||||
}
|
||||
};
|
||||
const next = structuredClone(runtime.database.state);
|
||||
next.local = { a: "b.md", b: "a.md" };
|
||||
await runtime.files.apply(next);
|
||||
assert(fired);
|
||||
await scanLocalFiles({
|
||||
next: structuredClone(runtime.database.state),
|
||||
changes,
|
||||
files: runtime.files,
|
||||
guard: () => {},
|
||||
ignored: () => false,
|
||||
oversized: () => false,
|
||||
commit: (next) => runtime.database.commit(next)
|
||||
});
|
||||
assert.equal(
|
||||
runtime.database.state.local.b,
|
||||
"editor.md",
|
||||
"Editor move was attributed to the wrong document"
|
||||
);
|
||||
assert.equal(
|
||||
Buffer.from(
|
||||
current.disk.userFiles().get(runtime.database.state.local.b)!
|
||||
).toString(),
|
||||
"B"
|
||||
);
|
||||
});
|
||||
|
|
|
|||
231
frontend/test-support/restart-reconciliation.test.ts
Normal file
231
frontend/test-support/restart-reconciliation.test.ts
Normal file
|
|
@ -0,0 +1,231 @@
|
|||
import assert from "node:assert/strict";
|
||||
import { test } from "node:test";
|
||||
import { fixture, bytes, randomUUID } from "./sync-server-fixture";
|
||||
import { assertCanonical } from "./canonical";
|
||||
import { SyncClient } from "../sync-client/src/sync-client";
|
||||
import { MemoryDisk, MemoryPersistence } from "./storage";
|
||||
|
||||
for (const powerLoss of [false, true])
|
||||
for (const operation of ["create", "replace", "move", "metadata"] as const)
|
||||
test(
|
||||
`${operation}: a ${powerLoss ? "power" : "process"} interruption and later user edits converge without replay`,
|
||||
{ timeout: 20_000 },
|
||||
async () => {
|
||||
const f = await fixture();
|
||||
let peer: SyncClient | undefined;
|
||||
try {
|
||||
const id = randomUUID();
|
||||
const head = await f.put(
|
||||
`/documents/${id}`,
|
||||
f.content("BASE\n")
|
||||
);
|
||||
const manifest = await f.put("/file-manifest", {
|
||||
requestId: randomUUID(),
|
||||
parentFileManifestId: 0,
|
||||
entries: { [id]: "note.md" }
|
||||
});
|
||||
if (operation !== "create") {
|
||||
await f.client.start();
|
||||
await f.client.setSetting("isSyncEnabled", false);
|
||||
if (operation === "move")
|
||||
await f.put("/file-manifest", {
|
||||
requestId: randomUUID(),
|
||||
parentFileManifestId: manifest.fileManifestId,
|
||||
entries: { [id]: "moved.md" }
|
||||
});
|
||||
else
|
||||
await f.put(
|
||||
`/documents/${id}`,
|
||||
f.content("REMOTE\nBASE\n", head.vaultUpdateId)
|
||||
);
|
||||
}
|
||||
let fired = false;
|
||||
let wrote = false;
|
||||
const interrupt = () => {
|
||||
fired = true;
|
||||
f.disk.crash(powerLoss);
|
||||
throw new Error("injected interruption");
|
||||
};
|
||||
f.disk.boundary = (label) => {
|
||||
if (fired) return;
|
||||
if (label === "durable:write:note.md") wrote = true;
|
||||
if (
|
||||
operation === "move" &&
|
||||
label === "visible:rename:note.md->moved.md"
|
||||
)
|
||||
interrupt();
|
||||
if (
|
||||
(operation === "create" ||
|
||||
operation === "replace") &&
|
||||
label === "visible:write:note.md"
|
||||
)
|
||||
interrupt();
|
||||
};
|
||||
f.store.boundary = (label) => {
|
||||
if (
|
||||
!fired &&
|
||||
operation === "metadata" &&
|
||||
wrote &&
|
||||
label === "before:save"
|
||||
)
|
||||
interrupt();
|
||||
};
|
||||
await assert.rejects(
|
||||
operation === "create"
|
||||
? f.client.start()
|
||||
: f.client.setSetting("isSyncEnabled", true),
|
||||
/injected interruption/
|
||||
);
|
||||
assert(
|
||||
fired,
|
||||
"The intended filesystem/save boundary must be reached"
|
||||
);
|
||||
f.disk.boundary = () => {};
|
||||
f.store.boundary = () => {};
|
||||
// Model torn content and further independent edits. Neither is
|
||||
// recoverable from an old filesystem intention, nor should it be.
|
||||
await f.disk.userWrite(
|
||||
operation === "move" ? "moved.md" : "note.md",
|
||||
bytes("partial\nUSER EDIT\n")
|
||||
);
|
||||
await f.disk.userWrite(
|
||||
"after.md",
|
||||
bytes("AFTER INTERRUPTION\n")
|
||||
);
|
||||
await f.reopenClient();
|
||||
await f.client.start();
|
||||
const inspectable = {
|
||||
files: () => f.disk.userFiles(),
|
||||
database: () => f.store.snapshot().database
|
||||
};
|
||||
const peerDisk = new MemoryDisk();
|
||||
const peerStore = new MemoryPersistence({
|
||||
settings: f.store.snapshot().settings
|
||||
});
|
||||
peer = await SyncClient.create({
|
||||
fs: peerDisk,
|
||||
persistence: peerStore
|
||||
});
|
||||
await peer.start();
|
||||
const clients = [
|
||||
inspectable,
|
||||
{
|
||||
files: () => peerDisk.userFiles(),
|
||||
database: () => peerStore.snapshot().database
|
||||
}
|
||||
];
|
||||
await assertCanonical(
|
||||
clients,
|
||||
f.url,
|
||||
"test-token-change-me"
|
||||
);
|
||||
assert.deepEqual(
|
||||
f.disk.userFiles().get("after.md"),
|
||||
bytes("AFTER INTERRUPTION\n")
|
||||
);
|
||||
assert(
|
||||
(await f.disk.listFilesRecursively()).every(
|
||||
(path) => !path.startsWith(".vault-link-sync/")
|
||||
)
|
||||
);
|
||||
const files = f.disk.userFiles();
|
||||
await f.reopenClient();
|
||||
await f.client.start();
|
||||
await assertCanonical(
|
||||
clients,
|
||||
f.url,
|
||||
"test-token-change-me"
|
||||
);
|
||||
assert.deepEqual(
|
||||
f.disk.userFiles(),
|
||||
files,
|
||||
"A second restart must not replay old writes"
|
||||
);
|
||||
} finally {
|
||||
f.disk.boundary = () => {};
|
||||
f.store.boundary = () => {};
|
||||
await peer?.destroy();
|
||||
await f.dispose();
|
||||
}
|
||||
}
|
||||
);
|
||||
|
||||
for (const operation of ["create", "replace", "move"] as const)
|
||||
test(
|
||||
`${operation}: metadata can survive while completed user-file writes are lost`,
|
||||
{ timeout: 20_000 },
|
||||
async () => {
|
||||
const f = await fixture();
|
||||
let peer: SyncClient | undefined;
|
||||
try {
|
||||
const id = randomUUID();
|
||||
const head = await f.put(
|
||||
`/documents/${id}`,
|
||||
f.content("BASE\n")
|
||||
);
|
||||
const manifest = await f.put("/file-manifest", {
|
||||
requestId: randomUUID(),
|
||||
parentFileManifestId: 0,
|
||||
entries: { [id]: "note.md" }
|
||||
});
|
||||
if (operation !== "create") {
|
||||
await f.client.start();
|
||||
await f.client.setSetting("isSyncEnabled", false);
|
||||
if (operation === "replace")
|
||||
await f.put(
|
||||
`/documents/${id}`,
|
||||
f.content("REMOTE\nBASE\n", head.vaultUpdateId)
|
||||
);
|
||||
else
|
||||
await f.put("/file-manifest", {
|
||||
requestId: randomUUID(),
|
||||
parentFileManifestId: manifest.fileManifestId,
|
||||
entries: { [id]: "moved.md" }
|
||||
});
|
||||
}
|
||||
const oldDisk = f.disk.image();
|
||||
if (operation === "create") await f.client.start();
|
||||
else await f.client.setSetting("isSyncEnabled", true);
|
||||
await f.client.destroy();
|
||||
// Metadata is newer than disk: completed filesystem calls have no
|
||||
// flush contract. This is not covered by failing a call mid-flight.
|
||||
f.disk.restore(oldDisk);
|
||||
await f.disk.userWrite(
|
||||
"after.md",
|
||||
bytes("USER AFTER POWER LOSS\n")
|
||||
);
|
||||
await f.reopenClient();
|
||||
await f.client.start();
|
||||
const peerDisk = new MemoryDisk();
|
||||
const peerStore = new MemoryPersistence({
|
||||
settings: f.store.snapshot().settings
|
||||
});
|
||||
peer = await SyncClient.create({
|
||||
fs: peerDisk,
|
||||
persistence: peerStore
|
||||
});
|
||||
await peer.start();
|
||||
await assertCanonical(
|
||||
[
|
||||
{
|
||||
files: () => f.disk.userFiles(),
|
||||
database: () => f.store.snapshot().database
|
||||
},
|
||||
{
|
||||
files: () => peerDisk.userFiles(),
|
||||
database: () => peerStore.snapshot().database
|
||||
}
|
||||
],
|
||||
f.url,
|
||||
"test-token-change-me"
|
||||
);
|
||||
assert.deepEqual(
|
||||
f.disk.userFiles().get("after.md"),
|
||||
bytes("USER AFTER POWER LOSS\n")
|
||||
);
|
||||
} finally {
|
||||
await peer?.destroy();
|
||||
await f.dispose();
|
||||
}
|
||||
}
|
||||
);
|
||||
|
|
@ -216,13 +216,6 @@ export class MemoryDisk implements FileSystemOperations {
|
|||
});
|
||||
}
|
||||
|
||||
public async flushPaths(paths: readonly string[]): Promise<void> {
|
||||
const affected = new Set(paths);
|
||||
for (const path of paths)
|
||||
for (let p = parent(path); p; p = parent(p)) affected.add(p);
|
||||
await this.change(`flush:${paths.join(",")}`, [...affected], () => {});
|
||||
}
|
||||
|
||||
public async delete(path: string): Promise<void> {
|
||||
this.check(path);
|
||||
assert(path !== "", "Cannot remove vault root");
|
||||
|
|
|
|||
|
|
@ -1,4 +1,5 @@
|
|||
import assert from "node:assert/strict";
|
||||
import { SUPPORTED_API_VERSION } from "../sync-client/src/consts";
|
||||
import { PermanentSyncError } from "../sync-client/src/errors/errors";
|
||||
import { test } from "node:test";
|
||||
import type { Database } from "../sync-client/src/persistence/database";
|
||||
|
|
@ -221,52 +222,6 @@ test("local undo of accepted-but-rejected manifest must survive replay", async (
|
|||
assert.equal(f.database.state.local.a, "a.md");
|
||||
});
|
||||
|
||||
test("disabled startup should recover an active journal before returning", async () => {
|
||||
const disk = new MemoryDisk();
|
||||
const key = JSON.stringify(["http://offline.test", "test"]);
|
||||
const state = emptyState(key);
|
||||
state.initialized = true;
|
||||
state.local = { a: "a.md" };
|
||||
state.documents = { a: { materialized: true } };
|
||||
const next = structuredClone(state);
|
||||
next.local = { a: "b.md" };
|
||||
const prefix = ".vault-link-sync/transactions/active/a";
|
||||
await disk.userWrite(prefix + ".source", bytes("ONLY COPY"));
|
||||
state.application = {
|
||||
id: "active",
|
||||
extensions: ["md"],
|
||||
next,
|
||||
steps: [
|
||||
{
|
||||
documentId: "a",
|
||||
from: "a.md",
|
||||
to: "b.md",
|
||||
staged: prefix + ".source",
|
||||
output: prefix + ".output",
|
||||
phase: "staged"
|
||||
}
|
||||
]
|
||||
};
|
||||
const store = new MemoryPersistence({
|
||||
settings: {
|
||||
remoteUri: "http://offline.test",
|
||||
vaultName: "test",
|
||||
isSyncEnabled: false
|
||||
},
|
||||
database: state
|
||||
});
|
||||
const client = await SyncClient.create({ fs: disk, persistence: store });
|
||||
try {
|
||||
await client.start();
|
||||
assert.deepEqual(
|
||||
(await disk.readSnapshot("b.md"))?.content,
|
||||
bytes("ONLY COPY")
|
||||
);
|
||||
} finally {
|
||||
await client.destroy();
|
||||
}
|
||||
});
|
||||
|
||||
test("a settings update resumes the enabled engine", async () => {
|
||||
const f = await settingsLifecycleFixture();
|
||||
try {
|
||||
|
|
@ -381,7 +336,7 @@ test("silent open WebSocket must eventually trigger catchup or reconnect", async
|
|||
const path = String(input);
|
||||
if (path.endsWith("/ping"))
|
||||
return Response.json({
|
||||
supportedApiVersion: 5,
|
||||
supportedApiVersion: SUPPORTED_API_VERSION,
|
||||
isAuthenticated: true,
|
||||
mergeableFileExtensions: ["md"],
|
||||
serverVersion: "test"
|
||||
|
|
@ -436,7 +391,6 @@ test("permanent destination failure replans the affected file and finishes the b
|
|||
next.fileManifest = { fileManifestId: 2, entries: { ...next.local } };
|
||||
await f.files.apply(next);
|
||||
await f.settings.setSettings({ ignorePatterns: ["unsupported/**"] });
|
||||
await f.files.recover();
|
||||
assert.deepEqual(
|
||||
[...f.disk.userFiles().values()]
|
||||
.map((b) => Buffer.from(b).toString())
|
||||
|
|
@ -469,63 +423,85 @@ test("conflict allocation terminates when a truncated extension ends in a space"
|
|||
assert.equal(result.status, 0, String(result.error ?? result.stderr));
|
||||
});
|
||||
|
||||
test("paged catchup folds remote edits and reversions before touching local bytes", async () => {
|
||||
const f = await fixture(
|
||||
{ "a.md": "base LOCAL" },
|
||||
{ getDocumentVersionContent: async () => bytes("base") }
|
||||
);
|
||||
f.database.state.documents.a.base = {
|
||||
...head("a", 1, "base"),
|
||||
hash: (await toStoredSnapshot({ content: bytes("base") })).hash
|
||||
};
|
||||
f.database.state.remoteHeads.a = head("a", 1, "base");
|
||||
f.database.state.lastSeenUpdateId = 1;
|
||||
const first = {
|
||||
headEventId: 3,
|
||||
endEventId: 2,
|
||||
events: [
|
||||
{
|
||||
eventId: 2,
|
||||
requestId: "remote-1",
|
||||
type: "content" as const,
|
||||
document: head("a", 2, "base REMOTE")
|
||||
for (const interrupted of [false, true]) {
|
||||
test(`paged catchup folds edits and reversions${interrupted ? " again after an interrupted fetch" : ""}`, async () => {
|
||||
const first = {
|
||||
headEventId: 3,
|
||||
endEventId: 2,
|
||||
events: [
|
||||
{
|
||||
eventId: 2,
|
||||
requestId: "remote-1",
|
||||
type: "content" as const,
|
||||
document: head("a", 2, "base REMOTE")
|
||||
}
|
||||
]
|
||||
};
|
||||
const last = {
|
||||
headEventId: 3,
|
||||
endEventId: 3,
|
||||
events: [
|
||||
{
|
||||
eventId: 3,
|
||||
requestId: "remote-2",
|
||||
type: "content" as const,
|
||||
document: head("a", 3, "base")
|
||||
}
|
||||
]
|
||||
};
|
||||
let fail = interrupted;
|
||||
let f: Awaited<ReturnType<typeof fixture>>;
|
||||
const fetchedAfter: number[] = [];
|
||||
const service: Partial<SyncService> = {
|
||||
getDocumentVersionContent: async () => bytes("base"),
|
||||
events: async (after) => {
|
||||
fetchedAfter.push(after);
|
||||
if (after === 1) return first;
|
||||
assert.equal(after, 2);
|
||||
assert.equal(f.database.state.lastSeenUpdateId, 1);
|
||||
assert.deepEqual(
|
||||
f.persistence.snapshot(),
|
||||
saved,
|
||||
"partial pages must not save metadata"
|
||||
);
|
||||
assert.deepEqual(
|
||||
(await f.disk.readSnapshot("a.md"))?.content,
|
||||
bytes("base LOCAL")
|
||||
);
|
||||
if (fail) {
|
||||
fail = false;
|
||||
throw new Error("interrupted page fetch");
|
||||
}
|
||||
return last;
|
||||
}
|
||||
]
|
||||
};
|
||||
await f.internals.incorporateEventBatch(first);
|
||||
assert.equal(
|
||||
f.database.state.lastSeenUpdateId,
|
||||
1,
|
||||
"partial history must not become an incorporated base"
|
||||
);
|
||||
assert.deepEqual(
|
||||
(await f.disk.readSnapshot("a.md"))?.content,
|
||||
bytes("base LOCAL")
|
||||
);
|
||||
assert.equal(
|
||||
await f.internals.preparePush(),
|
||||
false,
|
||||
"do not submit against a partially replayed history"
|
||||
);
|
||||
const last = {
|
||||
headEventId: 3,
|
||||
endEventId: 3,
|
||||
events: [
|
||||
{
|
||||
eventId: 3,
|
||||
requestId: "remote-2",
|
||||
type: "content" as const,
|
||||
document: head("a", 3, "base")
|
||||
}
|
||||
]
|
||||
};
|
||||
await f.internals.incorporateEventBatch(last);
|
||||
assert.equal(f.database.state.lastSeenUpdateId, 3);
|
||||
assert.deepEqual(
|
||||
(await f.disk.readSnapshot("a.md"))?.content,
|
||||
bytes("base LOCAL")
|
||||
);
|
||||
});
|
||||
};
|
||||
f = await fixture({ "a.md": "base LOCAL" }, service);
|
||||
f.database.state.documents.a.base = {
|
||||
...head("a", 1, "base"),
|
||||
hash: (await toStoredSnapshot({ content: bytes("base") })).hash
|
||||
};
|
||||
f.database.state.remoteHeads.a = head("a", 1, "base");
|
||||
f.database.state.lastSeenUpdateId = 1;
|
||||
await f.database.save();
|
||||
const saved = f.persistence.snapshot();
|
||||
const catchUp = async () =>
|
||||
f.internals.incorporateEventBatch(
|
||||
await service.events!(f.database.state.lastSeenUpdateId)
|
||||
);
|
||||
if (interrupted) {
|
||||
await assert.rejects(catchUp(), /interrupted page fetch/);
|
||||
f.disk.crash(true);
|
||||
f = await fixture({}, service, f);
|
||||
}
|
||||
await catchUp();
|
||||
assert.deepEqual(fetchedAfter, interrupted ? [1, 2, 1, 2] : [1, 2]);
|
||||
assert.equal(f.database.state.lastSeenUpdateId, 3);
|
||||
assert.deepEqual(
|
||||
(await f.disk.readSnapshot("a.md"))?.content,
|
||||
bytes("base LOCAL")
|
||||
);
|
||||
});
|
||||
}
|
||||
|
||||
test("an ignored occupant arriving after the scan keeps its path and privacy", async () => {
|
||||
let f: Awaited<ReturnType<typeof fixture>>;
|
||||
|
|
@ -580,6 +556,10 @@ test("content notifications during every snapshot do not starve a complete names
|
|||
for (const boundary of ["before:save", "durable:save"]) {
|
||||
test(`receipt incorporation after paged replay survives ${boundary} and process replacement`, async () => {
|
||||
const service: Partial<SyncService> = {
|
||||
events: async (after) => {
|
||||
assert.equal(after, 2);
|
||||
return page;
|
||||
},
|
||||
putFileContent: async () => {
|
||||
throw new PermanentSyncError("HTTP 413");
|
||||
},
|
||||
|
|
@ -604,18 +584,18 @@ for (const boundary of ["before:save", "durable:save"]) {
|
|||
const { requestId } = f.database.state.pending!.request;
|
||||
await f.internals.finishPending();
|
||||
await f.disk.userWrite("a.md", bytes("base"));
|
||||
await f.internals.incorporateEventBatch({
|
||||
const first = {
|
||||
headEventId: 3,
|
||||
endEventId: 2,
|
||||
events: [
|
||||
{
|
||||
eventId: 2,
|
||||
requestId,
|
||||
type: "content",
|
||||
type: "content" as const,
|
||||
document: head("a", 2, "base A")
|
||||
}
|
||||
]
|
||||
});
|
||||
};
|
||||
const page = {
|
||||
headEventId: 3,
|
||||
endEventId: 3,
|
||||
|
|
@ -632,16 +612,14 @@ for (const boundary of ["before:save", "durable:save"]) {
|
|||
if (label === boundary) throw new Error("injected crash");
|
||||
};
|
||||
await assert.rejects(
|
||||
f.internals.incorporateEventBatch(page),
|
||||
f.internals.incorporateEventBatch(first),
|
||||
/injected crash/
|
||||
);
|
||||
f.persistence.boundary = () => {};
|
||||
f.disk.crash(true);
|
||||
f = await fixture({}, service, f);
|
||||
await f.files.recover();
|
||||
assert.equal(f.database.state.eventReplay?.after, 2);
|
||||
await f.internals.incorporateEventBatch(page);
|
||||
assert.equal(f.database.state.eventReplay, undefined);
|
||||
assert.equal(f.database.state.lastSeenUpdateId, 1);
|
||||
await f.internals.incorporateEventBatch(first);
|
||||
assert.equal(f.database.state.lastSeenUpdateId, 3);
|
||||
assert.deepEqual(
|
||||
(await f.disk.readSnapshot("a.md"))?.content,
|
||||
|
|
@ -676,7 +654,7 @@ test("initial vault settings bind the engine and survive restart", async () => {
|
|||
fetch: async (input: RequestInfo | URL) => {
|
||||
if (String(input).endsWith("/ping"))
|
||||
return Response.json({
|
||||
supportedApiVersion: 5,
|
||||
supportedApiVersion: SUPPORTED_API_VERSION,
|
||||
isAuthenticated: true,
|
||||
mergeableFileExtensions: ["md"],
|
||||
serverVersion: "test"
|
||||
|
|
@ -804,7 +782,7 @@ async function settingsLifecycleFixture() {
|
|||
const path = String(input);
|
||||
if (path.endsWith("/ping"))
|
||||
return Response.json({
|
||||
supportedApiVersion: 5,
|
||||
supportedApiVersion: SUPPORTED_API_VERSION,
|
||||
isAuthenticated: true,
|
||||
mergeableFileExtensions: ["md"],
|
||||
serverVersion: "test"
|
||||
|
|
|
|||
|
|
@ -59,7 +59,6 @@ export async function fixture(
|
|||
recovered?.persistence ?? new MemoryPersistence({ database: initial });
|
||||
const logger = new Logger();
|
||||
const database = new Database(
|
||||
logger,
|
||||
initial,
|
||||
async (next) => persistence.save({ database: next }),
|
||||
"test",
|
||||
|
|
|
|||
Loading…
Reference in a new issue