no power loss
This commit is contained in:
parent
d56084e4bc
commit
2fcdbfd27d
43 changed files with 1374 additions and 3676 deletions
|
|
@ -1,3 +1,8 @@
|
|||
> Historical handover: its client journaling, recovery-archive and filesystem
|
||||
> durability descriptions are superseded by the direct-application model in
|
||||
> [the sync client README](frontend/sync-client/README.md). The current client
|
||||
> saves complete metadata and reconciles current disk contents after interruption.
|
||||
|
||||
# Sync engine v4 handover
|
||||
|
||||
Prepared on 2026-09-19. This records the implementation delivered in the preceding
|
||||
|
|
|
|||
|
|
@ -1,8 +1,7 @@
|
|||
# Sync algorithm
|
||||
|
||||
The current API v5 server stores immutable content versions and a separately
|
||||
versioned file manifest. Clients reconcile snapshots and apply them through a
|
||||
durable filesystem journal. The server does not merge competing edits.
|
||||
versioned file manifest. Clients reconcile snapshots with the current local files. The server does not merge competing edits.
|
||||
|
||||
## Identity and ordering
|
||||
|
||||
|
|
@ -19,10 +18,11 @@ UUID is rejected.
|
|||
|
||||
## Reconciliation
|
||||
|
||||
A new client reads a coherent vault snapshot, applies it durably, then replays
|
||||
subsequent events. Replayed pages are folded to their final document heads and
|
||||
manifest before changing local files. This avoids applying intermediate remote
|
||||
edits that were already reverted.
|
||||
A new client reads a coherent vault snapshot, applies it, then replays
|
||||
subsequent events over HTTP. Pages are folded in memory to their final document
|
||||
heads and manifest before changing local files. Interrupted fetching restarts from
|
||||
the last incorporated cursor without saved pagination state. This avoids applying
|
||||
intermediate remote edits that were already reverted.
|
||||
|
||||
For text, the client compares the common base, current local bytes and current
|
||||
remote bytes using `reconcile-text`. Configured extensions and valid UTF-8 without
|
||||
|
|
@ -31,34 +31,36 @@ they do not authorize the server to merge a stale base. Large diff operation set
|
|||
are sent as snapshots.
|
||||
|
||||
If only one side changed, that side is retained. Concurrent unmergeable changes
|
||||
keep the server version while the journal archives displaced local bytes for
|
||||
recovery. First contact with existing local text has no shared ancestor and uses
|
||||
keep the server version. First contact with existing local text has no shared ancestor and uses
|
||||
an empty parent. Concurrent edits may still require human review.
|
||||
|
||||
Paths use a separate three-way decision: keep a local change when the server path
|
||||
is unchanged, otherwise prefer the server path if both changed. Portable-name
|
||||
collisions receive deterministic conflict names. A manifest deletion is not
|
||||
reversed merely because another client edited the document; displaced bytes are
|
||||
retained locally.
|
||||
reversed merely because another client edited the document. Local backup archives
|
||||
are not created.
|
||||
|
||||
## Filesystem application
|
||||
|
||||
Before moving files, the client persists a journal describing sources,
|
||||
destinations and replacement snapshots. It stages sources before installing any
|
||||
destination, so swaps and rename cycles do not overwrite each other. Prepared
|
||||
content is persisted before installation so a crash does not repeat a merge.
|
||||
The client applies an in-memory plan directly to the visible namespace, then
|
||||
saves one metadata snapshot. Swaps and cycles move occupied sources to temporary
|
||||
conflict names that remain ordinary discoverable files if application stops.
|
||||
There is no persisted filesystem plan, staged payload, recovery replay or fsync
|
||||
protocol. An interruption may leave a subset of the operations visible, or partial
|
||||
file content; the next scan reconciles whatever is present then.
|
||||
|
||||
External notifications retain logical document identities. Notifications received
|
||||
during asynchronous filesystem reads or staging are incorporated before accepting
|
||||
a source. An editor save to a staged document's old path remains part of that
|
||||
document. Ignored and oversized physical occupants are protected during conflict
|
||||
allocation. Adapters report atomic editor saves as content updates and suppress
|
||||
notifications generated by the engine itself.
|
||||
Create, delete and move notifications persist logical document identities; content
|
||||
updates only wake the loop to reread current bytes. Namespace changes detected
|
||||
before mutations invalidate the plan. Notifications during application are incorporated
|
||||
between files using their document identities; the next scan rereads content. Content is reread
|
||||
before replacement. Ignored and oversized occupants stay in place. Adapters
|
||||
report atomic editor saves as updates and suppress engine-generated notifications.
|
||||
|
||||
The filesystem and metadata store have separate durability boundaries. Recovery
|
||||
replays the journal before scanning, including when network sync is disabled.
|
||||
Adapters must provide complete scans, coherent snapshots, exclusive vault access,
|
||||
no-replace renames and durable writes. Recovery artifacts are excluded from sync.
|
||||
Metadata persistence must replace a complete value atomically, leaving a readable
|
||||
old/new value after interruption. Metadata and user files need not describe the
|
||||
same moment: scans repair that disagreement. File creation and rename must reject
|
||||
occupied destinations; scans must be complete and reads coherent. No guarantee is
|
||||
made that user bytes or rename identities survive power loss.
|
||||
|
||||
## Server restores and failures
|
||||
|
||||
|
|
@ -68,8 +70,7 @@ The client persists and echoes it; the server rejects a checkpoint from a discar
|
|||
history before executing its request. Each new event gets a fresh token even if
|
||||
its numeric ID or request UUID was used on the discarded branch.
|
||||
|
||||
After a mismatch, the client durably records recovery intent, retains pending
|
||||
content, clears old history-dependent state and bootstraps against the restored
|
||||
After a mismatch, the client records reset intent, clears old history-dependent state and bootstraps against the restored
|
||||
server. Clean files reconcile against the restored snapshot. Unsent edits with an
|
||||
old base are preserved as separate local documents, with conflict names if needed.
|
||||
Interrupted recovery saves are replayable. Backups must be restored with the server
|
||||
|
|
|
|||
|
|
@ -9,8 +9,7 @@ VaultLink works well for most Obsidian vaults, but has some constraints you shou
|
|||
Only **`.md`** and **`.txt`** files get automatic conflict-free merging.
|
||||
|
||||
For other file types, a one-sided change is retained. Concurrent unmergeable
|
||||
changes keep the server's version and archive displaced local bytes for recovery.
|
||||
Review recovery artifacts when both devices edit a binary file.
|
||||
changes keep the server's version. The client does not archive displaced bytes.
|
||||
|
||||
### Binary Detection
|
||||
|
||||
|
|
@ -60,7 +59,7 @@ Rough estimates (varies by vault size and activity):
|
|||
|
||||
**Growth**: Version history grows with every change. A 10 MB vault with frequent edits might grow to 100+ MB over months.
|
||||
|
||||
Active journals, immutable merge bases and request receipts must not be independently deleted. There is no automatic safe history-pruning protocol.
|
||||
Immutable merge bases and request receipts must not be independently deleted. There is no automatic safe history-pruning protocol.
|
||||
|
||||
### Implications
|
||||
|
||||
|
|
@ -143,6 +142,14 @@ the restored server; unsent edits are preserved as separate local documents.
|
|||
Clients and server must be upgraded together. This does not replace independent
|
||||
backups or protect against arbitrary physical-media corruption.
|
||||
|
||||
## Interrupted sync
|
||||
|
||||
The client has no filesystem journal or power-loss durability protocol. Interrupted
|
||||
writes may leave missing, partial or moved files. On restart, normal scans reconcile
|
||||
the current files, including edits made while the client was stopped. Metadata
|
||||
saves must remain complete and readable, so the client can continue syncing; exact
|
||||
pre-interruption file contents and identities are not guaranteed.
|
||||
|
||||
## Known Edge Cases
|
||||
|
||||
### Simultaneous Deletes and Edits
|
||||
|
|
@ -150,10 +157,10 @@ backups or protect against arbitrary physical-media corruption.
|
|||
```
|
||||
User A deletes note.md
|
||||
User B edits note.md
|
||||
Result: Manifest deletion removes the file; unsent edited bytes are retained for recovery.
|
||||
Result: Manifest deletion removes the file, including unsent local edits.
|
||||
```
|
||||
|
||||
Content updates do not restore deleted membership. The client retains displaced bytes in its local recovery directory.
|
||||
Content updates do not restore deleted membership. There is no local recovery archive.
|
||||
|
||||
### Large File Uploads
|
||||
|
||||
|
|
|
|||
|
|
@ -74,7 +74,7 @@ Network retry timers remain enabled. Separate fake-clock tests in
|
|||
A barrier checks byte-for-byte replica equality **and** the server's canonical
|
||||
UUID/path map, file bytes, independently computed base/observed hashes, content
|
||||
and manifest CAS parents, event watermark,
|
||||
contiguous event log, and absence of pending requests/application journals.
|
||||
contiguous event log, and absence of pending requests.
|
||||
Internal recovery files are excluded from the user-visible file list, but are
|
||||
present in failure disk images. Scenario assertions separately check intended
|
||||
content, document identity, or preservation markers; equality alone is not an
|
||||
|
|
@ -106,8 +106,8 @@ distinct UUIDs at one path.
|
|||
- `wait-for-response-drop`: explicit checkpoint; every armed fault must fire.
|
||||
Every interrupted request must be retried with an identical request ID, target
|
||||
and body before a different CAS is sent. Fault kinds match exactly.
|
||||
- `rename-next-write`: inject an editor rename after the engine's durable install
|
||||
rename. The historical name is retained for fixtures; v4 never writes in place.
|
||||
- `rename-next-write`: inject an editor rename after the engine's content write,
|
||||
before its metadata save.
|
||||
- `pause-server`, `resume-server`: SIGSTOP/SIGCONT, not a crash.
|
||||
- `crash-server`, `restart-server`: SIGKILL and restart with the same database/port.
|
||||
Server-control scenarios have dedicated processes.
|
||||
|
|
|
|||
|
|
@ -43,12 +43,8 @@ export class DeterministicAgent {
|
|||
});
|
||||
this.disk.boundary = async (label) => {
|
||||
const rename = this.nextWriteRename;
|
||||
// v4 installs through an exclusive rename, never write-in-place.
|
||||
if (
|
||||
rename &&
|
||||
label.startsWith("durable:rename:") &&
|
||||
label.endsWith(`->${rename.oldPath}`)
|
||||
) {
|
||||
// Inject after the content write, before the metadata save.
|
||||
if (rename && label === `durable:write:${rename.oldPath}`) {
|
||||
this.nextWriteRename = undefined;
|
||||
await this.rename(rename.oldPath, rename.newPath);
|
||||
}
|
||||
|
|
|
|||
|
|
@ -13,10 +13,8 @@ export const sameDocIdCollapseAfterRemoteQuickWriteAndPendingRenameTest: TestDef
|
|||
steps: [
|
||||
{ type: "enable-sync", client: 0 },
|
||||
|
||||
// Create a deleted latest version before client 1 joins.
|
||||
// Catch-up will advance MinCovered with a non-contiguous id,
|
||||
// keeping client 1's create lastSeen low enough to exercise
|
||||
// the server's same-doc merge path from the e2e failure.
|
||||
// Seed a deleted document before client 1 bootstraps. Later creates
|
||||
// at the reused path must still receive separate identities.
|
||||
{
|
||||
type: "create",
|
||||
client: 0,
|
||||
|
|
|
|||
|
|
@ -33,11 +33,8 @@ export const selfMergePendingRenameAliasesSecondCreateTest: TestDefinition = {
|
|||
{ type: "enable-sync", client: 1 },
|
||||
{ type: "barrier" },
|
||||
|
||||
// Pause client 0's WS so its MinCovered watermark stays at 0
|
||||
// through the whole bug sequence. The merge condition the
|
||||
// server is going to fire is `creation_vuid > last_seen`; with
|
||||
// a non-zero gap the same-device second create gets merged
|
||||
// into the same-device first create.
|
||||
// Delay socket notifications while client 0 continues local work.
|
||||
// HTTP catchup must preserve the two generations at the reused path.
|
||||
{ type: "pause-websocket", client: 0 },
|
||||
|
||||
// Client 1 commits a doc to push the server's vuid above 0.
|
||||
|
|
|
|||
|
|
@ -134,7 +134,6 @@ export class NodeFileSystemOperations implements FileSystemOperations {
|
|||
await fs.unlink(fullPath).catch((error: unknown) => {
|
||||
if (!this.isMissing(error)) throw error;
|
||||
});
|
||||
await this.syncDirectory(path.dirname(fullPath));
|
||||
} catch (error) {
|
||||
throw new Error(
|
||||
`Failed to delete file ${fullPath}: ${error instanceof Error ? error.message : String(error)}`
|
||||
|
|
@ -170,34 +169,13 @@ export class NodeFileSystemOperations implements FileSystemOperations {
|
|||
const tmpPath = path.join(tmpDir, `atomic-write-${randomUUID()}.tmp`);
|
||||
try {
|
||||
await fs.writeFile(tmpPath, content, encoding);
|
||||
const fd = await fs.open(tmpPath, "r");
|
||||
try {
|
||||
await fd.datasync();
|
||||
} finally {
|
||||
await fd.close();
|
||||
}
|
||||
await fs.rename(tmpPath, fullPath);
|
||||
await this.syncDirectory(path.dirname(fullPath));
|
||||
} catch (error) {
|
||||
await fs.unlink(tmpPath).catch(() => undefined);
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
|
||||
// Make the rename durable by fsync'ing the destination's parent directory.
|
||||
// Skipped on Windows: fsync on a directory handle isn't supported there
|
||||
private async syncDirectory(dir: string): Promise<void> {
|
||||
if (process.platform === "win32") {
|
||||
return;
|
||||
}
|
||||
const fd = await fs.open(dir, "r");
|
||||
try {
|
||||
await fd.sync();
|
||||
} finally {
|
||||
await fd.close();
|
||||
}
|
||||
}
|
||||
|
||||
private async deleteDirectoryTree(fullPath: string): Promise<void> {
|
||||
const entry = await fs.lstat(fullPath).catch((error: unknown) => {
|
||||
if (this.isMissing(error)) return undefined;
|
||||
|
|
@ -220,7 +198,6 @@ export class NodeFileSystemOperations implements FileSystemOperations {
|
|||
}
|
||||
|
||||
await fs.rmdir(fullPath);
|
||||
await this.syncDirectory(path.dirname(fullPath));
|
||||
}
|
||||
|
||||
private isMissing(error: unknown): boolean {
|
||||
|
|
|
|||
|
|
@ -19,8 +19,8 @@
|
|||
"dev": "concurrently --kill-others \"npm run dev -w sync-client\" \"npm run dev -w obsidian-plugin\"",
|
||||
"test": "npm run test --workspaces",
|
||||
"check:harness": "tsc --noEmit -p test-support/tsconfig.json && tsc --noEmit -p deterministic-tests/tsconfig.json && tsc --noEmit -p test-client/tsconfig.json",
|
||||
"test:harness": "npm run check:harness && tsx --test 'test-support/harness.test.ts' 'test-support/oracle-regressions.test.ts' 'test-support/recovery.test.ts' 'test-support/journal-interleavings.test.ts' 'test-support/observer.test.ts' 'test-support/event-delivery.test.ts' 'test-support/polling.test.ts' 'test-support/engine-regressions.test.ts' 'test-support/sync-audit.test.ts' 'test-client/src/**/*.test.ts' '../scripts/e2e-runner.test.mjs'",
|
||||
"test:protocol": "tsx --test 'test-support/protocol.test.ts' 'test-support/journal-protocol.test.ts' 'test-support/content-limits.test.ts' 'test-support/server-restore.test.ts' 'test-support/protocol-audit.test.ts' 'test-support/real-disk.test.ts'",
|
||||
"test:harness": "npm run check:harness && tsx --test 'test-support/harness.test.ts' 'test-support/oracle-regressions.test.ts' 'test-support/recovery.test.ts' 'test-support/file-application.test.ts' 'test-support/observer.test.ts' 'test-support/event-delivery.test.ts' 'test-support/polling.test.ts' 'test-support/engine-regressions.test.ts' 'test-support/sync-audit.test.ts' 'test-client/src/**/*.test.ts' '../scripts/e2e-runner.test.mjs'",
|
||||
"test:protocol": "tsx --test 'test-support/protocol.test.ts' 'test-support/restart-reconciliation.test.ts' 'test-support/content-limits.test.ts' 'test-support/server-restore.test.ts' 'test-support/protocol-audit.test.ts' 'test-support/real-disk.test.ts'",
|
||||
"lint": "eslint --fix sync-client obsidian-plugin test-client deterministic-tests local-client-cli && prettier --write \"**/*.ts\"",
|
||||
"update": "ncu -u -ws"
|
||||
},
|
||||
|
|
|
|||
|
|
@ -4,28 +4,30 @@ The engine uses one serialized loop per vault and requires Web Crypto (SHA-256).
|
|||
messages wake it; complete disk scans and the durable server event log are the
|
||||
sources of truth. Periodic reconciliation is opt-in: set `syncIntervalMs` to a
|
||||
positive interval to repair missed notifications. Unset or `0` disables polling.
|
||||
`syncLocallyCreatedFile`, `syncLocallyUpdatedFile`, and `syncLocallyDeletedFile`
|
||||
durably record logical file changes before resolving, including while offline.
|
||||
Shutdown waits for those saves. Notifications describe logical external changes: adapters suppress
|
||||
Create, delete and move notifications persist logical identity changes before
|
||||
resolving, including while offline. Shutdown waits for those saves. Content-update
|
||||
notifications only wake reconciliation; scans read the latest bytes, including after
|
||||
restart. Notifications describe logical external changes: adapters suppress
|
||||
self-generated create/delete/move notifications, and report atomic editor saves
|
||||
as content updates. This distinguishes a deliberate delete/recreate from an edit.
|
||||
Use `waitUntilFinished()` to await the active attempt; errors leave
|
||||
pending work durable and are reported to the caller. Transient failures retry;
|
||||
pending requests saved and are reported to the caller. Transient failures retry;
|
||||
authentication and protocol errors wait for an explicit wake or settings change.
|
||||
|
||||
## State and policy
|
||||
|
||||
The persisted state contains the incorporated server file manifest, the current
|
||||
local identity map, and the exact pending submission. A received server file
|
||||
manifest is not a merge base until its local application is durable. Successful
|
||||
manifest becomes the merge base when its application metadata is saved. Successful
|
||||
submissions advance the base to the submitted content snapshot, preserving later
|
||||
local edits. Unknown outcomes are retried unchanged before incorporating newer
|
||||
remote state. Each contiguous event batch is folded into its final manifest and
|
||||
content heads before merging. Intermediate changes that the server has already
|
||||
reverted must not overwrite local edits.
|
||||
remote state. HTTP is the sole source of remote events; WebSocket messages only
|
||||
wake the loop. Contiguous event pages are folded in memory into their final manifest
|
||||
and content heads before merging. An interrupted fetch restarts from the last
|
||||
incorporated cursor; pagination progress is not persisted. Intermediate changes
|
||||
that the server has already reverted must not overwrite local edits.
|
||||
|
||||
A permanent upload rejection is recorded locally, with content retained for
|
||||
recovery. An unchanged rejected payload is skipped so other files can sync; the
|
||||
A permanent upload rejection is recorded in metadata. An unchanged rejected payload is skipped so other files can sync; the
|
||||
failure remains visible in history and `waitUntilFinished()`. Editing it, resetting
|
||||
the client, or changing settings permits a fresh attempt. Earlier attempts that
|
||||
committed despite a lost reply are incorporated through normal event replay.
|
||||
|
|
@ -44,10 +46,9 @@ are implicit; empty directories are not synced.
|
|||
- Content is uploaded before new membership; the client chooses and persists UUIDs.
|
||||
- Text uses three-way `reconcile`. First contact at a matching path adopts the
|
||||
remote ID and reconciles differing text with an empty parent.
|
||||
- Concurrent binary/unmergeable edits keep the server version and retain
|
||||
displaced local bytes in recovery. If only one side changed, keep that side.
|
||||
- Deletion follows the file manifest decision. Displaced edited bytes are retained
|
||||
locally, without automatically restoring membership.
|
||||
- Concurrent binary/unmergeable edits keep the server version. If only one side
|
||||
changed, keep that side. No local backup archive is created.
|
||||
- Deletion follows the file manifest decision, including when the file has local edits.
|
||||
- Notified moves preserve IDs. Offline inference requires a unique nonempty hash
|
||||
match on both sides; uncertain cases become delete/create.
|
||||
- Ignored and oversized files preserve their disk paths and remote membership.
|
||||
|
|
@ -65,9 +66,8 @@ writes. WebSocket events wake HTTP replay, which validates that checkpoint. A
|
|||
restored database cannot silently reuse a version number: a missing or different
|
||||
event incarnation returns a history mismatch before executing the request.
|
||||
|
||||
On mismatch the client first recovers its filesystem journal and scans local
|
||||
work. It durably records a reset before clearing the old checkpoint, archives
|
||||
pending content requests, and clears old receipts, cursors, bases and diff-cache
|
||||
On mismatch the client scans local work and records a reset before clearing the
|
||||
old checkpoint. It clears old requests, receipts, cursors, bases and diff-cache
|
||||
entries. Clean files reconcile against the restored snapshot. Unsent edits with
|
||||
an old base become separate local documents; path conflicts preserve them under
|
||||
conflict names instead of applying a diff to an unrelated restored version.
|
||||
|
|
@ -79,59 +79,37 @@ Their failures are reported, and remaining observers continue receiving updates.
|
|||
|
||||
## Required adapter contract
|
||||
|
||||
`FileSystemOperations` and `PersistenceProvider` define the adapter operations
|
||||
needed by the recovery model. The Obsidian and CLI adapters have intentionally
|
||||
not been updated in this rewrite.
|
||||
`FileSystemOperations` and `PersistenceProvider` define the adapter operations.
|
||||
The Obsidian and CLI adapters still need to adopt the current core interface.
|
||||
The sync client does not coordinate separate processes accessing the same vault.
|
||||
|
||||
The sync client does not coordinate separate processes accessing the same local
|
||||
vault.
|
||||
The filesystem adapter must provide complete scans, coherent snapshots, recursive
|
||||
directory creation, exclusive file creation, no-replace renames, and deletion of
|
||||
individual regular files. Directory pruning uses only `rmdir`, never recursive
|
||||
file removal. Reject links, special files and paths escaping the vault. An unreadable
|
||||
file or directory must fail the scan rather than look like a deletion. Optional
|
||||
editor cursor metadata is applied with file content.
|
||||
|
||||
The filesystem adapter must provide:
|
||||
User-file writes have no power-loss durability requirement. There is no filesystem
|
||||
journal, hidden staging area, replay, backup archive, or flush protocol. Swaps and
|
||||
cycles temporarily move occupied files to visible conflict names; a fresh scan
|
||||
can discover them if application stops partway through. Partial writes and edits
|
||||
made before, during or after an interruption are treated as current disk content.
|
||||
The next sync reconciles that content; it does not finish a saved filesystem plan.
|
||||
Power loss may lose bytes or rename identity and may cause a merge to run again.
|
||||
|
||||
- Complete scans and coherent content snapshots. Missing and unreadable must
|
||||
be distinguished; reject read races rather than return a partial snapshot.
|
||||
- Vault-relative operations that reject symlinks, hardlinks, special files, and
|
||||
ancestor symlink traversal. Staging and visible files share one filesystem.
|
||||
- Atomic, complete, exclusive file creation (a crash leaves absent or complete
|
||||
bytes, never a partial file) and atomic renames that **never replace** destinations.
|
||||
Writes apply optional cursor metadata.
|
||||
- `flushPaths` flushes existing files and ancestor directories, including parents
|
||||
of absent paths. Recovery uses it to complete interrupted rename durability.
|
||||
- Durable recursive directory creation, and directory pruning using only `rmdir`
|
||||
operations (never unlink descendant files). All affected data/directory entries
|
||||
must be flushed before a mutation resolves, including intermediate ancestors.
|
||||
Metadata is the one atomic boundary: persistence `save` replaces a complete value
|
||||
and an interrupted/failed save leaves an old or new complete value, never a torn
|
||||
object. A failed save is reloaded before retrying. No ordering between metadata and
|
||||
user-file durability is required. Settings and engine saves are serialized.
|
||||
Keep metadata outside the user-file namespace or in `.vault-link-sync`, which is
|
||||
excluded from scans. Changing a client's vault once it holds identities, offline
|
||||
notifications, pending requests or bootstrap state requires a separate state store.
|
||||
|
||||
Keep the persistence store outside the user file namespace or inside the reserved
|
||||
`.vault-link-sync` directory, so writing sync state cannot generate user edits.
|
||||
|
||||
Persistence `save` atomically replaces the complete state and resolves only after
|
||||
its data and directory entry are durable. A failed save may leave either complete
|
||||
value; the engine reloads before retrying. Settings saves are assumed to succeed
|
||||
and do not reload after errors. Settings and engine saves are serialized.
|
||||
Changing a client's vault once it holds identities, offline notifications,
|
||||
pending requests, or bootstrap/application state requires a separate state store
|
||||
and client. Lifecycle operations are serialized.
|
||||
|
||||
Before changing visible files, the engine persists an application journal. It
|
||||
stages every affected source before installing destinations, so swaps and cycles
|
||||
work. Recovery runs before scanning, including while the server is unavailable.
|
||||
Prepared content is persisted before installation, preventing repeated merges
|
||||
following a crash. Unexpected occupants are preserved and given conflict names.
|
||||
Notifications received while staging rebind the actual source identity before
|
||||
content is merged. Editor saves to a temporarily staged path continue the original
|
||||
document's journal, including saves racing preparation and installation. Ignored
|
||||
or oversized occupants are reserved before conflict allocation.
|
||||
|
||||
`.vault-link-sync/transactions` contains only active recovery artifacts. Staged
|
||||
sources and prepared output are removed after every install is durably recorded;
|
||||
completed journals are not archived. `.vault-link-sync/recovery` retains displaced
|
||||
original bytes and their document/path metadata before staging files are deleted.
|
||||
`.vault-link-sync/requests` retains rejected submission snapshots for explicit
|
||||
recovery. These directories are never scanned
|
||||
as user files. Pending requests and active recovery journals must not be
|
||||
independently discarded.
|
||||
|
||||
These guarantees assume the adapters/storage honor their durability contracts.
|
||||
Arbitrary external writers can change a file between observations; detected races
|
||||
are preserved/reconciled, but unobserved writes and offline identity cannot be
|
||||
reconstructed perfectly. There is no automated v2/v3 persisted-state migration.
|
||||
Filesystem application saves metadata once when it finishes. Namespace changes detected
|
||||
before mutations invalidate the plan. Changes during application are incorporated
|
||||
between files using their document identities; the next scan rereads content. Content is reread before
|
||||
replacement. Unexpected eligible occupants receive visible conflict names;
|
||||
ignored and oversized occupants stay in place. External writers can still change
|
||||
files between observations, so these checks do not promise transactional file
|
||||
contents or perfect identity reconstruction. There is no automated v2/v3 state migration.
|
||||
|
|
|
|||
|
|
@ -1,428 +0,0 @@
|
|||
import assert from "node:assert/strict";
|
||||
import { describe, it } from "node:test";
|
||||
import {
|
||||
Database,
|
||||
emptyState,
|
||||
type StoredDatabase
|
||||
} from "../persistence/database";
|
||||
import type { ServerConfig } from "../services/server-config";
|
||||
import { toStoredSnapshot } from "../sync-operations/content";
|
||||
import { isInternalPath } from "../utils/portable-path";
|
||||
import { Logger } from "../tracing/logger";
|
||||
import { FileOperations, type FileWrite } from "./file-operations";
|
||||
import type {
|
||||
FileSnapshot,
|
||||
FileSystemOperations
|
||||
} from "./filesystem-operations";
|
||||
|
||||
const bytes = (text: string) => new TextEncoder().encode(text);
|
||||
const text = (snapshot: FileSnapshot) =>
|
||||
new TextDecoder().decode(snapshot.content);
|
||||
const parent = (path: string) => path.split("/").slice(0, -1).join("/");
|
||||
|
||||
/** Atomic mutations with an injectable interruption before or after durability. */
|
||||
class MemoryFiles implements FileSystemOperations {
|
||||
public readonly files = new Map<string, FileSnapshot>();
|
||||
public readonly directories = new Set([""]);
|
||||
public boundary: (label: string) => void = () => {};
|
||||
|
||||
public change(label: string, action: () => void): void {
|
||||
this.boundary(`before ${label}`);
|
||||
action();
|
||||
this.boundary(`after ${label}`);
|
||||
}
|
||||
|
||||
public put(path: string, value: string): void {
|
||||
for (
|
||||
let directory = parent(path);
|
||||
directory;
|
||||
directory = parent(directory)
|
||||
)
|
||||
this.directories.add(directory);
|
||||
this.files.set(path, { content: bytes(value) });
|
||||
}
|
||||
|
||||
public async listFilesRecursively(root = ""): Promise<string[]> {
|
||||
assert(this.directories.has(root), `Cannot list non-directory ${root}`);
|
||||
return [...this.files.keys()].filter(
|
||||
(path) => !root || path.startsWith(`${root}/`)
|
||||
);
|
||||
}
|
||||
|
||||
public async readSnapshot(path: string): Promise<FileSnapshot | undefined> {
|
||||
assert(!this.directories.has(path), `Cannot read directory ${path}`);
|
||||
return structuredClone(this.files.get(path));
|
||||
}
|
||||
|
||||
public async stat(path: string) {
|
||||
for (let ancestor = parent(path); ancestor; ancestor = parent(ancestor))
|
||||
assert(
|
||||
!this.files.has(ancestor),
|
||||
`Cannot traverse file ${ancestor}`
|
||||
);
|
||||
const snapshot = this.files.get(path);
|
||||
if (snapshot)
|
||||
return { kind: "file" as const, size: snapshot.content.length };
|
||||
if (this.directories.has(path))
|
||||
return { kind: "directory" as const, size: 0 };
|
||||
return undefined;
|
||||
}
|
||||
|
||||
public async exists(path: string): Promise<boolean> {
|
||||
return this.files.has(path) || this.directories.has(path);
|
||||
}
|
||||
|
||||
public async createDirectory(path: string): Promise<void> {
|
||||
this.change(`mkdir ${path}`, () => {
|
||||
for (
|
||||
let directory = path;
|
||||
directory;
|
||||
directory = parent(directory)
|
||||
) {
|
||||
assert(
|
||||
!this.files.has(directory),
|
||||
`File blocks directory ${directory}`
|
||||
);
|
||||
this.directories.add(directory);
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
public async write(path: string, snapshot: FileSnapshot): Promise<void> {
|
||||
this.change(`write ${path}`, () => {
|
||||
assert(
|
||||
!this.files.has(path) && !this.directories.has(path),
|
||||
`Destination exists: ${path}`
|
||||
);
|
||||
assert(
|
||||
this.directories.has(parent(path)),
|
||||
`Parent missing: ${path}`
|
||||
);
|
||||
this.files.set(path, structuredClone(snapshot));
|
||||
});
|
||||
}
|
||||
|
||||
public async rename(from: string, to: string): Promise<void> {
|
||||
this.change(`rename ${from} -> ${to}`, () => {
|
||||
assert(this.files.has(from), `Source missing: ${from}`);
|
||||
assert(
|
||||
!this.files.has(to) && !this.directories.has(to),
|
||||
`Destination exists: ${to}`
|
||||
);
|
||||
assert(this.directories.has(parent(to)), `Parent missing: ${to}`);
|
||||
this.files.set(to, this.files.get(from)!);
|
||||
this.files.delete(from);
|
||||
});
|
||||
}
|
||||
|
||||
public async flushPaths(paths: readonly string[]): Promise<void> {
|
||||
this.change(`flush ${paths.join(",")}`, () => {});
|
||||
}
|
||||
|
||||
public async delete(path: string): Promise<void> {
|
||||
this.change(`rmdir ${path}`, () => {
|
||||
assert(!this.files.has(path));
|
||||
assert(
|
||||
![...this.files.keys()].some((file) =>
|
||||
file.startsWith(`${path}/`)
|
||||
)
|
||||
);
|
||||
for (const directory of this.directories)
|
||||
if (directory === path || directory.startsWith(`${path}/`))
|
||||
this.directories.delete(directory);
|
||||
});
|
||||
}
|
||||
|
||||
public async deleteFile(path: string): Promise<void> {
|
||||
this.change(`unlink ${path}`, () => {
|
||||
assert(!this.directories.has(path));
|
||||
this.files.delete(path);
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
async function setup(
|
||||
contents: Record<string, string>,
|
||||
local: Record<string, string>
|
||||
) {
|
||||
const fs = new MemoryFiles();
|
||||
for (const [path, value] of Object.entries(contents)) fs.put(path, value);
|
||||
let durable: StoredDatabase = emptyState("test");
|
||||
durable.local = local;
|
||||
for (const [id, path] of Object.entries(local))
|
||||
durable.documents[id] = {
|
||||
materialized: true,
|
||||
observedHash: (await toStoredSnapshot(fs.files.get(path)!)).hash
|
||||
};
|
||||
const config = {
|
||||
getConfig: async () => ({ mergeableFileExtensions: ["md"] })
|
||||
} as ServerConfig;
|
||||
const restart = () => {
|
||||
const database = new Database(
|
||||
new Logger(),
|
||||
structuredClone(durable),
|
||||
async (next) => {
|
||||
fs.change("save", () => {
|
||||
durable = structuredClone(next);
|
||||
});
|
||||
},
|
||||
"test",
|
||||
async () => structuredClone(durable)
|
||||
);
|
||||
return {
|
||||
database,
|
||||
operations: new FileOperations(fs, database, config)
|
||||
};
|
||||
};
|
||||
return { fs, restart, ...restart() };
|
||||
}
|
||||
|
||||
async function scenario() {
|
||||
const fixture = await setup(
|
||||
{
|
||||
"a.md": "A",
|
||||
"b.md": "B",
|
||||
"c.md": "C",
|
||||
"d.md": "Unmanaged",
|
||||
"e.md": "E",
|
||||
"folder/keep.md": "Keep",
|
||||
"deleted.md": "Deleted"
|
||||
},
|
||||
{ a: "a.md", b: "b.md", c: "c.md", e: "e.md", deleted: "deleted.md" }
|
||||
);
|
||||
const next = structuredClone(fixture.database.state);
|
||||
next.local = {
|
||||
a: "b.md",
|
||||
b: "a.md",
|
||||
c: "d.md",
|
||||
e: "folder",
|
||||
added: "new.md"
|
||||
};
|
||||
next.documents.added = { materialized: false };
|
||||
const writes: Record<string, FileWrite> = {
|
||||
a: {
|
||||
expected: await fixture.operations.snapshot("a.md"),
|
||||
replacement: await toStoredSnapshot({ content: bytes("A updated") })
|
||||
},
|
||||
added: {
|
||||
replacement: await toStoredSnapshot({
|
||||
content: bytes("Downloaded")
|
||||
})
|
||||
}
|
||||
};
|
||||
return { ...fixture, next, writes };
|
||||
}
|
||||
|
||||
async function verifyScenario(fs: MemoryFiles, database: Database) {
|
||||
assert.equal(database.state.application, undefined);
|
||||
for (const [id, expected] of Object.entries({
|
||||
a: "A updated",
|
||||
b: "B",
|
||||
c: "C",
|
||||
e: "E",
|
||||
added: "Downloaded"
|
||||
})) {
|
||||
const snapshot = fs.files.get(database.state.local[id])!;
|
||||
assert.equal(text(snapshot), expected, id);
|
||||
assert.equal(database.state.documents[id].materialized, true, id);
|
||||
assert.equal(
|
||||
database.state.documents[id].observedHash,
|
||||
(await toStoredSnapshot(snapshot)).hash,
|
||||
id
|
||||
);
|
||||
}
|
||||
assert.equal(database.state.local.a, "b.md");
|
||||
assert.equal(database.state.local.b, "a.md");
|
||||
assert.equal(database.state.local.c, "d.md");
|
||||
assert.notEqual(database.state.local.e, "folder");
|
||||
assert.equal(text(fs.files.get("folder/keep.md")!), "Keep");
|
||||
assert.equal(fs.files.has("deleted.md"), false);
|
||||
assert.equal(database.state.local.deleted, undefined);
|
||||
const visible = [...fs.files].filter(([path]) => !isInternalPath(path));
|
||||
assert.deepEqual(
|
||||
visible.map(([, value]) => text(value)).sort(),
|
||||
["A updated", "B", "C", "Downloaded", "E", "Keep", "Unmanaged"].sort()
|
||||
);
|
||||
assert.equal(
|
||||
[...fs.files.keys()].some((path) =>
|
||||
path.startsWith(".vault-link-sync/transactions/")
|
||||
),
|
||||
false
|
||||
);
|
||||
}
|
||||
|
||||
describe("filesystem journal recovery", () => {
|
||||
it("stages swaps, preserves unexpected files/directories, and cleans recovery artifacts", async () => {
|
||||
const { operations, database, fs, next, writes } = await scenario();
|
||||
await operations.apply(next, writes);
|
||||
await verifyScenario(fs, database);
|
||||
fs.boundary = () => {
|
||||
throw new Error("Completed recovery must not write");
|
||||
};
|
||||
await operations.recover();
|
||||
});
|
||||
|
||||
it("resumes after interruption at every filesystem and metadata mutation boundary", async (t) => {
|
||||
const baseline = await scenario();
|
||||
const boundaries: string[] = [];
|
||||
baseline.fs.boundary = (label) => boundaries.push(label);
|
||||
await baseline.operations.apply(baseline.next, baseline.writes);
|
||||
for (const [index, label] of boundaries.entries()) {
|
||||
await t.test(`${index}: ${label}`, async () => {
|
||||
const fixture = await scenario();
|
||||
const interrupted = new Error("Interrupted");
|
||||
let count = 0;
|
||||
fixture.fs.boundary = () => {
|
||||
if (count++ === index) throw interrupted;
|
||||
};
|
||||
await assert.rejects(
|
||||
fixture.operations.apply(fixture.next, fixture.writes),
|
||||
(error) => error === interrupted
|
||||
);
|
||||
fixture.fs.boundary = () => {};
|
||||
const { database, operations } = fixture.restart();
|
||||
if (
|
||||
!database.state.application &&
|
||||
database.state.local.a === "a.md"
|
||||
)
|
||||
await operations.apply(
|
||||
structuredClone(fixture.next),
|
||||
fixture.writes
|
||||
);
|
||||
else await operations.recover();
|
||||
await verifyScenario(fixture.fs, database);
|
||||
});
|
||||
}
|
||||
});
|
||||
|
||||
it("merges an edit made after planning exactly once across an interrupted install", async () => {
|
||||
const base = "First paragraph.\n\nLast paragraph.\n";
|
||||
const local = "Local first paragraph.\n\nLast paragraph.\n";
|
||||
const remote = "First paragraph.\n\nRemote last paragraph.\n";
|
||||
const fixture = await setup({ "note.md": base }, { note: "note.md" });
|
||||
const expected = await fixture.operations.snapshot("note.md");
|
||||
let edited = false;
|
||||
const interrupted = new Error("Interrupted after install");
|
||||
fixture.fs.boundary = (label) => {
|
||||
if (!edited && label === "after save") {
|
||||
fixture.fs.put("note.md", local);
|
||||
edited = true;
|
||||
}
|
||||
if (
|
||||
label.startsWith("after rename ") &&
|
||||
label.endsWith(" -> note.md")
|
||||
)
|
||||
throw interrupted;
|
||||
};
|
||||
await assert.rejects(
|
||||
fixture.operations.apply(structuredClone(fixture.database.state), {
|
||||
note: {
|
||||
expected,
|
||||
replacement: await toStoredSnapshot({
|
||||
content: bytes(remote)
|
||||
})
|
||||
}
|
||||
}),
|
||||
(error) => error === interrupted
|
||||
);
|
||||
fixture.fs.boundary = () => {};
|
||||
const { operations, database } = fixture.restart();
|
||||
await operations.recover();
|
||||
assert.equal(
|
||||
text(fixture.fs.files.get("note.md")!),
|
||||
"Local first paragraph.\n\nRemote last paragraph.\n"
|
||||
);
|
||||
assert.equal(
|
||||
database.state.documents.note.observedHash,
|
||||
(await operations.snapshot("note.md"))!.hash
|
||||
);
|
||||
assert.equal(
|
||||
[...fixture.fs.files.keys()].some((path) =>
|
||||
path.startsWith(".vault-link-sync/transactions/")
|
||||
),
|
||||
false
|
||||
);
|
||||
});
|
||||
|
||||
it("preserves an external deletion between planning and staging", async () => {
|
||||
const fixture = await setup(
|
||||
{ "note.md": "Local" },
|
||||
{ note: "note.md" }
|
||||
);
|
||||
const expected = await fixture.operations.snapshot("note.md");
|
||||
fixture.fs.boundary = (label) => {
|
||||
if (label === "after save") fixture.fs.files.delete("note.md");
|
||||
};
|
||||
await fixture.operations.apply(
|
||||
structuredClone(fixture.database.state),
|
||||
{
|
||||
note: {
|
||||
expected,
|
||||
replacement: await toStoredSnapshot({
|
||||
content: bytes("Remote")
|
||||
})
|
||||
}
|
||||
}
|
||||
);
|
||||
assert.equal(fixture.database.state.local.note, undefined);
|
||||
assert.equal(fixture.fs.files.has("note.md"), false);
|
||||
});
|
||||
|
||||
it("preserves a file created after the destination check", async () => {
|
||||
const fixture = await setup(
|
||||
{ "source.md": "Incoming" },
|
||||
{ doc: "source.md" }
|
||||
);
|
||||
const next = structuredClone(fixture.database.state);
|
||||
next.local.doc = "target.md";
|
||||
fixture.fs.boundary = (label) => {
|
||||
if (
|
||||
label.startsWith("before rename ") &&
|
||||
label.endsWith(" -> target.md")
|
||||
)
|
||||
fixture.fs.put("target.md", "Concurrent");
|
||||
};
|
||||
await assert.rejects(
|
||||
fixture.operations.apply(next),
|
||||
/Destination exists/
|
||||
);
|
||||
fixture.fs.boundary = () => {};
|
||||
const { operations, database } = fixture.restart();
|
||||
await operations.recover();
|
||||
assert.equal(text(fixture.fs.files.get("target.md")!), "Incoming");
|
||||
const displaced = Object.entries(database.state.local).find(
|
||||
([id]) => id !== "doc"
|
||||
)!;
|
||||
assert.equal(text(fixture.fs.files.get(displaced[1])!), "Concurrent");
|
||||
});
|
||||
|
||||
it("preserves an unexpected file blocking a destination's ancestor", async () => {
|
||||
const fixture = await setup(
|
||||
{ "source.md": "Incoming", target: "Occupant" },
|
||||
{ doc: "source.md" }
|
||||
);
|
||||
const next = structuredClone(fixture.database.state);
|
||||
next.local.doc = "target/child.md";
|
||||
await fixture.operations.apply(next);
|
||||
assert.equal(
|
||||
text(fixture.fs.files.get("target/child.md")!),
|
||||
"Incoming"
|
||||
);
|
||||
const displaced = Object.entries(fixture.database.state.local).find(
|
||||
([id]) => id !== "doc"
|
||||
)!;
|
||||
assert.equal(text(fixture.fs.files.get(displaced[1])!), "Occupant");
|
||||
});
|
||||
|
||||
it("handles paths becoming directories and directories becoming files", async () => {
|
||||
const fixture = await setup(
|
||||
{ a: "A", "b/deep/child": "B" },
|
||||
{ a: "a", b: "b/deep/child" }
|
||||
);
|
||||
const next = structuredClone(fixture.database.state);
|
||||
next.local = { a: "b", b: "a/deep/child" };
|
||||
await fixture.operations.apply(next);
|
||||
assert.equal(text(fixture.fs.files.get("b")!), "A");
|
||||
assert.equal(text(fixture.fs.files.get("a/deep/child")!), "B");
|
||||
});
|
||||
});
|
||||
File diff suppressed because it is too large
Load diff
|
|
@ -6,10 +6,9 @@ export interface FileSnapshot {
|
|||
cursors?: CursorPosition[];
|
||||
}
|
||||
|
||||
/** Required adapter contract for API v4; there is deliberately no weak fallback.
|
||||
/** Filesystem operations act on the current user-owned namespace.
|
||||
* Paths are vault-relative. Never follow symlinks (including ancestor symlinks).
|
||||
* Only independent regular files are supported; reject hardlinks/special files.
|
||||
* Every mutation must flush file data and affected directories before resolving.
|
||||
* A successful scan is complete: unreadable directories must reject, not vanish.
|
||||
*/
|
||||
export interface FileSystemOperations {
|
||||
|
|
@ -22,24 +21,15 @@ export interface FileSystemOperations {
|
|||
) => Promise<{ kind: "file" | "directory"; size: number } | undefined>;
|
||||
/** Includes directories. */
|
||||
exists: (path: string) => Promise<boolean>;
|
||||
/** Recursive, idempotent, durable directory creation. */
|
||||
/** Recursive, idempotent directory creation. */
|
||||
createDirectory: (path: string) => Promise<void>;
|
||||
/** Atomically create a complete file: never replace an existing object.
|
||||
* A crash leaves the path absent or complete, never a partial file.
|
||||
* Flush data and directory entries before resolving. */
|
||||
/** Create a file exclusively; never overwrite a raced destination.
|
||||
* Writes apply optional editor cursor metadata. No durability guarantee. */
|
||||
write: (path: string, snapshot: FileSnapshot) => Promise<void>;
|
||||
/** Atomic, durable rename on the same filesystem; fail if destination exists.
|
||||
* Flush the source's data and both affected directories before resolving. */
|
||||
/** Move a file without replacing an existing destination. */
|
||||
rename: (from: string, to: string) => Promise<void>;
|
||||
/** Flush existing files and all affected ancestor directories, even when a
|
||||
* named file is absent. Recovery uses this after a mutation became visible
|
||||
* before its caller learned that it was durable. */
|
||||
flushPaths: (paths: readonly string[]) => Promise<void>;
|
||||
/** Idempotently unlink one regular file and flush its parent directory.
|
||||
* The sync engine uses this only for its own internal recovery artifacts. */
|
||||
/** Idempotently unlink one regular file. */
|
||||
deleteFile: (path: string) => Promise<void>;
|
||||
/** Prune an implicit directory tree using rmdir only, durably. Reject
|
||||
* regular files and never unlink descendant files; reject if files remain.
|
||||
* Partial removal of empty child directories on failure is harmless. */
|
||||
/** Prune directories using rmdir only. Reject files and nonempty directories. */
|
||||
delete: (path: string) => Promise<void>;
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1,5 +1,4 @@
|
|||
import type { CursorPosition } from "reconcile-text";
|
||||
import type { Logger } from "../tracing/logger";
|
||||
import type { DocumentUpdateResponse } from "../services/types/DocumentUpdateResponse";
|
||||
import type { DocumentVersionWithoutContent } from "../services/types/DocumentVersionWithoutContent";
|
||||
import type { FileManifest } from "../services/types/FileManifest";
|
||||
|
|
@ -7,7 +6,6 @@ import type { FileManifestUpdateResponse } from "../services/types/FileManifestU
|
|||
import type { PushFileManifest } from "../services/types/PushFileManifest";
|
||||
import type { PutFileContent } from "../services/types/PutFileContent";
|
||||
import type { VaultSnapshot } from "../services/types/VaultSnapshot";
|
||||
import type { EventRecord } from "../services/types/EventRecord";
|
||||
import type { FileManifestEntries } from "../types/file-manifest-entries";
|
||||
|
||||
export type VaultUpdateId = number;
|
||||
|
|
@ -23,7 +21,6 @@ export interface StoredSnapshot {
|
|||
export interface DocumentMetadata {
|
||||
parentVersionId: number;
|
||||
hash: string;
|
||||
remoteRelativePath?: string;
|
||||
}
|
||||
|
||||
export interface DocumentState {
|
||||
|
|
@ -54,7 +51,7 @@ export type PendingRequest = { rejection?: string } & (
|
|||
);
|
||||
|
||||
export interface EngineState {
|
||||
/** Durable reset intent; clear the transport checkpoint before bootstrap. */
|
||||
/** Persisted reset intent; clear the transport checkpoint before bootstrap. */
|
||||
historyRecovery?: boolean;
|
||||
vaultKey: string;
|
||||
initialized: boolean;
|
||||
|
|
@ -64,17 +61,10 @@ export interface EngineState {
|
|||
lastSeenUpdateId: number;
|
||||
remoteHeads: Record<string, DocumentVersionWithoutContent>;
|
||||
bootstrap?: VaultSnapshot;
|
||||
/** Fold pages durably without applying intermediate remote snapshots. */
|
||||
eventReplay?: {
|
||||
after: number;
|
||||
manifest: FileManifest;
|
||||
heads: Record<string, DocumentVersionWithoutContent>;
|
||||
receipts: EventRecord[];
|
||||
};
|
||||
pending?: PendingRequest;
|
||||
/** A rejected retry does not prove an earlier attempt failed to commit. */
|
||||
unconfirmed?: PendingRequest[];
|
||||
/** Commits consumption of the durable notification queue with the identity map. */
|
||||
/** Commits consumption of the saved notification queue with the identity map. */
|
||||
lastAppliedLocalChangeId?: string;
|
||||
/** Physical exclusions, including files with no sync identity. */
|
||||
protectedPaths?: string[];
|
||||
|
|
@ -83,38 +73,13 @@ export interface EngineState {
|
|||
rejectedManifest?: { entries: FileManifestEntries; message: string };
|
||||
}
|
||||
|
||||
export interface FileStep {
|
||||
documentId: string;
|
||||
from?: string;
|
||||
to?: string;
|
||||
expected?: StoredSnapshot;
|
||||
replacement?: StoredSnapshot;
|
||||
staged: string;
|
||||
output: string;
|
||||
/** A permanent destination error has already selected a root fallback. */
|
||||
replanned?: boolean;
|
||||
/** Retained input of a later step for this identity; never projected as live. */
|
||||
superseded?: boolean;
|
||||
phase: "planned" | "staged" | "prepared" | "installing" | "installed";
|
||||
}
|
||||
|
||||
export interface ApplicationJournal {
|
||||
id: string;
|
||||
extensions: string[];
|
||||
next: EngineState;
|
||||
steps: FileStep[];
|
||||
}
|
||||
|
||||
export interface StoredDatabase extends EngineState {
|
||||
application?: ApplicationJournal;
|
||||
}
|
||||
export type StoredDatabase = EngineState;
|
||||
|
||||
// Read-only view used by cursor tracking and status reporting.
|
||||
export interface DocumentRecord {
|
||||
documentId: string;
|
||||
relativePath: string;
|
||||
metadata?: DocumentMetadata;
|
||||
isDeleted: boolean;
|
||||
}
|
||||
|
||||
export function emptyState(vaultKey: string): StoredDatabase {
|
||||
|
|
@ -134,9 +99,7 @@ function canRebindEmpty(state: Partial<StoredDatabase> | undefined): boolean {
|
|||
state !== undefined &&
|
||||
!state.initialized &&
|
||||
!state.pending &&
|
||||
!state.application &&
|
||||
!state.bootstrap &&
|
||||
!state.eventReplay &&
|
||||
(state.unconfirmed?.length ?? 0) === 0 &&
|
||||
Object.keys(state.local ?? {}).length === 0
|
||||
);
|
||||
|
|
@ -146,7 +109,6 @@ export class Database {
|
|||
public state: StoredDatabase;
|
||||
private needsReload = false;
|
||||
public constructor(
|
||||
private readonly logger: Logger,
|
||||
initial: Partial<StoredDatabase> | undefined,
|
||||
private readonly saveData: (data: StoredDatabase) => Promise<void>,
|
||||
vaultKey: string,
|
||||
|
|
@ -181,7 +143,7 @@ export class Database {
|
|||
const snapshot = structuredClone(next);
|
||||
if (this.needsReload)
|
||||
throw new Error(
|
||||
"Reload durable state before retrying an uncertain save"
|
||||
"Reload saved state before retrying an uncertain save"
|
||||
);
|
||||
this.needsReload = true;
|
||||
await this.saveData(snapshot);
|
||||
|
|
@ -204,10 +166,6 @@ export class Database {
|
|||
return Object.keys(this.state.local).length;
|
||||
}
|
||||
|
||||
public getLastSeenUpdateId(): number {
|
||||
return this.state.lastSeenUpdateId;
|
||||
}
|
||||
|
||||
public getDocumentByDocumentId(id: string): DocumentRecord | undefined {
|
||||
const relativePath = this.state.local[id];
|
||||
if (relativePath === undefined) return undefined;
|
||||
|
|
@ -215,11 +173,9 @@ export class Database {
|
|||
return {
|
||||
documentId: id,
|
||||
relativePath,
|
||||
isDeleted: false,
|
||||
metadata: base && {
|
||||
parentVersionId: base.vaultUpdateId,
|
||||
hash: base.hash,
|
||||
remoteRelativePath: this.state.fileManifest.entries[id]
|
||||
hash: base.hash
|
||||
}
|
||||
};
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
/** save() atomically replaces the entire value and resolves only after both the
|
||||
* data and its directory entry are durable across power loss. Failed saves must
|
||||
* leave either complete value readable; callers retain their recovery journal.
|
||||
/** Replace the complete metadata value atomically. Interrupted or failed saves
|
||||
* must leave an old/new complete value (or no value for the first save), never
|
||||
* partially parsed state. No ordering or durability of user files is required.
|
||||
*/
|
||||
export interface PersistenceProvider<T> {
|
||||
load: () => Promise<T | undefined>;
|
||||
|
|
|
|||
|
|
@ -25,12 +25,18 @@ import { isInternalPath } from "./utils/portable-path";
|
|||
import { FixedSizeDocumentCache } from "./utils/data-structures/fix-sized-cache";
|
||||
import { globsToRegexes } from "./utils/globs-to-regexes";
|
||||
import { Lock } from "./utils/data-structures/locks";
|
||||
import type { LocalChange } from "./sync-operations/local-changes";
|
||||
import {
|
||||
unappliedChanges,
|
||||
type LocalChange
|
||||
} from "./sync-operations/local-changes";
|
||||
|
||||
type StoredClient = Partial<{
|
||||
settings: Partial<SyncSettings>;
|
||||
database: Partial<StoredDatabase>;
|
||||
localChanges: LocalChange[];
|
||||
localChanges: (
|
||||
| LocalChange
|
||||
| { type: "update"; path: string; changeId?: string }
|
||||
)[];
|
||||
localChangesVaultKey: string;
|
||||
historyCheckpoint: { vaultKey: string; checkpoint?: string };
|
||||
}>;
|
||||
|
|
@ -96,7 +102,7 @@ export class SyncClient {
|
|||
let stored = (await persistence.load()) ?? {};
|
||||
let saving: Promise<void> = Promise.resolve();
|
||||
let reloadBeforeSave = false;
|
||||
// Settings and engine state share a store. Serialize complete durable
|
||||
// Settings and engine state share a store. Serialize complete metadata
|
||||
// replacements so one cannot overwrite a newer save from the other.
|
||||
const save = async (update: StoredClient): Promise<void> => {
|
||||
const operation = saving.then(async () => {
|
||||
|
|
@ -122,7 +128,6 @@ export class SyncClient {
|
|||
save({ settings: data })
|
||||
);
|
||||
const database = new Database(
|
||||
logger,
|
||||
stored.database,
|
||||
async (data) => save({ database: data }),
|
||||
vaultKey(settings.getSettings()),
|
||||
|
|
@ -132,8 +137,15 @@ export class SyncClient {
|
|||
return stored.database as StoredDatabase | undefined;
|
||||
}
|
||||
);
|
||||
// Older stores also queued content updates. Trim the acknowledged prefix
|
||||
// before discarding them, since its marker may belong to an update.
|
||||
const localChanges = unappliedChanges(
|
||||
database.state,
|
||||
stored.localChanges ?? []
|
||||
// eslint-disable-next-line no-restricted-syntax -- Migration needs a narrowed copy, not removal of a specific object.
|
||||
).filter((change) => change.type !== "update");
|
||||
if (
|
||||
stored.localChanges?.length &&
|
||||
localChanges.length &&
|
||||
stored.localChangesVaultKey !== vaultKey(settings.getSettings())
|
||||
) {
|
||||
throw new Error(
|
||||
|
|
@ -168,7 +180,6 @@ export class SyncClient {
|
|||
);
|
||||
const serverConfig = new ServerConfig(service);
|
||||
const notifier = new FileChangeNotifier();
|
||||
const localChanges = stored.localChanges ?? [];
|
||||
let syncer: Syncer;
|
||||
const files = new FileOperations(
|
||||
fs,
|
||||
|
|
@ -283,7 +294,7 @@ export class SyncClient {
|
|||
this.check();
|
||||
if (this.started)
|
||||
throw new Error("SyncClient has already been started");
|
||||
await this.syncer.recoverLocalState();
|
||||
await this.syncer.reloadLocalState();
|
||||
await this.database.bindVault(
|
||||
vaultKey(this.settings.getSettings())
|
||||
);
|
||||
|
|
@ -310,7 +321,7 @@ export class SyncClient {
|
|||
await stopped;
|
||||
}
|
||||
|
||||
/** Restart transports; never discard pending requests or filesystem journals. */
|
||||
/** Restart transports; never discard pending requests. */
|
||||
public async reset(): Promise<void> {
|
||||
await this.lifecycle.withLock(async () => {
|
||||
this.check();
|
||||
|
|
@ -335,12 +346,11 @@ export class SyncClient {
|
|||
this.database.state.initialized ||
|
||||
this.database.state.bootstrap ||
|
||||
this.database.state.pending ||
|
||||
this.database.state.application ||
|
||||
this.syncer.hasLocalChanges ||
|
||||
Object.keys(this.database.state.local).length)
|
||||
)
|
||||
throw new Error(
|
||||
"Changing vaults requires a separate state store and client; pending recovery data must not be reset"
|
||||
"Changing vaults requires a separate state store and client; existing sync state must not be reset"
|
||||
);
|
||||
|
||||
try {
|
||||
|
|
|
|||
|
|
@ -47,8 +47,7 @@ export async function mergeContent(
|
|||
remote.contentBase64
|
||||
].map(base64ToBytes);
|
||||
|
||||
// Concurrent unmergeable edits keep the server's content. The filesystem
|
||||
// journal retains displaced local bytes for recovery.
|
||||
// Concurrent unmergeable edits keep the server's content.
|
||||
if (!isFileTypeMergable(path, extensions) || inputs.some(isBinary))
|
||||
return remote;
|
||||
|
||||
|
|
|
|||
|
|
@ -18,7 +18,6 @@ it("retains future cursors and publishes them when content catches up", async ()
|
|||
getDocumentByDocumentId: () => ({
|
||||
documentId: "document",
|
||||
relativePath: "note.md",
|
||||
isDeleted: false,
|
||||
metadata: { parentVersionId: 2, hash: contentHash }
|
||||
})
|
||||
} as unknown as Database;
|
||||
|
|
|
|||
|
|
@ -11,13 +11,12 @@ export type LocalChange = {
|
|||
| { type: "move"; oldPath: string; relativePath: string }
|
||||
| { type: "create"; path: string; documentId?: string }
|
||||
| { type: "delete"; path: string }
|
||||
| { type: "update"; path: string }
|
||||
);
|
||||
|
||||
export function unappliedChanges(
|
||||
export function unappliedChanges<T extends { changeId?: string }>(
|
||||
state: EngineState,
|
||||
changes: readonly LocalChange[]
|
||||
): LocalChange[] {
|
||||
changes: readonly T[]
|
||||
): T[] {
|
||||
for (const change of changes) change.changeId ??= uuid();
|
||||
const applied = changes.findIndex(
|
||||
(change) => change.changeId === state.lastAppliedLocalChangeId
|
||||
|
|
@ -63,7 +62,6 @@ export function applyLocalChange(
|
|||
(path === from || path.startsWith(from + "/"))
|
||||
)
|
||||
);
|
||||
if (change.type === "update") return;
|
||||
for (const [id, path] of Object.entries(change.identities)) {
|
||||
if (change.type === "delete") {
|
||||
if (ignored(path)) {
|
||||
|
|
|
|||
|
|
@ -17,7 +17,6 @@ for (const change of ["version", "identity", "path", "delete"] as const)
|
|||
let current: DocumentRecord | undefined = {
|
||||
documentId: "a",
|
||||
relativePath: "a.md",
|
||||
isDeleted: false,
|
||||
metadata: { parentVersionId: 1, hash: digest }
|
||||
};
|
||||
const sent: { documentsWithCursors: DocumentWithCursors[] }[] = [];
|
||||
|
|
|
|||
Some files were not shown because too many files have changed in this diff Show more
Loading…
Reference in a new issue