181 lines
13 KiB
Markdown
181 lines
13 KiB
Markdown
# Sync client (API v4)
|
|
|
|
The sync coordinator uses one serialized loop per vault and requires Web Crypto (SHA-256). Local notifications and WebSocket
|
|
`vaultChanged` hints wake it; complete disk scans and the durable server event log are the
|
|
sources of truth. Periodic reconciliation is opt-in: set `syncIntervalMs` to a
|
|
positive interval to repair missed notifications. Unset or `0` disables polling.
|
|
Create, delete and move notifications persist logical identity changes before
|
|
resolving, including while offline. Shutdown waits for those saves. Content-update
|
|
notifications only wake reconciliation; scans read the latest bytes, including after
|
|
restart. Notifications describe logical external changes: adapters suppress
|
|
self-generated create/delete/move notifications, and report atomic editor saves
|
|
as content updates. This distinguishes a deliberate delete/recreate from an edit
|
|
when the host supplies reliable logical events. Raw OS watchers instead send
|
|
content-update hints for every event and let scans infer the current namespace.
|
|
Use `waitUntilFinished()` to await the active attempt; errors leave
|
|
pending requests saved and are reported to the caller. Transient failures retry;
|
|
authentication and protocol errors wait for an explicit wake or settings change.
|
|
Pausing or destroying the client aborts its HTTP session, including body reads;
|
|
resuming starts a fresh session. Connection checks can run while sync is disabled.
|
|
WebSocket shutdown retires callbacks immediately and drains message handlers
|
|
already running; it does not wait for the transport's close handshake.
|
|
|
|
## Code layout
|
|
|
|
Start with `src/sync-operations/syncer.ts` for the sync workflow: resolve the saved
|
|
upload, fetch the complete remote history, apply it, and prepare the next upload.
|
|
It coordinates the following components:
|
|
|
|
| Component | Responsibility |
|
|
| -------------------------------------- | --------------------------------------------------------------------------------------------- |
|
|
| `sync-scheduler.ts` | Serialize attempts, coalesce notifications, stop, and schedule polling or retries. |
|
|
| `local-change-queue.ts` | Persist create/delete/move notifications and invalidate plans when identities change. |
|
|
| `scan.ts` | Inventory disk files, infer identities, and commit the scan before consuming notifications. |
|
|
| `remote-changes.ts` | Fetch and validate every event page before returning the final remote state. |
|
|
| `content-sync.ts` | Fetch merge bases, plan content writes, encode uploads, and maintain the optional diff cache. |
|
|
| `upload-queue.ts` | Persist exact requests, handle acknowledgements, and retain rejected requests for retry. |
|
|
| `file-operations/` | Apply paths and content to disk, incorporating notifications between files. |
|
|
| `persistence/database.ts` | Define `StoredDatabase` and atomically commit or reload it. |
|
|
| `persistence/client-metadata-store.ts` | Serialize database, settings, checkpoint, and notification saves into the shared store. |
|
|
|
|
`requestSync()` schedules work; `synchronize()` runs an attempt; `refreshLocalFiles()`
|
|
flushes notifications and calls `scanLocalFiles()`. `initializeFromServerSnapshot()`
|
|
applies the initial server snapshot, while `applyRemoteChanges()` merges and applies
|
|
an observed remote state. These operations keep the same persisted database format.
|
|
|
|
The scheduler's `SyncPhase` enum makes its transitions explicit:
|
|
|
|
| Event | Transition |
|
|
| ----------------------------------- | ------------------------------------------------------------------- |
|
|
| Start or explicit request | Idle, waiting, or blocked → run requested |
|
|
| Begin a pass | Run requested → running |
|
|
| Notification during a pass | Running → run requested; repeated notifications coalesce |
|
|
| Successful completion | Running → idle; optionally schedule the polling timer |
|
|
| Transient failure | Running → waiting to retry |
|
|
| Authentication or permanent failure | Running → blocked until an explicit request |
|
|
| Stop | Running → stopping → stopped; idle timers are cancelled immediately |
|
|
|
|
From `frontend/`, run `npm run lint -w sync-client`, `npm run check -w sync-client`,
|
|
and `npm run test -w sync-client`. `npm run test:harness` covers interrupted saves,
|
|
file application, and recovery. Protocol types under `src/services/types/` are
|
|
generated from Rust and excluded from ESLint; handwritten TypeScript requires braces.
|
|
|
|
## State and policy
|
|
|
|
The persisted state contains the incorporated server file manifest, the current
|
|
local identity map, an optional event cursor, and exact pending submissions. An
|
|
absent cursor means the client must fetch a fresh vault snapshot; there are no
|
|
separate initialization or recovery flags and no saved bootstrap snapshot. Each
|
|
document stores its base version/hash, its last observed disk hash, and a remote
|
|
head only while that content is deferred. The observed hash also records local
|
|
file presence; missing remote downloads are not local deletions. Physical path
|
|
exclusions are rebuilt by scans. Rejection details live on the retained request,
|
|
without duplicate per-document or manifest rejection records. A received server file
|
|
manifest becomes the merge base when its application metadata is saved. Successful
|
|
submissions advance the base to the submitted content snapshot, preserving later
|
|
local edits. Unknown outcomes are retried unchanged before incorporating newer
|
|
remote state. HTTP is the sole source of remote events; WebSocket messages only
|
|
wake the loop. Contiguous event pages are folded in memory into their final manifest
|
|
and content heads before merging. An interrupted fetch restarts from the last
|
|
incorporated cursor; pagination progress is not persisted. Intermediate changes
|
|
that the server has already reverted must not overwrite local edits.
|
|
|
|
A permanent upload rejection pauses all sync, including after restart. The exact
|
|
pending request and rejection message stay in metadata, and the failure remains
|
|
visible in history and `waitUntilFinished()`. Resetting the client or changing
|
|
settings retries that request before reading newer remote changes. This recovers
|
|
earlier attempts that committed despite a lost reply without separate receipt
|
|
bookkeeping. Local edits made while paused are uploaded after the pending request
|
|
is resolved; editing a file alone does not retry a rejected request.
|
|
|
|
For each document ID (absence means deletion), use the server path if local is
|
|
unchanged, use local if the server is unchanged, and use the server when both
|
|
changed. Then resolve cross-document path conflicts: a retained server path wins,
|
|
UUID ordering breaks other ties, and displaced documents receive deterministic
|
|
`(conflict UUID)` names. This also handles file/ancestor and portable-name aliases.
|
|
Nonportable local names are repaired client-side before a file manifest is published.
|
|
Allocated filename components fit within 255 UTF-8 bytes, including conflict suffixes.
|
|
The protocol does not impose a universal total-path byte limit; filesystem-specific
|
|
limits surface from the adapter instead of silently flattening a name. Directories
|
|
are implicit; empty directories are not synced.
|
|
|
|
- Content is uploaded before new membership; the client chooses and persists UUIDs.
|
|
- Text uses three-way `reconcile`. First contact at a matching path adopts the
|
|
remote ID and reconciles differing text with an empty parent.
|
|
- In-memory snapshots contain bytes and their hash. Base64 encoding is limited to
|
|
wire payloads; pending requests persist the exact payload needed for retries.
|
|
- Concurrent binary/unmergeable edits keep the server version. If only one side
|
|
changed, keep that side. No local backup archive is created.
|
|
- Deletion follows the file manifest decision, including when the file has local edits.
|
|
- Notified moves preserve IDs. Offline inference requires a unique nonempty hash
|
|
match on both sides; uncertain cases become delete/create.
|
|
- Ignored and oversized files preserve their disk paths and remote membership.
|
|
The original manifest base is retained while excluded, so becoming eligible
|
|
resumes reconciliation without losing local renames. Not-yet-downloaded files
|
|
are not mistaken for local deletions.
|
|
- Bytes and line endings are preserved. File modification times are not tracked
|
|
or synchronized. Optional editor cursors are repositioned when text is reconciled.
|
|
|
|
## Restored server histories
|
|
|
|
Successful HTTP responses carry an `X-Vault-Link-History` checkpoint. The client
|
|
persists it before consuming the response and supplies it on subsequent reads and
|
|
writes. WebSocket events wake HTTP replay, which validates that checkpoint. A
|
|
restored database cannot silently reuse a version number: a missing or different
|
|
event incarnation returns a history mismatch before executing the request.
|
|
|
|
On mismatch the client scans local work and records a reset before clearing the
|
|
old checkpoint. It clears old requests, receipts, cursors, bases and diff-cache
|
|
entries. Clean files retain only their hash and reconcile against the restored
|
|
snapshot. A file edited after this reset is also preserved as a separate local
|
|
document; recovery never persists another copy of its content. Unsent edits with
|
|
an old base become separate local documents; path conflicts preserve them under
|
|
conflict names instead of applying a diff to an unrelated restored version.
|
|
Recovery resumes after interrupted state saves. Only the current persisted state
|
|
format is supported; old client stores are not converted.
|
|
|
|
Status observers cannot stop the background engine by throwing or rejecting.
|
|
Their failures are reported, and remaining observers continue receiving updates.
|
|
Remote cursor tracking retains the latest future update and the last usable
|
|
position for each device, so cursors remain visible while file content catches up.
|
|
|
|
## Required adapter contract
|
|
|
|
`FileSystemOperations` and `PersistenceProvider` define the adapter operations.
|
|
The CLI and desktop Obsidian share a Node filesystem adapter and atomic full-record
|
|
metadata persistence. Mobile Obsidian uses its host storage APIs. The E2E runner
|
|
builds and tests both integrations, including actual CLI processes.
|
|
The sync client does not coordinate separate processes accessing the same vault.
|
|
|
|
The filesystem adapter must provide complete scans, coherent snapshots, recursive
|
|
directory creation, direct file writes, atomic text updates, no-replace renames, and deletion of
|
|
individual regular files. Directory pruning uses only `rmdir`, never recursive
|
|
file removal. Reject links, special files and paths escaping the vault. An unreadable
|
|
file or directory must fail the scan rather than look like a deletion. Optional
|
|
editor cursor metadata is applied with file content. Text reconciliation runs inside
|
|
`atomicUpdateText` against the latest text and cursors. Files are updated before
|
|
metadata is saved; an interrupted save can safely repeat the content merge.
|
|
|
|
User-file writes have no power-loss durability requirement. The engine has no
|
|
filesystem journal, recovery staging area, replay, backup archive, or flush protocol.
|
|
Swaps and cycles temporarily move occupied files to visible conflict names; a fresh scan
|
|
can discover them if application stops partway through. Partial writes and edits
|
|
made before, during or after an interruption are treated as current disk content.
|
|
The next sync reconciles that content; it does not finish a saved filesystem plan.
|
|
Power loss may lose bytes or rename identity and may cause a merge to run again.
|
|
|
|
Metadata is the one atomic boundary: persistence `save` replaces a complete value
|
|
and an interrupted/failed save leaves an old or new complete value, never a torn
|
|
object. A failed save is reloaded before retrying. No ordering between metadata and
|
|
user-file durability is required. Settings and database saves are serialized.
|
|
Keep metadata outside the user-file namespace or in `.vault-link-sync`, which is
|
|
excluded from scans. Changing a client's vault once it holds identities, offline
|
|
notifications, pending requests or an incorporated event cursor requires a separate state store.
|
|
|
|
Filesystem application saves metadata once when it finishes. Namespace changes detected
|
|
before mutations invalidate the plan. Changes during application are incorporated
|
|
between files using their document identities; the next scan rereads content. Content is reread before
|
|
replacement. Unexpected eligible occupants receive visible conflict names;
|
|
ignored and oversized occupants stay in place. External writers can still change
|
|
files between observations, so these checks do not promise transactional file
|
|
contents or perfect identity reconstruction. There is no persisted-state migration.
|