diff --git a/sync-server/README.md b/sync-server/README.md index 9b11a061..90fc3e48 100644 --- a/sync-server/README.md +++ b/sync-server/README.md @@ -1,50 +1,106 @@ # Sync server -## Document pushes (API v3) +## API v4 -The server stores immutable document versions. One compare-and-swap endpoint, -`PUT /vaults/:vault_id/documents/:document_id`, handles every mutation. The client -generates the document UUID; the JSON body contains: +The server is a compare-and-swap store. Content versions do not contain paths or +deletions. A separately versioned file manifest owns the namespace and membership. +All routes below are under `/vaults/:vault_id` and require bearer authentication +(except the WebSocket, which authenticates its first message). -- `requestId`: a UUID identifying this exact push. -- `parentVersionId`: `null` to create an absent document, otherwise its base version. -- `relativePath`: the desired filename. -- `content`: `{ "type": "Snapshot", "value": "" }`, - `{ "type": "Diff", "value": [3, "inserted text", -2] }`, or `{ "type": "Delete" }`. +| Endpoint | Purpose | +| --- | --- | +| `PUT /documents/:document_id` | CAS an immutable content version. | +| `GET /documents/:document_id` | Current content version, including base64 bytes. | +| `GET /documents/:document_id/versions/:version_id/content` | Immutable raw bytes. | +| `GET /file_manifest` | Current file manifest. | +| `PUT /file_manifest` | CAS the complete file manifest. | +| `GET /vault_snapshot` | File manifest, referenced content heads, and event watermark from one read transaction. | +| `GET /events_since?after=N` | Every event after N, in order, with the current watermark. | +| `GET /ws` | Ordered event delivery and ephemeral cursor positions. | -All pushes return one of: +A content push includes a `Device-Id` header and this JSON body: -- `Accepted`: version metadata for the exact snapshot committed. A retry with the - same request ID returns the original acknowledgement, even after a restart or - subsequent updates. Clients must persist the ID with the unchanged request. - A payload fingerprint rejects reusing that ID for a different mutation. -- `StaleBase`: the current version, including `contentBase64`. Nothing was written; - the client merges locally and submits a new push with a new request ID. - -The base check, path allocation, new version, and durable acknowledgement share -one SQLite write transaction. A deleted document stays deleted; recreating a file -uses a new document ID. Occupied paths receive an available numbered filename. -Text updates accept reconcile-text transport diffs against the base version; -the server only reconstructs snapshots and never merges concurrent edits. - -Snapshot requests use base64, adding roughly one third to the raw file size. -Server and reverse-proxy request limits apply to the encoded JSON body. - -API v3 uses a fresh schema. Update clients together with the server and initialise -new vault databases; compatibility with previous releases is not supported. - -## Creating/resetting the Database for development - -```sh -rm -rf db.sqlite* -sqlx database create --database-url sqlite://db.sqlite3 -sqlx migrate run --source src/app_state/database/migrations --database-url sqlite://db.sqlite3 -cargo sqlx prepare --workspace +```json +{ + "requestId": "2e3ac3fe-a8d2-49ef-bdba-30ab5b7336f1", + "parentVersionId": null, + "content": { "type": "Snapshot", "value": "aGVsbG8=" } +} ``` -## Updating the DB schema through a migration +`parentVersionId: null` creates previously absent content using the UUID in the +URL. Subsequent pushes name its current version. `Diff` with a `value` array of +numbers/strings is also supported; the server only reconstructs the supplied +snapshot. File modification times are not part of the protocol. Paths and +`Delete` payloads are rejected. -```sh -sqlx migrate add --source src/app_state/database/migrations -sqlx migrate run --source src/app_state/database/migrations --database-url sqlite://db.sqlite3 +A file manifest push is: + +```json +{ + "requestId": "69e6a1ba-1f20-4d81-9ae2-e7812c321e65", + "parentFileManifestId": 0, + "entries": { "7534676e-b51a-42d6-8a62-e70a477f731d": "notes/hello.md" } +} ``` + +The initial empty file manifest has version 0. Upload a new document's content +first, then publish its ID in the file manifest. Removal from the file manifest +deletes membership; content updates never restore membership. Unreferenced +content remains readable. + +File manifest versions are stored in `file_manifests`, with one row per +UUID-to-path mapping in `file_manifest_entries`. Each version has a header row +even when its entries are empty. Entries are unique by document ID and exact path +within a version; portable path validation also rejects aliases and directory +conflicts. Reads reconstruct the complete map, and vault snapshots join these +rows to the latest content versions. The version, entries, and complete file +manifest event in `events.event_json` commit atomically. Event replay and retry +acknowledgements continue to use `event_json`. + +Both pushes return a tagged `Accepted` or `StaleBase` response. An accepted file +manifest response returns its assigned file manifest ID. A stale content response +includes bytes; a stale file manifest response includes the complete map. +Accepted requests store their unique request ID and fingerprint in the event row, +atomically with the version. Retrying the exact request reconstructs the original +`Accepted` response from that event before checking the current head, even after +later changes or a restart. Changing a payload while reusing its request ID is +rejected, including reuse across content and file manifest pushes. Session/device +identity is not part of the fingerprint. + +File manifest validation rejects duplicate UUID keys (including aliases), missing +content, path aliases, inconsistent directory spelling, and file/ancestor +conflicts. Paths use `/`, NFC normalization, at most 240 UTF-8 bytes, and portable +Windows filename restrictions. Alias comparison is NFC → Unicode uppercase → +NFC. `.vault-link-sync` and its aliases are reserved. The server never sanitizes +or allocates a replacement name. + +## Ordering and recovery + +Every accepted mutation appends one event in the same SQLite write transaction +as its state. Events also provide durable request tracking; there is no separate +receipt table. Event IDs are contiguous safe JavaScript integers; +content/file-manifest version IDs are the event IDs that created them. Rejected +requests and retries of accepted requests append no events. Timestamps never +determine event ordering. + +The WebSocket handshake retains `lastSeenVaultUpdateId`. The server sends +`{type: "vaultEvents", headEventId, events}` in increasing order, including events +originated by that client. Each connection drains the database through one sender. +In-memory notifications only wake it; polling covers missed wakes and cancellation +after commit. Reconnect with the last durably processed event ID. The HTTP replay +endpoint uses the same history, so losing the final notification is recoverable. +A fresh client reads `/vault_snapshot`, applies it durably, then replays after its +watermark. +Cursor notifications are ephemeral and do not consume event IDs. + +SQLite uses WAL, `synchronous=FULL`, and `fullfsync` where supported. On Unix, startup also +flushes database directory entries; the filesystem must support those durability +operations. Content, file manifests, and events (including request fingerprints) +are retained indefinitely. + +API v4 requires fresh vault databases and upgraded clients. Existing database +migration checksums intentionally reject earlier schemas; there is no automatic +erasure or migration. Use a new database directory and preserve old data. +Base64 increases snapshot body size by about one third; request limits apply to +the encoded JSON body. diff --git a/sync-server/src/app_state.rs b/sync-server/src/app_state.rs index 2019e08e..910564f9 100644 --- a/sync-server/src/app_state.rs +++ b/sync-server/src/app_state.rs @@ -20,7 +20,7 @@ pub struct AppState { impl AppState { pub async fn try_new(config: Config) -> Result { let broadcasts = Broadcasts::new(&config.server); - let database = Database::try_new(&config.database, &broadcasts).await?; + let database = Database::try_new(&config.database).await?; let cursors: Cursors = Cursors::new(&config.database, &broadcasts); Cursors::start_background_task(cursors.clone()); diff --git a/sync-server/src/errors.rs b/sync-server/src/errors.rs index 831b0e86..c505b8ae 100644 --- a/sync-server/src/errors.rs +++ b/sync-server/src/errors.rs @@ -5,7 +5,7 @@ use axum::{ http::StatusCode, response::{IntoResponse, Response}, }; -use log::{debug, error}; +use log::debug; use serde::Serialize; use thiserror::Error; use ts_rs::TS; diff --git a/sync-server/src/server.rs b/sync-server/src/server.rs index ecc01de9..27ff1155 100644 --- a/sync-server/src/server.rs +++ b/sync-server/src/server.rs @@ -93,18 +93,22 @@ pub async fn create_server(config: Config) -> Result<()> { fn get_authed_routes(app_state: AppState) -> Router { Router::new() .route( - "/vaults/:vault_id/manifest", - get(endpoints::get_manifest::get_manifest).put(endpoints::put_manifest::put_manifest), + "/vaults/:vault_id/file_manifest", + get(endpoints::get_file_manifest::get_file_manifest) + .put(endpoints::put_file_manifest::put_file_manifest), ) .route( - "/vaults/:vault_id/state", - get(endpoints::snapshot::snapshot), + "/vaults/:vault_id/vault_snapshot", + get(endpoints::vault_snapshot::vault_snapshot), + ) + .route( + "/vaults/:vault_id/events_since", + get(endpoints::events::events), ) - .route("/vaults/:vault_id/events", get(endpoints::events::events)) .route( "/vaults/:vault_id/documents/:document_id", get(endpoints::fetch_latest_document_version::fetch_latest_document_version) - .put(endpoints::push_document::push_document), + .put(endpoints::put_file_content::put_file_content), ) .route( "/vaults/:vault_id/documents/:document_id/versions/:vault_update_id/content", diff --git a/sync-server/src/utils.rs b/sync-server/src/utils.rs index 35d6a340..4798e39c 100644 --- a/sync-server/src/utils.rs +++ b/sync-server/src/utils.rs @@ -1,6 +1,4 @@ -pub mod dedup_paths; -pub mod find_first_available_path; pub mod log_level; pub mod normalize; +pub mod portable_path; pub mod rotating_file_writer; -pub mod sanitize_path; diff --git a/sync-server/src/utils/portable_path.rs b/sync-server/src/utils/portable_path.rs new file mode 100644 index 00000000..34551abf --- /dev/null +++ b/sync-server/src/utils/portable_path.rs @@ -0,0 +1,223 @@ +use crate::app_state::database::models::FileManifestEntries; +use anyhow::{Result, bail, ensure}; +use std::collections::BTreeMap; +use unicode_normalization::UnicodeNormalization; + +pub const INTERNAL_DIRECTORY: &str = ".vault-link-sync"; + +// Use the same locale-independent Unicode upper-case mapping in the client. +// NFC -> uppercase -> NFC also catches ß/SS and final/ordinary sigma aliases. +pub fn path_key(path: &str) -> String { + path.nfc() + .collect::() + .to_uppercase() + .nfc() + .collect() +} + +pub fn validate_file_manifest(entries: &FileManifestEntries) -> Result<()> { + let mut nodes: BTreeMap = BTreeMap::new(); + + for path in entries.values() { + ensure!( + !path.is_empty() && path.len() <= 240, + "Path must contain 1–240 UTF-8 bytes" + ); + + ensure!( + path.nfc().collect::() == *path, + "Paths must be NFC normalized" + ); + + let parts: Vec<_> = path.split('/').collect(); + + ensure!( + path_key(parts[0]) != path_key(INTERNAL_DIRECTORY), + "Reserved sync directory" + ); + + for (index, part) in parts.iter().enumerate() { + ensure!( + !part.is_empty() && *part != "." && *part != "..", + "Invalid path component" + ); + + ensure!(!part.ends_with(['.', ' ']), "Trailing dot or space in path"); + + ensure!( + !part + .chars() + .any(|c| c.is_control() || c == '\\' || "<>:\"|?*".contains(c)), + "Non-portable filename" + ); + + let stem = part + .split('.') + .next() + .unwrap_or_default() + .trim_end_matches(' ') + .to_uppercase(); + + ensure!( + !matches!( + stem.as_str(), + "CON" | "PRN" | "AUX" | "NUL" | "CONIN$" | "CONOUT$" + ), + "Reserved filename" + ); + + if stem.starts_with("COM") || stem.starts_with("LPT") { + ensure!( + !matches!( + &stem[3..], + "1" | "2" | "3" | "4" | "5" | "6" | "7" | "8" | "9" | "¹" | "²" | "³" + ), + "Reserved filename" + ); + } + + let prefix = parts[..=index].join("/"); + let is_file = index == parts.len() - 1; + + if let Some((spelling, previous_is_file)) = nodes.get(&path_key(&prefix)) { + if *spelling != prefix || *previous_is_file || is_file { + bail!("Conflicting path: {path}"); + } + } else { + nodes.insert(path_key(&prefix), (prefix, is_file)); + } + } + } + + Ok(()) +} + +#[cfg(test)] +mod tests { + use super::{FileManifestEntries, path_key, validate_file_manifest}; + use uuid::Uuid; + + fn manifest(paths: &[&str]) -> FileManifestEntries { + paths + .iter() + .enumerate() + .map(|(index, path)| (Uuid::from_u128(index as u128), (*path).to_owned())) + .collect() + } + + #[test] + fn path_keys_normalize_unicode_and_case_aliases() { + for (path, expected) in [ + ("Notes/Café.md", "NOTES/CAFÉ.MD"), + ("notes/Cafe\u{301}.md", "NOTES/CAFÉ.MD"), + ("Straße.md", "STRASSE.MD"), + ("σς.md", "ΣΣ.MD"), + ] { + assert_eq!(path_key(path), expected, "path: {path:?}"); + } + } + + #[test] + fn accepts_empty_manifests_and_portable_paths_with_shared_directories() { + validate_file_manifest(&manifest(&[])).unwrap(); + + let entries = manifest(&[ + "Notes/Café.md", + "Notes/日記.md", + "Notes/Archive/old.md", + ".gitkeep", + "COM10.txt", + "LPT0.txt", + ".vault-link-sync-backup/state.json", + "Notes/.vault-link-sync/state.json", + ]); + + validate_file_manifest(&entries).unwrap(); + } + + #[test] + fn enforces_the_path_length_limit_in_utf8_bytes() { + for path in ["a".repeat(240), "é".repeat(120)] { + validate_file_manifest(&manifest(&[&path])).unwrap(); + + let too_long = format!("{path}a"); + + assert!( + validate_file_manifest(&manifest(&[&too_long])).is_err(), + "accepted a path over 240 bytes: {too_long:?}" + ); + } + } + + #[test] + fn rejects_invalid_components_characters_and_non_normalized_paths() { + for path in [ + "", + "/notes.md", + "notes.md/", + "notes//a.md", + "./notes.md", + "notes/../a.md", + "notes./a.md", + "notes /a.md", + "Cafe\u{301}.md", + ] { + assert!( + validate_file_manifest(&manifest(&[path])).is_err(), + "accepted invalid path: {path:?}" + ); + } + + for character in ['\\', '<', '>', ':', '"', '|', '?', '*', '\0', '\u{85}'] { + let path = format!("notes/a{character}b.md"); + + assert!( + validate_file_manifest(&manifest(&[&path])).is_err(), + "accepted non-portable character in path: {path:?}" + ); + } + } + + #[test] + fn rejects_reserved_sync_and_windows_names() { + for path in [ + ".vault-link-sync", + ".VAULT-LINK-SYNC/state.json", + "con", + "prn.txt", + "AUX.md", + "nul .txt", + "CONIN$", + "conout$.log", + "com1.txt", + "LPT9", + "COM¹.txt", + "lpt²/file.md", + "COM³.txt", + ] { + assert!( + validate_file_manifest(&manifest(&[path])).is_err(), + "accepted reserved path: {path:?}" + ); + } + } + + #[test] + fn rejects_path_conflicts_in_either_entry_order() { + for [first, second] in [ + ["notes.md", "notes.md"], + ["Notes.md", "notes.md"], + ["Straße.md", "STRASSE.md"], + ["ς.md", "σ.md"], + ["Notes/a.md", "notes/b.md"], + ["notes", "notes/a.md"], + ] { + for paths in [[first, second], [second, first]] { + assert!( + validate_file_manifest(&manifest(&paths)).is_err(), + "accepted conflicting paths: {paths:?}" + ); + } + } + } +}