Wire it up

sync-server/README.md

@@ -1,50 +1,106 @@
 # Sync server
 
-## Document pushes (API v3)
+## API v4
 
-The server stores immutable document versions. One compare-and-swap endpoint,
+The server is a compare-and-swap store. Content versions do not contain paths or
-`PUT /vaults/:vault_id/documents/:document_id`, handles every mutation. The client
+deletions. A separately versioned file manifest owns the namespace and membership.
-generates the document UUID; the JSON body contains:
+All routes below are under `/vaults/:vault_id` and require bearer authentication
+(except the WebSocket, which authenticates its first message).
 
-- `requestId`: a UUID identifying this exact push.
+| Endpoint | Purpose |
-- `parentVersionId`: `null` to create an absent document, otherwise its base version.
+| --- | --- |
-- `relativePath`: the desired filename.
+| `PUT /documents/:document_id` | CAS an immutable content version. |
-- `content`: `{ "type": "Snapshot", "value": "<base64>" }`,
+| `GET /documents/:document_id` | Current content version, including base64 bytes. |
-  `{ "type": "Diff", "value": [3, "inserted text", -2] }`, or `{ "type": "Delete" }`.
+| `GET /documents/:document_id/versions/:version_id/content` | Immutable raw bytes. |
+| `GET /file_manifest` | Current file manifest. |
+| `PUT /file_manifest` | CAS the complete file manifest. |
+| `GET /vault_snapshot` | File manifest, referenced content heads, and event watermark from one read transaction. |
+| `GET /events_since?after=N` | Every event after N, in order, with the current watermark. |
+| `GET /ws` | Ordered event delivery and ephemeral cursor positions. |
 
-All pushes return one of:
+A content push includes a `Device-Id` header and this JSON body:
 
-- `Accepted`: version metadata for the exact snapshot committed. A retry with the
+```json
-  same request ID returns the original acknowledgement, even after a restart or
+{
-  subsequent updates. Clients must persist the ID with the unchanged request.
+  "requestId": "2e3ac3fe-a8d2-49ef-bdba-30ab5b7336f1",
-  A payload fingerprint rejects reusing that ID for a different mutation.
+  "parentVersionId": null,
-- `StaleBase`: the current version, including `contentBase64`. Nothing was written;
+  "content": { "type": "Snapshot", "value": "aGVsbG8=" }
-  the client merges locally and submits a new push with a new request ID.
+}
-
-The base check, path allocation, new version, and durable acknowledgement share
-one SQLite write transaction. A deleted document stays deleted; recreating a file
-uses a new document ID. Occupied paths receive an available numbered filename.
-Text updates accept reconcile-text transport diffs against the base version;
-the server only reconstructs snapshots and never merges concurrent edits.
-
-Snapshot requests use base64, adding roughly one third to the raw file size.
-Server and reverse-proxy request limits apply to the encoded JSON body.
-
-API v3 uses a fresh schema. Update clients together with the server and initialise
-new vault databases; compatibility with previous releases is not supported.
-
-## Creating/resetting the Database for development
-
-```sh
-rm -rf db.sqlite*
-sqlx database create --database-url sqlite://db.sqlite3
-sqlx migrate run --source src/app_state/database/migrations --database-url sqlite://db.sqlite3
-cargo sqlx prepare --workspace
 ```
 
-## Updating the DB schema through a migration
+`parentVersionId: null` creates previously absent content using the UUID in the
+URL. Subsequent pushes name its current version. `Diff` with a `value` array of
+numbers/strings is also supported; the server only reconstructs the supplied
+snapshot. File modification times are not part of the protocol. Paths and
+`Delete` payloads are rejected.
 
-```sh
+A file manifest push is:
-sqlx migrate add --source src/app_state/database/migrations <name>
+
-sqlx migrate run --source src/app_state/database/migrations --database-url sqlite://db.sqlite3
+```json
+{
+  "requestId": "69e6a1ba-1f20-4d81-9ae2-e7812c321e65",
+  "parentFileManifestId": 0,
+  "entries": { "7534676e-b51a-42d6-8a62-e70a477f731d": "notes/hello.md" }
+}
 ```
+
+The initial empty file manifest has version 0. Upload a new document's content
+first, then publish its ID in the file manifest. Removal from the file manifest
+deletes membership; content updates never restore membership. Unreferenced
+content remains readable.
+
+File manifest versions are stored in `file_manifests`, with one row per
+UUID-to-path mapping in `file_manifest_entries`. Each version has a header row
+even when its entries are empty. Entries are unique by document ID and exact path
+within a version; portable path validation also rejects aliases and directory
+conflicts. Reads reconstruct the complete map, and vault snapshots join these
+rows to the latest content versions. The version, entries, and complete file
+manifest event in `events.event_json` commit atomically. Event replay and retry
+acknowledgements continue to use `event_json`.
+
+Both pushes return a tagged `Accepted` or `StaleBase` response. An accepted file
+manifest response returns its assigned file manifest ID. A stale content response
+includes bytes; a stale file manifest response includes the complete map.
+Accepted requests store their unique request ID and fingerprint in the event row,
+atomically with the version. Retrying the exact request reconstructs the original
+`Accepted` response from that event before checking the current head, even after
+later changes or a restart. Changing a payload while reusing its request ID is
+rejected, including reuse across content and file manifest pushes. Session/device
+identity is not part of the fingerprint.
+
+File manifest validation rejects duplicate UUID keys (including aliases), missing
+content, path aliases, inconsistent directory spelling, and file/ancestor
+conflicts. Paths use `/`, NFC normalization, at most 240 UTF-8 bytes, and portable
+Windows filename restrictions. Alias comparison is NFC → Unicode uppercase →
+NFC. `.vault-link-sync` and its aliases are reserved. The server never sanitizes
+or allocates a replacement name.
+
+## Ordering and recovery
+
+Every accepted mutation appends one event in the same SQLite write transaction
+as its state. Events also provide durable request tracking; there is no separate
+receipt table. Event IDs are contiguous safe JavaScript integers;
+content/file-manifest version IDs are the event IDs that created them. Rejected
+requests and retries of accepted requests append no events. Timestamps never
+determine event ordering.
+
+The WebSocket handshake retains `lastSeenVaultUpdateId`. The server sends
+`{type: "vaultEvents", headEventId, events}` in increasing order, including events
+originated by that client. Each connection drains the database through one sender.
+In-memory notifications only wake it; polling covers missed wakes and cancellation
+after commit. Reconnect with the last durably processed event ID. The HTTP replay
+endpoint uses the same history, so losing the final notification is recoverable.
+A fresh client reads `/vault_snapshot`, applies it durably, then replays after its
+watermark.
+Cursor notifications are ephemeral and do not consume event IDs.
+
+SQLite uses WAL, `synchronous=FULL`, and `fullfsync` where supported. On Unix, startup also
+flushes database directory entries; the filesystem must support those durability
+operations. Content, file manifests, and events (including request fingerprints)
+are retained indefinitely.
+
+API v4 requires fresh vault databases and upgraded clients. Existing database
+migration checksums intentionally reject earlier schemas; there is no automatic
+erasure or migration. Use a new database directory and preserve old data.
+Base64 increases snapshot body size by about one third; request limits apply to
+the encoded JSON body.

sync-server/src/app_state.rs

@@ -20,7 +20,7 @@ pub struct AppState {
 impl AppState {
     pub async fn try_new(config: Config) -> Result<Self> {
         let broadcasts = Broadcasts::new(&config.server);
-        let database = Database::try_new(&config.database, &broadcasts).await?;
+        let database = Database::try_new(&config.database).await?;
         let cursors: Cursors = Cursors::new(&config.database, &broadcasts);
 
         Cursors::start_background_task(cursors.clone());

sync-server/src/errors.rs

@@ -5,7 +5,7 @@ use axum::{
     http::StatusCode,
     response::{IntoResponse, Response},
 };
-use log::{debug, error};
+use log::debug;
 use serde::Serialize;
 use thiserror::Error;
 use ts_rs::TS;

sync-server/src/server.rs

@@ -93,18 +93,22 @@ pub async fn create_server(config: Config) -> Result<()> {
 fn get_authed_routes(app_state: AppState) -> Router<AppState> {
     Router::new()
         .route(
-            "/vaults/:vault_id/manifest",
+            "/vaults/:vault_id/file_manifest",
-            get(endpoints::get_manifest::get_manifest).put(endpoints::put_manifest::put_manifest),
+            get(endpoints::get_file_manifest::get_file_manifest)
+                .put(endpoints::put_file_manifest::put_file_manifest),
         )
         .route(
-            "/vaults/:vault_id/state",
+            "/vaults/:vault_id/vault_snapshot",
-            get(endpoints::snapshot::snapshot),
+            get(endpoints::vault_snapshot::vault_snapshot),
+        )
+        .route(
+            "/vaults/:vault_id/events_since",
+            get(endpoints::events::events),
         )
-        .route("/vaults/:vault_id/events", get(endpoints::events::events))
         .route(
             "/vaults/:vault_id/documents/:document_id",
             get(endpoints::fetch_latest_document_version::fetch_latest_document_version)
-                .put(endpoints::push_document::push_document),
+                .put(endpoints::put_file_content::put_file_content),
         )
         .route(
             "/vaults/:vault_id/documents/:document_id/versions/:vault_update_id/content",

sync-server/src/utils.rs

@@ -1,6 +1,4 @@
-pub mod dedup_paths;
-pub mod find_first_available_path;
 pub mod log_level;
 pub mod normalize;
+pub mod portable_path;
 pub mod rotating_file_writer;
-pub mod sanitize_path;

sync-server/src/utils/portable_path.rs

@@ -0,0 +1,223 @@
+use crate::app_state::database::models::FileManifestEntries;
+use anyhow::{Result, bail, ensure};
+use std::collections::BTreeMap;
+use unicode_normalization::UnicodeNormalization;
+
+pub const INTERNAL_DIRECTORY: &str = ".vault-link-sync";
+
+// Use the same locale-independent Unicode upper-case mapping in the client.
+// NFC -> uppercase -> NFC also catches ß/SS and final/ordinary sigma aliases.
+pub fn path_key(path: &str) -> String {
+    path.nfc()
+        .collect::<String>()
+        .to_uppercase()
+        .nfc()
+        .collect()
+}
+
+pub fn validate_file_manifest(entries: &FileManifestEntries) -> Result<()> {
+    let mut nodes: BTreeMap<String, (String, bool)> = BTreeMap::new();
+
+    for path in entries.values() {
+        ensure!(
+            !path.is_empty() && path.len() <= 240,
+            "Path must contain 1–240 UTF-8 bytes"
+        );
+
+        ensure!(
+            path.nfc().collect::<String>() == *path,
+            "Paths must be NFC normalized"
+        );
+
+        let parts: Vec<_> = path.split('/').collect();
+
+        ensure!(
+            path_key(parts[0]) != path_key(INTERNAL_DIRECTORY),
+            "Reserved sync directory"
+        );
+
+        for (index, part) in parts.iter().enumerate() {
+            ensure!(
+                !part.is_empty() && *part != "." && *part != "..",
+                "Invalid path component"
+            );
+
+            ensure!(!part.ends_with(['.', ' ']), "Trailing dot or space in path");
+
+            ensure!(
+                !part
+                    .chars()
+                    .any(|c| c.is_control() || c == '\\' || "<>:\"|?*".contains(c)),
+                "Non-portable filename"
+            );
+
+            let stem = part
+                .split('.')
+                .next()
+                .unwrap_or_default()
+                .trim_end_matches(' ')
+                .to_uppercase();
+
+            ensure!(
+                !matches!(
+                    stem.as_str(),
+                    "CON" | "PRN" | "AUX" | "NUL" | "CONIN$" | "CONOUT$"
+                ),
+                "Reserved filename"
+            );
+
+            if stem.starts_with("COM") || stem.starts_with("LPT") {
+                ensure!(
+                    !matches!(
+                        &stem[3..],
+                        "1" | "2" | "3" | "4" | "5" | "6" | "7" | "8" | "9" | "¹" | "²" | "³"
+                    ),
+                    "Reserved filename"
+                );
+            }
+
+            let prefix = parts[..=index].join("/");
+            let is_file = index == parts.len() - 1;
+
+            if let Some((spelling, previous_is_file)) = nodes.get(&path_key(&prefix)) {
+                if *spelling != prefix || *previous_is_file || is_file {
+                    bail!("Conflicting path: {path}");
+                }
+            } else {
+                nodes.insert(path_key(&prefix), (prefix, is_file));
+            }
+        }
+    }
+
+    Ok(())
+}
+
+#[cfg(test)]
+mod tests {
+    use super::{FileManifestEntries, path_key, validate_file_manifest};
+    use uuid::Uuid;
+
+    fn manifest(paths: &[&str]) -> FileManifestEntries {
+        paths
+            .iter()
+            .enumerate()
+            .map(|(index, path)| (Uuid::from_u128(index as u128), (*path).to_owned()))
+            .collect()
+    }
+
+    #[test]
+    fn path_keys_normalize_unicode_and_case_aliases() {
+        for (path, expected) in [
+            ("Notes/Café.md", "NOTES/CAFÉ.MD"),
+            ("notes/Cafe\u{301}.md", "NOTES/CAFÉ.MD"),
+            ("Straße.md", "STRASSE.MD"),
+            ("σς.md", "ΣΣ.MD"),
+        ] {
+            assert_eq!(path_key(path), expected, "path: {path:?}");
+        }
+    }
+
+    #[test]
+    fn accepts_empty_manifests_and_portable_paths_with_shared_directories() {
+        validate_file_manifest(&manifest(&[])).unwrap();
+
+        let entries = manifest(&[
+            "Notes/Café.md",
+            "Notes/日記.md",
+            "Notes/Archive/old.md",
+            ".gitkeep",
+            "COM10.txt",
+            "LPT0.txt",
+            ".vault-link-sync-backup/state.json",
+            "Notes/.vault-link-sync/state.json",
+        ]);
+
+        validate_file_manifest(&entries).unwrap();
+    }
+
+    #[test]
+    fn enforces_the_path_length_limit_in_utf8_bytes() {
+        for path in ["a".repeat(240), "é".repeat(120)] {
+            validate_file_manifest(&manifest(&[&path])).unwrap();
+
+            let too_long = format!("{path}a");
+
+            assert!(
+                validate_file_manifest(&manifest(&[&too_long])).is_err(),
+                "accepted a path over 240 bytes: {too_long:?}"
+            );
+        }
+    }
+
+    #[test]
+    fn rejects_invalid_components_characters_and_non_normalized_paths() {
+        for path in [
+            "",
+            "/notes.md",
+            "notes.md/",
+            "notes//a.md",
+            "./notes.md",
+            "notes/../a.md",
+            "notes./a.md",
+            "notes /a.md",
+            "Cafe\u{301}.md",
+        ] {
+            assert!(
+                validate_file_manifest(&manifest(&[path])).is_err(),
+                "accepted invalid path: {path:?}"
+            );
+        }
+
+        for character in ['\\', '<', '>', ':', '"', '|', '?', '*', '\0', '\u{85}'] {
+            let path = format!("notes/a{character}b.md");
+
+            assert!(
+                validate_file_manifest(&manifest(&[&path])).is_err(),
+                "accepted non-portable character in path: {path:?}"
+            );
+        }
+    }
+
+    #[test]
+    fn rejects_reserved_sync_and_windows_names() {
+        for path in [
+            ".vault-link-sync",
+            ".VAULT-LINK-SYNC/state.json",
+            "con",
+            "prn.txt",
+            "AUX.md",
+            "nul .txt",
+            "CONIN$",
+            "conout$.log",
+            "com1.txt",
+            "LPT9",
+            "COM¹.txt",
+            "lpt²/file.md",
+            "COM³.txt",
+        ] {
+            assert!(
+                validate_file_manifest(&manifest(&[path])).is_err(),
+                "accepted reserved path: {path:?}"
+            );
+        }
+    }
+
+    #[test]
+    fn rejects_path_conflicts_in_either_entry_order() {
+        for [first, second] in [
+            ["notes.md", "notes.md"],
+            ["Notes.md", "notes.md"],
+            ["Straße.md", "STRASSE.md"],
+            ["ς.md", "σ.md"],
+            ["Notes/a.md", "notes/b.md"],
+            ["notes", "notes/a.md"],
+        ] {
+            for paths in [[first, second], [second, first]] {
+                assert!(
+                    validate_file_manifest(&manifest(&paths)).is_err(),
+                    "accepted conflicting paths: {paths:?}"
+                );
+            }
+        }
+    }
+}