Wire it up
This commit is contained in:
parent
69a1e91d4b
commit
8a85a68dbb
6 changed files with 336 additions and 55 deletions
|
|
@ -1,50 +1,106 @@
|
|||
# Sync server
|
||||
|
||||
## Document pushes (API v3)
|
||||
## API v4
|
||||
|
||||
The server stores immutable document versions. One compare-and-swap endpoint,
|
||||
`PUT /vaults/:vault_id/documents/:document_id`, handles every mutation. The client
|
||||
generates the document UUID; the JSON body contains:
|
||||
The server is a compare-and-swap store. Content versions do not contain paths or
|
||||
deletions. A separately versioned file manifest owns the namespace and membership.
|
||||
All routes below are under `/vaults/:vault_id` and require bearer authentication
|
||||
(except the WebSocket, which authenticates its first message).
|
||||
|
||||
- `requestId`: a UUID identifying this exact push.
|
||||
- `parentVersionId`: `null` to create an absent document, otherwise its base version.
|
||||
- `relativePath`: the desired filename.
|
||||
- `content`: `{ "type": "Snapshot", "value": "<base64>" }`,
|
||||
`{ "type": "Diff", "value": [3, "inserted text", -2] }`, or `{ "type": "Delete" }`.
|
||||
| Endpoint | Purpose |
|
||||
| --- | --- |
|
||||
| `PUT /documents/:document_id` | CAS an immutable content version. |
|
||||
| `GET /documents/:document_id` | Current content version, including base64 bytes. |
|
||||
| `GET /documents/:document_id/versions/:version_id/content` | Immutable raw bytes. |
|
||||
| `GET /file_manifest` | Current file manifest. |
|
||||
| `PUT /file_manifest` | CAS the complete file manifest. |
|
||||
| `GET /vault_snapshot` | File manifest, referenced content heads, and event watermark from one read transaction. |
|
||||
| `GET /events_since?after=N` | Every event after N, in order, with the current watermark. |
|
||||
| `GET /ws` | Ordered event delivery and ephemeral cursor positions. |
|
||||
|
||||
All pushes return one of:
|
||||
A content push includes a `Device-Id` header and this JSON body:
|
||||
|
||||
- `Accepted`: version metadata for the exact snapshot committed. A retry with the
|
||||
same request ID returns the original acknowledgement, even after a restart or
|
||||
subsequent updates. Clients must persist the ID with the unchanged request.
|
||||
A payload fingerprint rejects reusing that ID for a different mutation.
|
||||
- `StaleBase`: the current version, including `contentBase64`. Nothing was written;
|
||||
the client merges locally and submits a new push with a new request ID.
|
||||
|
||||
The base check, path allocation, new version, and durable acknowledgement share
|
||||
one SQLite write transaction. A deleted document stays deleted; recreating a file
|
||||
uses a new document ID. Occupied paths receive an available numbered filename.
|
||||
Text updates accept reconcile-text transport diffs against the base version;
|
||||
the server only reconstructs snapshots and never merges concurrent edits.
|
||||
|
||||
Snapshot requests use base64, adding roughly one third to the raw file size.
|
||||
Server and reverse-proxy request limits apply to the encoded JSON body.
|
||||
|
||||
API v3 uses a fresh schema. Update clients together with the server and initialise
|
||||
new vault databases; compatibility with previous releases is not supported.
|
||||
|
||||
## Creating/resetting the Database for development
|
||||
|
||||
```sh
|
||||
rm -rf db.sqlite*
|
||||
sqlx database create --database-url sqlite://db.sqlite3
|
||||
sqlx migrate run --source src/app_state/database/migrations --database-url sqlite://db.sqlite3
|
||||
cargo sqlx prepare --workspace
|
||||
```json
|
||||
{
|
||||
"requestId": "2e3ac3fe-a8d2-49ef-bdba-30ab5b7336f1",
|
||||
"parentVersionId": null,
|
||||
"content": { "type": "Snapshot", "value": "aGVsbG8=" }
|
||||
}
|
||||
```
|
||||
|
||||
## Updating the DB schema through a migration
|
||||
`parentVersionId: null` creates previously absent content using the UUID in the
|
||||
URL. Subsequent pushes name its current version. `Diff` with a `value` array of
|
||||
numbers/strings is also supported; the server only reconstructs the supplied
|
||||
snapshot. File modification times are not part of the protocol. Paths and
|
||||
`Delete` payloads are rejected.
|
||||
|
||||
```sh
|
||||
sqlx migrate add --source src/app_state/database/migrations <name>
|
||||
sqlx migrate run --source src/app_state/database/migrations --database-url sqlite://db.sqlite3
|
||||
A file manifest push is:
|
||||
|
||||
```json
|
||||
{
|
||||
"requestId": "69e6a1ba-1f20-4d81-9ae2-e7812c321e65",
|
||||
"parentFileManifestId": 0,
|
||||
"entries": { "7534676e-b51a-42d6-8a62-e70a477f731d": "notes/hello.md" }
|
||||
}
|
||||
```
|
||||
|
||||
The initial empty file manifest has version 0. Upload a new document's content
|
||||
first, then publish its ID in the file manifest. Removal from the file manifest
|
||||
deletes membership; content updates never restore membership. Unreferenced
|
||||
content remains readable.
|
||||
|
||||
File manifest versions are stored in `file_manifests`, with one row per
|
||||
UUID-to-path mapping in `file_manifest_entries`. Each version has a header row
|
||||
even when its entries are empty. Entries are unique by document ID and exact path
|
||||
within a version; portable path validation also rejects aliases and directory
|
||||
conflicts. Reads reconstruct the complete map, and vault snapshots join these
|
||||
rows to the latest content versions. The version, entries, and complete file
|
||||
manifest event in `events.event_json` commit atomically. Event replay and retry
|
||||
acknowledgements continue to use `event_json`.
|
||||
|
||||
Both pushes return a tagged `Accepted` or `StaleBase` response. An accepted file
|
||||
manifest response returns its assigned file manifest ID. A stale content response
|
||||
includes bytes; a stale file manifest response includes the complete map.
|
||||
Accepted requests store their unique request ID and fingerprint in the event row,
|
||||
atomically with the version. Retrying the exact request reconstructs the original
|
||||
`Accepted` response from that event before checking the current head, even after
|
||||
later changes or a restart. Changing a payload while reusing its request ID is
|
||||
rejected, including reuse across content and file manifest pushes. Session/device
|
||||
identity is not part of the fingerprint.
|
||||
|
||||
File manifest validation rejects duplicate UUID keys (including aliases), missing
|
||||
content, path aliases, inconsistent directory spelling, and file/ancestor
|
||||
conflicts. Paths use `/`, NFC normalization, at most 240 UTF-8 bytes, and portable
|
||||
Windows filename restrictions. Alias comparison is NFC → Unicode uppercase →
|
||||
NFC. `.vault-link-sync` and its aliases are reserved. The server never sanitizes
|
||||
or allocates a replacement name.
|
||||
|
||||
## Ordering and recovery
|
||||
|
||||
Every accepted mutation appends one event in the same SQLite write transaction
|
||||
as its state. Events also provide durable request tracking; there is no separate
|
||||
receipt table. Event IDs are contiguous safe JavaScript integers;
|
||||
content/file-manifest version IDs are the event IDs that created them. Rejected
|
||||
requests and retries of accepted requests append no events. Timestamps never
|
||||
determine event ordering.
|
||||
|
||||
The WebSocket handshake retains `lastSeenVaultUpdateId`. The server sends
|
||||
`{type: "vaultEvents", headEventId, events}` in increasing order, including events
|
||||
originated by that client. Each connection drains the database through one sender.
|
||||
In-memory notifications only wake it; polling covers missed wakes and cancellation
|
||||
after commit. Reconnect with the last durably processed event ID. The HTTP replay
|
||||
endpoint uses the same history, so losing the final notification is recoverable.
|
||||
A fresh client reads `/vault_snapshot`, applies it durably, then replays after its
|
||||
watermark.
|
||||
Cursor notifications are ephemeral and do not consume event IDs.
|
||||
|
||||
SQLite uses WAL, `synchronous=FULL`, and `fullfsync` where supported. On Unix, startup also
|
||||
flushes database directory entries; the filesystem must support those durability
|
||||
operations. Content, file manifests, and events (including request fingerprints)
|
||||
are retained indefinitely.
|
||||
|
||||
API v4 requires fresh vault databases and upgraded clients. Existing database
|
||||
migration checksums intentionally reject earlier schemas; there is no automatic
|
||||
erasure or migration. Use a new database directory and preserve old data.
|
||||
Base64 increases snapshot body size by about one third; request limits apply to
|
||||
the encoded JSON body.
|
||||
|
|
|
|||
|
|
@ -20,7 +20,7 @@ pub struct AppState {
|
|||
impl AppState {
|
||||
pub async fn try_new(config: Config) -> Result<Self> {
|
||||
let broadcasts = Broadcasts::new(&config.server);
|
||||
let database = Database::try_new(&config.database, &broadcasts).await?;
|
||||
let database = Database::try_new(&config.database).await?;
|
||||
let cursors: Cursors = Cursors::new(&config.database, &broadcasts);
|
||||
|
||||
Cursors::start_background_task(cursors.clone());
|
||||
|
|
|
|||
|
|
@ -5,7 +5,7 @@ use axum::{
|
|||
http::StatusCode,
|
||||
response::{IntoResponse, Response},
|
||||
};
|
||||
use log::{debug, error};
|
||||
use log::debug;
|
||||
use serde::Serialize;
|
||||
use thiserror::Error;
|
||||
use ts_rs::TS;
|
||||
|
|
|
|||
|
|
@ -93,18 +93,22 @@ pub async fn create_server(config: Config) -> Result<()> {
|
|||
fn get_authed_routes(app_state: AppState) -> Router<AppState> {
|
||||
Router::new()
|
||||
.route(
|
||||
"/vaults/:vault_id/manifest",
|
||||
get(endpoints::get_manifest::get_manifest).put(endpoints::put_manifest::put_manifest),
|
||||
"/vaults/:vault_id/file_manifest",
|
||||
get(endpoints::get_file_manifest::get_file_manifest)
|
||||
.put(endpoints::put_file_manifest::put_file_manifest),
|
||||
)
|
||||
.route(
|
||||
"/vaults/:vault_id/state",
|
||||
get(endpoints::snapshot::snapshot),
|
||||
"/vaults/:vault_id/vault_snapshot",
|
||||
get(endpoints::vault_snapshot::vault_snapshot),
|
||||
)
|
||||
.route(
|
||||
"/vaults/:vault_id/events_since",
|
||||
get(endpoints::events::events),
|
||||
)
|
||||
.route("/vaults/:vault_id/events", get(endpoints::events::events))
|
||||
.route(
|
||||
"/vaults/:vault_id/documents/:document_id",
|
||||
get(endpoints::fetch_latest_document_version::fetch_latest_document_version)
|
||||
.put(endpoints::push_document::push_document),
|
||||
.put(endpoints::put_file_content::put_file_content),
|
||||
)
|
||||
.route(
|
||||
"/vaults/:vault_id/documents/:document_id/versions/:vault_update_id/content",
|
||||
|
|
|
|||
|
|
@ -1,6 +1,4 @@
|
|||
pub mod dedup_paths;
|
||||
pub mod find_first_available_path;
|
||||
pub mod log_level;
|
||||
pub mod normalize;
|
||||
pub mod portable_path;
|
||||
pub mod rotating_file_writer;
|
||||
pub mod sanitize_path;
|
||||
|
|
|
|||
223
sync-server/src/utils/portable_path.rs
Normal file
223
sync-server/src/utils/portable_path.rs
Normal file
|
|
@ -0,0 +1,223 @@
|
|||
use crate::app_state::database::models::FileManifestEntries;
|
||||
use anyhow::{Result, bail, ensure};
|
||||
use std::collections::BTreeMap;
|
||||
use unicode_normalization::UnicodeNormalization;
|
||||
|
||||
pub const INTERNAL_DIRECTORY: &str = ".vault-link-sync";
|
||||
|
||||
// Use the same locale-independent Unicode upper-case mapping in the client.
|
||||
// NFC -> uppercase -> NFC also catches ß/SS and final/ordinary sigma aliases.
|
||||
pub fn path_key(path: &str) -> String {
|
||||
path.nfc()
|
||||
.collect::<String>()
|
||||
.to_uppercase()
|
||||
.nfc()
|
||||
.collect()
|
||||
}
|
||||
|
||||
pub fn validate_file_manifest(entries: &FileManifestEntries) -> Result<()> {
|
||||
let mut nodes: BTreeMap<String, (String, bool)> = BTreeMap::new();
|
||||
|
||||
for path in entries.values() {
|
||||
ensure!(
|
||||
!path.is_empty() && path.len() <= 240,
|
||||
"Path must contain 1–240 UTF-8 bytes"
|
||||
);
|
||||
|
||||
ensure!(
|
||||
path.nfc().collect::<String>() == *path,
|
||||
"Paths must be NFC normalized"
|
||||
);
|
||||
|
||||
let parts: Vec<_> = path.split('/').collect();
|
||||
|
||||
ensure!(
|
||||
path_key(parts[0]) != path_key(INTERNAL_DIRECTORY),
|
||||
"Reserved sync directory"
|
||||
);
|
||||
|
||||
for (index, part) in parts.iter().enumerate() {
|
||||
ensure!(
|
||||
!part.is_empty() && *part != "." && *part != "..",
|
||||
"Invalid path component"
|
||||
);
|
||||
|
||||
ensure!(!part.ends_with(['.', ' ']), "Trailing dot or space in path");
|
||||
|
||||
ensure!(
|
||||
!part
|
||||
.chars()
|
||||
.any(|c| c.is_control() || c == '\\' || "<>:\"|?*".contains(c)),
|
||||
"Non-portable filename"
|
||||
);
|
||||
|
||||
let stem = part
|
||||
.split('.')
|
||||
.next()
|
||||
.unwrap_or_default()
|
||||
.trim_end_matches(' ')
|
||||
.to_uppercase();
|
||||
|
||||
ensure!(
|
||||
!matches!(
|
||||
stem.as_str(),
|
||||
"CON" | "PRN" | "AUX" | "NUL" | "CONIN$" | "CONOUT$"
|
||||
),
|
||||
"Reserved filename"
|
||||
);
|
||||
|
||||
if stem.starts_with("COM") || stem.starts_with("LPT") {
|
||||
ensure!(
|
||||
!matches!(
|
||||
&stem[3..],
|
||||
"1" | "2" | "3" | "4" | "5" | "6" | "7" | "8" | "9" | "¹" | "²" | "³"
|
||||
),
|
||||
"Reserved filename"
|
||||
);
|
||||
}
|
||||
|
||||
let prefix = parts[..=index].join("/");
|
||||
let is_file = index == parts.len() - 1;
|
||||
|
||||
if let Some((spelling, previous_is_file)) = nodes.get(&path_key(&prefix)) {
|
||||
if *spelling != prefix || *previous_is_file || is_file {
|
||||
bail!("Conflicting path: {path}");
|
||||
}
|
||||
} else {
|
||||
nodes.insert(path_key(&prefix), (prefix, is_file));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
Ok(())
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::{FileManifestEntries, path_key, validate_file_manifest};
|
||||
use uuid::Uuid;
|
||||
|
||||
fn manifest(paths: &[&str]) -> FileManifestEntries {
|
||||
paths
|
||||
.iter()
|
||||
.enumerate()
|
||||
.map(|(index, path)| (Uuid::from_u128(index as u128), (*path).to_owned()))
|
||||
.collect()
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn path_keys_normalize_unicode_and_case_aliases() {
|
||||
for (path, expected) in [
|
||||
("Notes/Café.md", "NOTES/CAFÉ.MD"),
|
||||
("notes/Cafe\u{301}.md", "NOTES/CAFÉ.MD"),
|
||||
("Straße.md", "STRASSE.MD"),
|
||||
("σς.md", "ΣΣ.MD"),
|
||||
] {
|
||||
assert_eq!(path_key(path), expected, "path: {path:?}");
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn accepts_empty_manifests_and_portable_paths_with_shared_directories() {
|
||||
validate_file_manifest(&manifest(&[])).unwrap();
|
||||
|
||||
let entries = manifest(&[
|
||||
"Notes/Café.md",
|
||||
"Notes/日記.md",
|
||||
"Notes/Archive/old.md",
|
||||
".gitkeep",
|
||||
"COM10.txt",
|
||||
"LPT0.txt",
|
||||
".vault-link-sync-backup/state.json",
|
||||
"Notes/.vault-link-sync/state.json",
|
||||
]);
|
||||
|
||||
validate_file_manifest(&entries).unwrap();
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn enforces_the_path_length_limit_in_utf8_bytes() {
|
||||
for path in ["a".repeat(240), "é".repeat(120)] {
|
||||
validate_file_manifest(&manifest(&[&path])).unwrap();
|
||||
|
||||
let too_long = format!("{path}a");
|
||||
|
||||
assert!(
|
||||
validate_file_manifest(&manifest(&[&too_long])).is_err(),
|
||||
"accepted a path over 240 bytes: {too_long:?}"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn rejects_invalid_components_characters_and_non_normalized_paths() {
|
||||
for path in [
|
||||
"",
|
||||
"/notes.md",
|
||||
"notes.md/",
|
||||
"notes//a.md",
|
||||
"./notes.md",
|
||||
"notes/../a.md",
|
||||
"notes./a.md",
|
||||
"notes /a.md",
|
||||
"Cafe\u{301}.md",
|
||||
] {
|
||||
assert!(
|
||||
validate_file_manifest(&manifest(&[path])).is_err(),
|
||||
"accepted invalid path: {path:?}"
|
||||
);
|
||||
}
|
||||
|
||||
for character in ['\\', '<', '>', ':', '"', '|', '?', '*', '\0', '\u{85}'] {
|
||||
let path = format!("notes/a{character}b.md");
|
||||
|
||||
assert!(
|
||||
validate_file_manifest(&manifest(&[&path])).is_err(),
|
||||
"accepted non-portable character in path: {path:?}"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn rejects_reserved_sync_and_windows_names() {
|
||||
for path in [
|
||||
".vault-link-sync",
|
||||
".VAULT-LINK-SYNC/state.json",
|
||||
"con",
|
||||
"prn.txt",
|
||||
"AUX.md",
|
||||
"nul .txt",
|
||||
"CONIN$",
|
||||
"conout$.log",
|
||||
"com1.txt",
|
||||
"LPT9",
|
||||
"COM¹.txt",
|
||||
"lpt²/file.md",
|
||||
"COM³.txt",
|
||||
] {
|
||||
assert!(
|
||||
validate_file_manifest(&manifest(&[path])).is_err(),
|
||||
"accepted reserved path: {path:?}"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn rejects_path_conflicts_in_either_entry_order() {
|
||||
for [first, second] in [
|
||||
["notes.md", "notes.md"],
|
||||
["Notes.md", "notes.md"],
|
||||
["Straße.md", "STRASSE.md"],
|
||||
["ς.md", "σ.md"],
|
||||
["Notes/a.md", "notes/b.md"],
|
||||
["notes", "notes/a.md"],
|
||||
] {
|
||||
for paths in [[first, second], [second, first]] {
|
||||
assert!(
|
||||
validate_file_manifest(&manifest(&paths)).is_err(),
|
||||
"accepted conflicting paths: {paths:?}"
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
Loading…
Reference in a new issue