This commit is contained in:
parent
a2fbba682e
commit
b4259b767c
13 changed files with 608 additions and 24 deletions
38
README.md
38
README.md
|
|
@ -1,6 +1,6 @@
|
||||||
# `reconcile-text`: conflict-free 3-way text merging
|
# `reconcile-text`: conflict-free 3-way text merging
|
||||||
|
|
||||||
A Rust, TypeScript, and Python library for merging conflicting text edits without manual intervention. Unlike traditional 3-way merge tools that produce conflict markers, `reconcile-text` automatically resolves conflicts by applying both sets of changes (while updating cursor positions) using an algorithm inspired by Operational Transformation.
|
A Rust, TypeScript, and Python library for merging conflicting text edits without manual intervention. Unlike traditional 3-way merge tools that produce conflict markers, `reconcile-text` automatically resolves conflicts using an algorithm inspired by Operational Transformation. It combines edits where possible and prioritizes stable replay while updating cursor positions.
|
||||||
|
|
||||||
## Try it
|
## Try it
|
||||||
|
|
||||||
|
|
@ -132,12 +132,46 @@ Differential sync is implemented by [universal-sync][12], and it requires a merg
|
||||||
1. **Tokenisation** - Input texts are split into meaningful units (words, characters, etc.) for granular merging
|
1. **Tokenisation** - Input texts are split into meaningful units (words, characters, etc.) for granular merging
|
||||||
2. **Diff computation** - Myers' algorithm calculates differences between (parent ↔ left) and (parent ↔ right)
|
2. **Diff computation** - Myers' algorithm calculates differences between (parent ↔ left) and (parent ↔ right)
|
||||||
3. **Diff optimisation** - Operations are reordered and consolidated to maximise chained changes
|
3. **Diff optimisation** - Operations are reordered and consolidated to maximise chained changes
|
||||||
4. **Operational Transformation** - Edits are woven together using OT principles, preserving all modifications and updating cursors
|
4. **Operational Transformation** - Edits are woven together using OT principles, combining changes and updating cursors
|
||||||
|
5. **Replay verification** - Combined text is accepted only when it absorbs both inputs on replay; ambiguous cases keep one input deterministically
|
||||||
|
|
||||||
Whilst the primary goal of `reconcile-text` isn't to implement OT, it provides an elegant way to merge Myers' diff outputs. (For a dedicated Rust OT implementation, see [operational-transform-rs][13].) The same could be achieved with CRDTs, which many libraries implement well for text (see [Loro][14], [cola][15], and [automerge][16]).
|
Whilst the primary goal of `reconcile-text` isn't to implement OT, it provides an elegant way to merge Myers' diff outputs. (For a dedicated Rust OT implementation, see [operational-transform-rs][13].) The same could be achieved with CRDTs, which many libraries implement well for text (see [Loro][14], [cola][15], and [automerge][16]).
|
||||||
|
|
||||||
However, when only the end result of concurrent changes is observable, merge quality depends entirely on the quality of the underlying 2-way diffs. For instance, `move` operations cannot be supported because Myers' algorithm decomposes them into separate `insert` and `delete` operations, regardless of the merging algorithm used.
|
However, when only the end result of concurrent changes is observable, merge quality depends entirely on the quality of the underlying 2-way diffs. For instance, `move` operations cannot be supported because Myers' algorithm decomposes them into separate `insert` and `delete` operations, regardless of the merging algorithm used.
|
||||||
|
|
||||||
|
### Replay stability
|
||||||
|
|
||||||
|
Reapplying either unchanged input against the same base preserves the merged text:
|
||||||
|
|
||||||
|
```javascript
|
||||||
|
const merged = reconcile('a', 'b', 'x').text; // 'bx'
|
||||||
|
reconcile('a', merged, 'x').text; // 'bx'
|
||||||
|
reconcile('a', merged, 'b').text; // 'bx'
|
||||||
|
```
|
||||||
|
|
||||||
|
This holds for every built-in tokenizer and deterministic, lossless custom
|
||||||
|
tokenizers whose original token text concatenates back to the input.
|
||||||
|
The guarantee applies to text; cursors from both supplied inputs are mapped into
|
||||||
|
the result on each call.
|
||||||
|
|
||||||
|
The merge treats insertions already included in the other side's inserted text
|
||||||
|
as incorporated, including character subsequences inside joined words. It checks
|
||||||
|
that a tentative combined result absorbs both inputs. If it does not, the merge
|
||||||
|
keeps one input deterministically: more inserted characters take precedence,
|
||||||
|
then more deleted characters, then lexicographically greater text.
|
||||||
|
|
||||||
|
This gives retry stability precedence over preserving every concurrent edit.
|
||||||
|
For example, changing `a` to `abc` and `abcd` now produces `abcd`. Plain strings
|
||||||
|
cannot distinguish a replay from every independent edit containing the same
|
||||||
|
text, so some concurrent edits can be superseded. The fallback selects an input
|
||||||
|
without iterating until convergence; concurrent merges perform at most two
|
||||||
|
additional candidate merges to verify replay.
|
||||||
|
|
||||||
|
In Rust, use `reconcile` or `EditedText::merge_with_tokenizer` for this guarantee.
|
||||||
|
The lower-level `EditedText::merge` combines operations without the snapshot
|
||||||
|
replay check. Continue advancing the sync base after a successful application;
|
||||||
|
this guarantee covers replay of the unchanged merge result.
|
||||||
|
|
||||||
## Comparison with other approaches
|
## Comparison with other approaches
|
||||||
|
|
||||||
### Traditional 3-way merge (diff3, Git)
|
### Traditional 3-way merge (diff3, Git)
|
||||||
|
|
|
||||||
|
|
@ -143,6 +143,9 @@ export interface ReconcileApi {
|
||||||
* This is the primary function for 3-way text merging. Unlike traditional merge tools
|
* This is the primary function for 3-way text merging. Unlike traditional merge tools
|
||||||
* that produce conflict markers, this function automatically resolves conflicts by
|
* that produce conflict markers, this function automatically resolves conflicts by
|
||||||
* applying both sets of changes where possible.
|
* applying both sets of changes where possible.
|
||||||
|
* Replaying either unchanged input against the same base preserves the merged
|
||||||
|
* text. Replay stability can supersede ambiguous concurrent edits; the guarantee
|
||||||
|
* applies to text, not the combined cursor list.
|
||||||
*
|
*
|
||||||
* @param original - The original/base version of the text that both sides diverged from
|
* @param original - The original/base version of the text that both sides diverged from
|
||||||
* @param left - The left version of the text (either string or TextWithCursors with cursor positions)
|
* @param left - The left version of the text (either string or TextWithCursors with cursor positions)
|
||||||
|
|
|
||||||
|
|
@ -73,6 +73,40 @@ describe.each(backends)('reconcile [$name]', ({ api }) => {
|
||||||
expect(result.history.length).toBeGreaterThan(0);
|
expect(result.history.length).toBeGreaterThan(0);
|
||||||
});
|
});
|
||||||
|
|
||||||
|
it.each(['a', ''])(
|
||||||
|
'replaying conflicting replacements with base %p keeps both edits',
|
||||||
|
(base) => {
|
||||||
|
for (const tokenizer of ['Character', 'Line', 'Markdown', 'Word'] as const) {
|
||||||
|
const merged = reconcile(base, 'b', 'x', tokenizer);
|
||||||
|
expect(merged.text).toBe('bx');
|
||||||
|
expect(reconcile(base, merged, 'x', tokenizer)).toEqual(merged);
|
||||||
|
expect(reconcile(base, merged, 'b', tokenizer)).toEqual(merged);
|
||||||
|
expect(reconcileWithHistory(base, merged, 'x', tokenizer).text).toBe('bx');
|
||||||
|
}
|
||||||
|
}
|
||||||
|
);
|
||||||
|
|
||||||
|
it('preserves intentional repeats and Unicode cursor positions on replay', () => {
|
||||||
|
const repeated = reconcile('a', 'bb', 'x');
|
||||||
|
expect(repeated.text).toBe('bbx');
|
||||||
|
expect(reconcile('a', repeated, 'x')).toEqual(repeated);
|
||||||
|
const local = {
|
||||||
|
text: '🦊x',
|
||||||
|
cursors: [
|
||||||
|
{ id: 1, position: 1 },
|
||||||
|
{ id: 2, position: 2 },
|
||||||
|
],
|
||||||
|
};
|
||||||
|
const result = reconcile('a', local, {
|
||||||
|
text: 'x',
|
||||||
|
cursors: [{ id: 3, position: 1 }],
|
||||||
|
});
|
||||||
|
expect(result.text).toBe(local.text);
|
||||||
|
expect(result.cursors).toEqual(
|
||||||
|
expect.arrayContaining([...local.cursors, { id: 3, position: 2 }])
|
||||||
|
);
|
||||||
|
});
|
||||||
|
|
||||||
it('undiff accepts bigint entries (per the Array<number | bigint | string> type)', () => {
|
it('undiff accepts bigint entries (per the Array<number | bigint | string> type)', () => {
|
||||||
const original = 'Hello world';
|
const original = 'Hello world';
|
||||||
const changed = 'Hello cruel world';
|
const changed = 'Hello cruel world';
|
||||||
|
|
|
||||||
|
|
@ -27,6 +27,19 @@ class TestReconcile:
|
||||||
result = reconcile(parent, left, right)
|
result = reconcile(parent, left, right)
|
||||||
assert result["text"] == "With reconcile, merging documents is easy!"
|
assert result["text"] == "With reconcile, merging documents is easy!"
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("parent", ["", "a"])
|
||||||
|
def test_replaying_conflicting_replacements(self, parent: str) -> None:
|
||||||
|
merged = reconcile(parent, "b", "x")
|
||||||
|
assert merged["text"] == "bx"
|
||||||
|
assert reconcile(parent, merged, "x") == merged
|
||||||
|
assert reconcile(parent, merged, "b") == merged
|
||||||
|
assert reconcile_with_history(parent, merged, "x")["text"] == "bx"
|
||||||
|
|
||||||
|
def test_replaying_preserves_repeated_text(self) -> None:
|
||||||
|
merged = reconcile("a", "bb", "x")
|
||||||
|
assert merged["text"] == "bbx"
|
||||||
|
assert reconcile("a", merged, "x") == merged
|
||||||
|
|
||||||
def test_with_cursors(self) -> None:
|
def test_with_cursors(self) -> None:
|
||||||
result = reconcile(
|
result = reconcile(
|
||||||
"Hello",
|
"Hello",
|
||||||
|
|
|
||||||
|
|
@ -2,8 +2,9 @@
|
||||||
//!
|
//!
|
||||||
//! A library for merging conflicting text edits without manual intervention.
|
//! A library for merging conflicting text edits without manual intervention.
|
||||||
//! Unlike traditional 3-way merge tools that produce conflict markers,
|
//! Unlike traditional 3-way merge tools that produce conflict markers,
|
||||||
//! reconcile-text automatically resolves conflicts by applying both sets of
|
//! reconcile-text automatically resolves conflicts by combining edits where
|
||||||
//! changes (while updating cursor positions) using an algorithm inspired by
|
//! possible, prioritizing stable replay, and updating cursor positions with
|
||||||
|
//! an algorithm inspired by
|
||||||
//! Operational Transformation.
|
//! Operational Transformation.
|
||||||
//!
|
//!
|
||||||
//! ✨ **[Try the interactive demo](https://schmelczer.dev/reconcile)** to see it in action.
|
//! ✨ **[Try the interactive demo](https://schmelczer.dev/reconcile)** to see it in action.
|
||||||
|
|
|
||||||
|
|
@ -1,6 +1,7 @@
|
||||||
mod diff_error;
|
mod diff_error;
|
||||||
mod edited_text;
|
mod edited_text;
|
||||||
mod operation;
|
mod operation;
|
||||||
|
mod replay;
|
||||||
mod utils;
|
mod utils;
|
||||||
use std::fmt::Debug;
|
use std::fmt::Debug;
|
||||||
|
|
||||||
|
|
@ -10,22 +11,22 @@ pub use operation::Operation;
|
||||||
|
|
||||||
use crate::{Tokenizer, types::text_with_cursors::TextWithCursors};
|
use crate::{Tokenizer, types::text_with_cursors::TextWithCursors};
|
||||||
|
|
||||||
/// Given an `original` document and two concurrent edits to it,
|
/// Merge two concurrent snapshots, prioritizing stability when either input
|
||||||
/// return a document containing all changes from both `left`
|
/// is replayed against the same `original` document.
|
||||||
/// and `right`.
|
|
||||||
///
|
///
|
||||||
/// If both updated texts are identical, the resulting text is exactly that
|
/// If both updated texts are identical, the resulting text is exactly that
|
||||||
/// text, regardless of the original document.
|
/// text, regardless of the original document.
|
||||||
///
|
///
|
||||||
/// If a span has been inserted in either the `left` or `right`
|
/// With a deterministic, lossless tokenizer, if `merged = reconcile(original, left,
|
||||||
/// versions, it will be present in the return value. If both sides
|
/// right).apply()`, reconciling `merged` with either input again produces the
|
||||||
/// insert the same span with a common prefix, that prefix will only
|
/// same text. This guarantee concerns text, not the combined cursor list.
|
||||||
/// be present once in the output.
|
|
||||||
///
|
///
|
||||||
/// When both sides delete the same span, it will be deleted in the
|
/// Already incorporated insertions are recognized as character subsequences
|
||||||
/// return value. If one side deletes a span and the other side inserts
|
/// of the other side's inserted text, including inside joined words. A combined
|
||||||
/// into that span, the inserted text will be present in the return
|
/// result is kept only when it absorbs both inputs on replay. Otherwise one
|
||||||
/// value.
|
/// snapshot is selected, preferring more inserted characters, then more deleted
|
||||||
|
/// characters, then lexicographically greater text. Some independent concurrent
|
||||||
|
/// edits can therefore be superseded to keep retries stable.
|
||||||
///
|
///
|
||||||
/// Supports UTF-8. Arguments are tokenized using the provided `tokenizer`.
|
/// Supports UTF-8. Arguments are tokenized using the provided `tokenizer`.
|
||||||
///
|
///
|
||||||
|
|
@ -49,10 +50,7 @@ pub fn reconcile<'a, T>(
|
||||||
where
|
where
|
||||||
T: PartialEq + Clone + Debug,
|
T: PartialEq + Clone + Debug,
|
||||||
{
|
{
|
||||||
let left_operations = EditedText::from_strings_with_tokenizer(original, left, tokenizer);
|
replay::merge_stable(original, left, right, tokenizer)
|
||||||
let right_operations = EditedText::from_strings_with_tokenizer(original, right, tokenizer);
|
|
||||||
|
|
||||||
left_operations.merge(right_operations)
|
|
||||||
}
|
}
|
||||||
|
|
||||||
#[cfg(test)]
|
#[cfg(test)]
|
||||||
|
|
|
||||||
|
|
@ -95,10 +95,58 @@ where
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
pub(crate) fn inserted_text(&self) -> String {
|
||||||
|
let mut result = String::new();
|
||||||
|
for operation in &self.operations {
|
||||||
|
if let Operation::Insert { text, .. } = operation {
|
||||||
|
for token in text {
|
||||||
|
result.push_str(token.original());
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
result
|
||||||
|
}
|
||||||
|
|
||||||
|
pub(crate) fn edit_counts(&self) -> (usize, usize) {
|
||||||
|
self.operations
|
||||||
|
.iter()
|
||||||
|
.fold((0, 0), |(inserted, deleted), operation| match operation {
|
||||||
|
Operation::Insert { .. } => (inserted + operation.len(), deleted),
|
||||||
|
Operation::Delete {
|
||||||
|
deleted_character_count,
|
||||||
|
..
|
||||||
|
} => (inserted, deleted + deleted_character_count),
|
||||||
|
Operation::Equal { .. } => (inserted, deleted),
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
pub(crate) fn with_cursors_and_side(
|
||||||
|
mut self,
|
||||||
|
mut cursors: Vec<CursorPosition>,
|
||||||
|
side: Side,
|
||||||
|
) -> Self {
|
||||||
|
cursors.sort_by_key(|cursor| cursor.char_index);
|
||||||
|
self.cursors = cursors;
|
||||||
|
self.operation_sides.fill(side);
|
||||||
|
self
|
||||||
|
}
|
||||||
|
|
||||||
|
pub(crate) fn swap_sides(mut self) -> Self {
|
||||||
|
for side in &mut self.operation_sides {
|
||||||
|
*side = match side {
|
||||||
|
Side::Left => Side::Right,
|
||||||
|
Side::Right => Side::Left,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
self
|
||||||
|
}
|
||||||
|
|
||||||
/// Merge two `EditedText` instances. The two instances must be derived
|
/// Merge two `EditedText` instances. The two instances must be derived
|
||||||
/// from the same original text. The operations are merged using the
|
/// from the same original text. The operations are merged using the
|
||||||
/// principles of Operational Transformation. The cursors are updated
|
/// principles of Operational Transformation. The cursors are updated
|
||||||
/// accordingly to reflect the changes made by the merged operations.
|
/// accordingly to reflect the changes made by the merged operations.
|
||||||
|
/// This operation-level API does not guarantee snapshot replay stability;
|
||||||
|
/// use [`Self::merge_with_tokenizer`] or [`crate::reconcile`] for that contract.
|
||||||
///
|
///
|
||||||
/// # Panics
|
/// # Panics
|
||||||
///
|
///
|
||||||
|
|
@ -254,6 +302,23 @@ where
|
||||||
)
|
)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Merge snapshots with the same replay guarantees as [`crate::reconcile`].
|
||||||
|
/// The tokenizer must be deterministic, lossless, and match the tokenizer used
|
||||||
|
/// to create these edits. Operations are recalculated from their snapshots.
|
||||||
|
///
|
||||||
|
/// # Panics
|
||||||
|
///
|
||||||
|
/// Panics in debug builds if the original texts differ, or if a cursor index
|
||||||
|
/// overflows an `isize` during transformation.
|
||||||
|
#[must_use]
|
||||||
|
pub fn merge_with_tokenizer(&self, other: &Self, tokenizer: &Tokenizer<T>) -> Self {
|
||||||
|
debug_assert_eq!(
|
||||||
|
self.text, other.text,
|
||||||
|
"Edits must share the same original text"
|
||||||
|
);
|
||||||
|
super::reconcile(self.text, &self.apply(), &other.apply(), tokenizer)
|
||||||
|
}
|
||||||
|
|
||||||
/// Apply the operations to the text and return the resulting text
|
/// Apply the operations to the text and return the resulting text
|
||||||
#[must_use]
|
#[must_use]
|
||||||
pub fn apply(&self) -> TextWithCursors {
|
pub fn apply(&self) -> TextWithCursors {
|
||||||
|
|
|
||||||
172
src/operation_transformation/replay.rs
Normal file
172
src/operation_transformation/replay.rs
Normal file
|
|
@ -0,0 +1,172 @@
|
||||||
|
use std::fmt::Debug;
|
||||||
|
|
||||||
|
use super::EditedText;
|
||||||
|
use crate::{BuiltinTokenizer, Side, TextWithCursors, Tokenizer};
|
||||||
|
|
||||||
|
pub(super) fn merge_stable<'a, T>(
|
||||||
|
original: &'a str,
|
||||||
|
left: &TextWithCursors,
|
||||||
|
right: &TextWithCursors,
|
||||||
|
tokenizer: &Tokenizer<T>,
|
||||||
|
) -> EditedText<'a, T>
|
||||||
|
where
|
||||||
|
T: PartialEq + Clone + Debug,
|
||||||
|
{
|
||||||
|
// A combined result is accepted only if the candidate merge absorbs both
|
||||||
|
// inputs. On replay the candidate is then the same result, and its checks
|
||||||
|
// are unchanged or compare identical snapshots. If this fails, selecting
|
||||||
|
// one input is also stable: replay sees the original pair or identical inputs.
|
||||||
|
let candidate = merge_candidate(original, left, right, tokenizer);
|
||||||
|
if left.text_ref() == right.text_ref()
|
||||||
|
|| left.text_ref() == original
|
||||||
|
|| right.text_ref() == original
|
||||||
|
{
|
||||||
|
return candidate;
|
||||||
|
}
|
||||||
|
let merged = candidate.apply().text();
|
||||||
|
let without_cursors = merged.clone().into();
|
||||||
|
if [left, right].iter().all(|input| {
|
||||||
|
merge_candidate(
|
||||||
|
original,
|
||||||
|
&without_cursors,
|
||||||
|
&input.text_ref().into(),
|
||||||
|
tokenizer,
|
||||||
|
)
|
||||||
|
.apply()
|
||||||
|
.text()
|
||||||
|
== merged
|
||||||
|
}) {
|
||||||
|
return candidate;
|
||||||
|
}
|
||||||
|
|
||||||
|
let left_operations = EditedText::from_strings_with_tokenizer(original, left, tokenizer);
|
||||||
|
let right_operations = EditedText::from_strings_with_tokenizer(original, right, tokenizer);
|
||||||
|
if version_priority(&left_operations, left) >= version_priority(&right_operations, right) {
|
||||||
|
keep_version(left_operations, right_operations, left, right, Side::Left)
|
||||||
|
} else {
|
||||||
|
keep_version(right_operations, left_operations, right, left, Side::Right)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn is_subsequence(part: &str, whole: &str) -> bool {
|
||||||
|
let mut chars = whole.chars();
|
||||||
|
part.chars()
|
||||||
|
.all(|ch| chars.by_ref().any(|candidate| candidate == ch))
|
||||||
|
}
|
||||||
|
|
||||||
|
fn version_priority<'b, T>(
|
||||||
|
operations: &EditedText<'_, T>,
|
||||||
|
input: &'b TextWithCursors,
|
||||||
|
) -> (usize, usize, &'b str)
|
||||||
|
where
|
||||||
|
T: PartialEq + Clone + Debug,
|
||||||
|
{
|
||||||
|
let (inserted, deleted) = operations.edit_counts();
|
||||||
|
(inserted, deleted, input.text_ref())
|
||||||
|
}
|
||||||
|
|
||||||
|
fn merge_candidate<'a, T>(
|
||||||
|
original: &'a str,
|
||||||
|
left: &TextWithCursors,
|
||||||
|
right: &TextWithCursors,
|
||||||
|
tokenizer: &Tokenizer<T>,
|
||||||
|
) -> EditedText<'a, T>
|
||||||
|
where
|
||||||
|
T: PartialEq + Clone + Debug,
|
||||||
|
{
|
||||||
|
let left_operations = EditedText::from_strings_with_tokenizer(original, left, tokenizer);
|
||||||
|
let right_operations = EditedText::from_strings_with_tokenizer(original, right, tokenizer);
|
||||||
|
let left_text = left.text_ref();
|
||||||
|
let right_text = right.text_ref();
|
||||||
|
if left_text == right_text || left_text == original || right_text == original {
|
||||||
|
return merge_ordered(left_operations, right_operations, left_text, right_text);
|
||||||
|
}
|
||||||
|
|
||||||
|
let left_inserted = left_operations.inserted_text();
|
||||||
|
let right_inserted = right_operations.inserted_text();
|
||||||
|
// Treat insertions already present in the other side's edits as incorporated,
|
||||||
|
// including inside joined words or across separate inserted spans. This
|
||||||
|
// deliberately gives replay precedence over ambiguous independent edits.
|
||||||
|
let left_contains_right =
|
||||||
|
!right_inserted.is_empty() && is_subsequence(&right_inserted, &left_inserted);
|
||||||
|
let right_contains_left =
|
||||||
|
!left_inserted.is_empty() && is_subsequence(&left_inserted, &right_inserted);
|
||||||
|
match (left_contains_right, right_contains_left) {
|
||||||
|
(true, false) => keep_version(left_operations, right_operations, left, right, Side::Left),
|
||||||
|
(false, true) => keep_version(right_operations, left_operations, right, left, Side::Right),
|
||||||
|
(true, true) => {
|
||||||
|
if version_priority(&left_operations, left)
|
||||||
|
>= version_priority(&right_operations, right)
|
||||||
|
{
|
||||||
|
keep_version(left_operations, right_operations, left, right, Side::Left)
|
||||||
|
} else {
|
||||||
|
keep_version(right_operations, left_operations, right, left, Side::Right)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
(false, false) => merge_ordered(left_operations, right_operations, left_text, right_text),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn merge_ordered<'a, T>(
|
||||||
|
left: EditedText<'a, T>,
|
||||||
|
right: EditedText<'a, T>,
|
||||||
|
left_text: &str,
|
||||||
|
right_text: &str,
|
||||||
|
) -> EditedText<'a, T>
|
||||||
|
where
|
||||||
|
T: PartialEq + Clone + Debug,
|
||||||
|
{
|
||||||
|
// Canonical input order also stabilizes tokenizers with normalized or
|
||||||
|
// non-joinable tokens. Restore provenance to the caller's original sides.
|
||||||
|
if left_text <= right_text {
|
||||||
|
left.merge(right)
|
||||||
|
} else {
|
||||||
|
right.merge(left).swap_sides()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn keep_version<'a, T>(
|
||||||
|
operations: EditedText<'a, T>,
|
||||||
|
other_operations: EditedText<'a, T>,
|
||||||
|
kept: &TextWithCursors,
|
||||||
|
other: &TextWithCursors,
|
||||||
|
side: Side,
|
||||||
|
) -> EditedText<'a, T>
|
||||||
|
where
|
||||||
|
T: PartialEq + Clone + Debug,
|
||||||
|
{
|
||||||
|
let kept_text = kept.text_ref();
|
||||||
|
let traditional = merge_ordered(
|
||||||
|
operations.clone(),
|
||||||
|
other_operations,
|
||||||
|
kept_text,
|
||||||
|
other.text_ref(),
|
||||||
|
);
|
||||||
|
// Preserve the existing cursor mapping and provenance when the operation
|
||||||
|
// merge already produces the selected text. Character projection is needed
|
||||||
|
// only when replay suppression actually changes the operation result.
|
||||||
|
if traditional.apply().text_ref() == kept_text {
|
||||||
|
return match side {
|
||||||
|
Side::Left => traditional,
|
||||||
|
Side::Right => traditional.swap_sides(),
|
||||||
|
};
|
||||||
|
}
|
||||||
|
let mut cursors = kept.cursors();
|
||||||
|
if !other.cursors().is_empty() {
|
||||||
|
let other_text = other.text_ref();
|
||||||
|
// Character alignment maps cursors inside joined words and uses the
|
||||||
|
// same character offsets as the public cursor API, including Unicode.
|
||||||
|
let original = EditedText::from_strings_with_tokenizer(
|
||||||
|
other_text,
|
||||||
|
other,
|
||||||
|
&*BuiltinTokenizer::Character,
|
||||||
|
);
|
||||||
|
let updated = EditedText::from_strings_with_tokenizer(
|
||||||
|
other_text,
|
||||||
|
&kept_text.into(),
|
||||||
|
&*BuiltinTokenizer::Character,
|
||||||
|
);
|
||||||
|
cursors.extend(original.merge(updated).apply().cursors());
|
||||||
|
}
|
||||||
|
operations.with_cursors_and_side(cursors, side)
|
||||||
|
}
|
||||||
|
|
@ -21,4 +21,4 @@ expected: a b c d
|
||||||
parent: a
|
parent: a
|
||||||
left: abc
|
left: abc
|
||||||
right: abcd
|
right: abcd
|
||||||
expected: abcabcd
|
expected: abcd
|
||||||
|
|
|
||||||
|
|
@ -16,4 +16,4 @@ expected: original_1 edit_1| original_3|
|
||||||
parent: original_1 original_2 original_3
|
parent: original_1 original_2 original_3
|
||||||
left: original_1 edit_1| original_3
|
left: original_1 edit_1| original_3
|
||||||
right: original_1 conflicting_edit_1| original_3
|
right: original_1 conflicting_edit_1| original_3
|
||||||
expected: original_1 conflicting_edit_1| edit_1| original_3
|
expected: original_1 conflicting_edit_1|| original_3
|
||||||
|
|
|
||||||
|
|
@ -85,7 +85,7 @@ expected: Please submit your |completed |assignment |online |by Friday
|
||||||
parent: "a b "
|
parent: "a b "
|
||||||
left: "c d "
|
left: "c d "
|
||||||
right: "a b c d "
|
right: "a b c d "
|
||||||
expected: "c d c d "
|
expected: "a b c d "
|
||||||
|
|
||||||
---
|
---
|
||||||
parent: a b c d e
|
parent: a b c d e
|
||||||
|
|
|
||||||
|
|
@ -1,4 +1,6 @@
|
||||||
use reconcile_text::{BuiltinTokenizer, Token, reconcile};
|
use reconcile_text::{
|
||||||
|
BuiltinTokenizer, CursorPosition, EditedText, TextWithCursors, Token, reconcile,
|
||||||
|
};
|
||||||
use test_case::{test_case, test_matrix};
|
use test_case::{test_case, test_matrix};
|
||||||
|
|
||||||
#[test_matrix([
|
#[test_matrix([
|
||||||
|
|
@ -79,3 +81,265 @@ fn identical_edits_preserve_original_spelling(parent: &str, updated: &str) {
|
||||||
updated,
|
updated,
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[test_matrix([
|
||||||
|
BuiltinTokenizer::Character,
|
||||||
|
BuiltinTokenizer::Line,
|
||||||
|
BuiltinTokenizer::Markdown,
|
||||||
|
BuiltinTokenizer::Word
|
||||||
|
])]
|
||||||
|
fn replay_short_edits_preserves_merged_text(tokenizer: BuiltinTokenizer) {
|
||||||
|
let mut texts = vec![String::new()];
|
||||||
|
let mut level = texts.clone();
|
||||||
|
for _ in 0..2 {
|
||||||
|
level = level
|
||||||
|
.iter()
|
||||||
|
.flat_map(|text| ['a', 'b', ' ', '\n'].map(|ch| format!("{text}{ch}")))
|
||||||
|
.collect();
|
||||||
|
texts.extend(level.iter().cloned());
|
||||||
|
}
|
||||||
|
|
||||||
|
for parent in &texts {
|
||||||
|
for left in &texts {
|
||||||
|
for right in &texts {
|
||||||
|
let merged = reconcile(parent, &left.into(), &right.into(), &*tokenizer).apply();
|
||||||
|
assert_eq!(
|
||||||
|
reconcile(parent, &right.into(), &left.into(), &*tokenizer)
|
||||||
|
.apply()
|
||||||
|
.text(),
|
||||||
|
merged.text(),
|
||||||
|
"swapped inputs: tokenizer={tokenizer:?}, parent={parent:?}, left={left:?}, right={right:?}",
|
||||||
|
);
|
||||||
|
for replayed in [left, right] {
|
||||||
|
let actual = reconcile(parent, &merged, &replayed.into(), &*tokenizer).apply();
|
||||||
|
assert_eq!(
|
||||||
|
actual.text(),
|
||||||
|
merged.text(),
|
||||||
|
"tokenizer={tokenizer:?}, parent={parent:?}, left={left:?}, right={right:?}, replayed={replayed:?}"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test_matrix(["", "a"], [
|
||||||
|
BuiltinTokenizer::Character,
|
||||||
|
BuiltinTokenizer::Line,
|
||||||
|
BuiltinTokenizer::Markdown,
|
||||||
|
BuiltinTokenizer::Word
|
||||||
|
])]
|
||||||
|
fn replay_conflicting_replacements_keeps_both_edits(parent: &str, tokenizer: BuiltinTokenizer) {
|
||||||
|
let left = TextWithCursors::new("b".to_owned(), vec![CursorPosition::new(1, 1)]);
|
||||||
|
let right = TextWithCursors::new("x".to_owned(), vec![CursorPosition::new(2, 1)]);
|
||||||
|
let merged = reconcile(parent, &left, &right, &*tokenizer).apply();
|
||||||
|
assert_eq!(merged.text(), "bx");
|
||||||
|
assert_eq!(
|
||||||
|
merged.cursors(),
|
||||||
|
vec![CursorPosition::new(1, 1), CursorPosition::new(2, 2)]
|
||||||
|
);
|
||||||
|
for input in ["b", "x"] {
|
||||||
|
let replayed = reconcile(parent, &merged, &input.into(), &*tokenizer).apply();
|
||||||
|
assert_eq!(replayed, merged);
|
||||||
|
}
|
||||||
|
let left_operations = EditedText::from_strings_with_tokenizer(parent, &left, &*tokenizer);
|
||||||
|
let right_operations = EditedText::from_strings_with_tokenizer(parent, &right, &*tokenizer);
|
||||||
|
assert_eq!(
|
||||||
|
left_operations
|
||||||
|
.merge_with_tokenizer(&right_operations, &*tokenizer)
|
||||||
|
.apply(),
|
||||||
|
merged
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test_matrix([
|
||||||
|
BuiltinTokenizer::Character,
|
||||||
|
BuiltinTokenizer::Line,
|
||||||
|
BuiltinTokenizer::Markdown,
|
||||||
|
BuiltinTokenizer::Word
|
||||||
|
])]
|
||||||
|
fn replay_preserves_repeated_text_and_unicode_cursors(tokenizer: BuiltinTokenizer) {
|
||||||
|
let merged = reconcile("a", &"bb".into(), &"x".into(), &*tokenizer).apply();
|
||||||
|
assert_eq!(merged.text(), "bbx");
|
||||||
|
assert_eq!(
|
||||||
|
reconcile("a", &merged, &"x".into(), &*tokenizer).apply(),
|
||||||
|
merged
|
||||||
|
);
|
||||||
|
|
||||||
|
let left = TextWithCursors::new(
|
||||||
|
"🦊x".to_owned(),
|
||||||
|
vec![CursorPosition::new(1, 1), CursorPosition::new(2, 2)],
|
||||||
|
);
|
||||||
|
let right = TextWithCursors::new("x".to_owned(), vec![CursorPosition::new(3, 1)]);
|
||||||
|
let edited = reconcile("a", &left, &right, &*tokenizer);
|
||||||
|
let result = edited.apply();
|
||||||
|
assert_eq!(result.text(), "🦊x");
|
||||||
|
for (id, position) in [(1, 1), (2, 2), (3, 2)] {
|
||||||
|
assert!(
|
||||||
|
result
|
||||||
|
.cursors()
|
||||||
|
.contains(&CursorPosition::new(id, position))
|
||||||
|
);
|
||||||
|
}
|
||||||
|
let history_text: String = edited
|
||||||
|
.apply_with_history()
|
||||||
|
.iter()
|
||||||
|
.filter(|span| {
|
||||||
|
!matches!(
|
||||||
|
span.history(),
|
||||||
|
reconcile_text::History::RemovedFromLeft
|
||||||
|
| reconcile_text::History::RemovedFromRight
|
||||||
|
)
|
||||||
|
})
|
||||||
|
.map(reconcile_text::SpanWithHistory::text)
|
||||||
|
.collect();
|
||||||
|
assert_eq!(history_text, result.text());
|
||||||
|
assert_eq!(edited.apply_with_all().0, result);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test_matrix([
|
||||||
|
BuiltinTokenizer::Character,
|
||||||
|
BuiltinTokenizer::Line,
|
||||||
|
BuiltinTokenizer::Markdown,
|
||||||
|
BuiltinTokenizer::Word
|
||||||
|
])]
|
||||||
|
fn replay_longer_mixed_edits_is_stable(tokenizer: BuiltinTokenizer) {
|
||||||
|
let mut seed = 0x12ab_34cd_56ef_7890u64;
|
||||||
|
let mut next = || {
|
||||||
|
seed ^= seed << 13;
|
||||||
|
seed ^= seed >> 7;
|
||||||
|
seed ^= seed << 17;
|
||||||
|
seed
|
||||||
|
};
|
||||||
|
let alphabet = ['a', 'b', 'c', ' ', '\n', '\r', 'é', '🦊'];
|
||||||
|
for case in 0..2000 {
|
||||||
|
let mut text = || {
|
||||||
|
(0..next() % 16)
|
||||||
|
.map(|_| alphabet[usize::try_from(next() % 8).unwrap()])
|
||||||
|
.collect::<String>()
|
||||||
|
};
|
||||||
|
let parent = text();
|
||||||
|
let left = text();
|
||||||
|
let right = text();
|
||||||
|
let merged = reconcile(
|
||||||
|
&parent,
|
||||||
|
&left.clone().into(),
|
||||||
|
&right.clone().into(),
|
||||||
|
&*tokenizer,
|
||||||
|
)
|
||||||
|
.apply();
|
||||||
|
if case < 200 {
|
||||||
|
let left_with_cursors = TextWithCursors::new(
|
||||||
|
left.clone(),
|
||||||
|
(0..=left.chars().count())
|
||||||
|
.map(|position| CursorPosition::new(position, position))
|
||||||
|
.collect(),
|
||||||
|
);
|
||||||
|
let right_with_cursors = TextWithCursors::new(
|
||||||
|
right.clone(),
|
||||||
|
(0..=right.chars().count())
|
||||||
|
.map(|position| CursorPosition::new(100 + position, position))
|
||||||
|
.collect(),
|
||||||
|
);
|
||||||
|
let result = reconcile(
|
||||||
|
&parent,
|
||||||
|
&left_with_cursors,
|
||||||
|
&right_with_cursors,
|
||||||
|
&*tokenizer,
|
||||||
|
)
|
||||||
|
.apply();
|
||||||
|
assert_eq!(result.text(), merged.text());
|
||||||
|
assert_eq!(
|
||||||
|
result.cursors().len(),
|
||||||
|
left_with_cursors.cursors().len() + right_with_cursors.cursors().len()
|
||||||
|
);
|
||||||
|
assert!(
|
||||||
|
result
|
||||||
|
.cursors()
|
||||||
|
.iter()
|
||||||
|
.all(|cursor| cursor.char_index() <= result.text().chars().count())
|
||||||
|
);
|
||||||
|
for cursor in left_with_cursors
|
||||||
|
.cursors()
|
||||||
|
.into_iter()
|
||||||
|
.chain(right_with_cursors.cursors())
|
||||||
|
{
|
||||||
|
assert!(
|
||||||
|
result
|
||||||
|
.cursors()
|
||||||
|
.iter()
|
||||||
|
.any(|updated| updated.id() == cursor.id())
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
for input in [&left, &right] {
|
||||||
|
assert_eq!(
|
||||||
|
reconcile(&parent, &merged, &input.into(), &*tokenizer)
|
||||||
|
.apply()
|
||||||
|
.text(),
|
||||||
|
merged.text(),
|
||||||
|
"tokenizer={tokenizer:?}, parent={parent:?}, left={left:?}, right={right:?}, input={input:?}"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
assert_eq!(
|
||||||
|
reconcile(&parent, &right.into(), &left.into(), &*tokenizer)
|
||||||
|
.apply()
|
||||||
|
.text(),
|
||||||
|
merged.text()
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn replay_with_normalized_tokens_preserves_spelling() {
|
||||||
|
let tokenizer = |text: &str| {
|
||||||
|
text.chars()
|
||||||
|
.map(|ch| Token::new(ch.to_ascii_lowercase(), ch.to_string(), false, false))
|
||||||
|
.collect()
|
||||||
|
};
|
||||||
|
let texts = ["", "A", "a", "AB", "ba", "Aa", "a A", " A", "\nA"];
|
||||||
|
for parent in texts {
|
||||||
|
for left in texts {
|
||||||
|
for right in texts {
|
||||||
|
let merged = reconcile(parent, &left.into(), &right.into(), &tokenizer).apply();
|
||||||
|
for input in [left, right] {
|
||||||
|
assert_eq!(
|
||||||
|
reconcile(parent, &merged, &input.into(), &tokenizer)
|
||||||
|
.apply()
|
||||||
|
.text(),
|
||||||
|
merged.text()
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn ambiguous_ancestral_text_prefers_a_stable_input() {
|
||||||
|
let parent = "a b ";
|
||||||
|
let left = "a b a b ".into();
|
||||||
|
let right = "c d ".into();
|
||||||
|
let tokenizer = &*BuiltinTokenizer::Word;
|
||||||
|
let result = reconcile(parent, &left, &right, tokenizer);
|
||||||
|
let merged = result.apply();
|
||||||
|
assert_eq!(merged.text(), "a b a b ");
|
||||||
|
for input in [&left, &right] {
|
||||||
|
assert_eq!(
|
||||||
|
reconcile(parent, &merged, input, tokenizer).apply().text(),
|
||||||
|
merged.text()
|
||||||
|
);
|
||||||
|
}
|
||||||
|
assert!(
|
||||||
|
result
|
||||||
|
.apply_with_history()
|
||||||
|
.iter()
|
||||||
|
.any(|span| span.history() == reconcile_text::History::AddedFromLeft)
|
||||||
|
);
|
||||||
|
assert!(
|
||||||
|
!result
|
||||||
|
.apply_with_history()
|
||||||
|
.iter()
|
||||||
|
.any(|span| span.history() == reconcile_text::History::AddedFromRight)
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
|
||||||
|
|
@ -65,7 +65,7 @@ fn test_document_one_way_with_serialisation() {
|
||||||
|
|
||||||
doc.assert_eq_without_cursors(
|
doc.assert_eq_without_cursors(
|
||||||
&restored_left_operations
|
&restored_left_operations
|
||||||
.merge(restored_right_operations)
|
.merge_with_tokenizer(&restored_right_operations, &*BuiltinTokenizer::Word)
|
||||||
.apply()
|
.apply()
|
||||||
.text(),
|
.text(),
|
||||||
);
|
);
|
||||||
|
|
|
||||||
Loading…
Reference in a new issue