andras/reconcile
andras/reconcile
Watch
1
0
Fork
You've already forked reconcile
 0
Code Issues Pull requests 10 Projects Releases Packages Wiki Activity Actions

Expose word_tokenizer

Browse source
This commit is contained in:
Andras Schmelczer 2025-06-14 11:44:06 +01:00
commit 1038f9cee0
No known key found for this signature in database
GPG key ID: FC8F2C3D3D1A718C
1 changed files with 6 additions and 5 deletions
Download patch file Download diff file Expand all files Collapse all files

4
backend/reconcile/src/tokenizer/token.rs
View file

@ -3,11 +3,11 @@ use serde::{Deserialize, Serialize};
/// A token is a string that has been normalised in some way.
///
/// It's UTF-8 compatible.
///
/// A token consists of the normalised form is used for comparison, and the
/// original form used for subsequently applying `Operation`-s to a text
/// document.
///
/// It's UTF-8 compatible.
#[cfg_attr(feature = "serde", derive(Serialize, Deserialize))]
#[derive(Debug, Clone)]
pub struct Token<T>