andras/reconcile
andras/reconcile
Watch
1
0
Fork
You've already forked reconcile
 0
Code Issues Pull requests 10 Projects Releases Packages Wiki Activity Actions

Expose word_tokenizer

Browse source
This commit is contained in:
Andras Schmelczer 2025-06-14 11:44:06 +01:00
commit 1038f9cee0
No known key found for this signature in database
GPG key ID: FC8F2C3D3D1A718C
1 changed files with 6 additions and 5 deletions
Download patch file Download diff file Expand all files Collapse all files

1
backend/reconcile/src/tokenizer.rs
View file

@ -3,4 +3,5 @@ use token::Token;
pub mod token;
pub mod word_tokenizer;
/// A trait for tokenizers that take a string and return a list of tokens.
pub type Tokenizer<T> = dyn Fn(&str) -> Vec<Token<T>>;