andras/reconcile
andras/reconcile
Watch
1
0
Fork
You've already forked reconcile
 0
Code Issues Pull requests 10 Projects Releases Packages Wiki Activity Actions

Move files

Browse source
This commit is contained in:
Andras Schmelczer 2025-06-15 11:08:28 +01:00
commit 1fd6450484
No known key found for this signature in database
GPG key ID: FC8F2C3D3D1A718C
1 changed files with 579 additions and 3359 deletions
Download patch file Download diff file Expand all files Collapse all files

7
backend/reconcile/src/tokenizer.rs
View file

@ -1,7 +0,0 @@
use token::Token;
pub mod token;
pub mod word_tokenizer;
/// A trait for tokenizers that take a string and return a list of tokens.
pub type Tokenizer<T> = dyn Fn(&str) -> Vec<Token<T>>;