Meaning

1 senses

  1. A system that parses an input stream into its component tokens.

Origin

From tokenize + -er.

From tokenize + -er.

Wiktionary, CC BY-SA 4.0 · source

Forms

plural: tokenizers
Singulartokenizer
Pluraltokenizers

regular

See also

neighbours in the dictionary
toingtokentokitolerabletolerance
tokenizer — meaning, examples, forms, synonyms | Krumeto