Skip to content

verb · transitive · rare

tokenize

tokenize · tokenizes · tokenized · tokenizing

To reduce to a token or set of tokens by lexical analysisTo reduce to a token or set of tokens by lexical analysis.

US/ˈtoʊ.kən.aɪz/

Meaning

3 senses

(transitive)To reduce to a token or set of tokens by lexical analysis.
(transitive)To substitute sensitive data with meaningless placeholders.
(transitive)To treat as a token minority.

Forms

tokenizes, tokenizing, tokenized, tokenized

Base form	tokenize · to tokenize
3rd person singular	tokenizes
Present participle	tokenizing
Past simple	tokenized
Past participle	tokenized

regular · Lesson

Present simpleLesson

I, you, we, they	tokenize
he, she, it	tokenizes

Past simpleLesson

all persons

tokenized

Present perfectLesson

I, you, we, they	have tokenized
he, she, it	has tokenized

Future simple (will)Lesson

all persons

will tokenize

Word family

8 related words

Relatedtokenism

Deriveddetokenize mistokenize retokenize tokenizable tokenization tokenizer untokenized

Lessons for this word

3 lessons

A1Present simple — Because it is a verb
A1Past simple, regular verbs — Because of its past form
A2Present perfect: introduction — Because of its past participle

See also

neighbours in the dictionary

toing token toki tolerable tolerance

tokenize — meaning, examples, forms, synonyms | Krumeto