Text Tokenization Algorithm

From GM-RKB

(Redirected from tokenization algorithm)

Jump to navigation Jump to search

A Text Tokenization System is a text segmentation algorithm/string segmentation algorithm that can be applied by a text tokenization system (to solve a text tokenization task).

See: End-of-Sentence Detection Algorithm, Word Segmentation Algorithm.

Retrieved from "http://www.gabormelli.com/RKB/index.php?title=Text_Tokenization_Algorithm&oldid=783565"

Concept