tiktoken System

From GM-RKB

(Redirected from tiktoken)

Jump to navigation Jump to search

A tiktoken System is a text tokenization system.

See: SentencePiece.

References

https://github.com/openai/tiktoken

Retrieved from "http://www.gabormelli.com/RKB/index.php?title=tiktoken_System&oldid=827441"

Concept