Special Tokens

[thekid]

Support special tokens in encoding:

ENDOFTEXT = "<|endoftext|>"
FIM_PREFIX = "<|fim_prefix|>"
FIM_MIDDLE = "<|fim_middle|>"
FIM_SUFFIX = "<|fim_suffix|>"
ENDOFPROMPT = "<|endofprompt|>"

See https://github.com/openai/tiktoken/blob/main/tiktoken_ext/openai_public.py