Why LLM Glitch Tokens Produce Gibberish

HN points to it as the accessible explanation of why glitch tokens exist whenever tokenizers or odd GPT behavior come up.

Rob Miles explains "glitch" tokens, odd strings in large language model vocabularies that make the model produce gibberish. He covers how they likely arise from tokenizer training data, such as Reddit usernames, that never appeared in the model's own training.