ChatGPT Can Be Broken by Entering These Strange Words, And Nobody Is Sure Why -- Jerz's Literacy Weblog (est. 1999)

Reddit usernames like ‘SolidGoldMagikarp’ are somehow causing the chatbot to give bizarre responses.
[…]
“I’ve just found out that several of the anomalous GPT tokens (“TheNitromeFan”, ” SolidGoldMagikarp”, ” davidjl”, ” Smartstocks”, ” RandomRedditorWithNo”, ) are handles of people who are (competitively? collaboratively?) counting to infinity on a Reddit forum. I kid you not,” Watkins tweeted Wednesday morning. These users subscribe to the subreddit, r/counting, in which users have reached nearly 5,000,000 after almost a decade of counting one post at a time.
“There’s a hall of fame of the people who’ve contributed the most to the counting effort, and six of the tokens are people who are in the top ten last time I checked the listing. So presumably, they were the people who’ve done the most counting,” Watkins told Motherboard. “They were part of this bizarre Reddit community trying to count to infinity and they accidentally counted themselves into a kind of immortality.”