The AI That Writes One Character at a Time
Contents › Chapter 3 Pulling the lever
Always picking the top one
Picking only the top character tends to repeat the same words
View in 3DAlways picking the top one
Why not just always pick the top character? We tried it. Without spinning the reel, we picked the character in the widest slot every time. Look at the text in the box below. The same words keep repeating.
Text written by the AI: Once upon a time, and the stopped and she was a straw an
Why?If you only pick the top one, you keep drifting toward the most common words and easily go around in circles. Small AIs do this even more. That is why we spin the reel on purpose, so the second or third character sometimes gets picked too.
Learn moreAlways picking the top choice is called greedy decoding. For math questions with only one right answer, it can actually work better.
View this scene in 3D →More in the series
A small AI (a character-level language model with 62,652 dials) was really trained on 40 fairy tales and fables retold for this project (about 376,021 characters), and every number here was calculated by that AI. It is a tiny model for learning; real chatbots are far bigger.