Read the grid
Wall, floor, start, and goal become cell tokens with row and column positions.
Education Draft
Transformers
From a maze to a sequence of moves.
A Transformer learns patterns in sequences. Those sequences can represent words, pixels, or a route through a maze. These small experiments are trained from scratch so the inputs, predictions, and mistakes are easier to inspect.
01 / Maze → moves
One token. One move.
The encoder reads the complete maze. The decoder uses that representation and its previous tokens to predict UP, DOWN, LEFT, RIGHT, or END. Step through a recorded run and watch the route take shape.
MODEL OUTPUT / RECORDED REPLAY
Next-token probabilities
Ready
Green: start · Gold: goal
Blue: generated route
0 tokens
BOS
Recorded inference from the local 11×11 checkpoint (7,000 updates). The browser replays model outputs; it does not run a Transformer. Playback starts paused.
See the original Python viewer
Seed 1000003: 31 moves followed by END. This is one successful example.
Predict a move
Attention connects the maze representation with the moves generated so far.
Continue to END
Each prediction becomes input for the next step. Mistakes can lead into walls or end a route early.
Breadth-first search supplies training targets. During inference, the model chooses moves without search fallback or wall masking. Larger-maze generalization is not established.
Run the live Python demo & read the training notes ↗02 / Screen history → paddle moves
Pong, ten moves ahead.
Four recent screen matrices go in. Ten paddle actions come out. An encoder reads the numeric history; an autoregressive decoder predicts a queue of STAY, UP, and DOWN tokens. Each player executes all ten moves before asking the model for another sequence.
Numbered circles mark pending paddle-center positions; overlapping predictions show only the earliest visible move. The ball’s fading trail shows past motion. This is an edited highlight reel, not a continuous match or performance evaluation.
Read four frames
Each 24×32 matrix marks empty cells, walls, paddles, and the ball. The right player sees a mirrored view.
Generate ten actions
The encoder runs once per refill. The decoder generates the complete action sequence before any of those moves execute.
Finish the queue
The player runs all ten moves without corrections, then reads the latest history. Scoring a point does not clear pending moves.
Trained from scratch by imitating scripted teachers. During play, the learned players receive no coordinates, velocities, or expert fallback. This is imitation learning; the recorded match is not reinforcement-learning self-play.
Run Pong & explore the model and training notes ↗03 / Board history → a ten-move plan
Snake: close calls and escapes.
An encoder-decoder Transformer reads two recent 20×20 board matrices and generates ten moves: FORWARD, LEFT, or RIGHT, relative to the snake’s heading. It commits to the whole queue before reading the board again—even after eating food.
Read the board history
Two numeric boards encode walls, body, food, and the head’s direction. The model does not receive future food locations.
Rank allowed moves
A geometry mask and short search restrict candidate moves during generation. The Transformer ranks the allowed actions to form a ten-move queue.
Follow the whole queue
Pending moves run without per-tick correction or early replanning. New food does not clear the plan; death ends the episode.
Trained from scratch by imitating a search-based teacher. The planner conservatively treats visible body cells as occupied for the entire plan. It can still enter body traps; the animation includes failures as well as escapes.
Run Snake & read the implementation notes ↗04 / Text → pixels
A description becomes an image.
The colored stick-figure model reads English and generates a 48×48 image one palette pixel at a time. Compare the held-out target with two generated samples for each prompt.
Training images come from a renderer; inference predicts pixels directly. Held-out deep-bend poses with a raised left arm remain unreliable in the later model too.
Explore the image experiments ↗05 / English → French
English → French
An encoder reads English subword tokens. A decoder attends to that representation and predicts French tokens one at a time. The repository walks through tokenization, masks, training, and common mistakes in this small educational translator.
16.65M parameters / trained from scratch
Choose an English sentence
Hello.
English input tokens
Hello · .
French model output
Bonjour.
Bonjour · . · EOS
Recorded translation from the local checkpoint.
Choose one of five recorded examples, then press Play or Step. This demo replays actual greedy decoding from the local checkpoint; it does not translate new text in the browser. ␠ means a space: ␠is reads as “ is”. EOS ends the sequence.
These examples are demonstrations, not a translation-quality evaluation. This small educational model can make mistakes.
Open the translation walkthrough ↗Independent educational experiments supported by PseudoConscious Inc.