Personal project

Galley

A small GPT-style transformer built from scratch in PyTorch, with a workbench for inspecting tokens, attention and loss.

Python
PyTorch

Code

Attention heatmap from the Galley workbench, showing a causal triangle over the tokens of a short sentence. — One attention head in the Galley workbench. The triangle is the causal mask at work.

Problem

It's easy to use a language model without knowing what happens inside one. I wanted to write each part myself and be able to see it working, from raw bytes to generated text.

What I built

Galley covers a byte-level BPE tokenizer, causal multi-head attention, transformer blocks, a training loop with a cosine schedule, checkpoints, and greedy, temperature and top-k sampling. There is also code for LoRA and instruction fine-tuning.

A browser workbench shows the prompt, token spans, any attention head and the loss curve.

How

Byte-level BPE tokenizerTrained on the built-in paragraph
Sliding-window batchesToken ids to inputs and targets
EmbeddingsTokens and positions
Causal attentionMulti-head, masked
Transformer blocksThe GPT stack
TrainingCosine schedule, checkpoints
SamplingGreedy, temperature, top-k
WorkbenchTokens, attention, loss

The default debug config is deliberately tiny, so a full train and inspect loop runs on a laptop CPU: 2 layers, 4 heads, embedding width 32, context 32 and vocabulary 384.

A separate config matches the 124M GPT-2 tensor shapes, so an external checkpoint can be copied across.

Results

The debug model has 50,880 parameters. A 40-step training run took the loss from 5.97 to 4.92 on CPU.
The workbench exposes real attention weights for each layer and head.
The limit is explicit: it trains on one short paragraph, so the samples are not useful prose. Understanding was the goal, not output quality.

What I learned

Seeing the attention matrix made masking concrete in a way the equations never did.
A tiny model that trains in seconds is a far better debugging tool than a large one.

Next projectTranscribeMate