The first baseline I studied did not tell me whether I had a good model.

It told me which game I was actually playing.

That sounds like a small distinction, but it changed how I read every experiment afterwards. Parameter Golf is not just “train a small transformer”. It is a chain of constraints. A run can look good at one point in the chain and become worse later.

The important thing is to know where the score is coming from.

Terms I Needed First#

I did not need a whole transformer textbook to start reading the records, but I did need a small shared vocabulary:

The Artefact Is The Product#

The obvious version of the challenge is this:

Train the best small language model.

The real version is more precise:

Submit code plus a compressed model artefact under 16,000,000 bytes, then score the round-tripped model with validation BPB.

That means the final artefact is not a side effect. It is the product.

In normal model training, I would mostly think about validation loss. If validation loss goes down, the model got better. That is still useful here, but it is not the whole answer. The submitted model has to be serialised, compressed, decompressed, loaded again, and evaluated.

Any quality lost during that trip is part of the score.

This is the first mental model I needed:

training run
  -> raw model quality
  -> quantised/compressed artefact
  -> round-tripped model quality
  -> final BPB score

If the raw model gets better but the compressed version gets worse, the leaderboard does not care about my feelings. It scores the artefact.

BPB In Plain English#

The score is bits per byte, usually written as BPB.

I found it useful to avoid thinking of BPB as just another magic benchmark number. It is a compression-flavoured way of asking: how many bits does the model need, on average, to describe the validation text?

Lower is better.

Validation loss is measured over tokens. The challenge code takes token negative log likelihood in nats, converts it to bits, then normalises by the counted bytes of text. Nats are natural-log units; dividing by ln(2) converts them to bits:

BPB = total_negative_log_likelihood / ln(2) / counted_text_bytes

Equivalently, it is asking how many bits the model needs per byte of validation text. If two tokenizers split the same document differently, BPB tries to compare them against the same underlying bytes rather than only against their token boundaries.

That is separate from the compressed artefact size. The artefact budget asks how many bytes the submission file costs. BPB asks how many bits the model needs to describe validation text.

A toy intuition: if a model is less surprised by the next piece of text, it spends fewer bits describing it. Small BPB changes matter because score deltas were often measured in thousandths; a drop from 1.2244 to 1.2197 is meaningful, while a drop to 1.1925 is a much larger jump that usually deserves an explanation beyond “the model got better”.

A toy version:

That last row is the trap.

A model can have a decent pre-quant score, then lose quality after compression. The difference between those two numbers is not bookkeeping. It is evidence about which tensors, formats, and training choices are fragile.

There are two different compression ideas in play. Quantisation is lossy: it stores weights with fewer bits, so the loaded model may behave differently. zlib is lossless: it squeezes the bytes after they have already been represented. If quantisation damages an important tensor, zlib cannot fix the model quality; it only helps the file fit.

The Baseline Was Already A Real Model#

The baseline was not a toy in the sense of being careless.

It was small, but it had sensible choices baked in. It used a 1024-token SentencePiece vocabulary, a 9-layer transformer, 512-wide hidden states, grouped-query attention, tied embeddings, a relu^2 MLP, and int8 plus zlib compression for the final artefact.

In reader-facing terms:

The baseline stopped at 13,780 training steps under the 10-minute wallclock cap. Its pre-quant validation BPB was about 1.2172. After int8 plus zlib roundtrip, the final score was 1.22436570.

That gap mattered immediately.

The model was not just learning. It was also losing something when squeezed into the artefact.

How I Marked Evidence#

Once I understood the baseline, I needed a way to keep later numbers honest.

Unless stated otherwise, the BPB values in this series are record-reported values from public logs, README files, and submission metadata I inspected, not fresh reproduced measurements for the article. I use them as a map of the game board, not as clean ablations.

The hierarchy I used was rough but useful:

logs / submission metadata > code path > README prose > my inference

That is not a formal scientific protocol. It is the rule that kept me from giving every source the same weight. A final log line and a submission.json value can still be wrong, but they are closer to the run than a folder note written for humans. Code explains mechanisms. READMEs explain intent. My inference is useful only after it says what it is leaning on.

The convention is:

That convention matters because the same-looking number can mean different things. A log-reported final roundtrip BPB is stronger evidence than a stale README sentence. A metadata value that disagrees with the README is a flag, not a conclusion. Once this rule was in place, the later articles could spend less time apologising for every caveat and more time explaining what each record seemed to teach.

The Baseline Taught Me What To Distrust#

Before this, I would have been tempted to read “better validation loss” as “better model”. That is not wrong, but it is incomplete here.

The baseline made me distrust a few easy interpretations:

Pre-quant quality is not final quality.
If the model collapses under compression, the raw score is not enough.

The artefact budget creates strange incentives.
Sometimes a smaller model can be the better submitted artefact if the larger one compresses badly.

This is why I think the game board deserved its own article. If you skip it, the later technical details sound like random tricks. If you understand it, the later tricks become moves in a constrained system.

The Rule That Carried Forward#

By the end of the baseline read, I had a better frame for the series.

I was not simply asking:

How do I build a better small transformer?

I was asking:

Which part of the pipeline changed, and what evidence supports it?

That question became the rule for reading every later record. It kept the baseline article from becoming a bag of tricks, and it kept the later technical articles honest.

The way I think about it now: before I could learn from the leaderboard, I had to learn how the leaderboard was produced. Otherwise I would be learning from shadows.