After ChatGPT took off, I spent a lot of time building around AI models.

Chat interfaces, retrieval systems, agent workflows, evals, product demos. I could make these systems feel useful from the outside.

But there was still a gap I could feel.

I knew the words people used to explain the inside of these models: attention, context windows, quantisation, optimisers. Mostly, though, I knew them as vocabulary. I could follow along. I could read code with effort. I did not yet have real taste for why one model-design choice mattered more than another.

That was why Parameter Golf caught me: build the best language model you can, but squeeze the whole submission under 16MB.

That is smaller than many phone photos.

Then I started reading the records, and it stopped feeling like a toy.

A better checkpoint did not always mean a better submission. A model could look good before compression, then lose quality after it was packed into the final file. Some runs moved the score not because the model learned more, but because the evaluation gave it more useful context.

It was not just asking me to build a tiny model. It was asking me to understand the whole path from training run to submitted artefact to final score.

This is also a selected early-record learning series, not a final leaderboard recap. I followed the records that were most useful for building intuition: the baseline, compression-sensitive runs, context/evaluation moves, and a few ambiguous cases that taught me what not to overclaim. I did not independently rerun these scores for publication.

The map I kept coming back to was simple:

training run -> checkpoint -> compression -> submitted artefact -> reload -> evaluation -> BPB score

Here, a checkpoint means the trained weights before squeezing. The artefact is the file the judge actually reloads and scores.

That chain is the series. The next article is about the game board itself: what the artefact is, how BPB is counted, and why a good checkpoint is not automatically a good submission. After that, the series follows the bottlenecks that kept moving around the chain: compression, context, evaluation, architecture choices, and finally the agent workflow I used to keep the evidence straight.

I am putting that map up front because it is the easiest way to avoid a fake story. Parameter Golf is not just “make the model better”. Sometimes the checkpoint got better and the artefact got worse. Sometimes the model barely changed and the evaluation gave it more useful context. Sometimes the interesting part was not the score, but whether the score was supported by logs, metadata, or stale prose.

Why This Constraint Caught Me#

Most public model discussions reward scale. More parameters. More GPUs. More data. Larger context. Bigger benchmark tables. It is not that those things are unimportant, but they can make learning feel vague from the outside.

If a model improves because it is larger, better trained, better evaluated, better compressed, or simply run for longer, it is hard to tell which part actually mattered.

Parameter Golf flips the feeling of the problem.

The challenge is to train the best language model that fits into a 16,000,000 byte submission artefact. That artefact is not just the model in an abstract sense. It includes the saved model and the code needed to run it. The main track has a 10-minute 8xH100 training limit, a separate 10-minute 8xH100 evaluation limit, and a score measured as bits per byte on a FineWeb validation set, specifically the challenge’s fixed fineweb_val_* split.

In plain English: make a tiny language model as good as possible, but you have to pay for the file you submit.

That is a very different kind of pressure.

The other practical reason was compute. The challenge came with sponsored compute credits for the official setup, which made experimentation feel realistic instead of purely theoretical. I was not planning to burn through the most expensive setup just to chase the leaderboard, so my own exploratory runs leaned on cheaper 3090 rentals. Those runs were for learning and iteration; the published numbers in this series come from the public records I inspected, not from my exploratory rentals.

The game card looked roughly like this:

Under a byte budget, every comfortable default becomes suspicious. Maybe the embedding matrix should be tied. Maybe it should not be compressed the same way as every other tensor. Maybe full attention is wasteful. Maybe an extra layer helps, but only if the compression scheme changes. Maybe a better validation loss does not survive the final artefact roundtrip.

The constraint makes model design feel less like a list of architecture buzzwords and more like a set of tradeoffs you can actually hold.

The Personal Reason#

I wanted to do the challenge because I was annoyed at a gap in my own understanding.

I had become comfortable building around models. I could design systems that used LLMs. I could evaluate outputs, improve latency, shape prompts, manage context, and wire agents into workflows. But the model itself was still too much of a black box in my head.

Not completely. I knew enough to be dangerous. But not enough to have taste.

Taste is the part I cared about. It is one thing to say “use a smaller model” or “quantise the weights”. It is another thing to know which tensors are fragile, which dimensions are worth preserving, what evaluation is actually rewarding, and when a supposed architecture improvement is just a measurement artefact.

I did not go in expecting to win anything or top the charts. That was never really the point.

I went in like a student.

I did not expect to become a model researcher from one challenge. That would be silly.

But I did expect the challenge to force contact with the machinery. If I had to understand why a 17 million parameter model could fit under 16MB only after compression, why a tied embedding matrix could be both clever and fragile, or why scoring with more context could move the leaderboard more than training changes, I would learn more than I would from another passive reading session.

That was the bet: use the challenge as an excuse to learn the architecture properly, not as a shortcut to a badge.

The practical work behind the series was less glamorous than the leaderboard: read the rules, compare record folders, check logs against README prose, mark which numbers were clean and which were ambiguous, and use coding agents to split that evidence work into narrower passes. That process gets its own article later, but it mattered from the start because the technical lessons were only as useful as the evidence trail behind them.

Why “Golf” Is The Right Word#

The name is accurate.

In code golf, the game is not just to solve the problem. It is to solve it under an artificial constraint that changes what elegance means. A solution that would be ugly in normal production code might be brilliant in a code golf setting because the rules reward different things.

Parameter Golf has the same energy, but with models.

The byte limit does not simply ask “what is the best transformer?” It asks a narrower and more interesting question:

What is the best transformer-shaped artefact you can fit through this tiny doorway?

That doorway changes the incentives.

A model with better raw validation loss can lose after compression. A model with more parameters can be worse if the extra tensors do not compress well. A training improvement can matter less than an evaluation strategy. A small architectural trick can be worth more than a larger idea if it survives the artefact budget cleanly.

This is why the challenge felt like a useful learning environment. It punished vague thinking quickly.

If an idea helped, it should show up in the score. If it helped only before quantisation, that was a different lesson. If it helped only because the evaluation changed, that had to be named honestly. If it required extra files, lookup tables, or hidden compute, it had to be paid for or rejected.

The rules forced the question I wanted to ask:

What actually matters when the model has nowhere to hide?

What I Thought Would Matter#

Before reading deeply through the records, I assumed the centre of the series would be architecture.

That seemed natural. The challenge is about a model. So surely the story would be about better attention, better blocks, better depth, better width, better initialisation.

Some of that did matter.

The baseline model was already compact and reasonably tuned: a small GPT-style transformer with tied embeddings, grouped-query attention, a small vocabulary, and int8 plus zlib compression. Later submissions improved it with longer context, extra depth, mixed precision, Muon optimiser tweaks, and evaluation-time tricks.

But the first surprise was that “architecture” was too narrow a frame.

The model was only one part of the game.

The pipeline looked more like this:

I started wanting to understand transformer architecture.

I ended up learning that the boundary around the model is part of the model design problem.

What I Want To Capture#

This series is not meant to be a definitive guide to Parameter Golf. The repo already contains public records from several contributors, and the strongest lessons came from reading those attempts carefully, comparing logs, and trying to understand what each result actually supported.

So the series has one narrower job: to reconstruct what the challenge taught me as a learner moving from “I know the transformer vocabulary” toward “I can reason about the tradeoffs that move a constrained model.”

That means starting with the rules and evidence trail before jumping into attention heads or quantisation formats. Without the game board, the later results look like random tricks. With it, the tricks start to look like evidence.

The simple version is that I started with a transformer challenge and ended up with a measurement challenge. That did not make the architecture less interesting. It made the architecture harder to talk about lazily.

So the question I want the rest of the series to keep asking is:

What matters when the model has nowhere to hide?