Better validation loss - the training-time signal that says the model is getting less surprised by text - did not always mean a better submission.

That was the first result that made Parameter Golf feel different from a normal training exercise. I went in expecting the interesting part to be model architecture. The records kept reminding me that the model was only one piece of the system.

The bottleneck kept moving.

Sometimes it was training. Sometimes it was compression. Sometimes it was the evaluation protocol. Sometimes it was whether I trusted the evidence enough to compare two runs.

Using the evidence hierarchy from the previous article, the first records I studied looked like this:

Source anchors: official Parameter Golf records for Naive Baseline, Lower LR, FP16 Embed, 2048 context, 4096 context, and sliding-window eval.

This is the table that made the bottleneck story visible. The point is not that every row is a clean experiment. The point is that each lower score raised a different question.

That was also when I realised the challenge was less straightforward than I first thought. There was no playbook I could follow where step one is “pick the best attention layer”, step two is “compress it”, and step three is “win”. Every improvement seemed to compromise something while trying to buy something else.

The lower-LR record is a small example. Its README reported an eight-run learning-rate sweep where MATRIX_LR=0.06 made the post-quant score much worse, the default 0.04 was not optimal, and roughly halving the rates gave the best result in that set. That is not glamorous architecture work. It is the kind of experiment that teaches you that “train harder” and “train better” are not the same thing, especially once quantisation is waiting at the end.

When Compression Became The Problem#

The baseline model already showed the issue.

Before compression, it scored around 1.2172 BPB. After the int8 plus zlib roundtrip, it scored 1.22436570. The gap was not enormous, but in a leaderboard where improvements could be a few thousandths, it was very real.

This was my first compression lesson:

The model you trained is not exactly the model you submit.

For a small transformer, this is not just an implementation detail. It changes the design space. If a tensor is unusually sensitive to quantisation, the right move might be to spend more bytes preserving it and save bytes somewhere else.

That is the hypothesis the FP16 tied embedding run made concrete.

The embedding matrix was doing double duty. It mapped token IDs into vectors at the input, and because embeddings were tied, the same matrix also acted as the output head. Quantisation error there could affect both how the model reads and how it speaks.

Keeping that matrix in FP16 cost roughly 500KB, so the model had to give something back elsewhere. The run reduced MLP hidden size to stay under the 16MB budget. It also bundled schedule and learning-rate changes, so I would not treat it as a clean one-variable result. Still, the final BPB improved to about 1.2197, and the direction of the result matched the mechanism.

That is not a dramatic jump by itself, but the lesson was durable:

Training Was Still Real#

Compression mattered, but it did not make training irrelevant.

The lower-learning-rate run improved the baseline slightly, suggesting that the default learning rates were too aggressive for the constraint. It also had a hardware note: it was run on 8xH200, so faster step time may have explained part of the gain.

The long-context runs improved much more. Moving from 1024-token training windows to 2048 or 4096 tokens reduced the score substantially, even though each step became slower. Those records also changed other training details, especially in the 4096-token run, so the honest claim is narrower than “context alone won”.

That tradeoff is worth pausing on.

In a fixed wallclock budget, longer sequences mean fewer optimisation steps. The naive instinct is to prefer more steps. But language modelling is not just about update count. It is also about how much context the model sees while learning to predict the next token.

The 2048-token run reached about 1.2058 BPB. The 4096-token training optimisation reached about 1.2014. In these records, fewer steps with more context looked better than more short-context updates.

So the lesson was not “compression is all that matters”. It was more specific:

Learning rate: lower LR improved the baseline slightly, with a hardware caveat, which suggested the default optimisation was a little too aggressive.

Embedding compression: FP16 tied embeddings reduced post-quant damage, which suggested some tensors deserved special treatment, even though the run bundled other tuning changes.

Context length: longer-context records beat the short-context baseline, which suggested the model needed better examples, not just more updates, while still needing cleaner ablations.

Evaluation context: sliding-window eval moved the score sharply, which suggested the measurement itself had changed the effective task.

Each improvement answered a different question.

That is why the sequence of records felt less like a leaderboard and more like a diagnosis log.

The Evaluation Twist#

The sliding-window evaluation result was the point where my mental model really changed.

The baseline evaluation chops validation into non-overlapping windows. That means the first token in each window has almost no context, the next has one token of context, and so on. On average, tokens are scored with much less than the maximum available context.

Sliding-window evaluation changes that.

Instead of scoring every token in a fresh non-overlapping chunk, it moves a window forward by a smaller stride and only scores the rightmost part of the window. Those scored tokens have much richer context behind them.

The important fairness detail, as I read the record’s evaluation path, is that the stride changes how much previous context the model gets before the scored tokens; it is not meant to skip hard tokens or change the denominator.

chunked eval:
[score all 1024 tokens][score all 1024 tokens]
 
sliding eval with stride:
[context ............ score last 64]
        [context .... score last 64]
                [context score last 64]

The challenge rules allow evaluation methods to be pushed, as long as evaluation finishes within its own 10-minute 8xH100 budget and does not access training data unless those bits are paid for. That makes sliding-window evaluation a legal evaluation-side move, but it is a different kind of move from training a better model.

In the record, the training was basically baseline-like. The pre-quant BPB was not meaningfully better. But the final score dropped to 1.19250007.

That is a huge lesson.

It did not mean the trained model suddenly became much smarter. It meant the evaluation stopped asking the model to predict many tokens with artificially little context.

This distinction matters because it is very easy to tell the wrong story:

Bad story:

Sliding-window evaluation made the model better.

Better story:

Sliding-window evaluation let the same kind of model be scored in a way that used its context more effectively.

That difference is the kind of honesty I want to keep through the series. A leaderboard score moved. The reason it moved was evaluation context, not necessarily more learned language ability.

Document Boundaries Were Another Hidden Variable#

The LoRA test-time-training record made the evaluation story even more interesting.

At first glance, LoRA TTT sounds like the exciting part. LoRA means adding small low-rank adapter weights; TTT means test-time training. In this record, the idea was to adapt those tiny adapter weights at evaluation time, document by document, and use test-time compute to improve predictions without changing the submitted base model.

That is interesting. But the record’s own ablation is more subtle.

Most of the gain came before the LoRA step:

The LoRA adaptation helped, but the bigger lesson was that context structure mattered. If the model is evaluated across document boundaries as if the text were one continuous stream, it is being asked a slightly strange question. Resetting around documents and scoring with better context fixed much of that. The 1.2278 row is the baseline from this record’s own ablation table, so I read it within that local comparison rather than as a contradiction of the earlier 1.2244 leaderboard baseline.

Again, the bottleneck moved.

What looked like “maybe we need a clever test-time learner” partly became “maybe we need to stop scoring the model in a context-poor way”.

What Changed#

By this point, I stopped thinking of Parameter Golf as a search for one clever architecture.

It was a search for the active bottleneck.

If raw validation quality was the bottleneck, training and architecture mattered most. If quantisation was the bottleneck, tensor sensitivity and compression mattered. If evaluation context was the bottleneck, sliding windows and document boundaries mattered. If evidence quality was the bottleneck, logs and reproducibility mattered.

That is a much better way to learn than collecting tricks.

Once I had that frame, the architecture lessons became sharper. Tiny models punish assumptions quickly, but the punishment only teaches you something if you know which part of the pipeline delivered it.

That is what the next article is about: not a catalogue of transformer tricks, but the architecture defaults that started looking expensive once every byte had to justify itself.