Tiny transformers make every comfortable default suspicious.

That is the best summary of what I learned from the architecture side of Parameter Golf. Reading and comparing these records did not turn me into a transformer researcher, but it did make several abstract ideas feel concrete.

The useful question was rarely “is this architecture better?”

It was usually:

Is this architecture better after paying for bytes, compression, training time, and evaluation?

That extra clause changes everything.

The pattern repeats through the article:

comfortable default -> hidden byte/compression cost -> constrained replacement

Grouped-query attention, tied embeddings, mixed precision, longer context, and optimiser tuning are different techniques. What tied them together for me was that each one made an architecture default pay rent.

Attention Is Not Free#

The baseline used grouped-query attention: eight query heads, but fewer key/value heads. In plain terms, the model had more query views asking “what should I attend to?” than key/value projections storing “what information is available?”

Before this, I mostly understood grouped-query attention as an efficiency trick used in larger models. The Parameter Golf version made the tradeoff feel more immediate. Keys and values are not just compute. They are also parameters, activations, memory traffic, and eventually artefact bytes.

Full attention is the comfortable default. It is also expensive.

Grouped-query attention asks whether every query head really needs its own separate key/value projection. In a tiny model, that question is not aesthetic. It is budget.

full attention:
Q1 K1 V1 | Q2 K2 V2 | Q3 K3 V3 | Q4 K4 V4
 
grouped-query attention:
Q1 Q2 share K1 V1 | Q3 Q4 share K2 V2

In this challenge, the artefact-size effect is especially concrete: fewer key/value projection dimensions mean fewer saved weights to pay for. Runtime KV-cache savings are part of the broader attention story, but the byte-budget pressure here starts with the weights that land in the submitted artefact.

The baseline’s four KV heads were enough to be competitive early. That does not prove full multi-head attention is wasteful in every small model. It does show that a smaller attention shape can be a reasonable place to start when the model has to fit through a 16MB door.

The broader lesson:

Do not pay for symmetry just because the textbook diagram is symmetrical.

This was a recurring theme. The challenge rewarded spending capacity where it moved the final score, not where the architecture looked complete.

That sounds obvious now, but it was one of the architecture ideas that clicked for me. Before this, I would have mentally pictured “attention” as one standard block repeated across layers. Here, the useful question was more local: does every layer, every head, and every projection need to be paid for at full strength? In a tiny model, even the attention pattern is part of the budget conversation.

Tied Embeddings Are Clever And Fragile#

Tied embeddings were one of the most memorable tradeoffs.

The idea is simple: use the same matrix for token embeddings at the input and the language-model head at the output. That saves parameters because the model does not need a separate output matrix.

untied:
token -> input embedding matrix -> transformer -> output head matrix -> next-token scores
 
tied:
token -> shared embedding matrix -> transformer -> same shared matrix -> next-token scores

In a byte-limited model, that is attractive.

But the same property makes the matrix fragile. If the shared embedding is quantised poorly, the damage hits both sides of the model. The model reads tokens through that matrix and scores output tokens through that matrix.

That is why the FP16 tied embedding result made sense after I understood it. Keeping the shared embedding more precise cost bytes, and the record that did this also changed warmdown, learning rates, and MLP size. I would not treat it as a clean ablation. I would treat it as a good example of the accounting problem: if one tensor is unusually sensitive, preserving it may be worth shrinking something else.

A normal architecture summary might say:

Tied embeddings reduce parameter count.

Parameter Golf taught me the second half:

Tied embeddings concentrate importance into one tensor, so compression decisions around that tensor become unusually important.

That is the kind of lesson I wanted from the challenge. Not just what a technique does, but how it behaves under pressure.

Depth And Width Are Not Abstract#

The 10-layer mixed-precision run was another useful teaching example, though not a clean single-variable one.

The record changed depth, learning rate, and compression scheme together. Still, the shape of the move was instructive. The run tried to buy extra capacity with another transformer layer, but a plain int8-compressed 10-layer model was too large. The workaround was to compress some middle layers more aggressively, keeping early and late layers at higher precision.

That is a very golf-like move.

It says: I want the extra depth, but I cannot afford it uniformly. So which parts of the network can tolerate lower precision?

The answer in that record was to keep the first and last groups of layers at int8 and push the middle layers down to a coarser representation. Int8 means roughly eight bits per stored value after quantisation. In this record, int6 is better thought of as a coarser 64-level representation that can compress more cheaply, not necessarily as a neat six-bit packed tensor. The final score improved over the baseline, though later records found larger wins elsewhere.

The important lesson for me was not “middle layers are always safe to compress”. That would be too broad. The lesson was that depth, precision, and compression are coupled.

The byte budget turns architecture into accounting.

That sounds dry, but it is useful. It prevents a common mistake: talking about model shape as if it exists separately from the storage and evaluation pipeline.

Longer Context Made Step Count Less Obvious#

The long-context records were the architecture lesson I found easiest to feel.

Under a 10-minute budget, longer sequences are expensive. A 4096-token sequence gives the model more context per example, but each step takes longer. That means fewer updates before the wallclock limit.

The question becomes:

Would I rather give the model more updates, or better-context updates?

For the records I read, longer context was worth taking seriously. The 2048-token run beat the baseline significantly, though it also adjusted learning rates. The 4096-token run, combined with optimiser tuning and a smaller batch, improved further. That bundle makes the causal story messier, but it still changed my intuition about the tradeoff.

This helped me avoid a lazy mental model where “more steps” automatically sounds better. If the task rewards compression over real documents, the structure inside each training example matters.

It also connects back to sliding-window evaluation. Training context and evaluation context are different things, but they rhyme. A model that can use context should not be evaluated in a way that starves many tokens of it.

The result was a cleaner principle, not a universal rule:

Context is not just a maximum length in a config. It is part of the data the model is allowed to use when learning and being judged.

Optimisers Are Part Of Architecture Taste#

I used to mentally separate optimiser choices from architecture choices.

Architecture was the model. Optimisation was how you trained it.

Parameter Golf blurred that distinction. With only 10 minutes, the optimiser is part of the effective model design. If it cannot get useful weights into shape quickly, the architecture does not matter. If it produces weights that quantise badly, the final artefact suffers.

The baseline used a split optimiser setup, with Muon for matrix-shaped parameters and Adam for other pieces. Muon is an optimiser aimed at matrix-shaped weights; Adam is the more familiar general-purpose optimiser used elsewhere. Several records tuned learning rates, momentum, warmdown, and weight decay. Warmdown means deliberately decaying the learning rate toward the end of training so the weights settle before compression.

I am being careful here because I do not want to claim a universal Muon lesson from a small set of records. What I can say is narrower:

• default learning rates were not obviously optimal
• lower rates helped some runs
• warmdown and weight distributions mattered for post-quant quality
• optimiser choices affected both learning speed and compression robustness

That was enough to change how I read the experiments.

The optimiser was not background plumbing. It was one of the levers deciding whether a compact architecture could become a good artefact in time.

What I Would Not Generalise Yet#

This article is intentionally phrased as lessons from one constraint, not laws of transformer design.

I would not claim:

• grouped-query attention is always enough
• tied embeddings should always be kept in FP16
• middle layers are always less precision-sensitive
• longer context always beats more steps
• Muon tuning is always the decisive lever

Those claims would need cleaner ablations than this series has.

What I would claim is that Parameter Golf made the tradeoffs visible. It forced me to ask what each part costs, what it buys, and whether the final score is really measuring the thing I think it is measuring.

That is a better foundation than memorising architecture tricks.

What stuck with me was not any single trick. It was the feeling that architecture only became real once it had to pay rent: bytes, time, compression damage, evaluation context.

The hard part was not only understanding one run. It was keeping track of many partial runs, record folders, logs, stale READMEs, and competing hypotheses without losing the plot.

That became its own engineering problem.