The Next Parameter Golf Run
contents
If I took a clean run at this, I would spend less time hunting for the clever architecture and more time making the bottleneck visible earlier.
That is the cleanest postmortem.
The challenge started as a way for me to learn transformer architecture. It did that. But the stronger lesson was broader: small-model work is only partly about the block diagram. The rest is measurement, compression, evaluation, and research hygiene.
The next run would start there.
If I had to answer the opening question now, I would put it this way:
When the model has nowhere to hide, the move I trust is not just a clever block. It is knowing which part of the pipeline actually moved the score.
That is the synthesis the next run should test.
What I Would Keep#
The pieces I would keep are mostly boring, which is a good sign.
| Practice | Why I would keep it | Failure mode it prevents |
|---|---|---|
| Score the round-tripped artefact | The submitted file is the product | Celebrating a checkpoint that compression breaks |
| Track pre-quant and post-quant BPB | Compression damage is part of the model-design problem | Treating tensor fragility as bookkeeping |
| Separate training from evaluation | Sliding-window scores and better-trained models differ | Calling an evaluation win a model win |
| Attach every number to a source | Logs, metadata, READMEs, and git history have different weight | Turning stale prose into public evidence |
| Use agents for evidence passes | They help keep broad comparisons organised | Letting one confident summary flatten ambiguity |
That is the base layer for another run.
What I Would Be More Careful About#
I would be much more careful about claims that mix mechanisms.
For example, a run might change context length, learning rate, batch size, optimiser momentum, and evaluation method. If the score improves, the result is useful, but it does not tell me which lever mattered unless the ablations are clean.
That is fine for a competition. It is weaker for learning.
The next run would separate two modes:
| Mode | Goal | Success condition |
|---|---|---|
| Record chasing | Improve the final score | Stack whatever works under the rules |
| Understanding | Isolate a mechanism | Change one thing and measure the right gap |
Both modes are valid. The mistake is pretending they answer the same question.
If I am writing publicly, I need to mark which mode produced a claim. “This won” is not the same as “this explains why it won”.
Experiments I Would Run First#
My next pass would start with a small set of ranked hypotheses.
| Rank | Hypothesis | Controls | Primary metric | Decision rule |
|---|---|---|---|---|
| 1 | Some tensors deserve more bytes | Fixed checkpoint; vary only tensor precision/compression policy | Post-quant BPB delta per byte spent | Protect tensors that buy the most BPB per byte; demote the rest |
| 2 | Evaluation context is a separate lever | Fixed trained model; vary stride and eval sequence length within the eval budget | BPB plus evaluation wallclock | Keep the richest legal context that improves BPB, but label it as evaluation gain |
| 3 | Longer training context can beat more updates | Same tokenizer, model shape, and evaluation protocol where possible; compare 1024, 2048, 4096 | Post-quant BPB, steps completed, wallclock | Treat it as training signal only if it survives fixed evaluation |
| 4 | Extra depth and MLP bytes have different payoff | Compare depth and MLP size under the same compression policy | Final BPB, artefact bytes, post-quant gap | Spend bytes where final BPB improves after roundtrip |
| 5 | Test-time adaptation has marginal value after eval controls | Lock document isolation and stride first; compare no-LoRA vs LoRA | BPB improvement and eval-time cost | Keep adaptation only if it beats the fixed evaluation baseline within budget |
| 6 | Better records improve learning speed | Every run gets source of metric, code diff, hardware, artefact size, and claim status | Completeness of the claim ledger | No public claim until the evidence row is filled |
This list is less exciting than “try a wild new architecture”. That is the point.
I would rather make the bottleneck visible before trying to beat it.
That is the builder lesson I would carry outside the challenge too: instrument the pipeline before chasing cleverness. Otherwise the clever idea might work, and I still would not know why.
The Result I Would Reproduce Before Reusing#
The Warmdown/Int6 MLP3x result is the obvious example.
It is interesting because the log and submission metadata point to a very strong score, but the folder documentation does not cleanly match. Git history made the public status look unsettled, so I would keep it as an experiment seed, not as a settled claim.
The next run would isolate it:
- reproduce the exact final score
- confirm the artefact byte count
- separate normal roundtrip BPB from sliding-window BPB
- compare against the same model without the extra MLP
- document why it was removed or whether it was simply superseded
Until then, it belongs in the category of “interesting, not fully trusted”.
That category is useful. It stops me from either discarding the idea or overclaiming it.
What I Can Now Say#
The scoped claim is narrower than mastery and more useful: I can now separate model quality, artefact quality, evaluation effects, and evidence quality more clearly than when I started.
Before this series, I might have looked at a leaderboard jump and asked:
What architecture trick made the model better?
Now I would ask a different set of questions:
| Question | Why it matters |
|---|---|
| Did the raw model improve? | Separates learning from packaging |
| Did the compressed artefact hold up? | Separates checkpoint quality from submitted quality |
| Did evaluation context change? | Separates model quality from scoring protocol |
| Is the number log-backed or prose-backed? | Separates evidence from summary |
| Was the run a clean ablation or a record bundle? | Separates explanation from competition progress |
That is the skill I wanted from Parameter Golf. Not a certificate. A better instinct for where to look when a small model improves, fails, or lies through its metric.
What I Took From The Process#
The bigger personal takeaway is about learning with agents.
I do not think there is anything noble about refusing to use coding agents or AI tools just to prove that every line was handwritten. That feels like fighting the wrong battle. The more useful standard is whether I still understand the thing I am building, testing, or claiming.
That frame fits how this project changed me. The agents helped with execution, synthesis, comparison, and debate. They made it easier to pick up a difficult area quickly. But the understanding still had to be mine. I had to decide which sources counted, which claims were overreaching, and which tradeoffs were actually worth learning from.
That is the version of agentic work I want to keep using: not delegating the understanding, but using the tools to get to the understanding faster.
If I run Parameter Golf again, the goal is not just a lower BPB. It is a lower BPB with a cleaner explanation.
That is the part that feels most portable outside this challenge. Better systems do not only produce better numbers. They make it easier to tell why the number moved, what tradeoff paid for it, and whether I should trust it enough to build the next thing on top.