The agents became useful when I stopped asking them to be brilliant.

Parameter Golf was not hard because there was one impossible file to read.

It was hard because the story was spread everywhere: rules, code, logs, saved submissions, old notes, and git history. One file might say a run worked. Another file might show a different score. A README might describe the intention, while the final metadata told the version that actually got submitted.

If I read it linearly, I kept losing the comparison.

This is where coding agents helped.

Not as autonomous researchers. More like lab infrastructure.

The article earns itself on one concrete case: a Warmdown/Int6 result where the README, metadata, log, and git history did not tell the same story. The agents were useful because they made that mismatch visible before I accidentally turned it into a clean public claim.

This article is not a separate claim that agents made the model better. It is the method note for the previous articles: how I kept score claims, architecture claims, and evaluation claims from collapsing into one confident but unreliable story.

The Problem Was Not Reading One File#

Reading one training script is manageable.

The hard part is connecting evidence across many files without flattening the differences. One record might describe a training change. Another might describe an evaluation change. Another might have a README that no longer matches its log. Another might be a non-record submission that is technically interesting but not leaderboard-eligible.

The real task was synthesis.

I needed to answer questions like:

• Which results changed the model, and which changed the scoring setup?
• Which numbers came from the final compressed artefact?
• Which claims were supported by logs rather than README prose?
• Which ideas failed because they were bad, and which failed because they were too slow for 10 minutes?
• Which ambiguous results should not become public claims?

This is exactly the kind of work where a single agent can be both useful and dangerous.

Useful, because it can read broadly and summarise quickly.

Dangerous, because it can make a stale README sound just as authoritative as the final log line.

So I split the work.

Splitting The Lab Bench#

Instead of asking one agent to “understand the repo”, I divided the read into separate passes:

That structure mattered more than the agents themselves.

It forced each reader to bring back a different kind of evidence. It also made contradictions easier to catch. If the documentation reader trusted a folder note, but the records reader saw a different value in the submission metadata, and the git-history reader saw the result’s public status become unsettled, that inconsistency became visible.

That happened with the Warmdown/Int6 folder. Here, “Int6” means a more aggressive lower-bit compression choice, and “MLP3x” means a much larger feed-forward block than the baseline. The README told one story. The log and submission metadata told another. The git history made the result look unsettled. I would have missed the shape of that ambiguity if I had only read the folder once.

The actual evidence trace looked like this:

Source anchors: this mismatch comes from the Warmdown folder’s README, submission metadata, and training log. I am not treating the later MLP3x_QAT_Int6_SlidingWindow record as the source for the 1.15744040 value.

That table became the whole point of the agent workflow. The final public wording was not “this record supports Int6 MLP3x as the answer.” It was closer to: “this is an interesting but unsettled result, and I should not use it as clean evidence until I can reproduce and isolate it.”

The same loop also helped with cleaner results. For the sliding-window evaluation record, the README, submission metadata, and training log lined up around the same mechanism: pre-quant quality was basically baseline-like, but scoring tokens with richer context moved the final BPB sharply. That let me make a stronger claim there, while still naming it as an evaluation improvement rather than a smarter trained model.

The Autoresearch Temptation#

Around this project, I was also thinking about the broader idea of AI-assisted research loops. Andrej Karpathy’s autoresearch repo is an intentionally small version of that pattern: give an agent a training setup, a metric, and a fixed budget, then let it iterate on experiments.

I did not run a fully autonomous research loop for Parameter Golf. My agents were not independently designing, training, scoring, and committing new model variants overnight.

What I tried first was closer to the dream version: give the agent previous runs, ask it to analyse what worked, ask it to propose the next tranche, and let it execute. That sounded neat in theory. In practice, the hard part was not launching another run. The hard part was getting the model to understand the experiment history well enough to make the next move for the right reason.

It could summarise logs. It could suggest ideas. It could write code. But it was too easy for the loop to blur important distinctions: pre-quant versus post-quant, training improvement versus evaluation improvement, clean ablation versus bundled record attempt. If I let that happen, I would get a busier loop, not a smarter one.

So I moved to a more manual rhythm.

I would run work in batches. Each batch had a few tranches: one set of ideas to test, inspect, or compare. After the batch, I would talk through the results with the agent, ask what seemed supported, what seemed disproved, what was still ambiguous, and what tradeoffs we were actually seeing. Then I would decide the next batch.

That was much slower than the fantasy of a fully automatic researcher. It was also much better for learning.

The rough shape was:

The Lower LR record is a good public analogue for this loop. Early on, a simple hill-climbing instinct would say: try a lower rate, see if it improves, move again. The actual record ended up more like a small grid over learning rates, and the result was not “lower is always better”. 0.06 was worse, 0.04 was default-ish, 0.025 helped, 0.02 was best in that sweep, and 0.015 stopped improving. That is the kind of shape I wanted the agent to help surface: not just the winning row, but the boundary of the tradeoff.

The useful part of the autoresearch idea, for this project, was recognising that research has repeated infrastructure around the creative act: reading, comparing, logging, validating, and remembering what changed.

What Agents Were Good And Bad At#

The best agent outputs were not the most poetic ones. They were the ones that preserved distinctions.

For example, a useful summary did not just say:

Sliding-window evaluation improved the score.

It said:

Training was roughly baseline-like, pre-quant quality was not the main improvement, and the final score moved because tokens were scored with richer context.

That distinction matters.

The agents were also useful for:

Chronology.
Git history showed that some records landed after the leaderboard had already moved, and one strong-looking result was briefly added then removed.

Evidence separation.
The data pass confirmed the actual FineWeb shards were not present locally, so the record logs and submission metadata were the main evidence.

Implementation orientation.
The code pass identified the root script as more than a naive starter: it included Muon/Adam training, int8 plus zlib export, BPB evaluation, and LoRA test-time-training evaluation.

Contradiction hunting.
The Warmdown/Int6 mismatch became a named caveat instead of silently entering the article as a clean result.

Those are not glamorous tasks. But they make the later writing much better.

The weak spots were just as important.

Agents can over-smooth a messy repo. If a README says one thing and a log says another, a bad synthesis can accidentally merge them into one confident story. Agents also tend to preserve too much detail. They will happily produce long inventories of files, flags, and paths that are useful for internal work but boring or confusing for readers.

That matters because the blog series is not a repo report.

The public article needs mechanisms:

• what changed
• why it mattered
• what evidence supports it
• what should not be overclaimed

It does not need every path, every command, or every environment variable.

So the human job became editorial judgement. I had to keep asking:

Is this detail proof, mechanism, or noise?

If it was proof, I could translate it into a reader-facing number. If it was mechanism, I could explain it with a toy model. If it was noise, I should cut it.

The Loop I Would Reuse#

If I did this again, I would formalise the agent loop earlier.

The shape would be:

1. Define the metric and the evidence hierarchy.
2. Split readers by evidence type, not by vague topic.
3. Require every claim to say whether it came from docs, logs, code, or history.
4. Keep a small scoreboard table with only reader-facing fields.
5. Mark ambiguous results as ambiguous immediately.
6. Only then turn the findings into article structure.

The key phrase is “evidence hierarchy”.

For this challenge, final log lines and submission metadata beat README prose. Git chronology helps explain contradictions. Code explains mechanisms. Top-level docs explain intent and rules. Local file timestamps are weaker than git history.

That hierarchy made the agents much safer. It also stopped this article from becoming a celebration of agents for their own sake. The useful part was not that an agent “understood Parameter Golf”. The useful part was that separate passes made contradictions harder to miss.

What This Changed About Research For Me#

The main thing I learned is that agentic research does not have to start with full autonomy.

There is a smaller, very practical version:

Use agents to keep the lab bench organised while the human keeps responsibility for judgement.

That is not as dramatic as waking up to a better model every morning. But for a learning project, it might be more important. Before I can trust an agent to run experiments, I need to trust the measurement loop, the evidence trail, and my own ability to tell a real improvement from a benchmark-shaped illusion.

Parameter Golf made that obvious.

The challenge was small enough to study, but complex enough that process mattered. Agents helped me see the shape of the experiment space. They did not remove the need to think.

The simple version is that agents gave me a wider lab bench, not a better brain. I could lay out more evidence at once, argue with it faster, and notice contradictions earlier. The judgement still had to be mine.

That is the version of agentic research I trust right now: not a replacement for judgement, but a way to make the evidence trail harder to fool. The final question, then, is what I would actually do differently if I ran the challenge again.