Pretraining against the clock on one Trainium2 chip

1Problem

Phase 1 ran from 31 August to 30 September 2026. Each submission trains a language model from scratch on a single Trainium2 chip (a trn2.3xlarge instance) for 30 minutes of training clock, with compile and startup time excluded. The score is val bpb on a fixed shard of about 20 million tokens. The starting point is a dense GPT of roughly 50 to 60 million parameters derived from nanochat, using RMSNorm, rotary embeddings, a squared-ReLU MLP, and Muon plus AdamW. Almost everything can change except the data, tokenizer, preparation script and scoring harness. Teams get five submissions per week, and the top 10 advance.

The first thing I did was read the real training configuration instead of trusting the documentation. The baseline trained 59.6 million parameters on 1,000 steps of 65,536 tokens, about 1.1 tokens per parameter, at roughly 37.6 thousand tokens per second (about 13.4 TFLOP/s). That is a model far too large for the tokens it sees. I also found that my own first throughput estimate, made before reading the code, was off by a factor of 27, and I corrected it in the log.

2Approach

The work was a stack of small, separately measured changes rather than one recipe. By category:

• Shape for the hardware. Depth, width and which layers carry attention were re-chosen so matrix dimensions fit the 128 by 128 systolic array, and the model was made smaller for the number of tokens it can see.
• Batch and schedule. Batch size and the learning-rate schedule were swept and then fixed.
• Compile-clock handling. Compilation is outside the budget, so graph structure and packing were tuned to spend the 30 minutes on training.
• Custom kernels. Fused, hand-written NKI kernels replaced slow parts of the training step.
• Mixed precision. Higher-precision master weights, plus 8-bit floating point on part of the matrix multiplications late in the phase.

Profiling drove the order. Training ran at 90 to 98 percent of the chip's power cap, the matrix engine was the bottleneck and was throttled by power, and each new wide tensor cost about 10 percent of step time. Those three facts decided which ideas were worth a trial. In all, 49 numbered packages are in the submissions directory of the private repository, with hundreds of dated log entries.

3Protocol

Every idea went through the same gates, because a 30-minute run on a rented chip is the scarce resource.

• Hypothesis first. The expected effect and the rule for accepting or rejecting it were written before the run. An adversarial reviewer then tried to break the conclusion before anything was built on it.
• Validation ladder. CPU check, then a one-core probe, then a short smoke run, then the full 30-minute run with the exact package. Each rung exists because the one below it misses something: a CPU run does not show that a kernel executes on the chip.
• Noise characterisation. A change counts only against measured noise. Seed spread, pair noise and the local-to-official offset are given in the results.
• Elasticity. To turn a compute change into bpb, I measured bpb against ln(steps) instead of assuming a constant.
• Compliance. Anything that might read as outside the rules was checked before spending a submission slot.

4Results

Measurement: all values above are official leaderboard scores on the held-out evaluation, one submission each. The board at close put the 10th place at 0.955399, a margin of 0.0082 for my entry, and the leader about 0.0144 ahead. Official evaluations take about 40 minutes to appear.

Noise and the local-to-official offset

Measurement: local numbers are from 30-minute runs on the same chip type, not the leaderboard. The offset is not constant (it ranged from -0.0003 to -0.0033 across packages 22 to 32), so I quote it with its spread and never as a band around one point.

Two consequences shaped the plan. First, the slope of bpb against ln(steps) falls as the clock grows, so extra compute buys less than a constant slope suggests. Using the slope near the 30-minute budget, closing the 0.0144 gap to the leader would take roughly 30 to 37 percent more effective compute. Second, a single official evaluation cannot settle effects below about 0.003, so small gains were stacked locally and sent inside larger packages. I also measured the in-context-learning benchmark CORE once, on a single checkpoint on 20 September: 0.1073, against 0.1139 for GPT-2 124M. That is one measurement and says nothing about the lineage of packages.

5What did not work

A predicted gain that the leaderboard could not see

I predicted that a wider MLP variant would improve P26 by 0.00055. The official score for the package was 0.9697 against 0.969589 for P26, outside the predicted interval of 0.9686 to 0.9694. The local effect was smaller than what one official evaluation can separate from zero, so the submission slot bought no information.

A speedup that was a debug flag

An 8-bit MLP looked 27 percent faster in integration. The comparison arm without the flag showed that a debugging environment variable had been set on only one arm. Without it, the same operation ran 340 steps at 0.5212 s per step against 0.5308 s for bf16, about 1.8 percent faster.

Seed lottery

A sweep of six new seeds gave 0.965547 to 0.966626 locally (sd 0.00037), and none beat the existing two-run mean of 0.964177. Seed effects swing by about 0.0018, and a repeat of the same code and seed moved by 0.002 in the 8-bit setup, so a single run proves nothing about a seed. The last submission, P48, was a seed choice on the P46 code. It was worth about 0.0014, which is not a research result.

Dead ends in hardware and design

Attention in blocks of 128 crashed at run time. Sequence lengths below 512 were worse, and step time was flat between 512 and 1024. A scatter mode in a custom kernel aborted on the chip with a runtime error that the simulator never showed. A later architectural variant measured 0.001048 worse than the incumbent (n=2), and the optimizer and inter-replica agreement lines were closed with no measurable lever.

Measuring the wrong thing

My own CORE meter had three bugs worth +0.0480 that all passed 8 of 8 self-checks, and I caught them only later.

A compliance negative result

I built a scheme that used the training corpus as a non-parametric component at evaluation time. The organizers' review rejected it on 27 September, because what was evaluated differed from what was trained. The submission slot was spent, and I treat the rule as binding for anything similar.

Calendar

The rented chip block ended 30 minutes before its nominal end, which left about 20 hours at the end of Phase 1 with no chip to measure on.

6Lessons

• Read the real configuration before estimating anything. The baseline was badly mismatched to its token budget, and that was visible only in the code.
• Write the decision rule before the run. Most of my corrections came from a rule I had fixed in advance.
• A slope fitted in one range is not valid outside it. The elasticity of bpb against ln(steps) fell with clock length, so extrapolating upward overstated the value of compute.
• One run is one draw. Replicate in the same seed to test the number, and across seeds to test transfer.
• Treat a cheap probe on another backend as evidence about that backend only. CPU runs and the kernel simulator accepted things the chip rejected.
• Phase 2 is ongoing until 4 November 2026. On 19 October 2026 Team Quanta was 5th of 10 on the public board. I will write that phase up after it closes.