Biohub: cell tracking during development

1Problem

Each sample is a 3D+time movie of a developing zebrafish embryo. A submission is a graph: nodes are detected cells, edges link the same cell across frames, and a division is a node with two children. The score is adjusted_edge_jaccard + 0.1 * division_jaccard. The prize pool was $60,000 over seven places. Five submissions per day were allowed and two counted at the end.

Three properties of the metric shaped the work. The ground truth is sparse: only about 2.8% of nodes were annotated overall (133318 of 4725117, 6.1% on average per sample), so an edge only counts as a false positive if it touches an annotated node. The metric weights samples by size, so large dense movies dominate. And the whole training set holds only 151 divisions, which makes the division term both the largest opportunity and the largest source of variance.

In the early weeks more than a thousand teams were bunched between 0.91 and 0.93. On Sep 5 the seventh place stood at 0.953 and the leader at 0.970.

2Approach

I built on public notebooks: a pretrained detector pack released under CC0 and an ILP linker from tracksdata, plus a public notebook family that sat on a plateau around 0.933 to 0.940. I did not finish training a new detector; a local training run on a laptop was stopped before its first checkpoint. Estimated cost in the notes was 2.5 to 5 hours per epoch on a T4, with no resume, so training on the shared Kaggle GPU quota was out of reach.

Oracle experiments first

Before tuning anything I asked which stage could move the score. With perfect detection and nearest-neighbour linking, 97.8% of edges are recovered, and greedy linking ties the Hungarian assignment. The oracle (perfect detection plus nearest neighbour) scores 1.0674 with the official scorer, which says the metric is not capped by linking. On the real detector, node recall on ground-truth nodes was 0.984 weighted (20 graphs with ground truth, minimum 0.92), so detection headroom was about +0.01.

Local reproduction of the ILP

I dumped candidate graphs from two movies and reproduced Kaggle's ILP solution edge by edge locally with ilpy 0.6 and SCIP, in about 2 seconds. That gave an off-GPU bench for solver weights and thresholds. I also built a local run of the post-processing chain scored with the official scorer; on one movie it matched the Kaggle validator.

Cheap reruns

A stub notebook skips inference and reuses cached outputs, so a visible rerun cost 0.4 minutes of GPU instead of 15 to 60. That turned the daily submission limit, not GPU quota, into the real budget, and I used it to probe public-notebook parameters on the leaderboard.

3Results

Measurement: every score above is the Kaggle public leaderboard, which covers about 29% of the test set, one submission each (n=1). Public scores carry a shakeup risk; the final private-leaderboard placement is not reported here and I do not claim one. The 0.946 is my best public score, not a final result.

The most useful numbers were diagnostic rather than leaderboard moves. Switching the division module off cost 0.040 on the hidden test, so divisions are worth about 0.040 there (division Jaccard around 0.40 on the hidden test against 0.20 on my local bench of 16 graphs). Of 2598 edge failures over 60 training samples, 72% were association errors and 28% were detection errors. On two dumped movies the correct target was inside the top 8 candidates before thresholding in every case, and for 65% of the 31 failures with both endpoints detected, the 0.48 threshold had dropped the right candidate at rank 1.

4What did not work

Greedy linking and the per-column cross-entropy head

Replacing the ILP with greedy linking put submissions 7 and 8 below the pure public baseline of 0.894 to 0.896 (pack plus the tracksdata ILP); the best of them scored 0.881 on the leaderboard. Retraining the edge head with a per-column cross-entropy loss scored 0.874 on the leaderboard. The loss never sees detections without a parent, so it linked noise: 27% more nodes and nearly twice as many forks on one sample. My window benchmark was blind to exactly that case.

Pruning and division gates

An early short-track pruning rule cost 0.045. Sweeping the minimum track length on the official scorer (16 graphs) gave 0.9100 at 1, 0.9105 at 6 and 0.9010 at 10, so the public default of 6 was already the optimum. The public division gates were a local optimum in all 13 variants I swept; the one that looked better (+0.0016) turned out to be noise. On the leaderboard, halving the gate caps scored 0.941 and an eighth scored 0.922.

Changing the ILP division weight

Lowering the division weight to 0.5 scored 0.917, below the 0.933 floor. The reason is in the solver: with edge weight -1.0 and division weight 1.2, the margin of any division is non-negative, so the public ILP never divides (0 forks in 414256 nodes, against 2457 at weight 0.5). All divisions come from post-processing. At 0.5 the solver and the post-processing both emit forks and the false positives add up.

A motion model, and detection tuning

I refuted a motion model before building it: of 300 association failures with matched ground truth, 38% were gaps with no successor, and the local motion field was right in 7% and preferred the wrong rival in 90.9%. Detection threshold probes landed at 0.940 (0.98) and 0.933 (0.99). The second-seed weight curve is not smooth: 0.5 gave 0.945, while 0.3 and 0.0 gave 0.935 and 0.934. Eleven knob probes in total stayed between 0.933 and 0.946.

Local benches that did not transfer

Turning off the motion relink gained 0.016 in edges on a four-sample bench but scored 0.932 on the leaderboard, because it helped the small samples and hurt the large dense one that dominates the weighted metric. The local bench predicted the direction of leaderboard changes but not their size, which was two to three times larger for divisions.

5Lessons

• Run the oracle experiment before choosing a stage to improve. Here it showed that the stage I had already spent two submissions on was not the constraint.
• State an expected range per change from the per-sample spread, not the mean. With fewer than 10 samples the range must include "no change". Three of my declared ranges failed before I adopted this rule.
• Read the scorer. Its sample weighting and sparse ground truth explained results that looked inconsistent, and the 0.040 value of the division module only appeared by switching it off on the real test.
• Be wary of a solver weight whose name suggests a direction. Raising the division weight to 1.2 "to encourage divisions" made divisions unprofitable.
• I could not close the gap to the top because the top teams retrained the detector and my budget did not allow it. Building tools that do not need the GPU (the local ILP, the post-processing replay, the stub rerun) was the productive response to that limit, but it did not change the ceiling.