Merging fine-tuned experts gives you multi-task behaviour at the inference cost of one model, with no joint training run. The usual recipe is to fine-tune each expert, stop it at its best validation checkpoint, and merge from there. We wanted to know whether that checkpoint is the right one, and it turns out it usually isn't.

The sweep covered three Qwen 3.5 sizes, five domains from math to safety, and five merging methods. For each expert we found T*, the validation-optimal checkpoint, then kept training out to 5× T* and merged at every point. Simple averaging peaks while the experts are still undertrained, while TIES and DARE+TIES reach their best well past T*, in the range most would call overfitted. The effect grows with model size: TIES on the 4B model hits 68.5% refusal on HarmBench between 3× and 5× T*, more than 2.5 times what simple averaging manages at any duration.

The difference comes down to noise. Training past the optimum lowers bias but raises variance, and simple averaging weights every parameter equally, so all of it flows into the merged model. TIES and DARE trim the small updates and resolve sign conflicts first, which soaks most of it up, the same reason a random forest can afford deep trees.

Training duration and merging method are one decision rather than two. If you merge with a sparsification method, don't early-stop your experts. The paper has the per-domain breakdowns and a mode-connectivity analysis of why LoRA keeps overtrained experts mergeable.