通过最终Token偏好优化减少Doom Loops

Repetitive degeneration [1] is a common failure mode during inference: the model emits a span (often something like "Wait, let me reconsider…"), then repeats the same span again and again, until the context window is exhausted. We call this phenomenon the 'doom loop'. Small reasoning models are more prone to this behavior, especially on long thinking traces and hard problems [2].

重复退化 [1] 是推理过程中常见的失败模式：模型生成一段文本（通常是类似“等等，让我重新考虑……”的内容），然后一遍又一遍地重复同一段文本，直到上下文窗口耗尽。我们将这种现象称为'doom loop'。小型推理模型更容易出现这种行为，尤其是在长思维轨迹和困难问题上 [2]。

The commonly applied inference-time fix is to apply repetition_penalty to reweight the output distribution. However, this is a band-aid solution and can degrade performance. Reinforcement learning can target repetitive looping, but it typically requires carefully calibrated rewards and costly online rollouts.

通常应用的推理时修复方法是应用 repetition_penalty 来重新加权输出分布。然而，这只是一种治标不治本的解决方案，并且可能会降低性能。强化学习可以针对重复循环问题，但它通常需要仔细校准的奖励和昂贵的在线rollouts。

Our method takes a more targeted approach. We identify the exact token that begins a loop, train the model to prefer coherent alternatives at that single position, and leave the rest of the distribution largely untouched. The method adapts Antislop [3], training on chosen/rejected pairs that represent a single completion token, using Final Token Preference Optimization (FTPO). We call our approach “Antidoom”.

我们的方法采取了更具针对性的策略。我们识别出引发循环的确切 token，训练模型在该单一位置上偏好连贯的替代方案，并让分布的其余部分基本保持不变。该方法改编自 Antislop [3]，在代表单个补全 token 的 chosen/rejected 对上进行训练，使用最终 token 偏好优化（FTPO）。我们将我们的方法称为“Antidoom”。

On an early checkpoint of LFM2.5-2.6B, 10.2% of completions on hard math and coding prompts produced repetitive loops. After Antidoom training, that rate fell to 1.4%, with eval scores improving across the board as a direct result of reduced looping.

在 LFM2.5-2.6B 的早期 checkpoint 上，针对困难数学和编码提示的补全中有 10.2% 产生了重复循环。经过 Antidoom 训练后，该比例降至 1.4%，并且由于循环减少，各项 eval 分数均有所提高。

Anatomy of a doom loop

Doom loop 剖析

Doom loops can arise in inference from three mechanisms working together:

在推理过程中，Doom loops可能由三种机制共同作用而产生：

Me...