顶级 AI 实验室在 2026 年如何构建 RL agents（采用 Karpathy 的 system prompt 学习理念）

How Anthropic, OpenAI, and DeepSeek are converging on a single idea of using the system prompt as the reward function. A full breakdown of the RL evolution from RLHF to RULER, with code.

Anthropic、OpenAI 和 DeepSeek 在将系统提示作为奖励函数这一理念上趋于一致。从 RLHF 到 RULER 的 RL 演进全面解析，附带代码。

Reinforcement learning, at its core, is straightforward: a system takes an action, the environment rewards it, and the agent updates its behavior to maximize that reward over time.

强化学习的核心其实很简单：系统采取一个动作，环境给予奖励，智能体更新其行为以随着时间的推移最大化该奖励。

The interaction above works in discrete steps. At each step, three things happen in order:

上述交互以离散的步骤进行。在每个步骤中，依次发生三件事：

[

](https://x.com/_avichawla/article/2049037299334472015/media/2048506606695149568)

The agent observes the current state of the environment (S). A state is a description of the situation the agent is in, enough to decide what to do next. For instance, in chess, the state is the board position, and in a dialogue model, the state is the conversation history so far.

智能体观察环境的当前状态（S）。状态是对智能体所处情境的描述，足以决定下一步该做什么。例如，在国际象棋中，状态是棋盘局面；在对话模型中，状态是迄今为止的对话历史。
The agent picks an action (A) based on what it sees. The action is the agent's output, the only way it can influence the environment. For instance, in chess, an action is a legal move. For an LLM, an action is the generated response.

智能体根据其所见选择动作（A）。动作是智能体的输出，也是其影响环境的唯一方式。例如，在国际象棋中，动作是合法的一步。对于LLM而言，动作是生成的回复。
The environment then does two things: it transitions to a new state (S'), and it emits a reward (R), a scalar number that evaluates the action. The next step begins, and the loop continues.

然后环境执行两个操作：它转移到新状态（S'），并发出一个奖励（R），这是一个评估该动作的标量数值。下一步开始，循环继续。

Stringing these steps together gives a trajectory:

将这些步骤串联起来就得到了一条轨迹：

[

](https://x.com/_avichawla/article/2049037299334472015/media/2048291914454294529)

Reading left to right, this is the entire history of the agent's interaction with the envir...