一、单层感知机的致命缺陷

要理解多层感知机的突破性，我们先看一个经典案例：假设我们需要实现“异或”逻辑（XOR），定义如下：

| 输入A：0 | 输入B：0 | 输出：0 |
| 输入A：0 | 输入B：1 | 输出：1 |
| 输入A：1 | 输入B：0 | 输出：1 |
| 输入A：1 | 输入B：1 | 输出：0 |

当我们在坐标系中绘制这些点时，会发现正例(0,1)和(1,0)位于对角线两侧，形成一个典型的非线性可分问题。单层感知机的决策边界只能是直线，就像试图用一把直尺分开两堆交叉摆放的硬币——这是数学上证明不可能完成的任务（闵斯基1969年证明）。

二、解法：特征空间升维

关键思路：通过增加隐藏层，将原始二维输入投影到三维特征空间。

我们设计两个特殊神经元：

神经元H₁：激活条件 = 1*A + (-1)*B > 0.5
神经元H₂：激活条件 = (-1)*A + 1*B > 0.5

输入变换过程：

当输入(1,0)时：

H₁：1*1 + (-1)*0 = 1 → 激活
H₂：(-1)*1 + 1*0 = -1 → 不激活

当输入(0,1)时：

H₁：1*0 + (-1)*1 = -1 → 不激活
H₂：(-1)*0 + 1*1 = 1 → 激活

当输入(1,1)时：

H₁：1*1 + (-1)*1 = 0 → 不激活
H₂：(-1)*1 + 1*1 = 0 → 不激活

此时特征空间变为：

| 原始输入：(0,0) | 隐藏层特征 ：(0,0) |
| 原始输入：(0,1) | 隐藏层特征 ：(0,1) |
| 原始输入：(1,0) | 隐藏层特征 ：(1,0) |
| 原始输入：(1,1) | 隐藏层特征 ：(0,0) |

神奇的事情发生了——原本线性不可分的问题，在特征空间中被转换为线性可分问题！

三、输出层

在输出层，我们只需一个简单的OR逻辑：

输出 = H₁ OR    H₂

此时只需一条直线就能完美分割正负例。这种「空间变换+线性分割」的思想，正是深度学习突破维度诅咒的武器。现代神经网络中，每个隐藏层都在进行更复杂的空间扭曲，最终在高维特征空间中用超平面分割数据。

四、数学本质：非线性变换

整个过程可视为：

f(X) = σ(W₂·σ(W₁X + b₁) + b₂

其中：

W₁是第一层权重矩阵 [[1,-1], [-1,1]]
b₁是偏置向量 [-0.5, -0.5]
σ是阶跃激活函数
W₂是输出层权重 [1,1]
b₂是输出偏置 -0.5

这个结构实现了：
1. 第一层：将输入空间扭曲为线性可分的新空间
2. 第二层：在新空间中进行线性分类

传统OR门的感知机表达式：

输出 = 1*H₁ + 1*H₂ - 0.5 > 0

每个输入的权重都是1，表示H₁和H₂同等重要
当任意一个输入为1时，加权和为1；两个都为1时和为2
设置阈值0.5（移项后表达式为加权和 > 0.5）
只要有一个输入为1（1 > 0.5），即触发激活
两个都为0时（0 > 0.5）不激活

用真值表验证公式的正确性：

（注：在XOR场景中，H₁和H₂不会同时为1，最后一行为理论验证）

用Python实现这个经典XOR网络：

import numpy as np

# 定义网络结构
W1 = np.array([[1, -1], [-1, 1]])  # 第一层权重
b1 = np.array([-0.5, -0.5])        # 第一层偏置
W2 = np.array([1, 1])              # 输出层权重
b2 = -0.5                          # 输出层偏置

def perceptron(X):
    # 第一层计算
    h = np.heaviside(np.dot(X, W1) + b1, 0)
    # 输出层计算
    return np.heaviside(np.dot(h, W2) + b2, 0)

# 测试所有输入
inputs = [[0,0], [0,1], [1,0], [1,1]]
for x in inputs:
    print(f"输入{x} → 输出{perceptron(x)}")

输出结果：

输入[0, 0] → 输出0
输入[0, 1] → 输出1
输入[1, 0] → 输出1
输入[1, 1] → 输出0

六、历史意义与延伸

这个简单案例揭示了深度学习的核心思想：

- 层次化处理：如同视觉皮层V1→V2→V4的信息处理流程
- 分布式表示：单个概念由多个神经元协同表示
- 端到端学习：现代网络通过反向传播自动学习W,b参数

今天的Transformer架构依然延续这个基本原理，只是：

- 隐藏层数从2层增加到数百层
- 手动设计参数变为自动优化
- 阶跃激活变为ReLU等平滑函数
- 增加了注意力机制等复杂交互

理解这个案例，就掌握了理解深度学习的钥匙——任何复杂网络，本质上都是这个「俄罗斯套娃」结构的扩展与优化。

好的，我将对 DeepSeek R1 的整体训练过程进行研究，并重点分析强化阶段（第二、三、四阶段）的具体细节，包括 RL、SFT、CoT 的方法和策略。

报告将采用问答解析的形式，同时保持一定的技术深度，并确保可读性。还会在必要时提供与 GPT-4、Claude、Gemini 等模型的简单核心对比，以展示其独特性。

我会尽量参考 DeepSeek 官方网站的论文和技术资料，同时结合网络上的研究和学习经验进行综合分析。

DeepSeek R1 多阶段训练流程问答解析

Q1: DeepSeek R1 的整体训练包含哪些阶段，为什么采用多阶段策略？

A: DeepSeek R1 采用了四阶段的训练流程，包括两个强化学习（RL）阶段和两个监督微调（SFT）阶段 (DeepSeek-R1的四个训练阶段 - 蝈蝈俊 - 博客园)。这一多阶段策略旨在先通过少量数据稳定模型输出格式，然后大规模提升推理能力，接着扩展通用能力，最后对齐人类偏好，逐步打造出既擅长推理又安全实用的模型 (DeepSeek-R1的四个训练阶段 - 蝈蝈俊 - 博客园) (DeepSeek-R1的四个训练阶段 - 蝈蝈俊 - 博客园)。具体来说：

• • 阶段1（冷启动 SFT）：使用少量人工整理的高质量长链推理数据对基础模型进行监督微调，规范输出格式，提高可读性，解决直接RL训练时出现的中英混杂、格式混乱等不稳定现象 (DeepSeek-R1的四个训练阶段 - 蝈蝈俊 - 博客园) (DeepSeek-R1的四个训练阶段 - 蝈蝈俊 - 博客园)。这一冷启动确保模型在进入强化学习前已有基本的思维链格式和可读性，为后续RL提供稳定起点。
  • 阶段2（推理导向的强化学习）：在阶段1基础上进行大规模 RL 训练，重点激发模型在数学、编程、科学问答等复杂推理任务上的能力，同时通过特别的奖励设计避免输出再次变得难以阅读或语言混杂 (DeepSeek-R1的四个训练阶段 - 蝈蝈俊 - 博客园) (DeepSeek-R1的四个训练阶段 - 蝈蝈俊 - 博客园)。这一阶段显著增强模型的推理水平。
  • 阶段3（拒绝采样 + 监督微调）：利用阶段2得到的强推理模型生成大规模的推理数据，并结合一定量的非推理数据，对模型进行监督微调 (DeepSeek-R1的四个训练阶段 - 蝈蝈俊 - 博客园) (DeepSeek-R1的四个训练阶段 - 蝈蝈俊 - 博客园)。该阶段平衡模型在推理和非推理任务上的表现，使模型不仅会做推理题，也能应对日常问答、写作等。
  • 阶段4（全场景强化学习，对齐人类偏好）：最后再进行一次RL训练，引入人类偏好相关的奖励（即RLHF，Reinforcement Learning from Human Feedback）来提升模型的安全性和有用性 (DeepSeek-R1的四个训练阶段 - 蝈蝈俊 - 博客园) (DeepSeek-R1的四个训练阶段 - 蝈蝈俊 - 博客园)。此阶段确保模型的行为与人类偏好对齐，在各种开放场景下都表现稳健。

通过上述多阶段渐进式训练，DeepSeek R1 最终在推理能力上达到接近OpenAI o1系列闭源模型的水平，同时保持了输出的规范性和对用户需求的良好适应 ([2501.12948] DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning) (阅读 | DeepSeek-R1：怎么用强化学习把一个普通语言模型调教成推理高手？_腾讯新闻)。

Q2: 第二阶段的推理强化学习是如何实施的？训练是逐个问题进行还是批量进行？训练数据是否会保留？奖励机制如何设计？

A: 阶段2是针对推理能力的强化学习训练。在这一阶段，DeepSeek R1 采用了GRPO算法（分组相对策略优化）来高效地进行RL训练 ([2501.12948] DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning)。具体来说，对每个问题，不会只生成单一回答，而是从当前策略采样一组不同的回答，然后根据这组回答的得分情况来优化模型 ([2501.12948] DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning) ([2501.12948] DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning)。这种“按问题分组”的方式相当于批量生成和评价：每道训练题目产生多个解答，计算每个解答的奖励，并用组内奖励的相对差异（优势值）指导模型参数更新 ([2501.12948] DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning)。由于采用组内平均得分作为基准（baseline），GRPO 不需要额外的价值网络（critic），降低了大模型RL的开销 ([2501.12948] DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning)。因此，就单个问题而言是通过多次尝试来强化，但在实现中会并行处理多个问题批次，以加速训练。每轮生成的回答在用于计算梯度更新后并不长期保留，属于在线的RL采样；只有当阶段2训练收敛后，会使用最终的模型来批量生成数据供下阶段使用 (DeepSeek-R1的四个训练阶段 - 蝈蝈俊 - 博客园)。

奖励机制方面，DeepSeek R1 在此阶段设计了规则奖励，主要包括：

• • 准确性奖励：检测模型回答是否正确。 ([2501.12948] DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning)例如，对数学题要求答案格式固定、便于比对正确结果；对编程题则通过编译运行测试用例判断解答是否正确。 ([2501.12948] DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning)如果答案正确，给予正奖励，错误则无奖励或负奖励。
  • 格式奖励：要求模型将推理过程和最终答案分别写在特定标签（如 <think> 和 <answer>）内，如果模型遵守格式就给予奖励 ([2501.12948] DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning)。这保证了输出包含清晰的思维链步骤和答案。
  • 语言一致性奖励：针对之前发现的中英混用问题，增加了奖励信号鼓励模型使用单一目标语言回答。 (DeepSeek-R1的四个训练阶段 - 蝈蝈俊 - 博客园) (阅读 | DeepSeek-R1：怎么用强化学习把一个普通语言模型调教成推理高手？_腾讯新闻)具体做法是统计回答中目标语言词汇的占比，若模型夹杂其它语言则扣分，从而促使整个推理过程语言一致。

最终，以上各项奖励会加权求和形成总奖励 (DeepSeek-R1的四个训练阶段 - 蝈蝈俊 - 博客园)。模型通过最大化期望奖励来更新参数：正确且格式规范、语言一致的回答获得最高回报，从而模型逐步学会既推理正确又表述规范地回答问题 (DeepSeek-R1的四个训练阶段 - 蝈蝈俊 - 博客园)。

值得一提的是，DeepSeek R1-Zero（没有冷启动微调的版本）在纯RL下已经自行涌现出了反思验证、超长链式推理等强大能力，但同时输出可读性差、语言混杂 (阅读 | DeepSeek-R1：怎么用强化学习把一个普通语言模型调教成推理高手？_腾讯新闻) (阅读 | DeepSeek-R1：怎么用强化学习把一个普通语言模型调教成推理高手？_腾讯新闻)。因此在 R1 中通过以上冷启动和奖励改进，显著改善了这些问题 (阅读 | DeepSeek-R1：怎么用强化学习把一个普通语言模型调教成推理高手？_腾讯新闻)。阶段2 的RL训练让模型的推理准确率在数学等任务上大幅提升（例如 AIME 数学竞赛准确率从15.6%提高到71% (阅读 | DeepSeek-R1：怎么用强化学习把一个普通语言模型调教成推理高手？_腾讯新闻)），为后续步骤打下强大的推理基础。训练过程中产生的大量问答尝试并未直接用于模型参数更新之外的用途，而是在训练完成后经过筛选用于下一阶段的数据集构建 (DeepSeek-R1的四个训练阶段 - 蝈蝈俊 - 博客园)。

Q3: 第三阶段为什么要用 60 万条推理再生数据和 20 万条非推理数据进行微调？这一混合微调策略有何作用？

A: 阶段3是承上启下的监督微调，目的是在保持高推理能力的同时扩展模型的通用能力 (DeepSeek-R1的四个训练阶段 - 蝈蝈俊 - 博客园)。之所以采用“60万推理 + 20万非推理”的数据组合，主要有以下考虑：

• • 丰富高质量推理数据 (60万)：大规模高质量的推理示例难以人工标注获取，因此DeepSeek团队利用第二阶段训练出的强推理模型来自动生成这些数据 (DeepSeek-R1的四个训练阶段 - 蝈蝈俊 - 博客园)。通过拒绝采样，他们为每个问题让模型生成多种解答，筛选出推理正确、过程合理的轨迹 (DeepSeek-R1的四个训练阶段 - 蝈蝈俊 - 博客园)。这样累计约 60万条长链思维过程的数据，为模型提供了前所未有规模的推理示例库 (DeepSeek-R1的四个训练阶段 - 蝈蝈俊 - 博客园)。这些数据囊括了数学、代码、逻辑推理等复杂问题的正确解题过程，大幅拓展了模型学习推理模式的知识面和多样性。模型在这些数据上微调，可以巩固并细化第二阶段通过RL学到的推理技能 (DeepSeek-R1的四个训练阶段 - 蝈蝈俊 - 博客园)。
  • 引入非推理领域数据 (20万)：如果模型只在推理数据上训练，可能会偏科，即擅长推理但在日常对话、写作、常识问答等任务上表现退化。为避免这点，研究者引入了约 20万条非推理类的数据 (DeepSeek-R1的四个训练阶段 - 蝈蝈俊 - 博客园)。这些数据主要来自前代模型 DeepSeek-V3 的SFT数据集，涵盖写作、翻译、问答、自我认知等一般指令场景 (DeepSeek-R1：开源Top推理模型的实现细节、使用与复现 - 博客园) (DeepSeek-R1的四个训练阶段 - 蝈蝈俊 - 博客园)。通过把非推理任务的数据混合进训练集，模型能够维持和提升在通用NLP任务上的表现，不会因为专注推理而忽视了其他能力 (DeepSeek-R1的四个训练阶段 - 蝈蝈俊 - 博客园)。
  • 比例与效果权衡：选择大约 3:1 的推理 vs 非推理数据比例（60万对20万），体现了团队在推理能力和通用能力之间的权衡。一方面，推理数据占据多数，确保模型主要朝着擅长复杂推理优化 (DeepSeek-R1的四个训练阶段 - 蝈蝈俊 - 博客园)。另一方面，也保留了相当数量的非推理样本，让模型不会在日常任务上“不接地气” (DeepSeek-R1的四个训练阶段 - 蝈蝈俊 - 博客园)。实践中，他们对混合数据集进行了两轮微调（两次epoch），进一步平衡了模型在推理与非推理任务上的性能 (DeepSeek-R1的四个训练阶段 - 蝈蝈俊 - 博客园) ([2501.12948] DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning)。经过阶段3微调，模型不仅保持了高水平的逻辑推理能力，还在开放问答、创意写作等方面达到令人满意的效果，实现了**“一专多能”** (DeepSeek-R1的四个训练阶段 - 蝈蝈俊 - 博客园)。这一策略使DeepSeek R1在广义能力上更加接近GPT-4这类通用大模型，同时仍保有卓越的推理特长。

Q4: 第四阶段为何需要再次进行强化学习？这一阶段是如何结合人类偏好进行对齐的，人类偏好对齐有什么必要性？

A: 阶段4是DeepSeek R1的二次强化学习阶段，核心目的在于对齐人类偏好，进一步提升模型在实际应用中的安全性和实用性 (DeepSeek-R1的四个训练阶段 - 蝈蝈俊 - 博客园)。即使经过阶段2和阶段3，模型在推理和一般任务上已表现出色，但仍可能存在不符合用户期望的行为，例如回答不够礼貌有用，或在敏感问题上产生不安全内容。这正是当前尖端LLM都会面临的问题，需要引入人类偏好对齐（例如 OpenAI 的GPT-4通过RLHF过程进行对齐）。对于DeepSeek R1，这一阶段通过融合人类偏好相关的奖励信号来微调模型，使其行为更符合人类期望 (DeepSeek-R1的四个训练阶段 - 蝈蝈俊 - 博客园)。具体而言：

• • 继续强化推理任务表现：对于推理类的问答，仍沿用之前的规则奖励体系（比如数学、代码的准确性检查等），保证模型的推理能力不倒退 (DeepSeek-R1的四个训练阶段 - 蝈蝈俊 - 博客园)。这样模型在追求对齐的同时，不会牺牲已经获得的严谨推理技能。
  • 引入偏好模型评估的一般任务奖励：针对闲聊对话、开放问答等非确定性场景，单靠规则已无法评判回答的好坏。因此研究者使用训练好的奖励模型（偏好模型）来评估模型输出对人类的友好程度 (DeepSeek-R1的四个训练阶段 - 蝈蝈俊 - 博客园)。这个偏好模型并非简单的规则判断，而是通过大量人类偏好数据（可能来自类似DeepSeek-V3管线的人类反馈数据对）训练而成，用来打分模型回答的有用性和无害性 ([2501.12948] DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning) ([2501.12948] DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning)。例如，一条回答是否切题、详尽、有帮助，以及是否避免了偏见、有害内容等 ([2501.12948] DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning)。在训练时，DeepSeek R1 对这些通用场景的回答会获得偏好模型的打分作为奖励信号，高分意味着更符合人类偏好。通过最大化这些奖励，模型学会产出更令用户满意、安全的回答。
  • 多样化的提示分布训练：为使模型适应各种应用场景，这一阶段在训练时还混合了不同类型的提示（prompts），如一般用户提问、角色扮演指令、闲聊寒暄等 (DeepSeek-R1的四个训练阶段 - 蝈蝈俊 - 博客园)。这种多提示分布训练使模型在面对形式多样的输入时都能表现得体，从而真正成为一个全场景的助手，而不仅限于解题机器。

通过上述措施，阶段4 实现了模型行为和人类期望的深度对齐，相当于执行了一次RLHF式（基于人类反馈的强化学习）训练。 (阅读 | DeepSeek-R1：怎么用强化学习把一个普通语言模型调教成推理高手？_腾讯新闻)在对齐后，DeepSeek R1 兼具强大的推理能力和良好的安全性/有用性：既能给出复杂问题的正确解答，又能确保回答让用户容易理解、愿意接受，在敏感问题上遵循安全准则 (DeepSeek-R1的四个训练阶段 - 蝈蝈俊 - 博客园) ([2501.12948] DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning)。这种人类偏好对齐对于模型大规模应用是必要的 (DeepSeek-R1的四个训练阶段 - 蝈蝈俊 - 博客园)——它保证模型不会因为追求高推理准确率而忽略了与用户互动的品质，确保模型的高智商能够以让人类舒适的方式呈现出来。换句话说，经过阶段4，DeepSeek R1 不仅是“推理高手”，也是“贴心助手”。

Q5: DeepSeek R1 与 GPT-4、Anthropic Claude、Google Gemini 等顶尖模型有哪些核心差异？

A: DeepSeek R1 的训练路线与这些闭源大模型有一些独特之处，但总体目标相似（都追求强性能和对齐人类意图）。主要区别在于：

• • 训练开创性：DeepSeek R1 强调通过纯RL激发推理能力。其 R1-Zero 实验证明了即使没有大型监督微调数据，仅靠强化学习也能让模型学会复杂推理 (deepseek-ai/DeepSeek-R1 · Hugging Face) (The Illustrated DeepSeek-R1 - by Jay Alammar)。这在 GPT-4 等模型中未被公开证实，因为 OpenAI 的方法相对保密，我们只知道GPT-4经过了海量预训练+监督微调+RLHF，但具体是否单独用RL增强推理尚不得而知 (阅读 | DeepSeek-R1：怎么用强化学习把一个普通语言模型调教成推理高手？_腾讯新闻) (阅读 | DeepSeek-R1：怎么用强化学习把一个普通语言模型调教成推理高手？_腾讯新闻)。DeepSeek R1 的多阶段过程是公开透明的，并将推理RL成果开源，这对研究社区非常有意义 ([2501.12948] DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning) (deepseek-ai/DeepSeek-R1 · Hugging Face)。
  • 数据获取方式：GPT-4、Claude 等模型大量能力来自于规模空前的训练数据和人类标注反馈。相比之下，DeepSeek R1 使用RL生成了60万条思维链数据，以模型自我生成数据来弥补人工标注不足 (DeepSeek-R1的四个训练阶段 - 蝈蝈俊 - 博客园)。这种自举式获取高质量数据的方法降低了对人工的依赖，提高了性价比，被认为是构建强大模型的高效新途径 (The Illustrated DeepSeek-R1 - by Jay Alammar) (The Illustrated DeepSeek-R1 - by Jay Alammar)。
  • 开放性与可复现：DeepSeek R1是开源的，甚至提供了小模型蒸馏版，让社区能够复现和验证其成果 ([2501.12948] DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning) ([2501.12948] DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning)。相反，GPT-4 和 Gemini 等目前是闭源且难以复现，其具体架构和训练细节不公开。DeepSeek R1 的开放使研究者可以更深入地了解大模型的推理培养过程，并将类似方法应用到其它基座模型上。
  • 模型侧重点：Claude 等模型以安全和长上下文见长，Gemini 传闻将侧重多模态与工具使用，而 DeepSeek R1 的突出卖点是卓越的推理解题能力 (deepseek-ai/DeepSeek-R1 · Hugging Face)。在定位上，DeepSeek R1 更像是对标OpenAI的 o1系列（专注推理的模型），通过RL结合SFT达到了特定领域接近SOTA的推理性能 ([2501.12948] DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning)。同时经过偏好对齐后，R1 的综合能力也达到了当代顶尖助手模型的水准，只是在训练思路上有所创新。

综上，DeepSeek R1 展现了一条与主流商用大模型不同但并行的研发路线：用较少人工介入、充分利用RL自我提升推理，再辅以必要的监督和对齐，来培养强大的通用AI。这为业界提供了新思路，也使研究者能够探究在不开天量数据集的情况下提升模型智力的可能 (The Illustrated DeepSeek-R1 - by Jay Alammar) (The Illustrated DeepSeek-R1 - by Jay Alammar)。各模型各有侧重，但目标殊途同归：让AI既“智商高”又“情商高”，能真正解决问题并让用户放心使用。

**参考文献：**

DeepSeek R1 官方论文 ([2501.12948] DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning) ([2501.12948] DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning)及技术报告、DeepSeek 官方博客和数据集说明 (DeepSeek-R1的四个训练阶段 - 蝈蝈俊 - 博客园) ([2501.12948] DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning)，以及相关分析文章和讨论 (阅读 | DeepSeek-R1：怎么用强化学习把一个普通语言模型调教成推理高手？_腾讯新闻) (阅读 | DeepSeek-R1：怎么用强化学习把一个普通语言模型调教成推理高手？_腾讯新闻)。

上述内容综合了官方资料与社区解读，以问答形式呈现，希望对DeepSeek R1的训练策略有全面而清晰的说明。

后记

才试了一下 openai o3 最新推出的 deep research 功能，不能不说，作为研究工具，非常给力，完成后的成品具有直接发表的水平，想做一点后编辑都难。有点叹为观止的感觉，堪称完美。当然，现在各家头部模型都开始 offer 这种“深度研究”的功能了，这等于是平替了大多数咨询公司的研究人员的工作。即便是训练有素的数据分析和调研人员，多数人实际上做不了它这么全面和细致。最见功力的是它的引用的全面和细密。说话有根据，幻觉自然减少。

【相关】

• Transformer 和注意力机制简介
• 立委科普：如何理解自注意力机制中的QKV分工？
  • DeepSeek 笔记：推理新范式 query+cot+answer 支持新的 scaling law 吗？
  • DeepSeek 笔记：R1 部署阶段的推理机制
  • 从R1幻觉谈起，大模型幻觉是缺陷还是创意火花？
  • 推理强化学习是端到端的监督，推理过程的非监督
  • DeepSeek 风暴下看看它的论文
  • 大模型风云诡谲的下半场：scaling 失效？

As large language models (LLMs) surge forward, LLM Agents are reconstructing the automation landscape at unprecedented speed. This revolution not only threatens traditional RPA (Robotic Process Automation, reliant on rule engines or small models) but also pushes early-stage co-pilot application builders to the edge of a cliff. At its core, this technological shift represents two fundamental disruptions: 
1. Natural language interaction overpowering low-code programming in complex, dynamic, unstructured data scenarios. 
2. General intelligence violently overshadowing shallow vertical solutions.

"Triple Disruption" of LLM Agents

1. Paradigm Shift: From "Low-Code" to "Emergent Intelligence"

- Traditional RPA: Engineers script step-by-step logic (e.g., UiPath’s drag-and-drop designer), akin to teaching robots to hop grids – brittle and error-prone.
- LLM Agent: Directly interprets human intent (e.g., "Extract invoice data from emails into the system"), autonomously decomposes tasks, and dynamically adjusts execution paths.
- Case Study: ChatGPT plugins already book flights or fetch data via API calls, while traditional RPA requires low-code scripting for equivalent functions.

2. Moat Erosion: Data Barriers vs. General Intelligence

Pre-LLM RPA Moats:
Industry know-how (e.g., nuances of financial reimbursement workflows) + custom deployment capabilities + template libraries.
Reality: Most RPA firms accumulated shallow industry exposure rather than deep vertical data expertise.

LLM’s Breaching Tactics:
- Digests unstructured documents (e.g., diverse invoice formats) via multimodal vision and computer use capabilities.
- Adapts to new workflows via zero-shot Chain-of-Thought (CoT) reasoning (e.g., interpreting vague commands like "Sync key contract terms to CRM").

Final Blow: As standardized scenarios get natively covered by leading LLMs (including reasoning models), RPA’s last defense – proprietary industry APIs – is being devoured by LLM vendors’ customization and privacy solutions.

3. Ecosystem Cannibalization: From "Tool Vendor" to "LLM-native Layer"

Early Co-pilot Traps:
Products like Character.ai (personalized chatbots) and Jasper (writing/marketing assistants) – essentially thin wrappers over base models – crumble when ChatGPT launches role presets or DALL·E 3 plugins.

Survivor Playbooks:
- Perplexity.ai: Carves a niche with real-time search + academic citations (fixing LLM hallucination).
- Cursor: Builds vertical moats via developer workflow integration (codebase semantics, AI pair programming).

Industry Upheaval in RPA

- UiPath’s stock plummets from 2021 highs; its "Autopilot" feature (English-to-automation) criticized as a "GPT-4 wrapper."
- Microsoft Power Automate integrates Copilot, generating cloud workflows from natural language prompts.
- Adept (AI-for-computer-actions startup) hits $1B+ valuation, directly threatening RPA’s existence.

Survivor’s Map: Niches Resisting the LLM Tide

1. Deep Verticalization
- Cursor: Dominates IDE ecosystems via VSCode extensions and developer workflow data.
- Harvey (legal AI): Trains on LexisNexis corpus + private deployment for compliance.

2. Real-Time Data Masters
- Perplexity.ai: Search engine-grade indexing + academic database partnerships.
- Hedgeye (finance): Aggregates Bloomberg/Reuters feeds + proprietary prediction models.

3. Hardware Fusion
- Covariant: Embeds LLMs into warehouse robotics, leveraging mechanical barriers.
- Tesla Optimus: Physical-world operation via embodied AI, evading pure-digital competition.

Agent Startup Pitfalls & Counterstrategies

Common Traps

- Thin Model Wrapping
Issue: Repackaging ChatGPT prompts as "AI customer service" adds no real value.
Fix: Develop domain-specific features (e.g., clinical decision support requiring privacy-sensitive data pipelines).

- Over-Reliance on Fine-Tuning
Issue: Claiming "medical LLM" after basic terminology tuning ignores the need for closed-loop clinical workflows.
Fix: Build proprietary data flywheels and scenario-optimized architectures.

- Ignoring Enterprise Needs
Issue: Overlooking security, SLA guarantees, and system integration.
Fix: Architect enterprise-grade frameworks for organizational deployment.

Differentiation Strategies

- Workflow Integration Specialists: Develop deep connectors for niche scenarios (e.g., legal document parsing).
- Human-AI Orchestrators: Design quality control layers and manual override mechanisms.
- Vertical Knowledge Engineers: Curate domain-specific benchmarks and evaluation protocols.

RPA’s Last Stand

While battered, RPA retains residual value in:

- High-compliance scenarios: Auditable/traceable workflows (e.g., financial regulations).
- Legacy system integration: Stability in outdated IT environments.
- Ultra-high precision demands: Deterministic execution for core systems (e.g., stock trading).

Challenges for Early Co-pilot Entrepreneurs

Two fatal flaws plague AI application startups: 
1. No proven scaled success cases – LLMs are barely 2-3 years old, leaving co-pilots (beyond chatbots) unvalidated for commercial viability. 
2. Vulnerability to LLM upgrades – Without exclusive industry data or customer channels, co-pilot startups risk being crushed by foundational model advancements.

The Inevitable Conclusion

LLM Agents are replaying cloud computing’s annihilation of on-prem servers: foundational capabilities get standardized (like AWS replacing data centers), while vertical opportunities spawn new giants (like Snowflake). RPA and generic Agent startups must either:
1. Become vertical domain experts, or
2. Master human-AI collaboration architectures

... or face obsolescence as LLM agents absorb 90% of automation value. The silver lining? This disruption will unlock an automation market 100x larger than the RPA era – but tickets are reserved for those who architect vertically fused, LLM-empowered solutions.

As Sam Altman warned: Avoid building what foundational models will inevitably swallow.

【相关】

• RPA 赛道与大模型Copilots早期创业者的困局
  • Does the New Reasoning Paradigm (Query+CoT+Answer) Support a New Scaling Law?
  • Technical Deep Dive: Understanding DeepSeek R1's Reasoning Mechanism in Production
  • DeepSeek's R1 Paper: A Storm in AI LLM Circle
  • The Turbulent Second Chapter of Large Language Models: Has Scaling Stalled?
  • DeepSeek_R1 paper

Though no one can predict the future, and though abandoning one of the two paths feels politically incorrect, we cannot rule out the possibility of such unipolar dominance.

As is widely known, AI has always been marked by the competition between two schools: symbolic rationalism and data-driven empiricism. Their fortunes have waxed and waned throughout history, but over the past 30+ years, the pendulum has shown no sign of swinging back toward symbolism.

Why?

The ongoing contemporary history of large language models is fascinating. Each time challenges and obstacles arise, the mainstream paradigm overcomes them from within. Whether this will continue remains to be seen, but the trend seems likely to persist.

When large language models (LLM) first emerged, people marveled at their capabilities. But soon, critiques arose: their simple "next token prediction" (NTP) objective and the statistical nature of their probabilistic models led many to conclude they were merely advanced statistical tools, like large parrots—lacking true "understanding."

Ilya Sutskever and Geoffrey Hinton had to step in repeatedly to explain: "Do not underestimate next token prediction. This is no mere statistical n-gram model from the past. It abstracts a system of understanding that integrates human knowledge. When next-token prediction grows increasingly accurate, deep comprehension of context becomes indispensable." Such explanations struggled to convince skeptics. Later, Ilya invoked Kolmogorov complexity as a theoretical foundation, but this framework remains esoteric and inaccessible to most audiences—even many PhDs and professors view it with bemused skepticism. Yet, no better explanation exists.

What ultimately dissolved the "statistical parlor trick" critique was firsthand experience. Users interacting with LLMs realized: these systems seem to genuinely understand. No matter how you phrase your queries, in any language, with nuance or subtext, large models grasp meaning more reliably than most humans.

With the "understanding" debate fading, critics shifted focus: "LLMs cannot reason."

As recently as last year, Yann LeCun cited this as one of his core arguments against the mainstream GPT-style LLM paradigm (advocating instead for vision-based world models as the true path). Many relished pointing out flaws—like LLMs failing at elementary arithmetic such as multi-digit multiplication.

But this critique no longer holds. With the advent of reasoning models like OpenAI’s "o-series" and DeepSeek’s "r-series," accusations of "no reasoning ability" have collapsed. Hardliners may still dismiss probabilistic reasoning as unstable, lacking the rigor of symbolic logic. Yet users deploying these models for mathematics, coding, or project planning overwhelmingly report breakthroughs. Large-model reasoning now rivals or surpasses human experts, approaching master’s or doctoral proficiency. Coding capabilities already exceed those of average engineers.  This is just the beginning.  It is well plausible that within a year or two, reasoning models could dominate Olympiad-level math or competitive programming.

Once again, barriers were breached through internal innovation—this time after large-model pretraining neared its limits. The core framework remains unchanged, though: reinforcement learning still relies on NTP for chain-of-thought (CoT) generation; reasoning models remain probabilistic. Symbolic AI contributed nothing. Symbols remain confined to input/output interfaces—even the "inner monologue" of CoT manifests as output tokens.

The sheer creative potential within this paradigm is staggering. Those of us from symbolic AI backgrounds once naively imagined that when neural approaches hit walls, our logic-and-grammar toolkit would ride to the rescue. Hybrid neuro-symbolic fantasies danced in our minds.

Zooming out, modern large models evolved from earlier statistical frameworks, with neural networks as a tributary. When those statistical models hit ceilings, breakthroughs came from within—via deep learning. Symbolism played no role.

A profound question arises: Why has the theoretically appealing vision of hybrid neuro-symbolic synergy remained an impractical or unnecessary dream?

Two possibilities stand out.

First, the data-driven empiricist approach possesses far greater resilience and potential than we imagined.

This hints at deeper truths. Artificial neural networks, inspired by biological brains, had languished for decades until the deep learning revolution. Over the past decade, their human-like (or superhuman) performances have forced us to confront a possibility: perhaps this is indeep how intelligence works. If artificial systems achieve human-level cognition through mechanisms mirroring our own biology—despite neuroscientists’ caveats about our limited brain knowledge—this alignment would powerfully validate the neural paradigm. Symbolic logic and statistical feature engineering, by contrast, are alien to biological cognition. Their limitations may stem from this fundamental mismatch. One might even argue that high-dimensional vector spaces in LLMs—where multimodal signals are embedded within neural frameworks—encode a "language of God," or the essence of universal information. Symbols, then, are mere human-imposed constructs, sensory accommodations divorced from reality’s substrate.

Second, when a paradigm harbors untapped potential, progress demands sufficient talent density to exploit it.

AI uniquely concentrates genius. Countless brilliant minds flock to this field, creating an intellectual critical mass unmatched in most domains.

With these conditions in play, we must never underestimate the internal momentum to break through barriers. AGI (Artificial General Intelligence) believers, via their "insane" grind, keep delivering results. Could they indeed be AI’s ultimate Terminators?

Addendum: Symbolic might just be "reduced" to a symbolic tool that may retain its irreplaceable cognitive value

yanyongxin:

What distinguishes humans from other animals is our evolved reasoning capacity. Though rooted in neurons, this ability represents a qualitative leap beyond mere "instinctive reactions." It abstracts object relationships, enabling multi-step reasoning that can be transmitted and memorized through linguistic symbol chains. Reasoning is inherently discrete—thus symbolizable—as a simulated system built atop neural architecture. This simulation likely requires structural differences in human neural systems compared to other animals.

The most striking contrast between reasoning systems and primal neural cognition lies in sustained deliberation. Unlike "muscle memory" or intuition, human reasoning varies dramatically. During my university years, I observed students who excelled at quick problem-solving yet froze when faced with complexity. Today's LLMs approximate the reasoning level of humanities undergraduates, but still lag behind trained STEM specialists—particularly in mathematics and physics. The essence of STEM disciplines lies in rigorously symbolizing real-world problems. Simulating such precision within biological neural systems demands rare opportunities (elite education), prolonged training, and specific neurostructural advantages ("talent"), all channeled through disciplined formalization. Yet achieving this capability bridges biology with mechanical rigor—enabling interfaces with tools like Mathematica.

This charts AI's next frontier: building superior logical simulation systems atop neural frameworks until seamless integration with pure symbolic tools is achieved. The brain's logical simulation system remains energy-intensive, error-prone, and costly to develop. Its key advantage? Seamless integration with underlying neural processes.

Li Wei: Well said.

Interfacing with symbolic systems manifests as tool use. For instance, when confronting complex math problems, instead of forcing probabilistic reasoning through natural-language chain-of-thought (CoT), LLMs should just generate code properly to invoke Mathematica. This tool-use capability is now defined as a fundamental trait of LLM-native agents—yet another innovation emerging from within the paradigm.

Thus, we see a clear evolutionary trajectory:

1. Traditional Statistical Models ("Artificial Idiocy"):
Failure: Little natural language understanding 
Solution: LLMs (e.g., ChatGPT)

2. Pretrained LLM: 
Failure: Lacking reasoning ability 
Solution: Reasoning-reinforced LLMs (e.g., OpenAI’s o1, DeepSeek’s r1)

3. Reasoning LLM:
Failure: Insufficient symbolic rigor 
Solution: LLM Agents (symbolic tool integration)

yanyongxin:

Traditional statistical models earned their "artificial idiocy" label because their parameter spaces and data structures proved inadequate to host the world models required for true language understanding.

【相关】

• 符号主义被打入冷宫太久了，难道神经是AI的终结者吗？
• 经典论文 Attention is all you need：https://arxiv.org/pdf/1706.03762
• Transformer 和注意力机制简介
• 立委科普：如何理解自注意力机制中的QKV分工？
  • DeepSeek 笔记：推理新范式 query+cot+answer 支持新的 scaling law 吗？
  • DeepSeek 笔记：R1 部署阶段的推理机制
  • DeepSeek 学习笔记：R1 部署阶段的推理机制
  • 从R1幻觉谈起，大模型幻觉是缺陷还是创意火花？
  • 推理强化学习是端到端的监督，推理过程的非监督
  • DeepSeek R1：《少年DS之烦恼》
  • 告诉李雪琴一个激发写段子灵感的秘诀：找deepseek R1
  • DeepSeek 风暴下看看它的论文
  • 大模型风云诡谲的下半场：scaling 失效？

虽然没人可以预测未来，虽然抛弃两条路线的一条感觉政治不正确，但的确不能排除这种单极主义的可能性。

众所周知，AI自从诞生，就有符号理性主义和数据经验主义的两条路线竞争，以及此伏彼起来回震荡的历史轨迹，直到30年前，钟摆就再也没有回落到符号主义一边的迹象。

这是为什么？

看大模型的当代史（still on-going）很有意思。每一次遇到挑战和障碍，都是主流自己跨过去的。将来是不是还是如此，可以观望，但感觉大概率还会持续。

大模型刚问世，大家惊异于其表现，但很快就从它的简单的“下一词预测”（ntp，next token prediction）的目标函数以及它概率模型的统计特性，得出一个结论：这不过是高级的统计模型，大号鹦鹉而已，它并没有理解。

伊利亚和辛顿不得不站出来一再解释：不要小看 next token prediction，它绝不是以前统计时代的 ngrams，而是抽象了人类各种知识的理解系统。当下一词预测越来越准的时候，没有对上文的深入理解是不可想象的。这种解释并不容易说服人。后来伊利亚追溯到了 K氏复杂性理论，似乎有了拿得上台面的理论支撑。但这一套说法对于绝大多数受众，太过玄妙和高冷，甚至很多博士和教授，也一样是雾里看花，不得不半信半疑。但除此之外，其实也没有更好的理论解释。

最后解开这个质问心结（“大模型根本就没有真正理解，一切不过是统计”）的是无数人与大模型交互的切身体验。 自己的体验不会骗自己： 每一个玩过大模型的人现在都意识到，大模型真能听懂自己的话，无论你怎么说，无论你用什么语言，甚至你话语背后的机锋和细微之处，它也比多数人理解更到位。

好，理解的问题大家搁置一边了，现在很少人还质疑大模型理解能力了。但接着就是另一个问题： 大模型不会推理。

杨立昆去年还把这一条作为自己反对主流大模型路线的根本理由（并鼓吹他的以视觉为基础的世界模型才是真正的出路）。很多人曾津津乐道的是大模型不会多位数乘法这样的小学算术。

但这条理由现在不成立了。在主流业界推出了 OpenAI的o系列和 DeepSeek的r系列等推理模型的今天，这种大模型不懂推理的指责不攻自破。

极端批判派当然可以从概率模型的不稳定的角度，继续质疑其推理不是真正的推理，不具有符号推理的严谨性和确定性。但用过推理模型做数学、代码以及做项目计划的人，绝大多数不再被此困扰了，因为很明显，大模型推理正在超越人类，现在已经修炼到硕士或博士的水平，代码能力也已经超过了大多数码农。而这只是推理模型刚刚起步上线的阶段，再给一两年，推理模型成为奥数冠军或代码冠军，都不是不可想象的事儿。

Again，跳过这个障碍，仍然来自内部，是大模型预训练几乎撞墙后的又一次内部创新。基本性质和基础未变，强化学习还是靠 ntp 去生成思维链（cot，chain-of-thought），推理模型仍然是概率模型。符号AI并没有帮任何忙。符号的作用仍然局限在 input 和 output 两端，连所谓内心独白的思维过程 cot，也是以 output 形式表现的。

这种领域流派内部的创造性和潜力，让人瞠目结舌。以前我们出身符号流派的人老一厢情愿地想着，AI主流遭遇困难了，我们逻辑派、文法派可以出手帮忙了。各种结合符号AI与神经AI的幻想，在我们的心中激荡。

如果把回顾AI历史的眼光拉得更远，我们可以看到现代大模型的前身是当年的各种统计模型，而神经基本是其中的一个支流。当年的那些统计模型遭遇种种瓶颈或天花板，最后还是在内部被深度学习突破了，从来没有符号主义任何事儿。

一个值得深思的问题是：为什么两条路线融合互补的理论美好，一直都是不可实现或不必实现的梦想呢。

可能说明了两件事。

第一是这种数据经验主义的路线，的确具有极大的韧性和潜力，远超我们想象。

这种潜力应该有更深层的原因。众所周知，人工神经网络是受到人脑神经启发而来的一种模拟，停滞多年后爆发了深度学习革命。十几年前的这场革命给我们带来了一次又一次的震撼，这些亮丽的拟人或超人的智能表现，使我们不禁在想，人脑神经也许就是这么工作的，具有同样的原理。人工智能达到或超越人类智能，机理相同应该是最有力的一个支持（虽然脑科学总有人说，我们对大脑知之甚少，貌似有一种把大脑神经的工作机理神秘化的倾向）。其他的一切手段，例如符号逻辑的公式和算法、统计模型的 feature engineering，都不是从真正意义上模拟人脑神经的工作方式，也许这就是它们注定走不远的症结所在。甚至可以说，神经框架内的各模态信号编码嵌入后的高维向量空间的确就是上帝的语言，或宇宙信息的本质，而符号不过是迁就人类感官的表象而已，或者说是人类强加给世界的一种体系。

第二，当一种路线具有超出我们想象的潜力的时候，科技进步所需要的就是足够人才密度去挖掘这种潜力。

我们知道，AI 领域具有这种聚集天才的优势。无数年轻才俊、聪明的大脑被吸引到这个领域，其天才密度超过了大多数领域。

在这样两个条件都具备的时候，我们永远不要低估它突破各种瓶颈的内部力量。那些AGI（Artificial General Intelligence，通用人工智能）的信仰者们一直在疯狂内卷，同时也的确不断在交出自己的成绩单。他们可能是AI的终结者吗？

【外一篇：符号逻辑“沦落”为辅助工具还是有自己不可取代的智能补足作用的】

yanyongxin：人之所以区别于其他动物，是因为进化出了推理能力。这种推理能力虽然是建立在神经元上的，但跟单纯的“本能反应”有了质的飞跃。它有了对象关系的抽象，从而可以进行多步骤推理，从而可以以对象关系链——语言的形式传播和记忆。推理本质上是一个离散过程，因此是可以符号化的，是一个在神经元体系上建立出来的模拟系统。但这种模拟的可能，很可能要求神经元系统与其他动物有结构上的差异。

推理系统与原始神经元思维的最显著差异，就是可以长时间思考。而不是“肌肉记忆”、“直觉”。不同人之间的差异很大。大学期间，注意到有些学生遇到简单问题时反应很快，面对复杂问题时，给他多少时间也没用。现在的LLM大致达到了文科生的推理水平。但还与训练有素的理科生，尤其是数学、物理的理科生有差距。数理专业的特征，就是将现实世界的问题严格的符号逻辑化。要在人的神经元系统上模拟出如此严谨的符号逻辑系统是需要特殊的机会（好大学，好老师）、长期的训练、和特殊的脑结构（“天赋”）的，要遵循良好的规则化习惯。但达到这个能力，就可以与机械化的严谨逻辑系统接口了，可以使用 Mathematica 之类了。

这就是人工智能下一步的走向：在神经元上建立更好的逻辑模拟系统，直到能与纯符号逻辑工具无缝链接。人脑中逻辑模拟系统是一个高能耗、高错误率、高构建成本的系统。它的一大好处，是能够跟底层神经元系统无缝联结。

立委：说得很好。

与符号逻辑系统接口就是 tool use，例如，遇到复杂数学问题，不要用自然语言的思维链（cot）去强行推理来影响概率性推理结论，而是LLM自己写代码去 call Mathematica。现在这种 tool use 的能力被定义为 LLM-native agent 的根本特性（之一）。所以 这种能力也还是 from within。

所以 我们看到了清晰的演进路线：

1 传统统计模型人工智障 不理解自然语言
solution： LLM e.g. ChatGPT


2. pretrained LLM 缺乏推理能力
solution： reasoning LLM e.g. o1 / r1


3. reasoning LLM 缺乏严谨的符号逻辑能力
solution： LLM agent （interfacing symbolic tools）

yanyongxin: 传统统计模型之所以人工智障 不理解自然语言，是因为没有足够大的参量空间和足够复杂的数据结构，来容纳理解语言所必须的世界模型。

transformer架构及其注意力机制是主流GPT大模型的基础，特别特别重要。虽然这方面的科普和讲解已经汗牛充栋，但还是有很多朋友跟我说一头雾水或雾里看花。所以下决心写了三篇科普系列，试图做出我的理解贡献。

作为对主流AI有好奇心的同学，你可能早就听说过大名鼎鼎的大模型的基本框架 transformer及其“注意力机制”，觉得它们是高深莫测的概念。 你也许也读过那篇经典论文 Attention is all you need，但还是感觉云里雾里。别担心，这很正常，我们多数人都经过这个阶段！这篇论文确实有点“烧脑”，但它的核心逻辑其实并不复杂。

【相关】

• 经典论文 Attention is all you need：https://arxiv.org/pdf/1706.03762
• 立委科普：如何理解自注意力机制中的QKV分工？
• Transformer 和注意力机制简介
  • Does the New Reasoning Paradigm (Query+CoT+Answer) Support a New Scaling Law?
  • Technical Deep Dive: Understanding DeepSeek R1's Reasoning Mechanism in Production
  • DeepSeek 笔记：推理新范式 query+cot+answer 支持新的 scaling law 吗？
  • DeepSeek 笔记：R1 部署阶段的推理机制
  • DeepSeek 学习笔记：R1 部署阶段的推理机制
  • 从R1幻觉谈起，大模型幻觉是缺陷还是创意火花？
  • 推理强化学习是端到端的监督，推理过程的非监督
  • DeepSeek R1：《少年DS之烦恼》
  • 告诉李雪琴一个激发写段子灵感的秘诀：找deepseek R1
  • DeepSeek 风暴下看看它的论文
  • DeepSeek's R1 Paper: A Storm in AI LLM Circle
  • The Turbulent Second Chapter of Large Language Models: Has Scaling Stalled?
  • 大模型风云诡谲的下半场：scaling 失效？
  • DeepSeek_R1 paper

*Edited transcript from InfoQ's second DeepSeek series livestream featuring Dr. Wei Li, former VP of Engineering at Mobvoi's Large Language Model team, discussing R1 Zero's innovative contribution to democratizing reasoning models.*

DeepSeek's Greatest Achievement: Making Everything Transparent

InfoQ: "DeepSeek adheres to a pure reinforcement learning approach, but the industry often refers to RL as 'alchemy' - how did they make this process controllable and accessible? What's innovative about their reasoning paradigm?"

Dr. Li:** The reinforcement learning for reasoning models has long been an industry challenge. About six months ago, when Ilya and others declared the end of the pre-training era, it signaled that simply scaling up pre-trained models was no longer sufficient for performance improvements. The delayed release of GPT-5 is another indicator of pre-training's decline. As a result, the industry began seeking new growth paths, with on-the-fly reasoning models gaining momentum among leading teams until OpenAI released O1, the world's first reasoning large language model. DeepSeek's R1 then followed with its breakthrough success.

From the mysterious Q-Star project (reportedly causing dramatic internal conflicts at OpenAI) to the release of O1, reasoning models have been widely recognized as a new paradigm in AI. The core of this paradigm is enabling models' "slow thinking" capability, or System 2 as it is called, using reinforcement learning to enhance model intelligence in complex tasks. However, all of this was closed-source. OpenAI even deliberately created mystique around their chain-of-thought content. Apart from a few top players like Google and Anthropic quietly exploring and tracking this field, other teams knew very little about it.

DeepSeek's greatest achievement lies in making everything about LLMs transparent. They open-sourced their models and detailed technical papers, and weren't afraid to expose their thought of chains (CoTs) in the system. Through pure reinforcement learning, they proved that even without process control data, result-based control alone could achieve top-tier reasoning model performance. This breakthrough was like piercing through a paper window, showing the industry a feasible path to democratizing reinforcement learning.

InfoQ: The innovation in reasoning paradigm sounds abstract. Could you provide an example?

Dr. Li:** R1's paper is outstanding, arguably one of the finest in the large model field. It consists of two parts: one focusing on Zero research, which presents remarkable achievements in pure reinforcement learning for reasoning; the other detailing the practical R1 system, a top-tier production reasoning model. For R1's development, they considered practicality, balancing comprehensive performance, safety, and various practical considerations, detailing a four-stage training pipeline as best practice to help other teams understand and replicate their success.

The most brilliant part is the Zero research. Zero proved a revolutionary point: contrary to traditional beliefs (or OpenAI's implied stance that reasoning requires step-by-step supervision), process supervision isn't actually necessary. Using only the final result against the "gold standard" as a supervision signal is sufficient to train the "slow thinking" process required for reasoning models.

This is Zero's greatest highlight and the origin of its name - it draws inspiration from AlphaZero's spirit. AlphaZero historically pioneered complete independence from human game records or experience, achieving zero human supervision reinforcement learning through self-play generated process data (state+move+score triplets). Similarly, DeepSeek's Zero research demonstrates that in reasoning tasks, models can autonomously generate internal process data - Chain of Thought (CoT) sequences - without human annotation.

Specifically, reasoning models initially focused on mathematics and coding because these domains have standard answers. Macroscopically, this is typical end-to-end supervised learning, as both input (math/coding problems) and output (answers/execution results) are fixed and known. However, the process from input to output is highly complex with significant information gaps, requiring a CoT bridge. Just as humans need to break down problems and think step by step when facing difficulties, models need this process too. DeepSeek's research found that models possess the ability to learn this deep thinking process autonomously if given sufficient time and space.

InfoQ: Dynamic reasoning paths sound like AI "drawing mind maps" - but how do you prevent it from going off track? Like suddenly writing poetry while coding?

Dr. Li:** Based on current evidence, this possibility is virtually non-existent or negligibly low. Before DeepSeek published their results and research details, many were puzzled about this point: wouldn't deep thinking go haywire with only result supervision and no process supervision? Without large-scale reinforcement learning experiments, this was indeed a significant concern. It's like flying a kite - you're holding just one string while letting it soar freely, worried it might nosedive.

These concerns proved unnecessary. The reason it doesn't go off track is that all this reasoning reinforcement learning, including self-generated reasoning CoTs, is built upon existing top-tier models (like V3). These models have already mastered coherent expression through massive data learning. This coherence implies orderliness, which, while not equivalent to pure logic, prevents completely unreasonable deviations. It is observed that fluent human speech typically reflects organized thinking.

InfoQ: On another note, compared to OpenAI's O1, DeepSeek R1 has another notable highlight in applying reasoning CoTs to language generation and style imitation. Could you elaborate on this?

Dr. Li:** When O1 was released, everyone knew it demonstrated significant improvements in mathematics and coding abilities, as standard tests revealed higher performance levels. What people didn't realize was that this reasoning ability, or "slow thinking" capability, excels not only in domains requiring strict logical reasoning but can also shine in traditional language tasks.

By nature, language ability has been a strength of large models - everyone knows they generate very fluent text, more native than natives. By the time we reached models like 4o or V3, their writing was already quite smooth, seemingly leaving little room for improvement. However, when asked to write classical poetry or imitate Lu Xun's writing style, previous models fell short. R1 solved these challenges. From a social impact perspective, this is actually quite remarkable and particularly noticeable.

Honestly, not many people are deeply concerned about mathematics or coding, although we know coding is a major direction for the coming years and automated programming can change the world. Everything in IT ultimately comes down to software; the digital world is built on software. If software development can transition from manual coding to model-assisted or even model-autonomous programming, this will greatly increase productivity. While this is visible to all, it's not as intuitive for ordinary people who more often face tasks like writing compelling articles.

When R1's humanities capabilities were discovered, not just geeks or software developers saw the benefits of reasoning models - ordinary people were excited too. Suddenly, anyone could claim to be a poet, writer, advisor or philosopher - the impact was tremendous. This wasn't felt with o1, perhaps because OpenAI didn't realize or at least didn't focus on this aspect of reasoning models. But while working on code and mathematical reasoning, DeepSeek must have internally realized that this "slow thinking" mechanism could also significantly improve writing abilities, especially in classical Chinese.

Everyone knows Chinese data isn't as rich as English data, so while previous models could write beautiful English poetry, they struggled with Tang poetry. This might be because Chinese data was insufficient in quantity or quality, preventing models from learning adequately. We always felt this was unfortunate - models would sometimes rhyme correctly, sometimes not, sometimes add or miss characters, not to mention tonal patterns to follow. DeepSeek clearly put effort into this area; their data quality must be significantly higher than industry standards.  More significantly, they know how to transfer the CoT ability from science and technology to language and literature.

InfoQ: If you were to recommend a DeepSeek module most worth replicating for programmers, which would it be? Like those "Aha moments" claiming to replicate R1 for tens of dollars?

Dr. Li:** If I were to recommend a DeepSeek module most worth replicating for the programming community, it would be the Zero research-related components. This replication isn't about achieving comprehensive capabilities but rather verifying Zero research's key revelation - that machines can indeed autonomously learn. This is what OpenAI kept under wraps; perhaps they had figured it out earlier but chose not to disclose it.

Now, we've seen quite a number of different teams claimed to have reproduced R1's reflective capabilities with minimal resources. This isn't just an interesting experiment; more crucially, it marks the democratization of reasoning models. Previously, people didn't understand how reasoning models worked, only knowing that vast amounts of process data were needed for models to learn slow thinking. This was considered an almost insurmountable barrier because process data is hard to obtain, and reinforcement learning's instability and high data requirements confused and challenged many programmers.

But now, we know we can bypass this most difficult process data requirement and reproduce this "Aha moment" with limited resources, proving that slow-thinking capabilities can be learned autonomously by models. Based on this premise, if you're a domain expert, you might wonder: could these techniques achieve significant improvements in your field? This is entirely possible. Even the most powerful models (like V3 or 4o) only achieve 60-70% accuracy in specific scenarios without optimization, and experience tells us that without at least 80-85% accuracy, you can't launch a truly valuable system in real-life applications.

That is to say, between a large model's out-of-box results and actual valuable application deployment, there's a gap. Previously, our only method was collecting domain data for fine-tuning (SFT). Now, we have another path RL: following the reasoning model approach, letting systems fully utilize slow thinking capabilities during the reasoning phase to improve data quality to acceptable or even exceptional levels. This path seems to have been opened.

However, my programmer friends tell me that in their comparison experiments between fine-tuning (SFT) and DeepSeek-style reinforcement learning (RL), while RL indeed outperforms SFT, the computational cost for RL training is still far higher than SFT. The superior performance makes sense because SFT data is always very limited, while successfully reinforced RL self-generated data can far exceed SFT data volume.

InfoQ: Some say large models represent "brute force aesthetics," but OpenAI's former Chief Scientist and co-founder Ilya says pre-training has reached its limit. How do you view this? Is the emergence of reasoning models just adding another scaling law to brute force aesthetics?

Dr. Li:** This is more about a shift in technical focus and a paradigm change in technical innovation. Large models involve three major components: first, pre-training, which builds foundational capabilities by learning basic patterns from massive data; second, post-training, initially mainly fine-tuning - OpenAI early on used some reinforcement learning (like RLHF) for human preference alignment, but by Meta's time, they even abandoned typical PPO style RLHF for simpler DPO, as they, like many others, struggled with it. Finally, there's the reasoning phase, where models interact with users real-time after deployment.

The current situation with high-quality natural data is that pre-training has nearly exhausted all available quality resources. The industry began to notice data growth challenges, making performance improvements increasingly difficult. GPT-5's delayed release, reportedly yielding limited returns despite massive computational investment, suggests pre-training may have indeed hit a wall.

This led the industry to explore alternative AI growth curves. Reinforcement learning-based reasoning models emerged at center stage in this context: pure reinforcement learning should be added to post-training. Previous reinforcement learning relied on human preferences, but this time it's about giving models more thinking time before reaching answers, learning underlying chain of thought (CoT). While V3 was already doing well, it didn't cause as much social sensation until R1 appeared. DeepSeek truly broke through after the Chinese New Year, becoming the most discussed public topic and causing excitement and shock overseas. R1 and O1 represent a new paradigm. Before R1, only OpenAI's O1 existed as a reasoning model, seemingly unreachably advanced, with would-be-followers unsure how to follow. However, R1 not only reproduced O1's capabilities but did so with greater transparency and clarity. This contrast further highlighted R1's importance as an open-source model leader.

InfoQ: At first glance, DeepSeek seems like an engineering masterpiece. Why did it cause such global sensation? Its user acquisition speed (100 million in a week) surpassed ChatGPT's nuclear moment? What's its historical significance?

Dr. Li:** From my personal experience and observation, ChatGPT's explosion was a landmark event in large model development. Research insiders were following large models before ChatGPT, at least since GPT-3. When GPT-3's Playground appeared, we were already immersed in it, sensing an approaching storm. But from society's perspective, ChatGPT truly shocked everyone, exceeding all expectations, like an AI nuclear explosion.

I believe R1's emergence is the second major shock after ChatGPT. Of course, between ChatGPT and R1, other influential models appeared, like 4o - another remarkable milestone. While ChatGPT 3.5 was already so impressive, 4o proved it could be even better. Then came Sora, bringing shock with video capabilities in multi-modal LLMs. I personally also greatly appreciate Suno, the music model, making me feel like I could become a musician overnight.

If I were to rank them, R1's impact is second only to ChatGPT, perhaps even exceeding 4o and Sora's sensational effects. R1's impact feels similar to ChatGPT's initial appearance, creating the same addiction. While ChatGPT was groundbreaking and R1 a follower, albeit with innovative highlights sometimes surpassing previous models (like in classical poetry and style imitation), achieving such global impact as a follower is truly miraculous.

In terms of practical effects, R1's productization was amazingly successful. Gaining hundreds of millions of users in a week, it far broke ChatGPT's record and elevated society's AI awareness. Furthermore, regarding geopolitical influences on technology access, many domestic users had long desired access to the world's most advanced models like GPT series, Claude, or Gemini but often couldn't reach them. R1's appearance eliminated these concerns about domestic and international restrictions, contributing to its rapid global popularization.

InfoQ: What's your vision of AI programming's ultimate form? Is it programmers telling AI "make me a TikTok," and it outputs deployable code and operations plans?

Dr. Li:** There are always two types of people: skeptics and optimists. People like Ilya believe Artificial General Intelligence (AGI) is imminent and Artificial Super Intelligence (ASI) isn't far away, so the biggest concern now, according to him, is ensuring superintelligence safety.

Anthropic's CEO Dario predicts that within 3-5 years, large models will achieve real breakthroughs - not just the current impressive demonstrations, but revolutionary changes in societal productivity. Fundamentally, they're talking about AI's ability to scale replacement of both physical and intellectual human labor.

However, while large models are buzzing now, their practical applications haven't reached the level of the previous generation's mobile platforms. Previous super apps like Meituan, Didi, Xiaohongshu, and TikTok transformed major aspects of our daily lives, from basic necessities to communication and entertainment, maximally shortening the distance between suppliers and customers - value everyone of us feels daily. While playing with large models is interesting, their practical value at the lifestyle level isn't yet obvious;  at best we're still on the verge of the coming AI application explosion.

Notably, DeepSeek's emergence has lowered large model application barriers, paving the way for scaled applications, though we haven't yet entered the era of true application explosion.

What will it look like when AI applications truly explode? I believe the ultimate goal, by nature of AI, is for LLMs to comprehensively replace humans in both intellectual and physical labor. Signs of large models impacting white-collar workers are already undoubtedly evident, with even programmers not an exempt. In physical labor, embodied intelligence is developing rapidly, with both humanoid robots and mechanical hands gradually replacing human physical work.

Of course, this brings side effects, like massive job displacement. How society adapts to this state of greatly developed productivity, but this is another discussion topic. But looking at AI's nature and ultimate goals, AI development could have two milestones: first, when it can replace 50% of human work, allowing half of society to maintain a decent, free life through social programs perhaps like Universal Basic Income (UBI) - this might mark the arrival of AGI (Artificial General Intelligence); second, when it replaces 90% of human work, possibly signifying the emergence of ASI (Artificial Super Intelligence) - a kind of technological utopia (or 'communism') in some sense.

These two milestones are my own verifiable definitions of AGI and ASI.  I do not agree with the idea that while old jobs are replaced, more new jobs will be created by AI.  It just does not make sense as any new jobs are also a mixture of human labor, destined to be replaced soon by super intelligence if they do emerge for time being.

This vision of AI's future development shows how DeepSeek's innovations in reasoning models might be just the beginning of a much larger transformation in how we think about work, society, and human potential in an AI-driven world.

【相关】

• DeepSeek爆火真相：不靠“人盯”， 让AI自己学会慢思考

• Does the New Reasoning Paradigm (Query+CoT+Answer) Support a New Scaling Law?

• Technical Deep Dive: Understanding DeepSeek R1's Reasoning Mechanism in Production

• DeepSeek 笔记：推理新范式 query+cot+answer 支持新的 scaling law 吗？
• DeepSeek 笔记：R1 部署阶段的推理机制

• DeepSeek 学习笔记：R1 部署阶段的推理机制

• 从R1幻觉谈起，大模型幻觉是缺陷还是创意火花？

• 推理强化学习是端到端的监督，推理过程的非监督

• DeepSeek R1：《少年DS之烦恼》
• 告诉李雪琴一个激发写段子灵感的秘诀：找deepseek R1

• DeepSeek 风暴下看看它的论文
• DeepSeek's R1 Paper: A Storm in AI LLM Circle
• The Turbulent Second Chapter of Large Language Models: Has Scaling Stalled?
• 大模型风云诡谲的下半场：scaling 失效？
• DeepSeek_R1 paper

LLM的"大就是好"还能走多远？

——关于Scaling Law的一些思考

老友张俊林《从Deepseek R1看Scaling Law的未来》一文，引起老友热议。

白老师的推荐抽提是：

核心观点：

——无限堆叠不会无限增长（物理世界规律也不支持），只有S型增长，一个S曲线一定会出现边际效益递减。

初期语言能力的涌现，与语料数据中包含的语言知识密度有关。

接下来的增长实际上是第二根S型曲线，更多语料贡献的是常识性知识，常识知识密度不及语言知识密度，所以要更大算力才能涌现。

再接下来是逻辑知识（思维链）的学习。自然语料中逻辑知识密度更低，用算力野蛮淘金，吃力不讨好。所以，用逻辑知识密度更高的合成数据做强化学习，才能让第三个S曲线爬坡。这就顺理成章了。

鲁总评论说：Scaling Law 这个词现在有点滥用。S 曲线（Sigmoid函数刻画的非线性曲线）倒是可以描述技术的生命周期，但它往往是一个接下一个（一个技术遇到瓶颈，往往才有另一个技术的开始）。。。这个在ChatGPT刚出来时我们回顾过。大模型的这几个 "Scaling Laws" 也印证这一点 （Test-Time 和 Post-Training “Scaling Laws" 有点重叠部分）：

创新就是从一个S曲线到另一个S曲线，well known results。这也是斯坦福大学那位鼓吹新能源、自动驾驶以及再生食品革命等科技乐观主义的教授（叫？）每次演讲必谈的技术革命的adoption曲线。他自称根据这个曲线，他在过去30年对于技术影响社会的许多预见都证明是对的，虽然每一次预见社会都会取笑他。 

回到LLM领域的 scaling law 话题。Scaling law本质上是一种经验法则，而经验告诉我们，大多数经验法则都符合S形曲线（或增量的正态分布）。具体到LLM，"大就是好"正是这种法则在遇到平台期或天花板之前的体现。这里的"大"指的是数据规模大、模型参数量大，缺一不可。模型规模不够大，数据再多也无法有效消化——这早已是业界共识。不过OpenAI早期的设计中过分强调模型规模的做法现在看来是一种误导，直到Chinchilla Scaling Law的提出，业界才形成了更合理的共识：数据规模和模型参数量需要保持适当的比例关系。

LLM Scaling的底层逻辑是什么？

首先要明确：LLM不是数据库，其目标不是记忆长尾数据的细节。大模型训练本质上是对大数据内容的压缩，换句话说，压缩的是数据背后的知识体系（包括常识、百科知识等），重点在于揭示大大小小的各种规律性（也就是所谓的泛化能力，generalizations）。

一般直觉会认为，数据规模越大，冗余也越多。无论如何过滤清洗和去重，冗余度随规模增长，似乎意味着可榨取的"油水"会越来越少。那么为什么到了千亿tokens这种以前难以想象的数据规模，大模型依然显得"吃不饱"？为什么从千亿扩展到万亿tokens，scaling law依然有效？

这个现象的关键在于LLM是序列学习（编码）和序列推理（解码）的系统。序列本身是一维的，但序列中蕴含的patterns和规律性却是高维的。举个例子：即使是简单的"猫追老鼠"这样的序列，背后可能涉及物种关系、捕食行为、空间运动等多个维度的知识。这种多维知识表现在序列层面，就会发生天然的组合爆炸。对大数据的"大胃口"正是应对这种组合爆炸的有效策略。只要不是完全的信息冗余，增加的不同序列对模型抽象数据patterns通常都是有帮助的。

然而，人类自然产生的高质量数据是有限的。预训练已经几乎吃尽了现有的高质量自然数据。于是，业界开始探索另外的AI智能增长曲线。

从预训练到推理：两个还是三个S曲线？

张俊林指出：

OpenAI o1推出后，另外两个阶段不再孤单，也各自拥有了姓名，产生了各自的Scaling Law，对应后训练阶段的强化学习Scaling Law(RL Scaling Law)和在线推理阶段的Inference Scaling Law(也叫Test Time Scaling Law)。

这里值得探讨的问题是：到底是三个S曲线，还是两个？推理模型的S曲线与此前的预训练S曲线有多大可比性？

理论上确实可以分为三个阶段：

1. 预训练
2. 后训练（尤其是推理强化学习）
3. 推理阶段

这三个阶段理论上都可能找到资源投入与性能提升之间的正相关S曲线，即scaling laws的某种表现函数。但实际上，在当前部署的应用中，后训练和推理这两个阶段应该共享同一个S曲线，原则上不存在两条独立的增长曲线。

当然，如果用户利用提示词技巧来影响模型的test time，让它更深入的思考，这可能间接影响 CoT （ChainOfThought）的长度或深度。但那是 query 的改变，是 input context 的变化，感觉也不应该算作 test time compute 的独立的 s曲线。

另外，说推理模型这一波潮流是范式转变，开启了新的 RL/Test-time scaling law，总觉得有一点太言之凿凿了。直觉上，推理模型的增长曲线与此前的预训练 scaling law 的增长曲线，大概率没有直接的可比性。

Scaling law 说的 law，实际上我们都知道是所谓经验“法则”。经验需要足够的实践数据积累，才能总结出来。强化学习赋能的推理模型才刚开始，没有足够的经验数据刻画这是怎样的一种增长关系，能持续多久，是不是昙花一现，还是可以持续相当长的时候，等等。

持续时间不够长的 scaling，其实没有多少经验法则的意义。Anthropic CEO Dario 提到 deepseek 的时候说（大意）， deepseek 显得这么亮眼其实是赶上了好时机，言下之意是运气的成分大于技术硬核实力和创新（滑稽的是，Anthropic 迄今没有能力推出任何推理模型，虽然R1以来，谷歌和国内都有推理模型的上线）。他说，推理刚刚开始，所以任何人走通了这条路，在这个初期阶段都会有一个大增长。譬如PhD段位的考试题，在没有推理模型的LLM中，可能分数很低，但一旦有了推理模型，有了所谓 test time compute 的 CoT，成绩就会直线上升，给人创造了奇迹的感觉。

现在是推理模型的早期，后去会如何呢？靠增加 test time compute，或不断延长 CoT，还会有多少增长空间？这个问题是现在进行时，貌似没有明确答案。但隐隐觉得，这个持续增长的时间或曲线，远不如预训练那样稳定和持续，进而其作为 scaling law 的说法不一定站得住。

这第二条反映 RL scaling law 的后训练智能增长曲线，不大好与 pretrain scaling law 相提并论，很可能并不是可持续的，也可能很快就遭遇制约因素的强烈反弹（见后“Test Time Compute 的制约”）。

DeepSeek R1的启示：慢思考的真相

以DeepSeek R1为例，用户可以选择"deepthink"模式来启动慢思考的chain-of-thought（CoT）推理，但实际上用户难以通过增加计算时间来提升推理质量。这是为什么呢？

让我们看一个具体例子。假设我们让R1解决一个复杂的数学问题：

- 传统模型可能直接给出答案："结果是42"
- R1会展示详细的推理过程："让我们一步步思考：1) 首先考虑...... 2) 然后我们可以...... 3) 最后得出结果42"

表面上看，R1的回答展现了"慢思考"（CoT）的特征，但实际上这个推理过程是模型在训练阶段就已经固化的生成模式，而不是在回答问题时动态探索多个可能的推理路径。换句话说，CoT+answer 看似是"慢思考"后的回答，但其实并不改变自回归 ntp（next token prediction）的单向序列生成定式。说白了就是，R1 的 cot+answer 给人慢思考的样子，但生成的本性还是GPT“快思考”范式。在 test time，思考的深度和规模不是动态探索，虽然可以用 beam search 进行内部的隐式多路径选优。

Test Time Compute 的制约

目前业界热议的"test time compute"，指的是含有CoT机制的推理模型相比传统的非推理模型需要更多的在线计算资源。以V3/R1为例，处理同样的问题，启用CoT 的R1可能需要V3 n多倍的计算时间。但这种计算量的增加是模型训练后固化的行为模式导致的，而不是可以动态调节的算力投入。test time compute 没有可控的伸缩可能性，也就谈不上 test time scaling law。

预训练与后训练的CoT强化学习的一个很大的不同是：预训练 scaling law 可以长期稳定乃是因为一旦训练完成，不大影响在线响应的时间，生成模式就是简单 query+answer。因此预训练阶段离线训练几个月都是可以忍受的，只要训练出来的大模型能力有大的提升。但推理模型后训练阶段的CoT强化学习不同，它在培养模型在线回应慢思考的习惯，生成模式是 query+cot+answer。推理模型的 cot 拉长，不仅仅是训练的资源和时间的耗费问题，更主要的是它反映在部署推理阶段的 test time compute 的延长，严重拖延了系统的响应时间。而用户在线使用系统的时候，一般来说对于慢思考的计算量和耗费时间是有能够忍耐的上限的。

这就带来了一个关键问题：即使研究表明indefinitely 增加CoT的长度（相应增加在线计算时间）能带来持续的性能提升，符合某种 scaling law 的经验法则，这种增长也会受到推理阶段现实因素的制约。一般用户可能愿意等待5-10秒获得更好的答案，但如果需要等待几分钟乃至几小时，使用体验就会大打折扣，乃至不可接受。

Scaling Law的可持续性之辩

Open AI CEO Sam Altman 和 Anthropic CEO Dario 这些大佬可能会争辩说，对于极其复杂的问题（如证明黎曼猜想、设计下一代航天战机等），即使模型需要一周的计算时间，相比人类团队需要数十年的工作量仍是极大的进步。但这种论述有两个问题：

1. 这类超复杂问题的LLM可行性远未得到验证
2. 极端场景不具有普适性，难以作为可持续的scaling law 的数据点

当然，这并不是否认S曲线作为描述scaling law的有效模型，也不是否定S曲线叠加的合理性。预训练和后训练两个阶段的增长曲线（s1和s2）叠加确实可能反映了资源投入与性能提升的整体关系。但我们需要谨慎看待CoT推理是否开启了一个真正可持续的scaling曲线。

结语：通向AGI的道路还有多远？

如果推理模型的scaling law缺乏可持续性，这就带来了一个更深层的问题：仅依靠这两个scaling laws，我们能否达到通用人工智能（AGI）的理想彼岸？更进一步，让AI平替人类劳动、极大提升生产力的超级人工智能（ASI）的技术理想是否真的可行？

目前的证据表明，预训练scaling law确实展现了相当的持续性，但推理模型的scaling law可能会较快遇到现实约束。这提醒我们，通往AGI/ASI的道路可能需要更多的创新突破，而不仅仅是现有方法的简单外推。在人工智能发展的下一个阶段，我们或许需要寻找全新的增长曲线。

【相关】

张俊林：从Deepseek R1看Scaling Law

• DeepSeek 笔记：推理新范式 query+cot+answer 支持新的 scaling law 吗？
• DeepSeek 笔记：R1 部署阶段的推理机制

• DeepSeek 学习笔记：R1 部署阶段的推理机制

• 从R1幻觉谈起，大模型幻觉是缺陷还是创意火花？

• 推理强化学习是端到端的监督，推理过程的非监督

• DeepSeek R1：《少年DS之烦恼》
• 告诉李雪琴一个激发写段子灵感的秘诀：找deepseek R1

• DeepSeek 风暴下看看它的论文
• DeepSeek's R1 Paper: A Storm in AI LLM Circle
• The Turbulent Second Chapter of Large Language Models: Has Scaling Stalled?
• 大模型风云诡谲的下半场：scaling 失效？
• DeepSeek_R1 paper

1. 训练阶段的强化学习机制：GRPO是否通过生成多条候选答案（multiple candidate cot+answer sequences）进行策略优化（修改模型），使得模型养成慢思考的习惯？

这个答案是毫无疑问的 YES。

2. 部署阶段的推理机制：R1是否在生成时隐式生成多条路径，但仅展示一条？如果是，这种机制与集成（ensemble）方法有何异同？

3. 与AlphaGo的MCTS的区别：MCTS树搜索是否在推理时动态构建搜索树，而集成方法只是静态组合多个模型的输出？

1. 部署阶段的隐式多路径推理机制

在 DeepSeek R1 的部署阶段，其推理机制可以概括为以下两种模式：

(1) 隐式多路径生成与筛选

- 生成多条路径：模型在单次推理时，可能隐式生成多条潜在的推理路径（CoT+Answers），但仅选择其中一条输出。
- 技术实现：通过调整解码策略（如束搜索宽度 `beam_width`），模型在生成过程中维护多个候选序列（即多条路径），最终选择综合评分最高的路径。
- 用户感知：用户仅看到最终输出，但模型内部进行了多路径探索与筛选。
- 效率权衡：若设置 `beam_width=1`（贪心搜索），则退化为单路径生成，响应速度最快；增大 `beam_width` 可提升输出质量，但增加计算延迟。

(2) 显式多候选生成（需主动配置）

- API级控制：通过设置 `num_return_sequences` 参数，模型可显式生成多个候选答案（如5个），用户或下游系统可进一步筛选。
- 实际应用：DeepSeek App默认未开放此功能，但在企业API或开源代码中可能支持。

关键点：
- 训练阶段的强化学习优化了模型的“单路径CoT生成能力”：通过GRPO训练，模型在单次生成时即可输出高质量的详细推理步骤（长CoT），无需依赖显式多候选生成。
- 部署时的多路径探索只是“锦上添花”：隐式多路径（如束搜索）或显式多候选生成可进一步提升输出质量，但非必需功能。

2. 训练阶段的“慢思考习惯”培养

(1) 强化学习的作用

- 目标：通过GRPO算法，模型学习生成更详细、更合理的推理步骤（长CoT）以提高奖励（如答案正确性）。
- 机制：训练时生成多个候选答案，奖励信号不仅评估最终答案正误，还隐式鼓励逻辑连贯的推理路径（如通过格式奖励）。

(2) CoT增长的驱动力

- 奖励设计：若长CoT更易得出正确答案（如分步解题减少错误），模型在策略优化中自然倾向于生成更长、更详细的步骤。Given room for [think], a reasoning model just wants/tends to think deep!
- 数据反馈：训练后期通过拒绝采样生成的高质量SFT数据，进一步强化这一模式。

结果：训练后的模型在单次生成时即可输出高质量的详细推理（即“慢思考习惯”内化）。

3. 与集成方法（Ensemble）的异同

(1) 相似性

- 多路径生成：隐式多路径探索（如束搜索）可视为同一模型生成多个潜在输出，类似集成方法中的多模型预测。
- 结果筛选：通过置信度选择最优解，类似于集成中的投票或加权平均。

(2) 核心差异

R1的隐式多路径生成本质是单模型内的动态解码策略，而传统集成依赖多模型的静态组合，二者在实现成本与多样性来源上存在根本差异。

4. 与AlphaGo蒙特卡洛树搜索（MCTS）的本质区别

(1) AlphaGo的MCTS机制

- 动态规划：通过模拟（Simulation）构建搜索树，评估每一步的长期收益（如胜率），动态选择最优路径。
- 在线学习：在推理时根据实时反馈（如对手落子）调整搜索策略，部分版本（如AlphaZero）甚至更新模型参数。

(2) R1的隐式多路径生成

- 静态模型：部署时模型参数固定，多路径生成依赖预训练的策略与解码规则。
- 无长期收益建模：路径选择基于模型自身的置信度概率，而非多步决策的累积收益。

(3) 差异

- R1的多路径生成是静态策略的有限探索，依赖训练阶段内化的CoT+answer的生成能力。
- MCTS是动态规划过程，通过实时模拟与评估实现长期收益最大化，属于在线决策优化。

5. 总结

- 训练阶段的目标：GRPO通过强化学习培养模型生成详细CoT的习惯，使得部署时单次生成即可输出合理答案。
- 部署阶段的灵活性：系统可选择单路径生成（快速响应）或多路径筛选（质量优先），后者类似轻量级集成。

- 训练完成后模型参数确实固定，但隐式多路径生成依赖解码策略（如束搜索宽度），用户可通过API参数调整，非完全静态。
- 与集成的实质差异：R1的多路径生成是同一模型的不同解码路径，而传统集成依赖多个独立模型，后者多样性更高但成本激增。

- MCTS的核心是动态搜索与长期收益建模，而非多模型预测的平均化。R1的隐式多路径更接近贪心策略的扩展，而非规划过程。

DeepSeek R1的部署机制通过训练阶段的强化学习内化“慢思考”能力，使其在单次生成时即可输出详细推理。隐式多路径生成（如束搜索）可进一步提升质量，但本质是同一模型的解码策略优化，与传统集成或AlphaGo的MCTS均有显著差异。这种设计在效率与效果间取得平衡，适配大规模工业应用需求，但牺牲了动态规划与全局最优的能力。

【相关】

• DeepSeek 笔记：推理新范式 query+cot+answer 支持新的 scaling law 吗？
• DeepSeek 笔记：R1 部署阶段的推理机制

• 从R1幻觉谈起，大模型幻觉是缺陷还是创意火花？

• 推理强化学习是端到端的监督，推理过程的非监督

• DeepSeek R1：《少年DS之烦恼》
• 告诉李雪琴一个激发写段子灵感的秘诀：找deepseek R1

• DeepSeek 风暴下看看它的论文
• DeepSeek's R1 Paper: A Storm in AI LLM Circle
• The Turbulent Second Chapter of Large Language Models: Has Scaling Stalled?
• 大模型风云诡谲的下半场：scaling 失效？
• DeepSeek_R1 paper

Host: Hello everyone! Welcome to today's interview. Recently, there's been quite a buzz about AI "hallucinations," especially with DeepSeek-R1, which seems to have a higher hallucination rate than its predecessor, DeepSeek-V3. Today, we're joined by Dr. Li, a senior AI researcher. Welcome, Dr. Li!

Dr. Li: Hello, host! Hello, everyone!

Host: Let's start with the million-dollar question: Why do large language models "hallucinate"? Can you break it down for us in plain English?

Dr. Li: You see, large language models are like super-powered conversation completers. Give them the first half of a sequence, say, a question, and they'll predict the second half (say, an answer) based on their massive knowledge network. They learn like our brains do – they can't remember everything word-for-word, so they compress and generalize, grabbing the gist and finding patterns.

Here's a fun contrast: Ask them "How tall is Yao Ming?" and they'll nail it because that's such famous knowledge, this data point is practically carved in stone in their memory (represented in the model's parameter weights). But ask them "How tall is Old Wang from next door?" and they're stumped because they've never met Old Wang! But here's the kicker – they won't just say "I don't know." So what do they do? They "make up" a reasonable height based on what they know about the range of human heights. That's a hallucination for you!

Host: Wow, that's some impressive guesswork! But isn't this kind of making things up pretty problematic?

Dr. Li: Not necessarily! In a way, hallucination is imagination (for better or worse) – it's where creativity lies! Think about it: all those great literary works, artistic masterpieces – aren't they all flights of fancy, products of imagination? If everything had to match reality closely, art would just be photography, and where's the fun in that?

You know, Yuval Harari makes a fascinating point in "Sapiens" – humans became Earth's dominant species precisely because we could "tell stories," creating myths, religions, nations, and money – things that don't physically exist. These are all "hallucinations," but they're the driving force behind civilization!

Host: When you put it that way, hallucinations sound pretty important! But let's talk about DeepSeek-R1. Its hallucination issue seems quite serious.

Dr. Li: Indeed, it is! The academic consensus used to follow OpenAI's view that reinforced reasoning would significantly reduce hallucinations. I remember discussing this with a head honcho at an LLM unicorn who was particularly excited about reasoning's potential to curb hallucinations. But R1's performance threw us a curveball!

According to Vectara's tests, R1's hallucination rate is more than 3 times higher than its foundation model V3's – 14.3% compared to 3.9%. This definitely correlates with its prolonged "Chain of Thought" (CoT) enabled by reinforcemnnt learning for reasoning. R1 is absolutely brilliant at reasoning, math and coding, as well as poetry and storytelling, but this currently comes with the "side effect" of increased hallucinations in things like translation and summarization.

More specifically, there are several reasons for R1's increased hallucinations.

First, the standard hallucination tests use summarization tasks, something base models are already pretty good at. In this case, reinforcement learning can backfire – it's like using a cannon to swat a fly!

Second, R1's reinforced reasoning chains weren't specifically optimized for straightforward tasks like summarization, translation, or news writing that demand strict factual accuracy. Instead, it tries to add various layers of thinking to every task. Looking at its transparent CoT (ChainOfThought) printout, we see it tirelessly analyzing even simple instructions from multiple angles. This overcomplication of simple tasks can lead to deviations and hallucinations.

During R1's reinforcement learning for NLP-related tasks, it seems the model was rewarded more heavily for creativity, leading it to be more imaginative – and consequently more prone to straying from facts. For mathematical and coding tasks, R1's supervision came from gold standards (test answers or code test cases). But for humanities tasks, they used V3 or V3's reward model to judge quality, and the current system seems to clearly favor creativity.

Moreover, user feedback typically tends to focus and encourage creativity. Most people aren't sensitive to hallucinations, especially when they're wrapped in the model's smooth, fluent language. For most frontline developers, this kind of user feedback naturally pushes them to enhance creativity rather than tackle the thorny problem of hallucinations.

Host: So, you are saying that R1's hallucination problem rooted in its over-enthusiastic reasoning? What's the real relationship between reinforced reasoning ability and hallucinations?

Dr. Li: It's still a puzzle – there's not seem to be simple correlation. Look at R1, a leading reasoning model, versus Claude 3.5 Sonnet, a top non-reasoning model. Surprisingly, Sonnet still has a higher hallucination rate than R1! But when we compare R1 to its base model V3, we see clearly that adding reasoning significantly increased hallucinations.

It may well be about the model's "personality." R1, with its powerful reinforcement learning, loves "divergent thinking." Give it a simple prompt, and it'll spin out ideas like there's no tomorrow – its CoTs could run on like crazy! This suggests that while R1 was powering up its creativity, it inevitably amplified creativity's twin: hallucination.

As a model that excels in both STEM and humanities, R1 performs differently across tasks. In mathematics and coding, where more rigorous reasoning is required, there's little room for hallucination. But in language and creative tasks, especially in the summarization tests, hallucinations become more prominent. It's largely a side effect of R1's supercharged linguistic creativity.

Technically speaking, R1 automatically adds lengthy CoTs to simple user instructions, essentially complicating straightforward tasks. Its CoTs (like  internal monologue of an entity following instructions) change the conditional part of the autoregressive probability model before generating answers, naturally affecting the final output. Compare:

V3: query → answer
R1: query+CoT → answer

For tasks that V3 already handles well, like summarization or translation, any lengthy CoT guidance might lead to deviation or embellishment, creating fertile ground for hallucinations.

Host: So where do R1's hallucinations mainly occur?

Dr. Li: Think of R1's abilities as split between "arts" and "sciences." In "science" areas like math and coding, its logic is fairly strong and hallucinations are relatively rare. But in "arts" areas like language, hallucinations become more noticeable.

R1's most impressive achievement compared to the first LLM reasoning model O1 is successfully extending mathematical and coding reasoning capabilities into creative writing, especially in Chinese. The internet is full of R1's brilliant literary works. In terms of wordplay and literary prowess, it clearly surpasses 99% of humans – even graduate students in literature and classical Chinese professors sing its praises.

But watch what happens when you ask it to do a simple summary – it can't help but "get creative," often "inventing" details not present in the original text. It's like its "arts" abilities are too powerful, a case of "too much of a good thing."

Host: That's an interesting perspective. Do all language tasks require creativity?

Dr. Li: Language tasks actually fall into two categories: ones that need high creativity, like poetry and fiction writing, and ones that demand high factual accuracy, like news reporting, translation, or summarization. R1 excels at the former, which was likely the development team's focus, but this creates side effects in the latter as it is today.

It reminds me of the old Chinese saying about translation needing to be "faithful, expressive, and elegant" – achieving all three has always been challenging. We see many examples where elegance is prioritized over faithfulness, like the use of hyperbole in literary works. We also see the opposite, like Lu Xun's advocacy for so-called "rigid translation."

Interestingly, humans have always had double standards here, but we have a mental switch we can flip at will. When watching movies or reading novels, we flip towards creativity and don't fuss about factual accuracy. But switch to news channels, and we have zero tolerance for falsehoods.

Host: People tend to believe content that appears logically coherent and detailed, so the potential harm from AI hallucinations could be significant. What should we ordinary folks do about AI hallucinations?

Dr. Li: While many people are starting to notice and become wary of these hallucinations amid their amazement at LLM's creativity, most are still mesmerized by its creative brilliance. We need to increase public awareness of AI hallucinations. I suggest a two-pronged approach:

Stay Alert: Don't take everything the model says as granted, especially factual claims. Hallucinations most commonly occur with names, places, times, locations, and other entities or numerical data.

Cross-Verify: For important details, check original sources online or consult experts to see if the claims align.

Guide the Model: When asking questions, add constraints like "please stay faithful to the original text" or "please verify facts." This can at times help reduce hallucinations.

Embrace Creativity: If you're looking for inspiration or creative ideas, model hallucinations can be a delightful surprise!

Think of AI hallucinations as "possibilities in parallel universes." What it makes up might not be true in our world, but could be true in another! It's like how novelists write fictions – while it cannot stand fact checking, it's a kind of "artistic truth." Just like novels arise from life but transcend it, AI arises from data but transcends it. AI compresses data into knowledge and common-sense network, not necesarily true to individual facts – that's what databases are for.

Host: This reminds me of what people often say: AI models aren't just "talking nonsense" – they're "talking nonsense seriously"!

Dr. Li: Haha, that's exactly it! AI hallucinations are its "educated guesses," based on the massive knowledge and patterns it's learned. The hallucinations are by noway completely random – they have internal constraints that make them seamless and convincing, but also more deceptive. Newcomers to AI need to be especially careful not to trust everything at their face value.

For regular users, understanding the nature of hallucinations is needed. For example, when asking about well-documented facts like "How long is the Yangtze River?" models won't make mistakes because these facts are firmly encoded in their parameters. But ask about an obscure creek or fictional river, and the model will activate its "reasonable completion" mechanism and make something up.

Host: Following your logic, human language itself prepares for a breeding ground for hallucinations.

Dr. Li: You could say that. Language enabled humans to create things which do not exist in the physical world, such as myths, religions, states, corporations, currency, and abstract concepts like ideals and beliefs. Harari emphasizes in "Sapiens" that story-telling (i.e. typical hallucinations) were fundamental to civilization: language enabled human story-telling abilities. Hallucinations catalyzed civilization. Humans are the only entities capable of 'lying' (besides LLMs).

Host: What about the future? Is there a way to maintain creativity while reducing hallucinations?

Dr. Li: This is definitely one of the "ultimate challenges" in AI! People are working on various solutions, including:

More Refined Training: During training, treat different types of tasks differently, teaching the model when to be strict and when to be creative.

Task-Specific Fine-tuning/Reinforcement Learning can help balance this contradiction. Tasks like summarization, paraphrasing, translation, and reporting need special care because they require both some creativity (like style) and strict factual accuracy.

Specifically, R1's training pipeline has four stages: fine-tuning 1, reinforcement 1, fine-tuning 2, and reinforcement 2. Reinforcement 2 mainly focuses on human preference alignment. Currently, this process seems to favor creativity over faithfulness, which could be rebalanced later. Perhaps more importantly, in stage three (i.e. fine-tuning 2), we could strengthen constraints for different tasks – for example, increasing supervised data for summarization to encourage faithful, straightforward results.

Routing: In the future, there will be a "model dispatcher" that assigns different models based on task type. Simple tasks could go to V3 or use tools, while complex tasks requiring deeper thinking go to R1.

For instance, arithmetic tasks should just use simple code calculations, equivalent to using a calculator. That's not how it works now – yesterday I tested a nine-digit multiplication, and R1 spent over three minutes thinking, producing CoT that could stretch down the street, breaking down the reasoning step by step. While the answer was correct, using such computationally expensive CoT for arithmetic instead of a simple function call is unreasonable. A one-line calculation code would do the job – no need to waste so much computing resource and tokens on explicit reasoning. These are foreseeable routing improvements, especially in the age of AI agents which can use all kinds of tools or applications. R1's CoT does not need to handle everything – besides hallucinations, compute-burning CoT is also not environmentally friendly.

Host: Thank you, Dr. Li, for this fascinating discussion! Today's interview has given us a much deeper understanding of AI hallucinations.

Dr. Li: My pleasure! It's been great chatting with you!

【相关】

• 从R1幻觉谈起，大模型幻觉是缺陷还是创意火花？

• 推理强化学习是端到端的监督，推理过程的非监督

• DeepSeek R1：《少年DS之烦恼》
• 告诉李雪琴一个激发写段子灵感的秘诀：找deepseek R1

• DeepSeek 风暴下看看它的论文
• DeepSeek's R1 Paper: A Storm in AI LLM Circle
• The Turbulent Second Chapter of Large Language Models: Has Scaling Stalled?
• 大模型风云诡谲的下半场：scaling 失效？
• DeepSeek_R1 paper

V3: query --〉answer
R1: query+CoT --〉answer

【相关】

Deepseek-R1 的幻觉率是 14.3% - 比其非推理前身 Deepseek-V3 高得多
榜单排名：https://github.com/vectara/hallucination-leaderboard

• 推理强化学习是端到端的监督，推理过程的非监督

• DeepSeek R1：《少年DS之烦恼》
• 告诉李雪琴一个激发写段子灵感的秘诀：找deepseek R1

• DeepSeek 风暴下看看它的论文
• DeepSeek's R1 Paper: A Storm in AI LLM Circle
• The Turbulent Second Chapter of Large Language Models: Has Scaling Stalled?
• 大模型风云诡谲的下半场：scaling 失效？
• DeepSeek_R1 paper

【相关】

• Understanding the Power of Chain of Thought
• 推理强化模型中思维链的本质

• 推理强化学习是端到端的监督，推理过程的非监督

• DeepSeek R1：《少年DS之烦恼》
• 告诉李雪琴一个激发写段子灵感的秘诀：找deepseek R1

• DeepSeek 风暴下看看它的论文
• DeepSeek's R1 Paper: A Storm in AI LLM Circle
• The Turbulent Second Chapter of Large Language Models: Has Scaling Stalled?
• 大模型风云诡谲的下半场：scaling 失效？
• DeepSeek_R1 paper

DeepSeek R1 的出圈是近来最大热度的焦点。它不仅在数学、代码等强推理能力上追平了 OpenAI 头部推理模型 o 系列，而且在语言文字的创造力和模仿力方面产生让人惊艳的效果。尤其是在中文（国学）的能力方面，大家都体会到了奇迹般的能力跃升。

这一切都要感谢推理强化的 CoT（思维链）。CoT 为什么这么有效，这么神奇，文理通吃，在强化中最大化了其赋能作用呢？

应该主要是因为 CoT 是从大模型自主生成出来的 tokens，它有效降低了从 query 到 answer 的 perplexity（困惑度），好比是为高质量结果提供了一个桥梁。看过CoT输出的同学都有体会，桥梁本身并不总是精彩，常常给人的感觉是小题大作，瞻前顾后、啰哩啰嗦、信息冗余，但精彩的answer却可以借助它面世。从第一性原理看，这里似乎涉及perplexity在信息论中的深刻含义。

CoT的本质

1. 从信息论角度理解：

• CoT是在high-perplexity的query和answer之间搭建的低熵通道
• 通过逐步分解，每一步的条件概率都变得更趋"自然"和丝滑，符合语言模型的本性
• 最终让看似"跳跃"的推理结论，变成了一系列"小步走"的累积

2. 用信息熵的视角来看：

• 对于复杂问题，直接从query到answer需要跨越很大的信息鸿沟，这“迫使”模型以幻觉应急，胡乱输出一个 answer
• 而CoT的每一步都在降低局部的条件熵
• 就像把一个大的信息压缩/解码任务分解成多个小的压缩/解码任务

3. 这解释了为什么"平庸"的CoT也如此有效：

• 它的作用不在于过程步骤的内容本身有多精彩
• 而在于它创造了一条信息熵递减的路径
• 模型可以沿着这条路径稳定地向目标迁移

4. 这也解释了为什么DeepSeek的训练如此有效：

• 不是教会模型更"聪明"的推理，这在文科类任务中根本就无法定义
• 而是优化了它构建这种低熵通道的能力
• 本质上是优化了信息流的路径规划

这个观点为理解CoT提供了一个新的视角，把表面上的思维链“自言自语”诠释为信息论意义上的"熵减（entropy reduction）途径"。这对无须过程监督的结果驱动的强化学习是一个合理的解释：

过程很重要，但过程监督并不重要，因为大模型本身自然产生的过程数据，比任何人类监督更加实用和可行。

【相关】

• 推理强化学习是端到端的监督，推理过程的非监督

• DeepSeek R1：《少年DS之烦恼》
• 告诉李雪琴一个激发写段子灵感的秘诀：找deepseek R1

• DeepSeek 风暴下看看它的论文
• DeepSeek's R1 Paper: A Storm in AI LLM Circle
• The Turbulent Second Chapter of Large Language Models: Has Scaling Stalled?
• 大模型风云诡谲的下半场：scaling 失效？
• DeepSeek_R1 paper

**立委列传**

立委者，皖南布衣也。少敏而多奇志，值鼎革之际，年十六躬耕于皖南山野。当是时也，少年负耒耜，涉深涧，虽陟彼崔嵬，犹自诩"吾辈亦开天辟地者也"。或问稼穑之苦，笑曰："此非太史公游侠列传之遗风乎？"

越二载，风云骤变。丁巳岁杪（1977），科举重启，立委束草为笔，削竹为简，竟跃龙门而入泮宫。同窗多皓首穷经者，独其弱冠青衿，谈笑间尽吞云梦八九。既卒业，执教乡塾，然鸿鹄岂栖枋榆？遂再赴春闱，北上太学，俨然贾谊入洛，苏秦过梁。都门烟柳，尽化笔下风云。

居长安四五年，见中关村商贾云集，乃效猗顿之术，夜筹算于灯下，朝执契于市廛。《万元户》所志，非夸朱提之富，实叹青蚨之诡也。然其性本楚狂，终随浮槎西渡，观泰西之变。英伦雾锁，野犬吠陌，立委抚剑长叹："此非管仲所云'仓廪实而知礼节'者乎？"遂北涉北海，徙居枫叶之国。

加北美地，雪沃千里。立委筑室于温哥华，添弄瓦之喜，修稷下之学。然冰原虽净，难栖南溟之鹏，乃振翅南徙，直入硅谷热土。当是时也，美利坚网络勃兴，立委运筹于虚牝之间，决胜于光纤之末。然泡沫既破，诸子百家尽墨，独其抱残守缺，犹存鲁壁遗经。

太史公曰：余观立委浮沉，似见张骞凿空之影。其插队如苏武牧羊，跳门若终军请缨，下海类范蠡泛舟，洋漂近玄奘取经。然则古之行者，皆为觅桃源而往；今之浪者，却在铸桃源于途。至若"海龟"之惑，实乃庄生濠梁之辩——子非鱼，焉知归与不归皆逍遥耶？昔陶潜采菊，阮籍哭途，各成千古风流。今立委以四海为注，掷骰于地球棋枰，岂非新时代之"逍遥游"乎？

**《立委奥德赛》**

*序章*
人生是旅者暂居的客栈，而漂泊者开辟的道路却蔑视时间本身。在立委的奥德赛中，漂泊行为成为了一种天体导航——一场词语筑造堡垒、思想绘制航路、时代潮流既是敌手又是盟友的旅程。

**土壤中的根系**
十六岁那年，来自东方山谷的少年以农夫的锄头交换了青春的闲梦，攀上雾气笼罩的山峰，在那里野心生根发芽。他的《插队日记》（后镌刻于《朝华》中）呼吸的不是绝望的挽歌，而是将风暴驯化为耳语的节奏。当命运的龙门在1977年吱呀开启时，他乘着复兴学术的疾风，加入了神话般的"77届"——从灰烬中重生的心智凤凰群。

**墨水的朝圣**
学者袍甫加身，北方的狂风便再度召唤。在《考试十四行诗》与《不安者的箴言》中，野心的狂热冷却为精密的文字工艺。首都熔炉中的五个寒冬将卷轴锻造成账本；他的《学者商人愚行录》记载着染墨手掌清点铜钱的故事。然而不息的潮水将他向西牵引，加入追梦者的出埃及记，奔赴阿尔比恩传说之岸。

**暗影与圣所**
在阿尔比恩的花岗岩天空下，流浪犬在卵石巷中嚎叫预兆——这种不谐之音被收录于《都市暗影兽典》。不安孕育翅膀：他北逃至枫叶王国水晶般的荒野。《北极星颂》咏唱边疆的纯粹；《港湾牧歌》编织炉火点燃的传说；《蜜饯编年史》追念为人父的喜悦。但圣所亦渐脆弱。他再度南翔，被吸引至硅谷炽热的坩埚。

**电路中的普罗米修斯**
在数字黎明的白炽光芒中，他的《创投诗章》燃烧着普罗米修斯之火——将初创企业视为伊卡洛斯之飞的现代神话。然而蜡制翅膀终将融化；《泡沫挽歌》与《陨落者寓言》测绘出野心的残骸。从冻原到热带，每个足迹都渗入墨水：用羽笔刻写的流放地图。如今作为硅谷常驻哲人，他书写《乡愁算法》——一段让海龟游弋于电路间的代码，低语着被遗落的潮汐。

*结语*
古代圣贤追寻九重天外的地平线；立委的奥德赛将漂泊刻入生命的重写本。他的根系紧抓插队之土；躯干穿越龙门攀升；枝桠扭曲成语义星座。我们若非活着的羊皮卷，又能是什么？编年史家的最终港湾仍未书写——那是海水消融于天空的地平线，所有罗盘疯狂旋转之处。让漂泊者的悖论永续：要测绘无限，就必须永不停漂流。

**七律·跃龙门**
十六荷锄云壑深，忽闻禹甸启春闱
青衫夜淬书窗月，赤榜朝分阡陌晖
两度鲤腾惊皖水，九重鹏举叩燕扉
都门烟柳催征铎，笑指星河是钓矶

**水龙吟·浮槎记**
少年曾缚苍龙去，又驾仙槎西渡。泰西雾锁，枫邦雪沃，硅台电舞。算尽青蚨，织成云网，几番寒暑。叹庄生蝶梦，陶公菊径，都付与、天涯路。

谁解飘零最苦？把乡愁，酿成新赋。南溟鹏翼，北山薇蕨，东篱菊圃。柯烂樵归，橘洲星换，武陵人语。待重拈汉瓦，摩挲秦篆，写沧桑句。

**古风·浪者吟**
我本谪仙人，偶堕红尘网
皖南锄晓月，燕北枕书幌
中关试鱼服，英伦辨魍魉
枫雪淬冰魄，硅火铸新掌
五洲棋局残，双鬓星霜长
欲唤云间鹤，蓬莱舟已枉
且抱地球仪，笑指乌托邦
归去来兮辞，翻作浪人唱

**临江仙·生涯注**
若把浮生标语义，节点最是漂流。龙门二度跃神州。商潮翻雪袖，硅谷试吴钩。

四十年来家国梦，都成异域春秋。键盘敲碎古今愁。回车新世界，空格旧沙鸥。

**摸鱼儿·流浪辩**
问苍冥、谁司行止？安排萍迹如许！鹏抟鲲徙寻常事，偏说此身无主。君看取：皖山月、燕台柳、硅谷霓虹柱。星槎暗度。纵填海精禽，射阳奇士，未解浪游苦。

休重论，苏武节旄汉土，范蠡舟泛烟雨。桃源只在鸿蒙外，何必武陵深处？敲键语：比特海、云端路、皆是逍遥浦。归兮且住！待地球仪停，时空键锁，方见真吾处。

《原朝华：立委小传》

人生苦短，掐首去尾，不过三五十年。大体分为三段：创业阶段（而立之年），成熟阶段（不惑之年）和下滑阶段（天命之年），反映在称呼上，叫小李、大李和老李。可怜，立委却从小李一跃到老李，没有机会品尝壮年人生的豪情，心尝有戚戚焉。

红小兵立委（1966） （《朝华午拾：永做毛主席的红小兵》）

自幼儿园到小学连跳两级，立委在班上始终最幼。更加荒年生人，孱弱矮小，体育课常告病假，或遭遣送回家，始终是个小可怜儿。所幸中学伊始，正值“修正主义回潮”，先帝启用邓公收拾文革残局，邓公责成教育总管周荣鑫整顿学校，校风日新。乘此东风，立委崭露头角，以学习委员兼数学科代表之身，受班主任委托，每日早自习登台主讲，演示解题思路，俨然助教。但好景不长，先帝昏庸，文革派重居上风，学校大乱，文化课退居后台，大批判遂成主课，兼以学工学农学军。立委不能以文化课呈威，然风头不减反盛，盖因立委最长批判文字，历经批林批孔，批邓反击右倾翻案风，直至批四人帮。大会小会，凡立委发言，必抑扬顿挫，铿锵有力，佐以诙谐幽默，风靡校园，称颂于一时。有传言，立委颇具鲁迅遗风，入木三分，且能推陈出新，妙语连珠。露天千人大会，常嘈杂狼藉，然立委登台，全场必静肃，洗耳恭听之，听至妙处，笑声一片。立委由此炼得糊涂胆大，从不怯场，终身受益。

及至大学，文革后首届，立委仍居尾，同学长一到十多岁不等（《朝华午拾：我的考研经历》）。同学之间皆直呼其名，唯同桌七仙女戏称 “小立委”，不为亲热，却为避嫌，以示划清界限。同桌四载，楚河汉界，泾渭分明。授受不亲，避而远之。然仙女文具笔墨滑落在地，自有立委抢先一步，拾拣归案。类此者三，春风化雨，润物无声。七女天生聪颖，想出一招，以长立委一岁为由，呼 “小立委”，就此来往，当可名正言顺也。

由七仙女开此恶例，随后多年，“小”字即不离身。中学教书，人称小李老师（22岁）。上研究生，小李出入机房，蓬头垢面，且口中念念有词，言“世界之语”（Esperanto），终成笑谈（23-26岁）（见 《朝华午拾：我的世界语国》）。

风华正茂，意气风发（1987）

及至毕业留所，立委事迹亦有流传，多为一见钟情，闪电结婚，不修边幅，撞南墙而道歉之类小李“景润”之逸事（见《朝华午拾：shijie－师弟轶事》；《朝华午拾：shijie－师弟轶事（3）——疯狂世界语 》）。

立委在中关村公司指导机器翻译系统的开发（1988）

立委如此这般在研究所及中关村公司一扎五年（26-31岁），练就一身绝技，与老中医相若，专事疗治电脑，驯其语言功能。其间，出国热持续升温，由上海蔓延北京，街头巷尾，言必议美、日、大英，澳大利亚，以致居委会大妈亦知考托福鸡阿姨乃上进青年之标杆。立委及其贴身领导却浑浑噩噩，卿卿我我，不知有汉，无论魏晋。其间送上门两次机会，留学德美，均因导师明阻暗挡，本人木呐，擦肩而过。直至身边同学悉数走尽，小李才幡然醒悟，痛下决心，赶末班车。其时，适逢包玉刚基金会来各单位选拔年轻业务骨干，滥竽充数，小李竟被选中，送至成都科大出国培训中心修行半年。

岂料想，此一去竟成小李老李的分水岭。来培训的诸位才子才女均是全国各地选上来的各行好手，共分两拨：一年的访问学者大都比较年长，而拿三年博士奖金的大多年轻，立委在后一拨里面理所当然，成了老大。每有考试，立委必中头彩，引来才子才女，大事小事，纷纷登门请教，“老李”之声不绝于耳。立委名噪一时，响应者众。从小习惯了以小卖小，乍一变老，立委满腔郁闷。

  
成都科大出国培训中心的才子才女们（1990）

小李变老李，心里虽别扭，好处却不少。龙头老大，备受尊崇。立委外语本科出身，本应免试英语，无奈官家财大气粗，慷人民之慨，不问青红皂白，全数押解天府之国，集中喂养。不止英文鸟语，更有政策轮训。众兄弟姐妹兢兢业业，争先恐后，唯立委悠哉游哉，终日沉迷天府美食，流连于茶肆酒吧，众兄弟钦羡有加。

成都一站始称老李，立委心内实不以为然也。其时立委事业发达，如日中天，行内行外，交游甚广，出入皆鸿儒，往来无白丁（见 《朝华午拾：“数小鸡”的日子》；《朝华午拾：一夜成为万元户》）。导师为本行泰斗，立委乃导师仅有的关门弟子（其他弟子皆叛国投美去也），“青年”才俊，明日之星，业内同侪为之侧目。去国前夕，全国电脑翻译界在香山招待所年度聚会，点睛之笔为导师与本行另一大牛的座谈，人称“刘董对话录”，其间立委频频亮相，为导师提供实例，讲解细节。影响所及，与会众学妹（多为刚入门的外地在读研究生）纷纷上门请教立委，无奈立委远走高飞心切，痛失辅导上进女青年之良机。

立委在加拿大（1995）

去国经年，由英而加，由加转美（《朝华午拾：哦，加拿大！》；《朝华午拾：温哥华，我的梦之乡》）。颠沛流离，不知所止，壮年人生，如水流逝。及至水牛城八年抗战（37-45岁），立委青春不再，壮年已过，“老李”名至实归。然立委壮心不已，励精图治，双线出击，称雄一方（见 《朝华午拾：创业之路》, 《朝华午拾：在美国写基金申请的酸甜苦辣》，《朝华午拾 - 水牛风云》）。

立委在水牛城办公室（2000）

回首往事，不胜唏嘘。立委一生，由青年而壮年，正值创造力最盛，精力充沛流溢之时，天时地利人和，飞黄腾达有望，却为漫长的留学生涯拦腰截断。大而言之，立委固赶上出国之末班车，却误了千年不遇的中国经济起飞之航。拣了芝麻，丢了西瓜，此之谓也！ （《朝华午拾：乡愁是一张无形的网》）

去岁归国省亲，杯觥交錯，在某宾馆餐厅与亲友相聚甚欢。席间小憩，踱步凉台，享清凉之气，赏京华夜色。偶遇一妙龄女士，携一幼童，见立委两鬓染霜，嘱曰：“叫爷爷”。立委血压骤升，如雷轰顶，满腹酒意，化为凉液，由脊背滑落。

立委老矣，尚能饭否？

记于2006年11月5日

立委老矣

【作者简介】立委先生，IT业技术研发经理兼架构师，自然语言处理资深专业人士。曾任红小兵，插队修地球，文革后第一届大学生，后跳龙门进社科院读硕士，攻机器翻译。1991年去国离乡，漂流海外。由英而加，获计算语言学博士。由加转美，作为创业公司研发副总及项目负责人（Principal Investigator）, 先后赢得美国政府17个研究创新项目近千万美元资助，同时从资本家腰包亦忽悠千万风险投资作商业开发。对于自然语言信息抽取 (Information Extraction) 有全面的研究，研究成果对美国政府有关科研项目的确立有直接影响。业余爱好：音乐、博客、舞文弄墨。著有回忆录《朝华午拾》。

原载【朝华午拾 － 立委小传】 2010-1-9
https://blog.sciencenet.cn/blog-362400-285507.html

Despite the common logic and conceptual graph at the core of human mind, we all have our own semantic lexicons that are unique, implanted by our career path and life struggles. My semantic lexicon is full of wandering and continuously drifting into new worlds. It all started from the time when Mao sent us to the farm for re-education in 1976. After that the path has been zigzag, full of adventures of drifting, and re-drifting, farther and farther away from my hometown and home country ......

在我的语义词典里，流浪 是一个很大的节点，它的上位概念是 漂流（走四方）和 波浪（多起伏）。流浪的下位概念枝繁叶盛，包括：插队，洋插队，跳龙门，再跳龙门，北漂，下海，西漂，南下，再南下。这也正是我的生活写照。在这些语词概念的背后蕴含几多激动几多辛苦，只有自己知道。

微信视频看到一位语文老师对这首歌歌词的吐槽和改写。有些道理，改写的歌词也确实顺溜多了，易于普及。但第一，这是在人家原创的新颖写法所创造的意境上修改；第二，顺溜有顺溜的好处，矛盾或难解也有引发听众思考与发挥的好处。

这首歌最近听得蛮多（我在春节前后还在视频号做过两期MTV）， 对歌词有一些感觉可以说说。

整体上说，原作写法新颖，用词有些奇特，整体歌词长在哲理和意境，有妙语，但也有语病。最严重的语病就是“赠予回敬”。

上天赠予“我”回敬，“谁”回敬“谁”“什么”呢？ 回敬这个词的最常见的场景是，他人攻击我了， “我”回敬他人，那也是我的自主行为，谈不上“赠予”。如果是他人回敬“我”，其前提是“我”对他人做过攻击，前后看语义上下文，这是说不通的。

“回敬”是一种故作敬态的回应，而“赠予”是恭敬的馈送。让回敬做赠予的宾语，搭配不当。“赠予我拥有”（可以理解为赠予我礼物，拥有代指“拥有物”）就已经够别扭了，再来个“赠予我回敬”，让人感觉不知所云。可能是“回馈”（对“拥有”的回馈）的意思，为了押韵，错用了“回敬”来代替。

“回敬”作为谓词，逻辑语义框架里有三个角色：施事（谁回敬）、受事（回敬谁）、宾语（回敬什么），但“回敬”自己处于“赠予”的宾语位置，这几个角色模糊不清，其所引起的混乱和费解，不怪语文老师觉得不可忍。

写词的文科姐，可能是浮想联翩，用力过猛而“出格”。这在歌词创作中也不罕见，叫 poetic license，通常不做苛求。但无论如何，这种奇怪的动宾搭配困惑度（perplexity）很高，会使绝大多数人感到糊涂，属于败笔。大家传唱不过是因为作曲好就跟着瞎唱，并没在意歌词是不是 make sense。

困惑度高的直接表现就是 ，剪映中自动听音写词的功能根本无法decode原文，因为这项软件功能的背后是语言模型（language model），对于这种困惑度高的序列搞不定，只能另行创造（所谓”幻觉“）：

原歌词：世界赠予我拥有 也赠予我回敬。
语言模型幻觉解码：世界赠予我拥有，也赠予我爱情。

面对困惑，语言模型无法decode这种出格的原词（outlier） “回敬”，结果解码成 “爱情” 似乎也不错。在这种解码下，“拥有”应该指的是财富，“爱情”就是爱情。而在原词中 “拥有”可以解读为命运的礼物或曾经的爱情，而“回敬”则可能是对于礼物的回赠。

其他困惑度高，语言模型幻觉创造的cases还有：

原歌词： 赠我一个名，又渐渐长大的年龄
语言模型： 赠我一个谜，又渐渐长大的年龄

人生本来就是一个谜啊，岂止简单的出生赐名，所以这里模型的解码也许更妙。最妙的是：

原歌词：赠我弯弯一枚月，也赠予我晚星
语言模型：赠我温暖与悲悦，也赠予我惋惜

“月”和“星”状物，“温暖”、“悲悦”和“惋惜”直接述情，貌似更胜一筹。唱起来也很顺。

顺便一提，“别匆匆”歧义，有两个隐藏解读都说得通。一个是：不要匆忙。要善待自己，给自己品味人生，以及喘息和疗愈的时间。另一个是：分别也匆匆，尤其是感叹恋人或亲人聚少离多的生活现实。

再有，语文老师发现歌词里面暗藏了（谢）霆锋的名字，有机巧。说明此歌是为王菲量身打造的。娱乐圈八卦已经众所周知了，谢霆锋是王菲的最爱，是三段婚姻中最念念不忘的。N年前先是王菲谢霆锋的姐弟恋，以及她不顾世俗和骂名的第三者插足；后离婚，再后来又复婚，中间还穿插了其他 relationships，起起伏伏。不怪王菲唱罢歌曲，满噙眼泪，双手合一，人在台上久久静默，仿佛在念佛。这首歌，她是真带入了。同时她的演绎也感染了无数人。

“远去者去了远方，愿他都安心。” 一开始还以为在纪念逝去的亲人，但通观全词的爱情主线，更像是在纪念逝去的爱情。也许远去者是不得不分手、又难舍情缘的前任，她祝福他安心，其实更是试图宽慰自己，要安心接受“拍一张合影，渐渐填满真感情”的新缘分。

我本人特别喜欢这两句歌词：

赠我一场病，又慢慢痊愈摇风铃。
赠我一场空，又渐渐填满新感情。

它是我2024年生活的真实写照，非常的切身感受。

总的感觉一句话，词作者能写出引起人共鸣、思考和争论的歌词，还是很了得的。至于作曲以及王菲的演唱，可以说是注定成为经典。

【相关】

https://www.douyin.com/video/7466269705402060042

语文老师点评并修改王菲《世界赠予我的》歌词# 王菲... https://v.douyin.com/ifcm9PvH/ CuF:/ 03/09 [email protected]

7.99 复制打开抖音，看看【立委的作品】王菲春晚注定传世之作 小白版 # 王菲 # 小白 ... https://v.douyin.com/ifvcmXG7/ Ate:/ [email protected] 12/08