本文整理自InfoQ策划的DeepSeek系列直播第二期节目——DeepSeek爆火背后DeepSeek，纯强化学习路线到底有何不同。在直播中，出门问问大模型团队前工程副总李维博士聚焦推理范式的创新，分析了R1 Zero 对推理模型平民化的创新贡献。他提到，DeepSeek通过开源和透明化，证明了不需要过程监督，仅通过结果控制就能训练出优秀的推理模型，这大大颠覆了传统认知以及OpenAI 所暗示的需要在每一步监督推理强化学习的观点。
李维博士：实际上，推理模型的强化学习一直是业界的难题。大约半年前，IIya 等人宣称预训练时代已经结束，这意味着单纯依靠预训练模型的规模扩展来提高性能已经难以为继。GPT5迟迟不能上线也是预训练式微的一个迹象。因此，业界开始寻找新的增长道路，推理大模型在头部团队开始暗流涌动，直到 Open AI发布全球第一个推理大模型O1. 紧接着就是DeepSeek的R1出圈，这就是deepseek爆火的背景。
从 神神秘秘、据传引发了OpenAI宫斗的Q-Star 项目开始到 o1 大模型的推出，推理大模型被AI主流广泛公认为新的范式。这种范式的核心是开启模型的“慢思考”能力，即所谓 System 2，利用强化学习提升模型在复杂任务中的智能程度。然而，这一切都是闭源的，OpenAI 甚至故意制造了一些神秘感，遮掩其思维链的内容。除了少数头部玩家如 Google 和 Anthropic 在背后悄悄探索追踪外，其他团队对这一领域知之甚少。
李维博士：DeepSeek 的R1论文非常出色，堪称大模型领域中的一篇佳作。论文分为两部分：一部分是关于 Zero 的研究，这是纯粹的强化学习推理方向的成果，非常精彩；另一部分则是基于 Zero 研究成果的实用系统 R1，这是一个真正上线的头部推理大模型。在开发 R1 时，需要考虑实用性，包括综合性能、安全性以及各种实用考量等，因此论文中详细介绍了四阶段训练的最佳实践（best practice），帮助其他团队理解和复制这一成果。
论文最精彩的部分还是 Zero 的研究。Zero 的研究证明了一个颠覆性的观点：与传统认知（或 OpenAI 所暗示的需要在每一步监督推理强化学习的观点）不同，实际上并不需要过程监督。仅通过最终结果（即“黄金标准”）作为监督信号，就能训练出推理大模型所需的“慢思考”过程。
这是 Zero 的最大亮点，也是其名称的由来——它借鉴了 AlphaZero 的精神。AlphaZero 在人工智能历史上开创性地完全不依赖人类棋谱或经验学习，而是通过自我对弈的再生的过程数据（即：棋局状态+落子+评分的三元组步骤数据），实现了零人类监督的强化学习，并最终完全碾压了人类顶尖棋手。DeepSeek 的 Zero 研究也是如此，它表明在推理任务中，模型可以自主生成内部的过程数据，即思维链（CoT，Chain of Thought）序列，而无需人类标注。
老实说，真正关心数学或代码的人并不多，虽然我们知道代码是今后几年的一个大方向，自动编程能改变世界。所有 IT 方面的东西归根结底都是软件，数字世界是由软件构成的。如果软件能力可以从手工编写变成模型辅助，甚至模型自主编写，这将极大地提高我们的生产力。这是大家都能看到的，但对普通老百姓来说却没有那么直观，因为他们面对的更多是写文章如何出彩这类任务。
李维博士：DeepSeek 之所以能够产生复杂的思维链，背后是因为它是基于头部大模型V3训练的，而 V3 所涵盖的知识比我们任何个体所了解的都要广博得多得多。在这基础上，关键点是要给模型留下空间，让它有自主学习的机会。作为设计者或开发者，需要设计出这样的空间，让模型自己去填补、去学习。DeepSeek 就是这样实现的。它设计了一种格式，在输入问题question和输出答案answer之间，它留下了一个“思考”的空间，用标签 [think] 来标记: question+[think]+answer。这个 think 标签就是准备要学 思维链（cot） 的, 虽然开始为空，Zero 的 research 表明：只要留下think的标签，就给LLM自主填补cot 留下了空间。此后他们“啊哈”地惊喜发现，越来越条理化的cot 居然在 GRPO 组内选优的强化学习迭代算法的指引下，就自主学出来了。啥也不用做，模型就是自己要思考，而且能思考。LLM really wants/tends to think and think deep if given a chance. 比如，它可能会在推理过程中发现自己前面的某个结论与已知事实不符，于是就会自我纠正，说：“不对，这里可能有偏差。”这种反思和自我纠正的能力，是模型在学习过程中自然形成的。可以想像研究者当时的兴奋之情, 简直就是上帝给他们面授了天机。不但他们“啊哈”, 我们读论文追踪他们的人也感觉开了天目，不可思议，但 it just works。Zero research 的美丽就是没有人工的过程数据的任何干预，完完全全的纯强化出来的奇迹。
Anthropic 的 CEO 预计，在未来 3 到 5 年内，大模型将实现真正的突破，不仅仅是目前让我们震撼的表现和demos，而是真正能在生产力上对整个社会带来革命性的改变。他们所说的，归根结底就是 AI 能规模化平替人类的体力劳动和脑力劳动。目前大模型虽然很热闹，但在社会生活中的实际应用还远未达到上一代移动互联网平台的水平。上一代的 super apps，比如美团、滴滴、小红书、抖音等，它们改变了我们日常生后的主要方面，无论吃穿住行还是通信和娱乐，它们最大程度缩短了供应商和客户之间的距离，这些价值我们每天都能感受到。而玩大模型虽然有趣，但在生活层面的实际价值还不明显，应用层面还处于爆发的前夕。
值得指出的是，DeepSeek 的出现降低了大模型应用门槛，为应用铺平了道路，虽然目前我们还没有进入应用真正爆发的时代。未来，当AI应用真正爆发时，会是什么时候、什么样子呢？我认为，最终目标是 AI 在脑力劳动和体力劳动中全面代替人类。大模型对白领阶层的冲击，迹象已经很明显，甚至连程序员群体都难幸免。体力劳动方面，具身智能发展也很快，无论是人形机器人还是机械手，都在逐步代替人类的体力劳动。
— Reflections on LLM Scaling Laws and DeepSeek's R1
My friend Zhang Junlin's article "Looking at the Future of Scaling Laws through DeepSeek R1" has sparked interesting discussions among peers.
Core Insights from Initial Discussions
Professor Bai summarised the key highlights as follows:
Infinite stacking won't lead to infinite growth (physical laws don't support this)
Only S-shaped growth is possible, with diminishing returns inevitably appearing
The initial emergence of language capabilities relates to the density of linguistic knowledge in training data
The next growth phase represents a second S-curve, driven by common sense knowledge, which requires more computing power due to lower knowledge density
The third phase involves learning logical reasoning (Chain of Thought), where natural data has even lower density of such knowledge. Brute-force mining with computing power becomes inefficient, making reinforcement learning with synthetic data a more rational approach
As Dr. Lu points out: The term "Scaling Law" is becoming overloaded. While S-curves (nonlinear curves characterized by sigmoid functions) can describe technology adoption lifecycles, they typically occur in succession (one technology hits its ceiling, making way for another). Large language models' multiple "Scaling Laws" confirm this pattern, with some overlap between Test-Time and Post-Training "Scaling Laws".
The Nature of LLM Scaling
Let's examine the fundamental logic behind LLM scaling. First, it's crucial to understand that LLMs are not databases - they don't aim to memorize long-tail data details. Large model training essentially compresses big data, or more precisely, compresses the knowledge systems behind the data (including common sense and encyclopedic knowledge), focusing on capturing patterns and regularities of various patterns (what we call generalizations).
Conventional intuition suggests that as data scale increases, redundancy increases too. Regardless of filtering, cleaning, and deduplication, growing redundancy seems to imply diminishing returns. So why do large models still appear "hungry" even at the unprecedented scale of hundreds of billions of tokens? Why does the scaling law remain effective from hundreds of billions to trillions of tokens?
The key lies in LLMs being sequence learning and sequence decoding systems. While sequences are one-dimensional, the patterns and regularities behind are high-dimensional. For instance, even a simple sequence like "cat chases mouse" potentially involves multiple knowledge dimensions: species relationships, predatory behavior, spatial movement, actor-patient roles, etc. This multi-dimensional knowledge naturally leads to combinatorial explosion at the sequence level as information is flattened in language. The "appetite" for insatiable big data effectively addresses this combinatorial explosion. As long as there isn't complete information redundancy, additional diverse sequences will help models abstract data patterns more precisely.
The Two vs. Three S-curves Debate
Zhang Junlin observes that since OpenAI's O1, two other phases have gained recognition with their own Scaling Laws: the reinforcement learning Scaling Law (RL Scaling Law) for post-training, and the Inference Scaling Law (also called Test Time Scaling Law).
This raises a crucial question: Are there really three S-curves, or just two? How comparable is the reasoning model's S-curve to the pre-training S-curve?
In practice, post-training and inference phases likely share a single S-curve; there aren't two independent growth curves.
DeepSeek R1's Insights: The Truth About "Slow Thinking"
Consider DeepSeek R1: users can activate "deepthink" mode to enable Chain-of-Thought (CoT) reasoning, but they can't actually control reasoning quality by increasing computation time. Why is this?
Let's examine a concrete example. When R1 solves a complex mathematical problem:
Traditional models might directly answer: "The result is 42"
R1 shows detailed reasoning: "Let's think step by step: 1) First consider... 2) Then we can... 3) Finally, we get 42"
While R1's response appears to demonstrate "slow thinking" (CoT), this reasoning process reflects actually a generation pattern fixed during training, not dynamic exploration of multiple potential reasoning paths during response time. In other words, CoT+answer might look like "slow thinking," but it doesn't fundamentally change the unidirectional next-token prediction paradigm. R1's CoT+answer creates an illusion of slow thinking, but the generative nature remains fundamentally the GPT "fast thinking" paradigm. At test time, unlike AlphaGo, the depth and scale of thinking isn't dynamically explored, though beam search, if applied, can provide implicit multi-path optimization internally.
Test Time Compute Constraints
The industry's buzz word "test time compute" refers to reasoning models requiring more online computational resources compared to traditional non-reasoning models. For example, R1 with CoT enabled might need several times more computation time than its base model V3 for the same problem. However, this increased computation results from behavior patterns acquired during training, not dynamically adjustable compute investment. Without controllable scalability in test time compute, we can't really talk about a test time scaling law.
A major difference between pre-training and CoT reinforcement learning lies here: pre-training scaling laws can remain stable long-term because once training completes, it doesn't significantly impact online response time - the generation mode remains a simple query+answer. Therefore, offline training for months is acceptable if the resulting model shows significant capability improvements. However, reasoning models' post-training CoT reinforcement learning differs - it cultivates models' habits of responding with slow thinking, changing the generation mode to query+CoT+answer. Extending the CoT isn't just about the cost of training resources and time; more critically, it reflects in extended test time compute for each query during deployment, severely delaying system response time. Users generally have limited tolerance for slow thinking computation time and delays during online system use.
The Sustainability Debate
OpenAI's Sam Altman and Anthropic's Dario might argue that for extremely complex problems (like proving the Riemann hypothesis or designing next-generation aerospace vehicles), even if a model needs a week of computation time, it's still a massive improvement over human teams requiring decades. However, this argument has two issues:
LLM feasibility for such super-complex problems remains far from validated
Extreme scenarios lack universality and can't serve as data points for sustainable scaling laws
This isn't to deny S-curves as effective models for describing scaling laws, nor to reject the rationality of S-curve stacking. The combination of pre-training and post-training growth curves (s1 and s2) might indeed reflect the overall relationship between resource investment and performance improvement. However, we should carefully examine whether CoT reasoning truly opens a sustainable scaling curve.
Conclusion: How Far Is the LLM Road to AGI?
If reasoning models' scaling laws lack sustainability, this raises a deeper question: Can we reach the promised land of Artificial General Intelligence (AGI) through these two scaling laws alone? Furthermore, is the technical ideal of Artificial Super Intelligence (ASI) - AI replacing human labor and dramatically improving productivity - truly feasible?
Current evidence suggests that while pre-training scaling laws have shown considerable sustainability, reasoning models' scaling laws may quickly hit practical constraints. This reminds us that the path to AGI/ASI likely requires more innovative breakthroughs, not just simple extrapolation of existing methods. In the next phase of artificial intelligence development, we might need to discover entirely new growth curves.
A detailed analysis of how DeepSeek R1's inference mechanism works in production, and how it differs from training-time reinforcement learning.
Training vs. Deployment: Key Questions
1. Training Phase (GRPO): Does the reinforcement learning mechanism generate multiple candidate CoT+answer sequences to optimize the policy and cultivate "slow thinking" habits?
- The answer is definitively yes.
2. Deployment Phase: Does R1 implicitly generate multiple paths during inference but only display one? If so, how does this mechanism compare to traditional ensemble methods?
3. Comparison with AlphaGo's MCTS: How does R1's mechanism fundamentally differ from Monte Carlo Tree Search?
1. Inference Mechanism in Production
DeepSeek R1's real-time reasoning can be characterized by two modes:
A. Implicit Multi-path Generation and Selection
- Generation: The model may implicitly generate multiple potential reasoning paths (CoT+Answers) during a single inference but outputs only one.
- Technical Implementation: Through decoding strategies (e.g., beam width adjustment), the model maintains multiple candidate sequences, ultimately selecting the highest-scoring path.
- User Experience: Users see only the final output, though internal multi-path exploration occurs.
- Efficiency Trade-off: Setting beam_width=1 (greedy search) defaults to single-path generation for fastest response; increasing beam width improves quality at the cost of latency.
B. Explicit Multiple Candidate Generation (Optional)
- API Control: The num_return_sequences parameter allows explicit generation of multiple candidates.
- Practical Application: While not enabled by default in the DeepSeek App, this functionality may be available through enterprise APIs or open-source implementations.
2. Training Phase: Cultivating "Slow Thinking"
A. Role of Reinforcement Learning
- Objective: GRPO algorithm trains the model to generate more detailed, logical reasoning steps (longer CoT) to maximize rewards.
- Mechanism: Training generates multiple candidate answers, with rewards evaluating both answer correctness and format correctness.
B. Driving Forces Behind CoT Growth
- Reward Design: Longer CoTs naturally emerge when they lead to better answers.
- Data Feedback: High-quality SFT data generated through rejection sampling enhances this pattern.
3. Comparison with Ensemble Methods
Similarities
- Multi-path generation conceptually similar to ensemble predictions
- Result filtering comparable to voting/weighted averaging
Key Differences
R1's implicit multi-path generation is fundamentally a dynamic decoding strategy within a single model, distinct from traditional ensemble's static combination of multiple models.
4. Fundamental Distinction from AlphaGo's MCTS
AlphaGo's MCTS
- Dynamic Programming: Builds search trees through simulation
- Online Learning: Adjusts search strategy based on real-time feedback
R1's Implicit Multi-path Generation
- Static Model: Fixed parameters during deployment
- No Reward Modeling: Path selection based on model probability rather than cumulative rewards
Key Insights
1. Training phase GRPO cultivates detailed CoT capabilities for effective single-pass inference.
2. Deployment allows flexible trade-off between single-path (for speed) and multi-path (for quality) generation.
3. While model parameters are fixed post-training, decoding strategies offer some runtime flexibility.
4. R1's multi-path generation fundamentally differs from both traditional ensembles and MCTS-style dynamic planning.
This architecture achieves a practical balance between efficiency and effectiveness for large-scale industrial applications, though it sacrifices some dynamic planning and global optimization capabilities.
ASI 则是全面超越人类顶尖智能，包括发明创造的能力。ASI 的实现应该还可以商榷。现在就确信ASI可以在不太久的未来（有说三五年，也有说10年左右）实现的吹鼓手，主要是伊利亚、Dario（Anthropic CEO）这些AGI时代的“先知”们，他们是信仰者。奥特曼和马斯克貌似也在营销类似ASI的概念，但感觉更多是企业家需要画饼的驱动。
一个有意思的例子是，当 alpha zero 下棋到第30几步的时候，走了一步人类不能理解，连世界冠军也会判定是愚蠢的一步。但那却是超人智能的精彩过程，是制胜法宝的一个精妙环节。这种高招连冠军都不能理解，说明机器智能显然超越了人类智能的边界。如果在过程上依赖人类反馈，哪怕是围棋冠军来做标注，也会阻碍机器智能的超人潜力。
当这类超人智能大量产生的时候，人类很自然会感觉困扰。因为 by human nature，所有人多多少少都有某种控制欲，对于自己不理解、不能掌控的过程，总是持有戒心，至少是很不舒服。但可惜无解。未来会出现越来越多的不可理解的奇迹，或技术魔术。人类所能做的就是加强目标制定和结果控制，而不是“不自力量”试图过程控制。
Amid the waves of surprises brought by DeepSeek, an old friend pointed out that it struggles with simple math problems, using a popular elementary arithmetic question as an example:
Is 3.11 greater than 3.8?
What’s the core issue here?
In the wake of the DeepSeek frenzy, I looked into its research paper, which explains how its reasoning capabilities are enhanced through outcome-oriented reinforcement learning. The paper suggests that, in theory, outcome-oriented reinforcement learning can help a model learn proper reasoning processes. However, in practice, it’s not necessarily so.
Take the above math problem as an example. The answer is a binary yes/no question, meaning even random guessing has a 50% chance of being correct. This highlights a key potential flaw: outcome-oriented supervision signals are weak because they lack sufficient granularity. This kind of weak supervision inevitably hampers the model’s ability to learn proper reasoning processes.
Three Possible Solutions
Scaling Up the Model
One approach is to make the model larger and deeper, hoping that the theoretical concept of lossless compression based on Kolmogorov complexity can be pushed to its limit. In doing so, proper reasoning, as the "shortest program," might eventually be learned by the model. Theoretically, correct reasoning ensures accurate results. However, the gap between theory and practice makes it hard to place much confidence in this. The shortest program or lossless compression might just be an unreachable ideal.
Targeted Supervision Data
Another solution is to feed the model with problem-specific supervised data. For example, providing thousands or tens of thousands of reasoning cases involving such math problems. There’s no reason the model wouldn’t learn from this. However, solving one specific problem this way is merely a stopgap measure. Soon, others will come up with new edge cases involving weak supervision signals and reasoning pitfalls to challenge it.Another common challenge is the so-called “self-identification” problem. For instance, when asked “Who are you?”, many models, including DeepSeek (earlier versions), would claim they are ChatGPT developed by Open AI if no targeted supervised data is injected. After all, ChatGPT has dominated the internet in the two years since its explosive debut, and its data has inevitably influenced other models. However, this issue is already on the radar for specialized solutions and is gradually becoming less of a problem. Some Western media still claim that DeepSeek is just a distilled version of ChatGPT. Their evidence? Probably based on early versions they tested, the DeepSeek bot often self-claimed to be OpenAI’s ChatGPT. But if you test it now, you won’t see this problem anymore. Most likely, it was fixed with specialized training data. Their research paper also mentioned addressing the self-identification as a problem.
Similarly, the problem of comparing 3.11 and 3.8 can also be a transitional issue. If it disappears in the future, it won’t be a cause for celebration. Most likely, it will be resolved through targeted fixes rather than through fundamental improvements in intelligence brought about by algorithms or architecture changes or innovations.
Re-introducing Process Reward Models (PRM)?
The inherent weakness of outcome-oriented supervision signals is that it focuses only on the result while ignoring the checking of the process—a natural shortcoming of reinforcement learning driven by results-oriented pragmatism in RL (following the “black cat, white cat” principle, lol). This is essentially the cost of abandoning PRMs (Process Reward Models). So, would re-introducing process-based reward models solve the issue? Honestly, we don’t know. This is the third possible path, and it might be worth exploring. But again, as mentioned in my previous blog post (DeepSeek's R1 Paper: A Storm in AI LLM Circle), PRMs aren’t easy to work with—they’re unstable and difficult to implement, although, in theory, they could help correct nonsensical reasoning during the process.
In conclusion, the issue with DeepSeek struggling with problems like 3.11 vs. 3.8 lies in the limitations of weak supervision in results-oriented reinforcement learning. While there are potential solutions—scaling the model, targeted data, or process reward models—each comes with challenges and trade-offs. Whether any of these approaches can fundamentally improve reasoning capabilities remains an open question.
The recent Chinese podcast from Guangmi's quarterly report on large language models, discussing the "scaling paradigm shift" toward AGI (Artificial General Intelligence), is well worth a listen. It touches on many key topics related to the AI industry landscape, offering a unique perspective and style.
The term "paradigm shift" may sound a bit dramatic, but as a seasoned analyst, Guangmi uses it to describe the current turbulent landscape accurately. While the AI arms race among industry giants is still in full swing, real-world scalable applications of these models are struggling to materialize. The question of how to justify investments has become a significant pressure point, or perhaps even a looming bubble.
Let's revisit some AI basics. There are three main types of learning in LLMs (Large Language Models):
(i) supervised learning;
(ii) unsupervised learning (self-learning/pre-training); and
(iii) reinforcement learning (RL, self-play/post-training).
Ilya has emphasized the importance of RL in exploring new directions for LLMs. Guangmi's podcast highlights RL as the pathway to the paradigm shift in AGI through large models.
Historically, two key milestones in RL have stood out: AlphaZero's victory over human Go players, which shocked the world, and RLHF (Reinforcement Learning from Human Feedback), which aligned models with human preferences and paved the way for ChatGPT’s explosive growth.
Currently, discussions revolve around the potential of a new RL-driven ecosystem for large models (though there's no broad consensus—it's primarily a conversation within small Silicon Valley circles) and the emerging trends in the "arms race" of large models. Here’s the context:
1. Pre-training scaling seems to have hit a bottleneck, with GPT-5 still unreleased;
2. The overall momentum of the arms race remains unchanged among the major players (the billionaire clubs/giants);
3. Key tech figures are proposing new roadmaps or trying to construct new scaling laws to continue the AGI journey.
Guangmi closely monitors trends in Silicon Valley. His small team conducts in-depth research in the Bay Area and has established extensive contacts. Having chatted with them over coffee a couple of times, I’ve found them to be a dynamic, young team under his leadership—a small but sharp presence.
Guangmi’s thoughts are well-structured, and his breadth of knowledge and understanding of the larger context are impressive. This is no small feat, as the landscape of large models, both in terms of the models themselves and the industry, is often akin to the parable of the blind men and the elephant. Even top experts and business leaders struggle to assess the full picture. Just recently, Meta’s Zuckerberg responded to a question about whether the AI arms race would deliver the expected AGI returns, essentially saying: “No one really knows, but we can’t afford to miss out,” reflecting a typical FOMO (Fear Of Missing Out) mindset.
We’re currently in a delicate phase with little consensus. However, the few tech giants that have propelled Nvidia’s stock to astronomical levels won’t allow the arms race to slow anytime soon, as it is central to their tech and business dominance. OpenAI continues to raise funds, and Ilya, with his new company, recently secured more investment, all of which keeps the race heated.
At the same time, the obsession with scaling among tech elites and the mainstream AGI circles in Silicon Valley persists. The endless demand for resources driven by this scaling wave of large models means that only a small circle of tech insiders has the opportunity and resources to experiment, sense, and adjust the roadmap.
According to Guangmi, the so-called self-play RL scaling is currently gaining traction within a small circle of about 200 tech elites in Silicon Valley, indicating that this is still a nascent trend—one that even management leaders have not fully aligned with yet.
It seems Guangmi adopts a “prophet” mentality at times, perhaps exaggerating this trend to alert his audience. He even suggests that if he were a large-model entrepreneur, he would focus 200% of resources on RL, betting on it as the future path to victory.
In reality, for most people, this advice is neither practical nor actionable—it’s likely aimed at tech giants or unicorns, though even for them, it may fall on deaf ears.
Reinforcement learning is inherently challenging. Even the open-source leader Meta LLaMA 3 has chosen to sidestep RLHF in post-training alignment. So, it's even less realistic to expect large-model teams to fully bet on RL as the core of a new ecosystem. Furthermore, this trend is, at best, a “subtle undercurrent” in Silicon Valley. We’ll likely have to wait until OpenAI’s “Strawberry” or the new version of Claude releases later this year to fully assess its impact.
It seems the first chapter of LLM scaling has indeed come to an end. The actionable items in the so-called second chapter might not emerge from lofty, exploratory scaling directions with an uncertain roadmap. Instead, the focus should be on finding market entry points, accelerating applications, and addressing genuine market needs (PMF, product-market fit), especially as the inference costs of top models like GPT-4o/Claude 3.5 become more affordable, and multimodal capabilities (such as advancements in hyper-realistic full-duplex voice and video) further enhance application opportunities.
For the industry, the bottleneck in scaling large-model applications is the sword hanging over its future. This will determine whether the second chapter of the tech adoption curve ends with a soft landing and eventual recovery. As for the arms race, it’s best to leave that to Elon Musk, Zuckerberg, and the billionaire club to continue playing.
Reinforcement learning, as an extension of pre-training, belongs to the realm of “post-training.” When pre-training hits bottlenecks and diminishing returns, strengthening RL is a natural complement. In the simulation of human cognition, pre-training represents the accumulated knowledge of human civilization, while RL applies that knowledge in practice, learning from the environment. This overall approach to intelligent learning makes perfect sense and is the necessary direction for applying large models.
My old friend Lu said: “It’s intuitive that RL is the path we must take because there isn’t enough supervised learning data anymore.”
Indeed, utilizing regenerated data to varying degrees has become common practice. It’s inevitable. Models can already generate data of higher quality than humans, and this will only improve. However, this is not the same as self-play's proactive exploration and data regeneration.
As Mr. Mao pointed out: “RL aligns with the cognitive processes of humans and epistemology. It’s essentially the process of receiving external feedback and being tested in practice. RL is active learning, while training is passive.”
Guangmi's RL paradigm shift suggestion still lacks the necessary catalysts. But this potential trend is worth keeping in mind. It’s best to remain cautiously optimistic and open-minded while watching how things unfold.
Professor Ma is a compelling speaker, and his talk is definitely worth listening to. His paper on whitebox transformer, over 100 pages long, has just been released (Yi Ma’s white-box transformer paper is available here). Unfortunately, I haven’t had the time to dig into it yet. We’ll have to wait until more people have accepted or verified it before delving deeper.
His current claims revolve around using an extremely sparse approach to force transparency in transformers, with results that are reportedly on par with BERT and GPT-2 in many benchmarks. However, this doesn’t mean that he will be able to catch up with GPT-3 or later models anytime soon. But to be fair, it’s not a level playing field—he’s an academic without the resources to compete with mainstream AI in an arms race. What he does believe, however, is that he has opened a door—a path toward explainable AI in large models.
Honestly, I’ve always had a litttle bit doubts about Ilya’s theory explanation of shortest program compression (his Berkeley talk). From an ultimate theoretical perspective—where lossless compression is the ideal—the idea of continually scaling training, deepening, and lengthening learning makes sense, as it pushes the model toward becoming the smallest possible program for universal tasks. Ilya’s theory may hold up in this respect, at least in theory or as an end goal. But in any real-world scenario (e.g., under budgetary constraints, with methodological limitations), it’s hard to call a model purely derived through gradient descent the “shortest program,” because these models appear to be gigantic beasts with "huge circuits" inside, intuitively, should not be considered "short or small".
Models with hundreds of billions or even trillions of parameters are massive monstrosities, succeeding mainly through sheer size rather than through high regularity or elegance. Emphasizing how impressive their compression ratios are or how well they handle lossless compression may help explain the generalization and emergeng abilities in sequence learning from a theoretical standpoint. But in practice, any model at a given time is far from being the “shortest program.”
This highlights an unavoidable distance between theory and practice. Ilya essentially hedged practice with theory along a future time axis, but our immediate reality doesn’t seem to align with this. It’s like a clumsy wrestler trying to brand himself as sleek and slender fashion model. Visually not a fit, to most of our eyes.
Instinctively, LLMs feel full of rote memorization with significant redundancy. Under real-world conditions, achieving extreme or lossless compression seems impossible.
On the other hand, Professor Ma’s sparsity approach almost feels “over the top.” Enforcing the same weight for QKV directly seems a bit crude and simplistic, yet it still managed to be trained successfully. This shows that there’s a lot of flexibility within transformers—no matter what restrictions or pruning are applied, the model still finds a path out. In this sense, Professor Ma’s pursuit of the “shortest program” is more real and direct—it’s so short that even a human can interprete the process (hence the LLM explainability).
Yet the difference between these two extremes is still mind-boggling. On one side, we have gigantic models, and on the other, extreme simplicity to generate whitebox models. The fact that both approaches work is shocking.
Speaking of simplicity and explainability, here’s an interesting anecdote in AI history: Back in the day, during the era of symbolic MT, one of the earliest deployed systems (Siemens' METAL) for English-German translation used only eight symbolic features (such as human, animal, etc.). The rules were simple, transparent, and easy to explain. This shows that extreme simplicity and rule-based transparency can work in some rough application scenarios (where English and German are linguistically close, making translation easier).
Later, we MT-ers expanded the number of features to the thousands, trying to cover more of the long tail. Even then, it wasn’t perfect. At the time, we thought that with enough effort, we could match the quality of statistical MT. But now, we know that even if symbolic MT could catch up and match statistical MT, it’s still far from competing with neural MT.
So, could we have continued refining features further? It wasn’t because we didn’t want to keep extending symbolic features (similar to one-hot encoding, but with the internal structure of ontology/taxonomy). We wanted to go beyond thousands to tens of thousands of features. But in reality, thousands (of features in size) were already reaching the limit of human experts’ capacity to understand (AI explanability), manage and debug. Expanding further would have been unmanageable.
Meanwhile, how many parameters do mainstream Transformer neural networks have? And the space and granularity they represent are on a completely different scale. Given the vast difference in scale between the two, it’s natural to doubt any efforts to bridge this gap for AI explanability. How could that even be possible?
That’s why I’ve always felt that explainability in large models is an elusive goal. But Professor Ma is telling the world that they’ve achieved it.
我们后来把 8 个 features 扩展到千数量级，才擦了长尾的屁股。但也没擦干净。当时觉得，也许认真做可以对垒统计MT的品质（与董振东老师谈过，我们都觉得可以在翻译上最终用符号打败统计的，只是需要时间磨细活），但现在知道即便匹敌了统计MT，也远远不能与神经MT比高下。
那就把 features 往细做，成不？不是因为我们不想继续把 symbolic features （类似于 one hot encoding，但人为在 features 内部强加了类似于 HowNet 的 ontology/taxonomy 的结构性），从千这个量级进一步提升到万的量级。实际情况是，千几乎已经达到专家人脑的极限了，再扩大 features 的范围，我们就无法掌控和调试了。
下一个passion点 应该是 to b 场景，因为最终的应用大期待，大概率在垂直。To c 虽然很卷，但路线图和态势，能做什么，包括 aigc，已经基本清晰。但 to b 还在泥潭里挣扎，方向都还隔雾看花，闪闪烁烁，但也看到高人。例如白硕老师，感觉他就在捻须微笑，坐在金融交易的莲花池上，仗着to b 积淀。
I’ve now become the go-to expert for AIGC (AI-generated content) "custom services" among my old friends and classmates, just for fun. Below are nostalgic videos made from old photos that two of my classmates asked me to create.
Whenever I find the time, I’m more than happy to provide this kind of emotional value for friends and family because it’s truly satisfying to see their reactions of surprise.
The pianist is now a world-class piano master, frequently touring and performing in Europe, America, and China. These are precious old photos of him practicing and performing with our mutual friend, Brother Sun, in Philadelphia back in the early days.
Dr. Bai Shuo, a seasoned expert in NLP and a multi-talented musician, commented humorously: “Looks real for someone who pulls on the bow in Meditation as named, but the bowing and fingering are all wrong.”
Another old friend also left feedback noting that the visual model doesn’t understand music: "This needs improvement! It's obvious that the model was created by someone who doesn’t know how to play the violin or piano. The bowing and piano accompaniment are off. The first note has a two-and-a-half beat long tone, which should be played with a long bow. Additionally, the pianist’s right foot should never be raised or shaking like that—it should be on the sustain pedal.”
LOL
Even though the music's name Meditation was clearly specified in my prompt during generation, there is no model, in the foreseeable future, that can truly align the understanding of music with the intricate details of bodily movements during performance. Perhaps this can be reserved as one of the ultimate challenges for large models aiming for AGI, because theoretically, if enough alignment data of musical performance is available, based on the compression theory of "joint training", it’s possible to aim at perfect alignment across different modalities.
If simulating the objective world is the ultimate goal of visual models, then the current generation of visual models is at the level of “playing the piano to a cow” or “playing music to a tone-deaf audience”—completely unable to withstand scrutiny from musicians. For example, as someone with little musical knowledge, when I watch the nostalgic performance videos above, I wouldn’t notice the flaws as an expert would; instead, I find them vivid and emotionally engaging.
Of course, the standards of musicians might as well just be a "pseudo-demand" or a pseudo-goal (even if the visuals satisfy the picky “expert eye,” so what? Will it sell well?). It might not be worth the effort to pursue this. However, in theory, an ideal AGI should be capable of meeting these expert-level demands.
This is the challenge of musical performance alignment. Another challenge to Sora-like video generation models is character consistency in videos.
Achieving facial consistency in generative visual models is an extremely difficult. Don’t expect this issue to be resolved by video generation models alone in the short term, especially not through autoregressive methods.
Human eyes are extremely discerning with regards to face recognition, especially when it comes to familiar faces of friends and family—you can immediately tell when a character's appearance is off. For example, while playing with old photos recently, I used the KeLing model (top notch Video Model in China) to generate a video of myself. At the 5-second mark, it still looked passable, but by 10 seconds, it no longer resembled me.
10 second footage:
In the second 10-second video, just a slight turn of the head, and it’s no longer me—it looks more like my brother. How can a model handle such fine details? Especially when the starting image for video generation is not even a straightforward frontal shot, making the character information incomplete—how could it not go off track?
While the videos I've made for friends and family using KeLing during its public testing phase have generally been met with passionate surprise and amazement, most of them suffer from this issue of character consistency, which is a regret.
The current one-click video generation products on the market (including our own YuanChuang Island recently launched) tend to mainly use anime or manga styles. This is to avoid user scrutiny since these styles lack 3D distinct individual characteristics. As long as there is consistency in attire, no gender mix-ups, with age and race alignment, most people will accept it. The current one-click videos are generally rough, with entertainment value primarily in the story rather than character portrayal akin to a Hollywood blockbuster. However, as this path progresses, it will inevitably encounter the challenge of maintaining the consistency of digital IP actors and their roles.
My colleague, Lu, mentioned, "the consistency issue might require cross-checking from multiple video angles, which more or less touches on the core issue of whether modeling is necessary."
Indeed, some form of cross-checking is required, not just monotonic correction over time/sequence—that is indeed the key. There’s a need to decouple or separate the character's image from the storyline, rather than generating in a linear, one-way path. While sequence learning has indeed produced miracles in LLMs, sequence generation inherently has limitations, including random deviations over time. Although it's not as extreme as LeCun's criticism—where he says GPT's error accumulation is a tiny discrepancy that leads to a significant miss—his claim isn't entirely accurate because GPT's autoregressive operation also corrects and adjusts its course at every step in the context. Nevertheless, when it comes to fine-grained consistency, random deviations are almost impossible to handle, even with corrective mechanisms in place.
Hence decoupling, decoupling, decoupling! Decoupling can solve the problem. The world isn't limited to sequences. Beyond sequences and time, there is a constant abstraction (i.e., character image, or IP) that can be utilized. This is becoming increasingly clear. Take, for example, the digital IP character Maria (Xiao Ya) that I created using AIGC txt2img more than 2 years ago:：
Unless they’re fans, perhaps my numerous Maria videos might cause aesthetic fatigue—someone even called her “Dr. Li's fairy” (LOL). But indeed, there are fans; several of my old classmates are among them.
Why? Because she is an IP, and she has been decoupled.
现在市面上做的一键成片产品（包括我们的元创岛），其所以用二次元或其他夸张风格为主，是为了避免用户的挑剔，因为那些形象缺乏鲜明的个性，不是真正的 individual IP，只要保持穿戴一致性，男女不要错位，年龄和种族不要相左，一般人也就接受了。目前的一键成片普遍比较粗线条，娱乐价值更多是为视频里的故事，而不是好莱坞大片那样的角色形象刻画。但这条路往上走，就躲不开这种 digital IP 的演员角色定位及其一致性问题。
Overall, CRATE is similar to a transformer, with two differences:
in each attention head, the Q,K, and V weight matrices are weight-tied, i.e., set to be equal;
and the nonlinearity following each attention layer is no longer a multi-layer perceptron (MLP), but rather a more structured operator (ISTA) with sparse outputs.
咱们了解一下，ISTA（Iterative Soft-Thresholding Algorithm，迭代软阈值算法），是一种用于解决稀疏优化问题的算法，在机器学习领域有广泛应用。在CRATE架构中， ISTA被用来替代传统Transformer中的多层感知器(MLP)，还记得前不久的 KAN 的创新也是旨在平替 MLP。都是在 Transformer 里面开刀。
我浅薄的理解，ISTA 与 KAN for Science/Physics 的思路是一致的，就是要经过某种正则化或剪枝，最终拟合成稀疏化路径，从而获得可解释性。
工作原理: ISTA通过迭代的方式逐步接近问题的最优解。每次迭代包括两个步骤: a) 梯度下降步骤，这与主流同；b) 软阈值操作。增加这个操作，是要在两个目标之间找平衡：
Professor Ma is a prominent figure, renowned for his distinctive style and leadership in the field. His name is widely recognized and respected. Of particular interest recently are his critiques of mainstream large models and the bold claims he has made about his own work (see his post in Chinese below).
Recently, at a conference in Shenzhen (which I attended with my own talk too), Professor Ma sharply criticized mainstream large models, Ilya, and Kolmogorov complexity theory, dismissing them as being on the level of high school students and claiming that they lack a true understanding of theoretical concepts. He asserted that he has achieved breakthroughs in both theory and practice, particularly with the white-box Transformer developed by his team. According to him, this model not only demystifies the complexity of large models but also offers an engineering-feasible alternative.
When someone speaks with such confidence, it usually indicates genuine expertise and a commanding presence. Just as Yann LeCun in the U.S. criticized GPT as being inferior to a dog and called it a dead end, proposing his world model as an alternative, China has Professor Ma. Their critiques balance the global discourse, making the world feel less excluding. There is indeed hope that their work might address the "slow thinking" and "interpretability" shortcomings of current mainstream large models and contribute to the overall advancement of AI. Professor Ma’s academic and practical work deserves close study, though we may have to wait for time and peer reviews to fully test and validate their findings.
At the Shenzhen conference, after delivering his talk and sharp critiques, Professor Ma left immediately, likely due to his busy schedule.
The paper is over 100 pages long and is said to be released in a few days. Based on the current outline, the key points are as follows:
Overall, CRATE is similar to a transformer, with two differences:
- In each attention head, the Q, K, and V weight matrices are tied, i.e., set to be equal.
- The nonlinearity following each attention layer is no longer a multi-layer perceptron (MLP) but rather a more structured operator (ISTA) with sparse outputs.
Let's examine ISTA (Iterative Soft-Thresholding Algorithm), a widely used algorithm for solving sparse optimization problems in machine learning. In his CRATE architecture, ISTA replaces the traditional MLP in Transformers. Not long ago, KAN also introduced innovations aimed at replacing the MLP, both approaches representing surgeries within the Transformer architecture.
In my understanding, ISTA and KAN (for Science/Physics) share a common goal: through regularization or pruning, they ultimately fit a sparse path, thus achieving interpretability.
How it works
ISTA Iteratively approaches the optimal solution of a problem. Each iteration involves two steps: a) a gradient descent step, which aligns with mainstream methods; and b) a soft-thresholding operation. This operation is added to balance two objectives:
a) Maximizing model accuracy;
b) Achieving model sparsity, i.e., simplicity (as overly complex models are difficult for humans to interpret).
The soft-thresholding operation encourages internal elements to become zero, resulting in sparse outputs and increased interpretability. The weight-tied attention mechanism, combined with ISTA, promotes a deeper understanding of the input data structure, resembling a human-like structured analysis process that prioritizes key elements while regularizing the data.
Professor Ma claims that these two modifications naturally lead the model to learn the interpretability associated with human-like structuring and sparsity during supervised learning (and later as claimed successfully applied to self-supervised learning too).
For example, in image recognition, it was observed that certain attention heads correspond to different parts of animals. What's more remarkable is that this correspondence remains consistent across different animals and even different categories of animals. For instance, an attention head focused on the "head" consistently pays attention to the head area when processing different kinds of animals. This consistency suggests that CRATE has learned a general representation of visual features across categories.
However, those studying LLM interpretability have long discovered that at the end of MLP networks, various structured components (such as heads and feet) are also captured by attention mechanisms. Without this, it would be difficult to explain the generalization (or compression) capabilities exhibited by LLMs. The challenge lies in the early stages of the MLP network, where attention is more mixed, and mainstream researcher struggle to clarify what the attentions heads are focusing on. It seems that they are vaguely paying attention to the relationships between basic elements like pixels/dots and lines.
The core idea behind explainable AI is consistent: transforming the tangled, black-box, multi-layer network's internal data fitting paths into structured paths that are enabled with various constraints and pruning, leading to a sparse representation.
Who wouldn’t want a model to be interpretable? However, achieving sparsity and simplicity is extremely challenging, which is why, so far, these approaches have struggled to compete with the black-box methods that involve randomness.
Professor Ma’s confidence stems from the fact that, in the past six months to a year, he has begun to train models using the explainable white-box methods mentioned above, achieving results comparable to traditional transformers. At the Shenzhen conference, he mentioned that while he had always been confident that this was the correct approach, he remained cautious until results were obtained. Now, he believes that his cross-national team’s achievements with this approach have satisfied him enough to announce to the world that he has found a breakthrough in theory as well as practice, the correct method for white-boxing transformers, which could lead to a paradigm shift and a breakthrough in deep learning. This has made him both excited and confident. Therefore, he is no longer content with academic theoretical achievements alone; he feels compelled to take actions in industry as well. Professor Ma has recently founded a company to advance this work on an engineering level. At Shenzhen, he announced a directionally significant project challenging the mainstream, first time under the banner of his new company.
However, based on my years of NLP experience and intuition, I must point out a challenge (or potential issue): Human interpretability is built on a highly simplified finite set. If we consider symbolic features, a feature system with more than thousands of elements becomes incomprehensible to humans. But on the other hand, the number of parameters in transformers and the number of KQVs for attention heads are on a completely different scale. Reducing such complexity on this scale seems almost unimaginable.
KAN for Science succeeded because their target was extremely narrow—certain existing symbolic formulas in physics or potential formulas limited to a few parameters. With such a goal, pruning, along with scientist intervention or feedback, allowed KAN to claim interpretability.
Regardless, Professor Ma seems confident, so we would like to observe how his methods and results evolve and will, or will not, be accepted.
我们的目标客户群体包括内容创作者（ToPC，to professional consumer）和小型至中型企业（ToSMB，to small medium businesses）。内容创作者愿意为方便他们工作的工具付费，而我们正是为他们提供这样的工具。对于ToB客户，我们专注于为中小企业提供较为标准化的解决方案，因为大型客户的定制化需求较为复杂，不易操作。目前，我们拥有86万付费用户，这证明了我们的服务已经成功落地并得到市场的认可。下面是我们产品的一些展示。
Notes on the 92-page Paper Released with Meta's Super Large Model Llama 3.1
The super-large model Llama 3.1 is a milestone in the open-source large model community. As a leader, Meta's project involved over 500 participants/contributors (the authors of this paper are listed alphabetically in the appendix, similar to how the Central Committee members' names are displayed by stroke order). This original text is full of implementation details:
AIGC MV using Suno and keling （just for fun & cheering opensource milestone）
Notes：
Llama 3.1 doesn't use sparse techniques, it's not a multi-expert system like model 4, but a dense model.
405B parameters, 15.6T tokens: The number of tokens is 40 times the number of parameters. Large-scale top models now emphasize data growth far exceeding parameter growth. Is this 15T tokens of data open source? (No, because even if they were willing to open source it, they wouldn't dare, as it could lead to countless data infringement lawsuits)
Emphasizes three major levers for super-large foundation models: data, scale, and managing complexity.
Compared to the previous generation system Llama 2, computational power has increased 50 times (using 3.8 × 10^25 FLOPs).
Complexity management: (1) Choosing a standard dense Transformer architecture instead of a mixture of experts model to maximize training stability. (2) Adopting a relatively simple post-training procedure: Supervised Fine-Tuning (SFT), Rejection Sampling (RS), and Direct Preference Optimization (DPO). In other words, algorithm design and implementation tend towards simplification. Not using sparse techniques and multi-expert systems is for stability (but training challenges are greater, though they're not afraid). Using simpler, easier-to-implement DPO in the post-training phase instead of reinforcement learning is also for stability, as reinforcement learning has always been difficult to handle.
Benchmark tests cover: general, code, math, reasoning, tool use, long context, and multilingual. All performances are SOTA (state-of-the-art international level).
MMLU (Massive Multitask Language Understanding): 405B model achieves 87.3% (5-shot), 88.6% (0-shot, CoT).
Code generation (HumanEval): 405B model reaches 89.0%, close to GPT-4.
Math problems (GSM8K): 405B model achieves 96.8%, slightly higher than GPT-4.
Long context tasks: Excellent performance on some tasks, such as 95.2% on QuALITY.
Multilingual tasks (MGSM): 405B model reaches 91.6%, on par with top models. The 405B model is comparable or close to GPT-4 and Claude 3.5 Sonnet on many tasks. In short, open-source has caught up with closed-source.
Pre-training started with an 8k window, expanded to a 128k window in the later stages of pre-training (continued training).
After the foundation model pre-training was completed, multiple iterations of alignment "post-training" were performed. Including: (1) Aligning the model through human feedback, including multiple rounds of Supervised Fine-Tuning (SFT) and Direct Preference Optimization (DPO); (2) Integrating new capabilities, such as tool use; (3) Enhancing coding and reasoning abilities (specialized optimization); (4) Safety alignment.
Multimodal expansion (in progress, not yet released): Image, video, and speech capabilities. Including (1) Multimodal encoder pre-training: Image encoder trained on a large number of image-text pairs, aligning visual content and natural language in a unified space; (2) Speech self-training? (3) Experiments on video-text data alignment based on images.
Language model as the core, other modalities are added later (whether added to pre-training and/or post-training). When expanding to multimodal, the language model parameters remain unchanged, adapting to multimodality, allowing multimodal alignment in the same semantic space, closer to the language model. In other words, Llama follows a modular, step-by-step approach to gradually expand to multimodality. This is not the mainstream approach (mainly referring to Open AI and Google, at least in theory) advocating for "unified multimodal native data joint pre-training". The overall impression of Llama's algorithmic strategies is seeking stability rather than innovation or unification. It tends towards practicality, not caring about leading in algorithms. For example, the integration of speech first involves speech self-training (because speech is actually very similar to text, both being language systems), then alignment between speech and text (including Automatic Speech Recognition ASR and Text-to-Speech TTS). Integrating step by step into the cross-modal large model, this approach isn't cutting-edge in terms of advancement, but it's steady progress, beneficial for engineering development, integration, and iteration. It's unclear when they will be able to release multimodal capabilities online.
Data collection and cleaning work is very complex, but the Llama team is meticulous, which is also the data guarantee for its quality to catch up with SOTA. To recap: (1) De-duplication: URL-level de-duplication; Document-level de-duplication using MinHash algorithm; Row-level de-duplication: removing rows appearing more than 6 times every 30M documents. (2) Filtering: Removing low-quality documents, outliers, and excessively repetitive documents, using repetitive n-gram coverage to remove repetitive content (such as logs or error messages); using "dirty word" counts to filter adult websites not covered by blacklists; using token distribution KL divergence to filter documents with too many abnormal tokens. (3) Controlling data quality: Using fasttext classifier to identify text that might be cited by Wikipedia; using a Roberta-based classifier trained on Llama 2's predictions; using DistilRoberta to generate document quality scores. Also, fasttext language classifier can identify 176 languages; specially filtering two types of information: adult content and personal identity/privacy information. Special fine processing for code and math web pages.
Data proportions: For example, downsampling over-represented data categories on the web (such as art and entertainment); data mixing ratios determined by a series of small model experiments, final data mix summary: About 50% of tokens correspond to general knowledge; 25% of tokens involve math and reasoning; 17% of tokens are code; 8% of tokens are multilingual content.
Model architecture: Apart from empirical detail adjustments, the basic architecture of the dense model remains unchanged, so it's data and scaling that create top models. 405B model specific parameters: 126 layers; token representation dimension 16,384; 128 attention heads; model size of 405B determined according to scaling law, about the computational optimal size under 3.8 × 10^25 FLOPs training budget.
Vocabulary: Using a vocabulary of 128K tokens. Combines 100K tokens from the tiktoken3 tokenizer and 28K additional multilingual tokens to better support non-English languages.
Computing resources, including GPU clusters of tens of thousands of cards, massive storage, and high-speed networks, represent huge resource investments. Specific data as follows: Computing resources:
Used up to 16,000 H100 GPUs (a very powerful graphics processor).
Each GPU has 80GB of high-bandwidth memory, with a power of 700W.
These GPUs are installed on servers designed by Meta itself, with 8 GPUs and 2 CPUs per server. Storage system:
Uses a distributed file system called Tectonic.
Provides 240PB (1PB=1000TB) of storage space, distributed across 7,500 servers.
Can process 2TB of continuous data per second, with a peak of 7TB/second.
A major challenge is handling the large amount of burst writes generated when processing model checkpoints (the process of saving model states).
Three-step pre-training process: a) Initial pre-training; b) Long context continued pre-training; c) Annealing with high-quality data sources Key pre-training strategies:
Gradually increase batch size and sequence length to balance stability and efficiency.
Dynamically adjust data mixing to specifically enhance certain capabilities.
Increase context length in stages to avoid early computational overhead.
Use annealing and high-quality data in the late stages of training to fine-tune model performance.
[LLM Summary]
Llama 3: Meta's Open-Source Large Language Model Breakthrough**
1. Introduction and Overview
Meta has introduced Llama 3, a series of foundation language models designed to support various tasks including multilingual processing, programming, reasoning, and tool use. This model series includes versions with 8B, 70B, and 405B parameters, with the largest 405B parameter model adopting a dense Transformer architecture and supporting context windows of up to 128K tokens. The development of Llama 3 highlights three key factors: data quality and scale, computational scale, and complexity management.
2. Model Architecture and Pre-training Strategy
2.1 Model Architecture
Llama 3 retains the standard dense Transformer architecture rather than adopting a mixture of experts model. This choice aims to maximize training stability, reflecting Meta's emphasis on simplifying design to manage complexity. Key architectural improvements include:
- Using Grouped-Query Attention (GQA) mechanism, with 8 key-value heads per attention layer.
- Introducing attention masks to prevent self-attention between different documents in the same sequence.
- Expanding the vocabulary to 128K tokens, combining 100K tokens from the tiktoken3 tokenizer and 28K additional multilingual tokens.
- Increasing the RoPE base frequency hyperparameter to 500,000 to support longer contexts.
2.2 Pre-training Data Processing
Llama 3's pre-training data processing is extremely rigorous, including:
- Multi-level deduplication: URL-level, document-level (using MinHash algorithm), and row-level deduplication.
- Heuristic filtering: Removing low-quality documents, outliers, and excessively repetitive content.
- Model-based quality filtering: Using fasttext and Roberta-based classifiers for quality assessment.
- Special content processing: Developing specialized processing pipelines for code and mathematical content.
- Multilingual data processing: Using fasttext base language identification model, supporting 176 languages.
- Safety and privacy protection: Filtering website data containing personally identifiable information (PII) and unsafe content.
2.3 Pre-training Strategy
The pre-training process is divided into three main stages:
1. Initial pre-training: Conducted on about 15T multilingual tokens, far exceeding Llama 2's 1.8T tokens.
2. Long context pre-training: Gradually expanding from initial 8K tokens to 128K tokens context window.
3. Annealing phase: Fine-tuning with high-quality data in the final stage, using Polyak averaging to generate the final model.
Data mixing ratios are carefully designed:
- 50% general knowledge
- 25% mathematics and reasoning
- 17% code
- 8% multilingual content
3. Training Infrastructure and Challenges
3.1 Computational Resources
- Using up to 16K H100 GPUs, each equipped with 80GB HBM3 memory.
- Adopting a 4D parallel strategy: tensor parallelism, pipeline parallelism, context parallelism, and data parallelism.
3.2 Storage System
- Using the Tectonic distributed file system, providing 240PB of storage space.
- Supporting 2TB/s sustained throughput, with peak capacity of 7TB/s.
3.3 Network Optimization
- Developing the NCCLX communication library to improve network efficiency.
- Designing specific network topologies and load balancing strategies.
3.4 Training Challenges
- Experiencing 466 job interruptions during the 54-day training period, 419 of which were unexpected.
- Developing automated systems and specialized tools to handle hardware failures and network issues.
4. Post-training and Alignment
Llama 3 adopts a multi-round iterative post-training process, including:
1. Supervised Fine-Tuning (SFT)
2. Direct Preference Optimization (DPO)
3. Reward model training: Using human feedback data
4. Safety alignment: Implementing multiple rounds of safety measures
This process not only improves the model's instruction-following capabilities but also enhances safety and specific abilities (such as coding and reasoning).
5. Multimodal Expansion
Although not officially released yet, Llama 3 demonstrates promising multimodal capabilities:
- Image recognition: Training independent image encoders, integrated with the language model through adapters.
- Video understanding: Adding video adapters based on image adapters.
- Speech processing: Independently training speech encoders, then aligning with the language model.
This modular approach allows flexible addition of new modalities while maintaining core language capabilities.
These results indicate that Llama 3 405B is comparable or close to GPT-4 and Claude 3.5 Sonnet on multiple tasks, particularly excelling in document understanding and long context tasks.
7. Safety Considerations
Meta highly prioritizes safety in the development of Llama 3:
- Implementing strict safety measures in both pre-training and post-training stages.
- Developing the Llama Guard system-level safety solution.
- Conducting extensive red team testing and risk assessments.
8. Open Source Impact and Future Directions
Meta's decision to publicly release the entire Llama 3 series, including the 405B parameter version, may have far-reaching impacts on the AI research community:
- Promoting open, responsible AI development.
- Accelerating AGI research progress.
- Providing researchers with opportunities to examine and improve large-scale language models.
Future development directions may include:
- Further improving multimodal integration.
- Expanding context length.
- Continuously enhancing data quality and model scale.
9. Conclusion
The development of Llama 3 demonstrates Meta's deep experience and forward-thinking in large-scale AI systems. By focusing on three key levers - data quality, computational scale, and complexity management - Llama 3 has reached or approached the current state-of-the-art level on several key benchmarks. Its open-source release may drive a wave of innovation across the entire AI field, paving the way for responsible AGI development.
Llama 3: Meta's AI Chef's Latest "Divine Delicacy"
Attention, all tech enthusiasts! The Michelin three-star AI chef Meta has just unveiled a new dish! This divine delicacy named "Llama 3" is not only spicy enough but will elevate your taste buds to new heights!
1. The Chef's Secret Weapon
Imagine Llama 3 as a super nanny who speaks 8 languages, writes code, does math, and can be your personal assistant. She can handle a kindergarten full of rambunctious kids (8B version), manage a mid-sized company (70B version), or even govern a small country (405B version)! This 405B big sister can remember 128,000 "gossips" (oh no, I mean context) simultaneously, essentially a walking encyclopedia + supercomputer!
2. Ingredient Selection: Only the Freshest!
Llama 3's chefs are masters at picking ingredients:
They "fished" 15 trillion words from the internet, nearly 10 times more than the previous generation!
Half of these words are everyday life seasonings, a quarter are math problems and brain teasers, nearly a fifth are programmer spells, and the rest are dialects learned from world travels.
They even invented a super weed remover, filtering out all the online garbage, repetitive, and unhealthy stuff.
3. Cooking Process: Three-Step Stir-Fry Method
Step 1: "Slow Simmer" - Start with a regular stove (8K context) to cook it halfway. Step 2: "High Heat Stir-Fry" - Switch to a super stove (gradually increasing to 128K context), reducing the sauce to be thick and fragrant. Step 3: "Low Heat Finish" - Finally, a gentle simmer with the best ingredients, the legendary "annealing" (even the chefs don't know why it's called that), bringing the flavor to its peak!
4. Kitchen Equipment: Top-of-the-Line Luxury Version
16,000 super high-power induction cookers (H100 GPUs) firing simultaneously!
A refrigerator that could fit half the Pacific Ocean (240PB storage)!
A proprietary ingredient prep system faster than 5G (NCCLX communication library)!
Imagine all these stoves firing at once, making the kitchen feel like a sauna. But our chefs persevered through the heat, changing chef uniforms 466 times in 54 days to whip up this dish!
5. Training Method: Both Cute and Well-Mannered
Being a good cook isn't enough; you've got to have manners too! So our chefs began a long "training" process:
First came a round of "gentle education" (supervised fine-tuning)
Then the "carrot and stick" tactic (direct preference optimization)
Finally, they invited moral role models (safety alignment) for guidance
After all this fuss, Llama 3 not only cooks well but also knows how to please people, program, do math, and mind her manners - a true decathlon champion!
6. Special Side Dishes: Showcasing Multiple Talents
Don't think Llama 3 can only cook; she's a multi-talented "goddess":
Storytelling from images? Piece of cake!
Writing movie reviews? No problem!
Recognizing songs and even singing a bit? The karaoke queen!
Although these "talents" are still in practice, they already show the potential of Li Bai's "from black hair to snow white in a day"!
7. A True Powerhouse: Dazzling Test Scores
Llama 3 participated in a series of "Top Chef Competitions," with eye-popping scores:
College Entrance Exam (MMLU): 87.3 points (out of 100)
Programmer Interview (HumanEval): 89 points (out of 100)
Math Olympiad (GSM8K): 96.8 points (out of 100)
Long Novel Reading Comprehension (QuALITY): 95.2 points (out of 100)
Bring this report card home, and even a "Tiger Mom" would be grinning from ear to ear!
8. Safety First: AI's "Security Captain"
Meta's chefs know well the principle of "don't leave guns and ammo lying around." They've assigned Llama 3 a 24/7 bodyguard team (Llama Guard) to prevent her from accidentally saying or doing the wrong thing. They even arrange occasional "moral exams" to ensure she doesn't turn into a "Terminator."
9. Open Source Feast: Everyone Can Be a Master Chef!
The most impressive part is that Meta decided to make the recipe for this "divine delicacy" completely public! It's like a Michelin three-star restaurant putting their signature dish's recipe online. Now anyone who wants to can whip it up at home! This move not only shocked other master chefs but also made countless food lovers cheer with joy!
10. Future Outlook: Reaching New Heights
Meta's chefs aren't resting on their laurels; they're already pondering the next "divine delicacy":
Maybe a dancing Llama 4?
Or a painting Llama 5?
Who knows, one day we might see a Llama 6 composing symphonies!
In short, the AI world's "Michelin" journey has only just begun!
Epilogue
The birth of Llama 3 not only elevates Meta's status in the AI world but also brings a fresh breeze to the entire AI research community. This bowl of "Llama soup" is not only delicious but also brings unlimited imagination to everyone. What will the future of AI be like? Let's wait and see what flavor the next "divine delicacy" will be!
-- looking closely into his historical Berkeley talk
by Wei Li, Jia Gao
Introduction
When Ilya Sutskever left OpenAI and re-emerged with his new company, SSI (Safe Superintelligence Inc.), the move was both surprising and expected—he bypassed AGI and directly aimed at SSI (Safe Superintelligence). He confidently declared: Superintelligence is imminent, and establishing safe superintelligence (SSI) is the most important technological issue of our time.
Ilya, a legend in the field of deep learning and AI, and the former true soul of OpenAI, was at the center of the dramatic internal shift, addressing the issue—effective acceleration versus super alignment. Why was Ilya so steadfast about "super alignment" amid the underlying AI values and strategic path debate? Even after the storm settled, the outside world continued to speculate: what did Ilya see that compelled him to join the board in making the decision to oust CEO Sam Altman? Ilya remained hidden until recently, when he left OpenAI, leading to the dissolution of his super alignment team and the creation of his new company.
What did he see behind the push for "safe intelligence"?
Back on October 3, 2023, Ilya gave a talk at UC Berkeley titled "A Theory of Unsupervised Learning." Though obscure and known to few, it is destined to be one of the most significant moments in AI history. This talk was a theoretical reflection and summary by a top expert in deep learning on the GPT model he pioneered, now famous worldwide. Ilya revealed the core principles of large models and vividly described his obsession with, and excitement over, independently understanding the mechanisms of unsupervised learning. Despite the complexity, the talk was brilliant and enlightening.
Until recently, Leopold Aschenbrenner, a former member of his super alignment team, published a 165-page article, "Situation Awareness," preliminarily revealing the shock and concerns within OpenAI over the exponential evolution of GPT models. This partly answered the question of what Ilya saw, but Ilya himself remained silent until his official re-emergence not long ago.
Reflecting on his "confessional" talk at Berkeley, we might glimpse his "moment of enlightenment" when facing potential superintelligence and understand his original intent for safe intelligence. It was a rare deep sharing by Ilya, attempting to convey essential message to the world. But did the world hear him?
1. Machine Learning: Supervised Learning and Unsupervised Learning
To accommodate readers with varying mathematical backgrounds, this blog aims to explain Ilya's historical presentation in an accessible language. Purely technical explanations can be skipped by non-technical readers without affecting the understanding of the presentation's main ideas.
Before diving in, let's review the basic concepts of machine learning. Machine learning is like having computers as students and humans as teachers. By providing computers with numerous "practice problems" and "answer keys," they slowly learn to solve problems. This is supervised learning. But can computers really learn from practice problems instead of merely memorizing them? Ilya assures us there's theoretical proof of this.
Imagine a sea of problems before you, each paired with a standard answer. This is the model's training data. Model training is like diligently solving these problems until most of them are correct, meaning low training error. But even an extensive problem set has its limits. When new problems arise, can the model still get them right? These new problems are the test data, akin to exams. Whether the model performs well depends on its test error rate.
Mathematics tells us that as long as the problem set is large enough, far exceeding the model's size, excellent performance on training problems (low training error) ensures good performance on test problems (low testing error). In other words, if the model trains well, it will do well in exams! This is the mathematical guarantee for supervised learning.
However, if the model merely memorizes without extraction, no matter how large its memory or how strong its "memory power," it lacks real adaptive learning ability (called "generalization ability"). Only when the model isn't too smart, it will be forced to extract the essence (called "compression"), learning real skills from the problem set.
This explains why the model size shouldn't be too large, to avoid giving the model too much room to cut corners. In short, Ilya wants to say that "big labeled data + low training error" is the winning formula for supervised learning, guaranteed by mathematics. This point has been confirmed both theoretically and practically. Since the deep learning revolution 12 years ago, countless successful cases have shown that as long as the training data is sufficient, neural networks can excel, at all sorts of AI tasks, from recognizing cats and dogs to machine translation.
But what about unsupervised learning? Can computers learn intelligence from a problem set without standard answers? It sounds far-fetched, but Ilya is about to explain how he managed to seek a solid mathematical foundation for unsupervised learning as well.
2. Distribution Matching: A New Approach to Unsupervised Learning
Everyone knows that machine translation was a typical win of supervised learning, in fact, the only win among various NLP tasks (such as dialogue, information extraction, sentiment analysis, question answering, docuent understanding, etc.) prior to the large language model's era. Why? Because we have a vast amount of historical bilingual data. It's like students having workbooks with English on the left and Chinese on the right—supervised learning thrives on this setup.
But what if the teacher suddenly stops providing aligned bilingual data and only gives you English books and unrelated Chinese books, leaving you to figure out how to align and learn automatic translation? That's the problem unsupervised learning needs to solve. Ilya says unsupervised learning can also handle various language machine translations (which we've seen today with large models—specialized translation software is no longer needed), and even any input-to-output transformation tasks. What's the catch?
Ilya discovered a new approach called distribution matching. Essentially, if the English and Chinese book collections are large enough, containing various sentence structures, their linguistic regularities will be learned "without supervision". For example, the context distribution of "I/me/my" in English should correspond to "我" in Chinese; adjectives near nouns in English with semantic compatibility should have a similar pattern in Chinese, etc. This provides the basic condition for potential language alignment.
Ilya points out that if two languages' native data is sufficiently rich, the input in one language can almost uniquely determine the equivalent translation in the other language. This principle applies not only to machine translation but also to tasks like speech recognition and image recognition.
Ilya independently discovered this approach in 2015, fascinated by the underlying mathematical principle—compression theory. If we can find a method that maximally compresses both English and Chinese data, this approach will capture the common patterns of the two languages, which form the basis of translation.
So, Ilya proposes that unsupervised learning is essentially about finding the optimal data compression method. This perspective not only sounds cool but also provides a mathematical explanation for the effectiveness of unsupervised learning. Although real-world tasks are not idealized, this principle gives unsupervised learning a solid theoretical foundation, making it as convincing as supervised learning.
Next, Ilya will delve deeper into the mathematical principles behind it. Although somewhat abstract, he promises it’s full of insights. We'll see how he uses the magic of compression to explain the mysteries of unsupervised learning.
3. Ilya’s Ultimate Theory: From Conditional Modeling to Joint Modeling
This is the final and most intriguing slide of Ilya's talk, worthy of thorough analysis and contemplation. The goal of unsupervised learning is often defined as "learning the internal structure of data." Ilya suggests understanding unsupervised learning from the perspective of data compression: a good unsupervised learning algorithm should maximally compress the data, representing its content in the simplest form. This introduces the concept of Kolmogorov complexity.
The Kolmogorov complexity of a data object is the length of the shortest computer program that can fully describe this object. You can imagine this shortest program as a "compressed package" containing all the information needed to reconstruct the original data. From this perspective, the goal of unsupervised learning is to find the optimal compressed representation of the data, which is the Kolmogorov complexity.
The Kolmogorov complexity of a data object is the length of the shortest computer program that can fully describe this object. Imagine this shortest program as a "compressed package" containing all the information needed to reconstruct the original data. From this perspective, the goal of unsupervised learning is to find the optimal compressed representation of the data, which is the Kolmogorov complexity.
However, in practice, we often need to handle multiple related datasets. For instance, in machine translation, we have the source language dataset X and the target language dataset Y. We want to learn a model that can translate sentences from X to Y (or vice versa). Traditionally, this is viewed as a conditional probability problem: given X, what is the probability distribution of Y? Represented in terms of Kolmogorov complexity, this involves finding K(Y|X), the shortest description length of Y given X.
Ilya proposes a different approach. Instead of viewing X and Y as condition and result, like in supervised learning, he suggests viewing them as a whole and compressing them together within a massive model. Essentially, we seek the joint Kolmogorov complexity K(X,Y), the shortest program length that compresses both X and Y simultaneously. This approach must fully utilize the correlation between X and Y, using information in X to automatically align Y (or vice versa), much like how we use our native language knowledge to understand and remember foreign language expressions.
Ilya believes this joint compression idea is the true power of unsupervised learning. Real-world data is often interconnected, with numerous deep common patterns and regularities. If unsupervised learning can discover and utilize these regularities, it can significantly enhance learning efficiency and generalization ability. This explains the remarkable performance of large language models like GPT across various tasks: through massive unsupervised pretraining, they learn the deep regularities of the training data, and these regularities are transferable across related datasets.
Although Kolmogorov complexity is theoretically uncomputable, Ilya believes we can approximate this process using deep neural networks (like GPT). Through optimization algorithms such as gradient descent, neural networks can find the optimal compressed representation in massive data, capturing the essence of the data and its alignment patterns, even if not strictly in terms of Kolmogorov complexity.
Thus, Ilya’s theory can be seen as a new paradigm for unsupervised learning, elevating traditional independent modeling (like separate models for English and Chinese) to a unified associative modeling approach. In this paradigm, the goal of unsupervised learning is no longer just compressing individual datasets but finding the connections between them. This cross-modality learning represents an advanced form of artificial general intelligence (AGI).
Now, let’s closely examine this final slide. In it, X represents dataset 1 and Y represents dataset 2. The key point is extracting every bit of information from X (or Y) to help predict Y (or X). This is what Ilya refers to when he says training X and Y together yields the effect that unsupervised learning of X helps accomplish the task of transforming X to Y.
The crucial idea is: K(Y|X) becomes K(X, Y).
Ilya transforms the universally applicable functional AI task of "input X conditions output Y" into an approximate solving problem by jointly training X and Y without modal segmentation. This joint training approach is effectively the current multimodal unified training, abbreviated as K(X, Y).
Ilya aims to strengthen the theoretical basis, emphasizing his surprising discovery that self-learning of X has a strong predictive effect on Y.
The essence of unsupervised self-learning is that the self-learning of X is to compress X, and the self-learning of Y is to compress Y. This is straightforward because the essence of self-learning is involves only positive examples, without negative samples. Unsupervised self-learning lacks a specific task orientation; it learns language from language, images from images, music from music, and so on, continually abstracting various patterns from phenomena.
Ilya points out in the slide: conditioning on a dataset, not an example. The compression object is the dataset, not individual data points, which is crucial. This distinction separates superficial compression from content compression. Superficial compression is merely a mechanical process that does not produce intelligence. Only content compression can achieve artificial intelligence.
How do we understand the difference and connection between superficial lossless compression (e.g., digital music) and content lossless compression (e.g., Suno)? Compressing a specific song losslessly aims to ensure it can be restored to its original musical form (including noise and imperfections). This is traditional music compression, targeting individual sample, e.g., a specific song. Compressing a collection of music, whether using GPT or Diffusion, targets a group of samples, resulting in a large model like Suno.
When individual objects turn into group objects, formal compression naturally transforms into content compression. This is because, although the group comprises individuals, compressing the group is like "painting" a portrait of the group, outlining its characteristics. It may resemble an individual, but it is not a specific individual in the original data; otherwise, it would not be a model but a memory repository.
This is understandable because the purpose of large model compression is to identify the characteristics and regularities of the dataset. The text generated by GPT-4 might seem familiar; the music generated by Suno might sound familiar; the videos generated by Sora might look familiar; the images generated by MJ might seem familiar. However, they are virtual individuals "restored" based on prompts, abstracted or compressed from big data: derived from data, higher than data, mingling with data, indistinguishable from real and fake.
Given that the compression object is the entire dataset content, how do we measure its effectiveness after decompression? What is the gold standard?
This standard is each sample itself. However, this is not entirely accurate; the standard could have equivalent answers, as the same content can have various ways of expressions. The implementation method is "masking", and NTP simply masks the next token. Training involves calculating the loss for each sample, using backpropagation with gradient descent to adjust parameters continually, eventually lowering the loss in the group training of the dataset to an acceptable point, forming the large model.
This final slide and Ilya’s explanation emphasize a core point: Conditional Kolmogorov complexity K(Y|X) provides a theoretically optimal solution for unsupervised learning. K(Y|X) is defined as the length of the shortest program that produces the output dataset Y given access to the input dataset X. It represents the theoretical limit of extracting all valuable information from X to predict Y. An algorithm that can achieve K(Y|X) would be the best for predicting Y using unlabeled data X.
This can be seen as the theoretical basis for large models performing various language translations. Each language is potentially X and potentially Y. After self-learning with an huge amount of data, LLMs learn the relationships between languages, possessing the potential to translate from X to Y.
In practice, the machine translation task, like other tasks, initially involves few-shot examples in instruction-following fine-tuning to define the task, ultimately triggering the internal power of large models to translate various languages. This internal power of unsupervised learning for various tasks is the theme of his talk.
However, K(Y|X) is uncomputable in practice. Ilya proposes a feasible alternative, using joint Kolmogorov complexity K(X,Y) (joint compression of X and Y). He believes K(X,Y) can achieve the same effect as K(Y|X) in practical machine learning tasks.
Let us stop and think again: conditional modeling is now replaced by sequence modeling by Ilya. The widely known probability simplification in traditional machine learning, such as the Markov chain, has a similar effect.
Conclusion
Ilya's historic presentation at Berkeley on the theory of unsupervised learning reveals the secret behind the mainstream of self-learning large models, especially GPT. It seems that Ilya, after long contemplation, finally disclosed this "heavenly secret" in a cryptic manner at Berkeley. Although the theory and its proof appear complex, it is crucial for understanding why GPT's sequence learning method ("next token prediction") has become a universal simulator for AI tasks.
Ilya exudes a genius prophet aura, with a lonely invincibility and high-altitude isolation, blending a sense of deep realization, compassion, and the pure, focused, and idealistic earnestness of a graduate student nerd.
He claims to prefer compression but does not emphasize so-called lossless compression. He leaves room for himself and the mainstream, proposing the concept of "no regret"—though GPT may not achieve lossless or perfect compression, it theoretically proves there is no better way: GPT is the closest to lossless, "no-regret" modeling.
When Ilya officially re-emerges to establish SSI, emphasizing a single focus, a single goal, and a single product—to use technology to ensure the superintelligence brought by large models is safe for humanity—he asserts: AI will be eternal, its birth akin to the creation of heaven and earth. As Ilya passionately discusses AI's progress, he is most qualified to declare and lead the "exciting yet dangerous journey towards AGI."
谈谈我的看法。从序列学习的方式上看，数据驱动的模型学习是以 case based 的归纳（也叫压缩）作为起点和主干的，这个没有疑问。问题是，case based 的学习，到了一定的程度和量级的时候，是不是会非常逼近 rule-based 的学习。承认后者就是承认了大模型具有某种逻辑推理能力。大模型具有初步的逻辑推理能力这一点在大模型主流社区中本来不是问题，而是默契的共识，大模型测试的一个重要维度就是逻辑推理能力。但在更大的范围内（非主流圈子以及普罗大众），一直还是作为疑问存在的。
一个有意义的视角是看泛化中外推的理解。对于非解析的、没有对应符号规则的现象，外推本质上是不可计算的，也就是只能碰运气了。出路只有收集相关数据，把盲区带入雷达屏，化外推为内插。但是对于有解析解的高度规则化的数据分布，外推能力是泛化学习的自然期望，达不到期望就说明llm只是一个鹦鹉。达到了期望， 就说明 llm 跳过了鹦鹉的门槛，学会了某种推理规则。现在看来，头部大模型是跨越了这个门槛，继续拿鹦鹉学舌来比况大模型，彰显的是人类盲目的狂妄自大。
要摈弃削足适履的思维定式。只要模型展现出符号规则类似的推理逼近的能力，就应该承认它学会了初步的推理。更本质的，它融会贯通，对于规律现象，可以达到外推的能力。其实，小语种之间的机器翻译能力，就是外推的结果，因为训练数据中严重缺乏相关的数据。
前不久引起关注的一项关于KAN模型的研究中，KAN 的 AI for science 实验，其实已经展示了模型如何数据驱动去逼近解析解，等于是把模型学习逻辑推理的内部过程图示化了，非常生动 ，有相当的说服力。当然，KAN的实验表明对于简单的解析解，数据驱动可以逼近符号规则，但并不轻易就得出符号规则。实验中是加入了人为的剪枝等操作才得出了数据背后的符号规则。
与此对照，深度学习大佬杨立昆却坚决否认GPT有逻辑推理能力。杨立昆语录： AGI is a complete nonsense；GPT is a deadend，等等。矫枉过正反潮流，把话说死，并不是坏事。但轻信他，也可能就被带进沟里去了。
[verse 1]
In Suzhou's June, beneath a scorching sky,
A madman's blade flashed, evil drawing nigh.
Mother and child cried out in desperate fear,
Their screams of anguish piercing far and near.
[chorus]
With verse we mourn, our grief in words conveyed,
A hero's tribute, never to fade.
[verse 2]
Before the school bus, Madam Hu stood tall,
Her gentle hands became a shield for all.
No tiger-wrestler she, no dragon-slayer,
But love unbounded made her their savior.
[chorus]
With verse we mourn, our grief in words conveyed,
A hero's tribute, never to fade.
[verse 3]
Her blood stained red the soil of Jiangnan,
White clouds and grieving grass bore witness, wan.
Though snuffed, her candle's light forever gleams,
Like brave Feng Yuan of old, her courage beams.
[chorus]
With verse we mourn, our grief in words conveyed,
A hero's tribute, never to fade.
[verse 4]
Why must the kind so often suffer woe?
When will justice's path smooth waters show?
We question Heaven, tears fall like the rain,
In silence seek life's meaning through our pain.
[chorus]
With verse we mourn, our grief in words conveyed,
A hero's tribute, never to fade.
[verse 5]
Madam Hu's name shall echo through the years,
Half-masted flags, a nation draped in tears.
Her love, transcending life and death's divide,
One selfless act, as sun and moon abide.
[chorus]
With verse we mourn, our grief in words conveyed,
A hero's tribute, never to fade.
[verse 6]
Rest now in peace, return to native ground,
Let not your family grieve, all hearts are bound.
In old Wu Gate, by Suzhou's storied streams,
We offer flowers and wine to honor dreams.
[chorus]
With verse we mourn, our grief in words conveyed,
A hero's tribute, never to fade.
现在我们来看看AI界的'前传'！在达特茅斯会议这场AI盛宴前，麦卡锡大佬就在偷偷摸摸写'剧本'啦！他的文章《The inversion of functions defined by Turing machines》可不是在讲怎么把图灵机倒过来用。这篇'天书'其实在讨论如何设计一台超级解题机器。麦卡锡想象中的这台神机，能解决所有明确定义的智力问题。这不就是AI的雏形吗？"
Charles Bennett提出了逻辑深度的概念，它考虑了生成一个对象所需的最短程序的运行时间。大语言模型的参数可以看作是模型内部存储的信息量。因此，将模型参数比作柯氏复杂度是合理的。大语言模型的推理时间比作逻辑深度也是合理的。
李明是滑铁卢大学的杰出教授，在信息论和生物信息学领域做出了卓越贡献。他将K氏复杂性从单个序列扩展到两个序列，不仅可以测量单个序列内的信息，还可以测量两个序列之间的信息，这对通用大模型定义万能任务及其非监督学习完成各种任务意义重大。他与Paul Vitanyi合著的《An Introduction to Kolmogorov Complexity and Its Applications》被认为是该领域的经典著作，对信息科学的发展产生了深远影响。
Marcus Hutter是一位物理学家出身的计算机科学家，他提出了AIXI通用人工智能框架，并认为语言建模本质上就是压缩。他将所罗门诺夫归纳用于解释智能体和强化学习，认为学习过程就是压缩过程，并致力于研究通用人工智能。
Open AI 前灵魂人物伊利亚在伯克利演讲中，揭示监督学习与非监督或曰自监督学习的联系。伊利亚声称他在2016年独立想到了所有监督学习可以被归约为自监督学习的观点，并追溯到K氏复杂度为基础的压缩理论。伊利亚笃信简单的自回归GPT模型可以在超大数据展现超级智能。
回顾一下模型发展的时间线：深度神经Transformer架构于2017年6月提出，BERT模型于2018年10月提出。OpenAI的GPT系列模型从2018年6月开始，陆续推出了GPT、GPT2和GPT3，现在到了GPT4，成为业界主流。
总结一下，所罗门诺夫归纳第一步是收集观察数据。第二步形成假设解释数据： 假设可以是一个图灵机或一个数据驱动的大模型。第三步进行实验验证。如果数据证伪，则返回步骤2形成新的假设。
大模型遵循的是所罗门诺夫归纳的路线训练模型及其推理应用。
记得奥特曼当时是 YC 的 CEO，他大概把 Open AI 包装成 YC 孵化出来的 AI 企业，老马作为联合创始人和当时最大的投资人，在这一点不太满意。所以老马说，博客（说的Open AI计划）听上去不错，如果做些调整让新公司更加中立，而不是以YC为中心。
现在我们知道，是 Open AI 打开了 AGI 的大门，开启了人类文明的新时代，但走通这条路到 GPT3 或 ChatGPT 的核爆炸时刻，实在是太幸运的极小概率事件了。
老马与奥特曼这两位 AI 圈外但又接近 AI 的先知，与 Ilya 这样的圈内顶级科学家，在 AGI 的信念上，很早就非常默契：他们在计划这件事的时候，没有任何自我怀疑，好像就在谈一个事关人类命运的必然发生的事情一样。他们后来的分歧只是在实现的方式以及资源的局限上，并不在 AGI 本身。要知道那个时代，全球科学家和知识分子全体，几乎100%是不相信什么通用AI这种“鬼话”的，但地球上就有这么几个人，坚信AGI，并且能气味相投，凑在一起为之谋划，并开始担忧人类文明的命运。
他们默契，并决定成立 Open AI，是出于对于 AGI 可能被垄断的担心。具体说，是担心谷歌称霸世界：当时的谷歌已经搞出了 Alpha-go/-zero，让他们感觉此事无法缓行，必须立刻动手，以开源对抗谷歌。老马一半出于公心（为人类文明的前途忧虑），一半出于私心（希望自己成为谷歌AI的挑战者领袖，而不是放任奥特曼这些年轻人来领导）。
他对这个AGI事业和他可能扮演的角色非常投入，愿意做背后的金主，一开始就让奥特曼把第一笔融资提高一个量级，明确说，任何融资亏空他都可以补齐，隐含前提当然是他是 CEO 和 leader，最好是控股老板。按照 business 逻辑，这是完全合理的，毕竟在那样的早期，这样烧钱的AI“曼哈顿计划”，也只有老马这样识货的人才愿意成为金主。现代社会的铁律是，谁有钱，谁当家。可是奥特曼不甘心，他与Ilya几个是实际工作中的 Open AI 创始人和 AGI践行者，不甘心只做 COO 而把 CEO/Chairman 让给这个几乎是唯一靠谱的大金主。
于是上演了这一出最后分手的戏剧：老马在得不到他想得到的 CEO 或让 Open AI 依附于 Tesla 之后，决定退出。没有惊人的定力，奥特曼是不可能敢于把金主放跑的。而老马在决定离开的时候，宣判了 Open AI 的死刑：你们成功的希望为0，他说。不是老马对 AGI 的成功有丝毫怀疑，而是他觉得离开了他，Open AI 无法海量融资，只有死路一条。他当时列举了苹果和Facebook，判断这两家不可能有远见给 Open AI 输血，他却漏掉了微软，可能是根本没想到微软有此可能，他小看了微软CEO的眼光。
奥特曼怎么吸引和说服了微软，那是另一个故事了。但当时的情况是，除了老马，有钱人几乎没人能看懂 AGI 和前途，业内人士也看不懂，Open AI 就是一帮“疯子”在异想天开。融资几乎不可能，那么奥特曼怎么敢与老马分手，而不委曲求全让位给老马呢？
谁知道先知和天才不仅仅就是这几个疯子，微软CEO萨蒂亚·纳德拉（Satya Nadella）也是，虽然他离 AI 更远。萨蒂亚与奥特曼的“勾搭”是人类历史上最具浪漫色彩的一章，需要冲破种种桎梏。
现在我们似乎理解了，微软今天能超越苹果成为世界企业首富，就是英雄创造的历史：萨蒂亚是不可思议的领袖。他的悟性和远见让 Open AI 与微软结合，这是一桩非常奇特的姻缘：一方投入巨资，另一方短期看不到希望，巨资投入也带不来任何董事会决定权，萨蒂亚依然前行。世界上找不到微软这样的对象，它几乎是彼时彼刻唯一可以牵手 Open AI，摆脱它必死宿命的救星。呼唤的与被呼唤的，在千载难逢的那个时间点，没有错过。
后来的故事，所有人都知道了：这个“姻缘”彻底改变了AI，更重要的是，也改变了人类文明的走向。
其他都是花絮了：老马以维护人类的名义起诉 Open AI 违背初衷；Open AI 披露早期信件来往证明老马本人就梦想控股，并不真正在乎开源还是闭源，而他们则依然不忘初心。
微软的地位其实很尴尬。一方面，现在知道他们对于 Open AI 的巨额投资，已经从股价的飞升中得到了足够的回报，所以从投资角度，萨蒂亚是微软的英雄。但另一方面，这个“婚姻”始终无法稳定，也难以建立恒久的互信。微软不得不给自己做 Plan B，而 Open AI 也有自己的 Plan B：都需要在两人分手的时候有所准备。Open AI 这种独一无二的公益实体控股企业实体的架构，改变了人类历史进程，但却天然有矛盾和不稳定。上次奥特曼被踢出而复返的危机会不会重演？奥特曼本人会不会成为 AGI 沙皇，违背初心，一意孤行？
