月度归档： 2025 年 9 月

Advocates of “unified” models argue that the many signals of the world—text, images, audio, video—should be tokenized and mapped into the same semantic field (a shared hidden vector space), so they can be trained jointly and modeled with one brain.

A common recipe is to atomize other modalities the way we do text: turn them into token sequences so that sequence learning—the secret sauce behind LLMs' explosive success—can extend beyond text. Universal tokens lay the groundwork for this extension. Once text, images, audio, and video are all sliced into computable tokens of similar type, one step remains: make them play in the same band (for training and for generation).

The most straightforward trick is to give each instrument a passport. Before entering the model, every segment carries a modality tag—special start symbols telling the conductor whether the next stretch is image tokens, audio tokens, or text (e.g., <image>, <audio>). Positional encodings then act like a seating chart, indicating how these tokens are arranged in a sentence, on an image grid, or along a timeline. After a few rounds of training, the model internalizes these hints: look at pixels when needed, listen when needed, and weave everything into a coherent narrative when it speaks.

A deeper level of fusion is handled by attention. Think of attention as a rehearsal room with glass walls: text can “glance” at image tokens and images can “nod back.” Over time, some heads specialize in image–image relations, others in image–text translation. Flamingo makes this explicit: cross-modal attention layers are inserted between LLM layers so that the word currently being generated can continually “look back” at the relevant image region. To users this appears as abilities for things like step-by-step visual Q&A; under the hood, a stream of text tokens moves forward while aiming its attention at precisely the visual patches that matter.

In engineering practice, a common compromise is front-end experts first, then merge. PaLM-E in robotics is a good example: images are first distilled by a pretrained vision encoder (e.g., ViT) into compact representations, then projected into perception tokens; text is tokenized as usual; robot state vectors can be appended when needed. With appropriate modality tags, all of these enter a shared Transformer backbone, where dialog, reasoning, and decision-making happen on a single score. In this analogy the front end acts like a pickup: it cleans and compresses raw signals.

More concretely: the “pickup” is a modality-specific encoder (ViT for images, an acoustic encoder for speech, etc.). The “front end” consists of the pickup plus a small projection/adapter that turns features into tokens the backbone can digest. The “backbone” is the unified Transformer that handles cross-modal alignment, long-range memory, and reasoning. In short, the backbone processes music, not the raw electrical currents of each microphone. It doesn’t touch pixels or raw waveforms; it ingests track-like, distilled features produced by the front end and aligns/reasons over them. Redundant detail and noise are filtered at the front; during generation, pixel-level detail is filled back in by a diffusion renderer or a decoder according to the backbone’s plan.

For instance, a 224×224 image has over 150k pixels. If you hand all of them to the backbone, attention wastes budget on repetitive textures. A ViT-style pickup patches, encodes, and compresses the image into roughly a couple hundred visual tokens—each like a short riff, concentrating edges, shapes, and relations that matter for understanding. Audio is similar: raw 16 kHz samples are first converted to tens or hundreds of latent vectors; rhythm and timbre are distilled, while noise and redundancy are filtered. The backbone consumes these high-semantic-density tokens (i.e., semantic tokens), so compute is lighter and crucial information remains intact. When trained end-to-end, gradients from the backbone flow back to the front end, encouraging it to keep what matters for downstream generation and discard redundancy.

Those few, information-dense tokens inside the backbone don’t carry pixel-level detail. Detail returns through a wider channel on the rendering side during generation. The backbone sets the plan; the renderer does the fine work. Three practical stages or approaches are popular in generation.

1) Composition → Orchestration. The front end is paired with a decoder: the encoder compresses content into compact codes (discrete codewords/tokens or low-dimensional latents), and the decoder can reconstruct pixels. At generation time, the backbone predicts/plans decodable representations—a string of  codes or a per-frame latent vector—rather than pixels. The decoder then “orchestrates” texture, light, and materials from those codes. In image/video, the VQ-VAE/MAGVIT-type follows this predict-codes → decode path. With residual vector quantization (RVQ), detail is added coarse-to-fine: the backbone first emits higher-level codes; the renderer produces a solid base; lower-level residual codes then refine it layer by layer.

2) Storyboard → Cinematography. The backbone provides structural plans—a low-res sketch (blueprint), motion hints/optical flow, keypoints, or a camera trajectory. Each frame is then handed to in-frame diffusion (latent-space rendering) to “develop” the image from noise under those conditions. Diffusion doesn’t need the backbone to carry high-frequency detail; it iteratively reveals detail in a high-res latent space. This is the “next-frame prediction + in-frame diffusion” split: temporal coherence by the backbone, visual richness by diffusion.

3) From coarse to fine. The backbone first outputs a coarse result—low resolution or higher-level codes—then the rendering stack applies super-resolution (SR) and residual refinement to stack up resolution and texture. The farther down the pipeline, the wider the bandwidth, but this bandwidth is handled on the rendering side rather than bloating the backbone with pixel-long sequences. Many systems expose these as configurable quality gears: stop at 720p if the user is in a rush, or climb to 1080p/4K when desired.

One subtlety: the backbone does not simply “discard detail.” First, in joint end-to-end training, the front end and decoder co-adapt with the backbone so that what’s “compressed away” is true redundancy, while cues crucial for generation (edges, materials, style, rhythm) are preserved in a recoverable latent space. Second, many renderers look back at multi-scale front-end features (e.g., via U-Net skip connections or cross-attention during decoding), allowing them to query high-bandwidth details on demand—without hauling them through the backbone.

How the renderer “looks back” depends on the task:

• • From-scratch generation (text-to-image/video): there’s no high-res reference to query. The renderer relies on learned statistics to “hallucinate” detail from the backbone’s directional plan; the text front end builds latents during training but typically isn’t invoked during sampling in generation.

  • Conditional generation/editing (image-to-image, inpainting, colorization, video continuation, style transfer): the renderer does “look back.” The reference image or previous frame is encoded to multi-scale features; the decoder/diffusion network uses skip connections or cross-attention to pull in high-res cues, aligning edges, textures, and brushwork precisely.

In the autoencoder/vector-quantization track, encoder and decoder are two sides of the same coin: the encoder compresses images/video into a shared latent language; the decoder restores pixels from latents/codes. They are trained around the same latent interface with opposite roles, by two networks that translate the same latent language. Whether the encoder is used at generation time depends on the task: unconditional generation needs only the decoder; conditional/editing and video consistency bring the encoder back to supply high-res detail.

Put together, the pipeline is clear: the backbone sets the plan—semantic coherence and causal logic—without hauling pixels across long distances; the renderer does the fine work; it looks back when a reference exists, and otherwise lets diffusion “develop” detail locally. Decoding, diffusion, and super-resolution are the high-bandwidth back end that rebuilds the scene in place. Encoder and decoder share a latent interface, each doing its half of the job—two sides of the same coin.

• 谈谈大一统模型的万象token与统一建模的背后
• 立委关于大模型与AI的博客汇总

“Codec” is short for coder–decoder: an end-to-end machine for compressing and then restoring audio. The encoder compressess a signal into a more compact representation; the decoder reconstructs it as faithfully as possible.

In the LLM era, audio—like text—is often cut into short segments and encoded as a sequence of discrete tokens. The “audio dictionary” used for quantization is called a codebook, like a spice rack in a kitchen: discrete little vectors in separate slots, each with an index. Unlike text, which typically has a single vocabulary, neural audio coding often uses several codebooks at the same time step (audio unit), quantizing from coarse to fine in layers—a scheme known as RVQ (Residual Vector Quantization). During compression, the system picks one entry from each codebook, using their indices to “remember” that instant of sound in layers; during reconstruction, the vectors addressed by those indices are summed, layer by layer, to restore and refill the flavor.

Earlier TTS pipelines typically did “Mel-spectrogram draft” (a continuous-valued “score”) → a neural vocoder that plays it back into waveform. The LLM-native neural codec more often runs “semantic tokens → acoustic tokens → decode to waveform.” Both semantic tokens and acoustic tokens are discrete; they differ in granularity and division of labor.

Multi-layer token coding via RVQ is a key innovation for extending LLM methods to audio. Without it, token counts would explode. By its nature, layering simplifies the complex—divide and conquer. It’s an innovation in representation that pushes discretization all the way through.

“How many layers?” There’s no universal number. It’s a knob you turn together with bitrate, latency, and quality targets. Take Google’s SoundStream: it refines the same time step with residual quantization. The paper reports common 4/8/16-layer setups and even an extreme 80-layer experiment, showing training can remain stable with many layers. For a fixed target bitrate, you can trade fewer layers with larger codebooks against more layers with smaller ones—the design space is flexible.

Meta’s EnCodec follows a “multi-codebook, variable-bandwidth” approach: the 24 kHz model uses up to 32 codebooks (i.e., 32 RVQ layers), the 48 kHz model up to 16, with codebooks typically of size 1024. During training, it randomly selects a subset of layers to participate in quantization, so a single set of weights can serve 1.5/3/6/12/24 kbps, etc. In deployment you simply “open” as many layers as you need, striking a balance between quality and realtime latency.

Don’t confuse “number of layers” with “hierarchical scales.” OpenAI’s Jukebox uses three time-scale levels in a VQ-VAE: the top level is sparse, the bottom dense, carrying information from long-range song structure down to timbral details. That’s “horizontal” stratification over time, not stacking multiple residual quantizers at the same time step.

A rule of thumb: for realtime, low-latency speech, 4–16 layers are common; for music or higher fidelity at 24 kHz, a few dozen layers aren’t unusual. The final choice isn’t doctrinaire—it depends on your target bitrate, acceptable latency, and how exacting the ear is about texture.

A neural codec begins with a neural encoder that massages raw waveforms into compact latent vectors; then a quantizer selects entries from the layered codebooks—coarse strokes for contour, finer strokes for texture; finally a neural decoder turns these discrete entries back into an audible waveform. The whole chain is trained end to end, aiming to tell the ear what it cares about—timbre, prosody, articulation, even a touch of room tail—using as few indices as possible. The result is a stable, reversible, bitrate-tunable discrete sequence.

In this setup, “audio tokens” are the indices themselves. Every few tens of milliseconds, the quantizer picks a set of indices from the layered codebooks to describe that slice; over time you obtain a readable, writable “acoustic text.” This differs from the traditional Mel spectrogram: the Mel is a continuous “photo” better suited to a conventional neural vocoder; audio tokens are a discrete “word string” that can both decode back to high-fidelity waveforms and be continued, edited, and aligned by GPT-style autoregressive models like text.

In one line: a codec is the whole compress–restore machine; the codebook is just the rack holding the discrete basis vectors—what counts as tokens are their indices. A neural codec rebuilds this machine with neural networks so it can describe a sound using a string of discrete tokens and, when needed, sing that string back into real speech or music. Put simply, audio processing can now reuse the compression–and–reconstruction playbook that has proven so powerful for text LLMs—that’s the secret behind audio’s takeoff in the era of large models.

As large language models (LLMs) expand into audio, progress has been breathtaking. “LLM-native” speech technology reached practical maturity roughly half a year ago, and the entire industry has surged forward. Two features mark this maturity: ultra-realistic speech and full-duplex interaction.

Audio tokenization—akin to what we do with text—has not only produced “model musicians” (Suno is the signature product here; I’ll cover it in a separate tech blog), but more importantly, it has catalyzed two changes that truly transform language interaction: ultra-realistic speech synthesis, and natural conversations where the system can listen and speak at the same time and be interrupted at any moment—that is, full duplex.

Start with ultra-realism. Old-school TTS always sounded like a well-trained announcer—articulate and polished, yet lacking the grain and personality of everyday speech. Once neural codec manages speech into reversible discrete tokens, synthesis no longer “reads out” only the text/content; it can also faithfully reproduce the manner of speaking, hence everyday, down-to-earth speech made possible. The model first drafts a sketch in the space of “speech semantics”: where to take a breath, when to lower the voice, which word should gently rise. It then renders that as a string of audio tokens, and the decoder brings out the breath, sibilants, and the room’s tail reverberation. You can hear the rasp in a voice, the faint nasal smile, and even preserve a speaker’s “vocal persona” across languages. Interaction no longer has a one-size-fits-all voice; it feels like chatting with a neighbor or a friend. The difference from the pre-LLM era lives in those micro-textures carried by the layered discrete codes (tokens).

Now full duplex. Early voice assistants were like walkie-talkies: you speak, then they speak, taking turns. Natural conversation is more like a kitchen scene: you’re chopping vegetables, a friend reminds you to preheat the oven, you cut in—“wait, let’s halve the salt first”—and your friend immediately pivots and responds to the new instruction. Achieving that natural feel looks like an interaction design problem (“interruptible,” “barge-in”), but underneath it requires three things working in concert: continuous listening—streaming the incoming speech into tokens in real time; speaking while listening—generating its own audio tokens ready to speak out while always keeping room to brake; and low-latency understanding and revision—completing the loop of listen → update plan → change what it says in a few hundred milliseconds.

Neural codecs shine again here: by turning both what is heard and what will be said into the same kind of discrete sequence (token string), the model can read and write on the same timeline, managing turn-taking like a human in natural conversation.

These two advances are not isolated. The more ultra-realistic the synthesis, the more the token stream must explicitly encode pauses, stress, laughter, and other details. The smoother the full duplex, the more it relies on a stable, dense, and compact flow of audio tokens so the model can switch course on the fly without dropping a beat. Because the underlying representation has been unified into a “readable and writable token sequence,” we can naturally say, “Don’t be so formal—talk to me like you used to,” and the assistant instantly switches tone. We can also cut in mid-sentence—“Skip that—give me something concrete: go or not?”—and the back-and-forth remains fluid and unawkward.

In systems that truly deliver “ultra-realistic + full-duplex” today, the winning recipe is a hybrid of listen–think–speak. Up front, a streaming auditory model continuously turns your voice into a compact sequence of audio tokens; in the middle, a large model handles understanding and planning; at the back, the system writes its intended response as audio tokens and decodes them into a waveform in real time. The reason we don’t always hand everything to a single end-to-end autoregressive GPT that goes directly from audio-in to audio-out comes down to two constraints: millisecond-level latency requirements and the need for tight control over interruption and course correction.

Think of it as a well-rehearsed band. The streaming speech encoder (ASR) is the drummer, keeping time in tens-of-milliseconds beats. The large model in the middle is the conductor, continuously re-orchestrating based on the latest auditory tokens. The neural-codec decoder (TTS) is the lead vocalist, singing while leaving braking distance so an interruption can “hit the brakes” instantly. In practice, both drummer and vocalist are often smaller, faster specialist models (streaming recognition such as RNN-T/Conformer, paired with neural-codec-based fast synthesis) rather than folding every detail into one gigantic autoregressive stack. Otherwise, interruptions and back-ups might blow up latency.

This does not mean a return to the old ASR/TTS pipeline. The key change is that the base representation has become unified discrete audio tokens: the listening side no longer produces only text characters; it can emit semantic or acoustic units as well. The speaking side no longer merely concatenates phonemes; it writes a reversible stream of codec tokens that carries breath, stress, and reverberant tails. The large model either plans at the hidden layer or refines a two-step “semantic tokens → acoustic tokens” plan, then hands the result to the decoder to render. This keeps full-duplex latency low while preserving ultra-realistic texture.

Looking ahead, research is converging: end-to-end “spoken LLMs” are pulling listening and speaking into the same set of parameters, taking audio tokens directly as input and output (speech2speech). In engineering, however, conversation management—who speaks when, how to interrupt, how to revise—remains as guardrails for smoothness and robustness. Today’s best practice is like a hybrid car: the chassis is the unified, tokenized language; the engine is the large model; and the start/stop control is often delegated to the small specialized motors. The result is fast and stable—and it keeps the warmth of the human voice.