Choosing how far AI guesses ahead can speed replies
Original title: Adaptive Multi-Token Prediction: How far should we go?
How AI worksTry thisChatbots & assistants
Read by Charlie, a synthetic voice
The paper adds a small guide that chooses how many word pieces an AI should draft before checking them. It learns from each check, without training the model again, by picking the draft length most likely to survive. The authors report up to four times faster text output and matching or beating fixed drafting in eight of ten cases.
Our take: The evidence is mixed: theory and reported speedups exist, but the tests are limited, include a regression, and the runs were not repeated. It could still matter for people who wait on AI text.
- The method chooses the draft length before each check.
- It learns from check feedback without training the model again.
- The authors report up to four times faster text output.
17 claims checked against the source Read the paper on arXiv PDF
Checked against the source
Every statement in this brief had to match a word-for-word passage from the source; code checks each match, and a second AI model reviews the whole brief for errors. A brief that fails is not published. These are the 17 passages it rests on.
Show the 17 passages
Multi-token prediction can accelerate language-model decoding by drafting several tokens and verifying them in one target-model pass.
Multi-token prediction (MTP) can accelerate language-model decoding by drafting several tokens and verifying them in a single target-model pass.
Longer drafts can waste compute on discarded suffixes after rejection.
longer drafts offer greater amortization when reliable, but can waste compute on suffixes discarded after a rejection.
The central decision is how many tokens to propose before the next verification step.
how many tokens should be proposed before the next verification step?
Speculative decoding uses a cheaper draft mechanism to propose several tokens and evaluate them with the target model.
Speculative decoding reduces this sequential bottleneck by using a cheaper draft mechanism to propose several tokens and evaluating them in parallel with the target model
Autoregressive decoding generates text one token at a time through a target language model.
Autoregressive decoding generates each new token through an evaluation of a target language model.
AMTP is a lightweight controller that requires no retraining and leaves the exact verifier unchanged.
introduce Adaptive Multi-Token Prediction (AMTP): a lightweight controller that requires no retraining and leaves the exact verifier unchanged.
AMTP uses verification feedback to learn how far drafts can be trusted in each context.
using verification feedback to learn how far the draft can be trusted in each context
Before each round, AMTP predicts how far the draft is likely to survive, picks the horizon with highest expected tokens per second, and learns from every verification pass.
Before each round, AMTP predicts how far the draft is likely to survive and picks the horizon with the highest expected tokens per second, learning freely from every verification pass.
Each round chooses the horizon before drafting.
Each round of speculative decoding poses a one-step control problem: before drafting, choose the horizon
AMTP decodes up to 4.1x faster than standard decoding.
AMTP decodes up to 4.1×faster than standard decoding
AMTP matches or outperforms the fixed window in 8 of 10 settings.
matches or outperforms the fixed window in 8 of 10 settings
Experiments were run without busy serving with caching, and such serving changes horizon cost.
Our experiments decode one sequence at a time without a KV-cache: batched serving with caching changes the cost of each horizon and is a natural direction.
One reported regression was Meta MTP 7B on AIME by 15%.
The one regression is Meta MTP 7B on AIME (−15%).
Reported conditions were single unseeded runs.
each condition is a single unseeded run
AMTP drafts fewer tokens per round while committing a similar number.
Across all pairings, AMTP drafts fewer tokens per round while committing a similar number
The method can convert wasted draft computation into completed tokens.
converting wasted draft computation into completed tokens
The authors show prediction error can be related to wall-clock completion time.
any online learner’s prediction error translates into throughput regret against the best horizon for each context, and in turn into a bound on wall-clock completion time.