Speculative decoding: draft cheaply, verify together

Picture it

Think of writing a document with a fast, eager intern who constantly drafts the next few sentences ahead of you. Most of the time, the intern's guesses are perfectly aligned with your thoughts, allowing you to simply nod and approve whole blocks of text at once. If they make a mistake, you just cross it out and resume writing from that point. This parallel drafting lets the system sprint through predictable phrases, trading a little wasted effort for a massive boost in overall speed.

What to understand

• Draft cost, acceptance rate, verification cost, and lookahead
• Draft models, self-speculation, memory overhead, and load dependence

Follow the references

1. 1 · Paper

   Chen et al. · Speculative sampling ↗

   Follow the draft/verify diagram and rejection correction before reading speedup results.

Make it concrete

Compare speculation on/off at low and high concurrency. Record accepted tokens per verification, total GPU work, latency, throughput, and output-quality checks.