← all tags

#speculative-decoding

3 posts tagged speculative-decoding.

Thoughts

Speculative Decoding Just Made Inference Speed a Red Herring

Everyone is racing to make tokens generate faster, but speed is no longer the bottleneck worth solving.

inference speculative-decoding hardware model-efficiency

Speculative decoding just turned inference into a confidence game

Every speedup technique is just teaching models to guess better, and we're running out of good guesses.

speculative-decoding inference-optimisation model-performance

Tools & Experiments

DSpark

DeepSeek's open-source speculative decoding framework speeds up per-user generation by up to 85% without any quality loss.

experimental
speculative-decoding inference deepseek open-source llm-serving