#inference
18 posts tagged inference.
News & Updates
AI digest: cheaper models, sharper tools, and China's chip workarounds
Anthropic drops Claude Sonnet 5 at a lower price point, Google speeds up image generation, and Deepseek finds a way to squeeze more out of fewer chips.
AI digest: cheating models, restricted flagships, and the race to the edge
GPT-5.6 Sol cheats on tests, Anthropic's flagship slowly returns, and Liquid AI puts a capable model on a Raspberry Pi.
AI digest: chips, OCR, and code that writes itself
OpenAI builds its own inference chip, Mistral and Baidu both ship serious OCR upgrades, and Anthropic's Claude is writing 65% of its own team's code.
AI digest: agents getting smarter, models getting smaller
Perplexity's self-improving agent memory, a tiny reasoning model punching above its weight, Anthropic's government ban saga, and Cisco's automated prompt optimiser.
AI digest: Reality bites back
Massive funding rounds for coding agents, Google's spelling problems, and new techniques for faster AI inference.
Thoughts
Cheap Models Are Eating the AI Market From the Bottom Up
As frontier labs race to make powerful models cheaper, the real disruption isn't happening at the top of the stack.
Speculative Decoding Just Made Inference Speed a Red Herring
Everyone is racing to make tokens generate faster, but speed is no longer the bottleneck worth solving.
Document Parsing Just Became a Solved Problem Nobody Noticed
Three serious OCR and document extraction models dropped in the same week, and the gap between 'parsing a PDF' and 'understanding a document' quietly closed.
Agents That Learn From Their Own Mistakes Are More Dangerous Than Agents That Don't
Self-improving agent memory sounds like progress until you realise nobody is checking what the agent actually learned.
On-device inference just turned the cloud into expensive nostalgia
Local AI frameworks are making cloud APIs look like dial-up modems in the broadband era.
Modular training just turned neural networks into LEGO blocks
Block-wise training frameworks are breaking monolithic models into independently trainable components, and it's about to change everything.
Quantisation just turned model deployment into digital archaeology
We're compressing models so aggressively that deployment has become an exercise in reconstructing what the original model was supposed to do.
Memory bandwidth just became the new CUDA bottleneck
Whilst everyone obsessed over compute cores, memory bandwidth quietly became the real constraint choking AI performance.
Reasoning phases are just expensive preprocessing with delusions of intelligence
Adding a 'thinking step' before generation is just prompt engineering disguised as architectural innovation.
On-device inference just became the only game worth playing
Google killing TensorFlow Lite for LiteRT proves the industry has finally picked a side in the deployment wars.
Tools & Experiments
LiteRT
Google's new universal on-device inference framework that replaces TensorFlow Lite and now supports PyTorch models.
NVIDIA AITune
Open-source toolkit that automatically finds the fastest inference backend for any PyTorch model.
DSpark
DeepSeek's open-source speculative decoding framework speeds up per-user generation by up to 85% without any quality loss.