Source link : https://tech365.info/nvidias-new-approach-cuts-llm-reasoning-prices-by-8x-with-out-dropping-accuracy/
Researchers at Nvidia have developed a way that may scale back the reminiscence prices of enormous language mannequin reasoning by as much as eight instances. Their approach, known as dynamic reminiscence sparsification (DMS), compresses the important thing worth (KV) cache, the non permanent reminiscence LLMs generate and retailer as they course of prompts and cause via issues and paperwork.
Whereas researchers have proposed numerous strategies to compress this cache earlier than, most battle to take action with out degrading the mannequin’s intelligence. Nvidia’s method manages to discard a lot of the cache whereas sustaining (and in some circumstances enhancing) the mannequin’s reasoning capabilities.
Experiments present that DMS allows LLMs to “think” longer and discover extra options with out the standard penalty in pace or reminiscence prices.
The bottleneck of reasoning
LLMs enhance their efficiency on advanced duties by producing “chain-of-thought” tokens, basically writing out their reasoning steps earlier than arriving at a remaining reply. Inference-time scaling strategies leverage this by giving the mannequin a bigger funds to generate these pondering tokens or to discover a number of potential reasoning paths in parallel.
Nonetheless, this improved reasoning comes with a big computational value. Because the mannequin generates extra tokens, it builds up a KV cache.
For real-world functions, the KV cache is a significant bottleneck. Because the reasoning…
—-
Author : tech365
Publish date : 2026-02-12 23:03:00
Copyright for syndicated content belongs to the linked Source.
—-
1 – 2 – 3 – 4 – 5 – 6 – 7 – 8