Source link : https://tech365.info/indexcache-a-brand-new-sparse-consideration-optimizer-delivers-1-82x-quicker-inference-on-long-context-ai-fashions/
Processing 200,000 tokens by way of a big language mannequin is pricey and sluggish: the longer the context, the quicker the prices spiral. Researchers at Tsinghua College and Z.ai have constructed a way referred to as IndexCache that cuts as much as 75% of the redundant computation in sparse consideration fashions, delivering as much as 1.82x quicker time-to-first-token and 1.48x quicker era throughput at that context size.
The approach applies to fashions utilizing the DeepSeek Sparse Consideration structure, together with the newest DeepSeek and GLM households. It might probably assist enterprises present quicker person experiences for production-scale, long-context fashions, a functionality already confirmed in preliminary checks on the 744-billion-parameter GLM-5 mannequin.
The DSA bottleneck
Giant language fashions depend on the self-attention mechanism, a course of the place the mannequin computes the connection between each token in its context and all of the previous ones to foretell the subsequent token.
Nevertheless, self-attention has a extreme limitation. Its computational complexity scales quadratically with sequence size. For purposes requiring prolonged context home windows (e.g., massive doc processing, multi-step agentic workflows, or lengthy chain-of-thought reasoning), this quadratic scaling results in sluggish inference speeds and vital compute and reminiscence prices.
Sparse consideration affords a principled answer to this scaling drawback. As a…
—-
Author : tech365
Publish date : 2026-03-27 19:51:00
Copyright for syndicated content belongs to the linked Source.
—-
1 – 2 – 3 – 4 – 5 – 6 – 7 – 8