Deepseek Sparse Attention

00:00:00 Introduction to

... to MLA (decoupled RoPE) 22:18

Blog - https://opensuperintelligencelab.com/blog/

This week we review the

Long-context modeling is crucial for next-generation language models, yet the high computational cost of standard

Thanks to KiwiCo for sponsoring today's video! Go to https://www.kiwico.com/welchlabs and use code WELCHLABS for 50% off ...

This week's paper:

DeepSeek

China's

Support Content at - https://donorbox.org/etcg LinkedIn at - https://www.linkedin.com/in/eli-etherton-a15362211/

... Experts (MoE): https://youtu.be/0QQlYR1r6pQ -

DeepSeek Sparse Attention

Is the secret to smarter AI actually... paying less

... 00:00 - DeepSeek 3.2 01:52 - Standard Attention 02:15 -

Spotify - https://open.spotify.com/show/1KkKuQe82tf1bW78ReQ0wM Apple Podcasts ...

... how DeepSeek has created this model, and explaining DeepSeek's new secret weapon:

Heavily Compressed Attention (HCA) - Compressed

Sparse sliding window attention in DeepSeek v4 (dsv4)

DeepSeek

The paper introduces the