How Close Is FlashAttention to the Limit? Understanding Attention Through Data Movement

Louis Aeilot's Blog blog.aeilot.top 2026/10/05

Reading Barna Saha and Christopher Ye, The I/O Complexity of Attention, or How Optimal is Flash Attention? (2024). Original paper We often measure an algorithm’s cost in FLOPs. On a GPU, however, less…

讨论

还没有评论，来说第一句吧。

Louis Aeilot's Blog 的其他文章

Beyond FLOPs: How COSMA Builds Parallel Matrix Multiplication from Communication Bounds 2026/10/05
The Red-Blue Pebble Game: Why Faster Processors Still Have to Move Data 2026/10/05
Git Needs a Trash Can 2026/09/29
RoPE: Properties, Patterns, and Long-Context Behavior 2026/08/02
CS336 Assignment 1: Large Language Model Training and Inference 2026/08/01

分类

Star24k

Indi 收录中文独立博客，

每小时检查一次 RSS。

30 天内有更新的博客每小时检查一次，一年内有更新的每 3 小时，更久没更新的每 12 小时。抓取失败时会先照常重试，失败超过一天后逐步降低频率（12 小时、3 天、7 天）；连续 7 天抓不到标记为疑似失效，超过 90 天标记为已失效、每 90 天才检查一次。恢复后自动回到正常节奏。RSS 没有变化时只发一个很小的请求，不会重复下载。

欢迎在 GitHub 上提交你的博客。