Linear RNNs and State Space Models as Alternatives to Attention

HN commenters recommend it for building intuition about state space models and the tradeoffs between transformer and SSM layers.

Sasha Rush surveys linear RNNs and state space models as replacements for attention in transformer-style models, in a talk for MLSys. It covers the foundational preliminaries and predates Mamba.