Critical Points of Linear Self-Attention

Research assistant in the UCLA Department of Mathematics. Studying linear approximations to self-attention that avoid the quadratic cost of standard attention, and testing where they match it and where they break down. Manuscript in progress.