Paper Proposes Grassmann Flows as Attention Alternative

A paper titled 'Attention Is Not What You Need' was submitted to the Computer Science Machine Learning section on December 22, 2025. The authors argue that standard multi-head attention functions as a form of tensor lifting, mapping hidden vectors into a high-dimensional space of pairwise interactions. They propose an alternative architecture based on Grassmann flows instead of explicit self-attention.

The proposed Causal Grassmann layer linearly reduces token states and encodes local token pairs as two-dimensional subspaces on a Grassmann manifold via Plucker coordinates. The authors analyzed the complexity of this mixing process, showing linear scaling in sequence length for fixed rank.

On the Wikitext-2 benchmark, purely Grassmann-based models with 13 to 18 million parameters achieved validation perplexities within about 10 to 15 percent of size-matched Transformers. In natural language inference tasks, a Grassmann-Plucker head on top of DistilBERT slightly outperformed a Transformer head, recording a best validation accuracy of 0.8550 and a test accuracy of 0.8538.