/blog

Parallax upgrades softmax attention to a local linear fit via a single learned covariance correction.

We generalize diagonal gating from linear RNNs to softmax attention, introducing Wall Attention - a data-dependent positional encoding that achieves strong length generalization and beats RoPE across the board.

Muon kills neurons in tall MLP matrices. Aurora fixes it with a joint orthogonality–leverage constraint, resulting in an incredibly powerful optimizer.