/blog

We are launching One Layer Deeper, a competition for co-designing architectures, objectives, and optimizers to learn deeper serial computation and extrapolate beyond training depth.

Parallax upgrades softmax attention to a local linear fit via a single learned covariance correction.

We generalize diagonal gating from linear RNNs to softmax attention, introducing Wall Attention - a data-dependent positional encoding that achieves strong length generalization and beats RoPE across the board.

Muon kills neurons in tall MLP matrices. Aurora fixes it with a joint orthogonality–leverage constraint, resulting in an incredibly powerful optimizer.