1: Introduction

Why Position Matters

Standard attention is permutation-equivariant, and the operation does not recognize the temporal ordering in which tokens were introduced. Every effective transformer architecture injects some form of positional signal to break this symmetry and encode a recency bias.

Positional embeddings (PE), such as RoPE [1] and ALiBi [2], form the backbone of continual learning via in-context learning and long-context generalization, as they enable the model to adapt to the stream of tokens by selectively forgetting old memories [3][4][5][6], and so advances in PE are critical.

Data-Independent vs. Data-Dependent

The landscape of positional methods can be partitioned into methods that depend on context and those that do not.

Data-independent methods bake position into the architecture as a fixed function of token index. RoPE rotates query-key pairs by an angle proportional to their relative distance; ALiBi subtracts a linear penalty from attention logits. Both are static: the same two positions always receive the same bias regardless of content.

Data-dependent methods let the model decide, based on the per-token latent state, how to modulate its positional prior. Two recent examples are FoX [7] and PaTH [8], which we discuss in detail below.

Data-dependent PE is, in principle, superior because natural language has variable-rate information density. The model can learn end-to-end what to retain and what to forget as context grows.

From Linear to Softmax Attention

Modern linear RNNs compress the sequence into a fixed-size recurrent state $S_t$ that is updated at each step. Without gating, $S_t$ can only accumulate token features, and the model has no way to prioritize recent, relevant information over stale context. The forget gate^‡ is what gives these architectures selective memory, by decaying $S_t$ before writing new information in.

Mamba [9] and Gated DeltaNet [10] use scalar gates, applying the same decay uniformly across all features. GLA [11], RWKV-7 [12] and KDA [13] use diagonal gates $G_t = \mathrm{diag}(g_{t,1}, \ldots, g_{t,d})$, allowing different features to forget at different rates.

The diagonal structure is one of the core innovations behind the latest generation of linear RNNs, and it dramatically improves empirical performance.

Softmax attention has no finite-dimensional recurrent state, so it is not obvious how gating should translate. FoX [7], to our knowledge the first work to attempt this, takes the scalar gate from linear attention and formulates gating on the softmax state as an additive bias. The result is a data-dependent ALiBi, which is a content-dependent distance penalty.

PaTH [8] instead translates the generalized Householder products from DeltaNet's evolution rule [14], which are fundamental in state tracking. The result is loosely analogous to a data-dependent RoPE, as a content-dependent bilinear modulation of the query-key score, though generalized Householder products are not strictly orthogonal.

Both are instances of lifting linear RNN structures to softmax attention and producing data-dependent positional embeddings. The landscape of positional methods forms a natural taxonomy whether the bias is additive or multiplicative, and whether it is data-independent or data-dependent. Each quadrant allows a very specific temporal geometry.

From the Gate to the Wall

We introduce Wall Attention, which generalizes diagonal gates to softmax attention as a full replacement for RoPE. Wall is plug-and-play with existing training and inference frameworks, and we open-source reference Triton implementations for both. Along the way, we develop a general framework for translating RNN structures to softmax attention, unifying FoX, PaTH, and Wall as special cases.

2: Deriving Wall Attention

In Section 1, we described the gated linear RNN

where $A_t$ is some transition matrix. When we unroll this recurrence, the output at position $i$ is a weighted sum over past values. Define the unnormalized attention weight $w_{ij} = q_i^\top B_{ij}\, k_j$, where $B_{ij} = \prod_{r=j+1}^i A_r$ is the cumulative transition product from position $j$ to $i$.

To go from linear to softmax attention, we need $w_{ij} = \exp(q_i^\top B_{ij}\, k_j)$. Following convention, we omit the softmax normalization term throughout this derivation and work with unnormalized weights. The standard way to make this connection is to replace the dot product $q^\top k$ with a general kernel $\kappa(q, k) = \langle \phi(q), \phi(k) \rangle$ for some embedding $\phi$. If $\phi$ is the identity, $\kappa = q^\top k$ and we recover linear attention. If $\kappa = \exp(q^\top k)$, which is the exponential kernel, we recover softmax attention.

We are specifically interested in gating transition matrices that take the form $A_t = \mathrm{diag}(g)$. When we put a gate inside the recurrence, it interacts nontrivially with the feature map.

Primitives

Fix a feature map $\phi : \mathbb{R}^d \to \mathcal{H}$ from the input space to a (possibly infinite-dimensional) Hilbert space, which induces a corresponding positive-definite kernel $\kappa(x,y) = \langle \phi(x), \phi(y) \rangle_{\mathcal{H}}$. We consider two kernels:

• Linear kernel — $\phi(x) = x \ \ (\mathrm{identity})$, $\kappa(x,y) = x^\top y$
• Exponential kernel — $\phi(x) = \left(x^\alpha/\sqrt{\alpha !}\right)_{\alpha \in \mathbb{N}^d}$, $\kappa = \exp(x^\top y)$^*

To offer some brief intuition on the Taylor feature map for the exponential kernel, the resulting infinite-dimensional vector has as elements all of the monomials of elements of $x$ (up to scaling).

A generalized gated linear RNN maintains state $S_t \in \mathcal{H} \otimes \mathbb{R}^{d_v}$ with a gate $A_t: \mathcal{H} \to \mathcal{H}$ takes the following form

Unrolling, we obtain unnormalized attention weights

For a scalar gate $A_t = g_t \cdot I_\mathcal{H}$ where $g_t \in (0,1)$ is a single number, the cumulative product $\prod A_r = F_{ij} \cdot I_\mathcal{H}$ is also scalar. Since $F_{ij}$ is a scalar, it factors out of the inner product:

For the exponential kernel specifically

This is exactly the Forgetting Attention (FoX) formulation, whereby the scalar decay is absorbed into an additive logit bias.

Why Diagonal Gates Break for Exp

The gate $A_t: \mathcal{H} \to \mathcal{H}$ must match the dimension of the kernel's feature space. For the linear kernel ($\mathcal{H} = \mathbb{R}^d$), a $d$-dimensional diagonal gate $D_t = \mathrm{diag}(g_{t,1}, \dots, g_{t,d})$ can serve as $A_t$ directly. Unrolling gives the linear kernel score $w_{i,j} = \sum_n F_{ij,n}q_nk_n$, a per-channel decay-weighted dot product and precisely the score used in GLA, RWKV-7 & KDA.

For the exponential kernel, $\mathcal{H}$ is infinite-dimensional, so a $d$-dimensional diagonal gate cannot serve as $A_t$ directly. We need a lift $\tilde{D}_t: \mathcal{H} \to \mathcal{H}$ that extends the $d$-dimensional gate to the full feature space.

One might try to define $\tilde D_t$ as a diagonal operator on $\mathcal{H}$, assigning each monomial basis element a gate value. However, the monomials in the Taylor expansion correspond to combinations of multiple input coordinates at once, e.g. $x_1^2 x_3$ which involves both channels 1 and 3 simultaneously.

There is no canonical way to assign a single gate value from $D_t$ to a multi-channel monomial, so the scalar absorption trick that enables FoX has no analogue for diagonal gates in the exponential kernel.

The Induced Action

Instead of lifting $D_t$ to the feature space directly, we may let it act on the input space and induce the action through $\phi$. For any input-space operator $A$, define its induced action $\tilde{A}: \mathcal{H} \to \mathcal{H}$ by

Intuitively, this definition says "gating the input before embedding equals gating the embedded state". It is well-defined for any feature map and any input-space linear operator, beyond just diagonal gates. Note that this implicitly requires $\tilde{A}$ to be a linear operator on $\mathcal{H}$, i.e. that $\phi(Ax)$ is linear in $\phi(x)$. This holds for the Taylor feature map since $(Ax)^\alpha$ is a linear combination of monomials in $x$.

This resolves the obstruction from the previous section. The monomial $x_1^2 x_3$ had no single gate to inherit from $D = \mathrm{diag}(g)$, but the induced action $\tilde{D}$ assigns it the product of its constituent gates $g_1^2 g_3$, since $(Dx)^\alpha = g^\alpha x^\alpha$. In fact, for the Taylor feature map, the induced action coincides with the symmetric power lift $\mathrm{Sym}(A)$, the unique functorial extension of a linear operator to the symmetric algebra (see Appendix F for the full treatment).

Crucially, the induced action is a group homomorphism [18]. For any linear operators $A, B$ on $\mathbb{R}^d$, the induced action respects composition.

This means the cumulative induced transition equals the induced action of the cumulative product $\prod \tilde{A}_r = \widetilde{\prod A_r}$. When combined with the kernel trick, we obtain the following simplification for the exponential kernel

where $A_{j \to i} = \prod_{r=j+1}^i A_r$ and $\tilde{A}_{j \to i}$ is its induced action on $\mathcal{H}$.

This holds for any sequence of linear operators $A_r$. If $A_r = (I-\beta ww^\top)$ as a generalized Householder product, for example, we recover PaTH attention. Notably, if we restrict $A_r = g_t I$ we recover a different object than the FoX formulation discussed above, which we discuss in Appendix C.

Specializing to Diagonal Gates & Wall Attention

For diagonal gates where $A_r = D_r$, the cumulative product is also diagonal. From our induced action formulation, the attention score is then as follows

We can thus define Wall Attention.

Wall attention is an extremely powerful attention variant with the ability to selectively forget and remember per-channel.

Define $P_t = \sum_{u \leq t} \log g_u$ as the prefix sum in log-space of gate vectors. An equivalent, important formulation of Wall Attention is vanilla attention with modified queries and keys.

This decoupled formulation works because the cumulative diagonal product is trivially invertible, allowing a clean factorization into independent Q and K transforms. This factorization is what makes Wall fast in practice (Section 3).

Perspectives on Wall Attention

There are many ways to understand Wall beyond diagonal forget gates in attention. We present five further framings to build intuition.

3: Kernel Implementation & Practical Considerations

Recall, in the factorized form from Section 2, the attention score $s_{ij} = \tilde q_i^\top \tilde k_j$ where $\tilde{q}_i = \exp(P_i) \odot q_i$ and $\tilde{k}_j = \exp(-P_j) \odot k_j$. This factorization is what makes Wall practical. The per-channel gate reduces to a one-pass rescaling of $Q$ and $K$, after which the attention kernel is algorithmically identical to FlashAttention [19][20].

This section describes how to make this work at scale: the design choices that control overhead, the numerical problems that arise in finite precision, and the kernel that overcomes them.

Design Choices

Two (orthogonal) decisions control the cost of Wall's gate before we even touch the kernel.

Gate Tying. Wall produces a gate per query head, resulting in $H_Q \times d_k$ scalars per token (where $H_Q$ is the number of query heads and $d_k$ is the head dimension). In GQA, we can instead produce one gate per KV head and share it across the query group. This reduces the gate projection & overhead by a factor of the GQA ratio (typically 4–8x) and creates a shared decay geometry across query heads within a KV group. We find empirically that KV-head gating matches Q-head gating (Section 4).

Subdim Gating. Wall can be applied to only the first $K_g$ of $K$ head dimensions, leaving the rest ungated.

Unlike RoPE, where partial application maintains or improves performance, reducing $K_g$ for Wall degrades performance uniformly (Section 4).

Training Kernel

Decode

b_log_f = tl.sum(b_k * b_W_g[:, None], axis=0) + b_bias_g
b_log_f = log_sigmoid(b_log_f)
b_c = c_running + tl.cumsum(b_log_f, axis=0)
c_running = b_c[-1]
b_k_til = b_k * exp2((c_ref - b_c)[:, None] * RCP_LN2)

4: Empirical Results

Pretraining

We pretrained 400M and 1B transformers with Wall Attention on Nemotron CC v2 [23][28]. Training details are in the Appendix and broadly mirror the Aurora release [24].

At 400M, Wall (NoPE) outperforms the RoPE baseline under both Muon and Aurora optimizers. The consistency across optimizers confirms that the gains from diagonal gating are orthogonal to optimizer choice.

At 1B, Wall achieves a 0.01 nats gain over the RoPE baseline and defeats both FoX and Wall (RoPE), validating it as a standalone PE strategy.

We evaluate the 1B models on standard downstream benchmarks for small model pretraining.

Wall (NoPE) achieves the strongest performance across both pretraining and (short-context) downstream evals. Wall 1B is also a new SOTA on the pretraining tokens vs performance plot from Aurora, beating our old generation Aurora runs despite using only Muon.

The magnitude of pretraining gains is surprising. Many positional embedding variants do not show strong gains over RoPE in pretraining, since their focus is length extrapolation. Wall achieves significant improvements in pretraining convergence over RoPE.

Ablations

Gate Biases

Wall's per-channel retention is $r_t = \exp\!\bigl(f\bigl(\ln\sigma(W_g x_t + \colorbox{#ede9fe}{b})\bigr)\bigr)$, where $f$ is the soft clamp that bounds each step's log-gate to $(-g_{\max}, 0]$. The bias $\colorbox{#ede9fe}{b}$ sets the gate's operating point at initialization. Large positive $b$ starts retention near 1 (fully open, vanilla-attention-like); $b = 0$ starts at $\sigma(0) = 0.5$ before clamping, corresponding to retention $\approx 0.62$ after the soft clamp.

FoX prefers a gate bias of 0, matching the authors' findings. Wall prefers conservative biases of 6-8, starting fully open and gradually learning to close so that behavior matches vanilla attention at init.

Subdim & GQA

We ablated both efficiency strategies from Section 3: subdim gating (applying Wall to only $K_g < d_k$ head dims) and KV-head gate tying (one gate per KV head instead of per Q head under GQA).

Reducing the subdim degrades performance monotonically, but 2-4x subdim still substantially outperforms vanilla attention. KV-head gating matches the more expensive Q-head gating, confirming that gate tying is a free efficiency win.

Long-Context Extrapolation

We tested long-context extrapolation on CodeParrot, NarrativeQA, and PG-19 for the 1B models. Notably, these models were trained with a 4k maximum sequence length on a dataset where most sequences are significantly shorter than 4k tokens.

Wall (NoPE) generalizes to 160k+ sequence lengths, despite only having seen a maximum sequence length of 4k during training. At no point does Wall diverge in loss as context increases.

We further analyzed Wall on the Needle-in-a-Haystack (NIAH) retrieval task. We found Wall was able to extrapolate to 4x its training context length without difficulty, where all other methods failed.

Finally, for LongBench v1, Wall was SOTA over RoPE and FoX across categories including coding, summarization, and few-shot learning.

All the tests we ran here tested zero-shot extreme length generalization, which is an exceedingly difficult test of PE methods. With simple long-context midtraining, Wall is well-poised to generalize to enormous context lengths.

Mechanistic Analysis

We analyzed the learned gate distributions on LongCrawl64 (65K-token documents, 100 documents, 256-token bins).

Wall (NoPE) achieves lower NLL than RoPE at every position, with the gap widening as context grows. Wall (RoPE) falls between the two, suggesting the diagonal gate alone captures most of the positional structure that RoPE provides. We collected and analyzed per-channel gate values (retention scores) for LongCrawl64 document samples.

Wall learns a multi-timescale memory hierarchy across layers, with some layers acting as tight local windows and others maintaining signal over much longer ranges. The NoPE model shows wider variation than RoPE, consistent with the gates taking over more of the positional encoding role.

These layer-level aggregates, however, mask a more interesting finding at the channel level. Per-channel retention reveals two distinct populations - "always-on" channels with retention identically 1.0 (zero variance), and highly dynamic channels whose retention swings between the clamp floor and near-1 on a per-token basis. The dynamic channels respond to content, closing hard at semantic boundaries and opening elsewhere. The static channels provide unconditional long-range memory.

This bimodal structure is learned end-to-end. All channels start with effectively identical initialization due to strong gate biases, and the model discovers that some dimensions should serve as permanent memory while others should gate based on content. The cumulative product (Panel C) demonstrates how dynamic channels create sharp memory cutoffs whereas constant retention (dashed lines) gives only smooth exponential decay.

5: Inflection

Four things excite us about this work.

1. Sufficiency. Wall outperforms every positional variant we tested and does not benefit from stacking RoPE or FoX on top. This suggests Wall is sufficient for strong positional and temporal understanding.

2. Empirical success. Wall's pretraining gains are coupled with extreme length extrapolation. It improves upon both the convergence benefits normally afforded by RoPE and allows models trained at 4k sequence length extrapolate to 200k+ tokens.

3. Cross-pollination. Wall started as an attempt to bring diagonal gating from linear RNNs into softmax attention. The investigation forced us through a series of questions that, to our knowledge, have not been formalized before: how does a finite-dimensional gate act on an infinite-dimensional feature space? What does the induced action look like?

The induced action framework connects modern linear RNNs and softmax attention in a way that, to our knowledge, has not been formalized before. PaTH motivates the $q^\top Mk$ score from the linear RNN analogy; our framework derives it as a consequence of the kernel's feature map.

4. Production-ready. Many proposed attention alternatives claim superior expressivity but are difficult to scale to production training and inference. They require extensive kernel design and force users to accept overhead.

Wall retains the embarrassingly parallel structure of vanilla attention, is directly compatible with GQA and MLA, and can be made faster and lighter-weight through the parametrization and kernel strategies we developed.

Future Work

We list here future directions that may be valuable to the community.

Continued pretraining. Models pretrained with RoPE can be upcycled into Wall models for little cost. We would be excited to see larger-scale open-source models with Wall.

Kernel optimizations. Our open-source Triton implementation is correct and reasonably fast, but leaves performance on the table. Internal profiling suggests a prototype CuTE implementation of Wall/WallMLA closes much of the remaining gap to FlashAttention. An open-source CuTE kernel for Wall training and decoding would be a high-impact contribution.

Broader PE comparisons. We did not compare to PaTH because PaTH focuses on state tracking, whereas our focus is forgetting and long-context. We would like to see Wall evaluated against a wider set of positional embedding methods across diverse tasks and scales.

@article{pai2026wall,
  title   = {Wall Attention: Diagonal Gates for Softmax Attention},
  author  = {Pai, Dhruv and Averbuch, Timor and Zhang, Ashley and Keigwin, Ben and Dewulf, Alec},
  year    = {2026},
  url     = {https://blog.tilderesearch.com/blog/wall-attn}
}

Appendix

A: Open Source Release

We release reference Triton kernels for Wall Attention training and decoding at github.com/tilde-research/wall-attention-release.

B: Training Details

We report exact configurations and training settings for our 400M and 1B runs. We used an internal tokenizer with 128k vocab size for all experiments. For reproducibility, we trained on fully open-source internet data from NVIDIA Nemotron CC v2 [23][28]. Note the training setting broadly matches those used in Aurora [24]. We use per-head Muon (MuonSplit) [27] as the optimizer for all runs.

C: Induced Scalar Formulation

For a scalar gate $D_t = g_t I_d$, the general formula gives

This is very different from FoX. It's a multiplicative positional embedding, similar to the scalar case of Wall Attention. It is somewhat degenerate, as it amounts to merely a path-dependent global temperature.

PaTH [8] (footnote 3) observed that this formulation greatly underperforms FoX. This could be due to the bias initialization differences from our ablation study, whereby multiplicative gates require open-gate initialization.

D: Kernel

Off-diagonal / diagonal split. Each causal Wall output tile attends to many fully-causal key tiles plus one diagonal tile straddling the causal boundary. For Wall, this split coincides with a numerical split. Since $P_t=\sum_{n\le t} g_n$ (cumulative log-gate) is monotonically non-increasing, an off-diagonal tile anchored at $R$ has $P_i-R\le 0$ and $R-P_j\le 0$, so both rescale factors lie in $(0,1]$. The diagonal tile straddles $R$: $\exp_2(R-P_j)$ can grow to the within-tile budget, and masked entries can produce NaN. A single fused loop would force conservative handling (mask + clamp) onto all $O(T)$ tiles. Splitting lets the off-diagonal pass run branch-free and mask-free, confining the clamp to one $B_T\times B_S$ block per query.

bf16 diagonal matmuls. The off-diagonal matmuls ran in bf16 while the diagonal pass defaulted to fp32, an artifact of the reference FlashAttention implementation. This caused a throughput cliff: H100 WGMMA runs bf16 at ~2x TF32 and the backward is matmul-bound. We cast only the matmul inputs ($\tilde Q,\tilde K,dS$) to bf16 and keep fp32 accumulation. The diagonal cast leaves the error indistinguishable from the fp32 path.

Together with the optimizations discussed in Section 3, and block-shape autotuning, these reduce backward wall-clock by ~20% over the stable per-tile kernel.

The naive implementation is often faster than our Triton kernel, but it is numerically unstable and causes early model divergence during training.

E: KV Cache Optimization Improvements

Wall Attention offers a few unique opportunities for KV cache optimization.

Eviction. Per-channel gate values provide a principled heuristic for KV cache eviction. When all channels in an old key have decayed below a threshold, the key has been effectively forgotten and can be evicted at no cost.

Chunk Safety & MXFP8 Alignment. Within a chunk of size $C$, the maximum rescaling factor applied to any cached key is $\exp_2(|P_{j} - R_c|) \leq \exp_2(C \cdot |\log_2 g|_{\max})$. For typical learned gate magnitudes ($|\log_2 g| \approx 0.01\text{–}0.02$), we have

Even at chunk size 128 with aggressive gates, the rescaling is under 6x, and well within the representable range of fp8 E4M3 with negligible quantization error relative to the key magnitudes themselves.

The rescaled values within a chunk are monotonically ordered by distance from the anchor: keys near the chunk boundary have rescaling close to 1, keys far from it have larger rescaling. This smooth, predictable dynamic range aligns naturally with NVIDIA's MXFP8 format [25][26], which assigns a shared block-wise scaling factor to a small group of values. A single MXFP8 scale factor per block captures the local dynamic range with minimal quantization loss.

Reduced dynamic range for key-projected gates. Key-projected gates offer a distinct advantage for low-precision KV caches: since the cached keys are stored unmodified, they have the same dynamic range as vanilla attention and existing KV quantization techniques transfer directly. This contrasts with block-wise anchor decoding, where absorbed keys carry the gate rescaling in their magnitudes.

F: Symmetric Algebra Derivation

The feature space $\mathcal{H}$ spanned by the monomials $x^\alpha / \sqrt{\alpha!}$ is (a completion of) the symmetric algebra $\mathrm{Sym}(\mathbb{R}^d) = \bigoplus_{k \ge 0} \mathrm{Sym}^k(\mathbb{R}^d)$, and the Taylor feature map $\phi$ is the canonical exponential map $x \mapsto \sum_k x^{\odot k} / \sqrt{k!}$ sending an input vector to its coherent state.

Any linear map $A$ on $\mathbb{R}^d$ extends functorially to this algebra, acting on each graded piece by

and $\mathrm{Sym}(A) := \bigoplus_k \mathrm{Sym}^k(A)$ is the symmetric power lift of $A$. The induced action is exactly this lift: $\bar{A} = \mathrm{Sym}(A)$, since $\phi(Ax) = \mathrm{Sym}(A)\,\phi(x)$ holds by construction. This makes the induced action the canonical object rather than an ad hoc definition, as it is the unique functorial extension of $A$ to the feature space.

On the monomial basis, $\mathrm{Sym}(D)$ is diagonal with eigenvalue $g^\alpha = \prod_n g_n^{\alpha_n}$, since $(Dx)^\alpha = \prod_n (g_n x_n)^{\alpha_n} = g^\alpha x^\alpha$. This resolves the obstruction from the main text: each monomial receives the product of its constituent gates.

Because $\mathrm{Sym}(\cdot)$ is a functor, the induced action respects composition.

In particular, the cumulative induced transition equals the induced action of the cumulative product, $\prod_r \bar{A}_r = \overline{\prod_r A_r}$. Combined with the kernel identity, this collapses the feature-space inner product back to a finite-dimensional one.

This holds for any sequence of linear operators $A_r$, recovering PaTH when $A_r = (I - \beta_r w_r w_r^\top)$ and Wall when $A_r = D_r = \mathrm{diag}(g_r)$.

G: Gate Clamping Derivation

Even with per-tile anchoring, aggressive gates can overflow within a tile. The kernel accumulates prefix sums $P_t = \sum_{n \le t} \hat{g}_n$ where $\hat{g}_n = \ln\sigma(l_n)$ is the per-step log-gate in natural-log domain. The maximum exponent of any exp2 call in the backward is $(B_T - B_S) \cdot |g_{\max}| / \ln 2$; staying within fp32 range requires $|g_{\max}| < 0.87$ for our block sizes ($B_T = 128, B_S = 64$).

We enforce this with a soft clamp applied after logsigmoid:

This maps $\hat{g} \in (-\infty, 0]$ to $(-g_{\max}, 0]$ with a steep barrier near the bound but no gradient discontinuity. In retention terms, $r_t = e^{f(\hat{g}_t)}$, so the clamp imposes a floor of $e^{-g_{\max}} \approx 0.42$: each channel forgets at most 58% per step.

The soft clamp is asymptotically identity near zero (where most gates operate after training with high bias) and smoothly saturates toward $-g_{\max}$ for aggressive decays. A hard clamp would achieve the same bound but introduces a gradient discontinuity at the transition, which destabilizes training when channels hover near the boundary. Empirically, looser bounds yield better performance, so any kernel improvement that raises the overflow budget directly improves Wall.

H: Optimizer Ablation (Muon vs Aurora)

At 400M scale, we compared Muon and Aurora optimizers for both the RoPE baseline and Wall (NoPE). Both optimizers achieve comparable final loss on the baseline, and Wall (NoPE) outperforms the baseline under both optimizers. Aurora provides an edge over Muon, consistent with the findings in the Aurora post [24]. Since the gains from Wall are orthogonal to optimizer choice, all results in the main text use Muon except Figure 3, which shows both optimizers at 400M scale.