(Some of) The Models, They Just Don't Want to Learn

8.03.2026

×

Correspondence to Sean McLeish,  Ben Keigwin,  Mark Saroufim,  Rohan Anil

Questions and discussion on Discord

1: Introduction

"The models, they just want to learn" — Ilya Sutskever

Standard Transformers often seem to justify this perspective, since at sufficient scale, they exhibit generally stable optimization dynamics across a wide range of tasks.

The same is much less true for architectures with recurrent latent depth. These models are appealing because they can scale computation adaptively by adding depth at test time, but in practice they are often unstable, difficult to optimize, and unreliable when run for more steps than they saw during training.

It is tempting then to take the difficulty of training these models as evidence that vanilla Transformers are simply the better architecture. However, Transformers also benefit from years of accumulated knowledge about residual connections [8], normalization, initialization, and optimizers. Models with recurrent or adaptive latent depth may require the same degree of co-design between architecture and optimization before their utility becomes (more) clear.

To investigate this, we are releasing One Layer Deeper, a competition for learning latent serial computation end-to-end. The goal is simple to state but very difficult to achieve: train a model which can perform deep serial work and also productively use more computational depth at test time than it encountered during training.

2: Why Sequential Computation Matters

Some problems are naturally easy to parallelize, whereas others involve a sequence of dependencies such that each step can only begin once the result of the previous step is known. The computational depth of a problem is the minimum number of these sequentially dependent steps needed to solve it, which is notably distinct from the total amount of work performed.

Neural models can thus spend compute in different ways. A wider network can perform more work in parallel, while a deeper network can compose more transformations sequentially. Crucially, however, depth unlocks computational capacity that is much harder to obtain with width. Merrill and Sabharwal show that Transformers whose depth grows logarithmically with input length can solve tasks that fixed-depth Transformers cannot solve uniformly without superpolynomial width [1].

To obtain expressive models in depth, the question is therefore not simply whether a model performs enough operations, but rather whether those operations are arranged into the right number and orchestration of sequential steps based on problem difficulty.

3: More Tokens Are One Way to Think Longer

A standard Transformer forward pass has fixed depth, whereby information passes through a predetermined number of layers before the model produces its next-token distribution.

However, during chain-of-thought reasoning [2], the model repeatedly applies the same fixed-depth circuit, with generated tokens and the KV cache (fast weights) carrying state from one forward pass to the next. Chain of thought is therefore itself a form of recurrent computation through tokens, which has proven remarkably effective. It allows models to decompose difficult problems and spend additional test-time compute by generating longer traces. It is also a very general mechanism since the same language interface used to communicate the final answer can be used to store and manipulate the intermediate state.

However, this interface can impose an artificial computational structure since intermediate values must be serialized into tokens before they can influence later decoding steps. Computation which could happen in parallel within a latent representation must instead be spread over a sequence of autoregressive passes.

An alternative is to increase computation directly during the forward pass. Universal Transformers [3] repeatedly apply a shared Transformer block to the hidden state, utilizing Adaptive Computation Time [4] to allow different amounts of computation to be allocated across positions. Deep Equilibrium Models [5] instead solve for a fixed point of a learned transformation, effectively simulating the fixed point of an infinitely deep, weight-tied network. Deep Thinking models [6] and more recent looped or depth-recurrent Transformers [7] similarly reuse layers over multiple latent iterations.

These models investigate whether parameters can be reused across depth, whether computation can increase at test time without increasing parameter count, and whether the amount of latent computation can adapt to the difficulty of the input.

Looped layers are not the only possibility, however. Recurrent state updates, attractor-style models, and other forms of latent computation may offer different tradeoffs. The important distinction is not between Transformers and non-Transformers, but between different ways of structuring and allocating computation.

4: Learning to Compute Deeper Is Hard

Showing that an architecture can represent deeper computation is not the same as showing that it can learn it or utilize its depth well. There are at least three separate questions:

1. Can the instantiation of the architecture represent the required computation?
2. Can optimization discover the correct computation?
3. Does that computation remain useful when applied for more steps than were seen during training?

The first is largely a question of expressivity. Universal Transformers are computationally universal under reasonable assumptions [3], while deep equilibrium architectures can in principle represent computations much deeper than a standard fixed-depth forward pass [5]. Reusing the same transformation also provides a natural way to increase computation without increasing the number of parameters.

The harder question is whether gradient-based optimization will discover a useful solution. Repeatedly applying the same layers produces highly correlated latent updates, while small errors in the learned dynamics can compound over many iterations. Hidden states may explode, collapse, or converge to an unhelpful solution.

Providing a model with additional depth therefore does not guarantee that it will use that depth well. Even when a model solves problems at its training depth, it may fail when run for longer, and additional test-time compute can sometimes make a model's prediction worse. Adaptive computation adds another difficulty: the model must learn not only what computation to perform, but how much computation each problem requires in a generalizable way.

Unlike vanilla transformers, whose training methods have been refined over many years and at enormous scale, there are many aspects of parametrization & optimization we do not yet understand for models which perform long sequences of computation within their latent state.

The difficulty may not be that better model architectures do not exist, but rather that our current optimization knowledge disproportionately favors vanilla Transformers.

We fundamentally believe architecture cannot be studied independently from optimization. Models that perform useful computation over many latent steps may require different optimizers, losses, initializations, or intermediate supervision. An architecture that does not train under a standard Transformer setup is not necessarily bad; it may simply require a different training procedure.

5: Desiderata for a Clean Benchmark

Evaluating computational depth is surprisingly difficult. Training a model on a long serial procedure is not enough: the same answer may still be computable by a much shallower algorithm. The Euclidean algorithm, for example, proceeds through a chain of dependent remainders, but the greatest common divisor can also be found by checking candidate divisors in parallel. In particular, a serial algorithm does not necessarily imply a serial problem.

A useful benchmark for learned depth should therefore satisfy four properties:

1. The problem should be inherently serial. Solving harder instances should genuinely require a longer chain of dependent computation.
2. There should be no practical shortcut around that chain. Otherwise, a model may solve the benchmark with a shallower strategy than the one we intended to test. I.e., that the task is not as serial as we believe it to be.
3. Success should not come from memorization. The input–output space must be large enough that a model cannot replace computation with a lookup table.
4. Test depth must be larger than train depth. Test instances should require more sequential computation than any instance seen during training. This enforces genuine depth extrapolation.

We thus wanted a benchmark that satisfied each of the above characteristics and also possessed a clear "difficult dial" to allow people to derisk their ideas on a smaller scale.

6: One Layer Deeper

The public task in One Layer Deeper asks models to predict the result of repeatedly squaring a value modulo a semiprime.

Starting from

the model repeatedly applies

and predicts the final value after $T$ steps. Equivalently,

This task has a deliberately simple structure. Each squaring depends on the residue produced by the previous squaring. Without knowledge of the factorization of $N$, the best known general approach is to perform these modular squarings sequentially [9], and increasing $T$ directly increases the length of the dependency chain.

At the same time, neither the input nor output needs to grow with $T$. Both $x$ and the final residue remain bounded by $N$. We can therefore increase the required serial computation while keeping the problem length fixed.

It is also hard for the models to "cheat", so long as we use a semi-prime modulus, since factoring such numbers is difficult, there is no shortcut solution to solve the task without doing the serial computation itself.

Models are trained on smaller values of $N$ and $T$, then evaluated on larger moduli, greater computational depths, and held-out combinations of the two. The hidden Hard task tests whether approaches transfer beyond assumptions specific to the public repeated-squaring task.

7: What We Want Competitors to Explore

One Layer Deeper is not only an architecture competition.

Competitors are free to choose the model, optimizer, learning rate scheduler, and loss function. We want to encourage co-design between these choices, rather than assuming that a promising architecture should train successfully with an otherwise standard setup.

We want to see new ideas! E.g. we suspect successful submissions will require some combination of intentional serial architectures, new optimization methods to keep very deep execution stable enough to train, and new ways of performing adaptive computation.

More broadly, we hope to learn which combinations of architecture, objective, and optimizer make deep serial computation trainable.

8: How to Take Part

Thanks to Modal and Northflank, competitors can submit their ideas to hosted H100s through the One Layer Deeper website.

Training is limited to a single H100 budget and models are limited to 500M parameters. Submissions may not hard-code the task algorithm at runtime or use task-specific data augmentation intended to solve the benchmark directly. These constraints are intended to focus the search on learning the computation end-to-end through architecture, optimization, and loss design.

Submission instructions are available at: onelayerdeeper.ai/submit

You can also join the Discord to discuss ideas and share results: Discord

@article{tilde2026onelayerdeeper,
  title   = {(Some of) The Models, They Just Don't Want to Learn},
  author  = {Sean McLeish and Ben Keigwin and Mark Saroufim and Rohan Anil},
  year    = {2026},
  url     = {https://blog.tilderesearch.com/blog/one-layer-deeper}
}