Neural Networks Learn From the Bottom Up — A Visual Walkthrough

Language is hierarchical, in two ways

Train a network on data with hierarchical structure, like language, and two hierarchies are in play:

Syntactic — an open parenthesis predicts a close; structure nests.
Semantic — abstract concepts decompose into concrete ones.

Both run from an abstract top to a surface bottom.

In which direction does the network learn it?

Two ways the network could climb the abstraction tree over training:

Top-down — grasp the abstract gist first, fill in details later.
Bottom-up — nail local surface regularities first, reach the abstract rules only later, if ever.

Why it matters

Every task recruits capabilities latent at some level. Typo-fixing needs the bottom; creative writing needs the top.

The Random Hierarchy Model

Language's hierarchy is implicit. The RHM is a synthetic one we fully control.

A tree: the root expands, level by level, each feature picking one of \(m\) rules into \(s\) children, down to \(s^{L}\) observed tokens.

Cagnetta, Petrini, Tomasini, Favero & Wyart, Phys. Rev. X, 2024.

Features are a menu of synonyms

One latent feature, \(m\) interchangeable rules \(\Rightarrow\) many surface tuples that are synonyms: different strings, identical one level up.

The dial

Higher \(m\) \(\Rightarrow\) more synonyms to unify \(\Rightarrow\) the wave stalls sooner.

Every prediction tests exactly one level

Because the process is known, each token is tagged by the level it tests.

Predicting position \(p\) crosses the deepest boundary to its left — the level is how many times \(s\) divides \(p\) (its trailing zeros).

Payoff

A single held-out sequence becomes a per-level report card: group losses by level, watch each get learned on its own.

At \(L{=}6\) (64 tokens), the same rule tiles into a self-similar ruler:

The learning wave climbs the hierarchy bottom-up

Warmth = how far each level is learned (real per-level loss, \(m{=}2\) model). The order in which each level begins to fall is perfectly monotonic: leaves first, root last. Here that model is a 4-layer, 4-head transformer (\(d{=}128\), ~0.8M params) trained on 5M tokens of Random Hierarchy Model data at \(L{=}6,\ v{=}8,\ m{=}2\) (64-token sequences).

The same wave in the numbers — and a ceiling

Per-level cross-entropy (\(m{=}2\)). Uniform guessing \(=\ln 8 \approx 2.08\); the orange cell marks the frontier.

step	L0	L1	L2	L3	L4	L5
start	2.04	2.04	1.99	2.04	2.09	2.04
200	0.67	1.66	1.64	1.97	1.91	1.92
1,000	0.49	1.31	1.42	1.97	1.86	1.92
5,400	0.37	1.12	1.17	1.77	1.75	1.87

L1 moves only once L0 is underway; L2 waits for L1. Levels 3–5 barely leave the floor.

The ceiling

The wave stops after 2–3 levels. What sets that ceiling?

Model capacity?

a deeper hierarchy needs a bigger network

Signal in the data?

the cue simply runs out

Intuition says capacity. Here, intuition is wrong — these models had capacity to spare.

Hold one fixed, vary the other

Fix data (\(m{=}2\)), vary the model

Ten transformers spanning a 200× range of capacity:

All ten stall at L2. Depth buys almost nothing — 2 layers compose as many levels as 8.

Fix the model, vary \(m\)

One generous 8-layer network; \(v{=}8\), chance \(=0.125\):

\(m\)	L0	L1	L2	L3–5	climbs to
2	0.83	0.44	0.44	~0.27	~L2
4	0.59	0.31	0.21	~0.20	~L1
6	0.48	0.19	0.18	~0.16	~L0
8	0.34	0.18	0.18	~0.20	barely L0

Every extra synonym shortens the climb.

Verdict

In this setup, the ceiling is set by the data — once the model is big enough, adding capacity doesn't raise it.

Clean the signal, and the wave reaches the root

Same 8-layer network, same 40k steps in every panel — only the data's ambiguity changes.

Two compounding forces

1 The signal is loudest at the bottom

The cue is diluted by every random layer above it — the data to hear it grows exponentially with depth.

2 You cannot skip a step

A feature is identifiable only through the features beneath it — until parents are read, grandparents are invisible. The order is logically forced.

Together they make a signal ceiling — the wave stalls where the cue sinks below what the data reveals. Across the model sizes tested here, capacity was the cheap variable.

Implications for language modeling

Expect the same order in language:

First: syntax, local co-occurrences, the basic geometry of language.
Later, if ever: deep compositional rules and rich abstractions — only if the corpus carries a clean enough cue.

What buys the next level is signal — more data with a cleaner cue — against a threshold that rises exponentially with depth. In the data-limited regime, the reach is a property of language, not the model.

Consistent with Networks Learn Statistics of Increasing Complexity and Language Model Behavioral Phases.

Nothing accidental about a model that captures mostly surface semantics — it is where a data-limited model comes to rest.

Learning From the Bottom Up

Our motivations