HELIX is a sequence architecture that drops attention's quadratic cost without giving up exact long-range recall. Each block braids three mixers over one residual stream: an exact local window, a matrix-valued delta-rule recurrent state, and an index that retrieves whole past blocks by beam descent over a tree of learned landmarks. Storage grows linearly and is append-only, while each step holds O(1) hot state and makes O(log N) probes. In a reference implementation, attention pairs scale as N1.00 against N2.00 for full attention, 32 times fewer at 32k tokens; on multi-query associative recall HELIX ties full attention (30.8% against 32.0%, z = 0.72) while the same model without its index falls to chance. There are no trained checkpoints and no claims about language quality.
A transformer compares every token with every other [1]: doubling the input quadruples the work. Linear-time alternatives compress the past into a fixed-size state, and then they cannot quote back a name they saw once. That is a counting argument, not an engineering gap: a state of S bits cannot distinguish more than 2S histories [2]. HELIX (Hierarchical Episodic Linear IndeX) does not try to compress around it. It separates memory capacity from memory bandwidth: storage grows linearly, is append-only and can live in cold memory, while what any one token reads stays bounded. The claim is O(1) hot state and O(log N) probes per step, not O(1) total memory, which is impossible for anything that can quote an exact name from a million tokens back.
| Strand | Reads | Training cost | Held at decode |
|---|---|---|---|
| L, local | the last few blocks exactly, a different window per head group [3] | O(N x span) | a window-sized ring buffer |
| R, recurrent | everything, compressed into a matrix-valued delta-rule state [4, 5] | O(N x dk x dv) | one matrix and two convolution states per head |
| I, index | a few whole blocks chosen anywhere in the past by descending a landmark tree [6] | O(N x top-k x block) + O(N log N) routing | the key/value history and the tree, of which a step reads O(top-k x block + branching x log N) |
Table 1. Every block braids the three mixers back into the same residual stream. The right-hand column decides whether the model is usable at inference.
The past is cut into blocks. Each closed block gets a landmark, a learned summary mixing a plain mean with an attention pool (a mean alone erases the rare token a later query will be looking for), and landmarks are pooled again, a fixed number at a time, into a tree with one weight-shared pooler per level. A query block retrieves in four moves: start at the root using routing taken from the end of the previous block; keep a fixed beam of nodes per level, scoring the beam's children plus that level's frontier; descend to the landmarks; and read the top-k whole blocks exactly, as tokens rather than summaries. Selection scores are added to the attention logits along the path, so gradient reaches every level of the tree.
Two rules keep it causal and parallel. A node is eligible for query block J only if the whole span it summarises ends before J, so a summary is never consulted by a query it partly describes and every landmark is computed once. Routing for block J comes from the hidden state at the end of block J-1, so one gather serves the whole block; anything routing could not anticipate inside the block is covered exactly by the local strand. The frontiers at every level tile [0, J) exactly, so nothing reachable is cut off by a beam that descended elsewhere.
All on small models on four CPU cores, reproducible with helix bench and helix recall.
Attention pairs are counted by instrumenting the attention call, not estimated from a formula.
| Tokens | HELIX pairs/token | Full attention pairs/token | HELIX time | Full attention time |
|---|---|---|---|---|
| 4,096 | 8,192 | 32,776 | 0.4 s | 0.2 s |
| 8,192 | 8,192 | 65,544 | 0.9 s | 0.6 s |
| 16,384 | 8,192 | 131,080 | 1.7 s | 1.9 s |
| 32,768 | 8,192 | 262,152 | 4.3 s | 6.7 s |
Figure 1 and Table 2. HELIX fits N1.00 in attention pairs and full attention N2.00. At 32k tokens HELIX forms 32 times fewer pairs and runs 1.6 times faster, as unfused reference PyTorch against a fused attention kernel. At 4k tokens full attention is still faster; the crossover is between 8k and 16k on this hardware.
| Multi-query associative recall [7] | Recall | 95% interval |
|---|---|---|
| HELIX (local + recurrent + index) | 30.8% | 28.5-33.1% |
| HELIX without the index (local + recurrent) | 6.0% | 4.8-7.2% |
| Full attention | 32.0% | 29.7-34.3% |
| Chance | 6.2% |
Table 3. Bindings written near the start and queried about 200 tokens later, outside the local window. Removing the index leaves the same model at chance (a gap of about 21 standard errors): the window cannot see the bindings and the recurrent state cannot hold them. Against full attention it is a tie, 1.2 points with a 1.7-point standard error (z = 0.72), and reporting it as a win would be reading noise.
The test suite checks what can be checked without training: strict causality (editing token t leaves every earlier logit bitwise unchanged); token-by-token decoding from the fixed-size state reproduces the one-shot forward pass, as does unaligned chunked prefill; right padding is exactly invariant; each query reads the same number of keys at any length; the cache holds only what the architecture claims; attention tiling is bitwise inert; and every parameter, landmark poolers included, receives gradient.
NatureV1's attention engine, iHELIX, grew out of this design and applies it to the sphere.
pip install helix-lm helix bench # cost against context length helix recall # train with and without the index strand, and compare