CorX1.5 is a 157.8M-parameter decoder-only transformer whose architecture, 32,768-token tokenizer, pretraining pipeline and fine-tuning were all written at CorX Labs and trained from random weights on a single GPU. It adds learned role embeddings and assistant-only loss masking to a RoPE, SwiGLU and RMSNorm stack of ten 896-wide blocks with a 1,024-token context. Training ran in three stages: about three billion tokens of pretraining from 25 open datasets, distillation from a larger teacher with automated quality filtering, and reasoning training with explicit working. We give the memory arithmetic that makes the run fit on one card, the engineering that let it survive interruptions, and a plain statement of what the model is not for.
Most small open language models are fine-tunes: a released base model with new data on top. CorX1.5 is not. Its architecture, its tokenizer, its pretraining pipeline and its fine-tuning were written at CorX Labs and trained from random weights on a single GPU. This report records what was built, how a three-billion-token run fits on one card, and what the model is and is not good for. It is a small model trained on billions of tokens where comparable models from large labs see trillions, and it should be read that way.
CorX1.5 is a decoder-only transformer [1] with rotary position embeddings [2], SwiGLU feed-forward layers [3] and
RMSNorm [4]. The released config.json gives the exact shape:
| Setting | Value | Note |
|---|---|---|
Width (d_model) | 896 | 14 heads of 64 |
| Feed-forward width | 3,584 | 4 x width, SwiGLU |
| Blocks | 10 | each block applied once (n_loops = 1) |
| Context | 1,024 tokens | a budget decision, see section 4 |
| Vocabulary | 32,768 | byte-pair encoding trained from scratch |
| Dropout | 0.05 | training setting |
| Parameters | 157.8M |
Two choices are less common at this size. Role embeddings add a learned vector to every token marking it as user, assistant or raw text, so whose turn it is lives in the architecture rather than being inferred from special tokens. Assistant-only loss masking means that during instruction tuning the model reads the user's tokens as context but receives gradient only on the tokens it is meant to produce; no capacity is spent learning to predict the user.
Training ran in three stages, each with a held-out validation split carved out before training and early stopping on it.
<think> blocks before
answering, on verified worked arithmetic and on debugging exercises with a known, injected bug to check against.The whole corpus is tokenised once, offline, into a single flat file of token ids. Because the vocabulary is below
65,536, each id fits in a uint16, so three billion tokens occupy 6 GB on disk rather than 12. The file is read
through a memory map and batches are random windows sampled from it, so there is no shuffle step and resuming is a single
integer. Documents are concatenated with a separator and cut into exact 1,024-token rows; nothing is padded.
The tokens never have to fit in GPU memory; they stream past a batch at a time. What has to fit is the model state and the activations of one micro-batch.
| Per-parameter state at 157.8M | Bytes / param | Total |
|---|---|---|
| Weights, fp32 | 4 | 0.63 GB |
| Gradients, fp32 | 4 | 0.63 GB |
| Adam first and second moments | 8 | 1.26 GB |
| Total with Adam | 16 | 2.52 GB |
| Total with 8-bit Adam moments | 10 | 1.58 GB |
Table 1. The optimiser costs twice what the model does; quantising its moments to 8 bits returns almost a gigabyte.
Language models train best with hundreds of thousands of tokens per optimiser step. Gradient accumulation decouples that from memory: with a micro-batch of 8 sequences of 1,024 tokens and 64 accumulation steps, each step sees 524,288 tokens, and three billion tokens take about 5,700 optimiser steps. Forward and backward passes run in bfloat16 autocast, attention uses a fused kernel that never materialises the 1,024 x 1,024 score matrix [5], and gradient checkpointing is kept in reserve. The 1,024-token context is deliberate: attention cost grows with the square of length, and a 158M-parameter model gains more from more short sequences than from fewer long ones.
A multi-day run on one machine will be interrupted, so every checkpoint holds the model, the optimiser moments, the scheduler and step counter, the random-number state and the config; it is written to a temporary file and renamed, so a crash mid-write cannot destroy the last good one.
Messages indicating self-harm or crisis are caught by a hard-coded layer that returns a fixed set of crisis resources instead of a generated reply. It is implemented in code and is not left to the model's judgment.
Weights, tokenizer and the model definition are released under Apache-2.0 at
huggingface.co/CorxLabs/CorX1.5-160M. The architecture is custom and does not load with
AutoModel; the model card describes how to load it. A longer account of the run is on the lab's blog,
"3 billion tokens on one GPU".