SmallLM-Forge: A Small-Scale Language Model Training and Alignment Stack
What Is SmallLM-Forge?
SmallLM-Forge is a small-scale language-model stack for studying how the main components of modern language models are built and connected. It covers tokenization, decoder-only Transformer pretraining, autoregressive generation, parameter-efficient adaptation, supervised fine-tuning, and preference alignment in implementations that remain small enough to inspect and modify directly.
The project focuses on the ideas behind these components and how they become working code: how byte-level BPE constructs a vocabulary, how RoPE and grouped-query attention fit into a decoder-only Transformer, how low-rank adapters modify selected layers while the base model stays frozen, how supervised fine-tuning restricts the loss to response tokens, and how Direct Preference Optimization turns pairwise preferences into a training objective.
Small experiments are used throughout the repository to exercise these ideas in practice. They serve as examples of how the components can be assembled, while the main focus remains on the underlying mechanisms and their implementation.
From Tokens to Alignment
SmallLM-Forge follows three stages of language-model development:
Build — start from raw text, train a tokenizer, construct a decoder-only Transformer, and pretrain it with next-token prediction.
Adapt — keep an existing model as the starting point and modify its behavior efficiently with LoRA or DoRA and supervised fine-tuning.
Align — move from supervised targets to pairwise preferences and optimize the model with Direct Preference Optimization.
The three parts below follow the same progression: build, adapt, and align.
Explore the Stack
Why Build It at Small Scale?
Modern language-model libraries provide powerful high-level interfaces, but those abstractions can make the mechanics underneath a training run difficult to follow. SmallLM-Forge keeps the models, objectives, and training loops compact enough that the path from an idea or equation to the corresponding tensor operations can be inspected directly.
This makes the stack useful for experimenting with individual design choices. Tokenization, attention, feed-forward layers, generation, low-rank adaptation, supervised fine-tuning, and preference optimization remain separate components with clear responsibilities. Each one can be studied, modified, and tested without requiring a production-scale training system.
The small scale also keeps complete experiments practical. The goal is to make language-model architecture and training behavior concrete through executable implementations, while preserving enough structure to connect those implementations to the techniques used in larger systems.
Models, Artifacts, and Resources
The repository includes example workflows that exercise the components described in the three parts. The pretraining workflow has also produced two public Hugging Face artifacts:
- Byte-level BPE tokenizer:
liuhailin0123/wiki103-bpe-tokenizer - Pretrained mini model:
liuhailin0123/wiki103-llama-mini
The source code, playground notebooks, tests, and implementation details are available in the SmallLM-Forge repository.
