Part 2 — Adapt a Language Model with LoRA and SFT
A pretrained language model already knows how to model text. Adaptation changes its behavior for a particular task, response format, or domain.
SmallLM-Forge separates this problem into two parts. Parameter-efficient fine-tuning (PEFT) determines which parameters can change and how those changes are represented. Supervised fine-tuning (SFT) determines which outputs provide the training signal.
This part follows that path from a pretrained weight matrix to a response-only training objective.
From Pretraining to Parameter-Efficient Adaptation
A pretrained language model already contains a large set of parameters learned during pretraining. Adapting the model to a new task means changing some of those parameters so that the model assigns higher probability to the desired behavior.
For a linear layer,
\[ y = xW, \]standard fine-tuning makes the weight matrix \(W\) trainable and learns a new weight matrix
\[ W'. \]We can describe the change introduced by fine-tuning as
\[ W' = W + \Delta W, \]where \(\Delta W\) is the update learned for the new task.
For a large language model, repeating this process across every trainable weight means optimizing millions or billions of parameters. It also requires storing gradients and optimizer states for those parameters during training.
What Is Parameter-Efficient Fine-Tuning?
Parameter-efficient fine-tuning (PEFT) keeps most of the pretrained model fixed and introduces a much smaller set of trainable parameters for adaptation.
The basic idea is
pretrained parameters
↓
frozen
small adaptation parameters
↓
trainable
The pretrained model continues to provide the original representation and language-modeling capability. Training concentrates on a smaller parameter set that describes how the model should change for the new task.
LoRA is one way to construct such an update.
Instead of learning every entry of
\[ \Delta W \in \mathbb{R}^{d_{\text{in}}\times d_{\text{out}}}, \]LoRA asks whether the useful update can be represented inside a much smaller intermediate space.
Why Use a Low-Rank Update?
A general matrix
\[ \Delta W \in \mathbb{R}^{d_{\text{in}}\times d_{\text{out}}} \]contains
\[ d_{\text{in}}d_{\text{out}} \]independent parameters.
LoRA represents the same update through two matrices:
\[ \Delta W = \alpha AB, \]with
\[ A \in \mathbb{R}^{d_{\text{in}}\times r}, \qquad B \in \mathbb{R}^{r\times d_{\text{out}}}. \]The product \(AB\) still has the same shape as the original weight update:
\[ AB \in \mathbb{R}^{d_{\text{in}}\times d_{\text{out}}}, \]so it can be added directly to \(W\).
The difference is the number of trainable parameters.
A full update contains
\[ d_{\text{in}}d_{\text{out}} \]parameters, while the two LoRA matrices contain
\[ r(d_{\text{in}} + d_{\text{out}}). \]When
\[ r \ll \min(d_{\text{in}}, d_{\text{out}}), \]the adapter is much smaller than the original weight matrix.
What Does the Rank Mean?
The value \(r\) is the rank dimension of the LoRA adapter.
It is also the width of the intermediate representation created by the two projections:
The LoRA projections: A maps the input dimension down to the rank-r bottleneck, and B maps back up to d_out.
The first matrix \(A\) projects the input from the original feature dimension
\[ d_{\text{in}} \]into the smaller space
\[ r. \]The second matrix \(B\) projects that representation back to
\[ d_{\text{out}}. \]Therefore,
A : down projection
d_in → r
B : up projection
r → d_out
The matrix product \(AB\) has rank at most \(r\). Choosing \(r\) therefore controls how much freedom the adapter has to modify the original layer.
A smaller rank gives fewer trainable parameters and a tighter bottleneck. A larger rank gives the adapter more capacity to represent different weight updates.
LoRA uses the assumption that useful adaptation can often be represented with a rank much smaller than the dimensions of the original weight matrix. The rank is therefore a capacity choice rather than a new hidden dimension of the base Transformer.
Full fine-tuning updates the complete weight matrix. LoRA freezes the pretrained weight and learns a low-rank update through a down projection A and an up projection B, with r controlling the adapter bottleneck.
Why Initialize \(A\) and \(B\) Differently?
SmallLM-Forge initializes the two matrices as
\[ A \sim \mathcal N \left( 0,\frac{1}{r} \right), \]while
\[ B = 0. \]This choice has an important consequence.
At initialization,
\[ AB = 0, \]and therefore
\[ \Delta W = \alpha AB = 0. \]The adapted layer initially computes exactly the pretrained transformation:
\[ y = xW+\alpha xAB = xW. \]Adding a LoRA adapter therefore does not immediately perturb the pretrained model with a random weight update.
There is another reason that both matrices are not initialized to zero.
If
\[ A=0 \qquad\text{and}\qquad B=0, \]then the gradients through the product \(AB\) are also zero at the beginning of training. The adapter would have no useful direction from which to start learning.
The factor
\[ \frac{1}{\sqrt r} \]used when initializing \(A\) also keeps the scale of the random projection controlled as the rank changes.
Put the LoRA Update into the Forward Pass
With the low-rank update defined, the adapted linear layer becomes
\[ W' = W+\alpha AB. \]For an input \(x\),
\[ y = xW+\alpha xAB. \]This adds two contributions to the same output: the frozen pretrained transformation \(xW\), and a trainable low-rank correction \(\alpha xAB\). The first path preserves the pretrained behavior; the second learns a task-specific update through the rank-\(r\) adapter. The scaling factor \(\alpha\) controls how strongly that learned correction modifies the frozen base transformation.
The next implementation question is therefore how these LoRA branches are inserted into selected linear layers of an existing Transformer.
Insert Adapters into a Model
A Transformer contains many linear layers, so the next implementation problem is deciding where the adapter should be attached.
SmallLM-Forge recursively traverses the model tree and replaces selected leaf modules.
For example, a target projection can change from
q_proj
└── nn.Linear
to
q_proj
└── LinearWithLoRA
├── linear
└── lora
├── A
└── B
The caller specifies target module names, and apply_peft_to_module() performs
the replacement in place.
The adapter configuration is also stored with the model:
model._peft_meta = {
"adapter_class": ...,
"r": ...,
"alpha": ...,
"target_submodules": ...,
"kwargs": ...,
}
This keeps the structural modification and its configuration connected.
Freeze the Base Parameters
Once adapters have been injected, the pretrained parameters can be frozen.
SmallLM-Forge uses requires_grad to keep only selected parameters trainable:
for name, param in model.named_parameters():
if not any(
pattern in name
for pattern in patterns
):
param.requires_grad = False
A typical result is
base model parameters frozen
LoRA A trainable
LoRA B trainable
The utility also prints the number and percentage of trainable parameters. This is a useful verification step because adapter injection alone does not prevent the optimizer from updating the original model.
From LoRA to DoRA
SmallLM-Forge also supports DoRA.
LoRA constructs
\[ \widetilde W = W+\alpha AB. \]DoRA further separates the adapted weight into direction and magnitude.
The implementation first normalizes the adapted weight:
\[ D = \frac{\widetilde W}{\|\widetilde W\|_2}, \]then applies a trainable magnitude \(m\):
\[ W_{\text{DoRA}} = m\odot D. \]The corresponding implementation is
AB = (
self.lora.A.to(dtype)
@ self.lora.B.to(dtype)
).T
weight = (
self.linear.weight
+ self.lora.alpha * AB
)
direction = (
weight
/ weight.norm(
p=2,
dim=0,
keepdim=True,
).clamp(min=1e-12)
)
dora_weight = (
self.lora_magnitude.to(dtype)
* direction
)
The extra idea can be summarized as
LoRA
W + low-rank update
DoRA
W + low-rank update
↓
normalize direction
↓
trainable magnitude
Save Only What Changed
A parameter-efficient adapter can be stored separately from the base model.
SmallLM-Forge saves adapter tensors into
adapter_state.pt
and stores the adapter configuration in
adapter_config.json
The base model weights are excluded.
This gives a clean decomposition:
base model
+
adapter configuration
+
adapter parameters
At this point we know what changes in the model. The next question is which tokens provide the signal that drives those changes.
Turn a Pretrained Model into an Instruction-Following Model
A pretrained decoder-only language model learns to predict the next token:
\[ p(x_t \mid x_1,\dots,x_{t-1}) \]This teaches language structure, but it does not directly teach the model how to answer a user’s request.
Instruction tuning changes the objective from simply continuing text to learning a desired interaction:
instruction
+
expected response
↓
supervised example
The underlying mechanism remains causal language modeling. The difference is that the useful training signal is concentrated on the assistant response.
From Instructions to Training Targets
An instruction example contains:
context:
system instruction
user request
target:
assistant response
For example:
System:
You are a classifier.
User:
<task input>
Assistant:
<target response>
The model must read the complete conversation, but only the assistant output is the behavior we want to optimize.
Why Only Train on the Response?
The model reads the complete conversation, but only the assistant response is trained:
Response-only SFT: the input sequence stays fully visible, while the labels mask System and User with -100 and keep only the assistant token IDs.
The objective is:
\[ \mathcal L_{SFT} = -\sum_{t\in\text{response}} \log p_\theta \left( y_t \mid x, y_{\lt t} \right) \]The prompt remains available through attention, while only response tokens contribute gradients.
Implement Response-Only SFT
process_example() constructs each example in two forms: a prompt consisting of
the system and user turns, and the full sequence that appends the assistant
response:
prompt = [
{"role": "system", "content": system_prompt},
{"role": "user", "content": example["text"]},
]
full_prompt = [
*prompt,
{"role": "assistant", "content": example["str_label"]},
]
The two forms are tokenized separately into input_ids (prompt only) and
full_input_ids (prompt + response). Their length difference identifies the
response boundary.
Build the Batch
The collator combines examples, left-pads sequences, and creates the attention mask:
The batch layout: left-padded input_ids with the matching attention_mask over the full sequence.
Labels start as a copy of the full sequence, and the prompt region is then masked out:
labels = input_ids.clone()
labels[i, :prompt_end] = -100
The response-only objective is therefore encoded directly in the batch.
Train the Adapter
The model side contains:
frozen pretrained parameters
+
trainable LoRA / DoRA parameters
The data side contains:
input_ids
attention_mask
response-only labels
Together:
PEFT + SFT training: the response-only loss updates only the LoRA/DoRA adapter parameters while the base model stays frozen.
Run the SFT Loop
The training step is:
outputs = model(**batch)
loss = outputs.loss
loss.backward()
optimizer.step()
optimizer.zero_grad()
If freezing is configured correctly, gradients update only LoRA/DoRA parameters.
Evaluate the Adapted Model
Evaluation measures generated behavior:
prompt
↓
model.generate(...)
↓
generated tokens
↓
prediction
↓
metric
The sentiment example extracts generated labels and reports Macro F1.
Example: Sentiment Classification
A text example is converted into:
System:
You are a tweet sentiment classifier.
User:
<tweet>
Assistant:
negative | neutral | positive
The complete workflow is:
pretrained causal LM
↓
inject LoRA or DoRA
↓
freeze base parameters
↓
format instructions
↓
create response-only labels
↓
supervised fine-tuning
↓
evaluate generated outputs
The dataset demonstrates the adaptation pipeline. The same mechanism applies to other tasks where a context is followed by a desired response.
