Prompt masking: loss only where it matters
In an SFT sequence, the prompt tokens are inputs we condition on, not behavior we want to copy. Masking the prompt (loss weight 0) focuses every gradient on the completion positions, improving sample efficiency and preventing the model from learning to generate prompts.
Token Masking
labels = [-100] * prompt_len + completion_tokens. Cross-entropy ignores positions labelled -100, so only the assistant's tokens are scored.
Effective Epochs
Because fewer positions carry loss, the same token budget is 'spent' more densely on behavior. Effective epochs = steps × global_batch / dataset_tokens.
Overfit Signature
The classic SFT failure: training loss keeps falling while validation loss turns upward. Track the gap (valid − train) per checkpoint.
The loss is the mean negative log-likelihood over completion positions only. With masking, a 512-token example with a 384-token prompt trains on 128 scored positions.
- TRL and LLaMA-Factory expose this as train_on_inputs=False (Axolotl); getting it wrong silently halves training efficiency.
- Packing multiple short examples into one 4k sequence with correct attention separation is a standard throughput trick — and a standard correctness bug when masks leak across boundaries.