Fine-Tuning Language Models on Domain-Specific Data
Fine-tune language models on your domain with parameter-efficient LoRA and QLoRA: dataset preparation, training strategies and evaluation, on a small budget.
General-purpose language models know a bit about everything. They can write poetry, explain quantum physics, and generate code. But ask them about your company's internal APIs, your industry's specific terminology, or your organization's coding conventions, and they hallucinate confidently. Fine-tuning transforms a generalist into a specialist—a model that understands your domain's language, patterns, and constraints.
The barrier to fine-tuning has collapsed. Parameter-efficient methods like LoRA let you fine-tune a 7B parameter model on a single GPU in hours. You don't need a team of ML engineers or a cluster of A100s. You need good data, clear objectives, and the discipline to evaluate whether your fine-tuned model actually improves on the base model for your specific use case.
When Fine-Tuning Makes Sense
Fine-tuning isn't always the answer. Before investing in training, understand when it's the right tool.
## Decision framework
### Use fine-tuning when:
- You need the model to adopt a specific style or format
(medical reports, legal briefs, code in your framework)
- Domain-specific terminology that the base model gets wrong
- You have 100+ high-quality examples of desired behavior
- Prompt engineering alone can't achieve consistent quality
- Latency matters and you can't afford long prompts with
many examples
### Use RAG (retrieval) instead when:
- The knowledge changes frequently (product catalog, docs)
- You need citations and verifiable sources
- The base model understands the domain but lacks specifics
- You have structured data that's easy to retrieve
### Use prompt engineering instead when:
- You have fewer than 50 examples
- The task is well-defined and consistent
- Few-shot examples in the prompt achieve good results
- You need flexibility to change behavior without retrainingPreparing Training Data
The quality of your fine-tuning dataset determines the quality of your model. Garbage in, garbage out applies more strongly here than anywhere else in software.
# Dataset preparation for instruction fine-tuning
import json
from typing import TypedDict
class TrainingExample(TypedDict):
instruction: str
input: str
output: str
# ❌ Low-quality training data
bad_examples = [
{
"instruction": "Write code",
"input": "make a function",
"output": "def foo(): pass" # Too vague, teaches nothing
},
{
"instruction": "Fix the bug",
"input": "it doesn't work",
"output": "try restarting" # Not actionable
},
]
# ✅ High-quality training data
good_examples: list[TrainingExample] = [
{
"instruction": (
"Write a TypeScript function that validates "
"an email address using our company's domain rules."
),
"input": (
"Requirements: Must end with @company.com or "
"@subsidiary.com. Must not contain plus addressing. "
"Must be lowercase."
),
"output": (
"function validateCompanyEmail(email: string): "
"{ valid: boolean; error?: string } {\n"
" const normalized = email.toLowerCase().trim();\n"
" \n"
" if (normalized !== email) {\n"
" return { valid: false, error: "
"'Email must be lowercase' };\n"
" }\n"
" \n"
" if (normalized.includes('+')) {\n"
" return { valid: false, error: "
"'Plus addressing not allowed' };\n"
" }\n"
" \n"
" const allowedDomains = "
"['@company.com', '@subsidiary.com'];\n"
" const hasValidDomain = allowedDomains.some(\n"
" domain => normalized.endsWith(domain)\n"
" );\n"
" \n"
" if (!hasValidDomain) {\n"
" return { valid: false, error: "
"'Must use company email domain' };\n"
" }\n"
" \n"
" return { valid: true };\n"
"}"
),
},
]
def prepare_dataset(
examples: list[TrainingExample],
output_path: str
) -> None:
"""Format examples for fine-tuning frameworks."""
formatted = []
for ex in examples:
formatted.append({
"messages": [
{
"role": "system",
"content": (
"You are a senior engineer at Company. "
"Follow our coding standards and conventions."
),
},
{
"role": "user",
"content": f"{ex['instruction']}\n\n{ex['input']}",
},
{
"role": "assistant",
"content": ex["output"],
},
]
})
with open(output_path, "w") as f:
for item in formatted:
f.write(json.dumps(item) + "\n")LoRA: Fine-Tuning Without the GPU Bill
Low-Rank Adaptation (LoRA) freezes the original model weights and trains small adapter matrices. This reduces trainable parameters by 99%+ while achieving comparable quality to full fine-tuning.
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import LoraConfig, get_peft_model, TaskType
from datasets import load_dataset
from trl import SFTTrainer, SFTConfig
# Load base model
model_name = "meta-llama/Llama-3.1-8B-Instruct"
model = AutoModelForCausalLM.from_pretrained(
model_name,
torch_dtype="auto",
device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained(model_name)
# LoRA configuration
# r: rank of the adaptation matrices (lower = fewer params)
# alpha: scaling factor (typically 2x rank)
# target_modules: which layers to adapt
# ❌ Too aggressive: high rank, all modules
bad_config = LoraConfig(
r=128, # Way too many parameters
lora_alpha=256,
target_modules="all-linear", # Unnecessary
)
# ✅ Balanced: enough capacity for domain adaptation
lora_config = LoraConfig(
task_type=TaskType.CAUSAL_LM,
r=16,
lora_alpha=32,
lora_dropout=0.05,
target_modules=[
"q_proj", "k_proj", "v_proj", "o_proj",
"gate_proj", "up_proj", "down_proj",
],
bias="none",
)
model = get_peft_model(model, lora_config)
model.print_trainable_parameters()
# trainable params: 13,631,488 || all params: 8,043,167,744
# trainable%: 0.1695Training Loop and Hyperparameters
The training configuration matters more than most people realize. Learning rate, batch size, and number of epochs can mean the difference between a useful model and a catastrophically overfit one.
# Training configuration
training_config = SFTConfig(
output_dir="./checkpoints",
# Learning rate: too high = catastrophic forgetting
# too low = you're just wasting compute
learning_rate=2e-4,
# Warmup prevents early instability
warmup_steps=100,
lr_scheduler_type="cosine",
# Epochs: 1-3 for most datasets
# More epochs = more overfitting risk
num_train_epochs=2,
per_device_train_batch_size=4,
gradient_accumulation_steps=4, # Effective batch size: 16
# Evaluation during training to catch overfitting
eval_strategy="steps",
eval_steps=50,
save_strategy="steps",
save_steps=50,
# Mixed precision for memory efficiency
bf16=True,
logging_steps=10,
max_seq_length=2048,
)
# Load dataset
dataset = load_dataset("json", data_files={
"train": "data/train.jsonl",
"eval": "data/eval.jsonl",
})
# Initialize trainer
trainer = SFTTrainer(
model=model,
args=training_config,
train_dataset=dataset["train"],
eval_dataset=dataset["eval"],
processing_class=tokenizer,
)
# Train
trainer.train()
# Save the adapter (not the full model)
trainer.save_model("./final-adapter")
# Adapter is ~50MB vs 16GB for the full modelEvaluation: Does the Fine-Tuned Model Actually Help?
The most important step is rigorous evaluation. A model that scores well on training loss but produces worse outputs than the base model with good prompting is a waste of time.
# Evaluation framework
from dataclasses import dataclass
@dataclass
class EvalResult:
example_id: str
base_output: str
finetuned_output: str
reference_output: str
base_score: float
finetuned_score: float
def evaluate_model(
base_model,
finetuned_model,
eval_examples: list[dict],
scorer,
) -> list[EvalResult]:
"""Compare base vs fine-tuned on held-out examples."""
results = []
for example in eval_examples:
prompt = example["instruction"] + "\n" + example["input"]
base_output = generate(base_model, prompt)
ft_output = generate(finetuned_model, prompt)
base_score = scorer.score(
base_output, example["output"]
)
ft_score = scorer.score(
ft_output, example["output"]
)
results.append(EvalResult(
example_id=example["id"],
base_output=base_output,
finetuned_output=ft_output,
reference_output=example["output"],
base_score=base_score,
finetuned_score=ft_score,
))
# Summary statistics
base_avg = sum(r.base_score for r in results) / len(results)
ft_avg = sum(r.finetuned_score for r in results) / len(results)
print(f"Base model average: {base_avg:.3f}")
print(f"Fine-tuned average: {ft_avg:.3f}")
print(f"Improvement: {(ft_avg - base_avg) / base_avg * 100:.1f}%")
# Check for regressions: cases where fine-tuned is worse
regressions = [
r for r in results
if r.finetuned_score < r.base_score
]
print(f"Regressions: {len(regressions)}/{len(results)}")
return resultsCommon Pitfalls
# ❌ Pitfall 1: Overfitting on small datasets
# Symptom: training loss drops to near zero,
# eval loss increases
# Fix: fewer epochs, more dropout, more data
# ❌ Pitfall 2: Catastrophic forgetting
# Symptom: domain task improves but general
# capability degrades
# Fix: lower learning rate, mix in general data (5-10%)
# ❌ Pitfall 3: Poor data quality
# Symptom: model learns your mistakes and inconsistencies
# Fix: have domain experts review training examples
# Quality check: would you want a junior dev to
# learn from this example?
# ✅ Practical data quality checklist:
# - Every example has a clear, complete answer
# - No contradictions between examples
# - Consistent formatting and style
# - At least 100 examples for narrow tasks
# - At least 500 for broader domain adaptation
# - 10-20% held out for evaluationKey Takeaways
Fine-tuning is the right choice when you need consistent style, format, or domain terminology that prompt engineering alone can't reliably achieve—but if your knowledge changes frequently or you need citations, retrieval-augmented generation is usually a better fit. LoRA makes fine-tuning accessible on consumer hardware by training tiny adapter matrices (0.17% of total parameters) while achieving results comparable to full fine-tuning—start with rank 16, target the attention and feed-forward projection layers, and use a cosine learning rate schedule starting at 2e-4. Data quality matters more than quantity: 200 expert-curated examples where each one demonstrates exactly the behavior you want outperforms 10,000 noisy scraped examples, and every training example should pass the test of "would I want a junior developer to learn from this?" Always evaluate against the base model on held-out examples before deploying, because a fine-tuned model that scores well on training loss but produces worse outputs than the base model with good prompting has negative value—check for regressions and catastrophic forgetting, not just average improvement.
