What Is Fine-Tuning? Explained With Examples

Fine-tuning means taking a model that has already been trained on a huge general dataset and training it a little more on your own, much smaller dataset, so it gets good at your particular task. You are not teaching it from scratch. You are adjusting something that already works.

The idea in one analogy#

A newly qualified doctor has spent years learning general medicine. A specialist training programme does not start them again from anatomy; it builds on what they know, with a focused curriculum. Fine-tuning is the specialist programme. The general training is expensive and done once; the specialisation is cheap and done as often as needed.

What actually changes#

A trained model is a very large set of numbers called weights. Training adjusts them so the model’s outputs get closer to the answers in the training data. Fine-tuning continues that same process, but:

• starting from the already-trained weights rather than random ones,
• using a small dataset specific to your task,
• usually with a lower learning rate, so the adjustments are gentle and the general knowledge is not overwritten.

Modern methods often freeze most of the weights and train only a small added set — the LoRA technique is the common one for language models. That cuts memory and time enormously and makes the result a small file you can swap in and out.

Three examples#

1. An image classifier for your own categories#

A network trained on millions of general photos already knows edges, textures, shapes and objects. Fine-tune it on a few hundred photos of your products and it learns to tell them apart, because it only has to learn the last step, not how to see.

from torchvision import models
import torch.nn as nn

model = models.resnet50(weights="DEFAULT")     # pretrained on ImageNet
for param in model.parameters():
    param.requires_grad = False                # freeze everything...

model.fc = nn.Linear(model.fc.in_features, 4)  # ...except a new final layer for 4 classes

Training only that final layer on your images is the simplest form of fine-tuning, and it works surprisingly well.

2. A text classifier from a pretrained language model#

A model such as a small BERT variant already understands English. Give it a few thousand labelled support tickets and it learns to route them.

from transformers import AutoTokenizer, AutoModelForSequenceClassification, Trainer, TrainingArguments
from datasets import load_dataset

name = "distilbert-base-uncased"
tokenizer = AutoTokenizer.from_pretrained(name)
model = AutoModelForSequenceClassification.from_pretrained(name, num_labels=3)

data = load_dataset("csv", data_files={"train": "tickets_train.csv", "test": "tickets_test.csv"})
data = data.map(lambda batch: tokenizer(batch["text"], truncation=True, padding="max_length"), batched=True)

args = TrainingArguments(
    output_dir="out",
    learning_rate=2e-5,          # small: nudge, do not overwrite
    num_train_epochs=3,
    per_device_train_batch_size=16,
    eval_strategy="epoch",
)

Trainer(model=model, args=args, train_dataset=data["train"], eval_dataset=data["test"]).train()

Those thirty lines are a complete fine-tuning job. The learning rate is the number to notice: it is tiny, because a big one would destroy what the model already knows.

3. A large language model for a house style or format#

An LLM already writes fluently. Fine-tune it on a few thousand examples of your input-output pairs — questions and the answers your team would give, or documents and the summaries you want — and it adopts that behaviour without needing it spelled out in every prompt.

{"messages": [{"role": "user", "content": "Summarise: ..."}, {"role": "assistant", "content": "..."}]}
{"messages": [{"role": "user", "content": "Summarise: ..."}, {"role": "assistant", "content": "..."}]}

Most hosted LLM services accept a file of examples in a shape like that and run the fine-tuning for you.

When fine-tuning is the right tool#

The distinction that matters most: fine-tuning changes how a model behaves; it is a poor way to give it new facts. Facts go stale, and a model cannot cite which example it learned something from. Retrieval handles both, which is why the two are often combined.

How much data#

Far less than training from scratch, which is the point. Rough orders of magnitude:

• Image classifier, new final layer: tens to hundreds of examples per class.
• Text classifier: a few hundred to a few thousand labelled examples.
• LLM style or format: a few hundred high-quality examples often beats thousands of mediocre ones.

Quality dominates quantity. Every example teaches the model that this input should produce that output, so a wrong label is actively harmful.

The two ways it goes wrong#

Overfitting. Train too long on a small dataset and the model memorises it, scoring perfectly on the training examples and badly on anything new. Watch the score on a held-out set every epoch and stop when it stops improving.

Catastrophic forgetting. Train too aggressively and the model loses general ability while gaining your specific one — a chatbot fine-tuned on legal summaries that can no longer hold a conversation. A low learning rate, few epochs, and freezing most of the weights all guard against it.

Where to go next#

• Fine-tuning vs RAG — the decision most projects actually face.
• Supervised learning — the training loop this builds on.
• Machine learning with Python — the toolkit.