Transfer learning is the general idea: take a model trained on one large task and reuse what it learned for a different, usually smaller, task. Fine-tuning is the most common way of doing that — continuing to train some or all of the model’s weights on the new data. So the two are not alternatives; fine-tuning is a kind of transfer learning. The real decision is how much of the model to retrain, and that depends on how much data you have and how different your task is.
The relationship#
A model trained on millions of images has learned to detect edges, textures, shapes and objects. A model trained on the internet’s text has learned grammar, facts and reasoning patterns. Transfer learning says: do not throw that away; start from it. There are then several ways to build on it, from touching nothing to retraining everything:
| Approach | What is trained | Data needed | Risk |
|---|---|---|---|
| Feature extraction | Only a new final layer; the model is frozen | Very little | Underfits if the task is very different |
| Partial fine-tuning | The last few layers plus the new head | Moderate | Balanced |
| Full fine-tuning | Every weight, with a low learning rate | More | Overfitting; forgetting general ability |
| Adapter methods (LoRA) | Small added matrices; the model is frozen | Moderate | Slightly less capacity than full fine-tuning |
All four are transfer learning. Only the last three are fine-tuning, and the first is the one people mean when they contrast the terms.
Feature extraction#
Freeze the whole pretrained model and treat it as a fixed function that turns an input into a vector of features. Train a small new model — often a single layer — on top of those features.
from torchvision import models
import torch.nn as nn
model = models.resnet50(weights="DEFAULT")
for p in model.parameters():
p.requires_grad = False # frozen: transfer learning without fine-tuning
model.fc = nn.Linear(model.fc.in_features, 5) # only this layer trains
Fast, needs little data, hard to overfit. Works well when your images are the same kind the model was trained on: ordinary photographs, say. The same idea for text is taking a sentence-embedding model as-is and training a classifier on its embeddings.
Fine-tuning#
Unfreeze some or all of the layers and train them too, gently:
for p in model.parameters():
p.requires_grad = False
for p in model.layer4.parameters(): # unfreeze the last block
p.requires_grad = True
model.fc = nn.Linear(model.fc.in_features, 5)
optimizer = torch.optim.AdamW(
[p for p in model.parameters() if p.requires_grad], lr=1e-4) # small learning rate
The later layers hold the most task-specific knowledge, so unfreezing them adapts the model to your domain; the early layers, which detect generic edges and textures, are left alone. Full fine-tuning unfreezes everything, which works with enough data and a small enough learning rate.
For large language models, adapter methods such as LoRA are the practical form of fine-tuning: the original weights stay frozen and small trainable matrices are added alongside them, cutting memory and producing a swappable file of a few megabytes.
The two questions#
How much labelled data do you have? Hundreds of examples: feature extraction or an adapter. Thousands: partial fine-tuning. Tens of thousands or more: full fine-tuning becomes viable.
How similar is your task to the original? Classifying photographs of products, with a model trained on photographs: very similar, so freezing most of it works. Classifying X-rays or satellite images: the low-level features still transfer but the higher layers need to change, so fine-tune deeper. Text in a specialised jargon: similar story.
| Similar task | Different task | |
|---|---|---|
| Little data | Feature extraction | Feature extraction, or find a closer base model |
| Lots of data | Partial fine-tuning; full is unnecessary | Full fine-tuning |
What the base model brings#
None of this works without a good pretrained model for your kind of input. Image models trained on ImageNet, text models trained on web-scale corpora, audio models trained on speech: each transfers within its modality. Transferring across modalities — an image model to text — does not work; the features mean nothing to the new task. The single most useful choice is picking a base model trained on data as close to yours as exists.
Failure modes#
- Overfitting from fine-tuning too much of the model on too little data. Freeze more, or get more data.
- Catastrophic forgetting from a learning rate too high: the model gains your task and loses its general ability. Lower the rate, freeze more, or use an adapter.
- Negative transfer, where the pretrained knowledge actively hurts because the domains are too different. Rare with modern base models, but the signal is a frozen model scoring worse than a small model trained from scratch.
Questions people ask#
So is fine-tuning transfer learning or not?
It is transfer learning. The contrast people draw is really between fine-tuning and feature extraction, the frozen alternative.
Which is better for LLMs?
Adapter-based fine-tuning for behaviour and style; prompting or retrieval for knowledge. Full fine-tuning of a large model is rarely worth the cost outside research.
Can I fine-tune with unlabelled data?
Continued pretraining on domain text is a form of it: keep the original objective, feed it your corpus. It adapts vocabulary and style without labels.
Does transfer learning always help?
Almost always when a related base model exists, and increasingly so as base models improve. Training from scratch is now the exception, reserved for genuinely novel data types.
Where to go next#
- What is fine-tuning? — with a full worked example.
- Fine-tuning vs RAG — the other decision for language models.
- PyTorch for beginners — the code above, from the start.
