When should you fine-tune instead of prompting?
Your prompt has grown into a 2,000-word instruction manual and the output still misses. You have heard fine-tuning fixes this, but it can also burn weeks and lock you into a snapshot.
Prompting is the cheap first move
Per OpenAI's fine-tuning guide, optimization starts with prompting, not training. You give the model context, instructions, and goals, then measure the result. Prompting is fast to change and costs nothing but attention. Fine-tuning, by contrast, means additional training applied to the parameters of a pre-trained network, as Wikipedia's fine-tuning article explains. That is a heavier commitment than editing text.
Evals come before either choice
OpenAI's guide is explicit about ordering. First write evals that measure model output and establish a baseline for performance and accuracy. Without that baseline you cannot know whether a new prompt or a fine-tune actually helped. The guide frames evals, prompting, and fine-tuning as a flywheel, where each pass produces better prompts and better training data. Skipping the eval step means you are tuning blind.
Fine-tune when the prompt stops scaling
The signal to fine-tune is a prompt that has plateaued. If you keep stuffing examples and rules into the context and accuracy stops improving, per OpenAI's guide it may be desirable to fine-tune for some use cases. Fine-tuning bakes behavior into weights instead of paying for it in every request's context. Wikipedia notes the training can touch all layers or freeze most and adapt only some, which changes cost and flexibility.
What this does not solve
Fine-tuning is not stability. OpenAI's guide warns that LLM output is non-deterministic and behavior changes between model snapshots and families. A fine-tune tied to one snapshot can silently degrade when the base model moves. And the tooling shifts too: OpenAI notes parts of its evals and fine-tuning docs are moving into legacy documentation. You still need measurement after training, forever.
Do you have an eval baseline today that would tell you a fine-tune actually beat your best prompt?
Which of your prompts has stopped improving no matter how many examples you add?
Write one eval case for your worst-performing output and run your current prompt against it before lunch.
Sources
OpenAI: Improving model behavior
Recommended for you
- LLM Engineering
Why does your LLM feature fail silently in production?
Your LLM feature has no error signal when it gets worse. Drift, hidden hallucinations, and missing monitoring let decay hide inside plausible outputs.
- LLM Engineering
Why does prompt engineering still leave you without a product?
A clever prompt produces one good output. A product must survive users you never met, and the prompt cannot carry that weight alone.
- LLM Engineering
Why do your LLM bills stay high after you picked cheap models?
You switched to smaller models and costs barely moved. The problem is not which model you picked, it is that no one decides which request deserves which model.
- LLM Engineering
Why does your LLM feel slow when the latency chart looks fine?
Your p95 dashboard says the platform is healthy, but users still complain. LLM latency has two phases, and your monitoring probably only sees one.
Enjoyed this article?
Subscribe for new articles. No spam. Unsubscribe anytime.