The student who memorized the answers
Imagine two students preparing for an exam. Both are given the same 50 practice questions with answer keys. One student works to understand the ideas behind the questions. The other simply memorizes each question and its answer.
On the practice set, both score perfectly. On the real exam, which contains new questions on the same topics, the first student does well and the second struggles. The second student did not learn the subject. They learned the practice set.
That is overfitting. A model fits its training examples so closely, including their quirks and random noise, that it fails to capture the broader pattern and does badly on cases it has not seen.
Why it happens
A model’s job is to find a pattern that holds generally. But every collection of examples contains two things mixed together: genuine patterns and accidents. The accidents are noise, coincidences, and the particular choices of whoever gathered the data. A flexible model with enough capacity can keep adjusting until it explains all of it, accidents included.
Two things make this more likely:
- Too little data. With only a few examples, coincidences look like rules.
- Too much flexibility. A model with a huge number of adjustable settings can fit almost anything, which is part of why large models need large amounts of training data and careful checking.
A drawing example
Picture twelve points on a graph showing, hypothetically, the temperature at noon each day, which wobbles up and down slightly but trends gently upward over the season. A straight line through the middle captures the trend, though it misses each point slightly. A wildly wiggly curve can be drawn to pass through every one of the twelve points exactly. It has zero error on those points.
But if you ask the wiggly curve to predict the next day, it may swing sharply upward or downward because it has built the random wobble into its shape. The straight line would give a more sensible prediction. The wiggly curve is the overfit model.
A shopping example
Suppose a store builds a hypothetical model to predict who will buy a coat, using last winter’s customers. By chance, in that small group, many buyers had names starting with “M.” A flexible model might treat the first letter of the name as meaningful. It looks excellent on last winter’s data and is useless this winter. The pattern was a coincidence, which is closely related to the warning in correlation versus causation.
The opposite problem: underfitting
A model can also be too simple. A straight line would fail to capture a genuinely curved pattern, and a student who barely studied would miss obvious things. This is called underfitting. It shows up as poor performance on both the training examples and new ones. The art of building models is finding a middle ground: complex enough to capture the real pattern, but not so complex that it chases noise.
How people detect overfitting
The standard approach is to hold some data back.
- Split the examples into a training set and a test set (often also a middle set for tuning choices).
- Train the model only on the training set.
- Measure performance on the test set, which the model never saw.
If the training score is high and the test score is much lower, the model is probably overfit. A big gap is the telltale sign. Some practitioners also repeat the process several times with different splits, a technique called cross-validation, to make sure the result is not just luck.
One subtle trap is called leakage. If information from the test examples slips into training, for example through duplicates, the test score looks better than reality, and the overfitting is hidden. Another is repeatedly adjusting a model to do well on one test set, which turns that set into a second training set.
Ways to reduce it
- Collect more or more varied data. More examples make coincidences less convincing. The quality of those examples also matters, since poor training data stays poor in larger amounts.
- Use a simpler model when a simple one is good enough.
- Regularization. Add a penalty that discourages extreme settings, nudging the model toward smoother, simpler solutions.
- Early stopping. Stop training when performance on held-back data stops improving.
- Dropout and similar tricks. In neural networks, randomly switching off parts during training prevents over-reliance on any single path.
- Data cleaning. Fixing errors and duplicates removes some of the noise that models might otherwise memorize.
Why it matters outside the lab
An overfit model is risky because it looks good in the development phase. A company might report an impressive result that was measured on the data used to build the model. That is one reason to ask how a claim was tested, a theme in how to judge whether an AI claim is believable. In practice it is not unusual for real-world performance to fall short of laboratory numbers, because the world keeps changing and rarely looks exactly like the training set.
The takeaway
The goal of machine learning is not to remember the past perfectly, but to handle new cases sensibly. Overfitting is what happens when a model mistakes the first for the second, and the usual safeguard is simple: always judge a model on examples it has never seen.