Does testing a model on its own training data reliably measure generalization?

No, it can give an overly optimistic accuracy estimate

Yes, training accuracy always equals real-world performance

Yes, since a model never sees training data twice

What is a commonly used split ratio for training versus test data?

About 80% training and 20% test

Always exactly 50% training and 50% test

100% training with no separate test set

If a model performs well on training data but poorly on test data, what does that suggest?

The model is likely overfitting the training data

The model is underfitting and far too simple

The test set must contain incorrect labels

Why is it important to shuffle or randomize data before splitting into train and test sets?

To avoid a biased split caused by ordering in the data

Shuffling guarantees a higher R-squared score

It removes the need to scale features later

ML Train Test

Learn how train_test_split lets you honestly measure a model's performance and spot overfitting before it costs you.

Why You Need a Train/Test Split

A model's real job is to make good predictions on data it has never seen. If you train it and then measure its accuracy on the very same rows, you aren't testing that ability — you're just checking how well it memorized the answers. Overfitting happens when a model does exactly that: it captures the noise and quirks of the training set rather than the general pattern, so its training score looks great while its performance on new data is much worse.

scikit-learn's train_test_split function solves this by randomly splitting your dataset into two pieces before any training happens. The model only ever sees the training portion during fit(); the test portion is held back and used purely to check how the model behaves on data it has never touched.

test_size (or train_size) controls the split ratio — 0.2 to 0.3 for the test set is a common starting point.
random_state fixes the random shuffle so the same split — and the same results — can be reproduced later.
stratify=y keeps class proportions in classification problems consistent between the train and test sets.
shuffle=True (the default) randomizes row order before splitting; leave it off only for ordered data like time series.

Using train_test_split

Example

from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split

X, y = load_iris(return_X_y=True)

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, random_state=42
)

print("Training rows:", X_train.shape[0])
print("Test rows:    ", X_test.shape[0])

Note: Pick one random_state value and reuse it throughout a notebook or script. It makes your split — and any bug you're chasing — perfectly reproducible for yourself and anyone reviewing your code.

Overfitting vs Underfitting

Pattern	Training score	Test score	What's happening
Underfitting	Low	Low	Model is too simple to capture the pattern in the data
Good fit	High	High (close to training)	Model has learned the general pattern well
Overfitting	Very high (~100%)	Noticeably lower	Model has memorized training noise instead of generalizing

Example

from sklearn.tree import DecisionTreeClassifier
from sklearn.metrics import accuracy_score

deep_tree = DecisionTreeClassifier(max_depth=None, random_state=42)
deep_tree.fit(X_train, y_train)

print("Train accuracy:", accuracy_score(y_train, deep_tree.predict(X_train)))
print("Test accuracy: ", accuracy_score(y_test, deep_tree.predict(X_test)))

shallow_tree = DecisionTreeClassifier(max_depth=2, random_state=42)
shallow_tree.fit(X_train, y_train)

print("Shallow train accuracy:", accuracy_score(y_train, shallow_tree.predict(X_train)))
print("Shallow test accuracy: ", accuracy_score(y_test, shallow_tree.predict(X_test)))

Example

from sklearn.datasets import make_classification
from collections import Counter

X_imb, y_imb = make_classification(
    n_samples=200, weights=[0.9, 0.1], random_state=1
)

X_tr, X_te, y_tr, y_te = train_test_split(
    X_imb, y_imb, test_size=0.2, random_state=1, stratify=y_imb
)

print("Original distribution:", Counter(y_imb))
print("Test distribution:    ", Counter(y_te))

Note: Never use the test set to choose hyperparameters or pick between models — if you do, you are indirectly fitting to it and its score stops being an honest estimate of real-world performance. Use a separate validation split or cross_val_score for that, and touch the test set only once, at the very end.

Exercise: ML Train Test

Why is a dataset typically split into training and test sets?

Keep Going

Machine Learning Interview QuestionsPractice real questions asked for Machine Learning Machine Learning Engineer SalarySee what Machine Learning Engineer roles pay in the market Auto Apply for JobsApply to open roles related to Machine Learning and tweak your resume