What does it mean if a model performs well on training data but poorly on test data?

The model is likely overfitting to the training data

The model is perfectly generalized and ready for deployment

The test set must contain corrupted data

What is a key risk of evaluating a model only on its training data?

It gives an overly optimistic, unreliable estimate of real-world performance

It accurately reflects how the model will perform on new data

It guarantees the model has no bias whatsoever

Why is data typically split randomly rather than sequentially by row?

To help ensure both sets are representative and avoid ordering bias

To make the training set contain only the largest values

Random splitting is required by every machine learning library

What is a typical common ratio used to split data into training and testing sets?

Roughly 80% training and 20% testing

Exactly 50% training and 50% testing, always

100% training and 0% testing, since testing is optional

Data Science Train Test

Splitting data into a training set and a held-out test set lets you measure how well a model generalizes to data it has never seen, exposing overfitting before it reaches production.

Why Split Data into Train and Test Sets

A model that is only ever evaluated on the same data it learned from can look perfect while actually being useless. It may have memorized quirks, noise, and coincidences specific to that dataset rather than learning the underlying pattern. This failure mode is called overfitting: the model performs great on training data but poorly on new, unseen examples.

High training accuracy paired with much lower test accuracy is the classic sign of overfitting.
A model that overfits has essentially memorized the training examples, including their noise.
A model that underfits performs poorly on both training and test data, meaning it has not learned enough structure.

Using train_test_split

Scikit-learn's train_test_split function randomly divides your dataset into two portions before any training happens. The model only ever sees the training portion during fit(); the test portion is set aside and used purely for evaluation afterward, simulating how the model will behave on genuinely new data.

Example

from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.datasets import make_classification

X, y = make_classification(n_samples=200, n_features=5, random_state=7)

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=7, stratify=y
)

model = LogisticRegression()
model.fit(X_train, y_train)

print('Train accuracy:', round(model.score(X_train, y_train), 3))
print('Test accuracy:', round(model.score(X_test, y_test), 3))

Set	Purpose	Typical Size
Training set	Fit the model's parameters	70 to 80% of the data
Test set	Estimate real-world performance on unseen data	20 to 30% of the data

There is no single correct split ratio. An 80/20 split is a common default; with very large datasets you can afford a smaller test fraction (like 10%) since even that slice contains plenty of examples, while smaller datasets often use a 70/30 split to leave the test set statistically meaningful.

Note: Set random_state to a fixed integer so the split is reproducible every time you rerun the code. Use stratify=y for classification problems so the class proportions in the test set match the training set, which matters most when classes are imbalanced.

A single train/test split gives one estimate of performance, which can vary depending on which rows happened to land in the test set. Cross-validation, which repeats the split-train-evaluate cycle across several different folds and averages the results, gives a more robust estimate at the cost of extra computation.

Note: Never let information from the test set influence training, a mistake known as data leakage. This includes fitting scalers, imputers, or feature selectors on the full dataset before splitting instead of fitting them only on the training data.

The test set exists to reveal overfitting by measuring performance on data the model never trained on.
train_test_split randomly partitions data; use random_state for reproducibility and stratify for balanced classes.
Cross-validation extends the idea by averaging over multiple splits for a more reliable estimate.
Fit any preprocessing steps on the training data only to avoid leakage.

Exercise: Data Science Train Test

Why is a dataset typically split into training and testing sets?

Keep Going

Data Science Interview QuestionsPractice real questions asked for Data Science Data Scientist SalarySee what Data Scientist roles pay in the market Auto Apply for JobsApply to open roles related to Data Science and tweak your resume