ML Confusion Matrix

Reading a Confusion Matrix

A model that is 95% accurate sounds impressive, but if only 5% of the examples in your dataset belong to the class you actually care about, a model that predicts "negative" every single time is also 95% accurate — and completely useless. A confusion matrix fixes this blind spot by breaking predictions down into four specific outcomes instead of one aggregate score.

For a binary spam filter, every email falls into one of four buckets: a True Positive (TP) is spam correctly flagged as spam, a True Negative (TN) is a real email correctly left alone, a False Positive (FP) is a real email wrongly flagged as spam, and a False Negative (FN) is spam that slipped through to the inbox. Counting how many predictions land in each bucket tells you exactly what kind of mistakes a model is making, not just how many mistakes.

from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import confusion_matrix

X, y = make_classification(n_samples=500, weights=[0.85, 0.15], random_state=0)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, random_state=0, stratify=y
)

model = LogisticRegression().fit(X_train, y_train)
predictions = model.predict(X_test)

cm = confusion_matrix(y_test, predictions)
print("Confusion matrix:\n", cm)
print("TN:", cm[0, 0], "FP:", cm[0, 1], "FN:", cm[1, 0], "TP:", cm[1, 1])

Precision, Recall, and F1-Score

Precision answers "of everything the model flagged as positive, how much actually was?" — TP / (TP + FP). Recall answers "of everything that was actually positive, how much did the model catch?" — TP / (TP + FN). F1-score is the harmonic mean of the two, giving a single number that only stays high when both precision and recall are reasonably high.

• Prioritize precision when false positives are costly — e.g. flagging a legitimate email as spam and hiding it from the user.
• Prioritize recall when false negatives are costly — e.g. a medical screening test that misses an actual case of disease.
• Use F1-score when you need one balanced metric and neither type of mistake is clearly worse than the other.
• Look at both numbers together — a model with high precision and low recall behaves very differently from one with the reverse, even if their F1-scores happen to match.

from sklearn.metrics import classification_report

print(classification_report(y_test, predictions, target_names=["not spam", "spam"]))

from sklearn.metrics import precision_score, recall_score

probabilities = model.predict_proba(X_test)[:, 1]
high_recall_predictions = (probabilities >= 0.3).astype(int)  # lower threshold = catch more positives

print("Default threshold  - precision:", precision_score(y_test, predictions),
      "recall:", recall_score(y_test, predictions))
print("Lower threshold 0.3 - precision:", precision_score(y_test, high_recall_predictions),
      "recall:", recall_score(y_test, high_recall_predictions))