ML AUC ROC Curve

True Positives, False Positives, and the Decision Threshold

Most classifiers do not output a hard 0 or 1 label directly; they output a probability, and a threshold (0.5 by default) decides where the cutoff between the two classes falls. The true positive rate (TPR, also called recall) is the fraction of actual positives the model correctly flags, and the false positive rate (FPR) is the fraction of actual negatives the model incorrectly flags as positive. Moving the threshold up or down trades one of these off against the other: lowering it catches more true positives but also lets in more false positives.

The ROC (Receiver Operating Characteristic) curve plots FPR on the x-axis against TPR on the y-axis at every possible threshold, tracing out a curve from (0, 0), a threshold so strict nothing is predicted positive, to (1, 1), a threshold so loose everything is. A curve that hugs the top-left corner reaches a high TPR while keeping FPR low, which is what a strong classifier looks like; a curve that sits on the diagonal is indistinguishable from random guessing.

from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import roc_curve

X, y = make_classification(n_samples=500, n_features=10, weights=[0.6, 0.4], random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)

model = LogisticRegression(max_iter=1000)
model.fit(X_train, y_train)

probabilities = model.predict_proba(X_test)[:, 1]
fpr, tpr, thresholds = roc_curve(y_test, probabilities)

print('Number of thresholds evaluated:', len(thresholds))
print('First few FPR values:', fpr[:5])
print('First few TPR values:', tpr[:5])

AUC: A Single Number Summary

The area under the ROC curve (AUC) compresses the entire curve into one number between 0 and 1. It has a clean probabilistic interpretation: AUC is the probability that the model ranks a randomly chosen positive example higher than a randomly chosen negative one. Because it is computed across every threshold at once, AUC judges a model's ability to rank and separate classes without committing to any particular cutoff, which makes it a popular metric for comparing models before a specific threshold has been chosen for production.

from sklearn.metrics import roc_auc_score
from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split

X, y = make_classification(n_samples=500, n_features=10, random_state=1)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=1)

model = RandomForestClassifier(n_estimators=200, random_state=1)
model.fit(X_train, y_train)

probabilities = model.predict_proba(X_test)[:, 1]
auc = roc_auc_score(y_test, probabilities)
print('AUC:', round(auc, 3))

• AUC = 1.0 means the model perfectly separates the two classes at some threshold
• AUC = 0.5 means the model performs no better than random guessing
• AUC below 0.5 means the model's ranking is inverted, so flipping its predictions would actually help
• AUC is threshold-independent, so it summarizes ranking quality across every possible cutoff at once

import matplotlib.pyplot as plt
from sklearn.metrics import RocCurveDisplay
from sklearn.linear_model import LogisticRegression
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split

X, y = make_classification(n_samples=500, n_features=10, random_state=5)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=5)

model = LogisticRegression(max_iter=1000).fit(X_train, y_train)

RocCurveDisplay.from_estimator(model, X_test, y_test)
plt.plot([0, 1], [0, 1], linestyle='--', label='Random guess')
plt.legend()
plt.show()