Data Science Scale

Why Features Need a Common Scale

Real datasets mix wildly different units: a person's age might range from 18 to 90, while their income ranges from 20,000 to 500,000. If you feed these raw numbers into an algorithm that measures distance or computes gradients, the feature with the larger numeric range will dominate the calculation purely because of its scale, not because it is actually more informative.

• Distance-based algorithms: K-Nearest Neighbors, K-means clustering, and Support Vector Machines all compare points using distances (usually Euclidean), so a feature with a bigger range silently outweighs the others.
• Gradient-based algorithms: linear regression, logistic regression, and neural networks trained with gradient descent converge faster and more reliably when every feature varies over a similar range.
• Regularized models (Ridge, Lasso): the penalty term is applied uniformly to coefficients, so unscaled features get penalized unevenly.

The Z-score Standardization Formula

The most common scaling technique is standardization, which converts every value to a z-score: z = (x - mean) / standard_deviation. After this transform, each feature has a mean of 0 and a standard deviation of 1, so no single feature can dominate just because its raw numbers happen to be larger.

from sklearn.preprocessing import StandardScaler
import numpy as np

# Two features on very different scales: age (years) and income ($)
X = np.array([
    [25, 42000],
    [32, 58000],
    [47, 120000],
    [51, 95000],
])

scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)

print('Mean of each column after scaling:', X_scaled.mean(axis=0).round(2))
print('Std of each column after scaling:', X_scaled.std(axis=0).round(2))
print(X_scaled.round(2))

An alternative is min-max scaling, which squeezes every value into a fixed range, usually 0 to 1. This is useful when you know the algorithm needs bounded inputs (some neural network activations, for example) or when the data is not normally distributed and you want to preserve the shape of the original distribution.

from sklearn.preprocessing import MinMaxScaler

# Reuses X from the previous example
scaler = MinMaxScaler()  # squeezes every column into [0, 1]
X_scaled = scaler.fit_transform(X)
print(X_scaled.round(2))

• Standardization centers data at mean 0 with standard deviation 1 using the z-score formula.
• Min-max scaling compresses data into a fixed range, typically [0, 1].
• Scale before distance-based or gradient-based algorithms; skip it for tree-based models.
• Always fit the scaler on training data only, then apply it to the test data.