Pandas Cleaning Data

Why Clean Data Before You Analyze It

Every calculation you run on a DataFrame trusts that the data is accurate. A mean, a chart, or a machine learning model built on top of empty cells, garbled dates, impossible values, or duplicate rows will happily produce a number — it just won't be a number you can trust. Cleaning data is the step where you find and fix those problems before they quietly corrupt your results.

Four Common Data Quality Problems

• Empty cells — missing values that show up as NaN or None
• Wrong format — data stored as the wrong type, like dates saved as plain text
• Wrong data — values that are technically present but factually impossible or out of range
• Duplicates — the same row appearing more than once, which inflates counts and sums

Each of these problems has a specific downstream cost. Empty cells break aggregate functions or get silently excluded from them. Wrong formats make date arithmetic and sorting fail. Wrong data skews your mean and standard deviation. Duplicates double-count revenue, customers, or events. None of these announce themselves loudly — they just make your conclusions wrong.

import pandas as pd
import numpy as np

data = {
    'OrderID': [101, 102, 103, 104],
    'OrderDate': ['2021-03-01', '03/02/2021', 'Unknown', '2021-03-04'],
    'Age': [34, 29, 190, 41],
    'Amount': [250.0, np.nan, 175.5, 260.0]
}
df = pd.DataFrame(data)

print(df.info())
print(df.isnull().sum())

A Typical Cleaning Workflow

There is a natural order to cleaning: first inspect with info() and describe() to find the problems, then handle empty cells, then fix data types and formats, then fix or remove wrong values, then remove duplicates, and finally re-run info() and describe() to confirm the DataFrame is now trustworthy. Skipping the final check is a common mistake — always verify that your cleaning actually worked.

import pandas as pd
import numpy as np

data = {
    'OrderID': [101, 102, 103, 104, 104],
    'OrderDate': ['2021-03-01', '03/02/2021', 'Unknown', '2021-03-04', '2021-03-04'],
    'Age': [34, 29, 190, 41, 41],
    'Amount': [250.0, np.nan, 175.5, 260.0, 260.0]
}
df = pd.DataFrame(data)

df_clean = df.dropna().drop_duplicates()
print(df_clean)

import pandas as pd
import numpy as np

data = {
    'OrderID': [101, 102, 103, 104, 104],
    'OrderDate': ['2021-03-01', '03/02/2021', 'Unknown', '2021-03-04', '2021-03-04'],
    'Age': [34, 29, 190, 41, 41],
    'Amount': [250.0, np.nan, 175.5, 260.0, 260.0]
}
df = pd.DataFrame(data)

df_clean = df.dropna().drop_duplicates()
print(df_clean.info())
print(df_clean.describe())