Pandas Removing Duplicates

Detecting Duplicate Rows

A duplicate row is one where every column matches another row exactly. duplicated() scans the DataFrame and returns True for every row that is an exact repeat of an earlier one (the first occurrence is always marked False), so you can count or inspect duplicates before deciding what to do about them.

import pandas as pd

data = {
    'OrderID': [101, 102, 103, 102, 104],
    'Customer': ['Ann', 'Ben', 'Cleo', 'Ben', 'Dev'],
    'Amount': [250.0, 180.0, 175.5, 180.0, 320.0]
}
df = pd.DataFrame(data)

print(df.duplicated())
# 0    False
# 1    False
# 2    False
# 3     True   <- exact repeat of row 1
# 4    False
print('Duplicate rows:', df.duplicated().sum())

Removing Duplicates with drop_duplicates()

drop_duplicates() removes every row that duplicated() flagged as True, keeping the first occurrence of each unique row by default. Like most pandas cleaning methods, it returns a new DataFrame unless you pass inplace=True.

import pandas as pd

data = {
    'OrderID': [101, 102, 103, 102, 104],
    'Customer': ['Ann', 'Ben', 'Cleo', 'Ben', 'Dev'],
    'Amount': [250.0, 180.0, 175.5, 180.0, 320.0]
}
df = pd.DataFrame(data)

df.drop_duplicates(inplace=True)
print(df)

Duplicates Based on Specific Columns

Sometimes two rows should count as duplicates even if one column differs, like an OrderID that was re-generated on a resend. The subset parameter restricts the comparison to the columns you name, and keep controls which occurrence survives: 'first' (default), 'last', or False to drop every copy including the first.

import pandas as pd

data = {
    'OrderID': [101, 102, 103, 109, 104],
    'Customer': ['Ann', 'Ben', 'Cleo', 'Ben', 'Dev'],
    'Amount': [250.0, 180.0, 175.5, 180.0, 320.0]
}
df = pd.DataFrame(data)

# Treat rows as duplicates if Customer and Amount match, ignoring OrderID
df_unique = df.drop_duplicates(subset=['Customer', 'Amount'], keep='last')
print(df_unique)

• keep='first' (default) keeps the first occurrence and drops the rest
• keep='last' keeps the final occurrence, useful when later rows have more up-to-date data
• keep=False drops every row involved in a duplicate match, keeping none of them
• subset=[...] narrows the comparison to specific columns instead of the whole row