Data Cleaning using Pandas

Real-world data is almost never perfectly clean - it usually has missing values, duplicate rows, and inconsistent formatting. Data cleaning is the process of fixing these issues so the data is ready for reliable analysis.

Finding Missing Values

import pandas as pd

df = pd.read_csv("students.csv")

print(df.isnull().sum())   # count of missing values per column

Handling Missing Values

df.dropna(inplace=True)                     # remove rows with any missing value
df["Score"].fillna(df["Score"].mean(), inplace=True)   # fill with the column mean

Removing Duplicate Rows

print(df.duplicated().sum())   # count of duplicate rows
df.drop_duplicates(inplace=True)

Fixing Data Types

df["Score"] = df["Score"].astype(int)
df["JoinDate"] = pd.to_datetime(df["JoinDate"])

Renaming and Standardising Text

df.rename(columns={"Nm": "Name"}, inplace=True)
df["City"] = df["City"].str.strip().str.title()

Data cleaning typically takes up the largest share of time in any real data science project - a model is only as reliable as the data it's trained on, so this step is never worth rushing.

Coming Up Next

Next, you'll bring all of this together while exploring a real dataset from start to finish.