9 programs
Choose the best training program according to your career.
Improve the performance of your business with our innovative Industrial Training Program.
Read More →
Our instructor-led training classes are designed to make you an expert.
Read More →
Real-life case studies are a great way to demonstrate your practical knowledge.
Read More →
Explore blogs, latest technologies and industry updates with practical examples.
Read More →
Browse all available training programs and choose the right one for your career.
Read More →Know more about who we are and what we stand for.
Discover our mission to equip individuals with essential IT skills and industry expertise.
Read More →Learn, practice and plan your career — bilkul free.
Book a free video counselling session and pick the right course for you.
Read More →Download the Python data cleaning guide PDF from Uncodemy: missing values, duplicates, types, text and outliers with pandas examples.
This page offers Uncodemy's Python data cleaning guide as a PDF download. It shows how to detect and fix missing values, duplicates, wrong types, messy dates, inconsistent text and outliers using pandas, and how to wrap the steps into a function with checks so that the cleaning is repeatable. The examples build a small messy table in code.
Download the guide below. Each download is its own file, and everything in it is also explained on this page.
Fixing text, duplicates, types, dates, missing values and outliers with pandas, a reusable cleaning function with assertions, and checks to run after every job, on a small messy customers table.
A short form asks for your details before the download starts. Your details are used only to share study material and course information.
When data cleaning is written as code rather than done by hand, three good things happen. The steps are documented, because the code is the record of what changed. They are repeatable, because the next file can be cleaned by running the same script. And they are reviewable, because a colleague can read the rules and challenge them.
The price is a little more discipline: you must decide each rule explicitly instead of fixing things as you notice them. The example below uses a small messy customers table so that you can run every step and see the effect.
The table has duplicates, inconsistent text, missing values and dates in text form.
import pandas as pd
import numpy as np
raw = pd.DataFrame({
"customer_id": [1, 2, 2, 3, 4, 5],
"name": [" Asha ", "ravi", "ravi", "MEERA", None, "Kabir"],
"city": ["delhi", "Noida ", "Noida ", "NEW DELHI", "Pune", None],
"signup_date": ["2026-01-05", "05/02/2026", "05/02/2026", "2026-03-10", "bad date", "2026-04-01"],
"spend": ["500", "1,200", "1,200", None, "300", "-50"],
})
print(raw)
print(raw.isna().sum())
print(raw.duplicated(subset=["customer_id"]).sum(), "duplicate ids")Standardise text, map known variants to one label and remove duplicates deliberately, keeping the first row for each id.
df = raw.copy()
df["name"] = df["name"].str.strip().str.title()
df["city"] = (df["city"].str.strip().str.title()
.replace({"New Delhi": "Delhi"}))
df = df.drop_duplicates(subset=["customer_id"], keep="first")
print(df)Convert numbers stored as text, parse dates and flag impossible values rather than silently deleting them. errors="coerce" turns failures into missing values, which you can then count.
df["spend"] = pd.to_numeric(df["spend"].str.replace(",", "", regex=False), errors="coerce")
df["signup_date"] = pd.to_datetime(df["signup_date"], errors="coerce", dayfirst=False, format="mixed")
df["spend_invalid"] = df["spend"] < 0
df.loc[df["spend_invalid"], "spend"] = np.nan
print(df.dtypes)
print("unparsed dates:", df["signup_date"].isna().sum())After conversion you will have missing values from the original blanks and from values you could not convert. Choose a rule for each column. For a name, a label such as Unknown keeps the row. For spend, the median is a common neutral fill for skewed values, but if the amount drives a calculation you may prefer to leave it missing and report it.
Whatever you do, create a flag column or keep a count, so downstream users know which values were filled. A cleaning step that cannot be seen is a source of mistrust later.
Fill labels, flag the filled spend values and report how many were affected.
df["name"] = df["name"].fillna("Unknown")
df["city"] = df["city"].fillna("Unknown")
df["spend_was_missing"] = df["spend"].isna()
df["spend"] = df["spend"].fillna(df["spend"].median())
print(df[["customer_id", "name", "city", "spend", "spend_was_missing"]])
print("filled:", int(df["spend_was_missing"].sum()))Wrap the rules in a function and add assertions, so a bad input fails loudly instead of producing wrong numbers.
def clean_customers(raw):
df = raw.copy()
df["name"] = df["name"].str.strip().str.title().fillna("Unknown")
df["city"] = (df["city"].str.strip().str.title()
.replace({"New Delhi": "Delhi"}).fillna("Unknown"))
df = df.drop_duplicates(subset=["customer_id"], keep="first")
df["spend"] = pd.to_numeric(df["spend"].str.replace(",", "", regex=False), errors="coerce")
df.loc[df["spend"] < 0, "spend"] = np.nan
df["spend"] = df["spend"].fillna(df["spend"].median())
assert df["customer_id"].is_unique, "ids must be unique"
assert df["spend"].ge(0).all(), "negative spend remains"
return df
clean = clean_customers(raw)
print(len(raw), "->", len(clean), "rows")| Situation | Suggested action |
|---|---|
| Looks like a typing error (extra zero) | Correct from the source or set to missing, with a note |
| Genuine rare event (a very large order) | Keep; report medians or analyse separately |
| Impossible value (negative age) | Mark invalid and investigate |
| Unknown cause | Keep, flag and ask the data owner |
Cleaning rules deserve tests just like any other code. Create a tiny table that contains the exact problems you expect, such as a duplicate id, a negative amount and a date in the wrong format, run your function on it and assert that the output has the properties you want.
When a new problem appears in real data, add it to the test table first and then fix the code. Over time this builds a safety net, so changes to one rule do not silently break another and so a colleague can trust the function without reading every line.
| Check | Code idea |
|---|---|
| Row count explained | print(len(raw), len(clean)) |
| Keys unique | assert clean['customer_id'].is_unique |
| No impossible values | assert (clean['spend'] >= 0).all() |
| Remaining missing values | print(clean.isna().sum()) |
| Types as expected | print(clean.dtypes) |
Want to clean and prepare data the way analysts do at work? Explore Uncodemy's Data Analytics course and learn from industry trainers.