Data Reshaping and Cleaning

Raw data is rarely analysis-ready. tidyr reshapes messy data into "tidy" format, while a set of core techniques handles missing values and outliers — the unglamorous but essential 80% of any real project.

1. What is "Tidy Data"?

Tidy data follows three rules: (1) each variable is a column, (2) each observation is a row, (3) each value is a single cell. Most real-world data violates these rules and needs reshaping.

2. Reshaping with pivot_longer() and pivot_wider()

library(tidyr)

# Wide format: one column per year
wide_df <- data.frame(
  country = c("USA", "India"),
  y2023 = c(100, 200),
  y2024 = c(110, 220)
)

# Convert to long (tidy) format
long_df <- wide_df |>
  pivot_longer(cols = c(y2023, y2024),
               names_to = "year", values_to = "value")

# Convert back to wide format
long_df |> pivot_wider(names_from = year, values_from = value)

3. Splitting and Combining Columns

df |> separate(full_name, into = c("first", "last"), sep = " ")
df |> unite(full_name, first, last, sep = " ")

4. Handling Missing Values (NA)

R represents missing data with NA. It propagates through calculations unless explicitly handled.

df <- data.frame(x = c(1, NA, 3, NA, 5))

is.na(df$x)              # TRUE where missing
sum(is.na(df$x))         # count of missing values
mean(df$x, na.rm = TRUE) # ignore NAs in calculation

# Remove rows with any NA
df_clean <- df |> tidyr::drop_na()

# Replace NAs with a value
df |> tidyr::replace_na(list(x = 0))

5. Detecting Outliers

A common, simple approach uses the Interquartile Range (IQR):

Q1 <- quantile(df$x, 0.25, na.rm = TRUE)
Q3 <- quantile(df$x, 0.75, na.rm = TRUE)
IQR_val <- Q3 - Q1

lower <- Q1 - 1.5 * IQR_val
upper <- Q3 + 1.5 * IQR_val

outliers <- df$x[df$x < lower | df$x > upper]
print(outliers)

Boxplots (covered in the next lesson) are a fast visual way to spot the same outliers.

6. Standardizing Column Types and Text

library(stringr)

df$name <- str_trim(df$name)          # remove whitespace
df$name <- str_to_title(df$name)      # standardize casing
df$category <- as.factor(df$category) # convert to categorical

7. Production-Ready Checklist

• ✅ Understand tidy data principles — one variable per column, one observation per row.
• ✅ Use pivot_longer()/pivot_wider() — to reshape between wide and long formats.
• ✅ Handle NAs deliberately — investigate, then impute or remove intentionally.
• ✅ Detect outliers with IQR or visualization — before running statistical models.