How Uncodemy Trains Students for Data Engineer Roles

-- Week 1: Top 5 customers by revenue in the last 30 days
SELECT
  c.customer_id,
  c.name,
  SUM(o.amount) AS revenue
FROM customers c
JOIN orders o ON o.customer_id = c.customer_id
WHERE o.created_at >= CURRENT_DATE - INTERVAL '30 days'
GROUP BY c.customer_id, c.name
ORDER BY revenue DESC
LIMIT 5;

# Week 3: Clean a messy CSV in pandas
import pandas as pd

df = pd.read_csv("orders.csv")

# Normalise dates
df["created_at"] = pd.to_datetime(df["created_at"], errors="coerce")

# Drop duplicates and null keys
df = df.drop_duplicates(subset=["order_id"])
df = df.dropna(subset=["customer_id", "amount"])

# Standardise currency
df["currency"] = df["currency"].str.upper().fillna("INR")

# Save to parquet for the pipeline
df.to_parquet("orders_clean.parquet", index=False)

from datetime import datetime
from airflow import DAG
from airflow.operators.python import PythonOperator

def extract():
    ...

def transform():
    ...

def load():
    ...

def validate():
    ...

with DAG(
    "orders_pipeline",
    start_date=datetime(2026, 1, 1),
    schedule="0 2 * * *",
    catchup=False,
    default_args={"retries": 2, "retry_delay": 300},
) as dag:
    t1 = PythonOperator(task_id="extract", python_callable=extract)
    t2 = PythonOperator(task_id="transform", python_callable=transform)
    t3 = PythonOperator(task_id="load", python_callable=load)
    t4 = PythonOperator(task_id="validate", python_callable=validate)

    t1 >> t2 >> t3 >> t4

Before a Phase 2 pipeline is "done":
□ Extract step handles failures gracefully
□ Transform step is tested with sample data
□ Load step is idempotent (safe to re-run)
□ Row counts are logged in and out of every step
□ Data quality checks run at the end
□ Alerts fire on failure (email / Slack)
□ The DAG is versioned in Git
□ There is a runbook explaining what to do if it fails

Portfolio Project — Data Pipeline in a Box:
1. Problem statement
   - What data, what question, why it matters
2. Architecture
   - Source → ingestion → transformation → warehouse → dashboard
3. Stack
   - Python, Airflow, dbt, BigQuery or Snowflake
4. Quality
   - Tests for transformation logic
   - Data quality checks at every stage
5. Deployment
   - Dockerfile, docker-compose.yml, CI pipeline
6. Documentation
   - README with diagrams, screenshots, run instructions
7. Results
   - Sample queries, a small dashboard, sample insights

Before a real interview:
□ I can write JOINs and window functions fluently
□ I can explain the difference between star and snowflake schema
□ I can walk through my portfolio pipeline end-to-end
□ I can explain why I chose Airflow vs cron vs dbt
□ I can talk about idempotency and failure handling
□ I can debug a Spark job that runs out of memory
□ I can describe a data quality issue I caught and fixed
□ I can ask two thoughtful questions about the company