Big Data Career Path From Fresher to Expert Level

-- Daily check: did yesterday's load arrive completely?
SELECT
  load_date,
  COUNT(*)              AS rows_loaded,
  COUNT(DISTINCT order_id) AS unique_orders,
  SUM(CASE WHEN amount IS NULL THEN 1 ELSE 0 END) AS null_amounts
FROM staging_orders
WHERE load_date = CURRENT_DATE - 1
GROUP BY load_date;

-- Flag duplicates before moving to the final table
SELECT order_id, COUNT(*)
FROM staging_orders
WHERE load_date = CURRENT_DATE - 1
GROUP BY order_id
HAVING COUNT(*) > 1;

# Find failed job runs in a log file
grep -i "error" etl_job.log | tail -n 20

# Count errors by type
grep -io "error: [a-z ]*" etl_job.log | sort | uniq -c | sort -nr

# Check disk space before re-running a large job
df -h /data

# See the newest files in the landing folder
ls -lt /data/landing | head

from pyspark.sql import SparkSession, functions as F

spark = SparkSession.builder.appName("daily_orders").getOrCreate()

orders = spark.read.option("header", True).csv("/data/landing/orders/")

clean = (orders
    .withColumn("amount", F.col("amount").cast("double"))
    .dropna(subset=["order_id", "amount"])
    .dropDuplicates(["order_id"])
    .withColumn("order_date", F.to_date("order_ts")))

daily = clean.groupBy("order_date", "city").agg(
    F.sum("amount").alias("revenue"),
    F.countDistinct("customer_id").alias("customers"))

daily.write.mode("overwrite").partitionBy("order_date").parquet("/data/curated/daily_orders")

from datetime import datetime
from airflow import DAG
from airflow.operators.bash import BashOperator

with DAG(
    dag_id="daily_orders",
    start_date=datetime(2026, 1, 1),
    schedule="0 2 * * *",      # every day at 02:00
    catchup=False,
) as dag:
    ingest = BashOperator(task_id="ingest", bash_command="python ingest.py")
    transform = BashOperator(task_id="transform",
                             bash_command="spark-submit orders_pipeline.py")
    check = BashOperator(task_id="quality_check", bash_command="python checks.py")

    ingest >> transform >> check

Data pipeline design document

1. Context: which business need does this serve?
2. Sources: systems, volume, frequency, owners
3. Options considered:
   A) Daily batch  B) Micro-batch  C) Streaming
4. Decision and reasons (latency, cost, complexity)
5. Storage layout: zones, formats, partitioning
6. Data quality checks and alerting
7. Security: access roles, PII handling
8. Cost estimate and monitoring
9. Rollout and backfill plan
10. Open questions

Pipeline code review checklist (expert)

[ ] Idempotent: safe to re-run for the same date
[ ] Partitioning matches common query filters
[ ] Small files avoided or compacted
[ ] Schema changes handled explicitly
[ ] Data quality checks fail loudly
[ ] Secrets not in code
[ ] PII masked or restricted
[ ] Logs useful for on-call engineers
[ ] Cost impact understood

Where am I now?

Current role: ___________________
Tools I use weekly: ______________
Largest dataset I have worked with: ______
Have I built a pipeline end to end?  Yes / No
Have I fixed a production failure?  Yes / No
Can I explain partitioning and why it matters?  Yes / No

My level today: Fresher / Engineer / Senior / Expert

Target level: ____________  Target date: ________

Skills to add (max 3 at a time):
1. ____________  how I will practise: ____________
2. ____________  how I will practise: ____________
3. ____________  how I will practise: ____________

Proof I will create:
[ ] Project or pipeline
[ ] Design note or blog post
[ ] Certification (optional)

Person who can give me feedback: ____________

Quarter: ____

What I shipped:
-
What broke and what I learned:
-
New skill now used at work:
-
Feedback received:
-
Adjustments for next quarter:
-