Modern big data syllabus checklist
☐ SQL and Python refresher
☐ Distributed computing concepts (partitions, shuffles)
☐ Spark DataFrames and Spark SQL
☐ Parquet, partitioning, file sizing
☐ Joins and performance tuning in Spark
☐ Streaming concepts and Kafka basics
☐ Cloud data lake and managed Spark overview
☐ Orchestration basics
☐ End-to-end capstone

from pyspark.sql import SparkSession, functions as F

spark = SparkSession.builder.appName("orders").getOrCreate()
orders = spark.read.parquet("data/orders/")
daily = (orders.groupBy(F.to_date("order_ts").alias("day"), "city")
               .agg(F.sum("amount").alias("revenue"),
                    F.countDistinct("customer_id").alias("customers")))
daily.orderBy("day", "city").show(10)

from pyspark.sql import functions as F

orders = spark.read.parquet("data/orders/")          # large
cities = spark.read.parquet("data/city_lookup/")     # small lookup table

joined = orders.join(F.broadcast(cities), on="city_id", how="left")
joined.explain()   # look for BroadcastHashJoin in the plan

(daily
   .withColumn("month", F.date_format("day", "yyyy-MM"))
   .repartition("month")
   .write.mode("overwrite")
   .partitionBy("month")
   .parquet("output/daily_revenue/"))
# Partitioning by month lets later queries read only the months they need

Big data capstone (what to expect from a strong class)
1. Ingest raw event files into a data lake (Parquet)
2. Spark batch job: clean, deduplicate, aggregate daily metrics
3. Tune the job: partitions, broadcast join, file sizes
4. Streaming lab: consume events from Kafka, update running counts
5. Orchestrate the batch job on a schedule
6. Document architecture, costs and performance results

# Practise Spark locally between classes
pip install pyspark
python -c "from pyspark.sql import SparkSession; s=SparkSession.builder.getOrCreate(); print(s.version)"

# Larger practice: generate a few million rows to see partition effects
python - <<'PY'
from pyspark.sql import SparkSession, functions as F
spark = SparkSession.builder.getOrCreate()
df = spark.range(5_000_000).withColumn("city", (F.col("id") % 50).cast("string"))
print(df.rdd.getNumPartitions())
PY

Big Data Class Scorecard (1-5)
                       Class A   Class B   Class C
Modern syllabus          ___       ___       ___
Spark depth (tuning)     ___       ___       ___
Streaming coverage       ___       ___       ___
Cluster access           ___       ___       ___
Capstone quality         ___       ___       ___
Batch and support        ___       ___       ___
TOTAL                    ___       ___       ___

Before joining a big data class
☐ SQL: joins, aggregates, window functions
☐ Python: functions, files, pandas basics
☐ Linux: navigating, permissions, processes
☐ Basic data modelling
Without these, Spark classes become much harder.

Questions for the demo
1. What share of labs use Spark?
2. Is streaming covered with hands-on work?
3. Where do labs run — local, cluster or cloud?
4. Can I see a past capstone brief?
5. Which batches and support options are available?