Big Data Roadmap 2026 Step-by-Step Learning Path

# pip install pandas pyarrow
import os
import pandas as pd

df = pd.read_csv("trips.csv")
df.to_parquet("trips.parquet", index=False)

csv_mb = os.path.getsize("trips.csv") / 1e6
pq_mb = os.path.getsize("trips.parquet") / 1e6
print(f"CSV: {csv_mb:.1f} MB, Parquet: {pq_mb:.1f} MB")

# Read only the columns you need - a key Parquet benefit
subset = pd.read_parquet("trips.parquet", columns=["pickup_time", "fare"])
print(subset.head())

# Edit your crontab
crontab -e

# Run the clean-up script every day at 1:30 AM
# and append output and errors to a log file
30 1 * * * /usr/bin/python3 /home/user/etl/clean.py >> /home/user/etl/clean.log 2>&1

# List current cron jobs
crontab -l

# Check the log after the run
tail -n 20 /home/user/etl/clean.log

from pyspark.sql import SparkSession, functions as F

spark = SparkSession.builder.master("local[*]").appName("roadmap").getOrCreate()

trips = spark.read.parquet("trips.parquet")
print(trips.rdd.getNumPartitions())     # how work is split

hourly = (trips
    .withColumn("hour", F.hour("pickup_time"))
    .groupBy("hour")
    .agg(F.count("*").alias("trips"), F.avg("fare").alias("avg_fare"))
    .orderBy("hour"))

hourly.explain()   # nothing has run yet - lazy evaluation
hourly.show(24)    # an action triggers the job

# continuing from the DataFrame tab
trips.createOrReplaceTempView("trips")

result = spark.sql("""
    SELECT HOUR(pickup_time) AS hour,
           COUNT(*)          AS trips,
           ROUND(AVG(fare), 2) AS avg_fare
    FROM trips
    WHERE fare > 0
    GROUP BY HOUR(pickup_time)
    ORDER BY trips DESC
    LIMIT 5
""")
result.show()

# Open http://localhost:4040 while the session runs to see the Spark UI

from pyspark.sql import SparkSession, functions as F

# Requires the spark-sql-kafka package matching your Spark version
spark = SparkSession.builder.appName("clicks_per_minute").getOrCreate()

raw = (spark.readStream.format("kafka")
       .option("kafka.bootstrap.servers", "localhost:9092")
       .option("subscribe", "clicks")
       .load())

clicks = raw.selectExpr("CAST(value AS STRING) AS page", "timestamp")

per_minute = (clicks
    .withWatermark("timestamp", "2 minutes")
    .groupBy(F.window("timestamp", "1 minute"), "page")
    .count())

query = per_minute.writeStream.outputMode("update").format("console").start()
query.awaitTermination()

# Create a topic (Kafka installed locally)
bin/kafka-topics.sh --create --topic clicks \
  --bootstrap-server localhost:9092 --partitions 3 --replication-factor 1

# Send test messages (type a page name per line)
bin/kafka-console-producer.sh --topic clicks --bootstrap-server localhost:9092

# Read them back from the beginning
bin/kafka-console-consumer.sh --topic clicks \
  --bootstrap-server localhost:9092 --from-beginning

Step | Topic                  | Exercise done | Notes written | Date finished
-----|------------------------|---------------|---------------|--------------
1a   | SQL                    |               |               |
1b   | Python                 |               |               |
2a   | Linux + cron           |               |               |
2b   | Data formats           |               |               |
3    | Distributed storage    |               |               |
4    | Spark (API, internals) |               |               |
5    | Kafka + streaming      |               |               |
6    | Cloud data stack       |               |               |
7    | Airflow + capstone     |               |               |

Capstone: ____________________

Source: public API or open dataset ________
Ingestion: batch / streaming
Raw zone: object storage path ________
Processing: Spark job(s) ________
Curated tables: ________
Serving: warehouse table / dashboard ________
Schedule: Airflow DAG, daily at ____
Quality checks: row counts, nulls, duplicates
Monitoring: alerts on failure
Cost control: budget alert set? Yes / No

Week ___   Current step: ___

Hours spent: ___
Exercise completed: Yes / No
Concept I can now explain:
_______________________
Still confusing:
_______________________
Next week's focus:
_______________________