Big Data vs Regular Programming

# Regular programming — Python example
# Read orders from a database, compute totals, return response.

import sqlite3

def get_top_customers(db_path):
    conn = sqlite3.connect(db_path)
    cursor = conn.cursor()
    cursor.execute("""
        SELECT customer_id, SUM(amount) AS total
        FROM orders
        GROUP BY customer_id
        ORDER BY total DESC
        LIMIT 10
    """)
    rows = cursor.fetchall()
    conn.close()
    return rows

# Runs on one machine, one process, GB-scale data.

Regular programming — characteristics:
- Single machine (vertical scaling)
- KB to GB data
- ACID transactions
- Low latency per request
- Simple deployment
- Monolith or microservices
- Languages: Java, Python, C#, JS
- Storage: RDBMS, files, cache

# Big data programming — PySpark example
# Read orders from a data lake, compute totals, write results.

from pyspark.sql import SparkSession
from pyspark.sql.functions import sum, desc

spark = SparkSession.builder.appName("TopCustomers").getOrCreate()

orders = spark.read.parquet("s3://data-lake/orders/")

top_customers = (
    orders.groupBy("customer_id")
          .agg(sum("amount").alias("total"))
          .orderBy(desc("total"))
          .limit(10)
)

top_customers.write.mode("overwrite").parquet(
    "s3://data-lake/top_customers/"
)

# Runs across a cluster, TB-PB scale data.

Big data programming — characteristics:
- Distributed cluster (horizontal scaling)
- TB to PB+ data
- Batch + stream processing
- Eventual consistency (often)
- High throughput over low latency
- Fault tolerance via replication
- Tools: Hadoop, Spark, Kafka, Flink
- Storage: HDFS, S3, data lakehouse

# Regular: pandas (single machine, GB scale)
import pandas as pd

df = pd.read_csv("orders.csv")
result = (
    df.groupby("customer_id")["amount"]
      .sum()
      .sort_values(ascending=False)
      .head(10)
)

# Big data: PySpark (cluster, TB scale)
from pyspark.sql import SparkSession
from pyspark.sql.functions import sum, desc

spark = SparkSession.builder.getOrCreate()
df = spark.read.parquet("s3://data-lake/orders/")
result = (
    df.groupBy("customer_id")
      .agg(sum("amount").alias("total"))
      .orderBy(desc("total"))
      .limit(10)
)

When regular programming breaks:
- File no longer fits in RAM
- Query takes hours instead of seconds
- Data arrives from many sources (logs, events, IoT)
- Need batch + streaming in one pipeline
- Need historical + real-time views

When big data is overkill:
- Data fits comfortably on one machine
- Low latency is critical (e.g. payments)
- Team has no distributed systems experience
- Cost of cluster > value of insights

Choose Regular Programming when:
□ Data fits on a single machine (KB-GB)
□ Low latency matters (ms)
□ Strong ACID transactions required
□ Team is small and time-to-market is short
□ Cost of distributed system > value gained

Choose Big Data when:
□ Data is TB-PB+ and growing
□ Data arrives from many sources (logs, IoT, events)
□ Batch + stream processing required
□ Historical + real-time analytics needed
□ ML pipelines on large datasets
□ Cloud data lake/warehouse is available

Daily Learning Tracker:
Date: _________
☐ Concept study (1 hr)
☐ Hands-on coding (1 hr)
☐ Reading / docs (30 min)
☐ Notes + examples
☐ Reviewed progress

Today's Learnings:
1. 
2. 
3. 

Tomorrow's Targets:
1. 
2. 
3. 

Skills Tracker:
Regular Programming:
□ Python / Java / JS
□ SQL & RDBMS
□ REST APIs
□ Git & testing

Big Data:
□ Linux & shell
□ Hadoop HDFS / MapReduce
□ Spark (PySpark / Scala)
□ Kafka / streaming
□ Data lakehouse (Delta/Iceberg)
□ Cloud (S3, EMR, Databricks)

Status: ☐ Not started  ☐ Learning  ☐ Comfortable