Best Big Data Training Institute in Ghaziabad for 2026

from pyspark.sql import SparkSession
from pyspark.sql import functions as F

spark = SparkSession.builder.appName("sales").getOrCreate()
df = spark.read.csv("sales.csv", header=True, inferSchema=True)

summary = (df.groupBy("region")
             .agg(F.sum("amount").alias("total"),
                  F.countDistinct("order_id").alias("orders"))
             .orderBy(F.desc("total")))
summary.show()
summary.write.mode("overwrite").parquet("output/region_summary")

df.createOrReplaceTempView("sales")
spark.sql('''
    SELECT region,
           SUM(amount) AS total,
           RANK() OVER (ORDER BY SUM(amount) DESC) AS rnk
    FROM sales
    GROUP BY region
''').show()

# Common performance steps in Spark
df = spark.read.parquet("events/")        # columnar format
df = df.filter("event_date >= '2026-10-01'")
df = df.repartition(64, "user_id")        # balance work
df.cache()                                 # reuse in many steps
print(df.rdd.getNumPartitions())
# Then check stages and skew in the Spark UI (port 4040 locally)

# Practise Spark on your own laptop
pip install pyspark
python -c "from pyspark.sql import SparkSession; \
s=SparkSession.builder.getOrCreate(); print(s.version)"
# Spark UI while a job runs: http://localhost:4040

from pyspark.sql import SparkSession, functions as F

spark = SparkSession.builder.appName("clicks").getOrCreate()
clicks = (spark.readStream.format("kafka")
          .option("kafka.bootstrap.servers", "localhost:9092")
          .option("subscribe", "clicks")
          .load()
          .selectExpr("CAST(value AS STRING) AS page", "timestamp"))

counts = (clicks.withWatermark("timestamp", "2 minutes")
          .groupBy(F.window("timestamp", "1 minute"), "page").count())

query = counts.writeStream.outputMode("update").format("console").start()
query.awaitTermination()
# Needs the spark-sql-kafka package matching your Spark version

# Retail Sales Lakehouse (Spark)
Layers: bronze (raw CSV) -> silver (clean Parquet) -> gold (aggregates)
Checks: row counts, null keys, duplicate orders
Schedule: daily batch
Results: gold tables feed a sales dashboard
Lessons: partitioning by date cut job time noticeably

Big data course checklist             Course A   Course B
SQL + window functions                [ ]        [ ]
Hadoop/HDFS concepts                  [ ]        [ ]
Apache Spark (DataFrames + SQL)       [ ]        [ ]
Kafka / streaming                     [ ]        [ ]
Cloud data platform basics            [ ]        [ ]
I run jobs myself in labs             [ ]        [ ]
End-to-end pipeline project           [ ]        [ ]
Mock interviews                       [ ]        [ ]
Batch time fits my routine            [ ]        [ ]

1. Which Spark version and language (PySpark/Scala) is used?
2. How do learners practise - local, cluster or cloud?
3. What datasets are used in labs?
4. What does the final project include?
5. Are classes live, recorded or both?
6. Can I repeat a batch if I miss classes?
7. What is the current fee?

Phase 1: SQL + Python refresh
Phase 2: Spark DataFrames and Spark SQL
Phase 3: File formats, partitioning, performance
Phase 4: Kafka and Structured Streaming
Phase 5: Cloud storage + orchestration basics
Phase 6: Capstone pipeline + interview practice