Data Engineering Course Syllabus Full Curriculum Breakdown

CREATE TABLE dim_customer (
  customer_key  SERIAL PRIMARY KEY,
  customer_id   VARCHAR(20) NOT NULL,
  name          VARCHAR(100),
  city          VARCHAR(50),
  valid_from    DATE NOT NULL,
  valid_to      DATE,
  is_current    BOOLEAN DEFAULT TRUE
);

CREATE TABLE dim_date (
  date_key   INT PRIMARY KEY,      -- e.g. 20261106
  full_date  DATE NOT NULL,
  month      INT,
  year       INT
);

CREATE TABLE fact_sales (
  sale_id       BIGSERIAL PRIMARY KEY,
  customer_key  INT REFERENCES dim_customer(customer_key),
  date_key      INT REFERENCES dim_date(date_key),
  quantity      INT,
  amount        NUMERIC(12,2)
);

import csv
import requests

URL = "https://jsonplaceholder.typicode.com/users"

def fetch_users():
    resp = requests.get(URL, timeout=10)
    resp.raise_for_status()
    return resp.json()

def save_csv(rows, path="users.csv"):
    fields = ["id", "name", "email"]
    with open(path, "w", newline="") as f:
        writer = csv.DictWriter(f, fieldnames=fields)
        writer.writeheader()
        for r in rows:
            writer.writerow({k: r[k] for k in fields})

if __name__ == "__main__":
    save_csv(fetch_users())
    print("saved users.csv")

from pyspark.sql import SparkSession, functions as F

spark = SparkSession.builder.appName("sales_enrich").getOrCreate()

sales = spark.read.parquet("/lake/raw/sales")
products = spark.read.parquet("/lake/raw/products")   # small table

# Broadcast the small table to avoid a large shuffle
enriched = sales.join(F.broadcast(products), on="product_id", how="left")

result = (enriched
    .groupBy("category", F.to_date("sold_at").alias("sale_date"))
    .agg(F.sum("amount").alias("revenue")))

result.write.mode("overwrite").partitionBy("sale_date").parquet("/lake/curated/revenue")

-- models/marts/fct_daily_revenue.sql
select
    cast(sold_at as date)  as sale_date,
    p.category,
    sum(s.amount)          as revenue
from {{ ref('stg_sales') }} s
left join {{ ref('stg_products') }} p
    on s.product_id = p.product_id
group by 1, 2

-- models/marts/schema.yml
-- models:
--   - name: fct_daily_revenue
--     columns:
--       - name: sale_date
--         tests: [not_null]

from datetime import datetime, timedelta
from airflow import DAG
from airflow.operators.python import PythonOperator

def load_partition(ds, **_):
    # ds = the run date, e.g. '2026-11-06'; reload only that day
    print(f"loading partition for {ds}")

default_args = {"retries": 2, "retry_delay": timedelta(minutes=10)}

with DAG(
    dag_id="daily_sales_load",
    start_date=datetime(2026, 10, 1),
    schedule="@daily",
    catchup=True,            # allows backfilling past dates
    default_args=default_args,
) as dag:
    PythonOperator(task_id="load", python_callable=load_partition)

# pip install kafka-python
import json
from kafka import KafkaConsumer

consumer = KafkaConsumer(
    "orders",
    bootstrap_servers="localhost:9092",
    group_id="order-monitor",
    auto_offset_reset="earliest",
    value_deserializer=lambda b: json.loads(b.decode("utf-8")),
)

for message in consumer:
    order = message.value
    if order.get("amount", 0) > 50000:
        print("High-value order:", order["order_id"])

Module coverage                               Yes / No
Advanced SQL incl. window functions, CTEs      ___
Data modelling: star schema, SCD               ___
Python for ingestion and automation            ___
Linux, cron and Git                            ___
Spark DataFrames, partitions, joins            ___
Parquet and data lake layout                   ___
One cloud warehouse in depth                   ___
dbt or SQL transformation framework            ___
Airflow or similar orchestration               ___
Streaming basics with Kafka                    ___
Data quality, monitoring, cost                 ___

Ask the course team about the capstone:

1. Does it ingest from a real API or source?
2. Is data stored in a lake and a warehouse?
3. Are transformations tested?
4. Is it scheduled with retries and alerts?
5. Can it be re-run safely for a past date?
6. Is it deployed on a cloud account?
7. Will a mentor review the code?

                     | Course A | Course B
---------------------|----------|---------
Coverage score /11   |          |
Lab hours            |          |
Cloud access given   |          |
Capstone quality /7  |          |
Mentor reviews       |          |
Batch timings        |          |
Fees (ask directly)  |          |
Placement support    |          |