Why Every Data Engineer Should Practice Explaining Their Portfolio Aloud

"I have three projects, and I will start with the one
 I am most proud of.

 The first is a batch pipeline that loads public taxi
 trip data into Postgres every day using Airflow, with a
 dbt layer on top. It shows orchestration, modelling and
 testing.

 The second is a small streaming project that reads
 simulated order events from Kafka and keeps per-minute
 totals. It shows how I handle late events.

 The third is older: a Python script that cleans CSV
 exports. I keep it to show where I started.

 Shall I go deeper into the first one?"

# NYC Taxi Daily Pipeline

## What it does
Loads daily taxi trips into Postgres and builds
fct_trips and dim_zones with dbt.

## Architecture
Source files -> Airflow DAG -> raw schema -> dbt -> marts

## Run it locally
    make up       # starts Postgres and Airflow
    make backfill # loads one sample week
    make test     # runs dbt tests

## Design decisions
- Partition overwrite per day so reruns are safe
- Natural key trip_id is hashed from source columns

## What I built vs reused
- Reused: Airflow Docker setup from the official docs
- Built: DAG, dbt models, tests, data quality checks

## What I would improve
- Move from local Postgres to a cloud warehouse

services:
  postgres:
    image: postgres:16
    environment:
      POSTGRES_USER: etl
      POSTGRES_PASSWORD: etl
      POSTGRES_DB: warehouse
    ports:
      - "5432:5432"
    volumes:
      - pgdata:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U etl -d warehouse"]
      interval: 5s
      retries: 10

  etl:
    build: .
    depends_on:
      postgres:
        condition: service_healthy
    environment:
      PGHOST: postgres
      PGUSER: etl
      PGPASSWORD: etl
      PGDATABASE: warehouse
    command: python -m pipeline.run --date 2024-01-07

volumes:
  pgdata:

up:
	docker compose up -d postgres

run:
	docker compose run --rm etl

test:
	docker compose run --rm etl dbt test --project-dir dbt

down:
	docker compose down -v

# Spoken version:
# "If you clone the repo, 'make up' starts Postgres,
#  'make run' loads one sample day and 'make test' runs
#  the dbt tests. The ETL container waits for the database
#  health check, so it does not fail on a cold start."

# dbt/models/marts/schema.yml
version: 2

models:
  - name: fct_trips
    description: One row per completed taxi trip.
    columns:
      - name: trip_id
        tests:
          - unique
          - not_null
      - name: payment_type
        tests:
          - accepted_values:
              values: ['card', 'cash', 'no_charge', 'dispute']
      - name: pickup_zone_id
        tests:
          - relationships:
              to: ref('dim_zones')
              field: zone_id

Q: Is this from a course?

A: "Yes, the starting point was a course project that
 loaded one month of taxi data into a single table.

 After that I made three changes myself. I parameterised
 the load by date so I could backfill any period. I split
 the table into fct_trips and dim_zones with dbt. And I
 added tests, because I found duplicate trips when I
 reran a day.

 The relationships test also caught trips with pickup
 zones missing from the zone file, which I now map to
 an 'unknown' zone."

LIST ALL PROJECTS, THEN KEEP TWO OR THREE

Project | Skill it proves | Runs from README? | Tests?
________|_________________|___________________|_______
________|_________________|___________________|_______
________|_________________|___________________|_______

Keep projects that prove DIFFERENT skills.
Drop any that do not run from a fresh clone.
Order: strongest first.

MY TOUR (fill in, then say aloud)

Opening line: "I have __ projects, starting with ____."

Project 1 in two sentences: __________________________
What it proves: _____________________________________
How to run it: ______________________________________

Project 2 in two sentences: __________________________
What it proves: _____________________________________

Closing question: "Which one would you like to explore?"

THE DAY BEFORE

[ ] Fresh clone runs using only the README
[ ] No passwords, tokens or keys in the repo history
[ ] README has 'built vs reused' section
[ ] README has 'what I would improve' section
[ ] Pinned repos on profile match the tour order
[ ] I can explain every file in the main project
[ ] Tour timed at three minutes or less