How Git and GitHub Helps You Land a Data Engineer Job Faster

# Change a SQL model safely
git switch main
git pull
git switch -c feature/add-refund-column

# edit models/marts/fct_orders.sql
git add models/marts/fct_orders.sql
git commit -m "Add refund_amount to fct_orders"
git push -u origin feature/add-refund-column

# Open a pull request on GitHub, address review comments:
git add models/marts/fct_orders.sql
git commit -m "Use COALESCE for orders without refunds"
git push

# Who changed this metric logic, and when?
git log --oneline -- models/marts/fct_orders.sql

# Line-by-line authorship of a SQL file
git blame models/marts/fct_orders.sql

# What exactly changed between two commits?
git diff a1b2c3d e4f5a6b -- models/marts/fct_orders.sql

# Find the commit that broke a test (binary search)
git bisect start
git bisect bad HEAD
git bisect good v1.2.0
# run tests at each step, mark good/bad, then:
git bisect reset

sales-pipeline/
|-- README.md
|-- requirements.txt
|-- .env.example          # variable names only, no real values
|-- .gitignore
|-- src/
|   |-- extract.py         # pull data from API / source DB
|   |-- transform.py       # clean and reshape with pandas
|   `-- load.py            # write to the warehouse
|-- sql/
|   |-- staging_orders.sql
|   `-- fct_daily_sales.sql
|-- dags/
|   `-- sales_daily.py     # Airflow DAG
|-- tests/
|   |-- data/sample_orders.csv
|   `-- test_transform.py
`-- .github/
    `-- workflows/ci.yml

# Python
__pycache__/
*.pyc
.venv/

# Secrets - never commit
.env
*.pem
credentials*.json

# Data and outputs - too large and often private
data/raw/
data/processed/
*.parquet
*.duckdb
output/

# Notebooks checkpoints and editor files
.ipynb_checkpoints/
.vscode/
.DS_Store

# Keep small test fixtures
!tests/data/*.csv

# src/transform.py
import pandas as pd


def clean_orders(df: pd.DataFrame) -> pd.DataFrame:
    out = df.drop_duplicates(subset="order_id").copy()
    out = out.dropna(subset=["amount"])
    out["order_date"] = pd.to_datetime(out["order_date"])
    return out[["order_id", "customer_id", "order_date", "amount"]]


# tests/test_transform.py
import pandas as pd
from src.transform import clean_orders


def test_clean_orders_removes_duplicates_and_nulls():
    raw = pd.DataFrame({
        "order_id": [1, 1, 2, 3],
        "customer_id": [10, 10, 11, 12],
        "order_date": ["2026-01-05"] * 4,
        "amount": [500.0, 500.0, None, 250.0],
    })
    out = clean_orders(raw)
    assert list(out["order_id"]) == [1, 3]
    assert out["amount"].sum() == 750.0

# .github/workflows/ci.yml
name: data-pipeline-ci
on:
  pull_request:
  push:
    branches: [main]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install -r requirements.txt pytest ruff sqlfluff
      - name: Lint Python
        run: ruff check src tests
      - name: Lint SQL
        run: sqlfluff lint sql --dialect postgres
      - name: Unit tests
        run: pytest -q

# Daily Sales Pipeline

![CI](https://github.com/<you>/sales-pipeline/actions/workflows/ci.yml/badge.svg)

## What it does
Extracts orders from a public API, cleans them with pandas,
loads them into PostgreSQL and builds a daily sales table.

## Architecture
API -> extract.py -> transform.py -> load.py -> fct_daily_sales
Scheduled by Airflow (dags/sales_daily.py)

## Run locally
cp .env.example .env   # add your own DB settings
pip install -r requirements.txt
python -m src.extract && python -m src.transform && python -m src.load

## Tests
pytest -q   (also runs on every pull request)

## Design decisions
- Incremental load by order_date to avoid full reloads
- Duplicates removed on order_id before loading

## What I would add next
Data quality alerts and a backfill command

## What
Add refund_amount to fct_daily_sales.

## Why
Finance needs net sales (gross minus refunds) in the daily report.

## How
- Joined staging_refunds on order_id (LEFT JOIN, keeps orders without refunds)
- COALESCE(refund_amount, 0) so totals are never NULL

## How I tested
- New test: test_net_sales_with_refunds
- Ran the model on sample data; daily totals match a manual check

## Impact
- New column only; existing columns unchanged
- Dashboards using net_sales need a refresh after merge

GITHUB PORTFOLIO CHECKLIST - data engineer

Profile
[ ] Profile README: role target, skills, links to best repos
[ ] 2-4 pinned repositories, all data engineering related

Each pinned repo
[ ] README with problem, architecture, run steps
[ ] Small sample data in tests/, no large or private files
[ ] No secrets in code or history (.env.example only)
[ ] Tests plus a GitHub Actions workflow that passes
[ ] Meaningful commit messages, not 'update' x 40
[ ] At least one merged pull request showing review flow

Coverage across repos
[ ] Python ingestion
[ ] SQL modelling
[ ] Orchestration (Airflow or similar)
[ ] One cloud or warehouse component