Git & GitHub vs Traditional Tools

Traditional tools — common pain points:

□ No version history
  → who changed what? when? why?

□ Overwrites and "final_final_v2" files
  → data gets corrupted, no rollback

□ No branching
  → can't test changes without risking prod

□ No review process
  → bugs slip into production

□ No automated testing
  → pipeline breaks in production

□ No audit trail
  → can't prove what was run, when

□ Silo knowledge
  → only one person knows how it works

Real scenario — bad:

Team lead emails "final_pipeline.py" to 3
engineers.

- Engineer A edits locally and re-emails
  "final_pipeline_v2.py".
- Engineer B edits the original and uploads
  to a shared drive.
- Engineer C deploys a version from
  last week's email.

Result:
- Three different "final" versions exist.
- No one knows which one is in production.
- A bug fix yesterday is lost because it
  was overwritten by A's older copy.
- No way to roll back when prod breaks.

# Git — essential commands for data engineers

# Clone once
git clone https://github.com/team/pipelines.git

# Every task
git checkout -b feature/new-dbt-model

# Edit files, then
git add models/new_model.sql
git commit -m "Add daily revenue model"

# Push and open a PR
git push origin feature/new-dbt-model

# Update main after PR merge
git checkout main
git pull

# Revert a bad commit
git revert <commit-hash>

Typical data engineering workflow:

1. Pull latest main
   git checkout main && git pull

2. Create feature branch
   git checkout -b feature/daily-revenue

3. Add / edit:
   - SQL models
   - dbt YAML
   - Python scripts
   - Airflow DAGs

4. Commit small, meaningful changes
   git add ...
   git commit -m "Add daily revenue model"

5. Push and open a Pull Request

6. Team reviews, CI runs tests, merge

7. Deploy pipeline from main (auto)

Every step is tracked. Nothing is lost.

# .github/workflows/dbt-ci.yml
name: dbt CI

on:
  pull_request:
    branches: [main]

jobs:
  dbt-test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-python@v5
        with:
          python-version: '3.12'

      - run: pip install dbt-core dbt-postgres

      - run: dbt deps
      - run: dbt compile
      - run: dbt test
        env:
          DBT_PROFILES_DIR: .
          DBT_ENV_SECRET: ${{ secrets.DBT_PROD }}

# Every PR runs dbt tests automatically.
# Broken SQL never merges.

Branch protection (main):

Enable in Settings → Branches:

□ Require pull request before merging
□ Require 1-2 approvals
□ Require status checks (dbt tests, lint)
□ Require conversation resolution
□ Require linear history
□ Prevent force pushes
□ Prevent deletions

What this means in practice:
- No one can push directly to main
- Every change has a review
- Every change passes tests
- Every change is traceable

Data safety is no longer "trust-based" —
it's enforced by the platform.

dbt models — traditional vs Git

Traditional:
□ Model shared as .sql file in email
□ No review → bad join deployed
□ No CI → schema change breaks prod
□ No history → can't trace who changed
□ No rollback → must restore from
  "last good version" on someone's laptop

Git + GitHub:
□ Model lives in repo (models/revenue.sql)
□ PR triggers dbt compile + dbt test
□ Reviewers check logic + tests
□ Merge is blocked until tests pass
□ Full history: who, when, why
□ Rollback: git revert → redeploy

Airflow DAGs — traditional vs Git

Traditional:
□ DAG shared via shared drive
□ Multiple "final" versions
□ No test environment
□ Deploy = copy file to server via SCP
□ Breakages discovered in production
□ No one knows what changed

Git + GitHub:
□ DAG lives in repo (dags/daily_load.py)
□ Feature branch + PR
□ Airflow tests + DAG validation in CI
□ Deploy via GitHub Actions / Argo
□ Full audit trail
□ Rollback to any previous DAG

SQL changes — traditional vs Git

Traditional:
□ Analyst edits SQL in shared spreadsheet
□ Emails "updated_query.sql" to engineer
□ Engineer runs it in prod manually
□ No way to know which version is prod

Git + GitHub:
□ SQL lives in repo (queries/*.sql)
□ PR shows diff, reviewers comment
□ Automated tests run on PR
□ Merge = new version in main
□ Deploy pipeline uses main
□ Every change traceable + revertible

Git adoption plan (for a data team):

Week 1 — Setup
□ Create GitHub org + repo
□ Add team members
□ Add .gitignore (.env, target/, dbt_packages/)
□ Add README with contribution guide

Week 2 — Migration
□ Move pipeline code into repo
□ Move SQL / dbt models
□ Add dbt_project.yml and profiles
□ Commit existing state as "initial"

Week 3 — Process
□ Enable branch protection on main
□ Require PRs + 1 approval
□ Add GitHub Actions for dbt test

Week 4 — Onboarding
□ Train team on Git basics
□ Practice PR review
□ Document workflow in wiki

Then: keep every change in Git. No exceptions.

Git hygiene for data engineers:

□ Never commit secrets
  → use .env + GitHub Secrets

□ Small, meaningful commits
  → "Add daily revenue model" not "update"

□ One feature per branch
  → easier review, easier revert

□ Pull before push
  → avoid conflicts

□ Rebase feature branches on main
  → clean linear history

□ Delete merged branches
  → keep repo clean

□ Use PR descriptions
  → explain what + why, not just "fix"

□ Tag releases
  → data pipelines benefit from versions

Learning tracker:

Week: _____

Topics learned:
□ git clone / status / add / commit
□ branches (checkout -b, merge)
□ remotes (push / pull)
□ pull requests
□ conflicts + resolution
□ revert / reset basics
□ GitHub Actions basics

Practice done:
□ Committed daily
□ Opened at least 1 PR
□ Reviewed someone's PR
□ Fixed a merge conflict

Focus for next week:
- 
-