How Docker Helps You Land a Data Engineer Job Faster

#!/usr/bin/env bash
# Start a practice PostgreSQL database with persistent storage
docker volume create pgdata
docker run -d --name practice-pg \
  -e POSTGRES_PASSWORD=practice \
  -e POSTGRES_DB=analytics \
  -p 5432:5432 \
  -v pgdata:/var/lib/postgresql/data \
  postgres:16

# Open a SQL shell inside the container
docker exec -it practice-pg psql -U postgres -d analytics -c "SELECT version();"

docker ps                      # running containers
docker ps -a                   # all containers
docker logs practice-pg        # container logs
docker exec -it practice-pg bash   # shell inside the container
docker stop practice-pg        # stop
docker rm practice-pg          # remove container (volume stays)
docker images                  # list images
docker system df               # disk usage

services:
  db:
    image: postgres:16
    environment:
      POSTGRES_PASSWORD: practice
      POSTGRES_DB: analytics
    ports: ["5432:5432"]
    volumes: ["pgdata:/var/lib/postgresql/data"]
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U postgres"]
      interval: 5s
      retries: 10

  etl:
    build: ./etl
    environment:
      DATABASE_URL: postgresql://postgres:practice@db:5432/analytics
    depends_on:
      db:
        condition: service_healthy

volumes:
  pgdata:

#!/usr/bin/env bash
set -euo pipefail
docker compose up -d db          # start the database
docker compose run --rm etl      # run the pipeline once
docker compose exec db psql -U postgres -d analytics \
  -c "SELECT COUNT(*) FROM sales_daily;"
docker compose down              # stop everything (data volume kept)

FROM python:3.12-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY pipeline.py .
RUN useradd --create-home appuser
USER appuser
CMD ["python", "pipeline.py"]

# requirements.txt
# pandas==2.2.3
# SQLAlchemy==2.0.36
# psycopg2-binary==2.9.10

import os
import pandas as pd
from sqlalchemy import create_engine

engine = create_engine(os.environ["DATABASE_URL"])

raw = pd.DataFrame({
    "order_date": ["2026-11-04", "2026-11-04", "2026-11-05"],
    "city": [" Noida", "Delhi ", "Noida"],
    "amount": [500, 750, 300],
})
raw["city"] = raw["city"].str.strip()
daily = raw.groupby(["order_date", "city"], as_index=False)["amount"].sum()

daily.to_sql("sales_daily", engine, if_exists="replace", index=False)
print(f"loaded {len(daily)} rows into sales_daily")

Project: Containerised daily sales pipeline

- docker-compose.yml with Postgres (and optional MinIO)
- etl/ image: extract a public CSV, clean, aggregate, load
- SQL checks after load (row counts, no duplicates)
- Makefile or script: up, run, check, down
- README: architecture, how to run, what each table means

# Daily Sales Pipeline (Docker)

## Run it
    docker compose up -d db
    docker compose run --rm etl

## Architecture
CSV -> etl container -> Postgres (sales_daily)

## Data checks
Row count > 0; no duplicate (order_date, city) pairs.

## Design choices
Pinned images, env-var config, non-root container.

## Clean up
    docker compose down -v

[ ] Images pinned to versions
[ ] No passwords committed (use .env, add to .gitignore)
[ ] depends_on with a health check for the database
[ ] Pipeline is idempotent (safe to rerun)
[ ] Data quality checks after load
[ ] Works from a fresh clone with one or two commands
[ ] README explains how and why