What Does a Data Engineer Do on a Normal Work Day?

# Recent runs for one DAG
airflow dags list-runs -d orders_elt --no-backfill | head

# Task states for a specific run
airflow tasks states-for-dag-run orders_elt scheduled__2026-11-05T00:00:00+00:00

# Clear a failed task so it reruns (after fixing the cause)
airflow tasks clear orders_elt -t load_orders \
  -s 2026-11-05 -e 2026-11-05 --only-failed --yes

-- Freshness and today's row count vs the 7-day average
WITH daily AS (
    SELECT DATE(loaded_at) AS d, COUNT(*) AS rows_loaded
    FROM fct_orders
    WHERE loaded_at >= CURRENT_DATE - 8
    GROUP BY DATE(loaded_at)
)
SELECT (SELECT MAX(loaded_at) FROM fct_orders)                 AS newest_row,
       (SELECT rows_loaded FROM daily WHERE d = CURRENT_DATE)  AS today_rows,
       (SELECT ROUND(AVG(rows_loaded)) FROM daily WHERE d < CURRENT_DATE) AS avg_7d;

git checkout -b feature/add-crm-source
# write extractor + model + tests
dbt run --select stg_crm_contacts+
dbt test --select stg_crm_contacts+
git add .
git commit -m "Add daily CRM contacts ingestion with staging model and tests"
git push -u origin feature/add-crm-source
# open a pull request describing: what, why, how tested

## What
Adds daily ingestion of CRM contacts and a staging model.

## Why
Marketing analysts need contact attributes joined to orders.

## How tested
- Ran extractor against the sandbox API
- dbt tests: unique + not_null on contact_id (pass)
- Compared row count with CRM export (match)

## Risks / follow-ups
- API rate limit: added retries with backoff

Reviewing a pipeline pull request:
[ ] Is the load idempotent (safe to rerun)?
[ ] Are keys tested for unique / not null?
[ ] Are filters applied before heavy joins?
[ ] Are secrets read from config, not hard-coded?
[ ] Is the model documented (grain, owner)?
[ ] Is there a plan for backfilling history?

# models/marts/schema.yml (excerpt)
models:
  - name: fct_orders
    description: >
      One row per order. Refreshed daily by 06:00 IST.
      Excludes test orders. Owner: data engineering.
    columns:
      - name: order_id
        description: Unique order identifier from the shop database.
      - name: amount
        description: Order value in INR after discounts, before tax.

[ ] Orchestrator: any failed or long-running tasks?
[ ] Alerts channel: anything overnight?
[ ] Freshness of top 5 tables
[ ] Row counts vs usual range
[ ] Inform users if data is late (with ETA)

Date:
Incidents handled:
Work completed:
Pull requests opened / reviewed:
Questions answered for data users:
Blockers:
Tomorrow's first task:

Simulate a data engineer's day at home:
09:30 Check your mini pipeline's last run
10:00 Break something on purpose (rename a column)
10:30 Detect it with a test, fix it, backfill
12:00 Add a new small source
15:00 Write a PR description and self-review
16:00 Document the new table