Arts Graduate to Data Engineer: A Realistic Career Timeline

-- Works in SQLite, PostgreSQL and MySQL
CREATE TABLE customers (id INTEGER PRIMARY KEY, name TEXT, city TEXT);
CREATE TABLE orders (id INTEGER PRIMARY KEY, customer_id INTEGER, amount INTEGER, order_date TEXT);

INSERT INTO customers VALUES (1, 'Asha', 'Noida'), (2, 'Kabir', 'Delhi');
INSERT INTO orders VALUES
  (1, 1, 1200, '2026-10-01'),
  (2, 1, 800,  '2026-10-15'),
  (3, 2, 2500, '2026-10-20');

-- Total spend per customer, highest first
SELECT c.name, c.city, SUM(o.amount) AS total_spend, COUNT(*) AS orders
FROM customers c
JOIN orders o ON o.customer_id = c.id
GROUP BY c.id, c.name, c.city
ORDER BY total_spend DESC;

Stage 1 - Data basics checklist

Spreadsheets
[ ] Sort, filter, remove duplicates
[ ] VLOOKUP / XLOOKUP and pivot tables

SQL
[ ] SELECT, WHERE, ORDER BY, LIMIT
[ ] INNER and LEFT JOIN
[ ] GROUP BY with SUM, COUNT, AVG
[ ] HAVING and subqueries
[ ] CASE WHEN for simple rules

Habits
[ ] Check row counts before and after a join
[ ] Write down what each column means

# pip install pandas   (sqlite3 is built into Python)
import io
import sqlite3
import pandas as pd

raw_csv = io.StringIO("""order_id,city,amount,order_date
1, Noida ,1200,2026-10-01
2,Delhi,,2026-10-03
3,noida,800,2026-10-15
3,noida,800,2026-10-15
""")

# Extract
df = pd.read_csv(raw_csv)

# Transform
df["city"] = df["city"].str.strip().str.title()
df["amount"] = df["amount"].fillna(0).astype(int)
df["order_date"] = pd.to_datetime(df["order_date"])
df = df.drop_duplicates(subset="order_id")

# Load
conn = sqlite3.connect(":memory:")
df.to_sql("orders", conn, index=False, if_exists="replace")

# Check
print(pd.read_sql("SELECT city, SUM(amount) AS total FROM orders GROUP BY city", conn))
assert len(df) == 3, "unexpected row count"

Problems in the raw file and the fix:

" Noida " with spaces   -> str.strip()
"noida" lower case      -> str.title()  => "Noida"
blank amount            -> fillna(0)
text dates              -> pd.to_datetime()
duplicate order 3       -> drop_duplicates()

Lesson: real data is always messy.
A data engineer's job is to make it
clean, consistent and trustworthy.

# A simple way to schedule a pipeline before learning Airflow.
# Edit your schedule with:  crontab -e
# Format: minute hour day month weekday command

# Run the ETL script every day at 6:30 AM
30 6 * * * /usr/bin/python3 /home/me/pipelines/first_etl_pipeline.py >> /home/me/pipelines/etl.log 2>&1

# Check the log the next morning:
# tail -n 20 /home/me/pipelines/etl.log

Am I ready to apply for junior data roles?

[ ] Strong SQL: joins, window functions, CTEs
[ ] Python: pandas, functions, error handling
[ ] Built an ETL pipeline that re-runs safely
[ ] Loaded data into a cloud warehouse or database
[ ] Scheduled a job (cron or Airflow)
[ ] 2-3 projects on GitHub with README and diagram
[ ] Can explain my switch from arts in 60 seconds
[ ] Practised SQL interview questions regularly

Stage Tracker
Current stage: 1 Data Basics / 2 Python & ETL / 3 Cloud & Jobs

This week's focus: _______________
Hours studied: _____
SQL problems solved: _____

Built or improved:
_______________

Ready to move to next stage?  Y / N
If not, what is missing? _______________

Pipeline Project Brief

Name: _______________
Source data: CSV / API / database
Destination: database / warehouse

Transform rules:
1. _______________
2. _______________

Schedule: daily / hourly
Checks: row count, no nulls in key columns
Diagram drawn?  Y / N
README written? Y / N

Skill Matrix (rate 0-3)

SQL joins and aggregates       ___
Window functions and CTEs      ___
Python and pandas              ___
ETL design and re-runs         ___
Cloud storage basics           ___
Data warehouse basics          ___
Scheduling (cron / Airflow)    ___
Explaining my work clearly     ___