How Bank Employees Can Start a Career in Data Engineering

What a banking background actually gives you:

✓ DOMAIN
  You know how loans, credit, KYC, risk,
  reconciliation, IFRS, RBI reporting work.
  Most data engineers who join a bank spend
  6-12 months learning this. You already have it.

✓ SQL
  If you've written real banking queries,
  you know joins, aggregation, subqueries,
  and window functions (for statement
  generation, ageing, ranking).
  This is 60% of DE interviews.

✓ PROCESS DISCIPLINE
  You've worked with SLAs, audit trails,
  reconciliation checks, and sign-offs.
  Data quality and monitoring — the things
  that make a data engineer senior — feel
  natural to you.

✓ REGULATORY AWARENESS
  PII handling, retention, audit logs.
  Banks care deeply about these, and most
  DE candidates don't.

✓ STAKEHOLDER COMMUNICATION
  You can translate a business request
  into a spec. DEs who can do this get
  promoted faster than those who can't.

What this means:
You are NOT starting from behind.
You're starting from a different place.
The technical gap is real, but so is the
domain and process advantage.

Financial services DE teams actively
prefer candidates with banking experience.
They hire on domain + SQL + basic pipelines,
not on advanced Spark or deep CS theory.

What doesn't transfer (and you need to build):

✗ Python beyond Excel formulas
  (Pandas, functions, files, error handling)

✗ Data pipelines
  (Spark, Airflow, dbt, or any orchestration)

✗ Cloud data tools
  (S3, Glue, BigQuery, Redshift, IAM)

✗ Data modelling at scale
  (star schemas, slow-changing dimensions)

✗ Version control
  (Git, GitHub, PR workflow)

✗ Basic engineering practices
  (testing, logging, idempotency)

✗ Software fundamentals
  (how HTTP works, what a container is)

What you DON'T need:
✗ A CS degree
✗ Deep maths
✗ Understanding compilers
✗ Machine learning
✗ Kafka / streaming (until year 2)
✗ Kubernetes

What you DO need:
✓ 9 months of consistent work
✓ One real pipeline project
✓ A portfolio with the banking domain
  built into it
✓ The ability to explain what you built
  and why

Timeline is real: 9 months if you commit.
Faster if you're already technical.
Slower if you've never written Python.

The predictor of success in a banking
restart: consistency, not free time.

-- banking_queries.sql — SQL questions you should be able to solve

-- 1. Daily loan disbursement trend (last 30 days)
SELECT
  DATE(disbursement_date) AS d,
  SUM(amount) AS total_disbursed,
  COUNT(*) AS loans_count
FROM loans
WHERE disbursement_date >= CURRENT_DATE - INTERVAL '30 days'
GROUP BY d
ORDER BY d;

-- 2. Top 10 branches by loan amount (last quarter)
SELECT
  branch_id,
  SUM(amount) AS total,
  COUNT(*) AS loans
FROM loans
WHERE disbursement_date >= DATE_TRUNC('quarter', CURRENT_DATE) - INTERVAL '1 quarter'
  AND disbursement_date < DATE_TRUNC('quarter', CURRENT_DATE)
GROUP BY branch_id
ORDER BY total DESC
LIMIT 10;

-- 3. Accounts overdue by bucket (this is real banking work)
SELECT
  CASE
    WHEN days_past_due = 0 THEN 'current'
    WHEN days_past_due BETWEEN 1 AND 30 THEN '1-30'
    WHEN days_past_due BETWEEN 31 AND 60 THEN '31-60'
    WHEN days_past_due BETWEEN 61 AND 90 THEN '61-90'
    ELSE '90+'
  END AS bucket,
  COUNT(*) AS accounts,
  SUM(outstanding) AS exposure
FROM loan_accounts
WHERE status = 'active'
GROUP BY bucket
ORDER BY bucket;

-- 4. Running balance per customer (window function)
SELECT
  customer_id,
  txn_date,
  amount,
  SUM(amount) OVER (
    PARTITION BY customer_id
    ORDER BY txn_date
    ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW
  ) AS running_balance
FROM transactions
WHERE customer_id = 12345
ORDER BY txn_date;

-- 5. Rank loans by size within each branch
SELECT
  branch_id,
  loan_id,
  amount,
  RANK() OVER (PARTITION BY branch_id ORDER BY amount DESC) AS rnk
FROM loans
WHERE status = 'active';

-- Why these matter:
-- All five are interview questions at banks.
-- The window function is the one that
-- separates candidates who "know SQL" from
-- those who can actually solve real problems.

# banking_analysis.py — real Pandas on banking data
# (fake data — no real PII)

import pandas as pd
from pathlib import Path

# 1. Load — usual banking file is CSV or Excel
loans = pd.read_csv("data/loans.csv", parse_dates=["disbursement_date"])

# 2. Data cleaning (banks care about this)
loans["amount"] = pd.to_numeric(loans["amount"], errors="coerce")
loans = loans.dropna(subset=["amount", "customer_id"])
loans = loans[loans["amount"] > 0]

# 3. Aggregation by branch and month
loans["month"] = loans["disbursement_date"].dt.to_period("M").astype(str)
monthly = (
    loans.groupby(["month", "branch_id"])
         .agg(
             total=("amount", "sum"),
             count=("loan_id", "count"),
             avg=("amount", "mean"),
         )
         .reset_index()
)

# 4. Rank branches within each month
monthly["rank"] = monthly.groupby("month")["total"].rank(
    ascending=False, method="dense"
)
top_branches = monthly[monthly["rank"] <= 3]

# 5. Save for the reporting team
out = Path("output")
out.mkdir(exist_ok=True)
top_branches.to_csv(out / "top_branches.csv", index=False)
monthly.to_csv(out / "monthly_summary.csv", index=False)

print(f"Processed {len(loans):,} loans")
print(f"Generated {len(monthly)} monthly rows")

# What this shows interviewers:
# ✓ Pandas fluency
# ✓ Data cleaning (dropna, error handling)
# ✓ Grouping and aggregation
# ✓ File output for downstream use

# This is exactly what an analytics engineer
# does daily — and it's the closest banking
# task to a data engineering task.

-- models/staging/stg_loans.sql
-- first layer: clean and standardise raw banking data

with source as (
    select * from {{ source('raw', 'loans') }}
),

cleaned as (
    select
        cast(loan_id as varchar) as loan_id,
        cast(customer_id as varchar) as customer_id,
        cast(branch_id as varchar) as branch_id,
        cast(amount as numeric(18, 2)) as amount,
        cast(disbursement_date as date) as disbursement_date,
        lower(trim(status)) as status,
        case
            when status in ('active', 'closed', 'npa') then status
            else 'unknown'
        end as status_clean,
        _loaded_at
    from source
    where amount > 0
      and loan_id is not null
)

select * from cleaned

-- Why this layer matters:
-- Raw banking data is often dirty: mixed types,
-- inconsistent casing, null keys, negative amounts.
-- Staging is where you make it usable for
-- everything downstream.

# spark_loans.py — processing large banking data
# (used for nightly aggregations across all branches)

from pyspark.sql import SparkSession
from pyspark.sql import functions as F

spark = (
    SparkSession.builder
    .appName("loans-monthly")
    .getOrCreate()
)

# Read the raw loan data (Parquet in S3 or HDFS)
loans = spark.read.parquet("s3://bank-raw/loans/")

# Clean and enrich
cleaned = (
    loans
    .filter(F.col("amount") > 0)
    .filter(F.col("loan_id").isNotNull())
    .withColumn("month", F.date_trunc("month", "disbursement_date"))
)

# Aggregate per branch per month
monthly = (
    cleaned
    .groupBy("month", "branch_id")
    .agg(
        F.sum("amount").alias("total_disbursed"),
        F.count("loan_id").alias("loans_count"),
        F.avg("amount").alias("avg_loan_size"),
    )
    .orderBy("month", "branch_id")
)

# Write partitioned for downstream reads
(
    monthly.write
    .mode("overwrite")
    .partitionBy("month")
    .parquet("s3://bank-curated/loans_monthly/")
)

print(f"Processed: {monthly.count()} monthly aggregates")
spark.stop()

# What this teaches (and what interviewers ask):
# ✓ Spark transformations vs actions
# ✓ Partitioning for query performance
# ✓ Idempotency (mode overwrite, not append)
# ✓ Reading/writing Parquet
# ✓ Filtering before aggregation (cheap vs expensive)

# The 2 rules most candidates miss:
# 1. Always overwrite by partition, not append
#    (reruns must be safe)
# 2. Filter early — don't let Spark process
#    rows that will be thrown away

What cloud skills a banking DE actually needs:

AWS (or GCP — both are fine)
  ✓ S3 — object storage for raw and curated data
  ✓ IAM — roles and permissions (very important in banking)
  ✓ Athena or Redshift — query the data
  ✓ Glue — the managed Spark service
  ✓ CloudWatch — basic logging

  ✗ Lambda (useful but not needed for entry)
  ✗ Kinesis / streaming (year 2 topic)
  ✗ EKS / Kubernetes (year 2 topic)
  ✗ Deep infrastructure-as-code (year 2 topic)

Azure (if the bank is Azure-heavy — most Indian banks are)
  ✓ ADLS Gen2 — equivalent of S3
  ✓ Azure Data Factory — pipelines
  ✓ Azure SQL / Synapse — warehouse
  ✓ Key Vault — secrets
  ✓ Monitor — logging

GCP (if the bank is GCP-heavy — some fintechs)
  ✓ Cloud Storage
  ✓ BigQuery
  ✓ Dataflow / Dataproc
  ✓ Cloud IAM

What to actually learn:
- Pick ONE cloud
- Learn S3 + IAM + one warehouse + one Spark service
- Build one project that uses them
- The other clouds transfer in ~2 weeks once
  you know one

Reality: your first job will teach you
the specific services. What matters is
that you can talk about the concepts —
storage, compute, IAM, cost — with the
right vocabulary.

What a banking DE portfolio needs:

1. A DATA MODELLING PROJECT
   - Star schema for a banking domain
     (loans, cards, transactions, risk)
   - A dimension table with slowly
     changing dimensions
   - A fact table with the right grain
   - Documented in a README with an
     ER diagram

2. A PIPELINE PROJECT
   - Raw → staging → curated
   - dbt or Airflow (or both)
   - Scheduled or triggered
   - Idempotent (reruns are safe)
   - Data quality checks included
   - Error handling for bad rows

3. A SPARK PROCESSING PROJECT
   - Read a large file (public data)
   - Clean, aggregate, write partitioned
   - Explain the partitioning choice
   - Show a rerun is safe

4. A CLOUD DEPLOYMENT
   - At least one project deployed on
     a real cloud (AWS, Azure, GCP)
   - Live URL if possible (or clearly
     documented in the README)

5. A DOMAIN STORY
   - Every project's README explains
     the BUSINESS context
   - "This models loan disbursement
     for a bank with 3 branches"
   - Not "this is a Spark pipeline"

What NOT to include:
✗ Another Titanic or Iris dataset
✗ A Spark job with no clear output
✗ Notebooks (use scripts + README)
✗ Code without a "what I'd change" section
✗ Certificates (they're not portfolio)

The single strongest project:
A loan-servicing data model + dbt
pipeline + Spark aggregation, deployed
on cloud, with a README explaining
the business context.

That project is what gets you a job at
a bank's data engineering team — because
it looks exactly like what they build.

Where to apply as a banker-turned-DE:

TIER 1 (highest fit)
  - Data engineer roles at banks:
    HDFC, ICICI, Axis, Kotak, SBI
  - Data engineer roles at NBFCs:
    Bajaj Finserv, Tata Capital
  - Data engineer roles at BFSI IT services:
    TCS, Infosys, Wipro, Capgemini
  - Data platform roles at fintechs:
    Razorpay, Paytm, PhonePe

TIER 2 (good fit)
  - Analytics engineer roles at BFSI
  - Data engineer at payment companies
  - DE at banks' captive centres (GCCs)

TIER 3 (worth trying)
  - Generic DE roles at product companies
  - DE at consulting firms with BFSI clients

What to say in the interview:
"I've worked in banking for X years and
 I understand the domain — loans, risk,
 reconciliation, reporting. Over the past
 9 months I've built pipelines that model
 these in SQL, dbt, and Spark. Here's my
 portfolio."

That framing works because:
✓ It's honest
✓ It uses your real advantage
✓ It shows a portfolio, not just a certificate
✓ It gives them a reason to prefer you

What NOT to say:
✗ "I'm new to data engineering"
✗ "I'm still learning"
✗ "I was in banking before this"
  (framing yourself as "before", not "with")

Realistic salary:
  Bank DE (entry):   ₹6-10 LPA
  BFSI IT services:  ₹5-9 LPA
  Fintech DE:        ₹8-14 LPA
  (Ranges move 20-40% higher after year 3)

9-month planner — banking to data engineering:

MONTHS 1-3 (SQL + Python)
☐ 30 banking SQL queries solved
☐ Pandas analysis on a real dataset
☐ Python script fetching and saving data
☐ GitHub repo with both projects

MONTHS 4-5 (Modelling + dbt)
☐ Star schema for a banking domain
☐ dbt project with staging + marts
☐ Data quality tests in dbt
☐ Airflow DAG to run it

MONTH 6 (Spark)
☐ PySpark job processing 1M+ rows
☐ Partitioned output
☐ README explaining the choices

MONTHS 7-8 (Cloud + portfolio)
☐ Deploy one project on AWS or Azure
☐ Portfolio site or README index
☐ All projects have clean READMEs

MONTH 9 (Apply)
☐ Resume rewritten
☐ 50+ applications
☐ 10 mock interviews

End of month 9 must-haves:
☐ 3 portfolio projects
☐ 1 deployed on cloud
☐ 50+ applications sent
☐ 3+ interview responses

Weekly log:

Week _____

Hours spent:
  SQL:              ___
  Python / Pandas:  ___
  dbt / Airflow:    ___
  Spark:            ___
  Reading / docs:   ___

What I built this week:
_______________________________________________

What I shipped (committed or deployed):
_______________________________________________

What confused me:
_______________________________________________

Next week's single priority:
_______________________________________________

Rule: if nothing was "shipped", the week was
reading, not learning. A dbt model counts.
A SQL query counts. An article doesn't.

The banking DE path is 80% consistency,
20% cleverness. Keep showing up.

Skill matrix (rate 1-5 monthly):

SQL
Basic SELECT / WHERE           [ ]
Joins (inner, left, full)      [ ]
Aggregation + HAVING           [ ]
Subqueries and CTEs            [ ]
Window functions               [ ]

PYTHON
Pandas basics                  [ ]
Pandas cleaning + aggregation  [ ]
Error handling                 [ ]
Reading APIs / files           [ ]
Basic testing                  [ ]

MODELLING
Star schema                    [ ]
Fact vs dimension              [ ]
Slowly changing dimensions     [ ]
Grain decisions                [ ]

PIPELINES
dbt models + tests             [ ]
Airflow DAGs                   [ ]
Idempotency                    [ ]

BIG DATA
PySpark transformations        [ ]
Partitioning                   [ ]
File formats (Parquet)         [ ]

CLOUD
S3 / storage                   [ ]
IAM / security                 [ ]
Warehouse (Athena/Redshift)    [ ]
Managed Spark (Glue)           [ ]

DOMAIN (bonus)
Banking domain knowledge       [ ]

Target after 9 months: 3+ in every row.
Anything at 1-2 = next month's focus.
Anything at 4-5 = your differentiator.

The single highest-leverage skill:
data modelling. It's the difference
between an analyst who writes SQL and
a data engineer. Banks hire for it.