Data Engineer Career Opportunities in Telecom Industry

# telecom_cdr_pipeline.py — batch CDR processing
# (pySpark, running on EMR / Databricks / Glue)

from pyspark.sql import SparkSession
from pyspark.sql import functions as F

spark = (
    SparkSession.builder
    .appName("cdr-daily-aggregation")
    .getOrCreate()
)

# 1. Read raw CDRs for the day (usually Parquet on S3)
raw = spark.read.parquet("s3://telco-raw/cdr/dt=2026-11-12/")

# 2. Clean and enrich
clean = (
    raw
    .filter(F.col("duration_sec") >= 0)
    .filter(F.col("msisdn").isNotNull())
    .withColumn("event_hour",
                F.date_trunc("hour", F.col("event_ts")))
    .withColumn("is_international",
                F.col("roaming_flag") == 1)
)

# 3. Aggregate per subscriber per hour
agg = (
    clean
    .groupBy("msisdn", "event_hour", "is_international")
    .agg(
        F.count("*").alias("calls"),
        F.sum("duration_sec").alias("total_duration_sec"),
        F.sum("data_mb").alias("total_data_mb"),
    )
)

# 4. Write partitioned — downstream reads are cheap
(
    agg.write
    .mode("overwrite")
    .partitionBy("event_hour")
    .parquet("s3://telco-curated/usage_hourly/")
)

# Reality check:
# - One day is often 100GB-1TB compressed
# - Skewed partitions (a few heavy subscribers) will
#   kill performance if you don't salt keys
# - Reruns must be idempotent — overwrite, not append
# - Every step needs a data quality check
#   (row counts, null rates, referential checks)

# telecom_stream.py — streaming network events
# (PySpark Structured Streaming on Kafka)

from pyspark.sql import SparkSession
from pyspark.sql import functions as F
from pyspark.sql.types import (
    StructType, StructField, StringType, DoubleType, TimestampType
)

spark = SparkSession.builder.appName("network-alerts").getOrCreate()

schema = StructType([
    StructField("cell_id", StringType()),
    StructField("event_ts", TimestampType()),
    StructField("latency_ms", DoubleType()),
    StructField("error_rate", DoubleType()),
    StructField("region", StringType()),
])

stream = (
    spark.readStream
    .format("kafka")
    .option("kafka.bootstrap.servers", "broker:9092")
    .option("subscribe", "network.kpi")
    .load()
    .select(F.from_json(F.col("value").cast("string"), schema).alias("e"))
    .select("e.*")
)

# 5-minute windows per cell — rolling error rate
alerts = (
    stream
    .withWatermark("event_ts", "10 minutes")
    .groupBy(
        F.window("event_ts", "5 minutes"),
        "cell_id",
        "region",
    )
    .agg(
        F.avg("latency_ms").alias("avg_latency"),
        F.avg("error_rate").alias("avg_error"),
        F.count("*").alias("events"),
    )
    .filter((F.col("avg_error") > 0.05) |
            (F.col("avg_latency") > 200))
)

query = (
    alerts.writeStream
    .format("parquet")
    .option("checkpointLocation", "s3://telco-checkpoints/alerts")
    .option("path", "s3://telco-curated/alerts/")
    .trigger(processingTime="1 minute")
    .start()
)

query.awaitTermination()

# Reality check:
# - Exactly-once needs idempotent sinks
#   (overwrite by key + event_id dedup)
# - Late data happens. Watermarks are mandatory
# - Peak volume can be 10x baseline (match days,
#   festivals, network incidents)
# - Alerts must be quiet during known maintenance
#   or the on-call team will mute them

# dq_checks.py — telecom data quality, done properly
# Every CDR pipeline needs these checks
# before the curated data is trusted

import pandas as pd

def validate_cdr(df: pd.DataFrame) -> dict:
    report = {}
    n = len(df)
    report["row_count"] = n

    # 1. Referential integrity
    # Every MSISDN should exist in the subscriber table
    # (checked separately by joining; this is a placeholder)
    report["null_msisdn_pct"] = df["msisdn"].isna().mean()

    # 2. Range checks
    report["negative_duration_pct"] = (df["duration_sec"] < 0).mean()
    report["zero_data_pct"] = (df["data_mb"] == 0).mean()

    # 3. Distribution drift
    # Sudden shift in avg duration is a signal
    # that something changed upstream
    report["avg_duration_sec"] = df["duration_sec"].mean()
    report["p99_duration_sec"] = df["duration_sec"].quantile(0.99)

    # 4. Freshness
    report["max_event_ts"] = df["event_ts"].max()
    report["min_event_ts"] = df["event_ts"].min()

    # 5. Cardinality
    # The same MSISDN shouldn't appear more than N times
    # unless it's genuinely a heavy user
    counts = df["msisdn"].value_counts()
    report["top_msisdn_calls"] = int(counts.iloc[0]) if len(counts) else 0
    report["distinct_msisdn"] = int(df["msisdn"].nunique())

    return report

# What good looks like:
# - run on every batch, before downstream
# - fail the job if a threshold is breached
# - alert on drift, don't wait for the dashboard
#   to break three days later
# - store the report — you'll want the history

# Data engineers who own DQ get promoted faster
# than those who only write jobs.

Telecom vocabulary worth learning:

IDENTIFIERS
- MSISDN — the phone number you actually dial
- IMSI — the SIM card's internal ID
- IMEI — the phone's hardware ID
- Cell ID / eNodeB / gNodeB — the tower identifier

DATA TYPES
- CDR — Call Detail Record (per call)
- IPDR — IP Detail Record (per data session)
- TAP / RAP — roaming records exchange format
- KPI — network performance metrics (latency,
  drop rate, throughput)

SYSTEMS
- BSS — Business Support Systems (billing, CRM)
- OSS — Operations Support Systems (network ops)
- PCRF / PCF — policy and charging (5G)
- HLR / HSS — subscriber database
- SMSC — SMS centre

NETWORK
- RAN — Radio Access Network (towers)
- Core — the routing + services layer
- 5G SA / NSA — standalone vs non-standalone

BUSINESS
- ARPU — average revenue per user
- Churn — subscriber loss
- Data usage / voice usage
- Roaming — international usage

You don't need to memorise all of this.
But knowing 8-10 of these terms lets you
have a real conversation in a telecom
interview, and that changes how you're
perceived in the first 10 minutes.

The best data engineers in telecom usually
come in with generalist DE skills and pick
up the domain on the job.

How to choose a path at year 3-4:

1. "What part of the work do I actually enjoy?"
   - Building pipelines that never break → Platform
   - Making analysts' lives easier → Analytics
   - Sub-second latency problems → Streaming
   - Model serving + feature stores → MLOps
   - Hiring and strategy → Management

2. "What ceiling do I need in 5 years?"
   - ₹30-40 LPA: any path
   - ₹45-60 LPA: any path with depth
   - ₹65 LPA+: platform, streaming, ML infra,
     or management

3. "What am I willing to trade?"
   - Platform: high depth, on-call, invisible work
   - Analytics: stakeholder-facing, less on-call
   - Streaming: high pressure, high skill premium
   - MLOps: hybrid role, constant learning
   - Management: less hands-on, more politics

The honest advice:
Don't choose by title. Choose by what
you'll do on a Tuesday morning in year 5.

The single most valuable skill across all
paths: ownership of something end-to-end.
A pipeline you built, monitor, and improve
is worth more than five jobs you only
wrote once.

What each telecom DE level requires:

JUNIOR (0-2 yrs)
- Writes Spark jobs that work
- Understands partitioning, file formats
- Runs the jobs, fixes the failures
- Learns the domain vocabulary

MID (2-4 yrs)
- Owns one or two pipelines end-to-end
- Handles scale and skew
- Owns data quality for those tables
- Mentors juniors on tooling
- Designs the schema with the business

SENIOR (4-6 yrs)
- Owns a data domain (billing, network,
  fraud, etc.)
- Designs cross-pipeline contracts
- Defines data SLAs with consumers
- Handles incidents at 2 AM
- Reviews other engineers' designs

STAFF / LEAD (6+ yrs)
- Sets the platform strategy
- Owns the data architecture across teams
- Negotiates with vendors and cloud
- Hires and grows other engineers
- Represents data engineering to leadership

The jump from mid to senior is a jump in
ownership, not tooling. Tools barely change.
The decisions do.

The jump from senior to staff is a jump in
scope — from a pipeline to a domain.

If your work is invisible to leadership
after year 5, you'll plateau. This is why
data quality ownership and platform thinking
matter more than "I know Spark".

Three realistic entry routes:

ROUTE 1 — GENERALIST DE AT A TELCO/IT SERVICES (most common)
  Start: DE at an Indian operator or IT services
         company (TCS, Infosys, Tech Mahindra,
         Wipro, Accenture, Capgemini)
  Time: 6-12 months to a strong mid-level role
  Pay: ₹5-12 LPA
  Best for: 1-2 years of generalist DE work

ROUTE 2 — ANALYST → DATA ENGINEER
  Start: Data analyst who wants to build the
         pipelines instead of only using them
  Time: 9-12 months to switch internally
  Pay: ₹7-14 LPA
  Best for: Analysts with SQL and Python skills

ROUTE 3 — SOFTWARE DEVELOPER → DATA ENGINEER
  Start: Backend dev who moves to a data platform
         team inside the same company
  Time: 6-9 months with Spark and cloud
  Pay: ₹8-16 LPA
  Best for: Devs who like infrastructure and data

What's hard from a pure DS background:
✗ Streaming at scale — few DS projects touch it
✗ Platform design — DS rarely owns the infra
✗ On-call ownership — different from DS

What's accessible:
✓ Analytics DE (dbt, warehouse modelling)
✓ Batch pipelines (Spark, S3, Airflow)
✓ DQ + observability focus

Honest advice:
Get 1-2 years of generalist DE work first.
Telecom specifics (CDR, BSS, network) are
easier to learn on the job than the platform
fundamentals are.

Salary bands in India (2026), telecom DE:

JUNIOR (0-2 yrs)         ₹5-10 LPA
MID (2-4 yrs)            ₹10-18 LPA
SENIOR (4-6 yrs)         ₹18-28 LPA
STAFF/LEAD (6-9 yrs)     ₹28-45 LPA
PRINCIPAL (9+ yrs)       ₹45-70 LPA

FAANG-adjacent / product companies:
Typically 20-40% higher than services.

Telecom-specific uplift:
Telecom data teams often pay slightly more
than other industries at the same level,
because the scale requirements are high
and Spark/Kafka skills are scarce.

What the first year looks like:
- 50% building pipelines
- 20% fixing failures at 2 AM
- 15% data quality and monitoring
- 10% stakeholder conversations
- 5% learning the domain

The biggest mistake beginners make:
Trying to break into telecom DE by memorising
telecom terms. The portfolio matters more.
A Spark pipeline with real scale reasoning
beats any amount of vocabulary.

What nobody tells you:
- Some of the most interesting work is in
  streaming, not batch.
- "AI in telecom" is largely a data platform
  problem, not a modelling problem.
- Careers grow fastest when you combine
  strong platform skills with one deep domain.

Project 1 — synthetic CDR pipeline

BUSINESS QUESTION
"Given a day of call records, produce hourly
usage aggregations per subscriber, and flag
anomalous subscribers."

DATA
Synthetic CDR data (generated with Faker or
a public telecom dataset on Kaggle).

METHOD
1. Generate or load 50M+ synthetic CDRs
   (msisdn, event_ts, duration_sec, data_mb,
   roaming_flag)
2. Write a PySpark job to:
   - Clean invalid rows
   - Aggregate per subscriber per hour
   - Flag top 0.1% by call count as anomalous
3. Partition output by event_hour
4. Write DQ checks: nulls, negatives, row counts
5. Orchestrate with Airflow (or GitHub Actions)
6. Deploy on local Spark or Databricks community

DELIVERABLE
- GitHub repo with the Spark job
- DQ report sample
- README explaining scale decisions
  (partitioning, skew handling, idempotency)
- A note on cost (how much this would cost
  on AWS)

INTERVIEW VALUE
This is the closest project to what telecom
teams actually do. It shows you can handle
volume and think about failure modes.

Project 2 — streaming network alerts

BUSINESS QUESTION
"Detect cells whose error rate spikes above
threshold for 5 minutes, so the network team
can investigate before users complain."

DATA
Synthetic Kafka stream producing per-cell
events every second (cell_id, latency_ms,
error_rate, region, event_ts). Or use a public
IoT dataset as a stand-in.

METHOD
1. Produce a stream of events
   (Python script or Kafka producer)
2. Consume with PySpark Structured Streaming:
   - Watermark on event_ts
   - 5-minute tumbling window per cell
   - Aggregate avg_latency, avg_error, count
   - Filter on thresholds
3. Write alerts to a sink (Parquet or a table)
4. Add exactly-once by writing with
   checkpointLocation + idempotent sink
5. Run the pipeline end-to-end on local Kafka

DELIVERABLE
- Dockerfile + compose for Kafka + Spark
- Streaming job
- A sample of alerts
- README on exactly-once semantics and
  watermark reasoning

INTERVIEW VALUE
Streaming is where most DE candidates
are weakest. A working streaming job
with watermarks and exactly-once is rare
in a portfolio.

This single project will differentiate you
in interviews far more than another batch
ETL.

Project 3 — network KPI dashboard

BUSINESS QUESTION
"Build a dashboard that shows per-region
network quality (latency, error rate,
throughput) for the last 30 days."

DATA
Public network performance data (Ookla,
Kaggle telecom datasets) or synthetic.

METHOD
1. Ingest into a warehouse (BigQuery, DuckDB,
   or Postgres for local)
2. Model as:
   - fact_kpi_daily (region, cell, date, metrics)
   - dim_region (region metadata)
3. Build dbt models on top:
   - stg_kpi
   - agg_region_daily
   - fct_region_health
4. Schedule with Airflow
5. Build a dashboard (Metabase, Superset,
   Streamlit, or just a static HTML page
   using the aggregated data)

DELIVERABLE
- GitHub repo with dbt models + orchestration
- Deployed dashboard with a live URL
- README explaining the grain and dimensions
- Screenshot of the dashboard in the README

INTERVIEW VALUE
Shows you can model properly, not just
write Spark jobs. Analytics DE roles hire
on exactly this.

If you build one of these three, you'll be
ahead of 80% of telecom DE candidates
who only list tools.