Data Engineering Terms Every Beginner Should Know

Data Engineering · Terminology · Beginner Guide

Data Engineering Terms Every Beginner Should Know

AK Ankit Kaushik · Lead Trainer, Data Engineering & AI | Updated 12 November 2026 | 16 min read Free · Career Guide

Data engineering terms cluster around five stages of the pipeline — ingest, transform, store, serve, and monitor.

Quick summary — the glossary

Data engineering has its own vocabulary — and knowing the terms is the first step to understanding the job. Here's a practical glossary of 50+ terms you'll encounter in interviews, documentation, and daily work — with plain-English definitions and real examples.

In this guide you'll learn:

The core pipeline terms (ETL, ELT, CDC, streaming).
Storage terms (data lake, warehouse, lakehouse).
Tooling terms (Spark, dbt, Airflow, Kafka).
Quality + monitoring terms.
Interview Q&A + quiz to test your recall.

SECTION 01Pipeline core terms

These are the terms you'll hear in every data engineering conversation. Learn them first:

Term	Plain-English definition	Example / Why it matters
Pipeline	A series of steps that move data from source to destination	Extract from Postgres → clean → load into Snowflake
ETL	Extract, Transform, Load — transform before loading	Traditional approach: transform in-flight, then load into warehouse
ELT	Extract, Load, Transform — load first, then transform	Modern approach: load raw data into warehouse, transform with dbt
Source	The system where data originates	PostgreSQL, MySQL, Salesforce, S3, Kafka, APIs
Destination / Sink	Where data ends up after processing	Snowflake, BigQuery, Redshift, S3, Delta Lake
Connector	A pre-built integration that reads from or writes to a system	Airbyte / Fivetran connectors for Salesforce, Stripe
CDC	Change Data Capture — stream changes as they happen	Debezium reads Postgres WAL and emits every INSERT/UPDATE to Kafka
Batch processing	Process data in large chunks, on a schedule	Run a Spark job nightly at 2 AM to process yesterday's orders
Streaming	Process data continuously, event by event	Kafka → Flink → real-time fraud detection dashboard
Idempotent	Running the same job twice produces the same result	Critical for pipelines — re-running on failure shouldn't double-count
Backfill	Process historical data to catch up or fix past periods	New model needs 2 years of history — backfill from raw data
Watermark	A marker for how far streaming data has been processed	Prevents processing late-arriving events out of order

# ETL — extract, transform, load
# (transform happens before loading into warehouse)

import pandas as pd
from sqlalchemy import create_engine

# 1. Extract — pull from source database
source = create_engine("postgresql://source_db")
orders = pd.read_sql("SELECT * FROM orders WHERE updated_at > '2026-11-01'", source)

# 2. Transform — clean and reshape BEFORE loading
orders["total"] = orders["total"].fillna(0)
orders["order_date"] = pd.to_datetime(orders["order_date"])
daily = (
    orders
    .groupby(["order_date", "product_id"])
    .agg(revenue=("total", "sum"), units=("quantity", "sum"))
    .reset_index()
)

# 3. Load — write clean data into warehouse
warehouse = create_engine("snowflake://warehouse_db")
daily.to_sql("daily_sales", warehouse, if_exists="append", index=False)

# Pros: less storage in warehouse, cleaned data at rest
# Cons: transform logic is baked into pipeline code, harder to audit
# Classic ETL — used when warehouse is expensive or strict schema

# ELT — extract, load, transform
# (load raw, then transform with SQL/dbt inside the warehouse)

# Step 1 (extraction): use Airbyte or Fivetran to load raw data
# automatically into warehouse as tables like:
#   raw_orders, raw_products, raw_customers

# Step 2 (transformation): dbt model transforms inside the warehouse

# models/staging/stg_orders.sql
{{ config(materialized='view') }}

select
    order_id,
    customer_id,
    product_id,
    order_date::date as order_date,
    coalesce(total, 0) as total,
    quantity,
    created_at,
    updated_at
from {{ source('raw', 'orders') }}
where order_date is not null

# models/marts/daily_sales.sql
{{ config(materialized='table', partition_by={'field': 'order_date', 'data_type': 'date'}) }}

select
    order_date,
    product_id,
    sum(total) as revenue,
    sum(quantity) as units
from {{ ref('stg_orders') }}
group by 1, 2

# Pros: raw data always available for audit and re-transform
# Cons: more warehouse storage, transformation cost shifted to warehouse
# Modern ELT — used by every company building with Snowflake, BigQuery, Redshift

etl-vs-elt.md

Key insight: ETL vs ELT isn't just semantics — it's an architecture choice. ETL moves transformation to the pipeline layer. ELT pushes transformation to the warehouse (dbt). Modern data teams overwhelmingly choose ELT because raw data is preserved and transforms are version-controlled in SQL.

SECTION 02Storage terms

Where data lives after extraction — and why the choice matters:

Term	Definition	Example
Data warehouse	Structured storage optimised for analytics queries	Snowflake, BigQuery, Redshift
Data lake	Raw storage of any data — structured, semi, unstructured	S3, GCS, Azure Data Lake
Lakehouse	Combination: lake flexibility + warehouse analytics	Databricks + Delta Lake, Snowflake Iceberg
Parquet	Columnar file format — compressed, fast for analytics	Standard for data lakes and warehouses
Delta Lake / Iceberg	Table formats on top of Parquet with ACID + versioning	Enable time travel, safe concurrent writes
Partition	Physical organisation of data by a column value	Partition by date so queries scan only relevant files
Columnar storage	Store data by column instead of by row	Analytics queries only read needed columns → 10–100x faster
Schema-on-read	Schema defined when data is queried, not stored	Common in data lakes — flexible but slower
Schema-on-write	Schema enforced at write time	Warehouses — strict, faster queries
Materialisation	Persisting computed results as tables or views	dbt "materialized='table'" caches query results
Slowly changing dimension (SCD)	Tracking history of dimension changes over time	Type 2 SCD: track every customer address change with valid_from/valid_to
Star schema	Modelling pattern: fact tables + dimension tables	Fact: orders (transactions). Dimensions: customers, products, dates

Key insight: The lake vs warehouse debate is basically over — modern stacks use lakehouses (Databricks, Snowflake Iceberg) that combine both. But understanding the original distinction matters: lakes are flexible/cheap, warehouses are fast/structured, lakehouses aim to be both.

SECTION 03Transformation + orchestration

Tools that process and coordinate data pipelines:

Term	Definition	Why it matters
dbt	Data build tool — SQL-based transformation framework	The standard for modern data transformations; version-controlled SQL
Airflow	Python-based workflow orchestration tool	Schedules and monitors pipelines as DAGs
Dagster	Newer orchestration tool with stronger typing + assets	Alternative to Airflow with better developer experience
DAG	Directed Acyclic Graph — the dependency structure of tasks	defines which tasks run before others, and in what order
Spark	Distributed data processing engine	Processes terabytes of data across clusters — the default for large scale
PySpark	Python API for Apache Spark	Write Spark jobs in Python — the standard for data engineers
Polars	Fast in-memory data processing library (Rust-based)	Single-machine alternative to Spark for mid-size data
Trino / Presto	Distributed SQL query engine	Query many sources with one SQL engine (lake + warehouse)
Databricks	Managed Spark + lakehouse platform	Popular for large-scale ML + analytics workloads
Feature store	Centralised system for ML features	Feast, Tecton — share features between training and serving

Key insight: dbt + Airflow + Spark is the most common modern stack in Indian companies. Knowing all three — and when to use each — is what most data engineering interviews test.

SECTION 04Streaming + real-time

Real-time data processing terms — increasingly essential for modern data engineers:

Term	Definition	Example
Kafka	Distributed event streaming platform	Ingests millions of events/sec from apps, databases, sensors
Topic	Named stream in Kafka where events are published	"orders.created", "user.clicks"
Producer	Component that writes events to a Kafka topic	App sends every order to "orders.created" topic
Consumer	Component that reads events from a topic	Fraud detection service reads from "orders.created"
Consumer group	Multiple consumers sharing the workload of a topic	3 fraud detection instances each process 1/3 of events
Partition (Kafka)	Ordered subset of a topic	Enables parallelism — events with same key go to same partition
Offset	Position of a consumer within a partition	Tracks what's been processed; resume on failure
Exactly-once	Each event processed exactly once (no dupes, no misses)	Hard to guarantee — often "at-least-once" + idempotent writes
Kinesis	AWS managed Kafka-like streaming service	Similar role to Kafka, if you're on AWS
Flink	Distributed stream processing engine	Process, window, aggregate streaming data in real time

Key insight: Streaming is harder than batch — you have to handle late data, out-of-order events, backpressure, and exactly-once semantics. But it's increasingly expected in data engineering roles, especially at consumer tech companies (Swiggy, Zomato, Paytm).

SECTION 05Quality + monitoring

Terms for keeping data pipelines reliable and trustworthy:

Term	Definition	Example
Data quality	How accurate, complete, and reliable data is	Nulls, duplicates, out-of-range values — all need to be detected
Great Expectations	Python library for data quality testing	Defines expectations like "amount > 0" and fails pipeline if violated
Data contract	Formal agreement on data schema + semantics between teams	Producer team guarantees schema; consumer team relies on it
Data lineage	Tracking where data came from and where it goes	dbt's built-in lineage graph shows table dependencies
Data catalog	Central metadata registry — what data exists, who owns it	DataHub, Amundsen, Atlan — searchable inventory
Data observability	Monitoring data pipelines in production	Freshness, volume, schema, distribution monitoring
SLA / SLO	Service Level Agreement / Objective	"Orders dashboard refreshes within 15 minutes of event"
Freshness	How up-to-date data is	Tracked and alerted on — "last updated 3 hours ago" is stale
Schema drift	When source schema changes unexpectedly	Column added/removed/renamed — breaks downstream pipelines
Data incident	Production data issue requiring investigation	Pipeline failed, wrong values written, data stale

Key insight: Data quality is where most data engineering candidates are weakest — and where the most valuable data engineers are strongest. A pipeline that runs fast but produces wrong data is worse than no pipeline at all.

SECTION 06Interview Q&A — terms that get tested

Q1What's the difference between ETL and ELT?

ETL transforms data before loading into the warehouse. ELT loads raw data first, then transforms inside the warehouse (typically with dbt). ELT is now standard because raw data is preserved for re-processing, and transformations are version-controlled SQL rather than opaque pipeline code.

Q2What's the difference between a data lake and a data warehouse?

A data lake stores raw data of any format (S3, GCS) with schema-on-read — flexible and cheap but requires query-time processing. A data warehouse stores structured, modelled data (Snowflake, BigQuery) with schema-on-write — strict but fast. Modern "lakehouses" (Databricks, Iceberg) combine both.

Q3What is CDC and why does it matter?

Change Data Capture — streaming every insert/update/delete from a source database as events. Debezium reads the Postgres WAL and emits each change to Kafka. CDC matters because it enables real-time pipelines without full re-scans. Instead of reloading a table nightly, you stream only the changes.

Q4What is idempotency and why does it matter in pipelines?

Idempotency means running a pipeline twice produces the same result. Critical because pipelines fail and retry — without idempotency, a retry could double-count orders, duplicate rows, or corrupt aggregates. Techniques: use MERGE/upsert instead of INSERT, delete-and-insert by partition, or deduplicate by primary key.

Q5What is dbt and why is it popular?

dbt (data build tool) lets you write transformations in SQL and manages them like software — with version control, testing, documentation, and lineage. It replaces custom Python transform code with SQL models + dbt's orchestration. Popular because it makes ELT maintainable and lets analysts contribute transformations, not just engineers.

SECTION 07Test yourself — data engineering terms

Five questions. No sign-up.

0 / 5

Pick an answer to see why it is right or wrong.

SECTION 08Frequently asked questions

Do I need to memorize all these terms before interviews?

No — but you should be comfortable with the core 20: pipeline, ETL, ELT, source, sink, CDC, batch, streaming, data warehouse, data lake, lakehouse, Parquet, dbt, Airflow, Spark, Kafka, idempotency, backfill, data quality, and lineage. Those come up in nearly every interview.

What's the difference between Airflow and dbt?

Airflow orchestrates pipelines — it schedules tasks and manages dependencies (DAGs). dbt transforms data — it's a SQL framework for writing maintainable models. In a modern stack, Airflow triggers dbt runs; dbt does the transformation inside the warehouse.

What's a fact table vs a dimension table?

Fact tables store measurements/events (transactions, clicks) — typically large and append-only. Dimension tables store descriptive attributes (customers, products, dates) — typically smaller and updated. This is the star schema pattern, still the standard for analytics modelling.

What's the difference between Kafka and Kinesis?

They're both distributed streaming platforms. Kafka is open-source and runs anywhere (self-managed, Confluent Cloud, MSK). Kinesis is AWS-managed and integrated tightly with AWS. Functionally similar; choice usually comes down to cloud provider and team familiarity.

What's schema drift and how do I handle it?

Schema drift is when a source's schema changes unexpectedly (new column, renamed field, type change). It breaks downstream pipelines. Handle it via: (1) schema registry for Kafka, (2) contract testing at ingestion, (3) default columns + type coercion in staging models, (4) alerts on schema changes.

What I would do in the first 30 daysLayoff · 14 min Why sending 500 resumes won't workJob Search · 14 min Rebuild your resume for the AI job marketResume · 15 min Which skills to learn before applyingSkills · 14 min 90-day career recovery roadmapRecovery · 16 min How AI can help you prepare for interviewsAI · 11 min

Classroom & online · Noida

Master data engineering — from terms to projects

Our Data Engineering Programme covers SQL, Python, Spark, dbt, Airflow, and cloud — with real projects and 1:1 placement support.

₹28,500 · full programme ~~₹42,000~~

Explore the course Download the syllabus

Modern data stack curriculum
3 real projects (Spark, dbt, cloud)
1:1 placement support
Weekday & weekend batches

Data Engineering Terms Every Beginner Should Know