Data Engineering · Beginner Vocabulary

Data Engineering Terminology Every Beginner Should Learn

SECTION 01Core concepts — pipelines and ingestion

These are the terms you will hear in every data engineering conversation.

• Data pipeline: A series of steps that move data from one place to another — source to destination — with transformations along the way.
• Data ingestion: The process of bringing data into a system — from APIs, databases, files, or event streams.
• ETL (Extract, Transform, Load): Extract data from sources, transform it, then load into a destination. Transformation happens before loading.
• ELT (Extract, Load, Transform): Extract data, load it into the destination, then transform using the destination's compute power. Common with cloud warehouses.
• Batch processing: Processing data in large chunks at scheduled intervals — e.g., every night.
• Streaming: Processing data continuously as it arrives — real-time events, clicks, transactions.
• Source system: Where data comes from — a database, API, application, or log file.
• Sink / destination: Where data ends up — a warehouse, lake, or downstream system.
• Connector: A pre-built integration that connects a source or destination to your pipeline — e.g., a Postgres connector.
• Idempotency: Running the same pipeline twice produces the same result. Critical for reliable data engineering.
• Backfill: Re-processing historical data after a fix or logic change.
• Schema drift: When a source changes its structure unexpectedly, breaking downstream pipelines.

SECTION 02Storage terminology

Where data lives, and how the storage is structured.

• Data warehouse: A structured, centralised store optimised for analytics and reporting — e.g., Snowflake, BigQuery, Redshift.
• Data lake: A large store for raw data in any format — structured, semi-structured, unstructured. Cheap and flexible.
• Data lakehouse: A hybrid that combines the flexibility of a lake with the structure and performance of a warehouse — e.g., Databricks, Iceberg.
• Data mart: A smaller, focused subset of a warehouse for a specific team or purpose — e.g., a marketing data mart.
• Object storage: Storing data as objects with metadata — e.g., AWS S3, Google Cloud Storage, Azure Blob.
• Block storage: Raw storage volumes attached to servers — used for databases and file systems.
• Columnar storage: Storing data by column instead of row — much faster for analytics — e.g., Parquet, ORC.
• Partitioning: Splitting large tables into smaller chunks by a key — e.g., by date — to speed up queries.
• Bucketing: A finer split within partitions, often by hash, to optimise joins and sampling.
• Data retention: How long data is kept before it is deleted or archived.
• Cold storage: Cheap, slow storage for data rarely accessed — e.g., S3 Glacier.
• Hot storage: Fast, expensive storage for frequently accessed data.

SECTION 03Processing terminology

How data is transformed, computed, and moved at scale.

• Transformation: Changing data format, structure, or values — cleaning, joining, aggregating, deriving.
• MapReduce: An older distributed processing model — the foundation of Hadoop. Rarely used directly today.
• Spark: A fast, in-memory distributed processing engine — the modern standard for big data processing.
• Hadoop: An older ecosystem for distributed storage (HDFS) and processing (MapReduce). Still used in legacy systems.
• Distributed processing: Splitting work across many machines to process data faster than one machine could.
• Shuffle: The expensive step in distributed processing where data is redistributed across nodes — often the bottleneck.
• DAG (Directed Acyclic Graph): A graph of tasks with dependencies but no cycles — the core model of Airflow and Spark.
• Job: A unit of work in a processing system — e.g., a Spark job.
• Task: A smaller unit within a job — often a single partition processed by one worker.
• Data skew: When some partitions are much larger than others, slowing down the whole job.
• Watermark: A marker for how late data can arrive in a streaming system before it is dropped.
• Exactly-once semantics: A guarantee that each record is processed once and only once, even with failures.
• At-least-once semantics: A weaker guarantee — records may be processed more than once on failure, requiring deduplication.

SECTION 04Architecture and modelling terms

Higher-level concepts that describe how data systems are designed.

• OLTP (Online Transaction Processing): Systems optimised for fast, small transactions — e.g., order systems, banking apps.
• OLAP (Online Analytical Processing): Systems optimised for complex analytical queries over large data — e.g., warehouses.
• Normalisation: Organising data to reduce redundancy and improve integrity — common in OLTP.
• Denormalisation: Combining tables to improve query speed — common in OLAP.
• Star schema: A modelling pattern with a central fact table and surrounding dimension tables — simple and fast for analytics.
• Snowflake schema: A star schema with further normalised dimensions — more complex, less common.
• Fact table: The central table holding measurable events — sales, clicks, transactions.
• Dimension table: Descriptive tables that give context to facts — customer, product, date.
• Slowly Changing Dimension (SCD): A pattern for tracking how dimension attributes change over time — SCD Type 1, 2, 3.
• Data lineage: Tracing where data came from and how it was transformed — critical for debugging and governance.
• Data quality: Ensuring data is accurate, complete, and consistent — testing, validation, monitoring.
• Data governance: Policies for ownership, access, privacy, and compliance — who can see what, and why.
• Metadata: Data about data — schema, lineage, ownership, freshness.
• Data catalog: A searchable inventory of datasets and their metadata — e.g., DataHub, Amundsen.

SECTION 05Tools and cloud platforms

The tools and platforms you will use as a data engineer.

• Airflow: A workflow orchestration tool that schedules and monitors data pipelines using DAGs.
• dbt (data build tool): A transformation tool that lets you write SQL models with tests, documentation, and version control.
• Kafka: A distributed event streaming platform for real-time data pipelines.
• Flink: A stream processing engine for stateful, low-latency stream processing.
• Snowflake: A cloud data warehouse with separated compute and storage.
• BigQuery: Google's serverless cloud data warehouse.
• Redshift: Amazon's cloud data warehouse.
• Databricks: A unified platform for data engineering, data science, and ML, built on Spark.
• AWS S3: Amazon's object storage service — the default storage layer for data lakes.
• AWS Glue: A serverless ETL service on AWS.
• Docker: Containerisation technology used to package and run data applications consistently.
• Kubernetes: Container orchestration — used to run data workloads at scale.
• Terraform: Infrastructure as code — provisioning cloud resources through configuration files.
• CI/CD: Continuous Integration and Continuous Deployment — automating testing and deployment of data pipelines.

SECTION 06How to actually remember these

Vocabulary without context fades in a week. Here is how to make it stick.

• Build a simple pipeline: Load a CSV into Postgres, transform it with SQL, and export it — you will use 10+ terms at once.
• Use a real dataset: Public datasets from Kaggle or government portals make the terms concrete.
• Try Airflow locally: Docker Compose runs Airflow in minutes. Seeing a DAG makes the term unforgettable.
• Read error messages: "Schema drift", "skew", "OOM" — each is a term with a meaning, not just an annoyance.
• Keep a personal glossary: Write each new term in your own words. If you cannot explain it simply, you do not understand it.
• Follow data engineering blogs: The content will make more sense as your vocabulary grows.
• Explain terms to someone else: Teaching is the fastest way to solidify vocabulary.

SECTION 07Your first 60 days learning plan

A realistic plan to go from zero vocabulary to comfortable with the fundamentals.

• Days 1–10: Master SQL. Terms: joins, aggregation, window functions, CTEs, indexes.
• Days 11–20: Learn Python for data — pandas, file I/O, APIs. Terms: dataframe, API call, serialisation.
• Days 21–30: Learn storage basics. Terms: warehouse, lake, object storage, partitioning, Parquet.
• Days 31–40: Build your first pipeline. Terms: ETL, ELT, ingestion, transformation, orchestration.
• Days 41–50: Learn orchestration. Terms: DAG, Airflow, schedule, dependency, retry.
• Days 51–60: Learn cloud basics. Terms: S3, IAM, Terraform, Docker, CI/CD.
• Ongoing: Read one data engineering article per week and add new terms to your glossary.

SECTION 08Test yourself — Data engineering terminology

SECTION 09Frequently asked questions

SECTION 10Related reads