Data Engineering Glossary 50 Terms You Must Know in 2026

5.  Source system   Where data originates (app DB, CRM, API)
6.  Sink / target   Where data is written (warehouse, lake)
7.  Full load       Copy all rows every run
8.  Incremental     Copy only new or changed rows
9.  Watermark       Last processed value, e.g. max updated_at
10. CDC             Change Data Capture: read row changes
                    from database logs
11. Connector       Ready-made integration to a source
12. Message queue   Buffer holding events between systems
13. Topic           Named stream of events (Kafka)
14. Schema          Expected columns and types of data

-- Incremental extract using a watermark (last_loaded_at)
SELECT *
FROM app.orders
WHERE updated_at > :last_loaded_at
ORDER BY updated_at;

-- After a successful load, store the new watermark:
-- last_loaded_at = MAX(updated_at) from this batch

19. Lakehouse      Lake storage with warehouse-like tables
                   (Delta Lake, Apache Iceberg, Hudi)
20. Object storage Storage for files as objects (S3, GCS)
21. Columnar       Stores values column by column
22. Row-oriented   Stores whole rows together (OLTP DBs)
23. OLTP           Transaction systems (app databases)
24. OLAP           Analytical systems (warehouses)
25. Star schema    Fact table linked to dimension tables
26. Fact table     Events or measures (orders, payments)
27. Dimension      Descriptive attributes (customer, date)
28. SCD Type 2     Keep history by adding a new row
                   when an attribute changes

import pandas as pd

df = pd.read_csv("orders.csv", parse_dates=["order_date"])
df["dt"] = df["order_date"].dt.strftime("%Y-%m-%d")

# Writes folders like out/orders/dt=2026-11-06/...parquet
df.to_parquet("out/orders", partition_cols=["dt"], index=False)

33. Task          One step in a DAG
34. Retry         Automatic re-run after a failure
35. SLA           Time by which data must be ready
36. Spark         Engine for distributed data processing
37. Transformation Change data: clean, join, aggregate
38. dbt           Tool to write SQL transformations with
                  tests and documentation
39. Data quality  Checks that data is complete and valid
40. Freshness     How recent the latest data is
41. Lineage       Map of where data came from and goes
42. Upsert        Insert new rows, update existing ones

# dbt schema tests (models/schema.yml)
version: 2
models:
  - name: fct_orders
    columns:
      - name: order_id
        tests:
          - unique
          - not_null
      - name: customer_id
        tests:
          - relationships:
              to: ref('dim_customers')
              field: customer_id

43. Data contract
    An agreed schema and quality promise between the
    team producing data and the teams using it.

44. Exactly-once processing
    A streaming guarantee that each event affects the
    result once, even after failures and retries.

45. Schema evolution
    Safely adding or changing columns over time without
    breaking existing readers.

46. Data skew
    Uneven data distribution so a few workers do most
    of the work in a distributed job (e.g. one huge key).

47. Shuffle
    Moving data between workers for joins or group-bys
    in engines like Spark; often the slowest step.

48. Medallion architecture
    Layering data as bronze (raw), silver (cleaned)
    and gold (business-ready) tables.

49. Data mesh
    An approach where domain teams own and publish
    their data as products, with shared standards.

50. Reverse ETL
    Sending modelled warehouse data back into business
    tools such as a CRM or marketing platform.