How Kubernetes Helps You Land a Data Engineer Job Faster

apiVersion: apps/v1
kind: Deployment
metadata:
  name: airflow-web
spec:
  replicas: 1
  selector:
    matchLabels: { app: airflow-web }
  template:
    metadata:
      labels: { app: airflow-web }
    spec:
      containers:
        - name: web
          image: apache/airflow:2.9.3
          ports: [{ containerPort: 8080 }]
          readinessProbe:
            httpGet: { path: /health, port: 8080 }
---
apiVersion: v1
kind: Service
metadata:
  name: airflow-web
spec:
  selector: { app: airflow-web }
  ports: [{ port: 80, targetPort: 8080 }]

apiVersion: v1
kind: ConfigMap
metadata:
  name: pipeline-config
data:
  WAREHOUSE: "analytics"
  BATCH_SIZE: "50000"
---
apiVersion: v1
kind: Secret
metadata:
  name: pipeline-secrets
type: Opaque
stringData:
  DB_PASSWORD: "set-in-your-secret-manager"

apiVersion: batch/v1
kind: Job
metadata:
  name: nightly-aggregate
spec:
  backoffLimit: 2          # retry a couple of times on failure
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: aggregate
          image: registry.example.com/etl:1.4.0
          command: ["python", "aggregate.py", "--date", "2026-11-05"]
          resources:
            requests: { cpu: "500m", memory: "1Gi" }
            limits:   { cpu: "1",    memory: "2Gi" }

apiVersion: batch/v1
kind: CronJob
metadata:
  name: nightly-aggregate
spec:
  schedule: "0 2 * * *"     # 02:00 daily
  jobTemplate:
    spec:
      template:
        spec:
          restartPolicy: Never
          containers:
            - name: aggregate
              image: registry.example.com/etl:1.4.0
              command: ["python", "aggregate.py"]

kubectl get jobs -n data
kubectl get pods -n data --selector=job-name=nightly-aggregate
kubectl logs job/nightly-aggregate -n data --tail=100
kubectl describe pod <pod> -n data        # events: OOMKilled? Pending?

# Delete the failed job and recreate it (idempotent job only)
kubectl delete job nightly-aggregate -n data
kubectl apply -f batch-jobs.yaml

# Or trigger a one-off run from a CronJob
kubectl create job --from=cronjob/nightly-aggregate manual-run -n data

Day 1-2  : Containers and images; run a local cluster (kind/minikube)
Day 3-4  : Pods, Deployments, Services
Day 5-6  : ConfigMaps and Secrets for pipeline config
Day 7-8  : Jobs and CronJobs for batch work
Day 9-10 : Logs, describe, resource requests/limits
Day 11-12: Deploy Airflow (or a sample pipeline) on the cluster
Day 13-14: Diagnose a failed job; document recovery

Project: A small data platform on Kubernetes
- A CronJob that runs an idempotent ETL script
- ConfigMap + Secret for settings
- Resource requests and limits set
- A README showing how to deploy, run and recover
- An incident note: a job you broke and fixed

- Ran data pipelines and batch jobs on Kubernetes
  (Deployments, CronJobs, resource limits); diagnosed and
  recovered failed jobs using logs and events