Hard DevOps Interview Questions

The firmware (BIOS or UEFI) runs hardware checks and loads the bootloader, usually GRUB. GRUB loads the kernel and the initramfs, which contains drivers needed to mount the real root file system. The kernel then starts PID 1, which is systemd on most modern distributions. systemd mounts file systems and starts services in parallel according to unit dependencies and the default target.

Usually a large file was deleted while a process still holds it open, so the space is not released. Run lsof +L1 or lsof | grep deleted to find the process and file. Restarting or reloading that process frees the space; if you cannot restart, truncate the file through /proc/<pid>/fd/<n>. The fix is proper log rotation with copytruncate or a reopen signal.

Confirm with dmesg or journalctl -k, which shows the OOM message and the killed process. Check whether memory grows over time (a leak) or spikes with load. Set correct heap limits, for example JVM -Xmx below container memory, add swap only as a buffer, and set memory limits so one service cannot starve others. Then profile the app and right-size the instance.

The process has hit its file descriptor limit. Check usage with ls /proc/<pid>/fd | wc -l and the limit in /proc/<pid>/limits. Raise it through LimitNOFILE in the systemd unit or /etc/security/limits.conf, then restart. But also look for a leak: sockets or files not closed in code keep growing and will hit any limit. Raising the limit alone only buys time.

GitFlow uses long-lived develop and main branches plus feature, release and hotfix branches; it suits scheduled releases but causes big merges. Trunk-based development has everyone merge small changes into main at least daily, with short-lived branches and feature flags to hide unfinished work. It suits continuous delivery and is what high-performing DevOps teams usually prefer, but needs strong automated tests.

Treat the key as compromised immediately: deactivate and rotate it in IAM, then check CloudTrail for misuse. Removing it from history comes second, using git filter-repo or BFG and a force push, and asking everyone to re-clone. Then prevent repeats with secret scanning in CI, pre-commit hooks like gitleaks and storing secrets in a vault or secrets manager instead of code.

Short term, CI can use shallow clones with --depth 1 and partial clone filters. For future binaries, move them to Git LFS or an artefact store such as S3 or Nexus. To shrink existing history, use git filter-repo to remove the large files, coordinate a force push and ask everyone to re-clone. Add size checks in pre-receive hooks to prevent recurrence.

The cleanest approach is feature flags: both features merge to main, but the unfinished one stays switched off in production, so deployment is separated from release. Without flags, you cut a release branch and revert or exclude the second feature, which is error prone. For the future, keep branches short-lived and agree with product on flag-based releases.

DevSecOps moves security earlier into the delivery process instead of a final audit. A pipeline typically adds secret scanning, static code analysis (SAST) with tools like SonarQube, dependency scanning (SCA) for vulnerable libraries, container image scanning with Trivy, infrastructure-as-code checks with Checkov, and sometimes dynamic testing (DAST) on staging. Critical findings fail the build, while lower ones create tickets.

GitOps uses a Git repository as the single source of truth for the desired state of infrastructure and applications. An agent running in the cluster, such as Argo CD or Flux, continuously compares the live state with Git and pulls changes to reconcile drift. Deployments become pull requests, giving review, audit history and easy rollback by reverting a commit. CI builds images; GitOps handles deployment.

DORA metrics measure software delivery performance: deployment frequency, lead time for changes, change failure rate and time to restore service. The first two measure speed and the last two measure stability. Elite teams deploy on demand with lead times under a day and recover within an hour. DevOps teams use them to find bottlenecks, not to rank individual engineers.

Measure first: find which stages take longest. Then cache dependencies and Docker layers, run independent stages and test suites in parallel, split tests across agents, and run only affected modules in a monorepo. Move slow end-to-end tests to a later stage or nightly run while keeping fast unit tests on every commit. Use bigger or autoscaled agents if they are the bottleneck.

Make sure no builds run on the controller by setting its executors to zero. Add agents, ideally dynamic ones on Kubernetes or EC2 that scale with the queue. Check controller heap, disk and plugin count, and remove unused plugins. Discard old builds and artefacts with retention rules. For very large setups, split teams across multiple controllers or move suitable jobs to another CI tool.

Situation: a release passed staging but caused payment timeouts in production because a new connection pool setting was wrong. Task: as release owner, I had to restore service. Action: I triggered the rollback job within ten minutes, updated stakeholders, then found the config difference between environments. Result: service recovered quickly and we added config validation and a canary stage to the pipeline.

Linux namespaces give each container its own isolated view of process IDs, network interfaces, mount points, hostname, users and IPC, so it believes it is alone on the machine. Control groups (cgroups) limit and account for CPU, memory and I/O usage. A container runtime such as containerd with runc sets these up, plus an overlay file system for image layers.

Use small, trusted base images such as distroless or alpine and pin versions or digests. Run as a non-root user, use multi-stage builds, and never bake secrets into layers. Scan images with Trivy or similar in CI and rebuild regularly to pick up patches. At runtime, drop Linux capabilities, use read-only root file systems and sign images to verify their source.

docker stop sends SIGTERM to PID 1 and waits 10 seconds before SIGKILL. If the Dockerfile uses shell form like CMD node app.js, PID 1 is a shell that does not forward signals, so the app never gets SIGTERM. Use exec form CMD ["node", "app.js"], use exec in entrypoint scripts, or add an init such as tini with --init, and handle SIGTERM in code.

Requests are what the scheduler reserves for a container when placing it on a node; limits are the maximum it may use. Exceeding a CPU limit causes throttling, while exceeding a memory limit gets the container OOM killed. Pods are Guaranteed when requests equal limits, Burstable when requests are lower, and BestEffort with none; BestEffort pods are evicted first under node pressure.

RBAC controls who can do what on which resources. A Role defines allowed verbs like get, list or create on resources within a namespace; a ClusterRole does so cluster-wide. RoleBindings and ClusterRoleBindings attach those roles to users, groups or service accounts. Follow least privilege: give each application its own service account with only the permissions it needs.

Add a proper readiness probe so new pods get traffic only when ready. Set maxUnavailable to 0 and maxSurge to 1 or more. Handle SIGTERM gracefully and add a short preStop sleep so endpoints update before the old pod stops accepting connections, with terminationGracePeriodSeconds long enough to drain requests. Add a PodDisruptionBudget for node maintenance too.

Run kubectl describe node to see conditions such as MemoryPressure, DiskPressure or a kubelet that stopped posting status. SSH in and check systemctl status kubelet, the container runtime, journalctl -u kubelet, disk space and network to the API server. Pods on the node are evicted after a timeout. Cordon and drain it, fix or replace the node, and in cloud setups let the node group recreate it.

Situation: our monolith and five services ran on hand-managed VMs with frequent config drift. Task: I led the move to a managed Kubernetes cluster. Action: I containerised each service, wrote Helm charts, added probes and resource limits, set up Argo CD, and migrated one low-risk service first with a rollback plan. Result: deployments went from weekly to daily and VM costs dropped about 30 percent.

Drift is when real infrastructure differs from code, usually because someone changed it manually in the console. terraform plan, or plan -refresh-only, shows differences between state and reality. Teams run scheduled plans in CI and alert on any diff. Prevent drift by restricting console write access, applying only through pipelines, and either reverting manual changes or codifying them.

Without locking, both runs read the same state and write conflicting updates, which can corrupt state or create duplicate resources. Use a remote backend with state locking so the second run waits or fails. Better still, run apply only from a CI pipeline on merge to main, so changes are serialised and reviewed. If a lock is stuck after a crash, use terraform force-unlock carefully.

Run terraform plan; Terraform refreshes and sees the resource is missing, so it plans to recreate it, which is often right. If the resource should no longer exist, remove it from code, or use terraform state rm. If state itself is corrupted, restore a previous version from the versioned S3 bucket. Then find out why manual changes happened and tighten access.

Keep shared logic in versioned modules and create a separate root configuration per environment, each with its own backend state and tfvars, for example envs/dev and envs/prod. This isolates state so a mistake in dev cannot touch prod. Workspaces are an option for near-identical environments but share backend config. Promote module versions from dev to prod through pull requests.

Do not hard-code secrets in .tf files or playbooks. In Terraform, read them from AWS Secrets Manager or Vault data sources, mark variables sensitive, and protect state because it may still hold values. In Ansible, encrypt secret variable files with ansible-vault and supply the vault password from CI secrets, or look up values from Vault at runtime. Rotate anything exposed in Git.

I would start with their pain, such as slow environment setup or outages from manual changes, rather than the tool. Then I would codify one small, visible piece, for example a new staging environment, and show it rebuilt in minutes with a reviewed pull request. I would pair with team members, write simple docs and expand gradually, measuring time saved and incidents avoided.

Spread every tier across at least two Availability Zones. Put an ALB in front of an Auto Scaling group of stateless instances or containers. Use RDS Multi-AZ or Aurora for the database and ElastiCache for sessions. Serve static content from S3 through CloudFront and use Route 53 health checks. Remove single points of failure and test failover regularly.

Open Cost Explorer and group by service, account and tag to find the spike, then drill into usage type. Common causes are forgotten instances, unattached EBS volumes, NAT Gateway data transfer, oversized RDS, or logs retained forever. Fix the source, then set AWS Budgets alerts, enforce tagging, right-size using Compute Optimizer, and use Savings Plans or Spot for steady or batch workloads.

Load test beforehand to find bottlenecks. Pre-scale Auto Scaling groups or Kubernetes nodes with scheduled scaling, since reactive scaling may be too slow. Cache aggressively with CloudFront and Redis, add read replicas, and check service quotas. Use queues to absorb spikes in order processing, set up dashboards and alerts, freeze risky deployments, and keep an on-call war room ready.

Start from business RTO and RPO. Options range from backup and restore, which is cheapest and slowest, to pilot light, warm standby and multi-region active-active. Replicate data with S3 cross-region replication, Aurora Global Database or DynamoDB global tables, keep infrastructure in Terraform so it can be rebuilt, use Route 53 failover routing and rehearse failover drills.

Situation: a team saw our new pipeline checks as slowing them down. Task: get adoption without forcing it. Action: I sat with them to understand the pain, found that the scans added twelve minutes, moved slow scans to run in parallel and made results clearer. Result: pipeline time went below the old one, the team adopted it and later suggested further improvements.

Prometheus pulls, or scrapes, metrics over HTTP from targets that expose a /metrics endpoint, either apps instrumented with client libraries or exporters such as node exporter. Targets come from static config or service discovery, for example Kubernetes. Data is stored in a local time-series database and queried with PromQL. Alert rules fire to Alertmanager, which groups, silences and routes notifications. Grafana visualises the data.

An SLI is a measured indicator, for example the share of requests served under 300 ms. An SLO is the internal target for it, such as 99.9 percent over 30 days. An SLA is the external contract with penalties, usually looser than the SLO. The error budget is the allowed unreliability, 0.1 percent here; while budget remains teams ship features, and when it runs out they focus on reliability.

Distributed tracing follows one request across many services. Each operation is a span with timing and attributes, and spans share a trace ID passed in request headers, so tools like Jaeger or Tempo can show where time is spent. OpenTelemetry is the vendor-neutral standard and SDK for producing traces, metrics and logs, so you instrument once and send data to any backend.

A good alert is actionable, urgent and tied to user impact, such as an SLO burn rate, not just high CPU. Each page should have a runbook link and a clear owner. Reduce noise by removing alerts nobody acts on, using severity levels so non-urgent issues create tickets, grouping related alerts, and using for durations to ignore short spikes. Review alerts after each on-call week.

Averages can hide pain, so check p95 and p99 latency, not mean. Find out which users, regions or features are affected; it may be one endpoint, a third-party API or client-side slowness. Use traces to follow a slow request and check DNS, CDN and front-end metrics through real user monitoring. Then add the missing signal to dashboards so it is visible next time.

Look at memory graphs over days: a leak shows steady growth that drops only on restart, often with OOM kills. Correlate the start with a deployment. Take heap dumps or profiles, for example jmap for Java or pprof for Go, and compare them over time. As a short-term fix, set memory limits and schedule restarts or roll back, then fix the code and add a memory alert.

Declare the incident, open a channel and bridge, and assign roles: someone to investigate, someone to communicate and a scribe. My job is to coordinate, not to debug. Prioritise mitigation over root cause, such as rollback, failover or feature flags. Post regular status updates to stakeholders. After recovery, confirm it is stable, close it and schedule a blameless post-mortem with action items.

Find the top log sources by volume. Lower verbose debug logging in production, drop health check and noisy access logs, and sample high-volume info logs. Use structured logs to avoid duplicated fields. Set retention tiers: a few days hot, then archive to S3. Turn repeated log lines into metrics where only counts matter, which is far cheaper to store and query.

Following the agreed error budget policy, the team pauses risky feature launches and focuses on reliability work: fixing the top causes of failures from post-mortems, adding tests, improving rollbacks and canaries. Product and engineering leads review the plan together. Once the budget recovers in the rolling window, normal releases resume. If the SLO is unrealistic, revisit it with data.

Situation: our order service had repeated weekend outages from database connection exhaustion. Task: as the senior engineer I owned the fix. Action: I defined an availability SLO, added connection pool metrics and alerts, introduced a connection proxy, and added load tests to the pipeline. Result: monthly availability went from about 99.5 to 99.95 percent and pages dropped by more than half.