Scenario-Based DevOps Interview Questions

Usually a large file was deleted while a process still holds it open, so the space is not released. Run lsof +L1 or lsof | grep deleted to find the process and file. Restarting or reloading that process frees the space; if you cannot restart, truncate the file through /proc/<pid>/fd/<n>. The fix is proper log rotation with copytruncate or a reopen signal.

Start with uptime and top or htop to see load and which processes use CPU. Check whether it is user, system or iowait time, and use pidstat or ps -eo pid,pcpu,cmd --sort=-pcpu for detail. For a Java process, take thread dumps; for others, use perf or strace. Correlate with recent deployments, cron jobs or traffic spikes before deciding to scale, roll back or fix code.

Check reachability with ping and whether port 22 is open with nc -zv host 22. Then look at security groups, firewall rules and whether the sshd service is running, using the cloud console or serial access. Run ssh -vvv to see whether it fails at key exchange or authentication. Common causes are wrong key permissions, a full disk, a changed IP, or fail2ban blocking you.

Find what is holding the port with ss -ltnp | grep 8080 or lsof -i :8080. If it is an old instance of the same app that did not shut down, stop it gracefully or fix the service manager so only one runs. If it is another service, change one app's port through configuration rather than hard coding. Then restart and confirm with curl localhost:8080.

Confirm with dmesg or journalctl -k, which shows the OOM message and the killed process. Check whether memory grows over time (a leak) or spikes with load. Set correct heap limits, for example JVM -Xmx below container memory, add swap only as a buffer, and set memory limits so one service cannot starve others. Then profile the app and right-size the instance.

The process has hit its file descriptor limit. Check usage with ls /proc/<pid>/fd | wc -l and the limit in /proc/<pid>/limits. Raise it through LimitNOFILE in the systemd unit or /etc/security/limits.conf, then restart. But also look for a leak: sockets or files not closed in code keep growing and will hit any limit. Raising the limit alone only buys time.

Treat the key as compromised immediately: deactivate and rotate it in IAM, then check CloudTrail for misuse. Removing it from history comes second, using git filter-repo or BFG and a force push, and asking everyone to re-clone. Then prevent repeats with secret scanning in CI, pre-commit hooks like gitleaks and storing secrets in a vault or secrets manager instead of code.

Git marks conflicting files and inserts markers showing both versions. Open each file, decide the correct final code, often combining both sides after talking to the other author, and remove the markers. Then git add the files and finish with git commit or git rebase --continue. Run tests before pushing, because a conflict resolved without errors can still break logic.

Git keeps unreferenced commits for a while. Run git reflog to see recent HEAD positions and find the last commit of that branch, or check the hash printed when you deleted it. Then recreate the branch with git branch feature-x <hash>. Reflog is local only, so this works on the same machine, and entries expire after about 90 days by default.

First restore a green main quickly: run git revert <hash> and push, which adds an undo commit without rewriting history that others have pulled. Tell the team in the channel. Then fix the issue on a branch with a proper pull request and tests. Avoid git reset with force push on main; branch protection rules should block it anyway.

Short term, CI can use shallow clones with --depth 1 and partial clone filters. For future binaries, move them to Git LFS or an artefact store such as S3 or Nexus. To shrink existing history, use git filter-repo to remove the large files, coordinate a force push and ask everyone to re-clone. Add size checks in pre-receive hooks to prevent recurrence.

The cleanest approach is feature flags: both features merge to main, but the unfinished one stays switched off in production, so deployment is separated from release. Without flags, you cut a release branch and revert or exclude the second feature, which is error prone. For the future, keep branches short-lived and agree with product on flag-based releases.

Measure first: find which stages take longest. Then cache dependencies and Docker layers, run independent stages and test suites in parallel, split tests across agents, and run only affected modules in a monorepo. Move slow end-to-end tests to a later stage or nightly run while keeping fast unit tests on every commit. Use bigger or autoscaled agents if they are the bottleneck.

First identify flaky tests by tracking which tests fail and pass on the same commit. Quarantine them into a separate non-blocking job so the main pipeline stays trustworthy, and raise tickets with owners. Common causes are timing assumptions, shared test data, order dependence and external services. Blind automatic retries hide real bugs, so use them only briefly while fixing the root cause.

Assess impact quickly using monitoring and error rates. If users are affected, roll back to the last good version, which should be a single pipeline action redeploying the previous artefact. Communicate status to stakeholders. Then check pipeline logs and application logs to find the cause, fix it in a branch and redeploy. Afterwards, hold a blameless review and add a check to catch it earlier.

The environments differ. Compare tool versions such as JDK, Node or Python, environment variables, and dependency versions, especially if there is no lock file. Check for files present locally but not committed, and for hard-coded paths. Reproduce by running the build inside the same Docker image the CI agent uses. Long term, build inside containers so local and CI match.

Build and test once, publish a versioned artefact, then promote it through environments. Dev deploys automatically on merge, staging deploys after dev smoke tests, and production waits for a manual approval from authorised people using a Jenkins input step or GitHub environment protection rules. Keep environment-specific settings in config files or parameters, not separate pipelines, and record who approved each release.

Make sure no builds run on the controller by setting its executors to zero. Add agents, ideally dynamic ones on Kubernetes or EC2 that scale with the queue. Check controller heap, disk and plugin count, and remove unused plugins. Discard old builds and artefacts with retention rules. For very large setups, split teams across multiple controllers or move suitable jobs to another CI tool.

Run docker ps -a to see the exit code and docker logs <id> for output. Exit code 0 usually means the main process finished, for example a script that ran once or a service started in background mode, so the container has nothing to keep it alive. Non-zero codes point to errors such as missing config. Start an interactive shell with docker run -it --entrypoint sh to inspect.

Switch to a slimmer base image such as a slim or distroless variant. Use a multi-stage build so compilers and build caches stay out of the final image. Combine package installs and clean caches in the same RUN layer. Add a .dockerignore to exclude .git, node_modules and test data. Then inspect remaining layers with docker history or the dive tool.

Check that both are on the same user-defined network with docker network inspect; the default bridge does not resolve container names. Use the service or container name and the container's internal port, not localhost and not the published host port. Confirm the app listens on 0.0.0.0 rather than 127.0.0.1. Then test with curl or nc from inside one container.

Data written inside the container's writable layer is deleted when the container is removed. The database must store its data directory in a named volume or bind mount, for example -v pgdata:/var/lib/postgresql/data. Then containers can be upgraded or recreated without losing data. Also take regular backups, because volumes protect against container removal, not against disk failure.

Check usage with docker system df. Remove stopped containers, dangling images and unused networks with docker system prune, and old build cache with docker builder prune; be careful with the volumes flag. Container logs are a common culprit, so configure log rotation with max-size and max-file in daemon.json. In CI agents, schedule regular pruning jobs.

docker stop sends SIGTERM to PID 1 and waits 10 seconds before SIGKILL. If the Dockerfile uses shell form like CMD node app.js, PID 1 is a shell that does not forward signals, so the app never gets SIGTERM. Use exec form CMD ["node", "app.js"], use exec in entrypoint scripts, or add an init such as tini with --init, and handle SIGTERM in code.

The container starts and keeps crashing, and Kubernetes waits longer between restarts. Run kubectl describe pod to see events and exit codes, and kubectl logs <pod> --previous for the crashed container's output. Common causes are missing environment variables or secrets, bad config, failing database connections, a wrong command, an aggressive liveness probe, or exit code 137 meaning OOM kill.

Pending means it has not been scheduled or its volumes are not ready. kubectl describe pod shows the scheduler's reason. Common causes are insufficient CPU or memory on nodes for the requests, node selectors, affinity or taints that no node satisfies, a PersistentVolumeClaim that cannot bind, or a resource quota being exceeded. Fix the spec or add nodes with the cluster autoscaler.

The kubelet cannot pull the image. Check the image name and tag for typos and confirm the tag exists in the registry. For private registries, verify an imagePullSecret is set and has valid credentials, or that node IAM roles allow pulling from ECR. Also check network access from nodes to the registry and registry rate limits. kubectl describe pod shows the exact error.

Run kubectl get endpoints <svc>; if empty, the service selector does not match the pod labels or pods are not ready. Check that the service targetPort matches the container port. Test from a debug pod with curl svc-name.namespace:port to separate DNS from app issues. Then check network policies blocking traffic and, for external access, the ingress rules and load balancer health checks.

Add a proper readiness probe so new pods get traffic only when ready. Set maxUnavailable to 0 and maxSurge to 1 or more. Handle SIGTERM gracefully and add a short preStop sleep so endpoints update before the old pod stops accepting connections, with terminationGracePeriodSeconds long enough to drain requests. Add a PodDisruptionBudget for node maintenance too.

Run kubectl describe node to see conditions such as MemoryPressure, DiskPressure or a kubelet that stopped posting status. SSH in and check systemctl status kubelet, the container runtime, journalctl -u kubelet, disk space and network to the API server. Pods on the node are evicted after a timeout. Cordon and drain it, fix or replace the node, and in cloud setups let the node group recreate it.

Without locking, both runs read the same state and write conflicting updates, which can corrupt state or create duplicate resources. Use a remote backend with state locking so the second run waits or fails. Better still, run apply only from a CI pipeline on merge to main, so changes are serialised and reviewed. If a lock is stuck after a crash, use terraform force-unlock carefully.

Run terraform plan; Terraform refreshes and sees the resource is missing, so it plans to recreate it, which is often right. If the resource should no longer exist, remove it from code, or use terraform state rm. If state itself is corrupted, restore a previous version from the versioned S3 bucket. Then find out why manual changes happened and tighten access.

Write the resource blocks in code, then import the real resources into state. Since Terraform 1.5 you can declare import blocks with the resource ID and run plan; terraform plan -generate-config-out can even draft the configuration. Older versions use the terraform import CLI command. Run plan until it shows no changes, which proves code matches reality.

Keep shared logic in versioned modules and create a separate root configuration per environment, each with its own backend state and tfvars, for example envs/dev and envs/prod. This isolates state so a mistake in dev cannot touch prod. Workspaces are an option for near-identical environments but share backend config. Promote module versions from dev to prod through pull requests.

Check the output and the retry file or failed host list to see the error on those hosts, for example unreachable SSH, a missing package repo or full disk. Fix the cause, then rerun only those hosts with --limit. Because tasks are idempotent, rerunning on successful hosts is also safe. For rolling changes, use serial and max_fail_percentage to stop early on failures.

Do not hard-code secrets in .tf files or playbooks. In Terraform, read them from AWS Secrets Manager or Vault data sources, mark variables sensitive, and protect state because it may still hold values. In Ansible, encrypt secret variable files with ansible-vault and supply the vault password from CI secrets, or look up values from Vault at runtime. Rotate anything exposed in Git.

A private subnet has no direct route to the Internet Gateway. Check that its route table sends 0.0.0.0/0 to a NAT Gateway placed in a public subnet, that the NAT Gateway has an Elastic IP, and that the security group and NACL allow outbound traffic and return ports. For AWS services like S3, a VPC endpoint avoids NAT data charges.

Open Cost Explorer and group by service, account and tag to find the spike, then drill into usage type. Common causes are forgotten instances, unattached EBS volumes, NAT Gateway data transfer, oversized RDS, or logs retained forever. Fix the source, then set AWS Budgets alerts, enforce tagging, right-size using Compute Optimizer, and use Savings Plans or Spot for steady or batch workloads.

Load test beforehand to find bottlenecks. Pre-scale Auto Scaling groups or Kubernetes nodes with scheduled scaling, since reactive scaling may be too slow. Cache aggressively with CloudFront and Redis, add read replicas, and check service quotas. Use queues to absorb spikes in order processing, set up dashboards and alerts, freeze risky deployments, and keep an on-call war room ready.

Create an IAM role with a policy allowing only the needed actions, such as s3:GetObject on that bucket, attach it to the instance through an instance profile, and let the AWS SDK pick up temporary credentials automatically. Never put access keys in code or config files. Optionally add a bucket policy and a VPC endpoint so traffic stays inside AWS.

Start from business RTO and RPO. Options range from backup and restore, which is cheapest and slowest, to pilot light, warm standby and multi-region active-active. Replicate data with S3 cross-region replication, Aurora Global Database or DynamoDB global tables, keep infrastructure in Terraform so it can be rebuilt, use Route 53 failover routing and rehearse failover drills.

Immediately block public access on the bucket and remove the public policy or ACLs. Check S3 server access logs or CloudTrail data events to see whether objects were downloaded, and involve security if customer data was exposed. Then enable Block Public Access at the account level, use AWS Config rules to detect public buckets, and require bucket changes through reviewed IaC.

Confirm the endpoint, port, username and password. Check that the RDS security group allows inbound traffic on the database port from the application's security group, and that both are in reachable subnets of the same VPC or peered networks. Test with nc -zv endpoint 5432 from the app host. Also check the instance status, max connections and SSL requirements.

Averages can hide pain, so check p95 and p99 latency, not mean. Find out which users, regions or features are affected; it may be one endpoint, a third-party API or client-side slowness. Use traces to follow a slow request and check DNS, CDN and front-end metrics through real user monitoring. Then add the missing signal to dashboards so it is visible next time.

Collect alert data for a few weeks: which fired, how often, and whether anyone acted. Delete or downgrade non-actionable alerts, tune thresholds and add for durations, and group duplicates in Alertmanager. Fix the recurring root causes behind the top alerts and automate common fixes. Move to SLO-based alerts and make sure rotations are fair with follow-up time after bad nights.

Look at memory graphs over days: a leak shows steady growth that drops only on restart, often with OOM kills. Correlate the start with a deployment. Take heap dumps or profiles, for example jmap for Java or pprof for Go, and compare them over time. As a short-term fix, set memory limits and schedule restarts or roll back, then fix the code and add a memory alert.

Declare the incident, open a channel and bridge, and assign roles: someone to investigate, someone to communicate and a scribe. My job is to coordinate, not to debug. Prioritise mitigation over root cause, such as rollback, failover or feature flags. Post regular status updates to stakeholders. After recovery, confirm it is stable, close it and schedule a blameless post-mortem with action items.

Find the top log sources by volume. Lower verbose debug logging in production, drop health check and noisy access logs, and sample high-volume info logs. Use structured logs to avoid duplicated fields. Set retention tiers: a few days hot, then archive to S3. Turn repeated log lines into metrics where only counts matter, which is far cheaper to store and query.

Following the agreed error budget policy, the team pauses risky feature launches and focuses on reliability work: fixing the top causes of failures from post-mortems, adding tests, improving rollbacks and canaries. Product and engineering leads review the plan together. Once the budget recovers in the rolling window, normal releases resume. If the SLO is unrealistic, revisit it with data.

Acknowledge the alert so others know you are on it. Check whether it is really down from outside with curl or an uptime checker, and whether it is all users or one region. Look at recent deployments or changes, then load balancer health, application logs and the database. If a recent change caused it, roll back first. Inform the team and escalate if you cannot fix it quickly.