Site Reliability Engineer Interview Questions &amp; Answers

Show answer

Run kubectl describe node to see conditions such as MemoryPressure, DiskPressure or a kubelet that stopped posting status. SSH in and check systemctl status kubelet, the container runtime, journalctl -u kubelet, disk space and network to the API server. Pods on the node are evicted after a timeout. Cordon and drain it, fix or replace the node, and in cloud setups let the node group recreate it.

You are the incident commander for a major outage. What do you do?

Show answer

Declare the incident, open a channel and bridge, and assign roles: someone to investigate, someone to communicate and a scribe. My job is to coordinate, not to debug. Prioritise mitigation over root cause, such as rollback, failover or feature flags. Post regular status updates to stakeholders. After recovery, confirm it is stable, close it and schedule a blameless post-mortem with action items.

What is load average in Linux and how do you read it?

Infrastructure as Code (Terraform and Ansible)Hard

Show answer

Load average is the average number of processes running or waiting for CPU (and uninterruptible I/O) over 1, 5 and 15 minutes, shown by uptime or top. Compare it with the number of CPU cores: a load of 4 on a 4-core machine means full use, while 12 means work is queuing. A high load with low CPU use often points to disk or network I/O waits.

How do you manage dev, staging and production environments with Terraform?

Show answer

Keep shared logic in versioned modules and create a separate root configuration per environment, each with its own backend state and tfvars, for example envs/dev and envs/prod. This isolates state so a mistake in dev cannot touch prod. Workspaces are an option for near-identical environments but share backend config. Promote module versions from dev to prod through pull requests.

What is an inode in Linux?

Show answer

An inode is a data structure that stores a file's metadata: owner, permissions, size, timestamps, link count and pointers to data blocks, but not the file name. Names live in directory entries that map to inode numbers. A file system has a fixed number of inodes, so millions of tiny files can exhaust them while disk space is still free; check with df -i.

Tell me about a time you improved the reliability of a system.

Infrastructure as Code (Terraform and Ansible)Hard

Show answer

Situation: our order service had repeated weekend outages from database connection exhaustion. Task: as the senior engineer I owned the fix. Action: I defined an availability SLO, added connection pool metrics and alerts, introduced a connection proxy, and added load tests to the pipeline. Result: monthly availability went from about 99.5 to 99.95 percent and pages dropped by more than half.

What is configuration drift and how do you detect it with Terraform?

Show answer

Drift is when real infrastructure differs from code, usually because someone changed it manually in the console. terraform plan, or plan -refresh-only, shows differences between state and reality. Teams run scheduled plans in CI and alert on any diff. Prevent drift by restricting console write access, applying only through pipelines, and either reverting manual changes or codifying them.

How do you design a highly available web application on AWS?

Show answer

Spread every tier across at least two Availability Zones. Put an ALB in front of an Auto Scaling group of stateless instances or containers. Use RDS Multi-AZ or Aurora for the database and ElastiCache for sessions. Serve static content from S3 through CloudFront and use Route 53 health checks. Remove single points of failure and test failover regularly.

What makes a good alert, and how do you avoid alert fatigue?

Show answer

A good alert is actionable, urgent and tied to user impact, such as an SLO burn rate, not just high CPU. Each page should have a runbook link and a clear owner. Reduce noise by removing alerts nobody acts on, using severity levels so non-urgent issues create tickets, grouping related alerts, and using for durations to ignore short spikes. Review alerts after each on-call week.

Pods are running but the service is not reachable. How do you debug it?

KubernetesMedium

Show answer

Run kubectl get endpoints <svc>; if empty, the service selector does not match the pod labels or pods are not ready. Check that the service targetPort matches the container port. Test from a debug pod with curl svc-name.namespace:port to separate DNS from app issues. Then check network policies blocking traffic and, for external access, the ingress rules and load balancer health checks.

The kernel OOM killer keeps killing your application. How do you handle it?

Show answer

Confirm with dmesg or journalctl -k, which shows the OOM message and the killed process. Check whether memory grows over time (a leak) or spikes with load. Set correct heap limits, for example JVM -Xmx below container memory, add swap only as a buffer, and set memory limits so one service cannot starve others. Then profile the app and right-size the instance.

Tell me about a time you migrated an application to Kubernetes.

Show answer

Situation: our monolith and five services ran on hand-managed VMs with frequent config drift. Task: I led the move to a managed Kubernetes cluster. Action: I containerised each service, wrote Helm charts, added probes and resource limits, set up Argo CD, and migrated one low-risk service first with a rollback plan. Result: deployments went from weekly to daily and VM costs dropped about 30 percent.

Users complain the app is slow, but your dashboards look normal. What do you do?

Show answer

Averages can hide pain, so check p95 and p99 latency, not mean. Find out which users, regions or features are affected; it may be one endpoint, a third-party API or client-side slowness. Use traces to follow a slow request and check DNS, CDN and front-end metrics through real user monitoring. Then add the missing signal to dashboards so it is visible next time.

An application logs 'Too many open files'. How do you fix it?

Show answer

The process has hit its file descriptor limit. Check usage with ls /proc/<pid>/fd | wc -l and the limit in /proc/<pid>/limits. Raise it through LimitNOFILE in the systemd unit or /etc/security/limits.conf, then restart. But also look for a leak: sockets or files not closed in code keep growing and will hit any limit. Raising the limit alone only buys time.

The monthly AWS bill suddenly doubled. How do you investigate and fix it?

Show answer

Open Cost Explorer and group by service, account and tag to find the spike, then drill into usage type. Common causes are forgotten instances, unattached EBS volumes, NAT Gateway data transfer, oversized RDS, or logs retained forever. Fix the source, then set AWS Budgets alerts, enforce tagging, right-size using Compute Optimizer, and use Savings Plans or Spot for steady or batch workloads.

How does RBAC work in Kubernetes?

Show answer

RBAC controls who can do what on which resources. A Role defines allowed verbs like get, list or create on resources within a namespace; a ClusterRole does so cluster-wide. RoleBindings and ClusterRoleBindings attach those roles to users, groups or service accounts. Follow least privilege: give each application its own service account with only the permissions it needs.

Explain the Linux boot process briefly.

Show answer

The firmware (BIOS or UEFI) runs hardware checks and loads the bootloader, usually GRUB. GRUB loads the kernel and the initramfs, which contains drivers needed to mount the real root file system. The kernel then starts PID 1, which is systemd on most modern distributions. systemd mounts file systems and starts services in parallel according to unit dependencies and the default target.

You suddenly cannot SSH into a Linux server. What do you check?

Show answer

Check reachability with ping and whether port 22 is open with nc -zv host 22. Then look at security groups, firewall rules and whether the sshd service is running, using the cloud console or serial access. Run ssh -vvv to see whether it fails at key exchange or authentication. Common causes are wrong key permissions, a full disk, a changed IP, or fail2ban blocking you.

Explain liveness, readiness and startup probes.

KubernetesMedium

Show answer

A liveness probe checks whether the container is healthy; if it fails, the kubelet restarts it. A readiness probe checks whether it can serve traffic; if it fails, the pod is removed from service endpoints but not restarted. A startup probe gives slow-starting apps time to boot and holds off the other probes until it succeeds. Wrong liveness probes can cause restart loops.

You suspect a memory leak in a production service. How do you confirm and handle it?

Show answer

Look at memory graphs over days: a leak shows steady growth that drops only on restart, often with OOM kills. Correlate the start with a deployment. Take heap dumps or profiles, for example jmap for Java or pprof for Go, and compare them over time. As a short-term fix, set memory limits and schedule restarts or roll back, then fix the code and add a memory alert.

During rolling updates some user requests fail. How do you get true zero-downtime deployments?

Show answer

Add a proper readiness probe so new pods get traffic only when ready. Set maxUnavailable to 0 and maxSurge to 1 or more. Handle SIGTERM gracefully and add a short preStop sleep so endpoints update before the old pod stops accepting connections, with terminationGracePeriodSeconds long enough to drain requests. Add a PodDisruptionBudget for node maintenance too.

Tell me about a time you reduced cloud costs.

AWS Cloud for DevOpsMedium

Show answer

Situation: our dev and test AWS accounts cost almost as much as production. Task: my manager asked me to cut waste without slowing teams. Action: I tagged resources by owner, found idle instances and old snapshots, scheduled non-production instances to stop at night with a Lambda, and moved CI agents to Spot. Result: non-production spend fell about 40 percent within two months.

Logging costs are rising fast because of huge log volume. What do you do?

Show answer

Find the top log sources by volume. Lower verbose debug logging in production, drop health check and noisy access logs, and sample high-volume info logs. Use structured logs to avoid duplicated fields. Set retention tiers: a few days hot, then archive to S3. Turn repeated log lines into metrics where only counts matter, which is far cheaper to store and query.

What is the difference between an Ingress and a Service?

KubernetesMedium

Show answer

A Service exposes a set of pods at layer 4. An Ingress is a layer 7 HTTP routing resource that maps hostnames and paths to services, for example api.example.com to the api service and /admin to another. It needs an ingress controller such as NGINX or a cloud controller to work. One Ingress with one load balancer can serve many services, and it handles TLS termination.

Explain the Prometheus architecture.

Show answer

Prometheus pulls, or scrapes, metrics over HTTP from targets that expose a /metrics endpoint, either apps instrumented with client libraries or exporters such as node exporter. Targets come from static config or service discovery, for example Kubernetes. Data is stored in a local time-series database and queried with PromQL. Alert rules fire to Alertmanager, which groups, silences and routes notifications. Grafana visualises the data.

What is distributed tracing and how does OpenTelemetry help?

Show answer

Distributed tracing follows one request across many services. Each operation is a span with timing and attributes, and spans share a trace ID passed in request headers, so tools like Jaeger or Tempo can show where time is spent. OpenTelemetry is the vendor-neutral standard and SDK for producing traces, metrics and logs, so you instrument once and send data to any backend.

An e-commerce site expects 10 times normal traffic during a festive sale. How do you prepare?

Show answer

Load test beforehand to find bottlenecks. Pre-scale Auto Scaling groups or Kubernetes nodes with scheduled scaling, since reactive scaling may be too slow. Cache aggressively with CloudFront and Redis, add read replicas, and check service quotas. Use queues to absorb spikes in order processing, set up dashboards and alerts, freeze risky deployments, and keep an on-call war room ready.

How would you plan disaster recovery for a complete AWS region failure?

Show answer

Start from business RTO and RPO. Options range from backup and restore, which is cheapest and slowest, to pilot light, warm standby and multi-region active-active. Replicate data with S3 cross-region replication, Aurora Global Database or DynamoDB global tables, keep infrastructure in Terraform so it can be rebuilt, use Route 53 failover routing and rehearse failover drills.

df shows the disk is 100 percent full but du does not find large files. What is happening?

Show answer

Usually a large file was deleted while a process still holds it open, so the space is not released. Run lsof +L1 or lsof | grep deleted to find the process and file. Restarting or reloading that process frees the space; if you cannot restart, truncate the file through /proc/<pid>/fd/<n>. The fix is proper log rotation with copytruncate or a reopen signal.

Write a bash script that checks whether a service is running and restarts it if not.

Show answer

#!/bin/bash set -u SERVICE="nginx" if ! systemctl is-active --quiet "$SERVICE"; then echo "$(date) $SERVICE down, restarting" >> /var/log/svc-check.log systemctl restart "$SERVICE" if ! systemctl is-active --quiet "$SERVICE"; then echo "$SERVICE failed to start" | mail -s "ALERT" ops@example.com fi fi Run it from cron every few minutes; a real setup would rely on systemd Restart= plus monitoring.

What are resource requests and limits, and what are QoS classes?

Show answer

Requests are what the scheduler reserves for a container when placing it on a node; limits are the maximum it may use. Exceeding a CPU limit causes throttling, while exceeding a memory limit gets the container OOM killed. Pods are Guaranteed when requests equal limits, Burstable when requests are lower, and BestEffort with none; BestEffort pods are evicted first under node pressure.

How do you assume an IAM role in another account using the AWS CLI?

AWS Cloud for DevOpsMedium

Show answer

# ~/.aws/config [profile prod-admin] role_arn = arn:aws:iam::123456789012:role/DevOpsAdmin source_profile = default region = ap-south-1 aws s3 ls --profile prod-admin # manual way aws sts assume-role \ --role-arn arn:aws:iam::123456789012:role/DevOpsAdmin \ --role-session-name ravi-session The role's trust policy must allow your account. Temporary credentials expire, usually after an hour.

A developer accidentally pushed an AWS access key to a public repository. What do you do?

Git and Version ControlHard

Show answer

Treat the key as compromised immediately: deactivate and rotate it in IAM, then check CloudTrail for misuse. Removing it from history comes second, using git filter-repo or BFG and a force push, and asking everyone to re-clone. Then prevent repeats with secret scanning in CI, pre-commit hooks like gitleaks and storing secrets in a vault or secrets manager instead of code.

How does the Horizontal Pod Autoscaler work?

KubernetesMedium

Show answer

The HPA periodically reads metrics from the metrics server or a custom metrics adapter and changes the replica count of a Deployment or StatefulSet to keep a target, for example 70 percent average CPU. It calculates desired replicas from current usage divided by the target, within min and max bounds, with stabilisation windows to avoid flapping. CPU-based HPA needs resource requests set.

How do you work with developers who resist new DevOps processes?