Monitoring, Logging and SRE Interview Questions for DevOps

Show answer

Monitoring tracks known metrics and alerts when they cross thresholds, answering whether something is wrong. Observability is the ability to understand any internal state from outside data, answering why, even for problems you did not predict. It rests on three pillars: metrics for trends, logs for detailed events and traces for following one request across services. Good systems need both.

What is the ELK or EFK stack?

Show answer

ELK is Elasticsearch, Logstash and Kibana: Logstash or Beats collect and parse logs, Elasticsearch stores and indexes them, and Kibana provides search and dashboards. EFK replaces Logstash with Fluentd or Fluent Bit, which is common in Kubernetes as a DaemonSet. Centralised logging lets you search across all servers and containers in one place instead of SSH-ing into each.

What is Site Reliability Engineering, and what is toil?

Show answer

SRE, which began at a large search company, applies software engineering to operations: teams own reliability through SLOs, error budgets, automation, incident management and capacity planning. Toil is manual, repetitive, automatable work that grows with scale and adds no lasting value, such as manually restarting services. SRE teams aim to keep toil below about half their time and automate the rest.

You get an alert that the company website is down. What are your first steps?

Show answer

Acknowledge the alert so others know you are on it. Check whether it is really down from outside with curl or an uptime checker, and whether it is all users or one region. Look at recent deployments or changes, then load balancer health, application logs and the database. If a recent change caused it, roll back first. Inform the team and escalate if you cannot fix it quickly.

How do you configure Prometheus to scrape a target?

Show answer

# prometheus.yml global: scrape_interval: 15s scrape_configs: - job_name: node static_configs: - targets: ['10.0.1.10:9100', '10.0.1.11:9100'] - job_name: api metrics_path: /metrics static_configs: - targets: ['api.internal:8080'] # validate and reload promtool check config prometheus.yml curl -X POST http://localhost:9090/-/reload Port 9100 is the node exporter default. In Kubernetes, use service discovery or ServiceMonitors instead.

How much downtime does a 99.9 percent availability SLO allow?

Show answer

Allowed downtime = (1 - SLO) x period 99.9 percent over 30 days: 0.001 x 30 x 24 x 60 = 43.2 minutes per month 99.9 percent over a year = about 8.76 hours 99.95 percent per month = about 21.6 minutes 99.99 percent per month = about 4.3 minutes Each extra nine cuts the budget tenfold and usually needs much more redundancy and automation, so pick an SLO that matches real user needs.

How do you quickly search logs for errors from the last hour on a server or in Kubernetes?

Show answer

# systemd service journalctl -u api --since "1 hour ago" -p err # plain log files grep -i "error" /var/log/app/app.log | tail -50 grep -c "timeout" /var/log/app/app.log # Kubernetes kubectl logs deploy/api --since=1h | grep -i error kubectl logs -l app=api --since=1h --prefix | grep 500 For many hosts, search the central logging tool instead, filtering by service, level and time.

How do you handle the stress of on-call duty?

Show answer

I prepare before my shift by reading recent incidents and making sure runbooks and access work. During an incident I follow the runbook, stay calm and escalate early instead of struggling alone, because quick recovery matters more than solving it myself. Afterwards I note what was confusing and improve the docs or alerts, so each rotation gets easier for everyone.

What are the Prometheus metric types?

Show answer

A counter only increases, such as total requests, and you use rate() on it. A gauge goes up and down, such as memory in use or queue length. A histogram counts observations into configurable buckets, such as request duration, and lets you compute percentiles with histogram_quantile across instances. A summary computes quantiles on the client side but cannot be aggregated across instances.

What are the four golden signals, and the RED and USE methods?

Show answer

The four golden signals are latency, traffic, errors and saturation. RED, for request-driven services, tracks rate, errors and duration. USE, for resources such as CPU, disks and network, tracks utilisation, saturation and errors. RED tells you whether users are hurting; USE helps find which resource is the bottleneck. Together they give a solid starting set of dashboards and alerts.

The on-call team gets dozens of alerts every night and is burning out. How do you fix it?

Show answer

Collect alert data for a few weeks: which fired, how often, and whether anyone acted. Delete or downgrade non-actionable alerts, tune thresholds and add for durations, and group duplicates in Alertmanager. Fix the recurring root causes behind the top alerts and automate common fixes. Move to SLO-based alerts and make sure rotations are fair with follow-up time after bad nights.

Write PromQL queries for request rate, error rate and p95 latency.

Show answer

# requests per second per service over 5 minutes sum by (service) (rate(http_requests_total[5m])) # error ratio (5xx) sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m])) # p95 latency from a histogram histogram_quantile(0.95, sum by (le) (rate(http_request_duration_seconds_bucket[5m]))) rate() works on counters and handles resets; the le label holds histogram bucket bounds.

Write a Prometheus alerting rule for a high error rate.

Show answer

groups: - name: api-alerts rules: - alert: HighErrorRate expr: | sum(rate(http_requests_total{job="api",status=~"5.."}[5m])) / sum(rate(http_requests_total{job="api"}[5m])) > 0.05 for: 10m labels: severity: page annotations: summary: "API error rate above 5 percent" runbook: "https://wiki.example.com/runbooks/api-errors" for: 10m ignores short spikes; Alertmanager routes by the severity label.

Tell me about a blameless post-mortem you were part of.

Show answer

Situation: a certificate expired and our API was down for 40 minutes. Task: I wrote the post-mortem. Action: I built a timeline from logs and chat, and we focused on why the system allowed it rather than who forgot: renewal was manual and nothing alerted. Result: we automated renewal with cert-manager, added expiry alerts 21 days ahead, and tracked actions to closure.

How do you communicate with stakeholders during an outage?

Show answer

I post a short first update quickly saying what is affected, that we are working on it and when the next update will come. Then I update at a fixed interval, such as every 20 minutes, in plain language without guessing at causes. I keep technical detail in the engineering channel. After resolution, I send a summary with impact, cause and next steps.

Explain the Prometheus architecture.

Show answer

Prometheus pulls, or scrapes, metrics over HTTP from targets that expose a /metrics endpoint, either apps instrumented with client libraries or exporters such as node exporter. Targets come from static config or service discovery, for example Kubernetes. Data is stored in a local time-series database and queried with PromQL. Alert rules fire to Alertmanager, which groups, silences and routes notifications. Grafana visualises the data.

Explain SLI, SLO, SLA and error budget.

Show answer

An SLI is a measured indicator, for example the share of requests served under 300 ms. An SLO is the internal target for it, such as 99.9 percent over 30 days. An SLA is the external contract with penalties, usually looser than the SLO. The error budget is the allowed unreliability, 0.1 percent here; while budget remains teams ship features, and when it runs out they focus on reliability.

What is distributed tracing and how does OpenTelemetry help?

Show answer

Distributed tracing follows one request across many services. Each operation is a span with timing and attributes, and spans share a trace ID passed in request headers, so tools like Jaeger or Tempo can show where time is spent. OpenTelemetry is the vendor-neutral standard and SDK for producing traces, metrics and logs, so you instrument once and send data to any backend.

What makes a good alert, and how do you avoid alert fatigue?

Show answer

A good alert is actionable, urgent and tied to user impact, such as an SLO burn rate, not just high CPU. Each page should have a runbook link and a clear owner. Reduce noise by removing alerts nobody acts on, using severity levels so non-urgent issues create tickets, grouping related alerts, and using for durations to ignore short spikes. Review alerts after each on-call week.

Users complain the app is slow, but your dashboards look normal. What do you do?

Show answer

Averages can hide pain, so check p95 and p99 latency, not mean. Find out which users, regions or features are affected; it may be one endpoint, a third-party API or client-side slowness. Use traces to follow a slow request and check DNS, CDN and front-end metrics through real user monitoring. Then add the missing signal to dashboards so it is visible next time.

You suspect a memory leak in a production service. How do you confirm and handle it?

Show answer

Look at memory graphs over days: a leak shows steady growth that drops only on restart, often with OOM kills. Correlate the start with a deployment. Take heap dumps or profiles, for example jmap for Java or pprof for Go, and compare them over time. As a short-term fix, set memory limits and schedule restarts or roll back, then fix the code and add a memory alert.

You are the incident commander for a major outage. What do you do?

Show answer

Declare the incident, open a channel and bridge, and assign roles: someone to investigate, someone to communicate and a scribe. My job is to coordinate, not to debug. Prioritise mitigation over root cause, such as rollback, failover or feature flags. Post regular status updates to stakeholders. After recovery, confirm it is stable, close it and schedule a blameless post-mortem with action items.

Logging costs are rising fast because of huge log volume. What do you do?

Show answer

Find the top log sources by volume. Lower verbose debug logging in production, drop health check and noisy access logs, and sample high-volume info logs. Use structured logs to avoid duplicated fields. Set retention tiers: a few days hot, then archive to S3. Turn repeated log lines into metrics where only counts matter, which is far cheaper to store and query.

Your service has used its entire error budget halfway through the quarter. What happens next?

Show answer

Following the agreed error budget policy, the team pauses risky feature launches and focuses on reliability work: fixing the top causes of failures from post-mortems, adding tests, improving rollbacks and canaries. Product and engineering leads review the plan together. Once the budget recovers in the rolling window, normal releases resume. If the SLO is unrealistic, revisit it with data.

Tell me about a time you improved the reliability of a system.