Why I Always Ask Candidates About Failed Code, Not Just Wins

STRONG ANSWER — FIRST 90 SECONDS:

Interviewer: "Tell me about code that
failed."

Candidate: "Sure. The retry loop took down
the payment service for 40 minutes, and the
retry itself caused it.

The impact: 12K payments were delayed. The
error rate went from 0.1% to 40% because the
retry loop was hammering the downstream.

The root cause: the retry had exponential
backoff but no jitter. Every failed consumer
retried at the same second — at 500 failures
a second, that's 500 synchronized retries
every 2, 4, 8 seconds. The downstream got a
spike instead of a trickle. It was a
self-inflicted DDoS.

The fix: I added full jitter to the backoff —
randomize the delay between 0 and the
exponential value — and a circuit breaker
that trips after 20 consecutive failures and
sheds load for 30 seconds.

The prevention: the circuit breaker is the
key — it stops the retry loop from becoming
an outage when the downstream is already
sick.

The cost: full jitter means some retries
happen almost immediately — a bit more noise
on the downstream — and the circuit breaker
means some legitimate retries get shed during
a real outage. I accepted both because a
40-minute outage is worse than a few shed
retries.

The lesson: retries are a distributed-systems
problem, not a code problem. Every retry
needs jitter and a budget."

What the interviewer thinks in 90 seconds:
- "They named the code and the impact." ✅
- "They named the root cause: no jitter." ✅
- "They named the fix and the prevention." ✅
- "They named the cost: shed retries." ✅
- "They understand their own code." ✅

Signal: strong. The rest of the interview
confirms it.

WEAK ANSWER — FIRST 90 SECONDS:

Interviewer: "Tell me about code that
failed."

Candidate: "Yeah, there was a build that
failed once. I re-ran it and it worked. It
was probably a flaky test or something. I
don't really remember the details."

What the interviewer thinks in 90 seconds:
- "No code named." ❌
- "No impact — did anyone notice?" ❌
- "No root cause — 'flaky test' is not a
  cause." ❌
- "No fix, no prevention." ❌
- "They didn't understand the failure well
  enough to remember it." ❌
- "I'll ask a follow-up, but I already
  doubt the wins." ⚠️

What happens next:
- The interviewer asks "What was the flaky
  test?"
- The candidate says "I think it was a
  timing thing."
- The interviewer asks "How did you know
  it wasn't a real bug?"
- The candidate pauses.

Signal: weak. The candidate's wins now look
less impressive, because they can't explain
a failure.

Why it failed:
❌ No specific code
❌ No impact named
❌ No root cause
❌ No fix or prevention
❌ No lesson
❌ The first 90 seconds signaled "I ran
  code," not "I own code"

ROOT CAUSE ANSWER — WHAT DECIDES:

Interviewer: "How did you find the root
cause?"

WHAT THE CANDIDATE SAYS:
"I found it by reading the metrics, then
working backwards from the error rate.

Step one: I checked the downstream latency.
The payment service was at 4 seconds p99 —
normally 80ms. So the downstream was slow
first, then the retries made it worse.

Step two: I checked the retry logs. Every
retry cluster landed at 2s, 4s, and 8s
intervals — synchronized. That's the
signature of exponential backoff without
jitter.

Step three: I confirmed the math. 500
failures a second at peak, with a 2-second
base delay — that's 500 synchronized retries
every 2 seconds. The downstream couldn't
absorb the spike, so it stayed slow, so more
retries fired. A feedback loop.

The root cause: my retry design. The
downstream was sick, but the retries made it
a 40-minute outage instead of a 2-minute
blip.

The fix: full jitter — randomize the delay
between 0 and the exponential value — and a
circuit breaker.

The trade-off: full jitter means some
retries happen almost immediately, which adds
noise. I accepted that because the noise is
infinitesimal compared to a synchronized
spike.

The lesson: a retry without jitter is a
distributed-systems bug, not a code bug. I
now check every retry for jitter and a
budget."

What the interviewer thinks:
- "They checked metrics before guessing." ✅
- "They found the synchronized retry
  signature." ✅
- "They did the math: 500 at 2s." ✅
- "The root cause is a design flaw, not a
  data issue." ✅
- "The cost of the fix is named." ✅

Signal: confirmed judgment.

Why it decides:
✅ Metrics narrowed the scope
✅ The synchronized retry signature is
  specific
✅ The math is concrete: 500 at 2s
✅ Root cause is a design flaw
✅ The cost of the fix is named

PREVENTION ANSWER — WHAT DECIDES:

Interviewer: "How did you make sure it
didn't happen again?"

WHAT THE CANDIDATE SAYS:
"Three layers, and each catches a different
failure.

Layer one: full jitter on every retry. The
delay is randomized between 0 and the
exponential value, so 500 retries spread over
the window instead of firing together. That
breaks the synchronization.

Layer two: a circuit breaker. After 20
consecutive failures, it trips and sheds
load for 30 seconds. During that window,
retries go to a dead-letter queue instead of
hammering the downstream. That stops the
feedback loop even if the jitter isn't enough.

Layer three: a retry budget. Each request
gets at most 10% of its original timeout as
retry time. If the budget is spent, the
request fails fast instead of retrying
forever. That caps the blast radius.

The cost: full jitter adds noise — some
retries fire almost immediately — and the
circuit breaker sheds legitimate retries
during a real outage. The retry budget means
some requests fail when they could have
succeeded with more retries. I accepted all
three because a 40-minute outage is worse
than shed retries.

The result: the payment service has had zero
retry-induced outages in 18 months, and the
circuit breaker has tripped twice — both
times during real downstream outages, which is
exactly what it's for.

The lesson: retries need three things —
jitter, a budget, and a breaker. Any one
alone is insufficient."

What the interviewer thinks:
- "Three layers, each with a purpose." ✅
- "The checks target different failures." ✅
- "The cost is quantified." ✅
- "The circuit breaker has evidence: two
  trips." ✅
- "The lesson is a reusable rule." ✅

Signal: confirmed judgment.

Why it decides:
✅ Three layers, not one
✅ Each layer targets a failure class
✅ The cost is quantified
✅ The breaker has evidence
✅ The lesson is a reusable rule

MEMORY LEAK — STRONG VS. WEAK SIGNAL:

Scenario: "Tell me about code that failed in
production."

WEAK SIGNAL:
"One time the service ran out of memory. I
restarted it and it was fine. I think it was
a leak somewhere."
What the interviewer thinks:
- "No root cause — 'somewhere' is not a
  cause."
- "No prevention — would it happen again?"
- "They restarted and moved on."
- Signal: weak.

STRONG SIGNAL:
"The service ran out of memory every 6 hours,
and restarting was a band-aid. I spent a day
finding the real cause.

The symptom: the heap grew steadily and the
pod OOM-killed every 6 hours. Restarting
fixed it for 6 hours, which is why it was
hard to prioritize.

The root cause: an unclosed Stream from a
JPA query. The code opened a Stream on a
query result and never closed it, so the
underlying database connection leaked. The
thread dump showed 12K open connections —
the pool was supposed to be 50.

The fix: try-with-resources on the Stream,
so it closes the connection when the block
exits.

The prevention: I added a leak detector to
the staging environment that fails the build
if the connection count grows monotonically
over 10 minutes. It caught two more leaks in
a month.

The cost: the leak detector adds 10 minutes
to the staging build — the run has to be long
enough to see growth. I accepted that because
an OOM every 6 hours pages someone at 3 AM.

The lesson: 'restart fixes it' is a symptom,
not a solution. Every OOM has a leak; find
it."

What the interviewer thinks:
- "They named the code and the symptom." ✅
- "The root cause is a leaked Stream." ✅
- "The thread dump is evidence." ✅
- "The fix is try-with-resources." ✅
- "The prevention is a leak detector." ✅
- Signal: confirmed judgment.

Why it decides:
✅ Symptom is specific: 6-hour OOM
✅ Root cause is a leaked Stream
✅ Thread dump is evidence: 12K
✅ Fix is try-with-resources
✅ Prevention is a leak detector

INTERVIEW FOLLOW-UP — STRONG VS. WEAK
SIGNAL:

Scenario: "How did you know the fix worked?"

WEAK SIGNAL:
"I deployed it and the service stopped
crashing. It's been running for a week."
What the interviewer thinks:
- "A week isn't evidence against a 6-hour
  leak."
- "No measurement."
- Signal: weak.

STRONG SIGNAL:
"Three checks, and I can show the numbers.

Check one: the connection count. Before the
fix, it grew from 10 to 12K over 6 hours.
After the fix, it stays flat at 12 connections
for 72 hours. I watched it for 3 days because
the leak cycle was 6 hours — a 1-day check
wouldn't prove anything.

Check two: the heap. Before the fix, the heap
grew 200MB an hour. After the fix, it's flat
at 400MB with normal GC sawtooth. I compared
the heap graph before and after.

Check three: the leak detector. I ran it
against the fixed version for 72 hours in
staging — it passed. And I ran it against the
old version to confirm it fails — it caught
the leak in 90 minutes. That proves the
detector actually works, not just that it
passed.

The cost: the 72-hour watch was a real
investment — I set up an alert and checked
twice a day. I accepted that because a 6-hour
leak needs more than a 1-day proof.

The lesson: match the verification window to
the failure cycle. A 6-hour leak needs a
72-hour test, not a 24-hour one."

What the interviewer thinks:
- "Three checks with numbers." ✅
- "The verification window matches the
  failure cycle." ✅
- "The detector was tested against the old
  version." ✅
- "The 72-hour investment is named." ✅
- "The lesson is a reusable rule." ✅
- Signal: confirmed judgment.

Why it decides:
✅ Connection count is measured
✅ Heap is compared before/after
✅ Detector tested both ways
✅ The cost of verification is named
✅ The lesson is a rule

Failed-Code Story Checklist:

FIRST 90 SECONDS:
☐ Name the code and what it was supposed to
  do
☐ Name the impact: payments delayed,
  customers affected, revenue lost
☐ Name the root cause, not the blame
☐ Avoid "it was a flaky test" — name the
  actual cause

WHAT DECIDES:
☐ Root cause is a design flaw, a race
  condition, a leak, or a definition error
☐ The fix addresses the root cause, not a
  symptom
☐ The prevention targets the failure class
☐ Name how you knew the fix worked
☐ Name the cost of the fix or the
  prevention

WHAT CONFIRMS:
☐ A reusable lesson or rule
☐ Honest about who found it
☐ The follow-up "how did you know?" is
  answered with evidence
☐ "It's been running for a week" is not
  evidence
☐ The prevention has caught real issues
  since

RED FLAGS:
❌ "I don't remember the details"
❌ No code named
❌ No impact
❌ "It was a flaky test" as the cause
❌ No fix, no prevention
❌ "I re-ran it and it worked"
❌ Blame on the data or the downstream
❌ No lesson

Why this works:
- The first 90 seconds set the signal
- The rest of the story confirms or
  overturns it
- Impact, root cause, and prevention are
  the three things that can't be faked

30-DAY PRACTICE PLAN (FOR CANDIDATES):

Week 1 — Recall the failures
☐ List 5 code failures in the last 2 years
☐ For each, write: the code, the impact,
  the root cause
☐ Pick the 3 with the clearest impact
☐ Record yourself telling each in
  90 seconds
☐ Cut every "I don't remember" and "it
  was flaky"

Week 2 — Root causes
☐ For each story, write the root cause
  in one sentence
☐ Classify it: design flaw, race
  condition, leak, or definition error
☐ Write how you found it: metrics, logs,
  thread dump
☐ Write the fix and why it addresses
  the cause

Week 3 — Prevention and evidence
☐ For each story, write the prevention
  and what it targets
☐ Write how you knew the fix worked
☐ Write one reusable lesson or rule
☐ Add the cost of the fix or prevention

Week 4 — Mock interviews
☐ Run 2 mock interviews with a peer
☐ Ask them to challenge: "How did you
  know?" and "What was the root cause?"
☐ Re-run with stronger evidence
☐ Compare the two recordings

Signs it's working:
✅ You can name the impact without
  pausing
✅ You name a design flaw, not "a flaky
  test"
✅ You name how you found it
✅ You name how you knew the fix worked
✅ You name a reusable lesson
✅ The interviewer asks "what else?"
  instead of "what was the code?"