Why I Always Ask About Failed Automation Scripts, Not Just Wins

Interviewer: Tell me about a script that failed.

Candidate: Once my sales report script gave an error.
The data team had changed the file. I told them and
they fixed it. Then the script worked again.

Interviewer: How did you find out it failed?

Candidate: My manager said the report had not come.

Interviewer: Tell me about a script that failed.

Candidate: My daily sales script ran without errors
but produced a report with zero revenue for one region.
The source team had renamed 'Amount' to 'Net_Amount'.
My code used .get() with a default of 0, so it hid the
problem instead of failing.

I noticed because the row count was normal but the total
was far below the previous day. I changed the script to
check required columns at the start and stop with a clear
message, and to log the daily total so big drops stand out.

import pandas as pd

df = pd.read_csv('sales_today.csv')

# Fragile: if 'Amount' is missing, revenue becomes 0
# for every row and the script still 'succeeds'
if 'Amount' in df.columns:
    df['revenue'] = df['Amount']
else:
    df['revenue'] = 0

# Fragile: bad dates become NaT and are silently dropped
df['date'] = pd.to_datetime(df['date'], errors='coerce')
df = df.dropna(subset=['date'])

df.groupby('region')['revenue'].sum().to_excel('report.xlsx')

import logging, sys
import pandas as pd

logging.basicConfig(filename='sales_report.log',
                    level=logging.INFO,
                    format='%(asctime)s %(levelname)s %(message)s')

REQUIRED = {'date', 'region', 'Amount'}

df = pd.read_csv('sales_today.csv')
missing = REQUIRED - set(df.columns)
if missing:
    logging.error('Missing columns: %s', missing)
    sys.exit(1)

parsed = pd.to_datetime(df['date'], format='%Y-%m-%d', errors='coerce')
bad = parsed.isna().sum()
if bad:
    logging.warning('%d rows had unreadable dates', bad)
df['date'] = parsed
df = df.dropna(subset=['date'])

total = df['Amount'].sum()
logging.info('rows=%d total=%.2f', len(df), total)

df.groupby('region')['Amount'].sum().to_excel('report.xlsx')

FAILURE STORY CARD

Script:      daily sales report (Python, scheduled)
Symptom:     one region showed zero revenue
Noticed by:  me, from the daily total in the log
Investigated: compared today's columns with yesterday's
             -> 'Amount' renamed to 'Net_Amount'
Root cause:  my code defaulted missing column to 0
Fix:         handled new name, reran, sent corrected report
Prevention:  required-column check, exit with error,
             daily rows and total logged
Lesson:      fail loudly; never hide missing input

Q: Why did you use a default in the first place?
A: I wanted the script to never crash. I learned that
   a crash is better than a wrong report.

Q: How would you know if it failed overnight?
A: The scheduler task returns a non-zero exit code,
   and I added an email alert on failure. I also check
   the log each morning before the report goes out.

Q: What would you do differently now?
A: Agree a file format with the source team and add
   the column check from day one.

SCRIPT POSTMORTEM

Script name: ______________________
Date of failure: __________________
What the user saw: ________________
How it was noticed: alert / log / user / me
Time until noticed: _______________
Root cause: _______________________
Why my code allowed it: ___________
Immediate fix: ____________________
Prevention added: _________________
One-line lesson: __________________

BEFORE SCHEDULING ANY SCRIPT

[ ] Required columns checked at the start
[ ] Input file date and row count checked
[ ] Dates parsed with an explicit format
[ ] Bad or dropped rows counted and logged
[ ] Key totals logged each run
[ ] Script exits with non-zero code on error
[ ] Failure triggers an alert or email
[ ] Dependencies listed in requirements.txt

BREAK YOUR OWN SCRIPT (to find stories)

1. Rename one required column in the input file.
2. Delete the input file before the run.
3. Change dates to a different format.
4. Add duplicate rows.
5. Run from a different folder.

For each: did the script fail loudly, or produce
a wrong result quietly? Fix the quiet ones.