Why I Always Ask About Failed Notebooks, Not Just Wins

Interview Insights · Data Analytics

Why I Always Ask About Failed Notebooks, Not Just Wins

SECTION 01Why the failed notebook tells me more than the polished one

A polished notebook on GitHub has been cleaned up. Mistakes are removed, dead ends deleted, and the final version looks as if the analysis went in a straight line. That is fine for a portfolio, but it hides what I most want to know: how you behave when the data does not cooperate.

So I ask, 'Tell me about a notebook or analysis that did not work.' The answer shows three things I cannot see in a finished file. First, whether you are honest about mistakes. Second, whether you have the instinct to doubt a result, especially one that looks too good. Third, whether you learned something you now do by default. In real analyst jobs, a large part of the work is finding out why a number is wrong, so the failure story is a preview of how you will work on my team.

Interviewer: Tell me about a notebook that failed.

Candidate: Hmm, I don't think any of my projects
really failed. Sometimes there were errors in the
code, but I searched online and fixed them.

(Syntax errors are not analysis failures.
The interviewer learns nothing about judgement.)

Interviewer: Tell me about a notebook that failed.

Candidate: In my loan default project, my first model
looked almost perfect on the test set. That felt wrong
for such a messy problem, so I checked feature importance.

The top feature was 'recovery_amount', which is only
filled after a loan defaults. It was leaking the answer.
I removed every column created after the loan date,
and the score dropped to a believable level.

Now, before training anything, I write down when each
column becomes known.

The strong answer is not about a dramatic disaster. It is about a small, specific moment of doubt. That is exactly what makes it believable. You do not need a big failure; you need a real one.

SECTION 02Four kinds of notebook failure worth talking about

If you think you have no failure story, you probably have one and did not label it as a failure. Look back through your notebooks for any of the four patterns below. Each one makes a strong interview story because each shows a different skill: doubting results, checking definitions, testing assumptions and accepting a dead end.

The most common and most valuable is leakage, where information that would not be available at prediction time slips into the features. It produces results that look excellent and are worthless. A close second is a wrong denominator, where a rate or percentage was calculated over the wrong base. Both are mistakes almost every analyst makes once. Telling me about yours, and how you caught it, shows I will not have to catch it for you.

from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression

# Mistake 1: a column filled only AFTER default is a feature
features = ['income', 'loan_amount', 'tenure', 'recovery_amount']

# Mistake 2: scaler fitted on ALL rows before splitting,
# so test-set statistics leak into training
X = StandardScaler().fit_transform(loans[features])
y = loans['defaulted']

X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2, random_state=0)
model = LogisticRegression(max_iter=1000).fit(X_tr, y_tr)
print(model.score(X_te, y_te))   # suspiciously high

from sklearn.pipeline import make_pipeline
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score

# Only columns known on the loan start date
features = ['income', 'loan_amount', 'tenure']

# Time-based split: train on older loans, test on newer ones
loans = loans.sort_values('start_date')
cut = int(len(loans) * 0.8)
train, test = loans.iloc[:cut], loans.iloc[cut:]

# Scaler lives inside the pipeline, so it is fitted on train only
model = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000))
model.fit(train[features], train['defaulted'])

proba = model.predict_proba(test[features])[:, 1]
print('AUC:', roc_auc_score(test['defaulted'], proba))

Two separate leaks are fixed here. The obvious one is the post-outcome column. The subtle one is fitting the scaler before the split. Mentioning both in an interview shows you understand that leakage is about information flow, not just one bad column.

SECTION 03Telling a dead-end story without sounding like wasted time

Some failures are not mistakes at all. You had a reasonable hypothesis, tested it properly, and found nothing. Many candidates hide these because they feel unproductive. That is a pity, because a well-handled dead end shows something rare: the discipline to accept an answer you did not want.

Imagine a candidate who expected that customers who used a discount code would come back more often. They compared repeat purchase rates and found a difference, but when they split by customer segment, the difference disappeared. New customers used more discount codes and also returned less, so the overall comparison was mixed up with the segment. Telling that story shows segmentation skill, statistical caution and honesty in one answer.

# Overall repeat rate by discount use
overall = (customers
           .groupby('used_discount')['returned_90d']
           .mean())
print(overall)

# Same comparison inside each customer segment
by_segment = (customers
              .groupby(['segment', 'used_discount'])['returned_90d']
              .mean()
              .unstack())
print(by_segment)

# Share of each segment that used a discount
print(customers.groupby('segment')['used_discount'].mean())

## Result: no clear discount effect

Overall, discount users looked different from others,
but inside each segment (new, regular, loyal) the repeat
rates were close.

The overall gap came from the mix of segments:
segments differ both in how often they use discounts
and in how often they return.

Decision: I did not recommend more discounts based on
this data. A controlled test would be needed to measure
a real effect.

Keep a dead-end cell like this in your notebook instead of deleting the analysis. It is perfect interview material, and in a real job it saves the next analyst from repeating the same test.

When you tell the story, finish on the decision, not the disappointment. 'I recommended a controlled test instead of a discount push' is an outcome, and outcomes are what interviewers remember.

SECTION 04Prepare your failed notebook story

Use these tabs to find, structure and rehearse a failure answer from your own work. Most candidates discover they have at least two usable stories once they look.

Go through your notebooks and ask:

[ ] Did any result ever look too good to be true?
[ ] Did a number disagree with a source or dashboard?
[ ] Did a trend change when I split it by a group?
[ ] Did I test an idea and find no effect?
[ ] Did I redo a section after someone questioned it?
[ ] Did a notebook break when rerun on new data?

Any 'yes' is a candidate story. Pick the most specific.

FAILED NOTEBOOK STORY (60-90 seconds)

Context:    "In my ____ project, I was trying to ____."
The result: "My first result showed ____."
The doubt:  "It looked wrong because ____."
The debug:  "I checked ____ and found ____."
The fix:    "I changed ____, and the result became ____."
The habit:  "Since then, I always ____ before ____."

Rules: one project, one cause, one habit.
No blaming the data. No invented numbers.

Expect one of these after your story:

- How would you catch this earlier next time?
- Did anyone else see the wrong result first?
- What would have happened if you had not noticed?
- How do you check for leakage in a new project?
- What does your notebook look like now in that part?

Prepare a two-sentence answer for each.

SECTION 05Habits that turn failures into strong answers

The way you talk about a failure matters as much as the failure itself.

• Use 'I' for the mistake — Say 'I fitted the scaler on all rows', not 'the scaler was fitted'. Ownership of the mistake makes the fix sound like yours too.
• Name the signal — Explain what first made you doubt the result, such as a score that was too high or a number that did not match the source.
• Keep the emotion light — A short 'that was a useful lesson' is enough. Long descriptions of frustration take time away from the method.
• End with a default habit — Finish with a check you now run in every project. It proves the failure changed how you work.
• Keep old versions — Save the wrong version of a notebook in your repository history, so you can show before and after if the interviewer asks.

SECTION 06Interview Q&A — Failed notebooks

SECTION 07Test yourself — Failed notebooks quiz

SECTION 08Frequently asked questions

SECTION 09Related reads