The Data Scientist Answer That Instantly Builds My Confidence

from sklearn.dummy import DummyClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score, roc_auc_score

# X_train, X_test, y_train, y_test already split
baseline = DummyClassifier(strategy="most_frequent").fit(X_train, y_train)
simple = LogisticRegression(max_iter=1000).fit(X_train, y_train)

for name, model in [("baseline", baseline), ("logistic", simple), ("final", final_model)]:
    proba = model.predict_proba(X_test)[:, 1]
    pred = model.predict(X_test)
    print(f"{name:10s} accuracy={accuracy_score(y_test, pred):.3f} "
          f"auc={roc_auc_score(y_test, proba):.3f}")

# The dummy model's AUC is 0.5 because every row gets the same score.

Q: How good was your model?

A: "On the test set the final model had an AUC of 0.78.

 For context, the classes were imbalanced, so a model
 that predicts 'no churn' for everyone already gets very
 high accuracy. That is why I did not report accuracy.

 A plain logistic regression got 0.70 AUC. So the
 gradient boosting model added a real but modest lift,
 and I kept the logistic regression as a fallback because
 it is easier to explain."

from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score

# LEAKY: scaler sees every row, including future test folds
# X_scaled = StandardScaler().fit_transform(X)
# cross_val_score(LogisticRegression(), X_scaled, y, cv=5)

# SAFE: imputer and scaler are refitted inside each training fold
pipe = make_pipeline(
    SimpleImputer(strategy="median"),
    StandardScaler(),
    LogisticRegression(max_iter=1000),
)
scores = cross_val_score(pipe, X, y, cv=5, scoring="roc_auc")
print(scores.mean().round(3), scores.std().round(3))

import pandas as pd
from sklearn.ensemble import RandomForestClassifier

rf = RandomForestClassifier(n_estimators=300, random_state=42)
rf.fit(X_train, y_train)

importance = (pd.Series(rf.feature_importances_, index=X_train.columns)
              .sort_values(ascending=False))
print(importance.head(10))

# Spoken check:
# "If one feature dominates, I ask when it is recorded.
#  In one project, 'days_since_last_support_ticket' was
#  top. Tickets were often raised to cancel, so the
#  feature partly recorded the outcome. I rebuilt it to
#  count only tickets before the prediction date."

import pandas as pd
from sklearn.model_selection import TimeSeriesSplit, cross_val_score

# df has one row per customer-month, with a snapshot_date column
df = df.sort_values("snapshot_date")
X = df[feature_cols]
y = df["churned_next_month"]

tscv = TimeSeriesSplit(n_splits=5)
scores = cross_val_score(pipe, X, y, cv=tscv, scoring="roc_auc")
print(scores.round(3))

# Each fold trains on earlier rows and tests on later rows.
# Falling scores across folds can signal drift over time.

from sklearn.model_selection import GroupKFold, cross_val_score

# Several visits per patient: keep each patient in one fold only
gkf = GroupKFold(n_splits=5)
scores = cross_val_score(
    pipe, X, y,
    groups=df["patient_id"],
    cv=gkf,
    scoring="roc_auc",
)
print(scores.mean().round(3))

# Spoken: "With a random split the same patient appeared in
# train and test, and the model partly memorised patients.
# Grouping by patient gave a lower but honest score."

"On [split description], the final model scored
 [metric] = [value].

 The baseline was [baseline type] at [value], and a simple
 [simple model] reached [value].

 I chose [metric] because [reason, e.g. imbalance].

 To avoid leakage, I [removed X / built features only from
 data before the cut-off / fitted preprocessing in a
 Pipeline].

 The main limitation is [limitation]."

FOR EACH FEATURE, ANSWER:

Feature: __________________
[ ] Recorded before the prediction moment?
[ ] Available in production at prediction time?
[ ] Not a rewording of the target?
[ ] Not derived using statistics from the full dataset?

FOR THE WHOLE PROJECT:
[ ] Preprocessing fitted inside a Pipeline
[ ] No entity appears in both train and test
[ ] Test set touched only once, at the end

SAY THE ANSWER ALOUD FOR EACH

1. Your accuracy is 96%. Is that good?
2. What baseline did you compare against?
3. Which feature was most important? When is it recorded?
4. Did you scale before or after splitting?
5. Why a random split for time-ordered data?
6. Could the same customer be in train and test?
7. How many times did you look at the test set?
8. What would make you distrust your own result?