Why Confidence Without Pandas Skills Still Fails

import pandas as pd

# Confident claim: 'I build results with df.append.'
result = pd.DataFrame()
for i in range(3):
    row = pd.DataFrame({'id': [i], 'sq': [i * i]})
    # result = result.append(row)   # AttributeError in pandas 2.x

# Correct pattern: collect, then concat once
rows = []
for i in range(3):
    rows.append({'id': i, 'sq': i * i})
result = pd.DataFrame(rows)
print(result)

# Or with frames:
# result = pd.concat(list_of_frames, ignore_index=True)

import pandas as pd
import numpy as np

df = pd.DataFrame({'a': [1, np.nan, 3]})

out = df.dropna(inplace=True)
print(out)        # None  <- a common surprise

# Clearer and chain-friendly:
df = pd.DataFrame({'a': [1, np.nan, 3], 'b': ['x', 'y', None]})
clean = (df.dropna(subset=['a'])
           .rename(columns={'a': 'amount'})
           .reset_index(drop=True))
print(clean)

import pandas as pd
import numpy as np

df = pd.DataFrame({'amount': np.random.randint(100, 10000, size=200_000)})

# Slow: Python loop over rows
# labels = []
# for _, row in df.iterrows():
#     labels.append('high' if row['amount'] > 5000 else 'normal')
# df['label'] = labels

# Fast: one vectorised operation
df['label'] = np.where(df['amount'] > 5000, 'high', 'normal')

# Several bands with pd.cut
df['band'] = pd.cut(df['amount'],
                    bins=[0, 1000, 5000, np.inf],
                    labels=['low', 'mid', 'high'])
print(df['band'].value_counts())

# In Jupyter, compare approaches with %timeit

%timeit df['amount'] * 1.18
%timeit df['amount'].apply(lambda x: x * 1.18)

# Outside Jupyter:
import time
start = time.perf_counter()
_ = df['amount'].apply(lambda x: x * 1.18)
print('apply:', time.perf_counter() - start)

start = time.perf_counter()
_ = df['amount'] * 1.18
print('vectorised:', time.perf_counter() - start)

# Run this once on your own machine. Then your
# confidence about speed is based on evidence.

Q: How do you handle missing values in Pandas?

'Very easy. I always use fillna(0) on the whole
DataFrame. It is the best practice and it works
for every column.'

Problems:
- Filling a text column with 0 mixes types
- 0 is a real value for price or quantity
- Ignores why data is missing
- 'Always' and 'every' signal no real experience

Q: How do you handle missing values in Pandas?

'First I measure them: df.isna().sum() per column.
Then it depends on the column.'

df.isna().sum()

# numeric, few gaps: fill with median
df['age'] = df['age'].fillna(df['age'].median())

# categorical: explicit label
df['city'] = df['city'].fillna('Unknown')

# key column missing: drop those rows
df = df.dropna(subset=['customer_id'])

'I am less sure about time-series cases. There I
would look at ffill, but I would check with the
business whether carrying values forward makes sense.'

RUN EACH ONE AND WRITE THE RESULT

[ ] np.nan == np.nan  ->  ____
[ ] df.dropna(inplace=True) returns ____
[ ] hasattr(pd.DataFrame, 'append')  ->  ____
[ ] Time: df['x'] * 2  vs  df['x'].apply(lambda v: v * 2)
[ ] df.groupby('k')['v'].mean() type  ->  ____
[ ] df.merge(right, on='k').shape when right
    has duplicate keys  ->  ____

Any surprise = a claim you should stop making
with certainty.

FOR EACH PANDAS CLAIM IN MY PROJECT

Claim (e.g. 'cleaned 3 messy columns'):
______________________________________
Code cell that proves it: ______
Can I re-run it today?        Y / N
Can I explain each line?      Y / N
One thing I would improve:    ______

Confidence level to use:
( ) certain   ( ) know idea   ( ) read only

PHRASES THAT KEEP CONFIDENCE HONEST

'I have used this a lot. Here is how...'
'The approach is X. I would double-check the
 argument name in the docs.'
'I have read about it but not used it in a project.'
'In older versions it worked like X; I believe
 pandas 2 changed that.'
'Let me run a quick check rather than guess.'

AVOID
'Always', 'never', 'best practice' without a reason