D. E. Shaw Data Science Interview Questions & Answers

Convolution filters slide over the input to detect local patterns, pooling reduces size, and deeper layers combine edges into shapes and objects. Used mainly for images.

Small anonymous one-line functions, handy as arguments to sort, map, filter or Pandas apply.

The distribution of sample means approaches a normal distribution as the sample size grows (roughly 30+), whatever the population's shape. It lets us build confidence intervals and run hypothesis tests on means.

A range built from sample data that would contain the true parameter in a stated percentage (for example 95%) of repeated samples. It is not the probability that the parameter lies inside this one interval.

During training it randomly switches off a fraction of neurons so the network cannot depend on any single unit, acting as a regulariser against overfitting.

A mutex in CPython that lets only one thread execute Python bytecode at a time. Use multiprocessing for CPU-bound work and threads or asyncio for I/O-bound work.

It tries candidate splits and picks the one that most reduces impurity — Gini or entropy (information gain) for classification, variance / MSE for regression.

It projects data onto orthogonal directions of maximum variance to reduce dimensions, remove correlation and speed up models, at some cost of interpretability.

It applies Bayes' theorem assuming features are independent given the class. The assumption is rarely true but the model is fast and works well for text.

Both measure node impurity; Gini is cheaper to compute and entropy is slightly more sensitive to class balance, with similar trees in practice.

They build a list in one readable expression, for example [x*x for x in data if x > 0]. They are usually faster than an equivalent for-loop with append.

Gradients shrink as they pass through many layers (for example with sigmoid), so early layers barely learn. ReLU, residual connections and good initialisation help.

It normalises layer inputs per mini-batch and then rescales them, stabilising and speeding up training and allowing higher learning rates.

The ROC curve plots true positive rate against false positive rate across thresholds. AUC is the probability that a random positive is ranked above a random negative; 0.5 is chance.

A measure of asymmetry. Right-skewed data has a long tail on the right (mean greater than median); left-skewed is the opposite. Log transforms often reduce right skew.

Discriminative models learn P(y given x) directly (logistic regression, SVM). Generative models learn the joint P(x, y) and can generate data (Naive Bayes, GANs).

Precision = TP / (TP + FP). Recall = TP / (TP + FN). Prefer recall when missing a positive is costly — for example disease screening or fraud detection.

ReLU for hidden layers, sigmoid for binary outputs, softmax for multi-class outputs and tanh when zero-centred outputs help; Leaky ReLU avoids dead neurons.

df.drop_duplicates(subset=[...], keep='first') removes repeated rows; use duplicated() first to inspect what will be dropped.

Parametric tests assume a distribution (usually normal) and use means; non-parametric tests such as Mann-Whitney use ranks and make fewer assumptions.

Putting features on comparable ranges using standardisation or min-max scaling. Distance-based and gradient-based models (KNN, SVM, regression, neural nets) need it; tree models do not.

Regression predicts continuous values such as price; classification predicts discrete categories such as spam or not spam.

A network that compresses inputs into a small latent code and reconstructs them; used for denoising, dimensionality reduction and anomaly detection.

A shallow copy duplicates the outer object but shares nested objects. A deep copy (copy.deepcopy) recursively duplicates everything, so changes to nested items do not leak.

Use grid search, random search or Bayesian optimisation with cross-validation, starting with a coarse random search and narrowing the promising ranges.

Package the model behind an API with versioning, track latency and errors, monitor input drift and prediction quality, and automate retraining with rollback.

This is overfitting. Check for data leakage, simplify the model, add regularization, get more data or augment it, use cross-validation, early stopping and feature selection.

The square root of the variance; it measures how far values typically sit from the mean, in the same units as the data.

int, float, complex, bool, str, list, tuple, set, dict and NoneType. Mutable ones are list, set and dict; the others are immutable.

It fits a line minimising squared error. Assumes linearity, independent errors, constant variance (homoscedasticity), normally distributed residuals and no strong multicollinearity.

A decorator wraps a function to add behaviour such as logging, timing or caching without changing its code. It is applied with the @name syntax above the function.

Layers of connected neurons that apply weights, biases and activation functions to learn complex patterns, trained by backpropagation.

RNNs process sequences with a hidden state but struggle with long dependencies. LSTMs add gates that control what to remember or forget, easing the vanishing gradient.

Information from outside the training window, such as future data or test statistics, enters training and inflates performance. Split first, then fit scalers and encoders on training data only.

Bias occurs when the sample does not represent the population. Reduce it with random or stratified sampling, higher response rates and by checking sample demographics against the population.

Activations add non-linearity so networks can learn complex patterns. ReLU (max(0, x)) is cheap and reduces vanishing gradients compared with sigmoid/tanh; Leaky ReLU avoids 'dead' neurons.

As the number of independent trials grows, the sample average converges to the true expected value, which is why larger samples give more stable estimates.

pd.pivot_table(df, index='region', columns='month', values='sales', aggfunc='sum') reshapes long data into a summary matrix; melt() does the reverse.

Predictors that are highly correlated make coefficients unstable. Detect with VIF above about 5-10, then drop or combine features, use PCA or apply regularisation.

Mean is the average, median is the middle value after sorting and mode is the most frequent value. Median is more robust to outliers.

Use stratified splits, class weights, resampling such as SMOTE, threshold tuning and judge models with precision, recall, F1 or PR-AUC instead of accuracy.

Use stratified splits, class weights, resampling (SMOTE or undersampling) and threshold tuning, and judge the model with PR-AUC or F1 instead of accuracy.

Bagging trains models in parallel on bootstrap samples and averages them (reduces variance, e.g. Random Forest). Boosting trains models one after another, each fixing the previous errors (reduces bias, e.g. XGBoost).

The probability of seeing a result at least this extreme if the null hypothesis were true. A small p-value (for example < 0.05) is evidence against the null hypothesis.

Pearson measures linear relationships on continuous data; Spearman uses ranks, so it captures monotonic relationships and is less sensitive to outliers.

It assigns points to the nearest centroid and updates centroids until stable. Choose k with the elbow method, silhouette score or business context.

An optimisation method that updates weights in the opposite direction of the loss gradient. The learning rate sets the step size: too high and it diverges, too low and it is slow. Variants include SGD, mini-batch and Adam.

Training a meta-model on the predictions of several base models, often using cross-validated predictions to avoid leakage.

Term Frequency × Inverse Document Frequency: a word scores high if it is frequent in one document but rare across all documents. A classic way to turn text into numeric features.

Correlation measures linear association between variables (-1 to 1). A third factor, reverse causality or chance can create it, so causation needs controlled experiments.

Frequentists treat parameters as fixed and use long-run frequencies; Bayesians treat parameters as distributions updated with data via priors and posteriors, giving direct probability statements.

The model memorises noise and fails on new data. Prevent it with more data, simpler models, regularisation, cross-validation, early stopping and dropout.

L1 (Lasso) adds absolute weights and can shrink some to zero, giving feature selection. L2 (Ridge) adds squared weights and shrinks them smoothly without removing features.

It checks whether observed category counts differ from expected counts, used for independence between two categorical variables and for goodness of fit.