Machine Learning Fundamentals

Task families by output type: regression (continuous), classification (labels), ranking/recommendation, clustering (no label) — the family picks the metric. Paradigms by feedback: supervised (labels), unsupervised (structure), reinforcement (reward from actions). Features vs labels with the prediction-time rule: every column must exist, in code and meaning, strictly before the prediction moment or it is leakage. Leakage families: target, train–test contamination, temporal, entity — detect with the label-shuffle test (shuffled labels must collapse to baseline) and prevent with pipelines and honest splits. Baselines: always-mean / always-majority / current rule / simple linear — the floor every model must clear. EDA jobs: target first (balance sets metric and split; skew sets transform), feature–target relationships, quality bugs, split integrity. Missing data: mechanism matters (MCAR/MAR/MNAR), impute on train only, keep missingness indicators where absence is information. Encoding: one-hot for low cardinality, ordinal only when order is real, target encoding must be cross-fitted and smoothed. Scaling: mandatory for distance/gradient/penalty models, irrelevant for trees (threshold order is invariant). Imbalance: accuracy is meaningless at 99:1; weights, threshold tuning, PR-AUC, SMOTE inside the training fold only. Splits: stratified for rarity, time-ordered for temporal, grouped for repeated entities.

Least squares picks the line minimising squared residuals; slope = covariance/variance of x, intercept passes through the point of means. Interpretation: coefficient = change in y per unit of x holding others fixed — valid only without severe multicollinearity (check VIF; correlated predictors flip signs). Evaluation: MAE (typical error, robust), RMSE (same units, punishes big misses), R-squared (variance explained, can be negative on holdout, gamed by adding features — hence adjusted R-squared and baselines). Gradient descent: prediction, loss, gradient, step w minus lr times gradient; with squared loss on one point (x=2, y=5, w=1, lr=0.03) the step lands at 1.36. Batch/mini-batch/stochastic trade stability for speed; learning rate too high diverges, too low crawls. Regularisation: ridge (L2) shrinks all coefficients toward zero for stability under correlation; lasso (L1) lands on exact zeros for selection; elastic net combines both — and the penalty is scale-blind, so standardise first; alpha is searched on a log grid by CV, with the one-SE rule preferring the simpler model. Logistic regression: models log-odds through the sigmoid, trained with cross-entropy; exp(coefficient) is the odds ratio per unit; sigma(-2) is about 0.12; multi-class via softmax or one-vs-rest; the decision threshold is tuned on validation against asymmetric costs, never left at 0.5 by default.

Trees split greedily to maximise impurity gain: Gini = 1 minus sum of squared class shares (a 70/30 node is 0.42, also the chance two draws disagree), entropy as the log variant; depth is the overfitting dial, pruned pre- and post-hoc (cost-complexity). Trees ignore scaling (only threshold order matters) and need none (no gradient descent) — but they cannot extrapolate (leaves average seen values), prefer axis-aligned boundaries, and are unstable under resampling, which is why ensembles exist. Random forests: bootstrap samples leave about 36.8% OOB per tree, giving free validation; feature subsampling (sqrt for classification) decorrelates trees so averaging cuts variance; tune depth, leaf size, max_features — the tree count mainly buys stability. Boosting fits each tree to current residuals with shrinkage learning rate and depth control — small rate plus many trees usually beats one deep tree; watch the validation curve because boosting can overfit. k-NN: lazy, distance-based — scale first, choose k, weighted voting; distances concentrate as dimensions explode (curse of dimensionality; PCA helps). Naive Bayes: prior times likelihood over word counts, log-space sums, Laplace smoothing for unseen events; the independence assumption is false yet competitive at text ranking. SVM: maximise the margin, C trades margin width against violations, kernels separate in higher dimensions without computing the mapping explicitly. PCA: centred (usually standardised) data rotated onto directions of decreasing variance; explained-variance ratios sum to 1, cumulative threshold (95%) sets the component count, dropped eigenvalues equal reconstruction error; PCA is unsupervised and linear — variance retained is not accuracy retained.

Every metric is a cost model in disguise. Confusion matrix: TP/FP/FN/TN; precision = TP/(TP+FP) is what a flag means (80/179 is about 0.45 — fewer than half of alerts real); recall = TP/(TP+FN) is what fraction of positives you caught; F1 harmonises them, F-beta shifts weight; thresholds move along the trade — pick from FP versus FN costs on validation, not 0.5 by habit. ROC-AUC: probability a random positive outranks a random negative, judged against 0.5; PR-AUC: precision against recall with baseline equal to prevalence (0.01 on 1% data) — on rare events ROC-AUC looks fine while PR-AUC exposes useless precision. Pick the family from the base rate and harm. Regression: MAE (average absolute error, robust, in rupees), MSE/RMSE (squares punish the worst miss: errors 10, 20, -60 give MAE 30 and RMSE about 37), MAPE (percent framing, breaks on zeros and asymmetry); compare all against the naive baseline (mean/median/seasonal lag). Cross-validation: k folds refit the whole pipeline per fold per candidate — 20 candidates x 5 folds = 100 fits; stratified for rarity, grouped for entities, time-ordered for temporal (random k-fold on time leaks the future); report mean and spread, never the best fold; keep a sealed holdout for the final claim.

Expected error = bias-squared + variance + irreducible noise; tuning minimises the sum, never bias alone (worked: bias -2.6/sd 1.0 gives 11.76, bias -0.4/sd 2.5 gives 10.41, bias -1.5/sd 1.4 gives 8.21 — the least-biased model loses). Symptoms: high bias = train low and validation similarly low (needs capacity, features, weaker penalties); high variance = train high, validation far lower (needs data, stronger regularisation, simpler model, early stopping). Learning curves answer "will more data help?" — plateaued together means buy capacity, wide gap with validation still rising means buy data. Regularisation levers across families: L2/L1/elastic net (standardise first), tree depth/leaf size, boosting shrinkage, early stopping, dropout (nets); the one-SE rule picks the simplest model within one standard error of the best. Search: grid (systematic, wasteful — 500 fits where random takes 450 for better coverage), random (better spread per fit), Bayesian/Optuna (models the objective, suggests the next trial, prunes losers early — still optimising a noisy estimate). Best-of-N inflation: sigma times sqrt(2 ln N) — 100 trials at sigma 0.01 gains about 0.03 of pure luck; sigma 1 point at 50 variants is 1.4 points. Importance: impurity (free, biased to high-cardinality, optimistic on training rows) versus permutation on held-out data (honest, but correlated features mask each other — permute groups); importance is association, never causation. Leakage returns under tuning pressure: everything learned lives inside the Pipeline inside the splitter (SMOTE, encoders, selectors), and predict-time availability is re-audited before deployment.

K-means alternates nearest-centroid assignment and centroid-mean update until assignments stop; inertia only falls and reaches zero at k = n, so lowest inertia can never choose k; restarts (n_init) exist because only a local optimum is guaranteed. Distance is the model, so unscaled or duplicated columns silently decide the clusters — standardise inside a pipeline and inverse-transform centroids to describe segments in real units. Choose k honestly: elbow (shape), silhouette (cohesion a versus separation b, s = (b-a)/max(a,b): 1 clean, 0 boundary, negative misassigned) and downstream lift (the only one that pays), plus stability across seeds. Hierarchical: agglomerative merging with linkage defining cluster distance — single chains through noise (min cross distance 8 in the worked set), complete is compact but outlier-sensitive (max 10), average is the mean (9), ward minimises added within-cluster variance (heights on a different scale); a cut at height h keeps merges at or below h and replaces k with a defensible granularity dial; O(n-squared) memory caps the method at a few thousand rows. Anomalies: point, contextual and collective species need different tools; mean/sigma are vandalised by the outlier they measure (masking — the 95 ms spike scores only 2.6 sigma), median/MAD robust z = 0.6745(x - median)/MAD with threshold 3.5 survives half the sample; isolation forests flag items isolated by fewer random splits (contamination = alert budget); thresholds come from the cost matrix and the team capacity, flags are hypotheses for humans, review before retrain. PCA budgets: ratios from eigenvalues, cumulative threshold at 95%, trace equals feature count when standardised (units otherwise own component 1); t-SNE/UMAP preserve local neighbourhoods only — gaps, areas and island counts move with perplexity and seed, so cluster in real or PCA space and embed for eyes only.

Drift types decide repairs: covariate (P(X) moves, relationship stable) — fix the source or retrain; prior/label (P(Y) moves) — recalibrate; concept (P(Y|X) moves) — retrain on time-ordered windows; seasonal — model the calendar. PSI = sum over training-defined bins of (actual minus expected) times ln(actual/expected): below 0.10 stable, 0.10-0.25 investigate, above 0.25 act — and act means diagnose first (schema, units, joins, change point), because retraining on a broken feed entrenches the bug; the worked window (7/15/28/50 versus 25 each) sums to about 0.46. While labels are late, monitor features, score distributions, nulls and volume daily. Monitoring layers: data quality, input PSI, score PSI, operations (p95/p99), performance on label cadence, and slices with pre-registered floors, gap rules and sample sizes — the worked average (0.98 x 0.91 + 0.02 x 0.55 = 0.90) hides a slice at 0.55. Training-serving skew is silent: backfill training rows through the serving path and diff, shadow-compare scores. Alerts need thresholds, owners, runbooks and severity tiers or the team learns to ignore them. Fairness: start from harm (who pays for FP, who pays for FN), then compute demographic parity, equal opportunity (TPR), equalised odds (TPR and FPR) and predictive parity (PPV) per group with n; the worked groups (A: 0.90/0.15/0.60, B: 0.70/0.30/0.35) recover base rates 0.60 and 0.125 and PPV 0.90 versus 0.25 — with unequal base rates, calibration and equal error rates are provably incompatible, so you choose and document; removing the protected column hides the audit without removing the effect. When not to use ML: no trusted label, no signal beyond written rules, thin data, unreviewable harm, no monitoring capacity, no right to use the inputs; compare on operating numbers (rules 660 reviews versus ML 480 plus run cost fits a 500/day capacity with 60-day payback) and keep humans in the loop with an abstention band sized to capacity, measured combined performance and automation-bias checks.

Lifecycle gates, each with an artefact: framing (decision spec: decision, unit, prediction time, label timing rule), metrics (baselines, offline/online metrics, guardrails, ship criteria frozen before training), data (rights, availability audit), features (shared train/serve definitions, dumb baselines), validation (time/group-aware report with error analysis), release (shadow, canary, ramp, one-command rollback), monitoring (layers with owners), retirement (retrain and expiry triggers). The churn walkthrough separates the decision (who gets a call), the prediction (non-renewal probability at a fixed moment) and the intervention (measured against a holdback) — cancellation and offer-sent flags are banned as features because they arrive after the decision they encode. Experimentation: pre-register one primary metric with MDE, power (n = 2(1.96+0.84)^2 p(1-p)/delta^2: about 109,000 per arm for a 0.6-point effect at 50%), no peeking without always-valid intervals, guardrails with veto, SRM alarm (chi-square) stops the test before any metric is read, novelty needs full cycles, interference needs cluster or switchback designs; best-of-N inflation (sigma 0.5 over 50 variants = 1.4 points) kills most reported wins (observed +0.6 loses). Model cards: nine versioned sections with named out-of-scope prohibitions, a disaggregated evaluation table with n and floors, ethics and expiry — registry-gated with the model. Capstone defence: frame, data/split with leakage audit, modelling, evaluation with slices and limits, operations with triggers — answering offline-0.95-versus-production as leakage (shuffle test), split (time/group), skew (backfill diff), drift (PSI) or metric mismatch (operating point), plus the classic ten interview questions with definitions, numbers and boundary cases.