Advanced Machine Learning

20: Model Comparison & Calibration

Schedule (you are here )

Outline for the lecture

• Bias-Variance Trade-off & No Free Lunch Theorem
• Problems with Accuracy (Imbalance & Rebalancing Paradox)
• Confusion Matrix, Visual Quotients, Balanced Accuracy & $F_1$
• Cross-Validation, Hyperparameter Search & Expected Profit
• Profit Curves, ROC / AUC & Model Calibration

Bias-variance trade off revisited

Bias-Variance Decomposition for MSE

Bias-variance tradeoff

If data is all we have, how do we know which model is better?

Training Error Lies: 100 Random Dataset Splits

Each thin hairline is one random dataset draw $\CD_i$ (vertical spread = Variance). As complexity grows, training error $\downarrow$ monotonically while test error $\uparrow$ and fans out — never select model complexity on training error!

Select a Model That Generalizes Best

We pick hyperparameter $\hat{\theta}_i = \argmin_{\theta} g(\theta; \CD_i)$ by minimizing validation error $g(\theta; \CD_i)$ to approximate true test error $G(\theta)$:
$\EE_{\CD}\!\left[(g(\theta; \CD) - G(\theta))^2\right] = \text{Bias}(g)^2 + \text{Variance}(g)$

Takeaway: A small bias ($\EE[\hat{\theta}] \neq \theta^*$) with low variance stays consistently near $\theta^*$, whereas an unbiased estimator with high variance can pick disastrous $\hat{\theta}_i$ far from $\theta^*$!

From Classical Bias-Variance to Double Descent

Why the Second Descent? At $p = n$, the unique interpolant has exploding norm $\|\hat{\vth}\|_2 \sim \sigma_{\min}(\vec{X})^{-1} \to \infty$. For $p \gg n$, SGD finds the minimum-norm interpolant $\hat{\vth}_{\text{min}} = \argmin_{\vec{X}\vth = \vec{y}} \|\vth\|_2^2 = \vec{X}^{\top}(\vec{X}\vec{X}^{\top})^{-1}\vec{y}$, whose norm and variance shrink as $p$ grows!

No free lunch theorem

"Even after the observation of the frequent conjunction of objects, we have no reason to draw any inference concerning any object beyond those of which we have had experience."

David Hume, in A Treatise of Human Nature, Book I, part 3, Section 12.

No free lunch theorem (simply put)

There is not a universally good algorithm that's best on all possible test data!

Problems with accuracy

Accuracy on a Skewed Dataset (95% vs. 5%)

$$\text{accuracy} = \frac{\text{number of correct predictions}}{\text{number of total predictions}}$$

Naive Fix: Manually Rebalance Training Data (50% / 50%)

Suppose $N = 10^6$. Take all $5\%$ of Class $0$ ($50{,}000$) + $5\%$ from Class $1$ ($50{,}000$) $\implies 10^5$ balanced samples:

The Rebalancing Paradox: What Happens in the Wild?

Now deploy both classifiers on the true population ($5\%$ Class $0$, $95\%$ Class $1$):

Neither $84\%$ nor $99\%$ makes sense without knowing the cost of mistaking $0 \to 1$ vs. $1 \to 0$!

Confusion Matrix

Two-Class Confusion Matrix

Statistician view of the world

Visual Quotient Mnemonic: Accuracy

Family 1: Sensitivity & Specificity (Column Quotients)

Family 2: Precision & Recall (Information Retrieval)

Why One Metric Is Not Enough: How Models Cheat

• Maximize Recall (Sensitivity) alone?
  Predict Positive ($\mathbf{P}$) for everything $\implies \text{FN} = 0 \implies \mathbf{100\%\text{ Recall}}$ (with massive $\text{FP}$!).
• Maximize Specificity alone?
  Predict Negative ($\mathbf{N}$) for everything $\implies \text{FP} = 0 \implies \mathbf{100\%\text{ Specificity}}$ (catches $0$ positives!).
• Maximize Precision alone?
  Set threshold ultra-high and predict $\mathbf{P}$ on only $1$ certain sample $\implies \mathbf{100\%\text{ Precision}}$!

We must combine complementary quotients so the classifier cannot cheat!

Imbalanced Example: Raw Accuracy Rewards Ignoring P

Balanced Accuracy & F1 Score Catch the Cheater

Multiclass Confusion Matrix: Drug Repurposing

Which Metric is the right one?

it depends

Cross Validation

5-fold cross validation

Cross validation results

Grid search workflow

Extra testing data

Grid Search: Combinatorial Explosion

Nested Cross Validation

Hyperparameter optimization

Tools for hyperparameter optimization

• hyperopt
• Optuna
• Orion

Expected Value (profit)

Expected value

• Let's denote an outcome $i$ as $o_i$
• The probability of that outcome as $\prob{p}{o_i}$
• And its value as $\prob{v}{o_i}$
• The expected value is nothing but $$ EV = \sum_i^K \prob{p}{o_i}\cdot\prob{v}{o_i} $$

Example: targeted marketing

• A consumer buys the product for $\$200$ and our product-related costs are $\$100$.
• To target the consumer with the offer, we also incur a cost. Let’s say that we mail some flashy marketing materials, and the overall cost including postage is $\$1$, yielding a value (profit) of $\prob{v}{o_R} = \$99$ if the consumer responds (buys the product).
• Now, what about $v_{NR}$, the value to us if the consumer does not respond? We still mailed the marketing materials, incurring a cost of $\$1$ or equivalently a benefit of $-\$1$.

Example: shall we target this specific consumer?

• $\prob{p}{o_R} \cdot \$99 - [1 - \prob{p}{o_{R}}]\cdot \$1 \gt 0$
• $\prob{p}{o_R} \cdot \$99 \gt [1 - \prob{p}{o_{R}}]\cdot \$1$
• $\prob{p}{o_R} \gt 0.01$

• Send marketing materials if probability of responding is $\gt 1\%$

First, convert confusion matrix to probabilities

1. Raw Counts ($N = 110$)

Col $p$: $56+5=61$
Col $n$: $7+42=49$

2. Joint $\prob{p}{\text{Pred}, \text{Actual}}$

Divide all cells by $110$:
$\prob{p}{p} = \frac{61}{110} = \mathbf{0.55},\; \prob{p}{n} = \frac{49}{110} = \mathbf{0.45}$

3. Conditional $\prob{p}{\text{Pred} \mid \text{Actual}}$

Normalize each column by its sum:
$\prob{p}{Y\mid p}=0.92,\; \prob{p}{N\mid n}=0.86$

Second, estimate the cost benefit matrix

Expected Value Calculation

Example: targeted marketing — What's the profit?

What did we learn?

Decisions have costs

Visualizing model performance

Remember that Bayesian decision boundary?

Let's move the boundary around until we are happy

A classifier and confusion matrix

With a ranking classifier, a classifier plus a threshold produces a single confusion matrix.

Profit Curve: Example (Small Margin)

Profit Curves Across Classifiers

Receiver Operating Characteristics (ROC)

From Profit Curves to ROC

• Profit curves work well only when class priors and costs/benefits are known and stable.
• In practice, priors fluctuate (e.g., fraud rates vary month-to-month) and campaign budgets/costs change.
• Receiver Operating Characteristic (ROC) decouples performance from priors and costs by plotting the two column quotients!

ROC Axes: Hit Rate (TPR) vs. False Alarm Rate (FPR)

ROC Space

ROC: Each Point Is a Threshold

ROC Construction: Step by Step

Area Under the ROC Curve (AUC)

What Does AUC Mean Probabilistically?

Equivalent to the Mann-Whitney-Wilcoxon ranking statistic (and Gini coefficient):

$$\text{AUC} = \PP\Big( s(\vec{x}_{+}) > s(\vec{x}_{-}) \Big)$$ The probability that a randomly chosen positive sample $\vec{x}_{+}$ is ranked (scored) higher than a randomly chosen negative sample $\vec{x}_{-}$!

Threshold, ROC Space & Expected Profit Explorer

On Model Calibration

What is Model Calibration?

Model calibration is the problem of predicting probability estimates representative of the true correctness likelihood

Why do we need it?

We want the probability associated with the predicted class label to reflect its ground truth correctness likelihood.
If that is the case, we know when we can trust the model, and when to route the decision to a human!

Modern Best-Performing Models Are Lying

• 5-layer LeNet (1998) is well-calibrated
• 110-layer ResNet (2016) achieves much higher accuracy, yet is poorly calibrated (overconfident)
• Why does this happen, and how do we fix it?

Formal definitions

• Inputs $X \in {\cal X}$ and labels $Y \in {\cal Y} = \{1, \dots, K\}$ follow ground truth joint distribution $\pi(X,Y) = \pi(Y|X)\pi(X)$.
• A neural network $h(X) = (\hat{Y}, \hat{P})$ outputs a class prediction $\hat{Y}$ and a confidence score $\hat{P}$.
• We want the confidence estimate $\hat{P}$ to be calibrated, which intuitively means that $\hat{P}$ represents a true probability.
• Perfect calibration:

$$ \PP(\hat{Y} = Y \mid \hat{P} = p) = p, \quad \forall p \in [0, 1] $$

Reliability Diagrams

• Reliability Diagrams visualize model calibration (DeGroot & Fienberg, 1983)
• Plot expected sample accuracy as a function of confidence
• Perfect calibration ($\PP(\hat{Y} = Y | \hat{P} = p) = p$) lies on the identity diagonal
• Any deviation from the diagonal is miscalibration

Calibration Across Classical Classifiers

Reliability Diagrams: Binning

• Group predictions into $M$ interval bins (each of size $1/M$) and calculate the accuracy of each bin.
• Let $B_m$ be the set of indices of samples whose prediction confidence falls into the interval $I_m = (\frac{m-1}{M}, \frac{m}{M}]$.
• The accuracy of $B_m$ is: $$ \mbox{acc}(B_m) = \frac{1}{|B_m|} \sum_{i \in B_m} \mathbf{1}(\hat{y}_i = y_i) $$
• The average confidence within bin $B_m$ is: $$ \mbox{conf}(B_m) = \frac{1}{|B_m|} \sum_{i \in B_m} \hat{p}_i $$

Expected Calibration Error (ECE)

• We use empirical approximations directly: $$ \EE_{\hat{P}}\left[ \left| \PP(\hat{Y} = Y | \hat{P} = p) - p \right| \right] $$
• Expected Calibration Error (ECE) — weighted average of the red gap bars across all $M$ bins: $$ \mbox{ECE} = \sum_{m=1}^M \frac{|B_m|}{n} \left| \mbox{acc}(B_m) - \mbox{conf}(B_m) \right| $$
• Maximum Calibration Error (MCE) — worst-case bin gap for high-risk applications: $$ \mbox{MCE} = \max_{m \in \{1,\dots, M\}} \left| \mbox{acc}(B_m) - \mbox{conf}(B_m) \right| $$

Negative Log-Likelihood (NLL)

• A standard measure of a probabilistic model's quality (also known as cross-entropy loss in deep learning).
• Given a probabilistic model $\hat{\pi}(Y|X)$ and $n$ samples: $$ {\cal L} = - \sum_{i=1}^n \log\left(\hat{\pi}(y_i|x_i)\right) $$
• NLL is a proper scoring rule: in expectation, it is minimized if and only if $\hat{\pi}(Y|X)$ recovers the ground truth conditional distribution $\pi(Y|X)$.

What Affects Calibration?

• Increasing depth and width reduces classification error, but hurts calibration.
• Models trained with Batch Normalization or less weight decay are more miscalibrated.

Accuracy vs. NLL During Training

The network overfits NLL (becoming overconfident) long after 0/1 classification error plateaus!

Temperature Scaling

• The neural network computes a vector of non-probabilistic logits $\vec{z}_i$ for each input $\vec{x}_i$, before the softmax layer $\sigma_{\text{SM}}$: $$ \hat{p}_i = \max_k \sigma_{\text{SM}}(\vec{z}_i)^{(k)} = \max_k \frac{\exp(z_i^{(k)})}{\sum_{j=1}^K \exp(z_i^{(j)})} $$
• Temperature Scaling divides the logits by a single scalar parameter $T > 0$ before the softmax: $$ \hat{q}_i = \max_k \sigma_{\text{SM}}\left(\frac{\vec{z}_i}{T}\right)^{(k)} $$

Properties of Temperature Scaling

• $T = 1$: recovers the original uncalibrated probabilities $\hat{p}_i$.
• As $T > 1$ increases: softens the softmax distribution (raises output entropy) and cures overconfidence.
• As $T \to \infty$: $\hat{q}_i \to 1/K$ (uniform maximum uncertainty).
• As $T \to 0$: collapses to a point mass ($\hat{q}_i \to 1$).
• $T$ is optimized with respect to NLL on the validation set (with network weights frozen).
• Because $z_i^{(k)} / T$ is strictly monotonic for $T > 0$, $\argmax_k$ is unchanged — Top-1 Accuracy is 100% preserved!

Results: Expected Calibration Error (ECE)

Reliability Diagram, ECE & Temperature Scaling

Take Home Points

• Bias–Variance & Double Descent: Classical test risk balances $\text{Bias}^2$ and $\text{Variance}$, peaking at the interpolation threshold ($p = n$) before descending a second time as minimum-norm interpolants shrink $\|\hat{\vth}\|_2$ for $p \gg n$.
• Why Raw Accuracy Fails: Under class skew ($95\%/5\%$), predicting the majority class everywhere achieves $95\%$ accuracy while failing on the rare class; manually rebalancing to $50\%/50\%$ reverses the winner in the wild.
• Confusion Matrix Quotients: Sensitivity ($\text{TP}/\mathbf{P}$) & Specificity ($\text{TN}/\mathbf{N}$) normalize by true columns ($\to$ Balanced Accuracy); Precision ($\text{TP}/\text{Pred }\mathbf{P}$) & Recall ($\text{TP}/\mathbf{P}$) combine into $F_1$.
• Expected Profit & ROC / AUC: When priors & costs are known, optimize $EV(\tau)$; when unstable, plot $\text{TPR}$ vs. $\text{FPR}$ (ROC), where $\text{AUC} = \PP(s(\vec{x}_{+}) > s(\vec{x}_{-}))$.
• Model Calibration & Temperature Scaling: Deep networks overfit NLL and become overconfident; dividing logits by a single validation-tuned scalar $T > 0$ aligns $\hat{P}$ with true accuracy while preserving Top-1 Accuracy.