Advanced Machine Learning

08: Bayesian Decision Theory

Schedule

Outline for the lecture

• Preliminaries
• The Bayesian View

Preliminaries

Probability theory is nothing but common sense reduced to calculation
- Pierre-Simon Laplace, 1812

Deductive and plausible reasoning

Random Variables

Probability Distributions

Probability Distributions

The Bayesian view

Bayesian decision theory

Definition: Quantify the trade-offs between various classification decisions based on probabilities and the costs that accompany such decisions

• The decision problem is posed in probabilistic terms
• All relevant probability values are known

The Prior

• $\omega$ is a random variable: fish type
• $\omega\in \{\omega_1, \omega_2\}$ $\omega_1$ - salmon, $\omega_2$ - sea bass
• $\prob{P}{\omega_1}$ - the a priory probability (prior) that fish is a salmon
• $\prob{P}{\omega_2}$ - the a priory probability (prior) that fish is a sea bass

• Gives us the knowledge of how likely we are to get salmon before we see any fish

Priors-only decision rule

How do we make a decision between $\omega_1$ and $\omega_2$ if all we know is the priors $\prob{P}{\omega_1}$ and $\prob{P}{\omega_2}$?

$\prob{P}{\mbox{meet}} \gt \prob{P}{\overline{\mbox{meet}}}\rightarrow\mbox{shoot}$

Decide based just on the prior

• If $\prob{P}{\omega_1} \gg \prob{P}{\omega_2}$, you are right most of the time when deciding $\omega_1$
• If $\prob{P}{\omega_1} = \prob{P}{\omega_2}$, you are taking a random guess when deciding $\omega_1$
• No other decision rule can yield larger (expected) probability of being right

Use observations to increase our chances

• If we know the class conditional probability density $\prob{P}{\vec{x}|\omega_i}$ we can make better decisions
• For example, $\vec{x}$ is the observed lightness of a fish

Thomas Bayes

Bayes theorem

• Use the observed feature(s) $\vec{x}$ and the Bayes theorem

  \[ \prob{P}{\omega_i|\vec{x}} = \frac{\prob{P}{\vec{x}|\omega_i}\prob{P}{\omega_i}}{\prob{P}{\vec{x}}} \]

• In "plain English"

  \[ \mbox{posterior} = \frac{\mbox{likelihood}\times\mbox{prior}}{\mbox{evidence}} \]

• Evidence for $C$ classes $ \prob{P}{\vec{x}} = \sum_i^C\prob{P}{\vec{x}|\omega_i}\prob{P}{\omega_i} $

Put this to numbers

One person in 200,000 has a particular Progressive multifocal leukoencephalopathy (PML). There is a diagnostic test for the disease. It is correct 99% of time. Your test is positive what's the probability you have PML?

Decide based on posterior

• If $\prob{P}{\omega_1|\vec{x}} \gt \prob{P}{\omega_2|\vec{x}}$, decide $\omega_1$
• If $\prob{P}{\omega_1|\vec{x}} \lt \prob{P}{\omega_2|\vec{x}}$, decide $\omega_2$
• $\prob{P}{error|\vec{x}} = \min[\prob{P}{\omega_1|\vec{x}}, \prob{P}{\omega_2|\vec{x}}]$

General Bayesian Decision Theory

• Multiple features in $\vec{x}$
• More than two states of nature (classes)
• Allow actions (also refusal to act)
• Loss function more general that probability of error

General Bayesian Decision Theory: formally

• $\Omega = \{ \omega_1, \omega_2, \dots, \omega_C\}$ $c$ categories
• $\vec{x} = \{x_1, x_2, \dots, x_d\}$ $d$-dimensional feature vector
• $\vec{\alpha} = \{\alpha_1, \alpha_2, \dots, \alpha_A\}$ set of actions
• $\lambda(\alpha_i|\omega_i)$ loss incurred for taking action $\alpha_i$ in case of $\omega_i$
• Conditional risk \begin{array} \prob{R}{\alpha_i|\vec{x}} = \sum_j^C \lambda(\alpha_i|\omega_j)\prob{P}{\omega_j|\vec{x}} \end{array}
• Total risk \begin{array} \prob{R}{\vec{x}} = \sum_i^A \prob{R}{\alpha_i|\vec{x}} \end{array}

Assignment

Draw probability densities and find the decision regions for the following classes:

• $\Omega = \{ \omega_1, \omega_2\}$
• $\prob{P}{x|\omega_1} \sim \prob{N}{20, 4}$
• $\prob{P}{x|\omega_2} \sim \prob{N}{15, 2}$
• $\prob{P}{\omega_1} = \frac{1}{3}$
• $\prob{P}{\omega_2} = \frac{2}{3}$
• Classify $x = 17$

Maximum A Posteriori

The poor man's Bayes

$n = \alpha_H + \alpha_T$ increases $\rightarrow$

• Potentially, can know all there's to know about posterior
• But chose to go for a single parameter estimate

Can estimate full distributions

need to resort to sampling

Choosing conjugate priors

striving for simple analytical forms