.ipynb .pdf

repository

Contents

Frequentist Vs. Bayesian Methods

By: Chengyi (Jeff) Chen

---

Introduction

---

Maximum Likelihood Estimation

Derivation 1: KL Divergence

How to find the best $p(\mathcal{X},\mathcal{Z};\mathrm{\Theta }=\theta )$?

To learn the $p(\mathcal{X},\mathcal{Z};\mathrm{\Theta }=\theta )$, we need to first design a measure of success – how useful our model is / how accurate are we modelling the real life true data distribution. Because we can only observe $\mathcal{X}$, let’s define a “distance” measure between our incomplete data likelihood $p(\mathcal{X};\mathrm{\Theta }=\theta )$ (instead of complete data likelihood because we can’t observe it) and the true data distribution $f(\mathcal{X})$. The smaller the “distance” between our 2 distributions the better our model approximates the true data generating process. A common “distance” measure between probability distributions is the KL Divergence (“distance” because KL Divergence is asymmetric, does not satisfy triangle inequality, $D_{KL}(P||Q)\ne D_{KL}(Q||P)$). $D_{KL}(f(\mathcal{X})||p(\mathcal{X};\mathrm{\Theta }=\theta ))$ measures how well $p$ approximates $f$:

(17)$$\begin{array}{rl}{\theta }^{\ast }& =\arg \underset{\theta \in \mathrm{\Theta }}{min}{D}_{KL}(f||p)\\ & =\arg \underset{\theta \in \mathrm{\Theta }}{min}{\int }_{\mathbf{x}\in \mathcal{X}}f(\mathcal{X}=\mathbf{x})\log \frac{f(\mathcal{X}=\mathbf{x})}{p(\mathcal{X}=\mathbf{x};\mathrm{\Theta }=\theta )}d\mathbf{x}\\ & =\arg \underset{\theta \in \mathrm{\Theta }}{min}{\mathbb{E}}_{\mathbf{x}\sim f}[\log f(\mathcal{X}=\mathbf{x})]-{\mathbb{E}}_{\mathbf{x}\sim f}[\log p(\mathcal{X}=\mathbf{x};\mathrm{\Theta }=\theta )]\\ & =\arg \underset{\theta \in \mathrm{\Theta }}{min}-\mathbb{H}[f(\mathcal{X})]-{\mathbb{E}}_{\mathbf{x}\sim f}[\log p(\mathcal{X}=\mathbf{x};\mathrm{\Theta }=\theta )]\\ & =\arg \underset{\theta \in \mathrm{\Theta }}{max}{\mathbb{E}}_{\mathbf{x}\sim f}[\log p(\mathcal{X}=\mathbf{x};\mathrm{\Theta }=\theta )]\\ & \approx \arg \underset{\theta \in \mathrm{\Theta }}{max}\underset{N\to \mathrm{\infty }}{lim}\frac{1}{N}\sum _{{\mathbf{x}}_i\in {\mathbf{X}}_{\text{train}}}\log p(\mathcal{X}={\mathbf{x}}_i;\mathrm{\Theta }=\theta )\because \text{law of large numbers}\\ & =\arg \underset{\theta \in \mathrm{\Theta }}{max}\prod _{{\mathbf{x}}_i\in {\mathbf{X}}_{\text{train}}}p(\mathcal{X}={\mathbf{x}}_i;\mathrm{\Theta }=\theta )\because \log \text{ is a monotonic increasing function}\\ & =\arg \underset{\theta \in \mathrm{\Theta }}{max}p(\mathcal{X}={\mathbf{X}}_{\text{train}};\mathrm{\Theta }=\theta )\because \text{i.i.d. data assumption}\\ & ={\theta }_{\text{MLE}}\end{array}$$

We have thus arrived at Maximum Likelihood Estimation of parameters (you can read more about this derivation method here and here), a pointwise estimate of the parameters that maximizes the incomplete data likelihood (or complete data likelihood when we have no latent variables in the model).

Derivation 2: Posterior with Uniform Prior on Parameters

Why is MLE a “frequentist” inference technique?

The primary reason for why this technique is coined a “frequentist” method is because of the assumption that $\mathrm{\Theta }=\theta$ is a fixed parameter that needs to be estimated, while bayesians believe that $\mathrm{\Theta }=\theta$ should be a random variable, and hence, have a probability distribution that describes its behavior $p(\mathrm{\Theta })$, calling it our prior. In probabilistic programming / machine learning however, we don’t have to worry about these conflicting paradigms. To “convert” $\mathrm{\Theta }$ into a random variable instead, we just need to move $\mathrm{\Theta }$ into $\mathcal{Z}$ and as long as we have a way to model $\mathcal{Z}$, more specifically $p(\mathcal{Z}|\mathcal{X};\mathrm{\Theta }=\theta )$, the posterior distribution of our latent variables, we are good.

Can we simply find the $\theta$ that maximizes $p(\mathcal{X}={\mathbf{X}}_{\text{train}};\mathrm{\Theta }=\theta )$?

Unfortunately, because our model is specified with the latent variables $\mathcal{Z}$, we can’t directly maximize $p(\mathcal{X}={\mathbf{X}}_{\text{train}};\mathrm{\Theta }=\theta )$. We’ll have to marginalize out the latent variables first as follows:

(18)$$\begin{array}{rl}p(\mathcal{X}={\mathbf{X}}_{\text{train}};\mathrm{\Theta }=\theta )& ={\int }_{\mathbf{z}\in \mathcal{Z}}p(\mathcal{X}={\mathbf{X}}_{\text{train}},\mathcal{Z}=\mathbf{z};\mathrm{\Theta }=\theta )d\mathbf{z}\\ & ={\int }_{\mathbf{z}\in \mathcal{Z}}p(\mathcal{X}={\mathbf{X}}_{\text{train}}|\mathcal{Z}=\mathbf{z};\mathrm{\Theta }=\theta )p(\mathcal{Z}=\mathbf{z};\mathrm{\Theta }=\theta )d\mathbf{z}\end{array}$$

and hence, Maximum Likelihood Estimation becomes:

(19)$$\begin{array}{rl}{\theta }^{\ast }& =\arg \underset{\theta \in \mathrm{\Theta }}{max}{\int }_{\mathbf{z}\in \mathcal{Z}}p(\mathcal{X}={\mathbf{X}}_{\text{train}}|\mathcal{Z}=\mathbf{z};\mathrm{\Theta }=\theta )p(\mathcal{Z}=\mathbf{z};\mathrm{\Theta }=\theta )d\mathbf{z}\end{array}$$

However, this marginalization is often intractable (e.g. if $\mathcal{Z}$ is a sequence of events, so that the number of values grows exponentially with the sequence length, the exact calculation of the integral will be extremely difficult). Let’s instead try to find a lower bound for it by expanding it.

---

Maximum A Posteriori

Derivation 1: Computationally Inconvienient to calculate the full Posterior $p(\mathcal{\Theta }|\mathcal{X}={\mathbf{X}}_{\text{train}})$

Before continuing, realize that because

(20)$$\begin{array}{rl}p(\mathcal{X},\mathcal{Z};\mathrm{\Theta }=\theta )& =p(\mathcal{Z}|\mathcal{X};\mathrm{\Theta }=\theta )p(\mathcal{X};\mathrm{\Theta }=\theta )\end{array}$$

(21)$$\begin{array}{rl}p(\mathcal{Z}|\mathcal{X};\mathrm{\Theta }=\theta )& =\frac{}{}\end{array}$$

(22)$$\begin{array}{rl}& =\arg \underset{\theta \in \mathrm{\Theta }}{max}\frac{1}{N}\sum _{\mathbf{x}\in {\mathbf{x}}_{\text{train}}}{\int }_{\mathbf{z}\in \mathcal{Z}}\log p(\mathcal{X}=\mathbf{x},\mathcal{Z}=\mathbf{z};\mathrm{\Theta }=\theta )d\mathbf{z}\end{array}$$

Note

Mathematical Notation

The math notation of my content, including the ones in this post follow the conventions in Christopher M. Bishop’s Pattern Recognition and Machine Learning. In addition, I use caligraphic capitalized roman and capitalized greek symbols like $\mathcal{X},\mathcal{Y},\mathcal{Z},\mathrm{\Omega },\mathrm{\Psi },\mathrm{\Xi },\dots$ to represent BOTH a set of values that the random variables can take as well as the argument of a function in python (e.g. def p(Θ=θ)).

https://pyro.ai/examples/intro_long.html#Background:-inference,-learning-and-evaluation

Objective:

(23)$$\begin{array}{}\end{array}$$

Derivation 2:

Parameter Uncertainty

Frequentist: Uncertainty is estimated with confidence intervals

Bayesian: Uncertainty is estimated with credible intervals

Prediction Intervals

---

Empircal Bayes; Type II Maximum Likelihood Estimation

---

Hierarchical Bayes

Probabilistic Machine Learning Finding the Posterior of Latent Variables $\mathcal{Z}$