Machine Learning InFrequently Asked Questions¶
By: Chengyi (Jeff) Chen
The purpose of this section is to explore answers to questions that (in my opinion) are rarely asked but go beyond the conventional ML curriculum. Or perhaps just that I wasn’t paying enough attention during class HAHA.
Note: The answers here are subject to my own interpretation and should only be used as a secondary point of reference to the actual material that I’ve linked.
General Statistics¶
Q: What’s the difference between Law of Large Numbers and the Central Limit Theorem?
A:
Suppose \(X_1, X_2, \ldots , X_n\) are independent random variables with the same underlying distribution. In this case, we say that the \(X_i\) are independent and identically-distributed, or i.i.d. In particular, the \(X_i\) all have the same mean \(\mu\) and standard deviation \(\sigma\).
Let \(\bar{X_n}\) be the average of \(X_1, \ldots , X_n\):
Note that \(\bar{X_n}\) is itself a random variable. The law of large numbers and central limit theorem tell us about the value and distribution of \(\bar{X_n}\), respectively.
LoLN: As \(n\) grows, the probability that \(\bar{X_n}\) is close to \(\mu\) goes to 1.
CLT: As \(n\) grows, the distribution of \(\bar{X_n}\) converges to the normal distribution \(N(\mu, \sigma^2/n)\).