The following code block defines a function, fit_generative_model(), that takes as input a training set (x data and y labels) and fits a Gaussian generative model to it. It returns the following parameters of this generative model—for each label, j = 0,1,...,9, we have the following:
- πj: The frequency of the label (that is, the prior)
- μj: The 784-dimensional mean vector
- ∑j: The 784 x 784 covariance matrix
This means that π is a 10 x 1, μ is a 10 x 784, and ∑ is a 10 x 784 x 784 matrix. The Maximum Likelihood Estimates (MLE) for the parameters are the empirical estimates, as shown in the following diagram:
The empirical covariances are very likely to be singular ...
