Maximum likelihood estimation is a popular method in statistical inference. As a way of assessing the accuracy of the maximum likelihood estimate (MLE), the calculation of the covariance matrix of the MLE is of great interest in practice. Standard statistical theory shows that the normalized MLE is asymptotically normally distributed with covariance matrix being the inverse of the Fisher information matrix (FIM) at the unknown parameter. Two commonly used estimates for the covariance of the MLE are the inverse of the observed FIM (the same as the inverse Hessian of the negative log-likelihood) and the inverse of the expected FIM (the same as the inverse FIM). Both of the observed and expected FIM are evaluated at the MLE from the sample data. In this dissertation, we demonstrate that, under reasonable conditions similar to standard MLE conditions, the inverse expected FIM outperforms the inverse observed FIM under a mean squared error criterion. Specifically, in an asymptotic sense, the inverse expected FIM (evaluated at the MLE) has no greater mean squared error with respect to the true covariance matrix than the inverse observed FIM (evaluated at the MLE) at the element level. This result is different from widely accepted results showing preference for the observed FIM. In this dissertation, we present theoretical derivations that lead to the conclusion above. We also present numerical studies on three distinct problems to support the theoretical result. This dissertation also includes two appendices on topics of relevance to stochastic systems. The first appendix discusses optimal perturbation distributions for the simultaneous perturbation stochastic approximation (SPSA) algorithm. The second appendix considers Monte Carlo methods for computing FIMs when closed forms are not attainable.