We explore properties and applications of the Principal Inertia Components (PICs) between two discrete random variables $X$ and $Y$. The PICs lie in the intersection of information and estimation theory, and provide a fine-grained decomposition of the dependence between $X$ and $Y$. Moreover, the PICs describe which functions of $X$ can or cannot be reliably inferred (in terms of MMSE) given an observation of $Y$. We demonstrate that the PICs play an important role in information theory, and they can be used to characterize information-theoretic limits of certain estimation problems. In privacy settings, we prove that the PICs are related to fundamental limits of perfect privacy.