The \textit{eccentricity matrix} $\mathcal{E}(G)$ of a connected graph $G$ is obtained from the distance matrix of $G$ by keeping the largest non-zero entries in each row and each column, and leaving zeros in the remaining ones. The eigenvalues of $\mathcal{E}(G)$ are the \textit{$\mathcal{E}$-eigenvalues} of $G$. In this article, we find the inertia of the eccentricity matrices of trees. Interestingly, any tree on more than $4$ vertices with odd diameter has two positive and two negative $\mathcal{E}$-eigenvalues (irrespective of the structure of the tree). A tree with even diameter has the same number of positive and negative $\mathcal{E}$-eigenvalues, which is equal to the number of 'diametrically distinguished' vertices (see Definition 3.1). Besides we prove that the spectrum of the eccentricity matrix of a tree is symmetric with respect to the origin if and only if the tree has odd diameter. As an application, we characterize the trees with three distinct $\mathcal{E}$-eigenvalues.