How many adjacency matrix queries (also known as pair queries) are required to estimate the size of a maximum matching in an $n$-vertex graph $G$? We study this fundamental question in this paper. On the upper bound side, an algorithm of Bhattacharya, Kiss, and Saranurak [FOCS'23] gives an estimate that is within $εn$ of the right bound with $n^{2-Ω_ε(1)}$ queries, which is subquadratic in $n$ (and thus sublinear in the matrix size) for any fixed $ε> 0$. On the lower bound side, while there has been a lot of progress in the adjacency list model, no non-trivial lower bound has been established for algorithms with adjacency matrix query access. In particular, the only known lower bound is a folklore bound of $Ω(n)$, leaving a huge gap. In this paper, we present the first superlinear in $n$ lower bound for this problem. In fact, we close the gap mentioned above entirely by showing that the algorithm of [BKS'23] is optimal. Formally, we prove that for any fixed $δ> 0$, there is a fixed $ε> 0$ such that an estimate that is within $εn$ of the true bound requires $Ω(n^{2-δ})$ adjacency matrix queries. Our lower bound also has strong implications for estimating the earth mover's distance between distributions. For this problem, Beretta and Rubinstein [STOC'24] gave an $n^{2-Ω_ε(1)}$ time algorithm that obtains an additive $ε$-approximation and works for any distance function. Whether this can be improved generally, or even for metric spaces, had remained open. Our lower bound rules out the possibility of any improvements over this bound, even under the strong assumption that the underlying distances are in a (1, 2)-metric.