Let $n$ be a positive integer and $p$ a prime. The power mapping $x^{p^n-3}$ over $\mathbb{F}_{p^n}$ has desirable differential properties, and its differential spectra for $p=2,\,3$ have been determined. In this paper, for any odd prime $p$, by investigating certain quadratic character sums and some equations over $\mathbb{F}_{p^n}$, we determine the differential spectrum of $x^{p^n-3}$ with a unified approach. The obtained result shows that for any given odd prime $p$, the differential spectrum can be expressed explicitly in terms of $n$. Compared with previous results, a special elliptic curve over $\mathbb{F}_{p}$ plays an important role in our computation for the general case $p \ge 5$.