Permutation polynomials (PPs) and their inverses have applications in cryptography, coding theory and combinatorial design theory. In this paper, we make a brief summary of the inverses of PPs of finite fields, and give the inverses of all PPs of degree $\leq 6$ over finite fields $\mathbb{F}_{q}$ for all $q$ and the inverses of all PPs of degree $7$ over $\mathbb{F}_{2^n}$. The explicit inverse of a class of fifth degree PPs is the main result, which is obtained by using Lucas' theorem, some congruences of binomial coefficients, and a known formula for the inverses of PPs of finite fields.