A graph is called $k$-extendable if each $k$-matching can be extended to a perfect matching. We give spectral conditions for the $k$-extendability of graphs and bipartite graphs using Tutte-type and Hall-type structural characterizations. Concretely, we give a sufficient condition in terms of the spectral radius of the distance matrix for the $k$-extendability of a graph and completely characterize the corresponding extremal graphs. A similar result is obtained for bipartite graphs.