Let $G$ be a graph on $n$ vertices. The $k$-token graph (or symmetric $k$-th power) of $G$, denoted by $F_k(G)$ has as vertices the ${n\choose k}$ $k$-subsets of vertices from $G$, and two vertices are adjacent when their symmetric difference is a pair of adjacent vertices in $G$. In particular, $F_k(K_n)$ is the Johnson graph $J(n,k)$, which is a distance-regular graph used in coding theory. In this paper, we present some results concerning the (adjacency and Laplacian) spectrum of $F_k(G)$ in terms of the spectrum of $G$. For instance, when $G$ is walk-regular, an exact value for the spectral radius $ρ$ (or maximum eigenvalue) of $F_k(G)$ is obtained. When $G$ is distance-regular, other eigenvalues of its $2$-token graph are derived using the theory of equitable partitions. A generalization of Aldous' spectral gap conjecture (which is now a theorem) is proposed.