For any fixed positive integer $r$ and a given budget $k$, the $r$-\textsc{Eigenvalue Vertex Deletion} ($r$-EVD) problem asks if a graph $G$ admits a subset $S$ of at most $k$ vertices such that the adjacency matrix of $G\setminus S$ has at most $r$ distinct eigenvalues. The edge deletion, edge addition, and edge editing variants are defined analogously. For $r = 1$, $r$-EVD is equivalent to the Vertex Cover problem. For $r = 2$, it turns out that $r$-EVD amounts to removing a subset $S$ of at most $k$ vertices so that $G\setminus S$ is a cluster graph where all connected components have the same size. We show that $r$-EVD is NP-complete even on bipartite graphs with maximum degree four for every fixed $r > 2$, and FPT when parameterized by the solution size and the maximum degree of the graph. We also establish several results for the special case when $r = 2$. For the vertex deletion variant, we show that $2$-EVD is NP-complete even on triangle-free and $3d$-regular graphs for any $d\geq 2$, and also NP-complete on $d$-regular graphs for any $d\geq 8$. The edge deletion, addition, and editing variants are all NP-complete for $r = 2$. The edge deletion problem admits a polynomial time algorithm if the input is a cluster graph, while the edge addition variant is hard even when the input is a cluster graph. We show that the edge addition variant has a quadratic kernel. The edge deletion and vertex deletion variants are FPT when parameterized by the solution size alone. Our main contribution is to develop the complexity landscape for the problem of modifying a graph with the aim of reducing the number of distinct eigenvalues in the spectrum of its adjacency matrix. It turns out that this captures, apart from Vertex Cover, also a natural variation of the problem of modifying to a cluster graph as a special case, which we believe may be of independent interest.