Two boards are rook equivalent if they have the same number of non-attacking rook placements for any number of rooks. Define a rook equivalence graph of an equivalence set of Ferrers boards by specifying that two boards are connected by an edge if you can obtain one of the boards by moving squares in the other board out of one column and into a singe other column. Given such a graph, we categorize which boards will yield connected graphs. We also provide some cases where common graphs will or will not be the graph for some set of rook equivalent Ferrers boards. Finally, we extend this graph definition to the $m$-level rook placement generalization developed by Briggs and Remmel. This yields a graph on the set of rook equivalent singleton boards, and we characterize which singleton boards give rise to a connected graph.