In the chapter "Magic with a Matrix" in \emph{Hexaflexagons and Other Mathematical Diversions} (1988), Martin Gardner describes a delightful "party trick" to fill the squares of a $d$-by-$d$ chessboard with nonnegative integers such that the sum of the numbers covered by any placement of $d$ nonthreatening rooks is a given number $N$. We consider such chessboards from a geometric perspective which gives rise to a family of lattice polytopes. The polyhedral structure of these Gardner polytopes explains the underlying trick and enables us to count such chessboards for given $N$ in three different ways. We also observe a curious duality that relates Gardner polytopes to Birkhoff polytopes.