We introduce the family of graphical Hermite simplices and study the Smith normal forms of their matrices of vertex vectors, which is equivalent to studying the group structure of the cokernels for these matrices. Our motivation is to study the behavior of lattice simplices subject to small lattice perturbations of their vertices. In this case, a graphical Hermite simplex is a perturbation of a rectangular simplex, i.e., a simplex defined by a diagonal matrix and the origin, with the perturbation controlled by the structure of a directed graph. We first establish sufficient conditions on the graphs and diagonal entries of these matrices that imply having a single non-unit invariant factor, i.e., a cyclic cokernel. We then obtain bounds on the invariant factors of the defining matrices related to lengths of paths in the corresponding directed graph.