We investigate Cayley graphs of finite semigroups and monoids. First, we look at semigroup digraphs, i.e., directed Cayley graphs of semigroups, and give a Sabidussi-type characterization in the case of monoids. We then correct a proof of Zelinka from '81 that characterizes semigroup digraphs with outdegree $1$. Further, answering a question of Knauer and Knauer, we construct for every $k\geq 2$ connected $k$-outregular non-semigroup digraphs. On the other hand, we show that every sink-free directed graph is a union of connected components of a monoid digraph. Second, we consider monoid graphs, i.e., underlying simple undirected graphs of Cayley graphs of monoids. We show that forests and threshold graphs form part of this family. Conversely, we construct the -- to our knowledge -- first graphs, that are not monoid graphs. We present non-monoid graphs that are planar, have arboricity $2$, and treewidth $3$ on the one hand, and non-monoid graphs of arbitrarily high connectivity on the other hand. Third, we study generated monoid trees, i.e., trees that are monoid graphs with respect to a generating set. We give necessary and sufficient conditions for a tree to be in this family, allowing us to find large classes of trees inside and outside the family.