We show that every finitely generated group G with an element of order at least $(5rank(G))^{12}$ admits a locally finite directed Cayley graph with automorphism group equal to G. If moreover G is not generalized dihedral, then the above Cayley directed graph does not have bigons. On the other hand, if G is neither generalized dicyclic nor abelian and has an element of order at least $(2rank(G))^{36}$, then it admits an undirected Cayley graph with automorphism group equal to G. This extends classical results for finite groups and free products of groups. The above results are obtained as corollaries of a stronger form of rigidity which says that the rigidity of the graph can be observed in a ball of radius 1 around a vertex. This strong rigidity result also implies that the Cayley (di)graph covers very few (di)graphs. In particular, we obtain Cayley graphs of Tarski monsters which essentially do not cover other quasi-transitive graphs. We also show that a finitely generated group admits a locally finite labelled unoriented Cayley graph with automorphism group equal to itself if and only if it is neither generalized dicyclic nor abelian with an element of order greater than 2.