We estimate the number of graphical regular representations (GRRs) of a given group with large enough order. As a consequence, we show that almost all finite Cayley graphs have full automorphism groups 'as small as possible'. This confirms a conjecture of Babai-Godsil-Imrich-Lovasz on the proportion of GRRs, as well as a conjecture of Xu on the proportion of normal Cayley graphs, among Cayley graphs of a given finite group.