We address the worst-user bottleneck of wireless coded caching, which is known to severely diminish cache-aided multicasting gains due to the fundamental worst-channel limitation of multicasting transmission. We consider the quasi-static Rayleigh fading Broadcast Channel, for which we first show that the effective coded caching gain of the XOR-based standard coded-caching scheme completely vanishes in the low-SNR regime. Then, we reveal that this collapse is not intrinsic to coded caching. We do so by presenting a novel scheme that can fully recover the coded caching gains by capitalizing on one aspect that has to date remained unexploited: the shared side information brought about by the effectively unavoidable file-size constraint. As a consequence, the worst-user effect is dramatically ameliorated, as it is substituted by a much more subtle worst-group-of-users effect, where the suggested grouping is fixed, and it is decided before the channel or the demands are known. In some cases, the theoretical gains are completely recovered, and this is done without any user selection technique. We analyze the achievable rate performance of the proposed scheme and derive insightful performance approximations which prove to be very precise.