Wireless content caching has recently been considered as an efficient way in fog radio access networks (FRANs) to alleviate the heavy burden on capacity-limited fronthaul links and reduce delivery latency. In this paper, an advanced minimal delay association policy is proposed to minimize latency while guaranteeing spectral efficiency in F-RANs. By utilizing stochastic geometry and queueing theory, closed-form expressions of successful delivery probability, average ergodic rate, and average delivery latency are derived, where both the traditional association policy based on accessing the base station with maximal received power and the proposed minimal delay association policy are concerned. Impacts of key operating parameters on the aforementioned performance metrics are exploited. It is shown that the proposed association policy has a better delivery latency than the traditional association policy. Increasing the cache size of fog-computing based access points (F-APs) can more significantly reduce average delivery latency, compared with increasing the density of F-APs. Meanwhile, the latter comes at the expense of decreasing average ergodic rate. This implies the deployment of large cache size at F-APs rather than high density of F-APs can promote performance effectively in F-RANs.