Caches on the modern commodity CPUs have become one of the major sources of side-channel leakages and been abused as a new attack vector. To thwart the cache-based side-channel attacks, two types of countermeasures have been proposed: detection-based ones that limit the amount of microarchitectural traces an attacker can leave, and cache prefetching-and-locking techniques that claim to prevent such leakage by disallowing evictions on sensitive data. In this paper, we present the Prime+Retouch attack that completely bypasses these defense schemes by accurately inferring the cache activities with the metadata of the cache replacement policy. Prime+Retouch has three noticeable properties: 1) it incurs no eviction on the victim's data, allowing us to bypass the two known mitigation schemes, 2) it requires minimal synchronization of only one memory access to the attacker's pre-primed cache lines, and 3) it leaks data via non-shared memory, yet because underlying eviction metadata is shared. We demonstrate Prime+Retouch in two architectures: predominant Intel x86 and emerging Apple M1. We elucidate how Prime+Retouch can break the T-table implementation of AES with robust cache side-channel mitigations such as Cloak, under both normal and SGX-protected environments. We also manifest feasibility of the Prime+Retouch attack on the M1 platform imposing more restrictions where the precise measurement tools such as core clock cycle timer and performance counters are inaccessible to the attacker. Furthermore, we first demystify undisclosed cache architecture and its eviction policy of L1 data cache on Apple M1 architecture. We also devise a user-space noise-free cache monitoring tool by repurposing Intel TSX.