Partially-overlapping tones (POT) are known to help mitigate co-channel interference in uncoordinated multi-carrier networks by introducing intentional frequency offsets (FOs) to the transmitted signals. In this paper, we explore the use of (POT) with reinforcement learning (RL) in dense networks where multiple links access time-frequency resources simultaneously. We propose a novel framework based on Q-learning, to obtain the (FO) for the multi-carrier waveform used for each link. In particular, we consider filtered multi-tone (FMT) systems that utilize Gaussian, root-raised-cosine (RRC), and isotropic orthogonal transform algorithm (IOTA) based prototype filters. Our simulation results show that the proposed scheme enhances the capacity of the links by at least 30\% in additive white Gaussian noise (AWGN) channel at high signal-to-noise ratio (SNR), and even more so in the presence of severe multi-path fading. For a wide range of interfering link densities, we demonstrate substantial improvements in the outage probability and multi-user efficiency facilitated by (POT), with the Gaussian filter outperforming the other two filters.