Due to its high spectral confinement characteristics and spectral efficiency, QAM-FBMC is considered a candidate waveform to replace CP-OFDM. QAM-FBMC has inevitable non-orthogonality both in time and frequency, and the system and filter must be well-designed to minimize the interferences. However, existing QAM-FBMC studies utilize a matched filter as the receiver filter, which is not suitable for a non-orthogonal system. Therefore, in this paper, we design the prototype filters considering the MMSE criterion, and propose a system providing the highest SINR in QAM-FBMC which cannot avoid non-orthogonality. In addition, we confirm that the proposed filters show best performance at target SNR than the reference filters.