As constructing multi-D wavelets remains a challenging problem, we propose a new method called prime coset sum to construct multi-D wavelets. Our method provides a systematic way to construct multi-D non-separable wavelet filter banks from two 1-D lowpass filters, with one of whom being interpolatory. Our method has many important features including the following: 1) it works for any spatial dimension, and any prime scalar dilation, 2) the vanishing moments of the multi-D wavelet filter banks are guaranteed by certain properties of the initial 1-D lowpass filters, and furthermore, 3) the resulting multi-D wavelet filter banks are associated with fast algorithms that are faster than the existing fast tensor product algorithms.