In [Discrete Mathematics 306 (2005) 153-158], So proposed a conjecture saying that integral circulant graphs with different connection sets have different spectra. This conjecture is still open. We prove that this conjecture holds for integral circulant graphs whose orders have prime factorization of $4$ types.