Homomorphic encryption (HE) enables privacy-preserving deep learning by allowing computations on encrypted data without decryption. However, deploying convolutional neural networks (CNNs) with HE is challenging due to the need to convert input data into a two-dimensional matrix for convolution using the im2col technique, which rearranges the input for efficient computation. This restricts the types of CNN models that can be used since the encrypted data structure must be compatible with the specific model. UniHENN is a novel HE-based CNN architecture that eliminates the need for im2col, enhancing its versatility and compatibility with a broader range of CNN models. UniHENN flattens input data to one dimension without using im2col. The kernel performs convolutions by traversing the image, using incremental rotations and structured multiplication on the flattened input, with results spaced by the stride interval. Experimental results show that UniHENN significantly outperforms the state-of-the-art 2D CNN inference architecture named PyCrCNN in terms of inference time. For example, on the LeNet-1 model, UniHENN achieves an average inference time of 30.089 seconds, about 26.6 times faster than PyCrCNN's 800.591 seconds. Furthermore, UniHENN outperforms TenSEAL, an im2col-optimized CNN model, in concurrent image processing. For ten samples, UniHENN (16.247 seconds) was about 3.9 times faster than TenSEAL (63.706 seconds), owing to its support for batch processing of up to 10 samples. We demonstrate UniHENN's adaptability to various CNN architectures, including a 1D CNN and six 2D CNNs, highlighting its flexibility and efficiency for privacy-preserving cloud-based CNN services.