In this paper, a secure Convolutional Neural Network classifier is proposed using Fully Homomorphic Encryption (FHE). The secure classifier provides a user with the ability to out-source the computations to a powerful cloud server and/or setup a server to classify inputs without providing the model or revealing source data. To this end, a real number framework is developed over FHE by using a fixed point format with binary digits. This allows for real number computations for basic operators like addition, subtraction, and multiplication but also to include secure comparisons and max functions. Additionally, a rectified linear unit is designed and realized in the framework. Experimentally, the model was verified using a Convolutional Neural Network trained for handwritten digits. This encrypted implementation shows accurate results for all classification when compared against an unencrypted implementation.