The conventional digital beamforming technique needs one radio frequency (RF) chain per antenna element. High power consumption, significantly high cost of RF chain components per antenna and complex signal processing task at base band makes digital beamforming unsuitable for implementation in massive MIMO system. Hybrid beamforming takes the benefits of both analog and digital beamforming schemes. Near optimal performance can be achieved using very few RF chains in hybrid beamforming method. By optimizing the hybrid transmit and receive beamformers, the spectral efficiency of a system can be maximized. The hybrid beamforming optimization problem is non-convex optimization problem due to the non-convexity of the constraints. In this research work, the millimeter wave massive MIMO system performance implementing the hybrid precoding based on the conventional methods and deep neural network is studied considering the spectral efficiency, bit error rate and complexity. It is shown that the deep neural network based approach for hybrid precoding optimization outperforms conventional techniques by solving the non-convex optimization problem. Moreover, the DNN model accuracy is also analyzed by observing the training accuracy and test accuracy.