Photonic technologies have shown a promising way to build high-speed and high-energy-efficiency neural network accelerators. In previously presented photonic neural networks, architectures are mainly designed for fully-connected layers. When convolutional layers are executed in such neural networks, the large-scale electrooptic modulation array heavily increases the energy dissipation on chip. To increase the energy efficiency, here we show an integrated photonic architecture specifically for convolutional layer calculations. Optical delay lines replace electronics to execute data manipulations on optical chip, reducing the scale of electro-optic modulation array. Consequently, the energy dissipation of these parts is mitigated. Powered by wavelength division multiplexing, the footprint of delay lines is significantly reduced compared with previous art, thus being practical to fabricate. We evaluate the potential performance of the proposed architecture with respect to component flaws in practical fabrications. According to the results, with well-controlled system insertion loss, energy efficiency of the proposed architecture would surpass previously presented works and the state-of-art electronic processors. We anticipate the proposed architecture is beneficial for future fast and energy-efficient convolutional neural network accelerators.