Deep learning is able to functionally simulate the human brain and thus, it has attracted considerable interest. Optics-assisted deep learning is a promising approach to improve the forward-propagation speed and reduce the power consumption. However, present methods are based on a parallel processing approach that is inherently ineffective in dealing with serial data signals at the core of information and communication technologies. Here, we propose and demonstrate a serial optical deep learning concept that is specifically designed to directly process high-speed temporal data. By utilizing ultra-short coherent optical pulses as the information carriers, the neurons are distributed at different time slots in a serial pattern, and interconnected to each other through group delay dispersion. A 4-layer serial optical neural network (SONN) was constructed and trained for classification of both analog and digital signals with simulated accuracy rates of over 90% with proper individuality variance rates. Furthermore, we performed a proof-of-concept experiment of a pseudo-3-layer SONN to successfully recognize the ASCII (American Standard Code for Information Interchange) codes of English letters at a data rate of 12 Gbps. This concept represents a novel one-dimensional realization of artificial neural networks, enabling an efficient application of optical deep learning methods to the analysis and processing of serial data signals, while offering a new overall perspective for the temporal signal processing.