The article presents the system architecture for automatic decoding of railway track defectograms in real time. The system includes an ultrasound data preprocessing module, a set of neutral network classifiers, a decision block. Preprocessing of data includes affine transformations of measurement information into a format suitable for the operation of a neural network, as well as a combination of information on measurement channels, depending on the type of defect being defined. The classifier is built on a convolutional neural network. The proposed solution can be effectively implemented on a modern elemental basis for performing parallel computing, including tensor processor and GPUs.