This study targets to automatically annotate on arrhythmia by deep network. The investigated types include sinus rhythm, asystole (Asys), supraventricular tachycardia (Tachy), ventricular flutter or fibrillation (VF/VFL), ventricular tachycardia (VT). Methods: 13s limb lead ECG chunks from MIT malignant ventricular arrhythmia database (VFDB) and MIT normal sinus rhythm database were partitioned into subsets for 5-fold cross validation. These signals were resampled to 200Hz, filtered to remove baseline wandering, projected to 2D gray spectrum and then fed into a deep network with brand-new structure. In this network, a feature vector for a single time point was retrieved by residual layers, from which latent representation was extracted by variational autoencoder (VAE). These front portions were trained to meet a certain threshold in loss function, then fixed while training procedure switched to remaining bidirectional recurrent neural network (RNN), the very portions to predict an arrhythmia category. Attention windows were polynomial lumped on RNN outputs for learning from details to outlines. And over sampling was employed for imbalanced data. The trained model was wrapped into docker image for deployment in edge or cloud. Conclusion: Promising sensitivities were achieved in four arrhythmias and good precision rates in two ventricular arrhythmias were also observed. Moreover, it was proven that latent representation by VAE, can significantly boost the speed of convergence and accuracy.