A generative adversarial network (GAN) based fully automatic electrocardiogram (ECG) arrhythmia classification system with high performance is presented in this paper. The generator (G) in our GAN is designed to generate various coupling matrix inputs conditioned on different arrhythmia classes for data augmentation. Our designed discriminator (D) is trained on both real and generated ECG coupling matrix inputs, and is extracted as an arrhythmia classifier upon completion of training for our GAN. After fine-tuning the D by including patient-specific normal beats estimated using an unsupervised algorithm, and generated abnormal beats by G that are usually rare to obtain, our fully automatic system showed superior overall classification performance for both supraventricular ectopic beats (SVEB or S beats) and ventricular ectopic beats (VEB or V beats) on the MIT-BIH arrhythmia database. It surpassed several state-of-art automatic classifiers and can perform on similar levels as some expert-assisted methods. In particular, the F1 score of SVEB has been improved by up to 13% over the top-performing automatic systems. Moreover, high sensitivity for both SVEB (87%) and VEB (93%) detection has been achieved, which is of great value for practical diagnosis. We, therefore, suggest our ACE-GAN (Generative Adversarial Network with Auxiliary Classifier for Electrocardiogram) based automatic system can be a promising and reliable tool for high throughput clinical screening practice, without any need of manual intervene or expert assisted labeling.