Promptly discovering unknown network attacks is critical for reducing the risk of major loss imposed on system or equipment. This paper aims to develop an open-set intrusion detection model to classify known attacks as well as inferring unknown ones. To achieve this, we employ OpenMax and variational autoencoder to propose a dual detection model, VAEMax. First, we extract flow payload feature based on one-dimensional convolutional neural network. Then, the OpenMax is used to classify flows, during which some unknown attacks can be detected, while the rest are misclassified into a certain class of known flows. Finally, use VAE to perform secondary detection on each class of flows, and determine whether the flow is an unknown attack based on the reconstruction loss. Experiments performed on dataset CIC-IDS2017 and CSE-CIC-IDS2018 show our approach is better than baseline models and can be effectively applied to realistic network environments.