One important task in the study of information cascade is to predict the future recipients of a message given its past spreading trajectory. While the network structure serves as the backbone of the spreading, an accurate prediction can hardly be made without the knowledge of the dynamics on the network. The temporal information in the spreading sequence captures many hidden features, but predictions based on sequence alone have their limitations. Recent efforts start to explore the possibility of combining both the network structure and the temporal feature. Here, we propose a new end-to-end prediction method CasSeqGCN in which the structure and temporal feature are simultaneously taken into account. A cascade is divided into multiple snapshots which record the network topology and the state of nodes. The graph convolutional network (GCN) is used to learn the representation of a snapshot. A novel aggregation method based on dynamic routing is proposed to aggregate node representation and the long short-term memory (LSTM) model is used to extract temporal information. CasSeqGCN predicts the future cascade size more accurately compared with other state-of-art baseline methods. The ablation study demonstrates that the improvement mainly comes from the design of the input and the GCN layer. We explicitly design an experiment to show the quality of the cascade representation learned by our approach is better than other methods. Our work proposes a new approach to combine the structural and temporal features, which not only gives a useful baseline model for future studies of cascade prediction, but also brings new insights on a wide collection of problems related with dynamics on and of the network.