Link prediction on dynamic networks has been extensively studied and widely applied in various applications. However, temporal unlink prediction, which also plays an important role in the evolution of social networks, has not been paid much attention. Accurately predicting the links and unlinks on the future network greatly contributes to the network analysis that uncovers more latent relations between nodes. In this work, we assume that there are two kinds of relations between nodes, namely long-term relation and short-term relation, and we propose an effective algorithm called LULS for temporal link prediction and unlink prediction based on such relations. Specifically, for each snapshot of a dynamic network, LULS first collects higher-order structure as two topological matrices by applying short random walks. Then, LULS initializes and optimizes a global matrix and a sequence of temporary matrices for all the snapshots by using non-negative matrix factorization (NMF) based on the topological matrices, where the global matrix denotes long-term relation and the temporary matrices represent short-term relations of snapshots. Finally, LULS calculates the similarity matrix of the future snapshot and predicts the links and unlinks for the future network. Additionally, we further improve the prediction results by using graph regularization constraints to enhance the global matrix, resulting that the global matrix contains a wealth of topological information and temporal information. The conducted experiments on real-world networks illustrate that LULS outperforms other baselines for both link prediction and unlink prediction tasks.