With the rapid information explosion on online social network sites (SNSs), it becomes difficult for users to seek new friends or broaden their social networks in an efficient way. Link prediction, which can effectively conquer this problem, has thus attracted wide attention. Previous methods on link prediction fail to comprehensively capture the factors leading to new link formation: 1) few models have considered the varied impacts of users' short-term and long-term interests on link prediction. Besides, they fail to jointly model the influence from social influence and "weak links"; 2) considering that different factors should be derived from information sources of different modalities, there is a lack of effective multi-modal framework for link prediction. In this view, we propose a novel multi-modal framework for link prediction (referred as LP-UIT) which fuses a comprehensive set of features (i.e., user information and topological features) extracted from multi-modal information (i.e., textual information, graph information, and numerical information). Specifically, we adopt graph convolutional network to process the network information to capture topological features, employ natural language processing techniques (i.e., TF-IDF and word2Vec) to model users' short-term and long-term interests, and identify social influence and "weak links" from numerical features. We further use an attention mechanism to model the relationship between textual and topological features. Finally, a multi-layer perceptron (MLP) is designed to combine the representations from three modalities for link prediction. Extensive experiments on two real-world datasets demonstrate the superiority of LP-UIT over the state-of-the-art methods.