Scientific impact has been the center of extended debate regarding its accuracy and reliability. From hiring committees in academic institutions to governmental agencies that distribute funding, an author's scientific success as measured by the h-index is a vital point to their career. The objective of this work is to investigate whether the collaboration patterns of an author are good predictors of the author's future $h$-index. Although not directly related to each other, a more intense collaboration can result into increased productivity which can potentially have an impact on the author's future $h$-index. In this paper, we capitalize on recent advances in graph neural networks and we examine the possibility of predicting the $h$-index relying solely on the author's collaboration and the textual content of a subset of their papers. We perform our experiments on a large-scale network consisting of more than $1$ million authors that have published papers in computer science venues and more than $37$ million edges. The task is a six-months-ahead forecast, i.e. what the $h$-index of each author will be after six months. Our experiments indicate that there is indeed some relationship between the future $h$-index of an author and their structural role in the co-authorship network. Furthermore, we found that the proposed method outperforms standard machine learning techniques based on simple graph metrics along with node representations learned from the textual content of the author's papers.