The objective of this study is to address the mobility challenges faced by user equipment (UE) through the implementation of fluid antenna (FA) on the UE side. This approach aims to maintain the time-varying channel in a relatively stable state by strategically relocating the FA to an appropriate port. To the best of our knowledge, this paper introduces, for the first time, the application of large language models (LLMs) in the prediction of FA ports, presenting a novel model termed Port-LLM. Our proposed method for predicting the moving port of the FA is a two-step prediction method. To enhance the learning efficacy of our proposed Port-LLM model, we integrate low-rank adaptation (LoRA) fine-tuning technology. Additionally, to further exploit the natural language processing capabilities of pre-trained LLMs, we propose a framework named Prompt-Port-LLM, which is constructed upon the Port-LLM architecture and incorporates prompt fine-tuning techniques along with a specialized prompt encoder module. The simulation results show that our proposed models all exhibit strong generalization ability and robustness under different numbers of base station antennas and medium-to-high mobility speeds of UE. In comparison to existing methods, the performance of the port predicted by our models demonstrates superior efficacy. Moreover, both of our proposed models achieve millimeter-level inference speed.