Recent advances in pre-trained language models (PLMs) have demonstrated their capabilities in capturing universal knowledge, making them promising for radar signal processing applications. Nevertheless, directly fine-tuning PLMs on radar signals is both computationally expensive and prone to overfitting, particularly in low signal-to-clutter ratio (SCR) environments. To mitigate both issues, an effective fine-tuning framework for PLM-based marine radar target detection is proposed. First, we design a lightweight adaptation module, enabling computationally efficient fine-tuning while preserving the pre-trained model's general knowledge. Second, an effective selective fine-tuning strategy is developed to selectively optimize different feature patches based on their online-evaluated learning values, guiding the model to concentrate on those generalizable feature patterns and significantly reducing model overfitting to nosiy, anomalous, or overly simple patterns during optimization. Finally, a binary classification head is retrained based on autoencoder network to further enhance detection performance. Evaluations on real-world radar datasets highlight that the proposed RadarPLM framework considerably outperforms existing models, achieving a minimum of 6.35% gain in average detection performance under challenging low SCR conditions when using sequence features. In particular, under small-sample training conditions, RadarPLM also achieves highly significant average performance gains over prior methods, demonstrating the effectiveness of integrating the PLM.