A novel structure of polar codes is proposed for finite-state modulation (FSM), in order to improve polarization under it, and approach the polarization efficiency that conventional polar codes achieve under memoryless channels. We choose a particular class of FSM for research, termed bijective FSM, and observe an explicit polarization loss under bijective FSM. To eliminate the loss, we propose a novel polar coding structure by substituting the last kernel of each layer in polar coding structure with a swapping matrix, thereby termed last-pair swapping structure. We prove that under bijective FSM the proposed structure achieves identical polarization efficiency with that of conventional one on memoryless channels, and exceeds that of conventional one under bijective FSM. Furthermore, we give a plausible generalization of last-pair swapping polar code: on a broader class termed sub-injective FSM. Simulation corroborates that under sub-injective FSM polarization efficiency of the proposed polar code exceeds that of conventional one. And simulation results of error rate are given on continuous phase modulation (CPM) with additional white Gaussian noise (AWGN) channels, showing a considerable signal-to-noise power ratio (snr) gain of last-pair swapping polar code over conventional one, and identical performances between the proposed polar code under bijective FSM and conventional one on memoryless channels.