In this paper, a new precoder for a coherent optical filter bank multicarrier with offset quadrature amplitude modulation (FBMC/OQAM) system is proposed. The precoder is designed based on an iterative polynomial eigenvalue decomposition (PEVD) algorithm to jointly mitigate the inter-symbol interference (ISI) and inter-carrier interference (ICI). The PEVD algorithm is used to decompose a polynomial channel matrix into polynomial eigenvectors and eigenvalues matrices, then a precoder is designed based on the decomposed matrices. The precoder acts as a filter applied to each subcarrier and to its adjacent subcarriers. In addition, a new adaptive optimum energy algorithm is proposed to truncate the insignificant precoder filter coefficients produced by the PEVD algorithm and consequently reduce the computational complexity. A mathematical model and algorithm for implementing the precoder are also presented. Robustness of the proposed precoder has been analyzed and compared to existing precoder in optical channel with different dispersion effects. Numerical results demonstrate that the new precoder significantly outperforms the existing precoder in terms of error vector magnitude (EVM), bit error rate (BER), Frobenius norm of the error matrix, and computational complexity.