This work investigates the modeling and optimization of beyond diagonal reconfigurable intelligent surface (BD-RIS), which generalizes conventional RIS with diagonal phase shift matrices and provides additional flexibility for manipulating wireless channels, in wideband communication systems. Specifically, we start from the signal modeling of the BD-RIS-aided orthogonal frequency division multiplexing (OFDM) system, which bridges the time-domain and frequency-domain channels, and explicitly shows the frequency dependence of the BD-RIS response. We next characterize the frequency dependence of the BD-RIS response based on circuit models. Benefiting from the admittance parameter analysis, we model individually each tunable admittance component of BD-RIS and derive an approximated linear expression with respect to the frequency of the transmit signals. With the proposed signal model for the BD-RIS-aided OFDM system and the frequency-dependent BD-RIS model, we propose algorithms to optimize the BD-RIS and the power allocation at the transmitter to maximize the average rate for a BD-RIS-aided OFDM system. Finally, simulation results show that BD-RIS outperforms conventional RIS in the OFDM system. More importantly, the impact of wideband modeling of BD-RIS on the system performance becomes more significant as the circuit complexity of BD-RIS architectures increases.