The beyond-diagonal reconfigurable intelligent surface (BD-RIS) is a recent architecture in which scattering elements are interconnected to enhance the degrees of freedom for wave control, yielding performance gains over traditional single-connected RISs. For BD-RIS, channel estimation, which is well studied for conventional RIS, becomes more challenging due to complex connections and a larger number of coefficients. Previous works relied on pilot-assisted estimation followed by data decoding. This paper introduces a semi-blind tensor-based approach to joint channel and symbol estimation that eliminates the need for training sequences by directly leveraging data symbols. A practical scenario with time-varying user terminal-RIS channels under mobility is considered. By reformulating the received signal from a tensor-decomposition perspective, we develop two semi-blind receivers: a two-stage method that transforms the fourth-order PARATUCK model into a third-order PARAFAC model, and a single-stage iterative process based on the fourth-order TUCKER decomposition. Identifiability conditions for reliable joint recovery are derived, and numerical results demonstrate the performance advantages and trade-offs of the proposed schemes over existing solutions.