Beyond diagonal reconfigurable intelligent surface (BD-RIS) has emerged as an advancement and generalization of the conventional diagonal RIS (D-RIS) by introducing tunable interconnections between RIS elements, enabling smarter wave manipulation and enlarged coverage. While BD-RIS has demonstrated advantages over D-RIS in various aspects, most existing works rely on the assumption of a lossless model, leaving practical considerations unaddressed. This paper thus proposes a lossy BD-RIS model and develops corresponding optimization algorithms for various BD-RIS-aided communication systems. First, by leveraging admittance parameter analysis, we model each tunable admittance based on a lumped circuit with losses and derive an expression of a circle characterizing the real and imaginary parts of each tunable admittance. We then consider the received signal power maximization in single-user single-input single-output (SISO) systems with the proposed lossy BD-RIS model. To solve the optimization problem, we design an effective algorithm by carefully exploiting the problem structure. Specifically, an alternating direction method of multipliers (ADMM) framework is custom-designed to deal with the complicated constraints associated with lossy BD-RIS. Furthermore, we extend the proposed algorithmic framework to more general multiuser multiple-input single-output (MU-MISO) systems, where the transmit precoder and BD-RIS scattering matrix are jointly designed to maximize the sum-rate of the system. Finally, simulation results demonstrate that all BD-RIS architectures still outperform D-RIS in the presence of losses, but the optimal BD-RIS architectures in the lossless case are not necessarily optimal in the lossy case, e.g. group-connected BD-RIS can outperform fully- and tree-connected BD-RISs in SISO systems with relatively high losses, whereas the opposite always holds true in the lossless case.