This letter considers a beyond diagonal reconfigurable intelligent surface (BD-RIS) aided integrated sensing and communication (ISAC) system, where the BD-RIS can help a multi-antenna base station (BS) serve multiple user equipments (UEs) and localize a target simultaneously. We formulate an optimization problem that designs the BS beamforming matrix and the BD-RIS scattering matrix to maximize UEs' sum rate subject to a localization Cramer-Rao bound (CRB) constraint and an additional unitary matrix constraint for the scattering matrix. Because unitary matrices form a manifold, our problem belongs to constrained manifold optimization. This letter proposes a log-barrier based Riemannian steepest ascent method to solve this problem effectively. Numerical results verify the effectiveness of our algorithm and the performance gain of the BD-RIS aided ISAC systems over the conventional RIS aided ISAC systems.