We show that every finitely generated extension by $\mathbb{Z}$ of a locally normally finite group has Shalom's property $H_{\mathrm{FD}}$. This is no longer true without the normality assumption. This permits to answer some questions of Shalom, Erschler-Ozawa and Kozma. We also obtain a Neumann-Neumann embedding result that any countable locally finite group embedds into a two generated amenable group with property $H_{\mathrm{FD}}$.