Let $\boldsymbol{X}\in \mathbb{R}^p$ and $Y\in \mathbb{R}$. In this paper we propose an estimator of the conditional covariance matrix, $\mathrm{Cov}(\mathbb{E}[\boldsymbol{X}\vert Y])$, in an inverse regression setting. Based on the estimation of a quadratic functional, this methodology provides an efficient estimator from a semi parametric point of view. We consider a functional Taylor expansion of $\mathrm{Cov}(\mathbb{E}[\boldsymbol{X}\vert Y])$ under some mild conditions and the effect of using an estimate of the unknown joint distribution. The asymptotic properties of this estimator are also provided.