Let $X_1, \ldots, X_n$ be i.i.d. sample in $\mathbb{R}^p$ with zero mean and the covariance matrix $\mathbf{Σ^*}$. The classical PCA approach recovers the projector $\mathbf{P^*_{\mathcal{J}}}$ onto the principal eigenspace of $\mathbf{Σ^*}$ by its empirical counterpart $\mathbf{\widehat{P}_{\mathcal{J}}}$. Recent paper [Koltchinskii, Lounici (2017)] investigated the asymptotic distribution of the Frobenius distance between the projectors $\| \mathbf{\widehat{P}_{\mathcal{J}}} - \mathbf{P^*_{\mathcal{J}}} \|_2$, while [Naumov et al. (2017)] offered a bootstrap procedure to measure uncertainty in recovering this subspace $\mathbf{P^*_{\mathcal{J}}}$ even in a finite sample setup. The present paper considers this problem from a Bayesian perspective and suggests to use the credible sets of the pseudo-posterior distribution on the space of covariance matrices induced by the conjugated Inverse Wishart prior as sharp confidence sets. This yields a numerically efficient procedure. Moreover, we theoretically justify this method and derive finite sample bounds on the corresponding coverage probability. Contrary to [Koltchinskii, Lounici (2017), Naumov et al. (2017)], the obtained results are valid for non-Gaussian data: the main assumption that we impose is the concentration of the sample covariance $\mathbf{\widehatΣ}$ in a vicinity of $\mathbf{Σ^*}$. Numerical simulations illustrate good performance of the proposed procedure even on non-Gaussian data in a rather challenging regime.