This paper is concerned with the convergence and the error analysis for the feedback particle filter (FPF) algorithm. The FPF is a controlled interacting particle system where the control law is designed to solve the nonlinear filtering problem. For the linear Gaussian case, certain simplifications arise whereby the linear FPF reduces to one form of the ensemble Kalman filter. For this and for the more general nonlinear non-Gaussian case, it has been an open problem to relate the convergence and error properties of the finite-N algorithm to the mean-field limit (where the exactness results have been obtained). In this paper, the equations for empirical mean and covariance are derived for the finite-N linear FPF. Remarkably, for a certain deterministic form of FPF, the equations for mean and variance are identical to the Kalman filter. This allows strong conclusions on convergence and error properties based on the classical filter stability theory for the Kalman filter. It is shown that the error converges to zero even with finite number of particles. The paper also presents propagation of chaos estimates for the finite-N linear filter. The error estimates are illustrated with numerical experiments.