In this paper we answer a question of Kaimanovich by characterizing (jointly) bi-harmonic functions on countable, discrete groups with respect to a symmetric, generating measure. We also study the peripheral Poisson boundary of $L(\G)$ with respect to Markov operators arising from symmetric, generating probability measures on a countable, discrete group $\G$. We solve a recent conjecture of Bhat, Talwar and Kar regarding peripheral eigenvalues and their corresponding eigenvectors for such Markov operators, and provide a complete description of the peripheral Poisson boundary in the aforementioned scenario.