A graph $G$ is called a \emph{block graph} if each maximal $2$-connected component of $G$ is a clique. In this paper we study the Block Graph Vertex Deletion from the perspective of fixed parameter tractable (FPT) and kernelization algorithm. In particular, an input to Block Graph Vertex Deletion consists of a graph $G$ and a positive integer $k$ and the objective to check whether there exists a subset $S \subseteq V(G)$ of size at most $k$ such that the graph induced on $V(G)\setminus S$ is a block graph. In this paper we give an FPT algorithm with running time $4^kn^{O(1)}$ and a polynomial kernel of size $O(k^4)$ for Block Graph Vertex Deletion. The running time of our FPT algorithm improves over the previous best algorithm for the problem that ran in time $10^kn^{O(1)}$ and the size of our kernel reduces over the previously known kernel of size $O(k^9)$. Our results are based on a novel connection between Block Graph Vertex Deletion and the classical {\sc Feedback Vertex Set} problem in graphs without induced $C_4$ and $K_4-e$. To achieve our results we also obtain an algorithm for {\sc Weighted Feedback Vertex Set} running in time $3.618^kn^{O(1)}$ and improving over the running time of previously known algorithm with running time $5^kn^{O(1)}$.