In the Tree Deletion Set problem the input is a graph G together with an integer k. The objective is to determine whether there exists a set S of at most k vertices such that G-S is a tree. The problem is NP-complete and even NP-hard to approximate within any factor of OPT^c for any constant c. In this paper we give a O(k^4) size kernel for the Tree Deletion Set problem. To the best of our knowledge our result is the first counterexample to the "conventional wisdom" that kernelization algorithms automatically provide approximation algorithms with approximation ratio close to the size of the kernel. An appealing feature of our kernelization algorithm is a new algebraic reduction rule that we use to handle the instances on which Tree Deletion Set is hard to approximate.