We model the growth of a cell population by a piecewise deterministic Markov branching tree. Each cell splits into two offsprings at a division rate $B(x)$ that depends on its size $x$. The size of each cell grows exponentially in time, at a rate that varies for each individual. We show that the mean empirical measure of the model satisfies a growth-fragmentation type equation if structured in both size and growth rate as state variables. We construct a nonparametric estimator of the division rate $B(x)$ based on the observation of the population over different sampling schemes of size $n$ on the genealogical tree. Our estimator nearly achieves the rate $n^{-s/(2s+1)}$ in squared-loss error asymptotically. When the growth rate is assumed to be identical for every cell, we retrieve the classical growth-fragmentation model and our estimator improves on the rate $n^{-s/(2s+3)}$ obtained in \cite{DHRR, DPZ} through indirect observation schemes. Our method is consistently tested numerically and implemented on {\it Escherichia coli} data.