An impurity measures $I:{R}^k \to {R}^+$ maps a $k$-dimensional vector ${\bf v}$ to a non-negative value $I({\bf v})$ so that the more homogeneous ${\bf v}$, the larger its impurity. We study clustering based on impurity measures: given a collection $V$ of $n$ many $k$-dimensional vectors and an impurity measure $I$, the goal is to find a partition ${\cal P}$ of $V$ into $L$ groups $V_1,\ldots,V_L$ that minimizes the total impurities of the groups in ${\cal P}$, i.e., $I({\cal P})= \sum_{m=1}^{L} I(\sum_{{\bf v} \in V_m}{\bf v}).$ Impurity minimization is widely used as quality assessment measure in probability distribution clustering and in categorical clustering where it is not possible to rely on geometric properties of the data set. However, in contrast to the case of metric based clustering, the current knowledge of impurity measure based clustering in terms of approximation and inapproximability results is very limited. Our research contributes to fill this gap. We first present a simple linear time algorithm that simultaneously achieves $3$-approximation for the Gini impurity measure and $O(\log(\sum_{{\bf v} \in V} \| {\bf v} \|_1))$-approximation for the Entropy impurity measure. Then, for the Entropy impurity measure---where we also show that finding the optimal clustering is strongly NP-hard---we are able to design a polynomial time $O(\log^2(\min\{k,L\}))$-approximation algorithm. Our algorithm relies on a nontrivial characterization of a class of clusterings that necessarily includes a partition achieving $O(\log^2(\min\{k,L\}))$--approximation of the impurity of the optimal partition. Remarkably, this is the first polynomial time algorithm with approximation guarantee independent of the number of points/vector and not relying on any restriction on the components of the vectors for producing clusterings with minimum entropy.