In Defective Coloring we are given a graph $G$ and two integers $χ_d$, $Δ^*$ and are asked if we can $χ_d$-color $G$ so that the maximum degree induced by any color class is at most $Δ^*$. We show that this natural generalization of Coloring is much harder on several basic graph classes. In particular, we show that it is NP-hard on split graphs, even when one of the two parameters $χ_d$, $Δ^*$ is set to the smallest possible fixed value that does not trivialize the problem ($χ_d = 2$ or $Δ^* = 1$). Together with a simple treewidth-based DP algorithm this completely determines the complexity of the problem also on chordal graphs. We then consider the case of cographs and show that, somewhat surprisingly, Defective Coloring turns out to be one of the few natural problems which are NP-hard on this class. We complement this negative result by showing that Defective Coloring is in P for cographs if either $χ_d$ or $Δ^*$ is fixed; that it is in P for trivially perfect graphs; and that it admits a sub-exponential time algorithm for cographs when both $χ_d$ and $Δ^*$ are unbounded.