In this article we generalize the concepts that were used in the PhD thesis of Drudge to classify Cameron-Liebler line classes in PG$(n,q), n\geq 3$, to Cameron-Liebler sets of $k$-spaces in PG$(n,q)$ and AG$(n,q)$. In his PhD thesis, Drudge proved that every Cameron-Liebler line class in PG$(n,q)$ intersects every $3$-dimensional subspace in a Cameron-Liebler line class in that subspace. We are using the generalization of this result for sets of $k$-spaces in PG$(n,q)$ and AG$(n,q)$. Together with a basic counting argument this gives a very strong non-existence condition, $n\geq 3k+3$. This condition can also be improved for $k$-sets in AG$(n,q)$, with $n\geq 2k+2$.