We consider graphs for which the non-backtracking matrix has defective eigenvalues, or graphs for which the matrix does not have a full set of eigenvectors. The existence of these values results in Jordan blocks of size greater than one, which we call nontrivial. We show a relationship between the eigenspaces of the non-backtracking matrix and the eigenspaces of a smaller matrix, completely classifying their differences among graphs with at most one cycle. Finally, we provide several constructions of infinite graph families that have nontrivial Jordan blocks for both this smaller matrix and the non-backtracking matrix.