We confirm a recent conjecture of Xin and Zhang, which establishes a simple product formula for the characteristic polynomial of an $(n-1) \times (n-1)$ tridiagonal matrix $C$. This characteristic polynomial arises from a recurrence relation that enumerates $n \times n$ nonnegative integer matrices with all row and column sums equal to $t$, also called the Ehrhart polynomial of the $n$th Birkhoff polytope. Moreover, we extend our method to broader families of tridiagonal matrices.