We propose two types, namely Type-I and Type-II, quantum stabilizer codes using quadratic residue sets of prime modulus given by the form $p=4n\pm1$. The proposed Type-I stabilizer codes are of cyclic structure and code length $N=p$. They are constructed based on multi-weight circulant matrix generated from idempotent polynomial, which is obtained from a quadratic residue set. The proposed Type-II stabilizer codes are of quasi-cyclic (QC) structure and code length $N=pk$, where $k$ is the size of a quadratic residue set. They are constructed based on structured sparse-graphs codes derived from proto-matrix and circulant permutation matrix. With the proposed methods, we design rich classes of cyclic and quasi-cyclic quantum stabilizer codes with variable code length. We show how the commutative constraint (also referred to as the Symplectic Inner Product constraint) for quantum codes can be satisfied for each proposed construction method. We also analyze both the dimension and distance for Type-I stabilizer codes and the dimension of Type-II stabilizer codes. For the cyclic quantum stabilizer codes, we show that they meet the existing distance bounds in literature.