Restricted latent class models (RLCMs) have recently gained prominence in educational assessment, psychiatric evaluation, and medical diagnosis. Different from conventional latent class models, restrictions on the RLCM model parameters are imposed by a design matrix to respect practitioners' scientific assumptions. The design matrix, called $Q$-matrix in the cognitive diagnosis literature, is usually constructed by practitioners and domain experts, yet it is subjective and could be misspecified. To address this problem, researchers have proposed to estimate the $Q$-matrix from data. On the other hand, the fundamental learnability issue of the $Q$-matrix and model parameters remains underexplored and existing studies often impose stronger than needed or even impractical conditions. This paper proposes sufficient and necessary conditions for joint identifiability of the $Q$-matrix and the RLCM model parameters under different types of RLCMs. The developed identifiability conditions only depend on the design matrix and are easy to verify in practice.