We give an elementary proof for the non-$3$-colorability of $4$-chromatic Schrijver graphs thus providing such a proof also for $4$-chromatic Kneser graphs. To this end we use a complete description of the structure of $4$-chromatic Schrijver graphs that was already given by Braun and even earlier in an unpublished manuscript by Li. We also address connections to surface quadrangulations. In particular, we show that a spanning subgraph of $4$-chromatic Schrijver graphs quadrangulates the Klein bottle, while another spanning subgraph quadrangulates the projective plane. The latter is a special case of a result by Kaiser and Stehlík. We characterize the color-critical edges of $4$-chromatic Schrijver graphs and also present preliminary results toward the characterization of color-critical edges in Schrijver graphs of higher chromatic number. Finally, we show that (apart from two cases of small parameters) the subgraphs we present that quadrangulate the Klein bottle are edge-color-critical. The analogous result for the subgraphs quadrangulating the projective plane is an immediate consequence of earlier results by Gimbel and Thomassen and was already noted by Kaiser and Stehlík.