If $S=(s_1,s_2,\ldots)$ is a non-decreasing sequence of positive integers, then the $S$-packing $k$-coloring of a graph $G$ is a mapping $c: V(G)\rightarrow[k]$ such that if $c(u)=c(v)=i$ for $u\neq v\in V(G)$, then $d_G(u,v)>s_i$. The $S$-packing chromatic number of $G$ is the smallest integer $k$ such that $G$ admits an $S$-packing $k$-coloring. A graph $G$ is $χ_S$-vertex-critical if $χ_S(G-u) < χ_S(G)$ for each $u\in V(G)$. If $G$ is $χ_S$-vertex-critical and $χ_S(G) = k$, then $G$ is $k$-$χ_S$-vertex-critical. In this paper, $4$-$χ_S$-vertex-critical graphs are characterized for sequences $S = (1,s_2, s_3, \ldots)$ with $s_2 \ge 3$. There are $28$ sporadic examples and two infinite families of such graphs.