For a graph $G$ with vertex set $V(G)$ and a positive integer $i$, an $i$-packing in $G$ is a subset $X$ of $V(G)$ such that the distance between any two distinct vertices of $X$ is greater than $i$. The packing chromatic number of $G$, denoted by $χ_ρ(G)$, is the smallest positive integer $k$ for which there exists a partition $X_1, X_2, \ldots, X_k$ of $V(G)$ such that $X_i$ is an $i$-packing in $G$ for every $i \in [k]$. A graph $G$ is called $χ_ρ$-critical if $χ_ρ(H) < χ_ρ(G)$ holds for every proper subgraph $H$ of $G$. In this paper, we provide a structural characterization of $χ_ρ$-critical graphs with radius $1$, and completely determine the $χ_ρ$-critical cactus graphs with radius $2$ and diameter $2$ or $3$.