Given a graph $G$, a coloring $c:V(G)\longrightarrow \{1,\ldots,k\}$ such that $c(u)=c(v)=i$ implies that vertices $u$ and $v$ are at distance greater than $i$, is called a packing coloring of $G$. The minimum number of colors in a packing coloring of $G$ is called the packing chromatic number of $G$, and is denoted by $χ_ρ(G)$. In this paper, we propose the study of $χ_ρ$-critical graphs, which are the graphs $G$ such that for any proper subgraph $H$ of $G$, $χ_ρ(H)<χ_ρ(G)$. We characterize $χ_ρ$-critical graphs with diameter 2, and $χ_ρ$-critical block graphs with diameter 3. Furthermore, we characterize $χ_ρ$-critical graphs with small packing chromatic numbers, and we also consider $χ_ρ$-critical trees. In addition, we prove that for any graph $G$ with $e\in E(G)$, we have $(χ_ρ(G)+1)/2\le χ_ρ(G-e)\le χ_ρ(G)$, and provide a corresponding realization result, which shows that $χ_ρ(G-e)$ can achieve any of the integers between the bounds.