The packing chromatic number $χ_ρ(G)$ of a graph $G$ is the smallest integer $k$ such that the vertex set of $G$ can be partitioned into sets $V_i$, $i\in [k]$, where vertices in $V_i$ are pairwise at distance at least $i+1$. Packing chromatic vertex-critical graphs, $χ_ρ$-critical for short, are introduced as the graphs $G$ for which $χ_ρ(G-x) < χ_ρ(G)$ holds for every vertex $x$ of $G$. If $χ_ρ(G) = k$, then $G$ is $k$-$χ_ρ$-critical. It is shown that if $G$ is $χ_ρ$-critical, then the set $\{χ_ρ(G) - χ_ρ(G-x):\ x\in V(G)\}$ can be almost arbitrary. The $3$-$χ_ρ$-critical graphs are characterized, and $4$-$χ_ρ$-critical graphs are characterized in the case when they contain a cycle of length at least $5$ which is not congruent to $0$ modulo $4$. It is shown that for every integer $k\ge 2$ there exists a $k$-$χ_ρ$-critical tree and that a $k$-$χ_ρ$-critical caterpillar exists if and only if $k\le 7$. Cartesian products are also considered and in particular it is proved that if $G$ and $H$ are vertex-transitive graphs and ${\rm diam(G)} + {\rm diam}(H) \le χ_ρ(G)$, then $G\,\square\, H$ is $χ_ρ$-critical.