Gasharov introduced the combinatorial objects known as $P$-arrays to prove $s$-positivity for the chromatic symmetric functions of incomparability graphs of (3+1)-free posets. We define a crystal, a directed colored graph with some additional axioms, on the set of $P$-arrays. The components of the crystal have $s$-positive characters, thereby refining the $s$-positivity theorems of Gasharov, as well as Shareshian and Wachs. The crystal hints at a possible generalization of the Robinson-Schensted correspondence applied to $P$-arrays.