The Poisson distribution of order $k$ is a special case of a compound Poisson distribution. For $k=1$ it is the standard Poisson distribution. Although its probability mass function (pmf) is known, what is lacking is a $visual$ interpretation, which a sum over terms with factorial denominators does not supply. Unlike the standard Poisson distribution, the Poisson distribution of order $k$ can display a maximum of $four$ peaks simultaneously, as a function of two parameters: the order $k$ and the rate parameter $λ$. This note characterizes the shape of the pmf of the Poisson distribution of order $k$. The pmf can be partitioned into a single point at $n=0$, an increasing sequence for $n \in [1,k]$ and a mountain range for $n>k$ (explained in the text). The ``parameter space'' of the pmf is mapped out and the significance of each domain is explained, in particular the change in behavior of the pmf as a domain boundary is crossed. A simple analogy (admittedly unrelated) is that of the discriminant of a quadratic with real coefficients: its domains characterize the nature of the roots (real or complex), and the domain boundary signifies the presence of a repeated root. Something similar happens with the pmf of the Poisson distribution of order $k$. As an application, this note explains the mode structure of the Poisson distribution of order $k$. Improvements to various inequalities are also derived (sharper bounds, etc.). New conjectured upper and lower bounds for the median and the mode are also proposed.