In this article, we study a class of closed connected orientable PL $4$-manifolds admitting a semi-simple crystallization and which have an infinite cyclic fundamental group. We show that the manifold in the class admits a handle decomposition in which the number of $2$-handles depends upon its second Betti number and other $h$-handles ($h \leq 4$) are at most $2$. More precisely, our main result is the following. For a closed connected orientable PL $4$-manifold having a semi-simple crystallization with the fundamental group as $\mathbb{Z}$, we have constructed a handle decomposition for $M$ as one of the following types: $(1)$ one $0$-handle, two $1$-handles, $1+β_2(M)$ $2$-handles, one $3$-handle and one $4$-handle, $(2)$ one $0$-handle, one $1$-handle, $β_2(M)$ $2$-handles, one $3$-handle and one $4$-handle, where $β_2(M)$ denotes the second Betti number of manifold $M$ with $\mathbb{Z}$ coefficients.