Cocyclic Hadamard matrices (CHMs) were introduced by de Launey and Horadam as a class of Hadamard matrices with interesting algebraic properties. Ó Catháin and Röder described a classification algorithm for CHMs of order $4n$ based on relative difference sets in groups of order $8n$; this led to the classification of all CHMs of order at most 36. Based on work of de Launey and Flannery, we describe a classification algorithm for CHMs of order $4p$ with $p$ a prime; we prove refined structure results and provide a classification for $p \leqslant 13$. Our analysis shows that every CHM of order $4p$ with $p\equiv 1\bmod 4$ is equivalent to a Hadamard matrix with one of five distinct block structures, including Williamson type and (transposed) Ito matrices. If $p\equiv 3 \bmod 4$, then every CHM of order $4p$ is equivalent to a Williamson type or (transposed) Ito matrix.