Permutation decoding is a technique which involves finding a subset $S$, called PD-set, of the permutation automorphism group of a code $C$ in order to assist in decoding. An explicit construction of $\left \lfloor{\frac{2^m-m-1}{1+m}} \right \rfloor$-PD-sets of minimum size $\left \lfloor{\frac{2^m-m-1}{1+m}} \right \rfloor + 1$ for partial permutation decoding for binary linear Hadamard codes $H_m$ of length $2^m$, for all $m\geq 4$, is described. Moreover, a recursive construction to obtain $s$-PD-sets of size $l$ for $H_{m+1}$ of length $2^{m+1}$, from a given $s$-PD-set of the same size for $H_m$, is also established. These results are generalized to find $s$-PD-sets for (nonlinear) binary Hadamard codes of length $2^m$, called $\mathbb{Z}_4$-linear Hadamard codes, which are obtained as the Gray map image of quaternary linear codes of length $2^{m-1}$.