Recently, an image scrambling encryption algorithm of pixel bit based on chaos map was proposed. Considering the algorithm as a typical binary image scrambling/permutation algorithm exerting on plaintext of size $M\times (8N)$, this paper proposes a novel optimal method to break it with some known/chosen-plaintexts. The spatial complexity and computational complexity of the attack are only $O(32\cdot MN)$ and $O(16\cdot n_0\cdot MN)$ respectively, where $n_0$ is the number of known/chosen-plaintexts used. The method can be easily extended to break any permutation-only encryption scheme exerting on plaintext of size $M\times N$ and with $L$ different levels of values. The corresponding spatial complexity and computational complexity are only $O(MN)$ and $O(\lceil\log_L(MN)\rceil \cdot MN)$ respectively. In addition, some specific remarks on the performance of the image scrambling encryption algorithm are presented.