The linear complexity and the $k$-error linear complexity of a sequence have been used as important security measures for key stream sequence strength in linear feedback shift register design. By studying the linear complexity of binary sequences with period $2^n$, one could convert the computation of $k$-error linear complexity into finding error sequences with minimal Hamming weight. Based on Games-Chan algorithm, the $k$-error linear complexity distribution of $2^n$-periodic binary sequences is investigated in this paper. First, for $k=2,3$, the complete counting functions on the $k$-error linear complexity of $2^n$-periodic balanced binary sequences (with linear complexity less than $2^n$) are characterized. Second, for $k=3,4$, the complete counting functions on the $k$-error linear complexity of $2^n$-periodic binary sequences with linear complexity $2^n$ are presented. Third, as a consequence of these results, the counting functions for the number of $2^n$-periodic binary sequences with the $k$-error linear complexity for $k = 2$ and 3 are obtained. Further more, an important result in a recent paper is proved to be not completely correct.