In this paper, we present infinite families of permutations of $\mathbb{F}_{2^{2n}}$ with high nonlinearity and boomerang uniformity $4$ from generalized butterfly structures. Both open and closed butterfly structures are considered. It appears, according to experiment results, that open butterflies do not produce permutation with boomerang uniformity $4$. For the closed butterflies, we propose the condition on coefficients $α, β\in \mathbb{F}_{2^n}$ such that the functions $$V_i := (R_i(x,y), R_i(y,x))$$ with $R_i(x,y)=(x+αy)^{2^i+1}+βy^{2^i+1}$ are permutations of $\mathbb{F}_{2^n}^2$ with boomerang uniformity $4$, where $n\geq 1$ is an odd integer and $\gcd(i, n)=1$. The main result in this paper consists of two major parts: the permutation property of $V_i$ is investigated in terms of the univariate form, and the boomerang uniformity is examined in terms of the original bivariate form. In addition, experiment results for $n=3, 5$ indicates that the proposed condition seems to cover all coefficients $α, β\in \mathbb{F}_{2^n}$ that produce permutations $V_i$ with boomerang uniformity $4$. However, the experiment result shows that the quadratic permutation $V_i$ seems to be affine equivalent to the Gold function. Therefore, unluckily, we may not to obtain new permutations with boomerang uniformity $4$ from the butterfly structure.