Let $G$ be a graph with adjacency matrix $A(G)$ and degree diagonal matrix $D (G)$. In 2017, Nikiforov [Appl. Anal. Discrete Math., 11 (2017) 81--107] defined the matrix $A_α(G) = αD(G) + (1-α)A(G)$ for any real $α\in[0,1]$. The largest eigenvalue of $A(G)$ is called the spectral radius of $G$, while the largest eigenvalue of $A_α(G)$ is called the $A_α$ spectral radius of $G$. Let $\mathcal{G}_{n,i}$ be the set of graphs of order $n$ with independence number $i$. Recently, for all graphs in $\mathcal{G}_{n,i}$ having the minimum or the maximum $A$, $Q$ and $A_α$ spectral radius where $i\in\{1,2,\lfloor\frac{n}{2}\rfloor\,\lceil\frac{n}{2}\rceil+1,n-3,n-2,n-1\}$, there are some results have been given by Xu, Li and Sun et al., respectively. In 2021, Luo and Guo [Discrete Math., 345 (2022) 112778] determined all graphs in $\mathcal{G}_{n,n-4}$ having the minimum spectral radius. In this paper, we characterize the graphs in $\mathcal{G}_{n,n-4}$ having the minimum and the maximum $A_α$ spectral radius for $α\in[\frac{1}{2},1)$, respectively.