The vision-based relative localization can provide effective feedback for the cooperation of aerial swarm and has been widely investigated in previous works. However, the limited field of view (FOV) inherently restricts its performance. To cope with this issue, this letter proposes a novel distributed active vision-based relative localization framework and apply it to formation control in aerial swarms. Inspired by bird flocks in nature, we devise graph-based attention planning (GAP) to improve the observation quality of the active vision in the swarm. Then active detection results are fused with onboard measurements from Ultra-WideBand (UWB) and visual-inertial odometry (VIO) to obtain real-time relative positions, which further improve the formation control performance of the swarm. Simulations and experiments demonstrate that the proposed active vision system outperforms the fixed vision system in terms of estimation and formation accuracy.