Ray antenna array (RAA) is a novel multi-antenna architecture comprising massive low-cost antenna elements and a few radio-frequency (RF) chains. The antenna elements are arranged in a novel ray-like structure, where each ray corresponds to a simple uniform linear array (sULA) with deliberately designed orientation and all its antenna elements are directly connected. By further designing a ray selection network (RSN), appropriate sULAs are selected to connect to the RF chains for further baseband processing. RAA has three appealing advantages: (i) dramatically reduced hardware cost since no phase shifters are needed; (ii) enhanced beamforming gain as antenna elements with higher directivity can be used; (iii) uniform angular resolution across all signal directions. Such benefits make RAA especially appealing for integrated sensing and communication (ISAC), particularly for low-altitude unmanned aerial vehicle (UAV) swarm ISAC, where high-mobility aerial targets may easily move away from the boresight of conventional antenna arrays, causing severe communication and sensing performance degradation. Therefore, this paper studies RAA-based ISAC for low-altitude UAV swarm systems. First, we establish an input-output mathematical model for RAA-based UAV ISAC and rigorously show that RAA achieves uniform angular resolution for all directions. Besides, we design the RAA orientation and RSN. Furthermore, RAA-based ISAC with orthogonal frequency division multiplexing (OFDM) for UAV swarm is studied, and efficient algorithm is proposed for sensing target parameter estimation. Extensive simulation results demonstrate the significant performance improvement by RAA system over the conventional antenna arrays, in terms of sensing angular resolution and communication spectral efficiency, highlighting the great potential of the novel RAA system to meet the growing demands of low-altitude UAV ISAC.