Next generation wireless networks will exploit the large amount of spectrum available at millimeter wave (mmWave) frequencies. Design of mmWave systems, however, is challenging due to strict power, cost and hardware constraints at higher bandwidths. To achieve a good SNR for communication, mmWave systems use large antenna arrays. Beamforming with highly directional beams is one way to use the antennas. As the channel changes over time, the beams that maximize the SNR have to be estimated quickly to reduce the training overhead. Prior work has exploited the observation that mmWave channels are sparse to perform compressed sensing (CS) based beam alignment with few channel measurements. Most of the existing CS-based algorithms, however, assume perfect synchronization and fail in the presence of carrier frequency offset (CFO). This paper presents Swift-Link, a fast beam alignment algorithm that is robust against the offset. Swift-Link includes a novel randomized beam training sequence that minimizes the beam alignment errors due to CFO and a low-complexity algorithm that corrects these errors. Even with strict hardware constraints, our algorithm uses fewer channel measurements than comparable CS algorithms and has analytical guarantees. Swift-Link requires a small output dynamic range at the analog-to-digital converter compared to beam-scanning techniques.