This paper develops a novel methodology for designing analog beamforming codebooks for full-duplex millimeter wave (mmWave) transceivers, the first such codebooks to the best of our knowledge. Our design reduces the self-interference coupled by transmit-receive beam pairs and simultaneously delivers high beamforming gain over desired coverage regions, allowing mmWave full-duplex systems to support beam alignment while minimizing self-interference. To do so, our methodology allows some variability in beamforming gain to strategically shape beams that reject self-interference while still having substantial gain. We present an algorithm for approximately solving our codebook design problem while accounting for the non-convexity posed by digitally-controlled phase shifters and attenuators. Numerical results suggest that our design can outperform or nearly match existing codebooks in sum spectral efficiency across a wide range of self-interference power levels. Results show that our design offers an extra 20-50 dB of robustness to self-interference, depending on hardware constraints.