There is a growing interest in codebook-based beam-steering for millimeter-wave (mmWave) systems due to its potential for low complexity and rapid beam search. A key focus of recent research has been the design of codebooks that strike a trade-off between achievable gain and codebook size, which directly impacts beam search time. Statistical approaches have shown promise by leveraging the likelihood that certain beam directions (equivalently, sets of phase-shifter configurations) are more probable than others. Such approaches are shown to be valid for static, non-rotating transmission stations such as base stations. However, for the case of user terminals that are constantly changing orientation, the possible phase-shifter configurations become equally probable, rendering statistical methods less relevant. On the other hand, user terminals come with a large number of possible steering vector configurations, which can span up to six orders of magnitude. Therefore, efficient solutions to reduce the codebook size (set of possible steering vectors) without compromising array gain are needed. We address this challenge by proposing a novel and practical codebook refinement technique, aiming to reduce the codebook size while maintaining array gain within $γ$ dB of the maximum achievable gain at any random orientation of the user terminal. We project that a steering vector at a given angle could effectively cover adjacent angles with a small gain loss compared to the maximum achievable gain. We demonstrate experimentally that it is possible to reduce the codebook size from $1024^{16}$ to just a few configurations (e.g., less than ten), covering all angles while maintaining the gain within $γ=3$ dB of the maximum achievable gain.