Most research in the area of machine learning-based user beam selection considers a structure where the model proposes appropriate user beams. However, this design requires a specific model for each user-device beam codebook, where a model learned for a device with a particular codebook can not be reused for another device with a different codebook. Moreover, this design requires training and test samples for each antenna placement configuration/codebook. This paper proposes a device-agnostic beam selection framework that leverages context information to propose appropriate user beams using a generic model and a post processing unit. The generic neural network predicts the potential angles of arrival, and the post processing unit maps these directions to beams based on the specific device's codebook. The proposed beam selection framework works well for user devices with antenna configuration/codebook unseen in the training dataset. Also, the proposed generic network has the option to be trained with a dataset mixed of samples with different antenna configurations/codebooks, which significantly eases the burden of effective model training.