Driven by the visions of Internet of Things and 5G communications, recent years have seen a paradigm shift in personal mobile devices, from smartphones towards wearable devices. Wearable devices come in many different forms targeting different application scenarios. Among these, the fitness wearable devices (FWDs) are proven to be one of the forms that intrigue the market and occupy an increasing trend in terms of the market share. Nevertheless, although the fitness wearable devices nowadays are functionally self-contained based on the advanced sensor, computation, and communicative technologies, there is still a large gap to truly satisfy the target customer group, i.e., accessible to and usable by a larger quantity of users. This fuels the research area on applying the universal design principles to fitness wearable devices. In this survey, we first present the background of FWDs and show the acceptance and adaption challenges of the corresponding user groups. We then review the universal design principle and how it and its relative approaches could be used in FWDs. Further, we collect the available FWDs that bear the universal design principles in their development circles. Last, we open up the discussion based on the surveyed literature and provide the insight of potential future work.