In increasing number of electronic devices implement wideband radio technologies for localization and sensing purposes, like ultra-wideband (UWB). Such radio technologies benefit from a large number of antennas, but space for antennas is often limited, especially in devices for mobile and IoT applications. A common challenge is therefore to optimize the placement and orientations of a small number of antenna elements inside a device, leading to the best localization performance. We propose a method for systematically approaching the optimization of such sparse arrays by means of Cramér-Rao lower bounds (CRLBs) and vector spherical wave functions (VSWFs). The VSWFs form the basis of a wideband signal model considering frequency, direction and polarization-dependent characteristics of the antenna array under test (AUT), together with mutual coupling and distortions from surrounding obstacles. We derive the CRLBs for localization parameters like delay and angle-of-arrival for this model under additive white Gaussian noise channel conditions, and formulate optimization problems for determining optimal antenna positions and orientations via minimization of the CRLBs. The proposed optimization procedure is demonstrated by means of an exemplary arrangement of three Crossed Exponentially Tapered Slot (XETS) antennas.