Wireless digital twins (WDTs) enable site-specific learning, management, and evaluation in wireless networks. However, constructing and maintaining a high-fidelity WDT over large-scale complex environments can be prohibitively expensive, especially in terms of data acquisition, geometric reconstruction, storage, and ray tracing. To address this issue, a task-oriented nonuniform refinement framework for WDTs is proposed, where limited resources are selectively allocated to the WDT components that matter most to wireless fidelity. Specifically, a unified refinement framework is first developed, which maximizes task-level fidelity under resource constraints through fine-grained component-wise fidelity allocation. This framework is then instantiated for building-level geometry refinement in urban WDTs. It is found that different buildings exhibit highly heterogeneous impacts on wireless fidelity. Motivated by this observation, an ellipsoid-guided selective refinement algorithm (EGSR) is proposed. By jointly considering the relevance of each building to both line-of-sight (LoS) and non-line-of-sight (NLoS) propagation paths, its refinement priority can be estimated using only a low-fidelity WDT. Simulations across multiple urban scenarios show that EGSR can substantially improve radio-map fidelity and preserve beamforming effectiveness by refining only a small subset of buildings. These results demonstrate the potential of task-oriented fidelity allocation as a scalable principle for constructing efficient and performance-aware WDTs, thereby facilitating reliable site-specific learning and optimization.