We propose a novel conditional diffusion model for contextual portfolio optimization that learns the cross-sectional distribution of next-day stock returns conditioned on high-dimensional asset-specific factors. Our model leverages a Diffusion Transformer architecture with token-wise conditioning, which enables linking each asset's return to its own factor vector while capturing complex cross-asset dependencies. By drawing generative samples from the learned conditional return distribution, we perform daily mean-variance and mean-CVaR optimization, incorporating transaction costs and realistic constraints. Using data from the Chinese A-share market, we demonstrate that our approach consistently outperforms various standard benchmarks across multiple risk-adjusted performance metrics. Furthermore, we establish a 2-Wasserstein error bound for the conditional diffusion model and quantify how its distributional approximation errors propagate to the downstream portfolio optimization task. Our results demonstrate the potential of generative diffusion models for high-dimensional, risk-sensitive contextual stochastic optimization and financial decision making.