Ensuring strict safety guarantees is the paramount challenge for emerging 5G/6G wireless systems, particularly as they increasingly govern mission-critical applications ranging from autonomous UAV swarms to industrial automation. While deep reinforcement learning (DRL) offers a promising solution for complex resource allocation, standard algorithms frequently violate essential constraints, such as QoS mandates and power limits, posing unacceptable risks of system failure and regulatory non-compliance. We propose Safe-Deep Q-Learning, a novel algorithm that simultaneously addresses all three challenges: it handles mixed-integer nonconvex problems by approximating the Q-function, adapts to stochastic dynamics, and enforces dual-timescale constraints using integrated Lagrangian methods. Our framework features adaptive penalty scaling and constraint violation tracking, specifically tailored for wireless environments, and is designed to operate in both distributed and centralized architectural modes. We prove convergence to optimal constraint-satisfying policies under mild conditions and demonstrate robustness through dual variable stabilization. Validation on unmanned aerial vehicle (UAV) swarm control network and post-disaster emergency communications applications shows that Safe-Deep Q-Learning achieves stringent adherence to safety bounds with near-zero violation rates, significantly outperforming existing constrained RL baselines, establishing its effectiveness for safety-critical wireless deployments.