Dynamic voltage and frequency scaling (DVFS) and task-to-core allocation are critical for thermal management and balancing energy and performance in embedded systems. Existing approaches either rely on utilization-based heuristics that overlook stall times, or require extensive offline profiling for table generation, preventing runtime adaptation. Building upon hierarchical multi-agent scheduling, we contribute model-based reinforcement learning with accurate environment models that predict thermal dynamics and performance states, enabling synthetic training data generation and converging 20 times faster than model-free methods. We introduce Large Language Model (LLM)-based semantic feature extraction that characterizes OpenMP programs through code-level features without execution, enabling zero-shot deployment for new workloads in under 5 seconds without workload-specific profiling. Two collaborative agents decompose the exponential action space, achieving 358ms latency for subsequent decisions. Experiments on Barcelona OpenMP Tasks Suite (BOTS) and PolybenchC benchmarks across NVIDIA Jetson TX2, Jetson Orin NX, RubikPi, and Intel Core i7 demonstrate 7.09 times better energy efficiency, 4.0 times better makespan, and 358ms decision latency compared to existing power management techniques.