Secure Element (SE) in SoC sees an increasing adoption in industry. Many applications in IoT devices are bound to the SE because it provides strong cryptographic functions and physical protection. Though SE-in-SoC provides strong proven isolation for software programs, it also brings more design complexity and higher cost to PCB board building. More, SE-in-SoC may still have security concerns, such as malware installation and user impersonation. In this work, we employ TEE, a hardware-backed security technique, for protecting SE-in-SoC and RISCV. In particular, we construct various enclaves for isolating applications and manipulating the SE, with the inherently-secure primitives provided by RISC-V. Using hardware and software co-design, the solution ensures trusted execution and secure communication among applications. The security of SE is further protected by enforcing the SE to be controlled by a trusted enclave and making the RISC-V core resilient to side-channel attacks.