In this work, we present CEDR, a Compiler-integrated, Extensible Domain Specific System on Chip Runtime ecosystem to facilitate research towards addressing the challenges of architecture, system software and application development with distinct plug-and-play integration points in a unified compile time and run time workflow. We demonstrate the utility of CEDR on the Xilinx Zynq MPSoC-ZCU102 for evaluating performance of pre-silicon hardware in the trade space of SoC configuration, scheduling policy and workload complexity based on dynamically arriving workload scenarios composed of real-life signal processing applications scaling to thousands of application instances with FFT and matrix multiply accelerators. We provide insights into the trade-offs present in this design space through a number of distinct case studies. CEDR is portable and has been deployed and validated on Odroid-XU3, X86 and Nvidia Jetson Xavier based SoC platforms. Taken together, CEDR is a capable environment for enabling research in exploring the boundaries of productive application development, resource management heuristic development, and hardware configuration analysis for heterogeneous architectures.