An accurate, physically-based, and differentiable model of soft robots can unlock downstream applications in optimal control. The Finite Element Method (FEM) is an expressive approach for modeling highly deformable structures such as dynamic, elastomeric soft robots. In this paper, we compare virtual robot models simulated using differentiable FEM with measurements from their physical counterparts. In particular, we examine several soft structures with different morphologies: a clamped soft beam under external force, a pneumatically actuated soft robotic arm, and a soft robotic fish tail. We benchmark and analyze different meshing resolutions and elements (tetrahedra and hexahedra), numerical damping, and the efficacy of differentiability for parameter calibration using a simulator based on the fast Differentiable Projective Dynamics (DiffPD). We also advance FEM modeling in application to soft robotics by proposing a predictive model for pneumatic soft robotic actuation. Through our recipes and case studies, we provide strategies and algorithms for matching real-world physics in simulation, making FEM useful for soft robots