Developing kinesthetic haptic devices with advanced haptic rendering capabilities is challenging due to the limitations on driving mechanisms. In this study, we introduce a novel soft electrohydraulic actuator and develop a kinesthetic haptic device utilizing it as the driving unit. We established a mathematical model and conducted testing experiments to demonstrate the device's ability to stably output controllable feedback force. Our experiments also demonstrates that this device exhibits fast response characteristics. By utilizing the easily controllable nature of the soft electrohydraulic actuator, we were able to achieve high-resolution controllable feedback force output. Furthermore, by modulating the waveform of the driving high voltage, the device acquired the capability to render variable frequency haptic vibration without adding any extra vibration actuator. Using this kinesthetic haptic device, we built a teleoperated robotic system, showcasing the device's potential application as a haptic force feedback system in the field of robotics.