The nonlinearity and hysteresis of soft robot motions present challenges for control. To solve these issues, the Jacobian controller has been applied to approximate the nonlinear behaviors in a linear format. Accurate controllers like neural networks can handle delayed and nonlinear motions, but they require large datasets and exhibit low adaptability. Based on a novel analysis on these controllers, we propose an adaptive extended Jacobian controller, AdapJ, for soft manipulators. This controller retains the concise format of the Jacobian controller but introduces independent parameters. Similar to neural networks, its initialization and updating mechanism leverages the inverse model without building the corresponding forward model. In the experiments, we first compare the performance of the Jacobian controller, model predictive controller, neural network controller, iterative feedback controller, and AdapJ in simulation. We further analyze how AdapJ parameters adapt in response to the physical property change. Then, real-world experiments have validated that AdapJ outperforms the neural network controller, model predictive controller, and iterative feedback controller with fewer training samples and adapts robustly to varying conditions, including different control frequencies, material softness, and external disturbances. Future work may include online adjustment of the controller format and adaptability validation in more scenarios.