Natural control of limb motion is continuous and progressively adaptive to individual intent. While intuitive interfaces have the potential to rely on the neuromuscular input by the user for continuous adaptation, continuous volitional control of assistive devices that can generalize across various tasks has not been addressed. In this study, we propose a method to use spatiotemporal ultrasound features of the rectus femoris and vastus intermedius muscles of able-bodied individuals for task-invariant learning of continuous knee kinematics during steady-state and transient ambulation. The task-invariant learning paradigm was statistically evaluated against a task-specific paradigm for the steady-state (1) level-walk, (2) incline, (3) decline, (4) stair ascent, and (5) stair descent ambulation tasks. The transitions between steady-state stair ambulation and level-ground walking were also investigated. It was observed that the continuous knee kinematics can be learned using a task-invariant learning paradigm with statistically comparable accuracy to a task-specific paradigm. Statistical analysis further revealed that incorporating the temporal ultrasound features significantly improves the accuracy of continuous estimations (p < 0.05). The average root mean square errors (RMSEs) of knee angle and angular velocity estimation were 7.06° and 53.1°/sec, respectively, for the task-invariant learning compared to 6.00° and 51.8°/sec for the task-specific models. High accuracy of continuous task-invariant paradigms overcome the barrier of task-specific control schemes and motivate the implementation of direct volitional control of lower-limb assistive devices using ultrasound sensing, which may eventually enhance the intuitiveness and functionality of these devices towards a "free form" control approach.