Multi-access edge computing (MEC) is a promising technology to enhance the quality of service, particularly for low-latency services, by enabling computing offloading to edge servers (ESs) in close proximity. To avoid network congestion, collaborative edge computing has become an emerging paradigm to enable different ESs to collaboratively share their data and computation resources. However, most papers in collaborative edge computing only allow one-hop offloading, which may limit computing resource sharing due to either poor channel conditions or computing workload at ESs one-hop away. By allowing ESs multi-hop away to also share the computing workload, a multi-hop MEC enables more ESs to share their computing resources. Inspired by this observation, in this paper, we propose to leverage omnipresent vehicles in a city to form a data transportation network for task delivery in a multi-hop fashion. Here, we propose a general multi-hop task offloading framework for vehicle-assisted MEC where tasks from users can be offloaded to powerful ESs via potentially multi-hop transmissions. Under the proposed framework, we develop a reinforcement learning based task offloading approach to address the curse of dimensionality problem due to vehicular mobility and channel variability, with the goal to maximize the aggregated service throughput under constraints on end-to-end latency, spectrum, and computing resources. Numerical results demonstrate that the proposed algorithm achieves excellent performance with low complexity and outperforms existing benchmark schemes.