In many classification tasks, the set of target classes can be organized into a hierarchy. This structure induces a semantic distance between classes, and can be summarised under the form of a cost matrix, which defines a finite metric on the class set. In this paper, we propose to model the hierarchical class structure by integrating this metric in the supervision of a prototypical network. Our method relies on jointly learning a feature-extracting network and a set of class prototypes whose relative arrangement in the embedding space follows an hierarchical metric. We show that this approach allows for a consistent improvement of the error rate weighted by the cost matrix when compared to traditional methods and other prototype-based strategies. Furthermore, when the induced metric contains insight on the data structure, our method improves the overall precision as well. Experiments on four different public datasets - from agricultural time series classification to depth image semantic segmentation - validate our approach.