The expected signature of a family of paths need not be a signature of a path itself. Motivated by this, we consider the notion of a Lie group barycenter introduced by Buser and Karcher to propose a barycenter on path signatures. We show that every element of the free nilpotent Lie group is a barycenter of a group sample, where all but one sample element can be fixed arbitrarily. In the case of piecewise linear paths, we study the problem of recovering an underlying path corresponding to the barycenter of signatures. We determine the minimal number of segments required to learn from signature matrices, providing explicit transformations to the associated congruence normal forms.