Stroke fragmentation is one of the key steps in pen-based interaction. In this letter, we present a unified HMM-based stroke fragmentation technique that can do segment point location and primitive type determination simultaneously. The geometry features included are used to evaluate local features, and the HMM model is utilized to measure the global drawing context. Experiments prove that the model can efficiently represent smooth curves as well as strokes made up of arbitrary lines and circular arcs.