A partial automorphism of a finite graph is an isomorphism between its vertex induced subgraphs. The set of all partial automorphisms of a given finite graph forms an inverse monoid under composition (of partial maps). We describe the algebraic structure of such inverse monoids by the means of the standard tools of inverse semigroup theory, namely Green's relations and some properties of the natural partial order, and give a characterization of inverse monoids which arise as inverse monoids of partial graph automorphisms. We extend our results to digraphs and edge-colored digraphs as well.