Lattice polytopes which possess the integer decomposition property (IDP for short) turn up in many fields of mathematics. It is known that if the Cayley sum of lattice polytopes possesses IDP, then so does their Minkowski sum. In this paper, the Cayley sum of the order polytope of a finite poset and the stable set polytope of a finite simple graph is studied. We show that the Cayley sum of an order polytope and the stable set polytope of a perfect graph possesses a regular unimodular triangulation and IDP, and hence so does their Minkowski sum. Moreover, it turns out that, for an order polytope and the stable set polytope of a graph, the following conditions are equivalent: (i) the Cayley sum is Gorenstein; (ii) the Minkowski sum is Gorenstein; (iii) the graph is perfect.