Numerical modeling of polycrystal plasticity is computationally intensive. We employ Graph Neural Networks (GNN) to predict stresses on complex geometries for polycrystal plasticity from Finite Element Method (FEM) simulations. We present a novel message-passing GNN that encodes nodal strain and edge distances between FEM mesh cells, and aggregates to obtain embeddings and combines the decoded embeddings with the nodal strains to predict stress tensors on graph nodes. The GNN is trained on subgraphs generated from FEM mesh graphs, in which the mesh cells are converted to nodes and edges are created between adjacent cells. We apply the trained GNN to periodic polycrystals with complex geometries and learn the strain-stress maps based on crystal plasticity theory. The GNN is accurately trained on FEM graphs, in which the $R^2$ for both training and testing sets are larger than 0.99. The proposed GNN approach speeds up more than 150 times compared with FEM on stress predictions. We also apply the trained GNN to unseen simulations for validations and the GNN generalizes well with an overall $R^2$ of 0.992. The GNN accurately predicts the von Mises stress on polycrystals. The proposed model does not overfit and generalizes well beyond the training data, as the error distributions demonstrate. This work outlooks surrogating crystal plasticity simulations using graph data.