Efficient service composition in real time while providing necessary Quality of Service (QoS) guarantees has been a challenging research problem with ever growing complexity. Several heuristic based approaches with diverse proposals for taming the scale and complexity of web service composition, have been proposed in literature. In this paper, we present a new approach for efficient service composition based on abstraction refinement. Instead of considering individual services during composition, we propose several abstractions to form service groups and the composition is done on these abstract services. Abstraction reduces the search space significantly and thereby can be done reasonably fast. While this can expedite solution construction to a great extent, this also entails a possibility that it may fail to generate any solution satisfying the QoS constraints, though the individual services construct a valid solution. Hence, we propose to refine an abstraction to generate the composite solution with desired QoS values. A QoS satisfying solution, if one exists, can be constructed with multiple iterations of abstraction refinement. While in the worst case, this approach may end up exploring the complete composition graph constructed on individual services, on an average, the solution can be achieved on the abstract graph. The abstraction refinement techniques give a significant speed-up compared to the traditional composition techniques. Experimental results on real benchmarks show the efficiency of our proposed mechanism in terms of time and the number of services considered for composition.