With the advent of Unmanned Aerial Vehicles (UAV) and Micro Aerial Vehicles (MAV) in commercial sectors, their application for transporting and manipulating payloads has attracted many research work. A swarm of agents, cooperatively working to transport and manipulate a payload can overcome the physical limitations of a single agent, adding redundancy and tolerance against failures. In this paper, the dynamics of a swarm connected to a payload via flexible cables are modeled, and a decentralized control is designed using Artificial Potential Field (APF). The swarm is able to transport the payload through an unknown environment to a goal position while avoiding obstacles from the local information received from the onboard sensors. The key contributions are (a) the cables are modelled more accurately using lumped mass model instead of geometric constraints, (b) a decentralized swarm control is designed using potential field approach to ensure hover stability of system without payload state information, (c) the manipulation of payload elevation and azimuth angles are controlled by APF, and (d) the trajectory of the payload for transportation is governed by potential fields generated by goal point and obstacles. The efficacy of the method proposed in this work are evaluated through numerical simulations under the influence of external disturbances and failure of agents.