A remote collaboration setup allows an expert to instruct a remotely located novice user to help them with a task. Advanced solutions exist in the field of remote guidance and telemanipulation, however, we lack a device that combines these two aspects to grant the expert physical access to the work site. We explore a setup that involves three parties: a local worker, a remote helper and a handheld robot carried by the local worker. Using remote maintenance as an example task, the remote user assists the local user through diagnosis, guidance and physical interaction. We introduce a remote collaboration system with a handheld robot that provides task knowledge and enhanced motion and accuracy capabilities. We assess the proposed system in two different configurations: with and without the robot's assistance features enabled. We show that the handheld robot can mediate the helper's instructions and remote object interactions while the robot's semi-autonomous features improve task performance by 37%, reduce the workload for the remote user and decrease required verbal communication bandwidth. Our results demonstrate the effectiveness of task delegation with the remote user choosing an action and the robot autonomously completing it at the local level. This study is a first attempt to evaluate how this new type of collaborative robot works in a remote assistance scenario. We believe that the proposed setup and task-delegation concepts help to leverage current constrains in telemanipulation systems.