This paper describes a method of estimating the intention of a user's motion in a robot tele-operation scenario. One of the issues in tele-operation is latency, which occurs due to various reasons such as a slow robot motion and a narrow communication channel. An effective way of reducing the latency is to estimate the human intention of motions and to move the robot proactively. To enable a reliable early intention estimation, we use both hand motion and object affordances in a dual first-person vision (robot and user) with an HMD. Experimental results in an object pickup scenario show the effectiveness of the method.