Gaze-Enabled Grasping Assistance For Teleoperation of Robotic Manipulators

Abstract

Shared autonomy in robot teleoperation can ease task completion and lower cognitive load for operators by combining human intent with the autonomous capabilities of robots. As many manipulation control tasks involve the grasping of objects as the first step, augmenting assistance at this stage has the potential to improve user experience and task performance. Existing grasping assistance systems rely on classic grasp planners that limit their capabilities, while some utilize expensive hardware to provide the assistance. Virtual reality systems have been used for input and feedback in teleoperation and have also been applied in grasping assistance systems. Some virtual reality systems feature head-mounted displays that have eye-tracking capabilities, and research has been conducted to leverage the eye-tracking information for teleoperation applications.

This work introduces a novel grasping assistance framework that leverages user intent signalled through eye gaze. Specifically, the system retrieves eye gaze direction from a virtual reality headset during teleoperation. This gaze information is then used to automatically identify the operator's desired object for grasping, suggest suitable grasp options, and allow the operator to select a preferred grasp using their gaze. Once selected, the grasp is automatically executed via a predefined grasping sequence, eliminating the need for one-to-one motion mapping.

The grasping assistance system is implemented using ROS2 to control a Kinova Gen 3 robotic manipulator, with a Meta Quest Pro virtual reality headset providing the user interface. Additionally, the teleoperation system offers visual feedback from cameras in the manipulator's workspace, displayed through the head-mounted display, and incorporates a collision avoidance system to prevent unintended impacts.

A user study was performed with 30 participants using the developed system to compare the usability, workload, and performance of the grasping-assisted teleoperation with pure teleoperation (motion mapping). The study asks the operator to perform a pick-and-place task featuring four different objects in a specified order within the allotted time. Each participant performed the task with both pure teleoperation and grasping assistance modes in random order, and then completed questionnaires attempting to measure the usability of each system and measure their experienced workload. Results show that grasping assistance significantly reduces users' workload, but also leads to lower performance metrics with respect to pure teleoperation. The performance loss could be attributed to the implemented grasp planner. Therefore, a gaze-enabled grasping assistance framework such as the one presented in this thesis has the potential to reduce the workload experienced by users and improve performance metrics over standard motion mapping-based direct teleoperation frameworks.