Brenna Argall

Assistive robots offer agency to humans with severe motor impairments. Often, these users control high-DoF robots through low-dimensional interfaces—such as using a 1-D sip/puff interface to operate a 6-DoF robotic arm. This mismatch results in having access to only a subset of control dimensions at a given time, imposing unintended and artificial constraints on robot motion. As a result, interface-limited demonstrations embed suboptimal motions that reflect interface restrictions rather than user intent. To address this, we present a trajectory reconstruction algorithm that reasons about task, environment, and interface constraints to lift demonstrations into the robot’s full control space. We evaluate our approach using real-world demonstrations of ADL-inspired tasks performed via a 2-D joystick and 1-D sip/puff control interface, teleoperating two distinct 7-DoF robotic arms. Analyses of the reconstructed demonstrations and derived control policies show that lifted trajectories are faster and more efficient than their interface-constrained counterparts while respecting user preferences.

Assistive robots offer agency to humans with severe motor impairments. Often, these users control high-DoF robots through low-dimensional interfaces, such as using a 1-D sip-and-puff interface to operate a 6-DoF robotic arm. This mismatch results in having access to only a subset of control dimensions at a given time, imposing unintended and artificial constraints on robot motion. As a result, interface-limited demonstrations embed suboptimal motions that reflect interface restrictions rather than user intent. To address this, we present a trajectory reconstruction algorithm that reasons about task, environment, and interface constraints to lift demonstrations into the robot's full control space. We evaluate our approach using real-world demonstrations of ADL-inspired tasks performed via a 2-D joystick and 1-D sip-and-puff control interface, teleoperating two distinct 7-DoF robotic arms. Analyses of the reconstructed demonstrations and derived control policies show that lifted trajectories are faster and more efficient than their interface-constrained counterparts while respecting user preferences.

This paper details the development, implementation, and experimental evaluation of an interface-aware, task-agnostic assistance system for handling unintended interface operation during shared human-robot teleoperation, specifically applied to a 7-DoF robotic arm. The system addresses a limitation of current shared-control methods by considering the impact of control interfaces on user input precision and the robot agent’s incomplete understanding of the human’s policy. Additionally, the system addresses the issue of data efficiency in a data-scarce domain where gathering extensive data is impractical. The approach is evaluated in empirical case studies involving participants with spinal cord injuries and with data-driven models that are originally derived via direct statistical modeling, and later are derived via transfer learning. The paper overviews the limitations addressed and improvements made throughout this iterative process. The personalized assistance system is found to improve safety and reduce cognitive load across participants with spinal cord injuries in real-world assistive settings with minimal training data requirements.

In robot teleoperation, humans issue control signals through interfaces that require physical actuation.This interfacelevel interaction largely goes unmodeled within the field, yet the interpretation of an interface-level command can differ from what was intended by the user, as a result of diminished human ability or inadequate mappings from raw interface signals to robot control signals.Interface-aware systems aim to address this limitation in robot teleoperation by explicitly considering the impact of a control interface on user input quality when interpreting interface signals for robot control.This work presents an interface-aware formulation for the direct inference of intended interface-level commands given known interaction characteristics of a control interface using data-driven modeling, allowing for teleoperation assistance without knowledge of the human's policy.In our specific implementation, we tailor the formulation to model a user's operation of a sip/puff interface using a network of Gated Recurrent Units, chosen for their ability to model temporal patterns and suitability for data-scarce domains.The resulting model is agnostic to the robot being controlled, which allows for its use in task-and environment-agnostic robot teleoperation assistance.We deploy this model in two variations of assisted teleoperation frameworks using a 1-D sip/puff interface to control a 7-DoF robotic arm, and conduct a human subjects study with spinal cord injured participants to evaluate the efficacy of our method.Our proposed task-and environment-agnostic formulation is effective in reducing collisions during teleoperation, and is preferred by users over teleoperation baselines for ease and intuitiveness of robot operation.

The interfaces used to operate assistive robots typically employ fixed, predefined maps to associate interfacelevel commands to robot control commands. User-defined control maps instead consider an individual's preferences and capabilities, moving away from a one-size-fits-all mapping paradigm. This work presents novel methods for (1) eliciting user-defined control maps, (2) identifying and addressing issues in control signal data that arise from such a user-centered design, and (3) methodologies to filter erroneous signals and construct synthetic data in an effort to address these issues. We experimentally evaluate our proposed methods by conducting a user study that elicits user-defined, interface-level commands for controlling a powered wheelchair and a robotic arm through four control interfaces. Our results highlight the differing suitability of user-defined, data-driven control maps for the various interface-platform pairings, and provide an analysis of the errors generated during dataset definition and the impact of our postprocessing methods.

In this work, we present a case study evaluation that compares two methods of fitting the Body-Machine Interface (BoMI) to an individual with cervical spinal cord injury for the purpose of operating a robotic arm in 6-D. A BoMI is a control interface created by recording body movements and mapping them to the controls of a device or machine, and has shown promise for individuals with motor impairments whose access to standard interfaces is otherwise limited. Results from this case study show a one-size-fits-all placement strategy is not a sufficient fitting method for an individual with severe upper extremity limitations. However, when a flexible fitting protocol informed by a clinical evaluation is applied, we see improvement in two key categories: (1) success in map fitting and (2) robot task results. Informed by the case study analyses, we develop a novel method to customize and fit the BoMI to users in a way that is analogous to how commonly used assistive technologies are fit clinically: the BoMI Customization Evaluation (BCE). This new method of customization is determined from a physical evaluation conducted by a clinician in conjunction with participant feedback and BoMI engineers. Deployment of this novel method within a full evaluation study is underway. The current work focuses on the evolution to this protocol.

Eye gaze tracking is increasingly popular due to improved technology and availability. In the domain of assistive device control, however, eye gaze tracking is often used in discrete ways (e.g., activating buttons on a screen), and does not harness the full potential of the gaze signal. In this article, we present a method for collecting both reactionary and controlled eye gaze signals, via screen-based tasks designed to isolate various types of eye movements. The resulting data allows us to build an individualized characterization for eye gaze interface use. Results from a study conducted with participants with motor impairments are presented, offering insights into maximizing the potential of eye gaze for assistive device control. Importantly, we demonstrate the potential for incorporating direct continuous eye gaze inputs into gaze-based interface designs; generally seen as intractable due to the 'Midas touch' problem of differentiating between gaze movements for perception versus for interface operation. Our key insight is to make use of an individualized measure of smooth pursuit characteristics to differentiate between gaze for control and gaze for environment scanning. We also present results relating to gaze-based metrics for mental workload and show the potential for the concurrent use of eye gaze for control input as well as assessing a user's mental workload both offline and in real-time. These findings might inform the development of continuous control paradigms using eye gaze, as well as the use of eye tracking as the sole input modality to systems that share control between human-generated and autonomy-generated inputs.

In this work, we present an evolution of system designs and studies that aim to facilitate the operation of high-DoF assistive robotic arms by persons with upper limb paralysis. We highlight the experimental pipeline and note developments in our efforts to design a suitable control map that can convert low variance residual body motions from neuromotor-impaired populations into 6-D velocity control signals for use in teleoperating a 7-DOF robotic arm. Notably, we provide results from variance analyses on raw IMU control signals from both neuromotor-impaired and unimpaired populations, and an analysis of the intrinsic dimensionality of map-building datasets gathered with and without movement guidance. We then present a preliminary 13-session study that vets the control map developed in light of these findings.

When individuals are paralyzed from injury or damage to the brain, upper body movement and function can be compromised. While the use of body motions to interface with machines has shown to be an effective noninvasive strategy to provide movement assistance and to promote physical rehabilitation, learning to use such interfaces to control complex machines is not well understood. In a five session study, we demonstrate that a subset of an uninjured population is able to learn and improve their ability to use a high-dimensional Body-Machine Interface (BoMI), to control a robotic arm. We use a sensor net of four inertial measurement units, placed bilaterally on the upper body, and a BoMI with the capacity to directly control a robot in six dimensions. We consider whether the way in which the robot control space is mapped from human inputs has any impact on learning. Our results suggest that the space of robot control does play a role in the evolution of human learning: specifically, though robot control in joint space appears to be more intuitive initially, control in task space is found to have a greater capacity for longer-term improvement and learning. Our results further suggest that there is an inverse relationship between control dimension couplings and task performance.

Individuals who suffer from severe paralysis often lose the capacity to perform fundamental body movements and everyday activities. Empowering these individuals with the ability to operate robotic arms, in high degrees-of-freedom (DoFs), can help to maximize both functional utility and independence. However, robot teleoperation in high DoFs currently lacks accessibility due to the challenge in capturing high-dimensional control signals from the human, especially in the face of motor impairments. Body-machine interfacing is a viable option that offers the necessary high-dimensional motion capture, and it moreover is noninvasive, affordable, and promotes movement and motor recovery. Nevertheless, to what extent body-machine interfacing is able to scale to high-DoF robot control, and whether it is feasible for humans to learn, remains an open question. In this exploratory multi-session study, we demonstrate the feasibility of human learning to operate a body-machine interface to control a complex, assistive robotic arm. We use a sensor net of four inertial measurement unit sensors, bilaterally placed on the scapulae and humeri. Ten uninjured participants are familiarized, trained, and evaluated in reaching and Activities of Daily Living tasks, using the body- machine interface. Our results suggest the manner of control space mapping (joint-space control versus task-space control), from interface to robot, plays a critical role in the evolution of human learning. Though joint-space control shows to be more intuitive initially, task-space control is found to have a greater capacity for longer-term improvement and learning.

Eye gaze tracking is increasingly popular due to improved technology and availability. However, in assistive device control, eye gaze tracking is often limited to discrete control inputs. In this paper, we present a method for collecting both reactionary and control eye gaze signals to build an individualized characterization for eye gaze interface use. Results from a study conducted with motor-impaired participants are presented, offering insights into maximizing the potential of eye gaze for assistive device control. These findings can inform the development of continuous control paradigms using eye gaze.

Abstract Individuals who suffer from severe paralysis often lose the capacity to perform fundamental body movements and everyday activities. Empowering these individuals with the ability to operate robotic arms, in high-dimensions, helps to maximize both functional utility and human agency. However, high-dimensional robot teleoperation currently lacks accessibility due to the challenge in capturing high-dimensional control signals from the human, especially in the face of motor impairments. Body-machine interfacing is a viable option that offers the necessary high-dimensional motion capture, and it moreover is noninvasive, affordable, and promotes movement and motor recovery. Nevertheless, to what extent body-machine interfacing is able to scale to high-dimensional robot control, and whether it is feasible for humans to learn, remains an open question. In this exploratory multi-session study, we demonstrate the feasibility of human learning to operate a body-machine interface to control a complex, assistive robotic arm in reaching and Activities of Daily Living tasks. Our results suggest the manner of control space mapping, from interface to robot, to play a critical role in the evolution of human learning.