Vikram Kapila

As robots and humans start to share common spaces and perform collaborative tasks, it has become critical to facilitate information exchange between them for communicating and interpreting each other’s intentions. By overlaying virtual objects on a view of the physical world, mixed reality (MR) technology offers a compelling approach for designing innovative models of human–robot interaction (HRI). For robot manipulators, mobile MR frameworks that allow a user to communicate a goal position for the robot’s end effector have been widely studied. However, HRI applications that may require other relevant information for the manipulator to complete more complex tasks remain unexplored. Thus, we propose an MR-enhanced HRI framework, deployed on a touchscreen tablet, that utilizes a virtual arrow object to communicate force intent (i.e., location, direction, and magnitude) to the manipulator and provide visual force feedback to the user. To evaluate the system performance and user experience, we conducted a user study with 25 participants who used a manipulator robot to complete four insertion subtasks, reporting a task success score of 96%, a usability overall mean score of 4.35 out of 5, and a low task load index of 21.49 out of 100. The results show that the MR-HRI framework is intuitive to operate, allowing users to successfully perform assigned tasks by effectively communicating their intentions through the virtual arrow.

Abstract Background Restriction of movement at a joint due to disease or dysfunction can alter the range of motion (ROM) at other joints due to joint interactions. In this paper, we quantify the extent to which joint restrictions impact upper limb joint movements by conducting a disability simulation study that used wearable inertial sensors for three-dimensional (3D) motion capture. Methods We employed the Wearable Inertial Sensors for Exergames (WISE) system for assessing the ROM at the shoulder (flexion–extension, abduction–adduction, and internal–external rotation), elbow (flexion–extension), and forearm (pronation-supination). We recruited 20 healthy individuals to first perform instructed shoulder, elbow, and forearm movements without any external restrictions, and then perform the same movements with restriction braces placed to limit movement at the shoulder, elbow, and forearm, separately, to simulate disability. To quantify the extent to which a restriction at a non-instructed joint affected movement at an instructed joint, we computed average percentage reduction in ROM in the restricted versus unrestricted conditions. Moreover, we performed analysis of variance and post hoc Tukey tests ( q statistic) to determine the statistical significance ( p < 0.05 denoted using * ) of the differences in ROM of an instructed joint in the unrestricted versus restricted conditions. Results Restricting movement at the shoulder led to a large reduction in the average ROM for elbow flexion–extension (21.93%, q = 9.34 * ) and restricting elbow movement significantly reduced the average ROM for shoulder flexion–extension (17.77%, q = 8.05 * ), shoulder abduction–adduction (19.80%, q = 7.60 * ), and forearm pronation-supination (14.04%, q = 4.96 * ). Finally, restricting the forearm significantly reduced the average ROM for shoulder internal–external rotation (16.71%, q = 3.81 * ) and elbow flexion–extension (10.01%, q = 4.27 * ). Conclusions Joint interactions across non-instructed joints can reduce the ROM of instructed movements. Assessment of ROM in the real-world using 3D motion capture, for example using the WISE system, can aid in understanding movement limitations, informing interventions, and monitoring progress with rehabilitation.

Many neurological conditions, e.g., a stroke, can cause patients to experience upper limb (UL) motor impairments that hinder their daily activities. For such patients, while rehabilitation therapy is key for regaining autonomy and restoring mobility, its long-term nature entails ongoing time commitment and it is often not sufficiently engaging. Virtual reality (VR) can transform rehabilitation therapy into engaging game-like tasks that can be tailored to patient-specific activities, set goals, and provide rehabilitation assessment. Yet, most VR systems lack built-in methods to track progress over time and alter rehabilitation programs accordingly. We propose using arm kinematic modeling and capability maps to allow a VR system to understand a user's physical capability and limitation. Next, we suggest two use cases for the VR system to utilize the user's capability map for tailoring rehabilitation programs. Finally, for one use case, it is shown that the VR system can emphasize and assess the use of specific UL joints.Clinical relevance-This paper's VR-based system can tailor a rehabilitation tool to a user's capability and limit.

Time-series generative adversarial networks (TimeGAN) were recently developed to produce synthetic time-series data for varied applications. Most prior uses of TimeGAN in biomechanics and rehabilitation research did not consider data from inertial measurement unit (IMU) sensors for upper limb range of motion (ROM), especially in the context of disability simulation studies. In this paper, we used TimeGAN to produce synthetic three-dimensional (3D) ROM data for elbow flexion-extension movement performed under four experimental conditions within a disability simulation framework. In each experimental condition, we collected 3D ROM data of human subjects using the wearable inertial sensors for exergame (WISE) system and then produced synthetic ROM data using TimeGAN. Our goal was to produce accurate synthetic data to circumvent difficulties related to capturing data from a large number of human subjects. To assess the performance of TimeGAN in producing synthetic data, we used principal component analysis (PCA), t-distributed stochastic neighbor embedding (t-SNE), and discriminative score. The results show that TimeGAN produced synthetic ROM data of upper limb movements quite well. Specifically, the average discriminative score in the unrestricted, restricted shoulder, restricted elbow, and restricted forearm conditions was found to be 0.12, 0.08, 0.18, and 0.11, respectively.Clinical Relevance-This work illustrates the use of TimeGAN to produce synthetic 3D ROM data. A large 3D ROM dataset can help in training deep learning models to classify impairments in various upper limb joints, guide therapy interventions, and assess rehabilitation progress for patients suffering from movement ailments such as hemiplegia.

Several therapy routines require deep breathing exercises as a key component and patients undergoing such therapies must perform these exercises regularly. Assessing the outcome of a therapy and tailoring its course necessitates monitoring a patient's compliance with the therapy. While therapy compliance monitoring is routine in a clinical environment, it is challenging to do in an at-home setting. This is so because a home setting lacks access to specialized equipment and skilled professionals needed to effectively monitor the performance of a therapy routine by a patient. For some types of therapies, these challenges can be addressed with the use of consumer-grade hardware, such as earphones and smartphones, as practical solutions. To accurately monitor breathing exercises using wireless earphones, this paper proposes a framework that has the potential for assessing a patient's compliance with an at-home therapy. The proposed system performs real-time detection of breathing phases and channels with high accuracy by processing a 500 ms audio signal through two convolutional neural networks. The first network, called a channel classifier, distinguishes between nasal and oral breathing, and a pause. The second network, called a phase classifier, determines whether the audio segment is from inhalation or exhalation. According to k-fold cross-validation, the channel and phase classifiers achieved a maximum F1 score of 97.99% and 89.46%, respectively. The results demonstrate the potential of using commodity earphones for real-time breathing channel and phase detection for breathing therapy compliance monitoring.Clinical relevance-This paper introduces a real-time monitoring system for breathing that can facilitate therapy compliance for several breathing-based exercises.

Many neurological conditions, e.g., a stroke, can cause patients to experience upper limb (UL) motor impairments that hinder their daily activities. For such patients, while rehabilitation therapy is key for regaining autonomy and restoring mobility, its long-term nature entails ongoing time commitment and it is often not sufficiently engaging. Virtual reality (VR) can transform rehabilitation therapy into engaging game-like tasks that can be tailored to patient-specific activities, set goals, and provide rehabilitation assessment. Yet, most VR systems lack built-in methods to track progress over time and alter rehabilitation programs accordingly. We propose using arm kinematic modeling and capability maps to allow a VR system to understand a user's physical capability and limitation. Next, we suggest two use cases for the VR system to utilize the user's capability map for tailoring rehabilitation programs. Finally, for one use case, it is shown that the VR system can emphasize and assess the use of specific UL joints.

Several therapy routines require deep breathing exercises as a key component and patients undergoing such therapies must perform these exercises regularly. Assessing the outcome of a therapy and tailoring its course necessitates monitoring a patient's compliance with the therapy. While therapy compliance monitoring is routine in a clinical environment, it is challenging to do in an at-home setting. This is so because a home setting lacks access to specialized equipment and skilled professionals needed to effectively monitor the performance of a therapy routine by a patient. For some types of therapies, these challenges can be addressed with the use of consumer-grade hardware, such as earphones and smartphones, as practical solutions. To accurately monitor breathing exercises using wireless earphones, this paper proposes a framework that has the potential for assessing a patient's compliance with an at-home therapy. The proposed system performs real-time detection of breathing phases and channels with high accuracy by processing a $\mathbf{500}$ ms audio signal through two convolutional neural networks. The first network, called a channel classifier, distinguishes between nasal and oral breathing, and a pause. The second network, called a phase classifier, determines whether the audio segment is from inhalation or exhalation. According to $k$-fold cross-validation, the channel and phase classifiers achieved a maximum F1 score of $\mathbf{97.99\%}$ and $\mathbf{89.46\%}$, respectively. The results demonstrate the potential of using commodity earphones for real-time breathing channel and phase detection for breathing therapy compliance monitoring.

Abstract With the increasing demand for cross-disciplinary technical and professional skill sets in the workforce, Mechatronics and Robotics Engineering (MRE) is quickly emerging as its own engineering discipline. However, developing and implementing MRE courses and curricula is challenging for many potential MRE educators because there are no standardized course structures, curricula, hardware and software platforms, or course materials. To address these challenges, a multi-institutional team conducted several workshops starting in 2018 to provide support for curriculum development in MRE and to create a vibrant community of college instructors interested in MRE. Ranging from a half-day to two days, the workshops provided guidance and perspectives from leaders in MRE education. Based on participant feedback from these workshops and our goal for greater impact, we planned and delivered a more intensive three-day, virtual, yet hands-on workshop in January 2022. The objectives of the workshop were: 1) prepare current and future MRE educators, 2) familiarize MRE educators with advances in undergraduate MRE education, 3) help unify and standardize MRE curricula and courses, 4) pave the way toward accreditation for MRE degree programs, 5) generate enthusiasm and a sense of community among MRE educators, and 6) promote diversity and inclusivity within the MRE community. Notably, this workshop differed from previous ones by including a significant hands-on experiential learning component, which provided sample laboratory assignments and projects that could form the foundations of introductory and advanced courses in MRE. Remote assistance was provided by workshop leaders and student assistants. Participants actively engaged in the activities, including doing "homework" every evening. A post-workshop survey showed that participants overwhelmingly felt that the workshop met their expectations, they were better prepared to teach mechatronics, they belonged within the MRE community, and the workshop helped them develop a new MRE course for their institutions. Participants also suggested areas for future training and skill development, which could be incorporated into the development of future workshops.

with applications to natural and intuitive human-robot interaction, digital health, and STEM education. Under the Research Experience for Teachers Site, GK-12 Fellows, DR and ITEST projects, all

Abstract For the effective adoption of wearable robotic devices for use by workers with upper-limb disabilities, the training of occupational therapists is of paramount importance. It requires the creation of targeted training materials and programs that can be deployed to instruct the end-users effectively. This paper will describe a training program intended to support engineering professionals who are expected to translate technical research to implementable recommendations for occupational therapists while avoiding overloading them with too much information. The project seeks to employ pertinent social and educational theories to formulate and enact the education, training, and professional development through extensive face-to-face interactions and practical demonstrations. For illustrative purposes, the paper will include an example lesson based on the use of a low-cost, 3D-printed exoskeleton kit. The functional prototype being developed will include a user interface to allow the engineering professionals to modify the program's parameters controlling the robotic wearable device using a web or mobile application. The context of this paper is a hands-on and weeklong professional development workshop that seeks to incorporate effective pedagogical strategies for preparing trainers of occupational therapists to develop relevant curriculum and training materials. The professional development workshop participants will comprise the engineering researchers who are developing a wearable robotic device to be used by workers with upper-limb disabilities. On the first day of the workshop, the facilitators of the professional development program will introduce to the engineering researchers the concepts of project-based learning, 5E instructional model, social capital theory, and cultural-history activity theory (CHAT). These theories will provide engineering researchers with guiding frameworks to develop valuable lessons while considering the interactions between occupational therapists and their patients, including the interpersonal/communicative aspects of those relationships and the cultural, historical, political, and economic dimensions. They will also consider the resources embedded in social structures and how they are accessed and mobilized for purposive actions. For the next two days, the engineering researchers will work in groups with support from the facilitators to create the instructional materials to train occupational therapists, including the presentations, handouts, activity sheets, and other documents. The engineering researchers will present their work to the other participants and facilitators for feedback on the fourth day. They will improve and modify their work on the last day before presenting it to all participants and facilitators. As an example of a lesson using the 5E instructional model and project-based learning, we will use a ready-to-assemble, low-cost, 3D-printed exoskeleton kit equipped with a servo motor with position feedback and a force sensor. The participants will assemble the kit and test four control modes: a) record and play a trajectory control, b) point-to-point movement control, c) basic admittance control, and d) virtual wall control. By testing the different control modes and modifying their parameters, they will experience the working principles behind exoskeleton technologies and reflect on how this technology can help workers with upper-limb disabilities. The final paper will provide details on the features that the user interface will have, the various procedures of the example lesson, and an investigation of the extent to which the lessons and activities designed by the participants align with the proposed theoretical frameworks.

Transcranial doppler (TCD) ultrasound probes are an invaluable tool in cerebral blood flow (CBF) studies. Their operation demands maintaining consistent pose on the subject throughout the experimental protocol. However, the displacement of the TCD probe during vestibular studies is common and substantially prolongs the experiment or even terminates it. This is a significant challenge for integrating motion-based vestibular studies with CBF investigations. In response, a mechatronics system is designed to allow remote repositioning of the TCD probe during data collection experiments while the subject is wearing a head mounted virtual reality (VR) display and seated in a vestibular disorientation device. This paper presents the design, prototype, and operation of this mechatronics apparatus.Clinical Relevance- The mechatronics apparatus of this paper can enable motion-based vestibular studies that entail the use of CBF velocity measurement and head-mounted virtual reality display.

This book offers a synthesis of research, curriculum examples, pedagogy models, and classroom recommendations for the effective use of robotics in STEM teaching and learning. Authors Chauhan and Kapila demonstrate how the use of educational robotics can catalyze and enhance student learning and understanding within the STEM disciplines. The book explores the implementation of design-based research (DBR); technological, pedagogical, and content knowledge (TPACK); and the 5E instructional model; among others. Chapters draw on a variety of pedagogical scaffolds to help teachers deploy educational robotics for classroom use, including research-driven case studies, strategies, and standards-aligned lesson plans from real-life settings. This book will benefit STEM teachers, STEM teacher educators, and STEM education researchers.

When taught through traditional approaches, students find it challenging to attain a deep understanding of science, technology, engineering, and mathematics (STEM) concepts and fail to envision the implications of STEM disciplines for real-life situations. Inquiry-based learning can help students in deeper processing of concepts, better retention of ideas, and improved perceptions toward STEM fields. This chapter describes the 5E instructional model, a curriculum and instruction planning approach grounded in inquiry-based and constructivist learning theories, and its utilization to create robotics-based science learning units. The chapter first explores the five phases of the model—engage, explore, explain, elaborate, and evaluate—through which students ask questions, observe phenomena, analyze data, synthesize findings, and present conclusions to solve problems and investigate new ideas. Using two detailed examples, the chapter illustrates an implementation of the 5E model in designing robotics-based standards-aligned science learning experiences as well as how they promote hands-on engagement and make abstract concepts easier to understand. Finally, the chapter suggests recommendations for educators to effectively employ the 5E model in designing robotics-based learning experiences for students and teachers.