Robert D. Howe

The unpredictable nature of outdoor settings introduces numerous safety concerns, making hazard detection crucial for safe navigation. To address this issue, this paper proposes a sidewalk hazard detection system that combines a Variational Autoencoder (VAE) with a One-Class Support Vector Machine (OCSVM), using a wearable RGB camera as the primary sensing modality to enable low-cost, portable deployment and provide visual detail for detecting surface irregularities and unexpected objects. The VAE is trained exclusively on clean, obstruction-free sidewalk data to learn normal appearance patterns. At inference time, the reconstruction error produced by the VAE is used to identify spatial anomalies within each frame. These flagged anomalies are passed to an OCSVM, which determines whether they constitute a non-hazardous anomaly or a true hazardous anomaly that may impede navigation. To support this approach, we introduce a custom dataset consisting of over 20,000 training images of normal sidewalk scenes and 8000 testing frames containing both hazardous and non-hazardous anomalies. Experimental results demonstrate that the proposed VAE + OCSVM model achieves an AUC of 0.92 and an F1 score of 0.85, outperforming baseline anomaly detection models for outdoor sidewalk navigation. These findings indicate that the hybrid method offers a robust solution for sidewalk hazard detection in real-world outdoor environments.

This paper introduces a vision-based environment quantification pipeline designed to tailor the assistance provided by lower limb assistive devices during the transition from level walking to stair navigation. The framework consists of three components: staircase detection, transitional step prediction, and staircase dimension estimation. These components utilize an RGB-D camera worn on the chest and an Inertial Measurement Unit (IMU) worn at the hip. To detect ascending stairs, we employed a YOLOv3 model applied to continuous recordings, achieving an average accuracy of 98 . 1% . For descending stair detection, an edge detection algorithm was used, resulting in a pixel-wise edge localization accuracy of 89 . 1% . To estimate user locomotion speed and footfall, the IMU was positioned on the participant’s left waist, and the RGB-D camera was mounted at chest level. This setup accurately captured step lengths with an average accuracy of 94 . 4% across all participants and trials, enabling precise determination of the number of steps leading up to the transitional step on the staircase. As a result, the system accurately predicted the number of steps and localized the final footfall with an average error of 5 . 77 cm, measured as the distance between the predicted and actual placement of the final foot relative to the target destination. Finally, to capture the dimensions of the staircase’s tread depth and riser height, an

The reliability of manipulation in unstructured environments is unknown, but 1 in 10,000 dropped items may be acceptable.

Perception by touch is fundamentally linked to the motor system. A hallmark of this linkage takes the form of stereotyped haptic "exploratory procedures" [1], movement patterns that emerge when people set a perceptual goal such as judging the roughness of a textured surface. This paper expands the study of touch-directed movements by asking what patterns emerge when people encounter and interact with novel objects without explicitly specified goals. Participants were invited to freely interact with an art installation containing novel objects with distinct design features, intended to vary familiarity, structural affordance, and aesthetic response. Objects' affordances were additionally varied over time by utilizing jamming, a physical mechanism that induces changes in stiffness and plasticity. From video recordings, four categories of spontaneous "interactive procedures" differentiated by underlying goals were reliably identified: passive observational, active perceptual, constructive, and hedonic. Perceptual actions were most frequent, indicating an overriding goal of acquiring information about physical properties. The prevalence of other interactive procedures varied across objects, demonstrating the influence of perceptual affordances and prior knowledge. Changes in state further moderated interactions, such that interactions were longer in the stiff/jammed state, and the occurrence of a state change during an interactive procedure lengthened its duration. These findings extend our understanding of haptic exploration beyond explicitly goal-directed contexts, revealing how spontaneous responses in complex and dynamic environments are linked to perceptual outcomes and prior knowledge.

Remotely controlling a robotic platform could be challenging, specifically in the presence of uncertainties between the human and robot. A major problem is that human’s inputs to the system may not meet all task-specific criteria. For example, in teleoperated medical robotic applications, disrupted haptic signals (force feedback) can make it difficult for the human operator to command precise maneuvers, preventing the robot from maintaining stable and safe task execution. This paper addresses this issue by proposing an imitation learning strategy through human-robot interactions. Learning from successful task executions, the proposed methodology can condition teleoperation signals and human operator’s inputs to determine the legibility of the concurrent execution of the task. Utilizing collections of structured demonstration data of successful scenarios, the current study incorporates the renowned deep learning technique, recurrent neural network with long short-term memory units to achieve the goal. The developed framework learns optimum policies from expert demonstrations through human interactions. Once properly learned, the framework mimics the accomplished executions to protect and guarantee the performance in the case of disrupted teleoperation signals. The results demonstrate the effectiveness of the developed framework in performing teleoperation tasks under uncertainty-perturbed scenarios, tested over a long distance between Australia and the USA. Additionally, the proposed learning-based strategy outperforms state-of-the-art teleoperation methods. The developed framework enables the human operator to achieve a zero failure ratio, a critical factor in safety-sensitive applications such as remote clinical diagnosis.

Accurate estimation of human hand configuration and the forces they exert is critical for effective teleoperation and skill transfer in robotic manipulation. A deeper understanding of human interactions with objects can further enhance teleoperation performance. To address this need, researchers have explored methods to capture and translate human manipulation skills and applied forces to robotic systems. Among these, biosignal-based approaches, particularly those using forearm ultrasound data, have shown significant potential for estimating hand movements and finger forces. In this study, we present a method for simultaneously estimating manipulation skills and applied hand force using forearm ultrasound data. Data collected from seven participants were used to train deep learning models for classifying manipulation skills and estimating grasp force. Our models achieved an average classification accuracy of 94.87 percent plus or minus 10.16 percent for manipulation skills and an average root mean square error (RMSE) of 0.51 plus or minus 0.19 Newtons for force estimation, as evaluated using five-fold cross-validation. These results highlight the effectiveness of forearm ultrasound in advancing human-machine interfacing and robotic teleoperation for complex manipulation tasks. This work enables new and effective possibilities for human-robot skill transfer and tele-manipulation, bridging the gap between human dexterity and robotic control.

The unpredictable nature of outdoor settings introduces numerous safety concerns, making hazard detection crucial for safe navigation. This paper introduces a novel system for sidewalk safety navigation utilizing a hybrid approach that combines a Variational Autoencoder (VAE) with a One-Class Support Vector Machine (OCSVM). The system is designed to detect anomalies on sidewalks that could potentially pose walking hazards. A dataset comprising over 15,000 training frames and 5,000 testing frames was collected using video recordings, capturing various sidewalk scenarios, including normal and hazardous conditions. During deployment, the VAE utilizes its reconstruction mechanism to detect anomalies within a frame. Poor reconstruction by the VAE implies the presence of an anomaly, after which the OCSVM is used to confirm whether the anomaly is hazardous or non-hazardous. The proposed VAE model demonstrated strong performance, with a high Area Under the Curve (AUC) of 0.94, effectively distinguishing anomalies that could be potential hazards. The OCSVM is employed to reduce the detection of false hazard anomalies, such as manhole or water valve covers. This approach achieves an accuracy of 91.4%, providing a highly reliable system for distinguishing between hazardous and non-hazardous scenarios. These results suggest that the proposed system offers a robust solution for hazard detection in uncertain environments.

Controlling hand exoskeletons for assisting impaired patients in grasping tasks is challenging because it is difficult to infer user intent. We hypothesize that majority of daily grasping tasks fall into a small set of categories or modes which can be inferred through real-time analysis of environmental geometry from 3D point clouds. This paper presents a low-cost, real-time system for semantic image labeling of household scenes with the objective to inform and assist activities of daily living. The system consists of a miniature depth camera, an inertial measurement unit and a microprocessor. It is able to achieve 85% or higher accuracy at classification of predefined modes while processing complex 3D scenes at over 30 frames per second. Within each mode it can detect and localize graspable objects. Grasping points can be correctly estimated on average within 1 cm for simple object geometries. The system has potential applications in robotic-assisted rehabilitation as well as manual task assistance.

Wearable robots have shown great promise in aiding individuals with reduced mobility and enhancing human performance across various applications. To achieve optimal assistance, accurate estimation of muscle dynamics has shown promise in designing adaptive control strategies. Among different techniques, B-mode ultrasonography produced with linear array transducers have gained popularity as a gold-standard imaging tool, providing non-invasive solutions to measure in-vivo muscle dynamics. B-mode ultrasound has been employed to infer muscle thickness, fascicle lengths, pennation angles, and muscle force using neural networks, offering a valuable tool for designing individualized control strategies. The effectiveness of this measuring tool depends on integrating transducers into the wearable robot, but B-mode relies on large transducers. Studies have explored smaller single-element transducers for better wearability for muscle thickness estimation. However, their ability to infer more complex muscle architecture parameters using automated techniques is yet to be determined. In this study, we propose an approach to extract M-mode traces from B-mode images to simulate signals from single-element transducers. We then employ various machine learning architectures to infer muscle pennation angle and fascicle length. Preliminary results indicate promising performance from the CNN+Transformer (2-layer spatial) + Transformer (2-layer temporal) models, with results from the CNN+LSTM models (with a RMSE of 0.02 radian for pennation angle and 2.54mm for fascicle lengths). This study paves the way for enabling the use of smaller and more portable single-element transducers for wearable robotic applications. The link to the code is https: https://github.com/raku-slyu/AB-Mode-Utrasound.

Tendon-driven robotic catheters are capable of precise execution of minimally invasive cardiac procedures including ablations and imaging. These procedures require accurate mathematical models of not only the catheter and tendons but also their interactions with surrounding tissue and vasculature in order to control the robot path and interaction. This paper presents a mechanical model of a tendon-driven robotic catheter system based on Cosserat rods and integrated with a stable, implicit Euler scheme. We implement the Cosserat rod as a model for a simple catheter centerline and validate its physical accuracy against a large deformation analytical model and experimental data. The catheter model is then supplemented by adding a second Cosserat rod to model a single tendon, using penalty forces to define the constraints of the tendon-catheter system. All the model parameters are defined by the catheter properties established by the design. The combined model is validated against experimental data to confirm its physical accuracy. This model represents a new contribution to the field of robotic catheter modeling in which both the tendons and catheter are modeled by mechanical Cosserat rods and fully-validated against experimental data in the case of the single rod system.

Abstract The human body constantly experiences mechanical loading. However, quantifying internal loads within the musculoskeletal system remains challenging, especially during unconstrained dynamic activities. Conventional measures are constrained to laboratory settings, and existing wearable approaches lack muscle specificity or validation during dynamic movement. Here, we present a strategy for estimating corresponding joint torque from muscles with different architectures during various dynamic activities using wearable A-mode ultrasound. We first introduce a method to track changes in muscle thickness using single-element ultrasonic transducers. We then estimate elbow and knee torque with errors less than 7.6% and coefficients of determination ( R 2 ) greater than 0.92 during controlled isokinetic contractions. Finally, we demonstrate wearable joint torque estimation during dynamic real-world tasks, including weightlifting, cycling, and both treadmill and outdoor locomotion. The capability to assess joint torque during unconstrained real-world activities can provide new insights into muscle function and movement biomechanics, with potential applications in injury prevention and rehabilitation.

To enhance wearable robots, understanding user intent and environmental perception with novel vision approaches is needed.

Skeletal muscle velocity is a key indicator of neuromuscular function, and monitoring its changes plays important role in tracking the progression of musculoskeletal diseases, injuries, and fatigue. However, existing methods for non-invasive estimation of skeletal muscle velocity primarily use B- mode ultrasound, often with processing methods that are time-consuming or computationally expensive, with varying accuracy based on tissue structure. Here, we propose a spectral Doppler envelope estimation method designed for skeletal muscle measurements. When compared to the modified signal noise slope intersection (MSNSI) method, our method reduces the overall mean absolute error by 13.9% and the mean absolute zero error by 82.1%. We validated our method using a portable ultrasound system on a benchtop setup that mimics the acoustic properties, measurement angles, and velocity patterns of skeletal muscles. Ex vivo and in vivo muscle velocity estimates of parallel and pennate muscles will be compared to those obtained using the MSNSI method and manual tracking of B-Mode images. Our proposed method could enable automated estimation of skeletal muscle velocities during dynamic activities in unconstrained environments, providing new insight into neuromuscular function and movement biomechanics, with potential applications in monitoring fatigue, disease progression, or injury recovery. [Work supported by the National Science Foundation.]

Deployable structures provide adaptability and versatility for applications such as temporary architectures, space structures, and biomedical devices. Jamming is a mechanical phenomenon with which dramatic changes in stiffness can be achieved by increasing the frictional and kinematic coupling between constituents in a structure by applying an external pressure. This study applies jamming, which has been primarily used in medium-scale soft robotics applications to large-scale deployable structures with components that are soft and compact during transport, but rigid upon deployment. It proposes a new jamming structure with a novel built-in actuation mechanism which enables high-performance at large scales: a composite beam made of rectangular segments along a cable which can be pre-tensioned and thus jammed. Two theoretical models are developed to provide insights into the mechanical behavior of the composite beams and predict their performance under loading. A scale model of a deployable bridge is built using the tension-based composite beams, and the bridge is deployed and assembled by air with a drone demonstrating the versatility and viability of the proposed approach for robotics applications.

Brightness-mode (B-mode) ultrasound has been used to measure in vivo muscle dynamics for assistive devices. Estimation of fascicle length from B-mode images has now transitioned from time-consuming manual processes to automatic methods, but these methods fail to reach pixel-wise accuracy across extended locomotion. In this work, we aim to address this challenge by combining a U-net architecture with proven segmentation abilities with an LSTM component that takes advantage of temporal information to improve validation accuracy in the prediction of fascicle lengths. Using 64,849 ultrasound frames of the medial gastrocnemius, we semi-manually generated ground-truth for training the proposed U-net-LSTM. Compared with a traditional U-net and a CNNLSTM configuration, the validation accuracy, mean square error (MSE), and mean absolute error (MAE) of the proposed U-net-LSTM show better performance (91.4%, MSE =0.1± 0.03 mm, MAE =0.2± 0.05 mm). The proposed framework could be used for real-time, closed-loop wearable control during real-world locomotion.

Surface electromyography (EMG) for estimating neuromuscular activation suffers from several confounds, including changes in electrode-skin conditions due to contact pressure variation, sweat, and dehydration. These conditions lead to variation in bioimpedance across skin locations and thus variation in the EMG voltages measured at skin electrodes. This paper presents a system that combines standard EMG measurements with continuous bioimpedance sensing using only the electrodes already necessary for EMG. State of the art techniques for simultaneous EMG/Bioimpedance require two extra electrodes for bioimpedance sensing, away from the area of muscle activation. The key to our approach is stimulating the skin-electrode interfaces with voltage frequencies well above native EMG signals. By combining analog circuits, digital signal processing, and analytic calculations using bioimpedance principles, the system can decouple the intertwined signals. We provide design rationales for our system and benchtop characterizations to show accurate bioimpedance measurements (R<sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup>~0.96) under varying simulated EMG signals. We demonstrate system utility in vivo during controlled force generation tasks where controlled alteration to subjects' skin-electrode conditions produce changes in both EMG and bioimpedance. This work provides a basis for continuous, simultaneous EMG and bioimpedance sensing, so impedance variations can be used to normalize EMG signals for improved muscle effort estimates.

Reliable grasping and manipulation in daily tasks and unstructured environments require accurate contact modeling and grasp stability estimation. A key component is the coefficient of friction, which is variable and dependent on many factors. However, robotics applications often use Coulomb's model of friction, which ignores this variability and instead assumes that the coefficient of friction is a constant. In this work, we conducted sliding experiments with robot fingers and a robot hand, and show that rubber friction varies strongly with normal force <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$F_{n}$</tex-math></inline-formula> and contact velocity <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$v$</tex-math></inline-formula> , and includes a significant stochastic component. We present a framework for modeling the coefficient of friction <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\mu$</tex-math></inline-formula> as a distribution rather than a constant, and show how this distribution can be narrowed when given a prior on <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$F_{n}$</tex-math></inline-formula> or <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$v$</tex-math></inline-formula> . For a given distribution, the likelihood of slipping is a continuous function with respect to the tangential-to-normal force ratio, instead of a step function according to Coulomb's law. By modeling friction as a function of <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$F_{n}$</tex-math></inline-formula> and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$v$</tex-math></inline-formula> , we demonstrate that friction parameters can be estimated using regression models from a single sliding stroke of the fingertip against the object surface, and that strokes spanning a larger range of <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$F_{n}$</tex-math></inline-formula> - <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$v$</tex-math></inline-formula> space provide better friction estimates. These results can be applied to grasp control to enable a quantitative trade-off between the likelihood of slipping vs. grasp force levels, and to sliding manipulation planning by elucidating the relationship between desired velocity and anticipated force levels. Application of this model to machine learning has the potential to enhance reinforcement learning and sim-to-real transfer by providing more accurate representations of frictional behavior.

Cutaneous electrodes are routinely used for noninvasive electrophysiological sensing of signals from the brain, the heart, and the neuromuscular system. These bioelectronic signals propagate as ionic charge from their sources to the skin-electrode interface where they are then sensed as electronic charge by the instrumentation. However, these signals suffer from low signal-to-noise ratio arising from the high impedance at the tissue-to-electrode contact interface. This paper reports that soft conductive polymer hydrogels made purely of poly(3,4-ethylenedioxy-thiophene) doped with poly(styrene sulfonate) present nearly an order of magnitude decrease in the skin-electrode contact impedance (88%, 82%, and 77% at 10, 100, and 1 kHz, respectively) when compared to clinical electrodes in an ex vivo model that isolates the bioelectrochemical features of a single skin-electrode contact. Integrating these pure soft conductive polymer blocks into an adhesive wearable sensor enables high fidelity bioelectronic signals with higher signal-to-noise ratio (average 2.1 dB increase, max 3.4 dB increase) when compared to clinical electrodes across all subjects. The utility of these electrodes is demonstrated in a neural interface application. The conductive polymer hydrogels enable electromyogram-based velocity control of a robotic arm to complete a pick and place task. This work provides a basis for the characterization and use of conductive polymer hydrogels to better couple human and machine.

In the field of wearable robotics, assistance needs to be individualized for the user to maximize benefit. Information from muscle fascicles automatically recorded from brightness mode (B-mode) ultrasound has been used to design assistance profiles that are proportional to the estimated muscle force of young individuals. There is also a desire to develop similar strategies for older adults who may have age-altered physiology. This study introduces and validates a ResNet + 2x-LSTM model for extracting fascicle lengths in young and older adults. The labeling was generated in a semimanual manner for young (40,696 frames) and older adults (34,262 frames) depicting B-mode imaging of the medial gastrocnemius. First, the model was trained on young and tested on both young (R2 = 0.85, RMSE = 2.36 ± 1.51 mm, MAPE = 3.6%, aaDF = 0.48 ± 1.1 mm) and older adults (R2 = 0.53, RMSE = 4.7 ± 2.51 mm, MAPE = 5.19%, aaDF = 1.9 ± 1.39 mm). Then, the performances were trained across all ages (R2 = 0.79, RMSE = 3.95 ± 2.51 mm, MAPE = 4.5%, aaDF = 0.67 ± 1.8 mm). Although age-related muscle loss affects the error of the tracking methodology compared to the young population, the absolute percentage error for individual fascicles leads to a small variation of 3–5%, suggesting that the error may be acceptable in the generation of assistive force profiles.

This invention comprises an optical displacement sensor that uses the inverse-square attenuation of light reflected from a diffused surface to calculate the distance from the sensor to the reflecting surface. Light emerging from an optical fiber or the like is directed onto the surface whose distance is to be measured. The intensity I of reflected light is angle dependent, but within a sufficiently small solid angle it falls off as the inverse square of the distance from the surface. At least a pair of optical detectors are mounted to detect the reflected light within the small solid angle, their ends being at different distances R and R+.DELTA.R from the surface. The distance R can then be found in terms of the ratio of the intensity measurements and the separation length as ##EQU1##