Aaron M. Dollar

Abstract Habitat fragmentation, climate change, poaching, human-wildlife conflicts, and infectious diseases are the main threats to biodiversity conservation. They alter host-pathogen dynamics, reduce viable conservation areas, and promote genetic isolation, resulting in physiological stress among animal populations. Moreover, increased proximity between domestic and wild animals further facilitates disease spillovers exposing naïve host species and ecosystems to new pathogens. Of the more than 200 known zoonotic diseases, approximately 60% originate from animals, contributing significantly to the global infectious disease burden. Here, we describe the development of an innovative non-invasive approach for biological sampling that has been validated in mice and shelter cats. Our device consists of a disposable plastic cassette that through odor attractants lures animals to lick a filter paper. This saliva collection approach allowed for the detection of RNA viruses by RTqPCR and third-generation sequencing. RTqPCR oral swab and licked paper results showed that both methods significantly predicted serological status. Our sequencing results revealed the richness of the gene space, demonstrating the potential of this device for discovering rare or unknown species circulating in the saliva donor, enabling this player to be recognized as an environmental sentinel. This study demonstrates the feasibility of deploying this device in sheltered/captive animal settings as well as under laboratory simulations of different environments, providing necessary foundations for future field applications. Our methodology holds great potential for monitoring zoonotic pathogens in both captive and free-ranging animals, to even possibly allow proactive mitigation measures prior to spillover, without interfering with the natural animal behaviour and social structures. Visual abstract

The digitization of natural history collections over the past three decades has unlocked a treasure trove of specimen imagery and metadata. There is great interest in making this data more useful by further labeling it with additional trait data, and modern "deep learning" machine learning techniques utilizing convolutional neural nets (CNNs) and similar networks show particular promise to reduce the amount of required manual labeling by human experts, making the process much faster and less expensive. However, in most cases, the accuracy of these approaches is too low for reliable utilization of the automatic labeling, typically in the range of 80-85% accuracy. In this paper, we present and validate an approach that can greatly improve this accuracy, essentially by examining the "confidence" that the network has in the generated label as well as utilizing a user-defined threshold to reject labels that fall below a chosen level. We demonstrate that a naive model that produced 86% initial accuracy can achieve improved performance - over 95% accuracy (rejecting about 40% of the labels) or over 99% accuracy (rejecting about 65%) by selecting higher confidence thresholds. This gives flexibility to adapt existing models to the statistical requirements of various types of research and has the potential to move these automatic labeling approaches from being unusably inaccurate to being an invaluable new tool. After validating the approach in a number of ways, we annotate the reproductive state of a large dataset of over 600,000 herbarium specimens. The analysis of the results points at under-investigated correlations as well as general alignment with known trends. By sharing this new dataset alongside this work, we want to allow biologists to gather insights for their own research questions, at their chosen point of accuracy/coverage trade-off.

Traditional robot control relies on analytical methods that require precise system models, which are hard to apply in real-world settings and limit generalization to arbitrary tasks. However, systems like serial manipulators and passively adaptive hands feature inherently stable regions without control discontinuities like loss of contact or singularities. In these regions, approximate controllers focusing on the correct direction of motion enable successful coarse manipulation. When coupled with a rough estimation of the motion magnitude, precision manipulation is achieved. Leveraging this insight, we introduce a novel inverse Jacobian estimation method that independently estimates the primary motion direction and magnitude of the manipulator’s actuators. Our method efficiently estimates the direct mapping from task to actuator space with no need for a priori system knowledge enabling the same framework to control both hands and arms without compromising task performance. We present a novel control method with no a priori knowledge for precision manipulation. Experiments on the Yale Model O hand, Yale Stewart Hand, and a UR5e arm demonstrate that the inverse Jacobians estimated via our approach enable real-time control with submillimeter precision in manipulation tasks. These results highlight that online self-ID data alone is sufficient for precise real-world manipulation.

Camera-to-robot (also known as eye-to-hand) calibration is a critical component of vision-based robot manipulation. Traditional marker-based methods often require human intervention for system setup. Furthermore, existing autonomous markerless calibration methods typically rely on pre-trained robot tracking models that impede their application on edge devices and require fine-tuning for novel robot embodiments. To address these limitations, this paper proposes a model-based markerless camera-to-robot calibration framework, ARC-Calib, that is fully autonomous and generalizable across diverse robots and scenarios without requiring extensive data collection or learning. First, exploratory robot motions are introduced to generate easily trackable trajectory-based visual patterns in the camera’s image frames. Then, a geometric optimization framework is proposed to exploit the coplanarity and collinearity constraints from the observed motions to iteratively refine the estimated calibration result. Our approach eliminates the need for extra effort in either environmental marker setup or data collection and model training, rendering it highly adaptable across a wide range of real-world autonomous systems. Extensive experiments are conducted in both simulation and the real world to validate its robustness and generalizability.

Force-sensing capabilities are essential for robot manipulation systems. However, commonly used wrist-mounted force/torque sensors are heavy, fragile, and expensive, and tactile sensors require adding fragile circuitry to the robot fingers while only providing force information local to the contact. Here, we present a vision-based contact force estimator that serves as a more cost-effective and easier-to-implement alternative to existing force sensors by leveraging deformations of compliant hands upon contacts when compliant hands are in use. Our approach uses an estimator that visually observes a specialized compliant robot hand (available open source with easy fabrication through 3D printing) and predicts the contact force on the basis of its elastic deformation upon external forces. Because using wrist-mounted cameras to observe the gripper is common for robot manipulation systems, our method can obtain additional force information provided that the gripper is compliant. We optimized our compliant hand to minimize friction and avoid singularities in finger configurations, and we introduced memory to the estimator to combat the partial observability of the contact forces from the remaining friction and hysteresis. In addition, the estimator was made robust to background distractions and finger occlusions using vision foundation models to segment out the fingers. Although it is less accurate and slower than commercial force/torque sensors, we experimentally demonstrated the accuracy and robustness of our estimator (achieving between 0.2 newton and 0.4 newton error) and its utility during a variety of manipulation tasks using the gripper in the presence of noisy backgrounds and occlusions.

This paper introduces Model Q-II, an enhanced underactuated robotic hand designed to improve dexterous manipulation through expanded grasping modes and manipulation primitives. The Model Q-II incorporates tripod and enhanced power grasping modes, achieving increased versatility without adding additional actuators. The design employs passive mechanisms, such as lateral contact walls and a finger-locking system, to facilitate seamless transitions between modes, enabling precise pinch-to-tripod and pinch-to-power gating. These enhancements allow the hand to perform complex in-hand manipulations, including multi-directional object positioning. Theoretical analysis, simulations, and experimental evaluations validate the hand's performance, demonstrating improved grasping force, range, and manipulation capabilities. The results highlight Model Q-II's ability to handle various tasks, offering a robust, cost-effective solution for applications requiring both precise and powerful grasping.

Identifying predictive world models for robots from sparse online observations is essential for robot task planning and execution in novel environments. However, existing methods that leverage differentiable programming to identify world models are incapable of jointly optimizing the geometry, appearance, and physical properties of the scene. In this work, we introduce a novel rigid object representation that allows the joint identification of these properties. Our method employs a novel differentiable point-based geometry representation coupled with a grid-based appearance field, which allows differentiable object collision detection and rendering. Combined with a differentiable physical simulator, we achieve end-to-end optimization of world models or rigid objects, given the sparse visual and tactile observations of a physical motion sequence. Through a series of world model identification tasks in simulated and real environments, we show that our method can learn both simulation- and rendering-ready rigid world models from only one robot action sequence. The code and additional videos are available at our project website: <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://tianyi20.github.io/rigid-world-model.github.io/</uri>.

The robot manipulation ecosystem currently faces issues with integrating open-source components and reproducing results. This limits the ability of the community to benchmark and compare the performance of different solutions to one another in an effective manner, instead relying on largely holistic evaluations. As part of the COMPARE Ecosystem project, we are developing modular grasping and manipulation pipeline infrastructure in order to streamline performance benchmarking. The infrastructure will be used towards the establishment of standards and guidelines for modularity and improved open-source development and benchmarking. This paper provides a high-level overview of the architecture of the pipeline infrastructure, experiments conducted to exercise it during development, and future work to expand its modularity.

Camera-to-robot (also known as eye-to-hand) calibration is a critical component of vision-based robot manipulation. Traditional marker-based methods often require human intervention for system setup. Furthermore, existing autonomous markerless calibration methods typically rely on pre-trained robot tracking models that impede their application on edge devices and require fine-tuning for novel robot embodiments. To address these limitations, this paper proposes a model-based markerless camera-to-robot calibration framework, ARC-Calib, that is fully autonomous and generalizable across diverse robots and scenarios without requiring extensive data collection or learning. First, exploratory robot motions are introduced to generate easily trackable trajectory-based visual patterns in the camera's image frames. Then, a geometric optimization framework is proposed to exploit the coplanarity and collinearity constraints from the observed motions to iteratively refine the estimated calibration result. Our approach eliminates the need for extra effort in either environmental marker setup or data collection and model training, rendering it highly adaptable across a wide range of real-world autonomous systems. Extensive experiments are conducted in both simulation and the real world to validate its robustness and generalizability.

Identifying predictive world models for robots in novel environments from sparse online observations is essential for robot task planning and execution in novel environments. However, existing methods that leverage differentiable programming to identify world models are incapable of jointly optimizing the geometry, appearance, and physical properties of the scene. In this work, we introduce a novel rigid object representation that allows the joint identification of these properties. Our method employs a novel differentiable point-based geometry representation coupled with a grid-based appearance field, which allows differentiable object collision detection and rendering. Combined with a differentiable physical simulator, we achieve end-to-end optimization of world models, given the sparse visual and tactile observations of a physical motion sequence. Through a series of world model identification tasks in simulated and real environments, we show that our method can learn both simulation- and rendering-ready world models from only one robot action sequence. The code and additional videos are available at our project website: https://tianyi20.github.io/rigid-world-model.github.io/

The digitization of natural history collections over the past three decades has unlocked a treasure trove of specimen imagery and metadata. There is great interest in making this data more useful by further labeling it with additional trait data, and modern deep learning machine learning techniques utilizing convolutional neural nets (CNNs) and similar networks show particular promise to reduce the amount of required manual labeling by human experts, making the process much faster and less expensive. However, in most cases, the accuracy of these approaches is too low for reliable utilization of the automatic labeling, typically in the range of 80-85% accuracy. In this paper, we present and validate an approach that can greatly improve this accuracy, essentially by examining the confidence that the network has in the generated label as well as utilizing a user-defined threshold to reject labels that fall below a chosen level. We demonstrate that a naive model that produced 86% initial accuracy can achieve improved performance - over 95% accuracy (rejecting about 40% of the labels) or over 99% accuracy (rejecting about 65%) by selecting higher confidence thresholds. This gives flexibility to adapt existing models to the statistical requirements of various types of research and has the potential to move these automatic labeling approaches from being unusably inaccurate to being an invaluable new tool. After validating the approach in a number of ways, we annotate the reproductive state of a large dataset of over 600,000 herbarium specimens. The analysis of the results points at under-investigated correlations as well as general alignment with known trends. By sharing this new dataset alongside this work, we want to allow ecologists to gather insights for their own research questions, at their chosen point of accuracy/coverage trade-off.

Due to the nature of their implementation, nearly all low-level fabrication processes produce solidly filled structures. However, lattice structures are significantly stronger for the same amount of material, resulting in structures that are much lighter and more materially efficient. Here we propose an approach for fabricating lattice structures that echoes 3D printing techniques. In it, a modular chain of specially designed links is “extruded” onto a substrate to produce various lattices configurations depending on the chosen assembly algorithm, ranging from rigid regular lattices with nodal connectivity of 12, octet-truss, to significantly less dense configurations. Compared to conventional additive manufacturing methods, our approach allows for efficient use of nearly any material or combination of materials to construct lattices with programmed arrangements. We experimentally demonstrate that a 3x3x2 lattice structure (287 total links) is fabricated in 27 minutes via a modified robotic arm and can support approximately 1000 N in compression testing. Extrusion-based 3D printing, in which a filament of material is extruded through a nozzle has been widely adopted. Here, Zhe Xu and Aaron Dollar report an approach for fabricating lattice structures in which a modular chain of specially designed links is “extruded” onto a substrate allowing for construction of multiscale structures that are efficient in weight and varied in composition.

Calibrating robots into their workspaces is crucial for manipulation tasks. Existing calibration techniques often rely on sensors external to the robot (cameras, laser scanners, etc.) or specialized tools. This reliance complicates the calibration process and increases the costs and time requirements. Furthermore, the associated setup and measurement procedures require significant human intervention, which makes them more challenging to operate. Using the built-in force-torque sensors, which are nowadays a default component in collaborative robots, this work proposes a self-calibration framework where robot-environmental spatial relations are automatically estimated through compliant exploratory actions by the robot itself. The self-calibration approach converges, verifies its own accuracy, and terminates upon completion, autonomously purely through interactive exploration of the environment’s geometries. Extensive experiments validate the effectiveness of our self-calibration approach in accurately establishing the robot-environment spatial relationships without the need for additional sensing equipment or any human intervention.

We study the problem of rapidly identifying contact dynamics of unknown objects in partially known environments. The key innovation of our method is a novel formulation of the contact dynamics estimation problem as the joint estimation of contact geometries and physical parameters. We leverage DeepSDF, a compact and expressive neural-network-based geometry representation over a distribution of geometries, and adopt a particle filter to estimate both the geometries in contact and the physical parameters. In addition, we couple the estimator with an active exploration strategy that plans information-gathering moves to further expedite online estimation. Through simulation and physical experiments, we show that our method estimates accurate contact dynamics with fewer than 30 exploration moves for unknown objects touching partially known environments.

In closed-circuit racing such as NASCAR, vehicles often travel on highly-banked oval tracks that are dominated by left-hand corners. The aim of this paper is to study the generation and interpretation of the associated asymmetric GG diagrams and performance metrics. These include GG diagrams on inclined and/or cambered planar road surfaces, and GG diagrams on curved surfaces such as the interior face of an inverted cone. It is shown that a general fixed point on a traditional GG diagram is associated with acceleration or braking along a logarithmic spiral. Under pure acceleration and braking this spiral becomes a straight line, while under constant-speed cornering it becomes a circle. Some of these ideas are developed using a single-track car model. The paper then addresses the calculation and interpretation of GG-diagrams and performance metrics for a Generation 7 (Gen-7) NASCAR on curved road surfaces. The car's stability and performance limits under extreme lateral acceleration conditions are of particular interest. The main results come from a high-fidelity vehicle model and a representative set of parameters.

Precision gas analyzers are widely used in ecological research for manual measurement of soil carbon flux, a key metric used in the study of climate change. We present a generational update to the first low-cost, autonomous, closed-chamber style soil CO2 flux sensors (Fluxbots). Fluxbot 2.0 is the first such low-cost autonomous flux chamber capable of real-time wireless data transmission, which enables ecologists conducting in situ soil carbon flux surveys to set up their own wireless sensor arrays, reporting carbon flux data in real time at a very high level of temporal resolution. The system’s low cost (less than 500 USD per unit) and long-range cellular data transmission capabilities also allow for greatly improved spatial resolution. Additionally, the updated system consumes significantly less power, resulting in the ability to be deployed for longer than 10 × the battery lifetime of the original version on a single charge.

The ability to plan and control robotic in-hand manipulation is challenged by several issues, including the required amount of prior knowledge of the system and the sophisticated physics that varies across different robot hands or even grasp instances. One of the most direct models of in-hand manipulation is the inverse Jacobian, which can directly map from the desired in-hand object motions to the required hand actuator controls. However, acquiring such inverse Jacobians without complex hand-object system models is typically infeasible. We present a method for controlling in-hand manipulation using inverse Jacobians that are self-identified by a particle filter-based estimation scheme that leverages the ability of underactuated hands to maintain a passively stable grasp during self-identification movements. This method requires no a priori knowledge of the specific hand-object system and learns the system’s inverse Jacobian through small exploratory motions. Our system approximates the underlying inverse Jacobian closely, which can be used to perform manipulation tasks across a range of objects successfully. With extensive experiments on a Yale Model O hand, we show that the proposed system can provide accurate in-hand manipulation of sub-millimeter precision and that the inverse Jacobian-based controller can support real-time manipulation control of up to 900Hz.

Continuous exploration without interruption is important in scenarios such as search and rescue and precision agriculture, where consistent presence is needed to detect events over large areas. Ergodic search already derives continuous trajectories in these scenarios so that a robot spends more time in areas with high information density. However, existing literature on ergodic search does not consider the robot's energy constraints, limiting how long a robot can explore. In fact, if the robots are battery-powered, it is physically not possible to continuously explore on a single battery charge. Our paper tackles this challenge, integrating ergodic search methods with energy-aware coverage. We trade off battery usage and coverage quality, maintaining uninterrupted exploration by at least one agent. Our approach derives an abstract battery model for future state-of-charge estimation and extends canonical ergodic search to ergodic search under battery constraints. Empirical data from simulations and real-world experiments demonstrate the effectiveness of our energy-aware ergodic search, which ensures continuous exploration and guarantees spatial coverage.

This systems paper presents the implementation and design of RB5, a wheeled robot for autonomous long-term exploration with fewer and cheaper sensors. Requiring just an RGB-D camera and low-power computing hardware, the system consists of an experimental platform with rocker-bogie suspension. It operates in unknown and GPS-denied environments and on indoor and outdoor terrains. The exploration consists of a methodology that extends frontier- and sampling-based exploration with a path-following vector field and a state-of-the-art SLAM algorithm. The methodology allows the robot to explore its surroundings at lower update frequencies, enabling the use of lower-performing and lower-cost hardware while still retaining good autonomous performance. The approach further consists of a methodology to interact with a remotely located human operator based on an inexpensive long-range and low-power communication technology from the internet-of-things domain (i.e., LoRa) and a customized communication protocol. The results and the feasibility analysis show the possible applications and limitations of the approach.Code—The open-source software stack is made available on the project repository webpage<sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">†</sup>.

Joint absence in people with upper-limb-difference leads to compensatory motions. Such compensation has long been a topic of study, but typically only for a single object/user layout, which is unlikely to spatially generalize. We seek to understand how motion varies over a planar workspace for different target orientations and wrist mobility conditions. We therefore present a study that records arm and torso pose during grasping of 49 equally spaced cylindrical targets. Furthermore, we seek to validate the research practice of using wrist-immobilizing bypass sockets on able-bodied participants to simulate prostheses without wrists. Participants were 2 transradial amputees and 7 able-bodied individuals who conducted the study with and without wrist braces, generating 2450 trajectories. Heat-maps illustrate variation over the workspace in Mean Joint Angle, Range of Joint Motion and Distance Travelled by Body Segment. Results indicate that greater wrist restriction primarily exacerbated shoulder internal rotation and elbow flexion, not the trunk. We observed that bypass sockets do not fully simulate amputee behavior. Furthermore, amputee reaching with their intact limb is different to the reaching motion of normative participants, implying that transradial limb-difference affects both sides of the body. Differences in participant behavior were also observed between horizontal and vertical target orientations.

Calibrating robots into their workspaces is crucial for manipulation tasks. Existing calibration techniques often rely on sensors external to the robot (cameras, laser scanners, etc.) or specialized tools. This reliance complicates the calibration process and increases the costs and time requirements. Furthermore, the associated setup and measurement procedures require significant human intervention, which makes them more challenging to operate. Using the built-in force-torque sensors, which are nowadays a default component in collaborative robots, this work proposes a self-calibration framework where robot-environmental spatial relations are automatically estimated through compliant exploratory actions by the robot itself. The self-calibration approach converges, verifies its own accuracy, and terminates upon completion, autonomously purely through interactive exploration of the environment's geometries. Extensive experiments validate the effectiveness of our self-calibration approach in accurately establishing the robot-environment spatial relationships without the need for additional sensing equipment or any human intervention.

This systems paper presents the implementation and design of RB5, a wheeled robot for autonomous long-term exploration with fewer and cheaper sensors. Requiring just an RGB-D camera and low-power computing hardware, the system consists of an experimental platform with rocker-bogie suspension. It operates in unknown and GPS-denied environments and on indoor and outdoor terrains. The exploration consists of a methodology that extends frontier- and sampling-based exploration with a path-following vector field and a state-of-the-art SLAM algorithm. The methodology allows the robot to explore its surroundings at lower update frequencies, enabling the use of lower-performing and lower-cost hardware while still retaining good autonomous performance. The approach further consists of a methodology to interact with a remotely located human operator based on an inexpensive long-range and low-power communication technology from the internet-of-things domain (i.e., LoRa) and a customized communication protocol. The results and the feasibility analysis show the possible applications and limitations of the approach.

Humans regularly use all inner surfaces of the hand during manipulation, whereas traditional formulations for robots tend to use only the tips of their fingers, limiting overall dexterity. In this paper, we explore the use of the whole hand during spatial robotic dexterous within-hand manipulation. We present a novel four-fingered robotic hand called the Model B, which is designed and controlled using a straight-forward potential energy-based motion model that is based on the hand configuration and applied actuator torques. In this way the hand-object system is driven to a new desired configuration, often through sliding and rolling between the object and hand, and with the fingers "caging" the object to prevent ejection. This paper presents the first ever application of the energy model in three dimensions, which was used to compare the theoretical manipulability of popular robotic hands, which then inspired the design of the Model B. We experimentally validate the hand's performance with extensive benchtop experimentation with test objects and real world objects, as well as on a robotic arm, and demonstrate complex spatial caging manipulation on a variety of objects in all six object dimensions (three translation and three rotation) using all inner surfaces of the fingers and the palm.

Abstract Due to the nature of their implementation, nearly all low-level fabrication processes produce solidly filled structures. However, lattice structures are significantly stronger for the same amount of material, resulting in structures that are much lighter and more materially efficient. In this paper we propose an approach for fabricating lattice structures that echoes commercially successful additive manufacturing/3D printing techniques. In it, a modular chain of specially designed links is “extruded” onto a substrate to produce various lattice configurations depending on the chosen assembly algorithm, ranging from one of the strongest known rigid lattices, the octet-truss, to significantly less dense configurations. Like 3D printing, the process allows material to be compactly stored before being deployed into structures of nearly arbitrary geometry, but unlike it, the approach allows for the use of nearly any material or combination of materials. We demonstrate the concept experimentally with a nearly 300-link chain that is fed from a spool and autonomously laid down onto programmed lattice arrangements via a robotic arm modified for the task. In the current prototype implementation, ~300-link structures are fabricated in 27 minutes, and a 3x3x2 lattice structure (287 total links) is shown to support approximately 1000N in compression testing.

Continuous exploration without interruption is important in scenarios such as search and rescue and precision agriculture, where consistent presence is needed to detect events over large areas. Ergodic search already derives continuous trajectories in these scenarios so that a robot spends more time in areas with high information density. However, existing literature on ergodic search does not consider the robot's energy constraints, limiting how long a robot can explore. In fact, if the robots are battery-powered, it is physically not possible to continuously explore on a single battery charge. Our paper tackles this challenge, integrating ergodic search methods with energy-aware coverage. We trade off battery usage and coverage quality, maintaining uninterrupted exploration by at least one agent. Our approach derives an abstract battery model for future state-of-charge estimation and extends canonical ergodic search to ergodic search under battery constraints. Empirical data from simulations and real-world experiments demonstrate the effectiveness of our energy-aware ergodic search, which ensures continuous exploration and guarantees spatial coverage.

Building hand-object models for dexterous in-hand manipulation remains a crucial and open problem. Major challenges include the difficulty of obtaining the geometric and dynamical models of the hand, object, and time-varying contacts, as well as the inevitable physical and perception uncertainties. Instead of building accurate models to map between the actuation inputs and the object motions, this work proposes to enable the hand-object systems to continuously approximate their local models via a self-identification process where an underlying manipulation model is estimated through a small number of exploratory actions and non-parametric learning. With a very small number of data points, as opposed to most data-driven methods, our system self-identifies the underlying manipulation models online through exploratory actions and non-parametric learning. By integrating the self-identified hand-object model into a model predictive control framework, the proposed system closes the control loop to provide high accuracy in-hand manipulation. Furthermore, the proposed self-identification is able to adaptively trigger online updates through additional exploratory actions, as soon as the self-identified local models render large discrepancies against the observed manipulation outcomes. We implemented the proposed approach on a sensorless underactuated Yale Model O hand with a single external camera to observe the object's motion. With extensive experiments, we show that the proposed self-identification approach can enable accurate and robust dexterous manipulation without requiring an accurate system model nor a large amount of data for offline training.

Building hand-object models for dexterous in-hand manipulation remains a crucial and open problem. Major challenges include the difficulty of obtaining the geometric and dynamical models of the hand, object, and time-varying contacts, as well as the inevitable physical and perception uncertainties. Instead of building accurate models to map between the actuation inputs and the object motions, this work proposes to enable the hand-object systems to continuously approximate their local models via a self-identification process where an underlying manipulation model is estimated through a small number of exploratory actions and non-parametric learning. With a very small number of data points, as opposed to most data-driven methods, our system self-identifies the underlying manipulation models online through exploratory actions and non-parametric learning. By integrating the self-identified hand-object model into a model predictive control framework, the proposed system closes the control loop to provide high accuracy in-hand manipulation. Furthermore, the proposed self-identification is able to adaptively trigger online updates through additional exploratory actions, as soon as the self-identified local models render large discrepancies against the observed manipulation outcomes. We implemented the proposed approach on a sensorless underactuated Yale Model O hand with a single external camera to observe the object's motion. With extensive experiments, we show that the proposed self-identification approach can enable accurate and robust dexterous manipulation without requiring an accurate system model nor a large amount of data for offline training.

Contact can be conceptualized as a set of constraints imposed on two bodies that are interacting with one another in some way. The nature of a contact, whether a point, line, or surface, dictates how these bodies are able to move with respect to one another given a force, and a set of contacts can provide either partial or full constraint on a body's motion. Decades of work have explored how to explicitly estimate the location of a contact and its dynamics, e.g., frictional properties, but investigated methods have been computationally expensive and there often exists significant uncertainty in the final calculation. This has affected further advancements in contact-rich tasks that are seemingly simple to humans, such as generalized peg-in-hole insertions. In this work, instead of explicitly estimating the individual contact dynamics between an object and its hole, we approach this problem by investigating compliance-enabled contact formations. More formally, contact formations are defined according to the constraints imposed on an object's available degrees-of-freedom. Rather than estimating individual contact positions, we abstract out this calculation to an implicit representation, allowing the robot to either acquire, maintain, or release constraints on the object during the insertion process, by monitoring forces enacted on the end effector through time. Using a compliant robot, our method is desirable in that we are able to complete industry-relevant insertion tasks of tolerances <0.25mm without prior knowledge of the exact hole location or its orientation. We showcase our method on more generalized insertion tasks, such as commercially available non-cylindrical objects and open world plug tasks.

Robot manipulation today generally focuses on motions exclusively with a robot arm or a dexterous hand, but usually not a combination of both. However, complex manipulation tasks can require coordinating arm and hand motions that leverage capabilities of both, much like the coordinated arm and hand motions carried out by humans to perform everyday tasks. In this work, we evaluate unified manipulation with robot arms and dexterous hands, using a motion optimization framework that synthesizes a series of configuration states over the entire manipulation system. We characterize the possible benefits of unifying arm and dexterous hand capabilities within a single model via metrics such as pose accuracy, manipulability, joint-space smoothness, distance to joint-limits, distance to collisions, and more. Several arm-hand combinations are quantitatively compared in simulation on a variety of experiment tasks and performance measures. Our results suggest that combining motions from robot arms and dexterous hands indeed has compelling benefits, highlighting the exciting potential of continued progress in unified arm-hand motion synthesis for robotics applications.

Contact can be conceptualized as a set of constraints imposed on two bodies that are interacting with one another in some way. The nature of a contact, whether a point, line, or surface, dictates how these bodies are able to move with respect to one another given a force, and a set of contacts can provide either partial or full constraint on a body's motion. Decades of work have explored how to explicitly estimate the location of a contact and its dynamics, e.g., frictional properties, but investigated methods have been computationally expensive and there often exists significant uncertainty in the final calculation. This has affected further advancements in contact-rich tasks that are seemingly simple to humans, such as generalized peg-in-hole insertions. In this work, instead of explicitly estimating the individual contact dynamics between an object and its hole, we approach this problem by investigating compliance-enabled contact formations. More formally, contact formations are defined according to the constraints imposed on an object's available degrees-of-freedom. Rather than estimating individual contact positions, we abstract out this calculation to an implicit representation, allowing the robot to either acquire, maintain, or release constraints on the object during the insertion process, by monitoring forces enacted on the end effector through time. Using a compliant robot, our method is desirable in that we are able to complete industry-relevant insertion tasks of tolerances &lt;0.25mm without prior knowledge of the exact hole location or its orientation. We showcase our method on more generalized insertion tasks, such as commercially available non-cylindrical objects and open world plug tasks.

Recent rapid progress in HRI research makes it more crucial than ever to have systematic development and benchmarking methodologies to assess and compare different algorithms and strategies. Indeed, the lack of such methodologies results in inefficiencies and sometimes stagnation, since new methods cannot be effectively compared to prior work and the research gaps become challenging to identify. Moreover, lacking an active and effective mechanism to disseminate and utilize the available datasets and benchmarking protocols significantly reduces their impact and utility. A unified effort in the development, utilization, and dissemination of open-source assets amongst a governed community of users can advance these domains substantially; for HRI, this is particularly needed in the curation and generation of datasets for benchmarking. This workshop will take a step towards removing the roadblocks to the development and assessment of HRI by reviewing, discussing, and laying the groundwork for an open-source ecosystem at the intersection of HRI and robot manipulation. The workshop will play a crucial role for identifying the preconditions and requirements to develop an open-source ecosystem that provides open-source assets for HRI benchmarking and comparison, aiming to determine the needs and wants of HRI researchers. Invited speakers include those who have contributed to the development of open-source assets in HRI and robot manipulation and discussion topics will include issues related to the usage of open-source assets and the benefits of forming of an open-source ecosystem.