Hadas Kress‐Gazit

Robot manipulation has seen tremendous progress in recent years, with imitation learning policies enabling successful performance of dexterous and hard-to-model tasks. Concurrently, scaling data and model size has led to the development of capable language and vision foundation models, motivating large-scale efforts to create general-purpose robot foundation models. Although these models have garnered considerable enthusiasm and investment, meaningful evaluation of real-world performance remains a challenge, limiting the pace of development and inhibiting a nuanced understanding of current capabilities. Here, we rigorously evaluated multitask robot manipulation policies, referred to as large behavior models, by extending the diffusion policy paradigm across a corpus of simulated and real-world robot data. We proposed and validated an evaluation pipeline to rigorously analyze the capabilities of these models with statistical confidence. We compared against single-task baselines through blind, randomized trials in a controlled setting, using both simulation and real-world experiments. We found that multitask pretraining made the policies more successful and robust and enabled teaching complex new tasks more quickly, using a fraction of the data when compared with single-task baselines. Moreover, performance predictably increased as pretraining scale and diversity grows.

Emergency departments (EDs) are fast-paced, dynamic, safety-critical spaces where clinicians are overworked and underpaid. To support clinicians, researchers are exploring the contextualization and development of clinically assistive robots (CARs) that can assume non-critical tasks to reduce clinician overload. In this article, we introduce Clinically Assistive Robot System for Emergency Medicine (CAR-EM), collaboratively developed with ED clinicians. CAR-EM includes an autonomous robot and a task specification interface. It completes tasks by leveraging control synthesis, a framework that automatically transforms high-level tasks into control while providing guarantees and feedback. We conducted a feasibility study across two different hospital EDs, where interprofessional clinicians tasked the robot to perform patient assessments and item deliveries. Clinicians found the system easy to use, and particularly helpful to offload busywork. This work demonstrates control synthesis as a feasible tool to develop autonomy for robots in safety-critical spaces, and identifies considerations for failure interventions. We also discuss ethical considerations for deploying robots in hospitals, including healthcare worker displacement and work disruption. Thus, our work: (1) highlights the unique requirements of situating robots in real world hospital EDs, and (2) demonstrates a novel approach leveraging guarantees and feedback from control synthesis methods to successfully implement context-specific CAR behaviors. Through this work, we aim to further research for safer and more reliable robots in real world, uncertain environments.

We propose an integrated planning framework for quadrupedal locomotion over dynamically changing, unforeseen terrains. Existing approaches either rely on heuristics for instantaneous foothold selection–compromising safety and versatility–or solve expensive trajectory optimization problems with complex terrain features and long time horizons. In contrast, our framework leverages reactive synthesis to generate correct-by-construction controllers at the symbolic level, and mixed-integer convex programming (MICP) for dynamic and physically feasible footstep planning for each symbolic transition. We use a high-level manager to reduce the large state space in synthesis by incorporating local environment information, improving synthesis scalability. To handle specifications that cannot be met due to dynamic infeasibility, and to minimize costly MICP solves, we leverage a symbolic repair process to generate only necessary symbolic transitions. During online execution, re-running the MICP with real-world terrain data, along with runtime symbolic repair, bridges the gap between offline synthesis and online execution. We demonstrate, in simulation, our framework’s capabilities to discover missing locomotion skills and react promptly in safety-critical environments, such as scattered stepping stones and rebars.

In this work, we examine robots "asking for help" in unpredictable human spaces. We focus on an open question particularly relevant for robots deployed in public–"how do people help robots?" We present an elicitation study that shows how asking for help in a real-world field study yields valuable and sometimes unexpected information. From our study, we examine strangers’ responses toward a robot asking for spatial directions and extract valuable themes that can inform future asking-for-help systems. Our analysis provides a wide range of information, from geometric and topological information in natural language to details about rejection during an interaction. Further, we also provide anecdotes of valuable outlier behavior that can only be captured through a study in a real public space. Through our work, we highlight the importance of in-the-wild studies and discuss how the rich information they contribute will help robots effectively ask for help.

This paper presents INPROVF, an automatic framework that combines large language models (LLMs) and formal methods to speed up the repair process of high-level robot controllers. Previous approaches based solely on formal methods are computationally expensive and cannot scale to large state spaces. In contrast, INPROVF uses LLMs to generate repair candidates, and formal methods to verify their correctness. To improve the quality of these candidates, our framework first translates the symbolic representations of the environment and controllers into natural language descriptions. If a candidate fails the verification, INPROVF provides feedback on potential unsafe behaviors or unsatisfied tasks, and iteratively prompts LLMs to generate improved solutions. We demonstrate the effectiveness of INPROVF through 12 violations with various workspaces, tasks, and state space sizes.

Designing modular manipulators that can continue task execution despite joint failures remains a significant challenge in building reliable robotic systems. Due to the complexity of inverse kinematics calculations and the intractability of analyzing all failure conditions in continuous spaces, previous efforts have focused on constraining initial conditions or estimating generalized failure-tolerant regions in the workspace to limit the scope of the problem for analytical methods. In this work, we introduce a novel approach that distills the complex relationship between physical design and reachability into a neural network. This representation enables gradient-based optimization to efficiently generate designs tailored to specific task requirements, that are robust to arbitrary locked-joint failures. Simulation experiments demonstrate that our method produces manipulator designs with a higher task success rate under random failure conditions compared to baseline methods. Our code implementation can be found here.

Robot manipulation has seen tremendous progress in recent years, with imitation learning policies enabling successful performance of dexterous and hard-to-model tasks. Concurrently, scaling data and model size has led to the development of capable language and vision foundation models, motivating large-scale efforts to create general-purpose robot foundation models. While these models have garnered significant enthusiasm and investment, meaningful evaluation of real-world performance remains a challenge, limiting both the pace of development and inhibiting a nuanced understanding of current capabilities. In this paper, we rigorously evaluate multitask robot manipulation policies, referred to as Large Behavior Models (LBMs), by extending the Diffusion Policy paradigm across a corpus of simulated and real-world robot data. We propose and validate an evaluation pipeline to rigorously analyze the capabilities of these models with statistical confidence. We compare against single-task baselines through blind, randomized trials in a controlled setting, using both simulation and real-world experiments. We find that multi-task pretraining makes the policies more successful and robust, and enables teaching complex new tasks more quickly, using a fraction of the data when compared to single-task baselines. Moreover, performance predictably increases as pretraining scale and diversity grows. Project page: https://toyotaresearchinstitute.github.io/lbm1/

Abstract In this article, we present a framework that automatically selects a modular manipulator structure based on a user-defined task involving 3D trajectory tracking, handling heavy payloads with precision, and navigating around obstacles. For such tasks, hybrid structures combining serial and parallel components may offer advantages, leveraging the large reachable space of serial elements and the rigidity of parallel components for enhanced accuracy. Given the challenges in developing a comprehensive method for kinematic design and dynamic analysis of hybrid robots, existing works often constrain structures to 2D space or optimize predefined initial designs. Our approach explores both serial and hybrid structures, incorporating modular components such as open and closed loops. By widening the solution space, we uncover designs for tasks previously considered unfeasible due to structural constraints.We formulate task-to-manipulator matching as a constrained optimization problem in kinematics and statics, yielding both design and control solutions. To validate our approach, we demonstrate its feasibility through the physical implementation of manipulators capable of performing complex tasks.

We propose an integrated planning framework for quadrupedal locomotion over dynamically changing, unforeseen terrains. Existing approaches either rely on heuristics for instantaneous foothold selection--compromising safety and versatility--or solve expensive trajectory optimization problems with complex terrain features and long time horizons. In contrast, our framework leverages reactive synthesis to generate correct-by-construction controllers at the symbolic level, and mixed-integer convex programming (MICP) for dynamic and physically feasible footstep planning for each symbolic transition. We use a high-level manager to reduce the large state space in synthesis by incorporating local environment information, improving synthesis scalability. To handle specifications that cannot be met due to dynamic infeasibility, and to minimize costly MICP solves, we leverage a symbolic repair process to generate only necessary symbolic transitions. During online execution, re-running the MICP with real-world terrain data, along with runtime symbolic repair, bridges the gap between offline synthesis and online execution. We demonstrate, in simulation, our framework's capabilities to discover missing locomotion skills and react promptly in safety-critical environments, such as scattered stepping stones and rebars.

Given a collaborative high-level task and a team of heterogeneous robots with behaviors to satisfy it, this work focuses on the challenge of automatically adjusting the individual robot behaviors at runtime such that the task is still satisfied. We specifically address scenarios when robots encounter changes to their abilities–either failures or additional actions they can perform. We aim to minimize global teaming reassignments (and as a result, local resynthesis) when robots' capabilities change. The tasks are encoded in LTL <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$^\psi$</tex-math></inline-formula> , an extension of LTL introduced in our prior work. We increase the expressivity of LTL <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$^\psi$</tex-math></inline-formula> by including additional types of constraints on the overall teaming assignment that the user can specify, such as the minimum number of robots required for each assignment. We demonstrate the framework in a simulated warehouse scenario.

The robot learning community has made great strides in recent years, proposing new architectures and showcasing impressive new capabilities; however, the dominant metric used in the literature, especially for physical experiments, is "success rate", i.e. the percentage of runs that were successful. Furthermore, it is common for papers to report this number with little to no information regarding the number of runs, the initial conditions, and the success criteria, little to no narrative description of the behaviors and failures observed, and little to no statistical analysis of the findings. In this paper we argue that to move the field forward, researchers should provide a nuanced evaluation of their methods, especially when evaluating and comparing learned policies on physical robots. To do so, we propose best practices for future evaluations: explicitly reporting the experimental conditions, evaluating several metrics designed to complement success rate, conducting statistical analysis, and adding a qualitative description of failures modes. We illustrate these through an evaluation on physical robots of several learned policies for manipulation tasks.

Given a collaborative high-level task and a team of heterogeneous robots and behaviors to satisfy it, this work focuses on the challenge of automatically, at runtime, adjusting the individual robot behaviors such that the task is still satisfied, when robots encounter changes to their abilities--either failures or additional actions they can perform. We consider tasks encoded in LTL^ψand minimize global teaming reassignments (and as a result, local resynthesis) when robots' capabilities change. We also increase the expressivity of LTL^ψby including additional types of constraints on the overall teaming assignment that the user can specify, such as the minimum number of robots required for each assignment. We demonstrate the framework in a simulated warehouse scenario.

This paper presents a framework that enables robots to automatically recover from assumption violations of high-level specifications during task execution. In contrast to previous methods relying on user intervention to impose additional assumptions for failure recovery, our approach leverages synthesis-based repair to suggest new robot skills that, when implemented, repair the task. Our approach detects violations of environment safety assumptions during the task execution, relaxes the assumptions to admit observed environment behaviors, and acquires new robot skills for task completion. We demonstrate our approach with a Hello Robot Stretch in a factory-like scenario.

This paper presents a framework that enables robots to automatically recover from assumption violations of high-level specifications during task execution. In contrast to previous methods relying on user intervention to impose additional assumptions for failure recovery, our approach leverages synthesis-based repair to suggest new robot skills that, when implemented, repair the task. Our approach detects violations of environment safety assumptions during the task execution, relaxes the assumptions to admit observed environment behaviors, and acquires new robot skills for task completion. We demonstrate our approach with a Hello Robot Stretch in a factory-like scenario.

We propose a control synthesis framework for a heterogeneous multi-robot system to satisfy collaborative tasks, where actions may take varying duration of time to complete. We encode tasks using the discrete logic LTL^ψ, which uses the concept of bindings to interleave robot actions and express information about relationship between specific task requirements and robot assignments. We present a synthesis approach to automatically generate a teaming assignment and corresponding discrete behavior that is correct-by-construction for continuous execution, while also implementing synchronization policies to ensure collaborative portions of the task are satisfied. We demonstrate our approach on a physical multi-robot system.

We propose a new multi-agent task grammar to encode collaborative tasks for a team of heterogeneous agents that can have overlapping capabilities. The grammar allows users to specify the relationship between agents and parts of the task without providing explicit assignments or constraints on the number of agents required. We develop a method to automatically find a team of agents and synthesize correct-by-construction control with synchronization policies to satisfy the task. We demonstrate the scalability of our approach through simulation and compare our method to existing task grammars that encode multi-agent tasks.

We present a decentralized control algorithm for a minimalist robotic swarm lacking memory, explicit communication, or relative position information, to encapsulate multiple diffusive target sources in a bounded environment. The state-of-the-art approaches generally require either local communication or relative localization to provide guarantees of convergence and safety. We quantify trade-offs between task, control, and robot parameters for guaranteed safe convergence to all the sources. Furthermore, our algorithm is robust to occlusions and noise in the sensor measurements as we demonstrate in simulation.

We propose a new multi-agent task grammar to encode collaborative tasks for a team of heterogeneous agents that can have overlapping capabilities. The grammar allows users to specify the relationship between agents and parts of the task without providing explicit assignments or constraints on the number of agents required. We develop a method to automatically find a team of agents and synthesize correct-by-construction control with synchronization policies to satisfy the task. We demonstrate the scalability of our approach through simulation and compare our method to existing task grammars that encode multi-agent tasks.

In this paper we present a grammar and control synthesis framework for online modification of Event-based Signal Temporal Logic (STL) specifications, during execution. These modifications allow a user to change the robots' task in response to potential future violations, changes to the environment, or user-defined task changes. In cases where a modification is not possible, we provide feedback to the user and suggest alternative modifications. We demonstrate our task modification process using a Hello Robot Stretch.

We present a decentralized control algorithm for a robotic swarm given the task of encapsulating static and moving targets in a bounded unknown environment. We consider minimalist robots without memory, explicit communication, or localization information. The state-of-the-art approaches generally assume that the robots in the swarm are able to detect the relative position of neighboring robots and targets in order to provide convergence guarantees. In this work, we propose a novel control law for the guaranteed encapsulation of static and moving targets while avoiding all collisions, when the robots do not know the exact relative location of any robot or target in the environment. We make use of the Lyapunov stability theory to prove the convergence of our control algorithm and provide bounds on the ratio between the target and robot speeds. Furthermore, our proposed approach is able to provide stochastic guarantees under the bounds that we determine on task parameters for scenarios where a target moves faster than a robot. Finally, we present an analysis of how the emergent behavior changes with different parameters of the task and noisy sensor readings.

A typical approach to creating complex robot behaviors is to compose atomic controllers, or skills, such that the resulting behavior satisfies a high-level task; however, when a task cannot be accomplished with a given set of skills, it is difficult to know how to modify the skills to make the task possible. We present a method for combining symbolic repair with physical feasibility checking and implementation to automatically modify existing skills such that the robot can execute a previously infeasible task. We encode robot skills in linear temporal logic (LTL) formulas that capture both safety constraints and goals for reactive tasks. Furthermore, our encoding captures the full skill execution, as opposed to prior work where only the state of the world before and after the skill is executed are considered. Our repair algorithm suggests symbolic modifications, then attempts to physically implement the suggestions by modifying the original skills subject to Linear Temporal Logic (LTL) constraints derived from the symbolic repair. If skills are not physically possible, we automatically provide additional constraints for the symbolic repair. We demonstrate our approach with a Baxter and a Clearpath Jackal.

Integrated task and motion planning (TMP) offers a promising class of approaches for solving robot planning problems with intricate symbolic and geometric constraints. However, TMP planners rely on difficult-to-construct abstract models of robot actions. In this article, we propose a method for automatically constructing and continuously improving an abstraction of robot actions via observations of the robot performing the actions. This method, called <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">automatic abstraction repair</i> , allows action abstractions to be initially incorrect or incomplete and converge toward a correct model over time. Here, we demonstrate abstraction repair using constrained polynomial zonotopes (CPZs), an expressive nonconvex set representation for modeling predicates over joint symbolic and geometric state. The repair process performs a hybrid optimizing search over symbolic edit operations to predicate formulae and continuous predicate parameters to improve the grounding of the abstraction to the behavior of a physical robot. In this work, we describe the predicate model, introduce the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">symbolic-geometric abstraction repair</i> problem, and present an anytime algorithm for automatic abstraction repair. We demonstrate that abstraction repair can improve realistic action abstractions for common mobile manipulation actions from a handful of observations and discuss the tradeoffs of the CPZ model for predicate representation.

We present a method for calculating the probability that a robot successfully performs a task described using Signal Temporal Logic (STL). We focus on cases where the failure probability is very small, hence a traditional Monte-Carlo method becomes inefficient due to the large number of samples required to observe failures. Using elliptical sliced sampling, normalizing flows, and Bayesian optimization, we develop an algorithm that, under mild assumptions, is applicable to black-box systems, and can be applied to uncertainty sources with non-Gaussian probabilities. We demonstrate the application of our method on three different simulated robots.

Providing guarantees on the safe operation of robots against edge cases is challenging as testing methods such as traditional Monte-Carlo require too many samples to provide reasonable statistics. Built upon recent advancements in rare-event sampling, we present a model-based method to verify if a robotic system satisfies a Signal Temporal Logic (STL) specification in the face of environment variations and sensor/actuator noises. Our method is efficient and applicable to both linear and nonlinear and even black-box systems with arbitrary, but known, uncertainty distributions. For linear systems with Gaussian uncertainties, we exploit a feature to find optimal parameters that minimize the probability of failure. We demonstrate illustrative examples on applying our approach to real-world autonomous robotic systems.

We present a framework for the automatic encoding and repair of high-level tasks. Given a set of skills a robot can perform, our approach first abstracts sensor data into symbols and then automatically encodes the robot’s capabilities in Linear Temporal Logic (LTL). Using this encoding, a user can specify reactive high-level tasks, for which we can automatically synthesize a strategy that executes on the robot, if the task is feasible. If a task is not feasible given the robot’s capabilities, we present two methods, one enumeration-based and one synthesis-based, for automatically suggesting additional skills for the robot or modifications to existing skills that would make the task feasible. We demonstrate our framework on a Baxter robot manipulating blocks on a table, a Baxter robot manipulating plates on a table, and a Kinova arm manipulating vials, with multiple sensor modalities, including raw images.

In this paper we present a grammar and control synthesis framework for online modification of Event-based Signal Temporal Logic (STL) specifications, during execution. These modifications allow a user to change the robots' task in response to potential future violations, changes to the environment, or user-defined task design changes. In cases where a modification is not possible, we provide feedback to the user and suggest alternative modifications. We demonstrate our task modification process using a Hello Robot Stretch satisfying an Event-based STL specification.

Robots have the potential to assist in emergency evacuation tasks, but it is not clear how robots should behave to evacuate people who are not fully compliant, perhaps due to panic or other priorities in an emergency. In this paper, we compare two robot strategies: an actively nudging robot that initiates evacuation and pulls toward the exit and a passively waiting robot that stays around users and waits for instruction. Both strategies were automatically synthesized from a description of the desired behavior. We conduct a within participant study ( = 20) in a simulated environment to compare the evacuation effectiveness between the two robot strategies. Our results indicate an advantage of the nudging robot for effective evacuation when being exposed to the evacuation scenario for the first time. The waiting robot results in lower efficiency, higher mental load, and more physical conflicts. However, participants like the waiting robots equally or slightly more when they repeat the evacuation scenario and are more familiar with the situation. Our qualitative analysis of the participants' feedback suggests several design implications for future emergency evacuation robots.

Robots operating in human spaces need to be able to communicate with people. Understanding how humans and robots communicate about the shared space around them allows us to build robots that can interact fluidly with others. We performed a field study with a telepresence robot and a perceived autonomous robot to explore how humans give directions to robots and how the interactions differ based on the perceived identity of the robot operator. In this work we present some initial findings from our in-the-wild study including: 1) participants were more considerate to the robot in the telepresence condition, 2) participants considered the sensing and physical limitations of the robot when giving directions, and 3) participants were uncertain about the realness or identity of the robot and the robot operator.

We present a decentralized control algorithm for a robotic swarm given the task of encapsulating static and moving targets in a bounded unknown environment. We consider minimalist robots without memory, explicit communication, or localization information. The state-of-the-art approaches generally assume that the robots in the swarm are able to detect the relative position of neighboring robots and targets in order to provide convergence guarantees. In this work, we propose a novel control law for the guaranteed encapsulation of static and moving targets while avoiding all collisions, when the robots do not know the exact relative location of any robot or target in the environment. We make use of the Lyapunov stability theory to prove the convergence of our control algorithm and provide bounds on the ratio between the target and robot speeds. Furthermore, our proposed approach is able to provide stochastic guarantees under the bounds that we determine on task parameters for scenarios where a target moves faster than a robot. Finally, we present an analysis of how the emergent behavior changes with different parameters of the task and noisy sensor readings.