Satyandra K. Gupta

Abstract Legged robots can significantly increase human productivity by performing delivery tasks, especially in unstructured agricultural fields. In large outdoor environments, legged robots typically operate independently from tethered power sources, relying on onboard batteries. If a robot runs out of energy while executing a task, it will require human intervention, resulting in delays. On the other hand, frequent battery recharging or replacement could prolong task completion times. This article presents a systematic framework to enhance productivity for logistic tasks. The framework features a map construction utility, an energy consumption model to measure battery usage, and an energy-aware hierarchical planning approach that accounts for energy consumption and integrates appropriate battery replacement strategies to ensure that tasks are completed efficiently. Our algorithm first generates different scenarios, considering battery replacement options, payload partitioning, and speed reduction strategies. Subsequently, it employs graph search methods to identify the optimal plan that minimizes delivery completion time. We illustrate the effectiveness of our planning approach on a terrain with varying slopes and delivery tasks with different requirements. We also demonstrated that our robot can successfully traverse narrow furrows in broccoli and cabbage farms.

The evolution of artificial intelligence (AI) and machine learning (ML) is reshaping smart manufacturing by providing new capabilities for efficiency, adaptability, and autonomy across industrial value chains. However, the deployment of AI and ML in industrial settings still faces critical challenges, including the complexity of industrial big data, effective data management, integration with heterogeneous sensing and control systems, and the demand for trustworthy, explainable, and reliable operation in high-stakes industrial environments. In this roadmap, we present a comprehensive perspective on the foundations, applications, and emerging directions of AI and ML in smart manufacturing. It is structured in three parts. The first highlights the foundations and trends that frame the evolution of AI in smart manufacturing. The second focuses on key topics where AI is already enabling advances, including industrial big data analytics, advanced sensing and perception, autonomous systems, additive and laser-based manufacturing, digital twins, robotics, supply chain and logistics optimization, and sustainable manufacturing. The third section explores non-traditional ML approaches that are opening new frontiers, such as physics-informed AI, generative AI, semantic AI, advanced digital twins, explainable AI, RAMS, data-centric metrology, LLMs, and foundation models for highly connected and complex manufacturing systems. By identifying both opportunities and remaining barriers across these areas, this roadmap outlines the advances needed in methods, integration strategies, and industrial adoption. We hope this roadmap will serve as a guide for researchers, engineers, and practitioners to accelerate innovation, align academic and industrial priorities, and ensure that AI-driven smart manufacturing delivers reliable, sustainable, and scalable impact for the future of manufacturing ecosystems.

Robotic bin packing is widely deployed in warehouse automation, with current systems achieving robust performance through heuristic and learning-based strategies. These systems must balance compact placement with rapid execution, where selecting alternative items or reorienting them can improve space utilization but introduce additional time. We propose a selection-based formulation that explicitly reasons over this trade-off: at each step, the robot evaluates multiple candidate actions, weighing expected packing benefit against estimated operational time. This enables time-aware strategies that selectively accept increased operational time when it yields meaningful spatial improvements. Our method, STEP (Space-Time Efficient Packing), uses a preference-conditioned, Transformer-based reinforcement learning policy, and allows generalization across candidate set sizes and integration with standard placement modules. It achieves a 44% reduction in operational time without compromising packing density. Additional material is available at https://step-packing.github.io.

Robotic bin packing is widely deployed in warehouse automation, with current systems achieving robust performance through heuristic and learning-based strategies. These systems must balance compact placement with rapid execution, where selecting alternative items or reorienting them can improve space utilization but introduce additional time. We propose a selection-based formulation that explicitly reasons over this trade-off: at each step, the robot evaluates multiple candidate actions, weighing expected packing benefit against estimated operational time. This enables time-aware strategies that selectively accept increased operational time when it yields meaningful spatial improvements. Our method, STEP (Space-Time Efficient Packing), uses a preference-conditioned, Transformer-based reinforcement learning policy, and allows generalization across candidate set sizes and integration with standard placement modules. It achieves a 44% reduction in operational time without compromising packing density. Additional material is available at https://step-packing.github.io.

Autonomous in-space assembly requires mechanisms that can tolerate multi-axis misalignment without relying on active impedance control or large alignment features. This paper presents the design, fabrication, and experimental validation of a compact passive Remote Center of Compliance (RCC) device tailored for the constraints of in-space robotic assembly. The RCC employs space-qualified elastomeric elements arranged in a configuration to provide passive translational and rotational compliance, enabling correction of misalignment while maintaining low structural and control complexity. Bench-top experiments using a 7-DOF robotic arm demonstrate that the RCC accommodates lateral, pitch, and roll misalignments of up to 9 mm, 10°, and 14°, respectively, while reducing insertion forces and torques by 34-89% compared to a rigid end-effector. Because passive mechanisms provide no internal state feedback, we further develop a learning-based method for estimating RCC deflection and for detecting proximity to mechanical limits using short windows of force–torque and relative-pose data. Static stiffness and compliance models are shown to be insufficient due to strong nonlinearities and hysteresis, motivating a temporal convolutional network which infers full six-degree-of-freedom deflection. The model achieves normalized mean absolute errors of 0.14-0.18 and reliably classifies limit proximity using thresholded normalized displacement labels. Confusion matrices show high neutrality recall and strong directional consistency, with cross-sign errors below 10%, enabling conservative, low-latency limit detection suitable for real-time operation. Together, the proposed RCC device and learning-based safety-limit estimator provide a low-mass, low-power, and sensing-lightweight solution for misalignment-tolerant robotic assembly in the resource-constrained conditions of space environments.

Abstract Physical artificial intelligence (AI) is driving the next revolution in robotics by grounding perception, action, and cognition within a robot’s physical structure. Unlike traditional systems that rely on disembodied reasoning and preprogrammed control, physical AI leverages sensorimotor coupling to enable real-time adaptation, experiential learning, and generalized task performance. Advances in machine learning, high-fidelity simulations, and multimodal sensing have accelerated progress toward real-world deployment. This position article articulates a unifying perspective on physical AI, outlining its conceptual evolution, defining system-level principles, and analyzing key functional subsystems, such as situational awareness, mapping, planning, control, and human–robot interaction. It provides a domain-wise readiness assessment across manufacturing, healthcare, logistics, agriculture, service robotics, and space exploration, highlighting opportunities and limitations. Finally, it identifies critical challenges—real-time performance, cybersecurity, benchmarking, safety, interpretability, and energy efficiency—and proposes codesign principles and evaluation frameworks to guide future research. By synthesizing these elements, the article positions physical AI as a foundational paradigm for trustworthy, adaptive, and mission-ready robotic systems, offering readers a roadmap for research priorities, cross-domain insights, and practical implications that will shape the next era of robotics.

Diffusion-based policy learning has shown strong performance across diverse robotic tasks, often achieving high success rates. However, real-world deployment requires more than task success—it demands efficient execution and the ability to handle complex environments. In many assembly and disassembly settings, a single scene contains multiple potential task goals. This can confuse learned policies, leading to ambiguous behavior. Enabling explicit task selection via natural language is thus crucial for robust and flexible operation. In this paper, we address two key challenges: (1) improving task execution efficiency by structuring tasks into distinct sub-task modes using language, and (2) resolving goal ambiguity by allowing human operators to specify desired tasks through natural language commands. We further introduce an adaptive parameter selection mechanism that adjusts reliance on different sensory modalities depending on the active sub-task. We evaluate our approach on the NIST Task Board, a representative benchmark with multiple co-located task goals. Our method improves execution speed by 57% and increases task success rate by 19% compared to baseline approaches. Demonstration videos are available at: https://rros-lab.github.io/task-aware-diffusion/.

Efficient and safe transport of deformable packages using suction cups is crucial in warehouse automation. Unlike rigid packages, deformable packages exhibit complex oscillatory behaviors and can detach under aggressive motions. Traditional motion planners typically overlook these oscillations, often resulting in either unsafe trajectories or overly conservative, slow motions. This paper addresses that gap by formulating package oscillation dynamics as constraints and incorporating them into a Cartesian trajectory optimization framework. These constraints are formulated to be state-dependent - i.e., they adapt according to the instantaneous conditions along the planned trajectory (such as acceleration and gripper orientation) - to ensure that oscillations remain within safe limits. We derive a pendulum-like model to characterize package swing, enforcing constraints on peak oscillation angles. Our approach then optimizes end-effector trajectories under these state-dependent constraints, ensuring safe transport when the end-effector follows constant-acceleration profiles. Real-world experiments demonstrate that our optimized trajectories reduce transport time by up to 18% compared to baseline motions while strictly adhering to safety limits on package swing.

Legged robots can significantly increase human productivity by performing delivery tasks. Legged robots cannot be tethered to power sources when operating in large outdoor environments. If a robot runs out of energy while executing a task, it will require human intervention, resulting in delays. On the other hand, frequent battery recharging or replacement could also lead to significant delays in task completion. This paper presents an energy-aware hierarchical planning approach that accounts for energy consumption and integrates appropriate battery replacement strategies to ensure that tasks are completed efficiently. Our algorithm generates graph search instances for varying battery replacement actions, the option to split the payload into smaller portions, and reducing speed in the first level, while running graph search to determine the optimal plan that minimizes the time to complete the delivery task. We illustrate the effectiveness of our planning approach on a terrain with varying slopes and delivery tasks with different requirements.

Collaborative loco-manipulation by multiple quadruped manipulators enables handling bulky, heavy objects beyond the capabilities of individual robots. However, coordinating robot teams while navigating complex terrains and avoiding obstacles remains challenging. We propose a hierarchical control framework consisting of a model predictive control (MPC)-based manipulation planner with integrated obstacle avoidance, a geometry-aware mapping converting object trajectories into robot commands, and decentralized loco-manipulation MPC controllers. The framework supports collision-free collaborative manipulation tasks and enhances payload capacities. Validation through simulation and real-world hardware experiments with diverse quadruped robot teams demonstrates the approach’s effectiveness, robustness, and practical applicability.

Robotic assembly of complex, non-convex geometries with tight clearances remains a challenging problem, demanding precise state estimation for successful insertion. In this work, we propose a novel framework that relies solely on contact states to estimate the full SE(3) pose of a peg relative to a hole. Our method constructs an online submanifold of contact states through primitive motions with just 6 seconds of online execution, subsequently mapping it to an offline contact manifold for precise pose estimation. We demonstrate that without such state estimation, robots risk jamming and excessive force application, potentially causing damage. We evaluate our approach on five industrially relevant, complex geometries with 0.1 to 1.0 mm clearances, achieving a 96.7% success rate-a 6× improvement over primitive-based insertion without state estimation. Additionally, we analyze insertion forces, and overall insertion times, showing our method significantly reduces the average wrench, enabling safer and more efficient assembly.

Abstract Deformable packages are becoming increasingly prevalent in the logistics and warehouse industry, demanding robotic manipulation strategies that are robust and adaptive. Unlike rigid objects, these packages undergo significant shape changes under external forces, making their handling more complex. These deformable packages are often manipulated using suction cups and contain internal objects that shift during transport, introducing additional complexity. To ensure safe and efficient robotic manipulation at scale, high-fidelity simulation is crucial to support automated robot trajectory generation. In this work, we present a physics-driven simulation framework that accurately models multi-object deformable packages, capturing package deformations, suction-cup interactions, and internal object dynamics. Our approach employs a hyperelastic model to accurately simulate package-object interactions, including contact forces, suction-cup interactions, friction, and material deformation. By leveraging high-fidelity physics, we optimize simulation parameters using real-world trajectory and package deformation data, ensuring an accurate representation of package behavior. Running in real-time, our framework reduces real-to-sim discrepancies, making it viable for real-world deployment. Additionally, we develop a parallelized simulation environment for large-scale reinforcement learning and trajectory optimization, paving the way for scalable, efficient, and adaptable robotic manipulation of deformable packages in diverse warehouse settings. Our code, dataset, and videos are available on our project website: https://sites.google.com/usc.edu/deformable-sim/

Assembly operations in manufacturing, especially those involving precise alignment and force control, pose significant challenges for automation. Tasks like fitting a battery cover onto a housing require careful manipulation to ensure proper alignment and insertion without causing damage. We propose leveraging imitation learning by collecting demonstrations through hand-guided manipulation, capturing both vision and force/torque data from sensors mounted on the robot’s end-effector. Although hand-guided manipulation may introduce minor imprecisions, our approach compensates by integrating high-fidelity force/torque sensing and run-time visual feedback, along with post-processing filters, to ensure the precision required for complex robotic assembly. These demonstrations are used to train a bimanual robotic system where one arm holds the battery housing securely while the other inserts the top cover. To enable this, we extend the diffusion policy framework by incorporating run-time force feedback and visual observations. Additionally, we introduce data segmentation and augmentation methods to reduce the number of required demonstrations, enhancing the policy’s robustness to task failures. Our findings demonstrate that our approach, despite being trained on a limited dataset, improves success rate and efficiency over conventional diffusion techniques. In addition, we present a case study in which our bimanual robotic system performs precise alignment and insertion of the battery cover, highlighting its potential for complex assembly tasks in manufacturing settings. However, the approach remains sensitive to sensor drift and has not yet been tested on highly deformable or ultra-tight-tolerance assemblies, highlighting opportunities for future improvement.

Robotic assembly of complex, non-convex geometries with tight clearances remains a challenging problem, demanding precise state estimation for successful insertion. In this work, we propose a novel framework that relies solely on contact states to estimate the full SE(3) pose of a peg relative to a hole. Our method constructs an online submanifold of contact states through primitive motions with just 6 seconds of online execution, subsequently mapping it to an offline contact manifold for precise pose estimation. We demonstrate that without such state estimation, robots risk jamming and excessive force application, potentially causing damage. We evaluate our approach on five industrially relevant, complex geometries with 0.1 to 1.0 mm clearances, achieving a 96.7% success rate - a 6x improvement over primitive-based insertion without state estimation. Additionally, we analyze insertion forces, and overall insertion times, showing our method significantly reduces the average wrench, enabling safer and more efficient assembly.

Contact-rich assembly of complex, non-convex parts with tight tolerances remains a formidable challenge. Purely model-based methods struggle with discontinuous contact dynamics, while model-free methods require vast data and often lack precision. In this work, we introduce a hybrid framework that uses only contact-state information between a complex peg and its mating hole to recover the full SE(3) pose during assembly. In under 10 seconds of online execution, a sequence of primitive probing motions constructs a local contact submanifold, which is then aligned to a precomputed offline contact manifold to yield sub-mm and sub-degree pose estimates. To eliminate costly k-NN searches, we train a lightweight network that projects sparse contact observations onto the contact manifold and is 95x faster and 18% more accurate. Our method, evaluated on three industrially relevant geometries with clearances of 0.1-1.0 mm, achieves a success rate of 93.3%, a 4.1x improvement compared to primitive-only strategies without state estimation.

Disassembly is a critical challenge in maintenance and service tasks, particularly in high-precision operations such as electric vehicle (EV) battery recycling. Tasks like prying-open sealed battery covers require precise manipulation and controlled force application. In our approach we collect human demonstrations using a motion capture system, enabling the robot to learn from human-expert disassembly strategies. These demonstrations train a bimanual robotic system in which one arm exerts force with a specialized tool while the other manipulates and removes sealed components. Our method builds on a diffusion-based policy and integrates real-time force sensing to adapt its actions as contact conditions change. We decompose the demonstrations into distinct sub-tasks and apply data augmentation, thereby reducing the number of demonstrations needed and mitigating potential task failures. Our results show that the proposed method, even with a small dataset, achieves a high task success rate and efficiency compared to a standard diffusion technique. We demonstrate in a real-world application that the bimanual system effectively executes chiseling and peeling actions to separate bonded sheet from a substrate.

Uncrewed ground vehicle~(UGV) applications, such as warehouse operations, assembly-line production, infrastructure inspection, surveillance, precision farming, and search \& rescue, can benefit from shared control, in which a human can semi-automatically control the UGV when needed and let it operate fully-automatically, when desired. Many algorithms have been developed to permit a UGV to semi-autonomously conduct tasks, either individually, or in a group. However, a complete semi-autonomous system that works wherever, and whenever, needed is far from being implemented. Here, we develop a human-robot shared controller for the supervisory control of one or more UGVs by a single person. The shared controller blends an automatic control input with a human control input. The automatic control input consists of a trajectory tracking controller and a control barrier function based input term for collision avoidance. A joystick is used to provide the human control input. Human intent is measured employing a Lyapunov-like storage function, which is used in a convex function based blending law that continuously varies the magnitude of the control inputs coming from the human and the machine. The approach permits us to theoretically prove the asymptotic stability of the closed-loop system. The shared controller is validated using both a physical robot in a cluttered indoor environment, and a hardware-in-the-loop simulated robot operating in virtual warehouse environment.

Many manufacturing companies are facing an acute shortage of qualified workers. Deploying robotic cells is a potential solution to address this challenge. Historically robots have been deployed only in mass production applications in manufacturing. A large fraction of manufacturing is classified as high-mix manufacturing where a large variety of products are produced. Manually programming robots is not a viable solution in high-mix manufacturing applications. Robotic cells need to be powered by embodied AI to make them useful in high-mix manufacturing applications. This paper aims to build a bridge between smart manufacturing and AI communities to enable AI researchers to develop methods and tool that can be successfully deployed to realize smart robotic cells for high-mix manufacturing applications. This paper highlights key requirements for developing embodied AI for powering robotic cells for high-mix manufacturing applications. It also makes the case for approaches that combine model-based and data-driven methods to meet the needs of embodied AI in manufacturing applications and describes the role of generative AI approaches in smart manufacturing applications. Finally, it describes how AI can be used to enhance digital twins and augment human-machine interfaces in manufacturing applications.

The growing adoption of batteries in the electric vehicle industry and various consumer products has created an urgent need for effective recycling solutions. These products often contain a mix of compliant and rigid components, making robotic disassembly a critical step toward achieving scalable recycling processes. Diffusion policy has emerged as a promising approach for learning low-level skills in robotics. To effectively apply diffusion policy to contact-rich tasks, incorporating force as feedback is essential. In this paper, we apply diffusion policy with vision and force in a compliant object prying task. However, when combining low-dimensional contact force with high-dimensional image, the force information may be diluted. To address this issue, we propose a method that effectively integrates force with image data for diffusion policy observations. We validate our approach on a battery prying task that demands high precision and multi-step execution. Our model achieves a 96% success rate in diverse scenarios, marking a 57% improvement over the vision-only baseline. Our method also demonstrates zero-shot transfer capability to handle unseen objects and battery types.

The growing adoption of batteries in the electric vehicle industry and various consumer products has created an urgent need for effective recycling solutions. These products often contain a mix of compliant and rigid components, making robotic disassembly a critical step toward achieving scalable recycling processes. Diffusion policy has emerged as a promising approach for learning low-level skills in robotics. To effectively apply diffusion policy to contact-rich tasks, incorporating force as feedback is essential. In this paper, we apply diffusion policy with vision and force in a compliant object prying task. However, when combining low-dimensional contact force with high-dimensional image, the force information may be diluted. To address this issue, we propose a method that effectively integrates force with image data for diffusion policy observations. We validate our approach on a battery prying task that demands high precision and multi-step execution. Our model achieves a 96\% success rate in diverse scenarios, marking a 57\% improvement over the vision-only baseline. Our method also demonstrates zero-shot transfer capability to handle unseen objects and battery types. Supplementary videos and implementation codes are available on our project website. https://rros-lab.github.io/diffusion-with-force.github.io/

Large-scale metal additive manufacturing has become increasingly popular in aerospace and petroleum industries alike for sustainable fabrication of thin-shelled structural components. For example, wire-arc additive manufacturing (WAAM) offers high-deposition rates on large printing areas by robot-assisted welding of thick layers of material. However, WAAM technologies suffer from significant layer displacement and resultant part-scale distortion due to unstable high temperature deposition processes and lack of economically viable support structures. Therefore, geometric accuracy qualification at layer level is critical to process optimization and control. However, layer quality assessment relies on layer identification from large point clouds. Manual layer segmentation is experience-dependent and time-consuming due to high surface roughness, excessive layer remelting, and severe out-of-plane layer displacement. To enable automated layer segmentation for quality assessment, we computationally model a human operator’s intuition utilized in the process of finding layer boundaries, that is, locating nearby regions with a large number of possible boundary points, and finetuning boundaries by learning boundaries functions. In our proposed approach, geometrical features of the boundaries between printed layers are exploited to identify candidate boundary points. Cooperative multi-learning-agents efficiently process the large point clouds to locate sets of nearby regions with a high density of boundary points. Learning agents then sample promising boundary points. Gaussian process regression is employed to fine-tune layer boundaries through learning mean boundary functions and their uncertainties from the sampled points. Simulation studies demonstrate the accuracy and robustness of the procedure under severe surface roughness conditions. WAAM experimental studies illustrate the applicability of the methodology in practice.

Robotic in-space assembly is a key technology to enable fabrication of large structures in space. A critical task of robotic assembly is insertion, where uncertainty in pose from factors such as positional sensor errors and abruptly changing environmental conditions may cause misalignment and ultimately lead to jamming. In this work, we demonstrate a robotic arm performing an insertion task of geometry with significant complexity. The robotic system utilizes a two-stage vision-based algorithm to provide pose estimation prior to attempting insertion. The first stage provides a coarse pose estimate which allows the camera mounted to the wrist of the robotic arm to be moved to a closer view of the target. The second stage, a learning-based model, provides a fine pose estimate of the assembly misalignment. The fine pose estimate is then used to perform insertion using an impedance controller. In our experiments, we show the learning-based model's pose estimation capability under varied lighting conditions and demonstrate that the model can be trained to be robust to lighting conditions unseen in training data. Additionally, we tune an impedance controller to actively comply to misalignments during insertion, which further improves insertion success rate. Overall, we demonstrate an autonomous robotic insertion algorithm which provides robustness to uncertainty through state estimation as well as controls.

Multi-robot systems are becoming more common in various real-world applications, such as manufacturing and warehouse logistics. However, task allocation and scheduling for a multi-agent team face complex challenges due to the need to simultaneously consider time-extended tasks, task constraints, and uncertainties in execution. Potential task failures or contingencies can add additional tasks to recover from the failures, and reactively addressing contingencies can decrease teaming efficiency. To efficiently and proactively consider contingencies, this paper proposes treating the problem as a multi-robot task allocation under uncertainty problem. We suggest a hierarchical approach that divides the problem into two layers. We use mathematical program formulation for the lower layer to find the optimal solution for a deterministic multi-robot task allocation problem with known task outcomes. The higher-layer search intelligently generates more likely combinations of contingency scenarios and calls the inner-level search repeatedly to find the optimal task allocation sequence for the given scenario. We validate our results in simulation for manufacturing applications and demonstrate that our method can reduce the effect of potential delays from contingencies.Note to Practitioners—Automation engineers interested in deploying robotic cells in low-volume applications need to consider contingency handling. When the occurrence of contingencies can be characterized as probability distributions, it is often useful to consider using a proactive approach for task allocation and scheduling. To implement our algorithm, automation engineers will need to develop a hierarchical task network specified by domain experts that models task constraints and a task-agent duration model, which may be generated from simulation environments. Furthermore, they must identify tasks that can result in contingencies and describe them with a probabilistic model. This model can be generated from historical data and/or real-world experiments. Lastly, for addressing the contingency, the practitioner will need to specify a task procedure to recover from a specific contingency type. To run the algorithm, we found that repeatedly approximating the best proactive task allocation for a fixed computation budget and dispatching the best tasks worked well. The computation budget required to approximate the best task allocation is directly affected by the number of contingency scenarios that can be sampled. Therefore, the practitioner must determine a suitable computational budget empirically based on the number of contingencies that can occur.

Augmenting capabilities of human operators with multi-robot cells offers substantial advantages for increasing productivity in manufacturing applications. This synergy effectively combines the strengths of both robots and humans, maximizing operational efficiency and leveraging human capabilities. However, achieving these benefits requires real-time, reactive coordination of multi-robot motion generation in response to human motion. Current approaches face significant challenges, particularly in dealing with uncertainties in human motions. To address these issues, this paper introduces the Deep Reinforcement Learning (DRL) approach for end-to-end safe motion generation in human multi-robot collaborative workspaces. First, the DRL approach is augmented by adopting mixed-reality (MR) features to facilitate efficient state perception and representation of tasks, humans, robots, and scenes for enabling effective learning motion generation policy. Moreover, to better promote high-dimensional action generation of the multi-robot systems involving human, an advanced DRL approach is developed. The approach leverages memory-enhanced representation learning, intrinsic reward-guided exploration, and action space pruning to better address the motion generation challenges. Empirical testing demonstrates the effectiveness of the proposed system, with experiments showing high success rates across tasks with varying team sizes and difficulty levels, thereby demonstrating applicability in human-robot collaborative manufacturing tasks.

Uncrewed ground vehicle (UGV) applications, such as warehouse operations, assembly-line production, infrastructure inspection, surveillance, precision farming, and search & rescue, can benefit from shared control, in which a human can semi-automatically control the UGV when needed and let it operate fully-automatically, when desired. Many algorithms have been developed to permit a UGV to semi-autonomously conduct tasks, either individually, or in a group. However, a complete semi-autonomous system that works wherever, and whenever, needed is far from being implemented. Here, we develop a human-robot shared controller for the supervisory control of one or more UGVs by a single person. The shared controller blends an automatic control input with a human control input. The automatic control input consists of a trajectory tracking controller and a control barrier function based input term for collision avoidance. A joystick is used to provide the human control input. Human intent is measured employing a Lyapunov-like storage function, which is used in a convex function based blending law that continuously varies the magnitude of the control inputs coming from the human and the machine. The approach permits us to theoretically prove the asymptotic stability of the closed-loop system. The shared controller is validated using both a physical robot in a cluttered indoor environment, and a hardware-in-the-loop simulated robot operating in virtual warehouse environment.

Abstract Research in quadrupedal robotics is transitioning to studies into loco-manipulation, featuring fully articulated robotic arms mounted atop these robots. Integrating such arms enhances the practical utility of legged robots, paving the way for expanded applications like industrial inspection and search and rescue. Existing literature commonly employs a six-degree-of-freedom (six-DoF) arm directly mounted to the robot, which inherently adds significant weight and reduces the available payload for manipulation tasks. Our study explores an optimized combination of arm configuration and control framework by strategically reducing the DoFs and leveraging the quadruped robot’s inherent agile mobility. We demonstrate that by minimizing the DoFs to just one, a range of canonical loco-manipulation tasks can still be accomplished. Some tasks even show improved performance with fewer robotic arm DoFs due to the higher torque motor used in the design, allowing more of the robot’s payload to be used for manipulation. We designed our optimized one-DoF robotic arm and the control framework and tested it on top of a Unitree Aliengo. Our design outperforms conventional six-DoF counterparts in lifting capacity, achieving an impressive 8 kg payload compared to the 2 kg maximum payload of industry-standard six-DoF robotic arms on the same quadruped platform.

Suction cups are popular for picking and transporting packages in warehouse applications. To maximize throughput, high transport speeds are desired. Many packages are deformable and may detach from the suction cups due to inertial loading if trajectories use excessive velocities. This paper introduces a novel methodology that analyzes package deformation through its curvature at the package-suction cup contact interface to generate a Factor-of-Safety (FOS) score for each waypoint in a given trajectory. By maintaining the FOS above a predetermined threshold, the trajectory planner is able to generate transport trajectories that are both safe and time-optimized. Experimental results show the method’s efficacy, demonstrating a 21.92% reduction in transport times compared to a conservative trajectory generation. Our FOS predictor identified trajectories that ensured safe package transport with 100% accuracy across all 627 real-world experiments.

Bin-packing is an important problem in the robotic warehouse domain. Traditionally, this problem has been studied only for rigid packages (e.g., boxes or rigid objects). In this work, we tackle the problem of bin-packing with deformable packages that have become a popular choice for fulfillment needs. We present a system that incorporates a dual robot arm bimanual setup, uniquely combining suction and sweeping motions to stably and reliably pack deformable packages in a bin. Additionally, we propose a comprehensive action prediction framework to optimize for bin-packing efficiency by predicting optimal actions for both robots involved. Our methodology leverages a two-pronged learning strategy, where initially, we train a model in a self-supervised manner to predict a scoring metric indicative of bin-packing efficiency and then leverage an online optimization scheme to compute optimal actions in real time. The model is pre-trained in simulation in MuJoCo and fine-tuned on small-scale data from a real-world laboratory setting. Our packing score prediction model predicts bin-packing score ∈ [0, 1] with an MSE of 0.003. Real-world experiments validate our method’s adaptability to novel scenarios and its effectiveness in packing operations. Project Website: https://sites.google.com/usc.edu/bimanual-binpacking/

Multi-robot teams can be very useful in a wide variety of search and rescue missions in challenging environments. In a mission with considerable uncertainty due to intermittent communications, degraded information flow, and failures, humans need to assess both the current and expected future states and update task assignments in human-robot teams as quickly as possible. We have developed an alert generation framework that can perform risk assessment and robot tasking suggestions to assist human supervisors. Our approach for task assignment suggestion generation combines heuristics-based task selection with forward simulation-based probabilistic assessment. As the characteristics of decision aids can largely vary human performance, an alert system may or may not improve decision-making. We aim to configure our framework with a goal to improve human decision-making performance. Towards that, we present some preliminary user studies and design reasoning, which informed our final comprehensive human subject study. We demonstrate in the study that supervisors can improve their decision-making abilities, make faster decisions, and increase mission performance by using our alert generation framework. Our empirical findings also show that our framework does not require significant training and that people with a higher level of trust in automation perform better when provided with alerts. We also find that people with certain personality traits such as high agreeableness and conscientiousness are the most benefited by alerts.

In this work, we address the challenge of incorporating human preferences into the task-scheduling process for human-robot teams. Humans have various individual preferences that can be influenced by context and situational information. Incorporating these preferences can lead to improved team performance. Our main contribution is a framework that helps elicit and incorporate preferences during task scheduling. We achieve this by proposing 1) a constraint programming method to generate a range of plans, 2) an intelligent approach for selecting and presenting task schedules based on task features, and 3) a preference incorporation method that uses large language models to convert preferences into soft constraints. Our results demonstrate that we can efficiently generate diverse plans for preference elicitation and incorporate them into the task-scheduling process. We evaluate our framework using an assembly-inspired case study and show how it can effectively incorporate complex and realistic preferences. Our implementation can be found at github.com/RROS-Lab/Human-Robot-Preference-Planning.

The automation of composite sheet layup is essential to meet the increasing demand for composite materials in various industries. However, draping plans for the robotic layup of composite sheets are not robust. A plan that works well under a certain condition does not work well in a different condition. Changes in operating conditions due to either changes in material properties or working environment may lead a draping plan to exhibit suboptimal performance. In this paper, we present a comprehensive framework aimed at refining plans based on the observed execution performance. Our framework prioritizes the minimization of uncompacted regions while simultaneously improving time efficiency. To achieve this, we integrate human expertise with data-driven decision-making to refine expert-crafted plans for diverse production environments. We conduct experiments to validate the effectiveness of our approach, revealing significant reductions in the number of corrective paths required compared to initial expert-crafted plans. Through a combination of empirical data analysis, action-effectiveness modeling, and search-based refinement, our system achieves superior time efficiency in robotic layup. Experimental results demonstrate the efficacy of our approach in optimizing the layup process, thereby advancing the state-of-the-art in composite manufacturing automation.

Delays play an important role in the overall performance of multi-robot missions. Delays can be handled in a reactive manner by replanning or can be handled proactively by selecting task allocations that are robust to delays. Mixed Integer Linear Programming (MILP) has emerged as a useful tool for solving multi-robot task allocation problems. The possibility of delays can be incorporated by sampling many different delay scenarios and generating an optional solution for each scenario. The solution that leads to the most robust performance is selected as the overall solution. Unfortunately, generating a large number of delay scenarios and computing the robustness of each solution by evaluating it using every sampled delay scenario is computationally very slow. This paper presents a method for speeding up computations by pruning solutions based on a similarity index. We evaluate the proposed approach using assembly and disinfection applications. We show that our approach leads to very good solutions with a modest computational effort.

Abstract In this work, we present a step toward boosting human picker efficiency in strawberry harvesting using quadruped robots. Transitioning from manual to automated harvesting in the fruit industry, especially for fresh-market strawberries, remains a challenge due to the delicate nature of the fruit and unique field layouts. A notable inefficiency in manual harvesting is the time human pickers spend transporting filled trays to collection stations. Quadruped robots, renowned for their adaptability in challenging terrains, emerge as promising candidates for mobile crop transportation. Extending the foundational concepts from previous work, we introduce quadruped robots as auxiliary support units specifically tailored to boost human picker efficiency. Such legged platforms have demonstrated their mobility in agricultural scenarios. The robots can traverse various agricultural terrains, including rugged and muddy furrows in the field and main irrigation lines at the headland. Beyond advancements in transportation robot hardware, our system introduces a novel predictive scheduling algorithm, aiming to enhance harvesting efficiency. Specifically, this algorithm is engineered for real-time replanning, to account for unpredictable picker behaviors. Our empirical results show that the integration of quadruped robots leads to 15% decrease in total makespan of the task and a 72% decrease in non-picking time, compared to traditional manual methods.

Abstract This paper presents a learning framework for enabling robots to autonomously dispense granular materials on demand. This framework enables robots to scoop and transfer the requested material amount with milligram scale accuracy. Our approach is capable of handling challenging cases where the amount left in the source container is significantly less than the container volume. In such cases, robots must build piles before scooping the material to capture enough material within the scooper. We use Gaussian Process Regression (GPR) to predict granular material behavior during scooping and pouring tasks. GPR is effective in learning the behavior of granular material with task parameters, such as robot joint angles, joint accelerations, and end-effector geometry. During task execution, we use GPR to solve the inverse problem and determine the task parameters based on the desired scooping and pouring amounts. The system performance is evaluated by showing GPR’s ability to predict scooped and poured amounts with reasonable uncertainty. We benchmark our method against the traditional approach of fine-tuning the amount via closed-loop control from the scale sensor feedback. Our method shows 55.2% improvement in time taken to dispense the granular material over the benchmark approach. The proposed framework shows promising results in terms of reducing dispensing times.

Abstract Automating assembly operations effectively increase efficiency while decreasing the need for humans to perform ergonomically challenging tasks. However, full automation of these tasks is still a work in progress. Leveraging the complementary strengths of humans and robots offers a solution. Humans can handle tasks requiring dexterity by working in a team, while robots undertake routine, supportive roles. However, contingency situations created by task execution failures occur more frequently in high-mix applications because of the high variability in the types of tasks. Work cells that enable collaboration between humans and robots are not likely to be economically viable unless contingency situations are detected and efficiently managed. In such situations, additional contingency tasks must be executed to recover from the contingency, and productively utilizing human agents can help the work cell quickly recover from contingencies. This paper presents a framework for automatically assigning and scheduling tasks to humans and robots to complete multiple assemblies while managing the computational complexity of generating optimal task plans. Additionally, we present methods to generate recovery task plans that effectively resolve those contingencies automatically. In our case study, we present an approach to graphically determine the number of tasks and products to consider with limited computing time.

We address the challenge of transferring human preferences for action selection from simpler source tasks to complex target tasks. Our goal is to enable robots to support humans proactively by predicting their actions — without requiring demonstrations of their preferred action sequences in the target task. Previous research has relied on human experts to design or select a simple source task that can be used to effectively learn and transfer human preferences to a known target. However, identifying such source tasks for new target tasks can demand substantial human effort. Thus, we focus on automating the selection of source tasks, introducing two new metrics. Our first metric selects source tasks in which human preferences can be accurately learned from demonstrations, while our second metric selects source tasks in which the learned preferences, although not as accurate, can match the preferred human actions in the target task. We evaluate our metrics in simulated tasks and two human-led assembly studies. Our results indicate that selecting high-scoring source tasks on either metric improves the accuracy of predicting human actions in the target task. Notably, tasks chosen by our second metric can be simpler than the first, sacrificing learning accuracy but preserving prediction accuracy.

In recent years, the field of legged robotics has seen growing interest in enhancing the capabilities of these robots through the integration of articulated robotic arms. However, achieving successful loco-manipulation, especially involving interaction with heavy objects, is far from straightforward, as object manipulation can introduce substantial disturbances that impact the robot’s locomotion. This paper presents a novel framework for legged loco-manipulation that considers whole-body coordination through a hierarchical optimization-based control framework. First, an online manipulation planner computes the manipulation forces and manipulated object task-based reference trajectory. Then, pose optimization aligns the robot’s trajectory with kinematic constraints. The resultant robot reference trajectory is executed via a linear MPC controller incorporating the desired manipulation forces into its prediction model. Our approach has been validated in simulation and hardware experiments, highlighting the necessity of whole-body optimization compared to the baseline locomotion MPC when interacting with heavy objects. Experimental results with Unitree Aliengo, equipped with a custom-made robotic arm, showcase its ability to lift and carry an 8kg payload and manipulate doors.

Multi-robot systems are becoming increasingly prevalent in various real-world applications, such as manufacturing and warehouse logistics. These systems face complex challenges in 1) task allocation due to factors like time-extended tasks, and agent specialization, and 2) uncertainties in task execution. Potential task failures can add further contingency tasks to recover from the failure, thereby causing delays. This paper addresses the problem of Multi-Robot Task Allocation under Uncertainty by proposing a hierarchical approach that decouples the problem into two levels. We use a low-level optimization formulation to find the optimal solution for a deterministic multi-robot task allocation problem with known task outcomes. The higher-level search intelligently generates more likely combinations of failures and calls the inner-level search repeatedly to find the optimal task allocation sequence, given the known outcomes. We validate our results in simulation for a manufacturing domain and demonstrate that our method can reduce the effect of potential delays from contingencies. We show that our algorithm is computationally efficient while improving average makespan compared to other baselines.