Raul G. Longoria

Abstract This paper presents methods used to design and develop a smart hand tool system that takes advantage of low-cost sensing, machine learning, and real-time monitoring to optimize tool usage and improve human-tool interaction. A multidisciplinary team took a user-focused approach, balancing engineering design, prototyping, and testing with qualitative research and quantitative analysis to derive user needs and requirements. A prototype sensor unit (PSU) that can be adapted to various tools was developed to enable the real-time acquisition of data on motion, power consumption, orientation, and user activity. The prototype was experimentally validated on multiple tool types, and ML-enabled features including skill assessment, task recognition, battery life prediction, load estimation, and anomaly detection were developed and tested. Skill assessment ranking of user proficiency based on a Skill Index Score (SIS) correlated well with GD&T-based evaluations. Task recognition algorithms achieved over 77% accuracy, while battery life prediction closely matched real usage data. Load estimation was found to provide force predictions with an average error of ±1.18 N, and anomaly detection identified deviations such as excessive force and tool stoppages. These features were processed online using PSU and edge computing features. The results demonstrate the feasibility of further developing AI-enhanced power tools with real-time monitoring and performance evaluation, paving the way for advances in human-tool collaboration, skill development, training, and next-generation smart manufacturing applications.

Abstract The integration of low-cost sensors into hand tools increases opportunities to improve user productivity and safety. Human activity recognition (HAR) using sensor data is already a rich field focused on understanding human behavior in a wide range of practical applications, ranging from workplace environments to sports and medicine. This paper emphasizes human tool use. Tool manufacturers and instrumented tools used in industry have established the feasibility and utility of embedded sensors. A next step in tool ‘intelligence’ is to make judgments on human behavior and tool use for skill assessment. Skill assessment provides a unique challenge in that it is more subjective than HAR, in fact, the idea of skill can vary from task to task. This paper applies five different methods of skill analysis to rotary hand tool inertial measurement unit (IMU) and video data, while adopting a comparison to part tolerance using image-based GD&T techniques. While hand tool data provide unique challenges not currently present in the field of skill assessment, it is possible to use a variety of statistical and machine learning (ML) methods to determine user skill using both IMU and video data. The results suggest ways that skill monitoring functionality might be integrated into smart tool platforms.

This chapter reviews fundamental concepts useful in a more generalized approach to modeling thermodynamic systems and processes. In particular, this includes the introduction of a full multiport representation for Gibbs internal energy. Examples of equations of state are provided for gases and solids. This enables ways for modeling a wider range of thermo-fluid systems, including compressible flows, diffusion through membranes, and other types of open system effects. The methods introduced are also shown to lay the groundwork for dealing with multicomponent species, and for modeling chemical reactions within a bond graph framework.

Introductory text on nonlinear and continuous-time dynamic systems using bond graph methodology to enable readers to develop and apply physical system models Through an integrated and uniform approach to system modeling, analysis, and control, Modeling of Physical Systems uses realistic examples to link empirical, analytical, and numerical approaches and provide readers with the essential foundation needed to move towards more advanced topics in systems engineering. Rather than use only a linear modeling methodology, this book also incorporates nonlinear modeling approaches. The authors approach the topic using bond graph methodology, a well-known and highly effective method for the modeling and analysis of multi-energy domain systems at the physical level. With a strong focus on fundamentals, this book begins by reviewing core topics which engineering students will have been exposed to in their first two years of study. It then expands into introducing systematic model development using a bond graph approach. Later chapters expand on the fundamental understanding of systems, with insights regarding how to make decisions on what to model and how much complexity is needed for a particular problem. Written by two professors with nearly a century of combined research and industry experience, Modeling of Physical Systems explores topics including: Basic Kirchoff systems, covering mechanical translation and rotation, electrical, hydraulic, and thermal systems, and ideal couplersA complete introduction to bond graph methods and their application to practical engineering system modelingComputer-based analysis and simulation, covering algebraic analysis of system equation and semi-analytical analysis for linear system responseMultiport fields, distributed systems and transmission elements, covering heat and magnetism power lines and wave propagation modeling with W- and H-LinesSignal and power in measurement and control, covering derivative control and effect of feedback Modeling of Physical Systems is an essential learning resource for mechanical, mechatronics, and aerospace engineering students at the graduate and senior graduate level. The text is also valuable for professional engineers and researchers, controls engineers, and computer scientists seeking an understanding of engineering system modeling.

Synthetic data offers a versatile solution for generating realistic datasets. In this study, we introduce a methodology specifically focused on generating synthetic inertial acceleration data that effectively captures the dynamics of human arm gestures. Classical human biomechanical models for human motion—such as minimum jerk, minimum torque-change, and iterative Linear Quadratic Regulator (iLQR) are adapted and used to systematically decompose complex gestures into fundamental point-to-point movements. The models incorporate realistic sensor noise models to enhance the authenticity of the synthetic datasets, and experimental validation demonstrates that the synthetic trajectories generated closely align with actual inertial data recorded during predefined arm gestures.

ABSTRACT This paper explores an approach for applying Artificial Intelligence (AI) to co‐design smart hand tools to personalize learning for future skilled trade workers in workforce training programs. The purpose of this research is to better understand the perspectives of workers in the skilled trades and to respond with co‐designed socio‐technical interventions that empower workers. The research benefits from a collaboration between The University of Texas at Austin, the City of Austin, and Austin Community College (ACC) and incorporates insights from welding instructors and students, as well as skilled trade workers and supervisors. Social science findings derived from semi‐structured interviews inform tool design implemented by an interdisciplinary research team. The participatory design approach has resulted in two prototypes: a welding simulator that uses Augmented Reality (AR) and an AI‐enabled (smart) rotary tool. This paper has implications for workforce development to address skilled worker shortages. Additionally, it contributes to ongoing research into AI and skilled trade work which is understudied compared to AI and knowledge work.

Abstract This paper describes research toward developing smart hand tools that leverage artificial intelligence (AI) and sensors for use by human workers. Smart hand tools can provide useful feedback that can benefit human workers, contribute to worker training, and broaden participation in the skilled trade workforce. Specifically, the paper focuses on task recognition. Given the challenges of producing enough training data for machine learning (ML) using data purely from human-based testing, this paper shows how data synthetically-generated by a robot can be leveraged in the ML training process. The paper also demonstrates how fine-tuning ML models for individual physical tasks and workers can significantly scale up the benefits of using ML to provide this feedback. Experimental results show the effectiveness and scalability of this approach, including comparing test data size versus accuracy. In order for smart hand tools of the type introduced here to operate in real-time task recognition, as well as providing analytics on efficient and safe tool usage and operation, ML models need to be deployed ‘on tool’. This paper demonstrates how this can be accomplished by using a tinyML implementation. This paper provides a proof-of-concept for using automated platforms to help train smart tools, which will be essential given the wide range of uses for smart hand tools.

Abstract This study introduces two methodologies designed to monitor welding procedures, furnish feedback to users, and thereby refine the welding technique of novice welders. The first method introduces a low-cost welding simulator utilizing augmented reality (AR) technology. This simulator provides real-time visual feedback on essential welding parameters such as the contact tip to work distance (CTWD), work angle, travel angle, displacement, and travel speed. The second approach involves a welding sensor unit (WSU) attached to the actual welding torch, enabling monitoring and analysis of welding parameters in a real welding environment. Experimental verification of both methods demonstrates their effectiveness and accuracy. The simulator accurately replicates human performance within the real welding environment, enabling real-time visual feedback to enhance welding techniques. The WSU, equipped with economical sensors, is feasible for monitoring real-world welding scenarios. These methodologies have diverse applications, including welder training, quality control, and the generation of synthetic data for research and industrial purposes. An investigation into the influence of welding parameters on weld bead formation underscores the critical role these parameters play in shaping both the geometry and quality of welds. Through careful analysis, it becomes evident that factors such as Contact Tip to Work Distance (CTWD), travel speed, and working angles significantly impact the appearance and integrity of weld beads.

For skilled trade workers who use tools to do their work, technological innovation has often resulted in subjugation at the hand of their employers. Advances in artificial intelligence (AI) introduce the possibility of a new dynamic. While most of the hype around AI has focused on its potential to automate tasks and eliminate jobs, the intersection of AI and data cooperatives could allow for the development of smart hand tools that support and empower, rather than subjugate or replace, skilled trade workers. This paper explores smart hand tools as an emancipatory technology (ET) through interviews with supervisors of skilled trade workers employed by a municipal government. This paper responds to the call for additional research on how AI will influence work and workers and suggests that technology can enable skilled trade workers with new levels of agency.

The future of work does not require a choice between human and robot. Aside from explicit human-robot collaboration, robotics can play an increasingly important role in helping train workers as well as the tools they may use, especially in complex tasks that may be difficult to automate or effectively roboticize. This paper introduces a form of smart tool for use by human workers and shows how training the tool for task recognition, one of the key requirements, can be accomplished. Machine learning (ML) with purely human-based data can be extremely laborious and time-consuming. First, we show how data synthetically-generated by a robot can be leveraged in the ML training process. Later, we demonstrate how fine-tuning ML models for individual physical tasks and workers can significantly scale up the benefits of using ML to provide this feedback. Experimental results show the effectiveness and scalability of our approach, as we test data size versus accuracy. Smart hand tools of the type introduced here can provide insights and real-time analytics on efficient and safe tool usage and operation, thereby enhancing human participation and skill in a wide range of work environments. Using robotic platforms to help train smart tools will be essential, particularly given the diverse types of applications for which smart hand tools are envisioned for human use.

This paper lays out an approach to co-designing Artificial Intelligence (AI)-enhanced smart hand tools with skilled trade workers employed at a local municipality. Skilled trade workers contribute to society by building the infrastructure upon which the public depends. In addition, these technical interventions offer an opportunity for workers to benefit from the data they generate via smart hand tools, potentially creating a new empowerment dynamic with employers. Therefore, we consider technologies that support skilled trade workers in performing their work effectively, safely, and with increased levels of autonomy to be considered Public Interest Technologies (PIT). Interdisciplinary research is underway that aligns these approaches with Public Interest Design (PID) principles which informs researchers' desire to explore how emerging technologies - data cooperatives, blockchain, smart contracts, and data dividends - can further smart hand tools' empowerment dynamic. Future participatory design efforts may deliver additional insights and further impact technology deployment.