Research Overview

Our lab realizes robot teammates by developing the following three sequential system capabilities:

  1.  Plan:  A system to participate in early stages of human team planning to infer the agreed upon “idealized” shared plan.
  2.  Refine:  A system to refine task plans through observation and interaction with humans in real contexts.
  3.  Execute:  An online system to rapidly predict the spatio-temporal trajectory of future human actions and react accordingly.

Research Contribution #1: Plan

A system to participate in early stages of human team planning to infer the agreed upon “idealized” shared plan.

Human teams often form plans by achieving consensus among members of the group. In contrast, most existing models and algorithms for machine-assisted planning formalize the problem by assuming one human planner and one automated planner. In addition, machine support of team planning requires computationally tractable models to infer shared understanding of commitment.  However, prior models of human team planning are qualitative, or formal but not computationally tractable. Finally, machine participation in real world team planning is not possible when the machine assumes its models are correct and complete.

We have designed computational models of team planning that draw insight from well-established models of human cognition. The model structures are designed to perform real-time inference of the team's shared understanding of commitment.

Kim, J., M. E. Woicik, M. C. Gombolay, S-H. Son, and J. A. Shah, "Learning to Infer Final Plans in Human Team Planning", International Joint Conferences on Artificial Intelligence (IJCAI), 07/2018.
Kim, J., C. J. Banks, and J. A. Shah, "Collaborative Planning with Encoding of Users’ High-level Strategies", The AAAI Conference on Artificial Intelligence (AAAI), 02/2017.
Kim, J., and J. A. Shah, "Improving Team's Consistency of Understanding in Meetings", IEEE Transactions on Human-Machine Systems (THMS), vol. 46, issue 5, pp. 625-637, 04/2016.
Kim, B., C. M. Chacha, and J. A. Shah, "Inferring Team Task Plans from Human Meetings: A Generative Modeling Approach with Logic-Based Prior", Journal of Artificial Intelligence Research (JAIR) , pp. 361-398, 2015.
Kim, B., C. M. Chacha, and J. A. Shah, "Inferring Robot Task Plans from Human Team Meetings: A Generative Modeling Approach with Logic-Based Prior", AAAI Conference on Artificial Intelligence (AAAI) [29% Acceptance Rate], 07/2013.

Research Contribution #2: Refine

A system to refine task plans through observation and interaction with humans in real contexts.

The purpose of team planning is to form an idealized shared plan. But the team rarely follows the plan exactly. People change their collaboration strategies (i.e. rules and heuristics for task allocation, synchronization, timing) based on many factors including past experience, workload or personal preference. And people find it difficult to explicitly communicate how the team adapts their idealized plan to myriad real-world situations.

These challenges motivate a machine learning approach for refining idealized shared plans for real contexts. However there are no prior effective techniques for learning human task allocation and scheduling policies from human demonstration. There is typically little data available to learn from, no environment simulator/expert emulator, and many approaches to learning require regression through a very large state space.

Further, machine learning through remote observation is not sufficient to refine a human-robot team plan. People adapt their strategies through continued interactions with teammates. An effective machine teammate must co-adapt its collaboration strategies to support the human's learning process.

Our approach to policy learning is made possible with little data and no environment/expert emulator, through insight from models of human cognition. Our models and algorithms for jointly optimizing human-robot team performance are derived based on insights from effective human team training processes, including cross-training and perturbation training.

Gombolay, M., X. Jessie Yang, B. Hayes, N. Seo, Z. Liu, S. Wadhwania, T. Yu, N. Shah, T. Golen, and J. Shah, "Robotic assistance in the coordination of patient care", International Journal of Robotics Research (IJRR) (Accepted 02/2018—To Appear), 2018.
Gombolay, M., R. Jensen, J. Stigile, T. Golen, N. Shah, S-H. Son, and J. Shah, "Human-Machine Collaborative Optimization via Apprenticeship Scheduling", Journal of Artificial Intelligence Research (JAIR) (Accepted 02/2018—To Appear), 2018.
Ramakrishnan, R., C. Zhang, and J. Shah, "Perturbation Training for Human-Robot Teams", Journal of Artificial Intelligence Research (JAIR), vol. 59, 07/2017.
Gombolay, M., R. Jensen, J. Stigile, S-H. Son, and J. Shah, "Apprenticeship Scheduling: Learning to Schedule from Human Experts", International Joint Conferences on Artificial Intelligence (IJCAI), 07/2016.
Nikolaidis, S., P. Lasota, R. Ramakrishnan, and J. Shah, "Improved human–robot team performance through cross-training, an approach inspired by human team training practices", International Journal of Robotics Research (IJRR), vol. 34, issue 14, pp. 1711-1730, 12/2015.

Research Contribution #3: Execute

An online system to rapidly predict the spatio-temporal trajectory of future human actions and react accordingly.

Once a team has refined their plan, the members must coordinate while executing the plan. However, it is challenging for a machine to monitor a human's real-time progress through a plan. Prior activity recognition approaches are designed and tuned for specific motions or tasks. No existing single technique provides accurate predictions over short and long time horizons, in most scenarios. The robot must predict detailed space-time trajectories of human actions for short and long timescales (<1s to 10-20s) to react appropriately. 

Robots also need to make quick adjustments to their plans based on continually updating predictions of human actions.There are no prior task assignment and sequencing algorithms that scale to multi-agent factory size problems and support on-the-fly scheduling in the presence of temporal and spatial proximity constraints.

Our algorithms automatically leverage temporal and spatial models of activities to perform early classification of human activities and quickly update robot task plans.

Gombolay, M., R. Wilcox, and J. Shah, "Fast Scheduling of Robot Teams Performing Tasks with Temporospatial Constraints", IEEE Transactions on Robotics (T-RO), vol. 34, issue 1, pp. 220-239, 02/2018.
Hayes, B., and J. A. Shah, "Interpretable Models for Fast Activity Recognition and Anomaly Explanation During Collaborative Robotics Tasks", IEEE International Conference on Robotics and Automation (ICRA), 05/2017.
Lasota, P. A., and J. A. Shah, "A Multiple-Predictor Approach to Human Motion Prediction", IEEE International Conference on Robotics and Automation (ICRA), 05/2017.
Unhelkar, V. V., C. Pérez-D'Arpino, L. Stirling, and J. A. Shah, "Human-Robot Co-Navigation using Anticipatory Indicators of Human Walking Motion", IEEE International Conference on Robotics and Automation (ICRA), 05/2015.
Gombolay, M. C., R. J. Wilcox, and J. A. Shah, "Fast Scheduling of Multi-Robot Teams with Temporospatial Constraints", Robotics: Science and Systems (RSS) [30% Acceptance Rate], 06/2013.

Research