Conversational AI for human-robot interaction

Develop and evaluate new methods for advanced conversational interaction with robots
Description of the Project: 

There are many outstanding research problems in developing spoken conversational Natural Language interfaces for effective human-robot interaction (HRI).  For example:

  1. Collaborative planning of robot actions e.g. “go to the kitchen and get my mug, but first bring me my medicine”
  2. Visually-grounded language understanding and learning for robots,
  3. Socially intelligent conversational interaction with multiple humans,
  4. Interactive navigation in populated spaces.

Addressing these problems requires advances in Natural Language Processing and machine learning, applied to physically situated robots that can perceive objects and situations around them – including situations where several humans are present.  For the safe operation of robots, humans must be able to communicate with them naturally, rapidly, and robustly – for example asking a robot why it is doing something, or what it plans to do next.  This project will focus on machine learning methods for safe and efficient language understanding, dialogue management, and language generation for robots collaborating with humans. Please see the references for examples of research issues in this area. 

Resources required: 
GPU machines for model training. Robotarium HRI lab for data collection and evaluation experiments. Robot (Pepper/Tiago or similar)
Project number: 
First Supervisor: 
Heriot-Watt University
Second Supervisor(s): 
First supervisor university: 
Heriot-Watt University
Essential skills and knowledge: 
Background in AI
Desirable skills and knowledge: 
Experience with NLP and Machine Learning desirable
  1. Amanda Cercas Curry, Ioannis Papaioannou, Alessandro Suglia, Shubham Agarwal, Igor Shalyminov, Xinnuo Xu, Ondřej Dušek, Arash Eshghi, Ioannis Konstas, Verena Rieser and Oliver Lemon, "Alana v2: Entertaining and Informative Open-domain Social Dialogue using Ontologies and Entity Linking", Alexa Prize Proceedings, Amazon RE-INVENT, Las Vegas, 2018
  2. Igor Shalyminov, Ondrej Dusek, and Oliver Lemon "Neural Response Ranking for Social Conversation: A Data-Efficient Approach", Search-Oriented Conversational AI: EMNLP-SCAI, 2018
  3. Yanchao Yu, Arash Eshghi, and Oliver Lemon, "Learning how to learn: an adaptive dialogue agent for incrementally learning visually grounded word meanings", Robo-NLP workshop at ACL 2017 ** BEST PAPER AWARD **
  4. Ioannis Papaioannou, Christian Dondrup, and Oliver Lemon, "Human-Robot Interaction Requires More Than Slot Filling -  Multi-Threaded Dialogue for Collaborative Tasks and Social Conversation" AI-MHRI workshop 2018 
  5. Yanchao Yu, Arash Eshghi and Oliver Lemon, "An Incremental Dialogue System for Learning Visually Grounded Word Meanings" (demonstration system), Proc. Dialogue and Perception, 2018
  6. Verena Rieser and Oliver Lemon "Reinforcement Learning for Adaptive Dialogue Systems:   A Data-driven Methodology for Dialogue Management and Natural Language Generation" Book Series:   Theory and Applications of Natural Language Processing, Springer, 2011