Sabrina McCallum

Embodied environments are the ideal testbed for AI frameworks before they are deployed in real-world applications such as robots, Virtual and Augmented Reality, and multi-modal virtual assistants. In line with the embodiment hypothesis proposed by Cognitive Science, whereby intelligence and cognition emerge when a physical body interacts with its environment, Embodied AI research is inherently multi-modal, and agents operating in embodied environments need to be able to perceive and understand visual (‘see’) and audio signals (‘hear’) and use these abilities to act and reason. This involves encoding and learning from synthetic and/or real-world multi-modal experiences, for example by executing actions in the simulated environment, or by consuming real-world multi-media content. However, the majority of existing research in the field leverages only one (vision) or two (vision and language) modalities and is specific to the underlying simulated environment. This research seeks to address both shortcomings, by enabling artificial agents to learn from visual, audio and language signals and generalize across different embodied environments.