Crossmodal language grounding, learning, and teaching

Stefan Heinrich, Cornelius Weber, Stefan Wermter, Ruobing Xie, Yankei Lin, Zhiyuan Liu

Research output: Conference Article in Proceeding or Book/Report chapterArticle in proceedingsResearchpeer-review

Abstract

The human brain as one of the most complex dynamic systems enables us to communicate and externalise information by natural language. Despite extensive research, human-like communication with interactive robots is not yet possible, because we have not yet fully understood the mechanistic characteristics of the crossmodal binding between language, actions, and visual sensation that enable humans to acquire and use natural language. In this position paper we present visionand action-embodied language learning research as part of a project investigating multi-modal learning. Our research endeavour includes to develop a) a cortical neural-network model that learns to ground language into crossmodal embodied perception and b) a knowledge-based teaching framework to bootstrap and scale-up the language acquisition to a level of language development in children of age up to two years. We embed this approach of internally grounding embodied experience and externally teaching abstract experience into the developmental robotics paradigm, by means of developing and employing a neurorobot that is capable of multisensory perception and interaction. The proposed research contributes to designing neuroscientific experiments on discovering crossmodal integration particularly in language processing and to constructing future robotic companions capable of natural communication.
Original languageEnglish
Title of host publicationProceedings of the NIPS2016 Workshop on Cognitive Computation (CoCo@NIPS2016)
EditorsT.R. Besold, A. Bordes, A.d'A. Garcez, G. Wayne
Number of pages7
Place of PublicationBarcelona, ES
Publication date1 Dec 2016
Pages62-68
Publication statusPublished - 1 Dec 2016
Externally publishedYes

Fingerprint

Dive into the research topics of 'Crossmodal language grounding, learning, and teaching'. Together they form a unique fingerprint.

Cite this