Embodied Language Understanding with a Multiple Timescale Recurrent Neural Network

Stefan Heinrich; Cornelius Weber; Stefan Wermter

doi:10.1007/978-3-642-40728-4_27

Embodied Language Understanding with a Multiple Timescale Recurrent Neural Network

Stefan Heinrich, Cornelius Weber, Stefan Wermter

University of Hamburg

Research output: Conference Article in Proceeding or Book/Report chapter › Article in proceedings › Research › peer-review

Abstract

How the human brain understands natural language and what we can learn for intelligent systems is open research. Recently, researchers claimed that language is embodied in most – if not all – sensory and sensorimotor modalities and that the brain’s architecture favours the emergence of language. In this paper we investigate the characteristics of such an architecture and propose a model based on the Multiple Timescale Recurrent Neural Network, extended by embodied visual perception. We show that such an architecture can learn the meaning of utterances with respect to visual perception and that it can produce verbal utterances that correctly describe previously unknown scenes.

Original language	English
Title of host publication	Proceedings of the 23rd International Conference on Artificial Neural Networks (ICANN2013)
Editors	Valeri Mladenov, Petia Koprinkova-Hristova, Günther Palm, Alessandro E.P. Villa, Bruno Apolloni, Nicola Kasabov
Number of pages	8
Volume	8131
Publisher	Springer
Publication date	1 Sept 2013
Pages	216-223
DOIs	https://doi.org/10.1007/978-3-642-40728-4_27
Publication status	Published - 1 Sept 2013
Externally published	Yes

Series	Lecture Notes in Computer Science

Keywords

Embodied Language
MTRNN
Language Acquisition

Access to Document

10.1007/978-3-642-40728-4_27

https://stefanheinrich.net/files/2013_Heinrich_ICANN.pdf

Cite this

Heinrich, S., Weber, C., & Wermter, S. (2013). Embodied Language Understanding with a Multiple Timescale Recurrent Neural Network. In V. Mladenov, P. Koprinkova-Hristova, G. Palm, A. E. P. Villa, B. Apolloni, & N. Kasabov (Eds.), Proceedings of the 23rd International Conference on Artificial Neural Networks (ICANN2013) (Vol. 8131, pp. 216-223). Springer. https://doi.org/10.1007/978-3-642-40728-4_27

Heinrich, Stefan ; Weber, Cornelius ; Wermter, Stefan. / Embodied Language Understanding with a Multiple Timescale Recurrent Neural Network. Proceedings of the 23rd International Conference on Artificial Neural Networks (ICANN2013). editor / Valeri Mladenov ; Petia Koprinkova-Hristova ; Günther Palm ; Alessandro E.P. Villa ; Bruno Apolloni ; Nicola Kasabov. Vol. 8131 Springer, 2013. pp. 216-223 (Lecture Notes in Computer Science).

@inproceedings{df0ae69243f847b0bb8dc971e31e5b17,

title = "Embodied Language Understanding with a Multiple Timescale Recurrent Neural Network",

abstract = "How the human brain understands natural language and what we can learn for intelligent systems is open research. Recently, researchers claimed that language is embodied in most – if not all – sensory and sensorimotor modalities and that the brain{\textquoteright}s architecture favours the emergence of language. In this paper we investigate the characteristics of such an architecture and propose a model based on the Multiple Timescale Recurrent Neural Network, extended by embodied visual perception. We show that such an architecture can learn the meaning of utterances with respect to visual perception and that it can produce verbal utterances that correctly describe previously unknown scenes.",

keywords = "Embodied Language, MTRNN, Language Acquisition",

author = "Stefan Heinrich and Cornelius Weber and Stefan Wermter",

year = "2013",

month = sep,

day = "1",

doi = "10.1007/978-3-642-40728-4_27",

language = "English",

volume = "8131",

series = "Lecture Notes in Computer Science",

publisher = "Springer",

pages = "216--223",

editor = "Valeri Mladenov and Petia Koprinkova-Hristova and G{\"u}nther Palm and Villa, {Alessandro E.P.} and Bruno Apolloni and Nicola Kasabov",

booktitle = "Proceedings of the 23rd International Conference on Artificial Neural Networks (ICANN2013)",

address = "Germany",

}

Heinrich, S, Weber, C & Wermter, S 2013, Embodied Language Understanding with a Multiple Timescale Recurrent Neural Network. in V Mladenov, P Koprinkova-Hristova, G Palm, AEP Villa, B Apolloni & N Kasabov (eds), Proceedings of the 23rd International Conference on Artificial Neural Networks (ICANN2013). vol. 8131, Springer, Lecture Notes in Computer Science, pp. 216-223. https://doi.org/10.1007/978-3-642-40728-4_27

Embodied Language Understanding with a Multiple Timescale Recurrent Neural Network. / Heinrich, Stefan; Weber, Cornelius; Wermter, Stefan.
Proceedings of the 23rd International Conference on Artificial Neural Networks (ICANN2013). ed. / Valeri Mladenov; Petia Koprinkova-Hristova; Günther Palm; Alessandro E.P. Villa; Bruno Apolloni; Nicola Kasabov. Vol. 8131 Springer, 2013. p. 216-223 (Lecture Notes in Computer Science).

Research output: Conference Article in Proceeding or Book/Report chapter › Article in proceedings › Research › peer-review

TY - GEN

T1 - Embodied Language Understanding with a Multiple Timescale Recurrent Neural Network

AU - Heinrich, Stefan

AU - Weber, Cornelius

AU - Wermter, Stefan

PY - 2013/9/1

Y1 - 2013/9/1

N2 - How the human brain understands natural language and what we can learn for intelligent systems is open research. Recently, researchers claimed that language is embodied in most – if not all – sensory and sensorimotor modalities and that the brain’s architecture favours the emergence of language. In this paper we investigate the characteristics of such an architecture and propose a model based on the Multiple Timescale Recurrent Neural Network, extended by embodied visual perception. We show that such an architecture can learn the meaning of utterances with respect to visual perception and that it can produce verbal utterances that correctly describe previously unknown scenes.

AB - How the human brain understands natural language and what we can learn for intelligent systems is open research. Recently, researchers claimed that language is embodied in most – if not all – sensory and sensorimotor modalities and that the brain’s architecture favours the emergence of language. In this paper we investigate the characteristics of such an architecture and propose a model based on the Multiple Timescale Recurrent Neural Network, extended by embodied visual perception. We show that such an architecture can learn the meaning of utterances with respect to visual perception and that it can produce verbal utterances that correctly describe previously unknown scenes.

KW - Embodied Language

KW - MTRNN

KW - Language Acquisition

U2 - 10.1007/978-3-642-40728-4_27

DO - 10.1007/978-3-642-40728-4_27

M3 - Article in proceedings

VL - 8131

T3 - Lecture Notes in Computer Science

SP - 216

EP - 223

BT - Proceedings of the 23rd International Conference on Artificial Neural Networks (ICANN2013)

A2 - Mladenov, Valeri

A2 - Koprinkova-Hristova, Petia

A2 - Palm, Günther

A2 - Villa, Alessandro E.P.

A2 - Apolloni, Bruno

A2 - Kasabov, Nicola

PB - Springer

ER -

Heinrich S, Weber C, Wermter S. Embodied Language Understanding with a Multiple Timescale Recurrent Neural Network. In Mladenov V, Koprinkova-Hristova P, Palm G, Villa AEP, Apolloni B, Kasabov N, editors, Proceedings of the 23rd International Conference on Artificial Neural Networks (ICANN2013). Vol. 8131. Springer. 2013. p. 216-223. (Lecture Notes in Computer Science). doi: 10.1007/978-3-642-40728-4_27

Embodied Language Understanding with a Multiple Timescale Recurrent Neural Network

Abstract

Keywords

Access to Document

Fingerprint

Cite this