Adaptive Learning of Linguistic Hierarchy in a Multiple Timescale Recurrent Neural Network

Stefan Heinrich; Cornelius Weber; Stefan Wermter

doi:10.1007/978-3-642-33269-2_70

Adaptive Learning of Linguistic Hierarchy in a Multiple Timescale Recurrent Neural Network

Stefan Heinrich, Cornelius Weber, Stefan Wermter

Publikation: Konference artikel i Proceeding eller bog/rapport kapitel › Konferencebidrag i proceedings › Forskning › peer review

Abstract

Recent research has revealed that hierarchical linguistic structures can emerge in a recurrent neural network with a sufficient number of delayed context layers. As a representative of this type of network the Multiple Timescale Recurrent Neural Network (MTRNN) has been proposed for recognising and generating known as well as unknown linguistic utterances. However the training of utterances performed in other approaches demands a high training effort. In this paper we propose a robust mechanism for adaptive learning rates and internal states to speed up the training process substantially. In addition we compare the generalisation of the network for the adaptive mechanism as well as the standard fixed learning rates finding at least equal capabilities.

Originalsprog	Engelsk
Titel	Proceedings of the 22nd International Conference on Artificial Neural Networks (ICANN2012)
Redaktører	Alessandro E.P. Villa, Włodzisław Duch, Péter Érdi, Francesco Masulli, Günther Palm
Antal sider	8
Vol/bind	7552
Forlag	Springer
Publikationsdato	1 sep. 2012
Sider	555-562
DOI	https://doi.org/10.1007/978-3-642-33269-2_70
Status	Udgivet - 1 sep. 2012
Udgivet eksternt	Ja

Navn	Lecture Notes in Computer Science

Adgang til dokumentet

10.1007/978-3-642-33269-2_70

https://stefanheinrich.net/files/2012_Heinrich_ICANN.pdf

Citationsformater

Heinrich, S., Weber, C., & Wermter, S. (2012). Adaptive Learning of Linguistic Hierarchy in a Multiple Timescale Recurrent Neural Network. I A. E. P. Villa, W. Duch, P. Érdi, F. Masulli, & G. Palm (red.), Proceedings of the 22nd International Conference on Artificial Neural Networks (ICANN2012) (Bind 7552, s. 555-562). Springer. https://doi.org/10.1007/978-3-642-33269-2_70

Heinrich, Stefan ; Weber, Cornelius ; Wermter, Stefan. / Adaptive Learning of Linguistic Hierarchy in a Multiple Timescale Recurrent Neural Network. Proceedings of the 22nd International Conference on Artificial Neural Networks (ICANN2012). red. / Alessandro E.P. Villa ; Włodzisław Duch ; Péter Érdi ; Francesco Masulli ; Günther Palm. Bind 7552 Springer, 2012. s. 555-562 (Lecture Notes in Computer Science).

@inproceedings{bddf40c13ad747f888e88c90287f1e05,

title = "Adaptive Learning of Linguistic Hierarchy in a Multiple Timescale Recurrent Neural Network",

abstract = "Recent research has revealed that hierarchical linguistic structures can emerge in a recurrent neural network with a sufficient number of delayed context layers. As a representative of this type of network the Multiple Timescale Recurrent Neural Network (MTRNN) has been proposed for recognising and generating known as well as unknown linguistic utterances. However the training of utterances performed in other approaches demands a high training effort. In this paper we propose a robust mechanism for adaptive learning rates and internal states to speed up the training process substantially. In addition we compare the generalisation of the network for the adaptive mechanism as well as the standard fixed learning rates finding at least equal capabilities.",

keywords = "Adaptive Learning, MTRNN, Language",

author = "Stefan Heinrich and Cornelius Weber and Stefan Wermter",

year = "2012",

month = sep,

day = "1",

doi = "10.1007/978-3-642-33269-2_70",

language = "English",

volume = "7552",

series = "Lecture Notes in Computer Science",

pages = "555--562",

editor = "Villa, {Alessandro E.P.} and W{\l}odzis{\l}aw Duch and P{\'e}ter {\'E}rdi and Francesco Masulli and G{\"u}nther Palm",

booktitle = "Proceedings of the 22nd International Conference on Artificial Neural Networks (ICANN2012)",

publisher = "Springer",

address = "Germany",

}

Heinrich, S, Weber, C & Wermter, S 2012, Adaptive Learning of Linguistic Hierarchy in a Multiple Timescale Recurrent Neural Network. i AEP Villa, W Duch, P Érdi, F Masulli & G Palm (red), Proceedings of the 22nd International Conference on Artificial Neural Networks (ICANN2012). bind 7552, Springer, Lecture Notes in Computer Science, s. 555-562. https://doi.org/10.1007/978-3-642-33269-2_70

Adaptive Learning of Linguistic Hierarchy in a Multiple Timescale Recurrent Neural Network. / Heinrich, Stefan; Weber, Cornelius; Wermter, Stefan.
Proceedings of the 22nd International Conference on Artificial Neural Networks (ICANN2012). red. / Alessandro E.P. Villa; Włodzisław Duch; Péter Érdi; Francesco Masulli; Günther Palm. Bind 7552 Springer, 2012. s. 555-562 (Lecture Notes in Computer Science).

Publikation: Konference artikel i Proceeding eller bog/rapport kapitel › Konferencebidrag i proceedings › Forskning › peer review

TY - GEN

T1 - Adaptive Learning of Linguistic Hierarchy in a Multiple Timescale Recurrent Neural Network

AU - Heinrich, Stefan

AU - Weber, Cornelius

AU - Wermter, Stefan

PY - 2012/9/1

Y1 - 2012/9/1

N2 - Recent research has revealed that hierarchical linguistic structures can emerge in a recurrent neural network with a sufficient number of delayed context layers. As a representative of this type of network the Multiple Timescale Recurrent Neural Network (MTRNN) has been proposed for recognising and generating known as well as unknown linguistic utterances. However the training of utterances performed in other approaches demands a high training effort. In this paper we propose a robust mechanism for adaptive learning rates and internal states to speed up the training process substantially. In addition we compare the generalisation of the network for the adaptive mechanism as well as the standard fixed learning rates finding at least equal capabilities.

AB - Recent research has revealed that hierarchical linguistic structures can emerge in a recurrent neural network with a sufficient number of delayed context layers. As a representative of this type of network the Multiple Timescale Recurrent Neural Network (MTRNN) has been proposed for recognising and generating known as well as unknown linguistic utterances. However the training of utterances performed in other approaches demands a high training effort. In this paper we propose a robust mechanism for adaptive learning rates and internal states to speed up the training process substantially. In addition we compare the generalisation of the network for the adaptive mechanism as well as the standard fixed learning rates finding at least equal capabilities.

KW - Adaptive Learning

KW - MTRNN

KW - Language

U2 - 10.1007/978-3-642-33269-2_70

DO - 10.1007/978-3-642-33269-2_70

M3 - Article in proceedings

VL - 7552

T3 - Lecture Notes in Computer Science

SP - 555

EP - 562

BT - Proceedings of the 22nd International Conference on Artificial Neural Networks (ICANN2012)

A2 - Villa, Alessandro E.P.

A2 - Duch, Włodzisław

A2 - Érdi, Péter

A2 - Masulli, Francesco

A2 - Palm, Günther

PB - Springer

ER -

Heinrich S, Weber C, Wermter S. Adaptive Learning of Linguistic Hierarchy in a Multiple Timescale Recurrent Neural Network. I Villa AEP, Duch W, Érdi P, Masulli F, Palm G, red., Proceedings of the 22nd International Conference on Artificial Neural Networks (ICANN2012). Bind 7552. Springer. 2012. s. 555-562. (Lecture Notes in Computer Science). doi: 10.1007/978-3-642-33269-2_70

Adaptive Learning of Linguistic Hierarchy in a Multiple Timescale Recurrent Neural Network

Abstract

Adgang til dokumentet

Fingeraftryk

Citationsformater