Measuring Catastrophic Forgetting in Visual Question Answering

Claudio Greco; Barbara Plank; Raquel Fernandez; Raffaella Bernardi

doi:10.1007/978-981-15-9323-9_35

Measuring Catastrophic Forgetting in Visual Question Answering

Claudio Greco, Barbara Plank, Raquel Fernandez, Raffaella Bernardi

Research output: Conference Article in Proceeding or Book/Report chapter › Book chapter › Research › peer-review

Abstract

Catastrophic forgetting is a ubiquitous problem for the current generation of Artificial Neural Networks: When a network is asked to learn multiple tasks in a sequence, it fails dramatically as it tends to forget past knowledge. Little is known on how far multimodal conversational agents suffer from this phenomenon. In this paper, we study the problem of catastrophic forgetting in Visual Question Answering (VQA) and propose experiments in which we analyze pairs of tasks based on CLEVR, a dataset requiring different skills which involve visual or linguistic knowledge. Our results show that dramatic forgetting is at place in VQA, calling for studies on how multimodal models can be enhanced with continual learning methods.

Original language	English
Title of host publication	Tenth International Workshop on Spoken Dialogue Systems Technology (IWSDS) 2019 : Lecture Notes in Electrical Engineering book series (LNEE, volume 714)
Number of pages	387
Publisher	Springer
Publication date	2019
Pages	381
DOIs	https://doi.org/10.1007/978-981-15-9323-9_35
Publication status	Published - 2019

Series	Lecture Notes in Electrical Engineering
Volume	714
ISSN	1876-1100

Keywords

Catastrophic Forgetting
Artificial Neural Networks
Multimodal Conversational Agents
Visual Question Answering
Continual Learning Methods

Access to Document

10.1007/978-981-15-9323-9_35

Cite this

@inbook{1aa65b20105a4bf4a54d8d10a1c993ab,

title = "Measuring Catastrophic Forgetting in Visual Question Answering",

abstract = "Catastrophic forgetting is a ubiquitous problem for the current generation of Artificial Neural Networks: When a network is asked to learn multiple tasks in a sequence, it fails dramatically as it tends to forget past knowledge. Little is known on how far multimodal conversational agents suffer from this phenomenon. In this paper, we study the problem of catastrophic forgetting in Visual Question Answering (VQA) and propose experiments in which we analyze pairs of tasks based on CLEVR, a dataset requiring different skills which involve visual or linguistic knowledge. Our results show that dramatic forgetting is at place in VQA, calling for studies on how multimodal models can be enhanced with continual learning methods.",

keywords = "Catastrophic Forgetting, Artificial Neural Networks, Multimodal Conversational Agents, Visual Question Answering, Continual Learning Methods, Catastrophic Forgetting, Artificial Neural Networks, Multimodal Conversational Agents, Visual Question Answering, Continual Learning Methods",

author = "Claudio Greco and Barbara Plank and Raquel Fernandez and Raffaella Bernardi",

year = "2019",

doi = "10.1007/978-981-15-9323-9_35",

language = "English",

series = "Lecture Notes in Electrical Engineering",

publisher = "Springer",

pages = "381",

booktitle = "Tenth International Workshop on Spoken Dialogue Systems Technology (IWSDS) 2019",

address = "Germany",

}

Greco, C, Plank, B, Fernandez, R & Bernardi, R 2019, Measuring Catastrophic Forgetting in Visual Question Answering. in Tenth International Workshop on Spoken Dialogue Systems Technology (IWSDS) 2019: Lecture Notes in Electrical Engineering book series (LNEE, volume 714). Springer, Lecture Notes in Electrical Engineering, vol. 714, pp. 381. https://doi.org/10.1007/978-981-15-9323-9_35

Measuring Catastrophic Forgetting in Visual Question Answering. / Greco, Claudio; Plank, Barbara; Fernandez, Raquel et al.
Tenth International Workshop on Spoken Dialogue Systems Technology (IWSDS) 2019: Lecture Notes in Electrical Engineering book series (LNEE, volume 714). Springer, 2019. p. 381 (Lecture Notes in Electrical Engineering, Vol. 714).

Research output: Conference Article in Proceeding or Book/Report chapter › Book chapter › Research › peer-review

TY - CHAP

T1 - Measuring Catastrophic Forgetting in Visual Question Answering

AU - Greco, Claudio

AU - Plank, Barbara

AU - Fernandez, Raquel

AU - Bernardi, Raffaella

PY - 2019

Y1 - 2019

N2 - Catastrophic forgetting is a ubiquitous problem for the current generation of Artificial Neural Networks: When a network is asked to learn multiple tasks in a sequence, it fails dramatically as it tends to forget past knowledge. Little is known on how far multimodal conversational agents suffer from this phenomenon. In this paper, we study the problem of catastrophic forgetting in Visual Question Answering (VQA) and propose experiments in which we analyze pairs of tasks based on CLEVR, a dataset requiring different skills which involve visual or linguistic knowledge. Our results show that dramatic forgetting is at place in VQA, calling for studies on how multimodal models can be enhanced with continual learning methods.

AB - Catastrophic forgetting is a ubiquitous problem for the current generation of Artificial Neural Networks: When a network is asked to learn multiple tasks in a sequence, it fails dramatically as it tends to forget past knowledge. Little is known on how far multimodal conversational agents suffer from this phenomenon. In this paper, we study the problem of catastrophic forgetting in Visual Question Answering (VQA) and propose experiments in which we analyze pairs of tasks based on CLEVR, a dataset requiring different skills which involve visual or linguistic knowledge. Our results show that dramatic forgetting is at place in VQA, calling for studies on how multimodal models can be enhanced with continual learning methods.

KW - Catastrophic Forgetting

KW - Artificial Neural Networks

KW - Multimodal Conversational Agents

KW - Visual Question Answering

KW - Continual Learning Methods

KW - Catastrophic Forgetting

KW - Artificial Neural Networks

KW - Multimodal Conversational Agents

KW - Visual Question Answering

KW - Continual Learning Methods

U2 - 10.1007/978-981-15-9323-9_35

DO - 10.1007/978-981-15-9323-9_35

M3 - Book chapter

T3 - Lecture Notes in Electrical Engineering

SP - 381

BT - Tenth International Workshop on Spoken Dialogue Systems Technology (IWSDS) 2019

PB - Springer

ER -

Greco C, Plank B, Fernandez R, Bernardi R. Measuring Catastrophic Forgetting in Visual Question Answering. In Tenth International Workshop on Spoken Dialogue Systems Technology (IWSDS) 2019: Lecture Notes in Electrical Engineering book series (LNEE, volume 714). Springer. 2019. p. 381. (Lecture Notes in Electrical Engineering, Vol. 714). doi: 10.1007/978-981-15-9323-9_35

Measuring Catastrophic Forgetting in Visual Question Answering

Abstract

Keywords

Access to Document

Fingerprint

Cite this