Entity Decisions in Neural Language Modelling: Approaches and Problems

Research output: Conference Article in Proceeding or Book/Report chapterArticle in proceedingsResearchpeer-review

Abstract

We explore different approaches to explicit entity modelling in language models (LM). We independently replicate two existing models in a controlled setup, introduce a simplified variant of one of the models and analyze their performance in direct comparison. Our results suggest that today's models are limited as several stochastic variables make learning difficult. We show that the most challenging point in the systems is the decision if the next token is an entity token. The low precision and recall for this variable will lead to severe cascading errors. Our own simplified approach dispenses with the need for latent variables and improves the performance in the entity yes/no decision. A standard well-tuned baseline RNN-LM with a larger number of hidden units outperforms all entity-enabled LMs in terms of perplexity.
Original languageEnglish
Title of host publicationProceedings of the Second Workshop on Computational Models of Reference, Anaphora and Coreference
Number of pages5
Publication date7 Jun 2019
ISBN (Print)978-1-948087-97-1
DOIs
Publication statusPublished - 7 Jun 2019
Externally publishedYes

Fingerprint

Dive into the research topics of 'Entity Decisions in Neural Language Modelling: Approaches and Problems'. Together they form a unique fingerprint.

Cite this