What Factors Should Paper-Reviewer Assignments Rely On? Community Perspectives on Issues and Ideals in Conference Peer-Review

Terne Thorn Jakobsen; Anna Rogers

What Factors Should Paper-Reviewer Assignments Rely On? Community Perspectives on Issues and Ideals in Conference Peer-Review

Terne Thorn Jakobsen, Anna Rogers

Research output: Conference Article in Proceeding or Book/Report chapter › Article in proceedings › Research › peer-review

Abstract

Both scientific progress and individual researcher careers depend on the quality of peer review, which in turn depends on paper-reviewer matching. Surprisingly, this problem has been mostly approached as an automated recommendation problem rather than as a matter where different stakeholders (area chairs, reviewers, authors) have accumulated experience worth taking into account. We present the results of the first survey of the NLP community, identifying common issues and perspectives on what factors should be considered by paper-reviewer matching systems. This study contributes actionable recommendations for improving future NLP conferences, and desiderata for interpretable peer review assignments.

Original language	English
Title of host publication	Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies
Number of pages	14
Place of Publication	Seattle, United States
Publisher	Association for Computational Linguistics
Publication date	1 Jul 2022
Pages	4810-4823
Publication status	Published - 1 Jul 2022

Keywords

Peer Review
Paper-reviewer Matching
NLP Community Survey
Automated Recommendation Systems
Interpretable Peer Review Assignments

Access to Document

https://aclanthology.org/2022.naacl-main.354

Cite this

Thorn Jakobsen, T., & Rogers, A. (2022). What Factors Should Paper-Reviewer Assignments Rely On? Community Perspectives on Issues and Ideals in Conference Peer-Review. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (pp. 4810-4823). Association for Computational Linguistics. https://aclanthology.org/2022.naacl-main.354

Thorn Jakobsen, Terne ; Rogers, Anna. / What Factors Should Paper-Reviewer Assignments Rely On? Community Perspectives on Issues and Ideals in Conference Peer-Review. Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Seattle, United States : Association for Computational Linguistics, 2022. pp. 4810-4823

@inproceedings{c3027b3a72b642d7b6076cccfceb1356,

title = "What Factors Should Paper-Reviewer Assignments Rely On? Community Perspectives on Issues and Ideals in Conference Peer-Review",

abstract = "Both scientific progress and individual researcher careers depend on the quality of peer review, which in turn depends on paper-reviewer matching. Surprisingly, this problem has been mostly approached as an automated recommendation problem rather than as a matter where different stakeholders (area chairs, reviewers, authors) have accumulated experience worth taking into account. We present the results of the first survey of the NLP community, identifying common issues and perspectives on what factors should be considered by paper-reviewer matching systems. This study contributes actionable recommendations for improving future NLP conferences, and desiderata for interpretable peer review assignments.",

keywords = "Peer Review, Paper-reviewer Matching, NLP Community Survey, Automated Recommendation Systems, Interpretable Peer Review Assignments, Peer Review, Paper-reviewer Matching, NLP Community Survey, Automated Recommendation Systems, Interpretable Peer Review Assignments",

author = "{Thorn Jakobsen}, Terne and Anna Rogers",

year = "2022",

month = jul,

day = "1",

language = "English",

pages = "4810--4823",

booktitle = "Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies",

publisher = "Association for Computational Linguistics",

address = "United States",

}

Thorn Jakobsen, T & Rogers, A 2022, What Factors Should Paper-Reviewer Assignments Rely On? Community Perspectives on Issues and Ideals in Conference Peer-Review. in Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, Seattle, United States, pp. 4810-4823. <https://aclanthology.org/2022.naacl-main.354>

What Factors Should Paper-Reviewer Assignments Rely On? Community Perspectives on Issues and Ideals in Conference Peer-Review. / Thorn Jakobsen, Terne; Rogers, Anna.
Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Seattle, United States: Association for Computational Linguistics, 2022. p. 4810-4823.

Research output: Conference Article in Proceeding or Book/Report chapter › Article in proceedings › Research › peer-review

TY - GEN

T1 - What Factors Should Paper-Reviewer Assignments Rely On? Community Perspectives on Issues and Ideals in Conference Peer-Review

AU - Thorn Jakobsen, Terne

AU - Rogers, Anna

PY - 2022/7/1

Y1 - 2022/7/1

N2 - Both scientific progress and individual researcher careers depend on the quality of peer review, which in turn depends on paper-reviewer matching. Surprisingly, this problem has been mostly approached as an automated recommendation problem rather than as a matter where different stakeholders (area chairs, reviewers, authors) have accumulated experience worth taking into account. We present the results of the first survey of the NLP community, identifying common issues and perspectives on what factors should be considered by paper-reviewer matching systems. This study contributes actionable recommendations for improving future NLP conferences, and desiderata for interpretable peer review assignments.

AB - Both scientific progress and individual researcher careers depend on the quality of peer review, which in turn depends on paper-reviewer matching. Surprisingly, this problem has been mostly approached as an automated recommendation problem rather than as a matter where different stakeholders (area chairs, reviewers, authors) have accumulated experience worth taking into account. We present the results of the first survey of the NLP community, identifying common issues and perspectives on what factors should be considered by paper-reviewer matching systems. This study contributes actionable recommendations for improving future NLP conferences, and desiderata for interpretable peer review assignments.

KW - Peer Review

KW - Paper-reviewer Matching

KW - NLP Community Survey

KW - Automated Recommendation Systems

KW - Interpretable Peer Review Assignments

KW - Peer Review

KW - Paper-reviewer Matching

KW - NLP Community Survey

KW - Automated Recommendation Systems

KW - Interpretable Peer Review Assignments

M3 - Article in proceedings

SP - 4810

EP - 4823

BT - Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies

PB - Association for Computational Linguistics

CY - Seattle, United States

ER -

Thorn Jakobsen T, Rogers A. What Factors Should Paper-Reviewer Assignments Rely On? Community Perspectives on Issues and Ideals in Conference Peer-Review. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Seattle, United States: Association for Computational Linguistics. 2022. p. 4810-4823

What Factors Should Paper-Reviewer Assignments Rely On? Community Perspectives on Issues and Ideals in Conference Peer-Review

Abstract

Keywords

Access to Document

Fingerprint

Cite this