Peer Reviews

This chapter is based on the notes 2022-05-31_SPSE_Reviewing.pdf

There is no examiner above the professors! So how to evaluate work of professors and post-docs? (papers, grant proposals, etc.)

Reviews by research peers. This is everywhere:

  • conferences

  • journals

  • research funding

  • hiring decisions

  • salary decisions

Typical acceptance rates at conferences: 15% to 40%. (cf. conferences-computer.science and conference webpages for acceptance rates)

Note: ArXiv is not peer-reviewed, but:

  • quickly available possibility to ‘claim’ the research results

  • time stamp

  • versions

  • archived

  • avoids pay wall

Types of Reviews

  • open (software)

  • single anonymous (papers, grant proposals)

  • double anonymous (papers)

  • triple anonymous (papers)

  • signing (rare)

Depending on the context, reviewers know each other; at least the PC chairs/editors know the author.

Reviewing Systems

Screenshots EasyChair:

Roles in the Review Process

  • Chair (PC Chair, Editor)

  • Reviewers

  • Program committee (PC) members (at conference)

  • Subreviewer

  • Moderators (at larger conferences)

Organisation hierarchy: chair, moderators, pc-members.

Conference Schedule

This chapter is based on the notes 2022-06-14_SPSE_Reviewing-Part2.pdf

  • PC setup -> Abstract submission

  • Bidding/Conflicts of interest -> Full-paper submission

  • Review assignments

  • Review submission

  • PC discussion (together) -> Rebuttal

  • Author notification

  • PC is done, hand publication process to publisher -> Camera-ready submission

  • Publisher processes submissions, provides publications

Review Structure

Common scores in review:

  • Acceptance:

    • strong accept (3)

    • accept (2)

    • weak accept (1)

    • borderline (0)

    • weak reject (-1)

    • reject (-2)

    • strong reject (-3)

  • Own expertise/confidence:

    • eXpert (x)

    • knowledgeable (y)

    • unfamiliar (z)

  • Alternative schema for confidence:

    • expert (5)

    • high (4)

    • medium (3)

    • low (2)

    • none (1)

Review content:

  1. Summarize paper

  2. Explain contribution

  3. Evaluate in detail

    1. List strong points (pros) and weak points (cons)

    2. Evaluate criteria (may be given by conference):

      • Soundness (research methods)

      • Significance (scope, importance)

      • Novelty (related work)

      • Verifiability (reproducibility)

      • Presentation (understandability, structure, figures, form)

  4. Add comments for the authors

    1. Spelling, typos (not your job, though)

    2. Improvements (how to proceed with project, where to publish better, restructure, RQs)

  5. (If there’s a rebuttal): Questions to the authors

  6. Add confidential comments for the PC (plagiarism, opinion, explanations that authors shouldn’t see)

What if review assignment does not fit?

  • PC chair has to find more reviewers

  • Journal editor has to find fitting reviewers

Review Principles

  • Positive language

  • Constructive

  • Honest

  • Ethical and moral

Example: Review guidelines of ICSE22

Advantages of Doing Reviews

  • Broaden your own horizon

  • See how good papers are written

  • Experience what reviewers focus on

  • Training in writing reviews, become more efficient

Rebuttals

  • Reviews are sent to authors

  • Authors get short deadline to answer the reviews

  • PC discussion afterwards, only