Reviewer Instructions

Timeline

  • Paper submission: Sept 1, 2022
  • Reviewer and AC assignment: Sept 5, 2022
  • Reviewing period: Sept 6, 2022 - Sept 22, 2022
  • Author response period: Sept 30, 2022 - Oct 5, 2022
  • Discussion period: Oct 6, 2022 - Oct 11, 2022
  • Meta-reviewing period: Oct 12, 2022 - Oct 14, 2022
  • Decision released: Oct 23, 2022

Expected Workload

We’ve tried our best to grow a large pool of reviewers, so we expect that each reviewer/AC will be assigned a reasonable number of submissions. Please help us keep this number small by completing your assigned reviews on time -- and if you can complete them early, all the better!

  • We expect each reviewer to be assigned ~3 submissions (and no more than 5 submissions) from the proceedings track (9 pages) or extended abstract track (4 pages). A reviewer may be assigned submissions from both tracks.
  • We expect each AC to be assigned 6-10 submissions, which may be all from the proceedings track (9 pages) or extended abstract track (4 pages). An AC will only be assigned submissions from one track; you may specify your preference in the questionnaire (we will try our best to accommodate your preference but it is not guaranteed).

Confidentiality

All aspects of the review process should be confidential. Do not discuss or distribute the submissions or reviews or use any ideas from the submissions in your own work until they are publicly available.

Reviewer Assignment Process

Authors will be asked to label each paper with a set of data modalities and topic areas, and each reviewer/AC has been asked to list their data modalities and topic areas of expertise. This information will be used to assign reviewers/ACs to the most relevant papers possible.

Expertise Alignment & Conflicts

Once you receive your assigned submissions (Sept 5, 2022), please check ASAP that your assigned papers do not represent conflicts of interest and that you feel confident in serving as a reviewer/AC for a paper in this topic area. You should not recognize any paper you are reviewing as work done by someone you work closely with. If you notice a conflict or if you are not confident that you can review the submission, report it immediately to the organizers.

Format & Anonymity Violations

All submissions should be anonymous and formatted using the ML4H 2022 LaTeX template. Proceedings track papers should have their main content limited to 9 pages, and extended abstracts should be limited to 4 pages. Please note the page limit includes figures and text but excludes references and appendices. However, reviewers should not feel obligated to read any supplementary material. Your time is precious!

Our reviewing process is double-blind. The authors do not know the reviewers' identities, and the reviewers do not know the authors' identities. Of course, no process is perfect: the reviewers might be able to guess the authors' identities based on the data set or the approaches used, or by technical reports posted on the internet. As a reviewer, we expect you will not actively attempt to discover the identity of the author. We also caution you against assuming that you've discovered an author's identity based on a data set or approach: multiple independent inventions are common, and different groups often share data sets.

In general, we’d rather accept good work than nitpick about minor formatting issues (especially because some clinicians may not be familiar with LaTeX). However, major formatting violations (e.g. a 20-page journal-like paper) would be grounds for rejection. If you find a major violation, please report it to the organizers.

What counts as a good ML4H Paper/Abstract?

Similar to prior years, we offer two submission tracks at ML4H: a formal proceedings track as well as an extended abstract track. Our intention of providing two tracks is to establish ML4H as a strong venue for publishing excellent work at the intersection of machine learning and health while also providing a forum for insightful discussion of creative and probing ideas. While both tracks require high quality submissions, the criteria for acceptance are distinct -- the proceedings track is meant for polished work with technical sophistication and clear health impact, while the extended abstract track is for non-traditional research artifacts and thus should be judged on their likelihood to lead to a good discussion at the workshop. Please refer to the writing guidelines document for exemplary submissions in each track.

When should a paper switch to the abstract track?

Works submitted to the Proceedings track can be considered for conversion to the extended abstract track. If you are reviewing a paper and think that a shortened 4-page version of the work would be strong enough to warrant consideration in the extended abstract track, make sure to select that option in the reviewer form on OpenReview.

Keep in mind that the primary goal of the extended abstract track is to generate productive and interesting discussion amongst conference attendees. Creative, insightful, and/or potentially divisive contributions (even if less fully developed or not performant) make great additions to the abstract track. As a result, an extended abstract may not necessarily meet the same level of technical sophistication as a paper, as long as it meets the criteria of generating productive / interesting discussions.

For Reviewers

Main Responsibilities

  1. By Sept 1, 2022: Fill out the reviewer questionnaire
  2. Sept 5, 2022: Check assigned submissions for expertise alignment and possible conflicts
  3. Sept 6, 2022 - Sept 22, 2022: Write reviews
  4. Oct 6, 2022 - Oct 11, 2022: Participate in discussions with authors and area chairs

Goals of the Reviewing Process

There are two complementary goals of the reviewing process. One goal is to provide actionable, constructive, and respectful feedback to the authors that will help them improve the work, both through positive recognition of the paper’s strengths and targeted commentary on opportunities for improvement. The other goal is to help meta-reviewers and organizers make decisions such that the highest-impact work is identified and accepted.

This year, ML4H also features a dedicated consensus-building discussion period. In this period, we expect all reviewers to engage with one another and their meta-reviewer to present a unified consensus. We expect reviewers to consider the author’s rebuttal, align with other reviewers on commonly identified strengths and weaknesses, and update their review in light of these discussions. The goal of this period is not to induce homogeneity, but rather to build consensus and provide a unified recommendation to the author. If you feel after this period that your beliefs about the quality of the work are not reflected by other reviewers, it is acceptable and encouraged to present these beliefs in your final review, provided you have made a good-faith effort to engage with other reviewers and have been open to other perspectives or arguments. Meta-reviewers are encouraged to seek consensus in this period, so you will be asked by your meta-reviewer to justify your stance if you disagree with the majority of reviewers, and failing to do so may result in your review not being featured as heavily in the meta-reviewers overall recommendation.

How can I write a good review?

There are numerous external resources that provide helpful advice on writing reviews. To get a sense of what a good review might look like, the 2017 ACL PC chairs blog contains some useful advice. To understand the difference between a negative and a constructive review, check out the review content section of the NeurIPS 2021 guidelines (for inspiration only; follow the ML4H reviewer guidelines for any specific instructions). To assess the healthcare-specific aspects of a paper/abstract, Nature has a small paragraph on how to assess the authors’ contributions.

For Area Chairs

Main Responsibilities

  1. By Sept 1, 2022: Fill out the area chair questionnaire
  2. Sept 5, 2022 - Sept 22, 2022: Check assigned submissions for expertise alignment and possible conflicts. Notify the organizers ASAP if you are not comfortable meta-reviewing any assigned papers. a
    • You are not responsible for handling reviewer reassignment due to conflicts or expertise alignment. Reviewers have been instructed to report such cases directly to the organizers.
  3. Sept 23, 2022 - Sept 29, 2022: Examine all reviews submitted across your papers and flag any that should be replaced with an emergency review due to poor quality.
    • Identifying a review that need be replaced should not in general require reading the paper or forming your own opinion of the work; instead, this is an opportunity to flag reviews that are unambiguously lacking sufficient content to motivate a recommendation (e.g., the review is one sentence, and a score), lacking sufficient expertise to motivate trust (e.g., the reviewer states “I don’t know enough about this to review”), or are incomprehensible or severely inconsistent (e.g., the free-text says that they love the work, with no cons identified, but the score is a strong reject). Flagged reviews will then be replaced by an emergency review from a new reviewer.
  4. Oct 6, 2022 - Oct 11, 2022: Lead discussions with authors and reviewers.
    • Drive the formal discussion period and align reviewers to form updated, consensus reviews for their submissions. Reviewers should confront any inter-reviewer disagreements, respond to any points raised in author rebuttals, and be encouraged to update their scores if other reviewers or authors address previously identified major concerns during this process. The final output of this process should be a set of unified reviews. Note that the intent of this period is not to provide homogenous reviews, but rather to ensure that all reviews improve by taking into account the perspectives of other reviewers and that (insofar as is possible) reviewers can align on a consensus decision.
  5. Oct 12, 2022 - Oct 14, 2022: Write meta-reviews

Goals for Meta-reviewing Process

  1. Drive the consensus-building discussion period amongst reviewers such that reviewers both
    1. Align on an overall recommendation, such that (ideally) all final reviews and meta-reviews present a unified recommendation and aligned set of scores.
    2. Appropriately consider and address any new information offered in the author response.
  2. Write a meta-review that should:
    1. Summarize the opinions of the reviewers to aid in final decision making.
    2. Provide a summary of the reviews’ salient points for the author, focusing specifically on what critical aspects would be needed to improve the submission for acceptance in the future and/or what key strengths led the meta-reviewer to recommend acceptance.
  3. Flag low-quality reviews to give reviewers feedback on their review quality.
  4. Highlight high-quality papers and reviews for consideration for paper awards (more information coming soon) and Top Reviewer awards.