ML4H 2023 invites submissions describing innovative research that lies in the broad purview of Machine Learning for Health. Authors are invited to submit work on relevant problems in a variety of health-related disciplines including healthcare, biomedicine, and public health. This year, ML4H 2023 will accept submissions for two distinct tracks: the Proceedings track, for formal archival publications, and the non-archival Findings track.
In response to the growing ML4H community, ML4H has transitioned into a standalone symposium rather than a NeurIPS-affiliated workshop. This event represents a continuation of prior ML4H workshops/symposiums (2016, 2017, 2018, 2019, 2020, 2021, 2022) and will continue to be held in December directly before NeurIPS. ML4H 2023 will feature:
- Author Mentorship Program
- Career Mentorship Program
- Best Proceedings paper, Findings paper, Thematic paper awards
- Best Newcomer Paper award
- Thematic sessions on 1) health equity & global health and 2) generative AI for health.
If you are interested in being a reviewer, please reach out to us! We will be hosting a Reviewer Mentorship Program as well as Top Reviewer Awards.
Quick Submission Instructions
Submission Site: https://openreview.net/group?id=ML4H/2023/Symposium
ML4H 2023 LaTeX templates:
The abstract registration deadline is September 7th 11:59 PM AoE. For this deadline, the title, author list, abstract, track, subject area, and data modality should be submitted. Edits to these metadata after the abstract deadline will not be considered in the reviewing process. The submission deadline for ML4H 2023 has been extended and submissions are now due on September 12th 11:59 PM AoE in the form of anonymized PDF files. All submissions will be managed through the OpenReview system. Submissions must be formatted using the ML4H 2023 LaTeX templates with proper anonymization. Gross violations of formatting guidelines, malformed, non-blinded, non-health-related, or grossly insufficient works may be desk rejected by the organizing committee without undergoing additional review.
Aug 1st: Submission site opens
Sep 7th AoE: Abstract submission deadline
(Extended) Sep 12th AoE: Submission deadline
Oct 13th : Reviews available; Author response period starts
Oct 18th : Author response period ends
Oct 21st: Recommended deadline for uploading author response
Nov 1st: Decisions released
Nov 8th: Early Career Travel Award Deadline
Nov 15th [tentative]: Camera-ready deadline (Instructions)
Dec 10th: In-person event
Clarifications about author response periodWe would like to emphasize that a response is not required, and should only be submitted to directly address reviewers' questions or when there is a need for clarification. Responses can be submitted on OpenReview by posting "Official Comments" until Oct 31st. We recommend authors upload responses by Oct 21st to allow for full consideration by reviewers and meta-reviewers. The organizing committee will release decisions on Nov 1st and may consult author responses submitted after Oct 21st in exceptional cases.
As part of the submission, authors will indicate whether they would like the submission to be in the Proceedings track or the Findings track. Authors are also required to fill out a submission form that will be visible to reviewers to help them assess the work.
Data and Code: We encourage anonymized code and data submissions (if it can be made available with appropriate approval and guidelines) as supplemental material during review. If you are not sharing code, you must explicitly state this in the submission checklist. If your paper is accepted, we encourage public sharing of your code and/or data for the camera-ready version of the paper.
Ethics Board Approval: If your research requires IRB (or equivalent) approval or has been evaluated by your IRB as Not Human Subject Research, then for the camera-ready version of the paper, you must provide relevant information. At the time of submission for review, to preserve anonymity, it suffices to include a statement that relevant ethics approval information will be provided if the paper is accepted. If your research does not require IRB approval, please explicitly state this to be the case and provide a justification in the submission checklist.
ML4H 2023 will feature two submission tracks: a full, archival Proceedings track and a non-archival Findings track. Submissions to either track will undergo double-blind peer review. Accepted submissions to both tracks will be featured at the event's poster session.
Accepted works for both tracks will be chosen based on their technical merit and contribution to the event. More details on how to write an excellent ML4H full paper or findings paper can be found here. Below are the salient differences between both tracks.
(A) Proceedings Track
Excellent ML4H Proceedings papers should be compelling, cohesive works with a high degree of technical sophistication as well as clear and high-impact relevance to health. Accepted proceedings papers will be published in the Proceedings for Machine Learning Research (PMLR). Full proceedings papers can be up to 8 pages at submission (excluding references and appendices). If your submission is accepted, you will be allowed 1 additional content page for the camera-ready version.
Dual submission policy: Papers that are submitted to the ML4H proceedings track cannot be already published or under review in any other archival venue. Similarly, papers published to the ML4H proceedings may not be published again later at any other venue.
(B) Findings Track
An excellent findings paper is one that leads to insights at the event through interaction with other attendees. This can be through presenting new ideas/ways of thinking, leading to insightful discussion and feedback, dissemination of new valuable resources, or enabling new opportunities for collaborations. We also especially solicit “non-traditional research artifacts” as submissions to the findings track, such as papers highlighting novel datasets, insightful negative results, exciting preliminary results that warrant rapid dissemination, reproducibility studies, and opinion pieces or critiques.
Findings papers can be up to 4 pages at submission (excluding references and appendices), though additional information not critical for understanding the work can be included in an appendix without penalty (reviewers will review the work based predominantly on the main text). Findings papers will not appear in the ML4H proceedings, but upon acceptance, we invite (but do not require) authors to submit their findings (no page limit) to the ML4H arxiv.org index.
Authors of accepted findings papers (non-archival submissions) retain full copyright of their work, and acceptance of such a submission to ML4H 2023 does not preclude publication of the same material in another archival venue (e.g. journal or conference). Furthermore, findings submissions that are under review or have been recently published in a conference or a journal are allowed; if this is the case, authors should clearly state any overlapping published or submitted work at the time of submission (in the confidential comments), and must ensure that they are not violating any other venue’s dual submission policies.
Proceedings track submissions that are not accepted will automatically be considered for the Findings track. Findings track submissions cannot be considered for the Proceedings track. Decisions for both tracks will be released simultaneously.
Submitted papers should describe innovative machine learning research focused on relevant problems in health-related disciplines. This can mean new models, new algorithms, new datasets, and/or new applications. Topics of interest include but are not limited to data integration, temporal models, deep learning, semi-supervised learning, reinforcement learning, transfer learning, few/zero shot learning, learning from missing or biased data, learning from non-stationary data, causality, model biases, model evaluation, model criticism, model interpretability, model deployment, human-computer interaction, and privacy/security. We especially encourage submissions relevant to our two themes, Health Equity & Global Health and Generative AI (see below).
ML4H 2023 will feature two themes: 1) machine learning for health equity and global health, and 2) generative AI for health. We encourage technical, applied and socio-technical work. When submitting work to the symposium, authors will have the option to indicate whether or not their work fits within one or both themes (the two themes are not exclusive). Any work within the broad purview of machine learning for health will be considered for ML4H, regardless of its suitability to the themes, and relevance to the themes will have no impact on overall acceptance to the symposium. However, only papers submitted to the thematic submissions will be eligible for consideration for the Best Thematic Paper award(s).
Theme 1: Machine learning for health equity and global health
Machine learning has the potential to both address and exacerbate gaps in health equity. Despite the overwhelming evidence that these techniques can be beneficial to both health equity and global health, there remains a pressing need for methods to ensure the fairness & robustness of ML for health models (for example, to ensure unbiased performance across subpopulations). This theme solicits foundational, applied and socio-technical ML work designed to promote health equity and global health. Example topics within this area include but are not limited to:
- ML applied to health-related sustainable development goals
- Methods to ensure model robustness to equity-related distribution shifts
- Development of representative ML for health datasets
- Socio-technical insights and case-studies at the intersection of ML, health equity and global health (e.g., compelling qualitative studies on how clinicians use ML systems or how patients perceive ML interventions in health).
Example topical papers:
- Abebe et al. “Using search queries to understand health information needs in Africa”. ICWSM 2019.
- Chen et al. “Treating health disparities with artificial intelligence”. Nature Medicine, 2020.
- Ganapathi et al. “Tackling bias in AI health datasets through the STANDING Together initiative”. Nature Medicine, 2022.
- Mate et al. “Field study in deploying restless multi-armed bandits: Assisting non-profits in improving maternal and child health”. AAAI 2022.
- Pfohl et al. “Creating Fair Models of Atherosclerotic Cardiovascular Disease Risk”. AIES 2019.
- Pierson et al. “An algorithmic approach to reducing unexplained pain disparities in underserved populations”. Nature Medicine, 2021.
- Obermeyer et al. “Dissecting racial bias in an algorithm used to manage the health of populations”. Science, 2019.
- Seyyed-Kalantari et al. “Underdiagnosis bias of artificial intelligence algorithms applied to chest radiographs in under-served patient populations”. Nature Medicine, 2021.
- Zhao et al. “Machine learning and dengue forecasting: Comparing random forests and artificial neural networks for predicting dengue burden at national and sub-national scales in Colombia”. PLOS Neglected Tropical Diseases, 2020.
Theme 2: Generative AI for health: the road ahead
Generative AI, and large language models (LLMs) in particular, have transformed the landscape of AI, creating many new opportunities as well as challenges. This theme solicits work that explores the use and development of generative AI for key problems in health, including healthcare, biomedicine, and public health. Examples topics within this area include but are not limited to:
- Methods for training, evaluating, and benchmarking generative AI model on health-related tasks
- Evaluating the safety and robustness of generative AI in health
- Regulatory and implementation pathways for generative AI in health
- Applications of generative AI in health
Example topical papers:
- Agrawal et al. "Large language models are zero-shot clinical information extractors". EMNLP 2022.
- Lehman et al. "Do We Still Need Clinical Language Models?" CHIL 2023.
- Miura et al. "Improving factual completeness and consistency of image-to-text radiology report generation". NAACL 2021.
- Moor et al. "Foundation models for generalist medical artificial intelligence". Nature 2023.
- Nori et al. "Capabilities of GPT-4 on Medical Challenge Problems". arXiv 2023.
- Yang et al. "A large language model for electronic health records". npj Digital Medicine 2022.
Author Response Period
Initial reviews will be released on
October 9thOctober 13th. From October 9thOctober 13th to October 14thOctober 31st, 11:59 PM AoE, authors can submit responses to the reviews. Authors are encouraged to upload responses by Oct 21st for full consideration by the reviewers and meta-reviewers. Author responses may address any aspect of the reviews, including by adding specific types of new experimental results as requested by the reviewers, e.g. missing baselines. No conceptual changes to the original formulation are allowed beyond clarifications. After the author response period, the reviewers and meta-reviewer will discuss and reach a final decision for the papers. We reserve the right to solicit additional reviews after the author response period in the rare case that there are not sufficient high quality reviews to make a final decision.
Reviewer Discussion Period
During the reviewer discussion period, reviewers and meta-reviewers will discuss the paper, their reviews, and the author response. This process aims at seeking a consensus between reviewers and meta-reviewers. We ask reviewers to change their initially submitted review scores and recommendations during the discussion period, if applicable, and state this in the discussion along with justification. Discussions will take place within OpenReview by using the comment function in each respective submission and should remain double-blind, i.e. comments may not de-anonymize the authors or reviewers.
In general, these discussions will be between reviewers and meta-reviewers only. However, when further clarifications from the authors are necessary, reviewers may reach out to authors through OpenReview comments. It is only in response to such direct questions that authors should add comments beyond their author response, and said comments should be limited to directly answering the asked question. The reviewer discussion period formally ends on
October 14 October 23 11:59 PM AoE , but discussions may be finalized earlier.
This year, ML4H is offering an Author Mentorship Program which focuses on pairing less experienced authors with senior researchers to provide feedback on their paper submission, with the overall goal of improving submission quality and fostering future collaboration. More details on the Author Mentorship Program can be found at: https://ml4health.github.io/2023/author_mentorship.html
- Application form for mentees : https://forms.gle/eRwy68qSW4KVwg3U7
- Application form for mentors : https://forms.gle/RfL36L78SfQtA1Sz9
ML4H will also be offering a Reviewer Mentorship Program which will take place immediately after the submission deadline and its aim is to train junior reviewers, foster new connections and relationships in the ML4H community, and ultimately improve the quality of the review process. Closer to the symposium, ML4H will also be offering a Career Mentorship Program (coming soon). We especially encourage less experienced authors and reviewers and participants from underrepresented backgrounds to sign up as mentees, as well as more senior community members to serve as mentors for these programs. More details for these programs will be available soon, but if you are interested please drop us your email so we can reach out to you when the time comes!
This year, ML4H is also offering a Career Mentorship Program, where mentors will be paired with mentees to discuss career-related topics (e.g., developing a long-term research plan, doing healthcare research in industry, work-life balance). Mentors and mentees will be matched based on time availability and selected topics of expertise/interest for providing and receiving mentorship, respectively. To learn more, check out https://ml4health.github.io/2023/career_mentorship.html, where you will also find the forms to apply to the program!
To promote community interaction, at least one presenting author of accepted works must register for the event. Registration details are forthcoming.