Machine Learning for Science and Society

A Special Issue of Machine Learning

Cynthia Rudin and Kiri L. Wagstaff, guest editors

Reviewer Guidelines

• Include a clear discussion of impact, to science or society, of an ML innovation. Here we do not mean discussing potential impact as motivation. The paper should talk about actual, measurable impact.
• Measure impact (not just performance) quantitatively where possible.
• Have made an effort to make results accessible to a domain expert, in ways that domain experts can understand.
• Be novel in some way, in terms of either the application area or the means by which impact is achieved.

• Empirical comparisons of algorithms with little or no discussion of what the results mean.
• Papers where the main focus is to introduce an algorithm, rather than to solve a real problem (although it is perfectly fine to introduce a new algorithm as part of solving the problem).
• Papers that do not go beyond measuring performance on a test set to discuss impact. (Good performance on a test set in a new domain does not imply impact.)
• Papers that do not discuss paths to using the result or potential (or actual) deployment.
• Papers for which ML proves or verifies a fact that is already known, unless there is some quantifiable impact for verifying it more precisely.
• Extended versions of existing conference papers with little insight beyond what was in the conference version (these could be converted to regular submissions to the journal, though).

Review Form

1. Please summarize the paper's claims about impact achieved from a machine learning advance.
2. Impact: In your judgment, what is the significance of the impact described? Major or minor? Whom does it (or can it) affect? If the paper only discusses performance (e.g., accuracy on a task) but not the impact or potential impact of a system with that performance, it is not responsive to this call.
3. Novelty: Is this a novel application of ML, or is this a topic with an existing, established mechanism for success using ML? Is this a problem that truly can benefit from a new application of ML proposed by the authors?
4. Problem description: What is the problem domain? Is the problem described sufficiently to be understandable to those outside the problem domain?
5. Data preparation: Do the data preparation steps taken appear to be reproducible, given access to the data? (Note some data sources may be proprietary, and we do not expect authors to make data public.) Are they appropriate for the motivating problem and data available?
6. Machine learning: Is the machine learning component described in enough detail to understand what was done and how? Note that the machine learning technique need not be a novel advance for the field of machine learning, if it is applied in a novel way, or to a problem of unprecedented scale.
7. Results: Is the methodology clearly described and appropriate for the problem? Were problem-specific baselines and metrics employed? Are there additional experiments or metrics that should be conducted to evaluate impact? Are the results discussed and interpreted, including a discussion of the implications for the problem domain? Taken as a whole, do the results support the claims of impact?
8. Domain expert: Does the paper provide evidence from domain experts that the machine learning advance has (or can have) significant impact? Does the work described in the paper result in new knowledge or insights for the problem domain? Are most of the results written in such a way as to be interpretable to experts in that domain?
9. Infusion: Is there a clear description of how the machine learning advance was (or will be) incorporated into a deployed system? Are domain experts currently using the system? Have the authors provided a description of the steps needed for the technology to be adopted, if it isn’t already?
10. Lessons: Does the paper provide a summary of lessons learned from this experience that can benefit future machine learning researchers? What are they?
11. Overall judgment: Accept with minor revisions, Reject with encouragement to resubmit, or Reject.

• Impact (realized):
• Impact (potential):
• Clarity/writing/organization:
• References: