SMART: Scaling Machine Assessments of Research Trustworthiness

"The Center for Open Science (COS), along with its collaborators, is building on the work completed during the DARPA-funded SCORE program, which demonstrated the potential of using algorithms to efficiently evaluate research claims at scale. The SCORE program supplements existing research evaluation methods, including human judgment, evidence aggregation, and systematic replication.

Through a grant from the Robert Wood Johnson Foundation (RWJF), COS, in partnership with researchers at the University of Melbourne and Pennsylvania State University, have begun the SMART project, which seeks to advance the development of automated confidence evaluation of research claims (press release here). SMART will extend the research work initiated by the SCORE program through conducting user research and generating additional data to improve the algorithm and human assessment approaches developed during the program. The first phase of SMART will engage researcher authors to voluntarily submit their papers for evaluation. Papers will undergo two types of assessment by our partners: (1) human assessment and (2) AI assessment. These assessments will be shared with the authors of the papers who will then have the opportunity to provide feedback on the assessments...."