The second installment is out today and will discuss different peer review models, the effectiveness of peer review, and measures being taken to improve reviewer performance.
It is Peer Review Week and this year’s theme is “Research Integrity: Creating and supporting trust in research”. There are numerous experiments looking at traditional and more experimental models of peer review in an attempt to improve quality and ultimately increase trust. Some of these experiments and studies were presented at the most recent International Congress on Peer Review and Scientific Publication (PRC), which took place in Chicago Illinois from September 7th to 9th. As I mentioned yesterday, in celebration of Peer Review Week, I am recapping select sessions from the PRC. Today I highlight the papers that discuss different peer review models, the effectiveness of peer review, and measures being taken to improve reviewer performance.
Open peer review was a dominating theme, especially as it relates to improving trust and transparency. Some see trust and transparency as being in conflict with most current peer review practices, where the review process takes place in secret, with anonymous researchers providing closed analysis to the journal editor. There have been calls for a more open system for years and there are publishers, like Biomed Central and the British Medical Journal, that have been providing various forms of open peer review for a long time.
The plenary session that kicked off day 2, Improving the Research Culture to Increase Credibility of Research Findings, presented by Brian Nosek of the Open Science Foundation (OSF), was about improving trust and removing bias in peer review by involving reviewers earlier in the research process. Nosek stated that “current practice is sustained by dysfunctional incentives that prioritize publication of accuracy and transparency”. The OSF have advocated for a publishing format they called “Registered Reviews”, which emphasizes the purpose of the research and the protocols, rather than the outcomes. By bringing peer reviewers into the process early and having them evaluate the research plan keeps the focus on the quality of the research rather than the reported results. For example, today peer review takes place after the research has been conducted and the results are written up and submitted to a journal; this results in publication bias, which is the tendency to only publish confirmatory research. With Registered Reports, the peer review first takes place between the design phase and the actual collection and analysis of the data, and continues until publication. Nosek provided evidence that this method resulted in less publication bias and more rigorous review.
To assist with a more rigorous review Nosek presented a collaborative initiative called Systematizing Confidence in Open Research and Evidence (SCORE). The SCORE project is about developing artificial intelligence algorithms that flag potential issues in research papers, particularly in regard to replicability and reproducibility. These algorithms can be used to help peer reviewers and editors detect aspects of a research paper where they need to engage in closer examination. This initiative is still in early phases, with a prototype currently in testing and an open algorithm competition underway. Hundreds of researchers, mostly in social and behavioral sciences are participating, providing “claim extraction, credibility assessment, and reproducibility, robustness, and replication studies”.
The plenary session on day 3 continued the call for more openness in peer review. The presentation, Peer Review in the Age of Open Science, presented by Tony Ross-Hellauer, discussed the current state of open peer review by looking at uptake among journals and the varied forms that open peer review take. Ross Hellauer noted that editors and publishers show their commitment to openness through the policies they implement, and that current peer review processes are a black box because reviewers are anonymous, the reviews are not public, and editors select the reviewers they know rather than those who are most qualified.
Ross-Hellauer discussed a study by Wolfram et al, from 2020 which found steady growth in open peer review adoption, with adoption being most prevalent in medicine and science, and least prevalent in the humanities, social sciences and technology. The study showed that five publishers are responsible for 81% of journals that conduct open peer review. The study also showed that open peer review has been implemented in different ways, and that each methodology result in different levels of transparency. Ross-Hellauer conducted a systematic review in 2017 that attempted to document the variations of open peer review models, resulting in the collection of 122 definitions that were analyzed and categorized. What the study broadly found was that there are three primary aspects, open identities, open peer review reports, and community-wide participation. Benefits and drawbacks of each of these were also presented.
Open identities can increase accountability and transparency, expose conflict of interest, and reduce incivility in review reports. However, some reviewers feel that revealing identity might open them up to reprisal, especially for early career and under-represented researchers, which may I turn affect the quality and honesty of their opinions. The benefits of open peer review reports include an increase in review quality, participation in credit and reward systems like ORCID and Publons for peer review activity, and an increase in and higher utilization of training resources. It was also noted that predatory journals would find it difficult to provide open review reports because the additional work required is not financially viable for these illegitimate entities that simply collect publishing fees without providing any actual publishing services. Similar to drawbacks for open identities, some reviewers may be reticent to expose themselves to existing power dynamics, thus increasing refusal rates and the time it takes to get reviews submitted. Open community-wide participation shared the same benefits and drawbacks as open reports.
Another study by Ross-Hellauer is currently underway. It is a systematic review of the literature from 2017 to 2022 on the efficacy and uptake of the three core elements of open peer review just discussed: open identities, open reports and open participation. What has been discovered so far show that uptake of open peer review is growing, but still limited. There are many surveys and case studies, but they are limited in size and scope. Male and senior reviewers are more likely to agree to open identity, which further proves that power dynamics play an important role in academics and science. Open reports do not slow down the process, but this is not consistent across all disciplines. There is little evidence yet that show open peer review improves quality, and there is still not much data on open participation (community review). I look forward to following the results of this study over the next couple of years!
Openness in peer review may lead to more transparency, and thus trust in research. Concerns that I’ve heard raised about opening up the peer review process, in particular when reviewer identity is open and review comments are available during the review process, is that early reviews may affect the thinking of later reviewers, and that higher status reviewers may influence lower status reviewers, thus biasing the peer review process. A study on “herding”, which is the tendency of people to do what others are doing rather than choosing an independent course of action, tested these concerns. The presentation, Effect of Positive v Negative Reviewer-Led Discussion on Herding and acceptable Rates of Papers Submitted to a Machine Learning Conference, presented by Nihar Shah, reviewed an experiment carried out at the International Conference on Machine Learning (ICML) where reviewers openly discussed papers submitted to the conference. The process is different than the standard scholarly journal peer review process in that a paper submitted to the ICML undergoes an initial review, then an author rebuttal, then a discussion among the reviewers, then a decision. In the experiment the researchers took papers that had divergent initial reviews, then during the discussion phase one set of reviewers began by looking at the negative reviews, while the other set started with the positive reviews. It is a complex and interesting study with various caveats about the design of the discussions and how they compensated for the influence of the discussion initiators. In the end the results showed no statistically significant herding affect. As mentioned, the peer review process used was not the same as the standard scholarly journal, however, the study does provide some evidence that open peer review may not exacerbate bias. Shah’s study can be found here: https://arxiv.org/abs/2011.15083
Improving trust in research as it applies to the peer review process could include improvements to how reviewers approach their work. One would think that offering training and providing more detail on what to look for would help reviewers do a better job. However, two presentations from the PRC that reported on attempts to improve peer review quality showed very little efficacy. The first study, Reminding Peer Reviewers of the Most Important Reporting Guideline Items to Improve Completeness in Published Articles, presented by Benjamin Speich, asked reviewers to check ten specific reporting guideline items to be sure those items were accurately reported in the research paper. The study concluded that reminding reviewers to do this did not lead to better adherence to reporting guidelines in the published article. Speich speculates that peer reviewers may already be overworked and are not willing to take on additional tasks, and that those reviewers may not have adequate training on reporting and therefore their intervention would have little effect. The recommendation from Speich was that reporting assessment should be done by reporting experts, even if the journal has to pay for those services. Speich’s study can be found here: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7103787/
The other study was a bit more positive, though some may feel the reported level of improvements underwhelming. The study, Assessment of a Structured and Mentored Peer Review Curriculum on Quality of Peer Review, presented by Ariel Lyons-Warren, evaluated review quality before and after a mentored peer program for neurology residents and fellows. The study showed that peer reviews were better structured, but some of the qualitative aspects of the reviews were little changed, such as usefulness of review comments on the writing, the figures and the references, as well as the constructiveness of comments in general. In my opinion, the later items may have more to do with experience than skill, which leads me to think that mentoring is good for teaching technique, but experience in a discipline is important for quality. A blog post related to the mentor program can be found here: https://blogs.neurology.org/rf/improving-peer-reviewing-through-paired-mentor-mentee-reviews-with-the-rfs/
On a related note, Jessie Willis presented a systematic review entitled Online Training in Scholarly Peer Review that analyzed 22 openly available online peer review training activities, looking at time to completion (15 activities took an hour or less), target audience (usually researchers and journal reviewers), format (modules and webinars) and certification (5 provided certificates upon completion). The researchers also analyzed the content of each offering, identifying what topics they covered, from instruction on the peer review process, such as defining peer review (82%) and how to become a peer reviewer (77%), to training on how to conduct critical appraisal, such as understanding data and results (82%), analyzing methodology (77%) and identifying ethical concerns (64%). 91% focused on the very broad topic of writing effective peer review reports. Willis concluded that there is a need and a desire for peer review training, but very few high quality, openly accessible online opportunities exist. Willis’s study can be found here: Online training in manuscript peer review: a systematic review | medRxiv
Later this week I will wrap up my recap of the PRC by looking at some of the sessions that examined various aspects of preprints, including comparisons of preprints with their published articles. Preprints are controversial, with some people claiming that they erode trust in science, while others reject that premise pointing to the transparent nature of preprints.
For full list of the PRC sessions and for presenter affiliations, please click here: Peer Review Congress 2022 Program
By Tony Alves
