A quick rundown from the 9th Peer Review Congress – Part 3 – Preprints and their effectiveness in peer review

A quick rundown from the 9th Peer Review Congress – Part 3 – Preprints and their effectiveness in peer review

All week publishers have been celebrating Peer Review Week with various activities centered around this year’s theme, “Research Integrity: Creating and supporting trust in research”. Earlier this week I summarized sessions addressing research integrity from the most recent International Congress on Peer Review and Scientific Publication (PRC), which took place in Chicago Illinois from September 7th to 9th. On Monday I examined sessions that explored author and contributor misconduct as factors that can affect trust in science, after all trust in research starts with trust in researchers themselves. On Tuesday I highlighted the papers that discussed different peer review models, the effectiveness of peer review, and measures being taken to improve reviewer performance in an effort to find ways to increase trust in the peer review process. Today I will wrap up my recap of the PRC by looking at some of the sessions that examined various aspects of preprints, including comparisons of preprints with their published articles. Preprints are controversial, with some people claiming that they erode trust in science, while others reject that premise pointing to the transparent nature of preprints.

The previous set of summaries addressed integrity in the peer review process, and this next session summary does the same, with an eye toward preprints as a way to measure peer review effectiveness. In the abstract for this session the researchers say, “The ability of peer review to improve the scientific endeavor (eg the conduct, reporting, and validity of study findings) has been questioned, and calls have been made to showcase changes that occurred to each study due to peer review.” This is what the presentation, A Synthesis of Studies on Changes Manuscripts Underwent Between Submission or Preprint Posting and Peer-Reviewed Journal Publication, presented by Mario Malicki, attempts to do by examining papers that compared preprints and submitted versions of manuscripts with the peer reviewed versions of those same manuscripts. The researchers examined 28 studies, documenting how those studies went about comparing the sets of papers, along with other useful data for comparison like number of version pairs, manuscript sections used for comparison, and the metrics used. The results showed high similarity between the preprint or submitted manuscript and the final accepted version, though full quantification of the results is still underway. Malicki suggests that evaluation of peer review impact would be easier if there were greater standardization in the methods used to compare the different versions, including linking the preprint version with the published article, using open peer review for convenient analysis of suggested changes, and utilizing peer review statements that describe the changes that were made to the paper. Mario Malicki is the Editor-in-Chief of the journal Research Integrity and Peer Review.

Now to focus more on preprints, which are “research manuscripts yet to be certified by peer review and accepted for publication by a journal”, and preprint servers, which are “online platforms dedicated to the distribution of preprints”. These definitions were presented by Joseph Ross at the beginning of his talk, medRxiv Preprint Submissions, Posts, and Key Metrics, 2019-2021. It is a useful presentation for setting the stage for the other presentations I will summarize, as it take a very general look at one of the most influential and impactful preprint servers, medRxiv, and demonstrates why preprints contribute to both trust and mistrust in science. Some of the touted benefits of preprints are that they are an early way to demonstrate scientific productivity and provenance of ideas, they prompt feedback and collaboration, they expose results for less publishable yet useful studies, and they promote transparency by prompting researchers to experiment more before submitting to a peer review journal. I know there are several people out there that would argue against many of these points, and Ross does discuss preprint drawbacks, such as the possible harm to the public when incorrect information that could result in harmful treatment is posted and then magnified by the media, as well as misuse by commercial interests looking to promote themselves or denigrate a competitor. 

Looking specifically at medRxiv, there are policies in place that add a level of screening not seen at most other preprint servers. medRxiv is very different from other preprint servers because it is dealing with medical research that could potentially be used in diagnosis and treatment of disease and injury. They follow ICMJE guidelines on author disclosures of affiliations, funding and ethics, they only allow research articles and study protocols to be posted, and all preprints are screened before they are posted. These policies are important in raising the level of trust that researchers can have in the preprints found on medRxiv, but I don’t think that this necessarily translates into a general trust in research across preprint servers. Ross notes that from the launch of medRxiv in 2019 to the end of 2021, 83% of submitted preprints got posted after screening, and of those, only 0.17% were withdrawn. Although preprint usage statistics are impressive (133 million abstract views and 64 million downloads), the numbers on collaboration, at least via the preprint server, are quite low with only 8% of preprints with at least one comment. Most of this study was a presentation of facts from which I will not try to draw any conclusions regarding trust or the effect preprints have on research integrity. However, it is a useful case study in how a preprint server can impose signals of trust in what is otherwise a wild west scenario of players and policies.

Read: More in Common: Comparing Preprinted and Published Versions of an Article

One way to improve trust in preprints is to show that preprint versions of research papers are not radically different than their published counterparts. Showing that most researchers are taking the distribution of their research seriously and are not posting incomplete papers could be seen as evidence that preprints can be a trusted resource, though it must still be understood that preprints are not fully peer reviewed and should still be treated as working papers. In the session, Assessment of Concordance Between Reports of Clinical Studies Posted as medRxiv Preprints and Corresponding Publications in Peer-Reviewed Journals, presenter Guneet Janda looked at how much a research paper changes between preprint and published article. The study evaluated concordance between sample size, primary endpoints, results, and interpretation in preprints of clinical studies posted on medRxiv that were later published in peer review journals. Concordance for each of these means that the sample size was the same, there were no additional primary endpoints identified, all numeric results for primary endpoints were identical, and statements about the findings and the implications of the study were similar. Of 1853 papers submitted to medRxiv in September of 2020, 331 papers fit the study’s criteria, which included publication in a high-quality peer review journal. These papers showed overwhelming concordance between the preprint and published versions, with discordance rates for sample size at 11%, primary endpoint at 1%, results at 12%, and interpretation at 2%. Does this show that preprints are just as trusted as the published peer review version? No, this study only evaluated published preprints, and there are many preprints that do not get published. Also, the study looked at preprints published in high impact journals, which could mean that only well conducted studies made it through the gauntlet of peer review. 

Janda’s study can be found here: Assessment of Concordance and Discordance Among Clinical Studies Posted as Preprints and Subsequently Published in High-Impact Journals

A similar presentation, Comparison of Reports of Epidemiology Studies Posted as bioRxiv Preprints and Published in Peer-Reviewed Journals, by Mario Malicki, compared epidemiology studies posted on bioRxiv with their peer reviewed published versions. As of January 2021, there were 1538 epidemiology preprints posted on bioRxiv, and out of those, 844 were published as journal articles. 622 of those papers had a single preprint version to use in comparison, and the researchers then randomly selected 121 papers from that sample. Using MS Word track changes, the researchers analyzed the differences in the two versions of each pair, looking at elements like authorship, abstract, methods, results, discussion, conflict of interest, data sharing, and references. Although there were many minor changes between preprint and published manuscripts, important elements like methods and findings remained unchanged in over 90% of the pairs. This study did not examine who requested changes or why those changes were made, or if the changes actually improved validity or quality of the research. Similar to the previous study presented by Malicki, one takeaway is that understanding what changes took place as a result of peer review would help researchers better evaluate the impact of peer review and increase trust and transparency. 

Here is a link to a presentation given by Malicki on a similar topic: https://www.ti.ubc.ca/2021/08/25/aug-25-timss-do-we-need-journal-peer-review/

Here is a link to a poster presented at the EASE conference in 2021: Do we need journal peer review? Changes between 121 epidemiology preprints and their subsequent journal publications

Transparency contributes to trust in science, and reader-access to comments and feedback that might influence the final paper, can be helpful in achieving that. Preprint servers generally encourage comments from readers, since the concept of the preprint is that it is an early version of the paper. However, there are many questions surrounding the usefulness of these comments. The final session I want to summarize analyzes the comments made on bioRxiv and medRxiv by readers. The session Content Analysis of Comments on bioRxiv and medRxiv Preprints, presented by Gabriel Costa, described the content of comments on these preprint servers from 2020 to the present. Eleven evaluators looked at preprints that received between 1 and 20 comments using a standardized form. There were 1921 comments across 1037 preprints, with the vast majority receiving a single comment, and approximately 150 preprints on each server receiving 2 or more comments. General features of the comments included: comments from one of the article’s authors (11%); comments from an organized review effort (7%); summary description of the article (10%); and comments that included references (25%). Comments from authors usually discussed publication status, with additional information, study promotion and corrections being the next most common types of comments. Non-author comments most often included criticism and corrections (62%), compliments (38%), and questions (35%). Costa concluded that comments were sparse and brief, although those performed by organized review efforts were more complete and well organized. He also noted that a standardized taxonomy of the functions of peer review would also be helpful. 

It is clear from Costa’s presentation that commenting on preprints is not currently a regular part of the scientific process, and most comments are not providing meaningful value, and so perhaps not currently contributing to transparency. The exception, noted by Costa, was comments provided by organized review efforts. It is useful to point to a project by ASAPbio promoting organized preprint comments. They have compiled a registry of preprint review platforms called ReimagineReview. There are over thirty different services listed, and it will be very interesting to see how these evolve and expand across the preprint universe. Regarding a standardized peer review taxonomy, requested by Costa, there is a NISO and STM joint initiative currently underway developing “standard definitions and best practice recommendations for the communication of peer review processes”. Whether it is commenting on preprints or reviewer comments available alongside the published article, a commonly agreed vocabulary to describe peer review will provide structure and contribute to understanding and transparency of peer reviews.

Read: Credit for All Contributors Leads to More Transparent Research Outputs, An Ecosystem-Wide Benefit

Is it enough that the scholarly research community is spending the week thinking and talking about research integrity and trust in science? I know that the topic of research integrity is top-of-mind for many in scholarly publishing who are working with initiatives like STM’s Integrity Hub and NISO’s CORREC and CRediT, just to name a few. Both researchers and the public need to trust the work we all do to verify and distribute research, and this is becoming more and more difficult. In 2013, I was program co-chair for the Council of Science Editors annual meeting in Montreal, where the theme was Communicate Science Effectively – The World Depends On It! In a very prescient keynote address entitled, The New Science Communications Climate, science journalist Andrew Revkin stated that conventional science reporting was shrinking as budgets contracted, and that the public and policy makers were being inundated with disinformation and misinformation found online. We are now ten years down the road, and it seems that this situation has only gotten worse. A February 2020 Pew Research Center report shows that public confidence in scientists has declined since 2020. In that report, “29% of U.S. adults say they have a great deal of confidence in medical scientists to act in the best interests of the public, down from 40% who said this in November 2020. Similarly, the share with a great deal of confidence in scientists to act in the public’s best interests is down by 10 percentage points (from 39% to 29%)”. The eroding of trust in science has been attributed to various forces, such as miscommunication of science by the media, the lack of understanding that science is a process of collecting evidence and not a process of declaring truths, the political turmoil in the wake of the COVID-19 pandemic, and the use of pseudoscience to promote agendas. No, it’s not enough for us to have spent the week thinking about research integrity, it clearly needs to be at the forefront of how the scholarly publishing community approaches scholarly communications every day.

For full list of the PRC sessions and for presenter affiliations, please click here: Peer Review Congress 2022 Program 

By Tony Alves

Latest news and blog articles