Tony Alves attended the Association for Learned and Professional Society Publishers 2023 Annual Conference in Manchester, UK. Among the many interesting topics covered, Alves chose to focus this three-part blog series on three sessions that he felt reflected some of the big concerns facing scholarly publishers today, fairer metrics, technological disruption, and Chinese research priorities. In this first blog post, Alves reflects on the keynote address given by Elizabeth Gadd of Loughborough University. Gadd calls for a change in the way we evaluate research and researchers.
A Call for a New Way to Evaluate Research and Researchers
Tony Alves, SVP of Product Management, HighWire Press
The ALPSP conference kicked off with a keynote address from Elizabeth Gadd, Research Policy Manager at Loughborough University. Her opening contention was that the audit culture and obsession with metrics are toxic in academics. Gadd pointed out that research publications are very important according to researchers, while governments and funders are very interested in metrics. This means that bibliometric vendors have a strong incentive to supply the industry with data on article usage and researcher relevance. This in turn creates a culture where researchers are tempted to gain the system, driving publishers to build more ever more elaborate systems to catch fraud. Afterall, if there was no glory to publishing, paper mills, ghost authors and other bad actors would not exist. It also skews the scholarly record, for example, negative trials can be critical to scholarship, but citation bias suppresses negative studies.
Advocating for research reform, Gadd asked, “Are broader contributions actually valued?” In other words, can you take the Journal Impact Factor and the research h-index out of assessor’s head? Gadd feels that this needs to be an industry-wide effort. UK Research and Innovation (UKRI), an organization that connects researchers to public funding opportunities, has been working on the Research Excellence Framework (REF) for the past few years. REF is an initiative that seeks to reshape researcher incentives and broaden what is rewarded. The current REF is set to conclude in 2028 and will assess efforts from 2021 through 2027. Those efforts include changing the approach to research assessment, moving away from assessing individuals and their volume of work, and moving toward valuing a broader range of things and recognizing the diversity of contributions.
To help move away from the audit and metrics culture, and to refocus on the goals of the REF, Gadd introduced the concept of the Resume for Research and Innovation. This is a narrative CV template that helps researchers highlight their skills and experience, rather than just focusing on a list of published research articles, when seeking opportunities in their chosen research field. This narrative CV is designed to highlight a researcher’s:
- Contribution to generation of new ideas, tools, methodologies or knowledge
- Development and maintenance of effective working relationships with other researchers
- Contributions to the wider research and innovation community, including peer review
- Contributions that affect a wider social benefit
The goal of the narrative CV is to increase visibility in different types of research output and to surface how the research was done, not just what the researchers concluded. There is a greater emphasis on team collaboration, working with and leading teams, as well as bringing in peer review activity, to better recognize and reward peer review. There is also an effort to include the non-academic impact of the researchers’ work, measuring social media and other real world impact measures as well as looking at how research contributes to sustainable development goals?
Finally, Gadd discussed the principles of open scholarly infrastructure and the work that the Coalition for Advancing Research Assessment (CoARA) is doing in this area. CoARA seeks to have “researchers and research organizations recognizes the diverse outputs, practices and activities that maximize the quality and impact of research”. Similar to REF, CoARA would like research and researcher assessment to be based on qualitative judgement supported by quantitative evidence. CoARA puts peer review at the center of their efforts, advocating for open pre-publication of peer review, open post-publication peer review, and peer review credit (Publons, ORCID, etc.).
The message of Gadd’s keynote was clear. The scholarly publishing industry should put people first in assessment, probe evaluation designs to be sure they are fair to everyone, and rethink the use of assessment and rankings in hiring and promotion. What can publishers do? They need to realize that assessment does not provide the same benefit to researchers as it does to publishers. They need to move away from publication-dominant forms of assessment, redefine what should be valued, and change how they assess the research they publish.
By Tony Alves
