Changes to Cancer Research UK’s peer review process for grant applications

Although a valuable part of our grant application process, peer review is time consuming for both Cancer Research UK and our reviewers. We want to improve the way we make funding decisions and reduce the burden of written peer review on researchers. To that end, we are reducing the emphasis on written peer reviews and increasing our use of expert review panels.

We spoke with Dan Burkwood, Head of Research Funding Operations, to find out more about the changes and what they mean for future grant applications.

What’s changing with peer review?

db_photo.png

Over the last few years, we have looked at how we run our grants application process, and our analysis has shown that written peer review of grant applications has not been adding the value we think it should. So, we’ve decided to refocus our process by increasing the use of expert review panels in our decision making.

Of course, the important thing to say up front is that we continue to value peer review. It will always remain an important part of our scientific decision-making process, and we won’t be awarding any funding without it. Expert review panels are just a different way of doing it, and we think having in-person discussions is a more valuable way of reviewing research applications than soliciting written reviews. 

When will these changes take effect?

We will be using this new process in the upcoming round of funding committees, from spring 2021. Any proposals for grants going forward will now be assessed under these new changes.

What will the new peer review process look like?

All the applications will be reviewed in detail by an expert review panel.

When researchers apply for funding, we will group the applications by theme, and then convene expert review panels with relevant expertise in that area. These panel members will be the primary quality reviewers of the applications in that theme and will score the applications on the quality of the science. 

For larger grants, the applicant may also be invited to interview. For those applicants, we’ll share comments from the panel, which will be similar to the feedback they’d receive from written peer review. The panel will also highlight areas for discussion, so the applicants can prepare.

Once the applications have been reviewed by the panel, they will move on to the relevant funding committee. Committee members will make funding recommendations, considering both the quality scores for each application, and how the application fits strategically within our wider portfolio. 

Finally, funding decisions will then be fed back to applicants.

How will you select the members of these expert review panels? 

Our expert review panels will consist of researchers in defined areas, who can provide deep expertise on the grant proposals that we’re reviewing. 

In much the same way as we’ve always chosen peer reviewers, subject-matter experts in Cancer Research UK will decide on the selection criteria for panel members, based on the themes from that round of applications. They’ll look at people’s track records, who holds major grants, at people’s publication records and at other sources of information to find the best people to join the panel. They’ll also be mindful of the makeup of the panel and ensure we are meeting our standards for gender and ethnic diversity. 

One of the benefits we’ve seen in the past year, with moving to remote panels, is that we’ve been able to increase our international representation because you don’t have the difficulties of travel and people are more easily able to dial in and join them. 

Are there any cases where we’d still ask for a written peer review? 

Actually, yes, we will still use written peer review on occasion, but in much more limited circumstances. For example, if there is a particular technique or something in an application that’s a bit standalone, we’ll ask for a written peer review report on that aspect of the application to give the panel some extra input. 

Why are we making these changes?

In its current format, peer review takes a lot of time, both for us and for our research community. But when we compared written peer review scores and final funding decisions, we found in many cases they weren’t well correlated, so that tells us written review isn’t really doing what we need it to do.

My feeling is this reflects the burden on the research community. Cancer Research UK puts out somewhere between 3,000 and 5,000 written peer review requests a year, and we’re not the only people asking for peer reviews. If you look at the growth in scientific publishing in the last 10 years, that’s leading to more peer review requests – the system’s exhausted! Researchers are overloaded with review requests; it’s understandable if people don’t spend much time on them.

In written peer review, each reviewer looks at one application in isolation, so they don’t get the context of seeing other applications. Whereas a strength of expert panels is that you can bring people together for a day or two at a time, they can discuss the applications, really interrogate them at a deeper level and form views together as a collective We should get a much more constructive and valuable review out of that discussion. We’ve actually found that people are sometimes more willing to be on the panel than to write a peer review report, because on the panel they’ll get two days of lively scientific debate.

How will you evaluate whether the changes have the desired effects?

In terms of easing the burden on researchers, I think we can safely say we’ll be sending out far fewer written peer review requests, so the effect should be easy to see. 

To assess the quality of reviews, we’ll be looking for a closer correlation between how applications are scored at panels and the decisions being made by the funding committees. We’ll also be monitoring how those committees are working; whether people have everything they need in order to make those major decisions that affect all the research portfolio and people’s careers. While it’s quite hard to get metrics on that, we’ll monitor those discussions, and we’ll take feedback.

Dan Burkwood is Head of Research Funding Operations at Cancer Research UK. He is responsible for ensuring Cancer Research UK’s grant funding and peer review policies and processes enable the charity to fund the highest quality research.