Peer review analysis in the field of radiation oncology: results from a web-based survey of the Young DEGRO working group

The field of radiation oncology is continuously changing due to development of new radiation delivery and planning techniques, new systemic anticancer agents and demographic changes. Scientific journals have the important goal of disseminating these latest study results and contribute to the cumulative knowledge of the scientific community. With an increasing number of oncology journals listed in the Journal Citation Reports [1, 2], from 143 in 2008 to 230 in 2018, continuous and high-quality reviewing assessment is becoming more and more important. The increasing importance of the impact factor (IF) needs to be considered for quality and the number of manuscript submissions as well as for the motivation to contribute as a reviewer [3, 4]. At the same time, smaller or regional journals with a lower IF may encounter more difficulties in maintaining a proper review process. Based on the analysis of 3500 review experiences, Huisman et al. revealed that the satisfaction of authors as well as reviewers depends on the journal’s IF and duration of the review process [5]. Importantly, higher-ranked journals seemed to be more effective concerning the duration of the first review round, total review duration and immediate rejection time [5].

In the present survey, we aimed to evaluate and analyse the reviewing behaviour in German-speaking countries (Austria, Switzerland and Germany) in the field of radiation oncology. Based on the analysis, we sought to identify current trends and challenges in order to develop recommendations to increase the attractiveness of peer review activity.

A web-based questionnaire was developed within the survey platform “eSurveyCreator.com” (enuvo GmbH, Seefeldstraße 25, 8008 Zurich, Switzerland) containing 29 items regarding demographics and experience in publishing and reviewing scientific articles in the field of radiation oncology (see Supplementary I). Branching logic was used to tailor the questions on the basis of previous responses.

The data sample was collected through an online anonymized survey of radiation oncologists, biologists and physicists in German-speaking countries (Austria, Switzerland and Germany). In order to broadly address eligible participants, invitations were initially sent to members of the German Society of Radiation Oncology (DEGRO), German Society for Biological Radiation Research (DeGBS), German Society for Medical Physics (DGMP), Austrian Society of Radiooncology (ÖGRO) and the Swiss Society of Radiation Oncology (SRO) as well as through direct peer-to-peer contacts. A first invitation was sent to all participants on 8 October 2019 and a reminder email on 29 October 2019 to maximize the response rate. The survey was closed for any contribution on 24 November 2019. Multiple responses to the survey were prevented by locking the participants IP (internet protocol) address and setting cookies.

Raw data of the results were received directly from the online platform and were analysed using Microsoft Excel (Microsoft version 2001, Microsoft, Redmond, WA, USA) and SPSS (Mac version 25, IBM SPSS Statistics, IBM, Armonk, NY, USA).

We received a total of 281 responses from 8 October to 24 November 2019. Of these, 154 (55%) questionnaires were completed and included in the evaluation.

The vast majority of researchers consider peer review an integral part of their work and the scientific communication system [6, 7]. The peer review of manuscripts is often considered an altruistic duty of members of the research community [8]. However, the current scientific peer review process faces several obstacles such as increasing number of journals, submissions and non-adequate referee reports. To address these challenges, we conducted the first survey evaluating peer review activity in German-speaking countries (Austria, Switzerland and Germany), in order to develop recommendations to increase the attractiveness of peer review in the field of radiation oncology.

Our survey demonstrates that the most important criteria for peer reviewing are the scientific background of the manuscript (85%), the reputation of the journal (59%) and a high IF (40%). These findings are consistent with recent literature underlining the important role of the IF and the respectability of the journal for reputation, funding and career development [9, 10]. However, the suitability of the manuscript for the reviewer seems to be the most important criterion and should be considered in order to improve peer review activity in radiation oncology.

The majority of the participants (70%) of our survey prefer a double-blind review process over single-blind or open review. These findings go along with the published literature [6, 11]. Interestingly, only 14% would prefer the new concept of an “open review”. In fact, “open review” is described by more than 120 definitions in literature [12]. As a result, several characteristics may be incorporated, such as an open interaction between readership, authors and reviewers, non-blinded reviewing, reports and participation to the manuscript [12]. Single- or double-blind reviews have been suggested to protect reviewers from the author’s influence, allowing them to give objective feedback and thus minimizing the potential for tortious interference. Several studies have failed to show that single- or double-blind reviewing increases the quality of the review process [13‐15]. Due to effective online search methods and close scientific communities, blinded reviewing has been found to successfully conceal the author’s identity in only 54–74% of all cases [13, 14].

The increasing workload placed on peer reviewers due to a rapidly growing volume of publications and inappropriate selection of reviewers by the editorial board are frequently discussed [16]. Therefore, identifying qualified reviewers and keeping them motivated for future submissions is an important editorial task. Well-known researchers may be asked several times a day for their review contribution, resulting in a delay in the review process and decline of authors’ submissions due to increased processing time. An effective, timely review process is associated with higher IFs and satisfaction of authors and reviewers [5, 17]. In contrast to other studies, our survey revealed the important role of the scientific background of the manuscript (69%) and the effort of reviewing (63%) as the main reasons for declining peer review. With regard to the significance of the IF, our data is consistent with the results of the Global State of Peer Review Report 2018 [18].

The association between the likelihood of accepting an invitation for review and its scientific background as well as the choice of a journal based on high IF and visibility in the field creates particular strain for journals with a lower IF. Most manuscripts will at first be submitted to higher-IF journals where many of them will undergo a first round of peer review [19]. In case of rejection, they will then be re-submitted to journals with a lower IF, which—according to our analysis—will be less likely be able to facilitate a timely peer review process due to their lower attractiveness for potential peer reviewers and the increased workload due to several independent rounds of peer review.

High-IF journals perform a rigorous preliminary selection before external peer review, resulting in fewer review requests and a more effective peer review process. In order to improve the peer review process, pre-selection by the editorial office would result in significantly reduced review times. However, preliminary selection may undermine the neutral stance of the peer review process.

Especially for young researchers, the invitation to review a research article based on the recognition of expertise can be considered a great honour and important part on the way to becoming an independent researcher. However, as our results show, particularly established participants over 35 years of age with more than 10 years of experience in scientific publishing are willing to review manuscripts on a regular basis. A possible cause, according to younger scientists, could be the lack of relevance of peer reviewing for advancing their career and the lack of compensation for the often time-consuming work. Instead of conducting peer review, young scientists may be more interested in publishing their own research results and increasing their reputation. In addition to the reduced level of scientific expertise, this could explain why the participants were mainly postdoctoral researchers (45%), habilitated scientists (21%) and professors (24%). This hypothesis is underlined by survey results from Nicholson et al. [20], where about 70% of respondents indicated that they would include peer review as a professional service in their curriculum vitae. Importantly, 27% of respondents cited formal recognition in assessment as a factor that would motivate them to participate in public peer reviews. The results may also demonstrate a growing sense of frustration that for-profit companies, i.e., publishers, benefit in several ways from the scientific community. First, they receive direct or indirect payments for publication and for access to these articles. Second, peer review is conducted at no charge. In recent years, article-processing charges (APCs) have increased dramatically for several publishers, particularly open-access or hybrid open-access publishers, without any rewards for reviewers, resulting in enhancement of this questionable situation. As a result, predatory publishing has become a popular business model that involves charging publication fees to authors without providing editorial and publishing services or quality checks, which legitimate academic journals provide.

In our survey, only 12% of the participants have been rewarded for writing a peer review, namely partial waiver of publication fees (67%), free access to articles of the journal (56%), salary (6%) and vouchers for registration fees of various events such as scientific meetings (6%). Importantly, 59% of all participants stated that they would review more frequently if some kind of compensation or reward were offered. A Wiley study with more than 2900 participants revealed that reviewers are not satisfied regarding recognition for their work and stated that they should be rewarded for their effort [17].

Our survey demonstrates that scientific experience is correlated to review activity. However, due to the rising submission rates, we see an urgent need to actively engage early career researchers in the review process. In order to facilitate and promote peer review, peer review trainings for young researchers may improve both the motivation for and the quality of peer review reports [17]. The suitability of the reviewer for the submitted manuscript seems to be the most important criterion for peer review performance and should be critically recognized.

The results of this survey must be interpreted in light of its limitations: non-responders and participants with incomplete surveys (127 of 281 [45%]) might have answered differently and, therefore, our findings cannot be applied to a general statement. However, we believe that the diversity of our responder characteristics concerning age, gender, profession and scientific publishing expertise is comparable to the scientific community in radiation oncology and increases the informative value. Moreover, the used Likert scales may capture only broad perspectives in this complex issue of scientific publishing.

In summary, the suitability of the reviewer and the scientific background of the manuscript are the crucial criteria for willingness to peer review. Therefore, rigorous preliminary selection before external peer review may result in fewer review requests, more manuscripts that are suitable and a more effective peer review process. The review procedure should be performed in a double-blind manner. Rewards for peer review activity such as partial waiver of publication fees and free access to articles of the journal would increase the willingness to peer review. Peer review trainings for young researchers may improve both the motivation for and the quality of peer review reports.

Peer review activity in radiation oncology is determined by the suitability of the reviewer, reputation and a high IF of the journal. Rigorous pre-selection may result in fewer review requests and more effective peer review process. A double-blind peer review process is recommended by the majority of the participants. Review activity would increase if compensation were available. Free access to journal articles, discounts for publication costs or congress fees, or an expense allowance are ways to make the review process more attractive.