Intra-rater repeatability of a structured method of selecting abstracts for the annual euraps scientific meeting

To date, surprisingly little attention has been given to the process of selection of abstracts submitted to biomedical meetings and there are no reports on the intra-rater reliability of such a selection. We wanted to determine the intra-rater repeatability of the selection by multiple reviewers, of abstracts submitted to a plastic surgical scientific meeting. Prospective analysis of repeated structured ratings of five abstracts by three blinded reviewers of each of the 202 abstracts submitted to the annual scientific meeting of the European Association of Plastic Surgeons (EURAPS). The intra-class correlation coefficient of the score and repeated score of five abstracts and the kappa statistic of the dichotomy of acceptance of the top two rated abstracts vs rejection of the remaining three abstracts. Both were calculated for the set of repeated scores by each individual reviewer, as well as for the set of totals of scores by all reviewers. The median of the reviewers’ individual intra-class correlation coefficient was 0.86 (range, −0.56–0.92). Six out of 10 reviewers rated the same two abstracts as top abstracts during both reviews, resulting in kappa statistics ranging from −0.15 to 1.0 (median, 0.59). The median intra-class correlation coefficient of joined scores was 0.93 (range, 0.92–0.97), and the kappa statistic for the joined top-rated abstracts was 1.0. Excellent repeatability of the ranking and dichotomy of abstracts based on the joined scores of multiple peer reviewers gives confidence in EURAPS’ structured method of abstract selection.