Altmetric: Top 50 dental articles in 2014

Key Points

• Highlights Altmetric as a new and emerging scholarly tool that measures online attention surrounding journal articles.

• Reports the top 50 dental articles in 2014 according to Altmetric data.

• Suggests that dental clinical practitioners and research scientists should pay more attention to altmetrics as a rapid tool to measure the social impact of scholarly articles.

Abstract

Introduction Altmetrics is a new and emerging scholarly tool that measures online attention surrounding journal articles. Altmetric data resources include: policy documents, news outlets, blogs, online reference managers (eg Mendeley and CiteULike), post-publication peer-review forums (eg PubPeer and Publons), social media (eg Twitter, Facebook, Weibo, Google⁺, Pinterest, Reddit), Wikipedia, sites running Stack Exchange (Q&A), and reviews on F1000 and YouTube.

Methods To identify the top 50 dental articles in 2014, PubMed was searched using the following query “(“2014/1/1”[PDAT]:”2014/12/31”[PDAT]) and jsubsetd[text]” in December, 2015. Consequently, all PubMed records were extracted and sent to Altmetric LLP (London, UK) as a CSV file for examination. Data were analysed by Microsoft Office Excel 2010 using descriptive statistics and charts.

Results Using PubMed searches,15,132 dental articles were found in 2014. The mean Altmetric score of 50 top dental articles in 2014 was 69.5 ± 73.3 (95% CI: −74.14 to 213.14). The British Dental Journal (48%) and Journal of Dental Research (16%) had the maximum number of top articles. Twitter (67.13%), Mendeley (15.89%) and news outlets (10.92%) were the most popular altmetric data resources.

Discussion Altmetrics are intended to supplement bibliometrics, not replace them. Altmetrics is a fresh and emerging arena for the dental research community. We believe that dental clinical practitioners, research scientists, research directors and journal editors must pay more attention to altmetrics as a new and rapid tool to measure the social impact of scholarly articles.

Introduction

The term bibliometrics, statistical analysis of written publications (eg articles), was created by Alan Pritchard in 1969.¹ One of the most well-known aspects of bibliometrics is citation analysis. At the moment, a classic method to evaluate the impact of a research output is based on the number of citations for an article. There are many available citation databases, including Web of Science, Scopus, Google Scholar, etc. The most well-known bibliography tools to assess the impact of research output or journal performance are impact factor (IF), SCImago journal and country rank (SJR), source normalised impact per paper (SNIP), impact per publication (IPP) and Eigenfactor. The process of assessment of the impact is counting the number of times an article is cited by other works, with specific algorithms ranging from average citations per document to PageRank.

On the other hand, the emergence of new internet-based-technologies opens up new perspectives to evaluate the impact of research. Scholars are moving their everyday work to the web; biomedical researchers, healthcare professionals and patients are increasingly using social media and new scholarly e-tools to facilitate and improve their communication.^2,3,4 Large-scale analysis covering the entire spectrum of medical disciplines showed Twitter coverage has increased dramatically over time in the biomedical literature.⁵ The growing range of new online scholarly tools allow us to create new metrics for impact or use of scholarly publications, particularly for the public. To this end, 'altmetrics' was introduced by Jason Priem in 2010.⁶

Numerous websites and projects are computing altmetrics, including Impact Story, Plum Analytics and Altmetric.^7,8 A number of prestigious publishers have started providing altmetric data for their customers, including Elsevier, Wiley, BioMed Central, Nature Publishing Group, PLOS and Frontiers. Although altmetrics is a new term, interest in altmetrics is growing fast in comparison with bibliometrics (Fig. 1). However, an easy search of dental journals in PubMed by the key word “altmetric*” in December 11, 2015 showed no articled on this topic. In this article, we aimed to discuss only Altmetric (https://doi.org/www.altmetric.com); what Altmetric scores are, and how we should read and interpret them.

Figure 1

The results of Google trends search for 'altmetrics' (red line) and 'bibliometrics' (blue line). The horizontal axis of the graph shows time and the vertical one is the number of times a term is searched, relative to the total number of searches, globally. Data are from https://doi.org/trends.google.com (December 11, 2015)

A brief description of Altmetric

Altmetric permits consumers to access data on individual articles via a free bookmarklet. Users install the bookmarklet in their internet browsers by just dragging it into the browser's bookmark bar; then, when the user reads an article, they can click the bookmarklet button to obtain the Altmetric score and information pertaining to that article. Altmetrics automatically finds the DOI (digital objective identifier) or PMID (PubMed ID) on the article webpage and a report pops up in the right corner of the browser providing altmetrics that include a score indicating how much online attention the article has received. However, the main questions are: 'what data sources does Altmetric track?' and 'how is the Altmetric score calculated?' Data resources for Altmetric analysis include:

• 1 Policy documents

• 2 News (more than 1,000 English and non-English global news outlets which are available via: https://doi.org/www.altmetric.com/sources-news.php)

• 3 Blogs (over 8,000 academic and non-academic blogs)

• 4 Online reference managers, including Mendeley and CiteULike

• 5 Post-publication peer-review forums, including PubPeer and Publons

• 6 Social media, including Twitter (public comments and retweets only, no favourites), Facebook (public posts only, no likes), Weibo, Google⁺, Pinterest and Reddit (original posts only, not comments)

• 7 Other online sources, including Wikipedia, sites running Stack Exchange (Q&A), reviews on F1000 and YouTube.

After the collection of raw data, they are weighted according to the default values (Table 1) to reflect the relative reach of each source and then an Altmetric score is calculated. We should also bear in mind that Mendeley and Cite ULike (online reference managers) scores are calculated and presented but are never counted towards the Altmetric score. It is important to notice that the Altmetric score of a research output offers an indicator of the amount of online attention it has received.

Table 1 Weighted values for data resources of altmetric analysis

Methods

To identify dental articles in 2014, PubMed was searched using the following query: “(“2014/1/1”[PDAT]:”2014/12/31”[PDAT]) and jsubsetd [text]” in December, 2015. Consequently, all PubMed records were extracted and sent to Altmetric LLP (London, UK) as a CSV file for examination. Data were analysed by Microsoft Office Excel 2010 using descriptive statistics and charts.

Results

Using PubMed searches, 15,132 dental articles were found in 2014, from which 2,345 (15.49%) articles were open access, 673 (4.44%) articles were clinical trials and 1,010 (6.67%) articles were reviews.

The list of the top 50 dental articles with the highest Altmetric score is provided in Appendix 1 (in the online supplementary information available with this paper). The mean Altmetric score was 69.5 ± 73.3 (95% CI: −74.14 to 213.14) (Fig. 2). The British Dental Journal (48%) and Journal of Dental Research (16%) had the maximum number of top articles (Fig. 3). Twitter (67.13%), Mendeley (15.89%) and news outlets (10.92%) were the most popular altmetric data resources (Figs 4 and 5). Geographical analysis of the tweets showed that the United Kingdom (30.54%) and the USA (11.1%) had the highest number of tweets. Demographic breakdown of all the tweets (1,640) showed that 55% were by members of the public, 39.1% by practitioners (doctors, other healthcare professionals), 4.3% from scientists and 1.4% by science communicators (journalists, bloggers, editors) (Fig. 6).

Figure 2

Altmetric score of top 50 dental articles in 2014 (X axis). Clearly, 45 articles (90%) had an altmetric score lower than 125. Linear trend and R value are also shown. Box and whisker plot of data are shown on the left

Figure 3

Journals with the highest number of articles among the Altmetric top 50 dental articles in 2014

Figure 4

Total number of posts for all Altmetric data resources among top 50 dental articles in 2014 (Sum: 2,416, Mean: 161.1 ± 419.7, 95% CI: −661.6 to 983.7)

Figure 5

Number of posts for Twitter, news outlets and Mendeley for each of top 50 dental articles according to Altmetric in 2014 (X-axis). Linear trends and R values are also shown

Figure 6

Demographic breakdown of tweets among the top 50 dental articles according to Altmetric in 2014

We couldn't find any record from post-publication peer-review forums (for example, PubPeer, Publons, F1000), Wikipedia, sites running Stack Exchange (Q&A), Pinterest and policy documents. Other resources such as Facebook (2.69%), Weibo (2.19%), blogs (0.62%), Google⁺ (0.33%), Cite ULike (0.12%) and Reddit (0.08%) were used infrequently (Fig. 4). From among theses 50 articles, two (4%) articles did not have DOI. As a final point, readers should bear in mind that the Altmetric score may slightly fluctuate over time.

Discussion

To our knowledge, a persistent problem in dentistry is the slow recognition of new technologies by dental scholars and practitioners.⁹ For example, although X-rays were discovered in 1895, the first dental radiology, as a key diagnostic device, was introduced at 1913.¹⁰ The anti-caries efficacy of amorphous calcium phosphate had been shown in the 1960s, even though it was launched into the market only in 2004.¹¹ Now, we are experiencing this old problem with new online scholarly tools like altmetrics. Results of a recent survey showed 114 million English-language scholarly documents are accessible on the public web.¹² Turning a blind eye to what is happening to these documents in social media, news outlets, scientific blogs, policy documents, post-publication peer-review resources etc seems illogical.

Altmetric top 100 articles for major categories of science were published in 2014 and 2013.^13,14 So far, this study is the first attempt at this in the field of dental sciences. Dissemination of this list would increase the knowledge and awareness of dental research scientists' about new online scholarly tools such as altmetrics. Just like citation levels, altmetric scores of dental articles are low. The highest Altmetric score among dental articles in 2014 was 430, while it was 3,500 in medicine and 5,044 in all categories of science.¹³

A social media update in 2014 showed Facebook remained by far the most popular social medium and Twitter the least popular.¹⁵ However, among Altmetric top 50 dental articles, Twitter (67.88%) was much more popular than Facebook (2.69%) (Fig. 4). An interesting point is that Twitter is censored and blocked in some influential countries.¹⁶

Considering the principals of evidence-based dentistry, articles with both high (meta-analyses, systematic reviews and randomised controlled trials) and low (ideas, editorials and expert opinions) epistemological strength were seen among the Altmetric top 50 dental articles. It is no surprise that previous analysis of the top 100 cited articles in dentistry showed articles with the lowest epistemological strength (case series and narrative review/expert opinions) had the highest citation rate.^17,18

Post-publication peer-review services such as F1000, Publons and Pubpeer are opening up new horizons to the scientific community.¹⁹ A good example indicating the importance of this process is the rise and fall of STAP (stimulus-triggered acquisition of pluripotency).²⁰ According to Nature news: 'Two papers published in Nature in January 2014 promised to revolutionise the way stem cells are made by showing that simply putting differentiated cells under stress can 'reprogram' them and make them pluripotent – able to develop into any type of tissue in the body.'²⁰ After a relatively short time, critical post-publication peer reviews began to emerge on PubPeer by researchers, named and anonymous, unable to replicate the study. Consequently, 'the lead author was found guilty of misconduct, the papers were retracted and the RIKEN centre, where she worked, was rwstructured. The aftermath of the episode has been felt by scientists across Japan, in the form of new anti-misconduct policies.'²⁰ Nevertheless, despite the importance of this new concept, we could not find any post-publication peer-reviews among Altmetric's top 50 dental articles.

Evidence-informed health policy-making is one of the programmes of the World Health Organisation (WHO) that aims to ensure decision and policy makers are well-informed through the best available research evidence.²¹ However, we could not find any Altmetric top 50 dental articles that have been cited by policy documents. Only one systematic review about child dental neglect,²² whose Altmetric rank was #58, was cited by a UK government policy documents.²³

Despite several advantages, altmetrics have inherent weaknesses and altmetric findings should be interpreted with great caution. Advantages and limitations of altmetrics are discussed in Appendix 2.

Box 1: Appendix 2 Advantages and limitations of altmetrics

Limitations of bibliometrics are well known and have been widely discussed.^{24,25,26,27,28} Evaluation of research impact using bibliometric tools alone is not enough, and it is not predictive of subsequent clinical applications and health promotion.²⁹ Nevertheless, altmetrics is still a young arena, and research is needed to improve this field. Altmetrics cannot be seen as an alternative or replacement to bibliometrics; at most, it may function to complement bibliometrics.⁷ Bibliometrics measure scholarly impact; while altmetrics measure impact among scholars, member of public, practitioners (doctors, other healthcare professionals), science communicators (journalists, bloggers, editors), policy makers etc. Inclusion of post-publication peer-review resources, for example, F1000, Publons and Pubpeer into altmetrics means that academics now have easy access to what scholars are actually saying about each other's research findings. The geographical and demographic breakdown of readers by Twitter and Mendeley can show the extent of interest beyond a specific field or discipline. Because of the slow pace of academic publishing, it can take several months or years before an article begins to be analysed by bibliometric tools. With respect to the speed of communication today, this could be too long for researchers, readers, and grant funders. By tracking how research output is shared and discussed in real time, altmetrics could fill the gap between publication and citation.

However, though results of Google trends shows attention to altmetrics is growing fast (Fig. 1), the main question still is 'Do altmetrics work?' Results of a large scale study showed strong evidence that six of the altmetric data resources (tweets, Facebook wall posts, research highlights, blog mentions, mainstream media mentions and forum posts) do associate with citation counts, at least in medical and biological sciences.³⁰ Nowadays, the relationship between the impact factor and papers' citations is weakening and it might indicate that the usage of the impact factor as a sole tool to assess the quality of journals, articles, and researchers is coming to an end.³¹ On the other hand, tweets can forecast highly cited articles in the first 3 days of article publication³² and the extent of a general medical journal's Twitter followers is strongly related to its impact factor and citations.³³ Of more interest, the potential of altmetrics to inform funders about research impact is growing fast and attracts the attention of medical research funders and charities, eg the Welcome Trust.³⁴ Another good example would be the John Templeton Foundation, who award more than $100 million a year in research funding. The Director of the Planning and Evaluation team of the John Templeton Foundation, Steve Fitzmier, says 'At the core of the Foundation's mission is a desire to both fund high quality research and to generate greater public engagement with the research we support;...while analysing metrics such as citations can be helpful to assess impact, these methods provide an incomplete picture.'³⁵ The foundation decided to use Altmetric data in grant making and in communications so they can get a better sense of the broader impacts of the research they're funding, and they believe this will help them improve their future grant making process and communication strategy.³⁵

Nevertheless, despite several advantages, altmetrics has its inherent weaknesses and altmetric findings should be interpreted with great caution. As for bibliometrics, altmetrics do not rank the quality or scientific significance of articles. They only measure online attention surrounding journal articles. Social media posts may be done by a non-scientist who is attracted to buzzwords or 'sexy' words in titles.³⁶ Figure 6 showed 55% of tweets were made by members of the public who had not had access to full text articles.

However, as a good dental example, an author of this paper was surprised when reading the Altmetric score of their editorial 'What would be the tooth structure at non-carbon-based-life?'³⁷ Although it involves an innovative idea, it is not an article with high quality scientific content. Yet, this editorial ranked in the top 5% of all research outputs ever tracked by Altmetric presumably because of buzzwords in the title. Twitter analysis showed 63% of tweets were written by members of the public, rather than by scientist or practitioners.³⁸ Another example would be one of the most popular research articles published at Plos One 'Fellatio by fruit bats prolongs copulation time'.³⁹ To date, its Altmetric score is 489 and it achieved more than 362,461 views, 317 tweets from 268 users, with an upper bound of 1,139,028 followers, was considered by 410 Mendeley readers and featured in 32 scientific blogs. Yet it has been cited only 8 times in Web of Science and 10 times according to Scopus. The demographic breakdown shows 82% of tweets were written by members of the public.

However, we should keep in mind that having 'sexy' words in title is not necessarily associated with a high Altmetric score. For example a recent article 'Vaginal and oral sex initiation timing: A focus on gender and race/ethnicity' has an Altmetric score of just 3.

Another weakness of altmetrics could be gaming. It is well known that there are spam companies selling tweets; retweets, Twitter followers, Facebook 'likes', positive comments, etc. Altmetric gamers can use Hootsuite to automatically post all of the tweets to Facebook and Google+ for example. They also can pay for an in-stream advert on Twitter.⁴⁰ Nowadays, some anti-gaming strategies are available.⁴¹ Altmetric.com claims to have more than 1.5 million papers and can distinguish between organic vs. artificial patterns of attention. If they define any gaming activity, eg tweeting through a spambot, no value will be added to the Altmetric score and a big red mark will be added on the details page. Also Altmetric can detect social media automation tools, eg auto-tweeting dam.⁴⁰ A well-known case of unintentional auto-tweeting which was detected by Altmetric would be 'Microcystis aeruginosa and underwater light attenuation in a hypertrophic lake (Hartbeespoort Dam, South Africa)'.⁴² This article had 38 unintentional auto-tweets from one user which counted as only one tweet. However, gaming is not a new issue and it is a factor which not is limited to altmetrics. Scientific communities have experienced the same problems with bibliometrics, eg the Brazilian citation scheme and coercive citation in academic publishing.^43,44

Conclusion

Altmetrics is fresh and emerging arena for dental research community. Altmetrics are intended to supplement bibliometrics, not replace them. We believe that dental clinical practitioners, research scientists, and journal editors must pay more attention to altmetrics as a new diverse and rapid tool to measure scholarly social impact.