Accuracy of acute burns diagnosis made using smartphones and tablets: a questionnaire-based study among medical experts

The cases presented in the survey were selected in three steps, as described below.

A sample of clinical experts was purposively enrolled with the requirement of having a competence in diagnosing burns, either by training or by clinical practice. The participants were contacted through one of the following three groups: 1) the physicians enrolled as tele-experts in the mHealth project that the research team is involved in [32] (burn specialists n = 2 and emergency medicine specialists n = 11); 2) burns specialists from Sweden that have shown an interest for this field of research and are themselves involved in related studies (n = 7); 3) burns specialists from other provinces belonging to the network of the tele-experts in the first group (n = 6). We assigned the participants into three groups based on their speciality and country of practice; South African emergency medicine (EM) specialists (n = 11), South African burns specialists (n = 8) and Swedish burns specialists (n = 7).

A web-based questionnaire was developed using SurveyMonkey™. The questionnaire started with questions on participants’ background, demographics, prior experience of remote consultations and type of device used (smartphone or tablet). Thereafter, the 51 images were presented along with a brief description (age, sex and mechanism of injury) (Fig. 1). The participants were asked to assess burn depth (four categories: superficial partial, mid partial/indeterminate, deep partial, and full thickness [33]) and size, expressed as TBSA (in percentage, free text). Additional comments could be made separately about each image. The questionnaire ended with a few final questions about the participants’ confidence when making a diagnosis based on the images in the questionnaire, if they would feel comfortable advising in a telemedicine system with similar images and whether they perceive images as helpful in making a diagnosis.

The first and second author contacted eligible participants via email with an information letter introducing the aim and objective of the study. A link to the SurveyMonkey questionnaire was sent in the email and they were asked to reply to this questionnaire using a handheld device of their choice (smartphone or tablet). The images were presented in a randomised order. Participants were allowed to complete the survey on different occasions by using the same login and resuming working from where they ended on the last entry.

Eight participants were asked to answer the questionnaire again on a laptop computer (minimum two weeks after the first session) at a later time. This was done on a voluntary basis (convenience sampling) and we asked only those respondents who could have access to the computer specifically used for this part of the study. All three groups of clinicians were represented among those who repeated the survey. They all agreed to this and were provided with the computer that was designated for the purpose with a fixed screen resolution and luminance (HP EliteBook 1040 G3 256 SSD with the screen: 14 in., 1920 × 1080 resolution). The images were presented to them in random order and the participants were given the possibility to complete the survey at their own pace in a quiet place.

Real time bedside assessment was considered the gold standard for both TBSA and burn depth. Diagnostic accuracy was assessed versus gold standard using a two-way mixed effect intraclass correlation coefficient (ICC) for both TBSA [34] and burn depth (as an equivalent for weighted kappa [35]). Six answers for TBSA (0.5%) and 27 for depth (2%) out of 1326 were left unanswered. For the accuracy analyses, these were treated as missing and removed. ICC was also used for inter-rater reliability of participants’ diagnoses of burn depth and size, and for intra-rater reliability between those participants that conducted the survey twice. For TBSA, the standard error of measurement (SEM) was calculated. For these reliability analyses, the missing answers were replaced by the answer that deviated most from the gold standard for that specific image. For TBSA this was based on the answer of another participant; for depth, the most distant category was used. Analysis was performed for all images combined, as well as stratified on patient age (children/adults) and speciality and country of practice of participants (three categories). A confidence interval (CI) of 95% was used for all ICCs.

The interpretation of ICC by de Vet [36] used by Hop et al. [37] for photographic assessment of burn size and depth was applied and is as follows:

A Bland-Altman plot was created to illustrate the differences in assessments of TBSA compared to the gold standard of each image. To measure the sensitivity and specificity of the assessments of depth, cases were dichotomised based on whether one could expect that surgery would be required (deep partial and full thickness) or not (superficial partial or mid partial thickness/indeterminate). Sensitivity was compiled by calculating the proportion of those burns with a gold standard as either deep partial or full thickness that were assessed correctly by the participants. Specificity was compiled by the proportion of the burns with a gold standard as superficial partial or mid partial/indeterminate thickness that were assessed correctly. Missing answers were excluded from both the Bland-Altman plot and the sensitivity analysis of depth. Stata/IC 12.1 for Windows was used for all analyses [38].

Using handheld devices for image-based advice among clinicians is spreading rapidly in a range of clinical settings [20, 21, 39‐41], but the practice lacks solid evidence as regards diagnostic accuracy. To the best of our knowledge, this is the first scientific study to investigate this question more closely, with a focus on acute burns, i.e. a type of injury that challenges clinicians in emergency care services. By and large, we observed that the results concerning burn size are somewhat better than those relative to burn depth, just as is the case for assessments on computer screens [12].

We found that, in all three participant groups, the assessments of burn TBSA was high for all cases aggregated and for child and adult cases separately. In addition, compared to when using a computer, the assessments on handheld devices were of comparable accuracy. The results are similar to those from prior studies [29, 37, 42]. With regard to clinical expertise, the burns specialists in our study had the same ICC as the burns specialists in an earlier study [37].

In the case of burn depth, which is challenging even at bedside [43], the assessments were less accurate. South African burns specialists were slightly more accurate than the other two groups and the assessments of paediatric cases were better than those of adults. In addition, there was not a remarkable difference between making an assessment on a handheld device than on a computer screen. It is of note however, that these results were considerably better than those from one study using photographs of patients [37] but poorer than another [42]. The finding that assessments of depth among South Africans were more accurate among burns specialists (overall and in children) than emergency medicine specialists is most probably a reflection of differences in training and experience. Another plausible explanation could be that TBSA is more important for initial assessment at the emergency phase to work out resuscitation requirements, while depth has greater implication when assessing the need for surgery, which involves burns specialists to a greater extent. This point was raised by some of the participants in their free text comments.

There are mixed results as regards under- and over-estimation of burns severity. When considering whether the patients would require surgery or not according to gold standard of depth, three out of four burns that were deep partial or full thickness (75.7%), and almost the same proportion that were superficial partial or mid partial/indeterminate thickness (70.4%) were assessed accordingly on handheld devices. This finding suggests that overestimation of depth occurs in similar proportions as underestimation among the participants. These results differ from an investigation by Hop et al. [37], where overestimation was more common with reported 66.8–83.8% sensitivity and 40.9–46.2% specificity.

This study contributes new knowledge on the potential accuracy of remote assessments of acute burns made on handheld devices by experts which is, as mentioned above, a practice on the increase and rapidly substituting laptop screens. A few elements making the study more robust are worth mentioning. To start with, the design was such that images were presented in random order to avoid any order bias and the participants were given the possibility to use a handheld device of their choice. The latter gave a sense of real-life setting but it is of note that the use of smartphones was far more common than that of tablets. An additional design-related strength is that, for the intra-rater reliability, the time lag between the two measurements (minimum two weeks but many times longer than that) was most certainly sufficient to avoid memory bias given the length of the questionnaire and the high number of images to assess. The high number of images included was a way to make the study more robust, as well as having all participants assessing all images. In line with this, answering the questionnaire was relatively time-consuming but the participants could take a break and resume to allow for flexibility and limit the risk of response fatigue. The questionnaire had a low number of missing answers and different methods were tested in how to treat these but neither of them affected the results considerably.

One limitation of note is the fact that on some images there were several burn areas with different depths. Whereas it was important to include these since it represents the reality of emergency care situations, images of burns with one depth only would have been more straightforward for assessment and analyses purposes. Expecting some difficulties in that respect, the participants were instructed to choose only one depth for each image and to focus on the proportionally predominant depth. However, for 15 images, the proportions were relatively similar, which puts the participants in a challenging situation. Low agreement has been reported earlier when images include a mixture of depths [42]. Although the number of images with mixed depths herein was relatively small, it is possible that it could have lowered the agreement. Regarding the sensitivity analysis of burn depth, actual data on surgery status was not available and therefore the depth categories are only serving as an indicator for potential need of surgery. Bedside assessment was used as the gold standard rather than other techniques such as the laser Doppler [44] since these are not in practice in burn care in South Africa. An additional limitation is that the setup of our study differs slightly from clinical practice where an image sent for consultation often comes with more information than what was provided herein, for example more detailed patient and injury characteristics as well as the opportunity to chat or follow up on phone. This means that our experts were forced to make assessments under less than optimal circumstances. This was also a point that was raised in the free text comments by the participants. There is therefore the possibility that the results are on the lower side of the spectrum and that assessments made in clinical practice could be even more accurate. A third potential limitation is that a larger sample of images would be needed to allow for more detailed stratification in order to provide more information about the characteristics of cases that are particularly challenging to assess. A total of 50 images have been recommended for sample size of validation studies [36] but for stratification by typical cases, a larger number of images would have been needed. This was considered when designing the study but the number of images was limited by the time it would require in terms of dedication from each participant. The mixed-effect ICC limits the potential for generalisability since it is a model that assumes that the participants are not a random sample of a larger population [34]. This model was considered appropriate since all participants reviewed all images, and the participants were selected purposely. This also means that the representability of the results on the participants’ country and specialty could be debated due to the small selection of participants from each country. It is of note however that in any specific country, the number of burns specialists is relatively low. Lastly, there is some imbalance between the participants in terms of demographics and practice of using handheld devices for image-based remote assessments. Being more used to working with smartphones and tablets could also facilitate making a diagnosis through these devices.

With the increasing use of handheld devices for remote assessments [20‐22, 39‐41], knowledge of the accuracy of diagnosing through this means is necessary. This study provides encouraging evidence on the applicability of, and the positive attitude among participants towards using smartphones and tablets for diagnostic support on burns. The levels of accuracy observed may represent under-estimation of the accuracy that could be reached in more “real practice” situations. Indeed, in this study, the participants were only provided with minimal information on the cases they had to assess but in several systems under development or implementation, chat functions can be used that can work as a good complement [32].

Whether some types of burns are more challenging than others – or for some particular groups of assessors – is also a question that deserves additional consideration in future studies. Finally, given the increasing use of the practice and the potential importance of familiarity with the type of cases to be assessed for the accuracy of the diagnosis set, providing specialists with training on image-based diagnosis could be beneficial. On the other hand, the satisfactory results of the Swedish participants indicate that remote advice could also be provided by non-local specialists.