Comparing a tablet computer and paper forms for assessing patient-reported outcomes in edentulous patients

In oral health, tooth loss is one of the best examples of conditions that impact negatively on self-perceived health status. Due to the essential role of teeth for mastication, facial appearance, and speech, their loss usually leads to a poorer oral health-related quality of life (OHRQoL).1 The summit of this problem is complete edentulism, which is highly prevalent in most developed and developing countries.2 Although complete dentures are traditionally used for treating edentulism, their success is variable and often cannot satisfy patient needs in terms of comfort and mastication.3

The use of patient-reported outcomes (PRO) is of utmost importance for clinical investigations in prosthodontics due to their ability to detect patient perceptions and preferences towards different therapeutic approaches. Moreover, the assessment of patient perception can lead the practitioner to better understand individual needs and consequently make more accurate treatment decisions.4 Generally, clinician-reported outcomes cannot evaluate the impact of health conditions comprehensively, and they fail to disclose aspects directly relevant for patients. The impact of a health condition depends on the patient perception of physical and psychosocial aspects rather than specific clinician-assessed indices.5

Main PRO used for edentulous patients include6: (1) satisfaction with received prostheses, often reported according to items regarding retention, esthetics, and oral hygiene7; (2) masticatory ability8; and (3) OHRQoL, which can provide a broader appraisal of the impact of oral health on function and well-being.9 Those PRO are generally evaluated by questionnaires whose items are answered according to visual analogue scales (VAS)7, 8 or ordinal/Likert scales.9 Other common PRO with application in the specialty involve the report of symptoms and pain/discomfort.10

The most common method for PRO is the use of paper forms filled by a patient or a professional. In spite of the traditional use of print forms, the same can be done by using electronic devices, such as computers or tablets. The use of an electronic method for data collection presents potential advantages, including a faster obtainment of answers, less physical space needed for data storage, ergonomics, and possibly more accurate answers for delicate questions.11 The use of electronic data collection can also improve patient compliance and simplify the organization of collected data, as long as the results can go directly into a computer.12 Current evidence favors the use of electronic forms in diverse areas of health, showing its equivalence and good acceptability compared to the use of print forms.11, 13, 14, 15, 16

Although electronic forms are potentially interesting for research in prosthodontics, their efficacy may be limited among respondents without training in informatics, therefore leading to some degree of anxiety.11, 12 This might be critical for edentulous patients, who tend to be older and frequently present low instructional levels.17 Such characteristics can influence the performance of computer-based forms; moreover, previous studies about the electronic application of PRO enrolled younger and more instructed participants,11, 12, 13, 14, 15, 16 in contrast to randomized trials on the treatment for edentulism.6

The use of electronic forms in clinical trials involving edentulous participants may be advantageous, but the peculiarities of that population demand further investigation in order to be valid. Although previous studies reported the use of a computer,11, 12, 13, 14, 15, 16 a tablet could be even more useful. A tablet can be handled directly in the clinical setting without demanding more space than a clipboard. We also expect the friendlier look and portability compared to a desktop computer to be appealing although it might elicit different responses from respondents. Therefore, this study compared electronic and print forms for assessing PRO data in edentulous patients wearing conventional complete dentures. The time used for data collection by both methods was also compared.

This paper reports a test-retest assessment performed on a sample of edentulous participants wearing maxillary and mandibular complete dentures. Each participant answered forms to collect PRO data according to two application methods (electronic or print forms) in a random order. A one-month period was given between applications.

This study was approved by the institutional Ethics Committee. Enrollment was preceded by informing possible participants about the nature of the study and provision of signed consent.

The study enrolled patients seeking treatment by conventional or implant-retained complete dentures in the School of Dentistry of Ribeirão Preto, from September 2012 to March 2013.

Included participants were adults (18 or more years) of any gender or ethnic group, without debilitating systemic diseases that could hamper data collection either by altering cognition (e.g., dementia) or by unnecessary discomfort (e.g., malignant neoplasia). They should also have no natural teeth or implants and wear conventional maxillary and mandibular complete dentures. A maximum age of the existing prostheses was not considered. However, we considered a minimum post-insertion time of one month, as a few participants might seek treatment immediately after receiving new dentures in other institutions. This period was considered in order to avoid assessment during the functional adaptation and adjustment period.9, 18 Illiterate volunteers were excluded from the sample.

PRO data collection procedures involved the use of two forms with distinct rating scales, widely used in clinical studies involving edentulous individuals. First, participants answered a questionnaire for denture satisfaction.19 This questionnaire was composed of 11 items answered on a 100-mm visual analogue scale (VAS). Each item dealt with the following criteria: (1) ease in cleaning, (2) overall satisfaction, (3) ease in speaking, (4) comfort, (5) aesthetics, (6) retention/stability, and (7) mastication. The second form was the OHIP-EDENT (Brazilian version), which aimed to measure edentulous individuals' OHRQoL.9 The version used was similar to the original form, except for the provision of answers on a three-point Likert scale for each of the 19 items, instead of a five-point scale. The sum of responses for each item provided a summary score ranging from zero to 38, with higher values indicating worse OHRQoL. A researcher quantified the time needed for filling the forms according to each method using a chronometer and checked for missing answers or typing errors during their application.

In a first appointment, a single examiner collected PRO data by either the electronic (A) or print method (B). A second data collection was scheduled after one month and was made with the other method. In other words, each participant could answer the forms in a single sequence (A followed by B; or B followed by A), with a one-month interval between them. The sequence of methods was randomly assigned for each participant by tossing a coin before the first data collection.

The application of the electronic method used a tablet computer (iPad 3^rd Generation, Apple Inc., Cupertino, CA, USA). The satisfaction questionnaire was transcribed into the tablet by the VasQ software (BottleCube Inc., Tokyo, Japan). Due to the different rating scale, the OHIP-EDENT's electronic version was prepared by the Google Docs Forms software (Google Inc., Mountain View, CA, USA). Answers for both forms were sent to a single investigator by e-mail and analyzed with the statistical software.

We estimated sample size based on a minimum significant difference of 20% on a 100-mm VAS and standard deviations from partial results. A sample of 30 participants divided into the two sequences would suffice for comparing methods with α = 0.05 and β = 0.20. The sample size was added 20%, resulting in 36 participants, in order to compensate for eventual dropouts.

Results for each denture satisfaction item, summary OHIP-EDENT scores, and application time were evaluated according to mean differences between the methods and a respective .95 confidence interval (.95CI). On an average basis, similarity between methods was assumed if the null hypothesis of similarity could not be rejected.

The Cronbach's α coefficient was used to estimate internal consistency for each of the tested forms. Furthermore, intra-class correlation coefficients determined the association between the methods, in the case of denture satisfaction items and the summary OHIP-EDENT scores. For OHIP-EDENT, the agreement for each item was quantified by the weighed κ coefficient. The global results for denture satisfaction (item 2) and OHRQoL (summary OHIP-EDENT scores) were further evaluated by Bland-Altman plots.

Statistical tests were performed using the SPSS 20.0 package (IBM SPSS Inc., Chicago, IL, USA), with α = .05.

Initially, 36 participants were enrolled among 72 patients (Fig. 1). The major reason for exclusion was no use of complete dentures for one or two arches, corresponding to 31% of the screened patients. Illiteracy was an issue for 15% of the 72 patients. Six out of the 36 participants who entered the protocol were missed during the study, corresponding to 17% of losses or dropouts. Most of these cases were due to the insertion of new complete dentures before the second appointment. No participant received any interventions on their existing dentures during the study, such as relining or adjustments. A single participant refused to provide answers after the first appointment.

Fig. 1
Flow diagram of participants through the study.

• Click for larger image
• Download as PowerPoint slide

The 30 participants (83%) who completed the study presented mean age of 64.4 years (range: 52 - 80 years) and were mostly women (n = 24). Most participants (n = 18) did not finish elementary school, three were elementary school graduates, and the remaining nine presented higher levels of education.

The responses for denture satisfaction-related items presented wide variation. However, results tended to be similar on an average basis, without significant differences for the tested methods (Table 1). The .95CI found were near 10 mm around mean differences, considerably smaller than the minimum significant difference of 20% mentioned above. Overall scores for OHIP-EDENT were high for both methods and also presented wide variation. Nevertheless, the mean difference between the methods was practically negligible and non-significant.

Table 1
Mean results and standard deviations (SD) for each item of the denture satisfaction questionnaire and the summary score for OHIP-EDENT, differences between the methods and respective .95CI. The table also provides the intra-class correlation coefficients (ICC) for the association between the application methods for each item and OHIP-EDENT summary score (α = .05)

• Click for larger image
• Click for full table
• Download as Excel file

Mean time (SD in parentheses) for the electronic and print forms were 9.2 (2.5) and 8.5 (2.4) minutes, respectively. The difference of 0.7 minutes was not significant (paired t test, P = .06), with a .95CI ranging from −0.03 to 1.8 minutes. Participants who needed more time with the electronic method tended to take even longer with the print forms (Pearson's r = 0.70, P < .001).

The internal consistency found with the electronic method was similar to the results of the conventional approach (Table 2). The Cronbach's α coefficient was near 1.0 for all instances, suggesting that the items were highly correlated within the form for both methods.

Table 2
Internal consistency of the denture satisfaction questionnaire and OHIP-EDENT according to each application method

• Click for larger image
• Click for full table
• Download as Excel file

The association between application methods was generally moderate for each denture satisfaction item (Table 1). The correlation was significant for all items except for “ease in cleaning, maxillary denture” and “overall satisfaction”. In turn, OHIP-EDENT's overall scores for tested methods were strongly correlated.

Answers for isolated OHIP-EDENT items presented varied degrees of agreement (Table 3). Weighed κ coefficients ranged from 0.30 to 0.65 and were significant in all cases. Six of the 19 items had good agreement levels, whereas 12 of them had moderate levels. A single question presented a fair degree of agreement.

Table 3
Frequency of answers and agreement between the methods for each item of the OHIP-EDENT instrument according to the weighed κ coefficient. For the scale values, ‘0 ’, ‘1 ’, and ‘2 ’ stand for ‘never ’, ‘sometimes ’, and ‘almost always ’, respectively

• Click for larger image
• Click for full table
• Download as Excel file

Fig. 2 presents Bland-Altman plots for overall denture satisfaction (item 2) and summary OHIP-EDENT scores. The discrepancy between the methods is small, reinforcing that systematic differences are insignificant. In general, this discrepancy is not dependent on the magnitude of measurements, e.g. the variation between methods is similar for participants regardless of having a poor or good OHRQoL. An exception for this non-dependence was observed for overall denture satisfaction when average measures are below 20 mm; this means that dissatisfied participants tend to provide more similar responses by tested methods. This latter PRO measure presented considerably ample limits of agreement (approximately 75% of possible measurements), suggesting modest precision. The summary OHIP-EDENT scores presented relatively narrower limits of agreement, ranging nine units (24% of 38) from the average difference.

Fig. 2
Bland-Altman plots for denture satisfaction/item 2 (overall satisfaction) and OHIP-EDENT summary score. Solid reference lines and dashed lines represent the mean difference between methods and limits of agreement (mean difference ± 1.96 SD), respectively.

• Click for larger image
• Download as PowerPoint slide

This study evaluated the average results of two methods for collecting PRO data from edentulous participants, as well as their agreement. The presentation of forms by a tablet computer was comparable to the traditional paper-and-pencil method. This was true for a questionnaire for denture satisfaction answered on a 100-mm VAS as well as for an OHRQoL instrument answered on a Likert scale.

The insignificant differences indicated that both methods could provide similar average results in a clinical assessment. Moreover, the high level of association between methods suggested that participants tended to provide similar responses on both application sessions. For denture satisfaction, correlation levels were slightly lower than those found in a study on Japanese patients.20 This can be explained by the different levels of education for the two study samples. However, OHIP-EDENT results (ICC and between-method agreement for each question) are comparable to the finding of the test-retest assessment done for the same Brazilian version, in which the application format did not change.9

Two satisfaction items did not achieve significant correlation, i.e., ease in cleaning maxillary dentures and overall satisfaction, suggesting that the contents may be relatively vague for the participants. That item should be considered carefully due to the possible influence of several other aspects that respondents may ponder. An example of those aspects is the satisfaction with life in general, which can influence response about satisfaction with dentures.21 A participant feels more favorably towards his life may report a more optimistic viewpoint regarding his/her prostheses. Although one cannot completely discard an association between the methods regarding general satisfaction, the level of noise was higher than that for other tested conditions. Questions about specific aspects of participants' experiences with their dentures seem to be more reliable.

The overall satisfaction item evaluates the effect of provided treatment as a whole, whereas other items approach explanatory aspects that may explain eventual differences. We expect that results for separate overall satisfaction questions with maxillary and mandibular dentures would be similar to the studied items.

This study found good agreement between methods in terms of OHIP-EDENT summary scores, even considering the large number of elderly participants. This finding suggests that the participants were able to understand the instrument effectively, regardless of the application method. A systematic review showed that the agreement between electronic and paper-and-pencil methods is inversely correlated to the age of respondents.12 Nevertheless, the age-agreement correlation was mild and was based on studies that used less user-friendly electronic equipment out of date. The same review pointed out an acceptable agreement between methods for samples with the mean age higher than 60 years, represented by correlation coefficients above 0.75.

High internal consistency levels for the OHIP-EDENT instrument were also among the present results. Such levels were unaffected by changing from print forms to the electronic method. Both methods presented Cronbach's α statistics comparable to previous findings, reinforcing the reliability of the used form.9 Furthermore, coefficients should surpass 0.7 in order to affirm that a form is able to measure a single theoretical construct. In other words, applying the OHIP-EDENT by a tablet does not interfere with the internal consistency levels, which is adequate.

Time spent for providing answers was similar for the electronic and paper methods. Even if the difference found were statistically significant, it would not be clinically important. Although the electronic method could not achieve significantly lower application time, simplified organization of collected data may reduce work time indirectly.12 This aspect was not within our study goals and was not quantified; however, it was perceptible when handling forms in both formats. Time needed for each method is affected by the fact that application was done in an assisted format, as done routinely by the researchers. Similar time intervals suggest that the level of assistance needed for each participant was alike, but this was not within the study goals. Moreover, unassisted application of electronic forms could result in different results and deserves investigation.

The education level of our study sample was a major limitation of this study. As abovementioned, lack of training with computers would lead to some degree of anxiety and therefore interfere with responses. However, no participant reported any concern regarding the acceptance or understanding of the forms in either formats, as found in other studies.11, 12 Even with a sample composed by a large percent of elders with low education levels, results were favorable. It can be deduced that younger or more educated participants would present more favorable results, as supported by the results from different populations. A study of the OHIP instrument on German participants obtained even higher reproducibility following a test-retest assessment.22 Although distinct sociodemographic results may provide different results,9 comparability between the methods for developed countries' populations should be at least as good as that in this study.

This study sample was representative of edentulous patients looking for treatment with complete dentures. Most of the exclusions of possible volunteers happened due to the presence of natural teeth. Participants should also present previous complete dentures in order to answer the satisfaction questionnaire. Illiterate subjects were not enrolled in order to achieve a reliable comparison between the methods. As expected from a sample of Brazilian edentulous patients,9 they would demand the application of forms as an interview. Such interference would conceal possible differences, as long as the contact of the participant with the forms would be indirect, leading to an important confounder. A possibility with illiterate or cognitively impaired participants would be the use of specific presentation formats, such as figure-based scales, which could be the subject of future studies.

Lost participants following randomization were within acceptable numbers, and reasons for dropouts were mostly unrelated to the study protocol (i.e., insertion of new dentures before applying the second method). Therefore, it can be stated that the influence of such occurrences on results is minimal and has no significance.

This study provides direct evidence regarding the use of a tablet for OHIP-EDENT, but electronic forms may work well with other formats of the OHIP. As mentioned above, edentulous participants provide an unfavorable sociodemographic scenario for using an electronic form. Therefore, it is likely that a tablet computer may achieve promising results for other participant profiles. Moreover, different forms of the OHIP instrument may behave similarly when applied by this method, such as the original 49-item version and its 14-item23 and 5-item short versions.24

The substitution of print forms by electronic forms applied by a tablet computer is viable. The tablet computer is comparable to the traditional approach and provides advantages such as automated data storage and less physical space needed. It can be recommended in clinical studies for the acquisition of data regarding important PRO of treatments for edentulism, namely denture satisfaction and OHRQoL.