Background
Schizophrenia is distressing and disabling to the individual [
1], with an associated cost in the United Kingdom of around 6.7 billion pounds each year [
2]. Clinical outcome is usually poor despite treatment, with 80% relapsing by 5 years after the first episode. The major need is for better treatments. Treatment development is slow in this area, with a high rate of failed clinical trials. Currently, we assess treatments by asking patients to recall symptoms over the last 7-28 days, using widely-used semistructured symptom assessments such as the Positive and Negative Syndrome Scale (PANSS [
3]) and Calgary Depression Scale (CDS [
4]). This introduces bias and averaging, thus clinical information is lost. In addition, standard rating scales require training of raters to ensure high reliability, often difficult to achieve and maintain in multisite studies. For instance, a decrease in between-rater intraclass correlation from 0.9 (“high”) to 0.7 (“acceptable”) on PANSS full-scale score will reduce the power of a study to show an effect from 90% to 72% [
5,
6]), increasing the risk of a Type 2 error and a failed trial. An advance is needed in the real-time documentation of psychotic symptoms. One under-explored possibility is that of patient self-rating of symptoms.
There is scepticism as to the validity of self-report measures of psychosis. This view is often motivated by knowledge that cognitive deficits [
7] and lack of insight [
8] are common in patient populations. However, moderate concordance has often been observed between self-report measures and clinician based ratings of psychosis, which has been demonstrated in a range of symptom domains. This includes delusions [
9], hallucinations [
10], and negative symptoms [
11]. Self-report measures may be a more time and cost efficient method of assessing psychosis than clinical interviews, as they do not require the presence of a trained assessor. Thus, self-report measures may be the more attractive option for clinical assessment.
Over the past decade Personal Digital Assistants (PDAs) have been adapted for self-report symptom monitoring in individuals with severe mental illness [
12]. Studies evaluating PDAs have shown low rates of drop-out in community dwelling individuals with psychotic disorders [
13‐
15]. For example, Granholm and colleagues [
13] found that 87% of patients were compliant to PDA based momentary assessment as defined as completing at least four out of 28 data-points. Other studies have observed similarly low rates of drop-out when using more conservative definitions of compliance (eight out of 28 data-points) [
15].
PDAs are offline systems and whilst the data is collected in the real world it cannot be assessed until brought into the laboratory/clinic and downloaded. Assessing data in
vivo is desirable in that it could help to facilitate earlier and more immediate intervention, which in turn could help to reduce relapse, self-injury and the need for unscheduled acute care. Automated and personalised feedback could help clinicians to devise and review treatment strategies prior to consultation allowing for more effective care. An appropriately enabled mobile phone may have the advantage that people are accustomed to carrying and recharging it and are often familiar with the technology. Software applications are also easily uploaded to participants’ own smartphones ensuring that the individual does not have to carry with them an additional device. In a recent Ofcom report in the United Kingdom, 27% of adults and 47% of teenagers currently owned a smartphone [
16]. With advances in mobile phone technology PDAs are becoming increasingly obsolescent.
The first objective of this study was to evaluate and validate new mobile phone based self-report assessment scales for psychosis against the PANSS and the CDS, both widely used retrospective interview assessments of psychotic and related symptoms and considered to be benchmark scales accepted by regulatory authorities in clinical trials. Scales were specifically developed for purpose-built smartphone assessment software (i.e. ClinTouch) in order to monitor psychotic symptoms in real time. The second objective was to assess feasibility and acceptability to patients with serious mental illness, examining levels of compliance and drop-out to the procedure in individuals at different stages of psychosis. We also aimed to examine the internal consistency of the scales and their instability over time. In order to gauge the feasibility of installing this software onto participants own phones, we assessed the extent to which participants used mobile-phone technology in their everyday lives. In order to assess safety, in that that this approach did not cause distress, we assessed “reactivity” to the method as reported by participants at the end of sampling.
Thus, the study had two main hypotheses. First, that symptom data collected over a smartphone software application would show good correlations with corresponding data collected by conventional, gold standard rating scales. Second, high levels of compliance and low dropout from smartphone based assessment would be possible in individuals at different stages of psychosis (ultra-high risk, acute and remitted). In this study we also predicted that the self-report scales would show high internal consistency (α coefficients), but be sensitive to change, as represented by instability across time-points. No predictions were made as to participant’s level of phone use in their everyday lives.
Results
Adherence to the methodology
Initial verbal approach to participate was made by a member of the clinical care team and about 50% of those approached declined to take part. Of the 51 patients who agreed to be contacted about the study and had their contact details passed on to the research team, four subsequently declined, two were ineligible and one could no longer be contacted.
Compliance to the methodology was defined as completing at least 33% (14 or more) of all possible (42) entries. In all, 44 participants consented to and entered the study to ensure that 36 met this compliance criterion after 7 days: in other words, 82% of participants met the compliance criterion. Six acute and two remitted patients with psychosis failed to meet this criterion (Mean age: 31.5 (SD; 11.1), all male). Logistic regression analysis was performed to examine whether positive, negative and general subscales on the PANSS (prior to sampling), CDS total score (prior to sampling), or age significantly predicted whether an individual was compliant with the methodology. Positive symptom subscale severity was the only significant predictor (OR = 0.68, p = .033, CI: 0.48 – 0.97). The 8 non-compliant participants are excluded from all analyses subsequently presented in this manuscript.
A high number of entries were completed by the 12 acute (Min = 14, Max = 41, Mean = 28.5, SD = 8.1), 12 remitted (Min = 14, Max = 40, Mean = 29.5, SD = 9.3) and 12 UHR (Min = 21, Max = 38, Mean = 31.1, SD = 6.6) participants who were compliant with the procedure. Thus, on average, the aggregated sample completed 31.1 of all possible data-points (72%). A one-way ANOVA showed these differences to be non-significant across groups (F (2,35) = .312, p = .734). Multiple regression analysis was performed to investigate whether age, gender, PANSS subscales, and CDS total predicted the total number of diary entries completed by each individual. There were no statistically significant predictors.
Reactivity to the method
Reactivity (changes in thoughts or emotions) to filling in the questions was greatest in the acute group (mean: 3.6 (SD: 2.4)), and greater in the remitted (mean: 2.9 (SD: 1.5)) compared to UHR individuals (mean: 2.4 (SD: 1.7)). A Kruskall-Wallis test showed this difference to be statistically non-significant (x2 = 3.351 (df: 2), p = .187). Regression analysis was used to assess whether positive, negative or general symptoms on the PANSS, or CDS total score, significantly predicted reactivity across all three groups. Only negative symptoms predicted greater reactivity to the method (β =.54, p = .001).
Correlation between momentary assessment and interview subscales
Summary statistics for the mobile-phone assessment items and clinical interviews are provided in Table
2. The standard deviation
(SD) score reported in this table represents variability between individuals’ mean scores (not within individual variability). The CDS item 2, guilty ideas of reference, was only ever endorsed by two participants and was therefore not analysed.
Table 2
Summary statistics for interview and diary subscales, and the results to Spearman's correlations (in order of strength)
Hopelessness (CDS) | 0.87 | 1.0 | 6.7 | 3.3 | 1.5 | 1.0 | 3.0 | 1.8 | 0.8 | 0.80 |
p <.001 |
Delusions | 0.93 | 1.0 | 5.3 | 2.0 | 1.3 | 1.0 | 5.0 | 2.6 | 1.3 | 0.74 |
p <.001 |
Anxiety | 0.96 | 1.0 | 6.0 | 2.7 | 1.5 | 1.0 | 5.0 | 3.0 | 1.1 | 0.69 |
p < .001 |
Hallucinations | 0.96 | 1.0 | 6.4 | 2.6 | 1.7 | 1.0 | 5.0 | 2.4 | 1.5 | 0.68 |
p < .001 |
Suspiciousness | 0.95 | 1.0 | 6.2 | 1.0 | 6.2 | 1.0 | 5.0 | 2.5 | 1.3 | 0.63 |
p <.001 |
Grandiosity | 0.76 | 1.0 | 4.2 | 2.0 | 1.1 | 1.0 | 4.0 | 1.5 | 1.0 | 0.53 |
p <.001 |
Depression | 0.83 | 1.1 | 5.9 | 3.4 | 1.2 | 1.0 | 4.0 | 3.0 | 1.2 | 0.45* | 0.006* |
Guilt | 0.95 | 1.0 | 5.7 | 2.0 | 1.2 | 1.0 | 5.0 | 1.7 | 1.1 | 0.44 | 0.006 |
Somatic concern | 0.96 | 1.0 | 7.0 | 3.1 | 2.1 | 1.0 | 5.0 | 1.7 | 1.1 | 0.39 | 0.019 |
Passive apathetic social withdrawal | 0.93 | 1.0 | 6.9 | 4.1 | 1.6 | 1.0 | 3.0 | 1.8 | 0.9 | 0.26 | 0.131 |
Hostility | 0.86 | 1.0 | 5.0 | 2.4 | 1.2 | 1.0 | 7.0 | 1.9 | 1.2 | 0.25 | 0.145 |
Excitement | 0.89 | 1.0 | 6.8 | 3.7 | 1.7 | 1.0 | 4.0 | 1.5 | 0.9 | 0.06 | 0.712 |
Conceptual disorganisation | 0.95 | 1.0 | 5.0 | 2.0 | 1.1 | 1.0 | 3.0 | 1.6 | 0.8 | -0.04 | 0.832 |
The strength of the associations between the diary and corresponding interview subscales varied considerably (Table
2). Hopelessness, delusions, anxiety, hallucinations and suspiciousness diary items showed strong Spearman’s correlations with the corresponding items on the CDS and the PANSS (
rho > .60). Moderate and still statistically significant correlations were also observed for grandiosity, depression, guilt, and somatic concern (
rho > .35). However, passive and apathetic social withdrawal, hostility, excitement, and cognitive disorganisation were not significantly correlated with their corresponding PANSS subscales.
The internal consistency and instability of the scales
As can be seen in Table
2, the alpha scores for each of momentary assessment scales were high suggesting good internal consistency. The
MSSD and
SD scores for each momentary assessment scale are displayed in Table
3. A greater score represents greater instability across time. The delusion instability score was only calculated in individuals who triggered the delusion questions at briefing. All momentary assessment scales showed some instability across time. Passive apathetic social withdrawal was the least stable subscale, followed by excitement, conceptual disorganisation and anxiety. Delusions and grandiosity were the most stable self-report scales.
Table 3
With in subject instability metrics for all scales
Hopelessness - CDS | 1.3 (1.4) | 0.8 (0.5) |
Delusions | 0.7 (1.3) | 0.5 (0.4) |
Anxiety | 1.9 (1.7) | 1.1 (0.6) |
Hallucinations | 1.2 (1.6) | 0.6 (0.5) |
Suspiciousness | 1.1 (1.9) | 0.6 (0.5) |
Grandiosity | 0.9 (1.1) | 0.6 (0.5) |
Depression | 1.0 (1.0) | 0.8 (0.4) |
Guilt | 1.8 (2.1) | 0.9 (0.6) |
Somatic concern | 1.2 (1.4) | 0.7 (0.6) |
Passive apathetic social withdrawal | 2.7 (2.0) | 1.3 (0.6) |
Hostility | 1.9 (2.7) | 0.9 (0.5) |
Excitement | 2.2 (2.7) | 1.0 (0.6) |
Conceptual disorganisation | 2.1 (2.3) | 0.9 (0.6) |
General phone usage of the sample
A series of questions were asked to gauge the feasibility of using the ClinTouch software on participants’ own phones. Of the sample of 36, 83.3% currently owned a mobile phone, and 44.4% owned a smart phone (30.6% with a touch screen). Phone use included individuals who were acute (66.7%), remitted (91.7%) and UHR (91.7%). 63.9% of individuals with mobile phones reported that they kept these on them all or most of the time, with an identical number usually or always taking their phones with them when they went out. The sample had owned a mean of 8.3 (SD = 7.0) mobile-phones devices. 86.1% of the current sample reported that they would buy a new phone in the future.
Discussion
This study attempted to examine the validity and feasibility of a self-report scale for assessing psychotic symptoms on appropriately enabled mobile phones. The results suggest that the methodology is both feasible and acceptable across different stages of psychosis. Additionally, the data support the validity and reliability of several of the momentary items, suggesting that they pose a useful alternative to traditional symptom assessment.
The number of individuals dropping out of the study was relatively low across remitted and UHR samples, although slightly elevated in acute patients, where a third of individuals were non-compliant. This may explain the finding that positive symptoms significantly predicted non-compliance to the procedure. This supports the notion that momentary assessment is a relatively demanding approach, to which certain more symptomatic and chaotic patients may have difficulty in remaining compliant [
21]. Thus, in acute settings it may be beneficial to adapt the momentary assessment procedure (e.g. sampling rate, item number) to individual’s preferences and needs, or use an alternative method of assessment.
In compliant individuals, the number of assessment occasions was relatively high and similar to past momentary assessment research using PDAs in this population. For example, Swendsen and colleagues [
15] observed an identical completion rate of 72% of all data-point completed, whereas Granholm and colleagues [
13] found this to be 69%. In our study the number of entries was non-significantly different between the groups, suggesting that although a subgroup of acute patients struggled to complete the minimum number of entries, the majority were just as able to comply with the procedure as those with more attenuated symptoms. It should be noted that although compliance was high in this study rates of refusal to initially take part could not be assessed. Furthermore, socioeconomic status and reading ability were not considered, which may have predicted levels of non-compliance.
Reactivity to the methodology was minimal across the groups, although it was slightly elevated in individuals with greater levels of negative symptoms. This may explain why these symptoms have been found to predict drop-out in experience sampling studies (unpublished observation). Important to note is that reactivity could not be assessed in individuals who dropped out of this study and did not complete any diary entries. It is possible that greater levels of reactivity may be observed in non-compliant participants.
In line with the hypotheses, correlations with PANSS and CDS subscales were mainly significant, although there was considerable variability. Positive symptom scales (i.e. delusions, hallucinations, grandiosity, somatic concern and suspiciousness) generally showed moderate to strong correlations with their corresponding PANSS scales. Affective symptoms, including hopelessness, anxiety, guilt and depression, also significantly correlated with the interview measures. Therefore, ClinTouch appears to collect data which is comparable to traditionally used, gold standard assessments of psychotic symptoms and mood.
Passive apathetic social withdrawal, excitement, hostility and cognitive disorganisation items showed weak and non-significant correlations with their corresponding interview scores, requiring further consideration. There are several possible reasons for this finding. Most important is that the equivalent PANSS item ratings are based largely on observable behaviour during the interview, often supplemented by the reports of clinical staff and family members. Replicating this in a self-report item is a challenge. This is not to say that either holds a more valid or clinically useful viewpoint, but rather that they assess different constructs. Also, hostility and excitement represent socially undesirable behaviours, which patients may not associate with themselves or may wish to underplay in self-report measures. Finally, there was a limited range of scores observed on the apathetic social withdrawal and cognitive disorganisation PANSS subscales, which may have attenuated the correlations with the momentary assessment scales.
All of the mobile phone self-report scales showed instability (ie fluctuations) across time as shown by high within subject
MSSD and
SD scores, suggesting that they were sensitive to subtle shifts in symptomatology. Indeed, the mood scales (i.e. anxiety, depression and guilt) showed equivalent or greater levels of instability than typically employed experience sampling scales [
27]. Delusions and grandiosity were the most stable across time potentially suggesting that these reflect relatively fixed and inflexible belief systems. Passive apathetic social withdrawal showed the greatest instability, perhaps representing changes in the individual’s inclination to be around others. All of the self-report scales also showed good internal consistency.
The advantages of using technology to monitor mental illness have recently been documented [
28,
29]. Ambulant monitoring provides detailed information about an individual’s symptoms across a variety of situations and times of the day. This could generate discussion points for consultation; identify ‘relapse signatures’; and highlight momentary symptom triggers. It could also be used to monitor real-time acute phase medication treatment effects in the early stages of intervention [
30]. This is important given that most clinical improvement is now known to occur within the first 7-days after receiving antipsychotic treatment [
31,
32]. Furthermore, mobile assessment techniques can be adapted for use alongside psychosocial intervention [
33]. For example, person-tailored interventions could be triggered when an individual’s symptom score reaches a certain threshold or to facilitate ‘homework’ [
34]. In research, it will also potentially allow better clinical phenotyping, and stratification for clinical trials.
Perhaps the greatest strengths of ClinTouch are that it offers automatic wireless uploading of clinical information to a central server and can be installed on patients’ own phones, thus obviating the need to carry a special purpose device. Furthermore, smartphone technology may be more user-friendly and time-efficient than text-based systems [
35]. We observed that the majority of this sample currently owned and regularly used mobile phone technology, many of which were smart phones. With advances in technology it is likely that advanced mobile phones will become increasingly affordable and widespread, and this will make it a viable option for clinical assessment within clinical services. Future research will need to evaluate the merits and pitfalls of this approach.
Previous research in the area of telehealth and telecare devises suggests a need for deeper understanding of how ClinTouch is used in practice to identify the factors that facilitate implementation of this device. As the field of new technology in mental health aspires to moves beyond demonstration and towards the embedding of devices such as ClinTouch in everyday clinical practice, there is a need to engage methods and sub-studies that are able to describe the processes, identifying facilitators to context specific and successful implementation of telecare [
36]. Qualitative methods are being used to consider the social practices behind the integration and incorporation of the ClinTouch technology. Understanding their interactions with professionals and the synergy or otherwise with clinical expectations will inform its future use.
Competing interests
No competing interests, financial or otherwise, arise from this research.