Dynamic relationship between multiple START assessments and violent incidents over time: a prospective cohort study

The distinction between stable and dynamic risk factors in the structured approach to risk management is well-established and highly important in developing targeted interventions for risky individuals. Douglas and Skeem [1] differentiate the idea of a person having a relatively fixed risk status from that of being in a certain type of more-or-less temporary risk state. Risk status denotes an unchanging level of risk whilst the fluctuating nature of a risk state enables changes over time and situations to be considered in treatment planning. A focus on risk status prioritises risk assessment in terms of inter-individual ‘between persons’ variation to address the question ‘how risky is this person compared to others?’ Risk state adds to this an appreciation of intra-individual variation between and within-persons to ask ‘how risky is this person right now compared to others and compared to the past?’ If risk status is concerned with ‘who?’, risk state is concerned with ‘who when?’

The implication of adopting a focus on risk state is that risk assessments should be conducted repeatedly in order to track any potential changes over time. Since risk status is seen as a fixed aspect of an individual, one assessment will be sufficient to make predictions about risk and decisions about treatment over long time periods. In other words, dynamic risk assessments have short term validity whilst stable risk assessments are intended to have long-term applications and remain relatively unchanged between long term reviews. This time-limited function of dynamic risk assessment is acknowledged by both researchers and practitioners but most research on risk factors, whether stable or dynamic, is based on single point assessments and adopts a standard approach in examining the relationship between risk score and violent outcome i.e. inter-individual variation in terms of area under the curve (AUC) analysis [2]. Most mental health services do assess dynamic risk factors using repeated measurements as part of routine practice but the data from these repeated assessments seems rarely to be included in formal analyses of predictive validity [3].

A typical example is research relating to the Short-Term Assessment of Risk and Treatability (START) instrument [4]. This is an SPJ tool explicitly designed to focus on dynamic risk factors with 20 items rated in terms of their contribution as potential Strengths and Vulnerabilities to the individual and an overall specific risk estimate (Low, Medium or High) for a variety of target behaviours including risk to others, self-harm and unauthorised leave. These estimates are informed by, rather than calculated from, a composite of the individual item evaluations. Items can be highlighted for particular attention as ‘key’ factors on the Strengths scale and ‘critical’ factors on the Vulnerabilities scale and there is scope to note imminent danger under a separate ‘THREAT’ item. Assessments are based on case note reviews and discussion amongst multidisciplinary teams to establish the overall risk of various adverse events, including violence. The focus is on short-term risk assessment and evaluations can be conducted weekly if necessary. The instrument was developed from 2004 onwards primarily within forensic mental health settings but with intended applicability to other types of psychiatric service.

Given the short time since its inception, there is a relatively substantial empirical literature on the properties of the START [5]-[10]. This includes examination of its implementation, clinical utility [11],[12], internal consistency and inter-rater reliability [13]. Evidence of its predictive validity in seven studies has been summarised by Chu et al. [14] with AUC estimates for Vulnerabilities ranging from 0.66 to 0.83 and those for Strengths ranging from 0.65 to 0.77. However, despite the inherently dynamic nature of the instrument, all of these studies adopted the standard approach of calculating the relationship between an initial START score at a single time point and violence at a small number of subsequent points in the future (e.g. one year, five years and ten years). As a result the dynamic nature of the START is ignored and there is no attempt to examine how changes in risk factors are associated with changes in violence risk. This oversight is a common feature of the general risk assessment field, not just a problem with START research [15].

Wilson et al. [16] noted this problem with existing research and adopted a somewhat more dynamic approach by estimating the relationship between START scores at multiple baselines and subsequent violence at multiple endpoints. They found that “changes in dynamic risk factors were significantly associated with institutional violence, even after controlling for static risk factors”. However, the five time periods in this study were still relatively lengthy at three months when it is clear that many dynamic factors fluctuate much more rapidly, over days and even hours. There was also no attempt to examine intra-individual variability in this study.

The study reported here was designed to go one step further and examine the dynamic relationship between START scores and violence over periods of 30 days. Furthermore, it was intended to examine the variability in degree of dynamic changes between individual patients. Since some patients may be stable overall and others may be highly variable in their presentation, intra-individual variation in these fluctuations and their relationship to violence were examined.

The study took place on a Medium Secure Unit in England following approval by the North West England 1 Research Ethics Committee. The unit provides care and treatment to males and females who present a danger to the public due to mental illness but who do not require high secure care. It has 66 beds on six single-sex wards (one female).

The START was introduced into routine practice in 2008 with an intention for completion on every patient every two weeks to inform multidisciplinary team discussions. Assessments were conducted by all qualified nurses working in the unit at the time and some patients were assessed repeatedly by the same nurse. The assessments contributed to treatment and discharge planning but the final decision about discharge was taken by the responsible medical clinician. All START assessments completed from October 2008 to January 2011 (27 months) were eligible for inclusion in the study. Over this period there were 110 admissions to the unit. Assuming the 66 bed unit had permanent full occupancy for the entire 27 month period and two assessments were conducted per month as per the local procedure, 3,564 assessments should have been conducted.

Ultimately, 540 records were provided to the research team which represent 15.2% of all potential assessments. Furthermore, only 475 of these assessments (13.3%) achieved a threshold of having 14 or more completed items and were included in the study. Where data on up to six individual items were missing, scores for the missing items were imputed from the mean score for all completed items and added to the total for completed items to produce an imputed total score. Assessments with seven or more missing items were excluded from the analysis. As approved by the Research Ethics Committee, patient consent for data analysis was not required as it was based on anonymised routine clinical data.

The 475 included assessments were conducted on 50 patients (45.4% of admitted patients). The median number of START assessments was 8.5 (IQR = 10.8) with a range from one to 25. Five patients (10.0% of the sample) accounted for 22.3% of assessments (n = 107 assessments). The median number of days between START assessments was 22 (IQR = 20.1) but in some cases there was up to four months between assessments. The median from admission to the first START assessment was 312 days (IQR = 563.0) but this varied markedly from two to 1580 days (> four years), indicating substantial differences in mental state acuity on entry to the study. Data on index offences in the sample were not available.

Of the 50 patients, 44 (88.0%) were male and 43 (86.0%) had a diagnosis of schizophrenia. Their mean age was 38.6 years (range 19–65) and the median length of admission regardless of the study start or endpoint was 829.5 days (approximately 28 months, IQR 799.5, range 120–2297 days). Eighteen (36.0%) had been discharged by the study endpoint. Compared with the total number of admissions during the study period, the sample did not differ substantially in age or sex distribution, but the study sample had a higher proportion of patient with a diagnosis of schizophrenia (rather than schizoaffective disorder) than the overall unit population during the period.

Violence data for these 50 patients over the period up until November 2011 were obtained from the formal adverse incident recording system run by the health care trust. Violent incidents were defined as episodes of physical or verbal aggression, including property damage. There were 275 incidents involving 26 (52.0%) of the patients. The median number of incidents per patient for all patients (violent and non-violent) was 2.5 (IQR = 7.75) with a range of 0–58. Five patients (10.0% of the sample) accounted for 65.4% of incidents (n = 180 incidents).

This study sought to examine the nature of dynamic risk in a sample of mentally disordered offenders by tracking changes in risk over time in the overall group and by examining the relationship between change in risk factors and the occurrence of violence. The essential question is: how stable or fluid are the risk factors which are generally considered to be dynamic?

In terms of context, the mean START scores at the first assessment indicate that the degree of risk presented by this group is comparable to that in other samples studied during the development of the instrument [11]-[14]. In addition, there is some evidence of tool validity here in that discharged patients had a substantially better change in risk factors (reduced Vulnerabilities and improved Strengths) compared to those who were not discharged in the study period. Some circularity is possible as discharge decisions will be based on a range of information sources of which START assessments are only one. Increased discussion of the potential for discharge based on these other sources (e.g. unstructured clinical judgement) could lead to assumptions by the assessor about improvement which then influenced START assessments in a form of self-fulfilling prophecy. However, the strength of the differences in START scores between the discharged and non-discharged group is preliminary support for the concurrent validity of the instrument. Also, in this setting, the nurses making the assessments would participate in discharge discussions as part of the multidisciplinary team but would not make the final discharge decision.

Predictive validity as estimated in the standard single point approach was acceptable in both domains and within the range of AUCs produced from other studies adopting this approach to analysis [14]. AUCs for Vulnerabilities in both ROC analyses were comparable to those obtained for other risk assessment tools examined in a recent systematic review [18]. The effect size for Strengths was also comparable for the first assessment predictions but not when all assessments were considered.

The key finding here is that risk factors which are considered to have the capacity to fluctuate dynamically did so only in a few patients, and did not do so in most of the patients included in this group. The ICC values for both domains were approaching 1, indicating that a very high proportion of variance in scores could be attributed to stability within patients. In addition there were mixed results in terms of whether change in risk score is associated with the occurrence of violence. A ten point (i.e. 25%) increase in START Vulnerabilities scores was associated with a threefold increase in the likelihood of violence over a four week period though this elevated likelihood was less than double when within patient associations were examined in the subsample with available data. Changes in Strengths score were not associated with any increased risk in the whole group although again risk was approximately doubled when the within patient association was examined. These within patient associations are the most comprehensive analyses but they are inevitably compromised by the small sample (~15 patients) with violent incidents in the relevant period which means that the estimates, however suggestive, are extremely imprecise.

The lack of fluctuation in risk may be accounted for in a variety of ways. The setting was a medium secure unit where the average length of stay is over two years. Therefore, whilst some participants will have been assessed shortly after admission from prison or the community, many of the patients will have been under treatment in the unit for a long period and thus their mental state will have been relatively stabilised. Alternatively, it may be that ‘staff cognition’ is not as dynamic as the tool itself. If staff have become somewhat de-sensitized to changing risk presentations or are drawn away from more regular patient contact they may miss the nuances of change in each individual. They may then assume stability and be blind to changes when conducting the assessment. This may be compounded by the 3-point scale for completion of each item which is relatively insensitive to small changes. Regular completion every two weeks furthermore will inevitably lead toward a ‘tick-box’ mentality where the primary task becomes completing the documentation rather than using the documentation as a tool to engage with the patient. The pattern of missing data also suggests some tendencies which may distort accurate completion. In particular, the assessments here were always completed by nursing staff and items which were regularly missed e.g. medication adherence, material resources, may have been considered as the responsibility of other members of the multidisciplinary team i.e. doctors and social workers.

The main aim of this study has been to apply a new analytical approach to the study of dynamic violence risk. There are clear limitations in terms of the relatively small sample size, reliance on imputed data to fill gaps and the variability of timescales between START assessments and between admission and entry to the study. These limitations arise largely from the use of data collected in everyday practice for clinical rather than formal research purposes, a context which, on the other hand, increases the ecological validity of the assessment process. The clinical implications of the study are also limited as total scores were analysed rather than the summary risk ratings recommended by the instrument developers as the basis for clinical decisions. The next step is to address these weaknesses in data collection by conducting an adequately powered study with fully completed measures collected at regular time points. Such a study is under development and will enable a more valuable application of the approach to be conducted when it is completed.