Background
Features of the on-off design
Methods
Two examples of trials using on-off design
Trial 1. Controlled trial of a collaborative primary care team model for patients with diabetes and depression (TeamCare)
Trial 2. Healthy eating and active living for diabetes in primary care networks (HEALD)
Quantitative analysis
Qualitative data collection and analysis
Results and discussion
Our perceptions and experiences with the on-off design as researchers
Characteristic | Intervention (n = 95) | Active control (n = 62) | |
---|---|---|---|
n or mean (% or (SD) | n or mean (% or SD) | p valuea | |
Age (years) | 57.35 (9.97) | 59.18 (8.55) | 0.2 |
Sex (% female) | 58 (61 %) | 29 (47 %) | 0.08 |
BMI (kg/m2) | 37.07 (8.10) | 36.43 (8.19) | 0.6 |
Waist circumference (cm) | 114.0 (18.55) | 115.7 (14.50) | 0.5 |
PHQ-9 | 14.47 (3.85) | 14.59 (3.47) | 0.9 |
HbA1c (%) | 7.4 (1.78) | 7.8 (1.70) | 0.2 |
Systolic BP (mmHg) | 126.59 (15.46) | 123.54 (16.63) | 0.3 |
Diastolic BP (mmHg) | 75.98 (9.43) | 74.59 (8.22) | 0.4 |
LDL cholesterol (mmol/L) | 2.22 (0.86) | 2.15 (0.77) | 0.6 |
HDL cholesterol (mmol/L) | 1.09 (0.41) | 1.04 (0.36) | 0.4 |
Total cholesterol (mmol/L) | 4.38 (1.11) | 4.52 (0.98) | 0.4 |
Education to less than high school level | 11 (12 %) | 10 (16 %) | 0.4 |
Employed | 48 (52 %) | 31 (51 %) | 1.0 |
White | 81 (87 %) | 49 (80 %) | 0.1 |
Income (CAD) | |||
< $40,000 | 28 (30 %) | 16 (26 %) | 0.2 |
$40,000–$80,000 | 23 (25 %) | 16 (26 %) | |
> $80,000 | 26 (28 %) | 13 (21 %) | |
Refuse to answer | 3 (3 %) | 8 (13 %) | |
Smoking | 23 (25 %) | 15 (25 %) | 0.5 |
Alcohol use | 70 (75 %) | 41 (67 %) | 0.3 |
Psychoactive medicationsb | 46 (48 %) | 27 (44 %) | 0.6 |
Characteristic | Intervention (n =102) | Active control (n = 96) | |
---|---|---|---|
n or mean (% or SD) | n or mean (% or SD) | p value | |
Age (years) | 58.0 (8.2) | 61.3 (8.4) | 0.007 |
Sex (% female) | 46 (48 %) | 54 (54 %) | 0.4 |
BMI (kg/m2) | 34.6 (6.5) | 32.5 (6.5) | 0.04 |
Waist circumference (cm) | 112.5 (14.9) | 108.7 (15.4) | 0.09 |
Weight (kg) | 98.7 (20.6) | 93.6 (20.8) | 0.1 |
Resting heart rate (beats/minute) | 71.0 (10.9) | 70.7 (10.9) | 0.8 |
3-day pedometer steps | 16,761 (9238) | 19,075 (10,096) | 0.1 |
Glycemic index | 52.3 (4.6) | 51.1 (4.6) | 0.1 |
Glycemic load | 61.8 (15.3) | 57.7 (16.6) | 0.1 |
Energy intake (kcals)a | 1318 (452) | 1252 (467) | 0.3 |
Systolic BP (mmHg) | 125.8 (16.6) | 125.4 (15.9) | 0.8 |
Diastolic BP (mmHg) | 75.0 (8.7) | 76.9 (8.8) | 0.1 |
HbA1c (%) | 6.9 (1.2) | 6.6 (0.9) | 0.1 |
LDL cholesterol (mmol/L) | 2.2 (0.7) | 2.4 (0.9) | 0.1 |
HDL cholesterol (mmol/L) | 1.2 (0.4) | 1.20 (0.5) | 0.7 |
Total cholesterol (mmol/L) | 4.2 (0.9) | 4.5 (1.0) | 0.1 |
Education to less than high school level | 2 (1.9 %) | 4 (4.2 %) | 1.0 |
Employed | 67 (65.7 %) | 55 (57.3 %) | 0.6 |
White | 84 (82.3 %) | 78 (81.2 %) | 1.0 |
Income (CAD) | |||
< $40,000 | 26 (25.5 %) | 17 (17.7 %) | 0.8 |
$40,000–$80,000 | 39 (38.2 %) | 58 (50 %) | |
> $80,000 | 37 (36.3 %) | 31 (32.3 %) |
Healthcare providers’ experiences and perceptions of the on-off design
Discomfort with the on-off design
I think we would have had a lot more buy-in since this is already a protocol that has been proven in other places to have worked. We’re not necessarily testing the protocol so much as the applicability to this population and, honestly, is there that much difference between Canadians and Americans as far as whether or not a protocol would work? (PCN provider)
(The) On-Off has been one of the major challenges. And I know in a pure research laboratory [using], very scientific methods, it makes abundant sense to do that because that is the “gold standard.” But in reality, I think that’s why you see so few people-related studies that are having an On-Off group unless it’s like a pharmaceutical double-blind. (PCN management)
Conflict with professional commitment to provide care
In regard to the model of how the study is structured, I think the way that it’s not blinded – and I don’t know how else to look at the study. I have some research experience. But having an unblinded study where there’s the On group and the Off group and then this Off group doesn’t really get anything from us other than having to fill out questionnaires. (PCN provider)
And is the client gonna really go back to the doctor? I have no control over that. He may say, “Yeah I’ll go,” but whether he actually goes? I’m not following him up in 2 weeks and saying, “Hey, did you get over to your doctor yet? (PCN provider)
Patient safety should, in my mind as a clinician, have to come [first]. And it didn’t feel like it was. And that’s hard to do as a nurse. Especially as the legal expectation for myself that I would not compromise in any other setting. And I didn’t feel I did compromise because I felt trust in my existing team. But if we didn’t have the mental health team in place in order to refer somebody to, I would have felt that I was neglecting patient care. (PCN provider)
This fellow yesterday, just as an example, he was 21(on the PHQ)… his A1c was 11.1, his lipids were all elevated, blood pressure. And then I had to say, “Hmmm. Well thanks for coming out.” So that feels unethical to me somehow ‘cause we know we could offer something more. So I have a hard time with that. This fellow yesterday, he said, “I feel like my wife and I are falling through cracks,” and here he gave a hand out and I had to say, “Well thanks for coming out. Go back to your doctor” – you know? Just didn’t seem right somehow. Can we make him in my On group? Could I switch him? And then when he says things like, “I feel like my wife and I have fallen through the cracks,” it’s like oh here’s another one, you’re falling through the cracks ‘cause – he’s so depressed, he feels like he’s on a treadmill. He feels like giving up. He feels like jumping off a bridge. So now I have to say, “Hey wait a minute, go back to your doctor,” rather than saying, “Hey, why don’t you come back next week and see me and we’ll talk some more about this.” (PCN provider)
So that way you wouldn’t have this group of people that you feel some of them have been the people that needed it the worst, you know? Looking at their PHQ scores and talking with them even just a brief amount of time that you talk. Actually some of them I’ve spent quite a bit of time talking to because they’re a 20 (on the PHQ). You can’t really offer them everything. You think, “Oh I’d really like to follow up this individual,” and you can’t do it. So it’s kind of annoying – not right. Morally not right for them. So that’s one thing I would change. (PCN provider)
Support for research and suggestions for improving or altering study design
I think the whole On/Off, having the two groups and still seeing them in the same setting or context of a PCN. We realize you have to have a control but that was tough. Because you can’t really offer a whole lot to the people who are really struggling because of that Off group. It’s hard to kind of design. I don’t know if that was something that could be improved or worked on.’Cause you do have to have a control group somewhat. (PCN provider)
One of the challenges is the On and Off group idea. ‘Cause obviously it’s been really hard for the nurses to say, “Well, you’re demonstrating all these symptoms and maybe you’re not managing so well with your diabetes but you can’t be part of the program.” And so that’s been a struggle for us, for sure. Obviously a necessity in terms of how the project works and what we need the outcomes to be. But yeah, definitely struggling with that. (PCN provider)
See, it may be easier if somebody is used to research, this is just the way it goes. So you pull somebody in who’s a primary caregiver and put them into a position where they’re used to solving problems, they’re used to making suggestions. That’s what you do and then you have to take that away from them. It’s like you strip that ability away from them it’s like, “Okay, yeah.” (PCN management)
As far as ongoing challenges, I still think that the Off group and the design methodology has been a major barrier to success. And in the future, I have a very strong recommendation that that design methodology be reconsidered. (PCN management)
-
Intervention/on-group patients can act as their own controls using their past histories:
I’m not sure that you’re going to get better data or better results by having had this group of people that didn’t get the protocol. They could have been their own control group and then look at how many more people we could have offered it to. (PCN management)
-
Carefully match patients:
I think people can be their own controls or you can carefully match. (PCN management)
-
Focus on qualitative rather than quantitative evidence:
I know there’s other ways of doing research. I don’t know if you want it to be more qualitative than quantitative. (PCN provider)