Background
Evidence-based guidelines are overwhelmed by the rate of research publication
Limited literature addresses adoption of automation
Diffusion of Innovations
Research questions
Methods
Participants
Data collection and analysis
Stage 1: Assignment within predefined frameworks
Stage 2: Open coding within the Diffusion of Innovations framework
Stage 3: Generation of themes
Stage 4: Generation of matrices
Stage 5: Identifying patterns and outliers
Results
Participants
Overview
Compatibility
How you synthesize it, how you pull it together is kind of key. Participant 3
I think it would be a shame if humans weren’t involved in [synthesis]. Participant 9
Ability to double-check
I can see it could be done. But surely it would need to be checked by someone anyway. Because even if it’s done by a human with vast experience, it’s always important to have a second person to check it. Participant 5
At the minute the standard is for two operators. So you’d want it to have been checked by a second method, if not person. So that would be my only thing – reproducibility. Participant 7
Transparency as accountability
A group of experts can apply judgement to that body of evidence, and needs to know they can trust the evidence that you’d found. Participant 12
The key part of working with a face to face committee … Is you have they have to have total confidence in what the technical team has done. Participant 16
Relative Advantage
Freeing up human resources
In research time is always limited and you know there’s never enough grant money to help employ staff … by having a machine do it, it would be cost-effective, and spare the researchers’ time to do other research-related tasks. Participant 17
Time and cost saving
No matter how quickly a guideline’s done, everybody always wants it faster and to be of high quality. So anything that can improve on that would be welcome, I think. Participant 11
Observability
Need for evidence
I think at the moment it has a potentially high level of risk of being incorrect. But I don’t really know enough about it. I’d need to be convinced about it I think to consider it. Participant 9
If the whole process were done by some machine or machine learning application, I think it would need to be properly trialed. Participant 5
As long as there was clear data to support that … machine-learning is a reliable method, but you know, better than or equal to humans doing it. Participant 17
I do think it’s been well demonstrated for the screening aspects, for the hit rates of what gets included and what doesn’t, and how correct it is. Participant 11
Personal need for double-checking
The thing that’s sort of a little bit distressing from a novice point of view with machine-learning is not feeling like I have a way to check it… I’d need some way to be confident …. [I’d need] a way to check the algorithms. Participant 3
Complexity and Trialability
Whenever you try and really change things, I think there’s a degree of skepticism anyway…I think that might just be the nature of human beings. Participant 9
If they have to learn the process, and if it’s hard, then that sort of discourages them. Participant 18
So unless the technology offers a value add that’s substantial enough to overcome the learning curve…however much time it takes to do that has to not be more time than you’re gonna save. Participant 3
Contextual themes
Participant familiarity with automation
I’ve done a very little bit with machine-learning. Participant 3
It’s just my concern would be that I’ve not had any experience with it. Participant 7
I haven’t had much to do with machine-learning. Like I’ve kind of heard about it. Participant 17
I think that’s something I have no personal experience with. Participant 11
To be honest I actually haven’t had much experience with it. Participant 8
Yeah, I don’t know, I don’t really understand that process. Participant 5
Overall skepticism towards machine learning
It would be very difficult to train a machine to make the sort of value decisions that we have to make. Participant 10
I’m still a bit nervous about some of the interpretation of that…it just might be a distrust about it, I think? Participant 13
How can a computer apply judgement? …There’s judgement required when it comes to things like quality or – they are not things I expect to be evidence that could be accurate. Participant 12
I don’t think it could fully replace a human … I think there can be subtleties between how things can interact… I think there’s always going to be some sort of human element. Participant 9
I don’t know if we’re there yet. Maybe we’ll get to the point where we can do that, but to do that, like quality rating, or to do a level of evidence, or strength of evidence… I mean there’s still a lot of value judgements in that. And I don’t know how much machine learning could help with that at this point. Participant 3
Discussion
Cultural standards of practice greatly influence decision-making
Researcher effort will be redirected rather than replaced
For example, external stakeholders might believe the current vision is automated reviews devoid of valuable human control and input, that is, a general autonomous artificial intelligence system. That view, however, was neither represented nor sanctioned at the meeting. Therefore, improving the terminology associated with systematic review automation to reflect the goal more accurately is likely valuable.