Background
Methods
Search strategy
Inclusion and exclusion criteria
Study selection
Data extraction & critical appraisal
Results
Search results
Study author, year | Description of experts | Real or simulated cases | Types of opinions aggregated | Study design | Relevance to collective intelligence |
---|---|---|---|---|---|
Gagliardi, 2007 [25] | 20 general surgeons, 2 pathologists, 1 medical oncologist, 1 radiation oncologist | Real | Diagnosis, Treatment | Qualitative observational study to explore the role of multidisciplinary cancer conferences in practice | Describe collective output generated in multidisciplinary cancer conferences |
Douzgou, 2016 [24] | Physicians with patients with malformation syndromes | Real | Diagnosis | Descriptive study of a consultation tool which generates collective insight | Assess a collective intelligence tool |
Sternberg, 2017 [36] | International colleagues with urologic expertise | Real | Treatment | Use Twitter as a potential collective intelligence tool | Describe social media as a collective intelligence tool |
Sims, 2014 [35] | Clinicians affiliated with academic departments: 28 from pediatrics, 27 from neurology, 10 from internal medicine, 4 from psychiatric, 11 from pediatric neurology, 5 others | Real | Diagnosis, Treatment | Descriptive study of a clinical consultation system which generates collective insight and qualitative evaluation of the tool | Describe a collective intelligence tool |
Nault, 2009 [33] | 5 spinal deformity surgeons | Real | Treatment | Feasibility study of a surgical decision-making tool as compared to a group of experienced surgeons | Compare collective intelligence generated by experts with a technology tool |
Alby, 2015 [22] | 1 oncologist and others from hematology, anesthesiology, surgery, and nephrology | Real | Diagnosis | Qualitative observational study of conversations about cancer cases between the chief oncologist and other physicians at a hospital | Characterize collective intelligence generated in usual practice |
Kattan, 2013 [28] | 24 urologists and oncologists | Real | Prognosis | Analysis of physician group accuracy as compared to a nomogram | Compare collective intelligence generated by experts with a technology tool |
Kunina-Habenicht, 2015 [29] | 283 medical students, 20 expert physicians | Real | Diagnosis | Descriptive study of the development of a computerized test to assess diagnostic accuracy; results were compared among medical students and expert physicians | Compare computer-generated collective intelligence of experienced physicians to medical students |
Lajoie, 2012 [30] | 14 third-year medical students | Simulated | Diagnosis, Treatment | Qualitative observational study of team discussions with or without a technology tool to aid collaboration | Optimize metacognitive activities in collective intelligence with a technology tool |
Kalf, 1996 [27] | 21 geriatricians, 21 geriatric-psychiatrists, 21 internists | Simulated | Diagnosis | Analysis of diagnoses generated by different specialties | Compare collective intelligence among different specialists |
Larson, 1996 [31] | 24 first-year interns, 24 residents, 24 medical students | Simulated | Diagnosis | Qualitative observational study of team diagnostic discussions when teams are exposed to different case information | Characterize collective intelligence generated when groups have different amounts of information about a case |
Christensen, 2000 [23] | 24 first year interns, 24 residents, 24 medical students | Simulated | Diagnosis | Qualitative observational study of team diagnostic discussions when given different amounts of shared and unshared information | Characterize collective intelligence generated when groups have different amounts of information about a case |
Larson, 1998 [32] | 48 interns and 24 third-year medical students | Simulated | Diagnosis | Qualitative observational study of team diagnostic discussions when teams are exposed to different case information and given instructions about sharing information | Characterize collective intelligence generated when groups have different amounts of information about a case |
Semigran, 2016 [34] | 234 physicians, including fellows and residents | Simulated | Diagnosis | Analysis of a collective intelligence tool as compared to the accuracy of symptom checker websites | Compare a collective intelligence tool to online symptom checkers |
Hautz, 2015 [26] | 88 medical students | Simulated | Diagnosis | Analysis of diagnostic accuracy when participants worked in pairs or individually | Compare collective intelligence of pairs to individual aptitude |
Participants and decisions in included studies
Application of collective intelligence
Initial decision task
Method of aggregation
Availability of collective intelligence output
Outcomes
Study author, year | Initial decision task process | Method of Aggregation /Synthesis | Study Outcomes | Results | Collective intelligence available to participants |
---|---|---|---|---|---|
Alby, 2015 [22] | Group | In-person | Extent of diagnostic uncertainty and perceived diagnostic complexity in discussions among experts | Study participants relied on three collaborative practices during informal conversations about cancer cases to organize the diagnostic decision-making process. | Yes |
Christensen, 2000 [23] | Group | In-person | The ability of diagnostic teams to integrate shared and unshared case information into a differential diagnosis | Teams of study participants mentioned more shared than unshared information when diagnosing patient cases and were less likely to diagnosis a case accurately when team members had limited information. Experience of participants did not significantly impact diagnostic accuracy. | Yes |
Gagliardi, 2007 [25] | Group | In-person | Extent to which multidisciplinary cancer conferences can address cancer-related information needs of clinicians | Multidisciplinary cancer conferences resolved cancer-related information needs, including treatment, diagnosis, pathology, and staging. | Yes |
Larson, 1996 [31] | Group | In-person | The use and order of shared and unshared information in team diagnostic discussion and its contribution to diagnostic decision-making | Information that was known to all group members was more likely to be discussed than information unique to individuals. Team leaders performed an important function in ensuring quality group discussion and contributing to medical decision-making. | Yes |
Larson, 1998 [32] | Group | In-person | Relation of shared and unshared information to diagnostic accuracy among teams | Shared case information was pooled more than unshared information among study participants. Diagnoses were more accurate when teams pooled more unshared information. | Yes |
Sims, 2014 [35] | Group | Information technology | The utilization and user opinion of the crowdsourcing application in the clinical setting | A total of 170 consults were generated by 20 study participants, predominantly seeking assistance in medication use and complex decision-making from the crowd. Providers had a favorable opinion of using the tool in practice. | Yes |
Sternberg, 2017 [36] | Group | Information technology | Extent to which Twitter can be used to share ideas about clinical case management | Twitter facilitated discussion among 11 participants from 5 countries that resulted in treatment suggestions. | Yes |
Lajoie, 2012 [30] | Group | Information technology, in-person | Extent to which technology can enhance metacognitive activities in diagnostic discussion | Technology enabled more metacognitive activities in group discussion. | Yes |
Douzgou, 2016 [24] | Individual | Information technology | The ability of a web-based service to generate clinical diagnosis for providers using an expert crowd and add value to practice. | The web-based service added value through the case report generated. | Yes |
Kalf, 1996 [27] | Individual | Manual | Concordance in facts and diagnoses among different specialties examining clinical cases | Study participants differed systematically in the diagnoses they reached. | No |
Kattan, 2013 [28] | Individual | Manual | Comparison of the accuracy of physician predictions with a nomogram | The nomogram was more accurate than physicians, regardless of medical specialty. There was variability among the decisions made by physicians. | No |
Kunina-Habenicht, 2015 [29] | Individual | Information technology | Comparison of accuracy of diagnoses and time to diagnose between experts and medical students | Experts had higher accuracy rates and lower decision times than students. Diagnostic accuracy improved with year of study among students. | No |
Nault, 2009 [33] | Individual | Information technology | Concordance between surgeons and a fuzzy logic model tool | Study participants made diagnostic decisions that were generally in agreement with decisions made by fuzzy logic model tool. There was large variability among the decisions made by study participants. | No |
Semigran, 2016 [34] | Individual | Information technology | Comparison of the accuracy of differential diagnoses of physicians with online symptom checkers | Study participants listed the correct diagnosis first and within the top three diagnoses more often compared with symptom checkers. Study participants were more likely to list the correct diagnosis first for high-acuity vignettes and uncommon vignettes; symptom checkers were more likely to list the correct diagnosis first in low-acuity vignettes and common vignettes. | No |
Hautz, 2015 [26] | Individual and Group | Information technology | Comparison of diagnostic performance of individuals with those working in pairs. | Pairs of study participants were more accurate and confident than individuals, but confidence was not dependent upon decision accuracy. | Yes/No |