A multidisciplinary systematic review of the use of diagrams as a means of collecting data from research subjects: application, benefits and recommendations

Of the 80 articles included in our review, 53 were published studies[1, 4‐15, 19‐58], 19 were dissertations[59‐77], 2 were books[78, 79] and 6 represented grey literature[3, 80‐84], including unpublished working papers submitted by key experts and reports available on the internet. These articles were published between 1986 and 2010, with the majority published after 2000 and a substantial increase after 2006. This suggests that interest in these techniques has been increasing in recent years.

The most common discipline, determined by the lead author's affiliation and/or publication title, was from the education field. Other disciplines included healthcare, engineering, environmental science, geography, industrial design, psychology, and social science. The majority of articles clearly specified the study sample size, which averaged 36 research subjects, with a range of 2 to 243. Diagramming methods were used with a wide variety of research subjects, including students (elementary to graduate school), farmers, nurses, physicians, engineers, administrators and graphic designers.

The majority of articles specified at least one explicit reason why a form of diagramming was selected for data collection. These reasons fall into two broad categories: requirements or challenges of the research topic and the unique dataset that results from using diagrams.

The specific research topic examined was the most common reason for researchers choosing a diagramming technique for data collection. For some research topics, past studies have validated diagramming data collection techniques as a useful way to collect data. For example, research has established the usefulness of diagrams in collecting data about research subjects' knowledge or cognitive structures[19‐22, 59, 80]. Diagrams in data collection have also been validated as a means of measuring changes over time[6, 10, 20, 22‐24, 60, 85] and differences between participant groups[20, 25, 61, 81]. Diagramming methods were also sought out when research topics were not conducive to the more common qualitative data collection methods, such as interviews alone. These reasons include a research topic that deals with a population with linguistic, cultural, social or intuitional barriers the researcher wants to overcome[12, 14, 15, 26‐29, 79] or with highly complex subject matter[12, 14, 25, 30‐32, 85]. Examples of highly complex subject matters include the abstract nature of the research topic of 'pedagogical constructs'[25] and the multifaceted and diverse nature of 'clinical accountability relationships'[14].

Secondly, researchers sought out diagramming data collection methods because of the benefits previous studies found regarding the quality and uniqueness of the collected dataset. When research subjects drew diagrams without prompts, previous studies concluded that it minimized the influence of the researcher on the participant and their responses[1, 33‐35, 61, 62, 81, 82]. Studies have also found that diagramming is a reflective tool for the research subjects[28, 29, 36, 63]. Since diagrams can represent both concrete and theoretical notions[37], diagramming offers a more holistic coverage of the topic[29, 38, 61], with more uncensored and unique data gathered[1, 24, 28, 35, 39, 40, 58] than more traditional qualitative data collection methods.

A range of applications were identified, which varied widely based on the degree of instruction, degree of freedom in diagram creation, the number of diagrams created or edited and the use of diagram in conjunction with other data collection methods.

Half of the studies did not report the details of the instructions provided to the research subjects, except for describing the basic request to create a diagram. One study explicitly observed that specific instructions are needed to ensure the research participants create a diagram and not another form of written material[32], such as a drawing or table. Simple or short instructions were often given to research subjects when the diagram sought by the researchers did not have to conform to a rigid structure, such as life-cycle[13] and professional practice diagrams[32].

When the researcher sought highly structured diagrams, the degree of instruction provided to the research subjects ranged from the preferred method of giving specific and detailed instructions on what elements should be included (e.g. hierarchies, arrows) to showing an example of the type of diagram the researcher would like the participant to create. For example, in comparison to other diagramming techniques, concept maps have a fairly rigid definition and a very specific set of elements that the end diagram should contain. Such diagrams may require more detailed instructions. One study had associate nursing degree students draw their own concept maps after a 20-minute introductory tutorial, presentation of a sample diagram, discussion and question period, and instructions listing all elements to be included (e.g. arrangement of items, hierarchal order, linking concepts with arrow, labeling propositions, identifying cross-links/relationships)[9]. In some instances, research subjects were given the opportunity to practice the diagramming method and receive corrective feedback prior to data collection[19, 23, 41, 64].

It was most common for research subjects to create an original diagram on their own, in groups or a combination of both. Alternatively, some studies had the participant edit either a designated diagram provided by the researcher[3, 14, 15, 42] or a researcher-created diagram generated by the researcher during the interview with participant input[4, 5, 7, 43]. A few studies also chose the middle ground between original and prepared diagrams. For example, some provided a central concept or word to create the diagram around[14, 44], or included some words or shapes to fill in on a prepared diagram[45] and others gave research subjects a list of words to use in the creation of their diagram[65, 80].

Half of the studies used diagrams at multiple times within the data collection process as a means of comparison. A subset used a pre-/post- (or time series) approach to data collection, allowing researchers to track changes before, after, and sometimes during an intervention. This was found primarily within the discipline of education. For example, Rios asked a sample of teachers to create concept maps at multiple time intervals in order to identify their conceptual models and examine the impact of student interactions on the teachers' subject matter structure or vice versa[66].

Some researchers explicitly expressed the idea that diagrams alone would not capture complete perspectives from the research subjects[57, 77], suggesting that diagrams should be used in combination with other data collection methods. The majority of the studies did use diagramming data collection techniques in addition to other methods. In some cases the additional data collection methods were used explicitly in conjunction with the diagramming techniques, such as creating the diagram within interviews or discussion of the diagrams in later focus groups or interviews[8, 23, 39, 59, 60, 67].

The majority of the articles reported details about the analysis of diagram data. The use of only quantitative or qualitative analysis, or a mixture of both types of analysis was fairly equally distributed among the articles. Within each of these three categories of analysis, there were a variety of different techniques used that are briefly outlined below.

The majority of studies comparing diagrams across time or across research subjects chose either quantitative techniques only or a mix of qualitative and quantitative techniques for their analysis. Quantitative analysis techniques included counting (e.g. number of concepts identified, number of links between concepts, number of examples given, levels in hierarchy) and scoring. The two most common scoring methods were structural scoring and relational scoring[22]. Structural scoring refers to when weights are assigned to hierarchical structures, links between concepts and other elements. For example, a link between two concepts may be 10 points, an example 1 point and invalid examples and cross links 0 points[40]. Relational scoring reflects the quality or importance of each concept or link as determined by the researcher, by comparison to a similar diagram created by an expert or by other research subjects[36, 38, 60, 62, 68]. Some studies using quantitative analysis showed diagrams to illustrate how the final counting and scoring of a diagram was completed in the presentation of final study results[35, 40, 41, 46‐49, 54, 62]. For example, Kesby had focus groups in Zimbabwe create scored diagrams with local materials of rocks, string and bottle caps[54]. These diagrams were photographed and also reproduced on computer for legibility. Including an original diagram in the presentation of results also helps to orient the reader to the type of diagram that was used.

Studies which had diagrams completed or edited in the presence of the researcher, included additional data collection methods, and/or studies using less structured diagrams to collect data were likely to use either qualitative techniques only or a mix of qualitative and quantitative techniques for their analysis. For example, Haidet et al. gave medical school research subjects less structure and encouraged creativity in their diagramming exercise within interviews[13]. The diagram was used as a prompt to stimulate discussion, which was then analyzed through the interview transcripts. The diagram itself was then displayed in the final results to visually summarize the verbal exchange. Qualitative analysis included thematic and content analysis of the diagrams and additional data sources, such as transcripts, to identify prominent topics, themes and patterns in the diagrams[13, 25, 26, 34, 63, 66, 69‐71].

In some studies that used mixed-methods, the collected diagrams were the primary source of data and guided the analysis of additional data sources, for example, by providing the core themes for transcript analysis. In other cases the additional data sources played the dominant role in analysis and the diagrams were used almost as verification or visual representation to illustrate conclusions[72].

Some of the benefits to using diagrams for data collection have already been discussed in the section on why researchers chose diagramming data collection approaches. In addition to these, diagramming approaches that were seen as complementary to other data collection approaches were commonly used in interviews and focus groups[60, 73]. They were found to help focus discussions on particular themes[32] and enabled research subjects to more easily reflect on a topic or their beliefs by helping them to express thoughts in a more structured and organized manner[50, 51]. The use of diagrams was also seen to increase recall[52] and self-reflectiveness[53, 54]. In 1992, Powell found that interviews which made use of diagramming approaches were more introspective and tended to be more theoretical and philosophical than those that did not use diagramming methods[25].

Over half of the articles discussed at least one challenge of using a diagramming method for data collection. Interestingly, these challenges were often contradicted by other articles. All studies completed their data collection with diagrams, with some reporting that the diagramming allowed research subjects to overcome challenges of verbal communication[12, 14, 15, 26‐29, 75, 79]. However, many studies found that at least some of the research subjects expressed difficulty or discomfort with the diagramming task[1, 9, 10, 14, 28, 35, 55, 56, 58, 62, 74, 76, 83]. Some identified the ease and speed of data collection as benefits of using diagramming approaches[24, 31, 54], while others saw it as being time-intensive, particularly for analysis[6, 11, 26, 62]. Related to the visual organization and structure of knowledge that the diagrams presented, an advantage to using diagramming approaches for data collection is their ability to obtain unique and unsolicited data[7, 14, 15, 26, 29, 33, 42, 43, 58]. In contrast, there were also concerns regarding the data it did not collect, such as non-verbal communication, that require the discretion and experience of researchers to identify and interpret[15, 24, 84]. These contradictions illustrate that the benefits and challenges to using diagramming approaches for data collection depend on the application and type of diagram used in each research study.

The most important considerations for choosing the diagramming approach is the type of data needed to answer the research question (e.g., examining change over time, exploring people's experiences or views) and the type of analysis preferred by the researcher. For example, highly structured diagrams allowed for valid quantitative analysis, such as counting and ranking, which could be compared across research subjects. It should be noted that there is some controversy whether items should even be counted and that both an over- or under-reliance may be dangerous to the final research conclusions[28]. In comparison, other approaches that used a less structured diagrams relied heavily on qualitative analysis.

The appropriateness of different approaches to instructions to guide diagram creation is an important consideration in ensuring the validity of data. It is clear from our review that the initial instructions given to research subjects varied in structure but had a great impact on the resulting diagrams and their potential for different analysis techniques. If researchers require highly structured diagrams it may be useful to give research subjects more detailed instructions and the opportunity for practice and feedback[30, 64].

The last recommendation is that studies using diagramming data collection approaches should include visual presentations of the findings in their results sections. The use of diagrams often results in the collection of unique and unsolicited data through a visual component, which can then be displayed along with the final analysis. Just as diagrams can provide data not easily obtained through verbal data collection techniques, visual presentation of the collected diagrams may provide insights not as easily grasped through verbal communication of study results. While providing a scanned image of the original diagram can be difficult at times, it is also possible to present a photograph of the original diagram or a computer-generated recreation of the diagram. This is especially important given the variation in terminology between disciplines, as it relays to the reader the type and structure of diagrams created or used.

In addition to the recommendations for researchers considering the use of diagramming data collection approaches, this multidisciplinary review also identifies areas where future research is needed. This review required a substantial amount of preliminary work to understand the terminology used to describe diagrams and this data collection approach across different disciplines and fields. The intent was to devise a sensitive search, so as to cast a wide net in order to capture articles in a range of disciplines where the terminology is not standardized. Cole et al. illustrate this issue by identifying over a dozen different terms that are used to describe concept maps[56]. Thus far, development of terminology has focused on the end result, i.e. what type of diagram is created based on the elements it contains, rather than focusing on the actual data collection approach itself. This has created different data collection approaches that use diagrams separately, isolating research done across disciplines and even within disciplines. Therefore, it is our recommendation that efforts are directed towards standardizing the terminology for this data collection method. This would allow researchers to maintain the work they have done regarding specific types of diagrams, whilst providing an umbrella term to help with the sharing of best practices. Future research should also be directed at identification of the underpinning theory of the method as the review demonstrated a gap in this area[15, 32, 35]. Such a theory may help to further inform researchers regarding the appropriate use and applications for diagramming data collection approaches.

A limitation of our review is that the database search strategy did not capture general articles on visual data collection methods, which may include specific information on diagramming data collection approaches. However the non-indexed searches and articles identified by experts did pick up some of these articles. While efforts were made to contact authors to retrieve articles from our search, twenty-seven full-text articles were irretrievable. Ten of these were dissertations and five were books. This may have contributed to an incomplete representation of what the literature has to offer about diagramming data collection approaches.