Background
The primary health care sector delivers the majority of health care in western countries through small community-based organizations. In Australia, the vast majority of primary care is delivered through general practice. These small, busy organizations, composed of shifting configurations of staff, vary according to their local contexts, their size, their modes of funding, and the types of services they provide. To understand the processes that underpin the delivery of quality primary care, we need to have a nuanced idea of how individuals work within these organizations. But the methodological toolkit to collect and analyse data on staff and organizational function and activities is rather limited. Much research into primary care has relied on quantitative, or single-method qualitative research. Although the importance of qualitative research methods in primary care has been emphasized since the mid-1990s, spearheaded by the
British Medical Journal [
1]-[
3], studies typically focus on interviews or focus groups as the primary data source. Single-method studies do not capture the richness and variety of organizational functioning, the ways that staff members interact and use their time, and the impact of the spatial environment of the organization on their work.
In this paper we present an overview of a mixed-methods approach to researching small-scale primary care organizations that is rapid and rigorous (Q-RARA - Qualitative Rapid Appraisal, Rigorous Approach). We developed this method to study nurses in general practice in Australia, at a time when nurses were moving into general practice in large numbers [
4]. We wished to understand the impacts of individual features of the practice, of the nurses themselves, of the other staff and of professional culture on the nurses’ roles and work. Ethnography offered one route to study emerging practices and actions within a small organizational culture, but we were concerned that the length of time to conduct organizational ethnography would mean that we would only be able to study a few sites. Q-RARA is a methodological design that is fast, flexible, and capable of producing a wide range of data from different perspectives with minimal disruption of the organization. Since conducting the Australian General Practice Nurses Study we have modified and streamlined the research design. We believe that Q-RARA is likely to be useful for research into other small scale organizations, and for bounded areas of large health organizations.
Several reasons have been put forward to explain the reliance on single-method studies, mainly interviews, focus groups and surveys in primary care research. As a rule, general practices are small businesses with few staff members all working under strict time constraints and pressure [
5]. Because of the competing demands for time in general practice, staff may view research as peripheral to the purpose of the organization [
6]. Under these circumstances, researchers may choose methods that can be conducted out of work hours or away from the organization.
An additional challenge for researchers is the diversity of primary care organizations in terms of structure, funding, and function within and between countries [
7]. Such plurality may make inference problematic if research is based on a limited number of field-sites.
Q-RARA draws on two main influences: rapid appraisal and qualitative mixed methods research design (or QUAL-qual methods). The approach takes into account the need to minimize the impact of conducting research in small organizations, while maximizing the capacity to produce rich, detailed contextual findings. In this article we present: (1) an overview of the background to the approach - rapid appraisal and qualitative mixed methods design (2) the approach itself, and (3) a critical discussion of the broader literature regarding mixed methods design and issues of rigor. We identify the strengths and weaknesses of the approach, as well as its potential contribution to researching primary care organizations by extending the use of a qualitative mixed methods design.
Antecedents of Q-RARA
Rapid appraisal
Rapid appraisal was pioneered in the 1960s in the field of rural studies [
8], but not used systematically in health-related fields until the 1980s when rapid social science data were collected by trained [
9] and, more controversially, untrained [
10] researchers, on illness profiles, the understanding of disease terms, and health practices. A parallel endeavour had existed for some time in public health, where communicable disease epidemiologists used quantitative rapid appraisal methods to investigate disease outbreaks [
11].
Rapid appraisal has since been used in many other settings, including humanitarian crises, and its variations are now largely referred to under the banner of Rapid Evaluation and Assessment Methods (REAM) [
12]. REAM is an umbrella term that offers little detail about the processes involved in implementing a rapid appraisal approach, particularly using qualitative mixed methods. The epistemological underpinnings of REAM range from realist, objectivist epistemology (as in studies used to provide quick assessments of program performance [
13]) to constructionism (in studies used to assess the impacts of policy [
14] or roles [
15]). The epistemological diversity of these studies reflects an under-theorization of methodology in this research area.
Qualitative mixed methods
The other antecedent for our method is mixed methods research. Mixed methods research is research that is informed by and situated along a spectrum of quantitative through to qualitative paradigms, and generally refers to the combination of quantitative and qualitative methods carried out “for the broad purposes of breadth and depth of understanding and corroboration” [
16]. It is not new in general practice, having first been advocated over thirty years ago [
17],[
18]. One of the caveats of this method in primary care, however, is that it can prove time-consuming and overwhelming for small institutions [
19].
A particular strength of mixed methods research is the formal integration of individual methods at some point in the research process [
20]. Although integration produces a more detailed, richer understanding of the phenomena of interest, in health care research integration is frequently neglected [
21]. Furthermore, one of the pitfalls in mixed methods research is the conflation of integration with triangulation [
22],[
23]. In his influential book
The Research Act, Denzin [
24] popularized the concept of triangulation as “the combination of methodologies in the study of the same phenomenon” (p. 291). The many variants, processes and critical debate surrounding triangulation [
8],[
25] are beyond the purpose and scope of this article. We acknowledge the necessity to engage with the epistemological assumptions of the relative methods employed in any given mixed methods study so that issues of commensurability are addressed.
Morse [
26] challenged qualitative researchers to consider “if, when and how” the use of two methods from the same paradigm, referred to as QUAL-
qual methods, can be considered mixed methods. QUAL-
qual denotes a core project and a supplementary project whereby the latter cannot be a stand-alone project. According to Morse [
27] the data types, levels of analysis, or participant perspectives of the core and supplementary components need “to be handled differently and to be kept apart” (p 491).
Q-RARA attempts to marry the rapidity and limited intrusiveness of rapid appraisal methods with the rigour and integration of mixed methods research. In this paper, we subject the quality and rigor of Q-RARA to scrutiny through an assessment of its performance against Lincoln and Guba’s well-established frameworks of trustworthiness and authenticity [
28].
Results
We assessed the quality and rigor of the data and findings generated by Q-RARA, and interrogated its capacity to adequately reflect the stakeholders’ views, against the adequacy criteria suggested by McNall and Foster-Fishman [
12] in their adaptation of the framework by Guba and Lincoln [
28]. Guba and Lincoln proposed that
trustworthy data are credible, transferable, defendable and confirmable. Data should also be assessed for
authenticity, a domain that includes fairness, and educative, ontological, tactical and catalytic authenticity. These assessments are presented in Tables
3 and
4.
Table 3
Performance of rapid QUAL-qual Method against Guba and Lincoln’s trustworthiness criteria
Credibility | Extent to which findings accurately portray respondents’ constructions. Involves the following: | |
|
Prolonged engagement in targeted site to build rapport and trust between evaluators and setting members and provide evaluators with a deeper understanding of the relevant culture. |
Prolonged engagement: One day site visits precluded prolonged engagement. |
|
Persistent observation of site to provide sufficient understanding. |
Persistent observation: Although it is recognized that persistent observation was not carried out for this study, the researchers attempted to respond to this criterion through repeated observation periods. |
|
Peer debriefing: Extensive discussions of data and preliminary findings with one or more peers to refine thinking. |
Peer debriefing: RAs provided field notes and reflections for each site visit and were able to debrief with Research Manager (SH) as often as necessary. Furthermore, the questions in the qualitative interview schedule were clarified and streamlined in response to feedback from a research assistant. |
|
Negative case analysis: The constant reworking of hypotheses in light of disconfirming evidence. |
Negative case analysis: In most practices, leadership was vested in the general practitioner, and nurses were relative newcomers to the practice. Specific analytical attention looking for difference was paid to one practice where the nurse was the senior clinician who had worked longest in the practice as a negative case. |
|
Progressive subjectivity: Researchers identify and articulate any biases they hold, examine how their understandings shift during the project, and attend to how these biases might affect interpretations. |
Progressive subjectivity: Assumptions were regularly challenged during fortnightly analysis meetings (see Case Study 1). |
|
Member checks involve sharing and checking findings and interpretations with the people from whom the data were collected. |
Member checks: Summaries of our research were returned to each practice for verification. We also presented our data at general practice and nursing conferences, and posted evolving data on a website developed for the project, inviting feedback from readers on the blog who were practice nurses on the summary de-identified findings and our interpretations. The Reference Group, formed at the commencement of the study, met over the course of the project and gave their feedback on topics specifically raised with them. |
Transferability | Researchers describe features of targeted context in detail and suggest additional contexts to which findings might be generalized. | Extensive background and case information included in final report. |
Dependability | Concerned with stability over time in researchers and methods. Assessed by means of a dependability audit, which involves reviewing project records to determine the extent to which project procedures and changes are documented. | This team included clinicians, academics and individuals engaged in organization policy and advocacy, who assisted in recruitment and in ensuring that the understanding of the project by the field sites was consistent. Regular meetings were held with all team members to monitor adherence to project procedures and to document changes in protocols. |
The three chief investigators met regularly in person and via telephone, and a summary of the decisions made were routinely produced. |
Confirmability | Extent to which findings are grounded in the data. Assessed by means of confirmability audits, which involve reviewing research records to determine if findings can be traced to data and data to original sources. | Kept all case-summary, substantive theme and pattern analysis documents. |
Data and themes in all non-public documents were linked to subject IDs. |
At regular team meetings to discuss the ongoing analysis, members were encouraged to look for the “black swans”, that is, evidence that might contradict the finding under discussion. |
Table 4
Performance of rapid QUAL-qual Method against Guba and Lincoln’s Authenticity Criteria
Fairness | Extent to which different stakeholder perspectives are elicited and taken into account. Involves identifying all stakeholders, soliciting their perspectives, and engaging in open negotiations with them around recommendations and future actions. | We interviewed people holding a range of roles in each practice: practice nurses, a general practitioner, the practice manager and a receptionist. |
The Reference Group was particularly valuable in the final phases of the write-up, through the advice they gave on structuring the recommendations. |
Ontological authenticity | Extent to which stakeholders’ perceptions of the world have been improved or expanded. | The research gave “voice” to the participants by publishing and presenting information that they knew, but was not well understood or recognized more broadly. |
Educative authenticity | Extent to which individuals have developed a better understanding of other stakeholders’ experiences and perspectives. | This work was distributed in many forms (peer review journals, conference, trade press articles etc.) to both nurses and doctors, and gave support to the notion that nurses play multiple functions, some under-recognized, by general practitioners and nurses. |
Catalytic authenticity | Extent to which the research elicits action and change. | There is evidence at the macro level, that previously unnoticed element of practice nursing was the extent to which nurses were “educators” and yet “doctors tended not to recognize nurses’ educator [role] ... within the practice.” However, General Practice Education and Training, the national body responsible for preparing doctors for general practice, has now funded trials of practice nurses training general practice registrars. |
Tactical authenticity | Extent to which stakeholders feel empowered by the evaluation and by the ability to influence the actions taken. | The findings (e.g. six roles of nurses [ 4]) were taken up at multiple levels by nurses from the Chief Nurse in Department of Health and Ageing to individual practice nurses. Spin-offs have included further development of some of the under-recognized roles such as the nurse as educator role through specific project funding [ 40]. |
Trustworthiness
One of the clear challenges in assessing our approach against Guba and Lincoln’s criteria for trustworthiness was that they maintain that credible data are developed through prolonged engagement. Our method is predicated on short-term, intense engagement and cannot be compared with ethnography, which involves long-term (prolonged) engagement.
Despite their short period in the practice, research assistants were allowed access to the backstage of general practice. They were invited to observe practice meetings, accompanied nurses on their rounds when they left the practice, and followed her as she moved across all the spaces in the practice, including into doctor’s rooms. We considered that within a short-term engagement we had achieved a sense of engagement that was sufficient to meet the goals of the larger project in terms of data collection, but that this time frame may not be suitable for some other types of projects.
Against other criteria for the domain of trustworthiness the method performed well as outlined in Table
3.
Authenticity
The method was able to meet all the criteria contained within the domain of authenticity, as outlined in Table
4. The ability of this method to generate authentic data reflects, in part, its roots in rapid appraisal, a method that is pragmatic and purposive. On the other hand, the fact that the method also performed well against the trustworthiness criteria in general reflects the formalization of the mixed methods approach to frame rapid appraisal. An instructive comparison with our approach is research into the impact of financial incentives on clinical autonomy and internal motivation on family physicians in the UK [
41]. This ethnographic study developed detailed case studies of five British family practices at a time of structural policy change, including the introduction of the National Service Frameworks and the new General Medical Services contract. The study involved in-depth, long-term contact with each of the practices. The data from this study also scored highly on both trustworthiness and authenticity criteria, but the research itself was time-consuming and the generalizability of the data in some instances was limited.
Discussion
The approach to using Q-RARA, developed through the AGPNS, demonstrated three main features: (1) a high degree of acceptability in the field, (2) the operationalization of a modified QUAL-
qual method, and (3) quality and rigor. First, the approach was well received by the research participants, particularly the practice nurses who believed that the interest of the researchers validated their work. In common with traditional rapid appraisal methods, Q-RARA in general practices was carried out with minimal disruption to the research site while gathering a large quantity and range of data. Feedback from the nurses collected at the time and after leaving the site, indicated that Q-RARA did not disrupt their work unduly beyond the issues described above. It also has relevance for research in other small organizations, or can be employed, as Murray [
42] has demonstrated, as a mechanism for public involvement in the collection of data about social and health needs in primary care.
Second, in terms of the modification of the QUAL-
qual method, the original definition of mixed method design proposed by Morse and Neihaus [
43], p. 9 stated that it “consists of a complete method (i.e. the core component), plus one (or more) incomplete method(s) (i.e. the supplementary component[s]) that cannot be published alone, within a single study”. Morse [
27] went on to specify that the data types, level of analysis or participant perspectives must be sufficiently different that they need to be handled separately prior to integration. The stipulation that one of the qualitative methods must be seen as supplemental might imply an inadequacy in the primary method, and consequently, that the need for supplemental strategies arises from “a lack of clarity on the conceptual framework of the study” [
44], p. 281. In contrast, the qualitative methods used in the AGPNS were driven by the research questions and conceived prior to the site visits. Our study is not alone in this regard with many other studies in the broader health sector employing more than one complete, qualitative method [
45]-[
47].
In our approach we modified the QUAL-qual method by adding more than one standard qualitative method central to the study; both the in-depth interviews and the structured observation were core components. We suggest that the successful operationalization of this extended version of the QUAL-qual method may require alternate wording for the definition of mixed qualitative method design that addresses the concerns outlined above: QUAL-qual method design comprises several different qualitative methods, the choice of which is driven by the researchers’ epistemological position and theoretical perspective, and the research objectives. One or more of the methods may be able to generate sufficiently coherent and convincing findings to be published alone, but the authenticity and trustworthiness of the findings are increased by the planned use of additional research strategies that are insufficiently robust to generate defensible findings alone. The data types, level of analysis, or participant perspectives must be sufficiently different to warrant separate handling prior to integration. However, the findings generated by the different methods must be combined at some point in the research process.
The modifications to using the QUAL-
qual method that were made in our approach are a development from existing rapid appraisal studies in primary care in that they directly engage with epistemology and integration, unlike other studies-such as Murray et al. [
42],[
48] where the main research objective appears to be to advocate for the relevance of rapid appraisal methods in primary care, specifically interviews and focus groups or quantitative rapid appraisal methods, rather than a deeper engagement about method. Further to this, in studies where there is an exclusive qualitative mixed methods design, such as Manthorpe et al. [
49] it is typical for details about individual methods to be described but a discussion of integration to be absent.
Third, the use of exploratory mixed methods provided a range of data that would have been missing had we limited ourselves to the original definition proposed by Morse [
26],[
27]. These diverse data, in conjunction with the multidisciplinary team and an iterative approach to data analysis allowed us to engage in a dialogue with the data and produce authentic and trustworthy findings. The location of the nurse’s station as a way of reinforcing her centrality in the general practice provides an example of an evolving authentic understanding that would not have been otherwise achieved. The dialogic engagement with the data also enabled us to articulate previously unrecognized elements of the social world, that of the practice nurses’ role as agent of connectivity. The QUAL-
qual method also revealed the nuances of social life, particularly in the way that they relate to differences in power and authority. For instance, it became apparent that the general practitioners were unaware of the range of tasks undertaken by practice nurses and their corresponding skill set [
4].
The conduct of the AGPNS and development of our approach, like any field-based method, can be derailed by major events. A site visit had to be curtailed because the general practice principal collapsed and had to be taken to hospital. Nevertheless, the field researcher was able to observe and comment on the way the organization functioned in a crisis. The most difficult component of data collection was securing an interview with the general practitioner due to time pressures on their work. Consequently, some interviews with GPs were conducted by telephone after the practice visit. In small general practices, GPs have the most difficulty disposing of their time freely. Our results suggest that if this method had focused on those with least flexible time schedules in the organization, we would have had to allow more time or additional participant-responsive methods, such as video, or specific attention to informal meeting sites, such as tea-rooms.
The intensive nature of the day-long visits to practices could be draining for the field researchers, who were required to collect a great deal of data in a concentrated period of time, without disrupting the practice or losing rapport with practice staff. The two field researchers travelled very long distances to visit the sites, and were initially over-scheduled. Sufficient time between site visits is needed to collate field-notes, to recover from the trip and to prepare for the next one. Consequently, the 25 site visits were undertaken over a period of four months, equivalent to one site visit by each researcher every 10 days.
We believe that the one-day visit design is defensible even though it means that intra-clinic variation in workload across the week was not captured. All staff were asked if the observation day was typical for them, and how they structured their working week. Interviews were able to flesh out and extend the insights gained through observation; thus although there would have been differences in the observational data on different days, these did not change the broad categories of role expressions and determinants that were identified in each practice.
We noted that the structured observation tool could be improved by using more advanced technology. Nurses could be trained to use personal digital assistants or smart phone applications to record their own time use patterns. This may result in more valid data on the time use of nurses [
50] but at the loss of reflections-in-action, which the field researchers then used to refine the focus of their interview schedule. Alternatively, the field researchers could continue to record the data using tablet computers.
Focus groups were not included in our version of Q-RARA because of the very small size of the health care teams, and concerns that setting up a focus group would take away too many people from the workplace. We have subsequently used them in another study using Q-RARA where we were interested in the interaction between patients and volunteer-leaders in a falls prevention program [
51].
The structured observations were also limited because the researcher was not permitted to observe patient-nurse interactions. This meant that our analysis under-represented clinical care aspects of the nurse’s role. However, the fact that the nurse-patient interactions were not observed made the presence of the researcher more acceptable to the family practice itself. A method of this nature, where the researcher is only present for a short period, does not allow the researcher to develop trusting relationships with patients as, for example, an ethnographer may be able to do. To observe patient-nurse interactions would have imposed the burden of explaining the study to successive patients upon the reception and nursing staff, and slowed down the work of the practice.
Competing interests
The authors declare that they have no competing interests.