A new methodology for assessing health policy and systems research and analysis capacity in African universities

The participating partners in the CHEPSAA capacity assessment are identified in Table 1. The table summarises key features of these partners, at the time of the assessment. The institutional nature of the partners has been covered in detail elsewhere [7], therefore, in Table 1 below we emphasise their selected key features.

The assessment process included elements of self-assessment conducted by African partners, and external comparative synthesis conducted by a UK partner. Self-assessment was particularly attractive to the consortium because it was able to take advantage of existing research expertise within CHEPSAA. There was also a sense that the assessment process had potential to be a positive learning experience for all partners. The methodology development in the Consortium was led by one partner (University of Leeds), who initiated key documents, coordinated discussions at consortium meetings and at a distance through teleconferences and email exchange, and conducted comparative synthesis of partner findings.

Five principles underpinned the design of the methodology and conduct of the assessment.

The first step was to develop a shared understanding of the concept of capacity and capacity strengthening within the consortium. A concept note was circulated as a discussion document, which outlined key concepts, proposed the main principles, and suggested potential themes to assess HPSR + A capacity within African universities. The concept note was reviewed, discussed through at a teleconference involving all partners, and subsequently revised. During this process, a lack of clarity in understanding key concepts of the field became apparent. The Consortium Coordinator made a series of presentations and led discussions at the first annual consortium meeting to help develop a common sense of what constitutes HPSR + A.The second step was to map the contextual environment of HPSR + A for each CHEPSAA partner. A common guideline was developed, outlining the key areas for assessment, potential methods for data collection and analysis, and proposing a common structure of the report. The context mapping was anchored in a desk review but allowing for informal discussions with policy partners such as Ministries of Health, institutional partners, and reflection on personal experience by the HPSR + A academics. Partners developed individual reports (the three South African partners returned one combined report) against a minimum data requirement, which was subsequently synthesised using a framework approach based on assessment themes (Figure 2) to inform the next step in the assessment. The third step was a thematic organisational self-assessment incorporating individual assessment. The African partners returned individual reports that included relevant material from the context mapping. The fourth step was a comparative synthesis and subsequent discussion of assets and needs across the consortium. Again, a framework approach was used to analyse partner results and undertake a consortium-wide comparative synthesis. An assessment of current HPSR + A teaching materials used by partners was conducted separately.The assessment focused on specific thematic areas (Figure 2). The thematic areas in the context mapping fed into development of the organisational and individual assessment themes. Greater detail on these themes is set out in the list below.

Organisational and individual assessment: minimum information requirement

As part of the commitment to a semi-standardised approach, a menu of methods was proposed to allow CHEPSAA partners to select those most appropriate for their setting with different respondent groups (Table 2). These were: i) document review particularly internal documents; ii) in-depth interviews with key respondents; iii) focus group discussions; iv) a staff survey template; and v) participatory stakeholder workshop.

Partners used varying combinations of these methods (Table 3), although the participatory stakeholder workshop was not used.

There are limitations in the assessment methodology that has been presented here. The value of including elements of self-assessment has been discussed but there is also potential that important issues may have been missed because of the predominantly ‘insider’ perspective in collecting and analysing the results within each organisation. While the assessment was not solely self-assessment, the participating external partner was a partner of the research consortium and therefore not strictly an outsider. Limitations of self-reporting can be poor recall. Social desirability bias is also possible, meaning under reporting of what the reporter perceives to be negative and over reporting of what is perceived by a reporter to be positive. There is also scope for over reporting of deficits to make the case for future additional resourcing of HPSR + A units. The CHEPSAA methodology sought to take account of these limitations. There was active engagement of all partners in developing and applying methodology in capacity assessments. In addition, the semi-standardised methodology meant that common thematic areas (Figure 2) were accompanied by a list of required information agreed by all Consortium partners (see Organisational and individual assessment: minimum information requirement above). Therefore, a combined weight of experience from different disciplines, institutions, and individuals was brought to the assessment. The results of the capacity assessment have been published elsewhere [7] and are not reprised here.

The approach taken was felt by partners to be overwhelmingly positive (discussed later) suggesting that it can be considered appropriate and relevant to the contexts and concerns of HPSR + A units. From sustained reflection on the development and conduct of the CHEPSAA assessment methodology described above, four lessons can be communicated. First, the CHEPSAA methodology is new and unique to the field of HPSR + A. Second, in HPSR + A the unit of assessment can be a challenge, because HPSR + A groups within universities tend to overlap between academic departments and are embedded in different external networks. Third, a capacity assessment is not only a planning tool but is also capacity strengthening of itself. Last, based on the CHEPSAA experience, a number of methodological issues for consideration in future capacity assessments can be proposed. Each lesson is discussed in turn below.

Although universities play a vital role in HPSR + A, there are currently no capacity assessment methodologies for HPSR + A groups within universities. Assessment of administrative, decision making, and political capacity of health ministries [19, 20] and research and analytical capacities of independent research institutes [21], has been undertaken. Capacity assessments that have been conducted within universities have covered specific training courses, such as public health [22], post-graduate epidemiological research [23], and health services and systems training [24], or a transdisciplinary aspect of the academic remit such as knowledge translation [25] or running doctoral programmes [26]. None of these have assessed the field of HPSR + A solely within a university and none has done so comprehensively across three levels of capacity. The CHEPSAA assessment methodology is therefore new, unique, and implemented across seven institutions. It contains insights that could be of potential use to HPSR + A units in other universities around the world who can adopt or evolve the methodology for their local contexts. It can also be useful to HPSR + A units in other institutions such as Ministries of Health or civil society organisations. However, their specific scope of work will need to be reflected in the assessment themes (c.f. Figure 2). For instance, HPSR + A Quality Assurance may not be a priority for a Ministry of Health but other themes such as Leadership and Governance, Communications and Networking, and Resources would continue to be relevant.

Methodologically, it was often difficult to pin down ‘who’ was being assessed. The unit of assessment was more fluid than we were led to expect from accounts of other capacity assessments. In those accounts, the individuals assessed are seen as nested discretely within autonomous organisations that have complete control over its available resources. This was not the case for HPSR + A groups within universities.

The organisation in the CHEPSAA assessment was initially taken to be the contracted partner, which is an administrative unit within a university. Each unit has a different structural relationship to the university in which it is housed (Table 1). The Health Policy Research Group, College of Medicine, University of Nigeria Enugu-Campus (HPRG-COMMUNEC), for instance, is a cross college research group that sits within a department of the college but whose membership and remit is not constrained by the department. The Health Policy and Systems Programme/Health Economics Unit, University of Cape Town (HPSP/HEU-UCT) was, at the time of the assessment, a teaching and research programme closely allied with the Health Economics Unit within the School of Public Health and Family Medicine. The largest partner was the School of Public Health, University of the Western Cape (SOPH-UWC) where the whole school had an overt HPSR + A remit, and hence the School itself was assessed. The organisational assessment required partners to inquire into the assets and needs of different capacity themes such as organisational leadership, governance, and resourcing (Figure 1). However, this led to confusion. University financial governance structures, for instance, were not unique to those working in HPSR + A; provision of teaching and research aids were not solely generated or used by HPSR + A units (whether a research group, a department or a school). The School of Public Health, University of Ghana (SPH-UG) assessed all research individuals working in the School but noted that “HPS(R) + A type research is carried on in many schools and departments… [within] University of Ghana [this not only includes the School of Public Health but also] political science department, sociology department … [it is] difficult to be clear as to the exact extent given the absence of any coordinating mechanism and … that much of the output may remain in the grey literature” ([27], p. 20). The assessment captured some of the individuals working either full time or part time in HPSR + A since it was assumed that individuals participating in the assessment should be those nested within the organisational unit. However, this meant that individuals working on HPSR + A outside of the CHEPSAA partner were not included in the assessment. In other words, the CHEPSAA partner was not a freestanding organisational unit with a clear boundary.

Partners also included external partners as part of their assessment but this meant including the capacity assets and needs of these stakeholders. For instance, HPRG-COMMUNEC also included capacity assets and needs of policy makers; the Tropical Institute of Community Health and Development, Great Lakes University of Kisumu (TICH-GLUK) worked with non-academic organizations that are active in health policy as well as policy implementers such as the District Health Management teams. The inclusion of external partners was essential to the CHEPSAA assessment given that HPSR + A is an applied field.

However, when seeking to apply the generic capacity development levels used in existing frameworks, the boundary of the organisational unit in the CHEPSAA assessment became fuzzy. In retrospect, partners solved the confusion by approaching the unit flexibly, paying attention to the existence and quality of internal relationships between individuals and departments within the university, and externally with policy makers and practitioners. Future HPSR + A capacity assessments within universities need to be aware of the networked nature of the field and that this may change over time. At present, the field is embedded within different units and linkages internal to a university. This may change as the field grows when, for instance, there may be more Schools such as SOPH-UWC wholly focused on HPSR + A. However, the field is also heavily committed to strong relationships with external stakeholders (see earlier definition) and these should only endure and strengthen as the field matures.

Each HPSR + A academic unit in the CHEPSAA assessment has a very different institutional nature hence there can be no standard approach to charting a network. However, one possible way of doing so would be to revisit the tasks undertaken by HPSR + A staff and a HPSR + A unit to understand the relative balance between responsibility for tasks and governance/oversight of those tasks. For example, staff within a centre may undertake teaching across programs but employment of such staff is at the discretion of faculty. A future assessment would then explore how to strengthen those networks in the interests of building the field.

A capacity assessment is usually seen as a planning tool that can help organisations plan, strategize, and make decisions on future capacity strengthening activities [2, 12, 17]. The CHEPSAA assessment was used in this way. For instance, in CHEPSAA, the information gathered on the theme of Demand for Research and Teaching (Figure 2) and from staff surveys in which academic staff expressed a desire to teach more which, together with the concurrent review of HPSR + A course material, ultimately led to the development of two open access modules on complex health systems and health policy and systems research.

The assessment also had a powerful awareness raising effect that has not yet been reported in other assessments. There were two aspects to this. First, partners and respondents (Table 2) became more aware of their different conceptual understandings of HPSR + A and this led to efforts to develop a common understanding of the field. Given such differences, the researchers independently developed an information sheet to provide a coherent definition of HPSR + A and to act as a starting point for discussion about the field. Second, by working from this common base, respondents could more easily identify existing assets and recognise future opportunities, as well as future applications of these assets. For example, HPSP/HEU-UCT responded to ideas developed during assessment interviews to establish a journal club between interested academics and government managers in Cape Town to allow continued engagement and development of understandings around the terrain of HPSR + A. TICH-GLUK incorporated the assessment process into an existing mentorship programme by ensuring the assessment team were in mentor-mentee-peer support relationships. Following the assessment, the Centre for Health Policy, School of Public Health, University of Witwatersrand (CHP-WITS) developed a business case for HPSR + A teaching to present to university leadership and senior management and ultimately secured two centrally funded posts for early career academics in recognition of the unit’s contribution to teaching. Many respondents at the Institute of Development Studies, University of Dar Es Salaam (IDS-USDM) assessment expressed the need to establish a national network of HPSR + A academics and the first steps have now been taken to establishing a network of six universities teaching in the field. In other words, a significant unintended positive consequence of the capacity assessment process was that it produced outcomes associated with capacity strengthening interventions. This is consistent with Horton’s assertion that “every evaluation [assessment] of a capacity development effort should itself contribute to… capacity… and ultimately to the organization’s performance” [28].

It is acknowledged here that doing capacity assessments requires a certain level of capacity. The methodology described earlier was built through collaboration, but collaboration has a cost. Staff time, IT infrastructure and software, and a travel budget were required for collaboration by all participating partners. Resources were required to undertake data collection and analysis – for instance, ethical approvals were obtained for fieldwork; significant time was spent setting up appointments for interview and focus groups with policy makers, many of whom had competing schedules; TICH-GLUK notably trained staff to act as survey enumerators for an element of staff skills assessment. Not least, resources were also necessary to synthesise data, write reports, and conduct comparative analyses. This does not negate the capacity strengthening effect of assessments, but does mean that assessments should be designed to make the best use of existing capacity, rather than overstretch it.

The capacity assessment methodology reported here was developed and implemented across a diverse range of African universities. Even though our focus was specifically on African universities, we believe our results are also largely applicable to universities in other lower middle income countries with yet to fully establish HPSR + A within its institutions, because the nature of HPSR + A work is likely to be similar and will include different combinations of teaching, research including its communication, consultancies and advisory functions, and networking activities. However, one possible significant difference which may affect the adoption of our methodology relates to resource environments within different contexts. For HPSR + A units in other African universities and other continents wishing to conduct such an assessment, a number of methodological considerations are proposed.

First, consider including deliberate elements of self-assessment because it can be empowering for staff to engage directly with colleagues and leaders that can significantly influence the field. The potential limitations of self-assessment can be mitigated through standardising a number of components, ensuring a common understanding of key terms and methods, and continual engagement between partners in development and implementation of assessment methodology. Second, consider assets as well as needs because it is more likely to identify resources that teams may not be aware of or taking sufficient advantage of. Third, address the capacity potential of three levels – individual, organisational, wider environment – since doing so will help plan more comprehensive capacity strengthening activities. However, be aware that HPSR + A activities are embedded in different networks between and within these level – a more networked approach to assessing capacity has been proposed in an earlier section. Fourth, take a phased and incremental approach because it is both practical in terms of time and resources available and can give time to assessment teams to reflect on what they are learning in the process since it allows iterative data collection and analysis. Fifth, use methods and tools that assessment teams are familiar with because it makes best use of existing resources. Using unfamiliar methods and tools in an assessment incorporating self-assessment means that capacity development will be necessary before the assessment can begin. It is not always necessary to develop new methodological skills during a capacity assessment.

Finally, seek and fully cost genuine collaboration from the beginning because genuine collaboration means ownership of process and outcome but does require significant time, finance, and other resources to be invested (also noted in [21, 29]). When collaborating across large consortia, such investment will pay off in clear leadership at consortium and partner level and a clear methodological framework. Even where assessments do not include elements of self-assessment, genuine partnership between external assessors and those being evaluated can deepen external understanding of the everyday concerns of those participating in an assessment, and vice versa, and help those being evaluated to better understand the value of assessment for capacity strengthening.