Development and content validity of the Abilitator: a self-report questionnaire on work ability and functioning aimed at the population in a weak labour market position

The clients participating in services provided by the ESF Priority 5 projects represented the target group sample. They were mainly of working age, had been unemployed for several years, and faced various problems related to their health, lifestyles and life situations. They participated in the projects voluntarily to improve their work ability, functioning and employment opportunities. On average they were 40 years of age, had been unemployed from 3 to 7 years and had little post-compulsory education.

The essential development of the Abilitator was carried out by an internal group of experts (n = 8) from FIOH representing medical, health, sport, behavioural, and social sciences. One member of this group was from THL. The first external expert panel included academic specialists (n = 30) from the fields of work ability, functioning and social inclusion. The second external expert panel included both professionals and target group clients, bringing expertise through experience to the development. The professionals (n = 700) had varying lengths of work experience with the target population, and their occupational backgrounds ranged from university research scientists to social workers and sports coaches. They mostly worked in ESF Priority 5 projects. The target group experts (n = 28) were clients who received the services in the ESF Priority 5 projects.

The development of the Abilitator progressed through seven phases between 2014 and 2017 (Fig. 1). In Phase 1, we reviewed the related theoretical models and the existing PROMs of work ability and functioning used in Finland and more widely in the EU. We searched Google Scholar and PubMed using terms such as work ability, functioning, functional capacity, model, method, concept, theoretical, self-report, self-assessment, and questionnaire. We also identified self-report instruments from the Finnish TOIMIA network’s database, which contains guidelines for the measurement of functioning and evaluations of these measures [42]. The inclusion criteria for the instruments were their proven reliability, validity and wide use in previous research of the working-age population. Further inclusion criteria were free access, availability in the Finnish language and different occupational groups being able to use the instrument safely. The first version (0.1) of the Abilitator was formed based on the review and co-development with academic experts (n = 20).

After Abilitator 0.1 was implemented in 44 ESF Priority 5 projects in Phase 2, we co-developed it with professionals, academic experts and clients to form the second version (0.2) (Phase 3). The professionals’ (n = 600) experiences of and suggestions regarding the content of Abilitator 0.1 were collected in 22 local or national co-development workshops. Each workshop covered the content of the whole questionnaire in a similar manner to determine: 1) whether each question was relevant for the target group, 2) whether each question was formulated in a way that was appropriate for the target group, and 3) what kind of alterations should be made to each question for them to better suit the target groups’ needs or situations. Similar feedback was gathered from the professionals during 10 visits to different EFS Priority 5 projects. One group of clients (n = 7) also suggested question alterations and academic experts (n = 15) gave their input in separate encounters at this stage.

In Phase 4, Abilitator 0.2 was implemented in the ESF Priority 5 projects. After this, cognitive interviews were conducted with clients (n = 21) participating in five different national or local ESF Priority 5 projects (Phase 5) [43]. The interviewed groups had good geographical, gender and target group coverage. The aim of the interviews was to obtain information on how the respondents had processed and interpreted the questions of Abilitator 0.2. The interviews used a four-step question-answer process [44] related to the format, feasibility and comprehension of the questions. All the interviews were conducted by two interviewers and progressed following the same pattern. Each interview was recorded and transcribed.

In Phase 6, we sent an online survey to all the professionals (n = 144) using Abilitator 0.2 to collect additional feedback on the content, feasibility and format of the instrument. We also ran a second national co-development workshop with the professionals (n = 35). We used the information and feedback gained from the survey (n = 42), the workshop and the cognitive interviews to steer the development of Abilitator 0.2’s content and layout. The feedback was again systematically gathered in written format and reviewed by the internal group of experts. The suggestions were grouped into similar feedback units and the decisions regarding changes to the questionnaire were made in the internal expert group’s consensus meetings. At the end of Phase 6, the third version, i.e. Abilitator was ready. During each phase of the development, the professionals were given training and support materials on how to use the Abilitator with their clients and how to interpret the results.

When the content development was complete, the Abilitator was linked to the ICF (Phase 7). The main purpose of this was to translate the instrument’s content into the internationally unified and consistent language of human functioning, which can be used as a reference for comparing health information. This linking was conducted in co-operation with the national ICF concept working group and followed the updated linking rules [45]. It was first conducted by two research scientists separately and consensus was reached in two separate sessions with three other ICF experts. The second purpose was to position the Abilitator among the ICF-linked self-report instruments measuring work ability and functioning, and to compare the Abilitator’s ICF codes with the three ICF Core Sets most relevant to the target population; the generic set (7 codes), the brief vocational rehabilitation set (13 codes) and the minimal environmental set (12 codes) [46].

The Abilitator’s ICF codes were further compared with two validated, central self-report instruments: the Work Ability Index (WAI) (14 codes) and the World Health Organisation Disability Assessment Schedule (WHODAS 2.0) (27 codes). The WAI is used in occupational health services and research to assess employee work ability in health examinations and workplace surveys [39]. It is developed by FIOH to help define the necessary actions for maintaining and promoting work ability. WHODAS 2.0 is a generic assessment instrument that provides a standardised method for measuring health and disability across cultures. It was developed from a comprehensive set of ICF items that are sufficiently reliable for measuring activities and participation [47].

The content validation process of the Abilitator followed the guidelines of COSMIN [48, 49]. It was split into five phases: 1) definition of the construct to be measured and specification of the situation in which the instrument is used, 2) expert panels’ assessment of the instrument’s content during the development process, 3) consideration and provision of information on the instrument’s content, 4) assessment of whether the instrument’s content corresponded to the construct, and 5) assessment of whether the instrument’s construct corresponded to the ICF framework of functioning and relevant ICF Core Sets and other instruments measuring the same construct. Phase 2 included the evaluation of face validity, which is the degree to which the measurement instrument seems to be an adequate reflection of the measured construct [49].

The literature review conducted in Phase 1 (Fig. 1) found eight different theoretical models for work ability [50] and two models for functioning. In the bio-medical model of work ability, an existing illness, impediment or disability determines a person’s attributes and qualities as a worker [13, 51, 52]. In the balance model, work ability is the equilibrium between the individual and work-related factors [51, 53]. The psychosocial model emphasises the psychological and psychosocial factors connected to work participation and return to work [54‐57]. In the multi-dimensional models and the bio-psycho-social models, work ability is a holistic, comprehensive entity in which individual resources and work-related factors are combined by the operational environment and social support [13, 14, 27, 58]. In the employability model, work ability combines all the individual and societal actions that help a person become employed, stay employed and advance their career [59, 60]. According to the model emphasising the integration of the individual at the workplace, the concept of work ability is based on continuous change in work and work organisation [61, 62]. Work ability can also be considered a social construct that is constituted by and differs between different societies and systems [63].

As with the concept of work ability, biomedical and biopsychosocial models have been used to describe functioning and health [58]. An internationally accepted way of structuring the concept of functioning is ICF [15], the framework of which provides a standard language and multi-purpose classification of disability and health [64]. Functioning is a collective umbrella term of the ICF that describes a person’s body structures and functions and their capacity to perform daily activities in the environment in which they live. The ICF is a biopsychosocial model that combines the biomedical, social and environmental aspects of human functioning, health and disability [14, 65]. It can be used as an instrument to collect comparable data to support evidence-based decision-making in health and health-related sectors. WHO and the ICF Research Branch have created Core Sets of ICF which the essential relevant categories for specific health conditions and health care contexts [46].

The ICF framework reflects six different aspects of health and disability: health condition, body structure and body function, activity, participation, environmental factors, and personal factors [66]. Diseases or disorders, i.e. health conditions, are included in the conceptual model of health, but are classified in the International Classification of Diseases and Related Health Problems (ICD) [67]. Functioning should be understood as a continuum ranging from completely able (non-problematic) to completely disabled (problematic) and is the result of complex multifactorial interaction between the six components [68].

An example of the expert panels’ influence on the Abilitator content are questions D8 and D9 [Additional file 1]. Abilitator 0.1 contained two items for screening depression in primary care [74]. During Phase 3, systematic negative feedback from the target group, the professionals and the academic experts led to the removal of these items. The questions were considered too diagnostic to be used by professionals, too invasive to be answered by the respondents, and too difficult to evaluate in the context of their use. However, issues such as taking the initiative in everyday activities were still considered important. Therefore, the group of experts formulated two completely new items, D8 and D9. These questions were added to Abilitator 0.2 [Additional file 1] and the practical group of experts assessed their feasibility during Phases 5 and 6. Due to the systematically positive feedback received, Questions D8 and D9 of the Abilitator remained unchanged.

Abilitator 0.2 contained 76 questions and the online version also offered personal feedback. The content and format of this feedback was developed with the external expert panels along with the content of Abilitator 0.2. The internal expert group decided not to include all the questions in the feedback the Abilitator 0.2 gives the respondent because one single answer is not always enough to make meaningful assumptions about the respondent’s situation. However, the interpretations of all the questions were analysed for the professionals in the Abilitator user manual.

Based on the literature review on instruments in Phase 1 the internal group of experts created the structure and content of Abilitator 0.1 from pre-existing questionnaires and some newly formed questions into one self-report questionnaire. The literature review identified 55 self-report instruments of work ability and functioning, of which 14 were used based on face validity or partially by combining the most relevant questions. The consultations of academic specialists (n = 20) improved the 0.1 pilot version’s content. It was also decided that the Abilitator would retain the questions on overall functioning and work ability i.e. Questions B3 and B4 [Additional file 2] throughout the development process. This was to ensure that ESF Priority 5 projects could assess the overall change in work ability and functioning of their clients even if other parts of the Abilitator changed during its development.

We chose the following topics as the main elements of Abilitator 0.1: 1) Work ability and perceived health, 2) Everyday skills, 3) Social functioning and social involvement, 4) Psychological functioning, 5) Cognitive functioning, 6) Physical functioning, and 7) Background information. These topics covered the first three levels of the House of Work Ability and its dimensions of family, close community and society. Abilitator 0.1 contained 57 questions, of which 30 (54%) were taken directly from pre-existing questionnaires [Additional file 1]. The rest were newly-formed questions covering target group-specific topics that had either not been evaluated by a self-assessment method before or for which the formulation of the pre-existing questions did not directly meet the Abilitator criteria; for example, positive question format, equality, generality, and comprehensiveness.

During the development process, the content of the Abilitator was modified twice [Additional file 1]. All the feedback on Abilitator 0.1 and 0.2 was systematically gathered in written format and reviewed in detail by the internal group of experts (Phase 3, 5 and 6, Fig. 1). The suggestions were grouped into similar feedback units and the decisions regarding changes to the questionnaire were made in the internal expert group’s consensus meetings. As a result, 25% of the questions in Abilitator 0.1 remained unchanged, 50% were modified and 25% were removed. The unchanged questions were perceived as feasible for and by the target group and for evaluative purposes. The content or formulation of the questions was changed if: 1) the questions were not perceived as equal, 2) the questions’ original design was not perceived as suitable for the target group, 3) the questions’ original design did not reveal the desired issue precisely enough, 4) the questions required more text to support their comprehension, 5) the questions’ themes were perceived as too narrow or extensive, 6) the questions lacked important areas or response options and 7) the questions had too many or too few response options. The questions removed from Abilitator 0.1 were: 1) not answered as regularly as the others, 2) perceived as repetition, 3) not perceived as appropriate for or by the target population, 4) not perceived as covering the desired aspect, 5) not perceived as equal and 6) too difficult to answer. Nineteen completely new questions were added to Abilitator 0.2. If important issues or sub-issues were completely missing, or if new questions were needed to better suit the target groups’ situation, the removed question was replaced by a new one.

In Abilitator 0.1, the recall period varied from the present to 2 weeks or a month. According to the feedback, this was confusing to both the respondents and the professionals. Therefore, in Abilitator 0.2, the recall period was harmonised to the current situation, except for Section D (Mind) in which the recall period was set as one month. In addition, the scales were harmonised and presented either horizontally or vertically, and the best option was always at the furthest right or at the top, respectively.

When the final Abilitator was created, 60% of the questions in Abilitator 0.2 remained unchanged, 38% were modified and 2% were removed. The unchanged questions were perceived as feasible for and by the target group and for evaluation purposes. The content or formulation of the questions was changed if: 1) the questions required more text to support their comprehension, 2) the order of the questions was not logical within the sections of the questionnaire, 3) the question’s topic was too extensive to answer and needed splitting into two separate questions. Based on the feedback received, we added three new questions to the questionnaire to obtain a broader view of the respondent’s situation. At the end of the development process, the Abilitator contained 84 questions of which 17 were items from existing questionnaires, and 67 were either modifications or completely new items. The content of the personal feedback did not change significantly.

The Abilitator contains nine sections: A. Personal information, B. Well-being, C. Inclusion, D. Mind, E. Everyday life, F. Skills, G. Body, H. Background information, and I. Work and the Future (Fig. 3). Each section contains 4–14 questions. In Fig. 3, the Abilitator’s sections are further linked to general concepts and the Abilitator’s concept framework presented in Fig. 2. The whole questionnaire is presented in Additional file 2 and can also be accessed online [75].

The interpretation of the results as given in a respondent’s written feedback can be seen in Additional file 3. The feedback is built directly on the response options and has no external benchmark figures. The measure of each section is a summary scale of the selected item. The points received are converted into percentages: the minimum score is 0% and the maximum 100%. The feedback is grouped on the basis of the respondent’s situation per sections B–G: 1) the situation is good, 2) the situation is fairly good, but has some possible challenges and 3) the situation is fairly poor or poor. If the respondent evaluates some items as very poor and others as good, the feedback indicates possible challenges. The Abilitator’s content and its development versions 0.1 and 0.2, the scales, and the ICF codes by question are illustrated in Additional file 1. Another way in which to interpret the results is to do so question by question. The instructions for this are presented in the Abilitator’s user manual, currently only available in Finnish [75].

In practice, the Abilitator can be used in different ways. A service actor working in, for example, employment services can send the client a personal link to the Abilitator via email well before a scheduled appointment. On average the questionnaire takes 15–20 min to complete. The client can complete the questionnaire online independently or with a close person. Another option is that the service actor interviews the client and enters the responses directly into the online version of the Abilitator. A third option is that the service actor either gives or sends the Abilitator questionnaire in paper format to the client. The client then completes the questionnaire and returns it to the service actor, who enters the information into the online version.

The advantage of the online version of the Abilitator is that both the client and the service actor can see the results and personal feedback and prepare for their appointment accordingly. During the appointment, the client and the service actor can discuss the results, and plan targets and actions to improve or sustain the client’s work ability or functioning if necessary. In an ideal situation, they arrange a follow-up appointment during which they evaluate whether these targets have been met.

The Abilitator covered 79 ICF codes, of which 14 (18%) described body functions and structures (b), 40 (50%) activities and participation (d), 10 (13%) environmental factors (e) and 15 (19%) personal factors (pf). The ICF codes describing body structures and functions were related to global and specific mental and respiratory system functions. The codes related to activities and participation covered learning and applying knowledge, carrying out general tasks and demands, communication, mobility, self-care, domestic life, interpersonal interactions and relationships, and major life areas. The codes covering environmental factors described products and technology, support, relationships, and attitudes. The correspondence of all the Abilitator’s items to the ICF categories is illustrated in Fig. 4 and Additional file 1.

The direct equivalence of the Abilitator to the generic set was 4/7 codes (57%); to the brief vocational rehabilitation set, 10/13 codes (77%); and to the minimal environmental set, 1/12 codes (8%). In addition, two d4-category codes of the generic set, one e4-category code of the brief vocational rehabilitation set, and the e3-category codes of the minimal environmental set were indirectly represented in the Abilitator at another category level [Additional file 4].

The direct equivalence of the Abilitator was 5/14 (36%) WAI codes. In addition, similar aspects of four codes were indirectly covered in the e3- and e4-categories. The direct equivalence of the Abilitator was 12/27 (44%) codes of WHODAS 2.0, and there were only minor differences in the codes concerning categories d4 and d5 [Additional file 4].

The first strength of this study is that its structured, long-term, multidimensional development process was combined with extensive co-development. The use different of expert panels led to combining science and practice, which improved the Abilitator’s content, usability and accessibility. The second strength is its clear, well-structured content validation process and its documentation. The third strength relates to the extensive process of linking the Abilitator to ICF, in which the co-operation and work contribution of academic experts was crucial.

The limitations of this study relate to both the Abilitator as a self-report method and the study itself. The first Abilitator-related limitation concerns the interpretation of results. As the assessment is within-subject only and the scales within it are not yet validated as such, the Abilitator cannot yet be used to measure the level of work ability and functioning. On one hand, no benchmark data on the Abilitator are available because the instrument has only been used in the population in a weak labour market position. On the other hand, very strong benchmark data are available for some parts, because the Abilitator contains the same items as the nationally representative population’s health, functioning and welfare surveys [78, 79], including self-rated health and work ability. However, at this point, the Abilitator results may be presented as an approximation of work ability and functioning. The Abilitator should only be used as an indicative instrument, to indicate the respondent’s work ability- and functioning-related resources when the interpretation of the results is the same as the content of the questions.

The second limitation related to the Abilitator is its lack of coverage on work ability factors especially those of work-related specific skills, competence, values and attitudes. If the Abilitator is revised, it might be appropriate to consider whether these missing factors should be added to the questionnaire.

The third Abilitator-related limitation relates to the context of its use. First, the situation in which the Abilitator is applied needs to be considered, that is, 1) in what services the assessment is conducted, 2) how it is applied, 3) what information can be utilised in the context of use, and 4) what kind of actions can be carried out in the services in terms of the results. Furthermore, the process of applying the Abilitator needs to be evaluated, i.e. how well the method sits in the processes carried out in the services, how the method is implemented, and how is it realised. These aspects need to be investigated in future studies.

The first methodological limitation concerns the lack of a systematic review of existing self-report instruments on work ability and functioning. The literature review mainly focused on methods available in Finnish and with free access. Therefore, we may have missed some potential methods during the Abilitator’s initial development.

The second limitation relates to the groups participating in the co-development of the Abilitator, as they were mostly professionals and clients in the ESF Priority 5 projects. This may have led to the exclusion of some relevant views of other professionals working with or belonging to the target population.

Despite these limitations, the Abilitator has the potential to become a generic, feasible, multidimensional self-report instrument for evaluating the work ability and functioning of individuals in a weak labour market position who use different services. The information on work ability and functioning derived from the Abilitator could be used in these services to discuss and work together with individuals towards the most effective service plans. The positive and empowering form of the Abilitator could also support individuals’ agency. For organisations providing these different services, the Abilitator could be an instrument for systematic data collection on work ability and functioning. It has been suggested that different actors producing health and employment services should form closer partnerships and co-operative networks [33]. The most beneficial situation would be one in which the service clients, actors and organisations operate within the same system when assessing work ability and functioning. The Abilitator could be the unified instrument used in different services to strengthen this co-operation when working with the population in a weak labour market position.