Background
Methods
Design
Setting
Figures used for testing
Participants and recruitment
Data collection process
Expert assessment
Analyses
FIAT-health 2.0
Results
Comparison of potential user and expert assessments
Justifications provided by both experts and potential users | |
• The correctness of the methods | |
• Assumptions on which the model is based | |
• Match between the primary publication and the reported figure | |
• Transparency on the definition of the subject | |
• The conclusion that was made based on the results | |
• Previous knowledge of the subject | |
• Application of the figure in practice | |
• An extrapolation was made | |
• The geographical area the figure applies to | |
• It concerned an estimation | |
• No better figures are known about the subject | |
• Source of the figure | |
• Time period to which the figure relates | |
• Match between the population of the reported figure and the primary publication | |
• Generalization of the figure | |
• Interpretation of the journalist | |
• Difference in jargon between the primary publication and the reported figure | |
Justifications provided by potential users | |
• Credibility of the author | |
• Verifiability of the figure | |
• New information in the [publicly reported message] | |
Justifications provided by experts | |
• Method of modelling (the figure has no meaning as the expert considered the construct to be invalid) |
Evaluation of the FIAT-health 1.0
Topic | Policy Makers | Researchers | Communication officers | Students | Total |
---|---|---|---|---|---|
High time investment | 4 | 2 | 2 | 1 | 9 |
The questionnaire/Excel sheet is complex | 5 | 2 | 3 | 1 | 11 |
Time investment of checking the primary publication | 1 | 2 | 3 | ||
Reference to the primary publication is helpful | 2 | 1 | 1 | 4 | |
The structured way of assessing is good for creating awareness of the characteristics of the figure | 5 | 3 | 3 | 3 | 14 |
Language is complex | 1 | 3 | 1 | 2 | 7 |
Questions could be more in-depth | 1 | 1 | |||
Goal of the questionnaire is unclear | 2 | 1 | 3 | ||
Explanations clarify the question | 1 | 2 | 3 | ||
Not all questions are relevant for the figure | 1 | 1 | 2 | ||
Unclear meaning primary publication | 1 | 1 | |||
The FIAT-Health 1.0 is user friendly | 1 | 1 | 2 | 4 |
Consistency of the answers
-
3a, Is the figure expressed in absolute terms?
-
3c, Does the figure you are assessing match the figure in the primary publication?
-
4b, Does the definition of the subject of the figure you are assessing match the definition of the subject in the primary publication?
-
5b, Does the definition of the population of the figure you are assessing match the definition in the primary publication?
-
7a Is the time period in which the units are counted described in the primary publication?
-
7b, Does the time period to which the figure applies match the time period in the primary publication?
-
8a, Are the data on which the figure is based collected periodically?
-
10a, Were the data collected through an existing registration? and
-
13a, Was the figure constructed through modelling?
FIAT-health 2.0
FIAT-Health 2.0 | |
---|---|
What figure would you like to assess? (Provide the phrase in which the figure is mentioned.) | |
Question 1. Origin of the figure | |
1a. | Is the publication in which the figure is reported a primary publication?a |
1b. | Is the primary publication known? |
1c. | Is the primary publication verifiable? |
If the primary publication is not accessible, the FIAT-Health cannot be used. Access to the primary publication is necessary to assess the reported figure with the FIAT-Health 2.0.
| |
1d. | Does the figure you are assessing match the figure in the primary publication? |
Question 2. Credibility of the figure | |
2a. | Under the responsibility of what institute has the primary publications been published? |
2b. | How credible do you consider the author of the primary publication [in relation to this particular figure]? |
Question 3. Expression of the figure | |
3. | How is the reported figure expressed? (For example: in absolute terms, percentage, average, fraction) |
Question 4. Subject to which the figure applies | |
4a. | To what subject does the reported figure relate? |
4b. | Is the subject of the reported figure identical to the subject described in the primary publication? |
Question 5. Population to which the figure applies | |
5a. | To what population does the reported figure relate? |
5b. | Is the population of the reported figure identical to the population described in the primary publication? |
Question 6. Geographical area to which the figure applies | |
6a. | To what geographical area does the reported figure relate? |
6b. | Is the geographical area of the reported figure identical to the geographical area described in the primary publication? |
Question 7. Time period to which the figure applies | |
7a. | To what time period does the reported figure relate? |
7b. | Is the time period of the reported figure identical to the time period described in the primary publication? |
Question 8 to 13: Methods of counting and measuring | |
Multiple methods of counting and measuring are possible.
| |
Question 8. Data collection | |
8. | Are the data on which the figure is based collected only once, periodically or continuously? |
Question 9. Sample | |
9a. | Is the figure based on a sample? |
9b. | What is the sample size? |
9c. | What is the response rate? |
9d. | Were important groups disregarded in the calculation of the figure? If yes, which groups? |
9e. | How do you assess the representativeness of the sample? |
Question 10. Registration | |
10a. | Were the data collected through an existing registration? |
10b. | What registration was used? |
10c. | How do you assess the usability of this registration for the calculation of this specific figure? |
Question 11. Survey research | |
11a. | Were the data collected through survey research? |
11b. | Are the questions on which the figure is based described precisely? |
11c. | Are the answer categories of the questions described? |
11d. | How do you assess the conclusion which was made based on the questions and the answer categories? |
Question 12. Direct observations | |
12a. | Are the data collected through direct observations? |
12b. | How the direct observations take place? |
12c. | How do you assess the accuracy of the direct observations? |
Question 13. Modelling | |
13a. | Was the figure constructed through modelling? |
13b. | Are the assumptions which were made in the model known? |
13c. | How do you assess the plausibility of the assumptions made in the model? |
Final assessment | |
How do you assess the correctness of the figure in the primary publication? | |
How do you assess the interpretation of reported figure? |
Discussion
Limitations
Context
FIAT-Health 1.0 | FIAT-Health 2.0 | |
---|---|---|
Factual questions | Closed-ended questions on the characterization of the figure answered with ‘yes’ or ‘no’. | Includes both open-ended questions on the characterization of the figure, answered by taking information from the public report, and closed-ended questions, answered with ‘yes’ or ‘no’. |
Assessment questions | The user gives a rating on a scale from 1 (negative) to 4 (positive) on methodological aspects. | The user describes his or her assessment on methodological aspects providing a numerical rating. |
Questions on the primary publication | Questions on the consistency between the publicly reported figure and the figure as described in the primary publication. | Questions on the consistency between the publicly reported figure and the figure as described in the primary publication are rephrased. |
Final assessment | The user rates the ‘correctness of the original figure in the primary publication ‘and the ‘appropriateness of the figure in the report’ on a scale from 1 (negative) to 5 (positive). | The user describes his or her considerations on their assessment of the ‘correctness of the figure in the primary publication’, and the ‘interpretation of the reported figure’. |
Checklist | Can be used as a checklist. | Use of the FIAT-Health 2.0 as a checklist is made easier due to its simplified format. |