Incidence estimation from sentinel surveillance data; a simulation study and application to data from the Belgian laboratory sentinel surveillance

We will focus on laboratory sentinel surveillance. Our main objective is the estimation of the total number of newly lab-confirmed cases during a year in the area under investigation, Belgium, from a sample of lab-confirmed cases detected by the sentinel surveillance using IPW-methods. We compare how well design-based (adjusted) weights and the weights estimated by the Huff-model can estimate the total number of cases in a simulation study. We then apply the best performing method to data from the Belgian laboratory sentinel surveillance. Labs that are not reporting cases are considered non-participants to sentinel surveillance by default. We investigate the effect of this assumption, by including the true participation status (in the simulation study) or an estimate for participation (in the data examples) in the analysis.

Inverse probability weighting methods use weights to compute linearly weighted estimates of totals. The weights (w_a) are an estimate of the inverse of the probability of detection by the sentinel surveillance. To estimate the total number of cases a Horvitz-Thompson-type estimator is calculated as [22];

The estimated total \( {\hat{N}}_a \) for an area (a) is obtained by multiplying the number of cases detected by the sentinel surveillance by the inverse probability of detection. The sum of the areas needs to equal the total area of interest. The total area of interest of this study is Belgium (national). We use the existing administrative divisions to divide this area in smaller sub-areas: NIS5 < NIS2 < Provinces<Regions<National (from smallest to largest). We estimate the total number of cases by province using design-based weights.

The Huff-model is a spatial interaction model for retailing and services and belongs to the family of probabilistic market area models [16].

The Huff-model calculates the ‘market share’ (P_aj) of lab j for area a based on a spatial component S (a direct path between the centroid of a and address of j in km) and an aspatial component, attractiveness A (a simulated quantity for lab j). Areas are NIS5-areas (municipalities). Since the model is nonlinear (exponential weighting), parameters α and β are estimated by ordinary least squares regression after applying multiple step log-centering transformation to the variables [23]. Once α and β are estimated from the sentinel data set, the market share of every lab for every NIS5-area (a₃) and subsequently the market share of sentinel labs for area a \( \left({p}_{a_3. sentinel}\right) \) can be estimated.

In contrast to the other weights, the weight associated with the Huff-model allow for incidences and catchment populations to be calculated. In the Huff-model each sentinel lab potentially (depending on the distance decay function) contributes to the weights in all areas. With the other methods, a sentinel lab only contributes to the weight in the area in which it is located. In addition, a detected case contributes to the total in the area from which it originated with the Huff-model, while, with the other methods, cases contribute to the area in which the lab is located. Incidence estimates are obtained by dividing the estimated number of lab-confirmed cases by the total population/100 000 at the NIS5-area. The catchment population is the result of summing the product of the market share of a lab in a community and community population over the communities.

An overview of the different methods is provided (Table 1).

Because we assume that not only the distance determines the spatial relation between a lab and a NIS5-area, but also if and which regional borders have to be crossed and the clustering of labs, we extended the spatial component before we apply the model to the Belgian laboratory sentinel surveillance network. The spatial component is calculated as follows:

The areas a represent communities as defined by NIS5-codes (N = 589). D_aj represents the distance (in kilometers, most direct path) between the location of the lab and the centroid of the NIS5-area (since more detailed addresses are not available for the cases). While the Huff-model takes competition into account when estimating market shares by dividing an area’s market over all labs active in the area, we also allow for dependence between the distance function and the number of laboratories in a 20 km radius Rad_j. We introduce a categorical variable to capture the effect of the regional borders, which mostly coincides with the language borders; Regional Border (RB_aj). The three factors in the RB_aj-variable are the following:

Because we have no single measure for the attractiveness of a lab (A_j), we included the following variables to calculate the aspatial attractiveness component;

The number of reimbursements per lab (RS_j) is a continuous variable representing the number of tests for which lab j is reimbursed. Depending on the pathogen under investigation this can be a number specific to that pathogen or specific to a group of pathogens. A lab-concept variable (LC_j) is introduced to separate ‘hospital-associated’ laboratories from peripheral labs. The factor is coded ‘H’ for ‘hospital-associated’-laboratories and ‘P’ for peripheral labs. The number of different departments of a lab is coded LD_j. A department refers to a different location at which lab-tests are also performed. A normally distributed random effect for the laboratories (b_j) is also included. Data from the sentinel labs are used for the model fit. An intercept is included in the model. Model coefficients for which the 95% confidence interval contains zero are set to zero. Details on the extended Huff-model and the coefficient estimation are provided in Appendix.

A 95% bootstrap-based confidence interval is obtained by resampling the records used for the coefficient and market share estimation. A total of 1000 bootstrap samples are used for the computation of each confidence interval.

To limit the variability associated with small probabilities, we do not estimate the total number of cases in areas with a weight higher than 10 (or sentinel coverage lower than 10%). This is done for the design-based estimators and the Huff-model.

To study the influence of knowledge on the participation status of labs, in contrast to having to assume no participation when no cases are reported, we calculate estimates using prior knowledge on the participation status within the simulation study and estimate the participation of labs in our data example. An additional ‘.S’ (e.g. estHuff. NIS5. S) is added to the estimator to distinguish it from the estimators for which participation is inferred from reporting (e.g. estHuff. NIS5) in the simulation study. We estimate participation as a discrete quantity between 0 and 1 in the data example. Whenever a lab is reporting cases associated to a certain pathogen, its participation status is 1 for that pathogen. If a lab reports no cases for a certain pathogen, its participation status is estimated from that labs’ reporting of other pathogens. We calculate this quantity (Psentinellab) as how many of the twelve most frequently reported pathogens, including viruses (e.g. influenza, rotavirus), parasites (e.g. Cryptosporidium spp., Giardia spp.) and bacteria (e.g. Yersinia enterocolitica, Streptococcus pyogenes), are reported by that lab.

Data are provided by the Belgian laboratory sentinel surveillance network. This network was previously described in Muyldermans et al. [24]. The network relies on voluntary participating labs that submit data on a set of around 35 pathogens. The list of pathogens under surveillance is constituted during a yearly meeting. The reported data consist of patient demographic data; postal code, date of birth and gender and data on the diagnosis; date of diagnosis, subspecies and type of test. We limit the dataset to data from 2010 to 2015 and to four pathogens; Yersinia enterocolitica, influenza, campylobacter spp and Rotavirus. We chose four commonly reported pathogens that were under surveillance before, during and after the period 2010 to 2015. An additional motivation for the choice of pathogens was that these represent the different categories of the nomenclature numbers; a pathogen-specific, a group-specific and a non-specific number.

Data on reimbursed microbiology tests were obtained from the Belgian National Institute for Health and Disability Insurance (INAMI-RIZIV) for the period 2010–2015. The data consist of the number of reimbursed tests by nomenclature number, year and lab. A specific nomenclature number is available for Rotavirus antigen-test or culture. A number associated with a test for a group of enteric pathogens (< 6) is available for Yersinia enterocolitica and Campylobacter spp.. No specific nomenclature is available for influenza. When no specific reimbursement data is available the total number of reimbursements is used.

Some labs have reference activities with regards to specific pathogens. These reference-labs are identified and removed from the dataset for estimation of the total number of lab-confirmed cases. Once the estimation process is completed, their data are added to the estimated total. We exclude the data from reference-labs from the estimation process because these labs have several unique characteristics and are not representative for the other labs. For example, we assume their distance decay function will be less steep as compared to the average distance decay. They do however detect cases and it is essential to allow them to contribute to the total number of lab-confirmed cases.

Data on the location of microbiological labs is also obtained from the INAMI-RIZIV. Data on the Belgian demographics for the period 2010–2015 is obtained from the general directorate statistics Belgium.

A discussion of the performance of the design-based weights is included in Appendix.

We illustrate the model by presenting the reported and estimated incidence and estimated market share for Campylobacter spp. for 2015 (Fig. 5).

The simulation study mirrors some aspects of the Belgian sentinel labs surveillance network. These include the location of the labs, areas of aggregation and associated population sizes which were used to allocate cases to NIS5-areas. We explored this approach previously with the application of capture-recapture methods to surveillance data [25]. Such an approach however has different limitations, most importantly; the results of the simulation study are specific to this setting. In the simulation study several aspects are simplified, for example, we only simulate cases from a constant background incidence. Additionally we also opt for a simple case-definition without differences between cases and the heterogeneity in the detection probability of the labs is limited. With a real dataset, we might expect design-effects such as higher detection probabilities for more severe cases [26].

While adjusting for lab-density did not improve any of our estimates, such a variable could be of interest if it was computed differently. Souty et al. found that including the density of GPs over population improved their IPW-estimate [15]. In our simulation study it was driven by the administrative division in NIS2-areas. Future investigations could focus on the effect of lab-density calculated over custom areas with the labs at their center.

Market shares are, in contrast to design-based weights, not directly available. We need a model to estimate them. We find that the model fit is limited. This can be caused by both; a lack of power and unmodelled heterogeneity in the detection of cases by labs. To increase power, we could combine several years of observations and include a temporal component into the analysis. In addition, longitudinal analysis, e.g. a time-series analysis would allow for the smoothing of weights, which could be a solution for the high weights encountered in this study [27]. Furthermore, longitudinal models might be used as possible outbreak/early event detection tools. Variation in coefficients (e.g. positivity rate of reimbursed tests) over time has been proposed as an outbreak detection tool [28]. Keeping in mind however that temporal variation might also be the consequence of changes in lab tests or policies [29]. It is likely also possible to combine data from similar pathogens to increase power. In addition to the power necessary to estimate the coefficients of the extended Huff-model, sentinel surveillance should also have enough power, high enough market shares, in the areas of interest to detect events and describe trends. The Huff-model allows us to optimize power by taking lab-profiles into account [30]; market shares can be used to estimate both spatial and population coverage associated with a specific lab. We find that; the Belgian laboratory sentinel surveillance is a network of convenience, with a high geographical overlap in the north of Belgian and low market shares central and south. This unequal spatial coverage in combination with an exponential distance decay function will lead to very small market shares in some areas and consequently to unstable estimates and a reduction of the effective sample size [31]. An arbitrary cut-off or piecewise distance decay function can be used to avoid a laboratory’s small market shares in distant areas, but might lead to artificial observations. E.g. our choice to ignore weights larger than 10 reduces the instability of the IPW-estimator, but also leads to a considerable RMSE for the Huff-model when there are few sentinel labs in a simulated scenario with spatial heterogeneity. Other techniques to improve the efficiency of IPW have been explored in epidemiology, notably weight stabilization and augmented IPW [31].

Among the significant coefficients we did find, we found a significant and positive relation between the number of reimbursements and the market share in all data examples. We had however anticipated a higher effect size. Our results are in accordance with previous research that established a large variation in lab use by physicians [32]. We are likely missing important variables which help determine labs’ market shares. Several of these will be hard to capture as they can be pathogen and time-specific. For example; an informal referral system might exist among the labs. Referring samples is straightforward as they are easy to transport. What constitutes the attractiveness of a lab for cases will likely be different from what constitutes attractiveness for other labs. Another possible cause of heterogeneity in observed market shares are local outbreaks that are unaccompanied by an equal increase in testing behavior and that are not equally detected by all labs active in that area. Consider, for example, an outbreak in a hospital linked to one specific lab. An increase in tests due to a local screening program or change in attention/attitude that is not shared by all labs active in an area, can also cause a limited model fit. State specific health policies are a known cause of spatial heterogeneous reporting in sentinel surveillance and could further explain our limited model fit [11]. Health policies relevant to laboratory testing however are tied to the federal level and therefore common over all of Belgium. Public health policy did establish reference centers for laboratory tests. We excluded these labs from the estimation process because of their unique position.

In econometric studies ‘past performance’ is sometimes included as laboratory/store-specific variable [33]. With such a variable it is not necessary to identify all elements contributing to attractiveness. The significance of the random effect in our extended Huff-model points to the usefulness of such a parameter. Unfortunately such a variable would not be available for many labs as they have never participated in sentinel surveillance and it is not possible to estimate this for non-participating labs. A one-time measure, either through a survey or other data collection method could improve the model fit.

Finally, data quality issues cannot be remedied by our model and will contribute to a poor model fit. We assume reporting consistency on a yearly basis; a lab that is participating to sentinel surveillance should report all detected cases. Incomplete reporting (not reporting all detected cases), low data quality (typos, missing variables such as case location) and variations in the case-confirmation/definition among laboratories (only reporting on cases detected with a certain test or not being able to perform a certain test) will lead to a poor model fit.

A previous estimate of the representativeness of the Belgian laboratory sentinel surveillance network, obtained through analysis of the proportion of tests reimbursed by labs participating to sentinel surveillance, has found a coverage of around 50% [34]. The central assumption of that study was that sentinel labs report an equal number of cases as non-sentinel labs given correction for the number of reimbursed tests. We opt for a less strict assumption, by also correcting for other variables, such as lab-density, location and regional borders.

The absence of a participation status, not being able to distinguish zero-reporting labs (labs reporting no cases) from non-participating labs, is a hindrance to all methods and explains why the Huff-model has a larger RMSE is some simulation scenarios. This is an important limitation of the Belgian laboratory sentinel surveillance. This is further complicated by small yearly changes to the list of pathogens under surveillance. The assumption that a sentinel lab that did not report any cases did not participate and the assumption that laboratories that are not reporting any cases participate with a probability estimated from the reporting of other, frequently reported, pathogens are both strong. To illustrate this, we present some observations. Eighty labs, out of 161 microbiological labs, have participated to sentinel surveillance in 2015. A total of 35 pathogens were under surveillance in 2015. No lab reported cases for all the pathogens, two labs reported cases for more than 80% of the pathogens, 28 labs reported cases for more than 50% of the pathogens under surveillance in 2015. So even for common pathogens such as Rotavirus or influenza only 88 and 58% of all sentinel labs reported cases. In addition to a participation status to sentinel surveillance, it would be essential to know if a lab has the technical capacity to perform a certain test. Throughout this paper we assumed that all labs could detect all cases.

We apply a bootstrap algorithm for the estimation of confidence intervals. This is time consuming and analytical methods are available for direct confidence interval estimation even in a small area estimation setting. For example; using the weights obtained from the sampling design, effective sample sizes can be calculated [26]. Other methods are available to obtain a full probability model for small area estimation [35]. The nature of sentinel surveillance data, the quality and methodology of the systems, might however make additional variance inflation components necessary [36]. Additional research into the unmodelled heterogeneity is necessary to come to such a variance inflation component. Future extensions to the Huff-model include informative priors in a full Bayesian framework. Sample weights have already been incorporated in 3-stage Bayesian hierarchical models, including variance estimation [26]. Other methods (optimization algorithms such as harmony search) have also been developed to estimate the coefficients of the Huff-model [10].