Sales volume data for anti-malarials and RDTs were collected through the recall (RC) and the retail audit (RA) methods, together referred to as the sales level surveys (SLS) in retail and wholesale commercial anti-malarial providers. The aim of the SLS was to explore whether RC and RA for measuring sales volumes agreed sufficiently that they can be used interchangeably. The relative strengths and weaknesses of each method were also analysed from an implementation perspective using qualitative methods.
The sales level surveys
RA consisted of visiting each sampled outlet two times with a two-week time interval between each visit. At the first visit, referred to as the sales level survey 1 (SLS1), data on quantities stocked of each product were collected. At the second visit (SLS2), data on quantities stocked, quantities delivered between first (SLS1) and second (SLS2) visits, and quantities thrown away/transferred to other shops or sent back to wholesalers or confiscated were collected for each product in stock, including products in stock at either or both visits. Quantities stocked were physically counted where possible or providers were asked to state the quantities in stock. To collect data on quantities delivered and disposed of, providers were asked to check any available written records or sales receipts and in the absence of records, to recall these quantities. RC consisted of asking retailers and wholesalers to recall the quantities sold during the two-week time interval between SLS1 and SLS2. It was implemented at the start of SLS2 before collecting stock data in order to minimize bias as recall data may have been influenced by the process of counting stocks for the RA. A time interval of two weeks between the two visits at each outlet was chosen based on the existing literature in which a two-week time interval was considered reasonable for capturing wholesale deliveries [
16,
21]. A recall period of two weeks has also generally been used for collecting data such as fever episodes in household surveys [
23‐
25].
The SLS sampling strategy drew on data collected during the ACTwatch retail outlet and supply chain surveys, which are described in detail elsewhere [
18]. Briefly, for the ACTwatch retail outlet survey, a sample of 38 administrative clusters (health centre areas with catchment populations of 10–15,000 inhabitants) were selected with probability proportional to size from all 255 malaria endemic clusters in Cambodia [
1]. Then, a census of all public and private outlets in these 38 clusters was completed and a list of those stocking anti-malarial drugs was created [
1]. At each retail outlet, data were collected on the two most important wholesale supply sources for anti-malarial drugs. All anti-malarial wholesale supply sources mentioned by retailers were visited during the ACTwatch supply chain study. Data were collected on the two most important wholesale supply sources. This process was then repeated until the top of the chain was reached.
For the SLS, retail and wholesale outlets were purposively sampled from the list of anti-malarial retailers and wholesalers surveyed during the ACTwatch surveys. The geographical location of each commercial outlet and the number of outlets stocking anti-malarials in each location at the time of the outlet survey were used to select areas in which all outlets could be visited two times with a two-week time interval (in order to conduct the RA component).
A total of 107 retailers and 67 wholesale outlets were sampled. Wholesale and retail outlets not found, not stocking anti-malarials or not available at the time of the SLS were not replaced.
At retail level, the SLS was conducted as a standalone survey few months after the ACTwatch retail outlet survey. At wholesale level, the SLS took place during the ACTwatch supply chain survey. The questions relating to SLS1 (questions about quantities stocked) were asked after the supply chain survey questionnaire was administered, whilst SLS2 (questions about recall sales volumes, quantities stocked and quantities received and disposed of) was conducted two weeks later as a standalone survey.
All data collection tools were translated from English to Khmer and piloted before the start of data collection. A team of two interviewers entered each business, informed shopkeepers about the study objective and obtained consent. Interviews were conducted in Khmer, with the person most involved in the management of the business. Interviews were conducted in the premises, with breaks each time a customer arrived. Interviewers then asked whether they could return after two weeks and if so they arranged an appointment, and returned on that date.
All types of anti-malarial drugs in all dosage forms and packaging types, and RDTs were surveyed. For anti-malarials, data were collected in terms of both full packs and loose tablets (ie, those kept in containers/tins). Stock data for anti-malarials stored in half-full containers were estimated based on the height of the tablets in the pot measured using a ruler and the number of tablets in a full pot. RDT data were collected in terms of single RDT units.
For each anti-malarial observation, volume estimates were converted into adult equivalent treatment doses (AETDs) [
1]. One AETD was defined as the amount of the drug needed for a full adult course of treatment based on guidelines from the World Health Organization (WHO) where available, or else from peer-reviewed literature or manufacturers. Anti-malarials missing data required to calculate AETDs (eg, drug strength) were excluded from the sales volume estimation [
1].
RA estimates were calculated as: (total quantities stocked at SLS1) + (quantities delivered between SLS1 and SLS2) – (quantities disposed of between SLS1 and SLS2) – (total quantities stocked at SLS2).
Negative RA estimates indicating data collection errors during the SLS and anti-malarial/RDT observations without both RA and RC estimates were excluded from the analysis.
In outlets with sales data for more than one type of anti-malarial/RDT the sum of all RC estimates and of all RA estimates was calculated in order to obtain for each outlet single total sales volume estimates with each method.
The level of agreement between the two methods was explored following the Bland-Altman approach [
26,
27].
The first step was to calculate, for each outlet, the difference between RA and RC sales volume estimates for outlet
i. Formally:
where RAi and RCi are sales volumes estimated through the two different methods at outlet i.
The second step was to estimate the “bias of the measurement” between the two methods, which is the mean of the differences between the two different methods (b1), and its standard deviation (SD) (b2). Formally:
(b1)
(b2)
where is the difference between RA and RC in outleti the mean of the differences between RA and RC across all outlets and n the total number of outlets with a pair of RA and RC estimates.
Differences between sales volume estimates were plotted on a histogram (not shown) to verify that they were approximately normally distributed.
The third step was to investigate for each outlet whether there was an association between the total volume sold and the bias (ie, the mean of the difference, b1). This is because for the bias to be a meaningful estimate of the level of agreement between the two different methods, it should be constant throughout the range of measurements [
26,
27]. In the absence of a recognized gold standard method for measuring sales volumes, an outlet’s “true” total sales volume was proxied as the mean of RC and RA estimates. Formally:
The association between total volume sold (c) and measurement bias (b1) was explored graphically using a scatter plot of the differences against total volume sold and confirmed statistically using a correlation coefficient obtained through the STATA command
baplot[
28].
The fourth step was to calculate the interval within which 95% of paired estimates were expected to lie, referred to as the upper and lower limits of agreement (LoA) between the two methods [
26]. Formally:
(d)