A systematic review of methods for studying consumer health YouTube videos, with implications for systematic reviews

The searches

All reviews searched directly on the YouTube site, rather than through a third party interface such as Google advanced search. Ten of the 12 reviews reported the date of the search (Ache & Wallace, 2008; Pandey et al., 2010; Fat et al., 2012; Backinger et al., 2011; Keelan et al., 2012; Pant et al., 2012; Steinberg et al., 2010; Knösel & Jung, 2011). Most included several terms in the search, and these were presumably linked with “OR”. Most did not address the sort order, so presumably used the default values. Currently, YouTube search results are sorted by relevance as the default. One review sampled the top ranked items from searches sorted by relevance and number of views, using the default of searching all of YouTube, and then again searching only those classified by the person who posted the video as “educational” (4 samples in all) (Knösel, Jung & Bleckmann, 2011). Only one review explained how search terms were selected – by using Google Trends to determine which topical terms were most searched (Backinger et al., 2011). In the updated set, two additional studies used empirically derived search terms – most common brands and common search terms from Google Insights (Richardson & Vallone, 2012; Bromberg, Augustson & Backinger, 2012).

Several reviews attempted to make the search realistic, that is, searching as consumers might search (Fat et al., 2012; Pant et al., 2012; Knösel & Jung, 2011; Backinger et al., 2011); but all seemed to have worked from the search list rather than using a snowball technique (Grant, 2004). Snowballing is a technique used in sampling for qualitative studies - cases with connection to other cases are identified and selected (Giacomini & Cook, 2000) and in information retrieval - references to references are considered for relevance (Greenhalgh & Peacock, 2005). It is a useful adjunct when identifying all relevant candidates through a search engine is difficult for whatever reason (Horsley, Dingwall & Sampson, 2011).

However, three of the searches from the updated search did describe snowballing, as follows: “As clips were viewed, additional suggestions were offered by the site and these in turn led to further suggestions” (Stephen & Cumming, 2012). “For each of the top 10 videos, the top three related videos (ranked by YouTube) were also coded” (Thomas, Mackay & Salsbury, 2012). Finally “The search was supplemented by also reviewing the list of featured videos that accompany search results” (Kerber et al., 2012).

Only one review imposed filters on the search (in that case, that the video had been uploaded in the past three months) (Pandey et al., 2010).

The inclusion criteria

Eight of the 12 reviews stated that only English language videos were included (Backinger et al., 2011; Keelan et al., 2012; Singh, Singh & Singh, 2012; Lo, Esser & Gordon, 2010; Pant et al., 2012; Steinberg et al., 2010; Pandey et al., 2010; Tian, 2010). None of the other 4 reported that they were language inclusive. Nine reported that they excluded “off topic” videos (Keelan et al., 2012; Singh, Singh & Singh, 2012; Pant et al., 2012; Steinberg et al., 2010; Pandey et al., 2010; Tian, 2010; Fat et al., 2012; Knösel, Jung & Bleckmann, 2011; Ache & Wallace, 2008), but few gave clear criteria defining what was “on topic”. Eight reported that they removed duplicates (Singh, Singh & Singh, 2012; Pant et al., 2012; Steinberg et al., 2010; Pandey et al., 2010; Tian, 2010; Fat et al., 2012; Ache & Wallace, 2008; Backinger et al., 2011). Details of the treatment of duplicates were sparse; for instance, none stated how they selected which version to keep, or whether they aggregated view counts across all versions - although two stated that if videos had multiple parts, only one was kept, and both of these stated that they averaged views across all parts (Singh, Singh & Singh, 2012; Pant et al., 2012). Two reviews stated that the video must have sound to be eligible (Pant et al., 2012; Steinberg et al., 2010). One included only videos under 10 min in length (Steinberg et al., 2010) and one excluded videos blocked by their institutions’ internet filters (Backinger et al., 2011).

Reported characteristics of the review

Descriptive characteristics: All reviews reported on some characteristics of the videos. Elements most commonly reported were: number of views (Pant et al., 2012; Steinberg et al., 2010; Pandey et al., 2010; Keelan et al., 2012; Ache & Wallace, 2008; Backinger et al., 2011; Lo, Esser & Gordon, 2010; Knösel, Jung & Bleckmann, 2011; Fat et al., 2012; Singh, Singh & Singh, 2012; Tian, 2010; Knösel & Jung, 2011), length in minutes (Pant et al., 2012; Lo, Esser & Gordon, 2010; Steinberg et al., 2010; Knösel, Jung & Bleckmann, 2011; Singh, Singh & Singh, 2012; Pandey et al., 2010; Knösel & Jung, 2011; Keelan et al., 2012), and date posted (Pant et al., 2012; Lo, Esser & Gordon, 2010; Tian, 2010; Singh, Singh & Singh, 2012; Pandey et al., 2010). While most reported median, or mean number of views, often with some measure of dispersion, one reported the concentration of views – five videos accounted for 85% of total views (Kerber et al., 2012).

Other characteristics reported included number of “likes” (Pant et al., 2012; Singh, Singh & Singh, 2012; Fat et al., 2012), rating score (Steinberg et al., 2010; Tian, 2010; Keelan et al., 2012), times rated by viewers (Lo, Esser & Gordon, 2010; Tian, 2010), intended audience (Pant et al., 2012; Steinberg et al., 2010), amateur/pro, based on production quality (Fat et al., 2012; Lo, Esser & Gordon, 2010), type if non-standard (i.e., song, animation, advertisement) (Pant et al., 2012), country of origin or address of author (Tian, 2010). Importantly, one from the original set (Fat et al., 2012) and three from the update set, harvested self-reported demographics of viewers (Mukewar et al., 2012; Stephen & Cumming, 2012; Richardson & Vallone, 2012).

Several classified the videos according to the creating source (Pant et al., 2012; Ache & Wallace, 2008; Singh, Singh & Singh, 2012; Pandey et al., 2010; Knösel & Jung, 2011), each used its own typology, but common elements were: personal experience/patient, news reports, professional associations, NGOs such as WHO or Red Cross, pharmaceutical companies and medical institutions. Three reviews from the update set addressed the issue of covert advertising – two for a tobacco product (Richardson & Vallone, 2012; Bromberg, Augustson & Backinger, 2012), the other discussed the notion of paid testimonials appearing as consumer-posted videos (Mukewar et al., 2012).

Sample size

The number of videos assessed ranged from 10 to 622, with a mean of 145 and median 112. Some screened the entire search results (maximum 1634 videos). More common was an approach of taking a fixed sample size and screening this set, retaining those eligible after duplicates, off topic and other ineligible material was removed. Two reviews used a fixed sample size (Knösel, Jung & Bleckmann, 2011; Lo, Esser & Gordon, 2010). Several set a fixed sample size to screen (Fat et al., 2012; Knösel & Jung, 2011; Backinger et al., 2011). No reviews reported a formal sample size calculation.

Review methods

Two reviews reported saving all eligible videos offline (Ache & Wallace, 2008; Pandey et al., 2010). Some reported viewing, screening or assessing online, at the time of discovery. Two (both by Knosel) described the reviewing conditions in some detail; videos were viewed at the same time and under the same conditions by two assessors (Knösel, Jung & Bleckmann, 2011; Knösel & Jung, 2011). Knosel also described opportunities for the reviewers to communicate – required in one review (Knösel, Jung & Bleckmann, 2011) and prevented in the other (Knösel & Jung, 2011).

Eight of the 12 reviews described the reviewers (Backinger et al., 2011; Lo, Esser & Gordon, 2010; Keelan et al., 2012; Steinberg et al., 2010; Knösel & Jung, 2011; Singh, Singh & Singh, 2012; Knösel, Jung & Bleckmann, 2011; Fat et al., 2012). Most were health care professionals, however, one used lay raters – one potential patient (a youth) and one parent – to gain their perspective (Knösel & Jung, 2011).

Ten of 12 reported on the number of reviewers – 8 reported using 2 reviewers for each video (Backinger et al., 2011; Tian, 2010; Singh, Singh & Singh, 2012; Knösel, Jung & Bleckmann, 2011; Fat et al., 2012; Pandey et al., 2010; Keelan et al., 2012; Steinberg et al., 2010), one reported 3 (professional, parent, youth) (Knösel & Jung, 2011) and one implied multiple reviewers without specifying the number (Ache & Wallace, 2008). No review reported having only one reviewer make assessment of content. Four of the 10 with multiple reviewers reported using a third reviewer as arbitrator (Keelan et al., 2012; Singh, Singh & Singh, 2012; Steinberg et al., 2010; Backinger et al., 2011). Seven of the 10 computed kappa on reviewer agreement. It was not always made clear which rating was used if conflicts occurred – i.e., neither arbitration nor consensus was described (Fat et al., 2012; Keelan et al., 2012; Pandey et al., 2010; Knösel, Jung & Bleckmann, 2011; Steinberg et al., 2010; Tian, 2010; Backinger et al., 2011).

Only two reviews described a training or calibration exercise prior to undertaking assessments. Backinger described 4 h of training in which definitions were discussed, and 5 practice videos coded (Backinger et al., 2011). Tian described pre-testing their code book, using 20 videos and 40 text comments (Tian, 2010).

Blinding was used in two reviews. Pandey reported that reviewers were blind to the purpose of the study (Pandey et al., 2010). Lim Fat reported, “The individual who rated the comments was blinded to the classification of the video as being a positive, negative or neutral portrayal of Tourette syndrome. Likewise, the raters for classification of the videos were blinded to the classification of the comments” (Fat et al., 2012).

Ethical oversight

None of the original cohort of twelve reviews stated that IRB approval was obtained. One review explicitly stated that IRB approval was deemed unnecessary due to the nature of the study (Fat et al., 2012). In the 11 additional studies reviewed, four made explicit statements about IRB approval, one sought approval (Ehrlich, Richard & Woodward, 2012), three stated they were exempt (Richardson & Vallone, 2012; Mukewar et al., 2012; Kerber et al., 2012).

Follow-up

Three videos examined a cohort of reviews at two or more points in time looking changes in the number of hits. One review from the original set reported a follow-up at one and six months (Lo, Esser & Gordon, 2010). Two more from the update set included follow-up, one after 7 months (Mukewar et al., 2012) and one after 1 month and 7 months (Stephen & Cumming, 2012).

Outcomes of interest

The characteristics of the videos that were evaluated varied depending on the study objectives, but videos were commonly assessed as providing true or reliable information or being positive or negative toward the health issue addressed. One review from the original set (Singh, Singh & Singh, 2012) used a validated scale as part of the assessment - DISCERN: an instrument for judging the quality of written consumer health information on treatment choices (Charnock et al., 1999). One review identified from the update search (Mukewar et al., 2012) used two scales adapted from Inflammatory Bowel Disease patient education web sites; a detailed scale specific to IBD, and a 5-point global quality score.

Two looked at knowledge translation – one of these examined whether videos posted after a change in guidelines for cardiopulmonary resuscitation employed the new or old standard (Tourinho et al., 2012). Another investigated the integrity with which parents and carers implement the Picture Exchange Communication System (PECS), in a naturalistic setting (Jurgens, Anderson & Moore, 2012).

One review assessed the findability of the videos – having identified 33 videos that portrayed complete Epley maneuvers for benign paroxysmal positional vertigo, they searched again using very general terms – dizzy, dizziness, vertigo, positional dizziness, positional vertigo, dizziness treatment, and vertigo treatment. The investigators then determined where or if one of the videos depicting the Epley maneuver appeared in the list of relevance-ranked results for the less specific terms (Kerber et al., 2012).

Discussion

These reviews are recent, and for the most part, clearly reported. There are examples of excellent reporting in most facets of the review – study question, inclusion and exclusion criteria, search strategy, screening and data extraction methods. Few reviews, however, reported all elements well. Improved reporting would increase transparency and allow the reader to better assess the risk of bias in the study design (Tricco, Tetzlaff & Moher, 2010). The study design should be strong and reproducible, with methods in line with those of a well-conducted systematic review (Moher et al., 2009). Through this manuscript, we aim to describe the array of methods and data available to those planning to undertake this type of work, recognizing that, depending on the data examined and the objectives of the review, methods will vary.

While it is premature to put forth reporting guidelines – defined as a minimum set of elements that should be reported to enable the reader to understand the conduct of the study, assess the risk of bias and generalizability of results, the PRISMA checklist (Moher et al., 2009) and accompanying elaboration and explanation (Liberati et al., 2009) generalizes to many aspects of video reviews. The elements that have no real parallel in systematic reviews of research studies warrant the most consideration and some of these are elaborated in Table 3.

A number of factors make video searches less stable, and thus less replicable, than the sorts of database searches used in systematic reviews where results either match the search criteria or do not and results are sorted by date. In any relevance-ranked search results, the order will change as new entries are added, existing ones removed, or the proprietary ranking algorithm is changed. Thus, a systematic approach that will accomplish the goals of the review (whether they are exhaustive identification of eligible videos or a sample of fixed size that represents the videos that the target audience is most likely to find) is needed, and should be fully described.

There was one aspect of our upcoming review of immunization videos that was not informed by this survey of published studies - a discontinuation rule to stop screening when few additional studies were being found. Our initial search yielded 6,000 videos. Unlike the searches of bibliographic databases used for study identification in systematic reviews, this search result was ranked according to relevance. Spot checks showed that most of the lower ranked videos were irrelevant. Given the size of the list, screening the entire list in one sitting was not feasible. We did not know of a way to “download” the list, and had no assurance that the list would remain unchanged on subsequent screening days. Thus, a protocol was needed to discontinue screening when further screening was unlikely to yield additional eligible videos.

Discontinuation rules would allow one to manage relevance-ranked search results, and are essential when screening web search results that are often large, cannot be easily captured as a whole, and are not static from day to day. Given the absence of empirical guidance, we devised a pragmatic rule: screen until twenty consecutive ineligible videos are reviewed, then assess a margin of 50 more. Depending on the number of eligible videos found in the margin, a decision could be taken to stop screening or continue. With this discontinuation rule, one needs to accept that some additional eligible videos might have been found had the entire retrieval been screened, however, the likelihood of missing a large number is low given the relevance ranking.

Examples of discontinuation rules can be found in several health care fields. Computerized adaptive testing often use validated stopping rules to discontinue the test when it becomes statistically unlikely that administering additional items would improve the accuracy of the assessment (Babcock & Weiss, 2009). Clinical trials that have planned interim analyses may have pre-specified stopping rules designed “to ensure trials yield interpretable results while preserving the safety of study participants” (Cannistra, 2004). As search engine ranking algorithms improve, there is increasing opportunity for systematic reviewers to use sources that offer relevance ranking, such as Google Scholar or PubMed Related Citations. Experimental research has demonstrated that ranking algorithms can successfully place eligible records high in a search result (Cohen et al., 2006; Sampson et al., 2006). Yet, we were unable to identify any practical guidance on stopping rules for screening in the systematic review context, nor any explicit reports of when or if screening was stopped for Internet-based searches in systematic reviews. This is an area requiring more complete reporting on the part of systematic reviewers, as well as a useful area for further research.

It should be noted that factors such as screening order, the use of snowballing or other techniques to mimic consumer searching behaviour, and discontinuation rules are relevant only when there is a tacit acceptance that not all potentially relevant videos will be identified.

While we have focused our efforts on informing the conduct of systematic reviews, video producers hoping to reach consumers may wish to consider several factors. As the difference in number of views varies 1000 fold, a clear marketing plan is needed for any production effort to be worthwhile. Our review suggests that videos styled as home videos appeal to a broader audience than dyadic videos. As much as reviewers use empirically defined search terms, producers will want to select titles and keywords that are likely to match what consumers type in to the search bar. Producers will need to consider the factors that rank a video high in the related list as well as those that will make it appear in the search results. As the ranking algorithms for both search engine ranking and related sidebar ranking are proprietary and subject to change, video producers will want to seek up-to-date guidance on “Search Engine Optimization” for YouTube, or any other video channel they intend to use.

Limitations of this systematic review include the fact that there may be additional informative reviews that we did not identify and include. We only searched one traditional bibliographic database, and did not include social media such as blog postings. Also, reviews of consumer health videos are relatively new. YouTube was created in 2005, and given the time needed to conduct and publish reviews; there may be a large number in preparation. Certainly, as many appeared in the course of this project (April to November 2012) as were published prior to its start, suggesting that there may be innovations that we have not yet captured.