Deep learning for pollen allergy surveillance from twitter in Australia

According to Australian Institute of Health and Welfare (AIHW) [1], in 2014−15 nearly 1 in 5 Australian suffered from Pollen allergy, which amounts to 4.5 mln of citizens, predominantly working-aged adults. What is more, the expenditure on Allergic rhinitis medications doubled between 2001 and 2010, going from $107.8 mln to $226.8 mln per year, as reported by Australian pharmacies [1]. Overall allergies are increasing, but the reasons for an observed growth are not entirely clear [2, 3].

The potential of social media for public health mining has already been demonstrated in previous studies on Adverse Drug Reactions (ADRs) [4‐8], antibiotics misuse [9], influenza detection [10‐12], allergy surveillance [13‐17], and so on. Still, the automatic approaches frequently under-perform when exposed to novel/creative phrases, sarcasm, ambiguity and misspellings [6, 18, 19]. Consequently, the conventional machine learning classifiers struggle with correct identification of non-medical expressions such as ’hay fever sob’ or ’dribbling nose’, typical of social media discourse. On the other hand, the large proportion of user-generated content is of either commercial or informative nature - irrelevant for surveillance and knowledge discovery purposes. The news, warnings, products and services ads related to the condition can be published by both public as well as private accounts, limiting usability of the associated metadata. A critical challenge lies in abstracting essential information, in the context of Hay fever surveillance, from highly unstructured user-generated content to support public health monitoring from social media.

Deep learning emerged as a sub-field of machine learning and already benefited numerous Natural Language Processing (NLP) tasks [20]. The ability to learn the most salient aspects from text automatically eliminated the need for conventional classifiers dependent on manual feature-engineering. Further application of word embeddings allowed to account for syntactic and semantic regularities between the words, leading to classification performance improvement. As state-of-the-art approach, deep learning in public health mining domain is still in its infancy. Previous studies on allergies surveillance from social media conducted in the UK and US utilised either traditional machine learning classifiers such as Multinomial Naive Bayes [13, 17], or lexicon-based approaches [14‐16]. The application of deep learning for Hay fever-related user-generated content identification and knowledge discovery about the condition in Australia is yet to be explored in the literature.

Pollen allergy, commonly known as Hay Fever, significantly reduces the quality of life and affects physical, psychological and social functioning. The symptoms experienced are caused by body’s immune response to the inhaled pollen, resulting in chronic inflammation of eyes and nasal passages. Nasal congestion is often associated with sleep disturbance, resulting in daytime fatigue and somnolence. An increased irritability and self-consciousness along with a decreased level of energy and alertness are frequently observed during pollen season [21]. Moderate and severe symptoms of Hay fever considerably impair learning ability in children, while adults suffer from work absences and reduced productivity [21, 22]. According to World Allergy Organisation (WAO) [22], Hay fever is increasing in prevalence and severity, and will continue to be a concern.

Around the world, in both developed and developing countries, environments are undergoing profound changes [3]. An increased air pollution and global warming have a substantial impact on respiratory health of the population. Ziska et al. [23] has already reported that the duration of ragweed pollen season has been increasing in recent decades in North America. Any potential pattern changes, including prolonged pollen season, increased intensity of allergens or un-expected pollens detection directly affect the physical, psychological and social functioning of allergy sufferers [22]. The response to the external factors further differs among the individuals, which is particularly exacerbated in countries with high migration rates [3]. As for 2015, approx. 30% of the Australia’s Estimated Resident Population (ERP) was born overseas [24].

The ever-changing and unpredictable nature of Pollen allergies evolution necessitates the accurate and timely statistics about the state of the condition. The conventional, survey-based approaches involve a fraction of the population, and incur significant reporting delays (approx. 1 year in the case of official government reports [1]). Alternative approaches involve the number of hospital admissions and General Practitioners (GPs) reports of Hay fever instances. According to the study conducted in New South Wales - Australia [25], ’patients believe that Allergic rhinitis is the condition that should be self-managed’. Bypassing the Health Care Professionals (HCPs) and reliance on over-the-counter drugs can lead to statistics derived from services under-estimation. Also, the pharmacies supply data of oral antihistamines - the common Hay fever medicine - is used to indicate yearly start and peak of the season [1, 2]. Despite insightful, such analyses are not conducted systematically as the collection of data from drug manufacturers/pharmacy outlets across the country is required. Finally, the pollen rates assist in estimations of starting and peaking points of allergy seasons. Still, the actual condition prevalence may vary due to different responses to particular allergens among individuals.

Given the limitations of traditional approaches for allergies surveillance, the alternative sources of data increase in importance to closer reflect the state of the condition within the population. One domain that has grown by massive proportions in recent years, as well as continues to grow, is social media [6, 26]. Online platforms attract and encourage users to discuss their health issues, use of drugs, side effects and alternative treatments [6]. The updates range from generic signs of dissatisfaction (e.g. ’hay fever sucks’) to specific symptoms description (e.g. ’my head is killing me’). Also, it has been observed that individuals often prefer to share their health-related experiences with peers, rather than during clinical studies, or even physicians [27]. As a result, social media has become a source of valuable data, increasingly used for real-time detection and knowledge discovery [28].

Previous studies conducted in UK and US have already investigated the potential of Twitter for allergies surveillance. De Quincey et al. [15] observed that Twitter users are self-reporting the symptoms as well as medications, and the volume of Hay fever-related tweets strongly correlates (r=0.97, p<0.01) with incidents of Hay fever reported by Royal College of General Practitioners (RCGP) within the same year in the UK. Another correlation has been found in the work published by Cowie et al. [17], where the volume of Pollen allergy-related tweets collected in the UK over the period of 1 year resembled the pattern of pollen counts - grass pollen in particular. The study performed in the US has reported similar findings - strong correlations between (1) pollen rates and tweets reporting Hay fever symptoms (r=0.95), and (2) pollen rates and tweets reporting the use of antihistamines (r=0.93) [16]. Lee et al. [13] further observed the relationship between the weather conditions (daily maximum temperature), and number of conversations about allergies on Twitter. Additionally, the classification of actual allergy incidents and general awareness promotion was employed, along with the particular allergy types extraction. The correlations between the environmental factors and Hay fever-related tweets were also performed in the small-scale Australian study [29], where moderately strong dependencies were found for Temperature, Evaporation and Wind - all crucial factors in allergies development.

Gao et al. [30] demonstrated how deep learning approach can improve model performance for multiple information extraction tasks from unstructured cancer pathology reports compared to conventional methods. The corpus of 2505 reports was manually annotated for (1) primary site (9 labels), and (2) histological grade (4 labels) identification. The models tested were RNN, CNN, LSTM and GRU, and word embeddings were implemented for word-to-vector representation. Another study explored the effectiveness of domain-specific word embeddings on classification performance in Adverse Drug Reactions (ADRs) extraction from social media [5]. The data was collected from Twitter and DailyStrength (the online support community dedicated to health issues), followed by annotation of total of 7663 posts for presence of (1) adverse reactions, (2) beneficial effects, (3) condition suffered, and (4) other symptoms. The use of word embeddings enabled even the non-medical expressions correct identification in highly informal social media streams. The improved performance following the domain-specific embeddings development was also demonstrated in the classification of ADRs-related [12] (medical embeddings), and crisis-related tweets [31] (crisis embeddings). The former employed the bi-directional LSTM model for detection of ADRs, Drug Entities and others. The latter used CNN model for binary identification of useful versus non-useful posts during a crisis event. Similarly, CNN was successfully applied in personality identification [32], sarcasm detection [33], aspect extraction [34] or emotion recognition [35].

CNNs capture the most salient n-gram information by means of its convolution and max-pooling operations. In terms of NLP tasks, RNNs are found particularly suitable due to the ability to process variable length inputs as well as long-distance word relationships [36]. In text classification, the dependencies between the center and far-away words can be meaningful and contribute towards performance improvement [37]. The LSTMs (Long Short-Term Memory), as variants of RNNs - can leverage both short and long-distance word relationships [37]. Unlike LSTMs, GRUs (Gated Recurrent Unit) fully expose their memory content each timestep, and whenever a previously detected feature, or the memory content is considered to be important for later use, the update gate will be closed to carry the current memory content across multiple timesteps [38]. Based on empirical results, GRUs outperformed LSTMs in terms of convergence in CPU time and in terms of parameter updates and generalisation by using fixed number of parameters for all models on selected datasets [39].

The main contributions of the study can be stated as follows:

The objectives of the study are as follows:

The high-level methodological framework is presented in Fig. 1, and the particular steps are detailed in the following sub-sections.

The extraction phase inlcuded the following stages:

The full dataset of 4,148 posts (Sydney - 1,040, Melbourne - 1928), and Brisbane - 222) was annotated by two researchers, active in health informatics domain. Annotators performed the evaluation using the tweet text as well as link to the online tweet version if text was unclear, where certain commonly occurring emojis provided further context for tweets interpretation, e.g. nose or tears. The approach followed the methodological considerations for undertaking Twitter research outlined by Colditz et al. [40]. In case of potential disagreements, either the consensus was obtained or the ‘Unrelated/Ambiguous’ class was selected. The inter-rater reliability was calculated using Cohen’s kappa statistic [41], taking into account the probability of agreement by chance. The score achieved was κ = 0.78 and is considered significant [42]. The usernames have been removed from the posts given the privacy considerations.

The study conducted by Lee et al. [13] categorised the allergy-related posts into the actual incidents of the condition and general awareness promotion. Analogically, the posts were annotated into Informative and Non-Informative, as detailed in Table 2. The Informative category split was introduced to allow for (1) personal detailed reporting, and (2) personal generic reporting separation. Class 1 was further used for symptoms and/or treatments extraction, whereas combined classes 1 and 2 were used for quantitative analysis of the condition prevalence estimation. The Non-Informative category included public broadcasting (3), and unrelated content (4).

The experiments with 4 deep learning architectures were conducted due to various performances obtained on different datasets in previous studies. Pre-processing performed was minimal, and included removal of URLs, non-alphanumeric characters and lowercasing. In terms of emojis, their numerical representation was retained, following the punctuation removal. No excessive pre-processing was applied as models perform the operations on sequence of words in order they appear. Words are preserved in their original form without stemming/lemmatising due to their context-dependent representation, e.g. ’allergy’, ’allergic’, ’allergen’. Also, Sarker et al. [6] suggested that stop words can play a positive effect on classifier performance. Analogical pre-processing steps were implemented for the embeddings development.

For feature extraction, the word-to-vector representation was adopted due to its ability to effectively capture the relationships between the words, thus proving superior in text classification tasks. Additionally, the use of word embeddings naturally extends the feature set, which is particularly advantageous in the case of small to moderate datasets. The 2 word embeddings variants were implemented (1) GloVe embeddings - as default, and (2) HF embeddings - as alternative. The pre-trained Common Crawl 840B tokens GloVe embeddings were downloaded from the website². Both 50 dimensions (min) and 300 dimensions (max) options were tested. The HF embeddings were generated using 10 iterations and vector dimension of 50, given the moderate training data size. Previous study [4] reported improved classification performance with 50 dimensions while training domain-specific embeddings.

In terms of the parameters, the mini-batch size was set to default 32, the most popular non-linear activation function ReLU was selected, the number of recurrent units was set to standard 128, and the Nadam optimiser was used. The models were trained up to 50 epochs and implemented with open source neural network library Keras³.

Finally, the standard evaluation metrics were adopted, such as Accuracy, Precision (exactness) and Recall (completeness). The 5-fold cross-validation was followed, with 80:20 training and testing split as in [43]. The Confusion Matrices were further produced to examine in-detail the performances obtained for the particular classes.

As for the patterns investigation, the weather factors were superimposed on the tweet volume charts over the period of 6 months (2018/06/01−2018/12/31). The weekly averages of the number of Informative posts (class 1 + 2) were taken into account for Sydney, Melbourne, and Brisbane. The approach followed previous study conducted by Gesualdo et al. [16], where the weekly averages of tweets were used to avoid daily fluctuations for correlations with pollen rates and antihistamine prescriptions. The environmental data was obtained from Bureau of Meteorology⁴ (BOM) - Australia’s official weather forecast and weather radar. The following variables were extracted: Min Temp [ ^∘ C], Max Temp [ ^∘ C], Ave Temp [ ^∘ C], Sunshine [hrs], Rainfall [mm], Evaporation [mm], Relative Humidity [%], Max Wind [km\h], Ave Wind [km\h] and Pressure (hPa). Analogically, the weekly averages were considered.

In the case of gaps in data collection (Fig. 2), the compensation approach was adopted, i.e. given 1 day-worth of data missing within the week, the average of the remaining 6 days was calculated and considered as the 7th day tweet volume. The weekly average was then estimated based on the complete 7-days record.