An updated review of mechanistic potentials of melatonin against cancer: pivotal roles in angiogenesis, apoptosis, autophagy, endoplasmic reticulum stress and oxidative stress

Angiogenesis is a crucial event implicated in the progression of tumor as well as its metastasis [38]. Hypoxia in the central areas of solid tumor is a leading cause of angiogenesis via activation of angiogenic mediators [38, 39]. Vascular endothelial growth factor (VEGF), the specific mitogen of endothelial cells and the most active pro-angiogenic agent, is a powerful angiogenesis enhancer which increases vascular permeability. Numerous data suggest that, in tumor development, anti-VEGF therapy has important roles in the suppression of tumor cell growth, leading to a considerable amelioration in progression-free survival [40]. Hypoxia-inducible factor-1 (HIF-1) is another key factor in angiogenesis, which modulates hypoxia-activated genes transcription and consists of HIF-1α and HIF-1β heterodimer. The α subunit of HIF-1 is stabilized under hypoxia and degraded under normoxic situations, however, HIF-1β is expressed constitutively [41].

Melatonin has been shown to have regulatory role in angiogenesis process [42]. In other words, melatonin possesses various impacts on neovascularization under diverse pathological and physiological situations. In skin lesions, gastric ulcers, and some physiologic events, melatonin promotes angiogenesis, while in a hypoxic environment, in age-related ocular diseases, and in tumors melatonin suppresses neovascularization in tissues [43].

Melatonin exerts its antitumor potentials via inhibiting HIF-1-induced angiogenesis [44]. Furthermore, melatonin inhibits the accumulation of HIF-1α through suppressing the formation of ROS and the sphingosine kinase 1 (SPHK1) pathway in prostate cancer cells under hypoxic conditions [45]. Melatonin plays an important role in the paracrine interaction between proximal endothelial cells and malignant epithelial cells by a downmodulatory effect on the expression of VEGF in breast tumor cells, which reduces VEGF levels around endothelial cells [46].

Of note, anti-angiogenic potential of melatonin is a key factor resulting in the inhibition of proliferation of cancer cells, as demonstrated in various investigations. For instance, melatonin attenuates proliferation of prostate cancer cells triggered by epidermal growth factor [47]. Melatonin also hampers vasculogenic mimicry of oral cancer cells via inhibition of ROS-activated Akt and ERKs signaling pathway implicating the HIF-α pathway [48]. Melatonin up-regulates TGF-β1 expression in tumor tissues during the inhibition of gastric cancer tumor growth process [49]. Furthermore, apoptotic and anti-proliferative effects of melatonin on breast cancer cells are mediated by the simultaneous activation of the Apaf-1/caspase-dependent apoptotic pathway and the inhibition of PI3K/Akt, p300/NF-κB, and COX-2/PGE2 signaling pathways [32].

Endothelin-1 is a peptide acting as a survival factor in colon cancer, promoting angiogenesis and mediating cell proliferation. Melatonin suppresses endothelin-1 mRNA expression. Also, melatonin blocks the activity of endothelin-1 promoter modulated by NF-κβ and FoxO1 [50]. Melatonin represses ROCK-1, VEGF and HIF-1α genes expressions in oral cancer [51]. Melatonin alters the expression of inflammatory and angiogenic proteins in both co-culture and monoculture of cancer cells and cancer-associated fibroblasts [52]. Melatonin suppresses tumor angiogenesis and the growth of gastric cancer cells in tumor-bearing nude mice. Moreover, melatonin decreases the expression of VEGF and HIF-1α at translational and transcriptional levels within gastric cancer cells during tumorigenesis [53]. Reduced serum levels of VEGF have been reported in cancer subjects treated with melatonin [54]. Vimalraj et al. [55] showed that melatonin upregulates miR-424-5p expression in osteosarcoma cells suppressing VEGFA. Furthermore, it inhibits tumor angiogenesis, regulating surrounding endothelial cells migration and proliferation, and angiogenic growth factors and the morphology of blood vessels E (Table 1).

In normal cellular condition, there is a balance between the production of oxidants, so called reactive oxygen species (ROS), and their neutralizing compounds, named antioxidants. The state of excess ROS, in which the oxidant content of the cells dominates the neutralizing capacity of antioxidants, is defined as oxidative stress [56, 57]. Sustained oxidative stress increases the risk of cancer development either through inducing mutagenesis or by promoting the expression of proto-oncogenes such as cyclin D1. It also plays a signaling role in the activation of several genes involved in the cancer progression including the mitogen-activated protein kinase (MAPK), extracellular signal-regulated kinase (ERK) and JUN N-terminal kinase (JNK) [58, 59].

Melatonin role as a natural ally against oxidative stress has been revealed in many in vitro and in vivo studies. Detoxification of oxidants by melatonin is triggered by several direct or indirect mechanisms. In direct scenario, melatonin neutralizes the oxidants by its nonreceptor‐mediated free radical scavenging capacity. As indirect scenario, melatonin reduces the oxidative content through several mechanisms such as activating anti-oxidative enzymes and suppressing pro‐oxidative enzymes. It also stabilizes the mitochondrial inner membrane, thereby maintaining mitochondrial integrity leading to a reduced electron leakage and ROS generation [60, 61].

The inducing role of oxidative stress in cancer progression and preventive role of melatonin in the production and function of oxidants indicated a possible oncostatic property for melatonin [62]. Subsequently, it was revealed that melatonin reduces the oxidative damage to cellular components under conditions where toxic oxygen derivatives are acknowledged to be released [63, 64]. Moreover, in vitro studies demonstrated that melatonin treatment reduces the amount of oxidative contents in a variety of cancer cells, which was further supported by in vivo studies (Table 2).

Endoplasmic reticulum (ER) is an entry site for secretory proteins and most integral membrane proteins for proper folding and covalent modifications to assemble into a functional complex. In addition to the processing of proteins, ER is involved in various cellular functions including lipid synthesis, fatty acid turnover, detoxification, Ca²⁺ homeostasis. The ER network extends into all cell compartments to sense intrinsic and extrinsic perturbations and integrate the stress signals for maintenance of cellular homeostasis to preserve proper cellular and organismal function [65, 66]. However, the ER function can be impacted and disturbed by a multitude of exogenous and endogenous factors, leading to the accumulation of mis/unfolded proteins in the ER. This causes the imbalance between the client proteins load in the lumen of ER and the folding capacity of this organelle leading to the failure of the ER to cope with unusual high protein folding load, which is termed 'ER stress' [67]. To restore protein homeostasis, an adaptive signal transduction pathway called the unfolded protein response (UPR) is activated to induce compensatory responses to stressors for recovering normal ER function [68]. Signaling proteins sensing UPR include inositol-requiring protein-1α (IRE1α), activating transcription factor 6α (ATF6α) and protein kinase RNA (PKR)-like ER kinase (PERK). In nonstressed cells, UPR stress sensors are maintained in an inactive state through direct binding to the ER chaperone proteins, Bip (78-kDa glucose regulated protein, GRP78). Upon ER stress, aggregation of misfolded proteins leads to the dissociation of UPR sensors from Bip, which causes activation of UPR signals [69]. Although activation of UPR signaling pathways is a cellular strategy to increase survival, this pathway will instead activate cell death signaling pathways when the intensity or duration of cellular stress increases. Therefore, certain anti-cancer patterns may activate ER stress/UPR pathway to induce apoptosis in cancer cells [70].

Melatonin induces mitochondria-mediated apoptosis in colorectal cancer cells through reducing the expression of PrP^C and PINK1 resulting in the enhancement of superoxide production and induction of ER stress [71]. Melatonin also ameliorates ER-stress mediated insulin resistance. ER stress induces autophagy in pancreatic β cells, which this pathway plays an important role in insulin production and secretion. In the glucose analog 2-DG-treated rat insulinoma INS-1E cells, melatonin reduces insulin production via ER stress-induced autophagy [72]. Combination of melatonin with ER stress inducer tunicamycin increases the sensitivity of cancer cells to apoptosis through inhibiting the expression of COX-2 and increasing the Bax/Bcl-2 ratio and CHOP levels [73]. Selective inhibition of ATF-6 by melatonin results in the suppression of COX-2 production and enhancement of cancer cells to ER-stress induced apoptosis [74].

Melatonin increases apoptosis through enhancing caspase-3, -8 and -9 activities, Bax/Bcl-2 ratio, PARP cleavage and cytochrome c, p53 and Fas-L proteins concentrations in hepatocellular carcinoma, which this effect is mediated by the elevation of ER stress characterized by up-regulation of ATF6, CHOP and Bip [75]. Furthermore, melatonin increases the sensitivity of hepatocellular carcinoma cells to sorafenib through targeting the PERK-ATF4-Beclin1 pathway [76]. The same results have been reported in gastric cancer; melatonin inhibits cell proliferation through inducing activation of the IRE/JNK/Beclin1 signaling [77].

Melatonin in combination with the ER stressor thapsigargin increases the expression level of nuclear mammalian RNA-binding protein (HuD) resulting in the reduction of intracellular biosynthesis of insulin. Suppression of AKT/PI3K pathway and induction of nuclear mTOR (Ser2481, Ser2448) expressions by melatonin sensitizes rat insulinoma INS-1E cells to insulin through increasing the expression of insulin receptor substrate [78]. In contrast with these reports, melatonin has been reported to inhibit tunicamycin-induced ER stress in human hepatocellular carcinoma cells and increase the response of these cells to cytotoxic effects of doxorubicin; this is accompanied by inhibition of the PI3K/AKT pathway, elevation of CHOP and reduction of Survivin [79]. These evidences suggest that melatonin could improve the toxic effect of anti-cancer agents on cancer cells through regulating ER stress in cells (Table 3).

Autophagy is a complicated process maintaining intracellular homeostasis by eliminating degraded proteins and organelles during cellular stress. Autophagy is principally considered as a pro-survival process, but, excessive or inappropriate autophagy contributes to the cell death, a process known as autophagic cell death or type II programmed cell death [80].

Autophagy is a complicated process, which consists of five sequential steps, including: (a) initiation complex formation and double-membrane phagophore (nucleation) maturation; (b) membrane elongation and autophagosome formation sequestering cargo; (c) fusion with lysosome; (d) inner membrane disruption leading to degradation of cargo by hydrolases; and (e) macromolecular component utilization [81]. These steps of the autophagy pathway are regulated by more than 35 autophagy related genes (ATGs) and proteins most of which function in complexes. The initiation phase is regulated by Unc-51-like kinase1 (mammalian homologues of Atg1, ULK1)–Atg13–Atg101–FIP200 (mammalian homologues of Atg17) protein complex. Unc-51-like kinase1 phosphorylates and activates Beclin-1 (mammalian homologue of Atg6). Beclin-1 is a part of multiprotein-complex, class III PI3-kinase Vps34–p150 (mammalian homolog of Vps15)–Atg14-like protein (Atg14L)–Beclin-1, promoting nucleation [81‐83]. The elongation phase is regulated by two ubiquitin-like conjugation systems, Atg12 and LC3 (mammalian homologue of Atg8). In the first conjugation system, Atg12 is activated by Atg7 (E1-like enzyme), transferred to Atg10 (E2-like enzyme) and conjugated to Atg5. The Atg12–Atg5 conjugates further couples with Atg16 (Atg16L in mammals) to form the E3-like complex. In the LC3 conjugation system, LC3 is cleaved by a cysteine protease, Atg4, forming LC3-I. Thereafter, LC3-I is activated by Atg7 (E1-like enzyme), transferred to Atg3 (E1-like enzyme) and conjugated to phosphatidylethanolamine (PE) to form LC3-II; this process is facilitated by the E3-like complex. This lipidated form of LC3, LC3-II, is recruited to the autophagosome membrane. Finally, the Atg9 dependent pathway promotes autophagosome membrane expansion [81‐83].

Cargo sequestration can be selective or non-selective; the selectivity is based on autophagy receptors such as P62/SQSTM1, NBR1, NDP52, NIX/BNIP3L, BNIP3 and FUNDC1 [82]. Fusion of autophagosome with lysosome is the next step. The inner vesicle is degraded by lysosomal hydrolases, including cathepsin B, D (a homolog of proteinase A), and L. The degradation products are released to the cytosol and used in different anabolic pathways [84]. ER stress-induced activation of UPR pathways promotes induction of autophagy [85]. Activated PERK/ATF4 pathway up-regulates the expression of ATG genes including ATG5, ATG7, and ATG10 [86]. The conversion of LC3-I conversion to LC3-II is also induced by PERK pathway [87]. Activation of IRE1α pathway induces the expression of Beclin1 and the phosphorylation of Bcl-2 by JNK, which subsequently results in the Bcl-2–Beclin 1 dissociation [88‐90]. The release of Ca²⁺ from ER to cytosol triggers autophagy pathway through activating several mechanisms including (I) inhibition of mTOR by Ca²⁺/calmodulin dependent kinase kinase-β-mediated activation of AMP-activated protein kinase (AMPK) [91], and (II) dissociation of Bcl-2–Beclin 1 by inducing death-associated protein kinase (DAPK) 3-mediated Beclin 1 phosphorylation [92].

Melatonin has a modulatory effect on autophagy in various cell types and different conditions. Melatonin indirectly modulates autophagy through affecting oxidative stress, ER stress and inflammation [69]. Melatonin enhances the effectiveness of cisplatin and radiotherapy in head and neck squamous cell carcinoma, which this effect is mediated by the excessive activation of mitochondria leading to the over-production of ROS and subsequent induction of autophagy and apoptosis [93]. Melatonin also increases cytotoxic effects of rapamycin in cancer cells. Combination of rapamycin and melatonin suppresses AKT/mTOR pathway activation, which this effect leads to the enhancement of mitochondrial function and ROS production resulting in the induction of apoptosis and mitophagy [94]. Melatonin induces autophagy in clear cell renal cell carcinoma through activating transcriptional coactivator peroxisome proliferator-activated receptor gamma coactivator 1A (PGC1A) and uncoupling protein 1 (UCP1); this is associated with the elimination of lipid deposits without generating ATP, which subsequently leads to the tumor size reduction [95].

Melatonin reduces the viability liver cancer cells through transient induction autophagy by up-regulating JNK phosphorylation. However, ATG5 silencing sensitizes cancer cells to melatonin-induced apoptosis. This suggests that modulation of autophagy by melatonin has dual effect on cell death [96]. Similarly, disruption of autophagy sensitizes glioblastoma cells and tongue squamous cell carcinoma to melatonin-induced apoptosis [97]. Melatonin-induced autophagy is suggested to be mediated by activation of melatonin membrane receptor in tongue squamous cell carcinoma and suppression of melatonin membrane receptor-dependent autophagy may be strategy for treatment of tongue squamous cell carcinoma [98].

Several studies indicate that melatonin may induce apoptosis in cancer cells through inhibiting autophagy pathway. Melatonin down-regulates Beclin-1 and p62 expressions and LC3B-II/LC3B-I ratio in colitis-associated colon carcinogenesis in mice; this effect is associated with the increased level of Nrf2 and its downstream antioxidant enzymes including NAD(P)H:quinone oxidoreductase (NQO-1) and heme oxygenase-1 (HO-1). These suggest that the ameliorative effect of melatonin on inflammation and oxidative stress results in the reduction of autophagy [99]. Induction of ER stress is associated with the activation of autophagy in sorafenib-treated hepatocellular carcinoma cells, which this contributes to the resistance of cancer cells to apoptosis. Combination of melatonin with sorafenib inhibits ER stress-related autophagy through suppressing the PERK-ATF4-Beclin1 pathway leading to the sensitivity of hepatocellular carcinoma cells to sorafenib [76]. Co-stimulation of cancer cells with cisplatin and melatonin induce apoptosis in HeLa cells, which this effect is accompanied by inactivating mitophagy via blockade of JNK/Parkin pathway [100]. In contrast with this report, melatonin has been found to reversed the effects of cisplatin in HepG2 cells through suppression of mTOR and DNA excision repair cross complementary 1 (ERCC1) proteins expressions and up-regulation of Beclin-1 and LC3II expressions [101]. Taken together, different effects of melatonin on autophagy may be related to type of cancer cells, the stage of cancer and dose of melatonin (Table 4).

The balance between cell proliferation and death in tissues is maintained by apoptosis, a classical form of programmed cell death. Apoptosis is associated with the disassembly of apoptotic cells into membrane-enclosed vesicles, which are removed by macrophages without inducing inflammatory responses. Apoptosis is mediated by two principle signaling pathways, including extrinsic and intrinsic pathways [102]. The extrinsic apoptotic signaling pathways, defined as death receptor pathways, are initiated by the interaction of transmembrane death receptors (Fas, TNFR1, DR4 and DR5) with extracellular ligands (FasL, TNFα, TRAIL, and TNFSF10) resulting in the activation of adaptor proteins such as Fas-associated death domain (FADD). Activated FADD recruits initiator caspases (caspase 8 and caspase 10) to form the death-inducing signal complex (DISC). Formation of DISC leads to the proteolytic activation of caspase 8, which is the main initiator caspase of the extrinsic apoptotic signaling pathway. Caspase 8 activates executioner caspases (caspase 3, caspase 6, and caspase 7) and cleaves Bid, a BH3-only domain member of the B cell lymphoma-2 (Bcl-2) family. Truncated Bid (tBid) translocates to mitochondria and activates other proapoptotic Bcl-2 family members including Bak or Bax [102, 103].

The intrinsic apoptosis pathway, defined as mitochondrial-mediated apoptotic pathway, is activated by exogenous and endogenous stimuli such as DNA damage, oxidative stress, chemotherapy and radiotherapy. This apoptosis pathway is mediated by insertion of pro-apoptotic Bcl-2 family members (Bax/Bak) into mitochondrial membrane leading to the mitochondrial outer membrane permeabilization and release of pro-apoptotic factors such as cytochrome c, Smac/DIABLO, the nuclease EndoG, the oxidoreductase AIF, and the protease HtrA2/Omi [104]. Therefore, activation of pro-apoptotic Bcl-2 family members (Bax/Bak) is essential for cancer therapy. In contrast, elevation of anti-apoptotic Bcl-2 family proteins inhibits apoptosis in cancer cells through heterodimerization with Bax/Bak preventing the release of pro-apoptotic factors from mitochondria; this could result in the resistance of cancer cells to immune-surveillance [105, 106]. Once in the cytosol, cytochrome c combines with Apaf–1 and procaspase-9 to drive the assembly of the apoptosome; this molecular platform activates caspase 9, which this is followed by the activation of caspase-3 cascade of apoptosis [107]. Smac/DIABLO and HtrA2/Omi induce apoptosis through degrading inhibitor of apoptosis protein (IAP) family, neutralizing the inhibitory effect of IAPs on caspases [108]. The nuclease EndoG and the oxidoreductase AIF translocate to the nucleus, where they trigger internucleosomal DNA fragmentation independently of caspases [109].

As mentioned earlier, UPR signaling may promote the apoptotic pathways. Upon ER stress, apoptosis signal-regulating kinase 1 (ASK1) is recruited by IRE1α-TNF receptor-associated factor 2 (TRAF2) complex, causing activation of ASK1 and the downstream JNK pathway. Activation of JNK results in the phosphorylation of Bcl-2 and Bax; phosphorylation of Bcl-2 family suppresses antiapoptotic activity of Bcl-2, while induces mitochondrial translocation of Bax and activation of apoptosis pathway. Activated JNK also activates C/EBP homologous protein (CHOP), a stress-induced transcription factor inducing the expression of pro-apoptotic Bcl-2 family members. Furthermore, IRE1α-TRAF2 complex triggers the activation of caspase-12, which this caspase translocates from the ER to the cytosol, where it activates caspase-9, independent from the apoptosome pathway [110]. Furthermore, Activated PERK phosphorylates eIF2α, promoting the expression of activating transcriptional factor 4 (ATF4); ATF4 translocates to the nucleus where it induces CHOP expression [111].

Melatonin is reported to restrict tumor growth and cancer cell proliferation through inducing apoptosis in cancer cells (Table 5). As a powerful antioxidant melatonin inhibits ROS-induced activation of extracellular-regulated protein kinases (ERKs) and Akt pathways which are involved in the cancer cell survivor; inactivation of ROS-dependent Akt signaling contributes to the down-regulation of cyclin D1, PCNA, and Bcl-2 and up-regulation of Bax in cancer cells [48]. Inhibition of MDM2 expression is a mechanism by which melatonin induces apoptosis through upregulating the activity of caspase-3 and -9; MDM2 is an E3 ubiquitin ligase, which negatively regulates the p53 tumor suppressor [112, 113]. Under hypoxic conditions, tumor cells become resistant to TRAIL-induced cell apoptosis; this contributes to the up-regulation of anti-apoptotic protein expression and reduction of pro-apoptotic protein expression. Treatment with melatonin blocks hypoxic responses leading to the induction of apoptosis in TRAIL resistance tumor cells by the regulation of mitochondrial transmembrane potential and induction of Bax translocation [114]. Melatonin inhibits cancer cell growth by increasing cell cycle arrest in the G2/M phase, which this effect is coincident with the induction of apoptosis through up-regulating the expression of p53, p21, caspase-3/8/9, PARP, cytochrome c, Bax, JNK 1,-2 and -3 and p38 MAPKs in cancer cells [115]. Melatonin triggers two distinct apoptotic processes including TGFβ1 and caspase-independent early apoptosis and TGFβ1 and caspases-dependent late apoptosis. Early apoptosis is associated with the elevation level of p53/MDM2 ratio and up-regulation of AIF release; this process is independent to caspase activity or cleavage of PARP. Late apoptosis is associated with elevation of caspases-9 and -7 activity and cleaved-PARP level as well as reduction of Bcl-2/Bax ratio [116]. Melatonin also induces apoptosis through simultaneous suppression of COX-2/PGE2, p300/NF-κB, and PI3K/Akt signaling pathway. Inhibition of these pathways leads to the induction of Apaf-1 expression triggering cytochrome c release, and caspase-3 and -9 activation and cleavage [32].

Melatonin induces dephosphorylation and nuclear import of histone deacetylase 4 (HDAC4) in cancer cells; melatonin exerts this effect through inactivation of Ca²⁺/calmodulin-dependent protein kinase II alpha (CaMKIIα), leading to the H3 acetylation on Bcl-2 promoter and subsequent reduction of Bcl-2 expression [117]. Furthermore, inhibition of HDAC9 expression is a mechanism of melatonin to promote apoptosis in non-small cell lung cancer; the increased level of HDAC9 in patients with non-small cell lung cancer is correlated with worse overall survival and poor prognosis [118]. Melatonin promotes TNF-α-mediated apoptosis via inhibiting mitophagy in tumor cells. Since activation of mitophagy suppresses mitochondrial apoptosis, inhibition of mitophagy by melatonin results in the repression of mitochondrial potential, elevation of ROS generation, augmentation of mPTP opening rate and up-regulation of cytochrome c expression and caspases activity. Melatonin inhibits autophagy in tumor cells through inhibiting CaMKII activity leading to the suppression of Parkin expression [119]. In diethylnitrosamine (DEN)-induced hepatocellular carcinoma (HCC), melatonin increases therapeutic potential of mesenchymal stem cells (MSCs) through reduction of oxidative stress and inflammation, and induction of apoptosis [120].

Melatonin has been reported to increase therapeutic potential of anti-cancer agents, which this effect may result from its stimulatory effect on apoptosis. Co-treatment of melatonin and pterostilbene in colorectal cancer cells synergically enhances ROS production and apoptosis. Combination of these two agents upregulates the mRNA level of miR-25-5p, which this results in the activation of PARP and sex-determining region Y-Box10 (SOX10), and attenuation of Bcl-xL, neural precursor cell expressed developmentally downregulated protein 9 (NEDD9), and SOX9 expressions [121]. Melatonin synergically enhances anticancer potential of cisplatin through inducing apoptosis; melatonin increases the effect of cisplatin to the inhibition of ERK phosphorylation and induction of 90-kDa ribosomal S6 kinase (p90RSK) and heat shock protein 27 (HSP27) dephosphorylation [122]. Treatment with melatonin enhances ER stress–mediated apoptosis in tunicamycin-treated cancer cells; this effect is associated with the down-regulation of COX-2 and Bcl-2 expressions and up-regulation of Bim, CHOP and Bax expressions [73]. Melatonin inhibits tunicamycin-induced COX-2 activation in tumor cells through inhibiting NF-ĸB and p38 MAPK activation and p65 nuclear translocation [123]. Combination of melatonin with phenylarsine oxide also induces endoplasmic reticulum stress-induced cell death, accompanied by JNK activation, PARP cleavage, ROS generation and caspase-3 activation [124].