Serotonin receptors in depression: from A to B

Introduction

The serotonin hypothesis of depression has dominated the field of depression for over four decades¹. This theory is centered on the idea that reduced serotonin signaling is a risk factor in the etiology and/or pathophysiology of major depressive disorder (MDD)². However, the most robust body of evidence for the role of serotonin in depression is the efficacy of increasing extracellular serotonin for the treatment of depression. The discovery that the efficacy of tricyclic antidepressants (TCAs) and monoamine oxidase inhibitor (MAOI) antidepressants was largely due to their serotonergic actions, which prompted the use of serotonin selective reuptake inhibitors (SSRIs), the first among them fluoxetine, to treat depression^3–6. These drugs act at the serotonin transporter (5-HTT, also known as SERT) and cause increases in extracellular serotonin, which is the purported mechanism of action^6–8. Many subsequent drugs inhibiting serotonin reuptake have shown behavioral efficacy as antidepressant drugs, suggesting that increasing synaptic serotonin levels may lead to the treatment of depression^6,9.

Despite the relative success in treating depression by increasing extracellular serotonin, there is a lack of strong evidence supporting a direct correlation between low serotonin signaling and depression. While some studies report an association between levels of platelet serotonin and depression, this has not been a consistent finding in large sample sets, and it is also unclear how platelet levels are related to brain levels of serotonin^10,11. Additionally, few studies report direct correlations between cerebrospinal fluid 5-hydroxyindoleacetic acid (5-HIAA), a serotonin metabolite, and depression^12,13. Low levels of tryptophan have been consistently linked to depression; however, these effects could be independent of serotonin^14,15. The lack of consistent clear-cut abnormalities in global measures of serotonin signaling isn’t surprising if one considers the complexity of the receptors at which serotonin binds, the intricate neuroanatomical circuitry of the serotonin system, and the developmental role serotonin plays as a neurotrophic factor^16–18. Many recent studies have focused on understanding the mechanisms through which serotonin affects depression by studying the impact of 5-HTT and the 15 known receptors through gene-association studies, human brain imaging, and pharmacological and genetic mouse models¹⁹.

The success in treating depression by targeting the transporter with SSRIs prompted investigations into whether variability in 5-HTT expression levels could be involved in the etiology of depression. A highly cited study showed that there is an association between a polymorphism in the serotonin transporter (5-HTTLPR) and susceptibility to developing depression²⁰. This and other studies have shown that the short “s” allele, which results in lower levels of 5-HTT expression (at least in vitro) and therefore increased extracellular 5-HT, is associated with a higher risk of depression when combined with stressful life events^21,22. This discovery would be unexpected if developmental considerations were not considered. Although inhibiting the function of the transporter during adulthood decreases depressive symptoms as in the case of SSRIs, reduced expression of 5-HTT during development may increase depressive behavior in adulthood. A human functional magnetic resonance imaging (fMRI) study supports this, showing that short allele carriers show morphological and functional alterations in limbic circuits²³. Additionally, mice lacking 5-HTT throughout life display increased depressive-like behaviors, and pharmacological blockade of 5-HTT in mice exclusively during early postnatal development resulted in increased adult depressive behavior²⁴. These results highlight the differences in developmental versus adult effects of altered serotonin neurotransmission on depression.

In addition to the serotonin transporter, the majority of the 15 serotonin receptors have been implicated in the modulation of depression, depressive-like behaviors, or the response to anti-depressant treatment¹⁹. There are numerous pre-clinical studies which have investigated the role of serotonin receptors using pharmacological manipulations and genetic knockout (KO) models in rodents (Table 1). Given the breadth of this literature, this review will focus on two receptors that are among the most extensively studied for their role in modulating depression, the 5-HT_1A and 5-HT_1B receptor subtypes. In addition, attention will be paid to population-dependent and development-dependent effects of serotonin signaling at these receptors and will draw from both rodent and human studies.

The 5-HT_1A and 5-HT_1B receptors are both inhibitory Gi/o-coupled seven transmembrane receptors that are located throughout the brain^25–27. A major difference between these two receptors is their subcellular distribution²⁸. 5-HT_1A receptors are somatodendritic, while 5-HT_1B receptors are located on axon terminals^27,29. This difference is also reflected in their mechanisms of inhibitory action (Figure 1). Activation of either receptor causes decreased neurotransmitter release; however, 5-HT_1A receptor activation causes hyperpolarization, leading to decreased firing, while 5-HT_1B receptors inhibit voltage-gated calcium channels in the presynaptic terminal^30–32. Another mechanism for 5-HT_1B receptor-mediated inhibition is via effects on 5-HTT, and activation of the 5-HT_1B receptor increases serotonin reuptake^33,34.

Figure 1. Schematic illustrating the inhibitory effects of serotonin (5-hydroxytryptamine, 5-HT) 1A (5-HT_1A) (red) and 5-HT_1B (blue) receptors on the normal firing and neurotransmitter release of a neuron (top).

Activation of 5-HT_1A receptors results in decreased firing (middle), while activation of 5-HT_1B receptors causes decreased neurotransmitter release through actions in the presynaptic terminal (bottom).

Both 5-HT_1A and 5-HT_1B receptors act as autoreceptors located on serotonin neurons and also have heteroreceptor populations located on non-serotonin receptors (Figure 2). Although the mRNA in the raphe (corresponding to autoreceptors) is comparable between the two receptors, their heteroreceptors have distinct patterns of expression³⁵. 5-HT_1A receptors are enriched in the hippocampus and cortex, while 5-HT_1B receptors are highly expressed in the basal ganglia^36,37. These differences in mechanism of action and localization may play a role in the different functional effects of these receptors.

Figure 2. Diagram summarizing the roles of autoreceptor and heteroreceptor populations of serotonin (5-hydroxytryptamine, 5-HT) 1A (5-HT_1A) and 5-HT_1B receptors on behavior during development and adulthood.

5-HTT, serotonin transporter.

While this review focuses on the contribution of 5-HT_1A and 5-HT_1B receptors in depression and depressive-like behaviors, these receptors also modulate other psychiatric-relevant phenotypes. For example, alterations in 5-HT_1A receptor expression influence anxiety behavior, and 5-HT_1B receptor signaling affects reward- and impulsivity-related phenotypes. These receptor-based differences in serotonergic regulation of emotional behavior, which segment into endophenotypes, could contribute to the heterogeneity of symptoms found in MDD³⁸. Understanding the neural circuits that subserve these receptor-based and endophenotype-based differences can help clarify the often confusing and sometimes contradictory findings from various preclinical approaches. From a behavioral perspective, these phenotypes can be segmented through formal unsupervised factor analyses to better divide depressive behaviors into meaningful endophenotypes. Then predictors of the different endophenotypes could be tested by including genetic or pharmacological manipulations.

5-HT_1A and depression

Of the 15 known serotonin receptors, the 5-HT_1A receptor is the most studied for its role in depression³⁹. Quantification of 5-HT_1A receptor levels in humans from post mortem and positron emission tomography (PET) imaging studies reveals an increased level of 5-HT_1A receptors in patients diagnosed with MDD^40–42. Gene association studies have linked a polymorphism in the 5-HT_1A regulatory region (rs6295; G-1019C) with receptor levels in the brain and also to increased risk for depression^43–47. The GG genotype at this single nucleotide polymorphism (SNP) is associated with altered levels of 5-HT_1A receptor expression and reduced responsiveness to antidepressant treatment^43,48. Additionally, clinical studies have revealed antidepressant effects of buspirone and other 5-HT_1A receptor agonists^49,50.

Rodent models have also shown that 5-HT_1A receptor agonists, such as 8-hydroxy-2-(di-n-propylamino) tetralin (8-OH-DPAT), can have acute antidepressant-like effects^51–53. These effects are blocked by 5-HT_1A receptor antagonists, suggesting that the antidepressant-like response is specific to 5-HT_1A receptor signaling⁵⁴. 5-HT_1A heteroreceptors, expressed throughout the limbic system, are the likely site of action for these acute 5-HT_1A receptor-mediated effects^50,55. On the other hand, 5-HT_1A autoreceptors work in opposition to the heteroreceptors, leading to pro-depressive effects. Specifically, activation results in hyperpolarization and reduced firing of raphe neurons, leading to diminished serotonin release in projection regions⁵⁶. Therefore, stimulation of 5-HT_1A autoreceptors from increased extracellular serotonin following SSRI treatment is thought to oppose SSRI actions by downregulating serotonin neuron activity⁵⁷. Over the first few weeks of treatment, these receptors desensitize, which may underlie the delayed behavioral efficacy of SSRIs⁵⁸. Therefore, blocking 5-HT_1A autoreceptor activation has been introduced as an adjunctive therapy to SSRIs. 5-HT_1A receptor partial agonists such as pindolol, and more recently vilazodone, have been shown to be an effective adjunctive therapy to SSRIs in clinical studies^59–62. The development of new agonists that preferentially target subpopulations of 5-HT_1A receptors, for example autoreceptors versus heteroreceptors, potentially through biased agonism, may be useful tools for the treatment of MDD⁶³.

Differences in receptor levels have also been modeled in mice by using genetic loss-of-function models and have allowed causal links between receptor expression levels and depressive-like behavior. 5-HT_1A receptor KO mice have an anti-depressive phenotype^64,65. Tissue-specific KOs have been especially valuable for the dissection of this phenotype and have allowed investigations into the distinct roles of different populations of receptors⁶⁶. The absence of heteroreceptors results in increased depressive-like behavior- as measured in the forced swim test. This mouse model also allowed for temporal control of receptor expression, which revealed a developmental sensitive period for the effect of heteroreceptors on depressive-like behavior. Specifically, knockdown of 5-HT_1A heteroreceptors in adulthood was not sufficient to produce the depressive-like behavior. On the other hand, reduction of autoreceptors in adulthood increased mobility in the forced swim test, suggesting an “anti-depressed” phenotype.

Preclinical studies have also confirmed a causal role for alterations in 5-HT_1A receptor expression in antidepressant efficacy. 5-HT_1A receptor KO mice do not show a behavioral response to fluoxetine⁶⁷. As expected from the pharmacology work, this effect is not mediated by autoreceptors, since reduced expression of 5-HT_1A autoreceptors actually increases the speed and efficacy of SSRI response, requiring only 8 days of fluoxetine treatment to show a behavioral antidepressant-like response⁶⁸. Recent data show that 5-HT_1A heteroreceptors are critical for an effective behavioral response to an SSRI in mice⁶⁹. Genetic or viral deletion of 5-HT_1A receptors specifically in the dentate gyrus of the hippocampus reduced the behavioral response to fluoxetine. Furthermore, expression of 5-HT_1A receptors only in the dentate gyrus was sufficient for normal antidepressant-like responses. These results importantly demonstrate a mechanism for 5-HT_1A-mediated antidepressant effects localized in the mature granule cells of the dentate gyrus.

5-HT_1A and other psychiatric-relevant phenotypes

Anxiety behavior is also modulated strongly by the 5-HT_1A receptor, and, among depressed patients, almost half have a comorbid anxiety disorder⁷⁰. In preclinical studies, 5-HT_1A receptor agonists have anxiolytic effects, and 5-HT_1A receptor KO mice display increased anxiety-like behavior^64,65,71,72. The effect has a developmental sensitive period, since early developmental but not adult rescue of the receptor was sufficient to restore the normal phenotype in the KO⁷³. Consistent with this, early postnatal blockade of 5-HT_1A receptors, through genetic or pharmacological methods, also leads to increased anxiety^74,75. Recent work has shown that the sensitive period is peri-pubertal, and tissue-specific KO mice point to a role for autoreceptors during this period of development^66,76,77.

Other psychiatric disorders have also been linked to the 5-HT_1A receptor, including bipolar disorder and post-traumatic stress disorder^78,79. Additionally, the SNP rs6295 found in the premotor region that is associated with risk for depression is also linked with psychiatric hospitalization, a history of substance abuse, and prior suicide attempts⁴³. Consistent with the studies in depression, the G allele is associated with reduced expression of the 5-HT_1A receptor in the prefrontal cortex and an increased risk for psychiatric outcomes. Interestingly, the effects on receptor expression were also seen in the brain during early human embryonic development, suggesting its potential importance in mediating developmental contributions to adult depression. Finally, there were associations with childhood maltreatment with trends towards significant genotype by environment interactions⁴⁰.

5-HT_1B and depression

While the 5-HT_1B receptor is best known for its role in regulating aggressive and impulsive behavior, it also plays an important role in modulating depression. Activation of the 5-HT_1B receptor decreases serotonin levels in the brain through effects on release, synthesis, and reuptake^33,80,81. In humans, reduced 5-HT_1B receptor function is associated with MDD⁸². Additionally, patients with MDD are less responsive to 5-HT_1B receptor agonists, suggesting reduced expression or desensitization^83,84. This is consistent with clinical studies showing that 5-HT_1B receptor agonists produce antidepressant effects in humans^85–87. This has also been shown in mice, with specific agonists resulting in antidepressant-like behavior^88,89. However, genetic KO of the receptor also results in antidepressant-like behavior, suggesting that this is possibly caused by compensatory effects^90–94.

Both autoreceptor and heteroreceptor populations of 5-HT_1B receptors have been implicated in depressive-like behaviors using rodent models. However, since 5-HT_1B receptors are located on presynaptic terminals, heteroreceptors and autoreceptors have overlapping localization⁹⁵. This rules out brain imaging and pharmacological manipulations in preclinical models as tools to differentiate the role of the two populations of receptors. Therefore, it has been only the recent availability of a tissue-specific genetic mouse model that has allowed the dissection of the role of 5-HT_1B receptors in the regulation of behavior⁹⁶.

Our recent studies show that selective ablation of 5-HT_1B autoreceptors results in decreased depressive-like behaviors in mice⁹⁷. These mice also show increased elevations in serotonin levels compared to controls following SSRI administration, suggesting a potential mechanism of action for the behavioral effects. Specifically, removing the terminal auto-inhibition may result in increased serotonin in projection regions that are relevant to depressive behavior. Furthermore, we also showed that the impact of 5-HT_1B autoreceptors on behavior was not due to developmental expression, since the phenotype was not recapitulated in a mouse with developmental knockdown. These data are consistent with other evidence suggesting a pro-depressive role for the activation of 5-HT_1B autoreceptors^98,99. For example, 5-HT_1B mRNA is elevated in the raphe of rats following stress and in models of depression such as learned helplessness, and viral overexpression of 5-HT_1B receptors in the raphe results in depressive-like behavior following stress¹⁰⁰. In rats, reductions in 5-HT_1B receptor mRNA in the raphe are seen following SSRI treatment in post mortem brains^101,102. This effect isn’t seen in other brain regions such as the cortex, hippocampus, or striatum, suggesting that this effect is specific to autoreceptors. Additionally, another study showed that 5-HT_1B autoreceptors may desensitize following SSRI treatment, similar to 5-HT_1A autoreceptors¹⁰³. Finally, a recent PET study in humans reported that following effective cognitive behavioral therapy for depression, 5-HT_1B receptor binding was reduced in the brainstem¹⁰⁴.

There is evidence which suggests an opposing role for 5-HT_1B heteroreceptors in depressive behaviors. Activation of 5-HT_1B heteroreceptors in a rodent serotonin depletion model (to remove the contribution of autoreceptors) results in an antidepressant-like effect¹⁰⁵. Additionally, reduced expression of 5-HT_1B heteroreceptors in the ventral striatum is associated with depression in humans⁸². Finally, 5-HT_1B receptors located in the ventral striatum have been suggested to interact with p11 (a 5-HT_1B receptor-binding protein) to affect depression-related behaviors^106,107.

5-HT_1B and other psychiatric-related phenotypes

Reward dysfunction is a major symptom of MDD which is mediated, in part, by altered signaling in the mesolimbic reward system^108–112. 5-HT_1B receptors have been implicated in the neural basis of dysregulated reward sensitivity in a number of human studies and preclinical models^113–116, and both 5-HT_1B receptor protein and mRNA are located within the mesolimbic pathway in the nucleus accumbens (NAc) and ventral tegmental area (VTA)⁹⁵. Additionally, activation of 5-HT_1B receptors in the VTA increases dopamine levels in the NAc, potentially via effects on GABAergic signaling in the VTA¹¹⁷.

Many studies linking the receptor to functional deficits in reward processing have focused on addiction. Polymorphisms in the 5-HT_1B receptor gene have also been associated with drug and alcohol abuse^118–120. Additionally, a PET imaging study revealed increased 5-HT_1B receptor binding in pathological gamblers, who have known deficits in reward sensitivity, and gambling disorder is highly comorbid with depression and alcohol and substance use disorders^116,121. Another PET imaging study shows that there is reduced 5-HT_1B receptor binding in cocaine-dependent participants compared to in healthy controls¹²². In preclinical models, 5-HT_1B receptor KO mice are more motivated to self-administer cocaine¹²³. Consistent with this, 5-HT_1B receptor agonists attenuate the motivation for cocaine but paradoxically increase the rewarding effects of cocaine¹²⁴. These effects are mediated by 5-HT_1B receptor expression on medium spiny neurons in the NAc, likely through their projections to the VTA^125,126. Additionally, 5-HT_1B receptors are required for the rewarding properties of social interaction, supporting an impact on general reward systems¹¹⁴.

5-HT_1B receptors are also implicated in impulsive aggression. In humans, polymorphisms in the gene encoding 5-HT_1B receptors have been associated with aggression, suicide, and disorders that include impulsivity as a core phenotype, including attention deficit hyperactivity disorder and substance use disorder^115,120,127. In mice, 5-HT_1B receptor KOs are highly aggressive in tests of male and female aggression and also display increased impulsivity^128–130. Additionally, 5-HT_1B receptor agonists are known as “serenics” because they decrease aggression¹³¹. While the aggressive and impulsive phenotype was originally thought to be modulated by the same underlying circuits, our recent work shows that distinct populations of 5-HT_1B receptors modulate aggression and impulsivity⁹⁶. Furthermore, developmental expression of the 5-HT_1B receptor influences aggression, while adult expression modulates impulsive behavior.

Conclusion

There is a considerable body of research that implicates serotonin in the modulation of depression and depression-related behaviors. The preclinical work delineating the effects of signaling through the 5-HT_1A and 5-HT_1B receptors has been made possible because of careful pharmacological studies as well as the development of transgenic mouse models that have allowed for tissue-specific and inducible knockdown. These studies have highlighted the complexity of serotonin receptors, showing that their role varies through the lifespan and by cell-type population. Additionally, the availability of specific radioligands for PET imaging of these receptors has allowed for the translation of findings from preclinical work to humans. The large number of studies concerning the role of these receptors is partially due to the fact that the 5-HT₁ receptor subtypes were some of the first discovered, and it may be only a matter of time before the roles of more newly discovered receptors are clarified¹⁷.

Despite the amassing of evidence of serotonin receptor-specific involvement in depression, the primary pharmaceutical treatment strategy for depression remains the inhibition of serotonin reuptake. The lack of new treatment options is surprising given the need for them, since current SSRI treatments are ineffective in one-third of patients¹³². Additionally, the majority of patients, as seen in the STAR*D study, don’t respond to administration of the first SSRI treatment, requiring multi-step treatment plans that take months¹³². Furthermore, the considerable differences in treatment outcome also emphasize the heterogeneity of the depressed patient population. A better understanding of receptor signaling and neural circuit mechanisms by which serotonin affects depression may inform the development of novel, more targeted drugs that influence specific receptors, signaling cascades, or time periods. Also, personalized treatment plans could be developed based on symptoms, biomarkers, or pathophysiological presentation.